Turbogears, remember me

September 22nd, 2008

So a while back I implemented a remember me feature for chrss. I said I’d release the code for it and am finally now getting round to it.

Please note that this kind of “remember me” functionality can represent a potentially security hole. It makes sense for some sites where the convenience out weighs any problems that would occur if someone fraudulently gains access to the site. As I wrote this for a site that is concerned with playing chess online it seemed worth it.

So to get started this is meant to work with:

Also note that I’ve left some of the imports as they appear for my app (chrss), so you’ll need to change them as appropriate.

The idea

Conceptually a regular request with a remember me feature works thus:

  • If the user is not logged in, we check for a “remember me” cookie
  • If the cookie is present then we check to see if it matches a token (which maps to a user) in the database
  • If there’s a match to a user we can login the user and on future requests we can ignore the remember me cookie (everything works as before)

The token in the database is randomly generated when the user logs in (with the “remember me” option ticked on the login form) in a similar way to any kind of session tracking cookie. The different is that the token/cookie is meant to hang around for much longer than a regular session. It’s used in addition to Turbogears tg-visit cookie and is just a handy shortcut for logging in a user automatically. This means that it’s fairly non-invasive in so far as it interacts with the Turbogears identity framework.

The code

First of all we need a table in the database to connect the remember me token to a user. So in my models I defined the following entity:


class RememberMe(SQLObject):
    user_token = StringCol(length=40, alternateID=True,
            alternateMethodName="by_user_token")
    user_id = IntCol()
    expiry = DateTimeCol()

    expiry_index=DatabaseIndex(expiry)

The rest of the code then lives in remember_me.py.

First there’s the code to “remember” a user. This creates a RememberMe entity and sets a cookie on the user’s machine:


def generate_token():
    key_string= '%s%s%s%s' % (random.random(), datetime.now(),
                              cherrypy.request.remote_host,
                              cherrypy.request.remote_port)
    return sha.new(key_string).hexdigest()

def remember_user(user):
    from chrss.model import RememberMe

    user_token=generate_token()
    expiry=datetime.now() + timedelta(days=remember_me_age_days)
    remember=RememberMe(user_token=user_token, user_id=user.id,expiry=expiry)

    cookies= cherrypy.response.simple_cookie
    max_age = remember_me_age_days*24*60*60
    cookies[remember_cookie_name] = remember.user_token
    cookies[remember_cookie_name]['path'] = '/'
    cookies[remember_cookie_name]['expires'] = formatdate(time() + max_age)
    cookies[remember_cookie_name]['max-age'] = max_age

Here’s the reverse function to “un-remember” a user (which you would call from your logout method):


def unremember_user(user):
    cookies = cherrypy.request.simple_cookie
    if remember_cookie_name in cookies:
        user_token=cookies[remember_cookie_name].value

        if user_token:
            from chrss.model import RememberMe
            try:
                remember=RememberMe.by_user_token(user_token)
                remember.destroySelf()
            except SQLObjectNotFound:
                pass

            # now clear cookie
            cookies= cherrypy.response.simple_cookie
            cookies[remember_cookie_name] = ''
            cookies[remember_cookie_name]['path'] = '/'
            cookies[remember_cookie_name]['expires'] = 0
            cookies[remember_cookie_name]['max-age'] = 0

Before I get onto the two monkey patches, we need to make one more function, that we use to login the user given a user entity (bypassing the need for their username and password) and is based on code from here:


def login_user(user):
    ''' from http://docs.turbogears.org/1.0/IdentityRecipes'''
    visit_key = turbogears.visit.current().key
    IdentityObject = turbogears.identity.soprovider.SqlObjectIdentity

    from chrss.model import VisitIdentity
    try:
        link = VisitIdentity.by_visit_key(visit_key)
    except SQLObjectNotFound:
        link = None
    if not link:
        link = VisitIdentity(visit_key=visit_key, user_id=user.id)
    else:
        link.user_id = user.id
    user_identity = IdentityObject(visit_key);
    return user_identity

The monkey patches

Now we get to the meat of the code – the bit which does the actual “magic”. In both cases we are monkey-patching methods that belong to the IdentityVisitPlugin class in Turbogears (defined in turbogears.identity.visitor).

First up is identity_from_visit which normally just looks for the tg-visit cookie and then sees if that’s associated with a user login or not. We shall effectively override it, so that if no association is found then we will perform a further check to see if there is a remember me cookie that will let us log the user in:


# keep a reference to the original function
old_identity_from_visit=turbogears.identity.visitor.IdentityVisitPlugin.identity_from_visit

def identity_from_remember_me( self, visit_key ):
    identity=old_identity_from_visit( self, visit_key )
    if identity.anonymous:
        # not logged in so check for remember me cookie
        cookies = cherrypy.request.simple_cookie
        if remember_cookie_name in cookies:
            log.info("checking remember me cookie")
            user_token=cookies[remember_cookie_name].value

            from chrss.model import RememberMe, User
            try:
                remember=RememberMe.by_user_token(user_token)
                user=User.get(remember.user_id)
                return login_user(user)
            except SQLObjectNotFound:
                pass

    return identity

# monkey-patch the method
turbogears.identity.visitor.IdentityVisitPlugin.identity_from_visit=identity_from_remember_me

The next method we patch is identity_from_form. For this we just check whether there is a “remember_me” parameter in the request after a successful login (from calling the original method) and if so call the remember_user() function.


old_identity_from_form=turbogears.identity.visitor.IdentityVisitPlugin.identity_from_form

def identity_from_form(self, visit_key):
    identity=old_identity_from_form(self, visit_key)
    if identity is not None and not identity.anonymous:
        # login worked, so now see if 'remember me' set
        params=cherrypy.request.params
        remember_me=params.pop('remember_me', None)
        if remember_me:
            remember_user(identity.user)
    return identity

turbogears.identity.visitor.IdentityVisitPlugin.identity_from_form=identity_from_form

You’ll just import the remember_me module early on in starting up your Turbogears app and it will apply these monkey patches. Then if you modify your login template to include a “remember_me” checkbox you should have everything working.

As I said before it’s fairly non-invasive (as far as monkey patches go), so there shouldn’t really be a need to modify much beyond your login form and to add a call to unremember_user to your logout code. The only other thing is perhaps to setup a cron-script or other background task to delete expired entries in the database (which is why the RememberMe entity has an expiry column).

Source code

The remember_me module is available for download here.

A turbogears caching decorator

May 31st, 2008

A while back I wrote a caching decorator for chrss. It’s mostly used for the rss feeds, to help avoid having over-zealous rss readers slowing the site down. However I’m also now running it on a few other pages that were a bit slow (notably generating PGN files for games).

After letting it sit for a while Ian and Kyran also started using it on ShowMeDo. That was a couple of months ago. So now that I can be fairly certain it works it seemed like time to share it with the world.

So first off here’s a few features/comments:

  • It’s shamelessly based on code from Django (the caching backends at least)
  • It features an “anti-dogpiling” mechanism to try and make sure only one thread triggers a cache refresh
  • Multiple backends supported:
    • dummy – does no caching (for testing/development use)
    • simple – just uses a dictionary (for testing/development use)
    • localmem – thread-safe cache for use in process
    • file – uses file-system to cache data (this is what’s used with chrss and ShowMeDo)
    • memcache – uses memcached for caching (it should work, but not massively tested at the moment)

Now for some example usage:


from turbogears import expose, controllers
from cache import cache_result

class MyController(controllers.Controller):

    @expose(content_type="text/plain")
    @cache_result()
    def cache_some_text(self):
        ''' no template so it's pretty straightforward - expose just has to come first '''
        return 'this will be cached'

    @expose(template="my_template)
    @cache_result()
    def cache_data_only(self):
        ''' with a template we can just cache the data and not rendered html '''
        return dict(value='this dictionary will be cached')

    @expose()
    @cache_result()
    @expose(template="my_template)
    def cache_html(self):
        '''
        or we can cache the rendered html, but we have to use an outer expose()
        to make the method public
        '''
        return dict(value='will be cached with the html')

To see how you can use the @cache_result() you are probably best looking
at the source (there’s a fairly detailed comment explaining it). The following parameters can
be passed in:

  • key_fn – function used to derive a key to store the data in (default uses current url and user identity)
  • version_fn – can be used to control how a cached value expires (defaults to a function that returns the same value everytime)
  • timeout_seconds – how many seconds until the value start to expire

The default key function can be controlled via the config parameter cache.decorator.anon_only. If set to True (the default) it will only look in the cache for data when users are not logged in. Otherwise when users are logged in it will use a key just for them. The default is handy if you just want to avoid problems with a flood of anonymous users (e.g. from Slashdot/Digg etc).

The version function can be used to force expire cached values. The value of the version function is compared to the value stored in the cache and if different this triggers a cache refresh. For example if the version function was based on the number of comments in a blog post, then whenever a comment was added to the blog post the cache would get refreshed. This avoids having to wait for the cache to expire.

timeout_seconds specifies how many seconds before a value expires. It defaults to the value set in cache.decorator.timeout_seconds in your config file (or 1800 seconds if not set there).

anti-dogpiling

So first I’ll explain what I mean by “dogpiling” with respect to cache expiry.

The standard way to use a cache is to do something like:


value = cache.get('key', None)
if value is None:
    value = recompute_cached_value()
    cache['key'] = value
return value

Now this is fine normally. When the cached value expires the next request will simply call recompute_cached_value() and the cache will be updated for future requests.

The trouble arises when recompute_cached_value() takes a long time to run and you have have a lot of other requests running at the same time. If a request is still recalculating the value and another request comes along, then that will also attempt to recalculate the value. This will in turn probably slow down the calculation going on, making it more likely that the next request to arrive will also trigger a recalculation and so on. Very quickly you can end up with tens/hundreds/thousands of request all attempting to recalculate the cached value and you have lost most of the advantage of caching in the first place.

So to handle this situation more gracefully this caching decorator employs a two stage expiry.

First there is a hard cut off expiry that works like normal. This is set to occur later than the other expiry time and is the value that would be fed to memcache or equivalent.

The second expiry time set is the one normally used. Basically when we store/retrieve the cached data we also have access to this expiry time (and the version). If we see that we need to recalculate the value (due to the expiry time being in the past or the version being different), then we attempt to grab a lock to recalculate the value. If we don’t grab the lock, we assume another thread is doing the recalculation and rather than wait around we simply serve up the old (stale) data. This should mean that one thread (potentially per-process) will end up doing the recalculation rather than several.

This also means that we don’t have to remove a value from the cache to force a refresh (which might cause dogpiling). Instead we can update whatever value we use in our version function, to trigger a graceful refresh.

conclusion

So that’s a basic intro to this caching decorator. It’s quite a handy quick way of adding some caching to your turbogears app. You’ll need to see how well it works for you. I’m providing it “as is” and making no claims about anything. Feel free to incorporate it into your code and modify as you see fit. Just let me know if you have any issues or feedback.

bonus decorator

The cache code also includes a simple decorator to control the Expires header sent out with a response:


def expires(seconds=0):
    '''set expire headers for client-size caching'''
    def decorator(fn):
        def decorated(*arg,**kw):
            cherrypy.response.headers['Expires']=formatdate(_current_time()+seconds)
            return fn(*arg,**kw)
        return decorated
    return decorator

It’s handy for getting the client to cache some data for us too. I use it on some of the PIL generated images served up via my app.

source code

Download turbogears caching decorator

The source for the decorator(s) includes a simple test suite (to be run using nose).

chrss (chess by rss) update 22

December 6th, 2007

Another update for chrss. No major changes to the chess part itself, but I’ve just added a blog so I’ve got a dedicated place to discuss chrss.

This marks the start of me adding some actual “content” to chrss. Up until now chrss (as a site) has basically consisted of a front page and a bunch of chess games! It’s been extremely functional, but I felt it was time for chrss to grow up a bit. It might also help me attract a few more visitors, as currently there’s not exactly much for the search engines to search on!

chrss (chess by rss) update 21

November 26th, 2007

Another update to chrss.

  • tweaked front page layout to make it clearer and show different options depending on if you are logged in or not
  • addition of a help section (with backend CMS for myself to add content to it).

At the moment the help section is not directly linked from the front page of chrss. I need to get some more content in there and figure out where best to link it.

When developing in your spare time automated tests are your best friend

November 22nd, 2007

Well I just finished fixing a bug in chrss and it’s made me very glad that I’ve been using automated tests.

The bug looked like it might be very tricky to track down and I feared that it might get messy. In the end it proved to be a slight edge case I’d not catered for and was pretty easy to fix.

First I quickly tracked down the stack-trace in my logs:


  File "/home2/lilspikey/webapps/chrss2/chrss/controllers/game.py", line 226, in move
    game.make_move(game.turn,move)
  File "/home2/lilspikey/webapps/chrss2/chrss/model.py", line 239, in make_move
    self._record_move(chess_game,color,move)
  File "/home2/lilspikey/webapps/chrss2/chrss/model.py", line 202, in _record_move
    incheck=chess_game.in_mate()
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 213, in in_mate
    return self._board.calc_mate(self.game_state)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 915, in calc_mate
    can_move=self.any_legal_moves(game_state)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 990, in any_legal_moves
    legal=self.legal_moves(game_state,pos)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 984, in legal_moves
    return self._filter_check_moves(game_state,moves)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1029, in _filter_check_moves
    board.move_piece(move) # make the move
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1103, in move_piece
    moved_piece,taken_piece,castled,enpassant,updated_positions=self._calc_move(move)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1081, in _calc_move
    raise ValueError("illegal castling move")
ValueError: illegal castling move

Next step was to create a unit test to replay the moves from the game with the problem. Once I’d got it failing in the same way, I discovered a bug in my logic that meant rooks captured without moving would not update the “castling status” for that side. This led to falsely generating castling moves that weren’t possible and thus the failure in the stack trace (some defensive coding to stop this kind of thing). In this game in particular white’s queen-side rook had been captured without moving and my chess module was reporting that the king could still castle queen-side!

After getting it to work I now had a nice automated regression test for that bug. I also took the time to add a unit test for the actual function that I changed.

Brilliant stuff. Now I can be extra certain that bug is fixed and will stay fixed, as those tests will get run every time I run my test suite. I’m working on chrss in my spare time, so this is really important to me. I really don’t have time to spend manually testing and verifying that _everything_ works.

The more testing I can automate the less testing I have to do. Which means I’ve got more time for adding features or else playing games of chess!

chrss (chess by rss) update 20

November 6th, 2007
  • Updated the user registration process, so hopefully it’s a bit friendlier
  • Users now have a chance to resend their activation email if they have trouble activating their account
  • Tweaked filters for games on user’s page:
    • “active” – games that it’s your turn in or have had a move in the last 30 days
    • “waiting” – un-finished games that aren’t active
    • “finished”
    • “all”

Also improved the code coverage some more.

On a side note there is a preliminary mobile version of chrss available now. However I wouldn’t recommend it for regular use, as making a move currently involves a _huge_ drop down menu. Less than ideal. Still it can be handy for checking whether anyone has move. I’ve got some plans on improving the usability in the future, so check back later to see how it’s doing.

Turbogears and mobile mini-site logins

October 12th, 2007

I’m in the process of to writing a simple mobile version of chrss (chess by rss). This is spurred by the arrival of my new Nokia 6300 (which comes with Opera Mini). I’m aiming to end up with a mini-site running at http://chrss.co.uk/mob/ or similar.

The plan is to keep it _very_ simple. Main page lets you login, then lists your active games (showing which ones it’s your turn to move etc.). From there you’ll be able to get to a game, view the board and make a move. That’ll be it. Dead simple.

Turbogears works quite well for this in some respects – I can encapsulate the mini-site as a CherryPy controller and keep it fairly self contained. Though there is one minor drawback. That being that the identity framework is set up for the main site and will attempt to use the regular login page.

So how do we re-use the whole identity framework, but force access to the mobile mini-site to go via a different login page?

After a little poking around the TG source…

The skeleton controller for logging in:


class Mobile(controllers.Controller):
    @expose(template="cheetah:chrss.templates.mobile.index")
    @mob_login
    def index(self):
        return dict()

    @expose(template="cheetah:chrss.templates.mobile.login")
    def login(self,user=None,pwd=None):
        if identity.current.anonymous:
            msg=None
            visit_key = turbogears.visit.current().key
            ident=identity.current_provider.validate_identity(user,pwd,visit_key)
            if ident is None:
                msg="login failed"
                return dict(user=user,msg=msg)
        raise redirect('/mob/')

The login method looks up the current visit_key (effectively the value of the cookie TG uses to track visits) then uses identity.current_provider. validate_identity to try and log the user in.

NB: I’m using user and pwd as variables, instead of user_name and password. The identity framework intercepts parameters with those names and uses them to authenticate behind the scenes. Obviously I need those values to get through to my controller method – hence why I’m using different names.

After the call to identity.current_provider. validate_identity I check to see if I have an object returned and either report that the login failed or redirect back to /mob/. Easy.

The next bit is the decorator @mob_login shown on the index controller method. This works a bit like the @identity.require decorator, but isn’t as flexible. It simply forces the user to a mobile login page if they aren’t logged in:


def mob_login(fn):
    def decorated(*arg,**kw):
        if identity.current.anonymous:
            return dict(tg_template="cheetah:chrss.templates.mobile.login",user='',msg=None)
        return fn(*arg,**kw)
    return decorated

Obviously you’ll need a template for the login page. I’m using cheetah and my login page looks like this (there’s a master template not shown here):


#extends chrss.templates.mobile.master
#def body
#if $msg
    <div id="msg">${msg}</div>
#end if
#filter WebSafe
    <h1>chrss: chess by rss</h1>
    <form action="$tg.url('/mob/login')" method="POST">
         <label for="user">user name:</label>
         <input type="text" id="user" name="user" value="$user"/>
         <label for="pwd">password:</label>
         <input type="password" id="pwd" name="pwd"/>
         <input type="submit" value="login"/>
    </form>
#end filter
#end def

And that is more or less all I had to do to get a basic mobile/alternative login page working. Hope you found it informative. I’ll report back later on how well actually running it on a mobile device goes…

chrss (chess by rss) update 19

October 12th, 2007
  • Bug fix for PGN (Portable Game Notation) output (as noted by Calibius)
  • Faster PGN generation (evaluating only the legal moves I need to consider to avoid ambiguity, rather than _all_ legal moves)
  • URLs in comments should now automatically appear as click-able links
  • Ability to “rotate” the board, to view it from the other direction (handy when you’re playing black)

chrss (chess by rss) update 18

October 6th, 2007
  • simple bread crumbs added to aid navigation:

    breadcrumbs.gif

  • highlighting games that it’s your turn to move in:

    yourturn.gif

  • PGN (portable game notation) output for games:

    pgn.gif

  • improved test coverage

chrss (chess by rss) update 17

September 18th, 2007

This update adds two features:

  • you can now offer a draw to an opponent
  • comments are now included in rss feeds for games and users

The first new feature was needed so I could finish this game. A bit like when I had to add the ability to resign a game. That’s probably the last major missing feature added. I could add code for detecting threefold repetition or the fifty move rule, but that seemed overkill for what I have in mind for chrss. After all it’s meant to be for playing friendly games of correspondence chess – so agreeing whether to draw a game should not be a problem.

I also figured that adding comments to the rss feeds might be nice. With a game like chess, moves can be few and far between, so making the rss feeds a bit more active might be quite good. It also means you can nudge your opponent – as leaving a comment will show up in their feed reader and might well serve to remind them it’s their turn to play!

I’ve still got PGN in mind for my next feature. I imagine I’ll need to spend a bit of time to make sure that the exported PGN files will work with other chess programs.