Category Archives: Chrss

Porting chrss from Turbogears 1.0 to Django 1.3

For the a period of 18 months or so I slowly ported my chess by rss site (chrss) from Turbogears 1.0 to Django. When I started the port Django 1.1 was the latest version. I took so long to finish the port, that Django got to version 1.3 in the meantime! The slowness was mostly due to my then pending and now actual fatherhood. However by slowly toiling away, with the aid of quite a few automated tests I managed to deploy a Django version of the chrss the other week a couple few of months ago.

Python Framework to Python Framework

The good thing about moving from Turbogears to Python was that it’s all Python code. This meant that things like the core chess code didn’t need touching at all. It also meant that a lot of fairly standard code could be almost directly ported. Methods on controller objects in Turbogears became view functions in Django. Methods on model objects stayed methods and so on. A lot of the porting was fairly mechanistic. I moved all of the Turbogears code into a folder for easy referral and then built the Django version from nothing back up. Initially most of the work was done at the model/db level where I could copy over the code, convert it to Django style and then copy over and update the automated tests. I used Django Coverage to make sure the code was still all actually getting tested.

I could have opted to exactly match the database from the Turbogears version, but opted instead to make it a more Django like. This meant using the regular Django user models and so on. As Turbogears 1.0 involved a custom user model I had to create a separate user profile model in the Django port. There were a few other changes along these lines, but most of the porting was not so structural.

A lot of the hard work during porting came from missing bits of functionality that had far reaching effects. Testing was very awkward until a lot of the code had been ported already.

Cheetah templates to Django templates

Chrss used Cheetah for templates. Cheetah is not as restrictive with it’s templates as Django. It’s very easy to add lots of logic and code in there. Some pages in chrss had quite a bit of logic – in particular the main chess board page. This made porting rather tricky with Django. I had to put a lot of code into template tags and filters and carefully re-organise things. Porting the templates was probably the hardest part. Automated tests helped a bit with this, but a lot of the issues needed visual verification to ensure things really were working as they should.

One advantage of going from Cheetah to Django’s tempate language was the incompatible syntax. This meant I could simply leave bit’s of Cheetah code in a template and it would serve as quite a good visual reminder of something that was yet to be ported.

The second 90%

A good portion of the site was ported before my son’s birth. It seemed like maybe it wouldn’t take much longer, as it felt like 90% of the work was done. Of course it turned out there was another 90% yet to finish.

Beyond the usual tweaking and finishing of various odds and ends, the remaining work consisted of:

  • Completing the openid integration
  • Migrating the database

For the open id integration I opted to use Simon Willison’s Django OpenID app – hoping to be able to have a nice drop-in app. Possibly due to the slightly unfinished nature of the app and mostly due to my desire to customise the urls and general flow of the login/register process this took a fair while. It might have been quicker directly porting the original OpenID code I used with Turbogears, but it did work out ok in the end.

Of course it turns out that after all that hard work, that hardly anyone seems to use OpenID to login to sites anymore. I might have been better off integrating Django Social Auth instead, for Twitter and/or Facebook logings. However I decided that this would have been too much like feature creep and stuck with the original plan.

The chrss database isn’t very large. The table recording the moves for each game is currently over 70,000 rows, but each row is very small. So the database dump when tar gzipped is less than 3Mb in size. This gave me plenty of options for database migration. I’d roughly matched the schema used for the Turbogears site, but did take the opportunity to break from the past slightly. I created a Python script that used talked to the MySQL database and generated an sql dump in the new format. By using Python I was free to do any slightly more complex database manipulation I needed. The only real tricky part was converting to using Django’s user models.

One wrinkle with the database migrating was a bit disturbing. When I setup the app on webfaction I found some very odd behaviour. Logging in seemed to work, but after a couple of page requests you’d be logged out. The guys at webfaction were very helpful, but we were unable to get to the bottom of the problem. During testing I found that this problem did not happen with SQLite or Postgres, so was something to do with MySQL. This was one of those times when using an ORM paid off massively. Apart from the database migration script no code needed to be touched. If I’d had more time I might have persevered with MySQL, but Postgres has been working very well for me.

Conclusion

Chrss has been running using Django and Postgres for nearly eleven months now and has been working very well. I’ve had to fix a few things now and then, but doing this in Django has been very easy. I was also able to automate deployment using Fabric, so new code could be put live with the minimum of fuss. When you’ve only got a limited time to work with, automated deployment makes a huge difference.

Hopefully now that sleep routines are better established and my own sleep is less interrupted I’ll have a chance to add new features to chrss soon.

Turbogears, remember me

So a while back I implemented a remember me feature for chrss. I said I’d release the code for it and am finally now getting round to it.

Please note that this kind of “remember me” functionality can represent a potentially security hole. It makes sense for some sites where the convenience out weighs any problems that would occur if someone fraudulently gains access to the site. As I wrote this for a site that is concerned with playing chess online it seemed worth it.

So to get started this is meant to work with:

Also note that I’ve left some of the imports as they appear for my app (chrss), so you’ll need to change them as appropriate.

The idea

Conceptually a regular request with a remember me feature works thus:

  • If the user is not logged in, we check for a “remember me” cookie
  • If the cookie is present then we check to see if it matches a token (which maps to a user) in the database
  • If there’s a match to a user we can login the user and on future requests we can ignore the remember me cookie (everything works as before)

The token in the database is randomly generated when the user logs in (with the “remember me” option ticked on the login form) in a similar way to any kind of session tracking cookie. The different is that the token/cookie is meant to hang around for much longer than a regular session. It’s used in addition to Turbogears tg-visit cookie and is just a handy shortcut for logging in a user automatically. This means that it’s fairly non-invasive in so far as it interacts with the Turbogears identity framework.

The code

First of all we need a table in the database to connect the remember me token to a user. So in my models I defined the following entity:


class RememberMe(SQLObject):
    user_token = StringCol(length=40, alternateID=True,
            alternateMethodName="by_user_token")
    user_id = IntCol()
    expiry = DateTimeCol()
    
    expiry_index=DatabaseIndex(expiry)

The rest of the code then lives in remember_me.py.

First there’s the code to “remember” a user. This creates a RememberMe entity and sets a cookie on the user’s machine:


def generate_token():
    key_string= '%s%s%s%s' % (random.random(), datetime.now(),
                              cherrypy.request.remote_host,
                              cherrypy.request.remote_port)
    return sha.new(key_string).hexdigest()

def remember_user(user):
    from chrss.model import RememberMe
    
    user_token=generate_token()
    expiry=datetime.now() + timedelta(days=remember_me_age_days)
    remember=RememberMe(user_token=user_token, user_id=user.id,expiry=expiry)
    
    cookies= cherrypy.response.simple_cookie
    max_age = remember_me_age_days*24*60*60
    cookies[remember_cookie_name] = remember.user_token
    cookies[remember_cookie_name]['path'] = '/'
    cookies[remember_cookie_name]['expires'] = formatdate(time() + max_age)
    cookies[remember_cookie_name]['max-age'] = max_age

Here’s the reverse function to “un-remember” a user (which you would call from your logout method):


def unremember_user(user):
    cookies = cherrypy.request.simple_cookie
    if remember_cookie_name in cookies:
        user_token=cookies[remember_cookie_name].value
        
        if user_token:
            from chrss.model import RememberMe
            try:
                remember=RememberMe.by_user_token(user_token)
                remember.destroySelf()
            except SQLObjectNotFound:
                pass
            
            # now clear cookie
            cookies= cherrypy.response.simple_cookie
            cookies[remember_cookie_name] = ''
            cookies[remember_cookie_name]['path'] = '/'
            cookies[remember_cookie_name]['expires'] = 0
            cookies[remember_cookie_name]['max-age'] = 0

Before I get onto the two monkey patches, we need to make one more function, that we use to login the user given a user entity (bypassing the need for their username and password) and is based on code from here:


def login_user(user):
    ''' from http://docs.turbogears.org/1.0/IdentityRecipes'''
    visit_key = turbogears.visit.current().key
    IdentityObject = turbogears.identity.soprovider.SqlObjectIdentity
    
    from chrss.model import VisitIdentity
    try:
        link = VisitIdentity.by_visit_key(visit_key)
    except SQLObjectNotFound:
        link = None
    if not link:
        link = VisitIdentity(visit_key=visit_key, user_id=user.id)
    else:
        link.user_id = user.id
    user_identity = IdentityObject(visit_key);
    return user_identity

The monkey patches

Now we get to the meat of the code – the bit which does the actual “magic”. In both cases we are monkey-patching methods that belong to the IdentityVisitPlugin class in Turbogears (defined in turbogears.identity.visitor).

First up is identity_from_visit which normally just looks for the tg-visit cookie and then sees if that’s associated with a user login or not. We shall effectively override it, so that if no association is found then we will perform a further check to see if there is a remember me cookie that will let us log the user in:


# keep a reference to the original function
old_identity_from_visit=turbogears.identity.visitor.IdentityVisitPlugin.identity_from_visit

def identity_from_remember_me( self, visit_key ):
    identity=old_identity_from_visit( self, visit_key )
    if identity.anonymous:
        # not logged in so check for remember me cookie
        cookies = cherrypy.request.simple_cookie
        if remember_cookie_name in cookies:
            log.info("checking remember me cookie")
            user_token=cookies[remember_cookie_name].value
            
            from chrss.model import RememberMe, User
            try:
                remember=RememberMe.by_user_token(user_token)
                user=User.get(remember.user_id)
                return login_user(user)
            except SQLObjectNotFound:
                pass
            
    return identity

# monkey-patch the method
turbogears.identity.visitor.IdentityVisitPlugin.identity_from_visit=identity_from_remember_me

The next method we patch is identity_from_form. For this we just check whether there is a “remember_me” parameter in the request after a successful login (from calling the original method) and if so call the remember_user() function.


old_identity_from_form=turbogears.identity.visitor.IdentityVisitPlugin.identity_from_form

def identity_from_form(self, visit_key):
    identity=old_identity_from_form(self, visit_key)
    if identity is not None and not identity.anonymous:
        # login worked, so now see if 'remember me' set
        params=cherrypy.request.params
        remember_me=params.pop('remember_me', None)
        if remember_me:
            remember_user(identity.user)
    return identity

turbogears.identity.visitor.IdentityVisitPlugin.identity_from_form=identity_from_form

You’ll just import the remember_me module early on in starting up your Turbogears app and it will apply these monkey patches. Then if you modify your login template to include a “remember_me” checkbox you should have everything working.

As I said before it’s fairly non-invasive (as far as monkey patches go), so there shouldn’t really be a need to modify much beyond your login form and to add a call to unremember_user to your logout code. The only other thing is perhaps to setup a cron-script or other background task to delete expired entries in the database (which is why the RememberMe entity has an expiry column).

Source code

The remember_me module is available for download here.

A turbogears caching decorator

A while back I wrote a caching decorator for chrss. It’s mostly used for the rss feeds, to help avoid having over-zealous rss readers slowing the site down. However I’m also now running it on a few other pages that were a bit slow (notably generating PGN files for games).

After letting it sit for a while Ian and Kyran also started using it on ShowMeDo. That was a couple of months ago. So now that I can be fairly certain it works it seemed like time to share it with the world.

So first off here’s a few features/comments:

  • It’s shamelessly based on code from Django (the caching backends at least)
  • It features an “anti-dogpiling” mechanism to try and make sure only one thread triggers a cache refresh
  • Multiple backends supported:
    • dummy – does no caching (for testing/development use)
    • simple – just uses a dictionary (for testing/development use)
    • localmem – thread-safe cache for use in process
    • file – uses file-system to cache data (this is what’s used with chrss and ShowMeDo)
    • memcache – uses memcached for caching (it should work, but not massively tested at the moment)

Now for some example usage:


from turbogears import expose, controllers
from cache import cache_result

class MyController(controllers.Controller):

    @expose(content_type="text/plain")
    @cache_result()
    def cache_some_text(self):
        ''' no template so it's pretty straightforward - expose just has to come first '''
        return 'this will be cached'

    @expose(template="my_template)
    @cache_result()
    def cache_data_only(self):
        ''' with a template we can just cache the data and not rendered html '''
        return dict(value='this dictionary will be cached')

    @expose()
    @cache_result()
    @expose(template="my_template)
    def cache_html(self):
        ''' 
        or we can cache the rendered html, but we have to use an outer expose()
        to make the method public
        '''
        return dict(value='will be cached with the html')

To see how you can use the @cache_result() you are probably best looking
at the source (there’s a fairly detailed comment explaining it). The following parameters can
be passed in:

  • key_fn – function used to derive a key to store the data in (default uses current url and user identity)
  • version_fn – can be used to control how a cached value expires (defaults to a function that returns the same value everytime)
  • timeout_seconds – how many seconds until the value start to expire

The default key function can be controlled via the config parameter cache.decorator.anon_only. If set to True (the default) it will only look in the cache for data when users are not logged in. Otherwise when users are logged in it will use a key just for them. The default is handy if you just want to avoid problems with a flood of anonymous users (e.g. from Slashdot/Digg etc).

The version function can be used to force expire cached values. The value of the version function is compared to the value stored in the cache and if different this triggers a cache refresh. For example if the version function was based on the number of comments in a blog post, then whenever a comment was added to the blog post the cache would get refreshed. This avoids having to wait for the cache to expire.

timeout_seconds specifies how many seconds before a value expires. It defaults to the value set in cache.decorator.timeout_seconds in your config file (or 1800 seconds if not set there).

anti-dogpiling

So first I’ll explain what I mean by “dogpiling” with respect to cache expiry.

The standard way to use a cache is to do something like:


value = cache.get('key', None)
if value is None:
    value = recompute_cached_value()
    cache['key'] = value
return value

Now this is fine normally. When the cached value expires the next request will simply call recompute_cached_value() and the cache will be updated for future requests.

The trouble arises when recompute_cached_value() takes a long time to run and you have have a lot of other requests running at the same time. If a request is still recalculating the value and another request comes along, then that will also attempt to recalculate the value. This will in turn probably slow down the calculation going on, making it more likely that the next request to arrive will also trigger a recalculation and so on. Very quickly you can end up with tens/hundreds/thousands of request all attempting to recalculate the cached value and you have lost most of the advantage of caching in the first place.

So to handle this situation more gracefully this caching decorator employs a two stage expiry.

First there is a hard cut off expiry that works like normal. This is set to occur later than the other expiry time and is the value that would be fed to memcache or equivalent.

The second expiry time set is the one normally used. Basically when we store/retrieve the cached data we also have access to this expiry time (and the version). If we see that we need to recalculate the value (due to the expiry time being in the past or the version being different), then we attempt to grab a lock to recalculate the value. If we don’t grab the lock, we assume another thread is doing the recalculation and rather than wait around we simply serve up the old (stale) data. This should mean that one thread (potentially per-process) will end up doing the recalculation rather than several.

This also means that we don’t have to remove a value from the cache to force a refresh (which might cause dogpiling). Instead we can update whatever value we use in our version function, to trigger a graceful refresh.

conclusion

So that’s a basic intro to this caching decorator. It’s quite a handy quick way of adding some caching to your turbogears app. You’ll need to see how well it works for you. I’m providing it “as is” and making no claims about anything. Feel free to incorporate it into your code and modify as you see fit. Just let me know if you have any issues or feedback.

bonus decorator

The cache code also includes a simple decorator to control the Expires header sent out with a response:


def expires(seconds=0):
    '''set expire headers for client-size caching'''
    def decorator(fn):
        def decorated(*arg,**kw):
            cherrypy.response.headers['Expires']=formatdate(_current_time()+seconds)
            return fn(*arg,**kw)
        return decorated
    return decorator

It’s handy for getting the client to cache some data for us too. I use it on some of the PIL generated images served up via my app.

source code

Download turbogears caching decorator

The source for the decorator(s) includes a simple test suite (to be run using nose).

chrss (chess by rss) update 22

Another update for chrss. No major changes to the chess part itself, but I’ve just added a blog so I’ve got a dedicated place to discuss chrss.

This marks the start of me adding some actual “content” to chrss. Up until now chrss (as a site) has basically consisted of a front page and a bunch of chess games! It’s been extremely functional, but I felt it was time for chrss to grow up a bit. It might also help me attract a few more visitors, as currently there’s not exactly much for the search engines to search on!

When developing in your spare time automated tests are your best friend

Well I just finished fixing a bug in chrss and it’s made me very glad that I’ve been using automated tests.

The bug looked like it might be very tricky to track down and I feared that it might get messy. In the end it proved to be a slight edge case I’d not catered for and was pretty easy to fix.

First I quickly tracked down the stack-trace in my logs:


  File "/home2/lilspikey/webapps/chrss2/chrss/controllers/game.py", line 226, in move
    game.make_move(game.turn,move)
  File "/home2/lilspikey/webapps/chrss2/chrss/model.py", line 239, in make_move
    self._record_move(chess_game,color,move)
  File "/home2/lilspikey/webapps/chrss2/chrss/model.py", line 202, in _record_move
    incheck=chess_game.in_mate()
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 213, in in_mate
    return self._board.calc_mate(self.game_state)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 915, in calc_mate
    can_move=self.any_legal_moves(game_state)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 990, in any_legal_moves
    legal=self.legal_moves(game_state,pos)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 984, in legal_moves
    return self._filter_check_moves(game_state,moves)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1029, in _filter_check_moves
    board.move_piece(move) # make the move
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1103, in move_piece
    moved_piece,taken_piece,castled,enpassant,updated_positions=self._calc_move(move)
  File "/home2/lilspikey/webapps/chrss2/chrss/chess2.py", line 1081, in _calc_move
    raise ValueError("illegal castling move")
ValueError: illegal castling move

Next step was to create a unit test to replay the moves from the game with the problem. Once I’d got it failing in the same way, I discovered a bug in my logic that meant rooks captured without moving would not update the “castling status” for that side. This led to falsely generating castling moves that weren’t possible and thus the failure in the stack trace (some defensive coding to stop this kind of thing). In this game in particular white’s queen-side rook had been captured without moving and my chess module was reporting that the king could still castle queen-side!

After getting it to work I now had a nice automated regression test for that bug. I also took the time to add a unit test for the actual function that I changed.

Brilliant stuff. Now I can be extra certain that bug is fixed and will stay fixed, as those tests will get run every time I run my test suite. I’m working on chrss in my spare time, so this is really important to me. I really don’t have time to spend manually testing and verifying that _everything_ works.

The more testing I can automate the less testing I have to do. Which means I’ve got more time for adding features or else playing games of chess!

chrss (chess by rss) update 20

  • Updated the user registration process, so hopefully it’s a bit friendlier
  • Users now have a chance to resend their activation email if they have trouble activating their account
  • Tweaked filters for games on user’s page:
    • “active” – games that it’s your turn in or have had a move in the last 30 days
    • “waiting” – un-finished games that aren’t active
    • “finished”
    • “all”

Also improved the code coverage some more.

On a side note there is a preliminary mobile version of chrss available now. However I wouldn’t recommend it for regular use, as making a move currently involves a _huge_ drop down menu. Less than ideal. Still it can be handy for checking whether anyone has move. I’ve got some plans on improving the usability in the future, so check back later to see how it’s doing.

Turbogears and mobile mini-site logins

I’m in the process of to writing a simple mobile version of chrss (chess by rss). This is spurred by the arrival of my new Nokia 6300 (which comes with Opera Mini). I’m aiming to end up with a mini-site running at http://chrss.co.uk/mob/ or similar.

The plan is to keep it _very_ simple. Main page lets you login, then lists your active games (showing which ones it’s your turn to move etc.). From there you’ll be able to get to a game, view the board and make a move. That’ll be it. Dead simple.

Turbogears works quite well for this in some respects – I can encapsulate the mini-site as a CherryPy controller and keep it fairly self contained. Though there is one minor drawback. That being that the identity framework is set up for the main site and will attempt to use the regular login page.

So how do we re-use the whole identity framework, but force access to the mobile mini-site to go via a different login page?

After a little poking around the TG source…

The skeleton controller for logging in:


class Mobile(controllers.Controller):
    @expose(template="cheetah:chrss.templates.mobile.index")
    @mob_login
    def index(self):
        return dict()
    
    @expose(template="cheetah:chrss.templates.mobile.login")
    def login(self,user=None,pwd=None):
        if identity.current.anonymous: 
            msg=None
            visit_key = turbogears.visit.current().key
            ident=identity.current_provider.validate_identity(user,pwd,visit_key)
            if ident is None:
                msg="login failed"
                return dict(user=user,msg=msg)
        raise redirect('/mob/')

The login method looks up the current visit_key (effectively the value of the cookie TG uses to track visits) then uses identity.current_provider. validate_identity to try and log the user in.

NB: I’m using user and pwd as variables, instead of user_name and password. The identity framework intercepts parameters with those names and uses them to authenticate behind the scenes. Obviously I need those values to get through to my controller method – hence why I’m using different names.

After the call to identity.current_provider. validate_identity I check to see if I have an object returned and either report that the login failed or redirect back to /mob/. Easy.

The next bit is the decorator @mob_login shown on the index controller method. This works a bit like the @identity.require decorator, but isn’t as flexible. It simply forces the user to a mobile login page if they aren’t logged in:


def mob_login(fn):
    def decorated(*arg,**kw):
        if identity.current.anonymous:
            return dict(tg_template="cheetah:chrss.templates.mobile.login",user='',msg=None)
        return fn(*arg,**kw)
    return decorated

Obviously you’ll need a template for the login page. I’m using cheetah and my login page looks like this (there’s a master template not shown here):


#extends chrss.templates.mobile.master
#def body
#if $msg
    <div id="msg">${msg}</div>
#end if
#filter WebSafe
    <h1>chrss: chess by rss</h1>
    <form action="$tg.url('/mob/login')" method="POST">
         <label for="user">user name:</label>
         <input type="text" id="user" name="user" value="$user"/>
         <label for="pwd">password:</label>
         <input type="password" id="pwd" name="pwd"/>
         <input type="submit" value="login"/>
    </form>
#end filter
#end def

And that is more or less all I had to do to get a basic mobile/alternative login page working. Hope you found it informative. I’ll report back later on how well actually running it on a mobile device goes…