Psychic Origami

Ok so I blogged about a podcatcher/podcast client for the Chumby quite a while ago (August 2008 to be precise). At the time I said I’d tidy things up and release the source code etc. Well that didn’t quite happen, but I figured I might as well release the source code. So seeing as I’ve been using git a lot lately and github is easy to use I thought that’d I’d best just put online what I had.

So you can now find the hybrid Python, Javascript and Flash Chumby postcast client I dubbed chumbycast, on github. I’m providing the code just for the curious – I’ve had at least one request for it, but I’m not really going to detail too much of how it works. Though some of this is outlined in the original post where I mentioned the project.

I am tempted to revive this project a bit, but probably by trying a different approach. I mostly gave up on this as I was creating a UI on the Chumby in Flash and my Flash skills aren’t exactly great, plus I was trying to use mtasc which hampered things further. Which is not to belittle mtasc – it’s just with limited time and no previous Flash experience I wasn’t exactly helping myself by not using some nice friendly Flash IDE.

I’ve since realised that if I ditched the Flash UI I could probably get quite a bit done. The Python httpserver side of things was pretty easy. So if I focussed on that I would then have a few of ways of providing a UI by hijacking existing functionality on the Chumby.

• The Chumby runs a httpserver on port 8080 when an ipod is plugged in, which the UI uses to access playlists etc. I could mimic this approach and effectively make the podcast client look like an iPod as far as the Chumby’s UI was concerned. By plugging in a USB stick loaded up with the podcast client everything would behave the same as if you had plugged in an iPod.
• It’s possible to create playlist files and edit the “My Streams” section programmatically. Each podcast subscribed to would create a matching pls or m3u file and be added to the “My Streams” section.
• Create a javascript/web UI for controlling the playback and subscriptions to podcasts and other podcast management tasks (like removing old files, manually downloading older episodes etc) from another computer. Possibly adding bonjour/zero-conf support so the Chumby can be browsed to easily

I would need to see whether those first two could be made to work for my purpose, but it would make sense to just use the existing UI on the Chumby – rather than creating a new one. The existing chumbycast code already provides a javascript/web UI for controlling the playback on the Chumby. This was originally done so I could remote control the chumby, but also so that I could easily create and debug an API for the Flash UI to use.

The other major missing feature is allowing the podcasts to be paused part way through. The current client does not support this, as the audio files are streamed and not downloaded. So that would be the first change to make. Now that I’ve dug out the code maybe I’ll be inspired to play around some more. I’ve also been listening to a lot more podcasts (thanks to Huffduffer) so it might happen yet.

So quite a while ago I wrote about avoiding the n+1 query problem with SQLObject. I’ve since been using Django a lot more. In particular at my day job I’ve been improving a Django site we’ve inherited. This particular site suffered from a few too many SQL queries and needed speeding up. Some of the issues related to the classic n+1 problem. i.e. one initial query triggering a further n queries (where n is the number of results in the first query).

A liberal dose of select_related helped. However that was only useful in the cases where a ForeignKey needed to be pre-fetched.

In the case however there was a page that was selecting a set of objects that had tags. The tags for each object were being displayed along side a link to the main object. Given that the initial query returned over three hundred objects, this meant the page was performing another three hundred (plus) queries to fetch the individual tags for each object! Now we could cache the page and that’s was indeed what was being done. The trouble however was when the cache expired. It also made things painful when developing – as I’d typically want to disable caching whilst I’m making changes to pages frequently.

I came up with a specific solution for this project – to perform the initial query, then a second query to fetch *all* of the other tags in one go. The results of the second query could then be stitched into the original results, to make them available in a handy manor within the page’s template.

I took the original idea and made it re-usable and am now releasing that code as Django Batch Select. There are examples of how to use it over at github, but it works a bit like this:

>>> entries = Entry.objects.batch_select('tags').all()
>>> entry = entries[0]
>>> print entry.tags_all
[<Tag: Tag object>]

It’s a very early release – after the result of only a few hours coding, so use with care. It does have 100% test code coverage at the moment and I’m reasonable confident about it working. Please try it out and let me know whether it works for you.

The Django admin site is one of the best features of Django. It really lets you just get on with building your app, without having to worry too much about how you’ll administer your site.

The defaults are generally pretty good, but it’s often the case that you’ll want to tweak and change it (particularly when you have clients involved). Luckily it’s pretty easy to customize.

One common change that many people will want to do is to display a thumbnail of an uploaded image in the admin. The default image field (when an image has been uploaded) in the admin looks like:

and we want to change it to look like this:

In my case I had images that I knew would be fairly small so I didn’t need to use any auto-resizing or anything like that. I found this snippet for doing the same task, but decided to simplify it a fair bit and ended up with:

from django.contrib.admin.widgets import AdminFileWidget
from django.utils.translation import ugettext as _
from django.utils.safestring import mark_safe

class AdminImageWidget(AdminFileWidget):
    def render(self, name, value, attrs=None):
        output = []
        if value and getattr(value, "url", None):
            image_url = value.url
            file_name=str(value)
            output.append(u' <a href="%s" target="_blank"><img src="%s" alt="%s" /></a> %s ' % \
                (image_url, image_url, file_name, _('Change:')))
        output.append(super(AdminFileWidget, self).render(name, value, attrs))
        return mark_safe(u''.join(output))

Then to use it I simply override formfield_for_dbfield to return a field that uses that widget for the field I’m interested in:

class MyAdmin(admin.ModelAdmin):
    def formfield_for_dbfield(self, db_field, **kwargs):
        if db_field.name == 'profile_image':
            request = kwargs.pop("request", None)
            kwargs['widget'] = AdminImageWidget
            return db_field.formfield(**kwargs)
        return super(MyAdmin,self).formfield_for_dbfield(db_field, **kwargs)

It’s a fairly straightforward alteration to how the ImageField widget renders in the admin, but it is often a big help to be able to actually see the image in question.

UPDATED: Feb 24th, 2010. Based on Jeremy’s comment below and having upgraded to Django 1.1 I’ve modified this example to remove the “request” param from kwargs before passing it db_field.

I’ve been writing an application for my other half in recent months to help her organise observations of the children in her class. I’ve actually written two versions of the app. The first was dubbed “Observertron” and in retrospect was overly complex. After a bit of thought I was able to vastly simplify the app, whilst also making it more useful for other purposes beyond making observations.

The app is called NoteComb and is written in Python using wxPython. It’s essentially a specialised text-editor. The core feature is a grep-like search functionality coupled with the ability to edit text in-place during a search. Lines that don’t contain the search terms get hidden, leaving you free to edit the remaining lines as you want:

I’ve decided to take the time and try to package it up “properly”. So NoteComb is available as Mac and Windows apps, complete with icons etc. In fact to “regular” users it should appear that NoteComb is an app like any other – the fact it’s written using Python is largely incidental.

This packaging works pretty well overall and is pretty seamless once the app is downloaded, but it does tend to make for rather large apps. The Windows version when packaged as an installer runs in at about 4Mb, which isn’t too crazy, but the OS X app packed into a compressed DMG file weighs in at 15Mb! At the end of the day the app does include Python + wxPython + various libraries, so it’s not a surprise really, but I guess I was kind of hoping that I might develop in Python and get everything for free…

This is an early version of NoteComb (version 0.2.1), but it’s core functionality is there. Most of the extra work I’ve done has been on adding those little extra bits that aren’t core to the app, but are generally just expected (e.g. remembering previous window positions, copy/paste, undo/redo etc).

So feel free to download NoteComb and give it a go.

I’ve updated the relevant wxPython wiki page with this info, but thought i’d record it here for posterity too.

I’m currently playing around with writing a wxPython app. It’s at the point where I’m starting to package it into executables to be run without needing Python installed.

Py2exe is the tool I’m using for creating a windows app (and py2app for the mac). It’s possible to specify an icon for the executable you create with py2exe. However it’d be nice if this same icon would be used as the icon for frames created by that app. One could include an extra icon as a resource and load that up, but this seems counter to the whole DRY principle. Instead it’d be better to load the same icon from the executable we have created already.

Consulting the wxPython wiki showed that some enterprising souls had already had a go at doing this, but the solutions there relied on the win32api modules amongst others. My Windows machine didn’t have this module installed (as it was running Python 2.5) and looking at the 2nd solution proposed I saw a way to remove this dependency (and thus decrease the size of the final exe as it would not need to include the extra dll’s etc).

So here’s the sample code, from the wiki, I posted:

import wx, sys

class MyFrame(wx.Frame):
    def __init__(self, parent=None):
        wx.Frame.__init__(self, parent, wx.ID_ANY)
            # set window icon
            if sys.platform == 'win32':
                # only do this on windows, so we don't
                # cause an error dialog on other platforms
                exeName = sys.executable
                icon = wx.Icon(exeName, wx.BITMAP_TYPE_ICO)
                self.SetIcon(icon)

if __name__ == '__main__':
    app = wx.App(redirect=False)
    frame = MyFrame()
    frame.Show(True)
    app.MainLoop()

Basically it uses sys.executable to find the exe we are running in and then gets wxPython to load the first icon it finds from it (which should be the icon of the executable itself). So when you run this directly from python (not packaged), you end up with the icon from the python executable, but when you run the packaged version you get the icon from the exe instead.

It includes the platform check, as otherwise the app throws up an error dialog on the mac and obviously I want this to work on both platforms (and Linux too in the future).

This is probably my first attempt at “literary programming”, though using that phrase may be taking liberties a little. In this case by “literary programming” I mean – programming inspired by literature. The literature in question is Cryptonomicon by Neal Stephenson. Cryptonomicon already features a Perl script in it’s pages, but that’s not what I’m talking about here. Instead there’s a chapter later in the book when Randy Waterhouse has been incarcerated. However he’s been given the use of his laptop so that – his captors hope – he will decrypt a key piece of data whilst they are monitoring his screen via Van Eck Phreaking. Randy cottons onto this, so he creates an app that will convert text to morse code and have that morse code emitted via the LEDs on his laptop’s keyboard.

I suddenly decided to have a go at creating such a morse code script for my Macbook and here’s the result in action:

You can test the script out by running this at the command line (from the directory with the script in):

echo 'sos' | python morse.py -led

It depends on libraries that are only found in OS X 10.5 Leopard for controlling the capslock LED.

With a bit of compression it all fits well under 5K of code, so makes a worthy example of a 5K app. The morse part of the app is very simple – apart from the lookup table, it could probably fit into 2-3 lines of code! The bulk of the code is simply concerned with driving the capslock key’s LED via the HID. For this I cribbed heavily from some of Apple’s sample code and converted it to run in Python using ctypes.

The only problem I found was that for some reason they internal keyboard on my Macbook stops responding to the request to change the LED state after several seconds. However using this script with an external USB keyboard worked fine. I guess the internal keyboard has some sort of abuse-prevention built-in.

• Download the source (with build files etc)

The next Brighton and Hove Python User Meet will be the 18th of February 2009 at the Hampton Arms Farm Tavern. As per usual we’ll be sharing the venue with the Farm. Now that there’s a google group setup for these meets they might happen a bit more often, as Ian and I will no longer be the limiting factor!

A quick reminder too that the next £5app meet is now only a few weeks away on the 3rd of March. Jeremy Keith will be talking about Huffduffer (which I’ve been using a lot for getting great podcasts for the walk home after work) and we’ll be running the 5K app competition. The deadline for the 5K app is the Friday before the event (27th Feb), so there’s just under three weeks left to get something written.

If anyone wants to chat about the 5K app they’re welcome to pop along to the Python User Meet too. I’m more than happy to spend some time helping people get their code under the 5K budget.

After managing to organise two previous Python meetups and failing to organise many more it seemed like a good time to set-up mailing list of some sort so that things can self-organise a bit more. So please sign-up for the Brighton and Hove Python User Group Google Group (BHPUGGG) and hopefully there will be a few more meetups more often.

For any Pythonistas out there you might want to have a look at the 5K app competition. The idea is to write apps that are less than 5120 bytes in size. It’d be great to see some small Python apps entered and with Python (and a bit of self-extracting script fun) they should pack quite a punch!

After a 5K Java Twitter client and a 5K Javascript TODO list app, the next example 5K App is a wxPython fullscreen text editor. Much like Writeroom and Duncan’s JDarkroom (both of which are more fully featured) it lets you edit text files in “distraction-free” fullscreen. Ideal for creative writing or similar where you want to just get on with writing without the distraction of the outside world. So at the risk of invoking the apoplectic rage of Mark Pilgrim a fullscreen text editor seemed quite feasible in 5K of Python. In fact after using my self-extracting script code I actually had to add more features as I was way below budget!

Check out the video below or download 5KEdit for yourself.

Features:

• Fullscreen text editing
• Open/Save files
• Undo/Redo
• Retro looks
• Resizable fonts
• Configurable colours
• Word count
• Goto line number dialog
• Find text dialog
• Help page listing commands
• About box

It’s actually a bit more feature-full than I’d thought it would be. Part of this was that wxPython’s wx.StyledTextCtrl and wxPython in general provided quite a lot of what I needed (e.g. undo/redo) out of the box. Plus the self-extracting script technique really did a good job of shrinking the code down – allowing me to add more.

To run 5KEdit your need Python (2.4+) and wxPython (2.8+) installed. If you are running OS X Leopard you already have those installed, so you can simple switch to a terminal and run that 5KEdit script using the command:

pythonw 5KEdit.pyw

Make sure you run that from the directory 5KEdit.pyw is in. On a Windows system you may find that you can simply double-click on the .pyw file to run it. There’s a small repaint bug under Windows when you change colors, but it’s not a show stopper.

Here are the vital statistics:

• 17284 bytes/485 lines of original source-code
• 13767 bytes/379 lines of stripped source-code
• 5069 bytes of compressed/packed final version

As you can see the final version is less than a third the size of the original source code. A large part of this is due to the compression, but to help push it a bit further the source was also “stripped”. Blank-lines and comments were removed and the indentation was changed from my usual 4 spaces to a single space. The stripping doesn’t make a huge contribution, but without it the final version would be 5421 bytes in size. So for 400 bytes or so it’s probably worth it. Plus it means that you don’t have to worry about leaving yourself comments or spacing out your code nicely.

• Download the source (with build files etc)

So far for the 5K App I’ve written two example apps – one in Java and one in Javascript. This is partly so that I can figure out how the rules for the competition will need to work. In a lot of 4K competitions a single language is chosen, so things are a bit more straightforward. However Brighton seems like quite a varied place and allowing multiple languages for the competition seems like a nice friendly inclusive thing to do.

So the next example app I’ll be writing will be in Python. As Python source is the executable code there’s no compilation step (unlike Java) and un-like Javascript there aren’t really many tools for reducing code size. Python (like Ruby, Perl and PHP) tends to result in fairly concise code, so normally this isn’t too much of a problem. However 5K is quite a tough constraint and we really want to squeeze the most out of our code by compressing and packing it if possible.

So here’s a little script for compressing a Python script to make it much smaller:

import sys
import bz2
import base64

write=sys.stdout.write

for name in sys.argv[1:]:
    contents=bz2.compress(open(name).read())
    write('import bz2, base64\n')
    write('exec bz2.decompress(base64.b64decode(\'')
    write(base64.b64encode((contents)))
    write('\'))\n')

To use it save that code into a file (e.g. pack.py). It will write the compressed code to standard output (i.e. the screen), so you can redirect it to whatever file you want:

python pack.py my_script.py > my_script_packed.py

The compressed code looks like this:

import bz2, base64
exec bz2.decompress(base64.b64decode('<base64 encoded compressed data>'))

Which hopefully is fairly clear as to what it’s doing, but to summarize:

• The script data is base 64 decoded into bytes
• The bytes are then decompressed (bz2) into the text of the original script
• exec is then called to run the original script

One nice benefit to this way of compressing the script is that the final module namespace (after de-compression) will look essentially the same as it did if it was not compressed. The only difference is the bz2 and base64 modules will also be present. This should mean that you can actually compress multiple Python files and importing from them should still work. Though of course as is the case when adding an extra layer of complexity your mileage may vary…

For an idea of how effective this compression can be I took the Python script for calculating pi on wikipedia and ran it through the script. After confirming that it ran the same, a quick comparison revealed the compressed version had gone from 12658 bytes to 3421 bytes – less than a third of the original size.

It should be possible to create similar scripts for Ruby (using zlib and base64), Perl (using Compress::Zlib and MIME::Base64) and PHP (using bzdecompress and base64_decode).

As I work on the example Python 5K app I should hopefully get a good feel for how the competition rules might need changing to allow “scripting” languages like Python, Ruby, Perl and PHP. I think the plan will be that the app must be contained in a single file (less than 5120 bytes in size), with any resources embedded in the file. This is the norm for Java and Flash, but will probably require an extra packaging step for most other languages/runtimes. That single file can then be run either via a GUI (double-clicking) or via a standard invocation from the command line (e.g. python my_script_packed.py) using only a “standard” version of the language runtime. Note that the standard installed version of Python on MacOS X 10.5 (Leopard) includes quite a few extra libraries (e.g. wxPython) so these libraries would be eligible for use in the 5K app. The same will be true for Ruby and Perl, so that should hopefully help open things out. Otherwise Java’s large standard library might give it too much of an advantage…