Ok so I blogged about a podcatcher/podcast client for the Chumby quite a while ago (August 2008 to be precise). At the time I said I’d tidy things up and release the source code etc. Well that didn’t quite happen, but I figured I might as well release the source code. So seeing as I’ve been using git a lot lately and github is easy to use I thought that’d I’d best just put online what I had.
So you can now find the hybrid Python, Javascript and Flash Chumby postcast client I dubbed chumbycast, on github. I’m providing the code just for the curious – I’ve had at least one request for it, but I’m not really going to detail too much of how it works. Though some of this is outlined in the original post where I mentioned the project.
I am tempted to revive this project a bit, but probably by trying a different approach. I mostly gave up on this as I was creating a UI on the Chumby in Flash and my Flash skills aren’t exactly great, plus I was trying to use mtasc which hampered things further. Which is not to belittle mtasc – it’s just with limited time and no previous Flash experience I wasn’t exactly helping myself by not using some nice friendly Flash IDE.
I’ve since realised that if I ditched the Flash UI I could probably get quite a bit done. The Python httpserver side of things was pretty easy. So if I focussed on that I would then have a few of ways of providing a UI by hijacking existing functionality on the Chumby.
The Chumby runs a httpserver on port 8080 when an ipod is plugged in, which the UI uses to access playlists etc. I could mimic this approach and effectively make the podcast client look like an iPod as far as the Chumby’s UI was concerned. By plugging in a USB stick loaded up with the podcast client everything would behave the same as if you had plugged in an iPod.
It’s possible to create playlist files and edit the “My Streams” section programmatically. Each podcast subscribed to would create a matching pls or m3u file and be added to the “My Streams” section.
Create a javascript/web UI for controlling the playback and subscriptions to podcasts and other podcast management tasks (like removing old files, manually downloading older episodes etc) from another computer. Possibly adding bonjour/zero-conf support so the Chumby can be browsed to easily
I would need to see whether those first two could be made to work for my purpose, but it would make sense to just use the existing UI on the Chumby – rather than creating a new one. The existing chumbycast code already provides a javascript/web UI for controlling the playback on the Chumby. This was originally done so I could remote control the chumby, but also so that I could easily create and debug an API for the Flash UI to use.
The other major missing feature is allowing the podcasts to be paused part way through. The current client does not support this, as the audio files are streamed and not downloaded. So that would be the first change to make. Now that I’ve dug out the code maybe I’ll be inspired to play around some more. I’ve also been listening to a lot more podcasts (thanks to Huffduffer) so it might happen yet.
So quite a while ago I wrote about avoiding the n+1 query problem with SQLObject. I’ve since been using Django a lot more. In particular at my day job I’ve been improving a Django site we’ve inherited. This particular site suffered from a few too many SQL queries and needed speeding up. Some of the issues related to the classic n+1 problem. i.e. one initial query triggering a further n queries (where n is the number of results in the first query).
A liberal dose of select_related helped. However that was only useful in the cases where a ForeignKey needed to be pre-fetched.
In the case however there was a page that was selecting a set of objects that had tags. The tags for each object were being displayed along side a link to the main object. Given that the initial query returned over three hundred objects, this meant the page was performing another three hundred (plus) queries to fetch the individual tags for each object! Now we could cache the page and that’s was indeed what was being done. The trouble however was when the cache expired. It also made things painful when developing – as I’d typically want to disable caching whilst I’m making changes to pages frequently.
I came up with a specific solution for this project – to perform the initial query, then a second query to fetch *all* of the other tags in one go. The results of the second query could then be stitched into the original results, to make them available in a handy manor within the page’s template.
I took the original idea and made it re-usable and am now releasing that code as Django Batch Select. There are examples of how to use it over at github, but it works a bit like this:
It’s a very early release – after the result of only a few hours coding, so use with care. It does have 100% test code coverage at the moment and I’m reasonable confident about it working. Please try it out and let me know whether it works for you.
I’ve been working on a little kite physics demo app:
After chatting with Allistar at the £5app about how he did physics simulations I was pointed at “Gauss-Seidel” as an easy way to handle constraints. This led me to Advanced Character Physics by Thomas Jakobsen, which explained it all very nicely. That coupled with some info from NASA on forces on a kite helped me create this simple 2D “simulation”.
It’s not yet finished, but I thought I’d share some of my progress. I want to turn it into a slightly more rounded demo or perhaps even a simple game of some sort, with some scenery etc. I’m quite happy with how it’s turned out so far – even though I did waste a fair bit of time getting drop-shadows working…
It’s all written in Java, so it’ll be nice and easy to put online later. I think I’ll probably release the source too, as the physics and constraints code is fairly generalized. In fact through the use of generics and a few other tricks the code is setup to allow (potential) re-use for 3D physics too.
A while back I had a go at using the local storage features being added to javascript to create a simple TODO list app. The main focus of the app was getting it all to fit under 5K (5120 bytes), but it was a good test of using a client-side sql database in javascript.
As the app was written for size it didn’t have many frills. It did however work on the iPhone, as it’s version of Safari had support for the openDatabase call needed. However it didn’t look so good and although the TODO list items were stored locally the phone still had to be online to access the host webpage – which kind of undermined some of it’s utility.
With my recent acquisition of an iPhone I thought I’d revisit this TODO app and make it play nicely on the iPhone. In addition to using a client-side SQL database this new version features:
These features mean that if you have add a bookmark to your homescreen for this app, you might as well be running a native app. It looks pretty native, stores data locally, doesn’t require a net connection and even features standard app UI mechanisms. The main giveaway is of course the chrome associated with the Safari browser. Still not bad for some html, css and javascript. Not a total replacement for native Cocoa apps, but it does put the creation of client-side apps for the iPhone into the hands of even more people.
If you are on an iPhone or are running Safari 4.0 you can try out jTODO yourself or watch the video of it in action below:
I won’t rehash the sql-side of things in this post – instead I’d refer you to my original post on the matter.
The use of iui was also fairly straightforward – I’m only using the style-sheets and images. This means that there are no animations involved, but at this stage it seemed excessive to add them in. Perhaps I’ll add some in a future version.
This leaves the offline application cache and getting drag and drop to work.
Offline application cache
The offline application cache is really simple to implement. It’s currently supported by Safari on the iPhone, Safari 4.0 and Firefox 3+.
In my case I want all files to be cached, so I simple specify them in a “manifest” file:
This manifest file is then linked to the main html file thus:
<html manifest="todo.manifest">
That’s it. We now have an offline capable web-app. The main issue I had when doing this, was that Safari was very strict about the manifest – any file not mentioned in the manifest would not be loaded (even if it was normally accessible). The other issue of course is that we’ve now introduced another level of caching, so developing can be a bit annoying – as you think you’ve made a change, but then nothing shows up. I often ended up disabling the manifest for a while when debugging and then re-enabled it after things were working again. There’s also a version number in the manifest file, so that it well register as changed – to help refresh the caches after we’ve made a change.
With all this in place the app can be used when no net connection is available (e.g. in Airplane mode).
Drag and drop
The last job when developing this app was to enable drag and drop for re-ordering items. My first go at this worked pretty easily using jQuery UI’s sortable plugin – when running on my mac in Safari:
page.sortable({
axis: 'y',
handle: '.handle',
update: function(){
db.transaction(function(tx) {
$('.todo_item').each(function(i,item) {
var id = $(item).find(':input[type=checkbox]').attr('id');
id = Number(id.substring('todo_checkbox_'.length));
tx.executeSql('UPDATE todo SET todo_order=? WHERE id=?', [i, id]);
});
});
}
});
Essentially this is just setup to enabling drag and drop sorting only along the y axis (up and down), using the element with class handle to start the drag. Once the dragging has finished update gets called and I inspect the DOM to work out the current order of the todo_items and update the database accordingly.
That was pretty easy and working really well on the mac. Then of course I thought I’d test it on the iPhone. At that point I realised I’d forgotten that drag and drop doesn’t normally work in Safari in the iPhone. Holding and dragging normally scrolls the entire page – so drag and drop was a bit useless here.
However after a little digging I found someone had figured out how to get drag and drop working on the iPhone, by hijacking the various “touch” events that the phone generates. These work a little differently to the normal mouse events, but with some work can made to do our bidding:
function handleTouchEvent(event) {
/* from http://jasonkuhn.net/mobile/jqui/js/jquery.iphoneui.js
but changed a bit*/
var touches = event.changedTouches;
var first = touches[0];
var type = '';
// only want to do this for the drag handles
if ( !first.target || !first.target.className || first.target.className.indexOf("handle") == -1 ) {
return;
}
switch(event.type) {
case 'touchstart':
type = 'mousedown';
break;
case 'touchmove':
type = 'mousemove';
break;
case 'touchend':
type = 'mouseup';
break;
default:
return;
}
var simulatedEvent = document.createEvent('MouseEvent');
simulatedEvent.initMouseEvent(type, true, true, window, 1, first.screenX, first.screenY, first.clientX, first.clientY, false, false, false, false, 0/*left*/, null);
first.target.dispatchEvent(simulatedEvent);
if ( event.type == 'touchmove' ) {
event.preventDefault();
}
}
document.addEventListener("touchstart", handleTouchEvent, false);
document.addEventListener("touchmove", handleTouchEvent, false);
document.addEventListener("touchend", handleTouchEvent, false);
This registers listeners for the touch events, but only does any extra work if the target of the event has the class “handle”. If that’s the case a simulated mouse event is sent for the first touched item. To stop the page from scrolling when we want to drag instead event.preventDefault() is called just for the touchmove event. This is sufficient to let jquery UI do it’s job and enable drag and drop sorting of the TODO items.
The Django admin site is one of the best features of Django. It really lets you just get on with building your app, without having to worry too much about how you’ll administer your site.
The defaults are generally pretty good, but it’s often the case that you’ll want to tweak and change it (particularly when you have clients involved). Luckily it’s pretty easy to customize.
One common change that many people will want to do is to display a thumbnail of an uploaded image in the admin. The default image field (when an image has been uploaded) in the admin looks like:
and we want to change it to look like this:
In my case I had images that I knew would be fairly small so I didn’t need to use any auto-resizing or anything like that. I found this snippet for doing the same task, but decided to simplify it a fair bit and ended up with:
from django.contrib.admin.widgets import AdminFileWidget
from django.utils.translation import ugettext as _
from django.utils.safestring import mark_safe
class AdminImageWidget(AdminFileWidget):
def render(self, name, value, attrs=None):
output = []
if value and getattr(value, "url", None):
image_url = value.url
file_name=str(value)
output.append(u' <a href="%s" target="_blank"><img src="%s" alt="%s" /></a> %s ' % \
(image_url, image_url, file_name, _('Change:')))
output.append(super(AdminFileWidget, self).render(name, value, attrs))
return mark_safe(u''.join(output))
Then to use it I simply override formfield_for_dbfield to return a field that uses that widget for the field I’m interested in:
It’s a fairly straightforward alteration to how the ImageField widget renders in the admin, but it is often a big help to be able to actually see the image in question.
UPDATED: Feb 24th, 2010. Based on Jeremy’s comment below and having upgraded to Django 1.1 I’ve modified this example to remove the “request” param from kwargs before passing it db_field.
I’ve been writing an application for my other half in recent months to help her organise observations of the children in her class. I’ve actually written two versions of the app. The first was dubbed “Observertron” and in retrospect was overly complex. After a bit of thought I was able to vastly simplify the app, whilst also making it more useful for other purposes beyond making observations.
The app is called NoteComb and is written in Python using wxPython. It’s essentially a specialised text-editor. The core feature is a grep-like search functionality coupled with the ability to edit text in-place during a search. Lines that don’t contain the search terms get hidden, leaving you free to edit the remaining lines as you want:
I’ve decided to take the time and try to package it up “properly”. So NoteComb is available as Mac and Windows apps, complete with icons etc. In fact to “regular” users it should appear that NoteComb is an app like any other – the fact it’s written using Python is largely incidental.
This packaging works pretty well overall and is pretty seamless once the app is downloaded, but it does tend to make for rather large apps. The Windows version when packaged as an installer runs in at about 4Mb, which isn’t too crazy, but the OS X app packed into a compressed DMG file weighs in at 15Mb! At the end of the day the app does include Python + wxPython + various libraries, so it’s not a surprise really, but I guess I was kind of hoping that I might develop in Python and get everything for free…
This is an early version of NoteComb (version 0.2.1), but it’s core functionality is there. Most of the extra work I’ve done has been on adding those little extra bits that aren’t core to the app, but are generally just expected (e.g. remembering previous window positions, copy/paste, undo/redo etc).
So feel free to download NoteComb and give it a go.
I’ve updated the relevant wxPython wiki page with this info, but thought i’d record it here for posterity too.
I’m currently playing around with writing a wxPython app. It’s at the point where I’m starting to package it into executables to be run without needing Python installed.
Py2exe is the tool I’m using for creating a windows app (and py2app for the mac). It’s possible to specify an icon for the executable you create with py2exe. However it’d be nice if this same icon would be used as the icon for frames created by that app. One could include an extra icon as a resource and load that up, but this seems counter to the whole DRY principle. Instead it’d be better to load the same icon from the executable we have created already.
Consulting the wxPython wiki showed that some enterprising souls had already had a go at doing this, but the solutions there relied on the win32api modules amongst others. My Windows machine didn’t have this module installed (as it was running Python 2.5) and looking at the 2nd solution proposed I saw a way to remove this dependency (and thus decrease the size of the final exe as it would not need to include the extra dll’s etc).
So here’s the sample code, from the wiki, I posted:
import wx, sys
class MyFrame(wx.Frame):
def __init__(self, parent=None):
wx.Frame.__init__(self, parent, wx.ID_ANY)
# set window icon
if sys.platform == 'win32':
# only do this on windows, so we don't
# cause an error dialog on other platforms
exeName = sys.executable
icon = wx.Icon(exeName, wx.BITMAP_TYPE_ICO)
self.SetIcon(icon)
if __name__ == '__main__':
app = wx.App(redirect=False)
frame = MyFrame()
frame.Show(True)
app.MainLoop()
Basically it uses sys.executable to find the exe we are running in and then gets wxPython to load the first icon it finds from it (which should be the icon of the executable itself). So when you run this directly from python (not packaged), you end up with the icon from the python executable, but when you run the packaged version you get the icon from the exe instead.
It includes the platform check, as otherwise the app throws up an error dialog on the mac and obviously I want this to work on both platforms (and Linux too in the future).
When Jeremy talked about Huffduffer at March’s £5app he revealed that pretty much every page in Huffduffer had a RSS/JSON/XML version. A couple of weeks back, I asked whether Huffduffer supported JSONP. After a quick reply back from Jeremy that it wasn’t, there was another reply within an hour or two saying Jeremy had added JSONP support. At that point I felt my curiosity had got the better of me and I felt duty bound to a least do something with the newly added JSONP support…
Both the page used to generate the widget HTML and the widget code itself are 100% client-side. There’s no server-side code involved at all. For the widget itself this seemed pretty key. Such widgets may be embedded in pages that get a lot of hits, so doing anything beyond serving up a static file seemed best to avoid.
I’d not really written a widget like this before, but planned to create something similar for chrss at some point, so it seemed like a good thing to have a go at. Most of the techniques are based on code from Dr Nic’s DIY widgets – How to embed your site on another site article. However I tried to add a few more features/constraints into the mix:
Allow more than one widget with different data on the same page
Don’t pollute the global namespace
Don’t require special html embedded with the widget (just a single script tag)
The first point involved allowing a static javascript file to server different data. The easy way would be to have some server side code serve up some slightly different Javascript depending on the parameters given, but I decided there’s no reason why the Javascript can’t do that itself.
When the widget script runs it looks at the last known script element – which should be the element used to load the script itself to parse the parameters. In this case I just want a single parameter. So if the script tag used to embed the widget looks like this:
function findURL() {
// assume that last script in page is this script
var scripts = document.getElementsByTagName('script');
var script = scripts[scripts.length-1];
var scriptURL = script.getAttribute('src');
var m = scriptURL.match('^.*\\?(.*)');
if ( m ) {
return m[1];
}
return ''
}
So that solves the first problem.
The next problem is not polluting the global namespace. First I make sure that everything is wrapped inside an anonymous function that is called immediately. This reduces the scope of the functions declared in there. However we do need at least one global function to act as a callback for the JSONP data. In addition this function must be unique to each widget embedded in the page, so that the correct data gets given to the correct widget.
So first I generate a relatively unique id:
function uuid() {
var id = "_"+(new Date()).getTime();
for ( var i = 0; i < 8; i++ ) {
id += "0123456789".charAt(Math.floor(Math.random()*10));
}
return id;
}
var id = uuid();
var div_id = 'huffduffer'+id;
Then we use that id as a name for the function and create a callback that embeds the id of the div the widget will appear in via a closure:
function createCallback(div_id) {
// return a function with the div_id in a closure
return function(data) {
// real callback code here
}
}
// create a function with a generated name, to ensure
// we get the right div
window[id] = createCallback(div_id);
document.write("<div id='"+div_id+"'></div>");
This means that we can have multiple widgets on the page all showing different data and the global namespace pollution is limited to one randomly named function per widget.
In addition by creating this unique id we can tie the callback function to the relevant div in the page without needing any more magic. Which means we've also dealt with the third problem.
The "real" widget has been minimized using YUICompressor, but you can see the regular version of the code here:
This is probably my first attempt at “literary programming”, though using that phrase may be taking liberties a little. In this case by “literary programming” I mean – programming inspired by literature. The literature in question is Cryptonomicon by Neal Stephenson. Cryptonomicon already features a Perl script in it’s pages, but that’s not what I’m talking about here. Instead there’s a chapter later in the book when Randy Waterhouse has been incarcerated. However he’s been given the use of his laptop so that – his captors hope – he will decrypt a key piece of data whilst they are monitoring his screen via Van Eck Phreaking. Randy cottons onto this, so he creates an app that will convert text to morse code and have that morse code emitted via the LEDs on his laptop’s keyboard.
I suddenly decided to have a go at creating such a morse code script for my Macbook and here’s the result in action:
You can test the script out by running this at the command line (from the directory with the script in):
echo 'sos' | python morse.py -led
It depends on libraries that are only found in OS X 10.5 Leopard for controlling the capslock LED.
With a bit of compression it all fits well under 5K of code, so makes a worthy example of a 5K app. The morse part of the app is very simple – apart from the lookup table, it could probably fit into 2-3 lines of code! The bulk of the code is simply concerned with driving the capslock key’s LED via the HID. For this I cribbed heavily from some of Apple’s sample code and converted it to run in Python using ctypes.
The only problem I found was that for some reason they internal keyboard on my Macbook stops responding to the request to change the LED state after several seconds. However using this script with an external USB keyboard worked fine. I guess the internal keyboard has some sort of abuse-prevention built-in.
So after writing a 5K Twitter Client, a 5K TODO app and a 5K Fullscreen Text Editor I thought that’d be enough example apps to get things started. However while adding a list of app ideas to the 5K App page I suddenly decide that one of the ideas really appealed. The idea was to write a guitar tuner, that would listen on the computers mic and show the user what pitch it thought the note played was. I’m not much of a guitar player, but when I do play it’s always nice to have a guitar that’s in tune. As I’m unable to tune a guitar by ear I rely on guitar tuners and I always seem to be out of the 9V batteries my electronic tuner needs. Given my laptop is usually nearby, being able to use it as a guitar tuner would be most useful.
Check out the video below or download 5KTuner for yourself (requires Java):
The app is more or less non-interactive. It simply starts up tries to work out what note is being played. It shows a simple slider that moves according to what note is being played and indicates how close to that note you are. In addition the raw frequency (in hz) is shown, as well a chart that shows the autocorrelation function of the input sound. A red line is drawn on this chart when a “valid” note is heard and indicates the wavelength of the overall frequency.
The vital statistics are:
Final (obfuscated) app size: 3538 bytes
Final (un-obfuscated) app size: 4649 bytes
Final Source-code size: 8532 bytes
Lines of code: 245
This was my first foray into audio programming and I’m grateful that in the end it wasn’t too tricky. Initially I’d been looking into using the Fast Fourier Transform to work out the frequencies, but the autocorrelation function was much easier. It’s essentially just two for loops and a bit of maths. You listen for a certain amount of audio and then compare the first half of the sample with itself, but shifted. By working out the amount of shift where the difference is smallest you can find the overall frequency of the sample.
It could probably be made more robust, but if you want to see how it works (or improve it) you can download the source code.
