Psychic Origami

Here in Brighton a good number of the bus stops have electronic boards, that tell you when the next buses are due. The data for these boards is available online and even provides data for stops that don’t have the electronic boards. In fact there’s an free iPhone app available, so you can get the data easily on the move.

After playing around with iPhone javascript apps, I thought it would be interesting to try and create a javascript app for the bus times. I had some specific ideas about how I wanted the app to work. I was also keen on trying out some more “HTML 5″ tech. Plus I do like things to be (at least in principle) cross-platform, so this has a chance of working ok on Android phones (though I’ve not tested it on one).

There are still a few rough edges, but you can try out the app yourself at:

• http://buses.psychicorigami.com/

It will prompt you to add a stop. If you type in the text field, you’ll get a list of stops that match. If your browser supports geo-location, you’ll also see a “Find nearby” button, which will list the nearest few stops.

I had the pleasure of demoing this app at the Async Javascript show’n'tell last month. I quickly outlined how it worked, which involves the following:

• Offline App Cache – to make startup quick (key when you don’t want to miss your bus)
• Client-side Local Storage – to record which bus stops you add
• Geo-location – so you can find nearby stops
• JQuery – for front end
• Bottle.py – for back end
• BeautifulSoup – for scraping bus times

Getting the data

The buses.co.uk site has an API, which I used to scrape all of the bus stops in Brighton. I chose to scrape the update data from the website itself though, as I couldn’t get the API to give me the combined services for each stop (so you can see when the next 7 and 81 are coming for example). There are typically two stops for each name – one either side of the road. The updates are for both directions, so I actually merged the stops together – which makes searching easier.

All of the stop data was stored in a single SQLite database – mostly for convenience. The database is used in a read-only method in the actual running app. I’m sure I could use something a bit better, particularly when it comes to the distance queries. Currently, as there are a relatively small number of stops I’m using a brute force method to find the nearest stops, but I’m sure with the use of a proper spatial index this could be done a lot more efficiently. If this was scaled up to work for all stops in the UK, then that would be a necessity.

Geo-location

Initially the geo-location was pretty straightforward – I simply used the getCurrentPosition function and I got a latitude and longitude back. Testing in Firefox and Safari on the mac gave fairly reasonable results. However on the iPhone itself I started to notice it wasn’t very accurate. Sometimes it was out by several 100 metres, meaning that stops I was standing next to were sometimes not showing up at all! I had noticed, in the past, that the built-in map app sometimes has a bit of “lag” in getting an accurate position. So I switched to using watchPosition and polling for the position for several seconds. This worked pretty well, as the initial result would only be vaguely in the right place and then after a few seconds the position would become more accurate and the listing of nearby stops would start to look more sensible.

Look and feel

Originally I’d planned to mimic the look and feel of the iPhone’s built-in weather app. The app is built around a similar concept – you search for bus stops and add them to a list of stops that you can flick through. I tried to get a nice animated sliding action working, but kept on running into trouble on the device itself. In Safari and Firefox the animation all worked ok, but I suspect there’s an issue on the iPhone when you are dragging an area that limits redrawing. In the end I had to ditch the flip/swipe gesture too – interfering with the touch events again seemed to cause some rendering issues on occasion. So instead simply clicking on the left or right of the timetable moves between pages. It’s a slightly less complicated way of doing things, so there’s less that can go wrong really.

Deployment

As this was a nice simple app (on the back-end at least), I decided to take a little time to create a Fabric file to control deployment. The app is hosted on Webfaction, where I have already setup key-based login for ssh. The app is deployed using mod_wsgi with a daemon process, so that one merely needs to copy over the new code, then “touch” the .wsgi file restart the daemon process. Not exactly tricky to do, but making it a one-button process saves making any mistakes.

To trigger the deployment I run my fab file like this:

fab --user=lilspikey --hosts=lilspikey.webfactional.com deploy:webapps/buses/app/

I actually have that in a shell script, that isn’t checked into git, so that the deployment specific username, host name and remote directory aren’t checked into source control. If I wasn’t using key-based login, I’d need to specify a password and I definitely wouldn’t want to check that into git!

Source code

You can get the source code on github in my buses repository.

Ok so I blogged about a podcatcher/podcast client for the Chumby quite a while ago (August 2008 to be precise). At the time I said I’d tidy things up and release the source code etc. Well that didn’t quite happen, but I figured I might as well release the source code. So seeing as I’ve been using git a lot lately and github is easy to use I thought that’d I’d best just put online what I had.

So you can now find the hybrid Python, Javascript and Flash Chumby postcast client I dubbed chumbycast, on github. I’m providing the code just for the curious – I’ve had at least one request for it, but I’m not really going to detail too much of how it works. Though some of this is outlined in the original post where I mentioned the project.

I am tempted to revive this project a bit, but probably by trying a different approach. I mostly gave up on this as I was creating a UI on the Chumby in Flash and my Flash skills aren’t exactly great, plus I was trying to use mtasc which hampered things further. Which is not to belittle mtasc – it’s just with limited time and no previous Flash experience I wasn’t exactly helping myself by not using some nice friendly Flash IDE.

I’ve since realised that if I ditched the Flash UI I could probably get quite a bit done. The Python httpserver side of things was pretty easy. So if I focussed on that I would then have a few of ways of providing a UI by hijacking existing functionality on the Chumby.

• The Chumby runs a httpserver on port 8080 when an ipod is plugged in, which the UI uses to access playlists etc. I could mimic this approach and effectively make the podcast client look like an iPod as far as the Chumby’s UI was concerned. By plugging in a USB stick loaded up with the podcast client everything would behave the same as if you had plugged in an iPod.
• It’s possible to create playlist files and edit the “My Streams” section programmatically. Each podcast subscribed to would create a matching pls or m3u file and be added to the “My Streams” section.
• Create a javascript/web UI for controlling the playback and subscriptions to podcasts and other podcast management tasks (like removing old files, manually downloading older episodes etc) from another computer. Possibly adding bonjour/zero-conf support so the Chumby can be browsed to easily

I would need to see whether those first two could be made to work for my purpose, but it would make sense to just use the existing UI on the Chumby – rather than creating a new one. The existing chumbycast code already provides a javascript/web UI for controlling the playback on the Chumby. This was originally done so I could remote control the chumby, but also so that I could easily create and debug an API for the Flash UI to use.

The other major missing feature is allowing the podcasts to be paused part way through. The current client does not support this, as the audio files are streamed and not downloaded. So that would be the first change to make. Now that I’ve dug out the code maybe I’ll be inspired to play around some more. I’ve also been listening to a lot more podcasts (thanks to Huffduffer) so it might happen yet.

A while back I had a go at using the local storage features being added to javascript to create a simple TODO list app. The main focus of the app was getting it all to fit under 5K (5120 bytes), but it was a good test of using a client-side sql database in javascript.

As the app was written for size it didn’t have many frills. It did however work on the iPhone, as it’s version of Safari had support for the openDatabase call needed. However it didn’t look so good and although the TODO list items were stored locally the phone still had to be online to access the host webpage – which kind of undermined some of it’s utility.

With my recent acquisition of an iPhone I thought I’d revisit this TODO app and make it play nicely on the iPhone. In addition to using a client-side SQL database this new version features:

• Better looks (thanks to iui)
• Ability to run offline (thanks to html5′s offline application cache)
• Drag and drop item sorting (thanks to jquery UI + a bit of touch event hacking)

These features mean that if you have add a bookmark to your homescreen for this app, you might as well be running a native app. It looks pretty native, stores data locally, doesn’t require a net connection and even features standard app UI mechanisms. The main giveaway is of course the chrome associated with the Safari browser. Still not bad for some html, css and javascript. Not a total replacement for native Cocoa apps, but it does put the creation of client-side apps for the iPhone into the hands of even more people.

If you are on an iPhone or are running Safari 4.0 you can try out jTODO yourself or watch the video of it in action below:

I won’t rehash the sql-side of things in this post – instead I’d refer you to my original post on the matter.

The use of iui was also fairly straightforward – I’m only using the style-sheets and images. This means that there are no animations involved, but at this stage it seemed excessive to add them in. Perhaps I’ll add some in a future version.

This leaves the offline application cache and getting drag and drop to work.

Offline application cache

The offline application cache is really simple to implement. It’s currently supported by Safari on the iPhone, Safari 4.0 and Firefox 3+.

In my case I want all files to be cached, so I simple specify them in a “manifest” file:

CACHE MANIFEST

# version 1.0.2.37

iui/backButton.png
iui/blueButton.png
iui/cancel.png
iui/grayButton.png
iui/iui-logo-touch-icon.png
iui/iui.css
iui/iui.js
iui/iuix.css
iui/iuix.js
iui/listArrow.png
iui/listArrowSel.png
iui/listGroup.png
iui/loading.gif
iui/pinstripes.png
iui/selection.png
iui/thumb.png
iui/toggle.png
iui/toggleOn.png
iui/toolButton.png
iui/toolbar.png
iui/whiteButton.png

css/todo.css

js/jquery-1.3.2.min.js
js/jquery-ui-1.7.2.custom.min.js
js/todo.js

img/delete.png
img/deleting.png
img/redButton.png
img/handle.png
img/todo-touch-icon.png

This manifest file is then linked to the main html file thus:

<html manifest="todo.manifest">

That’s it. We now have an offline capable web-app. The main issue I had when doing this, was that Safari was very strict about the manifest – any file not mentioned in the manifest would not be loaded (even if it was normally accessible). The other issue of course is that we’ve now introduced another level of caching, so developing can be a bit annoying – as you think you’ve made a change, but then nothing shows up. I often ended up disabling the manifest for a while when debugging and then re-enabled it after things were working again. There’s also a version number in the manifest file, so that it well register as changed – to help refresh the caches after we’ve made a change.

With all this in place the app can be used when no net connection is available (e.g. in Airplane mode).

Drag and drop

The last job when developing this app was to enable drag and drop for re-ordering items. My first go at this worked pretty easily using jQuery UI’s sortable plugin – when running on my mac in Safari:

            page.sortable({
                axis: 'y',
                handle: '.handle',
                update: function(){
                    db.transaction(function(tx) {
                        $('.todo_item').each(function(i,item) {
                            var id = $(item).find(':input[type=checkbox]').attr('id');
                            id = Number(id.substring('todo_checkbox_'.length));
                            tx.executeSql('UPDATE todo SET todo_order=? WHERE id=?', [i, id]);
                        });
                    });
                }
            });

Essentially this is just setup to enabling drag and drop sorting only along the y axis (up and down), using the element with class handle to start the drag. Once the dragging has finished update gets called and I inspect the DOM to work out the current order of the todo_items and update the database accordingly.

That was pretty easy and working really well on the mac. Then of course I thought I’d test it on the iPhone. At that point I realised I’d forgotten that drag and drop doesn’t normally work in Safari in the iPhone. Holding and dragging normally scrolls the entire page – so drag and drop was a bit useless here.

However after a little digging I found someone had figured out how to get drag and drop working on the iPhone, by hijacking the various “touch” events that the phone generates. These work a little differently to the normal mouse events, but with some work can made to do our bidding:

    function handleTouchEvent(event) {
        /* from http://jasonkuhn.net/mobile/jqui/js/jquery.iphoneui.js
         but changed a bit*/

        var touches = event.changedTouches;
        var first = touches[0];
        var type = '';

        // only want to do this for the drag handles
        if ( !first.target || !first.target.className || first.target.className.indexOf("handle") == -1 ) {
            return;
        }

        switch(event.type) {
            case 'touchstart':
                type = 'mousedown';
                break;

            case 'touchmove':
                type = 'mousemove';
                break;        

            case 'touchend':
                type = 'mouseup';
                break;

            default:
                return;
        }

        var simulatedEvent = document.createEvent('MouseEvent');
        simulatedEvent.initMouseEvent(type, true, true, window, 1, first.screenX, first.screenY, first.clientX, first.clientY, false, false, false, false, 0/*left*/, null);

        first.target.dispatchEvent(simulatedEvent);

        if ( event.type == 'touchmove' ) {
            event.preventDefault();
        }
    }
    document.addEventListener("touchstart", handleTouchEvent, false);
    document.addEventListener("touchmove", handleTouchEvent, false);
    document.addEventListener("touchend", handleTouchEvent, false);

This registers listeners for the touch events, but only does any extra work if the target of the event has the class “handle”. If that’s the case a simulated mouse event is sent for the first touched item. To stop the page from scrolling when we want to drag instead event.preventDefault() is called just for the touchmove event. This is sufficient to let jquery UI do it’s job and enable drag and drop sorting of the TODO items.

When Jeremy talked about Huffduffer at March’s £5app he revealed that pretty much every page in Huffduffer had a RSS/JSON/XML version. A couple of weeks back, I asked whether Huffduffer supported JSONP. After a quick reply back from Jeremy that it wasn’t, there was another reply within an hour or two saying Jeremy had added JSONP support. At that point I felt my curiosity had got the better of me and I felt duty bound to a least do something with the newly added JSONP support…

So here I present a very simple embed-able Javascript widget for Huffduffer:

Both the page used to generate the widget HTML and the widget code itself are 100% client-side. There’s no server-side code involved at all. For the widget itself this seemed pretty key. Such widgets may be embedded in pages that get a lot of hits, so doing anything beyond serving up a static file seemed best to avoid.

I’d not really written a widget like this before, but planned to create something similar for chrss at some point, so it seemed like a good thing to have a go at. Most of the techniques are based on code from Dr Nic’s DIY widgets – How to embed your site on another site article. However I tried to add a few more features/constraints into the mix:

• Allow more than one widget with different data on the same page
• Don’t pollute the global namespace
• Don’t require special html embedded with the widget (just a single script tag)

The first point involved allowing a static javascript file to server different data. The easy way would be to have some server side code serve up some slightly different Javascript depending on the parameters given, but I decided there’s no reason why the Javascript can’t do that itself.

When the widget script runs it looks at the last known script element – which should be the element used to load the script itself to parse the parameters. In this case I just want a single parameter. So if the script tag used to embed the widget looks like this:

<script src="http://psychicorigami.com/huffduffer/build/huffduffer.js?lilspikey" type="text/javascript"></script>

I can extract everything after the ? thus:

    function findURL() {
        // assume that last script in page is this script
        var scripts = document.getElementsByTagName('script');
        var script = scripts[scripts.length-1];
        var scriptURL = script.getAttribute('src');
        var m = scriptURL.match('^.*\\?(.*)');
        if ( m ) {
            return m[1];
        }
        return ''
    }

So that solves the first problem.

The next problem is not polluting the global namespace. First I make sure that everything is wrapped inside an anonymous function that is called immediately. This reduces the scope of the functions declared in there. However we do need at least one global function to act as a callback for the JSONP data. In addition this function must be unique to each widget embedded in the page, so that the correct data gets given to the correct widget.

So first I generate a relatively unique id:


    function uuid() {
        var id = "_"+(new Date()).getTime();
        for ( var i = 0; i < 8; i++ ) {
            id += "0123456789".charAt(Math.floor(Math.random()*10));
        }
        return id;
    }
    var id = uuid();
    var div_id = 'huffduffer'+id;

Then we use that id as a name for the function and create a callback that embeds the id of the div the widget will appear in via a closure:

    function createCallback(div_id) {
        // return a function with the div_id in a closure
        return function(data) {
            // real callback code here
        }
    }

    // create a function with a generated name, to ensure
    // we get the right div
    window[id] = createCallback(div_id);
    document.write("<div id='"+div_id+"'></div>");

This means that we can have multiple widgets on the page all showing different data and the global namespace pollution is limited to one randomly named function per widget.

In addition by creating this unique id we can tie the callback function to the relevant div in the page without needing any more magic. Which means we've also dealt with the third problem.

The "real" widget has been minimized using YUICompressor, but you can see the regular version of the code here:

• huffduffer.js

I've tested everything in most of the major browsers, but can't guaranteed it'll work perfectly everywhere.

It occurred to me that I didn’t really go into much detail about the local storage part of my 5K TODO app. It’s quite a neat piece of tech and because it works in Safari it also works on the iPhone. That obviously means you can use javascript+local storage to create apps for your iPhone without needing to touch the iPhone SDK or learn objective-c. As everything is client-side you also don’t need server-side code to store your app state. The only downside is that local storage is at the mercy of the user. If they decide to reset Safari or generally clear out their user data the local storage goes too! Then again this would be true of files on a Mac’s file-system anyway…

So for any “serious” stuff you might want to look elsewhere, but in the meantime you can still build interesting client-side apps that use local-storage and run on your iPhone.

At the time of writing database storage in html5 is still a work in progress and is only available in Safari/Webkit. In the future it should appear in Firefox etc. Though there’s a HTML5 Wrapper for Google Gears that uses Gears to provide the html5 local storage functions (e.g. openDatabase).

Opening the database

The main function we’re concerned with is openDatabase which let’s us open/create a database that can only be accessed by the current domain (in a similar manner to cookies). In the TODO app I use this code to see if openDatabase is available and output a message if not and stop any further processing:

    if ( !window.openDatabase ) {
        $('add_note').innerHTML = "Browser does not support local storage."
        return;
    }

(note $ is a shortcut to document.getElementById in the TODO app code)

Assuming we find the openDatabase function we can then get a database object thus:

        var dbName = '5KTodo',
             db = openDatabase(dbName, '1.0' /*version*/, dbName, 65536 /*max size*/);

In this case we create/open a database called 5KTodo. We provide a version number (not really used at this stage, but is intended for allowing upgrading database schemas) and a max size. The max size is provided as a hint to the browser as to how big the database will grow. The upper limit for this is defined by the browser (and is about 5M in Safari).

Now once we’ve got our database object we can start running some SQL. We do this using the database object’s transaction method and the transaction’s (tx) executeSql function:

    db.transaction(
        function(tx) {
            tx.executeSql(sql, params, callback);
        }
    );

That will run the sql using the given params (replacing ? in the sql string with the param values to avoid sql-injection attacks) and then calls the callback with the results (if it succeeded).

Creating the tables

The first time we run the app the database will not exist and will contains no database tables, so we need to run some SQL to create the tables:

    function dbTransaction(fn) {
        db.transaction(fn);
    }

    /*create todo table*/
    dbTransaction(
        function (tx) {
            tx.executeSql('CREATE TABLE todo (id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, description TEXT NOT NULL DEFAULT "", todo_order INTEGER NOT NULL DEFAULT 0, done INTEGER NOT NULL DEFAULT 0 );');
        }
    );

On the next run of the application that will fail silently, which is fine as it only needs to run successfully once (to create the tables). It’s key that the create table statement runs in it’s own transaction though, so when it fails it doesn’t stop anything else running. Note that dbTransaction used here is only so that YUICompressor can better compress the code, as this code is from an example for the 5K App.

Reading from the database

After creating the tables (or doing nothing if they are already there) the next step is to see what’s already in the database. Unlike the create table statement we expect to get some data back, so we provide a callback to get the results:

    dbTransaction(
        function (tx) {
            listTODOs(tx, 'SELECT * FROM todo ORDER BY todo_order ASC', []);
        }
    );

    /*run the sql to read the notes and insert them into the DOM*/
    function listTODOs(tx, sql, params) {
        tx.executeSql(
            sql, params,
            function(tx, results) {
                var rows = results.rows;
                for ( var i = 0; i < rows.length; i++ ) {
                    var row = rows.item(i),
                        li = createElement('li'),
                        checkbox = createElement('input'),
                        label = createElement('label');
                    setAttribute(checkbox, 'type','checkbox');
                    setAttribute(checkbox, 'id', 'checkbox_'+row['id']);
                    setAttribute(label, 'for', 'checkbox_'+row['id']);
                    appendChild(label, createTextNode(row['description']));
                    appendChild(li,checkbox);
                    appendChild(li,label);
                    setAttribute(li,'id', 'note_'+row['id']);

                    if ( !row['done'] ) {
                        // not done, so insert above text field
                        notesList.insertBefore(li,$('add_note'));
                    }
                    else {
                        // done so put below text field
                        setAttribute(checkbox,'checked','checked');
                        li.className = 'done';
                        appendChild(notesList, li);
                    }

                    attachTODOHandlers(row['id'],li,checkbox);
                }
            });
    }

(note there are a lot of shortcut functions in there, but hopefully you should get the idea)

What that basically does is execute 'SELECT * FROM todo ORDER BY todo_order ASC' on the database in a transaction (tx) and then takes the results and inserts them into the DOM.

Inserting into the database

Inserting into the database is pretty straightforward - we just use an SQL insert statement. Though in the case of the 5K TODO app we need to do a couple of other things. The first is figuring out the next highest todo_order, then we insert the TODO item and next we re-select the TODO items from the database (with the id of the newly inserted item - results.insertId). The 2nd part is a bit overly obtuse in some respects, but was done so as to help cut down on code size (to fit under the 5K limit):

    function insertTODO(description) {
        dbTransaction(
            function(tx) {
                // set order to be one after items that aren't done
                tx.executeSql('SELECT MAX(todo_order) AS next_order FROM todo WHERE done=0', [],
                    function(tx,results) {
                        var order = results.rows.item(0)['next_order'] + 1;
                        tx.executeSql('INSERT INTO todo (description,todo_order) VALUES(?,?)', [description,order],
                            function(tx,results) {
                                listTODOs(tx, 'SELECT * FROM todo WHERE id=?', [results.insertId]);
                                updateTODOOrder(tx);
                            }
                        );
                    }
                );
            }
        );
    }

As the later SQL statements depend on results from the earlier statement(s) they actually get run in the statement callback functions.

It's fairly straightforward to access a database this way. For those familiar with databases it should all be pretty familiar territory. There's plenty of more info on Apple's site about using the Javascript database. You may also want to look at the source code the the 5K TODO app, though be warned a little bit of readability has been sacrificed for code size.

More 5K App preparations. After using Java to write a 5K Twitter Client it’s time to change languages and dive into some Javascript. So the next 5K app is a TODO list app (a bit like 37Signals Ta-da List). The app runs locally in a browser that support the local storage APIs (which I think is only Safari at the moment, but Firefox and others should support it soon).

Check out the video below, or try out 5K TODO for yourself.

The TODO list app has the following features:

• Add, edit, delete and re-order TODO items
• All TODO items stored in client-side database

So, pretty simple really, but all less than 5K in size. In fact that’s actually the un-compressed side. If that was served up with a gzip filter it would shrink down even further!

Here are the vital statistics:

• 13414 bytes of Javascript, 401 bytes of CSS and 425 bytes of HTML before packing
• 5101 bytes of combined Javascript, CSS and HTML (single file) after packing and running through YUICompressor
• 361 lines of Javascript

So, most of the credit for the final file size should probably be given to YUICompressor. Without much effort it easily knocked off half the size of the Javascript – without much work. However there were a few tricks used to get even more out of the compression.

Apart from removing unnecessary whitespace YUICompressor also renames all local variables and local functions to be as short as possible (typically a single letter). This only happens with local symbols, so the following code:

function myFunc() {
    alert("here");
}
window.onload = function() {
    myFunc();
}

Run through YUICompressor yields:

function myFunc(){alert("here")}window.onload=function(){myFunc()};

Whereas if myFunc() is instead declared inside of the onload handler:

window.onload = function() {
    function myFunc() {
        alert("here");
    }
    myFunc();
}

...it becomes a local function and YUICompressor can do it's work:

window.onload=function(){function a(){alert("here")}a()};

See how myFunc() has been renamed a() when the function is local.

So most of the strategies for reducing the javascript file size revolve around making more variables and functions local and hiding often used global functions (e.g. document.getElementById()) behind local functions.

Apart from that the only other thing I did was to write a script to take the separate html, css and javascript files and generate one html file containing everything. This didn't really save any space (maybe a few bytes), but it meant that the app was only in one file and hence easier to distribute. Plus it also means that on YSlow 5KTODO gets a perfect score - which is always a bonus.

• Download the source (with build files etc)