Category Archives: £5app

£5app – Things we built this summer

Just a quick reminder that it’s the next £5app this Tuesday. We’ve got quite a full line-up and we could have had more – it looks like people have been very busy this summer:

  • Ian on hacking a receipt printer with an Arduino
  • Shardcore on the geekery behind his enlightenment machine
  • Jon on an either digestly (or another app he’s been working on)
  • Emily on her iPhone app “Who’s There?”
  • Kyran with some Javascript visualisation demos using webgl
  • Mike with Tardis Bank: on-line pocket money banking for kids
  • Seb on how to be a gameshow host – making Flashy Fortunes with AIR Wiimotes and iPad
  • Jay on predicting the UK’s General Election with Twitter

I’ll be talking about my ultimate Arduino doorbell. It’ll be a short talk, but I hope to bring along the doorbell to demo. I’ve made sure to add a test mode so that I can trigger the bell without needing the the button too (don’t really fancy having to unscrew the button from the door frame).

Don’t forget to signup on upcoming and/or lanyrd if you are going.

5K Morse Code App Using Capslock LED

This is probably my first attempt at “literary programming”, though using that phrase may be taking liberties a little. In this case by “literary programming” I mean – programming inspired by literature. The literature in question is Cryptonomicon by Neal Stephenson. Cryptonomicon already features a Perl script in it’s pages, but that’s not what I’m talking about here. Instead there’s a chapter later in the book when Randy Waterhouse has been incarcerated. However he’s been given the use of his laptop so that – his captors hope – he will decrypt a key piece of data whilst they are monitoring his screen via Van Eck Phreaking. Randy cottons onto this, so he creates an app that will convert text to morse code and have that morse code emitted via the LEDs on his laptop’s keyboard.

I suddenly decided to have a go at creating such a morse code script for my Macbook and here’s the result in action:


You can test the script out by running this at the command line (from the directory with the script in):

echo 'sos' | python morse.py -led

It depends on libraries that are only found in OS X 10.5 Leopard for controlling the capslock LED.

With a bit of compression it all fits well under 5K of code, so makes a worthy example of a 5K app. The morse part of the app is very simple – apart from the lookup table, it could probably fit into 2-3 lines of code! The bulk of the code is simply concerned with driving the capslock key’s LED via the HID. For this I cribbed heavily from some of Apple’s sample code and converted it to run in Python using ctypes.

The only problem I found was that for some reason they internal keyboard on my Macbook stops responding to the request to change the LED state after several seconds. However using this script with an external USB keyboard worked fine. I guess the internal keyboard has some sort of abuse-prevention built-in.

Python Meet and 5K app reminder

The next Brighton and Hove Python User Meet will be the 18th of February 2009 at the Hampton Arms Farm Tavern. As per usual we’ll be sharing the venue with the Farm. Now that there’s a google group setup for these meets they might happen a bit more often, as Ian and I will no longer be the limiting factor!

A quick reminder too that the next £5app meet is now only a few weeks away on the 3rd of March. Jeremy Keith will be talking about Huffduffer (which I’ve been using a lot for getting great podcasts for the walk home after work) and we’ll be running the 5K app competition. The deadline for the 5K app is the Friday before the event (27th Feb), so there’s just under three weeks left to get something written.

If anyone wants to chat about the 5K app they’re welcome to pop along to the Python User Meet too. I’m more than happy to spend some time helping people get their code under the 5K budget.

A 5K Java Guitar Tuner

So after writing a 5K Twitter Client, a 5K TODO app and a 5K Fullscreen Text Editor I thought that’d be enough example apps to get things started. However while adding a list of app ideas to the 5K App page I suddenly decide that one of the ideas really appealed. The idea was to write a guitar tuner, that would listen on the computers mic and show the user what pitch it thought the note played was. I’m not much of a guitar player, but when I do play it’s always nice to have a guitar that’s in tune. As I’m unable to tune a guitar by ear I rely on guitar tuners and I always seem to be out of the 9V batteries my electronic tuner needs. Given my laptop is usually nearby, being able to use it as a guitar tuner would be most useful.

Check out the video below or download 5KTuner for yourself (requires Java):



The app is more or less non-interactive. It simply starts up tries to work out what note is being played. It shows a simple slider that moves according to what note is being played and indicates how close to that note you are. In addition the raw frequency (in hz) is shown, as well a chart that shows the autocorrelation function of the input sound. A red line is drawn on this chart when a “valid” note is heard and indicates the wavelength of the overall frequency.

The vital statistics are:

  • Final (obfuscated) app size: 3538 bytes
  • Final (un-obfuscated) app size: 4649 bytes
  • Final Source-code size: 8532 bytes
  • Lines of code: 245

This was my first foray into audio programming and I’m grateful that in the end it wasn’t too tricky. Initially I’d been looking into using the Fast Fourier Transform to work out the frequencies, but the autocorrelation function was much easier. It’s essentially just two for loops and a bit of maths. You listen for a certain amount of audio and then compare the first half of the sample with itself, but shifted. By working out the amount of shift where the difference is smallest you can find the overall frequency of the sample.

It could probably be made more robust, but if you want to see how it works (or improve it) you can download the source code.

A 5K Python Fullscreen Text Editor

After a 5K Java Twitter client and a 5K Javascript TODO list app, the next example 5K App is a wxPython fullscreen text editor. Much like Writeroom and Duncan’s JDarkroom (both of which are more fully featured) it lets you edit text files in “distraction-free” fullscreen. Ideal for creative writing or similar where you want to just get on with writing without the distraction of the outside world. So at the risk of invoking the apoplectic rage of Mark Pilgrim a fullscreen text editor seemed quite feasible in 5K of Python. In fact after using my self-extracting script code I actually had to add more features as I was way below budget!

Check out the video below or download 5KEdit for yourself.



Features:

  • Fullscreen text editing
  • Open/Save files
  • Undo/Redo
  • Retro looks
  • Resizable fonts
  • Configurable colours
  • Word count
  • Goto line number dialog
  • Find text dialog
  • Help page listing commands
  • About box

It’s actually a bit more feature-full than I’d thought it would be. Part of this was that wxPython’s wx.StyledTextCtrl and wxPython in general provided quite a lot of what I needed (e.g. undo/redo) out of the box. Plus the self-extracting script technique really did a good job of shrinking the code down – allowing me to add more.

To run 5KEdit your need Python (2.4+) and wxPython (2.8+) installed. If you are running OS X Leopard you already have those installed, so you can simple switch to a terminal and run that 5KEdit script using the command:

pythonw 5KEdit.pyw

Make sure you run that from the directory 5KEdit.pyw is in. On a Windows system you may find that you can simply double-click on the .pyw file to run it. There’s a small repaint bug under Windows when you change colors, but it’s not a show stopper.

Here are the vital statistics:

  • 17284 bytes/485 lines of original source-code
  • 13767 bytes/379 lines of stripped source-code
  • 5069 bytes of compressed/packed final version

As you can see the final version is less than a third the size of the original source code. A large part of this is due to the compression, but to help push it a bit further the source was also “stripped”. Blank-lines and comments were removed and the indentation was changed from my usual 4 spaces to a single space. The stripping doesn’t make a huge contribution, but without it the final version would be 5421 bytes in size. So for 400 bytes or so it’s probably worth it. Plus it means that you don’t have to worry about leaving yourself comments or spacing out your code nicely.

Packing Python scripts for the 5K app

So far for the 5K App I’ve written two example apps – one in Java and one in Javascript. This is partly so that I can figure out how the rules for the competition will need to work. In a lot of 4K competitions a single language is chosen, so things are a bit more straightforward. However Brighton seems like quite a varied place and allowing multiple languages for the competition seems like a nice friendly inclusive thing to do.

So the next example app I’ll be writing will be in Python. As Python source is the executable code there’s no compilation step (unlike Java) and un-like Javascript there aren’t really many tools for reducing code size. Python (like Ruby, Perl and PHP) tends to result in fairly concise code, so normally this isn’t too much of a problem. However 5K is quite a tough constraint and we really want to squeeze the most out of our code by compressing and packing it if possible.

So here’s a little script for compressing a Python script to make it much smaller:


import sys
import bz2
import base64

write=sys.stdout.write

for name in sys.argv[1:]:
    contents=bz2.compress(open(name).read())
    write('import bz2, base64\n')
    write('exec bz2.decompress(base64.b64decode(\'')
    write(base64.b64encode((contents)))
    write('\'))\n')

To use it save that code into a file (e.g. pack.py). It will write the compressed code to standard output (i.e. the screen), so you can redirect it to whatever file you want:


python pack.py my_script.py > my_script_packed.py

The compressed code looks like this:


import bz2, base64
exec bz2.decompress(base64.b64decode('<base64 encoded compressed data>'))

Which hopefully is fairly clear as to what it’s doing, but to summarize:

  • The script data is base 64 decoded into bytes
  • The bytes are then decompressed (bz2) into the text of the original script
  • exec is then called to run the original script

One nice benefit to this way of compressing the script is that the final module namespace (after de-compression) will look essentially the same as it did if it was not compressed. The only difference is the bz2 and base64 modules will also be present. This should mean that you can actually compress multiple Python files and importing from them should still work. Though of course as is the case when adding an extra layer of complexity your mileage may vary…

For an idea of how effective this compression can be I took the Python script for calculating pi on wikipedia and ran it through the script. After confirming that it ran the same, a quick comparison revealed the compressed version had gone from 12658 bytes to 3421 bytes – less than a third of the original size.

It should be possible to create similar scripts for Ruby (using zlib and base64), Perl (using Compress::Zlib and MIME::Base64) and PHP (using bzdecompress and base64_decode).

As I work on the example Python 5K app I should hopefully get a good feel for how the competition rules might need changing to allow “scripting” languages like Python, Ruby, Perl and PHP. I think the plan will be that the app must be contained in a single file (less than 5120 bytes in size), with any resources embedded in the file. This is the norm for Java and Flash, but will probably require an extra packaging step for most other languages/runtimes. That single file can then be run either via a GUI (double-clicking) or via a standard invocation from the command line (e.g. python my_script_packed.py) using only a “standard” version of the language runtime. Note that the standard installed version of Python on MacOS X 10.5 (Leopard) includes quite a few extra libraries (e.g. wxPython) so these libraries would be eligible for use in the 5K app. The same will be true for Ruby and Perl, so that should hopefully help open things out. Otherwise Java’s large standard library might give it too much of an advantage…

A 5K javascript, local storage backed, TODO list app

More 5K App preparations. After using Java to write a 5K Twitter Client it’s time to change languages and dive into some Javascript. So the next 5K app is a TODO list app (a bit like 37Signals Ta-da List). The app runs locally in a browser that support the local storage APIs (which I think is only Safari at the moment, but Firefox and others should support it soon).

Check out the video below, or try out 5K TODO for yourself.



The TODO list app has the following features:

  • Add, edit, delete and re-order TODO items
  • All TODO items stored in client-side database

So, pretty simple really, but all less than 5K in size. In fact that’s actually the un-compressed side. If that was served up with a gzip filter it would shrink down even further!

Here are the vital statistics:

  • 13414 bytes of Javascript, 401 bytes of CSS and 425 bytes of HTML before packing
  • 5101 bytes of combined Javascript, CSS and HTML (single file) after packing and running through YUICompressor
  • 361 lines of Javascript

So, most of the credit for the final file size should probably be given to YUICompressor. Without much effort it easily knocked off half the size of the Javascript – without much work. However there were a few tricks used to get even more out of the compression.

Apart from removing unnecessary whitespace YUICompressor also renames all local variables and local functions to be as short as possible (typically a single letter). This only happens with local symbols, so the following code:


function myFunc() {
    alert("here");
}
window.onload = function() {
    myFunc();
}

Run through YUICompressor yields:


function myFunc(){alert("here")}window.onload=function(){myFunc()};

Whereas if myFunc() is instead declared inside of the onload handler:


window.onload = function() {
    function myFunc() {
        alert("here");
    }
    myFunc();
}

...it becomes a local function and YUICompressor can do it's work:


window.onload=function(){function a(){alert("here")}a()};

See how myFunc() has been renamed a() when the function is local.

So most of the strategies for reducing the javascript file size revolve around making more variables and functions local and hiding often used global functions (e.g. document.getElementById()) behind local functions.

Apart from that the only other thing I did was to write a script to take the separate html, css and javascript files and generate one html file containing everything. This didn't really save any space (maybe a few bytes), but it meant that the app was only in one file and hence easier to distribute. Plus it also means that on YSlow 5KTODO gets a perfect score - which is always a bonus.

A 5K Twitter Client

So as part of preparations for the 5K App I’m writing a few example apps. The first one so far is a 5K Twitter Client, which makes it smaller than the Twitter logo itself (at 5.4K).

Check out the demo video to see it in action or download 5KTwit (requires Java):



The features are:

  • Update twitter status
  • View friends timeline (new tweets checked for every 3 minutes)
  • Easily change logged-in user
  • Count of characters remaining when typing status message

It doesn’t handle viewing direct messages, but I don’t tend to use them. Plus I wanted to make sure the features I had were done fairly well.

Anyway, I thought it would be a good idea to walk through making a 5K app, as there’s a fair bit of stuff just needs to be done to make it feasible. It’s the sort of stuff you don’t normally encounter daily when working on “real” software – unless you’ve been working on mobile or embedded apps. Nowadays most programmers don’t really need to worry about the size of the code they write, so how to make something that’s less that 5K in size isn’t immediately obvious.

The Twitter app (which I’ve lovingly christened “5KTwit”) is written in Java and runs as a standard client-side application. For some good pointers on how to make a Java app that’s really tiny, but packs a punch you should probably start with these Java 4K tips.

Now for a few numbers:

  • Final (obfuscated) app size: 5119 bytes
  • Final (un-obfuscated) app size: 6400 bytes
  • Final Source-code size: 10398 bytes
  • Lines of code: 310

As you can see the difference between the obfuscated and un-obfuscated versions is about 1K, so obviously obfuscation is pretty key. So it’s worth starting of with a good obfuscator. For those going “what’s an obfuscator?” it’s basically a tool to optimise, shrink and scramble your code. Originally they were intended to make code harder to decompile (hence “obfuscate”), but also have the added benefit of shrinking the compiled code to. I used Proguard and that’s probably your best bet too.

Proguard does a pretty good job of shrinking your code, but it’s worth reading up on how it works, so you can get an idea of what code changes you can make to get the most effect. For example Proguard will inline (and remove) small private functions for you (so they effectively have no overhead), but only if they are only called in one place. In general you probably want to use a source-control system/backup changes so you can rollback when a change doesn’t result in smaller obfuscated code.

Here’s the config file I used for Proguard (running on OS X):


-injars      dist/5KTwit.jar
-outjars     obfuscated/5KTwit.jar
-libraryjars <java.home>/../Classes/classes.jar
-optimizationpasses 2
-allowaccessmodification
-keep public class T  {
    public static void main(java.lang.String[]);
}

Proguard does a great job, but it won’t stop you writing code that’s bigger than it needs to be. In order to get the app under 5K I only used two classes (one for the main window and one for the username/password dialog) and one resource file. Each additional file has some overhead associated with it. Not only will each file tend to have some standard header info etc, but even if it was empty it would increase the size of the packaged jar file. So really cutting down on the number of files used makes a big difference. During testing I made a simple HelloWorld application and that was 700 bytes or so in size with a single class file and no real code. Most of that size would have been taken up with this sort of overhead.

As this app used swing for the GUI it also meant I had to be a bit creative about event handling. The norm is to have small event handling classes, but that would have incurred too much overhead. Instead I resorted to tricks like this:


    public void actionPerformed(ActionEvent e) {
        if ( e.getSource() == prefs ) {
            start();
        }
        else {
            String status = this.status.getText();
            if ( status.length() <= MAX_CHAR ) {
                tweetsDoc = loadDocument("http://twitter.com/statuses/update.xml", status);
                loadTweets();
                this.status.setText("");
            }
        }
    }

Where I checked the source of the event to see which button had been pressed, rather than having a separate event listener for each button.

I also had to deal with listening for window resize events, as there was some manual intervention required to make tweets wrap inside the scroll pane. So for that I circumvented the listener approach entirely and overrode processComponentEvent. In this code I had to try to estimate how big the panel containing the tweets would be, when I knew the width I wanted. So I resorted to a bit of nasty hackery and repeatedly set/unset the preferred size a few times so it would settle on the right height…

    protected void processComponentEvent(ComponentEvent e) {
        if ( e != null ) { super.processComponentEvent(e); }
        
        // force width of tweets to match width of window
        for ( int i = 0; i < 3; i++ ) {
            tweets.invalidate();
            tweets.setPreferredSize(null);
            int height = tweets.getPreferredSize().height;
            tweets.setPreferredSize(new Dimension(scrollPane.getViewportBorderBounds().width,height));
            scrollPane.validate();
        }
    }

Whereas in "real" code this would probably be done with a bit more finesse (and a few more classes).

Another tricky spot involved the background thread used to check for new tweets. This ran on another thread (rather than the main GUI thread), so as not to lock up the GUI when it runs (once every three minutes). However updating the GUI had to occur on the GUI thread. Normally this would simply be done by creating a few Runnable tasks, but that would have meant more classes. Instead I end up checking to see whether we're running on the GUI thread and change behaviour if that's the case:

    public void run() {
        if ( SwingUtilities.isEventDispatchThread() ) {
            // update UI as running on the right thread
            loadTweets();
        }
        else {
            while( currentThread == Thread.currentThread() ) {
                String friendsTimelineURL = "http://twitter.com/statuses/friends_timeline.xml?count=200";
                if ( lastTweetID != null ) {
                    friendsTimelineURL += "&since_id=" + lastTweetID;
                }
                tweetsDoc = loadDocument(friendsTimelineURL, null);
                try {
                    // invoke and wait, to run loadTweets()
                    // on awt thread
                    SwingUtilities.invokeAndWait(this);
                    Thread.sleep(1000*60*3); // 3 mins
                }
                catch( Exception e ) {
                    // break loop
                    break;
                }
            }
        }
    }

There are probably quite a few other places where I had to make this sort of "design" choice. It's feasible to do for this little code, but I wouldn't want to do this kind of thing in a big project. Though I guess if it was really needed you could write something that would take nice well written code and convert it to use these sorts of tricks...

Apart from that, most of the rest of the work came down to carefully weighing up which features I wanted and carefully tweaking and re-organising the code to make it smaller. I also found that the single image (for the preference button) was a lot smaller when saved as a gif rather than a png (90 bytes vs ~200 bytes).

Anyway, for 5K it's not a bad little client. I've actually been using it for the past couple of days (as I've been writing it). I doubt it would really survive under the hands of a more active user, but it does the job.

The 2nd £5app xmas special

Well whilst I’ve been ill it looks like Ian has been on an organizing rampage. So as well as Aleks talking about Spaceship! we’ve also got several shorter talks/demos on:

  • Lastminute.com Labs with mobile-phone based lightsabers (bluetooth+accelerometers)
  • Dominic Mason with a 3D iPhone game
  • Seb with Flash 3D Xmas games
  • Emily with fighting mini-Sumo robots
  • Tristan Ben Rubinstein with “eye-pong”

Plus from the flurry of other emails appearing it looks like there are quite a few other possibilities for the £5app meet next year too. Excellent work Ian!

The xmas (gaming) special is at the Werks* on the 10th of December, so don’t forget to signup on upcoming.

* If you are feeling charitable perhaps consider donating a little to the Werks projector fund.

Next £5app – September 18th 2008

The next £5app meet is a week tomorrow (Thursday 18th September). Just as last time it’s going to be a Democamp and so far we’ve got several demos lined up (including myself with the Chumby Podcatcher). See the wiki page for more details on the demos.

If you have something you want to demo please let us know (demo AT five pound app DOT com).

It should be a good night, so don’t forget to sign-up on upcoming and I hope to see you there.