Sunday, August 31, 2008

Memcache in Reusable Code

If you're writing code for Google App Engine, they highly recommend using Memcache. Here's the deal: when they're measuring what they're going to charge you for, they include CPU and bandwidth. Memory is free. Using memcache to avoid database accesses is a very easy way to increase the number of pages you can serve within your quota. This is especially true if you have complex queries that need to be done for every requested page. Of course, outside of App Engine, memcache is useful for the same reason that Google recommends it.

Memcache has a very simple API. It takes a single key for each cached object. In an application which uses memcache in different places, this means that you must make sure that keys are globally unique. In my App Engine application, I've created a module that I'm planning to release, so that others can use it in their applications (more on that later). But how do I ensure that my keys are unique and won't collide with any application that it gets used in?

The solution: I adopted a very simple key naming convention throughout my application, which I also used in the module. I searched and could find no standard key naming convention for memcache, so I hereby propose this convention as a standard.

All memcache keys consist of a prefix followed by a context-specific unique value. The prefix depends on what's being cached, as follows:

CaseFormatUnique value
Model instanceModel.field=uniqueField value
Class instanceClassName.attribute=uniqueAttribute value
Class dataClassName.meaning:uniqueMeaning-specific value

Notice the use of = for instances and : for class data. Why is this needed? Well, suppose inside a class you decide to cache some data that isn't the instance of the class, such as something that takes a long time to calculate. Now, suppose somebody else decides to cache instances of your class. You might have a key conflict. By only permitting the : prefixes within a class definition, such name conflicts are avoided.

If you're caching an instance of a model or a non-model class, use =. If you're caching something else within a class, use :.

If what you're caching has a single, obvious key, then you can omit the ".field" portion of the key without problems (for example, Model=key). In practice, I think most uses can use this short form, but I'm defining the longer forms now to avoid confusion where they are needed.

I also added this to the App Engine Cookbook.

Saturday, August 30, 2008

Invent Your Own Gadget

The folks over at Bug Labs just sponsored a contest at and after the latest Gnomedex. The question was: what gadget would you make with the BUG platform, a hardware platform of interchangeable modules?

I haven't read a lot about the BUG platform, but I like the idea. Simple, interchangeable modules with (I hope) simple protocols that allow you to quickly prototype a wide variety of hardware gadgets. The current modules provide GPS, motion sensor, accelerometer, camera, and display capabilities. More are on the way (and, in the contest, they said you could postulate future modules).

I posted three gadget ideas -- one that was pretty silly and two that I actually like. If I win the contest, I think I might try to build the last one. Here they are:

Ultimate Remote Control
Where is that remote control anyway? With the Ultimate Remote Control, you don't have to worry about it -- just wave at your TV. The Motion sensor detects when you're waving and activates the remote control. From there, simple hand signals read and recognized through the camera control the TV. Raise or lower your flat hand to change the volume. Hold out your palm to mute. Move your fist to the right or left to change channels or fast forward and rewind on your DVD. Form numbers with both hands to enter a channel code, or use sign language letters to pick a channel by name. Status and confirmations are shown on the LCD. Because there's no physical remote control, everybody in the room can use it any time.
Why I Want It: We have lots of remote controls and they always seem to be lost. If they're not lost, they get fought over.

Help Me Drive
Help Me Drive uses the new radio module to tune in to broadcasted traffic information and combines that with GPS information to alert me of traffic conditions that I care about -- and only those -- via the new text-to-speech module. I don't need to know about an accident 30 miles away unless I'm heading in that direction. Help Me Drive uses a combination of GPS tracking and recording of prior routes to predict which possible traffic issues are a concern. For example, if I'm on 520 heading West between Redmond and 405 and a significant percentage of the time that I'm on that road, I continue past 405 into downtown Seattle, then traffic between me and downtown Seattle is highly relevant. As a bonus, Help Me Drive downloads construction information via WiFi whenever I'm near a preferred or public WiFi network. Plus, by recording traffic information over time, and correlating those with my driving patterns (it knows my speed at every point on my route), it can provide a better predictor of how bad traffic will get on my regular routes. I can also specify traffic that I'm always interested in or that I'm always interested in when I leave a particular destination, like work or home, and I can specify that I'm always interested in traffic between my current location and home or work, depending on the time of day.
Why I Want It: I hate getting stuck in traffic when I could have taken a different route that wasn't congested.

Slow Window
There's a famous science fiction story by Bob Shaw, The Light of Other Days, that introduces the concept of Slow Glass -- glass through which light moves very slowly, possibly taking decades to go from one side to the other. It's an awesome story and highly recommended. My slow window wouldn't be quite so slow, but the camera constantly records and plays back on the screen at a specified delay -- possibly 15 seconds, possibly an hour. 24 hours happens to be a really interesting delay. See what happened yesterday. An optional accelerometer can cause the time dilation to change on-the-fly as you move it around, a pretty cool effect giving you that "special relativity" feel. And, an optional motion sensor can make the slow window delay variable -- it's always showing interesting stuff, looping as necessary.
Why I Want It: Concepts from SF have come to life in so many ways already. I've always wanted slow glass.

Thursday, August 28, 2008

I'm Impressed with Google App Engine

I had a great opportunity today to attend a Google "Hackathon" led by a few members of the Google App Engine team. I decided to go after attending a talk by Mano Marks at StartPad on Tuesday. In short, App Engine is a server infrastructure and SDK for building and deploying highly-scalable web applications. The goal is that App Engine apps are easy to build, maintain, and scale, and are cheap to run. App Engine competes with Amazon's EC2, S3, and SimpleDB, as well as lots of more complex solutions. Before Tuesday, I'd read some articles on App Engine and I'd signed up for an account, but I really knew little about it. But, the talk was pretty compelling, so I decided to jump on the opportunity to attend the Hackathon and learn more.

Walking away, I'm convinced that Google is on to something.

Let me set the stage. Google App Engine supplies a Python framework for developing web apps and all development must be in Python. Before this morning, I had never written a line of Python code. I hadn't even read any before Tuesday. App Engine provides the Django framework for web pages. All I knew about Django was its name. Other than the talk I'd been to on Tuesday, I knew nothing about the App Engine SDK. But Mano had certainly made it look easy.

I didn't go in completely unprepared -- I'm not crazy, after all. Last night, I spent an hour and a half reading the first few chapters of of Dive Into Python. I also bought an O'Reilly Python Pocket Reference that I have yet to open. And, I installed the SDK, though I forgot to install Eclipse with PyDev (the preferred editor for Windows), so I had to do that this morning.

I have a little web app I've been meaning to write for a few years, so I thought that would be the perfect thing to try to create in App Engine. I thought maybe I could actually do the whole thing in a day. That turned out to be aiming a little high. I got hung up on stupid Python bugs, some App Engine gotchas that seem obvious now that I know them, and I hadn't counted on the time it took to download, install, and setup Eclipse. If I had already known Python, I think those things wouldn't have bit me and/or I would have worked through them quicker. Maybe I still couldn't have done my whole app in a day, but I would have come a lot closer.

But -- and this is the amazing thing -- I did get the core part working, and I demoed it to the group at the end. My site actually could have gone live. Sure, it's missing some critical features (like keeping track of users) and it's pretty ugly right now, but it worked. Zero to Sixty in Eight hours. Having App Engine team members around to ask questions certainly helped. And the demos at the end showed me just how much some of the people who already knew what they were doing were able to accomplish in just a few hours.

App Engine is not perfect. It's still a work in progress and there are a few silly things. I got JavaScript errors on some admin pages on their live site. And debugging Python this way is a pain. I don't know what the state of the art on Python debuggers is yet, but I'm guessing it's not anywhere near what I'd like to see (my comparison is the Visual Studio debugger, which rocks). And, certainly, it's not appropriate for everything. Services like Sampa can't be built on it. And, it's not going to work if you have very particular database needs. But, it'll work for a lot of things. And it seems the team is interested in listening and it looks like they've already made changes and enhancements in response to feedback.

We've always talked about leveraging existing code, not repeating ourselves, and sticking to our own areas of expertise. And it's always been hard. Google is really delivering a way for us to do just that in a new and exciting way.

If you're interested in finding out when my app goes live, go to and sign up for the mailing list.