Not only is appalling that this web host didn’t have any off-site backups, but that those who are claiming to have had their businesses destroyed or livelihoods lost as a result of the hack would rely on their web host’s internal backups as their only data redundancy is completely ridiculous in this era.

Everyone, especially businesses, if you don’t have a backup and recovery procedure in place that can handle a complete loss of any one site, and any other servers whose keys are stored on machines at that site – get one today

On Saturday night one of my housemates thought it would be a good idea to apply the concept of Mark Zuckerberg’s now-infamous ‘FaceMash.com’ website to professional StarCraft II players. SC2 has a huge community that would probably find something like this very funny, and we were bored and needed something to code – the perfect opportunity, so at 8pm we got to work on StudCraft2.com.

Google App Engine

We weren’t sure how much traffic this site would generate, so a distributed, scalable hosting solution seemed like a good idea; we also wanted to use a framework with a simple deployment mechanism and an easy to use database API; so I decided to write the main site using appengine and Google’s ‘webapp’ framework, while my housemate worked on some Perl scripts to scrape team sites and the sc2wiki for player information.

This was my first real foray into appengine, and we were coding through the night hoping to deploy as soon as possible, so please take these code samples with a grain of salt, the approaches taken here may be suboptimal, non-idiomatic and brilliant examples of poor design.

First I had to design the models, like this one, that would be stored in the appengine datastore:

class Player(db.Model):
   """Models a StarCraft II player"""
   handle = db.StringProperty()
   name = db.StringProperty()
   team = db.ReferenceProperty(Team)
   race = db.ReferenceProperty(Race)
   nation = db.StringProperty()
   photo = db.StringProperty()
   score = db.FloatProperty()
   votecount = db.IntegerProperty()

Next was the class that handles the main page of the website:

class MainPage(webapp.RequestHandler):
   def post(self):
      if self.processForm():
         self.redirect("/")

   def get(self):
      self.showVotePage()

   def showVotePage(self):
      ...

   def processForm(self):
      ...

   def updateScore(self, winner, loser):
      ...

After processing the form in post(), we redirect to get() so that clicking ‘refresh’ doesn’t cause a browser message asking you to resubmit post data. The processForm method figures out which pair of players you are voting on, who won, and whether it’s a legal vote (more on this later). If all that checks out it calls updateScore.

Updating scores

def updateScore(self, winner, loser):
   if winner is not None and loser is not None
         and winner.key() != loser.key():
      K = 20
      winnerQ = math.pow(10, winner.score/400.0)
      loserQ  = math.pow(10, loser.score/400.0)
      winnerE = winnerQ/(winnerQ+loserQ)
      winnerS = 1.0
      loserE  = loserQ/(winnerQ+loserQ)
      loserS  = 0.0
      winner.score = winner.score + K*(winnerS - winnerE)
      loser.score  = loser.score + K*(loserS - loserE)
      winner.votecount = winner.votecount + 1
      loser.votecount = loser.votecount + 1
      winner.put()
      loser.put()

This is a simple implementation of the ELO rating system, as used in chess tournaments. The variables winnerE and loserE are the expected points of the two players in the matchup, based on their current scores. If winner and loser have equal scores, both expected values will be 0.5, the further ahead the winner is, the closer their expected value is to 1, and the loser’s expected value is to 0, in every case winnerE + loserE is equal to 1. The scores are then updated according to the difference between their expected scores and actual scores, multiplied by a fixed learning rate K, in this case 20.

This implementation is not thread-safe, because the score of a player may change between when it is read and when it is put, or ever between when the winner is put and the loser is put. Again, more on this later.

Duck-typing is your friend

A couple of hours into writing this app, we decided that we wanted to rank Starcraft II teams and races, not just individual players. So, I had a look at the above updateScore method to see what needed to change, the answer: nothing at all!

I often hear people (including myself) complaining about Python’s duck-typing being responsible for all sorts of hard-to-find bugs, but this was a situation where it really shines. All that is needed for an object to work with the updateScore method, is for it to have key() and put() methods, and score and votecount properties. So I designed the other models to match:

class Team(db.Model):
   """Models a StarCraft II team/clan"""
   name = db.StringProperty()
   score = db.FloatProperty()
   votecount = db.IntegerProperty()

class Race(db.Model):
   """Models a StarCraft II race (ZPTR)"""
   name = db.StringProperty()
   score = db.FloatProperty()
   votecount = db.IntegerProperty()

In other object-oriented languages like Java, to achieve the same effect of having a generic updateScore function that works with several object types, I would have to create a ScoredObject interface or abstract class with the required parameters; then have each of Player, Team and Race subclass or implement ScoredObject.

While you may argue that this more explicit type hierarchy makes things easier to reason about, the sheer number of lines of code that are required (in contrast to the duck-typing approach), makes it a poor choice for a rapidly developed application like this one.

Beware the script-kiddie

The StarCraft II community is home to a number of programmers and other tech-savvy users, who may attempt to sway the results of the voting site in favour of their personal preferences by cheating the system. To combat this, I wanted to design the site in such a way that it would detect repeated requests from the same user, or forms that have been modified to vote on a particular pair of users rather than the ones randomly allocated for that request. This is what I came up with:

class SpamHash(db.Model):
   player1 = db.StringProperty()
   player2 = db.StringProperty()
   secret = db.StringProperty()

Each time we serve up a page with a voting form, we create a new SpamHash object:

spamhash = SpamHash(player1=unicode(player1.key()),
                    player2=unicode(player2.key()),
                    secret=base64.b64encode(os.urandom(8), "*!"))

The form contains the players string-encoded keys, and a random 8-byte string, in hidden fields. The JavaScript on the page itself sets a ‘winner’ field to 0 or 1 when a user clicks a photo or presses the left or right arrow keys.

When we receive a post() request, we look up the matching SpamHash and delete it:

matchingSpamHash = SpamHash.gql("WHERE player1 = :1 AND player2 = :2
   AND secret = :3", player1ID, player2ID, spamhash).get()
if matchingSpamHash is not None:
   matchingSpamHash.delete()
else:
   ... # redirect to error page

This way each form we serve can only be used once, and only for the pair of players we randomly assigned. The system accumulates unused SpamHash objects over time, when people visit the home page but do not vote. If that ever gets out of control I’ll add timestamps to the SpamHash and create a CRON job that deletes old ones from time to time.

Going live

After a few finishing touches, a dodgy script to load up all the data, and a page for viewing the top-scoring Players, Teams and Races, it was time to go live. We started by just posting a link on the TeamLiquid.net forum, a popular hang-out for the SC2 community in the US, Australia and Europe, and sat back to watch the vote count grow.

We hit 1000 votes in the first 20 minutes, 5000 in the first hour and 8000 in the first two hours. Things have died off since then, but at our peak we were serving 24 page requests per second.

Transactions, Atomicity and Future Work

Earlier I mentioned that the score update function is not thread safe, in fact neither are the counter increments.

AppEngine provides a way to create transactions that are guaranteed to succeed or fail in their entirety, maintaining data integrity. To ensure that the ELO updates have a zero-sum net effect on the total number of points in the system, all the Player objects must be put in the same entity-group. When two simultaneous transactions that operate on the same entity-group are in progress, the one that tries to commit second will fail, effectively causing a vote not to be counted. Ideally we would like to add all votes to a queue of jobs to be processed atomically when the server has time (but I’m not sure how to do this with AppEngine).

As for the counters, there’s a simple pattern called a ‘sharded counter’ that gets around these sort of contention problems. A counter is split into N ‘shards’, each of which is a root entity, and when you want to increment the counter, you choose a random shard to increment in a transaction, then there’s only a 1/N chance of a collision, so setting N to a number a few times larger than your expected number of increments/second should allow almost all the transactions to succeed. When you want to print the counter, you just add up all the shards. Perhaps it’s possible to come up with a sharded version of the ELO update algorithm, but in general ELO updates are not commutative.

So, if we have time, we might update the StudCraft2 code to be thread safe so that we stop ‘leaking points’ as we are now, and perhaps make the system generic enough to be used for other types of pairwise-voting-comparison websites. Either way building this website was a lot of fun, and I learnt a lot about using Google AppEngine!

All the code is available on my GitHub account.

The current setup is thus:
The first disk is 500GB (465.76 logical) and has 3 partitions, a 100GB Windows XP partition (to be replaced), followed by a 2GB swap partition and the rest (370GB or so) is taken up by my Ubuntu root partition, no separate /home. Then there are 3 400GB drives in RAID0 using mdadm, that logical volume has one big ext4 partition (Barclay).

The plan is simply to erase the XP partition and install Windows 7 in its place, then update grub from a live CD. In the future I may end up shrinking the Ubuntu partition (since all the big stuff lives in Barclay), move the swap across, and grow the Windows 7 partition… apparently GParted can do that these days.

This all sounds fairly straightforward, hopefully the Windows installer doesn’t decide to do something dumb, like install onto one of the 400G drives it doesn’t recognise as being formatted without giving me a chance to tell it where to install. The trickiest part will be running updage-grub. Jayen has suggested that it would be a BadIdea^TM to just run the update-grub found on a live CD as I usually do, and should really run the one installed on my Linux system, since differences between versions could cause problems. Assuming my first HDD is sda, and the root partition is sda3, this should be a matter of firing up the live CD and running:


$ sudo su -
$ mount /dev/sda3 /mnt
$ mount -B /dev /mnt/dev
$ mount -B /proc /mnt/proc
$ mount -B /sys /mnt/sys
$ chroot /mnt
$ update-grub
$ grub-install /dev/sda
$ exit


I’ll update this post with how it all goes!

Update:
Everything went roughly as planned, except for one minor hitch. I was feeling paranoid about my RAID array, and I don’t have enough SATA cables for the CD drive as well as 4 hard drives, so I unplugged the RAID drives before the install. After doing the update-grub and plugging everything back in, the RAID array wouldn’t assemble! Turns out I swapped 2 of the SATA cables (and mdadm is that finnicky?).

Only other unexpected thing was the Windows installer making 2 partitions instead of one, so I needed to mount /dev/sda4 rather than /dev/sda3. Apparently the extra partition is a Windows equivalent of /boot.

I’m already getting annoyed at using Windows and associated tools:

• There’s no PDF reader built in. How am I supposed to read… anything?! Is Adobe paying them to leave this out?
• Seriously need package management. Why I have to follow a 3-step process to install a sequence of updates to the Windows Phone dev tools is beyond me. Also, I now have a dozen useless .exe’s sitting in my Downloads directory (perhaps if I downloaded to /tmp…).
• To do Windows Phone development, even if you’ve already installed the full-blown ‘Ultimate’ edition of VS2010, you have to go back and install the ‘Express’ version.
• Highlight-copy/middleclick-paste is definitely the coolest feature of X. Anyone know how I can make Windows Shell do this?

On the plus side, useful codecs are installed by default in WMP, and Daemon Tools is easier to use than ‘mount -o loop’.

I thought I’d take the quiz recently posted on PhD Comics, to see what score I get for ‘school spirit’ after 5 months in Texas.

Mascot’s name: Bevo – 1 point
Colors: Orange – 1 point
Fight Song: “The Eyes of Texas”, goes to the tune of “I’ve Been Working on the Railroad”, but I don’t know all the words.. I’ll give myself – 1 point out of 2
NCAA Division: What’s NCAA? Some sports thing? Big 12? Looking this up it turns out we’re in ‘Division 1′ and ‘Big 12′ is indeed the conference.. I’d heard the term but didn’t know what it was about – 0.5 points out of 1
Main Rival: Texas A&M – 1 point
Name of the Head Coach – No idea – 0 points out of 1
Bonus for going to the stadium – 2 points (been there once)

Final score: 6.5 out of 8

Since I’m not a grad student the comic’s interpretation doesn’t really make sense, but I feel like I did OK for a ring-in

Austin has a vibrant community of Python users, there were around 20-30 people in attendance, most of whom use Python at whichever company they work for, and some of whom have never used Python but are keen to learn. We had people from the fields of web development, scientific computing, HPC, the media, as well as a few students, including one who studies economics at UT.

What surprised me the most was the presence of two Kiwis and another Aussie, who happened to be a UNSW mathematics graduate now working at Enthought – small world!

The session kicked off with a presentation from Peter Wang, the organiser of APUG, on Cython, which I found particularly interesting as I am currently looking into various solutions for C++<->Python interaction for my work on RoboCup, including SWIG and Boost::Python. Today’s presentation made me think for the first time of perhaps doing it the other way around – writing the system entirely in Python, with all the performance-critical parts implemented as C(++) extensions.

The second talk was from one of the New Zealanders, visiting from the University of Auckland, who was presenting his Python library for linear programming, PuLP.

After the main talks everyone introduced themselves, and suggested ideas for 10 minute lightning talks for the December meetup. I volunteered to talk about embedding Python in soccer-playing robots, so it’ll probably be an expansion of the 5 minute talk I gave on the same topic at PyConAu, not too long ago.

An aside: I must apologise to everyone who I said I would keep updated about my time in Austin via this blog, as you can see I haven’t posted since I left New York back in August. The truth is I’ve been having too much fun and keeping too busy here in Austin to have time to sit down and write about it. I had the topics of several blog posts lined up, but I’ve now decided I’m going to write them in the reverse order to which I’d intended, so expect to see the following posts on the following coming up soon:

• Joel Spolsky’s FogBugz & Kiln World Tour – why you should all use Kiln or something like it
• The University of Texas’ innovative IP quota system
• Configuring the Atlassian tools (Confluence, Jira, Crowd, Fisheye & Crucible) for rUNSWift on a clean Ubuntu 10.04 system
• Why programming is the only thing I don’t do in my web browser
• ‘First’ impressions of Austin, UT and the public transport system

So.. hopefully I’ll get around to all that sometime soon

Update 15/08/11: I never got around to writing these, and probably never will. Sorry!

The flight left Sydney Airport on time, around 10am Wednesday August 4th, and the captain immediately told us that due to wind conditions we’d be landing about half an hour late. I was sitting next to a couple from Virginia who were keen to suggest a hundred and one things I need to see whilst in the USA.

On arrival in the Los Angeles area, we were again delayed because the runways were full-up with planes taking off, they’d been delayed by fog, so after 20 minutes of circling we were finally allowed to land at 6:40. Getting my 8:00 connecting flight to New York was starting to look pretty tight, as I still had to clear customs and immigration, and drop off my bags for the connecting flight.

Once on the ground, another 30 minute delay ensued, since our gate was still occupied by an aircraft that was supposed to have left half an hour earlier, while docking at 7:10, we were given stricter-than-usual instructions to stay in our seats. It soon became apparent why, the LAPD boarded the plane and arrested a drunk Australian traveller in his seat. Finally getting off the plane at 7:25, I was through customs at 7:50, and redirected to the ‘Special Services’ desk to have a replacement flight booked to New York.

‘Special Services’ is a row of just 3 check-in counters, with a queue of at least 400 people, as it seemed almost everyone on the flight from Sydney had missed a connection. Finally making it to the front, I was booked on a 2pm flight and handed off my bags. Not wanting to risk missing this next flight, though I had more than 4 hours to spare, I proceeded through security immediately, and just as well as it took over an hour. My overall impression of LAX is that it is the worst international airport I’ve been to: plain, crowded, disorganised, uncomfortable. The only upside was the fantastic Mexican food I had for breakfast/dinner (choose your time zone!) whilst waiting.

The flight to NYC proceeded on time and arrive an hour early, but naturally there was once again a 30 minute delay getting to the terminal. Immensely confused by the transport system, I took a cab to the hostel in Manhattan, arriving at 11pm, where I told by the Australian working at the front desk that despite my booking 4 months in advance, there were no beds available. After a lengthy chat with his manager, they managed to get me a bed in one of the 12-bed dorms rather than the 4-bed one I’d booked, where at least I was able to attempt sleep at half the price. I say ‘attempt sleep’ because of the sonorous snoring of my dorm-mates, the industrial air-conditioning unit next to my bed, as well as the constant arrivals of late night party-goers. Fortunately on the first night I was sufficiently jet-lagged that none of this mattered and I slept from midnight until 5am!

Thursday. After an early awakening, I ventured out into the New York subway system, which I must say is one of the finest features of the city. A very efficient and convenient way to travel, though I must say it took me a while to figure out what a ‘local stop’ was. A series of lines run North to South the length of Manhattan, and a few lines cross East to West, these are supplemented by city buses that take the same type of ‘Metrocard’ ticket as the subway. Arriving at Times Square before 7am, my sense of time still distorted, I got the impression that it was a reasonably busy part of the city. Going back there later on I realised that first morning it was practically deserted in comparison. I continued walking South down Broadway, amused by the M&M Megastore until the first CitiBank I reached, having just opened at 8am. The lady who assisted me in opening an account seemed to think it was a little over-zealous to spend the first hours of your first day on vacation organising details like bank accounts, perhaps so but I wanted to get things sorted out as early as possible. The issued me with a temporary ATM card on the spot, that I can use until my real debit card arrives in Austin.

The rest of the day was mostly spent walking and transiting around the city. The Staten Island Ferry is definitely the best way to see the Statue of Liberty without having to wait in hour-long lines for the ferry to Liberty Island. I stumbled across a neat little deli/green-grocer/kitchenwares store called ‘Dean & Deluca’, which seemed a much more friendly place to buy one’s dairy, meats, etc. compared to a typical supermarket. It turns out a number of other Dean & Deluca stores exist around th city, but all the other ones I saw were coffee shops. On Wall Street, there was a building with a touchscreen wall, where one can enter their email address to be sent coupons. The city has a very visible police presence, almost as big as that in London, and some very unusual police vehicles. After an exhausting 8 hours of walking, I went back to the hostel where I signed up for “Dave’s Downtown Tour” for 9am the next morning, and fell asleep at an early 8:30pm.

Rumours that there’s a Starbucks and 7-Eleven on the corner of every city block are greatly exaggerated, I didn’t see a single 7/11 in my time in New York, and the Starbucks are only as dense as they are in Sydney (which means there are quite a lot of them, but they’re not exactly crowding). There happened to be a Starbucks on the corner of Broadway and 103rd St, quite close to the hostel, so I went there for coffee and breakfast. Back at the hostel by 8:50, there was a group of about a` dozen gathered for the tour: two aussies, a kiwi, two from Holland, two from Germany, three from France and the rest from Brazil. Dave’s tour was quite extensive, covering some of the sights I’d seen the previous day: South Ferry, Wall Street and Ground Zero, a brief stop at the Seaport, before venturing into some of the less touristy neighbourhoods: TriBeCa (Triangle Below Canal) home to Robert De Niro’s 3 restaurants; SoHo, a more artsy part of town, where we visited Peter Lik’s photo gallery; Greenwich Villiage, the neighbourhood surrounding NYU; West Village, the most… bohemian part of town; and finally Chelsea, where we saw the old meat markets, the west-side railway which has been converted into a high-rise park, and Chelsea markets. The tour portrayed New York as an extremely diverse and interesting city, with a completely different vibe and theme in each neighbourhood. I ended the day by heading up North to Columbia University, such a pristine university campus I have never seen! Then it as only half a dozen blocks walk South to get back to the hostel.

Impressed with Dave’s Downtown Tour on Friday, I signed up for his tour of Central Park on Saturday morning. I’d seen small bits of the park when walking around on Thursday, but the park is huge and really needs a several hours to see send to end. We started at the North-West corner near the hostel, across to Harlem Meer and the Lasker Pool/Rink, weaved our way through Coventry Gardens, South-West to The Pool, back and forth back and forth, slowly heading south. We stopped for lunch at the Boathouse, before venturing back out towards Sheep’s Meadow and The Mall. It was a very thorough tour of the park, and Dave had a lot of insights to share about the history and design of the park, and the changes that have occurred along the way.

After the tour, ended at the South tip of the park, I walked down 5th Avenue, visiting the Apple ‘Cube’ Store, and the Rockerfeller Centre, climbing to the ‘Top of the Rock’ for 360 degree views of the city. It had been recommended as a better alternative for high-rise viewing than the Empire State Building (which is slightly taller), because the Rock has views of the Empire State Building itself, as well as Central Park, which can’t be seen from as far South as the Empire State. Saturday night was spent out on the town for a change, hanging out with other Australian tourists, where naturally we all joked about the peculiar popularity Fosters.

With a bus to Boston booked for Monday midday, Sunday felt like saying goodbye. A late breakfast at Columbus Circle was followed by a visit to St Patrick’s Cathedral – an enormous Gothic-revival style building, followed by a trip to Times Square to get a ticket to West Side Story in the afternoon. The show was fantastic, the acting flawless and the music exceptional, partly owing to the great acoustics in the small Broadway Orchestra theatre. Later in the evening I dropped by the famous ‘Century 21′ department store, where there are indeed great bargains.

Monday morning was a bit of a rush to get to the Australian Consulate to vote in the morning, before heading back to the hostel to check out, then back into town for the bus. I was impressed by the quality of the polling place setup, ballot sheets for every electorate in the country were laid out, sorted first by state and then alphabetically, the consular officers performed their duty of checking your enrolled electorate and signing the sealable postal voting envelopes efficiently. There was even a Liberal party member standing outside handing out ‘how-to-vote’ pamphlets, again indexed by state and electorate.

I checked out of the hostel with 45 minutes until the bus’ departure time, which seemed like plenty… and all was good until standing around at the Port Authority Bus Terminal with 10 minutes to spare, I realised my bus was actually departing from Pennsylvania Station, not Port Authority! Dragging my suitcase I rushed down 7th Avenue crossing a dozen streets until I reached Penn Station 5 minutes late. Turns out the rush wasn’t necessary, as the bus left half an hour late. After the initial delayed departure (for which I was thankful), it was a smooth ride, and the bus even had free WiFi!

I’m going to miss New York City, I’m not sure if it’s the skyscraper with no windows housing their phone switchboards, or free concerts at the Lincoln Center, or the magic of suddenly feeling like you’re in the middle of nowhere as you turn down a windy path in northern Central Park… but there’s something amazing about this place, and I hope to come back soon!

find . -type f -newer IMAG0232.jpg |xargs -I files mv files NYC

What’s wrong with it? Absolutely nothing! IMAG0232 was the last photo I took before arriving in NYC, that line is supposed to move everything newer than that into an existing directory called ‘NYC’.

The problem: I accidentally made a directory called NCY, not NYC, so instead of moving things to the *directory* NYC, they got moved to the *file* NYC.. each photo overriding the last one. Now only the most recent photo is left!

Sigh.

So I wrote this Python script.

import sys

def usage():
   print "Usage:   "+sys.argv[0]+"  "
   print "Example: "+sys.argv[0]+" -5 +10"
   sys.exit(1)

def printmap(tz1, tz2):
   print "timezone1 timezone2"
   diff = tz1-tz2
   for i in xrange(24):
      print i, (i-diff)%24

if len(sys.argv) != 3:
   usage()
try:
   tz1 = int(sys.argv[1])
   tz2 = int(sys.argv[2])
except ValueError:
   usage()
printmap(tz1, tz2)

Crude, but it does the trick!

Update: I never got around to writing a proper post about all this, but we ended up coming 2nd. There’s some great videos on YouTube of all our games if you’re interested.

For those of you who aren’t familiar with Robocup SPL, it is a competition consisting of 24 teams from universities around the world, who program the Aldebaran Nao humanoid robot to play soccer autonomously. The robot must identify objects in its environment, plan how it should react to the world in coordination with its team-mates, and execute those actions (such as walking and kicking) quickly and correctly. Our team, rUNSWift is described in this poster:

When we arrive in Singapore on Friday evening, we have one and a half days to calibrate our robots and software for the specific environmental peculiarities of the Suntec Convention Centre, before our first game which is a demonstration match for the opening ceremony. On Monday, we will play in the 1st pool round against two other teams:

Austin Villa

The team from the University of Texas, Austin are the champions from this year’s US Open and semi-finalists from last year’s World Championships in Graz, Austria. They’re notorious for having a diving goalie, which will prove a worthy challenge in this opening round.

BURST

This team is relatively new to the SPL league, but are not to be trifled with, they outperformed rUNSWift significantly in the 2009 competition, despite that being their first year participating, and the only Israeli team to have ever competed in a senior Robocup tournament. According to their team description paper, their research this year is focusing on machine-learning for walking and kicking as well as high end behavioural systems, while they using an augmented version of the code from the B-Human 2009 release for vision and localisation.

After the 1st pool round, we will either proceed directly to the 2nd pool round if we place 1st in our pool, or play in an intermediate round if we rank 2nd or 3rd. The quarter-finals take place on Wednesday and the semi and grand finals are on Thursday. The hot favourite for this year are B-Human, who won the competition last year, and also won this year’s German Open. We’re hoping to give them a run for their money.

This year rUNSWift has gone for a complete re-write of our software architecture, they key features being:

• An omni-directional walk significantly faster than the standard AL walk
• A hybrid Kalman-filter/Particle-filter system for localisation
• A Python-C++ mapping layer to allow for the rapid development of high-level behaviours
• A powerful suite of diagnostic and testing utilities, that allow most hardware and software glitches to be picked up at runtime, and often repaired

Watch this space for videos of set-up and games from Singapore over the coming days. In the mean time here’s a trailer for Robocup Singapore:

http://blog.daave.com/