Instant Badger

How many WTFs per language?

2012-01-23T18:32:00.000Z

Recently I've had to work with PHP an increasing amount. I've not been enjoying it. A few recent IRC conversations also got me thinking - is there an entirely arbitrary but kind of fun (and hey aren't they all arbitrary, really?) metric of programming language FAIL / coder FAILs per-language ?

So on a whim, I decided to invent one: number of entries on The Daily WTF which mention a given language.

Here we go - "web" languages only, numbers correct at time of writing:

.NET - 13,000
Java - 11,300
PHP - 8,100
Javascript - 6,020
Perl - 3,520
Python - 2,270
Ruby - 1,570
Coldfusion - 459

Or in chart form:

So what does this prove? Well, nothing - to draw any conclusions at all, we'd have to normalise this for all kinds of factors such as

number of coders / lines of code using each language (anyone have a decent set of figures?)
average experience level of coder (I'm pretty sure that more beginners use PHP than Perl, for instance)
you name it

...but it is kind of fun. Let the flamewar ~~commence~~ continue!

FeedWordpress duplicate posts with YD Domain Mapping

2012-01-10T12:03:00.001Z

Using FeedWordPress to pull in aggregated feeds to a Wordpress blog? Also using the Wordpress MU Domain Mapping plugin to map an arbitrary domain to your blog? Seeing posts get duplicated? Then read on ....

I've isolated the duplicate posts issue to some kind of interaction with the Wordpress MU Domain Mapping plugin. We're using this so that we can map arbitrary domains to our network sites.

E.g.

our Wordpress Network is set up as wp-network.foo.com, on a subdomain setup
a network site 'bar' would then be bar.wp-network.foo.com
we setup the feedwordpress plugin to pull in posts from a tag:uri feed
we get unique posts in, with GUIDs of the form bar.wp-network.foo.com?guid=(some guid)

All well and good. BUT, then we use the domain mapping plugin to map blog.bar.com to that network site...

It seems that when we hit a network site on its mapped domain (blog.bar.com), it instantly duplicates the syndicated posts with guids of the form blog.bar.com?guid=(the same guid)

I set up 2 identical blogs, subscribed them both to the same feed, and applied a domain mapping to one but not the other. All was fine until I hit the blog homepage on the new mapped domain- at which point all the posts got replicated with mapped domain GUIDs as above.

So I don't know if this is an issue with FeedWordPress or with the Wordpress MU Domain Mapping plugin, but they definitely don't seem to play nicely with each other.

Hope this helps someone!

Porting a Rails 2.3 app to Ruby 1.9

2011-11-22T14:49:00.001Z

We finally managed to get enough space in the schedule to take the plunge and port our monolithic Rails 2.3 app to Ruby 1.9, with a view to increasing scalability of our app. An upgrade to Rails 3 is also on the cards for later, but... one thing at a time.

As ever, the path to true Ruby nirvana is paved with good intentions, and tends to detour into dependency hell for a good portion of the way. Here's a quick shortlist of some of the issues we found along the way, and what we did to get round them.

In no particular order, here we go.....

MySQL2 version should be no later than 0.2.x

If you get the dreaded Please install the mysql2 adapter: `gem install activerecord-mysql2-adapter` error message, what it really means is You can't use MySQL2 version 0.3+ with Rails version less than 3
If you're already using MySQL2 0.2.x, and you're on Mac OS X, and you're still getting the error, then the other thing that it really means is I couldn't find the dynamic library libmysqlclient.18.dylib. There are two proposed fixes for this:

sudo install_name_tool -change libmysqlclient.18.dylib /usr/local/mysql/lib/libmysqlclient.18.dylib /Users/YOUR_USER_NAME/.rvm/gems/1.8/gems/mysql2-0.2 - I couldn't get this working
sudo ln -s /usr/local/mysql/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib - This worked for me on Lion

Use Bundler

If you weren't using it before (we weren't), use it now. Stop fighting the inevitable, just give in, bend over and take it - and use Bundler. Seriously, it makes things easier in the long run.

EventMachine does not compile (update: ..easily..) on 1.9.2

...at least not on my Lion Macbook Pro. It kept giving me the compiler error: cc1plus: error: unrecognized command line option ‘-Wshorten-64-to-32’. Under 1.9.3, however - no problems, compiled first time every time.

Update: actually, I did eventually get EventMachine 0.12.10 to compile under 1.9.2 with an evil hack. The error "-Wshorten-64-to-32" above made me think - the compiler is not part of RVM, so the only way the compiler would recognise a command-line option in one Ruby but not in another... is if it's not the same compiler! So I tried this:

$ rvm use 1.9.3@1.9.3rails2.3
Using /Users/aldavidson/.rvm/gems/ruby-1.9.3-p0 with gemset 1.9.3rails2.3
$ irb
ruby-1.9.3-p0 :002 > require 'mkmf'
 => true 
ruby-1.9.3-p0 :004 >  CONFIG['CXX']
 => "g++-4.2" 
ruby-1.9.3-p0 :005 > exit

Now let's see what 1.9.2 is using:

$ rvm use 1.9.2@1.9.2rails2.3
Using /Users/aldavidson/.rvm/gems/ruby-1.9.2-p290 with gemset 1.9.2rails2.3
$ irb 
ruby-1.9.2-p290 :001 > require 'mkmf'
 => true 
ruby-1.9.2-p290 :002 > CONFIG['CXX']
 => "g++" 
ruby-1.9.2-p290 :006 > exit

Ah-hah! So they are in fact using different C++ compilers! And although many native gems (e.g. MySQL) will respect and pass-through the CXX environment variable to the Makefile, sadly EventMachine is not one of them. So, the easiest fix was a bit of evil hackery - move g++ out of the way and symlink it to g++-4.2:

$ which g++
/usr/local/bin/g++

$ which g++-4.2
/usr/bin/g++-4.2

$ sudo mv /usr/local/bin/g++ /usr/local/bin/g++.bak && sudo ln -s /usr/bin/g++-4.2 /usr/local/bin/g++

looking good - let's go for it:

gem install eventmachineBuilding native extensions.  This could take a while...
Successfully installed eventmachine-0.12.10
1 gem installed

Yay!

Rails 2.3 is NOT SUPPORTED under Ruby 1.9.3!

There was a big fuss about Rails loading times under 1.9.3. The way that 'require' statements work has been changed, and it no longer checks to see if the file is there before requiring it. As a result, every controller MUST have a corresponding helper file, even if it's just a stub. This is majorly annoying, as we have 73 controllers across 3 namespaces, and only 17 helpers. The Rails team have "frozen" the 2.3 branch, and say that only security fixes will go in after 2.3.14 - in other words, they're not going to fix this. The good-old monkeypatch-via-plugin method doesn't work either, as the files get required AS Rails is loaded, not once it's initialised.

So there are two options, either:

Create a stub helper for each controller
...which seems a bit fugly to me, or..
Fork Rails, fix the issue, use that fork in the meantime until we can port to Rails 3.

This seems better, and who knows, if enough people complain about it, maybe we'll get a patch release (2.3.15?). I'm not holding my breath, mind.... but if we're in this situation, I'm sure others are too, so this will hopefully help some other people who are having the same problem.
So, here's the forked Rails 2.3.14 that we will fix the helper requires problem on (note: WILL fix, not "have already fixed"! :)

UPDATE: After having got round the EventMachine issue above, we no longer need to do this - so we're carrying on with 1.9.2 and standard rails 2.3.14.

MySQL idle connections still holding locks

2011-09-29T10:59:00.000+01:00

We had an interesting problem today. We were seeing very slow single-row updates (>30s) on our (innodb) scheduled_jobs table, and a large number of the update queries were failing with a LOCK WAIT TIMEOUT. The updates were using the primary key, so they should be pretty fast - but they weren't.

So we fired up a console and ran SHOW INNODB STATUS, and saw lots of transactions with long-held locks -

---TRANSACTION 0 200086649, ACTIVE 3000 sec, process no 29791, OS thread id 140353331377936
12 lock struct(s), heap size 3024, 6 row lock(s), undo log entries 10
MySQL thread id 243, query id 314676 ip-10-94-245-79.ec2.internal 10.94.245.79 tn

- however, when we cross-referenced the MySQL thread IDs with a SHOW PROCESSLIST, we found that lots of the threads weren't actually doing anything:

+------+------------+-----------+-----------------+----------------+------+--------------+------------------+
| Id   | User       | Host      | db              | Command        | Time | State        | Info             |
+------+------------+-----------+-----------------+----------------+------+--------------+------------------+
|  243 | tn         | (..snip..)| tn_production   | Sleep          | 3000 |              | NULL

This was strange - but after a bit of a Googling and a few minutes thought, we realised the cause.

When one of our worker processes dies - e.g. either because monit has killed it for taking up too much resource for too long, or because the EC2 instance it's running on has become unresponsive to ssh and been automatically restarted by our detect-and-rescue script - it may be in the middle of a transaction at the point it's killed.

If the transaction was holding locks, and never got to the "COMMIT" stage, and the connection wasn't closed normally, then that connection will persist - and maintain its locks - until it gets timed-out by the MySQL server.... and the default wait_timeout variable is 28800s - 8 hrs!.

So, we killed the idle connections, and the update queries started going through much more quickly again. We've now brought our wait_timeout variable down to 10 mins, and we'll see how we get on.

New EU Cookie / Privacy Law - Should you panic?

2011-05-11T10:30:00.000+01:00

There's been a lot of fuss recently about the new "EU Cookie Law", and what effect it will have on EU- and UK-based online businesses. The EU directive has been around and discussed with varying degrees of hyperbole for a while, what's caused the recent kerfuffle has been the adoption into UK legislation pretty-much as-is.

So, should you be panicking in order to meet the implementation date of 26th May 2011?

Well... maybe, but I'm not.

Allow me to explain.

There is cause for concern - just like we saw with the RIPA act circa 2000, the legislation is clearly well-intentioned, but has just-as-clearly been implemented by people with little understanding of the problem domain. The full legislation is available from the horse's mouth here: The Privacy and Electronic Communications (EC Directive) (Amendment) Regulations 2011, but I would recommend the implementation guidelines from the Information Commissioner's Office for a slightly lighter read.

Credit where it's due, the guidelines give a welcome degree of balance and reasonableness to the situation, but even so they manage to contradict themselves.

The important section is "What do the new rules say?", where it quotes the relevant section of the new legislation:

6 (1) Subject to paragraph (4), a person shall not store or gain access to information stored, in the terminal equipment of a subscriber or user unless the requirements of paragraph (2) are met.
(2) The requirements are that the subscriber or user of that terminal equipment--
(a) is provided with clear and comprehensive information about the purposes of the storage of, or access to, that information; and
(b) has given his or her consent.
(3) Where an electronic communications network is used by the same person to store or access information in the terminal equipment of a subscriber or user on more than one occasion, it is sufficient for the purposes of this regulation that the requirements of paragraph (2) are met in respect of the initial use.

(3A) For the purposes of paragraph (2), consent may be signified by a subscriber who amends or sets controls on the internet browser which the subscriber uses or by using another application or programme to signify consent.

(4) Paragraph (1) shall not apply to the technical storage of, or access to, information--
(a) for the sole purpose of carrying out the transmission of a communication over an electronic communications network; or
(b) where such storage or access is strictly necessary for the
provision

Now, correct me if I'm wrong, but that seems fairly clear - if the user sets the controls on their internet browser to accept cookies, consent may be taken to be signified, right? Riiiiight?

Well, apparently not. A little bit further down the document, it says:

I have heard that browser settings can be used to indicate
consent – can I rely on that?
(...)
At present, most browser settings are not sophisticated enough to
allow you to assume that the user has given their consent to allow
your website to set a cookie

Er...wut? Most browsers are set to accept cookies by default, but can be changed to reject them, or to reject third-party cookies, or prompt for each one. How is that not giving consent? And how does this guidance interact with paragraph 3A from the regulations themselves?

There's another implementation question as well. Let's walk through a workflow.

User X arrives at a site
Site wants to set a cookie so that it can identify that User X's next click comes from User X, and not any of its other users.
..So Site has to ask for consent, presumably by an irritating and obtrusive pop-up window, or some other interstitial means. Remember that cookies are sent back as part of the HTTP response, so the decision regarding whether or not to set a cookie is taken on the server, before a response is sent back to the user.
If the user says "YES", all well and good - Site sets a cookie and remembers that choice.
BUT... what if the user says no? How do you remember that choice, without using a cookie? You can't, right? So you're going to have to ask every request...

OK, that's admittedly a simplified scenario for the purposes of making a point. But the point is valid - it's going to be extremely tricky to think of ways of implementing this directive in any meaningful way without destroying your user experience. And what about the ubiquitous third-party cookies that form the basis of services such as Google Analytics? Again, the guidelines are specific:

An analytic cookie might not appear to be as intrusive as others that might track a user across multiple sites but you still need consent

And THAT, paradoxically, is why I'm not panicking. Laws that are very difficult to comply with are, in practice, difficult to enforce. This law, if it ever does get enforced, is most likely to be used as a political club to attack a high-profile mega-corporate that couldn't be brought down any other way - think MS and the eventually farcical browser-bundling lawsuit. Think Al Capone, and the fact that the only crime he ever got convicted of was tax evasion.

There have also been some surprisingly reasonable quotes from the Information Commissioner's Office and the Government :

[Information Commissioner Christopher] Graham said the ICO was clear the changes should not have a detrimental impact on consumers nor cause “an unnecessary burden on UK businesses.”

and

we do not expect the ICO to take enforcement action in the short term against businesses and organisations as they work out how to address their use of cookies (Ed Vaizey, Culture Minister)

Regardless, there may well end up being a high-profile test case at some point, which will garner huge amounts of publicity and huge amounts of lawyers saying things like "this is a very interesting case" - which, to paraphrase Terry Pratchett, is lawyer-speak for "at least six months at three grand a day, plus expenses, per lawyer, with a minimum team of ten". It will eventually establish a precedent for judicial interpretation of the act, and at that point the necessary course of action will become clear.

Until then, I'm not panicking. And neither should you.

Rails requests not timing out in time?

2011-02-22T20:51:00.000Z

The web server CPU was mostly idle, the 5 thin instances didn't seem to be doing much, but the requests were still taking forever to return anything. So what the flimmin' flip was taking so long?

Nginx was configured to timeout requests after 60s, but it didn't seem to be working. In fact, the key to figuring out what was going on, was that while tailing the logs, I saw an occasional query come through with a ridiculously long execution time - 400 seconds or more... Yowch!

So, if requests were supposed to time out after 60 seconds, how come a query was allowed to stay executing for 400 or more?

It all comes down to threads. We're still on CRuby 1.8.7 (MRI) in production - we'll move to 1.9 and JRuby at some point, but previous experience has made me careful to test thoroughly, test some more, then test again and again and again before making the move, and we're not quite ready to shift yet.

Now, the default (MRI) CRuby interpreter uses green threads, and the eventmachine gem underlying Thin server uses - like most Ruby libraries - timeout.rb for its timeout mechanism. This article gives the full details, but it boils down to the fact that:
it is a well-known limitations of green threads that when a green thread performs a blocking system call to the underlying operating systems, none of the green threads in the virtual machine will run until the system call returns ... From the operating system perspective, there is only a single thread in the Ruby interpreter (the native one)

So essentially, the monitor thread which is responsible for timing out the request has to wait for the blocking IO call to return, before it can resume. This means that when you have a long-running, block IO call (say, for instance, an ActiveRecord query that takes 400 seconds to run) running under MRI, the default timeout mechanism is useless.

The solution? If you can't just switch to JRuby, then here's a simple 5-minute fix that did the trick for us:

in environment.rb :


  config.gem 'SystemTimer', :lib => 'system_timer'

in application_controller.rb :

  around_filter :force_timeout

  def force_timeout( timeout=nil, &block )
    timeout ||= 60 # <= or whatever value is appropriate
    if defined?(SystemTimer) && timeout.to_i > 0
      begin
        SystemTimer.timeout_after(timeout) do
          yield
        end
      rescue Timeout::Error => e
        logger.error( "Timed out request after #{timeout}s!")
        raise "RequestTimeout" 
      end
    else
      yield
    end
  end

"cannot redeclare exchange" error on amqp and RabbitMQ 1.8 - fixed! (at last)

2010-07-02T14:27:00.003+01:00

I've been having some trouble with an inherited MacBook Pro, on Leopard (OS X 10.5), trying to get RabbitMQ and the tmm1-amqp gem up and running. The annoying thing is that it was all working fine a few weeks ago, but I ended up having to completely remove MacPorts and do several sudo rm -rf wipeouts and pretty much rebuild the dev environment from scratch. Since then, I re-installed MacPorts and did:
sudo port install rabbitmq-server

After a bit of fiddling about, I got this error (using exactly the same code that worked a few weeks ago):

vendor/gems/tmm1-amqp-0.6.4/lib/mq.rb:225:in `process_frame': PRECONDITION_FAILED - cannot redeclare exchange 'jobs' in vhost 'trigganews.com' with different type, durable or autodelete value in AMQP::Protocol::Exchange::Declare on 1 (MQ::Error)

....several hours of frustrating googling later.....

It turns out that I'd innocently upgraded my version of RabbitMQ 1.8, whereas previously I'd had v1.7.2. This is significant because RMQ 1.8 uses AMQP 0.9, whereas v1.7.2 uses AMQP 0.8 - and this matters because I'm using the tmm1-amqp gem to connect to it, and tmm1-amqp v0.6.4 only supports AMQP 0.8...

(HINT - check the files under the protocol directory to see what version your gem supports)

So - time to downgrade RabbitMQ to 1.7.2 then. Should be straightforward, no? Well, no.

It also turns out that the version of Erlang in MacPorts is now R14A - and guess what? RabbitMQ 1.7.2 doesn't build in Erlang R14A

So I had to create a local MacPorts repo (using the very helpful instructions HERE) from the previous checkin of the Erlang R13B4 portfile and point ports at that.

Then, with a sudo port uninstall rabbitmq-server; sudo port install rabbitmq-server @1.7.2, followed by a blat of the mnesia database files:
sudo rm -rf /opt/local/var/lib/rabbitmq/mnesia/rabbit

- and it all works again. YAY!

(PS - I had to port uninstall and re-install rabbitmq-server twice for it to work properly. So that's always worth a try too - I swear MacOS gets more and more like Windows every day.....:)

EAFNOSUPPORT socket connection issue on Mac OS X

2010-06-21T15:28:00.002+01:00

A frustrating problem cropped up this afternoon while trying to get a home-grown Ruby RabbitMQ client to talk to the broker on a MacBook.

The client is based on the EventMachine and AMQP gems, and although for once it worked perfectly on Windows -

(.... yes, frame that last sentence for posterity folks....)

- it was giving bizarre EAFNOSUPPORT socket errors on Mac.

It turns out that, for various obscure reasons, resolving "localhost" is not entirely straightforward on the Mac. There are two quick-and-easy ways round it:

If you have IPv6 mappings for localhost in your /etc/hosts file, comment them out
Replace "localhost" in your connection params with the explicit "127.0.0.1" instead

Sorted!

Memcached Cache Invalidation Made Easy

2009-12-08T20:54:00.003Z

There are only two hard problems in computer science - cache invalidation, and naming things
Phil Karlton

It's an oft-quoted truism that brings a knowing smile to most hardened programmers, but it's oft-quoted precisely because it's true - and during a recent enforced rush job to implement a cache, I came across a nifty solution to the first problem by judicious use of the second.

First, the problem - someone posted Cragwag on StumbleUpon, which led to an immediate spike in traffic on top of the slow increase I've been getting since I made it Tweet the latest news. All the optimisation work that I knew I needed to do at some point was more than a few hours work, and I had to get something out quickly - enter memcached.

Memcached is a simple, distributed-memory caching server that basically stores whatever data you give it in memory, associated with a given key. Rails has a built-in client that you can use simply as follows:

my_data = Cache.get(key) do { 
  ... do stuff to generate data
}

If the cache has an entry for the given key, it will return it straight from the cache. If not, the block will be called, and whatever is returned from the block will be cached with that key.

So far so good - but what exactly should you cache, and how should you do it?

The Complicated Way To Do It

A common pattern is to cache ActiveRecord objects, say by wrapping the finder method in a cache call, and generating a key of the class name and primary key. But this only works for single objects, which are usually pretty quick to retrieve anyway, and is no use for the more expensive queries, such as lists of objects plus related objects and metadata, or - often particularly slow - searches.

So you could extend that simple mechanism to cache lists of objects and search results, say by using the method name and the given parameters. But then you have an all-new headache - an object might be cached in many different collections, so how do you know which cache keys to purge? You have two options:

Try and keep track of which cache keys are caching which objects? Eep - that's starting to sound nasty - you're effectively creating a meta-index of cached entries and keys, which would almost certainly be comparable in size to your actual cache... and where's that index going to live and how are you going to make sure that it's faster to search this potentially large and complex index than to just hit the damn database?

Sidestep the invalidation problem by invalidating the entire cache whenever data is updated. This is much simpler, but there doesn't seem to be a "purge all" method - so you'd need to keep track of what keys are generated somewhere, then loop round them and delete them individually. You could do this with, say, an ActiveRecord class and delete the cache keys on a destroy_all - but still, that's icky.

The Easy Way To Do It

After a few minutes Googling, I found this post on the way Shopify have approached it, and suddenly it all became clear. You can solve the problem of Cache Invalidation by being cunning about Naming Things - in particular, your cache keys.

The idea is very simple - Be Specific about exactly what you're caching. Read that post for more details, or read on for how I've done it.

So I ripped out all of my increasingly-over-complicated caching code from the model, and went for a simple approach of caching the generated html in the controllers. At the start of each request, in a before_filter, I have one database hit - load the current CacheVersion - which just retrieves one integer from a table with only one record. Super fast - and if the data is cached, that's the only db hit for the whole request.

The current cache version number is stored as an instance variable of the application controller, and prepended to all cache keys. The rest of the key is generated from the controller name, the action, and a string constructed out of the passed parameters. Any model methods that aren't just simple retrievals but affect data, can just bump up the current cache version, and hey presto - everything then gets refreshed on next hit, and the old version just gets expired on the least-recently-used-goes-first rule.

This has a few very nice architectural benefits:

The caching code is then in the "right" place - in the bit you want to speed up - i.e. the interface
You also eliminate the overhead of rendering any complicated views - you just grab the html (or xml, or json) straight from the cache and spit it back.
It utilises, and fits in with, one of the fundamental ideas of resource-based IA - that the URL (including the query string) should uniquely identify the resource(s) requested
The application controller gives you a nice central place to generate your keys
If you have to display different data to users, no problem - just put the user id as part of the key.
Rails conveniently puts the controller and action names into the params hash, so your cache key generation is very simple
The admin interface can then easily work off up-to-date data
You can also provide an admin "Clear the cache" button that just has to bump up the current cache version number.

Etc etc - I could go on, but I won't. The net result is that pages which used to take several seconds to render now take just a few milliseconds, it's much much simpler and more elegant this way, and if you're not convinced by now, just give it a try. <mrsdoyle>Go on - ah go on now, ah you will now, won't you Father?</mrsdoyle>

app/models/cache_version.rb

class CacheVersion < ActiveRecord::Base
    def self.current
      CacheVersion.find(:last) || CacheVersion.new(:version=>0)
    end

    def self.increment
      cv = current 
      cv.version = cv.version + 1
      cv.save
    end 
end

app/controllers/application_controller.rb

require 'memcache_util'

class ApplicationController < ActionController::Base
  # load the current cache_version from the db
  # this is used to enable easy memcache "expiration"
  # by simply bumping up the current version whenever data changes
  include Cache
  before_filter :get_current_cache_version

private

  def cache_key
    "#{@cache_version.version}_#{params.sort.to_s.gsub(/ /, '_')}"
  end

  def get_current_cache_version
    @cache_version = CacheVersion.current
  end

  def with_cache( &block )
    @content, @content_type = Cache.get(cache_key) do
      block.call
      [@content, @content_type]
    end
    render :text=>@content, :content_type=>(@content_type||"text/html") 
  end
end

in your actual controller:

  def index 
    with_cache {
      # get data
      # NOTE: you must render to string and store it in @content
      respond_to do |format|
        format.html { 
          @content = render_to_string :action => "index", :layout => "application" 
        }
        format.xml {
          @content_type = "text/xml"
          @content = render_to_string :xml => @whatever, :layout=> false 
        }
      end
    }
  end

Specs failing with Daylight Saving Time change?

2009-10-26T15:57:00.003Z

So I've been banging my head for the past hour or so, trying to work out why some of our specs have suddenly started failing without the code having been touched, and it comes down to an inconsistency in how Rails is handling UTC offsets when Time objects are manipulated:


>> Time.now
 => Mon Oct 26 15:55:20 0000 2009
>> Time.now.utc
=> Mon Oct 26 15:55:26 UTC 2009

OK, that's fine. Now let's try a calculation:


>> (Time.now - 1.day).utc
[Sun Oct 25 14:55:41 UTC 2009

What? Where did that one hour offset come from??

It turns out THAT is the source of all the specs which suddenly failed for no apparent reason this morning.

The good news is, it's easy to fix:

>> Time.now.utc - 1.day
=> Sun Oct 25 15:56:08 UTC 2009

MUCH better!

Ooh, we're in IDC's "10 most innovative software companies"!

2009-10-26T13:41:00.003Z

Oooh, shiny! We just got named as one of IDC's 10 most innovative sub-$100m software companies to watch

I guess we scored more highly on the "Web 2.0-like functionality moves into the enterprise" category more than the other two, and by itself it might not mean much, but it's still great to be thought of in those terms. Makes me feel all warm and fuzzly.

Al's Ultimate Lasagne Recipe

2009-10-13T01:08:00.008+01:00

OK, ok, so I've never posted a recipe on here before. BUT, pretty much everyone I've ever made this lasagne for has said wow, you've GOT to give me the recipe for that! - the most recent example being a guy who had spent years living in Italy, no less - and I can't sleep tonight, so here goes.

Last year I added some refinements from Heston Blumenthal's 'Perfect' bolognese, but mercifully this version takes about 3-4hrs, rather than 3 days. You can add whatever embellishments you like, here I'm just going to describe the basic sauce preparation.

You will need:

Pans

A heavy-bottomed frying pan (our Le Creuset pan was perfect for this) that can get really hot

A large sauce pot / casserole dish

A large lasagne dish - deep enough for at least 3 layers of sauce, plus about half an inch of bechamel sauce

A medium sauce pan for the bechamel sauce

Ingredients

NOTE: all quantities are approximate. Don't be the kind of cook who has to measure everything to the 3rd significant figure - taste often and see what you think it needs, it's much more fun!

About 1kg of lean minced beef (preferably organic, or at least free range - we like Waitrose's, and 2 of their 500g packs works nicely)

About 400g of lardons (again, 2 packs of the Waitrose Free-Range lardons work nicely)

1 red onion, 1 white onion

1 stick of celery

1 or 2 carrots, depending on size - we're aiming for roughly equal quantities of diced carrot, red onion and celery, so adjust as needed

3 tins of chopped tomatoes

3 or 4 mushrooms - we like Portabellini
a bulb of garlic, the fresher the better

2 large bay leaves

2 large/4 small star anise

about 1tbsp Thai fish sauce (we like Squid Brand, which you should be able to get from any good Chinese food store)

about 1tbsp Lea & Perrins Worcester Sauce

about a third of a bottle of red wine

Maldon sea salt & freshly ground black pepper

A tablespoon of Marmite

Butter. Probably about 100g, maybe a bit more

Olive oil - it's best to NOT use extra virgin, you're going to be frying with it - but really, it doesn't make that much difference in the end

Lasagne sheets

A decent handful of basil leaves (MUST be fresh - dried is no good here)

A decent handful of fresh oregano (ditto)

The vine from some vine tomatoes (that's where most of the smell comes from, not the tomato itself)

For the bechamel sauce - maybe 50g of plain flour and about 1/3 pint of milk, plus about 50g of good mature cheddar (Cathedral City Extra Mature works well).

Preparation (20 mins)

Dice the carrot, the red onion and the celery. We're aiming for equal quantities of each, in equal-sized pieces.

Chop the white onion - these pieces don't need to be the same size as the previous lot, they can be bigger and rougher.

Slice the mushrooms, so they're maybe half a centimetre thick

Lightly crush and peel all the cloves of garlic. I use a good whack with my fist on top of a large knife on top the clove. It doesn't need to be obliterated, just kind of half-crushed, so the skin comes off easily.

Take half of the semi-crushed garlic and chop it finely.

In a small bowl, crush (with your fingers is fine) a good tablespoon of sea salt and grind about an equal quantity of black pepper (you're going to be handling raw beef next - you don't want the juices from your hands to hang around on the pepper mill, do you?)

Season the beef - I tip one pack of beef onto the other, then separate the strands back into the empty pack. Each time you make a full layer, sprinkle on some salt and pepper

Cooking stage 1 (20 mins)

In the big pot, pour a good layer of olive oil. Heat over a low-to-medium heat.

Add the semi-crushed (not chopped) garlic cloves. Cook until they're just going golden but still soft (burnt garlic is bitter and grim), then remove them and save for later

Increase the heat slightly, and add the diced red onion, carrot, and celery. Stir these occasionally while they soften.

Put the heavy-bottom frying pan over a medium-to-high heat, and add some olive oil

In the frying pan, add the chopped white onion and star anise. Fry until the onions are caramelising - i.e. going golden brown. Then tip the contents of the frying pan into the big pot (before the onions start to burn and go bitter.

In the frying pan, turn the heat up to full and start to sear the beef. Do this layer-by-layer - ALL the beef must be touching the pan, otherwise it'll broil instead of searing, so just do a little at a time. Keep it moving until it's browned outside but pink in the middle, then tip into the big pot. Repeat until all the beef is seared.

...remember to keep stirring the big pot every so often!

In the frying pan, add a big lump of butter - probably 50g or so. This will sizzle and spit for a while. Once it's stopped sizzling, all the water has gone out of it, so then add the chopped garlic and mushrooms and saute these until they're golden brown at the edges but still plump and juicy. Then tip them into the large pot (the butter will help give the sauce a nice sheen)

Make sure the frying pan is hot, then sear the lardons. You're looking for golden crispy edges, but plump and juicy pinkness. These will probably release a fair amount of fat into the pan - this is fine, it's all good for the sauce. When done, tip everything into the big pot.

Now de-glaze the frying pan by tipping the red wine into it and keep stirring it and scraping the tasty bits off the bottom until the wine is reduced by about half. Tip into the large pot. You can now wash the frying pan - everything from now on takes place in the large pot (except the bechamel sauce)

In the large pot, add the tomatoes and stir well.

Add the Thai fish sauce and Worcester sauce, stir, and reduce the heat to low-to-medium

Crack the bay leaves in half, and add to the pot

Chop and add the lightly-browned garlic that you used to flavour the oil with, right back at the start

Reducing the sauce (1hr-2hrs)

Leave the sauce pot simmering with the lid half-on for about an hour, stirring occasionally and adding pepper if needed. (Taste!)

Sometimes I add a tablespoon of Marmite if the sauce needs more depth, sometimes not - depends on the ingredients

It's done when it's done - i.e. when it looks and tastes like really good bolognese-style sauce, and isn't too runny or too dry.

When it *is* done, turn the heat off and leave it to cool with the lid on for about 45mins

Once it's cool, chop the basil and oregano and stir through. At this point, you can add the vine of the tomatoes and leave it to rest and infuse for about half an hour, then take the vine out again.

Layer the lasagne (5 mins)

Put the oven onto 180 degrees C to heat up

Put a layer of sauce in the bottom of the lasagne dish, then add a layer of lasagne

Repeat at least once, preferably twice (depending on the size of your dish and how much sauce you have) until you've used all the sauce - but make sure that your top layer is sauce, not pasta!

Bechamel Sauce (5 mins)

In the saucepan, melt about 50g of butter over a medium heat

Add the flour and stir quickly until you have a good consistency - still "smearable", not a dry lump

Begin adding milk a little at a time and stirring until absorbed. Make sure you don't add too much too quickly or it'll curdle.

When you have a nice medium-thick-but-still-easily-pourable sauce, add the grated cheese and grind some more pepper into it.

Stir until the cheese is all melted, then pour over the top of your lasagne, making sure it's all covered

Grate a bit more cheese on top

Final baking (45 mins)

Bake the lasagne in the oven at 180 degrees C for about 45 mins, or until the top is golden brown and ever so slightly crispy around the edges.

Remove from the oven and leave to rest for about ten minutes before serving

And that's it! Quality of ingredients counts for a lot, but the biggest clincher is the care taken to make sure that each ingredient is individually well prepared and cooked before adding to the pot. Enjoy!

Blogspot ATOM in Feed-normalizer / Simple-RSS

2009-10-03T10:03:00.003+01:00

Both Cragwag and Sybilline are using the excellent Feed-normalizer for parsing RSS and ATOM feeds, but there's been a niggling problem with the ATOM generated by Blogger / Blogspot in particular - the resulting links on each entry end up pointing to the comments, not the post itself.

So I just forked simple-rss at github and fixed this.

Turns out that simple-rss is just taking the first link tag that it comes across and using that as the link for a post, which in the case of Blogspot ATOM is the comments link.

On inspection of the ATOM RFC it says (section 4.2.7.2) :

atom:link elements MAY have a "rel" attribute that indicates the link relation type. If the "rel" attribute is not present, the link element MUST be interpreted as if the link relation type is "alternate".

Looking at the Blogspot ATOM, it looks like every element has a link rel="alternative" that points to the URL you would see if you navigated to the post from the blog homepage, so I've made it choose that link if it exists.

Github should build the gem automatically - but it's taking a long time to do it, so in the meantime, you can download it from http://github.com/aldavidson/simple-rss and build it locally:

gem uninstall simple-rss
cd (source root)
rake gem
cd pkg
gem install -l simple-rss

That should fix the problem

How to create and save an AMI image from a running instance

2009-09-08T19:44:00.004+01:00

One snag I encountered early on in my migration of Cragwag and Sybilline to Amazon's EC2 Cloud, was that I needed to take a snapshot of my running instance and save it as a new Amazon Machine Image (AMI).

I'd created a bare-bones Debian image from a public AMI (32-bit Lenny, 5.0, not much else) and then installed a few standard software packages on it - mysql, ruby, apache, etc etc etc. Once I'd got them configured the way I wanted, it had taken a couple of hours (I'll go into the configuration relating to EBS in a separate post) so I wanted to snapshot this instance as a new AMI image. That way, if and when I needed to create a new instance, all of this work would already have been done.

It actually took a fair amount of time to find out (well, more than a few seconds Googling, which is just eternity these days, y'know?) so I'll save you the pain and just give you the solution.

First, install Amazon's AMI tools, and API tools:


export $EC2_TOOLS_DIR=~/.ec2 #(or choose a directory here)
cd $EC2_TOOLS_DIR
mkdir ec2-ami-tools
cd ec2-ami-tools
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip ec2-ami-tools.zip
ln -s ec2-ami-tools-* current
cd ..
mkdir ec2-api-tools
cd ec2-api-tools
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip
ln -s ec2-api-tools-* current

echo "export EC2_AMITOOL_HOME=`dirname $EC2_TOOLS_DIR`/ec2-ami-tools/current" >> ~/.bashrc
echo "export EC2_APITOOL_HOME=`dirname $EC2_TOOLS_DIR`/ec2-api-tools/current" >> ~/.bashrc
echo "export PATH=${PATH}:`dirname $AMI_TOOLS_DIR`/ec2-ami-tools/current/bin:`dirname $AMI_TOOLS_DIR`/ec2-api-tools/current/bin" >> ~/.bashrc
source ~/.bashrc

Next, you'll need to get your security credentials. You can get a reminder of - or create as needed - these on the AWS "Your Account" > "Security Credentials" page.

I recommend you saving your X.509 certificate and your private key somewhere under /mnt/ - this directory is excluded from the bundled image. Quite important that, as otherwise your credentials would be bundled up in the image - and if you ever shared that image with anyone else, you'd be sharing your credentials too!

You'll also need to note your AWS access details - especially your access key and secret key - plus your Amazon account ID.

Now, we're at the main event.

To take a snapshot of your running instance:

First, choose a name for your AMI snapshot. We'll call it ami-instance-name :)


# make a directory for your image:
mkdir /mnt/ami-instance-name

# create the image (this will take a while!)
ec2-bundle-vol -d /mnt/ami-instance-name -k /path/to/your/pk-(long string).pem -c /path/to/your/cert-(long string).pem -u YOUR_AMAZON_ACCOUNT_ID_WITHOUT_DASHES

Once that's done, you should have a file called image.manifest.xml in your /mnt/ami-instance-name directory, along with all the bundle parts. Sometimes it will say Unable to read instance meta-data for product-codes - but this doesn't seem to cause any problems, and I've successfully ignored it so far :)

Next, upload the AMI image to S3. This command will create an S3 bucket of the given name if it doesn't exist - I've found it convenient to call my buckets the same as the instance name:

ec2-upload-bundle -b ami-instance-name -m /mnt/ami-instance-name/image.manifest.xml -a YOUR_AWS_ACCESS_KEY -s YOUR_AWS_SECRET_KEY

You should then be able to register the instance. I've done that using the rather spiffy AWS Management Console web UI, but you can also do it from the command line using:

ec2-register ami-instance-name/image.manifest.xml

And that's it!

Of course, you could be cunning and create a script that does it all in one. I've got my AWS/EC2 credentials stored in environment variables from my .bashrc:

export EC2_PRIVATE_KEY=/mnt/keys/pk-(long string).pem
export EC2_CERT=/mnt/keys/cert-(long string).pem
export AWS_ACCOUNT_ID=(my account id)
export AWS_ACCESS_KEY=(my AWS access key)
export AWS_SECRET_KEY=(my AWS secret key)

which means I can make, upload and register an instance in one, by running this script:

#!/bin/bash

$AMI_NAME=$1

ec2-bundle-vol -d /mnt/images/$1 -k $EC2_PRIVATE_KEY -c $EC2_CERT -u $AWS_ACCOUNT_ID
ec2-upload-bundle -b $1 -m /mnt/images/$1/image.manifest.xml -a $AWS_ACCESS_KEY  -s $AWS_SECRET_KEY
ec2-register $1/image.manifest.xml

...and giving it a parameter of ami-instance-name. I have that script saved as make_ami.sh, so I can just call, for instance:

make_ami.sh webserver-with-sites-up-and-running

...and go have a cup of coffee while it does it's thing.

Moving a site to The Cloud

2009-09-08T18:43:00.004+01:00

Last week I did a lot of reading and research into cloud hosting. The "Cloud" has been a buzzword for a while now, often bandied about by those who know no better as a simple sprinkle-on solution for all of your scale problems - much in the same way as AJAX was touted around a few years ago as a magic solution to all of your interface problems.

The perception can sometimes seem to be Hey, if we just shift X to The Cloud, we can scale it infinitely!. The reality, of course, is something rather more qualified. Yes, in theory, the cloud has all the capacity you're likely to need, unless you're going to be bigger than, say, Amazon - (are you? Are you reeeeeeeally? C'mon, be honest...) - provided - and it's a big proviso - that you architect it correctly. You can't just take an existing application and dump it into the cloud and expect to never have a transaction deadlock again, for instance. That's an application usage pattern issue that needs to be dealt with in your application, and no amount of hardware, physical or virtual, will solve it.

There are also some constraints that you'll need to work around, that may seem a little confusing at first. But once I got it, the light went on, and I became increasingly of the opinion that the cloud architecture is just sheer bloody genius.

What kind of constraints are they? Well, let's focus on Amazon's EC2, as it's the most well-known....

Your cloud-hosted servers are instances of an image
They're not physical machines - you can think of them as copies of a template Virtual Machine, if you like. Like a VMWare image. OK, that one's fairly straightforward. Got it? Good. Next:

Instances are transient - they do not live for ever
Bit more of the same, this means that you create and destroy instances as you need. The flipside is that there is no guarantee that the instance you created yesterday will still be there today. It should be, but it might not be. EC2 instances do die, and when they do, they can't be brought back - you need to create a new one. This is by design. Honestly!

Anything you write to an instance's disk after creation is non-persistent
Now we're getting down to it. This means that if you create an instance of, say, a bare-bones Linux install, and then install some more software onto it, and set up a website, then the instance dies - everything you've written to that instance's disk is GONE. There are good strategies for dealing with this, which we'll come onto next, but this is also by design. Yes, it is...

You can attach EBS persistent storage volumes to an instance - but only to one instance per volume
This one is maybe the most obscure constraint but is quite significant. Take a common architecture of two load-balanced web servers with a separate database server. It's obvious that the database needs to be stored on a persistent EBS volume - but what if the site involves users uploading files? Where do they live? A common pattern would be to have a shared file storage area mounted onto both web servers - but if an EBS volume can only be attached to one instance, you can't do that.

Think about that for a few seconds - this has some pretty serious implications for the architecture of a cloud-hosted site. BUT - and here's the sheer bloody genius - these are the kind of things you'd have to deal with for scaling out a site on physical servers anyway. Physical hardware and especially disks are not infallible and shouldn't be relied on. Servers can and do go down. Disks conk out. Scaling out horizontally needs up-front thought put into the architecture. The cloud constraints simply force you to accept that, and deal with it by designing your applications with horizontal scaling in mind from the start. And, coincidentally, provide some kick-ass tools to help you do that.

Take, for example, the last bullet point above - that EBS volumes can only be attached to one instance. So how do you have file storage shared between N load-balanced web servers? Well, the logical thing to do is to have a separate instance with a big persistent EBS volume attached to it, and have the web servers access it by some defined API - WebDAV, say, or something more application-specific.

But hang on.... isn't that what you should be doing anyway?. Isn't that a more scalable model? So that when your fileserver load becomes large, you could, say, create more instances to service your file requests, and maybe load-balance those, and....

See? It forces you to do the right thing - or, at least, put in the thought up front as to how you'll handle it. And if you then decide to stubbornly go ahead and do the wrong thing, then that's up to you... :)

So, anyway, I wanted to get my head round it, and thought I'd start by shifting Cragwag and Sybilline onto Amazons EC2 cloud hosting service. I did this over a two day period - most of which, it has to be said, was spent setting up Linux the way I was used to, rather than the cloud config - and I'll be blogging a few small, self-contained articles with handy tips I've learned along the way. Stay tuned....

Styling the Rails auto_complete plugin

2009-08-17T22:58:00.015+01:00

Over the weekend I was having a fiddle around with the layout on Cragwag, as the existing design was a bit, shall we say, emergent. I'd being trying to avoid a LHS column, because everything always seems to end up with one, but in the end I had to give up and just go with it. The auto-tag cloud just makes more sense over there, and there's no point fighting it.

Anyway, the next question was what else to put there? I was looking for an intermediate stop-gap search until I got round to putting a proper Lucene-based search in there, and so the thought struck - I'd been looking for a way to browse tags that weren't necessarily in the top 20 (e.g. show me all the posts about Dave McLeod's 2006 E11 route), so why not try an auto-suggest tag search?

So I found DHH's auto_complete plugin (it used to be part of Rails core until version 2) and got to work. This should be easy, right?

Well...

I'll cut out the frustrations and skip to the solutions. :)

The documentation on this plugin is virtually non-existent, and there some extra bits you'll need - but I found this post very helpful.

One minor irritation I found was that it writes out a bunch of css attributes directly into your HTML inside a style tag - including a hard-coded absolute width of 350px. (see the auto_complete_stylesheet method)

Argh! said I, as my search needed to fit into a 150px width.

So how can you get round this? Simple - you can override the inline CSS in your stylesheet, provided you provide a CSS selector with higher specificity

Now, CSS specificity can be a fairly complicated topic, but I usually just remember it like this - if you've got two rules that apply to a particular thing, the more specific rule wins.

In this case, the inline CSS selector from the plugin:

 div.auto_complete {
  width: 350px;
  background: #fff;
}

gets trumped by Cragwag's more specific selector:

div#content div#lh_sidebar div.auto_complete {
  width: 150px;
  background: #fff;
}

The first applies to any div with class="auto_complete", but the second applies only to divs with class="auto_complete" which are inside a div with id="lh_sidebar" which is inside a div with id="content". So that's a more specific rule, so it wins.

Yay!

OutOfMemoryError in ActiveRecord-JDBC on INSERT SELECT

2009-08-12T10:53:00.004+01:00

During some scale testing the other day, we came across this unusual / mildly amusing error in a database-bound command that just funnels INSERT SELECT statements down the ActiveRecord JDBC driver:


java/util/Arrays.java:2734:in `copyOf': java.lang.OutOfMemoryError: Java heap space (NativeException)
 from java/util/ArrayList.java:167:in `ensureCapacity'
 from java/util/ArrayList.java:351:in `add'
 from com/mysql/jdbc/StatementImpl.java:1863:in `getGeneratedKeysInternal'
 from com/mysql/jdbc/StatementImpl.java:1818:in `getGeneratedKeys'
 from org/apache/commons/dbcp/DelegatingStatement.java:318:in `getGeneratedKeys'
 from jdbc_adapter/JdbcAdapterInternalService.java:668:in `call'
 from jdbc_adapter/JdbcAdapterInternalService.java:241:in `withConnectionAndRetry'
 from jdbc_adapter/JdbcAdapterInternalService.java:662:in `execute_insert'
  ... 25 levels...

Now, there's a couple of things here that are worth pointing out.

I REALLY LOVE the fact that it blew heap space in a method called ensureCapacity. That makes me smile.
Why is it calling getGeneratedKeys() for an INSERT SELECT?

The getGeneratedKeys() method retrieves all the primary keys that are generated when you execute an INSERT statement. Fair enough - BUT the issue here is that we'd specifically structured the process and the SQL statements involved so as to be done with INSERT SELECTS, and hence avoid great chunks of data being transferred backwards and forwards between the app and the database.

It turns out that the ActiveRecord JDBC adapter is doing this :
(lib/active_record/connection_adapters/JdbcAdapterInternalService.java)


@JRubyMethod(name = "execute_insert", required = 1)
    public static IRubyObject execute_insert(final IRubyObject recv, final IRubyObject sql) throws SQLException {
        return withConnectionAndRetry(recv, new SQLBlock() {
            public IRubyObject call(Connection c) throws SQLException {
                Statement stmt = null;
                try {
                    stmt = c.createStatement();
                    smt.executeUpdate(rubyApi.convertToRubyString(sql).getUnicodeValue(), Statement.RETURN_GENERATED_KEYS);
                    return unmarshal_id_result(recv.getRuntime(), stmt.getGeneratedKeys());
                } finally {
                    if (null != stmt) {
                        try {
                            stmt.close();
                        } catch (Exception e) {
                        }
                    }
                }
            }
        });
    }

...in other words, explicitly telling the driver to return all the generated keys.
Hmm, OK, can we get round this by NOT calling the execute_insert method, and instead calling a raw execute method that doesn't return all the keys?

Well, no, unfortunately, because it also turns out that the ruby code is doing this:
(activerecord-jdbc-adapter-0.9/lib/active_record/connection_adapters/jdbc_adapter.rb)


      # we need to do it this way, to allow Rails stupid tests to always work
      # even if we define a new execute method. Instead of mixing in a new
      # execute, an _execute should be mixed in.
      def _execute(sql, name = nil)
        if JdbcConnection::select?(sql)
          @connection.execute_query(sql)
        elsif JdbcConnection::insert?(sql)
          @connection.execute_insert(sql)
        else
          @connection.execute_update(sql)
        end
      end

...and the JdbcConnection::insert? method is detecting if something's an insert by doing this:
(JdbcAdapterInternalService.java again)


@JRubyMethod(name = "insert?", required = 1, meta = true)
    public static IRubyObject insert_p(IRubyObject recv, IRubyObject _sql) {
        ByteList bl = rubyApi.convertToRubyString(_sql).getByteList();

        int p = bl.begin;
        int pend = p + bl.realSize;

        p = whitespace(p, pend, bl);

        if(pend - p >= 6) {
            switch(bl.bytes[p++]) {
            case 'i':
            case 'I':
                switch(bl.bytes[p++]) {
                case 'n':
                case 'N':
                    switch(bl.bytes[p++]) {
                    case 's':
                    case 'S':
                        switch(bl.bytes[p++]) {
                        case 'e':
                        case 'E':
                            switch(bl.bytes[p++]) {
                            case 'r':
                            case 'R':
                                switch(bl.bytes[p++]) {
                                case 't':
                                case 'T':
                                    return recv.getRuntime().getTrue();
                                }
                            }
                        }
                    }
                }
            }
        }
        return recv.getRuntime().getFalse();
    }

...in other words, if the sql contains the word INSERT, then it's an INSERT, and should be executed with an execute_insert call.

So, looks like we're a bit knacked here. There are two possible solutions:

The proper solution - fix the AR JDBC adapter (and, arguably, the MySQL connector/J as well, to stop it blowing heap space), submit a patch, wait for it to be accepted and make it into the next release.

Or, the pragmatic solution - rewrite the SQL-heavy command as a stored procedure and just call it with an EXECUTE and sod the DHH dogma.

We went with option 2 :)

Big kudos to D. R. MacIver for tracking down the source of the fail in double-quick time.

Introducing Cragwag.com!

2009-08-03T10:24:00.006+01:00

You know those conversations you end up having in the pub, where after a couple of beers you end up saying "you know what there should be? There should be a site that does X" (where X can be anything at all)

I've had so many of those over the years, and never quite managed to work up the free time / motivation to actually get on and put the ideas into practice..... (and then what's tended to happen is that a couple of years later, someone else goes and does them and makes a fortune, but that's just sods law)

Well a few months ago, I decided enough was enough, and the next time I had one of those ideas, I should just stop talking about it and actually do it - so in my evening and weekends here and there, I've been noodling away on a couple of ideas, mainly just for my own amusement, and to keep me in the habit of actually following through on things.

So here's one of them - Cragwag - a climbing news aggregator.

Those of you who know me in person know that I'm a keen amateur climber. I'm under no illusions - I'm definitely in the "amateur" category for good reason :) - but it struck me that although there's a "definitive" go-to site for uk-centric climbing news ( UK Climbing.com ) it's still editorially filtered - an editor keeps himself up to date on everything that's going on in the scene, and then publishes what he thinks is significant.

That's all well and good, but typically what's significant is what's going on at the forefront of ability. I felt there was also a place for the (admittedly by-now-a-little-cliched) long tail of climbing-related blogs - unfiltered, un-edited, everybody's tales. Whether of heart-stopping epics in the Himalaya, or scrabbling up a Stanage slab. If you felt it enough to blog about it, then someone wants to hear about it.

Plus it was an experiment in automatic news tagging and cross-relating of posts based on content, so it was kind of techie fun too. Which is important :) I'd like to do something with a crowd-sourced google map and iphone gps too, but that's much more experimental. Need to learn the iPhone SDK first...

So that's Cragwag - all the climbing news from punters and pros alike. Just for fun, for the sake of an experiment, and - to paraphprase the most famous answer to "why?" of all time - because it wasn't there :) Yay!

VC 2.0 - Crowdfunding is go!

2009-07-29T09:42:00.003+01:00

The Series B VC market sucks right now. The boom times of the pre-credit-crunch 2.0 world seem very far away. Although there's still enthusiasm to invest in early-stage companies at Angel or Series A rounds (small investment, large slice of equity, potential for large return) and Series C/D+ (where the company is commercialised and probably generating revenue or damn close to it, and therefore the risk is much lower) there's been a huge withdrawal of funds and appetite for Series B investments.

Which is were we are right now.

So in typical Trampo style, we've decided not to go down that route... and today we launched Trampoline's Crowdfunding process.

The idea is, simply put, that rather than persuading a few people to part with a large sum of cash, to go the Obama way and offer tranches of shares for much smaller amounts to many more people. This also opens up great possibilities for utilising the power of the extended network of many investors, all the way from getting a large network of potential contacts (it's almost like hiring a hundred part-time salesmen) to exploiting the wisdom of crowdfunders on key strategic decisions.

Of course, nothing's ever quite as simple as it sounds, and there are many FSA regulations about this kind of thing designed to protect Mom and Pop investors from the plethora of possible boiler room-type scams that could be represented in this light, such as:

We can't actively solicit investment in any public setting.
So this post is FYI only, ok? Ok...

We can't discuss any details with anyone, unless they're
1. A certified Sophisticated Investor, or
2. A certified High Net Worth Individual, or
3. A friend or family member (yes, apparently that's ok!)

So this is emphatically NOT an exchange-less IPO! 'Cause those are illegal! It's more of a self-build consortium, kind of thing

Anyway, bearing in mind the FSA regulations, that's about as much as I can say, apart from point you at Trampoline's Crowdfunding article in todays Financial Times (also on page 12 of the print edition), and the Crowdfunding Website if you want more information

This should not be possible - transactional fail?

2009-05-22T10:57:00.003+01:00

We're having a bizarre problem with MySQL 5.0.32 on Debian in a highly-concurrent environment, using the ActiveRecord JDBC adapter. We've also seen it on 5.0.51 on Ubuntu.

There's a particular sequence of 3 queries, all executed from within a ActiveRecord::Base.transaction { } block:


// Statement 1:
  INSERT INTO (InnoDB table 1)
         (type, ...loads of other fields...)
  SELECT 'ProtoMessageItem', ...other fields...
  FROM   (MyISAM table 1) LEFT OUTER JOIN (InnoDB table 2)
  WHERE  (anti-duplication clause)
  GROUP BY (synthetic identifier on MyISAM table 1, to be a unique key on InnoDB table 1)
 
// Statement 2:
  INSERT INTO (InnoDB table 2)
  SELECT (fields)
  FROM   (MyISAM table 1) LEFT OUTER JOIN (InnoDB table 3) INNER JOIN (InnoDB table 1)
  WHERE  (anti-duplication clause)

// Statement 3:
  UPDATE  (InnoDB table 1)
  SET     type = 'MessageItem' 
  WHERE   type = 'ProtoMessageItem'

Now, all of this takes place within an explicit transaction, and works well - this approach provides huge performance gains over the more simple and obvious methods.

The problem is that just very very occasionally, it breaks. We don't know how or why, but once in a blue moon, statement 2 throws a DUPLICATE KEY exception, and the transaction DOESN'T get rolled back. Statement 1 still gets COMMITTED. We can see ProtoMessageItems in the DB, and this blocks up the pipeline and stops further MessageItems from being created.

ProtoMessageItems should NEVER be visible - they're only ever created and then updated to full MessageItems in the same 3-statement transaction. Seeing them from outside that transaction is like seeing a naked singularity - it just shouldn't be possible.

This is all kinds of wrong in a very wrong way, as it means one of the following - either:

The DB is getting the correct BEGIN/COMMIT/ROLLBACK statements, but transactional atomicity is not holding for some reason

The JDBC driver is not sending the correct ROLLBACK statements, but thinks it's in autocommit mode

There's an obscure concurrency edge case somewhere with the ActiveRecord JDBC adapter, that means once in a while, for some reason, it starts that transaction in autocommit mode.

All of these possibilities are worrying. We've eliminated the possibility of it being some quirk of the particular data that causes this problem. We've turned on MySQL statement logging and watched it for hours, and as far as we can see, the correct ROLLBACK statements seem to be issued every time.

Has anyone encountered a similar problem? Any known issues with INSERT/SELECT on combinations of (transactional) InnoDB and (non-transactional) MyISAM tables?

Suggestions for GruntFuddlr !

2009-05-19T16:53:00.004+01:00

Following on from the previous post on The imminent death of Twitter, after several comments along the lines of "Oh Al, you have SOOOO got to buy that domain" followed by a brief moment of hastiness with the credit card, I am now the proud owner of GruntFuddlr.com !

So all I need to do now is decide what to do with it.... add your suggestions to the comments! The winner will win - erm - something to be decided!

Twitter Will Soon Die - the second Tipping Point approaches

2009-05-15T17:02:00.007+01:00

Maybe I'm sticking my neck out, but it's inevitable - hear me out here. By now you're surely familiar with the concept of The Tipping Point as espoused by Malcolm Gladwell -
the moment of critical mass, the threshold, the boiling point
the point at which momentum for change becomes unstoppable

This has been adopted as a central tenet of the Web 2.0+ business model, and is surely now almost as ubiquitous on a web startup's business plan as a long tail reference or hockey stick graph. This post is about the Tipping Point concept specifically as applied to social networking services.

It's certainly true in my experience that every successful social software service goes through some standard stages of growth:

Early adoption
The service gets adopted by a set of keenly curious early users. This is often through an initial restricted beta program, or through the conference circuit, or friends and families of those involved in the development. Many never get past this stage - maybe it's just not a particularly good idea, or useful service, or well-executed, or maybe it's just unlucky. Assuming the service does get past this stage, they move on to...

Viral spread
The early adopters have judged and found it worthy. Here's where they get to gain geek cool points by name-dropping it amongst their friends, on the conference circuit, in Friday-night bars, etc... often taking advantage of the opportunity to faintly patronise those who are still using an inferior / more mass-market / older service - I mean, how many times have you heard lines like these?

....Oh Digg totally jumped the shark, I moved off there to Reddit and now I've started using MangWurdlr (or whatever)
...Yeah I use GruntFuddlr for all my (whatever)-ing now, it's nice, they've got a really cool UI, and ....etc...etc...
...Oh, you should have a look at PigNudgr - they're still in beta, but it looks really cool, and they're a great bunch of guys - I used to work with Andy at (name of now-defunct late-nineties web agency, usually some combination of colour and animal) and he's doing some really neat stuff now...
etc etc.

Every time I have one of those conversations I'm reminded of when I was in the sixth form (that's school from 16-18, when you've passed your first exams - sorry I don't know what the US equivalent is) and people would compete for alpha-cool status by getting into ever-more obscure bands on ever-smaller labels.

Remember those? Bands like "Anna", "Spontaneous Cattle Combustion", "Generic me-too grunge band #241" - when they appeared in NME for the first time they were the coolest thing since sliced bread, and if you didn't have a copy of their debut EP you were nobody, maaannn - ? It's the same phenomenon. Web startups / bands - same thing, just slightly older protagonists. Same silly haircuts though.

The first Tipping Point

This is the Holy Grail for earlier-stage web services. This is where you succeed in creating enough word-of-mouth buzz that suddenly, you're unstoppable. You tip over into the mainstream. People's parents have heard of you (well, those that read the more techno-savvy broadsheets anyway) A whole new wave of joiners is poised to come on-board, often purely because of who is already on there. Prime example - Stephen Fry on Twitter.

Inflation

(apologies to Alan Guth!)
This is where the whole thing goes crazy. Faster growth than they ever thought possible. All those times the engineers said "Well, that's a high-class problem to have, and we'll just have to deal with that when we get there.." well - they're there right now. Seems like every week there's another digit on the hit counts. Often characterised by regular outages due to scale fail in completely unexpected ways ("Holy crap, we've run out of inodes??? That's possible?"). They're sticking in new servers by the dozen all the time, burning through all that VC funding and not making any money yet..

Meanwhile the user base seems to take on a life of its own. It's no longer a community, it instead takes on the characteristics of a population - splits, fragments, in-fighting, single-issue activism - often against the decisions of the founders, who in their increasingly urgent need to get revenue to support the spiralling infrastructure costs are taking commercially-led decisions and seen as increasingly out-of-touch with the still-trying-to-be-faithful users.

There's conflicting senses of excitement and disappointment, that it's just not the same as it used to be. And who are all these ****wits joining now anyway? What - my MUM just joined? And she wants to be my friend? Oh crap, better get rid of all of those tales of drunken debauchery and undeniable photographic evidence. Your boss probably joins up, adding further fuel to the conflict of identity. Is this personal? A work tool? Both?

The second Tipping Point

All good things must come to an end, and this is the end of the line for the early adopters. It's not new, edgy or cool anymore, now that it's talked about on Question Time and even the Government are starting to use it in some god-awful, half-assed but completely cringe-making attempt to be Down With Tha Kids. Meanwhile, you keep getting friend requests from people at school who you weren't friends with in the first place and have no intention of pretending to be friends with now.

The whole world and his hamster seems to be trying to spam you about it or on it. The number of z-list celebrities on it just keeps growing - Peter Andre? Really? Christ... - and they don't even post themselves, it's done via their PR person, or if they do they're just pushing their warez (yes, @WilliamShatner, I'm looking at you).

(sigh)

You've had enough. You can barely be bothered to check it any more, and all the third-party apps built on the API are just getting on your nerves, and all the cool kids have moved on to DangDurdlr.com anyway... apparently it's really cool, and apparently Dave from AvocadoAardvaark.com got together with Si from PurplePlatypus to start it up, and they've got a really neat UI, and it doesn't have all of these ****wits on it...

So that's the Second Tipping Point. When all of the people who joined before the first Tipping Point leave, partly due to their constant need for something new, partly because services just naturally evolve, but mostly because of all the people who joined just after the first. Who, in their turn, had mainly joined because of all the people who were already on it...

Facebook got there a few months ago. Twitter is rapidly approaching it, and you know what? It's natural. It's part of the inevitable evolution of ideas and the services built upon them. It's not that Twitter has anything fundamentally wrong with it, it's mainly that the problem with software that puts people in touch with people..... is people. The cliques and cooler-than-thouism that ruled school society never went away, they just changed form and went online.

MySpace took over from first-gen social software like SmartGroups, and while its user numbers may technically still be growing, in new-frontier terms, it's now dying. It was supplanted by Facebook, which has become supplanted by Twitter, which will soon be supplanted by some other new kid on the block. No-one knows what it will be yet, but that's part of the whole fun of this business - natural selection in the full glare of the web, with all the competition, cross-breeding, malformed offspring and occasional apparently-arbitrary successes. I love it.

Another day, another deployment

2009-04-07T15:15:00.005+01:00

Posts have been increasingly thin on the ground recently, due to a couple of things: my discovery of a reasonable lightweight API client for Twitter, but mainly lack of time - which itself is down to two things:

I'm organising my wedding to the ludicrously lovely Lise, and
we've been getting SONAR installed and running at two large corporates recently (who shall remain nameless)

It's a good feeling to have seen a product go right through from initial idea on a post-it note to being installed at a super-huge financial client, and it makes all the work worthwhile when you see people actually using it to do their job. I wish I could talk more about the details, but confidentiality dictates not, sadly.

I've also been meaning to blog about the app I whipped up to manage the football team I organise - being a geek at heart, it's amazing the amount of time and effort I devote to being effectively lazy, and this little Rails app which started out as a quick time-saver ended up being a case study in the power of stupidity as applied to a linear optimisation problem - "given this set of players, each of whom have a set of positions they prefer to play in, what is the best possible line-up ?" I'll get round to blogging about that at some point, honest..

Another little personal project that I started as a tool for myself and then ended up generalising and meaning to add to whenever I get time (chortle) is Cragwag - an RSS aggregator, plus some - aggregating many various climbing-related blogs, an only-just-started map of climbing areas plus B&B's plus parking plus convenient deli's, which I may end up mashing up with weather reports to answer questions like "where should we go climbing this weekend?"

Bleh. So that's why I've ended up micro-blogging rather than macro-blogging these days. I will try to get used to writing in more than 140 characters again soon...

And a jolly good time was had by all

2009-03-24T10:36:00.001Z

Karters
Originally uploaded by Dr Snooks

Stag do - survived!
Physically unscathed.
Some of the mental scars may take a little longer to expunge....

What was that about imitation / flattery?

2009-01-29T17:20:00.003Z

So, way back in April 2008, Techcrunch ran this story about us: The Enterprise Social Network Auto-Generated and Visually Mapped

Imagine our surprise here when we discovered this post today on TechcrunchIT:
When IBM Beats Facebook and Twitter

So IBM are now also doing a product called SONAR, that does almost exactly the same thing as our existing product called SONAR (Social Networks And Relevance)??? Spooky!
The article says that it's called SAND (Social Networks And Discovery), but the screenshots clearly show SONAR everywhere, and that's how it's referred to on the IBM Research Labs page

I felt compelled to respond to the claim that "No one else in consumer or enterprise is doing this yet.", so I posted a comment about an hour ago. Hasn't appeared yet though.....

...but can Evel jump it?

2009-01-07T12:13:00.003Z

Happy New Year to all!

It's time for my first bold tech prediction^* of 2009 - this year's "....but will it blend?" will be "...but can Evel jump it?"

Get your suggestions in - the more surreal, the better!

* Disclaimer: OK, OK, so for "bold tech prediction", read "blatant attempt to spread a meme and drive some traffic to my mate's site" :)

Goalkeeper wanted!

2008-10-02T09:56:00.003+01:00

It's off-topic, I know, but then I don't get much time to post anything at all right now, so here goes anyway.

Those who know me, know I'm a keen footballer. I play for a team in a midweek 11-a-side league, and we're currently DESPERATE for a goalie.

The matches are all on astroturf, so there's no studs and very few sliding tackles, and they vary in day, location and kick-off time as follows:

Willesden CCA 8pm kickoff, Mondays or Tuesdays

Paddington Recreation Ground 7:30pm kickoff, Mondays

Very occasionally, we may have a match on a Wednesday or Thursday night at Market Road or Linford Christie Stadium but these are rare.

The team vary in ages between 20-ish to 40-ish, and there's a similar variance in abilities. No-one is anywhere near Ronaldo standard, put it that way.

If you're interested, email me directly (address is on the right, obscured for spambots) or add a comment at the bottom. We've got our next match on Monday, so don't hang about!

Holistic User Experience

2008-08-06T22:46:00.003+01:00

SONAR v1.0 is just around the corner, and we've been thinking a lot about the user experience we offer. One of the key differences this time round is the recognition that User Experience is not just about the interface. Users in this sense are not just end users - they're *anyone* who interacts in any way with your software or your company. They include the sysadmins who have to install your stuff and keep it up and running, the managers who have to make a purchasing decision about whose software to spend their hard-won budget on.

So we believe that User Experience is about looking at the whole experience the user has in interacting with your software and you. It's about making the whole thing as easy as possible for them, right from the way they hear about it, through purchasing and finding the information they need to make an informed decision, through installation, configuration, administration, ongoing support, right the way through to migrating away from it if they want to.

Yes, I did say that you have to make it easy for them to migrate away from your
software. No, that's not a dumb thing to do.

There's two ways of looking at it - self-centred or user-centred. The first way often involves making it difficult to get their data out of your system once it's in there, end-of-lifing old versions and occasionally making new versions incompatible so that they have to keep buying your upgrades or they lose access to their data.

This is bad, m'kay? It's called Vendor Lock-In, and although it can do good things for your bottom line in the short term, ultimately it can create frustration and resentment among your users - Bad User Experience.

The thing to remember is that these days, people move around frequently. They change jobs, they change cities and countries at the drop of a hat, so if you create a Bad User Experience your bad reputation will spread along with the people who had it. The democratised web has created a situation where reputation has become more of a crucial commodity than ever before.

If you had to make a purchasing decision between system x and system y, and both of them had pretty much the same features for a pretty similar price, what factors would you use to make that decision? The first thing I'd do is ask around for anyone who's used either system before, and see what they thought of it. If I heard stories like "Yeah, it's OK, but once you buy it you're stuck with it" I'd hear alarm bells. I'm far more likely to go with a system that lets me get my data out easily if i change my mind for any reason.

You'd also look at questions like - whats the support like? Is there an active user community? Do they have forums / wikis / comprehensive FAQs? Do the reported bugs look they get fixed? Is it easy to find the information you need on the website?

You might give them a call to ask some questions, which opens up a whole new set of potential decision influencers - how quickly did they answer the phone? Did they sound happy to help? Did you have to navigate a labyrinthine set of automated menus for ten minutes before you actually got to speak to someone? Or even worse - did the menus have a voice recognition system that couldn't understand your accent?

All of the above factors - and many more - contribute to the user's experience of your company and your software, and all are potential factors that influence whether someone decides to spend their money on you. We know we've a long way to go in a lot of these areas, but we recognise this, and we're looking - and that's a great start.

Atheistic Error Message Of The Day

2008-06-26T19:22:00.002+01:00

while trying to install GOD, the Ruby process mgmt gem, I got a sudden attack of poignance:

[sonar@tryfan sonar-solr]$ gem install god
ERROR:  While executing gem ... (Gem::RemoteSourceException)
    HTTP Response 404

Discuss, in not less than 3000 words...

Rake gotcha on Windows

2008-04-03T14:20:00.002+01:00

We use Rake as our build system of choice for all of our projects, as it's much more flexible and pleasant to work with than make or ANT, and by and large it's pretty cross-platform. Except.....

I just spent about two hours tearing my hair out over something which should have worked transparently on Windows aswell as *NIX, but it just wasn't playing.

For reasons which would be tedious to go into, inside one particular rake task I needed to cd to a different directory and execute rake in there, to pick up a completely different set of tasks and ActiveRecord model classes, etc, and then resume execution of the containing task.

The following code worked fine on UNIX, but just completely failed to do anything on Windows:

Dir.chdir("../sonar-web") { system( "rake", "db:name_of_other_task" ) }

No error messages, nothing, it just wasn't doing anything.

I'll spare you the headbanging frustration and exhaustive list of things I tried that didn't work, and jump straight to the solution -

rake on Windows needs to execute c:/ruby/bin/rake.bat, not c:/ruby/bin/rake !

rake.bat will in turn call ruby.exe c:/ruby/bin/rake and pass on the command line parameters.

So at the top of my rakefile, I just added:

# on Windows, you can't invoke rake via a "system" cmd, as rake actually should invoke rake.bat
RAKE_CMD      = RUBY_PLATFORM.match(/win/) ? "rake.bat" : "rake"

and then changed my system call to :

system( "#{RAKE_CMD}", "name_of_other_task" )

....and everything worked fine.

I still miss CFOUTPUT

2008-02-26T09:50:00.015Z

It's been about a year now since I last coded CF in anger, and what coding I've done since then has been mainly in Java and Ruby On Rails. These days, most of my coding is done in my spare time, which is a resource in increasingly short supply - so coding actually tends to be done during my random bouts of insomnia. Like many similar turncoats, I've found RoR development to be settling into a fairly steady cycle :

Do some really complicated stuff really quickly.

Nod appreciatively and make coo-ing noises.

Stop yourself just before standing up and announcing that your name is Boris and you are invincible

Try to do something ostensibly simple, get stuck

Spend ages scouring the web for the simple solution that surely must be out there somewhere

Get pissed off because you found a blog post telling you that you shouldn't want to do that because it's "not the rails way"

Figure out a really ugly cumbersome way of doing it step-by-step

Fail to get to sleep because it's just bugging you that something so simple should be so hard in a framework that makes so many other more complicated things so simple

Two days later, discover entirely by accident that the fifty lines of hackery could have been done in one long line of twenty method calls ( things.each do{ |foo| foo.do.some.other.thing }.and.then.do.some.thing.else( bar ) ) if only you'd known that the method you needed was called (insert counter-intuitive method name here) and that there was a plugin called (insert name of obscure plugin here)

Go back and redo your hack with the one long line of twenty method calls. Feel like a l33t h4x0r d00d because you just replaced fifty lines with one.

Discover that somewhere in your one long line of twenty method calls, one of them is returning nil.

Think "hey, it's ok, I can debug this easily with the console!". Feel smug.

Spend what seems like an aeon starting the console, reproducing the conditions in which you get your nil, changing something, then restarting the console so you can test your change.

Resolve to write better unit tests in future.

Pine a little for the good old days of "make your change then hit f5" to see if something works.

Start a trawl through the source code looking for the cause of the problem

Reflect that while duck typing is indeed an orgy of sheer loveliness, in this particular case it would be nice if just this once you could know for sure that this particular object is a Foo and therefore all you need to know would be found in foo.rb

Discover that the cause of your woe is that at least one of your magical plugins doesn't work on Oracle, or SQL Server, or indeed anything other than MySQL.

Go back to the blog on which you found the plugin to see if it's a known problem with a new version

Discover that the blog is down.

Write a plugin to patch the plugin to work on Oracle

Feel vaguely uneasy that you now have a chain of umpteen plugins patching plugins patching plugins patching plugins patching ActiveRecord to do something that you apparently shouldn't want to do, but dammit, you needed to get it done by 5pm.

Go out and pull scary faces at small children to make them cry for a while until you feel better.

Come back feeling much better now that you've spread a little frustration around. Reflect that you're probably just not thinking about things in the "right" way.

Adjust your thought-angle, and come at it again

Repeat from step 1

To be clear, I do think that Ruby has some wonderful features. Blocks, open classes, method_missing - all of these little niceties make some fantastically cool things possible. The Enumerable#sort_by method in particular was one discovery that just gave me a wonderful warm fuzzy feeling - e.g.

some_collection.sort_by { |element| [ element.method1, element.method2, element.method3 ] }

Rails, also, has some really great features, and makes some of the donkey work so easy it's almost laughable. But sometimes it feels like all the thinking went into the elegance of the back-end design, and not enough thought went into the templating. RHTML feels like a tacked-on afterthought. HAML is better in some respects, but it still feels awkward. I haven't yet found any templating language that even comes close to the sheer simplicity and ease-of-use of CFOUTPUT.

The example that triggered this rant was grouped output. Let's keep this simple for the purpose of example - say I have a recordset with three fields:

Type	Sub-type	Title
Type 1	Sub-type 1	Foo 1
Type 1	Sub-type 1	Foo 2
Type 1	Sub-type 1	Foo 3
Type 1	Sub-type 2	Bar 1
Type 1	Sub-type 2	Bar 2
Type 1	Sub-type 2	Bar 3
Type 2	Sub-type 3	Foobar 1
Type 2	Sub-type 3	Foobar 2
Type 2	Sub-type 4	Foobar 3

- etc etc.

If you wanted to output these with headers and sub-headers whenever the type or sub-type changed, it would be almost trivially easy in CF:

<cfoutput query="myRecordset" group="type">
  <h2>#myRecordset.type#</h2>
  <cfoutput group="subtype">
    <h3>#myRecordset.subtype#</h3>
      <ul>
      <cfoutput>
         <li>#myRecordset.title#</li>
      </cfoutput>
      </ul>
  </cfoutput>
</cfoutput>

- but in Rails? I'm still at step 5. I know I'm not the first person to need to do this, not by a long shot, but I just don't yet know what the method that surely must exist would be called. I know you can use Enumerable#group_by to group an array of
objects into something approximating the raw recordset above, but that's kind of working backwards to me.

I also know that someone will probably post the answer in a comment, probably with some kind of dismissive one-word instruction like "Read." or "Learn." linking to the relevant part of the docs. And that's all well and good. I'm just in a temporary bout of misty-eyed nostalgia for the things that CF made easy - particularly CFOUTPUT.

Don't Mention The War!

2008-01-25T13:25:00.000Z

Well, in an office containing two Poles, two Germans, two Americans and many Brits, it had to happen sooner or later....

In response to an email about the petition for a Remembrance Day Bank Holiday, one of the Poles in the office said :
Personally I think it's a totally lame idea. People who want to go to Iraq and other remote places to shoot others should get a shrink not a holiday

At which one of the Brits said:
Hang on, what about the people who went to remote places like Poland and shot Germans?

And then the Poles started on about the lack of help they got in the early days of WWII, and just as that was kicking off, in walked one of the German guys.... and then the fun really started ...!

Ah, you can't beat a bit of good-natured dredging up national stereotypes from half a century ago to really get that Friday afternoon feeling into a place - it's still going, this is great, it's better than a movie! Where's the popcorn?

Ironically Apt Screenshot Of The Day

2008-01-23T13:23:00.001Z

Ironically Apt Screenshot Of The Day
Originally uploaded by Dr Snooks

Tomasz was trying to install the Java Enterprise Edition SDK. For some strange reason it was only giving him this one single button....

A Question (or two) of Ethics (part 1)

2008-01-22T12:59:00.000Z

A couple of thorny ethical questions have raised their head recently, and like most of these things, figuring out what's the "right" thing to do has not been easy.

The first situation came out of an interview. It's a classic problem - one that I've heard asked as a hypothetical "what would you do?" in interviews for a management post - but it does happen, and it happened to us.

We've been searching high and low for a damn good sysadmin, without much success, for months. We finally found a candidate with all the skills and a great attitude, he would have been absolutely perfect for us... there was just one small niggle - in his younger days, he'd hacked into a major credit card provider. Not just for a look-see, either, he actually figured out a way to use people's credit card numbers without their accounts being charged, and used it.

He didn't really know how much he took in total, but in the end he turned himself in, and did hard time for it. Now a completely reformed character, he was looking for a job that used his obviously abundant skills in a positive way.

To his credit, he was very open about this, and he talked quite freely about it. We really liked him, and we were satisfied that he wouldn't be a security problem. However, then there's the matter of our customers.....

We do work for some pretty security-conscious clients. Some of them have US Top Secret classified projects. Although we were quite happy that this guy wouldn't be a security risk, would they be? We're a small startup, battling for contracts along with the rest, and it's hard enough at times to get big corporations to take you at all seriously even at the moment. Anyone who has ever jumped through the zillion flaming hoops that they put you through in order to get a whiff of business from these kinds of companies knows just how insane their security procedures can be. (I'm still smiling at the badge I had to wear on one site visit recently - big bold capital letters, red font - "FOREIGN PERSON! ESCORT REQUIRED!") If one of them got wind of us having a genuine time-served cracker working for us, we'd be dead in the water.

In the end we had to make the uncomfortable decision that we couldn't hire this guy, despite him being one of the best candidates we've had. We're still looking for a suitable sysadmin.

That was a few months ago. The second incident is much more recent - yesterday, in fact - and quite similar, it's kind of approaching the same question of reform and change, but from an opposite direction. Mind you, this post has gotten quite long, so I'll put that in a post of it's own.

Error Message Of The Day - Sharepoint

2008-01-09T11:46:00.001Z

Error Message Of The Day - Sharepoint
Originally uploaded by Dr Snooks

I love sharepoint.
It's great.

It's sooo easy to configure and debug.
And so nice and rewarding to use.

SQL QOTW - Solution!

2007-12-21T16:30:00.001Z

To anyone who has been bashing their head against the desk with the SQL brain-buggerer I posted earlier, here is the solution I came up with. There may well be others, that may be more elegant, but here's the one that occurred to me...

OK, so you want to get all the documents up to a given cumulative total byte size.
A straightforward get-me-all-the-docs query looks like this:

SELECT id, filename, bytesize
FROM documents

We want to calculate the cumulative total size for all the records we've got so far, and then stop once we're about to pass the total. So the first problem is how do we work out the cumulative total?

Well, we're essentially dealing with an ordered list of records. We effectively want to work down the ordered list, calculating the cumulative total as we go.

So first lets impose an order on the list:


SELECT id, filename, bytesize
FROM documents
ORDER BY id

Now for each record, we want to work out the cumulative total. I had many thoughts on how to keep a running total, but I eventually realised that you don't actually need to. Following the old physics student's guiding principle of "if you find yourself with a problem you can't solve, rewrite it into a problem that you can solve"^**, here's the trick - we don't actually need to calculate this as "x + (next x)" - as long as we use the same imposed order, we can get this with a rather cunning subquery:


SELECT id, filename, bytesize,
       ( SELECT SUM(bytesize) 
         FROM documents d2 
         WHERE d2.id <= documents.id 
       ) AS cumulative_total
FROM documents
ORDER BY id

You see how it works? If we're working down an ordered list of ids-

- then keeping a running total is exactly equivalent to querying for the sum of all rows up-to-and-including the current row. So we can now move the subquery from the SELECT clause into the WHERE clause, and it gives us a sufficient criteria for accepting or rejecting rows:


SELECT id, filename, bytesize
FROM   documents
WHERE  ( SELECT SUM(bytesize) 
         FROM documents d2 
         WHERE d2.id <= documents.id 
       ) < @max_cumulative_size * 1024 * 1024
ORDER BY id

( where @max_cumulative_size is the maximum total size in megabytes, obviously )

It all hinges on the ordering. The one proviso with this query is that, although you are free to order the results by whatever you choose, the ordering must be the same as the implicit ordering in the subquery.

And - shock, horror, step back in amazement - that sql works as-is across MySQL and Oracle!

Yay!

So I don't know about you, but after that I feel the need to go hit the climbing wall and have a recuperative pint or two afterwards.

Have a great Christmas everyone!

^**This is a really useful skill to have, and it's not just applicable to Physics exams either. In fact, taken to extremes, it's perfectly possible, with a bit of cunning, to change a question about something you haven't revised, into a question about something you *have* revised....

I first realised the power of this at school, in English -
Q2.4 : Is Hamlet really mad? What is the purpose of his madness in the play and by the play? Discuss in not less than 3,000 words.

Answer : In any discussion concerning Hamlet, it is important to note that had he been Scottish, his situation would have been very similar to that of Macbeth....

:-)

SQL QOTW - SELECT all rows up to a cumulative total

2007-12-21T10:11:00.000Z

Here's a little SQL test that arose here the other day. Let's say you have a DB table called documents, in which you have the following fields:

id
filename
bytesize
date_created
full_content

full_content is a BLOB, containing all the text extracted from whatever document the record represents.
bytesize is, cunningly enough, the size of the content in bytes.

You want to get all the documents, and do something to them - what you want to do is not important here, just that it involves the full_content. However, a moment of pondering will show the nasty bit of this problem... Let's say you may have tens of millions of documents, and the average bytesize is around a meg. Clearly attempting to read the full table into memory is not a good idea (at least until we get Terabytes of RAM in our servers).

A far better idea is to limit the maximum memory usage to some value, and only process up to X MB of documents in one batch. The next time round, you'll get the next X MB of docs, and so on.

We turned this problem over and over for a while, and we thought of a few ways of doing it, but all involved multiple queries, and/or things like temporary tables... and, dammitall, it SHOULD be do-able in one query, dammit man! (cue much indignant harumphing and twizzling of large mutton chop side whiskers, in a Victorian man-of-letters kind of way) Oh yeah, and here's the killer - it has to be portable across Oracle 10g and MySQL.

So the problem is this - given this table, can you construct a single query that will read all the documents up to a given maximum number of megabytes, that will run on Oracle 10g AND MySQL?

Well, after pondering this in several isolated attempts for a few days, I went back to it yesterday, and the answer just popped out straight away - and it's actually deceptively simple. I was going to just give you the answer, but I thought it might be a bit more fun to leave it as a challenge, so a big fat Brucie Bonus goes to the first person who gets it - I'll put the answer up here later on today, if no-one gets it in the meantime.

Beware Case-Insensitive Comparisons on Oracle!

2007-11-19T10:41:00.000Z

Let's say you have a large database with lots of People in it. Each Person can have many EmailAddresses. Let's say that your code needs to be portable across multiple DB systems (specifically, MySQL and Oracle).

Let's also say that you process up to a million emails per day through your system, and for each email you check the sender and recipient addresses against your big DB. So that's potentially an awful lot of reads, and a potential performance bottleneck.

"No Problem!" you say, "I'll just stick an index on the EmailAddress.address field, and I'm sorted"

...but that's not quite it. EmailAddresses are case-insensitive, so that FOO@BAR.COM is the same as foo@bar.com. MySQL does a case-insensitive comparison by default, but Oracle doesn't.

So how do you get Oracle to do a case-insensitive comparison? Well, to cut a long story short, there are two parameters you can set (so long as you're on at least 10gR2):

alter session set NLS_COMP=LINGUISTIC;
alter session set NLS_SORT=BINARY_CI;

Setting NLS_COMP to LINGUISTIC tells Oracle to perform strict case insensitivity operations, and NLS_SORT governs the sort order. The _CI suffix tells it to be case-insensitive with respect to sorting.

Lovely jubbly, smashin' sorted and great - well, actually, no. Not sorted and great at all, because there's a lovely implementation gem there - "Setting NLS_SORT to anything other than BINARY causes a sort to use a full table scan, regardless of the path chosen by the optimizer.".

So Oracle's implementation of a case-insensitive comparison is, basically, "i'm going to go through every single row in the table one-by-one, checking to see if it matches".

As Mel Smith used to say - "Aw, it's marvellous, innit?"

So what can you do? Obviously a full table scan is BAAAAD, so we need to find a way to make it use an index. There are two things you can do^**:

1) Add a explicit lower-case-address field, that duplicates the address field and explicitly lower-cases the address before writing. You can then add an index on this field, and check against that. This means you duplicate info (which is bad) and use more storage than you need (which is also bad) but it's totally portable (which is good)

2) Add a functional index to the address field:
CREATE INDEX ix_my_index_name ON email_addresses( LOWER(address) )
This means you gain the vastly superior performance of an index range scan as opposed to a full table scan, but guess what - MySQL doesn't have functional indexes yet.

GRAAAAAAAAAAAAAAAAAAAARGH!

**OK, so there is a third and possibly fourth option in this particular case, to just lower-case the address field anyway, but the point of this post was meant to be general and it does have a downside too.

You potentially lose capitalisation of the personal part of an email address. For instance:
"Alistair Davidson" <apdavidson@some-domain-im-not-going-to-tell-you.com>
would get flattened to "alistair davidson" <apdavidson@some-domain-im-not-going-to-tell-you.com>

So you want to try to keep the capitalisation if possible.

Probably the best thing to do here is actually to split the personal part from the address, and have the personal part stored as-is, with the address part lower-cased.

Custom Blog Ads Rates?

2007-10-29T10:05:00.000Z

In blatant defiance of all known precedent, someone has not only read my climbing/mountaineering/great-outdoors-in-general blog, but actually thinks it's worth advertising on!

They emailed me to ask how much I'd charge per month for text ads - trouble is, I don't really know, I, like 99% of the rest of the blogosphere, just shoved AdSense on there and forgot about it. Like, really forgot about it, I haven't even had my first Google cheque yet.

So, er.... hmmm.... I don't want to undersell myself, but equally I don't want to give them a price that instantly marks me out as a no-hoper. Ponder, ponder, stick finger in the air...

But I'm a physicist by training, dammit, I should be able to work this out, surely. So let's do a quick calculation.....

It's a pretty low-traffic blog (a couple of hundred hits a day), but it's quite targetted, and AdSense shows an effective CPM of around $3 with a clickthrough ratio of just over 1%.

So, if I'm getting H hits per day, and the CPM (cost per thousand, I think?) is C, then the price per month should be (avg number of days in a month) * (H / 1000) * C, right?

That gives me a baseline price of around $30 / month. Obviously I can adjust that depending on page placement, as ads just below the banner get more clicks than in-page, but does that sound about right for a low-traffic, specialist enthusiast blog?

I didn't know they were so cheap either!

2007-10-29T09:43:00.001Z

i'm hurt we're so cheap
Originally uploaded by clurr

Planning for a 1.5 Terabyte Database

2007-10-13T00:36:00.000+01:00

Here's something to make you stroke your chin and stare into the distance for a moment or two -

A customer asked us about crunching a large amount of email. Some rough back-of-an-envelope calculations lead us to expect up to about 150GB of it in one go. Our experiments with the Enron emails on MySQL produced about a 1:10 ratio of data-in to database size, a ratio of roughly 1:1 on data-in to language model size and full-text index, and a roughly linear (not 1:1 though!) increase in processing time per email with elapsed cumulative crunching time.

So that means we can expect a database size of up to 1.5 Terabytes, plus another 300GB of language models and full-text index...

(We'd be using Oracle for the DB, as it comes with some very handy out of the box management and monitoring tools, performance advisor alerts, and all that kaboodle, and it may be that the 1:10 ratio is different on Oracle - we're looking at that at the moment)

So how do you go about planning for a database of that size?

We can't even defer the question and scale-as-we-go, as it's going to be growing to that size very quickly, within days of kicking off. It's got to be right, straight from the word go. I've worked with very large email datasets before, on Smartgroups, but in Freeserve we had a big team of specialist UNIX engineers to manage it.

How do you spec up the disk configuration, knowing that the database files are going to be that huge? We've tended to go for RAID 1 (mirrored) by default, as an until-now acceptable balance between simplicity and resilience. But this means that if you have, say, 2 x 500GB disks, you only actually get 500GB of storage. Dell provide up to 1TB disks on some servers, but damn it's pricey... we'll no doubt end up going for a combination of mirrored, striped configs, but that means more complexity of course.

On that subject, how do you organise the file system? And, crucially, how do you go about arranging back-ups for all that data? The last thing you want is for 1.5TB of data to get lost with no backup, and have to be re-calculated from scratch. The backups have to be stored somewhere, and even just the process of shifting that amount of data from one storage device to another is non-trivial. I mean, shifting 1.5x10¹³ bits, even over a dedicated, max-ed out Gigabit ethernet is going to take at least 1.5x10⁴ seconds - or just over 4hrs...

Hmm, questions, questions, questions, chin stroke, thousand-yard-stare, tap mouth absently... I'll be pondering this for a while, I think.

Mind you, my old university mate Dan once emailed me back in his PhD days, saying "Hey, you know about databases, don't you? Could you give me a quick intro? I've got to write one to handle data from the SLAC particle accelerator - it's got to handle terabytes of data per day...."
- and that was way back in about 1997 or so, when a 4GB drive would cost you nearly $500. Maybe time to give him a ping on Facebook....

It Lives..!

2007-10-12T23:55:00.000+01:00

OK, so posts have been a bit thin on the ground lately, due to being very busy at work but mostly on pre-sales stuff or client-confidential stuff.... but in general, just being so damn busy all the time.

I've been to the States on a client visit, where we also went to see the supremely-smashing Nellie McKay play at The Birchmere. I'd never heard her before, and Peter just said "She's kind of hard to describe..." - I ended up describing the gig as like Frank Zappa and Paul Simon jamming with Suzanne Vega and Phoebe from Friends, and I think that's about as close as I can get.

I've been sport-climbing in the magnificent El Chorro in Andalucia, which I'll post more about on Dynamove (which I've also been neglecting lately)

I've been studying the Social Dynamics of Werewolves and Lynch Mobs...and if you want an atmospheric bar in which to do so, you couldn't get better than Shunt - it was like walking into the bar in An American Werewolf In Paris

I've also been coding more in the last couple of weeks, working on LDAP importers and such, and I've come up against some interesting questions of scale, which I'll post more about later....

Comment Spam Extortion Racket?

2007-09-07T14:12:00.000+01:00

This comment on Trampoline Systems' blog made me smile -

hello , my name is Richard and I know you get a lot of spammy comments , I can help you with this problem . I know a lot of spammers and I will ask them not to post on your site. It will reduce the volume of spam by 30-50% .In return Id like to ask you to put a link to my site on the index page of your site. The link will be small and your visitors will hardly notice it , its just done for higher rankings in search engines. Contact me icq 454528835 or write me tedirectory(at)yahoo.com , i will give you my site url and you will give me yours if you are interested. thank you

I know you get a lot of spammy comments , I can help you with this problem . I know a lot of spammers....

Kind of like walking into a newsagent and saying "ya know, I can't help noticing you got a lot of paper in here... that's kind of a fire trap, ya know what I'm sayin?"

If this guy really does know a lot of spammers, I'm sure someone out there knows a lot of Federal Agents who'd like to talk to him - his ICQ number and email address are in the quote. ;-)

OMG, I'm a PHB....

2007-09-01T10:33:00.000+01:00

Last week was something of a milestone for me. While gathering my thoughts for the daily stand-up on Friday, the realisation suddenly dawned - I'd had a really busy week, but I hadn't actually written a single line of code all week. I hadn't even fired up my IDE....

So that's it - I am now officially "Management". How did this happen? Seems like just five minutes ago that we were four guys around a table - one CEO and three techies, building stuff. I'd sometimes spend entire days in the office coding away on my own. Now there's 13 of us, and that's just the permanent staff, not counting contractors and non-exec directors, and I've just spent the whole week running around like the proverbial blue-arsed fly, organising, cabling, battling with MS Project, on the phone with suppliers, dealing with customers, in meetings and conference calls, discussing product strategy, allocating work.... etc etc etc... anything, in fact, except coding.

It's funny how things change. Ten years ago I was the hardcore techie sniggering at the back of the room, thinking that the managers just didn't get it and that I would never be in their place. Now I'm the Pointy-Headed-Boss - and I'm actually OK with that... this stuff needs to be done, and I'd rather I dealt with it than have my team disturbed.

...I mentioned this in the stand-up, and was of course greeted with an appropriate chorus of cat-calls, ironic cheers, and a cry of "MAAAANAGEMEEENT WAAAAAAANNNNKKKKEEEEEERRRR!" (thanks Peter!)
So it's nice to know that although the faces change, the culture doesn't.

The Perennial RSS Authentication Dilemma

2007-08-16T20:49:00.000+01:00

It's a common problem, one that has cropped up many times for me over the last few years. You build a secure system, locked up behind a login so that only authenticated users can access the tightly-controlled data, and everything's fine - and then you come to the RSS feeds.

Simply put, RSS feeds and the corresponding use-case of syndicating data out of the application into another application - be it a desktop RSS reader, a web-based aggregator, or even another context within the same system - is in direct contradiction to your security. You can't have an RSS reader log in to your app using the standard login form, and most readers certainly don't support cookies, so you have to provide a bypass.... but what mechanism?

I've tried numerous approaches - most often HTTP AUTH (which some readers support, but not many) or an encrypted url token.

HTTP AUTH is somewhat like an old faithful, it's been around forever, every web server and browser worth mentioning has supported it for years, and it's simple to implement. But it has the disadvantage that once authenticated, the only way to log out is to close your browser completely. Also, many RSS readers don't support it.

Encrypting the users security credentials into a token that you can pass on the URL, is guaranteed to work on anything that can pass on a url correctly, but it has the disadvantage that then anyone who gets access to that url, to all intents and purposes, is that user - so you still have to be careful what you expose in the feed itself.

The main thing, as ever, is to establish exactly what behaviour is "intended". If the brief is for the user to be able to copy-and-paste RSS urls into readers / emails / other sites, then make sure everyone is clear on the implications of that - you're essentially allowing people and / or applications to impersonate a user without going through the login process.

Requirements, of course, vary wildly from app to app, but the approach I've tended to settle on is a combination of multiple methods - if there is a cookie identifying the user, then use that to establish ID, else if you have HTTP AUTH credentials, use them, authenticating appropriately as required. But remember that if you are dealing with automatic requests from readers via HTTP AUTH or encrypted tokens, you should ALWAYS clear out any session variables at the end of the request, otherwise you can quickly end up with thousands of persistent sessions for no apparent reason

Astroturfing in action : social media manipulation exchange

2007-08-15T16:08:00.000+01:00

"Astroturfing" is the art of faking "grassroots" - social media manipulation, in other words.

And here is where you can buy, sell, or exchange Diggs, Stumbles, Reddits, blog posts, reviews, you name it....

Apache Proxied Rewrite Character Set Gotcha

2007-08-13T18:12:00.000+01:00

Character sets, character sets, I'm perenially beseiged by character sets.... I just got to the bottom of a particularly strange character set issue with one of our clients' website.

Their domain points to our apache web server, which rewrites any requests for collaboration engine urls (e.g. somedomain.org/groups/ or somedomain.org/people ) in order to send them to the collab platform, and rewrites any other urls to go directly to a third-party hosted CMS-driven website.

The problem they were having was that when viewing the live, proxied site, strange characters were appearing in the generated content - e.g. "?" and that funny square box that handily tells you "you got the character set wrong, fool" - but when they looked at the site directly on the third-party CMS, it was fine.

Using Firebug to inspect the headers showed that when viewing the third-party CMS, the Content-Type was:

Content-Type text/html

but when proxied via our Apache web server, it was:

Content-Type text/html; charset=UTF-8

The meta tag in the HTML itself was :

< META http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" >

- and the ensuing character set mismatch was causing the dodgy character funkiness.

It turns out that buried deep within the bowels of our Apache config was this line:

#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8

Which was doing exactly what it said on the tin - forcing UTF-8 interpretation whatever the META tag said. Rather than changing it in the central config, I overrode it in their specific config with this line :

AddDefaultCharset Off

and it seems to work fine now. So that's a big "yay" for Firebug, a dependent "yay" for Firefox's plugin/extension architecture that allows and encourages third parties to develop these funky plugins, and a big slap round the face with a large fish for anyone who's not using UTF-8 in this day and age....

Rails Migration Version Headache - D'oh!

2007-07-31T16:08:00.000+01:00

I've been having real headbanging frustrations with Rails ActiveRecord migrations not picking up the correct target version. On the face of it, this should be about as simple as you can get -

- To determine the current version, the migrate task looks in your database for a table called "schema_info", and reads the value of the "version" field from it. If the table is not there, or is empty, then version is nil, and nil.to_i = 0.

- To determine the target version, the task gets the highest-numbered file from the db/migrate directory and parses an integer out of the filename.

OR

- You can override the target version by passing in a command-line parameter like so:
rake db:migrate VERSION=XXX

So, as the famous "outraged of Tunbridge Wells" might say, why oh why oh why was my migration task always reverting back to previous versions, despite having migration files numbered sequentially right the way up to version 19 (and counting) ?

In vain did I bang my head against the desk, curse migrations and all they stood for, implore the heavens to rain down brimstone upon the head of DHH and all his acolytes, and impugn the parentage of my PC, Windows, Oracle and databases in general....

On a trawl through the Railties source code, it turns out there's a third, more surreptitious way to tell the migrate task which version you'd like to migrate to - if there's an environment variable called VERSION, it will take the value from that. And when I looked at my System Variables in XP, what did I find?

VERSION=3.5.0

Ah.

OK.

My bad...

A Four-Way Grudge Match: Oracle vs. MySQL vs. Views vs. Derived Tables, eek..!

2007-06-28T12:13:00.000+01:00

I like Views. SQL Views are great. Unless, that is, you have to support certainly two (MySQL and Oracle) and ideally three (plus SQLServer) of the major DB systems with the same code. Yesterday I was forced into a dastardly hack that made me wince with it's ickiness, but was actually the most practical solution given the constraints.

The Situation

We have a view which combines a LEFT OUTER JOIN with an AVG and a GROUP BY, to "collapse" what may be many related records down in to one. The AVG gets an average link weighting, to give you an effective measure of how closely two items are related by all the various different linking mechanisms.

This view references a table - item_links - that is expected to grow very rapidly, to the size of many millions of records.

We INNER JOIN onto this view in application code, to get item records related to the "current" item.

The Problem

On Oracle, this view works really well. Nice and fast, at most a tenth of a second or so per query. Doing an EXPLAIN in Oracle's SQL Developer shows that the execution plan uses all the expected indexes. Luvvly jubbly, bosh, sorted, etc.

On MySQL, however, view support is much less mature. The query optimiser can only use the underlying indexes if the view is created with CREATE ALGORITHM=MERGE and the docs say:
"The MERGE algorithm requires a one-to-one relationship between the rows in the view and the rows in the underlying table. If this relationship does not hold, a temporary table must be used instead. Lack of a one-to-one relationship occurs if the view contains any of a number of constructs:

Aggregate functions (SUM(), MIN(), MAX(), COUNT(), and so forth)
DISTINCT
GROUP BY

...

... so instead, what MySQL does is select all rows into a temporary table, which is then used for executing the statement against.

Doing an EXPLAIN in MySQL shows that for the crucial operation of selecting the required rows from this potentially-huge table, there are no indexes used AT ALL. It is in fact selecting EVERY row into the temp table.

All of which leaves me - and, I must say, in blatant defiance of all known precedent - in the truly strange position of cursing MySQL whilst praising Oracle.....

Which worries me - I've never been here before.... :-)

The Solutions That Won't Work

Get rid of the view and just put the sql into each query,
i.e. change INNER JOIN vw_item_links to INNER JOIN item_links ON (blah) LEFT OUTER JOIN link_prefs ON (blah) etc etc
Hmmm.... it's possible, but I would say it's a last resort, because that would mean that we'd then have to GROUP BY every single field in every query, not just the fields in the view. This can get majorly cumbersome, because although MySQL allows you to just group by just-enough-fields-to-get-uniqueness (and thus gets a yay for ease-of-use), Oracle insists on grouping by every single field that's not an aggregate (and just about gets a begrudging yay for insisting on standards compliance)

Eliminate the view and the GROUP BY through using subqueries
i.e. SELECT (item fields), (SELECT AVG(weight) FROM item_links WHERE ...) AS weight, (SELECT other_item_id)
This is a bit of a null option, unfortunately, as we'd still have to INNER JOIN onto item_links in the FROM clause to get the related records, so we're not really gaining anything, and in fact we'd be introducing more overhead through a correlated subquery executed for every row.

So what else can we try? Anyone? Ah yes, you boy, there at the back....

The Solution That Did Work

You can, in fact, use subqueries in the FROM clause to create DERIVED TABLES, like so:

SELECT (fields) 
FROM items 
INNER JOIN ( your view definition SQL ) inline_vw_item_links
        ON inline_vw_item_links.other_item_id = items.id
WHERE inline_vw_item_links.item_id = ...
ORDER BY inline_vw_item_links.weight DESC

in effect, putting your view SQL into the query as if it were a regular table

This does mean that you lose some of the nice encapsulation of the view, and if the view definition is complex it makes your queries look ugly, but at least it works across both DB platforms. You can also get round the ickiness problem by holding the view definition SQL in a variable :

SELECT (fields)
FROM items 
INNER JOIN ( #getViewDefinitionSql()# ) inline_vw_item_links
        ON inline_vw_item_links.other_item_id = items.id
WHERE inline_vw_item_links.item_id = ...
ORDER BY inline_vw_item_links.weight DESC

But hang on, there's one more subtlety - (and this is the icky bit)

An EXPLAIN on Oracle shows that this query still uses all the appropriate indexes, as it should. Smashin'.

But MySQL, although it's now recognising that there are indexes there to be used, is still selecting ALL rows from item_links. Which, to be fair, is what the query is *strictly* telling it to do.

So here's the icky hack:

We put the item_links WHERE clause restrictions into the inline view definition SQL, so that it looks like this:

SELECT (fields) 
FROM items 
INNER JOIN ( 
    SELECT AVG(weight) AS weight, other_item_id, (etc...)
    FROM item_links LEFT OUTER JOIN (blah) ON (whatever)
    WHERE item_links.item_id = #my_item_id# AND shared=1
 ) inline_vw_item_links
        ON inline_vw_item_links.other_item_id = items.id
WHERE inline_vw_item_links.item_id = ...
ORDER BY inline_vw_item_links.weight DESC

which you can then prettify a bit for encapsulation like this:

SELECT (fields) 
FROM items 
INNER JOIN ( 
    #getViewDefinitionSql( "WHERE item_links.item_id = #my_item_id# AND shared=1" )
 ) inline_vw_item_links
        ON inline_vw_item_links.other_item_id = items.id
WHERE inline_vw_item_links.item_id = ...
ORDER BY inline_vw_item_links.weight DESC

Now this makes me feel icky. It's (effectively) mixing your selection criteria in with your join criteria, which violates my rules for "good" sql, and goes against what I've always thought of as best practise - letting the DB engine's query optimiser work out the execution plan for itself. And it's just...wrong...

...But it works. An EXPLAIN shows that MySQL is now using the indexes that it should be using, and only selecting the rows that it needs from the potentially-many-millions-of-rows item_links tables. And the benefit is blatantly obvious in execution times, turning what was a several-second query execution time into a few tenths of a second. And Oracle is still happy. I suspect that it was previously optimising the view SQL into effectively this query anyway, under the hood. So I'm swallowing my SQL purist pomposity, and going with it. But if anyone can think of a more "proper" way of doing this, I'm all ears.

Anyone fancy a football game TONIGHT? 8pm, Willesden CCA

2007-06-28T11:06:00.000+01:00

Due to some last-minute drop-outs, we're two men short for our 11-a-side match tonight, 8pm, at Willesden CCA. We're desperate enough, in fact, for me to test the power of the online social network and the blogosphere blah blah blah.... to the limit by posting this.

We ideally need a right-back, and a goalie, but to be honest we'd settle for any two reasonably competent players in any position, cause we're all fairly flexible.

So, if anyone fancies it, let me know ASAP!

Things to know:

We're not exactly stuffed full of Ronaldinhos, but you should at least be able to control and pass a ball, and run around for a full 90 minutes. Or play in goal!
It's on astroturf, so there's no studs and very few sliding tackles
I can bring spare shorts, shirt, socks, shinpads and a pair of size 11 trainers if needed - obviously it's best if you have your own trainers

How to make yourself feel old, instantly

2007-06-26T14:41:00.000+01:00

The baby on Nirvana's "Nevermind" album cover -

- now looks like this:

....sheesh....

"Folksonomy" : The Most Irritating Word On Teh Intarweb!

2007-06-22T11:02:00.000+01:00

It's official!

At least, according to YouGov that is.

And the second and third most annoying words on the internet?
"Blogosphere", followed by "blog". "Wiki" came in tenth - so evidently wikis still have some way to go before people are sick to death of them. Maybe as they gain more traction in the enterprise, the word will rise through the ranks?

So next time you trumpet the social software manifesto, bear this in mind...

When Facial Hair Becomes Compulsory

2007-06-14T15:44:00.001+01:00

DSC_5998
Originally uploaded by mccraig

It had to happen eventually - and today it did. By directorial decree, strange facial hair moved from a mere convention to a compulsory requirement at Trampoline. No exceptions!

I was aiming for a Zappa look, but unfortunately it turned out to be alarmingly close to George Michael... look out for me slumped at the wheel of a beemer in a Hampstead layby near you...

Last.fm - yours for £140m, squire....

2007-05-30T15:27:00.000+01:00

Woah - the Web 2.0 buyout bubble comes to Shoreditch, with our near 'n' dear neighbours last.fm announcing today that they've been bought out by none other than CBS for the eye-watering, pant-moistening, gast-flabbering sum of £140 million.

Congratulations guys! And there we were thinking these kind of deals only happened in California... GIT'chaw web 2 businesses 'ere, mate, get'em while they're hot...

Anti-Social Social Software - How far is Too Far?

2007-05-23T10:26:00.000+01:00

The social software revolution has been, more or less, about openness. Transparency and freedom have been the order of the day. "Information wants to be free!" goes the mantra, and by and large it's hard to argue against it. All kinds of information that has always been publically available is becoming gathered together, cross referenced and made searchable by the new wave of social software mashups and visualisations, and those who expressed doubt or indignation about this approach have been dismissed as relics of a bygone era, who "just don't get it".

(I'm including myself in this, by the way. Trampoline's Enron Explorer is a perfect case in point. The information was already public domain, and we weren't even the first to visualise it - we just put a Web 2.0 face on it, and got lucky with the timing.)

But now and again there comes an application of this philosophy that makes me wonder how far is too far? Enter the charmingly named www.whosarat.com.

According to the New York Times:

The site was started by Sean Bucci in 2004, after he was indicted in federal court in Boston on marijuana charges based on information from an informant.

Initially free, it now charges charges between $7.99 for a week and $89.99 for life. You don't have to be as cynical as me to take a guess as to the kind of people who will happily pay to see names and mugshots of the 4,300 informers and 400 undercover agents that the site says it has identified.

So is this too far?
Is it possible to define a clear moral line between a "good" use of public-domain data and "bad" ? Even if it is possible, should we?
Should some usages be permitted but not others? How do we tell the difference?

What is it that actually makes me uneasy about this?

The trouble is that any time you actually try to define what's acceptable and what isn't, in clear unambiguous language, you can always find a counter-example that destroys your mental partitioning of the world and means you have to start again.

The only way I can put my finger on what bothers me about the site is to refer to the intent behind it. Where the intent behind a mashup like Chicago Crime could reasonably be argued to be beneficial to society in some way, Who's A Rat is clearly not.

But even that definition is full of ambiguity. Who's A Rat may well be of immense benefit to certain segments of society (Hal Helms' "Vinny" persona, for one) and Chicago Crime could be argued to be detrimental to, say, property owners in high-crime areas who are looking to sell.

I guess the hoary old cliche applies just as much to information as to people : with great freedom comes great responsibility. We can't on the one hand argue that information needs to be free, yet on the other hand argue that it can only be free for usages that we like. Either the information is out there or it isn't, and I guess that if we want to gain the benefits of information freedom, we have to be prepared to tolerate the drawbacks.

Come Back XP, All Is Forgiven

2007-05-11T10:49:00.000+01:00

A couple of months ago, after the burglary, I replaced the nicked big-red-beast of a laptop with a shiny new Core 2 Duo / 2GB of RAM laptop, powerful enough to run SONAR, all its dependencies, and a Java IDE at the same time without dying. It's still heavier than your average laptop, but not quite so much as to generate its own gravitational pull, like the old one. (It's so much more calming to be able to site down and browse t'intarweb without small objects orbiting your laptop, you know? The accretion disk of pens and paperclips always got in my way, and the time distortions played hell with my google calendar...)

Anyway, at the time I bought it, the *ONLY* OS you could get from Dell was Vista. I ummed and aahed, and debated and discussed, and eventually decided to ignore my own standard policy of wait-till-at-least-service-pack-1-before-installing-any-new-MS-OS and take the plunge. It arrived, and I admit I was pleasantly surprised..... at first. Until I tried to put any software on it.

Firefox often (about half the time) won't start, the process just sits there using ~ 4M of RAM and never even getting the splash screen, and the process can't be killed. I have to reboot the laptop and try again.

The Apache 2.2 service won't start.

The Bluedragon 7 JX service will start, but immediately stop.

Windows Firewall drives me so nuts I had to turn it off to prevent an unfortunate episode involving me, a shopping mall, a chainsaw, and visions of GTA3.

User Account Control - yes, yes, I KNOW it's off! I turned it off deliberately, dammit! Stop telling me about it!

Office 2007 - a brave idea, to depart from the established interface so completely, but where the hell is the "Save As..." command? Where's it gone?! The help file tells you how to use it, but doesn't tell you where it is! Can't I load a document, make some changes and then save it to a different filename anymore? I'm downgrading.

There's a pre-installed copy of CineLink PowerDVD that I've NEVER run, but seems to be permanently resident and using up to 200MB of RAM - is this a DRM thing..?

Roll on service pack 1....

New Trampolinees Needed!

2007-03-21T09:39:00.000Z

We're hiring! Oh yes indeedy - techies are needed at Trampoline, with experience of some or all of these :

Java, Hibernate, Spring framework, JMS, Ruby, Ruby on Rails, Transactions, Asynchronous systems, Transactional messaging, SQL, MySQL, Oracle, Linux sysadmin, Windows sysadmin

But far more important than that is that they just fit. Come help us drag the enterprise market kicking and screaming into the 2.0 / 3.0 / buzzword-of-your-choice world. It's fun! It's hard work, but it's fun.

"Will Code For Equity" - sound like anyone you know?

2007-03-16T10:21:00.000Z

A friend of mine has an old CF/Fusebox site based around moving house, and finding companies to help you do that. She wants to completely revamp it and bring it in to the 2.0 era with user feedback, recommendations and reputations, etc, and she needs a developer.

Trouble is, although her existing site has been supporting her for two years, she doesn't have much money up front for a dev to revamp and relaunch it.

So the deal she's offering is, basically, take charge of the site and get free reign to do whatever you think is a good idea, in return for a decent chunk of equity and a corresponding share of the profits.

I'm not taking on any side work, I've got my hands more than full with Trampoline, but I told her I'd ask around, so if this sounds like anyone you know (and, crucially, you'd be happy to recommend them - she is a friend, after all, and I don't want to see her get ripped off) please give me a shout. Email address is in the "Other Links" section on the right, just under my gurning fizzog**

Sheesh. Coding for equity - does this sound like Bubble 2.0 to anyone else....?

**Translation for non-Northern people: grinning face

Finally, I Can Say It!

2007-03-15T11:19:00.000Z

Phew, at last I can say what's been going on, why I haven't been blogging much recently, etc etc - it's because we've been working so hard to close THIS: Trampoline Systems receives £3 million funding. YAAAAAAAAAAAAY!!! We did it!!! Woo and Yay and quite possibly Hoopla aswell! It's going to make a huge difference to the business, and let us step up a level or two in scale and approach.

I can also reveal that Dante was wrong. There is an eighth circle of hell - it's called "Due Diligence" and is full of lawyers. Lots of lawyers asking lots of questions that make you feel like you're being gradually eaten away by a swarm of bees, and make you want to stick your face in a deep fat fryer just to be rid of them.

But you'd have to make sure you kept paper copies, in triplicate, of the license agreement from the fryer company that gave you explicit permission to use it for face-boiling purposes, and prove that you complied with every letter of the fryer license in distributing the bits of hot fat around the room...

So here's a bit of advice. If you think that there's even the slightest possibility of your company ever getting outside investment, or being taken over, or floated in an IPO, make DAMN SURE that you :

Keep a record of all open-source software you use, whether you redistribute it or not

keep copies of the license agreements for ANY open-source software that you use, whether you distribute it or not

keep proof that anything you re-distribute has been distributed in accordance with the terms of the license
- i.e. if it's an MIT license, you have to provide a copy of the license and copyright notices along with any redistribution, so just keep a copy of what you provide to anyone else

read and understand the license terms before you redistribute!
- this is important, really! For instance, if you provide anyone with a copy of, say, the Oracle OCI client driver, pay close attention to the documentation requirements that it puts on you, and the implicit permission that gives Oracle to audit you at any time.

It's quite straightforward to do this whenever you download some open-source stuff. Just keep a directory somewhere that you can dump license agreements and copyright notices, and keep, say, a Google doc spreadsheet that just records the name of the library, the URL to download it, and the type of license it is under.

If you keep these up to date as and when you download OSS, it only takes a few minutes each time. If, like us, and I'm sure, most people, you didn't think to do it as you went along and waited till you were asked for all the information on everything, right now - then you're in for an awfully long slog of awfully tedious work. A little bit of diligence now saves a massive amount of retro-diligence in future.

Now, where's those Trampoline-branded space hoppers, baths full of champagne, and toilet roll stock option dispensers ? Boo.com 2.0, here we come.....

Irish Donkey Sex Case Brings Down Galway First

2007-03-02T13:10:00.000Z

I'm speechless - there are so many things to love about this story I don't even know where to start, right down to the name of the receptionist and the guy's claim that it was a "super-rabbit"... and is there really such a thing as the "Unlawful Accommodation of Donkeys Act"?

Irish Donkey Sex Case Shocks Net

Was there really so much dubious accomodation of donkeys going on in 1837 that they felt the need to pass a specific law against it?

[OT] [Rant] That's it, that's the last straw

2007-03-01T09:54:00.000Z

That's it - we can't pretend any more, it's finally undeniable - Hollywood has officially run out of ideas completely.

Honestly, in the last couple of years we've had execrable remakes of classic Asian films like The Ring, Ju-on (aka The Grudge) and The Eye, and now they're even plundering Battle Royale. I'm not even going to mention the appalling crime against humanity that was the remake of The Wicker Man.

I give up. I really do. Satire has become the new truth. You can't even satirize anymore, there's no point - reality has become the ultimate ironic piss-take of itself. Just take a look at this article from The Onion - U.S. Dept. Of Retro Warns: 'We May Be Running Out Of Past' dated 1997, and try to tell me it hasn't actually come to pass.

The saddest thing is that most westeners will only see the remakes, and so they'll associate the titles of these fantastic movies with the watered-down, over-produced, bastardised Hollywood interpretations that are (as far as I can see) mediocre at best, rather than the classic originals.

Honestly, can anyone give me one original idea that Hollywood has produced in the last couple of years? And yet, it's undoubtedly piracy that's causing declining revenues, oh yes - not the fact that virtually everything they put out these days is utter shite, oh no. It's them evil pirates, yes sir...

*sigh* I need coffee.

StatSVN - Ruby vs. Java and "Developer Of The Month"

2007-02-23T12:21:00.000Z

I'm a big fan of SVN, and I know it's in very common use throughout the development community, so I thought I'd give a shout out to an interesting project that extracts some fascinating SVN stats and draws lots of pretty pictures - StatSVN

We've now added a StatSVN task to our build scripts, and the figures it kicks out are just great - once you start looking, you can lose yourself in there for ages....

To give some random samples:

As of three days ago, there were 75130 lines of code in the Sonar trunk, of which Craig has contributed 33.6%, myself 32.1% and Jan 22.2%. As Jan has mostly been working on the Ruby On Rails interface, and Craig and I have mostly been writing the Java backend, what does this say about the relative expressive efficiency and productivity of each language?

Craig took "Developer Of The Month" for February, with 5286 lines, taking over from me in January with 7546 lines.

Over 80% of my changes are additions, with under 20% being updates. Craig is more or less the same, with a few percent more modifications - probably due to fixing my occasional "EVIL" quick hacks...

By far and away the most commits overall get done between 4pm and 5pm, but personally, I do three times as many commits between 2pm and 3pm than any other hour. Probably because I keep saying to myself got to finish this before I go to lunch...

Jan has not done a single commit before 12 noon :-) However, he's the only person to have committed between 1am and 2am!

I do a huge number of commits on a Monday (400+), followed by successively fewer every day until Friday (~60). Does this mean I'm fresher and more enthusiastic after the weekend, or is it that when I start the week, I go for the easiest tasks first, and ramp up towards the more difficult tasks as I go on? On the other hand, overall the most commits are done on a Friday, followed by Tuesday

You can also measure some metrics around files, rather than developers:

Overall, across all file types, we have an average of around 40 lines of code per file

58.7% of the files in the repository are .java, but they contribute 73.3% of the total lines of code

On average, each .rhtml template is just 26.6 lines of code

The most amended file in the whole repository is the Rakefile!

I could go on for hours....but it's nearly 2pm and I need to commit some code, dammit! So install StatSVN on your repository and have a play yourself. It's fun.

Roy Orbison In Clingfilm In Print!

2007-02-21T10:42:00.000Z

(ahem) Allow me to explain that title.... A few years back, myself and some of the Freeserve guys stumbled across the cringe-inducingly hysterical Ulli's Roy Orbison In Clingfilm site, a pastiche of Slash Fiction consisting of a selection of stories involving Roy Orbison turning up unexpectedly, and by increasingly unlikely chains of events, ending up wrapped in clingfilm.

If you haven't read it, and you fancy skirting that fine line between weeing yourself laughing and feeling unaccountably disturbed, go have a look now, and then come back... go on, it's ok, I'll wait...

Back? Good. I'll continue.

"Ulrich Haarbürste" is a nom-de-guerre of Michael Kelly, whose Page of Misery is a veritable treasure trove of warped, cynical, misanthropic comedy gold, and has been the source of several viral emails that you might have had at some point ( French Intellectuals In Afghanistan, anyone? ).

Anyway, when we discovered the delights of Roy Orbison In Clingfilm, I felt so inspired that I set the first story to music, using the Microsoft Agent control to read the text in a Steven Hawking stylee, and using various Cubase plugins (e.g. Voice Designer) to change the pitch and formants of the voices for Roy and Ulli. It was quite a fun experiment, and once I was done, I sent a copy to Michael. He loved it - and so did a growing number of people - if you Google for "Al Davidson Roy Orbison In Cling Film" it turns up in a surprising number of playlists...

So I'm genuinely chuffed to hear that Michael Kelly has produced a Roy Orbison In Clingfilm book! You can buy it HERE - go on, you know you want to, it'll make a delightful coffee table discussion piece, guaranteed to break the ice at parties, and scare off the neighbours' children too!

I should just add that I'm not getting any referral kickbacks off this - I just think the esteemed Mr Kelly's talent deserves all the exposure it can get, and I hope he gets showered with riches and future publication contracts from his book. Tell your friends. And after the week I've had, a bit of warped humour is exactly what I need. Tell the world! Lets see if we can get Roy Orbison In Clingfilm into some best seller lists!

PS - To celebrate the occasion, I moved the song from its previous home onto a more permanent home at last.fm, along with a couple of other self-indulgent noodly guitar instrumentals. I keep meaning to finish off the hundreds of tunes-in-progress I've got kicking around on my hard drive and upload those too, but somehow there's never quite enough time...

The Icing On The Cake

2007-02-20T09:26:00.000Z

Did someone break a mirror or something?

I spent the weekend clearing up after the burglary and running around like a blue-arsed fly, getting locks replaced and notifying the insurance company, getting some alarms and setting them up, trying and failing to get someone to come and replace the back door with its now-gaping-hole-where-the-catflap-used-to-be, and ending up hacking together a temporary repair with two plastic chopping boards and big tube of superglue (I kid you not!), and was finally starting to feel secure in my own home again.

So I made a conscious decision to try and relax on Sunday evening. I put my feet up, had a couple of beers, settled into the sofa for a wind-down in front of the TV followed by an early night, when about nine o' clock the phone rang. It was the tenant from one of Lisa's parents' flats.

"Can you ask him to call us urgently?" she said. "There's been an explosion!"

Lisa's parents own the whole building, and rent out the bottom floor to a restaurant, and the two flats above it to private tenants. We used to live in the first floor flat.

As it happens, the restauranteurs had been having some renovation work done, and there'd been a gas leak in the kitchen.... and on Sunday night, it went BOOM. Quite loudly, apparently. The police were round, and needed to get the landlord's details. One of the tenants in the first floor flat was hurt and had to go to hospital. The guy who'd been doing the renovation work has disappeared back to Morocco, and can't be contacted.

I'd already had a couple of beers, so I couldn't drive anywhere that night. So I dashed round there 7am the next morning to take a look at the first floor flat - the gas had obviously gone up the middle of the wall between the kitchen and bathroom, and when it ignited, it blew out that wall on both sides. The kitchen units were wrecked, and there's damage to the wall and ceiling.

Do you ever get the feeling that the universe is trying to tell you something?

At least I've still got my guitars - I feel a country song coming on...!

The Curse Continues....

2007-02-16T07:58:00.000Z

Every time Lise goes away without me, something bad happens - we joke about it, but I'm not really in the mood for laughing right now. She's currently in Malaysia with her family, eating superhuman quantities of food and visiting relatives, and me..? Well, I got back from work yesterday to discover we'd been burgled.

I knew something was strange as soon as I got back from work about 8pm, because I couldn't get in. My key was turning in the Yale lock, but it wasn't opening. I had to knock on the neighbour's door and ask them to let me through into their back garden, so that I could get into our garden and see if I'd left a window unlocked anywhere, although I was sure I hadn't.

At this point I thought I'd just managed to lock myself out somehow, or something had gone wrong with the lock... I fitted it myself, so it's perfectly possible that I messed it up somehow - I inherited lots of things from my Dad, including a tall slim build, a stubbornly-resilient head of hair and an overpowering love of guitars, but sadly his skill with DIY completely passed me by.

So, bumbling about in the back garden with my neighbour, I tried the back door, and it was open. "Yay for my lax security!" I cried, big slightly-embarrased grin with the neighbour, gaw-blimey-what-a-muppet-I-am-guvnor, forget-me-own-head-if-it-wasnt-screwed-on expression on my face as I went inside. And it was as soon as I walked into the kitchen that I realised what had happened.

It looks like they kicked in the catflap in the back door, then managed to get to the key, which was in the lock - either with a very long thin arm, or some bent coat-hanger kind of contraption, I don't know. But I'm sure that's how they got in. I figure they must have then gone straight through to the front door and flipped the deadlock switch - so they wouldn't be disturbed if I came home early and tried to let myself in through the front door.

The bastards had torn everything off the shelves and emptied every box onto the floor, in every room, evidently looking for money or small items they could fence quickly, like jewellery. They tore all our clothes out of the cupboards and drawers, went through everything, and trampled lots of back-garden mud into our beautiful fluffy white deep-pile bedroom carpets in the process.

Luckily, they didn't steal much. They must have been in and out within a few minutes. They took :

a Nikon Coolpix 6MP digital camera
possibly an old Sony Ericcson mobile, but most importantly
Lisa's Dad's Toshiba laptop.
possiby some of Lisa's jewellery.

Anyone who's seen me present at a London CFUG will probably remember the laptop - it's HUGE (17"), shiny and red, and weighs an absolute ton. So if by any chance you see it cropping up on eBay or in your local Cash Converters, email me.

It could have been far worse, of course. They didn't take:

any of my guitar collection
my main PC
the TV, DVD player, X-box, etc
my passport - phew! - despite all my paperwork living in orderly, clearly-labelled box files
Lisa's treasured Jimmy Choo shoes
any of the climbing/mountaineering equipment, despite going through it all
my stupidly expensive mountaineering watch, which I bought with my leaving present (gift vouchers) from Headshift

(Sadly they didn't even take the all-in-one printer that I ordered online ages ago, without realising what a huge great monstrous carbuncle it was going to be, and have regretted buying ever since)

And they didn't leave any nasty surprises on the carpet / up the wall / etc. When I got burgled back in Bridlington (by a guy who I thought was my best friend at the time) they left a nice big turd in the middle of the carpet, and killed the cat in the process. When my mum got burgled a few years ago, they smeared shit up the walls and left what I'll just call a "deposit" in her underwear drawer. I'm thankful that they didn't do any of that in our flat.

It must be a sign of the times that I was far more worried about them stealing my identity and information than my posessions. They now know my name, what I look like, where I live and where I work. I'm still worried about what was on the laptop. Did I have any banking information on there? Did Lise? Too late to check now. Hopefully if they didn't steal my passport, they're not sharp enough to realise what they have, and they'll just format the disk before flogging it on - but you just don't know, do you - and that's going to worry me.

Things like this make you realise, in fact, that when someone breaks in and steals your stuff, it's just stuff - and stuff doesn't actually matter that much. Stuff can be replaced. Mess can be tidied up - it took me until 2:30am this morning, but I cleared it all up. What matters is people, and I'm just grateful that Lise wasn't here when they came, and that she didn't get the huge shock I did when I walked in, and that she didn't have to see the mess they made of our beautiful home.

What gets me, though, is that feeling of being violated. "An Englishman's home is his castle" as the saying goes, and my castle feels...soiled. Fuckers.

I've now got a fun-packed day to look forward to today. The police were too busy to come round last night, so they're coming this morning, probably just to take my particulars and give me a crime number for the insurance. I'm sure they've got more urgent priorities, like trying to stop fifteen-year-olds shooting each other, which seems to be the preferred pastime-du-jour. I've got to file the insurance claim, get the locks changed, and try to get the back door replaced, if I can.

And I also have to tell Lise, and her Dad, over a dodgy mobile phone reception and an 8hr time difference, and I just know that it's going to break her heart. It;s going to ruin the remaining week of her holiday, and she's going to worry herself sick every day until she comes home.

So let this be a cautionary tale. Never leave your key in the lock! And if you, like me, keep looking at your door or windows and thinking "hmm, really ought to get round to getting that properly fixed-up at some point..." then get round to it TODAY! OK? Ok...

DevNotes On AJAX

2007-02-08T15:15:00.000Z

As a long-time user of the FLiP methodology, there have been many times where I've created a prototype and used Hal Helms' DevNotes as a plonk-and-play way of capturing clients' thoughts and feedback.

It's a nifty little tool that does the job it sets out to do - but since it was released way back when, teh intarweb has moved on, and I've had a few niggly things that I had to work around when using it :

it's rendered using "old-school" HTML, with things like FONT tags

sometimes you can get conflicts between the DevNotes form submissions and parameters that the containing prototype app requires

the dependency on cfx_make_tree.dll didn't play nicely with a non-Windows OS

So, while noodling around over the weekend recently, I decided to give it a bit of a facelift. One thing led to another, and it ended up as a pretty major update. So laydeez and gennullmun, allow me to present....(drum roll)

DevNotes On AJAX!

Main Changes:

Removed dependence on CFX_Make_Tree.dll,
replaced with udfMakeTree.cfm from cflib.org, by Michael Dinowitz
Ported to be based on Ajax requests,
de-coupling the DevNotes requests from the Prototype page requests, and allowing people to stay on the same Prototype page while adding notes
Added RSS syndication of notes.
Prefixed all request variables and JS functions with DEVNOTES
to prevent any potential conflicts with the containing app
Required parameters in the containing app are now:
// these fields required for DevNotes request.DEVNOTES.devAppName = "UniqueAppname"; request.DEVNOTES.attributeToKeyOn = "page"; request.DEVNOTES.DevNotesDSN = "YourDSNName"; request.DEVNOTES.devnotesRSSTitle = "Title for the RSS feed"; request.DEVNOTES.DevNotesRSSDescription = "Description for the RSS feed";
Updated HTML to be more semantic and standards-based.
Admittedly it might not be 100% XHTML-compliant, but I don't really have time to check it on that - be my guest...
Introduced CSS styling
and grayscale mini icons from Brand Spanking New
Re-factored the code into a fusebox3-stylee app structure.

Example usage:

Extract the archive into a subfolder called "DevNotes" under the root of your prototype app.

Make sure that the following variables are set up somewhere in your app - fbx_settings.cfm is usually the best place :


// these fields required for DevNotes
request.DEVNOTES.devAppName = "UniqueAppname";
request.DEVNOTES.attributeToKeyOn = "page";
request.DEVNOTES.DevNotesDSN = "YourDSNName";
request.DEVNOTES.devnotesRSSTitle = "Title for the RSS feed";
request.DEVNOTES.DevNotesRSSDescription = "Description for the RSS feed";

Then in your prototype's LAYOUT file, just add
<cfinclude template="../DevNotes/DevNotes.cfm" />
(replace the path as necessary)

NOTE: don't put this in OnRequestEnd.cfm, as it will get triggered for each AJAX request and land you in an infinite recursive loop! The layout file is the best place for it, as it's display-generating code.

Usual usage policy : you can do what you like with this code, so long as
a) you maintain the attribution comments
b) you accept that it's provided entirely as-is, with no warranty of any sort
c) you don't violate any terms in the other code that it includes (see credits within the code)
and most importantly
d) you don't have too much of a go at me for not commenting it particularly well!

Enjoy !

DevNotes On AJAX

We Have A New Paperweight

2007-02-08T12:44:00.001Z

New Paperweight
Originally uploaded by Dr Snooks.

We scooped a TrampoGong last night - the Oracle Partner Innovation Award for 2006. Yaaaay! Although, after sampling the varied delights of the hospitality bar, it's hard to tell whether the award or the coffee was more appreciated first thing this morning....

Oracle, Sonar and Schnow, oh my

2007-02-07T16:23:00.000Z

'ello, yes, I am still alive... bleh - been busy busy busy recently, it's been quite a week or two....

We had our first major deployment of Sonar to get out, so none of us got home until after midnight until Thursday. Eventually got it done, though, to huge relief and slaps on the back all round, thanks to some Herculean efforts from everyone on the team.

After a week like that, I needed to get right away from it all, so a weekend in the mountains of Scotland for some ice climbing was just the job. I learned quite a lot over that weekend -

- supporting my 12-and-a-half-stone on four tiny metal spikes sticking out of my toes gets pretty damn painful on the calves after four hours or so
- just because the weather is gorgeous right now, doesn't mean that it's not going to be bloody grim soon
- and many more

I got back to work yesterday, with just one day to recover before tonight's extravaganza at the Oracle Partner Innovation Awards - we've been nominated for the "Hottest Tech Prospect In The UK" award, so we're off to meet 'n' greet 'n' press the flesh with the industry bigwigs.

...which was nice...

Hopefully some point soon I'll be able to get back to some technical posts on this 'ere blog, as I have a couple of goodies to impart (even one CF-related, shock, horror) and I'll post them as soon as I get chance to package them up.

"Fed-Ex" Social Engineering / ID Theft Scam?

2007-01-22T09:45:00.000Z

At 7am this morning I was woken up by the phone ringing. I didn't make it to the phone in time, and it rang off. "Who the hell is ringing me at 7am?" I thought, in my usual semi-comatose, semi-neanderthal pre-coffee state, but a 1471 said "we do not have the caller's number". Hmmm....

Twenty minutes later it rang again. This time I got there in time to pick up -

"Hello?" I said.
(Well, admittedly, at that time in the morning it was probably more like "blrglhmph?", but you'll have to perform the necessary transliterations yourself for the rest of this transcript)

A man with a very strong Indian-type accent replied:
"Hello, can I speak to Mr. Alistair Davidson?"

"Speaking..."

"Is your address [my address and postcode] ?"

"Yes...."

"This is Fed-Ex, we have a package to deliver to you this afternoon"

"Oh ok..." ...so why do you need to ring me at 7am about it? I thought

"I need to confirm some security details - can you give me Mr Alistair Davidson's date of birth?"

Now at this point my tinfoil-hat alarm bells started ringing -

why would Fed Ex possibly want my date of birth in order to deliver a package?

come to think of it, how would they know my telephone number?

in fact, how did I know this guy was from Fed Ex at all? He said he was calling from Fed Ex, but I could ring up anyone at random and claim to be Kylie Minogue, it wouldn't make it true...

I decided to play it cautious -

"Why do you need my date of birth?" I asked

"I need to confirm these security details to deliver the package" he said

"Well I'm not going to give that information out," I replied

"But we won't be able to deliver this package" he was starting to get a little tell-tale this-isn't-the-way-it's-supposed-to-go tone of voice

"Ok," I paused, thinking - in fact, how did they get my number? I've never sent anything by Fed-Ex, but I'm fairly sure they won't require any information about the addressee other than the address? Now I was really suspicious - time to challenge back....

"So who's this package from?" I asked

"Er - it's a cash, door-to-door delivery, I can't tell you who it's from" he replied. Well, I'll give him a 5.7 on artistic impression for speedy improvisation, I thought, but there was a definite hesitation there, for which the Russian judges would mark him down on technical merit. Besides, if they'll happily accept packages for delivery without any information about the sender, then they certainly wouldn't need to know anything about the recipient

"So can you tell me Mr Alistair Davidson's date of birth?" he returned to what I was increasingly convinced was his script.

"No, I'm sorry, I'm not going to give that information out," I said firmly.

...and he hung up on me. A swift 1471 showed that he'd suppressed the number before he called.

OK, so there's a couple of possibilities here -

He really was from Fed Ex, and they really do have a policy of incredibly bizarre security procedures, ringing their addressees up at 7am, and hanging up on their customers. Call me Mr Idealistic, but I think that's unlikely

He, or an associate, got my name and address from somewhere - maybe from something as simple as a spam letter I threw away, or come to think of it, I'm in the phone book - and decided to try some phishing.

If I was a betting man, I'd put money on the second.

Which makes me wonder -

Does he / do they normally try this on US numbers? We don't really do Fed-Ex here in blighty, any delivery is far more likely to be via Parcel Force or even UPS.

Do they (I'll assume for now that there's more than one of these scammers) deliberately call early in the morning, to get people while they're fuzzy and bleary and not thinking straight?

How many other people have they managed to get identity details from like this?

I'm a suspicious sod, and paranoid about my personal information - it's a known side effect of having worked in "teh intarweb" for too long :-) - but I know plenty of people - like my mum - who will be incredibly dubious about giving their credit card details to an online retailer, but will quite happily answer any question asked by any random punter who rings them up and says "hi, this is (say) HSBC, can we check your account number and sort-code please?"

It's a sad fact that the weakest link in any security system is the people involved, and until there's a fundamental shift in human nature, that's unlikely to change any time soon.

In Defence Of Hungarian Notation

2007-01-15T19:15:00.000Z

I started typing a comment on Pete Bell's post Why Not Hungarian, but it got a bit too long so I've put my two-penn'orth here.

Hungarian notation, for me, can be very useful, but maybe not in the form that's most commonly understood. I've heard the argument that the original intent behind Hungarian notation was to declare what KIND OF THING a variable is - but this got mistranslated into what TYPE a variable is, and the two are not the same -

For instance, in the last big CF project I did, I took to declaring strings that contained markup with a "htm" prefix, and strings that may have character codes - such as from textarea input - with a "raw" prefix. Similarly URL-encoded strings get an "enc" prefix. They're all strings, and traditional Hungarian notation might give them all the same prefix - but if you do it this way, as soon as you started typing a line like:

<cfset htmContentForDisplay = rawInput />

you immediately know that something is wrong. The variable prefixes themselves make you think "Hmm - that's not going to work.... I need to do some intermediate processing to convert the formats there"

In a strongly-typed, pre-compiled language like Java or C++, then Hungarian notation can just get in the way - you've got the compiler as a type-safety net, and anyone who's ever tried to do anything meaningful with MFC will certainly testify to the horrors of things like "lpszhwndmyWindowHandle". But I think that in loosely-typed languages such as CF, Hungarian notation has it's place if you use it properly - remember, "a Hungarian is the only man who can follow you into a revolving door and come out first" :-)

(NSFW) Taking "User Generated Content" a bit too far

2006-12-20T15:44:00.000Z

The Register is reporting on a heroic altruist's solution to a sticky situation regarding copyright-infringing contributions to Wikipedia :

Wikipedia semen shortage filled by User Generated Content

....well, it's certainly very public-spirited of the honourable member.

Going loco with Mo'MoSoSo

2006-12-13T13:53:00.000Z

Nice to see that m'former'colleague Rik Abel is valiantly continuing his one-man crusade to get the gloriously-silly acronym MoSoSo more widely adopted, whilst simultaneously pointing out its majestic silliness in an ironic, post-modernist, self-referential way. Or something....

Don't get me wrong, I think Mobile Social Software could well be the killer app of Web 2-point-whatever-we're-up-to-now, but seriously folks - MoSoSo? That's Well Jackson!^*

Make sure you don't target your MoSoSo applications at the Small office / Home Office market, 'cause that would be SoHoMoSoSo.

And if your users wore black eyeshadow and jaggedy-cut hairstyles, would that make it SoHoEmoMoSoSo ?

And if it was in a Bohemian New Romantic style...? (BohoRomoSoHoEmoMoSoSo)

Yikes, this one could run...

^* - look here if you don't get the reference

Trusting The Magic Pixies - Hibernate HQL

2006-12-12T10:05:00.000Z

<rant>
I'm having trouble trusting the Magic Pixies. I admit, it's a common theme of mine, and maybe it's just NIH syndrome-by-proxy, but Magic Pixies - able and willing though they are - can only do your job for you if you ask them in just the right way. And sometimes they seem like they're being deliberately dumb and obstinate little buggers that just won't do as they're told, dammit!

(ahem) Perhaps I should elaborate here...

The Magic Pixies in question here are the ones that do the "hard" work for you in Hibernate, the near-as-dammit de facto standard ORM system for Java. Or Persistence Management. Or whatever term you prefer. Pete Bell has been blogging in great detail about the process of writing his own cf-based ORM framework, so if you're not familiar with ORM systems then you're probably best going to check out his blog, because this is a pure and simple straightforward bit of spleen venting.

The basic idea of Hibernate is that you can write your object model just as you choose, and then map your objects to persistent entities using (surprise surprise) an XML config file - or, if you're using Java 5 and EJB 2 and Hibernate 3, then you can dispense with the XML config file (yay!) and use annotations instead (double yay!) So long as you construct your mappings correctly, then you "don't need to worry about" the SQL - the Magic Pixies of Hibernate will auto-generate your db schema and SQL queries, and magically do your CRUD for you with a sprinkling of their Magic Pixie Dust™.

"But what if I want to do something a bit more complex than basic CRUD?" I cried.

"Like what?" said the Magic Pixies

"Like a left outer join?" I replied

"Oh, you don't need to worry about all that nasty SQL" said the Magic Pixies, " because that would tie your code to your database system, and that's a BAD THING! BAD Al! BAAAAAD!"

"Gosh, sorry Magic Pixies," I said, rather sheepishly, "I promise to use that nice abstracted Criteria API that you so generously provided in future"

"And so you should!" said the Magic Pixies, "Remember, you do the code, we do the data, otherwise we'll have harsh words with our union rep, OK?"

"Ok! Ok!" I said. "Now could you stop hitting me with that rolled-up newspaper please?"

"So long as you promise to be good"

"I do! I do!"

"Ok then"

"But what if I want to do something more complex than that?" I asked.

The Magic Pixies looked a bit puzzled.

"What on Earth could you want to do that's more complex than that?" they replied.

"Well, what if I wanted to find a set of objects of type X that didn't have any corresponding objects of type Y that match certain criteria?"

"Pfffft!" said the Magic Pixies, " that's easy, you just create Criteria along the association paths!"

"Er, huh?" I said.

"You just create a Criteria object for class X, and then create another Criteria object using that Criteria object by passing the name of the property of class X which refers to the encapsulated class Y contained within class X!"

"Huh?" I said.

"Or if you really want to, you can use HQL"

"HQL?"

"Yes, Hibernate Query Language. It's almost like SQL, but not quite. Because we wouldn't want you using SQL - SQL's tied to database platforms, and that's BAAAAAD"

"So how do I do it HQL?"

"You create a query that queries along the association paths and properties of the objects"

"Oh, right, OK" I said. "But what if class X doesn't have class Y as a property?"

"Er..... huh?" said the Magic Pixies.

"Class X has no property that refers to class Y"

"Well then, you won't need to query for it, will you?" said the Magic Pixies, a touch too smugly for my liking.

"But I do!" I insisted.

"Er.... huh?" said the Magic Pixies.

"Well, say if I had a table of ItemLinks...." I began.

"A WHAT of ItemLinks????"

"Sorrysorrysorry! I mean an ItemLink object..."

"That's better!"

"...that represented a weighted link between two Items, such as might be calculated by some very complicated fuzzy logic and Natural Language Processing"

"OK"

"...and a separate LinkPreference object that represented an preference expressed by a Person as to whether their ItemLink to a particular object would be public or not"

"Erm... can you give us an example?"

"Sure - this clever NLP stuff might detect that Bob from SysAdmin has been talking a lot about clustering database servers, and he might want to share that link so that he is known as an expert in that field."

"OK, with you so far..."

"But it might detect that the boss has been talking with his secretary about a dirty weekend in Brighton, and they really wouldn't want that shared at all, would they?"

"Erm, isn't that an outmoded stereotype that just reinforces age-old gender-typecast notions of sycophantic star-crossed secretaries as prey for the equally-stereotypical notion of amoral boss-as-alpha-male-predator?"

"Alright, alright, but you get the idea!" (that Magic Pixie was really starting to get on my titty ends)

"Yes, I follow you"

"So what if I want to query for all ItemLinks that have been created in the last, say, two weeks, and that don't have a corresponding LinkPreference?"

"Well, you could query for ItemLinks that have LinkPreference set to null"

At this point I was starting to snort quite heavily.

"But I told you, ItemLink doesn't have a property that refers to LinkPreference! The two are completely independent!"

"Well then you shouldn't want to query for them" said the Magic Pixies

"But I DO!"

"Well, can't you follow the association path up from ItemLink to Item and then down to LinkPreference?"

"Well, yes, I could, but wouldn't that result in the Items table being read in the query when there's absolutely no need for it?"

The Magic Pixies looked down at their feet

"...might do..."

"And isn't that horribly inefficient?"

They started fiddling with their shorts

"...might be..."

"And it's not that simple anyway, because it's a compond join on TWO properties!"

"...yes..."

"WHAT was that?"

"yes!" said the Magic Pixies, with bottom lip sticking out.

"So can you perform this raw SQL query in your own way?"

SELECT item_links.* 
FROM    item_links
LEFT OUTER JOIN link_prefs 
             ON item_links.item_id = link_prefs.owner_item_id 
            AND item_links.other_item_id = link_prefs.linked_item_Id
WHERE
 item_links.created_at < ?
AND link_prefs.shared IS NULL

"...might do, if you ask us nicely..."

(sigh) "OK, can you pleeeeeeease do it?"

They conferred for a moment in hushed whispers, and then turned back with a very smug-looking smile, and said

"Yes, we can - but we're not going to"

"WHAT?"

"You have to ask us in the right way"

Steam was starting to emerge from my ears

"And what IS the right way to ask you?"

They grinned even wider

"We're not going to tell you!"

And I stormed out of the room.

You see, the trouble I have with ORM systems is that they're all well and good as far as they go, and yes they can save large amounts of "donkey work" But sooner or later you nearly always come up against something that would be almost trivially easy to do with raw SQL, but the nice insulated ORM abstraction just can't deal with. I know that I'm probably looking at this from the "wrong" direction, I'm thinking about the data rather than the objects, but until the Magic Pixies start to play a bit more nicely, I'm always going to be a bit suspicious of them.
</rant>

(deep breaths..... calm.... happy thoughts..... nearly Christmas....)

Friday Brainf**k : How Unique Is A Phrase?

2006-12-08T10:58:00.000Z

Eep - posts have been a bit thin on the ground recently, due to big crunch time on SONAR, but here's an interesting question that's just cropped up, and I'm not sure of the answer to, and blogging the question might just help me get my own thoughts on it straight....

First, some context. (Yeah, yeah, skip the context and go to the question) I'm writing the DAO's for the system as a layer of abstract interfaces, with a default implementation based on Hibernate, and using the everything-is-an-item pattern that I've blogged about before.

Aside: This pattern, in conjunction with Java Generics - a Java 5 mechanism that's kind of like C++'s Templating system, but without the horrors that infest the STL Standard Template Library - has lead to a really nice way of getting lots of basic operations (e.g. CRUD) "for free", which deserves a post of its own and I'll blog about it next time I get chance.

In this design, for every type of item, there's a corresponding DAO. In the DAO, there's a save() method which either adds the passed object to the DB, or updates it if it already exists.

This save() method is also responsible for throwing an exception if the given object can't be saved, as it would break the application's business rules for uniqueness. So each implementation of the save() method calls an isDuplicate() method, which is defined by default on the abstract ItemDAO, and can be overridden as appropriate on the subclass DAOs. For instance, it's not acceptable to have two Email records with the same messageID, but it's perfectly fine to have two Person records called John Smith - and this is where the interesting question arises...

In our model, a Theme is also an Item. A Theme, in this case, being a word or phrase that has been extracted from the content of an Item due to it being "potentially interesting". There's a whole load of extreme Lisp cleverness being worked on by M'Colleague Craig McMillan, he of the prodigious beard and piratical proclivities, regarding how you determine a word or group of words is potentially interesting, but in the words of Frank Drebin, that's not important right now.

The question is, what makes a Theme unique? On a superficial level, you can say that a Theme is a group of characters that make up words, so that group of characters must be unique. In other words, no two Theme records must exist with the same String in the "Title" field. But if you think about it a bit more, that might not actually be the case.

For instance, the problems of homonyms (a word that has the same pronunciation and spelling as another word, but a different meaning - e.g. "bat" the animal and "bat" in cricket) and polysemy (capacity of a word or phrase with multiple, related meanings that derive from the same etymology - e.g. "bank on it" with bank meaning to rely upon something, which derives from the reputation of bank-the-financial-instution for reliability) are perennial problems for Natural Language Processing. Should we take that into account in the model? Can we?

In the above cases, I can probably take the reasonable position that because the words are spelled the same, I can assume that they are actually related in this sense, and if two emails refer frequently to "bush", they should be pointing to the same DB record for the "bush" theme, regardless of whether they were talking about a US President, a shrub, or the Australian outback - and homonymy and polysemy can just be swept under the carpet. (Hmm, starting to get a lot of bulges under this carpet in the office here...where's my hammer?)

However, when you start to think about the possibility for the system to be dealing with multiple locales and even multiple languages, which may well be the case in large multi-national corporates, another, related, linguistic term starts to rear it's ugly head - the Heterologue.

A Heterologue is a word that occurs in multiple languages, possibly with completely different meanings. For instance, the syllable "bat" in Cantonese means the number eight (at least, when pronounced with a high inflection), and having seen the way the lovely Lisa flits between English and Cantonese sometimes several times a sentence, even in emails where she types out the Cantonese word phonetically with English letters, this may well occur.

It's not just languages that show heterologues either - the same word or phrase in the same language in a different locale can have completely different meanings - the web is strewn with plenty of examples of British / American English ambiguities, the classic one being at Rocom when my newly-immigrated American colleague Stuart asked what Martin and I were doing for lunch, and I replied that we were driving into town so that Martin could pick up some fags, and did he want to come too?

So it boils down to this - should I include the locale of a Theme in the check for uniqueness along with the title (i.e. the actual string) or not? Is a given string of characters unique just within a particular locale, or globally? I'm tempted to say globally, but I have a nagging feeling that at some point in the not too distant future, that choice may turn round and bite me.

Integrating Applets With AJAX

2006-11-20T16:59:00.000Z

For our new SONAR product, as demo'ed in the Enron Explorer, we needed a way of visualising the social network of up to 80,000 people. Our first thought was of course, Flash, but we found that it just wasn't up to the job^* of visualising large numbers of nodes. If only there was a way of combining the scalability of a Java applet with a slick, whizz-bang AJAX interface... well, as luck would have it, there is!

There are certain problems you have to get round, particularly related to the issue of the applet showing through anything that you place over it, and the browser reloading the VM if you hide the applet in any way , but these are fairly straightforward to get round once you've figured it out (hint: try moving it off screen!) - m'colleague Jan Berkel blogs about the technique in more detail here.

It would be overstepping the mark to suggest that we invented the technique as speculative articles have been written on this subject before, but the feedback we've been getting from the Enron Explorer shows that a lot of people are taken most of all by the interface, and as far as we know we're the first to use it in a production application.

All we need now for the meme to take hold, of course, is a snappy acronym....
- APAX?
- APJAX?
- JAPAX?

Hmmm..... there HAS to be an amusing acronym we can tease out of this - all suggestions gratefully received!

* OK, I'm fairly sure that, given enough time, we could probably have found a way of getting round the scalability issues associated with doing it in Flash, but as Jan mentions in the article, we had a large amount of pre-written Java code that it would have been a shame to waste, and what you also get with Java is a vast library of free code and APIs out there, plus complete flexibility over how you use them.

10 Things to Check for Supporting International Characters In Your Web App

2006-11-13T11:46:00.000Z

These days, more and more of our web apps have to be ready and able to support international characters. It's a non-trivial problem, and one that causes many furrowed brows, because usually by the time you notice it, you've already screwed up some data. I've dealt with it many times, and found that it's generally much easier to prepare for the problem before it occurs, rather than hack a solution together after the fact. This is not a post about i18n-ing your display templates, that's a whole topic in itself, even though some standard mechanisms are pretty well-defined by now. It's about issues involved in storing and displaying content with non-English characters.

A full discussion of character encodings, and the headaches thereof, is WAAAY beyond the scope of this post. It could fill a pretty weighty multi-volume book all by itself. So I'll just refer you to Joel Spolsky's article on the topic - The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) - and say that the character set that's most commonly used for international characters is UTF-8.

So here's a quickie list of things to check for and bear in mind:

String Processing Issues

Use Unicode strings
Internally, CFMX is entirely based on Java, and so it "should" use Unicode strings by default. The main things that you need to to worry about are when data goes in - into the database - and when it comes out - gets presented to the user. Of these two, the most important is the first - so long as data is being stored correctly, you'll always be able to get it out again. If it's being stored incorrectly, you're a bit screwed :)

If you're using any Regular Expressions for processing or validating strings, BE CAREFUL!
It's very common to use expressions such as [a-zA-Z] to check for letters, or [a-zA-Z0-9] to check for alphanumerics. What do you think is going to happen if you pass an accented character such as à or é through this reg ex? Yup - é is NOT between a and z, so it will not match. How best to handle this is up to you - the POSIX regex elements such as [[:alpha:]] are good enough for some situations, but not others. For instance, POSIX does not allow more than 20 characters to be categorized as digits, whereas there are many more than 20 digit characters in Unicode. There's more detail on unicode.org

Data Storage

Store everything in UTF-8
Even if you're designing an app that "won't ever need anything except English", a little thought and effort at the design stage will mean that next time you have to write a similar app that does need international characters, you can re-use the code. Besides, it's just good practise, and the extra storage overhead of storing up-to-4-bytes-per-character is relatively easy to deal with in these days of 750GB disks for £250.

Check the collation on your databases, tables, and columns
Collations can be set at all levels of specificity, from the server right down to individual columns. Make sure that they're ALL UTF-8. Easiest way to do this is to generate a CREATE... script for your database (SQL Server) or a mysqldump file (MySQL), open it up in a text editor and search for COLLATE

In SQL Server, make sure that any character-based field that can be populated from user-entered data in any way, is specified as a Unicode field
i.e. the type starts with an N. varchar fields should be nvarchar. text fields should be ntext.

Plan ahead!
Plan to cope with multiple character sets up front, and you know your database can handle just about anything you're likely to throw at it. Sweeping it under the carpet and saying "we'll worry about that when it happens" is likely to end up with you taking your app offline while you rebuild tables and text indices. This can easily take several hours, for anything above a couple of thousand rows. Long downtimes make unhappy users.

Presentation

Explicitly declare the UTF-8 character set on every page
Make sure that every page has a meta tag like this:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">

...and also make sure that it's the FIRST tag in the Head of your doc, because as soon as a browser encounters a charset declaration, it starts re-parsing from the top.

You "should" specify the primary language in the HTML tag
In ordinary HTML, this is just :
<html lang="en-GB">

In XHTML 1.0, it's slightly different :
<html lang="en-GB" xml:lang="en-GB" xml‍ns="http://www.w3.org/1999/xhtml">

And in XHTML 1.1, you don't need the lang attribute:
<html xml:lang="en-GB" xml‍ns="http://www.w3.org/1999/xhtml">

This makes it much easier for text readers, translation software, and - crucially - search engines to recognise the language and take appropriate action. More details at W3.org

For multi-language markup, you can provide a comma-separated list of languages
You can (and should) also specify the language on any element within the page that is in a different language to the primary language. If your document structure doesn't break down to a logical tag that encompasses the different language part, then use a span tag:
<span xml:lang="fr-CA" >....</span>

Language is a CSS pseudo-class
This means that you can specify different styling for different languages, like so:
/* smaller font for documents in German */
HTML:lang(de){ font-size:90%; }
/* italicise any bits of French in any document */
:lang(fr){ font-style : italic; }
/* change the quotation marks for any Q tag INSIDE a French element */
:lang(fr) > Q { quotes: '« ' ' »' }

More details at W3.org

There are many many more things to think about, and this list is by no means exhaustive, but it should be enough to give you a starting point. Lots of the issues that are thrown up by this tend to be general "take-a-step-back" kind of issues that make you question your workflows and assumptions, rather than your programming expertise - e.g. if we're letting people enter their own nickname, and then using that nickname as part of the url, then what's going to happen if someone enters a nickname entirely in Russian? How are we going to handle generated emails to them? What about if they're text-only emails? (Hint: Content-type: text/plain; charset=utf-8 !)

But there are also deeper issues involved - if we start accepting and labelling content in different languages, what facilities do we need to provide to our users in order to filter out - or focus exclusively on - particular languages and character sets? Do we need to create a separate site for each language? Or do we accomodate all the content in one site, with filters?

Answers to those kinds of questions, I leave up to you :)

Alas, poor Smartgroups, I knew it well....

2006-11-07T12:19:00.000Z

It's not without a tinge of sadness and wistful sigh that I noticed that, in possibly the least-surprising announcement I've seen for a long time, Orange is to finally pull the plug on Smartgroups

Smartgroups was, in its day, the Daddy of first-generation community applications, and I worked on it for just over two years just after it got bought out by Freeserve, who became wholly-owned by Wanadoo, then France Telecom, then finally Orange UK.

(Remember those "Ten signs you are in a dotcom company" emails that went round about the turn of the millenium? One of them particularly stuck in my mind - "You've sat at the same desk for two years, and worked for four different companies")

I learnt a lot about managing large-scale systems from that job - SG handled more than 50 million emails per month - and a good percentage of the stories I'll come out with after several late-night-beers-with-other-techies hail from that time. I also learned a lot about human nature, and the myriad ways in which people will never fail to suprise you, even when you think you've seen it all. And I'm not just talking about the users there...

But ultimately, it was a failure to evolve and keep up with the competition (e.g. MySpace) that gradually put the nails in its coffin, and after about four years of knowing full well that its days were numbered, SG has finally been taken out the back and given a nice sunny wall to stand against, and a roll-up to smoke. And a blindfold.

Farewell Smutgropes - you will live on in the memories of all those who worked on you. (Despite some quite determined scrubbing with Mind Bleach in some cases) And if anyone manages to track down the new homes of the Stereo Stimming group, the Brent Spiner Data Lovers group, or my own personal favourite - Hairy Bearded Scotsmen In Kilts ("ONLY pictures of hairy bearded scotsmen in kilts will be accepted - any pictures of hairy bearded men in kilts who are NOT Scottish will be deleted...") then be sure to let me know. Such gems of the longest of long tails are surely too fine to be lost forever :)

Mutually Exclusive Interfaces

2006-10-27T10:39:00.000+01:00

Jason Nussbaum has posed a nice juicy thought experiment - how do you handle mutually exclusive interfaces? I started typing a comment, but it got too long, so here's my take on it :

The question was:

Had a thought: how do people handle mutually exclusive interfaces?

I suppose there's no such thing as mutually exclusive interfaces from a code perspective, but from a logic perspective, there may be. For example, let's say you do one of two things based on whether a class implements one of two interfaces. What do you do if something implements both?

It's a conceptual problem - as Jason says, there's not really a technical way that interfaces could be mutually exclusive. You can think of it this way - if an object X implements interface Y, then what you're really saying is "X can act like a Y". So there's no reason why X can't also act like a Z, depending on the situation.

There are so many ways you could tackle this problem. You could introduce a super-interface:

public interface DeadOrAlive{ public boolean isAlive(); }

public interface Dead extends DeadOrAlive{
public void startToSmell();
}
public interface Alive extends DeadOrAlive{
public void breathe();
}

...and thus force implementers of Dead or Alive to provide a method which indicates if they are alive or not. This feels like a bit of a kludge, but to my mind, less so than without.

You could add Exceptions onto the signature methods of Dead and Alive, which might throw NotDeadException and NotAliveException respectively - but this actually feels more kludgey to me than the first case.

I think the "proper" way to think of this is maybe the biggest kludge of all - as your object model is meant to reflect the real-world problem that you're modelling, if it produces this kind of conflict then maybe you need to re-think your model!

- unless, of course, your name happens to be Schrödinger...

public interface Alive{ public void breathe(); }
public interface Dead{ public void startToSmell(); }

public class Cat implements Dead, Alive
{
public void startToSmell();
public void breathe();
}

public class Box
{
private Cat cat;

public void putCatInBox();
}

...eep, lame quantum physics jokes on a Friday?! I think it's time for coffee...

Enron Explorer Going Viral

2006-10-25T12:12:00.000+01:00

Since we got Boing Boing'ed yesterday, Google Analytics shows that we've had over 21,000 page views so far - and that's just to the front page. Unfortunately the GA javascript can't be included in the main app, as it's pretty much all AJAX requests, so we'll have to wait for analysis of the Apache logs for full statistics.

I also put the Enron Explorer on del.icio.us 4 days ago - it's now been echoed by 112 people, including one who speaks Russian, by the looks of it. Google Analytics confirms that we've had visitors from 71 different countries, insterestingly enough including two from the Cayman Islands and one from an anonymising proxy - Kenny Boy, that's not you is it....?

Some of the comments are fantastic :

fpaulus
"Interesting (both technologically and economically) walk through the Enron e-mail archives"

markwithasee
"searchable database of all of enron's internal email from 99-02. WOW."

RStacy
"A look into what the future could look like in corporate transparency and analysis."

slightlyfleury
"Enron emails. Gawd, did I just come?"

some of the tags that people have applied to it (under the posting history on the right) are also interesting - I always find it intriguing to see how other people see what we've done.

We got Boing Boing'ed!

2006-10-24T18:21:00.000+01:00

Another way to hammer a webserver - the Enron Explorer got highlighted on Boing Boing, "The Most Popular Blog In The World" according to Technorati. And our traffic has gone through the roof, but the app is holding up thanks to judicious planning for heavy usage, and some hefty Squid proxying.

Remember kids, "Proper Planning Prevents Piss-Poor Performance"

....and Corny Cliches Create Copious Cringes...
and Aggravated Alliteration Attains Acute Annoyance.
etc.

The Enron Explorer!

2006-10-18T09:42:00.000+01:00

Phew - at long last we've gone live, and I can finally blog about what we've been working on here!

( yeah, yeah, skip all that, what about Enron? )

Our next major product is aimed squarely at the enterprise market, and it's a tool to analyse and extract meaning from corporate data stores and email traffic. Based on this information extraction, we can then map social networks and analyse information flow throughout the organisation, and use this information to give a shot in the arm to communications effectiveness - forwarding emails to you that are especially relevant to you but that you might otherwise have missed, and letting you set "volume level" for each of your interests.

There's no end of applications for this kind of tech - everything from expertise analysis to Sarbannes-Oxley compliance. The hardest question of all, though, was what to call it. After a tortuous voting and lobbying process second only to the IOC in labyrinthine complexity, we finally settled on the name SONAR, as it kind of implies what the product does (scans things and identifies things that you need to know about) and it's also a merely-mildly-icky acronym - SOcial Networks And Relevance.

Anyway, while we were working on the early stages of SONAR, we needed a large set of plausible test data to test the NLP algorithms - sadly, my usual plethora of furry animals and pitiful puns just wasn't enough in this case - and Jan came up with a stroke of genius - the Enron email archive.

In Oct. 2003, the FERC released ~200,000 Enron emails into the public domain, as part of the inquiry into the Enron fraud. So we grabbed the database, imported it into a very early version of SONAR, and set it chugging away. The results were so absorbing that we can't stop fiddling with it! We decided it was a great way to demo our system, even though it's a very early version, so laydeeeez an' gennulmen, (drum roll) allow me to present....

THE ENRON EXPLORER!

Just click on a name or a theme to get in, and off you go. There's some real gems in the archive, ranging from the hilariously obscene :

Jeff Skilling to Andy Zipper:
Fuck you, you piece of shit. I can't wait to see you go down with the ship like all the other vermin. Smug, paranoid, unhappy mother fucker. Eat shit.
...to which I think Andy Zipper responded with admirable restraint!

- to the heartbreaking:

Sato (Enron Japan) to Andy Fastow:
Please don't fire Enron Japan staff, we do nothing wrong! Please!!! [....] We know how serious is the situation but please don't fire us now. Our families are waiting for a happy chrismas and new year!!

We can (and often do) lose ourselves in this for ages - have an explore and enjoy finding out what they were saying to each other as the house of cards crumbled down around them.

On a technical note, the interface is written in RubyOnRails, with a Java backend. It's AJAX'ed up to the eyeballs - hey, we even have rounded corners and gradient fills on the AJAX loading indicators! - and although there are still a couple of issues to sort out, we've worked hard on maintaining the usual expectations of browser behaviour.

The back button works (with a couple of minor niggles), and you can bookmark and email the url of your current view, and it should (nearly!) always bring you back to the same point. There's some smooth integration between the Java applet visualiser and the AJAX calls too, although again there's still a couple of niggles to deal with.

There's a whole load of other niftiness going on behind the scenes, but I'll blog about that later. In the meantime, have fun with it, and if you find anything particularly juicy, leave a comment (either here or on the app itself) and share it with the world!

Styled Checkboxes and Radio Buttons

2006-10-11T11:26:00.000+01:00

I'm sure you've experienced the problem - most form elements can be styled pretty easily, but checkboxes and radio buttons? Forget it. If you're anything like me, you probably gave up by now and accepted that it's just one of those things you have to put up with. However, Philip Howard has released a nice CSS and JS solution that allows you to wrap a span of a certain class around the elements you want styled, and the magic pixies will do the rest.

I've seen this done before, but this solution is worth highlighting because :

It degrades gracefully back to the standard form elements if your browser does not support JS, CSS or images, and
He's put the extra bit of effort in to support the standard keyboard controls for form elements - space bar toggles status, left and right move the focus along a group of radio buttons, etc.

Nice one Philip.

Breaking Dependencies With Interfaces

2006-10-04T10:36:00.000+01:00

I just picked up on Mike Dinowitz's post Interfaces - Why Bother? asking for benefits that interfaces offer. I started typing a comment, but realised that my two-penn'orth was way too sprawling for a comment and needed a whole post of its own. So here goes - I'm sure this won't be telling someone as experienced as Mike anything he doesn't know, but there's also been a call for more introductory-level blog posts in the last couple of days, so hopefully someone out there will find this useful as a concrete example of where I'd be lost without interfaces.

I'm currently working on a large J2EE platform with over 2000 classes. The only sensible way to manage such complexity is to split it into several distinct modules, each of which is conceptually self-contained and compiled and unit-tested separately, in a strict order. The ANT build script builds the core framework module first, then the email server, then the groups and user modules, etc etc, finally finishing up with the Tapestry-based web interface.

This all works great, except that you're still left with cross-module dependencies. The email server needs to know that group emails should be forwarded to group members, the groups module needs to know about its users, and so on.

The way these dependencies are resolved, is through interfaces. In the email module, we create an interface that represents the bit of group- or member- related functionality that the email server needs to know about - in this case, the ability to accept and propagate an email - and make the group and member objects (which come after the email server) implement that interface.

So we make an interface called EmailPropagator with one method - propagate() - in the email server module, and make groups and members implement EmailPropagator.

This way, the email server doesn't need to know anything else about the "thing" it's sending email to - so long as it implements EmailPropagator, the email server can ask it to propagate email to whoever or wherever it feels like, and that's all that the email server needs to know about it in order to do its job.

We can make anything be accepted by the email server, so long as it implements that one method. It could be a group / mailing list, an individual person, a spam demon that forwards ten thousand copies to random addresses, a black hole that just swallows it up, a file store.... anything, so long as it implements that one method.

This is part of the power of OO, and particularly the power of interfaces. It's also different from inheritance - in Java, and hence CF too, a class can only extend one base class. However, it can implement as many different interfaces as you like. You've heard of "If it looks like a duck and quacks like a duck, it must be a duck...." - well, this extends to "If it quacks like a duck, I don't care what it is, so long as I can ask it to quack".

It's one of the guiding mantras of OO that you should "design to interfaces, not implementations" and this is exactly why - we can make a million different things plug into our email server and do a million different actions with an email, without ever needing to change the email server code. That's power, and that's reusability. And that's why I love interfaces.

How Enterprise 2.0 Products Can Succeed

2006-10-04T10:12:00.000+01:00

Much food for thought on Andrew McAfee's blog discussing John Gourville's concept of the "9x Email" problem which new technologies must overcome - "a mismatch of 9 to 1 between what innovators think consumers want and what consumers actually want."

People - real people, not tech people, who sometimes seem to be rabidly adopting the newest, most obscure technology for no other reason than to claim "geekier than thou" bragging rights amongst their peers - have an inbuilt tendency to stick with what they know. Gourville suggests that in order for new technology to go viral, it must offer a tenfold improvement over what's already out there. The problem is that in order to overcome the inertia associated with the status quo, en evolutionary, not revolutionary approach is called for:

Gourville's research suggests that the average person will underweight the prospective benefits of a replacement technology for it by about a factor of three, and overweight by the same factor everything they're being asked to give up

McAfee takes the example of email versus "groupware", and cites the intuitive nature of email interfaces as a point of comparison for Enterprise 2.0 apps. Just about everyone "gets" email as a concept, and the critical problem for Enterprise 2.0 technologies is one of interfaces. Rather than adding more and more bells and whistles to an interface, we should concentrate on making the interfaces clean, elegant, and instantly-comprehensible. This is a viewpoint I wholeheartedly share, but McAfee puts it very succinctly:

A great UI not only heightens the perceived benefits of a proposed collaboration technology, it also lowers the perceived costs.
...
The greatest challenge here, I think, doesn't have to do with making the browser sufficiently application-like ... It has to do with making technologists sufficiently user-like -- getting them to stop thinking in terms of bells and whistles and elaborate functionality, and to start thinking instead about busy users with short attention spans who need to get something done, and who can always reach for email

Insightful stuff.

How To Architect Your CSS

2006-09-29T09:38:00.000+01:00

We all have our own favourite strategies for architecting our application code, but CSS is often one of the aspects of a site that gets copmletely overlooked. As CSS - along with its browser support - matures and grows more powerful, how we structure our CSS becomes more and more important for maintainability and expansibility¹ of our applications.

I'm sure we've all experienced it - you start out with a prototype with a few simple CSS rules and the best of intentions, but as your application grows and changes, and more and more people have their input into the design process, the CSS grows and morphs and accumulates more and more quick fixes and cheesey hacks. Months later, you realise you've got to the point where something as simple as "can we make the comments link appear in red?" can take hours of navigating the jungle of interfering specificities and ids. Eventually you admit defeat, and put in a cheesey hack "just this once", and mentally promise to go back and fix it properly later - but somehow you never get round to it, and the process continues.

Digital Web has an interesting, if short, article on Architecting CSS, with some useful tips. If I had to pick two simple things that you should do RIGHT NOW, DAMMIT if you're not already, it would be :

Commenting your rules just as much as your code, and
Alphabetically sorting your attributes.

Oh, and avoiding the !important hack if at all possible.

Alright, three simple things....² You get the idea :)

1 - apologies for grimace-inducing linguistic contortions. It's Friday, it's early, and I haven't had coffee yet. Bleh.
2 - No-one expects the Spanish blog post!

Unfortunate Comedy Typos part 1

2006-09-21T16:57:00.000+01:00

Ever have one of those days where no matter what you do, you just can't type properly? Today, I'm finding it impossible to type StringBuffer - it keeps coming out as StringBugger.

Calling Dr. Freud, calling Dr. Freud.....

A couple of other Freudian ~~slaps~~ (ahem) SLIPS I keep having, mean that I keep declaring functino's (maybe a new fundamental particle for the Standard Model?) and functoni's - and I don't know if it's just me, but Func Toni conjures up a disturbing image of Swiss Toni in a gold lamé shirt with eighteen-inch collars, star-shaped shades, and a big-ass medallion, strutting his funky stuff to Graham Central Station - like this:

Damn, that's a funky bassline!

(sigh)

Thank crunchie it's Friday...

Demos wins award

2006-09-21T11:18:00.000+01:00

Congratulations to m'erstwhile colleagues at Headshift on scooping another award for the recently launched Demos site. I, along with the rest of the crew, put a huge amount of thought and care in to that site, and it gives me a nice warm fuzzy feeling to know that it's not just the client who appreciates it. Special kudos to Neil Roberts for doing the always-tricky task of picking up the reins on it after I left and teasing it lovingly into deployment.

Svn over ssh (svn+ssh://) on Windows via cygwin and PuTTy

2006-09-13T13:42:00.000+01:00

One of the more obvious things lacking from Windows is an ssh (secure shell) client, and this has caused me no end of grief trying to check out code from Subversion repositories on servers that require ssh access - such as our own.

I'd previously managed to get access by forcing my username in the URL of the repository along with a saved PuTTY session name:

svn+ssh://alistair@(saved PuTTY session name)/etc...

...where the saved PuTTY session used my imported private key from the live host.

(Note: you *must* use PuTTYGen to convert your OpenSSH keys to PuTTY-compatible keys)

This caused a certain amount of sneering from the Debian users in the office, who didn't have to jump through any of these hoops, but I could live with that...

The latest problem to have me gesticulating at the screen like a south-american-footballer-in-front-of-a-referee-who's-reaching-for-his-pocket was the use of externals.

An external in a Subversion repository is a URL reference to another resource - for instance, in a Java app that requires a certain library to build, you might add an external pointing to the download url of the particular jar file that you need.

All well and good, but you can also add externals pointing to a different location in the same repository, for instance, to share config files between back end and front end modules.

Now here was the problem - you can't have relative urls in externals, they have to be absolute. This particular external was pointing to a different lcoation in the same repository in the standard URL way, which works fine on UNIX (Debian) clients because they have a native ssh client :

svn+ssh//(host)/(path)

- but there was no way this was going to work on Windows via PuTTY - the only way I'd got svn+ssh:// urls to work previously was via the username@savedsessionname method. Every time I tried to update the working copy, it would get as far as that external, then give me this error message:

svn: Connection closed unexpectedly

So how do you get round this? Enter the cavalry - Cygwin.

Cygwin is a command shell for Windows that gives you a Bash-style environment, including....(drum roll)..... an ssh client!

With this, you can configure SVN (and the rather wonderful TortoiseSVN) to use this ssh client and - crucially - pass it a command line parameter that tells it to use your private key file for authentication.

If you open your SVN config file in a text editor, (you'll find this in C:\Documents and Settings\(username)\Application Data\Subversion\config) you'll see instructions on how to configure ssh. You can pretty much ignore it :-) and just use the following line:

ssh = "C:/(cygwin root path)/bin/ssh.exe" -i "C:/(path to your private key)"

...and that's it! It's one of those really simple solutions that took me a couple of minutes to implement, but hours of desk-headbanging frustration to find. You can also use the same command line in TortoiseSVN -> settings -> network -> ssh client.

One other note: make sure that you point ssh.exe at your openssh-compatible private key. If you've set up your key with PuTTYGen, you can load it in and export an OpenSSH-compatible key from there.

Lessons Learned From Kiko

2006-09-07T10:19:00.000+01:00

Richard White has posted an insightful write-up of his Actual Lessons From Kiko. Well worth a few minutes of anybody who is in the web app space.

RegEx to fully validate RFC822 email addresses

2006-08-30T16:18:00.000+01:00

I love Perl. No, I really do - I can sit and look at the really good examples for hours, and still have no clue what they're doing. And here is a fantastic example....

Mail::RFC822::Address - a module that tells you whether the given string is a valid email address or not.

Most email validators only check the absolute basics - e.g. some characters followed by an at-sign, followed by some more characters and at least one dot. But have you ever actually read RFC822, the critical RFC that defines the standard of what's acceptable in an email and what isn't ? It's surprisingly loose on what's an acceptable address.

(go on, read chapter 6, and try to summarise it - you know you want to...!)

But no more, thanks to Paul Warren, who has solved all our email-address-validation woes with one almighty mother of a reg ex - and here it is. You'll like this. No, you will.... here we go:


(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
)+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:
\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(
?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[
\t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\0
31]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+
(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:
(?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n) ?[
\t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\ r\n)?[
\t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[
\t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n) ?[
\t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t]
)*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[
\t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*
)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t]
)+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)
*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+
|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r \n)?[
\t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?: \r\n)?[
\t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t
]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031
]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](
?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?
:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?
:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)|(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?
:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)? [
\t]))*"(?:(?:\r\n)?[ \t])*)*:(?:(?:\r\n)?[ \t])*(?:(?:(?:[^()<>@,;:\\".\[\]
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|
\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>
@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"
(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t]
)*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?
:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[
\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:[^()<>@,;:\\".\[\] \000-
\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(
?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n)?[ \t])*(?:@(?:[^()<>@,;
:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([
^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\" .\[\]
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\ [\]
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\]
\000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]
|\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?(?:[^()<>@,;:\\".\[\] \0
00-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,
;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|"(?
:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*))*@(?:(?:\r\n)?[ \t])*
(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t])*(?:[
^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]
]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(?:\r\n)?[ \t])*)(?:,\s*(
?:(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(
?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[
\["()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t
])*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?
:\.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|
\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*|(?:
[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".\[]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)*\<(?:(?:\r\n) ?[
\t])*(?:@(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["
()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n) ?[
\t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>
@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*(?:,@(?:(?:\r\n)?[
\t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,
;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:\.(?:(?:\r\n)?[ \t]
)*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*)*:(?:(?:\r\n)?[ \t])*)?
(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\["()<>@,;:\\".
\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])*)(?:\.(?:(?:
\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z|(?=[\[
"()<>@,;:\\".\[\]]))|"(?:[^\"\r\\]|\\.|(?:(?:\r\n)?[ \t]))*"(?:(?:\r\n)?[ \t])
*))*@(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])
+|\Z|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*)(?:.(?:(?:\r\n)?[ \t])*(?:[^()<>@,;:\\".\[\] \000-\031]+(?:(?:(?:\r\n)?[ \t])+|\Z
|(?=[\["()<>@,;:\\".\[\]]))|\[([^\[\]\r\\]|\\.)*\](?:(?:\r\n)?[ \t])*))*\>(?:(
?:\r\n)?[ \t])*))*)?;\s*)

After a while, all you see is ....blonde......brunette.....

PS - hope Paul doesn't mind me posting it here. It's truly a thing of beauty, but I'll take it down if requested.

URGENT html prototype designer/coder needed today and tomorrow!

2006-08-30T12:05:00.000+01:00

Bit of a long shot this one, but Lise has been left high-and-dry by a contract developer taken ill. The job is mocking up flat HTML interfaces, and will probably require a bit of a late night tonight as it must be finished by close-of-play tomorrow for user testing on Friday. Ideally you'd be able to make it to Islington THIS AFTERNOON for a briefing and handover.

Very short notice, but they're a bit stuck - so if anyone finds themselves at a loose end for a couple of days, add a comment on this post and i'll pass your details on.

Flickr's Geotagging Let Down By Location Search

2006-08-29T11:18:00.000+01:00

Thomas Hawk of Geo-tagging mashup Zoomr does a good job of keeping his objectivity in his review of Flickr's new Geotagging facility.

I just had a play with it myself, and I have to agree with Thomas - the nicest aspect is the way that it's integrated with the Flickr Organiser (I still keep spelling that Organizr...) and the by-now-expected Ajax drag-and-drop wizardry. But the most disappointing aspect of it all is the underlying Yahoo! maps data. It's fine if you want to geotag photos down to city block level in most major US cities, but stray off the beaten path to tag photos of mountaineering treks or rock climbing venues - even world-famous rock climbing venues - and the map detail just isn't there.

To be fair, this isn't just a limitation of Yahoo!'s map data, it's similarly limited in Google Maps aswell - and I guess it's a reflection of the underlying business drivers behind the map data. Constructing an index of the whole globe is a massive undertaking, and it has to be funded somehow. The obvious channel is advertising, but only businesses are willing to pay for advertising, and most businesses tend to be centred around urban areas, therefore it's more important to the map provider that the advertisers are kept happy.

It just seems to take on an added dimension of disappointment when this limitation applies to photos. To be honest, one U.S. city block looks pretty much the same as any other U.S. city block, in the grand scheme of things, and part of the whole joy of Flickr comes from discovering fascinating, beautiful images that you may never otherwise have seen. Almost by its very nature, this is going to involve out-of-the-way places like the Sim Gang Glacier in Pakistan, La Dibona of Les Ecrins in the Alps, or K2, which requires 14 days of hard trekking from the nearest road before you even reach the mountain - often cited as the hardest trek in the world, but surely one of the most beautiful. Try to find any of these places in the Flickr map, and you'll have trouble. Even closer to home, in the rugged mountain landscape of North Wales around Tryfan, the map engine draws a blank.

To take the next step in geotagging, and in mapping as a whole, requires the next step to be taken in the search technology that takes a user-supplied string and works out what the hell you actually meant. This is a mammoth task in itself, considering how many different ways people can refer to a particular langitude and latitude, across languages and character sets, let alone local names vs. standard names. But whoever cracks that problem can look forward to a very bright future indeed, and that's just one reason why Natural Language Processing - clearing away the cobwebs of context and language and working with raw chunks of meaning - is becoming such a hot topic right now. Stay tuned...

So many parties, so little time...

2006-08-24T09:56:00.000+01:00

Well, I had the best intentions of making it to tonight's London CFUG, but once again my path is strewn with cowpats from the Devil's own Satanic herd** as it once again conflicts with something else that I just can't miss. This time, it's the Trampoline Systems Summer Bash, graced by the intriguing Czechoslovakian Alternative Folktronica of the most delectable Miss Eva Eden. Should be a larf - I've never had to mike-up a Bontempi organ before.... I'll whack the more-salacious photos onto Flickr tomorrow, plus any videos I happen to get of any embarrassingly drunk bigwigs saying spectacularly inappropriate things.

So have a pint and a whinge for me at the CFUG tonight, and hopefully I'll make it to the next one. Unless I'm halfway up some mountain somewhere or something.

**series and episode, anyone?

Who Wants To Buy An Ajax Calendar App?

2006-08-17T18:01:00.000+01:00

In a move that's not-at-all-designed-to-create-an-internet-buzz, AJAX calendar app Kiko is for sale on eBay.

Starting price US $49,999.99. No Bids yet.

In the interests of completeness, I should point out that they're not the first, by a long way - which market sector do you think got there first? Of course - porn!.

The listing states

We are selling Kiko because we want to have time to work on other projects as a development team.

...and not because they are now in direct competition with Google, or anything...

best of luck Kiko guys

A Pox Upon AppleMail!

2006-08-17T11:38:00.000+01:00

A lot of my time since joining Trampoline 6 weeks ago has been spent reaquainting myself with the black art of parsing and dismembering MIME emails with the JavaMail API. There's much I could say about the MIME format and particularly the JavaMail API, but those apopleptic rants deserve to be written up and nailed to church doors all of their own.

This post is about a problem I've been having with a mail generated by Apple Mail, that has been driving me nuts. It's not the first issue I've had with Apple Mail and it's funky attachment formatting, and I'm sure it won't be the last. However, it's the most maddening to date! Here's the problem:

In a multipart email, you separate each part with a unique string that must not occur in any of the parts. This is generated by the email client, and declared in the Content-Type header. RFC 2045 states that the boundary declaration is required for any multipart subtype. The header should look like this:

Content-Type: multipart/(whatever); boundary="----=(unique string)"

You will then get a set of Parts, each separated by an occurence of the boundary string, and each declaring what type of content it is by means of its own Content-Type header -

Content-Type: multipart/(whatever); boundary="----=(unique string)"

An example might be:


Content-Type: multipart/mixed; boundary="----=ABCDEFGHIJKLMNOP" 

----=ABCDEFGHIJKLMNOP
Content-Type: text/plain; charset=US-ASCII

Hi Al, 

Here's the schematic for the secret base under the island volcano. Note the new layout of the shark pools, and the trapdoor is now triggered from the pressure pad under your desk as requested. Will give the engineers a kick about the frickin' lasers and see what's taking them so long.

Cheers,

Dave

----=ABCDEFGHIJKLMNOP
Content-Type: image/png; name="plans.png"
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="plans.png"

(lots of data encoded into base 64 so that it can be transferred as text)
----=ABCDEFGHIJKLMNOP

All well and good so far.

It gets a bit more complicated when you introduce the fact that any part of the mail body can also be a multipart type, which must declare its own boundary string, but still, it should be parseable into a coherent tree structure, right?

Well yes - as long as you play by the rules.

Apple Mac files consist of two forks:

1) an apple-specific part called the RESOURCE fork which contains arbitrary information such as icon bitmaps and file info parameters,
2) a DATA fork which contains the actual file data.

This translates logically into a MIME multipart format - multipart/appledouble - with one part for each fork.

So, if we were to to send the example message above from a Mac using AppleMail, you would get something like this:


Content-Type: multipart/mixed; boundary="----=TOPLEVELBOUNDARY" 

----=TOPLEVELBOUNDARY
Content-Type: text/plain; charset=US-ASCII

Hi Al, 

Here's the schematic for the secret base under the island volcano. Note the new layout of the shark pools, and the trapdoor is now triggered from the pressure pad under your desk as requested. Will give the engineers a kick about the frickin' lasers and see what's taking them so long.

Cheers,

Dave

----=TOPLEVELBOUNDARY
Content-Type: multipart/appledouble; boundary="----=HEYIMTHEAPPLEDOUBLEBOUNDARY"

----=HEYIMTHEAPPLEDOUBLEBOUNDARY
Content-Type: application/applefile; name=plans.png
Content-Disposition: inline; filename="plans.png"

(apple-specific file information encoded into base 64)

----=HEYIMTHEAPPLEDOUBLEBOUNDARY
Content-Type: image/png; name=plans.png
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="plans.png"

(actual file data encoded into base 64)
----=HEYIMTHEAPPLEDOUBLEBOUNDARY

----=TOPLEVELBOUNDARY

Again, this is all well and good so far - apart from one or two minor irritations like that lack of quotes around the name of the file in the Content-Type header - which can cause some grief if the filename has spaces in it.... but that can be got round without much trouble using a bit of regex in pre-processing.

The problem comes when you have multiple appledouble-encoded attachments. What you would expect is something like this:


Content-Type: multipart/mixed; boundary="----=TOPLEVELBOUNDARY" 

----=TOPLEVELBOUNDARY
Content-Type: text/plain; charset=US-ASCII

blah - message text

ATTACHMENT 1:

----=TOPLEVELBOUNDARY
Content-Type: multipart/appledouble; boundary="----=HEYIMTHEAPPLEDOUBLEBOUNDARY"

----=HEYIMTHEAPPLEDOUBLEBOUNDARY
Content-Type: application/applefile; name=plans.png
Content-Disposition: inline; filename="plans.png"

(apple-specific file information encoded into base 64)

----=HEYIMTHEAPPLEDOUBLEBOUNDARY
Content-Type: image/png; name=plans.png
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="plans.png"

(actual file data encoded into base 64)
----=HEYIMTHEAPPLEDOUBLEBOUNDARY

ATTACHMENT 2:

----=TOPLEVELBOUNDARY
Content-Type: multipart/appledouble; boundary="----=DIFFERENTAPPLEDOUBLEBOUNDARY"

----=DIFFERENTAPPLEDOUBLEBOUNDARY
Content-Type: application/applefile; name=plans.png
Content-Disposition: inline; filename="plans.png"

(apple-specific file information encoded into base 64)

----=DIFFERENTAPPLEDOUBLEBOUNDARY
Content-Type: image/png; name=plans.png
Content-Transfer-Encoding: base64
Content-Disposition: inline; filename="plans.png"

(actual file data encoded into base 64)
----=DIFFERENTAPPLEDOUBLEBOUNDARY

----=TOPLEVELBOUNDARY

But what's actually happening is that in the second attachment, the all-important Content-Type: declaration -

Content-Type: multipart/appledouble; boundary="----=DIFFERENTAPPLEDOUBLEBOUNDARY"

- is missing!

This line is absolutely vital, as it not only declares that this part is in appledouble format, but more fundamentally it declares that this part is itself a multipart and is split with THIS boundary marker rather than any other.

If this line is missing, then ONLY the FIRST attachment gets recognised. Any subsequent attachments which don't get the content-type header are then considered to be text/plain by default, so you get an email which has the first attachment properly parsed as an image, but everything after that appears inline as text. So anyone reading the email gets a big long string of base64 encoded image data. Not nice.

Slovenia - The Game!

2006-08-16T14:28:00.000+01:00

At last, the Flash app you've all been waiting for - Slovenia, The Game! It's a very cute, virtual community based around various beauty spots in Slovenia. We went there last summer for a week, and completely fell in love with the place. It's just stupidly beautiful - everywhere you look, your jaw drops, and this quirky little game just brought it all back.... I'm sitting here with a silly grin on my face and a wistful look in my eye. Ah, memories...

by the way - you seem to get called "pedr" a lot - pedr loosely translates as "fag"

How to get a job in Silicon Valley

2006-08-14T09:58:00.000+01:00

Good-old Guy Kawasaki has an endearingly cynical but still-so-true guide to How To Get Hired In Silicon Valley. Well worth a read, and just as applicable outside California. Not that I've ever applied for a job in Silicon Valley (yet), but it certainly brought a wry smile to my face.

I've read many CVs and conducted many interviews, and it's amazing just how easy some people make it for you to put their CV on the "no" pile. I know we're now living in the txt-spk generation, but seriously - if you can't spell or punctuate on your CV when you're promoting yourself, how well are you going to promote the company? Interviewers are busy people, and will be looking for any reason to say no - spell well, communicate well, keep it short, and if you get an interview, read and learn from Mr Kawasaki.

What do atoms look like?

2006-08-11T11:12:00.000+01:00

....they look like THIS - Physics News Graphics have published a field-ion microscope image of the sharpest man-made object ever produced - a needle with a tip that's just a single tungsten atom. The thing I love about this image is the way some of the atoms look like blobs of mercury - because they moved during the 1-second-long imaging process.

Java Programmers are the Erotic Furries of Programming

2006-08-04T14:41:00.000+01:00

Nice chart of smugness here on Some Guy Ranting

....but where would CF programmers sit on this chart?

Demos is live

2006-08-03T09:49:00.000+01:00

Nice to see that Headshift have got the new Demos site live. This was the last site I worked on at Headshift, and there's quite a few aspecs to it of which I - and m'former colleagues Neil Roberts and Andy Birchall - feel quite proud.

It's architected with the "Everything-is-a-Item" approach that I learned from my Headshift predecessor, erstwhile colleague and all-round-good-egg Matt Perdeaux, which opens a lot of doors in terms of code reuse and making the most of your content.

Put simply, everything on the site - blog posts, comments, people, users, CMS pages - has the basic common data such as title, summary, and full description abstracted to an Items table, to which you can join as required in your SQL. All links between items and other entities (e.g. tags) are done at the Items level. This makes things like a global search across all content trivially easy (at least in theory - doesn't always work out that way of course... ) and means that once you've implemented tagging for, say, blog posts, you can have tagging on everything else as well with virtually zero extra effort - because everything is-a Item.

As with a lot of systems I've written over the years, it gets a little frustrating that some of the nicest bits are hidden away from public view in the admin interfaces. But you'll just have to take my word for it that there are lots of self-contained AJAX-driven custom tags that do things like tag administration and item-to-item linkages, which only had to be written once, and could be brought into any form for any kind of item and would work straight away - because everything is-a Item.

Other aspects of the site which give me that warm fluffy feeling inside include the friendly url scheme. The site is written with a Fusebox front-end, but there's a very thin layer above that which translates human-friendly urls into fusebox-friendly urls. This is done with Apache mod_rewrite, translating a url like

http://www.demos.co.uk/projects/demoswebsite/blog/artistaudience

into :

http://www.demos.co.uk/projects.cfm?sParams=/demoswebsite/blog/artistaudience

projects.cfm then parses the sParams string into its expected parameters, retrieves any referenced items into request scope, checks permissions, works out the appropriate fuseaction, and passes on to the FB framework by including index.cfm.

It would make a good CFUG case study, I reckon. (Hey Matt - fancy another vaudevillian double-act? Gaw blimey, roll out the barrel guvnor...) Mind you, I'm not sure if it's appropriate for me to reveal the technical innards of a system when I don't work for the company anymore... hmm... over to you then Neil ;)