Friday, May 22, 2009

This should not be possible - transactional fail?

We're having a bizarre problem with MySQL 5.0.32 on Debian in a highly-concurrent environment, using the ActiveRecord JDBC adapter. We've also seen it on 5.0.51 on Ubuntu.

There's a particular sequence of 3 queries, all executed from within a ActiveRecord::Base.transaction { } block:

// Statement 1:
INSERT INTO (InnoDB table 1)
(type, ...loads of other fields...)
SELECT 'ProtoMessageItem', ...other fields...
FROM (MyISAM table 1) LEFT OUTER JOIN (InnoDB table 2)
WHERE (anti-duplication clause)
GROUP BY (synthetic identifier on MyISAM table 1, to be a unique key on InnoDB table 1)

// Statement 2:
INSERT INTO (InnoDB table 2)
SELECT (fields)
FROM (MyISAM table 1) LEFT OUTER JOIN (InnoDB table 3) INNER JOIN (InnoDB table 1)
WHERE (anti-duplication clause)

// Statement 3:
UPDATE (InnoDB table 1)
SET type = 'MessageItem'
WHERE type = 'ProtoMessageItem'

Now, all of this takes place within an explicit transaction, and works well - this approach provides huge performance gains over the more simple and obvious methods.

The problem is that just very very occasionally, it breaks. We don't know how or why, but once in a blue moon, statement 2 throws a DUPLICATE KEY exception, and the transaction DOESN'T get rolled back. Statement 1 still gets COMMITTED. We can see ProtoMessageItems in the DB, and this blocks up the pipeline and stops further MessageItems from being created.

ProtoMessageItems should NEVER be visible - they're only ever created and then updated to full MessageItems in the same 3-statement transaction. Seeing them from outside that transaction is like seeing a naked singularity - it just shouldn't be possible.

This is all kinds of wrong in a very wrong way, as it means one of the following - either:

  1. The DB is getting the correct BEGIN/COMMIT/ROLLBACK statements, but transactional atomicity is not holding for some reason

  2. The JDBC driver is not sending the correct ROLLBACK statements, but thinks it's in autocommit mode

  3. There's an obscure concurrency edge case somewhere with the ActiveRecord JDBC adapter, that means once in a while, for some reason, it starts that transaction in autocommit mode.

All of these possibilities are worrying. We've eliminated the possibility of it being some quirk of the particular data that causes this problem. We've turned on MySQL statement logging and watched it for hours, and as far as we can see, the correct ROLLBACK statements seem to be issued every time.

Has anyone encountered a similar problem? Any known issues with INSERT/SELECT on combinations of (transactional) InnoDB and (non-transactional) MyISAM tables?

Tuesday, May 19, 2009

Suggestions for GruntFuddlr !

Following on from the previous post on The imminent death of Twitter, after several comments along the lines of "Oh Al, you have SOOOO got to buy that domain" followed by a brief moment of hastiness with the credit card, I am now the proud owner of !

So all I need to do now is decide what to do with it.... add your suggestions to the comments! The winner will win - erm - something to be decided!

Friday, May 15, 2009

Twitter Will Soon Die - the second Tipping Point approaches

Maybe I'm sticking my neck out, but it's inevitable - hear me out here. By now you're surely familiar with the concept of The Tipping Point as espoused by Malcolm Gladwell -
the moment of critical mass, the threshold, the boiling point
the point at which momentum for change becomes unstoppable

This has been adopted as a central tenet of the Web 2.0+ business model, and is surely now almost as ubiquitous on a web startup's business plan as a long tail reference or hockey stick graph. This post is about the Tipping Point concept specifically as applied to social networking services.

It's certainly true in my experience that every successful social software service goes through some standard stages of growth:

  1. Early adoption
    The service gets adopted by a set of keenly curious early users. This is often through an initial restricted beta program, or through the conference circuit, or friends and families of those involved in the development. Many never get past this stage - maybe it's just not a particularly good idea, or useful service, or well-executed, or maybe it's just unlucky. Assuming the service does get past this stage, they move on to...

  2. Viral spread
    The early adopters have judged and found it worthy. Here's where they get to gain geek cool points by name-dropping it amongst their friends, on the conference circuit, in Friday-night bars, etc... often taking advantage of the opportunity to faintly patronise those who are still using an inferior / more mass-market / older service - I mean, how many times have you heard lines like these?

    ....Oh Digg totally jumped the shark, I moved off there to Reddit and now I've started using MangWurdlr (or whatever)
    ...Yeah I use GruntFuddlr for all my (whatever)-ing now, it's nice, they've got a really cool UI, and ....etc...etc...
    ...Oh, you should have a look at PigNudgr - they're still in beta, but it looks really cool, and they're a great bunch of guys - I used to work with Andy at (name of now-defunct late-nineties web agency, usually some combination of colour and animal) and he's doing some really neat stuff now...
    etc etc.

    Every time I have one of those conversations I'm reminded of when I was in the sixth form (that's school from 16-18, when you've passed your first exams - sorry I don't know what the US equivalent is) and people would compete for alpha-cool status by getting into ever-more obscure bands on ever-smaller labels.

    Remember those? Bands like "Anna", "Spontaneous Cattle Combustion", "Generic me-too grunge band #241" - when they appeared in NME for the first time they were the coolest thing since sliced bread, and if you didn't have a copy of their debut EP you were nobody, maaannn - ? It's the same phenomenon. Web startups / bands - same thing, just slightly older protagonists. Same silly haircuts though.

  3. The first Tipping Point

    This is the Holy Grail for earlier-stage web services. This is where you succeed in creating enough word-of-mouth buzz that suddenly, you're unstoppable. You tip over into the mainstream. People's parents have heard of you (well, those that read the more techno-savvy broadsheets anyway) A whole new wave of joiners is poised to come on-board, often purely because of who is already on there. Prime example - Stephen Fry on Twitter.

  4. Inflation

    (apologies to Alan Guth!)
    This is where the whole thing goes crazy. Faster growth than they ever thought possible. All those times the engineers said "Well, that's a high-class problem to have, and we'll just have to deal with that when we get there.." well - they're there right now. Seems like every week there's another digit on the hit counts. Often characterised by regular outages due to scale fail in completely unexpected ways ("Holy crap, we've run out of inodes??? That's possible?"). They're sticking in new servers by the dozen all the time, burning through all that VC funding and not making any money yet..

    Meanwhile the user base seems to take on a life of its own. It's no longer a community, it instead takes on the characteristics of a population - splits, fragments, in-fighting, single-issue activism - often against the decisions of the founders, who in their increasingly urgent need to get revenue to support the spiralling infrastructure costs are taking commercially-led decisions and seen as increasingly out-of-touch with the still-trying-to-be-faithful users.

    There's conflicting senses of excitement and disappointment, that it's just not the same as it used to be. And who are all these ****wits joining now anyway? What - my MUM just joined? And she wants to be my friend? Oh crap, better get rid of all of those tales of drunken debauchery and undeniable photographic evidence. Your boss probably joins up, adding further fuel to the conflict of identity. Is this personal? A work tool? Both?

  5. The second Tipping Point

    All good things must come to an end, and this is the end of the line for the early adopters. It's not new, edgy or cool anymore, now that it's talked about on Question Time and even the Government are starting to use it in some god-awful, half-assed but completely cringe-making attempt to be Down With Tha Kids. Meanwhile, you keep getting friend requests from people at school who you weren't friends with in the first place and have no intention of pretending to be friends with now.

    The whole world and his hamster seems to be trying to spam you about it or on it. The number of z-list celebrities on it just keeps growing - Peter Andre? Really? Christ... - and they don't even post themselves, it's done via their PR person, or if they do they're just pushing their warez (yes, @WilliamShatner, I'm looking at you).


    You've had enough. You can barely be bothered to check it any more, and all the third-party apps built on the API are just getting on your nerves, and all the cool kids have moved on to anyway... apparently it's really cool, and apparently Dave from got together with Si from PurplePlatypus to start it up, and they've got a really neat UI, and it doesn't have all of these ****wits on it...

So that's the Second Tipping Point. When all of the people who joined before the first Tipping Point leave, partly due to their constant need for something new, partly because services just naturally evolve, but mostly because of all the people who joined just after the first. Who, in their turn, had mainly joined because of all the people who were already on it...

Facebook got there a few months ago. Twitter is rapidly approaching it, and you know what? It's natural. It's part of the inevitable evolution of ideas and the services built upon them. It's not that Twitter has anything fundamentally wrong with it, it's mainly that the problem with software that puts people in touch with people..... is people. The cliques and cooler-than-thouism that ruled school society never went away, they just changed form and went online.

MySpace took over from first-gen social software like SmartGroups, and while its user numbers may technically still be growing, in new-frontier terms, it's now dying. It was supplanted by Facebook, which has become supplanted by Twitter, which will soon be supplanted by some other new kid on the block. No-one knows what it will be yet, but that's part of the whole fun of this business - natural selection in the full glare of the web, with all the competition, cross-breeding, malformed offspring and occasional apparently-arbitrary successes. I love it.