Porting a Rails 2.3 app to Ruby 1.9

We finally managed to get enough space in the schedule to take the plunge and port our monolithic Rails 2.3 app to Ruby 1.9, with a view to increasing scalability of our app. An upgrade to Rails 3 is also on the cards for later, but... one thing at a time.

As ever, the path to true Ruby nirvana is paved with good intentions, and tends to detour into dependency hell for a good portion of the way. Here's a quick shortlist of some of the issues we found along the way, and what we did to get round them.

In no particular order, here we go.....

MySQL2 version should be no later than 0.2.x 

If you get the dreaded Please install the mysql2 adapter: `gem install activerecord-mysql2-adapter` error message, what it really means is You can't use MySQL2 version 0.3+ with Rails version less than 3
If you're already using MySQL2 0.2.x, and you're on Mac OS X, and you're still getting the error, then the other thing that it really means is I couldn't find the dynamic library libmysqlclient.18.dylib. There are two proposed fixes for this:

1. sudo install_name_tool -change libmysqlclient.18.dylib /usr/local/mysql/lib/libmysqlclient.18.dylib /Users/YOUR_USER_NAME/.rvm/gems/1.8/gems/mysql2-0.2 - I couldn't get this working
2. sudo ln -s /usr/local/mysql/lib/libmysqlclient.18.dylib /usr/lib/libmysqlclient.18.dylib - This worked for me on Lion

Use Bundler

If you weren't using it before (we weren't), use it now. Stop fighting the inevitable, just give in, bend over and take it - and use Bundler. Seriously, it makes things easier in the long run.

EventMachine does not compile (update: ..easily..) on 1.9.2

...at least not on my Lion Macbook Pro. It kept giving me the compiler error: cc1plus: error: unrecognized command line option ‘-Wshorten-64-to-32’. Under 1.9.3, however - no problems, compiled first time every time.

Update: actually, I did eventually get EventMachine 0.12.10 to compile under 1.9.2 with an evil hack. The error "-Wshorten-64-to-32" above made me think - the compiler is not part of RVM, so the only way the compiler would recognise a command-line option in one Ruby but not in another... is if it's not the same compiler! So I tried this:

$ rvm use 1.9.3@1.9.3rails2.3
Using /Users/aldavidson/.rvm/gems/ruby-1.9.3-p0 with gemset 1.9.3rails2.3
$ irb
ruby-1.9.3-p0 :002 > require 'mkmf'
 => true 
ruby-1.9.3-p0 :004 >  CONFIG['CXX']
 => "g++-4.2" 
ruby-1.9.3-p0 :005 > exit

Now let's see what 1.9.2 is using:

$ rvm use 1.9.2@1.9.2rails2.3
Using /Users/aldavidson/.rvm/gems/ruby-1.9.2-p290 with gemset 1.9.2rails2.3
$ irb 
ruby-1.9.2-p290 :001 > require 'mkmf'
 => true 
ruby-1.9.2-p290 :002 > CONFIG['CXX']
 => "g++" 
ruby-1.9.2-p290 :006 > exit

Ah-hah! So they are in fact using different C++ compilers! And although many native gems (e.g. MySQL) will respect and pass-through the CXX environment variable to the Makefile, sadly EventMachine is not one of them. So, the easiest fix was a bit of evil hackery - move g++ out of the way and symlink it to g++-4.2:

$ which g++
/usr/local/bin/g++

$ which g++-4.2
/usr/bin/g++-4.2

$ sudo mv /usr/local/bin/g++ /usr/local/bin/g++.bak && sudo ln -s /usr/bin/g++-4.2 /usr/local/bin/g++

looking good - let's go for it:

gem install eventmachineBuilding native extensions.  This could take a while...
Successfully installed eventmachine-0.12.10
1 gem installed

Yay!

Rails 2.3 is NOT SUPPORTED under Ruby 1.9.3!

There was a big fuss about Rails loading times under 1.9.3. The way that 'require' statements work has been changed, and it no longer checks to see if the file is there before requiring it. As a result, every controller MUST have a corresponding helper file, even if it's just a stub. This is majorly annoying, as we have 73 controllers across 3 namespaces, and only 17 helpers. The Rails team have "frozen" the 2.3 branch, and say that only security fixes will go in after 2.3.14 - in other words, they're not going to fix this. The good-old monkeypatch-via-plugin method doesn't work either, as the files get required AS Rails is loaded, not once it's initialised.

So there are two options, either:

1. Create a stub helper for each controller
   ...which seems a bit fugly to me, or..
2. Fork Rails, fix the issue, use that fork in the meantime until we can port to Rails 3.

This seems better, and who knows, if enough people complain about it, maybe we'll get a patch release (2.3.15?). I'm not holding my breath, mind.... but if we're in this situation, I'm sure others are too, so this will hopefully help some other people who are having the same problem.
So, here's the forked Rails 2.3.14 that we will fix the helper requires problem on (note: WILL fix, not "have already fixed"! :)

UPDATE: After having got round the EventMachine issue above, we no longer need to do this - so we're carrying on with 1.9.2 and standard rails 2.3.14.

MySQL idle connections still holding locks

We had an interesting problem today. We were seeing very slow single-row updates (>30s) on our (innodb) scheduled_jobs table, and a large number of the update queries were failing with a LOCK WAIT TIMEOUT. The updates were using the primary key, so they should be pretty fast - but they weren't.

So we fired up a console and ran SHOW INNODB STATUS, and saw lots of transactions with long-held locks -

---TRANSACTION 0 200086649, ACTIVE 3000 sec, process no 29791, OS thread id 140353331377936
12 lock struct(s), heap size 3024, 6 row lock(s), undo log entries 10
MySQL thread id 243, query id 314676 ip-10-94-245-79.ec2.internal 10.94.245.79 tn


- however, when we cross-referenced the MySQL thread IDs with a SHOW PROCESSLIST, we found that lots of the threads weren't actually doing anything:

+------+------------+-----------+-----------------+----------------+------+--------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+------------+-----------+-----------------+----------------+------+--------------+------------------+
| 243 | tn | (..snip..)| tn_production | Sleep | 3000 | | NULL


This was strange - but after a bit of a Googling and a few minutes thought, we realised the cause.

When one of our worker processes dies - e.g. either because monit has killed it for taking up too much resource for too long, or because the EC2 instance it's running on has become unresponsive to ssh and been automatically restarted by our detect-and-rescue script - it may be in the middle of a transaction at the point it's killed.

If the transaction was holding locks, and never got to the "COMMIT" stage, and the connection wasn't closed normally, then that connection will persist - and maintain its locks - until it gets timed-out by the MySQL server.... and the default wait_timeout variable is 28800s - 8 hrs!.

So, we killed the idle connections, and the update queries started going through much more quickly again. We've now brought our wait_timeout variable down to 10 mins, and we'll see how we get on.