CrudVision - Lisa Seelye

Yesterday we launched Mini Match, an application me and my colleagues from work wrote for Cartoon Network.

Late last year we opened a beta that was, unfortunately, short lived. Friday (May 30, 2008) we opened the system up on a much improved codebase for a few hours and had all systems “green”. Based on the positive success from Friday we opened it up yesterday.

It opened slowly at first with just a small advertisement on the Cartoon Network Games’ Page, and then a larger one on the same page and then we made the Cartoon Network home page with a small advertisement again. Today, I reckon, a larger advertisement will be put on the front page and we’ll really start to see traffic!

Some details on the application:

• Flash/Flex/AS/Whatever front-end GUI (really, it’s one of them)
• Java-based persistance server
• Rails-based funnel into the database with a bit of logic.

The Rails part is RESTful (for the most part) and is the “glue” of the application, to quote someone from IRC.

Today should be a fun day!

This coming Monday I’m flying off to Atlanta, Georgia for the week. Not keen on going as Atlanta is like descending into the armpit of summer. I’m hoping that the weather co-operates and I don’t melt.

I’m flying down there for work, we’re beta launching a product for a customer (that’s based in Atlanta - go figure) and they (the customer) have requested a contingent of us from work be on site for the release. So it’s me and two others from the UK.

We’re down on Monday then flying back home on Friday evening - I’m hoping to be in bed by midnight (landing 22:05 - 30 minute taxi home, getting bags and immigration and such). A SysAdmin that works for our customer suggested Zuma sushi bar to me for sushi. I like sushi a lot and hope to stop in for some glorious sushi.

I received an email to my work account today that started with “Dear Sirs…”. The email was sent, presumably, to a large number of customers at once about a networking issue by a UK hosting company.

I am female and at the time the salutation bugged me. In normal society one doesn’t generally greet a member of one gender with the salutations of the other. When I received the email my brain put together discussions from LinuxChix mail lists and other recent discussions on women in IT.

I took offence at the salutation “Sirs” because it implies that there are no women working in IT which is clearly not the case.

Anyways, I sent an email in reply to the email (which went to a support@ email) saying that I didn’t appreciate being addressed as a “Sir”. Less than 25 minutes later I received an email from the managing director of saying he wrote the message and thought the salutation “Sirs” was …was an acceptable greeting if the gender of the other party is unknown. I’m not sure I agree but I appreciated the feedback anyways and replied to him thanking him for taking the time to reply. Hopefully he will do as his email says and take my feedback on board and change the way his company addresses emails to women.

My good deed for the day is done.

First the problem:

I'm writing a load test framework at work and I need to consume JSON webservice and pass on the output to another request. But I don't always know what the data is that I need to pass on but I do know the basic "path" ("hpath" -- "hash path") to get to the data as the JSON data is uniform.

Now the method:

What's it do? It does magic!

I'll step through it...

Looks for @passon instance variable and doesn't do anything useful unless it exists.

Duplicate the input hash because the process done is destructive to it and we may need to reuse it.

For each new hash key and "hpath" pair from @passon split up the hpath into its parts and reverse it so Array#pop will work in the while loop to get the next first part to try.

Since each part of the hpath can examine an Array by index it has to be checked for and the index removed from the part (and saved).

Next, investigate the copy of the hash_data, nh by the computed key; if the value in the hash was an Array use the index to get the desired value. Then compute the next part!

Once we're out of parts stuff the new key and data into the returnhash and keep going til there's no more @passon pairs.

And thus some fun code was written.

With Rails there's a useful set of callback methods that can be used within model classes. They include after_create, before_validation and the one I want to talk about after_initialize.

A lot of cool things can be done with after_initialize, especially if one uses the database as a means to store meta-data about a model. Use the after_initialize method to transform your models with Ruby after they're fethced from the database. Consider the following User model definition:

It's pretty clear that with the :group => "group_id" that I want a unique set of Membership objects. That works well when I've already got a User and do user.memberships but not at all when I do User.find(:first, :include => :memberships); likely because of the way the join is set up and grouping on that may not be possible.

I still wanted to use eager loading so I thought that it would be an OK sacrifice to fetch all the Membership objects even if I wanted to pare it down after I initialize the User object (with eager loading). I defined:

Except it doesn't work. Quickly I found the following from active_record/base.rb (click link for source): right where it does the after_initialize callback! But it wasn't working for me. I needed to dig further.

Further research indicated that before the creation of the @memberships instance variable that would hold the collection of Membership objects the after_initialize callback was being fired off in the User model. Using after_initialize would not work due to the design of ActiveRecord. I'm still searching for an elegant way to work the way I want. I'll post again when I find it!

I've submitted my first patch to the Rails trac today to speed up the OracleAdapter#indexes method (used in migrations). A client at work uses Oracle and in the production environment this method was taking upwards of 45 seconds to run per table!One of their DBAs gave me the SQL there, so I can't take the credit, I just submitted it.

I hope it's accepted. Incidentally, it works and has saved me a lot of time with migrations.

Edit: Yeah, turns out that since turning these adapters into gems patches for them shouldn't be done through Rails trac. How irritating, but whatever. I've sent the patch do the current maintainer for review.

One part of code optimisation that is relatively easy is to cache the results of expensive operations. It's really simple. CACHE.set. But it isn't so simple. There are a seemingly endless possible ways the cache can be used improperly or otherwise fail. Handling these problems is the real work of caching.

The last checkout (around rev 250?) I did of Cache_fu (the acts_as_cached rewrite) it was pretty immature, chiefly in our need: An easy way to do arbitrary key names. n.b., I'm aware that there is a way but it seemed against the flow of this codebase. I use it, still, for the magic :all keyname.

But I need more, to store created JSON (To create what results in 100KB of JSON is a lot of objects and a lot of create time!) with a dynamic key in several distinct situations.

That round of caching improved API calls from 3 req/s to 300 req/s.

For the application most of the data can be cached but it needs to be done smartly: memcache server failures need to be handled (I suspect they are not, currently), server layout needs to be better (One daemon on each app server is not good, long term, IMHO), expiration of caches on data changes across the entire farm need to be smarter, deciding how long to cache data needs to be done.

CACHE.set and CACHE.get is really misleading; there's a lot of things to consider!

I relish these challenges and look forward to solving them.

On Monday one of our clients from work will begin a campaign to market the product to the public.

This means that the API I've written for it is stable and will be put to the true load test. I suspect I will be busy this week attempting to improve the performance of the Rails code. It's my hope that it will turn out to be perfect and we'll surpass Twitter in terms of concurrent users. Probably not since Twitter has about 450k registered accounts, the client doesn't expect that many.

With the bulk of the programming done the next phases come into play: optimisation and scaling. Both of these I'm a big fan of and I can't wait to get into it.

Clever title, eh?

Anywho... today was DHH's Keynote and a few other lectures that I was interested in. They included the Caching in a Multilanguage Environment by Benjamin Krause (of omdb.org), Rails Full Text Search with Ferret by Jens Kräuse and Really Scaling Rails by the Twitter guy(s).

Benjamin's multilanguage lecture consisted of introducing a couple plugins that altered Rails behaviour to enable (wait for it) multilanguage caching. The plugins (whose names I cleverly have forgotten but can likely be found somewhere on OMDB's trac, perhaps mlr.) were basic things. Change ActionController to handle ACCEPT_LANGUAGE and Routes to tack on a :lang fragment as in :id.:format.:lang (/people/1.html.en) to match Apache's mod_negotiation.

The plugins also provide test methods. All in all the lecture wasn't real revolutionary (to echo DHH's sentiment about how Rails should now move). Caching content based on a language. Of course! It makes perfect sense and is something many developers (especially in Europe?) would be interested in.

The next lecture was the Ferret lecture. At work we're really interested in using something like Ferret to offload some work from the database. One of our client's sites has already been bitten by DB load (see the Beast SQL Speedup) and have expressed an interest in having fulltext search with boolean operations. I'm very glad I sat in on this as it's something I've been wanting to do with evedb.info.

Jens's lecture was really cool. I admit to being naive when it comes to Ferret (really I mean acts_as_ferret): I know it can be used for searching but how to leverage its full power is (or was?) beyond me. Today was the first real taste of how to use it and it was like getting my head around REST and realising how useful it WILL be.

Finally the last lecture I attended today was the Really Scaling Rails by one of the Twitter guys. This was a cool lecture since Twitter is really what everyone using Rails hopes to achieve. And on an extremely limited way we (at work) have had to deal with these rapid scaling problems. We've had a client on the phone wondering why the forums took 12 seconds to load (if it did load) and why one browsing part of the site took 5 seconds per page. Quick fixes (For those two instances less than a day's work) and it keeps the client happy (and me! I like finding these solutions even if I end up cursing MySQL in the process).

The scaling lecture could be best summed up by: Don't optimise prematurely with a corollary stating "But be ready to optimise on a moment's notice." I agree.

I'll admit it. I haven't gotten around to learning Capistrano 2 yet so many or all of these issues may be solved.

One of our clients at work has a very bizarre deployment environment. To be brief (without getting sacked) it involves svn replication, shared NFS directories and a dedicated deploy box. They also have this setup twice, once for production and once for a "staging" environment.

Since they copy our SVN tagged release to their own SVN repository it means the Capistrano script doesn't actually do the mirroring (yet?); that is done by the client. I wrote a helper bash script that will take the tag, environment (production / "staging") and a capistrano command to select the right capistrano script, set the right environment (although now that I think about it this is redundant now) and run the command.

But there has to be an easier way to "One Click" this with Capistrano 1.x. It is my task to find a recipe to do it!

The problems are, due to the convoluted nature of the deploy environment I may have to abuse roles or invent my own or hack Capistrano to add qualifications to roles (similar to :primary): The dedicated deploy box is the box to which the application is deployed (acts like an app server!) but it doesn't/shouldn't run any mongrels. In a sense it's the primary app server since it has the authoritative source that's shared to other app servers.

But it isn't! It's a nightmare! It needs to be better done. I may try to one-shot this: sh deploy.sh SomeTag production deploy_first or deploy_upgrade.