CrudVision - Lisa Seelye

(n.b., the version of Beast we use may not be the most current. This post may not reflect what's in trunk.)

One of our clients from work uses Beast forum. Unfortunately last week it kind of died. Mongrel's logs were complaining that it couldn't allocate memory, the index (ForumsController#index) was taking upwards of 12 seconds to load...

The problem was SQL. Beast is very greedy when it displays the front page; it does a forums.posts.last.created_at to get the timestamp for when the last post was made. This is done for every top-level forum and it will destroy your site.

At about 21:30ish the previous night I disabled the site because my brain was mush and I couldn't focus enough to diagnose the problem to say nothing of finding a solution. The next day when I got to the office I spent the first 90 minutes of the day diagnosing and fixing the problem. I found that I could find the relevant data with a GROUP BY and ORDER BY with one query for all forums... Except the times were the FIRST post and not the LAST post.

So I spent the next two hours figuring out why MySQL didn't want to ORDER BY and formulating a query that was both quick, returned data for every forum and worked. This is the ugly thing that resulted:

This is fed into:

In the view we just substitute forum.posts.last with @last_posts[forum.id].first in all instances. This may have to bet tweaked for your setup because I'm not sure where Beast ends and client-specific stuff begins.

You see there's some parent_id things going on in the SQL. At work we made Beast forum nested. We do plan to make these changes available for others to use but it's just a matter of finding time to get rid of the client-specific modifications and merging to trunk.

This is one of the things that I'm proud about: the SQL query was a pain!

Yesterday at work I deployed the 'Oracle Project' to the customer's staging server. The only hiccup was (I think) a problem with the renumbering of migrations that was coming from two branch merges and a merge into trunk. I apparently misnumbered one and a critical migration wasn't run. Oops.

I've noticed that at the end of rake db:migrate that Rails rebuilds the db/scema.rb file (nothing new there) except with Oracle each table takes about 45 seconds to process! We've got about 35 models! That's almost 25 minutes! What gives?

Anyways, Oracle is conquered and it is my bitch.

At work I've been struggling with making one of our projects play nice with Oracle. I got Oracle XE installed through Windows XP in Parallels and just couldn't make my tests work. I kept getting errors that Rails couln't insert a nil value into a non-nil field (id).

What this eventually boiled down to is Rails fixtures MUST have an id attribute on them or you may run into errors! This was hidden behind the scenes since heretofore we were using auto_incrementing MySQL fields. To get this behaviour in Oracle one must do

(done by Rails automatically, normally) or set up a Trigger (not done by rails, and would have to be done in a create_table migration).

However, fixtures bypass the normal ActiveRecord::Base#create call and just does a raw execute insert into... Which means there's no value for id, unless it's specified in the fixture. This kept me at it for hours today and I only found it by hacking the fixtures source to make it print in the logs what it was doing!

Let this be a lesson! Always put in ids even if you don't care about their values.

Just finished registering for RailsConf Europe this morning. Travel is booked. Officially I am going on behalf of my work, unofficially I'm going to fangirl next to Rails's niftiest people.

More later!

Yesterday at work we discovered that Actionscript's HTTPService library is extremely crippled in its functionality. How bad is it? Well dear reader it's common practice when writing a webserver or webservice that one should return 404 Not Found if a resource can't be found or 201 Created when a resource has been created. The other status codes also have their usefulness. In my API project for work I believe I use (only for the non-HTML access): 200, 201, 400, 401, 403, 404, 409, and 500. Quite a rich set. I also use the POST, GET, PUT, DELETE verbs.

HTTPService, however, was built in such a way that anything other than 200 OK means "Raise an exception because a fault happened" - including 201 Created (a successful response code!). HTTPService does not permit anything other than GET and POST. Good thing that Rails permits the _method hidden field, eh?

What I am most likely going to have to do is to do the same sort of thing I would have had to do if the API was done with SOAP: Always return 200 OK and maintain a set of "response objects" to cope for shortcomings in other technoloties. Instead of a standard:

I'll probably have to do:

And I'll have to do that in every place. REST? What's that. Argh. Anyone know of a decent Actionscript HTTP library?

Just over four years ago I was given CVS commit access to the Gentoo portage tree. It's been quite a dramatic time and it's been very enlightening. The people I've met have been really nifty! Most of my the things that I maintain are stable. My main regret is that due to work completely sucking up my time I don't have much left over at the end of the day for Gentoo.

At work yesterday I found out that by default ActiveSupport::JSON.unquote_hash_key_identifiers is true. This means that a JSON string looks like:

{attributes: {name: "0-3VW8 I", coordinate_id: "6229", system_id: "30000995", id: "40063299", igbtype: "7"}}

Flash JSON decoders (and probably others) expect the above attributes word to be doublequoted. This is achieved (after much Googling) by setting ActiveSupport::JSON.unquote_hash_key_identifiers=false.

But it doesn't stop there. In the above JSON output the IDs are quoted, that means they're encoded as strings and not integers. This is against the spec.

This means that

The integers should not be double quoted.

With the help of lifo from #rails-contrib the problem seems to have been narrowed down to instance_values/instance_variable_get borking things up. From my poking around (n.b., I'm not all that experienced with the source for Rails itself! I may be wrong) the issue seems to lay within vendor/rails/activesupport/lib/active_support/json/encoders/object.rb. This bit of code seems to work for AR::Base derived models:

Edit (10 July 2007): It seems that this isn't exactly a drop-in replacement fix: If a model has associations loaded (Foo.find :first, :include => :bars) the above JSON-fixing snippet will not work because it looks at attributes only and not the loaded associations. I'll have to work around that.

I did something quite evil today at work.

A customer wants to be able to roll back changes made by its moderators. If there's a difference of opinion they want to roll back a change. Makes sense. (No, it isn't a wiki.)

I decided that a second-best effort (instead of doing:

I'd do:

Why is it evil? It requires two BLOB fields in MySQL, stores the ENTIRE object, twice and can't be searched upon.

The next iteration will likely just be a diff. Something like:

It is just a little better. They want to revert back to it so there's no need to do the diff after, upon reverting.

It works and I was impressed how easy it was because the revert action is simply to Marshal.load the old object and to just .save it.

Ruby is love. Just don't abuse it by stuffing crap into the database!

This is probably going to be the final part.

Last time I stopped just getting ready to talk about designing a JSON based RESTful API.

My initial design had no consideration for an administration component. Every action had a block that looks, at the bare minimum, like this:

I cared about JSON more than HTML or XML (I included XML where it was easy enough on the first pass simply because .to_xml is just as easy as .to_json). Another developer's concern was an HTML-based administration interface for the end users to use. My JSON part was going to be used by "meat" of the project to interact with the database.

At the start of the project I had about two dozen or so models to represent the application. This is all going to sound the same:

ruby script/generate resource -c Person
ruby script/generate resource -c Avatar

And so forth.

Add the basic four (really five) CRUD operations to each controller:

(Don't forget helpers!

)

And then fill in the blanks. Edit config/routes.rb to do resource nesting so editing the first Person's Avatar(s) is baked in to the URL and the caller doesn't have to worry about polluting the URL with parameters that aren't needed.

The tests become simple, too. There isn't much complexity to most resources, thankfully, it's always testing five basic actions.

My coworker that was doing the HTML only had to add a format.html in the respond_to block as well as two new actions (new and edit) to handle peripheral HTML forms that my JSON-using clients would never use. Handy, that.

By using CRUD it meant that REST could be used in Rails automatically. The actions just worked. Resource nesting just worked. I got to spend time thinking about cool stuff, like the application, and not how to make "it" work, kind of like why I liked Rails in the first place! No glue code.

CRUD/REST makes me a happy coder. Using CRUD means, for example, that buying a Product in a E-Commerce store it isn't an operation on a Product per se, but rather it is the creation of a CartEntry in a Person's ShoppingCart (POST /people/1/shopping_cart/cart_entries) and to check out it's just a creation of an action on the ShoppingCartController called, for example, checkout attached to the DELETE HTTP verb. It's debatable whether or not DELETE should semantically mean "Okay, now let me purchase everything" but that's another post.

In the checkout method there's then a concern of what do I do with all these CartEntries? What about a model called a ShippingQueue that the people in the shipping department at the store can use (with RESTful interfaces!) that has some state attached to it (ShippingStatus < AR::Base and then Shipped < ShippingStatus; Received < ShippingStatus, etc). Models are cool.

That's the end. CRUD helps me and that's all that matters in the end.

In the last part of what is probably going to obnoxiously long I realised that not using CRUD was making the application codebase ugly and making me not want to work with it. I rediscovered resources and discovered REST.

One of the projects at my new job was to migrate two Rails SVN repos into one (combining both Rails projects). I must admit that the codebase wasn't the best I had seen. It was clearly designed without REST in mind (especially considering there was no simply_restful and no rails 1.2) or CRUDdiness in mind. A lot of actions in controllers and a lot of scoping (addComment to some model in the model's controller, for example).

Tracing through the code was difficult: views had partials referencing partials that referenced other partials. If the project was started from scratch right now (the client would probably freak out) and it would definiately have a better structure.

After I had merged the two SVN repos into one work began on adding more features and bugfixing to the same application. The usual "Getting to know the codebase" time was applicable, but I can't help wonder what may have been if it had used REST concepts in the beginning adding to it and maintaining it would be much easier (but I may be biased).

Eventually I've become familiar with the codebase and understand how it works and its quirks. I'd love to talk the powers that be into letting us rewrite it RESTfully... but I doubt that will happen any time soon. Maybe Version 2.0.

The next project was one from scratch to expose an API to a database. In the beginning SOAP through ActionWebService was considered but ultimately discarded in favour of a RESTful design, much to my relief! ActionWebService::Struct derrived classes would have numbered in the hundreds in an effort to map the data combinations. Yick.

And so I got to design a RESTful design, this is where I really got to see the goodness of REST; prior to this project everything was just theory. Now I had to chose between scoping methods (PeopleController#addAvatar, which was actually creating an Avatar) or REST. I researched more into REST, played, and was thankful to see that it wasn't as hard as I had imagined. Rather, it was easy. Very easy, almost too easy.

The project in question is a large scale and involves a dozen or so people. There is a Java part, a Flash part, a Rails part and those crazy art-type people (who I believe practice dark arts to make such images appear on paper and computer screens!). The Java and Flash guys had the harder part. All I had to do was architect a bunch of models and controllers to do CRUD operations with an agreeable data envelope (we chose JSON). Everyone else had to knit everything together and make it all work. CRUD is easy, Working with Java and Flash is painful.

Part four will talk about the way I went about architecting the API. It's going to be very boring - CRUD is predictable (which is good!)