CrudVision – Lisa Seelye

December 8, 2007

Rails has_many gotcha

Filed under: activerecord,crud,rails,ruby,sql,work — Lisa Seelye @ 14:39

With Rails there's a useful set of callback methods that can be used within model classes. They include after_create, before_validation and the one I want to talk about after_initialize.

A lot of cool things can be done with after_initialize, especially if one uses the database as a means to store meta-data about a model. Use the after_initialize method to transform your models with Ruby after they're fethced from the database. Consider the following User model definition:

RUBY:
  1. class User <ActiveRecord::Base
  2.   has_many :memberships, :o rder => "memberships.group_id, memberships.expires_at desc", :group => "group_id"
  3. end

It's pretty clear that with the :group => "group_id" that I want a unique set of Membership objects. That works well when I've already got a User and do user.memberships but not at all when I do User.find(:first, :include => :memberships); likely because of the way the join is set up and grouping on that may not be possible.

I still wanted to use eager loading so I thought that it would be an OK sacrifice to fetch all the Membership objects even if I wanted to pare it down after I initialize the User object (with eager loading). I defined:

RUBY:
  1. class User <ActiveRecord::Base
  2.  
  3.   def after_initialize
  4.     do stuff with @memberships instance varible
  5.   end
  6. end

Except it doesn't work. Quickly I found the following from active_record/base.rb (click link for source): right where it does the after_initialize callback! But it wasn't working for me. I needed to dig further.

Further research indicated that before the creation of the @memberships instance variable that would hold the collection of Membership objects the after_initialize callback was being fired off in the User model. Using after_initialize would not work due to the design of ActiveRecord. I'm still searching for an elegant way to work the way I want. I'll post again when I find it!

July 29, 2007

Mundanity

Filed under: REST,crud,evedb.info,rails — Lisa Seelye @ 11:44

This whole bulk updating system I did a while ago works well, except it's very slow with a single mongrel. That problem could be solved with a simple proxy in front of many mongrels and a threaded updater application. For mass schema changes, for now, the easiest method is to dump from the CCP into my schema. I may be crazy but I'm not that crazy.

Things are coming to a close withthe RESTification of evedb. Next action once the RESTification is done is to get the site a proper design! I've got a sort of mental road map for evedb and so far I'm at what equates to "version 0.3". Long way to go but it's a fun ride. I think I need a vapourware release...

July 17, 2007

How NOT to write a HTTP socket library

Filed under: REST,actionscript,crud,work — Lisa Seelye @ 07:38

Yesterday at work we discovered that Actionscript's HTTPService library is extremely crippled in its functionality. How bad is it? Well dear reader it's common practice when writing a webserver or webservice that one should return 404 Not Found if a resource can't be found or 201 Created when a resource has been created. The other status codes also have their usefulness. In my API project for work I believe I use (only for the non-HTML access): 200, 201, 400, 401, 403, 404, 409, and 500. Quite a rich set. I also use the POST, GET, PUT, DELETE verbs.

HTTPService, however, was built in such a way that anything other than 200 OK means "Raise an exception because a fault happened" - including 201 Created (a successful response code!). HTTPService does not permit anything other than GET and POST. Good thing that Rails permits the _method hidden field, eh?

What I am most likely going to have to do is to do the same sort of thing I would have had to do if the API was done with SOAP: Always return 200 OK and maintain a set of "response objects" to cope for shortcomings in other technoloties. Instead of a standard:

RUBY:
  1. render :json =&gt; @product.to_json, :status =&gt; :created

I'll probably have to do:

RUBY:
  1. render :json =&gt; { :product =&gt; @product, :status =&gt; ErrorObject.new(201) }.to_json, :status =&gt; :o k

And I'll have to do that in every place. REST? What's that. Argh. Anyone know of a decent Actionscript HTTP library?

July 15, 2007

One more reason why REST is good

Filed under: REST,crud,evedb.info,postgres,rails,ruby — Lisa Seelye @ 09:30

In my quest to RESTify Evedb.info I keep finding more reasons to fawn ove REST (and CRUD): With the most recent Eve Online Database Dump for Revelations I have a need to migrate the data to my own schema. I also need to treat the database dump as a "Master" copy of what should appear in my site; stale data must be removed from my database.

I like to automate things. Prior to the RESTification efforts I would pump out raw SQL files and import them into Postgres in a batch. The SQL took about 45 minutes to import and was often held up due to foreign key issues.

Anyways, with REST I can funnel all of that through my controllers which have that functionality anyways (albeit heavily secured so normal users can't use this CUD API). I can have the same standard method for updating my version of CCP's data.

CCP has a lot of extra data in its Database dump (read: I don't use every column from every table). When I felt the need to import one of those I had to mess about with SQL files. Evedb.info was not using migrations! The SQL files were too big to be under source control. And a better solution was needed.

Enter: Evedb::Updater.

I wrote a class today to use the libcurl libraries for Ruby since the native Ruby Net::HTTP libraries were giving me trouble with the encoding of POST data. The general structure of the class I wrote is the following:

  1. Initalise the environment (Create a Postgres connection, Curl object, and store the base URL)
  2. Get a list of all of the IDs of the current model I'm dealing with (custom controller method all_ids since I have index paginated)
  3. Process an SQL query designed to do the schema change (
    SQL:
    1. SELECT agentid AS id FROM agtAgents

    ) for the specific model. This results in a list of things to create, update and destroy

  4. Do the deletions
  5. Do the creates
  6. Do the updates

It was a fun bit of code to write! The highlight is probably this bit:

RUBY:
  1. module Hash
  2.   def to_curl
  3.     [] if self.keys.empty?
  4.     ret = []
  5.     self.keys.each do |k|
  6.       self[k].each do |v,l|
  7.         ret <<Curl::PostField.content("#{k.to_s}[#{v.to_s}]",l.to_s)
  8.       end
  9.      end
  10.     ret
  11.   end
  12. end

The hash is generated by

RUBY:
  1. 0.upto(sql_results.num_tuples - 1) do |i|
  2.   m = { type => {} }
  3.   sql_results.fields.each do |field|
  4.     m[type].merge!( { field.to_sym => sql_results.getvalue(i,sql_results.fieldnum(field)).rstrip.fixmscrap.gsub(/'/,'\\\\\'') } )
  5. end

The result of the to_curl method is to make the POST data that how we rails folk like it! (agent[name]=foo).

One downside to this is that some models need to query the Rails database (not the CCP one) for information to use. I might be better off modifying those specific models to use virtual attributes and some clever use of before_create!

June 27, 2007

Why CRUD? – Part 4

Filed under: crud,rails,work — Lisa Seelye @ 13:59

This is probably going to be the final part.

Last time I stopped just getting ready to talk about designing a JSON based RESTful API.

My initial design had no consideration for an administration component. Every action had a block that looks, at the bare minimum, like this:

RUBY:
  1. respond_to do |format|
  2. format.js
  3. end

I cared about JSON more than HTML or XML (I included XML where it was easy enough on the first pass simply because .to_xml is just as easy as .to_json). Another developer's concern was an HTML-based administration interface for the end users to use. My JSON part was going to be used by "meat" of the project to interact with the database.

At the start of the project I had about two dozen or so models to represent the application. This is all going to sound the same:

ruby script/generate resource -c Person
ruby script/generate resource -c Avatar

And so forth.

Add the basic four (really five) CRUD operations to each controller:

RUBY:
  1. def index
  2. end
  3. def show
  4. end
  5. def create
  6. end
  7. def update
  8. end
  9. def destroy
  10. end

(Don't forget helpers!

RUBY:
  1. before_filter :get_thing, :o nly =&gt; [ :show, :update, :destroy ]
  2. protected
  3. def get_thing
  4. @thing = Thing.find params[:id]
  5. end

)

And then fill in the blanks. Edit config/routes.rb to do resource nesting so editing the first Person's Avatar(s) is baked in to the URL and the caller doesn't have to worry about polluting the URL with parameters that aren't needed.

The tests become simple, too. There isn't much complexity to most resources, thankfully, it's always testing five basic actions.

My coworker that was doing the HTML only had to add a format.html in the respond_to block as well as two new actions (new and edit) to handle peripheral HTML forms that my JSON-using clients would never use. Handy, that.

By using CRUD it meant that REST could be used in Rails automatically. The actions just worked. Resource nesting just worked. I got to spend time thinking about cool stuff, like the application, and not how to make "it" work, kind of like why I liked Rails in the first place! No glue code.

CRUD/REST makes me a happy coder. Using CRUD means, for example, that buying a Product in a E-Commerce store it isn't an operation on a Product per se, but rather it is the creation of a CartEntry in a Person's ShoppingCart (POST /people/1/shopping_cart/cart_entries) and to check out it's just a creation of an action on the ShoppingCartController called, for example, checkout attached to the DELETE HTTP verb. It's debatable whether or not DELETE should semantically mean "Okay, now let me purchase everything" but that's another post. ;)

In the checkout method there's then a concern of what do I do with all these CartEntries? What about a model called a ShippingQueue that the people in the shipping department at the store can use (with RESTful interfaces!) that has some state attached to it (ShippingStatus < AR::Base and then Shipped < ShippingStatus; Received < ShippingStatus, etc). Models are cool.

That's the end. CRUD helps me and that's all that matters in the end.

June 26, 2007

What Is CRUD?

Filed under: crud,rails — Lisa Seelye @ 21:47

CRUD is an acronym for the four basic types of SQL commands: Create, Read, Update, Delete.

Sticking to those four basic operations when dealing with models (of your data structures) and relationships between them can make handling the project easier as abstract things like join tables now have a real name and can hold real data (instead of two foreign keys).

It makes more sense in a Web application environment to match HTTP verbs with CRUD operations:

  • GET - Read
  • POST - Create
  • POST - Update
  • DELETE - Destroy/Delete

Each model in the project, for example in a store application one might have a collection of Products for sale, Categories of Products, and Customers that buy browse Categories and (buy) Products. In a RESTful application (one which does the above verb mapping) each instance of each model gets it's own URL, such as:

  • /products - refers to all of the Products (POST a new Product to this URL to add it to the collection)
  • /products/1 - refers to the Product whose ID is 1 (RUD operations are done on this URL)
  • /categories/1/products - In a properly set up application this refers to the Products that belong to the Category whose id is 1
  • /customers/1/products - This could refer to the products the Customer whose id is 1 has purchased.

This contrived example is incomplete (there is no shopping cart!) and there models are simple. However our store just had a holiday and we gave special codes to the top 100 customers to redeem for a free hat. There's now a model called Coupons (and they would live at /coupons with a cool GUI for the store manager to tinker with) but there's nowhere to redeem the Coupon. It doesn't make sense to do it with the Customer model or the Product model because there's nothing in those models that stores that the Customer has redeemed a Coupon. Looks like we need a join table with columns coupon_id and customer_id. But we don't know when they redeemed it and there's no way for them to store cool feedback saying congrats on the holiday.

At this point I would like to remind that this is very contrived.

There's got to be a better way to do the redemption of Coupons and get a free Product. And there is, I think (otherwise I wouldn't have this blog), CRUD to the rescue.

Call the new thing a UsedCoupon. It still has the attributes from the old join table and perhaps a timestamp and text field. Creating a UsedCoupon is equivelant to the act of redeeming (and getting the cool Product). Redeeming it is POSTing a new UsedCoupon object to /used_coupons; the associated product is given to the Customer and I have a reason to blog.

I happen to like the limitations because it's illogical to add unnecessary methods onto controllers (in MVC terms) when there's no need. Why not just make another URL, they're cheap. CRUDdy controllers will be lightweight since there aren't many methods (a typical max of 5!). Rails happens to afford the developer helpful methods to generate the right URLs from models.

To answer the question What is CRUD? I answer: CRUD is helpful.

June 25, 2007

Why CRUD? – Part 3

Filed under: crud,rails,work — Lisa Seelye @ 21:45

In the last part of what is probably going to obnoxiously long I realised that not using CRUD was making the application codebase ugly and making me not want to work with it. I rediscovered resources and discovered REST.

One of the projects at my new job was to migrate two Rails SVN repos into one (combining both Rails projects). I must admit that the codebase wasn't the best I had seen. It was clearly designed without REST in mind (especially considering there was no simply_restful and no rails 1.2) or CRUDdiness in mind. A lot of actions in controllers and a lot of scoping (addComment to some model in the model's controller, for example).

Tracing through the code was difficult: views had partials referencing partials that referenced other partials. If the project was started from scratch right now (the client would probably freak out) and it would definiately have a better structure.

After I had merged the two SVN repos into one work began on adding more features and bugfixing to the same application. The usual "Getting to know the codebase" time was applicable, but I can't help wonder what may have been if it had used REST concepts in the beginning adding to it and maintaining it would be much easier (but I may be biased).

Eventually I've become familiar with the codebase and understand how it works and its quirks. I'd love to talk the powers that be into letting us rewrite it RESTfully... but I doubt that will happen any time soon. ;-) Maybe Version 2.0.

The next project was one from scratch to expose an API to a database. In the beginning SOAP through ActionWebService was considered but ultimately discarded in favour of a RESTful design, much to my relief! ActionWebService::Struct derrived classes would have numbered in the hundreds in an effort to map the data combinations. Yick.

And so I got to design a RESTful design, this is where I really got to see the goodness of REST; prior to this project everything was just theory. Now I had to chose between scoping methods (PeopleController#addAvatar, which was actually creating an Avatar) or REST. I researched more into REST, played, and was thankful to see that it wasn't as hard as I had imagined. Rather, it was easy. Very easy, almost too easy.

The project in question is a large scale and involves a dozen or so people. There is a Java part, a Flash part, a Rails part and those crazy art-type people (who I believe practice dark arts to make such images appear on paper and computer screens!). The Java and Flash guys had the harder part. All I had to do was architect a bunch of models and controllers to do CRUD operations with an agreeable data envelope (we chose JSON). Everyone else had to knit everything together and make it all work. CRUD is easy, Working with Java and Flash is painful.

Part four will talk about the way I went about architecting the API. It's going to be very boring - CRUD is predictable (which is good!)

June 24, 2007

Why CRUD? – Part 2

Filed under: crud,rails — Lisa Seelye @ 10:01

This is Part 2, following Why CRUD? - Part 1

I realised some time later that I was not working on the site; the same thing was happening that had happened with PHP! I wasn't developing because it was a pain. The code was horrible. I had developed a code base that I didn't want to work with it!

Something needed to change. I liked Ruby and I liked Rails - something needed to change...

The thing that changed was me re-realising that URL don't mean "The address you put into the browser and hope you get something back" but rather it meant: Uniform RESOURCE Locator! A URL is a (fully qualified) way to point to a resource on the internet. I wrote about this in 2003 on my personal site and touted the glories of "Extensionless URLs" but had seemingly forgotten my own example. /images/mycity should give the requestor what they want based on the HTTP_ACCEPT headers sent to the server, so if they want a BMP more than a PNG more than a JPEG they should get it, in that order, if possible. /images/mycity is a resource, the extension simply specified the format that the resource was in.

And it's still damn cool.

Reading about REST and RESTful URL helpers in Rails made the somewhat dimmed lightbulb bright up again. I got it; I understood. The address in the bar is simply an address to a resource and we should give the user what they want (within reason).

Further limiting the thinking to basic CRUD (Create, Read, Update, Destroy) operations means that instead of getting /view/agents/:id I'd have /agents/:id linking to the show method in the AgentsController. Not to parrot too much but DHH was right: These constraints are liberating. No guessing that to talk about an Agent I have to look into the ViewController. How is that intuitive? Why should I have to worry about that crap? Let a convention deal with it.

So I was all about Rails again, which was good because about the time my interest in Rails was really really growing (again!) I was hired to do Rails coding.

Part 3 will deal with having to use an existing non CRUD/RESTful codebase and then being able to write an API to use CRUD/REST exclusively.

June 23, 2007

Why CRUD? – Part 1

Filed under: crud,rails — Lisa Seelye @ 16:30

To preface: I'm going to talk a lot about my pet project, evedb.info, which is a fansite for Eve Online which seems to be stuck in a perpetual development phase. The site was written in rails as a learning project.

Like many other web developers that use a MVC model of coding I thought the pinacle of design (and by "design" I mean code design and URL presentation) was to have a ViewController that would display Agents, the Corporations to which an Agent belongs, the Stations that a Corporation owns, the Systems in which the Stations are and the Items available in the game and so forth. This meant my ViewController had a bunch of methods: agents, corporations, stations, systems, items and so forth. The URL looks like /view/agents/[name of agent]. Station names are very long so I used their id instead of their name.

When I migrated to Rails 1.2 I had a problem. Since my URLs included the names of the thing being viewed (/view/agents/Hoken+Isikesu and /view/items/Dragon+F.O.F.+Cruise+Missile+I) sometimes included a period (.). When Rails 1.2 came about it introduced the respond_to method that can be used to handle requests for different types of representations for the same data:

RUBY:
  1. respond_to do |format|
  2. format.html
  3. format.js
  4. end

So /views/agents/1 will default to the HTML format and 1.xml will want the XML format.

So now the URL that looks like: /view/items/Dragon+F.O.F.+Cruise+Missile+I will want the /view/items/Dragon+F resource with the O.F.+Cruise+Missile+I format. Ooops. Bug!

Powered by WordPress