This whole bulk updating system I did a while ago works well, except it’s very slow with a single mongrel. That problem could be solved with a simple proxy in front of many mongrels and a threaded updater application. For mass schema changes, for now, the easiest method is to dump from the CCP into my schema. I may be crazy but I’m not that crazy.
Things are coming to a close withthe RESTification of evedb. Next action once the RESTification is done is to get the site a proper design! I’ve got a sort of mental road map for evedb and so far I’m at what equates to “version 0.3″. Long way to go but it’s a fun ride. I think I need a vapourware release…
DRY CRUD!
That is, of course, "Don't repeat yourself".
With my evedb.info site I have to essentially rsync two databases whilst mangling schema. The process will take FOREVER (Mainly because I'm not multithreading this process) but it's automated.
I kept having to write CRUD methods in all of my controllers to deal with updating evedb.info as I talked about in this linked post and they were ALL the same. So being the lazy programmer I am I put them all in the ApplicationController:
Here's the update method
RUBY:
-
def update
-
singular = params[:controller].singularize
-
camel = singular.camelize
-
klass = camel.constantize
-
@temp = klass.new params[singular]
-
@real = klass.find params[:id]
-
changes = object_diff(@real,@temp)
-
logger.info "Changes: #{changes.inspect}"
-
if changes.keys.size == 0
-
render :nothing => true, :status => :ok
-
return
-
end
-
if @real.update_attributes(changes)
-
render :nothing => true, :status => :ok
-
else
-
render :json => @real.errors.to_json, :status => 400
-
end
-
end
By abusing Rails's constantize I can mangle the controller's name ("corporations") into the model it's handling: Corporation and treat it klass as if it was actually spelled Corporation. Personally I think that is very cool.
object_diff is a method I wrote to find the differences between two ActiveRecord::Base derrived objects. Loop through attributes hash and note things that change. changes is those attributes that have changed.
The only part I haven't been able to DRY out is the index method which always returns YAML for my scripts but only the IDs of the objects.
RUBY:
-
if params[:format] != 'yaml'
-
@corporations = Corporation.find normal finder
-
end
-
respond_to do |format|
-
format.html
-
format.yaml { render :text => Corporation.find(:all, :select => 'id', :order => 'id').to_yaml }
-
end
But I don't think I can DRY it out becase Corporation YAML finder actually selects another column that my script uses. Oh well.
Aside from having my hair cut today I wrote a Ruby library for the Eve Online API. I also wrote the above page to act as the ad-hoc home for the library until something more formal can be arranged.
It's still early stages (and I forgot to remove my old API key from reve.rb!) but it seems to work. Have a look and post some feedback!
Just finished registering for RailsConf Europe this morning. Travel is booked. Officially I am going on behalf of my work, unofficially I'm going to fangirl next to Rails's niftiest people.
More later!
Following up to this post of mine. The schema can be found here and the full thing (including Schema) can be found here.
Yesterday at work we discovered that Actionscript's HTTPService library is extremely crippled in its functionality. How bad is it? Well dear reader it's common practice when writing a webserver or webservice that one should return 404 Not Found if a resource can't be found or 201 Created when a resource has been created. The other status codes also have their usefulness. In my API project for work I believe I use (only for the non-HTML access): 200, 201, 400, 401, 403, 404, 409, and 500. Quite a rich set. I also use the POST, GET, PUT, DELETE verbs.
HTTPService, however, was built in such a way that anything other than 200 OK means "Raise an exception because a fault happened" - including 201 Created (a successful response code!). HTTPService does not permit anything other than GET and POST. Good thing that Rails permits the _method hidden field, eh?
What I am most likely going to have to do is to do the same sort of thing I would have had to do if the API was done with SOAP: Always return 200 OK and maintain a set of "response objects" to cope for shortcomings in other technoloties. Instead of a standard:
RUBY:
-
render :json => @product.to_json, :status => :created
I'll probably have to do:
RUBY:
-
render :json => { :product => @product, :status => ErrorObject.new(201) }.to_json, :status => :ok
And I'll have to do that in every place. REST? What's that. Argh. Anyone know of a decent Actionscript HTTP library?
In my quest to RESTify Evedb.info I keep finding more reasons to fawn ove REST (and CRUD): With the most recent Eve Online Database Dump for Revelations I have a need to migrate the data to my own schema. I also need to treat the database dump as a "Master" copy of what should appear in my site; stale data must be removed from my database.
I like to automate things. Prior to the RESTification efforts I would pump out raw SQL files and import them into Postgres in a batch. The SQL took about 45 minutes to import and was often held up due to foreign key issues.
Anyways, with REST I can funnel all of that through my controllers which have that functionality anyways (albeit heavily secured so normal users can't use this CUD API). I can have the same standard method for updating my version of CCP's data.
CCP has a lot of extra data in its Database dump (read: I don't use every column from every table). When I felt the need to import one of those I had to mess about with SQL files. Evedb.info was not using migrations! The SQL files were too big to be under source control. And a better solution was needed.
Enter: Evedb::Updater.
I wrote a class today to use the libcurl libraries for Ruby since the native Ruby Net::HTTP libraries were giving me trouble with the encoding of POST data. The general structure of the class I wrote is the following:
- Initalise the environment (Create a Postgres connection, Curl object, and store the base URL)
- Get a list of all of the IDs of the current model I'm dealing with (custom controller method all_ids since I have index paginated)
- Process an SQL query designed to do the schema change (
SQL:
-
SELECT agentid AS id FROM agtAgents
) for the specific model. This results in a list of things to create, update and destroy
- Do the deletions
- Do the creates
- Do the updates
It was a fun bit of code to write! The highlight is probably this bit:
RUBY:
-
module Hash
-
def to_curl
-
[] if self.keys.empty?
-
ret = []
-
self.keys.each do |k|
-
self[k].each do |v,l|
-
ret <<Curl::PostField.content("#{k.to_s}[#{v.to_s}]",l.to_s)
-
end
-
end
-
ret
-
end
-
end
The hash is generated by
RUBY:
-
0.upto(sql_results.num_tuples - 1) do |i|
-
m = { type => {} }
-
sql_results.fields.each do |field|
-
m[type].merge!( { field.to_sym => sql_results.getvalue(i,sql_results.fieldnum(field)).rstrip.fixmscrap.gsub(/'/,'\\\\\'') } )
-
end
The result of the to_curl method is to make the POST data that how we rails folk like it! (agent[name]=foo).
One downside to this is that some models need to query the Rails database (not the CCP one) for information to use. I might be better off modifying those specific models to use virtual attributes and some clever use of before_create!
Last night I got my hands on the latest db dump and converted it to a Postgres format. Currently the "online version" (basically a PHP script to sort all the info!) is online for browsing for those who may be interested.
I think I'll pg_dump and pop the resulting SQL on Eve files so others can have it.
Just over four years ago I was given CVS commit access to the Gentoo portage tree. It's been quite a dramatic time and it's been very enlightening. The people I've met have been really nifty! Most of my the things that I maintain are stable. My main regret is that due to work completely sucking up my time I don't have much left over at the end of the day for Gentoo.
At work yesterday I found out that by default ActiveSupport::JSON.unquote_hash_key_identifiers is true. This means that a JSON string looks like:
{attributes: {name: "0-3VW8 I", coordinate_id: "6229", system_id: "30000995", id: "40063299", igbtype: "7"}}
Flash JSON decoders (and probably others) expect the above attributes word to be doublequoted. This is achieved (after much Googling) by setting ActiveSupport::JSON.unquote_hash_key_identifiers=false.
But it doesn't stop there. In the above JSON output the IDs are quoted, that means they're encoded as strings and not integers. This is against the spec.
This means that
RUBY:
-
ActiveSupport::JSON.unquote_hash_key_identifiers = false
-
planet = Planet.find :first
-
print planet.to_json # => {"attributes": {name: "0-3VW8 I", coordinate_id: "6229", system_id: "30000995", id: "40063299", igbtype: "7"}}
The integers should not be double quoted.
With the help of lifo from #rails-contrib the problem seems to have been narrowed down to instance_values/instance_variable_get borking things up. From my poking around (n.b., I'm not all that experienced with the source for Rails itself! I may be wrong) the issue seems to lay within vendor/rails/activesupport/lib/active_support/json/encoders/object.rb. This bit of code seems to work for AR::Base derived models:
RUBY:
-
class ActiveRecord::Base
-
def to_json
-
ret = ActiveSupport::JSON.unquote_hash_key_identifiers ? "{attributes: " : '{"attributes": '
-
ret + self.attributes.to_json + '}'
-
end
-
end
Edit (10 July 2007): It seems that this isn't exactly a drop-in replacement fix: If a model has associations loaded (Foo.find :first, :include => :bars) the above JSON-fixing snippet will not work because it looks at attributes only and not the loaded associations. I'll have to work around that.
