CrudVision - Lisa Seelye

Greetings dear readers.

The CCP guy (Garthagk) behind the Eve Online API we all use really has a hard task. As the API becomes more and more popular the stress on the webservers hosting the API and the database servers increases. Bossmen and bosswomen may cut the API from us to save resources (that would suck). It's up to us, the community, to do what we can to lessen the load on CCP's servers.

Garthagk provided us with a cacheduntil (Reve::API.cached_until in Reve). Up unil now Reve didn't privde any mechanism for really using this. Sure, you could use just the cached_until if it's just one character. But one of my hopes is that Reve will be used by operators of multi-user sites (like my vapourware evedb.info) that will aggregate many users' data. Cached_until isn't enough.

Enter the protected method Reve::API#compute_hash. (As of revision 22.)

After each API call Reve will compute a hash to represent that request. This will be available through the last_hash. This will tell you what the hash is after the call. But to respect the cached_until it's polite to check the current time against the "Last Call's" cached_until value. The last call will have a hash and that hash will be the key to the cached_until. Each API call now takes a hash for its parameters intead of the old style comma style. Treat the hash as a of transaction ID.

For a simple example let's get the alliances list:

For this simple call nothing has changed except the Reve library presents more info for the user. Store cached_until and last_hash in a database.

For the new parameter to get the hash:

A complex example:

Get the hash:

With luck this post is clear for Reve's users and all you guys will be kind to the API so we have it in the future.

Today I finally managed to migrate my Evedb.info SVN repository from an internal box (boron) to my repository at Dreamhost (Home to Reve, too). It is really simple to do: Just copy the local repository files from Subversion to the new repository base. This means the conf, dav, db, hooks and locks diretories.

The problem comes when one revision is copied (Call it revision 10) and is checked out to another machine (phosphorus), development is done, checked in (Dreamhost's Revision 11) and stupidly, development is also done on the local copy (boron) and checked in (Local revision 11). And just like that the migration away from a local file:// type.

It's a nightmare to sort but I've finally got it sorted and protected with an off-site source and SSH key file.

The moral is: Plan ahead, don't be stupid, and never make commits when you're really tired.

I'll admit it. I haven't gotten around to learning Capistrano 2 yet so many or all of these issues may be solved.

One of our clients at work has a very bizarre deployment environment. To be brief (without getting sacked) it involves svn replication, shared NFS directories and a dedicated deploy box. They also have this setup twice, once for production and once for a "staging" environment.

Since they copy our SVN tagged release to their own SVN repository it means the Capistrano script doesn't actually do the mirroring (yet?); that is done by the client. I wrote a helper bash script that will take the tag, environment (production / "staging") and a capistrano command to select the right capistrano script, set the right environment (although now that I think about it this is redundant now) and run the command.

But there has to be an easier way to "One Click" this with Capistrano 1.x. It is my task to find a recipe to do it!

The problems are, due to the convoluted nature of the deploy environment I may have to abuse roles or invent my own or hack Capistrano to add qualifications to roles (similar to :primary): The dedicated deploy box is the box to which the application is deployed (acts like an app server!) but it doesn't/shouldn't run any mongrels. In a sense it's the primary app server since it has the authoritative source that's shared to other app servers.

But it isn't! It's a nightmare! It needs to be better done. I may try to one-shot this: sh deploy.sh SomeTag production deploy_first or deploy_upgrade.

(n.b., the version of Beast we use may not be the most current. This post may not reflect what's in trunk.)

One of our clients from work uses Beast forum. Unfortunately last week it kind of died. Mongrel's logs were complaining that it couldn't allocate memory, the index (ForumsController#index) was taking upwards of 12 seconds to load...

The problem was SQL. Beast is very greedy when it displays the front page; it does a forums.posts.last.created_at to get the timestamp for when the last post was made. This is done for every top-level forum and it will destroy your site.

At about 21:30ish the previous night I disabled the site because my brain was mush and I couldn't focus enough to diagnose the problem to say nothing of finding a solution. The next day when I got to the office I spent the first 90 minutes of the day diagnosing and fixing the problem. I found that I could find the relevant data with a GROUP BY and ORDER BY with one query for all forums... Except the times were the FIRST post and not the LAST post.

So I spent the next two hours figuring out why MySQL didn't want to ORDER BY and formulating a query that was both quick, returned data for every forum and worked. This is the ugly thing that resulted:

This is fed into:

In the view we just substitute forum.posts.last with @last_posts[forum.id].first in all instances. This may have to bet tweaked for your setup because I'm not sure where Beast ends and client-specific stuff begins.

You see there's some parent_id things going on in the SQL. At work we made Beast forum nested. We do plan to make these changes available for others to use but it's just a matter of finding time to get rid of the client-specific modifications and merging to trunk.

This is one of the things that I'm proud about: the SQL query was a pain!

Whenever I need to make a change to data in a Rails-based table I very rarely use a GUI, in fact, the only time I have used a GUI is for Oracle (and we all know how I feel about that). Instead I favour the rails console.

For me I find it easier to use the console because it reinforces the ORM - an entry in the table is just an instance of a class that models that table. I find that Constellation.find(:all, :conditions => "faction_id = 0").each { |c| c.faction_id = nil ; c.save! } is more meaningful (and easier) than sorting by the value of that column and mass updating. Icky

When I need to construct weird SQL queries, however, I'll drop to the mysql or psql shells. GUIs are icky and their bulky interfaces tend to slow complex queries down. Of course, I'm a console geek so I'm slightly biased here.

Anyways, I cringe a little when I see people opening SQL GUIs to find table structure or to find data values and to change single values. But hey, different strokes for different folks!

This is good timing. I need to know about this stuff tomorrow to help a colleague out. Rails Polymorphic Associations were a bit mysterious to me and so I didn't use them when I first started with Rails but the time has come to put them to work.

First, the scenario:

A Player and a Corporation both may have many Wallet Balance Entries and a Wallet Balance Entry belongs to either a Player or a Corporation.

The Models are this:

And (just the Player model)

The relevant bits of the WalletBalanceEntry schema look like this:

Note that wallet_balance_owner_type and wallet_balance_owner_id match the name that the Player model specifies with :as => :wallet_balance_owner. The Corporation model will have the exact same line and it will work! Should Alliances come to have many wallet balance entries the Alliance model will have this same line as well.

Yesterday at work I deployed the 'Oracle Project' to the customer's staging server. The only hiccup was (I think) a problem with the renumbering of migrations that was coming from two branch merges and a merge into trunk. I apparently misnumbered one and a critical migration wasn't run. Oops.

I've noticed that at the end of rake db:migrate that Rails rebuilds the db/scema.rb file (nothing new there) except with Oracle each table takes about 45 seconds to process! We've got about 35 models! That's almost 25 minutes! What gives?

Anyways, Oracle is conquered and it is my bitch.

Finally Reve has a test suite.

Check out the source and just:

cd test
ruby reve_test.rb

I hope it works!

Download an archive for use:

• Reve Release 11
• Reve Release 11

At work I've been struggling with making one of our projects play nice with Oracle. I got Oracle XE installed through Windows XP in Parallels and just couldn't make my tests work. I kept getting errors that Rails couln't insert a nil value into a non-nil field (id).

What this eventually boiled down to is Rails fixtures MUST have an id attribute on them or you may run into errors! This was hidden behind the scenes since heretofore we were using auto_incrementing MySQL fields. To get this behaviour in Oracle one must do

(done by Rails automatically, normally) or set up a Trigger (not done by rails, and would have to be done in a create_table migration).

However, fixtures bypass the normal ActiveRecord::Base#create call and just does a raw execute insert into... Which means there's no value for id, unless it's specified in the fixture. This kept me at it for hours today and I only found it by hacking the fixtures source to make it print in the logs what it was doing!

Let this be a lesson! Always put in ids even if you don't care about their values.

It's fairly obvious that 5 tight knit people who know their shit will work better as a unit than 20 people who may perhaps not be up to level of an "expert" (to use the linked blog's lingo). Who doesn't dream of working exclusively with the rails core team?

Link to post