First the problem:
I'm writing a load test framework at work and I need to consume JSON webservice and pass on the output to another request. But I don't always know what the data is that I need to pass on but I do know the basic "path" ("hpath" -- "hash path") to get to the data as the JSON data is uniform.
Now the method:
RUBY:
-
# Picks out data from a (JSON) decoded hash based on the @passon hash,
-
# which looks like this:
-
# { "id" => "packet.products[0].attributes.id",
-
# "quantity" => "packet.products[0].attributes.quantity"
-
# }
-
# The "id" and "quantity" are the new keys for the return data;
-
# packet.products[0].attributes.id will look at the value of the id key
-
# in the attributes hash in the 0th element of the products array in the
-
# packet hash.
-
def pick_out_passon(hash_data)
-
return {} unless @passon
-
returnhash = {}
-
nh = hash_data.dup
-
@passon.each do |newkey,part_str|
-
parts = part_str.split(".").reverse
-
while (part = parts.pop) do
-
m = part.match(/\[(\d+)\]/)
-
index = nil
-
if m
-
index = m[1].to_i
-
part.gsub!(/\[#{m[0]}\]/,'')
-
end
-
nh = nh.values_at(part).first
-
nh = nh.at(index) if index
-
end
-
returnhash.merge!({ newkey=> nh})
-
nh = hash_data.dup
-
end
-
returnhash
-
end
What's it do? It does magic!
I'll step through it...
Looks for @passon instance variable and doesn't do anything useful unless it exists.
Duplicate the input hash because the process done is destructive to it and we may need to reuse it.
For each new hash key and "hpath" pair from @passon split up the hpath into its parts and reverse it so Array#pop will work in the while loop to get the next first part to try.
Since each part of the hpath can examine an Array by index it has to be checked for and the index removed from the part (and saved).
Next, investigate the copy of the hash_data, nh by the computed key; if the value in the hash was an Array use the index to get the desired value. Then compute the next part!
Once we're out of parts stuff the new key and data into the returnhash and keep going til there's no more @passon pairs.
And thus some fun code was written.
One part of code optimisation that is relatively easy is to cache the results of expensive operations. It's really simple. CACHE.set. But it isn't so simple. There are a seemingly endless possible ways the cache can be used improperly or otherwise fail. Handling these problems is the real work of caching.
The last checkout (around rev 250?) I did of Cache_fu (the acts_as_cached rewrite) it was pretty immature, chiefly in our need: An easy way to do arbitrary key names. n.b., I'm aware that there is a way but it seemed against the flow of this codebase. I use it, still, for the magic :all keyname.
But I need more, to store created JSON (To create what results in 100KB of JSON is a lot of objects and a lot of create time!) with a dynamic key in several distinct situations.
That round of caching improved API calls from 3 req/s to 300 req/s.
For the application most of the data can be cached but it needs to be done smartly: memcache server failures need to be handled (I suspect they are not, currently), server layout needs to be better (One daemon on each app server is not good, long term, IMHO), expiration of caches on data changes across the entire farm need to be smarter, deciding how long to cache data needs to be done.
CACHE.set and CACHE.get is really misleading; there's a lot of things to consider!
I relish these challenges and look forward to solving them.
At work yesterday I found out that by default ActiveSupport::JSON.unquote_hash_key_identifiers is true. This means that a JSON string looks like:
{attributes: {name: "0-3VW8 I", coordinate_id: "6229", system_id: "30000995", id: "40063299", igbtype: "7"}}
Flash JSON decoders (and probably others) expect the above attributes word to be doublequoted. This is achieved (after much Googling) by setting ActiveSupport::JSON.unquote_hash_key_identifiers=false.
But it doesn't stop there. In the above JSON output the IDs are quoted, that means they're encoded as strings and not integers. This is against the spec.
This means that
RUBY:
-
ActiveSupport::JSON.unquote_hash_key_identifiers = false
-
planet = Planet.find :first
-
print planet.to_json # => {"attributes": {name: "0-3VW8 I", coordinate_id: "6229", system_id: "30000995", id: "40063299", igbtype: "7"}}
The integers should not be double quoted.
With the help of lifo from #rails-contrib the problem seems to have been narrowed down to instance_values/instance_variable_get borking things up. From my poking around (n.b., I'm not all that experienced with the source for Rails itself! I may be wrong) the issue seems to lay within vendor/rails/activesupport/lib/active_support/json/encoders/object.rb. This bit of code seems to work for AR::Base derived models:
RUBY:
-
class ActiveRecord::Base
-
def to_json
-
ret = ActiveSupport::JSON.unquote_hash_key_identifiers ? "{attributes: " : '{"attributes": '
-
ret + self.attributes.to_json + '}'
-
end
-
end
Edit (10 July 2007): It seems that this isn't exactly a drop-in replacement fix: If a model has associations loaded (Foo.find :first, :include => :bars) the above JSON-fixing snippet will not work because it looks at attributes only and not the loaded associations. I'll have to work around that.
