If you are using the Apt cookbook for Chef, and you’re getting a “Parent directory /var/chef/cache does not exist.” error, make sure the “apt” (default recipe) is in your list of recipes.
Rails antipatterns
This unfinished post hails from 2013. Is any of it still true? No idea.
accepts_nested_attributes_for (use view models)
instance variables in the application controller – every controller and every view can see it. There’s no great place to put an object that lives for the whole request (in the request? in the response? in an object temporarily in the config?) People like Mongoid’s IdentityMap end up using static methods with thread locals for thread safety… that’s not a great way to do it.
Error messages put together in models, and i18n as the only alternative – Many people construct error messages directly in the model. In the best case, they may use the built-in i18n support to provide parameters to be substituted into a phrase. However, the model is still responsible for formatting the objects as appropriate in order to be substituted into the i18n string. If, for example, you need to include a list of objects in the error which should then be formatted to a more complex string (for example, a link to the article), it is not easy to make use of a view or helper to perform that functionality for you.
Warp: Multi-Key Transactions for Key-Value Stores
Interesting paper linked from YCombinator recently: http://hyperdex.org/papers/warp.pdf
Private static methods in Ruby?
Try making a private static method in Ruby and using it from an instance of the same class… wihout goofy stuff like send(). As far as I can tell, it’s impossible.
RSpec mock.with(Hash) expects an Array
The following problem, where RSpec would erroneously fail mock parameter checks in specs, has been fixed in either RSpec 2.13.0 or 2.13.1
RSpec was misinterpreting hash argument expections as arrays. If your expectation was “mock.with(a: 1, b: 2)”, you’d get:
RSpec::Mocks::MockExpectationError: <MyClass (class)> received :my_method with unexpected arguments expected: ({:a => 1, :b => 2}) got: ([:a, 1], [:b, 2])
If you tried wrapping your arguments in a hash “mock.with({a: 1, b: 2})” you’d get the very confusing:
expected: ([:a, 1], [:b, 2]) got: ([:a, 1], [:b, 2])
Luckly, as mentioned, this is fixed in the latest version. Update if possible!
Scaleable (threadsafe) Accumulators in JDK8
Found this on a social site today. Looks like JDK8 is getting some pretty cool-looking accumulators that are threadsafe and performant: http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/f21a3c273fb2
Reducers a la Rich Hickley in Ruby
This is a draft of an article which translates Rich Hickley’s “reducers” concept into the Ruby language.
http://clojure.com/blog/2012/05/08/reducers-a-library-and-model-for-collection-processing.html
collect, aka map, does:
1. Recursion (apply function to head, then call map recursively on the tail)
2. Consumes an ordered list (guarantees order of execution)
3. Produces an ordered list
reduce, as currently provided in Clojure, is ignorant of the collection structure & allows the collection to implement the coll-reduce protocol. Also, it can produce anything: an aggregate, a map, etc. Still inherently sequential (monotonic) in the order of the operations (left fold with seed).
Don’t create a new collection when we call map or filter… don’t add your result to the collection inside the reduce…
How to produce a result? A Reduction Transformer.
We want to be able to create a recipe for creating pies, maybe even before we have a bag of apples. This implies being able to compose multiple reductions. Composing might look something like: map(take_sticker_off_apple).map(reject_bad_apples)
reducing function: function you pass to reduce… a binary operator, such as addition. Arguments don’t have to be symmetric: order is important. First arg is seed/result/accumulator… second arg is input to process that’s building the result.
mapping a function: you have a reducing function
//f1 is the reducing function (a binary operator, etc. see above) // f(input) is "mess with that input" mapping(f) { // this is a reduction transformer ->(f1) { ->(result, input) { f1.call(result, f.call(input) } } do_stuff_to_input(input) { input.label = nil return input } filter_mapping = mapping(do_stuff_to_input) #we have bound f reducing_function(result, input) { return result + input } modified_reducing_function = filter_mapping.call(reduction_transformer) #we have bound f1 bag_of_apples.reduce(modified_reducing_function)
Mapping case: 1->1 case
Filtering case: 1->1/0
filtering(predicate_fn) { // this is a reduction transformer ->(reducing_fn) { ->(result, input) { if(predicate_fn.call(input)) { return reducing_fn.call(result, input) else return result } } } } bag_of_apples.reduce( filtering(is_good_apple?) //predicate_fn is bound .call(array_concat) //reducing_fn is bound )
takes a reducing function, returns one, and if predicate is true call reducing function on the result with the new input. What if predicate false? Just don’t touch the result. This is the right way to write filter.
One to many case: 1->M eg. mapcat
mapcatting(f) { ->(reducing_fn) { ->(result, input) { reduce(reducing_fn, result, f.call(input)) } } } bag_of_apples.reduce( mapcatting(slice_apple_into_pieces(5)).call(array_concat) )
There’s a lot of repeated stuff… we can make a macro to reduce it down to the minimum.
It’s also not pretty. Function that returns a function, for example.
Also, this is not the cake.
TO BE CONTINUED
Query languages in JSON? Wtf
MongoDB. Elasticsearch.
Custom query languages in JSON. Why?
This guy sums it up pretty well: http://smsohan.com/blog/2013/01/17/abusing-json/
Am I gonna have to embed a Coffeescript compiler in my program so that I can at least write Mongo queries that don’t make my eyes bleed?
Ruby HTTP request gets “end of file reached” after 60 seconds
Over the past couple days, I’ve been trying to figure out why long-running (due to large file upload & synchronous processing time) HTTP requests to our REST API are throwing “Faraday::Error::ConnectionFailed: end of file reached” after one minute.
The client is: Ruby 1.9.3-p194, Faraday 0.8.5 using the default Net::HTTP adapter, and Hyperclient 0.3.0.
The server is: Rails 3.2.12 on Ruby 1.9.3-p194, via Apache with Passenger.
The exception looks like:
Caught Faraday::Error::ConnectionFailed: end of file reached /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/protocol.rb:132:in `readline' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:2551:in `read_new' /usr/local/rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
The 60 second timing was rather mysterious for some time because:
- The timeout option of Net::HTTP in the client had no effect, and even if it did, I would expect to see a Timeout::Error, not an EOFError.
- Extending the Apache Timeout setting had no effect, even though Apache was sometimes logging the following error:
[ pid=6209 thr=140145517742048 file=ext/apache2/Hooks.cpp:808 time=2013-02-25 20:03:36.394 ]: Either the visitor clicked on the 'Stop' button in the web browser, or the visitor's connection has stalled and couldn't receive the data that Apache is sending to it. As a result, you will probably see a 'Broken Pipe' error in this log file. Please ignore it, this is normal. You might also want to increase Apache's TimeOut configuration option if you experience this problem often.
We were also seeing errors in Apache’s logs like:
[ pid=13250 thr=139854544902112 file=ext/apache2/Hooks.cpp:810 time=2013-02-22 20:19:49.359 ]: No data received from the back end application (process 27551) within 300000 msec. Either the backend application is frozen, or your TimeOut value of 300 seconds is too low. Please check whether your application is frozen, or increase the value of the TimeOut configuration directive.
That means Apache wasn’t hearing anything back from Passenger/Rails for 300 seconds. Sometimes, we’d also see several Ruby (Rails) processes running for long periods of time and using a large share of the CPU. Normally, Passenger only keeps one Ruby process around, but in this case Ruby processes were remaining and even being orphaned out of the passenger-status listing. I’m still not sure why those processes were getting orphaned (was their work really taking longer than 300 seconds?).
I was eventually able to get rid of the EOF problem by bypassing the Amazon Load Balancer (aka Elastic Load Balancer or ELB) in front of our EC2 instances, which finally allowed me to pinpoint the source of the problem. Browsing the internet eventually brought me to this thread: Elastic Load Balancer: HTTP Connections time out after 60 seconds. The potential solution mentioned in the thread is that Amazon can increase the timeout on your load balancer.
While I’m happy to have a possible solution, I’m not sure why the failure occurs exactly one minute after the request initiates. I would expect that sending the body of the request would take a significant amount of time (>30 seconds), during which the connection would have traffic and therefore the ELB timeout would not start ticking. However, the software always threw the EOF error after exactly one minute.
In any case, I’m hopeful that asking for an extension to the load balancer timeout will resolve the issue. I’ll try to remember to update this post once I verify the solution.
Determine century of two digit year based on proximity to another year
When writing software, it’s sometimes necessary to determine the value of a date which unfortunately uses only two digits to represent the year. Usually, it’s assumed that the year is somewhere in a specific 100-year range in order to use a cut-off date to decide whether the year is in the 1900’s or the 2000’s.
For example, from a stack overflow answer:
year4 = if year < cutoff "20" + year else "19" + year [/ruby] For example, 30 could be used as the cutoff value. Obviously, this assumes that you will never encounter the value "30" and need to interpret it as "2030" instead of "1930". If, on the other hand, you have another date available which is likely to be quite close to the date with the two-digit year (a reference date), it is possible to choose the century such that the date is closest to the reference date. The following code accomplishes the nearly the same task by determining the century of a two-digit year based on proximity to a reference year (not a date). [ruby] # Given a two-digit year, this method determines the absolute year (ie. the century) based on # proximity to a reference year. def interpret_two_digit_year(ref_year, two_digit_year) unless (0..99).include?(two_digit_year) raise "Invalid argument: year must be in range 0..99, year was #{two_digit_year}" end ref_nearest_century = ref_year.round(-2) ref_century_offset = ref_nearest_century - ref_year year_offset_from_ref = (((two_digit_year + ref_century_offset) + 50) % 100) - 50 ref_year + year_offset_from_ref end [/ruby] The approach is as follows:
- Translate the year into a new coordinate system where it is relative to the first year of the century nearest the reference year, instead of being relative to 0. The year will thereby be in the range (-50..149), relative to the reference year.
- Next, we will map the year to the range -50..50 relative to the reference year by first translating by +50 to bring the range to 0..199, modding by 100 to wrap the 100..199 back into the 0..100 range, and then reverting the initial translation by subtracting 50 to bring the range to the desired -50..50 range
- Lastly, add the resulting offset to the reference year to obtain the final result.
While the best approach is to cease using two-digit years entirely, this approach given above may be helpful in situations where two-digit years are unavoidable, and other nearby dates are available.