net-http-persistent 1.1
Eric Hodel | Wed, 19 May 2010 03:56:11 GMT
Persistent connections using Net::HTTP plus a speed fix for 1.8. It’s thread-safe too!
Changes
Minor Enhancements
Proxy support, see Net::HTTP::Persistent::new, Net::HTTP::Persistent#proxy_from_env
Added name parameter to Net::HTTP::Persistent::new for separation of connection pools.
Added Net::HTTP::Persistent#shutdown so you can clean up after yourself
Net::HTTP::Persistent now suppresses “peer certificate won’t be verified in this SSL session” warning.
Bug Fixes
Net::HTTP::Persistent retries requests in accordance with RFC 2616.
RubyGems 1.3.7
Eric Hodel | Fri, 14 May 2010 04:55:53 GMT
rubygems-update version 1.3.7 has been released!
RubyGems is a package management framework for Ruby.
This gem is an update for the RubyGems software. You must have an installation of RubyGems before this update can be applied.
See Gem for information on RubyGems (or `ri Gem`)
To upgrade to the latest RubyGems, run:
$ gem update --system # you might need to be an administrator or root
NOTE: RubyGems 1.1 and 1.2 have problems upgrading when there is no rubygems-update installed. You will need to use the following instructions if you see “Nothing to update”.
If you have an older version of RubyGems installed, then you can still do it in two steps:
$ gem install rubygems-update # again, might need to be admin/root $ update_rubygems # ... here too
If you don’t have any RubyGems install, there is still the pre-gem approach to getting software, doing it manually:
Download from: rubyforge.org/frs/?group_id=126
Unpack into a directory and cd there
Install with: ruby setup.rb # you may need admin/root privilege
For more details and other options, see:
ruby setup.rb --help
Changes
NOTE:
rubygems.org is now the default source for downloading gems.
You may have sources set via ~/.gemrc, so you should replace gems.rubyforge.org with rubygems.org
gems.rubyforge.org will continue to work for the forseeable future.
New features:
`gem` commands
`gem install` and `gem fetch` now report alternate platforms when a matching one couldn’t be found.
`gem contents` —prefix is now the default as specified in —help. Bug #27211 by Mamoru Tasaka.
`gem fetch` can fetch of old versions again. Bug #27960 by Eric Hankins.
`gem query` and friends output now lists platforms. Bug #27856 by Greg Hazel.
`gem server` now allows specification of multiple gem dirs for documentation. Bug #27573 by Yuki Sonoda.
`gem unpack` can unpack gems again. Bug #27872 by Timothy Jones.
`gem unpack` now unpacks remote gems.
—user-install is no longer the default. If you really liked it, see Gem::ConfigFile to learn how to set it by default. (This change was made in 1.3.6)
RubyGems now has platform support for IronRuby. Patch #27951 by Will Green.
Bug fixes:
Require rubygems/custom_require if —disable-gem was set. Bug #27700 by Roger Pack.
RubyGems now protects against exceptions being raised by plugins.
rubygems/builder now requires user_interaction. Ruby Bug #1040 by Phillip Toland.
Gem::Dependency support #version_requirements= with a warning. Fix for old Rails versions. Bug #27868 by Wei Jen Lu.
Gem::PackageTask depends on the package dir like the other rake package tasks so dependencies can be hooked up correctly.
Net::HTTP is not slow
Eric Hodel | Fri, 07 May 2010 08:56:00 GMT
You're just using it wrong.
Some time back there was a blog post about Net::HTTP being slow, but that's not true anymore, and probably wasn't as true then as it was claimed to be.
The way to make Net::HTTP go fast is to use a persistent connection so you don't have to re-connect to the server every time. Unfortunately the original benchmarks referenced above don't seem to make more than one request per implementation so Net::HTTP couldn't give its best possible showing.
If you're doing a one-off file transfer or only fetching content from one site at a time it's ok to avoid Net::HTTP for another library. If you're requesting data from the same server over and over, like a web service, it's nearly immoral to connect to it over and over.
In order to help you use Net::HTTP the right way I've released net-http-persistent. It's a thread-safe wrapper for Net::HTTP that performs persistent connections for you. Here's an example:
require 'net/http/persistent'
uri = URI.parse 'http://example.com/awesome/web/service'
http = Net::HTTP::Persistent.new
response = http.request uri # performs a GET
# perform a POST
post_uri = uri + 'create'
post = Net::HTTP::Post.new post_uri.path
post.set_form_data 'some' => 'cool data'
response = http.request post_uri, post # URI is always requirednet-http-persistent is incredibly tiny, so maybe you can add some convenience methods to it. I haven't had a need to.
Benchmark
I wrote the following three benchmark blocks to return the same request body for a URL I’m sure will work (return 200 OK with a payload). A static file was used to minimize server processing latency.
Each iteration:
sends an HTTP request
cleans up after itself (to be friendly to the network)
extracts the body
Loopback
When running across loopback with all three benchmarks I received the following result with N=20_000 using uri_2k:
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 1.330000 2.130000 3.460000 ( 9.601254)
Net::HTTP 8.410000 2.400000 10.810000 ( 17.333671)
Net::HTTP::Persistent 8.110000 0.880000 8.990000 ( 12.190094)
----------------------------------------------- total: 23.260000sec
user system total real
TCPSocket 1.340000 2.160000 3.500000 ( 9.759389)
Net::HTTP 8.390000 2.370000 10.760000 ( 17.381197)
Net::HTTP::Persistent 8.070000 0.880000 8.950000 ( 11.493741)
With N=50_000 and the Net::HTTP benchmark disabled:
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 3.290000 5.340000 8.630000 ( 24.503025)
Net::HTTP::Persistent 20.090000 2.160000 22.250000 ( 28.822468)
----------------------------------------------- total: 30.880000sec
user system total real
TCPSocket 3.290000 5.340000 8.630000 ( 23.874741)
Net::HTTP::Persistent 20.100000 2.150000 22.250000 ( 29.188237)
So raw TCPSocket is about 20% faster than Net::HTTP::Persistent.
This was expected as the initial connection setup and teardown round-trips will be very fast on the loopback interface which gives Net::HTTP::Persistent the worst-possible showing.
Unfortunately you miss out on easy error checking and all that other Net::HTTP and Net::HTTP::Persistent goodness using TCPSocket.
Real Internet
Depending upon your link speed, creating TCPSockets across the Real Internet may drastically reduce the performance of TCPSocket.
This benchmark was run with N=500 from my home internet connection and uri_2k. traceroute shows 16 hops between the client and server. At the time of the benchmark run ping -c 20 showed:
20 packets transmitted, 19 packets received, 5.0% packet loss
round-trip min/avg/max/stddev = 74.564/91.412/147.863/18.092 ms
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 0.180000 0.220000 0.400000 ( 99.048004)
Net::HTTP::Persistent 0.340000 0.120000 0.460000 ( 46.229385)
------------------------------------------------ total: 0.860000sec
user system total real
TCPSocket 0.210000 0.280000 0.490000 (112.646966)
Net::HTTP::Persistent 0.340000 0.140000 0.480000 ( 47.381403)
In this case Net::HTTP::Persistent is about 140% faster than TCPSocket.
Running this benchmark
The data files I used were created by dd:
dd if=/dev/zero of=~/Sites/zeros-1k bs=1024 count=1
If you’re running this benchmark repeatedly make sure you wait until the sockets fall out of TIME_WAIT before re-running, you should see 0 (or near 0):
netstat -an | grep TIME | lc
TCPSocket and Net::HTTP::Persistent should show similar times on a fast link (like loopback). If TCPSocket ends up vastly slower you’ve probably run out of sockets.
When running this benchmark with high N you may need to increase the ephemeral port range.
With an N of 50_000 and the following configuration I can run the TCPSocket or the Net::HTTP requests along with Net::HTTP::Persistent, but not both.
$ sysctl -a net.inet.ip.portrange net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 10000 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.hifirst: 10000 net.inet.ip.portrange.hilast: 65535
What about Curb?
I tried to write a benchmark using curb 0.7.1 but failed to make one that performed even as well as plain Net::HTTP.
I couldn’t get curb to use a persistent connection. curl_easy_perform(3) says that libcurl will create a persistent connection if you call it multiple times with the on the same handle. I can see this behavior using `strace curl URL URL`.
With curb I see a new socket created per sendto(2)/recvfrom(2) pair. I also see a bunch of calls to close(2) when ruby performs its final GC pass.
I couldn’t see a way to make curb shut down its socket manually. The only way to do this is to wait for the GC to collect the socket. Leaving file descriptors hanging around for the GC is not good. (It also seemed to spend most of the time in the benchmark waiting for sockets to close.)
I started looking through curb to see why it would behave this way, but in Curb::Easy::new it calls curb_easy_init(3) and doesn’t check the return value despite the man page saying it may return NULL and gave up.
I filed the issues 29, 30 and 31 on the curb github tracker for these problems instead.
The Code
require 'rubygems'
require 'benchmark'
require 'net/http'
require 'net/http/persistent'
uri_1k = URI.parse 'http://localhost/~drbrain/zeros-1k'
uri_2k = URI.parse 'http://localhost/~drbrain/zeros-2k'
uri_10k = URI.parse 'http://localhost/~drbrain/zeros-10k'
uri = uri_2k
N = 5_000
Benchmark.bmbm do |bm|
bm.report 'TCPSocket' do
# HTTP/1.1 requires handling of chunked transfer-encoding
tcp_request = <<-HTTP
GET #{uri.request_uri} HTTP/1.0\r
Host: #{uri.host}\r
Connection: close\r
\r
HTTP
N.times do
s = TCPSocket.open uri.host, uri.port
s.write tcp_request
data = s.read
s.close # hopefully reduces TIME_WAIT duration
data.split("\r\n\r\n", 2).last # get body
end
end
bm.report 'Net::HTTP' do
N.times do
response = nil
Net::HTTP.start uri.host, uri.port do |http|
# Net::HTTPRequest can't be recycled
request = Net::HTTP::Get.new uri.request_uri
response = http.request request
end
response.body
end
end
bm.report 'Net::HTTP::Persistent' do
http_p = Net::HTTP::Persistent.new
N.times do
response = http_p.request uri
response.body
end
end
endRuby 1.8.6 Policy
Eric Hodel | Fri, 23 Apr 2010 17:48:15 GMT
Ruby 1.8.6 is old and it's API is lacking the forward-compatibility that Ruby 1.8.7 has for moving to Ruby 1.9. Since I maintain two large ruby libraries that are shipped in 1.9 (RDoc and RubyGems) it is becoming hard to maintain 1.8.6 support inside them comfortably.
Currently I do not develop against 1.8.6 and correct operation on 1.8.6 is the last thing I investigate before release. Ensuring that my software works well in three versions of Ruby (1.8.6, 1.8.7 and 1.9.1) has become difficult. Supporting it in four once 1.9.2 is out will be too hard.
Ruby 1.8.7 offers many of the features of Ruby 1.9 making support for it fairly easy.
RubyGems 1.4.x will not support Ruby 1.8.6.
The RDoc 2.5.x releases do not guarantee support for Ruby 1.8.6.
Sorry Ruby 1.8.6, it's time to let you go.
rdoc-data 2.5.1
Eric Hodel | Thu, 01 Apr 2010 08:15:17 GMT
rdoc-data contains core ri data for use with RDoc 2.5 To install rungem install rdoc-data followed by rdoc-data.
This will allow you to look up RDoc for core and standard library files like Kernel, Array or Date.
rdoc-data contains ri information for Ruby 1.8.6, 1.8.7 and 1.9.1.
rdoc 2.5
Eric Hodel | Thu, 01 Apr 2010 06:18:55 GMT
rdoc version 2.5 has been released!
RDoc is an application that produces documentation for one or more Ruby source files. RDoc includes the rdoc and ri tools for generating and displaying online documentation.
At this point in time, RDoc 2.x is a work in progress and may incur further API changes beyond what has been made to RDoc 1.0.1. Command-line tools are largely unaffected, but internal APIs may shift rapidly.
See RDoc for a description of RDoc’s markup and basic use.
Changes:
NOTE:
You’ll need to:
gem install rdoc-data
then run:
rdoc-data
to have ri data for core and stdlib like Array or Kernel or Date.
9 Major Enhancements
Darkfish now has a “Home” button
ri no longer displays the value of a constant. There’s no easy way to make them presentable. Use irb or ruby -e instead. Ruby Bug #549.
New ri data format now uses Marshal and pre-builds caches
No support for old ri data format, too hard to maintain
To upgrade your core ri documentation, install the rdoc-data gem and run rdoc-data
RDoc now displays how well you’ve documented your library
New recursive-descent parser for RDoc::Markup. See RDoc::Markup::Parser
Updated ruby_lex and ruby_token
Removed threading support, RDoc is not thread-safe
Removed many unsupported options to rdoc
Future versions of RDoc will not support Ruby 1.8.6. Bugs filed for 1.8.6-only issues will be (largely) rejected.
17 Minor Enhancements
Source Parsing
RDoc now supports module aliasing via constant assignment.
RDoc now tracks superclasses correctly. Fixes File < IO for core docs.
RDoc now ignores methods inside methods.
RDoc now ignores Marshal and other binray files.
Removed "Skipping require of dynamic string" warning.
C parser now handles Document-method better. Bug #27329.
API enhancements for writing parsers like the Ruby parser, see RDoc::Parser::RubyTools
ri
Uses pager over less and more for Debian. Ruby Bug #1171.
ri will use the RI_PAGER environment variable to find a pager.
ri data generator now supports SIGINFO (^T)
When rdoc is in debug mode, ^C now prints a backtrace
RDoc::Markup::AttributeManager no longer uses global state.
RDoc::RDoc no longer passes around options. Patch #27167.
Darkfish won’t generate a file if its template is missing. Patch #25857.
Improved some wording for the RDoc main page. Patch #27264, #27268.
Removed diagram generation support (to return in the future).
Removed external support for RDoc::Task.
12 Bug Fixes
The :attr: directives now use the name given to create an attribute. See RDoc::Parser::Ruby#parse_meta_attr.
Fix crossrefs on paths with ’-’. Ruby Bug #883.
Fix ruby parser for alias with = in the name. Bug #27522.
Images are no longer executable. Bug #27156.
--op is no longer overridden by --ri. Bug #27054.
:method: now works when at the end of a class. Bug #26910.
Preserve elipsis from call-seq in Darkfish. Patch #26974.
Emacs-style coding: is handled properly. Patch #27388.
RDoc::RubyLex now parses UTF-8 identifiers. Bug #26946, #26947.
Fixed namespace lookup rules. Bug #26161.
Worked around bug in Selenium where they hide a .jar in a .txt file. Filed Selenium bug #27789.
Alias comments are no longer hidden. Reported by Adam Avilla.
ar_mailer 1.5.0
Eric Hodel | Fri, 16 Oct 2009 00:02:05 GMT
ar_mailer is a two-phase delivery agent for ActionMailer. Even delivering email to the local machine may take too long when you have to send hundreds of messages. ar_mailer allows you to store messages into the database for later delivery by a separate process, ar_sendmail.
Changes
- Added --pid-file option. See --help for details.
- Minor improvements to --help, argument error output
dnssd 1.3.1
Eric Hodel | Wed, 14 Oct 2009 03:05:58 GMT
dnssd version 1.3.1 has been released!
DNS Service Discovery (aka Bonjour, MDNS) API for Ruby. Implements browsing, resolving, registration and domain enumeration. Supports avahi's DNSSD compatibility layer for avahi 0.6.25 or newer.
Changes
- 1 bug fix
- Fix regexp for parsing the fullname. Patch by James Sadler.
imap_to_rss 1.1
Eric Hodel | Tue, 01 Sep 2009 22:47:00 GMT
imap_to_rss version 1.1 has been released!
IMAPToRSS turns messages on an IMAP server into RSS entries when the match a handler. Included handlers work for email from Amazon, HSBC and UPS. IMAPToRSS automatically loads handlers for any other mail.
Changes:
- 1 major enhancement
- Added iTunes handler
- 1 bug fix
- Fixed various bugs in the Amazon handler
dnssd 1.3
Eric Hodel | Wed, 19 Aug 2009 04:31:00 GMT
DNS Service Discovery (aka Bonjour, MDNS) API for Ruby. Implements browsing, resolving, registration and domain enumeration. Supports avahi’s DNSSD compatibility layer for avahi 0.6.25 or newer.
Changes
- 4 major enhancements
- Added DNSSD::Service.get_property
- Added DNSSD::Service#getaddrinfo
- Added DNSSD::Service#add_record
- Added DNSSD::Service#query_record
- 5 minor enhancements
- DNSSD::Reply#connect now uses DNSSD::Service#getaddrinfo (faster)
- DNSSD::Service#register behaves properly when blockless
- Broke up DNSSD::Reply into specific subclasses
- Added sample/query_record.rb
- sample/*.rb work with each other now for clarity of implementation
- 2 bug fixes
- Fix hierarchical domains like \.mac\.name.members.mac.com.
- Fix compilation against avahi 0.6.25
Articles