Net::HTTP is not slow
Eric Hodel | Fri, 07 May 2010 08:56:00 GMT
You're just using it wrong.
Some time back there was a blog post about Net::HTTP being slow, but that's not true anymore, and probably wasn't as true then as it was claimed to be.
The way to make Net::HTTP go fast is to use a persistent connection so you don't have to re-connect to the server every time. Unfortunately the original benchmarks referenced above don't seem to make more than one request per implementation so Net::HTTP couldn't give its best possible showing.
If you're doing a one-off file transfer or only fetching content from one site at a time it's ok to avoid Net::HTTP for another library. If you're requesting data from the same server over and over, like a web service, it's nearly immoral to connect to it over and over.
In order to help you use Net::HTTP the right way I've released net-http-persistent. It's a thread-safe wrapper for Net::HTTP that performs persistent connections for you. Here's an example:
require 'net/http/persistent'
uri = URI.parse 'http://example.com/awesome/web/service'
http = Net::HTTP::Persistent.new
response = http.request uri # performs a GET
# perform a POST
post_uri = uri + 'create'
post = Net::HTTP::Post.new post_uri.path
post.set_form_data 'some' => 'cool data'
response = http.request post_uri, post # URI is always requirednet-http-persistent is incredibly tiny, so maybe you can add some convenience methods to it. I haven't had a need to.
Benchmark
I wrote the following three benchmark blocks to return the same request body for a URL I’m sure will work (return 200 OK with a payload). A static file was used to minimize server processing latency.
Each iteration:
sends an HTTP request
cleans up after itself (to be friendly to the network)
extracts the body
Loopback
When running across loopback with all three benchmarks I received the following result with N=20_000 using uri_2k:
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 1.330000 2.130000 3.460000 ( 9.601254)
Net::HTTP 8.410000 2.400000 10.810000 ( 17.333671)
Net::HTTP::Persistent 8.110000 0.880000 8.990000 ( 12.190094)
----------------------------------------------- total: 23.260000sec
user system total real
TCPSocket 1.340000 2.160000 3.500000 ( 9.759389)
Net::HTTP 8.390000 2.370000 10.760000 ( 17.381197)
Net::HTTP::Persistent 8.070000 0.880000 8.950000 ( 11.493741)
With N=50_000 and the Net::HTTP benchmark disabled:
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 3.290000 5.340000 8.630000 ( 24.503025)
Net::HTTP::Persistent 20.090000 2.160000 22.250000 ( 28.822468)
----------------------------------------------- total: 30.880000sec
user system total real
TCPSocket 3.290000 5.340000 8.630000 ( 23.874741)
Net::HTTP::Persistent 20.100000 2.150000 22.250000 ( 29.188237)
So raw TCPSocket is about 20% faster than Net::HTTP::Persistent.
This was expected as the initial connection setup and teardown round-trips will be very fast on the loopback interface which gives Net::HTTP::Persistent the worst-possible showing.
Unfortunately you miss out on easy error checking and all that other Net::HTTP and Net::HTTP::Persistent goodness using TCPSocket.
Real Internet
Depending upon your link speed, creating TCPSockets across the Real Internet may drastically reduce the performance of TCPSocket.
This benchmark was run with N=500 from my home internet connection and uri_2k. traceroute shows 16 hops between the client and server. At the time of the benchmark run ping -c 20 showed:
20 packets transmitted, 19 packets received, 5.0% packet loss
round-trip min/avg/max/stddev = 74.564/91.412/147.863/18.092 ms
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin10.2.0]
Rehearsal ---------------------------------------------------------
TCPSocket 0.180000 0.220000 0.400000 ( 99.048004)
Net::HTTP::Persistent 0.340000 0.120000 0.460000 ( 46.229385)
------------------------------------------------ total: 0.860000sec
user system total real
TCPSocket 0.210000 0.280000 0.490000 (112.646966)
Net::HTTP::Persistent 0.340000 0.140000 0.480000 ( 47.381403)
In this case Net::HTTP::Persistent is about 140% faster than TCPSocket.
Running this benchmark
The data files I used were created by dd:
dd if=/dev/zero of=~/Sites/zeros-1k bs=1024 count=1
If you’re running this benchmark repeatedly make sure you wait until the sockets fall out of TIME_WAIT before re-running, you should see 0 (or near 0):
netstat -an | grep TIME | lc
TCPSocket and Net::HTTP::Persistent should show similar times on a fast link (like loopback). If TCPSocket ends up vastly slower you’ve probably run out of sockets.
When running this benchmark with high N you may need to increase the ephemeral port range.
With an N of 50_000 and the following configuration I can run the TCPSocket or the Net::HTTP requests along with Net::HTTP::Persistent, but not both.
$ sysctl -a net.inet.ip.portrange net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 10000 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.hifirst: 10000 net.inet.ip.portrange.hilast: 65535
What about Curb?
I tried to write a benchmark using curb 0.7.1 but failed to make one that performed even as well as plain Net::HTTP.
I couldn’t get curb to use a persistent connection. curl_easy_perform(3) says that libcurl will create a persistent connection if you call it multiple times with the on the same handle. I can see this behavior using `strace curl URL URL`.
With curb I see a new socket created per sendto(2)/recvfrom(2) pair. I also see a bunch of calls to close(2) when ruby performs its final GC pass.
I couldn’t see a way to make curb shut down its socket manually. The only way to do this is to wait for the GC to collect the socket. Leaving file descriptors hanging around for the GC is not good. (It also seemed to spend most of the time in the benchmark waiting for sockets to close.)
I started looking through curb to see why it would behave this way, but in Curb::Easy::new it calls curb_easy_init(3) and doesn’t check the return value despite the man page saying it may return NULL and gave up.
I filed the issues 29, 30 and 31 on the curb github tracker for these problems instead.
The Code
require 'rubygems'
require 'benchmark'
require 'net/http'
require 'net/http/persistent'
uri_1k = URI.parse 'http://localhost/~drbrain/zeros-1k'
uri_2k = URI.parse 'http://localhost/~drbrain/zeros-2k'
uri_10k = URI.parse 'http://localhost/~drbrain/zeros-10k'
uri = uri_2k
N = 5_000
Benchmark.bmbm do |bm|
bm.report 'TCPSocket' do
# HTTP/1.1 requires handling of chunked transfer-encoding
tcp_request = <<-HTTP
GET #{uri.request_uri} HTTP/1.0\r
Host: #{uri.host}\r
Connection: close\r
\r
HTTP
N.times do
s = TCPSocket.open uri.host, uri.port
s.write tcp_request
data = s.read
s.close # hopefully reduces TIME_WAIT duration
data.split("\r\n\r\n", 2).last # get body
end
end
bm.report 'Net::HTTP' do
N.times do
response = nil
Net::HTTP.start uri.host, uri.port do |http|
# Net::HTTPRequest can't be recycled
request = Net::HTTP::Get.new uri.request_uri
response = http.request request
end
response.body
end
end
bm.report 'Net::HTTP::Persistent' do
http_p = Net::HTTP::Persistent.new
N.times do
response = http_p.request uri
response.body
end
end
endRuby 1.8.6 Policy
Eric Hodel | Fri, 23 Apr 2010 17:48:15 GMT
Ruby 1.8.6 is old and it's API is lacking the forward-compatibility that Ruby 1.8.7 has for moving to Ruby 1.9. Since I maintain two large ruby libraries that are shipped in 1.9 (RDoc and RubyGems) it is becoming hard to maintain 1.8.6 support inside them comfortably.
Currently I do not develop against 1.8.6 and correct operation on 1.8.6 is the last thing I investigate before release. Ensuring that my software works well in three versions of Ruby (1.8.6, 1.8.7 and 1.9.1) has become difficult. Supporting it in four once 1.9.2 is out will be too hard.
Ruby 1.8.7 offers many of the features of Ruby 1.9 making support for it fairly easy.
RubyGems 1.4.x will not support Ruby 1.8.6.
The RDoc 2.5.x releases do not guarantee support for Ruby 1.8.6.
Sorry Ruby 1.8.6, it's time to let you go.
rdoc-data 2.5.1
Eric Hodel | Thu, 01 Apr 2010 08:15:17 GMT
rdoc-data contains core ri data for use with RDoc 2.5 To install rungem install rdoc-data followed by rdoc-data.
This will allow you to look up RDoc for core and standard library files like Kernel, Array or Date.
rdoc-data contains ri information for Ruby 1.8.6, 1.8.7 and 1.9.1.
rdoc 2.5
Eric Hodel | Thu, 01 Apr 2010 06:18:55 GMT
rdoc version 2.5 has been released!
RDoc is an application that produces documentation for one or more Ruby source files. RDoc includes the rdoc and ri tools for generating and displaying online documentation.
At this point in time, RDoc 2.x is a work in progress and may incur further API changes beyond what has been made to RDoc 1.0.1. Command-line tools are largely unaffected, but internal APIs may shift rapidly.
See RDoc for a description of RDoc’s markup and basic use.
Changes:
NOTE:
You’ll need to:
gem install rdoc-data
then run:
rdoc-data
to have ri data for core and stdlib like Array or Kernel or Date.
9 Major Enhancements
Darkfish now has a “Home” button
ri no longer displays the value of a constant. There’s no easy way to make them presentable. Use irb or ruby -e instead. Ruby Bug #549.
New ri data format now uses Marshal and pre-builds caches
No support for old ri data format, too hard to maintain
To upgrade your core ri documentation, install the rdoc-data gem and run rdoc-data
RDoc now displays how well you’ve documented your library
New recursive-descent parser for RDoc::Markup. See RDoc::Markup::Parser
Updated ruby_lex and ruby_token
Removed threading support, RDoc is not thread-safe
Removed many unsupported options to rdoc
Future versions of RDoc will not support Ruby 1.8.6. Bugs filed for 1.8.6-only issues will be (largely) rejected.
17 Minor Enhancements
Source Parsing
RDoc now supports module aliasing via constant assignment.
RDoc now tracks superclasses correctly. Fixes File < IO for core docs.
RDoc now ignores methods inside methods.
RDoc now ignores Marshal and other binray files.
Removed "Skipping require of dynamic string" warning.
C parser now handles Document-method better. Bug #27329.
API enhancements for writing parsers like the Ruby parser, see RDoc::Parser::RubyTools
ri
Uses pager over less and more for Debian. Ruby Bug #1171.
ri will use the RI_PAGER environment variable to find a pager.
ri data generator now supports SIGINFO (^T)
When rdoc is in debug mode, ^C now prints a backtrace
RDoc::Markup::AttributeManager no longer uses global state.
RDoc::RDoc no longer passes around options. Patch #27167.
Darkfish won’t generate a file if its template is missing. Patch #25857.
Improved some wording for the RDoc main page. Patch #27264, #27268.
Removed diagram generation support (to return in the future).
Removed external support for RDoc::Task.
12 Bug Fixes
The :attr: directives now use the name given to create an attribute. See RDoc::Parser::Ruby#parse_meta_attr.
Fix crossrefs on paths with ’-’. Ruby Bug #883.
Fix ruby parser for alias with = in the name. Bug #27522.
Images are no longer executable. Bug #27156.
--op is no longer overridden by --ri. Bug #27054.
:method: now works when at the end of a class. Bug #26910.
Preserve elipsis from call-seq in Darkfish. Patch #26974.
Emacs-style coding: is handled properly. Patch #27388.
RDoc::RubyLex now parses UTF-8 identifiers. Bug #26946, #26947.
Fixed namespace lookup rules. Bug #26161.
Worked around bug in Selenium where they hide a .jar in a .txt file. Filed Selenium bug #27789.
Alias comments are no longer hidden. Reported by Adam Avilla.
ar_mailer 1.5.0
Eric Hodel | Fri, 16 Oct 2009 00:02:05 GMT
ar_mailer is a two-phase delivery agent for ActionMailer. Even delivering email to the local machine may take too long when you have to send hundreds of messages. ar_mailer allows you to store messages into the database for later delivery by a separate process, ar_sendmail.
Changes
- Added --pid-file option. See --help for details.
- Minor improvements to --help, argument error output
dnssd 1.3.1
Eric Hodel | Wed, 14 Oct 2009 03:05:58 GMT
dnssd version 1.3.1 has been released!
DNS Service Discovery (aka Bonjour, MDNS) API for Ruby. Implements browsing, resolving, registration and domain enumeration. Supports avahi's DNSSD compatibility layer for avahi 0.6.25 or newer.
Changes
- 1 bug fix
- Fix regexp for parsing the fullname. Patch by James Sadler.
imap_to_rss 1.1
Eric Hodel | Tue, 01 Sep 2009 22:47:00 GMT
imap_to_rss version 1.1 has been released!
IMAPToRSS turns messages on an IMAP server into RSS entries when the match a handler. Included handlers work for email from Amazon, HSBC and UPS. IMAPToRSS automatically loads handlers for any other mail.
Changes:
- 1 major enhancement
- Added iTunes handler
- 1 bug fix
- Fixed various bugs in the Amazon handler
rubygems-isit19 1.0
Eric Hodel | Thu, 20 Aug 2009 05:26:00 GMT
Lets you figure out if your gems and gems you install might work on 1.9. Uses isitruby19.com as its datastore. Be sure to update the website with your experiences!
Features
- gem isit19 for checking your installed gems
- gem install plugin that tells you if your installed gem works on 1.9
Synopsis
$ gem install ZenTest ZenTest 4.1.4 might work, 100% say 4.1.3 works on 1.9 Update http://isitruby19.com/ZenTest with your experiences! Successfully installed ZenTest-4.1.4 1 gem installed
dnssd 1.3
Eric Hodel | Wed, 19 Aug 2009 04:31:00 GMT
DNS Service Discovery (aka Bonjour, MDNS) API for Ruby. Implements browsing, resolving, registration and domain enumeration. Supports avahi’s DNSSD compatibility layer for avahi 0.6.25 or newer.
Changes
- 4 major enhancements
- Added DNSSD::Service.get_property
- Added DNSSD::Service#getaddrinfo
- Added DNSSD::Service#add_record
- Added DNSSD::Service#query_record
- 5 minor enhancements
- DNSSD::Reply#connect now uses DNSSD::Service#getaddrinfo (faster)
- DNSSD::Service#register behaves properly when blockless
- Broke up DNSSD::Reply into specific subclasses
- Added sample/query_record.rb
- sample/*.rb work with each other now for clarity of implementation
- 2 bug fixes
- Fix hierarchical domains like \.mac\.name.members.mac.com.
- Fix compilation against avahi 0.6.25
imap_processor 1.3
Eric Hodel | Wed, 05 Aug 2009 02:38:30 GMT
IMAPProcessor is a client for processing messages on an IMAP server. It provides some basic mechanisms for connecting to an IMAP server, determining capabilities and handling messages.
IMAPProcessor ships with the executables imap_keywords which can query an IMAP server for keywords set on messages in mailboxes, imap_idle which can show new messages in a mailbox and imap_archive which will archive old messages to a new mailbox.
Changes
- IMAP IDLE support now matches ruby trunk’s support. See Net::IMAP#idle and Net::IMAP#idle_done
Articles