Forever-valid SSL certificates

drbrain | Wed, 11 Jan 2012 22:04:00 GMT

If your library uses X509 cryptography, naturally your tests will need a key and valid certificate to test against. Creating a key and certificate frequently can quickly drain your entropy pool which slows down your tests.

Instead of creating the key for every test startup you can create it once and load it off the disk like this: 

class TestMyGem < MyGem::TestCase
  private_key = File.expand_path '../../../test/private_key.pem', __FILE__
  private_key = File.read private_key
  PRIVATE_KEY = OpenSSL::PKey::RSA.new private_key

  # …

Sure, you can rebuild the certificate every time with a validity time of an hour, but why not create a forever-valid certificate to go with it? No reasonable person would ever use a key shipped with an open project anyhow. Here's how to generate such a key and certificate:

require 'openssl'

# purposefully short key length
key = OpenSSL::PKey::RSA.new 512

# bogus subject and issuer
name = OpenSSL::X509::Name.parse 'CN=nobody/DC=example'
cert = OpenSSL::X509::Certificate.new
cert.subject = name
cert.issuer = name
cert.version = 2
cert.serial = 1
cert.not_before = Time.now

# lasts as long as X509 allows
cert.not_after = Time.gm 9999, 12, 31, 23, 59, 59
cert.public_key = key.public_key

cert.sign key, OpenSSL::Digest::SHA1.new

open 'private_key.pem', 'w' do |io| io.write key.to_pem end
open 'public_cert.pem', 'w' do |io| io.write cert.to_pem end

You can load this certificate just like the key as described above: 

  public_cert = File.expand_path '../../../test/public_cert.pem', __FILE__
  public_cert = File.read public_cert
  PUBLIC_CERT = OpenSSL::X509::Certificate.new public_cert

Posted in ,  | 17 comments | no trackbacks

Replace your test helpers with reusable API

drbrain | Tue, 10 Jan 2012 18:37:50 GMT

test/test_helper.rb is a great idea Rails brought to the Ruby world as a place for functionality that helps you write better tests. There's now a standard place for you to implement common setup/teardown, shortcuts and custom assertions. However, a test helper is not the best place to store this functionality for a Ruby library.

One of the benefits you get out of writing tests is knowing where your API is clumsy and inadequate. If you have a test helper file full of methods to make your library easy to use in a test why is that not part of your library's API? Wouldn't your users also want a bunch of methods that make your library easier to use in their applications?

For example, in RubyGems we have a test helper that does this: gem_file = Gem::Builder.new(spec).build which is a little silly. Every time you create a Gem::Builder you want to build a gem. You don't create a Gem::Builder object for fun! To help out RubyGems users I added a new method: gem_file = Gem::Builder.build spec which immediately creates and builds them gem which is much nicer for everyone (but really, you should use Gem::PackageTask when building gems).

Whether you're writing a library or a Rails app, this kind of functionality belongs in your library (or application) code, not in the test helper where only your tests can benefit from it.

Even after you improve your API by moving helpful functionality back into your library there's still going to be some things that only make sense for tests. For example, you probably don't want to type t = Some::Deeply::Namespaced::Thing.new 1, 2, 3 many, many times in your tests, so you write a short wrapper method you can call like this: t = thing 1, 2, 3. Your tests may need setup and teardown to maintain a clean environment between tests, custom assertions for readability or you may want to include a pre-built stub or mock.

While this having this functionality in a test helper is fine for a Rails app, it shouldn't go in a library's test helper. When you keep testing functionality hidden in the test directory a user who wants to write a third-party extension for your gem can't access them. Why force a happy user to re-implement (possibly poorly or incorrectly) the work you've done to have nice, clean tests that are easy to read and write?

Instead of having a private test helper I have a public test case like MyGem::TestCase that lives in lib/my_gem/test_case.rb. This gives anyone who wants to extend my libraries a documented, ready-to-go API for writing tests for their extension.

My gem-specific test case typically contains all the requires needed to load the library (ideally require 'my_gem'), proper setup and teardown to sandbox the tests, any utility methods that don't belong in the library itself and possibly some custom assertions. This makes a brand new test easy to start: 

require 'my_gem/test_case'

class TestMyGemSomeClass < MyGem::TestCase
  def setup
    super

    # …
  end

  def test_something
    # …
  end
end

There is the minor downside that an extension writer must use minitest (my preferred testing library) to test their extension. Perhaps this inconvenience could be solved by a module providing setup, teardown and shortcuts that is included in the proper place for the extension writer's favorite testing library.

PS: Actually, Gem::Builder.build is Gem::Package.build since Gem::Format, Gem::Builder and Gem::Package are getting merged into one convenient class that deals with reading and writing gem files for RubyGems 2.0. This means there will only be one place to look for the API of messing with packages and it reduces the implementation of Gem::Installer a bit.

Posted in ,  | 1 comment | no trackbacks

return_bang 1.0

drbrain | Wed, 21 Dec 2011 00:52:48 GMT

return_bang version 1.0 has been released!

return_bang implements non-local exits from methods. Use return_bang to exit back to a processing loop from deeply nested code, or just to confound your enemies and your friends! What could possibly go wrong?

Features

  • Implements non-local exits for methods

  • Nestable

  • Named and stack-based exit points, go exactly where you need to be

  • Ignores pesky ensure blocks for when you really, really need to return

Synopsis

require 'return_bang/everywhere'

def some_method
  deeply_nested
  # never reached
end

def deeply_nested
  return!
end

return_here do
  some_method
end
# resumes here

Testimonials

“you’ll wind up with your cock in /dev/null somehow” – slyphon

“Haha! Right! This skips ensure… SO EVIL‼‼” – drbrain

“This is so evil that 6 def test_… have turned into: 16 tests, 65 assertions, 18 failures, 7 errors” – drbrain

Install

sudo gem install return_bang

Posted in  | 1 comment | no trackbacks

mechanize 2.1

drbrain | Tue, 20 Dec 2011 19:14:27 GMT

mechanize version 2.1 has been released!

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

2.1 / 2011-12-20

  • Deprecations

    • Mechanize#get no longer accepts an options hash.

    • Mechanize::Util::to_native_charset has been removed.

  • Minor enhancements

    • Mechanize now depends on net-http-persistent 2.3+. This new version brings idle timeouts to help with the dreaded “too many connection resets” issue when POSTing to a closed connection. Issue #123

    • SSL connections will be verified against the system certificate store by default.

    • Added Mechanize#retry_change_requests to allow mechanize to retry POST and other non-idempotent requests when you know it is safe to do so. Issue #123

    • Mechanize can now stream files directly to disk without loading them into memory first through Mechanize::Download, a pluggable parser for downloading files.

      All responses larger than Mechanize#max_file_buffer are downloaded to a Tempfile. For backwards compatibility Mechanize::File subclasses still load the response body into memory.

      To force all unknown content types to download to disk instead of memory set:

      agent.pluggable_parser.default = Mechanize::Download

    • Added Mechanize#content_encoding_hooks which allow handling of non-standard content encodings like “agzip”. Patch #125 by kitamomonga

    • Added dom_class to elements and the element matcher like dom_id. Patch #156 by Dan Hansen.

    • Added support for the HTML5 keygen form element. See dev.w3.org/html5/spec/Overview.html#the-keygen-element Patch #157 by Victor Costan.

    • Mechanize no longer follows meta refreshes that have no “url=” in the content attribute to avoid infinite loops. To follow a meta refresh to the same page set Mechanize#follow_meta_refresh_self to true. Issue #134 by Jo Hund.

    • Updated ‘Mac Safari’ User-Agent alias to Safari 5.1.1. ‘Mac Safari 4’ can be used for the old ‘Mac Safari’ alias.

    • When given multiple HTTP authentication options mechanize now picks the strongest method.

    • Improvements to HTTP authorization:

      • mechanize raises Mechanize::UnathorizedError for 401 responses which is a sublcass of Mechanize::ResponseCodeError.

      • Added support for NTLM authentication, but this has not been tested.

    • Mechanize::Cookie.new accepts attributes in a hash.

    • Mechanize::CookieJar#<<(cookie) (alias: add!) is added. Issue #139

    • Different mechanize instances may now have different loggers. Issue #122

    • Mechanize now accepts a proxy port as a service name or number string. Issue #167

  • Bug fixes

    • Mechanize now handles cookies just as most modern browsers do, roughly based on RFC 6265.

      • domain=.example.com (which is invalid) is considered identical to domain=example.com.

      • A cookie with domain=example.com is sent to host.sub.example.com as well as host.example.com and example.com.

      • A cookie with domain=TLD (no dots) is accepted and sent if the host name is TLD, and rejected otherwise. To retain compatibility and convention, host/domain names starting with “local” are exempt from this rule.

      • A cookie with no domain attribute is only sent to the original host.

      • A cookie with an Effective TLD is rejected based on the public suffix list. (cf. publicsuffix.org/)

      • “Secure” cookies are not sent via non-https connection.

      • Subdomain match is not performed against an IP address.

      • It is recommended that you clear out existing cookie jars for regeneration because previously saved cookies may not have been parsed correctly.

    • Mechanize takes more care to avoid saving files with certain unsafe names. You should still take care not to use mechanize to save files directly into your home directory ($HOME). Issue #163.

    • Mechanize#cookie_jar= works again. Issue #126

    • The original Referer value persists on redirection. Issue #150

    • Do not send a referer on a Refresh header based redirection.

    • Fixed encoding error in tests when LANG=C. Patch #142 by jinschoi.

    • The order of items in a form submission now match the DOM order. Patch #129 by kitamomonga

    • Fixed proxy example in EXAMPLE. Issue #146 by NielsKSchjoedt

Posted in  | 1 comment | no trackbacks

rdoc 3.12

drbrain | Thu, 15 Dec 2011 21:58:05 GMT

home

github.com/rdoc/rdoc

rdoc

docs.seattlerb.org/rdoc

bugs

github.com/rdoc/rdoc/issues

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying online documentation.

See RDoc for a description of RDoc’s markup and basic use.

3.12 / 2011-12-15

  • Minor enhancements

    • Added DEVELOPERS document which contains an overview of how RDoc works and how to add new features to RDoc.

    • Improved title for HTML output to include --title in the title element.

    • rdoc --pipe now understands --markup.

    • RDoc now supports irc-scheme hyperlinks. Issue #83 by trans.

  • Bug fixes

    • Fix title on HTML output for pages.

    • Fixed parsing of non-indented HEREDOC.

    • Fixed parsing of %w[] and other % literals. Issue #84 by Erik Hollensbe

    • Fixed arrow replacement in HTML output munging the spaceship operator. Issue #85 by eclectic923.

    • Verbatim sections with ERB that match the ruby code whitelist are no longer syntax-highlighted. Issue #86 by eclectic923

    • Line endings on windows are normalized immediately after reading with binmode. Issue #87 by Usa Nakamura

    • RDoc better understands directives for comments. Comment directives can now be found anywhere in multi-line comments. Issue #90 by Ryan Davis

    • Tidy links to methods show the label again. Issue #88 by Simon Chiang

    • RDoc::Parser::C can now find comments directly above rb_define_class_under. Issue #89 by Enrico

    • In rdoc, backspace and ansi formatters, labels and notes without bodies are now shown.

    • In rdoc, backspace and ansi formatters, whitespace between label or note and the colon is now stripped.

Posted in ,  | no comments | no trackbacks

Packaging from a gemspec is not the Best Way

drbrain | Fri, 09 Dec 2011 22:57:06 GMT

Or, why you should use Hoe (or a tool like it) to package your gems.

A while back Yehuda wrote Using .gemspecs as Intended and asserted that packaging directly from a .gemspec file was both the way you were supposed to build gems and the best way to build gems.

This is not true.

Rake::GemPackageTask (now Gem::PackageTask) was added to rake less than two weeks after the first commit on RubyGems. Which points to the recognition of a need to have a better way to describe and build a gem than just a file.

Using rake allows you to do more than just package a static list of files in a clean and reusable manner. You can write tasks to generate files that are then packaged in your gem at build time. Hoe builds atop Rake to support many types of generated files easily through task dependencies.

For example, generating a .gemtest file to work with GemTesters, integrating with rake-compiler for pre-compiled gems or generating parsers.

I can hear you thinking "you might have a point there, but I don't generate any files, so how does this apply to me?" Well, what if you wanted to include generated man pages in your gem using binman?

In the world of packaging directly from a gemspec the way to automatically generate files is to violate RubyGems' internals by overriding Gem::Specification#initialize. I can't blame Suraj Kurapati for this, he's working within the framework of the popular thing to do and this is the only way he could implement it.

(Hoe, on the other hand, handles such pre-package dependencies through a set of pre-defined rake tasks that you can add dependencies too. If you need to generate a file you make the package task dependent on it. See the examples above.)

How can we make this better?

First of all, stop using gemspecs as the One True Description of your gem. Let rake generate it. While Hoe is obviously the best way to do this, it doesn't support generating the gemspec by default. You can use one of the gemspec plugins for that. I'm sure there are other build tools have the equivalent built right in.

Second, if you've writing a tool that generates files that will be packaged into a gem, provide a rake task like the one that ships with binman that build tool authors can hook to make packaging seamless.

(Also, please, don't use system when you can use API.)

Posted in ,  | 6 comments | no trackbacks

mechanize 2.1 prerelease

drbrain | Wed, 09 Nov 2011 22:10:25 GMT

mechanize 2.1.pre.1 is up on rubygems.org. Notable improvements over mechanize 2.0.1 include:

  • Improvements in handling of connections closed by the server including a connection idle timeout setting and the ability to disable keep-alive connections. This addresses the "too many connection resets" error.
  • Improvements in cookie handling by Akinori MUSHA including cookie behavior like most modern browsers, proper TLD cookie domain handling via the domain_name gem
  • SSL peer certificate validation is enabled by default.

You can read the full list of changes in the mechanize release notes.

You can install the prerelease version of mechanize by running: 

gem install mechanize --prerelease

If you find bugs in the prerelease version please file an issue in the mechanize issues tracker on github.

Posted in  | no comments | no trackbacks

net-http-persistent 2.2

drbrain | Tue, 25 Oct 2011 00:57:18 GMT

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. Net::HTTP::Persistent supports reconnection and retry according to RFC 2616.

2.2 / 2011-10-24

  • Minor Enhancements

    • Added timeouts for idle connections which are set through #idle_timeout. The default timeout is 5 seconds. Reducing the idle timeout is preferred over setting #retry_change_requests to true if you wish to avoid the “too many connection resets” error when POSTing data.

    • Documented tunables and settings in one place in Net::HTTP::Persistent

Posted in  | no comments | no trackbacks

RDoc 3.10

drbrain | Sat, 08 Oct 2011 23:18:00 GMT

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying online documentation.

See RDoc for a description of RDoc’s markup and basic use.

Changes:

3.10 / 2011-10-08

  • Major enhancements

    • RDoc HTML output has been improved:

      • The search from Володя Колесников‘s (Vladimir Kolesnikov) Sdoc has been integrated.

        The search index generation is a reusable component through RDoc::Generator::JsonIndex

      • The table of contents is now a separate page and now shows links to headings and sections inside a page or class.

      • Class pages no longer show the namespace and no longer have file info pages.

      • HTML output is HTML 5.

      • Static files can be copied into RDoc using –copy-files

    • RDoc supports additional documentation formats:

      • TomDoc 1.0.0-rc1

      • RD format

      The default markup can be set via the --markup option.

      The format of documentation in a particular file can be specified by the :markup: directive. If the :markup: directive is in the first comment it is used as the default for the entire file. For other comments it overrides the default markup format.

      The markup format can be set for rake tasks using RDoc::Task#markup

    • RDoc can save and load an options file.

      To create an options file that defaults to using TomDoc markup run:

      rdoc --markup tomdoc --write-options

      This will create a .rdoc_options file. Check it in to your VCS and package it with your gem. RDoc will automatically load this file and combine it with the user’s options.

      Some options are not saved. See RDoc::Options@Saved+Options for full details.

  • Minor enhancements

    • RDoc autoloads everything. You only need to require ‘rdoc’ now.

    • HTML headings now have ids matching their titles.

      = Hello!

      Is rendered as

      <h1 id="label-Hello%21">Hello!</h1>

    • Labels for classes or methods can be linked-to by adding an @ following the class or method reference. For example, RDoc::Markup@Links

      See RDoc::Markup@Links for further details.

    • For HTML output RDoc uses SomeClass.method_name and SomeClass#method_name for remote methods and attributes and ::method_name and #method_name for local methods.

    • RDoc makes an effort to syntax-highlight ruby code in verbatim sections. See RDoc::Markup@Paragraphs+and+Verbatim

    • Added RDoc::TopLevel#text? and RDoc::Parser::Text to indicate a parsed file contains no ruby constructs.

    • Added rdoc-label link scheme which allows bidirectional links. See RDoc::Markup for details.

    • Image paths at HTTPS URLs will now be turned into +<img>+ tags. Pull Request #60 by James Mead

    • Added RDoc::Comment which encapsulates comment-handling functionality.

    • Added RDoc::Markup::PreProcess::post_process to allow arbitrary comment munging.

    • RDoc::RDoc::current is set for the entire RDoc run.

    • Split rdoc/markup/inline into individual files for its component classes.

    • Moved token stream HTML markup out of RDoc::AnyMethod#markup_code into RDoc::TokenStream::to_html

    • “Top” link in section headers is no longer inside the heading element.

    • RDoc avoids printing some warnings unless run with `rdoc –verbose`. For Rails issue #1646.

    • Finishing a paragraph with two or more spaces will result in a line break. This feature is experimental and may be modified or removed.

  • Bug fixes

    • Markup defined by RDoc::Markup#add_special inside a <tt> is no longer converted.

    • Performance of RDoc::RubyLex has been improved. Ruby Bug #5202 by Ryan Melton.

    • Add US-ASCII magic comments to work with ruby -Ku. Issue #63 by Travis D. Warlick, Jr.

    • Clicking a link in the method description now works. Issue #61 by Alan Hogan.

    • Fixed RDoc::Markup::Parser for CRLF line endings. Issue #67 by Marvin Gülker.

    • Fixed lexing of percent strings like %r{#}. Issue #68 by eclectic923.

    • The C parser now understands classes defined with rb_struct_define_without_accessor (like Range). Pull Request #73 by Dan Bernier

    • Fixed lexing of a b <<-HEREDOC. Issue #75 by John Mair.

    • Added LEGAL.rdoc with references to licenses in other files. Issue #78 by Dmitry Jemerov.

    • Block parameters are displayed in Darkfish output again. Issue #76 by Andrea Singh.

    • The method parameter coverage report no longer includes parameter default values. Issue #77 by Jake Goulding.

    • The module for an include is not looked up until parsed all the files are parsed. Unless your project includes nonexistent modules this avoids worst-case behavior (O(n!)) of RDoc::Include#module.

Posted in ,  | no comments | no trackbacks

net-http-persistent 2.1

drbrain | Tue, 20 Sep 2011 01:58:00 GMT

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. net-http-persistent supports reconnection and retry according to RFC 2616.

2.1 / 2011-09-19

  • Minor Enhancement

    • For HTTPS connections, SSL sessions are now reused avoiding the extra round trips and computations of extra SSL handshakes. If you have problems with SSL session reuse it can be disabled by Net::HTTP::Persistent#reuse_ssl_sessions

  • Bug Fixes

    • The default certificate store is now used even if #verify_mode was not set. Issue #7, Pull Request #8 by Matthew M. Boedicker

Posted in  | no comments | no trackbacks