mechanize 2.3

drbrain | Tue, 21 Feb 2012 01:18:20 GMT

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, and can follow links and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

2.3 / 2012-02-20

  • Minor enhancement

    • Add support for the Max-Age attribute in the Set-Cookie header.

    • Added Mechanize::Download#body for compatibility with Mechanize::File when using Mechanize#get_file with Mechanize::Image or other Download-based pluggable parser. Issue #202 by angas

    • Mechanize#max_file_buffer may be set to nil to disable creation of Tempfiles.

  • Bug fixes

    • Applied Mechanize#max_file_buffer to the Content-Encoding handlers as well to prevent extra Tempfiles for small gzip or deflate response

    • Increased the default Mechanize#max_file_buffer to 100,000 bytes. This gives ~5MB of response bodies in memory with the default history setting of 50 pages (depending on GC behavior).

    • Ignore empty path/domain attributes.

    • Cookies with an empty Expires attribute value were stored as session cookies but cookies without the Expires attribute were not. Issue #78

Posted in  | no comments | no trackbacks

mechanize 2.2

drbrain | Tue, 14 Feb 2012 00:00:22 GMT

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

2.2 / 2012-02-12

  • API changes

    • MetaRefresh#href is not normalized to an absolute URL, but set to the original value and resolved later. It is even set to nil when the Refresh URL is unspecified or empty.

  • Minor enhancements

    • Expose ssl_version from net-http-persistent. Patch by astera.

    • SSL parameters and proxy may now be set at any time. Issue #194 by dsisnero.

    • Improved Mechanize::Page with #image_with and #images_with and Mechanize::Page::Image various img element attribute accessors, #caption, #extname, #mime_type and #fetch. Pull request #173 by kitamomonga

    • Added MIME type parsing for content-types in Mechanize::PluggableParser for fine-grained parser choices. Parsers will be chosen based on exact match, simplified type or media type in that order. See Mechanize::PluggableParser#[]=.

    • Added Mechanize#download which downloads a response body to an IO-like or filename.

    • Added Mechanize::DirectorySaver which saves responses in a single directory. Issue #187 by yoshie902a.

    • Added Mechanize::Page::Link#noreferrer?

    • The documentation for Mechanize::Page#search and #at now show that both XPath and CSS expressions are allowed. Issue #199 by Shane Becker.

  • Bug fixes

    • Fixed handling of a HEAD request with Accept-Encoding: gzip. Issue #198 by Oleg Dashevskii

    • Use #resolve for resolving a Location header value. fixes #197

    • A Refresh value can have whitespaces around the semicolon and equal sign.

    • MetaRefresh#click no longer sends out Referer.

    • A link with an empty href is now resolved correctly where previously the query part was dropped.

Posted in  | 2 comments | no trackbacks

net-http-persistent 2.5

drbrain | Tue, 07 Feb 2012 23:47:19 GMT

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. Net::HTTP::Persistent supports reconnection and retry according to RFC 2616.

2.5 / 2012-02-07

  • Minor enhancements

    • The proxy may be changed at any time.

    • The allowed SSL version may now be set via #ssl_version. Issue #16 by astera

    • Added Net::HTTP::Persistent#override_headers which allows overriding

    • Net::HTTP default headers like User-Agent. See Net::HTTP::Persistent@Headers for details. Issue #17 by andkerosine

  • Bug fixes

    • The ruby 1.8 speed monkeypatch now handles EAGAIN for windows users. Issue #12 by Alwyn Schoeman

    • Fixed POST example in README. Submitted by injekt.

    • Fixed various bugs in the shutdown of connections especially cross-thread (which you shouldn’t be doing anyways).

Posted in  | 2 comments | no trackbacks

mechanize 2.1.1

drbrain | Sat, 04 Feb 2012 01:57:15 GMT

mechanize version 2.1.1 has been released!

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

2.1.1 / 2010-02-03

  • Bug fixes

    • Set missing idle_timeout default. Issue #196

    • Meta refresh URIs are now escaped (excluding %). Issue #177

    • Fix charset name extraction. Issue #180

    • A Referer URI sent on request no longer includes user information or fragment part.

    • Tempfiles for storing response bodies are unlinked upon creation to avoid possible lack of finalization. Issue #183

    • The default maximum history size is now 50 pages to avoid filling up a disk with tempfiles accidentally. Related to Issue #183

    • Errors in bodies with deflate and gzip responses now result in a Mechanize::Error instead of silently being ignored and causing future errors. Issue #185

    • Mechanize now raises an UnauthorizedError instead of crashing when a 403 response does not contain a www-authenticate header. Issue #181

    • Mechanize gives a useful exception when attempting to click buttons across pages. Issue #186

    • Added note to Mechanize#cert_store describing how to add certificates in case your system does not come with a default set. Issue #179

    • Invalid content-disposition headers are now ignored. Issue #191

    • Fix NTLM by recognizing the “Negotiation” challenge instead of endlessly looping. Issue #192

    • Allow specification of the NTLM domain through Mechanize#auth. Issue #193

    • Documented how to convert a Mechanize::ResponseReadError into a File or Page, along with a new method #force_parse. Issue #176

Posted in  | 1 comment | no trackbacks

net-http-persistent 2.4.1

drbrain | Sat, 04 Feb 2012 01:55:59 GMT

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. Net::HTTP::Persistent supports reconnection and retry according to RFC 2616.

2.4.1 / 2012-02-03

  • Bug fixes

    • When FakeWeb or WebMock are loaded SSL sessions will not be reused to prevent breakage of testing frameworks. Issue #13 by Matt Brictson, pull request #14 by Zachary Scott

    • SSL connections are reset when the SSL parameters change. Mechanize issue #194 by dsisnero

    • Last-use times are now cleaned up in #shutdown.

Posted in  | no comments | no trackbacks

net-http-persistent 2.4

drbrain | Tue, 31 Jan 2012 23:58:10 GMT

net-http-persistent version 2.4 has been released!

Manages persistent connections using Net::HTTP plus a speed fix for Ruby 1.8. It’s thread-safe too!

Using persistent HTTP connections can dramatically increase the speed of HTTP. Creating a new HTTP connection for every request involves an extra TCP round-trip and causes TCP congestion avoidance negotiation to start over.

Net::HTTP supports persistent connections with some API methods but does not handle reconnection gracefully. Net::HTTP::Persistent supports reconnection and retry according to RFC 2616.

2.4 / 2012-01-31

  • Minor Enhancement

    • net-http-persistent now complains if OpenSSL::SSL::VERIFY_PEER is equal to OpenSSL::SSL::VERIFY_NONE. If you have a platform that is broken this way you must define the constant:

      I_KNOW_THAT_OPENSSL_VERIFY_PEER_EQUALS_VERIFY_NONE_IS_WRONG = nil

      at the top level of your application to disable the warning.

  • Bug fix

    • Fix persisting SSL sessions through HTTP proxies. Mechanize issue #178 by Robert Poor, net-http-persistent issues #10, #11.

Posted in  | no comments | no trackbacks

return_bang 1.0

drbrain | Wed, 21 Dec 2011 00:52:48 GMT

return_bang version 1.0 has been released!

return_bang implements non-local exits from methods. Use return_bang to exit back to a processing loop from deeply nested code, or just to confound your enemies and your friends! What could possibly go wrong?

Features

  • Implements non-local exits for methods

  • Nestable

  • Named and stack-based exit points, go exactly where you need to be

  • Ignores pesky ensure blocks for when you really, really need to return

Synopsis

require 'return_bang/everywhere'

def some_method
  deeply_nested
  # never reached
end

def deeply_nested
  return!
end

return_here do
  some_method
end
# resumes here

Testimonials

“you’ll wind up with your cock in /dev/null somehow” – slyphon

“Haha! Right! This skips ensure… SO EVIL‼‼” – drbrain

“This is so evil that 6 def test_… have turned into: 16 tests, 65 assertions, 18 failures, 7 errors” – drbrain

Install

sudo gem install return_bang

Posted in  | 1 comment | no trackbacks

mechanize 2.1

drbrain | Tue, 20 Dec 2011 19:14:27 GMT

mechanize version 2.1 has been released!

The Mechanize library is used for automating interaction with websites. Mechanize automatically stores and sends cookies, follows redirects, can follow links, and submit forms. Form fields can be populated and submitted. Mechanize also keeps track of the sites that you have visited as a history.

2.1 / 2011-12-20

  • Deprecations

    • Mechanize#get no longer accepts an options hash.

    • Mechanize::Util::to_native_charset has been removed.

  • Minor enhancements

    • Mechanize now depends on net-http-persistent 2.3+. This new version brings idle timeouts to help with the dreaded “too many connection resets” issue when POSTing to a closed connection. Issue #123

    • SSL connections will be verified against the system certificate store by default.

    • Added Mechanize#retry_change_requests to allow mechanize to retry POST and other non-idempotent requests when you know it is safe to do so. Issue #123

    • Mechanize can now stream files directly to disk without loading them into memory first through Mechanize::Download, a pluggable parser for downloading files.

      All responses larger than Mechanize#max_file_buffer are downloaded to a Tempfile. For backwards compatibility Mechanize::File subclasses still load the response body into memory.

      To force all unknown content types to download to disk instead of memory set:

      agent.pluggable_parser.default = Mechanize::Download

    • Added Mechanize#content_encoding_hooks which allow handling of non-standard content encodings like “agzip”. Patch #125 by kitamomonga

    • Added dom_class to elements and the element matcher like dom_id. Patch #156 by Dan Hansen.

    • Added support for the HTML5 keygen form element. See dev.w3.org/html5/spec/Overview.html#the-keygen-element Patch #157 by Victor Costan.

    • Mechanize no longer follows meta refreshes that have no “url=” in the content attribute to avoid infinite loops. To follow a meta refresh to the same page set Mechanize#follow_meta_refresh_self to true. Issue #134 by Jo Hund.

    • Updated ‘Mac Safari’ User-Agent alias to Safari 5.1.1. ‘Mac Safari 4’ can be used for the old ‘Mac Safari’ alias.

    • When given multiple HTTP authentication options mechanize now picks the strongest method.

    • Improvements to HTTP authorization:

      • mechanize raises Mechanize::UnathorizedError for 401 responses which is a sublcass of Mechanize::ResponseCodeError.

      • Added support for NTLM authentication, but this has not been tested.

    • Mechanize::Cookie.new accepts attributes in a hash.

    • Mechanize::CookieJar#<<(cookie) (alias: add!) is added. Issue #139

    • Different mechanize instances may now have different loggers. Issue #122

    • Mechanize now accepts a proxy port as a service name or number string. Issue #167

  • Bug fixes

    • Mechanize now handles cookies just as most modern browsers do, roughly based on RFC 6265.

      • domain=.example.com (which is invalid) is considered identical to domain=example.com.

      • A cookie with domain=example.com is sent to host.sub.example.com as well as host.example.com and example.com.

      • A cookie with domain=TLD (no dots) is accepted and sent if the host name is TLD, and rejected otherwise. To retain compatibility and convention, host/domain names starting with “local” are exempt from this rule.

      • A cookie with no domain attribute is only sent to the original host.

      • A cookie with an Effective TLD is rejected based on the public suffix list. (cf. publicsuffix.org/)

      • “Secure” cookies are not sent via non-https connection.

      • Subdomain match is not performed against an IP address.

      • It is recommended that you clear out existing cookie jars for regeneration because previously saved cookies may not have been parsed correctly.

    • Mechanize takes more care to avoid saving files with certain unsafe names. You should still take care not to use mechanize to save files directly into your home directory ($HOME). Issue #163.

    • Mechanize#cookie_jar= works again. Issue #126

    • The original Referer value persists on redirection. Issue #150

    • Do not send a referer on a Refresh header based redirection.

    • Fixed encoding error in tests when LANG=C. Patch #142 by jinschoi.

    • The order of items in a form submission now match the DOM order. Patch #129 by kitamomonga

    • Fixed proxy example in EXAMPLE. Issue #146 by NielsKSchjoedt

Posted in  | 1 comment | no trackbacks

rdoc 3.12

drbrain | Thu, 15 Dec 2011 21:58:05 GMT

home

github.com/rdoc/rdoc

rdoc

docs.seattlerb.org/rdoc

bugs

github.com/rdoc/rdoc/issues

RDoc produces HTML and command-line documentation for Ruby projects. RDoc includes the rdoc and ri tools for generating and displaying online documentation.

See RDoc for a description of RDoc’s markup and basic use.

3.12 / 2011-12-15

  • Minor enhancements

    • Added DEVELOPERS document which contains an overview of how RDoc works and how to add new features to RDoc.

    • Improved title for HTML output to include --title in the title element.

    • rdoc --pipe now understands --markup.

    • RDoc now supports irc-scheme hyperlinks. Issue #83 by trans.

  • Bug fixes

    • Fix title on HTML output for pages.

    • Fixed parsing of non-indented HEREDOC.

    • Fixed parsing of %w[] and other % literals. Issue #84 by Erik Hollensbe

    • Fixed arrow replacement in HTML output munging the spaceship operator. Issue #85 by eclectic923.

    • Verbatim sections with ERB that match the ruby code whitelist are no longer syntax-highlighted. Issue #86 by eclectic923

    • Line endings on windows are normalized immediately after reading with binmode. Issue #87 by Usa Nakamura

    • RDoc better understands directives for comments. Comment directives can now be found anywhere in multi-line comments. Issue #90 by Ryan Davis

    • Tidy links to methods show the label again. Issue #88 by Simon Chiang

    • RDoc::Parser::C can now find comments directly above rb_define_class_under. Issue #89 by Enrico

    • In rdoc, backspace and ansi formatters, labels and notes without bodies are now shown.

    • In rdoc, backspace and ansi formatters, whitespace between label or note and the colon is now stripped.

Posted in ,  | no comments | no trackbacks

mechanize 2.1 prerelease

drbrain | Wed, 09 Nov 2011 22:10:25 GMT

mechanize 2.1.pre.1 is up on rubygems.org. Notable improvements over mechanize 2.0.1 include:

  • Improvements in handling of connections closed by the server including a connection idle timeout setting and the ability to disable keep-alive connections. This addresses the "too many connection resets" error.
  • Improvements in cookie handling by Akinori MUSHA including cookie behavior like most modern browsers, proper TLD cookie domain handling via the domain_name gem
  • SSL peer certificate validation is enabled by default.

You can read the full list of changes in the mechanize release notes.

You can install the prerelease version of mechanize by running: 

gem install mechanize --prerelease

If you find bugs in the prerelease version please file an issue in the mechanize issues tracker on github.

Posted in  | no comments | no trackbacks