Postmodern's Blog

It's simple, we kill eval

2013-03-07T00:00:00-08:00

... with define_method

In @tenderlove's blog post on dynamic method definitions, he dispels the old myth that define_method is slower than module_eval/class_eval. This myth has caused Ruby developers to prefer using module_eval/class_eval with String substitution to dynamically define methods:

def define_custom_reader(name)
  module_eval %{
    def #{name}
      get_data("#{name}")
    end
  }
end

The problem with the above code is that if user-input is passed to define_custom_reader, it allows an attacker to inject arbitrary Ruby code. If we rewrite the above code using define_method, we prevent code injection:

def define_custom_reader(name)
  define_method name do
    get_data(name)
  end
end

Now define_custom_reader will define a method using name, and when the method is called it will call get_data with name.

Why doesn't every Rubyist use define_method? One reason was the myth about the performance impact. The other was that blocks could not accept other blocks in 1.8.x. This prevented Rubyists from using define_method to define methods that accept blocks. However, blocks can accept other blocks in 1.9.x:

define_method name do |&block|
  value = get_data(name)
  block.call(value) if block
  value
end

Given that Ruby 1.8.7 will officially reach End-of-Life in June, Ruby developers should be upgrading to Ruby 1.9.3 or 2.0.0. In fact, according to RubyGems.org statistics the majority of Rubyists have already upgraded to 1.9.x. There is no excuse not to use define_method for your metaprogramming needs.

Some other things you don't need eval for

Injecting reader/writer methods:

module Mixin
  attr_accessor :foo
end

class Base
  include Mixin
end

obj.extend Mixin

Defining Constants:

klass.const_set(:FOO,foo)

Setting Instance Variables:

obj.instance_variable_set('@foo',foo)

Setting Class Variables:

klass.class_variable_set('@@foo',foo)

metaid.rb

As depicted above define_method works great for defining instance methods. If you want to define class methods with define_method, you must open the metaclass:

class << self
  define_method name do
    # ...
  end
end

The class << self syntax can be cumbersome. To alleviate this, there are a set of helper methods called [metaid]:

meta_def name do
  # ...
end

Much cleaner!

It's time to kill eval

eval is a security risk and is not necessary in most cases. It is time to ~~kill~~ eradicate eval from our code-bases, before it causes another embarrassing Remote Code Execution (RCE) vulnerability.

Challenge: grep through as much Ruby code as possible looking for _eval [%\"], and replace as many instances as possible with define_method, const_set,class_variable_set,instance_variable_setor even*_eval` with a block:

$ egrep -r "_eval [%\"<]" */lib/

RubyGems Tasks

2012-05-22T00:00:00-07:00

Warning: Controversial Content

If you are morally or ethically opposed to building/releasing Ruby Gems with rake, this blog post may anger you. If this is the case, you are advised to STOP READING and CLICK THE BACK BUTTON.

Ever since we could release Ruby Gems, we have had Gem helpers that could generate new projects and provided Rake tasks to automate the building/publishing of Gems. The most popular of these Gem helpers were Hoe and Jeweler. Both Hoe and Jeweler required a certain project layout and imposed a certain workflow onto the developer.

Sometime before the introduction of Bundler, Yehuda Katz proposed a radically simpler way of building Gems, using the gemspec file. Developers could use the built-in gem build command to build a .gem file from a .gemspec, and use the gem push command to publish the Gem to rubygems.org. This marked the start of an exodus of sorts, away from using Gem helpers such as Hoe and Jeweler.

All you need is a Gemspec?

With the advent of building gems from a .gemspec, a vocal minority formed within the Ruby community. They proclaimed that all one needs is a gemspec, and that all other tools (Hoe, Jeweler, Bundler and even Rake) are now obsolete!

Let us take a look at how they might release a Gem:

$ git status
$ git push
$ gem build my_project.gemspec
$ gem push my_project-1.2.3.gem
$ git tag 1.2.3
$ git push --tags

It takes roughly six commands to release a Gem to rubygems.org. A developer must remember to run each of these six commands, in order, every time they release a new version of their Gem. The possibility for human error increases.

In fact, I have found Gems that contained newer source-code than their Git repository; because the developer forgot to commit or push the changes before pushing the Gem. I have also found Git repositories with no tags, making it difficult to review what exactly changed between versions.

Now, let us see how developers release gems using Rake tasks:

$ rake release

That's it! That one command will run each of the above six commands. If any of the commands fail, the release process will halt. If there are uncommitted changes, the release process will halt. The possibility for human error has been greatly minimized.

Alternatives

Having used Hoe, Jeweler and then Bundler, I missed some of the workflow provided by these Gem helpers. So I began searching for lightweight alternatives.

I looked at Gem::PackageTask, MG, gem release and even bundler/gem_helper. Unfortunately they all were missing various features. Gem::PackageTask only built the .gem file, but did not check git status, tag the release or push the .gem. MG also did not check git status before releasing, did not tag releases, and only supports Git. gem release came close, but did not check git status before releasing, deleted the .gem file, only supports Git and tried to do too much (auto-magically bumping the version of your project for you). bundler/gem_helper came the closest, although while it defined Rake tasks it did not leverage Rake's powerful task => dependency system.

So I decided to cherry-pick all of the nice features from Hoe, Jeweler, MG, gem release and bundler/gem_helper, and leave out the opinionated features.

rubygems-tasks

rubygems-tasks provides agnostic and unobtrusive Rake tasks for building, installing and releasing Ruby Gems.

$ gem install rubygems-tasks

Adding rubygems-tasks to an existing project is easy as:

require 'rubygems/tasks'
Gem::Tasks.new

$ rake -T
rake build    # Builds all packages
rake console  # Spawns an Interactive Ruby Console
rake install  # Installs all built gem packages
rake release  # Performs a release

Features

Provides tasks to build, install and push gems to rubygems.org:
- Loads all project metadata from the .gemspec file.
- Supports loading multiple .gemspec files.
- Supports pushing gems to alternate Gemcutter servers.
Supports optionally building .tar.gz and .zip archives.
Supports Git, Mercurial and SubVersion SCMs.
- Supports creating PGP signed Git/Mercurial tags.
Supports generating checksums of built packages:
```
Gem::Tasks.new(:sign => {:checksum => true})
```
Supports generating PGP signatures for built packages:
```
Gem::Tasks.new(:sign => {:pgp => true})
```
Provides a console task, for jumping right into your code.
Defines task aliases for users coming from Jeweler or Hoe.
ANSI coloured messages!

Anti-Features

Does not parse project metadata from the README or the ChangeLog.
Does not generate or modify your code.
Does not automatically commit changes.
Does not inject dependencies into gems.
Zero dependencies.

Examples

Specifying an alternate Ruby Console to run:

Gem::Tasks.new do |tasks|
  tasks.console.command = 'pry'
end

Enable pushing gems to an in-house Gemcutter server:

Gem::Tasks.new do |tasks|
  tasks.push.host = 'gems.company.com'
end

Disable the push task:

Gem::Tasks.new(:push => false)

Enable building .tar.gz and .zip archives:

Gem::Tasks.new(:build => {:tar => true, :zip => true})

Enable Checksums and PGP signatures for built packages:

Gem::Tasks.new(:sign => {:checksum => true, :pgp => true})

Selectively defining tasks:

Gem::Build::Tar.new
Gem::SCM::Status.new
Gem::SCM::Tag.new(:format => 'REL-%s')
Gem::Sign::Checksum.new
Gem::Console.new

You don't have to use Bundler to create new RubyGems

2012-05-20T00:00:00-07:00

Warning: Controversial Content

If you are morally or ethically opposed to using Project Generators and prefer to create RubyGems by hand, STOP READING AND CLICK THE BACK BUTTON.

I am perfectly aware that one does not need any tools to create a RubyGem, and that all you really need is RubyGems and a *.gemspec file. However, the majority of users do not have the time or patience to create each Ruby project from scratch. Thus developers have historically used project generators such as Hoe, Jeweler and now Bundler.

If you are a diehard Bundler user, please read the entire blog post. This blog post is not putting Bundler down, nor is it posing a binary choice between Bundler and some new tool.

Enter Bundler

Bundler was initially created as a more robust way to manage dependencies of Rails3 applications. Once Bundler was integrated into the Rails3 generator templates, developers realized Bundler could also be used to manage the dependencies of any large Ruby application or library.

In order to help developers create projects with Bundler already setup, template files and a bundle gem command were added. Since Bundler was the "new hotness" and developers were becoming increasingly dissatisfied with Jeweler/Hoe, the community began to cargo cult Bundler as the defacto way to create RubyGems.

Looking Back

After having extensively used Bundler with Ronin, to keep it's many repositories in-sync with each other, I can definitely say Bundler solved dependency management for large Ruby projects. However, I began to question using bundle gem to create new RubyGems.

At first it made sense to provide a bundle gem command, but project generation is outside of the original scope of Bundler (dependency management). Furthermore, adding Bundler to an existing project isn't that difficult; just add a .gemspec file and a Gemfile.

Given that Bundler's stated goal is to "manage an application's dependencies", it doesn't seem very pragmatic to use Bundler in libraries with zero or only one runtime dependency.

Bundler's template files are a bit spartan as well. The Rakefile template does not include Rake tasks for RDoc or RSpec. This omission might encourage developers to not write documentation or tests, and rush to release.

The bundle gem command isn't very configurable either. If you want to generate a project with Textile markup, YARD documentation, RSpec tests and Mercurial, you are out of luck.

Enter Ore

Ore is a flexible Ruby project generator. Unlike other Ruby project generators, Ore provides many built in templates and allows custom templates to be installed from Git repositories.

$ gem install ore
$ mine my_project
Generating /home/postmodern/my_project
      create  lib
      create  lib/my_project
      create  spec
      create  .gitignore
      create  .rspec
      create  spec/my_project_spec.rb
      create  spec/spec_helper.rb
      create  .document
      create  my_project.gemspec
      create  ChangeLog.rdoc
      create  LICENSE.txt
      create  README.rdoc
      create  Rakefile
      create  lib/my_project/version.rb
      create  lib/my_project.rb
         run  git init from "."
         run  git add . from "."
         run  git commit -m "Initial commit." from "."
$ cd my_project/
$ rake -T
rake build         # Builds all packages
rake clobber_rdoc  # Remove RDoc HTML files
rake console       # Spawns an Interactive Ruby Console
rake install       # Installs all built gem packages
rake rdoc          # Build RDoc HTML files
rake release       # Performs a release
rake rerdoc        # Rebuild RDoc HTML files
rake spec          # Run RSpec code examples

Ore generates new Ruby projects with sensible defaults, such as a .gemspec file, rubygems-tasks, RDoc, RSpec, a .gitignore file and initializes the Git repository.

Ore also provides many different templates and options:

$ mine --help
Usage:
  mine PATH

Options:
      [--gemspec-yml]               
      [--jeweler-tasks]             
      [--bundler]                   
      [--rubygems-tasks]            
                                    # Default: true
      [--yard]                      
      [--bundler-tasks]             
      [--hg]                        
      [--rspec]                     
                                    # Default: true
      [--rvmrc]                     
      [--gem-test]                  
      [--gem-package-task]          
      [--gemspec]                   
                                    # Default: true
      [--test-unit]                 
      [--git]                       
                                    # Default: true
      [--bin]                       
      [--rdoc]                      
                                    # Default: true
      [--markdown]                  
      [--textile]                   
  -T, [--templates=TEMPLATE [...]]  
  -n, [--name=NAME]                 
  -V, [--version=VERSION]           
                                    # Default: 0.1.0
  -s, [--summary=SUMMARY]           
                                    # Default: TODO: Summary
  -D, [--description=DESCRIPTION]   
                                    # Default: TODO: Description
  -a, [--authors=NAME [...]]        
                                    # Default: ["postmodern"]
  -e, [--email=EMAIL]               
  -U, [--homepage=HOMEPAGE]         
  -B, [--bug-tracker=BUG_TRACKER]   
  -L, [--license=LICENSE]           
                                    # Default: MIT

As you can see, Ore is completely configurable and supports:

Git / Mercurial / SubVersion
.rvmrc files
Bundler
rubygems-tasks / Bundler tasks / Jeweler::Tasks / Gem::PackageTask
RSpec / Test::Unit
YARD / RDoc documentation
RDoc / Markdown / Textile markup

Unlike other project generators, Ore focuses only on project generation and does not force a specific project layout or workflow upon the developer. You can even generate projects with Bundler, YARD + Markdown and Mercurial instead of Git:

$ mine my_project --bundler --yard --markdown --hg
Generating /home/postmodern/my_project
      create  lib
      create  lib/my_project
      create  spec
      create  .hgignore
      create  .document
      create  .yardopts
      create  Gemfile
      create  .rspec
      create  spec/my_project_spec.rb
      create  spec/spec_helper.rb
      create  my_project.gemspec
      create  ChangeLog.md
      create  LICENSE.txt
      create  README.md
      create  Rakefile
      create  lib/my_project/version.rb
      create  lib/my_project.rb
         run  hg init from "."
         run  hg add . from "."
         run  hg commit -m "Initial commit." from "."
$ cd my_project && bundle install
$ rake spec
/home/postmodern/.rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec ./spec/my_project_spec.rb

MyProject
  should have a VERSION constant

Finished in 0.00593 seconds
1 example, 0 failures

Think Outside of the Bundle

Now that you have been introduced to Ore, I hope you will at the very least give it a try. I also hope you will understand that I am not simply anti-Bundler / pro-Ore. Ore gives you the option of generating Ruby projects with/without Bundler. The two tools are not mutually exclusive.

Reading RFID Cards on Linux (in the year 2012)

2012-04-04T00:00:00-07:00

A while back tenderlove blogged about using Ruby to interact with NFC tags. Near Field Communication (NFC) grew out of the Radio Frequency Identification (RFID) standard and has been used for contact-less payment with SmartPhones. Although, NFC tags are not yet widely deployed in the US, as opposed to Japan. On the other hand, RFID is widely deployed in the US, used in Payment systems, Asset management, Inventory systems, Product tracking, ID cards, Transportation tracking and even Passports. RFID is out there, and in those tags may be interesting information.

Not wanting to simply copy tenderlove and purchase an NFC starter kit, I went searching for a consumer grade RFID reader. I ended up purchasing an OmniKey 6321 USB reader/writer (ideal for mobile use) and a couple blank Mifare Read/Write RFID cards (with 8K EEPROM!). The next logical step is getting the reader working on Linux.

Unless you have a serial RFID reader (or can setup a USB Serial device with udev), the classic rfdump program is pretty much useless. librfid does not appear to be maintained any longer. Also, librfid-tool -S did not detect the OmniKey USB reader. Google quickly found many tutorials for using pcsc-lite to interact with SmartCard readers to read RFID tags.

Install pcscd

On Debian:

# apt-get install pcscd

On Fedora:

# yum install pcsc-lite

Install the OmniKey 6321 PC/SC Driver

Browser to [www.hidglobal.com/driverDownloads.php?techCat=19&prod_id=274][http://www.hidglobal.com/driverDownloads.php?techCat=19&prod_id=274].
Select your Operating System. Choose "Linux" for 32bit or "Linux x64" for 64bit.
Download the ifdokrfid_lnx tar archive file.

Extract the ifdokrfid_lnx tar archive:

$ tar xzvf ifdokrfid_lnx*.tar.gz
$ cd ifdokrfid_lnx*/

Install the proprietary PC/SC Driver:
```
$ ./install
```

Start pcscd

Plugin the OmniKey 6321 USB reader and start pcscd:

# pcscd -a -f -d

Then you should see the following output:

00000083 pcscdaemon.c:575:main() pcsc-lite 1.7.4 daemon ready.
00002003 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000196 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000189 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x047D, PID: 0x2043, path: /dev/bus/usb/002/002
00000174 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x047D, PID: 0x2043, path: /dev/bus/usb/002/002
00000176 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000180 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x046D, PID: 0xC069, path: /dev/bus/usb/002/003
00000207 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/003/001
00000184 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/003/001
00000190 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x076B, PID: 0x6321, path: /dev/bus/usb/003/002
00000081 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x076B, PID: 0x6321, path: /dev/bus/usb/003/002
00000056 hotplug_libudev.c:309:HPAddDevice() Adding USB device: OMNIKEY 6321
00000087 readerfactory.c:934:RFInitializeReader() Attempting startup of OMNIKEY 6321 (USB iClass Reader) 00 00 using /usr/lib64/pcsc/drivers/ifdokrfid_lnx_x64-2.10.0.1.bundle/Contents/Linux/ifdokrfid.so
00000423 readerfactory.c:824:RFBindFunctions() Loading IFD Handler 3.0
HID HID Global OMNIKEY RFID  X64 v2.10.0.1 
00304886 readerfactory.c:295:RFAddReader() Using the reader polling thread

To test the reader, briefly place an RFID card near the reader then remove it:

99999999 eventhandler.c:372:EHStatusHandlerThread() powerState: POWER_STATE_POWERED
00000030 eventhandler.c:387:EHStatusHandlerThread() Card inserted into OMNIKEY 6321 (USB iClass Reader) 00 01
00000015 Card ATR: 3B 8F 80 01 80 4F 0C A0 00 00 03 06 03 00 03 00 00 00 00 68 
00462848 eventhandler.c:446:EHStatusHandlerThread() powerState: POWER_STATE_UNPOWERED
00305960 eventhandler.c:325:EHStatusHandlerThread() Card Removed From OMNIKEY 6321 (USB iClass Reader) 00 01

DIY Pagination with DataMapper

2012-04-04T00:00:00-07:00

There are many pagination solutions out there. Of course there's the venerable will_pagination and the much newer Kaminari. However, all of the pagination solutions usually contain boiler-plate HTML. What if we only want the pagination logic, maybe in a JSON API, without using Rails or ActiveRecord, but instead Sinatra and DataMapper. As it turns out, DataMapper makes DIY pagination as simple as:

@posts = Post[((page - 1) * per_page), per_page]

As a Sinatra helper method this would look like:

def paginate(query)
  @page     = (params[:page] || 1).to_i
  @per_page = (params[:per_page] || 10).to_i

  query[((@page - 1) * @per_page), @per_page]
end

Now, what if we want to know the total number of pages and records? Enter the dm-chunked_query gem, which provides convenience methods for querying chunks of records:

require 'dm-chunked_query'

def paginate(query)
  @page        = (params[:page] || 1).to_i
  @per_page    = (params[:per_page] || 10).to_i

  @pages       = query.chunks_of(per_page)
  @total_count = @pages.count
  @page_count  = @pages.length

  @pages[@page - 1]
end

Pagination is that simple.

Hexdump 0.2.x

2011-06-11T00:00:00-07:00

Recently mephux began working on a hexdump.js library (epic demo). Naturally, I thought I should go back and improve my Ruby Hexdump library.

Hexdump 0.2.x

Hexdump now supports word-sizes and endianness. This is useful for dumping packed unsigned ints:

Hexdump.dump("\x12\x34\x00\x00\x42\x42", :word_size => 2, :endian => :big)
# 00000000  1234 0000 4242                           |ሴ䉂|

:word_size can have any value, and is not restricted to powers of 2.

Now that Hexdump parses multi-byte words from data, displaying Unicode characters was the next logical step:

Hexdump.dump("\xb6\x80" * 10, :word_size => 2)
# 00000000  80b6 80b6 80b6 80b6 80b6 80b6 80b6 80b6  |肶肶肶肶肶肶肶肶|
# 00000010  80b6 80b6                                |肶肶|

Finally, Hexdump received some performance tuning which cut benchmark times in half on Ruby 1.9.2:

Before

                                user     system      total        real
hexdump (block)             7.740000   0.030000   7.770000 (  8.138029)
hexdump                     9.590000   0.050000   9.640000 ( 10.178203)
hexdump width=256 (block)   7.280000   0.020000   7.300000 (  7.534507)
hexdump width=256           8.130000   0.030000   8.160000 (  8.342448)
hexdump ascii=true (block)  7.740000   0.030000   7.770000 (  7.958550)
hexdump ascii=true          9.550000   0.050000   9.600000 (  9.803758)

After

                                 user     system      total        real
hexdump (block)              3.010000   0.010000   3.020000 (  3.529396)
hexdump                      5.430000   0.030000   5.460000 (  6.216174)
hexdump width=256 (block)    3.010000   0.020000   3.030000 (  3.308961)
hexdump width=256            4.700000   0.040000   4.740000 (  5.520189)
hexdump ascii=true (block)   3.050000   0.010000   3.060000 (  3.501436)
hexdump ascii=true           5.450000   0.040000   5.490000 (  6.352144)
hexdump word_size=2 (block)  7.420000   0.050000   7.470000 (  9.174734)
hexdump word_size=2          9.500000   0.070000   9.570000 ( 11.228204)
hexdump word_size=4 (block)  4.110000   0.030000   4.140000 (  4.849785)
hexdump word_size=4          5.380000   0.060000   5.440000 (  6.209022)
hexdump word_size=8 (block)  3.350000   0.070000   3.420000 (  4.147304)
hexdump word_size=8          4.430000   0.040000   4.470000 (  5.930758)

Using ruby-prof and the Rubinius Profiler (rbx -Xprofile) I found that the majority of time spent was in:

Proc#call
Array#join
Integer#chr
String#%

Optimizing out these excess method calls involved:

Frequently called lambdas were replaced with methods.
Array#join was replaced with incremental String concatenating code.
A lookup table of bytes and printable characters were added, to reduce excess Integer#chr calls.
Interestingly, Kernel#sprintf is slightly faster than String#%.

Uses

Both hexdump.js and Ruby Hexdump aim to provide similar features and behaviors. hexdump.js is definitely useful for when you want to offload the work of Hexdumping to the Browser, or when writing node.js Apps.

The Ruby Hexdump library is more suited for Ruby CLI Apps, but can also be used in Web Apps:

dump = []

Hexdump.dump(data) do |index,numeric,printable|
  dump << [index, numeric, printable]
end

render :json => dump

Don't like WEBrick? Try net-http-server

2011-05-08T00:00:00-07:00

TL;DR

net-http-server:

$ gem install net-http-server

pure-Ruby HTTP Parser and Server
Small codebase
Fast-ish
Rack-like API
Rack Handler included
full YARD Documentation

WEBrick

Some have said that the WEBrick HTTP Server is a Ghetto. While WEBrick is very fast for a pure-Ruby HTTP Server, the parsing code is hand written and difficult to read. WEBrick is also one of the oldest Ruby HTTP Servers, but for some reason lacks documentation coverage. Given the rise of Rack, middleware and Rack applications, WEBricks API now seems awkward.

The Parser

When Parslet (a pure Ruby PEG Parser library) was announced, I wondered how hard would it be to write a HTTP Parser with Parslet. After researching the other Ragel based HTTP Parsers (Thin and Unicorn) and double checking RFC 2616, I suddenly had a pure-Ruby HTTP Parser. (in one file, that you can actually read!)

I found that the way in which Parslet nested matches into Arrays of Hashes resulted in data that looked very much like a Rack env Hash.

require 'net/http/server/parser'

parser = Net::HTTP::Server::Parser.new
parser.parse("GET /path?x=1&y=2 HTTP/1.1\r\nCookie: xyz;123\r\n\r\n")
# => {
#      :method=>"GET",
#      :uri=>{:path=>"path", :query=>"x=1&y=2"},
#      :version=>"1.1",
#      :headers=>[{:name=>"Cookie", :value=>"xyz;123"}]
#    }

The Daemon

The next step was to write an actual Daemon that would:

Receive Connections
Parse HTTP Requests
Pass HTTP Requests to a Request Handler
Receive HTTP Responses from the Request Handler
Send HTTP Responses
Close Connections

I settled on using the battle tested GServer class to handle the Connections for Nett:HTTP::Server::Daemon. I also borrowed some ideas from Rack, such as passing HTTP Requests via a call method and returning HTTP Responses as an Array (containing the HTTP Status, Headers and Body).

require 'net/http/server'
require 'pp'

Net::HTTP::Server.run(:host => '127.0.0.1', :port => 8080) do |request,socket|
  pp request

  [200, {'Content-Type' => 'text/html'}, ['Hello World']]
end

The Rack Handler

Given that the API was already very Rack-ish, writing a Rack handler on top of Net::HTTP::Server::Daemon was simple.

require 'rack/handler/http'
require 'sinatra'

class HelloWorld < Sinatra::Base

  get '/' do
    [200, {'Content-Type' => 'text/html'}, ["Hello World"]]
  end

end

Rack::Handler::HTTP.run HelloWorld, :Host => 'localhost', :Port => 1212

Benchmarks

By now your probably wondering, how fast is this pure Ruby HTTP Server?

require 'net/http/server'

Net::HTTP::Server.run(:port => 8080) do |request,socket|
  [200, {'Content-Type' => 'text/html'}, ['Hello World']]
end

$ ab -n 4000 -c 4 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
...

Finished 4000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      4
Time taken for tests:   73.405 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Total transferred:      220000 bytes
HTML transferred:       44000 bytes
Requests per second:    54.49 [#/sec] (mean)
Time per request:       73.405 [ms] (mean)
Time per request:       18.351 [ms] (mean, across all concurrent requests)
Transfer rate:          2.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       8
Processing:    24   73  31.5     62     236
Waiting:       24   72  31.4     60     236
Total:         25   73  31.5     62     236

Percentage of the requests served within a certain time (ms)
  50%     62
  66%     86
  75%     98
  80%    103
  90%    119
  95%    134
  98%    148
  99%    153
 100%    236 (longest request)

Definitely not as fast as Thin or even WEBrick, but not too bad considering its pure-Ruby and the size of the code-base.

Typos in your Documentation?

2011-03-04T00:00:00-08:00

More likely than you think.

Having good documentation (internal and external) can make or break bigger projects. Good documentation helps skilled users/developers get up to speed with your project. However, bad documentation will drive potential users away. Typos are probably the most embarrassing thing to find in documentation.

One day I was fixing a trivial bug in my code for a new-user, and glanced over the documentation. Low and behold I spot a typo right above the broken method. In this moment of embarrassment (in front of the new-user), I said enough is enough, no more typos!

Since I use YARD for all of my projects, I decided to write a YARD plugin which would scan all of my documentation text using the Hunspell spellchecking library (via ffi-hunspell). A couple days later, yard-spellcheck was printing typos with file-names, line-numbers and ANSI highlighting.

As I suspected, I was able to find and fix a handful of typos in my own libraries, thanks to yard-spellcheck. Next, I started scanning larger projects with YARD documentation. I found a couple typos in DataMapper and even YARD itself (all are fixed now).

The lesson of this short story is that typos are lurking everywhere; too numerous and well hidden for human eyes to catch them all. Luckily, we now have an automated-tool in the fight against typos. The Ruby Community is notably obsessed with testing and quality of software, we should feel the same way about our documentation.

$ gem install yard-spellcheck
$ cd my_project/
$ yard-spellcheck

Introducing DeploYML

2010-11-21T00:00:00-08:00

When Rails 3.0.0.beta.1 was first released, there was some confusion about how to deploy a Rails3 app with this new Bundler thing. A few blog posts were written with some monkey patches for injecting into Capistrano, which enable early Bundler support. Unfortunately, Bundler was still very new and none of the deployment tools had official support yet.

Around the same time, I needed to deploy a Rails3 app and I did not want to deal with the heavy weight of Capistrano; nor did I want to rely on a monkey patch from a blog post. Of course the first thing I did was look at the alternatives, which were Vlad The Deployer or write a bash script that used rsync / ssh / git. After trying to run the vlad rake tasks, I hit some weird bugs with how it was calling git or thin. Being on a dead-line and not wanting to wait on Capistrano or Vlad, I thought it should be easy to create a simple deployment utility with git and ssh.

DeploYML

Introducing DeploYML, a simple deployment solution for Ruby / Rails projects that uses a single YAML file, Git and ssh.

Configuring DeploYML requires at least two things:

# config/deploy.yml
source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site

Then one can deploy the project using the deployml command:

$ deployml deploy

After making some changes, one can re-deploy the project:

$ deployml redeploy

Configuration

Of course, one will want to specify more information, such as what server to run the project under:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
server: apache

Or what options to run the server with:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
server:
  name: thin
  options:
    servers: 4
    deamonize: true
    socket: /var/run/thin.sock
    rackup: true

Or more importantly, what Framework or ORM the project uses:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
framework: rails3
orm: datamapper
server: apache

One can even specify multiple-environments:

# config/deploy.yml
source: git@github.com:user/project.git
framework: rails3
orm: datamapper

# config/deploy/staging.yml
dest: ssh://deploy@www.example.com/srv/staging
server:
  name: thin
  options:
    config: /etc/thin/staging.yml
    socket: /tmp/thin.staging.sock

# config/deploy/production.yml
dest: ssh://deploy@www.example.com/srv/project
server:
  name: thin
  options:
    config: /etc/thin/example.yml
    socket: /tmp/thin.example.sock

Administration

DeploYML does more than just deploying, it also allows interacting with the deployment server and deployed project.

Need to quickly ssh into the server as the deploy user?

$ deployml ssh

Need to quickly execute a command remotely, within the directory of the deployed project?

$ deployml exec 'ps aux'

Need to execute a rake task remotely?

$ deployml rake db:autoupgrade

Interested?

Install it today:

$ gem install deployml

Note: I originally started DeploYML before Imploy was released, and just now got around to making a release. If you feel Capistrano is too heavy, I encourage you to also checkout Imploy for deploying Rails3 apps.

Mining RubyGems from Ore

2010-10-25T00:00:00-07:00

Recently there has been some interesting discussion on the role of Gem builders and gemspecs. Jeff Kreeftmeijer wrote about how easy it is to build a RubyGem using a hand-written .gemspec file. With just a stand-alone .gemspec file one can:

Build RubyGems: gem build my-project.gemspec.
Publish RubyGems: gem push my-project-0.1.0.gem.

One thing worries me is the fact that you either have to specify everything explicitly in the gemspec, or use inline git commands to query the files of a project. Copying and pasting all this boilerplate code around seems like a potential future maintenance hassle; also not very DRY.

The argument for just a .gemspec file did get me thinking. I do find myself regenerating the gemspec with Jeweler, almost once per-day. Also, these stand-alone gemspecs are pretty succinct. So when in doubt, see how other languages solved the problem.

Code vs. Data

I asked one of my Haskell friends how Cabal (the Haskell packager of choice) solves this problem. He pointed me to the Cabal file of his Haskell webapp serialist.net. Cabal uses easy to read, easy to parse and easy to edit plain-text. This reminded me, Code is for describing logic and flat-files are for describing static data. The majority of the information in the .gemspec file, is static data.

Now I am starting to really question the whole reason for an explicit gemspec. The .gemspec file only exists to create a Gem::Specification object, which is then passed to Gem::Builder or loaded by Bundler. Why are we writing Ruby code that normally would only exist in-memory? The gemspec purists state this is to allow things such as, dynamically loading the VERSION constant from the project or dynamically listing files tracked by Git. Although, both of these tasks can easily be automated by a library.

So I wondered, why not have a small library that loads the project information from a YAML file, fills in any missing information, and then creates a new Gem::Specification object. Furthermore, If we can call a method and get a Gem::Specification object back, we could place this in the .gemspec file for both gem build and Bundler to make use of.

Introducing Ore

Ore allows you to define all project information for a RubyGem in a single YAML file (gemspec.yml).

name: ore
version: 0.1.2
summary: Cut raw RubyGems from YAML
description:
  Ore is a simple RubyGem building solution. Ore handles the
  creation of Gem::Specification objects as well as building '.gem'
  files. Ore allows the developer to keep all of the project information
  in a single YAML file.

license: MIT
authors: Postmodern
email: postmodern.mod3@gmail.com
homepage: http://github.com/postmodern/ore
has_yard: true

dependencies:
  thor: ~> 0.14.3

development_dependencies:
  yard: ~> 0.6.1
  rspec: ~> 2.0.0

With Ore, one can write their description as free-text, no more using with awkward Ruby %Q{...} syntax.

Dependencies are listed in a YAML Hash, no more repeating add_dependency or add_development_dependency.

Ore can also infer missing information. If the version is not specified, Ore will search for and parse any VERSION or VERSION.yml files. Ore can even slurp up any VERSION, MAJOR, MINOR, PATCH or BUILD constants from a version.rb file in the lib/ directory. Also, notice that files is not listed, this is because Ore can detect the project is using Git, and list all files tracked by Git.

For a complete reference of everything that may go into a gemspec.yml file, and how Ore infers missing data, please see GemspecYML.

Building gems with Ore is easy as:

$ ore
Successfully built RubyGem
Name: ore
Version: 0.1.2
File: ore-0.1.2.gem
$ ls pkg/
ore-0.1.2.gem

One can still get the traditional gemspec from Ore:

$ ore gemspec
# -*- encoding: utf-8 -*-

Gem::Specification.new do |s|
  s.name = %q{ore}
  s.version = "0.1.2"

  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
  s.authors = ["Postmodern"]
  s.date = %q{2010-10-25}
  s.default_executable = %q{ore}
...

We can even use Ore in .gemspec files:

# -*- encoding: utf-8 -*-

begin
  Ore::Specification.new do |gemspec|
    # custom logic here
  end
rescue NameError
  begin
    require 'ore/specification'
    retry
  rescue LoadError
    STDERR.puts "The 'my-project.gemspec' file requires Ore."
    STDERR.puts "Run `gem install ore-core` to install Ore."
  end
end

Ore will still work with gem build and even Bundler.

Mining RubyGems from Ore

Ore also comes with an extendable generator, for creating new projects:

$ mine my-project
Generating /home/hal/my-project
  create  lib
  create  lib/my/project
  create  spec
  create  .rspec
  create  spec/my/project_spec.rb
  create  spec/spec_helper.rb
  create  .document
  create  .gitignore
  create  my-project.gemspec
  create  ChangeLog.rdoc
  create  LICENSE.txt
  create  README.rdoc
  create  Rakefile
  create  gemspec.yml
  create  lib/my/project/version.rb
  create  lib/my/project.rb
     run  git init from "."
     run  git add . from "."
     run  git commit -m "Initial commit." from "."

By default mine will generate an RDoc and test-unit project. mine can also generate very customized projects:

$ mine my-project --rspec --yard --markdown --bundler
Generating /home/hal/my-project
      create  lib
      create  lib/my/project
      create  spec
      create  .yardopts
      create  .rspec
      create  spec/my/project_spec.rb
      create  spec/spec_helper.rb
      create  Gemfile
      create  .document
      create  .gitignore
      create  my-project.gemspec
      create  ChangeLog.md
      create  LICENSE.txt
      create  README.md
      create  Rakefile
      create  gemspec.yml
      create  lib/my/project/version.rb
      create  lib/my/project.rb
         run  git init from "."
         run  git add . from "."
         run  git commit -m "Initial commit." from "."

mine simply renders Ore Templates. Unlike other generators which have their logic hard-coded in Ruby, Ore Templates are simply directories, containing static and ERb files. One can make their own Ore template by creating a directory, adding files and publishing a Git repository.

Users can install custom Ore templates from Git repositories:

$ ore install http://github.com/user/awesometest.git
$ ore list
Builtin templates:
  base
  rspec
  test_unit
  yard
  bundler
  jeweler_tasks
  rdoc
  ore_tasks
Installed templates:
  awesometest

Then use the -T option to specify additional custom templates:

$ mine my-project --yard --markdown -T awesometest

Workflow

By default Ore does not impose a workflow onto the developer. Even the mine utility does not add any additional Rake tasks to new projects. This allows the developer to use gem build and gem push, or even use Jeweler::Tasks with Ore.

For those just wanting simple Rake tasks to build, push and tag releases, there is ore-tasks. Ore::Tasks provides the following tasks:

rake build            # Only builds a Gem
rake console[script]  # Start IRB with all runtime dependencies loaded
rake gem              # Alias to the 'build' task
rake install          # Builds and installs a Gem
rake push             # Builds and pushes a Gem
rake release          # Builds and Pushes a new Gem / Build, Tags and Pushe...
rake tag              # Tags a release and pushes the tag
rake version          # Displays the current version

To generate a project with Ore::Tasks included:

$ mine my-project --ore-tasks

To generate a project with Jeweler::Tasks included:

$ mine my-project --jeweler-tasks

Dog Fooding

In order to do real-world testing with Ore, I created ore-example which uses Bundler, RSpec2, YARD and Ore::Tasks.

As of now, I have also migrated my chars and uri-query_params libraries to Ore.

Interested?

$ gem install ore
$ mine the-future

For questions or feedback, join #ruby-ore on irc.freenode.net.

All source-code is located on GitHub.

Ore is tested with RSpec2 and has extensive YARD documentation.

Introducing Combinatorics 0.2.0

2010-10-03T00:00:00-07:00

I decided to take my List Comprehension code with some other Combinatorics code I had floating around and release Combinatorics 0.2.0:

$ gem install combinatorics

After getting specs on the List Comprehension code, it became easy to refactor it and get the specs passing on Ruby 1.8.7, 1.9.2 and JRuby. Unfortunately, Rubinius does not yet support Enumerator#next.

The Combinatorics library also contains the powerset method, added to Array and Set:

require 'combinatorics'

[1,2,3].powerset
# => [[],
      [3],
      [2],
      [2, 3],
      [1],
      [1, 3],
      [1, 2],
      [1, 2, 3]]

In Combinatorics 0.2.0, Range#&, Range#upto and Range#downto were also added:

(1..50) & (20..100)
# => (20..50)

(1..5).upto(2..10).to_a
# => [1..5, 1..6, 1..7, 1..8, 1..9, 1..10,
      2..5, 2..6, 2..7, 2..8, 2..9, 2..10]

(2..10).downto(1..5).to_a
# => [2..10, 2..9, 2..8, 2..7, 2..6, 2..5,
      1..10, 1..9, 1..8, 1..7, 1..6, 1..5]

Fork the Combinatorics library today, and add your own Combinatoric methods.

List Comprehensions in Ruby

2010-10-01T00:00:00-07:00

Recently I have been interested in Haskell and been skimming the whimsical (yet informative) Learn a Haskell, for Great Good!. One feature I really like from Haskell is their implementation of list comprehensions:

Prelude> [(x,y) | x <- [1..10], y <- [2..8]]
[(1,2),(1,3),(1,4),(1,5),(1,6),(1,7),(1,8),
(2,2),(2,3),(2,4),(2,5),(2,6),(2,7),(2,8),
(3,2),(3,3),(3,4),(3,5),(3,6),(3,7),(3,8),
(4,2),(4,3),(4,4),(4,5),(4,6),(4,7),(4,8),
(5,2),(5,3),(5,4),(5,5),(5,6),(5,7),(5,8),
(6,2),(6,3),(6,4),(6,5),(6,6),(6,7),(6,8),
(7,2),(7,3),(7,4),(7,5),(7,6),(7,7),(7,8),
(8,2),(8,3),(8,4),(8,5),(8,6),(8,7),(8,8),
(9,2),(9,3),(9,4),(9,5),(9,6),(9,7),(9,8),
(10,2),(10,3),(10,4),(10,5),(10,6),(10,7),(10,8)]

This list comprehension is essentially a Set definition using first-order predicate logic, with universal quantifiers on x and y (where the Set contains the tuples composed of every x and every y from the ranges 1..10 and 2..8). Amazingly, the Haskell version looks almost exactly like the Set definitions from my Discrete Structures, Logic, and Computability (2nd edition) book, except without the curly-braces or universal quantifiers ().

One does not have to learn functional programming and Haskell to use list comprehensions, Python has them as well:

>>> [(x,y) for x in range(0,11) for y in range(2,9)]
[(0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8),
(1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 7), (1, 8),
(2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (2, 7), (2, 8),
(3, 2), (3, 3), (3, 4), (3, 5), (3, 6), (3, 7), (3, 8),
(4, 2), (4, 3), (4, 4), (4, 5), (4, 6), (4, 7), (4, 8),
(5, 2), (5, 3), (5, 4), (5, 5), (5, 6), (5, 7), (5, 8),
(6, 2), (6, 3), (6, 4), (6, 5), (6, 6), (6, 7), (6, 8),
(7, 2), (7, 3), (7, 4), (7, 5), (7, 6), (7, 7), (7, 8),
(8, 2), (8, 3), (8, 4), (8, 5), (8, 6), (8, 7), (8, 8),
(9, 2), (9, 3), (9, 4), (9, 5), (9, 6), (9, 7), (9, 8),
(10, 2), (10, 3), (10, 4), (10, 5), (10, 6), (10, 7), (10, 8)]

Granted, using the for loop syntax as constraints for the list comprehension is not as sexy as the Haskell version, but it gets the job done.

Unfortunately, Ruby does not support list comprehensions, and only has a couple methods for doing Combinatorics (Array#combination and Array#permutation). So, I implemented list comprehensions for the Ruby Array class:

>> require './array_comprehension'
=> true
>> [(1..10), (2..8)].comprehension.to_a
=> [[1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8],
    [2, 2], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8],
    [3, 2], [3, 3], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8],
    [4, 2], [4, 3], [4, 4], [4, 5], [4, 6], [4, 7], [4, 8],
    [5, 2], [5, 3], [5, 4], [5, 5], [5, 6], [5, 7], [5, 8],
    [6, 2], [6, 3], [6, 4], [6, 5], [6, 6], [6, 7], [6, 8],
    [7, 2], [7, 3], [7, 4], [7, 5], [7, 6], [7, 7], [7, 8],
    [8, 2], [8, 3], [8, 4], [8, 5], [8, 6], [8, 7], [8, 8],
    [9, 2], [9, 3], [9, 4], [9, 5], [9, 6], [9, 7], [9, 8],
    [10, 2], [10, 3], [10, 4], [10, 5], [10, 6], [10, 7], [10, 8]]

I used yield to quickly pass results to the given block, to not stress the Garbage Collector with building a huge Arrays. The method also returns an Enumerator object if no block is given, to fake Haskells lazy evaluation.

Note, that the Array may contain:

Enumerable objects:

  >> [('a'..'f'),(0..10).step(2)].comprehension.to_a
  => [["A", 0], ["A", 2], ["A", 4], ["A", 6], ["A", 8], ["A", 10],
      ["B", 0], ["B", 2], ["B", 4], ["B", 6], ["B", 8], ["B", 10],
      ["C", 0], ["C", 2], ["C", 4], ["C", 6], ["C", 8], ["C", 10],
      ["D", 0], ["D", 2], ["D", 4], ["D", 6], ["D", 8], ["D", 10],
      ["E", 0], ["E", 2], ["E", 4], ["E", 6], ["E", 8], ["E", 10],
      ["F", 0], ["F", 2], ["F", 4], ["F", 6], ["F", 8], ["F", 10]]

Non-enumerable objects:

  >> [5,(6..9),10].comprehension.to_a
  => [[5, 6, 10], [5, 7, 10], [5, 8, 10], [5, 9, 10]]

Even other list comprehensions:

  >> syllable = (['a'..'z'] * 2).comprehension
  >> word = [syllable] * 4
  >> number = (1..100)
  >> [*word, number].comprehension { |*s| puts s.join }

This code does require Ruby 1.9 since Enumerator#peek was not back-ported to Ruby 1.8.7.

$ rvm install 1.9.2

Introducing OpenNamespace

2010-09-03T00:00:00-07:00

I have always liked how frameworks such as Rails can autoload Classes. I wanted to provide similar behavior in Ronin and other frameworks, so I created the OpenNamespace library.

OpenNamespace allows namespaces to require and find classes and modules from RubyGems. Using OpenNamespace you can make a Plugins module able to load plugin modules/classes from other gems.

More specifically, OpenNamespace does not need to know where the files are, it just guesses the file path based on the constant name, attempts to require it then finds the constant in the namespace.

$ gem install open_namespace

Examples

require 'open_namespace'

module Project
  module Plugins
    include OpenNamespace
  end
end

Explicitly load constants:

Project::Plguins.require_const :foo_bar
# => Project::Plugins::FooBar

Explicitly load constants with odd capitalization:

Project::Plugins.require_const :tcp_session
# => Project::Plugins::TCPSession

Explicitly load constants via sub-paths:

Project::Plguins.require_const 'templates/erb'
# => Project::Plugins::Templates::Erb

Implicitly load constants via const_missing:

Project::Plugins::Other
# => Project::Plugins::Other

Enjoy.

Introducing uri-query_params

2010-09-02T00:00:00-07:00

uri-query_params is a new library which allows accessing the individual parameters in the query string of a HTTP URI in Ruby. The library monkey-patches the URI::HTTP class to provide similar behavior to PHPs famous $_GET hash.

$ gem install uri-query_params

Using uri-query_params you can, inspect the URI query_params:

require 'uri/query_params'

url = URI('http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=1HY&q=bob+ross&btnG=Search')

url.query_params
# => {"btnG"=>"Search", "hs"=>"1HY", "rls"=>"org.mozilla:en-US:official", "client"=>"firefox-a", "hl"=>"en", "q"=>"bob+ross"}

url.query_params['q']
# => "bob+ross"

Also, set the URI query_params:

url.query_params['q'] = 'Upright Citizens Brigade'
url.to_s
# => "http://www.google.com/search?btnG=Search&hs=1HY&rls=org.mozilla:en-US:official&client=firefox-a&hl=en&q=Upright%20Citizens%20Brigade"

Originally, the URI::QueryParams mixin was developed for Ronin and GScraper. I decided to split the code out into a common library, to prevent conflicts between the two separate versions.

Enjoy.

SimpleMarkup is dead. Long live MarkDown (and Textile)

2010-02-02T00:00:00-08:00

If you use RDoc, than you're probably familiar with it's markup, known as SimpleMarkup. If you've been using RDoc for a while, you are also probably aware that SimpleMarkup is horribly broken, when it comes to handling odd formatting.

Besides the usual formatting mistakes that can easily crash SimpleMarkup, I and other users began noticing odd error messages when installing YARD documentation with SimpleMarkup:

Unhandled special: Special: type=65, name=CROSSREF,_SPECIAL_, text="Spidr"

Even after ditching RDoc for YARD, SimpleMarkup (the default markup used by YARD) was able to reach forth from it's icy RubyGem and introduce bugs into my code. This was the last straw, SimpleMarkup is dead, time to move on.

Luckily, YARD supports using alternate markup engines, such as MarkDown or Textile. Having used MarkDown regularly in the past, and that it matched SimpleMarkup's syntax in some ways, I decided to convert all of my Ruby project's to MarkDown formatted YARD documentation.

First, I had to specify the --markup markdown option in the yard_options setting (provided by the hoe-yard plugin) in the Rakefile.

Second, I renamed the README.*, History.* files to README.md, History.md, respectively.

Finally, I began the tedious process of converting SimpleMarkup formatted text to MarkDown:

Replace the =, ==, === headings with #, ##, ###.
Indent all source-code examples not following a YARD @example tag by 4 spaces.
Convert any raw URLs to MarkDown links.
Replace any +code+ or <tt>code</tt> formatting with `code` or <code>code</code>. (Note: `code` can handle spaces, numbers and symbols within the ` ` characters, where as +code+ cannot.)

In conclusion, that is how I ridded myself of the last buggy vestiges of RDoc, and increased the reliability of my documentation using YARD.

Introducing static_paths 0.1.0

2010-01-25T00:00:00-08:00

Update 2: static_paths has now be renamed to data_paths with documentation available on rubydoc.info.

Update: Documentation has now been posted on static-paths.rubyforge.org.

When writing a library which bundles static-content in a directory named data/ or static/, you usually need to find the directory using something like.

File.expand_path(File.join(File.dirname(__FILE__),'..','..','..','..','static'))

But then you want to allow plugins to extend the functionality of your library, and even add their own static-content. Now you have to manage more than one directory path.

Enter StaticPaths, a Ruby library for managing and searching through directories containing static-content. StaticPaths helps manage directories across multiple libraries in much the same way that RubyGems manages lib/ directories using $LOAD_PATH. Except that StaticPaths does not use global variables.

Example Usage

require 'static_paths'

module MyLibrary
  include StaticPaths

  # define the static dir(s)
  register_static_dir File.join(File.dirname(__FILE__),'..','..','static')
end

register_static_dir will expand the path, make sure the path points to a directory, then adds the path to StaticPaths.paths and the local MyLibrary.static_paths.

One can also unregister directories using unregister_static_dir!.

MyLibrary.unregister_static_dir! File.join(File.dirname(__FILE__),'..','..','static')

Or, one can use unregister_static_dirs! to unregister all directories registered within a module/class.

module MyLibrary
  def MyLibrary.cleanup!
    unregister_static_dirs!
  end
end

To search through and access the content within registered directories, simply use the methods within the StaticPaths::Finders module.

module MyLibrary
  class UsesContent

    include StaticPaths::Finders

    def index
      find_static_file('index.html')
    end

    def file_dirs
      all_static_dirs('extra')
    end

    def copy(src,dest)
      each_static_dir(src) do |dir|
        FileUtils.cp_r(dir,dest)
      end
    end

  end
end

Installation:

$ sudo gem install static_paths

(Fully) Document your DataMapper models with YARD

2010-01-16T00:00:00-08:00

Any release quality software has to provide documentation, for the future maintainers and other developers. Traditionally, Ruby projects would use RDoc and add custom documentation blurbs to their classes, methods, attributes and constants. Unfortunately, there's a major limitation to RDoc, I can't be extended to recognize new meta-programming method calls.

This rigidness of RDoc really shows up when you need to document ORM backed projects, such as one containing DataMapper models. DataMapper allows one to define the properties and relations between models using handy class-methods:

class MyModel

  include DataMapper::Resource

  # The primary key of the model
  property :id, Serial

  # The name of the model
  property :name, String

  # The many authors contributing to the model
  has n, :authors

  # The user that owns the model
  belongs_to :user

end

RDoc will not recognize property, has or belongs_to. Nor will RDoc know that property adds a class-method and instance reader/writer methods with the given name to the model. Documentation fail.

Enter YARD

YARD is a documentation generation tool for the Ruby programming language. It enables the user to generate consistent, usable documentation that can be exported to a number of formats very easily, and also supports extending for custom Ruby constructs such as custom class level definitions.

http://github.com/lsegal/yard

YARD organizes most of it's parsing logic into multiple handlers; essentially classes that inherit YARD::Handlers::Ruby::Base and define a process method. YARD also supports a plugin system, by loading any RubyGems installed on the system that are prefixed with yard- or yard_. Using these two features of YARD, one can easily create a YARD plugin gem containing custom handlers, which YARD can automatically load and use.

yard-dm

yard-dm is a YARD plugin for parsing DataMapper model syntax. The plugin can handle the following statements:

property :name, Type
has n, :things
has 1, :thing
has 0..n, :things
has 1..n, :things
has 2..5, :things
belongs_to :stuff

Install It

$ sudo gem install yard-dm

Use It

$ cd dm-project/
$ yardoc

It's that easy.

Next time you need complete documentation on your DataMapper backed project, just install YARD and yard-dm.

Using YARD with Hoe, correctly

2010-01-11T00:00:00-08:00

When I first switched from the more traditional RDoc to YARD, I was only using the basic YARD task. In YARD 0.2.3.5, a gem plugin was added to YARD that would run yardoc on an installed gem if has_rdoc was set to "yard" in the Gem Specification. The yardoc command would also use the rdoc_options and extra_rdoc_files options from the Gem Specification.

Since I use Hoe for all my gem publishing needs, I got tired of having to specify all of this information explicitly in my Rakefiles; after all Hoe is suppose to make one's Rakefiles simple and elegant. So I took advantage of Hoe's plugin system and created a Hoe plugin to properly configure my projects for YARD.

Introducing hoe-yard, a Hoe plugin that will:

Automatically find the README and History files, irregardless of file extension.
Sets has_rdoc to yard.
Sets rdoc_options.
Sets extra_rdoc_files.
Adds YARD and hoe-yard as development dependencies.
Adds the yard and docs Rake tasks.

Install

$ sudo gem install hoe-yard

Usage

require 'rubygems'
require 'hoe'

Hoe.plugin :yard

Hoe.spec('my_project') do
  # ...

  # optional YARD settings
  self.yard_title = 'My Project (0.1.0)'
  self.yard_markup = :markdown
  self.yard_opts = ['--protected'] # any additional YARD options

  # ...
end

Not only does hoe-yard make setting up YARD with a project extremely easy, it also ensures the documentation that is automatically generated when a gem is installed will match the documentation published for that gem.

Spidr 0.2.2 released.

2010-01-11T00:00:00-08:00

Spidr 0.2.2 (code-named "next-level") has been released. This release contains a lot of changes that pushes Spidr into a new level of web spidering.

Web Spider Obstacle Course (WSOC)

Spidr 0.2.2 now requires and makes use of the new Web Spider Obstacle Course (WSOC) for testing. Before one runs the RSpec test-suite for Spidr, the WSOC server must first be started:

$ wsoc_server

Then simply run the specs as usual:

$ rake spec

Cookie support

As of 0.2.2, Spidr now comes with a CookieJar, thanks to the work of @zapnap. Now when the Spidr::Agent visits a page, any new cookie values will be merged into the CookieJar, and sent back with any future requests. Additionally, one can now access the Cookie values from a Spidr::Page object.

page.cookie
# => "COUNTRY=USA%2C97.100.45.38; expires=Mon, 18-Jan-2010 06:19:24 GMT; path=/; domain=.php.net"

page.cookies
# => ["COUNTRY=USA%2C97.100.45.38; expires=Mon, 18-Jan-2010 06:19:24 GMT; path=/; domain=.php.net"]

HTTP Basic Auth support

Spidr 0.2.2 now comes with a brand new AuthStore, for organizing HTTP Authentication credentials; also thanks to the work of @zapnap. Provided you have the credentials for the various HTTP Basic Auth protected areas that are to be spidered, Spidr can automatically respond to Basic Auth challenges. Simply specify the credentials to the Spidr::Agent and the agent will do the rest:

Spidr.host('corporation.com') do |agent|
  agent.authorized.add('http://corporation.com/private/', 'user1233', 'motivate synergize')

  agent.every_page do |page|
    if page.url.path =~ /private/
      # ...
    end
  end
end

URL Sanitization

A small yet important module was added in Spidr 0.2.2, and that is Spidr::Sanitizers. The Sanitizers module adds configuration settings to Spidr::Agent for how incoming URLs are to be sanitized.

For instance, URL fragments are removed by default, but this can be changed:

agent.strip_fragments
# => true
agent.strip_fragments = true

Additionally, perhaps one might wish to strip the query strings from all URLs:

agent.strip_query = true

Note: If YARD documentation generation fails when installing Spidr 0.2.2, this is due to a bug in RDoc/SimpleMarkup generation.

Introducing the new Web Spider Obstacle Course

2010-01-11T00:00:00-08:00

The Web Spider Obstacle Course was originally developed for testing the Spidr library. The Obstacle Course was essentially a set of static files containing valid links, invalid links, links to previously visited pages and links to non-existent hosts/ports. Having the course hosted on spidr.rubyforge.org made it difficult to do advanced testing (such as using HTTP Redirects), not to mention slow to access.

Now the Web Spider Obstacle Course (WSOC) has been rewritten as a Ruby Sinatra app that can be ran locally. Since most Ruby developers are familiar with Sinatra, it should not be too difficult to extend the Obstacle Course.

Currently WSOC tests a Web Spider's ability to navigate:

Empty links.
Circular links.
Links with relative-paths.
Links with absolute-paths.
Remote links.
javascript: links.
Links within frameset and iframe tags.
Cookie protected pages.
HTTP 300, 301, 302, 303 and 307 Redirects.
HTTP Baisc Auth protected pages.

3... 2... 1... Go

Install:

$ sudo gem install wsoc

Run:

wsoc_server -p 8080

Visit:

firefox http://localhost:8080/

The front page of WSOC will explain the rules of the Obstacle Course and providing the Start URL for any would-be Web Spiders. The WSOC server still contains the Obstacle Course Specs, available at http://localhost:8080/specs. The Obstace Course Specs are essentially a list of URLs, the expected behavior of the spider for the URL (visit, ignore or fail) and a message about the given URL. These specs can be access by Web Spider test suites in JSON (http://localhost:8080/specs.json) or YAML (http://localhost:8080/specs.yaml) formats.

Use It

If you actively maintain a Web Spider, or are thinking of writing a Web Spider/Crawler/Scanner, I highly recommend you use the Web Spider Obstacle Course as part of your testing solution. The Obstacle Course will test more than a Web Spiders ability to resolve relative links or only visit a link once.

The internet is a wild place, and will not always obey XHTML 1.1 Strict or HTTP 1.0 standards. Test your Web Spiders fully before setting them loose.

ToorCamp: Ronin - A Platform for Publishing and Mayhem

2009-11-01T00:00:00-07:00

I was pretty surprised and pleased when ToorCon finally released videos of the talks at ToorCamp 2009. At this years ToorCon: San Diego DVDs of the talks, pressed by MediaArchives, were handed out. Thanks to Mr. Pierce for grabbing a copy of my talk on Ronin and getting it back to me for transcoding/uploading. The full talk has been transcoded to Ogg Theora and uploaded to vimeo.com. If you have a Vimeo account, you can download the original ogg file as well.

ToorCamp: Ronin - A Platform for Publish and Mayhem

Original slides can be found here, in XHTML form.

Thanks go out to flatline and evoltech for going with me to ToorCamp, providing transportation and helping hitch-hike back when our van broke down. Also extra thanks to evoltech for letting me use his SUV sized laptop to run the slides on.

Introducing Wordlist 0.1.0

2009-10-21T00:00:00-07:00

Update: Evoltech correctly pointed out that the Wordlist::Builders::Website example was incorrect. The Website.build method must take a path argument and then any additional options.

For a while now people have asked/told me that I should write a Wordlist library for Ruby. People have also brought up the need to build wordlists from existing corpora; such as the text from a website. After seeing varying attempts at this task, I finally got serious and released Wordlist 0.1.0.

Currently, the design of the Wordlist Ruby library is quite simple. Wordlist::Builder is the base class for all wordlist builders. It handles parsing text, filtering out duplicate words, and writing the unique words to a file. The Builder class stores the CRC32 hash of each word in a Hash, mapping the word length to the Set of CRC32 hashes. Using a bucket system for the CRC32 hashes, keeps both word lookup time and memory usage to a minimum.

Wordlist::Builder.build('list.txt') do |builder|
  builder.parse(some_text)
  builder.parse_file('some/file.txt')
end

Wordlist also provides Wordlist::Builders::Website, which uses Spidr to completely spider a host, and build a wordlist from the inner-text recovered from every visited page.

require 'wordlist/builders/website'

Wordlist::Builders::Website.build('list.txt',:host => 'www.example.com')

Wordlist doesn't just build wordlists, it can also enumerate them. Wordlist defines Wordlist::List as the base class for all wordlist readers. Per convenience, Wordlist also defines Wordlist::FlatFile which handles the enumeration of flat-file wordlists.

list = Wordlist::FlatFile.new('list.txt')
list.each_word do |word|
  puts word
end

The Wordlist::List also provides methods for defining text-mutation rules, which are applied in series to each enumerated word.

list.mutate 'o', '0'
list.mutate '@', 0x41
list.mutate(/[hax]/i) { |match| match.swapcase }

list.each_mutation do |word|
  puts word
end

A mutation rule takes a String or Regexp (which is used to match a sub-string within a word) and another String, Integer or block which is used as a replacement for the matched sub-string. A mutation rule will perform every possible combination of sub-string replacement, passing each mutated word to the next mutation rule, and so on.

One can install Wordlist using rubygems, by simply running the following command:

$ sudo gem install wordlist

The git repository for Wordlist is located on GitHub.

As of 0.1.0, Wordlist is fairly simple but it gets the job done. The library was fun to design and test, as it involved borrowing some techniques typically used in blind-fuzzing. Expect more features and updates to Wordlist as time goes on.

Ronin "big-push" 0.3.0 released

2009-10-20T00:00:00-07:00

The long wait is over. Ronin 0.3.0, code-named "big-push", has finally been released.

DataMapper 0.10.0

Ronin was probably one of the first big Ruby apps to upgrade to DataMapper 0.10.0. As a result of this, we had to postpone the release of Ronin 0.3.0, until DataMapper 0.10.0 had finished it's rigorous two month Q/A cycle.

Once DataMapper 0.10.0 had been released, the two months of Q/A and almost a complete rewrite of the code-base really showed itself. I think @dkubb, @dbussink, @snusnu, @xshay, @myabc, @rsim, @namelessjon, @knowtheory and everyone else who helped with 0.10.0 deserves applause for their hard work and dedication.

YARD

As of 0.3.0, Ronin has successfully moved to YARD based documentation. The new Ronin API docs can be found in the usual location. YARD really helped improve the detail of the Ronin API docs, allowing developers to specify method arguments, accepted object-types, yield arguments, possible exceptions and return-types.

Additionally, YARD handlers were added to Ronin that parsed and regenerated documentation for DataMapper properties/relations. These additional YARD handlers helped improve documentation coverage for Ronin models, such as Ronin::Arch or Ronin::Exploits::Exploit.

Ronin::UI::CommandLine

Ronin::UI::CommandLine was almost entirely refactored in 0.3.0. Ronin::UI::CommandLine::Command was rewritten to use Thor. Switching to Thor greatly simplified the ronin commands, making it easier to write new commands. A good example of this, is the source-code for the install command from before and after 0.3.0.

Another benefit to using Thor, albeit cosmetic, was being able to use Thor's ANSI color-output. Normal output will now appear in green, warnings in yellow and error messages in red.

New convenience methods were also added to Ronin::UI::CommandLine::Command to help format and print various data: indent, print_title, print_array and print_hash.

Ronin::UI::Output

As of 0.3.0, Ronin::UI::Diagnostics and Ronin::UI::Verbose were replaced with the brand new Ronin::UI::Output module. Ronin::UI::Output is designed to delegate how output is handled. By default all output is sent to Ronin::UI::Output::Handler, which prints the output to the terminal. One can override the default output handler by using Ronin::UI::Output.handler=, to specify the new handler module; which must define print_info, print_debug, print_warning and print_error methods.

Ronin::UI::Output also provides the Helpers module, which can be included into any class and adds the print_info, print_debug, print_warning and print_error methods.

New Convenience Methods

New TCP and UDP networking methods were added to 0.3.0. Now one can easily create TCP/UDP servers using, Net.tcp_server, Net.tcp_server_session, Net.tcp_single_server, Net.udp_server and Net.udp_server_session.

Ronin::Network::HTTP::Proxy was also added to 0.3.0. The Ronin::Network::HTTP::Proxy helps parse and represent HTTP proxy addresses, and is now used by Ronin::Network::HTTP.proxy and all :proxy options.

proxy = Network::HTTP::Proxy.parse('148.233.239.24:80')
# => #<Ronin::Network::HTTP::Proxy: 148.233.239.24:80>

The Proxy class can also be used to test a proxy.

proxy.valid?
# => true
proxy.lag
# => 0.003881

Ronin::Platform

The overlay.xsl file got a make-over in 0.3.0. overlay.xsl is a XML StyleSheet, which helps render ronin.xml files from Overlays into proper XHTML when they are viewed in a web-browser. Now ronin.xml files should appear more like a standard "About" web-page.

Reloading Overlays/Extensions while ronin is running became possible with the addition of Ronin::Platform.reload!. Now if an Extension is modified, one can simply call Platform.reload! and get the updated version.

Ronin Extensions gained convenience methods for defining reader/writer methods to instance variables: attr_reader, attr_writer and attr_accessor. They can be used in Ronin Extensions, just as one would use them in a class.

ronin_extension do

  attr_accessor :var
  attr_reader :result

  setup do
    @var = 5
    @result = nil
  end
  ...
end

Ronin Extensions also gained their own on-demand temp directory. When Ronin::Platform::Extension#tmp_dir is called, a temp directory specifically for the extension will be created within ~/.ronin/tmp/, and the path of the new temp directory will be returned.

Upgrading

As always, the rubygem files for Ronin 0.3.0 are available on rubyforge.org. One can update Ronin using rubygems by simplying running:

$ gem update ronin

Likewise, one can also install Ronin using rubygems:

$ gem install ronin

Also, as of Ronin 0.3.0 all releases of Ronin will also be made available on gemcutter.org.

New project mailing lists

2009-10-18T00:00:00-07:00

As I've begun to get increasing amounts of email from other developers and users, I've wondered whether it's time to setup some mailing-lists.

Well, thanks to the efforts of Franck D'agostini, Spidr now has a mailing-list. I hope this mailing-list will help foster interesting discussions between users/developers and possible improvements to Spidr.

I also took the liberty of setting up an IRC channel for Spidr, since IRC is still a mainstay of developers (although I like XMPP a little more). Just join #spidr on irc.freenode.net.

Seeing the potential for how mailing-lists can help a project, I also setup a Google Group for Ronin. Obvious Privacy Warning: Since Google indexes pretty much everything, sketchy discussions relating to Ronin should probably happen somewhere beyond Google's reach, for one's own security and privacy.

So yeah, let's see what will happen.

Spidr "solid" 0.2.0

2009-10-14T00:00:00-07:00

After a period of refactoring, Spidr 0.2.0 (code-named "solid") has been released. Many things we're added to this release, along with some very important bug-fixes and optimizations.

Major changes

Spidr, along with many of my other projects, has moved to YARD based documentation. YARD's tag based documentation format really helped me annotate every method within Spidr. The new YARD docs can be found in the usual location.

Also following suite with many of my other projects, you can now find Spidr on the awesome and easy to use gemcutter.org.

Spidr should be a little faster now. Thanks to the work of justfalter, HTTP sessions with unique hosts/ports are now cached and resued. HTTP sessions no longer have to be re-initialized upon every request. Also, the history and failures lists are now Ruby Sets, yielding improved lookup times for checking if a link has been previously visted.

The code-base of Spidr should be a little more organized. Many methods within Spidr::Agent were grouped by functionality and moved to separate modules (Events and Filters) which are included back into Spidr::Agent.

The Spidr::Actions module was also added, which adds action methods that control spidering from within every_url, every_page, every_failed_url, all_headers event hooks. The pause! method will pause the spider, while skip_page! and skip_link! can manipulate the processing of pages/links.

The Spidr::Page#search, Spidr::Page#at, Spidr::Page#title methods were also added. These methods should make Spidr::Page feel alittle more like WWW::Mechanize::Page, allowing one to search the DOM (parsed by Nokogiri) with XPath/CSS-path expressions.

Lastly, many new examples which highlight some of the less-than-trivial things you can do with Spidr have been added to the website and docs.

Important bug-fixes

Spidr::Agent should now properly handle the spidering of SSL protected websites. Also thanks to justfalter, HTTPS sessions are now properly initialized and stored in the HTTP session cache; so the SSL-handshake only need be performed once per unique host/port.

Spidr::Agent#get_page will now correctly send the URI query along with the URI path for HTTP requests. Thanks go out to Damian Steer for reporting this.

Spidr::Page#doc now returns a Nokogiri::XML::Document object for RSS/RDF/Atom pages, allowing one to properly search RSS/Atom feeds.

Spidr::Page#code will now return the HTTP Status code as an Integer.

Spidr::Page#links now properly handles the HTTP Location header.

The URI expansion/normalization performed by Spidr::Page#to_absolute was greatly improved. Spidr::Page#to_absolute will now properly preserve trailing '/' characters on URI paths.

Shout Outs

A big thanks to everyone who helped with Spidr 0.2.0 by reporting bugs and testing new code. I hope this release will help users get more out of Spidr.

Ruby things worth looking into

2009-09-26T00:00:00-07:00

Over the past couple of months, I've been researching and playing with various up-and-coming Ruby libraries/services. So I thought it would be useful for other developers if I wrote something up.

Sinatra

If you already haven't heard, Sinatra is cool sauce. Sinatra is a Ruby Domain Specific Language for creating web-apps that can fit in a single file.

# myapp.rb
require 'rubygems'
require 'sinatra'
get '/' do
  'Hello world!'
end

$ gem install sinatra
$ ruby myapp.rb

It's that easy. But let's say you want to create an app accessible as a Class, just inherit Sinatra::Base.

require 'rubygems'
require 'sinatra'

class MyApp < Sinatra::Base

  get '/' do
    'Hello world!'
  end

end

I found the URL param parsing and request routing on-par with that of Merb/Rails. I also enjoyed the halt and pass methods, which will cause Sinatra to re-route a request to other handlers.

Defining helper methods which can be used within request handlers was as easy as creating a module with some instance methods, then registering the module using the helpers method.

module Helpers
  module Rendering
    include Rack::Utils

    alias h escape_html
  end
end

class MyApp < Sinatra::Base

  helpers Helpers::Rendering

end

Sinatra is built on top of Rack, so reviewing the Rack API might not hurt. The current Rack::Request object can be accessed via the request method. The return value of a request handler can be a String, Array of Strings or even a Rack::Response object. Sinatra also provides the Rack call method, which can be overridden for more custom request routing (such as vhost routing to other Rack apps). Given Sinatra's API Documentation, it's fairly easy to customize and extend your Sinatra app.

Thor

Thor is an alternative to Rake or Sake. Thor makes it very easy to define tasks as methods, and define command-line options/arguments for those tasks using a succinct syntax.

Thor tasks can be invoked via the thor command, or loading the Thor class and calling the start method directly.

I also noticed that Rails3 is using Thor::Group and Thor::Actions, to create code generators.

While documentation is scarce for Thor, I fould it's code-base fairly readable.

FFI

There's an easier way to write Ruby extensions to C libraries. It's called FFI, or Foreign Function Interface. Ruby FFI is a rubygem (named ffi) by Wayne Meissner which wraps around libffi, to provide a Ruby interface to load libraries, attach functions, variables and mapping in Structs/Unions/enums/callbacks.

To create a Ruby interface to a library, simply create a module which extends FFI::Library and attach some functions as module methods.

module FFI
  module TRE
    extend FFI::Library

    ffi_lib 'tre' # finds and loads the library

    # attaches a function named tre_regcomp, and defines the
    # types for it's arguments and return value.
    attach_function :tre_regcomp, [:pointer, :pointer, :int], :int
    # ...
  end
end

More examples can be found here.

If you need to quickly write bindings for a library, give FFI a shot. Another benefit of having FFI Ruby bindings, is that they will work seamlessly with JRuby and Rubinius, on any platform that is supported by libffi.

YARD

YARD is an alternative to the defacto RDoc documentation generation, that allows one to annotate code using a @tag based syntax and Markdown/Textile/RDoc formatting. YARD stores all the gathered documentation information in a Marshalable data-store file (.yardoc), which opens the door to on-the-fly documentation searching. YARD can also export it's data to XML or XHTML+CSS+jQuery.

Want to use YARD in your project? Drop this task into your Rakefile.

YARD::Rake::YardocTask.new do |t|
  t.files   = ['lib/**/*.rb']
  t.options = [
    '--protected',
    '--files', 'History.txt',
    '--title', 'MyProject'
  ]
end

task :docs => :yardoc

$ rake docs && firefox doc/index.html

Want to use YARD on someone else's (github) project, checkout rdoc.info, it's built on YARD and Ruby 1.8.7.

YARD can also be extended to detect and document meta-programming methods which define other methods at runtime. For example, this YARD handler (for Ruby 1.9.x) which documents parameter method-calls from the Parameters library.

Loren Segal's slides from his YARD presentation at Montreal.rb are also worth checking out.

Gemcutter

Gemcutter provides easy and fun RubyGem hosting, which is open-source and aims to become the the default gem hosting solution for Ruby. Did I mention that it looks classy (thanks to a redesign by ThoughtBot)?

Publishing gems to Gemcutter is as easy as running:

gem push pkg/my_project-0.1.0.gem

The gem push command can override previously uploaded versions of a gem. No more going into Rubyforge's File Admin panel to delete mistakenly uploaded versions, just run gem push again.

Also, Gemcutter is a Sinatra app running on Heruko.

Ronin "toorcamp" 0.2.4 finally released

2009-07-19T00:00:00-07:00

Update: As CG points out, I forgot to post the slides from the ToorCamp presentation. Here they are in XHTML form.

Update: @sanitybit discovered that parameters 0.1.7, which is required by ronin 0.2.4, was not released. Parameters 0.1.7 has now been released.

As promised in the Ronin: A Platform for Publishing and Mayhem talk at ToorCamp, Ronin 0.2.4 has finally been released. I was wanting to release 0.2.4 before ToorCamp, and hand out copies while there, but due to time constraints I had to wait till after the event.

$ gem update

Signed RubyGems

All released versions of Ronin, from 0.2.4 onward, will be signed. You can download the public certificate used to verify all of my gems here. A more in depth explanation of RubyGem Signing is given in Chapter 21 of the RubyGems Manual.

Bug Fixes

Mr. evoltech discovered and fixed a bug in the lookup of command names containing dashes, which was causing issues with the ronin-gen commands. flatline also improved the reliability of the caching of exploits from Overlays, now any exceptions raised during the caching of each exploit will be ignored.

Bytes and Chars

The 0.2.4 release now comes with new convenience methods to make working with byte and char Arrays easier:

[0x41, 0x41, 0x42].chars
# => ["A", "A", "B"]

["A", "B"].bytes
# => [0x41, 0x42]

[0x41, 0x41, 0x41].char_string
# => "AAB"

Un-Hexdumping

The File.unhexdump method was also added, making it even easier to un-hexdump those dumps.

Exceptions

Occasionally one needs to run some code which may raise exceptions, but you might not care about such exceptions, and would rather have them printed out. The catch_all method does exactly that. catch_all will catch all exceptions and print abbreviated back-traces.

require 'resolv'

catch_all do
  Resolv.getaddress('www.wired.com')
end

Another note-worth change in 0.2.4, was the renaming of the try method to attempt; so as not to conflict with JRuby's try method.

HTTP

The Net.http_request method was added, allowing one to make arbitrary HTTP Requests, just specify the :method option.

Net.http_request(:host => 'www.example.com', :method => :head)

Templates::Erb

The Templates::Erb module was added in 0.2.4, providing convenience methods for rendering Embedded Ruby (ERB) templates.

Scanners::Scanner

The Scanners::Scanner module was also added in 0.2.4. The Scanner module can be included into any class and allows one to define multiple scanner rules by name, which are ran against each target, returning results in real-time via a callback.

An example usage of Scanners::Scanner would be to add scanner rules to all IPAddr objects, having each IP address within a netmask scanned.

require 'ronin/scanners/scanner'
require 'ronin/extensions/ip_addr'

class IPAddr

  include Ronin::Scanners::Scanner

  scanner(:dns) do |ip,results|
    Resolve.getnames(ip).each do |name|
      results.call(name)
    end
  end

  def each_target(&block)
    each(&block)
  end

end

First we include Ronin::Scanners::Scanner into the IPAddr class. Then we define a simple scanner rule to perform reverse DNS lookup on an IP address and returns the results using the result callback. Finally we define the each_target method which enumerates over each IP address in the netmask, passing each to the block to be scanned.

To run all scanner rules on an IPAddr range:

ip = IPAddr.new('10.1.1.1/24')
ip.scan

To only run the DNS scanner rule:

ip.dns_scan

The SQL Injection, LFI and RFI scanning code has now been ported to use Scanners::Scanner.

Accessible Extensions

Extensions from Overlays are now more accessible. Within the Ronin console, they can be accessed as local variables:

puts milw0rm.remote.first_page

This is all thanks to the new Ronin#method_missing method; which catches missing instance method calls, and attempts to load the appropriate extension.

Command Name Changes

The ls and rm ronin commands have now been renamed to list and remove, respectively.

To list all installed Overlays:

$ ronin list

To remove (but not delete) an Overlay:

$ ronin remove overlay-name

New Dorks

ronin-dorks 0.1.2 saw the addition of the intext, allintext, string_intext, all_strings_intext, intitle, allintitle, string_intitle methods to Web::Dorks. The new intext and intitle convenience methods should simplify the creation of future dorks.

ronin.rubyforge.org Open Sourced

Last but not least, the source-code for the ronin.rubyforge.org website has been open-sourced on GitHub. Now if you want to make a correction or add a How-To, just fork it, commit your changes, then send me a pull-request and I'll upload your changes.

The collaborative editing is already happening, evoltech already wrote up a new badass Contribute page, that explains typical Git(Hub) workflow.

Sketches: Aftermath

2009-06-13T00:00:00-07:00

Apologies for not updating this blog more often. I have been very busy preparing for ToorCamp and releasing code.

One of my newer projects is Sketches. The idea for Sketches came from hacking sessions with Ronin. Eventually you need to automate some small task, this requires that you write a small method in the Ronin Console. I would find myself either writing methods directly into the console or cluttering my home directory with small Ruby scripts that I would load into the console. Usually after testing my code, I realized some changes needed to be made, and the code redefined/reloaded.

This became tedious and was cramping my work-flow. I just wanted to write code in my editor of choice and have it reloaded into the console like magic.

I remembered that Utility Belt had the ability to spawn an editor from IRB and load the resulting code after you exited it. Although, Utility Belt wont track the edited files and reload them whenever they change. Also, Utility Belt comes with a lot of OSX/S3 specific features that I didn't really need.

So I dug up reval.rb, borrowed the name Sketches from Processing's idea of code "sketches" and one week later I had Sketches working in IRB (and Ronin).

Sketches is easy to install:

$ sudo gem install sketches

Then just drop it into your .irbrc:

require 'sketches'

Sketches.config :editor => 'gvim'

When you want to pop open a new sketch:

sketch

You can also name your sketches:

sketch :foo

See all of your current sketches:

sketches

Don't worry about closing the editor, sketches will persist until you exit IRB and can be re-opened using the sketch method:

sketch :foo

After posting it to reddit I was surprised to see such an immediate positive reaction to the little project. I even saw posts on Twitter were people were using GNU Screen with vim for the editor command. I never thought of using screen, totally clever.

Ronin "faster" 0.2.3 has been released

2009-05-07T00:00:00-07:00

The wait is over, Ronin 0.2.3 (code-named "faster") has finally been released. This release contains new code, more specs, some very important architectural changes and a few bug-fixes.

Ronin on Ruby 1.9.1

Probably the most important news in Ronin 0.2.3, is that Ronin is now Ruby 1.9.1 compatible. Ronin can now take advantage of the considerable performance improvements in Ruby 1.9.1-p0. If you tend to do security research and find yourself having to use Ruby 1.9.1, you should look into using Ronin.

Faster Load-Times

Ronin also saw various architectural changes to help reduce load-times. The ronin/models.rb file was removed, which loaded models from the other Ronin libraries before the Database was setup. Now other Ronin libraries can call the Database.update! method, which will run non-destructive auto-migrations on the Database. Ronin::UI::CommandLine saw yet more refactoring. With the new Ronin::UI::CommandLine, sub-commands are loaded on-demand, instead of all at once.

Together these architectural changes have dramatically improved the load-time of Ronin's console. On systems that rarely run the Ruby interpreter, start-up times for Ronin should look like the following:

$ time (echo exit | (ronin > /dev/null))

real	0m3.841s
user	0m1.141s
sys	0m0.514s

On systems that regularly run the Ruby interpreter (thus caching frequently used memory and data) start-up times for Ronin will be a little quicker:

$ time (echo exit | (ronin > /dev/null))

real	0m1.656s
user	0m1.137s
sys	0m0.478s

New Convenience Methods

In 0.2.3 the IPAddr#each and IPAddr.each methods were added. It's somewhat common to need to iterate over a range of IP addresses. Say you have a CIDR notation IP address, and need to iterate over every IP address covered by it's netmask. Simply create a new IPAddr object and call each:

ip = IPAddr.new('10.1.1.1/24')
ip.each do |addr|
  ...
end

Well that's sort of cool, but what if you have a globbed IP address, similar to the ones nmap accepts? IPAddr.each has you covered:

IPAddr.each('10.1.1-5.*') do |addr|
  ...
end

Both IPAddr#each and IPAddr.each can iterate over IPv6 addresses.

Net.http_powered_by and Net.http_server also got added in 0.2.3. These methods provide quick access to the X-Powered-By and Server HTTP headers, respectively.

Net.http_powered_by(:url => 'http://www.stalkdaily.com/')
# => "PHP/5.2.9"

Net.http_server(:url => 'http://www.darkc0de.com/)
# => "Apache/2.2.11 (Unix) PHP/4.4.9 mod_ssl/2.2.11 OpenSSL/0.9.8c mod_fastcgi/2.4.6 Phusion_Passenger/2.1.2 DAV/2 SVN/1.4.2"

String#pad was also added in 0.2.3. The pad method doesn't do a lot, it merely pads a String out to a maximum length:

"hello".pad('A', 48)
# => "helloAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"

If you hate having to deal with exceptions in Ruby, the try method might prove useful. The try method simple attempts to run a block of code, catching and ignoring any exceptions that were raised:

require 'resolv'

try do
  Resolv.getaddress('might.not.exist.com')
end

Cacheable

Ronin::Objectify was replaced by the new Ronin::Cacheable module. Cacheable provides reliable caching and loading of Contextified objects with Ronin's Database. Using the new Cacheble module, the data you want cached into the database must be defined in a cache block:

ronin_exploit do
  cache do
    self.name = 'stupidhttpd'
    self.version = '0.2'
    self.author(:name => 'Postmodern')
  end

  ...
end

The use of a cache block creates a separation between the data to be cached and the code which will eventually be loaded.

Overlays

As of 0.2.3, overlays now support the automatic loading of the lib/init.rb file. So if you have code you'd like automatically loaded (maybe extensions to the Array class) from your Overlay, simply require it in the lib/init.rb file.

The loading of Extensions from Overlays became a little more robust in 0.2.3. If an exception is encountered when loading an extension file, the exception will be printed and the file ignored.

Ronin "theouterlibs" 0.2.2 released

2009-03-31T00:00:00-07:00

Ronin 0.2.2, code-named "theouterlibs", has been released. This release of Ronin has seen more action in the other libraries which support Ronin.

Ronin::Chars was split out of Ronin into the Chars 0.1.0 library, in order to help other frameworks and code which work with characters.

The ronin-overlay and ronin-ext sub-commands which handled generating skeleton Overlays and Extensions have been moved to the ronin-gen library. The ronin-gen library provides various code-generates which allow other Ronin libraries to create their own custom code-generators.

The R'epertoire library saw significant refactoring after various bugs were uncovered while testing Ronin 0.2.2. Now R'epertoire 0.2.1 has improved Git support, but CVS and Darcs support was removed.

Ronin also saw some minor improvements in 0.2.2. New convenience methods were added for formatting binary data:

Integer#bytes: returns the bytes that make up the Integer.

0xff41.bytes(2)
# => [65, 255]

0xff41.bytes(4, :big)
# => [0, 0, 255, 65]

String#hex_unescape: unescape those annoying hex-escaped Strings.
```
"\\x68\\x65\\x6c\\x6c\\x6f".hex_unescape
# => "hello"
```
String#unhexdump: un-hexdump a variety of hexdump formats.
```
File.read('data_dump.txt').unhexdump
# => "..."
```

Ronin::Static was also added in 0.2.2. The Ronin::Static module handles static resource directories which Ronin can search within for various files and directories. Ronin::Static::Finders provides path/file/directory finder methods that can be included into classes. Ronin Overlays and Extensions can now have static/ directories of their own, which are accessible via the Ronin::Static::Finders methods.

Loaded extensions can now be reloaded on the fly:

Platform.extensions.reload!

Ronin Extensions also can be accessed via constants within the Ronin namespace:

Ronin::HelloWord.name
# => "hello_word"

Introducing Chars 0.1.0

2009-03-17T00:00:00-07:00

Yesterday I released the first version of Chars. Chars is a Ruby library for working with character sets, generating random text and determining whether Strings or Integers belong to specific sets. Originally, Chars was part of the Ronin code-base, but was split-out in order to keep Ronin small and to promote code-reuse. So without further ado, here's some examples of what Chars can do:

Determine whether a byte belongs to a character set:
```
0x41.alpha?
# => true
```
Determine whether a String belongs to a character set:
```
"hello;".printable?
# => true
```

Find all sub-strings that belong to a character set within a String:

ls = File.read('/bin/ls')
Chars.printable.strings_in(ls)
# => ["/lib64/ld-linux-x86-64.so.2", "KIq/", "5J~!", "%L~!", ...]

Return a random character from the set of all characters:
```
Chars.all.random_char
# => "\x94"
```

Return a random Array of characters from the alpha-numeric character set:

Chars.alpha_numeric.random_array(10)
# => ["Q", "N", "S", "4", "x", "z", "3", "M", "F", "F"]

Return a random String from the set of all characters:

Chars.all.random_string(10)
# => "\xc2h\xad\xccm7\x1e6J\x13"

Return a random String with a random length between 5 and 10, from the set of space characters:
```
Chars.space.random_string(5..10)
# => "\r\v\n\t\n\f"
```

You can install Chars as a rubygem using the following command:

$ sudo gem install chars

If you want to hack on Chars, the code-base is available on GitHub.

Ronin 0.2.1 "notashellscript" released

2009-02-25T00:00:00-08:00

Yesterday Ronin 0.2.1, code-name "notashellscript", was released. That's right, we finally surpassed the awkward 0.1.x phase. Although, I completely forgot to write about Ronin 0.2.0, code-named "solid", which is where most of the action occurred.

Ronin 0.2.0 saw the complete refactoring and specing of the Platform code, which manages Overlays and their Extensions. Besides the huge amount of bug-fixes, modularization and renaming, Overlays can now have their own top-level lib/ directories. Also, all of the lib/ directories contained within an Overlay and it's Extensions are added to the $LOAD_PATH upon activation. This allows for easier requiring of code embedded within Overlays.

Ronin 0.2.0 also saw the addition of the ronin/environment file, which loads all of the convenience methods, configuration from ~/.ronin/config.rb and starts Ronin's Database. The environment file makes it easier to load up all of Ronin (minus the Platform code) in an IRB session and get hacking.

Grey bearded hackers bemoan how every exploit, library and framework have their own special leet diagnostic printing format. Some prefer the defacto "[*] Message" while others go for the saucy "{+} Message". Well grey beards you have one more reason to bemoan, UI::Diagnostics was added to Ronin 0.2.0. The UI::Diagnostics module adds the print_info, print_warning and print_error methods to a class. The output of these methods are controlled by the UI::Verbose module.

Ronin 0.2.1, code-name "notashellscript", had a couple but still important changes. Ronin 0.2.1 has dropped REXML in favour of Nokogiri for XML support. Nokogiri brings faster XML/HTML parsing and building to Ronin, providing it's own set of bindings to libxml2.

UI::CommandLine was rewritten in Ronin 0.2.1. Now sub-commands are accessible by both the ronin sub-command --options style and git style sub-commands, such as ronin-command --options. The git style sub-commands provide a more direct way of calling sub-commands.

Many of Ronin's other libraries received convenience sub-commands which simply start the Ronin console with the specific library pre-loaded. Ever wanted to jump right into Ronin Dorks or Ronin Scanners, now you totally can:

$ ronin-scanners
>> Scanners::Nmap.scan(:targets => '10.1.1.*', :syn_scan => true)
...

Finally, I've started to practice what I preach by setting up my own Ronin Overlay to experiment with. Of course, the Overlay is hosted on github.com, so feel free to clone and fork away. To install the Overlay under Ronin, use the following command:

$ ronin install git://github.com/postmodern/postmodern-ronin.git

So far, I've added an extension which parses the exploit lists on milw0rm.com. The milw0rm extensions is accessible within Ronin using the following code:

$ ronin
>> puts Ronin.milw0rm.remote.recent
...

Ironically, while testing the extension I noticed that milw0rm does not properly escape the titles of their exploits, resulting in unescaped < and > characters in their HTML.

The Push to Ruby 1.9

2009-02-06T00:00:00-08:00

So you probably heard that Ruby 1.9.1 has been released. That's right, no more testing release candidates, Ruby 1.9.1 has been marked stable. The 1.9.x series of Ruby comes with improved encoding support, includes RubyGems by default and has merged with YARV.

Including RubyGems was a good move, since it's always been a pain dealing with out-dated Debian packages of RubyGems, setting RUBYOPT or having gem update --system conflict with the customized Debian version of RubyGems. The inclusion of YARV, which implements a byte-code VM for Ruby, has brought considerable performance increases to Ruby. I've been having fun using the 1.9.1 release candidates to brute-force Project Euler problems.

So here's the catch, most of the RubyGems for Ruby do not work properly with the 1.9.x series and it's minor API changes. This poses a good dev challenge to the Ruby community: test, fix, fork or rewrite the offending RubyGems.

Some would say this will take some time, slowing the adoption of 1.9.1 or even stunting the Ruby community. But we came prepared, we got TDD, BDD, Unit::Test, Rspec and github.com.

Also, a handy site just popped up today, Is it Ruby 1.9: Community-powered gem compatibility for Ruby 1.9. People can search for RubyGems and add testing information to them.

To help with the push to Ruby 1.9.1, I threw together a shell script that installs Ruby 1.9.1 alongside 1.8.x. It's hosted on gist.github.com, so feel free to fork it.

#!/bin/sh
 
mkdir -p /usr/local/src && cd /usr/local/src
wget ftp://ftp.ruby-lang.org/pub/ruby/1.9/ruby-1.9.1-p0.tar.bz2
tar -xjvf ruby-1.9.1-p0.tar.bz2
cd ruby-1.9.1-p0
./configure --prefix=/usr --program-suffix=19 --enable-shared
make && make install

I'll be testing the RubyGems I use the most and my own projects. So far I've found that Contextify, Parameters, GScraper and Spidr all work with Ruby 1.9.1.

Ronin 0.1.3 "shake down" Released

2009-01-09T00:00:00-08:00

After many bug-fixes and testing Ronin 0.1.3, codenamed "shake down", has finally been released. I split out Ronin::Context into the Contextify library. Contextify is very handy for frameworks that want to load objects from Ruby files (without using YAML or defining specially named Classes/Modules). I've also refactored reverse-require, integrating it closer with RubyGems 1.3.0. Again, reverse-require is useful for frameworks that want to have a plugin system, but don't want to mess with the archaic gem_plugin. I eventually hope to get some of the code from reverse-requires merged into RubyGems. The Ronin::Objectify and Ronin::UI::CommandLine modules also got a fresh refactoring.

ronin-sql also received some refactoring love. It's SQL/Injection DSL was rewritten to uses a token emitter / formatter strategy for generating SQL syntax from the DSL. The use of tokens greatly simplified generating complex syntax from an Abstract Syntax Tree (AST).

>> require 'ronin/sql'
=> true
>> puts Code.sql_injection { has_table?(:users) }
AND (SELECT count(*) FROM users) = 1
=> nil

Not only were there updates, but also some libraries released for the very first time. After sitting on github for a while, ronin-exploits and ronin-scanners were finally released.

The ronin-exploits library provides the ability to define exploits and payloads, as well as caching them in Ronin's database. Since ronin-exploits uses Contextify, writing exploits becomes easy and elegant:

# test_exploit.rb
require 'ronin/sessions/tcp'

ronin_exploit do

  extend Sessions::TCP

  self.name = 'test'
  self.version = '0.2'
  self.license = License.cc_by_nc

  self.author(:name => 'postmodern', :organization => 'SophSec')

  def builder
    @buffer = 'some data'
  end

  def deployer
    tcp_send(@buffer)
  end

end

The library also allows one to associate Vulnerability Taxonomy information with Exploits or Payloads.

The ronin-scanners library provides interfaces to various security scanners. Currently, ronin-scanners provides a Rubyful interface to the Nmap network scanner and all of it's options.

>> require 'ronin/scanners'
=> true
>> puts Scanners::Nmap.scan(:targets => 'www.google.com', :ports => [80,21,25], :service_scan => true)
Starting Nmap 4.68 ( http://nmap.org ) at 2009-01-09 16:51 PST
Interesting ports on mh-in-f99.google.com (209.85.173.99):
PORT   STATE    SERVICE VERSION
21/tcp filtered ftp
25/tcp filtered smtp
80/tcp open     http    Google httpd 1.3 (GFE)
Service Info: OS: Linux

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.627 seconds
=> nil

The library also uses ScanDB, so that Nmap scan results can be stored or queried using ScanDB's database.

Now that I've completed another round of refactoring and releasing updated libraries, I can focus on other things, such as improving Ronin's website and build a presentation for Ronin. Recently I was invited by evoltech from HackBloc to trek down to the Bay Area and give a mini-presentation on Ronin at this years Hack Meet. Hack Meet is a semi-regular non-corporate hacker meet-up / mini-conference, where hackers gather and share their latest projects.

Spidr and Raingrams are back, now with specs

2008-11-13T00:00:00-08:00

Raingrams is back in action. After sitting on rubyforge for quite some time, I was asked to add some features to the general purpose Ngrams Ruby library. I ended up refactoring the code to handle probability calculations better (only recalculate the Maximum Likelihood Estimation (MLE) when the set of ngrams changes), removed the Unigram model (kinda pointless in a ngrams library), allow a trained model to be dumped to a file using Marshal and added the ability to generate random text from trained models. Raingrams also received a total of 133 new spec tests.

Install Raingrams:

$ sudo gem install raingrams

Spidr also received some new spec tests. After fixing a link handling bug in Spidr 0.1.1, I decided to create a Web Spider Obstacle Course for testing purposes. The course contains all manner of links (remote, local, relative, absolute, javascript, empty URLs and infinite looping links). The course also provides a JSON file containing spec information for how a web-spider should navigate the links. I also wrote a RSpec helper which imports the spec information from the JSON file and auto-generates spec tests for how Spidr::Agent should navigate the links in the obstacle course.

Install Spidr:

$ sudo gem install spidr

RDoc can let you down, can give you up, can run around and desert you...

2008-10-24T00:00:00-07:00

Update: The Ruby Core developers have finally fixed this bug in RDoc. Thanks go out to Eric Hodel (for recently refactoring RDoc) and the Ruby Core developers for maintaining Ruby.

Normally RDoc is there for you when it comes to auto-generating nice HTML documentation for your Ruby projects. But just now, it flat out failed while generating Graphviz diagrams of my code.

I was attempting to generate some documentation for the Ronin SQL library in order to test the formatting of README.txt. To my surprise rake docs failed without any informative error message describing which file caused RDoc to chock.

Here's the output of rake --trace docs:

[ gist.github.com/19353 ]


Generating HTML...
Diagrams: ...
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup/fragments.rb:291: warning: Object#type is deprecated; use Object#class
rake aborted!
undefined method `level' for nil:NilClass
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup/fragments.rb:292:in `add_list_breaks'
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup/fragments.rb:282:in `each'
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup/fragments.rb:282:in `add_list_breaks'
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup/fragments.rb:153:in `normalize'
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup.rb:459:in `group_lines'
/usr/lib64/ruby/1.8/rdoc/markup/simple_markup.rb:255:in `convert'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:246:in `markup'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:818:in `value_hash'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:865:in `write_on'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1293:in `gen_into'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1293:in `open'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1293:in `gen_into'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1289:in `each'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1289:in `gen_into'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1276:in `generate_html'
/usr/lib64/ruby/1.8/rdoc/generators/html_generator.rb:1197:in `generate'
/usr/lib64/ruby/1.8/rdoc/rdoc.rb:284:in `document'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake/rdoctask.rb:113:in `define'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:617:in `call'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:617:in `execute'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:612:in `each'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:612:in `execute'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:578:in `invoke_with_call_chain'
/usr/lib64/ruby/1.8/monitor.rb:242:in `synchronize'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:571:in `invoke_with_call_chain'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:588:in `invoke_prerequisites'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:585:in `each'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:585:in `invoke_prerequisites'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:577:in `invoke_with_call_chain'
/usr/lib64/ruby/1.8/monitor.rb:242:in `synchronize'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:571:in `invoke_with_call_chain'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:564:in `invoke'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:2019:in `invoke_task'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1997:in `top_level'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1997:in `each'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1997:in `top_level'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:2036:in `standard_exception_handling'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1991:in `top_level'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1970:in `run'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:2036:in `standard_exception_handling'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/lib/rake.rb:1967:in `run'
/usr/lib64/ruby/gems/1.8/gems/rake-0.8.3/bin/rake:31
/usr/bin/rake:19:in `load'
/usr/bin/rake:19

Suddenly, all that talk about YARD replacing RDoc is sounding a lot more pragmatic. YARD has certainly worked out for DataMapper and Merb thus far. Maybe Ronin will give YARD a try.