Postmodern's Blog

How to add a man page to your Ruby project

2023-12-05T00:00:00+00:00

If you have ever used the git command-line utility, you will be pleasantly surprised that if you run git clone --help it automatically displays the man page for git clone instead of the usual --help output.

This blog post will show you how to add the same functionality to your Ruby command-line utility.

Introducing kramdown-man

Man pages are written in the roff typesetting markup language, which uses macro tags that look like .PP and \fB. Needless to say writing roff by hand is not fun. Instead, we will use the kramdown-man gem to generate the roff man page from a similar looking pure markdown man page.

Step 1: Add kramdown-man

Add kramdown-man to your Gemfile and run bundle install:

gem 'kramdown-man', '~> 1.0'

Step 2: Write the markdown man page

First create the man/ directory in your project.

$ mkdir man

Then write the markdown man page, which should be named like man/mycli.1.md. The number in the file name indicates the man page Section number (Section 1 is for General Commands, Section 3 is for Library Functions).

Use kramdown-man’s own man page as an example for how to structure your markdown man page. It should look roughly like this:

# mycli 1 "2024-01-01" MyCLI "User Manuals"

## NAME

mycli - Does things and stuff

## SYNOPSIS

`mycli` [*options*] *ARG1* [*ARG2*]

## DESCRIPTION

The `mycli` utility does things and stuff. Bla bla bla bla.

## ARGUMENTS

*ARG1*
: This is a required argument.

*ARG2*
: This is an optional argument

## OPTIONS

`-f`, `--flag` *VALUE*
: This is an option flag that takes a *VALUE* argument.

`-h`, `--help`
: Prints the help information for the command.

## EXAMPLES

Example command description:

    $ mycli --flag file.txt

## AUTHOR

Your Name <you@example.com>

## SEE ALSO

[bash(1)](man:bash.1) [other-man-page](other-man-page.1.md)

To preview how the markdown man page will be rendered, use the kramdown-man command:

$ kramdown-man man/mycli.1.md

Markdown Man Page Layout Explained

The first line will be used for the man page’s header and footer lines. It has the following format:

# mycli 1 "2024-01-01" MyCLI "User Manuals"

# mycli - The command name.
1 - The section number.
2024-01-01 - The date the man page is being written, in the format YYYY-MM-DD.
MyCLI - The project’s name.
"User Manuals" - The man page section name.

Sections

Man pages typically have the follow main sections:

NAME - The command name and a short definition.
SYNOPSIS - The command’s usage, showing order of arguments.
DESCRIPTION - A more detailed description of the command.
ARGUMENTS - Defines the purpose of each argument.
OPTIONS - Defines the purpose of each option flag and it’s usage.
EXAMPLES - Show common examples of running the command.
AUTHORS - List the authors of the command.
SEE ALSO - Link to other man pages.

ARGUMENT / OPTION LISTS

Argument and option definitions must be defined using markdown definition lists (hence the : before the summary) for them to be properly indented.

*ARG1*
: Definition goes here.

*ARG2*
: Definition goes here.

  Multiple paragraphs may be given.

COMMAND USAGE SYNTAX

Codespans indicate a literal word:

`mycli`

Emphasis and all uppercase indicates a required argument:

*ARG*

Square brackets around an argument indicates an optional argument.

[*ARG*]

Curly-braces with pipe separates indicates one of the arguments is required:

{*ARG1* \| *ARG2* \| *ARG3*}

Man Page Links

To link to other man pages in your project’s man/ directory, use a regular markdown link that links to the .1.md file:

[other-man-page](other-man-page.1.md)

This will also generate a bolded man page reference which will look like other-man-page(1) in the displayed man page.

To link to other man pages that are already installed on a system, use a regular markdown link, but use a man:page-name.1 URL with the man page name and section number:

[bash(1)](man:bash.1)

This will generate a bolded man page reference which will look like bash(1) in the displayed man page.

Note: Firefox on Linux will actually recognize man: URIs and open them using Gnome’s yelp help browser.

Step 3: Add the rake task

Now that we have written the markdown man page, we need to setup a rake task to generate the roff formatted man page from the markdown man page.

Add the following code to your Rakefile:

require 'kramdown/man/task'
Kramdown::Man::Task.new

This will define a man rake task and define file dependencies between the man/*.1 output files and the man/*.1.md input files.

Step 4: Generate the man pages

To generate all man pages in the man/ directory run:

$ rake man

You can then view the generated man pages using the man command:

man ./man/mycli.1

Step 5: Add the code

In order for our CLI to automatically display the man page when the --help option is given, we will need to add this bit of code to the OptionParser’s --help option:

# The path to the man/mycli.1 generated man page
MAN_PAGE = File.join(__dir__,'..','..','..','man','mycli.1')

...

opts.on('-h','--help','Prints this kruft') do
  if $stdout.tty?
    system('man',MAN_PAGE)
  else
    puts opts
  end
  exit
end

The if $stdout.tty? check tests whether stdout is a TTY or being redirected to a file or another command. If we are running in a real TTY terminal, then display the man page. If we are not running in a real TTY terminal, then print the usual --help output. This is a polite thing to do, as users might want to view the --help output through less or might dump it to a file using --help >mycli.txt.

If you don’t want to copy/paste the above code into all of your Ruby projects, you can use the command_kit gem, which provides a CommandKit::Help::Man module that adds the same functionality to a command class.

Step 6: Package your man page

Now that we have generated our roff man pages, we will want to add them to either git or the gemspec’s files list. This way the generated roff man page will be included in the packaged gem.

If you prefer to not add the generated roff man page to git, you can manually add it to the list of files in your .gemspec file:

gem.files << 'man/mycli.1'

Then build and install your gem:

$ rake gem
$ gem install ./pkg/mycli-0.1.0.gem

Step 7: Test It!

Now your command should display it’s own man page when --help is given:

$ mycli --help

You should see something that looks like this:

Solving Advent of Cyber 2023 Day 2 using Ruby

2023-12-02T00:00:00+00:00

This year I decided to try my hand at the Advent of Cyber challenges.

The Day 2 challenge involves Data Science. We are given a Jupyter Notebook file containing a table of log data showing ports scan events. Now the challenge teaches you how to use Jupyter Notebook and Python, but we’re not going to solve it using Python. We are going to solve it using only Ruby!

While Python is very popular in the Data Science field, you can do Data Science with Ruby. Ruby standard library comes with many useful methods, such as map, select, group_by, group_by, which allow you to slice and dice large datasets.

First, we will need to liberate the data from the Jupyter Notebook. To do this, we open the Jupyter Notebook, navigate to the Table View, select all rows, copy the rows, and paste into a text file.

PacketNumber	Timestamp	Source	Destination	Protocol
1	05:49.5	10.10.1.7	10.10.1.9	HTTP
2	05:50.3	10.10.1.10	10.10.1.3	TCP
3	06:10.3	10.10.1.1	10.10.1.2	HTTP
4	06:10.4	10.10.1.9	10.10.1.3	ICMP
...

The rows will paste as Tab Separated Values (TSV). We will need to convert the rows into Comma Separated Values (CSV). Converting from TSV to CSV is as simple as the following vim substitution command %s/\v\t/,/g.

PacketNumber,Timestamp,Source,Destination,Protocol
1,05:49.5,10.10.1.7,10.10.1.9,HTTP
2,05:50.3,10.10.1.10,10.10.1.3,TCP
3,06:10.3,10.10.1.1,10.10.1.2,HTTP
4,06:10.4,10.10.1.9,10.10.1.3,ICMP
...

Much better. Finally, we save the file to data.csv.

Next, we will spawn an Interactive Ruby session using irb with the csv library preloaded:

$ irb -r csv
irb(main):001> 

Now we will load our data.csv file into a variable:

csv = CSV.read('data.csv', headers: true)

Now we just have to answer the Day 2 questions using pure Ruby.

How many packets were captured (looking at the PacketNumber)?

csv[-1]['PacketNumber']

What IP address sent the most amount of traffic during the packet capture?

csv.group_by { |row| row['Source'] }.max_by { |ip,events| events.count }.first

What was the most frequent protocol?

csv.group_by { |row| row['Source'] }.max_by { |ip,events| events.count }.first

As you can see, you don’t necessarily have to use Python for Data Science. Ruby is more than capable of doing basic Data Science.

How to use ActiveRecord in a library

2022-03-28T00:00:00+00:00

Recently, I wanted to use ActiveRecord in a library, not in a Rails app or a Rails engine. After doing a bit of Googling, I found this excellent tutorial. While it covered the basics, such as connecting to the database, defining a model and a migration, it missed a few things. In this blog post I will provide a more detailed example of how to connect to a default or arbitrary database, “connect” multiple models to the database, and run a directory of migrations if they have not already been ran.

Connecting to the Database

require 'active_record'

ActiveRecord::Base.establish_connection(
  adapter:  'sqlite3',
  database: 'database.sqlite3'
)

If you want to connect to a separate database for tests, you can use an in-memory sqlite3 database:

ActiveRecord::Base.establish_connection(
  adapter:  'sqlite3',
  database: ':memory:'
)

Migrations

# lib/library/migrations/0001_create_authors_table.rb
class CreateAuthorsTable < ActiveRecord::Migration[7.0]

  def change
    create_table :library_authors do |t|
      t.string :name, null: false
      t.index :name, unique: true
    end
  end

end

# lib/library/migrations/0002_create_books_table.rb
class CreateBooksTable < ActiveRecord::Migration[7.0]

  def change
    create_table :library_books do |t|
      t.string :title
      t.index :title, unique: true
    end
  end

end

# lib/library/migrations/0003_create_book_authors_table.rb
class CreateBookAuthorsTable < ActiveRecord::Migration[7.0]

  def change
    create_table :library_book_authors do |t|
      t.references :author, null: false,
                            foreign_key: {
                              to_table: :library_authors
                            }
      t.references :book, null: false,
                          foreign_key: {
                            to_table: :library_books
                          }
      t.index [:author_id, :book_id], unique: true
    end
  end

end

Note: due to how ActiveRecord loads migration classes they must not be defined in a module namespace and must not contain any all-uppercase acronym words (ex: CreateIpAddressesTable not CreateIPAddressesTable).

Running Migrations

ActiveRecord provides a ActiveRecord::MigrationContext class which can load and run migrations similar to rake db:migrate. This class can be initialized with our own migrations/ directory path, which will load all migrations within the directory and allow us to run the migrations.

The next step is to define our own Migrations module which defines migrate, migrate_up, and migrate_down methods which call the .migrate, .up, and .down methods on the initialized ActiveRecord::MigrationContext object:

# lib/library/migrations.rb
require_relative 'active_record'

module Library
  module Migrations
    DIR = File.join(__dir__,'migrations')

    def self.context
      @context ||= ActiveRecord::MigrationContext.new([DIR])
    end

    def self.migrate(target_version=nil)
      context.migrate(target_version)
    end

    def self.migrate_up(target_version=nil)
      context.up(target_version)
    end

    def self.migrate_down(target_version=nil)
      context.down(target_version)
    end
  end
end

Models

# lib/library/author.rb
module Library
  class Author < ActiveRecord::Base

    self.table_name_prefix = 'library_'

    attribute :name, :string
    validates :name, presence: true,
                     uniqueness: true

    has_many :authorships, class_name: 'BookAuthor'
    has_many :books, through:    :authorships,
                     inverse_of: :authors

  end
end

Note: the explicit self.table_name_prefix is required since ActiveRecord is unable to infer the table prefix based on the module namespace.

# lib/library/book_author.rb
module Library
  class BookAuthor < ActiveRecord::Base

    self.table_name_prefix = 'library_'

    belongs_to :book, required: true

    belongs_to :author, required: true

  end
end

Note: while the Internet says that all belongs_to associations are required by default since Rails 5, when using ActiveRecord outside of Rails one must explicitly add required: true to them.

# lib/library/book.rb
module Library
  class Book < ActiveRecord::Base

    self.table_name_prefix = 'library_'

    attribute :title, :string
    validates :title, presence: true,
                      uniqueness: true

    has_many :book_authors
    has_many :authors, through:    :book_authors,
                       inverse_of: :books

  end
end

To make it easy to load all models after establishing the database connection (ActiveRecord does not allow you to define any model classes before calling ActiveRecord::Base.establish_connection), we also create a models.rb file:

# lib/library/models.rb
require_relative 'author'
require_relative 'book'
require_relative 'book_author'

module Library
  MODELS = [
    Author,
    Book,
    BookAuthor
  ]
end

ActiveRecord::Base.connection

In order for your model’s inspect methods to work properly, the ActiveRecord models need to be “connected” to the established database connection. This can be done by calling connection on each model class.

Library::Book.inspect
# => "Library::Book (call 'Library::Book.connection' to establish a connection)"

Library::Book.connection

Library::Book.inspect
# => "Library::Book(id: integer, title: string)"

Note: for some reason it appears that this extra step is not necessary if the migrations are ran before loading the models.

I am not really sure why this extra step is necessary, as it seems like ActiveRecord could lazy-connect each model when ActiveRecord::Base#inspect is called.

Putting It All Together

Now we just need an entry-point method that can:

Connect to the database
Run migrations (if there are any pending migrations)
“connect” the models

We shall call this entry-point method Library.connect:

# lib/library.rb
module Library
  DEFAULT_DATABASE = {
    adapter:  'sqlite3',
    database: '/path/to/default/database.sqlite3'
  }

  def self.connect(database=DEFAULT_DATABASE)
    ActiveRecord::Base.establish_connection(database)

    require_relative 'library/migrations'
    Migrations.migrate_up

    require_relative 'library/models'
    MODELS.each(&:connection)
  end
end

If you are interested, I have pushed the working code for this blog post to GitHub.

Enumerating XKCD-style passwords with Ruby

2022-01-23T00:00:00+00:00

In XKCD comic #936 it explains how to measure password strength and how to come up with a more “secure” password. In this blog post I will show you how to enumerate such XKCD-style passwords using Ruby.

To do this we will use two libraries: wordlist and chars.

$ gem install wordlist chars

The wordlist library allows one to open/build wordlists, enumerate them, combine multiple wordlists together, and even mutate each word in the wordlist. The chars library allows working with common character sets, such as digits, alpha-numeric, punctuation, etc.

Tr0ub4dor

The first XKCD password (“Tr0ub4dor&3”) seems pretty difficult to guess, but it’s actually quite easy to enumerate. The original word is “troubador”, with some character/case substitutions, and one punctuation character along with one number appended to the end. Chaining together wordlist’s string manipulation methods we can enumerate over every possible mutation of “troubador”.

require 'wordlist'
require 'chars'

base_words = Wordlist::Words["troubador"]
passwords  = base_words * Chars.punctuation.chars * Chars.digits.chars
mutations  = passwords.mutate(/oa/, 'o' => '0', 'a' => '4').mutate_case

mutations.each { |password| puts password }

The above code creates a list of literal words, then combines them with all symbol characters and all digit characters, and finally it applies mutation rules to mutate the case of letters and substitutes o for 0 and a for 4.

Now let’s enumerate through all possible passwords. Let ‘er rip!

troubador 0
Troubador 0
tRoubador 0
trOubador 0
troUbador 0
trouBador 0
troubAdor 0
troubaDor 0
troubadOr 0
troubadoR 0
TRoubador 0
TrOubador 0
TroUbador 0
TrouBador 0
TroubAdor 0
TroubaDor 0
TroubadOr 0
TroubadoR 0
tROubador 0
tRoUbador 0
...
TrOuBADOR!9
TroUBADOR!9
tROUBADOr!9
tROUBADoR!9
tROUBAdOR!9
tROUBaDOR!9
tROUbADOR!9
tROuBADOR!9
tRoUBADOR!9
trOUBADOR!9
TROUBADOr!9
TROUBADoR!9
TROUBAdOR!9
TROUBaDOR!9
TROUbADOR!9
TROuBADOR!9
TRoUBADOR!9
TrOUBADOR!9
tROUBADOR!9
TROUBADOR!9

correcthorsebatterystapler

The second XKCD password (“correcthorsebatterystapler”) is four random words concatenated together. Assuming we have a wordlist of common English words, all we have to do is combine that wordlist with itself four times, then call .each like in the previous example.

require 'wordlist'

common_words = Wordlist.open('common_words.txt')
passwords    = common_words ** 4

passwords.each { |password| puts password }

The above code opens a .txt wordlist file, then creates a new wordlist object (passwords) that is simply the wordlist multiplied by itself four times; in Ruby ** is the power operator.

Let’s see if it works. Let ‘er rip!

aahedaahedaahedaahed
aahedaahedaahedaahing
aahedaahedaahedaalii
aahedaahedaahedaaliis
aahedaahedaahedaardvark
aahedaahedaahedaardvarks
aahedaahedaahedaardwolf
aahedaahedaahedaardwolves
aahedaahedaahedaargh
aahedaahedaahedaaron
aahedaahedaahedaaronic
aahedaahedaahedaaronical
aahedaahedaahedaaronite
aahedaahedaahedaaronitic
aahedaahedaahedaarrgh
aahedaahedaahedaarrghh
aahedaahedaahedaasvogel
aahedaahedaahedaasvogels
aahedaahedaahedababdeh
aahedaahedaahedababua
...

While it definitely took much longer to enumerate over every four common English words, it is still possible to easily enumerate over every possible password.

Bonus Challenge

What if we used different words for the first and second word, added random punctuation in between the words, and appended multiple digits?

require 'wordlist'
require 'chars'

wordlist1 = Wordlist.open('animals.txt')
wordlist2 = Wordlist.open('months.txt')
passwords = wordlist1 * Chars.punctuation.chars * wordlist2 * Chars.digits.strings_of_length(1..4)

passwords.each { |password| puts password }

The above code opens two wordlists for the first and second words, combines the first wordlist with all punctuation characters, combines the second wordlist, then combines all possible strings of lengths 1-4 of digit characters.

dog january0
dog january1
dog january2
dog january3
dog january4
dog january5
dog january6
dog january7
dog january8
dog january9
dog january00
dog january01
dog january02
dog january03
dog january04
dog january05
dog january06
dog january07
dog january08
dog january09
...
pony!december9980
pony!december9981
pony!december9982
pony!december9983
pony!december9984
pony!december9985
pony!december9986
pony!december9987
pony!december9988
pony!december9989
pony!december9990
pony!december9991
pony!december9992
pony!december9993
pony!december9994
pony!december9995
pony!december9996
pony!december9997
pony!december9998
pony!december9999

The Math

Using Combinatorics, if we wanted to calculate the total search space of a password we would use the simple equation n ** k, where n is the number of possibilities and k is the number of times they are repeated in succession. Example: if a 32bit unsigned integer has 32 bits, and each bit has 2 possibilities (1 and 0), then the total number of possible 32bit unsigned integers is 2 ** 32.

Assuming a password of length 26 ("correcthorsebatterystapler".length # => 26) composed entirely of printable ASCII characters, and there are 100 printable ASCII characters (Chars.printable.length # => 100), that would give us a search space of 100 ** 26 which is 10000000000000000000000000000000000000000000000000000.

Now what if we were a cunning hacker (or pentester), and guessed that the users password was probably four random English words? Then we would grab our trusty common English words wordlist, which happens to be 171,000 words long. This results in a search space of 171_000 ** 4, which is 855036081000000000000; much smaller than 10000000000000000000000000000000000000000000000000000.

The XKCD web comic made the assumption that a hacker (or pentester) would only resort to enumerating through every possible bit in a password string. Enumerating through every combination of bits of a 26 character long password would result in a search space of 2 ** (26 * 8) (1 char = 1 byte = 8 bits) or 411376139330301510538742295639337626245683966408394965837152256. This assumption is wrong as I have shown that one can use wordlists to reduce the search space to 855036081000000000000 which is visibly less than both 411376139330301510538742295639337626245683966408394965837152256 and 10000000000000000000000000000000000000000000000000000.

The smaller search space is the obvious winner, as it means less work to do and ultimately less time spent. Factor in the ever increasing speed of technology, the fact that you can distribute password bruteforcing across multiple IPs “in the cloud”, or how HashCat supports GPU accelerated password cracking using OpenCL (not to mention FPGAs and ASICs), and that further reduces the time it would take to bruteforce or crack a password using wordlists and common password patterns.

Conclusion

As you can see, we can easily enumerate complex password patterns using the wordlist and chars libraries, and it didn’t take that much code! This can be incredibly powerful when combined with a login bruteforcing or password cracker. By combining wordlists and character sets, instead of bruteforce enumerating over every ASCII character, we can reduce the size of the search space and thus reduce the time it takes to bruteforce/crack a password; fewer passwords to check means less time spent.

tl;dr Don’t get your security advice from a web comic. Use a password manager that generates truly random passwords and turn on 2FA.

Introducing command_mapper

2021-11-25T00:00:00+00:00

The Problem

Normally in Ruby if you need to write a method which accepts one or more arguments and executes a command, you would write something like this:

def git_pull(branch)
  system("git pull origin #{branch}")
end

However, there are a few problems with the above code:

Does not validate the input (ex: git_pull(nil), git_pull(""), git_pull(true), git_pull(false), git_pull([1,2,3]), git_pull({1=>2}), etc).
Vulnerable to arbitrary command injection (ex: git_pull(";evil_command_here#")).
Vulnerable to arbitrary option injection (ex: git_pull("--option-that-gives-an-attacker-control branch")).

A better version of the above code might look like this:

def git_pull(branch)
  args = %w[git pull origin]
  args << branch.to_s if branch
  system(*args)
end

Better. We have fixed the arbitrary command injection by passing multiple arguments to Kernel#system, which executes the command as its own sub-process, not in a sub-shell. We also added very basic validations for branch.

However, those basic validations are not enough and the above code is still vulnerable to option injection via branch or any additional argument that is appended to args. While the above code might be suitable for a Rakefile where we only call git_pull with explicit literal values, if we were to pass user input to git_pull, possibly from say a web app, we would need stronger input validations. It would take a lot of work to add support for all of git pull’s other options and arguments, and add validations for each of them.

Enter command_mapper

command_mapper is a new library for mapping external commands to Ruby classes.

require 'command_mapper/command'

#
# Represents the `grep` command
#
class Grep < CommandMapper::Command

  command "grep" do
    option "--extended-regexp"
    option "--fixed-strings"
    option "--basic-regexp"
    option "--perl-regexp"
    option "--regexp", equals: true, value: true
    option "--file", name: :patterns_file, equals: true, value: true
    option "--ignore-case"
    option "--no-ignore-case"
    option "--word-regexp"
    option "--line-regexp"
    option "--null-data"
    option "--no-messages"
    option "--invert-match"
    option "--version"
    option "--help"
    option "--max-count", equals: true, value: {type: Num.new}
    option "--byte-offset"
    option "--line-number"
    option "--line-buffered"
    option "--with-filename"
    option "--no-filename"
    option "--label", equals: true, value: true
    option "--only-matching"
    option "--quiet"
    option "--binary-files", equals: true, value: true
    option "--text"
    option "-I", name: 	# FIXME: name
    option "--directories", equals: true, value: true
    option "--devices", equals: true, value: true
    option "--recursive"
    option "--dereference-recursive"
    option "--include", equals: true, value: true
    option "--exclude", equals: true, value: true
    option "--exclude-from", equals: true, value: true
    option "--exclude-dir", equals: true, value: true
    option "--files-without-match", value: true
    option "--files-with-matches"
    option "--count"
    option "--initial-tab"
    option "--null"
    option "--before-context", equals: true, value: {type: Num.new}
    option "--after-context", equals: true, value: {type: Num.new}
    option "--context", equals: true, value: {type: Num.new}
    option "--group-separator", equals: true, value: true
    option "--no-group-separator"
    option "--color", equals: :optional, value: {required: false}
    option "--colour", equals: :optional, value: {required: false}
    option "--binary"

    argument :patterns
    argument :file, required: false, repeats: true
  end

end

Type System

An observant reader will notice type: Num.new in the above example code. All option values and arguments may have a type. All options and arguments default to the Str type. These types define their own validation and formatting rules. The available types are:

Str: string values
Num: numeric values
Hex: hexadecimal values
Map: maps true/false to yes/no, or enabled/disabled (aka --opt=yes|no or --opt=enabled|disabled values).
Enum: maps a finite set of Symbols to a finite set of Strings (aka --opt={foo|bar|baz} values).
List: comma-separated list (aka --opt VALUE,...).
KeyValue: maps a Hash or Array to key:value Strings (aka --opt KEY:VALUE or --opt KEY=VALUE values).
KeyValueList: a key-value list (aka --opt KEY:VALUE,... or --opt KEY=VALUE;... values).
InputPath: a path to a pre-existing file or directory
InputFile: a path to a pre-existing file
InputDir: a path to a pre-existing directory

Custom Types

Custom type classes can be defined by simply inheriting from CommandMapper::Types::Type then defining validate and format instance methods.

class PortRange < CommandMapper::Types::Type

  def validate(value)
    case value
    when Integer
      true
    when Range
      if value.begin.kind_of?(Integer)
        true
      else
        [false, "port range can only contain Integers"]
      end
    else
      [false, "port range must be an Integer or a Range of Integers"]
    end
  end

  def format(value)
    case value
    when Integer
      "#{value}"
    when Range
      "#{value.begin}-#{value.end}"
    end
  end

end

Then the custom type class can then be passed to any type: keyword argument:

option :ports, value: {required: true, type: PortRange.new}

Running Commands

Once a CommandMapper::Command class has been defined, it can then map the class’s attributes back to the command’s option flags, additional arguments, or subcommands, and then safely executed via system():

Grep.run(ignore_case: true, patterns: "foo", file: "file.txt")

Commands can also be initialized with a block and executed:

Grep.run do |grep|
  grep.ignore_case = true
  grep.patterns    = "foo"
  grep.file        = "file.txt"
end

Note: that if a required argument does not have a value or if an invalid value is given to an option or argument, a validation error will be raised:

Grep.run(file: 'file.txt')
# /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:494:in `block in command_argv': argument patterns is required (CommandMapper::ArgumentRequired)
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:490:in `each'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:490:in `command_argv'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:537:in `run_command'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:108:in `run'

Output from the command can also be captured (similar to \`...\`):

Grep.capture(ignore_case: true, patterns: "foo", file: "file.txt")
# => "..."

Output from the command can also be read via IO.popen:

Grep.popen(ignore_case: true, patterns: "foo", file: "file.txt")
# => #<IO:...>

The command can also be ran under sudo (see the CommandMapper::Sudo class):

Grep.sudo(patterns: "Error", file: "/var/log/syslog")
# Password: 
# ...

Finally, the command can even be safely embedded in another command string:

gre[ = Grep.new(ignore_case: true, patterns: "foo", file: "file.txt")
cmd = "#{grep} | less"
system(cmd)

Security

In order to prevent arbitrary command injection, any special shell characters in the command’s option or argument values will automatically be escaped using Shellwords:

grep = Grep.new(patterns: ';injected_command#', file: 'test.txt')
grep.command_string
# => "grep \\;injected_command\\# test.txt"

In order to prevent option injection, options will explicitly not allow values that begin with a - character:

Grep.run(label: '--injected-option', patterns: 'foo', file: 'test.txt')
# /home/postmodern/code/command_mapper.rb/lib/command_mapper/option.rb:273:in `emit_option_flag_and_value': option label formatted value ("--injected-option") cannot start with a '-' (CommandMapper::ValidationError)
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/option.rb:164:in `argv'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:485:in `block in command_argv'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:483:in `each'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:483:in `command_argv'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:542:in `run_command'
# 	from /home/postmodern/code/command_mapper.rb/lib/command_mapper/command.rb:108:in `run'

In order to prevent arbitrary option injection via additional arguments, a -- separator will be inserted between the options and the additional arguments if any of the arguments starts with a - character. This will force the CLI utility to stop parsing options after the -- separator, and prevents the CLI utility from parsing the additional arguments as option flags:

grep = Grep.new(ignore_case: true, patterns: '-foo', file: 'test.txt')
grep.command_argv
# => ["grep", "--ignore-case", "--", "-foo", "test.txt"]

But Wait, There’s More!

Now you might be thinking “gee that’s still a lot to type, must be tedious to setup”, and you’d be right. That’s where command_mapper-gen comes in. The command_mapper-gen CLI utility can parse a command’s --help output and/or man page, and automatically generate the above code:

$ command_mapper-gen grep
Failed to parse line in `grep --help`:

    -NUM                      same as --context=NUM

Failed to match sequence (('	' / SPACES) OPTION ','? ([ \\t]{1, } OPTION_SUMMARY)? !.) at line 1 char 5.

require 'command_mapper/command'

#
# Represents the `grep` command
#
class Grep < CommandMapper::Command

  command "grep" do
    option "--extended-regexp"
    option "--fixed-strings"
    option "--basic-regexp"
    option "--perl-regexp"
    option "--regexp", equals: true, value: true
    option "--file", equals: true, value: true
    option "--ignore-case"
    option "--no-ignore-case"
    option "--word-regexp"
    option "--line-regexp"
    option "--null-data"
    option "--no-messages"
    option "--invert-match"
    option "--version"
    option "--help"
    option "--max-count", equals: true, value: {type: Num.new}
    option "--byte-offset"
    option "--line-number"
    option "--line-buffered"
    option "--with-filename"
    option "--no-filename"
    option "--label", equals: true, value: true
    option "--only-matching"
    option "--quiet"
    option "--binary-files", equals: true, value: true
    option "--text"
    option "-I", name: 	# FIXME: name
    option "--directories", equals: true, value: true
    option "--devices", equals: true, value: true
    option "--recursive"
    option "--dereference-recursive"
    option "--include", equals: true, value: true
    option "--exclude", equals: true, value: true
    option "--exclude-from", equals: true, value: true
    option "--exclude-dir", equals: true, value: true
    option "--files-without-match", value: true
    option "--files-with-matches"
    option "--count"
    option "--initial-tab"
    option "--null"
    option "--before-context", equals: true, value: {type: Num.new}
    option "--after-context", equals: true, value: {type: Num.new}
    option "--context", equals: true, value: {type: Num.new}
    option "--group-separator", equals: true, value: true
    option "--no-group-separator"
    option "--color", equals: :optional, value: {required: false}
    option "--colour", equals: :optional, value: {required: false}
    option "--binary"

    argument :patterns
    argument :file, required: false, repeats: true
  end

end

Note: some keyword arguments are intentionally left blank with a # FIXME command because command_mapper-gen cannot infer the names of every option (ex: -I).

Importance to the Ruby Ecosystem

Beyond providing a Ruby interface to external commands, and preventing arbitrary command injection, command_mapper and command_mapper-gen allows Ruby to quickly interface with other CLI utilities written in other programming language ecosystems that Ruby cannot bind to, such as Elixir, Go, Rust, Crystal, Nim, or Zig.

Using command_mapper we can automate other CLI utilities, written in other languages, parse their output or output files, all seamlessly from Ruby as if you were calling another Ruby library.

RubyGems Tasks

2012-05-22T00:00:00+00:00

Warning: Controversial Content

If you are morally or ethically opposed to building/releasing Ruby Gems with rake, this blog post may anger you. If this is the case, you are advised to STOP READING and CLICK THE BACK BUTTON.

Ever since we could release Ruby Gems, we have had Gem helpers that could generate new projects and provided Rake tasks to automate the building/publishing of Gems. The most popular of these Gem helpers were Hoe and Jeweler. Both Hoe and Jeweler required a certain project layout and imposed a certain workflow onto the developer.

Sometime before the introduction of Bundler, Yehuda Katz proposed a radically simpler way of building Gems, using the gemspec file. Developers could use the built-in gem build command to build a .gem file from a .gemspec, and use the gem push command to publish the Gem to rubygems.org. This marked the start of an exodus of sorts, away from using Gem helpers such as Hoe and Jeweler.

All you need is a Gemspec?

With the advent of building gems from a .gemspec, a vocal minority formed within the Ruby community. They proclaimed that all one needs is a gemspec, and that all other tools (Hoe, Jeweler, Bundler and even Rake) are now obsolete!

Let us take a look at how they might release a Gem:

$ git status
$ git push
$ gem build my_project.gemspec
$ gem push my_project-1.2.3.gem
$ git tag 1.2.3
$ git push --tags

It takes roughly six commands to release a Gem to rubygems.org. A developer must remember to run each of these six commands, in order, every time they release a new version of their Gem. The possibility for human error increases.

In fact, I have found Gems that contained newer source-code than their Git repository; because the developer forgot to commit or push the changes before pushing the Gem. I have also found Git repositories with no tags, making it difficult to review what exactly changed between versions.

Now, let us see how developers release gems using Rake tasks:

$ rake release

That’s it! That one command will run each of the above six commands. If any of the commands fail, the release process will halt. If there are uncommitted changes, the release process will halt. The possibility for human error has been greatly minimized.

Alternatives

Having used Hoe, Jeweler and then Bundler, I missed some of the workflow provided by these Gem helpers. So I began searching for lightweight alternatives.

I looked at Gem::PackageTask, MG, gem release and even bundler/gem_helper. Unfortunately they all were missing various features. Gem::PackageTask only built the .gem file, but did not check git status, tag the release or push the .gem. MG also did not check git status before releasing, did not tag releases, and only supports Git. gem release came close, but did not check git status before releasing, deleted the .gem file, only supports Git and tried to do too much (auto-magically bumping the version of your project for you). bundler/gem_helper came the closest, although while it defined Rake tasks it did not leverage Rake’s powerful task => dependency system.

So I decided to cherry-pick all of the nice features from Hoe, Jeweler, MG, gem release and bundler/gem_helper, and leave out the opinionated features.

rubygems-tasks

rubygems-tasks provides agnostic and unobtrusive Rake tasks for building, installing and releasing Ruby Gems.

$ gem install rubygems-tasks

Adding rubygems-tasks to an existing project is easy as:

require 'rubygems/tasks'
Gem::Tasks.new

$ rake -T
rake build    # Builds all packages
rake console  # Spawns an Interactive Ruby Console
rake install  # Installs all built gem packages
rake release  # Performs a release

Features

Provides tasks to build, install and push gems to rubygems.org:
- Loads all project metadata from the .gemspec file.
- Supports loading multiple .gemspec files.
- Supports pushing gems to alternate Gemcutter servers.
Supports optionally building .tar.gz and .zip archives.
Supports Git, Mercurial and SubVersion SCMs.
- Supports creating PGP signed Git/Mercurial tags.
Supports generating checksums of built packages:
```
Gem::Tasks.new(:sign => {:checksum => true})
```
Supports generating PGP signatures for built packages:
```
Gem::Tasks.new(:sign => {:pgp => true})
```
Provides a console task, for jumping right into your code.
Defines task aliases for users coming from Jeweler or Hoe.
ANSI coloured messages!

Anti-Features

Does not parse project metadata from the README or the ChangeLog.
Does not generate or modify your code.
Does not automatically commit changes.
Does not inject dependencies into gems.
Zero dependencies.

Examples

Specifying an alternate Ruby Console to run:

Gem::Tasks.new do |tasks|
  tasks.console.command = 'pry'
end

Enable pushing gems to an in-house Gemcutter server:

Gem::Tasks.new do |tasks|
  tasks.push.host = 'gems.company.com'
end

Disable the push task:

Gem::Tasks.new(:push => false)

Enable building .tar.gz and .zip archives:

Gem::Tasks.new(:build => {:tar => true, :zip => true})

Enable Checksums and PGP signatures for built packages:

Gem::Tasks.new(:sign => {:checksum => true, :pgp => true})

Selectively defining tasks:

Gem::Build::Tar.new
Gem::SCM::Status.new
Gem::SCM::Tag.new(:format => 'REL-%s')
Gem::Sign::Checksum.new
Gem::Console.new

You don't have to use Bundler to create new RubyGems

2012-05-20T00:00:00+00:00

Warning: Controversial Content

If you are morally or ethically opposed to using Project Generators and prefer to create RubyGems by hand, STOP READING AND CLICK THE BACK BUTTON.

I am perfectly aware that one does not need any tools to create a RubyGem, and that all you really need is RubyGems and a *.gemspec file. However, the majority of users do not have the time or patience to create each Ruby project from scratch. Thus developers have historically used project generators such as Hoe, Jeweler and now Bundler.

If you are a diehard Bundler user, please read the entire blog post. This blog post is not putting Bundler down, nor is it posing a binary choice between Bundler and some new tool.

Enter Bundler

Bundler was initially created as a more robust way to manage dependencies of Rails3 applications. Once Bundler was integrated into the Rails3 generator templates, developers realized Bundler could also be used to manage the dependencies of any large Ruby application or library.

In order to help developers create projects with Bundler already setup, template files and a bundle gem command were added. Since Bundler was the “new hotness” and developers were becoming increasingly dissatisfied with Jeweler/Hoe, the community began to cargo cult Bundler as the defacto way to create RubyGems.

Looking Back

After having extensively used Bundler with Ronin, to keep it’s many repositories in-sync with each other, I can definitely say Bundler solved dependency management for large Ruby projects. However, I began to question using bundle gem to create new RubyGems.

At first it made sense to provide a bundle gem command, but project generation is outside of the original scope of Bundler (dependency management). Furthermore, adding Bundler to an existing project isn’t that difficult; just add a .gemspec file and a Gemfile.

Given that Bundler’s stated goal is to “manage an application’s dependencies”, it doesn’t seem very pragmatic to use Bundler in libraries with zero or only one runtime dependency.

Bundler’s template files are a bit spartan as well. The Rakefile template does not include Rake tasks for RDoc or RSpec. This omission might encourage developers to not write documentation or tests, and rush to release.

The bundle gem command isn’t very configurable either. If you want to generate a project with Textile markup, YARD documentation, RSpec tests and Mercurial, you are out of luck.

Enter Ore

Ore is a flexible Ruby project generator. Unlike other Ruby project generators, Ore provides many built in templates and allows custom templates to be installed from Git repositories.

$ gem install ore
$ mine my_project
Generating /home/postmodern/my_project
      create  lib
      create  lib/my_project
      create  spec
      create  .gitignore
      create  .rspec
      create  spec/my_project_spec.rb
      create  spec/spec_helper.rb
      create  .document
      create  my_project.gemspec
      create  ChangeLog.rdoc
      create  LICENSE.txt
      create  README.rdoc
      create  Rakefile
      create  lib/my_project/version.rb
      create  lib/my_project.rb
         run  git init from "."
         run  git add . from "."
         run  git commit -m "Initial commit." from "."
$ cd my_project/
$ rake -T
rake build         # Builds all packages
rake clobber_rdoc  # Remove RDoc HTML files
rake console       # Spawns an Interactive Ruby Console
rake install       # Installs all built gem packages
rake rdoc          # Build RDoc HTML files
rake release       # Performs a release
rake rerdoc        # Rebuild RDoc HTML files
rake spec          # Run RSpec code examples

Ore generates new Ruby projects with sensible defaults, such as a .gemspec file, rubygems-tasks, RDoc, RSpec, a .gitignore file and initializes the Git repository.

Ore also provides many different templates and options:

$ mine --help
Usage:
  mine PATH

Options:
      [--gemspec-yml]               
      [--jeweler-tasks]             
      [--bundler]                   
      [--rubygems-tasks]            
                                    # Default: true
      [--yard]                      
      [--bundler-tasks]             
      [--hg]                        
      [--rspec]                     
                                    # Default: true
      [--rvmrc]                     
      [--gem-test]                  
      [--gem-package-task]          
      [--gemspec]                   
                                    # Default: true
      [--test-unit]                 
      [--git]                       
                                    # Default: true
      [--bin]                       
      [--rdoc]                      
                                    # Default: true
      [--markdown]                  
      [--textile]                   
  -T, [--templates=TEMPLATE [...]]  
  -n, [--name=NAME]                 
  -V, [--version=VERSION]           
                                    # Default: 0.1.0
  -s, [--summary=SUMMARY]           
                                    # Default: TODO: Summary
  -D, [--description=DESCRIPTION]   
                                    # Default: TODO: Description
  -a, [--authors=NAME [...]]        
                                    # Default: ["postmodern"]
  -e, [--email=EMAIL]               
  -U, [--homepage=HOMEPAGE]         
  -B, [--bug-tracker=BUG_TRACKER]   
  -L, [--license=LICENSE]           
                                    # Default: MIT

As you can see, Ore is completely configurable and supports:

Git / Mercurial / SubVersion
.rvmrc files
Bundler
rubygems-tasks / Bundler tasks / Jeweler::Tasks / Gem::PackageTask
RSpec / Test::Unit
YARD / RDoc documentation
RDoc / Markdown / Textile markup

Unlike other project generators, Ore focuses only on project generation and does not force a specific project layout or workflow upon the developer. You can even generate projects with Bundler, YARD + Markdown and Mercurial instead of Git:

$ mine my_project --bundler --yard --markdown --hg
Generating /home/postmodern/my_project
      create  lib
      create  lib/my_project
      create  spec
      create  .hgignore
      create  .document
      create  .yardopts
      create  Gemfile
      create  .rspec
      create  spec/my_project_spec.rb
      create  spec/spec_helper.rb
      create  my_project.gemspec
      create  ChangeLog.md
      create  LICENSE.txt
      create  README.md
      create  Rakefile
      create  lib/my_project/version.rb
      create  lib/my_project.rb
         run  hg init from "."
         run  hg add . from "."
         run  hg commit -m "Initial commit." from "."
$ cd my_project && bundle install
$ rake spec
/home/postmodern/.rvm/rubies/ruby-1.9.3-p194/bin/ruby -S rspec ./spec/my_project_spec.rb

MyProject
  should have a VERSION constant

Finished in 0.00593 seconds
1 example, 0 failures

Think Outside of the Bundle

Now that you have been introduced to Ore, I hope you will at the very least give it a try. I also hope you will understand that I am not simply anti-Bundler / pro-Ore. Ore gives you the option of generating Ruby projects with/without Bundler. The two tools are not mutually exclusive.

Reading RFID Cards on Linux (in the year 2012)

2012-04-04T00:00:00+00:00

A while back tenderlove blogged about using Ruby to interact with NFC tags. Near Field Communication (NFC) grew out of the Radio Frequency Identification (RFID) standard and has been used for contact-less payment with SmartPhones. Although, NFC tags are not yet widely deployed in the US, as opposed to Japan. On the other hand, RFID is widely deployed in the US, used in Payment systems, Asset management, Inventory systems, Product tracking, ID cards, Transportation tracking and even Passports. RFID is out there, and in those tags may be interesting information.

Not wanting to simply copy tenderlove and purchase an NFC starter kit, I went searching for a consumer grade RFID reader. I ended up purchasing an OmniKey 6321 USB reader/writer (ideal for mobile use) and a couple blank Mifare Read/Write RFID cards (with 8K EEPROM!). The next logical step is getting the reader working on Linux.

Unless you have a serial RFID reader (or can setup a USB Serial device with udev), the classic rfdump program is pretty much useless. librfid does not appear to be maintained any longer. Also, librfid-tool -S did not detect the OmniKey USB reader. Google quickly found many tutorials for using pcsc-lite to interact with SmartCard readers to read RFID tags.

Install pcscd

On Debian:

# apt-get install pcscd

On Fedora:

# yum install pcsc-lite

Install the OmniKey 6321 PC/SC Driver

Browser to www.hidglobal.com/driverDownloads.php?techCat=19&prod_id=274.
Select your Operating System. Choose “Linux” for 32bit or “Linux x64” for 64bit.
Download the ifdokrfid_lnx tar archive file.

Extract the ifdokrfid_lnx tar archive:

$ tar xzvf ifdokrfid_lnx*.tar.gz
$ cd ifdokrfid_lnx*/

Install the proprietary PC/SC Driver:
```
$ ./install
```

Start pcscd

Plugin the OmniKey 6321 USB reader and start pcscd:

# pcscd -a -f -d

Then you should see the following output:

00000083 pcscdaemon.c:575:main() pcsc-lite 1.7.4 daemon ready.
00002003 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000196 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000189 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x047D, PID: 0x2043, path: /dev/bus/usb/002/002
00000174 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x047D, PID: 0x2043, path: /dev/bus/usb/002/002
00000176 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/002/001
00000180 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x046D, PID: 0xC069, path: /dev/bus/usb/002/003
00000207 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/003/001
00000184 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x1D6B, PID: 0x0001, path: /dev/bus/usb/003/001
00000190 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x076B, PID: 0x6321, path: /dev/bus/usb/003/002
00000081 hotplug_libudev.c:258:get_driver() Looking for a driver for VID: 0x076B, PID: 0x6321, path: /dev/bus/usb/003/002
00000056 hotplug_libudev.c:309:HPAddDevice() Adding USB device: OMNIKEY 6321
00000087 readerfactory.c:934:RFInitializeReader() Attempting startup of OMNIKEY 6321 (USB iClass Reader) 00 00 using /usr/lib64/pcsc/drivers/ifdokrfid_lnx_x64-2.10.0.1.bundle/Contents/Linux/ifdokrfid.so
00000423 readerfactory.c:824:RFBindFunctions() Loading IFD Handler 3.0
HID HID Global OMNIKEY RFID  X64 v2.10.0.1 
00304886 readerfactory.c:295:RFAddReader() Using the reader polling thread

To test the reader, briefly place an RFID card near the reader then remove it:

99999999 eventhandler.c:372:EHStatusHandlerThread() powerState: POWER_STATE_POWERED
00000030 eventhandler.c:387:EHStatusHandlerThread() Card inserted into OMNIKEY 6321 (USB iClass Reader) 00 01
00000015 Card ATR: 3B 8F 80 01 80 4F 0C A0 00 00 03 06 03 00 03 00 00 00 00 68 
00462848 eventhandler.c:446:EHStatusHandlerThread() powerState: POWER_STATE_UNPOWERED
00305960 eventhandler.c:325:EHStatusHandlerThread() Card Removed From OMNIKEY 6321 (USB iClass Reader) 00 01

DIY Pagination with DataMapper

2012-04-04T00:00:00+00:00

There are many pagination solutions out there. Of course there’s the venerable will_pagination and the much newer Kaminari. However, all of the pagination solutions usually contain boiler-plate HTML. What if we only want the pagination logic, maybe in a JSON API, without using Rails or ActiveRecord, but instead Sinatra and DataMapper. As it turns out, DataMapper makes DIY pagination as simple as:

@posts = Post[((page - 1) * per_page), per_page]

As a Sinatra helper method this would look like:

def paginate(query)
  @page     = (params[:page] || 1).to_i
  @per_page = (params[:per_page] || 10).to_i

  query[((@page - 1) * @per_page), @per_page]
end

Now, what if we want to know the total number of pages and records? Enter the dm-chunked_query gem, which provides convenience methods for querying chunks of records:

require 'dm-chunked_query'

def paginate(query)
  @page        = (params[:page] || 1).to_i
  @per_page    = (params[:per_page] || 10).to_i

  @pages       = query.chunks_of(per_page)
  @total_count = @pages.count
  @page_count  = @pages.length

  @pages[@page - 1]
end

Pagination is that simple.

Hexdump 0.2.x

2011-06-11T00:00:00+00:00

Recently mephux began working on a hexdump.js library (epic demo). Naturally, I thought I should go back and improve my Ruby Hexdump library.

Hexdump 0.2.x

Hexdump now supports word-sizes and endianness. This is useful for dumping packed unsigned ints:

Hexdump.dump("\x12\x34\x00\x00\x42\x42", :word_size => 2, :endian => :big)
# 00000000  1234 0000 4242                           |ሴ䉂|

:word_size can have any value, and is not restricted to powers of 2.

Now that Hexdump parses multi-byte words from data, displaying Unicode characters was the next logical step:

Hexdump.dump("\xb6\x80" * 10, :word_size => 2)
# 00000000  80b6 80b6 80b6 80b6 80b6 80b6 80b6 80b6  |肶肶肶肶肶肶肶肶|
# 00000010  80b6 80b6                                |肶肶|

Finally, Hexdump received some performance tuning which cut benchmark times in half on Ruby 1.9.2:

Before

                                user     system      total        real
hexdump (block)             7.740000   0.030000   7.770000 (  8.138029)
hexdump                     9.590000   0.050000   9.640000 ( 10.178203)
hexdump width=256 (block)   7.280000   0.020000   7.300000 (  7.534507)
hexdump width=256           8.130000   0.030000   8.160000 (  8.342448)
hexdump ascii=true (block)  7.740000   0.030000   7.770000 (  7.958550)
hexdump ascii=true          9.550000   0.050000   9.600000 (  9.803758)

After

                                 user     system      total        real
hexdump (block)              3.010000   0.010000   3.020000 (  3.529396)
hexdump                      5.430000   0.030000   5.460000 (  6.216174)
hexdump width=256 (block)    3.010000   0.020000   3.030000 (  3.308961)
hexdump width=256            4.700000   0.040000   4.740000 (  5.520189)
hexdump ascii=true (block)   3.050000   0.010000   3.060000 (  3.501436)
hexdump ascii=true           5.450000   0.040000   5.490000 (  6.352144)
hexdump word_size=2 (block)  7.420000   0.050000   7.470000 (  9.174734)
hexdump word_size=2          9.500000   0.070000   9.570000 ( 11.228204)
hexdump word_size=4 (block)  4.110000   0.030000   4.140000 (  4.849785)
hexdump word_size=4          5.380000   0.060000   5.440000 (  6.209022)
hexdump word_size=8 (block)  3.350000   0.070000   3.420000 (  4.147304)
hexdump word_size=8          4.430000   0.040000   4.470000 (  5.930758)

Using ruby-prof and the Rubinius Profiler (rbx -Xprofile) I found that the majority of time spent was in:

Proc#call
Array#join
Integer#chr
String#%

Optimizing out these excess method calls involved:

Frequently called lambdas were replaced with methods.
Array#join was replaced with incremental String concatenating code.
A lookup table of bytes and printable characters were added, to reduce excess Integer#chr calls.
Interestingly, Kernel#sprintf is slightly faster than String#%.

Uses

Both hexdump.js and Ruby Hexdump aim to provide similar features and behaviors. hexdump.js is definitely useful for when you want to offload the work of Hexdumping to the Browser, or when writing node.js Apps.

The Ruby Hexdump library is more suited for Ruby CLI Apps, but can also be used in Web Apps:

dump = []

Hexdump.dump(data) do |index,numeric,printable|
  dump << [index, numeric, printable]
end

render :json => dump

Don't like WEBrick? Try net-http-server

2011-05-08T00:00:00+00:00

TL;DR

net-http-server:

$ gem install net-http-server

pure-Ruby HTTP Parser and Server
Small codebase
Fast-ish
Rack-like API
Rack Handler included
full YARD Documentation

WEBrick

Some have said that the WEBrick HTTP Server is a Ghetto. While WEBrick is very fast for a pure-Ruby HTTP Server, the parsing code is hand written and difficult to read. WEBrick is also one of the oldest Ruby HTTP Servers, but for some reason lacks documentation coverage. Given the rise of Rack, middleware and Rack applications, WEBricks API now seems awkward.

The Parser

When Parslet (a pure Ruby PEG Parser library) was announced, I wondered how hard would it be to write a HTTP Parser with Parslet. After researching the other Ragel based HTTP Parsers (Thin and Unicorn) and double checking RFC 2616, I suddenly had a pure-Ruby HTTP Parser. (in one file, that you can actually read!)

I found that the way in which Parslet nested matches into Arrays of Hashes resulted in data that looked very much like a Rack env Hash.

require 'net/http/server/parser'

parser = Net::HTTP::Server::Parser.new
parser.parse("GET /path?x=1&y=2 HTTP/1.1\r\nCookie: xyz;123\r\n\r\n")
# => {
#      :method=>"GET",
#      :uri=>{:path=>"path", :query=>"x=1&y=2"},
#      :version=>"1.1",
#      :headers=>[{:name=>"Cookie", :value=>"xyz;123"}]
#    }

The Daemon

The next step was to write an actual Daemon that would:

Receive Connections
Parse HTTP Requests
Pass HTTP Requests to a Request Handler
Receive HTTP Responses from the Request Handler
Send HTTP Responses
Close Connections

I settled on using the battle tested GServer class to handle the Connections for Nett:HTTP::Server::Daemon. I also borrowed some ideas from Rack, such as passing HTTP Requests via a call method and returning HTTP Responses as an Array (containing the HTTP Status, Headers and Body).

require 'net/http/server'
require 'pp'

Net::HTTP::Server.run(:host => '127.0.0.1', :port => 8080) do |request,socket|
  pp request

  [200, {'Content-Type' => 'text/html'}, ['Hello World']]
end

The Rack Handler

Given that the API was already very Rack-ish, writing a Rack handler on top of Net::HTTP::Server::Daemon was simple.

require 'rack/handler/http'
require 'sinatra'

class HelloWorld < Sinatra::Base

  get '/' do
    [200, {'Content-Type' => 'text/html'}, ["Hello World"]]
  end

end

Rack::Handler::HTTP.run HelloWorld, :Host => 'localhost', :Port => 1212

Benchmarks

By now your probably wondering, how fast is this pure Ruby HTTP Server?

require 'net/http/server'

Net::HTTP::Server.run(:port => 8080) do |request,socket|
  [200, {'Content-Type' => 'text/html'}, ['Hello World']]
end

$ ab -n 4000 -c 4 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
...

Finished 4000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        11 bytes

Concurrency Level:      4
Time taken for tests:   73.405 seconds
Complete requests:      4000
Failed requests:        0
Write errors:           0
Total transferred:      220000 bytes
HTML transferred:       44000 bytes
Requests per second:    54.49 [#/sec] (mean)
Time per request:       73.405 [ms] (mean)
Time per request:       18.351 [ms] (mean, across all concurrent requests)
Transfer rate:          2.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       8
Processing:    24   73  31.5     62     236
Waiting:       24   72  31.4     60     236
Total:         25   73  31.5     62     236

Percentage of the requests served within a certain time (ms)
  50%     62
  66%     86
  75%     98
  80%    103
  90%    119
  95%    134
  98%    148
  99%    153
 100%    236 (longest request)

Definitely not as fast as Thin or even WEBrick, but not too bad considering its pure-Ruby and the size of the code-base.

Typos in your Documentation?

2011-03-04T00:00:00+00:00

More likely than you think.

Having good documentation (internal and external) can make or break bigger projects. Good documentation helps skilled users/developers get up to speed with your project. However, bad documentation will drive potential users away. Typos are probably the most embarrassing thing to find in documentation.

One day I was fixing a trivial bug in my code for a new-user, and glanced over the documentation. Low and behold I spot a typo right above the broken method. In this moment of embarrassment (in front of the new-user), I said enough is enough, no more typos!

Since I use YARD for all of my projects, I decided to write a YARD plugin which would scan all of my documentation text using the Hunspell spellchecking library (via ffi-hunspell). A couple days later, yard-spellcheck was printing typos with file-names, line-numbers and ANSI highlighting.

As I suspected, I was able to find and fix a handful of typos in my own libraries, thanks to yard-spellcheck. Next, I started scanning larger projects with YARD documentation. I found a couple typos in DataMapper and even YARD itself (all are fixed now).

The lesson of this short story is that typos are lurking everywhere; too numerous and well hidden for human eyes to catch them all. Luckily, we now have an automated-tool in the fight against typos. The Ruby Community is notably obsessed with testing and quality of software, we should feel the same way about our documentation.

$ gem install yard-spellcheck
$ cd my_project/
$ yard-spellcheck

Introducing DeploYML

2010-11-21T00:00:00+00:00

When Rails 3.0.0.beta.1 was first released, there was some confusion about how to deploy a Rails3 app with this new Bundler thing. A few blog posts were written with some monkey patches for injecting into Capistrano, which enable early Bundler support. Unfortunately, Bundler was still very new and none of the deployment tools had official support yet.

Around the same time, I needed to deploy a Rails3 app and I did not want to deal with the heavy weight of Capistrano; nor did I want to rely on a monkey patch from a blog post. Of course the first thing I did was look at the alternatives, which were Vlad The Deployer or write a bash script that used rsync / ssh / git. After trying to run the vlad rake tasks, I hit some weird bugs with how it was calling git or thin. Being on a dead-line and not wanting to wait on Capistrano or Vlad, I thought it should be easy to create a simple deployment utility with git and ssh.

DeploYML

Introducing DeploYML, a simple deployment solution for Ruby / Rails projects that uses a single YAML file, Git and ssh.

Configuring DeploYML requires at least two things:

# config/deploy.yml
source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site

Then one can deploy the project using the deployml command:

$ deployml deploy

After making some changes, one can re-deploy the project:

$ deployml redeploy

Configuration

Of course, one will want to specify more information, such as what server to run the project under:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
server: apache

Or what options to run the server with:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
server:
  name: thin
  options:
    servers: 4
    deamonize: true
    socket: /var/run/thin.sock
    rackup: true

Or more importantly, what Framework or ORM the project uses:

source: git@github.com:user/project.git
dest: deploy@www.example.com/var/www/site
framework: rails3
orm: datamapper
server: apache

One can even specify multiple-environments:

# config/deploy.yml
source: git@github.com:user/project.git
framework: rails3
orm: datamapper

# config/deploy/staging.yml
dest: ssh://deploy@www.example.com/srv/staging
server:
  name: thin
  options:
    config: /etc/thin/staging.yml
    socket: /tmp/thin.staging.sock

# config/deploy/production.yml
dest: ssh://deploy@www.example.com/srv/project
server:
  name: thin
  options:
    config: /etc/thin/example.yml
    socket: /tmp/thin.example.sock

Administration

DeploYML does more than just deploying, it also allows interacting with the deployment server and deployed project.

Need to quickly ssh into the server as the deploy user?

$ deployml ssh

Need to quickly execute a command remotely, within the directory of the deployed project?

$ deployml exec 'ps aux'

Need to execute a rake task remotely?

$ deployml rake db:autoupgrade

Interested?

Install it today:

$ gem install deployml

Note: I originally started DeploYML before Imploy was released, and just now got around to making a release. If you feel Capistrano is too heavy, I encourage you to also checkout Imploy for deploying Rails3 apps.

Mining RubyGems from Ore

2010-10-25T00:00:00+00:00

Recently there has been some interesting discussion on the role of Gem builders and gemspecs. Jeff Kreeftmeijer wrote about how easy it is to build a RubyGem using a hand-written .gemspec file. With just a stand-alone .gemspec file one can:

Build RubyGems: gem build my-project.gemspec.
Publish RubyGems: gem push my-project-0.1.0.gem.

One thing worries me is the fact that you either have to specify everything explicitly in the gemspec, or use inline git commands to query the files of a project. Copying and pasting all this boilerplate code around seems like a potential future maintenance hassle; also not very DRY.

The argument for just a .gemspec file did get me thinking. I do find myself regenerating the gemspec with Jeweler, almost once per-day. Also, these stand-alone gemspecs are pretty succinct. So when in doubt, see how other languages solved the problem.

Code vs. Data

I asked one of my Haskell friends how Cabal (the Haskell packager of choice) solves this problem. He pointed me to the Cabal file of his Haskell webapp serialist.net. Cabal uses easy to read, easy to parse and easy to edit plain-text. This reminded me, Code is for describing logic and flat-files are for describing static data. The majority of the information in the .gemspec file, is static data.

Now I am starting to really question the whole reason for an explicit gemspec. The .gemspec file only exists to create a Gem::Specification object, which is then passed to Gem::Builder or loaded by Bundler. Why are we writing Ruby code that normally would only exist in-memory? The gemspec purists state this is to allow things such as, dynamically loading the VERSION constant from the project or dynamically listing files tracked by Git. Although, both of these tasks can easily be automated by a library.

So I wondered, why not have a small library that loads the project information from a YAML file, fills in any missing information, and then creates a new Gem::Specification object. Furthermore, If we can call a method and get a Gem::Specification object back, we could place this in the .gemspec file for both gem build and Bundler to make use of.

Introducing Ore

Ore allows you to define all project information for a RubyGem in a single YAML file (gemspec.yml).

name: ore
version: 0.1.2
summary: Cut raw RubyGems from YAML
description:
  Ore is a simple RubyGem building solution. Ore handles the
  creation of Gem::Specification objects as well as building '.gem'
  files. Ore allows the developer to keep all of the project information
  in a single YAML file.

license: MIT
authors: Postmodern
email: postmodern.mod3@gmail.com
homepage: http://github.com/postmodern/ore
has_yard: true

dependencies:
  thor: ~> 0.14.3

development_dependencies:
  yard: ~> 0.6.1
  rspec: ~> 2.0.0

With Ore, one can write their description as free-text, no more using with awkward Ruby %Q{...} syntax.

Dependencies are listed in a YAML Hash, no more repeating add_dependency or add_development_dependency.

Ore can also infer missing information. If the version is not specified, Ore will search for and parse any VERSION or VERSION.yml files. Ore can even slurp up any VERSION, MAJOR, MINOR, PATCH or BUILD constants from a version.rb file in the lib/ directory. Also, notice that files is not listed, this is because Ore can detect the project is using Git, and list all files tracked by Git.

For a complete reference of everything that may go into a gemspec.yml file, and how Ore infers missing data, please see GemspecYML.

Building gems with Ore is easy as:

$ ore
Successfully built RubyGem
Name: ore
Version: 0.1.2
File: ore-0.1.2.gem
$ ls pkg/
ore-0.1.2.gem

One can still get the traditional gemspec from Ore:

$ ore gemspec
# -*- encoding: utf-8 -*-

Gem::Specification.new do |s|
  s.name = %q{ore}
  s.version = "0.1.2"

  s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
  s.authors = ["Postmodern"]
  s.date = %q{2010-10-25}
  s.default_executable = %q{ore}
...

We can even use Ore in .gemspec files:

# -*- encoding: utf-8 -*-

begin
  Ore::Specification.new do |gemspec|
    # custom logic here
  end
rescue NameError
  begin
    require 'ore/specification'
    retry
  rescue LoadError
    STDERR.puts "The 'my-project.gemspec' file requires Ore."
    STDERR.puts "Run `gem install ore-core` to install Ore."
  end
end

Ore will still work with gem build and even Bundler.

Mining RubyGems from Ore

Ore also comes with an extendable generator, for creating new projects:

$ mine my-project
Generating /home/hal/my-project
  create  lib
  create  lib/my/project
  create  spec
  create  .rspec
  create  spec/my/project_spec.rb
  create  spec/spec_helper.rb
  create  .document
  create  .gitignore
  create  my-project.gemspec
  create  ChangeLog.rdoc
  create  LICENSE.txt
  create  README.rdoc
  create  Rakefile
  create  gemspec.yml
  create  lib/my/project/version.rb
  create  lib/my/project.rb
     run  git init from "."
     run  git add . from "."
     run  git commit -m "Initial commit." from "."

By default mine will generate an RDoc and test-unit project. mine can also generate very customized projects:

$ mine my-project --rspec --yard --markdown --bundler
Generating /home/hal/my-project
      create  lib
      create  lib/my/project
      create  spec
      create  .yardopts
      create  .rspec
      create  spec/my/project_spec.rb
      create  spec/spec_helper.rb
      create  Gemfile
      create  .document
      create  .gitignore
      create  my-project.gemspec
      create  ChangeLog.md
      create  LICENSE.txt
      create  README.md
      create  Rakefile
      create  gemspec.yml
      create  lib/my/project/version.rb
      create  lib/my/project.rb
         run  git init from "."
         run  git add . from "."
         run  git commit -m "Initial commit." from "."

mine simply renders Ore Templates. Unlike other generators which have their logic hard-coded in Ruby, Ore Templates are simply directories, containing static and ERb files. One can make their own Ore template by creating a directory, adding files and publishing a Git repository.

Users can install custom Ore templates from Git repositories:

$ ore install http://github.com/user/awesometest.git
$ ore list
Builtin templates:
  base
  rspec
  test_unit
  yard
  bundler
  jeweler_tasks
  rdoc
  ore_tasks
Installed templates:
  awesometest

Then use the -T option to specify additional custom templates:

$ mine my-project --yard --markdown -T awesometest

Workflow

By default Ore does not impose a workflow onto the developer. Even the mine utility does not add any additional Rake tasks to new projects. This allows the developer to use gem build and gem push, or even use Jeweler::Tasks with Ore.

For those just wanting simple Rake tasks to build, push and tag releases, there is ore-tasks. Ore::Tasks provides the following tasks:

rake build            # Only builds a Gem
rake console[script]  # Start IRB with all runtime dependencies loaded
rake gem              # Alias to the 'build' task
rake install          # Builds and installs a Gem
rake push             # Builds and pushes a Gem
rake release          # Builds and Pushes a new Gem / Build, Tags and Pushe...
rake tag              # Tags a release and pushes the tag
rake version          # Displays the current version

To generate a project with Ore::Tasks included:

$ mine my-project --ore-tasks

To generate a project with Jeweler::Tasks included:

$ mine my-project --jeweler-tasks

Dog Fooding

In order to do real-world testing with Ore, I created ore-example which uses Bundler, RSpec2, YARD and Ore::Tasks.

As of now, I have also migrated my chars and uri-query_params libraries to Ore.

Interested?

$ gem install ore
$ mine the-future

For questions or feedback, join #ruby-ore on irc.freenode.net.

All source-code is located on GitHub.

Ore is tested with RSpec2 and has extensive YARD documentation.

Introducing Combinatorics 0.2.0

2010-10-03T00:00:00+00:00

I decided to take my List Comprehension code with some other Combinatorics code I had floating around and release Combinatorics 0.2.0:

$ gem install combinatorics

After getting specs on the List Comprehension code, it became easy to refactor it and get the specs passing on Ruby 1.8.7, 1.9.2 and JRuby. Unfortunately, Rubinius does not yet support Enumerator#next.

The Combinatorics library also contains the powerset method, added to Array and Set:

require 'combinatorics'

[1,2,3].powerset
# => [[],
      [3],
      [2],
      [2, 3],
      [1],
      [1, 3],
      [1, 2],
      [1, 2, 3]]

In Combinatorics 0.2.0, Range#&, Range#upto and Range#downto were also added:

(1..50) & (20..100)
# => (20..50)

(1..5).upto(2..10).to_a
# => [1..5, 1..6, 1..7, 1..8, 1..9, 1..10,
      2..5, 2..6, 2..7, 2..8, 2..9, 2..10]

(2..10).downto(1..5).to_a
# => [2..10, 2..9, 2..8, 2..7, 2..6, 2..5,
      1..10, 1..9, 1..8, 1..7, 1..6, 1..5]

Fork the Combinatorics library today, and add your own Combinatoric methods.

List Comprehensions in Ruby

2010-10-01T00:00:00+00:00

Recently I have been interested in Haskell and been skimming the whimsical (yet informative) Learn a Haskell, for Great Good!. One feature I really like from Haskell is their implementation of list comprehensions:

Prelude> [(x,y) | x <- [1..10], y <- [2..8]]
[(1,2),(1,3),(1,4),(1,5),(1,6),(1,7),(1,8),
(2,2),(2,3),(2,4),(2,5),(2,6),(2,7),(2,8),
(3,2),(3,3),(3,4),(3,5),(3,6),(3,7),(3,8),
(4,2),(4,3),(4,4),(4,5),(4,6),(4,7),(4,8),
(5,2),(5,3),(5,4),(5,5),(5,6),(5,7),(5,8),
(6,2),(6,3),(6,4),(6,5),(6,6),(6,7),(6,8),
(7,2),(7,3),(7,4),(7,5),(7,6),(7,7),(7,8),
(8,2),(8,3),(8,4),(8,5),(8,6),(8,7),(8,8),
(9,2),(9,3),(9,4),(9,5),(9,6),(9,7),(9,8),
(10,2),(10,3),(10,4),(10,5),(10,6),(10,7),(10,8)]

This list comprehension is essentially a Set definition using first-order predicate logic, with universal quantifiers on x and y (where the Set contains the tuples composed of every x and every y from the ranges 1..10 and 2..8). Amazingly, the Haskell version looks almost exactly like the Set definitions from my Discrete Structures, Logic, and Computability (2nd edition) book, except without the curly-braces or universal quantifiers ().

One does not have to learn functional programming and Haskell to use list comprehensions, Python has them as well:

>>> [(x,y) for x in range(0,11) for y in range(2,9)]
[(0, 2), (0, 3), (0, 4), (0, 5), (0, 6), (0, 7), (0, 8),
(1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (1, 7), (1, 8),
(2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (2, 7), (2, 8),
(3, 2), (3, 3), (3, 4), (3, 5), (3, 6), (3, 7), (3, 8),
(4, 2), (4, 3), (4, 4), (4, 5), (4, 6), (4, 7), (4, 8),
(5, 2), (5, 3), (5, 4), (5, 5), (5, 6), (5, 7), (5, 8),
(6, 2), (6, 3), (6, 4), (6, 5), (6, 6), (6, 7), (6, 8),
(7, 2), (7, 3), (7, 4), (7, 5), (7, 6), (7, 7), (7, 8),
(8, 2), (8, 3), (8, 4), (8, 5), (8, 6), (8, 7), (8, 8),
(9, 2), (9, 3), (9, 4), (9, 5), (9, 6), (9, 7), (9, 8),
(10, 2), (10, 3), (10, 4), (10, 5), (10, 6), (10, 7), (10, 8)]

Granted, using the for loop syntax as constraints for the list comprehension is not as sexy as the Haskell version, but it gets the job done.

Unfortunately, Ruby does not support list comprehensions, and only has a couple methods for doing Combinatorics (Array#combination and Array#permutation). So, I implemented list comprehensions for the Ruby Array class:

>> require './array_comprehension'
=> true
>> [(1..10), (2..8)].comprehension.to_a
=> [[1, 2], [1, 3], [1, 4], [1, 5], [1, 6], [1, 7], [1, 8],
    [2, 2], [2, 3], [2, 4], [2, 5], [2, 6], [2, 7], [2, 8],
    [3, 2], [3, 3], [3, 4], [3, 5], [3, 6], [3, 7], [3, 8],
    [4, 2], [4, 3], [4, 4], [4, 5], [4, 6], [4, 7], [4, 8],
    [5, 2], [5, 3], [5, 4], [5, 5], [5, 6], [5, 7], [5, 8],
    [6, 2], [6, 3], [6, 4], [6, 5], [6, 6], [6, 7], [6, 8],
    [7, 2], [7, 3], [7, 4], [7, 5], [7, 6], [7, 7], [7, 8],
    [8, 2], [8, 3], [8, 4], [8, 5], [8, 6], [8, 7], [8, 8],
    [9, 2], [9, 3], [9, 4], [9, 5], [9, 6], [9, 7], [9, 8],
    [10, 2], [10, 3], [10, 4], [10, 5], [10, 6], [10, 7], [10, 8]] 

I used yield to quickly pass results to the given block, to not stress the Garbage Collector with building a huge Arrays. The method also returns an Enumerator object if no block is given, to fake Haskells lazy evaluation.

Note, that the Array may contain:

Enumerable objects:

  >> [('a'..'f'),(0..10).step(2)].comprehension.to_a
  => [["A", 0], ["A", 2], ["A", 4], ["A", 6], ["A", 8], ["A", 10],
      ["B", 0], ["B", 2], ["B", 4], ["B", 6], ["B", 8], ["B", 10],
      ["C", 0], ["C", 2], ["C", 4], ["C", 6], ["C", 8], ["C", 10],
      ["D", 0], ["D", 2], ["D", 4], ["D", 6], ["D", 8], ["D", 10],
      ["E", 0], ["E", 2], ["E", 4], ["E", 6], ["E", 8], ["E", 10],
      ["F", 0], ["F", 2], ["F", 4], ["F", 6], ["F", 8], ["F", 10]] 

Non-enumerable objects:

  >> [5,(6..9),10].comprehension.to_a
  => [[5, 6, 10], [5, 7, 10], [5, 8, 10], [5, 9, 10]]

Even other list comprehensions:

  >> syllable = (['a'..'z'] * 2).comprehension
  >> word = [syllable] * 4
  >> number = (1..100)
  >> [*word, number].comprehension { |*s| puts s.join }

This code does require Ruby 1.9 since Enumerator#peek was not back-ported to Ruby 1.8.7.

$ rvm install 1.9.2

Introducing OpenNamespace

2010-09-03T00:00:00+00:00

I have always liked how frameworks such as Rails can autoload Classes. I wanted to provide similar behavior in Ronin and other frameworks, so I created the OpenNamespace library.

OpenNamespace allows namespaces to require and find classes and modules from RubyGems. Using OpenNamespace you can make a Plugins module able to load plugin modules/classes from other gems.

More specifically, OpenNamespace does not need to know where the files are, it just guesses the file path based on the constant name, attempts to require it then finds the constant in the namespace.

$ gem install open_namespace

Examples

require 'open_namespace'

module Project
  module Plugins
    include OpenNamespace
  end
end

Explicitly load constants:

Project::Plguins.require_const :foo_bar
# => Project::Plugins::FooBar

Explicitly load constants with odd capitalization:

Project::Plugins.require_const :tcp_session
# => Project::Plugins::TCPSession

Explicitly load constants via sub-paths:

Project::Plguins.require_const 'templates/erb'
# => Project::Plugins::Templates::Erb

Implicitly load constants via const_missing:

Project::Plugins::Other
# => Project::Plugins::Other

Enjoy.

Introducing uri-query_params

2010-09-02T00:00:00+00:00

uri-query_params is a new library which allows accessing the individual parameters in the query string of a HTTP URI in Ruby. The library monkey-patches the URI::HTTP class to provide similar behavior to PHPs famous $_GET hash.

$ gem install uri-query_params

Using uri-query_params you can, inspect the URI query_params:

require 'uri/query_params'

url = URI('http://www.google.com/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&hs=1HY&q=bob+ross&btnG=Search')

url.query_params
# => {"btnG"=>"Search", "hs"=>"1HY", "rls"=>"org.mozilla:en-US:official", "client"=>"firefox-a", "hl"=>"en", "q"=>"bob+ross"}

url.query_params['q']
# => "bob+ross"

Also, set the URI query_params:

url.query_params['q'] = 'Upright Citizens Brigade'
url.to_s
# => "http://www.google.com/search?btnG=Search&hs=1HY&rls=org.mozilla:en-US:official&client=firefox-a&hl=en&q=Upright%20Citizens%20Brigade"

Originally, the URI::QueryParams mixin was developed for Ronin and GScraper. I decided to split the code out into a common library, to prevent conflicts between the two separate versions.

Enjoy.