On Benchmarking

Sigh. It must be that time of year again. Another partially-completed Ruby implementation has started to get overhyped because of early performance numbers.

MacRuby has been mentioned on this blog before. It's a reimplementation of Ruby 1.9 targeting the Objective-C runtime--and now, targeting LLVM for immediately compiling Ruby code to native code. Initial performance results running some of my benchmark show an interesting mixed bag. For some, MacRuby's new "experimental" branch performs very well, in some cases a few times faster than JRuby. For others, performance is slow enough there must be something wrong. And there's a large number of my benchmarks that don't even run, due to broken features they'll be fixing over the next several months.

And yet, at least one Rubyist has already seen fit to declare MacRuby "the fastest Ruby implementation around". Really? When it's crashing for about half the scripts I ran and extremely slow for many others?

He bases this assertion on running the benchmarks MacRuby includes in its own repository. Because MacRuby usually performs much better on those benchmarks than Ruby 1.9, he has decided they're now "the fastest Ruby". Do we have to do this hype dance every year?

Look, I know I'm biased. I want JRuby to be the best Ruby implementation possible. I want it to be fast, and if possible, the fastest. I also want it to run existing Ruby applications and integrate well with Java libraries and applications and continue to be one of the best choices for running Ruby. So I can understand that it sounds like I'm throwing stones by pouring water on such a breathless proclamation as "fastest Ruby implementation around". But seriously guys...haven't we learned anything?

MacRuby's experimental branch is just that: experimental. Lots of stuff is fast, but lots of stuff is broken or slow. I'm sure the MacRuby guys are going to get everything resolved and working, and I'll admit these early results drive me to work on JRuby performance even harder. But I also know from experience that many of the missing features are exactly those that make Ruby performance a really difficult problem. That's why we've always focused on compatibility first (almost to a fault); it's really easy to paint yourself into a corner.

But this post isn't about MacRuby. They're doing awesome work, and I have no doubt at least some of the performance numbers will stick. This post is about the evils of benchmarking, especially prematurely.

Around this time last year, MagLev (Ruby based on the Gemstone VM) posted some crazy benchmarks and shocked the Ruby world at RailsConf. They had numbers even more stunning than MacRuby, running some simple numerical benchmarks orders of magnitude faster than either Ruby 1.8 or 1.9. Several Ruby bloggers immediately posted not just their enthusiasm, but their belief that MagLev had won the performance battle without ever firing a shot.

And I believe it was a great disservice to the MagLev team.

MagLev was, last spring, a very primitive and early implementation. It could run some useful Ruby code, but the majority of the core classes had not yet been implemented and very little work had been done on compatibility. Now we're approaching a year later, and MagLev is still in development, still closed source, still at a private alpha stage of life. Again, I'll admit I'm biased, so I need to state that I believe MagLev is also a really cool technology, at least as cool as MacRuby or JRuby. In many ways and for many domains both of them are going to be more compelling than JRuby, and I have no illusions that JRuby will never get leapfrogged in performance. But we need to remember a really important fact: these implementations are not done.

I could post blog entries with every experimental branch of JRuby I've ever tested. I could show you "fib" numbers 3-5 times faster than current JRuby and 10 times faster than Ruby 1.9. But honestly, what would be the point? I know it's experimental, I know we need to get there in a careful, measured way, and I know that my best experiments may never be reflected in real-world, real-application performance. And yet it seems like people just love to latch on to these early contenders, hyping them to death almost before they're out of the starting gate.

Listen, people: Ruby is hard to implement. Oh, it may look easy at a glance, and you can probably get 70, 80, or even 90% of the way pretty quickly. But there's some crazy stuff in that last 10% or 5% that totally blindsides you if you're not looking for it. An early Ruby implementation has not run that last mile of Ruby implementation, and it takes almost as much work to get there as it does to run the first 90%.

So let's try to be adults about this and give new implementations time to actually finish before we whip the community into a frenzy. Every time we go overboard in our declarations, we look like amateurs. And as certain as I am that MacRuby is going to be a major contender for the "fastest Ruby" crown, I think we'd be wise to hold judgment until it and other young Ruby implementations are actually finished.

BiteScript 0.0.1 - A Ruby DSL for JVM Bytecode

I have finally released the first version of BiteScript, my little DSL for generating JVM bytecode. Install it as a gem with "gem install bitescript".

require 'bitescript'

include BiteScript

fb = FileBuilder.build(__FILE__) do
public_class "SimpleLoop" do
 public_static_method "main", void, string[] do
   aload 0
   push_int 0
   aaload
   label :top
   dup
   aprintln
   goto :top
   returnvoid
 end
end
end

fb.generate do |filename, class_builder|
File.open(filename, 'w') do |file|
 file.write(class_builder.generate)
end
end

BiteScript grew out of my work on Duby. I did not want to call directly into a Java bytecode API like ASM, so I wrapped it with a nice Ruby-like layer. I also wanted the option of having blocks of bytecode look like raw assembly, but also callable as an API.

Currently only two projects I know of make use of BiteScript: Duby and the upcoming Ruby-to-Java "compiler2" in JRuby, which will also be released as a gem.

For a longer example, you can look at tool/compiler2.rb in JRuby, lib/duby/jvm/jvm_compiler.rb in Duby, or an example implementation of Fibonacci in BiteScript.

I'm open to suggestions for how to improve the API, and I'd also like to add the missing Java 5 features. The better BiteScript works, the better Duby and "compiler2" will work.

For folks interested in using BiteScript, the JVM Specification is an easy-to-read complete reference for targeting the JVM, and here is my favorite JVM opcode quickref.

More Compiling Ruby to Java Types

I did another pass on compiler2, and managed to wire in signature support. So let's look at a couple examples:

class MyRubyClass
 def helloWorld
   puts "Hello from Ruby"
 end
 def goodbyeWorld(a)
   puts a
 end

 signature :helloWorld, [] => Java::void
 signature :goodbyeWorld, [java.lang.String] => Java::void
end

In this case we have our friend MyRubyClass once again, with helloWorld and goodbyeWorld methods. You'll recall from my previous post that these two methods originally compiled as returning IRubyObject, and goodbyeWorld compiled as receiving a single IRubyObject parameter.

But with signature support, things are so much cooler! The two "signature" lines at the bottom of the class (syntax and structure are totally up for debate) associated signatures with the two methods. helloWorld receives no parameters and has a void return type. goodbyeWorld receives a single String parameter and has a void return type.

The compiler takes this new information, and produces a more normal-looking set of Java signatures:

Compiled from "MyObject.java.rb"
public class MyObject extends org.jruby.RubyObject{
   static {};
   public MyObject();
   public void helloWorld();
   public void goodbyeWorld(java.lang.String);
}

Huzzah! There's almost nothing here to give away that we're actually dealing with Ruby code under the covers. And the code that consumes this is just as simple:

public class MyObjectTest {
 public static void main(String[] args) {
   MyObject obj = new MyObject();
   obj.helloWorld();
   obj.goodbyeWorld("hello");
 }
}

And that's literally all there is to it. Here's a more advanced example:

class MyRubyClass
 %w[boolean byte short char int long float double].each do |type|
   java_type = Java.send type
   eval "def #{type}Method(a); a; end"
   signature "#{type}Method", [java_type] => java_type
 end
end

This time we're actually *generating* the methods, looping over a list of Java primitives and eval'ing a method for each. So this is *runtime* generation of methods, like any good Rubyist loves to do. And of course, this is absolutely no problem for compiler2:

Compiled from "MyObject2.java.rb"
public class MyObject2 extends org.jruby.RubyObject{
   static {};
   public MyObject2();
   public double doubleMethod(double);
   public int intMethod(int);
   public char charMethod(char);
   public short shortMethod(short);
   public boolean booleanMethod(boolean);
   public float floatMethod(float);
   public long longMethod(long);
   public byte byteMethod(byte);
}

All the methods are there, just as you'd expect them! Fantastic!!! (Though the ordering is a little peculiar; I think that's because we don't have an ordered method table in our class impl. Does it matter?)

Even better, the above methods are doing the same type coercion on the way in and out that we do for any other Java-based method calling. So your integral numerics are presented to Ruby as Fixnums, floating-point numerics are Floats, and booleans come through as Ruby true or false.

There's certainly more work to be done:

• There's no support for overloads at the moment, but I'll likely provide a method aliasing facility so you can define multiple Ruby methods and then say which one maps to which overload. And of course, you'll be able to define multiple overloads that go to the same method body if you wish.
• I also have not wired in varargs, but it will be an easy match to Ruby's restargs. And optional arguments could automatically generate different-arity Java signatures.
• Annotations will also be trivial to add; it's just a matter of attaching appropriate metadata and having compiler2 emit them. So you'll be able to use JavaEE 5, JUnit4, and any other frameworks that depend on having annotations present.

Of course this is all checked into JRuby trunk, so feel free to give it a try. Stop by JRuby mailing lists or IRC if you have questions. And it's all still written in Ruby; signature support bloated the compiler up to a whopping 178 lines of code, most of that for dealing with the JVM opcodes for primitive types.

This is just the beginning!

Compiling Ruby to Java Types

"Compiler #2" as it has been affectionately called is a compiler to turn normal Ruby classes into Java classes, so they can be constructed and called by normal Java code. When I asked for 1.3 priorities, this came out way at the top. Tom thought perhaps I asked for trouble putting it on the list, and he's probably right (like asking "prioritize these: sandwich, pencil, shiny gold ring with 5kt diamond, banana"), but I know this has been a pain point for people.

I have just landed an early prototype of the compiler on trunk. I made a few decisions about it today:

• It will use my bytecode DSL "BiteScript", just like Duby does
• It will use the *runtime* definition of a class to generate the Java version

The second point is an important one. Instead of having an offline compiler that inspects a file and generates code from it, the compiler will actually used the runtime class to create a Java version. This means you'll be able to use all the usual metaprogramming facilities, and at whatever point the compiler picks up your class it will see all those methods.

Here's an example:

# myruby.rb
require 'rbconfig'

class MyRubyClass
  def helloWorld
    puts "Hello from Ruby"
  end
  if Config::CONFIG['host_os'] =~ /mswin32/
    def goodbyeWorld(a)
      puts a
    end
  else
    def nevermore(*a)
      puts a
    end
  end
end

Here we have a class that defines two methods. The first, always defined, is helloWorld. The second is conditionally either goodbyeWorld or nevermore, based on whether we're on Windows. Yes, it's a contrived example...bear with me.

The compiler2 prototype can be invoked as follows (assuming bitescript is checked out into ../bitescript):

jruby -I ../bitescript/lib/ tool/compiler2.rb MyObject MyRubyClass myruby

A breakdown of these arguments is as follows:

• -I ../bitescript/lib includes bitescript
• tool/compiler2.rb is the compiler itself
• MyObject is the name we'd like the Java class to have
• MyRubyClass is the name of the Ruby class we want it to front
• myruby is the library we want it to require to load that class

Running this on OS X and dumping the resulting Java class gives us:

Compiled from "MyObject.java.rb"
public class MyObject extends org.jruby.RubyObject{
 static {};
 public MyObject();
 public org.jruby.runtime.builtin.IRubyObject helloWorld();
 public org.jruby.runtime.builtin.IRubyObject nevermore(org.jruby.runtime.builtin.IRubyObject[]);
}

The first thing to notice is that the compiler has generated a method for nevermore, since I'm not on Windows. I believe this will be unique among dynamic languages on the JVM: we will make the *runtime* set of methods available through the Java type, not just the static set present at compile time.

Because there are no type signatures specified for MyRubyClass, all types have defaulted to IRubyObject. Type signature logic will come along shortly. And notice also this extends RubyObject; a limitation of the current setup is that you won't be able to use compiler2 to create subclasses. That will come later.

Once you've run this, you've got a MyObject that can be instantiated and used directly. Behind the scenes, it uses a global JRuby instance, so JRuby's still there and you still need it in classpath, but you won't have to instantiate a runtime, pass it around, and so on. It should make integrating JRuby into Java frameworks that want a real class much easier.

So, thoughts? Questions? Have a look at the code under tool/compiler2.rb in JRuby's repository. The entire compiler is so far only 78 lines of Ruby code.