fidzu : Parrot

Earlier this month I released the new Reflect library in Rosella. I hadn't mentioned it before, but the library is sufficiently interesting that I want to talk about it at least a little bit. The Reflect library adds in tools for reflection. Somewhere, an etymologist weeps a tear of joy for the creative naming, I'm sure.

The Reflect library adds in wrappers for classes and packfiles that makes them easier to work with for many operations. First, I'd like to use a couple code examples to show the most basic API:

// Get the Sub PMC that we're currently executing
var s = Rosella.Reflect.get_current_sub();

// Get the current context
var c = Rosella.Reflect.get_current_context();

// Get the current object, if the current Sub is a method call
var obj = Rosella.Reflect.get_current_object();

// Get the class of the current object, if the current Sub is a method call
var c = Rosella.Reflect.get_current_class();

// Get a Module object for the packfile where the current Sub is defined
var m = Rosella.Reflect.get_current_module();

// Get a reflection wrapper object for the given Parrot Class PMC
var r = Rosella.Reflect.get_class_reflector(myClass);

// Get a Module object for the packfile in "foo/bar.pbc", loading it as
// necessary
var m = Rosella.Reflect.Module.load("foo/bar.pbc");

That's the basic API that the library provides to get basic information about where execution is happening at the moment when the call is made. Once you have a Module object or a Class reflector object, you can do all sorts of cool things that used to be a pain in the butt to do manually:

var m = Rosella.Reflect.get_current_module();
say(m);          // Stringified, produces the name and version of the packfile
m.load();        // Execute all :tag("load") and :load functions
n.init();        // Execute all :tag("init") and :init functions
say(m.version(); // Get the version string of the packfile "X.Y.Z"
say(m.path());   // The on-disk path to the current packfile

// Get a hash of all Class PMCs defined at compile-time (using the :method
// flag on Subs) defined in the packfile, keyed by name
var c = m.classes();

// Get a list of all non-:anon functions defined in the packfile
var f = m.functions();

// Get a hash of all non-:anon functions in the packfile, organized into
// a hash keyed by namespace
var f = m.functions_by_ns();

// Get a hash of all NameSpace PMCs defined at compile-time
var ns = m.namespaces();

Once you have Class and NameSpace PMCs from the packfile, you can start to do all sorts of cool operations and analyses on them. Once you have a Class reflector object, you can do even more stuff with that:

var c = Rosella.Reflect.get_current_class();

// Create a new object of the current type
var o = c.new();

// Say the name of the class
say(c.name());

// Attributes are encapsulated as objects. You can get an Attribute
// reflector and use it later to get and set values on objects of this
// type or subclasses
var attr = c.get_attr("foo");
var value = attr.get_value(o);
attr.set_value(o, "whatever");

// Methods are also encapsulated. You can get a method reflector now and
// invoke it on objects later (including objects of different types)
var method = c.get_method("bar");
var result = method.invoke(o);
var meths = c.get_all_methods();

// Basic capability detection. Determine if objects are members of the
// class or their subsets, and determine if the class can perform certain
// methods
if (c.isa(o)) { ... }
if (c.can("bar")) { ... }

I hope the code examples make up for the terse explanations.

The Reflect library is currently focused on reading data from things like Classes and Packfiles, not on creating these things like the new PACT project is supposed to do. I want to extend this library even further with abilities to further introspect functions down to the opcode level and then…Well, when we have a stream of opcodes to analyze the possibilities are endless. I'd also like the ability to get better introspection of the interpreter and global state, though a cleaner interface than the hodge-podge of interpinfo opcodes and ParrotInterpreter PMC methods and whatever else we currently use.

As always, using the interface Rosella provides will help to insulate you from changes to the various underlying mechanisms when we finally get around to cleaning them up and making them sane. There isn't a huge push to make such cleanups on a large scale yet, but I wouldn't be surprised if a few things started getting prettified in the coming months at a slow pace.

I've already started using the new library in several of the Rosella utility programs such as those that create a winxed header file or a test suite from an existing packfile. In all cases the updated programs are both cleaner and have more functionality than the previous incarnations. Expect to see this library improve and grow in 2012 and beyond, and expect to see it work closely with PACT, once that project gets moving forward.

01 Feb 2012 8:00am GMT

The first half of this month was dominated with some epic illnesses between my family members and myself, family functions and home maintenance. What little spare time I've had otherwise has been devoted to writing code, as opposed to writing blog posts about writing code. The blog has suffered.

The past couple days I've been working on Rosella's Test library. It's an old but good library and is, as far as I am aware, the most full-featured and easy to use testing tool in the Parrot ecosystem. With some of these most recent changes the library is better still.

Matchers

Kakapo had a series of Matcher routines and objects as part of it's testing facilities, and for a long time I've been wanting to port some of those ideas over to Rosella. As of last week, I have a simple version of them. Matchers allow you to ask the question "is this thing like that thing", with a custom set of rules. Let me give a basic example.

Previously in Rosella if you were unit testing a method which returned an array and you wanted to check that the array contained the right values, you would have to do something like this:

var result = obj.my_method();
var expected = [1, 2, 3, 4, 5];
self.assert.is_true(result != null);
self.assert.equal(elements(result), elements(expected));
for (int i = 0; i < elements(expected); i++) {
    self.assert.equal(result[i], expected[i]);
}

That's a lot of work, although you can cut it down a little bit if you know for certain that the array isn't null. With the new matcher functionality, you pass in two arrays and the Test library will match them for you:

var result = obj.my_method();
self.assert.is_match(result, [1, 2, 3, 4, 5]);

Internally the Test library maintains a list of matchers by name. When you pass in two objects, it loops over the list looking for a matcher that can handle the pair. In this case, one of the default matchers the library provides looks for objects which implement the "array" role, and then does element-wise matching on them. Another similar matcher does the same for hash-like objects that implement the "hash" role.

Another matcher checks to see if one or both of the two objects are strings, and then does a string comparison on them (converting the other, if it isn't a string already) and the last of the default matchers is used to compare floating point values with a certain error tolerance.

Since matchers are stored in a hash, you can access them by name, delete them, add your own, and replace existing ones if you want new matching semantics. This is especially useful in something like Parrot-Linear-Algebra, where I can say

$!assert.is_match($matrix_a, $matrix_b);

…and the library will automatically compare the dimensions of the matrices and the contents of them without needing nested loops and other distractions.

Nested TAP

Another item I've had on my wishlist for a while now has been nested TAP. I've always wanted to support it, and in theory at least the system was designed modularly enough to generate it without too much hassle. Last weekend I put on the finishing touches and now am proud to say that Rosella.Test can run nested tests and generate nested TAP. At the moment the interface to use it is a little ugly (I'm actively soliciting feedback!), but the capabilities are all there:

function my_test_method()
{
    self.status.suite().subtest(class MySubtestClass);
}

function my_vector_test_method()
{
    self.status.suite().subtest_vector(
        function(var a, var b) { ... },
        [1, 2, 3, 4, 5]
    );
}

function my_list_test_method()
{
    self.status.suite().subtest_list(
        function(var test) { ... },
        function(var test) { ... },
        function(var test) { ... }
    );
}

Here's an example of output from a similar test file in the Rosella suite:

1..4
    1..2
    ok 1 - test_1A
    ok 2 - test_1B
    # You passed all 2 subtests
ok 1 - test_1
    1..3
    ok 1 - test 1
    ok 2 - test 2
    ok 3 - test 3
    # You passed all 3 subtests
ok 2 - test_2
    1..5
    ok 1 - test 1
    ok 2 - test 2
    ok 3 - test 3
    ok 4 - test 4
    ok 5 - test 5
    # You passed all 5 subtests
ok 3 - test_3
    1..1
    not ok 1 - test 1
    # failure
    # Called from 'fail' (rosella/test.winxed : 481)
    # Called from '' (t/winxed_test/Nested.t : 40)
    # Called from '' (rosella/test.winxed : 1589)
    # Looks like you failed 1 of 1 subtests run
ok 4 - test_4
# You passed all 4 tests

The fourth test expects a failure in the subtest, which is why it says it passes when there is clearly some failure diagnostics appearing. This brings me to my next point…

Cleaner Diagnostics

Before when you ran a test and had a failure, you might see something like this:

not ok 2 - ooopsie_doopsies
# objects not equal '0' != '1'
# Called from 'throw' (rosella/test.winxed : 851)
# Called from 'internal_fail' (rosella/test.winxed : 1853)
# Called from 'fail' (rosella/test.winxed : 481)
# Called from 'equal' (rosella/test.winxed : 577)
# Called from 'ooopsie_doopsies' (t/core/Error.t : 18)
# Called from 'execute_test' (rosella/test.winxed : 1455)
# Called from '__run_test' (rosella/test.winxed : 1483)
# Called from 'run' (rosella/test.winxed : 1392)
# Called from 'test' (rosella/test.winxed : 1747)
# Called from '_block1000' (t/core/Error.t : 7)
# Called from '_block1177' ( : 158)
# Called from 'eval' ( : 151)
# Called from 'evalfiles' ( : 0)
# Called from 'command_line' ( : 0)
# Called from 'main' ( : 1)
# Called from '(entry)' ( : 0)

That's a huge mess, and it's a mess from two sides. At the top of the backtrace, you see all sorts of Rosella internal functions involved in the assertion and error handling. The bottom half of the backtrace is devoted to entry-way stuff. In this case there's NQP-related entry code and then the Rosella entry code. You, as the test writer, don't care about any of that. All you care about is the code you wrote and where its broken. If you have to dig through a huge backtrace to figure out where the error is, that's a big waste of time and effort.

Now, Rosella filters that crap out for you. Here's the same exact failure with the new backtrace reporting:

not ok 2 - ooopsie_doopsies
# objects not equal '0' != '1'
# Called from 'equal' (rosella/test.winxed : 577)
# Called from 'ooopsie_doopsies' (t/core/Error.t : 18)

Here you see the important parts of the backtrace only: The parts you wrote and the one assertion that failed. You don't see the internal garbage, you don't see the entry-way garbage, because those things aren't of interest to the test writer.

Parrot-Linear-Algebra

Another small project I did a few days ago was getting the PLA test suite working again. It's a testament to how stable both BLAS and Parrot's extending interfaces are. Recent Rosella refactors removed some of the special-purpose features that existed only for PLA and for no other reason (and which were a pain in the butt to maintain). I fixed up the test suite and PLA builds and runs perfectly now.

That's what I've been up to this month. I'm mostly done with my cleanups to the Test library now, barring a few more interface improvements I want to make. After that I've got a few projects to tackle inside libparrot itself. I'll write more about those topics when I have something to say.

30 Jan 2012 8:00am GMT

At one extreme, it is possible to approach the subject on a high mathematical
epsilon-delta level, which generally results in many undergraduate students not
knowing what's going on. At the other extreme, it is possible to wave away all
the subtleties until neither the student nor the teacher knows what's going on.

-Stanley J. Farlow, Preface to Partial Differential Equations for Scientists and
Engineers

On behalf of the Parrot team, I'm proud to announce Parrot 4.0.0,
also known as "Hyperstasis". Parrot

read more

18 Jan 2012 2:40am GMT

At one extreme, it is possible to approach the subject on a high mathematical epsilon-delta level, which generally results in many undergraduate students not knowing what's going on. At the other extreme, it is possible to wave away all the subtleties until neither the student nor the teacher knows what's going on.

-Stanley J. Farlow, Preface to Partial Differential Equations for Scientists and Engineers

On behalf of the Parrot team, I'm proud to announce Parrot 4.0.0, also known as "Hyperstasis". Parrot is a virtual machine aimed at running all dynamic languages.

Parrot 4.0.0 is available on Parrot's FTP site, or by following the download instructions at http://parrot.org/download. For those who would like to develop on Parrot, or help develop Parrot itself, we recommend using Git to retrieve the source code to get the latest and best Parrot code.

Parrot 4.0.0 News:
    - Core
        + Several cleanups to the interp subsystem API
        + Cleanups and documentation additions for green threads and timers
        + Iterator PMC and family now implement the "iterator" role
        + A bug in Parrot_ext_try was fixed where it was not popping a context correctly
    - Documentation
        + Docs for all versions of Parrot ever released are now available
          at http://parrot.github.com
    - Tests
        + Timer PMC tests were converted from PASM to PIR

The SHA256 message digests for the downloadable tarballs are:

a1e0bc3de509b247b2cea4863cc202cdceeaa329729416115d3c20a162a0dd88 parrot-4.0.0.tar.bz2
a63d45f50f7dd8ba76395cd2af14108412398ac24b8d827db369221cdb37fada parrot-4.0.0.tar.gz

Many thanks to all our contributors for making this possible, and our sponsors for supporting this project. Our next scheduled release is 21 February 2012.

Enjoy!

17 Jan 2012 8:00am GMT

The Advent calendar idea ended up terribly. Let us never speak of it again. The holiday season was particularly busy this year with a higher-than-average load of family- friend- and work-related activities. Combine that with an unexpected (and absolutely unappreciated) invasion of a particularly unpleasant and long-lasting stomach bug, and you have something of a perfect storm. I won't go into the details any further on this particular blog, but I will mention in passing that I've become extremely suspicious about the basic hygene habits of the other little turdbags that my son goes to daycare with.

To start 2012 off, and to get back into the swing of coding-for-pleasure, yesterday I went through and got the new Rosella Date library ready for prime time. The library is imperfect and incomplete, but those are things that can be fixed using the patented Andrew Style (tm) of meandering iterative development. For now, however, the library does seem to work well enough for basic tasks and it's already proven to be a valuable tool for some tasks. Here I'm going to introduce the new library and talk about some of the things I changed in Rosella to support it and some of the ways I've already integrated it into the rest of the collection.

String Formatters

When you use sprintf to print out an integer (for instance) you can use some basic modifiers to control how the value is printed. %d prints out the basic version, but you can specify field width, padding, alignment and a few other details by using a format specifier like %-02d. Or, if you want to print the same value out as hex instead of base-10, you can use %x or %X, and use modifiers with those as well.

The problem with something more complex like a date/time representation is that sprintf is not able to handle them natively and some kind of mapping is needed.

Rosella has added a new StringFormatter type to the Core library to help with this problem. A StringFormatter is a type that takes an object and a format string and outputs a new string according to the two. The default StringFormatter uses sprintf internally, but other formatters may use different mechanisms and syntaxes.

var sf = new Rosella.StringFormatter();
string s = sf.format(my_obj, "This is %s");

As an aside, this does demonstrate the fact that our get_string vtable really is insufficient for a lot of purposes. I suggest that get_string should take a parameter for a format string. We could easily incorporate that into the default sprintf implementation like this:

$S0 = sprintf "%{foobar}p", $P0

In that invocation, the get_string vtable on the first parameter would be called with the string argument "foobar". A normal invocation like this:

$S0 = sprintf "%p", $P0

…would call get_string with a null format and the behavior could be whatever the default string representation for that type is. By overriding get_string in your types to take different formats or to respond to common formats differently, you could have pretty detailed control over stringification at all levels.

Date Library

The new Date library provides a Date type for working with dates and times. You can create one in three ways:

var d = new Rosella.Date(t);
var d = new Rosella.Date(year, month, day);
var d = new Rosella.Date(year, month, day, hour, minute, second);

The first uses a system-specific time integer value to represent a time since the system epoch. This is the kind of value you get from the time opcode, or from stat calls on filesystem objects, for instance. In the third option, hours are specified on a 24-hour clock.

There are also a few functions you can call to get particular dates:

var d = Rosella.Date.now();
var n = Rosella.Date.min();
var x = Rosella.Date.max();

The first value should be the current date/time (as gotten from Parrot's time opcode). The second value is a minimum date object which corresponds to the minimum possible date value to display and is guaranteed to be evaluated as less than any other date. The third one similarly is a maximum date value which corresponds to the maximum possible date and is guaranteed to always be compared greater than any other date.

Dates are immutable. Once you create them, they cannot be modified in-place. Instead, several operations are provided to perform operations and return new Date objects with the results. Here are some examples:

var d = Rosella.Date.now();
var e = d.add_seconds(20);
e = d.add_hours(15);
e = d.add_months(24);
e = d.add_years(1000);

In each case, the variable e becomes a new date value and d is left unmodified. Two other methods let you pick out just the date components or just the time components:

var n = Rosella.Date.now();
var d = n.date();
var t = n.time();

Finally, you can use a new DateFormatter type to format the date value into a proper stringification:

var d = Rosella.Date.now();
string s = d.format_string("yyyy - MM - dd and the time is: hh:mm:ss");

At the moment the formatter is dirt simple and only supports a few formatting codes such as yyyy, MM, dd, hh, mm, and ss. I will be making it much more useful in the coming days, if I can settle on an algorithm which doesn't completely stink.

FileSystem

The FileSystem library now returns Date objects from certain file-time methods:

var f = new Rosella.FileSystem.File("t/harness");
var ct = f.change_time();
var at = f.access_time();
var mt = f.modify_time();

In each case, the value returned from the stat call on the file is used to create a new Date object.

Query

Dates are completely comparible, so you can sort them and work with them like other values in an iterable:

var a = [
    Rosella.Date.now(),
    Rosella.Date.max(),
    Rosella.Date.min()
];
Rosella.Query.iterable(a)
    .sort()
    .foreach(function(d) { say(d); });

This toy example sorts the three Date values, with the min first, then now, then max, and prints them out to the console. The stringified versions of the special min/max dates aren't particularly instructive, but they do get the point across.

So that's the new Date library. It does need more functionality and definitely needs more tests, but it is working pretty well for me now and has already proven itself useful for a number of purposes. Expect to see more of it in 2012.

15 Jan 2012 8:00am GMT

We've stopped posting to this blog, but we're not done working on Parort. If you're looking for more Parrot blogging goodness, head over to parrot.org or planet.parrot.org.

11 Jan 2012 5:24am GMT

This blog is to announce the completion of placing all of Parrot's documentation on 'parrot.github.com'. The documentation ranges from the present version (i.e., v3.11.0) to Parrot's release v0.0.6(0)[1]. To view the documentation, please navigate your preferred browser to http://parrot.github.com and select (or click) the "Parrot Documentation Releases (3.10.0 - 0.1.1)" link.

Thank you.

----------

read more

11 Jan 2012 5:07am GMT

Bailador is growing better and bigger and starts to resemble a real tool more and more. Let's see what new features it has gained recently.

Remember the first example in perldoc Dancer? It's not really different in Bailador:

use Bailador;
get '/hello/:name' => sub ($name) {
    return "Why, hello there $name"
}
baile;

Aside of being a little Spanished, what did it give us? We have subroutine signatures in Perl 6 so we can pass the :name parameter to the sub; there's no need to use param() now: it's gone.

You don't need to pass everything using GET of course. post keyword is also supported.

post '/' => sub {
    return request.params.perl
}

The above will print something like ("foo" => "bar").hash, if fed with appropriate request.

any() is a reserved keyword in Perl 6, and while you can use it, it means a completely different thing. Instead of any('get', 'post') you can just do it like this:

get post '/' => sub {
    if request.is_get {
        return "I am GET"
    } else {
        return request.params.perl
    }
}

post, as well as get return their arguments, so you can chain them like in the example above. It also shows the joy of request object, which you can use to inspect the request being processed. It's not as cool as Dancer::Request, but it does the job, being quite small and simple.

What else do we have? Let's show off a bit and write a simple-simple pastebin webapp.

use Bailador;

unless 'data'.IO ~~ :d {
    mkdir 'data'
}

get '/' => sub {
    template 'index.tt'
}

post '/new_paste' => sub {
    my $t  = time;
    my $c = request.params<content>;
    unless $c {
        return "No empty pastes please";
    }
    my $fh = open "data/$t", :w;
    $fh.print: $c;
    $fh.close;
    return "New paste available at paste/$t";
}

get /paste\/(.+)/ => sub ($tag) {
    content_type 'text/plain';
    if "data/$tag".IO.f {
        return slurp "data/$tag"
    }
    status 404;
    return "Paste does not exist";
}

baile;

Holy cow, what's that! Let's go there piece by piece. First, we'll create a data directory if it doesn't already exist. No black magic here, let's proceed. What's next? Templates! Here we just load index.tt, not passing any parameters, but that works too and some example apps use that in their example templates.

The handler of new_paste uses our well-known request object again, and creates a new file for a paste, identified by the current time.

The last get block uses some nifty features, so let's take a look. It uses regexes, and you can see that they also cooperate with subroutine parameters without black magic. We then set a content_type as we'll do in Dancer, and send status 404 if no paste have been found. Easy peasy? I suppose so. That's it, it works like a charm.

Thus we've covered all the features in Bailador as for now. I don't think it's that poor, as for about 100 lines of code.

What's next? What's missing? You tell me. Or you contribute; the code is dead simple and implementing stuff like before(), after(), before_template() etc should be a matter of 3-5 lines, I think. Feel encouraged to look into the code and hack on it. If you have any questions, suggestions or criticism, don't hesitate to tell, or poke me on #perl @ Freenode. Have fun!

06 Jan 2012 12:24am GMT

Over the summer of 2011 we had a GSOC project that was trying to build a JavaScript compiler for Parrot. That project made some interesting progress but ultimately I think the approach ended up being flawed. That project attempted to convert JavaScript into PIR code, which is a pretty big gap in terms of both syntax and semantics.

Towards the end of the summer, after it was too late to go back and start from the beginning, I had a different idea: What if we tried to generate Winxed code instead of PIR code? Winxed would handle things like register allocation, and the Winxed syntax is so similar in some places that the translation could almost be verbatim copying. I put those ideas together with node.js and the cafe compiler and Jaesop was born.

Jaesop intends to be a full JavaScript on Parrot compiler. The plan is to write as much of it in JavaScript itself as possible, and bootstrap upwards. The first component, "stage 0" uses node.js to run a converter that compiles Javascript code into winxed. That's the only part we have working right now, but what we do have is working pretty well.

In 2012 I want to get the next component, "stage 1" working. Stage 1 will use the stage 0 compiler to compile itself. It will be written in JavaScript (translated to Winxed en passant) and will be running entirely on Parrot. My goal for stage 1 is to be generating bytecode directly instead of generating either Winxed or PIR as an intermediary. That's going to require some serious help from a compiler toolkit of some variety and I have very high hopes for using PACT for this purpose when it's ready.

Stage 0 works very well and passes a small but interesting test suite I've set up for it. It does not self-compile, but there are only a few relatively small things standing in the way. For instance, regular expression support and pcre bindings are not complete yet, and the grammar currently requires semicolons at the end of statements but the code generated from Jison does not always contain semicolons. I also haven't built in the require() function, which is used by the stage 0 compiler to load in the various code files. These are all small issues and with a small amount of work I expect stage 0 to be able to self-host. Whether I want to expend that effort or focus attention on stage 1 instead is a different question entirely.

In 2012, once PACT has matured and 6model has been integrated into Parrot I expect to get back to work aggressively on Jaesop. I'm looking for helpers too, in case anybody reading this wants to get involved in the development of a new JavaScript compiler. I'll put out more of a call to arms when the prerequisites are in place and we're ready to get the wheels turning on stage 1. I estimate that by spring or early summer we should be ready to get started on the next phase of Jaesop development.

29 Dec 2011 8:00am GMT

I had already written a post about Rosella for my woefully inadequate advent calendar, but it got lost to the sands of time (I think it's on my other computer, committed but not pushed). Considering that I should have been on schedule to be on post #21 or #22 by now but am instead only on #8, I can't really afford to not write a post when I have the ideas inside me. This post is also going to be about Rosella, and if I ever find the first one I may post that as well. I clearly haven't had enough spare writing time this month to be even remotely picky about what I post or when.

I'm thinking I might like to try this little experiment again later in the year, when I've had more time to prepare and have fewer other things in real life demanding my undivided attention. Maybe we'll shoot for some kind of christmas-in-July thing. Until then, I'll happily conceed the "best Advent Calendar" crown back to moritz and the Perl peoples.

Rosella is a library project I started as a way to let me work on lots of different project ideas I had without having to duplicate build and test infrastructure. The goals from the project were solidified pretty early in the project lifecycle and have remained unchanged ever since: To provide solutions to common developer problems, in pure Parrot, in ways that are portable, reusable, and configurable. I've already talked about three of the oldest and most important libraries in the set: Test, Harness and MockObject on Day 5 of this Advent calendar. My other written-but-not-published post talked about three additional libraries as well: Query, FileSystem and Proxy. To avoid much duplicate effort in case I do ever find that post again, I'll write about a few different libraries: Container, Random and Template.

The Container library is one of the oldest libraries in Rosella. It is an implementation of a dependency injection container for Parrot, and originally lived in it's own Parrot-Container repository on github. My other library, for unit testing was also a separate repo. One day I was thinking about how I might like to use the Parrot-Container project to manage dependency injection in the Harness library, and trying to figure out how I wanted to manage the software dependencies. I wanted to use the Container library in the implementation of the unit test Harness, but I wanted to use the Harness and Test libraries to implement the test suite for the Container library repository. Combine that with a bunch of other ideas for new libraries I had brewing in the back of my mind and Rosella was quickly born.

The Container library is not quite as easy to use as a counter-part in the .NET or JVM environments because things are not always strongly typed, especially not at compile time or early in runtime when the container is initializing but the various bits of initialization logic in the program have not yet run. The Rakudo folks with their fancy object system and notion of gradual typing might have a different idea, but in terms of pure-parrot code, as Parrot exists right now, we can't query a Sub PMC and ask it what types of parameters it expects to receive. For people who might be familiar with other dependency injection (or "inversion of control") systems like Unity or Ninject, the syntax for setting up Rosella's container type might seem a little verbose. It's plenty powerful (and I have plenty of plans to upgrade things as Parrot's threading support and object system improve in 2012 and beyond), but it is verbose:

var container = new Rosella.Container();
container
    .register(class Foo)
    .register(class Bar,
        new Rosella.Container.Resolver.TypeConstructor(class Bar, "Bar", 1, 2, 3)
        new Rosella.Container.LifetimeManager.Permanent()
    )
    .register(class Baz,
        new Rosella.Container.Resolver.TypeConstructor(class Baz, "BUILD"
            new Rosella.Container.Argument.Resolve(class Foo),
            new Rosella.Container.Argument.Resolve(class Bar)
        )
        new Rosella.Container.Option.Attribute("my_attr", "value"),
        new Rosella.Container.LifetimeManager.Thread()
    )
    .alias(class Baz, "Baz");
var b = container.resolve(class Bar);

The part that is significantly more verbose here than you would expect in Unity for example is the part where I have to specify the types of arguments to pass to the constructor. In a platform like .NET, the container can read the type metadata from the constructor object itself and make that decision. In Parrot, since we currently don't make that kind of information available, you must specify. In Rosella's offering you do have many of the features that you would expect from one of the more well-known containers: the ability to specify registration lifetimes, the ability to initialize objects by making method calls and setting attribute values after construction, the ability to specify global singleton instances, etc. It is a pretty cool tool, but I haven't yet made as much use of it as I would have liked. In 2012 I expect to start integrating the container library into some of the other libraries, such as Harness, CommandLine and Template, to make user configuration easier and more straight-forward.

The Random library is a relative newcomer to Rosella, but is already demonstrating its usefulness in a variety of ways. The Random library was born from a few ideas I had turned into GCI tasks. An intrepid young student wrote an implementation of the Mersenne Twister algorithm for me in Winxed, and I set about writing up several other components such as a Box-Muller transform, a UUID generator, a Fisher-Yates array shuffler and a few other things. Now, you can do cool things with random numbers:

var prng = Rosella.Random.default_uniform_random()
int r = prng.get_int();

var uuid = Rosella.Random.UUID.new_uuid();
string id = string(uuid);

var a = [1, 2, 3, 4, 5, 6, 7, 8];
Rosella.Random.shuffle_array(a);

Implementations of things aren't all perfect and there are a handful of bugs to be worked out (especially bugs resulting from arithmetic differences between 32-bit and 64-bit machines) but it is already very usable and reliable for the most part. If you need a random number generator, or a UUID generator, or other random-related things, this is a very nice tool to have available.

The Template library is something I wanted to write for a long time, but had to wait until all the prerequisites were in place first. Template is a text-templating engine library. You create an Engine object and feed it two basic pieces of information: The string template to use and the data context object to fill in the blanks. Presto-chango, out comes the complete text.

The Template engine can execute basic logical operations depending on the values in the data context, it can compile and execute inline snippets of code, it can load and assemble pieces of template from separate files, and do several other things that you would expect a templating library to do. I could write many examples of templates and their use, but I'll stick with only one small example here for brevity:

Template:

Let's learn about <%
    var bar = context["animals"];
    return elements(bar);
%> new animals!
<$ for sound in animals $>
    The <# __KEY__ #> says <# sound #>
<$ endfor $>

Code:

string template = ...;
var context = {
    "animals" : {
        "Cow" : "Moo",
        "Bear" : "Growl!",
        "Cat" : "I CAN HAZ CHEESEBURGER"
    }
};
var engine = new Rosella.Template.Engine();
string output = engine.generate(template, context);

And the end result would be something like this (minus hash ordering concerns):

Let's learn about 3 new animals!
The Cow says Moo
The Bear says Growl!
The Cat says I CAN HAZ CHEESEBURGER

Rosella eats plenty of it's own dogfood here. Several test files in the Rosella test suite and a few boiler-plate source code files are generated using the Rosella Template library. I'm planning to start using it to generate skeleton files for Rosella documentation as well. In the future, I may also use it to help with generating content for this very blog!

In 2012 Rosella is going to be adding a bunch of cool new stuff. Considering how far Rosella has come by now and the fact that it's less than a year old, it's kind of hard to speculate where we will be next year around this time. I'm planning to add a new Date/Time library within the next few weeks. I'm also planning a new reflection/packfile library, a benchmarking library, a code assertions library, and rewrites to several of the existing libraries to add new functionality and optimize performance in some key ways. Those are only my plans for the first two or three months of 2012!

28 Dec 2011 8:00am GMT

When you want to find something on the internet, Google is a pretty popular tool to use. When you want to get some code written, Google turns out to also be a pretty good idea.

Google has been doing it's Summer Of Code (GSOC) program for several years now and every summer it's a smashing success. Last year they started up with a new program called the Google Code In (GCI). That program, while wildly different from GSOC, is also incredibly awesome. Every year Parrot receives a huge amount of code from these programs, and we would not be where we are today without them.

The Summer of Code program is very straight forward: Every year we receive applications from college age and some highschool-aged students for projects. Over the course of the summer the accepted applicants work diligently and at the end, if they are successful, they get paid. Also there are cool T-shirts. The GCI program is aimed at younger students, mostly those in high school. Instead of one large project spanning several months, the GCI students work on a large number of bite-sized tasks from many organizations. Each task is worth points, and the students who have the most points at the end of the program receive a prize. Also, I think there are monetary rewards for completing a certain number of tasks.

Last year we had a huge number of tasks completed by several GCI students. We had many bits of code written or re-written, some new documentation written, and many many tests added. Our project-wide test coverage metrics increased dramatically, since we had several tasks devoted to test coverage and several talented young coders who were chasing them down. Once you figure out how to write tests, subsequent tasks go much more quickly. We had some students who, after getting the hang of things, were able to complete several tasks per day at the end.

I heard one or two accusations that Parrot was inflating point values by offering tasks that could be completed so quickly. I pointed out that many tasks were considered very hard by students at the beginning, but as their familiarity of the code increased the relative difficulty decreased. It wasn't a problem of Parrot's tasks being too easy, but the students learning and improving much faster than we could keep up with. By the end of the program we were learning that the difficulty really needed to ramp up over the course of GCI, because students would be capable of much more towards the end than they would be at the beginning.

This year things are going a little bit more slowly. We have fewer of the "increase test coverage" tasks, because our test coverage is still so high from last year in the core VM. I've scoured several potential sources of tasks including searching for TODO notes in the Parrot source code, scanning through our myriad of trac tickets, digging into ecosystem projects (Plumage and Rosella especially) and begging other contributors for task ideas. Already we've seen some very useful bits of code contributed to the project, including new features and impressive cleanups and refactorings of old crufty code.

GCI this year is essentially closed off. There were two opportunities to publish new tasks and those have both passed. Students are slowly working their way through the remaining tasks in the queue until the program ends in a few weeks. However, next year I'm sure we're going to see both another GSOC and another round of GCI (At least, I hope we see them, since these are both so awesome!).

Despite being months away we're already starting to look for project ideas and potential applicants for GSOC. Getting good ideas picked early allows us time to refine them, and getting familiar with potential applications now helps us to learn more about them and be more comfortable with them when time comes to make final selections. GCI doesn't require so much preparatory work, but it would be nice if we went into next year with a larger pile of varied tasks for students to work on, instead of needing to scramble to create tasks in sufficient numbers at the last minute.

GSOC and GCI have both been amazingly successful programs in the past and I am hoping that trend continues into the future. Parrot has benefited from both programs to an amazing degree, and with a little bit of luck and a lot of planning we can keep the train moving into 2012 and beyond.

If you're a highschool or college-aged coder, or know somebody who is, and would like to talk about getting involved with either program in 2012 (especially if you would like to work with Parrot specifically!) please let me know and I can make sure you get pointed in the right direction.

22 Dec 2011 8:00am GMT

"If I had some duct tape, I could fix that."
 - MacGyver

On behalf of the Parrot team, I'm proud to announce Parrot 3.11.0, also known as "Duct tape". Parrot is a virtual machine aimed at running all dynamic languages.

read more

20 Dec 2011 8:07pm GMT

In late 2010 and early 2011 we spent a good amount of effort building a new embedding API for Parrot. I would like to say that the new API replaced an older, inferior API but that's not really the case. We didn't really have an old embedding API per se. We had a mismash of functions in a file called embed.c, but they hardly represented a consistent API, much less a complete set of things that an embedder would need. If anything the old embedding API was the entirety of all publicly exported functions from libparrot combined with a handful of utility functions that embedders in the past also needed.

In short, it was a mess. By early 2011 we had a much nicer API around to play with. Now that 2011 is almost over, the new API is considered to be extremely stable and robust.

Last December I started a project called ParrotSharp which embeds a Parrot Interpreter into the .NET CLI with C#. I haven't shown that project too much love in recent months, but as of today it's still building and seems to run correctly (although my IDE is telling me it can't find NUnit on my system, so it won't run my tests). That has to tell you something, when code I wrote months ago with the embedding API still works correctly even though so many things have changed in Parrot since then.

Parrot's embedding API is a little bit verbose but very easy and straight forward to use. Also, all API functions return a true/false status value, so calls can easily be chained together. Here is an example of the embedding API in action:

int main(int argc, char** argv) {
    Parrot_PMC interp, bytecodepmc, args;
    Parrot_Init_Args *initargs;
    Parrot_String filename;

    GET_INIT_STRUCT(initargs);

    if (!(
        Parrot_api_make_interpreter(NULL, 0, initargs, &interp) &&
        Parrot_api_set_executable_name(interp, argv[0]) &&
        Parrot_api_pmc_wrap_string_array(interp, argc, argv, &args) &&
        Parrot_api_string_import(interp, "foo.pbc", &filename) &&
        Parrot_api_load_bytecode_file(interp, filename, &bytecodepmc) &&
        Parrot_api_run_bytecode(interp, bytecodepmc, args)
    )) {
        Parrot_String errmsg, backtrace;
        Parrot_Int exit_code, is_error;
        Parrot_PMC exception;

        Parrot_api_get_result(interp, &is_error, &exception, &exit_code, &errmsg);
        if (is_error) {
            Parrot_api_get_exception_backtrace(interp, exception, &backtrace);
            // Print out exception information to the console, or whatever
        }
    }

    Parrot_api_destroy_interpreter(interp);
    exit(exit_code);
 }

This is, essentially, a simple program to execute a bytecode file. However, it does show some of the basics of the embedding API. Every function returns a true/false, pass/fail status bit. All data types passed around are properly wrapped Parrot_PMC or Parrot_String types and it almost never uses any other raw pointer types. Also since we're using PMC and STRING types, Parrot's GC manages all the memory for you and you don't need to be freeing things or cleaning things up (except for the interpreter itself).

This example above only shows a handful of API functions, but there are several dozen of them in the API and more can be easily added. We have API routines for performing a variety of actions on Strings and PMCs. We have API routines for loading, executing and writing bytecode. The API has decent defaults so you can just get an interpreter up and running quickly if you want, but it also have a variety of routines for tweaking and configuring the interpreter too. And, like I said (and I'll say it a million times more if I have to) we can always add new methods to the embedding API if there is a need.

We have at least the basics of a C# wrapper projects in the wings, and I've been planning a proper C++ wrapper for a while too but I haven't gotten around to it yet. That would make an excellent, smallish project for an intrepid newcomer to work on, especially one that knows C++. I like to think it should be easy to embed Parrot as a plugin for things like text editors or other pluggable unixy programs, but I haven't taken the time to really dig into any of them yet. This might make another great project for an eager new parrot hacker.

17 Dec 2011 8:00am GMT

In late 2010 and early 2011 we spent a good amount of effort building a new embedding API for Parrot. I would like to say that the new API replaced an older, inferior API but that's not really the case. We didn't really have an old embedding API per se. We had a mismash of functions in a file called embed.c, but they hardly represented a consistent API, much less a complete set of things that an embedder would need. If anything the old embedding API was the entirety of all publicly exported functions from libparrot combined with a handful of utility functions that embedders in the past also needed.

In short, it was a mess. By early 2011 we had a much nicer API around to play with. Now that 2011 is almost over, the new API is considered to be extremely stable and robust.

Last December I started a project called ParrotSharp which embeds a Parrot Interpreter into the .NET CLI with C#. I haven't shown that project too much love in recent months, but as of today it's still building and seems to run correctly (although my IDE is telling me it can't find NUnit on my system, so it won't run my tests). That has to tell you something, when code I wrote months ago with the embedding API still works correctly even though so many things have changed in Parrot since then.

Parrot's embedding API is a little bit verbose but very easy and straight forward to use. Also, all API functions return a true/false status value, so calls can easily be chained together. Here is an example of the embedding API in action:

int main(int argc, char** argv) {
    Parrot_PMC interp, bytecodepmc, args;
    Parrot_Init_Args *initargs;
    Parrot_String filename;

    GET_INIT_STRUCT(initargs);

    if (!(
        Parrot_api_make_interpreter(NULL, 0, initargs, &interp) &&
        Parrot_api_set_executable_name(interp, argv[0]) &&
        Parrot_api_pmc_wrap_string_array(interp, argc, argv, &args) &&
        Parrot_api_string_import(interp, "foo.pbc", &filename) &&
        Parrot_api_load_bytecode_file(interp, filename, &bytecodepmc) &&
        Parrot_api_run_bytecode(interp, bytecodepmc, args)
    )) {
        Parrot_String errmsg, backtrace;
        Parrot_Int exit_code, is_error;
        Parrot_PMC exception;

        Parrot_api_get_result(interp, &is_error, &exception, &exit_code, &errmsg);
        if (is_error) {
            Parrot_api_get_exception_backtrace(interp, exception, &backtrace);
            // Print out exception information to the console, or whatever
        }
    }

    Parrot_api_destroy_interpreter(interp);
    exit(exit_code);
 }

This is, essentially, a simple program to execute a bytecode file. However, it does show some of the basics of the embedding API. Every function returns a true/false, pass/fail status bit. All data types passed around are properly wrapped Parrot_PMC or Parrot_String types and it almost never uses any other raw pointer types. Also since we're using PMC and STRING types, Parrot's GC manages all the memory for you and you don't need to be freeing things or cleaning things up (except for the interpreter itself).

This example above only shows a handful of API functions, but there are several dozen of them in the API and more can be easily added. We have API routines for performing a variety of actions on Strings and PMCs. We have API routines for loading, executing and writing bytecode. The API has decent defaults so you can just get an interpreter up and running quickly if you want, but it also have a variety of routines for tweaking and configuring the interpreter too. And, like I said (and I'll say it a million times more if I have to) we can always add new methods to the embedding API if there is a need.

We have at least the basics of a C# wrapper projects in the wings, and I've been planning a proper C++ wrapper for a while too but I haven't gotten around to it yet. That would make an excellent, smallish project for an intrepid newcomer to work on, especially one that knows C++. I like to think it should be easy to embed Parrot as a plugin for things like text editors or other pluggable unixy programs, but I haven't taken the time to really dig into any of them yet. This might make another great project for an eager new parrot hacker.

17 Dec 2011 8:00am GMT

Ever since I started working with Parrot I've noticed something interesting about the community: They were very interested in unit testing. Parrot alone as a suite with over ten thousand tests (and I still feel like there are some portions of the VM that are heavily under-tested). When I first joined Parrot I had never written a unit test nor did I really understand the value of testing. I was a newbie fresh out of school where such practical aspects as that were never covered. Despite my unfamiliarity with testing, I very quickly decided it was a good idea and that more tests can definitely make software more awesome.

Writing tests for Parrot and Parrot-related projects is quite easy because we have the infrastructure for it. The easiest way to write tests, in my opinion, is with Rosella, but we have a Test::More library as well that can be used for great effect and is the primary testing tool used in Parrot's own test suite.

Several months ago we had Tapir, a simple TAP harness project written by dukeleto and others in NQP. There was also a project called Kakapo written by Austin Hastings which included several unit test and mock object utilities, also written in NQP. I absorbed a lot of ideas from both projects (and eventually rewrote those ideas in Winxed) for the Rosella Test, Harness and MockObject libraries.

Writing tests for Parrot is a great way to get involved in the project if you're a new user, a great way to get familiarized with the capabilities of the VM, and a very big benefit to the project in any case. First let's talk about Test::More as it's used in the Parrot test suite and then I'm going to talk about Rosella's test offerings.

Test::More is a very simple TAP producer library that implements a few standard test functions like plan(), is(), ok(), and a few others. You use Test::More like this:

.sub main :main
    .include 'test_more.pir'
    plan(5);
    ok(1, "This test passes");
    ok(0, "This test fails");
    is(1, 1, "These things are equal");
    isnt(1, 0, "These things aren't equal, test passes");
    is(1, 0, "These things aren't equal, so the test fails");
.end

The test harness reads the TAP output from the test file, checks the plan, checks the results of each individual tests, and gives you a readout of the overall pass/fail status of the test.

Test::More is very simple, and if you've used a TAP library before the basics of it should be very easy to grasp. Plus, if you look through the Parrot test suite (t/ directory and subdirectories) you'll see plenty of examples on usage.

Rosella offers several test-related libraries as part of it's collection: Test (a unit testing library), Harness (a library for building test harnesses) and MockObject (a mock-object testing extension) are all standard parts of the Rosella lineup. There's also an experimental Assert library that adds some new testing features as well.

Rosella ships with a default testing harness called rosella_harness which is available when you install Rosella. You can run it at the commandline with a list of directories like this:

$> rosella_harness t/foo t/bar t/baz

The harness will run through all *.t files in the given directories, reading the she-bang line in the file and using that program to execute the file. This is the fastest way to get started with testing. Of course, you can also use the Harness library to build your own harness in only a few lines of winxed code:

$include "rosella/harness.winxed";
function main[main]() {
    var harness = new Rosella.Harness();
    harness
        .add_test_dirs("Automatic", "t/foo", "t/bar", "t/baz", 1:[named("recurse")])
        .setup_test_run(1:[named("sort")]);
    harness.run();
    harness.show_results();
}

The listing for a harness written in NQP is almost as short, and I've shown it several times on this blog before.

Writing a unit test file with Rosella Test is similarly easy:

$include "rosella/test.winxed";
class MyTest {
    function test_one() {
        self.assert.equal(1, 1);
    }
}
function main[main]() { Rosella.Test.test(class MyTest); }

Each method in the MyTest class will be run as a test. Each test method may contain zero or more assertions. If all assertions pass, the test passes. If any assertion fails the whole test method immediately aborts and is marked as having failed. Unlike Test::More above, we don't need to explicitly count the number of tests for plan(). Instead, the library counts the number of methods for us automatically.

Rosella's MockObject library can be used together with the Test library to add MockObject support to your tests. Mock Objects, as I've said before and I'll say a million times again in the future, are tools that can do as much harm as good, especially if they are used incorrectly. Here's an example of a test using MockObject:

$include "rosella/test.winxed";
$include "rosella/mockobject.winxed";
class MyTest {
    function test_one() {
        var c = Rosella.MockObject.default_mock_factory()
            .create_typed(class MyTargetClass);
        c.expect_method("foo")
            .once()
            .with_args(1, 2, "test")
            .will_return("foobar");
        var o = c.mock();
        var result = o.foo(1, 2, "test");
        self.assert.equal(result, "foobar");
        c.verify();
    }
}
function main[main]() { Rosella.Test.test(class MyTest); }

You have to do more setup for the test with mockobjects, but you get a lot more flexibility to do black-box testing and unit testing with proper component isolation. I won't try to sell mock object testing here in this post, only demonstrating that it is both possible and easy with Rosella.

Parrot has several thousand tests in its suite. Winxed has a small but growing test suite. Rosella currently runs "728 tests in 116 files". parrot-libgit2 has a growing test suite. Rakudo has a gigantic spectest suite. Jaesop has a growing suite of tests. NQP has tests. PACT is going to have extensive tests, once it has code worth testing. Plumage has tests (though not nearly enough!). PLA has a relatively large suite of tests. Testing is a hugely important part of the Parrot ecosystem, and we currently have several tools to help with testing. Expect the trend to continue, with more tests being written for more projects in 2012.

16 Dec 2011 8:00am GMT

In GSOC two years ago my student Chandon started working on a very cool new project for Parrot: Hybrid Threads. I hadn't really put too much thought into the future of Parrot's concurrency architecture until that point and the idea certainly seemed novel and interesting to me. We went ahead with the project not quite sure how far he would get but very eager to see something happen with our ailing threading system.

The problem, needless to say, ended up being much larger than a project for a single summer and by the end of it Chandon had most of a green threads implementation set up, but nothing that was quite mergable to master. His branch sat, un-merged and un-molested, for a year before anybody decided to take a second stab at it.

Several months after Chandon's work ended I sat down and started seriously thinking about what I wanted Parrot's concurrency system to look like in the future. If we could have anything we wanted, what is it exactly that we would want? I started by reading up on all sorts of other technologies: Erlang, node.js, stackless threads in python, and others. I even read a few tangentially related materials, like some of the problems with common string optimizations. Ideas in hand, I began writing a short series of blog posts describing my personal conclusions. My plan for concurrency was very close to Chandon's hybrid threads idea but with a few extra details filled in. I suggested that we should stick with the hybrid threads approach, explicitly restrict cross-thread data contention by using messages and read-only proxies, avoid locking as much as possible, and rely on the immutability of certain data to keep things as simple as possible.

I didn't have the time to work on all this myself, at least not until several other projects cleared my TO-DO list. This is where hacker nine comes in. Nine wanted to work on adding threading to Parrot and liked the hybrid threads approach and some of the ideas I had been working on. So he checked out a copy of Chandon's green_threads work, updated to master, and started fixing things. A few weeks later green_threads was merged to master with Linux support only. I said I would work to port the system to Windows as well and nine would push forward with the hybrid portion of the hybrid threads design.

We're not ready with that yet, but we are getting painfully close to a working system. I've been stymied by the house hunt and the fact that I don't have a windows system at home to play with. Nine has had a few infrastructural difficulties, especially with GC, but he's making great progress regardless.

Here's an overview of the threading system we're working towards in brief: We will have two layers of concurrency: The first are the Tasks, or "green threads". Each Task represents an individual unit of executing work and multiple Tasks can execute together on a single thread. They do this through preemptive multitasking, the Parrot scheduler occasionally fires alarms and switches Tasks on the current thread if there is more than one in the queue. Since Parrot uses Continuation Passing Style internally, this mechanism is relatively simple to implement (It's the alarms that are surprisingly difficult and not cross-platform, but the actual Task switching is quite simple after that).

Multiple Tasks running on a single thread gives more of an illusion of concurrency than the real thing, because you aren't making use of multiple cores in your processor hardware or exploiting any context-switching optimizations at the lowest levels of threads. What we will have to take things to the next level is the OS threads implementation. Internally Parrot will maintain a pool of worker threads. When you create a Task you will have an option about where to dispatch it: On the current thread (useful where we need safe read/write access to PMCs on the same thread without locking), on a specific target thread, or on a completely new thread. Or, if you don't want to specify you can let the scheduler dispatch the Task to the best thread.

When you think about what this kind of system enables, it's actually pretty impressive: easy asynchronous IO (schedule an IO request Task on a dedicated IO thread), easy event handling (schedule a new Task on the current thread to keep data locality), easy threading (schedule a new Task, the scheduler will set up the Thread for you), easy eventing loops (main thread reads event sources and schedules Tasks), easy library callbacks (library callback schedules a Task in the owner thread), auto-threaded array operations (schedule tasks for subsets of the data, the scheduler puts the tasks on the threads with lowest latency) and a variety of other modern techniques. This system, once it's completed and all the bugs are ironed out, will really be a big boost for Parrot in terms of feature set and usability.

I don't know when we will be ready to merge, but I can't imagine it will happen before the 4.0 release. Sometime in early 2012 expect to see the new hybrid threading implementation for Parrot, even if it only works on Linux initially. By the 4.6 release I expect we will have a pretty robust hybrid threading system available on all our target platforms, and several of our HLLs and libraries will be making use of it.

15 Dec 2011 8:00am GMT