fidzu : Perl

(If this blog had had a tagging system, this post would have been tagged "meta". You have been warned.)

My blogging engine, psyde, is written in Perl 6. It's been serving me well in the past year-and-a-quarter since use.perl went read-only and forced my hand to move out. It's not too powerful, and it's kinda idiosyncratic, but it has definitely made me happy.

Howver, I've increasingly been picturing where I want to take the engine. It could do more, and I know what bits I want it to do. But I never seem to get to the point where I sit down and tinker with the engine to make it better. You know, 'cus of tuits and yaks and worse-is-better and all that.

So, here goes. Now I'm changing that. Here's a four-quarter plan for 2012:

• Q1: A way to create/edit posts online. All my posts sit in a repository, and theoretically I should be able to post from anywhere, just as one would expect from a blogging system. But in practice, I always forget to push the latest changes, which makes my local copy on my laptop at home the athoritative copy, and so I've no choice but to post from home. Better then to author all the posts online, to store the post sources on the server, and to have those be the authoritative copy.

  Also, while writing posts in Emacs is hard to beat from a text-editing perspective, one advantage of authoring posts on the web is that I would be able to see the final HTML in real time. Better than having to wait through a compilation process to see how things will turn out.

• Q2: A commenting system. Finally. This includes converting the old comments from my use.perl blog and retroactively adding them, too. I'd like to use something like openID as an authentication system, or more generally, something ready-made that makes it easy to just post a comment.

  I've had Disqus recommended to me as a way to add a commenting system. While it would certainly work, it's slightly against the spirit of psyde, which is more or less NIH. I believe it's sometimes healthy to reinvent wheels, as much to learn how to make them round as to figure out how they can be made to work better. More importantly, I wouldn't be as happy with someone else's wheel, even a well-oiled one, as with my own.

• Q3: A text-oriented graphic library. I often desire to inject an image/graph/diagram or other in my blog posts, explaining the relationship between some things. But there are too many steps involved:

  1. open up Inkscape
  2. painstakingly draw and position a bunch of rectangles and arrows and text fields
  3. export to .png
  4. upload the .png
  5. insert an <img> tag linking to where I uploaded the .png

  ...and besides, I always feel like I'm doing work that the computer should be doing for me, drawing the SVG. I've drawn graphs like that a hundred times before, and they all look the same.

  Imagine if I could just write a few symbols in my post saying "oh boy, here comes le graph!", a bit like \[ takes me into the displaymath environment in LaTeX. And, and! Because the groundwork I will have done in Q1 enables me to see things as I type in real time, and because the graphic library that I haven't written yet will be mostly implemented in JavaScript, those graphs will appear as I type them, too! Instant graphic!

  I haven't seen any other blog system do this. Let me know if there's prior art.

  The graphic library will support not just graphs, but any number of pre-set picture drawing modes: graphs, class diagrams, chess boards, hex boards, tables, Nim positions, pentomino pieces, sudoku boards, sparklines, basically anything. Adding another mode should be as simple (for a programmer) as adding a syntax parser and a generator of iamge elements. There will have to be an API for all of this; details pending.

• Q4: Productize psyde. After all of this, I actually think I'll have something worth sharing with people. Not everyone will prefer to leave the cozy confines of Wordpress or Movable Type or whatever it is you kids use these days, but some people will like the combination of features in psyde, and will want to set up their own blog with it.

  (This already happened once, by the way. I'm not at all prepared for this, so I just sent them a .tar.gz file with the minimal environment needed to get set up, and they succeeded. It was a slightly bumpy ride for them, of course, but they got it working.)

  Now psyde isn't really a blogging system at its core, but a static web page generator. Making a product out of it probably means turning a bunch of things into modules, and have them play nicely with the Perl 6 ecosystem. And to make setup a one-click operation, or as close to it as possible. Probably psyde the static web generator and the yet-unnamed blogging software will separate into a module-client relation as a part of this.

  When I say "product", by they way, I don't imagine I'll charge money for it. The source will remain free in all senses. I just want to make it easier to approach the software and get started with it. Removing various hard-coded things, as well as adding documentation and installation instructions, will be necessary.

Early on, I told myself that this would be a ten-year blog. We're now a few months into the second year. It's time to introduce some furniture and make this blog engine fit to live in. Fit to relax in after a hard day's work.

I want to be able to blog from my iPad. Or from a 40-inch TV screen. Or, in a pinch, from my Android phone. I want people to be able to comment. I want those graphs, oh nicely laid out, skinnable, auto-generated instant-gratification graphs! And I want to push psyde out into the wild, make it stand on its own legs.

I want to use Perl 6 more in production, because the time is here. Even though I know it's prudent to underpromise and overdeliver, I choose to publish my plans like this because so far I've only been scheming and dreaming - this somehow makes it all more concrete.

Onwards!

05 Feb 2012 3:15pm GMT

(This is a guest post by Moritz Lenz. If you're wondering what this is all about, it's the aftermath of The 2011 Perl 6 Coding Contest. If you're not wondering, it's still about that.)

Let's consider the first task from the Perl 6 Coding Contest 2011.

Here is the description of the task once more:

What non-negative integers can you write as expressions containing exactly
four occurrences the number 9, and any of the binary operators *, /, %, +,
-, prefix negations, and any number of matching pairs of parentheses you
care to use?

The program should accept an upper limit N as a command-line argument.
It should then print all integers 0..N in increasing order, along with
an expression with four nines, if any such was found.

It was probably the easiest of all tasks, and the one we got the most submissions for. Yet there were still some things that could go wrong, and some submissions got some of them wrong:

• non-negative implies that we start at 0 = 9 + 9 - 9 - 9, not at 1
• integers means that the result must be an integer; it does not mean that intermediate results are automatically rounded or truncated
• the expressions must consist of four times the number 9, not the digit nine. Thus 99 is not a valid expression of two nines.

All contestants solved this task with the following strategy: first find all possible expressions involving four nines (and possibly filter out those out of range), then iterate the numbers from 0 to N and check for each if an expression was found.

One possible approach to generate all expresions is to hard-code all expressions with one nine, namely 9 and -9. Then apply all operators to all combinations of those, and thus generate all expressions with two nines. This is a good time for filtering out duplicates, for example 18 == 9 + 9 == 9 - (-9). Then expressions of greater length can be generated from the shorter ones in the same manner.

There are just two small pitfalls: the first is that one has to remember that not all expressions can be generated by combining expressions of length (N-1) and 1. For example 324 == (9 + 9) * (9 + 9) can only be written as the combination of two expressions with two nines each. The second small pitfall is that three of the operators (%, / and -) are not commutative, so one has to explicitly try both combinations of $a op $b and $b op $a, or take care of that in some other way.

The task description allowed the prefix minus operator, but including it into the production does not produce any new numbers; it is sufficent to seed the expressions of length one with (9, -9).

As usual, we've published each contestant's code along with a review; feel free to have a look. This year, we're also publish solutions from people who sent in solutions but didn't sign up for the contest.

Isn't it noteworthy how, even with this narrow a task, people's solutions are all over the board in terms of the constructs used, code size, and style? We think so.

Stand by for the review on t2: "Sums of cubes".

03 Feb 2012 10:43pm GMT

Announce: Niecza Perl 6 v14

This is the fourteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. This month hasn't seen much of a
focus.

With this release, Niecza now passes a similar number of spectests as
Rakudo. (Note that they are not the same tests, and as such the test counts
are not completely comparable.) Solomon Foster has begun efforts to port
panda to niecza, with the goal of designing a multi-implementation ecosystem.

You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.


List of changes


[Breaking changes]

Multiple dispatch has been changed to reject all cases of ambiguity using
an explicit list of conflictors; some ambiguous cases were formerly accepted.

List iteration has been modified to throw an exception when the list generator
tries to access the unreified part of the list. The previous code exhibited
undefined behavior that some code actually relied on.

Parameters like @foo and %foo now insist that their arguments be Positional and
Associative respectively.

Rat and Complex stringification has been substantially changed.

Niecza now enforces "trusts".

&nextwith and CallFrame.args now conspire to hide the invocant parameter. If
you are calling nextwith directly, you no longer need to - and must no longer -
pass a self argument.

|$foo capture parameters now capture the logical "current" capture, rather
than the "initial" capture. In particular, 'method (|$foo)' no longer results
in $foo containing self.


[Major features]

END blocks are now suppported.

The subroutine entry process has been significantly modified to allow
signatures to reference variables and anonymous code blocks.

Roles have been overhauled to much more closely match the Rakudo nom behavior.
Importantly, roles now have a $?CLASS parameter, and role blocks are not run
until that is available. Role composition at compile time is now supported,
as is role summation, attributes in roles, role conflict detection, type
checking against roles, etc. &infix:<does> and &infix:<but> now support
much more of specced behaviors.

Accompanying that, the old tag and mixin classes CommonEnum, IntBasedEnum,
StrBasedEnum, Callable, Positional, Associative, Numeric, and Real have been
converted into roles.

Major signature improvements: Added support for sub-signatures, where blocks,
proper MMD with subtypes, values used as types. Parameter and Signature
objects are now reified into Perl 6 space and support a subset of the Rakudo
nom introspection API.

Niecza now supports constant folding! If you mark a sub 'is pure' and call
it with sufficiently constant arguments, it will be replaced at compile time
with the result of the call, provided said call does not throw an exception.
What constitutes "sufficiently constant" is not documented, poorly defined,
and subject to change.

Niecza now keeps attributes from different classes in different namespaces,
so you can have $!x in both a parent and a child class without issues.
Additionally, the sigil is part of the name, so you can now have both $!x
and @!x.

val() is now supported, and is used automatically on <> lists.

MAIN is now supported.

STD-imported syntax changes: \foo and |foo parameters no longer require a
sigil. my \foo = ... works to declare a "raw" variable. Initializer
assignment now binds tightly to the declarator, so that e.g. (5 + my $x = 3)
does something more useful.

.WHICH has been added, and === does the correct thing with value types now.

Perl 5 interop improvements: can now be used from any directory and builds
much more robustly. Also supports more pass and return cases. (Paweł Murias)


[Minor features]

No more pseudo-evaluators - all constructs which contain code that logically
is run at BEGIN time to produce a value, is now actually run rather than
attempting a static evaluation.

Binding to attributes is now supported.

Class attribute forms such as my $.foo, and the aliasing form has $foo, are
now supported.

Within a named 'anon sub', the name is visible, allowing for nicer recursion.

* now properly ignores assignments.

$obj.Foo::bar, $obj.::("Foo::bar"), and /<::("Foo::bar")>/ are now accepted.

You can now set 'is iffy' and 'is diffy' for fully custom operators.

Type adverbs :_ :U :D :T are now accepted.

Signatures like (Int, Int) are now allowed.

Phaser handling has been improved, and support added for CHECK.

Aliasing forms like <foo=$bar> are now allowed.

Version information is now embedded into the build, allowing for --version
and $?PERL support.

Printing of unhandled exceptions uses .gist.

Multiple dispatch now handles junctions.

Exporting multisubs from modules now approximately works.

Added ".Bridge" support. (Solomon Foster)

Attribute-binding parameters (:$!foo) now implemented.

:16() syntax is now supported.

Defaults and type constraints that are constants are now saved as such,
avoiding an unneeded block.

Str.perl now escapes special characters.

CLR interop now supports calls to shadowed and hidden methods, like
$obj.CLR::System::IDisposable.Dispose(). Note that this can NOT be used to
call overriden methods (callvirt semantics are used).

LTM processing ignores arguments and dispatchers forward the arguments to
multi regexes.

infix:<cmp> supports pairs, ±Inf (Solomon Foster)

Hash.perl sorts the output to be slightly more useful.

New setting things:
Cool.polar, Cool.roots, &roots, Array.delete, &infix:<minmax>, &rotate,
CommonEnum.pick, CommonEnum.roll, Any.min, Any.max, Any.minmax, Str.trans,
&elems, Any.reduce, &reduce, &shell, &categorize, &cwd, &chdir, $*CWD,
&printf, IO.copy, IO.chmod
(Solomon Foster)

Hash.push (Moritz Lenz)

Int.base, Array.splice, &splice (Will Coleda)

Set, Bag (Larry Wall, Solomon Foster)

&prefix:<sleep>, Mu.clone, &undefine, $*OS, Order, $*OUT, $*ERR

split and comb now support limits of Whatever (Solomon Foster)

Range coerces values to numbers according to spec (Solomon Foster)

&sort accepts a Callable first argument (Solomon Foster)

Added limit to &lines, :r and :rw to &open (Solomon Foster)

min and max support arbitrarily many arguments (Solomon Foster)

Changed sleep to return a value (Will Coleda)



[Other]

Daniel Ruoso is attempting an implementation of concurrent feeds.


Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

Next month is likely to see fewer tuits in general. The only concrete plan
I have is to continue with 6model convergence, hopefully reaching a point
where user-defined metaclasses are possible.


[1] https://github.com/downloads/sorear/niecza/niecza-14.zip
[2] https://github.com/sorear/niecza

30 Jan 2012 9:00pm GMT

I'm here to report that the macros grant is coming along nicely. I've been busier with $dayjob than anticipated, and so the schedule in the grant application is slipping a bit. But I have a fairly good view of the obstacles and to-do items ahead.

When we last saw each other, I was saying that "variable lookups from inside of the quasiquote end up confused". That's still true, and it's the big thing I want to fix before merging my work so far into the master branch, er, into the nom branch.

What I've done since last time:

• I wrote a thorough explanation of why ASTs need to carry around their original lexical environment. (Short explanation: they need to act like closures, but they aren't really so they need to fake it.)

• I re-based the macros branch on the latest nom, and fixed a bunch of regressions in my code that fell out of that.

• I started a test file with macro tests that are more adapted to the work that I'm doing than the tests that are already there. I hope to be able to absorb the ones that are there as we go along; right now they're not much use to me.

Now I'm well poised to go in and actually implement the lexical fixup I need for the ASTs to behave as if they're normal, honest-to-goblin closures. I just thought I'd blog this report in case I end up walking into the source code, never to return. :-)

The trick is this: .SET_BLOCK_OUTER_CTX. It lets you say "block, your OUTER is now this thingy". It's the kind of internal fixup that makes the cat walk by twice inside the Matrix.

Ok, I'm going in. See you on the other side.

29 Jan 2012 6:26pm GMT

Ding! Pencils down.

So, here we are, five weeks later. Let's sum up:

• 35 people signed up as contestants this year.
• 6 contestants submitted solutions.
• 27 submissions passed a base-test run and were accepted.

Last year's figures for the same things are (18, 5, 26), so somehow we ended up with many more contestants and only one more submission this year. That's fine; we know that the step from signing up to actually sending something in is quite a big one to take. Very interested to get feedback on how to make that step easier, though.

Just as last year, we'll synchronously publish people's solutions for each task in sequence, along with a blog post about the task and its solutions.

Unlike last year, I won't promise that these posts will be forthcoming in "the next few days/weeks". Last year it took over two months to process everything and get to the winner. Let's aim for one month this time - we're as expectant as you are to see solutions, blog post explanations, and an eventual winner!

Let's do this! First up: "Four 9s."

29 Jan 2012 12:04pm GMT

I've got my fork of Panda working on Niecza. There are two major issues which remain to be resolved, and they are both related. Panda assumes that the Perl 6 executable is named perl6, and modules should be installed to ~/.perl6. That's great if you have just one Perl 6 on your system.

But in this crazy modern world, I really want both Rakudo and Niecza on my system. And so assuming that everything can be named "perl6″ is a big problem.

I'm hoping the community can put their heads together and figure out a solution. I guess the obvious one (which would only need sorear to buy in) would be installing Niecza modules to ~/.niecza, and creating a "real" niecza executable. (It might well be as simple as a one-line script "mono path-to-Niecza.exe".)

Good idea or bad idea?

29 Jan 2012 3:09am GMT

So, we made it - a Rakudo Star release based on the "nom" development branch has landed. It's based on the compiler release moritz++ cut earlier this week, and pmichaud++, masak++ and myself have been involved in getting the Star-specific build and installation scripts in shape.

Getting to this release has been a lot of work; in many sense this is a revolution in Rakudo's development rather than a small evolutionary step over what we had before. It's taken a while (indeed, longer than I'd first hoped) and has been a lot of work - but it was worth it. Not just because of the many improvements that are in this release, but because of the enormous future potential that we now have.

Here's some of the things I'm happiest about in the release.

• The performance improvements in many areas. Yes, we've plenty of work to do here - but this is a solid step forward for a wide range of scripts, and in some cases an order of magnitude improvement.
• That 6model - something I started designing a year and a half ago - has not only cleanly supported all of the things we needed to do in Rakudo, but also opened up so many other doors. For example, the new NativeCall module uses its representation polymorphism support to great effect.
• Protoregexes doing real NFA-driven Longest Token Matching rather than the cheating version we had before that only operated on literals.
• The optimizer, along with the various extra compile time error reporting it gives. This will be an important future area for Rakudo.
• Initial native type support, and bigint semantics for the Int type.
• The POD6 support, thanks to tadzik++'s Google Summer of Code grant in summer.

So, what's next? Currently I'm working hard on getting true bounded serialization support in place. This should further improve BEGIN time support (including constructs that depend on BEGIN time), greatly cut down resource consumption during CORE.setting compilation (both time and memory) and give us faster startup. It's hard to guess at figures for the improvement, but I'm expecting it to be a noticeable improvement in all of these areas. I'm aiming at getting this landed for the next Rakudo compiler release (which I expect us to do a Star release based on too), though largely it depends on whether I can get it working and stable enough in time; while some parts are a simple matter or programming, other parts are tricky.

That aside, we've already got various other new features in the pipeline; even since last weekend's compiler release, there are multiple new regex-related things in place, moritz++ has continued with his typed exceptions work, we're catching a couple more errors at compile time rather than letting them slip through until runtime, and there's various other miscellaneous bug fixes. Also, masak++ is working on macros in a branch, and I'm optimistic that we'll have some initial macro support in place by the next release also. Busy times! :-)

29 Jan 2012 12:41am GMT

On behalf of the Rakudo and Perl 6 development teams, I'm happy to announce the January 2012 release of "Rakudo Star", a useful and usable distribution of Perl 6. The tarball for the January 2012 release is available from http://github.com/rakudo/star/downloads.

In the Perl 6 world, we make a distinction between the language ("Perl 6″) and specific implementations of the language such as "Rakudo Perl". This Star release includes release #48 of the Rakudo Perl 6 compiler [1], version 3.11 of the Parrot Virtual Machine [2], and various modules, documentation, and other resources collected from the Perl 6 community.

Significantly, this is the first distribution release based on the "nom" (New Object Model) development branch of Rakudo. This work has been carried out with the aim of increasing performance and correctness, as well as providing a better base for taking on a
range of missing features. Here are some of the major improvements in this release over the previous distribution release.

• Greatly improved performance in many areas. For some scripts, this release offers an order of magnitude performance improvement.
• POD6 support, including the $=POD variable to make the POD available at runtime and a -doc option to get at the POD
• The Int type now has big integer semantics
• Initial work on native types, which can be used to write much more efficient code
• LTM-driven protoregexes
• Meta-programming support, including custom meta-classes, overriding method dispatch and much more
• Exception handling is much closer to the specification, and thus much more useful
• Better package handling, including true separate compilation, lexical packages and better nested package handling
• An optimizer, which improves generated code as well as catching a range of issues at compile-time that previously made it to runtime
• Backslash sequences in character classes
• Stubbed methods from roles are now required, providing interface style functionality
• Typed arrays and hashes, as well as supporting for binding to array and hash elements

Due to improvements in the Perl 6 language specification, and changes to Rakudo to track them, some existing code will need changes. Here are some of the major differences to be aware of.

• Attributes can no longer be initialized using "new" unless they are declared as having an accessor; either give them one or write a BUILD submethod
• The proto keyword is no longer used to declare a multi-dispatch fallback
• You may no longer do 'filename'.lines; use 'filename'.IO.lines

We have maintained backwards compatibility with some changed pieces of syntax, but will drop them in an upcoming release:

• "<…>" in proto regex bodies; now this should be written "*"
• The use of "**" with a separator in regexes; this is now done by using "%" or "%%" on another quantifier

While this release does contain a great number of improvements, unfortunately we have regressed in a few places. Of note:

• Some cases of auto-vivification do not work
• The binding of a capture against a signature literal is broken

We will be working to restore this functionality for future Rakudo Star releases; if you depend heavily on it, you may wish to stick with the previous Rakudo Star release for
another month.

There are some key features of Perl 6 that Rakudo Star does not yet handle appropriately, although they will appear in upcoming releases. Some of the not-quite-there features include:

• pack and unpack
• macros
• threads and concurrency
• Unicode strings at levels other than codepoints
• pre and post constraints, and some other phasers
• interactive readline that understands Unicode
• non-blocking I/O
• much of Synopsis 9

There is a new online resource at http://perl6.org/compilers/features that lists the known implemented and missing features of Rakudo Star 2012.01 and other Perl 6 implementations.

In many places we've tried to make Rakudo smart enough to inform the programmer that a given feature isn't implemented, but there are many that we've missed. Bug reports about missing and broken features are welcomed at <rakudobug@perl.org>.

See http://perl6.org/ for links to much more information about Perl 6, including documentation, example code, tutorials, reference materials, specification documents, and other supporting resources. An updated draft of a Perl 6 book is available as <docs/UsingPerl6-draft.pdf> in the release tarball.

The development team thanks all of the contributors and sponsors for making Rakudo Star possible. If you would like to contribute, see <http://rakudo.org/how-to-help>, ask on the perl6-compiler@perl.org mailing list, or join us on IRC #perl6 on freenode.

[1] http://github.com/rakudo/rakudo
[2] http://parrot.org/

28 Jan 2012 2:34pm GMT

So, 2012 is here, and here's my first Perl 6 post of the year. Welcome! :-)

Looking Back

2011 brought us a faster Rakudo with vastly improved meta-programming capabilities, the first work on exploring native types in Perl 6, the start of a powerful type-driven optimizer and many other bits. It also took me to various conferences and workshops, which I greatly enjoyed. I'd like to take a moment to thank everyone involved in all of this!

This was all great, but slightly tainted by not managing to get a Rakudo Star distribution release out based on a compiler release with all of these improvements. I'd really hoped to get one out in December. So what happened? Simply, there were a certain set of things I wanted to get in place, and while many of them got done, they didn't all happen. While the compiler releases are time based - we do one every month - the distribution releases are more about stability and continuity. By the time I headed back to the UK to spend Christmas with family, we were still missing a some things I really wanted before a Star release was done. Given the first thing that happened when I started relaxing a little was that I immediately got unwell, I figured I should actually use my break as, well, a break - and come back recharged. So, I did that.

So, let's try again

So, the new goal is this month. I'm happy to report that in the week since I've got back to things, one of the things I really wanted to sort out is now done: Zavolaj, the native calling library, now does everything the pre-6model version of it did. In fact, it does a heck of a lot more. It's even documented now! It's also far cleaner; the original implementation was built in the space of a couple of days with mberends++ while I was moving apartment, and was decidedly hacky in places. The missing bits of the NativeCall library were important because they are depended on by MiniDBI, and I really didn't want to ship a Rakudo Star that can't connect to a database. So, next up is to make sure that is in working order. I'm not expecting that to be difficult.

That aside, there were some things to worry about in Rakudo itself. I've dealt with some of those things in the last week, and perhaps the one important remaining thing I want to deal with before Star is a nasty regex engine backtracking related bug (I've been hoping pmichaud++, the regex engine guru, might appear and magic it away, but it seems it's going to fall on my plate). But overall, we're well on track to cut the Star release this month.

What's the direction for the year ahead?

During 2011, Rakudo underwent a significant overhaul. It was somewhat painful, at times decidedly not much fun, but ultimately has been very much worth it: many long standing issues have been put to rest, performance has been improved and many things that were once hard to do are now relatively easy or at least accessible.

I think it goes without saying that we won't be doing any such wide-ranging overhaul in 2012. :-) The work in 2011 has opened many doors that we have yet to walk through, and 2012 will see us doing that.

At a high level, here's what I want:

• Less bugs, more stability: mostly this is about continuing to work through the ticket queue and fix things, adding tests as bugs are fixed to ensure they stay fixed.
• Better error reporting: there are things in STD, the Perl 6 standard grammar, that allow it to give much more informative error reports on syntax errors than we often can in Rakudo today. I want us to bring these things into Rakudo. Additionally, there's plenty of improvements to be made in runtime errors. Furthermore, I want to expand the various bits of static analysis that I have started doing in the optimizer to catch a much wider range of errors at compile time.
• Run programs faster: this process is helped by having decent profiling support these days. There's a lot more to be done here; the optimizer will help, as will code generation improvements.
• Compile programs faster: this will come from more efficient parsing and greatly improving the quality of the code NQP generates (NQP is the language we write much of the compiler in)
• Shorter startup time: this mostly involves finishing the bounded serialization work up. I think the best way to describe this stuff is "fiddly".
• Use less memory: Rakudo has got faster in no small part thanks to being able to understand its performance through profiling. Our understanding of its memory consumption is much more limited, which makes it harder to target improvements. That said, I've some good guesses, and some ideas for analyzing the situation.
• More features: while being able to do the things Rakudo can do today faster and with less bugs would give a very usable language for quite a lot of tasks, there's still various features to come. In particular, Rakudo's support for S05 and S09 needs work.
• VM Portability: we've made good progress towards being able to make a serious stab at this over the last year, while at the same time also managing to perform vastly better on Parrot too. With help from kshannon++, I'm currently working on completing the switch to QRegex (a regex engine with NFA-powered LTM, and written in NQP rather than PIR), which should carry on a pattern of simultaneously increasing performance and portability. Beyond that will be an overhaul of our AST and code generation, with the same goal in mind.

So, lot's of exciting things coming up, and I look forward to blogging about it here. :-)

A way to help

There are many ways to get involved, depending on what you're interested in. One way is to take a look at our fixed bugs that need tests writing. At the time of writing, just short of 100 tickets are in this state. No guts-y knowledge needed for this - you just need to understand enough Perl 6 (or be willing to learn enough) to know what the ticket is about and how to write a test for it. Drop by #perl6 for hints. :-)

15 Jan 2012 10:28pm GMT

So, last spring I did a series of posts on making a Pi spigot. Unfortunately, the project foundered a bit, as it turned out that only Rakudo had the subsignatures needed to make the code pretty, but only Niecza had the big integers needed to make the code work.

Fast forward to today. Now both Rakudo and Niecza can handle the best version of the code with no problem!

Let me emphasize that again. A year ago, neither implementation had those two features. Eight months ago, each had one. Today both have both. What's more, Rakudo seems to have made some pretty impressive speed improvements.

I think the awkward situation with Rakudo Star has helped obscure the fantastic good news in Perl 6. Right now we have two distinct Perl 6 implementations, each of which is markedly better than anything available a year ago. While there are still some rough edges, both implementations are making visible progress every single day.

Okay, enough cheerleading. Time to finish the Pi story. It turned out there was one last complication. I had never actually tried to use the code to generate an infinite stream of Pi. I always stopped at 40 digits. Confident that the algorithm must work, I tweaked the output to just continue generating digits until you stopped the program with control-C. And I ran it using Niecza, and it printed

3.14159265358979323846264338327950288419716
9399375105820974944592307816406286208998628
0348253421170679821480865

And then it just sat there, calculating. After a few minutes I stopped it and tried it in Rakudo, and got the exact same result.

Well, after adding a few debugging says here and there, I found the problem. The extr sub was returning NaN. It turns out that both compilers have issues with dividing huge numbers. Here's the division operation that was causing the problem:

826855066209083067690330954944954674053
707782399091459328155002954168455127712
564546723209828068849110223672691692080
858850302237001093531862737473606364113
314687502675869281622802970765988449203
963736097729699655628829895255493809983
868753943269929165690008254816168624365
041070395716948346309925280258763697273
816643106559428329680316113883598846477
019844021876290510680558354153412094804
165563855909020631086890050609449881578
622437959410200560054513816596644762131
226627968813825552929967132893776980417
525678140579476414867767644626389410380
794467097761379794479928269796859019439
705966555011741254554959832606241504043
482378842096776403191455346497512084739
323724281071973237937801014210278895804
940475966938880398182275335278425442994
287812050560074302564177393567873480740
249095636709741437469651121924884638352
523975466249955052660310789169884060356
70777782797813415527343750 / 7640045443
915776552858245682965495041201868477244
923723289802372673948570989301189643881
881673616027696317656701576457227272117
067947294675092324411286583110995372015
785893970194452345100095389207557064515
618905243737091067059065039684867000766
465399984513882758095027633673968549038
659642193965599458094646356792444696562
299054844575814305259223023977803302735
242307789179027935107449661143998428584
590618630170775872761731454567203230484
311106708425828778192240791257477924515
937573923664112071637127786446287936043
637833776529791999568414593746973068229
015816020732598109749879566833692821582
816119454436978125677364775318707235283
939392855143663977524974469973442065855
128922452382372338686111634084367257050
255499210918328304009454606504169283500
652033488755998721320134300149134205253
095344802815192912405314659633341062491
389270940370884860337596933277382049709
5584869384765625

Turns out the bottom number is bigger than can be presented by a Num, so the division operation ends up becoming N / Inf, which is NaN. This is a bit obscured in Rakudo because the division operation actually returns a Rat (which should be illegal according to the spec!), but then .floor is called on the result, which tries to convert to Num before doing the floor operation.

This opens several questions, like: Should Perl 6′s Int / Int operator be modified to try to cope with this sort of case?

But for the Pi problem, the good news is this: every time the extr sub is called, its result is fed to .floor. That means we simply needed to replace / with div and get rid of the .floors.

And with that, the code easily produces 2,000+ digits of Pi using either Rakudo or Niecza!

15 Jan 2012 5:43pm GMT

Bailador is growing better and bigger and starts to resemble a real tool more and more. Let's see what new features it has gained recently.

Remember the first example in perldoc Dancer? It's not really different in Bailador:

use Bailador;
get '/hello/:name' => sub ($name) {
    return "Why, hello there $name"
}
baile;

Aside of being a little Spanished, what did it give us? We have subroutine signatures in Perl 6 so we can pass the :name parameter to the sub; there's no need to use param() now: it's gone.

You don't need to pass everything using GET of course. post keyword is also supported.

post '/' => sub {
    return request.params.perl
}

The above will print something like ("foo" => "bar").hash, if fed with appropriate request.

any() is a reserved keyword in Perl 6, and while you can use it, it means a completely different thing. Instead of any('get', 'post') you can just do it like this:

get post '/' => sub {
    if request.is_get {
        return "I am GET"
    } else {
        return request.params.perl
    }
}

post, as well as get return their arguments, so you can chain them like in the example above. It also shows the joy of request object, which you can use to inspect the request being processed. It's not as cool as Dancer::Request, but it does the job, being quite small and simple.

What else do we have? Let's show off a bit and write a simple-simple pastebin webapp.

use Bailador;

unless 'data'.IO ~~ :d {
    mkdir 'data'
}

get '/' => sub {
    template 'index.tt'
}

post '/new_paste' => sub {
    my $t  = time;
    my $c = request.params<content>;
    unless $c {
        return "No empty pastes please";
    }
    my $fh = open "data/$t", :w;
    $fh.print: $c;
    $fh.close;
    return "New paste available at paste/$t";
}

get /paste\/(.+)/ => sub ($tag) {
    content_type 'text/plain';
    if "data/$tag".IO.f {
        return slurp "data/$tag"
    }
    status 404;
    return "Paste does not exist";
}

baile;

Holy cow, what's that! Let's go there piece by piece. First, we'll create a data directory if it doesn't already exist. No black magic here, let's proceed. What's next? Templates! Here we just load index.tt, not passing any parameters, but that works too and some example apps use that in their example templates.

The handler of new_paste uses our well-known request object again, and creates a new file for a paste, identified by the current time.

The last get block uses some nifty features, so let's take a look. It uses regexes, and you can see that they also cooperate with subroutine parameters without black magic. We then set a content_type as we'll do in Dancer, and send status 404 if no paste have been found. Easy peasy? I suppose so. That's it, it works like a charm.

Thus we've covered all the features in Bailador as for now. I don't think it's that poor, as for about 100 lines of code.

What's next? What's missing? You tell me. Or you contribute; the code is dead simple and implementing stuff like before(), after(), before_template() etc should be a matter of 3-5 lines, I think. Feel encouraged to look into the code and hack on it. If you have any questions, suggestions or criticism, don't hesitate to tell, or poke me on #perl @ Freenode. Have fun!

06 Jan 2012 12:24am GMT

The change of year is a good occasion to look back. Here I want to reflect on the development of Perl 6, its compilers and ecosystem.

At the start of the year, masak's Perl 6 Coding Contest continued from 2010, concluding in the announcement of the winner. I must admit that I still haven't read all the books I won :-)

Specification

2011 was a rather quiet year in terms of spec changes; they were a mixture of responses to compiler writer and user feedback, and some simplifications and cleanups.

Positional parameters used to be allowed to be called by name; this feature is now gone. That both makes the signature binder simpler, and removes accidental dependencies on names that weren't meant to be public. Read the full justification for more background.

A small change that illustrates the cleanup of old, p5-inherited features was the change that made &eval stop catching exceptions. There is really no good reason for it to catch them, except Perl 5 legacy.

say now uses a different stringification than print. The reasoning is that print is aimed at computer-readable output, whereas say is often used for debugging. As an example, undefined values stringify to the empty string (and produce a warning), whereas say calls the .gist method on the object to be said, which produces the type name on undefined values.

An area that has been greatly solidified due to implementation progress is Plain Old Documentation or Pod. Tadeusz Sośnierz' Google Summer of Code project ironed out many wrinkles and inconsistencies, and changed my perception of this part of the spec from "speculative" to "under development".

Rakudo

Rakudo underwent a huge refactoring this year; it is now bootstrapped by a new compiler called "nqp", and uses a new object model (nom).

It allows us to gain speed and memory advantages from gradual typing; for example the mandelbrot fractral generator used to take 18 minutes to run on a machine of mine, and now takes less than 40 seconds. Speedups in other areas are not as big, but there is still much room for improvement in the optimizer.

With the nom branch came support for different object representations. It makes it possible to store object attributes in simple C-like structs, which in turn makes it much easier and more convenient to interoperate with C libraries.

Tadeusz' work on Pod gave Rakudo support for converting Pod to plain text and HTML, and attach documentation objects to routines and other objects.

Rakudo now also has lazy lists, much better role handling, typed exceptions for a few errors, the -n and -p command line options, support for big integers, NFA-based support for proto regexes and improvements to many built-in functions, methods and operators.

Niecza

It is hard to accurately summarize the development of Niecza in a few sentences; instead of listing the many, many new features I should give an impression on how it feels and felt for the user.

At the start of 2011, programming in niecza was a real adventure. Running some random piece of Perl 6 code that worked with Rakudo rarely worked, most of the time it hit a missing built-in, feature or bug.

Now it often just works, and usually much faster than in Rakudo. There are still some missing features, but Stefan O'Rear and his fellow contributors work tirelessly on catching up to Rakudo, and it some areas Niecza is clearly ahead (for example Unicode support in regexes, and longest-token matching).

Since Niecza is implemented on top of the Common Language Runtime (CLR) (which means .NET or mono), it makes it easy to use existing CLR-based libraries. Examples include an interactive fractal generator and a small Tetris game in Perl 6.

Perlito

Perlito aims to be a minimal compiler with multiple backends, which can be used for embedding and experimenting with Perl 6. It had several releases in 2011, and has interesting features like a Javascript backend.

Ecosystem

The presence of two usable compilers (and in the case of Rakudo, two viable but very different branches) has led to many questions about the different compilers. The new Perl 6 Compiler Feature matrix tries to answer the questions about the state of the implemented features in the compilers.

With Panda we now have a module installer that actually works with Rakudo. It still has some lengths to go in terms of stability and feature completeness, but it is fun to work with.

The new Perl 6 Modules page gives an overview of existing Perl 6 modules; we hope to evolve it into a real CPAN equivalent.

Community

This year we had another Perl 6 Advent Calendar, with much positive feedback both from the Perl 6 community and the wider programming community.

We were also happy to welcome several new prolific contributors to the Perl 6 compilers and modules. The atmosphere in the community still feels relaxed, friendly and productive -- I quite enjoy it.

The year ends like it started: with a Perl 6 Coding Contest. This is a good opportunity to dive into Perl 6, provide feedback to compiler writers, and most of all have fun.

31 Dec 2011 6:00pm GMT

The p6cc contest is underway. Yay.

[Coke]++ discovered on the channel that Rakudo nom didn't have a -c flag. The five base-test files all syntax-check the corresponding code files using the -c flag. Which made Rakudo nom and the base-test files incompatible. Oh noes.

<moritz> moritz-- # not reviewing the test harness properly

The fault is even more mine, of course, since I wrote the test harness. And I may be an "early adopter" with Perl 6, but I'm always very late at switching over to a new Rakudo branch.

I was late at switching over to ng, back when it was still called ng. And I'm late this time in switching over from b (the new name for ng, since the n stands for "new" and ng isn't new anymore) to nom (the new "new").

I'll take the leap any day now, I promise.

moritz++ was quick in patching up the -c omission.

<dalek> rakudo/nom: a9bead6 | moritz++ | src/ (2 files):
<dalek> rakudo/nom: reimplement -c command line option
<dalek> rakudo/nom: review: https://github.com/rakudo/rakudo/commit/a9bead6d48
<moritz> masak: there you go
<masak> moritz++

What this means in practice is: you can't use the latest Rakudo nom compiler release to solve the p6cc problems. Not without modifying the base-test files anyway. But you can use the bleeding-edge git checkout of the nom branch.

If you're on either Niecza or Rakudo b, things should be fine: those have a working -c flag already.

Those are the breaks. Perl 6 is evolving, and the mat is constantly being pulled out from under us. To keep up, one has to do a jig now and then.

We added a NOTES file to keep track of information of this kind that we didn't manage to get into the contest instructions.

On the channel, we also had some nice concluding discussion about the nature of the -c flag.

<moritz> to me it felt a bit like a cheat
<moritz> because there is already some mechanism for specifying the target stage
<moritz> but it's too tightly coupled to the output from the existing stages to
         be easily usable
<moritz> so I feel like hijacking an existing mechanism
<masak> I guess.
<masak> in some sense, "checking syntax" isn't so much of a compiler stage as...
        a decision not to go past a certain compiler stage.
<TimToady> in a sense, -c adds the final CHECK, that just exits with status
<masak> right.
<TimToady> it can even be implemented that way, since CHECKS do lifo order
<masak> "everything turns out to be yet another setting" :)
<TimToady> yes, it could also be done with a variant setting, but that seems
           a bit heavyweight
<TimToady> otoh, it would be possible to sneak CHECK pushes in before the -c,
           so maybe a setting is the cleaner way

28 Dec 2011 8:49pm GMT

Announce: Niecza Perl 6 v13

This is the thirteenth release of Niecza Perl 6, as usual scheduled on
the last Monday of the month. I'm on winter break now, which means lots
of time for improvements. Muahahaha. A large portion of the improvements
have been in regular expression support.

Will Coleda and Solomon Foster are working on fudging spectests so they run on
Niecza; between that and actual improvements, we've gained 5000+ working
spectests since the last release. See [4] for a dramatic visual.


You can obtain a build of Niecza from [1]. This build contains a
working compiler as a set of .exe and .dll files suitable for use with
Mono or Microsoft .NET. If you wish to follow latest developments,
you can obtain the source from [2]; however, you will still need a
binary for bootstrapping, so you gain nothing from a "source is
better" perspective.

Niecza is a Perl 6 compiler project studying questions about the
efficient implementability of Perl 6 features. It currently targets
the Common Language Runtime; both Mono and Microsoft .NET are known to
work. On Windows, Cygwin is required for source builds only; see the
README for details.


List of changes



[Breaking changes]

/ <{ foo }> / is no longer accepted as a synonym for / <?{ foo }> /.

$0 is no longer allowed to mean $/ when there is no capture zero.

$/.ast no longer defaults to ~$/. Use $() if you want that.



[Major features]

New-style character class expressions like <:Uppercase & :Greek> are
now supported.

Unicode property access is now supported! In addition to the use in
regexes, there is also a minimal Niecza::UCD module which allows querying
the properties of characters. All non-Unihan properties defined in
Unicode 6.0.0 are available.

Runtime number parsing has been radically extended and now supports the
full gamut of Perl 6 number syntaxes.



[Minor features]

Supplementary characters are now generally supported, though StrPos-type
counting for chars, substr, etc is still in UTF-16 code units.

codes is now available, for when you need to actually count code points
(be aware that it is O(n)).

The regex infix operators & and && are now supported (currently treated
as synonyms, but don't rely on this).

&pow is a little bit smarter and needlessly returns NaN in fewer cases
(Solomon Foster, Stefan O'Rear).

Imported a few tweaks from STD, including a better message for say$_.

\h \v \s \w and \d now use the recommended UTS18[3] definitions.

\n, $$, and ^^ now match any vertical whitespace, including CRLF as
a single unit.

% is now supported for all quantifiers and %% is available too.

$() @() %() are now supported.

/$var/ now allows $var to be a Regex. Likewise /<$var>/, /@var/,
and /<@var>/ have been added. / <{...}> / now does the right thing.

The implementation of $<foo>=[...] has been considerably simplified
and depessimized.

/ <.$foo> / assertion syntax is now supported.

/ <foo> / will now call a lexical "my regex foo" if possible. To minimize
potential accidents, this applies ONLY to regexes, tokens, and rules;
despite using the same namespace, a "my sub foo" will not be called.

/ <&foo(...)> / now allows arguments.

@<foo> now correctly contextualizes.

Nontrivial regex protos, like "proto regex foo { "bar" {*} }", are now
implemented.

<( and )> are now supported.

Added Match methods: kv, keys, values, caps, chunks, prematch, postmatch.

Added \c[] syntax in strings and regexes.

Rat and FatRat now stringify as fractions.

Inf now stringifies as "Inf" rather than leaking C# Infinity.

Added predefined quasi-property rules like <alnum>.

Added Niecza::Benchmark, providing the barest minimum of functionality like
Perl 5's "Benchmark".

Inf, NaN correctly handled in rounding, Cool.truncate, &kv, Pair.invert,
&srand, allow :x(*) in subst, .chr, .ord, .chrs, .ords, &chrs, &ords,
trim methods, &roll, .roll(*), end, min, max, minmax (Solomon Foster)

Added .pick(*), corrected Str and Numeric for Range (Will Coleda)

classify (Moritz Lenz)



[Selected bug fixes]

Threads.pm6 is working again, now with exception-safe locking.



[Other]

There is now documentation on how to prepare releases.


Getting involved

Contact sorear in irc.freenode.net #perl6 or via the sender address of
this mailing. Also check out the TODO file; whether you want to work
on stuff on it, or have cool ideas to add to it, both are good.

Future directions

My current priorities are:
1. Make regexes much more feature-complete, including general Unicode
properties and grapheme mode
2. Prototype the debugger
3. 6model convergence work, including roles/native types
4. Figure out how modules and S11 stuff should work in Niecza. Do it.


[1] https://github.com/downloads/sorear/niecza/niecza-13.zip
[2] https://github.com/sorear/niecza
[3] http://www.unicode.org/reports/tr18/
[4] https://github.com/flussence/specgraphs/raw/master/impls.png

26 Dec 2011 3:55pm GMT

The kind elves who spend the rest of the year working in Santa's shop to bring you more of Perl 6 each year would like to wish you a very warm and fuzzy Christmas vacation. December is always a special time for us, because we get to interact with you all through the interface of the advent calendar. We think that's wonderful.

Be sure to check out this year's Perl 6 coding contest, where you can win €100 worth of books!

Merry Christmas!

25 Dec 2011 10:52am GMT

With slightly less gratuitous buildup of expectations this year, but with no less grandeur and excitement, we're proud to announce:

The 2011 Perl 6 Coding Contest

Today there's two of us arranging the contest: the esteemed Moritz Lenz, and me. Hello.

This is a contest for people who are aware that Perl 6 has been "officially released", and who want the perfect excuse to start playing around with the language. As the Perl 6 community steadily works its way towards production-readiness, we can already enjoy ourselves royally by solving interesting Computer Science problems!

Here's the deal:

do five tasks, and you might win Amazon books worth €100!

Addendum: Thanks to an anonymous donor, we're very happy to announce that there is now a second prize too: Amazon books worth 100 USD!

The contest starts now, today on 2011-12-25. It ends five weeks later, on 2012-01-29. Registration is open for two weeks, starting now. Just send an email to cmasak@gmail.com - saying "sign me up!" or even sending your Amazon wishlist so we'll know which books to buy you if you win.

You might be curious about what you need to do to win. Here's an extract from the file RULES:

Since "code quality" is a slightly subjective measure, let us provide a few
hints of what we'll be looking for:

* Correctness.
* Readability.
* Consistency.
* Clarity of intent.
* Algorithmic efficiency.
* Idiomatic use of Perl 6.
* Brevity.

In short, what we're looking for is top-quality code. That's how you win.

Here are the five tasks. Write Perl 6 programs to...

• Find a way to express an integer as an expression containing four 9s. (more)
• List the numbers which are sums of cubes in more ways than one. (more)
• Calculate addition chains. (more)
• Slide a hex-shaped piece across a board. (more)
• List all possible trees of a certain type. (more)

We've chosen the problems so that they're easy to explain, but allow contestants quite a bit of freedom to play around with various solutions.

If you're curious about the details of the contest, I recommend the file RULES. For easy downloading, here's a .zip file of everything you need for the contest.

Addendum: For useful afterthoughts, we've decided to keep a file NOTES around. Think of it as useful clarifications to the existing files in the repo. We won't pass or fail people based on things found in NOTES, but the items may help you with the tasks.

The rest is up to you. Hope hear from you - we want you to win those books!

25 Dec 2011 10:20am GMT

Planet Perl is going dormant. This will be the last post there for a while.

Why? There are better ways to get your Perl blog fix these days.

You might enjoy some of the following:

• blogs.perl.org
• PerlSphere
• Iron Man Perl Blogging Challenge
• Perl.com

Will Planet Perl awaken again in the future? It might! The universe is a big place, filled with interesting places, people and things. You never know what might happen, so keep your towel handy.

21 Mar 2011 2:04am GMT

A few years ago, I wrote about cheap wooden discs as D&D minis, and I've been using them ever since. They do a great job, and cost nearly nothing. For the most part, we've used a few for the PCs, marked with the characters' initials, and the rest for NPCs and enemies, usually marked with numbers.

With D&D 4E, we've tended to have combats with more and more varied enemies. (Minions are wonderful things.) Numbering has become insufficient. It's too hard to remember what numbers are what monster, and to keep initiative order separate from token numbers. In the past, I've colored a few tokens in with the red or green whiteboard markers, and that has been useful. So, this afternoon I found my old paints and painted six sets of five colors. (The black ones I'd already made with sharpies.)

I'm not sure what I'll want next: either I'll want five more of each color or I'll want five more colors. More colors will require that I pick up some white paint, while more of those colors will only require that I re-match the secondary colors when mixing. I think I'll wait to see which I end up wanting during real combats.

These colored tokens should work together well with my previous post about using a whiteboard for combat overview. Like-type monsters will get one color, and will all get grouped to one slot on initiative. Last night, for example, the two halfling warriors were red and acted in the same initiative slot. The three halfling minions were unpainted, and acted in another, later slot. Only PCs get their own initiative.

I think that it did a good amount to speed up combat, and that's even when I totally forgot to bring the combat whiteboard (and the character sheets!) with me. Next time, we'll see how it works when it's all brought together.

21 Mar 2011 12:47am GMT

In Pisa I gave a lightning talk about Perl Vogue. People enjoyed it and for a while I thought that it might actually turn into a project.

I won't though. It would just take far too much effort. And, besides, a couple of people have pointed out to be that the real Vogue are rather protective of their brand.

So it's not going to happen, I'm afraid. But as a subtle reminder of the ideas behind Perl Vogue I've created some t-shirts containing the article titles from the talk. You can get them from my Spreadshirt shop.

20 Mar 2011 12:02pm GMT

CPAN has gotten its first real update in a while tonight; the content is from the cpanorg git repository.

We tried to get the FAQ cleaned up a bit (though there's plenty of work left) and Leo Lapworth pretty heroically also did a first pass on cleaning up the ports page.

You might also notice a search box for search.cpan.org which we find appropriate, a list of recently uploaded modules on the homepage and a new page on how to mirror CPAN.

If you read the latter page, you'll see that the master mirror is now cpan-rsync.perl.org::CPAN (rsync only). In the coming weeks we'll work on encouraging the CPAN mirrors to switch to mirror from here to ease the load on FUnet, the sponsor of the master mirror for the last 15 years.

Work is also coming along well on the instant update mirroring system.

- ask

20 Mar 2011 9:10am GMT

LWP 6 makes hostname verification the default -- so note this from LWP::UserAgent:

If hostname verification is requested, and neither SSL_ca_file nor SSL_ca_path is set, then SSL_ca_file is implied to be the one provided by Mozilla::CA. If the Mozilla::CA module isn't available SSL requests will fail. Either install this module, set up an alternative SSL_ca_file or disable hostname verification.

If you use LWP and want SSL, you need IO::Socket::SSL (recommended) and Mozilla::CA.

19 Mar 2011 2:57am GMT

Remember use.perl? It's moth-balled now, but for years it provided two valuable services to the Perl community.

Firstly it provided a hosted blog platform which many people used to write about many things - sometimes even Perl. Of course we now have blogs.perl.org which provides a very similar service.

And secondly, it provided a place where people could submit stories related to Perl and then editors would approve the stories and publish them on the front page. Since use.perl closed down, the Perl community hasn't really had a centralised site for that.

Over the last eighteen months or so I've had conversations with people about building a site that replaced that part of use.perl. But there's always been something more interesting to work on.

Then, at the start of this week, Leo asked if I knew of a good Perl news feed that he could use on the front page of perl.org. And I realised that I'd been putting it off so too long. A few hours of WordPress configuration and Perl News was ready to go.

So if you have any interesting Perl news to share, please submit it to the site.

17 Mar 2011 2:01pm GMT

http://perlnews.org/ has just launched and will be providing a source for major announcements related to The Perl Programming Language (http://www.perl.org/). Find out more at http://perlnews.org/about/ - or if you have a story submit it http://perlnews.org/submit/.

All stories are approved to ensure relevance.

Thanks

The Perl News Team.

17 Mar 2011 1:44pm GMT

I'm still blogging five days a week, but obviously not here. That's largely because my new daughter is forcing me to choose where I spend my time and I can't blog too much about what I do lest I reveal trade secrets. So, just to keep my hand in, here's an ugly little "80% hack" that lets me find bugs like mad in OO code. I should really combine this with my warnings::unused hack and start building up a tool find find issues in legacy code.

First, an "80% Hack" is based on the Pareto Principle which states that 80% of the results stem from 20% of the effort. So I often write what I call 80% hacks which are simply quick and dirty tools which get things done.

The idea is simple. In legacy OO code where we're not using Moose, we have a nasty tendency to reach inside a blessed hashref. However, as classes start getting old and crufty, particularly in legacy code which is earning the company a ton of money, it's easy for someone to either misspell a hash key or refer to keys which are no longer used. What I've done is assume that each of these keys are used once and only once and I also assume they look like this:

$self->{ foo }
$_[0]  ->  { "bar" } # yeah, we need arbitrary whitespace
shift->{'something'} # and quotes

Yes, this code could be improved tremendously, but 80% hacks are personal hack which I simply don't pour a lot of time and effort into. Besides, they're fun.

#!/usr/bin/env perl                                                                                                                                                                                                                       

use strict;
use warnings;
use autodie ':all';
use Regexp::Common;

my $module = shift or die "usage: $0 pm_file";

#my $module = '/home/cpoe/git_tree/main/test_slot';

my $key_found = qr/
    (?: \$self | \$_\[0\] | shift )  # $self or $_[0] or shift
    \s* ->                         # ->
    \s* {                          # { 
    \s* ($RE{quoted}|\w*)          # $hash_key
    \s* }                          # }
/x;

open my $fh, '<', $module;

my %count_for;
while (<$fh>) {
    while (/$key_found/g) {
        my $key = $1;
        $key =~ s/^["']|['"]$//g;    # try and strip the quotes

        no warnings 'uninitialized';
        $count_for{$key}{count}++;
        $count_for{$key}{line} = $.;
    }
}

foreach my $key ( sort keys %count_for ) {
    next if $count_for{$key}{count} > 1;
    print "Possibly unused key '$key' at line $count_for{$key}{line}\n";
}

I run that with a .pm file as an argument and I get a report like:

Possibly unused key '_key1' at line 1338
Possibly unused key '_key2' at line 5325
...
Possibly unused key '_keyX' at line 4031

It's amazing how many bugs I've found with this.

I can't blog as much as I used to, but they make it all worth it.

17 Mar 2011 9:33am GMT

A goal of the MyCPAN work was to start with an existing Perl distribution and work backward to the MiniCPAN that would re-install the same thing. I hadn't had time to work on that part of the project until this month.

The first step I've had for awhile. I've created a database of any information I can collect about a file in the 150,000 distributions on BackPAN. There are about 3,000,000 candidate Perl module or script files. That includes basics such as MD5 digest of the file, the file size, the Perl packages declared in the file, and the package versions.

The next step is what I've been doing this week: collect the same information on the files in a Perl installation, which is much easier to do. There's not wacky distribution stuff involved.

Putting those two together should find the distributions that could make up the installation. With that list of distros, it's just a matter of creating the right 02packages file that a CPAN client can use. Easy peasy, I thought.

But, it's not that easy. Each file in the existing installation might have come from several distributions. That is, between different versions of a distribution, it's likely that many of the modules didn't change. So, looking at a single file doesn't lead to a single distribution. It might list several possible distributions.

But that's a start. Other files from that distribution should be present, and they each might come from several distributions even if one of them changed. If there's any file that only belongs to one distribution, that collapses everything for that distribution. If not, I have to find the overlap in possible distributions. There should be one distribution that overlaps more than all of the others, and that should be the right distribution.

That's not quite right either though, because some distribution versions don't change the module files. They update a test or the build file or something besides whatever is in lib. You'd think that at least the $VERSION would change, but think of any exception and you'll probably find it on BackPAN. That's not as horrible as it seems though. If all of the module files are the same, it doesn't matter which distribution I use, does it?

But then, there are some files that not only might come from more than one version of a particular distribution, but might also be in a completely different distribution. Some distributions have lifted files from other distributions. Files from the URI and LWP modules show up in other distributions. How should I figure out which one should be the candidate distribution?

The database I was using was just an extract of all of the information I have on each distribution and it's oriented to individual files. I select records to match up MD5 digests. However, when I get records back with different distributions, which one might be installed? If an installed file might have come from both Foo-Bar and Baz-Quux, I have to remove one of the distributions somehow. In that case, I have to step back to look at what else either distribution might have been installed. If the other files from Foo-Bar aren't there, it's probably not Foo-Bar.

That might be the end of the story, but what if both Foo-Bar and Baz-Quux are installed? That part I haven't figured out, but it's likely that the previous step will be inconclusive since the files from both distributions will all be there. However, there's also the chance that an older version of Foo-Bar and a newer Baz-Quux is there. If they both install a Foo.pm file, the older version in Foo-Bar might have been over written by an updated version from Baz-Quux. So, Every file except one from Foo-Bar is there. That means that there's possibly some path independence there so I would have to make sure I install modules in the right order to recreate the installation.

If the module installation order matters, I think that might rule out creating a Task::* distribution, which can't guarantee the installation order, I think. A Bundle::* might be able to do it though.

So, you think that's the end of it? Think about configure_requires and build_requires. Anything those need to be in the MiniCPAN too, even if they aren't in the installation. You have the option of not permanently installing those modules, so you might not see them in the analysis. Even when I get a list of distributions, I then have to check their dependencies to see if there's anything extra I need to add.

So, not so bad.

17 Mar 2011 8:37am GMT

I've been spending a fair bit of time working on Perl 5 core documentation. I started by editing and reorganizing some of the documents related to core hacking. This update will be in 5.14, due to be released in April. I'm also working on replacing the existing OO tutorials and updating the OO reference docs, though this won't make it into the 5.14 release.

There's been a lot of discussion on my OO doc changes, some of it useful, some of it useless, and some of it very rude (welcome to p5p!). Many of the people in the discussion don't have a clear vision of who the docs are for. Without that vision, it's really not possible to say whether a particular piece of documentation is good or not. A piece of documentation has to be good for a particular audience.

There's a number of audiences for the Perl 5 core docs, and they fall along several axes. Here are the axes I've identified.

Newbies vs experienced users

Newbie-ness is about being new to a particular concept. You could be an experienced Perl user and still be new to OO programming in general, or new to OO in Perl.

For my OO docs, I'm writing for two audiences. First, I'm writing for people who are learning OO. That's why the document starts with a general introduction to OO concepts. Second, I'm writing for people who want to learn more about how to do OO in Perl 5. For those people, the tutorial points them at several good OO systems on CPAN.

I'm not writing for people who already know Perl 5 OO and want to learn more, that's what the perlobj document is for.

From the discussion on p5p, I can see that many people there have trouble understanding how newbies think. I like how chromatic addresses these issues in a couple of his blog posts.

How the reader uses Perl

Perl is used for lots of different tasks, including sysadmin scripts, glue code in a mostly non-Perl environment, full app development, etc.

Ideally, we'd have tutorial documents that are appropriate for each of these areas. I think the OO tutorial is most likely to be of interest to people writing full Perl applications. If you're just whipping up some glue code, OO is probably overkill.

It would also be great to see some job-focused tutorials, like "Basic Perl Concepts for Sysadmins" or "Intro to Web Dev in Perl 5". Yes, I know there are books on these topics, but having at least some pointers to modules/books/websites in the core docs is useful.

Constraints on the reader's coding

If you're doing green field development, you have the luxury of using the latest and greatest stuff on CPAN. If you're maintaining a 10-year old Perl web app (I'm so sorry), then you probably don't. Some readers may not be able to install CPAN modules. Some readers are stuck with in house web frameworks.

People stuck with old code need good reference docs that explain all the weird shit they come across. People writing new code should be guided to modern best practices. They don't need to know that you can implement Perl 5 OO by hand using array references, ties, and lvalue methods

My OO tutorial is obviously aimed toward the green field developers. It's all about pointing them at good options on CPAN. As I revise perlobj, I'm trying to make sure that I cover every nook and cranny so that the poor developer stuck with (2001 Perl OO code can understand what they're maintaining.

(Sadly, that's probably my code they're stuck with.)

Conclusion

I'd like to see more explicit discussion of who the intended readers are when we discuss core documentation. Any major doc revision should start with a vision of who the docs are for.

There's probably other axes we can think about when writing documentation as well. Comments on this are most welcome.

16 Mar 2011 8:13pm GMT

Basic integration of software and web sites with Facebook, Twitter, and other social networking systems has become a litmus test for business these days. Depending on the software or site you might need to fetch some data, make a post, create events, upload photos, or use one or more of the social networking sites as a single sign-on system. This series will show you how to do exactly those things on Facebook using Facebook::Graph.

This first article starts small by using Facebook as an authentication mechanism. There are certainly simpler things to do, but this is one of the more popular things people want to be able to do. Before you can do anything, you need to have a Facebook account. Then register your new application (Figure 1).

Figure 1. Registering a Facebook application.

Then fill out the "Web Site" section of your new app (Figure 2).

Figure 2. Registering your application's web site.

Registering an application with Facebook gives you a unique identifier for your application as well as a secret key. This allows your app to communicate with Facebook and use its API. Without it, you can't do much (besides screen scraping and hoping).

Now you're ready to start creating your app. I've used the Dancer web app framework, but feel free to use your favorite. Start with a basic Dancer module:

package MyFacebook;

use strict;
use Dancer ':syntax';
use Facebook::Graph;

get '/' => sub {
  template 'home.tt'
};

true;

That's sufficient to give the app a home page. The next step is to force people to log in if they haven't already:

before sub {
    if (request->path_info !~ m{^/facebook}) {
        if (session->{access_token} eq '') {
            request->path_info('/facebook/login')
        }
    }
};

This little bit of Dancer magic says that if the path is not /facebook and the user has no access_token attached to their session, then redirect them to our login page. Speaking of our login page, create that now:

get '/facebook/login' => sub {
    my $fb = Facebook::Graph->new( config->{facebook} );
    redirect $fb->authorize->uri_as_string;
};

This creates a page that will redirect the user to Facebook, and ask them if it's ok for the app to use their basic Facebook information. That code passes Facebook::Graph some configuration information, so remember to add a section to Dancer's config.yml to keep track of that:

facebook:
    postback: "http://www.madmongers.org/facebook/postback/"
    app_id: "XXXXXXXXXXXXXXXX"
    secret: "XXXXXXXXXXXXXXXXXXXXXXXXXXX"

Remember, you get the app_id and the secret from Facebook's developer application after you create the app. The postback tells Facebook where to post back to after the user has granted the app authorization. Note that Facebook requires a slash (/) on the end of the URL for the postback. With Facebook ready to post to a URL, it's time to create it:

get '/facebook/postback/' => sub {
    my $authorization_code = params->{code};
    my $fb                 = Facebook::Graph->new( config->{facebook} );

    $fb->request_access_token($authorization_code);
    session access_token => $fb->access_token;
    redirect '/';
};

NOTE: I know it's called a postback, but for whatever reason Facebook does the POST as a GET.

Facebook's postback passes an authorization code-a sort of temporary password. Use that code to ask Facebook for an access token (like a session id). An access token allows you to request information from Facebook on behalf of the user, so all of those steps are, essentially, your app logging in to Facebook. However, unless you store that access token to use again in the future, the next request to Facebook will log you out. Therefore, the example shoves the access token into a Dancer session to store it for future use before redirecting the user back to the front page of the site.

NOTE: The access token we have will only last for two hours. After that, you have to request it again.

Now you can update the front page to include a little bit of information from Facebook. Replace the existing front page with this one:

get '/' => sub {
    my $fb = Facebook::Graph->new( config->{facebook} );

    $fb->access_token(session->{access_token});

    my $response = $fb->query->find('me')->request;
    my $user     = $response->as_hashref;
    template 'home.tt', { name => $user->{name} }
};

This code fetches the access token back out of the session and uses it to find out some information about the current user. It passes the name of that user into the home template as a template parameter so that the home page can display the user's name. (How do you know what to request and what responses you get? See the Facebook Graph API documentation.)

While there is a bit of a trick to using Facebook as an authentication system, it's not terribly difficult. Stay tuned for Part II where I'll show you how to post something to a user's wall.

15 Mar 2011 6:36pm GMT

For anyone who may have been affacted by the upgrade to LWP, the situation should now be resolved. David has put in place a 3rd party verified SSL certificate on the Metabase server, so all submissions should now be able to resolve certificate authenticity.

If you have implemented any short term fixes, you may need to remove them, before accepting the new certificate.

We now return you to your scheduled programming :)

Cross-posted from the CPAN Testers Blog

15 Mar 2011 2:08pm GMT

As Barbie reported, CPAN Testers broke under LWP version 6, as this version of LWP now defaults to rejecting unverifiable SSL connection (e.g. self-signed certificates). That meant that CPAN Testers upgrading their LWP could no longer submit reports (at least via https). The quick and obvious solution was to buy an SSL certificate and that's now done. If you visit https://metabase.cpantesters.org/, you can see the new certificate in action.

15 Mar 2011 1:19pm GMT

I spent a good few hours today attempting to use the MailboxGUID returned from the WMI Exchange provider to search for the associated Active Directory account, using the msExchMailboxGuid attribute.

Here's two functions I came up with in the end. One to convert MailboxGUID to something that a search on msExchMailboxGuid will like:

sub exch_to_ad {
  my $guid = shift;
  $guid =~ s/[\{\}]+//g;
  my $string = '';
  my $count = 0;
  foreach my $part ( split /\-/, $guid ) {
    $count++;
    if ( $count >= 4 ) {
      $string .= "\\$_" for unpack "(A2)*", $part;
    }
    else {
      $string .= "\\$_" for reverse unpack "(A2)*", $part;
    }
  }
  return $string;
}

And another to take a msExchMailboxGuid field, which is a byte array, and convert it to a MailboxGUID.

sub ad_to_exch {
  my $guid = shift;
  my @vals = map { sprintf("%.2X", ord $_) } unpack "(a1)*", $guid;
  my $string = '{';
  $string .= join '', @vals[3,2,1,0], '-', @vals[5,4], '-', 
     @vals[7,6], '-', @vals[8,9], '-', @vals[10..$#vals], '}';
  return $string;
}

Hopefully this should save other people some time.

14 Mar 2011 1:50pm GMT

If you're an existing CPAN Tester, and have recently upgraded LWP, you may have noticed that your report submissions have been failing. The reason being that LWP::UserAgent now requires that any https protocol request, needs to verify the certificate associated with it. With the Metabase having a self-signed certificate, this doesn't provide enough verification and so fails.

In the short term if you don't need to update LWP (libwww-perl), refrain from doing so for the time being. For those that have already done so, or have recently built test machines from a clean starting point, you will either need to wait until we have put a long term solution in place, or may wish to look at a solution from Douglas Wilson. Douglas has created a "hypothetical distribution", which you can see via a gist.

Others have also blogged about the problem, and have suggests and insights as to how to overcome this for the short term:

• CPAN::Reporter, Transport::Metabase, and libwww-lwp 6 by Douglas Wilson
• CPAN::Reporter fails to send through Metabase with new LWP 6 by Randy Stauner
• SSL host checking and LWP::UserAgent by Colin Newell

We will have more details of the longer term solution soon.

Cross-posted from the CPAN Testers Blog

14 Mar 2011 9:08am GMT

We've decided we're gonna start releasing Dancer under codenames that relate to people who've worked on the release.

This release (1.3020) we've seen the continued (and blessed!) involvement of a one "Michael G. Schwern". To some of you he might just be a "mike" or "michael" (or perhaps "the schwern"), but none of us in the core knew Schwern personally before his involvement with Dancer, and this came as a very welcomed and pleasant surprise.

Considering the storm of issues and pull requests done by Schwern, we decided the next version should be named after him, hence "The Schwern Cometh". :)

The latest version is only a week-or-so of development but carries the following statistics:

• 
  
• 6 contributors
• 
  
• 6 bug fixes
• 
  
• 12 features and enhancements
• 
  
• 10+ issues closed
• 
  

I really do see this as exceptional work. Other than Schwern I also want to thank Naveed Massjouni and Maurice Mengel for their contributions to this release (and any previous release!).

In the near future we'll also unveil the most elaborate hooks subsystem in the micro web framework world. I already know whose names will be splashes on that release. :)

14 Mar 2011 8:32am GMT