Jonathan Worthington — 45 minutes
Today, the only specified form of a Raku program is source code. This is fine enough for humans to read and write. However, it leaves those who might wish to write programs that deal with Raku programs in a bad place: they only official way to do that is to write their own parser. One may try and hook into the Rakudo compiler, however this is neither a supported nor stable API, not to mention being inconvenient for these tasks too.
Elsewhere, modules that use Raku as a compilation target resort to producing text to then `EVAL`, raising the risk of both injection attacks as well as being inherently inefficient. Nice things we might want to build - such as automatically deducing an OpenAPI specification from a Cro `route` block - are not really possible to implement today, given Raku's experimental macro support only provide access to the macro arguments as opaque objects that cannot be introspected.
These problems, and others, call for a second specified form of the Raku language aimed at programs rather than people. Around 18 months ago, I started thinking about how to make that happen, and a little over a year ago started implementation work. The task not only entails working out a good programmatic representation for the entire Raku language, but also building a new compiler frontend that works in terms of this representation.
It's the kind of journey one would probably not set out on without some amount of denial about just how much effort it was going to be. A year and around 300 AST node types later, I'm still working on it. The new compiler frontend has gone from limping through a few sanity tests to passing hundreds of specification test files in full. General solutions have been found to compilation challenges that cross-cut multiple language features. The resulting compiler implementation looks set to be rather easier to reason about that it aims to replace.
In this talk, I'll share the progress so far, discuss some of the things I've learned on the way, and speculate about the work that remains to be done.