A Lexical Mistake

During some of the early design work on the Mention programming language, I recognized that a fixed parser would be unable to deal with a great number of interesting language problems. Things like embedded SQL and other DSL language, or perhaps even a unit notation.

All of these things could be added into a syntax, I believe, with a carefully constructed extensible parser. Each extension would become a libraries (modules) adding or replacing function points in the existing parser later reusing some of them as part of the extended syntax. (The actual parser would be based on Parsec and other similar combinator parsers. OMeta is also related to this parser architecture.)

The problem with adding new syntaxes to a language isn’t just the parser implementation however. Most of the really tricky problems actually come from the rest of the tool chain. Things like the compiler, debugger, profilers, editor, and a bunch of other potential tools. It had become clear to me that I wasn’t just working on an extensible parser, but rather an extensible language, and that meant an extensible compiler architecture.

Basically I thinking about a completely extensible intermediate language, which would power a set of semi-extensible libraries for implementing the bulk of the tool-chain. These libraries will take the leg-work out of processing the source code, and will allow the tools to manipulate or interact with the code in its intermediate form (this includes editors and analytical tools). The only problem I saw was that I had absolutely no idea how to construct such an intermediate language, little own how to write tools and libraries to interact with it.

Anyway, recently I run into the latest (at the time) Steve Yegge post, with a somewhat inspiring discussion about the Prototype Pattern (The Universal Design Pattern). After musing about pattern (again) for a couple of days, it finally hit me, I had the architecture all wrong. This isn’t just some extensible compiler or parser…this was the Self programming language…redesigned, implemented and tailored for the task of compiler writing. The AST is simply a specialized variation of the Prototype pattern, or more succinctly, a compiler is just a constrained dialect of Self, designed for compiler tasks.

So this my proposal is this, use the prototype pattern for Mentions AST and build a framework of libraries and tools designed to aid in the maintenance and development of an AST and the compilers, parsers, debugger, et cetera, which depend on it.

There are already a few pitfalls that I can imagine coming out of this system. For one, a good namespace or typing system will probably be required to prevent name conflicts. That is, each extension or collection of extensions would add only private collections of attributes, while a smaller group of shared public attribute interfaces would be used to facilitate cooperation between extensions. And then there’s still the issue of expected behavior, which may be complicated by the potential for different combinations of compiler extensions to inadvertently interact in unexpected ways.

In any case, I believe that this will be one of the key features of the core Mention programming language and it may even make some of the more complicated features like type systems easier to implement and experiment with later on. At the very least, you can expect to see some follow up material on this when I start to get further into the implementation of the Mention.

– Lorenz