With all this Haskell programming and a rather lengthy predicted alpha data for the Haskell game engine, I started to wonder about my last game development project. Actually, it’s probably more accurate to say, I was looking over the Secret Maryo Chronicles forums and remembered how much I wanted to create a similar game, obviously with a somewhat different gameplay and tools.

See, this story goes further back to when I was still learning the basics of Haskell. I’d produced a simple game demo, but made no real progress in adapting that code base into something that would scale to a full game. That’s without considering the game specific tools either. My solution was to start a new project with the explicit goal of simply creating an actual game and level editor, based on the existing assets from Secret Maryo Chronicles placing the game under the same GPL license.

Since this isn’t the most graphic story from here, we’ll cut back to the present, which brings me to the first simple tech demo since I restarted the project. Its not much yet, just some code to test some basic view layout containers—the Shoes style stack and flow, although the larger design is inspired by Hopscotch more than Shoes.

I should probably point out that the entire program runs on wxPython and OpenGL, but doesn’t use the wxWidgets controls, but rather a uses a simple custom framework designed for rendering directly in OpenGL. This is meant to simplify creating the in-game menus and since the editor will need to utilize many of the in-game graphics anyway, not to mention the level test functionality, it will probably help there too.

This demo doesn’t do much really, it’s just a simple stack and flow layout container test, with a few image views. Also, you’ll need the wxPython, PyOpenGL and the Python Image Library (PIL) for this one. There is a fallback library, but I haven’t fixed a minor image flip but in the usage of said library, so the images will appear upside down. The demo should appear similarly to the screenshot below. (I found I had to move the window to get the OpenGL view to start drawing. Since I’m not developing on Mac OS I didn’t realize until I grabbed the screenshot for this post itself.)

To checkout the simple view framework demo for yourself, simply run the following commands (or for the git users out there, its on the demo-view-framework-1 branch.)

git clone git://github.com/krysole/smc-ma.git smc-ma
cd smc-ma
git checkout demo-view-framework-1

Then running the command python main.py should start the code.

– Lorenz

Well, its been somewhat slower going for the implementation of Fire lately. The interpreter I’m implementing is fairly simple, just an AST and direct interpretation of that, kind of like a Lisp meta-circular evaluator, but it has been surprisingly difficult to get around the complexity of the interpreter. I’m starting to thing this project may take a little longer than I originally expected, but that’s OK, it’s still been quite a lot of fun so far.

The biggest hurdle I’ve stumbled on over the past couple of weeks is getting the actual interpreter doing anything at all. There’s plenty of code sitting in the modules, but none of it can actually work since I haven’t quite managed to finish the implementation based on the design in use.

So Fire is basically just a heavily modified Smalltalk (or Self.) Most of the semantics are the same, with the significant exclusion of directly modifiable variables—but even those are supplemented with transaction variables and more functional programming facilities. At least that’s the design of the language so far.

The problem is that Fire, like Smalltalk, uses strict evaluation and at least some of the methods may need to, gasp, perform IO! To actually implement something like this, I have to implement the strict evaluation using the IO monad, even if it’s only by composing monads (or similar.) The story doesn’t quite end there though.

Programmers can also define local methods within a method in Fire. These methods are part of the lexical context, and correlate to the respective extension to the message send syntax. This means that the lexical context must be threaded through the evaluation of statement in a method.

Also, not all method calls return successfully. The return operator is also a variation of the normal return from a message send. The design of Fire here, calls for the usage of an Either return value, where the Left constructor provides a Fire exception object.

We would essentially like to hide the general flow using a monad for these two so that implementing methods for Fire in Haskell is as straight-forward as possible, not to mention the implementation of the interpreter itself.

So the decision I’m going with at the moment is to ignore any direct IO, which may eventually have to be reified back in later on, as well as transactions which should be a little simpler to put back in, and create a monad that captures both pieces of information. In theory, maybe it’s possible to implement this by combining monads—but the truth is, I’m just not that good yet. For that matter, this will be the first monad I’ve implemented (besides simple examples.)

Before I attempt the new design for the interpreter however, I also have another task I’m looking at which should be the topic of my next post. Testing Parsec parsers. All I’m going to say for now is that it involves QuickCheck, possibly because I haven’t actually finished writing the code yet. Until then, back to coding for me.

– Lorenz

Of all the programming languages I’m proficient in, not even Haskell is so curiously fun and exciting (in my mind) as the Smalltalk programming language. See the idea behind Smalltalk is deceptively simple, you have objects, which you send messages to. Those messages are handled by methods and finally, both messages and objects may maintain and carry references to other objects. This may just sound like any other object-oriented programming language, but all of this happens at run-time, including the creation of those methods, objects and classes!

Given my affinity for computer games, it’s not surprising then, that I’ve been eyeing off the usage of some dialect of Smalltalk for programming computer games. Even the performance of Smalltalk is competitive compared to many other high level programming languages. Since most high level game code is easier to program with the increased flexibility scripting/dynamic languages provide Smalltalk has always seemed to me, to be a very attractive counterpart to a C/C++ engine.

The same principle could also be applied to the Lua programming language, which is extraordinarily popular in game development right now, which is more or less designed to be attached to an engine written in the standard engine languages, C and C++.

As I pointed out in my last post, originally I only intended to create a game engine in Haskell, which could then help when developing games or frameworks dealing with the high level game code. But to make this easier I decided to put the conventional game development wisdom to good use, and add to the mix a scripting language instead. Also relevant are Haskell’s concurrency capabilities which, I believe, are drawing the most attention from more mainstream game developers.

A Counterpart To Haskell

It might be interesting to note that, in the early design of Fire, the language wasn’t actually going to be based on Smalltalk or any language specifically, rather, I was working out some of the design points to help in taking advantage of Haskell’s concurrency and functional programming capabilities. Eventually I came up with a basic features list along the lines of object-oriented, semi-functional, highly concurrent and prototype based (class-less).

While collecting my thoughts and trying to understand which kinds of syntax would be appropriate I noticed that regardless of which syntax I used, the language would have to almost completely separate the concept of mutable state from the objects themselves, allowing only the use of transaction variables. While I could technically have gone with almost any syntax from there, it’s a pretty sure thing that regardless of the syntax used, Fire was going to be a somewhat unfamiliar thing to work with. This is where Smalltalk syntax lends itself, not being difficult to adapt and extend, while also known for being easy to learn and unusually expressive.

There were some issues to deal with however. To start with, Smalltalk syntax isn’t technically a full language syntax, but only provides the syntax for method definitions. By proxy, the construction of objects and message sends is achieved, with a fair bit of help from the development environment (UI). It also helps to point out that image based languages are notoriously difficult to bootstrap, while many game developers these days use text based VCS anyway, it was obvious I would first have to invent the missing syntax.

After (admittedly) more than a few attempts, and the larger part of a notebook full of musings and hypothetical example code, I finally had a complete syntax and at least most of the basic semantics in mind. The resulting syntax now includes a module level notation, an object literal notation, there is a simplified notation for creating lists and everything else is a consequence of sending messages, as expected.

Difficulties

There are some issues in combining any object-oriented language with functional programming or concurrency. To make things simpler, and because Fire is designed for game development anyway, I’ve kept the basic semantics imperative, or more succinctly the language still has the conventional IO semantics of Smalltalk. The main differences, at this point, are the lack of mutable variables within the language itself (beyond TVars) and the extensive use of threads (asynchronous messages).

To make working with transaction variables easier, Fire does provides properties, implemented as objects responding to a ‘get’/’set:’ property protocol, and a property method syntax which maps a method pair (’var’/'var:’) to another object implementing the property protocol. Any object could theoretically be used here including IORef based objects, but that would generally risk concurrency issues and would probably be a bad idea. This is of course, alongside the usual method definition syntax, and a syntax for creating simply immutable variables, which is syntactic sugar for unary methods.

For those who have worked with Smalltalk, it’s immediately obvious that the main strength of the design has actually been lost however. That it, the main strength of Smalltalk has always been, at some level, the development environment itself. For those not familiar, the built in development tools allow any object, class or method to be reworked at any point while a program is running, even in the debugger. When developing an application, this level of flexibility is essential to the velocity a Smalltalk programmer can work at.

Such a development environment is probably going to be the most difficult thing to replicate in Fire, since both the UI frameworks as well as the tools themselves must be created from scratch. The VM will also likely require a way to allow programmers to ignore the immutability restrictions.

Luckily I do have something along the lines of a plan. What I believe will do the trick here is a simple dual mode VM, allowing (through the reflection layer) the modification of objects, while the finished games will use a release mode, preventing objects from being accidently scrambled. This solves both the development environment concerns, while keeping the language concurrency safe and hopefully simpler to integrate into the Haskell based engine. As as for the UI, I’ll get to that in a later post…

Clearly there’s a lot of stuff going on here, but for now, most of the work will focus on simply getting the spec written and the VM functioning. After then I can start moving only some of the details like the engine itself, and the libraries on either side of the language bridge. Actually, there’s the language bridge too, for that matter.

The purpose of this post is really just to give some idea of the reasoning behind a couple of the high level design choices used in Fire. There are plenty of other topics to consider yet, just at the language level, for example, prototypes systems are even more convention based than Smalltalk since the entire class system is replaced simply by methods for cloning and modifying the definition of objects. In any case, I think it should be interesting to see Smalltalk finally being applied to some serious game development, especially paired up to Haskell as well.

– Lorenz

Also, this project was killed some time ago. I’ve moved onto other things and when I look into more Haskell programming I will not be resurrecting this project (I have some more practical ideas in mind).

In my last attempt at coding gamedev, Haskell style, I found my true enemy to be my inability to fathom the modeling of high level game objects in basic Haskell records. The problem was pretty simple, I’m either not yet good enough to figure out how to implement high level game objects in Haskell, or the task requires something closer to the Functional Reactive type frameworks. The same Functional Reactive Programming that I still, to this day, don’t really understand.

In theory I should probably just keep reading through my copy Real World Haskell, but I always like to have one gamedev project and one language that I’m working on. This doesn’t necessarily include any reading or minor experiments of course, but it does help to limit how thinly I spread my free time and it’s nice to keep the gamedev and language design cravings in check (plus, it’s kind-of a fun, strange kind of duo, really).

Since my Haskell game project was essentially killed a while back, that left me with some time to work on a some gamedev projects in Python. The premise was, high level language, good for modeling game objects and apparently a solid library lineup. Solid at least, until I realized that almost all of my games require one simple critical feature. Tools.

The odd reality of any serious game development is that, no matter how clever your game is, you still need the tools, in which all of the content is created and developing those tools is, in my experience, a very hardcore UI intensive process, which given the current state of UI frameworks in Python is well…not good. Actually, perhaps that’s not fair, GTK may have done the trick nicely, but by the time I’d figured out how much UI I felt was really necessary, it seemed like I shouldn’t have to develop yet more UI code for the in-game UI.

Oh… and the Mention programming language is on temporary hiatus due to the heavy usage of untested extensions to the lambda calculus and type theory. While I slowly learn the math skills involved in learning the type theory (not that there aren’t other benefits to be had from learning mathematics properly anyway), I figure its going to take me at least a year to get it fully on track again.

Moving On…

So now there I was… back in Haskell… the Mention language stalled… and no game development project in sight… So I came up with an idea. What if I put some of the ideas I had floating around, and put them all together as a game, a language, and something to keep me working through Haskell. I would be developing a game engine designed specifically to beat the crap out of Haskell’s concurrency—and my CPU too, I suppose—while I also designed a high level game object language to implement the actual, high level game stuff.

It wasn’t actually until I’d been working on the language for a couple of weeks, when I realized what I was actually trying to implement, however. This link should probably actually be old news to anyone interested in gamedev, but here it is anyway: Tim Sweeney’s POPL 2006 talk “The Next Mainstream Programming Language: A Game Developer’s Perspective” (PPT Slides, Scribd Slides, I couldn’t find the video of the talk again.)

In the talk, he covers quite a few of the practical qualities that a modern commercial game contain in lines of code, complexity, and so on, but the point that stuck with me the most was the section on Concurrency. I swear, it literally screams “HASKELL!,” and not just because Haskell is mentioned later on, either. What I didn’t realize at the time, was that I’d end up trying to pull off a solution to Sweeney’s challenge, rather than simply using Python or Ruby and keeping the games… non-concurrent (or semi-concurrent).

There is actually a little section on why he doesn’t like Haskell at the very end of the slides. I’d just like to point out (for those who don’t realize this) that they’re all mostly irrelevant concerns since they’re all solvable in about five-seconds-flat by anyone who has more than two months of experience with Haskell, or aren’t significant problems anyway (laziness is not a performance hog, and targeting GHC optimizations isn’t impossible, et cetera). Honestly, I actually think it’ll probably be the two months of experience that causes the most problems (i.e. the unpopularity), but that may not be as much of an issue either, as better learning materials emerge.

Three Languages, for a Good Computer Game

So I’ve mentioned Haskell, and alluded to a scripting language, and of course we also have our shader language. To start with, I’d like to point out that the shader language is only partially bound to the game language, and even then it’s an indirect link, courtesy of the abstraction layer that typically accompanies the rendering code—that’s engine side—thus, the shader language may either be a separate language project, or (as I’ll be using) a prepackaged solution like GLSL. Regardless, at the moment only the engine and the scripting language are important.

This brings us to the title of the post then, a game scripting language, and it’s a strange one too. The problem is actually not as simple as just dropping in any kind of scripting language. No, the engine is intended to push Haskell’s concurrency potential to the limit (even on tomorrow’s hardware), but for this to happen, the scripting language must not become a bottleneck in the process. Sounds simple, right… but what happens when you build a game that scales big, really big even—beyond Supreme Commander big—and now it’s the scripting language that’s trying to handle unreasonable amounts of data (in addition to the engine). I don’t want to force game programmers to rewrite high level game code back in Haskell, simple to make it fast enough, that’s the entire purpose of the scripting language.

The solution then is the same as in the engine, make it massively concurrent, just like that Tim Sweeney talk was implying. But, a presumably object-oriented language, with extreme concurrency can only really mean one thing—it has to be semi-functional, and support either the process-oriented approach (Erlang), or the same Software Transaction Memory as Haskell below it—I chose the latter.

From Smalltalk, To Software Transactional Memory

Given the requirement for a Software Transactional Memory, it might seem obvious to simply write a variant of OCaml or Lua, but my experience with Smalltalk demanded otherwise. I believe that the best design pattern for implementing games is the prototype pattern. There are only really a few languages that make this pattern the global design pattern for the language, and in my opinion, JavaScript isn’t really the language of choice for writing semi-functional code in.

What I’ve done then is to start from a basic Smalltalk messaging syntax (that’s about the only real syntax in Smalltalk anyway) and then extend it to a full, text only syntax, removing all of the class and inheritance features and lastly, removing the main source of side effects, mutable variables. The result is a semi-functional object-oriented programming language, exclusively applying Software Transactional Memory for mutable object state, and making heavy use of prototypes and delegation.

There’s actually quite a lot, going into the design of Fire, oh yeah, that’s the name of this little programming language, and it would be quite pointless to go into great detail, at the end of an already lengthy post. What I can promise however, is a complete language specification, for those who are interested in the precise definition of the language and core libraries. And for everyone else, I should be following up this post later with essays covering some of the challenges faced in the unusual design requirements of The Fire Programming Language.

Anyway, that’s basically the status update, or the plan at least. The Fire Programming Language does actually have a complete syntax already too, and the first draft of said syntax chapter in the spec is also almost complete. Any question or reservations about the project are more than welcome, either in the comments or by email, and lastly… This is a long term project, one which I suspect may take anywhere up to year or more, before it hits a 1.0. What I do expect though, is some alphas and betas ready within 6 months or so and I’ll be setting up some git (public) repositories over the next couple of months for anyone who wants to muck about with it or give feedback (and the spec will slowly appear on a pages section of this blog).

– Lorenz

What I’m talking about exactly is game languages which allow the game objects themselves to participate in concurrent programming practices, without using intricate and complex locking sequences, or losing the right to talk directly to engine/library level services like physics or high-level game state (moving between the level and a menu for example).

Instead, almost all game level languages at the moment seem to be evangelizing the importance of being script level uncompiled beasts that require little to no real programming experience before becoming a productive member of the game code team!?

Of course I’m not saying that this is an inherently bad practice with the current level of technology floating around, but I do have to wonder whether this is still relevant for small team game development in the near multi-core parallelism future. Perhaps too far, but I think also relevant is the much slower pace of Open Source Game Development, which realistically speaking needs technology almost 3 years ahead of its time just to break even on an initial 1.0 release!

My question is actual reasonably simple despite the cynical tone up to this point… Can we realistically design a new object oriented DSL that contains the necessary concurrency techniques to allow programmers to write reliable and correct concurrent game code. This actually goes beyond just the engine code for which we’ve always been able to reasonably easily move to a more sophisticated language like Haskell but more towards the game specific stuff, which we cannot afford to bog down in complicated syntax and general side effect hostility.

I actually believe this is possible, and I’ve already begun planning the first steps towards writing some basic engine code that will underpin the project but I’m definitely more interested in seeing how effectively I can pin down the STM concurrency model into an object oriented syntax.

Ok… what about the language stuff…

I’ve actually been looking for a good excuse to both learn Haskell more fully (I’ve really only coded trivial examples in it so far), as well as properly exercise the Parsec combinator parser and finally, and perhaps most of all, design a new object model and object oriented programming language using that model.

The last, ironically may have to be restrained a little, but a concurrent perhaps slightly more conventional object oriented language may not be so bad. The point is that the project will involve a fairly comprehensive object oriented programming language, and one that is not only fully concurrency enabled but also tailored to game development tasks.

This is probably a good place to point out, that the conventional object oriented language I’m talking about here doesn’t include either C++, Java, Ruby or even the CLOS, but rather I’m talking specifically about the Smalltalk family of languages here. Its the work done on Smalltalk/Squeak, Self, Slate and more recently Newspeak that I find the most interesting, and the dialect I develop will likely incorporate many of the simpler innovations in the object systems they use.

The most critical issue that I’m aware of is the complexity added by the multi-dimensional objects used by most OO languages these days (including Smalltalk in this case). Rather than allowing the object state to exist in two (or more) dimensions I’ll be limiting them to a single flattened dimension similarly to record types, and the programmer will need to use either delegation/cooperation or composition in order to accomplish tasks in a reusable fashion. If you didn’t follow the dimension of state thing I’m actually just talking about Inheritance (another dimension might be Aspects, et cetera).

While I don’t want to go to much further into the syntax I would like to note that my intentions for this, apparently built in, concurrency will probably be in the form of an asynchronous send, paired with transaction support (which will actually be the only way to guarantee that data is written correctly). The asynchronous send, by definition, allows the receive to process it in a different thread… but since my little language is probably going to be more or less interpreted or at least green threaded, I’m pretty sure that I can just replace the async send operation as a plain old spawn operation.

The async message send in this case should provide a convenient way of signaling between game objects, and paired with the transactions (noting that very few Erlang processes don’t actively apply transactional semantics as well), I should have an effective way of implementing fully concurrent game code.

Well then… hopefully I don’t get killed between all of these projects and we’ll see something interesting come out of this stuff. Until then…

– Lorenz