The last Virtual JUG presentation session of 2015 was all about programming language development, both general purpose and domain specific languages or DSLs. We had the amazing duo of speakers that commit a large chunk of their time into the development on a language development framework, called Xtext. You might have heard of it, because it was behind the Xtend language that unconventionally compiles into Java rather than JVM bytecode.
First of all, a word about the speakers and why would you want to listen to them. Welcome, Sven Efftinge who works as consultant, coach and developer. He leads an open-source development lab in Kiel which focuses on language design, compiler development and Eclipse technologies. He is project lead of Xtend, a statically-typed programming language and Xtext, an award-winning framework for language development. You can find him on Twitter at @svenefftinge.
The second speaker, Dr. Miro Spönemann, is a software engineer and consultant for Itemis, also in Kiel, Germany. For five years or so he researched graph layout algorithms and their application to model-driven development. And now he is focused on domain-specific languages and their integration in web applications. Miro is also a core committer of Xtext.
The combination of Sven and Miro delivered a brilliant presentation on how general programming languages get developed, including a real demo showing how to build a small programming language, together with integrations with Java and the tooling support for the IDEs. All that in around 50 minutes! What a time to be alive!
Well, don’t give me the time to confuse you! You should just watch the session with your own eyes! The session replay is embedded below. If you prefer text as your medium of choice, keep reading this post, as I’ll explain what the session was about in more detail and recap what I learned from it. Maybe read it first and then dive head first into watching the presentation and demos.
Programming Language Development Made Easy
Let’s start at the beginning, you want to design a programming language? Do you think you need a general purpose language, like Java or Go or a DSL, which will shine in a particular domain?
A practical choice would, naturally, be the DSL: it might help real people, you might have a domain that is not yet covered by any specific programming language and you’ll have a much easier time maintaining it compared to the more abstract alternative.
So, what’s needed for a DSL? A language can contain the following parts that you, young padawan learners, might need to develop:
- lexer – to turn the strings of code into tokens that mean something to you
- parser – to understand the meaning of the tokens and construct an internal representation of them
- internal representation – to work with more suitable data structures later. An abstract syntax tree is a common choice
- code generator – to actually produce the code that will be executed by a machine
- interpreter – to execute the code without complex compilation procedures
- validators/verifiers – to ensure that the parts of the program won’t blow up the universe as we know it
- linker – to make multiple binaries play together
- type checker – if you want to have to use a type system, it is a good idea to verify the types of the data before using it
- type inference engine – to help people with the assignment of types and to help the engine to execute code faster
On top of that you’ll need to make your language usable, you’ll need to integrate with the existing tools like IDEs. Of course, IDE support only sounds simple! If you want to have anything better than a glorious notepad experience for your language you’ll need to add proper integration into the IDE. This might include tons of actions that we use every day without even thinking about them: syntax highlighting, content assist, validation, smart editing, toggling code blocks, finding references, goto declaration or implementation functionality, tooltips on hovers, and more, including:
Of course, this is just one aspect of the usability journey. Multiply this by the number of IDEs available, a good rule of thumb is that you should consider at least three IDEs if just looking at the JVM world.
Are you scared? No need to be! Welcome Xtext – the framework designed for language engineering. It lets you to define a language by its grammar, and it generates all the aspects needed for a language on your behalf. You just need to specify the semantics of the language and the infrastructure is taken care of.
The best part is that it uses a dependency injection mechanism to provide the default implementations of things like namespaces and scoping (how long your variables are accessible) and if you’re not happy with the default, you can easily change it yourself.
Your language even gets use of the Gradle and Maven integrations out of the box, so distributing it and incorporating it into the actual project should be a breeze. If that sounds fantastic, then you definitely have to listen to the session itself! Despite my best efforts in replicating the effect of watching the live demos, the original is always better.
Here’s a small example of a DSL that was developed during the session. The language talks about devices: window and heating, and a couple of rules that show how these rules can be transformed into a proper language.
Device Window can be OPEN, CLOSED Device Heating can be ON, OFF Rule "Close window when heating is turned on" when Heating.ON then fire(Window.CLOSE) Rule "Turn heating off when window is opened" when Window.OPEN fire(Heating.OFF)
All the code snippets and the result files are available on Github. Be sure to check them out as it will allow you to follow the session more effectively.
As part of the session, Sven and Miro showed how to start an Xtext project using Eclipse, where one can select which integrations are required and easily configure the base skeleton of the project.
After that the build system of your choice (if in doubt, pick Gradle) downloads the dependencies (maybe even the internet), so that you’ll have a project for the language ready. Now in the project you’ll have to specify the grammar of the language, the structure that will specify which words and identifiers can go in the source text in a specific order as well as the meaning of those.
During the session we used the following model, that states that there can be identifiable devices that can exist in certain states.
Model: declarations+=Declaration*; Declaration: Device | Rule; Device: 'Device' name=ID ('can' 'be' states+=State (',' states+=State)*)?; State: name=ID; Rule: 'Rule' description=STRING 'when' when=[State|QualifiedName] 'then' then=[State|QualifiedName]; QualifiedName: ID ('.' ID)*;
If the grammar syntax seems unfamiliar, watch the session, as Sven explains it in further details. In a nutshell, it defines the objects that we can deal with and what a state is for our devices as well as the syntax for the rules.
The best part of working with the Xtext framework is that it will automatically generate tests for your language, so you can test your domain model right out of the box. Watch the session to learn how Xtext generates the integrations with the frameworks for the target language, including a web view to access the language and specify its programs and all the necessary infrastructure to have a usable, practical language.
Here are a few links to the resources that you might find handy to browse during watching the presentation and for if you want to take it to the next level and try Xtext yourself.
- The snippets used for the demo on Github
- The main repository of Xtext
- Xtext website and docs
- Xtend – the JVM language build with Xtext
Now you’re fully equipped to create the programming language you always dreamed of!
After the session, both Miro and Sven joined us for a RebelLabs interview, where I had a chance to ask them about the current challenges of programming language development (not particularly related to the marvellous Xtext/Xtend combo), which hard problems, if any, we solved in language design and which tools are the most important for software engineering. Also, they answer if there’s any glory and fame left in the field of programming language design, or whether you have to go to academia to get a chance to develop your own language.
Watch for yourself, both Miro and Sven are incredibly smart and experienced. Learn from the best.