Xtext - DSL unleashed

Martin Reinhardt


  • Basics

  • Why should I use Xtext

  • Structuring your DSL

  • Code Generation

  • Testing

  • Continous Integration

  • Whats left?

  • Problems

  • Links


Xtext is a language development framework i.e., a technology supporting the activity of developing languages, so called Domain Specific Languages.

Xtext features

  • Or why to use Xtext?

Structuring your DSL

  • Why a DSL?
  • worthwhile if the language allows a particular type of problem, the language fits to solve a particular problem
  • Domain Specific Languages (DSL) have been hard sells in past
  • The most common objection is to the presumed effort required to create a DSL. Few have experience writing a language and those that do typically wrote languages some time ago.
  • With today’s tools though, writing a DSL is much easier, no need to build up on parser, lexer .. (know yacc/bison)
  • But how?

Designing a DSL with XText

  • Xtext delivers got defaults, but the more you want the more you pay
  • Uses ANTLr for lexer and parser generation (well proven, but check your grammar for left-recursion)
  • if you are fine with the (reasonable) defaults, your amount of work will be pretty low
  • otherwise, Xtext is highly configurable
  • Xtext is based on Ecore, so Knowledge of Ecore is required to exploit Xtext’s full potential
  • EMF is “a modeling framework and code generation facility for building tools and other applications based on a structured data model”

XText Grammar

  • grammar language (XGL) describe (textual) languages, looks a bit like EBNF
    • the syntax of the target language
    • structure of the target AST
      terminal NAME : expression ;
  • expression can contain
    • keywords
      EX: 'foo', "\foo", '"', "'"
    • wildcard
      EX: . 
    • rule calls (can only point to other terminal rules)
      EX: ID 
    • character ranges
      EX: 'a'..'z', 'A'..'Z', '0'..'9'
    • until token (useful for comments)
      EX: '/*' -> '*/'
    • negated token
      EX: !'a'		
    • cardinality operators (?, *, + or nothing)
      EX: '^'?, '\n'*, 'a'+
    • groups (token sequences)
      EX: 'a' . ID 
    • alternatives
      EX: ' ' | '\t' | '\r' | '\n' 
  • Terminal rules can hide each other, take care of the order !
  • Especially import when mixing grammars!
  • Parser Rules
    • keywords
    • rule calls
    • groups
    • cardinality operators
    • character ranges
    • alternatives
    • unordered groups (Elements can appear in any order but only once, Elements with cardinality * or + must appear continuously without interruption )
      EX: 'a' & ID*
  • You have to know your AST before defining the textual representation
  • Can use mindmap to structure your grammar before writing Xtext grammar
  • Start with less functionality and add feature by feature
    • Keep in mind besides the grammar there are the code generation, syntax highlighting ...
    • Smaller Steps are easier to cover all components
    • Also think about grammar splitting to split up to work between team members
  • Too much theory?
  • Let's go a bit more practical with an example
grammar org.example.domainmodel.Domainmodel with
generate domainmodel "http://www.example.org/domainmodel/Domainmodel"
Domainmodel :
  elements += Type*
  DataType | Entity
  'datatype' name = ID
  'entity' name = ID ('extends' superType = [Entity])? '{'
     features += Feature*
  many?='many'? name = ID ':' type = [Type]
  • Not that good practical sample?
  • How would I describe an UI with Xtext?
'webapp' (name=ID)? '{'		
	('actions' definedGlobalAction+=ActionElement)*
'page' name=ID ('path' path=STRING)?	'{'
	('input' input+=InputElement (','input+=InputElement)*)?
	('view' view+=ViewElement (','view+=ViewElement)*)?	
	('action' action+=ActionElement (','action+=ActionElement)*)?
'global' '{'
	('loaderElement' loader=Loader)?		
	('input' definedGlobalInputs+=InputElement (','definedGlobalInputs+=InputElement)*)?
	('view' definedGlobalViews+=ViewElement (','definedGlobalViews+=ViewElement)*)?	
	('action' definedGlobalActions+=ActionElement (','definedGlobalActions+=ActionElement)*)?
Loader: view=ViewElement;

ViewElement: type=ViewType name=ID 'selector =' selector=Selector ;

enum ViewType: ELEMENT ='element';

ActionElement: type=ActionType name=ID 'selector =' selector=Selector ;
enum ActionType: CLICK = 'click' ;

terminal ID: '^'?('a'..'z'|'A'..'Z'|'_'|':') ('a'..'z'|'-'|'A'..'Z'|'_'|'0'..'9'|':')*;

terminal INT returns ecore::EInt: ('-')?('0'..'9')+;
  • Let's start with a demo

Code generation

  • Xtend is used to generate code within Xtext, compiles to Java
  • Via JvmTypeReference
    • Good usage to generate Java classes, more robust when refactoring grammar, e.g. type safety
    • Litte bit more complex and doesn't support other languages, e.g. for generating text files
  • Via IGenerator
    • Litte easier to learn
    • No type safety
    • Generic approach, e.g. for generating text files

Good Practise

  • Split up generator via extension in Xtend
    class MyGenerator implements IGenerator {
    	//add extension 
    	@Inject extension anotherGenerator
  • As in Java add debug information, can use normal Java Logger
  • Use Xtend Generator for text generation (much easier as in Java)
  • Use Helper methods for common naming etc.

Example - IGenerator

  • Split up generator via extension in Xtend
    class MyDslGenerator implements IGenerator {
     override void doGenerate(Resource resource, IFileSystemAccess fsa) {
      fsa.generateFile(resource.className+".java", toJavaCode(resource.contents.head as Statemachine))
     def className(Resource res) {
      var name = res.URI.lastSegment
      name.substring(0, name.indexOf('.'))
     def toJavaCode(Statemachine sm) '''
     import java.io.BufferedReader;
     import java.io.IOException;
     import java.io.InputStreamReader;
     public class «sm.eResource.className» {
      public static void main(String[] args) {
       new «sm.eResource.className»().run();

Example - JvmTypeReference

  • Split up generator via extension in Xtend
     def dispatch void infer(Statemachine stm, 
                             IJvmDeclaredTypeAcceptor acceptor, boolean isPreIndexingPhase) {
       // create exactly one Java class per state machine
       acceptor.accept(stm.toClass(stm.className)).initializeLater [
         // add a field for each service annotated with @Inject
         members += stm.services.map[service|
           service.toField(service.name, service.type) [
             annotations += service.toAnnotation(typeof(Inject))
         // generate a method for each state having an action block
         members += stm.states.filter[action!=null].map[state|
           state.toMethod('do'+state.name.toFirstUpper, state.newTypeRef(Void::TYPE)) [
             visibility = PROTECTED
             // Associate the expression with the body of this method.
             body = state.action
         // generate a method containing the actual state machine code
         members += stm.toMethod("run", newTypeRef(Void::TYPE)) [
           // the run method has one parameter : an event source of type Provider 
           val eventProvider = stm.newTypeRef(typeof(Provider), stm.newTypeRef(typeof(String)))
           parameters += stm.toParameter("eventSource", eventProvider)
           // generate the body
           body = [append('''
             boolean executeActions = true;

Testing with Xtext

  • Native support for JUnit
  • Most of the tests can run headless
  • Can use Xtext Junit helper classes, e.g. for parametrized tests
  • integration test get really complex
  • But to to test the editor you should use tools like SWT-Bot

Testing examples

public class WebAppSetupModelTest {

  ParseHelper parseHelper;

   * Empty models should result in a null model
  public void testEmptyModel() throws Exception {
    StringBuilder modelString = new StringBuilder();
    WebAppSetup model = parseHelper.parse(modelString);
    assertNull("Model is not empty", model);


  public void testModelAccept() throws Exception {
    StringBuilder modelString = new StringBuilder();
    modelString.append("webapp ea { ");
    modelString.append("    global {");
    modelString.append("        loaderElement element loader selector = #ajax-loader");
    modelString.append("        action click next selector = #gotoNext");
    modelString.append("    }");
    modelString.append("    page login path \"/faces/pages/login/login.xhtml\" {");
    modelString.append("    input type user  selector = #login_input,");
    modelString.append("    type password selector = #password  ");
    modelString.append("    action click login selector = #login_submit  ");
    modelString.append("    }");
    WebAppSetup model = parseHelper.parse(modelString);
    assertNotNull("Model is null. ", model);
    assertTrue("Expected 'loader' as name for global loader element, but was "
        + model.getGlobal().getLoader().getView().getName(),
    assertTrue("Expected only one page, but was " + model.getPages().size(), model.getPages().size() == 1);



Future of testing with Xtext

Continous Integration

  • How build Eclipse on a continous integration plattform?
    By using Maven/Tycho
  • the whole eclipse plugin stuff gives you lot of maven projects, e.g. for the folder structure

Steps to successfull Maven build

  • Move projects to the appropiate folders
  • Add Parent-, Repository and Target-Plattform Maven-Projects
  • Create Feature Plugin with in eclipse
  • Create Product Plugin with in eclipse
  • More details on the wiki

Setting up a RCP product for a Xtext DSL

  • Create a Plugin Project for your product (PlugIn-Project in Eclipse)
  • Create a Feature Project for your DSL (Feature-Project in Eclipse)
  • Create a Feature Project for your DSL-SDK (Feature-Project in Eclipse)
  • Create a Feature Project for your DSL-Tests (Feature-Project in Eclipse)
  • Create the Product Configuration (see first step and add the needed plugins)
  • What about an example?

Configure your build Job

  • Build repository project in pre-buildstep
  • Add Maven buildstep for the parent-pom and just do a 'mvn clean install'
  • See sample job on CloudBees

Further topics

  • Formatting (Syntax-Highlighting, Indentention), Quickfixes, Refactoring ...
  • Learn about Developing Eclipse Plugins
  • Also keep in mind to setup a project wizard for the final plugin
  • Xtext isn't the only DSL-Toolset, see Groovy, Spoofax ...


  • Xtext allows to really start fast with DSL-development
  • but the more you want ...
  • The further configuration of the Editor
  • Maven/Tycho build, no real support for dependency management
  • testing support could be improved



BY Martin Reinhardt