Structuring your DSL

• Why a DSL?
• worthwhile if the language allows a particular type of problem, the language fits to solve a particular problem

• Domain Specific Languages (DSL) have been hard sells in past
• The most common objection is to the presumed effort required to create a DSL. Few have experience writing a language and those that do typically wrote languages some time ago.
With today’s tools though, writing a DSL is much easier, no need to build up on parser, lexer .. (know yacc/bison)
But how?

Designing a DSL with XText

• Xtext delivers got defaults, but the more you want the more you pay
• Uses ANTLr for lexer and parser generation (well proven, but check your grammar for left-recursion)
• if you are fine with the (reasonable) defaults, your amount of work will be pretty low
• otherwise, Xtext is highly configurable
• Xtext is based on Ecore, so Knowledge of Ecore is required to exploit Xtext’s full potential
• EMF is “a modeling framework and code generation facility for building tools and other applications based on a structured data model”

XText Grammar

• grammar language (XGL) describe (textual) languages, looks a bit like EBNF
• the syntax of the target language
• structure of the target AST

  terminal NAME : expression ;

• expression can contain
• keywords

  EX: 'foo', "\foo", '"', "'"

• wildcard

  EX: . 

• rule calls (can only point to other terminal rules)

  EX: ID 

• character ranges

  EX: 'a'..'z', 'A'..'Z', '0'..'9'

• until token (useful for comments)

  EX: '/*' -> '*/'

• negated token

  EX: !'a'		
   

• cardinality operators (?, *, + or nothing)

  EX: '^'?, '\n'*, 'a'+
   

• groups (token sequences)

  EX: 'a' . ID 

• alternatives

  EX: ' ' | '\t' | '\r' | '\n' 

• Terminal rules can hide each other, take care of the order !
• Especially import when mixing grammars!

• Parser Rules
• keywords
• rule calls
• groups
• cardinality operators
• character ranges
• alternatives
• unordered groups (Elements can appear in any order but only once, Elements with cardinality * or + must appear continuously without interruption )

  EX: 'a' & ID*

• You have to know your AST before defining the textual representation
• Can use mindmap to structure your grammar before writing Xtext grammar
• Start with less functionality and add feature by feature
• Keep in mind besides the grammar there are the code generation, syntax highlighting ...
• Smaller Steps are easier to cover all components
• Also think about grammar splitting to split up to work between team members
Too much theory?
Let's go a bit more practical with an example

grammar org.example.domainmodel.Domainmodel with
                                      org.eclipse.xtext.common.Terminals
 
generate domainmodel "http://www.example.org/domainmodel/Domainmodel"
 
Domainmodel :
  elements += Type*
;
  
Type:
  DataType | Entity
;
  
DataType:
  'datatype' name = ID
;
 
Entity:
  'entity' name = ID ('extends' superType = [Entity])? '{'
     features += Feature*
  '}'
;
 
Feature:
  many?='many'? name = ID ':' type = [Type]
;

Not that good practical sample?

How would I describe an UI with Xtext?

...
WebAppSetup:
'webapp' (name=ID)? '{'		
	(global=Global)?
	(pages+=Page)+ 
	('actions' definedGlobalAction+=ActionElement)*
'}';
	
Page:
'page' name=ID ('path' path=STRING)?	'{'
	('input' input+=InputElement (','input+=InputElement)*)?
	('view' view+=ViewElement (','view+=ViewElement)*)?	
	('action' action+=ActionElement (','action+=ActionElement)*)?
'}';
	
Global:
'global' '{'
	('loaderElement' loader=Loader)?		
	('input' definedGlobalInputs+=InputElement (','definedGlobalInputs+=InputElement)*)?
	('view' definedGlobalViews+=ViewElement (','definedGlobalViews+=ViewElement)*)?	
	('action' definedGlobalActions+=ActionElement (','definedGlobalActions+=ActionElement)*)?
'}';
		
Loader: view=ViewElement;

ViewElement: type=ViewType name=ID 'selector =' selector=Selector ;

enum ViewType: ELEMENT ='element';

ActionElement: type=ActionType name=ID 'selector =' selector=Selector ;
			
enum ActionType: CLICK = 'click' ;

terminal ID: '^'?('a'..'z'|'A'..'Z'|'_'|':') ('a'..'z'|'-'|'A'..'Z'|'_'|'0'..'9'|':')*;

terminal INT returns ecore::EInt: ('-')?('0'..'9')+;
...	

Code generation

• Xtend is used to generate code within Xtext, compiles to Java
Via JvmTypeReference
• Good usage to generate Java classes, more robust when refactoring grammar, e.g. type safety
• Litte bit more complex and doesn't support other languages, e.g. for generating text files
Via IGenerator
• Litte easier to learn
• No type safety
• Generic approach, e.g. for generating text files

Testing with Xtext

• Native support for JUnit
• Most of the tests can run headless
• Can use Xtext Junit helper classes, e.g. for parametrized tests
• integration test get really complex
• But to to test the editor you should use tools like SWT-Bot

Testing examples

@RunWith(XtextRunner.class)
@InjectWith(AppSetupInjectorProvider.class)
public class WebAppSetupModelTest {

  @Inject
  ParseHelper parseHelper;

  /**
   * Empty models should result in a null model
   */
  @Test
  public void testEmptyModel() throws Exception {
    StringBuilder modelString = new StringBuilder();
    modelString.append("");
    WebAppSetup model = parseHelper.parse(modelString);
    assertNull("Model is not empty", model);

  }

  @Test
  public void testModelAccept() throws Exception {
    StringBuilder modelString = new StringBuilder();
    modelString.append("webapp ea { ");
    modelString.append("    global {");
    modelString.append("        loaderElement element loader selector = #ajax-loader");
    modelString.append("        action click next selector = #gotoNext");
    modelString.append("    }");
    modelString.append("");
    modelString.append("    page login path \"/faces/pages/login/login.xhtml\" {");
    modelString.append("    input type user  selector = #login_input,");
    modelString.append("    type password selector = #password  ");
    modelString.append("    action click login selector = #login_submit  ");
    modelString.append("    }");
    modelString.append("}");
    WebAppSetup model = parseHelper.parse(modelString);
    assertNotNull("Model is null. ", model);
    assertTrue("Expected 'loader' as name for global loader element, but was "
        + model.getGlobal().getLoader().getView().getName(),
        model.getGlobal().getLoader().getView().getName().equalsIgnoreCase("loader"));
    assertTrue("Expected only one page, but was " + model.getPages().size(), model.getPages().size() == 1);

  }

}

Future of testing with Xtext

• better support for file handling for comparing generator artifacts
• More tools for easier unittesting

Continous Integration

• How build Eclipse on a continous integration plattform?
  By using Maven/Tycho
the whole eclipse plugin stuff gives you lot of maven projects, e.g. for the folder structure

docs
examples
 ->sample
features
 ->org.xtext.feature
releng
 ->org.xtext.parent
 ->org.xtext.product
 ->org.xtext.repository
 ->org.xtext.repository.parent
 ->org.xtext.targetplattform
 ->org.xtext.updatesite
plugins
 ->org.xtext.mysql1
 ->org.xtext.mydsl1.ui
 ->org.xtext.mysql2
 ->org.xtext.mydsl2.ui
tests
 ->org.xtext.mydsl1.tests
 ->org.xtext.mydsl2.tests

Steps to successfull Maven build

• Move projects to the appropiate folders
• Add Parent-, Repository and Target-Plattform Maven-Projects
• Create Feature Plugin with in eclipse
• Create Product Plugin with in eclipse
• More details on the wiki

Setting up a RCP product for a Xtext DSL

• Create a Plugin Project for your product (PlugIn-Project in Eclipse)
• Create a Feature Project for your DSL (Feature-Project in Eclipse)
• Create a Feature Project for your DSL-SDK (Feature-Project in Eclipse)
• Create a Feature Project for your DSL-Tests (Feature-Project in Eclipse)
• Create the Product Configuration (see first step and add the needed plugins)
What about an example?

Configure your build Job

• Build repository project in pre-buildstep
• Add Maven buildstep for the parent-pom and just do a 'mvn clean install'
• See sample job on CloudBees