Neverlang 2

Language Workbench


Questions and issues to Walter Cazzola.

A brief introduction to Neverlang

Domain Specific Languages (DSLs) are used to solve several problems, such as typesetting documents and code (TeX/LaTeX, lout, ...), to express and verify constraints in several domains (OCL, iLOG CP, C4J, ...) and to coordinate the computation (Linda, ...) and/or to data query (SQL, ...). In some cases, these are simply a bunch of programming features useless standalone embedded in a general purpose programming language or provided as external libraries (e.g., Linda and SQL). In these case performances and flexibility are often compromised especially when the DSL is realized as program transformation towards another high-level programming language.

A DSL integrating features from different programming languages and paradigms would be the best choice to express a concise and clean solution to a problem avoiding a cumbersome one due to the absence of a specific feature or to a farraginous one because you are using a general-purpose instead of a domain specific programming language. Unfortunately, to develop a domain specific or a general purpose programming language from scratch implies a considerable effort, a lot of time and skills that not always you can invest in a project.

To simplify and speed up the development of domain specific and general purpose programming languages we designed and developed the Neverlang language workbench. The Neverlang approach basically reflects the fact that programming languages have an intrinsically modular decomposition in language features that are implemented by language components that can be easily plugged and unplugged. A complete compiler/interpreter built up with Neverlang is the result of a compositional process involving several building blocks, i.e., the language components.

In this scenario, to design and implement a domain specific language just consists of composing existing―i.e., reusing―language features and components to automatically get all the needed support tooling—compiler, interpreter, debugger, IDE, etc. Each language component―dubbed as slice in the Neverlang parlance―embodies a single language concept—dubbed as language feature—and provides all the necessary code to support it such as its syntax, how it can be type checked, interpreted, translated to assembly code, debugged, how it should be supported by an IDE, etc. The whole structure of the generated tooling is the result of the composition of such a slices, in particular of the code necessary to compile/interpret each single feature.

How does it work?

Neverlang basically provides: a domain specific language for writing the building blocks and a mechanism for composing these blocks together and for generating the corresponding tooling.

The basic units composing a programming language developed by using Neverlang are modules. Each module encapsulates a specific feature of the language, e.g., a module can encapsulate the syntactically aspect of a loop, the type checking code of a comparison, or the code generation for a method call. Roles define how the composed modules form the compiler/interpreter. syntax, type-checking and evaluation are examples of roles, i.e., apart syntax all the other can be changed and there is no limit in the number of roles. Finally, modules regarding the same language structure but with different roles are grouped together in slices.

Let's see these concepts on a small example: a calculator for integers with only the + operator. The following code shows the implementation of the core of the DSL: the language feature (both module and slice) for the + operator. The final DSL is so simple that we only need a role for the evaluation of the operator.

module addlang.v1.PlusOpModule {
    reference syntax {
        add: AddExpression ← Integer "+" Expression;
    }
    
    role (evaluation) {
        add: .{
            $add.value = (Integer) $add[1].value + (Integer) $add[2].value;
        }.
    }
}

The module addlang.v1.PlusOpModule defines the syntax for the + operator and how it should be evaluated. The syntax is given by a rule of an attributed grammar; it defines the non-terminal AddExpression and use the non-terminals Expression that will be provided by other slices/modules.

Each role (in the example only the role evaluation) adds up on a reference syntax by adding semantics action to the non-terminals. Each role implements a visit on the program's parse tree applying at each node the corresponding semantic action.

The non-terminals are identified through their position in the rules numbering with 0 the top leftmost non-terminal and incrementing by one left-to-right and up-to-down all the non-terminals independently of repetitions and for the whole set of productions defined in the slice. As an alternative, a label can be associated to each rule and used to refer the non-terminals locally to a single rule. The heading of the action just figures out where the semantic action is anchored in the productions.

In the example, we have one semantic action. This enriches the head of the first production (labeled with add) and simply sums the value of the operands (stored in the corresponding non-terminal value attribute) and stores the result in the value attribute of the AddExpression non-terminal. Notes that this implementation rely on the fact that the Expression non-terminal defines a value attribute otherwise the composition will fail.

slice addlang.v1.PlusOpSlice {
  concrete syntax from addlang.v1.PlusOpModule
  module addlang.v1.PlusOpModule with role evaluation
}

These modules are composed in a slice, where we have to define which role is implemented by each imported module. In this example the slice could be omitted but it also shows a second dimension of composition that can be used, for example, to change either the semantics or the syntax of the operator and reusing the remaining part.

To complete the example we need some language features for the integers and for the expressions, whose non-terminal are used by the + but still dangling.

module addlang.v1.IntModule {
    reference syntax {
        int: Integer ← /[0-9]+/ ;
    }
    role (evaluation) {
        int: .{
            $int.value = Integer.parseInt(#0.text);
        }.
    }
}

The above module defines the integers through the regular expression (regex) /[0-9]+/. Note that, in Neverlang, a regex is equivalent to a terminal of the language even if it, de facto, defines several different terminals (all integers in this case). In the semantic action, we retrieve the current value for the regex the predefined attribute text associated to the terminal in position 0 (#0 refers to the regex); text is filled by the lexer with the lexeme matched in the input.

module addlang.v1.Expression {
    reference syntax {
        expression: Expression ← AddExpression;
        constant:   Expression ← Integer;
    }

    role (evaluation) {
        expression: .{
            $expression.value = (Integer) $expression[1].value;
        }.

        constant: .{
            $constant.value = (Integer) $constant[1].value;
        }.
    }
}

The addlang.v1.Expression is what we could call glue code. It just makes sure that both integers and the applications of the + operator are expressions and propagates the attribute values.

module addlang.v1.Program {
    reference syntax {
        axiom:   Program ← Expression;
    }

    role (evaluation) {
        axiom: .{
            System.out.println((Integer)$axiom[1].value);
        }.
    }
}

In Neverlang, the axiom for the grammar is always named Program. In this DSL a program is just a chain of sums of integers and the expected result is its evaluation. The semantic action associated to the axiom just prints the value of the calculated expression; this is possible because, by default, the parse tree visit is post order.

Now, we have all the language components and we have just to tell Neverlang how to compose them in a language.

language addlang.v1.AddLang {
    slices
        addlang.v1.Program
        addlang.v1.PlusOpSlice
        addlang.v1.IntModule
        addlang.v1.Expression
    roles syntax < evaluation

    left addlang.v1.PlusOpSlice
}

In the language construct we give a name to the language (addlang.v1.AddLang), list the full set of slices/modules (in the slices section) that should be composed in the language and which roles we have and in which order they should be applied (in the roles section).

Ok, our first DSL is ready. For your convenience you can download it from here

Neverlang Features

The Neverlang language workbench is though to enhance the support for variability and reuse of the language features. Its peculiar features are:

Getting started with Neverlang

Download the Neverlang package from the Resources section.

Note, Neverlang is developed under Linux, it runs under MacOS and since it runs on the JVM it should work without problems in Windows too. Anyway the above instructions are for a Linux system with bash installed.

Note Since version 1.2 Neverlang runs on top of Java 13.

Let's see how to use it with the AddLang example.

Resources

Here you can find some extra resource (more to come):

Neverlang Staff

The Neverlang2 project is led by Walter Cazzola.

References
  1. Walter Cazzola and Edoardo Vacchi, “On the Incremental Growth and Shrinkage of LR Goto-Graphs”, ACTA Informatica, vol. 51, no. 7, pp. 419–447, October 2014. [ACTA14]
  2. Edoardo Vacchi and Walter Cazzola, “Neverlang: A Framework for Feature-Oriented Language Development”, Computer Languages, Systems & Structures, vol. 43, no. 3, pp. 1–40, October 2015. [COMLAN15]
  3. Walter Cazzola and Albert Shaqiri, “Context-Aware Software Variability through Adaptable Interpreters”, IEEE Software, vol. 34, no. 6, pp. 83–88, November 2017, Special Issue on Context Variability Modeling. [IEEESW17]
  4. Walter Cazzola and Albert Shaqiri, “Open Programming Language Interpreters”, The Art, Science, and Engineering of Programming Journal, vol. 1, no. 2, pp. 5–1–5–34, April 2017. [PROG17]
  5. Walter Cazzola, Ruzanna Chitchyan, Awais Rashid, and Albert Shaqiri, “μ-DSU: A Micro-Language Based Approach to Dynamic Software Updating”, Computer Languages, Systems & Structures, vol. 51, pp. 71–89, January 2018. [COMLAN18]
  6. Walter Cazzola and Diego Mathias Olivares, “Gradually Learning Programming Supported by a Growable Programming Language”, IEEE Transactions on Emerging Topics in Computing, vol. 4, no. 3, pp. 404–415, September 2016. [TETC16]
  7. Thomas Kühn, Walter Cazzola, Nicola Pirritano Giampietro, and Massimiliano Poggi, “Piggyback IDE Support for Language Product Lines”, in Proceedings of the 23rd International Software Product Line Conference (SPLC'19), Paris, France, September 2019, ACM. [SPLC19]

Walter Cazzola

Didactics

Publications

Funded Projects

Research Projects

Related Events