One of the new cool features which came with Xtext 2.3.0 is a wizard which generates all the artifacts to build your Xtext DSL project with Buckminster; this will allow you to easily build your Xtext project p2 site and also to build your project and its tests headlessly (e.g., from the command line, and, more importantly from within a continuous integration system like Jenkins).

This new feature was not advertised much, nor documented, so I decided to write a tutorial about that, also with the permission of the developer of this feature Dennis Hübner.

In this tutorial I will show how to

• create the p2 repository from the IDE
• build your project (and run the tests) headlessly from the command line (through ant)
• build your project (and run the tests) headlessly in a continuous integration system like Jenkins

In the end, I will also try to describe/explain all the Buckminster files that the wizard created for you, so that one can customize them.

The sources used in this tutorial can be found at

https://github.com/LorenzoBettini/Xtext-Buckminster-Example

Create an Xtext project

So, first of all, let’s create a simple Xtext hello project; for this example I will use the following settings:

• project name: org.xtext.example.hellobuck
• name: org.xtext.example.hellobuck.HelloBuck
• extension: greetings
• check “Create SDK feature project”

and then, of course, we generate Xtext artifacts.

The DSL is basically the standard Greetings DSL; in this case, I’m using the JvmModelInferrer to generate Java classes from “Hello” specifications. Indeed, the DSL itself is not important in this tutorial. I’ve also added two Junit tests in the corresponding org.xtext.example.hellobuck.tests project:

Note the org.xtext.example.hellobuck.tests.launch Launch configuration that Xtext created for you (which basically runs all the Junit tests in this project. We will see how this will be used later.

Use the Xtext Build with Buckminster Wizard

Now we can use the Xtext wizard “Build with Buckminster”: New => Other => Xtext => Continuous Integration => Build with Buckminster:

You will have to specify the sdk feature of your Xtext project (that has been created by the Xtext new project wizard); the releng and site projects will have predefined names. For the Buckminster installation directory you have 3 choices:

• if you have already an headless installation of Buckminster in your system you can specify the path;
• you can install it using the link in the dialog
• you can leave that path empty and specify it later in the build.ant generated file (or pass it on the command line when invoking ant)

In any case, specifying the headless Buckminster installation path is only useful if you intend to build your projects headlessly with ant (it is not requested for building in the IDE, nor if you plan to build it with Jenkins by setting job using the Buckminster Jenkins plugin, as we will see later).

Before pressing Finish, we also want to specify the tests to launch during the headless build, so we press the button Add and we select the launch configuration in the .tests project which the Xtext wizard created for us.

If you want to build an existing Xtext project and you do not have that launch configuration, all you need to do is to simply create a new plain Junit launch configuration from the IDE, for instance, to run your existing Junit test suite, and save it in the .tests project yourself. You can also configure the headless build to run tests later on.

When the wizard finishes, you will end up with two additional projects

• org.xtext.example.hellobuck.buckminster with all the Buckminster (CQUERY and RMAP, build.ant, etc.) files to build your project headlessly;
• org.xtext.example.hellobuck.site which is a feature project to create the p2 repository for your Xtext project.

Build the p2 repository from the IDE

Although the .buckminster project created by the wizard is thought to be used in an headless environment, the .site project is useful also to build the p2 repository from the IDE itself! Indeed, if you created an old style update site, you could build the update site from the IDE itself. But if you switched to the new category.xml format, then building the corresponding p2 repository from the IDE is not straightforward.

If you want to run Buckminster from Eclipse, you first need to install it in the IDE of course, by using this repository: http://download.eclipse.org/tools/buckminster/updates-4.2

Before creating the p2 repository, you may want to tweak some Buckminster properties, for instance, you may want to create a buckminster.properties file in the .buckminster project (or in the .site project) like the following

so that Buckminster will generate all its artifacts in the tmp/hellobuck directory of your home folder.

We now right click on the .site project, and select Buckminster => Invoke Action… , we select site.p2 action, and optionally refer to the buckminster.properties file

When the action finishes, you will file the p2.site in the directory $HOME/tmp/hellobuck/build/org.xtext.example.hellobuck.site_1.0.0-eclipse.feature/site.p2/

Build with ant

The wizard created for us, in the .buckminster project a build.ant file that we can use to build the project, run the tests, and create the p2 repository with ant

Note that this file has the location of your Buckminster headless installation path hardcoded (recall the wizard we used before for creating Buckminster projects); if that is not correct, you can still pass this information to ant when invoking it

This ant script will build your project in the directory buildroot which will be created in the directory where all your projects are located (e.g., in the workspace or in another path); in particular it will

1. materialize the target platform for your project (the first time you run it), and this requires an Internet connection and might take some time
2. build your projects
3. run the Junit tests
4. build the p2 site repository

This should be the output (where WORKSPACE is actually the location of your projects):

It will also tell you where you can find the generated p2 site.

IMPORTANT: there’s a bug in the currently generated build.ant file, which prevent you from building with ant in Windows; until the fixed version of the wizard is released, you need to make sure that the build.ant has the right contents, as in this example source.

Build with Jenkins

To build your Xtext project in Jenkins you can either create a Job which uses ant and the build.ant script, or create a Job which uses the Buckminster Jenkins plugin (I’ve also blogged about that). I will detail both ways. Both jobs have in common the access to the git repository

A Job using ant

This requires that you configured an ant installation in your Jenkins, and that you already have a working version of Buckminster headless in Jenkins; you need to specify them in the ant build step configuration (the screeshots, of course, refer to the paths and values for my Jenkins installation):

then, we configure post build actions to archive both the artifacts (the p2 repository) and the Junit results:

A job using Buckminster plugin

To configure a Buckminster build step, we prepare a text file in the .buckminster projects with the Buckminster commands to execute (we take inspiration from the commands.txt file that the wizard created), we call it jenkins-commands.txt:

Then we configure a build step (this relies on a Buckminster installation that we have already configured in Jenkins), note that we refer to the jenkins-commands.txt we created above (an alternative would be to copy the commands directly in the text area “Commands”):

To configure the post build actions, since we used different output paths, we have to specify them accordingly (in particular, we have not used the output directory buildroot, thus the default will be used):

Details and Customization

It may be interesting to learn more about the files that the Xtext Buckminster wizard generated for you (at least, I personally found instructive to look at them, and learned something about Buckminster 🙂 most of which I described in another post of this blog). The details might also help you if you need to customize the generated files.

The .buckminster project contains a Component Specification (CSPEC), buckminster.cspec, which looks like

thus, due to its dependencies, it represents the target platform for your Xtext project. If your Xtext project required some further specific dependencies to build and test, this is the place to express them!

For instance, in an Xtext project of mine, Xsemantics, I run also some SwtBot tests, which also rely on pde, thus, the dependencies in the corresponding .buckminster project’s buckminster.cspec look like

To materialize components and dependencies with Buckminster, you need a Resource Map (RMAP), and the wizard created two maps: one containing p2 repositories location (for materializing the features of your target platform), called projects-platform.rmap and one for binding your projects to the workspace (when building headlessly), called project.rmap.

projects-platform.rmap looks like

This basically tells Buckminster to take all Xtext stuff (including additional components like Google Guice and Antlr) from the Xtext main repository, and to use the main Eclipse Juno release repository for everything else. However, it also redirects to project.rmap for resolving the main Buckminster component of our project org.xtext.example.hellobuck.buckminster.

Again, if your target platform needs additional features from different sites, this is the place where to express them; reusing the example above which also needed SwtBot the rmap would also have these additional stuff:

The project.rmap tells Buckminster where to find the projects of our Xtext project (so that it can bind it to the workspace when building headlessly):

This rmap assumes that all the Eclipse projects of your Xtext project are in the same base directory; if this is not the case, because you split them, for instance, in plugins, features, doc, etc. directories, then you need to tweak this rmap accordingly, for instance, again, taken from Xsemantics,

Then, you have to Component Query files (CQUERY): projects-platform.cquery, to materialize the target platform, and project.cquery, to materialize your projects in the workspace. The first one could also be used in the IDE to actually materialize the target platform in your workspace (instead of using a target definition file); the second one is useful when building headless.

projects-platform.cquery:

Note that this cquery has an advisor node to skip .source components: thus, the materialized target platform will not contain the sources for all the features. This is useful headlessly to reduce the time to materialize the target platform. But if you use it also for materializing the target platform in the IDE, then sources are useful to inspect Java classes, and you might want to remove this advisor node (or simply create another cquery only to be used in the IDE, without that advisor node). This query refers to projects-platform.rmap.

project.cquery:

As it usually happens when using Buckminster, the cquery for materializing the projects in the (headless) workspace refers to a single feature project which, transitively refers to all the bundles and additional features of your application. Usually, this is the feature project for building the p2 repository, like in this case.

“But wait! the site feature project does not refer to test projects, since I do not want to include them in my p2 repository! So, how can Buckminster run my tests headlessly if they are not materialized?”

If you look at the feature project .site, you will note that it contains a Component Extension (CSPEX) which allows to extend the automatically inferred Component Specification for the feature project with additional specifications. In this case the buckminster.cspex contains

You see that this specification adds a dependency to the org.xtext.example.hellobuck.tests project, so that, when Buckminster materializes org.xtext.example.hellobuck.site, and its dependencies, it will also materialize org.xtext.example.hellobuck.tests. You can use this technique to materialize additional projects; for instance, in from Xsemantics, I have a main feature for tests, and a tests project for each example, thus buckminster.cspex reads as follows

Finally, in the .site project, you will also find the feature.xml

which includes the .sdk feature of your project (Remember: when building the site.p2 on a feature project with Buckminster, you will build a p2 repository NOT for the very feature, but for the included features), and the category.xml to give categories to your features in the p2 repository:

Note that the .sdk.source feature is not a “real” feature project in your workspace, but, by default, Buckminster will automatically build a source jar for all your features and bundles.

That’s all! Many thanks to the author of this wizard, Dennis Hübner, for making it and for many suggestions!

Have fun with Bucky and Xtext 🙂