TL; DR: Xtext sources are now in a single Git monorepo (https://github.com/eclipse/xtext), and the build infrastructure is based entirely on Maven/Tycho (Gradle is not used anymore).
Background
A few years ago, Xtext sources were split into 6 separate GitHub repositories. I did not take part in that decision (I guess at that time, I wasn’t even a committer).
I think that at that time, the splitting was carefully thought out, aiming at making contributions and maintenance easier. In fact, while Xtext is mainly based on Eclipse, it has several core parts that are independent of Eclipse (including LSP support and plain Maven/Gradle projects support, without the Eclipse UI).
Those core parts were built with Gradle. I’m not a fan of Gradle: I’ve always found Gradle gives you too much power as a build tool, not to mention that, typically, the build breaks with new releases of Gradle. On the contrary, Maven has been around for several years and is rock solid. It’s rigid, so you have to obey its rules. But if you do, everything works.
Moreover, its files (POM) and build infrastructures always look the same in Maven projects, so you know where to look when you deal with a Maven project. For Eclipse projects, Tycho complements Maven. Tycho is not perfect, but everything works if you stick with its rules.
The rigidness of Maven/Tycho usually forces you to keep the build simple. And that’s good! 🙂
Of course, also previously, the Eclipse projects of Xtext had to be built with Maven/Tycho. Thus, the build of Xtext was split into 6 repositories with a mixture of Gradle builds and Maven/Tycho builds.
As a result, Xtext was a nightmare to build and maintain. That’s my opinion, of course, but if you read Xtext GitHub issues, PRs, and discussions, you see I wasn’t the only one to say that.
For example, the main problem was that if you changed something in one of the core projects (i.e., on its Git repository), you had to make sure that all the other projects did not break:
- once the core project was built successfully, archive its artifacts (p2 repository and Maven artifacts) in Jenkins (JIRO in Eclipse)
- trigger downstream jobs, making sure to use the archived artifacts corresponding to the branch with the changes
- wait for all downstream jobs to finish
- if anything breaks, go back from the beginning and start again!
Moreover, while it was easy to parametrize the Maven and Gradle files to use the archived Maven artifacts, it was a bit harder to use the archived p2 repositories in the target platforms of the Eclipse downstream projects. Typically, you had to temporarily modify and commit the target platform files to refer to the archived p2 repositories corresponding to the current feature branch. Before merging, you must remember to revert the changes to the target platform files.
In the end, the code in the single Git repositories was actually coupled, so the splitting in single Git repositories has always sounded wrong to me.
The release process used to be complex as well. Again, that was due to Git repository splitting and a mixture of Maven/Gradle builds.
A big change
I had always wanted to bring Xtext sources back into a single Git repository and to use Maven/Tycho only. I told the project lead Christian Dietrich about that many times, but he was always reluctant due to the time and effort it would have taken. On January 2023, I finally managed to convince him to give me a chance 🙂
I first started porting all single repositories to Maven/Tycho, removing the Gradle build.
Things were not so easy, but not even impossible. A few tests were failing in the new build infrastructure, but Christian and Sebastian Zarnekow helped me to bring everything to a working situation. We also took the chance to fix a few things in that respect.
In this stage, I also took the chance to set up the CI building also on GitHub Actions (our primary CI is, of course, Eclipse Jenkins “JIRO”).
Then, I moved to the repo merging. This part was easy. I had prepared the single Git repositories thinking about repository merging, so fixing the foreseen merge conflicts was just a matter of adjusting a few things in only a few files (mainly the parent POM).
Finally, I simplified the release infrastructure a lot. We used to have a vast Jenkinsfile for the release, with many complex operations based on Git cloning all the repositories, tagging, and tons of shell operations.
Now, the whole release is carried on during the Maven build (as it should be, in my opinion), including the uploading to the Eclipse web directories. The Jenkinsfile is very small.
The Oomph setup has been simplified as well. It takes time to have a development workspace, but that is due to dependency resolutions and the first Eclipse build. It takes place automatically.
Now, the complete Maven/Tycho build of Xtext, which consists of compilation and running all the tests (almost 40k tests!), takes about 1 hour and a half on Linux (Jenkins and GitHub Actions) and about 2 hours on macOS in GitHub Actions. It takes much less than before when you had to wait for the builds of the single Git repositories and downstream projects. The building of some single Git repositories was, of course, faster, but the overall time for building all the Git repositories was much more significant in the end.
Final thoughts
I hope this vast change improves the maintainability of the project. It should also help contributions from the community. Maybe I’m biased, but personally, I find working and contributing to Xtext much better now, as it used to be in the old days 😉
To me, the effort was very worthwhile!
In the end, it didn’t take long. Of course, I do not measure in months passed from the start of the porting (I started in the first days of January, and the first release with the new infrastructure, version 2.31.0, was at the end of May, though the first milestone with the new infrastructure was at the beginning of April). The time I spent working on that was much less. Of course, I hadn’t worked on that full-time: I did that in my spare time 🙂
In my humble opinion: If your project’s infrastructure and build are too complex, don’t change the build tools: change the latter and keep it simple. 😉 A single big Git monorepo is easier to deal with than 6 small Git repositories, especially when the single Git repositories contain code meant to belong together.
Many thanks to Christian and Sebastian for letting me work on this restructuring and promptly helping me during the process! 🙂