I recently started to port all my Java projects from Travis CI to GitHub Actions, since Travis CI changed its pricing model. (I’ll soon update also my book on TDD and Build Automation under that respect.)
I’ve always used caching mechanisms during the builds in Travis CI, to speed up the builds: caching Maven dependencies, especially in big projects, can save a lot of time. In my case, I’m mostly talking of Eclipse plug-in projects, built with Maven/Tycho, and the target platform resolution might have to download a few hundreds of megabytes. Thus, I wanted to use caching also in GitHub Actions, and there’s an action for that.
In this post, I’ll show my strategies for using the cache, in particular, using different workflows based on different operating systems, which are triggered only on some specific events. I’ll use a very simple example, but I’m using this strategy currently on this Xtext project: https://github.com/LorenzoBettini/edelta, which uses more than 300 Mb of dependencies.
The post assumes that you’re already familiar with GitHub Actions.
Warning: Please keep in mind that caches will also be evicted automatically (currently, the documentation says that “caches that are not accessed within the last week will also be evicted”). However, we can still benefit from caches if we are working on a project for a few days in a row.
To experiment with building mechanisms, I suggest you use a very simple example. I’m going to use a simple Maven Java project created with the corresponding Maven archetype: a Java class and a trivial JUnit test. The Java code is not important in this context, and we’ll concentrate on the build automation mechanisms.
The final project can be found here:
https://github.com/LorenzoBettini/github-actions-cache-example.
This is the initial POM for this project:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>io.github.lorenzobettini</groupId> <artifactId>example</artifactId> <version>0.0.1-SNAPSHOT</version> <name>example</name> <url>http://www.example.com</url> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <maven.compiler.source>1.8</maven.compiler.source> <maven.compiler.target>1.8</maven.compiler.target> </properties> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> </dependencies> <build> <pluginManagement><!-- lock down plugins versions to avoid using Maven defaults (may be moved to parent pom) --> <plugins> <!-- clean lifecycle, see https://maven.apache.org/ref/current/maven-core/lifecycles.html#clean_Lifecycle --> <plugin> <artifactId>maven-clean-plugin</artifactId> <version>3.1.0</version> </plugin> ... </plugins> </pluginManagement> </build> </project> |
This is the main workflow file (stored in .github/workflows/maven.yml):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# This workflow will build a Java project with Maven name: Java CI with Maven in Linux on: push: pull_request: jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up JDK 8 uses: actions/setup-java@v1 with: java-version: 1.8 - name: Cache Maven packages uses: actions/cache@v2 with: path: ~/.m2 key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} restore-keys: ${{ runner.os }}-m2- - name: Build with Maven run: mvn -f io.github.lorenzobettini.example/pom.xml verify |
This is a pretty standard workflow for a Java project built with Maven. This workflow runs for every push on any branch and every PR.
Note that we specify to cache the directory where Maven stores all the downloaded artifacts, ~/.m2.
For the cache key, we use the OS where our build is running, a constant string “-m2-” and the result of hashing all the POM files (we’ll see how we rely on this hashing later in this post).
Remember that the cache key will be used in future builds to restore the files saved in the cache. When no cache is found with the given key, the action searches for alternate keys if the restore-keys has been specified. As you see, we specified as the restore key something similar to the actual key: the running OS and the constant string “-m2-” but no hashing. This way, if we change our POMs, the hashing will be different, but if a previous cache exists we can still restore that and make use of the cached values. (See the official documentation for further details.) We’ll then have to download only the new dependencies if any. The cache will then be updated at the end of the successful job.
I usually rely on this strategy for the CI of my projects:
- build every pushes in any branch using a Linux build environment;
- build PRs in any branch also on a Windows and macOS environment (actually, I wasn’t using Windows with Travis CI since it did not provide Java support on that environment; that’s another advantage of GitHub Actions, which provides Java support also on Windows)
Thus, I have another workflow definition just for PRs (stored in .github/workflows/pr.yml):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# This workflow will build a Java project with Maven # This runs only on PR name: Java CI with Maven in Windows and MacOS on: pull_request: jobs: build: strategy: matrix: os: ['macos-latest', 'windows-latest'] runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v2 - name: Set up JDK 8 uses: actions/setup-java@v1 with: java-version: 1.8 - name: Cache Maven packages uses: actions/cache@v2 with: path: ~/.m2 key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} restore-keys: ${{ runner.os }}-m2- - name: Build with Maven run: mvn -f io.github.lorenzobettini.example/pom.xml verify |
Besides the build matrix for OSes, that’s basically the same as the previous workflow. In particular, we use the same strategy for defining the cache key (and restore key). Thus, we have a different cache for each different operating system.
Now, let’s have a look at the documentation of this action:
A workflow can access and restore a cache created in the current branch, the base branch (including base branches of forked repositories), or the default branch. For example, a cache created on the default branch would be accessible from any pull request. Also, if the branch feature-b has the base branch feature-a, a workflow triggered on feature-b would have access to caches created in the default branch (main), feature-a, and feature-b.
Access restrictions provide cache isolation and security by creating a logical boundary between different workflows and branches. For example, a cache created for the branch feature-a (with the base main) would not be accessible to a pull request for the branch feature-b (with the base main).
What does that mean in our scenario? Since the workflow running on Windows and macOS is executed only in PRs, this means that the cache for these two configurations will never be saved for the master branch. In turns, this means that each time we create a new PR, this workflow will have no chance of finding a cache to restore: the branch for the PR is new (so no cache is available for such a branch) and the base branch (typically, “master” or “main”) will have no cache saved for these two OSes. Summarizing, the builds for the PRs for these two configurations will always have to download all the Maven dependencies from scratch. Of course, if we don’t immediately merge the PR and we push other commits on the branch of the PR, the builds for these two OSes will find a saved cache (if the previous builds of the PR succeeded), but, in any case, the first build for each new PR for these two OSes will take more time (actually, much more time in a complex project with lots of dependencies).
Thus, if we want to benefit from caching also on these two OSes, we have to have another workflow on the OSes Windows and macOS that runs when merging a PR, so that the cache will be stored also for the master branch (actually we could use this strategy also when merging any PR with any base branch, not necessarily the main one).
Here’s this additional workflow (stored in .github/workflows/pr-merge.yml):
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 |
# This workflow will build a Java project with Maven # This runs only when a PR is merged, to create or update the cache # This is useful because the builds on Windows and macOS are triggered # only on PR, so, for new branches, the cache is always empty, since # the builds on PR will not update the cache on the master branch. # This workflow will update the cache for Windows and macOS on the master branch # when the PR is merged (actually it will update on the base branch of a merged PR). # The build consists in a Maven run but skipping tests name: Updates Cache on Windows and macOS on: push: # Sequence of patterns matched against refs/heads branches: # Push events on the main branch - master # Push events to branches matching - experiments* jobs: build: if: contains(github.event.head_commit.message, 'Merge pull request') strategy: matrix: os: ['macos-latest', 'windows-latest'] runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v2 - name: Set up JDK 8 uses: actions/setup-java@v1 with: java-version: 1.8 - name: Cache Maven packages uses: actions/cache@v2 with: path: ~/.m2 key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} restore-keys: ${{ runner.os }}-m2- - name: Build with Maven run: mvn -f io.github.lorenzobettini.example/pom.xml verify -DskipTests |
Note that we intercept the event push (since a merge of a PR is actually a push) but we have an if statement that enables the workflow only when the commit message contains the string “Merge pull request”, which is the default message when merging a PR on GitHub. In this example, we are only interested in PR merged with the master branch and with any branch starting with “experiments”, but you can adjust that as you see fit. Furthermore, since this workflow is only meant for updating the Maven dependency cache, we skip the tests (with -DskipTests) so that we save some time (especially in a complex project with lots of tests).
This way, after the first PR merged, the PR workflows running on Windows and macOS will find a cache (at least as a starting point).
We can also do better than that and avoid running the Maven build if there’s no need to update the cache. Remember that we use the result of hashing all the POM files in our cache key? We mean that if our POMs do not change then basically we don’t expect our dependencies to change (of course if we’re not using SNAPSHOT dependencies). Now, in the documentation, we also read
When key matches an existing cache, it’s called a cache hit, […] When key doesn’t match an existing cache, it’s called a cache miss, and a new cache is created if the job completes successfully.
The idea is to skip the Maven step in the above workflow “Updates Cache on Windows and macOS” if we have a cache hit since we expect no new dependencies are needed to be downloaded (our POMs haven’t changed). This is the interesting part to change:
|
1 2 3 4 5 6 7 8 9 10 |
- name: Cache Maven packages uses: actions/cache@v2 id: m2cache # note the cache id, see below for cache-hit with: path: ~/.m2 key: ${{ runner.os }}-m2-${{ hashFiles('**/pom.xml') }} restore-keys: ${{ runner.os }}-m2- - name: Build with Maven if: steps.m2cache.outputs.cache-hit != 'true' # note the use of the id 'm2cache' run: mvn -f io.github.lorenzobettini.example/pom.xml verify -DskipTests |
Note that we need to define an id for the cache to intercept the cache hit or miss and the id must match the id in the if statement.
This way, if we have a cache hit the workflow for updating the cache on Windows and macOS will be really fast since it won’t even run the Maven build for updating the cache.
If we change the POM, e.g., switch to JUnit 4.13.1, push the change, create a PR, and merge it, then, the workflow for updating the cache will actually run the Maven build since we have a cache miss: the key of the cache has changed. Of course, we’ll still benefit from the already cached dependencies (and all the Maven plugins already downloaded) and we’ll update the cache with the new dependencies for JUnit 4.13.1.
Final notes
One might think to intercept the merge of a PR by using on: pull_request: (as we did in the pr.yml workflow). However, “There’s no way to specify that a workflow should be triggered when a pull request is merged”. In the official forum, you can find a solution based on the “closed” PR event and the inspection of the PR “merged” event. So one might think to update the pr.yml workflow accordingly and get rid of the additional pr-merge.yml workflow. However, from my experiments, this solution will not make use of caching, which is the main goal of this post. The symptoms of such a problem are this message when the workflow initially tries to restore a cache:
Warning: Cache service responded with 403
and this message when the cache should be saved:
Unable to reserve cache with key …, another job may be creating this cache.
Another experiment that I tried was to remove the running OS from the cache key, e.g., m2-${{ hashFiles(‘**/pom.xml’) }} instead of ${{ runner.os }}-m2-${{ hashFiles(‘**/pom.xml’) }}, and to use a restore accordingly key, like m2- instead of ${{ runner.os }}-m2-. I was hoping to reuse the same cache across different OS environments. This seems to work for macOS, which seems to be able to reuse the Linux cache. Unfortunately, this does not work for Windows. Thus, I gave up that solution.