In a previous post I showed how to manage an Eclipse composite p2 repository and how to publish an Eclipse p2 composite repository on Sourceforge. In this post I’ll show a similar procedure to publish an Eclipse p2 composite repository on Bintray. The procedure is part of the Maven/Tycho build so that it is fully automated. Moreover, the pom.xml and the ant files can be fully reused in your own projects (just a few properties have to be adapted).
The complete example at https://github.com/LorenzoBettini/p2composite-bintray-example.
First of all, this procedure is quite different from the ones shown in other blogs (e.g., this one, this one and this one): in those approaches the p2 metadata (i.e., artifacts.jar and content.jar) are uploaded independently from a version, always in the same directory, thus overwriting the existing metadata. This leads to the fact that only the latest version of published features and bundles will be available to the end user. This is quite against the idea that old versions should still be available, and in general, all the versions should be available for the end users, especially if a new version has some breaking change and the user is not willing to update (see p2’s do’s and do not’s). For this reason, I always publish p2 composite repositories.
Quoting from https://wiki.eclipse.org/Equinox/p2/Composite_Repositories_(new)
The goal of composite repositories is to make this task easier by allowing you to have a parent repository which refers to multiple children. Users are then able to reference the parent repository and the children’s content will transparently be available to them.
In order to achieve this, all published p2 repositories must be available, each one with their own p2 metadata that should never be overwritten.
On the contrary, the metadata that we will overwrite will be the one for the composite metadata, i.e., compositeContent.xml and compositeArtifacts.xml.
In this example, all the binary artifacts are meant to be made available from: https://dl.bintray.com/lorenzobettini/p2-composite-example/. (NOTE: the artifacts are not effectively available from that URL anymore).
Directory Structure
What I aim at is to have the following remote paths on Bintray:
- releases: in this directory all p2 simple repositories will be uploaded, each one in its own directory, named after version.buildQualifier, e.g., 1.0.0.v20160129-1616/ etc. Your Eclipse users can then use the URL of one of these single update sites to stick to that specific version.
- updates: in this directory the composite metadata will be uploaded. The URL https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/ should be used by your Eclipse users to install the features in their Eclipse of for target platform resolution (depending on the kind of projects you’re developing). All versions will be available from this composite update site; I call this main composite. Moreover, you can provide the URL to a child composite update site that includes all versions for a given major.minor stream, e.g., https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.0/, https://dl.bintray.com/lorenzobettini/p2-composite-example/updates/1.1/, etc. I call each one of these, child composite.
- zipped: in this directory we will upload the zipped p2 repository for each version.
Summarizing we’ll end up with a remote directory structure like the following
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
root |-- releases | |-- 1.0.0.v2016... | | |-- artifacts.jar | | |-- content.jar | | |-- features | | | |-- your feature.jar | | |-- plugins | | | |-- your bundle.jar | |-- 1.1.0.v2016... | |-- 1.1.1.v2016... | |-- 2.0.0.v2016... | ... |-- updates | |-- compositeContent.xml | |-- compositeArtifacts.xml | |-- 1.0 | | |-- compositeContent.xml | | |-- compositeArtifact.xml | |-- 1.1 | | |-- compositeContent.xml | | |-- compositeArtifact.xml | |-- 2.0 ... | ... |-- zipped |-- your site 1.0.0.v2016....zip |-- your site 1.1.0.v2016....zip ... |
Uploading using REST API
In the posts I mentioned above, the typical line to upload contents with the REST API is of the shape
1 2 3 |
curl -X PUT -T $f \ -u ${BINTRAY_USER}:${BINTRAY_API_KEY} \ https://api.bintray.com/content/${BINTRAY_OWNER}/${BINTRAY_REPO}/$f;publish=1 |
For metadata, and
1 2 3 4 |
curl -X PUT -T $f \ -u ${BINTRAY_USER}:${BINTRAY_API_KEY} \ https://api.bintray.com/content/${BINTRAY_OWNER}/${BINTRAY_REPO}/\ ${PCK_NAME}/${PCK_VERSION}/$f;publish=1 |
For features and plugins.
But this has the drawback I was mentioning above.
Thanks to the Bintray Support, I managed to use a different scheme that allows me to store p2 metadata for a single p2 repository in the same directory of the p2 repository itself and to keep those metadata separate for each single release.
To achieve this, we need to use another URL scheme for uploading, using matrix params options or header options.
This means that we’ll upload everything with this URL
1 2 3 4 |
curl -XPUT -T $f \ -u${BINTRAY_USER}:${BINTRAY_API_KEY} \ "https://api.bintray.com/content/${BINTRAY_OWNER}/${BINTRAY_REPO}/\ ${TARGET_PATH}/$f;bt_package=${PCK_NAME};bt_version=${PCK_VERSION};publish=1" |
On the contrary, for uploading p2 composite metadata, we’ll use the schema of the other approaches, i.e., we will not associate it to any specific version; we just need to specify the desired remote path where we’ll upload the main and the child composite metadata.
Building Steps
During the build, we’ll have to update the composite site metadata, and we’ll have to do that locally.
The steps that we’ll perform during the Maven/Tycho build, which will rely on some Ant scripts can be summarized as follows:
- Retrieve the remote composite metadata compositeContent/Artifacts.xml, both for the main composite and the child composite. If these metadata cannot be found remotely, we fail gracefully: it means that it is the first time we release, or, if only the child composite cannot be found, that we’re releasing a new major.minor version. These will be downloaded in the directories target/main-composite and target/child-composite respectively. These will be created anyway.
- Preprocess possible downloaded composite metadata: if this property is present
1<property name='p2.atomic.composite.loading' value='true'/>
We must temporarily set it to false, otherwise we will not be able to add additional elements in the composite site with the p2 ant tasks. - Update the composite metadata using the version information passed from the Maven/Tycho build using the p2 Ant tasks for composite repositories
- Post process the composite metadata (i.e., put the property p2.atomic.composite.loading above to true, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356561 for further details about this property). UPDATE: Please have a look at the comment section, in particular, the comments from pascalrapicault, about this property.
- Upload everything to bintray: both the new p2 repository, its zipped version and all the composite metadata.
IMPORTANT: the pre and post processing of composite metadata that we’ll implement assumes that such metadata are not compressed. Anyway, I always prefer not to compress the composite metadata since it’s easier, later, to manually change them or reviewing.
Technical Details
You can find the complete example at https://github.com/LorenzoBettini/p2composite-bintray-example. Here I’ll sketch the main parts. First of all, all the mechanisms for updating the composite metadata and pushing to Bintray (i.e., the steps detailed above) are in the project p2composite.example.site, which is a Maven/Tycho project with eclipse-repository packaging.
The pom.xml has some properties that you should adapt to your project, and some other properties that can be left as they are if you’re OK with the defaults:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
<properties> <!-- The name of your own Bintray repository --> <bintray.repo>p2-composite-example</bintray.repo> <!-- The name of your own Bintray repository's package for releases --> <bintray.package>releases</bintray.package> <!-- The label for the Composite sites --> <site.label>Composite Site Example</site.label> <!-- If the Bintray repository is owned by someone different from your user, then specify the bintray.owner explicitly --> <bintray.owner>${bintray.user}</bintray.owner> <!-- Define bintray.user and bintray.apikey in some secret place, like .m2/settings.xml --> <!-- Default values for remote directories --> <bintray.releases.path>releases</bintray.releases.path> <bintray.composite.path>updates</bintray.composite.path> <bintray.zip.path>zipped</bintray.zip.path> <!-- note that the following must be consistent with the path schema used to publish child composite repositories and actual released p2 repositories --> <child.repository.path.prefix>../../releases/</child.repository.path.prefix> </properties> |
If you change the default remote paths it is crucial that you update the child.repository.path.prefix consistently. In fact, this is used to update the composite metadata for the composite children. For example, with the default properties the composite metadata will look like the following (here we show only compositeContent.xml):
1 2 3 4 5 6 7 8 9 10 11 12 13 |
<?xml version='1.0' encoding='UTF-8'?> <?compositeMetadataRepository version='1.0.0'?> <repository name='Composite Site Example 1.0' type='org.eclipse.equinox.internal.p2.metadata.repository.CompositeMetadataRepository' version='1.0.0'> <properties size='2'> <property name='p2.timestamp' value='1454086165279'/> <property name='p2.atomic.composite.loading' value='true'/> </properties> <children size='3'> <child location='../../releases/1.0.0.v20160129-1625'/> <child location='../../releases/1.0.0.v20160129-1630'/> <child location='../../releases/1.0.0.v20160129-1649'/> </children> </repository> |
You can also see that two crucial properties, bintray.user and, in particular, bintray.apikey should not be made public. You should keep these hidden, for example, you can put them in your local .m2/settings.xml file, associated to the Maven profile that you use for releasing (as illustrated in the following). This is an example of settings.xml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd/"> <profiles> <profile> <id>release-composite</id> <activation> <activeByDefault>false</activeByDefault> </activation> <properties> <bintray.user>YOUR BINTRAY USER HERE</bintray.user> <bintray.apikey>YOUR BINTRAY APIKEY HERE</bintray.apikey> </properties> </profile> </profiles> </settings> |
In the pom.xml of this project there is a Maven profile, release-composite, that should be activated when you want to perform the release steps described above.
We also make sure that the generated zipped p2 repository has a name with fully qualified version
1 2 3 4 5 6 7 8 9 |
<!-- make sure that zipped p2 repositories have the fully qualified version --> <plugin> <groupId>org.eclipse.tycho</groupId> <artifactId>tycho-p2-repository-plugin</artifactId> <version>${tycho-version}</version> <configuration> <finalName>${project.artifactId}-${qualifiedVersion}</finalName> </configuration> </plugin> |
In the release-composite Maven profile, we use the maven-antrun-plugin to execute some ant targets (note that the Maven properties are automatically passed to the Ant tasks): one to retrieve the remote composite metadata, if they exist, and the other one as the final step to deploy the p2 repository, its zipped version and the composite metadata to Bintray:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
<plugin> <artifactId>maven-antrun-plugin</artifactId> <version>${maven-antrun-plugin.version}</version> <executions> <execution> <!-- Retrieve possibly existing remote composite metadata --> <id>update-local-repository</id> <phase>prepare-package</phase> <configuration> <target> <ant antfile="${basedir}/bintray.ant" target="get-composite-metadata"> </ant> </target> </configuration> <goals> <goal>run</goal> </goals> </execution> <execution> <!-- Deploy p2 repository, p2 composite updated metadata and zipped p2 repository --> <id>deploy-repository</id> <phase>verify</phase> <configuration> <target> <ant antfile="${basedir}/bintray.ant" target="push-to-bintray"> </ant> </target> </configuration> <goals> <goal>run</goal> </goals> </execution> </executions> </plugin> |
The Ant tasks are defined in the file bintray.ant. Please refer to the example for the complete file. Here we sketch the main parts.
This Ant file relies on some properties with default values, and other properties that are expected to be passed when running these tasks, i.e., from the pom.xml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 |
<!-- These must be set from outside <property name="bintray.user" value="" /> <property name="bintray.apikey" value="" /> <property name="bintray.repo" value="" /> <property name="bintray.package" value="" /> <property name="bintray.releases.path" value="" /> <property name="bintray.composite.path" value="" /> <property name="bintray.zip.path" value="" /> --> <property name="bintray.url" value="https://dl.bintray.com/${bintray.owner}/${bintray.repo}" /> <property name="bintray.package.version" value="${unqualifiedVersion}.${buildQualifier}" /> <property name="bintray.releases.target.path" value="${bintray.releases.path}/${bintray.package.version}" /> <property name="main.composite.url" value="${bintray.url}/${bintray.composite.path}" /> <property name="target" value="target" /> <property name="composite.repository.directory" value="composite-child" /> <property name="main.composite.repository.directory" value="composite-main" /> <property name="compositeArtifacts" value="compositeArtifacts.xml" /> <property name="compositeContent" value="compositeContent.xml" /> <property name="local.p2.repository" value="target/repository" /> |
To retrieve the existing remote composite metadata we execute the following, using the standard Ant get task. Note that if there is no composite metadata (e.g., it’s the first release that we execute, or we are releasing a new major.minor version so there’s no child composite for that version) we ignore the error; however, we still create the local directories for the composite metadata:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
<!-- Take from the remote URL the possible existing metadata --> <target name="get-composite-metadata" depends="getMajorMinorVersion" > <get-metadata url="${main.composite.url}" dest="${target}/${main.composite.repository.directory}" /> <get-metadata url="${main.composite.url}/${majorMinorVersion}" dest="${target}/${composite.repository.directory}" /> <antcall target="preprocess-metadata" /> </target> <macrodef name="get-metadata" description="Retrieve the p2 composite metadata"> <attribute name="url" /> <attribute name="dest" /> <sequential> <echo message="Creating directory @{dest}..." /> <mkdir dir="@{dest}" /> <get-file file="${compositeArtifacts}" url="@{url}" dest="@{dest}" /> <get-file file="${compositeContent}" url="@{url}" dest="@{dest}" /> </sequential> </macrodef> <macrodef name="get-file" description="Use Ant Get task the file"> <attribute name="file" /> <attribute name="url" /> <attribute name="dest" /> <sequential> <!-- If the remote file does not exist then fail gracefully --> <echo message="Getting @{file} from @{url} into @{dest}..." /> <get dest="@{dest}" ignoreerrors="true"> <url url="@{url}/@{file}" /> </get> </sequential> </macrodef> |
For preprocessing/postprocessing composite metadata (in order to deal with the property p2.atomic.composite.loading as explained in the previous section) we have
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
<!-- p2.atomic.composite.loading must be set to false otherwise we won't be able to add a child to the composite repository without having all the children available --> <target name="preprocess-metadata" description="Preprocess p2 composite metadata"> <replaceregexp byline="true"> <regexp pattern="property name='p2.atomic.composite.loading' value='true'" /> <substitution expression="property name='p2.atomic.composite.loading' value='false'" /> <fileset dir="${target}"> <include name="${composite.repository.directory}/*.xml" /> <include name="${main.composite.repository.directory}/*.xml" /> </fileset> </replaceregexp> </target> <!-- p2.atomic.composite.loading must be set to true see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356561 --> <target name="postprocess-metadata" description="Preprocess p2 composite metadata"> <replaceregexp byline="true"> <regexp pattern="property name='p2.atomic.composite.loading' value='false'" /> <substitution expression="property name='p2.atomic.composite.loading' value='true'" /> <fileset dir="${target}"> <include name="${composite.repository.directory}/*.xml" /> <include name="${main.composite.repository.directory}/*.xml" /> </fileset> </replaceregexp> </target> |
Finally, to push everything to Bintray we execute curl with appropriate URLs, as we described in the previous section about REST API. The single tasks for pushing to Bintray are similar, so we only show one for uploading the p2 repository associated to a specific version, and the one for uploading p2 composite metadata. As detailed at the beginning of the post, we use different URL shapes.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
<target name="push-to-bintray" > <antcall target="postprocess-metadata" /> <antcall target="push-p2-repo-to-bintray" /> <antcall target="push-p2-repo-zipped-to-bintray" /> <antcall target="push-composite-to-bintray" /> <antcall target="push-main-composite-to-bintray" /> </target> <target name="push-p2-repo-to-bintray"> <apply executable="curl" parallel="false" relative="true" addsourcefile="false"> <arg value="-XPUT" /> <targetfile /> <fileset dir="${local.p2.repository}" /> <compositemapper> <mergemapper to="-T" /> <globmapper from="*" to="${local.p2.repository}/*" /> <mergemapper to="-u${bintray.user}:${bintray.apikey}" /> <globmapper from="*" to="https://api.bintray.com/content/${bintray.owner}/${bintray.repo}/${bintray.releases.target.path}/*;bt_package=${bintray.package};bt_version=${bintray.package.version};publish=1" /> </compositemapper> </apply> </target> <target name="push-composite-to-bintray" depends="getMajorMinorVersion" > <apply executable="curl" parallel="false" relative="true" addsourcefile="false"> <arg value="-XPUT" /> <targetfile /> <fileset dir="${target}/${composite.repository.directory}" /> <compositemapper> <mergemapper to="-T" /> <globmapper from="*" to="${target}/${composite.repository.directory}/*" /> <mergemapper to="-u${bintray.user}:${bintray.apikey}" /> <globmapper from="*" to="https://api.bintray.com/content/${bintray.owner}/${bintray.repo}/${bintray.composite.path}/${majorMinorVersion}/*;publish=1" /> </compositemapper> </apply> </target> |
To update composite metadata we execute an ant task using the tycho-eclipserun-plugin. This way, we can execute the Eclipse application org.eclipse.ant.core.antRunner, so that we can execute the p2 Ant tasks for managing composite repositories.
ATTENTION: in the following snipped, for the sake of readability, I split the <appArgLine> into several lines, but in your pom.xml it must be exactly in one (long) line.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
<plugin> <groupId>org.eclipse.tycho.extras</groupId> <artifactId>tycho-eclipserun-plugin</artifactId> <version>${tycho-version}</version> <configuration> <!-- Update p2 composite metadata or create it --> <!-- IMPORTANT: DO NOT split the arg line --> <appArgLine>-application org.eclipse.ant.core.antRunner -buildfile packaging-p2composite.ant p2.composite.add -Dsite.label="${site.label}" -Dproject.build.directory=${project.build.directory} -DunqualifiedVersion=${unqualifiedVersion} -DbuildQualifier=${buildQualifier} -Dchild.repository.path.prefix="${child.repository.path.prefix}"</appArgLine> <repositories> <repository> <id>mars</id> <layout>p2</layout> <url>http://download.eclipse.org/releases/mars</url> </repository> </repositories> <dependencies> <dependency> <artifactId>org.eclipse.ant.core</artifactId> <type>eclipse-plugin</type> </dependency> <dependency> <artifactId>org.apache.ant</artifactId> <type>eclipse-plugin</type> </dependency> <dependency> <artifactId>org.eclipse.equinox.p2.repository.tools</artifactId> <type>eclipse-plugin</type> </dependency> <dependency> <artifactId>org.eclipse.equinox.p2.core.feature</artifactId> <type>eclipse-feature</type> </dependency> <dependency> <artifactId>org.eclipse.equinox.p2.extras.feature</artifactId> <type>eclipse-feature</type> </dependency> <dependency> <artifactId>org.eclipse.equinox.ds</artifactId> <type>eclipse-plugin</type> </dependency> </dependencies> </configuration> <executions> <execution> <id>add-p2-composite-repository</id> <phase>package</phase> <goals> <goal>eclipse-run</goal> </goals> </execution> </executions> </plugin> |
The file packaging-p2-composite.ant is similar to the one I showed in a previous post. We use the p2 Ant tasks for adding a child to a composite p2 repository (recall that if there is no existing composite repository, the task for adding a child also creates new compositeContent.xml/Artifacts.xml; if a child with the same name exists the ant task will not add anything new).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 |
<?xml version="1.0"?> <project name="project"> <target name="getMajorMinorVersion"> <script language="javascript"> <![CDATA[ // getting the value buildnumber = project.getProperty("unqualifiedVersion"); index = buildnumber.lastIndexOf("."); counter = buildnumber.substring(0, index); project.setProperty("majorMinorVersion",counter); ]]> </script> </target> <target name="test_getMajorMinor" depends="getMajorMinorVersion"> <echo message="majorMinorVersion: ${majorMinorVersion}" /> </target> <!-- site.label The name/title/label of the created composite site unqualifiedVersion The version without any qualifier replacement buildQualifier The build qualifier child.repository.path.prefix The path prefix to access the actual p2 repo from the child repo, e.g., if child repo is in /updates/1.0 and the p2 repo is in /releases/1.0.0.something then this property should be "../../releases/" --> <target name="compute.child.repository.data" depends="getMajorMinorVersion"> <property name="full.version" value="${unqualifiedVersion}.${buildQualifier}" /> <property name="site.composite.name" value="${site.label} ${majorMinorVersion}" /> <property name="main.site.composite.name" value="${site.label} All Versions" /> <!-- composite.base.dir The base directory for the local composite metadata, e.g., from Maven, ${project.build.directory} --> <property name="composite.base.dir" value="target"/> <property name="main.composite.repository.directory" location="${composite.base.dir}/composite-main" /> <property name="composite.repository.directory" location="${composite.base.dir}/composite-child" /> <property name="child.repository" value="${child.repository.path.prefix}${full.version}" /> </target> <target name="p2.composite.add" depends="compute.child.repository.data"> <add.composite.repository.internal composite.repository.location="${main.composite.repository.directory}" composite.repository.name="${main.site.composite.name}" composite.repository.child="${majorMinorVersion}" /> <add.composite.repository.internal composite.repository.location="${composite.repository.directory}" composite.repository.name="${site.composite.name}" composite.repository.child="${child.repository}" /> </target> <!-- = = = = = = = = = = = = = = = = = macrodef: add.composite.repository.internal = = = = = = = = = = = = = = = = = --> <macrodef name="add.composite.repository.internal"> <attribute name="composite.repository.location" /> <attribute name="composite.repository.name" /> <attribute name="composite.repository.child" /> <sequential> <echo message=" " /> <echo message="Composite repository : @{composite.repository.location}" /> <echo message="Composite name : @{composite.repository.name}" /> <echo message="Adding child repository : @{composite.repository.child}" /> <p2.composite.repository> <repository compressed="false" location="@{composite.repository.location}" name="@{composite.repository.name}" /> <add> <repository location="@{composite.repository.child}" /> </add> </p2.composite.repository> <echo file="@{composite.repository.location}/p2.index">version=1 metadata.repository.factory.order=compositeContent.xml,\! artifact.repository.factory.order=compositeArtifacts.xml,\! </echo> </sequential> </macrodef> </project> |
Removing Released artifacts
In case you want to remove an existing released version, since we upload the p2 repository and the zipped version as part of a package’s version, we just need to delete that version using the Bintray Web UI. However, this procedure will never remove the metadata, i.e., artifacts.jar and content.jar. The same holds if you want to remove the composite metadata. For these metadata files, you need to use the REST API, e.g., with curl. I put a shell script in the example to quickly remove all the metadata files from a given remote Bintray directory.
Performing a Release
For performing a release you just need to run
1 |
mvn clean verify -Prelease-composite |
on the p2composite.example.tycho project.
Concluding Remarks
As I said, the procedure shown in this example is meant to be easily reusable in your projects. The ant files can be simply copied as they are. The same holds for the Maven profile. You only need to specify the Maven properties that contain values for your very project, and adjust your settings.xml with sensitive data like the bintray APIKEY.
Happy Releasing! 🙂