Gradle and TomEE Embedded FatJars

, Romain Manni-Bucau, 2016-09-02, 7 min and 49 sec read

Creating a fatjar, uber jar or a shade with maven leads quite often to maven-shade-plugin but gradle ecosystem is quite young on that area and even if it reused some maven-shade API/idea it is not always trivial to dig into it today. Let see how you can benefit from gradle scripting to build your fat TomEE jar!

So what is our goal? Merge a web application (let’s say JAX-RS + JPA) with TomEE to build an executable jar. In short do: java -jar myapp.jar.

What are the challenges? The standards one of a shade: how to merge the fullstack without breaking any part of the libraries/application.

Why should this break anything? Think to a part of the application using for its extensibility a service provider interface (SPI):

app-part1.jar contains META-INF/services/com.app.Extension with

com.app.Extension11
com.app.Extension12

app-part2.jar contains META-INF/services/com.app.Extension with

com.app.Extension21
com.app.Extension22

When the build will happen it will encounter both files and will just keep a single one of both. In other words: even if the classes are all there your application will only be half working, which means it will just not work.

Now you think it is your application so not a big deal? Actually most of the libraries use that, in TomEE stack to cite a few there are:

CXF
TomEE
MyFaces
BVal
OpenWebBeans
…

How to solve it? Configuring the way well known resources are merged. In maven-shade-plugin it is known as Transformer and the Gradle shadow pluigin just copied this notion.

Another challenge is to handle dependencies. Often in a shade you want to eclude some parts of the dependencies, you can do it directly on the dependencies or through the shading directly. Nice thing about gradle there is you can script it.

Finally last thing to keep in mind is how to ensure the jar is executable, ie how to set a Main-Class in the MANIFEST.MF.

Setup a project

First you need to create a normal gradle project, here is a build.gradle to start:

group 'com.github.rmannibucau'
version '1.0-SNAPSHOT'

apply plugin: 'idea'
apply plugin: 'java'

sourceCompatibility = 1.8

repositories {
    mavenLocal()
    mavenCentral()
}

dependencies {
    compile 'org.apache.tomee:tomee-embedded:7.0.2-SNAPSHOT'
}

If you need JPA/OpenJPA you can add the build time enhancement (highly recommanded for fatjars to avoid to have to set a javaagent on the running command):

buildscript {
    repositories {
        mavenLocal()
        jcenter()
    }
    dependencies {
        classpath 'at.schmutterer.oss.gradle:gradle-openjpa:0.2.0'
        classpath 'org.apache.openjpa:openjpa:2.4.1'
    }
}

group 'com.github.rmannibucau'
version '1.0-SNAPSHOT'

apply plugin: 'idea'
apply plugin: 'java'
apply plugin: 'openjpa'

sourceCompatibility = 1.8

repositories {
    mavenLocal()
    mavenCentral()
}

dependencies {
    compile 'org.apache.tomee:tomee-embedded:7.0.2-SNAPSHOT'
}

openjpa {
    files = fileTree(sourceSets.main.output.classesDir).matching {
        include '**/jpa/*.class'
    }
}

Now just develop your application in src/main/java and src/main/resources (don't forget the src/main/resources/META-INF/persistence.xml for JPA if needed).

Setup gradle shadow plugin

To build a fatjar with gradle you have several options but shadow plugin is the more advanced (actually pretty close to maven shade plugin). Here is a basic setup:

buildscript {
    repositories {
        mavenLocal()
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3'
    }
}

group 'com.github.rmannibucau'
version '1.0-SNAPSHOT'

// keep java, openjpa etc plugins if needed, removed for brievity
apply plugin: 'com.github.johnrengelman.shadow'

// remove the dependencies/openjpa config for brievity

shadowJar {
    classifier = 'bundle'
}

Note: this snippet didn't keep the previous blocks for brievity but we actually enrich previous build.gradle with that content.

So what does mean setting up shadow jar plugin: just adding the plugin dependency, declaring it as an applied plugin and configuring it in the shadowJar section.

But default setup will not allow you to run a TomEE fatjar, let's see what is missing!

Merging SPI files

First step is to ensure SPI files are still containing what is needed. For that aspect, shadow jar plugin copied the maven shade plugin concept of trasformer which are really aggregator of resources (from multiple jar resources to a merged one in the fatjar).

For ServiceLoader ones (META-INF/services/*) shadowJar provides a nice shortcut: "mergeServiceFiles()". For CXF we have to concatenate META-INF/cxf/bus-extensions.txt files, this can be done using the shortcut "append" which is an automatic transformer of show jar plugin. Finally we miss the aggregation of META-INF/openwebbeans/openwebbeans.properties files. That is properties so append should work? Actually not really cause these files have an ordinal so entries should be sorted. For that purpose OpenWebBeans will provide a custom transformer (one for maven, one for shadow jar) you can declare using transformer() method of shadow jar plugin:

import org.apache.openwebbeans.gradle.shadow.OpenWebBeansPropertiesTransformer

buildscript {
    repositories {
        mavenLocal()
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3'
        classpath 'org.apache.openwebbeans:openwebbeans-gradle:1.7.0'
    }
}


shadowJar {
    mergeServiceFiles()
    append 'META-INF/cxf/bus-extensions.txt'
    transform(OpenWebBeansPropertiesTransformer.class)
}

Filtering dependencies

Now we merge correctly our dependencies resources, we need to ensure we have all the depencencies we need...which also means most of the time removing the ones we don't need.

There are several syntaxes for that in shadow jar plugin but one common one is to code it using some exclusions/inclusions list configured in the build.gradle file. Personally i use two exclusions lists (but that's programmation so theorically you can do what you want):

excluded groups: used as categories of artifacts which are not required (activemq for instance if you don't need JMS)
excluded artifacts: when only a partial set of the artifacts of a groups are not needed (CXF JAX-WS artifacts when JAX-RS ones are needed for example)

Here is a sample of such an algorithm:

//
// customize exclusions depending your app
//

// first the not used dependencies like JSF, JAXWS, JMS ones
def excludedDependenciesGroups = [
        // gradle is buggy with poms, scope provided and optional I think
        'com.google.code.findbugs',
        'com.google.guava',
        'javax.annotation',
        'javax.ws.rs',
        'net.sf.ehcache',
        'org.apache.httpcomponents',
        'org.ow2.asm',
        // tomee jaxws, jms, etc...
        'commons-codec',
        'com.sun.xml.messaging.saaj',
        'joda-time',
        'junit',
        'net.shibboleth.utilities',
        'org.apache.activemq',
        'org.apache.activemq.protobuf',
        'org.apache.myfaces.core',
        'org.apache.neethi',
        'org.apache.santuario',
        'org.apache.ws.xmlschema',
        'org.apache.wss4j',
        'org.bouncycastle',
        'org.cryptacular',
        'org.fusesource.hawtbuf',
        'org.jasypt',
        'org.jvnet.mimepull',
        'org.opensaml',
        'wsdl4j',
        'xml-resolver'
]

// then cxf+tomee specific dependencies so we need to be more precise than the group
// to not exclude everything
def excludedDependenciesArtifacts = [
        'cxf-rt-bindings-soap',
        'cxf-rt-bindings-xml',
        'cxf-rt-databinding-jaxb',
        'cxf-rt-frontend-jaxws',
        'cxf-rt-frontend-simple',
        'cxf-rt-security-saml',
        'cxf-rt-ws-addr',
        'cxf-rt-wsdl',
        'cxf-rt-ws-policy',
        'cxf-rt-ws-security',
        'openejb-cxf',
        'openejb-webservices',
        'tomee-webservices',
        'geronimo-connector',
        'geronimo-javamail_1.4_mail'
]
shadowJar {
    // switch off JSF + JMS + JAXWS
    dependencies {
        exclude(dependency {
            excludedDependenciesGroups.contains(it.moduleGroup) ||
                    excludedDependenciesArtifacts.contains(it.moduleName)
        })
    }
}

We define the two sets and just use contains on the "current" dependency shadow jar plugin gives us in the depednencies clojure which behaves as a filter for dependencies.

Side note: of course shadow jar plugin also allows to include/exclude resources directly and for instance to skip JSF you can exclude META-INF/faces-config.xml file this way:

shadowJar {
    exclude 'META-INF/faces-config.xml'
}

Customizing MANIFEST.MF: Main-Class

So now we merged jar resources altogether, we included only the dependencies we want/need so we just need to ensure our jar is executable, for that we need to set in the MANIFEST.MF the Main-Class attribute. Since TomEE embedded provides one we can just reuse it: org.apache.tomee.embedded.FatApp.

Here what it looks like with shadow jar plugin:

shadowJar {
    // ensure we define the expected Main (if you wrap tomee main use your own class)
    manifest {
        attributes 'Main-Class': 'org.apache.tomee.embedded.FatApp'
    }
}

Run it

Now we have our plugin fully configured we can build our bundle:

gradle clean shadowJar

it will create a build/libs/*-bundle.jar file you can execute with the following command:

java -jar build/libs/*-bundle.jar

Note: the wildcard is used there just to not mention a too specific name but you should just use the actual jar name.

The main inherit of most tomee embedded options but it just enforces the fatjar ones (--as-war to say it is a jar to deploy as a war a,d --single-classloader to use the same classloader for the boot and the application).

Conclusion

Shadow jar plugin is very nice to build fatjar cause the usage of groovy in the build file allows to enrich the merge without creating another artifact if desired (nice for transformers). It also allows to fix programmatically and "easily" the bad parsing of maven dependencies of gradle (some excluded depencencies shouldn't be visible from included artifacts).

To conclude this article here is the full build.gradle file:

// run $ gradle clean shadowJar
// you can need to add apache snapshot repository: https://repository.apache.org/content/repositories/snapshots/
import org.apache.openwebbeans.gradle.shadow.OpenWebBeansPropertiesTransformer

buildscript {
    repositories {
        mavenLocal()
        jcenter()
    }
    dependencies {
        classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3'
        classpath 'org.apache.openwebbeans:openwebbeans-gradle:1.7.0'
        classpath 'at.schmutterer.oss.gradle:gradle-openjpa:0.2.0'
        classpath 'org.apache.openjpa:openjpa:2.4.1'
    }
}

group 'com.github.rmannibucau'
version '1.0-SNAPSHOT'

apply plugin: 'idea'
apply plugin: 'java'
apply plugin: 'openjpa'
apply plugin: 'com.github.johnrengelman.shadow'

sourceCompatibility = 1.8

repositories {
    mavenLocal()
    mavenCentral()
}

dependencies {
    compileOnly 'org.projectlombok:lombok:1.16.10'
    compile 'org.apache.tomee:tomee-embedded:7.0.2-SNAPSHOT'
}

openjpa { // in embedded module you have to set the javaagent if you don't enhance entities
    files = fileTree(sourceSets.main.output.classesDir).matching {
        include '**/User.class'
    }
}


//
// customize exclusions depending your app
//

// first the not used dependencies like JSF, JAXWS, JMS ones
def excludedDependenciesGroups = [
        // gradle is buggy with poms, scope provided and optional I think
        'com.google.code.findbugs',
        'com.google.guava',
        'javax.annotation',
        'javax.ws.rs',
        'net.sf.ehcache',
        'org.apache.httpcomponents',
        'org.ow2.asm',
        // tomee jaxws, jms, etc...
        'commons-codec',
        'com.sun.xml.messaging.saaj',
        'joda-time',
        'junit',
        'net.shibboleth.utilities',
        'org.apache.activemq',
        'org.apache.activemq.protobuf',
        'org.apache.myfaces.core',
        'org.apache.neethi',
        'org.apache.santuario',
        'org.apache.ws.xmlschema',
        'org.apache.wss4j',
        'org.bouncycastle',
        'org.cryptacular',
        'org.fusesource.hawtbuf',
        'org.jasypt',
        'org.jvnet.mimepull',
        'org.opensaml',
        'wsdl4j',
        'xml-resolver'
]

// then cxf+tomee specific dependencies so we need to be more precise than the group
// to not exclude everything
def excludedDependenciesArtifacts = [
        'cxf-rt-bindings-soap',
        'cxf-rt-bindings-xml',
        'cxf-rt-databinding-jaxb',
        'cxf-rt-frontend-jaxws',
        'cxf-rt-frontend-simple',
        'cxf-rt-security-saml',
        'cxf-rt-ws-addr',
        'cxf-rt-wsdl',
        'cxf-rt-ws-policy',
        'cxf-rt-ws-security',
        'openejb-cxf',
        'openejb-webservices',
        'tomee-webservices',
        'geronimo-connector',
        'geronimo-javamail_1.4_mail'
]
shadowJar {
    classifier = 'bundle'

    // merge SPI descriptors
    mergeServiceFiles()
    append 'META-INF/cxf/bus-extensions.txt'
    transform(OpenWebBeansPropertiesTransformer.class)

    // switch off JSF + JMS + JAXWS
    exclude 'META-INF/faces-config.xml'
    dependencies {
        exclude(dependency {
            excludedDependenciesGroups.contains(it.moduleGroup) ||
                    excludedDependenciesArtifacts.contains(it.moduleName)
        })
    }

    // ensure we define the expected Main (if you wrap tomee main use your own class)
    manifest {
        attributes 'Main-Class': 'org.apache.tomee.embedded.FatApp'
    }
}

From the same author:

Romain Manni-Bucau

In the same category:

Java/EE/Microprofile