Showing posts with label maven. Show all posts
Showing posts with label maven. Show all posts

Monday, 20 June 2016

Maven, SBT, Gradle local repository sharing

I've run into environment where different project were using different build tools - Maven, Gradle and SBT. Problem was how to reuse build artifacts between them and here goes small summary of my investigation. I will cover only local file repositories and NOT publishing/retrieving to/from remote repositories.

TL;DR every tool is able to use publish into and retrieve from Maven local repository

I'm using today's latest versions of Maven 3.3.9, Gradle 2.14, SBT 0.13.11 While Maven dependency management and artifact publishing is pretty stable, SBT and Gradle are still evolving quite a lot so check your versions carefully.

All build tools (unless you override default configuration) use your home directory to keep local repository inside of it. It is different on various operating systems and with my user name mvanek, then it will be

  • Linux ${USER_HOME} is /home/mvanek
  • Mac OS X ${USER_HOME} is /Users/mvanek
  • Windows 10 %USERPROFILE% is C:\Users\mvanek

Maven 3.3.9

  • mvn package - Operates inside target subdirectory of your project
  • mvn install - Also copy jar into ${USER_HOME}/.m2/repository

SBT 0.13.11

sbt build operates inside target subdirectory of your project

Let's have following simple build.sbt:
organization := "bt"
name := "bt-sbt"
version := "1.0.0-SNAPSHOT"

scalaVersion := "2.11.7"
sbtVersion := "0.13.11"

resolvers += Resolver.mavenLocal // Also use $HOME/.m2/repository

libraryDependencies += "commons-codec" % "commons-codec" % "1.10" 
libraryDependencies += "org.scalatest" %% "scalatest" % "2.2.6" % "test"

SBT -> Ivy

SBT is using Apache Ivy internally so executing sbt publish-local (which same as sbt publishLocal) will build and store produced jars into ${USER_HOME}/.ivy2/local/bt/bt-sbt_2.11 subdirectories Because Scala applications are bound to Scala version they are compiled with, Ivy module/Maven artifactId/Jar file name will be bt-sbt_2.11 to reflect that.

SBT -> Maven

Since SBT 0.13.0 publishing into Maven local repo is trivial. Execute sbt publish-m2 which is same as sbt publishM2 to build and store compiled binary, source and javadoc jars into ${USER_HOME}/.m2/repository/bt/bt-sbt_2.11/1.0.0-SNAPSHOT

Note that for file cache of remote repository's artifacts SBT uses ${USER_HOME}/.ivy2/cache directory.

Gradle 2.14

Gradle by itself does not have concept of local repository. gradle build - Operates inside build subdirectory of your project

Gradle's build behaviour changes with plugins applied inside build.gradle. Let's assume only java plugin is used in simple build.gradle file
apply plugin: 'java'

group = 'bt'
version = '1.0.0-SNAPSHOT'

task sourceJar(type: Jar) {
    from sourceSets.main.allJava
}

repositories {
    mavenCentral()
}

dependencies {
 compile group: 'commons-codec', name: 'commons-codec', version: '1.10'
 testCompile group: 'junit', name: 'junit', version: '4.12'
}

Name of the directory that contains Gradle project files is taken project name. This is important because project name will become Maven artifactId and it cannot be redefined in build.gradle. You can redefine it using settings.gradle file with line rootProject.name = 'bt-gradle'. There are also options to configure it just for individual plugins but I'd not recommend it because then you need to configure it for every plugin that might be affected (some publish plugin for example)

Gradle -> Maven

For sharing with Maven, probably easiest choice is maven plugin

Apply plugin in your build.gradle apply plugin: 'maven' then execute build via gradle install and your jar will land in ${USER_HOME}/.m2/repository/bt/bt-gradle/1.0.0-SNAPSHOT/bt-gradle-1.0.0-SNAPSHOT.jar

Another Gradle -> Maven option is maven-publish plugin which might be good choice if you also plan to publish artifacts into remote repository as well.

Add into build.gradle

apply plugin: 'maven-publish'

publishing {
    publications {
        mavenJava(MavenPublication) {
            from components.java
        }
    }
}
and execute build using gradle publishToMavenLocal

Gradle -> Ivy

Because SBT is using Ivy distribution instead of Maven, ivy-publish plugin should be weapon of choice for this.
apply plugin: 'ivy-publish'

publishing {
    publications {
        ivyJava(IvyPublication) {
            from components.java
        }
    }
    
    repositories {
        ivy {
            url "${System.properties['user.home']}/.ivy2/local"
        } 
    }
}
then execute buid using gradle publishIvyJavaPublicationToIvyRepository and jar will appear in ${USER_HOME}/.ivy2/local/bt/gradle-built/1.0.0-SNAPSHOT/gradle-built-1.0.0-SNAPSHOT.jar. This is unfortunately wrong as it is Maven layout, not Ivy. You can add layout explicitly
    repositories {
        ivy {
            layout "ivy"
            url "${System.properties['user.home']}/.ivy2/local"
        } 
    }
which is better and will work for simple jar build but it will fail when you attach sources. Yet another reason why I'll never use Gradle unless I'm tortured for a week at least.

Note that for file cache of remote repository's artifacts Gradle uses ${USER_HOME}/.gradle/caches/modules-2/files-2.1 directory

Happy local jar sharing!

Monday, 3 November 2014

Airbrake for logback

I've you've been observing this ticket for a while and it seems to be pretty ignored. Well not anymore or at least not by me. Undramatic sources of Airbrake Logback Appender are in GitHub airbrake-logback repo

Grab it from Maven central repo

<dependency>
    <groupId>net.anthavio</groupId>
    <artifactId>airbrake-logback</artifactId>
    <version>1.0.0</version>
</dependency>

Use it...well...as usual

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false" scan="true" scanPeriod="30 seconds">

    <appender name="AIRBRAKE" class="net.anthavio.airbrake.AirbrakeLogbackAppender">
        <apiKey>YOUR_AIRBRAKE_API_KEY</apiKey>
        <env>test</env>
        <enabled>true</enabled>

        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>ERROR</level>
        </filter>
    </appender>

    <root>
        <level value="info" />
        <appender-ref ref="AIRBRAKE" />
    </root>
</configuration>

Happy Logback based Airbraking

Wednesday, 23 April 2014

Java cloud hosted continuous integration services

Last year I resigned to maintain my own VPS hosted CI server with Git Repos, Jenkins, Sonar, Maven repository, LDAP, etc... and moved source code repositories to GitHub. I'm uploading build artifacts into Sonatype OSSRH in case of libraries and with web applications deployments, I'm still experimenting with Openshift and some other services. But I also lost possibility do build and deploy project on demand. Fortunately many cloud hosted CI services popped up last year or two.

CircleCI

Predefined, not very extensive toolset for Java projects is avaliable. It has recognized my pom.xml automaticaly without any configuration. Uses circle.yml configuration file. It is does not have free plan (only trial), cheapest Solo plan is for $19/month.

Travis-CI

Java is quite well supported. It is cofigured using .travis.yml. There is free (unlimited) plan and also quite expensive paid plans

Codeship

Neat interface, but you are allowed to use only few preinstalled java tools. Free plan is limited to 50 builds per month, while Basic plan will cost you $49 per month.

drone.io

Simplistic with basic java support. Free unlimited plan.

Cloudbees DEV@Cloud

Jenkins in the cloud with all it's awesomeness. You can choose pretty much any Java, Maven or Gradle version you can imagine plus zillions of Jenkins plugins. Pricing is trickier here because billing is build time based and offering includes application hosting (you might not be interested in). Free plan includes 100 build minutes and there is also FOSS plan. Starter plan with 40 hours of build per month will cost you $60 + $17 = $77 per month

Another Cloudbees service is BuildHive where you will get unlimited number of builds, but you will use shared and slower Jenkins instance with very limited configuration and only for GitHub repositories.

Test drive with Phanbedder

I have recently built little library named Phanbedder and blogged about it few days ago. While very tiny in java code size, it is pretty unusual because it launches separate processes of bundeled PhantomJS native binary during the tests. This makes it perfect candidate for testing cloud CI services.

Good news is that every above mentioned service managed to compile and test it using mvn clean test -Denforcer.skip=true command. I use maven-enforcer-plugin to enforce Java 6 to be used for compilation. But because many of CIs offers only Java 7, I had to switch enforcer off...

Trickier part is execution of mvn deploy. I'm uploading maven snapshot artifacts into Sonatype OSSRH and also release versions later into Maven central. For snapshot deployment into Sonatype OSS Nexus, username and password must be provided to make maven-deploy-plugin work, otherwise...

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.8.1:deploy (default-deploy) on project phanbedder-1.9.7: Failed to deploy artifacts: Could not transfer artifact net.anthavio:phanbedder-1.9.7:jar:1.0.1-20140422.184656-7 from/to sonatype-oss-snapshots (https://oss.sonatype.org/content/repositories/snapshots): Failed to transfer file: https://oss.sonatype.org/content/repositories/snapshots/net/anthavio/phanbedder-1.9.7/1.0.1-SNAPSHOT/phanbedder-1.9.7-1.0.1-20140422.184656-7.jar. Return code is: 401, ReasonPhrase: Unauthorized. -> [Help 1]

Normaly, deployment credentials are stored inside your personal maven settings.xml, but obviously this file is not present on cloud CI server. Workarounds exists for Travis-ci. Cloudbees DEV@Cloud has most elegant solution, but I guess no deployment with undisclosed credentials can be done from BuildHive and others.

Now, here goes links to public success build logs for Travis-CI, drone.io, BuildHive and DEV@Cloud. Sadly CircleCI and Codeship does not seem to support public projects.

Mentioned CI services can be webhook triggered, provide integrated browser testing and deployment into popular cloud hosting services like Heroku or Google App Engine. They evolve pretty quickly as they add more integrations as I write... Listing and comparing features here is not worth the effort so check documentation pages.

Everybody has different needs. All my projects are open source hobby projects so I guess I'll go with webhook triggered snapshot builds on BuildHive or I might use Travis-CI to have it with snapshot deployments.

For Maven central release deployment builds, I see only one option - Cloudbees DEV@Cloud. In some next blog post, I'll describe how fully automatic deployment can be achieved using Cloudbees DEV@Cloud, maven-release-plugin and Sonatype OSSRH.

Happy cloud hosted CI builds!

Monday, 21 April 2014

PhantomJS embedder for Selenium GhostDriver

There is quite a few Drivers for Selenium2/WebDriver. Two of them are particulary interesting, because they allow fast headless browser tests - HtmlUnitDriver and PhantomJSDriver.

Using HtmlUnitDriver is piece of cake, because HtmlUnit is pure java library, but disadvantage is limited JavaScript execution support, making it usable for mostly static html sites only.

PhantomJSDriver from GhostDriver project allows to employ PhantomJS, which is a headless WebKit, much much closer to real web browser. As native binary having it's dependencies staticaly linked, PhantomJS does not need any installation. Just unzip it somewhere.

Selenium 2 has annoying habit of needing to specify full path to browser binary when Driver instace is being created. For most common browsers like Chrome or Firefox, some basic discovery is performed, but it usualy fails for me and it is same story with PhantomJS now.

Most probably, you will get folowing exception

java.lang.IllegalStateException: The path to the driver executable must be set by the phantomjs.binary.path capability/system property/PATH variable; for more information, see https://github.com/ariya/phantomjs/wiki. The latest version can be downloaded from http://phantomjs.org/download.html
 at com.google.common.base.Preconditions.checkState(Preconditions.java:177)
 at org.openqa.selenium.phantomjs.PhantomJSDriverService.findPhantomJS(PhantomJSDriverService.java:237)
 at org.openqa.selenium.phantomjs.PhantomJSDriverService.createDefaultService(PhantomJSDriverService.java:182)
 at org.openqa.selenium.phantomjs.PhantomJSDriver.<init>(PhantomJSDriver.java:99)

Another obstacle usualy is different Operating System between developer's machine (MacOS, Windows) and continuous integration server (Linux). Because PhantomJS is native application, therefore every OS needs it's own binary to execute and you'll need to distribute right versions to every host that will ever build/test your project.

To escape from both annoyances, I've created Phanbedder. Tiny library that bundles PhantomJS binaries and unpacks right one for you on any of supported platform - Linux, Windows and Mac OSX.


File phantomjs = Phanbedder.unpack(); //Phanbedder to the rescue!
DesiredCapabilities dcaps = new DesiredCapabilities();
dcaps.setCapability(PhantomJSDriverService.PHANTOMJS_EXECUTABLE_PATH_PROPERTY, phantomjs.getAbsolutePath());
PhantomJSDriver driver = new PhantomJSDriver(dcaps);

//Usual Selenium stuff follows
try {
 driver.get("https://www.google.com");
 WebElement query = driver.findElement(By.name("q"));
 query.sendKeys("Phanbedder");
 query.submit();
 Assertions.assertThat(driver.getTitle()).contains("Phanbedder");
} finally {
 driver.quit();
}

To run code above you'll need following dependencies. Number 1.9.7. in artifactId stands for PhantomJS version bundeled inside.

    
      net.anthavio
      phanbedder-1.9.7
      1.0.0
    

    
      com.github.detro.ghostdriver
      phantomjsdriver
      1.1.0
    

Because this library is targeting various OSes, testing is tricky. I have tested it on Mac OS X 10.6, Windows 7 and I've also gave it a spin on travis-ci (good) and Cloudbees Jenkins (great). Source code is hosted on GitHub and deployed into maven central again using Cloudbees Cloud@DEV

Happy Phanbedding!

Wednesday, 15 January 2014

Deploy to Openshift from Github repository

On Cloudbees RUN@Cloud or Jelastic cloud PaaS, it is simple to build application war on your devbox and then use provided web interface to simply upload and deploy it.

Openshift does not offer such deployment option right away, and similarily to Heroku it encourages you to do builds and deployments via git push commit hook. First I'll do deployment to Openshift encouraged and documented way, but because it actually is possible to deploy externaly built war to Openshift, I'll describe how to perform it too.

I have existing quite simple, Maven built, multimodule application Wotan hosted in Github repository. One of it's submodules - wotan-browser is web application packaged as war. Plan is to deploy this webapp to Openshift.

Create Account and some Gears

Openshift account comes with 3 free Gears (simultaneously running JVMs). I've signed in and then created Gear named "wotan" with "Tomcat 7 (JBoss EWS 2.0)" cartridge. Same can be done using rhc app create wotan jbossews-2.0 command I also added "Jenkins Client" cartridge, which creates Jenkins instance that occupies second Gear. To execute Jenkins build, third and last free Gear is required. Since that escalated quickly, I'll later show how to build web application without Jenkins to save two Gears

In order to simplify authentication, I've configured same ssh public key I'm using for Github on Settings page. I strongly encourage you to do it as well. If it happend that you do not have any ssh key, create one using ssh-keygen or puttygen if you are on Windows.

What Openshift provides

Openshift gives you real linux system account, which is nice. It's big difference comparing to other cloud platforms, where you have only limited interface (web or SDK) to interact with your server and applications.
You can login into...
ssh 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com
...look around for a while...
[wotan-anthavio.rhcloud.com 52d184e45973ca0bc0000088]\> ls -gG
total 20
drwxr-xr-x.  4 4096 Jan 14 15:19 app-deployments
drwxr-xr-x.  4 4096 Jan 11 12:52 app-root
drwxr-xr-x.  3 4096 Jan 11 12:53 git
drwxr-xr-x. 13 4096 Jan 11 12:53 jbossews
drwxr-xr-x.  8 4096 Jan 11 12:57 jenkins-client
...but most interesting directories are...
  • jbossews - Server from Tomcat 7 (JBoss EWS 2.0) cartridge. This is ultimate place where webapps and logs will be. If you have picked another cartridge type for your application, your directory will be different of course
  • git - Git repository with default simple web application
Git repository comes prepopulated with default simple web application. I'm gonna replace it with my Github repository content, but it is worth to at least check it out and look how the vanilla pom.xml looks like.
git clone ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git/
Quite important is .openshift directory because it contains various Openshift metadata and configuration. See documentation page

Default simple application is also deployed on Gear creation and Jenkins instance with preconfigured build Job You can start new build and see on Applications page, how one Gear will become occupied by build process and disappears after while. It is also useful to examine build console output to see what precisely get executed and how - java version, maven parameters, etc...

Deploy using Openshift Jenkins

On my devbox I have my Github repo checked out (On Mac OSX and Windows7 PC with Cygwin)

git clone git@github.com:anthavio/wotan.git
Now merge Github repo into Openshift repo. Steps are taken from stackoverflow so I'll just list commands

git remote add openshift -f ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git/
git merge openshift/master -s recursive -X ours
git push openshift HEAD

Every git push openshift HEAD starts new Jenkins build. It fails at first because pom.xml is not ready for Openshift and tweaks has to be done. Let's examine these tweaks.

Maven pom.xml changes for Openshift

Whole pom.xml can be found here
  • To use dependencies not present in Maven Central repository, additional repository (sonatype-oss-public) has been added
  • You might seen in vanilla Openshift pom.xml slightly changed maven-war-plugin configuration. Contract for deployment is that all deployable artifacts must be present in webapps directory and have nice file name, because this name will became part of url
  • Turn off maven-enforcer-plugin - this might not be necessary for most of developers. I enforce conservative Java6, but Openshift runs on Java7

Skip OpenShift git repository

Actually you need not to use Openshift git repository at all. Just go to the Jenkins Job configuration and change Git repository URL from ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git to use directly Github repository https://github.com/anthavio/wotan.git and forget Openshift git repo forever...

Deploy pre-built war

Now I'll use another web application of mine - Disquo. I'll build war on my devbox and deploy it manualy. No Jenkins involved.
# build war localy
git clone git@github.com:anthavio/disquo.git /tmp/disquo.git
cd /tmp/disquo.git
mvn clean package -Dmaven.test.skip=true

# upload assembled war
scp disquo-web/target/disquo-web-1.0.0-SNAPSHOT.war 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com:~/app-root/repo/webapps/disquo.war
ssh 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com
# stop tomcat & deploy webapp & start tomcat
ctl_app stop jbossews
cp app-root/repo/webapps/disquo.war jbossews/webapps
ctl_app start jbossews

# Check Tomcat log file
tail -f jbossews/logs/catalina.out
Deploying this way has nice effects
  • Jenkins gear and building gear are not required, can be removed and all 3 gears are avaliable for running applications
  • Multiple applications can be deployed into single Tomcat gear
Happy Openshift deployments folks!