Friday, 17 January 2014

Openshift System properties and Environment variables

For cloud deployable application, it is advisable to create them as much self-contained as possible. If application allows some configuration, these settings should be packaged inside war file and used as default values, while value override mechanism is provided.

Linux Environment variables

Unfortunately, following (usual) way of setting environment variable will not work on Openshift.
ctl_app stop jbossews
export MY_ENV_VAR="my_env_var_value"
ctl_app start jbossews
To be precise, JAVA_OPTS_EXT variable set like this, will not visible to java process running your application. To set it properly, you have to do it using Openshift rhc tool or using shortcut
echo "my_env_var_value" > ~/.env/user_vars/MY_ENV_VAR
An now you can in your webapp get variable value
  String my_env_var = System.getenv("MY_ENV_VAR"); //"my_env_var_value"
  if (my_env_var == null) {
    my_env_var = "Default value..."; //I told you not to depend on external configuration and provide default values...
  }

Java System Properties

System Properties are most commonly used to pass parameters into Java application. They are specified as java "-D" prefixed command-line parameters:

java -Dmyapp.param=whatever ...
then they can be obtained inside application
String param = System.getProperty("myapp.param"); //"whatever"

But Openshift managed servers are started using ctl_app start ... (or gears start ...) command and we are not directly in control of executing java so we can't add "-D" parameters.

Trick is to use JAVA_OPTS_EXT evironment variable. Since we know how to set Openshift Environment variable, it is just matter of:

echo "-Dmyapp.param=myapp.value" > .env/user_vars/JAVA_OPTS_EXT
...then restart your server...
ctl_app stop jbossews
ctl_app start jbossews
...and enjoy fruits of your effort (look for -Dmyapp.param=myapp.value)...
[wotan-anthavio.rhcloud.com 52d184e45973ca0bc0000088]\> jps -mlv
29444 sun.tools.jps.Jps -mlv -Dapplication.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45 -Xms8m
24633 org.apache.catalina.startup.Bootstrap start -Xmx256m -XX:MaxPermSize=102m -XX:+AggressiveOpts -DOPENSHIFT_APP_UUID=52d184e45973ca0bc0000088 -Djava.util.logging.config.file=/var/lib/openshift/52d184e45973ca0bc0000088/jbossews//conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Dmyapp.param=myapp.value -Dcatalina.base=/var/lib/openshift/52d184e45973ca0bc0000088/jbossews/ -Dcatalina.home=/var/lib/openshift/52d184e45973ca0bc0000088/jbossews/ -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/lib/openshift/52d184e45973ca0bc0000088/jbossews//tmp

Wednesday, 15 January 2014

Deploy to Openshift from Github repository

On Cloudbees RUN@Cloud or Jelastic cloud PaaS, it is simple to build application war on your devbox and then use provided web interface to simply upload and deploy it.

Openshift does not offer such deployment option right away, and similarily to Heroku it encourages you to do builds and deployments via git push commit hook. First I'll do deployment to Openshift encouraged and documented way, but because it actually is possible to deploy externaly built war to Openshift, I'll describe how to perform it too.

I have existing quite simple, Maven built, multimodule application Wotan hosted in Github repository. One of it's submodules - wotan-browser is web application packaged as war. Plan is to deploy this webapp to Openshift.

Create Account and some Gears

Openshift account comes with 3 free Gears (simultaneously running JVMs). I've signed in and then created Gear named "wotan" with "Tomcat 7 (JBoss EWS 2.0)" cartridge. Same can be done using rhc app create wotan jbossews-2.0 command I also added "Jenkins Client" cartridge, which creates Jenkins instance that occupies second Gear. To execute Jenkins build, third and last free Gear is required. Since that escalated quickly, I'll later show how to build web application without Jenkins to save two Gears

In order to simplify authentication, I've configured same ssh public key I'm using for Github on Settings page. I strongly encourage you to do it as well. If it happend that you do not have any ssh key, create one using ssh-keygen or puttygen if you are on Windows.

What Openshift provides

Openshift gives you real linux system account, which is nice. It's big difference comparing to other cloud platforms, where you have only limited interface (web or SDK) to interact with your server and applications.
You can login into...
ssh 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com
...look around for a while...
[wotan-anthavio.rhcloud.com 52d184e45973ca0bc0000088]\> ls -gG
total 20
drwxr-xr-x.  4 4096 Jan 14 15:19 app-deployments
drwxr-xr-x.  4 4096 Jan 11 12:52 app-root
drwxr-xr-x.  3 4096 Jan 11 12:53 git
drwxr-xr-x. 13 4096 Jan 11 12:53 jbossews
drwxr-xr-x.  8 4096 Jan 11 12:57 jenkins-client
...but most interesting directories are...
  • jbossews - Server from Tomcat 7 (JBoss EWS 2.0) cartridge. This is ultimate place where webapps and logs will be. If you have picked another cartridge type for your application, your directory will be different of course
  • git - Git repository with default simple web application
Git repository comes prepopulated with default simple web application. I'm gonna replace it with my Github repository content, but it is worth to at least check it out and look how the vanilla pom.xml looks like.
git clone ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git/
Quite important is .openshift directory because it contains various Openshift metadata and configuration. See documentation page

Default simple application is also deployed on Gear creation and Jenkins instance with preconfigured build Job You can start new build and see on Applications page, how one Gear will become occupied by build process and disappears after while. It is also useful to examine build console output to see what precisely get executed and how - java version, maven parameters, etc...

Deploy using Openshift Jenkins

On my devbox I have my Github repo checked out (On Mac OSX and Windows7 PC with Cygwin)

git clone git@github.com:anthavio/wotan.git
Now merge Github repo into Openshift repo. Steps are taken from stackoverflow so I'll just list commands

git remote add openshift -f ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git/
git merge openshift/master -s recursive -X ours
git push openshift HEAD

Every git push openshift HEAD starts new Jenkins build. It fails at first because pom.xml is not ready for Openshift and tweaks has to be done. Let's examine these tweaks.

Maven pom.xml changes for Openshift

Whole pom.xml can be found here
  • To use dependencies not present in Maven Central repository, additional repository (sonatype-oss-public) has been added
  • You might seen in vanilla Openshift pom.xml slightly changed maven-war-plugin configuration. Contract for deployment is that all deployable artifacts must be present in webapps directory and have nice file name, because this name will became part of url
  • Turn off maven-enforcer-plugin - this might not be necessary for most of developers. I enforce conservative Java6, but Openshift runs on Java7

Skip OpenShift git repository

Actually you need not to use Openshift git repository at all. Just go to the Jenkins Job configuration and change Git repository URL from ssh://52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com/~/git/wotan.git to use directly Github repository https://github.com/anthavio/wotan.git and forget Openshift git repo forever...

Deploy pre-built war

Now I'll use another web application of mine - Disquo. I'll build war on my devbox and deploy it manualy. No Jenkins involved.
# build war localy
git clone git@github.com:anthavio/disquo.git /tmp/disquo.git
cd /tmp/disquo.git
mvn clean package -Dmaven.test.skip=true

# upload assembled war
scp disquo-web/target/disquo-web-1.0.0-SNAPSHOT.war 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com:~/app-root/repo/webapps/disquo.war
ssh 52d184e45973ca0bc0000088@wotan-anthavio.rhcloud.com
# stop tomcat & deploy webapp & start tomcat
ctl_app stop jbossews
cp app-root/repo/webapps/disquo.war jbossews/webapps
ctl_app start jbossews

# Check Tomcat log file
tail -f jbossews/logs/catalina.out
Deploying this way has nice effects
  • Jenkins gear and building gear are not required, can be removed and all 3 gears are avaliable for running applications
  • Multiple applications can be deployed into single Tomcat gear
Happy Openshift deployments folks!

Thursday, 28 November 2013

Generic throws

Suprisingly few developers really master Java generics in their full extent. As apparently not being one of them, generic throws feature surprised me lately.

Following code snippet shows the idea - generic <X extends Exception> is used in throws part of go() method. Implementing class defines Exception subclass that will be allowed to be thrown.

import java.io.FileNotFoundException;
interface Something<X extends Exception> {
  
 public abstract void go() throws X;
  
}
 
class SomethingImpl implements Something<FileNotFoundException> {
  
 public void go() throws FileNotFoundException {
  throw new FileNotFoundException("Yeah! Not found!");
 }
}

Interesting trick, but how can this be useful? Well, for example, you can pimp Strategy pattern with cunning Exception passing/handling!

In Strategy pattern, you basicaly have two choices how to deal with exceptions
  • Add throws Exception on both strategy interface and context method to pass every exception back to client - especially poor choice
  • Add throws Exception on strategy interface method, catch it inside context and rethrow it wrapped inside RuntimeExcetpion back to client - usual solution, but it mixes strategy and context exceptions together
But with generic throws, aditional options are available.

This is interface allowing Client to implement his own LoaderStrategy and it also allows him to choose Exception which can be thrown from his LoaderStrategy via generic throws X

/**
 * Strategy interface - To be implemented by client
 */
interface LoaderStrategy<X extends Exception> {

 public String load(long id) throws X;

}

And this is simple Context using LoaderStrategy interface. It is also passing chosen exceptions from strategy back to client, again via generic throws X

/**
 * Context using LoaderStrategy, passing selected exceptions (X)
 */
class StrategyContext {

 public <X extends Exception> void perform(LoaderStrategy<X> oader) throws X {

  long resourceId = getIdFromSomewhere();

  String data = loader.load(resourceId); //throws X - don't have to hadle exception here

  try {
   sendDataToRemoteSystem(data);
  } catch (RemoteException rx) {
   //Other checked Exception must be wrapped inside ContextException 
   throw new ContextException("Failed to send data for resourceId " + resourceId, rx);
  }
 }

 private long getIdFromSomewhere() {
  return System.currentTimeMillis(); //CurrentTimeMillis is best id ever!
 }

 private void sendDataToRemoteSystem(String data) throws RemoteException {
  //invoke some remoting operation here...
 }
}

/**
 * Context exception wrapper - delivers other then LoaderStrategy exceptions to client 
 */
class ContextException extends RuntimeException {

 public ContextException(String string, Exception cause) {
  super(string, cause);
 }
}

Use case 1 - Passing checked Exceptions

This is client's LoaderStrategy implementation. It throws IOException
/**
 * Client provided LoaderStrategy implementation throwing IOException
 */
class ReaderLoader implements LoaderStrategy<IOException> {

 private Reader reader;

 public ReaderLoader(Reader reader) {
  this.reader = reader;
 }

 public String load(long id) throws IOException {
  String sid = String.valueOf(id);
  BufferedReader br = new BufferedReader(reader);
  String line = null;
  while ((line = br.readLine()) != null) {
   if (line.startsWith(sid)) {
    return line;
   }
  }
  return null;
 }
}
Then IOException is propagated back into client code to be handeled here
 StrategyContext context = new StrategyContext();

 ReaderLoader readerLoader = new ReaderLoader(new StringReader("Blah! Blah! Blah!"));
 try {
  context.perform(readerLoader);
 } catch (IOException iox) {
  //ReaderLoader thrown IOException is propagated through StrategyContext and delivered here
 } catch (ContextException cx) {
  //StrategyContext thrown exception wrapper - All sorts of NON ReaderLoader originated Exceptions
  Throwable cause = cx.getCause();
 }
This does not brings much value, but in client code we can easily distinguish our ReaderLoader originated exceptions from Context originated ones and handle them differently. Simple two catch clauses.

Use case 2 - Customized checked Exceptions

Client also provides custom Exception he want to be thrown from his Strategy and propagated through StrategyContext back to client
/**
 * Client provided custom Exception
 */
class InvalidRecordCountException extends Exception {

 private long recordId;

 public InvalidRecordCountException(long recordId, String message) {
  super(message);
  this.recordId = recordId;
 }

 public InvalidRecordCountException(long recordId, SQLException exception) {
  super(exception);
  this.recordId = recordId;
 }

 public long getRecordId() {
  return recordId;
 }
}

/**
 * Client provided LoaderStrategy implementation throwing custom InvalidRecordCountException
 */
class JdbcLoader implements LoaderStrategy<InvalidRecordCountException> {

 private DataSource dataSource;

 private String select;

 public JdbcLoader(DataSource dataSource, String select) {
  this.dataSource = dataSource;
  this.select = select;
 }

 @Override
 public String load(long id) throws InvalidRecordCountException {
  Connection connection = null;
  PreparedStatement statement = null;
  ResultSet resultSet = null;
  try {
   connection = dataSource.getConnection();
   statement = connection.prepareStatement(select);
   statement.setLong(1, id);
   resultSet = statement.executeQuery();
   if (resultSet.next()) {
    String string = resultSet.getString(1);
    if (resultSet.next()) {
     //here we go...
     throw new InvalidRecordCountException(id, "Too many somethings in somewhere!");
    }
    return string;
   } else {
    //here we go...
    throw new InvalidRecordCountException(id, "Not a single something in somewhere!");
   }
  } catch (SQLException sqlx) {
   //here we go...
   throw new InvalidRecordCountException(id, sqlx);
  } finally {
   //TODO close resultSet, statement, connection
  }

 }
}
Then SQLExceptions are handled while InvalidRecordCountException propagated
 StrategyContext context = new StrategyContext();

 DataSource dataSource = null; //get it from somewhere
 JdbcLoader jdbcLoader = new JdbcLoader(dataSource, "SELECT something FROM somewhere WHERE column = ?");
 try {
  context.perform(jdbcLoader);
 } catch (InvalidRecordCountException ircx) {
  //JdbcLoader thrown InvalidRecordCountException is propagated through StrategyContext and delivered here
  long badRecordId = ircx.getRecordId(); //we can take some action when knowing failing record id
 } catch (ContextException cx) {
  //StrategyContext thrown exception wrapper - SQLException will be cause propably...
  Throwable cause = cx.getCause();
 }
Now we can easily pass additional error information inside custom exception.

Well, I understand that nobody loves to write lots of single purpose exceptions, but adopting "generic throws idiom" quite increases exception reusability.

All used code can by found in kitchensink repository

Wednesday, 27 November 2013

How many javax.annotation jars is out there?!?!

Today I've me very strange Exception. Here comes it's mighty stacktrace.

Caused by: java.lang.SecurityException: class "javax.annotation.Nullable"'s signer information does not match signer information of other classes in the same package
 at java.lang.ClassLoader.checkCerts(ClassLoader.java:806)
 at java.lang.ClassLoader.preDefineClass(ClassLoader.java:487)
 at java.lang.ClassLoader.defineClassCond(ClassLoader.java:625)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
 at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
 at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
 at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)


Application is Maven based and there are loads of transitive dependencies. Time for good classpath hunt!


Going through classpath revealed that only one javax.annotation.Nullable class is present in com.google.code.findbugs:jsr305:1.3.9 dependency. Aparently it is not a case of having multiple class definition, which happens quite often with javax.servlet API classes.
Searching further I found another dependency, with same javax.annotation package, but with diferent (jsr250) classes inside - org.eclipse.jetty.orbit:javax.annotation:1.1.0.v201108011116 This jar is special. It is (jarsigner) signed with certificate to guard it's content.
Case is clear now, package javax.annotation from inside of signed jsr250 jar is protected from being tampered and classes from unsigned jsr305 jar, having same package, are considered malicious as they are trying to sneak in signed jar's package. JVM performs check when such classes are loaded and eventually throws SecurityException.

My solution was simply to exclude com.google.code.findbugs:jsr305 dependency, because it was not really needed. When this is not an option, then signed orbit org.eclipse.jetty.orbit:javax.annotation dependency must be excluded and use some another (unsigned) added instead. Truly annoying and googling around revealed that poor package javax.annnotation is almost same transitive classpath blight as infamous commons-logging.jar

Short summary of javax.annnotation libraries

JSR 250: Common Annotations for the JavaTM Platform

  • javax.annotation (Generated, ManagedBean , PostConstruct, PreDestroy, Resource, Resources)
  • javax.annotation.security (DeclareRoles, DenyAll, PermitAll, RolesAllowed, RunAs) 
  • javax.annotation.sql (DataSourceDefinition, DataSourceDefinitions) - since version 1.1 
Usual Maven suspects:
  • jsr250-api.jar - http://mvnrepository.com/artifact/javax.annotation/jsr250-api
  • javax.annotation-api.jar - http://mvnrepository.com/artifact/javax.annotation/javax.annotation-api
  • Geronimo - http://mvnrepository.com/artifact/org.apache.geronimo.specs/geronimo-annotation_1.0_spec
  • Glassfish - http://mvnrepository.com/artifact/org.glassfish/javax.annotation
  • Jetty Orbit - http://mvnrepository.com/artifact/org.eclipse.jetty.orbit/javax.annotation
  • And loads of others - http://mavenhub.com/c/javax/annotation/postconstruct

JSR 305: Annotations for Software Defect Detection

  • javax.annotation (CheckForNull, ... , Nonnull, Nullable, ... , WillNotClose)
  • javax.annotation.concurrent (GuardedBy, ... , ThreadSafe)
  • javax.annotation.meta (Exclusive, ... , When)
Usual Maven suspects:
Oddly enough, Findbugs packaged jar is only Maven distributed version
  • jsr305.jar http://mavenhub.com/c/javax/annotation/nullable/jar 


Early version of jsr299 (WebBeans, later CDI) tried to jump into javax.annotation package too, but fortunately were kicked out
  • http://mavenhub.com/c/javax/annotation/named
  • javax.annotation.Named
  • javax.annotation.NonBinding
  • javax.annotation.Stereotype 
But wait, there is more!
For example obscure jsr308. And starting with Java 6, stub of jsr250 (Generated, PostConstruct, PreDestroy,Resource, Resources) is part of standard library -http://docs.oracle.com/javase/6/docs/api/javax/annotation/package-summary.html
Well this is real mess

Thursday, 25 April 2013

JDOM2 namespaces

JDOM2 is relatively new (2012) and it is replacing ancient, pre Java 5 generic, JDOM1 version. You can find nice set of basic examples here.

My requirement was to generate xml with single namespace without prefix. Quite common task you can say, but this happend to be little bit tricky. I thought that namespace, once set on Element, should be inherited by child Elements. Of course unless you intentionaly set another namespace on some child. I was wrong!

You must explicitly set very same namespace over and over again on every Element you create. Very annoying indeed. There is a page dedicated to namespace scoping but call me stupid, it haven't help me single bit.

Luckily, after you are finished with creating elements, you can "namespacize" them in one shot with following simple recursive method...

import org.jdom2.Element;
import org.jdom2.Namespace;

public static void setNamespace(Element element, Namespace namespace) {
 element.setNamespace(namespace);
 List children = element.getChildren();
 for (Element child : children) {
  setNamespace(child, namespace);
 }
}
Happy new namespaces!

Thursday, 18 April 2013

Building Fluent Builder

Recently I've rediscovered another wheel again, but this one is pretty interesting I think...

Simple Fluent Builder

Builder is rather simple desing pattern. It accumulates bunch of values and then uses them to construct some complex product.

Fluent builder is a builder with fluent interface. That means it returns reference to self from every method (except the build() method that returns product) so it allows method chaining.
Nice and simple example is StringBuilder from java standard library.

String hello = new StringBuilder()
  .append("h").append("e").append("l").append("l").append("o")
  .toString();

Generic getThis() trick

When Builder is single standalone class, using return this is dead simple solution, but when inheritance comes into play, builder base class methods cannot return this, because we need concrete builder to be returned to allow method chaining.

Solution is the getThis() trick, that allows to return reference to concrete builder even from base class that builder is extending. The price is that every concrete builder must implement getThis() method.

Here is a nice article with some background and explanation

Generic parent trick

I've found another slightly different scenario, when similar generic trick can be used. It employs two builders and uses generic parent (instead of generic self) to keep track of builders used in chaining. It is quite simple JSON builder.

JSON message is built on two structures: object and array, therefore we will have two concrete builders working together. We don't want builder user to be confused by offering object building methods to him when he constructs array and contrariwise. When user completes constructing array or object, previous builder must be restored and returned to user.

This builder completely prevents user from making wrong method call at any point of building.

JSR-353 - JSON-P

Brand new Java API for JSON processing is going to be part of the upcoming JEE 7 specification also contains JSON builder.
Surprisinly to me they came with simplest builder that is throwing exception when user calls object/array builder method out of correct object/array building context.
For example this code
JsonGenerator jg = javax.json.Json.createGenerator(System.out);
jg.writeStartObject().write("array_element").writeEnd();
executed will throw
javax.json.stream.JsonGenerationException: write(String) can only be called in array context
Very error prone API indeed! I've filed improvement into Jira, but since JSR already passed the Final Approval Ballot, I guess there is no hope.

Tuesday, 5 March 2013

nslookup vs /etc/hosts

TLDR: nslookup command does NOT use /etc/hosts for domain name to IP resolution.

Recently I needed to simulate outage of Disqus server providing REST endpoints for commenting.

Because client application, using this service, was deployed into horrible CMS server, changing URL for Disqus service was quite painful. OS on the server is Red Hat Enterprise Linux, so I decided to simply add entry into /etc/hosts translating "disqus.com" domain name to 127.0.0.1 (localhost). Purpose was to hijack DNS resolution into real public Disqus IP address, obtained from DNS server, to make disqus.com calls fail. Dead simple.

I've added line 127.0.0.1 disqus.com into /etc/hosts and executed

nslookup disqus.com
expecting to see the difference, but there wasn't any and nslookup was still returning Disqus public IP address. Suspecting DNS caching for causing this, I've started looking for a way to flush DNS translation cache. Linux standard way seems to have nscd daemon running and refreshing it
nscd -I hosts
should do the trick. But surprisingly, nscd was not running on that server. Neither did any of named, bind, rscd, dnsmasq...

Probably only way to flush dns cache in this situation could be restart of whole networking subsystem (/etc/init.d/network restart), but that wasn't something I could do on that server.

Then, just to be double sure, I executed

ping disqus.com
and spotted that it is pinging 127.0.0.1 as I wanted! But nslookup, executed again, was still showing public Disqus IP address!
As it has turned out that, nslookup is always doing DNS server lookup and it is ignoring your /etc/hosts file

Lesson learned - Don't test your /etc/hosts changes with nslookup. Use ping instead.

It is still some DNS caching done in Java, because ping reflected IP changes immediately, but application running in Java server changed IP after while. I haven't measured it precisely, but it seemed to be about one minute. As it turned out, this delay was actually caused by Varnish and has nothing to do with Java... sorry

All of this happened of RHEL 5.7 (cat /etc/*-release)

PS: Just for completeness. To make sure that /etc/hosts file will take precedence over DNS servers, in process of hostname resolution, following should be true
  • File /etc/host.conf should contain line order hosts,bind
  • File /etc/nsswitch.conf should contain line hosts: files dns