Certificates with SHA-1 and SunCertPathBuilderException

As SHA-1 is heading to deprecation as hashing algoritm for certificate signatures, unpleasant effects start to appear.

Our partners we need to communicate with over HTTPS have brand new certificate signed by GoDaddy Certificate Authority. Accessing their https secured site via browser does not show anything alarming.

But accessing REST endpoint hosted on same site using java HttpUrlConnection blows up with javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

What is going on?

Every internet browser comes with quite a big set of preinstalled Certificate Authorities (CA) trusted certificates, because your browser's vendors trusts them. And because they are CAs and they are trusted, then every certificate that is signed by them is trusted too.

Same story with JVM. There is truststore file inside every JVM named cacerts. In Oracle jdk1.7.0_67 there is 87 Certificate Authorities in it as they are trused by Oracle. GoDady is there too, so why that SunCertPathBuilderException? Let's examine it more closely.

Every JVM is also shipped with command line tool named keytool. Using it you can list and also modify any keystore in jks format such as cacerts Executing... (default password is changeit)

 
keytool -list -v -keystore ${JAVA_HOME}/jre/lib/security/cacerts -storepass changeit | grep -A 14 godaddy

...will print following...

Alias name: godaddyclass2ca
Creation date: 20-Jan-2005
Entry type: trustedCertEntry

Owner: OU=Go Daddy Class 2 Certification Authority, O="The Go Daddy Group, Inc.", C=US
Issuer: OU=Go Daddy Class 2 Certification Authority, O="The Go Daddy Group, Inc.", C=US
Serial number: 0
Valid from: Tue Jun 29 18:06:20 BST 2004 until: Thu Jun 29 18:06:20 BST 2034
Certificate fingerprints:
  MD5:  91:DE:06:25:AB:DA:FD:32:17:0C:BB:25:17:2A:84:67
  SHA1: 27:96:BA:E6:3F:18:01:E2:77:26:1B:A0:D7:77:70:02:8F:20:EE:E4
  SHA256: C3:84:6B:F2:4B:9E:93:CA:64:27:4C:0E:C6:7C:1E:CC:5E:02:4F:FC:AC:D2:D7:40:19:35:0E:81:FE:54:6A:E4
  Signature algorithm name: SHA1withRSA
  Version: 3

Now cmparing to certificate from HTTPS website... GoDaddyG2 cerificate

There is obvious mismatch. Apart from different certificate name, validity date range also notice that Signature Algorithm is "SHA-256 with RSA". GoDaddy's certificate in JVM is different from the one in use on website, therefore SunCertPathBuilderException.

To fix this, we need to add right (G2) GoDaddy's certificate into JVM cacert keystore. Visiting GoDaddy's certificate repository obvious candidate "GoDaddy Class 2 Certification Authority Root Certificate - G2" can be found there.

wget https://certs.godaddy.com/repository/gdroot-g2.crt
keytool -printcert -file gdroot-g2.crt

Will give us something we saw in website certificate...

Owner: CN=Go Daddy Root Certificate Authority - G2, O="GoDaddy.com, Inc.", L=Scottsdale, ST=Arizona, C=US
Issuer: CN=Go Daddy Root Certificate Authority - G2, O="GoDaddy.com, Inc.", L=Scottsdale, ST=Arizona, C=US
Serial number: 0
Valid from: Mon Aug 31 20:00:00 EDT 2009 until: Thu Dec 31 18:59:59 EST 2037
Certificate fingerprints:
  MD5:  80:3A:BC:22:C1:E6:FB:8D:9B:3B:27:4A:32:1B:9A:01
  SHA1: 47:BE:AB:C9:22:EA:E8:0E:78:78:34:62:A7:9F:45:C2:54:FD:E6:8B
  SHA256: 45:14:0B:32:47:EB:9C:C8:C5:B4:F0:D7:B5:30:91:F7:32:92:08:9E:6E:5A:63:E2:74:9D:D3:AC:A9:19:8E:DA
  Signature algorithm name: SHA256withRSA
  Version: 3

Now just import gdroot-g2.crt into JVM cacerts truststore

sudo keytool -import -alias godaddyg2ca -file gdroot-g2.cer -keystore ${JAVA_HOME}/jre/lib/security/cacerts -storepass changeit

Owner: CN=Go Daddy Root Certificate Authority - G2, O="GoDaddy.com, Inc.", L=Scottsdale, ST=Arizona, C=US
Issuer: CN=Go Daddy Root Certificate Authority - G2, O="GoDaddy.com, Inc.", L=Scottsdale, ST=Arizona, C=US
Serial number: 0
Valid from: Tue Sep 01 01:00:00 BST 2009 until: Thu Dec 31 23:59:59 GMT 2037
Certificate fingerprints:
  MD5:  80:3A:BC:22:C1:E6:FB:8D:9B:3B:27:4A:32:1B:9A:01
  SHA1: 47:BE:AB:C9:22:EA:E8:0E:78:78:34:62:A7:9F:45:C2:54:FD:E6:8B
  SHA256: 45:14:0B:32:47:EB:9C:C8:C5:B4:F0:D7:B5:30:91:F7:32:92:08:9E:6E:5A:63:E2:74:9D:D3:AC:A9:19:8E:DA
  Signature algorithm name: SHA256withRSA
  Version: 3

Extensions: 

#1: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
  CA:true
  PathLen:2147483647
]

#2: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  Key_CertSign
  Crl_Sign
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 3A 9A 85 07 10 67 28 B6   EF F6 BD 05 41 6E 20 C1  :....g(.....An .
0010: 94 DA 0F DE                                        ....
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

Problem solved and REST call should succeed from now using this JVM

For completeness sake, if you want to get rid of it, execute

keytool -delete -alias godaddyg2ca -keystore ${JAVA_HOME}/jre/lib/security/cacerts

What in case you are not allowed to modify JVM cacerts truststore?

Then make a copy of it, import gdroot-g2.cer into it and use this custom truststore instead of default JVM truststore using -Djavax.net.ssl.trustStore=/path/to/custom_cacerts -Djavax.net.ssl.trustStorePassword=changeit java parameters

What in case you need multiple keystores or something similarily complex?

Such scenarios cannot be solved simply by using JVM switches and parameters anymore and you have to roll your own X509TrustManager implementation. Then you need to plug it into your http client connection setup - (HttpsUrlConnection SSLSocketFactory) (Apache HttpClient 3 SecureProtocolSocketFactory) (Apache HttpClient 4 SSLConnectionSocketFactory ) (Jersey SslConfigurator)

Vaadin & Spring Boot & WebSockets & OpenShift & Java8

Spring Boot is around for a while and because Vaadin caught my interest lately, plus Spring4Vaadin appeared, I gave it a spin. To make it more challenging, I decided to use Java 8 and deploy it on OpenShift. All the code is hosted on GitHub

Application is simple chat with OAuth2 sign in using Facebook/Google/GitHub/LinkedIn/Disqus. High-tech is way it broadcasts chat messages to chat participants using server push which in turn uses WebSocket mechanism.

Some informations about Spring Boot and OpenShift are part of the Spring Boot Documentation and some more in this blog post.

I've created DIY gear using OpenShift application console but same can be done using rhc app create vinbudin -t diy-0.1. Application code was already on GitHub and to get it running on OpenShift, following steps must be performed.

1. Add git upstream repository (openshift remote)

git clone git@github.com:anthavio/vinbudin.git
git remote add openshift ssh://your_uuid@vinbudin-yourdomain.rhcloud.com/~/git/vinbudin.git/

2. Add OpenShift build hooks

Build hooks in .openshift/action_hooks are shell scripts that are executed when you push something into OpenShift remote repository. Typically they stop application, run maven/ant/sbt to build application and start it again.

3. Push to the openshift

That's it. When you execute following command, you will see build hooks executed and project will be built and deployed on your openshift gear.

git push openshift master

You can still git push origin master to GitHub repository without invoking build on OpenShift

Maven 3.2.3 and Java 8 on OpenShift

OpenShift gears (28 Oct 2014) have only Java 1.7.0_71 and Maven 3.0.5. If you want to use different, most probably newer versions, just download and unpack them in .openshift/action_hooks/deploy into ${OPENSHIFT_DATA_DIR}

WebSockets on OpenShift

Situation seems to be same as two years ago when this blog post was written. You still have to access port 8000 to get WebSockets working.

Thanks to Vaadin/Atmosphere push implementation which automaticaly downgrade to long-polling when websockets mechanism is unavailable, user experience is not affected, but observing messages exchanged between browser and server, you will easily spot the problem.

Open Chrome Developer Tool Network tab and navigate to http://vinbudin-openshift.anthavio.net/ui

Request URL:ws://vinbudin-openshift.anthavio.net/ui/PUSH/?v-uiId=0&v-csrfToken=731062c6-712d-4320-a92c-3742fe3b4451&X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=2.1.5.vaadin4-jquery&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&X-Cache-Date=0&Content-Type=application/json;%20charset=UTF-8&X-atmo-protocol=true
Request Method:GET
Status Code:501 Not Implemented

WebSocket upgrade request was rejected. Ouch!

Now doing same, but with port 8000 in url - http://vinbudin-openshift.anthavio.net:8000/ui

Request URL:ws://vinbudin-openshift.anthavio.net:8000/ui/PUSH/?v-uiId=0&v-csrfToken=731062c6-712d-4320-a92c-3742fe3b4451&X-Atmosphere-tracking-id=0&X-Atmosphere-Framework=2.1.5.vaadin4-jquery&X-Atmosphere-Transport=websocket&X-Atmosphere-TrackMessageSize=true&X-Cache-Date=0&Content-Type=application/json;%20charset=UTF-8&X-atmo-protocol=true
Request Method:GET
Status Code:101 Switching Protocols

WebSocket upgrade request was accepted and protocol was switched. Hurray!

Happy Vaadin & Spring Boot & OpenShift websocketing!

Logback with rsyslog

In company I work for, developers don't have any access to Live servers. While it is good from the responsibility division reasons, it makes problem and error tracking painful. Most often we need access to log files without asking somebody from operations team every single time.

To overcome this, we went predictable way of setting up company-wide log aggregation server. While there is plethora of solutions, one is quite easy to setup if you are on Unix/Linux - rsyslog It is widely used to provide standard syslog feature so there is good chance that you have it installed already.

I will not talk much about rsyslog server. It listens usualy on UDP port 514 and has configuration file /etc/rsyslog.conf. Excelent article about installation and basic configuration is here.

Any (client host) rsyslog can be configured to forward it's syslog messages into server rsyslog. This might be right way, when you develop native or RoR application and you log into local syslog, but Java world has it's well established logging libraries, namely Logback or Log4j. Both of them provide appanders capable of sending log events into rsyslog server, while logging into local files as usualy.

Anyway, if you are already logging directly into local syslog, and you need to forward messages into log server, just add to the end of /etc/rsyslog.conf file.

*.* @logs.mycompany.com:514

You can use DNS name or IP addres of your rsyslog server. 514 is default rsyslog port.

To log from java directly to rsyslog server, skipping local syslog completely, add SyslogAppender into logback.xml.
Logback does not provide explicit configuration for TAG aka APP-NAME aka $programname, but it can be emulated by using special syntax of the first word of the log message.
It must start with ' ' (whitespace) and end with ':' (colon) See ' mycoolapp:' in suffixPattern element value. Logback configuration feature ticket exist, but it seems to be abandoned. I don't use Log4J, so here is just a link to article about Log4j and syslog. But I don't think it covers TAG usage.

Back on server, log messages from muliple hosts and applications are written, by default, into single /vsr/log/messages file, making it quite messy. Fortunately rsyslog provides easy way of routing messages coming from different hosts or applications into different files.

You can split messages using hostname or ip address they are coming from, as described here or you can split them on log message content, as described here. You can throw them away too, as described here.

What we choose, is routing messages belonging to same application into single log file. Messages are coming from different hosts because application runs in cluster. Every log line contains hostname, so origin still can be recognized.

if $programname == 'mycoolapp' then /var/log/mycoolapp/mycoolapp.log

To make this work, client must send TAG ($programname) field correctly as described above in logback.xml

Happy rsyslogging

nslookup vs /etc/hosts

TLDR: nslookup command does NOT use /etc/hosts for domain name to IP resolution.

Recently I needed to simulate outage of Disqus server providing REST endpoints for commenting.

Because client application, using this service, was deployed into horrible CMS server, changing URL for Disqus service was quite painful. OS on the server is Red Hat Enterprise Linux, so I decided to simply add entry into /etc/hosts translating "disqus.com" domain name to 127.0.0.1 (localhost). Purpose was to hijack DNS resolution into real public Disqus IP address, obtained from DNS server, to make disqus.com calls fail. Dead simple.

I've added line 127.0.0.1 disqus.com into /etc/hosts and executed

nslookup disqus.com

expecting to see the difference, but there wasn't any and nslookup was still returning Disqus public IP address. Suspecting DNS caching for causing this, I've started looking for a way to flush DNS translation cache. Linux standard way seems to have nscd daemon running and refreshing it

nscd -I hosts

should do the trick. But surprisingly, nscd was not running on that server. Neither did any of named, bind, rscd, dnsmasq...

Probably only way to flush dns cache in this situation could be restart of whole networking subsystem (/etc/init.d/network restart), but that wasn't something I could do on that server.

Then, just to be double sure, I executed

ping disqus.com

and spotted that it is pinging 127.0.0.1 as I wanted! But nslookup, executed again, was still showing public Disqus IP address!
As it has turned out that, nslookup is always doing DNS server lookup and it is ignoring your /etc/hosts file

Lesson learned - Don't test your /etc/hosts changes with nslookup. Use ping instead.

It is still some DNS caching done in Java, because ping reflected IP changes immediately, but application running in Java server changed IP after while. I haven't measured it precisely, but it seemed to be about one minute. As it turned out, this delay was actually caused by Varnish and has nothing to do with Java... sorry

All of this happened of RHEL 5.7 (cat /etc/*-release)

PS: Just for completeness. To make sure that /etc/hosts file will take precedence over DNS servers, in process of hostname resolution, following should be true

• File /etc/host.conf should contain line order hosts,bind
• File /etc/nsswitch.conf should contain line hosts: files dns

ConnectTimeoutTest

From time to time I need to simulate one special type of timeout and that is connect timeout.

It differs from more common read timeout, which may happen when connection is already opened, request is sent, but it takes too long to get response. Connection refused is also different beast, meaning that server actively refused your request, probably while you have been accessing wrong host or port. You should not ignore this connect timeout just because it is less common. When it hits, it will get you down you down as quickly as read timeout.

Connect timeout can occur during connection opening process. That is after connection attempt was not refused and before any possible read timeout.

Fairly rare occasions lead to this type of timeout. It may be firewall misconfiguration silently throwing away your packets, network failure or server can be dying down being unable to even to open socket correctly. Any of those conditions is quite hard to meet so when you need to write some piece of software anticipating this type of timeout, some deterministic way to induce it on demand will come handy.

Basic idea is to create ServerSocket with one position long backlog. Backlog is basically request queue and if we make artificial request to fill it up, any consequent request is doomed.

Update: Backlog queue implementation is differs between platforms. This does work on Mac OS X and does NOT on Linux or Windows. Damn it!

Test method uses standard java HttpURLConnection to make the request and it's setConnectionTimeout to set connection timeout to some rasonable value.

Here comes the complete test case:

Oddly enough, different exception (java.net.ConnectException: Operation timed out) is thrown after 75 seconds in case that connection timeout is NOT set, comparing to case when it IS set (java.net.SocketTimeoutException: connect timed out).

Bonus code! Setup code for Apache Httpclient might look like:

HttpConnectionParamBean connectionBean = new HttpConnectionParamBean(httpParams);
connectionBean.setConnectionTimeout(getConnectTimeout());//httpParams.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, 1000L);
connectionBean.setSoTimeout(getReadTimeout());//httpParams.setParameter(CoreConnectionPNames.SO_TIMEOUT, 5000L);
DefaultHttpClient httpClient = new DefaultHttpClient(httpParams);

Enjoy your timeouts!