Using Charles Proxy with Play Framework

Overview

I got Charles Proxy working with Play Framework in such a way that I can record all of the web requests initiated by my Java-based web application. It was very easy to get Charles to intercept HTTP requests emanating directly from the browser; it was less easy getting Charles to intercept HTTP requests originating in the Java code.

Environment

Charles is a web debugging proxy application for Windows, Mac OS, and Linux. You run Charles on your computer, and it logs web traffic on your machine, whether that traffic is HTTP(S) requests initiated by your computer, or HTTP(S) requests received by your computer. You then examine the history of requests, responses and HTTP headers in Charles' user interface. As you can imagine, Charles is an extremely useful tool for web application developers.

Play Framework is a web framework on which you can build a web application in Java or Scala. Play Framework is currently at version 2.4, and that's the version that I'm using.

I'm using Firefox, which is currently at version 43.0.4.

I'm developing the web application on Mac OS X 10.9.5, and this blog post therefore relates to that operating system.

Installing Charles on Mac OS

Installing Charles is pretty easy. Just follow the directions in the documentation. Under Browser & System Configuration, the doco says this about installation on Mac OS:

"When you first install Charles you will be prompted to grant permissions to Charles to autoconfigure the proxy settings. After that, Charles will configure and then reconfigure the Mac OS X proxy settings whenever Charles is started or quit."

In other words, when you start Charles it automatically wedges itself into the networking components of your computer so that HTTP traffic originating in Safari and other Mac OS applications passes through Charles on its way out of your computer. After starting Charles, you'll be able to see the proxy enabled in your network settings:

network-proxy.png

Getting Charles to intercept HTTP requests from your browser

If you visit a web page in Safari you'll see the HTTP GET request, and the corresponding HTTP response listed in the log inside Charles.

If you're using Charles with Firefox, you need to ensure that the requisite plug-in is installed in that browser. The Charles configuration doco says:

"When you first install Charles, it will check for and prompt you to install the Firefox add-on. If you don’t install the Firefox add-on immediately you will later need to enable Firefox configuration in the Proxy Settings in the Proxy menu."

The good news is that once the plug-in has been installed in Firefox, you can forget about it. Whenever Charles starts it activates the Firefox plug-in, and when you quit Charles the Firefox plug-in is automatically deactivated. Sweet!

What about HTTPS?

The Charles documentation explains this issue better than I could. In short, Charles installs its own Root Certificate in the trusted certificate store on your machine so that it can act as a 'man in the middle' of HTTPS requests that your computer makes to particular websites.

To enable HTTPS man-in-the-middl-ing for a particular website, right-click on a host name in Charles' structure view and turn on SSL Proxying.

Getting Charles to intercept HTTP requests from the JVM running your Play Framework application

So far so good. If you type a URL into the address bar of your browser, a request is logged in Charles. If your Play Framework application fires off an HTTP redirect (302) request, which goes via your browser, a request is logged in Charles. If you use cURL or HTTPie or something similar to manually fire off an HTTP request, it is likewise logged in Charles.

But...

if your Java code running inside Play Framework fires off an HTTP request using the Play Framework web services API (WS API), then you won't see that HTTP request in Charles unless you do some more configuration work.

You need to follow the instructions under 'JAVA APPLICATIONS' on the SSL Certificates page in the Charles documentation.

Download Charles Root Certificate

In Charles go to the Help menu and choose "SSL Proxying > Save Charles Root Certificate". Save the root certificate (as a .crt) to ~/Desktop as charles-ssl-proxying-certificate.crt.

Verify the location of the Java cacerts file

In a terminal window, run the following command:

find $JAVA_HOME -name cacerts

It will return something like this:

/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts

Install the Charles Root Certificate in Java's key store

sudo keytool -import -alias charles \
-file ~/Desktop/charles-ssl-proxying-certificate.crt \
-keystore $JAVA_HOME/jre/lib/security/cacerts -storepass changeit

Note that the default keystore password is 'changeit', so that's the storepass value you should use unless you've explicitly changed it sometime in the past.

Verify that the certificate was installed

keytool -list -keystore $JAVA_HOME/jre/lib/security/cacerts \
-storepass changeit | grep charles

Set the HTTP_PROXY environment variable

Now set the HTTP_PROXY environment variable, as explained under 'Proxy Setup' in the Play Framework installation instructions.

export HTTP_PROXY=http://127.0.0.1:8888

Run the activator

You can now run the activator as normal. All being well, you'll find that Java-initiated HTTP requests are logged in Charles.