Writing your own Java application on Exalogic using SDP

I've written before about how Exalogic enables Oracle Middleware products to use Sockets Direct Protocol (SDP) under the covers, rather than TCP-IP, to achieve lower latency communication over an InfiniBand network. Originally, the capability to leverage SDP was limited to Oracle internal-only APIs in the JRockit JVM (Java 1.6) and thus was only usable by Oracle products like WebLogic.

However, SDP support has now been added as a general capability to Java 1.7 (Hotspot JVM), thus enabling any standalone Java application to be written to take advantage of SDP rather than TCP-IP, over InfiniBand. I found a new tutorial, Lesson: Understanding the Sockets Direct Protocol, describing how to write a Java application that can use SDP, so I gave it a go on an Exalogic X2-2 machine. Below I've recorded the steps that I took to test this, in case it's useful to others.

To leverage SDP from your application, you can still use the same Java socket APIs as normal and simply use a configuration file to indicate that SDP should be employed, not TCP-IP. The tutorial I found shows how to provide the SDP configuration file, but doesn't provide Java code examples to test this. So first of all I quickly wrote Java main classes for a server and a client and tested that they worked correctly on my Linux x86-64 laptop when using just TCP-IP over Ethernet.

If you want to try it out yourself, you can download a copy of the test Java project I wrote from here. Below is the key part of my server class that receives a line of text from the client over a socket, prints this text to the console and then replies with an acknowledgement.

  try (ServerSocket serverSocket = new ServerSocket(port)) {
    info("Running server on port " + port);

    while (true) {
      try (Socket socket = serverSocket.accept();
           BufferedReader in = new BufferedReader(
              new InputStreamReader(socket.getInputStream()));
           PrintWriter out = new PrintWriter(
              new BufferedWriter(new OutputStreamWriter(
                socket.getOutputStream())))) {
        String msg = in.readLine();
        info("Server received message:  " + msg);
        out.println("ACKNOWLEDGED (" + msg + ")");
        out.flush();
      }
    }
  }

And below is the key part of my client class which sends a line of text over a socket to the server and prints out the response text it receives.

  info("Running client connecting to " + host + ":" + port);

  try (Socket socket = new Socket(host, port);
       PrintWriter out = new PrintWriter(new BufferedWriter(
          new OutputStreamWriter(socket.getOutputStream())));
       BufferedReader in = new BufferedReader(
          new InputStreamReader(socket.getInputStream()))) {
    info("Client sent message:  " + SEND_MESSAGE_TEXT);
    out.println(SEND_MESSAGE_TEXT);
    out.flush();
    info("Client received message:  " + in.readLine());
  }

The observant among you will notice that all of above is just standard Java 1.7 using java.net.* & java.io.* APIs. Nothing special.

I then moved the test client and server apps over to two Exalogic compute nodes. Actually the compute nodes were virtualised in this case, rather than physical, with each Oracle Linux OS running as a guest OS (vServer) on top of the OVM hypervisor. As instructed in the tutorial, I added the following JVM arguments to my bash scripts for starting the Java server and client applications so that they can use SDP:

  -Dcom.sun.sdp.conf=/u01/myshare/sdp.conf
  -Djava.net.preferIPv4Stack=true
  -Dcom.sun.sdp.debug

I slipped the com.sun.sdp.debug argument in there too, because that makes the JVM print some information to the console, indicating if SDP is being used by the app. I created the sdp.conf file at the location /u01/myshare/sdp.conf, with the following content:

  bind * *
  connect 192.168.0.101  1234

In the first line I tell the JVM that if an application opens a server socket listening to all local network interface IP addresses, it should use SDP. The second line tells the JVM that if an application opens a new socket to the remote host:port of 192.168.0.101:1234, to use SDP. This is the host:port of one of the network interfaces on the vServer that my Java server will listen to, when it starts.

Then running my wrapper bash scripts to start the Java server main class with its SDP file present, on a vServer, and the Java client class, with its SDP file, on another vServer, I saw the following output:

  [paul@paul_vserver2]$ ./runserver.sh
    BIND to 0.0.0.0:1234 (socket converted to SDP protocol)
    INFO - Running server on port 1234
    INFO - Server received message:  Hello from Java socket client program

  [paul@paul_vserver1]$ ./runclient.sh
    INFO - Running client connecting to 192.168.0.101:1234
    CONNECT to 192.168.0.101:1234 (socket converted to SDP protocol)
    INFO - Client sent message:  Hello from Java socket client program
    INFO - Client received message:  ACKNOWLEDGED (Hello from Java socket client program)

As you can see the client successfully sends the server a message and receives a response. In bold I've highlighted the debug output from the JVM showing that SDP is being used. However, to prove that SDP is indeed being used I did some more analysis. Whilst my Java server class was still running with its server socket listening, I ran the following OS level networking command to see if my running application's SDP listener was present. The output displayed:

  [paul@paul_vserver2]$ sdpnetstat -al | grep sdp
    sdp  0   0 *:search-agent   *:*  LISTEN

This shows my server process running listening on all local IP addresses using SDP. As soon as I kill my server process and run the sdpnetstat command again, no SDP listeners are shown.

To further help prove this, I started up the server again listening on SDP, but in the client vServer, I changed the SDP conf file to have a 'rubbish' connect value to force the client application to use TCP-IP. Upon running the client application again, I see the following error, because the client is trying to use TCP to talk to a remote SDP listener. Also, notice in bold, the JVM debug output showing that no SDP match was found in sdp.conf.

  [paul@paul_vserver1]$ ./runclient.sh
    INFO - Running client connecting to 192.168.0.101:1234
    CONNECT to 192.168.0.101:1234 (no match)
    java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.(Socket.java:425)
        at java.net.Socket.(Socket.java:208)
        at testsocketcomms.Client.sendMessage(Client.java:35)
        at testsocketcomms.Client.main(Client.java:21)

Anyway, that's pretty much it. Want to use SDP from a custom application on Exalogic? Just use standard Java socket programming, specify the right settings in a configuration file and off you go!

In the future I hope to revisit this topic and performance test a bespoke application under load, comparing the performance difference between using TCP-IP over InfiniBand and SDP over InfiniBand. However, developing a 'realistic' performance test application, that doesn't contain other more prevalent bottlenecks, is not a simple task, hence it's not something I could have quickly demonstrated here.


Song for today: Floodlit World by Ultrasound