Anatomy of WebLogic TLOGs and considerations for transaction recovery

Introduction

As discussed in an earlier blog entry, the preservation and ability to restore WebLogic Transaction Logs (TLOGs) is often critical to the availability of an I.T. system and the consistency of the data it contains. With the loss of a machine hosting a WebLogic server, it may be necessary to re-create the environment on another machine, to re-start the failed WebLogic server. Sometimes it may even be necessary to start the failed Weblogic server in a different data-centre for disaster recovery purposes. Restoring a WebLogic server to a running state may be necessary to enable stuck transactions, recorded in its TLOG, to be pushed through to completion. The new host environment may have been reproduced from a previous backup, from re-running WLST scripts or just from manual re-creation. However, what if the new environment uses different hostnames, different IP addresses or different resource names? Will it be possible to reconcile the stuck transactions recorded in the old preserved TLOG? Will it be possible for the WebLogic instance, in the new environment, to successfully commit these stuck transactions?

In this blog entry, I examine the anatomy of a TLOG, to discover what environmental dependencies the data stored in a TLOG has. This will help me to formulate some recommendations outlining the configuration settings that can be changed, when moving a WebLogic server and its TLOG to a new host environment, without losing the ability to recover stuck transactions.

Anatomy of a TLOG

WebLogic's TLOG holds records of the state of in-flight transactions, that are marked to be committed, in something that WebLogic calls a 'persistent store'. The persistent store can either be files on a file-system or a table in database. The key factors to consider when planning the location of the persistent store are:

• the latency of writing to the storage
• the availability and resiliency of the storage

Whichever type of persistent store is used, the actual data stored is essentially the same. For the purposes of this investigation, we will look at the anatomy of a TLOG that uses a file store, but the principles gleaned will be exactly the same for database stores too.

To be able to study the contents of a TLOG, I needed to simulate a real system, processing distributed transactions, that would cause WebLogic to record data in its TLOG. I set-up an appropriate environment on my Linux x86-64 laptop, where I created a WebLogic version 12.1.1 domain with a single server defined, leaving the TLOG location as the default file-store. I created an Oracle version 11.2.0.1.0 single instance database with appropriate settings made to allow XA to function correctly. In the WebLogic domain I defined an XA data source to point to this database. I also created a WebLogic hosted JMS queue which is XA-enabled. Finally, I deployed a test application that I wrote, called BoxBurner, to the WebLogic server to help generate XA transactions for my tests.

BoxBurner is essentially a JEE application that continuously performs a repeated flow of steps. Each flow involves reading a message from a queue, performing a database insert operation, and then placing a new message back on to the same queue, ready for the next iteration. Each flow, composed of 'message dequeue', 'database insert' and 'message enqueue' operations is contained within a single distributed XA transaction. As a result, for each transaction being processed, WebLogic writes a commit decision to its TLOG. In BoxBurner, I also provide a small HTML user interface to allow a user to initiate seeding the queue with a number of messages to kick the continuous parallel flows off.  So with BoxBurner, I can easily simulate a highly transactional system under load, to assist in exploring the anatomy of a TLOG.

By default, the location of a WebLogic server's file-store containing the TLOG is at:

 <domainpath>/servers/<servername>/data/store/default/_WLS_<servername>000000.DAT

WebLogic rolls these log files over, under certain circumstances, incrementing the number in the end of the file name. Each file has a fixed size (approximately 1 MB) and when full (or when a checkpoint occurs), a new file is rolled over to. In my case, the server's initial TLOG data file is called:

  _WLS_MYDEMOADMSVR000000.DAT

So for my tests, running BoxBurner under load, on my laptop, I issued a kill -9 on the WebLogic process to terminate it without any warning, hoping to catch at least a few transactions in-flight. I then studied the state of the TLOG data file to see what transactions were recorded and stuck at the point when the failure occurred.

To study the contents of the TLOG data file, I ran a WebLogic dump utility that shows a summary of the important contents of the TLOG, using a small wrapper Bash script I'd created (dumptlogtext.sh):

 #!/bin/sh
 WEBLOGIC_HOME=/opt/oracle/Middleware/wlserver_12.1
 WEBLOGIC_JAR=$WEBLOGIC_HOME/server/lib/weblogic.jar
 WEBLOGIC_TX_JAR=$WEBLOGIC_HOME/../modules/com.bea.core.transaction_3.0.0.0.jar
 java -cp $WEBLOGIC_JAR:$WEBLOGIC_TX_JAR weblogic.transaction.internal.StoreTransactionLoggerImpl $*

I ran the script with the following arguments to tell it where to find the TLOG data file (note, if you run it with no arguments, the utility displays some useful help information):

 $ ./dumptlogtext.sh /u01/MyDemoSystem/stores/MyDemoAdmSvr_Stores MYDEMOADMSVR

The output essentially showed two different types of record, contained in the TLOG file:

• One checkpoint record, with the following example output below:

  | Class Name = weblogic.transaction.internal.ResourceCheckpoint                |
  | Object = ResourceCheckpoint={{BBDatasource_MyDemoDomain, props={}}, {WLStore |
  | _MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore, props={}}}                     |
  +------------------------------------------------------------------------------+

• Seven transaction records, each with output similar to the example below:

  | Class Name = weblogic.transaction.internal.ServerTransactionImpl             |
  | Object = Name=[EJB boxburner.msgprocessor.MsgProcessorMDB.onMessage(javax.jm |
  | s.Message)],Xid=BEA1-1800B10D6C88705836D9(381221099),Status=Active,numReplie |
  | sOwedMe=0,numRepliesOwedOthers=0,seconds since begin=1775,seconds left=30,XA |
  | ServerResourceInfo[WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore]=(Ser |
  | verResourceInfo[WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore]=(state= |
  | new,assigned=none),xar=null,re-Registered = false),XAServerResourceInfo[BBDa |
  | tasource_MyDemoDomain]=(ServerResourceInfo[BBDatasource_MyDemoDomain]=(state |
  | =new,assigned=none),xar=null,re-Registered = false),SCInfo[MyDemoDomain+MyDe |
  | moAdmSvr]=(state=active),properties=({weblogic.transaction.name=[EJB boxburn |
  | er.msgprocessor.MsgProcessorMDB.onMessage(javax.jms.Message)]}))             |
  +------------------------------------------------------------------------------+

The checkpoint record is made by WebLogic to enable it to track the different XA resources (eg. a database, a message queue) that have been incorporated in one or more XA trasactions that the WebLogic server has participated in. In this checkpoint record, we can see that the two XA resources are 'BBDatasource_MyDemoDomain' (the WebLogic datasource pointing at the database) and 'WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore' (the persistent store for the JMS queue).

The TLOG contained 7 pending transactions - XA transactions that WebLogic has marked for commit, but that may not yet have been committed in one or more of the back-end systems. Each transaction has a global ID (Xid), and in the example transaction shown, the WebLogic generated Xid is "BEA1-1800B10D6C88705836D9".

While the Weblogic server was still shut-down with pending transactions, I used SQL*Plus to connect to the database, and query the list of XA transactions that the database was tracking as pending and yet to commit. The query I issued was:

  SELECT LOCAL_TRAN_ID, GLOBAL_TRAN_ID, STATE, MIXED, HOST, COMMIT# 
  FROM DBA_2PC_PENDING

The outputted list of database pending transaction records was shown to be:

  LOCAL_TRAN_ID GLOBAL_TRAN_ID                 STATE    MIX HOST       COMMIT#
  ------------- ------------------------------ -------- --- ---------- ----------
  3.11.827      48801.1803B10D6C88705836D9     prepared no  pdthinkpad 1310237
  5.38.95       48801.1802B10D6C88705836D9     prepared no  pdthinkpad 1310234
  7.22.28       48801.1805B10D6C88705836D9     prepared no  pdthinkpad 1310241
  11.73.12      48801.1800B10D6C88705836D9     prepared no  pdthinkpad 1310235
  12.21.37      48801.1801B10D6C88705836D9     prepared no  pdthinkpad 1310228
  2.2.534       48801.1799B10D6C88705836D9     prepared no  pdthinkpad 1310236

Note: In my experience it can take a minute or two, from the point when failure occurs, for pending transactions to appear in the database's DBA_2PC_PENDING table. Therefore if you try this out yourself, and you see no records reported, but you expected to see some, wait a few minutes before executing the query again.

As you can see, the "BEA1-1800B10D6C88705836D9" recorded transaciton in the TLOG matches one of the recorded transactions ("48801.1800B10D6C88705836D9") in the Oracle database (the latter just uses a number representation for the text 'BEA1'). You may also notice that the database only lists 6 pending transaction whilst the WebLogic TLOG listed 7. Why would this be? Well this is likely to be because the database has committed the 7th transaction but didn't have time to notify WebLogic that this had occurred or Weblogic didn't have time to clean the transaction record up, in the TLOG, when the server process was killed. This is perfectly fine, because during Transaction Recovery, Weblogic will be notified by the database that the transaction has already been reconciled and no further action is required.

What is also interesting is that in the TLOG, no hostnames, IP addresses or ports appear to be directly recorded by WebLogic for the database resource or the JMS queue resource. However, in the database pending transaction table, we see that the database has tracked the hostname ('pdthinkpad') of the originator of the transaction, which in this case maps to my WebLogic server's listen address hostname. Later, we'll come back to why these observations may be important.

When running the WebLogic server instance I'd elected to turn on debug-level logging for the server, with the flag 'DebugJTATLOG' enabled. Below is an example of what WebLogic logged when processing one of the transactions and marking it to be committed in the TLOG,

     ####<19-Oct-2012 13:40:31 o'clock BST> <Debug> <JTATLOG> <pdthinkpad> <MyDemoAdmSvr> <[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-1800B10D6C88705836D9> <> <1342701631032> <BEA-000000> <TLOG writing log record, class=weblogic.transaction.internal.ServerTransactionImpl, obj=Name=[EJB boxburner.msgprocessor.MsgProcessorMDB.onMessage(javax.jms.Message)],Xid=BEA1-1800B10D6C88705836D9(347028305),Status=Logging,numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=0,seconds left=30,activeThread=Thread[[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)',5,Pooled Threads],XAServerResourceInfo[WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore]=(ServerResourceInfo[WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore]=(state=prepared,assigned=MyDemoAdmSvr),xar=WLStore_MyDemoDomain_MyDemoAdmSvr_BB_JMSRscJMSStore282730943,re-Registered = false),XAServerResourceInfo[BBDatasource_MyDemoDomain]=(ServerResourceInfo[BBDatasource_MyDemoDomain]=(state=prepared,assigned=MyDemoAdmSvr),xar=BBDatasource,re-Registered = false),...[TRUNCATED FOR BREVITY]...,CoordinatorURL=MyDemoAdmSvr+pdthinkpad:7001+MyDemoDomain+t3+)>

Finally, to test that WebLogic's Transaction Recovery process works correctly in normal circumstances, I simply re-started the WebLogic server on my laptop. Within half a minute, WebLogic's Transaction Recovery Service had kicked in and successfully recovered the stuck transactions that were recorded the TLOG and pushed them through to completion in the affected JMS Queue and Database resources. To prove this, I queried the database table DBA_2PC_PENDING again and this time no rows were returned.

Testing Transaction Recovery After Changing Specific Environment Settings

To determine which key environment setting changes will adversely affect the ability of WebLogic to recover TLOG recorded transactions, I ran a series of tests. For each test, I first ran the system as normal, to process messages. Then I killed the WebLogic server process and checked in the TLOG and in the database, to ensure there some stuck transactions were present. Then I made the necessary changes in the environment configuration settings for the test. Finally I re-started Weblogic to see whether the transactions were successfully recovered or not.

TEST: Changing JDBC Datasource Hostname and Port

In this test, after killing WebLogic, I changed the JDBC URL (hostname and port) value in the domain configuration settings for the WebLogic datasource ('BBDatasource'), to reference a non-existent Oracle DB listener address.  Upon re-starting WebLogic, as the Weblogic transaction recovery service kicked in, I saw lots of entries in the WebLogic log file like:

<19-Oct-2012 15:58:40 o'clock BST> <Warning> <JTA> <BEA-110486> <Transaction BEA1-0511D6A7A132705836D9 cannot complete commit processing because resource [BBDatasource_MyDemoDomain] is unavailable. The transaction will be abandoned after 431,741 seconds unless all resources acknowledge the commit decision.> 

Also, when querying the DBA_2PC_PENDING database table, I could see that there were still pending transactions present. As one would expect, this shows that WebLogic can't magically find a database and relies on the JDBC datasource configuration in the domain, to work out how to resolve reaching the database, rather than any information about a database host or port that may or may not have be stored in the TLOG, at the time when the transaction occurred. Upon correcting the JDBC URL in the domain's configuration and re-starting WebLogic, the pending transactions were all happily pushed through to completion, as expected.

TEST: Changing JDBC Datasource Name

For this test run, after killing the WebLogic server, I changed the name of the 'BBDatasource' data-source in the domain configuration (keeping the JNDI name it contains the same). Again I discovered that on server re-start, the pending transactions could not be recovered. This demonstrated that WebLogic's transaction recovery service uses the domain's current configuration settings for resources like data-sources, rather than anything that may have been recorded, before failure, in the TLOG. Again, correcting the datasource name and re-starting Weblogic resulted in the pending transactions beeing successfully cleared.

TEST: Changing Oracle Database Listener Host and Port

For this test run, after killing the WebLogic server, rather than changing anything in WebLogic's domain configuration, I instead changed the host and port of my local database listener (to listen on my laptop's wireless card address rather than my laptop's wired ethernet card address). On re-starting the Weblogic server, Weblogic was unable to recover the transactions, as expected, because it couldn't contact the database on the address it expected to. This time to help recovery to be possible, rather than reverting back the database changes I had made (Oracle listener host/port), I instead modified the JDBC datasource URL for the WebLogic domain, with the new host/port of the database. Upon re-starting WebLogic, the pending transactions were successfully recovered. What this showed is that, following a failure and before initiating system recovery, a database can be moved and a Weblogic server's configuration can be changed, to reflect the new database location. With these database location and WebLogic datasource changes, Weblogic is still able to successsfully recover transactions, which is potentially important for disaster recovery processes, where a database may be running in a different data-centre.

TEST: Changing WebLogic Default Listen Address

In my test environment, the WebLogic server was originally listening to the hostname "pdthinkpad" which mapped to my laptoep's wired Ethernet card. For this test, after killing the server, I made some changes in WebLogic's domain configuration, changing WebLogic's default channel listen address to a hostname ('temphost') which mapped to my laptop's wireless network card. I also changed my laptop's hostname to 'temphost' and added the appropriate entry for the hostname in '/etc/hosts', removing any trace of the old 'pdthinkpad' name. Before re-starting the WebLogic server to listen to the new address, I double checked the database, querying the DBA_2PC_PENDING table to be check that there were pending transactions. The example output was:

  LOCAL_TRAN_ID GLOBAL_TRAN_ID                 STATE    MIX HOST       COMMIT#
  ------------- ------------------------------ -------- --- ---------- ----------
  5.18.1100     48801.03542CFF3807705836D9     prepared no  pdthinkpad 1472769
  7.11.720      48801.03552CFF3807705836D9     prepared no  pdthinkpad 1472766
  12.5.42       48801.03512CFF3807705836D9     prepared no  pdthinkpad 1472778
  1.22.724      48801.034F2CFF3807705836D9     prepared no  pdthinkpad 1472772

Notice that the Oracle database is tracking that the pending transactions were originated from a server on host 'pdthinkpad'. So the question would be, if recovery of these transactions was now initiated by a seemingly different server (ie. one from a host called 'temphost'), would the database allow these transactions to be reconciled and completed?

With the new hostname and listen address in place, I re-started the WebLogic server and observed that WebLogic's transaction recovery service worked successfully, pushing through to completion the pending transactions. On querying the database DBA_2PC_PENDING table again, no records were listed, showing that the database had indeed committed the transactions. Therefore, the conclusion is that the hostname and listen address for a WebLogic server can be changed, without preventing the successful recovery of transactions recorded in the TLOG. Again, this is potentially important if the WebLogic server has been started in a different data-centre, with a copy of the original TLOG, as part of a disaster recovery process.

Conclusions From Tests

These tests were all based on cases where my WebLogic server, owning the TLOG, was the originator of XA transactions. In other words, the WebLogic server was the 'transaction coordinator'. In these cases, I was able to show that, after restoring or re-creating a Weblogic environment, the pending transactions that had been tracked in a preserved TLOG, could be successfully completed. I was able to make the following environmental changes without inhibiting the ability to recover transactions:

• WebLogic Server default channel listen addresses (Hostname/IP-address and Port)
• Oracle Database listener address (Hostname/IP-address and Port) plus related settings in the WebLogic domain datasource configuration file

What became evident during the investigation, is that it is important to keep the names of the following resources the same (stored in the domain configuration), between the stages of system failure and system recovery:

• Weblogic Domain Name
• WebLogic Server Names
• Weblogic JDBC Datasource Names
• Weblogic Persistent Store Names (File and/or DB)
• Weblogic JMS Server Names
• Weblogic JMS Destination Names

Word of caution

In WebLogic's online documentation for Managing Transactions there is a section titled Moving a Server. The help document highlights that in a specific situation, if the hostname and port of WebLogic's listen address is changed, transaction recovery will fail. This situation occurs if the WebLogic server is a transaction sub-coordinator as part of a larger transaction that has been propagated from another separate Weblogic server.

For example, an EJB client application running on Server A may initiate an XA transaction and invoke an EJB running on Server B which in turn performs an XA database update operation. In these cases, the transaction coordinator (Server A) will have recorded the URL (host and post) of the sub-coordinator (Server B) directly in the TLOG belonging to the coordinator server (Server A). During the transaction recovery process, if the sub-coordinator (Server B) listen address has changed, the coordinator (Server A) will not be able to contact the sub-coordinator (Server B) to inform it to commit or rollback pending transactions. The address that the coordinator server knew for the sub-coordinator server, was hard-wired directly in the TLOG file before the system failure occurred.  The coordinator server is thus unaware that a new listen address for the sub-coordinator server is being used.

Another example of where an XA transaction will be propagated between two WebLogic servers, is the situation where one server may be hosting a queue, and the second server is hosting a Message Driven Bean (MDB) based application, listening for messages appearing on the remote queue, with the MDB's container managed transaction demarcation set to 'required'.

As a result of these potential scenarios, the help documentation referenced above, states "Oracle recommends configuring server instances using DNS names rather than IP addresses to promote portability", to always allow for WebLogic instances to be restored on new machines, without inhibiting transactions recovery. While in this blog entry I have proven that this is not always necessary, depending on the nature of the XA transactions that occur (ones that don't propagate between WebLogic servers), it is a good rule to stick to, when possible, to be safe.

Summary

As we have seen, it is possible to change or move a database that has pending XA transactions, and still have WebLogic resolve and push through to completion those pending transactions, when a working system is re-stored.

We have also seen that it is important to keep key WebLogic resource names the same, when recovering WebLogic servers on new machines or in a whole new environment.

We have proved that in some circumstances, it is possible to change the hostnames and ports of the WebLogic servers, for example, restore the domain to run on a new set of host machines and still allow successful transaction recovery. The set of circumstances where this is possible is when XA transactions have not propagated between more than one WebLogic server.

However, we know that changing a WebLogic server's listen hostname and port breaks the ability to recover XA transactions that have been propagated. In addition, there may well be other places in the host application's metadata, completely unrelated to transactions, that hard-wires knowledge of server listen addresses. For example, an installed Oracle Middleware product running on Weblogic may be tracking server listen addresses in the its Meta-Data-Store (MDS). As another example, a bespoke JEE application hosted on WebLogic may be tracking server listen addresses in its own property files.

It is also worth noting that the official Oracle Fusion Middleware Disaster Recovery guide talks about the importance of using "hostname-based configuration instead of IP-based configuration" in the guide section titled Planning Host Names.

Therefore, I would echo the general advice that Oracle gives, to always use hostnames, rather than IP addresses for network address related resources. This maximises the ability to move servers, with new IP addresses, but retain the same hostnames during the move, and thus prevent breaking a functioning system. These principles are especially pertinent when planning for disaster recovery and the best way to re-establish a WebLogic domain and its TLOGs in another data-centre, so that the system can be restored to a fully working order.

Song for today: (I Don't Need You To) Set Me Free by Grinderman

Writing your own Java application on Exalogic using SDP

I've written before about how Exalogic enables Oracle Middleware products to use Sockets Direct Protocol (SDP) under the covers, rather than TCP-IP, to achieve lower latency communication over an InfiniBand network. Originally, the capability to leverage SDP was limited to Oracle internal-only APIs in the JRockit JVM (Java 1.6) and thus was only usable by Oracle products like WebLogic.

However, SDP support has now been added as a general capability to Java 1.7 (Hotspot JVM), thus enabling any standalone Java application to be written to take advantage of SDP rather than TCP-IP, over InfiniBand. I found a new tutorial, Lesson: Understanding the Sockets Direct Protocol, describing how to write a Java application that can use SDP, so I gave it a go on an Exalogic X2-2 machine. Below I've recorded the steps that I took to test this, in case it's useful to others.

To leverage SDP from your application, you can still use the same Java socket APIs as normal and simply use a configuration file to indicate that SDP should be employed, not TCP-IP. The tutorial I found shows how to provide the SDP configuration file, but doesn't provide Java code examples to test this. So first of all I quickly wrote Java main classes for a server and a client and tested that they worked correctly on my Linux x86-64 laptop when using just TCP-IP over Ethernet.

If you want to try it out yourself, you can download a copy of the test Java project I wrote from here. Below is the key part of my server class that receives a line of text from the client over a socket, prints this text to the console and then replies with an acknowledgement.

  try (ServerSocket serverSocket = new ServerSocket(port)) {
    info("Running server on port " + port);

    while (true) {
      try (Socket socket = serverSocket.accept();
           BufferedReader in = new BufferedReader(
              new InputStreamReader(socket.getInputStream()));
           PrintWriter out = new PrintWriter(
              new BufferedWriter(new OutputStreamWriter(
                socket.getOutputStream())))) {
        String msg = in.readLine();
        info("Server received message:  " + msg);
        out.println("ACKNOWLEDGED (" + msg + ")");
        out.flush();
      }
    }
  }

And below is the key part of my client class which sends a line of text over a socket to the server and prints out the response text it receives.

  info("Running client connecting to " + host + ":" + port);

  try (Socket socket = new Socket(host, port);
       PrintWriter out = new PrintWriter(new BufferedWriter(
          new OutputStreamWriter(socket.getOutputStream())));
       BufferedReader in = new BufferedReader(
          new InputStreamReader(socket.getInputStream()))) {
    info("Client sent message:  " + SEND_MESSAGE_TEXT);
    out.println(SEND_MESSAGE_TEXT);
    out.flush();
    info("Client received message:  " + in.readLine());
  }

The observant among you will notice that all of above is just standard Java 1.7 using java.net.* & java.io.* APIs. Nothing special.

I then moved the test client and server apps over to two Exalogic compute nodes. Actually the compute nodes were virtualised in this case, rather than physical, with each Oracle Linux OS running as a guest OS (vServer) on top of the OVM hypervisor. As instructed in the tutorial, I added the following JVM arguments to my bash scripts for starting the Java server and client applications so that they can use SDP:

  -Dcom.sun.sdp.conf=/u01/myshare/sdp.conf
  -Djava.net.preferIPv4Stack=true
  -Dcom.sun.sdp.debug

I slipped the com.sun.sdp.debug argument in there too, because that makes the JVM print some information to the console, indicating if SDP is being used by the app. I created the sdp.conf file at the location /u01/myshare/sdp.conf, with the following content:

  bind * *
  connect 192.168.0.101  1234

In the first line I tell the JVM that if an application opens a server socket listening to all local network interface IP addresses, it should use SDP. The second line tells the JVM that if an application opens a new socket to the remote host:port of 192.168.0.101:1234, to use SDP. This is the host:port of one of the network interfaces on the vServer that my Java server will listen to, when it starts.

Then running my wrapper bash scripts to start the Java server main class with its SDP file present, on a vServer, and the Java client class, with its SDP file, on another vServer, I saw the following output:

  [paul@paul_vserver2]$ ./runserver.sh
    BIND to 0.0.0.0:1234 (socket converted to SDP protocol)
    INFO - Running server on port 1234
    INFO - Server received message:  Hello from Java socket client program

  [paul@paul_vserver1]$ ./runclient.sh
    INFO - Running client connecting to 192.168.0.101:1234
    CONNECT to 192.168.0.101:1234 (socket converted to SDP protocol)
    INFO - Client sent message:  Hello from Java socket client program
    INFO - Client received message:  ACKNOWLEDGED (Hello from Java socket client program)

As you can see the client successfully sends the server a message and receives a response. In bold I've highlighted the debug output from the JVM showing that SDP is being used. However, to prove that SDP is indeed being used I did some more analysis. Whilst my Java server class was still running with its server socket listening, I ran the following OS level networking command to see if my running application's SDP listener was present. The output displayed:

  [paul@paul_vserver2]$ sdpnetstat -al | grep sdp
    sdp  0   0 *:search-agent   *:*  LISTEN

This shows my server process running listening on all local IP addresses using SDP. As soon as I kill my server process and run the sdpnetstat command again, no SDP listeners are shown.

To further help prove this, I started up the server again listening on SDP, but in the client vServer, I changed the SDP conf file to have a 'rubbish' connect value to force the client application to use TCP-IP. Upon running the client application again, I see the following error, because the client is trying to use TCP to talk to a remote SDP listener. Also, notice in bold, the JVM debug output showing that no SDP match was found in sdp.conf.

  [paul@paul_vserver1]$ ./runclient.sh
    INFO - Running client connecting to 192.168.0.101:1234
    CONNECT to 192.168.0.101:1234 (no match)
    java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.(Socket.java:425)
        at java.net.Socket.(Socket.java:208)
        at testsocketcomms.Client.sendMessage(Client.java:35)
        at testsocketcomms.Client.main(Client.java:21)

Anyway, that's pretty much it. Want to use SDP from a custom application on Exalogic? Just use standard Java socket programming, specify the right settings in a configuration file and off you go!

In the future I hope to revisit this topic and performance test a bespoke application under load, comparing the performance difference between using TCP-IP over InfiniBand and SDP over InfiniBand. However, developing a 'realistic' performance test application, that doesn't contain other more prevalent bottlenecks, is not a simple task, hence it's not something I could have quickly demonstrated here.


Song for today: Floodlit World by Ultrasound

Do I Need to Worry About the Availability and Recovery of WebLogic Transaction Logs?

When planning a WebLogic deployment that places a significant emphasis on High Availability or Disaster Recovery, it may be necessary to preserve WebLogic's Transaction Logs, to enable business-critical I.T. systems to be recovered to a correct and consistent state, following a system crash.

You may ask: What are WebLogic Transaction Logs?

Every WebLogic server has a persistent store (either on a file-system or in a database) to record information about the in-flight global transactions it co-ordinates. This is the Transaction Log, or TLOG for short. In the TLOG, WebLogic records each global transaction that has been flagged to commit but may not yet have committed in all the affected back-end data-stores. A global transaction is a special type of transaction, where the host application has encompassed a set of updates to two or more different data-stores as a seemingly single atomic operation. These data-stores could be relational databases, message queues or enterprise information systems, for example. A global transaction should either succeed or fail as a whole, without leaving any of the incorporated data-stores in an inconsistent state and as such, global transactions have ACID properties. When WebLogic co-ordinates a global transaction, it uses a Two-Phase-Commit (2PC) protocol to interact with the data-store managers (called resource managers). The interface between the transaction manager (e.g. WebLogic) and each resource manager (e.g. a database) is defined by the XA industry standard. To summarise, when processing global transactions, the transaction manager needs to persist its commit decision somewhere, and in WebLogic's case, this is in its TLOG.

So, what if WebLogic didn't persist transaction commit decisions?

If global transaction commit decisions are not persisted and the system fails, then under heavy load it is very likely that at least some transactions will still be in-flight, and temporarily at least, be in-doubt. At this point in time, for each transaction, the updates to one back-end data store may have committed, but the updates to another data-store, in the same transaction, may not yet have been instructed to commit (i.e. the data-store's updates are still pending). The system as a whole will have data in an inconsistent state. Once the failed parts of the system have been re-started, the data-stores holding pending updates will have no way of knowing whether the updates should be committed or rolled-back. The data in the system will then be permanently in an incorrect and inconsistent state. Even with manual human intervention, an administrator will have no way of knowing whether to commit or roll-back pending updates in a data-store, and so the correctness of a complete I.T. system will be forever in-doubt.

So, how does WebLogic recover pending transactions?

WebLogic's TLOGs are a key component of avoiding data in-consistency. Following a system crash, WebLogic's built-in Transaction Recovery Service automatically determines the global transactions that are still pending, by reading the TLOG and polling the relevant back-end data stores. WebLogic is then able to instruct the back-end data stores to either commit or roll-back each pending transaction. Once WebLogic's Transaction Recovery Service completes, the overall system will have been restored to a healthy and consistent state.

So, TLOGs are valuable assets then, that need to be preserved?

If you value and strive to protect and preserve the data in the databases and other data-stores in your enterprise, and your WebLogic hosted applications use global transactions, then you need to equally value and protect your WebLogic TLOGs, as both are inter-related. You need to ensure the persistence store for your TLOG is located on a highly available file-system storage or in a highly available database, and can survive such scenarios as irrevocable damage to a hard-disk platter, for example, or even the loss of a whole data-centre. You also need to plan for the ability to restore the WebLogic server referencing its highly available TLOG, during system recovery, to enable WebLogic to push the in-flight transactions through to completion and return the overall system to a consistent state.

For multi-data-centre deployments, it may be necessary to have a TLOG replicated between two data-centres. In the event of a complete data-centre failure, you can bring the WebLogic servers up in the other data centre, referencing the replicated copy of their TLOG, to allow the pending transactions to be correctly committed or rolled-back.

For enterprises that use WebLogic with global transactions, the preservation and recovery of TLOGs will need to be a critical component of the overall disaster recovery process.

So, investing in technologies and processes to preserve and recover TLOGs is absolutely necessary for all deployments?

Before you go ahead and invest in putting in place highly available storage, multi-site replication technologies and disaster recovery practices for TLOGs, it's worth considering that not all WebLogic deployments use global transactions. You need to be cognisant of this and perform an analysis of your WebLogic deployments, because such an investment cost may not be necessary for your particular system.

If your WebLogic deployed applications are bespoke JEE applications, developed in-house or by a partner, then the application's developers will be able to tell you if global "XA" transaction are employed or not.

If the WebLogic deployed application is built using Oracle Middleware or runs Oracle Applications, then XA global transactions may or may not be being used under the covers, depending on the specific product. You may need to consult the Oracle product documentation or contact Oracle Support. For example, Oracle SOA Suite inherently uses global transactions to track activity transitions belonging to running business processes. So if you value the integrity of these business processes and the data-stores they update, you need to value and protect the TLOGs.

If the WebLogic deployed application is provided by an ISV, you will need to study the ISV's product documentation and/or consult the ISV's Support organisation, to determine if global transactions are employed.

Final Words.....

It is worth stating that such transaction persistence and recovery requirements, and the implied investment required, are not unique to WebLogic. A TLOG is just a mechanism that WebLogic uses. Any enterprise that uses global transactions, regardless of technology vendor, will need to make similar considerations and investments, concerning the provision of highly available storage, multi-site replication technologies and disaster recovery practices.


Song for today: Miles Iz Ded by The Afghan Whigs

New Exa and Engineered Systems blog to watch

Just a quick post to say look out for a new technical blog by my friend and colleague, Donald Forbes. Given Don's expertise, inside track and ready access to real Exa* machines, there should be lots of insightful technical information coming over the next few months, especially on Exalogic and SPARC SuperCluster, so watch this (that?) space.

Song for today: Slipping Husband by The National

New release of DomainHealth - 1.0

I've just released a new version of DomainHealth, that by virtue of being the next increment after 0.9, means that this is the grand 1.0 release! No great fanfare or massive new features but this should [hopefully] be a nice stable release to rely on and live up to its 1.0 billing! :D

You can download DomainHealth 1.0 from here: http://sourceforge.net/projects/domainhealth

One new feature in 1.0 that is worth highlighting though, is the new optional capability to collect and show Processor, Memory and Network statistics from the underlying host Operating System and Machine that WebLogic is running on. DomainHealth only enables this feature if you've also deployed another small open source JEE application that I've created, called WLHostMachineStats. Below is a screenshot of DomainHealth 1.0 in action, displaying graphs of some of these host machine statistics (in this case it's running on an Exalogic system).

(click image for larger view)

WLHostMachineStats is a small agent (a JMX MBean deployed as a WAR file) that runs in every WebLogic Server in a WebLogic domain. It is used to retrieve OS data from the underlying machine hosting each WebLogic Server instance. For more information, including deployment instructions, and to download it, go to: http://sourceforge.net/projects/wlhostmchnstats

Here's another screenshot, just for fun:

(click image for larger view)

Some things to bear in mind....

...the WLHostMachineStats project is still in its infancy and currently places restrictions on what specific environments are supported. Right now, WLHostMachineStats can only be used for WebLogic domains running on Linux Intel (x86) 64-bit based machines (including Exalogic) and only for versions 10.3.0 or greater of WebLogic. This is partly because WLHostMachineStats relies on the SIGAR open source utility, that uses native C libraries and JNI. I hope to widen the list of supported platforms for WLHostMachineStats in the future.

Song for today: Dynamite Steps by The Twilight Singers

New release of DomainHealth (v0.9.1)

I've just released a new version of DomainHealth (version 0.9.1). This is primarily a maintenance/bug-fix release.

DomainHealth is an open source "zero-config" monitoring tool for WebLogic. It collects important server metrics over time, archives these into CSV files and provides a simple web interface for viewing graphs of current and historical statistics. It also works nicely on Exalogic.

To download (and see release notes) go to the project home (select 'files' menu option) at: http://sourceforge.net/projects/domainhealth/

For DomainHealth help docs see: http://sourceforge.net/apps/mediawiki/domainhealth/index.php

Song for today: Ascension Day by Talk Talk

Exalogic DCLI - run commands on all compute nodes at once

Exalogic includes a tool called DCLI (Distributed Command Line Interface) that can be used to run the same commands on all or a subset of compute nodes in parallel. This saves a lot of time and helps avoid the sorts of silly errors that often occur when running a command over and over again. DCLI is a tool that originally came with Exadata (as documented in the Oracle Exadata Storage Server Software User's Guide - E13861-05 chapter 9), and is now incorporated into the new Exalogic product too. It is worth noting that if you are ever involved in performing the initial configuration of a new Exalogic rack, using OneCommand to configure the Exalogic's networking, then under the covers OneCommand will be using DLCI to perform a lot of its work.
Introduction to Exalogic's DCLI
The Oracle Enterprise Linux 5.5 based factory image running on each Exalogic compute node has the exalogic.tools RPM package installed. This contains the DCLI tool in addition to other useful Exalogic command line utilities. Running 'rpm -qi exalogic.tools' on a compute node shows the following package information:

Name : exalogic.tools
Version : 1.0.0.0
Release : 1.0

When you run 'rpm -ql exalogic.tools' you will see that the set of command line utilities are all placed in a directory at '/opt/exalogic.tools'. Specifically, the DCLI tool is located at '/opt/exalogic.tools/tools/dcli'.

Running DCLI from the command line with the '-h' argument, will present you with a short help summary of DCLI and the parameters it can be given:

# /opt/exalogic.tools/tools/dcli -h

If you look at the contents of the '/opt/exalogic.tools/tools/dcli' file you will see that it is actually a Python script that, essentially, determines the list of compute nodes that a supplied command should be applied to and then runs the supplied command on each compute node using SSH under the covers. Conveniently, the Python script also captures the output from each compute node and prints it out in the shell that DCLI was run from. The output from each individual compute node is prefixed by that particular compute node's name so that it is easy for the administrator to see if something untoward occurred on one of the compute nodes only.

A good way of testing DCLI, is to SSH to your nominated 'master' compute node in the Exalogic rack (eg. the 1st one), as root user, and create a file (eg. called 'nodelist') which contains the hostnames of all the compute nodes in the rack (separated by newlines). For example, my nodelist file has the following entries in the first 3 lines:

el01cn01

el01cn02

el01cn03

....

Note: You can comment out one or more hostnames with a hash ('#') if you want DCLI to ignore particular hostnames.

As a reminder on Exalogic compute node naming conventions, 'el01' is the Exalogic rack's default name and 'cn01' contains the number of the specific compute node in that rack.

Once you've created the list of target compute nodes for DCLI to distribute commands to, a nice test is to run a DCLI command that just prints the date-time of each compute node to the shell output of your master compute node (using the /bin/date Linux command). For example:

# /opt/exalogic.tools/tools/dcli -t -g nodeslist /bin/date

Example output:

Target nodes: ['el01cn01', 'el01cn02', 'el01cn03',....]

el01cn01: Mon Feb 21 21:11:42 UTC 2011

el01cn02: Mon Feb 21 21:11:42 UTC 2011

el01cn03: Mon Feb 21 21:11:42 UTC 2011

....

When this runs, you will be prompted for the password for each compute node that DCLI contacts using SSH. The '-t' option tells DCLI to first print out all the names of all nodes it will run the operation on, which is useful for double-checking that you are hitting the compute nodes you intended. The -g command provides the name of the file that contains the list of nodes to operate on (in this case, 'nodelist' in the current directory).

SSH Trust and User Equivalence

To use DCLI without being prompted for a password for each compute node that is contacted, it is preferable to first set-up SSH Trust between the master compute node and all the other compute nodes. DCLI calls this "user equivalence"; a named user on one compute node will then be assumed to have the same identity as the same named user on all other compute nodes. On your nominated 'master' compute node (eg. 'el01cn01'), as root user, first generate an SSH public-private key for the root user. For example:

# ssh-keygen -N '' -f ~/.ssh/id_dsa -t dsa

This places the generated public and private key files in the '.ssh' sub-directory of the root user's home directory (note, '' in the command is two single quotes)

Now run the DCLI command with the '-k' option as shown below which pushes the current user's SSH public key to each other compute node's '.ssh/authorized_keys' file to establish SSH Trust. You will again be prompted to enter the password for each compute node, but this will be the last time you will need to. With the '-k' option, each compute node is contacted sequentially rather than in parallel, to give you chance to enter the password for each node in turn.

# /opt/exalogic.tools/tools/dcli -t -g nodeslist -k -s "\-o StrictHostKeyChecking=no"

In my example above, I also pass the SSH option 'StrictHostKeyChecking=no' so you avoid being prompted with the standard SSH question "Are you sure you want to continue connecting (yes/no)", for each compute node that is contacted. The master compute node will then be added to the list of SSH known hosts on each other compute node, so that this yes/no question will never occur again.

Once the DCLI command completes you have established SSH Trust and User Equivalence. Any subsequent DCLI commands that you issue, from now on, will occur without you being prompted fo passwords.

You can then run the original date-time test again, to satisfy yourself that SSH Trust and User Equivalence is indeed established between the master compute node and each other compute node and that no passwords are prompted for.

# /opt/exalogic.tools/tools/dcli -t -g nodeslist /bin/date

Useful Examples

Now lets have a look at some examples common DCLI commands you might need to issue for your new Exalogic system.

Example 1 - Add a new OS group to each compute node called 'oracle' with group id 500:

# /opt/exalogic.tools/tools/dcli -t -g nodeslist groupadd -g 500 oracle

Example 2 - Add a new OS user to each compute node called 'oracle' with user id 500 as a member of the new 'oracle' group:

# /opt/exalogic.tools/tools/dcli -t -g nodeslist useradd -g oracle -u 500 oracle

Example 3 - Set the password to 'welcome1' for the OS 'root' user and the new 'oracle' user on each compute node (this uses another feature of DCLI where, if multiple commands need to be run in one go, they can be added to a file, which I tend to suffix with '.scl' in my examples - 'scl' is the convention for 'source command line', and the '-x' parameter is provided to tell DCLI to run commands from the named file):

# vi setpasswds.scl
echo welcome1 | passwd root --stdin
echo welcome1 | passwd oracle --stdin # chmod u+x setpasswds.scl
# /opt/exalogic.tools/tools/dcli -t -g nodeslist -x setpasswds.scl

Example 4 - Create a new mount point directory and definition on each compute node for mounting the common/general NFS share which exists on Exalogic's ZFS Shared Storage appliance (the hostname of the HA shared storage on Exalogic's internal InfiniBand network in my example is 'el01sn-priv') and then from each compute node, permanently mount the NFS Share:

# /opt/exalogic.tools/tools/dcli -t -g nodeslist mkdir -p /u01/common/general
# /opt/exalogic.tools/tools/dcli -t -g nodeslist chown -R oracle:oracle /u01/common/general
# vi addmount.scl
cat >> /etc/fstab << EOF
el01sn-priv:/export/common/general /u01/common/general nfs rw,bg,hard,nointr,rsize=131072,wsize=131072,tcp,vers=3 0 0
EOF# chmod u+x addmount.scl

# /opt/exalogic.tools/tools/dcli -t -g nodeslist -x addmount.scl
# /opt/exalogic.tools/tools/dcli -t -g nodeslist mount /u01/common/general

Running DCLI As Non-Root User

In the default Exalogic set-up, DCLI executes as root user when issuing all of its commands regardless of what OS user's shell you use to enter the DCLI command from. Although root access is often necessary for creating things like OS users, groups and mount points, it is not desirable if you just want to use DCLI to execute non-privileged commands under a specific OS user on all computes nodes. For example, as a new 'coherence' OS user, you may want the ability to run a script that starts a Coherence Cache Server instance on every one of the compute nodes in the Exalogic rack, in one go, to automatically join the same Coherence cluster.

To enable DCLI to be used under any OS user and to run all its distributed commands on all compute nodes, as that OS user, we just need to make a few simple one-off changes on our master compute node where DCLI is being run from...

1. As root user, allow all OS users to access the Exalogic tools directory that contains the DCLI tool:

# chmod a+x /opt/exalogic.tools/tools

2. As root user, change the permissions of the DCLI tool to be executable by all users:

# chmod a+x /opt/exalogic.tools/tools/dcli

3. As root user, modify, the DCLI python script (/opt/exalogic.tools/tools/dcli) using 'vi' and replace the line....

USER_ID="root"

...with the line...

USER_ID=pwd.getpwuid(os.getuid())[0]

This script line uses some Python functions to set the DCLI user id to the name of the current OS user running the DCLI command, rather than the hard-coded 'root' username.

4. Whilst still editing the file using vi, add the following Python library import command near the top of the DCLI Python script to enable the 'pwd' Python library to be referenced by the code in step 3.

import pwd

Now log-on to your master compute node as your new non-root OS user (eg. 'coherence' user) and once you've done the one-off setup of your nodelist file and SSH-Trust/User-Equivalence (as described earlier), you will happily be able run DCLI commands accross all compute nodes as your new OS user.

For example, for a test Coherence project I've been playing with recently, I have a Cache Server 'start in-background' script in a Coherence project located on my Exalogic's ZFS Shared Storage. When I run script using the DCLI command below, from my 'coherence' OS user shell on my master compute node, 30 Coherence cache servers instances are started immediately, almost instantly forming a cluster across the compute nodes in the rack.

# /opt/exalogic.tools/tools/dcli -t -g nodeslist /u01/common/general/my-coh-proj/start-cache-server.sh

Just for fun I can run this again to allow 30 more Coherence servers to start-up and join the same Coherence cluster, now containing 60 members.

Summary

As you can see DCLI is pretty powerful yet very simple in both concept and execution!

Song for today: Death Rays by Mogwai

Exalogic Software Optimisations

[Update 19-March-2001 - this blog entry is actually a short summary of a much more detailed Oracle internal document I wrote in December 2010. A public whitepaper using the content from my internal document, has now been published on Oracle's Exalogic home page (see "White Papers" tab on right-hand side of the home page); for the public version, a revised introduction, summary and set of diagrams have been contributed by Oracle's Exalogic Product Managers.]

For version 1.0 of Exalogic there is a number of Exalogic-specific enhancements and optimisations that have been made to the Oracle Application Grid middleware products, specifically:

• the WebLogic application server product;
• the JRockit Java Virtual Machine (JVM) product;
• the Coherence in-memory clustered data-grid product.

In many cases, these product enhancements address performance limitations that are not present on general purpose hardware that uses Ethernet based networking. Typically, these limitations are only manifested when running on Exalogic's high-density computing nodes with InfiniBand's fast-networking infrastructure. Most of these enhancements are designed to enable the benefits of the high-end hardware components, that are unique to Exalogic, to be utilised to the full. This results in a well balanced hardware/software system.

I find it useful to categorise the optimisations in the following way:

1. Increased server scalability, throughput and responsiveness. Improvements to the networking, request handling, memory and thread management mechanisms, within WebLogic and JRockit, enable the products to scale better on the high-multi-core compute nodes that are connected to the fast InfiniBand fabric. WebLogic will use Java NIO based non-blocking server socket handlers (muxers) for more efficient request processing, multi-core aware thread pools and shared byte buffers to reduce data copies between sub-system layers. Coherence also includes changes to ensure more optimal network bandwidth usage when using InfiniBand networking.
2. Superior server session replication performance. WebLogic's In-Memory HTTP Session Replication mechanism is improved to utilise the large InfiniBand bandwidth available between clustered servers. A WebLogic server replicates more of the session data in parallel, over the network to a second server, using parallel socket connections (parallel "RJVMs") instead of just a single connection. WebLogic also avoids a lot of the unnecessary processing that usually takes place on the server receiving session replicas, by using "lazy de-serialisation". With the help of the underlying JRockit JVM, WebLogic skips the host node's TCP/IP stack, and uses InfiniBand's faster “native” networking protocol, called SDP, to enable the session payloads to be sent over the network with lower latency. As a result, for stateful web applications requiring high availability, end-user requests are responded to far quicker.
3. Tighter Oracle RAC integration for faster and more reliable database interaction. For Exalogic, WebLogic includes a new component called “Active Gridlink for RAC” that provides application server connectivity to Oracle RAC clustered databases. This supersedes the existing WebLogic capability for Oracle RAC connectivity, commonly referred to as “Multi-Data-Sources”. Active Gridlink provides intelligent Runtime Connection Load-Balancing (RCLB) across RAC nodes based on the current workload of each RAC node, by subscribing to the database's Fast Application Notification (FAN) events using Oracle Notification Services (ONS). Active Gridlink uses Fast Connection Failover (FCF) to enable rapid RAC node failure detection for greater application resilience (using ONS events as an input). Active GridLink also allows more transparent RAC node location management with support for SCAN and uses RAC node affinity for handling global (XA) transactions more optimally. Consequently, enterprise Java applications involving intensive database work, achieve a higher level of availability with better throughput and more consistent response times.
4. Reduced Exalogic to Exadata response times. When an Exalogic system is connected directly to an Exadata system (using the built-in Infiniband switches and cabling), WebLogic is able to use InfiniBand's faster “native” networking protocol, SDP, for JDBC interaction with the Oracle RAC database on Exadata. This incorporates enhancements to JRockit and the Oracle Thin JDBC driver in addition to WebLogic. With this optimisation, an enterprise Java application that interacts with Exadata, is able to respond to client requests quicker, especially where large JDBC result sets need to be passed back from Exadata to Exalogic.

To summarise, Exalogic provides a high performance, highly redundant hardware platform for any type of middleware application. If the middleware application happens to be running on Oracle's Application Grid software, further significant performance gains will be achieved.

Song for today: Come to Me by 65daysofstatic

Exalogic downloads and documentation links

Now that Exalogic has been released, the main Exalogic documentation is available at: http://download.oracle.com/docs/cd/E18476_01/index.htm

Worth particular attention is the "Machine Owner's Guide" and the "Enterprise Deployment Guide".

The Machine Owner's Guide will give you a good idea of the machine's internal specifications as well as the unit's external dimensions, power consumption needs, cooling needs, multi-rack cabling configurations, etc.

The Enterprise Deployment Guide (EDG) will point you in the right direction if you want to install and configure the Application Grid products on Exalogic in an optimal way for performance and highly-availability.

If you are about to take shipment of Exalogic and need copies of the software, then these can be accessed from the Oracle eDelivery website, using the following steps:

• Browse to the eDelivery site at http://edelivery.oracle.com/
• Press "Continue" link
• Submit the requested user info when prompted, accepting the restrictions
• In the resulting search page, for the Product Pack field, select "Oracle Fusion Middleware", and for Platform field select "Linux x86-64"
• In the results page, press the link for "Oracle Exalogic Elastic Cloud Software 11g Media Pack"

The Exalogic downloads include:

• Compute Node Base Image for Exalogic (parts 1 and 2) - this is the Oracle Enterprise Linux image including the Unbreakable Enterprise Kernel, OFED drivers for InfiniBand connectivity, and various supporting command line utilities
• Configuration Utilities for Exalogic - this is the set of "Middleware Machine Configurator" tools, including the spreadsheet and accompanying shell scripts to help users peform the base network configuration for all the compute nodes in an Exalogic rack (a.k.a. "OneCommand")
• Oracle WebLogic Server 11gR1 (10.3.4) - this is the combined WebLogic/JRockit/Coherence .bin installer for Exalogic (Linux x86-64)

Song for today: Cause = Time by Broken Social Scene

Exalogic 1.0 is here!

General availability of Oracle's brand new Exalogic Elastic Cloud product has just been publically announced.

Just in case you've somehow missed the buzz and haven't got a clue what Exalogic is, I'll describe it for you a little here...

Exalogic is an integrated hardware and software system that is engineered, tested, and tuned to run enterprise Java applications, as well as native applications, with an emphasis on high performance and high availability (HA). Exalogic incorporates redundant hardware with dense-computing resources, ZFS Shared Storage and InfiniBand networking. This hardware is sized and customised for optimum use by Oracle 'Application Grid' software products, to provide a balanced hardware/software system. Specifically, the WebLogic Application Server, JRockit Java Virtual Machine (JVM) and Coherence In-Memory Data Grid products have been enhanced to leverage some of the unique features of the underlying hardware, for maximum performance and HA.

If you are familiar with Exadata and it being the "database machine", then think of Exalogic as the "middleware machine". Physically linking the two together in a data-centre gives you the foundation for a very high-end Enterprise Java based OLTP solution.

Exalogic is a system rather than appliance, where users are able to install, develop or run what they want as long as it is Linux/Solaris x86-64 compatible. Even though some of the elements of Exalogic, like InfiniBand, are more often found in the supercomputing world, Exalogic is intended as a general purpose system for running enterprise business applications. Exalogic will just appear to the hosted applications as a set of general purpose operating systems and processors with common standards-based networking. This means that unlike the supercomputing world, developers don't have to create bespoke software specifically tailored to run on a high-end proprietary platform.

For further information, see the Oracle introduction to Exalogic whitepaper.

Song for today: Whipping Song by Sister Double Happiness

A simplified view of WLDF

The WebLogic Diagnostic Framework (WLDF) is a very powerful capability that has been in the product since WebLogic 9.0. WebLogic's product documentation does a good job of describing the fine detail of WLDF, but I believe that the high level overview it provides is too verbose and makes WLDF seem more complicated than it is. This is probably one reason why I don't see WLDF used quite as often as it should be, in place of custom coded monitoring solutions.

A couple of years ago I created my own diagram showing WLDF's composition, in an attempt to demonstrate that it's actually a fairly simple concept. However, I never got around to publishing it. Prompted by a recent request, I thought I'd address this ommision now, so here is that diagram...

(click image for larger view)

Hopefully this helps to better show the power of WLDF and that it's not as complex as one might think.


Song for today: Maps by Yeah Yeah Yeahs

Review of the new JRockit book

As promised, here's my review of the new JRockit book...


Having used JRockit for years (mostly sitting beneath WebLogic) I've been waiting for something like this book to arrive and lift the lid off the JVM to show what's inside. I'm glad to say that I am not disappointed. This book is a must-have for any serious enterprise Java developer or administrator who uses JRockit and wants to better understand how to tune, manage and monitor the JVM. The book also provides the Java developer with a clearer direction on the do's and don't's for developing a well behaving application on top of JRockit, especially in regard to memory management and thread management. Even for users of other JVMs, like Hotspot, much of the first half of the book, which concentrates on the JVM's internals, is relevant and insightful into the workings of Java virtual machines generally.

The first half of the book concentrates on providing in-depth knowledge of JRockit's internals and focusses on:

• Code Generation (Java bytecode, generation of native code, JRockit 's Just In Time compilation strategy with subsequent optimised code re-generation)
• Memory Management (heap usage, thread-local allocation, garbage collection strategies, deterministic-GC, 32-bit vs 64-bit memory management, potential pitfalls)
• Threads and Synchronisation (green vs O.S. threads, synchronisation patterns/anti-patterns, code generation strategies for locking/unlocking, thin and fat locks, lock profiling)
• Benchmarking and Tuning (throughput vs latency, tuning tips, benchmarking tips)

Don't be put off by the early exposure to Java bytecode samples in the book. The bytecode examples are only used liberally and only in a couple of early chapters. It's worth reading these bytecode samples carefully because the points these chapters make will resonate more strongly if you do.

As I read this book, it became evident that the JRockit engineers are extremely passionate about their work and that they live and breath it. It is re-assuring to know that they are constantly striving to improve the product, based on deep scientific theory yet always with an eye on pragmatism and real-world usage. I'm sure this is why JRockit is fast and powerful whilst remaining easy to manage and run applications on. Throughout the book, the ethos of JRockit's design is very clear. It's an adaptive JVM, where the internal runtime is constantly and automatically being re-evaluated and re-tuned, according to the current nature of the hosted application's usage and load. Reading the book, I can better appreciate the argument that server-side Java is faster than an equivalent native C application. A native application only has one-shot, at compile time, to generate optimal native code. JRockit, however, takes the opportunity to revisit this at runtime, when it has a much better idea of actual usage patterns.

The second half of the book provides an introduction and detailed reference into JRockit's rich and powerful management and monitoring toolset, including:

• JRockit Mission Control (JRMC), composed of the Management Console, the Flight Recorder (plus Runtime Analyzer which it replaces) and the Memory Leak Detector
• JRockit Command line tool (JRCMD), including in-depth examples of all the important commands
• JRockit Management APIs, including the Java API for direct in-line access to the JRockit JVM from hosted Java applications plus the JMX version of the API for remote access to the JRockit JVM

These chapters provide an easy to read introduction to the key features of JRockit's rich tools, lowering the barrier to use for newbies. Also, many of the illustrated examples are use-case driven, acting as a handy reference guide to come back to at a later date. For example, if you suspect that you have a memory leak in your application and want to work out how best to locate the root of the leak, using the tools, the Leak Detector chapter will take you by the hand through this process.

Finally, at the end of the book there is an introductory chapter covering JRockit Virtual Edition (VE). JRockit VE enables the JRockit JVM, accompanied by a small JRockit kernel, to be run in a virtualised environment. This runs directly on-top of a hypervisor (eg. Oracle VM) without requiring the use of a conventional, heavy-weight, general-purpose operations system such as Linux, Solaris or Windows. Such a fully functional operating systems would otherwise burden the system with a layer of unwanted latency. This chapter in particular makes me realise that the JRockit Engineers are proverbial Rocket Scientists (or should that be Rockit Scientists? :D ). I defy anyone not to be impressed by the ambition of JRockit VE and and the high level of technical expertise that must have gone into developing it!

To summarise, I highly recommend this book. Easy to read yet very insightful. Once you've read it, it will remain as a handy reference to come back to, especially when needing to use the JRMC tools to diagnose issues and tune the JVM and hosted applications.


Song for today: Speed Trials by Elliot Smith

New release of DomainHealth WebLogic monitoring tool

I've just released the latest version of DomainHealth - version 0.9.

DomainHealth is an open source "zero-config" monitoring tool for WebLogic. It collects important server metrics over time, archives these into CSV files and provides a simple web interface for viewing graphs of current and historical statistics.

(click image for larger view)

This release includes a new look and feel and a new page navigation mechanism (based on some contributions from Alain Gregoire). Other new features included are Web-App monitoring, EJB monitoring and various minor improvements and tweaks. For a full list of changes see the Release Notes document listed alongside the DomainHealth download files.

You can download DH from the project home page at http://sourceforge.net/projects/domainhealth.

The help docs for DH are at http://sourceforge.net/apps/mediawiki/domainhealth.

I'd like to say a big thank you to Alain Gregoire for his valuable design and code contributions to this version of DomainHealth.


Song for today: Lets Go For A Ride by Cracker

New JRockit book

A new book, "Oracle JRockit: The Definitive Guide", has just been published.

More info is available on the publisher's main landing page for the book.

I've got the book on order and I'll review it here once I've digested it. Judging by the content and the names of the esteemed JRockit engineers that wrote it, I'm expecting this to be an invaluable tome on all things JRockit.


Song for today: Mr.November by The National