Exalogic Software Optimisations

[Update 19-March-2001 - this blog entry is actually a short summary of a much more detailed Oracle internal document I wrote in December 2010. A public whitepaper using the content from my internal document, has now been published on Oracle's Exalogic home page (see "White Papers" tab on right-hand side of the home page); for the public version, a revised introduction, summary and set of diagrams have been contributed by Oracle's Exalogic Product Managers.]

For version 1.0 of Exalogic there is a number of Exalogic-specific enhancements and optimisations that have been made to the Oracle Application Grid middleware products, specifically:

• the WebLogic application server product;
• the JRockit Java Virtual Machine (JVM) product;
• the Coherence in-memory clustered data-grid product.

In many cases, these product enhancements address performance limitations that are not present on general purpose hardware that uses Ethernet based networking. Typically, these limitations are only manifested when running on Exalogic's high-density computing nodes with InfiniBand's fast-networking infrastructure. Most of these enhancements are designed to enable the benefits of the high-end hardware components, that are unique to Exalogic, to be utilised to the full. This results in a well balanced hardware/software system.

I find it useful to categorise the optimisations in the following way:

1. Increased server scalability, throughput and responsiveness. Improvements to the networking, request handling, memory and thread management mechanisms, within WebLogic and JRockit, enable the products to scale better on the high-multi-core compute nodes that are connected to the fast InfiniBand fabric. WebLogic will use Java NIO based non-blocking server socket handlers (muxers) for more efficient request processing, multi-core aware thread pools and shared byte buffers to reduce data copies between sub-system layers. Coherence also includes changes to ensure more optimal network bandwidth usage when using InfiniBand networking.
2. Superior server session replication performance. WebLogic's In-Memory HTTP Session Replication mechanism is improved to utilise the large InfiniBand bandwidth available between clustered servers. A WebLogic server replicates more of the session data in parallel, over the network to a second server, using parallel socket connections (parallel "RJVMs") instead of just a single connection. WebLogic also avoids a lot of the unnecessary processing that usually takes place on the server receiving session replicas, by using "lazy de-serialisation". With the help of the underlying JRockit JVM, WebLogic skips the host node's TCP/IP stack, and uses InfiniBand's faster “native” networking protocol, called SDP, to enable the session payloads to be sent over the network with lower latency. As a result, for stateful web applications requiring high availability, end-user requests are responded to far quicker.
3. Tighter Oracle RAC integration for faster and more reliable database interaction. For Exalogic, WebLogic includes a new component called “Active Gridlink for RAC” that provides application server connectivity to Oracle RAC clustered databases. This supersedes the existing WebLogic capability for Oracle RAC connectivity, commonly referred to as “Multi-Data-Sources”. Active Gridlink provides intelligent Runtime Connection Load-Balancing (RCLB) across RAC nodes based on the current workload of each RAC node, by subscribing to the database's Fast Application Notification (FAN) events using Oracle Notification Services (ONS). Active Gridlink uses Fast Connection Failover (FCF) to enable rapid RAC node failure detection for greater application resilience (using ONS events as an input). Active GridLink also allows more transparent RAC node location management with support for SCAN and uses RAC node affinity for handling global (XA) transactions more optimally. Consequently, enterprise Java applications involving intensive database work, achieve a higher level of availability with better throughput and more consistent response times.
4. Reduced Exalogic to Exadata response times. When an Exalogic system is connected directly to an Exadata system (using the built-in Infiniband switches and cabling), WebLogic is able to use InfiniBand's faster “native” networking protocol, SDP, for JDBC interaction with the Oracle RAC database on Exadata. This incorporates enhancements to JRockit and the Oracle Thin JDBC driver in addition to WebLogic. With this optimisation, an enterprise Java application that interacts with Exadata, is able to respond to client requests quicker, especially where large JDBC result sets need to be passed back from Exadata to Exalogic.

To summarise, Exalogic provides a high performance, highly redundant hardware platform for any type of middleware application. If the middleware application happens to be running on Oracle's Application Grid software, further significant performance gains will be achieved.

Song for today: Come to Me by 65daysofstatic

Exalogic 1.0 is here!

General availability of Oracle's brand new Exalogic Elastic Cloud product has just been publically announced.

Just in case you've somehow missed the buzz and haven't got a clue what Exalogic is, I'll describe it for you a little here...

Exalogic is an integrated hardware and software system that is engineered, tested, and tuned to run enterprise Java applications, as well as native applications, with an emphasis on high performance and high availability (HA). Exalogic incorporates redundant hardware with dense-computing resources, ZFS Shared Storage and InfiniBand networking. This hardware is sized and customised for optimum use by Oracle 'Application Grid' software products, to provide a balanced hardware/software system. Specifically, the WebLogic Application Server, JRockit Java Virtual Machine (JVM) and Coherence In-Memory Data Grid products have been enhanced to leverage some of the unique features of the underlying hardware, for maximum performance and HA.

If you are familiar with Exadata and it being the "database machine", then think of Exalogic as the "middleware machine". Physically linking the two together in a data-centre gives you the foundation for a very high-end Enterprise Java based OLTP solution.

Exalogic is a system rather than appliance, where users are able to install, develop or run what they want as long as it is Linux/Solaris x86-64 compatible. Even though some of the elements of Exalogic, like InfiniBand, are more often found in the supercomputing world, Exalogic is intended as a general purpose system for running enterprise business applications. Exalogic will just appear to the hosted applications as a set of general purpose operating systems and processors with common standards-based networking. This means that unlike the supercomputing world, developers don't have to create bespoke software specifically tailored to run on a high-end proprietary platform.

For further information, see the Oracle introduction to Exalogic whitepaper.

Song for today: Whipping Song by Sister Double Happiness

Review of the new JRockit book

As promised, here's my review of the new JRockit book...


Having used JRockit for years (mostly sitting beneath WebLogic) I've been waiting for something like this book to arrive and lift the lid off the JVM to show what's inside. I'm glad to say that I am not disappointed. This book is a must-have for any serious enterprise Java developer or administrator who uses JRockit and wants to better understand how to tune, manage and monitor the JVM. The book also provides the Java developer with a clearer direction on the do's and don't's for developing a well behaving application on top of JRockit, especially in regard to memory management and thread management. Even for users of other JVMs, like Hotspot, much of the first half of the book, which concentrates on the JVM's internals, is relevant and insightful into the workings of Java virtual machines generally.

The first half of the book concentrates on providing in-depth knowledge of JRockit's internals and focusses on:

• Code Generation (Java bytecode, generation of native code, JRockit 's Just In Time compilation strategy with subsequent optimised code re-generation)
• Memory Management (heap usage, thread-local allocation, garbage collection strategies, deterministic-GC, 32-bit vs 64-bit memory management, potential pitfalls)
• Threads and Synchronisation (green vs O.S. threads, synchronisation patterns/anti-patterns, code generation strategies for locking/unlocking, thin and fat locks, lock profiling)
• Benchmarking and Tuning (throughput vs latency, tuning tips, benchmarking tips)

Don't be put off by the early exposure to Java bytecode samples in the book. The bytecode examples are only used liberally and only in a couple of early chapters. It's worth reading these bytecode samples carefully because the points these chapters make will resonate more strongly if you do.

As I read this book, it became evident that the JRockit engineers are extremely passionate about their work and that they live and breath it. It is re-assuring to know that they are constantly striving to improve the product, based on deep scientific theory yet always with an eye on pragmatism and real-world usage. I'm sure this is why JRockit is fast and powerful whilst remaining easy to manage and run applications on. Throughout the book, the ethos of JRockit's design is very clear. It's an adaptive JVM, where the internal runtime is constantly and automatically being re-evaluated and re-tuned, according to the current nature of the hosted application's usage and load. Reading the book, I can better appreciate the argument that server-side Java is faster than an equivalent native C application. A native application only has one-shot, at compile time, to generate optimal native code. JRockit, however, takes the opportunity to revisit this at runtime, when it has a much better idea of actual usage patterns.

The second half of the book provides an introduction and detailed reference into JRockit's rich and powerful management and monitoring toolset, including:

• JRockit Mission Control (JRMC), composed of the Management Console, the Flight Recorder (plus Runtime Analyzer which it replaces) and the Memory Leak Detector
• JRockit Command line tool (JRCMD), including in-depth examples of all the important commands
• JRockit Management APIs, including the Java API for direct in-line access to the JRockit JVM from hosted Java applications plus the JMX version of the API for remote access to the JRockit JVM

These chapters provide an easy to read introduction to the key features of JRockit's rich tools, lowering the barrier to use for newbies. Also, many of the illustrated examples are use-case driven, acting as a handy reference guide to come back to at a later date. For example, if you suspect that you have a memory leak in your application and want to work out how best to locate the root of the leak, using the tools, the Leak Detector chapter will take you by the hand through this process.

Finally, at the end of the book there is an introductory chapter covering JRockit Virtual Edition (VE). JRockit VE enables the JRockit JVM, accompanied by a small JRockit kernel, to be run in a virtualised environment. This runs directly on-top of a hypervisor (eg. Oracle VM) without requiring the use of a conventional, heavy-weight, general-purpose operations system such as Linux, Solaris or Windows. Such a fully functional operating systems would otherwise burden the system with a layer of unwanted latency. This chapter in particular makes me realise that the JRockit Engineers are proverbial Rocket Scientists (or should that be Rockit Scientists? :D ). I defy anyone not to be impressed by the ambition of JRockit VE and and the high level of technical expertise that must have gone into developing it!

To summarise, I highly recommend this book. Easy to read yet very insightful. Once you've read it, it will remain as a handy reference to come back to, especially when needing to use the JRMC tools to diagnose issues and tune the JVM and hosted applications.


Song for today: Speed Trials by Elliot Smith

New JRockit book

A new book, "Oracle JRockit: The Definitive Guide", has just been published.

More info is available on the publisher's main landing page for the book.

I've got the book on order and I'll review it here once I've digested it. Judging by the content and the names of the esteemed JRockit engineers that wrote it, I'm expecting this to be an invaluable tome on all things JRockit.


Song for today: Mr.November by The National

Attempting to Quickly Diagnose Production Memory Leaks with JRockit

I just wanted to mention my favourite, not well known, JRockit diagnostic command which I've used a few times recently to resolve tricky JVM heap related issues in production environments.

Imagine the situation: You've performance tested and tuned your server and application to the hilt, but on day 3 after going live, the production servers are exhibiting excessive heap memory usage issues. You've been pushed into the noisy server room, the pressure's on and the project sponsor is pacing up and down behind you. Until a sys-admin sorts out a non-headless machine for you to run JRockit's Mission Control on, you're stuck with the command line on the production server you've just SSH'd into. You're desperate to be able to just issue a command or two into the shell to quickly see what's happening in the JVM heap before the sys-admin bounces the server to prevent it from going pop with out of memory errors. Well there is such a command that might just save you....

Using JRockit's JRCMD command line tool you can issue a command called heap_diagnostics.

This command was new in JRockit R26.4 which was bundled with some of the later WLS 9.x versions and with all WLS versions from 10.0 onwards. It produces a diagnostic text dump of the state of the live JRockit heap, into sys-out of your command shell. The dump includes a summary of the current heap size and heap free, the layout of the native memory used by the JVM, the layout of the JVM's heap, and the state of soft/weak/phantom references and finalizers at the last garbage collection. Most crucially though, the dump also includes the number, percentage and size of every object type currently held in the heap, in order of the largest consumer first. If you suspect a memory leak and look at the names of the classes listed at the top of this list, you have a good chance of very quickly discovering which objects are being leaked. This may not directly point you at the root cause, but it will give you a much better idea of where the problem may lie.

As an example, to produce a diagnostic dump on a Unix based host environment, run the following commands to: (i) make the JRockit JRE and tools visible to your shell, (ii) determine the operating system's process ID of the WebLogic instance you want to probe, and (iii) generate the diagnostic dump into a text file:

$ . /opt/oracle/WLS103/wlserver_10.3/server/bin/setWLSEnv.sh

$ jps -l | grep weblogic
9681 weblogic.Server

$ jrcmd 9681 heap_diagnostics > heapdump.txt

The command takes a second or two to complete (depending on the current size of your heap and how much is in there). In the environment where I was chasing a memory leak, the dump text file was about 0.5 MB in size, which I've truncated and shown below.

In this specific example, by looking at the first few entries in the "Detailed Heap Statistics" section of the dump, showing largest heap consumers first, I was able to take an educated guess that the cause of my memory leak was due to some SNMP related classes in the package 'monfox' in WebLogic 10.0.1 rather than in my application code. In my case, having so many of these types of objects in the heap was not to be expected. However in other cases, top consuming objects may be legitimate and not be leaked objects, so this is only a potential indicator. Incidentally, if you need a patch for this specific SNMP issue, raise an Oracle Support case using Metalink refering to Bug#8185278.

======== BEGIN OF HEAPDIAGNOSTIC =========================

Total memory in system: 3434975232 bytes
Available physical memory in system: 2001797120 bytes
-Xmx (maximal heap size) is 536870912 bytes
Heapsize: 536870912 bytes
Free heap-memory: 470024072 bytes

mmStartCompaction = 0x8c00000, mmEndCompaction = 0xac00000

Memory layout:
00000000-00002000 ---p 00000000 00:00 0
08048000-08057000 r-xp 00000000 08:06 11421468   /opt/oracle/wls1001/jrockit_150_11/bin/java
08057000-08058000 rwxp 0000f000 08:06 11421468   /opt/oracle/wls1001/jrockit_150_11/bin/java
08060000-080e0000 rwxp 08060000 00:00 0
08100000-08901000 rwxp 08100000 00:00 0
.........truncated.........

--------- Detailed Heap Statistics: ---------
26.1% 12083k   108775   -541k [C
12.8% 5924k     5586    +18k  [B
 7.2% 3338k   142444    -52k  java/lang/String
 6.4% 2974k    54389     -4k  monfox/toolkit/snmp/util/OidTree$Node
 3.8% 1753k    16028     +0k  [Lmonfox/toolkit/snmp/util/OidTree$Node;
 2.6% 1218k    38978    -39k  monfox/toolkit/snmp/agent/ext/table/SnmpMibTableAdaptor$AdaptorMibNode
 1.2% 555k     7069     +0k   java/lang/Class
 0.5% 225k     3213     -2k   java/util/TreeMap$Entry
.........truncated.........
     46258kB total ---
--------- End of Detailed Heap Statistics ---

------------------- Printing heap ---------------------
"o"/"p" = 1k normal/pinned objects
"O"/"P" = 50k normal/pinned objects
"."/"/" = <1k br="" fragmentation="" normal="" objects="" pinned="">" "/"_" = 1k/50k free space
-------------------------------------------------------
OOOOOOOOOOOOOOOOooooooooooooooooooo.oooooooooooo..      0x8ccb5e8
..o.oooooooooooooooooo.O.............OOooooooooooo      0x8cfb770
ooooooooooo.Oooooooooooooooooooooooooooooooooooooo      0x8d13a28
.........truncated.........
__________________________________________________      0x28a36420
____________________________________                    0x28bfd178
         
-------------- Done printing heap ---------------------

--- Verbose reference objects statistics - old collection --------
456.7 MB free memory (of 512.0 MB) after last heap GC, finished 0.137 s ago.
Soft references: 326 (146 only soft reachable, 0 cleared this GC)
    java/lang/ref/SoftReference: 215 (140, 0)
           71 ( 32, 0) java/lang/reflect/Method
           45 ( 45, 0) [Ljava/lang/reflect/Method;
.........truncated.........
    Softly reachable referents not used for at least 228.349 s cleared.
Weak references: 15268 (0 cleared this GC)
    java/lang/ref/WeakReference: 12834 (0)
        11355 (  0)    java/lang/Class
          888 (  0)    com/sun/jmx/interceptor/DefaultMBeanServerInterceptor$ListenerWrapper
.........truncated.........
Final handles: 743 (0 pending finalization, 0 became pending this GC)
          244 (  0, 0) java/util/zip/Inflater
          228 (  0, 0) java/util/jar/JarFile
           48 (  0, 0) com/rsa/jsafe/JA_RSAPublicKey
.........truncated.........
Weak object handles: 42096 (0 cleared this GC)
        26308 (  0)    java/lang/String
        12262 (  0)    sun/misc/Launcher$AppClassLoader
.........truncated.........
Phantom references: 366 (330 only phantom reachable, 0 became phantom reachable this GC)
    com/bea/xbean/store/Locale$Ref: 330 (330, 0)
          330 (330, 0) com/bea/xbean/store/Cursor
    jrockit/vm/ObjectMonitor: 22 (0, 0)
            4 (  0, 0) weblogic/work/ExecuteThread
            3 (  0, 0) weblogic/kernel/ServerExecuteThread
.........truncated.........
--- End of reference objects statistics - old collection ---------

Dark matter: 7207024 bytes
Heap size is not locked

======== END OF HEAPDIAGNOSTIC ===========================

Song for today: Turn Around by Whiskeytown

Is a JVM's Maximum Heap Size really 1.7 GB?

Many claim that the maximum heap size for a JVM is 1.7 GB, but is this really true?

Well yes and no....

Probably yes, where in most cases the person asking the question is a developer using a Windows based PC. Invariably, the PC uses a Windows 32-bit OS on 32 bit hardware and the JVM in use is Sun's 32-but Hotspot JVM. However, start to change any of these host environment characteristics and the maximum heap size limit can vary significantly.

In many production scenarios, the characteristics of the host environment are very different from development, such as having a 64-bit OS on 64-bit hardware (eg. Solaris/SPARC). However, its often only the developers and not the system administrators that understand the subtleties of JVM memory management to any significant degree, so the whole discussion gets lost and everyone ends up assuming 1.7GB is a hard heap size limit across the board..

So here's my attempt to map out the actual maximum heap sizes a little more clearly. Hopefully the figures in the table below are pretty accurate. I've tested many of these variables, but I've not had the luxury of having an environment to test them all. So please correct me if you have evidence of discrepancies.


The table assumes that the host machine has sufficient physical RAM installed and that other OS processes aren't consuming a significant portion of the available memory. It is assumed that the 64-bit OS is Windows, Linux or Solaris, the 32-bit hardware is x86 and the 64-bit hardware is x86-64, IA-64 or SPARC. Other operating system or hardware architectures may have different subtleties.

Depending on the JVM you use, your mileage may vary. For example, JRockit is pretty good at being able to hit the maximum heap size limits listed, whereas inherent limitations in other JVMs may prevent this.

It is worth discussing the Windows /3G boot option (with PAE) a little further. Most JVMs, like Sun Hotspot 1.6, require a contiguous memory space. Using the /3G option on Windows does not guarantee a single contiguous memory space dedicated to a single process. Therefore, this option cannot be used by JVMs that cannot cope with non-contiguous memory and these JVMs thus have a maximum heap size of 1.7 GB. JRockit can handle non-contiguous memory and therefore can use the /3G option on Windows, enabling a maximum heap size of approximately 2.7 GB.

Why the difference between maximum OS process size and maximum heap size?

Well the memory used by a JVM process is not just composed of heap. It is also made up of JVM internal structures, generated code, loaded native libraries and thread stacks which are all a lot less predictable in size. As a result, the stated maximum JVM process memory size can only ever be approximate allowing some headroom for non-heap memory usage. The more native C libraries your Java application uses and the more threads your application uses, then the more the memory, outside of the heap, will be consumed. Also, some JVMs, like Sun Hotspot, store class definitions outside of the heap (eg. in a “Perm Space”), further adding to the size of a JVM process over and above the maximum heap size.

So even though the figures in the table are achievable in certain circumstances, the actual choice of JVM and the nature of the hosted Java application and its usage of non-heap based memory may mean, in practice, that you are not able to get close to achieving the indicated maximum heap size for your application.


Supporting References:

[1] http://www.microsoft.com/whdc/system/platform/server/PAE/PAEmem.mspx
http://blogs.sun.com/moazam/entry/why_can_t_i_allocate
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4358809

[2] http://blogs.oracle.com/jrockit/2008/09/how_to_get_almost_3_gb_heap_on_windows.html (updated link following the move of the linked content from the old Dev2Dev site - 30-Jan-09)

[3] http://java.sun.com/docs/hotspot/HotSpotFAQ.html#gc_heap_32bit

Soundtrack for today: Radio Protector by 65daysofstatic