A. Sundararajan's Weblog

Querying Java heap with OQL

This is continuation of What's in my Java heap? post. Let us see more OQL (Object Query Language) examples. But before that we will digress into built-in functions supported in OQL.

The built-in functions in OQL fall into the following categories:

• Functions that operate on individual Java objects
  1. sizeof(o)-- returns size of Java object in bytes
  2. objectid(o)-- returns unique id of Java object
  3. classof(o)-- returns Class object for given Java object
  4. identical(o1, o2) -- returns (boolean) whether two given object are identical or not (essentially objectid(o1) == objectid(o2). Do not use simple JavaScript reference comparison for Java Objects!)
  5. referrers(o) -- returns array of objects refering to given Java object
  6. referees(o) -- returns array of objects referred by given Java object
  7. reachables(o) -- returns array of objects directly or indirectly referred from given Java object (transitive closure of referees of given object)
• Functions that operate operate on arrays
  1. contains(array, expr) -- returns array contains an element that satisfies given expression The expression can refer to built-in variable 'it'. This is current object iterated
  2. count(array, [expr]) -- returns number of elements satisfying given expression
  3. filter(array, expr) -- returns a new array containing elements satisfying given expression
  4. map(array, expr) -- returns a new array that contains results of applying given expression on each element of input array
  5. sort(array, [expr]) -- sorts the given array. optionally accepts comparison expression to use. if not given, sort uses numerical comparison
  6. sum(array) -- sums all elements of array
  As you can see, most array operating functions accept boolean expression -- the expression can refer to current object by it variable. This allows operating on arrays without loops -- the built-in functions loop through the array and 'apply' the expression on each element.

There is also built-in object called heap. There are various useful methods in heap object. For more details on built-in functions, object refer to "OQL Help" link in jhat's OQL page.

Now, let us see some interesting queries.

Select all objects referred by a SoftReference:

    select f.referent from java.lang.ref.SoftReference f 
    where f.referent != null

referent is a private field of java.lang.ref.SoftReference class (actually inherited field from java.lang.ref.Reference. You may use javap -p to find these!) We filter the SoftReferences that have been cleared (i.e., referent is null).

Show referents that are not referred by another object. i.e., the referent is reachable only by that soft reference:

    select f.referent from java.lang.ref.SoftReference f 
    where f.referent != null && referrers(f.referent).length == 1

Note that use of referrers built-in function to find the referrers of a given object. because referrers returns an array, the result supports length property.

Let us refine above query. We want to find all objects that referred only by soft references but we don't care how many soft references refer to it. i.e., we allow more than one soft reference to refer to it.

    select f.referent from java.lang.ref.SoftReference f 
    where f.referent != null &&
    filter(referrers(f.referent), "classof(it).name != 'java.lang.ref.SoftReference'").length == 0

Note that filter function filters the referrers array using a boolean expression. In the filter condition we check the class name of referrer is not java.lang.ref.SoftReference. Now, if the filtered arrays contains atleast one element, then we know that f.referent is referred from some object that is not of type java.lang.ref.SoftReference!

Find all finalizable objects (i.e., objects that are some class that has 'java.lang.Object.finalize()' method overriden)

    select f.referent from java.lang.ref.Finalizer f
    where f.referent != null

How does this work? When an instance of a class that overrides finalize() method is created (potentially finalizable object), JVM registers the object by creating an instance of java.lang.ref.Finalizer. The referent field of that Finalizer object refers to the newly created "to be finalized" object. (dependency on implementation detail!)

Find all finalizable objects and approximate size of the heap retained because of those.

    select { obj: f.referent, size: sum(map(reachables(f.referent), "sizeof(it)")) }
    from java.lang.ref.Finalizer f
    where f.referent != null

Ah! That looks really complex -- but, actually it is simple. I use JavaScript object literal to select multiple values in the select expression (obj and size properties). reachables finds objects reachable from given object. map creates a new array from input array by applying given expression on each element. The map call in this query would create an array of sizes of each reachable object. sum built-in adds all elements of array. So, we get total size of reachable objects from given object (f.referent in this case). Why do I say approximate size? HPROF binary heap dump format does not account for actual bytes used in live JVM. Instead sizes just enough to hold the data are used. For eg. JVMs would align smaller data types such as 'char' -- JVMs would use 4 bytes instead of 2 bytes. Also, JVMs tend to use one or two header words with each object. All these are not accounted in HPROF file dump. HPROF uses minimal size needed to hold the data - for example 2 bytes for a char, 1 byte for a boolean and so on.

That's all for now! We will more interesting OQL queries in future. Stay tuned...

---

JSON, AJAX and Mustang

JSON (JavaScript Object Notation) is a lightweight data interchange format. It uses (subset) of JavaScript object/array literal syntax as data interchange format. Now, Mustang Java SE 6 includes Rhino based JavaScript engine and javax.script API. This implies you can transfer and read data from web/app server in JSON format and read the same in client (web browser or JavaWebStart) side easily -- all you have to do is to make reader from your (URL) input stream and eval it!

You have all AJAX elements within JDK:

• JavaScript
• JSON or XML for data interchange
• asynchronous -- you can use java.lang.Thread from JavaScript or from Java code!

So, welcome to AJAX with JDK (with or without browser client?)

---

What's in my Java heap?

As I mentioned earlier, Mustang (Java SE 6) comes with jhat - Java Heap Analysis Tool. We have added more features to jhat as of build 53 (released on Sep, 22, 2005). jhat now comes with mechanism to query the heap. jhat supports OQL - Object Query Language -- a SQL-like language to query your Java heap! HAT always supported simple queries such as "show all instances of class X" (include subclasses or not), "show all referrers to a specific object", "show all objects reachable from a specific object" etc. But, these pre-defined queries (supported by hyperlinks) may always be sufficient to a specific problem in your hand. That is where OQL helps.

OQL query is of the form

select <expression>
[ from [instanceof] <class name> <identifier>
[ where <filter-expression> ] ]

where select, from, where, instanceof are keywords. expression, filter-expression are JavaScript expressions. class name is fully qualified Java class name (example: java.net.URL) or array class name. [C is char array name, [Ljava.io.File; is name of java.io.File[] and so on. If instanceof keyword is used, subtype objects are selected. If this keyword is not specified, only the instances of exact class specified are selected. Both from and where clauses are optional.

In select and (optional) where clauses, the expression used in JavaScript expression. Java heap objects are wrapped as convenient script objects so that fields may be accessed in natural syntax. For example, Java fields can be accessed with obj.field_name syntax and array elements can be accessed with array[index] syntax. Each Java object selected is bound to a JavaScript variable of the identifier name specified in from clause.

identifier is a name bound for each Java heap object that is matching from and where clause. Note that both from and where clauses are optional. Few simple examples, would help in understanding basic OQL.

OQL Examples

    select s from java.lang.String s
    where s.count > 100

will select and show all String instances that have length more than 100. Note that we use the 'count' field of java.lang.String class. If you treat your Java heap as "database", then "classes" are "tables", "objects" are "rows" and "fields" are "columns". How do you "schema" of the "database"? You can use "javap -p" option and get all fields of a specific class.

     select a from [I a where a.length >= 256

will show all int arrays that have 256 or more elements.

    select classof(cl).name 
    from instanceof java.lang.ClassLoader cl

show names of all ClassLoader classes. Note that classof is a OQL built-in function that returns Class object of a given Java object (similar to Object.getClass() method). We will see more OQL examples, built-in functions in future blogs. But, if you are in hurry and want to learn things as soon as possible, then download Mustang binary and run jhat! You may want to refer to Alan's blog on heap dumps to know more about how to get heap dump of your Java application ...

---

Mustang's HAT!

with

If you use Heap Analysis Tool (HAT), this is a great news for you. Mustang (Java SE 6) (build 51 onwards) includes HAT in $JDK/bin directory. This is called jhat - Java Heap Analysis Tool. So, you don't need to download HAT separately! JDK will include it as part of standard distribution.

Besides we are adding new features to jhat -- stay tuned! I'll blog more about it when the bits become available...

---

command line script shell in Mustang

Mustang (Java SE 6) includes (build 51 onwards) jrunscript - a command line scripting shell. This is under $JDK/bin directory. This shell supports script execution both batch and interactive modes. You can choose the scripting language by -l option and list all the available script engines by -q option. The default scripting language is JavaScript.

The usage message of jrunscript looks like:

Usage: jrunscript [options] [arguments...]

where [options] include:
  -classpath <path>    Specify where to find user class files 
  -cp <path>           Specify where to find user class files 
  -D<name>=<value>  Set a system property 
  -J<flag>             Pass <flag> directly to the runtime system 
  -l <language>        Use specified scripting language 
  -e <script>          Evaluate given script 
  -encoding <encoding> Specify character encoding used by script files 
  -f <script file>     Evaluate given script file 
  -f -                 Interactive mode, read script from standard input 
                       If this is used, this should be the last -f option 
  -help                Print this usage message and exit 
  -?                   Print this usage message and exit 
  -q                   List all scripting engines available and exit 
  
If [arguments..] are present and if no -e or -f option is used, then first
argument is script file and the rest of the arguments, if any, are passed
as script arguments. If [arguments..] and -e or -f option is used, then all
[arguments..] are passed as script arguments. If [arguments..], -e, -f are
missing, then interactive mode is used.

You can easily test your script one liners by -e option. For example,

    jrunscript -e "print('hello world')"

jrunscript supports interactive mode. Because of Java-JavaScript integration in Rhino JavaScript engine, interactive mode is a great way to learn Java APIs in Mustang. Nothing like writing few lines of code to learn new API. With jrunscript you can do that just by writing few lines of script!!

---