Prashanth's Weblog

1. Introduction
1. Probes
2. Probe Provider
3. Probe Listener & StatsProviders
   
2. DTrace*
3. Scripting Client*
4. eXtending Probes using XML (XP-XML)
5. Other Clients
1. JMX/AMX
2. REST
3. Admin Console
4. asadmin list-probes*
5. Help for probe*
   
6. asadmin get/list
7. asadmin monitor
6. Architecture
7. References
   

*- The features denoted with '*' are the value-add features available only for the paid customers.

1. Introduction

Monitoring in GlassFish v3 has taken a huge leap when compared to the earlier versions. We provide a way to dynamically and non-intrusively generate monitoring events from any of the glassfish runtime classes, the ability to listen to these events, collect the statistics and expose these statistics through various standard clients.

Salient features of Monitoring Infrastructure are

• Container(s) / sub-system only emit events/probes
• Example: onWebRequest(...), onWebResponse(...)
• Interesting values are passed as paramters
• Containers do NOT maintain any stats
• We use the term "providers"
• If there are no listeners registered for this, then there is no overhead
  
• Listeners receive the event that are emitted by providers
• Listeners can examine the parameters passed
• Listeners maintain the stats
• The Monitoring framework connects “providers” and “listeners”
• Supports various clients, like asadmin, JMX/AMX, REST, Admin GUI, Client-Scripting* (JavaScript)
• Monitoring events are exposed as probes in the DTrace* world
• Allows the Turning on/off, (configuring) the monitoring for components

*- Available as a value-add feature, made available only to the paid customers.

Before we dwelve into the details, let us understand the terminology used in this document.

i. Probes

Probe points are the monitoring events, emitted when glassfish goes through that code path. The Probes are defined as part of ProbeProvider definition. The Providers and its probes are discovered and registered automatically by the Monitoring infrastructure and thus exposed to the DTrace, Client-Scripting and Probe Listener clients.

ii. Probe Providers

You will use the provider API to expose the probes for your subsystem. A provider is either a Class or  an XML file, defined by the contianer/module. It defines the set of events that are emitted by the provider. Providers are discovered and registered automatically, as and when the modules are loaded into the glassfish.

The following are the two ways on how the ProbeProviders are defined.

Probe Provider as a Class

To define the Probes you could choose to write a simple ProbeProvider class (doesn't have to confirm to any interfaces) and annotate the class and methods with appropriate constructs to qualify the methods in the class as Probes. Once you define the probes, you would instantiate this class in the container code (or wherever you would like to fire this from) and just give a call to the probe method. See the below example.

Defining Provider as a Class @ProbeProvider(moduleProviderName="glassfish", moduleName="web", probeProviderName="jsp") Class JspProbeProvider { ...     @Probe(name="jspLoadedEvent")     public void jspLoadedEvent(         @ProbeParam("jsp")         String jsp,         @ProbeParam("hostName")         String hostName) { } .... } Emitting the probe Class JspProbeEmitterImpl.java { ... public JspProbeEmitterImpl() {            JspProbeProvider jspProvider = new JspProbeProvider();                 ......           }           public jspLoaded (String jsp, String vsId) {                jspProvider.jspLoadedEvent(jsp, vsId);           } ... } MANIFEST Entry probe-provider-class-names:  org.glassfish.web.admin.monitor.RequestProbeProvider,org.glassfish.web.admin.monitor.JspProbeProvider

Probe Provider as an XML

This will allow the user to define probes in a non-intrusive way. Just feed the XML file to the Monitoring infrastructure and make the probes active. This will allow the users to conviniently define probes based on existing code which we cannot change. The XML will be bundled as part of the module and you need to put an entry in the manifest file for the Monitoring infrastructure to discover it. See below for an example of ProbeProvider definition, Usage and Manifest entry.

Defining the Provider XML <probe-provider moduleProviderName="glassfish" moduleName="web" probeProviderName="jsp" >     <probe name="jspLoadedEvent"                                                                 <class>com.sun.enterprise.web.jsp.JspProbeEmitterImpl </class>        <method>jspLoaded</method>        <signature>void (String,String)</signature>        <probe-param type="String" name="jsp"/>        <probe-param type="String" name="hostName"/>        <return-param type="void" />        <location>RETURN/ENTRY/EXCEPTION</location>     </probe> </probe-provider> The Container class's method being called         Class JspProbeEmitterImpl.java{         ...       public jspLoaded (String jsp, String vsId) {             ....       }        } MANIFEST Entry probe-provider-class-names:  org.glassfish.web.admin.monitor.RequestProbeProvider,org.glassfish.web.admin.monitor.JspProbeProvider

iii. Probe Listener

A Probe listener (part of the functionality of StatsProvider Object) is basically an object that receives callbacks from the providers. A method in the listener can be "marked" as callback method for a particular event from a specific provider.

Stats Providers

StatsProvider are the objects which gather and expose the monitoring data of various components. They do the following functionality

• Allows configuration of a component for Monitoring i.e, Enable/Disable
• Gathers statistics
• Defines the Statistics Objects
  
• May Listen (Via. Probe Listeners) to Probe events emitted by Provider
• May collect statistics based on the probe events or however way it wants (for ex. using some specific container events)
• Also may collect statistics from non-Probe Provider, Ex. JVM statistics
  
• The statistics objects will be publised to the Monitoring Registry
• They are the target objects for AMX/JMX and REST clients

Writing a Probe Listener and StatsProvider @ManagedObject class JspStatsProvider() {     private CountStatistic activeJspsLoadedCount;     private CountStatistic totalJspsLoadedCount; ....     @ManagedAttribute("activejspsloadedcount")     public CountStatistic  getActiveJspsLoadedCount(){          return activeJspsLoadedCount;     }     @ManagedAttribute("totaljspsloadedcount")     public CountStatistic  getTotalJspsLoadedCount(){          return totalJspsLoadedCount;     }    @ProbeListener("glassfish:web:jsp:jspLoadedEvent")     public void jspLoadedEvent(         @ProbeParam("jsp")         String jsp,         @ProbeParam("hostName")         String hostName) {         activeJspsLoadedCount.increment();         totalJspsLoadedCount.increment();     } .... } Registering a StatsProvider Object StatsProviderManager.register("jsp-container", PluginPoint.SERVER, "applications/app1/jsp1", new JspStatsProvider());

In the above example the StatsProvider object will be registered under the “server” node in the Monitoring registry. Third parameter gives the intermediary sub-tree (followed after the 'server' node) in the Monitoring registry under which the Statistics objects (defined in the StatsProvider object) would be created as leaf nodes. The fourth parameter is the StatsProvider object itself.

>asadmin get server.applications.app1.jsp1.*   //will return these stats objects

2. DTrace

When the Probe Provider is registered with the monitoring infrastructure (Note that StatsProvider objects have nothing to do with this), the probes from glassfish components are automatically exposed to the amazing world of DTrace (for Solaris) so the DTrace clients can write their (D)scripts to listen to these probes. Note that as a provider, you wont have to do anything for DTrace to work.

3. Scripting Client*

This is a Value-add feature and available only to the paying customers. We expose all our probes to a scripting client that would allow GF user to enable/disable the probes and get the data they want. This is similar to DTrace philosophy but will work on any OS for GlassFish runtime. Monitoring infrastructure exposes a Scripting Container, also the asadmin and GUI clients for deploying and running the scripts written in JavaScript and JRuby (next release) and opening up a Comet channel to display the data as and when the probe is fired (a Push model, does it asynchronously). Note that the script is something similar to a StatsProvider object in this case, the difference being, the user could write adhoc script (no compilation is necessary) and just run it on the fly to see the data instantly.

Javascript example var njspLoaded=0; function jspLoaded(hostName) {     njspLoaded = njspLoaded + 1;     client.print( '\n js> jsp loaded event called on ' +             'host = ' + hostName +             //', app = ' + appName +             ' and count = ' + njspLoaded); } params = java.lang.reflect.Array.newInstance(java.lang.String, 1); params[0]="hostName"; scriptContainer.registerListener('glassfish:web:jsp:jspLoadedEvent', params, 'jspLoaded'); $> asadmin run-script <script-name> js> jsp loaded event called on host = localhost , app = helloApp and count = 1 js> jsp loaded event called on host = localhost , app = helloApp and count = 2 js> jsp loaded event called on host = localhost , app = helloApp and count = 3 js> jsp loaded event called on host = localhost , app = helloApp and count = 2 ^C

Apart from the adhoc scripting support, we will also provide pre-canned script capability, which user would be able to invoke directly. The output from the scripting container is name=value pairs with the value coming from the Object.toString(). Following is the example

asadmin run-script --probe "glassfish:web:request:requestStart" --paramnames "moduleName,hostName"

The module owner doesn't have to do anything to get this functionality. Its available by default, when the container (or module) comes up with its probe provider.

4. eXtending Probes using XML (XP-XML)

This would be a great feature considering the amount of work required by the user to expose any Class method in the glassfish runtime as a Probe. We give this ability using the XP-XML mechanism. Just write an XML with the probe provider information, drop it into your $INSTALL/lib/monitor directory (note that the extension should be .xml) and we discover it automatically and consume it to expose the probes you desire. Once the probes are exposed, you will have huge set of tools to use those probes ranging from writing a StatsProvider thus exposing it through JMX/REST/asadmin clients or writing D-Scripts to analyze the glassfish system or by writing an adhoc javascript to listen to these probes.

XP-XML - sample xml <?xml version="1.0" encoding="UTF-8"?> <probe-providers module="org.glassfish.web.glue"> <!--  glassfish:web:jsp-new:* --> <probe-provider moduleProviderName="glassfish" moduleName="web" probeProviderName="jsp-new" class="com.sun.enterprise.web.jsp.JspProbeEmitterImpl"> <!--  glassfish:web:jsp-new:loaded -->    <probe name="loaded">       <method>jspLoadedEvent</method>       <probe-param type="javax.servlet.Servlet" name="servlet" />       <return-param type="void" />    </probe> </probe-provider> </probe-providers>

5. Clients

We support various clients out of the box. You just need to follow the instructions to write the ProbeProviders and ProbeListeners and register with Monitoring Infrastructure, all the stats are made available with the following clients.

i. JMX/AMX

We use Gmbal infrastructure to expose the Statistics objects as JMX/AMX. As part of the MonitoringFramework.register API, we are exposing these Statistics objects as JMX/AMX automatically.

ii. REST

We use the REST infrastructure to expose the Statistics objects as REST API. As part of the MonitoringFramework.register API we notify the REST infrastructure to expose these Statistics objects as REST automatically. Please see REST blog (by Rajeshwar) which explains clearly the various REST features for Monitoring.

iii. Admin Console

Most of the statistics that you will see in the asadmin, will be exposed using the Admin Console.

iv. asadmin list-probes*

GlassFish related probes can be either viewed with dtrace -l 'glassfish*' command or using the asadmin list-probes command. This command will list all the probes in the GlassFish runtime.

v. Help for a probes

You can additionally view what each probe is all about, by doing

$> asadmin --help list-probes glassfish:web:jsp:jspLoaded

This will show the manpage for the specified probe.

vi. asadmin get/list

The asadmin get and list commands (with --monitor=true option to differentiate from the config get/list commands) will automatically expose the monitoring data, once you publish your statistic objects into the monitoring registry. It would look something like below

vii. asadmin monitor

To be able to plug in your own monitoring type and view its related output, you should be able to use the pluggability feature of the CLI monitor command. Please follow the link for an explanation on how this works.

This command displays commonly monitored statistics for Enterprise Server components and services. Monitoring output is displayed continuously in a tabular format; the --interval option can be used to display output at a particular interval (default 30 seconds).

$> asadmin monitor --type=webmodule server
asc  ast   rst   st    ajlc  mjlc tjlc   aslc    mslc      tslc
0     0     0     1     0     0      0     8        8          8

$> asadmin monitor --type=jvm server
                               JVM Monitoring
UpTime(ms)                          Heap and NonHeap Memory(bytes)
current                   min             max             low       high      count
11623062             8585216    167313408   0          0          38531072

6. Architecture

To get a bigger picture of overall v3 monitoring, look at the following architecture diagram.

7. References

Making Module Monitorable
Guidelines for writing ProbeProviders and StatsProviders
Writing Probes - Presentation
Monitoring Functional Specification