Olivier Rivat's Blog: Blog of Olivier Rivat

This blog describes how to integrate Glassfish V2.1 SNMP with HPOV

[Read More]

It is possible with Glassfish V2.1 to monitor different domains. In this blog, we will explain how to proceed.

Relationships between domain/DAS/ SNMP

The DAS (Domain administrative server) is the central piece of administration. A DAS is corresponding to a specific domain associated to a given administrative port.

 DAS with server instance, cluster instances within same domain

Glassfish V2.1 allows to create server and cluster instances instances within the same domain. Glassfish SNMP V2.1 allows to monitor SNMP at:

• DAS level
• standalone instance level
• cluster instance level

It is even possible possible to deploy the snmp war file on the DAS, and on cluster instances at the same time by using different SNMP port. This is described in the SNMP documentation at:

• http://docs.sun.com/app/docs/doc/820-7189/ghzpz?a=view

Configuring Glassfish SNMP to monitor different domains

In this section, is shown how to monitor different domains through Glassfish SNMP. Let's assume that we create 3 different domains: domain1, domain2, domain3. Those 3 domains need to have 3 different administrative ports (4849, 4850, 4851)

asadmin create-domain --port 4849 domain1 --user admin
asadmin create-domain --port 4850 domain2 --user admin
asadmin create-domain --port 4851 domain3 --user admin

The user needs to  assign a specfic SNMP port for each domain:

asadmin set --port 4849 --user admin server.property.snmp-adapter-port=10163
asadmin set --port 4850 --user admin server.property.snmp-adapter-port=10164
asadmin set --port 4851 --user admin server.property.snmp-adapter-port=10165

The user  should restart each domain for each snmp-adapter port value to be taken account

asadmin stop-domain domain1
asadmin start-domain domain1
asadmin stop-domain domain2
asadmin start-domain domain2
asadmin stop-domain domain3
asadmin start-domain domain3

The user can now deploy the Glassfish snmp war file within each domain

asadmin deploy --port 4849 __assnmp.war
asadmin deploy --port 4850 __assnmp.war
asadmin deploy --port 4851 __assnmp.war

Viewing snmp data within the different domains

snmpwalk -c public -v 1 <hostname>:10163 J2EE-MIB::j2eeDomTable
J2EE-MIB::j2eeDomMoName.1 = STRING: "domain1"
J2EE-MIB::j2eeDomEnterprise.1 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeDomMoStateManaged.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoStatProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoEventProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomSMState.1 = INTEGER: running(4)
J2EE-MIB::j2eeDomSMStartTime.1 = STRING: "Thu Aug 06 10:13:56 MEST 2009"

 snmpwalk -c public -v 1 <hostname>:10164 J2EE-MIB::j2eeDomTable
J2EE-MIB::j2eeDomMoName.1 = STRING: "domain2"
J2EE-MIB::j2eeDomEnterprise.1 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeDomMoStateManaged.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoStatProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoEventProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomSMState.1 = INTEGER: running(4)
J2EE-MIB::j2eeDomSMStartTime.1 = STRING: "Thu Aug 06 10:11:43 MEST 2009"

snmpwalk -c public -v 1 hostname:10165 J2EE-MIB::j2eeDomTable
J2EE-MIB::j2eeDomMoName.1 = STRING: "domain3"
J2EE-MIB::j2eeDomEnterprise.1 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeDomMoStateManaged.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoStatProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoEventProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomSMState.1 = INTEGER: running(4)
J2EE-MIB::j2eeDomSMStartTime.1 = STRING: "Thu Aug 06 10:14:37 MEST 2009" 
  

TroubleShooting Glassfish Snmp

If you are not able to see the SNMP values, you check the glassfish server.log log file, and see within the log file to check that the SNMP value is correct for this specific domain:

• SNMP Adapter Port: <value port> ( or example 10164 in case of domain2)
  

     SNMP security for Glassfish

This glassfish SNMP release supports SNMP V1 and V2 only. Snmp Security is quite weak within SNMP V1 and V2, as only some limited security can be put in place through community string.

A much more complete SNMP security solution will be available when Glassfish SNMP V3 support will be provided.

   Using Community String for Security

Hereafter is described how to configure the SNMP master agent with community string for security. Community string are supported in SNMP V1 and V2. With this approach, a minimal SNMP security support can be provided through SNMP master agent configuration.

The main idea consists of configuring SNMP security at SNMP master agent level. The SNMP master agent is using a configuration file. It is within this configuration file, that the user we will configure the SNMP community string. SNMP master agent configuration is operating system specific, and is described in the previous section. Instructions are provided for Solaris. For other platforms, the user should refer to the man page of snmp master agent of your operating system.


Before doing this, the user should configure the SNMP master agent to talk to glassfish SNMP agent (described in the previous section). It is also advised changing the GlassfishSnmp port adapter to something different from the default value 10161. The SNMP master Agent can be running on a different host, than the host on which Glassfish is running. All SNMP user's request should go to the SNMP master agent.

   Using community string on Solaris 10

The user needs to create a community string within the file snmpd.conf. It is possible to add the host or subnet from which this community can be accessed. The user should refer to the man page of snmpd.conf* for complete information.


rocommunity community [source] [OID]
Create read-only communities that can be used to access the agent. They are a quick wrapper around the more complex and powerful com2sec, group, access, and view directive lines. They are not as efficient as these, because groups are not created, so the tables are potentially larger. These directives are not recommended for complex environments. If your environment is relatively simple or you can sustain a small negative performance impact, use these directives.
Source can be a hostname, a subnet, or the word default. A subnet can be specified as IP/mask or IP/bits. The first source/community combination that matches the incoming packet is selected.
The OID token restricts access for that community to everything below that given OID.

     Example1

# access granted using community string mfwk
rocommunity mfwk
proxy -v1 -c public <ipaddress of glassfish installation>:10161 1.3.6.1.4.1.42.2.9999.1.1.1.1 Once the System Administrator has modified the snmpd.conf file, he has to restart the snmpd daemon:
/etc/init.d/init.sma start

User will have to indicate a community string when connecting to the SNMP master agent. Any other request not specifying the correct community string will be rejected.

•    snmpwalk -c mfwk -v 1 <hostname> J2EE-MIB::j2eeSrvMoName

---> J2EE-MIB::j2eeSrvMoName.1.1 = STRING: "name=server"

     Example2

# access granted using community string mfwk on the subnet 10.10.10.255
rocommunity mfwk 10.10.10.0/24
proxy -v1 -c public <ipaddress of glassfish installation>:10161 1.3.6.1.4.1.42.2.9999.1.1.1.1

snmpwalk -c mfwk -v 1 <hostname> J2EE-MIB::j2eeSrvMoName J2EE-MIB::j2eeSrvMoName.1.1 = STRING: "name=server"

In example2, snmpwalk user request will succeed only if the user's machine belongs to the specified subnet (i.e 10.10.10.25)

Glassfish SNMP Master Agent Integration

You can proxy the SNMP requests from the master agent for your operating system to the Glassfish SNMP agent of your deployment.

This allows to:

• define a specific SNMP entry point port (161), independent from the Glassfish SNMP adapter port (usually 10161, but which can be configured)
• allow to provide limited security support through community string.
  

The SNMP Master Agent can run on a different  host than the one where Glassfish is deployed. The configuration of the SNMP Master Agent is operating system specific.  The configuration of the SNMP master agent on Solaris 10 is provided as example.
Solaris 10 platform SNMP master agent configuration

For more complete information about SNMP master agent configuration, refer to  the man page for snmpd.conf.

To configure SNMP on the Solaris S10 platform, follow these steps:

• 1. Using a text editor, add the following lines to the end of the /etc/sma/snmp/snmpd.conf file.

   rocommunity public
   proxy -c public -v 1 <gf-server-hostname>:10161 1.3.6.1.4.1.42.2.9999.1.1.1

   gf-server-hostname is the fully qualified hostname on which glassfish server SNMP adapter port is running. 10161 is teh glassfishsnmp port number ( which can be customized). 

• 2. Restart the snmpd daemon using the following command:

   /etc/init.d/init.sma restart

• 3. Verify the snmpd status using the following command:

   /etc/init.d/init.sma status

Performing SNMP requests through SNMP Master Agent

snmp requests are directly done on the snmp master agent. Note that there is no snmp port supplied in this request, as teh request will go through the standard snmp port adapter 161. All Snmp requests sent to this port will be proxied  to Glassfish SNMP  adaptor port acccoring to the mapping done in snmpd.conf

snmpwalk -c mfwk -v 1 <hostname> -m J2EE-MIB J2EE-MIB::j2eeSrvMoName
J2EE-MIB::j2eeSrvMoName.1.1 = STRING: "cluster=cl1,name=cl1_ins1"
J2EE-MIB::j2eeSrvMoName.1.2 = STRING: "name=server"
J2EE-MIB::j2eeSrvMoName.1.3 = STRING: "name=sa_ins3"
J2EE-MIB::j2eeSrvMoName.1.4 = STRING: "cluster=cl1,name=cl1_ins2"

Presentation

There are many different MIB browser that you can use to browse a MIB.

The one I am using are:

•    snmpwalk

snmpwalk 

snmpwalk, snmpget, snmpgetnext snmp utilities are bundled with any Solaris distribution (after Solaris 10). (The same also for linux as well).  It is possible to browse very easily a MIB using those commands.

snmp utilities commands are described in UNIX man pages

• man snmpwalk

• man snmpget

snmpalk allows to browse very easily the J2EE-MIB

Example:

•    retrieving the snmp J2eesrvTable from the J2EE-MIB

snmpwalk -c public -v 1 <hostname>:10161 1.3.6.1.4.1.42.2.9999.1.1.1.1.1.2

iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.2.1.1 = STRING: "name=server"
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.3.1.1 = OID: iso.3.6.1.4.1.42
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.5.1.1 = STRING: "Sun Microsystems, Inc."
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.6.1.1 = STRING: "Sun Java System Application Server 9.1.1"
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.7.1.1 = INTEGER: 1
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.8.1.1 = INTEGER: 1
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.9.1.1 = INTEGER: 2
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.10.1.1 = INTEGER: 4
iso.3.6.1.4.1.42.2.9999.1.1.1.1.1.2.1.11.1.1 = STRING: "Fri Oct 31 15:47:37 MET 2008"


By providing as parameter the J2EE-MIB definition on the command line (using the -m option), the user is getting a more readable output

snmpwalk -c public -v 1 <hostname>:10161 -m J2EE-MIB J2EE-MIB::j2eeMoGroup | more
J2EE-MIB::j2eeDomMoName.1 = STRING: "domain1"
J2EE-MIB::j2eeDomEnterprise.1 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeDomMoStateManaged.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoStatProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomMoEventProv.1 = INTEGER: false(2)
J2EE-MIB::j2eeDomSMState.1 = INTEGER: other(1)
J2EE-MIB::j2eeDomSMStartTime.1 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeSrvMoName.1.1 = STRING: "cluster=cl1,name=cl1_ins1"
J2EE-MIB::j2eeSrvMoName.1.2 = STRING: "name=server"
J2EE-MIB::j2eeSrvMoName.1.3 = STRING: "name=sa_ins3"
J2EE-MIB::j2eeSrvMoName.1.4 = STRING: "cluster=cl1,name=cl1_ins2"
J2EE-MIB::j2eeSrvEnterprise.1.1 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeSrvEnterprise.1.2 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeSrvEnterprise.1.3 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeSrvEnterprise.1.4 = OID: J2EE-MIB::sun
J2EE-MIB::j2eeSrvVendor.1.1 = STRING: Sun Microsystems, Inc.
J2EE-MIB::j2eeSrvVendor.1.2 = STRING: Sun Microsystems, Inc.
J2EE-MIB::j2eeSrvVendor.1.3 = STRING: Sun Microsystems, Inc.
J2EE-MIB::j2eeSrvVendor.1.4 = STRING: Sun Microsystems, Inc.
J2EE-MIB::j2eeSrvVersion.1.1 = STRING: Sun GlassFish Communications Server 1.5
J2EE-MIB::j2eeSrvVersion.1.2 = STRING: Sun GlassFish Communications Server 1.5
J2EE-MIB::j2eeSrvVersion.1.3 = STRING: Sun GlassFish Communications Server 1.5
J2EE-MIB::j2eeSrvVersion.1.4 = STRING: Sun GlassFish Communications Server 1.5
J2EE-MIB::j2eeSrvMoStateManaged.1.1 = INTEGER: true(1)
J2EE-MIB::j2eeSrvMoStateManaged.1.2 = INTEGER: true(1)
J2EE-MIB::j2eeSrvMoStateManaged.1.3 = INTEGER: true(1)
J2EE-MIB::j2eeSrvMoStateManaged.1.4 = INTEGER: true(1)
J2EE-MIB::j2eeSrvMoStatProv.1.1 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoStatProv.1.2 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoStatProv.1.3 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoStatProv.1.4 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoEventProv.1.1 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoEventProv.1.2 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoEventProv.1.3 = INTEGER: false(2)
J2EE-MIB::j2eeSrvMoEventProv.1.4 = INTEGER: false(2)
J2EE-MIB::j2eeSrvSMState.1.1 = INTEGER: stopped(3)
J2EE-MIB::j2eeSrvSMState.1.2 = INTEGER: running(4)
J2EE-MIB::j2eeSrvSMState.1.3 = INTEGER: stopped(3)
J2EE-MIB::j2eeSrvSMState.1.4 = INTEGER: stopped(3)
J2EE-MIB::j2eeSrvSMStartTime.1.1 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeSrvSMStartTime.1.2 = STRING: "Thu Feb 05 16:06:55 MET 2009"
J2EE-MIB::j2eeSrvSMStartTime.1.3 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeSrvSMStartTime.1.4 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeJVMVersion.2.1 = STRING: 1.5.0_12
J2EE-MIB::j2eeJVMVendor.2.1 = STRING: Sun Microsystems Inc.
J2EE-MIB::j2eeJVMEnterprise.2.1 = OID: J2EE-MIB::sun
....
...

snmpget: Retrieving a specfic OID

Whereas snmpwalk is browsing the entire MIB, it is possible to retrieve a specific OID value using snmpget

Example:

snmpget -c public -v 1 jesma115:10161 J2EE-MIB::j2eeSrvVersion.1.1
  --->  J2EE-MIB::j2eeSrvVersion.1.1 = STRING: Sun Java System Application Server 9.1.1

Useful Tips

snmpwalk is very powerful, and allows to browse  very easily specific needs.
Example: Getting Server information

browsing the server list of the domain:

•    Getting the server list: 

snmpwalk -c public -v 1 <hostname>:10161 J2EE-MIB::j2eeSrvMoName
J2EE-MIB::j2eeSrvMoName.1.1 = STRING: "cluster=cl1,name=cl1_ins1"
J2EE-MIB::j2eeSrvMoName.1.2 = STRING: "name=server"
J2EE-MIB::j2eeSrvMoName.1.3 = STRING: "name=sa_ins3"
J2EE-MIB::j2eeSrvMoName.1.4 = STRING: "cluster=cl1,name=cl1_ins2"

•    Getting the Servers State

snmpwalk -c public -v 1 <hostname>:10161 J2EE-MIB::j2eeSrvSMState
J2EE-MIB::j2eeSrvSMState.1.1 = INTEGER: stopped(3)
J2EE-MIB::j2eeSrvSMState.1.2 = INTEGER: running(4)
J2EE-MIB::j2eeSrvSMState.1.3 = INTEGER: stopped(3)
J2EE-MIB::j2eeSrvSMState.1.4 = INTEGER: stopped(3)

•    Getting the Server Start Time

snmpwalk -c public -v 1 <hostname>:10161 J2EE-MIB::j2eeSrvSMStartTime
J2EE-MIB::j2eeSrvSMStartTime.1.1 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeSrvSMStartTime.1.2 = STRING: "Thu Feb 05 16:06:55 MET 2009"
J2EE-MIB::j2eeSrvSMStartTime.1.3 = STRING: "NOT_APPLICABLE"
J2EE-MIB::j2eeSrvSMStartTime.1.4 = STRING: "NOT_APPLICABLE"

In this deployment, the domain is made up of 4 different glassfish server instance:

•    one for the DAS (server)
• instance in cluster cl1 (cl1_ins1, cl1_ins2) 
• one standalone instance  (sa_ins3)
  

Only the DAS server is running and the other  server instances are stopped.

Example: getting information about a deployed webapplication

The following example is showing a deployed webapplication webapp1 , which contains modules and servlets

snmpwalk -c public -v 1 <hostname>:10161 J2EE-MIB::j2eeMoGroup | grep webapp1
J2EE-MIB::j2eeAppMoName.2.11 = STRING: "name=webapp1,server=server"
J2EE-MIB::j2eeModMoName.2.11.14 = STRING: "application=webapp1,name=//server/webapp1webmod2,server=server"
J2EE-MIB::j2eeModMoName.2.11.15 = STRING: "application=webapp1,name=//server/webapp1webmod1,server=server"
J2EE-MIB::j2eeSletMoName.2.11.14.36 = STRING: "application=webapp1,name=webapp1webmod2_Servlet2,webmodule=//server/webapp1webmod2,server=server"
J2EE-MIB::j2eeSletMoName.2.11.14.37 = STRING: "application=webapp1,name=jsp,webmodule=//server/webapp1webmod2,server=server"
J2EE-MIB::j2eeSletMoName.2.11.14.38 = STRING: "application=webapp1,name=default,webmodule=//server/webapp1webmod2,server=server"
J2EE-MIB::j2eeSletMoName.2.11.14.39 = STRING: "application=webapp1,name=webapp1webmod2_Servlet1,webmodule=//server/webapp1webmod2,server=server"
J2EE-MIB::j2eeSletMoName.2.11.15.40 = STRING: "application=webapp1,name=webapp1webmod1_Servlet1,webmodule=//server/webapp1webmod1,server=server"
J2EE-MIB::j2eeSletMoName.2.11.15.41 = STRING: "application=webapp1,name=jsp,webmodule=//server/webapp1webmod1,server=server"
J2EE-MIB::j2eeSletMoName.2.11.15.42 = STRING: "application=webapp1,name=default,webmodule=//server/webapp1webmod1,server=server"
J2EE-MIB::j2eeSletMoName.2.11.15.43 = STRING: "application=webapp1,name=webapp1webmod1_Servlet2,webmodule=//server/webapp1webmod1,server=server"

Glassfish Snmp J2EE MIBJ2EE-MIB Presentation

Glassfish V2.1 is providing SNMP J2EE-MIB support.

The J2EE-MIB is specified in the JSR77 specification, and also available at:

• http://java.sun.com/j2ee/tools/management/downloads/J2EE-MIB

The J2EE-MIB is made of 2 major groups:

• J2eeMoGroup: this group contains the definition of all the base Table

• J2eeStatistics: this group contains the definition of all the statistics Table

J2eeMoGroup Tables

The J2eeMoGroup contains the following base Tables

■ j2eeDomTable
■ j2eeSrvTable
■ j2eeJVMTable
■ j2eeAppTable
■ j2eeModTable
■ j2eeBeanTable
■ j2eeSletTable
■ j2eeAdapTable
■ j2eeRsrcTable
■ j2eeJCATable
■ j2eeJDBCTable

j2eeStatistics Group

The j2eeStatistics Group contains the following tables

■ j2eeStatistics Tables
■ j2eeServletStatTable
■ j2eeEjbEntityStatTable
■ j2eeEjbStatelessStatTable
■ j2eeEjbStatefulStatTable
■ j2eeMessageDrivenStatTable
■ j2eeJtaStatTable
■ j2eeJcaConnPoolStatTable
■ j2eeJDBCConnPoolStatTable
■ j2eeJVMStatTable


Glassfish has different monitoring level (OFF , LOW , HIGH). J2eeStatistics Table Entries (apart from j2eeJvmStatTable) are only accessible when monitoring level is to HIGH. The j2eeJvmStatTable is always accessible, even when monitoring is OFF.