Binu John's Weblog

GlassFish Tuning - HTTP Connection Queue & Keep Alive

In my last entry I discussed how to monitor and tune the HTTP thread pool in GlassFish. In this installment, I discuss the monitoring and tuning of the http listener's connection queue and the keep alive settings.

Each HTTP listener has an associated connection queue where requests that are waiting to be serviced are placed. The requests are taken out of the queue and serviced by one of the available processing threads. As expected, as the number of requests waiting to be serviced increases, the response times for those requests also increase. Additionally, the server will reject new requests if the number of items in the queue has reached a maximum configured value (default is 4096). The various parameters that can be configured for the connection queue and their default values are given below.

#asadmin get "server*connection-pool.*"

server.http-service.connection-pool.max-pending-count = 4096

server.http-service.connection-pool.queue-size-in-bytes = 4096

server.http-service.connection-pool.receive-buffer-size-in-bytes = 4096

server.http-service.connection-pool.send-buffer-size-in-bytes = 8192

The queue-size-in-bytes property is used to specify the maximum number of entries in the queue. Once the queue length has reached maximum capacity, the server will reject any new request. For most applications, there is no need to change the send and receive buffer sizes since the default values provide the optimum performance.

Monitoring the connection queue allows the server load to be evaluated and to take appropriate actions when the load increases beyond a certain level. The GlassFish monitoring framework can be used to inspect the various connection queue statistics using the admin CLI or through the Admin GUI.

# asadmin get -m *connection-queue.*count

Some of the important parameters, their description and how to interpret the values are given below.

Note: There is a bug in the measurement of  countoverflows-count that results in incorrect values being displayed. 

Attribute	Description	Comments
countqueued-count	Number of connections currently in the queue	A request in the queue will be processed when a processing thread becomes available. Consistently high values suggest high load on the system, incorrect thread pool tuning or lock contention within the application.
countqueued*minuteaverage-count	Average number of connections queued in the last 1, 5 or 15 minutes	Useful for filtering out short load spikes.
countoverflows-count	Number of times the queue has been too full to accommodate a connection	Rejection of client connections results in poor user experience. If clients can tolerate higher response times, increasing the queue size can reduce the connections rejections. Potential solution for a highly loaded system is to scale the application server tier vertically or horizontally. Verify that the bug (id: 7354) has been fixed.

HTTP/1.1 by default, uses persistent connections in which a single connection is used by the client to make multiple requests. The server maintains the connection in the keep-alive state enabling the user agent to make subsequent requests on the same connection rather than create a new connection for every request (HTTP/1.0). The server closes a connection if one of the the following conditions are met.

• The time elapsed between now and the last request has exceeded the value specified for the timeout-in-seconds parameter.

• The number of requests using a connection has exceeded the value specified by the max-connections parameter.

The max-connections parameter is used to prevent malicious clients from tying up a thread indefinitely when the Grizzly blocking connector is used. This restriction can be eliminated by setting the value to -1 (minus one) if the instance uses the Grizzly NIO connector or the instance is only accessed by trusted clients.

The default keep alive values are given below.

# asadmin get "server*keep-alive.*"

server.http-service.keep-alive.max-connections = 250

server.http-service.keep-alive.thread-count = 1

server.http-service.keep-alive.timeout-in-seconds = 30

The GlassFish monitoring framework can be used to inspect the various keep alive statistics using the admin CLI or through the Admin GUI as shown in the example below.

# asadmin get -m *keep-alive.*count

server.http-service.keep-alive.countconnections-count = 1869

server.http-service.keep-alive.countflushes-count = 0

server.http-service.keep-alive.counthits-count = 359873

server.http-service.keep-alive.countrefusals-count = 1428

server.http-service.keep-alive.counttimeouts-count = 0

server.http-service.keep-alive.maxconnections-count = 250

server.http-service.keep-alive.secondstimeouts-count = 30

Some of the important parameters, their description and how to interpret the values are given below. All values are totals since the start of monitoring, and hence will increase monotonically. To collect relevant values, the statistics for the period of interest can be obtained by collecting the values at the start of the measurement interval and then subtracting those numbers from the subsequent values. The discussions given below pertains to statistics collected during the measurement interval.

Attribute	Description	Comments
countconnections-count	Number of connections in keep-alive mode	If this value is consistently high (several hundreds/core), consider reducing max-connections or increasing the timeout.
counthits-count	Number of cache lookup hits	A high hit rate (keep-alive.counthits-count/request.countrequests-count) means that the current settings are working well.
countrefusals-count	Number of keep-alive connections that were rejected for exceeding the maximum number of requests allowed per connection.	The configuration attribute, keep-alive.max-connections, which limits the number of requests that is allowed for a connection. The client has to open a new connection for subsequent requests. It is advisable to keep this at the default value if the HTTP connector is run in blocking mode. A value of -1 (signifies unlimited requests) can be set for HTTP connectors running in non-blocking mode (default connector) or that servicing trusted clients.
counttimeouts-count	Number of keep-alive connections that timed out	The default keep alive time out is 30 seconds. Increase this value if most of your returning clients take longer than this interval for making subsequent requests. A very high value can cause the number of connections to be keep alive unnecessarily causing performance degradation.

Posted at 11:08AM Mar 16, 2009 by binu in GlassFish  |