Factotum: Documentation Handyman

As I mentioned previously, the topic of message build-up in the MTA message queues prompted this writeup. Here's the second part of this installment, providing some details on what you can do if you discover that your deployment is having an actual problem with message build-up.

Before You Begin

At a minimum, you need to enable logging for the channels that you want to troubleshoot. The amount of logging that you set (log level) depends on your situation. To enable logging on a channel and learn about other options, see Managing MTA Message and Connection Logs in Sun Java System Messaging Server 6.3 Administration Guide for more information.

Troubleshooting TCP Channels

This section describes a general approach to troubleshoot build up of messages in TCP message queues to determine if there is an actual problem.

1. When you suspect that there is something happening that is more than the store-and-forward aspect of messages building up in a message queue, begin by using the imsimta qm summarize command.

   For more information, see imsimta qm in Sun Java System Messaging Server 6.3 Administration Reference.

2. The imsimta qm summarize command can greatly impact your system if you have a large backlog of messages. Instead of running this command frequently, consider using the imsimta qm messages channel command instead.

   This command lists the destination hosts for which messages are queued in the specified channel. This command also lists how many messages are waiting for their next schedule retry (the delayed messages column) versus how many are ready to be retried now (active now). When you use the imsimta qm messages command, you must specify a channel name; a wildcard is not valid input. For example:

   # imsimta qm
   qm.maint> messsages tcp_local
   host                             active messages  delayed messages
                            example.com           0                 2000
                            sesta.com             0                 3000

   In this example, 2,000 messages are waiting for their next scheduled retry to be delivered to example.com, and 3,000 are waiting for their next scheduled retry for delivery to sesta.com.

   ---

   Note - The Messaging Server 6.3 release removed the imsimta qm messages command. However, Messaging Server 6.3 does contain a new, useful command—imsimta qm jobs—to help understand why messages are not being delivered.

   ---

   This information is also useful for the situation where messages have failed and are waiting to be retried. If you see there are many messages in a channel queue, but most of them are delayed, this probably indicates the problem is with the remote domain. See Destination Host Problems for information on how to workaround this problem.

   ---

   Note - A message can be in process of being tried by a job, or on a channel waiting to be tried, or on a channel waiting to be retried. The messages command active messages column includes messages which have not been tried yet and those which were previously delayed and are now ready to be tried again. That is why you might see a zero (0) in this column. In Messaging Server 6.3, you can see the messages being retried with the new jobs command.

   ---

   iPlanet Messaging Server 5. Use top -to channel or top -domain_to channel to analyze what is going on in that channel.

3. Look for trends on your system. For example, when most of the mail is all destined for one remote domain, check the status of that remote domain.

   Additionally, look in the mail.log_current file to determine what has happened in recent history when you tried to send mail to that remote domain.

4. Use the imsimta qm dir -to address command to select a group of messages. Then use this information to look at the delivery attempt history of some of the messages. (You use the sequence numbers from the dir listing). Often, you will find that these messages are all non-delivery notifications for spam, which was not deliverable. If this is the case, determine how those original spam messages got into the system in the first place. Verify that the messages are spam by using the imsimta qm subcommands dir, read, and history. If this is indeed the case, think about routing the non-delivery notifications through a different outbound channel, thereby preventing them from choking the normal tcp_local channel queue.

   For example, use the notificationchannel and dispositionchannel keywords to specify an alternate process channel to queue delivery status notifications and modify status notifications, respectively. Then you use source-specific rewrite rules to direct messages from these process channels to a particular tcp_* channel set to only use a few processes/threads. For more information, see Source-Channel-Specific Rewrite Rules ($M, $N) in Sun Java System Messaging Server 6.3 Administration Guide.

5. Verify that the master process for the channel is started. The tcp_* channels all use the smtp_client process. To find out which process is associated with which channel, in respect to dequeuing, see the master_command parameter in the associated channel block in the job_controller.cnf file.

Destination Host Problems

When you have determined that messages are queued to an unavailable remote host, you have two options:

1. Create a new channel for the host. If this host is consistently a problem, all future email will go to this new channel. For existing messages that are enqueued, you can either wait for the problem with the destination host to be resolved or delete the messages from the queue.

2. Increase the number of delivery threads for the channel, or set a ceiling on the number of queued messages that will trigger a new thread or process to start. See the max_client_threads parameter in the channel option file and the threaddepth channel keyword, respectively.

Troubleshooting the ims-ms Channel

This section describes a general approach to troubleshoot build up of messages in the ims-ms channel. The four general cases where the ims-ms channel shows a build-up of messages in the queue are:

• IMAP_MAILBOX_LOCKED. While you might see this error in a message file that is briefly in the queue area, typically such a message file doesn't remain for long. The error only repeats in a message file in the queue area if the mailbox is remaining locked for an extended period. The job controller retries delivery of these messages after short delays until either the message gets delivered or a different error is encountered.

• IMAP_MAILBOX_BADFORMAT, IMAP_MAILBOX_NOTSUPPORTED. The mailbox is most likely corrupted. This case rarely occurs. You might want to use the reconstruct command for these cases.

• IMAP_IOERROR. The message store is most likely corrupted or otherwise inaccessible. This case occurs even more rarely.

• IMAP_QUOTA_EXCEEDED. The user or users are over quota. This is the most common case, which this technical note discusses below.

---

Note - If the channel gets a permanent delivery failure error, then the message is immediately bounced and does not remain in the ims-ms queue area.

---

To troubleshoot the ims-ms channel, use the following high-level approach:

1. Perform a similar investigation as you would for tcp_* channels by using the imsimta qm summarize command to view what is happening on the system.

2. Use the imsimta qm history command to examine the message IDs to detect if there are different sorts of messages. For example, you might see:

   Message id: 800
   Filename: /opt/SUNWmsgsr/data/queue/tcp_local/001/ZZf0b0KaNZykG.00

   A message's file name starting with ZZ indicates that it has not been tried yet. The message file name is a counter starting at ZZ and decremented (ZY, ZX, and so on) each time the message is tried, fails, and is reenqueued for later retry. Thus, a ZZ* file name has not been tried yet, and there is no history.

   In general, but not always, when you have non-ZZ*, non-.HELD files in the queue area, you have the IMAP_QUOTA_EXCEEDED case. (The frequency with which you see IMAP_MAILBOX_LOCKED conditions probably depends upon user and email client characteristics. This condition is more common with users who like to receive and move around lots of large attachments but it should typically occur rarely.)

   For a site that enforces quota, probably most of the non-ZZ*, non-.HELD messages in the ims-ms queue area are there because of the recipient user being over quota. Verify this is the case by running the imsimta qm command with the history subcommand. You should see “over quota" in the history of the over-quota messages.

   ---

   Note - In iPlanet Messaging Server 5.2, the imsimta qm top command was enhanced to have more sorting options.

   ---

3. Are there any Q status messages in the mail.log_current file pertaining to the ims-ms channel? When you see “mailbox is busy” and Q status in the mail.log_current file, then the message is put back on the queue to be retried later as per the job controller's scheduling and the backoff keyword on the channel.

4. If not, check that the ims_master process is running. Are there any errors in its log file (the imta file)? The ims_master process could be hung. Use the imsimta process command to verify running processes.

Use the following strategies for users who become over quota:

1. Inform users of the need to perform mailbox maintenance to return to under quota status, or increase their quota.

2. Reduce the time that mail is queued for over quota accounts before being bounced back as over quota. See the store.quotagraceperiod configutil parameter. If you don't want to queue email for over quota accounts (and bounce the message straight back), set this parameter to 0 (that is, no grace period). This parameter is available in iPlanet Messaging Server 5 as well.

3. For Messaging Server 6, you can enable the local.store.overquotastatus configutil parameter. This enables quota enforcement before messages are enqueued in the MTA and prevents the MTA from filling up.

More About the ims-master Process

At times you might see the ims-master process shutting down and starting up in the log file:

# > imta /opt/SUNWmsgsr/logs
# grep "Sun Java" imta
[30/Aug/2006:17:05:05 -0400] learn ims_master[19736]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down
[30/Aug/2006:17:05:20 -0400] learn ims_master[28310]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) starting up
[30/Aug/2006:17:07:24 -0400] learn ims_master[28310]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down
[30/Aug/2006:17:07:32 -0400] learn ims_master[28380]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) starting up
[30/Aug/2006:17:19:31 -0400] learn ims_master[28380]: General Notice: Sun Java(t
m) System Messaging Server ims_master 6.2-7.02 (built Jun 13 2006) shutting down

This is normal operation and does note indicate a problem. This is a “notice” message (not an “error” or “critical” level message). As with all channel jobs, ims-ms channel jobs shut down from time to time based on either having nothing to do, or based on “timing out” (getting old). Then the job controller restarts new jobs as needed.

Configuration Issues

You might want to increase the number of processes the job controller can start for the tcp_local, tcp_intranet, or other tcp_* channels, or increase the number of threads each of those processes will start. You might also want to give the tcp_local channel its own pool. If you observe queued messages (total across all queues) to be greater than 100,000, increase the value of MAX_MESSAGES for the job_controller.cnf setting. See Job Controller Configuration File in Sun Java System Messaging Server 6.3 Administration Reference for more information.

Additional Information

This section contains additional information to help you understand MTA operations.

What Are .HELD Messages?

If the MTA detects that messages are bouncing between servers or channels, delivery is halted and the messages are stored in a file with the suffix .HELD in the msg-srv-base/data/queue/channel directory. Typically, a message loop occurs because each server or channel thinks the other is responsible for delivery of the message. You need to manually fix these .HELD messages with the imsimta process held command.

There is an unfortunate collision of terminology and concepts between .held messages and the hold channel. And worse still, the command to process .held messages is called release, whereas the command to process messages on the hold channel is called process_held.

You use the hold channel to hold messages of a recipient temporarily prevented from receiving new messages. For example, you might be moving a user's mailbox and want to hold new incoming messages. The hold channel is located in the msg-svr-base/queue/hold directory. Messages are written to this queue as ZZxxx.held files. Because the job controller doesn't “see” these .held files, they are not dequeued for delivery. You release these files with the imsimta qm release command, and the reprocess daemon reprocesses them.

See To Temporarily Hold Messages Using the Hold Channel in Sun Java System Messaging Server 6.3 Administration Guide for more information.

How Does a Message Become a .HELD Message?

Messaging Server makes use of MAX_*_RECEIVED_LINES options that you set in theoption.dat file to determine when a message is put into the .HELD state. The most relevant options and their default values are:

• MAX_LOCAL_RECEIVED_LINES=10

• MAX_RECEIVED_LINES=50

• MAX_TOTAL_RECEIVED_LINES=100

Once a message has looped through the MTA enough to accumulate MAX_RECEIVED_LINES header lines indicating the local MTA, then the message becomes .HELD. You can cause the MTA to immediately recognize that it has connected to itself, rather than waiting to accumulate MAX_LOCAL_RECEIVED_LINES local Received: headers, by specifying the loopcheck keyword on the appropriate channel(s) in the imta.cnf file.

For More Information on Troubleshooting the MTA

Use the following to aid in troubleshooting the MTA:

• Messages are Not Dequeued in Sun Java System Messaging Server 6.3 Administration Guide

• MTA Messages are Not Delivered in Sun Java System Messaging Server 6.3 Administration Guide

• Messages are Looping in Sun Java System Messaging Server 6.3 Administration Guide

I've been working with two of our Messaging Server experts to come up with an article on Messaging Server's Message Transfer Agent (MTA) and how it handles the build-up of messages in its message queues. This has been a fairly frequent topic of inquiry from the Comms community that presents itself as, 'I'm noticing a build-up of messages in my queues, should I be worried about it.'

The short of it is, not necessarily, because of the way the MTA is designed to work as a store-and-foward message system. The long of it is, maybe, but there's more to consider than just number of messages in the queue.

So, here's part one (of two) on my take on this situation, based on the experiences of two of our support engineers. I'd be interested in hearing from others and their experiences, to see if this article should be expanded/corrected.

Troubleshooting Sun Java System Messaging Server MTA Message Queues

Part I

This article describes how to troubleshoot the Sun Java System Messaging Server Message Transfer Agent (MTA), specifically message build-up in channels, including the TCP channels (tcp_local and tcp_intranet) and ims-ms channel.

Products covered by this article are:

• Sun Java System Messaging Server 6

• iPlanet Messaging Server 5

---

Note - This technical note assumes you are running Sun Java System Messaging Server 6. Where appropriate, iPlanet Messaging Server 5 commands are also mentioned.

---

This article contains the following topics:

• About the MTA and Channels

• When Is a Message Queue Having a Problem?

• Main Causes of Backed-up Message Queues

• Before You Begin<

• Troubleshooting TCP Channels

• Destination Host Problems

• Troubleshooting the ims-ms Channel

• Configuration Issues

• Additional Information

About the MTA and Channels

To begin understanding why messages build up in a channel queue, and whether an actual problem might be occurring on your system, you must first understand how mail messages flow through the MTA.

The MTA can be thought of as Messaging Server's central brain, responsible for message routing. The MTA takes in messages via SMTP sessions from other systems and decides what to do with those messages. The first stop for a message is the MTA SMTP server, which executes programs to handle the SMTP session. Based on numerous configuration possibilities, the SMTP server processes the message, which could include message blocking, address changing, or channel enqueueing.

Actually, the sequence of events is a bit more complex. The dispatcher spawns the SMTP server by listening on port 25 (or whichever port is defined for the SMTP server in the dispatcher.cnf file). When the dispatcher detects an attempt to connect to port 25, it starts an SMTP process to handle the incoming connection. The SMTP server typically decides whether to accept or reject the message based on numerous configuration possibilities. During the SMTP dialogue, the MTA machinery kicks in to decide what to do with the message. However, the PORT_ACCESS mapping table works with the dispatcher rather than the SMTP server to allow or block access to certain ports such as the SMTP port (port 25).

The focus of this technical note is the decision to route the message to a channel where the message is to be enqueued. A channel is a message connection with another system or destination. Once enqueued to a channel, the message's destination could be another server (either on the Internet or on your company's intranet), a remote message store, a specific domain name, a channel for extra processing, such as virus filtering, or a local message store.

When a channel contains messages but is not delivering them, the messages build up in the channel's message queue. The message queues are directories, located by default in the msg-svr-base/data/queue/channel/ directory. In this way the MTA holds the messages for future delivery when whatever situation preventing their delivery is resolved.

The specific channels discussed in this technical note are the TCP channels (tcp_local and tcp_intranet) and the ims-ms channel. The tcp_local channel is responsible for routing messages to the Internet, while the tcp_intranet channel is responsible for delivering messages to remote message stores on your company's intranet. The tcp_intranet channel also routes messages to any intermediary internal systems on their way to another system. The ims-ms channel is responsible for delivering messages to the local message store.

For a complete description of the MTA architecture and message flow, see MTA Architecture and Message Flow Overview in Sun Java System Messaging Server 6.3 Administration Guide.

When Is a Message Queue Having a Problem?

Though this might be counter-intuitive, in general, it is not a problem for messages to build up in a message queue. Indeed, the MTA is designed to handle this situation. Internet mail (SMTP) is a store-and-forward mail system. Internet mailers are designed to store the messages that cannot yet be delivered. Keep in mind that it is perfectly normal to not be able to deliver outgoing SMTP messages immediately. Network problems, problems on remote hosts, problems with remote users' mailboxes, and so forth, are common cases that cause the MTA to hold and not deliver messages. MTAs, including Messaging Servers', therefore can store and retry outgoing messages. An often encountered case has to do with message store users being over quota and hence their messages are waiting for retry. From the MTA design point of view, this exactly the same case as being unable to deliver a message because of a network problem. The MTA handles the over quota situation with exactly the same underlying mechanisms: the messages are stored in queues where they will be retried for delivery. Thus, just because you are seeing a build-up of messages in queues is not necessarily cause for alarm.

The real issue about backed-up queues is when should you have cause to worry and so try to troubleshoot the source of the problem. The next sections explain the main causes of queue backups and how to go about troubleshooting such situations.

Main Causes of Backed-up Message Queues

In general, four situations can cause messages to back up in the MTA message queues:

• Performance problems. Simply put, this occurs when your system is unable to keep up. Performance problems manifest themselves by high system loads and processes running flat out. In such situations, you need to perform an in-depth look at the system to be able to tune it and reduce the load.

• Destination host problems. Regardless of what is causing problems on the destination host (networking problems, system problems, and so on), this situation can cause all delivery threads to become “stuck,” stopping valid email from getting through, or causing a lot of queued emails. The “stuck” problem presents itself as a lot of active messages waiting to be processed but with a low load. The “queued” problem shows as lots of queued email with long delivery attempt history.

  ---

  Tip - To contact the administrator of a domain that is causing you problems, use the whois command or http://www.netsol.com/cgi-bin/whois/whois.

  ---

• Problems with Messaging Server itself. An example of a Messaging Server problem is when the stored process has an orphan lock for a user account, resulting in message build up for users in the ims-ms channel.

• Configuration issues. Two common configuration issues include the job controller ignoring queued email or queued email due to overquota accounts, or queued email due to slow directory response, common for big mailing lists.

Part Two of this article will discuss courses of action to the above situations, and provide more information on troubleshooting. Stay tuned.

There was an interesting conversation on the Info-iMS alias last week concerning "bounced email" that caught my eye. Ah, another learning opportunity! To recap, then, here is another installment of Comms 101, courtesy of our Comms experts.

The specific error sited was:

nsmail Illegal host/domain name found (Too many failures to this host during this run; skipping this host: No such host/domain)

The short reply as to what's going on is fairly intuitive: the remote domain in question failed to resolve to a legal result, meaning that almost certainly this is a DNS issue (and not a Messaging Server MTA issue). Note that "other mail" will continue to make it through to your system (expected behavior), so don't be mystified by that activity. In addition, the error message is communicating that the problem has happened multiple times during the current operation (run) of the channel. The message bouncing is thus occurring because of an underlying DNS problem. To troubleshoot this problem, you would need to enable debugging (such as on LOG_CONNECTION) to determine the specific issue. Very likely, such a problem is transitory in nature but it is clearly not an MTA issue. Of final note: fancy configuration workarounds to such a situation are typically more work than troubleshooting and fixing the real problem.

The full detail on this thread can be found here:

http://lists.balius.com/pipermail/info-ims-archive/2007-April/027757.html

Let's face it, Sun Java System Messaging Server is a complex product. The MTA is a very powerful piece of code, enabling you to do all kinds of email routing/filtering/etc. I won't even pretend to understand it to the extent that I should.

What I can do, as a writer, though, is to try and pass along those aspects of the product, that, if we had the time, would make up the basics, those concepts that seem to come up regularly within our internal aliases and set of questions. One of those examples would be defragmentation.

The MTA in Messaging Server contains a feature that enables automatic defragmentation of messages or partial messages. What does this mean? To quote from the Messaging Server 6 2005Q4 Administration Guide:

"The MIME standard provides the message/partial content type for breaking up messages into smaller parts. This is useful when messages have to traverse networks with size limits, or traverse unreliable networks where message fragmentation can provide a form of “checkpointing,” allowing for less subsequent duplication of effort when network failures occur during message transfer. Information is included in each part so that the message can be automatically reassembled after it arrives at its destination."

You implement defragmentation by using the defragment channel keyword and the defragmentation channel. Furthermore, the defragment channel only retains messages in the defragment channel queue only for a limited time. When one half of the time before the first nondelivery notice is sent has elapsed, the various parts of a message will be sent on without being reassembled. This choice of time value eliminates the possibility of a nondelivery notification being sent about a message in the defragment channel queue.

Okay, knowing all this, where does that leave us? Well, you might use the imsimta qm > summarize command to look at what is happening with your MTA message queues, and you might observe that there are x number of messages queued up in the defragmentation channel. Furthermore, in diving into reading the defrag'd files, you see that there are actually y number more parts to the messages. Is this a problem, you begin to wonder, with the defragmentation channel?

The short answer is no. In fact, the MTA and the defragmentation channel are working normally. What has happened is that a message has been fragmented into y parts but you only have x parts in the defragmentation channel queue. Because y is greater than x, no matter what the specifics of the messages in the queue, there are not enough parts present for reassembly to be performed. This situation continues until either all the parts arrive or the time limit for reassembly (noted above) is reached, at which point the message fragments will be sent on unassembled.

A basic, best practice for defragmentation is also to put the defragment keyword on a channel destined for a third-party scanner box, since scanners might take the approach of rejecting all fragmented messages (if the scanner cannot assemble the fragments together itself).

I hope to do more of these Messaging Server "basics" pieces over time. Stay tuned for more.