So, my mail services were working last night, but not today. A huge thunderstorm went through the city this morning. The Monster came screaming into the bedroom, the wife yelled, "The computers!", and I ran upstairs just in time for the UPSes to kick in. I powered everything down and went back to bed. I haven't gotten mail at my domain since then. Notice that my cable modem was up, so someone upstream could have gotten mad at me and cut me off. I am able to send mail out via the cox.net relay. Oh yeah, cox filters ports, but they claim they only filter port 25 outgoing. You have to connect to one of their machines.

Alright, lets start debugging this puppy. The first steps are to send email to both an account at the domain and somewhere else, say at work or gmail. If you get the remote one, you at least know email is flowing. I can check that off. As I said, I can also send email out - so I know something is working. The next thing I tried was local email:

[spud@adept ~]$ /usr/sbin/sendmail -v tdh@excfb.com
fkfkkjfsljklf11111111111111
.
tdh@excfb.com... Connecting to [127.0.0.1] via relay...
220 adept.internal.excfb.com ESMTP Sendmail 8.13.6/8.13.4; Tue, 6 Jun 2006 21:23:28 -0500
>>> EHLO adept.internal.excfb.com
250-adept.internal.excfb.com Hello localhost.localdomain [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH GSSAPI DIGEST-MD5 CRAM-MD5
250-DELIVERBY
250 HELP
>>> MAIL From: SIZE=28 AUTH=spud@adept.internal.excfb.com
250 2.1.0 ... Sender ok
>>> RCPT To:
>>> DATA
250 2.1.5 ... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 k572NSmc004845 Message accepted for delivery
tdh@excfb.com... Sent (k572NSmc004845 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 adept.internal.excfb.com closing connection

And all that tells me is that I can connect via the loopback - not very interesting at all, is it? How about a quick test from another machine on the subnet?

# telnet adept 25
Trying 192.168.2.108...
Connected to adept.internal.excfb.com.
Escape character is '^]'.
220 adept.internal.excfb.com ESMTP Sendmail 8.13.6/8.13.4; Tue, 6 Jun 2006 21:25:53 -0500
help
214-2.0.0 This is sendmail version 8.13.6
214-2.0.0 Topics:
214-2.0.0       HELO    EHLO    MAIL    RCPT    DATA
214-2.0.0       RSET    NOOP    QUIT    HELP    VRFY
214-2.0.0       EXPN    VERB    ETRN    DSN     AUTH
214-2.0.0       STARTTLS
214-2.0.0 For more info use "HELP ".
214-2.0.0 To report bugs in the implementation send email to
214-2.0.0       sendmail-bugs@sendmail.org.
214-2.0.0 For local information send email to Postmaster at your site.
214 2.0.0 End of HELP info

To recap, I've shown that sendmail is working on my mail server and is accepting protocol on port 25. Lets show that it fails miserably from a remote system:

Last login: Tue Jun  6 16:19:47 2006 from ip68-0-87-35.tu.ok.cox.net
-bash-2.05b$ uname -a
Linux virt18c.secure-wi.com 2.4.22-1.2199.5.legacy.nptlsmp #1 SMP Sat Apr 30 21:00:06 EDT 2005 i686 i686 i386 GNU/Linux
-bash-2.05b$ telnet mail.excfb.com 25
Trying 68.0.87.35...

telnet: connect to address 68.0.87.35: Connection timed out
-bash-2.05b$ telnet mail.excfb.com XXXX
Trying 68.0.87.35...
Connected to mail.excfb.com.
Escape character is '^]'.

Get /




Method Not Implemented
Get to /index.html not supported.



Apache/2.0.54 (Fedora) Server at www.excfb.com Port 80

Connection closed by foreign host.
-bash-2.05b$ man traceroute
-bash-2.05b$ man traceroute
-bash-2.05b$ traceroute -p 25 mail.excfb.com
traceroute to mail.excfb.com (68.0.87.35), 30 hops max, 38 byte packets
 1  207.158.22.1 (207.158.22.1)  0.455 ms  0.752 ms  0.448 ms
 2  sdtc.br02.g4-0-0.americanis.net (206.251.233.237)  0.229 ms  0.262 ms *
 3  unknown.Level3.net (209.245.56.201)  1.235 ms  1.192 ms  49.856 ms
 4  ge-7-0-0.mp2.SanDiego1.Level3.net (4.68.113.69)  1.216 ms  1.238 ms  1.077 ms
 5  ae-0-0.bbr2.Dallas1.Level3.net (64.159.1.110)  29.348 ms as-3-0.bbr1.Dallas1.Level3.net (64.159.3.214)  29.469 ms  29.750 ms
 6  ge-7-0-0-56.gar1.Dallas1.Level3.net (4.68.122.162)  29.972 ms ge-6-0-0-51.gar1.Dallas1.Level3.net (4.68.122.2)  29.941 ms ge-7-0-0-52.gar1.Dallas1.Level3.net (4.68.122.34)  30.055 ms
 7  COX-ENTERPRI.gar1.Level3.net (4.78.232.2)  38.508 ms  38.185 ms  38.366 ms
 8  68.12.14.34 (68.12.14.34)  38.207 ms 68.12.14.22 (68.12.14.22)  38.482 ms 68.12.14.34 (68.12.14.34) 38.232 ms
 9  68.12.14.65 (68.12.14.65)  44.302 ms 68.12.14.61 (68.12.14.61)  42.770 ms 68.12.14.65 (68.12.14.65) 44.176 ms
10  10.5.0.1 (10.5.0.1)  43.758 ms  43.521 ms  44.908 ms
11  ip68-0-87-35.tu.ok.cox.net (68.0.87.35)  53.471 ms  57.920 ms  53.876 ms
12  ip68-0-87-35.tu.ok.cox.net (68.0.87.35)  53.540 ms  53.614 ms  51.476 ms
-bash-2.05b$
-bash-2.05b$

Note that I'm not convinced that the ping -p 25 means anything valid. It does tell me that there is a path, but not much more. I happen to not have root on any remote boxes which can directly connect to my mail server, so the direct sendmail -v will not work. What will? How about this mail relay checker: whatsdown.net

This May Take a Minute or Two.. Please Wait...

Connecting to excfb.com (68.0.87.35)...
*Could not connect to excfb.com (68.0.87.35): Operation timed out
*
Test Failed

To recap, my mail is being serviced quite fine internally, but it looks like my port 25 is being blocked. This blockage could be my firewall, my cable modem, or my ISP (cox.net). I rebooted my router/firewall - it has a small syslog buffer and I can't see anything other than DOS type attacks. The next trick, since it is out of hours for customer support from my ISP, is to reboot the cable modem and then my router. I've had to wait until Gilmore Girls got Tivo'ed, otherwise I would be in a different world of trouble.

No, that did not improve anything. I'm going to go out on a limb and blame my ISP. If my router/firewall were hosed, I shouldn't be able to get in at all. I really hate cox.net for tech support. The original cable contract was in my wife's name and I have to know all of her security information in order to get help. You can't post URLs for their web pages, they use javascript to foil that. And I see contradictory information on whether or not they block port 25 inbound.

Frak...

The Top 100 Things I'd Do If I Ever Became An Evil Overlord