Upgraded a pair of ASA firewalls from 7.2(2) to 7.2(4) in an attempt to address some of the recent vulnerability announcements. Seems like an easy enough upgrade, since it's only going from one minor revision to the next....right? Wrong.
Jul 03 2008 08:11:16: %ASA-7-713236: IP = xx.xx.xx.xx, IKE_DECODE RECEIVED Message (msgid=1337c959) with payloads : HDR + HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NOTIFY (11) + NONE (0) total length : 176
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing hash payload
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing SA payload
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing nonce payload
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing ID payload
Jul 03 2008 08:11:16: %ASA-7-714011: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, ID_IPV4_ADDR ID received
Jul 03 2008 08:11:16: %ASA-7-713025: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, Received remote Proxy Host data in ID Payload: Address 172.16.16.1, Protocol 0, Port 0
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing ID payload
Jul 03 2008 08:11:16: %ASA-7-714011: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, ID_IPV4_ADDR_SUBNET ID received--xx.xx.xx.xx--255.255.255.0
Jul 03 2008 08:11:16: %ASA-7-713034: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, Received local IP Proxy Subnet data in ID Payload: Address xx.xx.xxx.xx, Mask 255.255.255.0, Protocol 0, Port 0
Jul 03 2008 08:11:16: %ASA-7-715047: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, processing notify payload
Jul 03 2008 08:11:16: %ASA-3-713227: IP = xx.xx.xx.xx, Rejecting new IPSec SA negotiation for peer xx.xx.xx.xx. A negotiation was already in progress for local Proxy xx.xx.xx.xx/255.255.255.0, remote Proxy xx.xx.xx.xx/255.255.255.255
Jul 03 2008 08:11:16: %ASA-3-713902: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, QM FSM error (P2 struct &0x6413a88, mess id 0x1337c959)!
Jul 03 2008 08:11:16: %ASA-7-715065: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, IKE QM Responder FSM error history (struct &0x6413a88) <state>, <event>: QM_DONE, EV_ERROR-->QM_BLD_MSG2, EV_IS_REKEY-->QM_BLD_MSG2, EV_CONFIRM_SA-->QM_BLD_MSG2, EV_PROC_MSG-->QM_BLD_MSG2, EV_HASH_OK-->QM_BLD_MSG2, NullEvent-->QM_BLD_MSG2, EV_COMP_HASH-->QM_BLD_MSG2, EV_VALIDATE_MSG
Jul 03 2008 08:11:16: %ASA-7-713906: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, sending delete/delete with reason message
Jul 03 2008 08:11:16: %ASA-7-713906: Received unexpected event EV_REMOVE in state MM_WAIT_DELETE
Jul 03 2008 08:11:16: %ASA-3-713902: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, Removing peer from correlator table failed, no match!
Jul 03 2008 08:11:16: %ASA-7-713906: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, IKE SA MM:5a58159b rcv'd Terminate: state MM_ACTIVE flags 0x0001c042, refcnt 1, tuncnt 0
Jul 03 2008 08:11:16: %ASA-7-720041: (VPN-Primary) Sending Phase 1 Terminate message (type L2L, remote addr xx.xx.xx.xx, my cookie 5A58159B, his cookie 94006566) to standby unit
Jul 03 2008 08:11:16: %ASA-7-713906: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, IKE SA MM:5a58159b terminating: flags 0x0101c002, refcnt 0, tuncnt 0
Jul 03 2008 08:11:16: %ASA-7-713906: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, sending delete/delete with reason message
Jul 03 2008 08:11:16: %ASA-7-715046: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, constructing blank hash payload
Jul 03 2008 08:11:16: %ASA-7-715046: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, constructing IKE delete payload
Jul 03 2008 08:11:16: %ASA-7-715046: Group = xx.xx.xx.xx, IP = xx.xx.xx.xx, constructing qm hash payload
Jul 03 2008 08:11:16: %ASA-7-713236: IP = xx.xx.xx.xx, IKE_DECODE SENDING Message (msgid=34d011ee) with payloads : HDR + HASH (8) + DELETE (12) + NONE (0) total length : 76
Jul 03 2008 08:11:16: %ASA-4-113019: Group = xx.xx.xx.xx, Username = xx.xx.xx.xx, IP = xx.xx.xx.xx, Session disconnected. Session Type: IPSecLAN2LAN, Duration: 0h:00m:00s, Bytes xmt: 0, Bytes rcv: 0, Reason: Unknown
A show isakmp sa give me a few tunnels in the state:
37 IKE Peer: xx.xx.xx.xx
Type : L2L Role : responder
Rekey : no State : MM_WAIT_DELETE
A peek over on the secondary:
18 IKE Peer: xx.xx.xx.xx
Type : L2L Role : responder
Rekey : yes State : MM_WAIT_DELETE
Well, after a bunch of troubleshooting, and finally calling TAC I find that this is realted to bug CSCse45327
Now, if you will notice this bug was FIXED in 7.2(2), and is not listed for any versions after 7.2(2), after all it was fixed. However, that appears to not be the case. Just a warning for all that might tread this way