Just to save some time, is the internal server running any kind of firewall
and was there a server (/server firewall) update done the night this went
all wrong?
My gut feeling about this is that you're starting to look in the wrong place
first.
-----Original Message-----
From: astalavista.box.sk@gmail.com [mailto:astalavista.box.sk@gmail.com]
Sent: 11 January 2006 15:00
To: firewalls@securityfocus.com
Subject: PIX replying with wrong SRC MAC?
Tried the cisco forums (on cisco.com) to no avail so I figured I would post
this here:
Having a problem here on a 506 6.3(4) that the user says popped up out of
nowhere one morning with no changes having been made to the config. There is
a 506 a layer 2 dumb switch, and then 1 internal server hanging off this
switch. the PIX can ping the server but the server cannot ping the PIX.
lots of troubleshooting was done but most all of those steps only answered
questions that are also answered by the following which is the result of a
capture for all traffic between the inside IP of the pix and the IP of the
server:
18:07:47.322065 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10920)
18:07:48.322325 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161:
icmp: echo request (ttl 255, id 25094)
18:07:48.322371 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10921)
18:07:49.322676 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161:
icmp: echo request (ttl 255, id 25095)
18:07:49.322737 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10922)
18:07:50.323011 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR > 12.12.3.161:
icmp: echo request (ttl 255, id 25096)
18:07:50.323072 0800.20c5.93f6 0800.20c5.93f6 0x0800 98: 12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10923)
18:08:40.136314 0003.e300.2851 0800.20c5.93f6 0x0800 135: 12.12.3.161.514 >
THOR.514: [udp sum ok] udp 93 (ttl 255, id 10924)
18:09:56.289932 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR:
icmp: echo request (ttl 255, id 10925)
18:09:56.290405 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161:
icmp: echo reply (DF) (ttl 255, id 25097)
18:09:57.288101 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR:
icmp: echo request (ttl 255, id 10926)
18:09:57.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161:
icmp: echo reply (DF) (ttl 255, id 25098)
18:09:58.288116 0003.e300.2851 0800.20c5.93f6 0x0800 74: 12.12.3.161 > THOR:
icmp: echo request (ttl 255, id 10927)
18:09:58.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR > 12.12.3.161:
icmp: echo reply (DF) (ttl 255, id 25099)
18:09:59.288452 0003.e300.2851 0800.20c5.93f6 0x0800 135: 12.12.3.161.514 >
THOR.514: [udp sum ok] udp 93 (ttl 255, id 10928)
as you can see, the top portion is where it was NOT working and th eproblem
is the PIX is replying to these echo-requests with an echo-reply packet that
has the server's mac as both the src and dst. On the bottom you can see a
ping from the PIX to the server (which works) where the Macs are as they
should be. This same behavior may be occuring on the outside interface but I
cannot tell because I do not have access to theupstream router to initiate a
ping.
Any ideas what could cause this?
(no input/output errors or collisions on interfaces, duplex settings are
fine, debug packet and debug icmp trace show packets coming and going, when
connected with just a hub in the middle the server can even see (in a
tcpdump) the ICMP echo replies but the pings still fail (bogus SRC addresson
the reply))
- - - - - - - - - - - - - - - - - - -
someone on the cisco forums blindly suggested proxyarp was the problem but
never returned to explain how this could be the case. my reply to that:
Host is on the inside, IP of .162 pinging inside interface of firewall .161
Process would be ARP request from .162 asking who has .161, response from
.161. Then the ICMP Echo request is sent, arrives at the firewall.
Firewall sends out an arp request "who has .162?" Which is where proxyarping
would come into play if anywhere since .162 is a host behind the firewall
with a static built in the firewall (static (inside, outside) x.x.x.x
x.x.x.x netmask 255.255.255.255). So if proxyarping was the problem it would
now allegedly use proxy arp to respond to its own ARP request (which doesnt
sound plausible to begin with) in which case the problem would be that the
src and dst macs would both be those of the firewall. That is not the case,
the source and destination MACs are both those of the server....how could
proxy arping change what the firewall uses as its own source mac in its
replies?
sh arp in the firewall shows the server's IP associated with its correct mac
also, if proxyarp was the problem wouldnt it affect the packets similarly
when the pix pinged the server? This problem does not exist when the pix
pings the server, only when the server pings the pix (see logs above).
Can you explain how proxy arp is causing this?
- - - - - - - - - - - - - - - -
how can a packet have the same src and dst mac address? If nothing else
wouldnt that cause problems with the switch's cam table since it would see
the incoming packet and say "oh my, the server is now on this interface
(referring the the FW interface the ICMP _reply_ just came in from)"
I have performed a write-erase, rebooted at least 20 times at all different
stages in the troubleshooting process, cleared arp, xlate, conn and
everything in between and the FW has even been upgraded to 6.3(5)
your thoughts?
smime.p7s
Description: S/MIME cryptographic signature
