Mr Wiggles,
"Capture" is the packet capture capability on the PIX. Unless you
configured the PIX to capture packets you won't see this bug.
Liberty for All,
Brian
-----Original Message-----
From: Mr Wiggles [mailto:astalavista.box.sk@gmail.com]
Sent: Thursday, January 12, 2006 7:46 AM
To: Andrew Shore
Cc: firewalls@securityfocus.com
Subject: Re: PIX replying with wrong SRC MAC?
That was my first thought too, but he says no changes were made to the
server or the firewall. One thing that would seem to dispell that
theory though is that the firewall can ping and send syslog messages to
the server, but the server cannot ping the firewall.
I mentioned we upgraded to 6.3(5) and he was still having the same
problem but I did not log in and troubleshoot when it was at version
6.3(5). This may be relevant as the release notes for this version say:
CSCea40885
Yes
PIX - Capture sometimes records wrong MAC addr for PIXs
Which would indicate this symptom which caused me to stop looking
elsewhere for the problem was likely just a result of a defective
capture as opposed to a defective packet being captured.
I have requested access to both the server and the PIX so that I can
perform TCPdumps and troubleshoot some more but this release note from
6.3(5) seems to answer my question about the bogus mac address..
On 1/12/06, Andrew Shore <andrew.shore@holistecs.com> wrote:
Just to save some time, is the internal server running any kind
of firewall
and was there a server (/server firewall) update done the night
this went
all wrong?
My gut feeling about this is that you're starting to look in the
wrong place
first.
-----Original Message-----
From: astalavista.box.sk@gmail.com
[mailto:astalavista.box.sk@gmail.com]
Sent: 11 January 2006 15:00
To: firewalls@securityfocus.com
Subject: PIX replying with wrong SRC MAC?
Tried the cisco forums (on cisco.com ) to no avail so I figured
I would pos
this here:
Having a problem here on a 506 6.3(4) that the user says popped
up out of
nowhere one morning with no changes having been made to the
config. There is
a 506 a layer 2 dumb switch, and then 1 internal server hanging
off this
switch. the PIX can ping the server but the server cannot ping
the PIX.
lots of troubleshooting was done but most all of those steps
only answered
questions that are also answered by the following which is the
result of a
capture for all traffic between the inside IP of the pix and the
IP of the
server:
18:07:47.322065 0800.20c5.93f6 0800.20c5.93f6 0x0800 98:
12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10920)
18:07:48.322325 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR >
12.12.3.161:
icmp: echo request (ttl 255, id 25094)
18:07:48.322371 0800.20c5.93f6 0800.20c5.93f6 0x0800 98:
12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10921)
18:07:49.322676 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR >
12.12.3.161:
icmp: echo request (ttl 255, id 25095)
18:07:49.322737 0800.20c5.93f6 0800.20c5.93f6 0x0800 98:
12.12.3.161 > THOR:
icmp: echo reply (ttl 255, id 10922)
18:07:50.323011 0800.20c5.93f6 0003.e300.2851 0x0800 98: THOR >
12.12.3.161:
icmp: echo request (ttl 255, id 25096)
18:07:50.323072 0800.20c5.93f6 0800.20c5.93f6 0x0800 98:
12.12.3.161 <http://12.12.3.161> > THOR:
icmp: echo reply (ttl 255, id 10923)
18:08:40.136314 0003.e300.2851 0800.20c5.93f6 0x0800 135:
12.12.3.161.514 >
THOR.514: [udp sum ok] udp 93 (ttl 255, id 10924)
18:09:56.289932 0003.e300.2851 0800.20c5.93f6 0x0800 74:
12.12.3.161 > THOR:
icmp: echo request (ttl 255, id 10925)
18:09:56.290405 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR >
12.12.3.161 <http://12.12.3.161> :
icmp: echo reply (DF) (ttl 255, id 25097)
18:09:57.288101 0003.e300.2851 0800.20c5.93f6 0x0800 74:
12.12.3.161 > THOR:
icmp: echo request (ttl 255, id 10926)
18:09:57.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR >
12.12.3.161:
icmp: echo reply (DF) (ttl 255, id 25098)
18:09:58.288116 0003.e300.2851 0800.20c5.93f6 0x0800 74:
12.12.3.161 <http://12.12.3.161> > THOR:
icmp: echo request (ttl 255, id 10927)
18:09:58.288406 0800.20c5.93f6 0003.e300.2851 0x0800 74: THOR >
12.12.3.161:
icmp: echo reply (DF) (ttl 255, id 25099)
18:09:59.288452 0003.e300.2851 0800.20c5.93f6 0x0800 135:
12.12.3.161.514 >
THOR.514: [udp sum ok] udp 93 (ttl 255, id 10928)
as you can see, the top portion is where it was NOT working and
th eproblem
is the PIX is replying to these echo-requests with an echo-reply
packet that
has the server's mac as both the src and dst. On the bottom you
can see a
ping from the PIX to the server (which works) where the Macs are
as they
should be. This same behavior may be occuring on the outside
interface but I
cannot tell because I do not have access to theupstream router
to initiate a
ping.
Any ideas what could cause this?
(no input/output errors or collisions on interfaces, duplex
settings are
fine, debug packet and debug icmp trace show packets coming and
going, when
connected with just a hub in the middle the server can even see
(in a
tcpdump) the ICMP echo replies but the pings still fail (bogus
SRC addresson
the reply))
- - - - - - - - - - - - - - - - - - -
someone on the cisco forums blindly suggested proxyarp was the
problem but
never returned to explain how this could be the case. my reply
to that:
Host is on the inside, IP of .162 pinging inside interface of
firewall .161
Process would be ARP request from .162 asking who has .161,
response from
.161. Then the ICMP Echo request is sent, arrives at the
firewall.
Firewall sends out an arp request "who has .162?" Which is where
proxyarping
would come into play if anywhere since .162 is a host behind the
firewall
with a static built in the firewall (static (inside, outside)
x.x.x.x
x.x.x.x netmask 255.255.255.255). So if proxyarping was the
problem it would
now allegedly use proxy arp to respond to its own ARP request
(which doesnt
sound plausible to begin with) in which case the problem would
be that the
src and dst macs would both be those of the firewall. That is
not the case,
the source and destination MACs are both those of the
server....how could
proxy arping change what the firewall uses as its own source mac
in its
replies?
sh arp in the firewall shows the server's IP associated with its
correct mac
also, if proxyarp was the problem wouldnt it affect the packets
similarly
when the pix pinged the server? This problem does not exist when
the pix
pings the server, only when the server pings the pix (see logs
above).
Can you explain how proxy arp is causing this?
- - - - - - - - - - - - - - - -
how can a packet have the same src and dst mac address? If
nothing else
wouldnt that cause problems with the switch's cam table since it
would see
the incoming packet and say "oh my, the server is now on this
interface
(referring the the FW interface the ICMP _reply_ just came in
from)"
I have performed a write-erase, rebooted at least 20 times at
all different
stages in the troubleshooting process, cleared arp, xlate, conn
and
everything in between and the FW has even been upgraded to
6.3(5)
your thoughts?
