RE: Localhost packets on WAN

> Once on the 26th and 8 times today we received packets from 
> 127.0.0.1:80 to an ephemeral port on one of our WAN IPs.  

Frequently, when the source port is 80 and the destination port is "ephemeral", 
I find problems like this are usually caused by buggy or misconfigured load 
balancers in front of a web site.  Some load balancers get your packet to the 
physical server by doing tricks with the network stack.  

It's not inconceivable that there's a technique that masquerades the request 
and makes the physical server think it's coming over the host's loopback 
adapter.

Perhaps one of your clients is attempting a connection against such a server 
farm, but one of the physical servers isn't accepting connections.  It might 
send an RST in reply to the connection request (via the load balancer via 
127.0.0.1).  If the load balancer doesn't know how to handle that situation 
(maybe it's rare on paper, since the load balancer would normally not route 
traffic there in the first place), it's possible it doesn't fix it up like it 
should and the source address remains 127.0.0.1 and leaks.

It's also conceivable that some other bug mangled a packet destined to a 
physical server, which responds by closing the connection.  As above, the load 
balancer doesn't handle that packet correctly and it leaks with an improper 
source address.

Of course, this is all just conjecture.  You might try examining some traffic 
just before this packet is received and see if there's any legitimate HTTP 
traffic going on.

Good luck,
 
David