I'm kind of surprised no one has suggested simply writing your own
script to deal with the data. There are some great text parsing tools
out there (grep, Awk, Sed, Perl, etc. etc.) that can be leveraged to
create a system that does exactly what you need. Another plus is it can
be customized to work with any firewall logging format and easily
tweaked to deal with multiple firewall products at the same time. Let me
give you a few examples to get the wheels turning.
Here's the general process:
*Look at your firewall log and ID patterns that you recognize (SMTP
to/from your mail server, HTTP to your Web server, etc.)
*Identify the criteria that makes those log entries unique Vs. other log
entries (IP(s), port(s), packet direction, transport, etc.)
*Create a command that will parse this log entry out of the main
firewall log and put it into its own unique file
*Rinse, lather, repeat
*Whatever is left (the stuff you don't expect or understand) gets put
into its own file
*Add these commands to a batch file or shell script
*Set this script up as a cron job or scheduled task to go off early in
the morning every day
*When you come in at the start of your work day focus on the file with
the interesting stuff and work out from there
Let's use a PIX log as an example with grep as the parsing tool:
Nov 13 21:02:28 firewall %PIX-2-106001: Inbound TCP connection denied
from 1.2.3.4/52716 to 192.168.1.10/25 flags SYN on interface outside
Nov 13 21:02:32 firewall %PIX-2-106001: Inbound TCP connection denied
from 1.4.3.2/1327 to 192.168.1.11/80 flags SYN on interface outside
PIX sends its logs to a Syslog server so there is probably more than
just firewall logs in the messages file. First thing we need to do is
extract only the firewall log entries. In the above example the PIX has
the unique host name "firewall" so we can key in on this. We can further
key in on "PIX" if we want to ensure we have no extraneous entries:
grep firewall /var/log/messages.1 | grep PIX > firewall.log
From here we can just focus in on the "firewall.log" file to parse out
what we understand. For example the first log entry is obviously SMTP to
our mail server. Consider the following set of commands:
grep '192.168.1.10/25' firewall.log > inbound_smtp.txt
grep -v '192.168.1.10/25' firewall.log > temp1.txt
The first line grabs all inbound SMTP to our mail server and shoves it
in an appropriately named file. The second line grabs all lines *except*
these SMTP entries and writes them to a temp file.
From there we can do something like this:
grep '192.168.1.11/80' temp1.txt > inbound_http.txt
grep -v '192.168.1.11/80' temp1.txt > temp2.txt
Similar process for our Web server, with one minor difference. Rather
than parse the original firewall.log file, we parse the temp file we
created in the last set of commands. This file will obviously be smaller
than firewall.log so this second search will go much faster. In the
second line we extract all entries but HTTP to our Web server and
generate another new temp file to work with.
You then just continue this process of stripping out what you understand
and dumping it to a file. When you are done, your last line will look
something like this:
ren temp20.txt interesting_stuff.txt
or on Linux/UNIX:
mv temp20.txt interesting_stuff.txt
So now when you go to do your analysis after the script has run, you can
focus right in on "interesting_stuff.txt" which will include all the
traffic patterns you don't normally see. Pretty easy way to clear out
large log files so you are left with something that is easy to review.
If you need to look for additional patterns (say the extent of a port
scan), you can always use grep to check for the pattern back in the
original firewall.log file.
A couple of caveats; grep is the easiest tool to learn but probably the
least flexible. It is also limited in that the entire log file must get
loaded into memory in order to be parsed. Not ideal if you are trying to
parse a 5 GB firewall log file.
Perl is probably the weapon of choice as it is far more flexible and can
deal with extremely large files as it can work line by line. Downside is
there is a much steeper learning curve to figuring out Perl compared to
grep as it is its own language.
I teach this process through the SANS perimeter track. The exercises
students do in class has them parsing a firewall log with just under
200,000 entries in order to look for six that are an indication of an
internally compromised system. By the end of the exercise they learn
that a properly written script can alert the firewall admin to these log
entries in less than a two minutes review process. Most of us can spare
two minutes out of our day to ID whacked boxes. ;-)
I've always been big on just writing your own scripts to parse your
logs. Its a bit more work on the front end, but IMHO not much more than
trying to figure out the GUI on a new tool that is probably only going
to work with one firewall product anyway.
HTH,
Chris
