Easy Unified2 File Reading in Python

I recently consolidated my Python code bits for dealing with Snort and Suricata unified2 log files into a project called idstools. While I’ll be adding more than just unified2 reading support, that is about it for now.

While it can be installed with pip (pip install idstools), if you just want to play around with it I suggest cloning the repo (git clone https://github.com/jasonish/idstools.py). You can then use the REPL or write test scripts from within the idstools.py directory without having to install the library (yeah, basic stuff for Python developers).

idstools does come with a few example programs that demonstrate unified2 file reading, namely, u2fast.py, u2tail.py and u2spewfoo.py (a simple clone of the Snort provided u2spewfoo).

Basic Unified2 File Reading

from idstools import unified2

reader = unified2.FileEventReader("tests/merged.log")
for event in reader:
    print("Event:n%s" % str(event))

This few lines of code will iterate through each record in the specified unified2 log files, aggregate the records into events and return each event as a dict.

If straight up record is reading is more what you are after then check out unified2.FileRecordReader, or the lower level unified2.read_record function.

Each event is represented as a dict containing the fields of a unified2 event record, with the associated packets represented as a list in event[“packets”] and extra data records represented as a list in event[“extra-data”].

Resolving Event Message and Classification Names

To make event reading just a little more useful, code to map signature and classifications IDs to descriptions is provided.
from idstools import maps

# Create and populate the signature message map.
sigmap = maps.MsgMap()
sigmap.load_genmsg_file("gen-msg.map")
sigmap.load_sidmsg_file("sid-msg.map")

# Get the description for 1:498.
print("Message for 1:498: %s" % (sigmap.get(1, 498).msg))

# Create and populate the classification map.
classmap = maps.ClassificationMap()
classmap.load_classification_file("classification.config")
print("The description for classification id 9 is %s, with priority %d." % (
        classmap.get(9).description, classmap.get(9).priority))
The example program u2fast.py is a complete example of reading events from one or more files, resolving event descriptions and classification names and printing the event in a “fast” like style.

Spool Reading

idstools also contains a spool reader for processing a unified2 spool directory as commonly used by Snort and Suricata.  It supports bookmarking, deleting files, and open and close hooks which can be used to implement custom archiving.
from idstools import spool

def my_open_hook(reader, filename):
    print("File %s has been opened." % (filename))

def my_close_hook(reader, filename):
    print("File %s has been closed." % (filename))

reader = spool.Unified2EventSpoolReader(
    "/var/log/snort", "merged.log", delete_on_close=False,
    bookmark=True,
    open_hook=my_open_hook,
    close_hook=my_close_hook)

for event in reader:
    print("Read event with generator-id %d, signature-id %d." % (
            event["signature-id"], event["generator-id"]))
To see a more complete directory spool process, check out the u2tail.py example program.To learn more checkout idstools over at GitHub, PyPI, or the work-in-progress documentation on Read the Docs.

Advertisements

3 thoughts on “Easy Unified2 File Reading in Python

  1. could tell me how to slove that
    Log:
    ———————————————————————————————————–
    Traceback (most recent call last):
    File “C:\Users\Rockey\Downloads\python-code\unified2_reader.py”, line 5, in
    for record in reader:
    File “C:\Python34\lib\idstools\unified2.py”, line 528, in next
    return read_record(self.fileobj)
    File “C:\Python34\lib\idstools\unified2.py”, line 501, in read_record
    raise err
    EOFError
    ———————————————————————————————————–

    • Looks like a bug. The intention is for idstools to catch when a full record can’t be read, rewind and return None so the caller can try again later. Instead I see that I do the rewind, but then raise the exception instead of returning None. Should be a simple fix that I’ll look at real soon now.

      If you want, file an issue at https://github.com/jasonish/py-idstools/issues/new

      • Might not be a bug. Can you share your code? If you are not using one of the File or Spool readers you will get an EOFError if you hit the end of file before the end of a record.

        If you are reading from static files, its time to stop. If you are reading from a growing file, you can try calling next again – ideally after waiting a moment for more data to arrive.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s