Snort, Logstash, Elastic Search and Kibana…

After having fun with Suricata’s new eve/json logging format and the Logstash/Elastic Search/Kibana combination (see this and this), I wanted to get my Snort events into Elastic Search as well.  Using my idstools python library I wrote u2json, a tool that will process a unified2 spool directory (much like barnyard) and convert the events to Suricata-style JSON.

Usage is relatively simple, assuming Snort is logging to /var/log/snort, the following command line should do:

  idstools-u2json -c /etc/snort/snort.conf 
--directory /var/log/snort
--prefix unified2.log
--follow --bookmark
--output /var/log/snort/alerts.json

As the output is in the same format as Suricata’s you can refer to this guide for the Logstash setup.

One extra step I did was use Logstash to add an “engine” field to each entry.  This can be accomplished by adapting the following Logstash configuration:

input {
file {
path => ["/var/log/suricata/eve.json"]
codec => json
type => "suricata-json"
}
file {
path => ["/var/log/snort/alerts.json"]
codec => json
type => "snort-json"
}
}

filter {
if [type] == "suricata-json" {
mutate {
add_field => {
"engine" => "suricata"
}
}
}

if [type] == "snort-json" {
mutate {
add_field => {
"engine" => "snort"
}
}
}
}

Checkout out the documentation for information.

Advertisements

9 thoughts on “Snort, Logstash, Elastic Search and Kibana…

  1. hi, im running the idstools-u2json (installed thru pip) via:

    idstools-u2json @/etc/snort/idstools-u2json.conf , my conf is:

    cat /etc/snort/idstools-u2json.conf
    –snort-conf=/etc/snort/snort.conf
    –directory=/var/log/snort
    –prefix=snort.log
    –follow
    –bookmark
    –delete
    –output=/var/log/snort/alerts.json
    ==== EOF ====

    after starting is see:
    Loaded 24831 rule message map entries.
    Loaded 38 classifications.
    Opened file /var/log/snort/snort.log.1432047824

    but the alerts.json file is empty (its created, but size 0)

    i can see the snort log growing, but the continous parsing is not doing much. Could You point me in the right direction

    • Looks like I need to cut a new release, as well as update the blog post.

      idstools-u2json is now idstools-u2eve, u2json is more a basic script not providing eve compatibility anymore, but serves more as an example application. You can probably get going by updating via pip:

      pip install –upgrade https://github.com/jasonish/py-idstools/archive/master.zip

      (this is likely what will be tagged as the next release after some documentation cleanup).

      Then call idstools-u2eve instead of idstools-u2json, using the same arguments. When running from the command line you can add –sdtout to see the json on stdout, as well as send it to the log file.

      • hi, thx for reply and sorry for confusion. The issue was on my snort.conf side. After i uncommeted the proper unified2 logtype, ur tool is working great. If theres anything to add to the parser – maybe a hardfail when a logfile is of different type than expected? Other than that – it working great! thanks

      • The behaviour is to silently wait for the log file to appear instead of failing out. Perhaps a warning message when it starts up with nothing to read – if the file exists, and its of a different, it will hard fail out.

  2. Hello. I know this is an old post, but I’ve been trying to get suricata logs to an ELK server.

    I am using filebeats on the suricata server to send the output to the ELK server.

    But from everything I have seen, it looks like I need to install an instance of logstash on the actual suricata server to get the correct formatting to send to ES. Is that right? Or is that just easier?

    I am seeing the output on the ELK server, but the detail is just combined in the message.

    So far I have not found anything about setting up suricata and filebeat and just sending it off that way.

    • I have only used logstash-forwarder to send the eve events to another logstash instance directly, or have run logstash alongside Suricata where it can handle the log files directly. Both worked without issue.

      I see that Filebeats has replaced logstash-forwarder, I have not tried it yet but will do so soon.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s