After having fun with Suricata’s new eve/json logging format and the Logstash/Elastic Search/Kibana combination (see this and this), I wanted to get my Snort events into Elastic Search as well. Using my idstools python library I wrote u2json, a tool that will process a unified2 spool directory (much like barnyard) and convert the events to Suricata-style JSON.
Usage is relatively simple, assuming Snort is logging to /var/log/snort, the following command line should do:
idstools-u2json -c /etc/snort/snort.conf
--directory /var/log/snort
--prefix unified2.log
--follow --bookmark
--output /var/log/snort/alerts.json
As the output is in the same format as Suricata’s you can refer to this guide for the Logstash setup.
One extra step I did was use Logstash to add an “engine” field to each entry. This can be accomplished by adapting the following Logstash configuration:
input {
file {
path => ["/var/log/suricata/eve.json"]
codec => json
type => "suricata-json"
}
file {
path => ["/var/log/snort/alerts.json"]
codec => json
type => "snort-json"
}
}
filter {
if [type] == "suricata-json" {
mutate {
add_field => {
"engine" => "suricata"
}
}
}
if [type] == "snort-json" {
mutate {
add_field => {
"engine" => "snort"
}
}
}
}
Checkout out the documentation for information.
[…] If you are not yet using Suricata, Snort can easily be used instead. For more info on sending Snort events to Elastic Search in “eve” format see my post Snort, Logstash, Elastic Search and Kibana… […]
hi, im running the idstools-u2json (installed thru pip) via:
idstools-u2json @/etc/snort/idstools-u2json.conf , my conf is:
cat /etc/snort/idstools-u2json.conf
–snort-conf=/etc/snort/snort.conf
–directory=/var/log/snort
–prefix=snort.log
–follow
–bookmark
–delete
–output=/var/log/snort/alerts.json
==== EOF ====
after starting is see:
Loaded 24831 rule message map entries.
Loaded 38 classifications.
Opened file /var/log/snort/snort.log.1432047824
but the alerts.json file is empty (its created, but size 0)
i can see the snort log growing, but the continous parsing is not doing much. Could You point me in the right direction
Looks like I need to cut a new release, as well as update the blog post.
idstools-u2json is now idstools-u2eve, u2json is more a basic script not providing eve compatibility anymore, but serves more as an example application. You can probably get going by updating via pip:
pip install –upgrade https://github.com/jasonish/py-idstools/archive/master.zip
(this is likely what will be tagged as the next release after some documentation cleanup).
Then call idstools-u2eve instead of idstools-u2json, using the same arguments. When running from the command line you can add –sdtout to see the json on stdout, as well as send it to the log file.
hi, thx for reply and sorry for confusion. The issue was on my snort.conf side. After i uncommeted the proper unified2 logtype, ur tool is working great. If theres anything to add to the parser – maybe a hardfail when a logfile is of different type than expected? Other than that – it working great! thanks
The behaviour is to silently wait for the log file to appear instead of failing out. Perhaps a warning message when it starts up with nothing to read – if the file exists, and its of a different, it will hard fail out.
Hello. I know this is an old post, but I’ve been trying to get suricata logs to an ELK server.
I am using filebeats on the suricata server to send the output to the ELK server.
But from everything I have seen, it looks like I need to install an instance of logstash on the actual suricata server to get the correct formatting to send to ES. Is that right? Or is that just easier?
I am seeing the output on the ELK server, but the detail is just combined in the message.
So far I have not found anything about setting up suricata and filebeat and just sending it off that way.
I have only used logstash-forwarder to send the eve events to another logstash instance directly, or have run logstash alongside Suricata where it can handle the log files directly. Both worked without issue.
I see that Filebeats has replaced logstash-forwarder, I have not tried it yet but will do so soon.
Hello. I want to define sensor-id by my own. Ex: ‘sensor-id’: 0 -> ‘sensor-id’: 1
It should pull the sensor ID out of the unified log. If you think it would be useful feature to have the script add/or change the field, please file an issue over at https://github.com/jasonish/py-idstools/issues