Dumpy – A Simple PCAP Spool File Frontend

Sometimes the best way to try out a new framework or language is to apply it to a domain you already know very well, even if it does happen to reinvent the wheel.  Tornado and Twitter Bootstrap are two such frameworks I’ve been meaning to play with for a while now. The result is Dumpy, a web front-end to pcap spool files as created by tcpdump, daemonlogger, or netsniff-ng with a very simple configuration and user interface:

Screen-2BShot-2B2012-12-18-2Bat-2B10.39.25-2BAM

Requirements are minimal, Python 2.6 (so it will run on CentOS 6 with little hassle), Tornado and py-bcrypt which are both trivially installed with pip. It provides its own http server with SSL support, and does not require a database.

Usage is also simple.  Simply enter a pcap filter, or paste in a Snort or Suricata event in “fast” format, choose  start and end times (or simply offsets) and hit download.

If interested, start a pcap spool (ie: sudo tcpdump -i eth0 -C 1000 -W10 -G 3600 -w /tmp/eth0.log.%Y%m%d.) then check out Dumpy over here https://github.com/jasonish/dumpy.

SpringMVC with Embedded Jetty and Thymeleaf

As mentioned in my previous post on using embedded Jetty with SpringMVC, I was going to look at simplifying the application by using Thymeleaf instead of JSPs as a view technology.

Well, it turns out that it is much more straightforward.

First we can remove the JSPC plugin from our pom.xml. Second we can completely remove the web.xml file.

Bootstrapping Jetty is much simpler now. We do not have to hook into the Jetty startup process with a lifecycle listener, instead we can directly create the dispatcher servlet and add it to a ServletContextHandler like we would any other servlet:

From JettyConfiguration.java:


@Bean
public ServletHolder dispatcherServlet() {
AnnotationConfigWebApplicationContext ctx =
new AnnotationConfigWebApplicationContext();
ctx.setParent(applicationContext);
ctx.register(MvcConfiguration.class);
DispatcherServlet servlet = new DispatcherServlet(ctx);
ServletHolder holder = new ServletHolder("dispatcher-servlet", servlet);
holder.setInitOrder(1);
return holder;
}

@Bean
public ServletContextHandler servletContext() throws IOException {
ServletContextHandler handler = new ServletContextHandler();
handler.setContextPath("/");
handler.setResourceBase(
new ClassPathResource("webapp").getURI().toString());
handler.addServlet(AdminServlet.class, "/metrics/*");
handler.addServlet(dispatcherServlet(), "/");
return handler;
}

@Bean(initMethod = "start", destroyMethod = "stop")
public Server jettyServer() throws IOException {
Server server = new Server();
server.setHandler(servletContext());
server.setConnectors(jettyConnectors());
return server;
}

This is much more straightforward, everything wired up with Spring without any lifecycle callback hooks.

The setup for Spring to render the Thymeleaf views, is a little more complex, but not fussy. What we do is replace:


@Bean
public InternalResourceViewResolver configureInternalResourceViewResolver() {
InternalResourceViewResolver resolver =
new InternalResourceViewResolver();
resolver.setPrefix("/WEB-INF/views/");
resolver.setSuffix(".jsp");
return resolver;
}

with


@Bean
public ServletContextTemplateResolver thymeleafTemplateResolver() {
ServletContextTemplateResolver resolver =
new ServletContextTemplateResolver();
resolver.setPrefix("/WEB-INF/thymeleaf/");
resolver.setSuffix(".html");
resolver.setTemplateMode("HTML5");
resolver.setCacheable(true);
return resolver;
}

@Bean
public SpringTemplateEngine thymeleafTemplateEngine() {
SpringTemplateEngine engine = new SpringTemplateEngine();
engine.setTemplateResolver(thymeleafTemplateResolver());
return engine;
}

@Bean
public ThymeleafViewResolver thymeleafViewResolver() {
ThymeleafViewResolver resolver = new ThymeleafViewResolver();
resolver.setTemplateEngine(thymeleafTemplateEngine());
return resolver;
}

in MvcConfiguration.java.

Our templates now look a lot more like plain HTML, and will render better when loaded directly in a browser:


<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org">
<head>
<title>Home</title>
</head>
<body>
<h1>Hello World!</h1>

<p>Server time: <span th:text="${serverTime}"></span></p>

<p>Here are some items:</p>
<ul>
<li th:each="item : ${someItems}" th:text="${item}"></li>
</ul>

<p>Do we have a message from the dummy service:</p>

<div th:if="${dummyService == null}">
<p>No, dummy service is null.</p>
</div>
<div th:if="${dummyService != null}">
<p>Yes: <span th:text="${dummyService.getMessage()}"></span></p>
</div>

<p><a href="resources/static.txt">A static file.</a></p>

<p><a href="metrics">Yammer Metrics</a></p>

</body>
</html>

With the build process being simple (no dependence on maven plugins) we can switch to a much less verbose Gradle build file:


apply plugin:'java'
apply plugin:'application'

version = '0.0.1-SNAPSHOT'

mainClassName = "ca.unx.template.Main"
applicationName = "jetty-springmvc-thymeleaf-template"

repositories {
mavenCentral()
}

dependencies {
compile("org.springframework:spring-webmvc:3.1.3.RELEASE") {
// Commons-logging excluded in favour of SLF4j.
exclude module: 'commons-logging'
}

compile "cglib:cglib:2.2.2"
compile "org.thymeleaf:thymeleaf-spring3:2.0.14"
compile "org.eclipse.jetty:jetty-webapp:8.1.8.v20121106"
compile "com.yammer.metrics:metrics-servlet:2.2.0"

/* Logging. */
def slf4jVersion = "1.7.1"
compile "ch.qos.logback:logback-classic:1.0.9"
compile "org.slf4j:slf4j-api:$slf4jVersion"
compile "org.slf4j:jcl-over-slf4j:$slf4jVersion"
compile "org.codehaus.groovy:groovy:1.8.6"
}

A complete template project can be found here. Note that this links to tag which is the state of the code at the time of this writing.

SpringMVC with Embedded Jetty

As an accidental Java developer I’ve never been comfortable deploying applications into a container, especially when the web interface is secondary to the primary purpose of the application. Instead I prefer to programatically create and manage the web interface rather than have it manage me. Currently SpringMVC is my Java web framework of choice (due to company convention more than anything else) and it is built around the idea of being managed by a container such as Jetty or Tomcat.

While there is no shortage of existing posts on using SpringMVC with embedded Jetty, they fail for me due to the following reasons:

  • They embed Jetty just enough to bootstrap something that looks like a classic Java web application – not what I want!
  • They don’t address JSPs – the default view technology used by SpringMVC.

Compile Those JSPs

By compiling the JSPs prior to deployment we can save ourselves from the hassle of classpath issues, especially relating to tag libs.


<plugin>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-jspc-maven-plugin</artifactId>
<version>${jetty.version}</version>
<executions>
<execution>
<phase>compile</phase>
<id>jspc</id>
<goals>
<goal>jspc</goal>
</goals>
<configuration>
<webAppSourceDirectory>
${basedir}/src/main/resources/webapp
</webAppSourceDirectory>
<webXml>
${basedir}/src/main/resources/webapp/WEB-INF/web.xml
</webXml>
<webXmlFragment>
${basedir}/target/classes/webapp/WEB-INF/web.xml-frag
</webXmlFragment>
</configuration>
</execution>
</executions>
</plugin>

This plugin will compile all the JSPs found under src/main/resources/webapp and modify the existing web.xml to direct requests for the JSPs to the pre-compiled versions. This avoids compiling them at runtime and speeds up the time it takes to respond to the first request for a JSP.

You may also notice that the JSPs are under src/main/resources/webapp instead of the more common src/main/webapp. This is to avoid extra lines in the pom.xml to pull in webapp as a resource, as we are packaging as a jar.

Bootstrapping Jetty

I use Spring annotations to configure Jetty and import its @Configuration class into my root context configuration class. In order for the SpringMVC dispatcher servlet to access beans created in the root context, this class needs to be application context aware:


@Configuration
public class JettyConfiguration implements ApplicationContextAware {
private ApplicationContext applicationContext;

@Override
public void setApplicationContext(ApplicationContext applicationContext)
throws BeansException {
this.applicationContext = applicationContext;
}

this is used as the parent context for the new context that will be created for the SpringMVC dispatcher servlet:


@Bean
public ServletHolder dispatcherServlet() {
AnnotationConfigWebApplicationContext ctx =
new AnnotationConfigWebApplicationContext();
ctx.setParent(applicationContext);
ctx.register(MvcConfiguration.class);
DispatcherServlet servlet = new DispatcherServlet(ctx);
ServletHolder holder = new ServletHolder("dispatcher-servlet",
servlet);
holder.setInitOrder(1);
return holder;
}

There are still a few more tricks required to get the dispatcher servlet registered without defining it in your web.xml. We must also create a Jetty web application context, this creates the JSP servlet as well as a default servlet as would be created during the instantiation of a classic war based application:


@Bean
public WebAppContext jettyWebAppContext() throws IOException {
WebAppContext ctx = new WebAppContext();
ctx.setContextPath("/");
ctx.setWar(new ClassPathResource("webapp").getURI().toString());

/* We can add the Metrics servlet right away. */
ctx.addServlet(AdminServlet.class, "/metrics/*");

return ctx;
}

Notice that I’ve added the metrics servlet, but have yet to add the dispatcher servlet. For some reason, adding the dispatcher servlet here causes JSP views to fail. One option is to fall back to defining the dispatcher servlet in your web.xml (which may be the cleanest option), or you can register the dispatcher servlet in the Jetty lifeCycleStarted callback which we’ll do here:


@Bean
public LifeCycle.Listener lifeCycleStartedListener() {
return new AbstractLifeCycle.AbstractLifeCycleListener() {
@Override
public void lifeCycleStarted(LifeCycle event) {
try {
ServletHolder dispatcherServlet = dispatcherServlet();
jettyWebAppContext().getServletHandler()
.addServletWithMapping(dispatcherServlet, "/");
dispatcherServlet.start();
} catch (Exception e) {
logger.error(
"Failed to start Spring MVC dispatcher servlet", e);
}
}
};
}

@Bean(initMethod = "start", destroyMethod = "stop")
public Server jettyServer() throws IOException {
Server server = new Server();
server.setHandler(jettyWebAppContext());

/* Add a life cycle listener so we can register the SpringMVC dispatcher
* servlet after the web application context has been started. */
server.addLifeCycleListener(lifeCycleStartedListener());

server.setConnectors(jettyConnectors());
return server;
}

In summary, this is just another variation of James Ward’s post on Containerless Spring MVC. The complete code for this project (which I use as a template) can be at https://github.com/jasonish/jetty-springmvc-jsp-template. The code as referenced in this post can be found in the 20121207 tag.

Next – As I’m not really a fan of JSPs, or the extra hoops required to make this work right I’ll probably look at updating the template with Thymeleaf support, which should greatly simplify things.