HILTI and Spicy

HILTI. A high-level intermediary language for traffic inspection.

Spicy. A next-generation parser generator for network protocols and file formats.

Note: This version of HILTI/Spicy is no longer maintained and kept here only for reference.

There is a new implementation on GitHub.

Download and Installation

The source code for both HILTI and Spicy is hosted at git://git.icir.org/hilti. There's is also a mirror on github. Follow the installation instructions to get it up and running.

We do not offer any further archives or packages for download right now. However, there's a Docker image that comes with everything preinstalled.

Documentation

There is the start of a manual. It's very preliminary at this point, and very incomplete.

For more background on HILTI, see the IMC paper.

For more background on Spicy, see the ACSAC paper.

For a demo of Spicy, watch the presentation at BroCon 2014.

Mailing List & Contact

For questions and discussion of HILTI and Spicy, join our mailing list. To report problems, please use the github tracker. To submit patches, file a github pull request. If you want to contact the development team directly, email us.

Frequently Asked Questions

What's the current implementation status?: HILTI and Spicy are currently in prototype state; they aren't production-ready yet. While they are functional and provide most of the initially targeted functionality at this point, that doesn't mean they are necessarily also stable, robust, efficient, portable, user-friendly, or documented.
Well, so when do you expect this all to be production-ready then?: There's quite a bit still left to do, and it will require more resources to make that happen. There's a Wiki page tracking the TODO list. Let us know if you want to help.
Which platforms are supported?: We develop HILTI and Spicy on 64-bit Linux and MacOS; other platforms are unlikely to work at the moment.
Can I use Spicy with Bro?: Yes, we have developed a Bro plugin that integrates Spicy transparently into Bro.
Oh, so is this all tied to Bro?: No, not at all. While coming out of the Bro Project, HILTI and Spicy are standalone systems designed with generic C APIs suitable for integration into a wide range of potentially quite different applications.
Alright, what's the license?: HILTI and Spicy are open-source and come with a BSD license, which allows for pretty much unrestricted use as long as you leave the license header in place.
I hear Spicy, I hear BinPAC++, that's confusing!?: BinPAC++ used to be the working title of the parser generator. We have since renamed the project to Spicy, as it's not actually related to the original 2006 BinPAC system at all—it's aiming at the same problem space but taking a very different approach.
Who's behind all this?: HILTI and Spicy are developed by the Networking and Security Group at the International Computer Science Institute, a non-profit research institute affiliated with the University of California, Berkeley. Development is funded in part by the National Science Foundation.

About HILTI

When developing networking systems such as firewalls, routers, and intrusion detection systems, one faces a striking gap between the ease with which one can often describe a desired analysis in high-level terms, and the tremendous amount of low-level implementation details that one must still grapple with to come to a robust solution. HILTI bridges this divide by providing an abstract execution environment for deep, stateful network traffic analysis. It offers platform to application developers that provides much of the low-level functionality, without tying it to a specific analysis structure.

HILTI consists of two parts: (1) an abstract machine model that caters specifically to the networking domain, directly supporting the field's common abstractions and idioms in its instruction set; and (2) a compiler toolchain, built on top of LLVM, for turning programs written for the abstract machine into optimized, natively executable code.

# cat hello-world.hlt
module Main

import Hilti

void run() {
  call Hilti::print ("Hello, HILTI world!")
}

# hiltic -j hello-world.hlt
Hello, HILTI world!

About Spicy

Spicy is a next-generation parser generator that makes it easy to build parsers for network protocols, file formats, and more. Spicy is more than just a "yacc for protocols": it's an all-in-one system that enables developers to write attributed grammars defining both syntax and semantics of an input format inside a single comprehensive scripting language.

The Spicy toolchain, built on top of HILTI, turns such grammars into efficient parsing code that exposes an well-defined C interface to its host application for feeding in input and retrieving results. At runtime, parsing proceeds fully incrementally—and potentially in parallel—on input streams of arbitrary size. Compilation takes place either statically at build time, or or just-in-time at startup.

# cat http-request.spicy
module HTTP;

const Token      = /[^ \t\r\n]+/;
const WhiteSpace = /[ \t]+/;
const NewLine    = /\r?\n/;

export type RequestLine = unit {
  method:  Token;
  :        WhiteSpace;
  uri:     Token;
  :        WhiteSpace;
  version: Version;
  :        NewLine;

  on %done {
    print self.method, self.uri, self.version.number;
    }
};

type Version = unit {
  :       /HTTP\//;
  number: /[0-9]+\.[0-9]+/;
};

# echo "GET /index.html HTTP/1.0" | spicy-driver http-request.spicy
GET /index.html 1.0