2.3. Bro Interface¶
Contents
Spicy comes with a Bro plugin that adds support for *.spicy
grammars to Bro. The following overview walks through installation and
usage. Keep in mind that everything is in development and not
necessarily stable right now.
Note
The Docker Image comes with both Bro itself and the plugin preinstalled, so you can skip the installation process when using that.
2.3.1. Installation¶
You currently need the current development version of Bro, as that adds support for dynamically loaded plugins. To get it, use git:
> git clone --recursive git://git.bro.org/bro
Now you need to build Bro with the same C++ compiler (and C++ standard
library) that you also use for compiling HILTI/Spicy. If
that’s, say, /opt/llvm/bin/clang++
, do:
> CXX="/opt/llvm/bin/clang++ --stdlib=libc++" ./configure && make
Then compile HILTI/Spicy, pointing it to your Bro clone:
> make BRO_DIST=/path/to/where/you/cloned/bro/into
Watch the output, it should indicate that it has found Bro.
If things went well, the Bro tests should now succeed:
> cd bro/tests && make
[...]
all XXX tests successful
Cool.
Note
The Bro plugin has no make install
for now, one needs to run
it right out of the build directory. See below for how to do that.
2.3.2. Usage¶
Writing a Spicy analyzer for Bro consists of two steps: writing the
analyzer as standard *.spicy
code; and defining the analyzer’s
interface to Bro, including the events it will generate, in a separate
*.evt
file. We look at the two separately in the following, using
a simple analyzer for parsing SSH banners as our running example. We
then show to run Bro so that it loads the Spicy plugin along with
any analyzers.
2.3.2.1. Spicy Grammar¶
Here’s a simple grammar ssh.spicy
to parse an SSH banner (e.g.,
SSH-2.0-OpenSSH_6.1
):
module SSH;
export type Banner = unit {
magic : /SSH-/;
version : /[^-]*/;
dash : /-/;
software: /[^\r\n]*/;
};
This is indeed just a standard Spicy grammar, just as it can be
used with spicy-driver [Missing]. It could include further hooks and
also global code; the latter will execute once at Bro initialization
time. If you add print
statements, they will generate the
corresponding output as Bro processes input, which can be helpful for
debugging.
2.3.2.2. Protocol Analyzer Interface¶
Next, we define a file ssh.evt
that refers to the grammar to
define a new Bro analyzer. An *.evt
file has three parts: (1) an
grammar
specification telling Bro which *.spicy
file to use; (2)
an analyzer
definition describing where to hook it into Bro’s
traffic processing; and (3) a series of event definitions specifying
how to turn what’s parsed into the Bro events. Here’s an ssh.evt
to go with our SSH grammar:
grammar ssh.spicy;
protocol analyzer spicy::SSH over TCP:
parse with SSH::Banner,
port 22/tcp,
replaces SSH;
on SSH::Banner -> event ssh::banner($conn, $is_orig, self.version, self.software);
The grammar
line specifies a *.spicy
file to load. The path is
relative to the location of the *.evt
file; if it’s not found
there, the plugin further searches for grammars inside the source tree
at hilti2/bro/spicy/
, and in any directories specified through the
environment variable BRO_SPICY_PATH
(using the standard syntax of
colon-separated paths). The grammar
line is optional, one can
leave it out if the *.spicy
grammars get loaded otherwise (e.g.,
via the command line; see below). One can also specify multiple
grammar
entries.
Note
Technically, the Bro plugin searches the *.spicy
files inside
the build directory at <build>/spicy
because that will later
correspond to the plugin’s top-level installation directory
(installing a plugin essentially means copying the build directory
somewhere else). The current build system setup however links that
back to the source directory hilti2/bro/spicy/
so that one can
edit files there and they will be pulled in directly.
The analyzer
block starts with giving the new analyzer a name
(spicy::SSH
). The namespace isn’t mandatory, but we use it to
differentiate Spicy analyzers from Bro’s standard analyzers. The
name corresponds to how we’ll refer to the new analyzer in Bro. For
example, just like there’s an analyzer::ANALYZER_SSH
enum for the
standard SSH analyzer, there’ll now be an
analyzer::ANALYZER_SPICY_SSH
. (As you can see there’s a bit of name
normalization going on, use bro -NN
to see the final name.).
over TCP
declares this to be an TCP application-layer analyzer
(over UDP
is the one other supported alternative right now).
Next comes set of of properties for the analyzer, separated by commas (order is not important):
parse with <unit>
links the new analyzer with the Spicy grammar:<unit>
is the fully-qualified name of the unit we want to use as the top-level entry point for parsing. When specified the way we do here, the unit will be used for both originator- and responder-side traffic. One can use different units for each side by specifyingparse originator with <orig-unit>
andparse responder with <resp-unit>
, respectively.port <port>
defines the well-known port for the analyzer; this translates into registering the port with Bro’s analyzer framework at runtime. This is optional; as usual analyzers can also be activated via DPD signatures or even manually from script-land. One can give multipleport
specifications (but separately).replaces <analyzer>
tells Bro that when this analyzer is activateed, it’s replacing one of the built-in analyzers, given by its name (see againbro -NN
for the names of all built-in analyzers). This has two effects: First, the built-in one will be completely disabled at startup, and second in script-land Bro will use the name of the original analyzer in place of this one (e.g., so that inconn.log
the service will show up asSSH
, notSPICY_SSH
).
Events are defined by lines of the form <hook> -> event
<name>(<parameters>)
. Let’s break it down:
<hook>
defines when an event should be triggered, and it works just like externally defined Spicy hooks, i.e., you give the fully-qualified path to the grammar element that is to trigger the event during parsing. In the example above,on SSH::Banner
will trigger an event whenever an instance of theSSH::Banner
unit has been fully parsed. One can also refer to individual unit fields and variables (e.g.,on SSH::Banner::version
). The unit-wide unit hooks work as well (e.g.,on SSH::Banner::%init
).
<name>
corresponds directly to the name of the event as visible in Bro. As you can see, one can (and should) use namespaces.The
<parameters>
define the event’s parameters, specifying both their values and (implicitly) their types at the same time. Each parameter is a complete Spicy expression (except for some “magic” ones starting with$
, see below). The type of each Spicy expression determines the Bro type of the corresponding event parameter, with an internal mapping defining how each Spicy type translates into a Bro type (e.g., abytes
objects translates into a Brostring
; see Type Mapping for the details). At runtime, when an event is raised, the expression is evaluated to determine the value passed to event’s handlers. That evaluation takes place in a hook context corresponding to what<hook>
triggers on, and it has access to the same scope as a manually written hook would have. For example, with theon SSH::Banner
event,self
refers to aSSH::Banner
unit instance andself.version
to itsversion
item. If, for example, you wanted an upper-case version of the software identification, you could useself.software.upper()
.In addition to expression parameters, there are three “magic” parameters that provide access to internal Bro state:
$conn
references the current connection that’s being parsed, and it translates into aconnection
record parameter for the corresponding Bro event.$is_orig
turns into a boolean value indicating whether Spicy is parsing the originator or responder side of the connection.$file
references the current file in case of defining a file analyzer rather than a protocol analyzer (see below).Note that the magic parameters aren’t expressions; you can’t further manipulate them.
2.3.2.3. Using a Spicy Analyzer¶
With both the grammar and the analyzer/event definition in place, we can now write an event handler:
event ssh::banner(c: connection, is_orig: bool, version: string, software: string)
{
print "SSH banner", c$id, is_orig, version, software, Hilti::is_compiled();
}
Note how the ssh::banner
definition from ssh.evt
maps into the
event’s parameter signature.
Before we can use it, we need to tell Bro where to find the Spicy
plugin. For that, we set the environment variable BRO_PLUGIN_PATH
to
the plugin’s build directory:
export BRO_PLUGIN_PATH=/path/to/hilti/build/bro
Note
If you get the path to the plugin wrong, or forget to set it at
all, you’ll get some misleading error messages below as Bro will
try to parse ssh.evt
file as a Bro script.
Now we can run Bro with the new analyzer by giving it the ssh.evt
on the command line. If not an absolute path, Bro will search for the
the file first in the current directory first, and then—just as
described above for *.spicy
files—in hilti2/bro/spicy/
and any
directories specified by BRO_SPICY_PATH
.
Let’s first just check that Bro indeeds loads the analyzer correctly.
bro -NN
will tell us:
# bro -NN ssh.evt
[...]
Plugin: Bro::Hilti - Dynamically compiled HILTI/Spicy functionality (dynamic, version 1)
[Analyzer] spicy_SSH (ANALYZER_SPICY_SSH, enabled)
[Event] ssh::banner
[...]
[...]
Todo
The -NN
output currently does not include all the information
shown above. Need to fix.
We see that Bro has found the Spicy plugin. The plugin indeed
provides our analyzer spicy_SSH
, which generates one event
ssh_banner
.
Let’s now try processing a trace containing a single SSH connection,
making sure to give Bro our event handler ssh-banner.bro
as well:
# bro -r ssh-single-conn.trace ssh.evt ssh-banner.bro
SSH banner, [orig_h=192.150.186.169, orig_p=49244/tcp, resp_h=131.159.14.23, resp_p=22/tcp], F, 1.99, OpenSSH_3.9p1
SSH banner, [orig_h=192.150.186.169, orig_p=49244/tcp, resp_h=131.159.14.23, resp_p=22/tcp], T, 2.0, OpenSSH_3.8.1p1
2.3.2.4. File Analyzers¶
Writing a Spicy file analyzer works pretty similar to a protocol
analyzer. The main difference is indeed just that we need to tell Bro
to use our Spicy unit for parsing files rather than protocols. We
do that via the *.evt
file by defining a file analyzer
section
instead of a protocol analyzer
. Here’s an example for a GZIP
decoder:
grammar gzip.spicy;
file analyzer spicy::GZIP:
parse with gzip::File,
mime-type application/x-gzip;
on gzip::Member -> event gzip::member($file, self.method, self.flags, self.mtime, self.xflags, self.os);
The parse with
works the same as with a protocol: it specifies the
top-level Spicy unit to use. Instead of giving a well-known port to
activate the analyzer automatically (as we do with protocols), we can
associate a file analyzer with a MIME type. Now whenever some part of
Bro finds data of that type, it will deploy our Spicy analyzer to
take it apart.
Here’s the main part of the corresponding grammar file gzip.spicy
(we skip the definition of the OS
enum for brevity):
export type File = unit {
member: Member;
};
type Member = unit {
%byteorder = Spicy::ByteOrder::Little;
magic : b"\x1f\x8b";
method : uint<8>;
flags : bitfield(8) {
ftext: 0;
fhrcr: 1;
fextra: 2;
fname: 3;
fcomment: 4;
};
mtime : uint<32> &convert=cast<time>($$);
xflags : uint<8>;
os : uint<8> &convert=OS($$);
xlen : uint<16> if ( self.flags.fextra != 0 );
xdata : bytes &length=self.xlen if ( self.flags.fextra != 0 );
name : bytes &until=b"\x00" if ( self.flags.fname != 0 );
comment: bytes &until=b"\x00" if ( self.flags.fcomment != 0 );
crc16 : uint<16> if ( self.flags.fhrcr != 0 );
};
For testing let’s also provide an event handler in gzip-header.bro
:
type Flags: record {
ftext: count;
fhrcr: count;
fextra: count;
With that all in place, we can run Bro like this on a trace with a GZIP file transferred via HTTP:
# bro -r gzip-single-request.trace gzip.evt gzip-header.bro
7aNwGytV9k4, 8, 0, 1380302739.000000, 0, gzip::UNIX
2.3.2.5. HILTI/Spicy Options¶
Bro’s Spicy plugin comes with a number of options defined in
hilti2/bro/scripts/bro/hilti/base/main.bro
. These include:
debug: bool
(default: false)- If true, compiles all Spicy/HILTI code in debug mode, meaning
that Debugging Support will be available (including the
HILTI_DEBUG
environment variable). dump_debug: bool
(default: false)- Dumps debug information about the compiled analyzers to the console. Can be helpful if Bro isn’t integrating a Spicy analyzer as expected.
optimize: bool
(default: true)- Activates code optimization. Should be on normally, but makes debugging in the debugger easier if off.
use_cache: bool
(default: true)
Todo
This is currently broken and disabled.
Enables caching of compiled Spicy/HILTI code. If on, you will notice that the first time you start Bro with a new (or modified) analyzer, it takes longer than on subsequent invocations. That’s because the first one needs to do all the leg-work of going from Spicy to native code. It then however caches the resulting code on disk and reuses it next time directly.
Normally you want this on, however currently the cache does not always pick up on changes made to
*.spicy
or*.evt
files (or Bro itself, or the Spicy plugin). If you notice that a change doesn’t seem to take effect, try removing the cache directory (.cache
) or simply disable it altogether.
See the script itself for the complete list of all options.
2.3.2.6. Type Mapping¶
The following table summarizes how Spicy types get mapped into Bro types. Types not listed are not yet implemented.
Spicy | Bro |
---|---|
addr |
addr |
bool |
bool |
bytes |
string |
double |
double |
enum |
enum [1] |
int<*> |
int |
string |
string |
time |
time |
uint<*> |
count |
tuple<*> |
record [2] |
list<*> |
vector |
set<*> |
set |
vector<*> |
vector |
[1] A corresponding Bro enum
type is created automatically.
[2] The record fields will by default be named f<n>
with n being
the zero-baed element index. However, if the tuple gets passed into a
Bro event, the initially created record will be coered into the type
the event expects and hence then use the corresponding field names
instead.