.. _spicy_bro-plugin: ============= Bro Interface ============= .. contents:: Spicy comes with a Bro plugin that adds support for ``*.spicy`` grammars to Bro. The following overview walks through installation and usage. Keep in mind that everything is in development and not necessarily stable right now. .. note:: The :ref:`docker` comes with both Bro itself and the plugin preinstalled, so you can skip the installation process when using that. Installation ============ You currently need the current development version of Bro, as that adds support for dynamically loaded plugins. To get it, use git:: > git clone --recursive git://git.bro.org/bro Now you need to build Bro with the same C++ compiler (and C++ standard library) that you also use for compiling HILTI/Spicy. If that's, say, ``/opt/llvm/bin/clang++``, do:: > CXX="/opt/llvm/bin/clang++ --stdlib=libc++" ./configure && make Then compile HILTI/Spicy, pointing it to your Bro clone:: > make BRO_DIST=/path/to/where/you/cloned/bro/into Watch the output, it should indicate that it has found Bro. If things went well, the Bro tests should now succeed:: > cd bro/tests && make [...] all XXX tests successful Cool. .. note:: The Bro plugin has no ``make install`` for now, one needs to run it right out of the build directory. See below for how to do that. Usage ===== Writing a Spicy analyzer for Bro consists of two steps: writing the analyzer as standard ``*.spicy`` code; and defining the analyzer's interface to Bro, including the events it will generate, in a separate ``*.evt`` file. We look at the two separately in the following, using a simple analyzer for parsing SSH banners as our running example. We then show to run Bro so that it loads the Spicy plugin along with any analyzers. Spicy Grammar ------------- Here's a simple grammar ``ssh.spicy`` to parse an SSH banner (e.g., ``SSH-2.0-OpenSSH_6.1``): .. literalinclude:: /../bro/spicy/ssh.spicy This is indeed just a standard Spicy grammar, just as it can be used with :ref:`spicy_spicy-driver`. It could include further hooks and also global code; the latter will execute once at Bro initialization time. If you add ``print`` statements, they will generate the corresponding output as Bro processes input, which can be helpful for debugging. Protocol Analyzer Interface --------------------------- Next, we define a file ``ssh.evt`` that refers to the grammar to define a new Bro analyzer. An ``*.evt`` file has three parts: (1) an ``grammar`` specification telling Bro which ``*.spicy`` file to use; (2) an ``analyzer`` definition describing where to hook it into Bro's traffic processing; and (3) a series of event definitions specifying how to turn what's parsed into the Bro events. Here's an ``ssh.evt`` to go with our SSH grammar: .. literalinclude:: /../bro/spicy/ssh.evt The ``grammar`` line specifies a ``*.spicy`` file to load. The path is relative to the location of the ``*.evt`` file; if it's not found there, the plugin further searches for grammars inside the source tree at ``hilti2/bro/spicy/``, and in any directories specified through the environment variable ``BRO_SPICY_PATH`` (using the standard syntax of colon-separated paths). The ``grammar`` line is optional, one can leave it out if the ``*.spicy`` grammars get loaded otherwise (e.g., via the command line; see below). One can also specify multiple ``grammar`` entries. .. note:: Technically, the Bro plugin searches the ``*.spicy`` files inside the *build* directory at ``/spicy`` because that will later correspond to the plugin's top-level installation directory (installing a plugin essentially means copying the build directory somewhere else). The current build system setup however links that back to the source directory ``hilti2/bro/spicy/`` so that one can edit files there and they will be pulled in directly. The ``analyzer`` block starts with giving the new analyzer a name (``spicy::SSH``). The namespace isn't mandatory, but we use it to differentiate Spicy analyzers from Bro's standard analyzers. The name corresponds to how we'll refer to the new analyzer in Bro. For example, just like there's an ``analyzer::ANALYZER_SSH`` enum for the standard SSH analyzer, there'll now be an ``analyzer::ANALYZER_SPICY_SSH``. (As you can see there's a bit of name normalization going on, use ``bro -NN`` to see the final name.). ``over TCP`` declares this to be an TCP application-layer analyzer (``over UDP`` is the one other supported alternative right now). Next comes set of of properties for the analyzer, separated by commas (order is not important): - ``parse with `` links the new analyzer with the Spicy grammar: ```` is the fully-qualified name of the unit we want to use as the top-level entry point for parsing. When specified the way we do here, the unit will be used for both originator- and responder-side traffic. One can use different units for each side by specifying ``parse originator with `` and ``parse responder with ``, respectively. - ``port `` defines the well-known port for the analyzer; this translates into registering the port with Bro's analyzer framework at runtime. This is optional; as usual analyzers can also be activated via DPD signatures or even manually from script-land. One can give multiple ``port`` specifications (but separately). - ``replaces `` tells Bro that when this analyzer is activateed, it's replacing one of the built-in analyzers, given by its name (see again ``bro -NN`` for the names of all built-in analyzers). This has two effects: First, the built-in one will be completely disabled at startup, and second in script-land Bro will use the name of the original analyzer in place of this one (e.g., so that in ``conn.log`` the service will show up as ``SSH``, not ``SPICY_SSH``). Events are defined by lines of the form `` -> event ()``. Let's break it down: - ```` defines when an event should be triggered, and it works just like externally defined Spicy :ref:`hooks `, i.e., you give the fully-qualified path to the grammar element that is to trigger the event during parsing. In the example above, ``on SSH::Banner`` will trigger an event whenever an instance of the ``SSH::Banner`` unit has been fully parsed. One can also refer to individual unit fields and variables (e.g., ``on SSH::Banner::version``). The unit-wide unit hooks work as well (e.g., ``on SSH::Banner::%init``). - ```` corresponds directly to the name of the event as visible in Bro. As you can see, one can (and should) use namespaces. - The ```` define the event's parameters, specifying both their values and (implicitly) their types at the same time. Each parameter is a complete Spicy expression (except for some "magic" ones starting with ``$``, see below). The type of each Spicy expression determines the Bro type of the corresponding event parameter, with an internal mapping defining how each Spicy type translates into a Bro type (e.g., a ``bytes`` objects translates into a Bro ``string``; see :ref:`spicy_bro-type-mapping` for the details). At runtime, when an event is raised, the expression is evaluated to determine the value passed to event's handlers. That evaluation takes place in a hook context corresponding to what ```` triggers on, and it has access to the same scope as a manually written hook would have. For example, with the ``on SSH::Banner`` event, ``self`` refers to a ``SSH::Banner`` unit instance and ``self.version`` to its ``version`` item. If, for example, you wanted an upper-case version of the software identification, you could use ``self.software.upper()``. In addition to expression parameters, there are three "magic" parameters that provide access to internal Bro state: - ``$conn`` references the current connection that's being parsed, and it translates into a ``connection`` record parameter for the corresponding Bro event. - ``$is_orig`` turns into a boolean value indicating whether Spicy is parsing the originator or responder side of the connection. - ``$file`` references the current file in case of defining a file analyzer rather than a protocol analyzer (see below). Note that the magic parameters aren't expressions; you can't further manipulate them. Using a Spicy Analyzer ---------------------- With both the grammar and the analyzer/event definition in place, we can now write an event handler: .. literalinclude:: /../bro/tests/spicy/ssh-banner.bro :lines: 6-9 Note how the ``ssh::banner`` definition from ``ssh.evt`` maps into the event's parameter signature. Before we can use it, we need to tell Bro where to find the Spicy plugin. For that, we set the environment variable ``BRO_PLUGIN_PATH`` to the plugin's build directory:: export BRO_PLUGIN_PATH=/path/to/hilti/build/bro .. note:: If you get the path to the plugin wrong, or forget to set it at all, you'll get some misleading error messages below as Bro will try to parse ``ssh.evt`` file as a Bro script. Now we can run Bro with the new analyzer by giving it the ``ssh.evt`` on the command line. If not an absolute path, Bro will search for the the file first in the current directory first, and then---just as described above for ``*.spicy`` files---in ``hilti2/bro/spicy/`` and any directories specified by ``BRO_SPICY_PATH``. Let's first just check that Bro indeeds loads the analyzer correctly. ``bro -NN`` will tell us:: # bro -NN ssh.evt [...] Plugin: Bro::Hilti - Dynamically compiled HILTI/Spicy functionality (dynamic, version 1) [Analyzer] spicy_SSH (ANALYZER_SPICY_SSH, enabled) [Event] ssh::banner [...] [...] .. todo:: The ``-NN`` output currently does not include all the information shown above. Need to fix. We see that Bro has found the Spicy plugin. The plugin indeed provides our analyzer ``spicy_SSH``, which generates one event ``ssh_banner``. Let's now try processing a trace containing a single SSH connection, making sure to give Bro our event handler ``ssh-banner.bro`` as well:: # bro -r ssh-single-conn.trace ssh.evt ssh-banner.bro SSH banner, [orig_h=192.150.186.169, orig_p=49244/tcp, resp_h=131.159.14.23, resp_p=22/tcp], F, 1.99, OpenSSH_3.9p1 SSH banner, [orig_h=192.150.186.169, orig_p=49244/tcp, resp_h=131.159.14.23, resp_p=22/tcp], T, 2.0, OpenSSH_3.8.1p1 File Analyzers -------------- Writing a Spicy file analyzer works pretty similar to a protocol analyzer. The main difference is indeed just that we need to tell Bro to use our Spicy unit for parsing files rather than protocols. We do that via the ``*.evt`` file by defining a ``file analyzer`` section instead of a ``protocol analyzer``. Here's an example for a GZIP decoder: .. literalinclude:: /../bro/spicy/gzip.evt The ``parse with`` works the same as with a protocol: it specifies the top-level Spicy unit to use. Instead of giving a well-known port to activate the analyzer automatically (as we do with protocols), we can associate a file analyzer with a MIME type. Now whenever some part of Bro finds data of that type, it will deploy our Spicy analyzer to take it apart. Here's the main part of the corresponding grammar file ``gzip.spicy`` (we skip the definition of the ``OS`` enum for brevity): .. literalinclude:: /../bro/spicy/gzip.spicy :lines: 6-32 For testing let's also provide an event handler in ``gzip-header.bro``: .. literalinclude:: /../bro/tests/spicy/gzip-header.bro :lines: 6-9 With that all in place, we can run Bro like this on a trace with a GZIP file transferred via HTTP:: # bro -r gzip-single-request.trace gzip.evt gzip-header.bro 7aNwGytV9k4, 8, 0, 1380302739.000000, 0, gzip::UNIX HILTI/Spicy Options ------------------- Bro's Spicy plugin comes with a number of options defined in ``hilti2/bro/scripts/bro/hilti/base/main.bro``. These include: ``debug: bool`` (default: false) If true, compiles all Spicy/HILTI code in debug mode, meaning that :ref:`spicy_debugging` will be available (including the ``HILTI_DEBUG`` environment variable). ``dump_debug: bool`` (default: false) Dumps debug information about the compiled analyzers to the console. Can be helpful if Bro isn't integrating a Spicy analyzer as expected. ``optimize: bool`` (default: true) Activates code optimization. Should be on normally, but makes debugging in the debugger easier if off. ``use_cache: bool`` (default: true) .. todo:: This is currently broken and disabled. Enables caching of compiled Spicy/HILTI code. If on, you will notice that the first time you start Bro with a new (or modified) analyzer, it takes longer than on subsequent invocations. That's because the first one needs to do all the leg-work of going from Spicy to native code. It then however caches the resulting code on disk and reuses it next time directly. Normally you want this on, however currently the cache does not always pick up on changes made to ``*.spicy`` or ``*.evt`` files (or Bro itself, or the Spicy plugin). If you notice that a change doesn't seem to take effect, try removing the cache directory (``.cache``) or simply disable it altogether. See the script itself for the complete list of all options. .. _spicy_bro-type-mapping: Type Mapping ------------ The following table summarizes how Spicy types get mapped into Bro types. Types not listed are not yet implemented. ============= ============== Spicy Bro ============= ============== ``addr`` ``addr`` ``bool`` ``bool`` ``bytes`` ``string`` ``double`` ``double`` ``enum`` ``enum`` [1] ``int<*>`` ``int`` ``string`` ``string`` ``time`` ``time`` ``uint<*>`` ``count`` ``tuple<*>`` ``record`` [2] ``list<*>`` ``vector`` ``set<*>`` ``set`` ``vector<*>`` ``vector`` ============= ============== [1] A corresponding Bro ``enum`` type is created automatically. [2] The record fields will by default be named ``f`` with *n* being the zero-baed element index. However, if the tuple gets passed into a Bro event, the initially created record will be coered into the type the event expects and hence then use the corresponding field names instead.