Binmap: a system scanner

System map warehouse

binmap builds a database of hashes and informations for systems. One of our goal is to provide a kind of warehouse with the database for several systems, and update the databases to track the systems as they evolve.

This is very useful when one wants to diff not only a binary but systems as a whole, to see what binaries have changed, which are new or removed.

We intend to provide a warehouse for databases, either created by us or sent by contributors. The files produced by binmap could be piped to gpg to ensure some kind of authentication and integrity of the various databases.

This needs to be automatized, so we need to setup a backend which will download and install an OS, run binmap to create the database, and update it when needed. Certainly, some tools from the devops community could help there.

Having a website accessible with hashes and some more information about clean files is something security community needs. We see it with IRMA, our asynchronous and customizable analysis platform for suspicious files, in order to quickly sort good and bad files.

Supporting static libraries

As explained, that is the biggest challenge. We did not want to go into that before having a proper piece of code. We believe that is ok for now, but this feature will require big brains and manpower, like for instance Silvio Cesare's work on binary similarities.

People working on malware have also the same issue: recognizing small piece of code (from the malware) into other binaries. The problem is the same and known to be hard. By open sourcing binmap, we hope people working on binary similarities will use it to do their research and tests.

Searching the information

Never ask a security guy to design a user interface. Security researchers tend to like it the hard way, but at some point, being smart is the right choice.

Gathering information is not the most difficult here. Presenting it, being able to retrieve the right information is. We have started to develop a web interface for binmap: it loads a database and can be used to explore the graph of relations or search for some functions. We also have an API to use the database and automatize some actions. We hope that people interested in UX will also join to improve it.

Some eye candy with the first draft of our prototype UI :-)

Risk map

When analyzing a system, it is really useful to know where previous bugs are located . By mixing the information about the binaries and libraries used by a reported vulnerability (e.g. CVE), one can draw a risk map , see what components are more prone to attacks and take action to buid a proper defense.

We have started to combine the information gathered by binmap with external sources. We focus on 2 kind of information:

1. The version of the files: we attempt to extract the version of a file either from the file itself or from its name. Then, we try to compare it with the latest known version from the official website. That way, we can quickly see how up-to-date is a system.

2. Many vulnerabilities are reported but not all of them are given a CVE. We can still map the name of a file with the known CVEs, for instance using the great CIRCL CVE Search

In the end, the information gathered on the system is compared with external information to learn more about its risks.

Installation

$ mkdir _build
$ cd _build
$ cmake ..
$ make

$ make install

$ cmake -DBoost_DEBUG=ON -G "Visual Studio 12" -DBoost_USE_STATIC_LIBS=ON -DBOOST_ROOT=D:\Programming\Libraries\boost_1_55_0 -DBOOST_LIBRARYDIR=D:\Programming\Libraries\boost_1_55_0\lib32-msvc-12.0 -DZLIB_LIBRARY=D:\Programming\Libraries\zlib-1.2.8 -DZLIB_INCLUDE_DIR=D:\Programming\Libraries\zlib-1.2.8

Usage

Using binmap is a two step process:

1. Scan a directory (or a file), for instance:

   $ ./binmap scan -v1 /usr/local -o local.dat
   

   This creates a database containing informations about the binaries that lie in this directory.

2. Dump the database to the dot format:

   $ ./binmap view -i local.dat -o local.dot
   

   or inspect the database using the Python API described below.

Python API

The blobmap module gives a read-only access to the content of a binmap database:

>>> import blobmap

First thing to do is to load a database:

>>> blobs = blobmap.BlobMap('local.dat')

A BlobMap is an ordered container of blobs, in chronological order, last being the most recent entry:

>>> blob = blobs.last()

A blob is basically a directed graph, where nodes are binaries and edges represent a use dependency---something like this program depends on this library. It can be indexed by paths, as in:

>>> clang_metadata = blob['/usr/local/bin/clang']
>>> print(str(clang_metadata))
clang: 8fcffc4a97cd4aaa1a32938a9e95d3b253476121(13223 exported symbols)(1303 imported symbols)(1 hardening features)

One can access the metadata for each node independently:

>>> clang_metadata.hash
8fcffc4a97cd4aaa1a32938a9e95d3b253476121
>>> clang_metadata.hardening_features
{'fortified'}
>>> help(clang_metadata)
[...]

The graph can be navigated using the successors and predecessors methods:

>>> blob.successors('/usr/local/bin/clang')
{'/lib/x86_64-linux-gnu/libtinfo.so.5',
 '/lib/x86_64-linux-gnu/libz.so.1',
 '/lib32/libc.so.6',
 ...}

It's also possible to make a diff between two blob, in order to gather intel concerning the changes of state of a system:

>>> from blobmap import BlobMap as BM
>>> b = BM('mynewprog.dat')
>>> g1, g0 = [b[k] for k in b.keys()][-2:]
>>> diff = g0.diff(g1)
>>> diff.added
{'/.../libmy1.so'}
>>> diff.removed
{'/.../libmy0.so'}
>>> diff.updated
{'/.../myprog'}