April 8, 2015

Envdb: Ask your environment questions.

I recently released a open source project called EnvDB. Envdb allows you to deploy osquery to your assets and ask them questions in an ad-hoc fashion. Think of each osquery install as a node in a database cluster. Envdb adds the glue to make this happen with next-to-no effort.

osquery #

Osquery is an open source project by Facebook. They describe their project as follows:

osquery allows you to easily ask questions about your Linux and OSX infrastructure. Whether your goal is intrusion detection, infrastructure reliability, or compliance, osquery gives you the ability to empower and ?inform a broad set of organizations within your company.

Using osquery box-by-box wasn’t working for me, but I was impressed by the amount of information I could gather from it. What was even cooler was the ability to join on other relevant information it collects just like I’m used to in sql. The idea for Envdb was pretty clear at this point. I need a way to query all my assets at once, in real-time, with setup as easy as possible.

Go + envdb #

I decided to write Envdb in Go mostly because I love the languag, but to also make it easy to release. Envdb contains a few components and with Go I was able to combine them all into an easy to use cli and a single binary for download.

What can I use this for? #

One of the main use-cases for envdb is security. Osquery offers an interface into data that’s critical to have when dealing with a compromise or when you are trying to determine if you have already been. One of the saved queries that ships with the current version of envdb is:

select * from processes where on_disk = 0

This is basically asking the system if there are any processes currently running that no longer have their original binary still on disk. A lot of malware will load itself into memory and delete themselves in an attempt to evade detection. It’s also nice to ask which processes are currently listening on ports.

SELECT DISTINCT process.name, listening.port, listening.address, process.pid
FROM processes AS process 
JOIN listening_ports 
AS listening ON process.pid = listening.pid;

This query highlights the power of osquery and being able to join on other tables/data it collects.

Envdb is also very useful in an operations role where you need to ask questions regarding performance, connectivity or reliability of your infrastructure. For example, asking if a particular process is running:

select name from processes where name = "redis"

Queries are hard, Give me something to start with. #

Since the first release of envdb I have added a new feature for saving and loading queries.

Right now it’s a pretty trivial set of saved queries but I plan to create a service in the future for Envdb users to share and import community queries. Keep an eye out for that.

Using Envdb #

I made a quick intro video to envdb that shows how to setup the server, connect nodes and use the web UI.

Below is the current envdb command usage as of writing this post:

usage: envdb [<flags>] <command> [<flags>] [<args> ...]

The Environment Database - SELECT * FROM awesome;

Flags:
  --help       Show help.
  --debug      Enable debug logging.
  --dev        Enable dev mode. (read assets from disk and enable debug
               output)
  -q, --quiet  Remove all output logging.

Commands:
  help [<command>]
    Show help for a command.

  server [<flags>]
    Start the tcp server for node connections.

  node --server=127.0.0.1 [<flags>] <node-name>
    Register a new node.

Server

envdb server
- Note: By default this will start the tcp server on port 3636 and the web server on port 8080.
Node Client

sudo envdb node --server <ip to server> SomeBoxName
That’s it - it’s really that simple.

Moving Forward #

I plan to add support for a plugin interface for extending what Envdb can request from a node. Currently that list of planned extentions includes: ~~yara~~ (osquery is adding support for this), bro and [memory](Volatility). The hope is to wrap these processes and query them using sql like osquery and allowing you to join on similar data points.

Example:

select * from listening_ports a join bro_conn b on a.port = b.source_port;

Download #

Pre-built versions of envdb are avaliable for linux 386/amd64.
linux downloads

NODE: envdb also builds and runs on macosx but we don’t currently have pre-build binaries for darwin.

Getting Involved & Sponsors #

EnvDB is open source software under the Apache 2 license. The source code can be found at https://github.com/mephux/envdb.

This project currently being sponsored by my employer Critical Stack. If you haven’t heard of Critical Stack, check out our free intel service at https://intel.criticalstack.com

Kudos