Kolide

Last Modification

Operating System Information

For my deployement of Kolide Fleet I used Ubuntu 16.04 server (64bit).

Distributor ID:   Ubuntu
Description:  Ubuntu 16.04.3 LTS
Release:  16.04
Codename: xenial

Prerequisites

Once you've installed Ubuntu you should update and upgrade the packages on they system.

sudo apt-get update && sudo apt-get -y upgrade

Now make sure you have curl installed as it is required for the getting Golang installed.

sudo apt-get install -y curl

According to the Kolide Launcher and Fleet documentation you will need to have Go 1.9 or greater installed. I followed Patrick Dahlke's "How to Install Go" post on Medium. The version of Go I installed for this project was 1.9.4.

Success

Setting your $GOPATH is critical for getting things to work!

PATH="$HOME/bin:$HOME/.local/bin:$PATH"
export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go

In addition go Go, you will need to install the build-essentials packages specifically for make, you will also need to install libc6-dev, gcc, g++ and dpkg-dev. Additional required software includes npm and yarn. NPM can be installed via the standard Ubuntu repositories. For yarn you will need to add their repository.

Warning

The version of yarn installed by cmdtest is incorrect. If you have it installed remove it and follow the instructions below:

sudo apt-get purge --auto-remove cmdtest

sudo apt-get install build-essential npm

curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn

The version of yarn that was installed after I ran through these steps was:

yarn --version 1.3.2

Node.js is also required. For me the version that was installed by default was v4.2.6 which is incompatible with fleet.

nodejs --version v4.2.6

To upgrade or install Node.js 9 run the following commands:

curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash -
sudo apt-get install -y nodejs

Install Docker CE and Docker Compose. I used the the installation guide found on docker docks.

Warning

Make sure you remove older versions of docker:

sudo apt-get remove docker docker-engine docker.io

Update the package index:

sudo apt-get update

Setup apt to use the docker repository over https.

sudo apt-get install apt-transport-https ca-certificates software-properties-common

Add the GPG key for the docker repo:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Verify the fingerprint of the key by searching for the last 8 characters:

sudo apt-key fingerprint 0EBFCD88

Now its time to add the "stable" docker repository. If you prefer you can add, edge or test.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Now we are ready to install Docker CE:

sudo apt-get update
sudo apt-get install -y docker-ce

And the final step is installing docker compose:

sudo apt-get install -y docker-compose

Now lets enable the docker service which will start the daemon after reboots, start the service, and check to make sure docker is running:

sudo systemctl enable docker
sudo systemctl start docker
sudo systemctl status docker

Installing Kolide Fleet

Now we need to clone the Fleet repository in our $GOPATH:

mkdir -p src/github.com/kolide
cd $GOPATH/src/github.com/kolide
git clone https://github.com/kolide/fleet.git
cd fleet

If we've taken care of all the prerequisites the following commands should work as application dependencies, javascript bundles, and the binary is built.

make deps
make generate
make

I found it necessary to add user: "1000:50" to the docker-compose.yml file to get things working properly, without it mysql will not startup, it throws an error, "mysqld: Can't create/write to file '/tmp/' (Errcode: 13 - Permission denied)"

services:
mysql:
  image: mysql:5.7
  user: "1000:50"
  volumes:
    - .:/tmp

Now its time to startup the infrastructure requirements for Fleet that includes redis, mysql server, and mailhog. You can use the -d flag for docker-compose up if you prefer and use docker-compose logs to check for startup errors if you prefer.

sudo docker-compose up

The next step is to setup the database schema, if you are did not use the -d flag in your docker-compose command you will need to run the follwing in another terminal shell.

./build/fleet prepare db

Its time to get the fleet server up and running. The following command is how you do it, but make sure to substitute generate your own 32 character key:

./build/fleet serve --auth_jwt_key=3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP

There are many ways you can generate the random key, the following is my prefered method (assuming openssl is installed):

openssl rand -base64 32

Once the server is running open your browser and go to https://localhost:8080 and follow the setup process.

Running Fleet as a Service. Create a configuration file for the fleet service:

vim /etc/systemd/system/fleet.service

Below is how I configured the service on my system:

[Unit]
Description=Kolide Fleet Application Server
[Service]
User=kolideuser
Group=kolidegroup
WorkingDirectory=/your/path/to/kolide/fleet
Environment="PATH=/your/path/to/kolide/fleet/build/"
ExecStart=/your/path/to/kolide/fleet/build/fleet serve --config /your/path/to/kolide/fleet/kolide.yml
[Install]
WantedBy=multi-user.target

Now modify the kolide.yml file with the custom options for your fleet server:

server:
cert: /your/path/to/kolide/kolide/fleet/certs/server.name.crt
key: /your/path/to/kolide/fleet/certs/server.name.key
address: 0.0.0.0:443
logging:
  json: true
auth:
  jwt_key: 3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP
Note

In the above kolide.yml file I added a option to have the server listen on port 443 instead of 8080. By default Linux does NOT allow processes to listen on Non-Ephemeral ports by default, so you will need to run the following command to enable fleet to run on port 443:

sudo setcap CAP_NET_BIND_SERVICE=+eip /your/path/to/kolide/fleet/build/fleet

Securing Your Server

Securing your kolide server is critical, so don't forget to enable the firewall and restrict access. For my environment I enabled 443 access in order for endpoints to connect, and ssh for management of the server. A good first step is to determine if you want to support IPv6 or not.

sudo vim /etc/default/ufw

Flip the IPV6= flag to yes or no depending upon your preference:

IPV6=no

Ensure the firewall is running:

sudo systemctl enable ufw
sudo systemctl start ufw
sudo ufw status

To disalbe IPv6 completely on the host, sudo edit /etc/sysctl.config and add the following lines:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

If IPv6 isn't disabled try the following:

sudo sysctl -p

You should see the outuput of the 3 lines added to the sysctl.config file, next:

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

It should return the value of 1.

Add rules to allow access to port 443 and restricted SSH access:

sudo ufw allow to any port 443
sudo ufw allow from 10.0.0.0/24 to any port 22
sudo ufw status

Some Custom Tweaks

I find myself tweaking/modifying files often in the fleet and launcher directories which is a long directory path so I find creating custom aliases valuable. Edit the bashrc (vim ~/.bashrc) file and add the following lines:

#My Custom Aliases
alias cdfleet='cd $GOPATH/src/github.com/kolide/fleet'
alias cdlaunch='cd $GOPATH/src/github.com/kolide/launcher'

Don't forget to source the file so the aliases are useable in the current terminal:

source ~/.bashrc

Installing Kolide Launcher

Success

From all of the documentation I've read the only way to currently get packages built for MacOS is to install launcher on a MacOS system. You will need to install Go (1.9 or greater). Again setting your $GOPATH is critical for getting things to work!

PATH="$HOME/bin:$HOME/.local/bin:$PATH"
export PATH=$PATH:/usr/local/go/bin
export GOPATH=$HOME/go

We need to clone the Launcher repository:

cd $GOPATH/src/github.com/kolide
git clone https://github.com/kolide/launcher.git
cd launcher

Just like we did with Fleet we need to install the dependencies and build the binaries.

make deps
make
./build/launcher --help

Package Builder will need the Enroll Secret code in order to build the packages specific to your kolide server. You can find the Enrol Secret by logging into kolide and clicking on the Add New Host button.

I prefer to utilize the package-builder for deploying osquery to MacOS and Linux systems. First, make sure you have zip installed.

make deps
make xp
make deps
make package-builder

For more information about how package-builder works you can run the following commands:

./build/package-builder --help
./build/package-builder make --help

To create the packages for MacOS and Linux you can run the following. Note if you have a valid certificate you can drop the --insecure flag.

./build/package-builder make -hostname=yoursystem.com:443 -enroll_secret="MzAUQ9KLGvF42axd57PNBFjVEXvXqruAoAJS2rinkeoG7Njk" --insecure

Adding the --autoupdate allows for push updates, wiithout it you will need ot create new packages for updates and push them out.

Installing Osquery on Ubuntu

I used the installation instructions found on the Osquery website.

export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY
sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main'
sudo apt-get update
sudo apt-get install osquery

Enable the osqueryd daemon:

systemctl enable osqueryd

Add the Erollment Secret to the enroll_secret file:

echo 'MzAUQ9KLGvF42axd57PNBFjVEXvXqruAoAJS2rinkeoG7Njk' > /var/osquery/enroll_secret

Installing Osquery on Windows

Warning

My Windows skills are lacking so that probably contributed with my struggles in setting osquery up. I performed so much troubleshooting that I didn't do a good job capturing everything I did, however, what I've provided I was able to replicate several times.

Installing osquery

I found the best way to install osquery on windows is using Chocolatey. Installation instructions for Chocolatey can be found here. Once installed you can run the following command. If you want osquery to run as a service make sure you use the params flag below. The official documenation can be found here.

choco install osquery --params='/InstallService'

Once installed we need to make some configuration changes in order to connect to our Kolide server. First add your configuration information to the osquery.flags file. I would have preferred to use a kolide.flags file, however, the service is already pointed to the osquery.flags file and I'm not sure how to change it and didn't really want to spend the time researching how to do it.

Success

With Linux and macOS the osquery.flags file begins with osqueryd followed by the list of flags. The Windows osquery service will not start with osqueryd in the flag file. I also removed the space before each flag.

--enroll_secret_path=c:\ProgramData\osquery\certs\enroll_secret
--tls_hostname=kolide.server.com
--host_identifier=uuid
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--logger_plugin=tls
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10

I also found if you are using a signed certificate for your Kolide server you do not need to have a flag pointing to the certificate. Having --tls_server_certs=c:\ProgramData\osquery\certs\kolide.server.crt in the flags file caused TLS connection errors. Once I removed it I was able to successfully connect the system to the Kolide server.

Next you will need to either add the enroll secret in your flag file with:

--enroll_secret_env=OSQUERY_ENROLL_SECRET

Or point to a file that contains the enroll secret like I chose to do in my flag file example:

--enroll_secret_path=c:\ProgramData\osquery\certs\enroll_secret

With those changes we can now test running osquweryd. I found it very useful to start the daemon via the command prompt because it outputs any errors there. Open the command prompt as Administrator and change directories to c:\ProgramData\osquery\osqueryd and run the following command.

osquerd.exe --flagfile=c:\ProgramData\osquery\osquery.flags --verbose --tls_dump

If everything is configured properly you should see plenty of output showing the system connecting to the Kolide server. You can double check in the Kolide interface under Hosts -> New. Now you can kill the daemon and start osqueryd via the Services.App.

Warning

In preparation for building the msi installer I did some "cleanup" where I deleted the certs.pem file located in the certs directory. Doing so completely killed my connection to the Kolide server. I have not investigated into why that happened, however, because I didn't have a backup of the file I and to reinstall osquery via choco.

Creating a MSI for Windows

Git is a requirement so ensure you have it installed. Once git is installed clone the osquery repo:

 git clone https://github.com/facebook/osquery.git

Now follow the next 3 steps:

1.

.\tools/make-win64-dev-env.bat

2.

.\tools\make-win64-binaries.bat

The first time I ran the command it failed with an error, "clang is not a full path and was not found in the path." To fix the problem I followed the steps outlined in this MSDN blog post.

3.

.\tools/deployment/make_windows_package.ps1 'msi'

When I executed the PowerShell command I received an error, "cannot be loaded because running scripts is disabled on this system." I made the following changes in order for the script to run.

Set-ExecutionPolicy RemoteSigned

At the time of documenting my install procedure I ran into a problem where a endpoint would partially connect to Kolide but never fully populate the data. On the client side I was receiving an error, "json: cannot unmarshal number into Go struct field distributedQueryResultsShim.statuses of type string."

There was a bug with running osquery version 3 so I had to do a build for version 2.11.2. In order to do that I did a git checkout with the version tag:

git checkout tags/2.11.2

I then went through the 3 steps listed above, make-win64-dev-env.bat and make-win64-binaries.bat followed by the make_windows_package.ps1.

.\tools\deployment\make_windows_package.ps1 -Type msi - ConfigFile C:\Path\To\osquery.conf -FlagFIle C:\Path\To\osquery.flags -Extras @("C:\Path\To\enroll_sercret_file")

Now for the fun part, installing msi's, rpm's, dep's, and pkgs on your endpoints, as well as, begining to play with osquery packages etc!

Resources:

Presentations

Blog Posts

Trail of Bits Series

Alternatives to Kolide