Operating System Information
For my deployement of Kolide Fleet I used Ubuntu 16.04 server (64bit).
Distributor ID: Ubuntu Description: Ubuntu 16.04.3 LTS Release: 16.04 Codename: xenial
Once you've installed Ubuntu you should update and upgrade the packages on they system.
sudo apt-get update && sudo apt-get -y upgrade
Now make sure you have curl installed as it is required for the getting Golang installed.
sudo apt-get install -y curl
According to the Kolide Launcher and Fleet documentation you will need to have Go 1.9 or greater installed. I followed Patrick Dahlke's "How to Install Go" post on Medium. The version of Go I installed for this project was 1.9.4.
Setting your $GOPATH is critical for getting things to work!
PATH="$HOME/bin:$HOME/.local/bin:$PATH" export PATH=$PATH:/usr/local/go/bin export GOPATH=$HOME/go
In addition go Go, you will need to install the build-essentials packages specifically for make, you will also need to install libc6-dev, gcc, g++ and dpkg-dev. Additional required software includes npm and yarn. NPM can be installed via the standard Ubuntu repositories. For yarn you will need to add their repository.
The version of yarn installed by cmdtest is incorrect. If you have it installed remove it and follow the instructions below:
sudo apt-get purge --auto-remove cmdtest
sudo apt-get install build-essential npmcurl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list sudo apt-get update && sudo apt-get install yarn
The version of yarn that was installed after I ran through these steps was:
yarn --version 1.3.2
Node.js is also required. For me the version that was installed by default was v4.2.6 which is incompatible with fleet.
nodejs --version v4.2.6
To upgrade or install Node.js 9 run the following commands:
curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash - sudo apt-get install -y nodejs
Install Docker CE and Docker Compose. I used the the installation guide found on docker docks.
Make sure you remove older versions of docker:
sudo apt-get remove docker docker-engine docker.io
Update the package index:
sudo apt-get update
Setup apt to use the docker repository over https.
sudo apt-get install apt-transport-https ca-certificates software-properties-common
Add the GPG key for the docker repo:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
Verify the fingerprint of the key by searching for the last 8 characters:
sudo apt-key fingerprint 0EBFCD88
Now its time to add the "stable" docker repository. If you prefer you can add, edge or test.
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
Now we are ready to install Docker CE:
sudo apt-get update sudo apt-get install -y docker-ce
And the final step is installing docker compose:
sudo apt-get install -y docker-compose
Now lets enable the docker service which will start the daemon after reboots, start the service, and check to make sure docker is running:
sudo systemctl enable docker sudo systemctl start docker sudo systemctl status docker
Installing Kolide Fleet
Now we need to clone the Fleet repository in our $GOPATH:
mkdir -p src/github.com/kolide cd $GOPATH/src/github.com/kolide git clone https://github.com/kolide/fleet.git cd fleet
make deps make generate make
I found it necessary to add user: "1000:50" to the docker-compose.yml file to get things working properly, without it mysql will not startup, it throws an error, "mysqld: Can't create/write to file '/tmp/' (Errcode: 13 - Permission denied)"
services: mysql: image: mysql:5.7 user: "1000:50" volumes: - .:/tmp
Now its time to startup the infrastructure requirements for Fleet that includes redis, mysql server, and mailhog. You can use the -d flag for docker-compose up if you prefer and use docker-compose logs to check for startup errors if you prefer.
sudo docker-compose up
The next step is to setup the database schema, if you are did not use the -d flag in your docker-compose command you will need to run the follwing in another terminal shell.
./build/fleet prepare db
Its time to get the fleet server up and running. The following command is how you do it, but make sure to substitute generate your own 32 character key:
./build/fleet serve --auth_jwt_key=3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP
There are many ways you can generate the random key, the following is my prefered method (assuming openssl is installed):
openssl rand -base64 32
Once the server is running open your browser and go to https://localhost:8080 and follow the setup process.
Running Fleet as a Service. Create a configuration file for the fleet service:
Below is how I configured the service on my system:
[Unit] Description=Kolide Fleet Application Server [Service] User=kolideuser Group=kolidegroup WorkingDirectory=/your/path/to/kolide/fleet Environment="PATH=/your/path/to/kolide/fleet/build/" ExecStart=/your/path/to/kolide/fleet/build/fleet serve --config /your/path/to/kolide/fleet/kolide.yml [Install] WantedBy=multi-user.target
Now modify the kolide.yml file with the custom options for your fleet server:
server: cert: /your/path/to/kolide/kolide/fleet/certs/server.name.crt key: /your/path/to/kolide/fleet/certs/server.name.key address: 0.0.0.0:443 logging: json: true auth: jwt_key: 3zqHl2cPa0tMmaCa9vPSEq6dcwN7oLbP
In the above kolide.yml file I added a option to have the server listen on port 443 instead of 8080. By default Linux does NOT allow processes to listen on Non-Ephemeral ports by default, so you will need to run the following command to enable fleet to run on port 443:
sudo setcap CAP_NET_BIND_SERVICE=+eip /your/path/to/kolide/fleet/build/fleet
Securing Your Server
Securing your kolide server is critical, so don't forget to enable the firewall and restrict access. For my environment I enabled 443 access in order for endpoints to connect, and ssh for management of the server. A good first step is to determine if you want to support IPv6 or not.
sudo vim /etc/default/ufw
Flip the IPV6= flag to yes or no depending upon your preference:
Ensure the firewall is running:
sudo systemctl enable ufw sudo systemctl start ufw sudo ufw status
To disalbe IPv6 completely on the host, sudo edit /etc/sysctl.config and add the following lines:
net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1
If IPv6 isn't disabled try the following:
sudo sysctl -p
You should see the outuput of the 3 lines added to the sysctl.config file, next:
It should return the value of 1.
Add rules to allow access to port 443 and restricted SSH access:
sudo ufw allow to any port 443 sudo ufw allow from 10.0.0.0/24 to any port 22 sudo ufw status
Some Custom Tweaks
I find myself tweaking/modifying files often in the fleet and launcher directories which is a long directory path so I find creating custom aliases valuable. Edit the bashrc (vim ~/.bashrc) file and add the following lines:
#My Custom Aliases alias cdfleet='cd $GOPATH/src/github.com/kolide/fleet' alias cdlaunch='cd $GOPATH/src/github.com/kolide/launcher'
Don't forget to source the file so the aliases are useable in the current terminal:
Installing Kolide Launcher
From all of the documentation I've read the only way to currently get packages built for MacOS is to install launcher on a MacOS system. You will need to install Go (1.9 or greater). Again setting your $GOPATH is critical for getting things to work!
PATH="$HOME/bin:$HOME/.local/bin:$PATH" export PATH=$PATH:/usr/local/go/bin export GOPATH=$HOME/go
We need to clone the Launcher repository:
cd $GOPATH/src/github.com/kolide git clone https://github.com/kolide/launcher.git cd launcher
Just like we did with Fleet we need to install the dependencies and build the binaries.
make deps make ./build/launcher --help
Package Builder will need the Enroll Secret code in order to build the packages specific to your kolide server. You can find the Enrol Secret by logging into kolide and clicking on the Add New Host button.
I prefer to utilize the package-builder for deploying osquery to MacOS and Linux systems. First, make sure you have zip installed.
make deps make xp make deps make package-builder
For more information about how package-builder works you can run the following commands:
./build/package-builder --help ./build/package-builder make --help
To create the packages for MacOS and Linux you can run the following. Note if you have a valid certificate you can drop the --insecure flag.
./build/package-builder make -hostname=yoursystem.com:443 -enroll_secret="MzAUQ9KLGvF42axd57PNBFjVEXvXqruAoAJS2rinkeoG7Njk" --insecure
Adding the --autoupdate allows for push updates, wiithout it you will need ot create new packages for updates and push them out.
Installing Osquery on Ubuntu
I used the installation instructions found on the Osquery website.
export OSQUERY_KEY=1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys $OSQUERY_KEY sudo add-apt-repository 'deb [arch=amd64] https://pkg.osquery.io/deb deb main' sudo apt-get update sudo apt-get install osquery
Enable the osqueryd daemon:
systemctl enable osqueryd
Add the Erollment Secret to the enroll_secret file:
echo 'MzAUQ9KLGvF42axd57PNBFjVEXvXqruAoAJS2rinkeoG7Njk' > /var/osquery/enroll_secret
Installing Osquery on Windows
My Windows skills are lacking so that probably contributed with my struggles in setting osquery up. I performed so much troubleshooting that I didn't do a good job capturing everything I did, however, what I've provided I was able to replicate several times.
I found the best way to install osquery on windows is using Chocolatey. Installation instructions for Chocolatey can be found here. Once installed you can run the following command. If you want osquery to run as a service make sure you use the params flag below. The official documenation can be found here.
choco install osquery --params='/InstallService'
Once installed we need to make some configuration changes in order to connect to our Kolide server. First add your configuration information to the osquery.flags file. I would have preferred to use a kolide.flags file, however, the service is already pointed to the osquery.flags file and I'm not sure how to change it and didn't really want to spend the time researching how to do it.
With Linux and macOS the osquery.flags file begins with osqueryd followed by the list of flags. The Windows osquery service will not start with osqueryd in the flag file. I also removed the space before each flag.
--enroll_secret_path=c:\ProgramData\osquery\certs\enroll_secret --tls_hostname=kolide.server.com --host_identifier=uuid --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_tls_refresh=10 --disable_distributed=false --distributed_plugin=tls --distributed_interval=10 --distributed_tls_max_attempts=3 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --logger_tls_period=10
I also found if you are using a signed certificate for your Kolide server you do not need to have a flag pointing to the certificate. Having --tls_server_certs=c:\ProgramData\osquery\certs\kolide.server.crt in the flags file caused TLS connection errors. Once I removed it I was able to successfully connect the system to the Kolide server.
Next you will need to either add the enroll secret in your flag file with:
Or point to a file that contains the enroll secret like I chose to do in my flag file example:
With those changes we can now test running osquweryd. I found it very useful to start the daemon via the command prompt because it outputs any errors there. Open the command prompt as Administrator and change directories to c:\ProgramData\osquery\osqueryd and run the following command.
osquerd.exe --flagfile=c:\ProgramData\osquery\osquery.flags --verbose --tls_dump
If everything is configured properly you should see plenty of output showing the system connecting to the Kolide server. You can double check in the Kolide interface under Hosts -> New. Now you can kill the daemon and start osqueryd via the Services.App.
In preparation for building the msi installer I did some "cleanup" where I deleted the certs.pem file located in the certs directory. Doing so completely killed my connection to the Kolide server. I have not investigated into why that happened, however, because I didn't have a backup of the file I and to reinstall osquery via choco.
Creating a MSI for Windows
Git is a requirement so ensure you have it installed. Once git is installed clone the osquery repo:
git clone https://github.com/facebook/osquery.git
Now follow the next 3 steps:
The first time I ran the command it failed with an error, "clang is not a full path and was not found in the path." To fix the problem I followed the steps outlined in this MSDN blog post.
When I executed the PowerShell command I received an error, "cannot be loaded because running scripts is disabled on this system." I made the following changes in order for the script to run.
At the time of documenting my install procedure I ran into a problem where a endpoint would partially connect to Kolide but never fully populate the data. On the client side I was receiving an error, "json: cannot unmarshal number into Go struct field distributedQueryResultsShim.statuses of type string."
There was a bug with running osquery version 3 so I had to do a build for version 2.11.2. In order to do that I did a git checkout with the version tag:
git checkout tags/2.11.2
I then went through the 3 steps listed above, make-win64-dev-env.bat and make-win64-binaries.bat followed by the make_windows_package.ps1.
.\tools\deployment\make_windows_package.ps1 -Type msi - ConfigFile C:\Path\To\osquery.conf -FlagFIle C:\Path\To\osquery.flags -Extras @("C:\Path\To\enroll_sercret_file")
Now for the fun part, installing msi's, rpm's, dep's, and pkgs on your endpoints, as well as, begining to play with osquery packages etc!
- Kolide - Official Website
- Kolide Fleet GitHub Site
- Kolide Launcher (package-builder) GitHub Site
- Windmill - TLS Endpoint for serving osquery configuration
- Unset Passwords
- Mac Managers - Operating Systems Analytics & Tools
- Mac Admins Episode 61: Combinatoric with Mike Arpaia
- Loco Moco Security Conference - Mike Arpaia: Starting, growing, and scaling your host intrusion detection efforts
- Osquery Community News
- Managing Osquery with Kolide Launcher and Fleet
- Install/Setup Kolide Fleet + Graylog + Osquery with Windows and Linux Deployment
- osquery Across the Enterprise
- Extending Osquery with Go
- Meet the SGT, an osquery Management Server Built Entirely on AWS!
- Trail of Bits osquery Extensions
- Announcing the Trail of Bits osquery extension repository
Trail of Bits Series
- Part One - Trail of Bits Blog - How are teams currently using osquery?
- Part Two - What are the current pain points of osquery?