macOS, Docker, Prometheus and Grafana

I like Grafana … the dashboards are just cool! Here (again) a tutorial about docker monitoring. In less minutes you should be done. As a comment … for Linux and Windows you can do that too! There are only partial changes.

Prepare Project

# create project
$ mkdir -p ~/Projects/DPG && cd ~/Projects/DPG

# show current IP
$ ifconfig | grep "inet " | grep -v 127.0.0.1

# create and edit prometheus.yml
$ vim prometheus.yml

Replace <yourLocalIP> with your IP. On Docker website you can find templates for Linux and Windows, too!

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['<yourLocalIP>:9323']

Configure Docker

This step is very easy. Just open Docker “Preferences” and specify in section “Daemon” -> “Advanced” the metrics-address. Just ensure that you use valid JSON!

macOS Docker Metrics

When you are done, press “Apply and Restart” button.

# view Docker metrics in browser
$ open -a Safari http://127.0.0.1:9323/metrics

Prepare Prometheus

# run Prometheus
$ docker run --name prometheus -p 9090:9090 -v $PWD/prometheus.yml:/etc /prometheus/prometheus.yml prom/prometheus

# open Prometheus WebUI
$ open -a Safari http://localhost:9090/targets

# get Prometheus IP
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' prometheus

Note: The space after /etc is just because of security settings of my provider! Please remove the space.

prometheus WebUI

Just for fun you can create already some graphs in Prometheus.

prometheus Graph

Prepare and run Grafana

# run Grafana
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) configure new DataSource for Prometheus.

Grafana DataSource Prometheus

Import Dashboard (ID: 1229)

Grafana Import Dashboard

… enter ID 1229 …

Grafana Dashboard Search

… be patient (don’t press any button) …

Docker Engine Metrics Dashboard

Select already created DataSource (Prometheus) and press “Import” button. Now you should see the awesome Grafana Dashboard.

Lunar – a UNIX security auditing tool

LUNAR is a open source UNIX security auditing tool written in Shell script. It offers the audit for various operating systems like Linux (RHEL, CentOS, Debian, Ubuntu), Solaris and Mac OS with less requirements. Services like Docker and AWS are also supported.

Download

Clone repository

# git clone
$ git clone https://github.com/lateralblast/lunar.git

Download via curl

# download via curl
$ curl -L -C - -o lunar.zip https://github.com/lateralblast/lunar/archive/master.zip

# extract archive
$ unzip lunar.zip

Usage

The use is very easy… but the outcome brings much values.

# show help
$ sh lunar.sh -h

# list functions
$ sh lunar.sh -S

# run ssh audit
$ sh lunar.sh -s audit_ssh_config

# run selinux audit in verbose mode
$ sh lunar.sh -s audit_selinux -v

# run all audits
$ sh lunar.sh -a

Test your infrastructure

Infrastructures can be very big. Luckily, there are provisioner like Chef, Salt, Ansible and etc. These provisioners can be very complex and possibly the developer has done something wrong. Therefore the infrastructure has to be tested! Tools like goss, Serverspec and Testinfra helps testers to validate. This tutorial show the first steps with Testinfra.

Testinfra is written in Python very small and easy to understand. Here is the GitHub repository.

Precondition

  • Vagrant (min. 1.9.3) installed
  • Python (min. 2.7) installed
  • pip (min. 9.0.1) and virtualenv (min. 15.1.0) installed
  • make (min. 3.81) installed

Project structure

To get used to it – i prepared some files for you. You only need to change the box name/url in Vagrantfile.

$ tree
.
├── Makefile
├── requirements.txt
├── Vagrantfile
└── tests.py
ENV_DIR = .env
CURRENT_DIR := $(shell pwd)
INTERPRETER = $(CURRENT_DIR)/$(ENV_DIR)/bin/
PATH := ${PATH}:$(INTERPRETER)

help:
	@echo "run make <target> with:"
	@echo " > env       : create virtualenv directory $(ENV_DIR)"
	@echo " > deps      : install dependencies from requirements.txt"
	@echo " > clean_env : delete virtualenv directory $(ENV_DIR)"
	@echo " > up        : run vagrant up"
	@echo " > destroy   : run vagrant destroy"
	@echo " > test      : run testinfra on vagrant environment"
	@echo " > clean_all : delete all files and directories"

env:
	@echo "[RUN]: create virtualenv"
	virtualenv $(ENV_DIR) && \
	. $(ENV_DIR)/bin/activate && \
	make deps

deps:
	@echo "[RUN]: install dependencies"
	$(INTERPRETER)/pip install -r requirements.txt

up:
	@echo "[RUN]: vagrant up"
	vagrant up

destroy:
	@echo "[RUN]: vagrant destroy -f"
	vagrant destroy -f
	rm -fr $(CURRENT_DIR)/.vagrant

test:
	@echo "[RUN]: run testinfr on vagrant environment"
	vagrant ssh-config > $(CURRENT_DIR)/ssh-config
	$(INTERPRETER)/pytest -v --hosts=default --ssh-config=$(CURRENT_DIR)/ssh-config tests.py

clean_all:
	@echo "[RUN]: delete all files and directories"
	rm -fr $(CURRENT_DIR)/.cache $(CURRENT_DIR)/__pycache__
	rm -f $(CURRENT_DIR)/ssh-config
	make destroy
	make clean_env

clean_env:
	@echo "[RUN]: delete virtualenv"
	rm -fr $(ENV_DIR)
testinfra==1.5.4
paramiko==2.1.2
#!/usr/bin/env python

def test_system_type(SystemInfo):
    '''Check OS type'''
    type = SystemInfo.type
    assert type == 'linux'

def test_user_exists(User):
    '''Check user exists'''
    user = User('vagrant')
    assert user.exists

def test_firewalld_is_installed(Package):
    '''Check firewalld is installed'''
    package = Package('firewalld')
    assert package.is_installed

def test_firewalld_running_and_enabled(Service):
    '''Check firewalld service is running and enabled'''
    service = Service('firewalld')
    assert service.is_running
    assert service.is_enabled
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|

  # disable ssh key update
  config.ssh.insert_key = false

  # vagrant box name
  config.vm.box = ""

  # vagrant box url
  config.vm.box_url = ""

  # disable box update
  config.vm.box_check_update = false

  # disable synced_folder
  config.vm.synced_folder ".", "/vagrant", disabled: true

  # 2nd network interface (public)
  # config.vm.network "public_network"

  # virtualbox settings
  config.vm.provider "virtualbox" do |vb|
    vb.name = "example_vm"
    vb.cpus = "2"
    vb.memory = "2048"
    vb.gui = false
  end

end

Usage

# create virtualenv and install dependencies
$ make env

# create vagrant environment
$ make up

# run tests
$ make test

# delete all generated files and directories
$ make clean_all

Testinfra offers several connections backends for remote command execution and can be used with python standard unit test framework: unittest. So the integration with build servers is easily possible.

TCP port scanner Brutescan

Brutescan is a fast and noisy TCP port scanner written in go.

Preparation

# install git and curl packages
$ sudo apt install -y curl git

# download go (do not install from Debian)
$ curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz

# unzip archive
$ tar xvf go1.8.linux-amd64.tar.gz

# set owner and group (recursive)
$ sudo chown -R root:root go

# move all into target directory
$ sudo mv go /usr/local/

Configure go (for user)

# create hidden go directory
$ mkdir ~/.go

# configure needed paths (inside .bashrc)
$ echo "GOPATH=$HOME/.go" >> ~/.bashrc
$ echo "export GOPATH" >> ~/.bashrc
$ echo "PATH=\$PATH:/usr/local/go/bin:\$GOPATH/bin" >> ~/.bashrc

# reload
$ source ~/.bashrc

# check go version
$ go version
go version go1.8 linux/amd64

Install brutescan

# install packages from github
$ go get github.com/asciimoo/brutescan

# show bombardier help
$ brutescan --help

Usage/Examples

# scan all ports on localhost
$ brutescan localhost
...
Scanning localhost (127.0.0.1)
Port range: 1-65535 (65534 ports)
Concurrent connections: 65512

port 22    open
port 25    open
port 111   open
port 42619 open

Scan finished in 2.970551852s
...

# scan port range with specific concurrent pool size
$ brutescan -pmin 22 -pmax 2000 -pool 100 heise.de
...
Scanning heise.de (193.99.144.80)
Port range: 22-2000 (1978 ports)
Concurrent connections: 100

no open ports found

Scan finished in 1m0.087341111s ...

HTTP benchmarking with Bombardier

Bombardier is a nice HTTP(S) benchmarking tool, written in Go language, for software performance testers.

Preparation

# install git and curl packages
$ sudo apt install -y curl git

# download go (do not install from Debian)
$ curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz

# unzip archive
$ tar xvf go1.8.linux-amd64.tar.gz

# set owner and group (recursive)
$ sudo chown -R root:root go

# move all into target directory
$ sudo mv go /usr/local/

Configure go (for user)

# create hidden go directory
$ mkdir ~/.go

# configure needed paths (inside .bashrc)
$ echo "GOPATH=$HOME/.go" >> ~/.bashrc
$ echo "export GOPATH" >> ~/.bashrc
$ echo "PATH=\$PATH:/usr/local/go/bin:\$GOPATH/bin" >> ~/.bashrc

# reload
$ source ~/.bashrc

# check go version
$ go version
go version go1.8 linux/amd64

Install bombardier

# install packages from github
$ go get -u github.com/codesenberg/bombardier

# show bombardier help
$ bombardier --help

Usage/Examples

# run with 5 connections on 10 sec.
$ bombardier -c 5 -k https://www.heise.de
...
Statistics        Avg      Stdev        Max
  Reqs/sec        32.44      37.83        201
  Latency      152.35ms    72.93ms      1.24s
  HTTP codes:
    1xx - 0, 2xx - 329, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     6.18MB/s
...

# run with 10 connections on 5 sec and show latency statistics.
$ bombardier -d 5s -c 10 -l -k https://www.heise.de
...
Statistics        Avg      Stdev        Max
  Reqs/sec        56.51      59.10        251
  Latency      173.10ms   102.95ms      1.32s
  Latency Distribution
     50%   155.83ms
     75%   164.06ms
     90%   174.99ms
     99%   542.91ms
  HTTP codes:
    1xx - 0, 2xx - 294, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:    10.63MB/s
...

 

HTTP inspection with Wuzz

Wuzz is a very easy command line tool for HTTP(S) inspection with very much potential. In this tutorial I will show the installation on Debian 8.7 (jessie).

Preparation

# install git and curl packages
$ sudo apt install -y curl git

# download go (do not install from Debian)
$ curl -O https://storage.googleapis.com/golang/go1.8.linux-amd64.tar.gz

# unzip archive
$ tar xvf go1.8.linux-amd64.tar.gz

# set owner and group (recursive)
$ sudo chown -R root:root go

# move all into target directory
$ sudo mv go /usr/local/

Configure go (for user)

# create hidden go directory
$ mkdir ~/.go

# configure needed paths (inside .bashrc)
$ echo "GOPATH=$HOME/.go" >> ~/.bashrc
$ echo "export GOPATH" >> ~/.bashrc
$ echo "PATH=\$PATH:/usr/local/go/bin:\$GOPATH/bin" >> ~/.bashrc

# reload
$ source ~/.bashrc

# check go version
$ go version
go version go1.8 linux/amd64

Install wuzz

# install packages from github
$ go get github.com/asciimoo/wuzz

# check wuzz version
$ wuzz --version wuzz 0.2.0

# show wuzz help
$ wuzz --help

# simple run
$ wuzz

If everything is going well, the terminal should look like this and you can start.

example wuzz cli

Tiny SSH audit

I wrote an bash script which makes the SSH server configuration audit a little bit easier. Here now an description for usage. The repository can found here.

Usage

# upload script to target host (tmp directory)
$ scp ssh_audit.sh <user>@<host>:/tmp

# login into target host
$ ssh <user>@<host>

# switch to root
$ su -

# change file permissions (optional)
$ chmod u+x /tmp/ssh_audit.sh

# run ssh audit
$ /tmp/ssh_audit.sh

The output is self-explanatory. If you need more details, just run following command.

# output the effective configuration to stdout (Extended test mode)
$ sshd -T

Install and configure Fail2Ban on CentOS 7

This tutorial presents the minimum SSH protection on CentOS 7 by Fail2Ban (without e-mail).

Preparation

# install epel-release
$ yum install -y epel-release && yum -y update

# install fail2ban packages
$ yum install -y fail2ban fail2ban-systemd

# update selinux-policies (if SELinux)
$ yum update -y selinux-policy*

Configuration

# change directory
$ cd /etc

# check content of 00-systemd.conf
$ cat fail2ban/jail.d/00-systemd.conf
...
[DEFAULT]
backend=systemd
...

# create custom default configuration
$ cp -p fail2ban/jail.conf fail2ban/jail.local

# edit custom default configuration
$ vim fail2ban/jail.local
...
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime  = 3600
maxretry = 3
...

# create custom sshd configuration
$ vim fail2ban/jail.d/sshd.local
...
[sshd]
enabled = true
port = ssh
logpath = %(sshd_log)s
...

Ready for startup

# enable fail2ban
$ systemctl enable fail2ban

# start fail2ban
$ systemctl start fail2ban

Check status

# check status (optional)
$ systemctl status fail2ban

# tail fail2ban logfile (optional)
$ tail -f /var/log/fail2ban.log

# tail secure logfile (optional)
$ tail -f /var/log/secure

# check iptables
$ iptables -L -n

# check status of jails
$ fail2ban-client status

# check status of sshd jail
$ fail2ban-client status sshd

Example

$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	347
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	1
   |- Total banned:	56
   `- Banned IP list:	185.110.132.202

$ whois 185.110.132.202
...
person:         Karamurzov Barasbi
abuse-mailbox:  abusemail@openstack.net.ua
address:        Belize, BE, Kolmo ave 11, apt 901
...