Shell linter evaluation and usage

Tomorrow, the 1st of August is a national holiday in Switzerland … So I do one day off and have some time. For a long time I wanted to deal with Shell lint. After some research, i found a few open-source tools. By the way … linters are being written for many programming languages and document formats.

Preparation

For evaluation i will not install the tools on my local system,… so Vagrant (with CentOS 7) is my choice.

# create project
$ mkdir -p ~/Projects/ShellLint && cd ~/Projects/ShellLint

# create example.sh
$ vim example.sh

# create Vagrantfile
$ vim Vagrantfile
#!/bin/bash

declare -r VERSION="1.0.0"
declare -r FILE_NAME=$(basename "$0")

function fc_usage()
{
 printf "Usage: %s" "$FILE_NAME"
 printf " [-h] [-V]\n"
}

function failure()
{
 print "here is a error"

syntax() {
 print "this line has simply to many chars ... with a simple shell lint you should see"
}

function fc_bashism()
{
 echo -e "hello world"
}

function main()
{
 fc_usage
 fc_bashism
}

exit 0
# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure("2") do |config|
  config.vm.box = "lupin/centos7"
  config.vm.box_check_update = false
  config.vm.synced_folder '.', '/vagrant', disabled: true

  config.vm.provision "file", source: "example.sh", destination: "example.sh"

  config.vm.provider "virtualbox" do |vb|
    vb.gui = false
    vb.name = "ShellLint"
    vb.memory = "1024"
    vb.cpus = 1
  end

end

Note: I created the Vagrant box “lupin/centos” via Packer … here my GitHub repository.

# create environment
$ vagrant up

# SSH into VM
$ vagrant ssh

Shell option -n

Many shell’s already offer a very simple script analysis. The option -n read commands in script, but do not execute them (syntax check).

# example bash
$ bash -n example.sh

# example shell
$ sh -n example.sh

Okay … but not really what I want… (more details are welcome)

shlint and checkbashisms

I found the repository here.

# install ruby (optional)
$ yum install -y ruby

# install json_pure (optional)
$ gem install json_pure

# install shlint
$ gem install shlint

# create config
$ echo -e 'shlint_shells="bash sh"\nshlint_debug=1' > .shlintrc

# run shlint
$ shlint example.sh

# run checkbashisms
$ checkbashisms -f example.sh

Note: for both tools you should change the shebang to “#!/bin/sh”

For shlint… I don’t get it. For checkbashisms … good if will write portable Shell scripts.

bashate

I found it here on Pypi.

# install epel repository
$ yum install -y epel-release

# install pip (python 2.x)
$ yum install -y python2-pip

# install bashate
$ pip install bashate

# run bashate
$ bashate example.sh

Nice … but not really all Standards.

Shellsheck

Shellcheck is known! Here the online service and here the repository.

# install epel repository
$ yum install -y epel-release

# update (optional)
$ yum update -y

# install ShellCheck
$ yum install -y ShellCheck

# run ShellCheck
$ shellcheck example.sh

I stay with that tool. Currently there are packages for almost every known OS.

Additional

Who knows me … knows that I do not like Installer and prefer Docker use. Here’s some fun.

# exit Vagrant and destroy
$ vagrant halt && vagrant destroy -f

# create Dockerfile
$ vim Dockerfile

# create Applescript
$ vim linter.scpt

# build Docker image
$ docker build -t alpine/shellcheck .

# change permission
$ chmod +x linter.scpt

# run Applescript
$ osascript linter.scpt
FROM alpine:latest

# install needed packages
RUN apk --update add wget

# download archive
RUN wget -q --no-check-certificate https://storage.googleapis.com/shellcheck/shellcheck-latest.linux.x86_64.tar.xz

# unzip archive
RUN tar xvfJ shellcheck-latest.linux.x86_64.tar.xz

# move binary
RUN mv /shellcheck-latest/shellcheck /usr/local/bin/shellcheck

# cleanup
RUN apk del wget
RUN rm -f shellcheck-latest.linux.x86_64.tar.xz
RUN rm -fr shellcheck-latest/

# change to mount directory
WORKDIR /mnt

# set entrypoint
ENTRYPOINT ["/usr/local/bin/shellcheck"]
#!/usr/bin/osascript

-- define global variables
global appName
global imageName

-- set magic values
set appName to "Docker"
set imageName to "alpine/shellcheck "

-- run docker linters
on LintShell(macPath)
	set posixPath to quoted form of POSIX path of macPath
	set fileName to do shell script "basename " & posixPath
	set dirName to do shell script "dirname " & posixPath
	set shellCmd to "docker run --rm -i -v " & dirName & ":/mnt " & imageName & fileName

	tell application "Terminal"
		set shell to do script shellCmd in front window
	end tell
end LintShell

-- display select box
on SelectFile()
	set dlTitle to "Nothing selected..."
	set dlMsg to "Process is terminated."

	try
		set macPath to choose file
	on error
		display dialog dlMsg buttons ["OK"] with title dlTitle
		return dlMsg
	end try

	LintShell(macPath)
end SelectFile

-- start Docker
on StartDocker()
	set dlTitle to "Docker cannot started"
	set dlMsg to "Something went wrong, could not start Docker!"

	try
		tell application appName
			activate
		end tell
	on error
		display dialog dlMsg buttons ["OK"] with title dlTitle
	end try
end StartDocker

-- check if Docker is running
on RunScript()
	set dlTitle to "Docker not running"
	set dlMsg to "Should Docker started?"

	if application appName is running then
		SelectFile()
	else
		display dialog dlMsg buttons ["OK", "No"] with title dlTitle
		if button returned of result is "OK" then
			StartDocker()
		end if
	end if
end RunScript

RunScript()

😉 just for fun…

Docker, Telegraf, InfluxDB and Grafana

I have already presented various tutorials on docker monitoring. This time we will use Telegraf.

Project preparation

# create new project
$ mkdir -p Projects/DTIG/influxdb && cd Projects/DTIG/

# create telegraf.conf
$ touch telegraf.conf

InfluxDB preparation

# start InfluxDB
$ docker run --name influxdb -p 8086:8086 -v $PWD/influxdb:/var/lib/influxdb influxdb

# create new user
$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE USER telegraf WITH PASSWORD 'password123' WITH ALL PRIVILEGES"

# create database for telegraf
$ curl -G http://localhost:8086/query -u telegraf:password123 --data-urlencode "q=CREATE DATABASE telegraf_db"

# show ip of influxdb container
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' influxdb
...
172.17.0.2
...

Telegraf preparation

# Telegraf Configuration

[agent]
  interval = "10s"
  round_interval = true
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_interval = "10s"
  flush_jitter = "0s"
  precision = ""
  debug = false
  quiet = false
  logfile = ""
  hostname = ""
  omit_hostname = false

[[outputs.influxdb]]
  urls = ["http://172.17.0.2:8086"] # required
  database = "telegraf_db" # required
  retention_policy = ""
  write_consistency = "any"
  timeout = "5s"

[[inputs.cpu]]
  percpu = true
  totalcpu = true
  collect_cpu_time = false

[[inputs.disk]]
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

[[inputs.docker]]
  endpoint = "unix:///var/run/docker.sock"
  container_names = []
  timeout = "5s"
  perdevice = true
  total = false
  # docker_label_include = []
  # docker_label_exclude = []

… Read more about Telegraf on documentation page …

# edit telegraf.conf
$ vim telegraf.conf

# start Telegraf
$ docker run --name telegraf -v $PWD/telegraf.conf:/etc /telegraf/telegraf.conf -v /var/run/docker.sock:/var/run/docker.sock telegraf

Note: The space after /etc is just because of security settings of my provider.

… after short time …

# show measurements (be patient)
$ curl -G http://localhost:8086/query -u telegraf:password123 --data-urlencode "db=telegraf_db" --data-urlencode "q=SHOW MEASUREMENTS"

Grafana preparation

# run grafana container
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) you can add new Data Source.

Grafana InfluxDB Telegraf DataSource

Okay … all done … you can start to create Dashboards or search for existing Dashboards for import.

Penetration testing report via Serpico

Penetration testing make fun but writing penetration testing reports is boring. When you start you will look for templates or software which supports you. Here comes Serpico into the game. Serpico is a collaboration and report generation tool. The best… it is open-source.

Usage via Docker

# search for Serpico (optional)
$ docker search serpico

# download official image
$ docker pull serpicoproject/serpico

# create and run container
$ docker run --name serpico -p 8888:443 -it serpicoproject/serpico /bin/bash -l

# run setup script (only 1st time)
$ ruby scripts/first_time.rb

# create new user
$ ruby scripts/create_user.rb admin test123 1

# start serpico
$ ruby serpico.rb

Now you can use Serpico in your favorite browser…

# macOS use Safari
$ open -a Safari https://localhost:8888

Login with created credentials (admin/test123) and create your reports. On Youtube is a good introduction. If you don’t have Microsoft Words installed, you can view your reports online.

Open Source GUI tools for macOS

Tiny list for Open Source GUI tools for macOS to increase productivity…

Compilation

Koala

Debug

Knuff

Database

Docker

hosts

Package Manager

Cakebrew

Proxy

REST

RegExp

Vagrant

Vagrant Manager

VPN

Tunnelblick

VCS

Docker, cAdvisor, InfluxDB and Grafana

In previous tutorials I showed the basics for docker monitoring with Prometheus and Jenkins. Here are now the basics with cAdvisor. Many steps are similar and can be implemented just as quickly.

Preparation

# create project
$ mkdir -p Projects/DCIG/influxdb && cd Projects/DCIG/

InfluxDB preparation

This time we start with InfluxDB, because cAdvisor needs it for connection!

# start InfluxDB
$ docker run --name influxdb -p 8086:8086 -v $PWD/influxdb:/var/lib/influxdb influxdb

# create new user
$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE USER cadvisor WITH PASSWORD 'password123' WITH ALL PRIVILEGES"

# create database for cadvisor
$ curl -G http://localhost:8086/query -u cadvisor:password123 --data-urlencode "q=CREATE DATABASE cadvisor_db"

# show ip of influxdb container
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' influxdb
...
172.17.0.2
...

cAdvisor preparation

Now we will use User, Password and Database name. You can find the documentation here.

# start cAdvisor
$ docker run -d --name=cadvisor -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro -p 8080:8080 google/cadvisor:latest -storage_driver=influxdb -storage_driver_db=cadvisor_db -storage_driver_user=cadvisor -storage_driver_password=password123 -storage_driver_host=172.17.0.2:8086

# check cAdvisor logs (optional)
$ docker logs -f cadvisor

# open cAdvisor WebUI
$ open -a Safari http://localhost:8080

After a while we can also see if cAdvisor sends metrics to InfluxDB.

# show measurements (be patient)
$ curl -G http://localhost:8086/query -u cadvisor:password123 --data-urlencode "db=cadvisor_db" --data-urlencode "q=SHOW MEASUREMENTS"

prepare and run Grafana

# run grafana container
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) configure new DataSource for InfluxDB.

DataSource InfluxDB

When DataSource is configured we import the Grafana Dashboard. (ID: 1367)

Dashboard Search cAdvisor

… be patient (don’t press any button) …

Import cAdvisor Dashboard

Select already created DataSource (InfluxDB) and press “Import” button. Now you should see the Grafana Dashboard.

macOS, Docker, Prometheus and Grafana

I like Grafana … the dashboards are just cool! Here (again) a tutorial about docker monitoring. In less minutes you should be done. As a comment … for Linux and Windows you can do that too! There are only partial changes.

Prepare Project

# create project
$ mkdir -p ~/Projects/DPG && cd ~/Projects/DPG

# show current IP
$ ifconfig | grep "inet " | grep -v 127.0.0.1

# create and edit prometheus.yml
$ vim prometheus.yml

Replace <yourLocalIP> with your IP. On Docker website you can find templates for Linux and Windows, too!

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['<yourLocalIP>:9323']

Configure Docker

This step is very easy. Just open Docker “Preferences” and specify in section “Daemon” -> “Advanced” the metrics-address. Just ensure that you use valid JSON!

macOS Docker Metrics

When you are done, press “Apply and Restart” button.

# view Docker metrics in browser
$ open -a Safari http://127.0.0.1:9323/metrics

Prepare Prometheus

# run Prometheus
$ docker run --name prometheus -p 9090:9090 -v $PWD/prometheus.yml:/etc /prometheus/prometheus.yml prom/prometheus

# open Prometheus WebUI
$ open -a Safari http://localhost:9090/targets

# get Prometheus IP
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' prometheus

Note: The space after /etc is just because of security settings of my provider! Please remove the space.

prometheus WebUI

Just for fun you can create already some graphs in Prometheus.

prometheus Graph

Prepare and run Grafana

# run Grafana
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

# open Grafana WebUI and login (admin:admin)
$ open -a Safari http://localhost:3000

After login (admin:admin) configure new DataSource for Prometheus.

Grafana DataSource Prometheus

Import Dashboard (ID: 1229)

Grafana Import Dashboard

… enter ID 1229 …

Grafana Dashboard Search

… be patient (don’t press any button) …

Docker Engine Metrics Dashboard

Select already created DataSource (Prometheus) and press “Import” button. Now you should see the awesome Grafana Dashboard.

macOS ScreenShots

I know there are already a lot of tutorials for this topic, but partly incomplete or no longer up to date. That’s why I’m trying now, since software testers create very often screenshots. 😉

via Preview.app

The first possibility is the Preview.app … here you can easily select with the mouse what you want to do.

Preview.app ScreenShots

via Keyboard

The next possibility are shortcuts. Here you don’t need to start Preview.app!

Capture the entire screen

[Shift] + [Command] + [3]

Capture via specific selection

[Shift] + [Command] + [4]

Capture window/menu

[Shift] + [Command] + [4]

here you can press [Space] to toggle

Capture the current Touch Bar

[Shift] + [Command] + [6]

For save to clipboard press [Control] on all actions.

via Terminal

# show man-pages
$ man screencapture

# show help
$ screencapture -h

# simple jpg
$ screencapture test.jpg

# specify file type (PNG, PDF, JPG, GIF, TIFF, BMP, PSD, PCT, etc.)
$ screencapture -t tiff test.tiff

# send to the clipboard (no name)
$ screencapture -c

# use timer (10 seconds)
$ screencapture -T 10 test.jpg

Note: please see man-pages – here you find more options!

ScreenShot Settings

# change your file format
$ defaults write com.apple.screencapture type jpg

# disable shadows
$ defaults write com.apple.screencapture disable-shadow -bool true
$ killall SystemUIServer

# enable shadows
$ defaults write com.apple.screencapture disable-shadow -bool false
$ killall SystemUIServer

# set default name
$ defaults write com.apple.screencapture name "ScreenShotName"
$ killall SystemUIServer

# set no default name
$ defaults write com.apple.screencapture name ""
$ killall SystemUIServer

# default location
$ defaults write com.apple.screencapture location ~/Pictures/
$ killall SystemUIServer

# revert default location
$ defaults write com.apple.screencapture location ~/Desktop/
$ killall SystemUIServer

Jenkins, InfluxDB and Grafana

Today an basic introduction to Jenkins, InfluxDB and Grafana. Docker is used to save some time. Okay,… let’s start.

Preparation

# create project and change directory
$ mkdir ~/Projects/JIG/influxdb && cd ~/Projects/JIG/

# download official jenkins image (latest)
$ docker pull jenkins

# download official influxdb image (latest)
$ docker pull influxdb

# download official grafana image (latest)
$ docker pull grafana/grafana

# list docker images
$ docker images
...
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jenkins             latest              59b08e8f6e37        4 days ago          704 MB
grafana/grafana     latest              2cdb407c0fa4        7 days ago          286 MB
influxdb            latest              fad81160f2de        13 days ago         224 MB
...

Jenkins preparation

# start Jenkins
$ docker run --name jenkins -p 8080:8080 jenkins

# copy password from cli
...
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

b49ffa5749724d61b43d3a159b181133
...

Now open your favorite browser with URL http://localhost:8080 and unlook Jenkins with following steps.

unlook jenkins

  1. unlook with password from cli
  2. install suggested plugins
  3. create your admin user
  4. start using jenkins

Next, the InfluxDB plug-in must be installed.

influxdb plugin jenkins

InfluxDB preparation

# start InfluxDB
$ docker run --name influxdb -p 8086:8086 -v $PWD/influxdb:/var/lib/influxdb influxdb

# check current configuration (optional)
$ docker exec -i influxdb influxd config

# create new user
$ curl -G http://localhost:8086/query --data-urlencode "q=CREATE USER jenkins WITH PASSWORD 'password123' WITH ALL PRIVILEGES"

# create database for jenkins
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "q=CREATE DATABASE jenkins_db"

# show users (optional)
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "q=SHOW USERS"
...
{"results":[{"statement_id":0,"series":[{"columns":["user","admin"],"values":[["jenkins",true]]}]}]}
...

# show databases (optional)
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "q=SHOW DATABASES"
...
{"results":[{"statement_id":0,"series":[{"name":"databases","columns":["name"],"values":[["_internal"],["jenkins_db"]]}]}]}
...

# show measurements
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "db=jenkins_db" --data-urlencode "q=SHOW MEASUREMENTS"
...
{"results":[{"statement_id":0}]}
...

Connect Jenkins with InfluxDB

# start jenkins container (if stopped)
$ docker start jenkins

# show ip of influxdb container
$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' influxdb
...
172.17.0.2
...

Add new InfluxDB target on Jenkins

jenkins influxdb settings

Save and create a new freestyle job. For example with following configuration.

jenkins job influxdb

When you are done, run the job.

# show measurements
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "db=jenkins_db" --data-urlencode "q=SHOW MEASUREMENTS"
...
{"results":[{"statement_id":0,"series":[{"name":"measurements","columns":["name"],"values":[["jenkins_data"]]}]}]}
...

# run select statement (optional)
$ curl -G http://localhost:8086/query -u jenkins:password123 --data-urlencode "db=jenkins_db" --data-urlencode "q=SELECT * FROM jenkins_data"
...
{"results":[{"statement_id":0,"series":[{"name":"jenkins_data","columns":["time","build_number","build_result","build_result_ordinal","build_status_message","build_successful","build_time","last_stable_build","last_successful_build","project_build_health","project_name","project_name_1"],"values":[["2017-06-08T17:27:24.487Z",9,"SUCCESS",0,"stable",true,16,9,9,100,"test","test"]]}]}]}
...

Add Grafana

# start containers (if stopped)
$ docker start influxdb && docker start jenkins

# run grafana container
$ docker run --name grafana -i -p 3000:3000 grafana/grafana

Open you browser with URL http://localhost:3000, login with credentials (admin/admin) and add a new InfluxDB data source.

grafana data source

From now on, you can create and share dashboards in Grafana, which shows all Jenkins metrics.