Real-time log monitoring

You may need to watch different log files on automated test runs. With log.io you can simply monitoring log files via browser! This tutorial shows how easy it is.

Preconditions

Preparation

Create new project with following structure and files.

# create new project LogIO
$ mkdir -p ~/Projects/LogIO/data

# go into new Project
$ cd ~/Projects/LogIO

# create needed files in data
$ touch data/{harvester.conf,log_server.conf,web_server.conf,log.io}

# create Vagrantfile
$ touch Vagrantfile

# show files
$ tree .
.
├── Vagrantfile
└── data
    ├── harvester.conf
    ├── log.io
    ├── log_server.conf
    └── web_server.conf

File contents

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.require_version ">= 1.8.1"
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.box = "centos/7"
  config.vm.network "public_network"
  config.vm.synced_folder "./data", "/vagrant", disabled: false

  config.vm.provider "virtualbox" do |vb|
    vb.name = "LogIO"
    vb.cpus = "2"
    vb.memory = "2048"
    vb.gui = false
  end

  config.vm.provision "shell", inline: <<-SHELL
    # install needed packages
    sudo yum update -y && sudo yum install -y epel-release
    sudo yum install -y vim net-tools npm nodejs
    sudo yum clean all
    # install log.io for user <root>
    sudo npm install -g log.io --user "root"
    # provide custom files for user <root>
    sudo rm -f /root/.log.io/*
    sudo cp /vagrant/*.conf /root/.log.io/
    sudo chown root:root /root/.log.io/*.conf
    # provide init.d for log.io
    sudo cp /vagrant/log.io /usr/local/bin/log.io
    sudo chmod +x /usr/local/bin/log.io
    sudo chown root:root /usr/local/bin/log.io
  SHELL

end

Configure your Harvesters…

exports.config = {
  nodeName: "application_server",
  logStreams: {
    apache: [
      "/var/log/apache2/access.log",
      "/var/log/apache2/error.log"
    ]
  },
  server: {
    // connect to log.io server
    host: '127.0.0.1',
    port: 28777
  }
}

Configure your log server…

exports.config = {
  host: '0.0.0.0',
  port: 28777
}

Configure your web server…

exports.config = {
  host: '0.0.0.0',
  port: 28778,

  /*
  // Enable HTTP Basic Authentication
  auth: {
    user: "admin",
    pass: "1234"
  },
  */

  /*
  // Enable HTTPS/SSL
  ssl: {
    key: '/path/to/privatekey.pem',
    cert: '/path/to/certificate.pem'
  },
  */

  /*
  // Restrict access to websocket (socket.io)
  // Uses socket.io 'origins' syntax
  restrictSocket: '*:*',
  */

  /*
  // Restrict access to http server (express)
  restrictHTTP: [
    "192.168.29.39",
    "10.0.*"
  ]
  */

}

Create simple init script…

#!/bin/bash

start() {
  echo "Starting log.io process..."
  /usr/bin/log.io-server &
  /usr/bin/log.io-harvester &
}

stop() {
  echo "Stopping io-log process..."
  pkill node
}

status() {
  echo "Status io-log process..."
  netstat -tlp | grep node
}

case "$1" in
  start) start;;
  stop) stop;;
  status) status;;
  *) echo "Usage: start|stop|status";;
esac

Usage

# start VM via vagrant
$ vagrant up

# SSH into VM
$ vagrant ssh

# become root
$ sudo su -

# start log.io
$ log.io start

# get ip
$ ip addr

Now open your browser with URL http://<ip>:28778

dnstwist with docker

What happens when users make a typo in URL input? It can display a fake websites. This similar-looking domains can be used to attack you (Domain Name Permutation). With dnstwist you can find such “evil neighbors”. This tutorial shows how to use in a few seconds dnstwist.

Precondition

  • docker installed and running
  • docker-machine installed (optional)

Instructions

# create VM (optional)
$ docker-machine create -d virtualbox dnstwist

# pointing shell (optional)
$ eval $(docker-machine env dnstwist)

# download docker image (optional)
$ docker pull jrottenberg/dnstwist

# start dnstwist
$ docker run --rm jrottenberg/dnstwist bitbucket.org

Monitor running docker containers with cAdvisor

As a software tester, you have several containers run in your environment. Here is an example how easily and quickly you can monitor your test-environment with cAdvisor.

Preconditions

Preparation

# create project (local)
$ mkdir -p ~/Projects/Monitoring && cd ~/Projects/Monitoring

# create shell script (local)
$ vim start-demo.sh
#!/usr/bin/env sh

docker run -d --name cadvisor -P -v /:/rootfs:ro -v /var/run:/var/run:rw -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro google/cadvisor:latest
docker run -d --name jenkins -P jenkins
docker run -d --name selenium-hub -P selenium/hub:2.53.0
docker run -d --name selenium-node_1 --link selenium-hub:hub selenium/node-chrome:2.53.0
docker run -d --name selenium-node_2 --link selenium-hub:hub selenium/node-firefox:2.53.0

Note: You can also assign the respective ports!

Run docker containers

# create new VM (local)
$ docker-machine create -d virtualbox monitor

# show status (local)
$ docker-machine ls
...
NAME      ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
monitor   -        virtualbox   Running   tcp://192.168.99.100:2376           v1.11.1  

# copy into VM (local) 
$ docker-machine scp ~/Projects/Monitoring/start-demo.sh monitor:/home/docker/

# ssh into VM (local into VM)
$ docker-machine ssh monitor

# change rights (VM)
$ chmod +x start-demo.sh && ls -la

# run shell script (VM)
$ ./start-demo.sh

# list running docker container (VM)
$ docker ps -a
...
CONTAINER ID        IMAGE                          COMMAND                  CREATED              STATUS              PORTS                                               NAMES
57c2598b4261        selenium/node-firefox:2.53.0   "/opt/bin/entry_point"   4 seconds ago        Up 4 seconds                                                            selenium-node_2
d79a5123bcfc        selenium/node-chrome:2.53.0    "/opt/bin/entry_point"   29 seconds ago       Up 29 seconds                                                           selenium-node_1
095f9844346d        selenium/hub:2.53.0            "/opt/bin/entry_point"   About a minute ago   Up About a minute   0.0.0.0:32771->4444/tcp                             selenium-hub
8db3ad58d8ce        jenkins                        "/bin/tini -- /usr/lo"   About a minute ago   Up About a minute   0.0.0.0:32770->8080/tcp, 0.0.0.0:32769->50000/tcp   jenkins
d1e1e1c36d6d        google/cadvisor:latest         "/usr/bin/cadvisor -l"   2 minutes ago        Up 2 minutes        0.0.0.0:32768->8080/tcp                             cadvisor

Open browser

cAdvisor

MDK3 and CentOS 7

Last time i showed you, how to install Aircrack-ng. This time we will install MDK3 on CentOS 7.

Preparation

Installation

# install needed software
$ yum install -y wget bzip2

# download mdk3
$ wget http://aspj.aircrack-ng.org/mdk3-v6.tar.bz2

# unzip mdk3
$ bzip2 -cd mdk3-v6.tar.bz2 | tar xvd -

# compiling
$ cd d mdk3-v6/
$ make
$ mv mdk3 /usr/local/bin/
$ make clean

Usage

# create text-file with fake AP`s
$ echo -e "the force is with you\ncheck me\nhave fun" > fakeAP

# kill interfering processes
$ airmon-ng check kill

# set interface into monitor mode (my interface is wlp0s11u1)
$ airmon-ng start wlp0s11u1

# start mdk3 to create fake AP`s
$ mdk3 wlp0s11u1mon b -f fakeAP

Aircrack-ng and CentOS 7

This time i will show you, how to install Aircrack-ng on CentOS 7. My CentOS 7 (CentOS Linux release 7.2.1511 x64) is a virtual maschine on VirtualBox (5.0). As wireless USB Adapter i use TP-Link TL-WN822N.

Preparation

  • CentOS 7 VM created and started
  • SSH access (optional)
# add epel repository
$ yum install -y epel-release

# install wireless-tools
$ yum install -y wireless-tools

# check wifi
$ iwconfig
$ ip addr

Install Aircrack-ng

# install needed software
$ yum install -y git-svn libpcap-devel sqlite-devel gcc gcc-c++ libnl-devel openssl-devel usbutils pciutils rfkill

# get latests version (my was Aircrack-ng 1.2 rc3 r2799)
$ svn co http://svn.aircrack-ng.org/trunk/ aircrack-ng

# Compiling
$ cd aircrack-ng/
$ make install clean

Usage

# kill interfering processes
$ airmon-ng check kill

# set interface into monitor mode (my interface is wlp0s11u1)
$ airmon-ng start wlp0s11u1

# start packet capturing
$ airodump-ng wlp0s11u1mon

# stop monitor mode
$ airmon-ng stop wlp0s11u1mon

Python profiling with PyCharm Community Edition

Before we start, if you don`t know what is profiling read this Wikipedia article! In my opinion profiling should be a part of every development/build process! Whether the responsibility lies with QA or development. Python profiler are supported only in PyCharm Professional Edition. This article show you the possibilities for the community edition.

Preparation

  • PyCharm installed
  • Virtualenv or similar installed (optional)
  • PyCharm BashSupport Plugin installed

The easiest Profiler

With Unix/Linux time command you have allready a simple profiler! Time writes a message to standard output. Here you will find some information on Stackoverflow.

#!/usr/bin/env python
# -*- coding: utf-8 -*-


def hello_world():

    for i in range(1, 5):
        print '%d Hello world from python...' % i


if __name__ == '__main__':
    hello_world()

With BashSupport Plugin we can setup the “Run/Debug Configuration” like:

unix time profiler

Better informations

But now we need better information. For this we use cProfile, cprofilev and snakeviz.

# cProfile is part of standard python library

# install snakeviz
$ pip install snakeviz

# install cprofildev
$ pip install cprofilev

“Run/Debug Configuration” example

cProfile simple

Now will store the results into a file

cProfile store output

With snakeviz you can open the profile in browser:

$ snakeviz output.prof

The other option is to use cprofilev:

cprofilev

Even more information

If that was not enough,… we install some more libraries.

# install line_profiler
$ pip install line_profiler

# install memory_profiler and psutil
$ pip install memory_profiler
$ pip install psutil

Now we need to change the example code. We add the decorator…

#!/usr/bin/env python
# -*- coding: utf-8 -*-


@profile
def hello_world():

    for i in range(1, 5):
        print '%d Hello world from python...' % i


if __name__ == '__main__':
    hello_world()

the line_profiler configuration

kernprofiler

the memory_profiler

memory profiler

All configurations could now startet via the “Run” button. There are even more Profiler that you can use with similar PyCharm.

Simple port scanner with Python

If you like Python and NMap … there is a very good wrapper from Alexandre Norman! This tutorial show a very simple example for usage.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

import nmap


def port_scan(target, ports):
    """
    Simple NMap port scanner example

    @param target: host for scan
    @type target: string
    @param ports: ports for scan
    @type ports: string
    """

    nmap_scan = nmap.PortScanner()
    nmap_scan.scan(str(target), str(ports))

    for host in nmap_scan.all_hosts():
        print '=' * 80
        print 'Host:\t%s' % host
        print 'State:\t%s\n' % nmap_scan[host].state()

        for protocol in nmap_scan[host].all_protocols():
            print 'Protocol(s): %s' % protocol

            port_list = list(nmap_scan[host][protocol].keys())
            port_list.sort()

            for port in port_list:
                print '\n[+] Port: %s' % port
                print '[+] State: %s' % nmap_scan[host][protocol][port]


if __name__ == '__main__':
    port_scan('192.168.192.1', '1-1000')

For running see following lines:

# change execution
$ chmod u+x example.py

# start script
$ python -B ./example.py

 

pylint and lxml

If you use Python virtualenv, pylint and lxml together, you may see error messages in pylint test results. It`s because only trusted C extension resources (the standard library) should be used. Here is an opportunity to improve the pylint test results.

Generate a .pylintrc file

# generate .pylintrc file
$ pylint --generate-rcfile > .pylintrc

Open the .pylintrc file to edit

# open with vim editor
$ vim .pylintrc

Add lxml to extension-pkg-whitelist

# A comma-separated list of package or module names from where C extensions may
# be loaded. Extensions are loading into the active Python interpreter and may
# run arbitrary code
extension-pkg-whitelist=lxml

If you now perform the pylint test again, no error relating to lxml should appear.