Curl via Socks5 proxy on macOS

SSH tunnel in Browsers are easy! What about curl via SSH tunnels? Yeah – it`s easy, too!

Preparation

Check minimal firewall rules and SSH configuration on target host.

# ensure ssh service is enabled
$ firewall-cmd --list-services
...
ssh

# ensure AllowTcpForwarding is allowed
$ sshd -T | grep -i AllowTcpForwarding
...
allowtcpforwarding yes

Create SSH tunnel

Some basics about SSH tunnel…

# create ssh tunnel (foreground process)
$ ssh -ND localhost:9000 <user>@<host>
$ ssh -C4ND localhost:9000 <user>@<host>
$ ssh -C4ND localhost:9000 <user>@<host> -v
$ ssh -C4ND localhost:9000 <user>@<host> -p 22 -v

# create ssh tunnel (background process)
$ ssh -C4ND localhost:9000 <user>@<host> -f
  • C: use compression (level can be set in configuration file)
  • 4: forces ssh to use IPv4 only
  • N: do not execute a remote command
  • D: specifies dynamic application-level port forwarding
  • v: verbose mode
  • f: go to background before command execution
  • p: port to connect to on the remote host

Check SSH tunnel

The following examples will help you to monitor the connection to the target server.

# check ssh tunnel (local)
$ ps aux | grep ssh

# check ssh tunnel via lsof (target)
$ lsof -nPi | grep ssh

# check ssh tunnel via ss (target)
$ ss -4t

Use SSH tunnel

Now we use the tunnel via curl. With the service ipinfo.io we can verify.

# curl for external ip (without tunnel)
$ curl https://ipinfo.io/ip

# curl for external ip
$ curl --socks5 localhost:9000 https://ipinfo.io/ip

# curl for external ip (since curl v7.21.7)
$ curl -x socks5://localhost:9000 https://ipinfo.io/ip

Note: There are two protocol prefixes socks5:// and socks5h://. The 2nd will let the SOCKS server handle DNS-queries.

Kill SSH tunnel

The simplest and hardest way to kill SSH tunnels (on background) is following example. But be careful it kills all ssh connections!

# kill all ssh tunnel
$ sudo killall ssh

Firewalld Rich Rules basics

This tutorial will help you to get started with the firewalld configuration. Basics on zones and rich rules are presented.

What we do

The shell provisioner will ensure that on all hosts firewalld and curl are installed. For “host_protected” the provisioner will install nginx for demo purposes, too. Furthermore, the firewall will configured on “host_protected”.

Every host has two interfaces NAT (enp0s3) and host-only (enp0s8). The provisioner will not touch the NAT interface (zone: public) rules! Only the host-only interface (zone: home) rules will modified!

# show result configuration public (local)
$ vagrant ssh host_protected -c 'sudo firewall-cmd --list-all --zone=public'
...
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: dhcpv6-client ssh
  ports:
  protocols:
  masquerade: no
  forward-ports:
  sourceports:
  icmp-blocks:
  rich rules:

# show result configuration home (local)
$ vagrant ssh host_protected -c 'sudo firewall-cmd --list-all --zone=home'
...
home (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s8
  sources:
  services:
  ports:
  protocols:
  masquerade: no
  forward-ports:
  sourceports:
  icmp-blocks:
  rich rules:
    rule family="ipv4" source address="192.168.33.10" service name="http" accept
    rule family="ipv4" source address="192.168.33.20" service name="ssh" accept

Project

Here are all needed files…

---
- name: host_http_access
  ip: 192.168.33.10
  hostname: http.local
- name: host_ssh_access
  ip: 192.168.33.20
  hostname: ssh.local
- name: host_protected
  ip: 192.168.33.30
  hostname: protected.local

Please add your values for box name/url!

# -*- mode: ruby -*-
# vi: set ft=ruby :

require 'yaml'

Vagrant.require_version ">= 1.9.3"
machines = YAML.load_file('hosts.yml')

Vagrant.configure("2") do |config|

  machines.each do |machines|

    config.vm.define machines["name"] do |machine|

      # disable ssh key update
      machine.ssh.insert_key = false

      # vagrant box name
      machine.vm.box = ""

      # vagrant box url
      machine.vm.box_url = ""

      # disable box update
      machine.vm.box_check_update = false

      # set hostname
      machine.vm.hostname = machines["hostname"]

      # disable synced_folder
      machine.vm.synced_folder ".", "/vagrant", disabled: true

      # 2nd network interface (private)
      machine.vm.network "private_network", ip: machines["ip"]

      # virtualbox settings
      machine.vm.provider :virtualbox do |vb|
        vb.name = machines["name"]
        vb.cpus = 1
        vb.memory = '1024'
        vb.gui = false
      end

      # run shell provisioner
      if machines["name"] == 'host_protected'
        machine.vm.provision "shell", path: "provisioner.sh", :args => "protected"
      else
        machine.vm.provision "shell", path: "provisioner.sh"
      end

    end

  end

end
#! /usr/bin/env bash

# install firewalld and curl
sudo yum install -y firewalld curl

# enable firewalld
sudo systemctl enable firewalld

# start firewalld
sudo systemctl start firewalld

# configure firewalld rich rules
if [ "${1}" == "protected" ]; then
  # install epel-release
  sudo yum install -y epel-release

  # install nginx
  sudo yum install -y nginx

  # enable nginx
  sudo systemctl enable nginx

  # start nginx
  sudo systemctl start nginx

  # change enp0s8 to home zone
  sudo firewall-cmd --zone=home --change-interface=enp0s8 --permanent

  # restart firewalld service
  sudo systemctl restart firewalld

  # remove all services form zone home
  sudo firewall-cmd --zone=home --remove-service dhcpv6-client --permanent
  sudo firewall-cmd --zone=home --remove-service mdns --permanent
  sudo firewall-cmd --zone=home --remove-service samba-client --permanent
  sudo firewall-cmd --zone=home --remove-service ssh --permanent

  # add rich rules
  sudo firewall-cmd --zone=home --add-rich-rule='rule family="ipv4" service name="http" source address="192.168.33.10" accept' --permanent
  sudo firewall-cmd --zone=home --add-rich-rule='rule family="ipv4" service name="ssh" source address="192.168.33.20" accept' --permanent

  # reload firewall
  sudo firewall-cmd --reload
fi

Usage

# start vagrant environment (local)
$ vagrant up

# show status (optional - local)
$ vagrant status
...
Current machine states:

host_http_access          running (virtualbox)
host_ssh_access           running (virtualbox)
host_protected            running (virtualbox)

# ssh into host_http_access (local)
$ vagrant ssh host_http_access

# try http via curl (host_http_access)
$ curl -I http://192.168.33.30

# try ssh (host_http_access)
$ ssh vagrant@192.168.33.30

# ssh into host_ssh_access (local)
$ vagrant ssh host_ssh_access

# try http via curl (host_ssh_access)
$ curl -I http://192.168.33.30

# try ssh (host_ssh_access)
$ ssh vagrant@192.168.33.30

# destroy vagrant environment
$ vagrant destroy -f

Note: before you destroy the vagrant environment, have a look on zones xml files for “host_protected”!

# ssh into host_protected (local)
$ vagrant ssh host_protected

# change to root (host_protected)
$ sudo su -

# change directory
$ cd /etc

# cat actual zone xml files
$ cat firewalld/zones/*.xml

Kickstart Configurator

With kickstart configurator you don`t need to remember the correct syntax of kickstart files. An graphical user interface helps to create or to edit kickstart files on the fly.

Preparation

# install KDE
$ yum groupinstall -y "KDE Plasma Workspaces"

# install kickstart configurator
$ yum install -y system-config-kickstart

# start KDE
$ startx

Note: Of course you can also use Gnome, Cinnamon, MATE or Xfce!

Usage

To start the kickstart configurator search the application “Kickstart” [Applications => System Tools => Kickstart] or run terminal command:

# start kickstart configurator
$ system-config-kickstart
Kickstart-Konfigurator

Note: Via [File => Preview] you can review your current selections before saving.

Validation

After the creation, you should check the kickstart file!

# install pykickstart
$ yum install -y pykickstart

# start validate a kickstart file
$ ksvalidator </path/to/kickstart file>

Install and configure Fail2Ban on CentOS 7

This tutorial presents the minimum SSH protection on CentOS 7 by Fail2Ban (without e-mail).

Preparation

# install epel-release
$ yum install -y epel-release && yum -y update

# install fail2ban packages
$ yum install -y fail2ban fail2ban-systemd

# update selinux-policies (if SELinux)
$ yum update -y selinux-policy*

Configuration

# change directory
$ cd /etc

# check content of 00-systemd.conf
$ cat fail2ban/jail.d/00-systemd.conf
...
[DEFAULT]
backend=systemd
...

# create custom default configuration
$ cp -p fail2ban/jail.conf fail2ban/jail.local

# edit custom default configuration
$ vim fail2ban/jail.local
...
[DEFAULT]
ignoreip = 127.0.0.1/8
bantime  = 3600
maxretry = 3
...

# create custom sshd configuration
$ vim fail2ban/jail.d/sshd.local
...
[sshd]
enabled = true
port = ssh
logpath = %(sshd_log)s
...

Ready for startup

# enable fail2ban
$ systemctl enable fail2ban

# start fail2ban
$ systemctl start fail2ban

Check status

# check status (optional)
$ systemctl status fail2ban

# tail fail2ban logfile (optional)
$ tail -f /var/log/fail2ban.log

# tail secure logfile (optional)
$ tail -f /var/log/secure

# check iptables
$ iptables -L -n

# check status of jails
$ fail2ban-client status

# check status of sshd jail
$ fail2ban-client status sshd

Example

$ fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	347
|  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned:	1
   |- Total banned:	56
   `- Banned IP list:	185.110.132.202

$ whois 185.110.132.202
...
person:         Karamurzov Barasbi
abuse-mailbox:  abusemail@openstack.net.ua
address:        Belize, BE, Kolmo ave 11, apt 901
...

Install Pappy Proxy on CentOS7

Pappy Proxy (Proxy Attack Proxy ProxY) is an open source intercepting proxy for performing web application security tests. It is similar to BurpSuite, ZAP and so on. The benefit is the command line!

Installation

# update system
$ yum update -y

# install epel
$ yum -y install epel-release

# install needed packages
$ yum install -y git python-pip python-devel gcc libffi-devel libxml2 libxml2-devel libxslt libxslt-devel openssl-devel

# clone repository
$ git clone --recursive https://github.com/roglew/pappy-proxy.git

# change directory
$ cd pappy-proxy/

# install pappy-proxy
$ pip install .

# run help
$ pappy --help

# start up in lite mode
$ pappy -l

So pappy is started … we can configure and test it (via lite-mode).

# generate certificates
pappy> gencerts

# test proxy from other terminal
$ curl -x http://localhost:8000 -L http://google.com

# list requests
pappy> ls

# prints the full response to a request
pappy> vfs <id>

# exit pappy-proxy lite mode
pappy> quit

Prepare a project

# create project directory
$ mkdir myProject

# change directory
$ cd myProject

We need to start pappy shortly to create config.json file.

# start pappy-proxy
$ pappy

# exit pappy-proxy
pappy> quit

# show directory content
$ ls -la

# modify settings
$ vim config.json

# start up pappy-proxy
$ pappy
...

Note: By default the proxy is running on port 8000 – bound to localhost. You need to modify the config.json. For more information read the docs.

Explain Shell direct from terminal

Explainshell.com rocks! Nevertheless, you lose time to leave the terminal (open browser, copy-paste). But there is a cool solution from ManKier. All what you need is curl.

Usage

# curl request for whoami
$ curl -Gs "https://www.mankier.com/api/explain/?cols="$(tput cols) --data-urlencode "q=whoami"

# curl request for df -h
$ curl -Gs "https://www.mankier.com/api/explain/?cols="$(tput cols) --data-urlencode "q=df -h"

Simpler usage

With a tiny script it will be more comfortable! Add the following to your .bashrc or .bash_profile (MAC OS X).

# explain.sh begins
explain () {
	if [ "$#" -eq 0 ]; then
		while read  -p "Command: " cmd; do
			curl -Gs "https://www.mankier.com/api/explain/?cols="$(tput cols) --data-urlencode "q=$cmd"
		done
		echo "Bye!"
	elif [ "$#" -eq 1 ]; then
		curl -Gs "https://www.mankier.com/api/explain/?cols="$(tput cols) --data-urlencode "q=$1"
	else
		echo "Usage"
		echo "explain                  interactive mode."
		echo "explain 'cmd -o | ...'   one quoted command to explain it."
	fi
}

Now you can do …

# explain one command
$ explain 'df -h'
...
df(1)
  df displays the amount of disk space available on the file system containing each file name
  argument. If no file name is given, the space available on all currently mounted file systems is
  shown. Disk space is shown in 1K blocks by default, unless the environment variable
  POSIXLY_CORRECT is set, in which case 512-byte blocks are used. If an argument is the absolute
  file name of a disk device node containing a mounted file system, df shows the space available on
  that file system rather than on the file system containing the device node. This version of df
  cannot show the space available on unmounted file systems, because on most kinds of systems doing
  so requires very nonportable intimate knowledge of file system structures.

  -h (-H, --HUMAN-READABLE)
    print sizes in powers of 1024 (e.g., 1023M)

… if you insert only “explain” an interactive mode will started!

SSH jump host example with vagrant

This time shows the tutorial two topics in one. The first topic is: “How an easy to configure SSH host jump”. The 2nd topic is: “Provisioning examples for Vagrant”.

Precondition

  • Vagrant installed
  • VirtualBox installed

File content

---
- name: jump_host
  box: centos/7
  ip: 192.168.100.10
  cpus: 1
  memory: 1024
- name: host_a
  box: centos/7
  ip: 192.168.100.20
  cpus: 1
  memory: 1024
- name: host_b
  box: centos/7
  ip: 192.168.100.30
  cpus: 1
  memory: 1024
Host *
  Compression yes
  AddressFamily inet
  Protocol 2
  ServerAliveInterval 60
  ServerAliveCountMax 30

Host jump_host
  HostName 192.168.x.x
  User testuser

Host host_*
  ProxyCommand ssh jump_host -W %h:%p
  User testuser
# -*- mode: ruby -*-

require 'yaml'

machines = YAML.load_file('machines.yml')
API_VERSION = "2"
KEY_LOCATION = "~/.ssh/id_rsa.pub"

Vagrant.require_version ">= 1.8.1"
Vagrant.configure(API_VERSION) do |config|

  # loop
  machines.each do |machines|
    # vm setup + provisioning
    config.vm.define machines["name"] do |machine|
      machine.vm.box = machines["box"]
      machine.vm.synced_folder ".", "/vagrant", disabled: true
      machine.vm.network "private_network", ip: machines["ip"]

      machine.vm.provider :virtualbox do |vb|
        vb.name = machines["name"]
        vb.cpus = machines["cpus"]
        vb.memory = machines["memory"]
        vb.gui = false
      end

      # provisioning: only jump_host
      if machines["name"] == 'jump_host'
        # prompt for interface
        machine.vm.network "public_network"

        machine.vm.provision "shell", inline: <<-SHELL
          cd /etc && sudo sed -i '/#AllowTcpForwarding yes/c\AllowTcpForwarding yes' ssh/sshd_config
          sudo systemctl restart sshd.service
          cd /etc && sudo grep -q 'host_a' hosts || echo '192.168.100.20  host_a' >> hosts
          cd /etc && sudo grep -q 'host_b' hosts || echo '192.168.100.30  host_b' >> hosts
        SHELL
      end

      # provisioning: all
      machine.vm.provision "file", source: KEY_LOCATION, destination: "/tmp/pub.key"

      machine.vm.provision "shell", inline: <<-SHELL
        sudo useradd testuser
        sudo mkdir /home/testuser/.ssh
        sudo touch /home/testuser/.ssh/authorized_keys
        sudo cat /tmp/pub.key > /home/testuser/.ssh/authorized_keys
        sudo rm -f /tmp/pub.key
        sudo chmod 0700 /home/testuser/.ssh
        sudo chmod 0644 /home/testuser/.ssh/authorized_keys
        sudo chown -R testuser:testuser /home/testuser/.ssh
      SHELL
    end
  end
end

Note: Replace values for HostName “192.168.x.x” by real IP from jump_host and KEY_LOCATION “~/.ssh/id_rsa.pub”!

Steps

# create and goto project
$ mkdir ~/Projects/JumpHostExample && cd ~/Projects/JumpHostExample

# create yaml file
$ vim machines.yml

# create Vagrantfile
$ vim Vagrantfile

# start Vagrant
$ vagrant up

# get IP from jump_host (for config)
$ ssh jump_host -c "ip addr show enp0s9"

# create or edit ssh config
$ vim ~/.ssh/config

# ssh into hosts via jump_host
$ ssh host_a
$ ssh host_b

Minimal CentOS 7 base box with Packer

The title says it, … this tutorial is about Packer, CentOS 7 and Vagrant. After that, you should be able to integrate the creation of Vagrant base boxes into your Build-server. There is on small exception to other – the VirtualBox Guest Additions will be provided via PlugIn! Because other users could may have different versions.

Preconditions

Project structure

$ tree
.
├── Makefile
├── packer.json
├── src
│   ├── Vagrantfile.tpl
│   └── ks.cfg
└── target

File contents

CURRENT_DIR := $(shell pwd)

.PHONY: clean

help:
	@echo "Run make with:"
	@echo " > validate       ...to run packer validation"
	@echo " > build          ...to start packer build"
	@echo " > up             ...to start vagrant"
	@echo " > reload         ...to reload vagrant"
	@echo " > ssh            ...to ssh into vm"
	@echo " > clean          ...to cleanup for next build"

validate:
	packer validate $(CURRENT_DIR)/packer.json

build:
	packer build $(CURRENT_DIR)/packer.json
	cp $(CURRENT_DIR)/src/Vagrantfile.tpl $(CURRENT_DIR)/target/Vagrantfile

up:
	vagrant box add packer/centos7 $(CURRENT_DIR)/target/virtualbox-CentOS-7.box
	cd $(CURRENT_DIR)/target && vagrant up

reload:
	cd $(CURRENT_DIR)/target && vagrant reload

ssh:
	cd $(CURRENT_DIR)/target && vagrant ssh

clean:
	cd $(CURRENT_DIR)/target && vagrant halt
	cd $(CURRENT_DIR)/target && vagrant destroy -f
	rm -fr $(CURRENT_DIR)/builds/
	rm -fr $(CURRENT_DIR)/target/* $(CURRENT_DIR)/target/.* 2> /dev/null
	vagrant box remove packer/centos7
{
  "variables": {
    "file": "http://linuxsoft.cern.ch/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso",
    "checksum": "88c0437f0a14c6e2c94426df9d43cd67",
    "type": "md5",
    "non_gui": "false"
  },
  "builders": [
    {
      "type": "virtualbox-iso",
      "iso_url": "{{ user `file` }}",
      "iso_checksum": "{{ user `checksum` }}",
      "iso_checksum_type": "md5",
      "headless": "{{ user `non_gui` }}",
      "output_directory": "builds",
      "vm_name": "CentOS7_to_Vagrant",
      "guest_os_type": "RedHat_64",
      "disk_size": "10240",
      "vboxmanage": [
        ["modifyvm", "{{.Name}}", "--memory", "2048"],
        ["modifyvm", "{{.Name}}", "--cpus", "2"],
        ["modifyvm", "{{.Name}}", "--audio", "none"],
        ["modifyvm", "{{.Name}}", "--usb", "off"]
      ],
      "http_directory": "src",
      "boot_wait": "5s",
      "boot_command": [
        "<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ks.cfg<enter><wait>"
      ],
      "ssh_username": "vagrant",
      "ssh_password": "vagrant",
      "ssh_port": 22,
      "ssh_wait_timeout": "600s",
      "guest_additions_path": "disable",
      "shutdown_command": "sudo shutdown -h 0"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "sudo yum update -y",
        "sudo rm -rf /tmp/*",
        "sudo rm -f /var/log/wtmp /var/log/btmp ",
        "sudo yum clean all",
        "sudo rm -rf /var/cache/* /usr/share/doc/*",
        "rm -f .bash_history",
        "history -c"
      ]
    }
  ],
  "post-processors": [
    {
      "type": "vagrant",
      "keep_input_artifact": false,
      "compression_level": 9,
      "output": "target/{{.Provider}}-CentOS-7.box"
    }
  ]
}
install
cdrom

lang en_US.UTF-8
keyboard us
timezone UTC

network --bootproto=dhcp
firewall --disabled

rootpw --plaintext packer
user --name=vagrant --password=vagrant
auth --enableshadow --passalgo=sha512 --kickstart
selinux --permissive

text
skipx

clearpart --all --initlabel
zerombr
autopart
bootloader --location=mbr

firstboot --disable
reboot

%packages --instLangs=en_US.utf8 --nobase --ignoremissing --excludedocs
@^minimal
@core

-aic94xx-firmware
-atmel-firmware
-b43-openfwwf
-bfa-firmware
-ipw2100-firmware
-ipw2200-firmware
-ivtv-firmware
-iwl100-firmware
-iwl105-firmware
-iwl135-firmware
-iwl1000-firmware
-iwl2000-firmware
-iwl2030-firmware
-iwl3160-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6000g2b-firmware
-iwl6050-firmware
-iwl7260-firmware
-libertas-usb8388-firmware
-ql2100-firmware
-ql2200-firmware
-ql23xx-firmware
-ql2400-firmware
-ql2500-firmware
-rt61pci-firmware
-rt73usb-firmware
-xorg-x11-drv-ati-firmware
-zd1211-firmware
%end

%post --log=/root/ks.log
SEE NEXT PICTURE!!!! The security settings of my provider does not allow this content!
%end

ks content

# -*- mode: ruby -*-

Vagrant.require_version ">= 1.8.1"

Vagrant.configure("2") do |config|

  config.vm.box = "packer/centos7"
  config.vm.box_url = "target/virtualbox-CentOS-7.box"
  config.vm.synced_folder ".", "/vagrant", disabled: true

  config.vm.provider "virtualbox" do |vb|
    vb.name = "CentOS-7"
    vb.cpus = "2"
    vb.memory = "2048"
    vb.gui = false
  end

end

Usage

# run packer build (via make)
$ make build

# run vagrant up (via make)
$ make run

# run vagrant reload (via make)
$ make reload

# run vagrant ssh (via make)
$ make ssh

# destroy everything (via make)
$ make clean