Skip to main content

Protecting a debian head end

 

The following configuration protects a debian head end from defined threats. This is based on this script. Dependencies are ipset and iprange.

/usr/local/sbin/updatethreatblock.sh

#!/usr/bin/env bash
#
# usage updatethreatblock.sh <configuration file>
# eg: updatethreatblock.sh /etc/ipset-threatblock/ipset-threatblock.conf
#
function exists() { command -v "$1" >/dev/null 2>&1 ; }

if [[ -z "$1" ]]; then
  echo "Error: please specify a configuration file, e.g. $0 /etc/ipset-threatblock/ipset-threatblock.conf"
  exit 1
fi

# shellcheck source=ipset-threatblock.conf
if ! source "$1"; then
  echo "Error: can't load configuration file $1"
  exit 1
fi

if ! exists curl && exists egrep && exists grep && exists ipset && exists iptables && exists sed && exists sort && exists wc ; then
  echo >&2 "Error: searching PATH fails to find executables among: curl egrep grep ipset iptables sed sort wc"
  exit 1
fi

DO_OPTIMIZE_CIDR=no
if exists iprange && [[ ${OPTIMIZE_CIDR:-yes} != no ]]; then
  DO_OPTIMIZE_CIDR=yes
fi

if [[ ! -d $(dirname "$IP_BLACKLIST") || ! -d $(dirname "$IP_BLACKLIST_RESTORE") ]]; then
  echo >&2 "Error: missing directory(s): $(dirname "$IP_BLACKLIST" "$IP_BLACKLIST_RESTORE"|sort -u)"
  exit 1
fi

if [ -f "$IP_BLACKLIST_EXCEPTIONS" ]; then
EXCEPTIONS_TMP=$(mktemp)
for exception in $(sed -r -e 's/\s*#.*$//;/^$/d;/^(0.0.0.0|10.|127.|172.1[6-9].|172.2[0-9].|172.3[0-1].|192.168.|22[4-9].|23[0-9].)/d' "$IP_BLACKLIST_EXCEPTIONS")
do
exception_array+=( "$exception" )
echo $exception >> $EXCEPTIONS_TMP
done
fi

# create the ipset if needed (or abort if does not exists and FORCE=no)
if ! ipset list -n|command grep -q "$IPSET_BLACKLIST_NAME"; then
  if [[ ${FORCE:-no} != yes ]]; then
    echo >&2 "Error: ipset does not exist yet, add it using:"
    echo >&2 "# ipset create $IPSET_BLACKLIST_NAME -exist hash:net family inet hashsize ${HASHSIZE:-16384} maxelem ${MAXELEM:-65536}"
    exit 1
  fi
  if ! ipset create "$IPSET_BLACKLIST_NAME" -exist hash:net family inet hashsize "${HASHSIZE:-16384}" maxelem "${MAXELEM:-65536}"; then
    echo >&2 "Error: while creating the initial ipset"
    exit 1
  fi
fi

# create the iptables binding if needed (or abort if does not exists and FORCE=no)
if ! iptables -nvL INPUT|command grep -q "match-set $IPSET_BLACKLIST_NAME"; then
  # we may also have assumed that INPUT rule n°1 is about packets statistics (traffic monitoring)
  if [[ ${FORCE:-no} != yes ]]; then
    echo >&2 "Error: iptables does not have the needed ipset INPUT rule, add it using:"
    echo >&2 "# iptables -I INPUT ${IPTABLES_IPSET_RULE_NUMBER:-1} -m set --match-set $IPSET_BLACKLIST_NAME src -j DROP"
    exit 1
  fi
  if ! iptables -I INPUT "${IPTABLES_IPSET_RULE_NUMBER:-1}" -m set --match-set "$IPSET_BLACKLIST_NAME" src -j DROP; then
    echo >&2 "Error: while adding the --match-set ipset rule to iptables"
    exit 1
  fi
fi

IP_BLACKLIST_TMP=$(mktemp)
for i in "${BLACKLISTS[@]}"
do
  IP_TMP=$(mktemp)
  (( HTTP_RC=$(curl -L -A "blacklist-update/script/github" --connect-timeout 10 --max-time 10 -o "$IP_TMP" -s -w "%{http_code}" "$i") ))
  if (( HTTP_RC == 200 || HTTP_RC == 302 || HTTP_RC == 0 )); then # "0" because file:/// returns 000
    command grep -Po '^(?:\d{1,3}\.){3}\d{1,3}(?:/\d{1,2})?' "$IP_TMP" | sed -r 's/^0*([0-9]+)\.0*([0-9]+)\.0*([0-9]+)\.0*([0-9]+)$/\1.\2.\3.\4/' >> "$IP_BLACKLIST_TMP"
    [[ ${VERBOSE:-yes} == yes ]] && echo -n "."
  elif (( HTTP_RC == 503 )); then
    echo -e "\\nUnavailable (${HTTP_RC}): $i"
  else
    echo >&2 -e "\\nWarning: curl returned HTTP response code $HTTP_RC for URL $i"
  fi
  rm -f "$IP_TMP"
done

# sort -nu does not work as expected
sed -r -e '/^(0\.0\.0\.0|10\.|127\.|172\.1[6-9]\.|172\.2[0-9]\.|172\.3[0-1]\.|192\.168\.|22[4-9]\.|23[0-9]\.)/d' "$IP_BLACKLIST_TMP"|sort -n|sort -mu >| "$IP_BLACKLIST"
if [[ ${DO_OPTIMIZE_CIDR} == yes ]]; then
  if [[ ${VERBOSE:-no} == yes ]]; then
    echo -e "\\nAddresses before CIDR optimization: $(wc -l "$IP_BLACKLIST" | cut -d' ' -f1)"
  fi
  < "$IP_BLACKLIST" iprange --optimize - > "$IP_BLACKLIST_TMP" 2>/dev/null
  if [[ ${#exception_array[@]} > 0 ]]; then
    echo "Allowing for ${#exception_array[@]} exclusions from blacklist"
    echo "Addresses before removing exclusions: $(wc -l "$IP_BLACKLIST_TMP" | cut -d' ' -f1)"
    IP_BLACKLIST_WITH_EXCEPT_TMP=$(mktemp)
    iprange "$IP_BLACKLIST_TMP" --except "$EXCEPTIONS_TMP" > "$IP_BLACKLIST_WITH_EXCEPT_TMP" 2>/dev/null
    cp "$IP_BLACKLIST_WITH_EXCEPT_TMP" "$IP_BLACKLIST_TMP"
  fi
  if [[ ${VERBOSE:-no} == yes ]]; then
    echo "Addresses after CIDR optimization:  $(wc -l "$IP_BLACKLIST_TMP" | cut -d' ' -f1)"
  fi
  cp "$IP_BLACKLIST_TMP" "$IP_BLACKLIST"
fi

rm -f "$IP_BLACKLIST_TMP"

# family = inet for IPv4 only
cat >| "$IP_BLACKLIST_RESTORE" <<EOF
create $IPSET_TMP_BLACKLIST_NAME -exist hash:net family inet hashsize ${HASHSIZE:-16384} maxelem ${MAXELEM:-65536}
create $IPSET_BLACKLIST_NAME -exist hash:net family inet hashsize ${HASHSIZE:-16384} maxelem ${MAXELEM:-65536}
EOF

# can be IPv4 including netmask notation
# IPv6 ? -e "s/^([0-9a-f:./]+).*/add $IPSET_TMP_BLACKLIST_NAME \1/p" \ IPv6
sed -rn -e '/^#|^$/d' \
  -e "s/^([0-9./]+).*/add $IPSET_TMP_BLACKLIST_NAME \\1/p" "$IP_BLACKLIST" >> "$IP_BLACKLIST_RESTORE"

cat >> "$IP_BLACKLIST_RESTORE" <<EOF
swap $IPSET_BLACKLIST_NAME $IPSET_TMP_BLACKLIST_NAME
destroy $IPSET_TMP_BLACKLIST_NAME
EOF

ipset -file  "$IP_BLACKLIST_RESTORE" restore

if [[ ${VERBOSE:-no} == yes ]]; then
  echo
  echo "Threatblock addresses found: $(wc -l "$IP_BLACKLIST" | cut -d' ' -f1)"
fi

/usr/local/sbin/startubl.sh

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
cd /usr/local/sbin
date > /var/log/threatblock/tb-`date +%d%H`.run
/usr/local/sbin/updatethreatblock.sh /etc/ipset-threatblock/ipset-threatblock.conf
date >> /var/log/threatblock/tb-`date +%d%H`.run

/usr/local/sbin/rebootubl.sh

#!/bin/bash
# Threat intelligence
/sbin/ipset restore < /etc/ipset-threatblock/ip-threatblock.restore
/sbin/iptables -I INPUT   1 -i enp4s0f1 -m set --match-set threatblock src -j DROP -m comment --comment "threat intel"
/sbin/iptables -I FORWARD 1 -i enp4s0f1 -m set --match-set threatblock src -j DROP -m comment --comment "threat intel"
/sbin/iptables -I FORWARD 1 -o enp4s0f1 -m set --match-set threatblock src -j DROP -m comment --comment "threat intel"

crontab

#
# m h  dom mon dow   command
@reboot         sleep 180 && /usr/local/sbin/rebootubl.sh
@daily          /usr/local/sbin/startubl.sh

/etc/ipset-threatblock/ipset-threatblock.conf

IPSET_BLACKLIST_NAME=threatblock # change it if it collides with a pre-existing ipset list
IPSET_TMP_BLACKLIST_NAME=${IPSET_BLACKLIST_NAME}-tmp

# ensure the directory for IP_BLACKLIST/IP_BLACKLIST_RESTORE exists (it won't be created automatically)
IP_BLACKLIST_RESTORE=/etc/ipset-threatblock/ip-threatblock.restore
IP_BLACKLIST=/etc/ipset-threatblock/ip-threatblock.list

IP_BLACKLIST_EXCEPTIONS=/etc/ipset-threatblock/threatblock.exceptions

VERBOSE=yes # probably set to "no" for cron jobs, default to yes
FORCE=yes # will create the ipset-iptable binding if it does not already exist
let IPTABLES_IPSET_RULE_NUMBER=1 # if FORCE is yes, the number at which place insert the ipset-match rule (default to 1)

# Sample (!) list of URLs for IP blacklists. Currently, only IPv4 is supported in this script, everything else will be filtered.
BLACKLISTS=(
    "file:///etc/ipset-blacklist/ip-threatblock-custom.list" # optional, for your personal nemeses (no typo, plural)
    "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1" # Project Honey Pot Directory of Dictionary Attacker IPs
    "https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1"  # TOR Exit Nodes
    "http://danger.rulez.sk/projects/bruteforceblocker/blist.php" # BruteForceBlocker IP List
    "https://www.spamhaus.org/drop/drop.lasso" # Spamhaus Don't Route Or Peer List (DROP)
    "https://cinsscore.com/list/ci-badguys.txt" # C.I. Army Malicious IP List
    "https://lists.blocklist.de/lists/all.txt" # blocklist.de attackers
    "https://blocklist.greensnow.co/greensnow.txt" # GreenSnow
    "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/firehol_level1.netset" # Firehol Level 1
    "https://raw.githubusercontent.com/firehol/blocklist-ipsets/master/stopforumspam_7d.ipset" # Stopforumspam via Firehol
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_shadowserver.txt" # Shadow server
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_shodan.txt" # Shodan
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_sogou.txt" # Sogou search engine
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_evil.txt" # Hostile IPs
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_internet_cens.txt" # Internet census
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_diverseenvironment.txt" # Diverse environment
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_netsysres.txt" # Net Systems Research
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_rwth-aachen.txt" # AAChen
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_onyphe.txt" # onephe
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_stretchoid.txt" # Strechoid
    "https://gitlab.com/ohisee/block-shodan-stretchoid-census/raw/master/pf_table_openportsstats.txt" # OpenPortStats
    # "http://ipverse.net/ipblocks/data/countries/xx.zone" # Ban an entire country, see http://ipverse.net/ipblocks/data/countries/
)
MAXELEM=131072

/etc/ipset-threatblock/ip-threatblock-custom.list

104.152.52.0/24
# Proxylogon attacking IPs via Bluehexagon
86.105.18.116
89.34.111.11
182.18.152.105
103.77.192.219
104.140.114.110
104.248.49.97
104.250.191.110
108.61.246.56
149.28.14.163
157.230.221.198
161.35.1.207
161.35.1.225
165.232.154.116
167.99.168.251
167.99.239.29
185.250.151.72
192.81.208.169
203.160.69.66
211.56.98.146
5.2.69.14
5.254.43.18
80.92.205.81
91.192.103.43
104.248.49.97
13.231.174.2
161.35.45.41
194.87.69.35
45.155.205.225
45.76.110.29
45.77.252.175
112.66.255.71
139.59.56.239
161.35.51.41
161.35.76.1
188.166.162.201
77.61.36.169
161.129.64.124
46.30.188.60
139.162.123.108
194.68.44.19
172.105.18.72
77.83.159.15
185.125.231.175
185.224.83.137
107.173.83.123
201.162.109.184
68.2.82.62
182.215.181.200
45.15.9.45
141.164.40.193
172.105.87.139

/etc/ipset-threatblock/threatblock.exceptions

13.107.42.2
13.107.42.11
23.227.38.65
100.64.0.0/10
This article was originally published over on LinkedIn: Protecting a debian head end 

Comments

Popular posts from this blog

Why Madge Networks, the token-ring company, went titsup

There I was shooting the breeze with an old mate. The conversation turned to why Madge Networks which I wrote about here went titsup. My analysis is that Madge Networks had a solution and decided to go out and find a problem. They deferred to more incorrect strategic technology choices. The truth of the matter is that when something goes titsup, its not because of one reason only, but a myriad of them all contributing to the negative consequence. There are the immediate or visual ones, which are underpinned by intermediate ones and finally after digging right down, there are the root causes. There is never a singular root cause for anything but I'll present my opinion and encourage everyone else to chip in. All of them together are more likely the reason the company went titsup. As far as technology brainfarts go there is no better example than Kodak . They invented the digital camera that killed them. However, they were so focused on milking people in their leg

Flawed "ITIL aligned"​ Incident Management

Many "ITIL aligned" service desk tools have flawed incident management. The reason is that incidents are logged with a time association and some related fields to type in some gobbledygook. The expanded incident life cycle is not enforced and as a result trending and problem management is not possible. Here is a fictitious log of an incident at PFS, a financial services company, which uses CGTSD, an “ITIL-aligned” service desk tool. Here is the log of an incident record from this system: Monday, 12 August: 09:03am (Bob, the service desk guy): Alice (customer in retail banking) phoned in. Logged an issue. Unable to assist over the phone (there goes our FCR), will escalate to second line. 09:04am (Bob, the service desk guy): Escalate the incident to Charles in second line support. 09:05am (Charles, technical support): Open incident. 09:05am (Charles, technical support): Delayed incident by 1 day. Tuesday, 13 August: 10:11am (Charles, technical support): Phoned Alice.

The best social media requires no batteries

  Today it is all about social media such as whatsapp, facebook, twitter or even LinkedIn. However, the best social media is Craic. No, it is not to be confused with substance abuse. Let me explain. Often when people meet around a braai , dinner table, or share either a pot of beer, bottle of wine, a cup of tea or a mug of coffee a conversation is likely to happen. This conversation is invariably about things and is referred to as Craic. And it is best reinforced with a good bottle of whisky (typically an Irish one, which would be known as a whiskey). Now talking about why some people call it whisky, and other whiskey is good Craic. Craic is often a discussion about things that spark a debate or lead to an extensive and prolonged engagement. Things in our world are objects that exist or have existed for a long time period. We typically assume that things in our modern world have only been around a short time period but invariably many