Skip to content
This repository has been archived by the owner on Dec 6, 2023. It is now read-only.

Handlers in MiTM server for detection of PII in HTTP and HTTPS traffic #100

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Handlers in MiTM server for detection of PII in HTTP and HTTPS traffic #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 9 additions & 10 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,23 @@
#nogotofail
# nogotofail


Nogotofail is a network security testing tool designed to help developers and
security researchers spot and fix weak TLS/SSL connections and sensitive
cleartext traffic on devices and applications in a flexible, scalable, powerful way.
It includes testing for common SSL certificate verification issues, HTTPS and TLS/SSL
library bugs, SSL and STARTTLS stripping issues, cleartext issues, and more.
Nogotofail is a network security testing tool designed to help developers and security researchers spot and fix weak TLS/SSL connections and sensitive cleartext traffic on devices and applications in a flexible, scalable, powerful way.
It includes testing for common SSL certificate verification issues, HTTPS and TLS/SSL library bugs, SSL and STARTTLS stripping issues, cleartext issues, personally identifiable information (PII) disclosure issues and more.

##Design
See [docs/pii_analysis.md](docs/pii_analysis.md) for an overview of PII detection features.

## Design
Nogotofail is composed of an on-path network MiTM and optional clients for the devices being tested.
See [docs/design.md](docs/design.md) for the overview and design goals of nogotofail.

##Dependencies
## Dependencies
Nogotofail depends only on Python 2.7 and pyOpenSSL>=0.13. The MiTM is designed to work on Linux
machines and the transparent traffic capture modes are Linux specific and require iptables as well.

Additionally the Linux client depends on [psutil](https://pypi.python.org/pypi/psutil).

##Getting started
## Getting started
See [docs/getting_started.md](docs/getting_started.md) for setup and a walkthrough of nogotofail.

##Discussion
## Discussion
For discussion please use our [nogotofail Google Group](https://groups.google.com/forum/#!forum/nogotofail).
169 changes: 169 additions & 0 deletions docs/create_tls_proxy_cert.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
# Creating a Certificate to performing MitM TLS Proxying

nogotofail-pii can be configured to operate as a man-in-the-middle (MitM) TLS proxy and inspect encrypted (HTTPS) traffic for PII. The method described here is using a self-signed certificate and requires two certificate chain files (PEM format) to be created:

- **ca-chain-cleartext.key.cert.pem** certificate chain file contains the two certificate public key files (root and intermediate) and the intermediate certificate private key (the private key is unencrypted).
- **ca-chain.cert.pem** certificate chain file contains the two certificate public key files (for the root and intermediate certificates).

The recommended procedure below and is based on the method used here: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html

## 1. Setting up the Certificate Authority

### a. Preparation

Create a folder to store the Certificate Authority (CA) files.

``` mkdir /root/ca ```

Text files index.txt and serial are setup to act as a kind of flat file database to keep track of signed certificates.
```
cd /root/ca
mkdir certs crl newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial
```
An OpenSSL configuration file openssl.cnf needs to be created for the CA. The format used is based on the following instructions: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html#prepare-the-configuration-file

### b. Creating the root key

The root key is encrypted using AES 256-bit encryption and a strong password should be used.
```
cd /root/ca
openssl genrsa -aes256 -out private/ca.key.pem 4096
```
Enter pass phrase for ca.key.pem: secretpassword
Verifying - Enter pass phrase for ca.key.pem: secretpassword

```chmod 400 private/ca.key.pem```

### c. Create the root certificate

The root certficate (ca.cert.pem) is created using the root key (ca.key.pem). The expiry date of the root certificate was set to approx 20 years (7300) days.
```
cd /root/ca
openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem

Enter pass phrase for ca.key.pem: secretpassword
You are about to be asked to enter information that will be incorporated
into your certificate request.

Country Name (2 letter code) [XX]:AU
State or Province Name []:Australia
Locality Name []:
Organization Name []:PII MitM Ltd
Organizational Unit Name []:PII MitM Ltd Certificate Authority
Common Name []:pii.mitm.ca
Email Address []:

chmod 444 certs/ca.cert.pem
```
The root certificate should be verified using the instructions at: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html#verify-the-root-certificate

## 2. Create the TLS man-in-the-middle certificate key pair

A new certificate will be created to perform the TLS man-in-the-middle (MitM) inspection between the mobile device and server. The certificate keys will be generated from the root CA.

### a. Preparation

The new certificate files will be stored in a different directory. The suggested folder name is tlsmitm and should be created under the CA folder:

```mkdir /root/ca/tlsmitm```

Create the folders needed for this certificate using:
```
cd /root/ca/tlsmitm
mkdir certs crl csr newcerts private
chmod 700 private
touch index.txt
echo 1000 > serial
```
Add a crlnumber file to the intermediate CA directory tree to keep track of certificate revocation lists.

```echo 1000 > /root/ca/intermediate/crlnumber```

Copy the intermediate CA configuration file to /root/ca/mitm/openssl.cnf. The following five options need to be changed for this certificate:
```
[ CA_default ]
dir = /root/ca/tlsmitm
private_key = $dir/private/tlsmitm.key.pem
certificate = $dir/certs/tlsmitm.cert.pem
crl = $dir/crl/tlsmitm.crl.pem
policy = policy_loose
```

### b. Create the certificate key

Create the tls mitm key tls.pii.mitm.ca. The intermediate key is encrypted using AES 256-bit encryption and a strong password.
```
cd /root/ca
openssl genrsa -aes256 -out tlsmitm/private/tlsmitm.key.pem 4096

Enter pass phrase for tlsmitm.key.pem: secretpassword
Verifying - Enter pass phrase for tlsmitm.key.pem: secretpassword

chmod 400 tlsmitm/private/tlsmitm.key.pem
```

### c. Create the TLS MitM certificate

The TLS MitM key is used to create a certificate signing request (CSR). The details should generally match the root CA, except the Common Name which must be different.
```
cd /root/ca
openssl req -config tlsmitm/openssl.cnf -new -sha256 -key tlsmitm/private/tlsmitm.key.pem -out tlsmitm/csr/tlsmitm.csr.pem

Enter pass phrase for tlsmitm.key.pem: secretpassword
You are about to be asked to enter information that will be incorporated
into your certificate request.
-----
Country Name (2 letter code) [XX]:AU
State or Province Name []:Australia
Locality Name []:
Organization Name []:PII MitM Ltd
Organizational Unit Name []:PII MitM Ltd Certificate Authority
Common Name []:tls.pii.mitm.ca
Email Address []:
```
To create the TLS MitM certificate, use the root CA with the v3_intermediate_ca extension to sign the intermediate CSR.
```
cd /root/ca
openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in tlsmitm/csr/tlsmitm.csr.pem -out tlsmitm/certs/tlsmitm.cert.pem

Enter pass phrase for ca.key.pem: secretpassword
Sign the certificate? [y/n]: y

chmod 444 tlsmitm/certs/tlsmitm.cert.pem
```
To verify the details of this certificate are correct use the instructions at: https://jamielinux.com/docs/openssl-certificate-authority/create-the-intermediate-pair.html#verify-the-intermediate-certificate

## 3. Setting up the TLS MitM certificates

### a. Creating the certificate chain file

To create the certificate chain file ca-chain.cert.pem containing the two certificate public key files (root and TLS MitM) the two files are concatinated:
```
cat tlsmitm/certs/tlsmitm.cert.pem certs/ca.cert.pem > tlsmitm/certs/ca-chain.cert.pem
chmod 444 tlsmitm/certs/ca-chain.cert.pem
```

### b. Creating the certificate chain file with TLS MitM private key

Firstly, an unencrypted version of the TLS MitM private key needs to be created by removing the passphrase:
```
openssl rsa -in tlsmitm/private/tlsmitm.key.pem -out tlsmitm/private/tlsmitm.unencrypted.key.pem
```
Note. You will prompted to enter the passphrase.

To create the certificate chain file ca-chain-cleartext.key.cert.pem containing the two certificate public key files (root and TLS MitM) and the intermediate certificate private key (private key unencrypted), the private key and certificate chain file (form part a.) need to be concatinated:
```
cat tlsmitm/private/tlsmitm.unencrypted.key.pem tlsmitm/certs/ca-chain.cert.pem > tlsmitm/certs/ca-chain-cleartext.cert.pem
chmod 444 tlsmitm/certs/ca-chain-cleartext.cert.pem
```

### c. Installing the TLS MitM certificates

The two PEM files need to be installed before TLS MitM functionality can be enabled.

The file containing the two public keys ca-chain.cert.pem needs to be installed in the Android device's certificate key store (under the Settings > Security > Trusted Credentials option).

The file containing the two public keys and private key ca-chain-cleartext.cert.pem must be copied onto the server in the /opt/nogotofail folder.
66 changes: 66 additions & 0 deletions docs/gce/_update_dev.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
#!/bin/sh

set -e

# Directory paths used for nogotofail.
INSTALL_DIR=/opt/nogotofail
CONFIG_DIR=/etc/nogotofail
LOG_DIR=/var/log/nogotofail

# Stop the nogotofail-mitm and other associated services if they're running.
if (ps ax | grep -v grep | grep nogotofail-mitm > /dev/null) then
sudo /etc/init.d/nogotofail-mitm stop
fi
if (ps ax | grep -v grep | grep dnsmasq > /dev/null) then
sudo /etc/init.d/dnsmasq stop
fi
if (ps ax | grep -v grep | grep openvpn > /dev/null) then
sudo /etc/init.d/openvpn stop
fi
# Remove Python files and compiled versions i.e. *.py and *.pyc files.
# TODO: Find a more elegant method for uninstalling a Python program.
#rm -rf $INSTALL_DIR
#rm -rf $CONFIG_DIR
#rm -rf $LOG_DIR
find $INSTALL_DIR -type f -name '*.py' -delete
find $INSTALL_DIR -type f -name '*.pyc' -delete

# Install toolchain dependencies
sudo apt-get update
sudo apt-get -y upgrade
#sudo apt-get -y install patch make gcc libssl-dev python-openssl liblzo2-dev libpam-dev

# Install OpenVPN and dnsmasq
#sudo apt-get -y install openvpn dnsmasq

# Build and install a patched version of OpenVPN.
# This is needed because the OpenVPN 2.3.x still does not properly handle
# floating clients (those whose source IP address as seen by the server changes
# from time to time) which is a regular occurrence in the mobile world.
# OpenVPN 2.4 might ship with proper support out of the box. In that case, this
# kludge can be removed.
#./build_openvpn.sh

# Build and install a patched version of dnsmasq.
# This is needed because GCE does not support IPv6. We thus blackhole IPv6
# traffic from clients so that they are forced to use IPv4. However, default
# DNS servers will still resolve hostnames to IPv6 addresses causing clients to
# attempt IPv6. To avoid clients attempting IPv6, we run a patched dnsmasq DNS
# server which empties AAAA records thus causing clients to go for A records
# which provide IPv4 addresses.
#./build_dnsmasq.sh

# Set up OpenVPN server
#sudo ./setup_openvpn.sh

# Set up the MiTM daemons
sudo ./setup_mitm.sh

# Move dev mitm.conf file into /etc/nogotofail directory
sudo cp /home/michael/noseyp_setup/mitm.conf /etc/nogotofail/mitm.conf

# Restart all the relevant daemons
sudo /etc/init.d/dnsmasq start
sudo /etc/init.d/openvpn start
#sudo /etc/init.d/nogotofail-mitm stop || true
sudo /etc/init.d/nogotofail-mitm start
13 changes: 12 additions & 1 deletion docs/gce/mitm.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,22 @@
#verbose=True
#port=8080
#attacks=selfsigned invalidhostname
attacks=httpspii
#data=httpdetection httpauthdetection
data=httppii

probability=0.5
probability=0.2
debug=True

serverssl=/etc/nogotofail/mitm_controller_cert_and_key.pem
logfile=/var/log/nogotofail/mitm.log
eventlogfile=/var/log/nogotofail/mitm.event
trafficfile=/var/log/nogotofail/mitm.traffic

[nogotofail.pii]
[email protected]
ip_address=55.66.77.88
email = [email protected]
first_name = joe
last_name = blogs
postal_address = "1 Long Road, Towns-ville"
63 changes: 63 additions & 0 deletions docs/gce/update.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#!/bin/sh

set -e

# Directory paths used for nogotofail.
INSTALL_DIR=/opt/nogotofail
CONFIG_DIR=/etc/nogotofail
LOG_DIR=/var/log/nogotofail

# Stop the nogotofail-mitm and other associated services if they're running.
if (ps ax | grep -v grep | grep nogotofail-mitm > /dev/null) then
sudo /etc/init.d/nogotofail-mitm stop
fi
if (ps ax | grep -v grep | grep dnsmasq > /dev/null) then
sudo /etc/init.d/dnsmasq stop
fi
if (ps ax | grep -v grep | grep openvpn > /dev/null) then
sudo /etc/init.d/openvpn stop
fi
# Remove Python files and compiled versions i.e. *.py and *.pyc files.
# TODO: Find a more elegant method for uninstalling a Python program.
#rm -rf $INSTALL_DIR
#rm -rf $CONFIG_DIR
#rm -rf $LOG_DIR
find $INSTALL_DIR -type f -name '*.py' -delete
find $INSTALL_DIR -type f -name '*.pyc' -delete

# Install toolchain dependencies
sudo apt-get update
sudo apt-get -y upgrade
#sudo apt-get -y install patch make gcc libssl-dev python-openssl liblzo2-dev libpam-dev

# Install OpenVPN and dnsmasq
#sudo apt-get -y install openvpn dnsmasq

# Build and install a patched version of OpenVPN.
# This is needed because the OpenVPN 2.3.x still does not properly handle
# floating clients (those whose source IP address as seen by the server changes
# from time to time) which is a regular occurrence in the mobile world.
# OpenVPN 2.4 might ship with proper support out of the box. In that case, this
# kludge can be removed.
#./build_openvpn.sh

# Build and install a patched version of dnsmasq.
# This is needed because GCE does not support IPv6. We thus blackhole IPv6
# traffic from clients so that they are forced to use IPv4. However, default
# DNS servers will still resolve hostnames to IPv6 addresses causing clients to
# attempt IPv6. To avoid clients attempting IPv6, we run a patched dnsmasq DNS
# server which empties AAAA records thus causing clients to go for A records
# which provide IPv4 addresses.
#./build_dnsmasq.sh

# Set up OpenVPN server
#sudo ./setup_openvpn.sh

# Set up the MiTM daemon
sudo ./setup_mitm.sh

# Restart all the relevant daemons
sudo /etc/init.d/dnsmasq start
sudo /etc/init.d/openvpn start
#sudo /etc/init.d/nogotofail-mitm stop || true
sudo /etc/init.d/nogotofail-mitm start
Loading