-
Notifications
You must be signed in to change notification settings - Fork 418
Commit
New Handlers httppii and httpspii added to detect PII in HTTP and HTTPS traffic. Also added documentation describing how to use the PII handlers.
- Loading branch information
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,24 +1,23 @@ | ||
#nogotofail | ||
This comment has been minimized.
Sorry, something went wrong. |
||
# nogotofail | ||
This comment has been minimized.
Sorry, something went wrong. |
||
|
||
|
||
Nogotofail is a network security testing tool designed to help developers and | ||
security researchers spot and fix weak TLS/SSL connections and sensitive | ||
cleartext traffic on devices and applications in a flexible, scalable, powerful way. | ||
It includes testing for common SSL certificate verification issues, HTTPS and TLS/SSL | ||
library bugs, SSL and STARTTLS stripping issues, cleartext issues, and more. | ||
Nogotofail is a network security testing tool designed to help developers and security researchers spot and fix weak TLS/SSL connections and sensitive cleartext traffic on devices and applications in a flexible, scalable, powerful way. | ||
It includes testing for common SSL certificate verification issues, HTTPS and TLS/SSL library bugs, SSL and STARTTLS stripping issues, cleartext issues, personally identifiable information (PII) disclosure issues and more. | ||
|
||
This comment has been minimized.
Sorry, something went wrong. |
||
##Design | ||
See [docs/pii_analysis.md](docs/pii_analysis.md) for an overview of PII detection features. | ||
This comment has been minimized.
Sorry, something went wrong. |
||
|
||
## Design | ||
This comment has been minimized.
Sorry, something went wrong. |
||
Nogotofail is composed of an on-path network MiTM and optional clients for the devices being tested. | ||
See [docs/design.md](docs/design.md) for the overview and design goals of nogotofail. | ||
|
||
##Dependencies | ||
This comment has been minimized.
Sorry, something went wrong. |
||
## Dependencies | ||
This comment has been minimized.
Sorry, something went wrong. |
||
Nogotofail depends only on Python 2.7 and pyOpenSSL>=0.13. The MiTM is designed to work on Linux | ||
machines and the transparent traffic capture modes are Linux specific and require iptables as well. | ||
|
||
Additionally the Linux client depends on [psutil](https://pypi.python.org/pypi/psutil). | ||
|
||
##Getting started | ||
## Getting started | ||
See [docs/getting_started.md](docs/getting_started.md) for setup and a walkthrough of nogotofail. | ||
|
||
##Discussion | ||
## Discussion | ||
For discussion please use our [nogotofail Google Group](https://groups.google.com/forum/#!forum/nogotofail). |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,169 @@ | ||
# Creating a Certificate to performing MitM TLS Proxying | ||
This comment has been minimized.
Sorry, something went wrong. |
||
|
||
nogotofail-pii can be configured to operate as a man-in-the-middle (MitM) TLS proxy and inspect encrypted (HTTPS) traffic for PII. The method described here is using a self-signed certificate and requires two certificate chain files (PEM format) to be created: | ||
|
||
- **ca-chain-cleartext.key.cert.pem** certificate chain file contains the two certificate public key files (root and intermediate) and the intermediate certificate private key (the private key is unencrypted). | ||
- **ca-chain.cert.pem** certificate chain file contains the two certificate public key files (for the root and intermediate certificates). | ||
|
||
The recommended procedure below and is based on the method used here: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html | ||
|
||
## 1. Setting up the Certificate Authority | ||
|
||
### a. Preparation | ||
|
||
Create a folder to store the Certificate Authority (CA) files. | ||
|
||
``` mkdir /root/ca ``` | ||
|
||
Text files index.txt and serial are setup to act as a kind of flat file database to keep track of signed certificates. | ||
``` | ||
cd /root/ca | ||
mkdir certs crl newcerts private | ||
chmod 700 private | ||
touch index.txt | ||
echo 1000 > serial | ||
``` | ||
An OpenSSL configuration file openssl.cnf needs to be created for the CA. The format used is based on the following instructions: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html#prepare-the-configuration-file | ||
|
||
### b. Creating the root key | ||
|
||
The root key is encrypted using AES 256-bit encryption and a strong password should be used. | ||
``` | ||
cd /root/ca | ||
openssl genrsa -aes256 -out private/ca.key.pem 4096 | ||
``` | ||
Enter pass phrase for ca.key.pem: secretpassword | ||
Verifying - Enter pass phrase for ca.key.pem: secretpassword | ||
|
||
```chmod 400 private/ca.key.pem``` | ||
|
||
### c. Create the root certificate | ||
|
||
The root certficate (ca.cert.pem) is created using the root key (ca.key.pem). The expiry date of the root certificate was set to approx 20 years (7300) days. | ||
``` | ||
cd /root/ca | ||
openssl req -config openssl.cnf -key private/ca.key.pem -new -x509 -days 7300 -sha256 -extensions v3_ca -out certs/ca.cert.pem | ||
Enter pass phrase for ca.key.pem: secretpassword | ||
You are about to be asked to enter information that will be incorporated | ||
into your certificate request. | ||
Country Name (2 letter code) [XX]:AU | ||
State or Province Name []:Australia | ||
Locality Name []: | ||
Organization Name []:PII MitM Ltd | ||
Organizational Unit Name []:PII MitM Ltd Certificate Authority | ||
Common Name []:pii.mitm.ca | ||
Email Address []: | ||
chmod 444 certs/ca.cert.pem | ||
``` | ||
The root certificate should be verified using the instructions at: https://jamielinux.com/docs/openssl-certificate-authority/create-the-root-pair.html#verify-the-root-certificate | ||
|
||
## 2. Create the TLS man-in-the-middle certificate key pair | ||
|
||
A new certificate will be created to perform the TLS man-in-the-middle (MitM) inspection between the mobile device and server. The certificate keys will be generated from the root CA. | ||
|
||
### a. Preparation | ||
|
||
The new certificate files will be stored in a different directory. The suggested folder name is tlsmitm and should be created under the CA folder: | ||
|
||
```mkdir /root/ca/tlsmitm``` | ||
|
||
Create the folders needed for this certificate using: | ||
``` | ||
cd /root/ca/tlsmitm | ||
mkdir certs crl csr newcerts private | ||
chmod 700 private | ||
touch index.txt | ||
echo 1000 > serial | ||
``` | ||
Add a crlnumber file to the intermediate CA directory tree to keep track of certificate revocation lists. | ||
|
||
```echo 1000 > /root/ca/intermediate/crlnumber``` | ||
|
||
Copy the intermediate CA configuration file to /root/ca/mitm/openssl.cnf. The following five options need to be changed for this certificate: | ||
``` | ||
[ CA_default ] | ||
dir = /root/ca/tlsmitm | ||
private_key = $dir/private/tlsmitm.key.pem | ||
certificate = $dir/certs/tlsmitm.cert.pem | ||
crl = $dir/crl/tlsmitm.crl.pem | ||
policy = policy_loose | ||
``` | ||
|
||
### b. Create the certificate key | ||
|
||
Create the tls mitm key tls.pii.mitm.ca. The intermediate key is encrypted using AES 256-bit encryption and a strong password. | ||
``` | ||
cd /root/ca | ||
openssl genrsa -aes256 -out tlsmitm/private/tlsmitm.key.pem 4096 | ||
Enter pass phrase for tlsmitm.key.pem: secretpassword | ||
Verifying - Enter pass phrase for tlsmitm.key.pem: secretpassword | ||
chmod 400 tlsmitm/private/tlsmitm.key.pem | ||
``` | ||
|
||
### c. Create the TLS MitM certificate | ||
|
||
The TLS MitM key is used to create a certificate signing request (CSR). The details should generally match the root CA, except the Common Name which must be different. | ||
``` | ||
cd /root/ca | ||
openssl req -config tlsmitm/openssl.cnf -new -sha256 -key tlsmitm/private/tlsmitm.key.pem -out tlsmitm/csr/tlsmitm.csr.pem | ||
Enter pass phrase for tlsmitm.key.pem: secretpassword | ||
You are about to be asked to enter information that will be incorporated | ||
into your certificate request. | ||
----- | ||
Country Name (2 letter code) [XX]:AU | ||
State or Province Name []:Australia | ||
Locality Name []: | ||
Organization Name []:PII MitM Ltd | ||
Organizational Unit Name []:PII MitM Ltd Certificate Authority | ||
Common Name []:tls.pii.mitm.ca | ||
Email Address []: | ||
``` | ||
To create the TLS MitM certificate, use the root CA with the v3_intermediate_ca extension to sign the intermediate CSR. | ||
``` | ||
cd /root/ca | ||
openssl ca -config openssl.cnf -extensions v3_intermediate_ca -days 3650 -notext -md sha256 -in tlsmitm/csr/tlsmitm.csr.pem -out tlsmitm/certs/tlsmitm.cert.pem | ||
Enter pass phrase for ca.key.pem: secretpassword | ||
Sign the certificate? [y/n]: y | ||
chmod 444 tlsmitm/certs/tlsmitm.cert.pem | ||
``` | ||
To verify the details of this certificate are correct use the instructions at: https://jamielinux.com/docs/openssl-certificate-authority/create-the-intermediate-pair.html#verify-the-intermediate-certificate | ||
|
||
## 3. Setting up the TLS MitM certificates | ||
|
||
### a. Creating the certificate chain file | ||
|
||
To create the certificate chain file ca-chain.cert.pem containing the two certificate public key files (root and TLS MitM) the two files are concatinated: | ||
``` | ||
cat tlsmitm/certs/tlsmitm.cert.pem certs/ca.cert.pem > tlsmitm/certs/ca-chain.cert.pem | ||
chmod 444 tlsmitm/certs/ca-chain.cert.pem | ||
``` | ||
|
||
### b. Creating the certificate chain file with TLS MitM private key | ||
|
||
Firstly, an unencrypted version of the TLS MitM private key needs to be created by removing the passphrase: | ||
``` | ||
openssl rsa -in tlsmitm/private/tlsmitm.key.pem -out tlsmitm/private/tlsmitm.unencrypted.key.pem | ||
``` | ||
Note. You will prompted to enter the passphrase. | ||
|
||
To create the certificate chain file ca-chain-cleartext.key.cert.pem containing the two certificate public key files (root and TLS MitM) and the intermediate certificate private key (private key unencrypted), the private key and certificate chain file (form part a.) need to be concatinated: | ||
``` | ||
cat tlsmitm/private/tlsmitm.unencrypted.key.pem tlsmitm/certs/ca-chain.cert.pem > tlsmitm/certs/ca-chain-cleartext.cert.pem | ||
chmod 444 tlsmitm/certs/ca-chain-cleartext.cert.pem | ||
``` | ||
|
||
### c. Installing the TLS MitM certificates | ||
|
||
The two PEM files need to be installed before TLS MitM functionality can be enabled. | ||
|
||
The file containing the two public keys ca-chain.cert.pem needs to be installed in the Android device's certificate key store (under the Settings > Security > Trusted Credentials option). | ||
|
||
The file containing the two public keys and private key ca-chain-cleartext.cert.pem must be copied onto the server in the /opt/nogotofail folder. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
#!/bin/sh | ||
|
||
set -e | ||
|
||
# Directory paths used for nogotofail. | ||
INSTALL_DIR=/opt/nogotofail | ||
CONFIG_DIR=/etc/nogotofail | ||
LOG_DIR=/var/log/nogotofail | ||
|
||
# Stop the nogotofail-mitm and other associated services if they're running. | ||
if (ps ax | grep -v grep | grep nogotofail-mitm > /dev/null) then | ||
sudo /etc/init.d/nogotofail-mitm stop | ||
fi | ||
if (ps ax | grep -v grep | grep dnsmasq > /dev/null) then | ||
sudo /etc/init.d/dnsmasq stop | ||
fi | ||
if (ps ax | grep -v grep | grep openvpn > /dev/null) then | ||
sudo /etc/init.d/openvpn stop | ||
fi | ||
# Remove Python files and compiled versions i.e. *.py and *.pyc files. | ||
# TODO: Find a more elegant method for uninstalling a Python program. | ||
#rm -rf $INSTALL_DIR | ||
#rm -rf $CONFIG_DIR | ||
#rm -rf $LOG_DIR | ||
find $INSTALL_DIR -type f -name '*.py' -delete | ||
find $INSTALL_DIR -type f -name '*.pyc' -delete | ||
|
||
# Install toolchain dependencies | ||
sudo apt-get update | ||
sudo apt-get -y upgrade | ||
#sudo apt-get -y install patch make gcc libssl-dev python-openssl liblzo2-dev libpam-dev | ||
|
||
# Install OpenVPN and dnsmasq | ||
#sudo apt-get -y install openvpn dnsmasq | ||
|
||
# Build and install a patched version of OpenVPN. | ||
# This is needed because the OpenVPN 2.3.x still does not properly handle | ||
# floating clients (those whose source IP address as seen by the server changes | ||
# from time to time) which is a regular occurrence in the mobile world. | ||
# OpenVPN 2.4 might ship with proper support out of the box. In that case, this | ||
# kludge can be removed. | ||
#./build_openvpn.sh | ||
|
||
# Build and install a patched version of dnsmasq. | ||
# This is needed because GCE does not support IPv6. We thus blackhole IPv6 | ||
# traffic from clients so that they are forced to use IPv4. However, default | ||
# DNS servers will still resolve hostnames to IPv6 addresses causing clients to | ||
# attempt IPv6. To avoid clients attempting IPv6, we run a patched dnsmasq DNS | ||
# server which empties AAAA records thus causing clients to go for A records | ||
# which provide IPv4 addresses. | ||
#./build_dnsmasq.sh | ||
|
||
# Set up OpenVPN server | ||
#sudo ./setup_openvpn.sh | ||
|
||
# Set up the MiTM daemons | ||
sudo ./setup_mitm.sh | ||
|
||
# Move dev mitm.conf file into /etc/nogotofail directory | ||
sudo cp /home/michael/noseyp_setup/mitm.conf /etc/nogotofail/mitm.conf | ||
|
||
# Restart all the relevant daemons | ||
sudo /etc/init.d/dnsmasq start | ||
sudo /etc/init.d/openvpn start | ||
#sudo /etc/init.d/nogotofail-mitm stop || true | ||
sudo /etc/init.d/nogotofail-mitm start |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -8,11 +8,22 @@ | |
#verbose=True | ||
#port=8080 | ||
#attacks=selfsigned invalidhostname | ||
attacks=httpspii | ||
#data=httpdetection httpauthdetection | ||
data=httppii | ||
|
||
probability=0.5 | ||
probability=0.2 | ||
debug=True | ||
|
||
serverssl=/etc/nogotofail/mitm_controller_cert_and_key.pem | ||
logfile=/var/log/nogotofail/mitm.log | ||
eventlogfile=/var/log/nogotofail/mitm.event | ||
trafficfile=/var/log/nogotofail/mitm.traffic | ||
|
||
[nogotofail.pii] | ||
[email protected] | ||
ip_address=55.66.77.88 | ||
email = [email protected] | ||
first_name = joe | ||
last_name = blogs | ||
postal_address = "1 Long Road, Towns-ville" |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
#!/bin/sh | ||
|
||
set -e | ||
|
||
# Directory paths used for nogotofail. | ||
INSTALL_DIR=/opt/nogotofail | ||
CONFIG_DIR=/etc/nogotofail | ||
LOG_DIR=/var/log/nogotofail | ||
|
||
# Stop the nogotofail-mitm and other associated services if they're running. | ||
if (ps ax | grep -v grep | grep nogotofail-mitm > /dev/null) then | ||
sudo /etc/init.d/nogotofail-mitm stop | ||
fi | ||
if (ps ax | grep -v grep | grep dnsmasq > /dev/null) then | ||
sudo /etc/init.d/dnsmasq stop | ||
fi | ||
if (ps ax | grep -v grep | grep openvpn > /dev/null) then | ||
sudo /etc/init.d/openvpn stop | ||
fi | ||
# Remove Python files and compiled versions i.e. *.py and *.pyc files. | ||
# TODO: Find a more elegant method for uninstalling a Python program. | ||
#rm -rf $INSTALL_DIR | ||
#rm -rf $CONFIG_DIR | ||
#rm -rf $LOG_DIR | ||
find $INSTALL_DIR -type f -name '*.py' -delete | ||
find $INSTALL_DIR -type f -name '*.pyc' -delete | ||
|
||
# Install toolchain dependencies | ||
sudo apt-get update | ||
sudo apt-get -y upgrade | ||
#sudo apt-get -y install patch make gcc libssl-dev python-openssl liblzo2-dev libpam-dev | ||
|
||
# Install OpenVPN and dnsmasq | ||
#sudo apt-get -y install openvpn dnsmasq | ||
|
||
# Build and install a patched version of OpenVPN. | ||
# This is needed because the OpenVPN 2.3.x still does not properly handle | ||
# floating clients (those whose source IP address as seen by the server changes | ||
# from time to time) which is a regular occurrence in the mobile world. | ||
# OpenVPN 2.4 might ship with proper support out of the box. In that case, this | ||
# kludge can be removed. | ||
#./build_openvpn.sh | ||
|
||
# Build and install a patched version of dnsmasq. | ||
# This is needed because GCE does not support IPv6. We thus blackhole IPv6 | ||
# traffic from clients so that they are forced to use IPv4. However, default | ||
# DNS servers will still resolve hostnames to IPv6 addresses causing clients to | ||
# attempt IPv6. To avoid clients attempting IPv6, we run a patched dnsmasq DNS | ||
# server which empties AAAA records thus causing clients to go for A records | ||
# which provide IPv4 addresses. | ||
#./build_dnsmasq.sh | ||
|
||
# Set up OpenVPN server | ||
#sudo ./setup_openvpn.sh | ||
|
||
# Set up the MiTM daemon | ||
sudo ./setup_mitm.sh | ||
|
||
# Restart all the relevant daemons | ||
sudo /etc/init.d/dnsmasq start | ||
sudo /etc/init.d/openvpn start | ||
#sudo /etc/init.d/nogotofail-mitm stop || true | ||
sudo /etc/init.d/nogotofail-mitm start |
1 comment
on commit 407e535
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pull
pull