Skip to content

Commit

Permalink
Separate loki-upgrader
Browse files Browse the repository at this point in the history
  • Loading branch information
Florian Roth committed Jun 12, 2017
1 parent e58867f commit 63f4313
Show file tree
Hide file tree
Showing 5 changed files with 313 additions and 124 deletions.
45 changes: 34 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,29 @@ Loki currently includes the following IOCs:

Loki is the new generic scanner that combines most of the features from my recently published scanners: [ReginScanner](https://github.com/Neo23x0/ReginScanner) and [SkeletonKeyScanner](https://github.com/Neo23x0/SkeletonKeyScanner).

## Update

Since version 0.21.0 LOKI includes a separate updater tool named `loki-upgrader.exe` or `loki-upgrader.py`.

```
usage: loki-upgrader.py [-h] [-l log-file] [--sigsonly] [--progonly] [--nolog]
[--debug]
Loki - Upgrader
optional arguments:
-h, --help show this help message and exit
-l log-file Log file
--sigsonly Update the signatures only
--progonly Update the program files only
--nolog Don't write a local log file
--debug Debug output
```

It allows to update the compiled loki.exe for Windows and the signature-base sources.

When running `loki.exe --update` it will create an new upgrader process and exits LOKI in order to replace the `loki.exe` with the newer one, which would be locked otherwise.

## Requirements

No requirements if you use the compiled EXE.
Expand Down Expand Up @@ -106,9 +129,9 @@ c:\Python27\Scripts\pip.exe install pylzma
[-w warning-level] [-n notice-level] [--printAll]
[--allreasons] [--noprocscan] [--nofilescan] [--noindicator]
[--reginfs] [--dontwait] [--intense] [--csv] [--onlyrelevant]
[--nolog] [--update] [--debug]
[--nolog] [--update] [--debug]

Loki - Simple IOC Scanner
Loki - Simple IOC Scanner

optional arguments:
-h, --help show this help message and exit
Expand Down Expand Up @@ -136,11 +159,11 @@ c:\Python27\Scripts\pip.exe install pylzma

## Signature and IOCs

Since version 0.15 the Yara signatures reside in the sub-repository [signature-base](https://github.com/Neo23x0/signature-base). You can just download the LOKI release ZIP archive and run LOKI once to download the 'signature-base' sub repository with all the signatures.
Since version 0.15 the Yara signatures reside in the sub-repository [signature-base](https://github.com/Neo23x0/signature-base). You can just download the LOKI release ZIP archive and run LOKI once to download the 'signature-base' sub repository with all the signatures.

The IOC files for hashes and filenames are stored in the './signature-base/iocs' folder. All '.yar' files placed in the './signature-base/yara' folder will be initialized together with the rule set that is already included. Use the 'score' value to define the level of the message upon a signature match.
The IOC files for hashes and filenames are stored in the './signature-base/iocs' folder. All '.yar' files placed in the './signature-base/yara' folder will be initialized together with the rule set that is already included. Use the 'score' value to define the level of the message upon a signature match.

You can add hash, c2 and filename IOCs by adding files to the './signature-base/iocs' subfolder. All hash IOCs and filename IOC files must be in the format used by LOKI (see the default files). The files must have the strings "hash", "filename" or "c2" in their name to get pulled during initialization.
You can add hash, c2 and filename IOCs by adding files to the './signature-base/iocs' subfolder. All hash IOCs and filename IOC files must be in the format used by LOKI (see the default files). The files must have the strings "hash", "filename" or "c2" in their name to get pulled during initialization.

For Hash IOCs (divided by newline; hash type is detected automatically)
```
Expand All @@ -154,7 +177,7 @@ Filename as Regex;Description [Reference]

# User-Defined Scan Excludes

Since version v0.16.2 LOKI supports the definition of user-defined excludes via "excludes.cfg" in the new "./config" folder. Each line represents a regular expression thats gets applied to the full file path during the directory walk. This way you can exclude certain directories regardless of their drive name, file extensions in certain folders and all files and directories that belong to a product that is sensitive to antivirus scanning.
Since version v0.16.2 LOKI supports the definition of user-defined excludes via "excludes.cfg" in the new "./config" folder. Each line represents a regular expression thats gets applied to the full file path during the directory walk. This way you can exclude certain directories regardless of their drive name, file extensions in certain folders and all files and directories that belong to a product that is sensitive to antivirus scanning.

The '''exclude.cfg''' looks like this:

Expand All @@ -173,7 +196,7 @@ The '''exclude.cfg''' looks like this:
# Matches: /var/log/test.log
# Not Matches: /var/log/test.gz
#

# Useful examples
\\Ntfrs\\
\\Ntds\\
Expand All @@ -185,9 +208,9 @@ The '''exclude.cfg''' looks like this:

Since version v0.10 LOKI includes various threat intel receivers using the public APIs of these services to retrieve and store the IOCs in a format that LOKI understands. It is no problem if these indicators overlap with the ones already included. Loki uses a filename regex or hash only once. (no preformance impact)

The threat intel receivers have also been moved to the [signature-base](https://github.com/Neo23x0/signature-base) sub repository with version 0.15 and can be found in "./signature-base/threatintel".
The threat intel receivers have also been moved to the [signature-base](https://github.com/Neo23x0/signature-base) sub repository with version 0.15 and can be found in "./signature-base/threatintel".

Provide your API key via ```-k APIKEY``` or set it in the script header.
Provide your API key via ```-k APIKEY``` or set it in the script header.

## Open Threat Exchange (OTX) Receiver

Expand Down Expand Up @@ -273,10 +296,10 @@ This will create a `loki.exe` in the subfolder `./loki/dist`.
To include the msvcr100.dll to improve the target os compatibility change the line in the file `./loki/loki.spec` that contains `a.bianries,` to the following:

a.binaries + [('msvcr100.dll', 'C:\Windows\System32\msvcr100.dll', 'BINARY')],

# Use LOKI on Mac OS X

- Download Yara sources from [here](https://github.com/plusvic/yara/releases/tag/v3.4.0)
- Download Yara sources from [here](https://github.com/VirusTotal/yara/releases)
- Install openssl (brew install openssl, then sudo cp -r /usr/local/Cellar/openssl/1.0.2h_1/include /usr/local)
- ./build.sh
- sudo make install
Expand Down
2 changes: 1 addition & 1 deletion lib/helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -268,4 +268,4 @@ def getAgeString(filePath):
timestring = "CREATED: %s MODIFIED: %s ACCESSED: %s" % ( time.ctime(ctime), time.ctime(mtime), time.ctime(atime) )
except Exception,e:
timestring = "CREATED: not_available MODIFIED: not_available ACCESSED: not_available"
return timestring
return timestring
56 changes: 35 additions & 21 deletions lib/lokilogger.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,14 @@ class LokiLogger():
only_relevant = False
debug = False

def __init__(self, no_log_file, log_file, hostname, csv, only_relevant, debug):
def __init__(self, no_log_file, log_file, hostname, csv, only_relevant, debug, caller):
self.no_log_file = no_log_file
self.log_file = log_file
self.hostname = hostname
self.csv = csv
self.only_relevant = only_relevant
self.debug = debug
self.caller = caller

# Colorization ----------------------------------------------------
init()
Expand Down Expand Up @@ -148,26 +149,39 @@ def log_to_file(self, message, mes_type):
print "Cannot print to log file {0}".format(self.log_file)

def print_welcome(self):
print Back.GREEN + " ".ljust(79) + Back.BLACK + Fore.GREEN

print " __ ____ __ ______ "
print " / / / __ \/ //_/ _/ "
print " / /__/ /_/ / ,< _/ / "
print " /____/\____/_/|_/___/ "
print " ________ _____ ____ "
print " / _/ __ \/ ___/ / __/______ ____ ___ ___ ____ "
print " _/ // /_/ / /__ _\ \/ __/ _ `/ _ \/ _ \/ -_) __/ "
print " /___/\____/\___/ /___/\__/\_,_/_//_/_//_/\__/_/ "

print Fore.WHITE
print " (C) Florian Roth"
print " June 2017"
print " Version %s" % __version__
print " "
print " DISCLAIMER - USE AT YOUR OWN RISK"
print " "
print Back.GREEN + " ".ljust(79) + Back.BLACK
print Fore.WHITE+''+Back.BLACK

if self.caller == 'main':
print Back.GREEN + " ".ljust(79) + Back.BLACK + Fore.GREEN

print " __ ____ __ ______ "
print " / / / __ \/ //_/ _/ "
print " / /__/ /_/ / ,< _/ / "
print " /____/\____/_/|_/___/ "
print " ________ _____ ____ "
print " / _/ __ \/ ___/ / __/______ ____ ___ ___ ____ "
print " _/ // /_/ / /__ _\ \/ __/ _ `/ _ \/ _ \/ -_) __/ "
print " /___/\____/\___/ /___/\__/\_,_/_//_/_//_/\__/_/ "

print Fore.WHITE
print " (C) Florian Roth"
print " June 2017"
print " Version %s" % __version__
print " "
print " DISCLAIMER - USE AT YOUR OWN RISK"
print " "
print Back.GREEN + " ".ljust(79) + Back.BLACK
print Fore.WHITE+''+Back.BLACK

else:
print " "
print Back.GREEN + " ".ljust(79) + Back.BLACK + Fore.GREEN

print " "
print " LOKI UPGRADER "

print " "
print Back.GREEN + " ".ljust(79) + Back.BLACK
print Fore.WHITE + '' + Back.BLACK

def getSyslogTimestamp():
date_obj = datetime.datetime.utcnow()
Expand Down
213 changes: 213 additions & 0 deletions loki-upgrader.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
#!/usr/bin/env python
# -*- coding: iso-8859-1 -*-
# -*- coding: utf-8 -*-
#
# LOKI Upgrader

import urllib2
import json
import zipfile
import shutil
from StringIO import StringIO
import os
import argparse
import traceback
from sys import platform as _platform

# Win32 Imports
if _platform == "win32":
try:
import wmi
import win32api
from win32com.shell import shell
except Exception, e:
platform = "linux" # crazy guess


from lib.lokilogger import *

class LOKIUpdater(object):

UPDATE_URL_SIGS = "https://github.com/Neo23x0/signature-base/archive/master.zip"
UPDATE_URL_LOKI = "https://api.github.com/repos/Neo23x0/Loki/releases/latest"

def __init__(self, debug, logger, application_path):
self.debug = debug
self.logger = logger
self.application_path = application_path

def update_signatures(self):
try:

# Downloading current repository
try:
self.logger.log("INFO", "Downloading %s ..." % self.UPDATE_URL_SIGS)
response = urllib2.urlopen(self.UPDATE_URL_SIGS)
except Exception, e:
if self.debug:
traceback.print_exc()
self.logger.log("ERROR", "Error downloading the signature database - check your Internet connection")
sys.exit(1)

# Preparations
try:
sigDir = os.path.join(self.application_path, './signature-base/')
for outDir in ['', 'iocs', 'yara', 'misc']:
fullOutDir = os.path.join(sigDir, outDir)
if not os.path.exists(fullOutDir):
os.makedirs(fullOutDir)
except Exception, e:
if self.debug:
traceback.print_exc()
self.logger.log("ERROR", "Error while creating the signature-base directories")
sys.exit(1)

# Read ZIP file
try:
zipUpdate = zipfile.ZipFile(StringIO(response.read()))
for zipFilePath in zipUpdate.namelist():
sigName = os.path.basename(zipFilePath)
if zipFilePath.endswith("/"):
continue
self.logger.log("DEBUG", "Extracting %s ..." % zipFilePath)
if "/iocs/" in zipFilePath and zipFilePath.endswith(".txt"):
targetFile = os.path.join(sigDir, "iocs", sigName)
elif "/yara/" in zipFilePath and zipFilePath.endswith(".yar"):
targetFile = os.path.join(sigDir, "yara", sigName)
elif "/misc/" in zipFilePath and zipFilePath.endswith(".txt"):
targetFile = os.path.join(sigDir, "misc", sigName)
else:
continue

# New file
if not os.path.exists(targetFile):
self.logger.log("INFO", "New signature file: %s" % sigName)

# Extract file
source = zipUpdate.open(zipFilePath)
target = file(targetFile, "wb")
with source, target:
shutil.copyfileobj(source, target)

except Exception, e:
if self.debug:
traceback.print_exc()
self.logger.log("ERROR", "Error while extracting the signature files from the download package")
sys.exit(1)

except Exception, e:
if self.debug:
traceback.print_exc()
return False
return True


def update_loki(self):
try:

# Downloading the info for latest release
try:
self.logger.log("INFO", "Checking location of latest release %s ..." % self.UPDATE_URL_LOKI)
response_info = urllib2.urlopen(self.UPDATE_URL_LOKI)
data = json.load(response_info)
# Get download URL
zip_url = data['assets'][0]['browser_download_url']
self.logger.log("INFO", "Downloading latest release %s ..." % zip_url)
response_zip = urllib2.urlopen(zip_url)
except Exception, e:
if self.debug:
traceback.print_exc()
self.logger.log("ERROR", "Error downloading the loki update - check your Internet connection")
sys.exit(1)

# Read ZIP file
try:
zipUpdate = zipfile.ZipFile(StringIO(response_zip.read()))
for zipFilePath in zipUpdate.namelist():
if zipFilePath.endswith("/") or "/config/" in zipFilePath:
continue

source = zipUpdate.open(zipFilePath)
targetFile = "/".join(zipFilePath.split("/")[1:])

self.logger.log("INFO", "Extracting %s ..." %targetFile)

try:
target = file(targetFile, "wb")
with source, target:
shutil.copyfileobj(source, target)
except Exception,e:
self.logger.log("ERROR", "Cannot extract %s" % targetFile)
if self.debug:
traceback.print_exc()

except Exception, e:
if self.debug:
traceback.print_exc()
self.logger.log("ERROR", "Error while extracting the signature files from the download package")
sys.exit(1)

except Exception, e:
if self.debug:
traceback.print_exc()
return False
return True


def get_application_path():
try:
if getattr(sys, 'frozen', False):
application_path = os.path.dirname(os.path.realpath(sys.executable))
else:
application_path = os.path.dirname(os.path.realpath(__file__))
if "~" in application_path and platform == "windows":
# print "Trying to translate"
# print application_path
application_path = win32api.GetLongPathName(application_path)
#if args.debug:
# logger.log("DEBUG", "Application Path: %s" % application_path)
return application_path
except Exception, e:
print "Error while evaluation of application path"
traceback.print_exc()


if __name__ == '__main__':

# Parse Arguments
parser = argparse.ArgumentParser(description='Loki - Upgrader')
parser.add_argument('-l', help='Log file', metavar='log-file', default='loki-upgrade.log')
parser.add_argument('--sigsonly', action='store_true', help='Update the signatures only', default=False)
parser.add_argument('--progonly', action='store_true', help='Update the program files only', default=False)
parser.add_argument('--nolog', action='store_true', help='Don\'t write a local log file', default=False)
parser.add_argument('--debug', action='store_true', default=False, help='Debug output')
parser.add_argument('--detached', action='store_true', default=False, help=argparse.SUPPRESS)

args = parser.parse_args()

# Computername
if _platform == "win32":
t_hostname = os.environ['COMPUTERNAME']
else:
t_hostname = os.uname()[1]

# Logger
logger = LokiLogger(args.nolog, args.l, t_hostname, False, False, args.debug, caller='upgrader')

# Update Loki
updater = LOKIUpdater(args.debug, logger, get_application_path())

# Updating LOKI
if not args.sigsonly:
logger.log("INFO", "Updating LOKI ...")
updater.update_loki()
if not args.progonly:
logger.log("INFO", "Updating Signatures ...")
updater.update_signatures()

logger.log("INFO", "Update complete")

if args.detached:
logger.log("INFO", "Press any key to return ...")

sys.exit(0)
Loading

0 comments on commit 63f4313

Please sign in to comment.