Improvements to recent malicious document analysis

Update Guide

Improvements to recent malicious document analysis:

• Added XLMMacroDeobfuscator analyzer, refer #196 thanks to @0ssigeno
• Updated oletools to last available changes

Other:

• updated black to 20.8b1 and little fix in the docs

Unpacme + whoisxml API + checkdmarc analyzer + Fix VT2

Update Guide

• 3 new analyzers which can be used out of the box:
  • UnpacMe_EXE_Unpacker: UnpacMe is an automated malware unpacking service. (Thanks to @0ssigeno)
  • CheckDMARC: checdmarc provides SPF and DMARC DNS records validator for domains. (Thanks to @goodlandsecurity)
  • Whoisxmlapi: Fetch WHOIS record data, of a domain name, an IP address, or an email address. (Thanks to @tamthaitu)
• Some fixes to Cymru Malware and VT2 analyzers.
• Now you or your organization can get paid support/extra features/custom integrations for IntelOwl via xscode platform. Details.

[Patch] fixed version number - Added SpeakEasy, upgraded Capa and updated docs

This patch allows to download the most recent docker image of IntelOwl. Previous version was downloading the old (v.1.5.1) docker image.

Please see v1.6.0 for release details.

Upgrade guide

Added SpeakEasy, upgraded Capa and updated docs

• added new analyzer for FireEye speakeasy
• updated FireEye Capa to 1.1.0
• updated docs, including instructions for Remnux users and a new "How to use pyintelowl" video

[Patched] IntelX phonebook API + Dynamic Analyzer's Conf.

Patch after v1.5.0.

• Fixed runtime_configuration JSON serialization bug when requesting file scan.

IntelX phonebook API + Dynamic Analyzer's Conf. + more..

This release contains a bug that was fixed in v1.5.1. We recommend cloning the master branch.

Features:

• Ability to pass a JSON field runtime_configuration for dynamic configuration per scan request. Demo GIF.
• IntelligenceX's phonebook API for observables.
• Increased JWT token lifetime for webapp. (Ref.).

Breaking Changes:

• Moved ldap_config.py under configuration/ directory. If you were using LDAP before this release, please refer the updated docs.

Fixes:

• Updates and fixes to: Doc_info, PE_Info, VirusTotal v3 and Shodan_Honeyscore analyzers.
• Added migration files for DB.

Quark Engine, Pulsedive, Python 3.7, GKE Deployment docs

Upgrade Guide

• Inbuilt Integration for Pulsedive analyzer for IP, URL, Domain and Hash observables. Works without API key with rate limit of 30 requests/minute.
• Inbuilt integration for Integrated Quark-engine for APKs - An Obfuscation-Neglect Android Malware Scoring System.
• Increase max_length for file_mimetype column. Thanks to @skygrip for the report.
• Index the fields that are used in ask_analysis_availability for faster fetching.
• Update LDAP documentation, add section about GKE deployments.
• Fixed: is_test issue in _docker_run. Thanks to @colbyprior.
• Fixed: active_dns now returns proper result.
• The base docker image is now based on Python 3.7.
• Refactor test cases/classes to reduce duplicate code.

Elastic Search + LDAP + groups/permissions + specific docker tags

Read at release v1.3.0 for details.

• The images on hub.docker.com are now tagged with the same version number as the GitHub release tags.

Elastic Search + LDAP + groups/permissions + some fixes

• Added the ability to leverage Django's permissions system to organize users into groups, allow/restrict different permissions to different groups, mark particular jobs as private so they are not visible to other users. Docs on how to use this.
• Added support for Elastic Search. If elastic search is enabled, all analysis are auto synced between the postgreSQL database and the Elastic Search index. Docs.
  • As a bonus, a preconfigured Kibana configuration (having some helpful visualizations and dashboard) is also provided which can be imported as a "Saved Object" into Kibana.
• Added basic support for LDAP authentication mechanism. Docs.
• Fixed: CUCKOO_API_KEY variable missing from env_file_app_template.
• Increased observable_name field's max_length to support upto 512 chars. Up from the previous 128 limit. (Issue #144)
• Cleaner log messages throughout analyzer related functions.
• Various other under-the-hood improvements, fixes and optimizations.

For users upgrading to v1.3.0 from prior versions - Please follow the steps described here.

Capa + Box-JS + APKiD + logging issue fixed (Stable Release)

• Integrations for analyzers: Capa by FireEye, Box-JS and APKiD. All of these are available as optional analyzers which can be enabled as per user's need.
• Fix for issue #129. Now supports max length of 128 chars so SHA256/512 hashes can be scanned.
• Refactoring and various bug fixes in Docker based optional analyzers, especially the logging issue.
• changed flush_expired_tokens cron schedule from every 6h to 3h. So the user's DB is not cluttered.
• Cleaner log messages throughout analyzer related functions.

Note: To update the web-client, please run docker pull intelowlproject/intelowl_ng:latest before starting Intel Owl.