Skip to content

Commit

Permalink
Merge pull request #2291 from intelowlproject/develop
Browse files Browse the repository at this point in the history
* Fix migration signal

Signed-off-by: 0ssigeno <[email protected]>

* Fix kwargs

Signed-off-by: 0ssigeno <[email protected]>

* Fail if in production mode

Signed-off-by: 0ssigeno <[email protected]>

* Validated disable with 429

Signed-off-by: 0ssigeno <[email protected]>

* Fix is_from_org information

Signed-off-by: 0ssigeno <[email protected]>

* Fix greynoise api requirement

Signed-off-by: 0ssigeno <[email protected]>

* fixes #1758 - greynoise labs analyzer (#2225)

* greynoise labs analyzer

* fix

* update migration numbers

* update maximum tlp

* fix

---------

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* removed scanner issue template

* fixes #1663 - Abusix analyzer (#2233)

* fixes #1663 - abusix analyzer

* fix

* updated migration numbers

---------

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* Bump uwsgitop from 0.11 to 0.12 in /requirements (#2237)

Bumps [uwsgitop](https://github.com/xrmx/uwsgitop) from 0.11 to 0.12.
- [Commits](xrmx/uwsgitop@v0.11...v0.12)

---
updated-dependencies:
- dependency-name: uwsgitop
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix ses email

Signed-off-by: 0ssigeno <[email protected]>

* Jobs now requires both analyzers AND connectors to be empty

Signed-off-by: 0ssigeno <[email protected]>

* Fixes for quad9

Signed-off-by: 0ssigeno <[email protected]>

* Removed comment on enable back

Signed-off-by: 0ssigeno <[email protected]>

* HTTPerror should not have the entire traceback

Signed-off-by: 0ssigeno <[email protected]>

* Trying to unlock the git process deleting the lock file

Signed-off-by: 0ssigeno <[email protected]>

* Fix ingestor periodic task

Signed-off-by: 0ssigeno <[email protected]>

* Fix retrieval of the task

Signed-off-by: 0ssigeno <[email protected]>

* Fix ingestor python class retrieval

Signed-off-by: 0ssigeno <[email protected]>

* Fix test

Signed-off-by: 0ssigeno <[email protected]>

* Fix tests

Signed-off-by: 0ssigeno <[email protected]>

* Fixes

Signed-off-by: 0ssigeno <[email protected]>

* specified version of nginx

* Priority mgmt (#2242)

* Priority mgmt

Signed-off-by: 0ssigeno <[email protected]>

* Blake

Signed-off-by: 0ssigeno <[email protected]>

* Docs

Signed-off-by: 0ssigeno <[email protected]>

* Changed order of cache table

Signed-off-by: 0ssigeno <[email protected]>

* Manage case where cache it is not initialized

Signed-off-by: 0ssigeno <[email protected]>

* Fixes

Signed-off-by: 0ssigeno <[email protected]>

* Blake

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* Bump pillow from 10.2.0 to 10.3.0 in /requirements (#2240)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.2.0...10.3.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:production
...

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* fixed rescan observable (#2243)

* Added pivot buttons (#2239)

* added pivot buttons

* fixes

* linter

* docs

* updated doc

* changes

---------

Co-authored-by: Matteo Lodi <[email protected]>

* added tlp info icon (#2246)

* adjusted 'Investigation Overview' button (#2244)

* adjusted investigation button

* fix

* Allow filtering for disabled boolean

Signed-off-by: 0ssigeno <[email protected]>

* Fix cache

Signed-off-by: 0ssigeno <[email protected]>

* Fix env variables + healthcheck

Signed-off-by: 0ssigeno <[email protected]>

* Fix visualizable table (#2249)

* Fix images url

* Fixed table images using absolute url

* improved frontend validation in visualizer framework

* modified JobIsRunningAlert component (#2256)

* changed JobIsRunningAlert

* prettier

* mocked flow

* adjusts to default domain reputation visualizer + analyzers urls and abstractmethods (#2250)

* little adjusts

* adjust

* refactored base_url in url to enable healthchecks

* adjust

* added update abstract method

* added logging of intel_owl package

* adjusted containers dependencies

* Fix

Signed-off-by: 0ssigeno <[email protected]>

* support for Elastic8 and removal of support for older versions (#2262)

* support for elastic 8

* elastic8

* elastic8

* removed elasticsearch sniffing

* Update migration guide from postgres 12 to 16 (#2260)

* Improved migration from postgres 12 to postgres 16

* Trimmed extra whitespaces

* Fix volume

Signed-off-by: 0ssigeno <[email protected]>

* fixes #1698 - hfinger analyzer (#2241)

* hfinger analyzer

* dependency for hfinger analyzer

* migrations for hfinger analyzer

* overridden update method

* default config fix

* modified usage.md

* fix

* fix

* Fix phoneinfo

Signed-off-by: 0ssigeno <[email protected]>

* Minor fix

Signed-off-by: 0ssigeno <[email protected]>

* Fixes

Signed-off-by: 0ssigeno <[email protected]>

* Fixing logs

Signed-off-by: 0ssigeno <[email protected]>

* Bump channels from 4.0.0 to 4.1.0 in /requirements (#2255)

Bumps [channels](https://github.com/django/channels) from 4.0.0 to 4.1.0.
- [Changelog](https://github.com/django/channels/blob/main/CHANGELOG.txt)
- [Commits](django/channels@4.0.0...4.1.0)

---
updated-dependencies:
- dependency-name: channels
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump quark-engine from 24.2.1 to 24.4.1 in /requirements (#2254)

Bumps [quark-engine](https://github.com/quark-engine/quark-engine) from 24.2.1 to 24.4.1.
- [Release notes](https://github.com/quark-engine/quark-engine/releases)
- [Commits](quark-engine/quark-engine@v24.2.1...v24.4.1)

---
updated-dependencies:
- dependency-name: quark-engine
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* adjusted guide (#2264)

* Fix docs

Signed-off-by: 0ssigeno <[email protected]>

* Little adjusts (#2263)

* little adjusts

* adjust

* refactored base_url in url to enable healthchecks

* adjust

* added update abstract method

* added logging of intel_owl package

* adjusted threatfox

* threatfox analyzer fix

* threatfox analyzer fix

* threatfox analyzer fix

* threatfox analyzer fix

* threatfox analyzer fix

* elastic shards/replicas reduced

* Fix

Signed-off-by: 0ssigeno <[email protected]>

* typo

* fix

Signed-off-by: 0ssigeno <[email protected]>

* adjusted NodeToolbar (#2273)

* Fix migration

Signed-off-by: 0ssigeno <[email protected]>

* Cache are adjusted through celery

Signed-off-by: 0ssigeno <[email protected]>

* fix

Signed-off-by: 0ssigeno <[email protected]>

* Added stop step for intermediate container (#2266)

* create_or_update

Signed-off-by: 0ssigeno <[email protected]>

* Typo

Signed-off-by: 0ssigeno <[email protected]>

* Fix

Signed-off-by: 0ssigeno <[email protected]>

* Fixes

Signed-off-by: 0ssigeno <[email protected]>

* Remove dns0 analyzers (#2274)

* Added migrations to remove analyzers and playbook

* Removed analyzers sources

* Removed dns0 analyzer from docs

* Fix

Signed-off-by: 0ssigeno <[email protected]>

* Fix black

* Added if conditions for saver migrations

---------

Signed-off-by: 0ssigeno <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>

* Frontend - adjusted job metadata section (#2272)

* adjusted JobInfoCard

* fixed  fitView in chrome

* Frontend - improvements (#2278)

* improvements

* fixed test

* Fix_ci (#2284)

* Fix_ci

Signed-off-by: 0ssigeno <[email protected]>

* Typo

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* Optimization

Signed-off-by: 0ssigeno <[email protected]>

* Optimization

Signed-off-by: 0ssigeno <[email protected]>

* ASN maxmind integration (#2282)

* Changed library from maxminddb to geoip2

* Refactoring maxmind not finished

* Refactoring maxmind analyzer

* Added methods for query db

* Made a method 'private'

* Renamed method

* Made attributes 'private'

* Added return type

* Improved log message

* Renamed back to update() because of updating cron

* Fixed media_root settings

* Added log to tar extraction

* Removed unnecessary variable

* Improved log messages

* Readded maxminddb library

* Update api_app/analyzers_manager/observable_analyzers/maxmind.py

Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>

* Fix_ci (#2284)

* Fix_ci

Signed-off-by: 0ssigeno <[email protected]>

* Typo

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* Changed library from maxminddb to geoip2

* Refactoring maxmind not finished

* Refactoring maxmind analyzer

* Added methods for query db

* Made a method 'private'

* Renamed method

* Made attributes 'private'

* Added return type

* Improved log message

* Renamed back to update() because of updating cron

* Fixed media_root settings

* Added log to tar extraction

* Removed unnecessary variable

* Improved log messages

* Readded maxminddb library

* Moved functions and improved logs

* Changed error handling

* Fixed deepsource warning

---------

Signed-off-by: 0ssigeno <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>

* Abusix send email (#2283)

* Added AbuseSubmitter connector

* Renamed and updated connector EmailSender

* Added monkeypatch to email sender connector

* Updated EmailSender parameters

* Added AbuseDomainToAbuseIp pivot

* Updated AbuseDomainToAbuseIp pivot

* Updated pivots

* Changed health_check_status to editable

* Changed health_check_status back to not editable

* receiver update

* subject and body update

* added abusix update migration

* fix investigation creation

* updated AbuseDomainToIp should_run

* updated AbuseIpToSubmission should_run

* added plugin migrations

* fix migrations

* Changed email sender body

* fixed migrations dependencies

* updates email sender and abuse submitter

* Fix migrations

* Fix

* Fix

* Fix playbooks migrations

* Update api_app/connectors_manager/connectors/abuse_submitter.py

Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>

* Added AbuseSubmitter connector

* Renamed and updated connector EmailSender

* Added monkeypatch to email sender connector

* Updated EmailSender parameters

* Added AbuseDomainToAbuseIp pivot

* Updated AbuseDomainToAbuseIp pivot

* Updated pivots

* Changed health_check_status to editable

* Changed health_check_status back to not editable

* receiver update

* subject and body update

* added abusix update migration

* fix investigation creation

* updated AbuseDomainToIp should_run

* updated AbuseIpToSubmission should_run

* added plugin migrations

* fix migrations

* Changed email sender body

* fixed migrations dependencies

* updates email sender and abuse submitter

* Fix migrations

* Fix

* Fix

* Fix playbooks migrations

* added update method

* added update method

* removed AbuseSubmitter values

Co-authored-by: Simone Berni <[email protected]>

* removed EmailSender values

* Update should_run

Co-authored-by: Simone Berni <[email protected]>

* changed AbuseDomainToAbuseIp to AnyCompare

* update compare

* added update

* removed AbuseIpToSubmission

* Added AbuseSubmitter connector

* Renamed and updated connector EmailSender

* Added monkeypatch to email sender connector

* Updated EmailSender parameters

* Added AbuseDomainToAbuseIp pivot

* Updated AbuseDomainToAbuseIp pivot

* Updated pivots

* Changed health_check_status to editable

* Changed health_check_status back to not editable

* receiver update

* subject and body update

* added abusix update migration

* fix investigation creation

* updated AbuseDomainToIp should_run

* updated AbuseIpToSubmission should_run

* added plugin migrations

* fix migrations

* Changed email sender body

* fixed migrations dependencies

* updates email sender and abuse submitter

* Fix migrations

* Fix

* Fix

* Fix playbooks migrations

* added update method

* Added AbuseSubmitter connector

* Changed health_check_status to editable

* Changed health_check_status back to not editable

* added abusix update migration

* added plugin migrations

* fix migrations

* fixed migrations dependencies

* Fix migrations

* removed AbuseSubmitter values

Co-authored-by: Simone Berni <[email protected]>

* removed EmailSender values

* Update should_run

Co-authored-by: Simone Berni <[email protected]>

* changed AbuseDomainToAbuseIp to AnyCompare

* update compare

* added update

* removed AbuseIpToSubmission

* Fix pivot migrations

* Renamed playbook Abuse_Domain to Takedown_Request

* Update api_app/connectors_manager/connectors/abuse_submitter.py

Co-authored-by: Matteo Lodi <[email protected]>

* Update api_app/connectors_manager/connectors/abuse_submitter.py

Co-authored-by: Matteo Lodi <[email protected]>

* Added EmailSender header and footer

* Fix linters

* Fix test

Signed-off-by: 0ssigeno <[email protected]>

* Fix pivot test

* Fix

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>

* Fixed EmailSender output and plugin descriptions (#2290)

* Updated EmailSender output

* Updated plugin descriptions

* Update authentication/models.py

Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>

* Update authentication/models.py

Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>

* deepsource adjusts

* bump and changelog

* fixes for Yara

* removing Yara visualizer from Static Analysis Playbook and updated and fixed Floss Analyzer

* Removed mkdir

Signed-off-by: 0ssigeno <[email protected]>

* blake post merge

* Yara already made in dockerfile

Signed-off-by: 0ssigeno <[email protected]>

* Removed import

Signed-off-by: 0ssigeno <[email protected]>

* update yara visualizer

* Docs update (#2293)

* Added list of pre-built pivots

* Added list of pre-built connectors

* Added list of pre-built playbooks

* Added already implemented modules for pivots

* Added running a plugin

* tried adjust

* Adjust field_to_compare description

---------

Co-authored-by: Matteo Lodi <[email protected]>

* Fix dependencies

Signed-off-by: 0ssigeno <[email protected]>

* Typo

Signed-off-by: 0ssigeno <[email protected]>

* Upgrade docs

Signed-off-by: 0ssigeno <[email protected]>

* Fixed update with None token (#2287)

* Bump django-ses from 3.5.0 to 4.0.0 in /requirements (#2280)

Bumps [django-ses](https://github.com/django-ses/django-ses) from 3.5.0 to 4.0.0.
- [Release notes](https://github.com/django-ses/django-ses/releases)
- [Changelog](https://github.com/django-ses/django-ses/blob/main/CHANGES.md)
- [Commits](django-ses/django-ses@v3.5.0...v4.0.0)

---
updated-dependencies:
- dependency-name: django-ses
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump djangorestframework from 3.14.0 to 3.15.1 in /requirements (#2219)

Bumps [djangorestframework](https://github.com/encode/django-rest-framework) from 3.14.0 to 3.15.1.
- [Release notes](https://github.com/encode/django-rest-framework/releases)
- [Commits](encode/django-rest-framework@3.14.0...3.15.1)

---
updated-dependencies:
- dependency-name: djangorestframework
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump library/nginx from 1.25.4-alpine to 1.26.0-alpine in /docker (#2285)

Bumps library/nginx from 1.25.4-alpine to 1.26.0-alpine.

---
updated-dependencies:
- dependency-name: library/nginx
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Starting playbooks (#2297)

* starting playbooks

Signed-off-by: 0ssigeno <[email protected]>

* Not starting

Signed-off-by: 0ssigeno <[email protected]>

* Fix migration

Signed-off-by: 0ssigeno <[email protected]>

* Prettier

Signed-off-by: 0ssigeno <[email protected]>

* Fix

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* Add IP2WHOIS (#2288)

* Add IP2WHOIS

* Update Usage.md

* Update ip2whois.py

- Add an example JSON result to showcase the fields in the result.
- Attempt to fix an error reported by DeepSource: Python.

* Update ip2whois.py

Fix formatting error.

* Update ip2whois.py

Fix the E501 line too long error by changed to another JSON example.

* Update ip2whois.py

* fixed migration

---------

Co-authored-by: Matteo Lodi <[email protected]>

* Bump gunicorn from 20.1.0 to 22.0.0 in /integrations/pcap_analyzers (#2276)

Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 20.1.0 to 22.0.0.
- [Release notes](https://github.com/benoitc/gunicorn/releases)
- [Commits](benoitc/gunicorn@20.1.0...22.0.0)

---
updated-dependencies:
- dependency-name: gunicorn
  dependency-type: direct:production
...

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* Bump gunicorn from 20.1.0 to 22.0.0 in /integrations/tor_analyzers (#2277)

Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 20.1.0 to 22.0.0.
- [Release notes](https://github.com/benoitc/gunicorn/releases)
- [Commits](benoitc/gunicorn@20.1.0...22.0.0)

---
updated-dependencies:
- dependency-name: gunicorn
  dependency-type: direct:production
...

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* Bump django-auth-ldap from 4.7.0 to 4.8.0 in /requirements (#2270)

Bumps [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap) from 4.7.0 to 4.8.0.
- [Release notes](https://github.com/django-auth-ldap/django-auth-ldap/releases)
- [Changelog](https://github.com/django-auth-ldap/django-auth-ldap/blob/master/docs/changes.rst)
- [Commits](django-auth-ldap/django-auth-ldap@4.7.0...4.8.0)

---
updated-dependencies:
- dependency-name: django-auth-ldap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump gunicorn in /integrations/malware_tools_analyzers/requirements (#2275)

Bumps [gunicorn](https://github.com/benoitc/gunicorn) from 20.1.0 to 22.0.0.
- [Release notes](https://github.com/benoitc/gunicorn/releases)
- [Commits](benoitc/gunicorn@20.1.0...22.0.0)

---
updated-dependencies:
- dependency-name: gunicorn
  dependency-type: direct:production
...

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>

* updated frontend dependencies

* Fix serializer

Signed-off-by: 0ssigeno <[email protected]>

* added linkedin button (#2299)

* Fixed tests and fixed maxmind update (#2298)

* added plugin info icon in raw data report (#2296)

* added plugin info icon

* fix test

* adjusted tests

* refactor pluginReportTables

* Change investigation default name (#2301)

* Change investigation default name

Signed-off-by: 0ssigeno <[email protected]>

* Blake

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* adjusted old job removal

* Graph prevention with scan_mode check_previous_analysis (#2302)

* Graph prevention with scan_mode check_previous_analysis

Signed-off-by: 0ssigeno <[email protected]>

* Wops

Signed-off-by: 0ssigeno <[email protected]>

---------

Signed-off-by: 0ssigeno <[email protected]>

* Fix + test for runtime config

Signed-off-by: 0ssigeno <[email protected]>

* Fixes

Signed-off-by: 0ssigeno <[email protected]>

* domain playbook adjusts (#2304)

* domain playbook adjusts

* black

* Added few Docstrings in authentication\views.py  (#2236)

* Added few Docstrings

* Updated with few changes.

* linter

---------

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>
Co-authored-by: Your Name <[email protected]>

* fixes #1699 Permhash analyzer (#2258)

* added permhash analyzer and updated project-requirements.txt

* name change

* migrations

* changes

* update mimetypes and migration file

* update permhash

* fix migration issues

* raise AnalyzerRunException in perm_hash.py

* use mimetype instead of file extension

* added monkeypatch patches and changed permhash version from 0.1.4.2 to 0.1.4

* add permhash to free analyzers

* updated test_classes

 => added AndroidManifest.xml, manifest.json and sample.crx to
test_files.zip
 => added test cases for xml, json and crx in
tests/api_app/analyzers_manager/test_classes.py

* updated perm_hash.py

* added permhash to both free analyzers and static analyzers

* updated Usage.md

* updated migrations

* fix

* migration

* migrations

* migrations

---------

Co-authored-by: Matteo Lodi <[email protected]>

* Bump pywatchman from 1.4.1 to 2.0.0 in /requirements (#2169)

Bumps [pywatchman](https://github.com/facebook/watchman) from 1.4.1 to 2.0.0.
- [Release notes](https://github.com/facebook/watchman/releases)
- [Commits](https://github.com/facebook/watchman/commits/v2.0)

---
updated-dependencies:
- dependency-name: pywatchman
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Added traefik switch to script (#2307)

* Removed depends on

Signed-off-by: 0ssigeno <[email protected]>

* Fixes Blint#2232 (#2257)

* blint

* report directory, code quality, docs

adjusts to default domain reputation visualizer + analyzers urls and abstractmethods (#2250)

* little adjusts

* adjust

* refactored base_url in url to enable healthchecks

* adjust

* added update abstract method

* added logging of intel_owl package

adjusted containers dependencies

Fix

Signed-off-by: 0ssigeno <[email protected]>

report directory, code quality, docs

adjusts to default domain reputation visualizer + analyzers urls and abstractmethods (#2250)

* little adjusts

* adjust

* refactored base_url in url to enable healthchecks

* adjust

* added update abstract method

* added logging of intel_owl package

adjusted containers dependencies

Fix

Signed-off-by: 0ssigeno <[email protected]>

RED tlp

* dict response, log remove

* cleanup

* migratiuons

* migrations

* code quality

* supportfiles

* updated blint

* updated python in ci

* repo_downloader_fix

* codeql

* fix migrations

* fix migrations

* cleaned code

* cleaned code

* fix

* try fix repo downloader

* don't work

---------

Co-authored-by: g4ze <[email protected]>
Co-authored-by: Matteo Lodi <[email protected]>

* adjusted JobIsRunningFlow (#2300)

* adjusted JobIsRunningFlow

* refactor

* deepsource

* adjusted test + permission edge case

* adjusted Update view

---------

Signed-off-by: 0ssigeno <[email protected]>
Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Shivam Purohit <[email protected]>
Co-authored-by: 0ssigeno <[email protected]>
Co-authored-by: Moon Patel <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: Martina Carella <[email protected]>
Co-authored-by: Daniele Rosetti <[email protected]>
Co-authored-by: fgibertoni <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Simone Berni <[email protected]>
Co-authored-by: code-review-doctor[bot] <72320148+code-review-doctor[bot]@users.noreply.github.com>
Co-authored-by: Shivam Purohit <[email protected]>
Co-authored-by: Cristina Ascari <[email protected]>
Co-authored-by: IP2Location <[email protected]>
Co-authored-by: suryapavan1611 <[email protected]>
Co-authored-by: Your Name <[email protected]>
Co-authored-by: Nilay Gupta <[email protected]>
Co-authored-by: g4ze <[email protected]>
  • Loading branch information
17 people committed May 13, 2024
2 parents 1d7f566 + f4dd486 commit 98197f7
Show file tree
Hide file tree
Showing 227 changed files with 8,071 additions and 2,475 deletions.
12 changes: 11 additions & 1 deletion .github/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,23 @@

[**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)

## [v6.0.2](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.1)
Major fixes and adjustments. We improved the documentation to help the transition to the new major version.

We added **Pivot** buttons to enable manual Pivoting from an Observable/File analysis to another. See [Doc](https://intelowl.readthedocs.io/en/latest/Usage.html#pivots) for more info

As usual, we add new plugins. This release brings the following new ones:
* a complete **TakedownRequest** playbook to automate TakeDown requests for malicious domains
* new File Analyzers for tools like [HFinger](https://github.com/CERT-Polska/hfinger), [Permhash](https://github.com/google/permhash) and [Blint](https://github.com/owasp-dep-scan/blint)
* improvement of the existing Maxmind analyzer: it now downloads the ASN database too.

## [v6.0.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.1)
Little fixes for the major.

## [v6.0.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.0.0)
This major release is another important milestone for this project! We have been working hard to transform IntelOwl from a *Data Extraction Platform* to a complete *Investigation Platform*!

One of the most noticeable feature is the addition of the [**Investigation** framework](https://intelowl.readthedocs.io/en/latest/Usage.md#investigations-framework)!
One of the most noticeable feature is the addition of the [**Investigation** framework](https://intelowl.readthedocs.io/en/latest/Usage.html#investigations-framework)!

Thanks to the this new feature, analysts can leverage IntelOwl as the starting point of their "Investigations", register their findings, correlate the information found, and collaborate...all in a single place.

Expand Down
21 changes: 0 additions & 21 deletions .github/ISSUE_TEMPLATE/new_scanner.md

This file was deleted.

2 changes: 1 addition & 1 deletion .github/workflows/codeql-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
- name: Set up Python
uses: actions/[email protected]
with:
python-version: '3.9'
python-version: '3.11'

- name: Install dependencies
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pull_request_automation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
- name: Set up Python
uses: actions/[email protected]
with:
python-version: 3.9
python-version: 3.11

- name: Install Dependencies
run: |
Expand Down
1 change: 1 addition & 0 deletions api_app/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,7 @@ def get_secrets(self, obj: PythonModule):
class AbstractConfigAdminView(CustomAdminView):
list_display = ("name", "description", "disabled", "disabled_in_orgs")
search_fields = ("name",)
list_filter = ("disabled",)
# allow to clone the object
save_as = True

Expand Down
37 changes: 37 additions & 0 deletions api_app/analyzers_manager/file_analyzers/blint_scan.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import logging
import os
import shutil

from blint.analysis import AnalysisRunner
from django.conf import settings

from api_app.analyzers_manager.classes import FileAnalyzer
from intel_owl.settings._util import set_permissions

logger = logging.getLogger(__name__)


class BlintAnalyzer(FileAnalyzer):
"""
Wrapper for Blint static analysis tool
"""

def update(self) -> bool:
pass

def run(self) -> dict:
logger.info(f"Running Blint on {self.filepath} for {self.md5}")

reports_dir = settings.BLINT_REPORTS_PATH / f"blint_analysis_{self.md5}"
os.mkdir(reports_dir)
set_permissions(reports_dir)

analyzer = AnalysisRunner()
findings, reviews, fuzzables = analyzer.start(
files=[self.filepath], reports_dir=reports_dir
)
response = {"findings": findings, "reviews": reviews, "fuzzables": fuzzables}

shutil.rmtree(reports_dir)

return response
16 changes: 15 additions & 1 deletion api_app/analyzers_manager/file_analyzers/floss.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from json import dumps as json_dumps

from api_app.analyzers_manager.classes import DockerBasedAnalyzer, FileAnalyzer
from api_app.analyzers_manager.exceptions import AnalyzerRunException


class Floss(FileAnalyzer, DockerBasedAnalyzer):
Expand All @@ -23,15 +24,28 @@ class Floss(FileAnalyzer, DockerBasedAnalyzer):
max_no_of_strings: dict
rank_strings: dict

@classmethod
def update(cls) -> bool:
pass

def run(self):
# get binary
binary = self.read_file_bytes()
# make request data
fname = str(self.filename).replace("/", "_").replace(" ", "_")
args = [f"@{fname}"]
# From floss v3 there is prompt that can be overcome
# by using the flag --no static.
# We can lose static strings considering that we can easily
# retrieve them with more simple tools
args = [f"@{fname}", "--json", "--no", "static"]
req_data = {"args": args, "timeout": self.timeout}
req_files = {fname: binary}
result = self._docker_run(req_data, req_files)
if not isinstance(result, dict):
raise AnalyzerRunException(
f"result from floss tool is not a dict but is {type(result)}."
f" Full dump: {result}"
)
result["exceeded_max_number_of_strings"] = {}
# we are changing the endpoint of _docker_run to stringsifter
self.url = self.ranking_url
Expand Down
57 changes: 57 additions & 0 deletions api_app/analyzers_manager/file_analyzers/hfinger.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl
# See the file 'LICENSE' for copying permission.

from hfinger.analysis import hfinger_analyze

from api_app.analyzers_manager.classes import FileAnalyzer
from tests.mock_utils import if_mock_connections, patch


class Hfinger(FileAnalyzer):
"""
Create fingerprints of malware HTTP
requests stored in pcap files.
"""

fingerprint_report_mode: int = 2

def run(self):
return hfinger_analyze(self.filepath, self.fingerprint_report_mode)

@classmethod
def update(cls) -> bool:
pass

@classmethod
def _monkeypatch(cls):
patches = [
if_mock_connections(
patch(
"hfinger.analysis.hfinger_analyze",
return_value=[
{
"epoch_time": "1388111476.787707000",
"ip_src": "192.168.1.138",
"ip_dst": "173.194.115.80",
"port_src": "49209",
"port_dst": "80",
"fingerprint": "2.4|1|0.5||2.4|1.2|GE|1|ac,ac-la,us-ag,\
ac-en,ho,co|ac:te-ht,ap-xh+xm,as-as/ac-la:75ef792f/\
us-ag:ca0c4d71/ac-en:gz,de/co:Ke-Al|||",
},
{
"epoch_time": "1388111477.142485000",
"ip_src": "192.168.1.138",
"ip_dst": "66.225.230.141",
"port_src": "49220",
"port_dst": "80",
"fingerprint": "1.5|3|1.0|html|||GE|1|ac,re,ac-la,us-ag,\
ac-en,ho,co|ac:te-ht,ap-xh+xm,as-as/ac-la:75ef792f/\
us-ag:ca0c4d71/ac-en:gz,de/co:Ke-Al|||",
},
],
)
)
]

return super()._monkeypatch(patches=patches)
84 changes: 84 additions & 0 deletions api_app/analyzers_manager/file_analyzers/perm_hash.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl
# See the file 'LICENSE' for copying permission.

import logging

import magic
from permhash.functions import (
APK_MANIFEST_MIMETYPES,
APK_MIMETYPES,
CRX_MANIFEST_MIMETYPES,
CRX_MIMETYPES,
permhash_apk,
permhash_apk_manifest,
permhash_crx,
permhash_crx_manifest,
)

from api_app.analyzers_manager.classes import FileAnalyzer
from api_app.analyzers_manager.exceptions import AnalyzerRunException
from tests.mock_utils import if_mock_connections, patch

logger = logging.getLogger(__name__)


class Permhash(FileAnalyzer):
"""
Create permissions hash of APK, Chrome extensions,
Android manifest and Chrome extension manifest files.
"""

def run(self):
result = {}
mimetype = magic.from_file(self.filepath, mime=True)

hash_val = ""

if mimetype in APK_MIMETYPES:
hash_val = permhash_apk(self.filepath)
elif mimetype in APK_MANIFEST_MIMETYPES:
hash_val = permhash_apk_manifest(self.filepath)
elif mimetype in CRX_MIMETYPES:
hash_val = permhash_crx(self.filepath)
elif mimetype in CRX_MANIFEST_MIMETYPES:
hash_val = permhash_crx_manifest(self.filepath)
else:
raise AnalyzerRunException(f"Mimetype {mimetype} not supported.")

# permhash returns False if for some reason the hash value can't be found
if hash_val:
result["hash"] = hash_val
else:
result["error"] = "Could not find permissions in the file."

return result

@classmethod
def update(cls) -> bool:
pass

@classmethod
def _monkeypatch(cls):
hash_val = "aad106ceb64ac2a636ddec77c3feed4c2ffc5c27ab353660d8cb3e1c971ef278"
patches = [
if_mock_connections(
patch(
"permhash.functions.permhash_apk",
return_value=hash_val,
),
patch(
"permhash.functions.permhash_apk_manifest",
return_value=hash_val,
),
patch(
"permhash.functions.permhash_crx",
return_value=hash_val,
),
patch(
"permhash.functions.permhash_crx_manifest",
return_value=hash_val,
),
)
]

return super()._monkeypatch(patches=patches)
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,10 @@
class XlmMacroDeobfuscator(FileAnalyzer):
passwords_to_check: list

@classmethod
def update(cls) -> bool:
pass

def run(self):
results = {}
try:
Expand All @@ -24,7 +28,7 @@ def run(self):
if not results:
results["error"] = "Can't decrypt with current passwords"
except SoftTimeLimitExceeded:
self._handle_base_exception("Soft Time Limit Exceeded")
self._handle_exception("Soft Time Limit Exceeded", is_base_err=True)
return results

def decrypt(self, xlmpassword=""):
Expand Down
15 changes: 12 additions & 3 deletions api_app/analyzers_manager/file_analyzers/yara_scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,13 @@ def _update_git(self):
try:
o.pull(allow_unrelated_histories=True, rebase=True)
except git.exc.GitCommandError as e:
logger.exception(e)
return
if "index.lock" in e.stderr:
# for some reason the git process did not exit correctly
self.delete_lock_file()
o.pull(allow_unrelated_histories=True, rebase=True)
else:
logger.exception(e)
return
else:
logger.info(f"About to clone {self.url} at {self.directory}")
git.Repo.clone_from(self.url, self.directory, depth=1)
Expand All @@ -151,6 +156,10 @@ def _update_git(self):
if settings.GIT_KEY_PATH.exists():
os.remove(settings.GIT_KEY_PATH)

def delete_lock_file(self):
lock_file_path = self.directory / ".git" / "index.lock"
lock_file_path.unlink(missing_ok=False)

@property
def compiled_file_name(self):
return "intel_owl_compiled.yas"
Expand Down Expand Up @@ -328,7 +337,7 @@ def __repr__(self):
class YaraScan(FileAnalyzer):
ignore: list
repositories: list
_private_repositories: dict
_private_repositories: dict = {}
local_rules: str

def _get_owner_and_key(self, url: str) -> Tuple[Union[str, None], Union[str, None]]:
Expand Down
28 changes: 28 additions & 0 deletions api_app/analyzers_manager/migrations/0075_adjust_greynoise.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from django.db import migrations


def migrate(apps, schema_editor):
PythonModule = apps.get_model("api_app", "PythonModule")

pm = PythonModule.objects.get(
module="greynoiseintel.GreyNoiseAnalyzer",
base_path="api_app.analyzers_manager.observable_analyzers",
)
param = pm.parameters.get(name="api_key_name")
param.required = False
param.values.filter(owner=None, for_organization=False).delete()
param.save()


def reverse_migrate(apps, schema_editor):
...


class Migration(migrations.Migration):
dependencies = [
("api_app", "0062_alter_parameter_python_module"),
("analyzers_manager", "0074_adjust_maximum_tlp"),
]
operations = [
migrations.RunPython(migrate, reverse_migrate),
]
Loading

0 comments on commit 98197f7

Please sign in to comment.