Merge pull request #1822 from intelowlproject/develop

v5.1.0

.github/CHANGELOG.md

@@ -2,6 +2,28 @@
 
 [**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)
 
+## [v5.1.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v5.1.0)
+With this release we announce our new official site created by [Abheek Tripathy](https://twitter.com/abheekblahblah)!
+Feel free to check it out! Official [blog post here](https://intelowlproject.github.io/blogs/official_site_revamped)!
+
+**Important changes**
+* We added a new type of Plugin called [Ingestor](https://intelowl.readthedocs.io/en/latest/Usage.html#ingestors). **Ingestors** allow to automatically insert IOC streams from outside sources to IntelOwl itself.
+* Visualizers are not connected anymore to Analyzers/Connectors. They are connected to a single Playbook instead. This allows the users to create and manage the Visualizers in an easier way.
+* We added the new **Pivot** framework in the backend which allows to connect jobs to each other and to _pivot_ from one indicator to another. This is the first step to give the chance to the users to create more broader and complex investigation in IntelOwl. The next step will be to add the Frontend changes that allows the user to fully leverage the framework 
+
+**New/Improved Plugins:**
+* Added new `DNS` playbook that collects the analyzers which performs DNS queries to various providers
+* Added more option for `CapeSandbox` analyzer
+
+**Fixes / adjusts / minor changes**
+* added chance to change the password of the account from the personal section in the application
+* added a lot of Frontend tests for the "Scan" page to improve stability
+* some frontend changes to improve overall experience (#1743, #1741, #1754, #1772, #1780, #1807, #1806)
+* added new partial statuses for the Job which allow to better track the job progression [#1740)]
+* Added new public Yara rules
+* updated installation instructions
+* upgraded a lot of packages
+
 ## [v5.0.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v5.0.1)
 
 **Bug fixing for the v5.0.0 release**

.github/ISSUE_TEMPLATE/new_visualizer.md

@@ -10,10 +10,7 @@ assignees: ''
 ## Name
 
 
-## Analyzers
-
-
-## Connectors
+## Playbooks
 
 
 ## Why should we create it

.github/legal_notice.md

@@ -38,11 +38,13 @@ license terms.
 [InQuest Yara rules](https://github.com/InQuest/yara-rules),
 [dr4k0nia Yara rules](https://github.com/dr4k0nia/yara-rules),
 [Facebook Yara rules](https://github.com/facebook/malware-detection),
+[edelucia Yara rules](https://github.com/edelucia/rules/tree/main/yara),
 [JPCERTCC Yara rules](https://github.com/JPCERTCC/jpcert-yara),
 [HuntressLab Yara rules](https://github.com/embee-research/Yara),
 [elceef Yara Rules](https://github.com/elceef/yara-rulz),
 [Elastic Yara Rules](https://github.com/elastic/protections-artifacts), 
 [SIFalcon Yara Rules](https://github.com/SIFalcon/Detection/),
+[LOLDrivers Yara Rules](https://github.com/magicsword-io/LOLDrivers),
 [APKiD](https://github.com/rednaga/APKiD/blob/master/LICENSE.COMMERCIAL),
 [Box-JS](https://github.com/CapacitorSet/box-js/blob/master/LICENSE),
 [Capa](https://github.com/fireeye/capa/blob/master/LICENSE.txt),

.github/pull_request_template.md

@@ -16,7 +16,7 @@ Please delete options that are not relevant.
 
 - [ ] I have read and understood the rules about [how to Contribute](https://intelowl.readthedocs.io/en/latest/Contribute.html) to this project
 - [ ] The pull request is for the branch `develop`
-- [ ] A new plugin (analyzer, connector, visualizer or playbook) was added or changed, in which case:
+- [ ] A new plugin (analyzer, connector, visualizer, playbook or ingestor) was added or changed, in which case:
     - [ ] [Usage](https://github.com/intelowlproject/IntelOwl/blob/master/docs/source/Usage.md) file was updated.
     - [ ] [Advanced-Usage](./Advanced-Usage.md) was updated (in case the plugin provides additional optional configuration).
     - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.

.github/workflows/codeql-analysis.yml

@@ -7,6 +7,8 @@ permissions: read-all
 name: "CodeQL"
 
 on:
+  push:
+    branches: [master, develop]
   pull_request:
     # The branches below must be a subset of the branches above
     branches: [master, develop]
@@ -16,6 +18,14 @@ jobs:
     name: Analyze
     runs-on: ubuntu-latest
 
+    permissions:
+      # required for all workflows
+      security-events: write
+
+      # only required for workflows in private repositories
+      actions: read
+      contents: read
+
     strategy:
       fail-fast: false
       matrix:
@@ -34,7 +44,7 @@ jobs:
         fetch-depth: 2
 
     - name: Set up Python
-      uses: actions/setup-python@v4.6.1
+      uses: actions/setup-python@v4.7.0
       with:
         python-version: '3.9'
 

.github/workflows/pull_request_automation.yml

@@ -37,7 +37,7 @@ jobs:
         uses: actions/checkout@v3
 
       - name: Set up Python
-        uses: actions/setup-python@v4.6.1
+        uses: actions/setup-python@v4.7.0
         with:
           python-version: 3.9
 
@@ -90,6 +90,7 @@ jobs:
           DOCKER_BUILDKIT: 1
           BUILDKIT_PROGRESS: "plain"
           STAGE: "ci"
+          REPO_DOWNLOADER_ENABLED: false
 
       - name: Startup script launch (Fast)
         if: "!contains(github.base_ref, 'master')"
@@ -99,6 +100,7 @@ jobs:
           DOCKER_BUILDKIT: 1
           BUILDKIT_PROGRESS: "plain"
           STAGE: "ci"
+          REPO_DOWNLOADER_ENABLED: false
 
       - name: Docker debug
         run: |

.github/workflows/scorecard.yml

@@ -37,7 +37,7 @@ jobs:
           persist-credentials: false
 
       - name: "Run analysis"
-        uses: ossf/scorecard-action@e38b1902ae4f44df626f11ba0734b14fb91f8f86 # v2.1.2
+        uses: ossf/scorecard-action@08b4669551908b1024bb425080c797723083c031 # v2.2.0
         with:
           results_file: results.sarif
           results_format: sarif

.github/workflows/sphinx_build_on_release.yml

@@ -14,7 +14,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
       - name: Set up Python
-        uses: actions/setup-python@v4.6.1
+        uses: actions/setup-python@v4.7.0
         with:
           python-version: 3.9
       - name: Install docs requirements

.pre-commit-config.yaml

@@ -36,7 +36,7 @@ repos:
         args: ["--fix"]
         files: frontend/src/
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: v2.6.2
+    rev: v3.0.0
     hooks:
       - id: prettier
         files: frontend/src/

api_app/admin.py

@@ -2,8 +2,12 @@
 # See the file 'LICENSE' for copying permission.
 
 from django.contrib import admin
+from django.db.models import JSONField
+from prettyjson.widgets import PrettyJSONWidget
 
-from .models import Job, PluginConfig, Tag
+from .forms import ParameterInlineForm
+from .models import AbstractConfig, Job, Parameter, PluginConfig, Tag
+from .tabulars import PluginConfigInline
 
 
 @admin.register(Job)
@@ -53,8 +57,15 @@ class TagAdminView(admin.ModelAdmin):
 
 
 @admin.register(PluginConfig)
-class PluginCredentialAdminView(admin.ModelAdmin):
-    list_display = ("id", "value", "parameter_name", "for_organization", "owner_name")
+class PluginConfigAdminView(admin.ModelAdmin):
+    list_display = (
+        "id",
+        "value",
+        "parameter_name",
+        "for_organization",
+        "owner_name",
+        "plugin",
+    )
     search_fields = ["parameter__name", "value"]
     list_filter = (
         "for_organization",
@@ -64,12 +75,104 @@ class PluginCredentialAdminView(admin.ModelAdmin):
         "parameter__visualizer_config__name",
     )
 
+    @staticmethod
+    def plugin(instance: PluginConfig):
+        return instance.parameter.config.name
+
     @staticmethod
     def parameter_name(instance: PluginConfig):
         return instance.parameter.name
 
     @staticmethod
     def owner_name(instance: PluginConfig):
         if instance.owner:
-            return instance.owner.name
+            return instance.owner.username
         return None
+
+
+class AbstractReportAdminView(admin.ModelAdmin):
+    list_display = (
+        "id",
+        "config",
+        "job",
+        "status",
+        "start_time",
+        "end_time",
+    )
+    list_display_links = ("id",)
+    search_fields = ("config",)
+
+    @staticmethod
+    def has_add_permission(request):
+        return False
+
+
+class JsonViewerAdminView(admin.ModelAdmin):
+    formfield_overrides = {
+        JSONField: {"widget": PrettyJSONWidget(attrs={"initial": "parsed"})}
+    }
+
+
+@admin.register(Parameter)
+class ParameterAdminView(admin.ModelAdmin):
+    inlines = [PluginConfigInline]
+    search_fields = ["name"]
+    list_filter = ["is_secret"]
+    list_display = ParameterInlineForm.Meta.fields + ["plugin"]
+    fields = list_display
+
+    @staticmethod
+    def plugin(obj: Parameter):
+        config = (
+            obj.analyzer_config
+            or obj.connector_config
+            or obj.visualizer_config
+            or obj.ingestor_config
+        )
+        return config.name
+
+
+class ParameterInline(admin.TabularInline):
+    model = Parameter
+    list_display = ParameterInlineForm.Meta.fields
+    fields = list_display + [
+        "default",
+    ]
+    extra = 0
+    show_change_link = True
+    form = ParameterInlineForm
+
+
+class AbstractConfigAdminView(JsonViewerAdminView):
+    list_display = ("name", "description", "disabled", "disabled_in_orgs")
+    search_fields = ("name",)
+    # allow to clone the object
+    save_as = True
+
+    @staticmethod
+    def disabled_in_orgs(instance: AbstractConfig):
+        return [org.name for org in instance.disabled_in_organizations.all()]
+
+
+class PythonConfigAdminView(AbstractConfigAdminView):
+    inlines = [ParameterInline]
+    list_display = (
+        "name",
+        "python_module",
+        "params",
+        "secrets",
+        "disabled",
+        "disabled_in_orgs",
+    )
+
+    @staticmethod
+    def params(instance: AbstractConfig):
+        return list(
+            instance.parameters.filter(is_secret=False).values_list("name", flat=True)
+        )
+
+    @staticmethod
+    def secrets(instance: AbstractConfig):
+        return list(
+            instance.parameters.filter(is_secret=True).values_list("name", flat=True)
+        )

api_app/analyzers_manager/admin.py

@@ -2,9 +2,9 @@
 # See the file 'LICENSE' for copying permission.
 from django.contrib import admin
 
+from api_app.admin import AbstractReportAdminView, PythonConfigAdminView
 from api_app.analyzers_manager.forms import AnalyzerConfigAdminForm
 from api_app.analyzers_manager.models import AnalyzerConfig, AnalyzerReport
-from api_app.core.admin import AbstractConfigAdminView, AbstractReportAdminView
 
 
 @admin.register(AnalyzerReport)
@@ -13,10 +13,12 @@ class AnalyzerReportAdminView(AbstractReportAdminView):
 
 
 @admin.register(AnalyzerConfig)
-class AnalyzerConfigAdminView(AbstractConfigAdminView):
-    list_display = AbstractConfigAdminView.list_display + (
+class AnalyzerConfigAdminView(PythonConfigAdminView):
+    list_display = PythonConfigAdminView.list_display + (
         "type",
         "docker_based",
         "maximum_tlp",
+        "update_schedule",
     )
     form = AnalyzerConfigAdminForm
+    exclude = ["update_task"]

api_app/analyzers_manager/apps.py

@@ -6,3 +6,7 @@
 
 class AnalyzersManagerConfig(AppConfig):
     name = "api_app.analyzers_manager"
+
+    @staticmethod
+    def ready() -> None:
+        from . import signals  # noqa

api_app/analyzers_manager/classes.py

@@ -10,11 +10,11 @@
 import requests
 from django.conf import settings
 
-from api_app.core.classes import Plugin
 from certego_saas.apps.user.models import User
 from tests.mock_utils import MockUpResponse, if_mock_connections, patch
 
-from ..core.models import AbstractConfig
+from ..classes import Plugin
+from ..models import AbstractConfig
 from .constants import HashChoices, ObservableTypes, TypeChoices
 from .exceptions import AnalyzerConfigurationException, AnalyzerRunException
 from .models import AnalyzerConfig, AnalyzerReport
@@ -66,16 +66,6 @@ def get_exceptions_to_catch(self):
             AnalyzerRunException,
         )
 
-    def get_error_message(self, err, is_base_err=False):
-        """
-        Returns error message for
-        *_handle_analyzer_exception* and *_handle_base_exception* fn
-        """
-        return (
-            f"{self.__repr__()}."
-            f" {'Unexpected error' if is_base_err else 'Analyzer error'}: '{err}'"
-        )
-
     def _validate_result(self, result, level=0, max_recursion=190):
         """
         function to validate result, allowing to store inside postgres without errors.
@@ -221,8 +211,8 @@ def filepath(self) -> str:
             )
         return self.__filepath
 
-    def before_run(self, *args, **kwargs):
-        super().before_run(**kwargs)
+    def before_run(self):
+        super().before_run()
         logger.info(
             f"STARTED analyzer: {self.__repr__()} -> "
             f"File: ({self.filename}, md5: {self.md5})"