Merge branch 'develop' of github.com:intelowlproject/IntelOwl into ip…

…2-test

api_app/visualizers_manager/visualizers/yara.py

@@ -57,7 +57,6 @@ def run(self) -> List[Dict]:
         ]
         page1 = self.Page(name="Yara first page")
         h1 = self.HList(value=[self._yara_analyzer()])
-        logger.debug(h1.to_dict())
         page1.add_level(
             self.Level(position=1, size=self.LevelSize.S_3, horizontal_list=h1)
         )
@@ -67,17 +66,11 @@ def run(self) -> List[Dict]:
                 self._yara_signatures(signatures),
             ]
         )
-        logger.debug(h2.to_dict())
-        page2 = self.Page(name="Yara second page")
-        page2.add_level(
-            self.Level(position=1, size=self.LevelSize.S_3, horizontal_list=h1)
-        )
-        page2.add_level(
+        page1.add_level(
             self.Level(position=2, size=self.LevelSize.S_5, horizontal_list=h2)
         )
         logger.debug(page1)
-        logger.debug(page2)
-        return [page1.to_dict(), page2.to_dict()]
+        return [page1.to_dict()]
 
     @classmethod
     def _monkeypatch(cls):

docker/default.yml

@@ -48,6 +48,10 @@ services:
       timeout: 2s
       start_period: 2s
       retries: 6
+    depends_on:
+      uwsgi:
+        condition: service_healthy
+
 
   nginx:
     image: intelowlproject/intelowl_nginx:${REACT_APP_INTELOWL_VERSION}
@@ -81,6 +85,9 @@ services:
     env_file:
       - env_file_app
     <<: *no-healthcheck
+    depends_on:
+      uwsgi:
+        condition: service_healthy
 
 
   celery_worker_default:

docs/source/Contribute.md

@@ -282,6 +282,13 @@ After having written the new python module, you have to remember to:
    5. *Soft_time_limit: maximum time for the task execution
    6. *Playbook to Execute: Playbook that **will** be executed in the Job generated by the Pivot
 
+Most of the times you don't need to create a new Pivot Module. There are already some base modules that can be extended.
+The most important ones are the following 2:
+1.`AnyCompare`: use this module if you want to create a custom Pivot from a specific value extracted from the results of the analyzers/connectors. How? you should populate the parameter `field_to_compare` with the dotted path to the field you would like to extract the value from.
+2.`SelfAnalyzable`: use this module if you want to create a custom Pivot that would analyze again the same observable/file.
+
+
+
 ### How to add a new Visualizer
 
 #### Configuration

docs/source/Usage.md

@@ -298,6 +298,8 @@ The following is the list of the available connectors. You can also navigate the
 - `OpenCTI`: automatically creates an observable and a linked report on your OpenCTI instance, linking the the successful analysis on IntelOwl.
 - `YETI`: YETI = Your Everyday Threat Intelligence. find or create observable on YETI, linking the successful analysis on IntelOwl.
 - `Slack`: Send the analysis link to a Slack channel (useful for external notifications)
+- `EmailSender`: Send a generic email.
+- `AbuseSubmitter`: Send an email to request to take down a malicious domain.
 
 
 ### Pivots
@@ -309,7 +311,8 @@ Pivots are designed to create a job from another job. This plugin allows the use
 This is a "SOAR" feature that allows the users to connect multiple analysis together.
 
 #### List of pre-built Pivots
-None
+- `TakedownRequestToAbuseIp`: This Plugin leverages results from DNS resolver analyzers to extract a valid IP address to pivot to the Abusix analyzer.
+- `AbuseIpToSubmission`: This Plugin leverages results from the Abusix analyzer to extract the abuse contacts of an IP address to pivot to the AbuseSubmitter connector.
 
 You can build your own custom Pivot with your custom logic with just few lines of code. See the [Contribute](https://intelowl.readthedocs.io/en/latest/Contribute.html#how-to-add-a-new-pivot) section for more info.
 
@@ -391,6 +394,9 @@ The following is the list of the available pre-built playbooks. You can also nav
 - `Popular_URL_Reputation_Services`: Collection of the most popular and free reputation analyzers for URLs and Domains
 - `Popular_IP_Reputation_Services`: Collection of the most popular and free reputation analyzers for IP addresses
 - `Dns`: A playbook containing all dns providers
+- `Takedown_Request`: Start investigation to request to take down a malicious domain. A mail will be sent to the domain's abuse contacts found
+- `Abuse_IP`: Playbook containing the Abusix analyzer. It is executed after the Takedown_Request playbook
+- `Send_Abuse_Email`: Playbook containing the AbuseSubmitter connector to send an email to request to take down a malicious domain. It is executed after the Abuse_IP playbook
 
 #### Playbooks creation and customization
 
@@ -573,6 +579,17 @@ These is how every available TLP value behaves once selected for an analysis exe
 4. `RED`: disable analyzers that could impact privacy, limit view permissions to my group and do not use any external service
 
 
+### Running a plugin
+A plugin can be run when all of the following requirements have been satisfied:
+1. All the required parameters of the plugin have been configured
+2. The plugin is not disabled
+3. The plugin is not disabled for the user's organization
+4. If the plugin has a health check schedule, the last check has to be successful
+5. The TLP selected to run the plugin cannot be higher than the maximum TLP configured for that plugin
+6. The observable classification or the file mimetype has to be supported by the plugin
+
+
+
 ## Investigations Framework
 
 *Investigations* are a new framework introduced in IntelOwl v6 with the goal to allow the users to connect the analysis they do with each other.

intel_owl/settings/commons.py

@@ -1,6 +1,5 @@
 # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl
 # See the file 'LICENSE' for copying permission.
-import os
 from pathlib import Path, PosixPath
 
 from ._util import get_secret
@@ -15,7 +14,6 @@
 MEDIA_ROOT = BASE_DIR / "files_required"
 CONFIG_ROOT = PROJECT_LOCATION / "configuration"
 YARA_RULES_PATH = MEDIA_ROOT / "yara"  # path for manual yara rules
-os.makedirs(YARA_RULES_PATH, exist_ok=True)
 
 LOG_DIR = Path("/") / "var" / "log" / "intel_owl"
 # test / ci

requirements/docs-requirements.txt

@@ -1,4 +1,5 @@
-sphinx_rtd_theme==1.3.0
+sphinx_rtd_theme==2.0.0
 sphinxcontrib-openapi==0.8.3
 sphinxcontrib-redoc==1.6.0
 recommonmark==0.7.1
+docutils==0.20.1