bump version to 4.1.0 + added changelog + improved documentation for …

…release

.github/CHANGELOG.md

@@ -2,6 +2,33 @@
 
 [**Upgrade Guide**](https://intelowl.readthedocs.io/en/latest/Installation.md#update-to-the-most-recent-version)
 
+## [v4.1.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.1.0)
+
+This release marks the end of the Google Summer of Code for this year (2022)!
+Each contributor wrote a blog post regarding his work for IntelOwl during this summer:
+  - [Aditya Narayan Sinha](https://twitter.com/0x0elliot): [Creating Playbooks for IntelOwl](https://www.honeynet.org/2022/10/06/gsoc-2022-project-summary-creating-playbooks-for-intelowl/)
+  - [Aditya Pratap Singh](https://twitter.com/devmrfitz): [IntelOwl v4 improvements](https://www.honeynet.org/2022/09/26/gsoc-2022-project-summary-intelowl-v4-improvements/)
+  - [Hussain Khan](https://twitter.com/Hussain41099635): [IntelOwl Go Client](https://www.honeynet.org/2022/09/06/gsoc-2022-project-summary-intelowl-go-client-go-intelowl/)
+
+I would like to thank them and all the mentors (@sp35, @eshaan7, @0ssigeno, @drosetti) for the efforts put in the place during the last months!
+
+Looking forward for the Google Summer of Code 2023!
+
+**Time savers features**
+- New Plugin Type to allow to easily replicate the same type of analysis without having to select and/or configure groups of analyzers/connectors every time: **Playbooks** ([docs reference](https://intelowl.readthedocs.io/en/latest/Usage.html#playbooks))
+- Default Plugins Parameters can be customized from the GUI and are defined at user/org level instead of globally ([docs reference](https://intelowl.readthedocs.io/en/latest/Advanced-Usage.html#customize-analyzer-execution))
+- Plugins Secrets can now be managed from the GUI and are defined at user/org level instead of globally ([docs reference](https://intelowl.readthedocs.io/en/latest/Installation.html#deprecated-environment-configuration))
+- Organization admins can enable/disable analyzers for all the org ([docs reference](https://intelowl.readthedocs.io/en/latest/Usage.html#multi-tenancy))
+- Google Oauth authentication support ([docs reference](https://intelowl.readthedocs.io/en/latest/Advanced-Usage.html#google-oauth2))
+- Added support for `extends` key to simplify Analyzer configuration and customization ([docs reference](https://intelowl.readthedocs.io/en/latest/Usage.html#analyzers-customization))
+
+**Others**
+- Adjusted default time limits and configuration of some analyzers
+- various fixes and stability contributions
+- a lot of dependencies upgrades
+- other minor updates
+
+
 ## [v4.0.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v4.0.1)
 
 **New/Improved Analyzers:**
@@ -321,7 +348,7 @@ This is a minor patch release.
 - New `ClamAV` analyzer: scan files for viruses/malwares/trojans using [ClamAV antivirus engine](https://docs.clamav.net/).
 - Fixed `Tranco` Analyzer pointing to the wrong `python_module`
 - Removed `CirclePDNS` default value in `env_file_app_template`
-- VirusTotal v3: New configuration options: `include_behaviour_summary` for behavioral analysis and `include_sigma_analyses` for sigma analysis report of the file. See [Customize Analyzers](https://intelowl.readthedocs.io/en/master/Advanced-Usage.html#customize-analyzer-execution-at-time-of-request).
+- VirusTotal v3: New configuration options: `include_behaviour_summary` for behavioral analysis and `include_sigma_analyses` for sigma analysis report of the file. See [Customize Analyzers](https://intelowl.readthedocs.io/en/master/Advanced-Usage.html#customize-analyzer-execution).
 
 **REST API changes:**
 

.gitignore

@@ -9,6 +9,7 @@ docker/custom.override.yml
 venv/
 intel_owl_test_env/
 compose-elk.yml
+docs_env/
 docs/build/
 configuration/service_account_keyfile.json
 configuration/custom_yara/*

README.md

@@ -6,10 +6,11 @@
 [![Twitter Follow](https://img.shields.io/twitter/follow/intel_owl?style=social)](https://twitter.com/intel_owl)
 [![Official Site](https://img.shields.io/badge/official-site-blue)](https://intelowlproject.github.io)
 
-[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/intelowlproject/IntelOwl.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/intelowlproject/IntelOwl/context:python)
 [![CodeFactor](https://www.codefactor.io/repository/github/intelowlproject/intelowl/badge)](https://www.codefactor.io/repository/github/intelowlproject/intelowl)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)
+[![CodeQL](https://github.com/intelowlproject/IntelOwl/actions/workflows/codeql-analysis.yml/badge.svg)](https://github.com/intelowlproject/IntelOwl/actions/workflows/codeql-analysis.yml)
+[![Dependency Review](https://github.com/intelowlproject/IntelOwl/actions/workflows/dependency_review.yml/badge.svg)](https://github.com/intelowlproject/IntelOwl/actions/workflows/dependency_review.yml)
 [![Build & Tests](https://github.com/intelowlproject/IntelOwl/workflows/Build%20&%20Tests/badge.svg)](https://github.com/intelowlproject/IntelOwl/actions)
 [![codecov](https://codecov.io/gh/intelowlproject/IntelOwl/branch/master/graph/badge.svg?token=R097M4TYA6)](https://codecov.io/gh/intelowlproject/IntelOwl)
 # Intel Owl

authentication/views.py

@@ -40,7 +40,7 @@ def get_client_obj(request) -> Client:
         return client
 
     def post(self, request, *args, **kwargs):
-        response = super(LoginView, self).post(request, *args, **kwargs)
+        response = super().post(request, *args, **kwargs)
         uname = request.user.username
         logger.info(f"LoginView: received request from '{uname}'.")
         if request.user.is_superuser:
@@ -63,7 +63,7 @@ def post(self, request, *args, **kwargs):
                 logger.info(f"administrator: '{uname}' was logged out.")
             except Exception:
                 logger.exception(f"administrator: '{uname}' session logout failed.")
-        return super(LogoutView, self).post(request, format=None)
+        return super().post(request, format=None)
 
 
 APIAccessTokenView = durin_views.APIAccessTokenView

configuration/analyzer_config.json

@@ -2753,24 +2753,8 @@
     }
   },
   "Shodan_Search": {
-    "type": "observable",
-    "python_module": "shodan.Shodan",
+    "extends": "Shodan_Honeyscore",
     "description": "scan an IP against Shodan Search API",
-    "disabled": false,
-    "external_service": true,
-    "leaks_info": false,
-    "observable_supported": ["ip"],
-    "config": {
-      "soft_time_limit": 30,
-      "queue": "default"
-    },
-    "secrets": {
-      "api_key_name": {
-        "env_var_key": "SHODAN_KEY",
-        "description": "",
-        "required": true
-      }
-    },
     "params": {
       "shodan_analysis": {
         "value": "search",

docker/.env

@@ -1,3 +1,3 @@
 ### DO NOT CHANGE THIS VALUE !!
 ### It should be updated only when you pull latest changes off from the 'master' branch of IntelOwl.
-INTELOWL_TAG_VERSION=v4.0.1
+INTELOWL_TAG_VERSION=v4.1.0

docker/.version

@@ -1 +1 @@
-REACT_APP_INTELOWL_VERSION="v4.0.1"
+REACT_APP_INTELOWL_VERSION="v4.1.0"

docs/source/Advanced-Usage.md

@@ -4,7 +4,7 @@ This page includes details about some advanced features that Intel Owl provides
 
 - [Advanced Usage](#advanced-usage)
   - [Optional Analyzers](#optional-analyzers)
-  - [Customize analyzer execution at time of request](#customize-analyzer-execution-at-time-of-request)
+  - [Customize analyzer execution](#customize-analyzer-execution)
     - [View and understand different parameters](#view-and-understand-different-parameters)
     - [from the GUI](#from-the-gui)
     - [from Pyintelowl](#from-pyintelowl)
@@ -121,16 +121,16 @@ Otherwise you can enable just one of the cited integration by using the related
 python3 start.py prod --tor_analyzers up
 ```
 
-## Customize analyzer execution at time of request
+## Customize analyzer execution
 
 Some analyzers and connectors provide the chance to customize the performed analysis based on parameters (`params` attr in the configuration file) that are different for each analyzer.
 
-- You can set a custom default values by changing their `value` attribute directly from the configuration files.
+- You can set a custom default values by changing their `value` attribute directly from the configuration files. Since IntelOwl v4, it is possible to change these values directly from the GUI in the section "Your plugin configuration".
 - You can choose to provide runtime configuration when requesting an analysis that will be merged with the default overriding it. This override is done only for the specific analysis.
 
 <div class="admonition info">
 <p class="admonition-title">Info</p>
-Connectors parameters can only be changed from it's configuration file, not at the time of analysis request.
+Connectors parameters can only be changed from either their configuration file or the "Your plugin configuration" section, not at the time of analysis request.
 </div>
 
 ##### View and understand different parameters
@@ -211,11 +211,17 @@ Jobs with either AMBER or RED TLP value will be accessible to only members withi
 
 ## Notifications
 
-IntelOwl integrated the notification system from the certego_saas package, allowing the admins to create notification that every user will be able to see.
+Since v4, IntelOwl integrated the notification system from the `certego_saas` package, allowing the admins to create notification that every user will be able to see.
 
-It is possible to create a new notification from the django admin interface:
-in body it is possible to even use html syntax, allowing to embed images, links, etc;
-in the app_name field, please remember to use `intelowl` as the app name.
+The user would find the Notifications button on the top right of the page:
+
+<img style="border: 0.2px solid black" width=220 height=210 src="https://raw.githubusercontent.com/intelowlproject/IntelOwl/master/docs/static/notifications.png">
+
+There the user can read notifications provided by either the administrators or the IntelOwl Maintainers.
+
+As an Admin, if you want to add a notification to have it sent to all the users, you have to login to the Django Admin interface, go to the "Notifications" section and add it there.
+While adding a new notification, in the `body` section it is possible to even use HTML syntax, allowing to embed images, links, etc;
+in the `app_name field`, please remember to use `intelowl` as the app name.
 
 Everytime a new release is installed, once the backend goes up it will automatically create a new notification,
 having as content the latest changes described in the [CHANGELOG.md](https://github.com/intelowlproject/IntelOwl/blob/master/.github/CHANGELOG.md),

docs/source/Installation.md

@@ -100,12 +100,16 @@ In the `env_file_app`, configure different variables as explained below.
 * `DJANGO_SECRET`: random 50 chars key, must be unique. If you do not provide one, Intel Owl will automatically set a new secret on every run.
 * `INTELOWL_WEB_CLIENT_DOMAIN` (example: `localhost`/`mywebsite.com`): the web domain of your instance, this is used for generating links to analysis results.
 
+**Optional configuration**:
+* `OLD_JOBS_RETENTION_DAYS`: Database retention for analysis results (default: 3 days). Change this if you want to keep your old analysis longer in the database.
 
 ### Deprecated environment configuration
-The following variables are deprecated and will be removed in the future.
-The new way to configure plugin secrets is to use the `Plugin Secrets` page in the GUI.
-If you had previously specified any variables in the environment, run `docker exec -ti intelowl_uwsgi python3 manage.py migrate_secrets`
-and then remove those secrets from the env file.
+The following variables are related to the specific services integrated in IntelOwl.
+They are deprecated and will be removed in the future: the new way to configure plugin secrets is to use the `Plugin Secrets` page in the GUI.
+This change not only promotes a better user experience and overall security, but allows to configure these secrets at either user or org level instead of globally.
+
+If you had previously specified any variables in the environment, IntelOwl will migrate those variables for you once you update the software to the v4.1.0 and restart the containers.
+(Under the hood IntelOwl runs `docker exec -ti intelowl_uwsgi python3 manage.py migrate_secrets`). Then you can remove those secrets from the env file.
 
 ```text
 **Optional** variables needed to enable specific analyzers:
@@ -161,8 +165,6 @@ and then remove those secrets from the env file.
 * `CONNECTOR_YETI_KEY`: your own YETI instance key to use with `YETI` connector
 * `CONNECTOR_YETI_URL`: your own YETI instance API URL to use with `YETI` connector
 
-**Advanced** additional configuration:
-* `OLD_JOBS_RETENTION_DAYS`: Database retention for analysis results (default: 3 days). Change this if you want to keep your old analysis longer in the database.
 ```
 ### Database configuration (required)
 In the `env_file_postgres`, configure different variables as explained below.