Consider removing Dynaconf

[noklam]

Extend Discussion

Settings management (default value, template, validation, parsing)
Doesn't seem to be particular useful as overlap with configloader already. Validation these days are not really useful, there are probably better choices. The only thing that I recalled is custom config loader validation but 0.19 there is only OmegaConfigLoader and we expect most project stick with it. If the validation is useful, maybe we should consider validating datasets etc?

Fully support of environment to override existing config
? KEDRO_LOGGING_CONFIG?

Optional layer environment

• Kedro's ConfigLoader (base, local, prod etc)

Secret Vault & Credential Management
Seems like the most interesting part for Kedro

CLI
Unknown, the dynaconf init creates a set of files that are very different from Kedro. dynaconf write py -v key=value is a way to update settings.py dynamically.

Dynaconf comes with many dependencies and currently we are only using it for schema validation.

Alternative:
Actually leverage features of dyanconf like dotenv native support, credentials management etc

More:

Settings https://www.dynaconf.com/settings_files/

.py - Not Recommended but supported for backwards compatibility.

[noklam]

Some historical context:

Refactor the way Kedro loads settings.py (private link)

Refactor the way Kedro loads settings and make it similar to Django (?)

Here's a list of links for inspiration

1. https://docs.djangoproject.com/en/dev/topics/settings/#using-settings-without-setting-django-settings-module
2. https://docs.djangoproject.com/en/dev/topics/settings/#using-settings-in-python-code
3. https://docs.djangoproject.com/en/dev/topics/settings/#default-settings
4. https://github.com/django/django/blob/master/django/conf/global_settings.py
5. https://github.com/django/django/blob/master/django/conf/init.py#L96
6. https://github.com/django/django/blob/master/django/conf/init.py#L131

Double-check that this works with multiprocessing on Windows, i.e. spawn multiprocessing

Example resulting API

from kedro.framework.project import settings, configure_project
configure_project(package_name) to be called in all places immediately after _add_src_to_path

• All plugins, Kedro code and user code can access settings by settings.CONF_ROOT, etc.
• No need to pass package_name to KedroSession and KedroContext
• Make the change backwards compatible and issue a deprecation warning for package_name, saying that it is no longer needed and will be removed from 0.18

[astrojuanlu]

I'll be very honest, I was never a fan of configure_project and its aunt bootstrap_project. With all our talk about not using global variables in our training materials, I've always found it weird that we seem to get a pass:

kedro/kedro/framework/project/__init__.py

Lines 293 to 294 in a53500a

	global PACKAGE_NAME # noqa: PLW0603
	PACKAGE_NAME = package_name

[astrojuanlu]

In any case, the critical thing here is probably settings.<VAR> access, I guess?

[astrojuanlu]

If we ever wanted native support for Hashicorp (or OpenBao https://github.com/openbao/openbao) we could do that ourselves, possibly through a plugin:

https://github.com/dynaconf/dynaconf/blob/0177b68a14036f32adc06f3e3596d31f945c31aa/dynaconf/loaders/vault_loader.py#L14-L15

[noklam]

Found this today