feat(NODE-1475): Configuration revamp (define config structure and co…

…nfig tool) (#1539)

[NODE-1360](https://dfinity.atlassian.net/browse/NODE-1360)

Changes pulled out of: #1190

This PR defines the config structure and config tool, but does not yet
integrate the config tool into SetupOS. This PR should not have an
impact on the ic-os. Later, the config tool will be utilized for config
sanitization, organization, access, and propagation.

[NODE-1360]:
https://dfinity.atlassian.net/browse/NODE-1360?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

ic-os/components/init/bootstrap-ic-node/guestos/bootstrap-ic-node.sh

@@ -135,6 +135,10 @@ write_metric_attr "guestos_boot_action" \
     "GuestOS boot action" \
     "gauge"
 
+# /boot/config/CONFIGURED serves as a tag to indicate that the one-time bootstrap configuration has been completed.
+# If the `/boot/config/CONFIGURED` file is not present, the boot sequence will
+# search for a virtual USB stick (the bootstrap config image)
+# containing the injected configuration files, and create the file.
 if [ -f /boot/config/CONFIGURED ]; then
     echo "Bootstrap completed already"
 fi

ic-os/docs/Configuration.adoc

@@ -20,7 +20,7 @@ SetupOS validates, sanitizes, and copies all of its configuration files to the H
   deployment.json                 # Deployment-specific configurations
   nns_public_key.pem              # NNS public key
 
-Refer to link:../components/setupos-scripts/config.sh[config.sh] link:../components/setupos-scripts/setup-hostos-config.sh[setup-hostos-config.sh]
+Refer to link:../../rs/ic_os/config/README.md[rs/ic_os/config] & link:../components/setupos-scripts/setup-hostos-config.sh[setup-hostos-config.sh]
 
 === HostOS -> GuestOS
 

ic-os/guestos/docs/ConfigStore.adoc

@@ -1,6 +1,6 @@
 = GuestOS Config Store
 
-This document describes the contents of the GuestOS *config* partition (*/dev/vda3* in the GuestOS disk image). The config partition stores information that must be preserved across system upgrades and needs to be available during early boot time. Consequently, this information cannot reside within the encrypted payload data partition.
+This document calls out some of the contents of the GuestOS *config* partition (*/dev/vda3* in the GuestOS disk image). The config partition stores information that must be preserved across system upgrades and needs to be available during early boot time. Consequently, this information cannot reside within the encrypted payload data partition.
 
 Currently, all contents in the config partition are stored as plain-text without integrity protection.
 
@@ -22,108 +22,10 @@ This file contains the key material used to derive the wrapping key for all bloc
 
 In the absence of a sealing key (which will be available in SEV-protected trusted execution environments), the `store.keyfile` is stored as plain-text. Once a sealing key becomes available, it should be used to wrap the contents of this file.
 
-=== ssh/
-
-This directory contains SSH host keys. These keys must be persisted across upgrades and are transferred to `/etc/ssh` during the boot process.
-
-=== node_exporter/
-
-This directory contains the Node Exporter TLS keys. These keys must be persisted across upgrades and are transferred to `/etc/node_exporter` during the boot process.
-
-=== accounts_ssh_authorized_keys/
-
-This directory contains individual files named `admin`, `backup`, `readonly` and `root`. The contents of these files serve as `authorized_keys` for their respective role account. This means that, for example, `accounts_ssh_authorized_keys/admin` is transferred to `~admin/.ssh/authorized_keys` on the target system.
-
-* *admin*: ...
-* *backup / readonly*: These files can only be modified via an NNS proposal, and are in place for subnet recovery or issue debugging purposes.
-* *root*: Used for development/debug builds only.
-
-This directory and any file in it is optional. By default, no authorized key is installed for any account, meaning no one has SSH access to the GuestOS.
-
-=== node_operator_private_key.pem
-
-This file contains the Node Operator private key, which is registered with the NNS and used to sign the IC join request. The private key can be generated using one of the following commands:
-
-    dfx identity new --disable-encryption node_operator
-    cp ~/.config/dfx/identity/node_operator/identity.pem ./node_operator_private_key.pem
-
-Or
-
-    quill generate --pem-file node_operator_private_key.pem
-
-=== network.conf
-
-Network configuration parameters.
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- *ipv6_address*: The IPv6 address of the node, used for the node to "identify" itself (via the registry). All public IC services are offered through this address, which will be assigned to the enp1s0 interface. It is used as the "private" management access to the node. If left blank, SLAAC is used on the interface.
-
-- *ipv6_gateway*: The default IPv6 gateway, only meaningful if ipv6_address is also provided.
-
-- *hostname*: The hostname, which can be any text in principle but is generally derived from the ID of the physical host (e.g., MAC address).
-
-Note: if this file is not given, the system will fall back to network auto configuration.
-
-=== nns.conf
-
-The IP address(es) of NNS node(s).
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- *nns_url*: The URL (HTTP) of the NNS node(s). If multiple URLs are provided, separate them with whitespace. If this key is not specified, http://127.0.0.1:8080 is assumed by default (which only works for nodes that do not require registration).
-
-This configuration is used when generating the replica configuration to fill in the nns_url placeholder.
-
-=== nns_public_key.pem
-
-This file must be a text file containing the public key of the NNS to be used.
-
 == Development configuration files
 
 These configuration files should only be used for development and testing purposes.
 
-=== backup.conf
-
-This file configures the usage of the backup spool directory.
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- *backup_retention_time_secs*: The maximum age of any file or directory kept in the backup spool.
-
-- *backup_purging_interval_secs*: The interval at which the backup spool directory will be scanned for files to delete.
-
-This configuration file should only be used for testnet deployments (to achieve shorter retention times) and must be missing for production deployments, where suitable production default values are assumed.
-
-=== filebeat.conf
-
-Configures filebeat to export logs out of the system.
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- elasticsearch_hosts: Space-separated lists of hosts to ship logs to.
-- elasticsearch_tags: Space-separated list of tags to apply to exported log records.
-
-If left unspecified, filebeat will be left unconfigured and no logs are exported.
-
-=== socks_proxy.conf
-
-Configuration for socks proxy.
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- socks_proxy: URL of the socks proxy to use. E.g socks5://socksproxy.com:1080
-
-=== bitcoin_addr.conf
-
-Configuration for bitcoin adapter.
-
-Must be a file of key/value pairs separated by "=" (one per line) with the following possible keys:
-
-- bitcoind_addr: Address of the bitcoind to be contacted by bitcoin adapter service.
-
-If left unspecified, the bitcoin adapter will not work properly due to lack of external system to contact.
-
 == Injecting external state
 
 *Typical bootstrap process:* On first boot, the system will perform technical initialization (filesystems, etc.) and afterwards, initialize itself to act as a node in the IC. The node is initialized using key generation on the node itself (such that the private key never leaves the node) and through joining the IC (the node gets the rest of its state via joining the IC). "Registration" to the target IC is initiated by the node itself by sending a Node Operator-signed "join" request to its NNS. 
@@ -141,14 +43,3 @@ This behavior is suitable for the following use cases:
 
 - Externally controlled join of a node to a subnet: In this case, ic-prep is used to prepare key material to the node, while ic-admin is used to modify the target NNS such that it "accepts" the new node as part of the IC.
 
-=== ic_crypto
-
-Externally generated cryptographic keys.
-
-Must be a directory with contents matching the internal representation of the ic_crypto directory. When given, this provides the private keys of the node. If not given, the node will generate its own private/public key pair.
-
-=== ic_registry_local_store 
-
-Initial registry state.
-
-Must be a directory with contents matching the internal representation of the ic_registry_local_store. When given, this provides the initial state of the registry. If not given, the node will fetch (initial) registry state from the NNS.

rs/ic_os/config/BUILD.bazel

@@ -1,20 +1,54 @@
-load("@rules_rust//rust:defs.bzl", "rust_library")
+load("@rules_rust//rust:defs.bzl", "rust_binary", "rust_library", "rust_test")
 
 package(default_visibility = ["//rs:ic-os-pkg"])
 
 DEPENDENCIES = [
     # Keep sorted.
+    "//rs/ic_os/utils",
+    "//rs/types/types",
     "@crate_index//:anyhow",
+    "@crate_index//:clap",
+    "@crate_index//:regex",
+    "@crate_index//:serde",
+    "@crate_index//:serde_json",
+    "@crate_index//:serde_with",
+    "@crate_index//:url",
 ]
 
+DEV_DEPENDENCIES = [
+    # Keep sorted.
+    "@crate_index//:once_cell",
+    "@crate_index//:tempfile",
+]
+
+MACRO_DEPENDENCIES = []
+
+ALIASES = {}
+
 rust_library(
-    name = "config",
+    name = "config_lib",
     srcs = glob(
         ["src/**/*.rs"],
+        exclude = ["src/main.rs"],
     ),
-    aliases = {},
     crate_name = "config",
     edition = "2021",
-    proc_macro_deps = [],
     deps = DEPENDENCIES,
 )
+
+rust_binary(
+    name = "config",
+    srcs = ["src/main.rs"],
+    aliases = ALIASES,
+    crate_name = "config",
+    edition = "2021",
+    proc_macro_deps = MACRO_DEPENDENCIES,
+    deps = [":config_lib"] + DEPENDENCIES,
+)
+
+rust_test(
+    name = "config_lib_test",
+    crate = ":config_lib",
+    # You may add other deps that are specific to the test configuration
+    deps = DEV_DEPENDENCIES,
+)

rs/ic_os/config/Cargo.toml

@@ -5,3 +5,23 @@ edition = "2021"
 
 [dependencies]
 anyhow = { workspace = true }
+ic-types = { path = "../../types/types" }
+clap = { workspace = true }
+utils = { path = "../utils" }
+url = { workspace = true }
+serde_json = { workspace = true }
+serde = { workspace = true }
+serde_with = "1.6.2"
+regex = { workspace = true }
+
+[dev-dependencies]
+once_cell = "1.8"
+tempfile = { workspace = true }
+
+[lib]
+name = "config"
+path = "src/lib.rs"
+
+[[bin]]
+name = "config"
+path = "src/main.rs"

rs/ic_os/config/README.md

@@ -0,0 +1,11 @@
+# IC-OS Config
+
+IC-OS Config is responsible for managing the configuration of IC-OS images. 
+
+SetupOS transforms user-facing configuration files (like `config.ini`, `deployment.json`, etc.) into a SetupOSConfig struct. Then, in production, configuration is propagated from SetupOS → HostOS → GuestOS (→ replica) via the HostOSConfig and GuestOSConfig structures.
+
+All access to configuration and the config partition should go through the config structures.
+
+For testing, IC-OS Config is also used to create HostOS and GuestOS configuration directly.
+
+For details on the IC-OS configuration mechanism, refer to [ic-os/docs/Configuration.adoc](../../../ic-os/docs/Configuration.adoc)