From aae91937f6e466a55c379c33835d1b896b107125 Mon Sep 17 00:00:00 2001 From: Ruslan Kuprieiev Date: Wed, 9 Aug 2023 02:44:33 +0300 Subject: [PATCH] docs: list/import/fetch: document --config/--remote/--remote-config (#4722) Fixes #4712 --- content/docs/command-reference/get.md | 27 +++- content/docs/command-reference/import.md | 116 ++++++++++++++++++ content/docs/command-reference/list.md | 24 ++++ .../user-guide/project-structure/dvc-files.md | 14 ++- 4 files changed, 174 insertions(+), 7 deletions(-) diff --git a/content/docs/command-reference/get.md b/content/docs/command-reference/get.md index 711b74833a..eb74e4995e 100644 --- a/content/docs/command-reference/get.md +++ b/content/docs/command-reference/get.md @@ -9,7 +9,10 @@ directory. ```usage usage: dvc get [-h] [-q | -v] [-o ] [--rev ] - [--show-url] [-j ] [-f] url path + [--show-url] [-j ] [-f] + [--config ] [--remote ] + [--remote-config [= ...]] + url path positional arguments: url Location of DVC or Git repository to download from @@ -82,6 +85,16 @@ name. storage location (URL) of the target data. If `path` is a Git-tracked file, this option is ignored. +- `--config ` - path to a [config file](/doc/command-reference/config) + that will be merged with the config in the target repository. + +- `--remote ` - name of the `dvc remote` to set as a default in the target + repository. + +- `--remote-config [= ...]` - `dvc remote` config options to merge + with a remote's config (default or one specified by `--remote`) in the target + repository. + - `-h`, `--help` - prints the usage/help message, and exit. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no @@ -219,3 +232,15 @@ Untracked files: [get started example repo]: https://github.com/iterative/example-get-started [switching between versions]: /doc/start/data-management/data-versioning#switching-between-versions + +## Example: Set AWS profile for default remote + +```cli +$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile +``` + +## Example: Set default remote + +```cli +$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote +``` diff --git a/content/docs/command-reference/import.md b/content/docs/command-reference/import.md index c28e999990..9963774dc1 100644 --- a/content/docs/command-reference/import.md +++ b/content/docs/command-reference/import.md @@ -12,6 +12,8 @@ usage: dvc import [-h] [-q | -v] [-o ] [--rev ] [--no-exec | --no-download] [-j ] + [--config ] [--remote ] + [--remote-config [= ...]] url path positional arguments: @@ -118,6 +120,16 @@ file. speed up the operation. Note that the default value can be set in the source repo using the `jobs` config option of `dvc remote modify`. +- `--config ` - path to a [config file](/doc/command-reference/config) + that will be merged with the config in the target repository. + +- `--remote ` - name of the `dvc remote` to set as a default in the target + repository. + +- `--remote-config [= ...]` - `dvc remote` config options to merge + with a remote's config (default or one specified by `--remote`) in the target + repository. + - `-h`, `--help` - prints the usage/help message, and exit. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no @@ -386,3 +398,107 @@ C, otherwise the import chain resolution would fail. The `dvc remote default` for all repos in the import chain must also be accessible (repo C needs to have all the appropriate credentials). + +## Example: Set default remote + +```cli +$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote +... +$ cat prepared.dvc +deps: + - path: data/prepared + repo: + url: https://github.com/iterative/example-get-started-s3 + rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b + remote: myremote +outs: + - md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir + size: 27 + nfiles: 3 + path: prepared +``` + +## Example: Set AWS profile for default remote + +```cli +$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile +... +$ cat prepared.dvc +deps: + - path: data/prepared + repo: + url: https://github.com/iterative/example-get-started-s3 + rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b + remote: + profile: myprofile +outs: + - md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir + size: 27 + nfiles: 3 + path: prepared +``` + +## Example: Create new AWS S3 remote and set it as default + +If remote with that name already exists, its config will be merged with options +provided by `--remote-config`. + +```cli +$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared \ + --remote myremote \ + --remote-config url=s3://mybucket/mypath profile=myprofile +... +$ cat prepared.dvc +deps: + - path: data/prepared + repo: + url: https://github.com/iterative/example-get-started-s3 + rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b + config: + core: + remote: myremote + remote: + myremote: + url: s3://mybucket/mypath + profile: myprofile +outs: + - md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir + size: 27 + nfiles: 3 + path: prepared +``` + +## Example: Set AWS secret keys for particular remote + +In this example, instead of using `--remote myremote` with `--remote-config` and +exposing your secrets in dvcfile, you could use `--config` to use a gitignored +config file. The format of the config file is the same as produced by +`dvc config`. + +```cli +$ cat myconfig +[core] + remote = myremote + +[remote "myremote"] + access_key_id = myaccesskeyid + secret_access_key = mysecretaccesskey +$ cat .gitignore # make sure you are not commiting this file to git +... +/myconfig +... +$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --config myconfig +... +$ cat prepared.dvc +deps: + - path: data/prepared + repo: + url: https://github.com/iterative/example-get-started-s3 + rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b + config: myconfig +outs: + - md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir + size: 27 + nfiles: 3 + path: prepared +``` diff --git a/content/docs/command-reference/list.md b/content/docs/command-reference/list.md index 7f73d933b3..7fe902d606 100644 --- a/content/docs/command-reference/list.md +++ b/content/docs/command-reference/list.md @@ -12,6 +12,8 @@ and by Git. ```usage usage: dvc list [-h] [-q | -v] [-R] [--dvc-only] [--json] [--rev []] + [--config ] [--remote ] + [--remote-config [= ...]] url [path] positional arguments: @@ -71,6 +73,16 @@ accessed with `dvc get`, `dvc import`, or `dvc.api`. - `--json` - prints the command's output in easily parsable JSON format, instead of a human-readable table. +- `--config ` - path to a [config file](/doc/command-reference/config) + that will be merged with the config in the target repository. + +- `--remote ` - name of the `dvc remote` to set as a default in the target + repository. + +- `--remote-config [= ...]` - `dvc remote` config options to merge + with a remote's config (default or one specified by `--remote`) in the target + repository. + - `-h`, `--help` - prints the usage/help message, and exit. - `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no @@ -161,3 +173,15 @@ installed): ```cli $ dvc list . -R --dvc-only | xargs python -m zipfile -c data.zip ``` + +## Example: Set AWS profile for default remote + +```cli +$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile +``` + +## Example: Set default remote + +```cli +$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote +``` diff --git a/content/docs/user-guide/project-structure/dvc-files.md b/content/docs/user-guide/project-structure/dvc-files.md index 4288411937..6f3bbf958e 100644 --- a/content/docs/user-guide/project-structure/dvc-files.md +++ b/content/docs/user-guide/project-structure/dvc-files.md @@ -73,14 +73,16 @@ The following subfields may be present under `deps` entries: | `md5`
`etag`
`checksum` | Only in external dependencies created with `dvc import-url`: Hash value of the imported file or directory. MD5 is used for local paths and SSH; [ETag] for HTTP, S3, GCS, and Azure; and a special _checksum_ for HDFS and WebHDFS. | | `size` | Size of the file or directory (sum of all files). | | `nfiles` | If this dependency is a directory, the number of files inside (recursive). | -| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, and `rev_lock` (detailed below). | +| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, `rev_lock`, `config` and `remote` (detailed below). | ### Dependency `repo` subfields: -| Field | Description | -| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------- | -| `url` | URL of Git repository with source DVC project | -| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. | -| `rev_lock` | Git commit hash of the external DVC repository at the time of importing or updating the dependency (with `dvc update`) | +| Field | Description | +| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `url` | URL of Git repository with source DVC project | +| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. | +| `rev_lock` | Git commit hash of the external DVC repository at the time of importing or updating the dependency (with `dvc update`) | +| `config` | When `dvc import --config` is used: Path to a [config file](/doc/command-reference/config) that will be merged with the config in the target repository. When both `--remote` and `--remote-config` are used: config options that will be merged with the config in the target repository. See examples section in`dvc import`. | +| `remote` | Only when `dvc import --remote` or `--remote-config` is used: name of the `dvc remote` to set as a default or remote config options to merge with a default remote's config in the target repository. See examples section in `dvc import`. | [git revision]: https://git-scm.com/docs/revisions