Skip to content

Commit

Permalink
docs: list/import/fetch: document --config/--remote/--remote-config (#…
Browse files Browse the repository at this point in the history 
…4722)

Fixes #4712
  • Loading branch information
@efiop
efiop committed Aug 8, 2023
1 parent 5f70156 commit aae9193
Show file tree
Hide file tree
Showing 4 changed files with 174 additions and 7 deletions.
27 changes: 26 additions & 1 deletion content/docs/command-reference/get.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ directory.

```usage
usage: dvc get [-h] [-q | -v] [-o <path>] [--rev <commit>]
[--show-url] [-j <number>] [-f] url path
[--show-url] [-j <number>] [-f]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url path
positional arguments:
url Location of DVC or Git repository to download from
Expand Down Expand Up @@ -82,6 +85,16 @@ name.
storage location (URL) of the target data. If `path` is a Git-tracked file,
this option is ignored.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -219,3 +232,15 @@ Untracked files:
[get started example repo]: https://github.com/iterative/example-get-started
[switching between versions]:
/doc/start/data-management/data-versioning#switching-between-versions

## Example: Set AWS profile for default remote

```cli
$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
```

## Example: Set default remote

```cli
$ dvc get https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
```
116 changes: 116 additions & 0 deletions content/docs/command-reference/import.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ usage: dvc import [-h] [-q | -v]
[-o <path>] [--rev <commit>]
[--no-exec | --no-download]
[-j <number>]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url path
positional arguments:
Expand Down Expand Up @@ -118,6 +120,16 @@ file.
speed up the operation. Note that the default value can be set in the source
repo using the `jobs` config option of `dvc remote modify`.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -386,3 +398,107 @@ C, otherwise the import chain resolution would fail.

The `dvc remote default` for all repos in the import chain must also be
accessible (repo C needs to have all the appropriate credentials).

## Example: Set default remote

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
remote: myremote
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Set AWS profile for default remote

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
remote:
profile: myprofile
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Create new AWS S3 remote and set it as default

If remote with that name already exists, its config will be merged with options
provided by `--remote-config`.

```cli
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared \
--remote myremote \
--remote-config url=s3://mybucket/mypath profile=myprofile
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
config:
core:
remote: myremote
remote:
myremote:
url: s3://mybucket/mypath
profile: myprofile
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```

## Example: Set AWS secret keys for particular remote

In this example, instead of using `--remote myremote` with `--remote-config` and
exposing your secrets in dvcfile, you could use `--config` to use a gitignored
config file. The format of the config file is the same as produced by
`dvc config`.

```cli
$ cat myconfig
[core]
remote = myremote
[remote "myremote"]
access_key_id = myaccesskeyid
secret_access_key = mysecretaccesskey
$ cat .gitignore # make sure you are not commiting this file to git
...
/myconfig
...
$ dvc import https://github.com/iterative/example-get-started-s3 data/prepared --config myconfig
...
$ cat prepared.dvc
deps:
- path: data/prepared
repo:
url: https://github.com/iterative/example-get-started-s3
rev_lock: 8141b41c5be682ced15136ed84b59468b68fd66b
config: myconfig
outs:
- md5: e784c380dd9aa9cb13fbe22e62d7b2de.dir
size: 27
nfiles: 3
path: prepared
```
24 changes: 24 additions & 0 deletions content/docs/command-reference/list.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ and by Git.
```usage
usage: dvc list [-h] [-q | -v] [-R] [--dvc-only]
[--json] [--rev [<commit>]]
[--config <path>] [--remote <name>]
[--remote-config [<name>=<value> ...]]
url [path]
positional arguments:
Expand Down Expand Up @@ -71,6 +73,16 @@ accessed with `dvc get`, `dvc import`, or `dvc.api`.
- `--json` - prints the command's output in easily parsable JSON format, instead
of a human-readable table.

- `--config <path>` - path to a [config file](/doc/command-reference/config)
that will be merged with the config in the target repository.

- `--remote <name>` - name of the `dvc remote` to set as a default in the target
repository.

- `--remote-config [<name>=<value> ...]` - `dvc remote` config options to merge
with a remote's config (default or one specified by `--remote`) in the target
repository.

- `-h`, `--help` - prints the usage/help message, and exit.

- `-q`, `--quiet` - do not write anything to standard output. Exit with 0 if no
Expand Down Expand Up @@ -161,3 +173,15 @@ installed):
```cli
$ dvc list . -R --dvc-only | xargs python -m zipfile -c data.zip
```

## Example: Set AWS profile for default remote

```cli
$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote-config profile=myprofile
```

## Example: Set default remote

```cli
$ dvc list https://github.com/iterative/example-get-started-s3 data/prepared --remote myremote
```
14 changes: 8 additions & 6 deletions content/docs/user-guide/project-structure/dvc-files.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,16 @@ The following subfields may be present under `deps` entries:
| `md5`<br/>`etag`<br/>`checksum` | Only in <abbr>external dependencies</abbr> created with `dvc import-url`: Hash value of the imported file or directory. MD5 is used for local paths and SSH; [ETag] for HTTP, S3, GCS, and Azure; and a special _checksum_ for HDFS and WebHDFS. |
| `size` | Size of the file or directory (sum of all files). |
| `nfiles` | If this dependency is a directory, the number of files inside (recursive). |
| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, and `rev_lock` (detailed below). |
| `repo` | Only in external dependencies created with `dvc import`: It can contain `url`, `rev`, `rev_lock`, `config` and `remote` (detailed below). |

### Dependency `repo` subfields:

| Field | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | URL of Git repository with source DVC project |
| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. |
| `rev_lock` | Git commit hash of the external <abbr>DVC repository</abbr> at the time of importing or updating the dependency (with `dvc update`) |
| Field | Description |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `url` | URL of Git repository with source DVC project |
| `rev` | Only when `dvc import --rev` is used: Specific commit hash, branch or tag name, etc. (a [Git revision]) used to import the dependency from. |
| `rev_lock` | Git commit hash of the external <abbr>DVC repository</abbr> at the time of importing or updating the dependency (with `dvc update`) |
| `config` | When `dvc import --config` is used: Path to a [config file](/doc/command-reference/config) that will be merged with the config in the target repository. When both `--remote` and `--remote-config` are used: config options that will be merged with the config in the target repository. See examples section in`dvc import`. |
| `remote` | Only when `dvc import --remote` or `--remote-config` is used: name of the `dvc remote` to set as a default or remote config options to merge with a default remote's config in the target repository. See examples section in `dvc import`. |

[git revision]: https://git-scm.com/docs/revisions

0 comments on commit aae9193

Please sign in to comment.