Skip to content

Commit

Permalink
misc README updates (#1240)
Browse files Browse the repository at this point in the history 
* some quick readme updates

* cml pr create

* review feedback

* scrub  refs
  • Loading branch information
@dacbd
dacbd committed Nov 1, 2022
1 parent 1a2b245 commit bff9df1
Showing 1 changed file with 112 additions and 96 deletions.
208 changes: 112 additions & 96 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,14 @@ jobs:
run:
runs-on: ubuntu-latest
# optionally use a convenient Ubuntu LTS + DVC + CML image
# container: docker://ghcr.io/iterative/cml:0-dvc2-base1
# container: ghcr.io/iterative/cml:0-dvc2-base1
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
# may need to setup NodeJS & Python3 on e.g. self-hosted
# - uses: actions/setup-node@v2
# - uses: actions/setup-node@v3
# with:
# node-version: '16'
# - uses: actions/setup-python@v2
# - uses: actions/setup-python@v4
# with:
# python-version: '3.x'
- uses: iterative/setup-cml@v1
Expand All @@ -103,17 +103,17 @@ jobs:
run: |
# Post reports as comments in GitHub PRs
cat results.txt >> report.md
cml send-comment report.md
cml comment create report.md
```
## Usage
We helpfully provide CML and other useful libraries pre-installed on our
[custom Docker images](https://github.com/iterative/cml/blob/master/Dockerfile).
In the above example, uncommenting the field
`container: docker://ghcr.io/iterative/cml:0-dvc2-base1`) will make the runner
pull the CML Docker image. The image already has NodeJS, Python 3, DVC and CML
set up on an Ubuntu LTS base for convenience.
`container: ghcr.io/iterative/cml:0-dvc2-base1`) will make the runner pull the
CML Docker image. The image already has NodeJS, Python 3, DVC and CML set up on
an Ubuntu LTS base for convenience.

### CML Functions

Expand All @@ -124,18 +124,17 @@ report.
Below is a table of CML functions for writing markdown reports and delivering
those reports to your CI system.

| Function | Description | Example Inputs |
| ----------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------- |
| `cml runner` | Launch a runner locally or hosted by a cloud provider | See [Arguments](https://github.com/iterative/cml#arguments) |
| `cml publish` | Publicly host an image for displaying in a CML report | `<path to image> --title <image title> --md` |
| `cml send-comment` | Return CML report as a comment in your GitLab/GitHub workflow | `<path to report> --head-sha <sha>` |
| `cml send-github-check` | Return CML report as a check in GitHub | `<path to report> --head-sha <sha>` |
| `cml pr` | Commit the given files to a new branch and create a pull request | `<path>...` |
| `cml tensorboard-dev` | Return a link to a Tensorboard.dev page | `--logdir <path to logs> --title <experiment title> --md` |
| Function | Description | Example Inputs |
| ------------------------- | ---------------------------------------------------------------- | ----------------------------------------------------------- |
| `cml runner launch` | Launch a runner locally or hosted by a cloud provider | See [Arguments](https://github.com/iterative/cml#arguments) |
| `cml comment create` | Return CML report as a comment in your GitLab/GitHub workflow | `<path to report> --head-sha <sha>` |
| `cml check create` | Return CML report as a check in GitHub | `<path to report> --head-sha <sha>` |
| `cml pr create` | Commit the given files to a new branch and create a pull request | `<path>...` |
| `cml tensorboard connect` | Return a link to a Tensorboard.dev page | `--logdir <path to logs> --title <experiment title> --md` |

#### CML Reports

The `cml send-comment` command can be used to post reports. CML reports are
The `cml comment create` command can be used to post reports. CML reports are
written in markdown ([GitHub](https://github.github.com/gfm),
[GitLab](https://docs.gitlab.com/ee/user/markdown.html), or
[Bitbucket](https://confluence.atlassian.com/bitbucketserver/markdown-syntax-guide-776639995.html)
Expand All @@ -153,11 +152,12 @@ cat results.txt >> report.md

:framed_picture: **Images** Display images using the markdown or HTML. Note that
if an image is an output of your ML workflow (i.e., it is produced by your
workflow), you will need to use the `cml publish` function to include it a CML
report. For example, if `graph.png` is output by `python train.py`, run:
workflow), it can be uploaded and included automaticlly to your CML report. For
example, if `graph.png` is output by `python train.py`, run:

```bash
cml publish graph.png --md >> report.md
echo "![](./graph.png)" >> report.md
cml comment create report.md
```

### Getting Started
Expand Down Expand Up @@ -189,8 +189,8 @@ jobs:
run:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
- uses: iterative/setup-cml@v1
- name: Train model
env:
Expand All @@ -200,8 +200,8 @@ jobs:
python train.py
cat metrics.txt >> report.md
cml publish plot.png --md >> report.md
cml send-comment report.md
echo "![](./plot.png)" >> report.md
cml comment create report.md
```

3. In your text editor of choice, edit line 16 of `train.py` to `depth = 5`.
Expand Down Expand Up @@ -253,9 +253,9 @@ on: [push]
jobs:
run:
runs-on: ubuntu-latest
container: docker://ghcr.io/iterative/cml:0-dvc2-base1
container: ghcr.io/iterative/cml:0-dvc2-base1
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Train model
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -278,16 +278,16 @@ jobs:
echo "## Plots" >> report.md
echo "### Class confusions" >> report.md
dvc plots diff --target classes.csv --template confusion -x actual -y predicted --show-vega master > vega.json
vl2png vega.json -s 1.5 > plot.png
cml publish --md plot.png >> report.md
vl2png vega.json -s 1.5 > confusion_plot.png
echo "![](./confusion_plot.png)" >> report.md
# Publish regularization function diff
echo "### Effects of regularization" >> report.md
dvc plots diff --target estimators.csv -x Regularization --show-vega master > vega.json
vl2png vega.json -s 1.5 > plot.png
cml publish --md plot.png >> report.md
echo "![](./plot.png)" >> report.md
cml send-comment report.md
cml comment create report.md
```

> :warning: If you're using DVC with cloud storage, take note of environment
Expand Down Expand Up @@ -423,14 +423,14 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: iterative/setup-cml@v1
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Deploy runner on EC2
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
run: |
cml runner \
cml runner launch \
--cloud=aws \
--cloud-region=us-west \
--cloud-type=g4dn.xlarge \
Expand All @@ -440,10 +440,10 @@ jobs:
runs-on: [self-hosted, cml-gpu]
timeout-minutes: 50400 # 35 days
container:
image: docker://iterativeai/cml:0-dvc2-base1-gpu
image: ghcr.io/iterative/cml:0-dvc2-base1-gpu
options: --gpus all
steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v3
- name: Train model
env:
REPO_TOKEN: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
Expand All @@ -452,7 +452,7 @@ jobs:
python train.py
cat metrics.txt > report.md
cml send-comment report.md
cml comment create report.md
```

In the workflow above, the `deploy-runner` step launches an EC2 `g4dn.xlarge`
Expand All @@ -466,72 +466,83 @@ newly-launched instance. See [Environment Variables] below for details on the

#### Docker Images

The CML Docker image (`docker://iterativeai/cml`) comes loaded with Python,
CUDA, `git`, `node` and other essentials for full-stack data science. Different
versions of these essentials are available from different `iterativeai/cml`
The CML Docker image (`ghcr.io/iterative/cml` or `iterativeai/cml`) comes loaded
with Python, CUDA, `git`, `node` and other essentials for full-stack data
science. Different versions of these essentials are available from different
image tags. The tag convention is `{CML_VER}-dvc{DVC_VER}-base{BASE_VER}{-gpu}`:

| `{BASE_VER}` | Software included (`-gpu`) |
| ------------ | --------------------------------------------- |
| 0 | Ubuntu 18.04, Python 2.7 (CUDA 10.1, CuDNN 7) |
| 1 | Ubuntu 20.04, Python 3.8 (CUDA 11.2, CuDNN 8) |

For example, `docker://iterativeai/cml:0-dvc2-base1-gpu`, or
`docker://ghcr.io/iterative/cml:0-dvc2-base1`.
For example, `iterativeai/cml:0-dvc2-base1-gpu`, or
`ghcr.io/iterative/cml:0-dvc2-base1`.

#### Arguments

The `cml runner` function accepts the following arguments:
The `cml runner launch` function accepts the following arguments:

```
--help Show help [boolean]
--version Show version number [boolean]
--log Maximum log level
[choices: "error", "warn", "info", "debug"] [default: "info"]
--labels One or more user-defined labels for this runner
(delimited with commas) [default: "cml"]
--idle-timeout Seconds to wait for jobs before shutting down. Set
to -1 to disable timeout [default: 300]
--name Name displayed in the repository once registered
cml-{ID}
--no-retry Do not restart workflow terminated due to instance
disposal or GitHub Actions timeout
[boolean] [default: false]
--single Exit after running a single job
[boolean] [default: false]
--reuse Don't launch a new runner if an existing one has
the same name or overlapping labels
[boolean] [default: false]
--driver Platform where the repository is hosted. If not
specified, it will be inferred from the
environment [choices: "github", "gitlab"]
--repo Repository to be used for registering the runner.
If not specified, it will be inferred from the
environment
--token Personal access token to register a self-hosted
runner on the repository. If not specified, it
will be inferred from the environment
[default: "infer"]
--cloud Cloud to deploy the runner
[choices: "aws", "azure", "gcp", "kubernetes"]
--cloud-region Region where the instance is deployed. Choices:
[us-east, us-west, eu-west, eu-north]. Also
accepts native cloud regions [default: "us-west"]
--cloud-type Instance type. Choices: [m, l, xl]. Also supports
native types like i.e. t2.micro
--cloud-gpu GPU type.
[choices: "nogpu", "k80", "v100", "tesla"]
--cloud-hdd-size HDD size in GB
--cloud-ssh-private Custom private RSA SSH key. If not provided an
automatically generated throwaway key will be used
[default: ""]
--cloud-spot Request a spot instance [boolean]
--cloud-spot-price Maximum spot instance bidding price in USD.
Defaults to the current spot bidding price
[default: "-1"]
--cloud-startup-script Run the provided Base64-encoded Linux shell script
during the instance initialization [default: ""]
--cloud-aws-security-group Specifies the security group in AWS [default: ""]
--labels One or more user-defined labels for
this runner (delimited with commas)
[string] [default: "cml"]
--idle-timeout Time to wait for jobs before
shutting down (e.g. "5min"). Use
"never" to disable
[string] [default: "5 minutes"]
--name Name displayed in the repository
once registered
[string] [default: cml-{ID}]
--no-retry Do not restart workflow terminated
due to instance disposal or GitHub
Actions timeout [boolean]
--single Exit after running a single job
[boolean]
--reuse Don't launch a new runner if an
existing one has the same name or
overlapping labels [boolean]
--reuse-idle Creates a new runner only if the
matching labels don't exist or are
already busy [boolean]
--docker-volumes Docker volumes, only supported in
GitLab [array] [default: []]
--cloud Cloud to deploy the runner
[string] [choices: "aws", "azure", "gcp", "kubernetes"]
--cloud-region Region where the instance is
deployed. Choices: [us-east,
us-west, eu-west, eu-north]. Also
accepts native cloud regions
[string] [default: "us-west"]
--cloud-type Instance type. Choices: [m, l, xl].
Also supports native types like i.e.
t2.micro [string]
--cloud-permission-set Specifies the instance profile in
AWS or instance service account in
GCP [string] [default: ""]
--cloud-metadata Key Value pairs to associate
cml-runner instance on the provider
i.e. tags/labels "key=value"
[array] [default: []]
--cloud-gpu GPU type. Choices: k80, v100, or
native types e.g. nvidia-tesla-t4
[string]
--cloud-hdd-size HDD size in GB [number]
--cloud-ssh-private Custom private RSA SSH key. If not
provided an automatically generated
throwaway key will be used [string]
--cloud-spot Request a spot instance [boolean]
--cloud-spot-price Maximum spot instance bidding price
in USD. Defaults to the current spot
bidding price [number] [default: -1]
--cloud-startup-script Run the provided Base64-encoded
Linux shell script during the
instance initialization [string]
--cloud-aws-security-group Specifies the security group in AWS
[string] [default: ""]
--cloud-aws-subnet, Specifies the subnet to use within
--cloud-aws-subnet-id AWS [string] [default: ""]
```

#### Environment Variables
Expand All @@ -556,12 +567,13 @@ CML support proxy via known environment variables `http_proxy` and

#### On-premise (Local) Runners

This means using on-premise machines as self-hosted runners. The `cml runner`
function is used to set up a local self-hosted runner. On a local machine or
on-premise GPU cluster, [install CML as a package](#local-package) and then run:
This means using on-premise machines as self-hosted runners. The
`cml runner launch` function is used to set up a local self-hosted runner. On a
local machine or on-premise GPU cluster,
[install CML as a package](#local-package) and then run:

```bash
cml runner \
cml runner launch \
--repo=$your_project_repository_url \
--token=$PERSONAL_ACCESS_TOKEN \
--labels="local,runner" \
Expand All @@ -577,9 +589,13 @@ pre-installed in a custom Docker image pulled by a CI runner. You can also
install CML as a package:

```bash
npm i -g @dvcorg/cml
npm install --location=global @dvcorg/cml
```

You can use `cml` without node by downloading the correct standalone binary for
your system from the asset section of the
[releases](https://github.com/iterative/cml/releases).

You may need to install additional dependencies to use DVC plots and Vega-Lite
CLI commands:

Expand All @@ -599,15 +615,15 @@ CML and Vega-Lite package installation require the NodeJS package manager
use a set up action to install NodeJS:

```bash
uses: actions/setup-node@v2
uses: actions/setup-node@v3
with:
node-version: '16'
```

- **GitLab**: Requires direct installation.

```bash
curl -sL https://deb.nodesource.com/setup_12.x | bash
curl -sL https://deb.nodesource.com/setup_16.x | bash
apt-get update
apt-get install -y nodejs
```
Expand Down

0 comments on commit bff9df1

Please sign in to comment.