cml-runner self-hosted issues

[casperdcl]

• ~/.cml/cml-*/ is not named according to --name
• ~/.cml/cml-*/ is not cleaned up on exit (should be cleaned esp. if no --name)
• don't pass sensitive env vars (Add exclusion list for environment variables #802)
• cache runner setup script in ~/.cml/ (Runner package is downloaded every single time #615)
• feature request: local secrets (cml runner --env flag similar to docker run --env)
• feature request: user ID (cml runner --user flag similar to docker run --user)
• feature request: auto-mount sockets for docker-in-docker support (runner: Docker socket mounts #799)

[0x2b3bfa0]

Variables stored in the GitHub Actions self–hosted runner .env file will update the existing ones, and not replace them, as you can see in actions/runner@f259e57 → src/Runner.Listener/Program.cs:152.

We can use env(1) or the env option on Node JS' child_process functions to reset the environment and avoid the propagation of variables other than the ones in that file.

The same goes for GitLab and the environment configuration, and chances are Bitbucket Pipelines self–hosted runners will behave similarly.

Shell

env --ignore-environment ./run.sh

Node JS

It would suffice to filter out the environment on utils.js so it doesn't propagate to the child process.

cml/src/utils.js

Line 13 in 2e9a21d

{ ...process.env },

cml/src/drivers/github.js

Lines 227 to 237 in 2e9a21d

	await exec(
	`${resolve(
	workdir,
	'config.sh'
	)} --unattended --token "${await this.runnerToken()}" --url "${
	this.repo
	}" --name "${name}" --labels "${labels}" --work "${resolve(
	workdir,
	'_work'
	)}"`
	);

[DavidGOrtega]

feature request: local secrets (cml runner --env flag similar to docker run --env)

Why do we exactly want this feature @casperdcl ?

[0x2b3bfa0]

https://iterativeai.slack.com/archives/C01900GSB4J/p1632232145260800?thread_ts=1632223956.257700&cid=C01900GSB4J

[casperdcl]

just realised this is also a problem on all runners (not just manual self-hosted)

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: iterative/setup-cml@v1
    - uses: actions/checkout@v2
    - run: |
        cml runner --labels=cml-runner --cloud=aws --cloud-type=t2.micro
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        REPO_TOKEN: ${{ secrets.CML_CI_TOKEN }}
  train:
    runs-on: [self-hosted, cml-runner]
    steps:
    - run: env # why does this list AWS_ACCESS_KEY_ID & AWS_SECRET_ACCESS_KEY ??

should really either

• whitelist-only vanilla CI vars (GITHUB_ENV etc) or
• at least blacklist CML config vars (AWS_*ACCESS_KEY etc)

[dacbd]

What do you mean @casperdcl ? These envs are set by the TPI with the via cml.sh which runs exec on cml runner and starts the GitHub runner with spawn. The default for child_process.spawn is to inherit process.env

cml/src/drivers/github.js

Line 267 in 68ddbe4

return spawn(resolve(workdir, 'run.sh'), {

Maybe you want to override that by adding env: {} into the options from the above?

[dacbd]

I miss read that, I suspect the above is how they are populated.

I agree they probably shouldn't be there and the user should re-authenticate if they need them so there isn't any accidental use of creds that are meant for cml to create/run/destroy instances.

[DavidGOrtega]

There is not a lot of work to do here.
As we discussed the GH runner has the "ability" to propagate its ENV vars.
Here GH runner inherits the CML wrapper vars.
What we should do to fix this and not allow the GH runner to behave this way is simply pass an empty ENV vars in the exec. This way the ENV vars that will always access the runner will be the CI vars and workflow.