no longer produces /neurodocker/startup.sh in docker recipes, in effect ignores neurodocker/templates/_header.yaml?

[yarikoptic]

Discovered while working on

• Update (neuro)docker image PeerHerholz/BIDSonym#81

although there is neurodocker/templates/_header.yaml stating that

# Instructions to be run in the beginning of every Dockerfile generated by
# Neurodocker.

such promise is not fulfilled:

$ docker run -i --rm repronim/neurodocker:1.0.0 generate docker --base-image ubuntu:20.04 --pkg-manager apt
# Generated by Neurodocker and Reproenv.

FROM ubuntu:20.04

# Save specification to JSON.
RUN printf '{ \
  "pkg_manager": "apt", \
  "existing_users": [ \
    "root" \
  ], \
  "instructions": [ \
    { \
      "name": "from_", \
      "kwds": { \
        "base_image": "ubuntu:20.04" \
      } \
    } \
  ] \
}' > /.reproenv.json
# End saving to specification to JSON.

[yarikoptic]

archaeology expedition notes:

in 0.7.0 we had --base not --base-image and that worked

❯ neurodocker generate docker --pkg-manager apt --base neurodebian
[NEURODOCKER 2024-06-12 10:37:09,907 WARNING]: A newer version (1.0.0) of ReproNim/neurodocker is available. You are using 0.7.0
# Generated by: Neurodocker version 0.7.0
# Latest release: Neurodocker version 1.0.0
# Timestamp: 2024/06/12 14:37:09 UTC
# 
# Thank you for using Neurodocker. If you discover any issues
# or ways to improve this software, please submit an issue or
# pull request on our GitHub repository:
# 
#     https://github.com/ReproNim/neurodocker

FROM neurodebian

USER root

ARG DEBIAN_FRONTEND="noninteractive"

ENV LANG="en_US.UTF-8" \
    LC_ALL="en_US.UTF-8" \
    ND_ENTRYPOINT="/neurodocker/startup.sh"
RUN export ND_ENTRYPOINT="/neurodocker/startup.sh" \
    && apt-get update -qq \
    && apt-get install -y -q --no-install-recommends \
           apt-utils \
           bzip2 \
           ca-certificates \
           curl \
           locales \
           unzip \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* \
    && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen \
    && dpkg-reconfigure --frontend=noninteractive locales \
    && update-locale LANG="en_US.UTF-8" \
    && chmod 777 /opt && chmod a+s /opt \
    && mkdir -p /neurodocker \
    && if [ ! -f "$ND_ENTRYPOINT" ]; then \
         echo '#!/usr/bin/env bash' >> "$ND_ENTRYPOINT" \
    &&   echo 'set -e' >> "$ND_ENTRYPOINT" \
    &&   echo 'export USER="${USER:=`whoami`}"' >> "$ND_ENTRYPOINT" \
    &&   echo 'if [ -n "$1" ]; then "$@"; else /usr/bin/env bash; fi' >> "$ND_ENTRYPOINT"; \
    fi \
    && chmod -R 777 /neurodocker && chmod a+s /neurodocker

ENTRYPOINT ["/neurodocker/startup.sh"]

RUN echo '{ \
    \n  "pkg_manager": "apt", \
    \n  "instructions": [ \
    \n    [ \
    \n      "base", \
    \n      "neurodebian" \
    \n    ] \
    \n  ] \
    \n}' > /neurodocker/neurodocker_specs.json

in 0.8.0 neurodocker switched to --base-image and lost that addition:

❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian
# Generated by Neurodocker and Reproenv.

FROM neurodebian

# Save specification to JSON.
RUN printf '{ \
  "pkg_manager": "apt", \
  "existing_users": [ \
    "root" \
  ], \
  "instructions": [ \
    { \
      "name": "from_", \
      "kwds": { \
        "base_image": "neurodebian" \
      } \
    } \
  ] \
}' > /.reproenv.json
# End saving to specification to JSON.

unfrortunately there were no changelog whatsoever back in those dark times, so hard to tell the intent etc.

trying to use --_header KEY=VALUE revealed that it leads to reading that _header template

❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian   --_header KEY=VALUE
Traceback (most recent call last):
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 212, in from_dict
    renderer.add_registered_template(method_or_template, **kwds)
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 364, in add_registered_template
    self.add_template(template=template, method=method)
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 263, in add_template
    template_method.validate_kwds()
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/template.py", line 145, in validate_kwds
    raise TemplateKeywordArgumentError(
neurodocker.reproenv.exceptions.TemplateKeywordArgumentError: Keyword argument provided is not specified in template: 'KEY'.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yoh/proj/repronim/neurodocker/venv-3/bin/neurodocker", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 464, in docker
    _base_generate(
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 441, in _base_generate
    r = renderer.from_dict(renderer_dict)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 214, in from_dict
    raise RendererError(
neurodocker.reproenv.exceptions.RendererError: Error on template '_header'. Please see above for more information.

apparently neurodocker adds that odd --name_header automagically as per each template, so since it is named _header, even if there is no parameters -- it adds it

def _get_params_for_registered_templates() -> list[click.Parameter]:
    """Return list of click parameters for registered templates."""
    params: list[click.Parameter] = []
    names_tmpls = list(registered_templates_items())
    names_tmpls.sort(key=lambda r: r[0])  # sort by name
    for name, tmpl in names_tmpls:
        hlp = _create_help_for_template(Template(tmpl))
        param = OptionEatAll(
            [f"--{name.lower()}"], type=KeyValuePair(), multiple=True, help=hlp
        )
        params.append(param)
    return params

and trying --_header method=source without anu

❯ neurodocker generate docker --pkg-manager apt --base-image neurodebian --_header method=source
Traceback (most recent call last):
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 76, in _render_string_from_template
    source = tmpl.render(template=template)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render
    self.environment.handle_exception()
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/jinja2/environment.py", line 487, in getattr
    return getattr(obj, attribute)
           ^^^^^^^^^^^^^^^^^^^^^^^
jinja2.exceptions.UndefinedError: '_header' is undefined

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/yoh/proj/repronim/neurodocker/venv-3/bin/neurodocker", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/venv-3/lib/python3.11/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 464, in docker
    _base_generate(
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/cli/generate.py", line 441, in _base_generate
    r = renderer.from_dict(renderer_dict)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 212, in from_dict
    renderer.add_registered_template(method_or_template, **kwds)
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 364, in add_registered_template
    self.add_template(template=template, method=method)
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 328, in add_template
    command = _render_string_from_template(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/repronim/neurodocker/neurodocker/reproenv/renderers.py", line 78, in _render_string_from_template
    raise RendererError(err) from e
neurodocker.reproenv.exceptions.RendererError: A template included in this renderer raised an error. Please check the template definition. A required argument might not be included in the required arguments part of the template. Variables in the template should start with `self.`.

so remains unclear on how to make use of that _header

[yarikoptic]

@kaczmarj do you remember by any chance if it was a conscious decision to drop adding _header instructions?

[kaczmarj]

the _header instructions should be added, because it includes important setup things. like installing curl, setting locales, etc.

did i remove the addition of _header?? it should definitely be included in each dockerfile

users aren't meant to add --_header on their own. that's why it has the underscore.