You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm hitting this problem with a private gitlab-hosted repo, containing circa 1500 documents that get rendered with quarto. I'm not able to share this set-up, however the approach I'm taking is similar to what is shown publically here, where I'm scraping some document meta-data and pushing this to files in a data folder which are then published as resources in the rendered docs. I'm experimenting with alternative ways to work with the document meta-data and wanted to leverage the pre-render capability within quarto.
The gitlab-runner that is doing the rendering is based on this docker image, a debian 12 OS, with quarto 1.5.57 on-board. If I comment-out the pre-render scripts, then things run fine. However, when I enable the scripts/metadata-scrape.py on the gitlab repo (similar in pattern to this scripts/metadata-scrape.py, only longer to handle custom meta-data), I'm getting this Argument list too long error. Can you shed any light on why this might be happening when quarto is handling pre-render scripts.
Per this, I tried to set a longer command-line buffer with ulimit -s 65536 on the VM running this dockerized gitlab-runner, but also included in the .gitlab-ci.yml so that it gets applied within the runner itself (see image above), but none of these have helped.
The text was updated successfully, but these errors were encountered:
The only way I can see this happening is that we're passing the list of input files as the QUARTO_PROJECT_INPUT_FILES env variable, and that is triggering the error (even though the error talks about the argument list being too long instead of an env variable being too long).
I'm not sure how to fix this in a backwards-compatible way. What we need to do for large files is to pass the path of a temporary file that contains the list of input files; but if we do that, we'll break the very many existing pre-render scripts that work just fine.
I'm not sure how to fix this in a backwards-compatible way.
One suggested fix in the second link above is to use a file-type variable. Maybe creating both a variable-type variable and file-type variable would allow users facing the Argument list too long problem to revert to using the file-type variable, somehow.
Also, for avoidance of doubt, if I disable the pre-render scripts section of _quarto.yml, then the render proceeds, per below (file names and folders are fictitious), for now, but I do feel like I'm at the upper-end of file count handled, since I'm sometimes bumping up against this problem which I know you are addressing separately.
$ quarto render --output-dir public
WARN: The file /xxx/xxx/xxx.qmd contains a theme property which is being ignored. Website projects do not support per document themes since all pages within a website share the website's theme.
WARN: The file /xxx/xxx/xxx.qmd contains a theme property which is being ignored. Website projects do not support per document themes since all pages within a website share the website's theme.
....
[ 1/1516] docs/folder1/folder2/folder3/folder4/01_file.ipynb
[ 2/1516] docs/folder1/folder2/folder3/folder4/02_file.ipynb
[ 3/1516] docs/folder1/folder2/folder3/folder4/03_file.ipynb
[ 4/1516] docs/folder1/folder2/folder3/folder4/04_file.ipynb
...
I'm not sure what form QUARTO_PROJECT_INPUT_FILES takes and how this is influenced by whether pre-render scripts are enabled or not. Just for further colour, one of my scripts generates an array of file-paths to process meta-data for documents .. that looks something like what's depicted below. Not sure if QUARTO_PROJECT_INPUT_FILES contains a richer set of data beyond file paths or not.
Discussed in #10823
Originally posted by Analect September 17, 2024
Description
I'm hitting this problem with a private gitlab-hosted repo, containing circa 1500 documents that get rendered with quarto. I'm not able to share this set-up, however the approach I'm taking is similar to what is shown publically here, where I'm scraping some document meta-data and pushing this to files in a
data
folder which are then published asresources
in the rendered docs. I'm experimenting with alternative ways to work with the document meta-data and wanted to leverage thepre-render
capability within quarto.The gitlab-runner that is doing the rendering is based on this docker image, a debian 12 OS, with quarto 1.5.57 on-board. If I comment-out the
pre-render
scripts, then things run fine. However, when I enable thescripts/metadata-scrape.py
on the gitlab repo (similar in pattern to this scripts/metadata-scrape.py, only longer to handle custom meta-data), I'm getting thisArgument list too long
error. Can you shed any light on why this might be happening when quarto is handlingpre-render
scripts.Per this, I tried to set a longer command-line buffer with
ulimit -s 65536
on the VM running this dockerized gitlab-runner, but also included in the.gitlab-ci.yml
so that it gets applied within the runner itself (see image above), but none of these have helped.The text was updated successfully, but these errors were encountered: