myst build --execute doesn't re-evaluate Markdown cells with {eval} directive

[srprca]

Description

When running myst build --execute, Markdown cells with {eval} directives are not re-evaluated which leads to out-of-date output.

Versions

As installed from conda-forge conda channel:

jupyterlab                4.2.4              pyhd8ed1ab_0    conda-forge
jupyterlab-myst           2.4.2              pyhd8ed1ab_0    conda-forge
mystmd                    1.3.5              pyhd8ed1ab_0    conda-forge

Reproduction

1. Create a new directory for the purposes of bug reproduction: mkdir myst_repro
2. Enter this directory: cd myst_repro
3. Initialize a MyST project: myst init, answer n to the suggestion to run myst start
4. Run JupyterLab with a jupyterlab-myst extension installed and enabled: jupyter lab
5. Create a new Python Jupyter Notebook
6. Create the first cell: a code cell with code a = 3
7. Evaluate this cell, there is no output
8. Create the second cell: a Markdown cell with code

{eval}`a`

9. Evaluate it once, the symbol a is rendered as a regular code block (see bug eval role not working as expected jupyterlab-myst#175)
10. Evaluate it again, value 3 is rendered
11. Change the first cell to read a = 5, evaluate it
12. Save the Notebook, shutdown Jupyter Lab
13. Observe that in the Notebook's textual representation the second (Markdown) cell is recorded as

{
   "cell_type": "markdown",
   "id": "f521db5a-a795-4091-9e9b-51020f109334",
   "metadata": {
    "user_expressions": [
     {
      "expression": "a",
      "result": {
       "data": {
        "text/plain": "3"
       },
       "metadata": {},
       "status": "ok"
      }
     }
    ]
   },
   "source": [
    "{eval}`a`"
   ]
  }

14. Run myst build --execute, observe the line starting with "Executing notebook..." in the output
15. Run myst start, open the URL printed in the output with your browser
16. Observe that the code cell reads a = 5, but the Markdown cell is rendered as 3
17. Run Jupyter Lab again, open the same Jupyter Notebook, select "Kernel -> Restart kernel and run all cells" from the menu
18. Observe that the Markdown cell has been re-rendered to 5

[agoose77]

Thanks for this reproducer @srprca.

What's happening here is technically not a bug, rather a confusion with how execution works. This doesn't mean that we shouldn't change something in mystmd to clarify this.

JupyterLab's MyST extension has to store user-expression results in the cell metadata in order to persist them. mystmd knows how to pull out those outputs, and regular code cell outputs, from the .ipynb into the MyST document.

When you use mystmd build --execute, we overwrite those outputs in our AST with execution results. If we do not need to re-execute the notebook, then we re-use our cached outputs. Crucially, this cache is entirely separate to the .ipynb file altogether.

So, when you write myst start without --execute, you're effectively ignoring our execution cache, and re-using whatever is in the notebook.

There are bigger questions about whether the execution cache should be configurable and/or more standardised, and I think this UX question ties into it.

[lwasser]

Hey @agoose77 👋🏻 I actually have run into this as well. I do understand there are more significant questions associated with it!! BUT from a user perspective, it's unexpected and appears to the user as a bug. Even if, technically, it's not a bug in terms of how Jupyter works! I am not sure what the solution is, but could mystmd "rerender" markdown cells too when you run execute, or is there some command that would do that for the user? I have had to continually "run" markdown cells. I could be confused here too. just noting it's a pain point for a user while appreciating the complexity on the dev side of things.

[agoose77]

@lwasser could you clarify your point regarding rerunning markdown cells? myst will always run cells and expressions if they have changed (and you set - -execute)

[srprca]

@lwasser, I believe the main conclusion from @agoose77's explanation is that unless you pass --execute to myst start, it will just re-use whatever is stored in the Notebook file. If you haven't explicitly re-evaluated the Markdown cells with {eval} directives in your Notebook from JupyterLab interface, what is stored in the Notebook will be outdated. Both myst build --execute and myst start --execute will re-evaluate everything in the Notebook, including such cells, but if you simply run myst start without --execute, mystmd will just use whatever is already stored in the Notebook, regardless of any prior calls to mystmd build --execute.

@agoose77, I wonder if it would be possible to document this somewhat counter-intuitive behaviour? I believe a typical user will have an expectation that myst start will serve whatever was previously built by myst build ..., unless explicitly overriden by arguments to myst start. If myst start instead works independently of myst build, perhaps this can be mentioned in myst start --help...?

[agoose77]

It's true that we don't document this nuance strongly enough, but it looks to me as though we might need to change the behavior altogether. This is clearly an unintuitive UX given that two people have encountered it.

A nice feature of MyST's execution cache is that it's always clear whether the cached state is up-to-date. Whereas, an ipynb does not have any way of guaranteeing that the input of a code cell/expression matches the output saved in the notebook.

That said, it feels the execution pipeline being one-way (not writing the cache back to the notebook) is the right move, so I'm not sure what the next step will be here besides improvement of our documentation in the near term.