Intregrating Manubot and Idyll

[dhimmel]

Idyll stands for Interactive Document Language and is a "markup language for interactive documents." The current description reads:

Idyll extends the ubiquitous Markdown format to enable the creation of dynamic, interactive narratives for the web. The language and toolchain aim to empower journalists, researchers, and technical experts to create compelling content using familiar tools and processes.

Idyll can be used to create explorable explanations, to power blog engines and content management systems, and to generate dynamic technical reports. The tool can generate standalone webpages or be embedded inside of your existing site.

Taking a look at an example was helpful. See Idyll on GitHub at idyll-lang/idyll.

@cgreene met the Idyll folks recently and wondered whether it'd be helpful for the Deep Review in greenelab/deep-review#842.

This issue is for discussing whether there is synergy between Idyll and Manubot, and whether there's an opportunity to integrate them in some form.

CC @mathisonian @AndrewGYork @marciovm.

@AndrewGYork is also working on interactive papers hosted via GitHub (example).

[dhimmel]

Thus far Manubot has focused on bringing the open source workflow to "traditional" manuscripts. Currently, manuscripts are written in markdown with pandoc extensions and some manubot-specific extensions for citations. We haven't explored interactive figures. However, the markdown input should support HTML elements that get passedthrough the HTML manuscript.

Therefore, it's possible we could convert Idyll snippets to HTML that would render in Manubot's HTML output. I need to learn more about the Idyll workflow... i.e. how the idyll source is converted to a webpage. Or perhaps Idyll would like to take some the continuous deployment and citation infrastructure from Manubot? Or perhaps there is no synergy and Idyll should remain focused on interactivity and Manubot on open workflows for composing static manuscripts?

[agitter]

The remote_refocus manuscript you linked above presents a good vision for how an interactive Manubot manuscript might work. The HTML includes interactive components, but it is still possible to download a static snapshot as a PDF. I haven't dug in to see how that is implemented.

Thus far, we have intended for Manubot to be compatible with traditional journals (e.g. the push for JATS support #82). For now, that may mean that we need to continue supporting some form of a static manuscript even if we introduce a parallel interactive version.

[mathisonian]

Hi @dhimmel,

Idyll's build process takes as input a markup file (written in idyll's markup language, or idyll-flavored markdown if you will) and some options and outputs HTML, CSS, and JS files that serve as a self contained web page. The HTML that idyll outputs can be templated, so manubot could inject anything that it wanted into the output via an html template. You can run this build step via a node.js API or with the command line tool, docs on this are here: https://idyll-lang.org/docs/configuration-and-styles

There is also a way to do all of this in the browser, but it seems like doing it in a build step makes more sense so that static html is rendered.

[mathisonian]

Idyll would have to make some updates to make it straightforward to generate static (pdf) output, but its not a huge change. I've already successfully experimented with latex output, and could provide an api to add other targets.

[dhimmel]

Idyll would have to make some updates to make it straightforward to generate static (pdf) output

Okay note that currently Manubot creates it's PDF output from the HTML webpage using WeasyPrint. So if you can print an Idyll webpage and it renders properly, then it should work with WeasyPrint.

[mathisonian]

Ah cool - I think it should work fine in that case, we'll just have to double check that any interactive components do something reasonable for that conversion. I suspect there'd only be an issue if something was only being rendered on the client-side and not in the static output, but components are rendered into the static output by default.

[mathisonian]

Hi all, I've been busy with some paper deadlines but coming back to this.

In the meantime I've updated the Idyll compiler to work with user provided plugins which can be use to extend Idyll's syntax if there is anything missing (for example, we can add footnote/references support with this).

[dhimmel]

I've updated the Idyll compiler to work with user provided plugins

Great! I will make it a goal to play around with some Idyll docs and perhaps write a plugin.

Given that Manubot and Idyll are two nascent projects, both without large userbases yet, and both somewhat complex, I think it makes sense to let them evolve separately for the the time being. However, we'll keep a close eye out for any synergy or code reuse that makes sense between the projects.

[mathisonian]

Sounds good @dhimmel! Let me know if you run into any issues or if there are any features that would make a big difference on your end - happy to prioritize.

If you end up going for a plugin the best docs currently are this thread idyll-lang/idyll#280