How to signal customisation

[finanalyst]

Intro

• RakuDoc allows for customisation with custom blocks and custom markup
  • customisation might be through a custom block, eg. =LeaflletMap, which exposes the Leaflet js library for maps.
  • customisation might be a custom code, eg. M<pay here|PayPal; user-token;product-info>
• RakuDoc v2 has naming rules to distinguish between built in and customisation
• An author of a RakuDoc source will know where the code/formatting information can be found to render a custom block/format code that he/she has used
• The current specification is that any unknown custom block is rendered into plain text, and for some warning mechanism to be triggered.
• The specification has a reference to DOC use XXX, where XXX is a module. The current RakuAST compiler triggers an error when DOC use ... is encountered.
• The aim when designing RakuDoc was to allow for developers to create their own custom blocks/markup codes, which could then be adopted by other authors.

How should customisation be signalled to the renderer?

Problems

• RakuDoc is format neutral (eg., HTML, Markdown, epub), but output is obviously format specific, so any customisation needs also to be aware of the output format.
• It is recognised that some customisation, eg. PayPal links and LeaflletMaps, may be specific to HTML and not have a usable rendering in (eg) epub.
• If DOC use is included, then where is it used? Consider:

DOC use Some::Module;
=begin rakudoc

=SomeNewBlock Rendering this information is specified in C<Some::Module>

=end rakudoc

This indicates that code from Some::Module needs to be executed

• The code in Some::Module is not a part of the RakuDoc source
• How is DOC use to be interpretted by eg Podlite, which is not Raku based?
• Is the code in Some::Module related to the way in which the AST is created, or the way in which the AST data structures are rendered?

To keep the issue separate from the plan for RakuDoc::Render, I'll outline the plan for the Renderer below.

Thoughts? @thoughtstream @codesections @vrurg @lizmat @zag

[finanalyst]

The RakuDoc::Render module I am writing is planned to use templates and plugins to handle customisation.

A generic renderer processor handles the RakuDoc source, and each RakuDoc instruction is handled using templates.

The templates define how the source for each block is formatted into the output. The renderer for a particular output, eg. HTML, generates an instance of the generic processor, then attaches templates to the processor, then passes the RakuAST rakudoc statement list to the processor.

Customisation can be handled by attaching templates for customisations to the processor instance.

If the terminal invocation is used, eg

raku --doc=RakuDoc::To::HTML test.rakudoc > test.html

RakuDoc::To::HTML instantiates a generic processor and attaches HTML templates.

A programmatic implementation for an HTML file would be

use RakuDoc::Render;
sub MAIN( $rakudoc-source ) {
   my $p = RakuDoc::Processor.new;
   my $source = "$rakudoc-source.rakudoc".IO.slurp.AST;
   $p.templates( &get-html-templates);
   $p.render( $source.rakudoc );
   "$rakudoc-source.html".IO.spurt: $p.file-render;
   "standard.css".IO.spurt: $*RESOURCE/standard.css;
}

The question is where to find the information for a custom block.

If the information is in a 'DOC use' statement, then the $source statement list could be searched for the module name, and the templates could be extracted from the module referenced there. The templates could then be attached to $p, and the file rendered.

[vrurg]

I finally got a couple of minutes on this.

• How is DOC use to be interpretted by eg Podlite, which is not Raku based?

Ignored. Here is the point: it is only the compiler which can handle all of a language. No IDE could do it without compiler support.

• Is the code in Some::Module related to the way in which the AST is created, or the way in which the AST data structures are rendered?

Here I just repeat what we discussed earlier in person: any DOC use module can only affect how a document is parsed. Rendering must be a separate concern, sole responsibility of a renderer, whatever it is.

The question is where to find the information for a custom block.

A rendered, apparently, knows nothing about document source. It deals with RakuAST where block representation depends on the parser in first place. If a DOC use module converts the custom block into a standard set of RakuAST nodes then the renderer is unlikely to even know the block ever existed.

Slightly different case is if the block gets some special representation in the AST. In this case it is up to the renderer to decide how to handle it and where to get the necessary modules or whatever it takes to render the block correctly. Say, a config file would do.

On the documentation creator side they may leave a note in distribution README about recommended renderer, modules, etc. needed to produce best expected output. The parser module from DOC use could provide install-time message with similar advises. Like, say, "if you install this RakuDOC::CustomBlock module then consider installing RakuDOC::Render::CustomBlock for the 'rakudoc' utility'.

It is also up to the renderer to chose how to manage unknown/unsupported AST nodes. Simply skip; or insert as verbatims; or die.

To sum up, my point remains the same: we must clearly separate parsing from rendering with no way for a RakuDOC document to be in charge of its own rendering. It may provide hints by setting configuration values that can be understood by renderers. Or even be specific to a particular renderer. But they would remain nothing more but hints.

[finanalyst]

@vrurg I understand your points, but it takes a bit of re-reading.
Take the following situation:

• An developer is not a maintainer of RakuAST::Render (the module I am currently working on, but this could be generalised to any Renderer of RakuDoc v2)
• The developer creates a custom block that is used in a RakuDoc source as =MyNewBlock
• The developer creates the customisation code for eg HTML & Markdown as defined by RakuAST::Render
  • my current idea is that customisation code would be managed by sub-classing RakuAST::Render::Custom
  • the sub-class is declared in a module, eg. RakuASTRender::MyCustomisations
• The developer initially wants to have the customisation code in a private location

Currently, RakuDoc v2 specifies that if a Renderer does not recognise a custom block, then it does a minimal renderering and issues an error message

How does the developer, who will only use RakuAST::Render for her own purposes?

Current strategy with Raku::Pod::Render

I solved this problem with RPR by creating a generic Renderer class. The actual renderer (eg. Pod::To::HTML2) creates an instance of the generic class and adds the customisations to the instance.

Pod::To::HTML2 will also look in the root directory for sub-directories that have a _ in the name, and interprets these as sources for customisation code. Before rendering, these customisations are added to the generic instance.

Consequently, there is no need to add any explicit text to the source files. If a custom block is present in the customisations, then it will be rendered properly, if not, it will be minimally rendered.

It seems from your comments above that this approach is to be preferred to having explicit indications in the source.

[vrurg]

It seems from your comments above that this approach is to be preferred to having explicit indications in the source.

I'd rather be a config file zealot where customizations are explicitly mentioned. There are ways how to do it, like referring to installed module names; or, for totally private projects – paths to customization distributions, like $HOME/src/Raku/custom-block-a where a META6 can be found. I mean, details may vary. The config file in project's directory may, at the same time, serve as developer's hints as to how their docs are to be rendered correctly.

Overall, most used scenario is likely to be the one where a distro developer provides their own rendering. It could be available through managers like App::Mi6, or via a Makefile, as I do.

But, at the same time, raku.land, for example, may use their own rendering with their own configs for producing unified look for all docs. This is where the problem of different customizations could hurt. But it's not unsolvable.

[finanalyst]

@vrurg Where would the config file be placed? In the same directory as the .rakudoc or .rakumod file?

One of my goals is to create a web-site served locally that shows all of the documentation of all the modules installed locally.

If as in Problem-solving 393 the module developer can place documentation in arbitrary places, where would the config file be?

[vrurg]

where would the config file be?

To me this is a technical detail. Let's say, a project provides their own local config or configs in project's root, a typical setup. The question is how the renderer, you're going to use to produce your local site, takes these configs into account? Or doesn't take, if necessary.

I'm not ready to get into these details right now, and they'd be mostly off-topic here anyway. Out of the box I foresee a couple of technical problems to be resolved. But it is barely possible to be prepared for all of them anyway.

[finanalyst]

Summarising before closing:

• RakuDoc documents may make no assumption about the renderer
• The user who is rendering a RakuDoc document into some form will decide, and the user can set up the render environment. Hence configuration - if needed -should be in some external configuration file.

Closing issue as resolved