Index pages

[alexvoss]

Added mechanism to create index pages with information based on .example.yml files that describe each example. The code is loosely based on the shortcodes mechanism used in the documentation plus the archive.pyhook.

Would like to know if you think this is going roughtly in the right direction or if I am off track here.

One question I have not wrapped my head around yet is how to create index pages for the tags without pre-defining what tags exist. Same for authors. I might have a peek at how this is implemented in the blog plugin.

[squidfunk]

Could you explain how and where the markers will be used? Note that we should not put those markers inside of examples themselves, because examples must at all times be self-contained and runnable standalone, so they can be downloaded and run.

If we only use those markers for overview pages that are not part of examples, it might be a good tool. However, what index pages are there? We likely only need a single top-level page for branching into examples, and inject a link into examples to return to the overview. At least from the top of my head.

One question I have not wrapped my head around yet is how to create index pages for the tags without pre-defining what tags exist. Same for authors. I might have a peek at how this is implemented in the blog plugin.

I'm not exactly sure what you mean.

[alexvoss]

where the markers will be used

Ah, sorry, should have mentioned that. The markers would go into the documentation for the whole set of examples, so docs/ at the top of the repository. They can then be used to filter and further configure the list of examples shown on each page. At the moment there is only index.md with <!-- index:all --> but I thought we should also have pages for different tags that group examples together.

The kinds of tags I have imagined we'd use:

- public        # works without Insiders
- insiders      # requires Insiders to run
- simple        # requires no extra plugins or other magic
- integration   # demonstrates integration, use of other plugins
- blog          # functionality demonstrated

If you think we should stick with a single page then we could add classes to these and let people selective hide things by tag - rather than having separate index pages. That might actually make more sense.

[squidfunk]

To be honest, it sounds a little like we're reinventing a categorization system for the third time 😉 We already have tags and categories in blogs. If you think that it is necessary, we can go ahead, but I only want to bring up this point, since it feels like we're reinventing the wheel.

IMHO, at first, sticking to a single page and moving things under certain sections sounds like a less overengineered approach at the current stage. Take our plugins page for example – we have 12 plugins in three categories. I'm not sure how many examples we have right now, but it's definitely of the same magnitude. Thus, I'd say: keep it simple and we'll consider tagging once we gravitate towards ++magnitude.

I'm absolutely fine starting with a lean approach quickly, and reconsidering as we grow. We've followed the same approach successfully on our documentation. To put it more graphically: documentation is like a plant – you should repot it from time to time. Disclaimer: this quote is not from me, but from somebody from Twitter (can't remember who)

[alexvoss]

Hi, I have pushed some changes to this branch and also merged in changes from master (which, IIRC was the blog-update example). The code should do the job for now and produces just one list on the main index page.

At the moment, there are only two examples on there as only two have complete .example.yml files but we can fix that quickly.

I am rendering the tags as I think they convey important information, especially the public/insiders distinction. We should have a discussion about how filtering by them might work.

There is the question how we should order the examples on the index page, which is of particular importance if we have no way to filter them.

One thing I would like to do is to provide author information for the examples, partly to get some exposure but also to encourage others to contribute. I note the discussion about what the ambassadors could get credit for - perhaps this would be something to offer? Just a thought.

[squidfunk]

I'm currently short on time verifying this, but if you're confident that it works, I can merge this through. Having an index page that might not be perfect yet is better than having no index page ☺️ I trust you in pushing this forward in a meaningful way, and the approach with generating the index page from descriptive yaml files gives us a lot of flexibility in changing it later.

One thing I would like to do is to provide author information for the examples, partly to get some exposure but also to encourage others to contribute. I note the discussion about what the ambassadors could get credit for - perhaps this would be something to offer? Just a thought.

That is a very interesting idea, but we'd first see how it goes with discussions, and we're still in iteration-mode on the examples repository. Once our process reaches maturity here, we can for sure onboard individuals to contribute examples and get compensated with Insiders access. That being said, you can definitely explore how we could incorporate author information, so we can leverage it later on.

[alexvoss]

Having an index page that might not be perfect yet is better than having no index page

I don't want to extend this for too long but did some work on filtering the examples. It would be good if you could have a look - not for correctness of the implementation but to see if you think the overall approach is ok. I do still need to add a "show all" button.

The logic is implemented in CSS at the moment, turning off examples that match a filter that is turned off, which at the moment means the user has to tick all the pills that match an example they want to find. This seemed to make sense when thinking about a query like "show me all the examples for the blog plugin in the public version". Not sure it captures all uses, though. I had an issue with the tags-with-icons example, which might have wanted to be tagged with icons and tags but people would then need to tick both to find it. Perhaps a different logic would be better but that would likely mean switching to JS.

One other thing I thought would be good was to move the example metadata into the headers of the README.md files. An unintended side effect of doing this is that the metadata show on Github. Can you help me decide if that is a good or a bad thing?

[alexvoss]

Right, I forgot: also need to decide how the produce a short description of the example for the list. One approach would be to mandate that there be a <!-- more --> separator and all that comes before that would get copied through. Would mean adding a markdown attribute to all HTML but that should not keep us, in fact it would be good if Markdown markup was available.

[squidfunk]

I'll take a look.

One other thing I thought would be good was to move the example metadata into the headers of the README.md files. An unintended side effect of doing this is that the metadata show on Github. Can you help me decide if that is a good or a bad thing?

That's perfectly fine.

[squidfunk]

Okay, so honestly Im not really a fan of the filtering logic yet. I have to switch on all filters to get results, and there will likely be much more tags when we have more examples. I'd say for now, we go without filters and list the examples on the top-level page. Let's first build 40-50 examples and then decide how we make those more accessible. Remember, example should be linked from the documentation, which is considered the primary entrypoint.

[alexvoss]

Ok, will park the filter logic somewhere. Perhaps just commenting it out in the template?

I assume the rendering of the examples as such is ok? What do you think about the description? Should this live in the header of the README.md, potentially duplicating what the README itself says? Or should we try and find a way to parse it out of the Markdown, perhaps with a `?

[squidfunk]

We could just parse the content from the corresponding README.md, but I'd say this is something for later. Let's keep it as lean as possible right now. I'll revisit this topic later and we'll find a good solution together. For now, we should concentrate on the actual examples.

[alexvoss]

I would like to take up our discussion of the tags again. It seems to me that it should be possible to tweak either or both the tags and project plugins to aggregate the tags over multiple sub-projects. If that was supported, it would be easy to just equip the README.md files here in examples with tag metadata and have a single index page that lists them.

Any further improvement such as grouping tag index content by the other tags used (as in squidfunk/mkdocs-material#6443) would then also benefit out index here, which, in turn, can serve as a test-case.

If this is not something you would slip in between other work when you want to take an 'active break', I would be happy to have a go. It would seem to me worthwhile because it adds value to projects, tags and blog, all in one go - as well as resolving this open discussion here! If you want to do this as a quick win then I will just plod on with the other things I already have taken on and finish some of them off. Will start with the simplified index page here. Should not take that long and be with you tomorrow.

[alexvoss]

It just strikes me that I need to study the existing plugins to see if the hook in our parent project here is not entirely superfluous if we can do what I suggested above. It might reduce down to a simple check to see if all README.md files do have tags? (So no example goes unlisted.)

[squidfunk]

The problem is that investing into the tags plugin makes no sense at the current stage. It was the first plugin I wrote, when I was new to Python and just learned how to write MkDocs plugins. I have several ideas how to make the tags plugin much more powerful. I also have a working prototype, but it's not yet production ready. The general idea is to remove the tags index entirely and allow to use markers to create an arbitrary number of tags index that are link targets (so tags link to the closest index). Currently, there can only be a single index to link to – all extra tags indexes have no links.

I'll bump it up my – as you love to call it – guilt pile 😅 I will check whether I can finish a good prototype soon, so we can take it for a spin. Before, we need to understand how to integrate both plugins, so we can automatically collect tags from projects and link to them in indexes.

It just strikes me that I need to study the existing plugins to see if the hook in our parent project here is not entirely superfluous if we can do what I suggested above. It might reduce down to a simple check to see if all README.md files do have tags? (So no example goes unlisted.)

Jup, might be a good idea, but might also not be – the metadata will show up in the README.md when viewed on GitHub, and might be confusing. I'm not sure what's better. However, we could add a preprocessing step via a custom transform to set all metadata from the .example.yml in the README.md when building projects, so it would be local to our build only. The tags could then be read and linked with the logic mentioned above.

[alexvoss]

For a moment I thought the meta pluging could put the tags into the Markdown but then it would be in all pages and would get in the way when we warn to showcase the meta plugin itself. So, some custom code to hoist from .example.yml into the page header of the README.md would be good.

Btw. where should the README file live? (Sub-)project root or docs/? For GitHub the project root would be better but then we would need to do something for it to show up in the built example (pmydownx snippet?) At the moment, we have it in docs/. Will go and check if we could symlink to it.

[squidfunk]

For a moment I thought the meta pluging could put the tags into the Markdown but then it would be in all pages and would get in the way when we warn to showcase the meta plugin itself. So, some custom code to hoist from .example.yml into the page header of the README.md would be good.

I really like your thinking – however, it's kind of a non-use case for the meta plugin, because we would need some functionality to only apply to a specific file or level, and that would defeat the whole case, so the transform is probably a better idea. It is explicitly intended to make changes to projects before building them without touching the projects themselves. Sometimes you cannot touch the projects, as you're building docs inside git submodules that you might not govern, or similar use cases.

Btw. where should the README file live? (Sub-)project root or docs/? For GitHub the project root would be better but then we would need to do something for it to show up in the built example (pmydownx snippet?) At the moment, we have it in docs/. Will go and check if we could symlink to it.

In the root of the docs_dir, because then, MkDocs will just pick it up as the index page and GitHub will also render it as well, which I didn't know before trying it myself, e.g., here.

[alexvoss]

oh, right, so no need for symlinks. This is documented behaviour, too:

If you put your README file in your repository's hidden .github, root, or docs directory, GitHub will recognize and automatically surface your README to repository visitors.

[squidfunk]

Okay, let me draw something off my guilt pile today – oh look! It's the tags plugin! I'll invest the day into checking if I can finish the rewrite fast, as it's quite far actually, and we can at least close the open issue. We will definitely need to think about how to integrate the tags plugin with the projects plugin, since you want to link tags from different projects, so this is something I will definitely keep in mind when finishing my prior work.

It would be really awesome to build an overreaching tags index, essentially sourcing tags from other sites. This might be possible by emitting something like a tags.json that can be consumed by other projects, or by the tags plugin itself. All of that just from the top of my head, so please expect heavy back and forth.

[squidfunk]

I made substantial progress 🥳 Still not ready for a spin, but I think I'll get there tomorrow. I will open a PR on the Insiders repository as soon as this is ready for testing, so we can gravitate towards a sensical solution together. Now:

• tags_file and tags_extra_files are now essentially obsolete, as you can now integrate a tags indexes wherever you want by using an inline comment inside a Markdown file. Syntax is as easy as <!-- @tags --> inside any Markdown file.
• The tags index is now rendered as a child below the nearest enclosing section. This means if there's an h3 above the inline comment, the tags will use h4 headlines. This also means that you can now pre- and suffix the tags index with arbitrary content, which makes it particularly useful, because it allows the author to define very narrow, specialized indexes at particular locations. Why is that cool? Well...
• ... because we can now use the new tags plugin to implement things like Blog: embed blog posts in other pages squidfunk/mkdocs-material#5074. Authors will be able to tag blog posts with specific tags, e.g., blog, which they can then reference in inline comments, possibly similar to <!-- @tags blog -->. We can add further syntax to limit, sort, and randomize the content, to implement things like "most recent 3 blog posts", "most recent posts in category foo" or even "suggested pages", which can be put at the bottom of a page to draw random (or weighted) pages from a collection of tagged pages.
• It must be possible to define new templates for the tags index, so that authors can render specific tags indexes with specific templates. For example, for a blog post overview, another template should be used as for a regular index of pages. We will provide some templates for the author's convenience.
• We'll likely need new functionality to mark certain tags as hidden, so that those tags can be used for pages to be assigned to indexes, but they are not rendered on the page. With the help of the meta plugin, the author will be able to tag entire subtrees of the documentation. Edit: I think this is even simpler: just use a second instance of the tags plugin with a different key, e.g. classification, and don't render tags on pages themselves.
• The tags plugin now uses a customizable front matter property, so you can run multiple tags plugins alongside each other. The default is obviously tags, but the author can now configure the name of the property.
• We can implement a hierarchical mode, where a tags index includes only the pages that are below the subsection in which the tags index is located. If you have very large documentation, you can create specialized tags indexes for subsections just by using the default syntax, no configuration necessary. Might work something like <!-- @tags mode:scope -->.
• It should be possible for the author to define named collections of tags in mkdocs.yml, in order to group tags declaratively. These collections could then be referenced in the inline comment, possibly combining them with scope, something like <!-- @tags collection:foo mode:scope -->. This will render all tags in the current subtree of your documentation which belong to the collection foo, as defined in mkdocs.yml.
• The plugin should emit a tags.json index that can be consumed by other plugins like the projects plugin, but not necessarily only by this plugin. This could be implemented by a new setting in the tags plugin where an author can define extra indexes to be included. The projects plugin could, in a second step, auto-configure the tags plugin to consider those extra indexes, which can then be consolidated when building tags indexes. We will need to invest some brain power into that, because I'm not sure how the syntax and UX should work here. This is just meant as a lookout what would be possible to achieve, and what would essentially solve the same things in this PR with new capabilities.

That's just from the top of my head. In essence, we'll be introducing several new degrees of freedom that will make the tags plugin an excellent companion for building second-level structures besides the navigation structure. I'm curious what you think about that and if you have further ideas that we should consider to evaluate.

[alexvoss]

This looks great. Might take a bit of time for me to wrap my head around all the use cases and implications. I would love to test-drive this in create-blog and examples. Are you already including the idea of aggregating tags indexes across subprojects or is that a next step and you just keep it in the back of your mind?

One thought I had was how the tags functionality might work with the 'related links' in the blog plugin? More generally speaking, would it make sense to have a related-tags function? Imagine, for example, pages tagged with 'customization'. A related tag might be 'plugins', 'hooks'? These are distinct, at least in current parlance, but are all related to "how do I get what I need out of Material and MkDocs?"

[squidfunk]

Are you already including the idea of aggregating tags indexes across subprojects or is that a next step and you just keep it in the back of your mind?

It's a next step. As I've written, we first need to expose the tags of a project via something like tags.json, which can then be consumed by other plugins like the projects plugin.

One thought I had was how the tags functionality might work with the 'related links' in the blog plugin? More generally speaking, would it make sense to have a related-tags function? Imagine, for example, pages tagged with 'customization'. A related tag might be 'plugins', 'hooks'? These are distinct, at least in current parlance, but are all related to "how do I get what I need out of Material and MkDocs?"

Sounds a lot like squidfunk/mkdocs-material#6463 to me, and the answer is yes. However, we don't even need to explicitly define the relation but can use relative co-occurrence to derive degree of relatedness. We're essentially building a bipartite graph of pages and tags.

[squidfunk]

I've distilled our conversation into squidfunk/mkdocs-material#6517. Let's collect some feedback and user input along the way.