Automatically generate contracts documentation #138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We are creating a GH Actions workflow which automatically generates the contracts documentation in Markdown, based on the functions and the NatSpec-format comments in the Solidity files stored in the `contracts` folder in `threshold-network/solidity-contracts`. For certain workflow triggers, the generated documentation will be pushed to the `threshold-network/threshold` repository, under `./docs/app-development/staking-contract-and-dao/staking-contract-and-dao-api`. As the `.docs` folder is synched with Threshold docs GitBook space, the pushed docs will be displayed in HTML under `docs.threshold.network` domain. There are two main jobs in the workflow: * `contracts-docs-publish-preview` * `contracts-docs-publish` Both call a reusable workflow `reusable-solidity-docs.yml` which resides in the `keep-network/ci` repository under `.github/workflows`. The jobs differ in parameters with witch the reusable action is called. The common part of the jobs is the beginning stage, where the Solidity files are being prepared for Markdown generation. During that stage the jobs will remove the spaces chars between the NatSpec comments markers (`///`) and the text of the comment. This is done to ensure proper rendering of the lists in the output docs and is a default action made by `reusable-solidity-docs.yml`. Once files are ready, the jobs use Docgen tool to generate the Markdown docs. The tool needs to be installed in the project and configured in the `hardhat.config.ts`. The configuration that we use specifies that the docs should be generated to the `geerated-docs` subfolder, to one single `index.md` file. The `.sol` files in the `./solidity/contracts/test` folder will be ignored during generation (those are test/stub contracts which are not used on Mainnet). A custom template will be used during docs generations. The template was created based on the default https://github.com/OpenZeppelin/solidity-docgen/blob/master/src/themes/markdown/common.hbs template, but it removes the cursive from the `@dev` type comments (because the cursive didn't work well with formatting of the lists). Once the Docgen is run and the `index.md` file is generated, the jobs will add a Table of Contents to the file, to improve the navigation. The TOC will be added by the `markdown-toc` tool and will be generated using `--maxdepth 2` param, which results in listing all the contract names, but not the functions. Once the TOC is added, we'll start to see the difference in the behavior of the jobs. The `contracts-docs-publish-preview` job is triggered for: * `pull_request` events, if the PR modifies either files in `./contracts` or the workflow file itself, * `push` events, when the push is made to a branch which name starts with `releases/mainnet/v`, * `workflow_dispatch` events. The job publishes the generated artifacts as artifacts of the GH Actions workflow run. For PRs, link to the run with the artifacts will be posted as a PR comment. The `contracts-docs-publish` job is triggered for: * published releases, which names start with `refs/tags/v`. The job pushes the generated file to the `./docs/app-development/staking-contract-and-dao/staking-contract-and-dao-api` folder in the `threshold-network/threshold` repo, to the `main` branch. The `.docs` folder is synched with the GitBook Threshold documentation, so thanks to the job, the changes in the contracts should be reflected on the docs.threshold.network. In order for the job to work, we need to configure a user which will create the commit The user needs to have right to commit to the `threshold-network/threshold` repo. We also need to configure following secrets: - [ ] `THRESHOLD_DOCS_GITHUB_TOKEN` - [ ] `GPG_PRIVATE_KEY` - [ ] `GPG_PASSPHRASE` We're using a GPG key to be able to do verified commits.
This was referenced Apr 19, 2023
We decided to push the docs updates not directly to branch synched with GitBook, but create a PR and set a GitBook-synched branch as a base branch for that PR. This resulted in changes in the `keep-network/ci/.github/workflows/reusable-solidity-docs.yml` workflow's inputs.
nkuba
added a commit
to keep-network/ci
that referenced
this pull request
May 15, 2023
Add reusable workflow for creation of Solidity API docs
In this commit we create a reusable workflow which automatically generates the Markdown contracts documentation based on the functions and the NatSpec-format comments in the Solidity files of the specified project. Depending on the value of the workflow's inputs, the workflow may sync the generated files with the specified directory in the external repository (we will use this to push the documentation to a directory that's synchronized with GitBook).
### How Markdown documentation gets created and pushed
1. The documentation gets created based on the content of the Solidity files in `contracts` folder of the specified project (`projectSubfolder`). Before we run the Docgen tool generating Markdown files, we may need to perform some slight changes to the input files, as some of the formatting we use in the `.sol` files of the projects where we want to use the action is interpreted by Docgen not the way we would like or is not completely in line with the NatSpec format:
- In many `@dev` clauses in the Solidity files we have the lists of requirements or other items which are constructed like this:
```
/// @dev Requirements:
/// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
/// Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt
/// auctor velit non eleifend. Nunc sit amet est non ligula condimentum
/// mattis.
/// - Item two. Quisque purus massa, pellentesque in viverra tempus,
/// aliquet nec urna.
```
This doesn't get recognized by Docgen as a list and is translated to
regular text, which is displayed as one continuous line. But when the space
characters between `///` and the text get removed from he `.sol` files, the
lists are recognized correctly (without breaking interpretation of other
features). That's why (unless `removeTrailingSpacesInComments` is set to
`false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity
files.
- There may be other problems with the formatting of the input files that may
need correction. To handle that we allow for specifying
`preProcessingCommand`, which can modify the input files according to need.
An example of such command:
`sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol`.
2. Once the files are ready, we install dependencies. It is assumed that the
project in which action will be run has the `solidity-docgen` package added
to the list of dev dependencies in `package.json`.
3. After project is installed we run the Docgen tool. In order for the tool to
work, the Hardhat config needs to be updated (see
https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The
workflow expects `outputDir` to be set to `generated-docs`. You may also need
to specify other configs, like `pages`, `templates` or `exclude`, according
to the needs of particular project.
The tool will generate to `generated-docs` either one `index.md` file (if
`pages` is set to `single`) or a set of files (if `pages` is set to `items`
or `files`), with names reflecting their content.
4. If docs are generated to one file, it may be useful to add a Table Of Contents
to that file, to ease the navigation. This will be done if `addTOC` is set
to `true`. We use `markdown-toc` tool for TOC generation. The specific
options that should be used during generation can be specified in the
`tocOptions` input.
5. Then (if `exportAsGHArtifacts` is set to true) we export the generated
file(s) as GH Actions artifacts. They will be available for download in the
details of the workflow run.
6. If artifacts were exported, workflow was triggered by a PR and the
`commentPR` was set to true, a comment with information where the artifacts
can be found will be posted in the PR.
7. If `publish` input was set, we then proceed to the steps pushing the
generated file(s) to a repository/path/branch specified in the workflow's
inputs. To do that we first need to make sure we use correct config to be
able to publish to the destination repository. We need to provide email and
a username of the user who will be author of the commit. If repository allows
only for commits from verified users, the `verifyCommits` input should be
set to `true` and the user have the GPG key configured (see
https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key).
The GPG private key and a passphrase to the key should be stored as GH
secrets in the repositories where the workflow will be used. The name of the
secrets should be passed to the workflow as `git_user_signingkey` and
`git_commit_gpgsign`.
Apart from configuring aforementioned commiter-related params, a GH secret
storing GitHub API token (personal access token) for the destination
repository needs to be provided in the `githubToken` secret.
If everything is configured, the workflow syncs the content of the folder
storing generated docs with the content of the destination directory (using
`rsync`). If `rsyncDelete` input is set to true, the files which exist in the
destination repository, but don't exist in the `generated-docs` folder will
be removed from the destination repo. One may want to set this option if
Docgen's `pages` attribute is set to `pages` or `items` (so that when a
contract or its function gets removed, we would remove the corresponding doc
from the destination repo). The setting should be used with caution!
Refs:
keep-network/tbtc-v2#584
keep-network/keep-core#3534
threshold-network/solidity-contracts#138
Change introduced to fix the 'Your push would publish a private email address.' error.
|
Solidity API documentation preview available in the artifacts of the https://github.com/threshold-network/solidity-contracts/actions/runs/5334375626 check. |
The workflow is meant to publish the docs on new releases, but as we already had releases when workflow wasn't implemented, we need to publish the docs representing the current state of the API. We are satisfied with docummenting based on the `main` branch state. After docs get publish, we need to revert this change.
|
Solidity API documentation preview available in the artifacts of the https://github.com/threshold-network/solidity-contracts/actions/runs/5387296030 check. |
|
The docs got published to this PR: https://github.com/threshold-network/threshold/pull/32/files. Slither check is failing, we need to merge #147 to fix this. |
|
Solidity API documentation preview available in the artifacts of the https://github.com/threshold-network/solidity-contracts/actions/runs/5388961465 check. |
|
Solidity API documentation preview available in the artifacts of the https://github.com/threshold-network/solidity-contracts/actions/runs/5389053900 check. |
michalinacienciala
added
the
documentation
Instead of running the `contracts-docs-publish-preview` job on update of every PR, we want to run it only when PR modifies contracts or the workflow itself.
pdyraga
approved these changes
Jun 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We are creating a GH Actions workflow which automatically generates the contracts documentation in Markdown, based on the functions and the NatSpec-format comments in the Solidity files stored in the
contractsfolder inthreshold-network/solidity-contracts. For certain workflow triggers, the generated documentation will be pushed to thethreshold-network/thresholdrepository, under./docs/app-development/staking-contract-and-dao/staking-contract-and-dao-api. As the.docsfolder is synched with Threshold docs GitBook space, the pushed docs will be displayed in HTML underdocs.threshold.networkdomain.There are two main jobs in the workflow:
contracts-docs-publish-previewcontracts-docs-publishBoth call a reusable workflow
reusable-solidity-docs.ymlwhich resides in thekeep-network/cirepository under.github/workflows. The jobs differ in parameters with witch the reusable action is called. The common part of the jobs is the beginning stage, where the Solidity files are being prepared for Markdown generation. During that stage the jobs will remove the spaces chars between the NatSpec comments markers (///) and the text of the comment. This is done to ensure proper rendering of the lists in the output docs and is a default action made byreusable-solidity-docs.yml.Once files are ready, the jobs use Docgen tool to generate the Markdown docs. The tool needs to be installed in the project and configured in the
hardhat.config.ts. The configuration that we use specifies that the docs should be generated to thegeerated-docssubfolder, to one singleindex.mdfile. The.solfiles in the./solidity/contracts/testfolder will be ignored during generation (those are test/stub contracts which are not used on Mainnet). A custom template will be used during docs generations. The template was created based on the defaulthttps://github.com/OpenZeppelin/solidity-docgen/blob/master/src/themes/markdown/common.hbs template, but it removes the cursive from the
@devtype comments (because the cursive didn't work well with formatting of the lists). Once the Docgen is run and theindex.mdfile is generated, the jobs will add a Table of Contents to the file, to improve the navigation. The TOC will be added by themarkdown-toctool and will be generated using--maxdepth 2param, which results in listing all the contract names, but not the functions. Once the TOC is added, we'll start to see the difference in the behavior of the jobs.The
contracts-docs-publish-previewjob is triggered for:pull_requestevents, if the PR modifies either files in./contractsor the workflow file itself,pushevents, when the push is made to a branch which name starts withreleases/mainnet/v,workflow_dispatchevents. The job publishes the generated artifacts as artifacts of the GH Actions workflow run. For PRs, link to the run with the artifacts will be posted as a PR comment.The
contracts-docs-publishjob is triggered for:refs/tags/v. The job pushes the generated file to the./docs/app-development/staking-contract-and-dao/staking-contract-and-dao-apifolder in thethreshold-network/thresholdrepo, to themainbranch. The.docsfolder is synched with the GitBook Threshold documentation, so thanks to the job, the changes in the contracts should be reflected on the docs.threshold.network. In order for the job to work, we need to configure a user which will create the commit The user needs to have right to commit to thethreshold-network/thresholdrepo. We also need to configure following secrets:THRESHOLD_DOCS_GITHUB_TOKENTHRESHOLD_DOCS_GPG_PRIVATE_KEY_BASE64THRESHOLD_DOCS_GPG_PASSPHRASEWe're using a GPG key to be able to do verified commits.Refs:
keep-network/ci#43
keep-network/tbtc-v2#584
keep-network/keep-core#3534