Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatically generate contracts documentation #3534

Merged
merged 12 commits into from
Jun 28, 2023
Merged

Automatically generate contracts documentation #3534

merged 12 commits into from
Jun 28, 2023

Conversation

michalinacienciala
Copy link
Contributor

@michalinacienciala michalinacienciala commented Apr 19, 2023 •
edited
Loading

We are creating GH Actions workflows which automatically generates the contracts documentation in Markdown, based on the functions and the NatSpec-format comments in the Solidity files stored in the contracts folders in keep-network/keep-core/solidity/random-beacon and keep-network/keep-core/solidity/ecdsa. For certain workflow triggers, the generated documentation will be pushed to the threshold-network/threshold repository, under ./docs/app-development/random-beacon/random-beacon-api / ./docs/app-development/ecdsa/ecdsa-api. As the .docs folder is synched with Threshold docs GitBook space, the pushed docs will be displayed in HTML under docs.threshold.network domain.

There are two main jobs in the workflows:

  • contracts-docs-publish-preview
  • contracts-docs-publish

Both call a reusable workflow reusable-solidity-docs.yml which resides in the keep-network/ci repository under .github/workflows. The jobs differ in parameters with witch the reusable action is called. The common part of the jobs is the beginning stage, where the Solidity files are being prepared for Markdown generation. During that stage the jobs will remove the spaces chars between the NatSpec comments markers (///) and the text of the comment. This is done to ensure proper rendering of the lists in the output docs and is a default action made by reusable-solidity-docs.yml.
Once files are ready, the jobs use Docgen tool to generate the Markdown docs. The tool needs to be installed in the project and configured in the hardhat.config.ts. The configuration that we use specifies that the docs should be generated to the geerated-docs subfolder, to one single index.md file. The .sol files in the ./solidity/contracts/test folder will be ignored during generation (those are test/stub contracts which are not used on Mainnet). A custom template will be used during docs generations. The template was created based on the default
https://github.com/OpenZeppelin/solidity-docgen/blob/master/src/themes/markdown/common.hbs template, but it removes the cursive from the @dev type comments (because the cursive didn't work well with formatting of the lists). Once the Docgen is run and the index.md file is generated, the jobs will add a Table of Contents to the file, to improve the navigation. The TOC will be added by the markdown-toc tool and will be generated using --maxdepth 2 param, which results in listing all the contract names, but not the functions. Once the TOC is added, we'll start to see the difference in the behavior of the jobs.

The contracts-docs-publish-preview job is triggered for:

  • pull_request events, if the PR modifies either files in ./solidity/<project>/contracts or the workflow files itself,
  • push events, when the push is made to a branch which name starts with releases/mainnet/solidity/ and project's files are modified,
  • workflow_dispatch events. The job publishes the generated artifacts as artifacts of the GH Actions workflow run. For PRs, link to the run with the artifacts will be posted as a PR comment.

The contracts-docs-publish job is triggered for:

  • published releases, which names start with refs/tags/solidity/. The job pushes the generated file to the
    ./docs/app-development/random-beacon/random-beacon-api / ./docs/app-development/ecdsa/ecdsa-apifolder in the threshold-network/threshold repo, to the main branch. The .docs folder is synched with the GitBook Threshold documentation, so thanks to the job, the changes in the contracts should be reflected on the docs.threshold.network. In order for the job to work, we need to configure a user which will create the commit The user needs to have right to commit to the threshold-network/threshold repo. We also need to configure following secrets:
  • THRESHOLD_DOCS_GITHUB_TOKEN
  • THRESHOLD_DOCS_GPG_PRIVATE_KEY_BASE64
  • THRESHOLD_DOCS_GPG_PASSPHRASE We're using a GPG key to be able to do verified commits.

Refs:
keep-network/ci#43
keep-network/tbtc-v2#584
threshold-network/solidity-contracts#138

@michalinacienciala
Automatically generate contracts documentation
2ca6598 
We are creating GH Actions workflows which automatically generates the contracts
documentation in Markdown, based on the functions and the NatSpec-format
comments in the Solidity files stored in the `contracts` folders in
`keep-network/keep-core/solidity/random-beacon` and
`keep-network/keep-core/solidity/ecdsa`. For certain workflow triggers, the
generated documentation will be pushed to the `threshold-network/threshold`
repository, under `./docs/app-development/random-beacon/random-beacon-api` /
`./docs/app-development/ecdsa/ecdsa-api`. As the `.docs` folder is synched with
Threshold docs GitBook space, the pushed docs will be displayed in HTML under
`docs.threshold.network` domain.

There are two main jobs in the workflows:

* `contracts-docs-publish-preview`
* `contracts-docs-publish`

Both call a reusable workflow `reusable-solidity-docs.yml` which resides in the
`keep-network/ci` repository under `.github/workflows`. The jobs differ in
parameters with witch the reusable action is called. The common part of the jobs
is the beginning stage, where the Solidity files are being prepared for Markdown
generation. During that stage the jobs will remove the spaces chars between the
NatSpec comments markers (`///`) and the text of the comment. This is done to
ensure proper rendering of the lists in the output docs and is a default
action made by `reusable-solidity-docs.yml`. Another pre-processing is
running a `sed` command on a `./contracts/bridge/BitcoinTx.sol` that removes
incorrectly used line with `//` blank comment in the middle of section with
NatSpec's `///` comments (which was breaking the formatting of that comment in
Markdown).
Once files are ready, the jobs use Docgen tool to generate the Markdown docs.
The tool needs to be installed in the project and configured in the
`hardhat.config.ts`. The configuration that we use specifies that the docs
should be generated to the `geerated-docs` subfolder, to one single `index.md`
file. The `.sol` files in the `./solidity/contracts/test` folder will be
ignored during generation (those are test/stub contracts which are not used on
Mainnet). A custom template will be used during docs generations. The template
was created based on the default
`https://github.com/OpenZeppelin/solidity-docgen/blob/master/src/themes/markdown/common.hbs`
template, but it removes the cursive from the `@dev` type comments (because the
cursive didn't work well with formatting of the lists).
Once the Docgen is run and the `index.md` file is generated, the jobs will add
a Table of Contents to the file, to improve the navigation. The TOC will be
added by the `markdown-toc` tool and will be generated using `--maxdepth 2`
param, which results in listing all the contract names, but not the functions.
Once the TOC is added, we'll start to see the difference in the behavior of the
jobs.

The `contracts-docs-publish-preview` job is triggered for:
* `pull_request` events, if the PR modifies either files if
  `./solidity/<project>/contracts` or the workflow files itself,
* `push` events, when the push is made to a branch which name starts with
  `releases/mainnet/solidity/` and project's files are modified,
* `workflow_dispatch` events.
The job publishes the generated artifacts as artifacts of the GH Actions
workflow run. For PRs, link to the run with the artifacts will be posted as a PR
comment.

The `contracts-docs-publish` job is triggered for:
* published releases, which names start with `refs/tags/solidity/`.
The job pushes the generated file to the
`./docs/app-development/random-beacon/random-beacon-api` /
`./docs/app-development/ecdsa/ecdsa-api`folder in the
`threshold-network/threshold` repo, to the `main` branch. The `.docs` folder
is synched with the GitBook Threshold documentation, so thanks to the job, the
changes in the contracts should be reflected on the docs.threshold.network.
In order for the job to work, we need to configure a user which will create the
commit The user needs to have right to commit to the
`threshold-network/threshold` repo. We also need to configure following secrets:
- [ ] `THRESHOLD_DOCS_GITHUB_TOKEN`
- [ ] `GPG_PRIVATE_KEY`
- [ ] `GPG_PASSPHRASE`
We're using a GPG key to be able to do verified commits.
This was referenced Apr 19, 2023
Merged
Merged
Merged
@michalinacienciala michalinacienciala linked an issue Apr 19, 2023 that may be closed by this pull request
Generate documentation for smart contracts keep-network/tbtc-v2#553
Closed
@michalinacienciala
Rename input with project directory
62aaae9 
The name of the input in the reusable action has changed, we're updating the
name in the jobs that use it.
@michalinacienciala
Update destination branch input's name
daaf3e1 
We decided to push the docs updates not directly to branch synched with GitBook,
but create a PR and set a GitBook-synched branch as a base branch for that PR.
This resulted in changes in the
`keep-network/ci/.github/workflows/reusable-solidity-docs.yml`
workflow's inputs.
nkuba added a commit to keep-network/ci that referenced this pull request May 15, 2023
@nkuba
Merge pull request #43 from keep-network/generate-solidity-docs
6cb91a9 
Add reusable workflow for creation of Solidity API docs

In this commit we create a reusable workflow which automatically generates the Markdown contracts documentation based on the functions and the NatSpec-format comments in the Solidity files of the specified project. Depending on the value of the workflow's inputs, the workflow may sync the generated files with the specified directory in the external repository (we will use this to push the documentation to a directory that's synchronized with GitBook).

### How Markdown documentation gets created and pushed
1. The documentation gets created based on the content of the Solidity files in `contracts` folder of the specified project (`projectSubfolder`). Before we run the Docgen tool generating Markdown files, we may need to perform some slight changes to the input files, as some of the formatting we use in the `.sol` files of the projects where we want to use the action is interpreted by Docgen not the way we would like or is not completely in line with the NatSpec format:
   - In many `@dev` clauses in the Solidity files we have the lists of requirements or other items which are constructed like this:
```
      /// @dev Requirements:
      /// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
      ///   Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt
      ///   auctor velit non eleifend. Nunc sit amet est non ligula condimentum
      ///   mattis.
      /// - Item two. Quisque purus massa, pellentesque in viverra tempus,
      ///   aliquet nec urna.
```
     This doesn't get recognized by Docgen as a list and is translated to
     regular text, which is displayed as one continuous line. But when the space
     characters between `///` and the text get removed from he `.sol` files, the
     lists are recognized correctly (without breaking interpretation of other
     features). That's why (unless `removeTrailingSpacesInComments` is set to
     `false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity
     files.
   - There may be other problems with the formatting of the input files that may
     need correction. To handle that we allow for specifying
     `preProcessingCommand`, which can modify the input files according to need.
     An example of such command:
     `sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol`.
2. Once the files are ready, we install dependencies. It is assumed that the
   project in which action will be run has the `solidity-docgen` package added
   to the list of dev dependencies in `package.json`.
3. After project is installed we run the Docgen tool. In order for the tool to
   work, the Hardhat config needs to be updated (see
   https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The
   workflow expects `outputDir` to be set to `generated-docs`. You may also need
   to specify other configs, like `pages`, `templates` or `exclude`, according
   to the needs of particular project.
   The tool will generate to `generated-docs` either one `index.md` file (if
   `pages` is set to `single`) or a set of files (if `pages` is set to `items`
   or `files`), with names reflecting their content.
4. If docs are generated to one file, it may be useful to add a Table Of Contents
   to that file, to ease the navigation. This will be done if `addTOC` is set
   to `true`. We use `markdown-toc` tool for TOC generation. The specific
   options that should be used during generation can be specified in the
   `tocOptions` input.
5. Then (if `exportAsGHArtifacts` is set to true) we export the generated
   file(s) as GH Actions artifacts. They will be available for download in the
   details of the workflow run.
6. If artifacts were exported, workflow was triggered by a PR and the
   `commentPR` was set to true, a comment with information where the artifacts
   can be found will be posted in the PR.
7. If `publish` input was set, we then proceed to the steps pushing the
   generated file(s) to a repository/path/branch specified in the workflow's
   inputs. To do that we first need to make sure we use correct config to be
   able to publish to the destination repository. We need to provide email and
   a username of the user who will be author of the commit. If repository allows
   only for commits from verified users, the `verifyCommits` input should be
   set to `true` and the user have the GPG key configured (see
   https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key).
   The GPG private key and a passphrase to the key should be stored as GH
   secrets in the repositories where the workflow will be used. The name of the
   secrets should be passed to the workflow as `git_user_signingkey` and
   `git_commit_gpgsign`.
   Apart from configuring aforementioned commiter-related params, a GH secret
   storing GitHub API token (personal access token) for the destination
   repository needs to be provided in the `githubToken` secret.
   If everything is configured, the workflow syncs the content of the folder
   storing generated docs with the content of the destination directory (using
   `rsync`). If `rsyncDelete` input is set to true, the files which exist in the
   destination repository, but don't exist in the `generated-docs` folder will
   be removed from the destination repo. One may want to set this option if
   Docgen's `pages` attribute is set to `pages` or `items` (so that when a
   contract or its function gets removed, we would remove the corresponding doc
   from the destination repo). The setting should be used with caution!

Refs:
keep-network/tbtc-v2#584
keep-network/keep-core#3534
threshold-network/solidity-contracts#138
@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5334377481 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5334377479 check.

@michalinacienciala
Merge remote-tracking branch 'origin/main' into document-contracts-gi…
f247ad1 
…tbook
@michalinacienciala
Add temporary config to publish the docs from main
c7f233c 
The workflow is meant to publish the docs on new releases, but as we already
had releases when workflow wasn't implemented, we need to publish the docs
representing the current state of the API. We are satisfied with docummenting
based on the `main` branch state. After docs get publish, we need to revert this
change.
@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5387366465 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5387366462 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388134304 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388134297 check.

@michalinacienciala
Avoid executing contracts-docs-publish parallelly
19e2a6a 
As triggers for the `Publish contracts documentation` jobs in the workflows
generating Random Beacon and ECDSA API docs are the same (and also the
previously executed steps are very similar, it's likely that the both jobs will
start and the same or very similar time.
When that happens it's very likely that the both jobs will pull the
`auto-update-solidity-api-docs` branch at the same state and one of the jobs
will push a commit with updated docs, while the other won't be able to do that,
as the state of the branch will be different than when the pull happened.
The error we see in that case is following:
```
error: failed to push some refs to 'https://github.com/threshold-network/threshold.git'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```
To avoid such situations we add a 4 minutes wait before we execute the 'Publish
contracts documentation' job in ECDSA workflow. This should be more than
enught for the Random Beacon's `Publish contracts documentation` to finish.

There is still a small change that both `Publish contracts documentation` jobs
will run at similar time (if something slows down the execution of the Random
Beacon workflow) and cause a failure of one of them, but in that unlikely
scenario, we can always rerun the failing job and it should pass on that second
run.
@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388647526 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388647527 check.

@michalinacienciala michalinacienciala marked this pull request as ready for review June 27, 2023 10:33
@michalinacienciala
Copy link
Contributor Author

The docs got published to this PR: https://github.com/threshold-network/threshold/pull/32/files.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388749047 check.

@github-actions
Copy link

Solidity API documentation preview available in the artifacts of the https://github.com/keep-network/keep-core/actions/runs/5388749037 check.

@michalinacienciala michalinacienciala added the 📖 documentation People need to know things! label Jun 27, 2023
pdyraga
pdyraga approved these changes Jun 28, 2023
View reviewed changes
.github/workflows/contracts-ecdsa-docs.yml
# This job will be triggered for releases which name starts with
# `refs/tags/solidity/`. It will generate contracts documentation in
# Markdown and sync it with a specific path of
# `threshold-network/threshold` repository. If changes will be detected,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep in mind to update this comment as a part of the threshold-network/threshold repo rename work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already taken care of in #3653 👍

@pdyraga pdyraga merged commit b2c6e58 into main Jun 28, 2023
@pdyraga pdyraga deleted the document-contracts-gitbook branch June 28, 2023 08:02
This was referenced Jun 28, 2023
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pdyraga pdyraga pdyraga approved these changes

Assignees

@michalinacienciala michalinacienciala

Labels
📖 documentation People need to know things!
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Generate documentation for smart contracts
2 participants
@michalinacienciala @pdyraga