Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add reusable workflow for creation of Solidity API docs #43

Merged
merged 3 commits into from
May 15, 2023

Conversation

michalinacienciala
Copy link
Collaborator

@michalinacienciala michalinacienciala commented Apr 15, 2023

In this commit we create a reusable workflow which automatically generates the Markdown contracts documentation based on the functions and the NatSpec-format comments in the Solidity files of the specified project. Depending on the value of the workflow's inputs, the workflow may sync the generated files with the specified directory in the external repository (we will use this to push the documentation to a directory that's synchronized with GitBook).

How Markdown documentation gets created and pushed

  1. The documentation gets created based on the content of the Solidity files in contracts folder of the specified project (projectSubfolder). Before we run the Docgen tool generating Markdown files, we may need to perform some slight changes to the input files, as some of the formatting we use in the .sol files of the projects where we want to use the action is interpreted by Docgen not the way we would like or is not completely in line with the NatSpec format:
    • In many @dev clauses in the Solidity files we have the lists of requirements or other items which are constructed like this:
      /// @dev Requirements:
      /// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
      ///   Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt
      ///   auctor velit non eleifend. Nunc sit amet est non ligula condimentum
      ///   mattis.
      /// - Item two. Quisque purus massa, pellentesque in viverra tempus,
      ///   aliquet nec urna.
 This doesn't get recognized by Docgen as a list and is translated to
 regular text, which is displayed as one continuous line. But when the space
 characters between `///` and the text get removed from he `.sol` files, the
 lists are recognized correctly (without breaking interpretation of other
 features). That's why (unless `removeTrailingSpacesInComments` is set to
 `false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity
 files.
  • There may be other problems with the formatting of the input files that may
    need correction. To handle that we allow for specifying
    preProcessingCommand, which can modify the input files according to need.
    An example of such command:
    sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol.
  1. Once the files are ready, we install dependencies. It is assumed that the
    project in which action will be run has the solidity-docgen package added
    to the list of dev dependencies in package.json.
  2. After project is installed we run the Docgen tool. In order for the tool to
    work, the Hardhat config needs to be updated (see
    https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The
    workflow expects outputDir to be set to generated-docs. You may also need
    to specify other configs, like pages, templates or exclude, according
    to the needs of particular project.
    The tool will generate to generated-docs either one index.md file (if
    pages is set to single) or a set of files (if pages is set to items
    or files), with names reflecting their content.
  3. If docs are generated to one file, it may be useful to add a Table Of Contents
    to that file, to ease the navigation. This will be done if addTOC is set
    to true. We use markdown-toc tool for TOC generation. The specific
    options that should be used during generation can be specified in the
    tocOptions input.
  4. Then (if exportAsGHArtifacts is set to true) we export the generated
    file(s) as GH Actions artifacts. They will be available for download in the
    details of the workflow run.
  5. If artifacts were exported, workflow was triggered by a PR and the
    commentPR was set to true, a comment with information where the artifacts
    can be found will be posted in the PR.
  6. If publish input was set, we then proceed to the steps pushing the
    generated file(s) to a repository/path/branch specified in the workflow's
    inputs. To do that we first need to make sure we use correct config to be
    able to publish to the destination repository. We need to provide email and
    a username of the user who will be author of the commit. If repository allows
    only for commits from verified users, the verifyCommits input should be
    set to true and the user have the GPG key configured (see
    https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key).
    The GPG private key and a passphrase to the key should be stored as GH
    secrets in the repositories where the workflow will be used. The name of the
    secrets should be passed to the workflow as git_user_signingkey and
    git_commit_gpgsign.
    Apart from configuring aforementioned commiter-related params, a GH secret
    storing GitHub API token (personal access token) for the destination
    repository needs to be provided in the githubToken secret.
    If everything is configured, the workflow syncs the content of the folder
    storing generated docs with the content of the destination directory (using
    rsync). If rsyncDelete input is set to true, the files which exist in the
    destination repository, but don't exist in the generated-docs folder will
    be removed from the destination repo. One may want to set this option if
    Docgen's pages attribute is set to pages or items (so that when a
    contract or its function gets removed, we would remove the corresponding doc
    from the destination repo). The setting should be used with caution!

Refs:
keep-network/tbtc-v2#584
keep-network/keep-core#3534
threshold-network/solidity-contracts#138

In this commit we create a reusable workflow which automatically generates the
Markdown contracts documentation based on the functions and the NatSpec-format
comments in the Solidity files of the specified project. Depending on the value
of the workflow's inputs, the workflow may sync the generated files with the
specified directory in the external repository (we will use this to push the
documentation to a directory that's synchronized with GitBook).

---How Markdown documentation gets created and pushed---
1. The documentation gets created based on the content of the Solidity files in
   `contracts` folder of the specified project (`projectSubfolder`). Before we
   run the Docgen tool generating Markdown files, we may need to perform some
   slight changes to the input files, as some of the formatting we use in the
   `.sol` files of the projects where we want to use the action is interpreted
   by Docgen not the way we would like or is not completely in line with the
   NatSpec format:
   - In many `@dev` clauses in the Solidity files we have the lists of
     requirements or other items which are constructed like this:
     ```
      /// @dev Requirements:
      /// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
      ///   Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt
      ///   auctor velit non eleifend. Nunc sit amet est non ligula condimentum
      ///   mattis.
      /// - Item two. Quisque purus massa, pellentesque in viverra tempus,
      ///   aliquet nec urna.
     ```
     This doesn't get recognized by Docgen as a list and is translated to
     regular text, which is displayed as one continuous line. But when the space
     characters between `///` and the text get removed from he `.sol` files, the
     lists are recognized correctly (without breaking interpretation of other
     features). That's why (unless `removeTrailingSpacesInComments` is set to
     `false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity
     files.
   - There may be other problems with the formatting of the input files that may
     need correction. To handle that we allow for specifying
     `preProcessingCommand`, which can modify the input files according to need.
     An example of such command:
     `sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol`.
2. Once the files are ready, we install dependencies. It is assumed that the
   project in which action will be run has the `solidity-docgen` package added
   to the list of dev dependencies in `package.json`.
3. After project is installed we run the Docgen tool. In order for the tool to
   work, the Hardhat config needs to be updated (see
   https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The
   workflow expects `outputDir` to be set to `generated-docs`. You may also need
   to specify other configs, like `pages`, `templates` or `exclude`, according
   to the needs of particular project.
   The tool will generate to `generated-docs` either one `index.md` file (if
   `pages` is set to `single`) or a set of files (if `pages` is set to `items`
   or `files`), with names reflecting their content.
4. If docs are generated to one file, it may e useful to add a Table Of Contents
   to that file, to ease the navigation. This will be done if `addTOC` is set
   to `true`. We use `markdown-toc` tool for TOC generation. The specific
   options that should be used during generation can be specified in the
   `tocOptions` input.
5. Then (if `exportAsGHArtifacts` is set to true) we export the generated
   file(s) as GH Actions artifacts. They will be available for download in the
   details of the workflow run.
6. If artifacts were exported, workflow was triggered by a PR and the
   `commentPR` was set to true, a comment with information where the artifacts
   can be found will be posted in the PR.
7. If `publish` input was set, we then proceed to the steps pushing the
   generated file(s) to a repository/path/branch specified in the workflow's
   inputs. To do that we first need to make sure we use correct config to be
   able to publish to the destination repository. We need to provide email and
   a username of the user who will be author of the commit. If repository allows
   only for commits from verified users, the `verifyCommits` input should be
   set to `true` and the user have the GPG key configured (see
   https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key).
   The GPG private key and a passphrase to the key should be stored as GH
   secrets in the repositories where the workflow will be used. The name of the
   secrets should be passed to the workflow as `git_user_signingkey` and
   `git_commit_gpgsign`.
   Apart from configuring aforementioned commiter-related params, a GH secret
   storing GitHub API token (personal access token) for the destination
   repository needs to be provided in the `githubToken` secret.
   If everything is configured, the workflow syncs the content of the folder
   storing generated docs with the content of the destination directory (using
   `rsync`). If `rsyncDelete` input is set to true, the files which exist in the
   destination repository, but don't exist in the `generated-docs` folder will
   be removed from the destination repo. One may want to set this option if
   Docgen's `pages` attribute is set to `pages` or `items` (so that when a
   contract or its function gets removed, we would remove the corresponding doc
   from the destination repo). The setting should be used with caution!
Copy link
Member

@nkuba nkuba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impressive work! 💯

.github/workflows/reusable-solidity-docs.yml Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Outdated Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Outdated Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Outdated Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Outdated Show resolved Hide resolved
.github/workflows/reusable-solidity-docs.yml Outdated Show resolved Hide resolved
Previous version of the workflow assumed the workflow will be used to push the
updated docs directly to a branch that's synchronized with GitBook. But we
decided it will be safer to create a PR in the destination repo instead, that
will have the GitBook-synchronized branch set as a PR base branch.

Workflows that will use our reusable workflow will need to specify
`destinationBaseBranch` input (if not, `main` will be set as a default).

Objectives:
The destination repo may be in a bunch of different states when workflow is
executed, we wanted to handle those states. It is assumed that the reusable
worklow will be triggered if there's a `release` event in the project using this
workflow.

The release may not change anything in the contracts functions and comments
resulting in no changes between docs which are already in the destination repo
and those which got generated by the workflow. In that case we don't want to
push anything and create/update any PRs.

Probably the most common situation will be a case where a release modifies
something in contracts' functions/comments, resulting in a need for the docs
update. In such case we want to commit the changes and create a PR in the
destination repo. If there is already an earlier open PR updating the docs, we
want to update it instead of creating a new one.

To handle above objectives:
1. We configure `base_branch` and `head_branch` environment variables. The
   `base_branch` variable is set to value passed from `destinationBaseBranch`
   input. The `head_branch` will be set to `auto-update-solidity-api-docs`.
2. Next, we clone the `destinationRepo` and name it `dest-repo-clone` (we change
   the name because we want to avoid a situation where there's already a folder
   with the name of the destination repo present in repo where we run the
   workflow). All branches are cloned, and `head_branch` gets checked out.
3. We enter the cloned repo and checkout the `head_branch` (if it does not
   exist, we create a new branch under this name - based on `head_branch`- and
   checkout it).
4. If destination directory for docs does not exist, we create it. Then we
   synchronize the content of the `../generated-docs` folder with the content of
   destination folder.
5. Then we stage all the changes made in the `destinationRepo` and check if
   anything was staged. If nothing was staged, this means there were no changes
   to the dosc and we can finish the workflow without doing any changes to the
   destination repo (we end at step 5). If something was staged, we proceed to
   step 6.
6. We configure commiter in Git and commit the changes.
7. We push the commit to the `head_branch`.
8. Then we check if there's already an open PR existing for the `head_branch`. We
   do that by executing a curl command that lists all the open PRs in the
   `destinationRepo` that have the head branch set to `head_branch`. If no PRs
   match the criteria, the curl returns `[\n\n]`. If there's already a PR
   existing, then we're ok and we don't need to do anything else (PR got updated
   in step 7). If not, we create a PR using `hub` CLI tool
   (https://github.com/github/hub).
# projects's dev dependencies and is configured in Hardhat config (see
# https://github.com/OpenZeppelin/solidity-docgen#usage).
# The workflow expects `outputDir` to be set to `generated-docs`. You may
# also need to specify other configs, like `temlates` or `exclude`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# also need to specify other configs, like `temlates` or `exclude`.
# also need to specify other configs, like `templates` or `exclude`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Generate documentation for smart contracts
2 participants