Merge pull request #43 from keep-network/generate-solidity-docs

Add reusable workflow for creation of Solidity API docs

In this commit we create a reusable workflow which automatically generates the Markdown contracts documentation based on the functions and the NatSpec-format comments in the Solidity files of the specified project. Depending on the value of the workflow's inputs, the workflow may sync the generated files with the specified directory in the external repository (we will use this to push the documentation to a directory that's synchronized with GitBook).

### How Markdown documentation gets created and pushed
1. The documentation gets created based on the content of the Solidity files in `contracts` folder of the specified project (`projectSubfolder`). Before we run the Docgen tool generating Markdown files, we may need to perform some slight changes to the input files, as some of the formatting we use in the `.sol` files of the projects where we want to use the action is interpreted by Docgen not the way we would like or is not completely in line with the NatSpec format:
   - In many `@dev` clauses in the Solidity files we have the lists of requirements or other items which are constructed like this:
```
      /// @dev Requirements:
      /// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
      ///   Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt
      ///   auctor velit non eleifend. Nunc sit amet est non ligula condimentum
      ///   mattis.
      /// - Item two. Quisque purus massa, pellentesque in viverra tempus,
      ///   aliquet nec urna.
```
     This doesn't get recognized by Docgen as a list and is translated to
     regular text, which is displayed as one continuous line. But when the space
     characters between `///` and the text get removed from he `.sol` files, the
     lists are recognized correctly (without breaking interpretation of other
     features). That's why (unless `removeTrailingSpacesInComments` is set to
     `false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity
     files.
   - There may be other problems with the formatting of the input files that may
     need correction. To handle that we allow for specifying
     `preProcessingCommand`, which can modify the input files according to need.
     An example of such command:
     `sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol`.
2. Once the files are ready, we install dependencies. It is assumed that the
   project in which action will be run has the `solidity-docgen` package added
   to the list of dev dependencies in `package.json`.
3. After project is installed we run the Docgen tool. In order for the tool to
   work, the Hardhat config needs to be updated (see
   https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The
   workflow expects `outputDir` to be set to `generated-docs`. You may also need
   to specify other configs, like `pages`, `templates` or `exclude`, according
   to the needs of particular project.
   The tool will generate to `generated-docs` either one `index.md` file (if
   `pages` is set to `single`) or a set of files (if `pages` is set to `items`
   or `files`), with names reflecting their content.
4. If docs are generated to one file, it may be useful to add a Table Of Contents
   to that file, to ease the navigation. This will be done if `addTOC` is set
   to `true`. We use `markdown-toc` tool for TOC generation. The specific
   options that should be used during generation can be specified in the
   `tocOptions` input.
5. Then (if `exportAsGHArtifacts` is set to true) we export the generated
   file(s) as GH Actions artifacts. They will be available for download in the
   details of the workflow run.
6. If artifacts were exported, workflow was triggered by a PR and the
   `commentPR` was set to true, a comment with information where the artifacts
   can be found will be posted in the PR.
7. If `publish` input was set, we then proceed to the steps pushing the
   generated file(s) to a repository/path/branch specified in the workflow's
   inputs. To do that we first need to make sure we use correct config to be
   able to publish to the destination repository. We need to provide email and
   a username of the user who will be author of the commit. If repository allows
   only for commits from verified users, the `verifyCommits` input should be
   set to `true` and the user have the GPG key configured (see
   https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key).
   The GPG private key and a passphrase to the key should be stored as GH
   secrets in the repositories where the workflow will be used. The name of the
   secrets should be passed to the workflow as `git_user_signingkey` and
   `git_commit_gpgsign`.
   Apart from configuring aforementioned commiter-related params, a GH secret
   storing GitHub API token (personal access token) for the destination
   repository needs to be provided in the `githubToken` secret.
   If everything is configured, the workflow syncs the content of the folder
   storing generated docs with the content of the destination directory (using
   `rsync`). If `rsyncDelete` input is set to true, the files which exist in the
   destination repository, but don't exist in the `generated-docs` folder will
   be removed from the destination repo. One may want to set this option if
   Docgen's `pages` attribute is set to `pages` or `items` (so that when a
   contract or its function gets removed, we would remove the corresponding doc
   from the destination repo). The setting should be used with caution!

Refs:
keep-network/tbtc-v2#584
keep-network/keep-core#3534
threshold-network/solidity-contracts#138

.github/workflows/reusable-solidity-docs.yml

@@ -0,0 +1,273 @@
+name: Generate and push Solidity API documentation
+
+# To use this workflow, make sure `solidity-docgen` package is added to the 
+# projects's dev dependencies and is configured in Hardhat config (see
+# https://github.com/OpenZeppelin/solidity-docgen#usage).
+# The workflow expects `outputDir` to be set to `generated-docs`. You may
+# also need to specify other configs, like `temlates` or `exclude`.
+on:
+  workflow_call:
+    inputs:
+      projectDir:
+        description: "A root's subfolder containing the `contracts` folder and
+          the Yarn / Hardhat configuration. Leave empty ('') if the project
+          config resides in the root. If config resides in a subfolder, provide
+          the subfolder name/path, without leading `.` and ending `/`. For
+          example, type `/v2/solidity` if the path to the `contracts` folder is
+          `./v2/solidity/contracts`."
+        type: string
+        required: false
+        default: ""
+      trimComments:
+        description: "True if you want to remove space characters between `///`
+          and a comment text in the Solidity files, before generating `.md`
+          files. This may be needed to improve the look of the lists in the
+          generated files."
+        type: boolean
+        required: false
+        default: true
+      preProcessingCommand:
+        description: "An optional additional bash command to be executed before
+          transforming the Solidity files to an HTML file. The command will be
+          executed in the path specified by `projectDir`."
+        type: string
+        required: false
+      addTOC:
+        description: "True if you want to add Table Of Contents to the generated
+          Markdown file. Uses `markdown-toc` tool to generate TOC. Shouldn't be
+          set to `true` if Docgen's `pages` attribute in `hardhat.config.ts` is
+          set to `items` or `files`."
+        type: boolean
+        required: false
+        default: true
+      tocOptions:
+        description: "Options to be passed to the `markdown-toc` tool used for
+          generation of Table Of Contents. See
+          https://github.com/jonschlinkert/markdown-toc#cli for more details."
+        type: string
+        required: false
+        default: "-i --maxdepth 2"
+      publish:
+        description: "True if you want to push the generated file(s) to the
+          destination repository."
+        type: boolean
+        required: false
+        default: true
+      verifyCommits:
+        description: "True if you want to sign commits updating the docs using
+          GPG key. You'll need to specify `gpgPrivateKey` and `gpgPassphrase`
+          secrets."
+        type: boolean
+        required: false
+        default: false
+      destinationRepo:
+        description: "A name of the repository where the generated Solidity docs
+          will be pushed to. Required if `publish=true`."
+        type: string
+        required: false
+      destinationFolder:
+        description: "A path in the destination repository where the generated
+          Solidity docs will be pushed to. For example, `./docs-api/tbtc-v2`."
+        type: string
+        required: false
+        default: "."
+      destinationBaseBranch:
+        description: "A branch on the destination repository which will be set
+          as a base branch of the PR updating Solidity docs (must exist)."
+        type: string
+        required: false
+        default: "main"
+      userEmail:
+        description: "The email address of a GitHub user associated with the
+          `githubToken` secret. Will be used to sign commits. Required if
+          `publish=true`."
+        type: string
+        required: false
+      userName:
+        description: "The name of the GitHub user associated with the
+          `githubToken` secret. Will be used to sign commits. Required if
+          `publish=true`."
+        type: string
+        required: false
+      rsyncDelete:
+        description: "True if you want to delete from the destination folder
+          the files which were not generated during current run of the workflow.
+          Usually there shouldn't be need to set this option to `true` if
+          Docgen's `pages` attribute in `hardhat.config.ts` is set to `single`.
+          In other cases the option should be use with caution."
+        type: boolean
+        required: false
+        default: false
+      commentPR:
+        description: "True if you want to add a comment with the path to the
+          generated files in the PR invoking the workflow. If the workflow is
+          not triggered by the `pull_request` event, having this input set to
+          `true` will not brake the execution."
+        type: boolean
+        required: false
+        default: false
+      exportAsGHArtifacts:
+        description: "True if you want to see the artifacts generated on various
+          stages of workflow execution in the GH run details."
+        type: boolean
+        required: false
+        default: false
+    secrets:
+      githubToken:
+        description: "A GitHub API token for the destination organization.
+          Should have `repo` scope. Required if `publish=true`."
+        required: false
+      gpgPrivateKey:
+        description: "A GPG private key needed when `verifyCommits` is set to
+          true. See
+          https://github.com/crazy-max/ghaction-import-gpg#prerequisites for
+          instructions on key generation."
+        required: false
+      gpgPassphrase:
+        description: "A passphrase of the GPG key. Needed when `verifyCommits`
+          is set to true."
+        required: false
+
+jobs:
+  docs-generate-html-and-publish:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: .${{ inputs.projectDir }}
+    steps:
+      - uses: actions/checkout@v3
+
+      # In this step we modify the format of the comments in the Solidity
+      # contracts files. We do that because our original formatting is not
+      # processed by Docgen in the way we would like.
+      # To nicely display lists (like the list of requirements) we need to
+      # remove multiple space chars after the `///` comment. We do that by
+      # executing `sed 's_///[[:blank:]]*_///_'` on all contracts files, which
+      # substitutes `///` + 0 or more spaces/tabs with just `///`.
+
+      - name: Prepare contract files for further processing
+        if: inputs.trimComments
+        shell: bash
+        run: |
+          find ./contracts \
+            -name "*.sol" \
+            -type f \
+            -exec sed -i 's_///[[:blank:]]*_///_' {} \;
+
+      - name: Execute additional command
+        if: inputs.preProcessingCommand != null
+        shell: bash
+        run: ${{ inputs.preProcessingCommand }}
+
+      - name: Export artifacts
+        if: inputs.exportAsGHArtifacts == true
+        uses: actions/upload-artifact@master
+        with:
+          name: contracts-after-preprocessing
+          path: .${{ inputs.projectDir }}/contracts
+
+      # We may need this step in case we execute the workflow in a project
+      # that has a dependency to `@summa-tx/[email protected]` package, which
+      # downloads one of its sub-dependencies via unathenticated `git://`
+      # protocol. That protocol is no longer supported. Thanks to this step
+      # `https://` is used instead of `git://`.
+      - name: Configure git to don't use unauthenticated protocol
+        shell: bash
+        run: git config --global url."https://".insteadOf git://
+
+      - name: Install dependencies
+        shell: bash
+        run: yarn install --frozen-lockfile
+
+      # Generates `.md` file(s) based on config in the `hardhat.config.ts`.
+      - name: Build Markdown docs
+        run: yarn run hardhat docgen
+
+      - name: Add Table of Contents
+        if: inputs.addTOC == true
+        run: |
+          yarn global add markdown-toc 
+          sed -i '2s/^/\<!-- toc --\> \n/' ./generated-docs/index.md
+          sed -i '2s/^/## Table Of Contents \n/' ./generated-docs/index.md
+          markdown-toc ${{ inputs.tocOptions }} ./generated-docs/index.md
+
+      - name: Export artifacts
+        if: inputs.exportAsGHArtifacts == true
+        uses: actions/upload-artifact@master
+        with:
+          name: contracts-final-output
+          path: .${{ inputs.projectDir }}/generated-docs
+
+      - name: Post link to artifacts in PR comment
+        if: |
+          inputs.exportAsGHArtifacts == true
+            && inputs.commentPR == true
+            && startsWith(github.ref, 'refs/pull')
+        uses: actions/github-script@v6
+        with:
+          script: |
+            github.rest.issues.createComment({
+              issue_number: context.issue.number,
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              body: 'Solidity API documentation preview available in the artifacts of the https://github.com/${{ github.repository}}/actions/runs/${{ github.run_id}} check.'
+            })
+
+      - name: Import GPG key
+        if: inputs.publish == true && inputs.verifyCommits == true
+        uses: crazy-max/ghaction-import-gpg@111c56156bcc6918c056dbef52164cfa583dc549 # v5.2.0
+        with:
+          gpg_private_key: ${{ secrets.gpgPrivateKey }}
+          passphrase: ${{ secrets.gpgPassphrase }}
+          git_user_signingkey: true
+          git_commit_gpgsign: true
+
+      - name: Sync generated docs with the specified repository and create PR
+        if: inputs.publish == true
+        env:
+          API_TOKEN_GITHUB: ${{ secrets.githubToken }}
+        run: |
+          echo "▶ Configure environment variables"
+            head_branch=auto-update-solidity-api-docs
+            base_branch=${{ inputs.destinationBaseBranch }}
+          echo "▶ Checkout destination repo"
+            git clone --branch $base_branch \
+              https://x-access-token:[email protected]/${{ inputs.destinationRepo }}.git \
+              dest-repo-clone
+          echo "▶ Create/checkout head branch"
+            cd dest-repo-clone
+            git checkout $head_branch || git checkout -b $head_branch
+          echo "▶ Synchronize docs"
+            mkdir -p ${{ inputs.destinationFolder }}
+            rsync -avh ${{ inputs.rsyncDelete && '--delete' || '' }} \
+              ../generated-docs \
+              ${{ inputs.destinationFolder }}
+          echo "▶ Commit changes"
+            git add -A
+            if git status | grep -q "Changes to be committed"
+            then
+              git config --global user.email ${{ inputs.userEmail }}
+              git config --global user.name ${{ inputs.userName }}
+              git commit ${{ inputs.verifyCommits && '-S' || '' }} \
+                -m "Update docs by https://github.com/${{ github.repository}}/actions/runs/${{ github.run_id}}"
+              echo "▶ Push commit"
+                git push --set-upstream origin HEAD:$head_branch
+              echo "▶ Check if PR for the head branch already exists"
+                dest_org=${${{ inputs.destinationRepo }}%%/*}
+                pr_for_head=$(curl -L \
+                  -H "Accept: application/vnd.github+json" \
+                  -H "Authorization: Bearer $API_TOKEN_GITHUB" \
+                  -H "X-GitHub-Api-Version: 2022-11-28" \
+                  "https://api.github.com/repos/${{ inputs.destinationRepo }}/pulls?status=open&head=$dest_org:$head_branch")
+                if [[ $pr_for_head == $'[\n\n]' ]]; then
+                  echo "▶ Checked. A PR for the head branch ($head_branch) will be created"
+                    hub pull-request --base $base_branch \
+                      --message "Update Solidity API docs" \
+                      --message "Docs updated by workflow: https://github.com/${{ github.repository}}/actions/runs/${{ github.run_id}}"
+                else
+                  echo "▶ Checked. A PR for head branch ($head_branch) already exists and got updated."
+                fi
+            else
+              echo "▶ No changes detected, no commits will be made."
+              exit 0
+            fi