From b5c7e63355846c238b0e2a7da0dfed351d7658de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Michalina=20Ciencia=C5=82a?= Date: Thu, 13 Apr 2023 07:36:33 +0200 Subject: [PATCH 1/3] Add reusable workflow for creation of Solidity API docs In this commit we create a reusable workflow which automatically generates the Markdown contracts documentation based on the functions and the NatSpec-format comments in the Solidity files of the specified project. Depending on the value of the workflow's inputs, the workflow may sync the generated files with the specified directory in the external repository (we will use this to push the documentation to a directory that's synchronized with GitBook). ---How Markdown documentation gets created and pushed--- 1. The documentation gets created based on the content of the Solidity files in `contracts` folder of the specified project (`projectSubfolder`). Before we run the Docgen tool generating Markdown files, we may need to perform some slight changes to the input files, as some of the formatting we use in the `.sol` files of the projects where we want to use the action is interpreted by Docgen not the way we would like or is not completely in line with the NatSpec format: - In many `@dev` clauses in the Solidity files we have the lists of requirements or other items which are constructed like this: ``` /// @dev Requirements: /// - Item one. Lorem ipsum dolor sit amet, consectetur adipiscing elit. /// Nulla sed porttitor enim, sit amet venenatis enim. Donec tincidunt /// auctor velit non eleifend. Nunc sit amet est non ligula condimentum /// mattis. /// - Item two. Quisque purus massa, pellentesque in viverra tempus, /// aliquet nec urna. ``` This doesn't get recognized by Docgen as a list and is translated to regular text, which is displayed as one continuous line. But when the space characters between `///` and the text get removed from he `.sol` files, the lists are recognized correctly (without breaking interpretation of other features). That's why (unless `removeTrailingSpacesInComments` is set to `false`) we run `sed -i 's_///[[:blank:]]*_///_' command on all Solidity files. - There may be other problems with the formatting of the input files that may need correction. To handle that we allow for specifying `preProcessingCommand`, which can modify the input files according to need. An example of such command: `sed -i ':a;N;$!ba;s_///\n//\n_///\n_g' ./contracts/bridge/BitcoinTx.sol`. 2. Once the files are ready, we install dependencies. It is assumed that the project in which action will be run has the `solidity-docgen` package added to the list of dev dependencies in `package.json`. 3. After project is installed we run the Docgen tool. In order for the tool to work, the Hardhat config needs to be updated (see https://github.com/OpenZeppelin/solidity-docgen#usage for more details). The workflow expects `outputDir` to be set to `generated-docs`. You may also need to specify other configs, like `pages`, `templates` or `exclude`, according to the needs of particular project. The tool will generate to `generated-docs` either one `index.md` file (if `pages` is set to `single`) or a set of files (if `pages` is set to `items` or `files`), with names reflecting their content. 4. If docs are generated to one file, it may e useful to add a Table Of Contents to that file, to ease the navigation. This will be done if `addTOC` is set to `true`. We use `markdown-toc` tool for TOC generation. The specific options that should be used during generation can be specified in the `tocOptions` input. 5. Then (if `exportAsGHArtifacts` is set to true) we export the generated file(s) as GH Actions artifacts. They will be available for download in the details of the workflow run. 6. If artifacts were exported, workflow was triggered by a PR and the `commentPR` was set to true, a comment with information where the artifacts can be found will be posted in the PR. 7. If `publish` input was set, we then proceed to the steps pushing the generated file(s) to a repository/path/branch specified in the workflow's inputs. To do that we first need to make sure we use correct config to be able to publish to the destination repository. We need to provide email and a username of the user who will be author of the commit. If repository allows only for commits from verified users, the `verifyCommits` input should be set to `true` and the user have the GPG key configured (see https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key). The GPG private key and a passphrase to the key should be stored as GH secrets in the repositories where the workflow will be used. The name of the secrets should be passed to the workflow as `git_user_signingkey` and `git_commit_gpgsign`. Apart from configuring aforementioned commiter-related params, a GH secret storing GitHub API token (personal access token) for the destination repository needs to be provided in the `githubToken` secret. If everything is configured, the workflow syncs the content of the folder storing generated docs with the content of the destination directory (using `rsync`). If `rsyncDelete` input is set to true, the files which exist in the destination repository, but don't exist in the `generated-docs` folder will be removed from the destination repo. One may want to set this option if Docgen's `pages` attribute is set to `pages` or `items` (so that when a contract or its function gets removed, we would remove the corresponding doc from the destination repo). The setting should be used with caution! --- .github/workflows/reusable-solidity-docs.yml | 248 +++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 .github/workflows/reusable-solidity-docs.yml diff --git a/.github/workflows/reusable-solidity-docs.yml b/.github/workflows/reusable-solidity-docs.yml new file mode 100644 index 0000000..de846b0 --- /dev/null +++ b/.github/workflows/reusable-solidity-docs.yml @@ -0,0 +1,248 @@ +name: Generate and push Solidity API documentation + +# To use this workflow, make sure `solidity-docgen` package is added to the +# projects's dev dependencies and is configured in Hardhat config (see +# https://github.com/OpenZeppelin/solidity-docgen#usage). +# The workflow expects `outputDir` to be set to `generated-docs`. You may +# also need to specify other configs, like `temlates` or `exclude`. +on: + workflow_call: + inputs: + projectSubfolder: + description: "A root's subfolder containing the `contracts` folder and + the Yarn / Hardhat configuration. Leave empty ('') if the project + config resides in the root. If config resides in a subfolder, provide + the subfolder name/path, without leading `.` and ending `/`. For + example, type `/v2/solidity` if the path to the `contracts` folder is + `./v2/solidity/contracts`." + type: string + required: false + default: "" + removeTrailingSpacesInComments: + description: "True if you want to remove space characters between `///` + and a comment text in the Solidity files, before generating `.md` + files. This may be needed to improve the look of the lists in the + generated files." + type: boolean + required: false + default: true + preProcessingCommand: + description: "An optional additional bash command to be executed before + transforming the Solidity files to an HTML file. The command will be + executed in the path specified by `projectSubfolder`." + type: string + required: false + addTOC: + description: "True if you want to add Table Of Contents to the generated + Markdown file. Uses `markdown-toc` tool to generate TOC. Shouldn't be + set to `true` if Docgen's `pages` attribute in `hardhat.config.ts` is + set to `items` or `files`." + type: boolean + required: false + default: true + tocOptions: + description: "Options to be passed to the `markdown-toc` tool used for + generation of Table Of Contents. See + https://github.com/jonschlinkert/markdown-toc#cli for more details." + type: string + required: false + default: "-i --maxdepth 2" + publish: + description: "True if you want to push the generated file(s) to the + destination repository." + type: boolean + required: false + default: true + verifyCommits: + description: "True if you want to sign commits updating the docs using + GPG key. You'll need to specify XXX and YYY secrets." + type: boolean + required: false + default: false + destinationRepo: + description: "A name of the repository where the generated Solidity docs + will be pushed to. Required if `publish=true`." + type: string + required: false + destinationFolder: + description: "A path in the destination repository where the generated + Solidity docs will be pushed to. For example, `./docs-api/tbtc-v2`." + type: string + required: false + default: "." + destinationBranch: + description: "A branch on the destination repository where the generated + Solidity docs will be pushed to (must exist)." + type: string + required: false + default: "main" + userEmail: + description: "The email address of a GitHub user associated with the + `githubToken` secret. Will be used to sign commits. Required if + `publish=true`." + type: string + required: false + userName: + description: "The name of the GitHub user associated with the + `githubToken` secret. Will be used to sign commits. Required if + `publish=true`." + type: string + required: false + rsyncDelete: + description: "True if you want to delete from the destination folder + the files which were not generated during current run of the workflow. + Usually there shouldn't be need to set this option to `true` if + Docgen's `pages` attribute in `hardhat.config.ts` is set to `single`. + In other cases the option should be use with caution." + type: boolean + required: false + default: false + commentPR: + description: "True if you want to add a comment with the path to the + generated files in the PR invoking the workflow. If the workflow is + not triggered by the `pull_request` event, having this input set to + `true` will not brake the execution." + type: boolean + required: false + default: false + exportAsGHArtifacts: + description: "True if you want to see the artifacts generated on various + stages of workflow execution in the GH run details." + type: boolean + required: false + default: false + secrets: + githubToken: + description: "A GitHub API token for the destination organization. + Should have `repo` scope. Required if `publish=true`." + required: false + gpgPrivateKey: + description: "A GPG private key needed when `verifyCommits` is set to + true. See + https://github.com/crazy-max/ghaction-import-gpg#prerequisites for + instructions on key generation." + required: false + gpgPassphrase: + description: "A passphrase of the GPG key. Needed when `verifyCommits` + is set to true." + required: false + +jobs: + docs-generate-html-and-publish: + runs-on: ubuntu-latest + defaults: + run: + working-directory: .${{ inputs.projectSubfolder }} + steps: + - uses: actions/checkout@v3 + + # In this step we modify the format of the comments in the Solidity + # contracts files. We do that because our original formatting is not + # processed by Docgen in the way we would like. + # To nicely display lists (like the list of requirements) we need to + # remove multiple space chars after the `///` comment. We do that by + # executing `sed 's_///[[:blank:]]*_///_'` on all contracts files, which + # substitutes `///` + 0 or more spaces/tabs with just `///`. + + - name: Prepare contract files for further processing + if: inputs.removeTrailingSpacesInComments + shell: bash + run: | + find ./contracts \ + -name "*.sol" \ + -type f \ + -exec sed -i 's_///[[:blank:]]*_///_' {} \; + + - name: Execute additional command + if: inputs.preProcessingCommand != null + shell: bash + run: ${{ inputs.preProcessingCommand }} + + - name: Export artifacts + if: inputs.exportAsGHArtifacts == true + uses: actions/upload-artifact@master + with: + name: contracts-after-preprocessing + path: .${{ inputs.projectSubfolder }}/contracts + + # We may need this step in case we execute the workflow in a project + # that has a dependency to `@summa-tx/relay-sol@2.0.2` package, which + # downloads one of its sub-dependencies via unathenticated `git://` + # protocol. That protocol is no longer supported. Thanks to this step + # `https://` is used instead of `git://`. + - name: Configure git to don't use unauthenticated protocol + shell: bash + run: git config --global url."https://".insteadOf git:// + + - name: Install dependencies + shell: bash + run: yarn install --frozen-lockfile + + # Generates `.md` file(s) based on config in the `hardhat.config.ts`. + - name: Build Markdown docs + run: yarn run hardhat docgen + + - name: Add Table of Contents + if: inputs.addTOC == true + run: | + yarn global add markdown-toc + sed -i '2s/^/\