Add a database bootstrap guide #9390

nirbosl · 2024-09-17T20:43:00Z

Description:

Added a new script named bootstrap.sh which is used to import the exported mirrornode content found in the mirrornode-db-export bucket
Added a new document named bootstrap.md which contains a guide with info and instructions on how to setup a fresh mirrornode DB, and how to import the exported data available in the bucket linked above.
Updated docs/database/README.md with a paragraph linking to bootstrap.md

Related issue(s):

Checklist

Documented
- In-script comments
- bootstrap.md as a how-to guide
Tested by importing a small dataset taken from an actual mainnet DB export

…cript, and linking the bootstrap guide to the main DB README.md doc Signed-off-by: Nir Ben-Or <[email protected]>

Signed-off-by: Nir Ben-Or <[email protected]>

codecov · 2024-09-17T20:49:29Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.56%. Comparing base (b4522b8) to head (e526a4f).
Report is 6 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9390      +/-   ##
============================================
+ Coverage     92.55%   92.56%   +0.01%     
- Complexity     7039     7047       +8     
============================================
  Files           912      914       +2     
  Lines         29760    29810      +50     
  Branches       3760     3767       +7     
============================================
+ Hits          27544    27595      +51     
  Misses         1445     1445              
+ Partials        771      770       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

nirbosl · 2024-09-18T16:37:31Z

I've tested the import script using a small dataset and it has completed successfully.

docs/database/bootstrap.md

hedera-mirror-importer/src/main/resources/db/scripts/bootstrap.sh

Co-authored-by: Steven Sheehy <[email protected]> Signed-off-by: Nir Ben-Or <[email protected]>

…to come Signed-off-by: Nir Ben-Or <[email protected]>

Signed-off-by: Nir Ben-Or <[email protected]>

Co-authored-by: Steven Sheehy <[email protected]> Signed-off-by: Nir Ben-Or <[email protected]>

hedera-mirror-importer/src/main/resources/db/scripts/init.sh

Signed-off-by: Nir Ben-Or <[email protected]>

…voke of the extra grant for GCP cloud sql instances Signed-off-by: Nir Ben-Or <[email protected]>

steven-sheehy · 2024-09-19T22:24:02Z

docs/database/bootstrap.md

+export PGHOST="DB_IP_ADDRESS"
+export PGPORT="DB_PORT"


Better to show common examples for these instead of replacement text.

Suggested change

export PGHOST="DB_IP_ADDRESS"

export PGPORT="DB_PORT"

export PGHOST="127.0.0.1"

export PGPORT="5432"

steven-sheehy · 2024-09-19T22:24:54Z

docs/database/bootstrap.md

+
+### 3. Run the Initialization Script
+
+Download the initialization script [`init.sh`](../../hedera-mirror-importer/src/main/resources/db/scripts/init.sh) from the repository:


I meant relative to root, not current directory.

Suggested change

Download the initialization script [`init.sh`](../../hedera-mirror-importer/src/main/resources/db/scripts/init.sh) from the repository:

Download the initialization script [`init.sh`](/hedera-mirror-importer/src/main/resources/db/scripts/init.sh) from the repository:

steven-sheehy · 2024-09-20T22:01:34Z

docs/database/bootstrap.md

+  - [3. Run the Import Script](#3-run-the-import-script)
+- [Handling Failed Imports](#handling-failed-imports)
+  - [Steps to Handle Failed Imports:](#steps-to-handle-failed-imports)
+- [Additional Notes](#additional-notes)


This ToC doesn't seem to be updated to reflect the headers. Please scan through all sections and update.

steven-sheehy · 2024-09-20T22:06:03Z

docs/database/bootstrap.md

+export GRAPHQL_PASSWORD="SET_PASSWORD"
+export GRPC_PASSWORD="SET_PASSWORD"
+export IMPORTER_PASSWORD="SET_PASSWORD"
+export OWNER_PASSWORD="SET_PASSWORD"
+export REST_PASSWORD="SET_PASSWORD"
+export REST_JAVA_PASSWORD="SET_PASSWORD"
+export ROSETTA_PASSWORD="SET_PASSWORD"
+export WEB3_PASSWORD="SET_PASSWORD"


A better experience would be to create a bootstrap.env with all the exports and ask the user to just adjust all defaults. Then inside bootstrap.sh run source ./bootstrap.env.

steven-sheehy · 2024-09-20T22:08:22Z

docs/database/bootstrap.md

+Download the initialization script [`init.sh`](../../hedera-mirror-importer/src/main/resources/db/scripts/init.sh) from the repository:
+
+```bash
+curl -O https://raw.githubusercontent.com/hashgraph/hedera-mirror-node/main/hedera-mirror-importer/src/main/resources/db/scripts/init.sh


Technically they should download the init.sh version that's specified in MIRRORNODE_VERSION

steven-sheehy · 2024-09-20T22:11:09Z

docs/database/bootstrap.md

+Import the database schema:
+
+```bash
+psql -f schema.sql


It's confusing that both schema.sql and MIRRORNODE_VERSION files are referenced in the docs before they are requested to be downloaded. Users following the steps sequentially will get stuck.

steven-sheehy · 2024-09-20T22:23:10Z

docs/database/bootstrap.md

+**Important Notes:**
+
+- The bucket is **read-only** to the public.
+- It is configured as **Requester Pays**, meaning you need a GCP account with a valid billing account attached to download the data.


It might be useful to link to the Hedera docs that show how to create such an account.

steven-sheehy · 2024-09-20T22:26:13Z

docs/database/bootstrap.md

+   **Detach from the `screen` Session:**
+
+   Press `Ctrl+A` then `D`.
+
+   - This allows the import process to continue running in the background.
+
+   **Reattach to the `screen` Session Later:**
+
+   ```bash
+   screen -r db_import
+   ```


Screen/tmux might be a bit more complex then just adding nohup or disown to the bootstrap.sh command.

steven-sheehy · 2024-09-20T22:27:12Z

docs/database/bootstrap.md

+   After the script completes, check the exit status:
+
+   ```bash
+   echo "EXIT STATUS: $?"


This only works if they are running it in the foreground. Better to have them check the final few lines of the log.

nirbosl added 2 commits September 17, 2024 15:27

Initial commit, adding the bootstrap from export guide, bootstrap.shs…

7784939

…cript, and linking the bootstrap guide to the main DB README.md doc Signed-off-by: Nir Ben-Or <[email protected]>

Improve comments

563ccab

Signed-off-by: Nir Ben-Or <[email protected]>

nirbosl force-pushed the exported-db-bootstrap branch from 03f4d6d to 563ccab Compare September 17, 2024 20:46

nirbosl marked this pull request as ready for review September 18, 2024 16:37

steven-sheehy changed the title ~~Initial commit, adding the bootstrap from export guide, bootstrap.sh script, and linking the bootstrap guide to the main DB README.md doc~~ Add a database bootstrap guide Sep 18, 2024

steven-sheehy assigned nirbosl Sep 18, 2024

steven-sheehy added documentation Type: Improvements or additions to documentation database Area: Database labels Sep 18, 2024

steven-sheehy requested a review from a team September 18, 2024 17:06

steven-sheehy requested changes Sep 18, 2024

View reviewed changes

nirbosl and others added 3 commits September 19, 2024 10:33

Update title - suggestion accepted

bba77e9

Co-authored-by: Steven Sheehy <[email protected]> Signed-off-by: Nir Ben-Or <[email protected]>

Several changes and updates following Steven's review comments; More …

26dffb1

…to come Signed-off-by: Nir Ben-Or <[email protected]>

Continuation of the review comments resolution

e3e0b9c

Signed-off-by: Nir Ben-Or <[email protected]>

nirbosl force-pushed the exported-db-bootstrap branch from 4105ac2 to e3e0b9c Compare September 19, 2024 17:22

Remove redundant "mirror node" in the title

a4528cd

Co-authored-by: Steven Sheehy <[email protected]> Signed-off-by: Nir Ben-Or <[email protected]>

xin-hedera reviewed Sep 19, 2024

View reviewed changes

hedera-mirror-importer/src/main/resources/db/scripts/init.sh Show resolved Hide resolved

Several more changes based on comments

1bc16eb

Signed-off-by: Nir Ben-Or <[email protected]>

nirbosl force-pushed the exported-db-bootstrap branch from 1e68dfe to 1bc16eb Compare September 19, 2024 19:34

Almost done, two more items left - will be in the next commit

cd6fe75

Signed-off-by: Nir Ben-Or <[email protected]>

steven-sheehy added this to the 0.115.0 milestone Sep 20, 2024

Changed the import command to a one-liner, and added a conditional re…

e526a4f

…voke of the extra grant for GCP cloud sql instances Signed-off-by: Nir Ben-Or <[email protected]>

nirbosl requested review from xin-hedera and steven-sheehy September 20, 2024 19:49

steven-sheehy requested changes Sep 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a database bootstrap guide #9390

Add a database bootstrap guide #9390

nirbosl commented Sep 17, 2024 •

edited by steven-sheehy

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading

nirbosl commented Sep 18, 2024

steven-sheehy Sep 19, 2024

steven-sheehy Sep 19, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024

steven-sheehy Sep 20, 2024


		### 3. Run the Initialization Script

		Download the initialization script [`init.sh`](../../hedera-mirror-importer/src/main/resources/db/scripts/init.sh) from the repository:

Add a database bootstrap guide #9390

Are you sure you want to change the base?

Add a database bootstrap guide #9390

Conversation

nirbosl commented Sep 17, 2024 • edited by steven-sheehy Loading

codecov bot commented Sep 17, 2024 • edited Loading

Codecov Report

nirbosl commented Sep 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nirbosl commented Sep 17, 2024 •

edited by steven-sheehy

Loading

codecov bot commented Sep 17, 2024 •

edited

Loading