This repository has been archived by the owner on May 15, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 5
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add integration test for full deal flow, fix retrieval bugs (#118)
# Goals Write an integration test that verifies data is actually sent into singularity and deals are proposed to boost. Test retrieval including: - Test the retrieval process from both the datasource and Filecoin. - Verify the correct functioning of the read seeker. - Test the HTTP endpoint for file retrieval. - Verify Lassie integration and CAR file retrieval. fixes #103 # Implementation - Various bug fixes already shipped and merged to singularity discovered in testing - Move most execution of devnet integration test out of github action into a Makefile. See `integration/test/README.md` for more information. This enables running tests and devnets locally with ease - Switch from SINGULARITY_TAG to SINGULARITY_REF, removing the automatically inserted colon. This allows us to run from direct SHA tags as needed. It also means the code will run is no ref is specified, as opposed to before where it would fail if no SINGULARITY_TAG was present - Fix bug with go-routine execution of singularity preparation introduced in cleanup data PR - Fix bug with blob descriptors when local file is deleted - Fix bug where local data was deleted before the deal status was published or active - Add several hook scripts to enable acceptance of boost deals (such as pricing deals to zero) - Various configuration fixes to make dealmaking in test run smoothly - Integration test that now does a MUCH more full cycle -- pushes, retrieves, waits for full storage on filecoin, then retrieves with Lassie # For discussion - I had to add a custom entrypoint boost in order to properly configure boost to advertise booster-http. We should probably upstream this. - There is a bug fix for singularity that should get in before we merge this. --------- Co-authored-by: Masih H. Derkani <[email protected]> Co-authored-by: Elijah <[email protected]>
- Loading branch information
3 people
authored
Oct 5, 2023
1 parent
d3eb162
commit f870783
Showing
17 changed files
with
363 additions
and
75 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| .PHONY: devnet/up devnet/down motionlarity/up motionlarity/down test | ||
|
|
||
| ./devnet/.up: | ||
| echo "Run Boost Devnet from pre-built images" | ||
| rm -rf ./devnet/data && docker compose -f ./devnet/docker-compose.yaml up -d | ||
| echo "Await Lotus full node startup" | ||
| docker compose -f ./devnet/docker-compose.yaml exec lotus lotus wait-api --timeout=20m | ||
| echo "Await Lotus-miner full node startup" | ||
| docker compose -f ./devnet/docker-compose.yaml exec lotus-miner lotus-miner wait-api --timeout=20m | ||
| touch ./devnet/.up | ||
|
|
||
| devnet/up: ./devnet/.up | ||
|
|
||
| ./devnet/.boostready: ./devnet/.up | ||
| ./boost-setup.sh | ||
| touch ./devnet/.boostready | ||
|
|
||
| ./motionlarity/.env.local: ./devnet/.up | ||
| echo "Set up Motion Wallet and API endpoints" | ||
| cat ./motionlarity/.env > ./motionlarity/.env.local | ||
| echo "" >> ./motionlarity/.env.local | ||
| ./integration-setup.sh ./motionlarity/.env.local | ||
|
|
||
| motionlarity/setup: ./motionlarity/.env.local ./devnet/.boostready | ||
|
|
||
| devnet/down: motionlarity/down | ||
| docker compose -f ./devnet/docker-compose.yaml down && sleep 2 && rm -rf ./devnet/data | ||
| rm ./devnet/.up || true | ||
|
|
||
| ./motionlarity/.up: ./motionlarity/.env.local ./devnet/.boostready | ||
| ./buildsingularity.sh | ||
| echo "Run Motionlarity" | ||
| docker compose -f ./motionlarity/docker-compose.yaml --env-file ./motionlarity/.env.local up -d | ||
| ./waitmotion.sh | ||
| touch ./motionlarity/.up | ||
|
|
||
| motionlarity/up: ./motionlarity/.up | ||
|
|
||
| motionlarity/down: | ||
| docker compose -f ./motionlarity/docker-compose.yaml down --rmi=all --volumes | ||
| rm ./motionlarity/.up || true | ||
| rm ./motionlarity/.env.local || true | ||
| rm ./devnet/.boostready || true | ||
|
|
||
| test: motionlarity/up | ||
| echo "Run Motion integration tests" | ||
| MOTION_INTEGRATION_TEST='true' go test . -v --count=1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Motion Devnet Integration Test Suite | ||
|
|
||
| This integration allows you to spin up a complete devnet including Lotus, Boost, Singularity, and Motion, in order to test round trip deal flow. It includes an integration test that covers all steps of a motion deal workflow, including storing to a filecoin provider and retrieving data back via trustless HTTP retrieval. | ||
|
|
||
| It's intended both as an integration test to run in CI, and a platform for developers to spin up test nets so they can try out new Motion features they are working on. | ||
|
|
||
| ### Command reference | ||
|
|
||
| All devnets are spun up with `make`. Key commands: | ||
|
|
||
| - `make devnet/up` - will spin up a boost / lotus devnet | ||
| - `make motionlarity/up` - will spin up motion and singularity pre-configured to make deals with a lotus/boost devnet. if a lotus / boost devnet is not already spun up, this will spin one up | ||
| - `make test` - this will run the integration test on top of the motion/singularity/boost/lotus devnets. It will spin up all required networks that are not already running | ||
| - `make motionalirity/down` -- this will shut down motion and singularity processes | ||
| - `make devnet/down` -- this will shut down the boost/lotus devnet. If singularity and motion processes running, it will shut them down as well | ||
|
|
||
| ### Using local singularity | ||
|
|
||
| Motion processes that are spun up with this test suite automatically use the code that is in the local repository on the current branch. | ||
|
|
||
| However, Singularity processes are by default spun up with a remote image. If you want to spin up Singularity using a local code repository as the base, you'll want to specify `SINGULARITY_LOCAL_DOCKERFILE` as the path to your local singularity repo. For example, if you store your go repositories on the traditional go src folder hierarchy, you can use: | ||
|
|
||
| `SINGULARITY_LOCAL_DOCKERFILE=../../../../data-preservation-programs/singularity make motionlarity/up` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| echo "setup boost pricing" | ||
| for i in {1..10} | ||
| do | ||
| curl -X POST -d '{"operationName":"AppStorageAskUpdateMutation","variables":{"update":{"Price":"0", "VerifiedPrice": 0}},"query":"mutation AppStorageAskUpdateMutation($update: StorageAskUpdate!) {\n storageAskUpdate(update: $update)\n}\n"}' http://localhost:8080/graphql/query && break || sleep 5 | ||
| done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| if [ -n "$SINGULARITY_LOCAL_DOCKERFILE" ] | ||
| then | ||
| echo "Building singularity from local source" | ||
| source ./motionlarity/.env | ||
| docker build -t ghcr.io/data-preservation-programs/singularity${SINGULARITY_REF} ${SINGULARITY_LOCAL_DOCKERFILE} | ||
| fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,87 @@ | ||
| #!/usr/bin/env bash | ||
| set -e | ||
|
|
||
| echo Wait for lotus is ready ... | ||
| lotus wait-api | ||
| echo Wait for lotus-miner is ready ... | ||
| lotus-miner wait-api | ||
| echo BOOST_PATH=$BOOST_PATH | ||
| export DEFAULT_WALLET=`lotus wallet default` | ||
| export FULLNODE_API_INFO=`lotus auth api-info --perm=admin | cut -f2 -d=` | ||
| export MINER_API_INFO=`lotus-miner auth api-info --perm=admin | cut -f2 -d=` | ||
|
|
||
| if [ ! -f $BOOST_PATH/.init.boost ]; then | ||
| echo Init wallets ... | ||
| export COLLAT_WALLET=`lotus wallet new bls` | ||
| export PUBMSG_WALLET=`lotus wallet new bls` | ||
| export CLIENT_WALLET=`lotus wallet new bls` | ||
| echo MINER_API_INFO=$MINER_API_INFO | ||
| echo FULLNODE_API_INFO=$FULLNODE_API_INFO | ||
| echo PUBMSG_WALLET=$PUBMSG_WALLET | ||
| echo COLLAT_WALLET=$COLLAT_WALLET | ||
|
|
||
| lotus send --from $DEFAULT_WALLET $COLLAT_WALLET 10 | ||
| lotus send --from $DEFAULT_WALLET $PUBMSG_WALLET 10 | ||
| lotus send --from $DEFAULT_WALLET $CLIENT_WALLET 10 | ||
| lotus wallet market add --from $DEFAULT_WALLET --address $CLIENT_WALLET 5 | ||
| lotus wallet market add --address $COLLAT_WALLET 5 | ||
|
|
||
| until lotus-miner actor control set --really-do-it ${PUBMSG_WALLET}; do echo Waiting for storage miner API ready ...; sleep 1; done | ||
|
|
||
| echo Init boost on first run ... | ||
|
|
||
| boostd -vv --boost-repo $BOOST_PATH init --api-sealer=$MINER_API_INFO \ | ||
| --api-sector-index=$MINER_API_INFO \ | ||
| --wallet-publish-storage-deals=$PUBMSG_WALLET \ | ||
| --wallet-deal-collateral=$COLLAT_WALLET \ | ||
| --max-staging-deals-bytes=2000000000 | ||
|
|
||
| # echo exit code: $? | ||
|
|
||
| echo Setting port in boost config... | ||
| sed 's|ip4/0.0.0.0/tcp/0|ip4/0.0.0.0/tcp/50000|g' $BOOST_PATH/config.toml > $BOOST_PATH/config.toml.tmp; cp $BOOST_PATH/config.toml.tmp $BOOST_PATH/config.toml; rm $BOOST_PATH/config.toml.tmp | ||
| sed 's|127.0.0.1|0.0.0.0|g' $BOOST_PATH/config.toml > $BOOST_PATH/config.toml.tmp; cp $BOOST_PATH/config.toml.tmp $BOOST_PATH/config.toml; rm $BOOST_PATH/config.toml.tmp | ||
|
|
||
| echo Done | ||
| touch $BOOST_PATH/.init.boost | ||
| fi | ||
|
|
||
| # TODO(anteva): fixme: hack as boostd fails to start without this dir | ||
| mkdir -p /var/lib/boost/deal-staging | ||
|
|
||
| if [ ! -f $BOOST_PATH/.register.boost ]; then | ||
| echo Temporary starting boost to get maddr... | ||
|
|
||
| boostd -vv run &> $BOOST_PATH/boostd.log & | ||
| BOOST_PID=`echo $!` | ||
| echo Got boost PID = $BOOST_PID | ||
|
|
||
| until cat $BOOST_PATH/boostd.log | grep maddr; do echo "Waiting for boost..."; sleep 1; done | ||
| echo Looks like boost started and initialized... | ||
|
|
||
| echo Registering to lotus-miner... | ||
| MADDR=`cat $BOOST_PATH/boostd.log | grep maddr | cut -f3 -d"{" | cut -f1 -d:` | ||
| echo Got maddr=${MADDR} | ||
|
|
||
| lotus-miner actor set-peer-id ${MADDR} | ||
| lotus-miner actor set-addrs /dns/boost/tcp/50000 | ||
| echo Registered | ||
|
|
||
| touch $BOOST_PATH/.register.boost | ||
| echo Try to stop boost... | ||
| kill -15 $BOOST_PID || kill -9 $BOOST_PID | ||
| rm -f $BOOST_PATH/boostd.log | ||
| echo Super. DONE! Boostd is now configured and will be started soon | ||
| fi | ||
|
|
||
| ## Override config options | ||
| echo Updating config values | ||
| sed 's|ServiceApiInfo = ""|ServiceApiInfo = "ws://localhost:8042"|g' $BOOST_PATH/config.toml > $BOOST_PATH/config.toml.tmp; cp $BOOST_PATH/config.toml.tmp $BOOST_PATH/config.toml; rm $BOOST_PATH/config.toml.tmp | ||
| sed 's|ExpectedSealDuration = "24h0m0s"|ExpectedSealDuration = "0h0m10s"|g' $BOOST_PATH/config.toml > $BOOST_PATH/config.toml.tmp; cp $BOOST_PATH/config.toml.tmp $BOOST_PATH/config.toml; rm $BOOST_PATH/config.toml.tmp | ||
|
|
||
| echo Setting HTTP addr | ||
| sed 's|HTTPRetrievalMultiaddr = ""|HTTPRetrievalMultiaddr = "/dns4/booster-http/tcp/7777/http"|g' $BOOST_PATH/config.toml > $BOOST_PATH/config.toml.tmp; cp $BOOST_PATH/config.toml.tmp $BOOST_PATH/config.toml; rm $BOOST_PATH/config.toml.tmp | ||
|
|
||
| echo Starting LID service and boost in dev mode... | ||
| trap 'kill %1' SIGINT | ||
| exec boostd-data run leveldb --addr=0.0.0.0:8042 & boostd -vv run --nosync=true |
Oops, something went wrong.