Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Integrate PeerDASKZG library #6396

Open
wants to merge 21 commits into
base: kzgpeerdas
Choose a base branch
from
Open

chore: Integrate PeerDASKZG library #6396

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
36b51e2
add script for building rust lib
kevaundray Jul 1, 2024
c3dd731
update commit hash
kevaundray Jul 1, 2024
10e4f52
example usage
kevaundray Jul 1, 2024
26f34b9
show how to use one of the methods in rust-kzg library
kevaundray Jul 1, 2024
222599d
modify test to use compile time flag
kevaundray Jul 2, 2024
644c108
temporarily add make commands to make testing a bit easier
kevaundray Jul 2, 2024
b280e45
add a kzg.nim shim file
kevaundray Jul 2, 2024
69719ff
edit tests to use kzg shim
kevaundray Jul 2, 2024
215f484
camelCase
kevaundray Jul 2, 2024
19148b6
add copyright information
kevaundray Jul 3, 2024
2b4c36e
do not put rust lib in vendor folder since nimbus-build-system uses t…
kevaundray Jul 3, 2024
6b76b2b
add rust to the CI to build
kevaundray Jul 3, 2024
b81dd39
Setup -> Set up
kevaundray Jul 3, 2024
086d100
address rest of feedback
kevaundray Jul 3, 2024
1c7a542
update copyright year
kevaundray Jul 3, 2024
1847e35
install gnu target for windows instead of default msvc
kevaundray Jul 3, 2024
7947784
update to remove java building
kevaundray Jul 3, 2024
9b99d48
chore: update commit hash
kevaundray Jul 14, 2024
d4de379
chore: fix import paths
kevaundray Jul 14, 2024
590bb89
Patch by agnish
kevaundray Jul 16, 2024
1a3ea46
Copy agnish patch
kevaundray Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions build_peerdas_lib.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the rust library, we call ./scripts/compile.sh and it will compile the rust code for the host platform and copy the static libs into the nim directory. From thereon, you can call the nim code as if it were regular nim code. This script:

  • clones the rust code
  • calls ./scripts/compile.sh
  • Copies the nim directory to the destination folder
  • deletes the rust code since we no longer need it


# Current usage: ./build_peerdas_lib.sh vendor/nimpeerdaskzg

set -e # Exit immediately if a command exits with a non-zero status.
tersec marked this conversation as resolved.
Show resolved Hide resolved

if [ "$#" -ne 1 ]; then
echo "Usage: $0 <destination_folder>"
exit 1
fi


DESTINATION_FOLDER=$1 # Destination folder to copy the built nim library with the static lib include to

COMMIT_HASH="4205937b69945f23731d90ba2970f19fa4a5e06b" # commit to checkout rust lib at
REPO_URL="https://github.com/crate-crypto/peerdas-kzg"

echo "Building peerdas-kzg with commit hash: $COMMIT_HASH and destination: $DESTINATION_FOLDER"

TEMP_DIR=$(mktemp -d)
echo "Created temporary directory: $TEMP_DIR"

echo "Cloning repository..."
git clone "$REPO_URL" "$TEMP_DIR"

cd "$TEMP_DIR"

echo "Checking out commit: $COMMIT_HASH"
git checkout "$COMMIT_HASH"

echo "Building Rust Library: Running ./scripts/compile.sh nim"
if [ -f "./scripts/compile.sh" ]; then
./scripts/compile.sh nim
else
echo "Error: ./scripts/compile.sh not found"
exit 1
fi

if [ ! -d "bindings/nim" ]; then
echo "Error: bindings/nim directory not found"
exit 1
fi

# Move back to the original directory that the script was called from
cd -

echo "Creating destination folder: $DESTINATION_FOLDER"
mkdir -p "$DESTINATION_FOLDER"

# Copy the nim code to the destination folder (includes the built static lib)
echo "Copying contents of bindings/nim/nim_code to $DESTINATION_FOLDER"
cp -a "$TEMP_DIR/bindings/nim/nim_code/." "$DESTINATION_FOLDER"

# Clean up the temporary directory
rm -rf "$TEMP_DIR"

echo "Successfully built peerdas-kzg library and copied the nimble package to $DESTINATION_FOLDER"
7 changes: 7 additions & 0 deletions tests/consensus_spec/test_fixture_kzg.nim
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import
std/json,
yaml,
kzg4844/kzg_ex,
../../../vendor/nimpeerdaskzg/nim_peerdas_kzg/nim_peerdas_kzg,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A hack right now, to reference the peerdas-kzg lib

stint,
chronicles,
stew/[byteutils, results],
Expand Down Expand Up @@ -39,7 +40,10 @@ func fromHex[N: static int](s: string): Opt[array[N, byte]] =
except ValueError:
Opt.none array[N, byte]

var ctx: nim_peerdas_kzg.KZGCtx

block:
ctx = newKZGCtx()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be modified in the future to allow one to say how many threads the library should use

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, Nimbus except for a couple specific cryptography libraries doesn't use threads -- would these threads be wholly contained within the KZG library?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep they would be wholly contained in the cryptography library, you can set the number to 1 to use a single thread -- I can also look into making it a compilation flag in the future

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fine, then -- we also have

nimbus-eth2/beacon_chain/conf.nim

Lines 264 to 267 in cac63a3

numThreads* {.
defaultValue: 0,
desc: "Number of worker threads (\"0\" = use as many threads as there are CPU cores available)"
name: "num-threads" .}: int

already to configure the number of threads to be modified by the end-user, so could potentially hook into that.

It's already used for BLS signature verification threads, though, so one would have to consider how to keep it from becoming a misleading or double-counted setting

template sourceDir: string = currentSourcePath.rsplit(DirSep, 1)[0]
doAssert Kzg.loadTrustedSetup(
sourceDir &
Expand All @@ -61,10 +65,13 @@ proc runBlobToKzgCommitmentTest(suiteName, suitePath, path: string) =
check output.kind == JNull
else:
let commitment = blobToKzgCommitment(blob.get)
# var blobObject = Blob(bytes: blob.get)
# let commitment = ctx.blobToKzgCommitment(blobObject)
check:
if commitment.isErr:
output.kind == JNull
else:
# commitment.get.bytes == fromHex[48](output.getStr).get
commitment.get == fromHex[48](output.getStr).get

proc runVerifyKzgProofTest(suiteName, suitePath, path: string) =
Expand Down