[DO NOT MERGE] Visualize diff to upstream v3.1.0 #200

sveitser · 2024-08-19T08:57:20Z

This PR is there to visualize the diff between our integration branch and the last-upstream-sync branch. Currently last-upstream-sync is at tag v3.1.0.

Test the winner in e2e test

Fixes compilation on OSX. Not sure why.

llvm 17 -> 16

…ile for osp test

VID Namespace Validation

* Fix the delayed transactions * Update system_tests/espresso_e2e_test.go Co-authored-by: Mathis <[email protected]> --------- Co-authored-by: Mathis <[email protected]>

fix typo

* Update espresso-sequencer-go * Support builder-guaranteed tx * Update the error message

* Save digest artifacts into own directory per run Currently it seems that we have too many digests in the folder and I think this breaks the assembly of the multiarch docker image. With this change only digests for the same run are saved in the same directory. * Fail docker image merge on wrong number of digests * Clean digests dir

Clean the log with the log helper

* Add missing dependency * Use renamed variable, fix typo

Pin the golint version in CI

* Fix pebble panic * lint

sveitser · 2024-08-19T09:55:01Z

execution/gethexec/node.go

@@ -185,7 +185,20 @@ func CreateExecutionNode(
 		if err != nil {
 			return nil, err
 		}
-		txPublisher = sequencer
+		if config.Sequencer.Espresso {


Create EspressoSequencer in espresso mode.

sveitser · 2024-08-19T09:56:34Z

execution/gethexec/node.go

+			if err != nil {
+				return nil, err
+			}
+			switchSequencer, err := NewSwitchSequencer(sequencer, espressoSequencer, parentChainReader.Client(), seqConfigFetcher)


Enable the automated escape hatch to restore liveness during HotShot liveness failures.

sveitser · 2024-08-19T09:57:58Z

execution/gethexec/sequencer.go

+
+	// Espresso specific flags
+	Espresso                bool          `koanf:"espresso"`
+	EnableEspressoSovereign bool          `koanf:"enable-espresso-sovereign"`


Enable the sovereign sequencer mode, where nitro remains the block builder.

codecov-commenter · 2024-08-19T10:29:35Z

Codecov Report

Attention: Patch coverage is 12.09741% with 1119 lines in your changes missing coverage. Please review.

Please upload report for BASE (last-upstream-sync@7d1d84c). Learn more about missing BASE report.

Additional details and impacted files

@@                  Coverage Diff                  @@
##             last-upstream-sync     #200   +/-   ##
=====================================================
  Coverage                      ?   24.44%           
=====================================================
  Files                         ?      266           
  Lines                         ?    38773           
  Branches                      ?        0           
=====================================================
  Hits                          ?     9479           
  Misses                        ?    27666           
  Partials                      ?     1628

sveitser · 2024-08-19T10:44:53Z

arbnode/transaction_streamer.go

+					if err != nil {
+						return 0, false, nil, err
+					}
+					if oldJst.BlockMerkleJustification == nil && newJst.BlockMerkleJustification != nil {


@ImJeremyHe @nomaxg why do we need to have this extra logic to check for duplicates? We should probably add more explicit comments to the code.

Agree that there should be more comments.

That is because that batch poster should wait for the justifications from hotshot and fill the messages with them before posting them to L1. This means that actually the batch poster are modifying the existing messages.

For some validators, they receive the l2 messages without any jst first and maintain the l2 blocks. And they also fetch the messages from L1. No matter where the messages are from, they are eventually stored in the transaction streamer. So, the transaction streamer will check if the these messages from different sources are the same. If not, it will consider a reorg happening.

That's why we have to add these code here to get around the misunderstanding of reorg

sveitser · 2024-08-19T10:47:49Z

arbnode/transaction_streamer.go

@@ -1136,7 +1202,310 @@ func (s *TransactionStreamer) executeMessages(ctx context.Context, ignored struc
 	return s.config().ExecuteMessageLoopDelay
 }

+func (s *TransactionStreamer) PollSubmittedTransactionForFinality(ctx context.Context) time.Duration {


Remaining changes in this file are for the sovereign sequencer mode. They handle the submission of L2 blocks to the Espresso Sequencer for notarization.

sveitser · 2024-08-19T10:49:47Z

arbos/arbostypes/incomingmessage.go

+	BlockMerkleComm *espressoTypes.TaggedBase64
+}
+
+type EspressoBlockJustification struct {


The Justification is used to justify that the L2 blocks are created from the transactions sequenced by the Espresso sequencer, or in other words went through the HotShot consensus.

sveitser · 2024-08-19T10:58:56Z

arbos/block_processor.go

@@ -288,6 +313,22 @@ func ProduceBlockAdvanced(
 				return nil, nil, err
 			}

+			if chainConfig.ArbitrumChainParams.EnableEspresso {


This is for a feature we called "builder guaranteed transactions". It allows wallets to "tag" their transactions with an Ethereum address of a builder to only allow that builder to include the transaction in a block.

sveitser · 2024-08-19T11:00:46Z

arbos/parse_l2.go

@@ -95,6 +107,8 @@ const (
 	L2MessageKind_Heartbeat          = 6 // deprecated
 	L2MessageKind_SignedCompressedTx = 7
 	// 8 is reserved for BLS signed batch
+	L2MessageKind_EspressoSequencedTx = 10


Using the espresso sequencer as sequencer for the L2 (this is deprecated, in favor of the sovereign sequencer).

sveitser · 2024-08-19T11:01:06Z

arbos/parse_l2.go

@@ -95,6 +107,8 @@ const (
 	L2MessageKind_Heartbeat          = 6 // deprecated
 	L2MessageKind_SignedCompressedTx = 7
 	// 8 is reserved for BLS signed batch
+	L2MessageKind_EspressoSequencedTx = 10
+	L2MessageKind_EspressoSovereignTx = 11


Using the Espresso sequencer as notarization layer.

sveitser · 2024-08-19T11:40:53Z

cmd/replay/main.go

+				commitment,
+			)
+			if jst.Proof != nil {
+				espressocrypto.VerifyNamespace(chainConfig.ChainID.Uint64(), *jst.Proof, *jst.Header.PayloadCommitment, *jst.Header.NsTable, txs, *jst.VidCommon)


This call dispatches to the espresso crypto helper to verify the integrity of the Espresso namespace. Currently this checks that the L2 transactions are exactly the ones that were finalized by Espresso consensus.

sveitser · 2024-08-19T11:45:56Z

execution/gethexec/switch_sequencer.go

@@ -0,0 +1,148 @@
+package gethexec


This module allows the nitro sequencer running on Espresso to automatically bypass the Espresso sequencer in case of a liveness failure to avoid stalling Nitro.

sveitser · 2024-08-19T11:48:48Z

validator/execution_state.go

+	SendRoot      common.Hash
+	Batch         uint64
+	PosInBatch    uint64
+	HotShotHeight uint64


The block height of hotshot is used to enforce a continuous mapping between Espresso and nitro blocks (barring the Escape hatch being turned on).

sveitser · 2024-08-19T12:01:40Z

validator/validation_entry.go

@@ -22,4 +23,9 @@ type ValidationInput struct {
 	DelayedMsg    []byte
 	StartState    GoGlobalState
 	DebugChain    bool
+


Extra validation input for validation of Espresso integration functionality.

The height is used for continuity check of blocks.

The hotshot commitment is used to verify the integrity of transactions.

The hotshot liveness is used to validate correct use of the escape hatch.

sveitser · 2024-08-19T12:09:19Z

arbitrator/prover/src/host.rs

@@ -286,6 +292,15 @@ impl Hostio {
                opcode!(LocalGet, 1);
                opcode!(ReadPreImage, PreimageType::Sha2_256);
            }
+            WavmReadHotShotCommitment => {
+                opcode!(LocalGet, 0);


@nomaxg do you know what these opcode do, and why it's 0, 1 for the commitment and just 0 for the liveness?

Here defines how the WavmReadHotShotCommitment should be converted into OpCodes. LocalGet means getting something from somewhere and putting them to the value_stack.
Since the ReadHotShotCommitment opcode requires 2 arguments, so we have to LocalGet two times. (One pointer for the place where the commitment will be stored, and the other for the hotshot height)
As for the Liveness, we don't need to store the bool value into the module memory. (I don't quite understand why, but readInboxMessage stores the messages into the memory and the hotshot commitment seems something similar to the InboxMessage)

sveitser · 2024-08-19T12:44:54Z

arbitrator/prover/src/lib.rs

@@ -247,6 +279,56 @@ pub unsafe extern "C" fn arbitrator_step_until_host_io(
    ptr::null_mut()
 }

+#[no_mangle]
+#[cfg(feature = "native")]
+pub unsafe extern "C" fn arbitrator_step_until_is_hotshot_live(


@nomaxg @ImJeremyHe this function and the one below are exactly the same. Is that intentional?

Sorry. I believe this is a bug. But this function is only used in the espresso_osp_test.

Ok. We should investigate why the test passes despite this bug.

sveitser · 2024-08-19T13:23:59Z

arbitrator/prover/src/lib.rs

+            if mach.is_halted() {
+                return ptr::null_mut();
+            }
+            if mach.next_instruction_is_read_hotshot() {


@ImJeremyHe @nomaxg feels like this should be next_instruction_is_hotshot_live

* Improve help texts for flags

* Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]>

sveitser · 2024-08-28T13:14:36Z

I'm closing this PR to prevent it from triggering the already troubled CI.

ImJeremyHe and others added 30 commits February 23, 2024 19:19

Fix

5d1e690

Fix TestEspresso

c0dfa1e

Delete system_tests/espresso-e2e/1.log

26d345e

Go back to go 1.20

b4644ba

Merge pull request #84 from EspressoSystems/jh/e2e

a64afd8

Test the winner in e2e test

Merge branch 'integration' into update-espresso-go

29a8e8a

final fixes

22df1d7

add tech debt todos

bae34d0

Update wasi stub

b9648b7

llvm 17 -> 16

b0d613e

Fixes compilation on OSX. Not sure why.

Merge pull request #90 from EspressoSystems/llvm-16

cf475d1

llvm 17 -> 16

ensure that sequencer and validator namespaces are consistent

888ab78

use chain id as namespace

1e97220

delete espresso test

24aa9e6

add test helpers to e2e file

1c11309

update e2e test environment with state variable

209c50a

update common data

3d3d7c1

WIP: A simple test for error validation. Please RESET this commit later

c2b3287

switch to uncompressed bytes

85ca8ea

comment out one step proving e2e test section

d4e5e65

delete unecessary inclusion proof file, bring back validation input f…

e9d2ac9

…ile for osp test

increase context timeout

c481fc7

comment out unused staker test code

14c760b

Add time limit and update the validation_input.json

e642b89

add some comments, env var for long staker test

49c93cd

Merge pull request #86 from EspressoSystems/update-espresso-go

afed0fb

VID Namespace Validation

Fix the delayed transactions (#93)

9ddd100

* Fix the delayed transactions * Update system_tests/espresso_e2e_test.go Co-authored-by: Mathis <[email protected]> --------- Co-authored-by: Mathis <[email protected]>

fix typo

581ddbc

Merge pull request #95 from EspressoSystems/fix-typo

06f7c3d

fix typo

Skip executing the transactions guaranteed by other builders (#94)

2293be6

* Update espresso-sequencer-go * Support builder-guaranteed tx * Update the error message

sveitser and others added 8 commits August 8, 2024 17:11

Add dependabot rule to update nitro-testnode (#189)

1e0c9b1

Merge pull request #114 from EspressoSystems/jh/log

60ade09

Clean the log with the log helper

Fix build (#191)

fd0a5b4

* Add missing dependency * Use renamed variable, fix typo

dependabot: update contracts submodule (#193)

9b473c3

Pin the golint version in CI

51ff425

Merge pull request #196 from EspressoSystems/jh/fix-go-ci

4d547c9

Pin the golint version in CI

Fix pebble panic (#192)

ea5c47e

* Fix pebble panic * lint

sveitser changed the title ~~[DO NOT MERGE] Visualize diff to upstream~~ [DO NOT MERGE] Visualize diff to upstream v3.1.0 Aug 19, 2024

sveitser commented Aug 19, 2024

View reviewed changes

sveitser and others added 2 commits August 20, 2024 09:09

Update some outdated help on flags (#203)

2efed6d

* Improve help texts for flags

Update e2e tests to use Sovereign sequencer (#204)

06581bf

* Update e2e tests to use Sovereign sequencer Co-authored-by: ImJeremyHe <[email protected]>

sveitser closed this Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DO NOT MERGE] Visualize diff to upstream v3.1.0 #200

[DO NOT MERGE] Visualize diff to upstream v3.1.0 #200

sveitser commented Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

codecov-commenter commented Aug 19, 2024 •

edited

Loading

sveitser Aug 19, 2024 •

edited

Loading

ImJeremyHe Aug 20, 2024 •

edited

Loading

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

sveitser Aug 19, 2024

ImJeremyHe Aug 20, 2024 •

edited

Loading

sveitser Aug 19, 2024

ImJeremyHe Aug 20, 2024 •

edited

Loading

sveitser Aug 20, 2024

sveitser Aug 19, 2024 •

edited

Loading

sveitser commented Aug 28, 2024

[DO NOT MERGE] Visualize diff to upstream v3.1.0 #200

[DO NOT MERGE] Visualize diff to upstream v3.1.0 #200

Conversation

sveitser commented Aug 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 19, 2024 • edited Loading

Codecov Report

sveitser Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

ImJeremyHe Aug 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ImJeremyHe Aug 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ImJeremyHe Aug 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sveitser Aug 19, 2024 • edited Loading

Choose a reason for hiding this comment

sveitser commented Aug 28, 2024

codecov-commenter commented Aug 19, 2024 •

edited

Loading

sveitser Aug 19, 2024 •

edited

Loading

ImJeremyHe Aug 20, 2024 •

edited

Loading

ImJeremyHe Aug 20, 2024 •

edited

Loading

ImJeremyHe Aug 20, 2024 •

edited

Loading

sveitser Aug 19, 2024 •

edited

Loading