Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grandine stops all validator activity when slasher constructs slashings #12

Open
InvisibleSymbol opened this issue Jan 30, 2024 · 1 comment

Comments

@InvisibleSymbol
Copy link

InvisibleSymbol commented Jan 30, 2024

network: holesky
validators: 3.3k attached through web3signer

some of them appear to be invalid (i.e the same attester index doesnt appear in both logged attestations), but there are indeed valid detected double votes, such as validator 967119 for example.

noticed that to they have been detected after they were already slashed on the chain? beaconcha.in says the slashing for 967119 was included in slot 889285, but the below logs are at slot 889349? thats exactly 64 slots later.

anyways this caused all attached validators to be offline until restarted

[2024-01-30T00:29:37.523+00:00] [INFO] [validator::validator] started work on execution payload with id Capella(0x02d2c34f5115bade) for head 0x9fd0b5e41a7b8cdeaf6ca125f935db1a25ec8da72592f9632c865754ed64cbf3 at slot 889349
[2024-01-30T00:29:39.000+00:00] [INFO] [validator::validator] ProposeFourth tick in slot 889348
[2024-01-30T00:29:40.000+00:00] [INFO] [validator::validator] Attest tick in slot 889348
[2024-01-30T00:29:40.129+00:00] [INFO] [validator::validator] validators [1316016, 1316864, 1316129, 1315144, 1315758, 1316669, 1316325, 1317236, 1316831, 1315085, 1316020, 1315244, 1315087, 1316177, 1316882, 1317008, 1315030, 1317107, 1317423, 1317202, 1317508, 1315176, 1316002, 1317138, 1315819, 1315150, 1316353, 1315211, 1318077, 1316036, 1315179, 1315980, 1316065, 1317208, 1316040, 1315533, 1315364, 1315370, 1317989, 1315004, 1316575, 1315485, 1315497, 1317131, 1317193, 1315692, 1318138, 1316548, 1317873, 1315623, 1317514, 1318060, 1315543, 1317327, 1315396, 1317333, 1317268, 1315771, 1318058, 1316241, 1317655, 1317123, 1316027, 1317394, 1317080, 1316420, 1318074, 1316725, 1315882, 1315261, 1316391, 1315329, 1317175, 1318168, 1318001, 1316778, 1317069, 1316403, 1318084, 1315339, 1317792, 1316105, 1316008, 1315489, 1315208, 1316784, 1316702, 1315200, 1317062, 1315734, 1318160, 1315378, 1315897, 1316127, 1315262, 1315919, 1316017, 1316908, 1315305, 1316792, 1317850, 1317300, 1315937, 1315051, 1317815, 1317302, 1315357, 1316518, 1317766, 1316868, 1316641] attesting in slot 889348
[2024-01-30T00:29:41.534+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:41.536+00:00] [INFO] [eth1::download_manager] downloaded 1 Eth1 blocks from block 836233 to block 836233
[2024-01-30T00:29:43.001+00:00] [INFO] [validator::validator] AttestFourth tick in slot 889348
[2024-01-30T00:29:44.001+00:00] [INFO] [validator::validator] Aggregate tick in slot 889348
[2024-01-30T00:29:44.003+00:00] [INFO] [validator::validator] validators [1315758, 1316725, 1316105] aggregating in slot 889348
[2024-01-30T00:29:46.538+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:47.001+00:00] [INFO] [validator::validator] AggregateFourth tick in slot 889348
[2024-01-30T00:29:48.094+00:00] [INFO] [validator::validator] Propose tick in slot 889349
[2024-01-30T00:29:50.284+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889349, root: 0x03d7a4cef2a668618b54c0ba91e40918bf4ee28b626d8f454dd69c8f780df23e)
[2024-01-30T00:29:50.645+00:00] [INFO] [validator::validator] started work on execution payload with id Capella(0x02a931aac064c2fe) for head 0x03d7a4cef2a668618b54c0ba91e40918bf4ee28b626d8f454dd69c8f780df23e at slot 889350
[2024-01-30T00:29:51.001+00:00] [INFO] [validator::validator] ProposeFourth tick in slot 889349
[2024-01-30T00:29:51.540+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:29:51.544+00:00] [INFO] [eth1::download_manager] downloaded 1 Eth1 blocks from block 836234 to block 836234
[2024-01-30T00:29:52.001+00:00] [INFO] [validator::validator] Attest tick in slot 889349
[2024-01-30T00:29:52.177+00:00] [INFO] [validator::validator] validators [1315802, 1315777, 1316411, 1316063, 1317331, 1317689, 1315739, 1316584, 1315573, 1316705, 1316911, 1316821, 1315559, 1317556, 1316979, 1317493, 1315048, 1317094, 1316846, 1317602, 1316216, 1316313, 1318047, 1315740, 1317400, 1316220, 1317132, 1316441, 1317524, 1317857, 1315584, 1317951, 1317943, 1318171, 1316513, 1315591, 1317491, 1315574, 1317976, 1316333, 1316218, 1315059, 1316737, 1315231, 1316149, 1315173, 1316734, 1315912, 1316095, 1317323, 1317450, 1317337, 1316061, 1315215, 1315005, 1316157, 1316787, 1316497, 1316351, 1317097, 1316529, 1315228, 1316021, 1316929, 1317271, 1315546, 1315526, 1315913, 1316773, 1316327, 1317927, 1316223, 1317999, 1315983, 1315995, 1318146, 1315035, 1318139, 1317624, 1315071, 1316053, 1316245, 1317324, 1315587, 1315006, 1315952, 1316416, 1318153, 1317142, 1315205, 1317710, 1318274, 1317286, 1315227, 1315997, 1317283, 1317784, 1318174, 1317718, 1315188, 1317146, 1317795, 1316515, 1316532, 1315956, 1316907, 1316736, 1318180, 1316756, 1317677, 1315308, 1317950, 1315814, 1316273, 1317104, 1315924, 1315326, 1318188, 1318073, 1317073, 1316268, 1315307] attesting in slot 889349
[2024-01-30T00:29:54.162+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [946254], data: AttestationData { slot: 889249, index: 11, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x977f7b0b51b21bb5f1d440e836cf061c00130e7c57a7d200dc2d3a6798dbcb8cb7ef85d0824f1dbaeb81da64f2405e7810c9c95ab7f11108800b3d4ce6c87c991367ef584a3d9fa6ace80e873f0d13db58e88b6cf08b5a405ed0e0acb657b8c7 }, attestation_2: IndexedAttestation { attesting_indices: [985739], data: AttestationData { slot: 889249, index: 11, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xae78c6d5614c2382f8fe7563cd592cfc88754fa7bf61e29b20dc34cce4996964bcb38235d8697ea77fd1786d9a520fc11122a3b76d0924b30a3f4ed30eb354c2405546a01d0830a20231857b11789f208436851f33734a78fde94b7c58e34185 } }, reason: DoubleVote }
[2024-01-30T00:29:54.162+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [972765], data: AttestationData { slot: 889249, index: 39, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xb8aef84c3def06d5d50221cb3386620d7da8daa07cd46af3694875740df5ad2a62dbe268ef0271d45dac10a6e0f1db540c35c6be6890a5065213cd1a0e0b3c0817f509d410110d498246fba873c44c792cefad49ffc6185ad7f7b8ce9a679af4 }, attestation_2: IndexedAttestation { attesting_indices: [960499], data: AttestationData { slot: 889249, index: 39, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xae096981ddcb8e0f3159b9b097e4e6e97b3d05515294d0decfdd6a5dd13d03f2607a164bea71679da43d1813ba64c1e9113ba26f6a0146ef4d9cdaa73a68dbd7092e072a32bf24cd26caa1265c3507cf1766f28f7ca5cbaaefcac212505651ae } }, reason: DoubleVote }
[2024-01-30T00:29:54.165+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:54.165+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.405+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [967119], data: AttestationData { slot: 889249, index: 21, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa3a5347a40cdf788dde1b14fc6d79c1f3e2db7b84d5e6529990c5b1f2b89b4a24773bec1ebe346a00dd10f0753e5cf7e11ac55cfe67ded16a0d5312f9652b19fa49218037bbff0120eea1f52be52cd06ce0993bf17d7f0f80beb082c11e4167a }, attestation_2: IndexedAttestation { attesting_indices: [967119], data: AttestationData { slot: 889249, index: 21, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x9140a0243f531391a4b090f91d74e9e027958a69a373ca1055f5be9501a855a4be5ed7bc3bd97efb1dcc2cec501f58300573138e260046f230ee5f1f945e34812dc6a602789937cf3fa730855801db6a9c106c978ffe94d11a8e9900a8c3317e } }, reason: DoubleVote }
[2024-01-30T00:29:55.407+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.408+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x828b46381ba93a3d0b82cec2278feb605e3067c627a02d3c5d36a58a4a9001fe9fb1e5c021ac83a60ac97921e81e18c60bcab1afce32a48b04044e9fb246b983de0e470d0a76ea9555eb32611934368d143f60ba3be1cc2b9d3c67501df72278 }, attestation_2: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa25ac0a17263e93a6fd6744fecf6b0903253f50ea7f3f0b28a9ba8b08ba58bc9a9afca83aa5b0b19fa8bba03fde2f4d0023d4aa62777d7888ec4c39f3e23cd1dc20dadf20e0e412180178b162c2cfb01b83a62bb59bf348949b28a6fba49a32b } }, reason: DoubleVote }
[2024-01-30T00:29:55.408+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [979682], data: AttestationData { slot: 889249, index: 41, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xa5a8de5ead44cabca0d238ace4358425fdf0c32a2dd1b614262502271b8302210cb4ef0086e132aa0656916c58375d180398276bbe651f96f5005551efed66d850a2189137ac2267f96f10db39fac2e2489d1670ec66f5457eaf9352df0bc2e3 }, attestation_2: IndexedAttestation { attesting_indices: [979682], data: AttestationData { slot: 889249, index: 41, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x8057b0f5a98283cf6f292c769610c1f9ca020670c4fcc6eaa1c30d474a5a46cfba18b1ff166038ea29386915dab35bff1189670c5ebe28dd7d8b2d9a8e27bcc93ba90c59b374b239fd3b341587fa5958ab6f67d39261093011013a3308c3862e } }, reason: DoubleVote }
[2024-01-30T00:29:55.411+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.412+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:55.576+00:00] [INFO] [slasher::slasher] attester slashing constructed: ExplainedAttesterSlashing { slashing: AttesterSlashing { attestation_1: IndexedAttestation { attesting_indices: [944920], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0x828b46381ba93a3d0b82cec2278feb605e3067c627a02d3c5d36a58a4a9001fe9fb1e5c021ac83a60ac97921e81e18c60bcab1afce32a48b04044e9fb246b983de0e470d0a76ea9555eb32611934368d143f60ba3be1cc2b9d3c67501df72278 }, attestation_2: IndexedAttestation { attesting_indices: [931404], data: AttestationData { slot: 889249, index: 44, beacon_block_root: 0x694d5b71e186639dc54e0b509997744a0abbb7e56dde613139c7893602bf11d2, source: Checkpoint { epoch: 27788, root: 0x4bae73a8f746a022a42b49cd5a192da5112d87a3e1f493ebd9afef4dad221fff }, target: Checkpoint { epoch: 27789, root: 0x6566d041dc3d61f6604af21f4f06b50464cefd46037daa0843bd6356640512d1 } }, signature: 0xb6dbc20388ce0bf8a3b79ea01a76c791036592fe9cfab6da9d271757aa17bff594a64cb1370509c78efaae025a02a2730eae6a80076c284bbbe60960acdd1675302a824e8e426e116ce8ac3b05b42c7b78fa090343201eba707d2c9f363a41b3 } }, reason: DoubleVote }
[2024-01-30T00:29:55.578+00:00] [WARN] [fork_choice_control::mutator] attester slashing rejected (error: no attesters slashed, origin: Own)
[2024-01-30T00:29:56.546+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:01.547+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:02.546+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889350, root: 0x8a7287a9a1b4f6c6c291056ec71fb55394ad2184439e4dc7ab52d26db5973aa6)
[2024-01-30T00:30:06.549+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:11.549+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:12.963+00:00] [INFO] [p2p::network] [Peers: 249/256] received beacon block as gossip (slot: 889351, root: 0xe85eccd4f0a65d4505e8a5d93fa34d6875c87c2f19b72698360b89bc39f43ceb)
[2024-01-30T00:30:14.721+00:00] [INFO] [metrics::service] metrics sent to external service
[2024-01-30T00:30:16.551+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:21.552+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:26.285+00:00] [INFO] [p2p::network] [Peers: 252/256] received beacon block as gossip (slot: 889352, root: 0x7a404547bf5d17fd76f105308250b4c993f6063ebf431c3bcedfcfe527e6c022)
[2024-01-30T00:30:26.553+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:31.554+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:32.401+00:00] [WARN] [p2p::network] [Peers: 251/256] request 107043 to peer 16Uiu2HAmGmFmaKxGkmDTLjA4WAsocfb7GLcSh3FvK9BDvpWvxPSC failed: Stream Timeout
[2024-01-30T00:30:36.555+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:36.687+00:00] [INFO] [p2p::network] [Peers: 252/256] received beacon block as gossip (slot: 889353, root: 0x27b8695db5580da2b7b7d9d1c64894ac235ddf46bdcc4c7911409b159e8ec3c1)
[2024-01-30T00:30:41.557+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:46.558+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:51.560+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:30:56.562+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
[2024-01-30T00:31:01.564+00:00] [INFO] [eth1::eth1_chain] started Eth1 block download task
@sauliusgrigaitis
Copy link
Member

Thanks for the report. I'm afraid that we even forgot that we have slasher and didn't run it for more than 2 years. We will look at it at some point, but for now it's best to not activate built-in slasher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants