From 01457e0aa7574b8929584e999075a161a36af4ae Mon Sep 17 00:00:00 2001 From: Ognjen Maric Date: Tue, 24 Sep 2024 10:40:27 +0200 Subject: [PATCH 1/3] A note and another strategy for ID-based deduplication --- .../smart-contracts/best-practices/idempotency.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx index bd449af9d8..9884349a5c 100644 --- a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx +++ b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx @@ -56,6 +56,8 @@ Moreover, the ICP/ICRC ledgers use the `created_at_time` parameter to limit the But even with this improvement used in the ledgers, the time window approach implicitly assumes that the client will be able to get a definite answer to their call within the time window. For example, after the 24 hours expire, the user cannot easily tell if their ledger transfer happened; their only option is to analyze the ledger blocks, which is somewhat tedious, and has to be done carefully to avoid asynchrony issues; see the section on [queryable call results](#queryable-call-results). +Furthermore, just using a time window for deduplication does not guarantee that the memory will stay bounded. In principle, arbitrarily many updates could happen within the time window, although this is bounded in practice by the scaling limits of the ICP. The ICP/ICRC ledgers thus also define a maximum capacity: the number of deduplicated transactions (i.e., deduplication IDs) that they will store in their deduplication store. Further transactions are simply rejected after the capacity is reached, until the window passes and old transactions can be removed from the deduplication store. Yet another extension of the approach is to make the time window simply the guaranteed lower bound, and just store deduplication IDs until the capacity is reached. This way, the clients obtain a hard deduplication guarantee for the time window, and a best-effort attempt to deduplicate transactions even past the window. + An alternative is to do away with the time window, and store the deduplication data forever. This requires multiple canisters to prevent exhausting the canister memory, similar to how the ICP/ICRC ledgers store the transaction data in the archive canister. This shifts the tedious part of querying the deduplication data (e.g., ledger blocks) from the user to the canister. Summarizing, the advantages of this approach are: From 560458252d670fda44330e48cae987b0f1ac6b68 Mon Sep 17 00:00:00 2001 From: oggy-dfin <89794951+oggy-dfin@users.noreply.github.com> Date: Wed, 25 Sep 2024 09:16:36 +0200 Subject: [PATCH 2/3] Update docs/developer-docs/smart-contracts/best-practices/idempotency.mdx Co-authored-by: Roel Storms --- .../smart-contracts/best-practices/idempotency.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx index 9884349a5c..d9daad0d0f 100644 --- a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx +++ b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx @@ -56,7 +56,7 @@ Moreover, the ICP/ICRC ledgers use the `created_at_time` parameter to limit the But even with this improvement used in the ledgers, the time window approach implicitly assumes that the client will be able to get a definite answer to their call within the time window. For example, after the 24 hours expire, the user cannot easily tell if their ledger transfer happened; their only option is to analyze the ledger blocks, which is somewhat tedious, and has to be done carefully to avoid asynchrony issues; see the section on [queryable call results](#queryable-call-results). -Furthermore, just using a time window for deduplication does not guarantee that the memory will stay bounded. In principle, arbitrarily many updates could happen within the time window, although this is bounded in practice by the scaling limits of the ICP. The ICP/ICRC ledgers thus also define a maximum capacity: the number of deduplicated transactions (i.e., deduplication IDs) that they will store in their deduplication store. Further transactions are simply rejected after the capacity is reached, until the window passes and old transactions can be removed from the deduplication store. Yet another extension of the approach is to make the time window simply the guaranteed lower bound, and just store deduplication IDs until the capacity is reached. This way, the clients obtain a hard deduplication guarantee for the time window, and a best-effort attempt to deduplicate transactions even past the window. +Relying solely on a time window for deduplication does not guarantee bounded memory usage. In theory, an unlimited number of updates could occur within the time window, though in practice, this is constrained by the scaling limits of the ICP. The ICP/ICRC ledgers thus also define a maximum capacity: a limit on the number of deduplicated transactions (i.e., deduplication IDs) that can be stored in their deduplication store. Once this capacity is reached, further transactions are rejected until older transactions expire from the deduplication store at the end of the time window. Yet another extension of the approach is to make the time window simply the guaranteed lower bound, and just store deduplication IDs until the capacity is reached. This way, the clients obtain a hard deduplication guarantee for the time window, and a best-effort attempt to deduplicate transactions even past the window. An alternative is to do away with the time window, and store the deduplication data forever. This requires multiple canisters to prevent exhausting the canister memory, similar to how the ICP/ICRC ledgers store the transaction data in the archive canister. This shifts the tedious part of querying the deduplication data (e.g., ledger blocks) from the user to the canister. From b12d0ea69974e9d0180d2be028d327627419e6bb Mon Sep 17 00:00:00 2001 From: Ognjen Maric Date: Wed, 25 Sep 2024 09:20:28 +0200 Subject: [PATCH 3/3] Robin's suggestion --- .../smart-contracts/best-practices/idempotency.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx index d9daad0d0f..7d81422115 100644 --- a/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx +++ b/docs/developer-docs/smart-contracts/best-practices/idempotency.mdx @@ -56,7 +56,7 @@ Moreover, the ICP/ICRC ledgers use the `created_at_time` parameter to limit the But even with this improvement used in the ledgers, the time window approach implicitly assumes that the client will be able to get a definite answer to their call within the time window. For example, after the 24 hours expire, the user cannot easily tell if their ledger transfer happened; their only option is to analyze the ledger blocks, which is somewhat tedious, and has to be done carefully to avoid asynchrony issues; see the section on [queryable call results](#queryable-call-results). -Relying solely on a time window for deduplication does not guarantee bounded memory usage. In theory, an unlimited number of updates could occur within the time window, though in practice, this is constrained by the scaling limits of the ICP. The ICP/ICRC ledgers thus also define a maximum capacity: a limit on the number of deduplicated transactions (i.e., deduplication IDs) that can be stored in their deduplication store. Once this capacity is reached, further transactions are rejected until older transactions expire from the deduplication store at the end of the time window. Yet another extension of the approach is to make the time window simply the guaranteed lower bound, and just store deduplication IDs until the capacity is reached. This way, the clients obtain a hard deduplication guarantee for the time window, and a best-effort attempt to deduplicate transactions even past the window. +Relying solely on a time window for deduplication does not guarantee bounded memory usage. In theory, an unlimited number of updates could occur within the time window, though in practice, this is constrained by the scaling limits of the ICP. The ICP/ICRC ledgers thus also define a maximum capacity: a limit on the number of deduplicated transactions (i.e., deduplication IDs) that can be stored in their deduplication store. Once this capacity is reached, further transactions are rejected until older transactions expire from the deduplication store at the end of the time window. Yet another extension of the approach is to guaranteed deduplication for the stated time window as above, but keep storing deduplication IDs even beyond that window, as long as the capacity is not reached. This way, the clients obtain a hard deduplication guarantee for the time window, and a best-effort attempt to deduplicate transactions even past the window. An alternative is to do away with the time window, and store the deduplication data forever. This requires multiple canisters to prevent exhausting the canister memory, similar to how the ICP/ICRC ledgers store the transaction data in the archive canister. This shifts the tedious part of querying the deduplication data (e.g., ledger blocks) from the user to the canister.