Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi tree column option #232

Merged
merged 54 commits into from
Feb 15, 2024
Merged
Show file tree
Hide file tree
Changes from 42 commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
a284ebc
Multitree root commit with blocking log write
MattHalpinParity May 24, 2023
f088832
fmt
MattHalpinParity May 24, 2023
5353e83
Implement get_root and get_node
MattHalpinParity May 24, 2023
05367c2
Initial work on readers
MattHalpinParity May 24, 2023
ccb0155
Working readers
MattHalpinParity May 26, 2023
27c3d06
Working iterator
MattHalpinParity May 29, 2023
a479ce1
Added TreeReader for accessing tree root and nodes
MattHalpinParity Jun 18, 2023
d4afd3e
get_tree returns RwLock reader
MattHalpinParity Jun 20, 2023
f8d825c
Track ValueTable free entries in memory
MattHalpinParity Jun 21, 2023
5a50c1d
Added ability to claim free entries from ValueTable. Used this to mak…
MattHalpinParity Jun 23, 2023
902fa02
Make ChainGenerator generate trees that share nodes from previous trees
MattHalpinParity Jul 13, 2023
a057eed
Tree commits share existing nodes
MattHalpinParity Jul 18, 2023
366cb7b
Stress test tree pruning. Tree removal (Currently just removes root).
MattHalpinParity Jul 19, 2023
4461eb4
Implemented tree removal with reference counting for shared nodes
MattHalpinParity Jul 24, 2023
73cf3ec
Empty on shutdown option. This removes all trees and waits for value …
MattHalpinParity Jul 24, 2023
ba5a913
Depth based age histograms for more accurate chain generation. Increa…
MattHalpinParity Aug 1, 2023
7365866
Prepare for using claim_contiguous_entries
MattHalpinParity Aug 3, 2023
5703cc2
Append-only mode
arkpar Aug 7, 2023
6cdf863
Check RC on dereferencing root
arkpar Aug 7, 2023
634a314
Correctly use full key or hash
MattHalpinParity Aug 11, 2023
ee11d9f
Safer entry claiming. Deal with tree removal while a commit is being …
MattHalpinParity Aug 19, 2023
06b96e2
Separate tree operations
MattHalpinParity Aug 25, 2023
b04121d
Added various checks for correct usage
MattHalpinParity Aug 25, 2023
1d73416
Reference count tables
MattHalpinParity Sep 28, 2023
5aa61af
fmt
MattHalpinParity Sep 28, 2023
8545181
Remove value table verification of ref counts
MattHalpinParity Sep 29, 2023
91d10ad
Only create and use ref count table when needed
MattHalpinParity Oct 3, 2023
5709d13
Multitree stress fix for appending to existing database
MattHalpinParity Oct 5, 2023
29e2070
On restart table data needs to be generated after all log files have …
MattHalpinParity Oct 12, 2023
29c94d3
In memory ref count cache. Verifies with table.
MattHalpinParity Oct 19, 2023
8ac418e
Only access ref count table when needed
MattHalpinParity Oct 19, 2023
bacc28f
Merge with master
MattHalpinParity Oct 23, 2023
b943f21
Rename
MattHalpinParity Oct 24, 2023
d8fb0a0
Remove commented out lines
MattHalpinParity Oct 24, 2023
551bbab
madvise
MattHalpinParity Oct 25, 2023
9e7032f
Ref count overlay reclaiming
MattHalpinParity Oct 25, 2023
a0e66e1
Chunk buffer alignment
MattHalpinParity Oct 25, 2023
001672a
Test fix
MattHalpinParity Oct 26, 2023
29db340
Windows fix
MattHalpinParity Oct 26, 2023
5ffa528
Fix
MattHalpinParity Oct 26, 2023
08ae7d1
Fix
MattHalpinParity Oct 26, 2023
3d263d7
Loom RwLock requires Sized
MattHalpinParity Oct 29, 2023
59594a2
Debug asserts
MattHalpinParity Jan 10, 2024
b1e90f0
Use INFO_COLUMN as column index
MattHalpinParity Jan 10, 2024
9662b92
Avoid changing existing log action values
MattHalpinParity Jan 10, 2024
0077b00
Typo
MattHalpinParity Jan 10, 2024
115b5c8
Fix
MattHalpinParity Jan 11, 2024
0424bdf
Remove ordered for now
MattHalpinParity Jan 23, 2024
dff334d
Remove claim_next_free
MattHalpinParity Feb 1, 2024
e3e9032
Read lock
MattHalpinParity Feb 1, 2024
b566577
Improved node data packing and unpacking
MattHalpinParity Feb 2, 2024
fad6255
Simple ref_count tests
MattHalpinParity Feb 8, 2024
963d7ef
Remove unused multi tree node compression
MattHalpinParity Feb 8, 2024
d4caa3f
Added allow_direct_node_access column option
MattHalpinParity Feb 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ Cargo.lock
*.log
test_db/
test_db_stress/
test_db_multitree_stress/

# vim
*.swp
2 changes: 2 additions & 0 deletions admin/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,10 @@ env_logger = { version = "0.10.0", default-features = false, features = ["auto-c
fdlimit = "0.2.1"
log = "0.4.8"
parity-db = { path = ".." }
parking_lot = "0.12.0"
rand = { version = "0.8.5", features = ["small_rng"] }
blake2 = "0.10.4"
siphasher = "0.3.10"

[target.'cfg(not(target_env = "msvc"))'.dependencies]
jemallocator = "0.5.0"
42 changes: 42 additions & 0 deletions admin/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
use std::path::PathBuf;

mod bench;
mod multitree_bench;

/// Command line admin client entry point.
/// Uses default column definition.
Expand Down Expand Up @@ -127,6 +128,44 @@ pub fn run() -> Result<(), String> {
let db = parity_db::Db::open_or_create(&db_options).unwrap();
bench::run_internal(args, db);
},
SubCommand::MultiTreeStress(bench) => {
let args = bench.get_args();
// avoid deleting folders by mistake.
options.path.push("test_db_multitree_stress");
if options.path.exists() && !args.append {
std::fs::remove_dir_all(options.path.as_path())
.map_err(|e| format!("Error clearing multitree stress db: {e:?}"))?;
}

let mut db_options = options.clone();

for c in &mut db_options.columns {
c.multitree = true;
if args.pruning == 0 {
c.append_only = true;
}
}

if args.compress {
for c in &mut db_options.columns {
c.compression = parity_db::CompressionType::Lz4;
}
}

if db_options.columns.len() < 2 {
let info_column: parity_db::ColumnOptions = Default::default();
db_options.columns.push(info_column);
}

let info_column = &mut db_options.columns[1];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could use multitree_bench::INFO_COLUMN rather than 1 and 2 .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

info_column.uniform = false;
info_column.multitree = false;
info_column.ref_counted = false;
info_column.preimage = false;

let db = parity_db::Db::open_or_create(&db_options).unwrap();
multitree_bench::run_internal(args, db)?;
},
}
Ok(())
}
Expand Down Expand Up @@ -187,6 +226,8 @@ pub enum SubCommand {
Check(Check),
/// Stress tests.
Stress(bench::Stress),
/// Multitree stress test.
MultiTreeStress(multitree_bench::MultiTreeStress),
}

impl SubCommand {
Expand All @@ -207,6 +248,7 @@ impl Cli {
SubCommand::Flush(flush) => &flush.shared,
SubCommand::Check(check) => &check.shared,
SubCommand::Stress(bench) => &bench.shared,
SubCommand::MultiTreeStress(bench) => &bench.shared,
}
}
}
Expand Down
Loading
Loading