emo/thin-provisioning-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,881 Commits 2 Branches 0 Tags 3.7 MiB
041ed7858c
Commit Graph

1881 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber
c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber
9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber
16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber
c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Joe Thornber
774111b34d
Merge pull request #136 from mingnus/thin-generate-metadata-wip 
[thin_generate_metadata] Introduce the new transactional metadata generate tool
 2020-05-28 10:47:04 +01:00
Ming-Hung Tsai
86139cf6bd [sm_recursive] Fix atomicity when applying BOPs 
Any BOP should be taken only once. Therefore, we should remove BOPs
that are being processed or had been processed from the uncommitted
list.
 2020-05-28 17:30:15 +08:00
Joe Thornber
4cdffafe88 [sm_recursive] Fix bug that allowed the same block to be allocated twice. 
See issue 70.  Soln from mingnus.
 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0f2918e989 [thin_generate_metadata] Implement format_metadata and create_thin 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0c6d132774 [thin_generate_metadata] Preparation for revising the features 2020-05-28 15:09:20 +08:00
Joe Thornber
128ecc27f2 [space-maps] remove count_possibly_greater_than_one 
This is only ever called with a recursive space map wrapper,
and that causes too many false positives.  False positives trigger
a copy of a metadata page, so this is a false economy.
 2020-05-28 07:38:58 +01:00
Joe Thornber
29cfdd8979 [space-maps/core] rewrite the core space map to use less memory. 2020-05-27 12:00:40 +01:00
Joe Thornber
e92ae10408 [thin_metadata_pack/unpack] Add version field to header. 2020-05-26 10:59:13 +01:00
Joe Thornber
9daa726870 [thin_metadata_pack/unpack] functional tests 2020-05-26 10:55:03 +01:00
Joe Thornber
a934a34d22 [thin_metadata_pack/unpack] Move the file magic nr before the compressed stream. 
Makes it easier to spot whether it's a pack file or not.
 2020-05-26 10:53:14 +01:00
Joe Thornber
d9e18fecf1
Merge pull request #135 from mingnus/master 
[thin_metadata_pack] Improve block scanning performance
 2020-05-26 09:53:27 +01:00
Joe Thornber
371d1bbd6e [space-maps/recursive] fix bug in flush_ops() 
Decrements weren't being passed down properly.
 2020-05-26 09:24:15 +01:00
Joe Thornber
50f8e792d3 [space-map] Make the version of inc/dec that take a count the only one. 2020-05-26 09:12:45 +01:00
Joe Thornber
cc806a0daa [space-maps/disk] add implementation for the variant of inc/dec that take a count 2020-05-26 08:57:13 +01:00
Joe Thornber
12b6157c5b [space-maps/recursive] lift a lock, and add a lock 2020-05-26 08:45:35 +01:00
Joe Thornber
0e54de95e1 [space-maps/recursive] Remove some code duplication. 2020-05-26 08:38:05 +01:00
Joe Thornber
65fcda062f [space-maps/recursive] Aggregate the BOPs 2020-05-26 08:30:52 +01:00
Joe Thornber
2ed089a83f [space-map] Add new virtual methods to inc or dec by a specific amount. 
Provide default implementation that just loops
 2020-05-26 08:29:52 +01:00
Joe Thornber
5e6eec1bb0 [recursive space map] switch to new form 'for' statements 2020-05-26 07:49:26 +01:00
Ming-Hung Tsai
c0106fc0b3 [thin_metadata_pack] Improve block scanning performance 2020-05-23 20:32:09 +08:00
Joe Thornber
0e1700fbe9 [thin_metadata_pack] First pass at pack/unpack 2020-05-22 14:11:48 +01:00
Joe Thornber
b7d20bce48 [dev-tools] Update dev tools to use non-template version of block_manager. 2020-05-01 14:11:33 +01:00
Joe Thornber
0882022120 [build] remove [DEP] compilation messages. 
It just means each file appears twice.
 2020-04-30 15:50:58 +01:00
Joe Thornber
4313469475 [all] Switch from boost::shared_ptr -> std::shared_ptr. 
Shared_ptr has moved into the standard library since these tools were
first written.
 2020-04-30 15:02:43 +01:00
Joe Thornber
e801cc607b [block_manager] Hard code block size to 4k. 
We're never going to use anything other than 4k, and by hard coding it we
avoid making block_manager a template.
 2020-04-30 14:30:01 +01:00
Joe Thornber
3e5de399a7 [functional tests] Remove dependency on the ThunderChez library. 
I've just moved the relevant code into the functional-tests dir.
 2020-04-30 12:07:42 +01:00
Joe Thornber
ad79b627a4 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-04-08 12:30:26 +01:00
Joe Thornber
0098803345 [thinp_*] Make mtree_ref_counter derive from ref_counter. 
Slight performance hit, but I'd rather everything was derived
from ref_counter.

Also implement it!  Not sure why this wasn't done before.
 2020-04-08 12:24:37 +01:00
Joe Thornber
ed22bfcdda [space-map-disk] Add virtual destructor to index_store. 
Bug. Cleaning up index stores would not be correct.
 2020-04-08 12:23:28 +01:00
Joe Thornber
1481ed0f8a [ref_counter] Make ref_counter an abstract base class. 
Forces people to override correctly.

Also make no_op_ref_counter final.
 2020-04-08 12:22:28 +01:00
Joe Thornber
501299a4ee [cache_*] array ref_counter wasn't overloaded correctly. 
Major bug, explains some space map issues I've seen when
restoring cache metadata.
 2020-04-08 12:19:55 +01:00
Joe Thornber
8db3800e2c [base/application] One of the run methods wasn't overriding 2020-04-08 12:19:06 +01:00
Joe Thornber
0ccee1759e [various] Squash a couple of clang warning wrt friend declarations. 2020-04-08 12:18:24 +01:00
Joe Thornber
5e347dddbf [various] remove dead code. 
Squashes warnings from clang++
 2020-04-08 12:12:59 +01:00
Joe Thornber
ed44a1128f
Merge pull request #134 from Arusekk/issue-12 
Fix compilation with uClibc
 2020-03-25 15:00:28 +00:00
Arusekk
3ed3925370 Fix compilation with uClibc 
Closes #12
 2020-03-25 15:58:02 +01:00
Joe Thornber
47da81b6b7
Merge pull request #133 from mingnus/master 
Improve performance, and remove unused code
 2020-03-02 11:22:49 +00:00
Ming-Hung Tsai
7a1c6dc4bf [space-map-disk] improve performance of finding a free bitmap entry 
* Simply test the raw bitmap entries against zero, to avoid
  time-consumed reference-count value extraction.

* Improve the way of iterating entries inside a bitmap:
  Extract 64-bit of bitmap entries at once, and use bitwise shift
  to iterate through the entries.
 2020-02-29 17:39:17 +08:00
Ming-Hung Tsai
cb0a77e2ae [block-manager] remove unused copy-assignment operator 
block_cache::block is non-copyable, and so are the containing structures.
 2020-02-29 00:29:01 +08:00
Joe Thornber
70f11e0866
Merge pull request #132 from mingnus/master 
Fix potential file descriptor leak and compile flags
 2020-02-25 16:08:36 +00:00
Ming-Hung Tsai
57cae67ff1 [unit-tests] fix googletest compile flags 
googletest now requires c++11 compiler
 2020-02-23 15:41:20 +08:00
Ming-Hung Tsai
955e11bc28 [block-cache] fix potential file descriptor leak 
Encapsulate file descriptor into an object, to ensure that an fd will be
closed properly while exception raised, e.g., the block_cache throws
exception during the block_manager's construction.
 2020-02-23 15:41:16 +08:00
Ming-Hung Tsai
c85ea5ef76 [thin_check] factor out metadata_checker 2020-02-17 11:42:24 +00:00
Ming-Hung Tsai
d6a8c03aa2 [btree_damage_visitor] factor out non-template functions 2020-02-17 11:42:24 +00:00
Ming-Hung Tsai
e6a4ba53f0 [build] add options for gprof 2020-02-17 11:42:24 +00:00