emo/thin-provisioning-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,550 Commits 2 Branches 0 Tags 3.7 MiB
7f7ba950ef
Commit Graph

1550 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ming-Hung Tsai
7f7ba950ef [thin_pool] Complete device creation 
- Set up the device details
- Cache device details of opened devices
- Update changed devices at once while committing the pool
 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
77adb08c3f [thin_pool] Replace the member pointer by references 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
0b058a2850 [thin_pool] Encapsulate metadata into the pool class 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
5810846409 [thin] Add more constructors to device_details 2020-06-17 19:09:24 +08:00
Joe Thornber
dc5bb3559b [build] remove need for zlib. 
Disappeared with the C++ version of thin_metadata_pack
 2020-06-12 19:09:19 +01:00
Joe Thornber
3e24264442 Merge branch '2020-05-27-check-data-space-map' 2020-06-12 13:42:35 +01:00
Joe Thornber
0b5afc6cb0 [thin_check] fix bugs in thin_check -m 2020-06-12 13:41:47 +01:00
Joe Thornber
0ab509578f [*_check] Remove the inane "This is not the btree you are looking for." message. 2020-06-11 13:05:03 +01:00
Joe Thornber
7a85e47a20 [rust] squash clippy (linter) warnings 2020-06-10 16:46:38 +01:00
Joe Thornber
ace8b39a8a Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 15:12:46 +01:00
Joe Thornber
f90010e22a [pack/unpack] fix bug in file_utils::file_size() 2020-06-10 15:07:40 +01:00
Joe Thornber
380632565d [thin_metadata_{pack/unpack}] Fix get_nr_blocks() 
It didn't work with block devices.
 2020-06-10 14:38:42 +01:00
Joe Thornber
324c0050bf [install-rust-tools] add dep to generate man pages 2020-06-10 10:32:36 +01:00
Joe Thornber
549c0b9499 Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-10 10:15:16 +01:00
Joe Thornber
821628b004 [pack/unpack] add missing file. 2020-06-09 14:59:13 +01:00
Joe Thornber
86704deacb [thin_check] Add support for --metadata-snap. 
Not tested yet.
 2020-06-09 14:25:19 +01:00
Joe Thornber
01a7462ece Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-09 13:39:46 +01:00
Joe Thornber
409a660082 [pack/unpack] Get the functional tests working again. 
There's some hard coded version numbers in the tests, but I'm
leaving for now since I'll rewrite in Rust to avoid too much
of a proliferation of languages.
 2020-06-09 13:03:39 +01:00
Joe Thornber
db5a71a53c [rust tools] Get the tools version from autoconf 2020-06-09 10:23:52 +01:00
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber
c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber
9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber
16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber
c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Joe Thornber
774111b34d
Merge pull request #136 from mingnus/thin-generate-metadata-wip 
[thin_generate_metadata] Introduce the new transactional metadata generate tool
 2020-05-28 10:47:04 +01:00
Ming-Hung Tsai
86139cf6bd [sm_recursive] Fix atomicity when applying BOPs 
Any BOP should be taken only once. Therefore, we should remove BOPs
that are being processed or had been processed from the uncommitted
list.
 2020-05-28 17:30:15 +08:00
Joe Thornber
4cdffafe88 [sm_recursive] Fix bug that allowed the same block to be allocated twice. 
See issue 70.  Soln from mingnus.
 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0f2918e989 [thin_generate_metadata] Implement format_metadata and create_thin 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0c6d132774 [thin_generate_metadata] Preparation for revising the features 2020-05-28 15:09:20 +08:00
Joe Thornber
128ecc27f2 [space-maps] remove count_possibly_greater_than_one 
This is only ever called with a recursive space map wrapper,
and that causes too many false positives.  False positives trigger
a copy of a metadata page, so this is a false economy.
 2020-05-28 07:38:58 +01:00
Joe Thornber
29cfdd8979 [space-maps/core] rewrite the core space map to use less memory. 2020-05-27 12:00:40 +01:00
Joe Thornber
e92ae10408 [thin_metadata_pack/unpack] Add version field to header. 2020-05-26 10:59:13 +01:00
Joe Thornber
9daa726870 [thin_metadata_pack/unpack] functional tests 2020-05-26 10:55:03 +01:00
Joe Thornber
a934a34d22 [thin_metadata_pack/unpack] Move the file magic nr before the compressed stream. 
Makes it easier to spot whether it's a pack file or not.
 2020-05-26 10:53:14 +01:00
Joe Thornber
d9e18fecf1
Merge pull request #135 from mingnus/master 
[thin_metadata_pack] Improve block scanning performance
 2020-05-26 09:53:27 +01:00
Joe Thornber
371d1bbd6e [space-maps/recursive] fix bug in flush_ops() 
Decrements weren't being passed down properly.
 2020-05-26 09:24:15 +01:00
Joe Thornber
50f8e792d3 [space-map] Make the version of inc/dec that take a count the only one. 2020-05-26 09:12:45 +01:00
Joe Thornber
cc806a0daa [space-maps/disk] add implementation for the variant of inc/dec that take a count 2020-05-26 08:57:13 +01:00
Joe Thornber
12b6157c5b [space-maps/recursive] lift a lock, and add a lock 2020-05-26 08:45:35 +01:00
Joe Thornber
0e54de95e1 [space-maps/recursive] Remove some code duplication. 2020-05-26 08:38:05 +01:00
Joe Thornber
65fcda062f [space-maps/recursive] Aggregate the BOPs 2020-05-26 08:30:52 +01:00
Joe Thornber
2ed089a83f [space-map] Add new virtual methods to inc or dec by a specific amount. 
Provide default implementation that just loops
 2020-05-26 08:29:52 +01:00
Joe Thornber
5e6eec1bb0 [recursive space map] switch to new form 'for' statements 2020-05-26 07:49:26 +01:00
Ming-Hung Tsai
c0106fc0b3 [thin_metadata_pack] Improve block scanning performance 2020-05-23 20:32:09 +08:00
Joe Thornber
0e1700fbe9 [thin_metadata_pack] First pass at pack/unpack 2020-05-22 14:11:48 +01:00
Joe Thornber
b7d20bce48 [dev-tools] Update dev tools to use non-template version of block_manager. 2020-05-01 14:11:33 +01:00
Joe Thornber
0882022120 [build] remove [DEP] compilation messages. 
It just means each file appears twice.
 2020-04-30 15:50:58 +01:00
Joe Thornber
4313469475 [all] Switch from boost::shared_ptr -> std::shared_ptr. 
Shared_ptr has moved into the standard library since these tools were
first written.
 2020-04-30 15:02:43 +01:00
Joe Thornber
e801cc607b [block_manager] Hard code block size to 4k. 
We're never going to use anything other than 4k, and by hard coding it we
avoid making block_manager a template.
 2020-04-30 14:30:01 +01:00
Joe Thornber
3e5de399a7 [functional tests] Remove dependency on the ThunderChez library. 
I've just moved the relevant code into the functional-tests dir.
 2020-04-30 12:07:42 +01:00