emo/thin-provisioning-tools
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,637 Commits 2 Branches 0 Tags 3.7 MiB
3f19818c56
Commit Graph

414 Commits

Author SHA1 Message Date
Ming-Hung Tsai
3f19818c56 [thin] Adopt stateful random number generators 2020-08-05 22:14:01 +08:00
Ming-Hung Tsai
90f2d43a95 [thin_generate_damage] Add help text 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai
b278f4ffa4 [thin_check] Change the policy of --clear-needs-check-flag to prevent error recurrence 
- Disallow clearing the needs_check flag if there's any error,
  i.e., the metadata must be fully examined, and the result must
  be NO_ERROR.
- Disallow combining --clear-needs-check with -m, --super-blocks-only,
  --skip-mappings, --override-mapping-root, or --ignore-non-fatal-errors.
 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai
49efa06ffa [thin_check] Support fixing leaked metadata blocks 
- The leaked blocks are defined as allocated but not used ones,
  i.e., expected ref-count == 0 and actual ref-count <= 2.
- Blocks with expected ref-count > 0, or actual ref-count > 2,
  are treated as unrepairable leaks.
- The fixing processing won't be executed if there's any
  unrepairable leak or errors.
- The fixing process implicitly clearing the superblock
  needs_check flag if there's no error.
 2020-07-30 23:00:50 +08:00
Ming-Hung Tsai
7be6b91420 [metadata_checker] Tidying: rename variables 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai
ecb5226acc [thin] Fix indent and trailing whitespaces 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai
b693c7d6b0 [metadata_checker] Fix missing member initialization 2020-07-29 16:47:22 +08:00
Ming-Hung Tsai
e62022a200 [base] Move run_set to namespace base 2020-07-29 15:33:25 +08:00
Ming-Hung Tsai
061d966e7a [io_generator] Support mixed sequential and random io pattern 2020-07-25 17:46:59 +08:00
Ming-Hung Tsai
d1fed5f5ec [thin_generate_damage] Introduce tools to generate metadata damage 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai
e46fabde1a [thin_generate_metadata] Fix command line options 
- Remove unused options
- Fix potential option conflict
 2020-07-25 16:53:44 +08:00
Ming-Hung Tsai
8d8ee51599 [thin_generate_metadata] Support setting the needs_check flag 2020-07-25 16:53:44 +08:00
Joe Thornber
fec11289b0 [thin_check/dump] Under populated nodes are now 'non fatal errors' 
thin_dump always ignores non-fatal-errors.
 2020-07-02 16:03:23 +01:00
Joe Thornber
9adcc7911f Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-07-02 10:53:10 +01:00
Joe Thornber
a407c831df [thin_repair] Fix segfault with metadata containing loops (bz1853241) 
'seen' bit was being set too late.
 2020-07-02 10:51:10 +01:00
Ming-Hung Tsai
19e482fd2f [thin_generate_mappings] Update help text 2020-06-24 12:26:15 +08:00
Ming-Hung Tsai
8006e18173 [thin_pool] Fix the number of mapped blocks 2020-06-24 12:26:14 +08:00
Ming-Hung Tsai
084ac4b950 [io_generator] Simplify the class interface 2020-06-24 12:26:09 +08:00
Ming-Hung Tsai
8fa59553c4 [thin_generate_mappings] Introduce the mapping generation tool 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
621b466333 [all] Eliminate duplicated sector_t typedefs 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
5260a87c0b [base] Move math_utils to namespace base 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
ef8e94b22c [thin_generate_metadata] Support setting transaction id and metadata snapshots 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
524de134a4 [thin_pool] Support creating metadadata snapshot 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
e33c60c7e2 [thin_pool] Add close_thin() 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
89409927b4 [thin_generate_metadata] Support device deletion 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
a5614f7db2 [thin_pool] Complete device deletion 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
4e1ec9326b [thin_pool] Tidying 
- Remove unused headers
- Use predefined types
 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
598696bc32 [thin_pool] Implement synchronous mapping insertion and removal 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
383ae78871 [thin_pool] Determine shared blocks while looking up mappings 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
a842b4ecfd [thin_generate_metadata] Make use of thin_pool.cc 2020-06-19 18:07:55 +08:00
Ming-Hung Tsai
66b519db83 [thin_pool] Fix snapshot creation 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
2cce8a51a7 [thin_pool] Fix mapped block time 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
7f7ba950ef [thin_pool] Complete device creation 
- Set up the device details
- Cache device details of opened devices
- Update changed devices at once while committing the pool
 2020-06-19 17:22:19 +08:00
Ming-Hung Tsai
77adb08c3f [thin_pool] Replace the member pointer by references 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
0b058a2850 [thin_pool] Encapsulate metadata into the pool class 2020-06-17 19:09:24 +08:00
Ming-Hung Tsai
5810846409 [thin] Add more constructors to device_details 2020-06-17 19:09:24 +08:00
Joe Thornber
0b5afc6cb0 [thin_check] fix bugs in thin_check -m 2020-06-12 13:41:47 +01:00
Joe Thornber
86704deacb [thin_check] Add support for --metadata-snap. 
Not tested yet.
 2020-06-09 14:25:19 +01:00
Joe Thornber
01a7462ece Merge branch 'master' into 2020-05-27-check-data-space-map 2020-06-09 13:39:46 +01:00
Joe Thornber
61de3f9287 [thin_metadata_pack/unpack] Replace C++ implementation with a Rust one. 
The Rust implementation is multithreaded, performs better in general and
does custom compression of btree nodes to achieve much better compression
ratios.  unpack also checksums expanded metadata to validate it.

Format version has jumped to 3, no backwards compatibility, but I think
that's ok since we never made a release that contained the C++ version
of these tools.

Benchmarks
==========

On an 8 core, 16 hyperthread machine.

metadata 1G, full:

      Pack size    pack time     unpack time
------------------------------------------------------
C++      193M        50.3s          6.9s (no verify)
Rust      70M         1.4s          1.8s (verify)

metadata 16G, sparse:

       Pack size    pack time     unpack time
------------------------------------------------------
C++      21M          68s           1s   (no verify)
Rust      4M           8.6s         0.5s (verify)
 2020-06-09 09:15:00 +01:00
Joe Thornber
c48851e747 Merge branch 'master' of github.com:jthornber/thin-provisioning-tools 2020-06-01 11:14:24 +01:00
Joe Thornber
9f2b747ffa [thin_metadata_pack] Prefetch blocks. 
Doubles performance
 2020-06-01 11:13:51 +01:00
Joe Thornber
16a10d2554 [thin_check] Remove the metadata_checker base class. 
It's really only a single method.
 2020-05-28 14:43:03 +01:00
Joe Thornber
c94f560be8 [thin_check] Check the ref counts in the data space map. 
Hadn't realised this was being done.
 2020-05-28 14:35:51 +01:00
Ming-Hung Tsai
0f2918e989 [thin_generate_metadata] Implement format_metadata and create_thin 2020-05-28 17:30:15 +08:00
Ming-Hung Tsai
0c6d132774 [thin_generate_metadata] Preparation for revising the features 2020-05-28 15:09:20 +08:00
Joe Thornber
e92ae10408 [thin_metadata_pack/unpack] Add version field to header. 2020-05-26 10:59:13 +01:00
Joe Thornber
a934a34d22 [thin_metadata_pack/unpack] Move the file magic nr before the compressed stream. 
Makes it easier to spot whether it's a pack file or not.
 2020-05-26 10:53:14 +01:00
Ming-Hung Tsai
c0106fc0b3 [thin_metadata_pack] Improve block scanning performance 2020-05-23 20:32:09 +08:00
Joe Thornber
0e1700fbe9 [thin_metadata_pack] First pass at pack/unpack 2020-05-22 14:11:48 +01:00