diff options
| author | Matt Benjamin <mbenjamin@redhat.com> | 2019-09-24 23:12:02 +0200 |
|---|---|---|
| committer | Matt Benjamin <mbenjamin@redhat.com> | 2024-07-03 20:21:46 +0200 |
| commit | 8ff3417193906bb276a8f90249cb2c19759d6507 (patch) | |
| tree | 7d49dcf4173cd92a6a11a957c827c01ed0fd84c7 /src/rgw/rgw_file_int.h | |
| parent | common/rgw: add BLAKE3 submodule (diff) | |
| download | ceph-8ff3417193906bb276a8f90249cb2c19759d6507.tar.xz ceph-8ff3417193906bb276a8f90249cb2c19759d6507.zip | |
rgw: add checksum and digest machinery
Adds new Blake3 digest format (native), a concrete type to
represent digests, and static_visitor machinery to unify varying
checksum computations.
This framework, together with new trailing checksum header support,
is used to implement S3 additional checksum verification. Parts of
the AWS content checksum API work build on a prior contribution from
imtzw <tongzhiwei_yewu@cmss.chinamobile.com>.
Thank you!
Fixes: https://tracker.ceph.com/issues/42080
Fixes: https://tracker.ceph.com/issues/63951
squashed commits:
* rgw_cksum: add trival test vectors for sha-format digests
Computed digests match those produced by sha1sum, sha256sum,
and sha512sum utilities.
* rgw_cksum: add test vectors for blake3
Tests the same input strings with digests validated by
b3sum (https://crates.io/crates/b3sum).
* rgw_ckum: switch to accel crc32c
The internal Ceph convention appears to be to omit a final
xor where ceph_crc32c is used, but it's required for compatibility
with AWS implementations.
* rgw_cksum: add XXH3 digest
* rgw_cksum: write class encoder for rgw::digest::Cksum
* rgw_cksum: also reverse crc32c (REBASEME)
Mark noticed that the crc32c output was being tested against a
byteswapped value (crc32c also needs byteswap on LE).
* rgw_cksum: add digest::Cksum serde tests
* rgw_cksum: fix main(...) linkage
(so we run our main unit and not the one in gmock
* rgw_cksum: convenience extensions for integration with RGW/S3
* introduce rgw_cksum unique_ptr factory
* rgw_cksum: mark string transform accessors const
* rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass
* rgw_cksum: hook up put-object checksum workflow
* tweaks to report on content checksum mismatch
* rgw_cksum: match SDK as well as general checksum header
* make it more efficient
* initialize RGWPutObj_Cksum::digest
* rgw_cksum: write parse_cksum_type w/const char* arg
* initialize _type correctly; doing armored wrong
* fix expected checksum header name, clean up verify
* fix output on checksum verify fail, cleanup
* introduce Cksum::to_armor(); all AWS cases pass
* oops, extra 0-byte at end of to_armor() result
* use to_armor() with decoded checksums (i.e., for all S3 presentation)
* remove unnecessary finalize() in RGWPutObj_Cksum dtor
* RGWPutObj_Cksum::Factory fixes
* fixes test_object_checksum_sha256
* choose preferred checksum algorithm header if both are present
* log verified checksums in RGWPutObj::execute at 16
* checksum not needed in policy condition
* fix checksum trailing header format
* move Blake3 to rgw_digest_blake3.h
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Diffstat (limited to 'src/rgw/rgw_file_int.h')
| -rw-r--r-- | src/rgw/rgw_file_int.h | 1 |
1 files changed, 1 insertions, 0 deletions
diff --git a/src/rgw/rgw_file_int.h b/src/rgw/rgw_file_int.h index d61ef33c8a8..0a1db645207 100644 --- a/src/rgw/rgw_file_int.h +++ b/src/rgw/rgw_file_int.h @@ -37,6 +37,7 @@ #include "rgw_aio_throttle.h" #include "rgw_compression.h" #include "rgw_perf_counters.h" +#include "rgw_cksum.h" /* XXX |