| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
encode-decode comparison
Currently, ceph-dencoder lacks certain rgw types, preventing us from accurately checking the ceph corpus for encode-decode mismatches.
This pull request aims to address this issue by adding the missing types to ceph-dencoder.
To successfully incorporate these types into ceph-dencoder, we need to introduce the necessary `dump` and `generate_test_instances`
functions that was missing in some types. These functions are essential for proper encode and decode of the added types.
This PR will enhance the functionality of ceph-dencoder by including the missing types, enabling a comprehensive analysis of encode-decode consistency.
With the addition of these types, we can ensure the robustness and correctness of the ceph corpus.
This update will significantly contribute to improving the overall reliability and accuracy of ceph-dencoder.
It allows for a more comprehensive assessment of the encode-decode behavior, leading to enhanced data integrity and stability within the ceph ecosystem.
Fixes: https://tracker.ceph.com/issues/61788
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>
|
| |
|
|
|
|
|
|
| |
add implementation for full object encryption stack on compression. Compressing first and then encrypting.
Fixed: https://tracker.ceph.com/issues/19988
Signed-off-by: Dai Zhiwei <daizhiwei3@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>
|
| |
|
|
|
|
|
|
|
| |
move the encoder and decoder methods into their associated class
files to eliminate undefined references to the class vtable
https://tracker.ceph.com/issues/53596
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
* add "std::" prefix in headers
* add "using" declarations in .cc files.
so we don't rely on "using namespace std" in one or more included
headers.
Signed-off-by: Kefu Chai <kchai@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Upstreaming / rebase of #24500
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: change io_ctx pool per storage class
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: Changing free() to delete()
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: Addressing review comments
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: Fixing seg fault
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Moving CacheRequest out of librados
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: cache initialization fix
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: fix missing spawn.hpp compile errors
resolves compilation errors similar to:
```
[ 15%] Building CXX object src/tools/immutable_object_cache/CMakeFiles/ceph_immutable_object_cache_lib.dir/CacheController.cc.o
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_common.h:31,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_rados.h:17,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/librados/IoCtxImpl.h:30,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/librados/RadosClient.h:35,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/neorados/RADOSImpl.h:27,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/neorados/RADOS.cc:37:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/common/async/yield_context.h:31:10: fatal error: spawn/spawn.hpp: No such file or directory
31 | #include <spawn/spawn.hpp>
| ^~~~~~~~~~~~~~~~~
compilation terminated.
src/neorados/CMakeFiles/neorados_api_obj.dir/build.make:62: recipe for target 'src/neorados/CMakeFiles/neorados_api_obj.dir/RADOS.cc.o' failed
make[3]: *** [src/neorados/CMakeFiles/neorados_api_obj.dir/RADOS.cc.o] Error 1
```
Signed-off-by: Mark Kogan <mkogan@redhat.com>
Resolving merge conflict
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Removing all logs and unnecessary comments
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Cache Read and Write working
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
Initial Commit L1 Cache
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
post re-rebase merge, update fixes
Signed-off-by: Mark Kogan <mkogan@redhat.com>
fixup! post re-rebase merge, update fixes
rgw: continuation of rgwcache branch rebase
Signed-off-by: Mark Kogan <mkogan@redhat.com>
RGW: DataCache: post merge fixes
Signed-off-by: Mark Kogan <mkogan@redhat.com>
fixes of segmentation fault caused by oid
Signed-off-by: E. Ugur Kaynar <ukaynar@bu.edu>
rgw: fixes for segmentation faults and configuration processing
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: data cache first commit
Signed-off-by: Mania Abdi <mania.abdi287@gmail.com>
rgw: cleanup addressing PR comments
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: cleanup addressing PR comments, continuation.
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: pr cleanup addressing second review round
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: Addressing review comments, removing all D3N code from librados
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: for compilation err from removal of mydout() helper
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rge: addressing review comments
rgw: move d3n datacache into separate files
rgw: 1st part of datacache rebranding to d3n
fix forward declaration compile err (only with clang):
../src/rgw/rgw_cache.h:396:4: error: member access into incomplete type 'struct get_obj_data'
d->data_lock.lock();
^
../src/rgw/rgw_cache.h:365:8: note: forward declaration of 'get_obj_data'
struct get_obj_data;
^
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: addressing review comments, datacache rebranding to d3n cache
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: Cleaning up unused D3N cache code
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: cont. cleaning up of rgw_obj_data()
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: Removing redundant code, fix for multipart S3 objects
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: fix: incorrect content len on multipart get
in s3tests_boto3.functional.test_s3:test_multipart_copy_versioned
when d3n cache is disabled
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: fix segfault reading from cache
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: fix segfault in multisite
sync on secondary site
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: fix segfault in multisite teuthology tests, cont.
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: Adding drain to wait for all AIO reads to complete
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
rgw: fix for using read() by liabio or posix io per config
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: improve persistent data cache directory handling
on start create the persistent datacache directory if necessary
and add an option to evict it's content if already exists
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: fix possible segfault during eviction
Signed-off-by: Mark Kogan <mkogan@redhat.com>
Co-authored-by: Mania Abdi <mania.abdi287@gmail.com>
Co-authored-by: E. Ugur Kaynar <ukaynar@bu.edu>
Co-authored-by: Aishwarya Mathuria <amathuri@redhat.com>
Co-authored-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: addressing latest review comments
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: add debug logs for cache in/out flow
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: move the L2 cache functionality to separate PR
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: addressing review comments
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: address java_s3tests teuthology issues
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: do not handle compressed objects fro now
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: l2 cleanup and log fixups + post dpp
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: thread dpp thru get_obj_iterate_cb() and related
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: bypass reading versioned objects from cache
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: cleanup and fix s3tests
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: allow to enable cache only on beast
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: delete the content of the cache directory on rgw start
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: re-enable d3n cache with civetweb frontend
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: rebase post zipper 10
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: address teuthoogy valgrind leaks detected
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: mitigating valgrind leaks
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: remove rgw_d3n_l1_libaio_read option
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: wip segfault fix
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: mitigate libaio SIGEV_THREAD cb race
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: change libaio signaling mechanism
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: wip cont. libaio cb thread race
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: wip libaio cb thread race
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: libaio cleanups and edge case handling fixes
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: narrow the libaio locking scope
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: add libaio req ordering mechanism
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: fix lock regression
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: addressing reviwe comments and cleasnup
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: libaio locks cleanup
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: refactor libaio abstraction to share the ioc implementation
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: addressing latest review comments and cleanup
Signed-off-by: Mark Kogan <mkogan@redhat.com>
rgw: d3n: address review comments, cont.
Signed-off-by: Mark Kogan <mkogan@redhat.com>
Co-authored-by: Mania Abdi <mania.abdi287@gmail.com>
Co-authored-by: E. Ugur Kaynar <ukaynar@bu.edu>
Co-authored-by: Aishwarya Mathuria <amathuri@redhat.com>
Co-authored-by: Ali Maredia <amaredia@redhat.com>
Co-authored-by: Feng Hualong <hualong.feng@intel.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
as the left-hand operator is promoted to off_t which is a signed
integer, while rgw_max_chunk_size will be an unsigned after the
yaml-to-cxx migration. so let's cast it to `off_t` before comparing
them.
the same applies to rgw_copy_obj_progress_every_bytes.
Signed-off-by: Kefu Chai <kchai@redhat.com>
|
| |
|
|
| |
Signed-off-by: Casey Bodley <cbodley@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The motivation is that `claim(list&)` seems to actually be a pre-C++11
counterpart of the already available `operator=(list&&)`.
This commit deprecates the `claim()` method but doesn't drop it yet.
All occurrences of `buffer::list::claim(list&)` are switched to
* `list::operator=(list&&)` or
* reworked to use `list::list(list&&)` instead.
Changes are applied to: rgw, osdc, osd, os/memstore, os/filestore,
os/bluestore, os, msg, mgr, messages, mds, librbd, librados, crimson,
common, cls, mon.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Signed-off-by: WangPengfei <wpf_1253@qq.com>
modified: src/common/legacy_config_opts.h
modified: src/common/options.cc
modified: src/compressor/Compressor.h
modified: src/compressor/QatAccel.cc
modified: src/compressor/QatAccel.h
modified: src/compressor/brotli/BrotliCompressor.cc
modified: src/compressor/brotli/BrotliCompressor.h
modified: src/compressor/lz4/LZ4Compressor.h
modified: src/compressor/snappy/SnappyCompressor.h
modified: src/compressor/zlib/ZlibCompressor.cc
modified: src/compressor/zlib/ZlibCompressor.h
modified: src/compressor/zstd/ZstdCompressor.h
modified: src/os/bluestore/BlueStore.cc
modified: src/os/bluestore/bluestore_types.cc
modified: src/os/bluestore/bluestore_types.h
modified: src/rgw/rgw_compression.cc
modified: src/rgw/rgw_compression.h
modified: src/rgw/rgw_compression_types.h
modified: src/rgw/rgw_json_enc.cc
modified: src/rgw/rgw_op.cc
modified: src/rgw/rgw_rados.cc
modified: src/test/compressor/compressor_example.h
modified: src/test/compressor/test_compression.cc
modified: src/test/rgw/test_rgw_compression.cc
|
| |
|
|
| |
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
|
| |
|
|
| |
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
|
| |\
| |
| |
| |
| | |
rgw: metadata refactoring
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
|
| | |
| |
| |
| | |
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
|
| |/
|
|
|
|
| |
The zlib compression takes effect in RGW。
When getting objects failed because of decompress-failed。“ceph-client.rgw” log printed “Compression failed with exit code......”，it should be “deCompression failed with exit code......”。
Signed-off-by: Han Fengzhe <hanfengzhe@hisilicon.com>
|
| |
|
|
|
|
|
|
|
|
| |
Most, although not all, source code files have a comment on the first
two lines describing the C++ formatting conventions used. This commit
adds that directive to the files lacking it. Also, vertical whitespace
was made more consistent. Note: this only touches files specifically
for RGW.
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
|
| |
|
|
| |
Signed-off-by: Casey Bodley <cbodley@redhat.com>
|
| |
|
|
|
|
|
|
|
|
| |
Adjust the interfaces to provide the ability for the read callback
to pause the reads. While doing that, define a new class interface
for this instead of RGWGetDataCB. This had a butterfly effect that
required modifications to the obj read filters, but the end result
is a bit cleaner.
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
|
| |
|
|
|
|
|
|
| |
This is an omission in #20226
Fixes: http://tracker.ceph.com/issues/23146
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>
|
| |
|
|
|
|
| |
Fixes: http://tracker.ceph.com/issues/22852
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>
|
| |
|
|
| |
Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
compression enable
large zero object has a large compression rate,
even 4M compressed data can decompress several GB data.
Handle so much data in single process lead strange issue.
Fixes: http://tracker.ceph.com/issues/20098
Signed-off-by: fang yuxiang fang.yuxiang@eisoo.com
|
| |
|
|
|
|
|
|
| |
compression,
followup of http://tracker.ceph.com/issues/20231
Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
compression was enabled
look at the prototype:
RGWGetObj_Decompress::handle_data(bufferlist& bl, off_t bl_ofs, off_t bl_len)
we should trim the bl using bl_ofs and bl_len.
Fixes: http://tracker.ceph.com/issues/20100
Signed-off-by: fang yuxiang <fang.yuxiang@eisoo.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduced AES_256_CTR block encryptor.
Added filters for encryptors to RGWPostObj and RGWPutObj.
Exposed cur_stripe from RGWObjManifest::obj_iterator
Moved RGWPutObjProcessor_Multipart from rgw_op.cc to rgw_rados.h.
Implemented actual AES256_CTR and AES256_ECB modes.
Refactored RGWGetObj_BlockDecrypt to process multipart manifest differently.
Initial work of server-side-encryption customer key (Amazon SSE-C).
Initial work of "auto encryption".
Add prepare_encryption method to RGWInitMultipart operation.
Changed naming of client encryption mode to SSE-C-AES256.
Eliminated AWS encryption related headers from being stored with object's metadata.
Added set_attr utility functions.
split RGW_ATTR_CRYPT_KEY => RGW_ATTR_CRYPT_KEYSEL + RGW_ATTR_CRYPT_KEYID
Added interfaces create_random_key_selector and get_actual_key_from_kms
Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
|
| |
|
|
|
|
|
| |
This drags in multiple related changes that are needed in order to
support that.
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
|
| |
|
|
| |
Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>
|
| |
|
|
| |
Signed-off-by: Casey Bodley <cbodley@redhat.com>
|
| |
|
|
|
|
|
| |
if rgw_compression_type is set to "random", we need to use the same
plugin for each block of data
Signed-off-by: Casey Bodley <cbodley@redhat.com>
|
| |
|
|
| |
Signed-off-by: Adam Kupczyk <akupczyk@mirantis.com>
|
|
|
Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>
|
