diff options
558 files changed, 21143 insertions, 6224 deletions
diff --git a/.githubmap b/.githubmap index 68c711aa587..5265fa59bed 100644 --- a/.githubmap +++ b/.githubmap @@ -188,3 +188,4 @@ robbat2 Robin H. Johnson <robbat2@orbis-terrarum.net> leonid-s-usov Leonid Usov <leonid.usov@ibm.com> ffilz Frank S. Filz <ffilzlnx@mindspring.com> Jayaprakash-ibm Jaya Prakash Madaka <jayaprakash@ibm.com> +spuiuk Sachin Prabhu <sp@spui.uk> diff --git a/PendingReleaseNotes b/PendingReleaseNotes index 146cab64d6f..b4824a65584 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -1,5 +1,16 @@ >=20.0.0 +* RGW: The User Account feature introduced in Squid provides first-class support for + IAM APIs and policy. Our preliminary STS support was instead based on tenants, and + exposed some IAM APIs to admins only. This tenant-level IAM functionality is now + deprecated in favor of accounts. While we'll continue to support the tenant feature + itself for namespace isolation, the following features will be removed no sooner + than the V release: + * tenant-level IAM APIs like CreateRole, PutRolePolicy and PutUserPolicy, + * use of tenant names instead of accounts in IAM policy documents, + * interpretation of IAM policy without cross-account policy evaluation, + * S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'` + * RBD: All Python APIs that produce timestamps now return "aware" `datetime` objects instead of "naive" ones (i.e. those including time zone information instead of those not including it). All timestamps remain to be in UTC but @@ -34,6 +45,8 @@ (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. +* RADOS: Added convenience function `librados::AioCompletion::cancel()` with + the same behavior as `librados::IoCtx::aio_cancel()`. * mgr/restful, mgr/zabbix: both modules, already deprecated since 2020, have been finally removed. They have not been actively maintenance in the last years, @@ -48,6 +61,10 @@ CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. +* The HeadBucket API now reports the `X-RGW-Bytes-Used` and `X-RGW-Object-Count` + headers only when the `read-stats` querystring is explicitly included in the + API request. + >=19.2.1 * CephFS: Command `fs subvolume create` now allows tagging subvolumes through option @@ -57,6 +74,9 @@ `ceph fs subvolume earmark set`, `ceph fs subvolume earmark get` and `ceph fs subvolume earmark rm` have been added to set, get and remove earmark from a given subvolume. +* RADOS: A performance botteneck in the balancer mgr module has been fixed. + Related Tracker: https://tracker.ceph.com/issues/68657 + >=19.0.0 * cephx: key rotation is now possible using `ceph auth rotate`. Previously, @@ -324,6 +344,8 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config * NFS: The export create/apply of CephFS based exports will now have a additional parameter `cmount_path` under the FSAL block, which specifies the path within the CephFS to mount this export on. If this and the other `EXPORT { FSAL {} }` options are the same between multiple exports, those exports will share a single CephFS client. If not specified, the default is `/`. +* CephFS: MDS emits a warning with estimated replay completion time when replay + runs for more than 30 seconds. >=18.0.0 diff --git a/README.md b/README.md index 56257697e9a..f8fcf35e8b7 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ To build Ceph, follow this procedure: ninja -j3 - > [IMPORTANT] + > [!IMPORTANT] > > [Ninja](https://ninja-build.org/) is the build system used by the Ceph > project to build test builds. The number of jobs used by `ninja` is @@ -126,6 +126,9 @@ To build Ceph, follow this procedure: 5. Install the vstart cluster: ninja install + + + ### CMake Options @@ -177,6 +180,36 @@ The diagnostic colors will be visible when the following command is run: Other available values for `DIAGNOSTICS_COLOR` are `auto` (default) and `never`. +## Tips and Tricks + + * Use "debug builds" only when needed. Debugging builds are helpful for + development, but they can slow down performance. Use + `-DCMAKE_BUILD_TYPE=Release` when debugging isn't necessary. + * Enable Selective Daemons when testing specific components. Don't start + unnecessary daemons. + * Preserve Existing Data skip cluster reinitialization between tests by + using the `-n` flag. + * To manage a vstart cluster, stop daemons using `./stop.sh` and start them + with `./vstart.sh --daemon osd.${ID} [--nodaemonize]`. + * Restart the sockets by stopping and restarting the daemons associated with + them. This ensures that there are no stale sockets in the cluster. + * To track RocksDB performance, set `export ROCKSDB_PERF=true` and start + the cluster by using the command `./vstart.sh -n -d -x --bluestore`. + * Build with `vstart-base` using debug flags in cmake, compile, and deploy + via `./vstart.sh -d -n --bluestore`. + * To containerize, generate configurations with `vstart.sh`, and deploy with + Docker, mapping directories and configuring the network. + * Manage containers using `docker run`, `stop`, and `rm`. For detailed + setups, consult the Ceph-Container repository. + +## Troubleshooting + + * Cluster Fails to Start: Look for errors in the logs under the `out/` + directory. + * OSD Crashes: Check the OSD logs for errors. + * Cluster in a `Health Error` State: Run the `ceph status` command to + identify the issue. + * RocksDB Errors: Look for RocksDB-related errors in the OSD logs. ## Building a source tarball diff --git a/ceph.spec.in b/ceph.spec.in index ece1ebf2ec8..1e4c1f0850c 100644 --- a/ceph.spec.in +++ b/ceph.spec.in @@ -1186,18 +1186,41 @@ Obsoletes: libcephfs1 < %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-libs < %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-libcephfs < %{_epoch_prefix}%{version}-%{release} %endif +Recommends: libcephfs-proxy2 = %{_epoch_prefix}%{version}-%{release} +Requires: libcephfs-daemon %description -n libcephfs2 Ceph is a distributed network file system designed to provide excellent performance, reliability, and scalability. This is a shared library allowing applications to access a Ceph distributed file system via a POSIX-like interface. +%package -n libcephfs-proxy2 +Summary: Proxy for libcephfs +%if 0%{?suse_version} +Group: System/Libraries +%endif +Recommends: libcephfs-daemon = %{_epoch_prefix}%{version}-%{release} +%description -n libcephfs-proxy2 +This package contains the libcephfs_proxy.so library that allow applications +to share cephfs mounts to reduce resource consumption. + +%package -n libcephfs-daemon +Summary: Deamon for the libcephfs proxy +%if 0%{?suse_version} +Group: System/Filesystems +%endif +Requires: libcephfs2 = %{_epoch_prefix}%{version}-%{release} +%description -n libcephfs-daemon +This package contains the libcephfsd daemon that allows applications to share +cephfs mounts to reduce resource consumption. + %package -n libcephfs-devel Summary: Ceph distributed file system headers %if 0%{?suse_version} Group: Development/Libraries/C and C++ %endif Requires: libcephfs2 = %{_epoch_prefix}%{version}-%{release} +Requires: libcephfs-proxy2 = %{_epoch_prefix}%{version}-%{release} Requires: librados-devel = %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-devel < %{_epoch_prefix}%{version}-%{release} Provides: libcephfs2-devel = %{_epoch_prefix}%{version}-%{release} @@ -2524,6 +2547,16 @@ fi %postun -n libcephfs2 -p /sbin/ldconfig +%files -n libcephfs-proxy2 +%{_libdir}/libcephfs_proxy.so.* + +%post -n libcephfs-proxy2 -p /sbin/ldconfig + +%postun -n libcephfs-proxy2 -p /sbin/ldconfig + +%files -n libcephfs-daemon +%{_sbindir}/libcephfsd + %files -n libcephfs-devel %dir %{_includedir}/cephfs %{_includedir}/cephfs/libcephfs.h @@ -2532,6 +2565,8 @@ fi %dir %{_includedir}/cephfs/metrics %{_includedir}/cephfs/metrics/Types.h %{_libdir}/libcephfs.so +%{_libdir}/libcephfs_proxy.so +%{_libdir}/pkgconfig/cephfs.pc %files -n python%{python3_pkgversion}-cephfs %{python3_sitearch}/cephfs.cpython*.so diff --git a/debian/.gitignore b/debian/.gitignore index 32ca866d753..1d6ef3a34b5 100644 --- a/debian/.gitignore +++ b/debian/.gitignore @@ -38,4 +38,6 @@ /python-cephfs /libcephfs-java /libcephfs-jni +/libcephfs-proxy0-dbg +/libcephfs-proxy0 /tmp diff --git a/debian/control b/debian/control index d31a82bbc75..a7d2dbb4c3a 100644 --- a/debian/control +++ b/debian/control @@ -891,6 +891,7 @@ Conflicts: libceph, Replaces: libceph, libceph1, libcephfs, +Recommends: libcephfs-proxy2 (= ${binary:Version}) Architecture: linux-any Section: libs Depends: ${misc:Depends}, @@ -919,10 +920,61 @@ Description: debugging symbols for libcephfs2 . This package contains debugging symbols for libcephfs2. +Package: libcephfs-proxy2 +Architecture: linux-any +Section: libs +Depends: ${misc:Depends}, + ${shlibs:Depends}, +Recommends: libcephfs-daemon (= ${binary:Version}) +Description: Libcephfs proxy library + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + +Package: libcephfs-proxy2-dbg +Architecture: linux-any +Section: debug +Priority: extra +Depends: libcephfs-proxy2 (= ${binary:Version}), + ${misc:Depends}, +Description: debugging symbols for libcephfs-proxy2 + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + . + This package contains debugging symbols for libcephfs-proxy2. + +Package: libcephfs-daemon +Architecture: linux-any +Depends: libcephfs2 (= ${binary:Version}), + ${misc:Depends}, +Description: Libcephfs proxy daemon + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + +Package: libcephfs-daemon-dbg +Architecture: linux-any +Section: debug +Priority: extra +Depends: libcephfs-daemon (= ${binary:Version}), + ${misc:Depends}, +Description: debugging symbols for libcephfs-daemon + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + . + This package contains debugging symbols for libcephfs-proxy2. + Package: libcephfs-dev Architecture: linux-any Section: libdevel Depends: libcephfs2 (= ${binary:Version}), + libcephfs-proxy2 (= ${binary:Version}), ${misc:Depends}, Conflicts: libceph-dev, libceph1-dev, @@ -944,10 +996,10 @@ Package: librgw2 Architecture: linux-any Section: libs Depends: librados2 (= ${binary:Version}), + liblua5.3-0, ${misc:Depends}, ${shlibs:Depends}, - liblua5.3-dev, - luarocks, +Suggests: luarocks, Description: RADOS Gateway client library RADOS is a distributed object store used by the Ceph distributed storage system. This package provides a REST gateway to the diff --git a/debian/libcephfs-daemon.install b/debian/libcephfs-daemon.install new file mode 100644 index 00000000000..454de46d2d7 --- /dev/null +++ b/debian/libcephfs-daemon.install @@ -0,0 +1 @@ +usr/sbin/libcephfsd diff --git a/debian/libcephfs-dev.install b/debian/libcephfs-dev.install index cf22dce62d4..47431dd5526 100644 --- a/debian/libcephfs-dev.install +++ b/debian/libcephfs-dev.install @@ -3,3 +3,5 @@ usr/include/cephfs/libcephfs.h usr/include/cephfs/types.h usr/include/cephfs/metrics/Types.h usr/lib/libcephfs.so +usr/lib/libcephfs_proxy.so +usr/lib/pkgconfig/cephfs.pc diff --git a/debian/libcephfs-proxy2.install b/debian/libcephfs-proxy2.install new file mode 100644 index 00000000000..fc363125bc2 --- /dev/null +++ b/debian/libcephfs-proxy2.install @@ -0,0 +1 @@ +usr/lib/libcephfs_proxy.so.* diff --git a/debian/rules b/debian/rules index 3fbed3f3a2e..6c0ab5e12c6 100755 --- a/debian/rules +++ b/debian/rules @@ -121,6 +121,8 @@ override_dh_strip: dh_strip -plibradosstriper1 --dbg-package=libradosstriper1-dbg dh_strip -plibrbd1 --dbg-package=librbd1-dbg dh_strip -plibcephfs2 --dbg-package=libcephfs2-dbg + dh_strip -plibcephfs-proxy2 --dbg-package=libcephfs-proxy2-dbg + dh_strip -plibcephfs-daemon --dbg-package=libcephfs-daemon-dbg dh_strip -plibrgw2 --dbg-package=librgw2-dbg dh_strip -pradosgw --dbg-package=radosgw-dbg dh_strip -pceph-test --dbg-package=ceph-test-dbg diff --git a/doc/_ext/ceph_commands.py b/doc/_ext/ceph_commands.py index 0697c71f0e1..d96eab08853 100644 --- a/doc/_ext/ceph_commands.py +++ b/doc/_ext/ceph_commands.py @@ -94,7 +94,7 @@ class CmdParam(object): self.goodchars = goodchars self.positional = positional != 'false' - assert who == None + assert who is None def help(self): advanced = [] diff --git a/doc/cephadm/services/osd.rst b/doc/cephadm/services/osd.rst index 831bd238c79..90ebd86f897 100644 --- a/doc/cephadm/services/osd.rst +++ b/doc/cephadm/services/osd.rst @@ -198,6 +198,18 @@ There are a few ways to create new OSDs: .. warning:: When deploying new OSDs with ``cephadm``, ensure that the ``ceph-osd`` package is not already installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior. +* OSDs created via ``ceph orch daemon add`` are by default not added to the orchestrator's OSD service, they get added to 'osd' service. To attach an OSD to a different, existing OSD service, issue a command of the following form: + + .. prompt:: bash * + + ceph orch osd set-spec-affinity <service_name> <osd_id(s)> + + For example: + + .. prompt:: bash # + + ceph orch osd set-spec-affinity osd.default_drive_group 0 1 + Dry Run ------- diff --git a/doc/cephadm/services/rgw.rst b/doc/cephadm/services/rgw.rst index ed0b149365a..3df8ed2fc56 100644 --- a/doc/cephadm/services/rgw.rst +++ b/doc/cephadm/services/rgw.rst @@ -173,6 +173,32 @@ Then apply this yaml document: Note the value of ``rgw_frontend_ssl_certificate`` is a literal string as indicated by a ``|`` character preserving newline characters. +Disabling multisite sync traffic +-------------------------------- + +There is an RGW config option called ``rgw_run_sync_thread`` that tells the +RGW daemon to not transmit multisite replication data. This is useful if you want +that RGW daemon to be dedicated to I/O rather than multisite sync operations. +The RGW spec file includes a setting ``disable_multisite_sync_traffic`` that when +set to "True" will tell cephadm to set ``rgw_run_sync_thread`` to false for all +RGW daemons deployed for that RGW service. For example + +.. code-block:: yaml + + service_type: rgw + service_id: foo + placement: + label: rgw + spec: + rgw_realm: myrealm + rgw_zone: myzone + rgw_zonegroup: myzg + disable_multisite_sync_traffic: True + +.. note:: This will only stop the RGW daemon(s) from sending replication data. + The daemon can still receive replication data unless it has been removed + from the zonegroup and zone replication endpoints. + Service specification --------------------- diff --git a/doc/cephfs/health-messages.rst b/doc/cephfs/health-messages.rst index 0f171c6ccc9..7aa1f2e44ee 100644 --- a/doc/cephfs/health-messages.rst +++ b/doc/cephfs/health-messages.rst @@ -269,3 +269,11 @@ other daemons, please see :ref:`health-checks`. To evict and permanently block broken clients from connecting to the cluster, set the ``required_client_feature`` bit ``client_mds_auth_caps``. + +``MDS_ESTIMATED_REPLAY_TIME`` +----------------------------- + Message + "HEALTH_WARN Replay: x% complete. Estimated time remaining *x* seconds + + Description + When an MDS journal replay takes more than 30 seconds, this message indicates the estimated time to completion. diff --git a/doc/cephfs/index.rst b/doc/cephfs/index.rst index 57ea336c00b..630d29f1956 100644 --- a/doc/cephfs/index.rst +++ b/doc/cephfs/index.rst @@ -93,6 +93,7 @@ Administration CephFS Top Utility <cephfs-top> Scheduled Snapshots <snap-schedule> CephFS Snapshot Mirroring <cephfs-mirroring> + Purge Queue <purge-queue> .. raw:: html @@ -147,6 +148,7 @@ CephFS Concepts LazyIO <lazyio> Directory fragmentation <dirfrags> Multiple active MDS daemons <multimds> + Snapshots <snapshots> .. raw:: html diff --git a/doc/cephfs/mount-using-kernel-driver.rst b/doc/cephfs/mount-using-kernel-driver.rst index 22ede055d0b..bf229099a19 100644 --- a/doc/cephfs/mount-using-kernel-driver.rst +++ b/doc/cephfs/mount-using-kernel-driver.rst @@ -4,32 +4,32 @@ Mount CephFS using Kernel Driver ================================= -The CephFS kernel driver is part of the Linux kernel. It allows mounting -CephFS as a regular file system with native kernel performance. It is the -client of choice for most use-cases. +The CephFS kernel driver is part of the Linux kernel. It makes possible the +mounting of CephFS as a regular file system with native kernel performance. It +is the client of choice for most use-cases. -.. note:: CephFS mount device string now uses a new (v2) syntax. The mount - helper (and the kernel) is backward compatible with the old syntax. - This means that the old syntax can still be used for mounting with - newer mount helpers and kernel. However, it is recommended to use - the new syntax whenever possible. +.. note:: The CephFS mount device string now uses a new syntax ("v2"). The + mount helper is backward compatible with the old syntax. The kernel is + backward-compatible with the old syntax. This means that the old syntax can + still be used for mounting with newer mount helpers and with the kernel. Prerequisites ============= Complete General Prerequisites ------------------------------ -Go through the prerequisites required by both, kernel as well as FUSE mounts, -in `Mount CephFS: Prerequisites`_ page. +Go through the prerequisites required by both kernel and FUSE mounts, +as described on the `Mount CephFS: Prerequisites`_ page. Is mount helper present? ------------------------ -``mount.ceph`` helper is installed by Ceph packages. The helper passes the +The ``mount.ceph`` helper is installed by Ceph packages. The helper passes the monitor address(es) and CephX user keyrings, saving the Ceph admin the effort of passing these details explicitly while mounting CephFS. If the helper is not present on the client machine, CephFS can still be mounted using the kernel -driver, but only by passing these details explicitly to the ``mount`` command. -To check whether ``mount.ceph`` is present on your system, run the following command: +driver but only by passing these details explicitly to the ``mount`` command. +To check whether ``mount.ceph`` is present on your system, run the following +command: .. prompt:: bash # @@ -38,70 +38,88 @@ To check whether ``mount.ceph`` is present on your system, run the following com Which Kernel Version? --------------------- -Because the kernel client is distributed as part of the linux kernel (not -as part of packaged ceph releases), you will need to consider which kernel +Because the kernel client is distributed as part of the Linux kernel (and not +as part of the packaged Ceph releases), you will need to consider which kernel version to use on your client nodes. Older kernels are known to include buggy -ceph clients, and may not support features that more recent Ceph clusters +Ceph clients and may not support features that more recent Ceph clusters support. -Remember that the "latest" kernel in a stable linux distribution is likely -to be years behind the latest upstream linux kernel where Ceph development +Remember that the "latest" kernel in a stable Linux distribution is likely +to be years behind the latest upstream Linux kernel where Ceph development takes place (including bug fixes). As a rough guide, as of Ceph 10.x (Jewel), you should be using a least a 4.x kernel. If you absolutely have to use an older kernel, you should use the fuse client instead of the kernel client. -This advice does not apply if you are using a linux distribution that -includes CephFS support, as in this case the distributor will be responsible -for backporting fixes to their stable kernel: check with your vendor. +This advice does not apply if you are using a Linux distribution that includes +CephFS support. In that case, the distributor is responsible for backporting +fixes to their stable kernel. Check with your vendor. Synopsis ======== -In general, the command to mount CephFS via kernel driver looks like this:: +This is the general form of the command for mounting CephFS via the kernel driver: - mount -t ceph {device-string}={path-to-mounted} {mount-point} -o {key-value-args} {other-args} +.. prompt:: bash # + + mount -t ceph {device-string}={path-to-mounted} {mount-point} -o {key-value-args} {other-args} Mounting CephFS =============== -On Ceph clusters, CephX is enabled by default. Use ``mount`` command to -mount CephFS with the kernel driver:: +CephX authentication is enabled by default in Ceph clusters. Use the ``mount`` +command to use the kernel driver to mount CephFS: + +.. prompt:: bash # - mkdir /mnt/mycephfs - mount -t ceph <name>@<fsid>.<fs_name>=/ /mnt/mycephfs + mkdir /mnt/mycephfs + mount -t ceph <name>@<fsid>.<fs_name>=/ /mnt/mycephfs -``name`` is the username of the CephX user we are using to mount CephFS. -``fsid`` is the FSID of the ceph cluster which can be found using -``ceph fsid`` command. ``fs_name`` is the file system to mount. The kernel -driver requires MON's socket and the secret key for the CephX user, e.g.:: +#. ``name`` is the username of the CephX user we are using to mount CephFS. +#. ``fsid`` is the FSID of the Ceph cluster, which can be found using the + ``ceph fsid`` command. ``fs_name`` is the file system to mount. The kernel + driver requires a ceph Monitor's address and the secret key of the CephX + user. For example: - mount -t ceph cephuser@b3acfc0d-575f-41d3-9c91-0e7ed3dbb3fa.cephfs=/ -o mon_addr=192.168.0.1:6789,secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== + .. prompt:: bash # -When using the mount helper, monitor hosts and FSID are optional. ``mount.ceph`` -helper figures out these details automatically by finding and reading ceph conf -file, .e.g:: + mount -t ceph cephuser@b3acfc0d-575f-41d3-9c91-0e7ed3dbb3fa.cephfs=/ -o mon_addr=192.168.0.1:6789,secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== - mount -t ceph cephuser@.cephfs=/ -o secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== +When using the mount helper, monitor hosts and FSID are optional. The +``mount.ceph`` helper discovers these details by finding and reading the ceph +conf file. For example: -.. note:: Note that the dot (``.``) still needs to be a part of the device string. +.. prompt:: bash # -A potential problem with the above command is that the secret key is left in your -shell's command history. To prevent that you can copy the secret key inside a file -and pass the file by using the option ``secretfile`` instead of ``secret``:: + mount -t ceph cephuser@.cephfs=/ -o secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o secretfile=/etc/ceph/cephuser.secret +.. note:: Note that the dot (``.`` in the string ``cephuser@.cephfs``) must be + a part of the device string. -Ensure the permissions on the secret key file are appropriate (preferably, ``600``). +A weakness of this method is that it will leave the secret key in your shell's +command history. To avoid this, copy the secret key inside a file and pass the +file by using the option ``secretfile`` instead of ``secret``. For example: -Multiple monitor hosts can be passed by separating each address with a ``/``:: +.. prompt:: bash # - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o mon_addr=192.168.0.1:6789/192.168.0.2:6789,secretfile=/etc/ceph/cephuser.secret + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o secretfile=/etc/ceph/cephuser.secret -In case CephX is disabled, you can omit any credential related options:: +Ensure that the permissions on the secret key file are appropriate (preferably, +``600``). - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs +Multiple monitor hosts can be passed by separating addresses with a ``/``: -.. note:: The ceph user name still needs to be passed as part of the device string. +.. prompt:: bash # + + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o + mon_addr=192.168.0.1:6789/192.168.0.2:6789,secretfile=/etc/ceph/cephuser.secret + +If CephX is disabled, omit any credential-related options. For example: + +.. prompt:: bash # + + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs + +.. note:: The Ceph user name must be passed as part of the device string. To mount a subtree of the CephFS root, append the path to the device string:: @@ -111,29 +129,40 @@ Backward Compatibility ====================== The old syntax is supported for backward compatibility. -To mount CephFS with the kernel driver:: +To mount CephFS with the kernel driver, run the following commands: + +.. prompt:: bash # + + mkdir /mnt/mycephfs + mount -t ceph :/ /mnt/mycephfs -o name=admin - mkdir /mnt/mycephfs - mount -t ceph :/ /mnt/mycephfs -o name=admin +The key-value argument right after the option ``-o`` is the CephX credential. +``name`` is the username of the CephX user that is mounting CephFS. -The key-value argument right after option ``-o`` is CephX credential; -``name`` is the username of the CephX user we are using to mount CephFS. +To mount a non-default FS (in this example, ``cephfs2``), run commands of the following form. These commands are to be used in cases in which the cluster +has multiple file systems: -To mount a non-default FS ``cephfs2``, in case the cluster has multiple FSs:: +.. prompt:: bash # + + mount -t ceph :/ /mnt/mycephfs -o name=admin,fs=cephfs2 - mount -t ceph :/ /mnt/mycephfs -o name=admin,fs=cephfs2 +or - or +.. prompt:: bash # - mount -t ceph :/ /mnt/mycephfs -o name=admin,mds_namespace=cephfs2 + mount -t ceph :/ /mnt/mycephfs -o name=admin,mds_namespace=cephfs2 -.. note:: The option ``mds_namespace`` is deprecated. Use ``fs=`` instead when using the old syntax for mounting. +.. note:: The option ``mds_namespace`` is deprecated. Use ``fs=`` instead when + using the old syntax for mounting. Unmounting CephFS ================= -To unmount the Ceph file system, use the ``umount`` command as usual:: +To unmount the Ceph file system, use the ``umount`` command, as in this +example: + +.. prompt:: bash # - umount /mnt/mycephfs + umount /mnt/mycephfs .. tip:: Ensure that you are not within the file system directories before executing this command. @@ -150,11 +179,12 @@ For example:: cephuser@.cephfs=/ /mnt/ceph ceph mon_addr=192.168.0.1:6789,noatime,_netdev 0 0 -If the ``secret`` or ``secretfile`` options are not specified then the mount helper -will attempt to find a secret for the given ``name`` in one of the configured keyrings. +If the ``secret`` or ``secretfile`` options are not specified, the mount +helper will attempt to find a secret for the given ``name`` in one of the +configured keyrings. -See `User Management`_ for details on CephX user management and mount.ceph_ -manual for more options it can take. For troubleshooting, see +See `User Management`_ for details on CephX user management and the mount.ceph_ +manual for a list of the options it recognizes. For troubleshooting, see :ref:`kernel_mount_debugging`. .. _fstab: ../fstab/#kernel-driver diff --git a/doc/cephfs/purge-queue.rst b/doc/cephfs/purge-queue.rst new file mode 100644 index 00000000000..d7a68e7fa55 --- /dev/null +++ b/doc/cephfs/purge-queue.rst @@ -0,0 +1,106 @@ +============ +Purge Queue +============ + +MDS maintains a data structure known as **Purge Queue** which is responsible +for managing and executing the parallel deletion of files. +There is a purge queue for every MDS rank. Purge queues consist of purge items +which contain nominal information from the inodes such as size and the layout +(i.e. all other un-needed metadata information is discarded making it +independent of all metadata structures). + +Deletion process +================ + +When a client requests deletion of a directory (say ``rm -rf``): + +- MDS queues the files and subdirectories (purge items) from pq (purge queue) + journal in the purge queue. +- Processes the deletion of inodes in background in small and manageable + chunks. +- MDS instructs underlying OSDs to clean up the associated objects in data + pool. +- Updates the journal. + +.. note:: If the users delete the files more quickly than the + purge queue can process then the data pool usage might increase + substantially over time. In extreme scenarios, the purge queue + backlog can become so huge that it can slacken the capacity reclaim + and the linux ``du`` command for CephFS might report inconsistent + data compared to the CephFS Data pool. + +There are a few tunable configs that MDS uses internally to throttle purge +queue processing: + +.. confval:: filer_max_purge_ops +.. confval:: mds_max_purge_files +.. confval:: mds_max_purge_ops +.. confval:: mds_max_purge_ops_per_pg + +Generally, the defaults are adequate for most clusters. However, in +case of pretty huge clusters, if the need arises like ``pq_item_in_journal`` +(counter of things pending deletion) reaching gigantic figure then the configs +can be tuned to 4-5 times of the default value as a starting point and +further increments are subject to more requirements. + +Start from the most trivial config ``filer_max_purge_ops``, which should help +reclaim the space more quickly:: + + $ ceph config set mds filer_max_purge_ops 40 + +Incrementing ``filer_max_purge_ops`` should just work for most +clusters but if it doesn't then move ahead with tuning other configs:: + + $ ceph config set mds mds_max_purge_files 256 + $ ceph config set mds mds_max_purge_ops 32768 + $ ceph config set mds mds_max_purge_ops_per_pg 2 + +.. note:: Setting these values won't immediately break anything except + inasmuch as they control how many delete ops we issue to the + underlying RADOS cluster, but might eat up some cluster performance + if the values set are staggeringly high. + +.. note:: The purge queue is not an auto-tuning system in terms of its work + limits as compared to what is going on. So it is advised to make + a conscious decision while tuning the configs based on the cluster + size and workload. + +Examining purge queue perf counters +=================================== + +When analysing MDS perf dumps, the purge queue statistics look like:: + + "purge_queue": { + "pq_executing_ops": 56655, + "pq_executing_ops_high_water": 65350, + "pq_executing": 1, + "pq_executing_high_water": 3, + "pq_executed": 25, + "pq_item_in_journal": 6567004 + } + +Let us understand what each of these means: + +.. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - Name + - Description + * - pq_executing_ops + - Purge queue operations in flight + * - pq_executing_ops_high_water + - Maximum number of executing purge operations recorded + * - pq_executing + - Purge queue files being deleted + * - pq_executing_high_water + - Maximum number of executing file purges + * - pq_executed + - Purge queue files deleted + * - pq_item_in_journal + - Purge items (files) left in journal + +.. note:: ``pq_executing`` and ``pq_executing_ops`` might look similar but + there is a small nuance. ``pq_executing`` tracks number of files + in the purge queue while ``pq_executing_ops`` is the count of RADOS + objects from all the files in purge queue. diff --git a/doc/cephfs/snap-schedule.rst b/doc/cephfs/snap-schedule.rst index a94d938040f..48e79047864 100644 --- a/doc/cephfs/snap-schedule.rst +++ b/doc/cephfs/snap-schedule.rst @@ -197,6 +197,15 @@ this happens, the next snapshot will be schedule as if the previous one was not delayed, i.e. one or more delayed snapshots will not cause drift in the overall schedule. +If a volume is deleted while snapshot schedules are active on the volume, then +there might be cases when Python Tracebacks are seen in the log file or on the +command-line when commands are executed on such volumes. Although measures have +been taken to take note of the fs_map changes and delete active timers and +close database connections to avoid Python Tracebacks, it is not possible to +completely mute the tracebacks due to the inherent nature of problem. In the +event that such tracebacks are seen, the only solution to get the system to a +stable state is the disable and re-enable the snap_schedule Manager Module. + In order to somewhat limit the overall number of snapshots in a file system, the module will only keep a maximum of 50 snapshots per directory. If the retention policy results in more then 50 retained snapshots, the retention list will be diff --git a/doc/cephfs/snapshots.rst b/doc/cephfs/snapshots.rst new file mode 100644 index 00000000000..a60be96ed53 --- /dev/null +++ b/doc/cephfs/snapshots.rst @@ -0,0 +1,85 @@ +================ +CephFS Snapshots +================ + +CephFS snapshots create an immutable view of the file system at the point +in time they are taken. CephFS support snapshots which is managed in a +special hidden subdirectory named ``.snap`` .Snapshots are created using +``mkdir`` inside this directory. + +Snapshots can be exposed with a different name by changing the following client configurations. + +- ``snapdirname`` which is a mount option for kernel clients +- ``client_snapdir`` which is a mount option for ceph-fuse. + +Snapshot Creation +================== + +CephFS snapshot feature is enabled by default on new file systems. To enable +it on existing file systems, use the command below. + +.. code-block:: bash + + $ ceph fs set <fs_name> allow_new_snaps true + +When snapshots are enabled, all directories in CephFS will have a special ``.snap`` +directory. (You may configure a different name with the client snapdir setting if +you wish.) +To create a CephFS snapshot, create a subdirectory under ``.snap`` with a name of +your choice. +For example, to create a snapshot on directory ``/file1/``, invoke ``mkdir /file1/.snap/snapshot-name`` + +.. code-block:: bash + + $ touch file1 + $ cd .snap + $ mkdir my_snapshot + +Using snapshot to recover data +=============================== + +Snapshots can also be used to recover some deleted files. + +- ``create a file1 and create snapshot snap1`` + +.. code-block:: bash + + $ touch /mnt/cephfs/file1 + $ cd .snap + $ mkdir snap1 + +- ``create a file2 and create snapshot snap2`` + +.. code-block:: bash + + $ touch /mnt/cephfs/file2 + $ cd .snap + $ mkdir snap2 + +- ``delete file1 and create a new snapshot snap3`` + +.. code-block:: bash + + $ rm /mnt/cephfs/file1 + $ cd .snap + $ mkdir snap3 + +- ``recover file1 using snapshot snap2 using cp command`` + +.. code-block:: bash + + $ cd .snap + $ cd snap2 + $ cp file1 /mnt/cephfs/ + +Snapshot Deletion +================== + +Snapshots are deleted by invoking ``rmdir`` on the ``.snap`` directory they are +rooted in. (Attempts to delete a directory which roots the snapshots will fail; +you must delete the snapshots first.) + +.. code-block:: bash + + $ cd .snap + $ rmdir my_snapshot diff --git a/doc/conf.py b/doc/conf.py index 4fdc9a53b75..5293ff1b212 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -76,7 +76,7 @@ html_show_sphinx = False html_static_path = ["_static"] html_sidebars = { '**': ['smarttoc.html', 'searchbox.html'] - } +} html_css_files = ['css/custom.css'] @@ -133,13 +133,23 @@ extensions = [ 'sphinxcontrib.mermaid', 'sphinxcontrib.openapi', 'sphinxcontrib.seqdiag', - ] +] ditaa = shutil.which("ditaa") if ditaa is not None: # in case we don't have binfmt_misc enabled or jar is not registered - ditaa_args = ['-jar', ditaa] - ditaa = 'java' + _jar_paths = [ + '/usr/share/ditaa/lib/ditaa.jar', # Gentoo + '/usr/share/ditaa/ditaa.jar', # deb + '/usr/share/java/ditaa.jar', # rpm + ] + _jar_paths = [p for p in _jar_paths if os.path.exists(p)] + if _jar_paths: + ditaa = 'java' + ditaa_args = ['-jar', _jar_paths[0]] + else: + # keep ditaa from shutil.which + ditaa_args = [] extensions += ['sphinxcontrib.ditaa'] else: extensions += ['plantweb.directive'] diff --git a/doc/dev/crimson/pipeline.rst b/doc/dev/crimson/pipeline.rst index e9115c6d7c3..6e47b79d80b 100644 --- a/doc/dev/crimson/pipeline.rst +++ b/doc/dev/crimson/pipeline.rst @@ -2,96 +2,34 @@ The ``ClientRequest`` pipeline ============================== -In crimson, exactly like in the classical OSD, a client request has data and -ordering dependencies which must be satisfied before processing (actually -a particular phase of) can begin. As one of the goals behind crimson is to -preserve the compatibility with the existing OSD incarnation, the same semantic -must be assured. An obvious example of such data dependency is the fact that -an OSD needs to have a version of OSDMap that matches the one used by the client -(``Message::get_min_epoch()``). - -If a dependency is not satisfied, the processing stops. It is crucial to note -the same must happen to all other requests that are sequenced-after (due to -their ordering requirements). - -There are a few cases when the blocking of a client request can happen. - - - ``ClientRequest::ConnectionPipeline::await_map`` - wait for particular OSDMap version is available at the OSD level - ``ClientRequest::ConnectionPipeline::get_pg`` - wait a particular PG becomes available on OSD - ``ClientRequest::PGPipeline::await_map`` - wait on a PG being advanced to particular epoch - ``ClientRequest::PGPipeline::wait_for_active`` - wait for a PG to become *active* (i.e. have ``is_active()`` asserted) - ``ClientRequest::PGPipeline::recover_missing`` - wait on an object to be recovered (i.e. leaving the ``missing`` set) - ``ClientRequest::PGPipeline::get_obc`` - wait on an object to be available for locking. The ``obc`` will be locked - before this operation is allowed to continue - ``ClientRequest::PGPipeline::process`` - wait if any other ``MOSDOp`` message is handled against this PG - -At any moment, a ``ClientRequest`` being served should be in one and only one -of the phases described above. Similarly, an object denoting particular phase -can host not more than a single ``ClientRequest`` the same time. At low-level -this is achieved with a combination of a barrier and an exclusive lock. -They implement the semantic of a semaphore with a single slot for these exclusive -phases. - -As the execution advances, request enters next phase and leaves the current one -freeing it for another ``ClientRequest`` instance. All these phases form a pipeline -which assures the order is preserved. - -These pipeline phases are divided into two ordering domains: ``ConnectionPipeline`` -and ``PGPipeline``. The former ensures order across a client connection while -the latter does that across a PG. That is, requests originating from the same -connection are executed in the same order as they were sent by the client. -The same applies to the PG domain: when requests from multiple connections reach -a PG, they are executed in the same order as they entered a first blocking phase -of the ``PGPipeline``. - -Comparison with the classical OSD ----------------------------------- -As the audience of this document are Ceph Developers, it seems reasonable to -match the phases of crimson's ``ClientRequest`` pipeline with the blocking -stages in the classical OSD. The names in the right column are names of -containers (lists and maps) used to implement these stages. They are also -already documented in the ``PG.h`` header. - -+----------------------------------------+--------------------------------------+ -| crimson | ceph-osd waiting list | -+========================================+======================================+ -|``ConnectionPipeline::await_map`` | ``OSDShardPGSlot::waiting`` and | -|``ConnectionPipeline::get_pg`` | ``OSDShardPGSlot::waiting_peering`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::await_map`` | ``PG::waiting_for_map`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::wait_for_active`` | ``PG::waiting_for_peered`` | -| +--------------------------------------+ -| | ``PG::waiting_for_flush`` | -| +--------------------------------------+ -| | ``PG::waiting_for_active`` | -+----------------------------------------+--------------------------------------+ -|To be done (``PG_STATE_LAGGY``) | ``PG::waiting_for_readable`` | -+----------------------------------------+--------------------------------------+ -|To be done | ``PG::waiting_for_scrub`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::recover_missing`` | ``PG::waiting_for_unreadable_object``| -| +--------------------------------------+ -| | ``PG::waiting_for_degraded_object`` | -+----------------------------------------+--------------------------------------+ -|To be done (proxying) | ``PG::waiting_for_blocked_object`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::get_obc`` | *obc rwlocks* | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::process`` | ``PG::lock`` (roughly) | -+----------------------------------------+--------------------------------------+ - - -As the last word it might be worth to emphasize that the ordering implementations -in both classical OSD and in crimson are stricter than a theoretical minimum one -required by the RADOS protocol. For instance, we could parallelize read operations -targeting the same object at the price of extra complexity but we don't -- the -simplicity has won. +RADOS requires writes on each object to be ordered. If a client +submits a sequence of concurrent writes (doesn't want for the prior to +complete before submitting the next), that client may rely on the +writes being completed in the order in which they are submitted. + +As a result, the client->osd communication and queueing mechanisms on +both sides must take care to ensure that writes on a (connection, +object) pair remain ordered for the entire process. + +crimson-osd enforces this ordering via Pipelines and Stages +(crimson/osd/osd_operation.h). Upon arrival at the OSD, messages +enter the ConnectionPipeline::AwaitActive stage and proceed +through a sequence of pipeline stages: + +* ConnectionPipeline: per-connection stages representing the message handling + path prior to being handed off to the target PG +* PerShardPipeline: intermediate Pipeline representing the hand off from the + receiving shard to the shard with the target PG. +* CommonPGPipeline: represents processing on the target PG prior to obtaining + the ObjectContext for the target of the operation. +* CommonOBCPipeline: represents the actual processing of the IO on the target + object + +Because CommonOBCPipeline is per-object rather than per-connection or +per-pg, multiple requests on different objects may be in the same +CommonOBCPipeline stage concurrently. This allows us to serve +multiple reads in the same PG concurrently. We can also process +writes on multiple objects concurrently up to the point at which the +write is actually submitted. + +See crimson/osd/osd_operations/client_request.(h|cc) for details. diff --git a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst index 34dfd521eaa..6964012ef31 100644 --- a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst +++ b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst @@ -6,7 +6,8 @@ Integration Tests using Teuthology Workflow Infrastructure -------------- -Components: +Components +********** 1. `ceph-ci`_: Clone of the main Ceph repository, used for triggering Jenkins Ceph builds for development. @@ -44,7 +45,27 @@ Components: Each Teuthology test *run* contains multiple test *jobs*. Each job runs in an environment isolated from other jobs, on a different collection of test nodes. -To test a change in Ceph, follow these steps: +Workflow Overview +***************** + +.. image:: workflow.png + + +To test a change in Ceph, start by pushing a branch with your changes to the +`ceph-ci`_ repository. This will automatically trigger the Jenkins process +to build Ceph binaries - the status of the build can be observed on `Shaman`_. +These built packages will be uploaded on `Chacra`_. + +To schedule a Teuthology integration test against this new build, you will +need access to the Sepia lab. Once you have access, log into the Teuthology +machine and complete the one-time initial Teuthology setup required to run +Teuthology commands. After the setup, use the ``teuthology-suite`` command to schedule +a Teuthology run. In this command, use the ``-c <ceph-ci branch name>`` option to +specify your build. The results of your test can be observed on `Pulpito`_. +Log into a `developer playground machine`_ to review the Teuthology run's archive logs. + + +The rest of the document will explain these steps in detail: 1. Getting binaries - Build Ceph. 2. Scheduling Test Run: @@ -98,6 +119,31 @@ Ceph binaries must be built for your branch before you can use teuthology to run .. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph +Pushing to the ceph-ci repository +********************************* + +Follow these steps to push to the ceph-ci repository. After pushing, a new build will +automatically be scheduled. + +1. Add the ceph-ci repository as a remote to your local clone of the Ceph repository: + + .. prompt:: bash $ + + git remote add ceph-ci git@github.com:ceph/ceph-ci.git + + $ git remote -v + origin git@github.com:ceph/ceph.git (fetch) + origin git@github.com:ceph/ceph.git (push) + ceph-ci git@github.com:ceph/ceph-ci.git (fetch) + ceph-ci git@github.com:ceph/ceph-ci.git (push) + +2. Push your branch upstream by running a command of the following form: + + .. prompt:: bash $ + + $ git push ceph-ci wip-yourname-feature-x + + Naming the ceph-ci branch ************************* Prepend your branch with your name before you push it to ceph-ci. For example, @@ -110,15 +156,14 @@ the name of that stable branch in your ceph-ci branch name. For example, the ``feature-x`` PR branch should be named ``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.* -You can choose to only trigger a CentOS 9.Stream build (excluding other distro like ubuntu) -by adding "centos9-only" at the end of the ceph-ci branch name. For example, -``wip-$yourname-feature-centos9-only``. This helps to get quicker builds and save resources -when you don't require binaries for other distros. - Delete the branch from ceph-ci when you no longer need it. If you are logged in to GitHub, all your branches on ceph-ci can be found here: https://github.com/ceph/ceph-ci/branches. +.. note:: You can choose to only trigger a CentOS 9.Stream build (excluding other + distro like ubuntu) by adding "centos9-only" at the end of the ceph-ci branch name. + For example, ``wip-$yourname-feature-centos9-only``. This helps to get quicker builds + and save resources when you don't require binaries for other distros. Scheduling Test Run ------------------- diff --git a/doc/dev/developer_guide/testing_integration_tests/workflow.png b/doc/dev/developer_guide/testing_integration_tests/workflow.png Binary files differnew file mode 100644 index 00000000000..610baf683bc --- /dev/null +++ b/doc/dev/developer_guide/testing_integration_tests/workflow.png diff --git a/doc/dev/development-workflow.rst b/doc/dev/development-workflow.rst index 904269c5121..530944bf580 100644 --- a/doc/dev/development-workflow.rst +++ b/doc/dev/development-workflow.rst @@ -243,6 +243,10 @@ differences: * All commits are cherry-picked with ``git cherry-pick -x`` to reference the original commit +.. note: If a backport is appropriate, the submitter is responsible for + determining appropriate target stable branches to which backports must be + made. + See `the backporter manual <https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst>`_ for more information. diff --git a/doc/dev/libcephfs_proxy.rst b/doc/dev/libcephfs_proxy.rst new file mode 100644 index 00000000000..baa96f765c9 --- /dev/null +++ b/doc/dev/libcephfs_proxy.rst @@ -0,0 +1,289 @@ +Design of the libcephfs proxy +============================= + +Description of the problem +-------------------------- + +When an application connects to a Ceph volume through the *libcephfs.so* +library, a cache is created locally inside the process. The *libcephfs.so* +implementation already deals with memory usage of the cache and adjusts it so +that it doesn't consume all the available memory. However, if multiple +processes connect to CephFS through different instances of the library, each +one of them will keep a private cache. In this case memory management is not +effective because, even configuring memory limits, the number of libcephfs +instances that can be created is unbounded and they can't work in a coordinated +way to correctly control ressource usage. Due to this, it's relatively easy to +consume all memory when all processes are using data cache intensively. This +causes the OOM killer to terminate those processes. + +Proposed solution +----------------- + +High level approach +^^^^^^^^^^^^^^^^^^^ + +The main idea is to create a *libcephfs_proxy.so* library that will provide the +same API as the original *libcephfs.so*, but won't cache any data. This library +can be used by any application currently using *libcephfs.so* in a transparent +way (i.e. no code modification required) just by linking against +*libcephfs_proxy.so* instead of *libcephfs.so*, or even using *LD_PRELOAD*. + +A new *libcephfsd* daemon will also be created. This daemon will link against +the real *libcephfs.so* library and will listen for incoming connections on a +UNIX socket. + +When an application starts and initiates CephFS requests through the +*libcephfs_proxy.so* library, it will connect to the *libcephfsd* daemon +through the UNIX socket and it will forward all CephFS requests to it. The +daemon will use the real *libcephfs.so* to execute those requests and the +answers will be returned back to the application, potentially caching data in +the *libcephfsd* process itself. All this will happen transparently, without +any knowledge from the application. + +The daemon will share low level *libcephfs.so* mounts between different +applications to avoid creating an instance for each application, which would +have the same effect on memory as linking each application directly to the +*libcephfs.so* library. This will be done only if the configuration defined by +the applications is identical. Otherwise new independent instances will still +be created. + +Some *libcephfs.so* functions will need to be implemented in an special way +inside the *libcephfsd* daemon to hide the differences caused by sharing the +same mount instance with more than one client (for example chdir/getcwd cannot +rely directly on the ``ceph_chdir()``/``ceph_getcwd()`` of *libcephfs.so*). + +Initially, only the subset of the low-level interface functions of +*libcephfs.so* that are used by the Samba's VFS CephFS module will be provided. + +Design of common components +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Network protocol +"""""""""""""""" + +Since the connection through the UNIX socket is to another process that runs on +the same machine and the data we need to pass is quite simple, we'll avoid all +the overhead of generic XDR encoding/decoding and RPC transmissions by using a +very simple serialization implemented in the code itself. For the future we may +consider using cap'n proto (https://capnproto.org), which claims to have zero +overhead for encoding and decoding, and would provide an easy way to support +backward compatibility if the network protocol needs to be modified in the +future. + +Design of the *libcephfs_proxy.so* library +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This library will basically connect to the UNIX socket where the *libcephfsd* +daemon is listening, wait for requests coming from the application, serialize +all function arguments and send them to the daemon. Once the daemon responds it +will deserialize the answer and return the result to the application. + +Local caching +""""""""""""" + +While the main purpose of this library is to avoid independent caches on each +process, some preliminary testing has shown a big performance drop for +workloads based on metadata operations and/or small files when all requests go +through the proxy daemon. To minimize this, metadata caching should be +implemented. Metadata cache is much smaller than data cache and will provide a +good trade-off between memory usage and performance. + +To implement caching in a safe way, it's required to correctly invalidate data +before it becomes stale. Currently libcephfs.so provides invalidation +notifications that can be used to implement this, but its semantics are not +fully understood yet, so the cache in the libcephfs_proxy.so library will be +designed and implemented in a future version. + + +Design of the *libcephfsd* daemon +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The daemon will be a regular process that will centralize libcephfs requests +coming from other processes on the same machine. + +Process maintenance +""""""""""""""""""" + +Since the process will work as a standalone daemon, a simple systemd unit file +will be provided to manage it as a regular system service. Most probably this +will be integrated inside cephadm in the future. + +In case the *libcephfsd* daemon crashes, we'll rely on systemd to restart it. + + +Special functions +^^^^^^^^^^^^^^^^^ + +Some functions will need to be handled in a special way inside the *libcephfsd* +daemon to provide correct functionality since forwarding them directly to +*libcephfs.so* could return incorrect results due to the sharing of low-level +mounts. + +**Sharing of underlying struct ceph_mount_info** + +The main purpose of the proxy is to avoid creating a new mount for each process +when they are accessing the same data. To be able to provide this we need to +"virtualize" the mount points and let the application believe it's using its +own mount when, in fact, it could be using a shared mount. + +The daemon will track the Ceph account used to connect to the volume, the +configuration file and any specific configuration changes done before mounting +the volume. Only if all settings are exactly the same as another already +mounted instance, then the mount will be shared. The daemon won't understand +CephFS settings nor any potential dependencies between settings. For this +reason, a very strict comparison will be done: the configuration file needs to +be identical and any other changes made afterwards need to be set to the exact +same value and in the same order so that two configurations can be considered +identical. + +The check to determine whether two configurations are identical or not will be +done just before mounting the volume (i.e. ``ceph_mount()``). This means that +during the configuration phase, we may have many simultaneous mounts allocated +but not yet mounted. However only one of them will become a real mount. The +others will remain unmounted and will be eventually destroyed once users +unmount and release them. + +The following functions will be affected: + +* **ceph_create** + + This one will allocate a new ceph_mount_info structure, and the provided id + will be recorded for future comparison of potentially matching mounts. + +* **ceph_release** + + This one will release an unmounted ceph_mount_info structure. Unmounted + structures won't be shared with anyone else. + +* **ceph_conf_read_file** + + This one will read the configuration file, compute a checksum and make a + copy. The copy will make sure that there are no changes in the configuration + file since the time the checksum was computed, and the checksum will be + recorded for future comparison of potentially matching mounts. + +* **ceph_conf_get** + + This one will obtain the requested setting, recording it for future + comparison of potentially matching mounts. + + Even though this may seem unnecessary, since the daemon is considering the + configuration as a black box, it could be possible to have some dynamic + setting that could return different values depending on external factors, so + the daemon also requires that any requested setting returns the same value to + consider two configurations identical. + +* **ceph_conf_set** + + This one will record the modified value for future comparison of potentially + matching mounts. + + In normal circumstances, some settings may be set even after having mounted + the volume. The proxy won't allow that to avoid potential interferences with + other clients sharing the same mount. + +* **ceph_init** + + This one will be a no-op. Calling this function triggers the allocation of + several resources and starts some threads. This is just a waste of resources + if this *ceph_mount_info* structure is not finally mounted because it matches + with an already existing mount. + + Only if at the time of mount (i.e. ``ceph_mount()``) there's no match with + already existing mounts, then the mount will be initialized and mounted at + the same time. + +* **ceph_select_filesystem** + + This one will record the selected file system for future comparison of + potentially matching mounts. + +* **ceph_mount** + + This one will try to find an active mount that matches with all the + configurations defined for this *ceph_mount_info* structure. If none is + found, it will be mounted. Otherwise, the already existing mount will be + shared with this client. + + The unmounted *ceph_mount_info* structures will be kept around associated + with the mounted one. + + All "real" mounts will be made against the absolute root of the volume + (i.e. "/") to make sure they can be shared with other clients later, + regardless of whether they use the same mount point or not. This means that + just after mounting, the daemon will need to resolve and store the root inode + of the "virtual" mount point. + + The CWD (Current Working Directory) will also be initialized to the same + inode. + +* **ceph_unmount** + + This one will detach the client from the mounted *ceph_mount_info* structure + and reattach it to one of the associated unmounted structures. If this was + the last user of the mount, it's finally unmounted instead. + + After calling this function, the client continues using a private + *ceph_mount_info* structure that is used exclusively by itself, so other + configuration changes and operations can be done safely. + +**Confine accesses to the intended mount point** + +Since the effective mount point may not match the real mount point, some +functions could be able to return inodes outside of the effective mount point +if not handled with care. To avoid it and provide the result that the user +application expects, we will need to simulate some of them inside the +*libcephfsd* daemon. + +There are three special cases to consider: + +1. Handling of paths starting with "/" +2. Handling of paths containing ".." (i.e. parent directory) +3. Handling of paths containing symbolic links + +When these special paths are found, they need to be handled in a special way to +make sure that the returned inodes are what the client expects. + +The following functions will be affected: + +* **ceph_ll_lookup** + + Lookup accepts ".." as the name to resolve. If the parent directory is the + root of the "virtual" mount point (which may not be the same as the real + mount point), we'll need to return the inode corresponding to the "virtual" + mount point stored at the time of mount, instead of the real parent. + +* **ceph_ll_lookup_root** + + This one needs to return the root inode stored at the time of mount. + +* **ceph_ll_walk** + + This one will be completely reimplemented inside the daemon to be able to + correctly parse each path component and symbolic link, and handle "/" and + ".." in the correct way. + +* **ceph_chdir** + + This one will resolve the passed path and store it along the corresponding + inode inside the current "virtual" mount. The real ``ceph_chdir()`` won't be + called. + +* **ceph_getcwd** + + This one will just return the path stored in the "virtual" mount from + previous ``ceph_chdir()`` calls. + +**Handle AT_FDCWD** + +Any function that receives a file descriptor could also receive the special +*AT_FDCWD* value. These functions need to check for that value and use the +"virtual" CWD instead. + +Testing +------- + +The proxy should be transparent to any application already using +*libcephfs.so*. This also applies to testing scripts and applications. So any +existing test against the regular *libcephfs.so* library can also be used to +test the proxy. diff --git a/doc/man/8/cephadm.rst b/doc/man/8/cephadm.rst index b2cad6cb505..3c23a9867f7 100644 --- a/doc/man/8/cephadm.rst +++ b/doc/man/8/cephadm.rst @@ -13,7 +13,7 @@ Synopsis | [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] | [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] | [--retry RETRY] [--no-container-init] -| {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,list-images} +| {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,list-images,update-osd-service} | ... @@ -106,6 +106,7 @@ Synopsis | **cephadm** **list-images** +| **cephadm** **update-osd-service** [-h] [--fsid FSID] --osd-ids OSD_IDS --service-name SERVICE_NAME Description @@ -535,6 +536,18 @@ list-images List the default container images for all services in ini format. The output can be modified with custom images and passed to --config flag during bootstrap. +update-osd-service +------------------ + +Update the OSD service for specific OSDs + +Arguments: + +* [--fsid FSID] cluster FSID +* --osd-ids OSD_IDS Comma-separated OSD IDs +* --service-name SERVICE_NAME OSD service name + + Availability ============ diff --git a/doc/man/8/radosgw-admin.rst b/doc/man/8/radosgw-admin.rst index c7750c348ad..3cd4338a5ec 100644 --- a/doc/man/8/radosgw-admin.rst +++ b/doc/man/8/radosgw-admin.rst @@ -541,6 +541,13 @@ Options Generate random secret key. +.. option:: --generate-key + + create user with or without credentials. + If this option set to false, then user cannot set --gen-access-key/--gen-secret/--secret-key/--access-key. + If this option set to true, then user cannot set --secret-key/--access-key and bypass options for --gen-secret/--gen-access-key. + Default is true. + .. option:: --key-type=<type> Key type, options are: swift, s3. diff --git a/doc/mgr/dashboard.rst b/doc/mgr/dashboard.rst index b0448bd0eef..32824fab4b5 100644 --- a/doc/mgr/dashboard.rst +++ b/doc/mgr/dashboard.rst @@ -1310,9 +1310,9 @@ redirection on standby nodes. mode tcp option httpchk GET / http-check expect status 200 - server x <HOST>:<PORT> ssl check verify none - server y <HOST>:<PORT> ssl check verify none - server z <HOST>:<PORT> ssl check verify none + server x <HOST>:<PORT> check check-ssl verify none + server y <HOST>:<PORT> check check-ssl verify none + server z <HOST>:<PORT> check check-ssl verify none .. _dashboard-auditing: diff --git a/doc/rados/operations/health-checks.rst b/doc/rados/operations/health-checks.rst index f5d38948150..a1498a09fd0 100644 --- a/doc/rados/operations/health-checks.rst +++ b/doc/rados/operations/health-checks.rst @@ -1665,6 +1665,14 @@ Some of the gateways are in the GW_UNAVAILABLE state. If a NVMeoF daemon has crashed, the daemon log file (found at ``/var/log/ceph/``) may contain troubleshooting information. +NVMEOF_GATEWAY_DELETING +_______________________ + +Some of the gateways are in the GW_DELETING state. They will stay in this +state until all the namespaces under the gateway's load balancing group are +moved to another load balancing group ID. This is done automatically by the +load balancing process. If this alert persist for a long time, there might +be an issue with that process. Miscellaneous ------------- diff --git a/doc/rados/operations/stretch-mode.rst b/doc/rados/operations/stretch-mode.rst index e8be5e13e6a..7a4fa46117d 100644 --- a/doc/rados/operations/stretch-mode.rst +++ b/doc/rados/operations/stretch-mode.rst @@ -119,13 +119,29 @@ See https://tracker.ceph.com/issues/68338 for more information. Stretch Mode ============ -Stretch mode is designed to handle deployments in which you cannot guarantee the -replication of data across two data centers. This kind of situation can arise -when the cluster's CRUSH rule specifies that three copies are to be made, but -then a copy is placed in each data center with a ``min_size`` of 2. Under such -conditions, a placement group can become active with two copies in the first -data center and no copies in the second data center. +Stretch mode is designed to handle netsplit scenarios between two data zones as well +as the loss of one data zone. It handles the netsplit scenario by choosing the surviving zone +that has the better connection to the ``tiebreaker monitor``. It handles the loss of one zone by +reducing the ``size`` to ``2`` and ``min_size`` to ``1``, allowing the cluster to continue operating +with the remaining zone. When the lost zone comes back, the cluster will recover the lost data +and return to normal operation. + +Connectivity Monitor Election Strategy +--------------------------------------- +When using stretch mode, the monitor election strategy must be set to ``connectivity``. +This strategy tracks network connectivity between the monitors and is +used to determine which zone should be favored when the cluster is in a netsplit scenario. + +See `Changing Monitor Elections`_ + +Stretch Peering Rule +-------------------- +One critical behavior of stretch mode is its ability to prevent a PG from going active if the acting set +contains only replicas from a single zone. This safeguard is crucial for mitigating the risk of data +loss during site failures because if a PG were allowed to go active with replicas only in a single site, +writes could be acknowledged despite a lack of redundancy. In the event of a site failure, all data in the +affected PG would be lost. Entering Stretch Mode --------------------- @@ -271,7 +287,7 @@ possible, if needed). .. _Changing Monitor elections: ../change-mon-elections Exiting Stretch Mode -===================== +-------------------- To exit stretch mode, run the following command: .. prompt:: bash $ diff --git a/doc/radosgw/account.rst b/doc/radosgw/account.rst index 6dab997d93e..0e4ede5a50a 100644 --- a/doc/radosgw/account.rst +++ b/doc/radosgw/account.rst @@ -174,6 +174,11 @@ An existing user can be adopted into an account with ``user modify``:: .. note:: Account membership is permanent. Once added, users cannot be removed from their account. +.. note:: The IAM User API imposes additional requirements on the format + of ``UserName``, which is enforced when migrating users into an account. + If migration fails with "UserName contains invalid characters", the + ``--display-name`` should be modified to match ``[\w+=,.@-]+``. + .. warning:: Ownership of the user's notification topics will not be transferred to the account. Notifications will continue to work, but the topics will no longer be visible to SNS Topic APIs. Topics and diff --git a/doc/radosgw/admin.rst b/doc/radosgw/admin.rst index 7c7d9d6df14..8dbf8c10b04 100644 --- a/doc/radosgw/admin.rst +++ b/doc/radosgw/admin.rst @@ -262,6 +262,7 @@ include: - ``--secret-key=<key>`` manually specifies a S3 secret key or a Swift secret key. - ``--gen-access-key`` automatically generates a random S3 access key. - ``--gen-secret`` automatically generates a random S3 secret key or a random Swift secret key. +- ``--generate-key`` create user with or without credentials. If sets to false, then user cannot set ``gen-secret/gen-access-key/access-key/secret-key`` Adding S3 keys ~~~~~~~~~~~~~~ diff --git a/doc/radosgw/bucket_logging.rst b/doc/radosgw/bucket_logging.rst index d96ffbe2758..cb9f8465d20 100644 --- a/doc/radosgw/bucket_logging.rst +++ b/doc/radosgw/bucket_logging.rst @@ -27,6 +27,8 @@ This time (in seconds) could be set per source bucket via a Ceph extension to th or globally via the `rgw_bucket_logging_obj_roll_time` configuration option. If not set, the default time is 5 minutes. Adding a log object to the log bucket is done "lazily", meaning, that if no more records are written to the object, it may remain outside of the log bucket even after the configured time has passed. +To counter that, you can flush all logging objects on a given source bucket to log them, +regardless if enough time passed or if no more records are written to the object. Standard ```````` @@ -38,8 +40,8 @@ Journal If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed. This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client. An exception to the above are "multi/delete" log records: if writing these log records fail, the operation continues and may still be successful. -Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written -before such a failure, there will be no indication for that in the log records. +Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular-expression matching can also be used on these to create filters. +Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written. Bucket Logging REST API diff --git a/doc/radosgw/config-ref.rst b/doc/radosgw/config-ref.rst index edc6a90b0f9..b4aa56fff54 100644 --- a/doc/radosgw/config-ref.rst +++ b/doc/radosgw/config-ref.rst @@ -78,7 +78,7 @@ These values can be tuned based upon your specific workload to further increase aggressiveness of lifecycle processing. For a workload with a larger number of buckets (thousands) you would look at increasing the :confval:`rgw_lc_max_worker` value from the default value of 3 whereas for a workload with a smaller number of buckets but higher number of objects (hundreds of thousands) -per bucket you would consider decreasing :confval:`rgw_lc_max_wp_worker` from the default value of 3. +per bucket you would consider increasing :confval:`rgw_lc_max_wp_worker` from the default value of 3. .. note:: When looking to tune either of these specific values please validate the current Cluster performance and Ceph Object Gateway utilization before increasing. diff --git a/doc/radosgw/notifications.rst b/doc/radosgw/notifications.rst index 05653956be1..897c280facf 100644 --- a/doc/radosgw/notifications.rst +++ b/doc/radosgw/notifications.rst @@ -188,6 +188,7 @@ updating, use the name of an existing topic and different endpoint values). [&Attributes.entry.15.key=Policy&Attributes.entry.15.value=<policy-JSON-string>] [&Attributes.entry.16.key=user-name&Attributes.entry.16.value=<user-name-string>] [&Attributes.entry.17.key=password&Attributes.entry.17.value=<password-string>] + [&Attributes.entry.18.key=kafka-brokers&Attributes.entry.18.value=<kafka-broker-list>] Request parameters: @@ -296,6 +297,8 @@ Request parameters: - "broker": Messages are considered "delivered" if acked by the broker. (This is the default.) + - kafka-brokers: A command-separated list of host:port of kafka brokers. These brokers (may contain a broker which is defined in kafka uri) will be added to kafka uri to support sending notifcations to a kafka cluster. + .. note:: - The key-value pair of a specific parameter need not reside in the same @@ -571,6 +574,7 @@ Valid AttributeName that can be passed: - mechanism: may be provided together with user/password (default: ``PLAIN``). - kafka-ack-level: No end2end acknowledgement is required. Messages may persist in the broker before being delivered to their final destinations. + - kafka-brokers: Set endpoint with broker(s) as a comma-separated list of host or host:port (default port 9092). Notifications ~~~~~~~~~~~~~ diff --git a/doc/radosgw/s3/bucketops.rst b/doc/radosgw/s3/bucketops.rst index 2e5cc48646d..c33a8c0f410 100644 --- a/doc/radosgw/s3/bucketops.rst +++ b/doc/radosgw/s3/bucketops.rst @@ -751,6 +751,14 @@ Parameters are XML encoded in the body of the request, in the following format: <TargetPrefix>string</TargetPrefix> <LoggingType>Standard|Journal</LoggingType> <ObjectRollTime>integer</ObjectRollTime> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> </LoggingEnabled> </BucketLoggingStatus> @@ -881,6 +889,14 @@ Response is XML encoded in the body of the request, in the following format: <TargetPrefix>string</TargetPrefix> <LoggingType>Standard|Journal</LoggingType> <ObjectRollTime>integer</ObjectRollTime> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> </LoggingEnabled> </BucketLoggingStatus> @@ -894,3 +910,27 @@ HTTP Response | ``404`` | NoSuchBucket | The bucket does not exist | +---------------+-----------------------+----------------------------------------------------------+ +Flush Bucket Logging +-------------------- + +Flushes all logging objects for a given source bucket (logging bucket are written lazily). + +Syntax +~~~~~~ + +:: + + POST /{bucket}?logging HTTP/1.1 + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``201`` | Created | Flushed all logging objects successfully | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + diff --git a/doc/radosgw/s3/objectops.rst b/doc/radosgw/s3/objectops.rst index 2ac52607fe3..ddc5fb910c4 100644 --- a/doc/radosgw/s3/objectops.rst +++ b/doc/radosgw/s3/objectops.rst @@ -115,7 +115,7 @@ Request Headers +---------------------------+------------------------------------------------+--------------------------------+------------+ | **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ -| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +| **if-none-match** | Gets only if object ETag doesn't match. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ Response Headers @@ -155,7 +155,7 @@ Request Headers +---------------------------+------------------------------------------------+--------------------------------+------------+ | **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ -| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +| **if-none-match** | Gets only if object ETag doesn't match | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ Get Object ACL diff --git a/doc/radosgw/uadk-accel.rst b/doc/radosgw/uadk-accel.rst index fdf99f891f0..aaafe1c21df 100644 --- a/doc/radosgw/uadk-accel.rst +++ b/doc/radosgw/uadk-accel.rst @@ -2,9 +2,9 @@ UADK Acceleration for Compression =============================================== -UADK is a framework for applications to access hardware accelerators in a -unified, secure, and efficient way. UADK is comprised of UACCE, libwd and many -other algorithm libraries. +UADK is a framework that makes it possible for applications to access hardware +accelerators in a unified, secure, and efficient way. UADK is comprised of +UACCE, libwd, and many other algorithm libraries. See `Compressor UADK Support`_. @@ -12,31 +12,31 @@ See `Compressor UADK Support`_. UADK in the Software Stack ========================== -UADK is a general-purpose user space accelerator framework that uses shared -virtual addressing (SVA) to provide a unified programming interface for hardware -acceleration of cryptographic and compression algorithms. +UADK is a general-purpose user space accelerator framework that uses Shared +Virtual Addressing (SVA) to provide a unified programming interface for +hardware acceleration of cryptographic and compression algorithms. UADK includes Unified/User-space-access-intended Accelerator Framework (UACCE), which enables hardware accelerators that support SVA to adapt to UADK. Currently, HiSilicon Kunpeng hardware accelerators have been registered with UACCE. Through the UADK framework, users can run cryptographic and compression -algorithms using hardware accelerators instead of CPUs, freeing up CPU computing -power and improving computing performance. +algorithms using hardware accelerators instead of CPUs, which frees up CPU +computing power and improves computing performance. -A user can access the hardware accelerators by performing user-mode operations on -the character devices, or the use of UADK can be done via frameworks that have -been enabled by others including UADK support (for example, OpenSSL* libcrypto*, -DPDK, and the Linux* Kernel Crypto Framework). +Users can access the hardware accelerators by performing user-mode operations +on the character devices, or the use of UADK can be achieved via frameworks +that have been enabled by others including UADK support (for example, OpenSSL* +libcrypto*, DPDK, and the Linux* Kernel Crypto Framework). See `OpenSSL UADK Engine`_. UADK Environment Setup ====================== -UADK consists of UACCE, vendors’ drivers, and an algorithm layer. UADK requires the -hardware accelerator to support SVA, and the operating system to support IOMMU and -SVA. Hardware accelerators from different vendors are registered as different character -devices with UACCE by using kernel-mode drivers of the vendors. +UADK consists of UACCE, vendor drivers, and an algorithm layer. UADK requires +the hardware accelerator to support SVA, and the operating system to support +IOMMU and SVA. Hardware accelerators are registered as different character +devices with UACCE by kernel-mode drivers. :: @@ -77,11 +77,12 @@ Configuration #. Kernel Requirement -User needs to make sure that UACCE is already supported in Linux kernel. The kernel version -should be at least v5.9 with SVA (Shared Virtual Addressing) enabled. +Users must ensure that UACCE is supported by the Linux kernel release in use, +which should be 5.9 or later with SVA (Shared Virtual Addressing) enabled. -UACCE may be built as a module or built into the kernel. Here's an example to build UACCE -with hardware accelerators for the HiSilicon Kunpeng platform. +UACCE may be built as a loadable module or built into the kernel. Here's an +example to build UACCE with hardware accelerators for the HiSilicon Kunpeng +platform. .. prompt:: bash $ @@ -97,13 +98,17 @@ with hardware accelerators for the HiSilicon Kunpeng platform. Make sure all these above kernel configurations are selected. #. UADK enablement -If the architecture is aarch64, it will automatically download the UADK source code to build -the static library. If it runs on other architecture, user can enable it with build parameters -`-DWITH_UADK=true` - -#. Manual Build UADK -As the above paragraph shows, the UADK is enabled automatically, no need to build manually. -For developer who is interested in UADK, you can refer to the below steps for building. +If the architecture is ``aarch64``, it will automatically download the UADK +source code to build the static library. When building on other CPU +architectures, the user may enable UADK by adding ``-DWITH_UADK=true`` to the +compilation command line options. Note that UADK may not be compatible with all +architectures. + +#. Manually Building UADK +As implied in the above paragraph, if the architecture is ``aarch64``, the UADK +is enabled automatically and there is no need to build it manually. However, +below we provide the procedure for manually building UADK so that developers +can study how it is built. .. prompt:: bash $ @@ -115,9 +120,9 @@ For developer who is interested in UADK, you can refer to the below steps for bu make make install - .. note:: Without –prefix, UADK will be installed to /usr/local/lib by - default. If get error:"cannot find -lnuma", please install - the `libnuma-dev`. + .. note:: Without ``--prefix``, UADK will be installed under + ``/usr/local/lib`` by default. If you get the error: + ``cannot find -lnuma``, install the ``libnuma-dev`` package. #. Configure @@ -126,7 +131,8 @@ For developer who is interested in UADK, you can refer to the below steps for bu uadk_compressor_enabled=true - The default value in `global.yaml.in` for `uadk_compressor_enabled` is false. + The default value in `global.yaml.in` for `uadk_compressor_enabled` is + ``false``. .. _Compressor UADK Support: https://github.com/ceph/ceph/pull/58336 .. _OpenSSL UADK Engine: https://github.com/Linaro/uadk_engine diff --git a/doc/releases/index.rst b/doc/releases/index.rst index a8015c65465..1393770878f 100644 --- a/doc/releases/index.rst +++ b/doc/releases/index.rst @@ -23,7 +23,6 @@ security fixes. Squid (v19.2.*) <squid> Reef (v18.2.*) <reef> - Quincy (v17.2.*) <quincy> .. ceph_releases:: releases.yml current @@ -40,6 +39,7 @@ receive bug fixes or backports). :maxdepth: 1 :hidden: + Quincy (v17.2.*) <quincy> Pacific (v16.2.*) <pacific> Octopus (v15.2.*) <octopus> Nautilus (v14.2.*) <nautilus> diff --git a/doc/releases/releases.yml b/doc/releases/releases.yml index 948f9eab278..6a76cc7c92c 100644 --- a/doc/releases/releases.yml +++ b/doc/releases/releases.yml @@ -32,6 +32,7 @@ releases: quincy: target_eol: 2024-06-01 + actual_eol: 2025-01-13 releases: - version: 17.2.8 released: 2024-11-25 diff --git a/doc/start/hardware-recommendations.rst b/doc/start/hardware-recommendations.rst index 3c3c781a815..3d5e44d8e02 100644 --- a/doc/start/hardware-recommendations.rst +++ b/doc/start/hardware-recommendations.rst @@ -311,7 +311,7 @@ media cost. Moreover, when using NVMe SSDs, you do not need *any* HBA. This additionally reduces the HDD vs SSD cost gap when the system as a whole is considered. The initial cost of a fancy RAID HBA plus onboard cache plus battery backup (BBU or supercapacitor) can easily exceed more than 1000 US -dollars even after discounts - a sum that goes a log way toward SSD cost parity. +dollars even after discounts - a sum that goes a long way toward SSD cost parity. An HBA-free system may also cost hundreds of US dollars less every year if one purchases an annual maintenance contract or extended warranty. diff --git a/examples/rgw/boto3/bucket_logging.py b/examples/rgw/boto3/bucket_logging.py index fdc219c5765..7a972dac8bc 100644 --- a/examples/rgw/boto3/bucket_logging.py +++ b/examples/rgw/boto3/bucket_logging.py @@ -39,6 +39,17 @@ bucket_logging_conf = {'LoggingEnabled': { }, 'ObjectRollTime': 60, 'LoggingType': 'Journal', + "Filter": { + "Key": { + "FilterRules": + [ + { + "Name": "prefix", + "Value": "myfile" + } + ] + } + } } } diff --git a/examples/rgw/boto3/post_bucket_logging.py b/examples/rgw/boto3/post_bucket_logging.py new file mode 100644 index 00000000000..130fc53b50a --- /dev/null +++ b/examples/rgw/boto3/post_bucket_logging.py @@ -0,0 +1,23 @@ +import boto3 +import sys + + +if len(sys.argv) == 2: + # bucket name as first argument + bucketname = sys.argv[1] +else: + print('Usage: ' + sys.argv[0] + ' <bucket>') + sys.exit(1) + +# endpoint and keys from vstart +endpoint = 'http://127.0.0.1:8000/'+bucketname +access_key='0555b35654ad1656d804' +secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' + +client = boto3.client('s3', + endpoint_url=endpoint, + aws_access_key_id=access_key, + aws_secret_access_key=secret_key) + +# flushing the logs for bucket logging +print(client.post_bucket_logging(Bucket=bucketname)) diff --git a/examples/rgw/boto3/service-2.sdk-extras.json b/examples/rgw/boto3/service-2.sdk-extras.json index 5c22ee9f248..b81667ecd09 100644 --- a/examples/rgw/boto3/service-2.sdk-extras.json +++ b/examples/rgw/boto3/service-2.sdk-extras.json @@ -13,6 +13,17 @@ "documentationUrl":"https://docs.ceph.com/docs/master/radosgw/s3/bucketops/#delete-notification", "documentation":"<p>Deletes the notification configuration from the bucket.</p>" }, + "PostBucketLogging":{ + "name":"PostBucketLogging", + "http":{ + "method":"POST", + "requestUri":"/{Bucket}?logging", + "responseCode":201 + }, + "input":{"shape":"PostBucketLoggingRequest"}, + "documentationUrl":"https://docs.ceph.com/docs/master/radosgw/s3/bucketops/#post-bucket-logging", + "documentation":"<p>Flushes the logging objects of the buckets.</p>" + }, "GetUsageStats":{ "name":"GetUsageStats", "http":{ @@ -146,6 +157,18 @@ } } }, + "PostBucketLoggingRequest":{ + "type":"structure", + "required":["Bucket"], + "members":{ + "Bucket":{ + "shape":"BucketName", + "documentation":"<p>Name of the bucket to flush its logging objects.</p>", + "location":"uri", + "locationName":"Bucket" + } + } + }, "FilterRule":{ "type":"structure", "members":{ @@ -287,6 +310,10 @@ "RecordsBatchSize":{ "shape":"RecordsBatchSize", "documentation":"indicates how many records to batch in memory before writing to the object. if set to zero, records are written syncronously to the object. if <code>ObjectRollTime</code>e is reached, the batch of records will be written to the object regardless of the number of records. </p>" + }, + "Filter":{ + "shape":"LoggingConfigurationFilter", + "documentation":"<p>A filter for all log object. Filter for the object by its key (prefix, suffix and regex).</p>" } }, "documentation":"<p>Describes where logs are stored the prefix assigned to all log object keys for a bucket, and their format. also, the level the delivery guarantee of the records.</p>" @@ -340,6 +367,18 @@ "Standard", "Journal" ] + }, + "LoggingConfigurationFilter":{ + "type":"structure", + "members":{ + "Key":{ + "shape":"S3KeyFilter", + "documentation":"<p/>", + "locationName":"S3Key" + } + }, + "documentation":"<p>A filter for all log object. Filter for the object by its key (prefix, suffix and regex).</p>", + "locationName":"Filter" } }, "documentation":"<p/>" diff --git a/monitoring/ceph-mixin/config.libsonnet b/monitoring/ceph-mixin/config.libsonnet index a15b88422fc..e917b4c2dac 100644 --- a/monitoring/ceph-mixin/config.libsonnet +++ b/monitoring/ceph-mixin/config.libsonnet @@ -9,12 +9,12 @@ CephNodeNetworkPacketDropsPerSec: 10, CephRBDMirrorImageTransferBandwidthThreshold: 0.8, CephRBDMirrorImagesPerDaemonThreshold: 100, - NVMeoFMaxGatewaysPerGroup: 4, - NVMeoFMaxGatewaysPerCluster: 4, + NVMeoFMaxGatewaysPerGroup: 8, + NVMeoFMaxGatewaysPerCluster: 32, NVMeoFHighGatewayCPU: 80, NVMeoFMaxSubsystemsPerGateway: 128, - NVMeoFMaxNamespaces: 1024, - NVMeoFHighClientCount: 32, + NVMeoFMaxNamespaces: 2048, + NVMeoFHighClientCount: 128, NVMeoFHighHostCPU: 80, // // Read/Write latency is defined in ms diff --git a/monitoring/ceph-mixin/dashboards/rgw.libsonnet b/monitoring/ceph-mixin/dashboards/rgw.libsonnet index 79a4b7a14eb..c0c548b79c8 100644 --- a/monitoring/ceph-mixin/dashboards/rgw.libsonnet +++ b/monitoring/ceph-mixin/dashboards/rgw.libsonnet @@ -298,7 +298,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_get_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_get_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -314,7 +314,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_put_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_put_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -331,7 +331,7 @@ local g = import 'grafonnet/grafana.libsonnet'; sum by (rgw_host) ( label_replace( rate(ceph_rgw_req{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ) @@ -351,7 +351,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_get_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_get_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -385,7 +385,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace(sum by (instance_id) ( rate(ceph_rgw_op_get_obj_bytes{%(matchers)s}[$__rate_interval]) + rate(ceph_rgw_op_put_obj_bytes{%(matchers)s}[$__rate_interval])) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -404,7 +404,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_put_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_put_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), diff --git a/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json b/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json index 5e185b63b7f..5bf8279c27c 100644 --- a/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json +++ b/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json @@ -108,14 +108,14 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "GET {{rgw_host}}", "refId": "A" }, { - "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "PUT {{rgw_host}}", @@ -210,7 +210,7 @@ "steppedLine": false, "targets": [ { - "expr": "sum by (rgw_host) (\n label_replace(\n rate(ceph_rgw_req{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n )\n)\n", + "expr": "sum by (rgw_host) (\n label_replace(\n rate(ceph_rgw_req{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n )\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -305,7 +305,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -502,7 +502,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(sum by (instance_id) (\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval]) +\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval])) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(sum by (instance_id) (\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval]) +\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval])) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -597,7 +597,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", diff --git a/monitoring/ceph-mixin/prometheus_alerts.libsonnet b/monitoring/ceph-mixin/prometheus_alerts.libsonnet index 143e65f20e7..5d1ab49b533 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.libsonnet +++ b/monitoring/ceph-mixin/prometheus_alerts.libsonnet @@ -856,6 +856,16 @@ }, }, { + alert: 'NVMeoFMultipleNamespacesOfRBDImage', + 'for': '1m', + expr: 'count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1', + labels: { severity: 'warning', type: 'ceph_default' }, + annotations: { + summary: 'RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace ', + description: 'Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups.', + }, + }, + { alert: 'NVMeoFTooManyGateways', 'for': '1m', expr: 'count(ceph_nvmeof_gateway_info) by (cluster) > %.2f' % [$._config.NVMeoFMaxGatewaysPerCluster], diff --git a/monitoring/ceph-mixin/prometheus_alerts.yml b/monitoring/ceph-mixin/prometheus_alerts.yml index 3eb8a8db4fa..7c0da4d51a4 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.yml +++ b/monitoring/ceph-mixin/prometheus_alerts.yml @@ -765,20 +765,29 @@ groups: labels: severity: "warning" type: "ceph_default" + - alert: "NVMeoFMultipleNamespacesOfRBDImage" + annotations: + description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups." + summary: "RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace " + expr: "count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1" + for: "1m" + labels: + severity: "warning" + type: "ceph_default" - alert: "NVMeoFTooManyGateways" annotations: - description: "You may create many gateways, but 4 is the tested limit" + description: "You may create many gateways, but 32 is the tested limit" summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}" - expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 4.00" + expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 32.00" for: "1m" labels: severity: "warning" type: "ceph_default" - alert: "NVMeoFMaxGatewayGroupSize" annotations: - description: "You may create many gateways in a gateway group, but 4 is the tested limit" + description: "You may create many gateways in a gateway group, but 8 is the tested limit" summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}" - expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 4.00" + expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 8.00" for: "1m" labels: severity: "warning" @@ -823,7 +832,7 @@ groups: annotations: description: "Although you may continue to create namespaces in {{ $labels.gateway_host }}, the configuration may not be supported" summary: "The number of namespaces defined to the gateway exceeds supported values on cluster {{ $labels.cluster }}" - expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\"gateway_host\",\"$1\",\"instance\",\"(.*?)(?::.*)?\")) > 1024.00" + expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\"gateway_host\",\"$1\",\"instance\",\"(.*?)(?::.*)?\")) > 2048.00" for: "1m" labels: severity: "warning" @@ -839,9 +848,9 @@ groups: type: "ceph_default" - alert: "NVMeoFHighClientCount" annotations: - description: "The supported limit for clients connecting to a subsystem is 32" + description: "The supported limit for clients connecting to a subsystem is 128" summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}" - expr: "ceph_nvmeof_subsystem_host_count > 32.00" + expr: "ceph_nvmeof_subsystem_host_count > 128.00" for: "1m" labels: severity: "warning" diff --git a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml index a4e63bbcf73..83b4ff80375 100644 --- a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml +++ b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml @@ -2270,6 +2270,54 @@ tests: summary: "wah subsystem has reached its maximum number of namespaces on cluster mycluster" description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to wah" +# NVMeoFMultipleNamespacesOfRBDImage + - interval: 1m + input_series: + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev1", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev1", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev2", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage2"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev2", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage2"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev3", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev3", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev4", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' # bdev with no ns + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="1", bdev_name="bdev1", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="1", bdev_name="bdev1", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="2", bdev_name="bdev2", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="2", bdev_name="bdev2", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn2", nsid="1", bdev_name="bdev3", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn2", nsid="1", bdev_name="bdev3", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + promql_expr_test: + - expr: count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1 + eval_time: 1m + exp_samples: + - labels: '{pool_name="mypool", rbd_name="myimage1"}' + value: 2 + alert_rule_test: + - eval_time: 5m + alertname: NVMeoFMultipleNamespacesOfRBDImage + exp_alerts: + - exp_labels: + pool_name: mypool + rbd_name: myimage1 + severity: warning + type: ceph_default + exp_annotations: + summary: "RBD image mypool/myimage1 cannot be reused for multiple NVMeoF namespace " + description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups." + # NVMeoFTooManyGateways - interval: 1m input_series: @@ -2283,12 +2331,69 @@ tests: values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.5",cluster="mycluster"}' values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.6",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.7",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.8",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.9",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.10",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.11",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.12",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.13",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.14",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.15",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.16",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.17",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.18",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.19",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.20",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.21",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.22",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.23",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.24",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.25",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.26",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.27",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.28",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.29",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.30",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.31",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.32",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.33",cluster="mycluster"}' + values: '1+0x20' + promql_expr_test: - - expr: count(ceph_nvmeof_gateway_info) by (cluster) > 4.00 + - expr: count(ceph_nvmeof_gateway_info) by (cluster) > 32.00 eval_time: 1m exp_samples: - labels: '{cluster="mycluster"}' - value: 5 + value: 33 alert_rule_test: - eval_time: 5m alertname: NVMeoFTooManyGateways @@ -2299,7 +2404,7 @@ tests: type: ceph_default exp_annotations: summary: "Max supported gateways exceeded on cluster mycluster" - description: "You may create many gateways, but 4 is the tested limit" + description: "You may create many gateways, but 32 is the tested limit" # NVMeoFMaxGatewayGroupSize - interval: 1m @@ -2314,16 +2419,24 @@ tests: values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.12",cluster="mycluster"}' values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.10",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.14",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.11",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.13",cluster="mycluster"}' + values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4",cluster="mycluster"}' values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5",cluster="mycluster"}' values: '1+0x20' promql_expr_test: - - expr: count(ceph_nvmeof_gateway_info) by (cluster, group) > 4.00 + - expr: count(ceph_nvmeof_gateway_info) by (cluster, group) > 8.00 eval_time: 1m exp_samples: - labels: '{cluster="mycluster",group="group-1"}' - value: 5 + value: 9 alert_rule_test: - eval_time: 5m alertname: NVMeoFMaxGatewayGroupSize @@ -2335,7 +2448,7 @@ tests: type: ceph_default exp_annotations: summary: "Max gateways within a gateway group (group-1) exceeded on cluster mycluster" - description: "You may create many gateways in a gateway group, but 4 is the tested limit" + description: "You may create many gateways in a gateway group, but 8 is the tested limit" # NVMeoFSingleGatewayGroup - interval: 1m @@ -2719,12 +2832,14 @@ tests: values: '200+0x10' - series: 'ceph_nvmeof_subsystem_namespace_count{instance="node-1:10008",nqn="nqn10",cluster="mycluster"}' values: '200+0x10' + - series: 'ceph_nvmeof_subsystem_namespace_count{instance="node-1:10008",nqn="nqn11",cluster="mycluster"}' + values: '200+0x10' promql_expr_test: - - expr: sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,"gateway_host","$1","instance","(.*):.*")) > 1024 + - expr: sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,"gateway_host","$1","instance","(.*):.*")) > 2048 eval_time: 1m exp_samples: - labels: '{gateway_host="node-1", cluster="mycluster"}' - value: 2000 + value: 2200 alert_rule_test: - eval_time: 5m alertname: NVMeoFTooManyNamespaces @@ -2767,15 +2882,15 @@ tests: - interval: 1m input_series: - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn1",cluster="mycluster"}' - values: '2 2 2 4 4 8 8 8 10 10 20 20 32 34 34 38 38 40 44 44' + values: '2 4 8 10 20 30 40 50 62 74 80 95 100 110 130 130 130 130 130 130' - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn2",cluster="mycluster"}' - values: '2 2 2 8 8 8 16 16 16 16 16 16 16 16 16 16 16 16 16 16' + values: '2 8 16 16 16 16 16 16 16 16 20 20 32 34 34 36 37 37 37 37' promql_expr_test: - - expr: ceph_nvmeof_subsystem_host_count > 32.00 + - expr: ceph_nvmeof_subsystem_host_count > 128.00 eval_time: 15m exp_samples: - labels: '{__name__="ceph_nvmeof_subsystem_host_count",nqn="nqn1",cluster="mycluster"}' - value: 38 + value: 130 alert_rule_test: - eval_time: 20m alertname: NVMeoFHighClientCount @@ -2787,7 +2902,7 @@ tests: type: ceph_default exp_annotations: summary: "The number of clients connected to nqn1 is too high on cluster mycluster" - description: "The supported limit for clients connecting to a subsystem is 32" + description: "The supported limit for clients connecting to a subsystem is 128" # NVMeoFMissingListener - interval: 1m diff --git a/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature b/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature index 8d96dcdd610..a34d5759437 100644 --- a/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature +++ b/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature @@ -7,6 +7,7 @@ Scenario: "Test Average GET Latencies" | ceph_rgw_op_get_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 20 60 80 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `Average GET/PUT Latencies by RGW Instance` with legend `GET {{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1.5 | @@ -18,6 +19,7 @@ Scenario: "Test Average PUT Latencies" | ceph_rgw_op_put_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 10 30 50 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `Average GET/PUT Latencies by RGW Instance` with legend `PUT {{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1 | @@ -28,6 +30,7 @@ Scenario: "Test Total Requests/sec by RGW Instance" | ceph_rgw_req{instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 10 50 100 | | ceph_rgw_metadata{ceph_daemon="rgw.1", hostname="localhost", instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.1` Then Grafana panel `Total Requests/sec by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {rgw_host="1"} | 1.5 | @@ -39,6 +42,7 @@ Scenario: "Test GET Latencies by RGW Instance" | ceph_rgw_op_get_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 20 60 80 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `GET Latencies by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1.5 | @@ -71,6 +75,7 @@ Scenario: "Test Bandwidth by RGW Instance" | ceph_rgw_metadata{ceph_daemon="rgw.1", hostname="localhost", instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 1 1 1 | When evaluation time is `1m` And interval is `30s` + And variable `rgw_servers` is `rgw.1` Then Grafana panel `Bandwidth by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.1", instance_id="92806566", rgw_host="1"} | 2.25 | @@ -83,6 +88,7 @@ Scenario: "Test PUT Latencies by RGW Instance" | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When evaluation time is `1m` And interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `PUT Latencies by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1 | diff --git a/qa/config/crimson_bluestore.yaml b/qa/config/crimson_bluestore.yaml new file mode 100644 index 00000000000..d5ba487b9bf --- /dev/null +++ b/qa/config/crimson_bluestore.yaml @@ -0,0 +1,25 @@ +overrides: + ceph: + fs: xfs + conf: + osd: + # crimson's osd objectstore option + crimson osd objectstore: bluestore + debug alienstore: 20 + bluestore block size: 96636764160 + debug bluestore: 20 + debug bluefs: 20 + debug rocksdb: 10 + bluestore compression mode: aggressive + bluestore fsck on mount: true + bluestore compression algorithm: snappy + # lower the full ratios since we can fill up a 100gb osd so quickly + mon osd full ratio: .9 + mon osd backfillfull_ratio: .85 + mon osd nearfull ratio: .8 + osd failsafe full ratio: .95 + bluestore rocksdb cf: false + log to stderr: true + err to stderr: true + log flush on exit: true + log to file: false diff --git a/qa/config/crimson_qa_overrides.yaml b/qa/config/crimson_qa_overrides.yaml index fa8f49a4986..a10c59d77cc 100644 --- a/qa/config/crimson_qa_overrides.yaml +++ b/qa/config/crimson_qa_overrides.yaml @@ -9,6 +9,7 @@ overrides: osd pool default crimson: true osd: crimson osd obc lru size: 10 + debug ms: 20 flavor: crimson workunit: env: diff --git a/qa/config/seastore.yaml b/qa/config/crimson_seastore.yaml index 6158563eedf..d1919456ab1 100644 --- a/qa/config/seastore.yaml +++ b/qa/config/crimson_seastore.yaml @@ -1,13 +1,13 @@ overrides: ceph: - fs: xfs conf: osd: - osd objectstore: seastore + # crimson's osd objectstore option + crimson osd objectstore: seastore debug seastore: 20 debug seastore onode: 20 debug seastore odata: 20 - debug seastore ompap: 20 + debug seastore omap: 20 debug seastore tm: 20 debug seastore t: 20 debug seastore cleaner: 20 diff --git a/qa/crontab/teuthology-cronjobs b/qa/crontab/teuthology-cronjobs index c979e5b105f..c558a1382ef 100644 --- a/qa/crontab/teuthology-cronjobs +++ b/qa/crontab/teuthology-cronjobs @@ -52,7 +52,6 @@ TEUTHOLOGY_SUITE_ARGS="--non-interactive --newest=100 --ceph-repo=https://git.ce 00 05 * * 0,2,4 $CW $SS 1 --ceph main --suite smoke -p 100 --force-priority 08 05 * * 0 $CW $SS 1 --ceph squid --suite smoke -p 100 --force-priority 16 05 * * 0 $CW $SS 1 --ceph reef --suite smoke -p 100 --force-priority -24 05 * * 0 $CW $SS 1 --ceph quincy --suite smoke -p 100 --force-priority ## ********** windows tests on main branch - weekly # 00 03 * * 1 CEPH_BRANCH=main; MACHINE_NAME=smithi; $CW teuthology-suite -v -c $CEPH_BRANCH -n 100 -m $MACHINE_NAME -s windows -k distro -e $CEPH_QA_EMAIL @@ -122,7 +121,6 @@ TEUTHOLOGY_SUITE_ARGS="--non-interactive --newest=100 --ceph-repo=https://git.ce 16 00 * * 1 $CW $SS 1 --ceph quincy --suite upgrade-clients/client-upgrade-pacific-quincy --suite-branch pacific -p 820 24 00 * * 1 $CW $SS 120000 --ceph quincy --suite upgrade:octopus-x -p 820 32 00 * * 1 $CW $SS 120000 --ceph quincy --suite upgrade:pacific-x -p 820 -40 00 * * 1 $CW $SS 1 --ceph quincy --suite upgrade/quincy-p2p -p 820 ### upgrade runs for reef release ###### on smithi diff --git a/qa/standalone/scrub/osd-scrub-repair.sh b/qa/standalone/scrub/osd-scrub-repair.sh index 491e46603f7..6dd5b10ae8f 100755 --- a/qa/standalone/scrub/osd-scrub-repair.sh +++ b/qa/standalone/scrub/osd-scrub-repair.sh @@ -5833,7 +5833,7 @@ function TEST_periodic_scrub_replicated() { flush_pg_stats # Request a regular scrub and it will be done - pg_schedule_scrub $pg + pg_scrub $pg grep -q "Regular scrub request, deep-scrub details will be lost" $dir/osd.${primary}.log || return 1 # deep-scrub error is no longer present diff --git a/qa/standalone/scrub/osd-scrub-test.sh b/qa/standalone/scrub/osd-scrub-test.sh index 8015e023bdd..85a45a421c1 100755 --- a/qa/standalone/scrub/osd-scrub-test.sh +++ b/qa/standalone/scrub/osd-scrub-test.sh @@ -544,6 +544,9 @@ function TEST_dump_scrub_schedule() { --osd_op_queue=wpq \ --osd_stats_update_period_not_scrubbing=1 \ --osd_stats_update_period_scrubbing=1 \ + --osd_scrub_retry_after_noscrub=1 \ + --osd_scrub_retry_pg_state=2 \ + --osd_scrub_retry_delay=2 \ --osd_scrub_sleep=0.2" for osd in $(seq 0 $(expr $OSDS - 1)) @@ -683,6 +686,234 @@ function TEST_pg_dump_objects_scrubbed() { teardown $dir || return 1 } +function wait_initial_scrubs() { + local -n pg_to_prim_dict=$1 + local extr_dbg=1 # note: 3 and above leave some temp files around + + # set a long schedule for the periodic scrubs. Wait for the + # initial 'no previous scrub is known' scrubs to finish for all PGs. + bin/ceph tell osd.* config set osd_scrub_min_interval 7200 + bin/ceph tell osd.* config set osd_deep_scrub_interval 14400 + bin/ceph tell osd.* config set osd_max_scrubs 32 + bin/ceph tell osd.* config set osd_scrub_sleep 0 + bin/ceph tell osd.* config set osd_shallow_scrub_chunk_max 10 + bin/ceph tell osd.* config set osd_scrub_chunk_max 10 + + for pg in "${!pg_to_prim_dict[@]}"; do + (( extr_dbg >= 1 )) && echo "Scheduling initial scrub for $pg" + bin/ceph tell $pg scrub || return 1 + done + + sleep 1 + (( extr_dbg >= 1 )) && bin/ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' + + tout=20 + while [ $tout -gt 0 ] ; do + sleep 0.5 + (( extr_dbg >= 2 )) && bin/ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' + not_done=$(bin/ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' | wc -l ) + # note that we should ignore a header line + if [ "$not_done" -le 1 ]; then + break + fi + not_done=$(( (not_done - 2) / 4 )) + echo "Still waiting for $not_done PGs to finish initial scrubs (timeout $tout)" + tout=$((tout - 1)) + done + (( tout == 0 )) && return 1 + return 0 +} + + +# Whenever a PG is being scrubbed at a regular, periodic, urgency, and is queued +# for its replicas: +# if the operator is requesting a scrub of the same PG, the operator's request +# should trigger an abort of the ongoing scrub. +# +# The test process: +# - a periodic scrub is initiated of a PG. That scrub is set to be a very slow one. +# - a second PG, which shares some of its replicas, is intrcuted to be scrubbed. That one +# should be stuck in replica reservation. We will verify that. +# - now - the operator is requesting that second PG to be scrubbed. The original (pending) +# scrub should be aborted. We would check for: +# - the new, operator's scrub to be scheduled +# - the replicas' reservers to be released +function TEST_abort_periodic_for_operator() { + local dir=$1 + local -A cluster_conf=( + ['osds_num']="5" + ['pgs_in_pool']="16" + ['pool_name']="test" + ) + local extr_dbg=1 # note: 3 and above leave some temp files around + + standard_scrub_wpq_cluster "$dir" cluster_conf 3 || return 1 + local poolid=${cluster_conf['pool_id']} + local poolname=${cluster_conf['pool_name']} + echo "Pool: $poolname : $poolid" + + #turn off '-x' (but remember previous state) + local saved_echo_flag=${-//[^x]/} + set +x + + # fill the pool with some data + TESTDATA="testdata.$$" + dd if=/dev/urandom of=$TESTDATA bs=320 count=1 + for i in $( seq 1 256 ) + do + rados -p "$poolname" put "obj${i}" $TESTDATA 2>/dev/null 1>/dev/null + done + rm -f $TESTDATA + if [[ -n "$saved_echo_flag" ]]; then set -x; fi + + # create the dictionary of the PGs in the pool + declare -A pg_pr + declare -A pg_ac + declare -A pg_po + build_pg_dicts "$dir" pg_pr pg_ac pg_po "-" + (( extr_dbg >= 2 )) && echo "PGs table:" + for pg in "${!pg_pr[@]}"; do + (( extr_dbg >= 2 )) && echo "Got: $pg: ${pg_pr[$pg]} ( ${pg_ac[$pg]} ) ${pg_po[$pg]}" + done + + wait_initial_scrubs pg_pr || return 1 + + # limit all OSDs to one scrub at a time + bin/ceph tell osd.* config set osd_max_scrubs 1 + bin/ceph tell osd.* config set osd_stats_update_period_not_scrubbing 1 + + # configure for slow scrubs + bin/ceph tell osd.* config set osd_scrub_sleep 3 + bin/ceph tell osd.* config set osd_shallow_scrub_chunk_max 2 + bin/ceph tell osd.* config set osd_scrub_chunk_max 2 + (( extr_dbg >= 2 )) && bin/ceph tell osd.2 dump_scrub_reservations --format=json-pretty + + # the first PG to work with: + local pg1="1.0" + # and another one, that shares its primary, and at least one more active set member + local pg2="" + for pg in "${!pg_pr[@]}"; do + if [[ "${pg_pr[$pg]}" == "${pg_pr[$pg1]}" ]]; then + local -i common=0 + count_common_active $pg $pg1 pg_ac common + if [[ $common -gt 1 ]]; then + pg2=$pg + break + fi + fi + done + if [[ -z "$pg2" ]]; then + # \todo handle the case when no such PG is found + echo "No PG found with the same primary as $pg1" + return 1 + fi + + # the common primary is allowed two concurrent scrubs + bin/ceph tell osd."${pg_pr[$pg1]}" config set osd_max_scrubs 2 + echo "The two PGs to manipulate are $pg1 and $pg2" + + set_query_debug "$pg1" + # wait till the information published by pg1 is updated to show it as + # not being scrubbed + local is_act + for i in $( seq 1 3 ) + do + is_act=$(bin/ceph pg "$pg1" query | jq '.scrubber.active') + if [[ "$is_act" = "false" ]]; then + break + fi + echo "Still waiting for pg $pg1 to finish scrubbing" + sleep 0.7 + done + bin/ceph pg dump pgs + if [[ "$is_act" != "false" ]]; then + bin/ceph pg "$pg1" query + echo "PG $pg1 appears to be still scrubbing" + return 1 + fi + sleep 0.5 + + echo "Initiating a periodic scrub of $pg1" + (( extr_dbg >= 2 )) && bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + bin/ceph tell $pg1 schedule-deep-scrub || return 1 + sleep 1 + (( extr_dbg >= 2 )) && bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + for i in $( seq 1 14 ) + do + sleep 0.5 + stt=$(bin/ceph pg "$pg1" query | jq '.scrubber') + is_active=$(echo $stt | jq '.active') + is_reserving_replicas=$(echo $stt | jq '.is_reserving_replicas') + if [[ "$is_active" = "true" && "$is_reserving_replicas" = "false" ]]; then + break + fi + echo "Still waiting for pg $pg1 to start scrubbing: $stt" + done + if [[ "$is_active" != "true" || "$is_reserving_replicas" != "false" ]]; then + bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + echo "The scrub is not active or is reserving replicas" + return 1 + fi + (( extr_dbg >= 2 )) && bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + + # PG 1 is scrubbing, and has reserved the replicas - soem of which are shared + # by PG 2. As the max-scrubs was set to 1, that should prevent PG 2 from + # reserving its replicas. + + (( extr_dbg >= 1 )) && bin/ceph tell osd.* dump_scrub_reservations --format=json-pretty + + # now - the 2'nd scrub - which should be blocked on reserving + set_query_debug "$pg2" + bin/ceph tell "$pg2" schedule-deep-scrub + sleep 0.5 + (( extr_dbg >= 2 )) && echo "====================================================================================" + (( extr_dbg >= 2 )) && bin/ceph pg "$pg2" query -f json-pretty | jq '.scrubber' + (( extr_dbg >= 2 )) && bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + sleep 1 + (( extr_dbg >= 2 )) && echo "====================================================================================" + (( extr_dbg >= 2 )) && bin/ceph pg "$pg2" query -f json-pretty | jq '.scrubber' + (( extr_dbg >= 2 )) && bin/ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + # make sure pg2 scrub is stuck in the reserving state + local stt2=$(bin/ceph pg "$pg2" query | jq '.scrubber') + local pg2_is_reserving + pg2_is_reserving=$(echo $stt2 | jq '.is_reserving_replicas') + if [[ "$pg2_is_reserving" != "true" ]]; then + echo "The scheduled scrub for $pg2 should have been stuck" + bin/ceph pg dump pgs + return 1 + fi + + # now - issue an operator-initiated scrub on pg2. + # The periodic scrub should be aborted, and the operator-initiated scrub should start. + echo "Instructing $pg2 to perform a high-priority scrub" + bin/ceph tell "$pg2" scrub + for i in $( seq 1 10 ) + do + sleep 0.5 + stt2=$(bin/ceph pg "$pg2" query | jq '.scrubber') + pg2_is_active=$(echo $stt2 | jq '.active') + pg2_is_reserving=$(echo $stt2 | jq '.is_reserving_replicas') + if [[ "$pg2_is_active" = "true" && "$pg2_is_reserving" != "true" ]]; then + break + fi + echo "Still waiting: $stt2" + done + + if [[ "$pg2_is_active" != "true" || "$pg2_is_reserving" = "true" ]]; then + echo "The high-priority scrub for $pg2 is not active or is reserving replicas" + return 1 + fi + echo "Done" +} + + + main osd-scrub-test "$@" # Local Variables: diff --git a/qa/standalone/scrub/scrub-helpers.sh b/qa/standalone/scrub/scrub-helpers.sh index 49b8346b8d2..0b14d6028b6 100644 --- a/qa/standalone/scrub/scrub-helpers.sh +++ b/qa/standalone/scrub/scrub-helpers.sh @@ -240,8 +240,8 @@ function standard_scrub_cluster() { local saved_echo_flag=${-//[^x]/} set +x - run_mon $dir a --osd_pool_default_size=$OSDS || return 1 - run_mgr $dir x || return 1 + run_mon $dir a --osd_pool_default_size=3 || return 1 + run_mgr $dir x --mgr_stats_period=1 || return 1 local ceph_osd_args="--osd_deep_scrub_randomize_ratio=0 \ --osd_scrub_interval_randomize_ratio=0 \ @@ -249,9 +249,12 @@ function standard_scrub_cluster() { --osd_pool_default_pg_autoscale_mode=off \ --osd_pg_stat_report_interval_max_seconds=1 \ --osd_pg_stat_report_interval_max_epochs=1 \ + --osd_stats_update_period_not_scrubbing=3 \ + --osd_stats_update_period_scrubbing=1 \ --osd_scrub_retry_after_noscrub=5 \ --osd_scrub_retry_pg_state=5 \ --osd_scrub_retry_delay=3 \ + --osd_pool_default_size=3 \ $extra_pars" for osd in $(seq 0 $(expr $OSDS - 1)) @@ -297,6 +300,107 @@ function standard_scrub_wpq_cluster() { } +# Parse the output of a 'pg dump pgs_brief' command and build a set of dictionaries: +# - pg_primary_dict: a dictionary of pgid -> acting_primary +# - pg_acting_dict: a dictionary of pgid -> acting set +# - pg_pool_dict: a dictionary of pgid -> pool +# If the input file is '-', the function will fetch the dump directly from the ceph cluster. +function build_pg_dicts { + local dir=$1 + local -n pg_primary_dict=$2 + local -n pg_acting_dict=$3 + local -n pg_pool_dict=$4 + local infile=$5 + + local extr_dbg=0 # note: 3 and above leave some temp files around + + #turn off '-x' (but remember previous state) + local saved_echo_flag=${-//[^x]/} + set +x + + # if the infile name is '-', fetch the dump directly from the ceph cluster + if [[ $infile == "-" ]]; then + local -r ceph_cmd="bin/ceph pg dump pgs_brief -f=json-pretty" + local -r ceph_cmd_out=$(eval $ceph_cmd) + local -r ceph_cmd_rc=$? + if [[ $ceph_cmd_rc -ne 0 ]]; then + echo "Error: the command '$ceph_cmd' failed with return code $ceph_cmd_rc" + fi + (( extr_dbg >= 3 )) && echo "$ceph_cmd_out" > /tmp/e2 + l0=`echo "$ceph_cmd_out" | jq '[.pg_stats | group_by(.pg_stats)[0] | map({pgid: .pgid, pool: (.pgid | split(".")[0]), acting: .acting, acting_primary: .acting_primary})] | .[]' ` + else + l0=`jq '[.pg_stats | group_by(.pg_stats)[0] | map({pgid: .pgid, pool: (.pgid | split(".")[0]), acting: .acting, acting_primary: .acting_primary})] | .[]' $infile ` + fi + (( extr_dbg >= 2 )) && echo "L0: $l0" + + mapfile -t l1 < <(echo "$l0" | jq -c '.[]') + (( extr_dbg >= 2 )) && echo "L1: ${#l1[@]}" + + for item in "${l1[@]}"; do + pgid=$(echo "$item" | jq -r '.pgid') + acting=$(echo "$item" | jq -r '.acting | @sh') + pg_acting_dict["$pgid"]=$acting + acting_primary=$(echo "$item" | jq -r '.acting_primary') + pg_primary_dict["$pgid"]=$acting_primary + pool=$(echo "$item" | jq -r '.pool') + pg_pool_dict["$pgid"]=$pool + done + + if [[ -n "$saved_echo_flag" ]]; then set -x; fi +} + + +# a function that counts the number of common active-set elements between two PGs +# 1 - the first PG +# 2 - the second PG +# 3 - the dictionary of active sets +function count_common_active { + local pg1=$1 + local pg2=$2 + local -n pg_acting_dict=$3 + local -n res=$4 + + local -a a1=(${pg_acting_dict[$pg1]}) + local -a a2=(${pg_acting_dict[$pg2]}) + + local -i cnt=0 + for i in "${a1[@]}"; do + for j in "${a2[@]}"; do + if [[ $i -eq $j ]]; then + cnt=$((cnt+1)) + fi + done + done + + res=$cnt +} + + +# given a PG, find another one with a disjoint active set +# - but allow a possible common Primary +# 1 - the PG +# 2 - the dictionary of active sets +# 3 - [out] - the PG with a disjoint active set +function find_disjoint_but_primary { + local pg=$1 + local -n ac_dict=$2 + local -n p_dict=$3 + local -n res=$4 + + for cand in "${!ac_dict[@]}"; do + if [[ "$cand" != "$pg" ]]; then + local -i common=0 + count_common_active "$pg" "$cand" ac_dict common + if [[ $common -eq 0 || ( $common -eq 1 && "${p_dict[$pg]}" == "${p_dict[$cand]}" )]]; then + res=$cand + return + fi + fi + done +} + + + # A debug flag is set for the PG specified, causing the 'pg query' command to display # an additional 'scrub sessions counter' field. # diff --git a/qa/suites/crimson-rados-experimental/.qa b/qa/suites/crimson-rados-experimental/.qa index fea2489fdf6..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/.qa +++ b/qa/suites/crimson-rados-experimental/.qa @@ -1 +1 @@ -../.qa
\ No newline at end of file +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml deleted file mode 120000 index bd9854e7029..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/distros/supported/centos_latest.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml deleted file mode 100644 index d8e5898b99f..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml +++ /dev/null @@ -1,14 +0,0 @@ -overrides: - ceph-deploy: - conf: - global: - osd pool default size: 2 - osd crush chooseleaf type: 0 - osd pool default pg num: 128 - osd pool default pgp num: 128 - ceph: - conf: - osd: - osd shutdown pgref assert: true -roles: -- [mon.a, mgr.x, osd.0, osd.1, osd.2, client.0] diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml deleted file mode 100644 index c22f08eecf8..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml +++ /dev/null @@ -1,18 +0,0 @@ -overrides: - install: - ceph: - flavor: crimson -tasks: -- install: -- ceph: - conf: - osd: - debug monc: 20 - mon: - mon min osdmap epochs: 50 - paxos service trim min: 10 - # prune full osdmaps regularly - mon osdmap full prune min: 15 - mon osdmap full prune interval: 2 - mon osdmap full prune txsize: 2 - flavor: crimson diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml deleted file mode 120000 index 6a70c381709..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/config/seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml deleted file mode 100644 index ad8c921425b..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml +++ /dev/null @@ -1,28 +0,0 @@ -overrides: - ceph: - log-ignorelist: - - reached quota - - but it is still running - - overall HEALTH_ - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(CACHE_POOL_NO_HIT_SET\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(PG_DEGRADED\) - conf: - client: - debug ms: 1 - mon: - mon warn on pool no app: false - osd: - osd class load list: "*" - osd class default list: "*" - osd blocked scrub grace period: 3600 -tasks: -- workunit: - clients: - client.0: - - rados/test.sh - - rados/test_pool_quota.sh diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml deleted file mode 100644 index 25efcdac83d..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml +++ /dev/null @@ -1,18 +0,0 @@ -overrides: - ceph: - crush_tunables: optimal - conf: - mon: - mon osd initial require min compat client: luminous - osd: - osd_discard_disconnected_ops: false -tasks: -- rados: - clients: [client.0] - ops: 4000 - objects: 500 - max_attr_len: 8192 - op_weights: - read: 45 - write: 45 - delete: 10 diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/% b/qa/suites/crimson-rados-experimental/thrash/% index e69de29bb2d..e69de29bb2d 100644 --- a/qa/suites/crimson-rados-experimental/seastore/basic/% +++ b/qa/suites/crimson-rados-experimental/thrash/% diff --git a/qa/suites/crimson-rados-experimental/seastore/.qa b/qa/suites/crimson-rados-experimental/thrash/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/.qa diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/.qa b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled new file mode 120000 index 00000000000..5393a75548a --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled @@ -0,0 +1 @@ +.qa/overrides/2-size-2-min-size.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml new file mode 120000 index 00000000000..5ff70eadf75 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml @@ -0,0 +1 @@ +.qa/overrides/3-size-2-min-size.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/.qa b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml diff --git a/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/short_pg_log.yaml index abd86d7d986..abd86d7d986 120000 --- a/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml.disabled +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/short_pg_log.yaml diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/.qa b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled new file mode 120000 index 00000000000..47afd70202d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled @@ -0,0 +1 @@ +.qa/overrides/more-active-recovery.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled new file mode 100644 index 00000000000..0bbc72db754 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled @@ -0,0 +1,6 @@ +overrides: + ceph: + conf: + global: + osd_async_recovery_min_cost: 1 + osd_object_clean_region_max_num_intervals: 1000 diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled new file mode 100644 index 00000000000..4aed086bcc3 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + global: + osd_async_recovery_min_cost: 1 diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled new file mode 100644 index 00000000000..88f15f2f691 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + global: + osd_object_clean_region_max_num_intervals: 1000 diff --git a/qa/suites/crimson-rados-experimental/thrash/clusters/+ b/qa/suites/crimson-rados-experimental/thrash/clusters/+ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/+ diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/.qa b/qa/suites/crimson-rados-experimental/thrash/clusters/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/.qa diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-2.yaml b/qa/suites/crimson-rados-experimental/thrash/clusters/fixed-2.yaml index 9774de6887b..79641f695ab 100644 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-2.yaml +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/fixed-2.yaml @@ -6,6 +6,15 @@ overrides: conf: osd: osd shutdown pgref assert: true + crimson alien thread cpu cores: 6-7 + osd.0: + crimson seastar cpu cores: 0-2 + osd.1: + crimson seastar cpu cores: 3-5 + osd.2: + crimson seastar cpu cores: 0-2 + osd.3: + crimson seastar cpu cores: 3-5 global: ms cluster mode: crc ms service mode: crc diff --git a/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled new file mode 100644 index 00000000000..e559d9126e8 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled @@ -0,0 +1,4 @@ +openstack: + - volumes: # attached to each instance + count: 4 + size: 10 # GB diff --git a/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro b/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro new file mode 120000 index 00000000000..a5b729b9efa --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro @@ -0,0 +1 @@ +.qa/distros/crimson-supported-all-distro/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/crimson_qa_overrides.yaml b/qa/suites/crimson-rados-experimental/thrash/crimson_qa_overrides.yaml index 2bf67af1b18..2bf67af1b18 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/crimson_qa_overrides.yaml +++ b/qa/suites/crimson-rados-experimental/thrash/crimson_qa_overrides.yaml diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/.qa b/qa/suites/crimson-rados-experimental/thrash/deploy/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml b/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml new file mode 100644 index 00000000000..ecad09cfe3a --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml @@ -0,0 +1,11 @@ +overrides: + install: + ceph: + flavor: crimson +tasks: +- install: +- ceph: + conf: + osd: + debug monc: 20 + flavor: crimson diff --git a/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled new file mode 100644 index 00000000000..0c2062240ee --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled @@ -0,0 +1,16 @@ +# no need to verify os + flavor + sha1 +verify_ceph_hash: false +tasks: +- cephadm: + conf: + mgr: + debug ms: 1 + debug mgr: 20 + debug osd: 10 +- cephadm.shell: + mon.a: + - ceph orch status + - ceph orch ps + - ceph orch ls + - ceph orch host ls + - ceph orch device ls diff --git a/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa b/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml b/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa b/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml b/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml new file mode 100644 index 00000000000..aa44b6101ff --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml @@ -0,0 +1,34 @@ +overrides: + ceph: + log-ignorelist: + - but it is still running + - objects unfound and apparently lost + conf: + osd: + osd debug reject backfill probability: .3 + osd scrub min interval: 60 + osd scrub max interval: 120 + osd max backfills: 3 + osd snap trim sleep: 2 + osd delete sleep: 1 + mon: + mon min osdmap epochs: 50 + paxos service trim min: 10 + # prune full osdmaps regularly + mon osdmap full prune min: 15 + mon osdmap full prune interval: 2 + mon osdmap full prune txsize: 2 +tasks: +- thrashosds: + timeout: 2400 + dump_ops_enable: false + sighup_delay: 0 + min_in: 3 + noscrub_toggle_delay: 0 + chance_thrash_pg_upmap: 0 + reweight_osd: 0 + thrash_primary_affinity: false + ceph_objectstore_tool: false + chance_inject_pause_short: 0 + chance_thrash_cluster_full: 0 + chance_reset_purged_snaps_last: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml b/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml new file mode 120000 index 00000000000..9124eb1aa29 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml @@ -0,0 +1 @@ +.qa/tasks/thrashosds-health.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/.qa b/qa/suites/crimson-rados-experimental/thrash/workloads/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml new file mode 100644 index 00000000000..8c9764ade84 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml @@ -0,0 +1,13 @@ +overrides: + ceph: + conf: + client.0: + admin socket: /var/run/ceph/ceph-$name.asok +tasks: +- radosbench: + clients: [client.0] + time: 150 +- admin_socket: + client.0: + objecter_requests: + test: "http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}" diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml new file mode 100644 index 00000000000..d35e8421ab4 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml @@ -0,0 +1,20 @@ +overrides: + conf: + osd: + osd deep scrub update digest min age: 0 +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + pool_snaps: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml new file mode 100644 index 00000000000..902c4b56a1e --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml @@ -0,0 +1,49 @@ +overrides: + ceph: + conf: + client.0: + debug ms: 1 + debug objecter: 20 + debug rados: 20 +tasks: +- full_sequential: + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml new file mode 100644 index 00000000000..071f55e3928 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + conf: + client.0: + debug ms: 1 + debug objecter: 20 + debug rados: 20 +tasks: +- full_sequential: + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml new file mode 100644 index 00000000000..afe04229898 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + balance_reads: true + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml new file mode 100644 index 00000000000..445b582ea42 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + localize_reads: true + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml new file mode 100644 index 00000000000..e7e8070fd76 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml @@ -0,0 +1,23 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml new file mode 100644 index 00000000000..1161c3cc253 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml @@ -0,0 +1,15 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + balance_reads: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml new file mode 100644 index 00000000000..80af0def0e4 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml @@ -0,0 +1,15 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + localize_reads: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml new file mode 100644 index 00000000000..0694ffcd0d6 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml @@ -0,0 +1,14 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml new file mode 100644 index 00000000000..606dcae6922 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml @@ -0,0 +1,8 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 500 + write_fadvise_dontneed: true + op_weights: + write: 100 diff --git a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore b/qa/suites/crimson-rados/singleton/objectstore deleted file mode 120000 index dbccf5ad928..00000000000 --- a/qa/suites/crimson-rados/singleton/objectstore +++ /dev/null @@ -1 +0,0 @@ -../thrash/objectstore
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/.qa b/qa/suites/crimson-rados/singleton/objectstore/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml new file mode 120000 index 00000000000..481e393be4a --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml.disabled b/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml index abd86d7d986..abd86d7d986 120000 --- a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml.disabled +++ b/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml diff --git a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml b/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml new file mode 120000 index 00000000000..abd86d7d986 --- /dev/null +++ b/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml @@ -0,0 +1 @@ +.qa/overrides/short_pg_log.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/fs/multifs/tasks/failover.yaml b/qa/suites/fs/multifs/tasks/failover.yaml index 55dde639c23..b7a0338566c 100644 --- a/qa/suites/fs/multifs/tasks/failover.yaml +++ b/qa/suites/fs/multifs/tasks/failover.yaml @@ -8,6 +8,7 @@ overrides: - \(MDS_DAMAGE\) - \(FS_DEGRADED\) - \(MDS_CACHE_OVERSIZED\) + - \(MDS_ESTIMATED_REPLAY_TIME\) ceph-fuse: disabled: true tasks: diff --git a/qa/suites/fs/nfs/tasks/nfs.yaml b/qa/suites/fs/nfs/tasks/nfs.yaml index aa966bff214..2dd668c9f88 100644 --- a/qa/suites/fs/nfs/tasks/nfs.yaml +++ b/qa/suites/fs/nfs/tasks/nfs.yaml @@ -1,3 +1,10 @@ +overrides: + install: + extra_system_packages: + rpm: + - fio + deb: + - fio tasks: - cephfs_test_runner: modules: diff --git a/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml b/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml index 602d3416263..aa327b0cdf5 100644 --- a/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml +++ b/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml @@ -5,6 +5,7 @@ overrides: - "mds.dir_split" tasks: - workunit: + timeout: 5h clients: all: - kernel_untar_build.sh diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml index 7c97edae552..0416ae2ea4e 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml @@ -1,7 +1,8 @@ +# runs on default nvmeof image (i.e. DEFAULT_NVMEOF_IMAGE) tasks: - nvmeof: installer: host.a - gw_image: quay.io/ceph/nvmeof:latest # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" + gw_image: default # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" rbd: pool_name: mypool image_name_prefix: myimage diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml index 9ef37004427..dfe31380bb6 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml @@ -18,6 +18,7 @@ tasks: clients: client.0: - nvmeof/setup_subsystem.sh + - nvmeof/basic_tests.sh env: RBD_POOL: mypool RBD_IMAGE_PREFIX: myimage @@ -27,7 +28,6 @@ tasks: timeout: 30m clients: client.0: - - nvmeof/basic_tests.sh - nvmeof/fio_test.sh --rbd_iostat client.1: - nvmeof/basic_tests.sh diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml index 12cb50b408d..d66b6fc8093 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml @@ -31,8 +31,11 @@ tasks: no_coverage_and_limits: true timeout: 30m clients: - client.0: + client.3: - nvmeof/scalability_test.sh nvmeof.a,nvmeof.b - nvmeof/scalability_test.sh nvmeof.b,nvmeof.c,nvmeof.d + - nvmeof/scalability_test.sh nvmeof.b,nvmeof.c env: SCALING_DELAYS: '50' + RBD_POOL: mypool + NVMEOF_GROUP: mygroup0 diff --git a/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml b/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml new file mode 100644 index 00000000000..83d54cdf5c3 --- /dev/null +++ b/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml @@ -0,0 +1,37 @@ +tasks: +- nvmeof: + installer: host.a + gw_image: quay.io/ceph/nvmeof:latest # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" + rbd: + pool_name: mypool + image_name_prefix: myimage + gateway_config: + subsystems_count: 10 + namespaces_count: 90 # each subsystem + cli_image: quay.io/ceph/nvmeof-cli:latest + +- cephadm.wait_for_service: + service: nvmeof.mypool.mygroup0 + +- cephadm.exec: + host.a: + - ceph orch ls nvmeof --export > /tmp/nvmeof-orig.yaml + - cp /tmp/nvmeof-orig.yaml /tmp/nvmeof-no-huge-page.yaml + - "sed -i '/ pool: mypool/a\\ spdk_mem_size: 4096' /tmp/nvmeof-no-huge-page.yaml" + - cat /tmp/nvmeof-no-huge-page.yaml + - ceph orch ls --refresh + - ceph orch apply -i /tmp/nvmeof-no-huge-page.yaml + - ceph orch redeploy nvmeof.mypool.mygroup0 + +- cephadm.wait_for_service: + service: nvmeof.mypool.mygroup0 + +- workunit: + no_coverage_and_limits: true + clients: + client.0: + - nvmeof/setup_subsystem.sh + - nvmeof/basic_tests.sh + env: + RBD_POOL: mypool + RBD_IMAGE_PREFIX: myimage diff --git a/qa/suites/nvmeof/thrash/gateway-initiator-setup/3-subsys-60-namespace.yaml b/qa/suites/nvmeof/thrash/gateway-initiator-setup/120-subsys-8-namespace.yaml index b4755a6433b..0f7ac011a60 100644 --- a/qa/suites/nvmeof/thrash/gateway-initiator-setup/3-subsys-60-namespace.yaml +++ b/qa/suites/nvmeof/thrash/gateway-initiator-setup/120-subsys-8-namespace.yaml @@ -6,8 +6,8 @@ tasks: pool_name: mypool image_name_prefix: myimage gateway_config: - subsystems_count: 3 - namespaces_count: 20 # each subsystem + subsystems_count: 120 + namespaces_count: 8 # each subsystem cli_image: quay.io/ceph/nvmeof-cli:latest - cephadm.wait_for_service: diff --git a/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml b/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml index 19fa2ec605d..46037784d31 100644 --- a/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml +++ b/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml @@ -11,6 +11,7 @@ overrides: - NVMEOF_SINGLE_GATEWAY - NVMEOF_GATEWAY_DOWN - are in unavailable state + - is unavailable - is in error state - failed cephadm daemon diff --git a/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml b/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml index 80bf0527715..b58dc14d87b 100644 --- a/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml +++ b/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml @@ -6,9 +6,11 @@ overrides: - NVMEOF_SINGLE_GATEWAY - NVMEOF_GATEWAY_DOWN - are in unavailable state + - is unavailable - is in error state - failed cephadm daemon tasks: - nvmeof.thrash: checker_host: 'client.0' + randomize: False diff --git a/qa/suites/nvmeof/thrash/workloads/fio.yaml b/qa/suites/nvmeof/thrash/workloads/fio.yaml index b042b92d6ae..f9a0d0ebde5 100644 --- a/qa/suites/nvmeof/thrash/workloads/fio.yaml +++ b/qa/suites/nvmeof/thrash/workloads/fio.yaml @@ -1,11 +1,11 @@ tasks: - workunit: no_coverage_and_limits: true - timeout: 30m + timeout: 60m clients: client.0: - - nvmeof/fio_test.sh --rbd_iostat + - nvmeof/fio_test.sh --random_devices 200 env: RBD_POOL: mypool IOSTAT_INTERVAL: '10' - RUNTIME: '600' + RUNTIME: '1800' diff --git a/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml b/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml index 372bf2561fa..8b3c4c11ac6 100644 --- a/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml +++ b/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml @@ -15,6 +15,7 @@ overrides: # causing tests to fail due to health warns, even if # the tests themselves are successful. - \(OSDMAP_FLAGS\) + - \(PG_DEGRADED\) tasks: - workunit: clients: diff --git a/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml b/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml new file mode 100644 index 00000000000..7cd47898544 --- /dev/null +++ b/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml @@ -0,0 +1,8 @@ +roles: +- [mon.a, mgr.x, osd.0, osd.1, client.0] +tasks: +- install: +- exec: + client.0: + - mkdir $TESTDIR/ceph_test_bluefs && cd $TESTDIR/ceph_test_bluefs && ceph_test_bluefs --log-file $TESTDIR/archive/ceph_test_bluefs.log --debug-bluefs 5/20 --gtest_catch_exceptions=0 + - rm -rf $TESTDIR/ceph_test_bluefs diff --git a/qa/suites/rados/verify/validater/valgrind.yaml b/qa/suites/rados/verify/validater/valgrind.yaml index e2dc29b5f7e..17cf141b0cd 100644 --- a/qa/suites/rados/verify/validater/valgrind.yaml +++ b/qa/suites/rados/verify/validater/valgrind.yaml @@ -27,6 +27,7 @@ overrides: - \(SLOW_OPS\) - slow request - OSD bench result + - OSD_DOWN valgrind: mon: [--tool=memcheck, --leak-check=full, --show-reachable=yes] osd: [--tool=memcheck] diff --git a/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml b/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml new file mode 100644 index 00000000000..d2072c41a68 --- /dev/null +++ b/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml @@ -0,0 +1,8 @@ +tasks: + - exec: + client.0: + - mkdir /home/ubuntu/cephtest/migration + - qemu-img create -f qcow2 /home/ubuntu/cephtest/migration/empty.qcow2 1G + - echo '{"type":"qcow","stream":{"type":"http","url":"https://download.ceph.com/qa/ubuntu-12.04.qcow2"}}' | rbd migration prepare --import-only --source-spec-path - client.0.0 + - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.1 + - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.2 diff --git a/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml b/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml index 462570e7727..303f98d540e 100644 --- a/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml +++ b/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml @@ -1,7 +1,7 @@ tasks: - kafka: client.0: - kafka_version: 2.6.0 + kafka_version: 3.8.1 - notification-tests: client.0: extra_attr: ["kafka_test", "data_path_v2_kafka_test"] diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/+ b/qa/suites/rgw/notifications/tasks/kafka_failover/+ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/+ diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml b/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml new file mode 100644 index 00000000000..5c83d5c0d23 --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml @@ -0,0 +1,20 @@ +tasks: +- install: +- ceph: +- openssl_keys: +- rgw: + client.0: + +overrides: + install: + ceph: + extra_system_packages: + rpm: + - java + deb: + - default-jre + ceph: + conf: + global: + osd_min_pg_log_entries: 10 + osd_max_pg_log_entries: 10 diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros b/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros new file mode 120000 index 00000000000..46280a42a96 --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros @@ -0,0 +1 @@ +../../.qa/distros/supported-random-distro$/
\ No newline at end of file diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml b/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml new file mode 100644 index 00000000000..01d6fc637de --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml @@ -0,0 +1,8 @@ +tasks: +- kafka-failover: + client.0: + kafka_version: 3.8.1 +- notification-tests: + client.0: + extra_attr: ["kafka_failover"] + rgw_server: client.0 diff --git a/qa/suites/upgrade/quincy-x/parallel/0-start.yaml b/qa/suites/upgrade/quincy-x/parallel/0-start.yaml index 40fbcefe728..62fb6427f72 100644 --- a/qa/suites/upgrade/quincy-x/parallel/0-start.yaml +++ b/qa/suites/upgrade/quincy-x/parallel/0-start.yaml @@ -32,13 +32,22 @@ overrides: osd: osd shutdown pgref assert: true log-ignorelist: - - \(POOL_APP_NOT_ENABLED\) + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum + - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED diff --git a/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml b/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml index e27c7c0f092..f7167975aa9 100644 --- a/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml +++ b/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml @@ -1,11 +1,8 @@ overrides: ceph: log-ignorelist: - - mons down - - mon down - - MON_DOWN - - out of quorum - - PG_AVAILABILITY + - Telemetry requires re-opt-in + - telemetry module includes new collections tasks: - install: branch: quincy diff --git a/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml b/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml index 005514292ce..5641471629e 100644 --- a/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml +++ b/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml @@ -1,17 +1,25 @@ overrides: ceph: log-ignorelist: - - \(POOL_APP_NOT_ENABLED\) + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum + - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED - OSDMAP_FLAGS - - PG_AVAILABILITY + - OSD_UPGRADE_FINISHED tasks: - install: branch: quincy diff --git a/qa/suites/upgrade/reef-x/parallel/0-start.yaml b/qa/suites/upgrade/reef-x/parallel/0-start.yaml index 146bd57960d..62fb6427f72 100644 --- a/qa/suites/upgrade/reef-x/parallel/0-start.yaml +++ b/qa/suites/upgrade/reef-x/parallel/0-start.yaml @@ -32,4 +32,22 @@ overrides: osd: osd shutdown pgref assert: true log-ignorelist: - - PG_DEGRADED + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down + - OSD_DOWN + - mons down + - mon down + - MON_DOWN + - out of quorum + - PG_AVAILABILITY + - PG_DEGRADED + - Reduced data availability + - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED + - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED diff --git a/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml b/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml index ce4e0cc228b..b5160c2dd00 100644 --- a/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml +++ b/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml @@ -1,12 +1,8 @@ overrides: ceph: log-ignorelist: - - mons down - - mon down - - MON_DOWN - - out of quorum - - PG_AVAILABILITY - - PG_DEGRADED + - Telemetry requires re-opt-in + - telemetry module includes new collections tasks: - install: branch: reef diff --git a/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml b/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml index 5e995da7d2c..fa93b2f2ece 100644 --- a/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml +++ b/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml @@ -1,20 +1,19 @@ overrides: ceph: log-ignorelist: - - \(MDS_ALL_DOWN\) - - \(MDS_UP_LESS_THAN_MAX\) - - \(OSD_SLOW_PING_TIME + - MDS_ALL_DOWN + - MDS_UP_LESS_THAN_MAX + - OSD_SLOW_PING_TIME - reached quota + - running out of quota - overall HEALTH_ - - \(CACHE_POOL_NO_HIT_SET\) - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(SLOW_OPS\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(OBJECT_MISPLACED\) + - CACHE_POOL_NO_HIT_SET + - pool\(s\) full + - POOL_FULL + - SMALLER_PGP_NUM + - SLOW_OPS + - CACHE_POOL_NEAR_FULL + - OBJECT_MISPLACED - slow request - - \(MON_DOWN\) - noscrub - nodeep-scrub diff --git a/qa/suites/upgrade/reef-x/stress-split/1-start.yaml b/qa/suites/upgrade/reef-x/stress-split/1-start.yaml index 992f9e1bc36..59ccfe2cd02 100644 --- a/qa/suites/upgrade/reef-x/stress-split/1-start.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/1-start.yaml @@ -1,11 +1,25 @@ overrides: ceph: log-ignorelist: + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down + - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY + - PG_DEGRADED + - Reduced data availability + - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED + - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED tasks: - install: branch: reef diff --git a/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml b/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml index 5e995da7d2c..fa93b2f2ece 100644 --- a/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml @@ -1,20 +1,19 @@ overrides: ceph: log-ignorelist: - - \(MDS_ALL_DOWN\) - - \(MDS_UP_LESS_THAN_MAX\) - - \(OSD_SLOW_PING_TIME + - MDS_ALL_DOWN + - MDS_UP_LESS_THAN_MAX + - OSD_SLOW_PING_TIME - reached quota + - running out of quota - overall HEALTH_ - - \(CACHE_POOL_NO_HIT_SET\) - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(SLOW_OPS\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(OBJECT_MISPLACED\) + - CACHE_POOL_NO_HIT_SET + - pool\(s\) full + - POOL_FULL + - SMALLER_PGP_NUM + - SLOW_OPS + - CACHE_POOL_NEAR_FULL + - OBJECT_MISPLACED - slow request - - \(MON_DOWN\) - noscrub - nodeep-scrub diff --git a/qa/tasks/cephfs/test_exports.py b/qa/tasks/cephfs/test_exports.py index 346f139874b..468378fce3d 100644 --- a/qa/tasks/cephfs/test_exports.py +++ b/qa/tasks/cephfs/test_exports.py @@ -153,6 +153,8 @@ class TestExportPin(CephFSTestCase): # vstart.sh sets mds_debug_subtrees to True. That causes a ESubtreeMap # to be written out every event. Yuck! self.config_set('mds', 'mds_debug_subtrees', False) + # make sure ESubtreeMap is written frequently enough: + self.config_set('mds', 'mds_log_minor_segments_per_major_segment', '4') self.config_rm('mds', 'mds bal split size') # don't split /top self.mount_a.run_shell_payload("rm -rf 1") diff --git a/qa/tasks/cephfs/test_failover.py b/qa/tasks/cephfs/test_failover.py index 29af1e76a4f..46139163ddd 100644 --- a/qa/tasks/cephfs/test_failover.py +++ b/qa/tasks/cephfs/test_failover.py @@ -1,3 +1,4 @@ +import re import time import signal import logging @@ -342,6 +343,60 @@ class TestClusterResize(CephFSTestCase): self.fs.wait_for_daemons(timeout=90) +class TestFailoverBeaconHealth(CephFSTestCase): + CLIENTS_REQUIRED = 1 + MDSS_REQUIRED = 1 + + def initiate_journal_replay(self, num_files=100): + """ Initiate journal replay by creating files and restarting mds server.""" + + self.config_set("mds", "mds_delay_journal_replay_for_testing", "5000") + self.mounts[0].test_files = [str(x) for x in range(num_files)] + self.mounts[0].create_files() + self.fs.fail() + self.fs.set_joinable() + + def test_replay_beacon_estimated_time(self): + """ + That beacon emits warning message with estimated time to complete replay + """ + self.initiate_journal_replay() + self.wait_for_health("MDS_ESTIMATED_REPLAY_TIME", 60) + # remove the config so that replay finishes and the cluster + # is HEALTH_OK + self.config_rm("mds", "mds_delay_journal_replay_for_testing") + self.wait_for_health_clear(timeout=60) + + def test_replay_estimated_time_accuracy(self): + self.initiate_journal_replay(250) + def replay_complete(): + health = self.ceph_cluster.mon_manager.get_mon_health(debug=False, detail=True) + codes = [s for s in health['checks']] + return 'MDS_ESTIMATED_REPLAY_TIME' not in codes + + def get_estimated_time(): + completion_percentage = 0.0 + time_duration = pending_duration = 0 + with safe_while(sleep=5, tries=360) as proceed: + while proceed(): + health = self.ceph_cluster.mon_manager.get_mon_health(debug=False, detail=True) + codes = [s for s in health['checks']] + if 'MDS_ESTIMATED_REPLAY_TIME' in codes: + message = health['checks']['MDS_ESTIMATED_REPLAY_TIME']['detail'][0]['message'] + ### sample warning string: "mds.a(mds.0): replay: 50.0446% complete - elapsed time: 582s, estimated time remaining: 581s" + m = re.match(".* replay: (\d+(\.\d+)?)% complete - elapsed time: (\d+)s, estimated time remaining: (\d+)s", message) + if not m: + continue + completion_percentage = float(m.group(1)) + time_duration = int(m.group(3)) + pending_duration = int(m.group(4)) + log.debug(f"MDS_ESTIMATED_REPLAY_TIME is present in health: {message}, duration: {time_duration}, completion_percentage: {completion_percentage}") + if completion_percentage >= 50: + return (completion_percentage, time_duration, pending_duration) + _, _, pending_duration = get_estimated_time() + # wait for 25% more time to avoid false negative failures + self.wait_until_true(replay_complete, timeout=pending_duration * 1.25) + class TestFailover(CephFSTestCase): CLIENTS_REQUIRED = 1 MDSS_REQUIRED = 2 diff --git a/qa/tasks/cephfs/test_nfs.py b/qa/tasks/cephfs/test_nfs.py index faa35be6926..0a1c07dce04 100644 --- a/qa/tasks/cephfs/test_nfs.py +++ b/qa/tasks/cephfs/test_nfs.py @@ -369,6 +369,45 @@ class TestNFS(MgrTestCase): except CommandFailedError as e: self.fail(f"expected read/write of a file to be successful but failed with {e.exitstatus}") + def _mnt_nfs(self, pseudo_path, port, ip): + ''' + Mount created export + :param pseudo_path: It is the pseudo root name + :param port: Port of deployed nfs cluster + :param ip: IP of deployed nfs cluster + ''' + tries = 3 + while True: + try: + self.ctx.cluster.run( + args=['sudo', 'mount', '-t', 'nfs', '-o', f'port={port}', + f'{ip}:{pseudo_path}', '/mnt']) + break + except CommandFailedError: + if tries: + tries -= 1 + time.sleep(2) + continue + raise + + self.ctx.cluster.run(args=['sudo', 'chmod', '1777', '/mnt']) + + def _test_fio(self, pseudo_path, port, ip): + ''' + run fio with libaio on /mnt/fio + :param mnt_path: nfs mount point + ''' + try: + self._mnt_nfs(pseudo_path, port, ip) + self.ctx.cluster.run(args=['mkdir', '/mnt/fio']) + fio_cmd=['sudo', 'fio', '--ioengine=libaio', '-directory=/mnt/fio', '--filename=fio.randrw.test', '--name=job', '--bs=16k', '--direct=1', '--group_reporting', '--iodepth=128', '--randrepeat=0', '--norandommap=1', '--thread=2', '--ramp_time=20s', '--offset_increment=5%', '--size=5G', '--time_based', '--runtime=300', '--ramp_time=1s', '--percentage_random=0', '--rw=randrw', '--rwmixread=50'] + self.ctx.cluster.run(args=fio_cmd) + except CommandFailedError as e: + self.fail(f"expected fio to be successful but failed with {e.exitstatus}") + finally: + self.ctx.cluster.run(args=['sudo', 'rm', '-rf', '/mnt/fio']) + self.ctx.cluster.run(args=['sudo', 'umount', '/mnt']) + def _write_to_read_only_export(self, pseudo_path, port, ip): ''' Check if write to read only export fails @@ -627,6 +666,18 @@ class TestNFS(MgrTestCase): self._test_data_read_write(self.pseudo_path, port, ip) self._test_delete_cluster() + def test_async_io_fio(self): + ''' + Test async io using fio. Expect completion without hang or crash + ''' + self._test_create_cluster() + self._create_export(export_id='1', create_fs=True, + extra_cmd=['--pseudo-path', self.pseudo_path]) + port, ip = self._get_port_ip_info() + self._check_nfs_cluster_status('running', 'NFS Ganesha cluster restart failed') + self._test_fio(self.pseudo_path, port, ip) + self._test_delete_cluster() + def test_cluster_info(self): ''' Test cluster info outputs correct ip and hostname diff --git a/qa/tasks/kafka.py b/qa/tasks/kafka.py index 5e6c208ca30..833f03babf6 100644 --- a/qa/tasks/kafka.py +++ b/qa/tasks/kafka.py @@ -4,6 +4,7 @@ Deploy and configure Kafka for Teuthology import contextlib import logging import time +import os from teuthology import misc as teuthology from teuthology import contextutil @@ -33,6 +34,13 @@ def install_kafka(ctx, config): assert isinstance(config, dict) log.info('Installing Kafka...') + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + for (client, _) in config.items(): (remote,) = ctx.cluster.only(client).remotes.keys() test_dir=teuthology.get_testdir(ctx) @@ -40,7 +48,8 @@ def install_kafka(ctx, config): kafka_file = kafka_prefix + current_version + '.tgz' - link1 = 'https://archive.apache.org/dist/kafka/' + current_version + '/' + kafka_file + link1 = '{apache_mirror_url_front}/kafka/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + current_version + '/' + kafka_file ctx.cluster.only(client).run( args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'wget', link1], ) diff --git a/qa/tasks/kafka_failover.py b/qa/tasks/kafka_failover.py new file mode 100644 index 00000000000..3ca60ab84fc --- /dev/null +++ b/qa/tasks/kafka_failover.py @@ -0,0 +1,244 @@ +""" +Deploy and configure Kafka for Teuthology +""" +import contextlib +import logging +import time +import os + +from teuthology import misc as teuthology +from teuthology import contextutil +from teuthology.orchestra import run + +log = logging.getLogger(__name__) + +def get_kafka_version(config): + for client, client_config in config.items(): + if 'kafka_version' in client_config: + kafka_version = client_config.get('kafka_version') + return kafka_version + +kafka_prefix = 'kafka_2.13-' + +def get_kafka_dir(ctx, config): + kafka_version = get_kafka_version(config) + current_version = kafka_prefix + kafka_version + return '{tdir}/{ver}'.format(tdir=teuthology.get_testdir(ctx),ver=current_version) + + +@contextlib.contextmanager +def install_kafka(ctx, config): + """ + Downloading the kafka tar file. + """ + assert isinstance(config, dict) + log.info('Installing Kafka...') + + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + + for (client, _) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + test_dir=teuthology.get_testdir(ctx) + current_version = get_kafka_version(config) + + kafka_file = kafka_prefix + current_version + '.tgz' + + link1 = '{apache_mirror_url_front}/kafka/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + current_version + '/' + kafka_file + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'wget', link1], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'tar', '-xvzf', kafka_file], + ) + + kafka_dir = get_kafka_dir(ctx, config) + # create config for second broker + second_broker_config_name = "server2.properties" + second_broker_data = "{tdir}/data/broker02".format(tdir=kafka_dir) + second_broker_data_logs_escaped = "{}/logs".format(second_broker_data).replace("/", "\/") + + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=kafka_dir), run.Raw('&&'), + 'cp', '{tdir}/config/server.properties'.format(tdir=kafka_dir), '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'mkdir', '-p', '{tdir}/data'.format(tdir=kafka_dir) + ], + ) + + # edit config + ctx.cluster.only(client).run( + args=['sed', '-i', 's/broker.id=0/broker.id=1/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/localhost:19092/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/localhost:19092/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/log.dirs=\/tmp\/kafka-logs/log.dirs={}/g'.format(second_broker_data_logs_escaped), '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'cat', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name) + ] + ) + + try: + yield + finally: + log.info('Removing packaged dependencies of Kafka...') + test_dir=get_kafka_dir(ctx, config) + current_version = get_kafka_version(config) + for (client,_) in config.items(): + ctx.cluster.only(client).run( + args=['rm', '-rf', '{tdir}/logs'.format(tdir=test_dir)], + ) + + ctx.cluster.only(client).run( + args=['rm', '-rf', test_dir], + ) + + ctx.cluster.only(client).run( + args=['rm', '-rf', '{tdir}/{doc}'.format(tdir=teuthology.get_testdir(ctx),doc=kafka_file)], + ) + + +@contextlib.contextmanager +def run_kafka(ctx,config): + """ + This includes two parts: + 1. Starting Zookeeper service + 2. Starting Kafka service + """ + assert isinstance(config, dict) + log.info('Bringing up Zookeeper and Kafka services...') + for (client,_) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + kafka_dir = get_kafka_dir(ctx, config) + + second_broker_data = "{tdir}/data/broker02".format(tdir=kafka_dir) + second_broker_java_log_dir = "{}/java_logs".format(second_broker_data) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + './zookeeper-server-start.sh', + '{tir}/config/zookeeper.properties'.format(tir=kafka_dir), + run.Raw('&'), 'exit' + ], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + './kafka-server-start.sh', + '{tir}/config/server.properties'.format(tir=get_kafka_dir(ctx, config)), + run.Raw('&'), 'exit' + ], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + run.Raw('LOG_DIR={second_broker_java_log_dir}'.format(second_broker_java_log_dir=second_broker_java_log_dir)), + './kafka-server-start.sh', '{tdir}/config/server2.properties'.format(tdir=kafka_dir), + run.Raw('&'), 'exit' + ], + ) + + try: + yield + finally: + log.info('Stopping Zookeeper and Kafka Services...') + + for (client, _) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-server-stop.sh', + '{tir}/config/kafka.properties'.format(tir=get_kafka_dir(ctx, config)), + ], + ) + + time.sleep(5) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './zookeeper-server-stop.sh', + '{tir}/config/zookeeper.properties'.format(tir=get_kafka_dir(ctx, config)), + ], + ) + + time.sleep(5) + + ctx.cluster.only(client).run(args=['killall', '-9', 'java']) + + +@contextlib.contextmanager +def run_admin_cmds(ctx,config): + """ + Running Kafka Admin commands in order to check the working of producer anf consumer and creation of topic. + """ + assert isinstance(config, dict) + log.info('Checking kafka server through producer/consumer commands...') + for (client,_) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-topics.sh', '--create', '--topic', 'quickstart-events', + '--bootstrap-server', 'localhost:9092' + ], + ) + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + 'echo', "First", run.Raw('|'), + './kafka-console-producer.sh', '--topic', 'quickstart-events', + '--bootstrap-server', 'localhost:9092' + ], + ) + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-console-consumer.sh', '--topic', 'quickstart-events', + '--from-beginning', + '--bootstrap-server', 'localhost:9092', + run.Raw('&'), 'exit' + ], + ) + + try: + yield + finally: + pass + + +@contextlib.contextmanager +def task(ctx,config): + """ + Following is the way how to run kafka:: + tasks: + - kafka: + client.0: + kafka_version: 2.6.0 + """ + assert config is None or isinstance(config, list) \ + or isinstance(config, dict), \ + "task kafka only supports a list or dictionary for configuration" + + all_clients = ['client.{id}'.format(id=id_) + for id_ in teuthology.all_roles_of_type(ctx.cluster, 'client')] + if config is None: + config = all_clients + if isinstance(config, list): + config = dict.fromkeys(config) + + log.debug('Kafka config is %s', config) + + with contextutil.nested( + lambda: install_kafka(ctx=ctx, config=config), + lambda: run_kafka(ctx=ctx, config=config), + lambda: run_admin_cmds(ctx=ctx, config=config), + ): + yield + diff --git a/qa/tasks/mgr/dashboard/test_rbd.py b/qa/tasks/mgr/dashboard/test_rbd.py index a872645e33e..83b3bf520c2 100644 --- a/qa/tasks/mgr/dashboard/test_rbd.py +++ b/qa/tasks/mgr/dashboard/test_rbd.py @@ -869,7 +869,19 @@ class RbdTest(DashboardTestCase): self.assertEqual(clone_format_version, 2) self.assertStatus(200) + # if empty list is sent, then the config will remain as it is value = [] + res = [{'section': "global", 'value': "2"}] + self._post('/api/cluster_conf', { + 'name': config_name, + 'value': value + }) + self.wait_until_equal( + lambda: _get_config_by_name(config_name), + res, + timeout=60) + + value = [{'section': "global", 'value': ""}] self._post('/api/cluster_conf', { 'name': config_name, 'value': value diff --git a/qa/tasks/mgr/dashboard/test_rgw.py b/qa/tasks/mgr/dashboard/test_rgw.py index 5c7b0329675..a9071bc2a3a 100644 --- a/qa/tasks/mgr/dashboard/test_rgw.py +++ b/qa/tasks/mgr/dashboard/test_rgw.py @@ -785,7 +785,7 @@ class RgwUserSubuserTest(RgwTestCase): 'access': 'readwrite', 'key_type': 'swift' }) - self.assertStatus(200) + self.assertStatus(201) data = self.jsonBody() subuser = self.find_object_in_list('id', 'teuth-test-user:tux', data) self.assertIsInstance(subuser, object) @@ -808,7 +808,7 @@ class RgwUserSubuserTest(RgwTestCase): 'access_key': 'yyy', 'secret_key': 'xxx' }) - self.assertStatus(200) + self.assertStatus(201) data = self.jsonBody() subuser = self.find_object_in_list('id', 'teuth-test-user:hugo', data) self.assertIsInstance(subuser, object) diff --git a/qa/tasks/notification_tests.py b/qa/tasks/notification_tests.py index b4697a6f797..f1eae3c89c4 100644 --- a/qa/tasks/notification_tests.py +++ b/qa/tasks/notification_tests.py @@ -220,7 +220,7 @@ def run_tests(ctx, config): for client, client_config in config.items(): (remote,) = ctx.cluster.only(client).remotes.keys() - attr = ["!kafka_test", "!data_path_v2_kafka_test", "!amqp_test", "!amqp_ssl_test", "!kafka_security_test", "!modification_required", "!manual_test", "!http_test"] + attr = ["!kafka_test", "!data_path_v2_kafka_test", "!kafka_failover", "!amqp_test", "!amqp_ssl_test", "!kafka_security_test", "!modification_required", "!manual_test", "!http_test"] if 'extra_attr' in client_config: attr = client_config.get('extra_attr') diff --git a/qa/tasks/nvmeof.py b/qa/tasks/nvmeof.py index 42e357294d9..691a6f7dd86 100644 --- a/qa/tasks/nvmeof.py +++ b/qa/tasks/nvmeof.py @@ -128,12 +128,11 @@ class Nvmeof(Task): total_images = int(self.namespaces_count) * int(self.subsystems_count) log.info(f'[nvmeof]: creating {total_images} images') + rbd_create_cmd = [] for i in range(1, total_images + 1): imagename = self.image_name_prefix + str(i) - log.info(f'[nvmeof]: rbd create {poolname}/{imagename} --size {self.rbd_size}') - _shell(self.ctx, self.cluster_name, self.remote, [ - 'rbd', 'create', f'{poolname}/{imagename}', '--size', f'{self.rbd_size}' - ]) + rbd_create_cmd += ['rbd', 'create', f'{poolname}/{imagename}', '--size', f'{self.rbd_size}', run.Raw(';')] + _shell(self.ctx, self.cluster_name, self.remote, rbd_create_cmd) for role, i in daemons.items(): remote, id_ = i @@ -251,9 +250,9 @@ class NvmeofThrasher(Thrasher, Greenlet): daemon_max_thrash_times: For now, NVMeoF daemons have limitation that each daemon can - be thrashed only 3 times in span of 30 mins. This option + be thrashed only 5 times in span of 30 mins. This option allows to set the amount of times it could be thrashed in a period - of time. (default: 3) + of time. (default: 5) daemon_max_thrash_period: This option goes with the above option. It sets the period of time over which each daemons can be thrashed for daemon_max_thrash_times @@ -306,17 +305,17 @@ class NvmeofThrasher(Thrasher, Greenlet): self.max_thrash_daemons = int(self.config.get('max_thrash', len(self.daemons) - 1)) # Limits on thrashing each daemon - self.daemon_max_thrash_times = int(self.config.get('daemon_max_thrash_times', 3)) + self.daemon_max_thrash_times = int(self.config.get('daemon_max_thrash_times', 5)) self.daemon_max_thrash_period = int(self.config.get('daemon_max_thrash_period', 30 * 60)) # seconds self.min_thrash_delay = int(self.config.get('min_thrash_delay', 60)) self.max_thrash_delay = int(self.config.get('max_thrash_delay', self.min_thrash_delay + 30)) - self.min_revive_delay = int(self.config.get('min_revive_delay', 100)) + self.min_revive_delay = int(self.config.get('min_revive_delay', 60)) self.max_revive_delay = int(self.config.get('max_revive_delay', self.min_revive_delay + 30)) def _get_devices(self, remote): GET_DEVICE_CMD = "sudo nvme list --output-format=json | " \ - "jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == \"Ceph bdev Controller\") | .DevicePath'" + "jq -r '.Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == \"Ceph bdev Controller\")) | .Namespaces | sort_by(.NSID) | .[] | .NameSpace'" devices = remote.sh(GET_DEVICE_CMD).split() return devices @@ -347,6 +346,7 @@ class NvmeofThrasher(Thrasher, Greenlet): run.Raw('&&'), 'ceph', 'orch', 'ps', '--daemon-type', 'nvmeof', run.Raw('&&'), 'ceph', 'health', 'detail', run.Raw('&&'), 'ceph', '-s', + run.Raw('&&'), 'sudo', 'nvme', 'list', ] for dev in self.devices: check_cmd += [ @@ -421,13 +421,11 @@ class NvmeofThrasher(Thrasher, Greenlet): while not self.stopping.is_set(): killed_daemons = defaultdict(list) - weight = 1.0 / len(self.daemons) - count = 0 + thrash_daemon_num = self.rng.randint(1, self.max_thrash_daemons) + selected_daemons = self.rng.sample(self.daemons, thrash_daemon_num) for daemon in self.daemons: - skip = self.rng.uniform(0.0, 1.0) - if weight <= skip: - self.log('skipping daemon {label} with skip ({skip}) > weight ({weight})'.format( - label=daemon.id_, skip=skip, weight=weight)) + if daemon not in selected_daemons: + self.log(f'skipping daemon {daemon.id_} ...') continue # For now, nvmeof daemons can only be thrashed 3 times in last 30mins. @@ -445,17 +443,11 @@ class NvmeofThrasher(Thrasher, Greenlet): continue self.log('kill {label}'.format(label=daemon.id_)) - # daemon.stop() kill_method = self.kill_daemon(daemon) killed_daemons[kill_method].append(daemon) daemons_thrash_history[daemon.id_] += [datetime.now()] - # only thrash max_thrash_daemons amount of daemons - count += 1 - if count >= self.max_thrash_daemons: - break - if killed_daemons: iteration_summary = "thrashed- " for kill_method in killed_daemons: @@ -468,7 +460,7 @@ class NvmeofThrasher(Thrasher, Greenlet): self.log(f'waiting for {revive_delay} secs before reviving') time.sleep(revive_delay) # blocking wait - self.log('done waiting before reviving') + self.log(f'done waiting before reviving - iteration #{len(summary)}: {iteration_summary}') self.do_checks() self.switch_task() @@ -487,7 +479,7 @@ class NvmeofThrasher(Thrasher, Greenlet): if thrash_delay > 0.0: self.log(f'waiting for {thrash_delay} secs before thrashing') time.sleep(thrash_delay) # blocking - self.log('done waiting before thrashing') + self.log('done waiting before thrashing - everything should be up now') self.do_checks() self.switch_task() diff --git a/qa/tasks/rgw_multisite.py b/qa/tasks/rgw_multisite.py index e83a54efc2b..f93ca017fa2 100644 --- a/qa/tasks/rgw_multisite.py +++ b/qa/tasks/rgw_multisite.py @@ -361,6 +361,8 @@ def create_zonegroup(cluster, gateways, period, config): if endpoints: # replace client names with their gateway endpoints config['endpoints'] = extract_gateway_endpoints(gateways, endpoints) + if not config.get('api_name'): # otherwise it will be set to an empty string + config['api_name'] = config['name'] zonegroup = multisite.ZoneGroup(config['name'], period) # `zonegroup set` needs --default on command line, and 'is_master' in json args = is_default_arg(config) diff --git a/qa/tasks/s3a_hadoop.py b/qa/tasks/s3a_hadoop.py index 7b77359fcf2..4518a6f397c 100644 --- a/qa/tasks/s3a_hadoop.py +++ b/qa/tasks/s3a_hadoop.py @@ -1,5 +1,6 @@ import contextlib import logging +import os from teuthology import misc from teuthology.orchestra import run @@ -40,7 +41,7 @@ def task(ctx, config): # get versions maven_major = config.get('maven-major', 'maven-3') - maven_version = config.get('maven-version', '3.6.3') + maven_version = config.get('maven-version', '3.9.9') hadoop_ver = config.get('hadoop-version', '2.9.2') bucket_name = config.get('bucket-name', 's3atest') access_key = config.get('access-key', 'EGAQRD2ULOIFKFSKCT4F') @@ -48,11 +49,19 @@ def task(ctx, config): 'secret-key', 'zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb') + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + # set versions for cloning the repo apache_maven = 'apache-maven-{maven_version}-bin.tar.gz'.format( maven_version=maven_version) - maven_link = 'http://archive.apache.org/dist/maven/' + \ - '{maven_major}/{maven_version}/binaries/'.format(maven_major=maven_major, maven_version=maven_version) + apache_maven + maven_link = '{apache_mirror_url_front}/maven/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + '{maven_major}/{maven_version}/binaries/'.format(maven_major=maven_major, maven_version=maven_version) + \ + apache_maven hadoop_git = 'https://github.com/apache/hadoop' hadoop_rel = 'hadoop-{ver} rel/release-{ver}'.format(ver=hadoop_ver) if hadoop_ver == 'trunk': @@ -204,6 +213,7 @@ def run_s3atest(client, maven_version, testdir, test_options): run.Raw('&&'), run.Raw(rm_test), run.Raw('&&'), + run.Raw('JAVA_HOME=$(alternatives --list | grep jre_1.8.0 | head -n 1 | awk \'{print $3}\')'), run.Raw(run_test), run.Raw(test_options) ] diff --git a/qa/workunits/nvmeof/basic_tests.sh b/qa/workunits/nvmeof/basic_tests.sh index dc6fd1669da..9e7a1f5134e 100755 --- a/qa/workunits/nvmeof/basic_tests.sh +++ b/qa/workunits/nvmeof/basic_tests.sh @@ -38,8 +38,10 @@ disconnect_all() { connect_all() { sudo nvme connect-all --traddr=$NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --transport=tcp -l 3600 sleep 5 - output=$(sudo nvme list --output-format=json) - if ! echo "$output" | grep -q "$SPDK_CONTROLLER"; then + expected_devices_count=$1 + actual_devices=$(sudo nvme list --output-format=json | jq -r ".Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == \"$SPDK_CONTROLLER\")) | .Namespaces[].NameSpace" | wc -l) + if [ "$actual_devices" -ne "$expected_devices_count" ]; then + sudo nvme list --output-format=json return 1 fi } @@ -72,11 +74,13 @@ test_run connect test_run list_subsys 1 test_run disconnect_all test_run list_subsys 0 -test_run connect_all +devices_count=$(( $NVMEOF_NAMESPACES_COUNT * $NVMEOF_SUBSYSTEMS_COUNT )) +test_run connect_all $devices_count gateways_count=$(( $(echo "$NVMEOF_GATEWAY_IP_ADDRESSES" | tr -cd ',' | wc -c) + 1 )) multipath_count=$(( $gateways_count * $NVMEOF_SUBSYSTEMS_COUNT)) test_run list_subsys $multipath_count + echo "-------------Test Summary-------------" echo "[nvmeof] All nvmeof basic tests passed!" diff --git a/qa/workunits/nvmeof/fio_test.sh b/qa/workunits/nvmeof/fio_test.sh index 57d355a6318..f7f783afc67 100755 --- a/qa/workunits/nvmeof/fio_test.sh +++ b/qa/workunits/nvmeof/fio_test.sh @@ -5,6 +5,7 @@ sudo yum -y install sysstat namespace_range_start= namespace_range_end= +random_devices_count= rbd_iostat=false while [[ $# -gt 0 ]]; do @@ -17,6 +18,10 @@ while [[ $# -gt 0 ]]; do namespace_range_end=$2 shift 2 ;; + --random_devices) + random_devices_count=$2 + shift 2 + ;; --rbd_iostat) rbd_iostat=true shift @@ -29,7 +34,7 @@ done fio_file=$(mktemp -t nvmeof-fio-XXXX) all_drives_list=$(sudo nvme list --output-format=json | - jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == "Ceph bdev Controller") | .DevicePath') + jq -r '.Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == "Ceph bdev Controller")) | .Namespaces | sort_by(.NSID) | .[] | .NameSpace') # When the script is passed --start_ns and --end_ns (example: `nvmeof_fio_test.sh --start_ns 1 --end_ns 3`), # then fio runs on namespaces only in the defined range (which is 1 to 3 here). @@ -37,6 +42,8 @@ all_drives_list=$(sudo nvme list --output-format=json | # run on first 3 namespaces here. if [ "$namespace_range_start" ] || [ "$namespace_range_end" ]; then selected_drives=$(echo "${all_drives_list[@]}" | sed -n "${namespace_range_start},${namespace_range_end}p") +elif [ "$random_devices_count" ]; then + selected_drives=$(echo "${all_drives_list[@]}" | shuf -n $random_devices_count) else selected_drives="${all_drives_list[@]}" fi diff --git a/qa/workunits/nvmeof/scalability_test.sh b/qa/workunits/nvmeof/scalability_test.sh index 5a26b6284f7..8ede4b7eda2 100755 --- a/qa/workunits/nvmeof/scalability_test.sh +++ b/qa/workunits/nvmeof/scalability_test.sh @@ -3,37 +3,64 @@ GATEWAYS=$1 # exmaple "nvmeof.a,nvmeof.b" DELAY="${SCALING_DELAYS:-50}" +POOL="${RBD_POOL:-mypool}" +GROUP="${NVMEOF_GROUP:-mygroup0}" +source /etc/ceph/nvmeof.env if [ -z "$GATEWAYS" ]; then echo "At least one gateway needs to be defined for scalability test" exit 1 fi -pip3 install yq - status_checks() { - ceph nvme-gw show mypool '' - ceph orch ls - ceph orch ps - ceph -s + expected_count=$1 + + output=$(ceph nvme-gw show $POOL $GROUP) + nvme_show=$(echo $output | grep -o '"AVAILABLE"' | wc -l) + if [ "$nvme_show" -ne "$expected_count" ]; then + return 1 + fi + + orch_ls=$(ceph orch ls) + if ! echo "$orch_ls" | grep -q "$expected_count/$expected_count"; then + return 1 + fi + + output=$(ceph orch ps --service-name nvmeof.$POOL.$GROUP) + orch_ps=$(echo $output | grep -o 'running' | wc -l) + if [ "$orch_ps" -ne "$expected_count" ]; then + return 1 + fi + + ceph_status=$(ceph -s) + if ! echo "$ceph_status" | grep -q "HEALTH_OK"; then + return 1 + fi } +total_gateways_count=$(( $(echo "$NVMEOF_GATEWAY_IP_ADDRESSES" | tr -cd ',' | wc -c) + 1 )) +scaled_down_gateways_count=$(( total_gateways_count - $(echo "$GATEWAYS" | tr -cd ',' | wc -c) - 1 )) + echo "[nvmeof.scale] Setting up config to remove gateways ${GATEWAYS}" +ceph orch ls --service-name nvmeof.$POOL.$GROUP --export > /tmp/nvmeof-gw.yaml ceph orch ls nvmeof --export > /tmp/nvmeof-gw.yaml cat /tmp/nvmeof-gw.yaml -yq "del(.placement.hosts[] | select(. | test(\".*($(echo $GATEWAYS | sed 's/,/|/g'))\")))" /tmp/nvmeof-gw.yaml > /tmp/nvmeof-gw-new.yaml + +pattern=$(echo $GATEWAYS | sed 's/,/\\|/g') +sed "/$pattern/d" /tmp/nvmeof-gw.yaml > /tmp/nvmeof-gw-new.yaml cat /tmp/nvmeof-gw-new.yaml echo "[nvmeof.scale] Starting scale testing by removing ${GATEWAYS}" -status_checks -ceph orch rm nvmeof.mypool && sleep 20 # temp workaround +status_checks $total_gateways_count ceph orch apply -i /tmp/nvmeof-gw-new.yaml # downscale +ceph orch redeploy nvmeof.$POOL.$GROUP sleep $DELAY -status_checks -ceph orch rm nvmeof.mypool && sleep 20 # temp workaround +status_checks $scaled_down_gateways_count +echo "[nvmeof.scale] Downscale complete - removed gateways (${GATEWAYS}); now scaling back up" ceph orch apply -i /tmp/nvmeof-gw.yaml #upscale +ceph orch redeploy nvmeof.$POOL.$GROUP sleep $DELAY -status_checks +status_checks $total_gateways_count echo "[nvmeof.scale] Scale testing passed for ${GATEWAYS}" diff --git a/qa/workunits/nvmeof/setup_subsystem.sh b/qa/workunits/nvmeof/setup_subsystem.sh index cc4024323eb..b573647b1e3 100755 --- a/qa/workunits/nvmeof/setup_subsystem.sh +++ b/qa/workunits/nvmeof/setup_subsystem.sh @@ -26,14 +26,21 @@ list_subsystems () { done } +list_namespaces () { + for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do + subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" + sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT --format plain namespace list --subsystem $subsystem_nqn + done +} + +echo "[nvmeof] Starting subsystem setup..." + # add all subsystems for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT subsystem add --subsystem $subsystem_nqn --no-group-append done -list_subsystems - # add all gateway listeners for i in "${!gateway_ips[@]}" do @@ -65,11 +72,5 @@ done list_subsystems -# list namespaces -for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do - subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" - sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT --format plain namespace list --subsystem $subsystem_nqn -done - echo "[nvmeof] Subsystem setup done" diff --git a/qa/workunits/rbd/cli_generic.sh b/qa/workunits/rbd/cli_generic.sh index 2aa27d3d655..0ceb9ff54cf 100755 --- a/qa/workunits/rbd/cli_generic.sh +++ b/qa/workunits/rbd/cli_generic.sh @@ -914,6 +914,11 @@ test_namespace() { rbd group create rbd/test1/group1 rbd group image add rbd/test1/group1 rbd/test1/image1 + rbd group image add --group-pool rbd --group-namespace test1 --group group1 \ + --image-pool rbd --image-namespace test1 --image image2 + rbd group image rm --group-pool rbd --group-namespace test1 --group group1 \ + --image-pool rbd --image-namespace test1 --image image1 + rbd group image rm rbd/test1/group1 rbd/test1/image2 rbd group rm rbd/test1/group1 rbd trash move rbd/test1/image1 diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 43bab75680d..9cbe350b388 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -16,6 +16,7 @@ endif() set(bindir ${CMAKE_INSTALL_FULL_BINDIR}) set(sbindir ${CMAKE_INSTALL_FULL_SBINDIR}) set(libdir ${CMAKE_INSTALL_FULL_LIBDIR}) +set(includedir ${CMAKE_INSTALL_FULL_INCLUDEDIR}) set(sysconfdir ${CMAKE_INSTALL_FULL_SYSCONFDIR}) set(libexecdir ${CMAKE_INSTALL_FULL_LIBEXECDIR}) set(pkgdatadir ${CMAKE_INSTALL_FULL_DATADIR}) @@ -31,6 +32,12 @@ configure_file(ceph-post-file.in configure_file(ceph-crash.in ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/ceph-crash @ONLY) +if(WITH_LIBCEPHFS) + configure_file( + ${CMAKE_SOURCE_DIR}/src/cephfs.pc.in + ${CMAKE_BINARY_DIR}/src/cephfs.pc @ONLY) +endif(WITH_LIBCEPHFS) + # the src/.git_version file may be written out by make-dist; otherwise # we pull the git version from .git option(ENABLE_GIT_VERSION "build Ceph with git version string" ON) @@ -832,10 +839,12 @@ if(WITH_LIBCEPHFS) target_link_libraries(cephfs PRIVATE client ceph-common ${CRYPTO_LIBS} ${EXTRALIBS}) if(ENABLE_SHARED) + set(libcephfs_version 2.0.0) + set(libcephfs_soversion 2) set_target_properties(cephfs PROPERTIES OUTPUT_NAME cephfs - VERSION 2.0.0 - SOVERSION 2) + VERSION ${libcephfs_version} + SOVERSION ${libcephfs_soversion}) if(NOT APPLE AND NOT (WIN32 AND CMAKE_CXX_COMPILER_ID STREQUAL Clang)) foreach(name ceph-common client osdc) @@ -848,6 +857,9 @@ if(WITH_LIBCEPHFS) install(DIRECTORY "include/cephfs" DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + install(FILES + ${CMAKE_BINARY_DIR}/src/cephfs.pc + DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig) set(ceph_syn_srcs ceph_syn.cc client/SyntheticClient.cc) @@ -857,6 +869,7 @@ if(WITH_LIBCEPHFS) if(LINUX) add_subdirectory(mount) endif() + add_subdirectory(libcephfs_proxy) endif(WITH_LIBCEPHFS) if(WITH_LIBCEPHSQLITE) diff --git a/src/bash_completion/radosgw-admin b/src/bash_completion/radosgw-admin index 023a83f87e4..d9e36d8ef29 100644 --- a/src/bash_completion/radosgw-admin +++ b/src/bash_completion/radosgw-admin @@ -19,7 +19,7 @@ _radosgw_admin() if [[ ${cur} == -* ]] ; then COMPREPLY=( $(compgen -W "--uid --subuser --access-key --os-user --email --auth_uid --secret --os-secret --gen-access-key --gen-secret \ - --access --display-name --bucket --object --date --conf --name --id --version -s -w" -- ${cur}) ) + --access --display-name --bucket --object --date --conf --name --id --version -s -w --generate-key" -- ${cur}) ) return 0 fi diff --git a/src/ceph-volume/ceph_volume/devices/lvm/zap.py b/src/ceph-volume/ceph_volume/devices/lvm/zap.py index c278de43eb0..a6d82c7f0fa 100644 --- a/src/ceph-volume/ceph_volume/devices/lvm/zap.py +++ b/src/ceph-volume/ceph_volume/devices/lvm/zap.py @@ -119,13 +119,12 @@ class Zap: osd_uuid = details.get('osd_uuid') break - for osd_uuid, details in raw_report.items(): + for _, details in raw_report.items(): device: str = details.get('device') if details.get('osd_uuid') == osd_uuid: raw_devices.add(device) return list(raw_devices) - def find_associated_devices(self) -> List[api.Volume]: """From an ``osd_id`` and/or an ``osd_fsid``, filter out all the Logical Volumes (LVs) in the diff --git a/src/ceph-volume/ceph_volume/main.py b/src/ceph-volume/ceph_volume/main.py index f8eca65ec49..4f27f429e89 100644 --- a/src/ceph-volume/ceph_volume/main.py +++ b/src/ceph-volume/ceph_volume/main.py @@ -11,8 +11,16 @@ try: from importlib.metadata import entry_points def get_entry_points(group: str): # type: ignore - return entry_points().get(group, []) # type: ignore + eps = entry_points() + if hasattr(eps, 'select'): + # New importlib.metadata uses .select() + return eps.select(group=group) + else: + # Fallback to older EntryPoints that returns dicts + return eps.get(group, []) # type: ignore + except ImportError: + # Fallback to `pkg_resources` for older versions from pkg_resources import iter_entry_points as entry_points # type: ignore def get_entry_points(group: str): # type: ignore diff --git a/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py b/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py index cca64e83ab0..c971b7776ef 100644 --- a/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py +++ b/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py @@ -22,7 +22,7 @@ ceph_bluestore_tool_output = ''' "whoami": "0" }, "/dev/vdx": { - "osd_uuid": "d5a496bc-dcb9-4ad0-a12c-393d3200d2b6", + "osd_uuid": "d5a496bc-dcb9-4ad0-a12c-393d3200d2b7", "size": 214748364800, "btime": "2024-10-16T10:51:05.955279+0000", "description": "main", diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 77b55314f66..921e61a4534 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -347,12 +347,21 @@ def lsblk_all(device: str = '', return result -def is_device(dev): +def is_device(dev: str) -> bool: """ - Boolean to determine if a given device is a block device (**not** - a partition!) + Determines whether the given path corresponds to a block device (not a partition). - For example: /dev/sda would return True, but not /dev/sdc1 + This function checks whether the provided device path represents a valid block device, + such as a physical disk (/dev/sda) or an allowed loop device, but excludes partitions + (/dev/sdc1). It performs several validation steps, including file existence, path format, + device type, and additional checks for loop devices if allowed. + + Args: + dev (str): The path to the device (e.g., "/dev/sda"). + + Returns: + bool: True if the path corresponds to a valid block device (not a partition), + otherwise False. """ if not os.path.exists(dev): return False @@ -364,7 +373,7 @@ def is_device(dev): TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'loop'] # fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev) diff --git a/src/ceph-volume/ceph_volume/util/prepare.py b/src/ceph-volume/ceph_volume/util/prepare.py index 9c863b83d93..ff7fc023fc4 100644 --- a/src/ceph-volume/ceph_volume/util/prepare.py +++ b/src/ceph-volume/ceph_volume/util/prepare.py @@ -9,6 +9,7 @@ import logging import json from ceph_volume import process, conf, terminal from ceph_volume.util import system, constants, str_to_int, disk +from typing import Optional logger = logging.getLogger(__name__) mlogger = terminal.MultiLogger(__name__) @@ -121,7 +122,7 @@ def get_block_wal_size(lv_format=True): return wal_size -def create_id(fsid, json_secrets, osd_id=None): +def create_id(fsid: str, json_secrets: str, osd_id: Optional[str]=None) -> str: """ :param fsid: The osd fsid to create, always required :param json_secrets: a json-ready object with whatever secrets are wanted diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py index d2ddf564116..a8616980e4d 100755 --- a/src/cephadm/cephadm.py +++ b/src/cephadm/cephadm.py @@ -111,6 +111,7 @@ from cephadmlib.file_utils import ( unlink_file, write_new, write_tmp, + update_meta_file, ) from cephadmlib.net_utils import ( build_addrv_params, @@ -3453,6 +3454,7 @@ def list_daemons( detail: bool = True, legacy_dir: Optional[str] = None, daemon_name: Optional[str] = None, + type_of_daemon: Optional[str] = None, ) -> List[Dict[str, str]]: host_version: Optional[str] = None ls = [] @@ -3489,6 +3491,8 @@ def list_daemons( if os.path.exists(data_dir): for i in os.listdir(data_dir): if i in ['mon', 'osd', 'mds', 'mgr', 'rgw']: + if type_of_daemon and type_of_daemon != i: + continue daemon_type = i for j in os.listdir(os.path.join(data_dir, i)): if '-' not in j: @@ -3525,6 +3529,8 @@ def list_daemons( if daemon_name and name != daemon_name: continue (daemon_type, daemon_id) = j.split('.', 1) + if type_of_daemon and type_of_daemon != daemon_type: + continue unit_name = get_unit_name(fsid, daemon_type, daemon_id) @@ -4705,6 +4711,34 @@ def command_list_images(ctx: CephadmContext) -> None: # print default images cp_obj.write(sys.stdout) + +def update_service_for_daemon(ctx: CephadmContext, + available_daemons: list, + update_daemons: list) -> None: + """ Update the unit.meta file of daemon with required service name for valid daemons""" + + data = {'service_name': ctx.service_name} + # check if all the daemon names are valid + if not set(update_daemons).issubset(set(available_daemons)): + raise Error(f'Error EINVAL: one or more daemons of {update_daemons} does not exist on this host') + for name in update_daemons: + path = os.path.join(ctx.data_dir, ctx.fsid, name, 'unit.meta') + update_meta_file(path, data) + print(f'Successfully updated daemon {name} with service {ctx.service_name}') + + +@infer_fsid +def command_update_osd_service(ctx: CephadmContext) -> int: + """update service for provided daemon""" + update_daemons = [f'osd.{osd_id}' for osd_id in ctx.osd_ids.split(',')] + daemons = list_daemons(ctx, detail=False, type_of_daemon='osd') + if not daemons: + raise Error(f'Daemon {ctx.osd_ids} does not exists on this host') + available_daemons = [d['name'] for d in daemons] + update_service_for_daemon(ctx, available_daemons, update_daemons) + return 0 + + ################################## @@ -5571,6 +5605,14 @@ def _get_parser(): parser_list_images = subparsers.add_parser( 'list-images', help='list all the default images') parser_list_images.set_defaults(func=command_list_images) + + parser_update_service = subparsers.add_parser( + 'update-osd-service', help='update service for provided daemon') + parser_update_service.set_defaults(func=command_update_osd_service) + parser_update_service.add_argument('--fsid', help='cluster FSID') + parser_update_service.add_argument('--osd-ids', required=True, help='Comma-separated OSD IDs') + parser_update_service.add_argument('--service-name', required=True, help='OSD service name') + return parser diff --git a/src/cephadm/cephadmlib/daemons/monitoring.py b/src/cephadm/cephadmlib/daemons/monitoring.py index 9a9402632b0..4ba00daaefb 100644 --- a/src/cephadm/cephadmlib/daemons/monitoring.py +++ b/src/cephadm/cephadmlib/daemons/monitoring.py @@ -16,7 +16,13 @@ from ..daemon_form import register as register_daemon_form from ..daemon_identity import DaemonIdentity from ..deployment_utils import to_deployment_container from ..exceptions import Error -from ..net_utils import get_fqdn, get_hostname, get_ip_addresses, wrap_ipv6 +from ..net_utils import ( + get_fqdn, + get_hostname, + get_ip_addresses, + wrap_ipv6, + EndPoint, +) @register_daemon_form @@ -89,11 +95,6 @@ class Monitoring(ContainerDaemonForm): 'image': DefaultImages.ALERTMANAGER.image_ref, 'cpus': '2', 'memory': '2GB', - 'args': [ - '--cluster.listen-address=:{}'.format( - port_map['alertmanager'][1] - ), - ], 'config-json-files': [ 'alertmanager.yml', ], @@ -248,11 +249,14 @@ class Monitoring(ContainerDaemonForm): ip = meta['ip'] if 'ports' in meta and meta['ports']: port = meta['ports'][0] - if daemon_type == 'prometheus': - config = fetch_configs(ctx) + config = fetch_configs(ctx) + if daemon_type in ['prometheus', 'alertmanager']: ip_to_bind_to = config.get('ip_to_bind_to', '') if ip_to_bind_to: ip = ip_to_bind_to + web_listen_addr = str(EndPoint(ip, port)) + r += [f'--web.listen-address={web_listen_addr}'] + if daemon_type == 'prometheus': retention_time = config.get('retention_time', '15d') retention_size = config.get( 'retention_size', '0' @@ -276,9 +280,11 @@ class Monitoring(ContainerDaemonForm): r += ['--web.route-prefix=/prometheus/'] else: r += [f'--web.external-url={scheme}://{host}:{port}'] - r += [f'--web.listen-address={ip}:{port}'] if daemon_type == 'alertmanager': - config = fetch_configs(ctx) + clus_listen_addr = str( + EndPoint(ip, self.port_map[daemon_type][1]) + ) + r += [f'--cluster.listen-address={clus_listen_addr}'] use_url_prefix = config.get('use_url_prefix', False) peers = config.get('peers', list()) # type: ignore for peer in peers: @@ -294,13 +300,11 @@ class Monitoring(ContainerDaemonForm): if daemon_type == 'promtail': r += ['--config.expand-env'] if daemon_type == 'prometheus': - config = fetch_configs(ctx) try: r += [f'--web.config.file={config["web_config"]}'] except KeyError: pass if daemon_type == 'node-exporter': - config = fetch_configs(ctx) try: r += [f'--web.config.file={config["web_config"]}'] except KeyError: diff --git a/src/cephadm/cephadmlib/file_utils.py b/src/cephadm/cephadmlib/file_utils.py index 27e70e31756..4dd88cc3671 100644 --- a/src/cephadm/cephadmlib/file_utils.py +++ b/src/cephadm/cephadmlib/file_utils.py @@ -5,6 +5,7 @@ import datetime import logging import os import tempfile +import json from contextlib import contextmanager from pathlib import Path @@ -157,3 +158,26 @@ def unlink_file( except Exception: if not ignore_errors: raise + + +def update_meta_file(file_path: str, update_key_val: dict) -> None: + """Update key in the file with provided value""" + try: + with open(file_path, 'r') as fh: + data = json.load(fh) + file_stat = os.stat(file_path) + except FileNotFoundError: + raise + except Exception: + logger.exception(f'Failed to update {file_path}') + raise + data.update( + {key: value for key, value in update_key_val.items() if key in data} + ) + + with write_new( + file_path, + owner=(file_stat.st_uid, file_stat.st_gid), + perms=(file_stat.st_mode & 0o777), + ) as fh: + fh.write(json.dumps(data, indent=4) + '\n') diff --git a/src/cephadm/cephadmlib/net_utils.py b/src/cephadm/cephadmlib/net_utils.py index 9a7f138b1c6..bfa61d933ef 100644 --- a/src/cephadm/cephadmlib/net_utils.py +++ b/src/cephadm/cephadmlib/net_utils.py @@ -24,12 +24,22 @@ class EndPoint: def __init__(self, ip: str, port: int) -> None: self.ip = ip self.port = port + self.is_ipv4 = True + try: + if ip and ipaddress.ip_network(ip).version == 6: + self.is_ipv4 = False + except Exception: + logger.exception('Failed to check ip address version') def __str__(self) -> str: - return f'{self.ip}:{self.port}' + if self.is_ipv4: + return f'{self.ip}:{self.port}' + return f'[{self.ip}]:{self.port}' def __repr__(self) -> str: - return f'{self.ip}:{self.port}' + if self.is_ipv4: + return f'{self.ip}:{self.port}' + return f'[{self.ip}]:{self.port}' def attempt_bind(ctx, s, address, port): diff --git a/src/cephadm/tests/test_deploy.py b/src/cephadm/tests/test_deploy.py index c5094db335f..1736639ed55 100644 --- a/src/cephadm/tests/test_deploy.py +++ b/src/cephadm/tests/test_deploy.py @@ -316,7 +316,7 @@ def test_deploy_a_monitoring_container(cephadm_fs, funkypatch): runfile_lines = f.read().splitlines() assert 'podman' in runfile_lines[-1] assert runfile_lines[-1].endswith( - 'quay.io/titans/prometheus:latest --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://10.10.10.10:9095 --web.listen-address=1.2.3.4:9095' + 'quay.io/titans/prometheus:latest --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.listen-address=1.2.3.4:9095 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://10.10.10.10:9095' ) assert '--user 8765' in runfile_lines[-1] assert f'-v /var/lib/ceph/{fsid}/prometheus.fire/etc/prometheus:/etc/prometheus:Z' in runfile_lines[-1] diff --git a/src/cephfs.pc.in b/src/cephfs.pc.in new file mode 100644 index 00000000000..3c5761495bb --- /dev/null +++ b/src/cephfs.pc.in @@ -0,0 +1,10 @@ +prefix=@prefix@ +exec_prefix=${prefix} +libdir=@libdir@ +includedir=@includedir@ + +Name: libcephfs +Description: Ceph distributed file system client library +Version: @libcephfs_version@ +Cflags: -I${includedir}/cephfs -D_FILE_OFFSET_BITS=64 +Libs: -L${libdir} -lcephfs diff --git a/src/client/Client.cc b/src/client/Client.cc index c404057b929..00b85a8e746 100644 --- a/src/client/Client.cc +++ b/src/client/Client.cc @@ -8907,7 +8907,6 @@ int Client::chownat(int dirfd, const char *relpath, uid_t new_uid, gid_t new_gid tout(cct) << new_gid << std::endl; tout(cct) << flags << std::endl; - filepath path(relpath); InodeRef in; InodeRef dirinode; @@ -8917,10 +8916,24 @@ int Client::chownat(int dirfd, const char *relpath, uid_t new_uid, gid_t new_gid return r; } - r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), 0, dirinode); - if (r < 0) { - return r; + if (!strcmp(relpath, "")) { +#if defined(__linux__) && defined(AT_EMPTY_PATH) + if (flags & AT_EMPTY_PATH) { + in = dirinode; + goto out; + } +#endif + return -CEPHFS_ENOENT; + } else { + filepath path(relpath); + r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), 0, dirinode); + if (r < 0) { + return r; + } } + +out: + struct stat attr; attr.st_uid = new_uid; attr.st_gid = new_gid; @@ -11740,8 +11753,12 @@ int64_t Client::_write(Fh *f, int64_t offset, uint64_t size, const char *buf, cond_iofinish = new C_SaferCond(); filer_iofinish.reset(cond_iofinish); } else { - //Register a wrapper callback for the C_Write_Finisher which takes 'client_lock' - filer_iofinish.reset(new C_Lock_Client_Finisher(this, iofinish.get())); + //Register a wrapper callback C_Lock_Client_Finisher for the C_Write_Finisher which takes 'client_lock'. + //Use C_OnFinisher for callbacks. The op_cancel_writes has to be called without 'client_lock' held because + //the callback registered here needs to take it. This would cause incorrect lock order i.e., objecter->rwlock + //taken by objecter's op_cancel and then 'client_lock' taken by callback. To fix the lock order, queue + //the callback using the finisher + filer_iofinish.reset(new C_OnFinisher(new C_Lock_Client_Finisher(this, iofinish.get()), &objecter_finisher)); } get_cap_ref(in, CEPH_CAP_FILE_BUFFER); @@ -12230,6 +12247,7 @@ int Client::statxat(int dirfd, const char *relpath, unsigned mask = statx_to_mask(flags, want); + InodeRef in; InodeRef dirinode; std::scoped_lock lock(client_lock); int r = get_fd_inode(dirfd, &dirinode); @@ -12237,12 +12255,24 @@ int Client::statxat(int dirfd, const char *relpath, return r; } - InodeRef in; - filepath path(relpath); - r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), mask, dirinode); - if (r < 0) { - return r; + if (!strcmp(relpath, "")) { +#if defined(__linux__) && defined(AT_EMPTY_PATH) + if (flags & AT_EMPTY_PATH) { + in = dirinode; + goto out; + } +#endif + return -CEPHFS_ENOENT; + } else { + filepath path(relpath); + r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), mask, dirinode); + if (r < 0) { + return r; + } } + +out: + r = _getattr(in, mask, perms); if (r < 0) { ldout(cct, 3) << __func__ << " exit on error!" << dendl; diff --git a/src/client/Client.h b/src/client/Client.h index f8c39e2fdd6..d5108c12262 100644 --- a/src/client/Client.h +++ b/src/client/Client.h @@ -219,6 +219,7 @@ struct dir_result_t { ordered_count = 0; cache_index = 0; buffer.clear(); + fd = -1; } InodeRef inode; diff --git a/src/cls/rgw/cls_rgw_types.cc b/src/cls/rgw/cls_rgw_types.cc index d5f6ba4bdee..9fd60aaff3f 100644 --- a/src/cls/rgw/cls_rgw_types.cc +++ b/src/cls/rgw/cls_rgw_types.cc @@ -194,7 +194,9 @@ void rgw_bucket_dir_entry_meta::dump(Formatter *f) const utime_t ut(mtime); encode_json("mtime", ut, f); encode_json("etag", etag, f); - encode_json("storage_class", storage_class, f); + encode_json("storage_class", + rgw_placement_rule::get_canonical_storage_class(storage_class), + f); encode_json("owner", owner, f); encode_json("owner_display_name", owner_display_name, f); encode_json("content_type", content_type, f); diff --git a/src/common/CMakeLists.txt b/src/common/CMakeLists.txt index ea3cce16609..c607839a8d2 100644 --- a/src/common/CMakeLists.txt +++ b/src/common/CMakeLists.txt @@ -13,6 +13,7 @@ if(WIN32) endif() add_subdirectory(io_exerciser) +add_subdirectory(json) add_subdirectory(options) set(common_srcs diff --git a/src/common/Formatter.cc b/src/common/Formatter.cc index fd3b2be0221..cd12e4f9885 100644 --- a/src/common/Formatter.cc +++ b/src/common/Formatter.cc @@ -23,6 +23,8 @@ #include <algorithm> #include <set> #include <limits> +#include <utility> +#include <boost/container/small_vector.hpp> // ----------------------- namespace ceph { @@ -365,10 +367,21 @@ std::ostream& JSONFormatter::dump_stream(std::string_view name) void JSONFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; - add_value(name, buf, quoted); + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } + + add_value(name, buf.data(), quoted); } int JSONFormatter::get_len() const @@ -550,15 +563,27 @@ std::ostream& XMLFormatter::dump_stream(std::string_view name) void XMLFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - size_t len = vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } + auto e = get_xml_name(name); print_spaces(); if (ns) { - m_ss << "<" << e << " xmlns=\"" << ns << "\">" << xml_stream_escaper(std::string_view(buf, len)) << "</" << e << ">"; + m_ss << "<" << e << " xmlns=\"" << ns << "\">" << xml_stream_escaper(std::string_view(buf.data(), len)) << "</" << e << ">"; } else { - m_ss << "<" << e << ">" << xml_stream_escaper(std::string_view(buf, len)) << "</" << e << ">"; + m_ss << "<" << e << ">" << xml_stream_escaper(std::string_view(buf.data(), len)) << "</" << e << ">"; } if (m_pretty) @@ -927,14 +952,26 @@ void TableFormatter::dump_format_va(std::string_view name, const char *fmt, va_list ap) { finish_pending_string(); - char buf[LARGE_SIZE]; - vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } size_t i = m_vec_index(name); if (ns) { - m_ss << ns << "." << buf; - } else - m_ss << buf; + m_ss << ns << "." << buf.data(); + } else { + m_ss << buf.data(); + } m_vec[i].push_back(std::make_pair(get_section_name(name), m_ss.str())); m_ss.clear(); diff --git a/src/common/HTMLFormatter.cc b/src/common/HTMLFormatter.cc index e7e985531d8..1bc8d864cb6 100644 --- a/src/common/HTMLFormatter.cc +++ b/src/common/HTMLFormatter.cc @@ -23,6 +23,8 @@ #include <stdlib.h> #include <string> #include <string.h> // for strdup +#include <boost/container/small_vector.hpp> +#include <utility> // for std::cmp_greater_equal #include "common/escape.h" @@ -138,17 +140,27 @@ std::ostream& HTMLFormatter::dump_stream(std::string_view name) void HTMLFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - size_t len = vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + size_t len = vsnprintf(buf.data(), buf.size(), fmt, ap); + va_end(ap_copy); + + if(std::cmp_greater_equal(len, buf.size())){ + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + } std::string e(name); print_spaces(); if (ns) { m_ss << "<li xmlns=\"" << ns << "\">" << e << ": " - << xml_stream_escaper(std::string_view(buf, len)) << "</li>"; + << xml_stream_escaper(std::string_view(buf.data(), len)) << "</li>"; } else { m_ss << "<li>" << e << ": " - << xml_stream_escaper(std::string_view(buf, len)) << "</li>"; + << xml_stream_escaper(std::string_view(buf.data(), len)) << "</li>"; } if (m_pretty) diff --git a/src/common/bit_vector.hpp b/src/common/bit_vector.hpp index 961d9a0192e..c5fd491ed29 100644 --- a/src/common/bit_vector.hpp +++ b/src/common/bit_vector.hpp @@ -29,8 +29,8 @@ private: static const uint8_t MASK = static_cast<uint8_t>((1 << _bit_count) - 1); // must be power of 2 - BOOST_STATIC_ASSERT((_bit_count != 0) && !(_bit_count & (_bit_count - 1))); - BOOST_STATIC_ASSERT(_bit_count <= BITS_PER_BYTE); + static_assert((_bit_count != 0) && !(_bit_count & (_bit_count - 1))); + static_assert(_bit_count <= BITS_PER_BYTE); template <typename DataIterator> class ReferenceImpl { diff --git a/src/common/ceph_time.h b/src/common/ceph_time.h index 01feff4c063..0b05be5372e 100644 --- a/src/common/ceph_time.h +++ b/src/common/ceph_time.h @@ -342,6 +342,23 @@ public: } }; +// Please note time_guard is not thread safety -- multiple threads +// updating same diff_accumulator can corrupt it. +template <class ClockT = mono_clock> +class time_guard { + const typename ClockT::time_point start; + timespan& diff_accumulator; + +public: + time_guard(timespan& diff_accumulator) + : start(ClockT::now()), + diff_accumulator(diff_accumulator) { + } + ~time_guard() { + diff_accumulator += ClockT::now() - start; + } +}; + namespace time_detail { // So that our subtractions produce negative spans rather than // arithmetic underflow. diff --git a/src/common/intrusive_lru.h b/src/common/intrusive_lru.h index 3ed3625d8a0..f9132773e4d 100644 --- a/src/common/intrusive_lru.h +++ b/src/common/intrusive_lru.h @@ -94,6 +94,8 @@ public: friend void intrusive_ptr_add_ref<>(intrusive_lru_base<Config> *); friend void intrusive_ptr_release<>(intrusive_lru_base<Config> *); + unsigned get_use_count() const { return use_count; } + virtual ~intrusive_lru_base() {} }; @@ -176,6 +178,25 @@ class intrusive_lru { evict(lru_target_size); } + /// clear [from, to) invoking f upon and invalidating any live references + template <typename F> + void clear_range( + typename lru_set_t::iterator from, typename lru_set_t::iterator to, + F &&f) { + while (from != to) { + if (!(*from).lru) { + unreferenced_list.erase(lru_list_t::s_iterator_to(*from)); + from = lru_set.erase_and_dispose(from, [](auto *p) { delete p; } ); + } else { + std::invoke(f, static_cast<T&>(*from)); + from->lru = nullptr; + assert(from->is_invalidated()); + from = lru_set.erase_and_dispose( + from, [](auto *p) { assert(p->use_count > 0); }); + } + } + } + public: /** * Returns the TRef corresponding to k if it exists or @@ -198,38 +219,28 @@ public: } } - /* - * Clears unreferenced elements from the lru set [from, to] + /** + * clear_range + * + * Clears elements from the lru set in [from, to] invoking F upon and + * invalidating any with outstanding references */ - void clear_range( - const K& from, - const K& to) { - auto from_iter = lru_set.lower_bound(from); - auto to_iter = lru_set.upper_bound(to); - for (auto i = from_iter; i != to_iter; ) { - if (!(*i).lru) { - unreferenced_list.erase(lru_list_t::s_iterator_to(*i)); - i = lru_set.erase_and_dispose(i, [](auto *p) - { delete p; } ); - } else { - i++; - } - } + template <typename F> + void clear_range(const K& from, const K& to, F &&f) { + auto from_iter = lru_set.lower_bound(from); + auto to_iter = lru_set.upper_bound(to); + clear_range(from_iter, to_iter, std::forward<F>(f)); } - /// drop all elements from lru, invoke f on any with outstanding references + /** + * clear + * + * Clears all elements from the lru set invoking F upon and + * invalidating any with outstanding references + */ template <typename F> void clear(F &&f) { - evict(0); - assert(unreferenced_list.empty()); - for (auto &i: lru_set) { - std::invoke(f, static_cast<T&>(i)); - i.lru = nullptr; - assert(i.is_invalidated()); - } - lru_set.clear_and_dispose([](auto *i){ - assert(i->use_count > 0); /* don't delete, still has a ref count */ - }); + clear_range(lru_set.begin(), lru_set.end(), std::forward<F>(f)); } template <class F> diff --git a/src/common/io_exerciser/CMakeLists.txt b/src/common/io_exerciser/CMakeLists.txt index 07091df86e1..ab2e64fc222 100644 --- a/src/common/io_exerciser/CMakeLists.txt +++ b/src/common/io_exerciser/CMakeLists.txt @@ -5,9 +5,11 @@ add_library(object_io_exerciser STATIC Model.cc ObjectModel.cc RadosIo.cc + EcIoSequence.cc ) target_link_libraries(object_io_exerciser - librados + librados global + json_structures )
\ No newline at end of file diff --git a/src/common/io_exerciser/DataGenerator.cc b/src/common/io_exerciser/DataGenerator.cc index 9aa77eeb6e9..701c32fa9ec 100644 --- a/src/common/io_exerciser/DataGenerator.cc +++ b/src/common/io_exerciser/DataGenerator.cc @@ -2,32 +2,28 @@ // vim: ts=8 sw=2 smarttab #include "DataGenerator.h" -#include "ObjectModel.h" +#include <chrono> +#include <iostream> +#include <stdexcept> +#include "ObjectModel.h" #include "common/debug.h" #include "common/dout.h" - #include "fmt/format.h" #include "fmt/ranges.h" -#include <chrono> -#include <iostream> -#include <stdexcept> - #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context using DataGenerator = ceph::io_exerciser::data_generation::DataGenerator; -using SeededRandomGenerator = ceph::io_exerciser::data_generation - ::SeededRandomGenerator; -using HeaderedSeededRandomGenerator = ceph::io_exerciser::data_generation - ::HeaderedSeededRandomGenerator; +using SeededRandomGenerator = + ceph::io_exerciser::data_generation ::SeededRandomGenerator; +using HeaderedSeededRandomGenerator = + ceph::io_exerciser::data_generation ::HeaderedSeededRandomGenerator; std::unique_ptr<DataGenerator> DataGenerator::create_generator( - GenerationType generationType, const ObjectModel& model) -{ - switch(generationType) - { + GenerationType generationType, const ObjectModel& model) { + switch (generationType) { case GenerationType::SeededRandom: return std::make_unique<SeededRandomGenerator>(model); case GenerationType::HeaderedSeededRandom: @@ -39,28 +35,25 @@ std::unique_ptr<DataGenerator> DataGenerator::create_generator( return nullptr; } -bufferlist DataGenerator::generate_wrong_data(uint64_t offset, uint64_t length) -{ +bufferlist DataGenerator::generate_wrong_data(uint64_t offset, + uint64_t length) { bufferlist retlist; uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; - for (uint64_t block_offset = offset; - block_offset < offset + length; - block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { std::memset(buffer, 0, block_size); retlist.append(ceph::bufferptr(buffer, block_size)); } return retlist; } -bool DataGenerator::validate(bufferlist& bufferlist, uint64_t offset, uint64_t length) -{ +bool DataGenerator::validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length) { return bufferlist.contents_equal(generate_data(offset, length)); } -ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) -{ +ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) { uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; @@ -70,29 +63,26 @@ ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) constexpr size_t generation_length = sizeof(uint64_t); - for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) - { + for (uint64_t i = 0; i < block_size; + i += (2 * generation_length), rand1++, rand2--) { std::memcpy(buffer + i, &rand1, generation_length); std::memcpy(buffer + i + generation_length, &rand2, generation_length); } size_t remainingBytes = block_size % (generation_length * 2); - if (remainingBytes > generation_length) - { + if (remainingBytes > generation_length) { size_t remainingBytes2 = remainingBytes - generation_length; std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); - } - else if (remainingBytes > 0) - { + } else if (remainingBytes > 0) { std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); } return ceph::bufferptr(buffer, block_size); } -ceph::bufferptr SeededRandomGenerator::generate_wrong_block(uint64_t block_offset) -{ +ceph::bufferptr SeededRandomGenerator::generate_wrong_block( + uint64_t block_offset) { uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; @@ -102,141 +92,134 @@ ceph::bufferptr SeededRandomGenerator::generate_wrong_block(uint64_t block_offse constexpr size_t generation_length = sizeof(uint64_t); - for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) - { + for (uint64_t i = 0; i < block_size; + i += (2 * generation_length), rand1++, rand2--) { std::memcpy(buffer + i, &rand1, generation_length); std::memcpy(buffer + i + generation_length, &rand2, generation_length); } size_t remainingBytes = block_size % (generation_length * 2); - if (remainingBytes > generation_length) - { + if (remainingBytes > generation_length) { size_t remainingBytes2 = remainingBytes - generation_length; std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); - } - else if (remainingBytes > 0) - { + } else if (remainingBytes > 0) { std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); } return ceph::bufferptr(buffer, block_size); } -bufferlist SeededRandomGenerator::generate_data(uint64_t offset, uint64_t length) -{ +bufferlist SeededRandomGenerator::generate_data(uint64_t offset, + uint64_t length) { bufferlist retlist; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { retlist.append(generate_block(block_offset)); } return retlist; } -bufferlist SeededRandomGenerator::generate_wrong_data(uint64_t offset, uint64_t length) -{ +bufferlist SeededRandomGenerator::generate_wrong_data(uint64_t offset, + uint64_t length) { bufferlist retlist; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { retlist.append(generate_wrong_block(block_offset)); } return retlist; } -HeaderedSeededRandomGenerator - ::HeaderedSeededRandomGenerator(const ObjectModel& model, - std::optional<uint64_t> unique_run_id) : - SeededRandomGenerator(model), - unique_run_id(unique_run_id.value_or(generate_unique_run_id())) -{ - -} +HeaderedSeededRandomGenerator ::HeaderedSeededRandomGenerator( + const ObjectModel& model, std::optional<uint64_t> unique_run_id) + : SeededRandomGenerator(model), + unique_run_id(unique_run_id.value_or(generate_unique_run_id())) {} -uint64_t HeaderedSeededRandomGenerator::generate_unique_run_id() -{ +uint64_t HeaderedSeededRandomGenerator::generate_unique_run_id() { std::mt19937_64 random_generator = - std::mt19937_64(duration_cast<std::chrono::milliseconds>( - std::chrono::system_clock::now().time_since_epoch()).count()); + std::mt19937_64(duration_cast<std::chrono::milliseconds>( + std::chrono::system_clock::now().time_since_epoch()) + .count()); - return random_generator(); + return random_generator(); } -ceph::bufferptr HeaderedSeededRandomGenerator::generate_block(uint64_t block_offset) -{ +ceph::bufferptr HeaderedSeededRandomGenerator::generate_block( + uint64_t block_offset) { SeedBytes seed = m_model.get_seed(block_offset); - TimeBytes current_time = duration_cast<std::chrono::milliseconds>( - std::chrono::system_clock::now().time_since_epoch()).count(); + TimeBytes current_time = + duration_cast<std::chrono::milliseconds>( + std::chrono::system_clock::now().time_since_epoch()) + .count(); - ceph::bufferptr bufferptr = SeededRandomGenerator::generate_block(block_offset); + ceph::bufferptr bufferptr = + SeededRandomGenerator::generate_block(block_offset); - std::memcpy(bufferptr.c_str() + uniqueIdStart(), &unique_run_id, uniqueIdLength()); + std::memcpy(bufferptr.c_str() + uniqueIdStart(), &unique_run_id, + uniqueIdLength()); std::memcpy(bufferptr.c_str() + seedStart(), &seed, seedLength()); std::memcpy(bufferptr.c_str() + timeStart(), ¤t_time, timeLength()); return bufferptr; } -ceph::bufferptr HeaderedSeededRandomGenerator::generate_wrong_block(uint64_t block_offset) -{ +ceph::bufferptr HeaderedSeededRandomGenerator::generate_wrong_block( + uint64_t block_offset) { return HeaderedSeededRandomGenerator::generate_block(block_offset % 8); } const HeaderedSeededRandomGenerator::UniqueIdBytes - HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist) { UniqueIdBytes read_unique_run_id = 0; - std::memcpy(&read_unique_run_id, - &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()], - uniqueIdLength()); + std::memcpy( + &read_unique_run_id, + &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()], + uniqueIdLength()); return read_unique_run_id; } const HeaderedSeededRandomGenerator::SeedBytes - HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset, + const bufferlist& bufferlist) { SeedBytes read_seed = 0; - std::memcpy(&read_seed, - &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()], - seedLength()); + std::memcpy( + &read_seed, + &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()], + seedLength()); return read_seed; } const HeaderedSeededRandomGenerator::TimeBytes - HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset, + const bufferlist& bufferlist) { TimeBytes read_time = 0; - std::memcpy(&read_time, - &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()], - timeLength()); + std::memcpy( + &read_time, + &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()], + timeLength()); return read_time; } bool HeaderedSeededRandomGenerator::validate(bufferlist& bufferlist, - uint64_t offset, uint64_t length) -{ + uint64_t offset, uint64_t length) { std::vector<uint64_t> invalid_block_offsets; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { - bool valid_block - = validate_block(block_offset, - (bufferlist.c_str() + ((block_offset - offset) * - m_model.get_block_size()))); - if (!valid_block) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { + bool valid_block = validate_block( + block_offset, (bufferlist.c_str() + + ((block_offset - offset) * m_model.get_block_size()))); + if (!valid_block) { invalid_block_offsets.push_back(block_offset); } } - if (!invalid_block_offsets.empty()) - { + if (!invalid_block_offsets.empty()) { printDebugInformationForOffsets(offset, invalid_block_offsets, bufferlist); } @@ -244,59 +227,51 @@ bool HeaderedSeededRandomGenerator::validate(bufferlist& bufferlist, } bool HeaderedSeededRandomGenerator::validate_block(uint64_t block_offset, - const char* buffer_start) -{ + const char* buffer_start) { // We validate the block matches what we generate byte for byte // however we ignore the time section of the header ceph::bufferptr bufferptr = generate_block(block_offset); bool valid = strncmp(bufferptr.c_str(), buffer_start, timeStart()) == 0; - valid = valid ? strncmp(bufferptr.c_str() + timeEnd(), - buffer_start + timeEnd(), - m_model.get_block_size() - timeEnd()) == 0 : valid; + valid = valid + ? strncmp(bufferptr.c_str() + timeEnd(), buffer_start + timeEnd(), + m_model.get_block_size() - timeEnd()) == 0 + : valid; return valid; } const HeaderedSeededRandomGenerator::ErrorType - HeaderedSeededRandomGenerator::getErrorTypeForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist) -{ - try - { - UniqueIdBytes read_unique_run_id = readUniqueRunId(block_offset - read_offset, - bufferlist); - if (unique_run_id != read_unique_run_id) - { +HeaderedSeededRandomGenerator::getErrorTypeForBlock( + uint64_t read_offset, uint64_t block_offset, const bufferlist& bufferlist) { + try { + UniqueIdBytes read_unique_run_id = + readUniqueRunId(block_offset - read_offset, bufferlist); + if (unique_run_id != read_unique_run_id) { return ErrorType::RUN_ID_MISMATCH; } SeedBytes read_seed = readSeed(block_offset - read_offset, bufferlist); - if (m_model.get_seed(block_offset) != read_seed) - { + if (m_model.get_seed(block_offset) != read_seed) { return ErrorType::SEED_MISMATCH; } if (std::strncmp(&bufferlist[((block_offset - read_offset) * - m_model.get_block_size()) + bodyStart()], + m_model.get_block_size()) + + bodyStart()], generate_block(block_offset).c_str() + bodyStart(), - m_model.get_block_size() - bodyStart()) != 0) - { + m_model.get_block_size() - bodyStart()) != 0) { return ErrorType::DATA_MISMATCH; } - } - catch(const std::exception& e) - { + } catch (const std::exception& e) { return ErrorType::DATA_NOT_FOUND; } return ErrorType::UNKNOWN; } -void HeaderedSeededRandomGenerator - ::printDebugInformationForBlock(uint64_t read_offset, uint64_t block_offset, - const bufferlist& bufferlist) -{ - ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, bufferlist); +void HeaderedSeededRandomGenerator ::printDebugInformationForBlock( + uint64_t read_offset, uint64_t block_offset, const bufferlist& bufferlist) { + ErrorType blockError = + getErrorTypeForBlock(read_offset, block_offset, bufferlist); TimeBytes read_time = 0; std::time_t ttp; @@ -304,433 +279,361 @@ void HeaderedSeededRandomGenerator char read_bytes[m_model.get_block_size()]; char generated_bytes[m_model.get_block_size()]; - if (blockError == ErrorType::DATA_MISMATCH || blockError == ErrorType::UNKNOWN) - { + if (blockError == ErrorType::DATA_MISMATCH || + blockError == ErrorType::UNKNOWN) { read_time = readDateTime(block_offset - read_offset, bufferlist); - std::chrono::system_clock::time_point time_point{std::chrono::milliseconds{read_time}}; + std::chrono::system_clock::time_point time_point{ + std::chrono::milliseconds{read_time}}; ttp = std::chrono::system_clock::to_time_t(time_point); - std::memcpy(&read_bytes, - &bufferlist[((block_offset - read_offset) * m_model.get_block_size())], - m_model.get_block_size() - bodyStart()); - std::memcpy(&generated_bytes, - generate_block(block_offset).c_str(), + std::memcpy( + &read_bytes, + &bufferlist[((block_offset - read_offset) * m_model.get_block_size())], + m_model.get_block_size() - bodyStart()); + std::memcpy(&generated_bytes, generate_block(block_offset).c_str(), m_model.get_block_size() - bodyStart()); } std::string error_string; - switch(blockError) - { - case ErrorType::RUN_ID_MISMATCH: - { - UniqueIdBytes read_unique_run_id = readUniqueRunId((block_offset - read_offset), - bufferlist); - error_string = fmt::format("Header (Run ID) mismatch detected at block {} " - "(byte offset {}) Header expected run id {} but found id {}. " - "Block data corrupt or not written from this instance of this application.", - block_offset, - block_offset * m_model.get_block_size(), - unique_run_id, - read_unique_run_id); - } - break; - - case ErrorType::SEED_MISMATCH: - { + switch (blockError) { + case ErrorType::RUN_ID_MISMATCH: { + UniqueIdBytes read_unique_run_id = + readUniqueRunId((block_offset - read_offset), bufferlist); + error_string = fmt::format( + "Header (Run ID) mismatch detected at block {} " + "(byte offset {}) Header expected run id {} but found id {}. " + "Block data corrupt or not written from this instance of this " + "application.", + block_offset, block_offset * m_model.get_block_size(), unique_run_id, + read_unique_run_id); + } break; + + case ErrorType::SEED_MISMATCH: { SeedBytes read_seed = readSeed((block_offset - read_offset), bufferlist); - if (m_model.get_seed_offsets(read_seed).size() == 0) - { - error_string = fmt::format("Data (Seed) mismatch detected at block {}" - " (byte offset {}). Header expected seed {} but found seed {}. " - "Read data was not from any other recognised block in the object.", - block_offset, - block_offset * m_model.get_block_size(), - m_model.get_seed(block_offset), - read_seed); - } - else - { + if (m_model.get_seed_offsets(read_seed).size() == 0) { + error_string = fmt::format( + "Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}. " + "Read data was not from any other recognised block in the object.", + block_offset, block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), read_seed); + } else { std::vector<int> seed_offsets = m_model.get_seed_offsets(read_seed); - error_string = fmt::format("Data (Seed) mismatch detected at block {}" - " (byte offset {}). Header expected seed {} but found seed {}." - " Read data was from a different block(s): {}", - block_offset, - block_offset * m_model.get_block_size(), - m_model.get_seed(block_offset), - read_seed, + error_string = fmt::format( + "Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}." + " Read data was from a different block(s): {}", + block_offset, block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), read_seed, fmt::join(seed_offsets.begin(), seed_offsets.end(), "")); } - } - break; - - case ErrorType::DATA_MISMATCH: - { - error_string = fmt::format("Data (Body) mismatch detected at block {}" - " (byte offset {}). Header data matches, data body does not." - " Data written at {}\nExpected data: \n{:02x}\nRead data:{:02x}", - block_offset, - block_offset * m_model.get_block_size(), + } break; + + case ErrorType::DATA_MISMATCH: { + error_string = fmt::format( + "Data (Body) mismatch detected at block {}" + " (byte offset {}). Header data matches, data body does not." + " Data written at {}\nExpected data: \n{:02x}\nRead data:{:02x}", + block_offset, block_offset * m_model.get_block_size(), std::ctime(&ttp), - fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), + ""), fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); - } - break; + } break; - case ErrorType::DATA_NOT_FOUND: - { + case ErrorType::DATA_NOT_FOUND: { uint64_t bufferlist_length = bufferlist.to_str().size(); - error_string = fmt::format("Data (Body) could not be read at block {}" - " (byte offset {}) offset in bufferlist returned from read: {}" - " ({} bytes). Returned bufferlist length: {}.", - block_offset, - block_offset * m_model.get_block_size(), + error_string = fmt::format( + "Data (Body) could not be read at block {}" + " (byte offset {}) offset in bufferlist returned from read: {}" + " ({} bytes). Returned bufferlist length: {}.", + block_offset, block_offset * m_model.get_block_size(), (block_offset - read_offset), (block_offset - read_offset) * m_model.get_block_size(), bufferlist_length); - } - break; + } break; case ErrorType::UNKNOWN: - [[ fallthrough ]]; - - default: - { - error_string = fmt::format("Data mismatch detected at block {}" - " (byte offset {}).\nExpected data:\n{:02x}\nRead data:\n{:02x}", - block_offset, - block_offset * m_model.get_block_size(), - fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + [[fallthrough]]; + + default: { + error_string = fmt::format( + "Data mismatch detected at block {}" + " (byte offset {}).\nExpected data:\n{:02x}\nRead data:\n{:02x}", + block_offset, block_offset * m_model.get_block_size(), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), + ""), fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); - } - break; + } break; } dout(0) << error_string << dendl; } -void HeaderedSeededRandomGenerator - ::printDebugInformationForRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - ErrorType rangeError, - const bufferlist& bufferlist) -{ - switch(rangeError) - { - case ErrorType::RUN_ID_MISMATCH: - printDebugInformationForRunIdMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::SEED_MISMATCH: - printDebugInformationForSeedMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::DATA_MISMATCH: - printDebugInformationDataBodyMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::DATA_NOT_FOUND: - printDebugInformationDataNotFoundRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::UNKNOWN: - [[ fallthrough ]]; - default: - printDebugInformationCorruptRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; +void HeaderedSeededRandomGenerator ::printDebugInformationForRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, ErrorType rangeError, + const bufferlist& bufferlist) { + switch (rangeError) { + case ErrorType::RUN_ID_MISMATCH: + printDebugInformationForRunIdMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::SEED_MISMATCH: + printDebugInformationForSeedMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_MISMATCH: + printDebugInformationDataBodyMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_NOT_FOUND: + printDebugInformationDataNotFoundRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::UNKNOWN: + [[fallthrough]]; + default: + printDebugInformationCorruptRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForRunIdMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForRunIdMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { uint64_t range_start = start_block_offset; uint64_t range_length = 0; - UniqueIdBytes initial_read_unique_run_id = readUniqueRunId(start_block_offset - read_offset, - bufferlist); + UniqueIdBytes initial_read_unique_run_id = + readUniqueRunId(start_block_offset - read_offset, bufferlist); for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { - ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) - == ErrorType::RUN_ID_MISMATCH); + i < start_block_offset + range_length_in_blocks; i++) { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) == + ErrorType::RUN_ID_MISMATCH); - UniqueIdBytes read_unique_run_id = readUniqueRunId(i - read_offset, bufferlist); + UniqueIdBytes read_unique_run_id = + readUniqueRunId(i - read_offset, bufferlist); if (initial_read_unique_run_id != read_unique_run_id || - i == (start_block_offset + range_length_in_blocks - 1)) - { - if (range_length == 1) - { + i == (start_block_offset + range_length_in_blocks - 1)) { + if (range_length == 1) { printDebugInformationForBlock(read_offset, i, bufferlist); - } - else if (range_length > 1) - { - dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {} ({} bytes)" - " and spanning a range of {} blocks ({} bytes). " - "Expected run id {} for range but found id {}" - " for all blocks in range. " - "Block data corrupt or not written from this instance of this application.", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - unique_run_id, - initial_read_unique_run_id) << dendl; + } else if (range_length > 1) { + dout(0) + << fmt::format( + "Data (Run ID) Mismatch detected from block {} ({} bytes)" + " and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id {}" + " for all blocks in range. " + "Block data corrupt or not written from this instance of " + "this application.", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + unique_run_id, initial_read_unique_run_id) + << dendl; } range_start = i; range_length = 1; initial_read_unique_run_id = read_unique_run_id; - } - else - { + } else { range_length++; } } - if (range_length == 1) - { - printDebugInformationForBlock(read_offset, - start_block_offset + range_length_in_blocks - 1, - bufferlist); - } - else if (range_length > 1) - { - dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Expected run id {} for range but found id for all blocks in range. " - "Block data corrupt or not written from this instance of this application.", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - unique_run_id, - initial_read_unique_run_id) + if (range_length == 1) { + printDebugInformationForBlock( + read_offset, start_block_offset + range_length_in_blocks - 1, + bufferlist); + } else if (range_length > 1) { + dout(0) << fmt::format( + "Data (Run ID) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id for all blocks " + "in range. " + "Block data corrupt or not written from this instance of " + "this application.", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + unique_run_id, initial_read_unique_run_id) << dendl; } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForSeedMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForSeedMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { uint64_t range_start = start_block_offset; uint64_t range_length = 0; // Assert here if needed, as we can't support values // that can't be converted to a signed integer. - ceph_assert(m_model.get_block_size() < (std::numeric_limits<uint64_t>::max() / 2)); + ceph_assert(m_model.get_block_size() < + (std::numeric_limits<uint64_t>::max() / 2)); std::optional<int64_t> range_offset = 0; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { - ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) - == ErrorType::SEED_MISMATCH); + i < start_block_offset + range_length_in_blocks; i++) { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) == + ErrorType::SEED_MISMATCH); SeedBytes read_seed = readSeed(i - read_offset, bufferlist); std::vector<int> seed_found_offsets = m_model.get_seed_offsets(read_seed); if ((seed_found_offsets.size() == 1 && - (static_cast<int64_t>(seed_found_offsets.front() - i) == range_offset)) || - range_length == 0) - { - if (range_length == 0) - { + (static_cast<int64_t>(seed_found_offsets.front() - i) == + range_offset)) || + range_length == 0) { + if (range_length == 0) { range_start = i; - if (seed_found_offsets.size() > 0) - { + if (seed_found_offsets.size() > 0) { range_offset = seed_found_offsets.front() - i; - } - else - { + } else { range_offset = std::nullopt; } } range_length++; - } - else - { - if (range_length == 1) - { + } else { + if (range_length == 1) { printDebugInformationForBlock(read_offset, i - 1, bufferlist); - } - else if (range_length > 1 && range_offset.has_value()) - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Returned data located starting from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, range_length * m_model.get_block_size(), - static_cast<uint64_t>(*range_offset) + range_start, - (static_cast<uint64_t>(*range_offset) + range_start) - * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) - << dendl; - } - else - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Data seed mismatch spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, range_length * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) - << dendl; + } else if (range_length > 1 && range_offset.has_value()) { + dout(0) + << fmt::format( + "Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + static_cast<uint64_t>(*range_offset) + range_start, + (static_cast<uint64_t>(*range_offset) + range_start) * + m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) + << dendl; + } else { + dout(0) + << fmt::format( + "Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Data seed mismatch spanning a range of {} blocks ({} " + "bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) + << dendl; } range_length = 1; range_start = i; - if (seed_found_offsets.size() > 0) - { + if (seed_found_offsets.size() > 0) { range_offset = seed_found_offsets.front() - i; - } - else - { + } else { range_offset = std::nullopt; } } } - if (range_length == 1) - { - printDebugInformationForBlock(read_offset, - start_block_offset + range_length_in_blocks - 1, - bufferlist); - } - else if (range_length > 1 && range_offset.has_value()) - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes). " - "Returned data located starting from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - *range_offset + range_start, - (*range_offset + range_start) * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) + if (range_length == 1) { + printDebugInformationForBlock( + read_offset, start_block_offset + range_length_in_blocks - 1, + bufferlist); + } else if (range_length > 1 && range_offset.has_value()) { + dout(0) << fmt::format( + "Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + *range_offset + range_start, + (*range_offset + range_start) * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) << dendl; - } - else - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes). " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) + } else { + dout(0) << fmt::format( + "Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) << dendl; } } -void HeaderedSeededRandomGenerator -::printDebugInformationDataBodyMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " - "Headers look as expected for range, " - "but generated data body does not match. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationDataBodyMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationCorruptRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " - "Headers look as expected for range, " - "but generated data body does not match. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationCorruptRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationDataNotFoundRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data not found for blocks from {} to {}. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationDataNotFoundRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data not found for blocks from {} to {}. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; - for (uint64_t i = start_block_offset; i < start_block_offset + range_length_in_blocks; i++) - { + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForOffsets(uint64_t read_offset, - std::vector<uint64_t> offsets, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForOffsets( + uint64_t read_offset, std::vector<uint64_t> offsets, + const bufferlist& bufferlist) { uint64_t range_start = 0; uint64_t range_length = 0; ErrorType rangeError = ErrorType::UNKNOWN; - for (const uint64_t& block_offset : offsets) - { - ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, - bufferlist); + for (const uint64_t& block_offset : offsets) { + ErrorType blockError = + getErrorTypeForBlock(read_offset, block_offset, bufferlist); - if (range_start == 0 && range_length == 0) - { + if (range_start == 0 && range_length == 0) { range_start = block_offset; range_length = 1; rangeError = blockError; - } - else if (blockError == rangeError && - range_start + range_length == block_offset) -{ + } else if (blockError == rangeError && + range_start + range_length == block_offset) { range_length++; - } - else - { - if (range_length == 1) - { + } else { + if (range_length == 1) { printDebugInformationForBlock(read_offset, range_start, bufferlist); - } - else if (range_length > 1) - { + } else if (range_length > 1) { printDebugInformationForRange(read_offset, range_start, range_length, rangeError, bufferlist); } @@ -741,12 +644,9 @@ void HeaderedSeededRandomGenerator } } - if (range_length == 1) - { + if (range_length == 1) { printDebugInformationForBlock(read_offset, range_start, bufferlist); - } - else if (range_length > 1) - { + } else if (range_length > 1) { printDebugInformationForRange(read_offset, range_start, range_length, rangeError, bufferlist); } diff --git a/src/common/io_exerciser/DataGenerator.h b/src/common/io_exerciser/DataGenerator.h index 1e5784a54cc..c497c78ed61 100644 --- a/src/common/io_exerciser/DataGenerator.h +++ b/src/common/io_exerciser/DataGenerator.h @@ -3,8 +3,8 @@ #include <memory> #include <random> -#include "include/buffer.h" #include "ObjectModel.h" +#include "include/buffer.h" /* Overview * @@ -23,149 +23,139 @@ * * class HeaderedSeededRandomGenerator * Inherits from SeededDataGenerator. Generates entirely random patterns - * based on the seed retrieved by the model, however also appends a + * based on the seed retrieved by the model, however also appends a * header to the start of each block. This generator also provides * a range of verbose debug options to help disagnose a miscompare * whenever it detects unexpected data. */ namespace ceph { - namespace io_exerciser { - namespace data_generation { - enum class GenerationType { - SeededRandom, - HeaderedSeededRandom - // CompressedGenerator - // MixedGenerator - }; - - class DataGenerator { - public: - virtual ~DataGenerator() = default; - static std::unique_ptr<DataGenerator> - create_generator(GenerationType generatorType, - const ObjectModel& model); - virtual bufferlist generate_data(uint64_t length, uint64_t offset)=0; - virtual bool validate(bufferlist& bufferlist, uint64_t offset, - uint64_t length); - - // Used for testing debug outputs from data generation - virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length); - - protected: - const ObjectModel& m_model; - - DataGenerator(const ObjectModel& model) : m_model(model) {} - }; - - class SeededRandomGenerator : public DataGenerator - { - public: - SeededRandomGenerator(const ObjectModel& model) - : DataGenerator(model) {} - - virtual bufferptr generate_block(uint64_t offset); - virtual bufferlist generate_data(uint64_t length, uint64_t offset); - virtual bufferptr generate_wrong_block(uint64_t offset); - virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length) override; - }; - - class HeaderedSeededRandomGenerator : public SeededRandomGenerator - { - public: - HeaderedSeededRandomGenerator(const ObjectModel& model, - std::optional<uint64_t> unique_run_id = std::nullopt); - - bufferptr generate_block(uint64_t offset) override; - bufferptr generate_wrong_block(uint64_t offset) override; - bool validate(bufferlist& bufferlist, uint64_t offset, - uint64_t length) override; - - private: - using UniqueIdBytes = uint64_t; - using SeedBytes = int; - using TimeBytes = uint64_t; - - enum class ErrorType { - RUN_ID_MISMATCH, - SEED_MISMATCH, - DATA_MISMATCH, - DATA_NOT_FOUND, - UNKNOWN - }; - - constexpr uint8_t headerStart() const - { return 0; }; - constexpr uint8_t uniqueIdStart() const - { return headerStart(); }; - constexpr uint8_t uniqueIdLength() const - { return sizeof(UniqueIdBytes); }; - constexpr uint8_t seedStart() const - { return uniqueIdStart() + uniqueIdLength(); }; - constexpr uint8_t seedLength() const - { return sizeof(SeedBytes); }; - constexpr uint8_t timeStart() const - { return seedStart() + seedLength(); }; - constexpr uint8_t timeLength() const - { return sizeof(TimeBytes); }; - constexpr uint8_t timeEnd() const - { return timeStart() + timeLength(); }; - constexpr uint8_t headerLength() const - { return uniqueIdLength() + seedLength() + timeLength(); }; - constexpr uint8_t bodyStart() const - { return headerStart() + headerLength(); }; - - const UniqueIdBytes readUniqueRunId(uint64_t block_offset, - const bufferlist& bufferlist); - const SeedBytes readSeed(uint64_t block_offset, - const bufferlist& bufferlist); - const TimeBytes readDateTime(uint64_t block_offset, +namespace io_exerciser { +namespace data_generation { +enum class GenerationType { + SeededRandom, + HeaderedSeededRandom + // CompressedGenerator + // MixedGenerator +}; + +class DataGenerator { + public: + virtual ~DataGenerator() = default; + static std::unique_ptr<DataGenerator> create_generator( + GenerationType generatorType, const ObjectModel& model); + virtual bufferlist generate_data(uint64_t length, uint64_t offset) = 0; + virtual bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length); + + // Used for testing debug outputs from data generation + virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length); + + protected: + const ObjectModel& m_model; + + DataGenerator(const ObjectModel& model) : m_model(model) {} +}; + +class SeededRandomGenerator : public DataGenerator { + public: + SeededRandomGenerator(const ObjectModel& model) : DataGenerator(model) {} + + virtual bufferptr generate_block(uint64_t offset); + bufferlist generate_data(uint64_t length, uint64_t offset) override; + virtual bufferptr generate_wrong_block(uint64_t offset); + bufferlist generate_wrong_data(uint64_t offset, + uint64_t length) override; +}; + +class HeaderedSeededRandomGenerator : public SeededRandomGenerator { + public: + HeaderedSeededRandomGenerator( + const ObjectModel& model, + std::optional<uint64_t> unique_run_id = std::nullopt); + + bufferptr generate_block(uint64_t offset) override; + bufferptr generate_wrong_block(uint64_t offset) override; + bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length) override; + + private: + using UniqueIdBytes = uint64_t; + using SeedBytes = int; + using TimeBytes = uint64_t; + + enum class ErrorType { + RUN_ID_MISMATCH, + SEED_MISMATCH, + DATA_MISMATCH, + DATA_NOT_FOUND, + UNKNOWN + }; + + constexpr uint8_t headerStart() const { return 0; }; + constexpr uint8_t uniqueIdStart() const { return headerStart(); }; + constexpr uint8_t uniqueIdLength() const { return sizeof(UniqueIdBytes); }; + constexpr uint8_t seedStart() const { + return uniqueIdStart() + uniqueIdLength(); + }; + constexpr uint8_t seedLength() const { return sizeof(SeedBytes); }; + constexpr uint8_t timeStart() const { return seedStart() + seedLength(); }; + constexpr uint8_t timeLength() const { return sizeof(TimeBytes); }; + constexpr uint8_t timeEnd() const { return timeStart() + timeLength(); }; + constexpr uint8_t headerLength() const { + return uniqueIdLength() + seedLength() + timeLength(); + }; + constexpr uint8_t bodyStart() const { + return headerStart() + headerLength(); + }; + + const UniqueIdBytes readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist); + const SeedBytes readSeed(uint64_t block_offset, const bufferlist& bufferlist); + const TimeBytes readDateTime(uint64_t block_offset, + const bufferlist& bufferlist); + + const UniqueIdBytes unique_run_id; + + uint64_t generate_unique_run_id(); + + bool validate_block(uint64_t block_offset, const char* buffer_start); + + const ErrorType getErrorTypeForBlock(uint64_t read_offset, + uint64_t block_offset, const bufferlist& bufferlist); - const UniqueIdBytes unique_run_id; - - uint64_t generate_unique_run_id(); - - bool validate_block(uint64_t block_offset, const char* buffer_start); - - const ErrorType getErrorTypeForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist); - - void printDebugInformationForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist); - void printDebugInformationForRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - ErrorType rangeError, - const bufferlist& bufferlist); - - void printDebugInformationForRunIdMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationForSeedMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationDataBodyMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationDataNotFoundRange(uint64_t ßread_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationCorruptRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - - void printDebugInformationForOffsets(uint64_t read_offset, - std::vector<uint64_t> offsets, - const bufferlist& bufferlist); - }; - } - } -} + void printDebugInformationForBlock(uint64_t read_offset, + uint64_t block_offset, + const bufferlist& bufferlist); + void printDebugInformationForRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + ErrorType rangeError, + const bufferlist& bufferlist); + + void printDebugInformationForRunIdMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationForSeedMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationDataBodyMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationDataNotFoundRange(uint64_t ßread_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationCorruptRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + + void printDebugInformationForOffsets(uint64_t read_offset, + std::vector<uint64_t> offsets, + const bufferlist& bufferlist); +}; +} // namespace data_generation +} // namespace io_exerciser +} // namespace ceph diff --git a/src/common/io_exerciser/EcIoSequence.cc b/src/common/io_exerciser/EcIoSequence.cc new file mode 100644 index 00000000000..611920c96e0 --- /dev/null +++ b/src/common/io_exerciser/EcIoSequence.cc @@ -0,0 +1,267 @@ +#include "EcIoSequence.h" + +#include <memory> + +using IoOp = ceph::io_exerciser::IoOp; +using Sequence = ceph::io_exerciser::Sequence; +using IoSequence = ceph::io_exerciser::IoSequence; +using EcIoSequence = ceph::io_exerciser::EcIoSequence; +using ReadInjectSequence = ceph::io_exerciser::ReadInjectSequence; + +bool EcIoSequence::is_supported(Sequence sequence) const { return true; } + +std::unique_ptr<IoSequence> EcIoSequence::generate_sequence( + Sequence sequence, std::pair<int, int> obj_size_range, int k, int m, + int seed) { + switch (sequence) { + case Sequence::SEQUENCE_SEQ0: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ1: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ2: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ3: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ4: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ5: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ6: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ7: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ8: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ9: + return std::make_unique<ReadInjectSequence>(obj_size_range, seed, + sequence, k, m); + case Sequence::SEQUENCE_SEQ10: + return std::make_unique<Seq10>(obj_size_range, seed, k, m); + default: + ceph_abort_msg("Unrecognised sequence"); + } +} + +EcIoSequence::EcIoSequence(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + setup_inject(false), + clear_inject(false), + shard_to_inject(std::nullopt) {} + +void EcIoSequence::select_random_data_shard_to_inject_read_error(int k, int m) { + shard_to_inject = rng(k - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_data_shard_to_inject_write_error(int k, + int m) { + // Write errors do not support injecting to the primary OSD + shard_to_inject = rng(1, k - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_shard_to_inject_read_error(int k, int m) { + shard_to_inject = rng(k + m - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_shard_to_inject_write_error(int k, int m) { + // Write errors do not support injecting to the primary OSD + shard_to_inject = rng(1, k + m - 1); + setup_inject = true; +} + +void EcIoSequence::generate_random_read_inject_type() { + inject_op_type = static_cast<InjectOpType>( + rng(static_cast<int>(InjectOpType::ReadEIO), + static_cast<int>(InjectOpType::ReadMissingShard))); +} + +void EcIoSequence::generate_random_write_inject_type() { + inject_op_type = static_cast<InjectOpType>( + rng(static_cast<int>(InjectOpType::WriteFailAndRollback), + static_cast<int>(InjectOpType::WriteOSDAbort))); +} + +ceph::io_exerciser::ReadInjectSequence::ReadInjectSequence( + std::pair<int, int> obj_size_range, int seed, Sequence s, int k, int m) + : EcIoSequence(obj_size_range, seed) { + child_sequence = IoSequence::generate_sequence(s, obj_size_range, seed); + select_random_data_shard_to_inject_read_error(k, m); + generate_random_read_inject_type(); +} + +Sequence ceph::io_exerciser::ReadInjectSequence::get_id() const { + return child_sequence->get_id(); +} + +std::string ceph::io_exerciser::ReadInjectSequence::get_name() const { + return child_sequence->get_name() + + " running with read errors injected on shard " + + std::to_string(*shard_to_inject); +} + +std::unique_ptr<IoOp> ReadInjectSequence::next() { + step++; + + if (nextOp) { + std::unique_ptr<IoOp> retOp = nullptr; + nextOp.swap(retOp); + return retOp; + } + + std::unique_ptr<IoOp> childOp = child_sequence->next(); + + switch (childOp->getOpType()) { + case OpType::Remove: + nextOp.swap(childOp); + switch (inject_op_type) { + case InjectOpType::ReadEIO: + return ClearReadErrorInjectOp::generate(*shard_to_inject, 0); + case InjectOpType::ReadMissingShard: + return ClearReadErrorInjectOp::generate(*shard_to_inject, 1); + case InjectOpType::WriteFailAndRollback: + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 0); + case InjectOpType::WriteOSDAbort: + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 3); + case InjectOpType::None: + [[fallthrough]]; + default: + ceph_abort_msg("Unsupported operation"); + } + break; + case OpType::Create: + switch (inject_op_type) { + case InjectOpType::ReadEIO: + nextOp = InjectReadErrorOp::generate( + *shard_to_inject, 0, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::ReadMissingShard: + nextOp = InjectReadErrorOp::generate( + *shard_to_inject, 1, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::WriteFailAndRollback: + nextOp = InjectWriteErrorOp::generate( + *shard_to_inject, 0, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::WriteOSDAbort: + nextOp = InjectWriteErrorOp::generate( + *shard_to_inject, 3, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::None: + [[fallthrough]]; + default: + ceph_abort_msg("Unsupported operation"); + } + break; + default: + // Do nothing in default case + break; + } + + return childOp; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> +ceph::io_exerciser::ReadInjectSequence::_next() { + ceph_abort_msg( + "Should not reach this point, " + "this sequence should only consume complete sequences"); + + return DoneOp::generate(); +} + +ceph::io_exerciser::Seq10::Seq10(std::pair<int, int> obj_size_range, int seed, + int k, int m) + : EcIoSequence(obj_size_range, seed), + offset(0), + length(1), + inject_error_done(false), + failed_write_done(false), + read_done(false), + successful_write_done(false), + test_all_lengths(false), // Only test length(1) due to time constraints + test_all_sizes( + false) // Only test obj_size(rand()) due to time constraints +{ + select_random_shard_to_inject_write_error(k, m); + // We will inject specifically as part of our sequence in this sequence + setup_inject = false; + if (!test_all_sizes) { + select_random_object_size(); + } +} + +Sequence ceph::io_exerciser::Seq10::get_id() const { + return Sequence::SEQUENCE_SEQ10; +} + +std::string ceph::io_exerciser::Seq10::get_name() const { + return "Sequential writes of length " + std::to_string(length) + + " with queue depth 1" + " first injecting a failed write and read it to ensure it rolls back, " + "then" + " successfully writing the data and reading the write the ensure it " + "is applied"; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq10::_next() { + if (!inject_error_done) { + inject_error_done = true; + return InjectWriteErrorOp::generate(*shard_to_inject, 0, 0, + std::numeric_limits<uint64_t>::max()); + } else if (!failed_write_done) { + failed_write_done = true; + read_done = false; + barrier = true; + return SingleFailedWriteOp::generate(offset, length); + } else if (failed_write_done && !read_done) { + read_done = true; + barrier = true; + return SingleReadOp::generate(offset, length); + } else if (!clear_inject_done) { + clear_inject_done = true; + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 0); + } else if (!successful_write_done) { + successful_write_done = true; + read_done = false; + barrier = true; + return SingleWriteOp::generate(offset, length); + } else if (successful_write_done && !read_done) { + read_done = true; + return SingleReadOp::generate(offset, length); + } else if (successful_write_done && read_done) { + offset++; + inject_error_done = false; + failed_write_done = false; + read_done = false; + clear_inject_done = false; + successful_write_done = false; + + if (offset + length >= obj_size) { + if (!test_all_lengths) { + remove = true; + done = true; + return BarrierOp::generate(); + } + + offset = 0; + length++; + if (length > obj_size) { + if (!test_all_sizes) { + remove = true; + done = true; + return BarrierOp::generate(); + } + + length = 1; + return increment_object_size(); + } + } + + return BarrierOp::generate(); + } else { + ceph_abort_msg("Sequence in undefined state. Aborting"); + return DoneOp::generate(); + } +}
\ No newline at end of file diff --git a/src/common/io_exerciser/EcIoSequence.h b/src/common/io_exerciser/EcIoSequence.h new file mode 100644 index 00000000000..37283b3906b --- /dev/null +++ b/src/common/io_exerciser/EcIoSequence.h @@ -0,0 +1,65 @@ +#include "IoSequence.h" + +namespace ceph { +namespace io_exerciser { +class EcIoSequence : public IoSequence { + public: + virtual bool is_supported(Sequence sequence) const override; + static std::unique_ptr<IoSequence> generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int k, int m, int seed); + + protected: + bool setup_inject; + bool clear_inject; + std::optional<uint64_t> shard_to_inject; + InjectOpType inject_op_type; + + EcIoSequence(std::pair<int, int> obj_size_range, int seed); + + // Writes cannot be sent to injected on shard zero, so selections seperated + // out + void select_random_data_shard_to_inject_read_error(int k, int m); + void select_random_data_shard_to_inject_write_error(int k, int m); + void select_random_shard_to_inject_read_error(int k, int m); + void select_random_shard_to_inject_write_error(int k, int m); + void generate_random_read_inject_type(); + void generate_random_write_inject_type(); +}; + +class ReadInjectSequence : public EcIoSequence { + public: + ReadInjectSequence(std::pair<int, int> obj_size_range, int seed, Sequence s, + int k, int m); + + Sequence get_id() const override; + std::string get_name() const override; + virtual std::unique_ptr<IoOp> next() override; + std::unique_ptr<IoOp> _next() override; + + private: + std::unique_ptr<IoSequence> child_sequence; + std::unique_ptr<IoOp> nextOp; +}; + +class Seq10 : public EcIoSequence { + public: + Seq10(std::pair<int, int> obj_size_range, int seed, int k, int m); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + + bool inject_error_done; + bool failed_write_done; + bool read_done; + bool clear_inject_done; + bool successful_write_done; + bool test_all_lengths; + bool test_all_sizes; +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/IoOp.cc b/src/common/io_exerciser/IoOp.cc index cd855ba6fff..493d1f435b4 100644 --- a/src/common/io_exerciser/IoOp.cc +++ b/src/common/io_exerciser/IoOp.cc @@ -1,188 +1,316 @@ #include "IoOp.h" -using IoOp = ceph::io_exerciser::IoOp; +#include "fmt/format.h" +#include "include/ceph_assert.h" -IoOp::IoOp( OpType op, - uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) : - op(op), - offset1(offset1), length1(length1), - offset2(offset2), length2(length2), - offset3(offset3), length3(length3) -{ +using IoOp = ceph::io_exerciser::IoOp; +using OpType = ceph::io_exerciser::OpType; -} +using DoneOp = ceph::io_exerciser::DoneOp; +using BarrierOp = ceph::io_exerciser::BarrierOp; +using CreateOp = ceph::io_exerciser::CreateOp; +using RemoveOp = ceph::io_exerciser::RemoveOp; +using SingleReadOp = ceph::io_exerciser::SingleReadOp; +using DoubleReadOp = ceph::io_exerciser::DoubleReadOp; +using TripleReadOp = ceph::io_exerciser::TripleReadOp; +using SingleWriteOp = ceph::io_exerciser::SingleWriteOp; +using DoubleWriteOp = ceph::io_exerciser::DoubleWriteOp; +using TripleWriteOp = ceph::io_exerciser::TripleWriteOp; +using SingleFailedWriteOp = ceph::io_exerciser::SingleFailedWriteOp; +using DoubleFailedWriteOp = ceph::io_exerciser::DoubleFailedWriteOp; +using TripleFailedWriteOp = ceph::io_exerciser::TripleFailedWriteOp; -std::string IoOp::value_to_string(uint64_t v) const -{ +namespace { +std::string value_to_string(uint64_t v) { if (v < 1024 || (v % 1024) != 0) { return std::to_string(v); - }else if (v < 1024*1024 || (v % (1024 * 1024)) != 0 ) { + } else if (v < 1024 * 1024 || (v % (1024 * 1024)) != 0) { return std::to_string(v / 1024) + "K"; - }else{ + } else { return std::to_string(v / 1024 / 1024) + "M"; } } +} // namespace -std::unique_ptr<IoOp> IoOp - ::generate_done() { +IoOp::IoOp() {} - return std::make_unique<IoOp>(OpType::Done); -} +template <OpType opType> +ceph::io_exerciser::TestOp<opType>::TestOp() : IoOp() {} + +DoneOp::DoneOp() : TestOp<OpType::Done>() {} -std::unique_ptr<IoOp> IoOp - ::generate_barrier() { +std::string DoneOp::to_string(uint64_t block_size) const { return "Done"; } - return std::make_unique<IoOp>(OpType::BARRIER); +std::unique_ptr<DoneOp> DoneOp::generate() { + return std::make_unique<DoneOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_create(uint64_t size) { +BarrierOp::BarrierOp() : TestOp<OpType::Barrier>() {} - return std::make_unique<IoOp>(OpType::CREATE,0,size); +std::unique_ptr<BarrierOp> BarrierOp::generate() { + return std::make_unique<BarrierOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_remove() { - - return std::make_unique<IoOp>(OpType::REMOVE); +std::string BarrierOp::to_string(uint64_t block_size) const { + return "Barrier"; } -std::unique_ptr<IoOp> IoOp - ::generate_read(uint64_t offset, uint64_t length) { +CreateOp::CreateOp(uint64_t size) : TestOp<OpType::Create>(), size(size) {} - return std::make_unique<IoOp>(OpType::READ, offset, length); +std::unique_ptr<CreateOp> CreateOp::generate(uint64_t size) { + return std::make_unique<CreateOp>(size); } -std::unique_ptr<IoOp> IoOp - ::generate_read2(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2) { +std::string CreateOp::to_string(uint64_t block_size) const { + return "Create (size=" + value_to_string(size * block_size) + ")"; +} - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } +RemoveOp::RemoveOp() : TestOp<OpType::Remove>() {} - return std::make_unique<IoOp>(OpType::READ2, - offset1, length1, - offset2, length2); +std::unique_ptr<RemoveOp> RemoveOp::generate() { + return std::make_unique<RemoveOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_read3(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) { +std::string RemoveOp::to_string(uint64_t block_size) const { return "Remove"; } - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); +template <OpType opType, int numIOs> +ceph::io_exerciser::ReadWriteOp<opType, numIOs>::ReadWriteOp( + std::array<uint64_t, numIOs>&& offset, + std::array<uint64_t, numIOs>&& length) + : TestOp<opType>(), offset(offset), length(length) { + auto compare = [](uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2) { + if (offset1 < offset2) { + ceph_assert(offset1 + length1 <= offset2); + } else { + ceph_assert(offset2 + length2 <= offset1); + } + }; + + if (numIOs > 1) { + for (int i = 0; i < numIOs - 1; i++) { + for (int j = i + 1; j < numIOs; j++) { + compare(offset[i], length[i], offset[j], length[j]); + } + } } - if (offset1 < offset3) { - ceph_assert( offset1 + length1 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset1 ); +} + +template <OpType opType, int numIOs> +std::string ceph::io_exerciser::ReadWriteOp<opType, numIOs>::to_string( + uint64_t block_size) const { + std::string offset_length_desc; + if (numIOs > 0) { + offset_length_desc += fmt::format( + "offset1={}", value_to_string(this->offset[0] * block_size)); + offset_length_desc += fmt::format( + ",length1={}", value_to_string(this->length[0] * block_size)); + for (int i = 1; i < numIOs; i++) { + offset_length_desc += fmt::format( + ",offset{}={}", i + 1, value_to_string(this->offset[i] * block_size)); + offset_length_desc += fmt::format( + ",length{}={}", i + 1, value_to_string(this->length[i] * block_size)); + } } - if (offset2 < offset3) { - ceph_assert( offset2 + length2 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset2 ); + switch (opType) { + case OpType::Read: + [[fallthrough]]; + case OpType::Read2: + [[fallthrough]]; + case OpType::Read3: + return fmt::format("Read{} ({})", numIOs, offset_length_desc); + case OpType::Write: + [[fallthrough]]; + case OpType::Write2: + [[fallthrough]]; + case OpType::Write3: + return fmt::format("Write{} ({})", numIOs, offset_length_desc); + case OpType::FailedWrite: + [[fallthrough]]; + case OpType::FailedWrite2: + [[fallthrough]]; + case OpType::FailedWrite3: + return fmt::format("FailedWrite{} ({})", numIOs, offset_length_desc); + default: + ceph_abort_msg( + fmt::format("Unsupported op type by ReadWriteOp ({})", opType)); } - return std::make_unique<IoOp>(OpType::READ3, - offset1, length1, - offset2, length2, - offset3, length3); } -std::unique_ptr<IoOp> IoOp::generate_write(uint64_t offset, uint64_t length) { - return std::make_unique<IoOp>(OpType::WRITE, offset, length); +SingleReadOp::SingleReadOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::Read, 1>({offset}, {length}) {} + +std::unique_ptr<SingleReadOp> SingleReadOp::generate(uint64_t offset, + uint64_t length) { + return std::make_unique<SingleReadOp>(offset, length); } -std::unique_ptr<IoOp> IoOp::generate_write2(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2) { - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } - return std::make_unique<IoOp>(OpType::WRITE2, - offset1, length1, - offset2, length2); +DoubleReadOp::DoubleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2) + : ReadWriteOp<OpType::Read2, 2>({offset1, offset2}, {length1, length2}) {} + +std::unique_ptr<DoubleReadOp> DoubleReadOp::generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2) { + return std::make_unique<DoubleReadOp>(offset1, length1, offset2, length2); } -std::unique_ptr<IoOp> IoOp::generate_write3(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) { - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } - if (offset1 < offset3) { - ceph_assert( offset1 + length1 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset1 ); - } - if (offset2 < offset3) { - ceph_assert( offset2 + length2 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset2 ); - } - return std::make_unique<IoOp>(OpType::WRITE3, - offset1, length1, - offset2, length2, - offset3, length3); -} - -bool IoOp::done() { - return (op == OpType::Done); -} - -std::string IoOp::to_string(uint64_t block_size) const -{ - switch (op) { - case OpType::Done: - return "Done"; - case OpType::BARRIER: - return "Barrier"; - case OpType::CREATE: - return "Create (size=" + value_to_string(length1 * block_size) + ")"; - case OpType::REMOVE: - return "Remove"; - case OpType::READ: - return "Read (offset=" + value_to_string(offset1 * block_size) + - ",length=" + value_to_string(length1 * block_size) + ")"; - case OpType::READ2: - return "Read2 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + ")"; - case OpType::READ3: - return "Read3 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + - ",offset3=" + value_to_string(offset3 * block_size) + - ",length3=" + value_to_string(length3 * block_size) + ")"; - case OpType::WRITE: - return "Write (offset=" + value_to_string(offset1 * block_size) + - ",length=" + value_to_string(length1 * block_size) + ")"; - case OpType::WRITE2: - return "Write2 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + ")"; - case OpType::WRITE3: - return "Write3 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + - ",offset3=" + value_to_string(offset3 * block_size) + - ",length3=" + value_to_string(length3 * block_size) + ")"; - default: - break; - } - return "Unknown"; +TripleReadOp::TripleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::Read3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleReadOp> TripleReadOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleReadOp>(offset1, length1, offset2, length2, + offset3, length3); +} + +SingleWriteOp::SingleWriteOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::Write, 1>({offset}, {length}) {} + +std::unique_ptr<SingleWriteOp> SingleWriteOp::generate(uint64_t offset, + uint64_t length) { + return std::make_unique<SingleWriteOp>(offset, length); +} + +DoubleWriteOp::DoubleWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) + : ReadWriteOp<OpType::Write2, 2>({offset1, offset2}, {length1, length2}) {} + +std::unique_ptr<DoubleWriteOp> DoubleWriteOp::generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2) { + return std::make_unique<DoubleWriteOp>(offset1, length1, offset2, length2); +} + +TripleWriteOp::TripleWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::Write3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleWriteOp> TripleWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleWriteOp>(offset1, length1, offset2, length2, + offset3, length3); +} + +SingleFailedWriteOp::SingleFailedWriteOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::FailedWrite, 1>({offset}, {length}) {} + +std::unique_ptr<SingleFailedWriteOp> SingleFailedWriteOp::generate( + uint64_t offset, uint64_t length) { + return std::make_unique<SingleFailedWriteOp>(offset, length); +} + +DoubleFailedWriteOp::DoubleFailedWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) + : ReadWriteOp<OpType::FailedWrite2, 2>({offset1, offset2}, + {length1, length2}) {} + +std::unique_ptr<DoubleFailedWriteOp> DoubleFailedWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2) { + return std::make_unique<DoubleFailedWriteOp>(offset1, length1, offset2, + length2); +} + +TripleFailedWriteOp::TripleFailedWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::FailedWrite3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleFailedWriteOp> TripleFailedWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleFailedWriteOp>(offset1, length1, offset2, + length2, offset3, length3); +} + +template <ceph::io_exerciser::OpType opType> +ceph::io_exerciser::InjectErrorOp<opType>::InjectErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : TestOp<opType>(), + shard(shard), + type(type), + when(when), + duration(duration) {} + +template <ceph::io_exerciser::OpType opType> +std::string ceph::io_exerciser::InjectErrorOp<opType>::to_string( + uint64_t blocksize) const { + std::string_view inject_type = get_inject_type_string(); + return fmt::format( + "Inject {} error on shard {} of type {}" + " after {} successful inject(s) lasting {} inject(s)", + inject_type, shard, type.value_or(0), when.value_or(0), + duration.value_or(1)); +} + +ceph::io_exerciser::InjectReadErrorOp::InjectReadErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : InjectErrorOp<OpType::InjectReadError>(shard, type, when, duration) {} + +std::unique_ptr<ceph::io_exerciser::InjectReadErrorOp> +ceph::io_exerciser ::InjectReadErrorOp::generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) { + return std::make_unique<InjectReadErrorOp>(shard, type, when, duration); +} + +ceph::io_exerciser::InjectWriteErrorOp::InjectWriteErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : InjectErrorOp<OpType::InjectWriteError>(shard, type, when, duration) {} + +std::unique_ptr<ceph::io_exerciser::InjectWriteErrorOp> +ceph::io_exerciser ::InjectWriteErrorOp::generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) { + return std::make_unique<InjectWriteErrorOp>(shard, type, when, duration); +} + +template <ceph::io_exerciser::OpType opType> +ceph::io_exerciser::ClearErrorInjectOp<opType>::ClearErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : TestOp<opType>(), shard(shard), type(type) {} + +template <ceph::io_exerciser::OpType opType> +std::string ceph::io_exerciser::ClearErrorInjectOp<opType>::to_string( + uint64_t blocksize) const { + std::string_view inject_type = get_inject_type_string(); + return fmt::format("Clear {} injects on shard {} of type {}", inject_type, + shard, type.value_or(0)); +} + +ceph::io_exerciser::ClearReadErrorInjectOp::ClearReadErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : ClearErrorInjectOp<OpType::ClearReadErrorInject>(shard, type) {} + +std::unique_ptr<ceph::io_exerciser::ClearReadErrorInjectOp> +ceph::io_exerciser ::ClearReadErrorInjectOp::generate( + int shard, const std::optional<uint64_t>& type) { + return std::make_unique<ClearReadErrorInjectOp>(shard, type); +} + +ceph::io_exerciser::ClearWriteErrorInjectOp::ClearWriteErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : ClearErrorInjectOp<OpType::ClearWriteErrorInject>(shard, type) {} + +std::unique_ptr<ceph::io_exerciser::ClearWriteErrorInjectOp> +ceph::io_exerciser ::ClearWriteErrorInjectOp::generate( + int shard, const std::optional<uint64_t>& type) { + return std::make_unique<ClearWriteErrorInjectOp>(shard, type); }
\ No newline at end of file diff --git a/src/common/io_exerciser/IoOp.h b/src/common/io_exerciser/IoOp.h index 60c02a93d4e..1887eafcc1f 100644 --- a/src/common/io_exerciser/IoOp.h +++ b/src/common/io_exerciser/IoOp.h @@ -1,94 +1,248 @@ #pragma once -#include <string> +#include <array> #include <memory> -#include "include/ceph_assert.h" +#include <optional> +#include <string> + +#include "OpType.h" /* Overview * - * enum OpType - * Enumeration of different types of I/O operation - * * class IoOp * Stores details for an I/O operation. Generated by IoSequences * and applied by IoExerciser's */ namespace ceph { - namespace io_exerciser { - - enum class OpType { - Done, // End of I/O sequence - BARRIER, // Barrier - all prior I/Os must complete - CREATE, // Create object and pattern with data - REMOVE, // Remove object - READ, // Read - READ2, // 2 Reads in one op - READ3, // 3 Reads in one op - WRITE, // Write - WRITE2, // 2 Writes in one op - WRITE3 // 3 Writes in one op - }; - - class IoOp { - protected: - std::string value_to_string(uint64_t v) const; - - public: - OpType op; - uint64_t offset1; - uint64_t length1; - uint64_t offset2; - uint64_t length2; - uint64_t offset3; - uint64_t length3; - - IoOp( OpType op, - uint64_t offset1 = 0, uint64_t length1 = 0, - uint64_t offset2 = 0, uint64_t length2 = 0, - uint64_t offset3 = 0, uint64_t length3 = 0 ); - - static std::unique_ptr<IoOp> generate_done(); - - static std::unique_ptr<IoOp> generate_barrier(); - - static std::unique_ptr<IoOp> generate_create(uint64_t size); - - static std::unique_ptr<IoOp> generate_remove(); - - static std::unique_ptr<IoOp> generate_read(uint64_t offset, +namespace io_exerciser { + +class IoOp { + public: + IoOp(); + virtual ~IoOp() = default; + virtual std::string to_string(uint64_t block_size) const = 0; + virtual constexpr OpType getOpType() const = 0; +}; + +template <OpType opType> +class TestOp : public IoOp { + public: + TestOp(); + constexpr OpType getOpType() const override { return opType; } +}; + +class DoneOp : public TestOp<OpType::Done> { + public: + DoneOp(); + static std::unique_ptr<DoneOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +class BarrierOp : public TestOp<OpType::Barrier> { + public: + BarrierOp(); + static std::unique_ptr<BarrierOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +class CreateOp : public TestOp<OpType::Create> { + public: + CreateOp(uint64_t size); + static std::unique_ptr<CreateOp> generate(uint64_t size); + std::string to_string(uint64_t block_size) const override; + uint64_t size; +}; + +class RemoveOp : public TestOp<OpType::Remove> { + public: + RemoveOp(); + static std::unique_ptr<RemoveOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +template <OpType opType, int numIOs> +class ReadWriteOp : public TestOp<opType> { + public: + std::array<uint64_t, numIOs> offset; + std::array<uint64_t, numIOs> length; + + protected: + ReadWriteOp(std::array<uint64_t, numIOs>&& offset, + std::array<uint64_t, numIOs>&& length); + std::string to_string(uint64_t block_size) const override; +}; + +class SingleReadOp : public ReadWriteOp<OpType::Read, 1> { + public: + SingleReadOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleReadOp> generate(uint64_t offset, + uint64_t length); +}; + +class DoubleReadOp : public ReadWriteOp<OpType::Read2, 2> { + public: + DoubleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleReadOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleReadOp : public ReadWriteOp<OpType::Read3, 3> { + public: + TripleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleReadOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +class SingleWriteOp : public ReadWriteOp<OpType::Write, 1> { + public: + SingleWriteOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleWriteOp> generate(uint64_t offset, uint64_t length); +}; + +class DoubleWriteOp : public ReadWriteOp<OpType::Write2, 2> { + public: + DoubleWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleWriteOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleWriteOp : public ReadWriteOp<OpType::Write3, 3> { + public: + TripleWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleWriteOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +class SingleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite, 1> { + public: + SingleFailedWriteOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleFailedWriteOp> generate(uint64_t offset, + uint64_t length); +}; + +class DoubleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite2, 2> { + public: + DoubleFailedWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleFailedWriteOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite3, 3> { + public: + TripleFailedWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleFailedWriteOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +template <ceph::io_exerciser::OpType opType> +class InjectErrorOp : public TestOp<opType> { + public: + InjectErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + std::string to_string(uint64_t block_size) const override; + + int shard; + std::optional<uint64_t> type; + std::optional<uint64_t> when; + std::optional<uint64_t> duration; + + protected: + virtual inline constexpr std::string_view get_inject_type_string() const = 0; +}; + +class InjectReadErrorOp : public InjectErrorOp<OpType::InjectReadError> { + public: + InjectReadErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + static std::unique_ptr<InjectReadErrorOp> generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "read"; + } +}; + +class InjectWriteErrorOp : public InjectErrorOp<OpType::InjectWriteError> { + public: + InjectWriteErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + static std::unique_ptr<InjectWriteErrorOp> generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "write"; + } +}; + +template <ceph::io_exerciser::OpType opType> +class ClearErrorInjectOp : public TestOp<opType> { + public: + ClearErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + std::string to_string(uint64_t block_size) const override; + + int shard; + std::optional<uint64_t> type; + + protected: + virtual inline constexpr std::string_view get_inject_type_string() const = 0; +}; + +class ClearReadErrorInjectOp + : public ClearErrorInjectOp<OpType::ClearReadErrorInject> { + public: + ClearReadErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + static std::unique_ptr<ClearReadErrorInjectOp> generate( + int shard, const std::optional<uint64_t>& type); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "read"; + } +}; + +class ClearWriteErrorInjectOp + : public ClearErrorInjectOp<OpType::ClearWriteErrorInject> { + public: + ClearWriteErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + static std::unique_ptr<ClearWriteErrorInjectOp> generate( + int shard, const std::optional<uint64_t>& type); - static std::unique_ptr<IoOp> generate_read2(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2); - - static std::unique_ptr<IoOp> generate_read3(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2, - uint64_t offset3, - uint64_t length3); - - static std::unique_ptr<IoOp> generate_write(uint64_t offset, - uint64_t length); - - static std::unique_ptr<IoOp> generate_write2(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2); - - static std::unique_ptr<IoOp> generate_write3(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2, - uint64_t offset3, - uint64_t length3); - - bool done(); - - std::string to_string(uint64_t block_size) const; - }; + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "write"; } -}
\ No newline at end of file +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.cc b/src/common/io_exerciser/IoSequence.cc index 4a7ca0593d1..83f1cc595a5 100644 --- a/src/common/io_exerciser/IoSequence.cc +++ b/src/common/io_exerciser/IoSequence.cc @@ -1,12 +1,12 @@ #include "IoSequence.h" +using IoOp = ceph::io_exerciser::IoOp; using Sequence = ceph::io_exerciser::Sequence; using IoSequence = ceph::io_exerciser::IoSequence; -std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& seq) -{ - switch (seq) - { +std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, + const Sequence& seq) { + switch (seq) { case Sequence::SEQUENCE_SEQ0: os << "SEQUENCE_SEQ0"; break; @@ -37,6 +37,9 @@ std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& s case Sequence::SEQUENCE_SEQ9: os << "SEQUENCE_SEQ9"; break; + case Sequence::SEQUENCE_SEQ10: + os << "SEQUENCE_SEQ10"; + break; case Sequence::SEQUENCE_END: os << "SEQUENCE_END"; break; @@ -44,19 +47,12 @@ std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& s return os; } -IoSequence::IoSequence(std::pair<int,int> obj_size_range, - int seed) : - min_obj_size(obj_size_range.first), max_obj_size(obj_size_range.second), - create(true), barrier(false), done(false), remove(false), - obj_size(min_obj_size), step(-1), seed(seed) -{ - rng.seed(seed); +bool IoSequence::is_supported(Sequence sequence) const { + return sequence != Sequence::SEQUENCE_SEQ10; } -std::unique_ptr<IoSequence> IoSequence::generate_sequence(Sequence s, - std::pair<int,int> obj_size_range, - int seed) -{ +std::unique_ptr<IoSequence> IoSequence::generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int seed) { switch (s) { case Sequence::SEQUENCE_SEQ0: return std::make_unique<Seq0>(obj_size_range, seed); @@ -78,24 +74,39 @@ std::unique_ptr<IoSequence> IoSequence::generate_sequence(Sequence s, return std::make_unique<Seq8>(obj_size_range, seed); case Sequence::SEQUENCE_SEQ9: return std::make_unique<Seq9>(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ10: + ceph_abort_msg( + "Sequence 10 only supported for erasure coded pools " + "through the EcIoSequence interface"); + return nullptr; default: break; } return nullptr; } -int IoSequence::get_step() const -{ - return step; +IoSequence::IoSequence(std::pair<int, int> obj_size_range, int seed) + : min_obj_size(obj_size_range.first), + max_obj_size(obj_size_range.second), + create(true), + barrier(false), + done(false), + remove(false), + obj_size(min_obj_size), + step(-1), + seed(seed) { + rng.seed(seed); } -int IoSequence::get_seed() const -{ - return seed; +std::string ceph::io_exerciser::IoSequence::get_name_with_seqseed() const { + return get_name() + " (seqseed " + std::to_string(get_seed()) + ")"; } -void IoSequence::set_min_object_size(uint64_t size) -{ +int IoSequence::get_step() const { return step; } + +int IoSequence::get_seed() const { return seed; } + +void IoSequence::set_min_object_size(uint64_t size) { min_obj_size = size; if (obj_size < size) { obj_size = size; @@ -105,23 +116,20 @@ void IoSequence::set_min_object_size(uint64_t size) } } -void IoSequence::set_max_object_size(uint64_t size) -{ +void IoSequence::set_max_object_size(uint64_t size) { max_obj_size = size; if (obj_size > size) { done = true; } } -void IoSequence::select_random_object_size() -{ +void IoSequence::select_random_object_size() { if (max_obj_size != min_obj_size) { obj_size = min_obj_size + rng(max_obj_size - min_obj_size); } } -std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::increment_object_size() -{ +std::unique_ptr<IoOp> IoSequence::increment_object_size() { obj_size++; if (obj_size > max_obj_size) { done = true; @@ -129,106 +137,118 @@ std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::increment_object_size() create = true; barrier = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } -std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::next() -{ +Sequence IoSequence::getNextSupportedSequenceId() const { + Sequence sequence = get_id(); + ++sequence; + for (; sequence < Sequence::SEQUENCE_END; ++sequence) { + if (is_supported(sequence)) { + return sequence; + } + } + + return Sequence::SEQUENCE_END; +} + +std::unique_ptr<IoOp> IoSequence::next() { step++; if (remove) { remove = false; - return IoOp::generate_remove(); + return RemoveOp::generate(); } if (barrier) { barrier = false; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } if (done) { - return IoOp::generate_done(); + return DoneOp::generate(); } if (create) { create = false; barrier = true; - return IoOp::generate_create(obj_size); + return CreateOp::generate(obj_size); } return _next(); } - - -ceph::io_exerciser::Seq0::Seq0(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0) -{ +ceph::io_exerciser::Seq0::Seq0(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0) { select_random_object_size(); length = 1 + rng(obj_size - 1); } -std::string ceph::io_exerciser::Seq0::get_name() const -{ +Sequence ceph::io_exerciser::Seq0::get_id() const { + return Sequence::SEQUENCE_SEQ0; +} + +std::string ceph::io_exerciser::Seq0::get_name() const { return "Sequential reads of length " + std::to_string(length) + - " with queue depth 1 (seqseed " + std::to_string(get_seed()) + ")"; + " with queue depth 1"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq0::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq0::_next() { std::unique_ptr<IoOp> r; if (offset >= obj_size) { done = true; barrier = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } if (offset + length > obj_size) { - r = IoOp::generate_read(offset, obj_size - offset); + r = SingleReadOp::generate(offset, obj_size - offset); } else { - r = IoOp::generate_read(offset, length); + r = SingleReadOp::generate(offset, length); } offset += length; return r; } - - -ceph::io_exerciser::Seq1::Seq1(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed) -{ +ceph::io_exerciser::Seq1::Seq1(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed) { select_random_object_size(); count = 3 * obj_size; } -std::string ceph::io_exerciser::Seq1::get_name() const -{ - return "Random offset, random length read/write I/O with queue depth 1 (seqseed " - + std::to_string(get_seed()) + ")"; +Sequence ceph::io_exerciser::Seq1::get_id() const { + return Sequence::SEQUENCE_SEQ1; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq1::_next() -{ +std::string ceph::io_exerciser::Seq1::get_name() const { + return "Random offset, random length read/write I/O with queue depth 1"; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq1::_next() { barrier = true; if (count-- == 0) { done = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } uint64_t offset = rng(obj_size - 1); uint64_t length = 1 + rng(obj_size - 1 - offset); - return (rng(2) != 0) ? IoOp::generate_write(offset, length) : - IoOp::generate_read(offset, length); -} + if (rng(2) != 0) { + return SingleWriteOp::generate(offset, length); + } else { + return SingleReadOp::generate(offset, length); + } +} +ceph::io_exerciser::Seq2::Seq2(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0), length(0) {} -ceph::io_exerciser::Seq2::Seq2(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(0) {} +Sequence ceph::io_exerciser::Seq2::get_id() const { + return Sequence::SEQUENCE_SEQ2; +} -std::string ceph::io_exerciser::Seq2::get_name() const -{ +std::string ceph::io_exerciser::Seq2::get_name() const { return "Permutations of offset and length read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() { length++; if (length > obj_size - offset) { length = 1; @@ -239,24 +259,23 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() return increment_object_size(); } } - return IoOp::generate_read(offset, length); + return SingleReadOp::generate(offset, length); } - - -ceph::io_exerciser::Seq3::Seq3(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(0) -{ +ceph::io_exerciser::Seq3::Seq3(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(0) { set_min_object_size(2); } -std::string ceph::io_exerciser::Seq3::get_name() const -{ +Sequence ceph::io_exerciser::Seq3::get_id() const { + return Sequence::SEQUENCE_SEQ3; +} + +std::string ceph::io_exerciser::Seq3::get_name() const { return "Permutations of offset 2-region 1-block read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() { offset2++; if (offset2 >= obj_size - offset1) { offset2 = 1; @@ -267,24 +286,23 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() return increment_object_size(); } } - return IoOp::generate_read2(offset1, 1, offset1 + offset2, 1); + return DoubleReadOp::generate(offset1, 1, offset1 + offset2, 1); } - - -ceph::io_exerciser::Seq4::Seq4(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(1) -{ +ceph::io_exerciser::Seq4::Seq4(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(1) { set_min_object_size(3); } -std::string ceph::io_exerciser::Seq4::get_name() const -{ +Sequence ceph::io_exerciser::Seq4::get_id() const { + return Sequence::SEQUENCE_SEQ4; +} + +std::string ceph::io_exerciser::Seq4::get_name() const { return "Permutations of offset 3-region 1-block read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() { offset2++; if (offset2 >= obj_size - offset1) { offset2 = 2; @@ -295,33 +313,35 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() return increment_object_size(); } } - return IoOp::generate_read3(offset1, 1, - offset1 + offset2, 1, - (offset1 * 2 + offset2)/2, 1); + return TripleReadOp::generate(offset1, 1, (offset1 + offset2), 1, + (offset1 * 2 + offset2) / 2, 1); } +ceph::io_exerciser::Seq5::Seq5(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + offset(0), + length(1), + doneread(false), + donebarrier(false) {} +Sequence ceph::io_exerciser::Seq5::get_id() const { + return Sequence::SEQUENCE_SEQ5; +} -ceph::io_exerciser::Seq5::Seq5(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(1), - doneread(false), donebarrier(false) {} - -std::string ceph::io_exerciser::Seq5::get_name() const -{ +std::string ceph::io_exerciser::Seq5::get_name() const { return "Permutation of length sequential writes"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() { if (offset >= obj_size) { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } doneread = false; donebarrier = false; @@ -333,33 +353,36 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() } } uint64_t io_len = (offset + length > obj_size) ? (obj_size - offset) : length; - std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len); + std::unique_ptr<IoOp> r = SingleWriteOp::generate(offset, io_len); offset += io_len; return r; } +ceph::io_exerciser::Seq6::Seq6(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + offset(0), + length(1), + doneread(false), + donebarrier(false) {} +Sequence ceph::io_exerciser::Seq6::get_id() const { + return Sequence::SEQUENCE_SEQ6; +} -ceph::io_exerciser::Seq6::Seq6(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(1), - doneread(false), donebarrier(false) {} - -std::string ceph::io_exerciser::Seq6::get_name() const -{ +std::string ceph::io_exerciser::Seq6::get_name() const { return "Permutation of length sequential writes, different alignment"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() { if (offset >= obj_size) { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } doneread = false; donebarrier = false; @@ -374,74 +397,72 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() if (io_len == 0) { io_len = length; } - std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len); + std::unique_ptr<IoOp> r = SingleWriteOp::generate(offset, io_len); offset += io_len; return r; } - - -ceph::io_exerciser::Seq7::Seq7(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed) -{ +ceph::io_exerciser::Seq7::Seq7(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed) { set_min_object_size(2); offset = obj_size; } -std::string ceph::io_exerciser::Seq7::get_name() const -{ +Sequence ceph::io_exerciser::Seq7::get_id() const { + return Sequence::SEQUENCE_SEQ7; +} + +std::string ceph::io_exerciser::Seq7::get_name() const { return "Permutations of offset 2-region 1-block writes"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq7::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq7::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } if (offset == 0) { doneread = false; donebarrier = false; - offset = obj_size+1; + offset = obj_size + 1; return increment_object_size(); } offset--; - if (offset == obj_size/2) { + if (offset == obj_size / 2) { return _next(); } doneread = false; donebarrier = false; - return IoOp::generate_write2(offset, 1, obj_size/2, 1); + return DoubleReadOp::generate(offset, 1, obj_size / 2, 1); } - - -ceph::io_exerciser::Seq8::Seq8(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(1) -{ +ceph::io_exerciser::Seq8::Seq8(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(1) { set_min_object_size(3); } -std::string ceph::io_exerciser::Seq8::get_name() const -{ +Sequence ceph::io_exerciser::Seq8::get_id() const { + return Sequence::SEQUENCE_SEQ8; +} + +std::string ceph::io_exerciser::Seq8::get_name() const { return "Permutations of offset 3-region 1-block write I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } offset2++; if (offset2 >= obj_size - offset1) { @@ -455,34 +476,30 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() } doneread = false; donebarrier = false; - return IoOp::generate_write3(offset1, 1, - offset1 + offset2, 1, - (offset1 * 2 + offset2)/2, 1); + return TripleWriteOp::generate(offset1, 1, offset1 + offset2, 1, + (offset1 * 2 + offset2) / 2, 1); } +ceph::io_exerciser::Seq9::Seq9(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0), length(0) {} - -ceph::io_exerciser::Seq9::Seq9(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(0) -{ - +Sequence ceph::io_exerciser::Seq9::get_id() const { + return Sequence::SEQUENCE_SEQ9; } -std::string ceph::io_exerciser::Seq9::get_name() const -{ +std::string ceph::io_exerciser::Seq9::get_name() const { return "Permutations of offset and length write I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } length++; if (length > obj_size - offset) { @@ -496,5 +513,5 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() } doneread = false; donebarrier = false; - return IoOp::generate_write(offset, length); + return SingleWriteOp::generate(offset, length); }
\ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.h b/src/common/io_exerciser/IoSequence.h index 114ff76303f..b6c254cf096 100644 --- a/src/common/io_exerciser/IoSequence.h +++ b/src/common/io_exerciser/IoSequence.h @@ -3,7 +3,6 @@ #pragma once #include "IoOp.h" - #include "include/random.h" /* Overview @@ -29,195 +28,209 @@ */ namespace ceph { - namespace io_exerciser { - - enum class Sequence { - SEQUENCE_SEQ0, - SEQUENCE_SEQ1, - SEQUENCE_SEQ2, - SEQUENCE_SEQ3, - SEQUENCE_SEQ4, - SEQUENCE_SEQ5, - SEQUENCE_SEQ6, - SEQUENCE_SEQ7, - SEQUENCE_SEQ8, - SEQUENCE_SEQ9, - // - SEQUENCE_END, - SEQUENCE_BEGIN = SEQUENCE_SEQ0 - }; - - inline Sequence operator++( Sequence& s ) - { - return s = (Sequence)(((int)(s) + 1)); - } - - std::ostream& operator<<(std::ostream& os, const Sequence& seq); - - /* I/O Sequences */ - - class IoSequence { - public: - virtual ~IoSequence() = default; - - virtual std::string get_name() const = 0; - int get_step() const; - int get_seed() const; - - std::unique_ptr<IoOp> next(); - - static std::unique_ptr<IoSequence> - generate_sequence(Sequence s, std::pair<int,int> obj_size_range, int seed ); - - protected: - uint64_t min_obj_size; - uint64_t max_obj_size; - bool create; - bool barrier; - bool done; - bool remove; - uint64_t obj_size; - int step; - int seed; - ceph::util::random_number_generator<int> rng = - ceph::util::random_number_generator<int>(); - - IoSequence(std::pair<int,int> obj_size_range, int seed); - - virtual std::unique_ptr<IoOp> _next() = 0; - - void set_min_object_size(uint64_t size); - void set_max_object_size(uint64_t size); - void select_random_object_size(); - std::unique_ptr<IoOp> increment_object_size(); - - }; - - class Seq0: public IoSequence { - public: - Seq0(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - }; - - class Seq1: public IoSequence { - public: - Seq1(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next(); - - private: - int count; - }; - - class Seq2: public IoSequence { - public: - Seq2(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - }; - - class Seq3: public IoSequence { - public: - Seq3(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - private: - uint64_t offset1; - uint64_t offset2; - }; - - class Seq4: public IoSequence { - public: - Seq4(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset1; - uint64_t offset2; - }; - - class Seq5: public IoSequence { - public: - Seq5(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - bool doneread; - bool donebarrier; - }; - - class Seq6: public IoSequence { - public: - Seq6(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - bool doneread; - bool donebarrier; - }; - - class Seq7: public IoSequence { - public: - Seq7(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - bool doneread = true; - bool donebarrier = false; - }; - - class Seq8: public IoSequence { - public: - Seq8(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - private: - uint64_t offset1; - uint64_t offset2; - bool doneread = true; - bool donebarrier = false; - }; - - class Seq9: public IoSequence { - private: - uint64_t offset; - uint64_t length; - bool doneread = true; - bool donebarrier = false; - - public: - Seq9(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - - std::unique_ptr<IoOp> _next() override; - }; - } -}
\ No newline at end of file +namespace io_exerciser { + +enum class Sequence { + SEQUENCE_SEQ0, + SEQUENCE_SEQ1, + SEQUENCE_SEQ2, + SEQUENCE_SEQ3, + SEQUENCE_SEQ4, + SEQUENCE_SEQ5, + SEQUENCE_SEQ6, + SEQUENCE_SEQ7, + SEQUENCE_SEQ8, + SEQUENCE_SEQ9, + SEQUENCE_SEQ10, + + SEQUENCE_END, + SEQUENCE_BEGIN = SEQUENCE_SEQ0 +}; + +inline Sequence operator++(Sequence& s) { + return s = (Sequence)(((int)(s) + 1)); +} + +std::ostream& operator<<(std::ostream& os, const Sequence& seq); + +/* I/O Sequences */ + +class IoSequence { + public: + virtual ~IoSequence() = default; + + virtual Sequence get_id() const = 0; + virtual std::string get_name_with_seqseed() const; + virtual std::string get_name() const = 0; + int get_step() const; + int get_seed() const; + + virtual Sequence getNextSupportedSequenceId() const; + virtual std::unique_ptr<IoOp> next(); + + virtual bool is_supported(Sequence sequence) const; + static std::unique_ptr<IoSequence> generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int seed); + + protected: + uint64_t min_obj_size; + uint64_t max_obj_size; + bool create; + bool barrier; + bool done; + bool remove; + uint64_t obj_size; + int step; + int seed; + ceph::util::random_number_generator<int> rng = + ceph::util::random_number_generator<int>(); + + IoSequence(std::pair<int, int> obj_size_range, int seed); + + virtual std::unique_ptr<IoOp> _next() = 0; + + void set_min_object_size(uint64_t size); + void set_max_object_size(uint64_t size); + void select_random_object_size(); + std::unique_ptr<IoOp> increment_object_size(); +}; + +class Seq0 : public IoSequence { + public: + Seq0(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; +}; + +class Seq1 : public IoSequence { + public: + Seq1(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + int count; +}; + +class Seq2 : public IoSequence { + public: + Seq2(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; +}; + +class Seq3 : public IoSequence { + public: + Seq3(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; +}; + +class Seq4 : public IoSequence { + public: + Seq4(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; +}; + +class Seq5 : public IoSequence { + public: + Seq5(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; +}; + +class Seq6 : public IoSequence { + public: + Seq6(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; +}; + +class Seq7 : public IoSequence { + public: + Seq7(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + bool doneread = true; + bool donebarrier = false; +}; + +class Seq8 : public IoSequence { + public: + Seq8(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; + bool doneread = true; + bool donebarrier = false; +}; + +class Seq9 : public IoSequence { + private: + uint64_t offset; + uint64_t length; + bool doneread = true; + bool donebarrier = false; + + public: + Seq9(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/Model.cc b/src/common/io_exerciser/Model.cc index 50812ecbb15..6548e1eda7a 100644 --- a/src/common/io_exerciser/Model.cc +++ b/src/common/io_exerciser/Model.cc @@ -4,25 +4,11 @@ using Model = ceph::io_exerciser::Model; -Model::Model(const std::string& oid, uint64_t block_size) : -num_io(0), -oid(oid), -block_size(block_size) -{ +Model::Model(const std::string& oid, uint64_t block_size) + : num_io(0), oid(oid), block_size(block_size) {} -} +const uint64_t Model::get_block_size() const { return block_size; } -const uint64_t Model::get_block_size() const -{ - return block_size; -} +const std::string Model::get_oid() const { return oid; } -const std::string Model::get_oid() const -{ - return oid; -} - -int Model::get_num_io() const -{ - return num_io; -}
\ No newline at end of file +int Model::get_num_io() const { return num_io; }
\ No newline at end of file diff --git a/src/common/io_exerciser/Model.h b/src/common/io_exerciser/Model.h index 58d107409a6..9e421e79a78 100644 --- a/src/common/io_exerciser/Model.h +++ b/src/common/io_exerciser/Model.h @@ -1,15 +1,13 @@ #pragma once -#include "IoOp.h" - #include <boost/asio/io_context.hpp> -#include "librados/librados_asio.h" - -#include "include/interval_set.h" -#include "global/global_init.h" -#include "global/global_context.h" +#include "IoOp.h" #include "common/Thread.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/interval_set.h" +#include "librados/librados_asio.h" /* Overview * @@ -21,29 +19,27 @@ */ namespace ceph { - namespace io_exerciser { - - class Model - { - protected: - int num_io{0}; - std::string oid; - uint64_t block_size; - - public: - Model(const std::string& oid, uint64_t block_size); - virtual ~Model() = default; - - virtual bool readyForIoOp(IoOp& op) = 0; - virtual void applyIoOp(IoOp& op) = 0; - - const std::string get_oid() const; - const uint64_t get_block_size() const; - int get_num_io() const; - }; - - /* Simple RADOS I/O generator */ - - - } -}
\ No newline at end of file +namespace io_exerciser { + +class Model { + protected: + int num_io{0}; + std::string oid; + uint64_t block_size; + + public: + Model(const std::string& oid, uint64_t block_size); + virtual ~Model() = default; + + virtual bool readyForIoOp(IoOp& op) = 0; + virtual void applyIoOp(IoOp& op) = 0; + + const std::string get_oid() const; + const uint64_t get_block_size() const; + int get_num_io() const; +}; + +/* Simple RADOS I/O generator */ + +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/ObjectModel.cc b/src/common/io_exerciser/ObjectModel.cc index 589f6434282..454d7254cf2 100644 --- a/src/common/io_exerciser/ObjectModel.cc +++ b/src/common/io_exerciser/ObjectModel.cc @@ -6,25 +6,20 @@ using ObjectModel = ceph::io_exerciser::ObjectModel; -ObjectModel::ObjectModel(const std::string& oid, uint64_t block_size, int seed) : - Model(oid, block_size), created(false) -{ +ObjectModel::ObjectModel(const std::string& oid, uint64_t block_size, int seed) + : Model(oid, block_size), created(false) { rng.seed(seed); } -int ObjectModel::get_seed(uint64_t offset) const -{ +int ObjectModel::get_seed(uint64_t offset) const { ceph_assert(offset < contents.size()); return contents[offset]; } -std::vector<int> ObjectModel::get_seed_offsets(int seed) const -{ +std::vector<int> ObjectModel::get_seed_offsets(int seed) const { std::vector<int> offsets; - for (size_t i = 0; i < contents.size(); i++) - { - if (contents[i] == seed) - { + for (size_t i = 0; i < contents.size(); i++) { + if (contents[i] == seed) { offsets.push_back(i); } } @@ -32,8 +27,7 @@ std::vector<int> ObjectModel::get_seed_offsets(int seed) const return offsets; } -std::string ObjectModel::to_string(int mask) const -{ +std::string ObjectModel::to_string(int mask) const { if (!created) { return "Object does not exist"; } @@ -48,107 +42,127 @@ std::string ObjectModel::to_string(int mask) const return result; } -bool ObjectModel::readyForIoOp(IoOp& op) -{ - return true; -} - -void ObjectModel::applyIoOp(IoOp& op) -{ - auto generate_random = [&rng = rng]() { - return rng(); - }; - - switch (op.op) { - case OpType::BARRIER: - reads.clear(); - writes.clear(); - break; - - case OpType::CREATE: - ceph_assert(!created); - ceph_assert(reads.empty()); - ceph_assert(writes.empty()); - created = true; - contents.resize(op.length1); - std::generate(std::execution::seq, contents.begin(), contents.end(), - generate_random); - break; - - case OpType::REMOVE: - ceph_assert(created); - ceph_assert(reads.empty()); - ceph_assert(writes.empty()); - created = false; - contents.resize(0); - break; - - case OpType::READ3: - ceph_assert(created); - ceph_assert(op.offset3 + op.length3 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset3, op.length3)); - reads.union_insert(op.offset3, op.length3); - [[fallthrough]]; - - case OpType::READ2: - ceph_assert(created); - ceph_assert(op.offset2 + op.length2 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset2, op.length2)); - reads.union_insert(op.offset2, op.length2); - [[fallthrough]]; - - case OpType::READ: - ceph_assert(created); - ceph_assert(op.offset1 + op.length1 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset1, op.length1)); - reads.union_insert(op.offset1, op.length1); - num_io++; - break; - - case OpType::WRITE3: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset3, op.length3)); - ceph_assert(!writes.intersects(op.offset3, op.length3)); - writes.union_insert(op.offset3, op.length3); - ceph_assert(op.offset3 + op.length3 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset3), - std::next(contents.begin(), op.offset3 + op.length3), - generate_random); - [[fallthrough]]; - - case OpType::WRITE2: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset2, op.length2)); - ceph_assert(!writes.intersects(op.offset2, op.length2)); - writes.union_insert(op.offset2, op.length2); - ceph_assert(op.offset2 + op.length2 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset2), - std::next(contents.begin(), op.offset2 + op.length2), - generate_random); - [[fallthrough]]; - - case OpType::WRITE: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset1, op.length1)); - ceph_assert(!writes.intersects(op.offset1, op.length1)); - writes.union_insert(op.offset1, op.length1); - ceph_assert(op.offset1 + op.length1 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset1), - std::next(contents.begin(), op.offset1 + op.length1), - generate_random); - num_io++; - break; - default: - break; +bool ObjectModel::readyForIoOp(IoOp& op) { return true; } + +void ObjectModel::applyIoOp(IoOp& op) { + auto generate_random = [&rng = rng]() { return rng(); }; + + auto verify_and_record_read_op = + [&contents = contents, &created = created, &num_io = num_io, + &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N>& readOp) { + ceph_assert(created); + for (int i = 0; i < N; i++) { + ceph_assert(readOp.offset[i] + readOp.length[i] <= contents.size()); + // Not allowed: read overlapping with parallel write + ceph_assert(!writes.intersects(readOp.offset[i], readOp.length[i])); + reads.union_insert(readOp.offset[i], readOp.length[i]); + } + num_io++; + }; + + auto verify_write_and_record_and_generate_seed = + [&generate_random, &contents = contents, &created = created, + &num_io = num_io, &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N> writeOp) { + ceph_assert(created); + for (int i = 0; i < N; i++) { + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(writeOp.offset[i], writeOp.length[i])); + ceph_assert(!writes.intersects(writeOp.offset[i], writeOp.length[i])); + writes.union_insert(writeOp.offset[i], writeOp.length[i]); + ceph_assert(writeOp.offset[i] + writeOp.length[i] <= contents.size()); + std::generate(std::execution::seq, + std::next(contents.begin(), writeOp.offset[i]), + std::next(contents.begin(), + writeOp.offset[i] + writeOp.length[i]), + generate_random); + } + num_io++; + }; + + auto verify_failed_write_and_record = + [&contents = contents, &created = created, &num_io = num_io, + &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N> writeOp) { + // Ensure write should still be valid, even though we are expecting OSD + // failure + ceph_assert(created); + for (int i = 0; i < N; i++) { + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(writeOp.offset[i], writeOp.length[i])); + ceph_assert(!writes.intersects(writeOp.offset[i], writeOp.length[i])); + writes.union_insert(writeOp.offset[i], writeOp.length[i]); + ceph_assert(writeOp.offset[i] + writeOp.length[i] <= contents.size()); + } + num_io++; + }; + + switch (op.getOpType()) { + case OpType::Barrier: + reads.clear(); + writes.clear(); + break; + + case OpType::Create: + ceph_assert(!created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = true; + contents.resize(static_cast<CreateOp&>(op).size); + std::generate(std::execution::seq, contents.begin(), contents.end(), + generate_random); + break; + + case OpType::Remove: + ceph_assert(created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = false; + contents.resize(0); + break; + + case OpType::Read: { + SingleReadOp& readOp = static_cast<SingleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + case OpType::Read2: { + DoubleReadOp& readOp = static_cast<DoubleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + case OpType::Read3: { + TripleReadOp& readOp = static_cast<TripleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + + case OpType::Write: { + ceph_assert(created); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::Write2: { + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::Write3: { + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::FailedWrite: { + ceph_assert(created); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + case OpType::FailedWrite2: { + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + case OpType::FailedWrite3: { + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + default: + break; } } diff --git a/src/common/io_exerciser/ObjectModel.h b/src/common/io_exerciser/ObjectModel.h index 93c70f41429..cad1307b84e 100644 --- a/src/common/io_exerciser/ObjectModel.h +++ b/src/common/io_exerciser/ObjectModel.h @@ -14,40 +14,41 @@ */ namespace ceph { - namespace io_exerciser { - /* Model of an object to track its data contents */ - - class ObjectModel : public Model { - private: - bool created; - std::vector<int> contents; - ceph::util::random_number_generator<int> rng = - ceph::util::random_number_generator<int>(); - - // Track read and write I/Os that can be submitted in - // parallel to detect violations: - // - // * Read may not overlap with a parallel write - // * Write may not overlap with a parallel read or write - // * Create / remove may not be in parallel with read or write - // - // Fix broken test cases by adding barrier ops to restrict - // I/O exercisers from issuing conflicting ops in parallel - interval_set<uint64_t> reads; - interval_set<uint64_t> writes; - public: - ObjectModel(const std::string& oid, uint64_t block_size, int seed); - - int get_seed(uint64_t offset) const; - std::vector<int> get_seed_offsets(int seed) const; - - std::string to_string(int mask = -1) const; - - bool readyForIoOp(IoOp& op); - void applyIoOp(IoOp& op); - - void encode(ceph::buffer::list& bl) const; - void decode(ceph::buffer::list::const_iterator& bl); - }; - } -}
\ No newline at end of file +namespace io_exerciser { +/* Model of an object to track its data contents */ + +class ObjectModel : public Model { + private: + bool created; + std::vector<int> contents; + ceph::util::random_number_generator<int> rng = + ceph::util::random_number_generator<int>(); + + // Track read and write I/Os that can be submitted in + // parallel to detect violations: + // + // * Read may not overlap with a parallel write + // * Write may not overlap with a parallel read or write + // * Create / remove may not be in parallel with read or write + // + // Fix broken test cases by adding barrier ops to restrict + // I/O exercisers from issuing conflicting ops in parallel + interval_set<uint64_t> reads; + interval_set<uint64_t> writes; + + public: + ObjectModel(const std::string& oid, uint64_t block_size, int seed); + + int get_seed(uint64_t offset) const; + std::vector<int> get_seed_offsets(int seed) const; + + std::string to_string(int mask = -1) const; + + bool readyForIoOp(IoOp& op); + void applyIoOp(IoOp& op); + + void encode(ceph::buffer::list& bl) const; + void decode(ceph::buffer::list::const_iterator& bl); +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/OpType.h b/src/common/io_exerciser/OpType.h new file mode 100644 index 00000000000..7cddb805e45 --- /dev/null +++ b/src/common/io_exerciser/OpType.h @@ -0,0 +1,91 @@ +#pragma once + +#include <fmt/format.h> +#include <include/ceph_assert.h> + +/* Overview + * + * enum OpType + * Enumeration of different types of I/O operation + * + */ + +namespace ceph { +namespace io_exerciser { +enum class OpType { + Done, // End of I/O sequence + Barrier, // Barrier - all prior I/Os must complete + Create, // Create object and pattern with data + Remove, // Remove object + Read, // Read + Read2, // Two reads in a single op + Read3, // Three reads in a single op + Write, // Write + Write2, // Two writes in a single op + Write3, // Three writes in a single op + FailedWrite, // A write which should fail + FailedWrite2, // Two writes in one op which should fail + FailedWrite3, // Three writes in one op which should fail + InjectReadError, // Op to tell OSD to inject read errors + InjectWriteError, // Op to tell OSD to inject write errors + ClearReadErrorInject, // Op to tell OSD to clear read error injects + ClearWriteErrorInject // Op to tell OSD to clear write error injects +}; + +enum class InjectOpType { + None, + ReadEIO, + ReadMissingShard, + WriteFailAndRollback, + WriteOSDAbort +}; +} // namespace io_exerciser +} // namespace ceph + +template <> +struct fmt::formatter<ceph::io_exerciser::OpType> { + constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); } + + auto format(ceph::io_exerciser::OpType opType, + fmt::format_context& ctx) const -> fmt::format_context::iterator { + switch (opType) { + case ceph::io_exerciser::OpType::Done: + return fmt::format_to(ctx.out(), "Done"); + case ceph::io_exerciser::OpType::Barrier: + return fmt::format_to(ctx.out(), "Barrier"); + case ceph::io_exerciser::OpType::Create: + return fmt::format_to(ctx.out(), "Create"); + case ceph::io_exerciser::OpType::Remove: + return fmt::format_to(ctx.out(), "Remove"); + case ceph::io_exerciser::OpType::Read: + return fmt::format_to(ctx.out(), "Read"); + case ceph::io_exerciser::OpType::Read2: + return fmt::format_to(ctx.out(), "Read2"); + case ceph::io_exerciser::OpType::Read3: + return fmt::format_to(ctx.out(), "Read3"); + case ceph::io_exerciser::OpType::Write: + return fmt::format_to(ctx.out(), "Write"); + case ceph::io_exerciser::OpType::Write2: + return fmt::format_to(ctx.out(), "Write2"); + case ceph::io_exerciser::OpType::Write3: + return fmt::format_to(ctx.out(), "Write3"); + case ceph::io_exerciser::OpType::FailedWrite: + return fmt::format_to(ctx.out(), "FailedWrite"); + case ceph::io_exerciser::OpType::FailedWrite2: + return fmt::format_to(ctx.out(), "FailedWrite2"); + case ceph::io_exerciser::OpType::FailedWrite3: + return fmt::format_to(ctx.out(), "FailedWrite3"); + case ceph::io_exerciser::OpType::InjectReadError: + return fmt::format_to(ctx.out(), "InjectReadError"); + case ceph::io_exerciser::OpType::InjectWriteError: + return fmt::format_to(ctx.out(), "InjectWriteError"); + case ceph::io_exerciser::OpType::ClearReadErrorInject: + return fmt::format_to(ctx.out(), "ClearReadErrorInject"); + case ceph::io_exerciser::OpType::ClearWriteErrorInject: + return fmt::format_to(ctx.out(), "ClearWriteErrorInject"); + default: + ceph_abort_msg("Unknown OpType"); + return fmt::format_to(ctx.out(), "Unknown OpType"); + } + } +};
\ No newline at end of file diff --git a/src/common/io_exerciser/RadosIo.cc b/src/common/io_exerciser/RadosIo.cc index 44b82260263..a78c074228b 100644 --- a/src/common/io_exerciser/RadosIo.cc +++ b/src/common/io_exerciser/RadosIo.cc @@ -1,300 +1,453 @@ #include "RadosIo.h" +#include <fmt/format.h> +#include <json_spirit/json_spirit.h> + +#include <ranges> + #include "DataGenerator.h" +#include "common/ceph_json.h" +#include "common/json/OSDStructures.h" using RadosIo = ceph::io_exerciser::RadosIo; -RadosIo::RadosIo(librados::Rados& rados, - boost::asio::io_context& asio, - const std::string& pool, - const std::string& oid, - uint64_t block_size, - int seed, - int threads, - ceph::mutex& lock, - ceph::condition_variable& cond) : - Model(oid, block_size), - rados(rados), - asio(asio), - om(std::make_unique<ObjectModel>(oid, block_size, seed)), - db(data_generation::DataGenerator::create_generator( - data_generation::GenerationType::HeaderedSeededRandom, *om)), - pool(pool), - threads(threads), - lock(lock), - cond(cond), - outstanding_io(0) -{ +namespace { +template <typename S> +int send_osd_command(int osd, S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, + Formatter* f) { + encode_json(name, s, f); + + std::ostringstream oss; + f->flush(oss); + int rc = rados.osd_command(osd, oss.str(), inbl, outbl, nullptr); + return rc; +} + +template <typename S> +int send_mon_command(S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, + Formatter* f) { + encode_json(name, s, f); + + std::ostringstream oss; + f->flush(oss); + int rc = rados.mon_command(oss.str(), inbl, outbl, nullptr); + return rc; +} +} // namespace + +RadosIo::RadosIo(librados::Rados& rados, boost::asio::io_context& asio, + const std::string& pool, const std::string& oid, + const std::optional<std::vector<int>>& cached_shard_order, + uint64_t block_size, int seed, int threads, ceph::mutex& lock, + ceph::condition_variable& cond) + : Model(oid, block_size), + rados(rados), + asio(asio), + om(std::make_unique<ObjectModel>(oid, block_size, seed)), + db(data_generation::DataGenerator::create_generator( + data_generation::GenerationType::HeaderedSeededRandom, *om)), + pool(pool), + cached_shard_order(cached_shard_order), + threads(threads), + lock(lock), + cond(cond), + outstanding_io(0) { int rc; rc = rados.ioctx_create(pool.c_str(), io); ceph_assert(rc == 0); allow_ec_overwrites(true); } -RadosIo::~RadosIo() -{ -} +RadosIo::~RadosIo() {} -void RadosIo::start_io() -{ +void RadosIo::start_io() { std::lock_guard l(lock); outstanding_io++; } -void RadosIo::finish_io() -{ +void RadosIo::finish_io() { std::lock_guard l(lock); ceph_assert(outstanding_io > 0); outstanding_io--; cond.notify_all(); } -void RadosIo::wait_for_io(int count) -{ +void RadosIo::wait_for_io(int count) { std::unique_lock l(lock); while (outstanding_io > count) { cond.wait(l); } } -void RadosIo::allow_ec_overwrites(bool allow) -{ +void RadosIo::allow_ec_overwrites(bool allow) { int rc; bufferlist inbl, outbl; - std::string cmdstr = - "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + "\", \ + std::string cmdstr = "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + + "\", \ \"var\": \"allow_ec_overwrites\", \"val\": \"" + - (allow ? "true" : "false") + "\"}"; + (allow ? "true" : "false") + "\"}"; rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr); ceph_assert(rc == 0); } -RadosIo::AsyncOpInfo::AsyncOpInfo(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3 ) : - offset1(offset1), length1(length1), - offset2(offset2), length2(length2), - offset3(offset3), length3(length3) -{ +template <int N> +RadosIo::AsyncOpInfo<N>::AsyncOpInfo(const std::array<uint64_t, N>& offset, + const std::array<uint64_t, N>& length) + : offset(offset), length(length) {} -} - -bool RadosIo::readyForIoOp(IoOp &op) -{ - ceph_assert(ceph_mutex_is_locked_by_me(lock)); //Must be called with lock held +bool RadosIo::readyForIoOp(IoOp& op) { + ceph_assert( + ceph_mutex_is_locked_by_me(lock)); // Must be called with lock held if (!om->readyForIoOp(op)) { return false; } - switch (op.op) { - case OpType::Done: - case OpType::BARRIER: - return outstanding_io == 0; - default: - return outstanding_io < threads; + + switch (op.getOpType()) { + case OpType::Done: + case OpType::Barrier: + return outstanding_io == 0; + default: + return outstanding_io < threads; } } -void RadosIo::applyIoOp(IoOp &op) -{ - std::shared_ptr<AsyncOpInfo> op_info; - +void RadosIo::applyIoOp(IoOp& op) { om->applyIoOp(op); // If there are thread concurrent I/Os in flight then wait for // at least one I/O to complete - wait_for_io(threads-1); - - switch (op.op) { - case OpType::Done: - [[ fallthrough ]]; - case OpType::BARRIER: - // Wait for all outstanding I/O to complete - wait_for_io(0); - break; - - case OpType::CREATE: - { + wait_for_io(threads - 1); + + switch (op.getOpType()) { + case OpType::Done: + [[fallthrough]]; + case OpType::Barrier: + // Wait for all outstanding I/O to complete + wait_for_io(0); + break; + + case OpType::Create: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(0, op.length1); - op_info->bl1 = db->generate_data(0, op.length1); - op_info->wop.write_full(op_info->bl1); - auto create_cb = [this] (boost::system::error_code ec, - version_t ver) { + uint64_t opSize = static_cast<CreateOp&>(op).size; + std::shared_ptr<AsyncOpInfo<1>> op_info = + std::make_shared<AsyncOpInfo<1>>(std::array<uint64_t, 1>{0}, + std::array<uint64_t, 1>{opSize}); + op_info->bufferlist[0] = db->generate_data(0, opSize); + op_info->wop.write_full(op_info->bufferlist[0]); + auto create_cb = [this](boost::system::error_code ec, version_t ver) { ceph_assert(ec == boost::system::errc::success); finish_io(); }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, create_cb); + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, + create_cb); + break; } - break; - case OpType::REMOVE: - { + case OpType::Remove: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(); + auto op_info = std::make_shared<AsyncOpInfo<0>>(); op_info->wop.remove(); - auto remove_cb = [this] (boost::system::error_code ec, - version_t ver) { + auto remove_cb = [this](boost::system::error_code ec, version_t ver) { ceph_assert(ec == boost::system::errc::success); finish_io(); }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, remove_cb); + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, + remove_cb); + break; } - break; + case OpType::Read: + [[fallthrough]]; + case OpType::Read2: + [[fallthrough]]; + case OpType::Read3: + [[fallthrough]]; + case OpType::Write: + [[fallthrough]]; + case OpType::Write2: + [[fallthrough]]; + case OpType::Write3: + [[fallthrough]]; + case OpType::FailedWrite: + [[fallthrough]]; + case OpType::FailedWrite2: + [[fallthrough]]; + case OpType::FailedWrite3: + applyReadWriteOp(op); + break; + case OpType::InjectReadError: + [[fallthrough]]; + case OpType::InjectWriteError: + [[fallthrough]]; + case OpType::ClearReadErrorInject: + [[fallthrough]]; + case OpType::ClearWriteErrorInject: + applyInjectOp(op); + break; + default: + ceph_abort_msg("Unrecognised Op"); + break; + } +} - case OpType::READ: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1); - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - auto read_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read_cb); - num_io++; +void RadosIo::applyReadWriteOp(IoOp& op) { + auto applyReadOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> readOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(readOp.offset, readOp.length); + + for (int i = 0; i < N; i++) { + op_info->rop.read(readOp.offset[i] * block_size, + readOp.length[i] * block_size, &op_info->bufferlist[i], + nullptr); } - break; + auto read_cb = [this, op_info](boost::system::error_code ec, version_t ver, + bufferlist bl) { + ceph_assert(ec == boost::system::errc::success); + for (int i = 0; i < N; i++) { + ceph_assert(db->validate(op_info->bufferlist[i], op_info->offset[i], + op_info->length[i])); + } + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->rop, 0, nullptr, read_cb); + num_io++; + }; - case OpType::READ2: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, - op.length1, - op.offset2, - op.length2); - - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - op_info->rop.read(op.offset2 * block_size, - op.length2 * block_size, - &op_info->bl2, nullptr); - auto read2_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - ceph_assert(db->validate(op_info->bl2, - op_info->offset2, - op_info->length2)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read2_cb); - num_io++; + auto applyWriteOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> writeOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(writeOp.offset, writeOp.length); + for (int i = 0; i < N; i++) { + op_info->bufferlist[i] = + db->generate_data(writeOp.offset[i], writeOp.length[i]); + op_info->wop.write(writeOp.offset[i] * block_size, + op_info->bufferlist[i]); } - break; + auto write_cb = [this](boost::system::error_code ec, version_t ver) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, write_cb); + num_io++; + }; - case OpType::READ3: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2, - op.offset3, op.length3); - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - op_info->rop.read(op.offset2 * block_size, - op.length2 * block_size, - &op_info->bl2, nullptr); - op_info->rop.read(op.offset3 * block_size, - op.length3 * block_size, - &op_info->bl3, nullptr); - auto read3_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - ceph_assert(db->validate(op_info->bl2, - op_info->offset2, - op_info->length2)); - ceph_assert(db->validate(op_info->bl3, - op_info->offset3, - op_info->length3)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read3_cb); - num_io++; + auto applyFailedWriteOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> writeOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(writeOp.offset, writeOp.length); + for (int i = 0; i < N; i++) { + op_info->bufferlist[i] = + db->generate_data(writeOp.offset[i], writeOp.length[i]); + op_info->wop.write(writeOp.offset[i] * block_size, + op_info->bufferlist[i]); } - break; + auto write_cb = [this, writeOp](boost::system::error_code ec, + version_t ver) { + ceph_assert(ec != boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, write_cb); + num_io++; + }; - case OpType::WRITE: - { + switch (op.getOpType()) { + case OpType::Read: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - auto write_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write_cb); - num_io++; + SingleReadOp& readOp = static_cast<SingleReadOp&>(op); + applyReadOp(readOp); + break; } - break; - - case OpType::WRITE2: - { + case OpType::Read2: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - op_info->bl2 = db->generate_data(op.offset2, op.length2); - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - op_info->wop.write(op.offset2 * block_size, op_info->bl2); - auto write2_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write2_cb); - num_io++; + DoubleReadOp& readOp = static_cast<DoubleReadOp&>(op); + applyReadOp(readOp); + break; + } + case OpType::Read3: { + start_io(); + TripleReadOp& readOp = static_cast<TripleReadOp&>(op); + applyReadOp(readOp); + break; + } + case OpType::Write: { + start_io(); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + applyWriteOp(writeOp); + break; + } + case OpType::Write2: { + start_io(); + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + applyWriteOp(writeOp); + break; + } + case OpType::Write3: { + start_io(); + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + applyWriteOp(writeOp); + break; } - break; - case OpType::WRITE3: - { + case OpType::FailedWrite: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2, - op.offset3, op.length3); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - op_info->bl2 = db->generate_data(op.offset2, op.length2); - op_info->bl3 = db->generate_data(op.offset3, op.length3); - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - op_info->wop.write(op.offset2 * block_size, op_info->bl2); - op_info->wop.write(op.offset3 * block_size, op_info->bl3); - auto write3_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write3_cb); - num_io++; + SingleFailedWriteOp& writeOp = static_cast<SingleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; + } + case OpType::FailedWrite2: { + start_io(); + DoubleFailedWriteOp& writeOp = static_cast<DoubleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; + } + case OpType::FailedWrite3: { + start_io(); + TripleFailedWriteOp& writeOp = static_cast<TripleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; } - break; - default: - break; + default: + ceph_abort_msg( + fmt::format("Unsupported Read/Write operation ({})", op.getOpType())); + break; } } + +void RadosIo::applyInjectOp(IoOp& op) { + bufferlist osdmap_inbl, inject_inbl, osdmap_outbl, inject_outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + int osd = -1; + std::vector<int> shard_order; + + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, get_oid(), ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", osdmap_inbl, + &osdmap_outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(osdmap_outbl.c_str(), osdmap_outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply; + reply.decode_json(&p); + + osd = reply.acting_primary; + shard_order = reply.acting; + + switch (op.getOpType()) { + case OpType::InjectReadError: { + InjectReadErrorOp& errorOp = static_cast<InjectReadErrorOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECErrorRequest<InjectOpType::ReadEIO> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 1) { + ceph::messaging::osd::InjectECErrorRequest< + InjectOpType::ReadMissingShard> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort_msg("Unsupported inject type"); + } + break; + } + case OpType::InjectWriteError: { + InjectWriteErrorOp& errorOp = static_cast<InjectWriteErrorOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECErrorRequest< + InjectOpType::WriteFailAndRollback> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 3) { + ceph::messaging::osd::InjectECErrorRequest<InjectOpType::WriteOSDAbort> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + + // This inject is sent directly to the shard we want to inject the error + // on + osd = shard_order[errorOp.shard]; + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + case OpType::ClearReadErrorInject: { + ClearReadErrorInjectOp& errorOp = + static_cast<ClearReadErrorInjectOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECClearErrorRequest<InjectOpType::ReadEIO> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 1) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::ReadMissingShard> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + case OpType::ClearWriteErrorInject: { + ClearReadErrorInjectOp& errorOp = + static_cast<ClearReadErrorInjectOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::WriteFailAndRollback> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 3) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::WriteOSDAbort> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + default: + ceph_abort_msg( + fmt::format("Unsupported inject operation ({})", op.getOpType())); + break; + } +}
\ No newline at end of file diff --git a/src/common/io_exerciser/RadosIo.h b/src/common/io_exerciser/RadosIo.h index 179c5bba3ae..a5c66ad4768 100644 --- a/src/common/io_exerciser/RadosIo.h +++ b/src/common/io_exerciser/RadosIo.h @@ -10,71 +10,65 @@ * in the object. Uses DataBuffer to create and validate * data buffers. When there are not barrier I/Os this may * issue multiple async I/Os in parallel. - * + * */ namespace ceph { - namespace io_exerciser { - namespace data_generation { - class DataGenerator; - } - - class RadosIo: public Model { - protected: - librados::Rados& rados; - boost::asio::io_context& asio; - std::unique_ptr<ObjectModel> om; - std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator> db; - std::string pool; - int threads; - ceph::mutex& lock; - ceph::condition_variable& cond; - librados::IoCtx io; - int outstanding_io; +namespace io_exerciser { +namespace data_generation { +class DataGenerator; +} + +class RadosIo : public Model { + protected: + librados::Rados& rados; + boost::asio::io_context& asio; + std::unique_ptr<ObjectModel> om; + std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator> db; + std::string pool; + std::optional<std::vector<int>> cached_shard_order; + int threads; + ceph::mutex& lock; + ceph::condition_variable& cond; + librados::IoCtx io; + int outstanding_io; + + void start_io(); + void finish_io(); + void wait_for_io(int count); + + public: + RadosIo(librados::Rados& rados, boost::asio::io_context& asio, + const std::string& pool, const std::string& oid, + const std::optional<std::vector<int>>& cached_shard_order, + uint64_t block_size, int seed, int threads, ceph::mutex& lock, + ceph::condition_variable& cond); - void start_io(); - void finish_io(); - void wait_for_io(int count); - - public: - RadosIo(librados::Rados& rados, - boost::asio::io_context& asio, - const std::string& pool, - const std::string& oid, - uint64_t block_size, - int seed, - int threads, - ceph::mutex& lock, - ceph::condition_variable& cond); + ~RadosIo(); - ~RadosIo(); + void allow_ec_overwrites(bool allow); - void allow_ec_overwrites(bool allow); + template <int N> + class AsyncOpInfo { + public: + librados::ObjectReadOperation rop; + librados::ObjectWriteOperation wop; + std::array<ceph::bufferlist, N> bufferlist; + std::array<uint64_t, N> offset; + std::array<uint64_t, N> length; - class AsyncOpInfo { - public: - librados::ObjectReadOperation rop; - librados::ObjectWriteOperation wop; - ceph::buffer::list bl1; - ceph::buffer::list bl2; - ceph::buffer::list bl3; - uint64_t offset1; - uint64_t length1; - uint64_t offset2; - uint64_t length2; - uint64_t offset3; - uint64_t length3; + AsyncOpInfo(const std::array<uint64_t, N>& offset = {}, + const std::array<uint64_t, N>& length = {}); + ~AsyncOpInfo() = default; + }; - AsyncOpInfo(uint64_t offset1 = 0, uint64_t length1 = 0, - uint64_t offset2 = 0, uint64_t length2 = 0, - uint64_t offset3 = 0, uint64_t length3 = 0 ); - ~AsyncOpInfo() = default; - }; + // Must be called with lock held + bool readyForIoOp(IoOp& op); + void applyIoOp(IoOp& op); - // Must be called with lock held - bool readyForIoOp(IoOp& op); - - void applyIoOp(IoOp& op); - }; - } -}
\ No newline at end of file + private: + void applyReadWriteOp(IoOp& op); + void applyInjectOp(IoOp& op); +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/BalancerStructures.cc b/src/common/json/BalancerStructures.cc new file mode 100644 index 00000000000..48dfb843761 --- /dev/null +++ b/src/common/json/BalancerStructures.cc @@ -0,0 +1,38 @@ +#include "BalancerStructures.h" + +#include "common/ceph_json.h" + +using namespace ceph::messaging::balancer; + +void BalancerOffRequest::dump(Formatter* f) const { + encode_json("prefix", "balancer off", f); +} + +void BalancerOffRequest::decode_json(JSONObj* obj) {} + +void BalancerStatusRequest::dump(Formatter* f) const { + encode_json("prefix", "balancer status", f); +} + +void BalancerStatusRequest::decode_json(JSONObj* obj) {} + +void BalancerStatusReply::dump(Formatter* f) const { + encode_json("active", active, f); + encode_json("last_optimization_duration", last_optimization_duration, f); + encode_json("last_optimization_started", last_optimization_started, f); + encode_json("mode", mode, f); + encode_json("no_optimization_needed", no_optimization_needed, f); + encode_json("optimize_result", optimize_result, f); +} + +void BalancerStatusReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("active", active, obj); + JSONDecoder::decode_json("last_optimization_duration", + last_optimization_duration, obj); + JSONDecoder::decode_json("last_optimization_started", + last_optimization_started, obj); + JSONDecoder::decode_json("mode", mode, obj); + JSONDecoder::decode_json("no_optimization_needed", no_optimization_needed, + obj); + JSONDecoder::decode_json("optimize_result", optimize_result, obj); +}
\ No newline at end of file diff --git a/src/common/json/BalancerStructures.h b/src/common/json/BalancerStructures.h new file mode 100644 index 00000000000..bbf5c748eb3 --- /dev/null +++ b/src/common/json/BalancerStructures.h @@ -0,0 +1,35 @@ +#pragma once + +#include <string> + +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace balancer { +struct BalancerOffRequest { + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct BalancerStatusRequest { + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct BalancerStatusReply { + bool active; + std::string last_optimization_duration; + std::string last_optimization_started; + std::string mode; + bool no_optimization_needed; + std::string optimize_result; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; +} // namespace balancer +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/CMakeLists.txt b/src/common/json/CMakeLists.txt new file mode 100644 index 00000000000..1497daf93db --- /dev/null +++ b/src/common/json/CMakeLists.txt @@ -0,0 +1,4 @@ +add_library(json_structures STATIC + BalancerStructures.cc ConfigStructures.cc OSDStructures.cc) + + target_link_libraries(json_structures global)
\ No newline at end of file diff --git a/src/common/json/ConfigStructures.cc b/src/common/json/ConfigStructures.cc new file mode 100644 index 00000000000..651278d002a --- /dev/null +++ b/src/common/json/ConfigStructures.cc @@ -0,0 +1,20 @@ +#include "ConfigStructures.h" + +#include "common/ceph_json.h" + +using namespace ceph::messaging::config; + +void ConfigSetRequest::dump(Formatter* f) const { + encode_json("prefix", "config set", f); + encode_json("who", who, f); + encode_json("name", name, f); + encode_json("value", value, f); + encode_json("force", force, f); +} + +void ConfigSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("who", who, obj); + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("value", value, obj); + JSONDecoder::decode_json("force", force, obj); +}
\ No newline at end of file diff --git a/src/common/json/ConfigStructures.h b/src/common/json/ConfigStructures.h new file mode 100644 index 00000000000..554229d75f4 --- /dev/null +++ b/src/common/json/ConfigStructures.h @@ -0,0 +1,24 @@ +#pragma once + +#include <optional> +#include <string> + +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace config { +struct ConfigSetRequest { + std::string who; + std::string name; + std::string value; + std::optional<bool> force; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; +} // namespace config +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/OSDStructures.cc b/src/common/json/OSDStructures.cc new file mode 100644 index 00000000000..aaac5f6e169 --- /dev/null +++ b/src/common/json/OSDStructures.cc @@ -0,0 +1,150 @@ +#include "OSDStructures.h" + +#include "common/ceph_json.h" +#include "common/io_exerciser/OpType.h" + +using namespace ceph::messaging::osd; + +void OSDMapRequest::dump(Formatter* f) const { + encode_json("prefix", "osd map", f); + encode_json("pool", pool, f); + encode_json("object", object, f); + encode_json("nspace", nspace, f); + encode_json("format", format, f); +} + +void OSDMapRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("object", object, obj); + JSONDecoder::decode_json("nspace", nspace, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDMapReply::dump(Formatter* f) const { + encode_json("epoch", epoch, f); + encode_json("pool", pool, f); + encode_json("pool_id", pool_id, f); + encode_json("objname", objname, f); + encode_json("raw_pgid", raw_pgid, f); + encode_json("pgid", pgid, f); + encode_json("up", up, f); + encode_json("up_primary", up_primary, f); + encode_json("acting", acting, f); + encode_json("acting_primary", acting_primary, f); +} + +void OSDMapReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("epoch", epoch, obj); + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("pool_id", pool_id, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("raw_pgid", raw_pgid, obj); + JSONDecoder::decode_json("pgid", pgid, obj); + JSONDecoder::decode_json("up", up, obj); + JSONDecoder::decode_json("up_primary", up_primary, obj); + JSONDecoder::decode_json("acting", acting, obj); + JSONDecoder::decode_json("acting_primary", acting_primary, obj); +} + +void OSDPoolGetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool get", f); + encode_json("pool", pool, f); + encode_json("var", var, f); + encode_json("format", format, f); +} + +void OSDPoolGetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("var", var, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDPoolGetReply::dump(Formatter* f) const { + encode_json("erasure_code_profile", erasure_code_profile, f); +} + +void OSDPoolGetReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("erasure_code_profile", erasure_code_profile, obj); +} + +void OSDECProfileGetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool get", f); + encode_json("name", name, f); + encode_json("format", format, f); +} + +void OSDECProfileGetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDECProfileGetReply::dump(Formatter* f) const { + encode_json("crush-device-class", crush_device_class, f); + encode_json("crush-failure-domain", crush_failure_domain, f); + encode_json("crush-num-failure-domains", crush_num_failure_domains, f); + encode_json("crush-osds-per-failure-domain", crush_osds_per_failure_domain, + f); + encode_json("crush-root", crush_root, f); + encode_json("jerasure-per-chunk-alignment", jerasure_per_chunk_alignment, f); + encode_json("k", k, f); + encode_json("m", m, f); + encode_json("plugin", plugin, f); + encode_json("technique", technique, f); + encode_json("w", w, f); +} + +void OSDECProfileGetReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("crush-device-class", crush_device_class, obj); + JSONDecoder::decode_json("crush-failure-domain", crush_failure_domain, obj); + JSONDecoder::decode_json("crush-num-failure-domains", + crush_num_failure_domains, obj); + JSONDecoder::decode_json("crush-osds-per-failure-domain", + crush_osds_per_failure_domain, obj); + JSONDecoder::decode_json("crush-root", crush_root, obj); + JSONDecoder::decode_json("jerasure-per-chunk-alignment", + jerasure_per_chunk_alignment, obj); + JSONDecoder::decode_json("k", k, obj); + JSONDecoder::decode_json("m", m, obj); + JSONDecoder::decode_json("plugin", plugin, obj); + JSONDecoder::decode_json("technique", technique, obj); + JSONDecoder::decode_json("w", w, obj); +} + +void OSDECProfileSetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd erasure-code-profile set", f); + encode_json("name", name, f); + encode_json("profile", profile, f); +} + +void OSDECProfileSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("profile", profile, obj); +} + +void OSDECPoolCreateRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool create", f); + encode_json("pool", pool, f); + encode_json("pool_type", pool_type, f); + encode_json("pg_num", pg_num, f); + encode_json("pgp_num", pgp_num, f); + encode_json("erasure_code_profile", erasure_code_profile, f); +} + +void OSDECPoolCreateRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("pool_type", pool_type, obj); + JSONDecoder::decode_json("pg_num", pg_num, obj); + JSONDecoder::decode_json("pgp_num", pgp_num, obj); + JSONDecoder::decode_json("erasure_code_profile", erasure_code_profile, obj); +} + +void OSDSetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd set", f); + encode_json("key", key, f); + encode_json("yes_i_really_mean_it", yes_i_really_mean_it, f); +} + +void OSDSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("key", key, obj); + JSONDecoder::decode_json("yes_i_really_mean_it", yes_i_really_mean_it, obj); +}
\ No newline at end of file diff --git a/src/common/json/OSDStructures.h b/src/common/json/OSDStructures.h new file mode 100644 index 00000000000..3e4528a099f --- /dev/null +++ b/src/common/json/OSDStructures.h @@ -0,0 +1,189 @@ +#pragma once + +#include <memory> +#include <string> +#include <vector> + +#include "common/ceph_json.h" +#include "common/io_exerciser/OpType.h" +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace osd { +struct OSDMapRequest { + std::string pool; + std::string object; + std::string nspace; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDMapReply { + epoch_t epoch; + std::string pool; + uint64_t pool_id; + std::string objname; + std::string raw_pgid; + std::string pgid; + std::vector<int> up; + int up_primary; + std::vector<int> acting; + int acting_primary; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDPoolGetRequest { + std::string pool; + std::string var = "erasure_code_profile"; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDPoolGetReply { + std::string erasure_code_profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileGetRequest { + std::string name; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileGetReply { + std::string crush_device_class; + std::string crush_failure_domain; + int crush_num_failure_domains; + int crush_osds_per_failure_domain; + std::string crush_root; + bool jerasure_per_chunk_alignment; + int k; + int m; + std::string plugin; + std::string technique; + std::string w; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileSetRequest { + std::string name; + std::vector<std::string> profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECPoolCreateRequest { + std::string pool; + std::string pool_type; + int pg_num; + int pgp_num; + std::string erasure_code_profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDSetRequest { + std::string key; + std::optional<bool> yes_i_really_mean_it = std::nullopt; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +// These structures are sent directly to the relevant OSD +// rather than the monitor +template <io_exerciser::InjectOpType op_type> +struct InjectECErrorRequest { + std::string pool; + std::string objname; + int shardid; + std::optional<uint64_t> type; + std::optional<uint64_t> when; + std::optional<uint64_t> duration; + + void dump(Formatter* f) const { + switch (op_type) { + case io_exerciser::InjectOpType::ReadEIO: + [[fallthrough]]; + case io_exerciser::InjectOpType::ReadMissingShard: + ::encode_json("prefix", "injectecreaderr", f); + break; + case io_exerciser::InjectOpType::WriteFailAndRollback: + [[fallthrough]]; + case io_exerciser::InjectOpType::WriteOSDAbort: + ::encode_json("prefix", "injectecwriteerr", f); + break; + default: + ceph_abort_msg("Unsupported Inject Type"); + } + ::encode_json("pool", pool, f); + ::encode_json("objname", objname, f); + ::encode_json("shardid", shardid, f); + ::encode_json("type", type, f); + ::encode_json("when", when, f); + ::encode_json("duration", duration, f); + } + void decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("shardid", shardid, obj); + JSONDecoder::decode_json("type", type, obj); + JSONDecoder::decode_json("when", when, obj); + JSONDecoder::decode_json("duration", duration, obj); + } +}; + +template <io_exerciser::InjectOpType op_type> +struct InjectECClearErrorRequest { + std::string pool; + std::string objname; + int shardid; + std::optional<uint64_t> type; + + void dump(Formatter* f) const { + switch (op_type) { + case io_exerciser::InjectOpType::ReadEIO: + [[fallthrough]]; + case io_exerciser::InjectOpType::ReadMissingShard: + ::encode_json("prefix", "injectecclearreaderr", f); + break; + case io_exerciser::InjectOpType::WriteFailAndRollback: + [[fallthrough]]; + case io_exerciser::InjectOpType::WriteOSDAbort: + ::encode_json("prefix", "injectecclearwriteerr", f); + break; + default: + ceph_abort_msg("Unsupported Inject Type"); + } + ::encode_json("pool", pool, f); + ::encode_json("objname", objname, f); + ::encode_json("shardid", shardid, f); + ::encode_json("type", type, f); + } + void decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("shardid", shardid, obj); + JSONDecoder::decode_json("type", type, obj); + } +}; +} // namespace osd +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/options/crimson.yaml.in b/src/common/options/crimson.yaml.in index 69b3a615576..132a4a09e89 100644 --- a/src/common/options/crimson.yaml.in +++ b/src/common/options/crimson.yaml.in @@ -2,6 +2,17 @@ --- options: +- name: crimson_osd_objectstore + type: str + level: advanced + desc: backend type for a Crimson OSD (e.g seastore or bluestore) + default: bluestore + enum_values: + - bluestore + - seastore + - cyanstore + flags: + - create - name: crimson_osd_obc_lru_size type: uint level: advanced diff --git a/src/common/options/mds.yaml.in b/src/common/options/mds.yaml.in index 94824faef6b..03a53cd7cea 100644 --- a/src/common/options/mds.yaml.in +++ b/src/common/options/mds.yaml.in @@ -1713,6 +1713,12 @@ options: default: 1000 services: - mds +- name: mds_delay_journal_replay_for_testing + type: millisecs + level: dev + desc: Delay the journal replay to verify the replay time estimate + long_desc: Jorunal replay warning is activated if the mds has been in replay state for more than 30 seconds. This config delays replay for validating the replay warning in tests. + default: 0 flags: - runtime - name: mds_server_dispatch_killpoint_random @@ -1739,4 +1745,4 @@ options: default: 16 services: - mds - min: 8 + min: 4 diff --git a/src/common/options/mon.yaml.in b/src/common/options/mon.yaml.in index ab1634bc154..1307030e3fb 100644 --- a/src/common/options/mon.yaml.in +++ b/src/common/options/mon.yaml.in @@ -91,6 +91,13 @@ options: default: 1000 services: - mon +- name: mon_nvmeofgw_delete_grace + type: secs + level: advanced + desc: Issue NVMEOF_GATEWAY_DELETING health warning after this amount of time has elapsed + default: 15_min + services: + - mon - name: mon_mgr_inactive_grace type: int level: advanced diff --git a/src/common/pick_address.cc b/src/common/pick_address.cc index d125d7171e0..a0629a15686 100644 --- a/src/common/pick_address.cc +++ b/src/common/pick_address.cc @@ -642,17 +642,24 @@ int get_iface_numa_node( bool is_addr_in_subnet( CephContext *cct, const std::string &networks, - const std::string &addr) + const entity_addr_t &addr) { const auto nets = get_str_list(networks); ceph_assert(!nets.empty()); - unsigned ipv = CEPH_PICK_ADDRESS_IPV4; - struct sockaddr_in public_addr; - public_addr.sin_family = AF_INET; - - if(inet_pton(AF_INET, addr.c_str(), &public_addr.sin_addr) != 1) { - lderr(cct) << "unable to convert chosen address to string: " << addr << dendl; + struct sockaddr_in6 public_addr6; + struct sockaddr_in public_addr4; + + if (addr.is_ipv4() && + inet_pton(AF_INET, addr.ip_only_to_str().c_str(), &public_addr4.sin_addr) == 1) { + public_addr4.sin_family = AF_INET; + } else if (addr.is_ipv6() && + inet_pton(AF_INET6, addr.ip_only_to_str().c_str(), &public_addr6.sin6_addr) == 1) { + public_addr6.sin6_family = AF_INET6; + ipv = CEPH_PICK_ADDRESS_IPV6; + } else { + std::string_view addr_type = addr.is_ipv4() ? "IPv4" : "IPv6"; + lderr(cct) << "IP address " << addr << " is not parseable as " << addr_type << dendl; return false; } @@ -660,10 +667,16 @@ bool is_addr_in_subnet( struct ifaddrs ifa; memset(&ifa, 0, sizeof(ifa)); ifa.ifa_next = nullptr; - ifa.ifa_addr = (struct sockaddr*)&public_addr; + if (addr.is_ipv4()) { + ifa.ifa_addr = (struct sockaddr*)&public_addr4; + } else if (addr.is_ipv6()) { + ifa.ifa_addr = (struct sockaddr*)&public_addr6; + } + if(matches_with_net(cct, ifa, net, ipv)) { return true; } } + lderr(cct) << "address " << addr << " is not in networks '" << networks << "'" << dendl; return false; } diff --git a/src/common/pick_address.h b/src/common/pick_address.h index 40575d7d155..c28a6037ded 100644 --- a/src/common/pick_address.h +++ b/src/common/pick_address.h @@ -98,6 +98,6 @@ int get_iface_numa_node( bool is_addr_in_subnet( CephContext *cct, const std::string &networks, - const std::string &addr); + const entity_addr_t &addr); #endif diff --git a/src/compressor/lz4/LZ4Compressor.cc b/src/compressor/lz4/LZ4Compressor.cc index a209a5ac149..1504a2fe65d 100644 --- a/src/compressor/lz4/LZ4Compressor.cc +++ b/src/compressor/lz4/LZ4Compressor.cc @@ -121,16 +121,12 @@ int LZ4Compressor::decompress(ceph::buffer::list::const_iterator &p, LZ4_streamDecode_t lz4_stream_decode; LZ4_setStreamDecode(&lz4_stream_decode, nullptr, 0); - ceph::buffer::ptr cur_ptr = p.get_current_ptr(); - ceph::buffer::ptr *ptr = &cur_ptr; - std::optional<ceph::buffer::ptr> data_holder; - if (compressed_len != cur_ptr.length()) { - data_holder.emplace(compressed_len); - p.copy_deep(compressed_len, *data_holder); - ptr = &*data_holder; - } - - char *c_in = ptr->c_str(); + ceph::buffer::list indata; + // this does a shallow copy + p.copy(compressed_len, indata); + // if the input isn't fragmented, c_str() costs almost nothing. + // otherwise rectifying copy will be taken + const char* c_in = indata.c_str(); char *c_out = dstptr.c_str(); for (unsigned i = 0; i < count; ++i) { int r = LZ4_decompress_safe_continue( diff --git a/src/crimson/admin/osd_admin.cc b/src/crimson/admin/osd_admin.cc index de9626a2f2d..41da72c9fde 100644 --- a/src/crimson/admin/osd_admin.cc +++ b/src/crimson/admin/osd_admin.cc @@ -14,6 +14,7 @@ #include "common/config.h" #include "crimson/admin/admin_socket.h" #include "crimson/common/log.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/osd/exceptions.h" #include "crimson/osd/osd.h" #include "crimson/osd/pg.h" diff --git a/src/crimson/common/logclient.cc b/src/crimson/common/logclient.cc index d402ecd1901..a3c30227bc7 100644 --- a/src/crimson/common/logclient.cc +++ b/src/crimson/common/logclient.cc @@ -7,6 +7,7 @@ #include "crimson/net/Messenger.h" #include "crimson/mon/MonClient.h" #include "mon/MonMap.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/Graylog.h" using std::map; diff --git a/src/crimson/common/shared_lru.h b/src/crimson/common/shared_lru.h index 92d99d332c4..0d73658e709 100644 --- a/src/crimson/common/shared_lru.h +++ b/src/crimson/common/shared_lru.h @@ -25,12 +25,17 @@ class SharedLRU { SimpleLRU<K, shared_ptr_t, false> cache; std::map<K, std::pair<weak_ptr_t, V*>> weak_refs; + // Once all of the shared pointers are destoryed, + // erase the tracked object from the weak_ref map + // before actually destorying it struct Deleter { - SharedLRU<K,V>* cache; + SharedLRU<K,V>* shared_lru_ptr; const K key; - void operator()(V* ptr) { - cache->_erase_weak(key); - delete ptr; + void operator()(V* value_ptr) { + if (shared_lru_ptr) { + shared_lru_ptr->_erase_weak(key); + } + delete value_ptr; } }; void _erase_weak(const K& key) { @@ -42,9 +47,19 @@ public: {} ~SharedLRU() { cache.clear(); + // initially, we were assuming that no pointer obtained from SharedLRU // can outlive the lru itself. However, since going with the interruption // concept for handling shutdowns, this is no longer valid. + // Moreover, before clearing weak_refs, invalidate each deleter + // cache pointer as this SharedLRU is being destoryed. + for (const auto& [key, value] : weak_refs) { + shared_ptr_t val; + val = value.first.lock(); + auto this_deleter = get_deleter<Deleter>(val); + this_deleter->shared_lru_ptr = nullptr; + } + weak_refs.clear(); } /** diff --git a/src/crimson/common/tmap_helpers.cc b/src/crimson/common/tmap_helpers.cc index 9c14ebc450e..58c4fc7e218 100644 --- a/src/crimson/common/tmap_helpers.cc +++ b/src/crimson/common/tmap_helpers.cc @@ -7,6 +7,8 @@ #include "include/encoding.h" #include "include/rados.h" +#include <map> + namespace detail { #define decode_or_return(v, bp) \ diff --git a/src/crimson/mon/MonClient.cc b/src/crimson/mon/MonClient.cc index 4919f0bf21f..4c076cf43c6 100644 --- a/src/crimson/mon/MonClient.cc +++ b/src/crimson/mon/MonClient.cc @@ -13,6 +13,7 @@ #include "auth/AuthClientHandler.h" #include "auth/RotatingKeyRing.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/hostname.h" #include "include/utime_fmt.h" diff --git a/src/crimson/net/Socket.cc b/src/crimson/net/Socket.cc index 2c729f4e8c2..3a7aeaf9651 100644 --- a/src/crimson/net/Socket.cc +++ b/src/crimson/net/Socket.cc @@ -8,6 +8,7 @@ #include <seastar/net/packet.hh> #include "crimson/common/log.h" +#include "include/random.h" // for ceph::util::generate_random_number() #include "Errors.h" using crimson::common::local_conf; diff --git a/src/crimson/net/io_handler.cc b/src/crimson/net/io_handler.cc index b93124f3c12..bc5e9bf404c 100644 --- a/src/crimson/net/io_handler.cc +++ b/src/crimson/net/io_handler.cc @@ -347,7 +347,7 @@ void IOHandler::do_set_io_state( { ceph_assert_always(seastar::this_shard_id() == get_shard_id()); auto prv_state = get_io_state(); - logger().debug("{} got {}do_set_io_state(): prv_state={}, new_state={}, " + logger().debug("{} got {} do_set_io_state(): prv_state={}, new_state={}, " "fa={}, set_notify_out={}, at {}", conn, cc_seq.has_value() ? fmt::format("{} ", *cc_seq) : "", @@ -984,7 +984,7 @@ void IOHandler::notify_out_dispatch() }); }); } - if (shard_states->try_enter_out_dispatching()) { + if (shard_states->try_enter_out_dispatching(conn)) { shard_states->dispatch_in_background( "do_out_dispatch", conn, [this] { return do_out_dispatch(*shard_states); diff --git a/src/crimson/net/io_handler.h b/src/crimson/net/io_handler.h index 5986fcb16ac..41c76ab925b 100644 --- a/src/crimson/net/io_handler.h +++ b/src/crimson/net/io_handler.h @@ -309,7 +309,7 @@ public: in_exit_dispatching = std::nullopt; } - bool try_enter_out_dispatching() { + bool try_enter_out_dispatching(SocketConnection &conn) { assert(seastar::this_shard_id() == sid); if (out_dispatching) { // already dispatching out @@ -327,6 +327,9 @@ public: // do not dispatch out return false; default: + crimson::get_logger(ceph_subsys_ms).error( + "{} try_enter_out_dispatching() got wrong io_state {}", + conn, io_state); ceph_abort("impossible"); } } @@ -574,6 +577,8 @@ struct fmt::formatter<crimson::net::IOHandler::io_state_t> case switched: name = "switched"; break; + default: + name = "undefined"; } return formatter<string_view>::format(name, ctx); } diff --git a/src/crimson/os/alienstore/alien_store.cc b/src/crimson/os/alienstore/alien_store.cc index f390823a8a0..db6decd84f9 100644 --- a/src/crimson/os/alienstore/alien_store.cc +++ b/src/crimson/os/alienstore/alien_store.cc @@ -141,7 +141,8 @@ seastar::future<> AlienStore::stop() AlienStore::base_errorator::future<bool> AlienStore::exists( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { return op_gates.simple_dispatch("exists", [=, this] { return tp->submit(ch->get_cid().hash_to_shard(tp->size()), [=, this] { @@ -212,7 +213,8 @@ seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> AlienStore::list_objects(CollectionRef ch, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const + uint64_t limit, + uint32_t op_flags) const { logger().debug("{}", __func__); assert(tp); @@ -348,7 +350,8 @@ AlienStore::readv(CollectionRef ch, AlienStore::get_attr_errorator::future<ceph::bufferlist> AlienStore::get_attr(CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { logger().debug("{}", __func__); assert(tp); @@ -376,7 +379,8 @@ AlienStore::get_attr(CollectionRef ch, AlienStore::get_attrs_ertr::future<AlienStore::attrs_t> AlienStore::get_attrs(CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { logger().debug("{}", __func__); assert(tp); @@ -397,7 +401,8 @@ AlienStore::get_attrs(CollectionRef ch, auto AlienStore::omap_get_values(CollectionRef ch, const ghobject_t& oid, - const set<string>& keys) + const set<string>& keys, + uint32_t op_flags) -> read_errorator::future<omap_values_t> { logger().debug("{}", __func__); @@ -421,7 +426,8 @@ auto AlienStore::omap_get_values(CollectionRef ch, auto AlienStore::omap_get_values(CollectionRef ch, const ghobject_t &oid, - const std::optional<string> &start) + const std::optional<string> &start, + uint32_t op_flags) -> read_errorator::future<std::tuple<bool, omap_values_t>> { logger().debug("{} with_start", __func__); @@ -429,8 +435,21 @@ auto AlienStore::omap_get_values(CollectionRef ch, return do_with_op_gate(omap_values_t{}, [=, this] (auto &values) { return tp->submit(ch->get_cid().hash_to_shard(tp->size()), [=, this, &values] { auto c = static_cast<AlienCollection*>(ch.get()); - return store->omap_get_values(c->collection, oid, start, - reinterpret_cast<map<string, bufferlist>*>(&values)); + return store->omap_iterate( + c->collection, oid, + ObjectStore::omap_iter_seek_t{ + .seek_position = start.value_or(std::string{}), + // FIXME: classical OSDs begins iteration from LOWER_BOUND + // (or UPPER_BOUND if filter_prefix > start). However, these + // bits are not implemented yet + .seek_type = ObjectStore::omap_iter_seek_t::UPPER_BOUND + }, + [&values] + (std::string_view key, std::string_view value) mutable { + values[std::string{key}].append(value); + // FIXME: there is limit on number of entries yet + return ObjectStore::omap_iter_ret_t::NEXT; + }); }).then([&values] (int r) -> read_errorator::future<std::tuple<bool, omap_values_t>> { if (r == -ENOENT) { @@ -578,7 +597,8 @@ unsigned AlienStore::get_max_attr_name_length() const seastar::future<struct stat> AlienStore::stat( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { assert(tp); return do_with_op_gate((struct stat){}, [this, ch, oid](auto& st) { @@ -604,7 +624,8 @@ seastar::future<std::string> AlienStore::get_default_device_class() } auto AlienStore::omap_get_header(CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) -> get_attr_errorator::future<ceph::bufferlist> { assert(tp); @@ -630,7 +651,8 @@ AlienStore::read_errorator::future<std::map<uint64_t, uint64_t>> AlienStore::fie CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { assert(tp); return do_with_op_gate(std::map<uint64_t, uint64_t>(), [=, this](auto& destmap) { diff --git a/src/crimson/os/alienstore/alien_store.h b/src/crimson/os/alienstore/alien_store.h index 853585dac9c..1d39411450e 100644 --- a/src/crimson/os/alienstore/alien_store.h +++ b/src/crimson/os/alienstore/alien_store.h @@ -36,7 +36,8 @@ public: base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; mkfs_ertr::future<> mkfs(uuid_d new_osd_fsid) final; read_errorator::future<ceph::bufferlist> read(CollectionRef c, const ghobject_t& oid, @@ -49,29 +50,36 @@ public: uint32_t op_flags = 0) final; - get_attr_errorator::future<ceph::bufferlist> get_attr(CollectionRef c, - const ghobject_t& oid, - std::string_view name) const final; - get_attrs_ertr::future<attrs_t> get_attrs(CollectionRef c, - const ghobject_t& oid) final; + get_attr_errorator::future<ceph::bufferlist> get_attr( + CollectionRef c, + const ghobject_t& oid, + std::string_view name, + uint32_t op_flags = 0) const final; + get_attrs_ertr::future<attrs_t> get_attrs( + CollectionRef c, + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; /// Retrieves paged set of values > start (if present) read_errorator::future<std::tuple<bool, omap_values_t>> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; ///< @return <done, values> values.empty() iff done seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; seastar::future<CollectionRef> open_collection(const coll_t& cid) final; @@ -97,16 +105,19 @@ public: unsigned get_max_attr_name_length() const final; seastar::future<struct stat> stat( CollectionRef, - const ghobject_t&) final; + const ghobject_t&, + uint32_t op_flags = 0) final; seastar::future<std::string> get_default_device_class() final; get_attr_errorator::future<ceph::bufferlist> omap_get_header( CollectionRef, - const ghobject_t&) final; + const ghobject_t&, + uint32_t) final; read_errorator::future<std::map<uint64_t, uint64_t>> fiemap( CollectionRef, const ghobject_t&, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags) final; FuturizedStore::Shard& get_sharded_store() final { return *this; diff --git a/src/crimson/os/alienstore/thread_pool.cc b/src/crimson/os/alienstore/thread_pool.cc index 277055ec51e..2d208548b32 100644 --- a/src/crimson/os/alienstore/thread_pool.cc +++ b/src/crimson/os/alienstore/thread_pool.cc @@ -7,6 +7,7 @@ #include <pthread.h> #include "include/ceph_assert.h" +#include "include/intarith.h" // for round_up_to() #include "crimson/common/config_proxy.h" using crimson::common::local_conf; diff --git a/src/crimson/os/cyanstore/cyan_store.cc b/src/crimson/os/cyanstore/cyan_store.cc index 3f861a9271f..41819fb5eb6 100644 --- a/src/crimson/os/cyanstore/cyan_store.cc +++ b/src/crimson/os/cyanstore/cyan_store.cc @@ -12,6 +12,7 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" +#include "crimson/common/perf_counters_collection.h" #include "cyan_collection.h" #include "cyan_object.h" @@ -207,7 +208,8 @@ CyanStore::Shard::list_objects( CollectionRef ch, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const + uint64_t limit, + uint32_t op_flags) const { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {} {} {}", @@ -256,7 +258,8 @@ CyanStore::Shard::list_collections() CyanStore::Shard::base_errorator::future<bool> CyanStore::Shard::exists( CollectionRef ch, - const ghobject_t &oid) + const ghobject_t &oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); if (!c->exists) { @@ -332,7 +335,8 @@ CyanStore::Shard::get_attr_errorator::future<ceph::bufferlist> CyanStore::Shard::get_attr( CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {}", @@ -351,7 +355,8 @@ CyanStore::Shard::get_attr( CyanStore::Shard::get_attrs_ertr::future<CyanStore::Shard::attrs_t> CyanStore::Shard::get_attrs( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {}", @@ -366,7 +371,8 @@ CyanStore::Shard::get_attrs( auto CyanStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t& oid, - const omap_keys_t& keys) + const omap_keys_t& keys, + uint32_t op_flags) -> read_errorator::future<omap_values_t> { auto c = static_cast<Collection*>(ch.get()); @@ -387,7 +393,8 @@ auto CyanStore::Shard::omap_get_values( auto CyanStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const std::optional<string> &start) + const std::optional<string> &start, + uint32_t op_flags) -> CyanStore::Shard::read_errorator::future<std::tuple<bool, omap_values_t>> { auto c = static_cast<Collection*>(ch.get()); @@ -408,7 +415,8 @@ auto CyanStore::Shard::omap_get_values( auto CyanStore::Shard::omap_get_header( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) -> CyanStore::Shard::get_attr_errorator::future<ceph::bufferlist> { auto c = static_cast<Collection*>(ch.get()); @@ -976,7 +984,8 @@ CyanStore::Shard::fiemap( CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); @@ -991,7 +1000,8 @@ CyanStore::Shard::fiemap( seastar::future<struct stat> CyanStore::Shard::stat( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); auto o = c->get_object(oid); diff --git a/src/crimson/os/cyanstore/cyan_store.h b/src/crimson/os/cyanstore/cyan_store.h index e9394991bc2..1d481ef5829 100644 --- a/src/crimson/os/cyanstore/cyan_store.h +++ b/src/crimson/os/cyanstore/cyan_store.h @@ -34,11 +34,13 @@ public: seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; base_errorator::future<bool> exists( CollectionRef ch, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<ceph::bufferlist> read( CollectionRef c, @@ -56,33 +58,39 @@ public: get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const final; + std::string_view name, + uint32_t op_flags = 0) const final; get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; read_errorator::future<std::tuple<bool, omap_values_t>> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; get_attr_errorator::future<ceph::bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; @@ -101,7 +109,8 @@ public: CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags) final; unsigned get_max_attr_name_length() const final; diff --git a/src/crimson/os/futurized_store.h b/src/crimson/os/futurized_store.h index 51ef2331014..e7d4c8546de 100644 --- a/src/crimson/os/futurized_store.h +++ b/src/crimson/os/futurized_store.h @@ -54,7 +54,8 @@ public: virtual base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; using get_attr_errorator = crimson::errorator< crimson::ct_error::enoent, @@ -62,42 +63,49 @@ public: virtual get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const = 0; + std::string_view name, + uint32_t op_flags = 0) const = 0; using get_attrs_ertr = crimson::errorator< crimson::ct_error::enoent>; using attrs_t = std::map<std::string, ceph::bufferlist, std::less<>>; virtual get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; virtual seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; using omap_values_t = attrs_t; using omap_keys_t = std::set<std::string>; virtual read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) = 0; + const omap_keys_t& keys, + uint32_t op_flags = 0) = 0; using omap_values_paged_t = std::tuple<bool, omap_values_t>; virtual read_errorator::future<omap_values_paged_t> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) = 0; ///< @return <done, values> values.empty() only if done virtual get_attr_errorator::future<bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; virtual seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const = 0; + uint64_t limit, + uint32_t op_flags = 0) const = 0; virtual seastar::future<CollectionRef> create_new_collection(const coll_t& cid) = 0; @@ -153,7 +161,8 @@ public: CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) = 0; + uint64_t len, + uint32_t op_flags = 0) = 0; virtual unsigned get_max_attr_name_length() const = 0; }; diff --git a/src/crimson/os/seastore/CMakeLists.txt b/src/crimson/os/seastore/CMakeLists.txt index e5b8960c38c..3da5e65ceec 100644 --- a/src/crimson/os/seastore/CMakeLists.txt +++ b/src/crimson/os/seastore/CMakeLists.txt @@ -1,5 +1,6 @@ set(crimson_seastore_srcs cached_extent.cc + lba_mapping.cc seastore_types.cc segment_manager.cc segment_manager/ephemeral.cc @@ -19,7 +20,6 @@ set(crimson_seastore_srcs omap_manager.cc omap_manager/btree/btree_omap_manager.cc omap_manager/btree/omap_btree_node_impl.cc - btree/btree_range_pin.cc btree/fixed_kv_node.cc onode.cc onode_manager/staged-fltree/node.cc diff --git a/src/crimson/os/seastore/async_cleaner.cc b/src/crimson/os/seastore/async_cleaner.cc index cecdb985532..64e6749562e 100644 --- a/src/crimson/os/seastore/async_cleaner.cc +++ b/src/crimson/os/seastore/async_cleaner.cc @@ -609,6 +609,7 @@ JournalTrimmerImpl::trim_alloc() return extent_callback->with_transaction_intr( Transaction::src_t::TRIM_ALLOC, "trim_alloc", + CACHE_HINT_NOCACHE, [this, FNAME](auto &t) { auto target = get_alloc_tail_target(); @@ -653,6 +654,7 @@ JournalTrimmerImpl::trim_dirty() return extent_callback->with_transaction_intr( Transaction::src_t::TRIM_DIRTY, "trim_dirty", + CACHE_HINT_NOCACHE, [this, FNAME](auto &t) { auto target = get_dirty_tail_target(); @@ -1125,6 +1127,7 @@ SegmentCleaner::do_reclaim_space( return extent_callback->with_transaction_intr( src, "clean_reclaim_space", + CACHE_HINT_NOCACHE, [this, &backref_extents, &pin_list, &reclaimed](auto &t) { return seastar::do_with( @@ -1142,8 +1145,7 @@ SegmentCleaner::do_reclaim_space( pin->get_key(), pin->get_val(), pin->get_length(), - pin->get_type(), - JOURNAL_SEQ_NULL); + pin->get_type()); } for (auto &cached_backref : cached_backref_entries) { if (cached_backref.laddr == L_ADDR_NULL) { @@ -1241,6 +1243,7 @@ SegmentCleaner::clean_space_ret SegmentCleaner::clean_space() return extent_callback->with_transaction_intr( Transaction::src_t::READ, "retrieve_from_backref_tree", + CACHE_HINT_NOCACHE, [this, &weak_read_ret](auto &t) { return backref_manager.get_mappings( t, @@ -1507,6 +1510,7 @@ bool SegmentCleaner::check_usage() SpaceTrackerIRef tracker(space_tracker->make_empty()); extent_callback->with_transaction_weak( "check_usage", + CACHE_HINT_NOCACHE, [this, &tracker](auto &t) { return backref_manager.scan_mapped_space( t, @@ -1813,6 +1817,7 @@ bool RBMCleaner::check_usage() RBMSpaceTracker tracker(rbms); extent_callback->with_transaction_weak( "check_usage", + CACHE_HINT_NOCACHE, [this, &tracker, &rbms](auto &t) { return backref_manager.scan_mapped_space( t, diff --git a/src/crimson/os/seastore/async_cleaner.h b/src/crimson/os/seastore/async_cleaner.h index 424247c5bdc..1cef771aeb8 100644 --- a/src/crimson/os/seastore/async_cleaner.h +++ b/src/crimson/os/seastore/async_cleaner.h @@ -17,6 +17,7 @@ #include "crimson/os/seastore/randomblock_manager_group.h" #include "crimson/os/seastore/transaction.h" #include "crimson/os/seastore/segment_seq_allocator.h" +#include "crimson/os/seastore/backref_mapping.h" namespace crimson::os::seastore { @@ -299,24 +300,29 @@ public: /// Creates empty transaction /// weak transaction should be type READ virtual TransactionRef create_transaction( - Transaction::src_t, const char *name, bool is_weak=false) = 0; + Transaction::src_t, + const char *name, + cache_hint_t cache_hint = CACHE_HINT_TOUCH, + bool is_weak=false) = 0; /// Creates empty transaction with interruptible context template <typename Func> auto with_transaction_intr( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, Func &&f) { return do_with_transaction_intr<Func, false>( - src, name, std::forward<Func>(f)); + src, name, cache_hint, std::forward<Func>(f)); } template <typename Func> auto with_transaction_weak( const char* name, + cache_hint_t cache_hint, Func &&f) { return do_with_transaction_intr<Func, true>( - Transaction::src_t::READ, name, std::forward<Func>(f) + Transaction::src_t::READ, name, cache_hint, std::forward<Func>(f) ).handle_error( crimson::ct_error::eagain::assert_failure{"unexpected eagain"}, crimson::ct_error::pass_further_all{} @@ -385,9 +391,10 @@ private: auto do_with_transaction_intr( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, Func &&f) { return seastar::do_with( - create_transaction(src, name, IsWeak), + create_transaction(src, name, cache_hint, IsWeak), [f=std::forward<Func>(f)](auto &ref_t) mutable { return with_trans_intr( *ref_t, diff --git a/src/crimson/os/seastore/backref/btree_backref_manager.h b/src/crimson/os/seastore/backref/btree_backref_manager.h index 38084bb00e6..24897dd55da 100644 --- a/src/crimson/os/seastore/backref/btree_backref_manager.h +++ b/src/crimson/os/seastore/backref/btree_backref_manager.h @@ -9,44 +9,28 @@ namespace crimson::os::seastore::backref { -constexpr size_t BACKREF_BLOCK_SIZE = 4096; - -class BtreeBackrefMapping : public BtreeNodeMapping<paddr_t, laddr_t> { - extent_types_t type; +class BtreeBackrefMapping : public BackrefMapping { public: BtreeBackrefMapping(op_context_t<paddr_t> ctx) - : BtreeNodeMapping(ctx) {} + : BackrefMapping(ctx) {} BtreeBackrefMapping( op_context_t<paddr_t> ctx, CachedExtentRef parent, uint16_t pos, backref_map_val_t &val, backref_node_meta_t &&meta) - : BtreeNodeMapping( + : BackrefMapping( + val.type, ctx, parent, pos, val.laddr, val.len, - std::forward<backref_node_meta_t>(meta)), - type(val.type) - {} - extent_types_t get_type() const final { - return type; - } - - bool is_clone() const final { - return false; - } - -protected: - std::unique_ptr<BtreeNodeMapping<paddr_t, laddr_t>> _duplicate( - op_context_t<paddr_t> ctx) const final { - return std::unique_ptr<BtreeNodeMapping<paddr_t, laddr_t>>( - new BtreeBackrefMapping(ctx)); - } + std::forward<backref_node_meta_t>(meta)) {} }; +constexpr size_t BACKREF_BLOCK_SIZE = 4096; + using BackrefBtree = FixedKVBtree< paddr_t, backref_map_val_t, BackrefInternalNode, BackrefLeafNode, BtreeBackrefMapping, BACKREF_BLOCK_SIZE, false>; diff --git a/src/crimson/os/seastore/backref_entry.h b/src/crimson/os/seastore/backref_entry.h new file mode 100644 index 00000000000..5f9becc9565 --- /dev/null +++ b/src/crimson/os/seastore/backref_entry.h @@ -0,0 +1,127 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include <memory> +#include <iostream> + +#if FMT_VERSION >= 90000 +#include <fmt/ostream.h> +#endif + +#include <boost/intrusive/set.hpp> + +#include "crimson/os/seastore/seastore_types.h" + +namespace crimson::os::seastore { + +struct backref_entry_t { + using ref_t = std::unique_ptr<backref_entry_t>; + + backref_entry_t( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) + : paddr(paddr), + laddr(laddr), + len(len), + type(type) { + assert(len > 0); + } + paddr_t paddr = P_ADDR_NULL; + laddr_t laddr = L_ADDR_NULL; + extent_len_t len = 0; + extent_types_t type = extent_types_t::NONE; + friend bool operator< ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr < r.paddr; + } + friend bool operator> ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr > r.paddr; + } + friend bool operator== ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr == r.paddr; + } + + using set_hook_t = + boost::intrusive::set_member_hook< + boost::intrusive::link_mode< + boost::intrusive::auto_unlink>>; + set_hook_t backref_set_hook; + using backref_set_member_options = boost::intrusive::member_hook< + backref_entry_t, + set_hook_t, + &backref_entry_t::backref_set_hook>; + using multiset_t = boost::intrusive::multiset< + backref_entry_t, + backref_set_member_options, + boost::intrusive::constant_time_size<false>>; + + struct cmp_t { + using is_transparent = paddr_t; + bool operator()( + const backref_entry_t &l, + const backref_entry_t &r) const { + return l.paddr < r.paddr; + } + bool operator()(const paddr_t l, const backref_entry_t &r) const { + return l < r.paddr; + } + bool operator()(const backref_entry_t &l, const paddr_t r) const { + return l.paddr < r; + } + }; + + static ref_t create_alloc( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type)); + assert(laddr != L_ADDR_NULL); + return std::make_unique<backref_entry_t>( + paddr, laddr, len, type); + } + + static ref_t create_retire( + const paddr_t& paddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type) || + is_retired_placeholder_type(type)); + return std::make_unique<backref_entry_t>( + paddr, L_ADDR_NULL, len, type); + } + + static ref_t create(const alloc_blk_t& delta) { + return std::make_unique<backref_entry_t>( + delta.paddr, delta.laddr, delta.len, delta.type); + } +}; + +inline std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent) { + return out << "backref_entry_t{" + << ent.paddr << "~0x" << std::hex << ent.len << std::dec << ", " + << "laddr: " << ent.laddr << ", " + << "type: " << ent.type + << "}"; +} + +using backref_entry_ref = backref_entry_t::ref_t; +using backref_entry_mset_t = backref_entry_t::multiset_t; +using backref_entry_refs_t = std::vector<backref_entry_ref>; +using backref_entryrefs_by_seq_t = std::map<journal_seq_t, backref_entry_refs_t>; +using backref_entry_query_set_t = std::set<backref_entry_t, backref_entry_t::cmp_t>; + +} // namespace crimson::os::seastore + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::os::seastore::backref_entry_t> : fmt::ostream_formatter {}; +#endif diff --git a/src/crimson/os/seastore/backref_manager.h b/src/crimson/os/seastore/backref_manager.h index 3feedb997b4..8c746b571b2 100644 --- a/src/crimson/os/seastore/backref_manager.h +++ b/src/crimson/os/seastore/backref_manager.h @@ -6,6 +6,7 @@ #include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/transaction.h" +#include "crimson/os/seastore/backref_mapping.h" namespace crimson::os::seastore { diff --git a/src/crimson/os/seastore/backref_mapping.h b/src/crimson/os/seastore/backref_mapping.h new file mode 100644 index 00000000000..d0a6a0ea6ff --- /dev/null +++ b/src/crimson/os/seastore/backref_mapping.h @@ -0,0 +1,27 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include "crimson/os/seastore/btree/btree_range_pin.h" + +namespace crimson::os::seastore { + +class BackrefMapping : public BtreeNodeMapping<paddr_t, laddr_t> { + extent_types_t type; +public: + BackrefMapping(op_context_t<paddr_t> ctx) + : BtreeNodeMapping(ctx) {} + template <typename... T> + BackrefMapping(extent_types_t type, T&&... t) + : BtreeNodeMapping(std::forward<T>(t)...), + type(type) {} + extent_types_t get_type() const { + return type; + } +}; + +using BackrefMappingRef = std::unique_ptr<BackrefMapping>; +using backref_pin_list_t = std::list<BackrefMappingRef>; + +} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/btree/btree_range_pin.cc b/src/crimson/os/seastore/btree/btree_range_pin.cc deleted file mode 100644 index f0d507a24c4..00000000000 --- a/src/crimson/os/seastore/btree/btree_range_pin.cc +++ /dev/null @@ -1,54 +0,0 @@ -// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- -// vim: ts=8 sw=2 smarttab - -#include "crimson/os/seastore/btree/btree_range_pin.h" -#include "crimson/os/seastore/btree/fixed_kv_node.h" - -namespace crimson::os::seastore { - -template <typename key_t, typename val_t> -get_child_ret_t<LogicalCachedExtent> -BtreeNodeMapping<key_t, val_t>::get_logical_extent( - Transaction &t) -{ - ceph_assert(is_parent_viewable()); - assert(pos != std::numeric_limits<uint16_t>::max()); - ceph_assert(t.get_trans_id() == ctx.trans.get_trans_id()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - auto v = p.template get_child<LogicalCachedExtent>(ctx, pos, k); - if (!v.has_child()) { - this->child_pos = v.get_child_pos(); - } - return v; -} - -template <typename key_t, typename val_t> -bool BtreeNodeMapping<key_t, val_t>::is_stable() const -{ - assert(!this->parent_modified()); - assert(pos != std::numeric_limits<uint16_t>::max()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - return p.is_child_stable(ctx, pos, k); -} - -template <typename key_t, typename val_t> -bool BtreeNodeMapping<key_t, val_t>::is_data_stable() const -{ - assert(!this->parent_modified()); - assert(pos != std::numeric_limits<uint16_t>::max()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - return p.is_child_data_stable(ctx, pos, k); -} - -template class BtreeNodeMapping<laddr_t, paddr_t>; -template class BtreeNodeMapping<paddr_t, laddr_t>; -} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/btree/btree_range_pin.h b/src/crimson/os/seastore/btree/btree_range_pin.h index 91751801e5d..bfd350a8bed 100644 --- a/src/crimson/os/seastore/btree/btree_range_pin.h +++ b/src/crimson/os/seastore/btree/btree_range_pin.h @@ -7,11 +7,12 @@ #include "crimson/common/log.h" -#include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/seastore_types.h" +#include "crimson/os/seastore/transaction.h" namespace crimson::os::seastore { +class Cache; template <typename node_key_t> struct op_context_t { @@ -116,8 +117,6 @@ protected: extent_len_t len = 0; fixed_kv_node_meta_t<key_t> range; uint16_t pos = std::numeric_limits<uint16_t>::max(); - - virtual std::unique_ptr<BtreeNodeMapping> _duplicate(op_context_t<key_t>) const = 0; fixed_kv_node_meta_t<key_t> _get_pin_range() const { return range; } @@ -139,11 +138,7 @@ public: len(len), range(meta), pos(pos) - { - if (!parent->is_pending()) { - this->child_pos = {parent, pos}; - } - } + {} CachedExtentRef get_parent() const final { return parent; @@ -162,11 +157,6 @@ public: return len; } - extent_types_t get_type() const override { - ceph_abort("should never happen"); - return extent_types_t::ROOT; - } - val_t get_val() const final { if constexpr (std::is_same_v<val_t, paddr_t>) { return value.get_paddr(); @@ -180,16 +170,6 @@ public: return range.begin; } - PhysicalNodeMappingRef<key_t, val_t> duplicate() const final { - auto ret = _duplicate(ctx); - ret->range = range; - ret->value = value; - ret->parent = parent; - ret->len = len; - ret->pos = pos; - return ret; - } - bool has_been_invalidated() const final { return parent->has_been_invalidated(); } @@ -215,9 +195,6 @@ public: return unviewable; } - get_child_ret_t<LogicalCachedExtent> get_logical_extent(Transaction&) final; - bool is_stable() const final; - bool is_data_stable() const final; bool is_parent_viewable() const final { ceph_assert(parent); if (!parent->is_valid()) { diff --git a/src/crimson/os/seastore/cache.cc b/src/crimson/os/seastore/cache.cc index cdd9c542d95..86f816e1648 100644 --- a/src/crimson/os/seastore/cache.cc +++ b/src/crimson/os/seastore/cache.cc @@ -28,15 +28,6 @@ SET_SUBSYS(seastore_cache); namespace crimson::os::seastore { -std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent) { - return out << "backref_entry_t{" - << ent.paddr << "~0x" << std::hex << ent.len << std::dec << ", " - << "laddr: " << ent.laddr << ", " - << "type: " << ent.type << ", " - << "seq: " << ent.seq << ", " - << "}"; -} - Cache::Cache( ExtentPlacementManager &epm) : epm(epm), @@ -1348,21 +1339,39 @@ record_t Cache::prepare_record( io_stat_t retire_stat; std::vector<alloc_delta_t> alloc_deltas; alloc_delta_t rel_delta; + backref_entry_refs_t backref_entries; rel_delta.op = alloc_delta_t::op_types_t::CLEAR; for (auto &i: t.retired_set) { auto &extent = i.extent; get_by_ext(efforts.retire_by_ext, extent->get_type()).increment(extent->get_length()); retire_stat.increment(extent->get_length()); - DEBUGT("retired and remove extent -- {}", t, *extent); + DEBUGT("retired and remove extent {}~0x{:x} -- {}", + t, extent->get_paddr(), extent->get_length(), *extent); commit_retire_extent(t, extent); - if (is_backref_mapped_extent_node(extent) || - is_retired_placeholder_type(extent->get_type())) { + + // Note: commit extents and backref allocations in the same place + if (is_backref_mapped_type(extent->get_type()) || + is_retired_placeholder_type(extent->get_type())) { + DEBUGT("backref_entry free {}~0x{:x}", + t, + extent->get_paddr(), + extent->get_length()); rel_delta.alloc_blk_ranges.emplace_back( - extent->get_paddr(), - L_ADDR_NULL, - extent->get_length(), - extent->get_type()); + alloc_blk_t::create_retire( + extent->get_paddr(), + extent->get_length(), + extent->get_type())); + backref_entries.emplace_back( + backref_entry_t::create_retire( + extent->get_paddr(), + extent->get_length(), + extent->get_type())); + } else if (is_backref_node(extent->get_type())) { + remove_backref_extent(extent->get_paddr()); + } else { + ERRORT("Got unexpected extent type: {}", t, *extent); + ceph_abort("imposible"); } } alloc_deltas.emplace_back(std::move(rel_delta)); @@ -1399,27 +1408,40 @@ record_t Cache::prepare_record( if (modify_time == NULL_TIME) { modify_time = commit_time; } + laddr_t fresh_laddr; + if (i->is_logical()) { + fresh_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + fresh_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + fresh_laddr = L_ADDR_NULL; + } record.push_back(extent_t{ i->get_type(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), + fresh_laddr, std::move(bl) }, modify_time); - if (i->is_valid() - && is_backref_mapped_extent_node(i)) { + + if (!i->is_valid()) { + continue; + } + if (is_backref_mapped_type(i->get_type())) { + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + assert(i->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL); + alloc_laddr = L_ADDR_MIN; + } alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), - i->get_length(), - i->get_type()); + alloc_blk_t::create_alloc( + i->get_paddr(), + alloc_laddr, + i->get_length(), + i->get_type())); } } @@ -1430,14 +1452,20 @@ record_t Cache::prepare_record( get_by_ext(efforts.fresh_ool_by_ext, i->get_type()).increment(i->get_length()); i->prepare_commit(); - if (is_backref_mapped_extent_node(i)) { + if (is_backref_mapped_type(i->get_type())) { + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else { + assert(is_lba_node(i->get_type())); + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin, - i->get_length(), - i->get_type()); + alloc_blk_t::create_alloc( + i->get_paddr(), + alloc_laddr, + i->get_length(), + i->get_type())); } } @@ -1455,19 +1483,57 @@ record_t Cache::prepare_record( i->state = CachedExtent::extent_state_t::CLEAN; assert(i->is_logical()); i->clear_modified_region(); - touch_extent(*i, &trans_src); + touch_extent(*i, &trans_src, t.get_cache_hint()); DEBUGT("inplace rewrite ool block is commmitted -- {}", t, *i); } + auto existing_stats = t.get_existing_block_stats(); + DEBUGT("total existing blocks num: {}, exist clean num: {}, " + "exist mutation pending num: {}", + t, + existing_stats.valid_num, + existing_stats.clean_num, + existing_stats.mutated_num); for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), + assert(is_logical_type(i->get_type())); + if (!i->is_valid()) { + continue; + } + + if (i->is_exist_clean()) { + i->state = CachedExtent::extent_state_t::CLEAN; + } else { + assert(i->is_exist_mutation_pending()); + // i->state must become DIRTY in complete_commit() + } + + // exist mutation pending extents must be in t.mutated_block_list + add_extent(i); + const auto t_src = t.get_src(); + if (i->is_dirty()) { + add_to_dirty(i, &t_src); + } else { + touch_extent(*i, &t_src, t.get_cache_hint()); + } + + alloc_delta.alloc_blk_ranges.emplace_back( + alloc_blk_t::create_alloc( + i->get_paddr(), i->cast<LogicalCachedExtent>()->get_laddr(), i->get_length(), - i->get_type()); - } + i->get_type())); + + // Note: commit extents and backref allocations in the same place + // Note: remapping is split into 2 steps, retire and alloc, they must be + // committed atomically together + backref_entries.emplace_back( + backref_entry_t::create_alloc( + i->get_paddr(), + i->cast<LogicalCachedExtent>()->get_laddr(), + i->get_length(), + i->get_type())); } + alloc_deltas.emplace_back(std::move(alloc_delta)); for (auto b : alloc_deltas) { @@ -1521,6 +1587,9 @@ record_t Cache::prepare_record( record.push_back(std::move(delta)); } + apply_backref_mset(backref_entries); + t.set_backref_entries(std::move(backref_entries)); + ceph_assert(t.get_fresh_block_stats().num == t.inline_block_list.size() + t.ool_block_list.size() + @@ -1620,26 +1689,35 @@ record_t Cache::prepare_record( return record; } -void Cache::backref_batch_update( - std::vector<backref_entry_ref> &&list, - const journal_seq_t &seq) +void Cache::apply_backref_byseq( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq) { - LOG_PREFIX(Cache::backref_batch_update); - DEBUG("inserting {} entries at {}", list.size(), seq); - ceph_assert(seq != JOURNAL_SEQ_NULL); - - for (auto &ent : list) { - backref_entry_mset.insert(*ent); + LOG_PREFIX(Cache::apply_backref_byseq); + DEBUG("backref_entry apply {} entries at {}", + backref_entries.size(), seq); + assert(seq != JOURNAL_SEQ_NULL); + if (backref_entries.empty()) { + return; } - - auto iter = backref_entryrefs_by_seq.find(seq); - if (iter == backref_entryrefs_by_seq.end()) { - backref_entryrefs_by_seq.emplace(seq, std::move(list)); + if (backref_entryrefs_by_seq.empty()) { + backref_entryrefs_by_seq.insert( + backref_entryrefs_by_seq.end(), + {seq, std::move(backref_entries)}); + return; + } + auto last = backref_entryrefs_by_seq.rbegin(); + assert(last->first <= seq); + if (last->first == seq) { + last->second.insert( + last->second.end(), + std::make_move_iterator(backref_entries.begin()), + std::make_move_iterator(backref_entries.end())); } else { - iter->second.insert( - iter->second.end(), - std::make_move_iterator(list.begin()), - std::make_move_iterator(list.end())); + assert(last->first < seq); + backref_entryrefs_by_seq.insert( + backref_entryrefs_by_seq.end(), + {seq, std::move(backref_entries)}); } } @@ -1652,7 +1730,7 @@ void Cache::complete_commit( SUBTRACET(seastore_t, "final_block_start={}, start_seq={}", t, final_block_start, start_seq); - std::vector<backref_entry_ref> backref_list; + backref_entry_refs_t backref_entries; t.for_each_finalized_fresh_block([&](const CachedExtentRef &i) { if (!i->is_valid()) { return; @@ -1681,24 +1759,30 @@ void Cache::complete_commit( add_extent(i); assert(!i->is_dirty()); const auto t_src = t.get_src(); - touch_extent(*i, &t_src); + touch_extent(*i, &t_src, t.get_cache_hint()); epm.commit_space_used(i->get_paddr(), i->get_length()); - if (is_backref_mapped_extent_node(i)) { - DEBUGT("backref_list new {} len 0x{:x}", + + // Note: commit extents and backref allocations in the same place + if (is_backref_mapped_type(i->get_type())) { + DEBUGT("backref_entry alloc {}~0x{:x}", t, i->get_paddr(), i->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + assert(i->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL); + alloc_laddr = L_ADDR_MIN; + } + backref_entries.emplace_back( + backref_entry_t::create_alloc( i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), + alloc_laddr, i->get_length(), - i->get_type(), - start_seq)); + i->get_type())); } else if (is_backref_node(i->get_type())) { add_backref_extent( i->get_paddr(), @@ -1735,9 +1819,10 @@ void Cache::complete_commit( epm.mark_space_free(extent->get_paddr(), extent->get_length()); } for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - epm.mark_space_used(i->get_paddr(), i->get_length()); + if (!i->is_valid()) { + continue; } + epm.mark_space_used(i->get_paddr(), i->get_length()); } for (auto &i: t.mutated_block_list) { @@ -1751,64 +1836,10 @@ void Cache::complete_commit( for (auto &i: t.retired_set) { auto &extent = i.extent; extent->dirty_from_or_retired_at = start_seq; - if (is_backref_mapped_extent_node(extent) || - is_retired_placeholder_type(extent->get_type())) { - DEBUGT("backref_list free {} len 0x{:x}", - t, - extent->get_paddr(), - extent->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - extent->get_paddr(), - L_ADDR_NULL, - extent->get_length(), - extent->get_type(), - start_seq)); - } else if (is_backref_node(extent->get_type())) { - remove_backref_extent(extent->get_paddr()); - } else { - ERRORT("{}", t, *extent); - ceph_abort("not possible"); - } } - auto existing_stats = t.get_existing_block_stats(); - DEBUGT("total existing blocks num: {}, exist clean num: {}, " - "exist mutation pending num: {}", - t, - existing_stats.valid_num, - existing_stats.clean_num, - existing_stats.mutated_num); - for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - if (i->is_exist_clean()) { - i->state = CachedExtent::extent_state_t::CLEAN; - } else { - assert(i->state == CachedExtent::extent_state_t::DIRTY); - } - DEBUGT("backref_list new existing {} len 0x{:x}", - t, - i->get_paddr(), - i->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - i->get_paddr(), - i->cast<LogicalCachedExtent>()->get_laddr(), - i->get_length(), - i->get_type(), - start_seq)); - add_extent(i); - const auto t_src = t.get_src(); - if (i->is_dirty()) { - add_to_dirty(i, &t_src); - } else { - touch_extent(*i, &t_src); - } - } - } - if (!backref_list.empty()) { - backref_batch_update(std::move(backref_list), start_seq); - } + apply_backref_byseq(t.move_backref_entries(), start_seq); + commit_backref_entries(std::move(backref_entries), start_seq); for (auto &i: t.pre_alloc_list) { if (!i->is_valid()) { @@ -1931,7 +1962,7 @@ Cache::replay_delta( alloc_delta_t alloc_delta; decode(alloc_delta, delta.bl); - std::vector<backref_entry_ref> backref_list; + backref_entry_refs_t backref_entries; for (auto &alloc_blk : alloc_delta.alloc_blk_ranges) { if (alloc_blk.paddr.is_relative()) { assert(alloc_blk.paddr.is_record_relative()); @@ -1939,17 +1970,10 @@ Cache::replay_delta( } DEBUG("replay alloc_blk {}~0x{:x} {}, journal_seq: {}", alloc_blk.paddr, alloc_blk.len, alloc_blk.laddr, journal_seq); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - alloc_blk.paddr, - alloc_blk.laddr, - alloc_blk.len, - alloc_blk.type, - journal_seq)); - } - if (!backref_list.empty()) { - backref_batch_update(std::move(backref_list), journal_seq); + backref_entries.emplace_back( + backref_entry_t::create(alloc_blk)); } + commit_backref_entries(std::move(backref_entries), journal_seq); return replay_delta_ertr::make_ready_future<std::pair<bool, CachedExtentRef>>( std::make_pair(true, nullptr)); } @@ -2002,7 +2026,7 @@ Cache::replay_delta( [](CachedExtent &) {}, [this](CachedExtent &ext) { // replay is not included by the cache hit metrics - touch_extent(ext, nullptr); + touch_extent(ext, nullptr, CACHE_HINT_TOUCH); }, nullptr) : _get_extent_if_cached( diff --git a/src/crimson/os/seastore/cache.h b/src/crimson/os/seastore/cache.h index 07647f6c7cf..a239b861726 100644 --- a/src/crimson/os/seastore/cache.h +++ b/src/crimson/os/seastore/cache.h @@ -3,14 +3,13 @@ #pragma once -#include <iostream> - #include "seastar/core/shared_future.hh" #include "include/buffer.h" #include "crimson/common/errorator.h" #include "crimson/common/errorator-loop.h" +#include "crimson/os/seastore/backref_entry.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/extent_placement_manager.h" #include "crimson/os/seastore/logging.h" @@ -38,86 +37,6 @@ class FixedKVBtree; class BackrefManager; class SegmentProvider; -struct backref_entry_t { - backref_entry_t( - const paddr_t paddr, - const laddr_t laddr, - const extent_len_t len, - const extent_types_t type, - const journal_seq_t seq) - : paddr(paddr), - laddr(laddr), - len(len), - type(type), - seq(seq) - {} - backref_entry_t(alloc_blk_t alloc_blk) - : paddr(alloc_blk.paddr), - laddr(alloc_blk.laddr), - len(alloc_blk.len), - type(alloc_blk.type) - {} - paddr_t paddr = P_ADDR_NULL; - laddr_t laddr = L_ADDR_NULL; - extent_len_t len = 0; - extent_types_t type = - extent_types_t::ROOT; - journal_seq_t seq; - friend bool operator< ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr < r.paddr; - } - friend bool operator> ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr > r.paddr; - } - friend bool operator== ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr == r.paddr; - } - - using set_hook_t = - boost::intrusive::set_member_hook< - boost::intrusive::link_mode< - boost::intrusive::auto_unlink>>; - set_hook_t backref_set_hook; - using backref_set_member_options = boost::intrusive::member_hook< - backref_entry_t, - set_hook_t, - &backref_entry_t::backref_set_hook>; - using multiset_t = boost::intrusive::multiset< - backref_entry_t, - backref_set_member_options, - boost::intrusive::constant_time_size<false>>; - - struct cmp_t { - using is_transparent = paddr_t; - bool operator()( - const backref_entry_t &l, - const backref_entry_t &r) const { - return l.paddr < r.paddr; - } - bool operator()(const paddr_t l, const backref_entry_t &r) const { - return l < r.paddr; - } - bool operator()(const backref_entry_t &l, const paddr_t r) const { - return l.paddr < r; - } - }; -}; - -std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent); - -using backref_entry_ref = std::unique_ptr<backref_entry_t>; -using backref_entry_mset_t = backref_entry_t::multiset_t; -using backref_entry_refs_t = std::vector<backref_entry_ref>; -using backref_entryrefs_by_seq_t = std::map<journal_seq_t, backref_entry_refs_t>; -using backref_entry_query_set_t = std::set< - backref_entry_t, backref_entry_t::cmp_t>; - /** * Cache * @@ -205,6 +124,7 @@ public: TransactionRef create_transaction( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, bool is_weak) { LOG_PREFIX(Cache::create_transaction); @@ -218,7 +138,8 @@ public: [this](Transaction& t) { return on_transaction_destruct(t); }, - ++next_id + ++next_id, + cache_hint ); SUBDEBUGT(seastore_t, "created name={}, source={}, is_weak={}", *ret, name, src, is_weak); @@ -365,7 +286,7 @@ public: SUBDEBUGT(seastore_cache, "{} {} is present in cache -- {}", t, type, offset, *ret); t.add_to_read_set(ret); - touch_extent(*ret, &t_src); + touch_extent(*ret, &t_src, t.get_cache_hint()); return ret->wait_io().then([ret] { return get_extent_if_cached_iertr::make_ready_future< CachedExtentRef>(ret); @@ -422,7 +343,7 @@ public: t, T::TYPE, offset, length); auto f = [&t, this, t_src](CachedExtent &ext) { t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent<T>( @@ -470,7 +391,7 @@ public: ++stats.access.s.load_absent; t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent<T>( @@ -568,7 +489,7 @@ public: ++access_stats.cache_lru; ++stats.access.s.cache_lru; } - touch_extent(*p_extent, &t_src); + touch_extent(*p_extent, &t_src, t.get_cache_hint()); } else { if (p_extent->is_dirty()) { ++access_stats.trans_dirty; @@ -915,7 +836,7 @@ private: t, type, offset, length, laddr); auto f = [&t, this, t_src](CachedExtent &ext) { t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent_by_type( @@ -957,7 +878,7 @@ private: ++stats.access.s.load_absent; t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent_by_type( @@ -984,7 +905,7 @@ private: for (auto it = start_iter; it != end_iter; it++) { - res.emplace(it->paddr, it->laddr, it->len, it->type, it->seq); + res.emplace(it->paddr, it->laddr, it->len, it->type); } return res; } @@ -1553,11 +1474,10 @@ private: /// Update lru for access to ref void touch_extent( CachedExtent &ext, - const Transaction::src_t* p_src) + const Transaction::src_t* p_src, + cache_hint_t hint) { - if (p_src && - is_background_transaction(*p_src) && - is_logical_type(ext.get_type())) { + if (hint == CACHE_HINT_NOCACHE && is_logical_type(ext.get_type())) { return; } if (ext.is_stable_clean() && !ext.is_placeholder()) { @@ -1907,9 +1827,23 @@ private: seastar::metrics::metric_group metrics; void register_metrics(); - void backref_batch_update( - std::vector<backref_entry_ref> &&, - const journal_seq_t &); + void apply_backref_mset( + backref_entry_refs_t& backref_entries) { + for (auto& entry : backref_entries) { + backref_entry_mset.insert(*entry); + } + } + + void apply_backref_byseq( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq); + + void commit_backref_entries( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq) { + apply_backref_mset(backref_entries); + apply_backref_byseq(std::move(backref_entries), seq); + } /// Add extent to extents handling dirty and refcounting /// diff --git a/src/crimson/os/seastore/cached_extent.cc b/src/crimson/os/seastore/cached_extent.cc index 085a519cb68..49fede1d9a8 100644 --- a/src/crimson/os/seastore/cached_extent.cc +++ b/src/crimson/os/seastore/cached_extent.cc @@ -7,6 +7,7 @@ #include "crimson/common/log.h" #include "crimson/os/seastore/btree/fixed_kv_node.h" +#include "crimson/os/seastore/lba_mapping.h" namespace { [[maybe_unused]] seastar::logger& logger() { @@ -38,12 +39,6 @@ void intrusive_ptr_release(CachedExtent *ptr) #endif -bool is_backref_mapped_extent_node(const CachedExtentRef &extent) { - return extent->is_logical() - || is_lba_node(extent->get_type()) - || extent->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL; -} - std::ostream &operator<<(std::ostream &out, CachedExtent::extent_state_t state) { switch (state) { @@ -148,6 +143,12 @@ void LogicalCachedExtent::on_replace_prior() { parent->children[off] = this; } +void LogicalCachedExtent::maybe_set_intermediate_laddr(LBAMapping &mapping) { + laddr = mapping.is_indirect() + ? mapping.get_intermediate_base() + : mapping.get_key(); +} + parent_tracker_t::~parent_tracker_t() { // this is parent's tracker, reset it auto &p = (FixedKVNode<laddr_t>&)*parent; @@ -156,32 +157,6 @@ parent_tracker_t::~parent_tracker_t() { } } -std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs) -{ - out << "LBAMapping(" << rhs.get_key() - << "~0x" << std::hex << rhs.get_length() << std::dec - << "->" << rhs.get_val(); - if (rhs.is_indirect()) { - out << ",indirect(" << rhs.get_intermediate_base() - << "~0x" << std::hex << rhs.get_intermediate_length() - << "@0x" << rhs.get_intermediate_offset() << std::dec - << ")"; - } - out << ")"; - return out; -} - -std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs) -{ - bool first = true; - out << '['; - for (const auto &i: rhs) { - out << (first ? "" : ",") << *i; - first = false; - } - return out << ']'; -} - bool BufferSpace::is_range_loaded(extent_len_t offset, extent_len_t length) const { assert(length > 0); diff --git a/src/crimson/os/seastore/cached_extent.h b/src/crimson/os/seastore/cached_extent.h index 2a54e8ded77..9dc60d719eb 100644 --- a/src/crimson/os/seastore/cached_extent.h +++ b/src/crimson/os/seastore/cached_extent.h @@ -6,6 +6,7 @@ #include <iostream> #include <boost/intrusive/list.hpp> +#include <boost/intrusive/set.hpp> #include <boost/intrusive_ptr.hpp> #include <boost/smart_ptr/intrusive_ref_counter.hpp> @@ -1096,8 +1097,6 @@ protected: std::ostream &operator<<(std::ostream &, CachedExtent::extent_state_t); std::ostream &operator<<(std::ostream &, const CachedExtent&); -bool is_backref_mapped_extent_node(const CachedExtentRef &extent); - /// Compare extents by paddr struct paddr_cmp { bool operator()(paddr_t lhs, const CachedExtent &rhs) const { @@ -1280,7 +1279,6 @@ private: }; class ChildableCachedExtent; -class LogicalCachedExtent; class child_pos_t { public: @@ -1338,48 +1336,18 @@ using PhysicalNodeMappingRef = std::unique_ptr<PhysicalNodeMapping<key_t, val_t> template <typename key_t, typename val_t> class PhysicalNodeMapping { public: + PhysicalNodeMapping() = default; + PhysicalNodeMapping(const PhysicalNodeMapping&) = delete; virtual extent_len_t get_length() const = 0; - virtual extent_types_t get_type() const = 0; virtual val_t get_val() const = 0; virtual key_t get_key() const = 0; - virtual PhysicalNodeMappingRef<key_t, val_t> duplicate() const = 0; - virtual PhysicalNodeMappingRef<key_t, val_t> refresh_with_pending_parent() { - ceph_abort("impossible"); - return {}; - } virtual bool has_been_invalidated() const = 0; virtual CachedExtentRef get_parent() const = 0; virtual uint16_t get_pos() const = 0; - // An lba pin may be indirect, see comments in lba_manager/btree/btree_lba_manager.h - virtual bool is_indirect() const { return false; } - virtual key_t get_intermediate_key() const { return min_max_t<key_t>::null; } - virtual key_t get_intermediate_base() const { return min_max_t<key_t>::null; } - virtual extent_len_t get_intermediate_length() const { return 0; } virtual uint32_t get_checksum() const { ceph_abort("impossible"); return 0; } - // The start offset of the pin, must be 0 if the pin is not indirect - virtual extent_len_t get_intermediate_offset() const { - return std::numeric_limits<extent_len_t>::max(); - } - - virtual get_child_ret_t<LogicalCachedExtent> - get_logical_extent(Transaction &t) = 0; - - void link_child(ChildableCachedExtent *c) { - ceph_assert(child_pos); - child_pos->link_child(c); - } - - // For reserved mappings, the return values are - // undefined although it won't crash - virtual bool is_stable() const = 0; - virtual bool is_data_stable() const = 0; - virtual bool is_clone() const = 0; - bool is_zero_reserved() const { - return !get_val().is_real(); - } virtual bool is_parent_viewable() const = 0; virtual bool is_parent_valid() const = 0; virtual bool parent_modified() const { @@ -1392,24 +1360,8 @@ public: } virtual ~PhysicalNodeMapping() {} -protected: - std::optional<child_pos_t> child_pos = std::nullopt; }; -using LBAMapping = PhysicalNodeMapping<laddr_t, paddr_t>; -using LBAMappingRef = PhysicalNodeMappingRef<laddr_t, paddr_t>; - -std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs); - -using lba_pin_list_t = std::list<LBAMappingRef>; - -std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs); - -using BackrefMapping = PhysicalNodeMapping<paddr_t, laddr_t>; -using BackrefMappingRef = PhysicalNodeMappingRef<paddr_t, laddr_t>; - -using backref_pin_list_t = std::list<BackrefMappingRef>; - /** * RetiredExtentPlaceholder * @@ -1523,6 +1475,8 @@ private: return out; } }; + +class LBAMapping; /** * LogicalCachedExtent * @@ -1557,11 +1511,7 @@ public: laddr = nladdr; } - void maybe_set_intermediate_laddr(LBAMapping &mapping) { - laddr = mapping.is_indirect() - ? mapping.get_intermediate_base() - : mapping.get_key(); - } + void maybe_set_intermediate_laddr(LBAMapping &mapping); void apply_delta_and_adjust_crc( paddr_t base, const ceph::bufferlist &bl) final { @@ -1661,8 +1611,6 @@ using lextent_list_t = addr_extent_list_base_t< } #if FMT_VERSION >= 90000 -template <> struct fmt::formatter<crimson::os::seastore::lba_pin_list_t> : fmt::ostream_formatter {}; template <> struct fmt::formatter<crimson::os::seastore::CachedExtent> : fmt::ostream_formatter {}; template <> struct fmt::formatter<crimson::os::seastore::LogicalCachedExtent> : fmt::ostream_formatter {}; -template <> struct fmt::formatter<crimson::os::seastore::LBAMapping> : fmt::ostream_formatter {}; #endif diff --git a/src/crimson/os/seastore/journal.h b/src/crimson/os/seastore/journal.h index a5c9029c43c..298935bd22e 100644 --- a/src/crimson/os/seastore/journal.h +++ b/src/crimson/os/seastore/journal.h @@ -59,13 +59,13 @@ public: crimson::ct_error::erange, crimson::ct_error::input_output_error >; - using submit_record_ret = submit_record_ertr::future< - record_locator_t - >; - virtual submit_record_ret submit_record( + using on_submission_func_t = std::function< + void(record_locator_t)>; + virtual submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle - ) = 0; + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) = 0; /** * flush @@ -101,9 +101,6 @@ public: virtual replay_ret replay( delta_handler_t &&delta_handler) = 0; - virtual seastar::future<> finish_commit( - transaction_type_t type) = 0; - virtual ~Journal() {} virtual backend_type_t get_type() = 0; diff --git a/src/crimson/os/seastore/journal/circular_bounded_journal.cc b/src/crimson/os/seastore/journal/circular_bounded_journal.cc index 9ee8b1b997f..41ff8318aba 100644 --- a/src/crimson/os/seastore/journal/circular_bounded_journal.cc +++ b/src/crimson/os/seastore/journal/circular_bounded_journal.cc @@ -58,35 +58,52 @@ CircularBoundedJournal::close_ertr::future<> CircularBoundedJournal::close() return record_submitter.close(); } -CircularBoundedJournal::submit_record_ret +CircularBoundedJournal::submit_record_ertr::future<> CircularBoundedJournal::submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) { LOG_PREFIX(CircularBoundedJournal::submit_record); DEBUG("H{} {} start ...", (void*)&handle, record); assert(write_pipeline); - return do_submit_record(std::move(record), handle); + return do_submit_record( + std::move(record), handle, std::move(on_submission) + ).safe_then([this, t_src] { + if (is_trim_transaction(t_src)) { + return update_journal_tail( + trimmer.get_dirty_tail(), + trimmer.get_alloc_tail()); + } else { + return seastar::now(); + } + }); } -CircularBoundedJournal::submit_record_ret +CircularBoundedJournal::submit_record_ertr::future<> CircularBoundedJournal::do_submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + on_submission_func_t &&on_submission) { LOG_PREFIX(CircularBoundedJournal::do_submit_record); if (!record_submitter.is_available()) { DEBUG("H{} wait ...", (void*)&handle); return record_submitter.wait_available( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } auto action = record_submitter.check_action(record.size); if (action == RecordSubmitter::action_t::ROLL) { return record_submitter.roll_segment( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } @@ -99,13 +116,16 @@ CircularBoundedJournal::do_submit_record( return handle.enter(write_pipeline->device_submission ).then([submit_fut=std::move(submit_ret.future)]() mutable { return std::move(submit_fut); - }).safe_then([FNAME, this, &handle](record_locator_t result) { + }).safe_then([FNAME, this, &handle, on_submission=std::move(on_submission) + ](record_locator_t result) mutable { return handle.enter(write_pipeline->finalize - ).then([FNAME, this, result, &handle] { + ).then([FNAME, this, result, &handle, + on_submission=std::move(on_submission)] { DEBUG("H{} finish with {}", (void*)&handle, result); auto new_committed_to = result.write_result.get_end_seq(); record_submitter.update_committed_to(new_committed_to); - return result; + std::invoke(on_submission, result); + return seastar::now(); }); }); } @@ -392,13 +412,4 @@ Journal::replay_ret CircularBoundedJournal::replay( }); } -seastar::future<> CircularBoundedJournal::finish_commit(transaction_type_t type) { - if (is_trim_transaction(type)) { - return update_journal_tail( - trimmer.get_dirty_tail(), - trimmer.get_alloc_tail()); - } - return seastar::now(); -} - } diff --git a/src/crimson/os/seastore/journal/circular_bounded_journal.h b/src/crimson/os/seastore/journal/circular_bounded_journal.h index 874bd8dc086..16278df6cfe 100644 --- a/src/crimson/os/seastore/journal/circular_bounded_journal.h +++ b/src/crimson/os/seastore/journal/circular_bounded_journal.h @@ -80,9 +80,11 @@ public: return backend_type_t::RANDOM_BLOCK; } - submit_record_ret submit_record( + submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission ) final; seastar::future<> flush( @@ -148,8 +150,6 @@ public: return cjs.get_records_start(); } - seastar::future<> finish_commit(transaction_type_t type) final; - using cbj_delta_handler_t = std::function< replay_ertr::future<bool>( const record_locator_t&, @@ -160,7 +160,10 @@ public: cbj_delta_handler_t &&delta_handler, journal_seq_t tail); - submit_record_ret do_submit_record(record_t &&record, OrderingHandle &handle); + submit_record_ertr::future<> do_submit_record( + record_t &&record, + OrderingHandle &handle, + on_submission_func_t &&on_submission); void try_read_rolled_header(scan_valid_records_cursor &cursor) { paddr_t addr = convert_abs_addr_to_paddr( diff --git a/src/crimson/os/seastore/journal/segmented_journal.cc b/src/crimson/os/seastore/journal/segmented_journal.cc index 6be2ad4936a..67c0b3fb8ac 100644 --- a/src/crimson/os/seastore/journal/segmented_journal.cc +++ b/src/crimson/os/seastore/journal/segmented_journal.cc @@ -368,25 +368,30 @@ seastar::future<> SegmentedJournal::flush(OrderingHandle &handle) }); } -SegmentedJournal::submit_record_ret +SegmentedJournal::submit_record_ertr::future<> SegmentedJournal::do_submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + on_submission_func_t &&on_submission) { LOG_PREFIX(SegmentedJournal::do_submit_record); if (!record_submitter.is_available()) { DEBUG("H{} wait ...", (void*)&handle); return record_submitter.wait_available( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } auto action = record_submitter.check_action(record.size); if (action == RecordSubmitter::action_t::ROLL) { DEBUG("H{} roll, unavailable ...", (void*)&handle); return record_submitter.roll_segment( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } else { // SUBMIT_FULL/NOT_FULL DEBUG("H{} submit {} ...", @@ -398,22 +403,27 @@ SegmentedJournal::do_submit_record( return handle.enter(write_pipeline->device_submission ).then([submit_fut=std::move(submit_ret.future)]() mutable { return std::move(submit_fut); - }).safe_then([FNAME, this, &handle](record_locator_t result) { + }).safe_then([FNAME, this, &handle, on_submission=std::move(on_submission) + ](record_locator_t result) mutable { return handle.enter(write_pipeline->finalize - ).then([FNAME, this, result, &handle] { + ).then([FNAME, this, result, &handle, + on_submission=std::move(on_submission)] { DEBUG("H{} finish with {}", (void*)&handle, result); auto new_committed_to = result.write_result.get_end_seq(); record_submitter.update_committed_to(new_committed_to); - return result; + std::invoke(on_submission, result); + return seastar::now(); }); }); } } -SegmentedJournal::submit_record_ret +SegmentedJournal::submit_record_ertr::future<> SegmentedJournal::submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) { LOG_PREFIX(SegmentedJournal::submit_record); DEBUG("H{} {} start ...", (void*)&handle, record); @@ -429,7 +439,8 @@ SegmentedJournal::submit_record( return crimson::ct_error::erange::make(); } - return do_submit_record(std::move(record), handle); + return do_submit_record( + std::move(record), handle, std::move(on_submission)); } } diff --git a/src/crimson/os/seastore/journal/segmented_journal.h b/src/crimson/os/seastore/journal/segmented_journal.h index 891de7ec306..3f51de70fb3 100644 --- a/src/crimson/os/seastore/journal/segmented_journal.h +++ b/src/crimson/os/seastore/journal/segmented_journal.h @@ -44,9 +44,11 @@ public: close_ertr::future<> close() final; - submit_record_ret submit_record( + submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle) final; + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) final; seastar::future<> flush(OrderingHandle &handle) final; @@ -59,9 +61,6 @@ public: backend_type_t get_type() final { return backend_type_t::SEGMENTED; } - seastar::future<> finish_commit(transaction_type_t type) { - return seastar::now(); - } bool is_checksum_needed() final { // segmented journal always requires checksum @@ -69,10 +68,10 @@ public: } private: - submit_record_ret do_submit_record( + submit_record_ertr::future<> do_submit_record( record_t &&record, - OrderingHandle &handle - ); + OrderingHandle &handle, + on_submission_func_t &&on_submission); SegmentSeqAllocatorRef segment_seq_allocator; SegmentAllocator journal_segment_allocator; diff --git a/src/crimson/os/seastore/lba_manager.h b/src/crimson/os/seastore/lba_manager.h index a050b2cdf47..9a34bf56157 100644 --- a/src/crimson/os/seastore/lba_manager.h +++ b/src/crimson/os/seastore/lba_manager.h @@ -19,6 +19,7 @@ #include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/seastore_types.h" +#include "crimson/os/seastore/lba_mapping.h" namespace crimson::os::seastore { diff --git a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc index 007737ff450..888d3c359ac 100644 --- a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc +++ b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc @@ -94,6 +94,45 @@ void unlink_phy_tree_root_node<laddr_t>(RootBlockRef &root_block) { namespace crimson::os::seastore::lba_manager::btree { +get_child_ret_t<LogicalCachedExtent> +BtreeLBAMapping::get_logical_extent(Transaction &t) +{ + ceph_assert(is_parent_viewable()); + assert(pos != std::numeric_limits<uint16_t>::max()); + ceph_assert(t.get_trans_id() == ctx.trans.get_trans_id()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + auto v = p.template get_child<LogicalCachedExtent>(ctx, pos, k); + if (!v.has_child()) { + this->child_pos = v.get_child_pos(); + } + return v; +} + +bool BtreeLBAMapping::is_stable() const +{ + assert(!this->parent_modified()); + assert(pos != std::numeric_limits<uint16_t>::max()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + return p.is_child_stable(ctx, pos, k); +} + +bool BtreeLBAMapping::is_data_stable() const +{ + assert(!this->parent_modified()); + assert(pos != std::numeric_limits<uint16_t>::max()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + return p.is_child_data_stable(ctx, pos, k); +} + BtreeLBAManager::mkfs_ret BtreeLBAManager::mkfs( Transaction &t) diff --git a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h index ef10ff9623b..e0902053d0e 100644 --- a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h +++ b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h @@ -23,11 +23,15 @@ #include "crimson/os/seastore/lba_manager/btree/lba_btree_node.h" #include "crimson/os/seastore/btree/btree_range_pin.h" +namespace crimson::os::seastore { +class LogicalCachedExtent; +} + namespace crimson::os::seastore::lba_manager::btree { struct LBALeafNode; -class BtreeLBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { +class BtreeLBAMapping : public LBAMapping { // To support cloning, there are two kinds of lba mappings: // 1. physical lba mapping: the pladdr in the value of which is the paddr of // the corresponding extent; @@ -61,14 +65,14 @@ class BtreeLBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { // their keys. public: BtreeLBAMapping(op_context_t<laddr_t> ctx) - : BtreeNodeMapping(ctx) {} + : LBAMapping(ctx) {} BtreeLBAMapping( op_context_t<laddr_t> c, LBALeafNodeRef parent, uint16_t pos, lba_map_val_t &val, lba_node_meta_t meta) - : BtreeNodeMapping( + : LBAMapping( c, parent, pos, @@ -190,8 +194,12 @@ public: SUBDEBUGT(seastore_lba, "new pin {}", ctx.trans, static_cast<LBAMapping&>(*new_pin)); return new_pin; } + bool is_stable() const final; + bool is_data_stable() const final; + get_child_ret_t<LogicalCachedExtent> get_logical_extent(Transaction &t); + protected: - std::unique_ptr<BtreeNodeMapping<laddr_t, paddr_t>> _duplicate( + LBAMappingRef _duplicate( op_context_t<laddr_t> ctx) const final { auto pin = std::unique_ptr<BtreeLBAMapping>(new BtreeLBAMapping(ctx)); pin->key = key; diff --git a/src/crimson/os/seastore/lba_mapping.cc b/src/crimson/os/seastore/lba_mapping.cc new file mode 100644 index 00000000000..90fae09ce21 --- /dev/null +++ b/src/crimson/os/seastore/lba_mapping.cc @@ -0,0 +1,44 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#include "lba_mapping.h" + +namespace crimson::os::seastore { + +std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs) +{ + out << "LBAMapping(" << rhs.get_key() + << "~0x" << std::hex << rhs.get_length() << std::dec + << "->" << rhs.get_val(); + if (rhs.is_indirect()) { + out << ",indirect(" << rhs.get_intermediate_base() + << "~0x" << std::hex << rhs.get_intermediate_length() + << "@0x" << rhs.get_intermediate_offset() << std::dec + << ")"; + } + out << ")"; + return out; +} + +std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs) +{ + bool first = true; + out << '['; + for (const auto &i: rhs) { + out << (first ? "" : ",") << *i; + first = false; + } + return out << ']'; +} + +LBAMappingRef LBAMapping::duplicate() const { + auto ret = _duplicate(ctx); + ret->range = range; + ret->value = value; + ret->parent = parent; + ret->len = len; + ret->pos = pos; + return ret; +} + +} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/lba_mapping.h b/src/crimson/os/seastore/lba_mapping.h new file mode 100644 index 00000000000..338d4d53f55 --- /dev/null +++ b/src/crimson/os/seastore/lba_mapping.h @@ -0,0 +1,73 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include "crimson/os/seastore/cached_extent.h" +#include "crimson/os/seastore/btree/btree_range_pin.h" + +namespace crimson::os::seastore { + +class LBAMapping; +using LBAMappingRef = std::unique_ptr<LBAMapping>; + +class LogicalCachedExtent; + +class LBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { +public: + LBAMapping(op_context_t<laddr_t> ctx) + : BtreeNodeMapping<laddr_t, paddr_t>(ctx) {} + template <typename... T> + LBAMapping(T&&... t) + : BtreeNodeMapping<laddr_t, paddr_t>(std::forward<T>(t)...) + { + if (!parent->is_pending()) { + this->child_pos = {parent, pos}; + } + } + + // An lba pin may be indirect, see comments in lba_manager/btree/btree_lba_manager.h + virtual bool is_indirect() const = 0; + virtual laddr_t get_intermediate_key() const = 0; + virtual laddr_t get_intermediate_base() const = 0; + virtual extent_len_t get_intermediate_length() const = 0; + // The start offset of the pin, must be 0 if the pin is not indirect + virtual extent_len_t get_intermediate_offset() const = 0; + + virtual get_child_ret_t<LogicalCachedExtent> + get_logical_extent(Transaction &t) = 0; + + void link_child(ChildableCachedExtent *c) { + ceph_assert(child_pos); + child_pos->link_child(c); + } + virtual LBAMappingRef refresh_with_pending_parent() = 0; + + // For reserved mappings, the return values are + // undefined although it won't crash + virtual bool is_stable() const = 0; + virtual bool is_data_stable() const = 0; + virtual bool is_clone() const = 0; + bool is_zero_reserved() const { + return !get_val().is_real(); + } + + LBAMappingRef duplicate() const; + + virtual ~LBAMapping() {} +protected: + virtual LBAMappingRef _duplicate(op_context_t<laddr_t>) const = 0; + std::optional<child_pos_t> child_pos = std::nullopt; +}; + +std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs); +using lba_pin_list_t = std::list<LBAMappingRef>; + +std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs); + +} // namespace crimson::os::seastore + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::os::seastore::LBAMapping> : fmt::ostream_formatter {}; +template <> struct fmt::formatter<crimson::os::seastore::lba_pin_list_t> : fmt::ostream_formatter {}; +#endif diff --git a/src/crimson/os/seastore/seastore.cc b/src/crimson/os/seastore/seastore.cc index 5b51083f344..6a866cb1f9b 100644 --- a/src/crimson/os/seastore/seastore.cc +++ b/src/crimson/os/seastore/seastore.cc @@ -408,6 +408,7 @@ SeaStore::Shard::mkfs_managers() return transaction_manager->with_transaction_intr( Transaction::src_t::MUTATE, "mkfs_seastore", + CACHE_HINT_TOUCH, [this](auto& t) { LOG_PREFIX(SeaStoreS::mkfs_managers); @@ -897,9 +898,10 @@ get_ranges(CollectionRef ch, seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> SeaStore::Shard::list_objects(CollectionRef ch, - const ghobject_t& start, - const ghobject_t& end, - uint64_t limit) const + const ghobject_t& start, + const ghobject_t& end, + uint64_t limit, + uint32_t op_flags) const { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -910,13 +912,14 @@ SeaStore::Shard::list_objects(CollectionRef ch, return seastar::do_with( RetType(std::vector<ghobject_t>(), start), std::move(limit), - [this, ch, start, end](auto& ret, auto& limit) { - return repeat_eagain([this, ch, start, end, &limit, &ret] { + [this, ch, start, end, op_flags](auto& ret, auto& limit) { + return repeat_eagain([this, ch, start, end, &limit, &ret, op_flags] { ++(shard_stats.repeat_read_num); return transaction_manager->with_transaction_intr( Transaction::src_t::READ, "list_objects", + op_flags, [this, ch, start, end, &limit, &ret](auto &t) { LOG_PREFIX(SeaStoreS::list_objects); @@ -1054,6 +1057,7 @@ SeaStore::Shard::list_collections() return transaction_manager->with_transaction_intr( Transaction::src_t::READ, "list_collections", + CACHE_HINT_TOUCH, [this, &ret](auto& t) { LOG_PREFIX(SeaStoreS::list_collections); @@ -1137,6 +1141,7 @@ SeaStore::Shard::read( Transaction::src_t::READ, "read", op_type_t::READ, + op_flags, [this, offset, len, op_flags](auto &t, auto &onode) { return _read(t, onode, offset, len, op_flags); }).finally([this] { @@ -1148,7 +1153,8 @@ SeaStore::Shard::read( SeaStore::Shard::base_errorator::future<bool> SeaStore::Shard::exists( CollectionRef c, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { LOG_PREFIX(SeaStoreS::exists); ++(shard_stats.read_num); @@ -1160,6 +1166,7 @@ SeaStore::Shard::exists( Transaction::src_t::READ, "exists", op_type_t::READ, + op_flags, [FNAME](auto& t, auto&) { DEBUGT("exists", t); return seastar::make_ready_future<bool>(true); @@ -1240,7 +1247,8 @@ SeaStore::Shard::get_attr_errorator::future<ceph::bufferlist> SeaStore::Shard::get_attr( CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1251,6 +1259,7 @@ SeaStore::Shard::get_attr( Transaction::src_t::READ, "get_attr", op_type_t::GET_ATTR, + op_flags, [this, name](auto &t, auto& onode) { return _get_attr(t, onode, name); }).handle_error( @@ -1296,7 +1305,8 @@ SeaStore::Shard::_get_attrs( SeaStore::Shard::get_attrs_ertr::future<SeaStore::Shard::attrs_t> SeaStore::Shard::get_attrs( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1307,6 +1317,7 @@ SeaStore::Shard::get_attrs( Transaction::src_t::READ, "get_attrs", op_type_t::GET_ATTRS, + op_flags, [this](auto &t, auto& onode) { return _get_attrs(t, onode); }).handle_error( @@ -1338,7 +1349,8 @@ seastar::future<struct stat> SeaStore::Shard::_stat( seastar::future<struct stat> SeaStore::Shard::stat( CollectionRef c, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1349,6 +1361,7 @@ seastar::future<struct stat> SeaStore::Shard::stat( Transaction::src_t::READ, "stat", op_type_t::STAT, + op_flags, [this, oid](auto &t, auto &onode) { return _stat(t, onode, oid); }).handle_error( @@ -1364,9 +1377,10 @@ seastar::future<struct stat> SeaStore::Shard::stat( SeaStore::Shard::get_attr_errorator::future<ceph::bufferlist> SeaStore::Shard::omap_get_header( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { - return get_attr(ch, oid, OMAP_HEADER_XATTR_KEY); + return get_attr(ch, oid, OMAP_HEADER_XATTR_KEY, op_flags); } SeaStore::base_iertr::future<SeaStore::Shard::omap_values_t> @@ -1389,7 +1403,8 @@ SeaStore::Shard::read_errorator::future<SeaStore::Shard::omap_values_t> SeaStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const omap_keys_t &keys) + const omap_keys_t &keys, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1400,6 +1415,7 @@ SeaStore::Shard::omap_get_values( Transaction::src_t::READ, "omap_get_values", op_type_t::OMAP_GET_VALUES, + op_flags, [this, keys](auto &t, auto &onode) { return do_omap_get_values(t, onode, keys); }).finally([this] { @@ -1529,7 +1545,8 @@ SeaStore::Shard::read_errorator::future<SeaStore::Shard::omap_values_paged_t> SeaStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const std::optional<std::string> &start) + const std::optional<std::string> &start, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1540,6 +1557,7 @@ SeaStore::Shard::omap_get_values( Transaction::src_t::READ, "omap_get_values2", op_type_t::OMAP_GET_VALUES2, + op_flags, [this, start](auto &t, auto &onode) { return do_omap_get_values(t, onode, start); }).finally([this] { @@ -1589,7 +1607,8 @@ SeaStore::Shard::fiemap( CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1600,6 +1619,7 @@ SeaStore::Shard::fiemap( Transaction::src_t::READ, "fiemap", op_type_t::READ, + op_flags, [this, off, len](auto &t, auto &onode) { return _fiemap(t, onode, off, len); }).finally([this] { @@ -2677,6 +2697,7 @@ seastar::future<> SeaStore::Shard::write_meta( return transaction_manager->with_transaction_intr( Transaction::src_t::MUTATE, "write_meta", + CACHE_HINT_NOCACHE, [this, &key, &value](auto& t) { LOG_PREFIX(SeaStoreS::write_meta); diff --git a/src/crimson/os/seastore/seastore.h b/src/crimson/os/seastore/seastore.h index fd7e177da63..e2a993b9e20 100644 --- a/src/crimson/os/seastore/seastore.h +++ b/src/crimson/os/seastore/seastore.h @@ -101,7 +101,8 @@ public: seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<ceph::bufferlist> read( CollectionRef c, @@ -118,32 +119,38 @@ public: base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const final; + std::string_view name, + uint32_t op_flags = 0) const final; get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; /// Retrieves paged set of values > start (if present) read_errorator::future<omap_values_paged_t> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; ///< @return <done, values> values.empty() iff done get_attr_errorator::future<bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; /// std::get<1>(ret) returns end if and only if the listing has listed all /// the items within the range, otherwise it returns the next key to be listed. @@ -151,7 +158,8 @@ public: CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; seastar::future<CollectionRef> open_collection(const coll_t& cid) final; @@ -170,7 +178,8 @@ public: CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags = 0) final; unsigned get_max_attr_name_length() const final { return 256; @@ -251,7 +260,8 @@ public: return seastar::do_with( internal_context_t( ch, std::move(t), - transaction_manager->create_transaction(src, tname)), + transaction_manager->create_transaction( + src, tname, t.get_fadvise_flags())), std::forward<F>(f), [this, op_type](auto &ctx, auto &f) { assert(shard_stats.starting_io_num); @@ -298,20 +308,22 @@ public: Transaction::src_t src, const char* tname, op_type_t op_type, + cache_hint_t cache_hint_flags, F &&f) const { auto begin_time = std::chrono::steady_clock::now(); return seastar::do_with( oid, Ret{}, std::forward<F>(f), - [this, ch, src, op_type, begin_time, tname + [this, ch, src, op_type, begin_time, tname, cache_hint_flags ](auto &oid, auto &ret, auto &f) { - return repeat_eagain([&, this, ch, src, tname] { + return repeat_eagain([&, this, ch, src, tname, cache_hint_flags] { assert(src == Transaction::src_t::READ); ++(shard_stats.repeat_read_num); return transaction_manager->with_transaction_intr( src, tname, + cache_hint_flags, [&, this, ch, tname](auto& t) { LOG_PREFIX(SeaStoreS::repeat_with_onode); diff --git a/src/crimson/os/seastore/seastore_types.h b/src/crimson/os/seastore/seastore_types.h index 483ac995ee3..5930469ca07 100644 --- a/src/crimson/os/seastore/seastore_types.h +++ b/src/crimson/os/seastore/seastore_types.h @@ -3,6 +3,7 @@ #pragma once +#include <deque> #include <limits> #include <numeric> #include <optional> @@ -14,13 +15,47 @@ #include "include/byteorder.h" #include "include/denc.h" +#include "include/encoding.h" #include "include/buffer.h" #include "include/intarith.h" #include "include/interval_set.h" #include "include/uuid.h" +#include "include/rados.h" namespace crimson::os::seastore { +class cache_hint_t { + enum hint_t { + TOUCH, + NOCACHE + }; +public: + static constexpr cache_hint_t get_touch() { + return hint_t::TOUCH; + } + static constexpr cache_hint_t get_nocache() { + return hint_t::NOCACHE; + } + cache_hint_t(uint32_t flags) { + if (unlikely(flags & CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) || + unlikely(flags & CEPH_OSD_OP_FLAG_FADVISE_NOCACHE)) { + hint = NOCACHE; + } + } + bool operator==(const cache_hint_t &other) const { + return hint == other.hint; + } + bool operator!=(const cache_hint_t &other) const { + return hint != other.hint; + } +private: + constexpr cache_hint_t(hint_t hint) : hint(hint) {} + hint_t hint = hint_t::TOUCH; +}; + +inline constexpr cache_hint_t CACHE_HINT_TOUCH = cache_hint_t::get_touch(); +inline constexpr cache_hint_t CACHE_HINT_NOCACHE = cache_hint_t::get_nocache(); + /* using a special xattr key "omap_header" to store omap header */ const std::string OMAP_HEADER_XATTR_KEY = "omap_header"; @@ -1226,7 +1261,6 @@ constexpr laddr_t L_ADDR_MAX = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX); constexpr laddr_t L_ADDR_MIN = laddr_t::from_raw_uint(0); constexpr laddr_t L_ADDR_NULL = L_ADDR_MAX; constexpr laddr_t L_ADDR_ROOT = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX - 1); -constexpr laddr_t L_ADDR_LBAT = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX - 2); struct __attribute__((packed)) laddr_le_t { ceph_le64 laddr; @@ -1467,6 +1501,23 @@ constexpr bool is_physical_type(extent_types_t type) { } } +constexpr bool is_backref_mapped_type(extent_types_t type) { + if ((type >= extent_types_t::LADDR_INTERNAL && + type <= extent_types_t::OBJECT_DATA_BLOCK) || + type == extent_types_t::TEST_BLOCK || + type == extent_types_t::TEST_BLOCK_PHYSICAL) { + assert(is_logical_type(type) || + is_lba_node(type) || + type == extent_types_t::TEST_BLOCK_PHYSICAL); + return true; + } else { + assert(!is_logical_type(type) && + !is_lba_node(type) && + type != extent_types_t::TEST_BLOCK_PHYSICAL); + return false; + } +} + constexpr bool is_real_type(extent_types_t type) { if (type <= extent_types_t::OBJECT_DATA_BLOCK || (type >= extent_types_t::TEST_BLOCK && @@ -1943,12 +1994,13 @@ struct __attribute__((packed)) root_t { struct alloc_blk_t { alloc_blk_t( - paddr_t paddr, - laddr_t laddr, + const paddr_t& paddr, + const laddr_t& laddr, extent_len_t len, extent_types_t type) - : paddr(paddr), laddr(laddr), len(len), type(type) - {} + : paddr(paddr), laddr(laddr), len(len), type(type) { + assert(len > 0); + } explicit alloc_blk_t() = default; @@ -1964,6 +2016,25 @@ struct alloc_blk_t { denc(v.type, p); DENC_FINISH(p); } + + static alloc_blk_t create_alloc( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type)); + assert(laddr != L_ADDR_NULL); + return alloc_blk_t(paddr, laddr, len, type); + } + + static alloc_blk_t create_retire( + const paddr_t& paddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type) || + is_retired_placeholder_type(type)); + return alloc_blk_t(paddr, L_ADDR_NULL, len, type); + } }; // use absolute address diff --git a/src/crimson/os/seastore/transaction.h b/src/crimson/os/seastore/transaction.h index 9b95161a404..cd8c333c69f 100644 --- a/src/crimson/os/seastore/transaction.h +++ b/src/crimson/os/seastore/transaction.h @@ -8,11 +8,12 @@ #include <boost/intrusive/list.hpp> #include "crimson/common/log.h" +#include "crimson/os/seastore/backref_entry.h" +#include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/logging.h" #include "crimson/os/seastore/ordering_handle.h" -#include "crimson/os/seastore/seastore_types.h" -#include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/root_block.h" +#include "crimson/os/seastore/seastore_types.h" #include "crimson/os/seastore/transaction_interruptor.h" namespace crimson::os::seastore { @@ -408,12 +409,14 @@ public: src_t src, journal_seq_t initiated_after, on_destruct_func_t&& f, - transaction_id_t trans_id + transaction_id_t trans_id, + cache_hint_t cache_hint ) : weak(weak), handle(std::move(handle)), on_destruct(std::move(f)), src(src), - trans_id(trans_id) + trans_id(trans_id), + cache_hint(cache_hint) {} void invalidate_clear_write_set() { @@ -460,6 +463,7 @@ public: ool_write_stats = {}; rewrite_stats = {}; conflicted = false; + assert(backref_entries.empty()); if (!has_reset) { has_reset = true; } @@ -571,10 +575,23 @@ public: return pre_alloc_list; } + cache_hint_t get_cache_hint() const { + return cache_hint; + } + private: friend class Cache; friend Ref make_test_transaction(); + void set_backref_entries(backref_entry_refs_t&& entries) { + assert(backref_entries.empty()); + backref_entries = std::move(entries); + } + + backref_entry_refs_t move_backref_entries() { + return std::move(backref_entries); + } + /** * If set, *this may not be used to perform writes and will not provide * consistentency allowing operations using to avoid maintaining a read_set. @@ -669,6 +686,10 @@ private: transaction_id_t trans_id = TRANS_ID_NULL; seastar::lw_shared_ptr<rbm_pending_ool_t> pending_ool; + + backref_entry_refs_t backref_entries; + + cache_hint_t cache_hint = CACHE_HINT_TOUCH; }; using TransactionRef = Transaction::Ref; @@ -681,7 +702,8 @@ inline TransactionRef make_test_transaction() { Transaction::src_t::MUTATE, JOURNAL_SEQ_NULL, [](Transaction&) {}, - ++next_id + ++next_id, + CACHE_HINT_TOUCH ); } diff --git a/src/crimson/os/seastore/transaction_manager.cc b/src/crimson/os/seastore/transaction_manager.cc index 94e9b3b9ab1..807d88b2cbc 100644 --- a/src/crimson/os/seastore/transaction_manager.cc +++ b/src/crimson/os/seastore/transaction_manager.cc @@ -66,6 +66,7 @@ TransactionManager::mkfs_ertr::future<> TransactionManager::mkfs() return with_transaction_intr( Transaction::src_t::MUTATE, "mkfs_tm", + CACHE_HINT_TOUCH, [this, FNAME](auto& t) { cache->init(); @@ -131,6 +132,7 @@ TransactionManager::mount() journal->get_trimmer().set_journal_head(start_seq); return with_transaction_weak( "mount", + CACHE_HINT_TOUCH, [this](auto &t) { return cache->init_cached_extents(t, [this](auto &t, auto &e) { @@ -461,8 +463,12 @@ TransactionManager::do_submit_transaction( } SUBTRACET(seastore_t, "submitting record", tref); - return journal->submit_record(std::move(record), tref.get_handle() - ).safe_then([this, FNAME, &tref](auto submit_result) mutable { + return journal->submit_record( + std::move(record), + tref.get_handle(), + tref.get_src(), + [this, FNAME, &tref](record_locator_t submit_result) + { SUBDEBUGT(seastore_t, "committed with {}", tref, submit_result); auto start_seq = submit_result.write_result.start_seq; journal->get_trimmer().set_journal_head(start_seq); @@ -473,10 +479,8 @@ TransactionManager::do_submit_transaction( journal->get_trimmer().update_journal_tails( cache->get_oldest_dirty_from().value_or(start_seq), cache->get_oldest_backref_dirty_from().value_or(start_seq)); - return journal->finish_commit(tref.get_src() - ).then([&tref] { - return tref.get_handle().complete(); - }); + }).safe_then([&tref] { + return tref.get_handle().complete(); }).handle_error( submit_transaction_iertr::pass_further{}, crimson::ct_error::assert_all{"Hit error submitting to journal"} diff --git a/src/crimson/os/seastore/transaction_manager.h b/src/crimson/os/seastore/transaction_manager.h index dc6cc20cf59..e574460894a 100644 --- a/src/crimson/os/seastore/transaction_manager.h +++ b/src/crimson/os/seastore/transaction_manager.h @@ -741,8 +741,9 @@ public: TransactionRef create_transaction( Transaction::src_t src, const char* name, + cache_hint_t cache_hint = CACHE_HINT_TOUCH, bool is_weak=false) final { - return cache->create_transaction(src, name, is_weak); + return cache->create_transaction(src, name, cache_hint, is_weak); } using ExtentCallbackInterface::submit_transaction_direct_ret; diff --git a/src/crimson/osd/backfill_facades.h b/src/crimson/osd/backfill_facades.h index 64544d4c870..ce649303d4f 100644 --- a/src/crimson/osd/backfill_facades.h +++ b/src/crimson/osd/backfill_facades.h @@ -82,6 +82,9 @@ struct PGFacade final : BackfillState::PGFacade { } PGFacade(PG& pg) : pg(pg) {} + std::ostream &print(std::ostream &out) const override { + return out << pg; + } }; } // namespace crimson::osd diff --git a/src/crimson/osd/backfill_state.cc b/src/crimson/osd/backfill_state.cc index 837fd2eb2af..f957f072c93 100644 --- a/src/crimson/osd/backfill_state.cc +++ b/src/crimson/osd/backfill_state.cc @@ -8,11 +8,7 @@ #include "crimson/osd/backfill_state.h" #include "osd/osd_types_fmt.h" -namespace { - seastar::logger& logger() { - return crimson::get_logger(ceph_subsys_osd); - } -} +SET_SUBSYS(osd); namespace crimson::osd { @@ -27,22 +23,23 @@ BackfillState::BackfillState( progress_tracker( std::make_unique<BackfillState::ProgressTracker>(backfill_machine)) { - logger().debug("{}:{}", __func__, __LINE__); + LOG_PREFIX(BackfillState::BackfillState); + DEBUGDPP("", *backfill_machine.pg); backfill_machine.initiate(); } template <class S> BackfillState::StateHelper<S>::StateHelper() { - logger().debug("enter {}", - boost::typeindex::type_id<S>().pretty_name()); + LOG_PREFIX(BackfillState::StateHelper); + DEBUGDPP("enter {}", pg(), boost::typeindex::type_id<S>().pretty_name()); } template <class S> BackfillState::StateHelper<S>::~StateHelper() { - logger().debug("exit {}", - boost::typeindex::type_id<S>().pretty_name()); + LOG_PREFIX(BackfillState::StateHelper); + DEBUG("exit {}", boost::typeindex::type_id<S>().pretty_name()); } BackfillState::~BackfillState() = default; @@ -63,13 +60,16 @@ BackfillState::BackfillMachine::~BackfillMachine() = default; BackfillState::Initial::Initial(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::Initial::Initial); backfill_state().last_backfill_started = peering_state().earliest_backfill(); - logger().debug("{}: bft={} from {}", - __func__, peering_state().get_backfill_targets(), - backfill_state().last_backfill_started); + DEBUGDPP("{}: bft={} from {}", + pg(), + __func__, + peering_state().get_backfill_targets(), + backfill_state().last_backfill_started); for (const auto& bt : peering_state().get_backfill_targets()) { - logger().debug("{}: target shard {} from {}", - __func__, bt, peering_state().get_peer_last_backfill(bt)); + DEBUGDPP("{}: target shard {} from {}", + pg(), __func__, bt, peering_state().get_peer_last_backfill(bt)); } ceph_assert(peering_state().get_backfill_targets().size()); ceph_assert(!backfill_state().last_backfill_started.is_max()); @@ -80,7 +80,8 @@ BackfillState::Initial::Initial(my_context ctx) boost::statechart::result BackfillState::Initial::react(const BackfillState::Triggered& evt) { - logger().debug("{}: backfill triggered", __func__); + LOG_PREFIX(BackfillState::Initial::react::Triggered); + DEBUGDPP("", pg()); ceph_assert(backfill_state().last_backfill_started == \ peering_state().earliest_backfill()); ceph_assert(peering_state().is_backfilling()); @@ -93,26 +94,10 @@ BackfillState::Initial::react(const BackfillState::Triggered& evt) if (Enqueuing::all_enqueued(peering_state(), backfill_state().backfill_info, backfill_state().peer_backfill_info)) { - logger().debug("{}: switching to Done state", __func__); + DEBUGDPP("switching to Done state", pg()); return transit<BackfillState::Done>(); } else { - logger().debug("{}: switching to Enqueuing state", __func__); - return transit<BackfillState::Enqueuing>(); - } -} - -boost::statechart::result -BackfillState::Cancelled::react(const BackfillState::Triggered& evt) -{ - logger().debug("{}: backfill re-triggered", __func__); - ceph_assert(peering_state().is_backfilling()); - if (Enqueuing::all_enqueued(peering_state(), - backfill_state().backfill_info, - backfill_state().peer_backfill_info)) { - logger().debug("{}: switching to Done state", __func__); - return transit<BackfillState::Done>(); - } else { - logger().debug("{}: switching to Enqueuing state", __func__); + DEBUGDPP("switching to Enqueuing state", pg()); return transit<BackfillState::Enqueuing>(); } } @@ -120,9 +105,10 @@ BackfillState::Cancelled::react(const BackfillState::Triggered& evt) // -- Enqueuing void BackfillState::Enqueuing::maybe_update_range() { + LOG_PREFIX(BackfillState::Enqueuing::maybe_update_range); if (auto& primary_bi = backfill_state().backfill_info; primary_bi.version >= pg().get_projected_last_update()) { - logger().info("{}: bi is current", __func__); + INFODPP("bi is current", pg()); ceph_assert(primary_bi.version == pg().get_projected_last_update()); } else if (primary_bi.version >= peering_state().get_log_tail()) { if (peering_state().get_pg_log().get_log().empty() && @@ -136,31 +122,31 @@ void BackfillState::Enqueuing::maybe_update_range() ceph_assert(primary_bi.version == eversion_t()); return; } - logger().debug("{}: bi is old, ({}) can be updated with log to {}", - __func__, - primary_bi.version, - pg().get_projected_last_update()); + DEBUGDPP("{}: bi is old, ({}) can be updated with log to {}", + pg(), + primary_bi.version, + pg().get_projected_last_update()); auto func = [&](const pg_log_entry_t& e) { - logger().debug("maybe_update_range(lambda): updating from version {}", - e.version); + DEBUGDPP("maybe_update_range(lambda): updating from version {}", + pg(), e.version); if (e.soid >= primary_bi.begin && e.soid < primary_bi.end) { if (e.is_update()) { - logger().debug("maybe_update_range(lambda): {} updated to ver {}", - e.soid, e.version); + DEBUGDPP("maybe_update_range(lambda): {} updated to ver {}", + pg(), e.soid, e.version); primary_bi.objects.erase(e.soid); primary_bi.objects.insert(std::make_pair(e.soid, e.version)); } else if (e.is_delete()) { - logger().debug("maybe_update_range(lambda): {} removed", - e.soid); + DEBUGDPP("maybe_update_range(lambda): {} removed", + pg(), e.soid); primary_bi.objects.erase(e.soid); } } }; - logger().debug("{}: scanning pg log first", __func__); + DEBUGDPP("{}: scanning pg log first", pg()); peering_state().scan_log_after(primary_bi.version, func); - logger().debug("{}: scanning projected log", __func__); + DEBUGDPP("{}: scanning projected log", pg()); pg().get_projected_log().scan_log_after(primary_bi.version, func); primary_bi.version = pg().get_projected_last_update(); } else { @@ -244,6 +230,7 @@ void BackfillState::Enqueuing::trim_backfilled_object_from_intervals( BackfillState::Enqueuing::result_t BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) { + LOG_PREFIX(BackfillState::Enqueuing::remove_on_peers); // set `new_last_backfill_started` to `check` result_t result { {}, check }; for (const auto& bt : peering_state().get_backfill_targets()) { @@ -255,8 +242,8 @@ BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) backfill_listener().enqueue_drop(bt, pbi.begin, version); } } - logger().debug("{}: BACKFILL removing {} from peers {}", - __func__, check, result.pbi_targets); + DEBUGDPP("BACKFILL removing {} from peers {}", + pg(), check, result.pbi_targets); ceph_assert(!result.pbi_targets.empty()); return result; } @@ -264,7 +251,8 @@ BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) BackfillState::Enqueuing::result_t BackfillState::Enqueuing::update_on_peers(const hobject_t& check) { - logger().debug("{}: check={}", __func__, check); + LOG_PREFIX(BackfillState::Enqueuing::update_on_peers); + DEBUGDPP("check={}", pg(), check); const auto& primary_bi = backfill_state().backfill_info; result_t result { {}, primary_bi.begin }; std::map<hobject_t, std::pair<eversion_t, std::vector<pg_shard_t>>> backfills; @@ -325,6 +313,7 @@ bool BackfillState::Enqueuing::Enqueuing::all_emptied( BackfillState::Enqueuing::Enqueuing(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::Enqueuing::Enqueuing); auto& primary_bi = backfill_state().backfill_info; // update our local interval to cope with recent changes @@ -334,8 +323,7 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) // that backfill will be spinning here over and over. For the sake // of performance and complexity we don't synchronize with entire PG. // similar can happen in classical OSD. - logger().warn("{}: bi is old, rescanning of local backfill_info", - __func__); + WARNDPP("bi is old, rescanning of local backfill_info", pg()); post_event(RequestPrimaryScanning{}); return; } else { @@ -347,13 +335,14 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) primary_bi)) { // need to grab one another chunk of the object namespace and restart // the queueing. - logger().debug("{}: reached end for current local chunk", __func__); + DEBUGDPP("reached end for current local chunk", pg()); post_event(RequestPrimaryScanning{}); return; } do { if (!backfill_listener().budget_available()) { + DEBUGDPP("throttle failed, turning to Waiting", pg()); post_event(RequestWaiting{}); return; } else if (should_rescan_replicas(backfill_state().peer_backfill_info, @@ -392,16 +381,25 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) } } while (!all_emptied(primary_bi, backfill_state().peer_backfill_info)); - if (backfill_state().progress_tracker->tracked_objects_completed() - && Enqueuing::all_enqueued(peering_state(), - backfill_state().backfill_info, - backfill_state().peer_backfill_info)) { - backfill_state().last_backfill_started = hobject_t::get_max(); - backfill_listener().update_peers_last_backfill(hobject_t::get_max()); + if (should_rescan_primary(backfill_state().peer_backfill_info, + primary_bi)) { + // need to grab one another chunk of the object namespace and restart + // the queueing. + DEBUGDPP("reached end for current local chunk", pg()); + post_event(RequestPrimaryScanning{}); + return; + } else { + if (backfill_state().progress_tracker->tracked_objects_completed() + && Enqueuing::all_enqueued(peering_state(), + backfill_state().backfill_info, + backfill_state().peer_backfill_info)) { + backfill_state().last_backfill_started = hobject_t::get_max(); + backfill_listener().update_peers_last_backfill(hobject_t::get_max()); + } + DEBUGDPP("reached end for both local and all peers " + "but still has in-flight operations", pg()); + post_event(RequestWaiting{}); } - logger().debug("{}: reached end for both local and all peers " - "but still has in-flight operations", __func__); - post_event(RequestWaiting{}); } // -- PrimaryScanning @@ -416,16 +414,45 @@ BackfillState::PrimaryScanning::PrimaryScanning(my_context ctx) boost::statechart::result BackfillState::PrimaryScanning::react(PrimaryScanned evt) { - logger().debug("{}", __func__); + LOG_PREFIX(BackfillState::PrimaryScanning::react::PrimaryScanned); + DEBUGDPP("", pg()); backfill_state().backfill_info = std::move(evt.result); - return transit<Enqueuing>(); + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } + return discard_event(); +} + +boost::statechart::result +BackfillState::PrimaryScanning::react(CancelBackfill evt) +{ + LOG_PREFIX(BackfillState::PrimaryScanning::react::SuspendBackfill); + DEBUGDPP("suspended within PrimaryScanning", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::PrimaryScanning::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::PrimaryScanning::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } boost::statechart::result BackfillState::PrimaryScanning::react(ObjectPushed evt) { - logger().debug("PrimaryScanning::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::PrimaryScanning::react::ObjectPushed); + DEBUGDPP("PrimaryScanning::react() on ObjectPushed; evt.object={}", + pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, true); return discard_event(); } @@ -443,11 +470,11 @@ bool BackfillState::ReplicasScanning::replica_needs_scan( BackfillState::ReplicasScanning::ReplicasScanning(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::ReplicasScanning::ReplicasScanning); for (const auto& bt : peering_state().get_backfill_targets()) { if (const auto& pbi = backfill_state().peer_backfill_info.at(bt); replica_needs_scan(pbi, backfill_state().backfill_info)) { - logger().debug("{}: scanning peer osd.{} from {}", - __func__, bt, pbi.end); + DEBUGDPP("scanning peer osd.{} from {}", pg(), bt, pbi.end); backfill_listener().request_replica_scan(bt, pbi.end, hobject_t{}); ceph_assert(waiting_on_backfill.find(bt) == \ @@ -469,8 +496,9 @@ BackfillState::ReplicasScanning::~ReplicasScanning() boost::statechart::result BackfillState::ReplicasScanning::react(ReplicaScanned evt) { - logger().debug("{}: got scan result from osd={}, result={}", - __func__, evt.from, evt.result); + LOG_PREFIX(BackfillState::ReplicasScanning::react::ReplicaScanned); + DEBUGDPP("got scan result from osd={}, result={}", + pg(), evt.from, evt.result); // TODO: maybe we'll be able to move waiting_on_backfill from // the machine to the state. ceph_assert(peering_state().is_backfill_target(evt.from)); @@ -479,12 +507,17 @@ BackfillState::ReplicasScanning::react(ReplicaScanned evt) if (waiting_on_backfill.empty()) { ceph_assert(backfill_state().peer_backfill_info.size() == \ peering_state().get_backfill_targets().size()); - return transit<Enqueuing>(); + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } } } else { - // we canceled backfill for a while due to a too full, and this + // we suspended backfill for a while due to a too full, and this // is an extra response from a non-too-full peer - logger().debug("{}: canceled backfill (too full?)", __func__); + DEBUGDPP("suspended backfill (too full?)", pg()); } return discard_event(); } @@ -492,17 +525,30 @@ BackfillState::ReplicasScanning::react(ReplicaScanned evt) boost::statechart::result BackfillState::ReplicasScanning::react(CancelBackfill evt) { - logger().debug("{}: cancelled within ReplicasScanning", - __func__); - waiting_on_backfill.clear(); - return transit<Cancelled>(); + LOG_PREFIX(BackfillState::ReplicasScanning::react::SuspendBackfill); + DEBUGDPP("suspended within ReplicasScanning", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::ReplicasScanning::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::ReplicasScanning::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } boost::statechart::result BackfillState::ReplicasScanning::react(ObjectPushed evt) { - logger().debug("ReplicasScanning::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::ReplicasScanning::react::ObjectPushed); + DEBUGDPP("ReplicasScanning::react() on ObjectPushed; evt.object={}", + pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, true); return discard_event(); } @@ -517,17 +563,45 @@ BackfillState::Waiting::Waiting(my_context ctx) boost::statechart::result BackfillState::Waiting::react(ObjectPushed evt) { - logger().debug("Waiting::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::Waiting::react::ObjectPushed); + DEBUGDPP("Waiting::react() on ObjectPushed; evt.object={}", pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, false); - return transit<Enqueuing>();; + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } + return discard_event(); +} + +boost::statechart::result +BackfillState::Waiting::react(CancelBackfill evt) +{ + LOG_PREFIX(BackfillState::Waiting::react::SuspendBackfill); + DEBUGDPP("suspended within Waiting", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::Waiting::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::Waiting::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } // -- Done BackfillState::Done::Done(my_context ctx) : my_base(ctx) { - logger().info("{}: backfill is done", __func__); + LOG_PREFIX(BackfillState::Done::Done); + INFODPP("backfill is done", pg()); backfill_listener().backfilled(); } @@ -537,13 +611,6 @@ BackfillState::Crashed::Crashed() ceph_abort_msg("{}: this should not happen"); } -// -- Cancelled -BackfillState::Cancelled::Cancelled(my_context ctx) - : my_base(ctx) -{ - ceph_assert(peering_state().get_backfill_targets().size()); -} - // ProgressTracker is an intermediary between the BackfillListener and // BackfillMachine + its states. All requests to push or drop an object // are directed through it. The same happens with notifications about @@ -577,8 +644,8 @@ void BackfillState::ProgressTracker::complete_to( const pg_stat_t& stats, bool may_push_to_max) { - logger().debug("{}: obj={}", - __func__, obj); + LOG_PREFIX(BackfillState::ProgressTracker::complete_to); + DEBUGDPP("obj={}", pg(), obj); if (auto completion_iter = registry.find(obj); completion_iter != std::end(registry)) { completion_iter->second = \ @@ -619,4 +686,19 @@ void BackfillState::enqueue_standalone_push( backfill_machine.backfill_listener.enqueue_push(obj, v, peers); } +void BackfillState::enqueue_standalone_delete( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers) +{ + progress_tracker->enqueue_drop(obj); + for (auto bt : peers) { + backfill_machine.backfill_listener.enqueue_drop(bt, obj, v); + } +} + +std::ostream &operator<<(std::ostream &out, const BackfillState::PGFacade &pg) { + return pg.print(out); +} + } // namespace crimson::osd diff --git a/src/crimson/osd/backfill_state.h b/src/crimson/osd/backfill_state.h index 072c91e079d..517a02ea4df 100644 --- a/src/crimson/osd/backfill_state.h +++ b/src/crimson/osd/backfill_state.h @@ -62,6 +62,8 @@ struct BackfillState { struct CancelBackfill : sc::event<CancelBackfill> { }; + struct ThrottleAcquired : sc::event<ThrottleAcquired> { + }; private: // internal events struct RequestPrimaryScanning : sc::event<RequestPrimaryScanning> { @@ -136,34 +138,10 @@ public: explicit Crashed(); }; - struct Cancelled : sc::state<Cancelled, BackfillMachine>, - StateHelper<Cancelled> { - using reactions = boost::mpl::list< - sc::custom_reaction<Triggered>, - sc::custom_reaction<PrimaryScanned>, - sc::custom_reaction<ReplicaScanned>, - sc::custom_reaction<ObjectPushed>, - sc::transition<sc::event_base, Crashed>>; - explicit Cancelled(my_context); - // resume after triggering backfill by on_activate_complete(). - // transit to Enqueuing. - sc::result react(const Triggered&); - sc::result react(const PrimaryScanned&) { - return discard_event(); - } - sc::result react(const ReplicaScanned&) { - return discard_event(); - } - sc::result react(const ObjectPushed&) { - return discard_event(); - } - }; - struct Initial : sc::state<Initial, BackfillMachine>, StateHelper<Initial> { using reactions = boost::mpl::list< sc::custom_reaction<Triggered>, - sc::transition<CancelBackfill, Cancelled>, sc::transition<sc::event_base, Crashed>>; explicit Initial(my_context); // initialize after triggering backfill by on_activate_complete(). @@ -174,12 +152,9 @@ public: struct Enqueuing : sc::state<Enqueuing, BackfillMachine>, StateHelper<Enqueuing> { using reactions = boost::mpl::list< - sc::transition<CancelBackfill, Cancelled>, sc::transition<RequestPrimaryScanning, PrimaryScanning>, sc::transition<RequestReplicasScanning, ReplicasScanning>, sc::transition<RequestWaiting, Waiting>, - sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, sc::transition<sc::event_base, Crashed>>; explicit Enqueuing(my_context); @@ -237,12 +212,15 @@ public: sc::custom_reaction<ObjectPushed>, sc::custom_reaction<PrimaryScanned>, sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, + sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, sc::transition<sc::event_base, Crashed>>; explicit PrimaryScanning(my_context); sc::result react(ObjectPushed); // collect scanning result and transit to Enqueuing. sc::result react(PrimaryScanned); + sc::result react(CancelBackfill); + sc::result react(Triggered); }; struct ReplicasScanning : sc::state<ReplicasScanning, BackfillMachine>, @@ -251,6 +229,7 @@ public: sc::custom_reaction<ObjectPushed>, sc::custom_reaction<ReplicaScanned>, sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, sc::transition<RequestDone, Done>, sc::transition<sc::event_base, Crashed>>; explicit ReplicasScanning(my_context); @@ -259,6 +238,7 @@ public: sc::result react(ObjectPushed); sc::result react(ReplicaScanned); sc::result react(CancelBackfill); + sc::result react(Triggered); // indicate whether a particular peer should be scanned to retrieve // BackfillInterval for new range of hobject_t namespace. @@ -277,10 +257,14 @@ public: using reactions = boost::mpl::list< sc::custom_reaction<ObjectPushed>, sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, + sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, + sc::transition<ThrottleAcquired, Enqueuing>, sc::transition<sc::event_base, Crashed>>; explicit Waiting(my_context); sc::result react(ObjectPushed); + sc::result react(CancelBackfill); + sc::result react(Triggered); }; struct Done : sc::state<Done, BackfillMachine>, @@ -308,6 +292,11 @@ public: const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers); + void enqueue_standalone_delete( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers); + bool is_triggered() const { return backfill_machine.triggering_event() != nullptr; @@ -325,6 +314,26 @@ public: } } private: + struct backfill_suspend_state_t { + bool suspended = false; + bool should_go_enqueuing = false; + } backfill_suspend_state; + bool is_suspended() const { + return backfill_suspend_state.suspended; + } + void on_suspended() { + ceph_assert(!is_suspended()); + backfill_suspend_state = {true, false}; + } + bool on_resumed() { + auto go_enqueuing = backfill_suspend_state.should_go_enqueuing; + backfill_suspend_state = {false, false}; + return go_enqueuing; + } + void go_enqueuing_on_resume() { + ceph_assert(is_suspended()); + backfill_suspend_state.should_go_enqueuing = true; + } hobject_t last_backfill_started; BackfillInterval backfill_info; std::map<pg_shard_t, BackfillInterval> peer_backfill_info; @@ -405,8 +414,10 @@ struct BackfillState::PGFacade { virtual const eversion_t& get_projected_last_update() const = 0; virtual const PGLog::IndexedLog& get_projected_log() const = 0; + virtual std::ostream &print(std::ostream &out) const = 0; virtual ~PGFacade() {} }; +std::ostream &operator<<(std::ostream &out, const BackfillState::PGFacade &pg); class BackfillState::ProgressTracker { // TODO: apply_stat, @@ -433,6 +444,9 @@ class BackfillState::ProgressTracker { BackfillListener& backfill_listener() { return backfill_machine.backfill_listener; } + PGFacade& pg() { + return *backfill_machine.pg; + } public: ProgressTracker(BackfillMachine& backfill_machine) @@ -447,3 +461,9 @@ public: }; } // namespace crimson::osd + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::osd::BackfillState::PGFacade> + : fmt::ostream_formatter {}; +#endif + diff --git a/src/crimson/osd/heartbeat.cc b/src/crimson/osd/heartbeat.cc index 03986952b4f..5902fc8c14f 100644 --- a/src/crimson/osd/heartbeat.cc +++ b/src/crimson/osd/heartbeat.cc @@ -9,6 +9,7 @@ #include "messages/MOSDPing.h" #include "messages/MOSDFailure.h" +#include "msg/msg_types.h" #include "crimson/common/config_proxy.h" #include "crimson/common/formatter.h" diff --git a/src/crimson/osd/main.cc b/src/crimson/osd/main.cc index fa387804dcd..0bfd3e2266b 100644 --- a/src/crimson/osd/main.cc +++ b/src/crimson/osd/main.cc @@ -24,6 +24,7 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" #include "crimson/common/fatal_signal.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/mon/MonClient.h" #include "crimson/net/Messenger.h" #include "crimson/osd/stop_signal.h" @@ -201,7 +202,7 @@ int main(int argc, const char* argv[]) true); } auto store = crimson::os::FuturizedStore::create( - local_conf().get_val<std::string>("osd_objectstore"), + local_conf().get_val<std::string>("crimson_osd_objectstore"), local_conf().get_val<std::string>("osd_data"), local_conf().get_config_values()); diff --git a/src/crimson/osd/main_config_bootstrap_helpers.cc b/src/crimson/osd/main_config_bootstrap_helpers.cc index 3596929527f..e4920eb870f 100644 --- a/src/crimson/osd/main_config_bootstrap_helpers.cc +++ b/src/crimson/osd/main_config_bootstrap_helpers.cc @@ -17,10 +17,13 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" #include "crimson/common/fatal_signal.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/mon/MonClient.h" #include "crimson/net/Messenger.h" #include "crimson/osd/main_config_bootstrap_helpers.h" +#include <sys/wait.h> // for waitpid() + using namespace std::literals; using crimson::common::local_conf; using crimson::common::sharded_conf; diff --git a/src/crimson/osd/object_context.h b/src/crimson/osd/object_context.h index 6f51045931d..4195e5dc597 100644 --- a/src/crimson/osd/object_context.h +++ b/src/crimson/osd/object_context.h @@ -9,6 +9,7 @@ #include <seastar/core/shared_future.hh> #include <seastar/core/shared_ptr.hh> +#include "common/fmt_common.h" #include "common/intrusive_lru.h" #include "osd/object_state.h" #include "crimson/common/exception.h" @@ -73,6 +74,8 @@ public: using watch_key_t = std::pair<uint64_t, entity_name_t>; std::map<watch_key_t, seastar::shared_ptr<crimson::osd::Watch>> watchers; + CommonOBCPipeline obc_pipeline; + ObjectContext(hobject_t hoid) : lock(hoid), obs(std::move(hoid)) {} @@ -128,14 +131,49 @@ public: } bool is_valid() const { - return !invalidated_by_interval_change; + return !invalidated; } private: boost::intrusive::list_member_hook<> obc_accessing_hook; uint64_t list_link_cnt = 0; + + /** + * loading_started + * + * ObjectContext instances may be used for pipeline stages + * prior to actually being loaded. + * + * ObjectContextLoader::load_and_lock* use loading_started + * to determine whether to initiate loading or simply take + * the desired lock directly. + * + * If loading_started is not set, the task must set it and + * (syncronously) take an exclusive lock. That exclusive lock + * must be held until the loading completes, at which point the + * lock may be relaxed or released. + * + * If loading_started is set, it is safe to directly take + * the desired lock, once the lock is obtained loading may + * be assumed to be complete. + * + * loading_started, once set, remains set for the lifetime + * of the object. + */ + bool loading_started = false; + + /// true once set_*_state has been called, used for debugging bool fully_loaded = false; - bool invalidated_by_interval_change = false; + + /** + * invalidated + * + * Set to true upon eviction from cache. This happens to all + * cached obc's upon interval change and to the target of + * a repop received on a replica to ensure that the cached + * state is refreshed upon subsequent replica read. + */ + bool invalidated = false; friend class ObjectContextRegistry; friend class ObjectContextLoader; @@ -156,6 +194,15 @@ public: } } + template <typename FormatContext> + auto fmt_print_ctx(FormatContext & ctx) const { + return fmt::format_to( + ctx.out(), "ObjectContext({}, oid={}, refcount={})", + (void*)this, + get_oid(), + get_use_count()); + } + using obc_accessing_option_t = boost::intrusive::member_hook< ObjectContext, boost::intrusive::list_member_hook<>, @@ -186,12 +233,14 @@ public: void clear_range(const hobject_t &from, const hobject_t &to) { - obc_lru.clear_range(from, to); + obc_lru.clear_range(from, to, [](auto &obc) { + obc.invalidated = true; + }); } void invalidate_on_interval_change() { obc_lru.clear([](auto &obc) { - obc.invalidated_by_interval_change = true; + obc.invalidated = true; }); } diff --git a/src/crimson/osd/object_context_loader.cc b/src/crimson/osd/object_context_loader.cc index 869ca91504c..483251a23b5 100644 --- a/src/crimson/osd/object_context_loader.cc +++ b/src/crimson/osd/object_context_loader.cc @@ -16,19 +16,20 @@ ObjectContextLoader::load_and_lock_head(Manager &manager, RWState::State lock_ty LOG_PREFIX(ObjectContextLoader::load_and_lock_head); DEBUGDPP("{} {}", dpp, manager.target, lock_type); auto releaser = manager.get_releaser(); - // no users pre-populate head_state on this path, so we don't bother to - // handle it ceph_assert(manager.target.is_head()); - ceph_assert(manager.head_state.is_empty()); + + if (manager.head_state.is_empty()) { + auto [obc, _] = obc_registry.get_cached_obc(manager.target); + manager.set_state_obc(manager.head_state, obc); + } ceph_assert(manager.target_state.is_empty()); - auto [obc, existed] = obc_registry.get_cached_obc(manager.target); - manager.set_state_obc(manager.target_state, obc); - manager.set_state_obc(manager.head_state, obc); + manager.set_state_obc(manager.target_state, manager.head_state.obc); - if (existed) { + if (manager.target_state.obc->loading_started) { co_await manager.target_state.lock_to(lock_type); } else { manager.target_state.lock_excl_sync(); + manager.target_state.obc->loading_started = true; co_await load_obc(manager.target_state.obc); manager.target_state.demote_excl_to(lock_type); } @@ -36,7 +37,8 @@ ObjectContextLoader::load_and_lock_head(Manager &manager, RWState::State lock_ty } ObjectContextLoader::load_and_lock_fut -ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_type) +ObjectContextLoader::load_and_lock_clone( + Manager &manager, RWState::State lock_type, bool lock_head) { LOG_PREFIX(ObjectContextLoader::load_and_lock_clone); DEBUGDPP("{} {}", dpp, manager.target, lock_type); @@ -46,16 +48,20 @@ ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_t ceph_assert(manager.target_state.is_empty()); if (manager.head_state.is_empty()) { - auto [obc, existed] = obc_registry.get_cached_obc(manager.target.get_head()); + auto [obc, _] = obc_registry.get_cached_obc(manager.target.get_head()); manager.set_state_obc(manager.head_state, obc); + } - if (existed) { - co_await manager.head_state.lock_to(RWState::RWREAD); - } else { - manager.head_state.lock_excl_sync(); - co_await load_obc(manager.head_state.obc); - manager.head_state.demote_excl_to(RWState::RWREAD); - } + if (!manager.head_state.obc->loading_started) { + // caller is responsible for pre-populating a loaded obc if lock_head is + // false + ceph_assert(lock_head); + manager.head_state.lock_excl_sync(); + manager.head_state.obc->loading_started = true; + co_await load_obc(manager.head_state.obc); + manager.head_state.demote_excl_to(RWState::RWREAD); + } else if (lock_head) { + co_await manager.head_state.lock_to(RWState::RWREAD); } if (manager.options.resolve_clone) { @@ -95,13 +101,14 @@ ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_t manager.head_state.state = RWState::RWNONE; } } else { - auto [obc, existed] = obc_registry.get_cached_obc(manager.target); + auto [obc, _] = obc_registry.get_cached_obc(manager.target); manager.set_state_obc(manager.target_state, obc); - if (existed) { + if (manager.target_state.obc->loading_started) { co_await manager.target_state.lock_to(RWState::RWREAD); } else { manager.target_state.lock_excl_sync(); + manager.target_state.obc->loading_started = true; co_await load_obc(manager.target_state.obc); manager.target_state.obc->set_clone_ssc(manager.head_state.obc->ssc); manager.target_state.demote_excl_to(RWState::RWREAD); diff --git a/src/crimson/osd/object_context_loader.h b/src/crimson/osd/object_context_loader.h index 6d007d65176..49f8f1572bf 100644 --- a/src/crimson/osd/object_context_loader.h +++ b/src/crimson/osd/object_context_loader.h @@ -2,9 +2,11 @@ #include <seastar/core/future.hh> #include <seastar/util/defer.hh> +#include "crimson/common/coroutine.h" #include "crimson/common/errorator.h" #include "crimson/common/log.h" #include "crimson/osd/object_context.h" +#include "crimson/osd/osd_operation.h" #include "crimson/osd/pg_backend.h" #include "osd/object_state_fmt.h" @@ -143,9 +145,7 @@ public: if (s.is_empty()) return; s.release_lock(); - SUBDEBUGDPP( - osd, "released object {}, {}", - loader.dpp, s.obc->get_oid(), s.obc->obs); + SUBDEBUGDPP(osd, "releasing obc {}, {}", loader.dpp, *(s.obc), s.obc->obs); s.obc->remove_from(loader.obc_set_accessing); s = state_t(); } @@ -165,11 +165,13 @@ public: ObjectContextRef &get_obc() { ceph_assert(!target_state.is_empty()); + ceph_assert(target_state.obc->is_loaded()); return target_state.obc; } ObjectContextRef &get_head_obc() { ceph_assert(!head_state.is_empty()); + ceph_assert(head_state.obc->is_loaded()); return head_state.obc; } @@ -188,19 +190,49 @@ public: release(); } }; - Manager get_obc_manager(hobject_t oid, bool resolve_clone = true) { + + class Orderer { + friend ObjectContextLoader; + ObjectContextRef orderer_obc; + public: + CommonOBCPipeline &obc_pp() { + ceph_assert(orderer_obc); + return orderer_obc->obc_pipeline; + } + + ~Orderer() { + LOG_PREFIX(ObjectContextLoader::~Orderer); + SUBDEBUG(osd, "releasing obc {}, {}", *(orderer_obc)); + } + }; + + Orderer get_obc_orderer(const hobject_t &oid) { + Orderer ret; + std::tie(ret.orderer_obc, std::ignore) = + obc_registry.get_cached_obc(oid.get_head()); + return ret; + } + + Manager get_obc_manager(const hobject_t &oid, bool resolve_clone = true) { Manager ret(*this, oid); ret.options.resolve_clone = resolve_clone; return ret; } + Manager get_obc_manager( + Orderer &orderer, const hobject_t &oid, bool resolve_clone = true) { + Manager ret = get_obc_manager(oid, resolve_clone); + ret.set_state_obc(ret.head_state, orderer.orderer_obc); + return ret; + } + using load_and_lock_ertr = load_obc_ertr; using load_and_lock_iertr = interruptible::interruptible_errorator< IOInterruptCondition, load_and_lock_ertr>; using load_and_lock_fut = load_and_lock_iertr::future<>; private: load_and_lock_fut load_and_lock_head(Manager &, RWState::State); - load_and_lock_fut load_and_lock_clone(Manager &, RWState::State); + load_and_lock_fut load_and_lock_clone(Manager &, RWState::State, bool lock_head=true); public: load_and_lock_fut load_and_lock(Manager &, RWState::State); @@ -244,7 +276,7 @@ public: // locks on head as part of this call. manager.head_state.obc = head; manager.head_state.obc->append_to(obc_set_accessing); - co_await load_and_lock(manager, State); + co_await load_and_lock_clone(manager, State, false); co_await std::invoke(func, head, manager.get_obc()); } diff --git a/src/crimson/osd/ops_executer.cc b/src/crimson/osd/ops_executer.cc index 97b241fdce4..cbc35c21a04 100644 --- a/src/crimson/osd/ops_executer.cc +++ b/src/crimson/osd/ops_executer.cc @@ -15,12 +15,15 @@ #include <seastar/core/thread.hh> +#include "crimson/common/log.h" #include "crimson/osd/exceptions.h" #include "crimson/osd/pg.h" #include "crimson/osd/watch.h" #include "osd/ClassHandler.h" #include "osd/SnapMapper.h" +SET_SUBSYS(osd); + namespace { seastar::logger& logger() { return crimson::get_logger(ceph_subsys_osd); @@ -464,10 +467,7 @@ auto OpsExecuter::do_const_op(Func&& f) { template <class Func> auto OpsExecuter::do_write_op(Func&& f, OpsExecuter::modified_by m) { ++num_write; - if (!osd_op_params) { - osd_op_params.emplace(); - fill_op_params(m); - } + check_init_op_params(m); return std::forward<Func>(f)(pg->get_backend(), obc->obs, txn); } OpsExecuter::call_errorator::future<> OpsExecuter::do_assert_ver( @@ -822,25 +822,100 @@ OpsExecuter::do_execute_op(OSDOp& osd_op) } } -void OpsExecuter::fill_op_params(OpsExecuter::modified_by m) +OpsExecuter::rep_op_fut_t +OpsExecuter::flush_changes_and_submit( + const std::vector<OSDOp>& ops, + SnapMapper& snap_mapper, + OSDriver& osdriver) { - osd_op_params->req_id = msg->get_reqid(); - osd_op_params->mtime = msg->get_mtime(); - osd_op_params->at_version = pg->get_next_version(); - osd_op_params->pg_trim_to = pg->get_pg_trim_to(); - osd_op_params->pg_committed_to = pg->get_pg_committed_to(); - osd_op_params->last_complete = pg->get_info().last_complete; - osd_op_params->user_modify = (m == modified_by::user); + const bool want_mutate = !txn.empty(); + // osd_op_params are instantiated by every wr-like operation. + assert(osd_op_params || !want_mutate); + assert(obc); + + auto submitted = interruptor::now(); + auto all_completed = interruptor::now(); + + if (cloning_ctx) { + ceph_assert(want_mutate); + } + + apply_stats(); + if (want_mutate) { + osd_op_params->at_version = pg->get_next_version(); + osd_op_params->pg_trim_to = pg->get_pg_trim_to(); + osd_op_params->pg_committed_to = pg->get_pg_committed_to(); + osd_op_params->last_complete = pg->get_info().last_complete; + + std::vector<pg_log_entry_t> log_entries; + + if (cloning_ctx) { + log_entries.emplace_back(complete_cloning_ctx()); + } + + log_entries.emplace_back(prepare_head_update(ops, txn)); + + if (auto log_rit = log_entries.rbegin(); log_rit != log_entries.rend()) { + ceph_assert(log_rit->version == osd_op_params->at_version); + } + + /* + * This works around the gcc bug causing the generated code to incorrectly + * execute unconditionally before the predicate. + * + * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101244 + */ + auto clone_obc = cloning_ctx + ? std::move(cloning_ctx->clone_obc) + : nullptr; + auto [_submitted, _all_completed] = co_await pg->submit_transaction( + std::move(obc), + std::move(clone_obc), + std::move(txn), + std::move(*osd_op_params), + std::move(log_entries) + ); + + submitted = std::move(_submitted); + all_completed = std::move(_all_completed); + } + + if (op_effects.size()) [[unlikely]] { + // need extra ref pg due to apply_stats() which can be executed after + // informing snap mapper + all_completed = + std::move(all_completed).then_interruptible([this, pg=this->pg] { + // let's do the cleaning of `op_effects` in destructor + return interruptor::do_for_each(op_effects, + [pg=std::move(pg)](auto& op_effect) { + return op_effect->execute(pg); + }); + }); + } + + co_return std::make_tuple( + std::move(submitted), + std::move(all_completed)); } -std::vector<pg_log_entry_t> OpsExecuter::prepare_transaction( - const std::vector<OSDOp>& ops) +pg_log_entry_t OpsExecuter::prepare_head_update( + const std::vector<OSDOp>& ops, + ceph::os::Transaction &txn) { - // let's ensure we don't need to inform SnapMapper about this particular - // entry. + LOG_PREFIX(OpsExecuter::prepare_head_update); assert(obc->obs.oi.soid.snap >= CEPH_MAXSNAP); - std::vector<pg_log_entry_t> log_entries; - log_entries.emplace_back( + + update_clone_overlap(); + if (cloning_ctx) { + obc->ssc->snapset = std::move(cloning_ctx->new_snapset); + } + if (snapc.seq > obc->ssc->snapset.seq) { + // update snapset with latest snap context + obc->ssc->snapset.seq = snapc.seq; + obc->ssc->snapset.snaps.clear(); + } + + pg_log_entry_t ret{ obc->obs.exists ? pg_log_entry_t::MODIFY : pg_log_entry_t::DELETE, obc->obs.oi.soid, @@ -849,15 +924,38 @@ std::vector<pg_log_entry_t> OpsExecuter::prepare_transaction( osd_op_params->user_modify ? osd_op_params->at_version.version : 0, osd_op_params->req_id, osd_op_params->mtime, - op_info.allows_returnvec() && !ops.empty() ? ops.back().rval.code : 0); + op_info.allows_returnvec() && !ops.empty() ? ops.back().rval.code : 0}; + if (op_info.allows_returnvec()) { // also the per-op values are recorded in the pg log - log_entries.back().set_op_returns(ops); - logger().debug("{} op_returns: {}", - __func__, log_entries.back().op_returns); + ret.set_op_returns(ops); + DEBUGDPP("op returns: {}", *pg, ret.op_returns); + } + ret.clean_regions = std::move(osd_op_params->clean_regions); + + + if (obc->obs.exists) { + obc->obs.oi.prior_version = obc->obs.oi.version; + obc->obs.oi.version = osd_op_params->at_version; + if (osd_op_params->user_modify) + obc->obs.oi.user_version = osd_op_params->at_version.version; + obc->obs.oi.last_reqid = osd_op_params->req_id; + obc->obs.oi.mtime = osd_op_params->mtime; + obc->obs.oi.local_mtime = ceph_clock_now(); + + obc->ssc->exists = true; + pg->get_backend().set_metadata( + obc->obs.oi.soid, + obc->obs.oi, + obc->obs.oi.soid.is_head() ? &(obc->ssc->snapset) : nullptr, + txn); + } else { + // reset cached ObjectState without enforcing eviction + obc->obs.oi = object_info_t(obc->obs.oi.soid); } - log_entries.back().clean_regions = std::move(osd_op_params->clean_regions); - return log_entries; + + DEBUGDPP("entry: {}", *pg, ret); + return ret; } // Defined here because there is a circular dependency between OpsExecuter and PG @@ -871,25 +969,26 @@ version_t OpsExecuter::get_last_user_version() const return pg->get_last_user_version(); } -std::unique_ptr<OpsExecuter::CloningContext> OpsExecuter::execute_clone( +void OpsExecuter::prepare_cloning_ctx( const SnapContext& snapc, const ObjectState& initial_obs, const SnapSet& initial_snapset, PGBackend& backend, ceph::os::Transaction& txn) { + LOG_PREFIX(OpsExecuter::prepare_cloning_ctx); const hobject_t& soid = initial_obs.oi.soid; logger().debug("{} {} snapset={} snapc={}", __func__, soid, initial_snapset, snapc); - auto cloning_ctx = std::make_unique<CloningContext>(); + cloning_ctx = std::make_unique<CloningContext>(); cloning_ctx->new_snapset = initial_snapset; // clone object, the snap field is set to the seq of the SnapContext // at its creation. - hobject_t coid = soid; - coid.snap = snapc.seq; + cloning_ctx->coid = soid; + cloning_ctx->coid.snap = snapc.seq; // existing snaps are stored in descending order in snapc, // cloned_snaps vector will hold all the snaps stored until snapset.seq @@ -900,49 +999,63 @@ std::unique_ptr<OpsExecuter::CloningContext> OpsExecuter::execute_clone( return std::vector<snapid_t>{std::begin(snapc.snaps), last}; }(); - auto clone_obc = prepare_clone(coid, osd_op_params->at_version); - osd_op_params->at_version.version++; + // make clone here, but populate in metadata in complete_cloning_ctx + backend.clone_for_write(soid, cloning_ctx->coid, txn); - // make clone - backend.clone(clone_obc->obs.oi, initial_obs, clone_obc->obs, txn); + cloning_ctx->clone_obc = prepare_clone(cloning_ctx->coid, initial_obs); delta_stats.num_objects++; - if (clone_obc->obs.oi.is_omap()) { + if (cloning_ctx->clone_obc->obs.oi.is_omap()) { delta_stats.num_objects_omap++; } delta_stats.num_object_clones++; // newsnapset is obc's ssc - cloning_ctx->new_snapset.clones.push_back(coid.snap); - cloning_ctx->new_snapset.clone_size[coid.snap] = initial_obs.oi.size; - cloning_ctx->new_snapset.clone_snaps[coid.snap] = cloned_snaps; + cloning_ctx->new_snapset.clones.push_back(cloning_ctx->coid.snap); + cloning_ctx->new_snapset.clone_size[cloning_ctx->coid.snap] = initial_obs.oi.size; + cloning_ctx->new_snapset.clone_snaps[cloning_ctx->coid.snap] = cloned_snaps; // clone_overlap should contain an entry for each clone // (an empty interval_set if there is no overlap) - auto &overlap = cloning_ctx->new_snapset.clone_overlap[coid.snap]; + auto &overlap = cloning_ctx->new_snapset.clone_overlap[cloning_ctx->coid.snap]; if (initial_obs.oi.size) { overlap.insert(0, initial_obs.oi.size); } // log clone - logger().debug("cloning v {} to {} v {} snaps={} snapset={}", - initial_obs.oi.version, coid, - osd_op_params->at_version, cloned_snaps, cloning_ctx->new_snapset); + DEBUGDPP("cloning v {} to {} v {} snaps={} snapset={}", *pg, + initial_obs.oi.version, cloning_ctx->coid, + osd_op_params->at_version, cloned_snaps, cloning_ctx->new_snapset); +} - cloning_ctx->log_entry = { +pg_log_entry_t OpsExecuter::complete_cloning_ctx() +{ + ceph_assert(cloning_ctx); + const auto &coid = cloning_ctx->coid; + cloning_ctx->clone_obc->obs.oi.version = osd_op_params->at_version; + + osd_op_params->at_version.version++; + + pg->get_backend().set_metadata( + cloning_ctx->coid, + cloning_ctx->clone_obc->obs.oi, + nullptr /* snapset */, + txn); + + pg_log_entry_t ret{ pg_log_entry_t::CLONE, coid, - clone_obc->obs.oi.version, - clone_obc->obs.oi.prior_version, - clone_obc->obs.oi.user_version, + cloning_ctx->clone_obc->obs.oi.version, + cloning_ctx->clone_obc->obs.oi.prior_version, + cloning_ctx->clone_obc->obs.oi.user_version, osd_reqid_t(), - clone_obc->obs.oi.mtime, // will be replaced in `apply_to()` + cloning_ctx->clone_obc->obs.oi.mtime, // will be replaced in `apply_to()` 0 }; - encode(cloned_snaps, cloning_ctx->log_entry.snaps); - cloning_ctx->log_entry.clean_regions.mark_data_region_dirty(0, initial_obs.oi.size); - cloning_ctx->clone_obc = clone_obc; - - return cloning_ctx; + ceph_assert(cloning_ctx->new_snapset.clone_snaps.count(coid.snap)); + encode(cloning_ctx->new_snapset.clone_snaps[coid.snap], ret.snaps); + ret.clean_regions.mark_data_region_dirty(0, cloning_ctx->clone_obc->obs.oi.size); + ret.mtime = cloning_ctx->clone_obc->obs.oi.mtime; + return ret; } void OpsExecuter::update_clone_overlap() { @@ -965,47 +1078,16 @@ void OpsExecuter::update_clone_overlap() { delta_stats.num_bytes += osd_op_params->modified_ranges.size(); } -void OpsExecuter::CloningContext::apply_to( - std::vector<pg_log_entry_t>& log_entries, - ObjectContext& processed_obc) -{ - log_entry.mtime = processed_obc.obs.oi.mtime; - log_entries.insert(log_entries.begin(), std::move(log_entry)); - processed_obc.ssc->snapset = std::move(new_snapset); -} - -std::vector<pg_log_entry_t> -OpsExecuter::flush_clone_metadata( - std::vector<pg_log_entry_t>&& log_entries, - SnapMapper& snap_mapper, - OSDriver& osdriver, - ceph::os::Transaction& txn) -{ - assert(!txn.empty()); - update_clone_overlap(); - if (cloning_ctx) { - cloning_ctx->apply_to(log_entries, *obc); - } - if (snapc.seq > obc->ssc->snapset.seq) { - // update snapset with latest snap context - obc->ssc->snapset.seq = snapc.seq; - obc->ssc->snapset.snaps.clear(); - } - logger().debug("{} done, initial snapset={}, new snapset={}", - __func__, obc->obs.oi.soid, obc->ssc->snapset); - return std::move(log_entries); -} - ObjectContextRef OpsExecuter::prepare_clone( const hobject_t& coid, - eversion_t version) + const ObjectState& initial_obs) { ceph_assert(pg->is_primary()); ObjectState clone_obs{coid}; clone_obs.exists = true; - clone_obs.oi.version = version; - clone_obs.oi.prior_version = obc->obs.oi.version; - clone_obs.oi.copy_user_bits(obc->obs.oi); + // clone_obs.oi.version will be populated in complete_cloning_ctx + clone_obs.oi.prior_version = initial_obs.oi.version; + clone_obs.oi.copy_user_bits(initial_obs.oi); clone_obs.oi.clear_flag(object_info_t::FLAG_WHITEOUT); auto [clone_obc, existed] = pg->obc_registry.get_cached_obc(std::move(coid)); @@ -1036,11 +1118,12 @@ OpsExecuter::OpsExecuter(Ref<PG> pg, { if (op_info.may_write() && should_clone(*obc, snapc)) { do_write_op([this](auto& backend, auto& os, auto& txn) { - cloning_ctx = execute_clone(std::as_const(snapc), - std::as_const(obc->obs), - std::as_const(obc->ssc->snapset), - backend, - txn); + prepare_cloning_ctx( + std::as_const(snapc), + std::as_const(obc->obs), + std::as_const(obc->ssc->snapset), + backend, + txn); }); } } diff --git a/src/crimson/osd/ops_executer.h b/src/crimson/osd/ops_executer.h index 51436f19da7..f5554bd6919 100644 --- a/src/crimson/osd/ops_executer.h +++ b/src/crimson/osd/ops_executer.h @@ -195,26 +195,26 @@ private: SnapContext snapc; // writer snap context struct CloningContext { + /// id of new clone, populated in prepare_cloning_ctx + hobject_t coid; + /// new snapset, populated in prepare_cloning_ctx SnapSet new_snapset; - pg_log_entry_t log_entry; + /// populated in complete_cloning_ctx ObjectContextRef clone_obc; - - void apply_to( - std::vector<pg_log_entry_t>& log_entries, - ObjectContext& processed_obc); }; std::unique_ptr<CloningContext> cloning_ctx; - /** - * execute_clone + * prepare_cloning_ctx * * If snapc contains a snap which occurred logically after the last write * seen by this object (see OpsExecuter::should_clone()), we first need - * make a clone of the object at its current state. execute_clone primes - * txn with that clone operation and returns an - * OpsExecuter::CloningContext which will allow us to fill in the corresponding - * metadata and log_entries once the operations have been processed. + * make a clone of the object at its current state. prepare_cloning_ctx + * primes txn with that clone operation and populates cloning_ctx with + * an obc for the clone and a new snapset reflecting the clone. + * + * complete_cloning_ctx later uses the information from cloning_ctx to + * generate a log entry and object_info versions for the clone. * * Note that this strategy differs from classic, which instead performs this * work at the end and reorders the transaction. See @@ -227,13 +227,15 @@ private: * @param backend [in,out] interface for generating mutations * @param txn [out] transaction for the operation */ - std::unique_ptr<CloningContext> execute_clone( + void prepare_cloning_ctx( const SnapContext& snapc, const ObjectState& initial_obs, const SnapSet& initial_snapset, PGBackend& backend, ceph::os::Transaction& txn); + /// complete clone, populate clone_obc, return log entry + pg_log_entry_t complete_cloning_ctx(); /** * should_clone @@ -264,12 +266,6 @@ private: */ void update_clone_overlap(); - std::vector<pg_log_entry_t> flush_clone_metadata( - std::vector<pg_log_entry_t>&& log_entries, - SnapMapper& snap_mapper, - OSDriver& osdriver, - ceph::os::Transaction& txn); - private: // this gizmo could be wrapped in std::optional for the sake of lazy // initialization. we don't need it for ops that doesn't have effect @@ -400,15 +396,22 @@ public: std::tuple<interruptible_future<>, interruptible_future<>>; using rep_op_fut_t = interruptible_future<rep_op_fut_tuple>; - template <typename MutFunc> - rep_op_fut_t flush_changes_n_do_ops_effects( + rep_op_fut_t flush_changes_and_submit( const std::vector<OSDOp>& ops, SnapMapper& snap_mapper, - OSDriver& osdriver, - MutFunc mut_func) &&; - std::vector<pg_log_entry_t> prepare_transaction( - const std::vector<OSDOp>& ops); - void fill_op_params(modified_by m); + OSDriver& osdriver); + pg_log_entry_t prepare_head_update( + const std::vector<OSDOp>& ops, + ceph::os::Transaction &txn); + + void check_init_op_params(OpsExecuter::modified_by m) { + if (!osd_op_params) { + osd_op_params.emplace(); + osd_op_params->req_id = msg->get_reqid(); + osd_op_params->mtime = msg->get_mtime(); + osd_op_params->user_modify = (m == modified_by::user); + } + } ObjectContextRef get_obc() const { return obc; @@ -443,7 +446,7 @@ public: ObjectContextRef prepare_clone( const hobject_t& coid, - eversion_t version); + const ObjectState& initial_obs); void apply_stats(); }; @@ -485,76 +488,6 @@ auto OpsExecuter::with_effect_on_obc( return std::forward<MainFunc>(main_func)(ctx_ref); } -template <typename MutFunc> -OpsExecuter::rep_op_fut_t -OpsExecuter::flush_changes_n_do_ops_effects( - const std::vector<OSDOp>& ops, - SnapMapper& snap_mapper, - OSDriver& osdriver, - MutFunc mut_func) && -{ - const bool want_mutate = !txn.empty(); - // osd_op_params are instantiated by every wr-like operation. - assert(osd_op_params || !want_mutate); - assert(obc); - - auto submitted = interruptor::now(); - auto all_completed = interruptor::now(); - - if (cloning_ctx) { - ceph_assert(want_mutate); - } - - apply_stats(); - if (want_mutate) { - auto log_entries = flush_clone_metadata( - prepare_transaction(ops), - snap_mapper, - osdriver, - txn); - - if (auto log_rit = log_entries.rbegin(); log_rit != log_entries.rend()) { - ceph_assert(log_rit->version == osd_op_params->at_version); - } - - /* - * This works around the gcc bug causing the generated code to incorrectly - * execute unconditionally before the predicate. - * - * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101244 - */ - auto clone_obc = cloning_ctx - ? std::move(cloning_ctx->clone_obc) - : nullptr; - auto [_submitted, _all_completed] = co_await mut_func( - std::move(txn), - std::move(obc), - std::move(*osd_op_params), - std::move(log_entries), - std::move(clone_obc)); - - submitted = std::move(_submitted); - all_completed = std::move(_all_completed); - } - - if (op_effects.size()) [[unlikely]] { - // need extra ref pg due to apply_stats() which can be executed after - // informing snap mapper - all_completed = - std::move(all_completed).then_interruptible([this, pg=this->pg] { - // let's do the cleaning of `op_effects` in destructor - return interruptor::do_for_each(op_effects, - [pg=std::move(pg)](auto& op_effect) { - return op_effect->execute(pg); - }); - }); - } - - co_return std::make_tuple( - std::move(submitted), - std::move(all_completed)); -} - template <class Func> struct OpsExecuter::RollbackHelper { void rollback_obc_if_modified(); diff --git a/src/crimson/osd/osd_operation.h b/src/crimson/osd/osd_operation.h index fd8b049c0bf..394375c1129 100644 --- a/src/crimson/osd/osd_operation.h +++ b/src/crimson/osd/osd_operation.h @@ -50,24 +50,36 @@ struct PGPeeringPipeline { }; struct CommonPGPipeline { - struct WaitForActive : OrderedExclusivePhaseT<WaitForActive> { - static constexpr auto type_name = "CommonPGPipeline:::wait_for_active"; - } wait_for_active; - struct RecoverMissing : OrderedConcurrentPhaseT<RecoverMissing> { - static constexpr auto type_name = "CommonPGPipeline::recover_missing"; - } recover_missing; - struct CheckAlreadyCompleteGetObc : OrderedExclusivePhaseT<CheckAlreadyCompleteGetObc> { - static constexpr auto type_name = "CommonPGPipeline::check_already_complete_get_obc"; - } check_already_complete_get_obc; - struct LockOBC : OrderedConcurrentPhaseT<LockOBC> { - static constexpr auto type_name = "CommonPGPipeline::lock_obc"; - } lock_obc; + struct WaitPGReady : OrderedConcurrentPhaseT<WaitPGReady> { + static constexpr auto type_name = "CommonPGPipeline:::wait_pg_ready"; + } wait_pg_ready; + struct GetOBC : OrderedExclusivePhaseT<GetOBC> { + static constexpr auto type_name = "CommonPGPipeline:::get_obc"; + } get_obc; +}; + +struct PGRepopPipeline { + struct Process : OrderedExclusivePhaseT<Process> { + static constexpr auto type_name = "PGRepopPipeline::process"; + } process; + struct WaitCommit : OrderedConcurrentPhaseT<WaitCommit> { + static constexpr auto type_name = "PGRepopPipeline::wait_repop"; + } wait_commit; + struct SendReply : OrderedExclusivePhaseT<SendReply> { + static constexpr auto type_name = "PGRepopPipeline::send_reply"; + } send_reply; +}; + +struct CommonOBCPipeline { struct Process : OrderedExclusivePhaseT<Process> { - static constexpr auto type_name = "CommonPGPipeline::process"; + static constexpr auto type_name = "CommonOBCPipeline::process"; } process; struct WaitRepop : OrderedConcurrentPhaseT<WaitRepop> { - static constexpr auto type_name = "ClientRequest::PGPipeline::wait_repop"; + static constexpr auto type_name = "CommonOBCPipeline::wait_repop"; } wait_repop; + struct SendReply : OrderedExclusivePhaseT<SendReply> { + static constexpr auto type_name = "CommonOBCPipeline::send_reply"; + } send_reply; }; @@ -205,6 +217,9 @@ protected: public: static constexpr bool is_trackable = true; + virtual bool requires_pg() const { + return true; + } }; template <class T> @@ -326,6 +341,18 @@ public: with_throttle_while(std::forward<Args>(args)...), *this); } + // Returns std::nullopt if the throttle is acquired immediately, + // returns the future for the acquiring otherwise + std::optional<seastar::future<>> + try_acquire_throttle_now(crimson::osd::scheduler::params_t params) { + if (!max_in_progress || in_progress < max_in_progress) { + ++in_progress; + --pending; + return std::nullopt; + } + return acquire_throttle(params); + } + private: void dump_detail(Formatter *f) const final; diff --git a/src/crimson/osd/osd_operation_external_tracking.h b/src/crimson/osd/osd_operation_external_tracking.h index d2786a95e4d..6a2d7e3ccbd 100644 --- a/src/crimson/osd/osd_operation_external_tracking.h +++ b/src/crimson/osd/osd_operation_external_tracking.h @@ -25,24 +25,23 @@ struct LttngBackend ConnectionPipeline::AwaitMap::BlockingEvent::Backend, ConnectionPipeline::GetPGMapping::BlockingEvent::Backend, PerShardPipeline::CreateOrWaitPG::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent::Backend, + CommonPGPipeline::GetOBC::BlockingEvent::Backend, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, PGMap::PGCreationBlockingEvent::Backend, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent::Backend, PG_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitForActive::BlockingEvent::Backend, PGActivationBlocker::BlockingEvent::Backend, scrub::PGScrubber::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing:: - BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::Process::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::SendReply::BlockingEvent::Backend, - ClientRequest::CompletionEvent::Backend + ClientRequest::CompletionEvent::Backend, + CommonOBCPipeline::Process::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, + CommonOBCPipeline::SendReply::BlockingEvent::Backend, + PGRepopPipeline::Process::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent::Backend, + PGRepopPipeline::SendReply::BlockingEvent::Backend { void handle(ClientRequest::StartEvent&, const Operation&) override {} @@ -72,24 +71,28 @@ struct LttngBackend const PerShardPipeline::CreateOrWaitPG& blocker) override { } - void handle(PGMap::PGCreationBlockingEvent&, - const Operation&, - const PGMap::PGCreationBlocker&) override { + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent& ev, + const Operation& op, + const CommonPGPipeline::WaitPGReady& blocker) override { } - void handle(ClientRequest::PGPipeline::AwaitMap::BlockingEvent& ev, + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { + } + + void handle(CommonPGPipeline::GetOBC::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::AwaitMap& blocker) override { + const CommonPGPipeline::GetOBC& blocker) override { } - void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + void handle(PGMap::PGCreationBlockingEvent&, const Operation&, - const PG_OSDMapGate::OSDMapBlocker&) override { + const PGMap::PGCreationBlocker&) override { } - void handle(ClientRequest::PGPipeline::WaitForActive::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::WaitForActive& blocker) override { + void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + const Operation&, + const PG_OSDMapGate::OSDMapBlocker&) override { } void handle(PGActivationBlocker::BlockingEvent& ev, @@ -102,51 +105,47 @@ struct LttngBackend const scrub::PGScrubber& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent& ev, + void handle(CommonOBCPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::RecoverMissing& blocker) override { + const CommonOBCPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { - } - - void handle(ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc& blocker) override { + const CommonOBCPipeline::WaitRepop& blocker) override { } - - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::LockOBC& blocker) override { + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { + void handle(CommonOBCPipeline::SendReply::BlockingEvent& ev, + const Operation& op, + const CommonOBCPipeline::SendReply& blocker) override { } - void handle(ClientRequest::PGPipeline::Process::BlockingEvent& ev, + void handle(PGRepopPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::Process& blocker) override { + const PGRepopPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::WaitRepop& blocker) override { + const PGRepopPipeline::WaitCommit& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::SendReply::BlockingEvent& ev, + void handle(PGRepopPipeline::SendReply::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::SendReply& blocker) override { + const PGRepopPipeline::SendReply& blocker) override { } void handle(ClientRequest::CompletionEvent&, const Operation&) override {} + }; struct HistoricBackend @@ -155,24 +154,23 @@ struct HistoricBackend ConnectionPipeline::AwaitMap::BlockingEvent::Backend, ConnectionPipeline::GetPGMapping::BlockingEvent::Backend, PerShardPipeline::CreateOrWaitPG::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent::Backend, + CommonPGPipeline::GetOBC::BlockingEvent::Backend, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, PGMap::PGCreationBlockingEvent::Backend, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent::Backend, PG_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitForActive::BlockingEvent::Backend, PGActivationBlocker::BlockingEvent::Backend, scrub::PGScrubber::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing:: - BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::Process::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::SendReply::BlockingEvent::Backend, - ClientRequest::CompletionEvent::Backend + ClientRequest::CompletionEvent::Backend, + CommonOBCPipeline::Process::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, + CommonOBCPipeline::SendReply::BlockingEvent::Backend, + PGRepopPipeline::Process::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent::Backend, + PGRepopPipeline::SendReply::BlockingEvent::Backend { void handle(ClientRequest::StartEvent&, const Operation&) override {} @@ -202,24 +200,28 @@ struct HistoricBackend const PerShardPipeline::CreateOrWaitPG& blocker) override { } - void handle(PGMap::PGCreationBlockingEvent&, - const Operation&, - const PGMap::PGCreationBlocker&) override { + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent& ev, + const Operation& op, + const CommonPGPipeline::WaitPGReady& blocker) override { + } + + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { } - void handle(ClientRequest::PGPipeline::AwaitMap::BlockingEvent& ev, + void handle(CommonPGPipeline::GetOBC::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::AwaitMap& blocker) override { + const CommonPGPipeline::GetOBC& blocker) override { } - void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + void handle(PGMap::PGCreationBlockingEvent&, const Operation&, - const PG_OSDMapGate::OSDMapBlocker&) override { + const PGMap::PGCreationBlocker&) override { } - void handle(ClientRequest::PGPipeline::WaitForActive::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::WaitForActive& blocker) override { + void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + const Operation&, + const PG_OSDMapGate::OSDMapBlocker&) override { } void handle(PGActivationBlocker::BlockingEvent& ev, @@ -232,55 +234,52 @@ struct HistoricBackend const scrub::PGScrubber& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::RecoverMissing& blocker) override { - } - - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { + static const ClientRequest& to_client_request(const Operation& op) { +#ifdef NDEBUG + return static_cast<const ClientRequest&>(op); +#else + return dynamic_cast<const ClientRequest&>(op); +#endif } - void handle(ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent& ev, + void handle(CommonOBCPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc& blocker) override { + const CommonOBCPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::LockOBC& blocker) override { + const CommonOBCPipeline::WaitRepop& blocker) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::Process::BlockingEvent& ev, + void handle(CommonOBCPipeline::SendReply::BlockingEvent& ev, + const Operation& op, + const CommonOBCPipeline::SendReply& blocker) override { + } + + void handle(PGRepopPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::Process& blocker) override { + const PGRepopPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::WaitRepop& blocker) override { + const PGRepopPipeline::WaitCommit& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::SendReply::BlockingEvent& ev, + void handle(PGRepopPipeline::SendReply::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::SendReply& blocker) override { + const PGRepopPipeline::SendReply& blocker) override { } - static const ClientRequest& to_client_request(const Operation& op) { -#ifdef NDEBUG - return static_cast<const ClientRequest&>(op); -#else - return dynamic_cast<const ClientRequest&>(op); -#endif - } void handle(ClientRequest::CompletionEvent&, const Operation& op) override { if (crimson::common::local_conf()->osd_op_history_size) { diff --git a/src/crimson/osd/osd_operations/client_request.cc b/src/crimson/osd/osd_operations/client_request.cc index 61a56600a57..fcd0f318db2 100644 --- a/src/crimson/osd/osd_operations/client_request.cc +++ b/src/crimson/osd/osd_operations/client_request.cc @@ -101,7 +101,7 @@ PerShardPipeline &ClientRequest::get_pershard_pipeline( return shard_services.get_client_request_pipeline(); } -ClientRequest::PGPipeline &ClientRequest::client_pp(PG &pg) +CommonPGPipeline &ClientRequest::client_pp(PG &pg) { return pg.request_pg_pipeline; } @@ -140,6 +140,15 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{} start", *pgref, *this); PG &pg = *pgref; + + DEBUGDPP("{}.{}: entering wait_pg_ready stage", + *pgref, *this, this_instance_id); + // The prior stage is OrderedExclusive (PerShardPipeline::create_or_wait_pg) + // and wait_pg_ready is OrderedConcurrent. This transition, therefore, cannot + // block and using enter_stage_sync is legal and more efficient than + // enter_stage. + ihref.enter_stage_sync(client_pp(pg).wait_pg_ready, *this); + if (!m->get_hobj().get_key().empty()) { // There are no users of locator. It was used to ensure that multipart-upload // parts would end up in the same PG so that they could be clone_range'd into @@ -156,28 +165,22 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{}: discarding {}", *pgref, *this, this_instance_id); co_return; } - DEBUGDPP("{}.{}: entering await_map stage", - *pgref, *this, this_instance_id); - co_await ihref.enter_stage<interruptor>(client_pp(pg).await_map, *this); - DEBUGDPP("{}.{}: entered await_map stage, waiting for map", - pg, *this, this_instance_id); + auto map_epoch = co_await interruptor::make_interruptible( ihref.enter_blocker( *this, pg.osdmap_gate, &decltype(pg.osdmap_gate)::wait_for_map, m->get_min_epoch(), nullptr)); - DEBUGDPP("{}.{}: map epoch got {}, entering wait_for_active", + DEBUGDPP("{}.{}: waited for epoch {}, waiting for active", pg, *this, this_instance_id, map_epoch); - co_await ihref.enter_stage<interruptor>(client_pp(pg).wait_for_active, *this); - - DEBUGDPP("{}.{}: entered wait_for_active stage, waiting for active", - pg, *this, this_instance_id); co_await interruptor::make_interruptible( ihref.enter_blocker( *this, pg.wait_for_active_blocker, &decltype(pg.wait_for_active_blocker)::wait)); + co_await ihref.enter_stage<interruptor>(client_pp(pg).get_obc, *this); + if (int res = op_info.set_from_op(&*m, *pg.get_osdmap()); res != 0) { co_await reply_op_error(pgref, res); @@ -238,10 +241,6 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{}.{}: process[_pg]_op complete, completing handle", *pgref, *this, this_instance_id); co_await interruptor::make_interruptible(ihref.handle.complete()); - - DEBUGDPP("{}.{}: process[_pg]_op complete," - "removing request from orderer", - *pgref, *this, this_instance_id); } seastar::future<> ClientRequest::with_pg_process( @@ -257,10 +256,13 @@ seastar::future<> ClientRequest::with_pg_process( auto instance_handle = get_instance_handle(); auto &ihref = *instance_handle; return interruptor::with_interruption( - [this, pgref, this_instance_id, &ihref]() mutable { + [FNAME, this, pgref, this_instance_id, &ihref]() mutable { return with_pg_process_interruptible( pgref, this_instance_id, ihref - ).then_interruptible([this, pgref] { + ).then_interruptible([FNAME, this, this_instance_id, pgref] { + DEBUGDPP("{}.{}: with_pg_process_interruptible complete," + " completing request", + *pgref, *this, this_instance_id); complete_request(*pgref); }); }, [FNAME, this, this_instance_id, pgref](std::exception_ptr eptr) { @@ -268,9 +270,10 @@ seastar::future<> ClientRequest::with_pg_process( *pgref, *this, this_instance_id, eptr); }, pgref, pgref->get_osdmap_epoch()).finally( [this, FNAME, opref=std::move(opref), pgref, - this_instance_id, instance_handle=std::move(instance_handle), &ihref] { + this_instance_id, instance_handle=std::move(instance_handle), &ihref]() mutable { DEBUGDPP("{}.{}: exit", *pgref, *this, this_instance_id); - ihref.handle.exit(); + return ihref.handle.complete( + ).finally([instance_handle=std::move(instance_handle)] {}); }); } @@ -345,7 +348,13 @@ ClientRequest::process_op( instance_handle_t &ihref, Ref<PG> pg, unsigned this_instance_id) { LOG_PREFIX(ClientRequest::process_op); - ihref.enter_stage_sync(client_pp(*pg).recover_missing, *this); + ihref.obc_orderer = pg->obc_loader.get_obc_orderer(m->get_hobj()); + auto obc_manager = pg->obc_loader.get_obc_manager( + *(ihref.obc_orderer), + m->get_hobj()); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().process, *this); + if (!pg->is_primary()) { DEBUGDPP( "Skipping recover_missings on non primary pg for soid {}", @@ -365,16 +374,6 @@ ClientRequest::process_op( } } - /** - * The previous stage of recover_missing is a concurrent phase. - * Checking for already_complete requests must done exclusively. - * Since get_obc is also an exclusive stage, we can merge both stages into - * a single stage and avoid stage switching overhead. - */ - DEBUGDPP("{}.{}: entering check_already_complete_get_obc", - *pg, *this, this_instance_id); - co_await ihref.enter_stage<interruptor>( - client_pp(*pg).check_already_complete_get_obc, *this); DEBUGDPP("{}.{}: checking already_complete", *pg, *this, this_instance_id); auto completed = co_await pg->already_complete(m->get_reqid()); @@ -402,12 +401,6 @@ ClientRequest::process_op( DEBUGDPP("{}.{}: past scrub blocker, getting obc", *pg, *this, this_instance_id); - auto obc_manager = pg->obc_loader.get_obc_manager(m->get_hobj()); - - // initiate load_and_lock in order, but wait concurrently - ihref.enter_stage_sync( - client_pp(*pg).lock_obc, *this); - int load_err = co_await pg->obc_loader.load_and_lock( obc_manager, pg->get_lock_type(op_info) ).si_then([]() -> int { @@ -425,13 +418,8 @@ ClientRequest::process_op( co_return; } - DEBUGDPP("{}.{}: got obc {}, entering process stage", + DEBUGDPP("{}.{}: obc {} loaded and locked, calling do_process", *pg, *this, this_instance_id, obc_manager.get_obc()->obs); - co_await ihref.enter_stage<interruptor>( - client_pp(*pg).process, *this); - - DEBUGDPP("{}.{}: in process stage, calling do_process", - *pg, *this, this_instance_id); co_await do_process( ihref, pg, obc_manager.get_obc(), this_instance_id ); @@ -553,12 +541,14 @@ ClientRequest::do_process( std::move(ox), m->ops); co_await std::move(submitted); } - co_await ihref.enter_stage<interruptor>(client_pp(*pg).wait_repop, *this); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().wait_repop, *this); co_await std::move(all_completed); } - co_await ihref.enter_stage<interruptor>(client_pp(*pg).send_reply, *this); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().send_reply, *this); if (ret) { int err = -ret->value(); diff --git a/src/crimson/osd/osd_operations/client_request.h b/src/crimson/osd/osd_operations/client_request.h index 6d1043e2783..91a6728fd4b 100644 --- a/src/crimson/osd/osd_operations/client_request.h +++ b/src/crimson/osd/osd_operations/client_request.h @@ -11,6 +11,7 @@ #include "osd/osd_op_util.h" #include "crimson/net/Connection.h" #include "crimson/osd/object_context.h" +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osdmap_gate.h" #include "crimson/osd/osd_operation.h" #include "crimson/osd/osd_operations/client_request_common.h" @@ -41,21 +42,9 @@ class ClientRequest final : public PhasedOperationT<ClientRequest>, unsigned instance_id = 0; public: - class PGPipeline : public CommonPGPipeline { - public: - struct AwaitMap : OrderedExclusivePhaseT<AwaitMap> { - static constexpr auto type_name = "ClientRequest::PGPipeline::await_map"; - } await_map; - struct SendReply : OrderedExclusivePhaseT<SendReply> { - static constexpr auto type_name = "ClientRequest::PGPipeline::send_reply"; - } send_reply; - friend class ClientRequest; - friend class LttngBackend; - friend class HistoricBackend; - friend class ReqRequest; - friend class LogMissingRequest; - friend class LogMissingRequestReply; - }; + epoch_t get_epoch_sent_at() const { + return m->get_map_epoch(); + } /** * instance_handle_t @@ -93,20 +82,18 @@ public: // don't leave any references on the source core, so we just bypass it by using // intrusive_ptr instead. using ref_t = boost::intrusive_ptr<instance_handle_t>; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; std::tuple< - PGPipeline::AwaitMap::BlockingEvent, + CommonPGPipeline::WaitPGReady::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, - PGPipeline::WaitForActive::BlockingEvent, PGActivationBlocker::BlockingEvent, - PGPipeline::RecoverMissing::BlockingEvent, + CommonPGPipeline::GetOBC::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, scrub::PGScrubber::BlockingEvent, - PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - PGPipeline::LockOBC::BlockingEvent, - PGPipeline::Process::BlockingEvent, - PGPipeline::WaitRepop::BlockingEvent, - PGPipeline::SendReply::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, + CommonOBCPipeline::SendReply::BlockingEvent, CompletionEvent > pg_tracking_events; @@ -293,7 +280,7 @@ private: unsigned this_instance_id); bool is_pg_op() const; - PGPipeline &client_pp(PG &pg); + CommonPGPipeline &client_pp(PG &pg); template <typename Errorator> using interruptible_errorator = diff --git a/src/crimson/osd/osd_operations/internal_client_request.cc b/src/crimson/osd/osd_operations/internal_client_request.cc index 4790025065a..b8f7646bc74 100644 --- a/src/crimson/osd/osd_operations/internal_client_request.cc +++ b/src/crimson/osd/osd_operations/internal_client_request.cc @@ -52,39 +52,17 @@ CommonPGPipeline& InternalClientRequest::client_pp() } InternalClientRequest::interruptible_future<> -InternalClientRequest::do_process( - crimson::osd::ObjectContextRef obc, - std::vector<OSDOp> &osd_ops) -{ - LOG_PREFIX(InternalClientRequest::do_process); - auto params = get_do_osd_ops_params(); - OpsExecuter ox( - pg, obc, op_info, params, params.get_connection(), SnapContext{}); - co_await pg->run_executer( - ox, obc, op_info, osd_ops - ).handle_error_interruptible( - crimson::ct_error::all_same_way( - [this, FNAME](auto e) { - ERRORDPPI("{}: got unexpected error {}", *pg, *this, e); - ceph_assert(0 == "should not return an error"); - return interruptor::now(); - }) - ); - - auto [submitted, completed] = co_await pg->submit_executer( - std::move(ox), osd_ops); - - co_await std::move(submitted); - co_await std::move(completed); -} - -InternalClientRequest::interruptible_future<> InternalClientRequest::with_interruption() { LOG_PREFIX(InternalClientRequest::with_interruption); assert(pg->is_active()); - co_await enter_stage<interruptor>(client_pp().recover_missing); + obc_orderer = pg->obc_loader.get_obc_orderer(get_target_oid()); + auto obc_manager = pg->obc_loader.get_obc_manager( + *obc_orderer, + get_target_oid()); + + co_await enter_stage<interruptor>(obc_orderer->obc_pp().process); bool unfound = co_await do_recover_missing( pg, get_target_oid(), osd_reqid_t()); @@ -94,10 +72,8 @@ InternalClientRequest::with_interruption() std::make_error_code(std::errc::operation_canceled), fmt::format("{} is unfound, drop it!", get_target_oid())); } - co_await enter_stage<interruptor>( - client_pp().check_already_complete_get_obc); - DEBUGI("{}: getting obc lock", *this); + DEBUGI("{}: generating ops", *this); auto osd_ops = create_osd_ops(); @@ -107,26 +83,37 @@ InternalClientRequest::with_interruption() std::as_const(osd_ops), pg->get_pgid().pgid, *pg->get_osdmap()); assert(ret == 0); - auto obc_manager = pg->obc_loader.get_obc_manager(get_target_oid()); - - // initiate load_and_lock in order, but wait concurrently - enter_stage_sync(client_pp().lock_obc); - co_await pg->obc_loader.load_and_lock( obc_manager, pg->get_lock_type(op_info) ).handle_error_interruptible( crimson::ct_error::assert_all("unexpected error") ); - DEBUGDPP("{}: got obc {}, entering process stage", - *pg, *this, obc_manager.get_obc()->obs); - co_await enter_stage<interruptor>(client_pp().process); + auto params = get_do_osd_ops_params(); + OpsExecuter ox( + pg, obc_manager.get_obc(), op_info, params, params.get_connection(), + SnapContext{}); + co_await pg->run_executer( + ox, obc_manager.get_obc(), op_info, osd_ops + ).handle_error_interruptible( + crimson::ct_error::all_same_way( + [this, FNAME](auto e) { + ERRORDPPI("{}: got unexpected error {}", *pg, *this, e); + ceph_assert(0 == "should not return an error"); + return interruptor::now(); + }) + ); + + auto [submitted, completed] = co_await pg->submit_executer( + std::move(ox), osd_ops); + + co_await std::move(submitted); - DEBUGDPP("{}: in process stage, calling do_process", - *pg, *this); - co_await do_process(obc_manager.get_obc(), osd_ops); + co_await enter_stage<interruptor>(obc_orderer->obc_pp().wait_repop); + + co_await std::move(completed); - logger().debug("{}: complete", *this); + DEBUGDPP("{}: complete", *pg, *this); co_await interruptor::make_interruptible(handle.complete()); co_return; } @@ -148,7 +135,7 @@ seastar::future<> InternalClientRequest::start() return seastar::now(); }).finally([this] { logger().debug("{}: exit", *this); - handle.exit(); + return handle.complete(); }); } diff --git a/src/crimson/osd/osd_operations/internal_client_request.h b/src/crimson/osd/osd_operations/internal_client_request.h index 6023db0a8db..1cfde4ab080 100644 --- a/src/crimson/osd/osd_operations/internal_client_request.h +++ b/src/crimson/osd/osd_operations/internal_client_request.h @@ -4,6 +4,7 @@ #pragma once #include "crimson/common/type_helpers.h" +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osd_operation.h" #include "crimson/osd/osd_operations/client_request_common.h" #include "crimson/osd/pg.h" @@ -45,11 +46,10 @@ private: crimson::osd::ObjectContextRef obc, std::vector<OSDOp> &osd_ops); - seastar::future<> do_process(); - Ref<PG> pg; epoch_t start_epoch; OpInfo op_info; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; public: @@ -57,12 +57,8 @@ public: std::tuple< StartEvent, - CommonPGPipeline::WaitForActive::BlockingEvent, - PGActivationBlocker::BlockingEvent, - CommonPGPipeline::RecoverMissing::BlockingEvent, - CommonPGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - CommonPGPipeline::LockOBC::BlockingEvent, - CommonPGPipeline::Process::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, CompletionEvent > tracking_events; }; diff --git a/src/crimson/osd/osd_operations/logmissing_request.cc b/src/crimson/osd/osd_operations/logmissing_request.cc index 8147c969260..274744cdd92 100644 --- a/src/crimson/osd/osd_operations/logmissing_request.cc +++ b/src/crimson/osd/osd_operations/logmissing_request.cc @@ -58,9 +58,9 @@ PerShardPipeline &LogMissingRequest::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &LogMissingRequest::client_pp(PG &pg) +PGRepopPipeline &LogMissingRequest::repop_pipeline(PG &pg) { - return pg.request_pg_pipeline; + return pg.repop_pipeline; } seastar::future<> LogMissingRequest::with_pg( @@ -73,7 +73,7 @@ seastar::future<> LogMissingRequest::with_pg( return interruptor::with_interruption([this, pg] { LOG_PREFIX(LogMissingRequest::with_pg); DEBUGI("{}: pg present", *this); - return this->template enter_stage<interruptor>(client_pp(*pg).await_map + return this->template enter_stage<interruptor>(repop_pipeline(*pg).process ).then_interruptible([this, pg] { return this->template with_blocking_event< PG_OSDMapGate::OSDMapBlocker::BlockingEvent diff --git a/src/crimson/osd/osd_operations/logmissing_request.h b/src/crimson/osd/osd_operations/logmissing_request.h index 51c9d540cb5..fe4761c4ab4 100644 --- a/src/crimson/osd/osd_operations/logmissing_request.h +++ b/src/crimson/osd/osd_operations/logmissing_request.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -77,14 +80,14 @@ public: ConnectionPipeline::AwaitMap::BlockingEvent, ConnectionPipeline::GetPGMapping::BlockingEvent, PerShardPipeline::CreateOrWaitPG::BlockingEvent, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent, + PGRepopPipeline::Process::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, PGMap::PGCreationBlockingEvent, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); + PGRepopPipeline &repop_pipeline(PG &pg); crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/logmissing_request_reply.cc b/src/crimson/osd/osd_operations/logmissing_request_reply.cc index fb122a95cd1..5640610bd01 100644 --- a/src/crimson/osd/osd_operations/logmissing_request_reply.cc +++ b/src/crimson/osd/osd_operations/logmissing_request_reply.cc @@ -56,11 +56,6 @@ PerShardPipeline &LogMissingRequestReply::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &LogMissingRequestReply::client_pp(PG &pg) -{ - return pg.request_pg_pipeline; -} - seastar::future<> LogMissingRequestReply::with_pg( ShardServices &shard_services, Ref<PG> pg) { diff --git a/src/crimson/osd/osd_operations/logmissing_request_reply.h b/src/crimson/osd/osd_operations/logmissing_request_reply.h index c741b41bd0f..bdb6c2ac6ac 100644 --- a/src/crimson/osd/osd_operations/logmissing_request_reply.h +++ b/src/crimson/osd/osd_operations/logmissing_request_reply.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -82,8 +85,6 @@ public: > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); - crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/peering_event.h b/src/crimson/osd/osd_operations/peering_event.h index 85de5c711d6..aa6b8a95a94 100644 --- a/src/crimson/osd/osd_operations/peering_event.h +++ b/src/crimson/osd/osd_operations/peering_event.h @@ -44,6 +44,10 @@ protected: float delay = 0; PGPeeringEvent evt; + epoch_t get_epoch_sent_at() const { + return evt.get_epoch_sent(); + } + const pg_shard_t get_from() const { return from; } @@ -84,6 +88,10 @@ public: evt(std::forward<Args>(args)...) {} + bool requires_pg() const final { + return evt.requires_pg; + } + void print(std::ostream &) const final; void dump_detail(ceph::Formatter* f) const final; seastar::future<> with_pg( diff --git a/src/crimson/osd/osd_operations/pg_advance_map.h b/src/crimson/osd/osd_operations/pg_advance_map.h index 43be7319545..21702f6ff4f 100644 --- a/src/crimson/osd/osd_operations/pg_advance_map.h +++ b/src/crimson/osd/osd_operations/pg_advance_map.h @@ -50,6 +50,10 @@ public: PGPeeringPipeline::Process::BlockingEvent > tracking_events; + epoch_t get_epoch_sent_at() const { + return to; + } + private: PGPeeringPipeline &peering_pp(PG &pg); }; diff --git a/src/crimson/osd/osd_operations/recovery_subrequest.h b/src/crimson/osd/osd_operations/recovery_subrequest.h index 17c2faf97ea..2fe8ff372b3 100644 --- a/src/crimson/osd/osd_operations/recovery_subrequest.h +++ b/src/crimson/osd/osd_operations/recovery_subrequest.h @@ -39,6 +39,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return m->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return m->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); diff --git a/src/crimson/osd/osd_operations/replicated_request.cc b/src/crimson/osd/osd_operations/replicated_request.cc index 5ca11e5dd15..ec607758c55 100644 --- a/src/crimson/osd/osd_operations/replicated_request.cc +++ b/src/crimson/osd/osd_operations/replicated_request.cc @@ -5,6 +5,7 @@ #include "common/Formatter.h" +#include "crimson/common/coroutine.h" #include "crimson/osd/osd.h" #include "crimson/osd/osd_connection_priv.h" #include "crimson/osd/osd_operation_external_tracking.h" @@ -58,39 +59,57 @@ PerShardPipeline &RepRequest::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &RepRequest::client_pp(PG &pg) +PGRepopPipeline &RepRequest::repop_pipeline(PG &pg) { - return pg.request_pg_pipeline; + return pg.repop_pipeline; +} + +RepRequest::interruptible_future<> RepRequest::with_pg_interruptible( + Ref<PG> pg) +{ + LOG_PREFIX(RepRequest::with_pg_interruptible); + DEBUGI("{}", *this); + co_await this->template enter_stage<interruptor>(repop_pipeline(*pg).process); + co_await interruptor::make_interruptible(this->template with_blocking_event< + PG_OSDMapGate::OSDMapBlocker::BlockingEvent + >([this, pg](auto &&trigger) { + return pg->osdmap_gate.wait_for_map( + std::move(trigger), req->min_epoch); + })); + + if (pg->can_discard_replica_op(*req)) { + co_return; + } + + auto [commit_fut, reply] = co_await pg->handle_rep_op(req); + + // Transitions from OrderedExclusive->OrderedConcurrent cannot block + this->template enter_stage_sync(repop_pipeline(*pg).wait_commit); + + co_await std::move(commit_fut); + + co_await this->template enter_stage<interruptor>( + repop_pipeline(*pg).send_reply); + + co_await interruptor::make_interruptible( + pg->shard_services.send_to_osd( + req->from.osd, std::move(reply), pg->get_osdmap_epoch()) + ); } seastar::future<> RepRequest::with_pg( ShardServices &shard_services, Ref<PG> pg) { LOG_PREFIX(RepRequest::with_pg); - DEBUGI("{}: RepRequest::with_pg", *this); + DEBUGI("{}", *this); IRef ref = this; return interruptor::with_interruption([this, pg] { - LOG_PREFIX(RepRequest::with_pg); - DEBUGI("{}: pg present", *this); - return this->template enter_stage<interruptor>(client_pp(*pg).await_map - ).then_interruptible([this, pg] { - return this->template with_blocking_event< - PG_OSDMapGate::OSDMapBlocker::BlockingEvent - >([this, pg](auto &&trigger) { - return pg->osdmap_gate.wait_for_map( - std::move(trigger), req->min_epoch); - }); - }).then_interruptible([this, pg] (auto) { - return pg->handle_rep_op(req); - }).then_interruptible([this] { - logger().debug("{}: complete", *this); - return handle.complete(); - }); + return with_pg_interruptible(pg); }, [](std::exception_ptr) { return seastar::now(); }, pg, pg->get_osdmap_epoch()).finally([this, ref=std::move(ref)] { logger().debug("{}: exit", *this); - handle.exit(); + return handle.complete(); }); } diff --git a/src/crimson/osd/osd_operations/replicated_request.h b/src/crimson/osd/osd_operations/replicated_request.h index ff5dea6d6db..c2494b3715f 100644 --- a/src/crimson/osd/osd_operations/replicated_request.h +++ b/src/crimson/osd/osd_operations/replicated_request.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -68,6 +71,9 @@ public: r_conn = make_local_shared_foreign(std::move(conn)); } + interruptible_future<> with_pg_interruptible( + Ref<PG> pg); + seastar::future<> with_pg( ShardServices &shard_services, Ref<PG> pg); @@ -77,14 +83,16 @@ public: ConnectionPipeline::AwaitMap::BlockingEvent, ConnectionPipeline::GetPGMapping::BlockingEvent, PerShardPipeline::CreateOrWaitPG::BlockingEvent, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent, + PGRepopPipeline::Process::BlockingEvent, + PGRepopPipeline::WaitCommit::BlockingEvent, + PGRepopPipeline::SendReply::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, PGMap::PGCreationBlockingEvent, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); + PGRepopPipeline &repop_pipeline(PG &pg); crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/scrub_events.h b/src/crimson/osd/osd_operations/scrub_events.h index 02a5d852bb7..8bed90e4c14 100644 --- a/src/crimson/osd/osd_operations/scrub_events.h +++ b/src/crimson/osd/osd_operations/scrub_events.h @@ -27,11 +27,11 @@ class RemoteScrubEventBaseT : public PhasedOperationT<T> { crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; - epoch_t epoch; spg_t pgid; protected: using interruptor = InterruptibleOperation::interruptor; + epoch_t epoch; template <typename U=void> using ifut = InterruptibleOperation::interruptible_future<U>; @@ -40,7 +40,7 @@ protected: public: RemoteScrubEventBaseT( crimson::net::ConnectionRef conn, epoch_t epoch, spg_t pgid) - : l_conn(std::move(conn)), epoch(epoch), pgid(pgid) {} + : l_conn(std::move(conn)), pgid(pgid), epoch(epoch) {} PGPeeringPipeline &get_peering_pipeline(PG &pg); @@ -117,6 +117,10 @@ public: : RemoteScrubEventBaseT<ScrubRequested>(std::forward<Args>(base_args)...), deep(deep) {} + epoch_t get_epoch_sent_at() const { + return epoch; + } + void print(std::ostream &out) const final { out << "(deep=" << deep << ")"; } @@ -141,6 +145,10 @@ public: ceph_assert(scrub::PGScrubber::is_scrub_message(*m)); } + epoch_t get_epoch_sent_at() const { + return epoch; + } + void print(std::ostream &out) const final { out << "(m=" << *m << ")"; } diff --git a/src/crimson/osd/osd_operations/snaptrim_event.cc b/src/crimson/osd/osd_operations/snaptrim_event.cc index 8cab6125682..f8fb7aef6f2 100644 --- a/src/crimson/osd/osd_operations/snaptrim_event.cc +++ b/src/crimson/osd/osd_operations/snaptrim_event.cc @@ -388,20 +388,24 @@ SnapTrimObjSubEvent::remove_or_update( SnapTrimObjSubEvent::snap_trim_obj_subevent_ret_t SnapTrimObjSubEvent::start() { + obc_orderer = pg->obc_loader.get_obc_orderer( + coid); + ceph_assert(pg->is_active_clean()); - auto exit_handle = seastar::defer([this] { - logger().debug("{}: exit", *this); - handle.exit(); + auto exit_handle = seastar::defer([this, opref = IRef(this)] { + logger().debug("{}: exit", *opref); + std::ignore = handle.complete().then([opref = std::move(opref)] {}); }); co_await enter_stage<interruptor>( - client_pp().check_already_complete_get_obc); + obc_orderer->obc_pp().process); logger().debug("{}: getting obc for {}", *this, coid); auto obc_manager = pg->obc_loader.get_obc_manager( + *obc_orderer, coid, false /* resolve_oid */); co_await pg->obc_loader.load_and_lock( @@ -411,43 +415,39 @@ SnapTrimObjSubEvent::start() crimson::ct_error::assert_all{"unexpected error in SnapTrimObjSubEvent"} ); - co_await process_and_submit( - obc_manager.get_head_obc(), obc_manager.get_obc() - ).handle_error_interruptible( - remove_or_update_iertr::pass_further{}, - crimson::ct_error::assert_all{"unexpected error in SnapTrimObjSubEvent"} - ); - - logger().debug("{}: completed", *this); - co_await interruptor::make_interruptible(handle.complete()); -} - -ObjectContextLoader::load_obc_iertr::future<> -SnapTrimObjSubEvent::process_and_submit(ObjectContextRef head_obc, - ObjectContextRef clone_obc) { - logger().debug("{}: got clone_obc={}", *this, clone_obc->get_oid()); + logger().debug("{}: got obc={}", *this, obc_manager.get_obc()->get_oid()); - co_await enter_stage<interruptor>(client_pp().process); - - logger().debug("{}: processing clone_obc={}", *this, clone_obc->get_oid()); - - auto txn = co_await remove_or_update(clone_obc, head_obc); - - auto [submitted, all_completed] = co_await pg->submit_transaction( - std::move(clone_obc), - nullptr, - std::move(txn), - std::move(osd_op_p), - std::move(log_entries) - ); + auto all_completed = interruptor::now(); + { + // as with PG::submit_executer, we need to build the pg log entries + // and submit the transaction atomically + co_await interruptor::make_interruptible(pg->submit_lock.lock()); + auto unlocker = seastar::defer([this] { + pg->submit_lock.unlock(); + }); - co_await std::move(submitted); + logger().debug("{}: calling remove_or_update obc={}", + *this, obc_manager.get_obc()->get_oid()); + + auto txn = co_await remove_or_update( + obc_manager.get_obc(), obc_manager.get_head_obc()); + + auto submitted = interruptor::now(); + std::tie(submitted, all_completed) = co_await pg->submit_transaction( + ObjectContextRef(obc_manager.get_obc()), + nullptr, + std::move(txn), + std::move(osd_op_p), + std::move(log_entries) + ); + co_await std::move(submitted); + } - co_await enter_stage<interruptor>(client_pp().wait_repop); + co_await enter_stage<interruptor>(obc_orderer->obc_pp().wait_repop); co_await std::move(all_completed); - co_return; + logger().debug("{}: completed", *this); } void SnapTrimObjSubEvent::print(std::ostream &lhs) const diff --git a/src/crimson/osd/osd_operations/snaptrim_event.h b/src/crimson/osd/osd_operations/snaptrim_event.h index 1164b3169d2..a2b4d357568 100644 --- a/src/crimson/osd/osd_operations/snaptrim_event.h +++ b/src/crimson/osd/osd_operations/snaptrim_event.h @@ -6,6 +6,7 @@ #include <iostream> #include <seastar/core/future.hh> +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osdmap_gate.h" #include "crimson/osd/osd_operation.h" #include "crimson/common/subop_blocker.h" @@ -112,10 +113,6 @@ public: private: object_stat_sum_t delta_stats; - ObjectContextLoader::load_obc_iertr::future<> process_and_submit( - ObjectContextRef head_obc, - ObjectContextRef clone_obc); - snap_trim_obj_subevent_ret_t remove_clone( ObjectContextRef obc, ObjectContextRef head_obc, @@ -158,6 +155,7 @@ private: } Ref<PG> pg; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; osd_op_params_t osd_op_p; const hobject_t coid; @@ -169,9 +167,8 @@ public: std::tuple< StartEvent, - CommonPGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - CommonPGPipeline::Process::BlockingEvent, - CommonPGPipeline::WaitRepop::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, CompletionEvent > tracking_events; }; diff --git a/src/crimson/osd/pg.cc b/src/crimson/osd/pg.cc index 1e2988efbbe..2746e730f2b 100644 --- a/src/crimson/osd/pg.cc +++ b/src/crimson/osd/pg.cc @@ -868,46 +868,18 @@ std::ostream& operator<<(std::ostream& os, const PG& pg) return os; } -void PG::mutate_object( - ObjectContextRef& obc, - ceph::os::Transaction& txn, - osd_op_params_t& osd_op_p) +void PG::enqueue_push_for_backfill( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers) { - if (obc->obs.exists) { - obc->obs.oi.prior_version = obc->obs.oi.version; - obc->obs.oi.version = osd_op_p.at_version; - if (osd_op_p.user_modify) - obc->obs.oi.user_version = osd_op_p.at_version.version; - obc->obs.oi.last_reqid = osd_op_p.req_id; - obc->obs.oi.mtime = osd_op_p.mtime; - obc->obs.oi.local_mtime = ceph_clock_now(); - - // object_info_t - { - ceph::bufferlist osv; - obc->obs.oi.encode_no_oid(osv, CEPH_FEATURES_ALL); - // TODO: get_osdmap()->get_features(CEPH_ENTITY_TYPE_OSD, nullptr)); - txn.setattr(coll_ref->get_cid(), ghobject_t{obc->obs.oi.soid}, OI_ATTR, osv); - } - - // snapset - if (obc->obs.oi.soid.snap == CEPH_NOSNAP) { - logger().debug("final snapset {} in {}", - obc->ssc->snapset, obc->obs.oi.soid); - ceph::bufferlist bss; - encode(obc->ssc->snapset, bss); - txn.setattr(coll_ref->get_cid(), ghobject_t{obc->obs.oi.soid}, SS_ATTR, bss); - obc->ssc->exists = true; - } else { - logger().debug("no snapset (this is a clone)"); - } - } else { - // reset cached ObjectState without enforcing eviction - obc->obs.oi = object_info_t(obc->obs.oi.soid); - } + assert(recovery_handler); + assert(recovery_handler->backfill_state); + auto backfill_state = recovery_handler->backfill_state.get(); + backfill_state->enqueue_standalone_push(obj, v, peers); } -void PG::enqueue_push_for_backfill( +void PG::enqueue_delete_for_backfill( const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers) @@ -915,7 +887,7 @@ void PG::enqueue_push_for_backfill( assert(recovery_handler); assert(recovery_handler->backfill_state); auto backfill_state = recovery_handler->backfill_state.get(); - backfill_state->enqueue_standalone_push(obj, v, peers); + backfill_state->enqueue_standalone_delete(obj, v, peers); } PG::interruptible_future< @@ -1039,8 +1011,15 @@ PG::interruptible_future<eversion_t> PG::submit_error_log( const std::error_code e, ceph_tid_t rep_tid) { - logger().debug("{}: {} rep_tid: {} error: {}", - __func__, *m, rep_tid, e); + // as with submit_executer, need to ensure that log numbering and submission + // are atomic + co_await interruptor::make_interruptible(submit_lock.lock()); + auto unlocker = seastar::defer([this] { + submit_lock.unlock(); + }); + LOG_PREFIX(PG::submit_error_log); + DEBUGDPP("{} rep_tid: {} error: {}", + *this, *m, rep_tid, e); const osd_reqid_t &reqid = m->get_reqid(); mempool::osd_pglog::list<pg_log_entry_t> log_entries; log_entries.push_back(pg_log_entry_t(pg_log_entry_t::ERROR, @@ -1061,47 +1040,45 @@ PG::interruptible_future<eversion_t> PG::submit_error_log( log_entries, t, peering_state.get_pg_trim_to(), peering_state.get_pg_committed_to()); - return seastar::do_with(log_entries, set<pg_shard_t>{}, - [this, t=std::move(t), rep_tid](auto& log_entries, auto& waiting_on) mutable { - return interruptor::do_for_each(get_acting_recovery_backfill(), - [this, log_entries, waiting_on, rep_tid] - (auto& i) mutable { - pg_shard_t peer(i); - if (peer == pg_whoami) { - return seastar::now(); - } - ceph_assert(peering_state.get_peer_missing().count(peer)); - ceph_assert(peering_state.has_peer_info(peer)); - auto log_m = crimson::make_message<MOSDPGUpdateLogMissing>( - log_entries, - spg_t(peering_state.get_info().pgid.pgid, i.shard), - pg_whoami.shard, - get_osdmap_epoch(), - get_last_peering_reset(), - rep_tid, - peering_state.get_pg_trim_to(), - peering_state.get_pg_committed_to()); - waiting_on.insert(peer); - logger().debug("submit_error_log: sending log" - "missing_request (rep_tid: {} entries: {})" - " to osd {}", rep_tid, log_entries, peer.osd); - return shard_services.send_to_osd(peer.osd, - std::move(log_m), - get_osdmap_epoch()); - }).then_interruptible([this, waiting_on, t=std::move(t), rep_tid] () mutable { - waiting_on.insert(pg_whoami); - logger().debug("submit_error_log: inserting rep_tid {}", rep_tid); - log_entry_update_waiting_on.insert( - std::make_pair(rep_tid, - log_update_t{std::move(waiting_on)})); - return shard_services.get_store().do_transaction( - get_collection_ref(), std::move(t) - ).then([this] { - peering_state.update_trim_to(); - return seastar::make_ready_future<eversion_t>(projected_last_update); - }); - }); - }); + + set<pg_shard_t> waiting_on; + for (const auto &peer: get_acting_recovery_backfill()) { + if (peer == pg_whoami) { + continue; + } + ceph_assert(peering_state.get_peer_missing().count(peer)); + ceph_assert(peering_state.has_peer_info(peer)); + auto log_m = crimson::make_message<MOSDPGUpdateLogMissing>( + log_entries, + spg_t(peering_state.get_info().pgid.pgid, peer.shard), + pg_whoami.shard, + get_osdmap_epoch(), + get_last_peering_reset(), + rep_tid, + peering_state.get_pg_trim_to(), + peering_state.get_pg_committed_to()); + waiting_on.insert(peer); + + DEBUGDPP("sending log missing_request (rep_tid: {} entries: {}) to osd {}", + *this, rep_tid, log_entries, peer.osd); + co_await interruptor::make_interruptible( + shard_services.send_to_osd( + peer.osd, + std::move(log_m), + get_osdmap_epoch())); + } + waiting_on.insert(pg_whoami); + DEBUGDPP("inserting rep_tid {}", *this, rep_tid); + log_entry_update_waiting_on.insert( + std::make_pair(rep_tid, + log_update_t{std::move(waiting_on)})); + co_await interruptor::make_interruptible( + shard_services.get_store().do_transaction( + get_collection_ref(), std::move(t) + )); + + peering_state.update_trim_to(); + co_return projected_last_update; } PG::run_executer_fut PG::run_executer( @@ -1157,27 +1134,25 @@ PG::submit_executer_fut PG::submit_executer( OpsExecuter &&ox, const std::vector<OSDOp>& ops) { LOG_PREFIX(PG::submit_executer); - // transaction must commit at this point - return std::move( + DEBUGDPP("", *this); + + // we need to build the pg log entries and submit the transaction + // atomically to ensure log ordering + co_await interruptor::make_interruptible(submit_lock.lock()); + auto unlocker = seastar::defer([this] { + submit_lock.unlock(); + }); + + auto [submitted, completed] = co_await std::move( ox - ).flush_changes_n_do_ops_effects( + ).flush_changes_and_submit( ops, snap_mapper, - osdriver, - [FNAME, this](auto&& txn, - auto&& obc, - auto&& osd_op_p, - auto&& log_entries, - auto&& new_clone) { - DEBUGDPP("object {} submitting txn", *this, obc->get_oid()); - mutate_object(obc, txn, osd_op_p); - return submit_transaction( - std::move(obc), - std::move(new_clone), - std::move(txn), - std::move(osd_op_p), - std::move(log_entries)); - }); + osdriver + ); + co_return std::make_tuple( + std::move(submitted).then_interruptible([unlocker=std::move(unlocker)] {}), + std::move(completed)); } PG::interruptible_future<MURef<MOSDOpReply>> PG::do_pg_ops(Ref<MOSDOp> m) @@ -1251,13 +1226,10 @@ void PG::update_stats(const pg_stat_t &stat) { ); } -PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) +PG::handle_rep_op_fut PG::handle_rep_op(Ref<MOSDRepOp> req) { LOG_PREFIX(PG::handle_rep_op); DEBUGDPP("{}", *this, *req); - if (can_discard_replica_op(*req)) { - co_return; - } ceph::os::Transaction txn; auto encoded_txn = req->get_data().cbegin(); @@ -1279,7 +1251,8 @@ PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) txn, false); DEBUGDPP("{} do_transaction", *this, *req); - co_await interruptor::make_interruptible( + + auto commit_fut = interruptor::make_interruptible( shard_services.get_store().do_transaction(coll_ref, std::move(txn)) ); @@ -1290,10 +1263,7 @@ PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) req.get(), pg_whoami, 0, map_epoch, req->get_min_epoch(), CEPH_OSD_FLAG_ONDISK); reply->set_last_complete_ondisk(lcod); - co_await interruptor::make_interruptible( - shard_services.send_to_osd(req->from.osd, std::move(reply), map_epoch) - ); - co_return; + co_return handle_rep_op_ret(std::move(commit_fut), std::move(reply)); } PG::interruptible_future<> PG::update_snap_map( @@ -1324,20 +1294,21 @@ void PG::log_operation( bool transaction_applied, ObjectStore::Transaction &txn, bool async) { - logger().debug("{}", __func__); + LOG_PREFIX(PG::log_operation); + DEBUGDPP("", *this); if (is_primary()) { ceph_assert(trim_to <= peering_state.get_pg_committed_to()); } auto last = logv.rbegin(); if (is_primary() && last != logv.rend()) { - logger().debug("{} on primary, trimming projected log", - __func__); + DEBUGDPP("on primary, trimming projected log", *this); projected_log.skip_can_rollback_to_to_head(); projected_log.trim(shard_services.get_cct(), last->version, nullptr, nullptr, nullptr); } if (!is_primary()) { // && !is_ec_pg() + DEBUGDPP("on replica, clearing obc", *this); replica_clear_repop_obc(logv); } if (!logv.empty()) { @@ -1354,13 +1325,13 @@ void PG::log_operation( void PG::replica_clear_repop_obc( const std::vector<pg_log_entry_t> &logv) { - logger().debug("{} clearing {} entries", __func__, logv.size()); - for (auto &&e: logv) { - logger().debug(" {} get_object_boundary(from): {} " - " head version(to): {}", - e.soid, - e.soid.get_object_boundary(), - e.soid.get_head()); + LOG_PREFIX(PG::replica_clear_repop_obc); + DEBUGDPP("clearing obc for {} log entries", logv.size()); + for (auto &&e: logv) { + DEBUGDPP("clearing entry for {} from: {} to: {}", + e.soid, + e.soid.get_object_boundary(), + e.soid.get_head()); /* Have to blast all clones, they share a snapset */ obc_registry.clear_range( e.soid.get_object_boundary(), e.soid.get_head()); diff --git a/src/crimson/osd/pg.h b/src/crimson/osd/pg.h index 15aeec0e4f3..06038c0aa00 100644 --- a/src/crimson/osd/pg.h +++ b/src/crimson/osd/pg.h @@ -10,6 +10,7 @@ #include <seastar/core/shared_future.hh> #include "common/dout.h" +#include "common/ostream_temp.h" #include "include/interval_set.h" #include "crimson/net/Fwd.h" #include "messages/MOSDRepOpReply.h" @@ -77,7 +78,8 @@ class PG : public boost::intrusive_ref_counter< using ec_profile_t = std::map<std::string,std::string>; using cached_map_t = OSDMapService::cached_map_t; - ClientRequest::PGPipeline request_pg_pipeline; + CommonPGPipeline request_pg_pipeline; + PGRepopPipeline repop_pipeline; PGPeeringPipeline peering_request_pg_pipeline; ClientRequest::Orderer client_request_orderer; @@ -594,7 +596,13 @@ public: using with_obc_func_t = std::function<load_obc_iertr::future<> (ObjectContextRef, ObjectContextRef)>; - interruptible_future<> handle_rep_op(Ref<MOSDRepOp> m); + using handle_rep_op_ret = std::tuple< + interruptible_future<>, // resolves upon commit + MURef<MOSDRepOpReply> // reply message + >; + // outer future resolves upon submission + using handle_rep_op_fut = interruptible_future<handle_rep_op_ret>; + handle_rep_op_fut handle_rep_op(Ref<MOSDRepOp> m); void update_stats(const pg_stat_t &stat); interruptible_future<> update_snap_map( const std::vector<pg_log_entry_t> &log_entries, @@ -663,6 +671,7 @@ private: const OpInfo &op_info, std::vector<OSDOp>& ops); + seastar::shared_mutex submit_lock; using submit_executer_ret = std::tuple< interruptible_future<>, interruptible_future<>>; @@ -675,6 +684,8 @@ private: struct do_osd_ops_params_t; interruptible_future<MURef<MOSDOpReply>> do_pg_ops(Ref<MOSDOp> m); + +public: interruptible_future< std::tuple<interruptible_future<>, interruptible_future<>>> submit_transaction( @@ -683,6 +694,8 @@ private: ceph::os::Transaction&& txn, osd_op_params_t&& oop, std::vector<pg_log_entry_t>&& log_entries); + +private: interruptible_future<> repair_object( const hobject_t& oid, eversion_t& v); @@ -891,10 +904,11 @@ private: const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers); - void mutate_object( - ObjectContextRef& obc, - ceph::os::Transaction& txn, - osd_op_params_t& osd_op_p); + void enqueue_delete_for_backfill( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers); + bool can_discard_replica_op(const Message& m, epoch_t m_map_epoch) const; bool can_discard_op(const MOSDOp& m) const; void context_registry_on_change(); diff --git a/src/crimson/osd/pg_backend.cc b/src/crimson/osd/pg_backend.cc index 24a381b4cf7..79895de06de 100644 --- a/src/crimson/osd/pg_backend.cc +++ b/src/crimson/osd/pg_backend.cc @@ -1283,22 +1283,6 @@ PGBackend::rm_xattr( return rm_xattr_iertr::now(); } -void PGBackend::clone( - /* const */object_info_t& snap_oi, - const ObjectState& os, - const ObjectState& d_os, - ceph::os::Transaction& txn) -{ - // See OpsExecuter::execute_clone documentation - txn.clone(coll->get_cid(), ghobject_t{os.oi.soid}, ghobject_t{d_os.oi.soid}); - { - ceph::bufferlist bv; - snap_oi.encode_no_oid(bv, CEPH_FEATURES_ALL); - txn.setattr(coll->get_cid(), ghobject_t{d_os.oi.soid}, OI_ATTR, bv); - } - txn.rmattr(coll->get_cid(), ghobject_t{d_os.oi.soid}, SS_ATTR); -} - using get_omap_ertr = crimson::os::FuturizedStore::Shard::read_errorator::extend< crimson::ct_error::enodata>; @@ -1341,9 +1325,10 @@ maybe_get_omap_vals( PGBackend::ll_read_ierrorator::future<ceph::bufferlist> PGBackend::omap_get_header( const crimson::os::CollectionRef& c, - const ghobject_t& oid) const + const ghobject_t& oid, + uint32_t op_flags) const { - return store->omap_get_header(c, oid) + return store->omap_get_header(c, oid, op_flags) .handle_error( crimson::ct_error::enodata::handle([] { return seastar::make_ready_future<bufferlist>(); @@ -1356,10 +1341,13 @@ PGBackend::ll_read_ierrorator::future<> PGBackend::omap_get_header( const ObjectState& os, OSDOp& osd_op, - object_stat_sum_t& delta_stats) const + object_stat_sum_t& delta_stats, + uint32_t op_flags) const { if (os.oi.is_omap()) { - return omap_get_header(coll, ghobject_t{os.oi.soid}).safe_then_interruptible( + return omap_get_header( + coll, ghobject_t{os.oi.soid}, CEPH_OSD_OP_FLAG_FADVISE_DONTNEED + ).safe_then_interruptible( [&delta_stats, &osd_op] (ceph::bufferlist&& header) { osd_op.outdata = std::move(header); delta_stats.num_rd_kb += shift_round_up(osd_op.outdata.length(), 10); @@ -1723,7 +1711,8 @@ PGBackend::fiemap( CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { return store->fiemap(c, oid, off, len); } @@ -1835,3 +1824,32 @@ PGBackend::read_ierrorator::future<> PGBackend::tmapget( read_errorator::pass_further{}); } +void PGBackend::set_metadata( + const hobject_t &obj, + object_info_t &oi, + const SnapSet *ss /* non-null iff head */, + ceph::os::Transaction& txn) +{ + ceph_assert((obj.is_head() && ss) || (!obj.is_head() && !ss)); + { + ceph::bufferlist bv; + oi.encode_no_oid(bv, CEPH_FEATURES_ALL); + txn.setattr(coll->get_cid(), ghobject_t{obj}, OI_ATTR, bv); + } + if (ss) { + ceph::bufferlist bss; + encode(*ss, bss); + txn.setattr(coll->get_cid(), ghobject_t{obj}, SS_ATTR, bss); + } +} + +void PGBackend::clone_for_write( + const hobject_t &from, + const hobject_t &to, + ceph::os::Transaction &txn) +{ + // See OpsExecuter::execute_clone documentation + txn.clone(coll->get_cid(), ghobject_t{from}, ghobject_t{to}); + txn.rmattr(coll->get_cid(), ghobject_t{to}, SS_ATTR); +} + diff --git a/src/crimson/osd/pg_backend.h b/src/crimson/osd/pg_backend.h index 813218983fd..9c2230375b0 100644 --- a/src/crimson/osd/pg_backend.h +++ b/src/crimson/osd/pg_backend.h @@ -308,11 +308,6 @@ public: ObjectState& os, const OSDOp& osd_op, ceph::os::Transaction& trans); - void clone( - /* const */object_info_t& snap_oi, - const ObjectState& os, - const ObjectState& d_os, - ceph::os::Transaction& trans); interruptible_future<struct stat> stat( CollectionRef c, const ghobject_t& oid) const; @@ -320,7 +315,8 @@ public: CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len); + uint64_t len, + uint32_t op_flags = 0); write_iertr::future<> tmapput( ObjectState& os, @@ -380,11 +376,13 @@ public: object_stat_sum_t& delta_stats); ll_read_ierrorator::future<ceph::bufferlist> omap_get_header( const crimson::os::CollectionRef& c, - const ghobject_t& oid) const; + const ghobject_t& oid, + uint32_t op_flags = 0) const; ll_read_ierrorator::future<> omap_get_header( const ObjectState& os, OSDOp& osd_op, - object_stat_sum_t& delta_stats) const; + object_stat_sum_t& delta_stats, + uint32_t op_flags = 0) const; interruptible_future<> omap_set_header( ObjectState& os, const OSDOp& osd_op, @@ -411,6 +409,20 @@ public: ceph::os::Transaction& trans, osd_op_params_t& osd_op_params, object_stat_sum_t& delta_stats); + + /// sets oi and (for head) ss attrs + void set_metadata( + const hobject_t &obj, + object_info_t &oi, + const SnapSet *ss /* non-null iff head */, + ceph::os::Transaction& trans); + + /// clone from->to and clear ss attribute on to + void clone_for_write( + const hobject_t &from, + const hobject_t &to, + ceph::os::Transaction& trans); + virtual rep_op_fut_t submit_transaction(const std::set<pg_shard_t> &pg_shards, const hobject_t& hoid, diff --git a/src/crimson/osd/pg_recovery.cc b/src/crimson/osd/pg_recovery.cc index ec3af0d2b00..5eef584c776 100644 --- a/src/crimson/osd/pg_recovery.cc +++ b/src/crimson/osd/pg_recovery.cc @@ -67,8 +67,6 @@ PGRecovery::start_recovery_ops( if (max_to_start > 0) { max_to_start -= start_replica_recovery_ops(trigger, max_to_start, &started); } - using interruptor = - crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>; return interruptor::parallel_for_each(started, [] (auto&& ifut) { return std::move(ifut); @@ -609,8 +607,21 @@ void PGRecovery::update_peers_last_backfill( bool PGRecovery::budget_available() const { - // TODO: the limits! - return true; + crimson::osd::scheduler::params_t params = + {1, 0, crimson::osd::scheduler::scheduler_class_t::background_best_effort}; + auto &ss = pg->get_shard_services(); + auto futopt = ss.try_acquire_throttle_now(std::move(params)); + if (!futopt) { + return true; + } + std::ignore = interruptor::make_interruptible(std::move(*futopt) + ).then_interruptible([this] { + assert(!backfill_state->is_triggered()); + using BackfillState = crimson::osd::BackfillState; + backfill_state->process_event( + BackfillState::ThrottleAcquired{}.intrusive_from_this()); + }); + return false; } void PGRecovery::on_pg_clean() diff --git a/src/crimson/osd/pg_recovery.h b/src/crimson/osd/pg_recovery.h index 657e6d3e888..5c7b5c5ef2b 100644 --- a/src/crimson/osd/pg_recovery.h +++ b/src/crimson/osd/pg_recovery.h @@ -25,6 +25,8 @@ class PGBackend; class PGRecovery : public crimson::osd::BackfillState::BackfillListener { public: + using interruptor = + crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>; template <typename T = void> using interruptible_future = RecoveryBackend::interruptible_future<T>; PGRecovery(PGRecoveryListener* pg) : pg(pg) {} diff --git a/src/crimson/osd/pg_shard_manager.h b/src/crimson/osd/pg_shard_manager.h index b9879c8c9dd..f7bd7a6c08e 100644 --- a/src/crimson/osd/pg_shard_manager.h +++ b/src/crimson/osd/pg_shard_manager.h @@ -256,18 +256,40 @@ public: auto &opref = *op; return opref.template with_blocking_event< PGMap::PGCreationBlockingEvent - >([&target_shard_services, &opref](auto &&trigger) { - return target_shard_services.wait_for_pg( - std::move(trigger), opref.get_pgid()); - }).safe_then([&logger, &target_shard_services, &opref](Ref<PG> pgref) { - logger.debug("{}: have_pg", opref); - return opref.with_pg(target_shard_services, pgref); - }).handle_error( - crimson::ct_error::ecanceled::handle([&logger, &opref](auto) { - logger.debug("{}: pg creation canceled, dropping", opref); - return seastar::now(); - }) - ).then([op=std::move(op)] {}); + >([&target_shard_services, &opref, &logger](auto &&trigger) mutable { + auto pg = target_shard_services.get_pg(opref.get_pgid()); + auto fut = ShardServices::wait_for_pg_ertr::make_ready_future<Ref<PG>>(pg); + if (!pg) { + if (opref.requires_pg()) { + auto osdmap = target_shard_services.get_map(); + if (!osdmap->is_up_acting_osd_shard( + opref.get_pgid(), target_shard_services.local_state.whoami)) { + logger.debug( + "pg {} for {} is no longer here, discarding", + opref.get_pgid(), opref); + opref.get_handle().exit(); + auto _fut = seastar::now(); + if (osdmap->get_epoch() > opref.get_epoch_sent_at()) { + _fut = target_shard_services.send_incremental_map( + std::ref(opref.get_foreign_connection()), + opref.get_epoch_sent_at() + 1); + } + return _fut; + } + } + fut = target_shard_services.wait_for_pg( + std::move(trigger), opref.get_pgid()); + } + return fut.safe_then([&logger, &target_shard_services, &opref](Ref<PG> pgref) { + logger.debug("{}: have_pg", opref); + return opref.with_pg(target_shard_services, pgref); + }).handle_error( + crimson::ct_error::ecanceled::handle([&logger, &opref](auto) { + logger.debug("{}: pg creation canceled, dropping", opref); + return seastar::now(); + }) + ); + }).then([op=std::move(op)] {}); } seastar::future<> load_pgs(crimson::os::FuturizedStore& store); diff --git a/src/crimson/osd/replicated_backend.cc b/src/crimson/osd/replicated_backend.cc index f09cd147ea9..6c8abecffaf 100644 --- a/src/crimson/osd/replicated_backend.cc +++ b/src/crimson/osd/replicated_backend.cc @@ -96,11 +96,18 @@ ReplicatedBackend::submit_transaction( bufferlist encoded_txn; encode(txn, encoded_txn); + bool is_delete = false; for (auto &le : log_entries) { le.mark_unrollbackable(); + if (le.is_delete()) { + is_delete = true; + } } + co_await pg.update_snap_map(log_entries, txn); + std::vector<pg_shard_t> to_push_clone; + std::vector<pg_shard_t> to_push_delete; auto sends = std::make_unique<std::vector<seastar::future<>>>(); for (auto &pg_shard : pg_shards) { if (pg_shard == whoami) { @@ -115,12 +122,17 @@ ReplicatedBackend::submit_transaction( m = new_repop_msg( pg_shard, hoid, encoded_txn, osd_op_p, min_epoch, map_epoch, log_entries, false, tid); - if (_new_clone && pg.is_missing_on_peer(pg_shard, hoid)) { - // The head is in the push queue but hasn't been pushed yet. - // We need to ensure that the newly created clone will be - // pushed as well, otherwise we might skip it. - // See: https://tracker.ceph.com/issues/68808 - to_push_clone.push_back(pg_shard); + if (pg.is_missing_on_peer(pg_shard, hoid)) { + if (_new_clone) { + // The head is in the push queue but hasn't been pushed yet. + // We need to ensure that the newly created clone will be + // pushed as well, otherwise we might skip it. + // See: https://tracker.ceph.com/issues/68808 + to_push_clone.push_back(pg_shard); + } + if (is_delete) { + to_push_delete.push_back(pg_shard); + } } } pending_txn->second.acked_peers.push_back({pg_shard, eversion_t{}}); @@ -130,8 +142,6 @@ ReplicatedBackend::submit_transaction( pg_shard.osd, std::move(m), map_epoch)); } - co_await pg.update_snap_map(log_entries, txn); - pg.log_operation( std::move(log_entries), osd_op_p.pg_trim_to, @@ -157,7 +167,8 @@ ReplicatedBackend::submit_transaction( return seastar::now(); } return peers->all_committed.get_shared_future(); - }).then_interruptible([pending_txn, this, _new_clone, + }).then_interruptible([pending_txn, this, _new_clone, &hoid, + to_push_delete=std::move(to_push_delete), to_push_clone=std::move(to_push_clone)] { auto acked_peers = std::move(pending_txn->second.acked_peers); pending_trans.erase(pending_txn); @@ -167,6 +178,9 @@ ReplicatedBackend::submit_transaction( _new_clone->obs.oi.version, to_push_clone); } + if (!to_push_delete.empty()) { + pg.enqueue_delete_for_backfill(hoid, {}, to_push_delete); + } return seastar::make_ready_future< crimson::osd::acked_peers_t>(std::move(acked_peers)); }); diff --git a/src/crimson/osd/replicated_recovery_backend.cc b/src/crimson/osd/replicated_recovery_backend.cc index 76f24196b51..0d6c9d38236 100644 --- a/src/crimson/osd/replicated_recovery_backend.cc +++ b/src/crimson/osd/replicated_recovery_backend.cc @@ -35,6 +35,15 @@ ReplicatedRecoveryBackend::recover_object( logger().debug("recover_object: loading obc: {}", soid); return pg.obc_loader.with_obc<RWState::RWREAD>(soid, [this, soid, need](auto head, auto obc) { + if (!obc->obs.exists) { + // XXX: this recovery must be triggered by backfills and the corresponding + // object must have been deleted by some client request after the object + // is enqueued for push but before the lock is acquired by the recovery. + // + // Abort the recovery in this case, a "recover_delete" must have been + // added for this object by the client request that deleted it. + return interruptor::now(); + } logger().debug("recover_object: loaded obc: {}", obc->obs.oi.soid); auto& recovery_waiter = get_recovering(soid); recovery_waiter.obc = obc; @@ -306,7 +315,10 @@ ReplicatedRecoveryBackend::recover_delete( } return seastar::make_ready_future<>(); }).then_interruptible([this, soid, &stat_diff] { - pg.get_recovery_handler()->on_global_recover(soid, stat_diff, true); + const auto &missing = pg.get_peering_state().get_pg_log().get_missing(); + if (!missing.is_missing(soid)) { + pg.get_recovery_handler()->on_global_recover(soid, stat_diff, true); + } return seastar::make_ready_future<>(); }); }); @@ -568,14 +580,17 @@ ReplicatedRecoveryBackend::read_metadata_for_push_op( return seastar::make_ready_future<eversion_t>(ver); } return interruptor::make_interruptible(interruptor::when_all_succeed( - backend->omap_get_header(coll, ghobject_t(oid)).handle_error_interruptible<false>( + backend->omap_get_header( + coll, ghobject_t(oid), CEPH_OSD_OP_FLAG_FADVISE_DONTNEED + ).handle_error_interruptible<false>( crimson::os::FuturizedStore::Shard::read_errorator::all_same_way( [oid] (const std::error_code& e) { logger().debug("read_metadata_for_push_op, error {} when getting omap header: {}", e, oid); return seastar::make_ready_future<bufferlist>(); })), - interruptor::make_interruptible(store->get_attrs(coll, ghobject_t(oid))) - .handle_error_interruptible<false>( + interruptor::make_interruptible( + store->get_attrs(coll, ghobject_t(oid), CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) + ).handle_error_interruptible<false>( crimson::os::FuturizedStore::Shard::get_attrs_ertr::all_same_way( [oid] (const std::error_code& e) { logger().debug("read_metadata_for_push_op, error {} when getting attrs: {}", e, oid); @@ -613,8 +628,14 @@ ReplicatedRecoveryBackend::read_object_for_push_op( return seastar::make_ready_future<uint64_t>(offset); } // 1. get the extents in the interested range - return interruptor::make_interruptible(backend->fiemap(coll, ghobject_t{oid}, - 0, copy_subset.range_end())).safe_then_interruptible( + return interruptor::make_interruptible( + backend->fiemap( + coll, + ghobject_t{oid}, + 0, + copy_subset.range_end(), + CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) + ).safe_then_interruptible( [=, this](auto&& fiemap_included) mutable { interval_set<uint64_t> extents; try { @@ -630,8 +651,12 @@ ReplicatedRecoveryBackend::read_object_for_push_op( push_op->data_included.span_of(extents, offset, max_len); // 3. read the truncated extents // TODO: check if the returned extents are pruned - return interruptor::make_interruptible(store->readv(coll, ghobject_t{oid}, - push_op->data_included, 0)); + return interruptor::make_interruptible( + store->readv( + coll, + ghobject_t{oid}, + push_op->data_included, + CEPH_OSD_OP_FLAG_FADVISE_DONTNEED)); }).safe_then_interruptible([push_op, range_end=copy_subset.range_end()](auto &&bl) { push_op->data.claim_append(std::move(bl)); uint64_t recovered_to = 0; diff --git a/src/crimson/osd/shard_services.cc b/src/crimson/osd/shard_services.cc index c2340898929..e1acb34636f 100644 --- a/src/crimson/osd/shard_services.cc +++ b/src/crimson/osd/shard_services.cc @@ -783,6 +783,11 @@ seastar::future<> ShardServices::dispatch_context_transaction( co_return; } +Ref<PG> ShardServices::get_pg(spg_t pgid) +{ + return local_state.get_pg(pgid); +} + seastar::future<> ShardServices::dispatch_context_messages( BufferedRecoveryMessages &&ctx) { diff --git a/src/crimson/osd/shard_services.h b/src/crimson/osd/shard_services.h index fb86418aba2..f1ed9b8d911 100644 --- a/src/crimson/osd/shard_services.h +++ b/src/crimson/osd/shard_services.h @@ -10,6 +10,7 @@ #include "include/common_fwd.h" #include "osd_operation.h" +#include "osd/osd_types_fmt.h" #include "msg/MessageRef.h" #include "crimson/common/exception.h" #include "crimson/common/shared_lru.h" @@ -482,6 +483,8 @@ public: return pg_to_shard_mapping.remove_pg_mapping(pgid); } + Ref<PG> get_pg(spg_t pgid); + crimson::common::CephContext *get_cct() { return &(local_state.cct); } @@ -588,6 +591,7 @@ public: FORWARD_TO_OSD_SINGLETON(get_pool_info) FORWARD(with_throttle_while, with_throttle_while, local_state.throttler) + FORWARD(try_acquire_throttle_now, try_acquire_throttle_now, local_state.throttler) FORWARD_TO_OSD_SINGLETON(build_incremental_map_msg) FORWARD_TO_OSD_SINGLETON(send_incremental_map) diff --git a/src/crimson/tools/perf_crimson_msgr.cc b/src/crimson/tools/perf_crimson_msgr.cc index e5f56361fff..5623438f821 100644 --- a/src/crimson/tools/perf_crimson_msgr.cc +++ b/src/crimson/tools/perf_crimson_msgr.cc @@ -1,6 +1,7 @@ // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab +#include <iomanip> #include <map> #include <boost/program_options.hpp> #include <boost/iterator/counting_iterator.hpp> diff --git a/src/crimson/tools/store_nbd/tm_driver.cc b/src/crimson/tools/store_nbd/tm_driver.cc index 389ecd78afc..870809c5153 100644 --- a/src/crimson/tools/store_nbd/tm_driver.cc +++ b/src/crimson/tools/store_nbd/tm_driver.cc @@ -25,6 +25,7 @@ seastar::future<> TMDriver::write( return tm->with_transaction_intr( Transaction::src_t::MUTATE, "write", + CACHE_HINT_TOUCH, [this, offset, &ptr](auto& t) { return tm->remove(t, laddr_t::from_byte_offset(offset) @@ -112,6 +113,7 @@ seastar::future<bufferlist> TMDriver::read( return tm->with_transaction_intr( Transaction::src_t::READ, "read", + CACHE_HINT_TOUCH, [=, &blret, this](auto& t) { return read_extents(t, laddr_t::from_byte_offset(offset), size diff --git a/src/exporter/ceph_exporter.cc b/src/exporter/ceph_exporter.cc index 44b67c7e615..2232851c094 100644 --- a/src/exporter/ceph_exporter.cc +++ b/src/exporter/ceph_exporter.cc @@ -30,13 +30,13 @@ static void handle_signal(int signum) static void usage() { std::cout << "usage: ceph-exporter [options]\n" << "options:\n" - " --sock-dir: The path to ceph daemons socket files dir\n" - " --addrs: Host ip address where exporter is deployed\n" - " --port: Port to deploy exporter on. Default is 9926\n" - " --cert-file: Path to the certificate file to use https\n" - " --key-file: Path to the certificate key file to use https\n" + " --sock-dir: The path to Ceph daemon sockets (*.asok)\n" + " --addrs: Host IP address on which the exporter is to listen\n" + " --port: TCP Port on which the exporter is to listen. Default is 9926\n" + " --cert-file: Path to the certificate file when using HTTPS\n" + " --key-file: Path to the certificate key file when using HTTPS\n" " --prio-limit: Only perf counters greater than or equal to prio-limit are fetched. Default: 5\n" - " --stats-period: Time to wait before sending requests again to exporter server (seconds). Default: 5s" + " --stats-period: Interval between daemon scrapes (seconds). Default: 5s" << std::endl; generic_server_usage(); } diff --git a/src/include/cephfs/ceph_ll_client.h b/src/include/cephfs/ceph_ll_client.h index ac5b7c22471..458edce590b 100644 --- a/src/include/cephfs/ceph_ll_client.h +++ b/src/include/cephfs/ceph_ll_client.h @@ -110,6 +110,9 @@ struct ceph_statx { * others in the future, we disallow setting any that aren't recognized. */ #define CEPH_REQ_FLAG_MASK (AT_SYMLINK_NOFOLLOW|AT_STATX_DONT_SYNC) +#if defined(__linux__) && defined(AT_EMPTY_PATH) +#define CEPH_AT_EMPTY_PATH (CEPH_REQ_FLAG_MASK|AT_EMPTY_PATH) +#endif /* fallocate mode flags */ #ifndef FALLOC_FL_KEEP_SIZE diff --git a/src/include/cephfs/libcephfs.h b/src/include/cephfs/libcephfs.h index ba0b76e072b..f26eabfbd5c 100644 --- a/src/include/cephfs/libcephfs.h +++ b/src/include/cephfs/libcephfs.h @@ -937,7 +937,7 @@ int ceph_fstatx(struct ceph_mount_info *cmount, int fd, struct ceph_statx *stx, * @param relpath to the file/directory to get statistics of * @param stx the ceph_statx struct that will be filled in with the file's statistics. * @param want bitfield of CEPH_STATX_* flags showing designed attributes - * @param flags bitfield that can be used to set AT_* modifier flags (AT_STATX_SYNC_AS_STAT, AT_STATX_FORCE_SYNC, AT_STATX_DONT_SYNC and AT_SYMLINK_NOFOLLOW) + * @param flags bitfield that can be used to set AT_* modifier flags (AT_STATX_DONT_SYNC, AT_SYMLINK_NOFOLLOW and AT_EMPTY_PATH) * @returns 0 on success or negative error code on failure. */ int ceph_statxat(struct ceph_mount_info *cmount, int dirfd, const char *relpath, @@ -1104,7 +1104,7 @@ int ceph_lchown(struct ceph_mount_info *cmount, const char *path, int uid, int g * @param relpath the relpath of the file/directory to change the ownership of. * @param uid the user id to set on the file/directory. * @param gid the group id to set on the file/directory. - * @param flags bitfield that can be used to set AT_* modifier flags (AT_SYMLINK_NOFOLLOW) + * @param flags bitfield that can be used to set AT_* modifier flags (AT_SYMLINK_NOFOLLOW and AT_EMPTY_PATH) * @returns 0 on success or negative error code on failure. */ int ceph_chownat(struct ceph_mount_info *cmount, int dirfd, const char *relpath, diff --git a/src/include/rados/librados.hpp b/src/include/rados/librados.hpp index 4a7ac3ea6e0..0f5a9036eff 100644 --- a/src/include/rados/librados.hpp +++ b/src/include/rados/librados.hpp @@ -202,6 +202,8 @@ inline namespace v14_2_0 { int set_complete_callback(void *cb_arg, callback_t cb); int set_safe_callback(void *cb_arg, callback_t cb) __attribute__ ((deprecated)); + /// Request immediate cancellation as if by IoCtx::aio_cancel(). + int cancel(); int wait_for_complete(); int wait_for_safe() __attribute__ ((deprecated)); int wait_for_complete_and_cb(); @@ -772,17 +774,30 @@ inline namespace v14_2_0 { void tier_evict(); }; - /* IoCtx : This is a context in which we can perform I/O. - * It includes a Pool, + /** + * @brief A handle to a RADOS pool used to perform I/O operations. * * Typical use (error checking omitted): - * + * @code * IoCtx p; * rados.ioctx_create("my_pool", p); - * p->stat(&stats); - * ... etc ... + * p.stat("my_object", &size, &mtime); + * @endcode + * + * IoCtx holds a pointer to its underlying implementation. The dup() + * method performs a deep copy of this implementation, but the copy + * construction and assignment operations perform shallow copies by + * sharing that pointer. + * + * Function names starting with aio_ are asynchronous operations that + * return immediately after submitting a request, and whose completions + * are managed by the given AioCompletion pointer. The IoCtx's underlying + * implementation is involved in the delivery of these completions, so + * the caller must guarantee that its lifetime is preserved until then - + * if not by preserving the IoCtx instance that submitted the request, + * then by a copied/moved instance that shares the same implementation. * - * NOTE: be sure to call watch_flush() prior to destroying any IoCtx + * @note Be sure to call watch_flush() prior to destroying any IoCtx * that is used for watch events to ensure that racing callbacks * have completed. */ @@ -791,9 +806,13 @@ inline namespace v14_2_0 { public: IoCtx(); static void from_rados_ioctx_t(rados_ioctx_t p, IoCtx &pool); + /// Construct a shallow copy of rhs, sharing its underlying implementation. IoCtx(const IoCtx& rhs); + /// Assign a shallow copy of rhs, sharing its underlying implementation. IoCtx& operator=(const IoCtx& rhs); + /// Move construct from rhs, transferring its underlying implementation. IoCtx(IoCtx&& rhs) noexcept; + /// Move assign from rhs, transferring its underlying implementation. IoCtx& operator=(IoCtx&& rhs) noexcept; ~IoCtx(); @@ -1150,7 +1169,8 @@ inline namespace v14_2_0 { int aio_stat2(const std::string& oid, AioCompletion *c, uint64_t *psize, struct timespec *pts); /** - * Cancel aio operation + * Request immediate cancellation with error code -ECANCELED + * if the operation hasn't already completed. * * @param c completion handle * @returns 0 on success, negative error code on failure diff --git a/src/include/random.h b/src/include/random.h index f2e3e37bcd7..6b7c9405efd 100644 --- a/src/include/random.h +++ b/src/include/random.h @@ -16,9 +16,9 @@ #define CEPH_RANDOM_H 1 #include <mutex> +#include <optional> #include <random> #include <type_traits> -#include <boost/optional.hpp> // Workaround for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85494 #ifdef __MINGW32__ @@ -123,7 +123,7 @@ void randomize_rng() template <typename EngineT> EngineT& engine() { - thread_local boost::optional<EngineT> rng_engine; + thread_local std::optional<EngineT> rng_engine; if (!rng_engine) { rng_engine.emplace(EngineT()); diff --git a/src/kv/KeyValueDB.h b/src/kv/KeyValueDB.h index 858742d511e..d926840180e 100644 --- a/src/kv/KeyValueDB.h +++ b/src/kv/KeyValueDB.h @@ -9,6 +9,7 @@ #include <map> #include <optional> #include <string> +#include <string_view> #include <boost/scoped_ptr.hpp> #include "include/encoding.h" #include "common/Formatter.h" @@ -211,6 +212,10 @@ public: return ""; } virtual ceph::buffer::list value() = 0; + // When valid() returns true, value returned as string-view + // is guaranteed to be valid until iterator is moved to another + // position; that is until call to next() / seek_to_first() / etc. + virtual std::string_view value_as_sv() = 0; virtual int status() = 0; virtual ~SimplestIteratorImpl() {} }; @@ -220,7 +225,12 @@ public: virtual ~IteratorImpl() {} virtual int seek_to_last() = 0; virtual int prev() = 0; + // When valid() returns true, key returned as string-view + // is guaranteed to be valid until iterator is moved to another + // position; that is until call to next() / seek_to_first() / etc. + virtual std::string_view key_as_sv() = 0; virtual std::pair<std::string, std::string> raw_key() = 0; + virtual std::pair<std::string_view, std::string_view> raw_key_as_sv() = 0; virtual ceph::buffer::ptr value_as_ptr() { ceph::buffer::list bl = value(); if (bl.length() == 1) { @@ -247,7 +257,9 @@ public: virtual int next() = 0; virtual int prev() = 0; virtual std::string key() = 0; + virtual std::string_view key_as_sv() = 0; virtual std::pair<std::string,std::string> raw_key() = 0; + virtual std::pair<std::string_view, std::string_view> raw_key_as_sv() = 0; virtual bool raw_key_is_prefixed(const std::string &prefix) = 0; virtual ceph::buffer::list value() = 0; virtual ceph::buffer::ptr value_as_ptr() { @@ -258,6 +270,7 @@ public: return ceph::buffer::ptr(); } } + virtual std::string_view value_as_sv() = 0; virtual int status() = 0; virtual size_t key_size() { return 0; @@ -315,15 +328,24 @@ private: std::string key() override { return generic_iter->key(); } + std::string_view key_as_sv() override { + return generic_iter->key_as_sv(); + } std::pair<std::string, std::string> raw_key() override { return generic_iter->raw_key(); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return generic_iter->raw_key_as_sv(); + } ceph::buffer::list value() override { return generic_iter->value(); } ceph::buffer::ptr value_as_ptr() override { return generic_iter->value_as_ptr(); } + std::string_view value_as_sv() override { + return generic_iter->value_as_sv(); + } int status() override { return generic_iter->status(); } diff --git a/src/kv/RocksDBStore.cc b/src/kv/RocksDBStore.cc index ca63ea06484..51d224b67c0 100644 --- a/src/kv/RocksDBStore.cc +++ b/src/kv/RocksDBStore.cc @@ -6,6 +6,7 @@ #include <memory> #include <set> #include <string> +#include <string_view> #include <errno.h> #include <unistd.h> #include <sys/types.h> @@ -47,6 +48,7 @@ using std::ostream; using std::pair; using std::set; using std::string; +using std::string_view; using std::unique_ptr; using std::vector; @@ -1992,7 +1994,7 @@ int RocksDBStore::split_key(rocksdb::Slice in, string *prefix, string *key) // Find separator inside Slice char* separator = (char*) memchr(in.data(), 0, in.size()); - if (separator == NULL) + if (separator == nullptr) return -EINVAL; prefix_len = size_t(separator - in.data()); if (prefix_len >= in.size()) @@ -2006,6 +2008,27 @@ int RocksDBStore::split_key(rocksdb::Slice in, string *prefix, string *key) return 0; } +// TODO: deduplicate the code, preferrably by removing the string variant +int RocksDBStore::split_key(rocksdb::Slice in, string_view *prefix, string_view *key) +{ + size_t prefix_len = 0; + + // Find separator inside Slice + char* separator = (char*) memchr(in.data(), 0, in.size()); + if (separator == nullptr) + return -EINVAL; + prefix_len = size_t(separator - in.data()); + if (prefix_len >= in.size()) + return -EINVAL; + + // Fetch prefix and/or key directly from Slice + if (prefix) + *prefix = string_view(in.data(), prefix_len); + if (key) + *key = string_view(separator + 1, in.size() - prefix_len - 1); + return 0; +} + void RocksDBStore::compact() { dout(2) << __func__ << " starting" << dendl; @@ -2226,7 +2249,13 @@ int RocksDBStore::RocksDBWholeSpaceIteratorImpl::prev() string RocksDBStore::RocksDBWholeSpaceIteratorImpl::key() { string out_key; - split_key(dbiter->key(), 0, &out_key); + split_key(dbiter->key(), nullptr, &out_key); + return out_key; +} +string_view RocksDBStore::RocksDBWholeSpaceIteratorImpl::key_as_sv() +{ + string_view out_key; + split_key(dbiter->key(), nullptr, &out_key); return out_key; } pair<string,string> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key() @@ -2235,6 +2264,12 @@ pair<string,string> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key() split_key(dbiter->key(), &prefix, &key); return make_pair(prefix, key); } +pair<string_view,string_view> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key_as_sv() +{ + string_view prefix, key; + split_key(dbiter->key(), &prefix, &key); + return make_pair(prefix, key); +} bool RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key_is_prefixed(const string &prefix) { // Look for "prefix\0" right in rocksb::Slice @@ -2267,6 +2302,12 @@ bufferptr RocksDBStore::RocksDBWholeSpaceIteratorImpl::value_as_ptr() return bufferptr(val.data(), val.size()); } +std::string_view RocksDBStore::RocksDBWholeSpaceIteratorImpl::value_as_sv() +{ + rocksdb::Slice val = dbiter->value(); + return std::string_view{val.data(), val.size()}; +} + int RocksDBStore::RocksDBWholeSpaceIteratorImpl::status() { return dbiter->status().ok() ? 0 : -1; @@ -2348,9 +2389,15 @@ public: string key() override { return dbiter->key().ToString(); } + string_view key_as_sv() override { + return dbiter->key().ToStringView(); + } std::pair<std::string, std::string> raw_key() override { return make_pair(prefix, key()); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return make_pair(prefix, dbiter->key().ToStringView()); + } bufferlist value() override { return to_bufferlist(dbiter->value()); } @@ -2358,6 +2405,10 @@ public: rocksdb::Slice val = dbiter->value(); return bufferptr(val.data(), val.size()); } + std::string_view value_as_sv() override { + rocksdb::Slice val = dbiter->value(); + return std::string_view{val.data(), val.size()}; + } int status() override { return dbiter->status().ok() ? 0 : -1; } @@ -2668,6 +2719,15 @@ public: } } + std::string_view key_as_sv() override + { + if (smaller == on_main) { + return main->key_as_sv(); + } else { + return current_shard->second->key_as_sv(); + } + } + std::pair<std::string,std::string> raw_key() override { if (smaller == on_main) { @@ -2677,6 +2737,15 @@ public: } } + std::pair<std::string_view,std::string_view> raw_key_as_sv() override + { + if (smaller == on_main) { + return main->raw_key_as_sv(); + } else { + return { current_shard->first, current_shard->second->key_as_sv() }; + } + } + bool raw_key_is_prefixed(const std::string &prefix) override { if (smaller == on_main) { @@ -2695,6 +2764,15 @@ public: } } + std::string_view value_as_sv() override + { + if (smaller == on_main) { + return main->value_as_sv(); + } else { + return current_shard->second->value_as_sv(); + } + } + int status() override { //because we already had to inspect key, it must be ok @@ -3017,9 +3095,15 @@ public: string key() override { return iters[0]->key().ToString(); } + string_view key_as_sv() override { + return iters[0]->key().ToStringView(); + } std::pair<std::string, std::string> raw_key() override { return make_pair(prefix, key()); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return make_pair(prefix, iters[0]->key().ToStringView()); + } bufferlist value() override { return to_bufferlist(iters[0]->value()); } @@ -3027,6 +3111,10 @@ public: rocksdb::Slice val = iters[0]->value(); return bufferptr(val.data(), val.size()); } + std::string_view value_as_sv() override { + rocksdb::Slice val = iters[0]->value(); + return std::string_view{val.data(), val.size()}; + } int status() override { return iters[0]->status().ok() ? 0 : -1; } diff --git a/src/kv/RocksDBStore.h b/src/kv/RocksDBStore.h index 477b209854c..50b91be2bf6 100644 --- a/src/kv/RocksDBStore.h +++ b/src/kv/RocksDBStore.h @@ -386,10 +386,13 @@ public: int next() override; int prev() override; std::string key() override; + std::string_view key_as_sv() override; std::pair<std::string,std::string> raw_key() override; + std::pair<std::string_view,std::string_view> raw_key_as_sv() override; bool raw_key_is_prefixed(const std::string &prefix) override; ceph::bufferlist value() override; ceph::bufferptr value_as_ptr() override; + std::string_view value_as_sv() override; int status() override; size_t key_size() override; size_t value_size() override; @@ -419,6 +422,7 @@ public: } static int split_key(rocksdb::Slice in, std::string *prefix, std::string *key); + static int split_key(rocksdb::Slice in, std::string_view *prefix, std::string_view *key); static std::string past_prefix(const std::string &prefix); diff --git a/src/libcephfs.cc b/src/libcephfs.cc index 7eea6665f61..60da6145787 100644 --- a/src/libcephfs.cc +++ b/src/libcephfs.cc @@ -982,7 +982,11 @@ extern "C" int ceph_statxat(struct ceph_mount_info *cmount, int dirfd, const cha { if (!cmount->is_mounted()) return -CEPHFS_ENOTCONN; +#ifdef CEPH_AT_EMPTY_PATH + if (flags & ~CEPH_AT_EMPTY_PATH) +#else if (flags & ~CEPH_REQ_FLAG_MASK) +#endif return -CEPHFS_EINVAL; return cmount->get_client()->statxat(dirfd, relpath, stx, cmount->default_perms, want, flags); diff --git a/src/libcephfs_proxy/CMakeLists.txt b/src/libcephfs_proxy/CMakeLists.txt new file mode 100644 index 00000000000..e19841241e7 --- /dev/null +++ b/src/libcephfs_proxy/CMakeLists.txt @@ -0,0 +1,18 @@ +set(proxy_common_srcs proxy_link.c proxy_log.c) +set(libcephfsd_srcs libcephfsd.c proxy_manager.c proxy_mount.c proxy_helpers.c ${proxy_common_srcs}) +set(libcephfs_proxy_srcs libcephfs_proxy.c ${proxy_common_srcs}) + +add_executable(libcephfsd ${libcephfsd_srcs}) +add_library(cephfs_proxy ${CEPH_SHARED} ${libcephfs_proxy_srcs}) + +target_link_libraries(libcephfsd cephfs ${CRYPTO_LIBS}) + +if(ENABLE_SHARED) + set_target_properties(cephfs_proxy PROPERTIES + OUTPUT_NAME cephfs_proxy + VERSION 2.0.0 + SOVERSION 2) +endif(ENABLE_SHARED) + +install(TARGETS libcephfsd DESTINATION ${CMAKE_INSTALL_SBINDIR}) +install(TARGETS cephfs_proxy DESTINATION ${CMAKE_INSTALL_LIBDIR}) diff --git a/src/libcephfs_proxy/libcephfs_proxy.c b/src/libcephfs_proxy/libcephfs_proxy.c new file mode 100644 index 00000000000..149fae123f7 --- /dev/null +++ b/src/libcephfs_proxy/libcephfs_proxy.c @@ -0,0 +1,869 @@ + +#include <stdlib.h> + +#include "include/cephfs/libcephfs.h" + +#include "proxy_log.h" +#include "proxy_helpers.h" +#include "proxy_requests.h" + +/* We override the definition of the ceph_mount_info structure to contain + * internal proxy information. This is already a black box for libcephfs users, + * so this won't be noticed. */ +struct ceph_mount_info { + proxy_link_t link; + uint64_t cmount; +}; + +/* The global_cmount is used to stablish an initial connection to serve requests + * not related to a real cmount, like ceph_version or ceph_userperm_new. */ +static struct ceph_mount_info global_cmount = { PROXY_LINK_DISCONNECTED, 0 }; + +static bool client_stop(proxy_link_t *link) +{ + return false; +} + +static int32_t proxy_connect(proxy_link_t *link) +{ + CEPH_REQ(hello, req, 0, ans, 0); + char *path, *env; + int32_t sd, err; + + path = PROXY_SOCKET; + env = getenv(PROXY_SOCKET_ENV); + if (env != NULL) { + path = env; + } + + sd = proxy_link_client(link, path, client_stop); + if (sd < 0) { + return sd; + } + + req.id = LIBCEPHFS_LIB_CLIENT; + err = proxy_link_send(sd, req_iov, 1); + if (err < 0) { + goto failed; + } + err = proxy_link_recv(sd, ans_iov, 1); + if (err < 0) { + goto failed; + } + + proxy_log(LOG_INFO, 0, "Connected to libcephfsd version %d.%d", + ans.major, ans.minor); + + if ((ans.major != LIBCEPHFSD_MAJOR) || + (ans.minor != LIBCEPHFSD_MINOR)) { + err = proxy_log(LOG_ERR, ENOTSUP, "Version not supported"); + goto failed; + } + + return sd; + +failed: + proxy_link_close(link); + + return err; +} + +static void proxy_disconnect(proxy_link_t *link) +{ + proxy_link_close(link); +} + +static int32_t proxy_global_connect(void) +{ + int32_t err; + + err = 0; + + if (!proxy_link_is_connected(&global_cmount.link)) { + err = proxy_connect(&global_cmount.link); + } + + return err; +} + +static int32_t proxy_check(struct ceph_mount_info *cmount, int32_t err, + int32_t result) +{ + if (err < 0) { + proxy_disconnect(&cmount->link); + proxy_log(LOG_ERR, err, "Disconnected from libcephfsd"); + + return err; + } + + return result; +} + +/* Macros to simplify communication with the server. */ +#define CEPH_RUN(_cmount, _op, _req, _ans) \ + ({ \ + int32_t __err = \ + CEPH_CALL((_cmount)->link.sd, _op, _req, _ans); \ + __err = proxy_check(_cmount, __err, (_ans).header.result); \ + __err; \ + }) + +#define CEPH_PROCESS(_cmount, _op, _req, _ans) \ + ({ \ + int32_t __err = -ENOTCONN; \ + if (proxy_link_is_connected(&(_cmount)->link)) { \ + (_req).cmount = (_cmount)->cmount; \ + __err = CEPH_RUN(_cmount, _op, _req, _ans); \ + } \ + __err; \ + }) + +__public int ceph_chdir(struct ceph_mount_info *cmount, const char *path) +{ + CEPH_REQ(ceph_chdir, req, 1, ans, 0); + + CEPH_STR_ADD(req, path, path); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CHDIR, req, ans); +} + +__public int ceph_conf_get(struct ceph_mount_info *cmount, const char *option, + char *buf, size_t len) +{ + CEPH_REQ(ceph_conf_get, req, 1, ans, 1); + + req.size = len; + + CEPH_STR_ADD(req, option, option); + CEPH_BUFF_ADD(ans, buf, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_GET, req, ans); +} + +__public int ceph_conf_read_file(struct ceph_mount_info *cmount, + const char *path_list) +{ + CEPH_REQ(ceph_conf_read_file, req, 1, ans, 0); + + CEPH_STR_ADD(req, path, path_list); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_READ_FILE, req, ans); +} + +__public int ceph_conf_set(struct ceph_mount_info *cmount, const char *option, + const char *value) +{ + CEPH_REQ(ceph_conf_set, req, 2, ans, 0); + + CEPH_STR_ADD(req, option, option); + CEPH_STR_ADD(req, value, value); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_SET, req, ans); +} + +__public int ceph_create(struct ceph_mount_info **cmount, const char *const id) +{ + CEPH_REQ(ceph_create, req, 1, ans, 0); + struct ceph_mount_info *ceph_mount; + int32_t sd, err; + + ceph_mount = proxy_malloc(sizeof(struct ceph_mount_info)); + if (ceph_mount == NULL) { + return -ENOMEM; + } + + err = proxy_connect(&ceph_mount->link); + if (err < 0) { + goto failed; + } + sd = err; + + CEPH_STR_ADD(req, id, id); + + err = CEPH_CALL(sd, LIBCEPHFSD_OP_CREATE, req, ans); + if ((err < 0) || ((err = ans.header.result) < 0)) { + goto failed_link; + } + + ceph_mount->cmount = ans.cmount; + + *cmount = ceph_mount; + + return 0; + +failed_link: + proxy_disconnect(&ceph_mount->link); + +failed: + proxy_free(ceph_mount); + + return err; +} + +__public const char *ceph_getcwd(struct ceph_mount_info *cmount) +{ + static char cwd[PATH_MAX]; + int32_t err; + + CEPH_REQ(ceph_getcwd, req, 0, ans, 1); + + CEPH_BUFF_ADD(ans, cwd, sizeof(cwd)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_GETCWD, req, ans); + if (err >= 0) { + return cwd; + } + + errno = -err; + + return NULL; +} + +__public int ceph_init(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_init, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_INIT, req, ans); +} + +__public int ceph_ll_close(struct ceph_mount_info *cmount, + struct Fh *filehandle) +{ + CEPH_REQ(ceph_ll_close, req, 0, ans, 0); + + req.fh = ptr_value(filehandle); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_CLOSE, req, ans); +} + +__public int ceph_ll_create(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, int oflags, + Inode **outp, Fh **fhp, struct ceph_statx *stx, + unsigned want, unsigned lflags, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_create, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.oflags = oflags; + req.want = want; + req.flags = lflags; + + CEPH_STR_ADD(req, name, name); + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_CREATE, req, ans); + if (err >= 0) { + *outp = value_ptr(ans.inode); + *fhp = value_ptr(ans.fh); + } + + return err; +} + +__public int ceph_ll_fallocate(struct ceph_mount_info *cmount, struct Fh *fh, + int mode, int64_t offset, int64_t length) +{ + CEPH_REQ(ceph_ll_fallocate, req, 0, ans, 0); + + req.fh = ptr_value(fh); + req.mode = mode; + req.offset = offset; + req.length = length; + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_FALLOCATE, req, ans); +} + +__public int ceph_ll_fsync(struct ceph_mount_info *cmount, struct Fh *fh, + int syncdataonly) +{ + CEPH_REQ(ceph_ll_fsync, req, 0, ans, 0); + + req.fh = ptr_value(fh); + req.dataonly = syncdataonly; + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_FSYNC, req, ans); +} + +__public int ceph_ll_getattr(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_statx *stx, unsigned int want, + unsigned int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_getattr, req, 0, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.want = want; + req.flags = flags; + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_GETATTR, req, ans); +} + +__public int ceph_ll_getxattr(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, void *value, size_t size, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_getxattr, req, 1, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = size; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, value, size); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_GETXATTR, req, ans); +} + +__public int ceph_ll_link(struct ceph_mount_info *cmount, struct Inode *in, + struct Inode *newparent, const char *name, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_link, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.parent = ptr_value(newparent); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LINK, req, ans); +} + +__public int ceph_ll_listxattr(struct ceph_mount_info *cmount, struct Inode *in, + char *list, size_t buf_size, size_t *list_size, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_listxattr, req, 0, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = buf_size; + + CEPH_BUFF_ADD(ans, list, buf_size); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LISTXATTR, req, ans); + if (err >= 0) { + *list_size = ans.size; + } + + return err; +} + +__public int ceph_ll_lookup(struct ceph_mount_info *cmount, Inode *parent, + const char *name, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_lookup, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_lookup_inode(struct ceph_mount_info *cmount, + struct inodeno_t ino, Inode **inode) +{ + CEPH_REQ(ceph_ll_lookup_inode, req, 0, ans, 0); + int32_t err; + + req.ino = ino; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP_INODE, req, ans); + if (err >= 0) { + *inode = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_lookup_root(struct ceph_mount_info *cmount, Inode **parent) +{ + CEPH_REQ(ceph_ll_lookup_root, req, 0, ans, 0); + int32_t err; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP_ROOT, req, ans); + if (err >= 0) { + *parent = value_ptr(ans.inode); + } + + return err; +} + +__public off_t ceph_ll_lseek(struct ceph_mount_info *cmount, + struct Fh *filehandle, off_t offset, int whence) +{ + CEPH_REQ(ceph_ll_lseek, req, 0, ans, 0); + int32_t err; + + req.fh = ptr_value(filehandle); + req.offset = offset; + req.whence = whence; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LSEEK, req, ans); + if (err >= 0) { + return ans.offset; + } + + return err; +} + +__public int ceph_ll_mkdir(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_mkdir, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_MKDIR, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_mknod(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, dev_t rdev, + Inode **out, struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_mknod, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.rdev = rdev; + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_MKNOD, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_open(struct ceph_mount_info *cmount, struct Inode *in, + int flags, struct Fh **fh, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_open, req, 0, ans, 0); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.flags = flags; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_OPEN, req, ans); + if (err >= 0) { + *fh = value_ptr(ans.fh); + } + + return err; +} + +__public int ceph_ll_opendir(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_dir_result **dirpp, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_opendir, req, 0, ans, 0); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_OPENDIR, req, ans); + if (err >= 0) { + *dirpp = value_ptr(ans.dir); + } + + return err; +} + +__public int ceph_ll_put(struct ceph_mount_info *cmount, struct Inode *in) +{ + CEPH_REQ(ceph_ll_put, req, 0, ans, 0); + + req.inode = ptr_value(in); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_PUT, req, ans); +} + +__public int ceph_ll_read(struct ceph_mount_info *cmount, struct Fh *filehandle, + int64_t off, uint64_t len, char *buf) +{ + CEPH_REQ(ceph_ll_read, req, 0, ans, 1); + + req.fh = ptr_value(filehandle); + req.offset = off; + req.len = len; + + CEPH_BUFF_ADD(ans, buf, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_READ, req, ans); +} + +__public int ceph_ll_readlink(struct ceph_mount_info *cmount, struct Inode *in, + char *buf, size_t bufsize, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_readlink, req, 0, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = bufsize; + + CEPH_BUFF_ADD(ans, buf, bufsize); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_READLINK, req, ans); +} + +__public int ceph_ll_releasedir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dir) +{ + CEPH_REQ(ceph_ll_releasedir, req, 0, ans, 0); + + req.dir = ptr_value(dir); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RELEASEDIR, req, ans); +} + +__public int ceph_ll_removexattr(struct ceph_mount_info *cmount, + struct Inode *in, const char *name, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_removexattr, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_REMOVEXATTR, req, ans); +} + +__public int ceph_ll_rename(struct ceph_mount_info *cmount, + struct Inode *parent, const char *name, + struct Inode *newparent, const char *newname, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_rename, req, 2, ans, 0); + + req.userperm = ptr_value(perms); + req.old_parent = ptr_value(parent); + req.new_parent = ptr_value(newparent); + CEPH_STR_ADD(req, old_name, name); + CEPH_STR_ADD(req, new_name, newname); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RENAME, req, ans); +} + +__public void ceph_rewinddir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dirp) +{ + CEPH_REQ(ceph_rewinddir, req, 0, ans, 0); + + req.dir = ptr_value(dirp); + + CEPH_PROCESS(cmount, LIBCEPHFSD_OP_REWINDDIR, req, ans); +} + +__public int ceph_ll_rmdir(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_rmdir, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RMDIR, req, ans); +} + +__public int ceph_ll_setattr(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_statx *stx, int mask, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_setattr, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.mask = mask; + CEPH_BUFF_ADD(req, stx, sizeof(*stx)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SETATTR, req, ans); +} + +__public int ceph_ll_setxattr(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const void *value, size_t size, + int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_setxattr, req, 2, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = size; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + CEPH_BUFF_ADD(req, value, size); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SETXATTR, req, ans); +} + +__public int ceph_ll_statfs(struct ceph_mount_info *cmount, struct Inode *in, + struct statvfs *stbuf) +{ + CEPH_REQ(ceph_ll_statfs, req, 0, ans, 1); + + req.inode = ptr_value(in); + + CEPH_BUFF_ADD(ans, stbuf, sizeof(*stbuf)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_STATFS, req, ans); +} + +__public int ceph_ll_symlink(struct ceph_mount_info *cmount, Inode *in, + const char *name, const char *value, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_symlink, req, 2, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + CEPH_STR_ADD(req, target, value); + + CEPH_BUFF_ADD(req, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SYMLINK, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_unlink(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_unlink, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_UNLINK, req, ans); +} + +__public int ceph_ll_walk(struct ceph_mount_info *cmount, const char *name, + Inode **i, struct ceph_statx *stx, unsigned int want, + unsigned int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_walk, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, path, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_WALK, req, ans); + if (err >= 0) { + *i = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_write(struct ceph_mount_info *cmount, + struct Fh *filehandle, int64_t off, uint64_t len, + const char *data) +{ + CEPH_REQ(ceph_ll_write, req, 1, ans, 0); + + req.fh = ptr_value(filehandle); + req.offset = off; + req.len = len; + CEPH_BUFF_ADD(req, data, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_WRITE, req, ans); +} + +__public int ceph_mount(struct ceph_mount_info *cmount, const char *root) +{ + CEPH_REQ(ceph_mount, req, 1, ans, 0); + + CEPH_STR_ADD(req, root, root); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_MOUNT, req, ans); +} + +__public struct dirent *ceph_readdir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dirp) +{ + static struct dirent de; + int32_t err; + + CEPH_REQ(ceph_readdir, req, 0, ans, 1); + + req.dir = ptr_value(dirp); + + CEPH_BUFF_ADD(ans, &de, sizeof(de)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_READDIR, req, ans); + if (err < 0) { + errno = -err; + return NULL; + } + if (ans.eod) { + return NULL; + } + + return &de; +} + +__public int ceph_release(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_release, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_RELEASE, req, ans); +} + +__public int ceph_select_filesystem(struct ceph_mount_info *cmount, + const char *fs_name) +{ + CEPH_REQ(ceph_select_filesystem, req, 1, ans, 0); + + CEPH_STR_ADD(req, fs, fs_name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_SELECT_FILESYSTEM, req, ans); +} + +__public int ceph_unmount(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_unmount, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_UNMOUNT, req, ans); +} + +__public void ceph_userperm_destroy(UserPerm *perms) +{ + CEPH_REQ(ceph_userperm_destroy, req, 0, ans, 0); + + req.userperm = ptr_value(perms); + + CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_USERPERM_DESTROY, req, ans); +} + +__public UserPerm *ceph_userperm_new(uid_t uid, gid_t gid, int ngids, + gid_t *gidlist) +{ + CEPH_REQ(ceph_userperm_new, req, 1, ans, 0); + int32_t err; + + req.uid = uid; + req.gid = gid; + req.groups = ngids; + CEPH_BUFF_ADD(req, gidlist, sizeof(gid_t) * ngids); + + err = proxy_global_connect(); + if (err >= 0) { + err = CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_USERPERM_NEW, req, + ans); + } + if (err >= 0) { + return value_ptr(ans.userperm); + } + + errno = -err; + + return NULL; +} + +__public const char *ceph_version(int *major, int *minor, int *patch) +{ + static char cached_version[128]; + static int32_t cached_major = -1, cached_minor, cached_patch; + + if (cached_major < 0) { + CEPH_REQ(ceph_version, req, 0, ans, 1); + int32_t err; + + CEPH_BUFF_ADD(ans, cached_version, sizeof(cached_version)); + + err = proxy_global_connect(); + if (err >= 0) { + err = CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_VERSION, + req, ans); + } + + if (err < 0) { + *major = 0; + *minor = 0; + *patch = 0; + + return "Unknown"; + } + + cached_major = ans.major; + cached_minor = ans.minor; + cached_patch = ans.patch; + } + + *major = cached_major; + *minor = cached_minor; + *patch = cached_patch; + + return cached_version; +} + +__public UserPerm *ceph_mount_perms(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_mount_perms, req, 0, ans, 0); + int32_t err; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_MOUNT_PERMS, req, ans); + if (err < 0) { + errno = -err; + return NULL; + } + + return value_ptr(ans.userperm); +} diff --git a/src/libcephfs_proxy/libcephfsd.c b/src/libcephfs_proxy/libcephfsd.c new file mode 100644 index 00000000000..ee2d99a0aae --- /dev/null +++ b/src/libcephfs_proxy/libcephfsd.c @@ -0,0 +1,1823 @@ + +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> +#include <getopt.h> +#include <endian.h> + +#include "include/cephfs/libcephfs.h" + +#include "proxy_manager.h" +#include "proxy_link.h" +#include "proxy_helpers.h" +#include "proxy_log.h" +#include "proxy_requests.h" +#include "proxy_mount.h" + +typedef struct _proxy_server { + proxy_link_t link; + proxy_manager_t *manager; +} proxy_server_t; + +typedef struct _proxy_client { + proxy_worker_t worker; + proxy_link_t *link; + proxy_random_t random; + void *buffer; + uint32_t buffer_size; + int32_t sd; +} proxy_client_t; + +typedef struct _proxy { + proxy_manager_t manager; + proxy_log_handler_t log_handler; + const char *socket_path; +} proxy_t; + +typedef int32_t (*proxy_handler_t)(proxy_client_t *, proxy_req_t *, + const void *data, int32_t data_size); + +/* This is used for requests that are not associated with a cmount. */ +static proxy_random_t global_random; + +static int32_t send_error(proxy_client_t *client, int32_t error) +{ + proxy_link_ans_t ans; + struct iovec iov[1]; + + iov[0].iov_base = &ans; + iov[0].iov_len = sizeof(ans); + + return proxy_link_ans_send(client->sd, error, iov, 1); +} + +static uint64_t uint64_checksum(uint64_t value) +{ + value = (value & 0xff00ff00ff00ffULL) + + ((value >> 8) & 0xff00ff00ff00ffULL); + value += value >> 16; + value += value >> 32; + + return value & 0xff; +} + +static uint64_t ptr_checksum(proxy_random_t *rnd, void *ptr) +{ + uint64_t value; + + if (ptr == NULL) { + return 0; + } + + value = (uint64_t)(uintptr_t)ptr; + /* Many current processors don't use the full 64-bits for the virtual + * address space, and Linux assigns the lower 128 TiB (47 bits) for + * user-space applications on most architectures, so the highest 8 bits + * of all valid addressess are always 0. + * + * We use this to encode a checksum in the high byte of the address to + * be able to do a verification before dereferencing the pointer, + * avoiding crashes if the client passes an invalid or corrupted pointer + * value. + * + * Alternatives like using indexes in a table or registering valid + * pointers require access to a shared data structure that will require + * thread synchronization, making it slower. */ + if ((value & 0xff00000000000007ULL) != 0) { + proxy_log(LOG_ERR, EINVAL, + "Unexpected pointer value"); + abort(); + } + + value -= uint64_checksum(value) << 56; + + return random_scramble(rnd, value); +} + +static int32_t ptr_check(proxy_random_t *rnd, uint64_t value, void **pptr) +{ + if (value == 0) { + *pptr = NULL; + return 0; + } + + value = random_unscramble(rnd, value); + + if ((uint64_checksum(value) != 0) || ((value & 7) != 0)) { + proxy_log(LOG_ERR, EFAULT, "Unexpected pointer value"); + return -EFAULT; + } + + *pptr = (void *)(uintptr_t)(value & 0xffffffffffffffULL); + + return 0; +} + +/* Macro to simplify request handling. */ +#define CEPH_COMPLETE(_client, _err, _ans) \ + ({ \ + int32_t __err = (_err); \ + if (__err < 0) { \ + __err = send_error(_client, __err); \ + } else { \ + __err = CEPH_RET(_client->sd, __err, _ans); \ + } \ + __err; \ + }) + +#ifdef PROXY_TRACE +#define TRACE(_fmt, _args...) printf(_fmt "\n", ##_args) +#else +#define TRACE(_fmt, _args...) do { } while (0) +#endif + +static int32_t libcephfsd_version(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_version, ans, 1); + const char *text; + int32_t major, minor, patch; + + text = ceph_version(&major, &minor, &patch); + TRACE("ceph_version(%d, %d, %d) -> %s", major, minor, patch, text); + + ans.major = major; + ans.minor = minor; + ans.patch = patch; + + CEPH_STR_ADD(ans, text, text); + + return CEPH_RET(client->sd, 0, ans); +} + +static int32_t libcephfsd_userperm_new(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_userperm_new, ans, 0); + UserPerm *userperm; + int32_t err; + + userperm = ceph_userperm_new(req->userperm_new.uid, + req->userperm_new.gid, + req->userperm_new.groups, (gid_t *)data); + TRACE("ceph_userperm_new(%u, %u, %u) -> %p", req->userperm_new.uid, + req->userperm_new.gid, req->userperm_new.groups, userperm); + + err = -ENOMEM; + if (userperm != NULL) { + ans.userperm = ptr_checksum(&global_random, userperm); + err = 0; + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_userperm_destroy(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_userperm_destroy, ans, 0); + UserPerm *perms; + int32_t err; + + err = ptr_check(&global_random, req->userperm_destroy.userperm, + (void **)&perms); + + if (err >= 0) { + ceph_userperm_destroy(perms); + TRACE("ceph_userperm_destroy(%p)", perms); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_create(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_create, ans, 0); + proxy_mount_t *mount; + const char *id; + int32_t err; + + id = CEPH_STR_GET(req->create, id, data); + + err = proxy_mount_create(&mount, id); + TRACE("ceph_create(%p, '%s') -> %d", mount, id, err); + + if (err >= 0) { + ans.cmount = ptr_checksum(&client->random, mount); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_release(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_release, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->release.cmount, (void **)&mount); + if (err >= 0) { + err = proxy_mount_release(mount); + TRACE("ceph_release(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_read_file(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_conf_read_file, ans, 0); + proxy_mount_t *mount; + const char *path; + int32_t err; + + err = ptr_check(&client->random, req->conf_read_file.cmount, + (void **)&mount); + if (err >= 0) { + path = CEPH_STR_GET(req->conf_read_file, path, data); + + err = proxy_mount_config(mount, path); + TRACE("ceph_conf_read_file(%p, '%s') ->%d", mount, path, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_get(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_conf_get, ans, 1); + proxy_mount_t *mount; + const char *option; + void *buffer; + uint32_t size; + int32_t err; + + buffer = client->buffer; + size = client->buffer_size; + if (req->conf_get.size < size) { + size = req->conf_get.size; + } + err = ptr_check(&client->random, req->conf_get.cmount, (void **)&mount); + if (err >= 0) { + option = CEPH_STR_GET(req->conf_get, option, data); + + err = proxy_mount_get(mount, option, buffer, size); + TRACE("ceph_conf_get(%p, '%s', '%s') -> %d", mount, option, + (char *)buffer, err); + + if (err >= 0) { + CEPH_DATA_ADD(ans, value, buffer, strlen(buffer) + 1); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_set(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_conf_set, ans, 0); + proxy_mount_t *mount; + const char *option, *value; + int32_t err; + + err = ptr_check(&client->random, req->conf_set.cmount, (void **)&mount); + if (err >= 0) { + option = CEPH_STR_GET(req->conf_set, option, data); + value = CEPH_STR_GET(req->conf_set, value, + data + req->conf_set.option); + + err = proxy_mount_set(mount, option, value); + TRACE("ceph_conf_set(%p, '%s', '%s') -> %d", mount, option, + value, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_init(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_init, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->init.cmount, (void **)&mount); + if (err >= 0) { + err = proxy_mount_init(mount); + TRACE("ceph_init(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_select_filesystem(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_select_filesystem, ans, 0); + proxy_mount_t *mount; + const char *fs; + int32_t err; + + err = ptr_check(&client->random, req->select_filesystem.cmount, + (void **)&mount); + if (err >= 0) { + fs = CEPH_STR_GET(req->select_filesystem, fs, data); + + err = proxy_mount_select(mount, fs); + TRACE("ceph_select_filesystem(%p, '%s') -> %d", mount, fs, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_mount(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_mount, ans, 0); + proxy_mount_t *mount; + const char *root; + int32_t err; + + err = ptr_check(&client->random, req->mount.cmount, (void **)&mount); + if (err >= 0) { + root = CEPH_STR_GET(req->mount, root, data); + + err = proxy_mount_mount(mount, root); + TRACE("ceph_mount(%p, '%s') -> %d", mount, root, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_unmount(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_unmount, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->unmount.cmount, (void **)&mount); + + if (err >= 0) { + err = proxy_mount_unmount(mount); + TRACE("ceph_unmount(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_statfs(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_statfs, ans, 1); + struct statvfs st; + proxy_mount_t *mount; + struct Inode *inode; + int32_t err; + + err = ptr_check(&client->random, req->ll_statfs.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_statfs.inode, + (void **)&inode); + } + + if (err >= 0) { + CEPH_BUFF_ADD(ans, &st, sizeof(st)); + + err = ceph_ll_statfs(proxy_cmount(mount), inode, &st); + TRACE("ceph_ll_statfs(%p, %p) -> %d", mount, inode, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *out; + const char *name; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_lookup.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_lookup.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_lookup.want; + flags = req->ll_lookup.flags; + name = CEPH_STR_GET(req->ll_lookup, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + if (name == NULL) { + err = proxy_log(LOG_ERR, EINVAL, + "NULL name passed to ceph_ll_lookup()"); + } else { + // Forbid going outside of the root mount point + if ((parent == mount->root) && + (strcmp(name, "..") == 0)) { + name = "."; + } + + err = ceph_ll_lookup(proxy_cmount(mount), parent, name, + &out, &stx, want, flags, perms); + } + + TRACE("ceph_ll_lookup(%p, %p, '%s', %p, %x, %x, %p) -> %d", + mount, parent, name, out, want, flags, perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, out); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup_inode(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup_inode, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + struct inodeno_t ino; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup_inode.cmount, + (void **)&mount); + if (err >= 0) { + ino = req->ll_lookup_inode.ino; + + err = ceph_ll_lookup_inode(proxy_cmount(mount), ino, &inode); + TRACE("ceph_ll_lookup_inode(%p, %lu, %p) -> %d", mount, ino.val, + inode, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup_root(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup_root, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup_root.cmount, + (void **)&mount); + if (err >= 0) { + /* The libcephfs view of the root of the mount could be + * different than ours, so we can't rely on + * ceph_ll_lookup_root(). We fake it by returning the cached + * root inode at the time of mount. */ + err = proxy_inode_ref(mount, mount->root_ino); + TRACE("ceph_ll_lookup_root(%p, %p) -> %d", mount, mount->root, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, mount->root); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_put(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_put, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + int32_t err; + + err = ptr_check(&client->random, req->ll_put.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_put.inode, + (void **)&inode); + } + + if (err >= 0) { + err = ceph_ll_put(proxy_cmount(mount), inode); + TRACE("ceph_ll_put(%p, %p) -> %d", mount, inode, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_walk(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_walk, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + const char *path; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_walk.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&global_random, req->ll_walk.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_walk.want; + flags = req->ll_walk.flags; + path = CEPH_STR_GET(req->ll_walk, path, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = proxy_path_resolve(mount, path, &inode, &stx, want, flags, + perms, NULL); + TRACE("ceph_ll_walk(%p, '%s', %p, %x, %x, %p) -> %d", mount, + path, inode, want, flags, perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_chdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_chdir, ans, 0); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + const char *path; + char *realpath; + int32_t err; + + err = ptr_check(&client->random, req->chdir.cmount, (void **)&mount); + if (err >= 0) { + path = CEPH_STR_GET(req->chdir, path, data); + + /* Since the libcephfs mount may be shared, we can't really + * change the current directory to avoid interferences with + * other users, so we just lookup the new directory and keep an + * internal reference. */ + err = proxy_path_resolve(mount, path, &inode, &stx, + CEPH_STATX_INO, 0, mount->perms, + &realpath); + TRACE("ceph_chdir(%p, '%s') -> %d", mount, path, err); + if (err >= 0) { + ceph_ll_put(proxy_cmount(mount), mount->cwd); + mount->cwd = inode; + mount->cwd_ino = stx.stx_ino; + + /* TODO: This path may become outdated if the parent + * directories are moved, however this seems the + * best we can do for now. */ + proxy_free(mount->cwd_path); + mount->cwd_path = realpath; + mount->cwd_path_len = strlen(realpath); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_getcwd(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_getcwd, ans, 1); + proxy_mount_t *mount; + const char *path; + int32_t err; + + err = ptr_check(&client->random, req->getcwd.cmount, (void **)&mount); + + if (err >= 0) { + /* We just return the cached name from the last chdir(). */ + path = mount->cwd_path; + TRACE("ceph_getcwd(%p) -> '%s'", mount, path); + CEPH_STR_ADD(ans, path, path); + err = 0; + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_readdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_readdir, ans, 1); + struct dirent de; + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->readdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->readdir.dir, + (void **)&dirp); + } + + if (err >= 0) { + err = ceph_readdir_r(proxy_cmount(mount), dirp, &de); + TRACE("ceph_readdir_r(%p, %p, %p) -> %d", mount, dirp, &de, + err); + ans.eod = true; + if (err > 0) { + ans.eod = false; + CEPH_BUFF_ADD(ans, &de, + offset_of(struct dirent, d_name) + + strlen(de.d_name) + 1); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_rewinddir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_rewinddir, ans, 0); + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->rewinddir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->rewinddir.dir, + (void **)&dirp); + } + + if (err >= 0) { + ceph_rewinddir(proxy_cmount(mount), dirp); + TRACE("ceph_rewinddir(%p, %p)", mount, dirp); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_open(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_open, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + struct Fh *fh; + int32_t flags, err; + + err = ptr_check(&client->random, req->ll_open.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_open.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_open.userperm, + (void **)&perms); + } + if (err >= 0) { + flags = req->ll_open.flags; + + err = ceph_ll_open(proxy_cmount(mount), inode, flags, &fh, + perms); + TRACE("ceph_ll_open(%p, %p, %x, %p, %p) -> %d", mount, inode, + flags, fh, perms, err); + + if (err >= 0) { + ans.fh = ptr_checksum(&client->random, fh); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_create(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_create, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + struct Fh *fh; + const char *name; + UserPerm *perms; + mode_t mode; + uint32_t want, flags; + int32_t oflags, err; + + err = ptr_check(&client->random, req->ll_create.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_create.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_create.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_create.mode; + oflags = req->ll_create.oflags; + want = req->ll_create.want; + flags = req->ll_create.flags; + name = CEPH_STR_GET(req->ll_create, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_create(proxy_cmount(mount), parent, name, mode, + oflags, &inode, &fh, &stx, want, flags, + perms); + TRACE("ceph_ll_create(%p, %p, '%s', %o, %x, %p, %p, %x, %x, " + "%p) -> %d", + mount, parent, name, mode, oflags, inode, fh, want, flags, + perms, err); + + if (err >= 0) { + ans.fh = ptr_checksum(&client->random, fh); + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_mknod(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_mknod, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + dev_t rdev; + mode_t mode; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_mknod.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_mknod.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_mknod.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_mknod.mode; + rdev = req->ll_mknod.rdev; + want = req->ll_mknod.want; + flags = req->ll_mknod.flags; + name = CEPH_STR_GET(req->ll_mknod, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_mknod(proxy_cmount(mount), parent, name, mode, + rdev, &inode, &stx, want, flags, perms); + TRACE("ceph_ll_mknod(%p, %p, '%s', %o, %lx, %p, %x, %x, %p) -> " + "%d", + mount, parent, name, mode, rdev, inode, want, flags, + perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_close(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_close, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int32_t err; + + err = ptr_check(&client->random, req->ll_close.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_close.fh, + (void **)&fh); + } + + if (err >= 0) { + err = ceph_ll_close(proxy_cmount(mount), fh); + TRACE("ceph_ll_close(%p, %p) -> %d", mount, fh, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_rename(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_rename, ans, 0); + proxy_mount_t *mount; + struct Inode *old_parent, *new_parent; + const char *old_name, *new_name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_rename.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rename.old_parent, + (void **)&old_parent); + } + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rename.new_parent, + (void **)&new_parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_rename.userperm, + (void **)&perms); + } + if (err >= 0) { + old_name = CEPH_STR_GET(req->ll_rename, old_name, data); + new_name = CEPH_STR_GET(req->ll_rename, new_name, + data + req->ll_rename.old_name); + + err = ceph_ll_rename(proxy_cmount(mount), old_parent, old_name, + new_parent, new_name, perms); + TRACE("ceph_ll_rename(%p, %p, '%s', %p, '%s', %p) -> %d", mount, + old_parent, old_name, new_parent, new_name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lseek(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_lseek, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + off_t offset, pos; + int32_t whence, err; + + err = ptr_check(&client->random, req->ll_lseek.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_lseek.fh, + (void **)&fh); + } + if (err >= 0) { + offset = req->ll_lseek.offset; + whence = req->ll_lseek.whence; + + pos = ceph_ll_lseek(proxy_cmount(mount), fh, offset, whence); + err = -errno; + TRACE("ceph_ll_lseek(%p, %p, %ld, %d) -> %ld (%d)", mount, fh, + offset, whence, pos, -err); + + if (pos >= 0) { + ans.offset = pos; + err = 0; + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_read(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_read, ans, 1); + proxy_mount_t *mount; + struct Fh *fh; + void *buffer; + uint64_t len; + int64_t offset; + uint32_t size; + int32_t err; + + buffer = client->buffer; + + err = ptr_check(&client->random, req->ll_read.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_read.fh, (void **)&fh); + } + if (err >= 0) { + offset = req->ll_read.offset; + len = req->ll_read.len; + + size = client->buffer_size; + if (len > size) { + buffer = proxy_malloc(len); + if (buffer == NULL) { + err = -ENOMEM; + } + } + if (err >= 0) { + err = ceph_ll_read(proxy_cmount(mount), fh, offset, len, + buffer); + TRACE("ceph_ll_read(%p, %p, %ld, %lu) -> %d", mount, fh, + offset, len, err); + + if (err >= 0) { + CEPH_BUFF_ADD(ans, buffer, err); + } + } + } + + err = CEPH_COMPLETE(client, err, ans); + + if (buffer != client->buffer) { + proxy_free(buffer); + } + + return err; +} + +static int32_t libcephfsd_ll_write(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_write, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + uint64_t len; + int64_t offset; + int32_t err; + + err = ptr_check(&client->random, req->ll_write.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_write.fh, + (void **)&fh); + } + if (err >= 0) { + offset = req->ll_write.offset; + len = req->ll_write.len; + + err = ceph_ll_write(proxy_cmount(mount), fh, offset, len, data); + TRACE("ceph_ll_write(%p, %p, %ld, %lu) -> %d", mount, fh, + offset, len, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_link(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_link, ans, 0); + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_link.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_link.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&client->random, req->ll_link.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_link.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_link, name, data); + + err = ceph_ll_link(proxy_cmount(mount), inode, parent, name, + perms); + TRACE("ceph_ll_link(%p, %p, %p, '%s', %p) -> %d", mount, inode, + parent, name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_unlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_unlink, ans, 0); + proxy_mount_t *mount; + struct Inode *parent; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_unlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_unlink.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_unlink.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_unlink, name, data); + + err = ceph_ll_unlink(proxy_cmount(mount), parent, name, perms); + TRACE("ceph_ll_unlink(%p, %p, '%s', %p) -> %d", mount, parent, + name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_getattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_getattr, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_getattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_getattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_getattr.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_getattr.want; + flags = req->ll_getattr.flags; + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_getattr(proxy_cmount(mount), inode, &stx, want, + flags, perms); + TRACE("ceph_ll_getattr(%p, %p, %x, %x, %p) -> %d", mount, inode, + want, flags, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_setattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_setattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + int32_t mask, err; + + err = ptr_check(&client->random, req->ll_setattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_setattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_setattr.userperm, + (void **)&perms); + } + if (err >= 0) { + mask = req->ll_setattr.mask; + + err = ceph_ll_setattr(proxy_cmount(mount), inode, (void *)data, + mask, perms); + TRACE("ceph_ll_setattr(%p, %p, %x, %p) -> %d", mount, inode, + mask, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_fallocate(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_fallocate, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int64_t offset, len; + mode_t mode; + int32_t err; + + err = ptr_check(&client->random, req->ll_fallocate.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_fallocate.fh, + (void **)&fh); + } + if (err >= 0) { + mode = req->ll_fallocate.mode; + offset = req->ll_fallocate.offset; + len = req->ll_fallocate.length; + + err = ceph_ll_fallocate(proxy_cmount(mount), fh, mode, offset, + len); + TRACE("ceph_ll_fallocate(%p, %p, %o, %ld, %lu) -> %d", mount, + fh, mode, offset, len, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_fsync(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_fsync, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int32_t dataonly, err; + + err = ptr_check(&client->random, req->ll_fsync.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_fsync.fh, + (void **)&fh); + } + if (err >= 0) { + dataonly = req->ll_fsync.dataonly; + + err = ceph_ll_fsync(proxy_cmount(mount), fh, dataonly); + TRACE("ceph_ll_fsync(%p, %p, %d) -> %d", mount, fh, dataonly, + err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_listxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_listxattr, ans, 1); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_listxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_listxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_listxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_listxattr.size; + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_listxattr(proxy_cmount(mount), inode, + client->buffer, size, &size, perms); + TRACE("ceph_ll_listxattr(%p, %p, %lu, %p) -> %d", mount, inode, + size, perms, err); + + if (err >= 0) { + ans.size = size; + CEPH_BUFF_ADD(ans, client->buffer, size); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_getxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_getxattr, ans, 1); + proxy_mount_t *mount; + struct Inode *inode; + const char *name; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_getxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_getxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_getxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_getxattr.size; + name = CEPH_STR_GET(req->ll_getxattr, name, data); + + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_getxattr(proxy_cmount(mount), inode, name, + client->buffer, size, perms); + TRACE("ceph_ll_getxattr(%p, %p, '%s', %p) -> %d", mount, inode, + name, perms, err); + + if (err >= 0) { + CEPH_BUFF_ADD(ans, client->buffer, err); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_setxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_setxattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + const char *name, *value; + UserPerm *perms; + size_t size; + int32_t flags, err; + + err = ptr_check(&client->random, req->ll_setxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_setxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_setxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_setxattr, name, data); + value = data + req->ll_setxattr.name; + size = req->ll_setxattr.size; + flags = req->ll_setxattr.flags; + + err = ceph_ll_setxattr(proxy_cmount(mount), inode, name, value, + size, flags, perms); + TRACE("ceph_ll_setxattr(%p, %p, '%s', %p, %x, %p) -> %d", mount, + inode, name, value, flags, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_removexattr(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_removexattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_removexattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_removexattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_removexattr.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_removexattr, name, data); + + err = ceph_ll_removexattr(proxy_cmount(mount), inode, name, + perms); + TRACE("ceph_ll_removexattr(%p, %p, '%s', %p) -> %d", mount, + inode, name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_readlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_readlink, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_readlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_readlink.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_readlink.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_readlink.size; + + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_readlink(proxy_cmount(mount), inode, + client->buffer, size, perms); + TRACE("ceph_ll_readlink(%p, %p, %p) -> %d", mount, inode, perms, + err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_symlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_symlink, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + UserPerm *perms; + const char *name, *value; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_symlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_symlink.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_symlink.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_symlink, name, data); + value = CEPH_STR_GET(req->ll_symlink, target, + data + req->ll_symlink.name); + want = req->ll_symlink.want; + flags = req->ll_symlink.flags; + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_symlink(proxy_cmount(mount), parent, name, value, + &inode, &stx, want, flags, perms); + TRACE("ceph_ll_symlink(%p, %p, '%s', '%s', %p, %x, %x, %p) -> " + "%d", + mount, parent, name, value, inode, want, flags, perms, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_opendir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_opendir, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + struct ceph_dir_result *dirp; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_opendir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_opendir.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_opendir.userperm, + (void **)&perms); + } + + if (err >= 0) { + err = ceph_ll_opendir(proxy_cmount(mount), inode, &dirp, perms); + TRACE("ceph_ll_opendir(%p, %p, %p, %p) -> %d", mount, inode, + dirp, perms, err); + + if (err >= 0) { + ans.dir = ptr_checksum(&client->random, dirp); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_mkdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_mkdir, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + mode_t mode; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_mkdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_mkdir.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_mkdir.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_mkdir.mode; + want = req->ll_mkdir.want; + flags = req->ll_mkdir.flags; + name = CEPH_STR_GET(req->ll_mkdir, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_mkdir(proxy_cmount(mount), parent, name, mode, + &inode, &stx, want, flags, perms); + TRACE("ceph_ll_mkdir(%p, %p, '%s', %o, %p, %x, %x, %p) -> %d", + mount, parent, name, mode, inode, want, flags, perms, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_rmdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_rmdir, ans, 0); + proxy_mount_t *mount; + struct Inode *parent; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_rmdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rmdir.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_rmdir.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_rmdir, name, data); + + err = ceph_ll_rmdir(proxy_cmount(mount), parent, name, perms); + TRACE("ceph_ll_rmdir(%p, %p, '%s', %p) -> %d", mount, parent, + name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_releasedir(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_releasedir, ans, 0); + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->ll_releasedir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_releasedir.dir, + (void **)&dirp); + } + + if (err >= 0) { + err = ceph_ll_releasedir(proxy_cmount(mount), dirp); + TRACE("ceph_ll_releasedir(%p, %p) -> %d", mount, dirp, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_mount_perms(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_mount_perms, ans, 0); + proxy_mount_t *mount; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->mount_perms.cmount, + (void **)&mount); + if (err >= 0) { + perms = ceph_mount_perms(proxy_cmount(mount)); + TRACE("ceph_mount_perms(%p) -> %p", mount, perms); + + ans.userperm = ptr_checksum(&global_random, perms); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static proxy_handler_t libcephfsd_handlers[LIBCEPHFSD_OP_TOTAL_OPS] = { + [LIBCEPHFSD_OP_VERSION] = libcephfsd_version, + [LIBCEPHFSD_OP_USERPERM_NEW] = libcephfsd_userperm_new, + [LIBCEPHFSD_OP_USERPERM_DESTROY] = libcephfsd_userperm_destroy, + [LIBCEPHFSD_OP_CREATE] = libcephfsd_create, + [LIBCEPHFSD_OP_RELEASE] = libcephfsd_release, + [LIBCEPHFSD_OP_CONF_READ_FILE] = libcephfsd_conf_read_file, + [LIBCEPHFSD_OP_CONF_GET] = libcephfsd_conf_get, + [LIBCEPHFSD_OP_CONF_SET] = libcephfsd_conf_set, + [LIBCEPHFSD_OP_INIT] = libcephfsd_init, + [LIBCEPHFSD_OP_SELECT_FILESYSTEM] = libcephfsd_select_filesystem, + [LIBCEPHFSD_OP_MOUNT] = libcephfsd_mount, + [LIBCEPHFSD_OP_UNMOUNT] = libcephfsd_unmount, + [LIBCEPHFSD_OP_LL_STATFS] = libcephfsd_ll_statfs, + [LIBCEPHFSD_OP_LL_LOOKUP] = libcephfsd_ll_lookup, + [LIBCEPHFSD_OP_LL_LOOKUP_INODE] = libcephfsd_ll_lookup_inode, + [LIBCEPHFSD_OP_LL_LOOKUP_ROOT] = libcephfsd_ll_lookup_root, + [LIBCEPHFSD_OP_LL_PUT] = libcephfsd_ll_put, + [LIBCEPHFSD_OP_LL_WALK] = libcephfsd_ll_walk, + [LIBCEPHFSD_OP_CHDIR] = libcephfsd_chdir, + [LIBCEPHFSD_OP_GETCWD] = libcephfsd_getcwd, + [LIBCEPHFSD_OP_READDIR] = libcephfsd_readdir, + [LIBCEPHFSD_OP_REWINDDIR] = libcephfsd_rewinddir, + [LIBCEPHFSD_OP_LL_OPEN] = libcephfsd_ll_open, + [LIBCEPHFSD_OP_LL_CREATE] = libcephfsd_ll_create, + [LIBCEPHFSD_OP_LL_MKNOD] = libcephfsd_ll_mknod, + [LIBCEPHFSD_OP_LL_CLOSE] = libcephfsd_ll_close, + [LIBCEPHFSD_OP_LL_RENAME] = libcephfsd_ll_rename, + [LIBCEPHFSD_OP_LL_LSEEK] = libcephfsd_ll_lseek, + [LIBCEPHFSD_OP_LL_READ] = libcephfsd_ll_read, + [LIBCEPHFSD_OP_LL_WRITE] = libcephfsd_ll_write, + [LIBCEPHFSD_OP_LL_LINK] = libcephfsd_ll_link, + [LIBCEPHFSD_OP_LL_UNLINK] = libcephfsd_ll_unlink, + [LIBCEPHFSD_OP_LL_GETATTR] = libcephfsd_ll_getattr, + [LIBCEPHFSD_OP_LL_SETATTR] = libcephfsd_ll_setattr, + [LIBCEPHFSD_OP_LL_FALLOCATE] = libcephfsd_ll_fallocate, + [LIBCEPHFSD_OP_LL_FSYNC] = libcephfsd_ll_fsync, + [LIBCEPHFSD_OP_LL_LISTXATTR] = libcephfsd_ll_listxattr, + [LIBCEPHFSD_OP_LL_GETXATTR] = libcephfsd_ll_getxattr, + [LIBCEPHFSD_OP_LL_SETXATTR] = libcephfsd_ll_setxattr, + [LIBCEPHFSD_OP_LL_REMOVEXATTR] = libcephfsd_ll_removexattr, + [LIBCEPHFSD_OP_LL_READLINK] = libcephfsd_ll_readlink, + [LIBCEPHFSD_OP_LL_SYMLINK] = libcephfsd_ll_symlink, + [LIBCEPHFSD_OP_LL_OPENDIR] = libcephfsd_ll_opendir, + [LIBCEPHFSD_OP_LL_MKDIR] = libcephfsd_ll_mkdir, + [LIBCEPHFSD_OP_LL_RMDIR] = libcephfsd_ll_rmdir, + [LIBCEPHFSD_OP_LL_RELEASEDIR] = libcephfsd_ll_releasedir, + [LIBCEPHFSD_OP_MOUNT_PERMS] = libcephfsd_mount_perms, +}; + +static void serve_binary(proxy_client_t *client) +{ + proxy_req_t req; + CEPH_DATA(hello, ans, 0); + struct iovec req_iov[2]; + void *buffer; + uint32_t size; + int32_t err; + + /* This buffer will be used by most of the requests. For requests that + * require more space (probably just some writes), a new temporary + * buffer will be allocated by proxy_link_req_recv() code. */ + size = 65536; + buffer = proxy_malloc(size); + if (buffer == NULL) { + return; + } + + ans.major = LIBCEPHFSD_MAJOR; + ans.minor = LIBCEPHFSD_MINOR; + err = proxy_link_send(client->sd, ans_iov, ans_count); + if (err < 0) { + proxy_free(buffer); + return; + } + + while (true) { + req_iov[0].iov_base = &req; + req_iov[0].iov_len = sizeof(req); + req_iov[1].iov_base = buffer; + req_iov[1].iov_len = size; + + err = proxy_link_req_recv(client->sd, req_iov, 2); + if (err > 0) { + if (req.header.op >= LIBCEPHFSD_OP_TOTAL_OPS) { + err = send_error(client, -ENOSYS); + } else if (libcephfsd_handlers[req.header.op] == NULL) { + err = send_error(client, -EOPNOTSUPP); + } else { + err = libcephfsd_handlers[req.header.op]( + client, &req, req_iov[1].iov_base, + req.header.data_len); + } + } + + if (req_iov[1].iov_base != buffer) { + /* Free the buffer if it was temporarily allocated. */ + proxy_free(req_iov[1].iov_base); + } + + if (err < 0) { + break; + } + } + + proxy_free(buffer); +} + +static void serve_connection(proxy_worker_t *worker) +{ + CEPH_DATA(hello, req, 0); + proxy_client_t *client; + int32_t err; + + client = container_of(worker, proxy_client_t, worker); + + err = proxy_link_recv(client->sd, req_iov, req_count); + if (err >= 0) { + if (req.id == LIBCEPHFS_LIB_CLIENT) { + serve_binary(client); + } else { + proxy_log(LOG_ERR, EINVAL, + "Invalid client initial message"); + } + } + + close(client->sd); +} + +static void destroy_connection(proxy_worker_t *worker) +{ + proxy_client_t *client; + + client = container_of(worker, proxy_client_t, worker); + + proxy_free(client->buffer); + proxy_free(client); +} + +static int32_t accept_connection(proxy_link_t *link, int32_t sd) +{ + proxy_server_t *server; + proxy_client_t *client; + int32_t err; + + server = container_of(link, proxy_server_t, link); + + client = proxy_malloc(sizeof(proxy_client_t)); + if (client == NULL) { + err = -ENOMEM; + goto failed_close; + } + + client->buffer_size = 65536; + client->buffer = proxy_malloc(client->buffer_size); + if (client->buffer == NULL) { + err = -ENOMEM; + goto failed_client; + } + + random_init(&client->random); + client->sd = sd; + client->link = link; + + /* TODO: Make request management asynchronous and avoid creating a + * thread for each connection. */ + err = proxy_manager_launch(server->manager, &client->worker, + serve_connection, destroy_connection); + if (err < 0) { + goto failed_buffer; + } + + return 0; + +failed_buffer: + proxy_free(client->buffer); + +failed_client: + proxy_free(client); + +failed_close: + close(sd); + + return err; +} + +static bool check_stop(proxy_link_t *link) +{ + proxy_server_t *server; + + server = container_of(link, proxy_server_t, link); + + return proxy_manager_stop(server->manager); +} + +static int32_t server_start(proxy_manager_t *manager) +{ + proxy_server_t server; + proxy_t *proxy; + + proxy = container_of(manager, proxy_t, manager); + + server.manager = manager; + + return proxy_link_server(&server.link, proxy->socket_path, + accept_connection, check_stop); +} + +static void log_print(proxy_log_handler_t *handler, int32_t level, int32_t err, + const char *msg) +{ + printf("[%d] %s\n", level, msg); +} + +static struct option main_opts[] = { + {"socket", required_argument, NULL, 's'}, + {} +}; + +int32_t main(int32_t argc, char *argv[]) +{ + struct timespec now; + proxy_t proxy; + char *env; + int32_t err, val; + + clock_gettime(CLOCK_MONOTONIC, &now); + srand(now.tv_nsec); + + random_init(&global_random); + + proxy_log_register(&proxy.log_handler, log_print); + + proxy.socket_path = PROXY_SOCKET; + + env = getenv(PROXY_SOCKET_ENV); + if (env != NULL) { + proxy.socket_path = env; + } + + while ((val = getopt_long(argc, argv, ":s:", main_opts, NULL)) >= 0) { + if (val == 's') { + proxy.socket_path = optarg; + } else if (val == ':') { + proxy_log(LOG_ERR, ENODATA, + "Argument missing for '%s'\n", optopt); + return 1; + } else if (val == '?') { + proxy_log(LOG_ERR, EINVAL, + "Unknown option '%s'\n", optopt); + return 1; + } else { + proxy_log(LOG_ERR, EINVAL, + "Unexpected error parsing the options\n"); + return 1; + } + } + if (optind < argc) { + proxy_log(LOG_ERR, EINVAL, + "Unexpected arguments in command line"); + return 1; + } + + err = proxy_manager_run(&proxy.manager, server_start); + + proxy_log_deregister(&proxy.log_handler); + + return err < 0 ? 1 : 0; +} diff --git a/src/libcephfs_proxy/proxy.h b/src/libcephfs_proxy/proxy.h new file mode 100644 index 00000000000..cfb69072f19 --- /dev/null +++ b/src/libcephfs_proxy/proxy.h @@ -0,0 +1,67 @@ + +#ifndef __LIBCEPHFSD_PROXY_H__ +#define __LIBCEPHFSD_PROXY_H__ + +#include <string.h> +#include <errno.h> +#include <stdarg.h> +#include <stdint.h> +#include <stdbool.h> + +#define LIBCEPHFSD_MAJOR 0 +#define LIBCEPHFSD_MINOR 2 + +#define LIBCEPHFS_LIB_CLIENT 0xe3e5f0e8 // 'ceph' xor 0x80808080 + +#define PROXY_SOCKET "/run/libcephfsd.sock" +#define PROXY_SOCKET_ENV "LIBCEPHFSD_SOCKET" + +#define offset_of(_type, _field) ((uintptr_t) & ((_type *)0)->_field) + +#define container_of(_ptr, _type, _field) \ + ((_type *)((uintptr_t)(_ptr) - offset_of(_type, _field))) + +struct _list; +typedef struct _list list_t; + +struct _proxy_buffer_ops; +typedef struct _proxy_buffer_ops proxy_buffer_ops_t; + +struct _proxy_buffer; +typedef struct _proxy_buffer proxy_buffer_t; + +struct _proxy_output; +typedef struct _proxy_output proxy_output_t; + +struct _proxy_log_handler; +typedef struct _proxy_log_handler proxy_log_handler_t; + +struct _proxy_worker; +typedef struct _proxy_worker proxy_worker_t; + +struct _proxy_manager; +typedef struct _proxy_manager proxy_manager_t; + +struct _proxy_link; +typedef struct _proxy_link proxy_link_t; + +typedef int32_t (*proxy_output_write_t)(proxy_output_t *); +typedef int32_t (*proxy_output_full_t)(proxy_output_t *); + +typedef void (*proxy_log_callback_t)(proxy_log_handler_t *, int32_t, int32_t, + const char *); + +typedef void (*proxy_worker_start_t)(proxy_worker_t *); +typedef void (*proxy_worker_destroy_t)(proxy_worker_t *); + +typedef int32_t (*proxy_manager_start_t)(proxy_manager_t *); + +typedef int32_t (*proxy_link_start_t)(proxy_link_t *, int32_t); +typedef bool (*proxy_link_stop_t)(proxy_link_t *); + +struct _list { + list_t *next; + list_t *prev; +}; + +#endif diff --git a/src/libcephfs_proxy/proxy_helpers.c b/src/libcephfs_proxy/proxy_helpers.c new file mode 100644 index 00000000000..149d84d34bb --- /dev/null +++ b/src/libcephfs_proxy/proxy_helpers.c @@ -0,0 +1,81 @@ + +#include "proxy_helpers.h" + +#include <openssl/evp.h> + +static const char hex_digits[] = "0123456789abcdef"; + +int32_t proxy_hash(uint8_t *hash, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data) +{ + EVP_MD_CTX *ctx; + void *ptr; + uint32_t bytes; + int32_t i, err, len; + + if (size < 32) { + return proxy_log(LOG_ERR, ENOBUFS, + "Digest buffer is too small"); + } + + ctx = EVP_MD_CTX_new(); + if (ctx == NULL) { + return proxy_log(LOG_ERR, ENOMEM, "EVP_MD_CTX_new() failed"); + } + + if (!EVP_DigestInit_ex2(ctx, EVP_sha256(), NULL)) { + err = proxy_log(LOG_ERR, ENOMEM, "EVP_DigestInit_ex2() failed"); + goto done; + } + + i = 0; + while ((len = feed(&ptr, data, i)) > 0) { + if (!EVP_DigestUpdate(ctx, ptr, len)) { + err = proxy_log(LOG_ERR, ENOMEM, + "EVP_DigestUpdate() failed"); + goto done; + } + i++; + } + if (len < 0) { + err = len; + goto done; + } + + if (!EVP_DigestFinal_ex(ctx, hash, &bytes)) { + err = proxy_log(LOG_ERR, ENOMEM, "EVP_DigestFinal_ex() failed"); + goto done; + } + + err = 0; + +done: + EVP_MD_CTX_free(ctx); + + return err; +} + +int32_t proxy_hash_hex(char *digest, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data) +{ + uint8_t hash[32]; + int32_t i, err; + + if (size < 65) { + return proxy_log(LOG_ERR, ENOBUFS, + "Digest buffer is too small"); + } + + err = proxy_hash(hash, sizeof(hash), feed, data); + if (err < 0) { + return err; + } + + for (i = 0; i < 32; i++) { + *digest++ = hex_digits[hash[i] >> 4]; + *digest++ = hex_digits[hash[i] & 15]; + } + *digest = 0; + + return 0; +} diff --git a/src/libcephfs_proxy/proxy_helpers.h b/src/libcephfs_proxy/proxy_helpers.h new file mode 100644 index 00000000000..b4f58e7e3b3 --- /dev/null +++ b/src/libcephfs_proxy/proxy_helpers.h @@ -0,0 +1,311 @@ + +#ifndef __LIBCEPHFS_PROXY_HELPERS_H__ +#define __LIBCEPHFS_PROXY_HELPERS_H__ + +#include <stdlib.h> +#include <signal.h> +#include <pthread.h> +#include <stdint.h> +#include <stdbool.h> + +#include "proxy_log.h" + +#define __public __attribute__((__visibility__("default"))) + +#define ptr_value(_ptr) ((uint64_t)(uintptr_t)(_ptr)) +#define value_ptr(_val) ((void *)(uintptr_t)(_val)) + +typedef struct _proxy_random { + uint64_t mask; + uint64_t factor; + uint64_t factor_inv; + uint64_t shift; +} proxy_random_t; + +/* Generate a 64-bits random number different than 0. */ +static inline uint64_t random_u64(void) +{ + uint64_t value; + int32_t i; + + do { + value = 0; + for (i = 0; i < 4; i++) { + value <<= 16; + value ^= (random() >> 8) & 0xffff; + } + } while (value == 0); + + return value; +} + +/* Randomly initialize the data used to scramble pointers. */ +static inline void random_init(proxy_random_t *rnd) +{ + uint64_t inv; + + rnd->mask = random_u64(); + + /* Generate an odd multiplicative factor different than 1. */ + do { + rnd->factor = random_u64() | 1; + } while (rnd->factor == 1); + + /* Compute the inverse of 'factor' modulo 2^64. */ + inv = rnd->factor & 0x3; + inv *= 0x000000012 - rnd->factor * inv; + inv *= 0x000000102 - rnd->factor * inv; + inv *= 0x000010002 - rnd->factor * inv; + inv *= 0x100000002 - rnd->factor * inv; + rnd->factor_inv = inv * (2 - rnd->factor * inv); + + rnd->shift = random_u64(); +} + +/* Obfuscate a pointer. */ +static inline uint64_t random_scramble(proxy_random_t *rnd, uint64_t value) +{ + uint32_t bits; + + bits = __builtin_popcountll(value); + + /* rnd->shift is rotated by the amount of bits set to 1 in the original + * value, and the lowest 6 bits are extracted. This generates a + * pseudo-random number that depends on the number of bits of the + * value. */ + bits = ((rnd->shift >> bits) | (rnd->shift << (64 - bits))) & 0x3f; + + /* The value is rotated by the amount just computed. */ + value = (value << bits) | (value >> (64 - bits)); + + /* The final result is masked with a random number. */ + value ^= rnd->mask; + + /* And multiplied by a random factor modulo 2^64. */ + return value * rnd->factor; +} + +/* Recover a pointer. */ +static inline uint64_t random_unscramble(proxy_random_t *rnd, uint64_t value) +{ + uint32_t bits; + + /* Divide by the random factor (i.e. multiply by the inverse of the + * factor). */ + value *= rnd->factor_inv; + + /* Remove the mask. */ + value ^= rnd->mask; + + /* Get the number of bits the pointer was rotated. */ + bits = __builtin_popcountll(value); + bits = ((rnd->shift >> bits) | (rnd->shift << (64 - bits))) & 0x3f; + + /* Undo the rotation to recover the original value. */ + return (value >> bits) | (value << (64 - bits)); +} + +static inline void *proxy_malloc(size_t size) +{ + void *ptr; + + ptr = malloc(size); + if (ptr == NULL) { + proxy_log(LOG_ERR, errno, "Failed to allocate memory"); + } + + return ptr; +} + +static inline int32_t proxy_realloc(void **pptr, size_t size) +{ + void *ptr; + + ptr = realloc(*pptr, size); + if (ptr == NULL) { + return proxy_log(LOG_ERR, errno, "Failed to reallocate memory"); + } + + *pptr = ptr; + + return 0; +} + +static inline void proxy_free(void *ptr) +{ + free(ptr); +} + +static inline char *proxy_strdup(const char *str) +{ + char *ptr; + + ptr = strdup(str); + if (ptr == NULL) { + proxy_log(LOG_ERR, errno, "Failed to copy a string"); + return NULL; + } + + return ptr; +} + +static inline int32_t proxy_mutex_init(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_init(mutex, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, "Failed to initialize a mutex"); + } + + return 0; +} + +static inline void proxy_mutex_lock(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_lock(mutex); + if (err != 0) { + proxy_abort(err, "Mutex cannot be acquired"); + } +} + +static inline void proxy_mutex_unlock(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_unlock(mutex); + if (err != 0) { + proxy_abort(err, "Mutex cannot be released"); + } +} + +static inline int32_t proxy_rwmutex_init(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_init(mutex, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, + "Failed to initialize a rwmutex"); + } + + return 0; +} + +static inline void proxy_rwmutex_rdlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_rdlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be acquired for read"); + } +} + +static inline void proxy_rwmutex_wrlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_wrlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be acquired for write"); + } +} + +static inline void proxy_rwmutex_unlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_unlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be released"); + } +} + +static inline int32_t proxy_condition_init(pthread_cond_t *condition) +{ + int32_t err; + + err = pthread_cond_init(condition, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, + "Failed to initialize a condition variable"); + } + + return 0; +} + +static inline void proxy_condition_signal(pthread_cond_t *condition) +{ + int32_t err; + + err = pthread_cond_signal(condition); + if (err != 0) { + proxy_abort(err, "Condition variable cannot be signaled"); + } +} + +static inline void proxy_condition_wait(pthread_cond_t *condition, + pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_cond_wait(condition, mutex); + if (err != 0) { + proxy_abort(err, "Condition variable cannot be waited"); + } +} + +static inline int32_t proxy_thread_create(pthread_t *tid, + void *(*start)(void *), void *arg) +{ + int32_t err; + + err = pthread_create(tid, NULL, start, arg); + if (err != 0) { + proxy_log(LOG_ERR, err, "Failed to create a thread"); + } + + return err; +} + +static inline void proxy_thread_kill(pthread_t tid, int32_t signum) +{ + int32_t err; + + err = pthread_kill(tid, signum); + if (err != 0) { + proxy_abort(err, "Failed to send a signal to a thread"); + } +} + +static inline void proxy_thread_join(pthread_t tid) +{ + int32_t err; + + err = pthread_join(tid, NULL); + if (err != 0) { + proxy_log(LOG_ERR, err, "Unable to join a thread"); + } +} + +static inline int32_t proxy_signal_set(int32_t signum, struct sigaction *action, + struct sigaction *old) +{ + if (sigaction(signum, action, old) < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to configure a signal"); + } + + return 0; +} + +int32_t proxy_hash(uint8_t *hash, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data); + +int32_t proxy_hash_hex(char *digest, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data); + +#endif diff --git a/src/libcephfs_proxy/proxy_link.c b/src/libcephfs_proxy/proxy_link.c new file mode 100644 index 00000000000..20d9086ffa9 --- /dev/null +++ b/src/libcephfs_proxy/proxy_link.c @@ -0,0 +1,421 @@ + +#include <stdio.h> +#include <unistd.h> +#include <sys/uio.h> + +#include "proxy_link.h" +#include "proxy_manager.h" +#include "proxy_helpers.h" +#include "proxy_log.h" + +static int32_t iov_length(struct iovec *iov, int32_t count) +{ + int32_t len; + + len = 0; + while (count > 0) { + len += iov->iov_len; + iov++; + count--; + } + + return len; +} + +static int32_t proxy_link_prepare(struct sockaddr_un *addr, const char *path) +{ + struct sigaction action; + int32_t sd, len, err; + + memset(&action, 0, sizeof(action)); + action.sa_handler = SIG_IGN; + err = proxy_signal_set(SIGPIPE, &action, NULL); + if (err < 0) { + return err; + } + + memset(addr, 0, sizeof(*addr)); + addr->sun_family = AF_UNIX; + len = snprintf(addr->sun_path, sizeof(addr->sun_path), "%s", path); + if (len < 0) { + return proxy_log(LOG_ERR, EINVAL, + "Failed to copy Unix socket path"); + } + if (len >= sizeof(addr->sun_path)) { + return proxy_log(LOG_ERR, ENAMETOOLONG, + "Unix socket path too long"); + } + + sd = socket(AF_UNIX, SOCK_STREAM, 0); + if (sd < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to create a Unix socket"); + } + + return sd; +} + +int32_t proxy_link_client(proxy_link_t *link, const char *path, + proxy_link_stop_t stop) +{ + struct sockaddr_un addr; + int32_t sd, err; + + link->stop = stop; + link->sd = -1; + + sd = proxy_link_prepare(&addr, path); + if (sd < 0) { + return sd; + } + + err = 0; + while (err >= 0) { + if (connect(sd, (struct sockaddr *)&addr, sizeof(addr)) < 0) { + if (errno == EINTR) { + continue; + } + + err = proxy_log(LOG_ERR, errno, + "Failed to connect to libcephfsd"); + } else { + link->sd = sd; + return sd; + } + } + + close(sd); + + return err; +} + +void proxy_link_close(proxy_link_t *link) +{ + close(link->sd); + link->sd = -1; +} + +int32_t proxy_link_server(proxy_link_t *link, const char *path, + proxy_link_start_t start, proxy_link_stop_t stop) +{ + struct sockaddr_un addr; + socklen_t len; + int32_t cd, err; + + link->stop = stop; + link->sd = -1; + + err = proxy_link_prepare(&addr, path); + if (err < 0) { + return err; + } + link->sd = err; + + if ((unlink(path) < 0) && (errno != ENOENT) && (errno != ENOTDIR)) { + err = proxy_log(LOG_ERR, errno, + "Failed to remove existing socket"); + goto done; + } + + if (bind(link->sd, (struct sockaddr *)&addr, sizeof(addr)) < 0) { + err = proxy_log(LOG_ERR, errno, "Failed to bind Unix socket"); + goto done; + } + + if (listen(link->sd, SOMAXCONN) < 0) { + err = proxy_log(LOG_ERR, errno, + "Failed to listen from Unix socket"); + goto done; + } + + while (!stop(link)) { + len = sizeof(addr); + cd = accept(link->sd, (struct sockaddr *)&addr, &len); + if (cd < 0) { + if (errno != EINTR) { + proxy_log(LOG_ERR, errno, + "Failed to accept a connection"); + } + } else { + start(link, cd); + } + } + + err = 0; + +done: + close(link->sd); + + return err; +} + +int32_t proxy_link_read(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size) +{ + ssize_t len; + + do { + len = read(sd, buffer, size); + if (len < 0) { + if (errno == EINTR) { + if (link->stop(link)) { + return -EINTR; + } + continue; + } + return proxy_log(LOG_ERR, errno, + "Failed to read from socket"); + } + } while (len < 0); + + return len; +} + +int32_t proxy_link_write(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size) +{ + ssize_t len; + int32_t total; + + total = size; + while (total > 0) { + len = write(sd, buffer, total); + if (len < 0) { + if (errno == EINTR) { + if (link->stop(link)) { + return -EINTR; + } + continue; + } + return proxy_log(LOG_ERR, errno, + "Failed to write to socket"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENOBUFS, + "No data written to socket"); + } + + buffer += len; + total -= len; + } + + return size; +} + +int32_t proxy_link_send(int32_t sd, struct iovec *iov, int32_t count) +{ + struct iovec iov_copy[count]; + ssize_t len; + int32_t total; + + memcpy(iov_copy, iov, sizeof(struct iovec) * count); + iov = iov_copy; + + total = 0; + while (count > 0) { + len = writev(sd, iov, count); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "Failed to send data"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENOBUFS, "Partial write"); + } + total += len; + + while ((count > 0) && (iov->iov_len <= len)) { + len -= iov->iov_len; + iov++; + count--; + } + + if (count > 0) { + iov->iov_base += len; + iov->iov_len -= len; + } + } + + return total; +} + +int32_t proxy_link_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + struct iovec iov_copy[count]; + ssize_t len; + int32_t total; + + memcpy(iov_copy, iov, sizeof(struct iovec) * count); + iov = iov_copy; + + total = 0; + while (count > 0) { + len = readv(sd, iov, count); + if (len < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to receive data"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENODATA, "Partial read"); + } + total += len; + + while ((count > 0) && (iov->iov_len <= len)) { + len -= iov->iov_len; + iov++; + count--; + } + + if (count > 0) { + iov->iov_base += len; + iov->iov_len -= len; + } + } + + return total; +} + +int32_t proxy_link_req_send(int32_t sd, int32_t op, struct iovec *iov, + int32_t count) +{ + proxy_link_req_t *req; + + req = iov[0].iov_base; + + req->header_len = iov[0].iov_len; + req->op = op; + req->data_len = iov_length(iov + 1, count - 1); + + return proxy_link_send(sd, iov, count); +} + +int32_t proxy_link_req_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + proxy_link_req_t *req; + void *buffer; + int32_t err, len, total; + + len = iov->iov_len; + iov->iov_len = sizeof(proxy_link_req_t); + err = proxy_link_recv(sd, iov, 1); + if (err < 0) { + return err; + } + total = err; + + req = iov->iov_base; + + if (req->data_len > 0) { + if (count == 1) { + return proxy_log(LOG_ERR, ENOBUFS, + "Request data is too long"); + } + if (iov[1].iov_len < req->data_len) { + buffer = proxy_malloc(req->data_len); + if (buffer == NULL) { + return -ENOMEM; + } + iov[1].iov_base = buffer; + } + iov[1].iov_len = req->data_len; + } else { + count = 1; + } + + if (req->header_len > sizeof(proxy_link_req_t)) { + if (len < req->header_len) { + return proxy_log(LOG_ERR, ENOBUFS, + "Request is too long"); + } + iov->iov_base += sizeof(proxy_link_req_t); + iov->iov_len = req->header_len - sizeof(proxy_link_req_t); + } else { + iov++; + count--; + if (count == 0) { + return total; + } + } + + err = proxy_link_recv(sd, iov, count); + if (err < 0) { + return err; + } + + return total + err; +} + +int32_t proxy_link_ans_send(int32_t sd, int32_t result, struct iovec *iov, + int32_t count) +{ + proxy_link_ans_t *ans; + + ans = iov->iov_base; + + ans->header_len = iov->iov_len; + ans->flags = 0; + ans->result = result; + ans->data_len = iov_length(iov + 1, count - 1); + + return proxy_link_send(sd, iov, count); +} + +int32_t proxy_link_ans_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + proxy_link_ans_t *ans; + int32_t err, len, total; + + len = iov->iov_len; + iov->iov_len = sizeof(proxy_link_ans_t); + err = proxy_link_recv(sd, iov, 1); + if (err < 0) { + return err; + } + total = err; + + ans = iov->iov_base; + + if (ans->data_len > 0) { + if ((count == 1) || (iov[1].iov_len < ans->data_len)) { + return proxy_log(LOG_ERR, ENOBUFS, + "Answer data is too long"); + } + iov[1].iov_len = ans->data_len; + } else { + count = 1; + } + + if (ans->header_len > sizeof(proxy_link_ans_t)) { + if (len < ans->header_len) { + return proxy_log(LOG_ERR, ENOBUFS, + "Answer is too long"); + } + iov->iov_base += sizeof(proxy_link_ans_t); + iov->iov_len = ans->header_len - sizeof(proxy_link_ans_t); + } else { + iov++; + count--; + if (count == 0) { + return total; + } + } + + err = proxy_link_recv(sd, iov, count); + if (err < 0) { + return err; + } + + return total + err; +} + +int32_t proxy_link_request(int32_t sd, int32_t op, struct iovec *req_iov, + int32_t req_count, struct iovec *ans_iov, + int32_t ans_count) +{ + int32_t err; + + err = proxy_link_req_send(sd, op, req_iov, req_count); + if (err < 0) { + return err; + } + + return proxy_link_ans_recv(sd, ans_iov, ans_count); +} diff --git a/src/libcephfs_proxy/proxy_link.h b/src/libcephfs_proxy/proxy_link.h new file mode 100644 index 00000000000..01a32d94377 --- /dev/null +++ b/src/libcephfs_proxy/proxy_link.h @@ -0,0 +1,67 @@ + +#ifndef __LIBCEPHFS_PROXY_LINK_H__ +#define __LIBCEPHFS_PROXY_LINK_H__ + +#include <sys/socket.h> +#include <sys/un.h> + +#include "proxy.h" + +#define PROXY_LINK_DISCONNECTED { NULL, -1 } + +struct _proxy_link { + proxy_link_stop_t stop; + int32_t sd; +}; + +typedef struct _proxy_link_req { + uint16_t header_len; + uint16_t op; + uint32_t data_len; +} proxy_link_req_t; + +typedef struct _proxy_link_ans { + uint16_t header_len; + uint16_t flags; + int32_t result; + uint32_t data_len; +} proxy_link_ans_t; + +static inline bool proxy_link_is_connected(proxy_link_t *link) +{ + return link->sd >= 0; +} + +int32_t proxy_link_client(proxy_link_t *link, const char *path, + proxy_link_stop_t stop); + +void proxy_link_close(proxy_link_t *link); + +int32_t proxy_link_server(proxy_link_t *link, const char *path, + proxy_link_start_t start, proxy_link_stop_t stop); + +int32_t proxy_link_read(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size); + +int32_t proxy_link_write(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size); + +int32_t proxy_link_send(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_req_send(int32_t sd, int32_t op, struct iovec *iov, + int32_t count); + +int32_t proxy_link_req_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_ans_send(int32_t sd, int32_t result, struct iovec *iov, + int32_t count); + +int32_t proxy_link_ans_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_request(int32_t sd, int32_t op, struct iovec *req_iov, + int32_t req_count, struct iovec *ans_iov, + int32_t ans_count); + +#endif diff --git a/src/libcephfs_proxy/proxy_list.h b/src/libcephfs_proxy/proxy_list.h new file mode 100644 index 00000000000..3dcb2ff9791 --- /dev/null +++ b/src/libcephfs_proxy/proxy_list.h @@ -0,0 +1,121 @@ + +#ifndef __LIBCEPHFS_PROXY_LIST_H__ +#define __LIBCEPHFS_PROXY_LIST_H__ + +#include "proxy.h" + +#define LIST_INIT(_list) { _list, _list } + +#define list_entry(_ptr, _type, _field) container_of(_ptr, _type, _field) + +#define list_first_entry(_list, _type, _field) \ + list_entry((_list)->next, _type, _field) + +#define list_last_entry(_list, _type, _field) \ + list_entry((_list)->prev, _type, _field) + +#define list_next_entry(_ptr, _field) \ + list_first_entry(&_ptr->_field, __typeof(*_ptr), _field) + +#define list_for_each_entry(_ptr, _list, _field) \ + for (_ptr = list_first_entry(_list, __typeof(*_ptr), _field); \ + &_ptr->_field != _list; _ptr = list_next_entry(_ptr, _field)) + +static inline void list_init(list_t *list) +{ + list->next = list; + list->prev = list; +} + +static inline bool list_empty(list_t *list) +{ + return list->next == list; +} + +static inline void list_add_between(list_t *item, list_t *prev, list_t *next) +{ + item->next = next; + item->prev = prev; + prev->next = item; + next->prev = item; +} + +static inline void list_add(list_t *item, list_t *list) +{ + list_add_between(item, list, list->next); +} + +static inline void list_add_tail(list_t *item, list_t *list) +{ + list_add_between(item, list->prev, list); +} + +static inline void list_del(list_t *list) +{ + list->next->prev = list->prev; + list->prev->next = list->next; +} + +static inline void list_del_init(list_t *list) +{ + list_del(list); + list_init(list); +} + +static inline void list_move(list_t *item, list_t *list) +{ + list_del(item); + list_add(item, list); +} + +static inline void list_move_tail(list_t *item, list_t *list) +{ + list_del(item); + list_add_tail(item, list); +} + +static inline void list_splice_between(list_t *src, list_t *prev, list_t *next) +{ + list_t *first, *last; + + first = src->next; + last = src->prev; + + first->prev = prev; + prev->next = first; + + last->next = next; + next->prev = last; +} + +static inline void list_splice(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst, dst->next); + } +} + +static inline void list_splice_tail(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst->prev, dst); + } +} + +static inline void list_splice_init(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst, dst->next); + list_init(src); + } +} + +static inline void list_splice_tail_init(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst->prev, dst); + list_init(src); + } +} + +#endif diff --git a/src/libcephfs_proxy/proxy_log.c b/src/libcephfs_proxy/proxy_log.c new file mode 100644 index 00000000000..dc1afed63de --- /dev/null +++ b/src/libcephfs_proxy/proxy_log.c @@ -0,0 +1,110 @@ + +#include <stdio.h> +#include <stdarg.h> + +#include "proxy_log.h" +#include "proxy_helpers.h" +#include "proxy_list.h" + +#define PROXY_LOG_BUFFER_SIZE 4096 + +static __thread char proxy_log_buffer[PROXY_LOG_BUFFER_SIZE]; + +static pthread_rwlock_t proxy_log_mutex = PTHREAD_RWLOCK_INITIALIZER; +static list_t proxy_log_handlers = LIST_INIT(&proxy_log_handlers); + +static void proxy_log_write(int32_t level, int32_t err, const char *msg) +{ + proxy_log_handler_t *handler; + + proxy_rwmutex_rdlock(&proxy_log_mutex); + + list_for_each_entry(handler, &proxy_log_handlers, list) { + handler->callback(handler, level, err, msg); + } + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +__public void proxy_log_register(proxy_log_handler_t *handler, + proxy_log_callback_t callback) +{ + handler->callback = callback; + + proxy_rwmutex_wrlock(&proxy_log_mutex); + + list_add_tail(&handler->list, &proxy_log_handlers); + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +__public void proxy_log_deregister(proxy_log_handler_t *handler) +{ + proxy_rwmutex_wrlock(&proxy_log_mutex); + + list_del_init(&handler->list); + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +static void proxy_log_msg(char *buffer, const char *text) +{ + int32_t len; + + len = strlen(text) + 1; + + memcpy(buffer, text, len); +} + +int32_t proxy_log_args(int32_t level, int32_t err, const char *fmt, + va_list args) +{ + static __thread bool busy = false; + int32_t len; + + if (busy) { + return -err; + } + busy = true; + + len = vsnprintf(proxy_log_buffer, sizeof(proxy_log_buffer), fmt, args); + if (len < 0) { + proxy_log_msg(proxy_log_buffer, + "<log message formatting failed>"); + } else if (len >= sizeof(proxy_log_buffer)) { + proxy_log_msg(proxy_log_buffer + sizeof(proxy_log_buffer) - 6, + "[...]"); + } + + proxy_log_write(level, err, proxy_log_buffer); + + busy = false; + + return -err; +} + +int32_t proxy_log(int32_t level, int32_t err, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + err = proxy_log_args(level, err, fmt, args); + va_end(args); + + return err; +} + +void proxy_abort_args(int32_t err, const char *fmt, va_list args) +{ + proxy_log_args(LOG_CRIT, err, fmt, args); + abort(); +} + +void proxy_abort(int32_t err, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + proxy_abort_args(err, fmt, args); + va_end(args); +} diff --git a/src/libcephfs_proxy/proxy_log.h b/src/libcephfs_proxy/proxy_log.h new file mode 100644 index 00000000000..02f45f9b110 --- /dev/null +++ b/src/libcephfs_proxy/proxy_log.h @@ -0,0 +1,28 @@ + +#ifndef __LIBCEPHFSD_PROXY_LOG_H__ +#define __LIBCEPHFSD_PROXY_LOG_H__ + +#include "proxy.h" + +enum { LOG_CRIT, LOG_ERR, LOG_WARN, LOG_INFO, LOG_DBG }; + +struct _proxy_log_handler { + list_t list; + proxy_log_callback_t callback; +}; + +int32_t proxy_log_args(int32_t level, int32_t err, const char *fmt, + va_list args); + +int32_t proxy_log(int32_t level, int32_t err, const char *fmt, ...); + +void proxy_abort_args(int32_t err, const char *fmt, va_list args); + +void proxy_abort(int32_t err, const char *fmt, ...); + +void proxy_log_register(proxy_log_handler_t *handler, + proxy_log_callback_t callback); + +void proxy_log_deregister(proxy_log_handler_t *handler); + +#endif diff --git a/src/libcephfs_proxy/proxy_manager.c b/src/libcephfs_proxy/proxy_manager.c new file mode 100644 index 00000000000..ea57083e700 --- /dev/null +++ b/src/libcephfs_proxy/proxy_manager.c @@ -0,0 +1,247 @@ + +#include <signal.h> + +#include "proxy_manager.h" +#include "proxy_helpers.h" +#include "proxy_list.h" +#include "proxy_log.h" + +static void proxy_manager_signal_handler(int32_t signum, siginfo_t *info, + void *ctx) +{ +} + +static void proxy_worker_register(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + list_add_tail(&worker->list, &manager->workers); + + proxy_mutex_unlock(&manager->mutex); +} + +static void proxy_worker_deregister(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + list_del_init(&worker->list); + if (list_empty(&manager->workers)) { + proxy_condition_signal(&manager->condition); + } + + proxy_mutex_unlock(&manager->mutex); +} + +static void proxy_worker_finished(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + if (list_empty(&manager->finished)) { + proxy_condition_signal(&manager->condition); + } + + list_move_tail(&worker->list, &manager->finished); + + proxy_mutex_unlock(&manager->mutex); +} + +static void *proxy_worker_start(void *arg) +{ + proxy_worker_t *worker; + + worker = arg; + + worker->start(worker); + + proxy_worker_finished(worker); + + return NULL; +} + +static void *proxy_manager_start(void *arg) +{ + proxy_manager_t *manager; + proxy_worker_t *worker; + + manager = arg; + + proxy_mutex_lock(&manager->mutex); + + while (true) { + while (!list_empty(&manager->finished)) { + worker = list_first_entry(&manager->finished, + proxy_worker_t, list); + list_del_init(&worker->list); + + proxy_mutex_unlock(&manager->mutex); + + proxy_thread_join(worker->tid); + + if (worker->destroy != NULL) { + worker->destroy(worker); + } + + proxy_mutex_lock(&manager->mutex); + } + + if (manager->stop && list_empty(&manager->workers)) { + break; + } + + proxy_condition_wait(&manager->condition, &manager->mutex); + } + + manager->done = true; + proxy_condition_signal(&manager->condition); + + proxy_mutex_unlock(&manager->mutex); + + return NULL; +} + +static int32_t proxy_manager_init(proxy_manager_t *manager) +{ + int32_t err; + + list_init(&manager->workers); + list_init(&manager->finished); + + manager->stop = false; + manager->done = false; + + manager->main_tid = pthread_self(); + + err = proxy_mutex_init(&manager->mutex); + if (err < 0) { + return err; + } + + err = proxy_condition_init(&manager->condition); + if (err < 0) { + pthread_mutex_destroy(&manager->mutex); + } + + return err; +} + +static void proxy_manager_destroy(proxy_manager_t *manager) +{ + pthread_cond_destroy(&manager->condition); + pthread_mutex_destroy(&manager->mutex); +} + +static int32_t proxy_manager_setup_signals(struct sigaction *old) +{ + struct sigaction action; + + /* The CONT signal will be used to wake threads blocked in I/O. */ + memset(&action, 0, sizeof(action)); + action.sa_flags = SA_SIGINFO; + action.sa_sigaction = proxy_manager_signal_handler; + + return proxy_signal_set(SIGCONT, &action, old); +} + +static void proxy_manager_restore_signals(struct sigaction *action) +{ + proxy_signal_set(SIGCONT, action, NULL); +} + +static void proxy_manager_terminate(proxy_manager_t *manager) +{ + proxy_worker_t *worker; + + proxy_mutex_lock(&manager->mutex); + + list_for_each_entry(worker, &manager->workers, list) { + worker->stop = true; + proxy_thread_kill(worker->tid, SIGCONT); + } + + while (!manager->done) { + proxy_condition_wait(&manager->condition, &manager->mutex); + } + + proxy_mutex_unlock(&manager->mutex); + + proxy_thread_join(manager->tid); +} + +int32_t proxy_manager_run(proxy_manager_t *manager, proxy_manager_start_t start) +{ + struct sigaction old_action; + int32_t err; + + err = proxy_manager_init(manager); + if (err < 0) { + return err; + } + + err = proxy_manager_setup_signals(&old_action); + if (err < 0) { + goto done_destroy; + } + + err = proxy_thread_create(&manager->tid, proxy_manager_start, manager); + if (err < 0) { + goto done_signal; + } + + err = start(manager); + + proxy_manager_terminate(manager); + +done_signal: + proxy_manager_restore_signals(&old_action); + +done_destroy: + proxy_manager_destroy(manager); + + return err; +} + +void proxy_manager_shutdown(proxy_manager_t *manager) +{ + proxy_mutex_lock(&manager->mutex); + + manager->stop = true; + proxy_condition_signal(&manager->condition); + + proxy_mutex_unlock(&manager->mutex); + + /* Wake the thread if it was blocked in an I/O operation. */ + proxy_thread_kill(manager->main_tid, SIGCONT); +} + +int32_t proxy_manager_launch(proxy_manager_t *manager, proxy_worker_t *worker, + proxy_worker_start_t start, + proxy_worker_destroy_t destroy) +{ + int32_t err; + + worker->manager = manager; + worker->start = start; + worker->destroy = destroy; + worker->stop = false; + + proxy_worker_register(worker); + + err = proxy_thread_create(&worker->tid, proxy_worker_start, worker); + if (err < 0) { + proxy_worker_deregister(worker); + } + + return err; +} diff --git a/src/libcephfs_proxy/proxy_manager.h b/src/libcephfs_proxy/proxy_manager.h new file mode 100644 index 00000000000..6a539be8d5b --- /dev/null +++ b/src/libcephfs_proxy/proxy_manager.h @@ -0,0 +1,43 @@ + +#ifndef __LIBCEPHFSD_PROXY_MANAGER_H__ +#define __LIBCEPHFSD_PROXY_MANAGER_H__ + +#include <pthread.h> + +#include "proxy.h" + +struct _proxy_worker { + list_t list; + pthread_t tid; + proxy_manager_t *manager; + proxy_worker_start_t start; + proxy_worker_destroy_t destroy; + bool stop; +}; + +struct _proxy_manager { + list_t workers; + list_t finished; + pthread_t main_tid; + pthread_t tid; + pthread_mutex_t mutex; + pthread_cond_t condition; + bool stop; + bool done; +}; + +int32_t proxy_manager_run(proxy_manager_t *manager, + proxy_manager_start_t start); + +void proxy_manager_shutdown(proxy_manager_t *manager); + +int32_t proxy_manager_launch(proxy_manager_t *manager, proxy_worker_t *worker, + proxy_worker_start_t start, + proxy_worker_destroy_t destroy); + +static inline bool proxy_manager_stop(proxy_manager_t *manager) +{ + return manager->stop; +} + +#endif diff --git a/src/libcephfs_proxy/proxy_mount.c b/src/libcephfs_proxy/proxy_mount.c new file mode 100644 index 00000000000..abfef1232c2 --- /dev/null +++ b/src/libcephfs_proxy/proxy_mount.c @@ -0,0 +1,1246 @@ + +#include "proxy_mount.h" +#include "proxy_helpers.h" + +#include <stdio.h> +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +/* Maximum number of symlinks to visit while resolving a path before returning + * ELOOP. */ +#define PROXY_MAX_SYMLINKS 16 + +struct _proxy_linked_str; +typedef struct _proxy_linked_str proxy_linked_str_t; + +/* This structure is used to handle symlinks found during the walk of a path. + * + * We'll start with an initial string representing a path. If one of the + * components is found to be a symlink, a new proxy_linked_str_t will be + * created with the content of the symlink. Then the new string will point + * to the old string, which may still contain some additional path components. + * The new string will be traversed resolving symlinks as they are found in the + * same way. Once it finished, the old string is recovered and traversal + * continues from the point it was left. */ +struct _proxy_linked_str { + proxy_linked_str_t *next; + char *remaining; + char data[]; +}; + +/* This structure is used to traverse a path while resolving any symlink + * found. At the end, it will contain the realpath of the entry and its + * inode. */ +typedef struct _proxy_path_iterator { + struct ceph_statx stx; + struct ceph_mount_info *cmount; + proxy_linked_str_t *lstr; + UserPerm *perms; + struct Inode *root; + struct Inode *base; + char *realpath; + uint64_t root_ino; + uint64_t base_ino; + uint32_t realpath_size; + uint32_t realpath_len; + uint32_t symlinks; + bool release; + bool follow; +} proxy_path_iterator_t; + +typedef struct _proxy_config { + int32_t src; + int32_t dst; + int32_t size; + int32_t total; + void *buffer; +} proxy_config_t; + +typedef struct _proxy_change { + list_t list; + uint32_t size; + char data[]; +} proxy_change_t; + +typedef struct _proxy_iter { + proxy_instance_t *instance; + list_t *item; +} proxy_iter_t; + +typedef struct _proxy_instance_pool { + pthread_mutex_t mutex; + list_t hash[256]; +} proxy_mount_pool_t; + +static proxy_mount_pool_t instance_pool = { + .mutex = PTHREAD_MUTEX_INITIALIZER, +}; + +/* Ceph client instance sharing + * + * The main purpose of the libcephfs proxy is to avoid the multiple independent + * data caches that are created when libcephfs is used from different processes. + * However the cache is not created per process but per client instance, so each + * call to `ceph_create()` creates its own private data cache instance. Just + * forwarding the libcephfs API calls to a single proxy process is not enough to + * solve the problem. + * + * The proxy will try to reuse existing client instances to reduce the number of + * independent caches. However it's not always possible to map all proxy clients + * to a single libcephfs instance. When different settings are used, separate + * Ceph instances are required to avoid unwanted behaviors. + * + * Even though it's possible that some Ceph options may be compatible even if + * they have different values, the proxy won't try to handle these cases. It + * will consider the configuration as a black box, and only 100% equal + * configurations will share the Ceph client instance. + */ + +/* Ceph configuration file management + * + * We won't try to parse Ceph configuration files. The proxy only wants to know + * if a configuration is equal or not. To do so, when a configuration file is + * passed to the proxy, it will create a private copy and compute an SHA256 + * hash. If the hash doesn't match, the configuration is considered different, + * even if it's not a real difference (like additional empty lines or the order + * of the options). + * + * The private copy is necessary to enforce that the settings are not changed + * concurrently, which could make us believe that two configurations are equal + * when they are not. + * + * Besides a configuration file, the user can also make manual configuration + * changes by using `ceph_conf_set()`. These changes are also tracked and + * compared to be sure that the active configuration matches. Only if the + * configuration file is exactly equal and all the applied changes are the same, + * and in the same order, the Ceph client instance will be shared. + */ + +int32_t proxy_inode_ref(proxy_mount_t *mount, uint64_t inode) +{ + inodeno_t ino; + struct Inode *tmp; + int32_t err; + + /* There's no way to tell libcephfs to increase the reference counter of + * an inode, so we do a full lookup for now. */ + + ino.val = inode; + + err = ceph_ll_lookup_inode(proxy_cmount(mount), ino, &tmp); + if (err < 0) { + proxy_log(LOG_ERR, -err, "ceph_ll_loolkup_inode() failed"); + } + + return err; +} + +static proxy_linked_str_t *proxy_linked_str_create(const char *str, + proxy_linked_str_t *next) +{ + proxy_linked_str_t *lstr; + uint32_t len; + + len = strlen(str) + 1; + lstr = proxy_malloc(sizeof(proxy_linked_str_t) + len); + if (lstr != NULL) { + lstr->next = next; + if (len > 1) { + lstr->remaining = lstr->data; + memcpy(lstr->data, str, len); + } else { + lstr->remaining = NULL; + } + } + + return lstr; +} + +static proxy_linked_str_t *proxy_linked_str_next(proxy_linked_str_t *lstr) +{ + proxy_linked_str_t *next; + + next = lstr->next; + proxy_free(lstr); + + return next; +} + +static void proxy_linked_str_destroy(proxy_linked_str_t *lstr) +{ + while (lstr != NULL) { + lstr = proxy_linked_str_next(lstr); + } +} + +static bool proxy_linked_str_empty(proxy_linked_str_t *lstr) +{ + return lstr->remaining == NULL; +} + +static char *proxy_linked_str_scan(proxy_linked_str_t *lstr, char ch) +{ + char *current; + + current = lstr->remaining; + lstr->remaining = strchr(lstr->remaining, ch); + if (lstr->remaining != NULL) { + *lstr->remaining++ = 0; + } + + return current; +} + +static int32_t proxy_path_iterator_init(proxy_path_iterator_t *iter, + proxy_mount_t *mount, const char *path, + UserPerm *perms, bool realpath, + bool follow) +{ + uint32_t len; + char ch; + + if (path == NULL) { + return proxy_log(LOG_ERR, EINVAL, "NULL path received"); + } + + memset(&iter->stx, 0, sizeof(iter->stx)); + iter->cmount = proxy_cmount(mount); + iter->perms = perms; + iter->root = mount->root; + iter->root_ino = mount->root_ino; + iter->base = mount->cwd; + iter->base_ino = mount->cwd_ino; + iter->symlinks = 0; + iter->release = false; + iter->follow = follow; + + len = strlen(path) + 1; + + ch = *path; + if (ch == '/') { + iter->base = mount->root; + iter->base_ino = mount->root_ino; + path++; + } + + iter->realpath = NULL; + iter->realpath_len = 0; + iter->realpath_size = 0; + + if (realpath) { + if (ch != '/') { + len += mount->cwd_path_len; + } + len = (len + 63) & ~63; + iter->realpath_size = len; + + iter->realpath = proxy_malloc(len); + if (iter->realpath == NULL) { + return -ENOMEM; + } + if (ch != '/') { + memcpy(iter->realpath, mount->cwd_path, + mount->cwd_path_len + 1); + iter->realpath_len = mount->cwd_path_len; + } else { + iter->realpath[0] = '/'; + iter->realpath[1] = 0; + iter->realpath_len = 1; + } + } + + iter->lstr = proxy_linked_str_create(path, NULL); + if (iter->lstr == NULL) { + proxy_free(iter->realpath); + return -ENOMEM; + } + + return 0; +} + +static char *proxy_path_iterator_next(proxy_path_iterator_t *iter) +{ + while (proxy_linked_str_empty(iter->lstr)) { + iter->lstr = proxy_linked_str_next(iter->lstr); + if (iter->lstr == NULL) { + return NULL; + } + } + + return proxy_linked_str_scan(iter->lstr, '/'); +} + +static bool proxy_path_iterator_is_last(proxy_path_iterator_t *iter) +{ + proxy_linked_str_t *lstr; + + lstr = iter->lstr; + while (proxy_linked_str_empty(iter->lstr)) { + lstr = lstr->next; + if (lstr == NULL) { + return true; + } + } + + return false; +} + +static void proxy_path_iterator_destroy(proxy_path_iterator_t *iter) +{ + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + + proxy_free(iter->realpath); + proxy_linked_str_destroy(iter->lstr); +} + +static int32_t proxy_path_iterator_resolve(proxy_path_iterator_t *iter) +{ + static __thread char path[PATH_MAX]; + proxy_linked_str_t *lstr; + char *ptr; + int32_t err; + + if (++iter->symlinks > PROXY_MAX_SYMLINKS) { + return proxy_log(LOG_ERR, ELOOP, "Too many symbolic links"); + } + + err = ceph_ll_readlink(iter->cmount, iter->base, path, sizeof(path), + iter->perms); + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_ll_readlink() failed"); + } + + ptr = path; + if (*ptr == '/') { + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + iter->base = iter->root; + iter->base_ino = iter->root_ino; + iter->release = false; + if (iter->realpath != NULL) { + iter->realpath[1] = 0; + iter->realpath_len = 1; + } + + ptr++; + } + + lstr = proxy_linked_str_create(ptr, iter->lstr); + if (lstr == NULL) { + return -ENOMEM; + } + iter->lstr = lstr; + + return 0; +} + +static int32_t proxy_path_iterator_append(proxy_path_iterator_t *iter, + const char *name) +{ + uint32_t len, size; + int32_t err; + + len = strlen(name) + 1; + size = iter->realpath_size; + if (iter->realpath_len + len >= size) { + do { + size <<= 1; + } while (iter->realpath_len + len >= size); + err = proxy_realloc((void **)&iter->realpath, size); + if (err < 0) { + return err; + } + iter->realpath_size = size; + } + + if (iter->realpath_len > 1) { + iter->realpath[iter->realpath_len++] = '/'; + } + memcpy(iter->realpath + iter->realpath_len, name, len); + iter->realpath_len += len - 1; + + return 0; +} + +static void proxy_path_iterator_remove(proxy_path_iterator_t *iter) +{ + while ((iter->realpath_len > 0) && + (iter->realpath[--iter->realpath_len] != '/')) { + } +} + +static int32_t proxy_path_lookup(struct ceph_mount_info *cmount, + struct Inode *parent, const char *name, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms) +{ + int32_t err; + + err = ceph_ll_lookup(cmount, parent, name, inode, stx, want, flags, + perms); + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_ll_lookup() failed"); + } + + return err; +} + +static int32_t proxy_path_iterator_lookup(proxy_path_iterator_t *iter, + const char *name) +{ + struct Inode *inode; + int32_t err; + + if (S_ISLNK(iter->stx.stx_mode)) { + return proxy_path_iterator_resolve(iter); + } + + err = proxy_path_lookup(iter->cmount, iter->base, name, &inode, + &iter->stx, CEPH_STATX_INO | CEPH_STATX_MODE, + AT_SYMLINK_NOFOLLOW, iter->perms); + if (err < 0) { + return err; + } + + if (iter->realpath != NULL) { + if ((name[0] == '.') && (name[1] == '.') && (name[2] == 0)) { + proxy_path_iterator_remove(iter); + } else { + err = proxy_path_iterator_append(iter, name); + if (err < 0) { + ceph_ll_put(iter->cmount, inode); + return err; + } + } + } + + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + iter->base = inode; + iter->base_ino = iter->stx.stx_ino; + iter->release = true; + + if (iter->follow && S_ISLNK(iter->stx.stx_mode) && + proxy_path_iterator_is_last(iter)) { + return proxy_path_iterator_resolve(iter); + } + + return 0; +} + +/* Implements a path walk ensuring that it's not possible to go higher than the + * root mount point used in ceph_mount(). This means that it handles absolute + * paths and ".." entries in a special way, including paths found in symbolic + * links. */ +int32_t proxy_path_resolve(proxy_mount_t *mount, const char *path, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms, + char **realpath) +{ + proxy_path_iterator_t iter; + char *name, c; + int32_t err; + + err = proxy_path_iterator_init(&iter, mount, path, perms, + realpath != NULL, + (flags & AT_SYMLINK_NOFOLLOW) == 0); + if (err < 0) { + return err; + } + + while ((err >= 0) && + ((name = proxy_path_iterator_next(&iter)) != NULL)) { + c = *name; + if (c == '.') { + c = name[1]; + if ((c == '.') && (iter.base == mount->root)) { + c = name[2]; + } + } + if (c == 0) { + continue; + } + + err = proxy_path_iterator_lookup(&iter, name); + } + + if (err >= 0) { + err = proxy_path_lookup(proxy_cmount(mount), iter.base, ".", + inode, stx, want, flags, iter.perms); + } + + if ((err >= 0) && (realpath != NULL)) { + *realpath = iter.realpath; + iter.realpath = NULL; + } + + proxy_path_iterator_destroy(&iter); + + return err; +} + +static int32_t proxy_config_source_prepare(const char *config, struct stat *st) +{ + int32_t fd, err; + + fd = open(config, O_RDONLY); + if (fd < 0) { + return proxy_log(LOG_ERR, errno, "open() failed"); + } + + if (fstat(fd, st) < 0) { + err = proxy_log(LOG_ERR, errno, "fstat() failed"); + goto failed; + } + + if (!S_ISREG(st->st_mode)) { + err = proxy_log(LOG_ERR, EINVAL, + "Configuration file is not a regular file"); + goto failed; + } + + return fd; + +failed: + close(fd); + + return err; +} + +static void proxy_config_source_close(int32_t fd) +{ + close(fd); +} + +static int32_t proxy_config_source_read(int32_t fd, void *buffer, size_t size) +{ + ssize_t len; + + len = read(fd, buffer, size); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "read() failed"); + } + + return len; +} + +static int32_t proxy_config_source_validate(int32_t fd, struct stat *before, + int32_t size) +{ + struct stat after; + + if (fstat(fd, &after) < 0) { + return proxy_log(LOG_ERR, errno, "fstat() failed"); + } + + if ((before->st_size != size) || (before->st_size != after.st_size) || + (before->st_blocks != after.st_blocks) || + (before->st_ctim.tv_sec != after.st_ctim.tv_sec) || + (before->st_ctim.tv_nsec != after.st_ctim.tv_nsec) || + (before->st_mtim.tv_sec != after.st_mtim.tv_sec) || + (before->st_mtim.tv_nsec != after.st_mtim.tv_nsec)) { + proxy_log(LOG_WARN, 0, + "Configuration file has been modified while " + "reading it"); + + return 0; + } + + return 1; +} + +static int32_t proxy_config_destination_prepare(void) +{ + int32_t fd; + + fd = openat(AT_FDCWD, ".", O_TMPFILE | O_WRONLY, 0600); + if (fd < 0) { + return proxy_log(LOG_ERR, errno, "openat() failed"); + } + + return fd; +} + +static void proxy_config_destination_close(int32_t fd) +{ + close(fd); +} + +static int32_t proxy_config_destination_write(int32_t fd, void *data, + size_t size) +{ + ssize_t len; + + len = write(fd, data, size); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "write() failed"); + } + if (len != size) { + return proxy_log(LOG_ERR, ENOSPC, "Partial write"); + } + + return size; +} + +static int32_t proxy_config_destination_commit(int32_t fd, const char *name) +{ + char path[32]; + + if (fsync(fd) < 0) { + return proxy_log(LOG_ERR, errno, "fsync() failed"); + } + + if (linkat(fd, "", AT_FDCWD, name, AT_EMPTY_PATH) < 0) { + if (errno == EEXIST) { + return 0; + } + + /* This may fail if the user doesn't have CAP_DAC_READ_SEARCH. + * In this case we attempt to link it using the /proc + * filesystem. */ + } + + snprintf(path, sizeof(path), "/proc/self/fd/%d", fd); + if (linkat(AT_FDCWD, path, AT_FDCWD, name, AT_SYMLINK_FOLLOW) < 0) { + if (errno != EEXIST) { + return proxy_log(LOG_ERR, errno, "linkat() failed"); + } + } + + return 0; +} + +static int32_t proxy_config_transfer(void **ptr, void *data, int32_t idx) +{ + proxy_config_t *cfg; + int32_t len, err; + + cfg = data; + + len = proxy_config_source_read(cfg->src, cfg->buffer, cfg->size); + if (len <= 0) { + return len; + } + + err = proxy_config_destination_write(cfg->dst, cfg->buffer, len); + if (err < 0) { + return err; + } + + cfg->total += len; + + *ptr = cfg->buffer; + + return len; +} + +/* Copies and checksums a given configuration to a file and makes sure that it + * has not been modified. */ +static int32_t proxy_config_prepare(const char *config, char *path, + int32_t size) +{ + char hash[65]; + proxy_config_t cfg; + struct stat before; + int32_t err; + + cfg.size = 4096; + cfg.buffer = proxy_malloc(cfg.size); + if (cfg.buffer == NULL) { + return -ENOMEM; + } + cfg.total = 0; + + cfg.src = proxy_config_source_prepare(config, &before); + if (cfg.src < 0) { + err = cfg.src; + goto done_mem; + } + + cfg.dst = proxy_config_destination_prepare(); + if (cfg.dst < 0) { + err = cfg.dst; + goto done_src; + } + + err = proxy_hash_hex(hash, sizeof(hash), proxy_config_transfer, &cfg); + if (err < 0) { + goto done_dst; + } + + err = proxy_config_source_validate(cfg.src, &before, cfg.total); + if (err < 0) { + goto done_dst; + } + + err = snprintf(path, size, "ceph-%s.conf", hash); + if (err < 0) { + err = proxy_log(LOG_ERR, errno, "snprintf() failed"); + goto done_dst; + } + if (err >= size) { + err = proxy_log(LOG_ERR, ENOBUFS, + "Insufficient space to store the name"); + goto done_dst; + } + + err = proxy_config_destination_commit(cfg.dst, path); + +done_dst: + proxy_config_destination_close(cfg.dst); + +done_src: + proxy_config_source_close(cfg.src); + +done_mem: + proxy_free(cfg.buffer); + + return err; +} + +/* Record changes to the configuration. */ +static int32_t proxy_instance_change_add(proxy_instance_t *instance, + const char *arg1, const char *arg2, + const char *arg3) +{ + proxy_change_t *change; + int32_t len[3], total; + + len[0] = strlen(arg1) + 1; + if (arg2 == NULL) { + arg2 = "<null>"; + } + len[1] = strlen(arg2) + 1; + len[2] = 0; + if (arg3 != NULL) { + len[2] = strlen(arg3) + 1; + } + + total = len[0] + len[1] + len[2]; + + change = proxy_malloc(sizeof(proxy_change_t) + total); + if (change == NULL) { + return -ENOMEM; + } + change->size = total; + + memcpy(change->data, arg1, len[0]); + memcpy(change->data + len[0], arg2, len[1]); + if (arg3 != NULL) { + memcpy(change->data + len[0] + len[1], arg3, len[2]); + } + + list_add_tail(&change->list, &instance->changes); + + return 0; +} + +static void proxy_instance_change_del(proxy_instance_t *instance) +{ + proxy_change_t *change; + + change = list_last_entry(&instance->changes, proxy_change_t, list); + list_del(&change->list); + + proxy_free(change); +} + +/* Destroy a Ceph client instance */ +static void proxy_instance_destroy(proxy_instance_t *instance) +{ + if (instance->mounted) { + ceph_unmount(instance->cmount); + } + + if (instance->cmount != NULL) { + ceph_release(instance->cmount); + } + + while (!list_empty(&instance->changes)) { + proxy_instance_change_del(instance); + } + + proxy_free(instance); +} + +/* Create a new Ceph client instance with the provided id */ +static int32_t proxy_instance_create(proxy_instance_t **pinstance, + const char *id) +{ + struct ceph_mount_info *cmount; + proxy_instance_t *instance; + int32_t err; + + instance = proxy_malloc(sizeof(proxy_instance_t)); + if (instance == NULL) { + return -ENOMEM; + } + + list_init(&instance->siblings); + list_init(&instance->changes); + instance->cmount = NULL; + instance->inited = false; + instance->mounted = false; + + err = proxy_instance_change_add(instance, "id", id, NULL); + if (err < 0) { + goto failed; + } + + err = ceph_create(&cmount, id); + if (err < 0) { + proxy_log(LOG_ERR, -err, "ceph_create() failed"); + goto failed; + } + + instance->cmount = cmount; + + *pinstance = instance; + + return 0; + +failed: + proxy_instance_destroy(instance); + + return err; +} + +static int32_t proxy_instance_release(proxy_instance_t *instance) +{ + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot release an active connection"); + } + + proxy_instance_destroy(instance); + + return 0; +} + +/* Assign a configuration file to the instance. */ +static int32_t proxy_instance_config(proxy_instance_t *instance, + const char *config) +{ + char path[128], *ppath; + int32_t err; + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot configure a mounted instance"); + } + + ppath = NULL; + if (config != NULL) { + err = proxy_config_prepare(config, path, sizeof(path)); + if (err < 0) { + return err; + } + ppath = path; + } + + err = proxy_instance_change_add(instance, "conf", ppath, NULL); + if (err < 0) { + return err; + } + + err = ceph_conf_read_file(instance->cmount, ppath); + if (err < 0) { + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_option_get(proxy_instance_t *instance, + const char *name, char *value, + size_t size) +{ + int32_t err, res; + + if (name == NULL) { + return proxy_log(LOG_ERR, EINVAL, "NULL option name"); + } + + res = ceph_conf_get(instance->cmount, name, value, size); + if (res < 0) { + return proxy_log( + LOG_ERR, -res, + "Failed to get configuration from a client instance"); + } + + err = proxy_instance_change_add(instance, "get", name, value); + if (err < 0) { + return err; + } + + return res; +} + +static int32_t proxy_instance_option_set(proxy_instance_t *instance, + const char *name, const char *value) +{ + int32_t err; + + if ((name == NULL) || (value == NULL)) { + return proxy_log(LOG_ERR, EINVAL, "NULL value or option name"); + } + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot configure a mounted instance"); + } + + err = proxy_instance_change_add(instance, "set", name, value); + if (err < 0) { + return err; + } + + err = ceph_conf_set(instance->cmount, name, value); + if (err < 0) { + proxy_log(LOG_ERR, -err, + "Failed to configure a client instance"); + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_select(proxy_instance_t *instance, const char *fs) +{ + int32_t err; + + if (instance->mounted) { + return proxy_log( + LOG_ERR, EISCONN, + "Cannot select a filesystem on a mounted instance"); + } + + err = proxy_instance_change_add(instance, "fs", fs, NULL); + if (err < 0) { + return err; + } + + err = ceph_select_filesystem(instance->cmount, fs); + if (err < 0) { + proxy_log(LOG_ERR, -err, + "Failed to select a filesystem on a client instance"); + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_init(proxy_instance_t *instance) +{ + if (instance->mounted || instance->inited) { + return 0; + } + + /* ceph_init() does start several internal threads. However, an instance + * may not end up being mounted if the configuration matches with + * another mounted instance. Since ceph_mount() also calls ceph_init() + * if not already done, we avoid initializing it here to reduce resource + * consumption. */ + + instance->inited = true; + + return 0; +} + +static int32_t proxy_instance_hash(void **ptr, void *data, int32_t idx) +{ + proxy_iter_t *iter; + proxy_change_t *change; + + iter = data; + + if (iter->item == &iter->instance->changes) { + return 0; + } + + change = list_entry(iter->item, proxy_change_t, list); + iter->item = iter->item->next; + + *ptr = change->data; + + return change->size; +} + +/* Check if an existing instance matches the configuration used for the current + * one. If so, share the mount. Otherwise, create a new mount. */ +static int32_t proxy_instance_mount(proxy_instance_t **pinstance) +{ + proxy_instance_t *instance, *existing; + proxy_iter_t iter; + list_t *list; + int32_t err; + + instance = *pinstance; + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot mount and already mounted instance"); + } + + iter.instance = instance; + iter.item = instance->changes.next; + + /* Create a hash that includes all settings. */ + err = proxy_hash(instance->hash, sizeof(instance->hash), + proxy_instance_hash, &iter); + if (err < 0) { + return err; + } + + list = &instance_pool.hash[instance->hash[0]]; + + proxy_mutex_lock(&instance_pool.mutex); + + if (list->next == NULL) { + list_init(list); + } else { + list_for_each_entry(existing, list, list) { + if (memcmp(existing->hash, instance->hash, 32) == 0) { + /* A match has been found. Instead of destroying + * the current instance, it's stored as a + * sibling of the one found. It will be + * reassigned to an instance when someone + * unmounts. */ + list_add(&instance->list, &existing->siblings); + goto found; + } + } + } + + /* No matching instance has been found. Just create a new one. The root + * is always "/". Each virtual mount point will locally store its root + * path. */ + err = ceph_mount(instance->cmount, "/"); + if (err >= 0) { + err = ceph_ll_lookup_root(instance->cmount, &instance->root); + if (err >= 0) { + instance->inited = true; + instance->mounted = true; + list_add(&instance->list, list); + } else { + ceph_unmount(instance->cmount); + } + } + + existing = NULL; + +found: + proxy_mutex_unlock(&instance_pool.mutex); + + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_mount() failed"); + } + + if (existing != NULL) { + proxy_log(LOG_INFO, 0, "Shared a client instance (%p)", + existing); + *pinstance = existing; + } else { + proxy_log(LOG_INFO, 0, "Created a new client instance (%p)", + instance); + } + + return 0; +} + +static int32_t proxy_instance_unmount(proxy_instance_t **pinstance) +{ + proxy_instance_t *instance, *sibling; + int32_t err; + + instance = *pinstance; + + if (!instance->mounted) { + return proxy_log(LOG_ERR, ENOTCONN, + "Cannot unmount an already unmount instance"); + } + + sibling = NULL; + + proxy_mutex_lock(&instance_pool.mutex); + + if (list_empty(&instance->siblings)) { + /* This is the last mount using this instance. We unmount it. */ + list_del(&instance->list); + instance->mounted = false; + } else { + /* There are other mounts sharing this instance. Take one of the + * saved siblings, which share the exact same configuration but + * are not mounted, to assign it to the current mount. */ + sibling = list_first_entry(&instance->siblings, + proxy_instance_t, list); + list_del_init(&sibling->list); + } + + proxy_mutex_unlock(&instance_pool.mutex); + + if (sibling == NULL) { + ceph_ll_put(instance->cmount, instance->root); + + err = ceph_unmount(instance->cmount); + if (err < 0) { + return proxy_log(LOG_ERR, -err, + "ceph_unmount() failed"); + } + } else { + *pinstance = sibling; + } + + return 0; +} + +int32_t proxy_mount_create(proxy_mount_t **pmount, const char *id) +{ + proxy_mount_t *mount; + int32_t err; + + mount = proxy_malloc(sizeof(proxy_mount_t)); + if (mount == NULL) { + return -ENOMEM; + } + mount->root = NULL; + + err = proxy_instance_create(&mount->instance, id); + if (err < 0) { + proxy_free(mount); + return err; + } + + *pmount = mount; + + return 0; +} + +int32_t proxy_mount_config(proxy_mount_t *mount, const char *config) +{ + return proxy_instance_config(mount->instance, config); +} + +int32_t proxy_mount_set(proxy_mount_t *mount, const char *name, + const char *value) +{ + return proxy_instance_option_set(mount->instance, name, value); +} + +int32_t proxy_mount_get(proxy_mount_t *mount, const char *name, char *value, + size_t size) +{ + return proxy_instance_option_get(mount->instance, name, value, size); +} + +int32_t proxy_mount_select(proxy_mount_t *mount, const char *fs) +{ + return proxy_instance_select(mount->instance, fs); +} + +int32_t proxy_mount_init(proxy_mount_t *mount) +{ + return proxy_instance_init(mount->instance); +} + +int32_t proxy_mount_mount(proxy_mount_t *mount, const char *root) +{ + struct ceph_statx stx; + struct ceph_mount_info *cmount; + int32_t err; + + err = proxy_instance_mount(&mount->instance); + if (err < 0) { + return err; + } + + cmount = proxy_cmount(mount); + + mount->perms = ceph_mount_perms(cmount); + + if (root == NULL) { + root = "/"; + } + + /* Temporarily set the root and cwd inodes to make proxy_path_resolve() + * to work correctly. */ + mount->root = mount->instance->root; + mount->root_ino = CEPH_INO_ROOT; + + mount->cwd = mount->instance->root; + mount->cwd_ino = CEPH_INO_ROOT; + + /* Resolve the desired root directory. */ + err = proxy_path_resolve(mount, root, &mount->root, &stx, + CEPH_STATX_ALL_STATS, 0, mount->perms, NULL); + if (err < 0) { + goto failed; + } + if (!S_ISDIR(stx.stx_mode)) { + err = proxy_log(LOG_ERR, ENOTDIR, + "The root path is not a directory"); + goto failed_root; + } + + mount->cwd_path = proxy_strdup("/"); + if (mount->cwd_path == NULL) { + err = -ENOMEM; + goto failed_root; + } + mount->cwd_path_len = 1; + + mount->root_ino = stx.stx_ino; + + err = proxy_inode_ref(mount, stx.stx_ino); + if (err < 0) { + goto failed_path; + } + + mount->cwd = mount->root; + mount->cwd_ino = stx.stx_ino; + + return 0; + +failed_path: + proxy_free(mount->cwd_path); + +failed_root: + ceph_ll_put(proxy_cmount(mount), mount->root); + +failed: + proxy_instance_unmount(&mount->instance); + + return err; +} + +int32_t proxy_mount_unmount(proxy_mount_t *mount) +{ + ceph_ll_put(proxy_cmount(mount), mount->root); + mount->root = NULL; + mount->root_ino = 0; + + ceph_ll_put(proxy_cmount(mount), mount->cwd); + mount->cwd = NULL; + mount->cwd_ino = 0; + + proxy_free(mount->cwd_path); + + return proxy_instance_unmount(&mount->instance); +} + +int32_t proxy_mount_release(proxy_mount_t *mount) +{ + int32_t err; + + err = proxy_instance_release(mount->instance); + if (err >= 0) { + proxy_free(mount); + } + + return err; +} diff --git a/src/libcephfs_proxy/proxy_mount.h b/src/libcephfs_proxy/proxy_mount.h new file mode 100644 index 00000000000..14bd58fabb2 --- /dev/null +++ b/src/libcephfs_proxy/proxy_mount.h @@ -0,0 +1,64 @@ + +#ifndef __LIBCEPHFSD_PROXY_MOUNT_H__ +#define __LIBCEPHFSD_PROXY_MOUNT_H__ + +#include "proxy.h" +#include "proxy_list.h" + +#include "include/cephfs/libcephfs.h" + +typedef struct _proxy_instance { + uint8_t hash[32]; + list_t list; + list_t siblings; + list_t changes; + struct ceph_mount_info *cmount; + struct Inode *root; + bool inited; + bool mounted; +} proxy_instance_t; + +typedef struct _proxy_mount { + proxy_instance_t *instance; + UserPerm *perms; + struct Inode *root; + struct Inode *cwd; + char *cwd_path; + uint64_t root_ino; + uint64_t cwd_ino; + uint32_t cwd_path_len; +} proxy_mount_t; + +static inline struct ceph_mount_info *proxy_cmount(proxy_mount_t *mount) +{ + return mount->instance->cmount; +} + +int32_t proxy_inode_ref(proxy_mount_t *mount, uint64_t inode); + +int32_t proxy_mount_create(proxy_mount_t **pmount, const char *id); + +int32_t proxy_mount_config(proxy_mount_t *mount, const char *config); + +int32_t proxy_mount_set(proxy_mount_t *mount, const char *name, + const char *value); + +int32_t proxy_mount_get(proxy_mount_t *mount, const char *name, char *value, + size_t size); + +int32_t proxy_mount_select(proxy_mount_t *mount, const char *fs); + +int32_t proxy_mount_init(proxy_mount_t *mount); + +int32_t proxy_mount_mount(proxy_mount_t *mount, const char *root); + +int32_t proxy_mount_unmount(proxy_mount_t *mount); + +int32_t proxy_mount_release(proxy_mount_t *mount); + +int32_t proxy_path_resolve(proxy_mount_t *mount, const char *path, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms, + char **realpath); + +#endif diff --git a/src/libcephfs_proxy/proxy_requests.h b/src/libcephfs_proxy/proxy_requests.h new file mode 100644 index 00000000000..4e3739276bb --- /dev/null +++ b/src/libcephfs_proxy/proxy_requests.h @@ -0,0 +1,343 @@ + +#ifndef __LIBCEPHFSD_PROXY_REQUESTS_H__ +#define __LIBCEPHFSD_PROXY_REQUESTS_H__ + +#include "proxy.h" +#include "proxy_link.h" + +/* Macros to add and get data from communication buffers. */ + +#define CEPH_BUFF_ADD(_data, _ptr, _size) \ + do { \ + _data##_iov[_data##_count].iov_base = (void *)(_ptr); \ + _data##_iov[_data##_count].iov_len = (_size); \ + _data##_count++; \ + } while (0) + +#define CEPH_DATA_ADD(_data, _field, _ptr, _size) \ + do { \ + (_data)._field = (_size); \ + CEPH_BUFF_ADD(_data, _ptr, (_data)._field); \ + } while (0) + +#define CEPH_STR_ADD(_data, _field, _str) \ + do { \ + if ((_str) != NULL) { \ + CEPH_DATA_ADD(_data, _field, _str, strlen(_str) + 1); \ + } else { \ + (_data)._field = 0; \ + } \ + } while (0) + +#define CEPH_STR_GET(_data, _field, _ptr) \ + ({ \ + const void *__ptr = (_ptr); \ + if ((_data)._field == 0) { \ + __ptr = NULL; \ + } \ + __ptr; \ + }) + +#define CEPH_DATA(_name, _data, _data_count) \ + proxy_##_name##_##_data##_t _data; \ + struct iovec _data##_iov[_data_count + 1]; \ + int32_t _data##_count = 0; \ + CEPH_BUFF_ADD(_data, &_data, sizeof(_data)) + +#define CEPH_REQ(_name, _req, _req_count, _ans, _ans_count) \ + CEPH_DATA(_name, _req, _req_count); \ + CEPH_DATA(_name, _ans, _ans_count) + +#define CEPH_CALL(_sd, _op, _req, _ans) \ + proxy_link_request((_sd), _op, _req##_iov, _req##_count, _ans##_iov, \ + _ans##_count) + +#define CEPH_RET(_sd, _res, _ans) \ + proxy_link_ans_send((_sd), (_res), _ans##_iov, _ans##_count) + +enum { + LIBCEPHFSD_OP_NULL = 0, + + LIBCEPHFSD_OP_VERSION, + LIBCEPHFSD_OP_USERPERM_NEW, + LIBCEPHFSD_OP_USERPERM_DESTROY, + LIBCEPHFSD_OP_CREATE, + LIBCEPHFSD_OP_RELEASE, + LIBCEPHFSD_OP_CONF_READ_FILE, + LIBCEPHFSD_OP_CONF_GET, + LIBCEPHFSD_OP_CONF_SET, + LIBCEPHFSD_OP_INIT, + LIBCEPHFSD_OP_SELECT_FILESYSTEM, + LIBCEPHFSD_OP_MOUNT, + LIBCEPHFSD_OP_UNMOUNT, + LIBCEPHFSD_OP_LL_STATFS, + LIBCEPHFSD_OP_LL_LOOKUP, + LIBCEPHFSD_OP_LL_LOOKUP_INODE, + LIBCEPHFSD_OP_LL_LOOKUP_ROOT, + LIBCEPHFSD_OP_LL_PUT, + LIBCEPHFSD_OP_LL_WALK, + LIBCEPHFSD_OP_CHDIR, + LIBCEPHFSD_OP_GETCWD, + LIBCEPHFSD_OP_READDIR, + LIBCEPHFSD_OP_REWINDDIR, + LIBCEPHFSD_OP_LL_OPEN, + LIBCEPHFSD_OP_LL_CREATE, + LIBCEPHFSD_OP_LL_MKNOD, + LIBCEPHFSD_OP_LL_CLOSE, + LIBCEPHFSD_OP_LL_RENAME, + LIBCEPHFSD_OP_LL_LSEEK, + LIBCEPHFSD_OP_LL_READ, + LIBCEPHFSD_OP_LL_WRITE, + LIBCEPHFSD_OP_LL_LINK, + LIBCEPHFSD_OP_LL_UNLINK, + LIBCEPHFSD_OP_LL_GETATTR, + LIBCEPHFSD_OP_LL_SETATTR, + LIBCEPHFSD_OP_LL_FALLOCATE, + LIBCEPHFSD_OP_LL_FSYNC, + LIBCEPHFSD_OP_LL_LISTXATTR, + LIBCEPHFSD_OP_LL_GETXATTR, + LIBCEPHFSD_OP_LL_SETXATTR, + LIBCEPHFSD_OP_LL_REMOVEXATTR, + LIBCEPHFSD_OP_LL_READLINK, + LIBCEPHFSD_OP_LL_SYMLINK, + LIBCEPHFSD_OP_LL_OPENDIR, + LIBCEPHFSD_OP_LL_MKDIR, + LIBCEPHFSD_OP_LL_RMDIR, + LIBCEPHFSD_OP_LL_RELEASEDIR, + LIBCEPHFSD_OP_MOUNT_PERMS, + + LIBCEPHFSD_OP_TOTAL_OPS +}; + +#define CEPH_TYPE_REQ(_name, _fields...) \ + struct _proxy_##_name##_req; \ + typedef struct _proxy_##_name##_req proxy_##_name##_req_t; \ + struct _proxy_##_name##_req { \ + _fields \ + } + +#define CEPH_TYPE_ANS(_name, _fields...) \ + struct _proxy_##_name##_ans; \ + typedef struct _proxy_##_name##_ans proxy_##_name##_ans_t; \ + struct _proxy_##_name##_ans { \ + _fields \ + } + +#define FIELDS(_fields...) _fields +#define REQ(_fields...) FIELDS(proxy_link_req_t header; _fields) +#define REQ_CMOUNT(_fields...) REQ(uint64_t cmount; _fields) +#define ANS(_fields...) FIELDS(proxy_link_ans_t header; _fields) +#define ANS_CMOUNT(_fields...) ANS(uint64_t cmount; _fields) + +#define CEPH_TYPE(_name, _req, _ans) \ + CEPH_TYPE_REQ(_name, _req); \ + CEPH_TYPE_ANS(_name, _ans) + +/* Declaration of types used to transder requests and answers. */ + +CEPH_TYPE(hello, FIELDS(uint32_t id;), FIELDS(int16_t major; int16_t minor;)); + +CEPH_TYPE(ceph_version, REQ(), + ANS(int32_t major; int32_t minor; int32_t patch; int16_t text;)); + +CEPH_TYPE(ceph_userperm_new, REQ(uint32_t uid; uint32_t gid; uint32_t groups;), + ANS(uint64_t userperm;)); + +CEPH_TYPE(ceph_userperm_destroy, REQ(uint64_t userperm;), ANS()); + +CEPH_TYPE(ceph_create, REQ(int16_t id;), ANS_CMOUNT()); + +CEPH_TYPE(ceph_release, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_conf_read_file, REQ_CMOUNT(uint16_t path;), ANS()); + +CEPH_TYPE(ceph_conf_get, REQ_CMOUNT(uint32_t size; uint16_t option;), + ANS(uint16_t value;)); + +CEPH_TYPE(ceph_conf_set, REQ_CMOUNT(uint16_t option; uint16_t value;), ANS()); + +CEPH_TYPE(ceph_init, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_select_filesystem, REQ_CMOUNT(uint16_t fs;), ANS()); + +CEPH_TYPE(ceph_mount, REQ_CMOUNT(uint16_t root;), ANS()); + +CEPH_TYPE(ceph_unmount, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_ll_statfs, REQ_CMOUNT(uint64_t inode;), ANS()); + +CEPH_TYPE(ceph_ll_lookup, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint32_t want; + uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_lookup_inode, REQ_CMOUNT(struct inodeno_t ino;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_lookup_root, REQ_CMOUNT(), ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_put, REQ_CMOUNT(uint64_t inode;), ANS()); + +CEPH_TYPE(ceph_ll_walk, + REQ_CMOUNT(uint64_t userperm; uint32_t want; uint32_t flags; + uint16_t path;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_chdir, REQ_CMOUNT(uint16_t path;), ANS()); + +CEPH_TYPE(ceph_getcwd, REQ_CMOUNT(), ANS(uint16_t path;)); + +CEPH_TYPE(ceph_readdir, REQ_CMOUNT(uint64_t dir;), ANS(bool eod;)); + +CEPH_TYPE(ceph_rewinddir, REQ_CMOUNT(uint64_t dir;), ANS()); + +CEPH_TYPE(ceph_ll_open, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; int32_t flags;), + ANS(uint64_t fh;)); + +CEPH_TYPE(ceph_ll_create, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + int32_t oflags; uint32_t want; uint32_t flags; + uint16_t name;), + ANS(uint64_t inode; uint64_t fh;)); + +CEPH_TYPE(ceph_ll_mknod, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + dev_t rdev; uint32_t want; uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_close, REQ_CMOUNT(uint64_t fh;), ANS()); + +CEPH_TYPE(ceph_ll_rename, + REQ_CMOUNT(uint64_t userperm; uint64_t old_parent; + uint64_t new_parent; uint16_t old_name; + uint16_t new_name;), + ANS()); + +CEPH_TYPE(ceph_ll_lseek, REQ_CMOUNT(uint64_t fh; off_t offset; int32_t whence;), + ANS(off_t offset;)); + +CEPH_TYPE(ceph_ll_read, REQ_CMOUNT(uint64_t fh; int64_t offset; uint64_t len;), + ANS()); + +CEPH_TYPE(ceph_ll_write, REQ_CMOUNT(uint64_t fh; int64_t offset; uint64_t len;), + ANS()); + +CEPH_TYPE(ceph_ll_link, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint64_t parent; + uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_unlink, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_getattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint32_t want; + uint32_t flags;), + ANS()); + +CEPH_TYPE(ceph_ll_setattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; int32_t mask;), ANS()); + +CEPH_TYPE(ceph_ll_fallocate, + REQ_CMOUNT(uint64_t fh; int64_t offset; int64_t length; + int32_t mode;), + ANS()); + +CEPH_TYPE(ceph_ll_fsync, REQ_CMOUNT(uint64_t fh; int32_t dataonly;), ANS()); + +CEPH_TYPE(ceph_ll_listxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size;), + ANS(size_t size;)); + +CEPH_TYPE(ceph_ll_getxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size; + uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_setxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size; + int32_t flags; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_removexattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint16_t name;), ANS()); + +CEPH_TYPE(ceph_ll_readlink, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size;), ANS()); + +CEPH_TYPE(ceph_ll_symlink, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint32_t want; + uint32_t flags; uint16_t name; uint16_t target;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_opendir, REQ_CMOUNT(uint64_t userperm; uint64_t inode;), + ANS(uint64_t dir;)); + +CEPH_TYPE(ceph_ll_mkdir, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + uint32_t want; uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_rmdir, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_releasedir, REQ_CMOUNT(uint64_t dir;), ANS()); + +CEPH_TYPE(ceph_mount_perms, REQ_CMOUNT(), ANS(uint64_t userperm;)); + +typedef union _proxy_req { + proxy_link_req_t header; + + proxy_ceph_version_req_t version; + proxy_ceph_userperm_new_req_t userperm_new; + proxy_ceph_userperm_destroy_req_t userperm_destroy; + proxy_ceph_create_req_t create; + proxy_ceph_release_req_t release; + proxy_ceph_conf_read_file_req_t conf_read_file; + proxy_ceph_conf_get_req_t conf_get; + proxy_ceph_conf_set_req_t conf_set; + proxy_ceph_init_req_t init; + proxy_ceph_select_filesystem_req_t select_filesystem; + proxy_ceph_mount_req_t mount; + proxy_ceph_unmount_req_t unmount; + proxy_ceph_ll_statfs_req_t ll_statfs; + proxy_ceph_ll_lookup_req_t ll_lookup; + proxy_ceph_ll_lookup_inode_req_t ll_lookup_inode; + proxy_ceph_ll_lookup_root_req_t ll_lookup_root; + proxy_ceph_ll_put_req_t ll_put; + proxy_ceph_ll_walk_req_t ll_walk; + proxy_ceph_chdir_req_t chdir; + proxy_ceph_getcwd_req_t getcwd; + proxy_ceph_readdir_req_t readdir; + proxy_ceph_rewinddir_req_t rewinddir; + proxy_ceph_ll_open_req_t ll_open; + proxy_ceph_ll_create_req_t ll_create; + proxy_ceph_ll_mknod_req_t ll_mknod; + proxy_ceph_ll_close_req_t ll_close; + proxy_ceph_ll_rename_req_t ll_rename; + proxy_ceph_ll_lseek_req_t ll_lseek; + proxy_ceph_ll_read_req_t ll_read; + proxy_ceph_ll_write_req_t ll_write; + proxy_ceph_ll_link_req_t ll_link; + proxy_ceph_ll_unlink_req_t ll_unlink; + proxy_ceph_ll_getattr_req_t ll_getattr; + proxy_ceph_ll_setattr_req_t ll_setattr; + proxy_ceph_ll_fallocate_req_t ll_fallocate; + proxy_ceph_ll_fsync_req_t ll_fsync; + proxy_ceph_ll_listxattr_req_t ll_listxattr; + proxy_ceph_ll_getxattr_req_t ll_getxattr; + proxy_ceph_ll_setxattr_req_t ll_setxattr; + proxy_ceph_ll_removexattr_req_t ll_removexattr; + proxy_ceph_ll_readlink_req_t ll_readlink; + proxy_ceph_ll_symlink_req_t ll_symlink; + proxy_ceph_ll_opendir_req_t ll_opendir; + proxy_ceph_ll_mkdir_req_t ll_mkdir; + proxy_ceph_ll_rmdir_req_t ll_rmdir; + proxy_ceph_ll_releasedir_req_t ll_releasedir; + proxy_ceph_mount_perms_req_t mount_perms; +} proxy_req_t; + +#endif diff --git a/src/librados/librados_asio.h b/src/librados/librados_asio.h index 0aedc376575..3e5b7c57c6f 100644 --- a/src/librados/librados_asio.h +++ b/src/librados/librados_asio.h @@ -14,6 +14,9 @@ #ifndef LIBRADOS_ASIO_H #define LIBRADOS_ASIO_H +#include <boost/asio/associated_cancellation_slot.hpp> +#include <boost/asio/cancellation_type.hpp> + #include "include/rados/librados.hpp" #include "common/async/completion.h" #include "librados/AioCompletionImpl.h" @@ -74,6 +77,7 @@ struct Invoker<void> { template <typename Result> struct AsyncOp : Invoker<Result> { unique_aio_completion_ptr aio_completion; + boost::asio::cancellation_slot slot; using Signature = typename Invoker<Result>::Signature; using Completion = ceph::async::Completion<Signature, AsyncOp<Result>>; @@ -83,6 +87,7 @@ struct AsyncOp : Invoker<Result> { auto p = std::unique_ptr<Completion>{static_cast<Completion*>(arg)}; // move result out of Completion memory being freed auto op = std::move(p->user_data); + op.slot.clear(); // clear our cancellation handler // access AioCompletionImpl directly to avoid locking const librados::AioCompletionImpl* pc = op.aio_completion->pc; const int ret = pc->rval; @@ -94,11 +99,46 @@ struct AsyncOp : Invoker<Result> { op.dispatch(std::move(p), ec, ver); } + struct op_cancellation { + AioCompletion* completion = nullptr; + bool is_read = false; + + void operator()(boost::asio::cancellation_type type) { + if (completion == nullptr) { + return; // no AioCompletion attached + } else if (type == boost::asio::cancellation_type::none) { + return; // no cancellation requested + } else if (is_read) { + // read operations produce no side effects, so can satisfy the + // requirements of 'total' cancellation. the weaker requirements + // of 'partial' and 'terminal' are also satisfied + completion->cancel(); + } else if (type == boost::asio::cancellation_type::terminal) { + // write operations only support 'terminal' cancellation because we + // can't guarantee that no osd has succeeded (or will succeed) in + // applying the write + completion->cancel(); + } + } + }; + template <typename Executor1, typename CompletionHandler> - static auto create(const Executor1& ex1, CompletionHandler&& handler) { + static auto create(const Executor1& ex1, bool is_read, + CompletionHandler&& handler) { + op_cancellation* cancel_handler = nullptr; + auto slot = boost::asio::get_associated_cancellation_slot(handler); + if (slot.is_connected()) { + cancel_handler = &slot.template emplace<op_cancellation>(); + } + auto p = Completion::create(ex1, std::move(handler)); p->user_data.aio_completion.reset( Rados::aio_create_completion(p.get(), aio_dispatch)); + if (cancel_handler) { + cancel_handler->completion = p->user_data.aio_completion.get(); + cancel_handler->is_read = is_read; + p->user_data.slot = std::move(slot); + } return p; } }; @@ -108,6 +148,9 @@ struct AsyncOp : Invoker<Result> { /// Calls IoCtx::aio_read() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, size_t len, uint64_t off, CompletionToken&& token) @@ -117,7 +160,8 @@ auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, size_t len, uint64_t off) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = true; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_read(oid, op.aio_completion.get(), &op.result, len, off); @@ -132,6 +176,9 @@ auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_write() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, const bufferlist &bl, size_t len, uint64_t off, @@ -142,7 +189,8 @@ auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, const bufferlist &bl, size_t len, uint64_t off) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_write(oid, op.aio_completion.get(), bl, len, off); @@ -157,6 +205,9 @@ auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_operate() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, ObjectReadOperation *read_op, int flags, @@ -167,7 +218,8 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, ObjectReadOperation *read_op, int flags) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = true; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_operate(oid, op.aio_completion.get(), read_op, @@ -183,6 +235,9 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_operate() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, ObjectWriteOperation *write_op, int flags, @@ -194,7 +249,8 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, [] (auto handler, auto ex, IoCtx& io, const std::string& oid, ObjectWriteOperation *write_op, int flags, const jspan_context* trace_ctx) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_operate(oid, op.aio_completion.get(), write_op, flags, trace_ctx); @@ -209,6 +265,9 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_notify() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_notify(ExecutionContext& ctx, IoCtx& io, const std::string& oid, bufferlist& bl, uint64_t timeout_ms, CompletionToken &&token) @@ -218,7 +277,8 @@ auto async_notify(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, bufferlist& bl, uint64_t timeout_ms) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_notify(oid, op.aio_completion.get(), diff --git a/src/librados/librados_cxx.cc b/src/librados/librados_cxx.cc index 2167eeade3c..60217b99b41 100644 --- a/src/librados/librados_cxx.cc +++ b/src/librados/librados_cxx.cc @@ -1103,6 +1103,14 @@ void librados::AioCompletion::release() delete this; } +int librados::AioCompletion::cancel() +{ + if (!pc->io) { + return 0; // no operation was started + } + return pc->io->aio_cancel(pc); +} + ///////////////////////////// IoCtx ////////////////////////////// librados::IoCtx::IoCtx() : io_ctx_impl(NULL) { diff --git a/src/librbd/ObjectMap.cc b/src/librbd/ObjectMap.cc index 65e3fc4a4c2..160bb4dcf9e 100644 --- a/src/librbd/ObjectMap.cc +++ b/src/librbd/ObjectMap.cc @@ -107,32 +107,6 @@ bool ObjectMap<I>::object_may_exist(uint64_t object_no) const } template <typename I> -bool ObjectMap<I>::object_may_not_exist(uint64_t object_no) const -{ - ceph_assert(ceph_mutex_is_locked(m_image_ctx.image_lock)); - - // Fall back to default logic if object map is disabled or invalid - if (!m_image_ctx.test_features(RBD_FEATURE_OBJECT_MAP, - m_image_ctx.image_lock)) { - return true; - } - - bool flags_set; - int r = m_image_ctx.test_flags(m_image_ctx.snap_id, - RBD_FLAG_OBJECT_MAP_INVALID, - m_image_ctx.image_lock, &flags_set); - if (r < 0 || flags_set) { - return true; - } - - uint8_t state = (*this)[object_no]; - bool nonexistent = (state != OBJECT_EXISTS && state != OBJECT_EXISTS_CLEAN); - ldout(m_image_ctx.cct, 20) << "object_no=" << object_no << " r=" - << nonexistent << dendl; - return nonexistent; -} - -template <typename I> bool ObjectMap<I>::update_required(const ceph::BitVector<2>::Iterator& it, uint8_t new_state) { ceph_assert(ceph_mutex_is_locked(m_lock)); diff --git a/src/librbd/ObjectMap.h b/src/librbd/ObjectMap.h index 35ea4cb88f9..5e7fcbbe9dd 100644 --- a/src/librbd/ObjectMap.h +++ b/src/librbd/ObjectMap.h @@ -65,7 +65,6 @@ public: void close(Context *on_finish); bool set_object_map(ceph::BitVector<2> &target_object_map); bool object_may_exist(uint64_t object_no) const; - bool object_may_not_exist(uint64_t object_no) const; void aio_save(Context *on_finish); void aio_resize(uint64_t new_size, uint8_t default_object_state, diff --git a/src/librbd/migration/HttpClient.cc b/src/librbd/migration/HttpClient.cc index 6a504d3a9ac..d212981a917 100644 --- a/src/librbd/migration/HttpClient.cc +++ b/src/librbd/migration/HttpClient.cc @@ -63,14 +63,13 @@ public: m_on_shutdown = on_finish; auto current_state = m_state; + m_state = STATE_SHUTTING_DOWN; + if (current_state == STATE_UNINITIALIZED) { // never initialized or resolve/connect failed on_finish->complete(0); return; - } - - m_state = STATE_SHUTTING_DOWN; - if (current_state != STATE_READY) { + } else if (current_state != STATE_READY) { // delay shutdown until current state transition completes return; } @@ -118,7 +117,7 @@ public: ceph_assert(m_http_client->m_strand.running_in_this_thread()); auto cct = m_http_client->m_cct; - ldout(cct, 20) << "work=" << work.get() << ", r=" << -ec.value() << dendl; + ldout(cct, 20) << "work=" << work.get() << ", ec=" << ec.what() << dendl; ceph_assert(m_in_flight_requests > 0); --m_in_flight_requests; @@ -187,13 +186,14 @@ protected: virtual void connect(boost::asio::ip::tcp::resolver::results_type results, Context* on_finish) = 0; virtual void disconnect(Context* on_finish) = 0; + virtual void reset_stream() = 0; void close_socket() { auto cct = m_http_client->m_cct; ldout(cct, 15) << dendl; boost::system::error_code ec; - boost::beast::get_lowest_layer(derived().stream()).socket().close(ec); + derived().stream().lowest_layer().close(ec); } private: @@ -229,7 +229,6 @@ private: auto cct = m_http_client->m_cct; ldout(cct, 15) << dendl; - shutdown_socket(); m_resolver.async_resolve( m_http_client->m_url_spec.host, m_http_client->m_url_spec.port, [this, on_finish](boost::system::error_code ec, auto results) { @@ -358,8 +357,7 @@ private: } int shutdown_socket() { - if (!boost::beast::get_lowest_layer( - derived().stream()).socket().is_open()) { + if (!derived().stream().lowest_layer().is_open()) { return 0; } @@ -367,7 +365,7 @@ private: ldout(cct, 15) << dendl; boost::system::error_code ec; - boost::beast::get_lowest_layer(derived().stream()).socket().shutdown( + derived().stream().lowest_layer().shutdown( boost::asio::ip::tcp::socket::shutdown_both, ec); if (ec && ec != boost::beast::errc::not_connected) { @@ -414,7 +412,7 @@ private: void handle_receive(boost::system::error_code ec, std::shared_ptr<Work>&& work) { auto cct = m_http_client->m_cct; - ldout(cct, 15) << "work=" << work.get() << ", r=" << -ec.value() << dendl; + ldout(cct, 15) << "work=" << work.get() << ", ec=" << ec.what() << dendl; ceph_assert(m_in_flight_requests > 0); --m_in_flight_requests; @@ -445,10 +443,10 @@ private: ldout(cct, 5) << "remote peer stream closed, retrying request" << dendl; m_receive_queue.push_front(work); } else if (ec == boost::beast::error::timeout) { - lderr(cct) << "timed-out while issuing request" << dendl; + lderr(cct) << "timed-out while receiving response" << dendl; work->complete(-ETIMEDOUT, {}); } else { - lderr(cct) << "failed to issue request: " << ec.message() << dendl; + lderr(cct) << "failed to receive response: " << ec.message() << dendl; work->complete(-ec.value(), {}); } @@ -473,7 +471,7 @@ private: r = -EACCES; } else if (boost::beast::http::to_status_class(result) != boost::beast::http::status_class::successful) { - lderr(cct) << "failed to retrieve size: HTTP " << result << dendl; + lderr(cct) << "failed to retrieve resource: HTTP " << result << dendl; r = -EIO; } @@ -501,7 +499,10 @@ private: << "next_state=" << next_state << ", " << "r=" << r << dendl; - m_state = next_state; + if (current_state != STATE_SHUTTING_DOWN) { + m_state = next_state; + } + if (current_state == STATE_CONNECTING) { if (next_state == STATE_UNINITIALIZED) { shutdown_socket(); @@ -512,14 +513,17 @@ private: return; } } else if (current_state == STATE_SHUTTING_DOWN) { + ceph_assert(m_on_shutdown != nullptr); if (next_state == STATE_READY) { // shut down requested while connecting/resetting disconnect(new LambdaContext([this](int r) { handle_shut_down(r); })); return; } else if (next_state == STATE_UNINITIALIZED || - next_state == STATE_SHUTDOWN || next_state == STATE_RESET_CONNECTING) { - ceph_assert(m_on_shutdown != nullptr); + shutdown_socket(); + m_on_shutdown->complete(r); + return; + } else if (next_state == STATE_SHUTDOWN) { m_on_shutdown->complete(r); return; } @@ -528,6 +532,7 @@ private: ceph_assert(next_state == STATE_RESET_CONNECTING); ceph_assert(on_finish == nullptr); shutdown_socket(); + reset_stream(); resolve_host(nullptr); return; } else if (current_state == STATE_RESET_CONNECTING) { @@ -589,7 +594,7 @@ public: this->close_socket(); } - inline boost::beast::tcp_stream& + inline boost::asio::ip::tcp::socket& stream() { return m_stream; } @@ -601,20 +606,25 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - m_stream.async_connect( - results, - [on_finish](boost::system::error_code ec, const auto& endpoint) { - on_finish->complete(-ec.value()); - }); + ceph_assert(!m_stream.is_open()); + boost::asio::async_connect(m_stream, + results, + [on_finish](boost::system::error_code ec, + const auto& endpoint) { + on_finish->complete(-ec.value()); + }); } void disconnect(Context* on_finish) override { on_finish->complete(0); } -private: - boost::beast::tcp_stream m_stream; + void reset_stream() override { + // no-op -- tcp_stream object can be reused after shut down + } +private: + boost::asio::ip::tcp::socket m_stream; }; #undef dout_prefix @@ -633,7 +643,7 @@ public: this->close_socket(); } - inline boost::beast::ssl_stream<boost::beast::tcp_stream>& + inline boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream() { return m_stream; } @@ -645,7 +655,9 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - boost::beast::get_lowest_layer(m_stream).async_connect( + ceph_assert(!m_stream.lowest_layer().is_open()); + async_connect( + m_stream.lowest_layer(), results, [this, on_finish](boost::system::error_code ec, const auto& endpoint) { handle_connect(-ec.value(), on_finish); @@ -657,19 +669,25 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - if (!m_ssl_enabled) { - on_finish->complete(0); - return; - } - m_stream.async_shutdown( - asio::util::get_callback_adapter([this, on_finish](int r) { - shutdown(r, on_finish); })); + [this, on_finish](boost::system::error_code ec) { + handle_disconnect(ec, on_finish); + }); + } + + void reset_stream() override { + auto http_client = this->m_http_client; + auto cct = http_client->m_cct; + ldout(cct, 15) << dendl; + + // ssl_stream object can't be reused after shut down -- move-in + // a freshly constructed instance + m_stream = boost::asio::ssl::stream<boost::asio::ip::tcp::socket>( + http_client->m_strand, http_client->m_ssl_context); } private: - boost::beast::ssl_stream<boost::beast::tcp_stream> m_stream; - bool m_ssl_enabled = false; + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> m_stream; void handle_connect(int r, Context* on_finish) { auto http_client = this->m_http_client; @@ -728,33 +746,38 @@ private: // Perform the SSL/TLS handshake m_stream.async_handshake( boost::asio::ssl::stream_base::client, - asio::util::get_callback_adapter( - [this, on_finish](int r) { handle_handshake(r, on_finish); })); + [this, on_finish](boost::system::error_code ec) { + handle_handshake(ec, on_finish); + }); } - void handle_handshake(int r, Context* on_finish) { + void handle_handshake(boost::system::error_code ec, Context* on_finish) { auto http_client = this->m_http_client; auto cct = http_client->m_cct; - ldout(cct, 15) << "r=" << r << dendl; + ldout(cct, 15) << "ec=" << ec.what() << dendl; - if (r < 0) { - lderr(cct) << "failed to complete handshake: " << cpp_strerror(r) + if (ec) { + lderr(cct) << "failed to complete SSL handshake: " << ec.message() << dendl; - disconnect(new LambdaContext([r, on_finish](int) { - on_finish->complete(r); })); + on_finish->complete(-ec.value()); return; } - m_ssl_enabled = true; on_finish->complete(0); } - void shutdown(int r, Context* on_finish) { + void handle_disconnect(boost::system::error_code ec, Context* on_finish) { auto http_client = this->m_http_client; auto cct = http_client->m_cct; - ldout(cct, 15) << "r=" << r << dendl; + ldout(cct, 15) << "ec=" << ec.what() << dendl; - on_finish->complete(r); + if (ec && ec != boost::asio::ssl::error::stream_truncated) { + lderr(cct) << "failed to shut down SSL: " << ec.message() << dendl; + on_finish->complete(-ec.value()); + return; + } + + on_finish->complete(0); } }; diff --git a/src/librbd/migration/HttpClient.h b/src/librbd/migration/HttpClient.h index 3997e6159e7..5844f918693 100644 --- a/src/librbd/migration/HttpClient.h +++ b/src/librbd/migration/HttpClient.h @@ -13,13 +13,12 @@ #include <boost/asio/strand.hpp> #include <boost/asio/ip/tcp.hpp> #include <boost/asio/ssl/context.hpp> +#include <boost/asio/ssl/stream.hpp> #include <boost/beast/version.hpp> -#include <boost/beast/core/tcp_stream.hpp> #include <boost/beast/http/empty_body.hpp> #include <boost/beast/http/message.hpp> #include <boost/beast/http/string_body.hpp> #include <boost/beast/http/write.hpp> -#include <boost/beast/ssl/ssl_stream.hpp> #include <functional> #include <memory> #include <string> @@ -97,7 +96,7 @@ public: completion(r, std::move(response)); } - void operator()(boost::beast::tcp_stream& stream) override { + void operator()(boost::asio::ip::tcp::socket& stream) override { preprocess_request(); boost::beast::http::async_write( @@ -110,7 +109,7 @@ public: } void operator()( - boost::beast::ssl_stream<boost::beast::tcp_stream>& stream) override { + boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream) override { preprocess_request(); boost::beast::http::async_write( @@ -152,9 +151,9 @@ private: virtual bool need_eof() const = 0; virtual bool header_only() const = 0; virtual void complete(int r, Response&&) = 0; - virtual void operator()(boost::beast::tcp_stream& stream) = 0; + virtual void operator()(boost::asio::ip::tcp::socket& stream) = 0; virtual void operator()( - boost::beast::ssl_stream<boost::beast::tcp_stream>& stream) = 0; + boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream) = 0; }; template <typename D> struct HttpSession; diff --git a/src/librbd/operation/FlattenRequest.cc b/src/librbd/operation/FlattenRequest.cc index 7bc34681924..8034637e8e6 100644 --- a/src/librbd/operation/FlattenRequest.cc +++ b/src/librbd/operation/FlattenRequest.cc @@ -49,15 +49,6 @@ public: return -ERESTART; } - { - std::shared_lock image_lock{image_ctx.image_lock}; - if (image_ctx.object_map != nullptr && - !image_ctx.object_map->object_may_not_exist(m_object_no)) { - // can skip because the object already exists - return 1; - } - } - if (!io::util::trigger_copyup( &image_ctx, m_object_no, m_io_context, this)) { // stop early if the parent went away - it just means diff --git a/src/mds/Beacon.cc b/src/mds/Beacon.cc index 6fbfc79d416..1c1eeb4ecf8 100644 --- a/src/mds/Beacon.cc +++ b/src/mds/Beacon.cc @@ -26,6 +26,7 @@ #include "mds/MDSRank.h" #include "mds/MDSMap.h" #include "mds/Locker.h" +#include "mds/mdstypes.h" #include "Beacon.h" @@ -550,6 +551,19 @@ void Beacon::notify_health(MDSRank const *mds) } } } + if (mds->is_replay()) { + CachedStackStringStream css; + auto estimate = mds->mdlog->get_estimated_replay_finish_time(); + // this probably should be configurable, however, its fine to report + // if replay is running for more than 30 seconds. + if (estimate.elapsed_time > std::chrono::seconds(30)) { + *css << "replay: " << estimate.percent_complete << "% complete - elapsed time: " + << estimate.elapsed_time << ", estimated time remaining: " + << estimate.estimated_time; + MDSHealthMetric m(MDS_HEALTH_ESTIMATED_REPLAY_TIME, HEALTH_WARN, css->strv()); + health.metrics.push_back(m); + } + } } MDSMap::DaemonState Beacon::get_want_state() const diff --git a/src/mds/MDLog.cc b/src/mds/MDLog.cc index 4bbf2a1a141..d041e3b2fc8 100644 --- a/src/mds/MDLog.cc +++ b/src/mds/MDLog.cc @@ -221,7 +221,46 @@ uint64_t MDLog::get_safe_pos() const return journaler->get_write_safe_pos(); } +// estimate the replay completion time based on mdlog journal pointers +EstimatedReplayTime MDLog::get_estimated_replay_finish_time() { + ceph_assert(mds->is_replay()); + EstimatedReplayTime estimated_time{0, std::chrono::seconds::zero(), std::chrono::seconds::zero()}; + if (!journaler) { + return estimated_time; + } + + auto read_pos = journaler->get_read_pos(); + auto write_pos = journaler->get_write_pos(); + auto trimmed_pos = journaler->get_trimmed_pos(); + + dout(20) << __func__ << ": read_pos=" << read_pos << ", write_pos=" + << write_pos << ", trimmed_pos=" << trimmed_pos << dendl; + + if (read_pos == trimmed_pos || write_pos == trimmed_pos) { + return estimated_time; + } + + auto total_bytes = write_pos - trimmed_pos; + double percent_complete = ((double)(read_pos - trimmed_pos)) / (double)total_bytes; + auto elapsed_time = std::chrono::duration_cast<std::chrono::seconds> + (ceph::coarse_mono_clock::now() - replay_start_time); + auto time = ((1 - percent_complete) / percent_complete) * elapsed_time; + + dout(20) << __func__ << "percent_complete=" << percent_complete + << ", elapsed_time=" << elapsed_time + << ", estimated_time=" << std::chrono::round<std::chrono::seconds>(time) + << dendl; + + estimated_time.percent_complete = percent_complete * 100; + estimated_time.elapsed_time = elapsed_time; + estimated_time.estimated_time = std::chrono::round<std::chrono::seconds>(time); + dout(20) << __func__ << "estimated_time.percent_complete=" << estimated_time.percent_complete + << ", estimated_time.elapsed_time=" << estimated_time.elapsed_time + << ", estimated_time.estimated_time=" << estimated_time.estimated_time + << dendl; + return estimated_time; +} void MDLog::create(MDSContext *c) { @@ -1137,6 +1176,7 @@ void MDLog::_recovery_thread(MDSContext *completion) { std::lock_guard l(mds->mds_lock); journaler = front_journal; + replay_start_time = ceph::coarse_mono_clock::now(); } C_SaferCond recover_wait; @@ -1374,11 +1414,17 @@ void MDLog::_reformat_journal(JournalPointer const &jp_in, Journaler *old_journa // i am a separate thread void MDLog::_replay_thread() { - dout(10) << "_replay_thread start" << dendl; + dout(10) << __func__ << ": start time: " << replay_start_time << ", now: " + << ceph::coarse_mono_clock::now() << dendl; // loop int r = 0; while (1) { + auto sleep_time = g_conf().get_val<std::chrono::milliseconds>("mds_delay_journal_replay_for_testing"); + if (unlikely(sleep_time > 0ms)) { + dout(10) << __func__ << ": sleeping for " << sleep_time << "ms" << dendl; + std::this_thread::sleep_for(sleep_time); + } // wait for read? journaler->check_isreadable(); if (journaler->get_error()) { diff --git a/src/mds/MDLog.h b/src/mds/MDLog.h index a858b40fa03..180a34c9d82 100644 --- a/src/mds/MDLog.h +++ b/src/mds/MDLog.h @@ -53,6 +53,7 @@ enum { #include "LogSegment.h" #include "MDSMap.h" #include "SegmentBoundary.h" +#include "mdstypes.h" #include <list> #include <map> @@ -162,6 +163,7 @@ public: void reopen(MDSContext *onopen); void append(); void replay(MDSContext *onfinish); + EstimatedReplayTime get_estimated_replay_finish_time(); void standby_trim_segments(); @@ -328,5 +330,7 @@ private: std::atomic<bool> upkeep_log_trim_shutdown{false}; std::map<uint64_t, std::vector<Context*>> waiting_for_expire; // protected by mds_lock + + ceph::coarse_mono_time replay_start_time = ceph::coarse_mono_clock::zero(); }; #endif diff --git a/src/mds/MetricsHandler.cc b/src/mds/MetricsHandler.cc index 9fc4c6122a4..d9c09e06b27 100644 --- a/src/mds/MetricsHandler.cc +++ b/src/mds/MetricsHandler.cc @@ -20,15 +20,6 @@ MetricsHandler::MetricsHandler(CephContext *cct, MDSRank *mds) mds(mds) { } -bool MetricsHandler::ms_can_fast_dispatch2(const cref_t<Message> &m) const { - return m->get_type() == CEPH_MSG_CLIENT_METRICS || m->get_type() == MSG_MDS_PING; -} - -void MetricsHandler::ms_fast_dispatch2(const ref_t<Message> &m) { - bool handled = ms_dispatch2(m); - ceph_assert(handled); -} - bool MetricsHandler::ms_dispatch2(const ref_t<Message> &m) { if (m->get_type() == CEPH_MSG_CLIENT_METRICS && m->get_connection()->get_peer_type() == CEPH_ENTITY_TYPE_CLIENT) { diff --git a/src/mds/MetricsHandler.h b/src/mds/MetricsHandler.h index 0b75b024860..25ee208aa95 100644 --- a/src/mds/MetricsHandler.h +++ b/src/mds/MetricsHandler.h @@ -25,11 +25,6 @@ class MetricsHandler : public Dispatcher { public: MetricsHandler(CephContext *cct, MDSRank *mds); - bool ms_can_fast_dispatch_any() const override { - return true; - } - bool ms_can_fast_dispatch2(const cref_t<Message> &m) const override; - void ms_fast_dispatch2(const ref_t<Message> &m) override; bool ms_dispatch2(const ref_t<Message> &m) override; void ms_handle_connect(Connection *c) override { diff --git a/src/mds/mdstypes.cc b/src/mds/mdstypes.cc index 680218e62e3..f9424eed6dc 100644 --- a/src/mds/mdstypes.cc +++ b/src/mds/mdstypes.cc @@ -1042,3 +1042,8 @@ void snaprealm_reconnect_t::generate_test_instances(std::list<snaprealm_reconnec ls.back()->realm.seq = 2; ls.back()->realm.parent = 1; } + +void EstimatedReplayTime::print(std::ostream& out) { + out << "replay: " << percent_complete << "% complete - elapsed time: " + << elapsed_time << ", estimated time remaining: " << estimated_time; +} diff --git a/src/mds/mdstypes.h b/src/mds/mdstypes.h index 3b8269006cb..742d7b23432 100644 --- a/src/mds/mdstypes.h +++ b/src/mds/mdstypes.h @@ -1044,4 +1044,12 @@ inline bool operator==(const MDSCacheObjectInfo& l, const MDSCacheObjectInfo& r) } WRITE_CLASS_ENCODER(MDSCacheObjectInfo) +struct EstimatedReplayTime { + double percent_complete; + std::chrono::seconds estimated_time; + std::chrono::seconds elapsed_time; + + void print(std::ostream& out); +}; + #endif diff --git a/src/messages/MMDSBeacon.h b/src/messages/MMDSBeacon.h index c157c33e758..526285aae8c 100644 --- a/src/messages/MMDSBeacon.h +++ b/src/messages/MMDSBeacon.h @@ -48,6 +48,7 @@ enum mds_metric_t { MDS_HEALTH_CLIENTS_LAGGY, MDS_HEALTH_CLIENTS_LAGGY_MANY, MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH, + MDS_HEALTH_ESTIMATED_REPLAY_TIME, MDS_HEALTH_DUMMY, // not a real health warning, for testing }; @@ -69,6 +70,7 @@ inline const char *mds_metric_name(mds_metric_t m) case MDS_HEALTH_CLIENTS_LAGGY: return "MDS_CLIENTS_LAGGY"; case MDS_HEALTH_CLIENTS_LAGGY_MANY: return "MDS_CLIENTS_LAGGY_MANY"; case MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH: return "MDS_CLIENTS_BROKEN_ROOTSQUASH"; + case MDS_HEALTH_ESTIMATED_REPLAY_TIME: return "MDS_ESTIMATED_REPLAY_TIME"; case MDS_HEALTH_DUMMY: return "MDS_DUMMY"; default: return "???"; @@ -107,6 +109,8 @@ inline const char *mds_metric_summary(mds_metric_t m) return "%num% client(s) laggy due to laggy OSDs"; case MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH: return "%num% MDS report clients with broken root_squash implementation"; + case MDS_HEALTH_ESTIMATED_REPLAY_TIME: + return "%num% estimated journal replay time"; default: return "???"; } diff --git a/src/mgr/PyModule.cc b/src/mgr/PyModule.cc index cff63ef4a6b..4f996489ba0 100644 --- a/src/mgr/PyModule.cc +++ b/src/mgr/PyModule.cc @@ -38,6 +38,18 @@ std::string PyModule::mgr_store_prefix = "mgr/"; #define BOOST_BIND_GLOBAL_PLACEHOLDERS // Boost apparently can't be bothered to fix its own usage of its own // deprecated features. + +// Fix instances of "'BOOST_PP_ITERATION_02' was not declared in this scope; did +// you mean 'BOOST_PP_ITERATION_05'" and related macro error bullshit that spans +// 300 lines of errors +// +// Apparently you can't include boost/python stuff _and_ have this header +// defined +// +// Thanks to the ceph-aur folks for the fix at: +// https://github.com/bazaah/aur-ceph/commit/8c5cc7d8deec002f7596b6d0860859a0a718f12b +#undef BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS + #include <boost/python/extract.hpp> #include <boost/python/import.hpp> #include <boost/python/object.hpp> diff --git a/src/mgr/PyModule.h b/src/mgr/PyModule.h index 177447c2cb3..a47db3a47ef 100644 --- a/src/mgr/PyModule.h +++ b/src/mgr/PyModule.h @@ -161,9 +161,9 @@ public: } const std::string &get_name() const { - std::lock_guard l(lock) ; return module_name; + return module_name; } - const std::string &get_error_string() const { + std::string get_error_string() const { std::lock_guard l(lock) ; return error_string; } bool get_can_run() const { diff --git a/src/mon/FSCommands.cc b/src/mon/FSCommands.cc index 6220a357ff0..cc53d2869f7 100644 --- a/src/mon/FSCommands.cc +++ b/src/mon/FSCommands.cc @@ -1211,6 +1211,11 @@ class RemoveFilesystemHandler : public FileSystemCommandHandler fsmap.erase_filesystem(fsp->get_fscid()); + ss << "If there are active snapshot schedules associated with this " + << "file-system, you might see EIO errors in the mgr logs or at the " + << "snap-schedule command-line due to the missing file-system. " + << "However, these errors are transient and will get auto-resolved."; + return 0; } }; diff --git a/src/mon/NVMeofGwMap.cc b/src/mon/NVMeofGwMap.cc index 719403925ad..2d2735f1e7c 100755 --- a/src/mon/NVMeofGwMap.cc +++ b/src/mon/NVMeofGwMap.cc @@ -171,6 +171,8 @@ int NVMeofGwMap::cfg_delete_gw( << state.availability << " Resulting GW availability: " << state.availability << dendl; state.subsystems.clear();//ignore subsystems of this GW + utime_t now = ceph_clock_now(); + mon->nvmegwmon()->gws_deleting_time[group_key][gw_id] = now; return 0; } } @@ -895,10 +897,12 @@ struct CMonRequestProposal : public Context { } }; -void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const +void NVMeofGwMap::get_health_checks(health_check_map_t *checks) { list<string> singleGatewayDetail; list<string> gatewayDownDetail; + list<string> gatewayInDeletingDetail; + int deleting_gateways = 0; for (const auto& created_map_pair: created_gws) { const auto& group_key = created_map_pair.first; auto& group = group_key.second; @@ -915,9 +919,37 @@ void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const ostringstream ss; ss << "NVMeoF Gateway '" << gw_id << "' is unavailable." ; gatewayDownDetail.push_back(ss.str()); + } else if (gw_created.availability == gw_availability_t::GW_DELETING) { + deleting_gateways++; + utime_t now = ceph_clock_now(); + bool found_deleting_time = false; + auto gws_deleting_time = mon->nvmegwmon()->gws_deleting_time; + auto group_it = gws_deleting_time.find(group_key); + if (group_it != gws_deleting_time.end()) { + auto& gw_map = group_it->second; + auto gw_it = gw_map.find(gw_id); + if (gw_it != gw_map.end()) { + found_deleting_time = true; + utime_t delete_time = gw_it->second; + if ((now - delete_time) > g_conf().get_val<std::chrono::seconds>("mon_nvmeofgw_delete_grace").count()) { + ostringstream ss; + ss << "NVMeoF Gateway '" << gw_id << "' is in deleting state."; + gatewayInDeletingDetail.push_back(ss.str()); + } + } + } + if (!found_deleting_time) { + // DELETING gateway not found in gws_deleting_time, set timeout now + mon->nvmegwmon()->gws_deleting_time[group_key][gw_id] = now; + } } } } + if (deleting_gateways == 0) { + // no gateway in GW_DELETING state currently, flush old gws_deleting_time + mon->nvmegwmon()->gws_deleting_time.clear(); + } + if (!singleGatewayDetail.empty()) { ostringstream ss; ss << singleGatewayDetail.size() << " group(s) have only 1 nvmeof gateway" @@ -934,6 +966,15 @@ void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const ss.str(), gatewayDownDetail.size()); d.detail.swap(gatewayDownDetail); } + if (!gatewayInDeletingDetail.empty()) { + ostringstream ss; + ss << gatewayInDeletingDetail.size() << " gateway(s) are in deleting state" + << "; namespaces are automatically balanced across remaining gateways, " + << "this should take a few minutes."; + auto& d = checks->add("NVMEOF_GATEWAY_DELETING", HEALTH_WARN, + ss.str(), gatewayInDeletingDetail.size()); + d.detail.swap(gatewayInDeletingDetail); + } } int NVMeofGwMap::blocklist_gw( diff --git a/src/mon/NVMeofGwMap.h b/src/mon/NVMeofGwMap.h index 5f657733012..85fd62b3a07 100755 --- a/src/mon/NVMeofGwMap.h +++ b/src/mon/NVMeofGwMap.h @@ -144,7 +144,7 @@ public: DECODE_FINISH(bl); } - void get_health_checks(health_check_map_t *checks) const; + void get_health_checks(health_check_map_t *checks); }; #include "NVMeofGwSerialize.h" diff --git a/src/mon/NVMeofGwMon.cc b/src/mon/NVMeofGwMon.cc index 0fe5c3e655f..c9a6b789b89 100644 --- a/src/mon/NVMeofGwMon.cc +++ b/src/mon/NVMeofGwMon.cc @@ -66,11 +66,6 @@ void NVMeofGwMon::on_shutdown() void NVMeofGwMon::tick() { - if (++tick_ratio == 10) { - global_rebalance_index++; - dout(20) << "rebalance index " << global_rebalance_index << dendl; - tick_ratio = 0; - } if (!is_active() || !mon.is_leader()) { dout(10) << "NVMeofGwMon leader : " << mon.is_leader() << "active : " << is_active() << dendl; @@ -329,8 +324,9 @@ bool NVMeofGwMon::preprocess_command(MonOpRequestRef op) if (HAVE_FEATURE(mon.get_quorum_con_features(), NVMEOFHA)) { f->dump_string("features", "LB"); if (map.created_gws[group_key].size()) { - uint32_t index = (global_rebalance_index % - map.created_gws[group_key].size()) + 1; + time_t seconds_since_1970 = time(NULL); + uint32_t index = ((seconds_since_1970/60) % + map.created_gws[group_key].size()) + 1; f->dump_unsigned("rebalance_ana_group", index); } } @@ -625,15 +621,15 @@ bool NVMeofGwMon::prepare_beacon(MonOpRequestRef op) avail = gw_availability_t::GW_CREATED; dout(20) << "No-subsystems condition detected for GW " << gw_id <<dendl; } else { - bool listener_found = true; + bool listener_found = false; for (auto &subs: sub) { - if (subs.listeners.size() == 0) { - listener_found = false; - dout(10) << "No-listeners condition detected for GW " << gw_id << " for nqn " << subs.nqn << dendl; + if (subs.listeners.size()) { + listener_found = true; break; } } if (!listener_found) { + dout(10) << "No-listeners condition detected for GW " << gw_id << dendl; avail = gw_availability_t::GW_CREATED; } }// for HA no-subsystems and no-listeners are same usecases diff --git a/src/mon/NVMeofGwMon.h b/src/mon/NVMeofGwMon.h index 2d13e153bd2..d7f5fd89cde 100644 --- a/src/mon/NVMeofGwMon.h +++ b/src/mon/NVMeofGwMon.h @@ -82,10 +82,9 @@ public: void check_subs(bool type); void check_sub(Subscription *sub); + std::map<NvmeGroupKey, std::map<NvmeGwId, utime_t>> gws_deleting_time; + private: - // used for calculate pool & group GW responsible for rebalance - uint32_t global_rebalance_index = 1; - uint8_t tick_ratio = 0; void synchronize_last_beacon(); void process_gw_down(const NvmeGwId &gw_id, const NvmeGroupKey& group_key, bool &propose_pending, diff --git a/src/msg/async/Event.cc b/src/msg/async/Event.cc index 08e117ea54a..16abb1368b0 100644 --- a/src/msg/async/Event.cc +++ b/src/msg/async/Event.cc @@ -347,7 +347,7 @@ void EventCenter::wakeup() return ; ldout(cct, 20) << __func__ << dendl; - char buf = 'c'; + static constexpr char buf = 'c'; // wake up "event_wait" #ifdef _WIN32 int n = send(notify_send_fd, &buf, sizeof(buf), 0); diff --git a/src/nvmeof/NVMeofGwMonitorClient.cc b/src/nvmeof/NVMeofGwMonitorClient.cc index ce3328aec51..1b128055e08 100644 --- a/src/nvmeof/NVMeofGwMonitorClient.cc +++ b/src/nvmeof/NVMeofGwMonitorClient.cc @@ -42,7 +42,6 @@ NVMeofGwMonitorClient::NVMeofGwMonitorClient(int argc, const char **argv) : monc{g_ceph_context, poolctx}, client_messenger(Messenger::create(g_ceph_context, "async", entity_name_t::CLIENT(-1), "client", getpid())), objecter{g_ceph_context, client_messenger.get(), &monc, poolctx}, - client{client_messenger.get(), &monc, &objecter}, timer(g_ceph_context, beacon_lock), orig_argc(argc), orig_argv(argv) @@ -134,7 +133,6 @@ int NVMeofGwMonitorClient::init() // Initialize Messenger client_messenger->add_dispatcher_tail(this); client_messenger->add_dispatcher_head(&objecter); - client_messenger->add_dispatcher_tail(&client); client_messenger->start(); poolctx.start(2); @@ -190,7 +188,6 @@ int NVMeofGwMonitorClient::init() objecter.init(); objecter.enable_blocklist_events(); objecter.start(); - client.init(); timer.init(); { @@ -302,8 +299,7 @@ void NVMeofGwMonitorClient::shutdown() std::lock_guard bl(beacon_lock); timer.shutdown(); } - // client uses monc and objecter - client.shutdown(); + // Stop asio threads, so leftover events won't call into shut down // monclient/objecter. poolctx.finish(); diff --git a/src/nvmeof/NVMeofGwMonitorClient.h b/src/nvmeof/NVMeofGwMonitorClient.h index 6dd167e4e58..e01c823afb5 100644 --- a/src/nvmeof/NVMeofGwMonitorClient.h +++ b/src/nvmeof/NVMeofGwMonitorClient.h @@ -21,7 +21,6 @@ #include "common/Timer.h" #include "common/LogClient.h" -#include "client/Client.h" #include "mon/MonClient.h" #include "osdc/Objecter.h" #include "messages/MNVMeofGwMap.h" @@ -58,7 +57,6 @@ protected: MonClient monc; std::unique_ptr<Messenger> client_messenger; Objecter objecter; - Client client; std::map<NvmeGroupKey, NvmeGwMonClientStates> map; ceph::mutex lock = ceph::make_mutex("NVMeofGw::lock"); // allow beacons to be sent independently of handle_nvmeof_gw_map diff --git a/src/os/DBObjectMap.cc b/src/os/DBObjectMap.cc index 7da9a67be62..65627b5f818 100644 --- a/src/os/DBObjectMap.cc +++ b/src/os/DBObjectMap.cc @@ -519,6 +519,11 @@ bufferlist DBObjectMap::DBObjectMapIteratorImpl::value() return cur_iter->value(); } +std::string_view DBObjectMap::DBObjectMapIteratorImpl::value_as_sv() +{ + return cur_iter->value_as_sv(); +} + int DBObjectMap::DBObjectMapIteratorImpl::status() { return r; diff --git a/src/os/DBObjectMap.h b/src/os/DBObjectMap.h index 444f21eb815..1e1452010e7 100644 --- a/src/os/DBObjectMap.h +++ b/src/os/DBObjectMap.h @@ -393,6 +393,7 @@ private: int next() override { ceph_abort(); return 0; } std::string key() override { ceph_abort(); return ""; } ceph::buffer::list value() override { ceph_abort(); return ceph::buffer::list(); } + std::string_view value_as_sv() override { ceph_abort(); return std::string_view(); } int status() override { return 0; } }; @@ -431,6 +432,7 @@ private: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; int status() override; bool on_parent() { diff --git a/src/os/ObjectStore.h b/src/os/ObjectStore.h index 521435b6c31..df3ae920a2f 100644 --- a/src/os/ObjectStore.h +++ b/src/os/ObjectStore.h @@ -29,6 +29,7 @@ #include <errno.h> #include <sys/stat.h> +#include <functional> #include <map> #include <memory> #include <vector> @@ -735,15 +736,6 @@ public: std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) = 0; -#ifdef WITH_SEASTAR - virtual int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) = 0; -#endif - /// Filters keys into out which are defined on oid virtual int omap_check_keys( CollectionHandle &c, ///< [in] Collection containing oid @@ -766,6 +758,48 @@ public: const ghobject_t &oid ///< [in] object ) = 0; + struct omap_iter_seek_t { + std::string seek_position; + enum { + // start with provided key (seek_position), if it exists + LOWER_BOUND, + // skip provided key (seek_position) even if it exists + UPPER_BOUND + } seek_type = LOWER_BOUND; + static omap_iter_seek_t min_lower_bound() { return {}; } + }; + enum class omap_iter_ret_t { + STOP, + NEXT + }; + /** + * Iterate over object map with user-provided callable + * + * Warning! The callable is executed under lock on bluestore + * operations in c. Do not use bluestore methods on c while + * iterating. (Filling in a transaction is no problem). + * + * @param c collection + * @param oid object + * @param start_from where the iterator should point to at + * the beginning + * @param visitor callable that takes OMAP key and corresponding + * value as string_views and controls iteration + * by the return. It is executed for every object's + * OMAP entry from `start_from` till end of the + * object's OMAP or till the iteration is stopped + * by `STOP`. Please note that if there is no such + * entry, `visitor` will be called 0 times. + * @return error code, zero on success + */ + virtual int omap_iterate( + CollectionHandle &c, + const ghobject_t &oid, + omap_iter_seek_t start_from, + std::function<omap_iter_ret_t(std::string_view, + std::string_view)> visitor + ) = 0; + virtual int flush_journal() { return -EOPNOTSUPP; } virtual int dump_journal(std::ostream& out) { return -EOPNOTSUPP; } diff --git a/src/os/bluestore/BlueFS.cc b/src/os/bluestore/BlueFS.cc index 5f4f1a4d48a..2f88acdc93b 100644 --- a/src/os/bluestore/BlueFS.cc +++ b/src/os/bluestore/BlueFS.cc @@ -5,6 +5,7 @@ #include "bluestore_common.h" #include "BlueFS.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/debug.h" #include "common/errno.h" #include "common/perf_counters.h" @@ -12,6 +13,12 @@ #include "include/ceph_assert.h" #include "common/admin_socket.h" +#if defined(WITH_SEASTAR) && !defined(WITH_ALIEN) +#include "crimson/common/perf_counters_collection.h" +#else +#include "common/perf_counters_collection.h" +#endif + #define dout_context cct #define dout_subsys ceph_subsys_bluefs #undef dout_prefix @@ -1699,7 +1706,8 @@ int BlueFS::_replay(bool noop, bool to_stdout) << " fnode=" << fnode << " delta=" << delta << dendl; - ceph_assert(delta.offset == fnode.allocated); + // be leanient, if there is no extents just produce error message + ceph_assert(delta.offset == fnode.allocated || delta.extents.empty()); } if (cct->_conf->bluefs_log_replay_check_allocations) { int r = _check_allocations(fnode, @@ -3823,6 +3831,7 @@ int BlueFS::truncate(FileWriter *h, uint64_t offset)/*_WF_L*/ fnode.size = offset; fnode.allocated = new_allocated; fnode.reset_delta(); + fnode.recalc_allocated(); log.t.op_file_update(fnode); // sad, but is_dirty must be set to signal flushing of the log h->file->is_dirty = true; diff --git a/src/os/bluestore/BlueStore.cc b/src/os/bluestore/BlueStore.cc index a024a0c2105..25e6c4fe596 100644 --- a/src/os/bluestore/BlueStore.cc +++ b/src/os/bluestore/BlueStore.cc @@ -4830,7 +4830,7 @@ void BlueStore::Onode::rewrite_omap_key(const string& old, string *out) out->append(old.c_str() + out->length(), old.size() - out->length()); } -void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) +size_t BlueStore::Onode::calc_userkey_offset_in_omap_key() const { size_t pos = sizeof(uint64_t) + 1; if (!onode.is_pgmeta_omap()) { @@ -4840,9 +4840,15 @@ void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) pos += sizeof(uint64_t); } } - *user_key = key.substr(pos); + return pos; } +void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) +{ + *user_key = key.substr(calc_userkey_offset_in_omap_key()); +} + + void BlueStore::Onode::finish_write(TransContext* txc, uint32_t offset, uint32_t length) { while (true) { @@ -5519,7 +5525,13 @@ BlueStore::OmapIteratorImpl::OmapIteratorImpl( if (o->onode.has_omap()) { o->get_omap_key(string(), &head); o->get_omap_tail(&tail); + auto start1 = mono_clock::now(); it->lower_bound(head); + c->store->log_latency( + __func__, + l_bluestore_omap_seek_to_first_lat, + mono_clock::now() - start1, + c->store->cct->_conf->bluestore_log_omap_iterator_age); } } BlueStore::OmapIteratorImpl::~OmapIteratorImpl() @@ -5654,6 +5666,13 @@ bufferlist BlueStore::OmapIteratorImpl::value() return it->value(); } +std::string_view BlueStore::OmapIteratorImpl::value_as_sv() +{ + std::shared_lock l(c->lock); + ceph_assert(it->valid()); + return it->value_as_sv(); +} + // ===================================== @@ -13601,52 +13620,6 @@ int BlueStore::omap_get_values( return r; } -#ifdef WITH_SEASTAR -int BlueStore::omap_get_values( - CollectionHandle &c_, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<string> &start_after, ///< [in] Keys to get - map<string, bufferlist> *output ///< [out] Returned keys and values - ) -{ - Collection *c = static_cast<Collection *>(c_.get()); - dout(15) << __func__ << " " << c->get_cid() << " oid " << oid << dendl; - if (!c->exists) - return -ENOENT; - std::shared_lock l(c->lock); - int r = 0; - OnodeRef o = c->get_onode(oid, false); - if (!o || !o->exists) { - r = -ENOENT; - goto out; - } - if (!o->onode.has_omap()) { - goto out; - } - o->flush(); - { - ObjectMap::ObjectMapIterator iter = get_omap_iterator(c_, oid); - if (!iter) { - r = -ENOENT; - goto out; - } - if (start_after) { - iter->upper_bound(*start_after); - } else { - iter->seek_to_first(); - } - for (; iter->valid(); iter->next()) { - output->insert(make_pair(iter->key(), iter->value())); - } - } - -out: - dout(10) << __func__ << " " << c->get_cid() << " oid " << oid << " = " << r - << dendl; - return r; -} -#endif - int BlueStore::omap_check_keys( CollectionHandle &c_, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -13724,6 +13697,94 @@ ObjectMap::ObjectMapIterator BlueStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(logger,c, o, it)); } +int BlueStore::omap_iterate( + CollectionHandle &c_, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) +{ + Collection *c = static_cast<Collection *>(c_.get()); + dout(10) << __func__ << " " << c->get_cid() << " " << oid << dendl; + if (!c->exists) { + return -ENOENT; + } + std::shared_lock l(c->lock); + OnodeRef o = c->get_onode(oid, false); + if (!o || !o->exists) { + dout(10) << __func__ << " " << oid << "doesn't exist" <<dendl; + return -ENOENT; + } + o->flush(); + dout(10) << __func__ << " has_omap = " << (int)o->onode.has_omap() <<dendl; + if (!o->onode.has_omap()) { + // nothing to do + return 0; + } + + KeyValueDB::Iterator it; + { + auto bounds = KeyValueDB::IteratorBounds(); + std::string lower_bound, upper_bound; + o->get_omap_key(string(), &lower_bound); + o->get_omap_tail(&upper_bound); + bounds.lower_bound = std::move(lower_bound); + bounds.upper_bound = std::move(upper_bound); + it = db->get_iterator(o->get_omap_prefix(), 0, std::move(bounds)); + } + + // seek the iterator + { + std::string key; + o->get_omap_key(start_from.seek_position, &key); + auto start = ceph::mono_clock::now(); + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it->lower_bound(key); + c->store->log_latency( + __func__, + l_bluestore_omap_lower_bound_lat, + ceph::mono_clock::now() - start, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + } else { + it->upper_bound(key); + c->store->log_latency( + __func__, + l_bluestore_omap_upper_bound_lat, + ceph::mono_clock::now() - start, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + } + } + + // iterate! + std::string tail; + o->get_omap_tail(&tail); + const std::string_view::size_type userkey_offset_in_dbkey = + o->calc_userkey_offset_in_omap_key(); + ceph::timespan next_lat_acc{0}; + while (it->valid()) { + const auto& db_key = it->raw_key_as_sv().second; + if (db_key >= tail) { + break; + } + std::string_view user_key = db_key.substr(userkey_offset_in_dbkey); + omap_iter_ret_t ret = f(user_key, it->value_as_sv()); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + ceph::time_guard<ceph::mono_clock>{next_lat_acc}; + it->next(); + } else { + ceph_abort(); + } + } + c->store->log_latency( + __func__, + l_bluestore_omap_next_lat, + next_lat_acc, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + return 0; +} + // ----------------- // write helpers diff --git a/src/os/bluestore/BlueStore.h b/src/os/bluestore/BlueStore.h index 99f8d057cf0..5549f97ffea 100644 --- a/src/os/bluestore/BlueStore.h +++ b/src/os/bluestore/BlueStore.h @@ -1457,6 +1457,7 @@ public: } void rewrite_omap_key(const std::string& old, std::string *out); + size_t calc_userkey_offset_in_omap_key() const; void decode_omap_key(const std::string& key, std::string *user_key); void finish_write(TransContext* txc, uint32_t offset, uint32_t length); @@ -1753,6 +1754,7 @@ public: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; std::string tail_key() override { return tail; } @@ -3416,15 +3418,6 @@ public: std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) override; -#ifdef WITH_SEASTAR - int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) override; -#endif - /// Filters keys into out which are defined on oid int omap_check_keys( CollectionHandle &c, ///< [in] Collection containing oid @@ -3438,6 +3431,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override { fsid = u; } diff --git a/src/os/bluestore/bluefs_types.cc b/src/os/bluestore/bluefs_types.cc index e18dd490140..fe77f7f74d8 100644 --- a/src/os/bluestore/bluefs_types.cc +++ b/src/os/bluestore/bluefs_types.cc @@ -154,7 +154,9 @@ mempool::bluefs::vector<bluefs_extent_t>::iterator bluefs_fnode_t::seek( assert(it != extents_index.begin()); --it; assert(offset >= *it); - p += it - extents_index.begin(); + uint32_t skip = it - extents_index.begin(); + ceph_assert(skip <= extents.size()); + p += skip; offset -= *it; } diff --git a/src/os/bluestore/bluefs_types.h b/src/os/bluestore/bluefs_types.h index 627118c12f8..08b3ca0cf41 100644 --- a/src/os/bluestore/bluefs_types.h +++ b/src/os/bluestore/bluefs_types.h @@ -89,6 +89,7 @@ struct bluefs_fnode_t { void recalc_allocated() { allocated = 0; extents_index.reserve(extents.size()); + extents_index.clear(); for (auto& p : extents) { extents_index.emplace_back(allocated); allocated += p.length; diff --git a/src/os/bluestore/bluestore_tool.cc b/src/os/bluestore/bluestore_tool.cc index d62721b4366..16f1e6434e0 100644 --- a/src/os/bluestore/bluestore_tool.cc +++ b/src/os/bluestore/bluestore_tool.cc @@ -1136,7 +1136,7 @@ int main(int argc, char **argv) } return r; } - } else if (action == "free-dump" || action == "free-score" || action == "fragmentation") { + } else if (action == "free-dump" || action == "free-score" || action == "free-fragmentation") { AdminSocket *admin_socket = g_ceph_context->get_admin_socket(); ceph_assert(admin_socket); std::string action_name = action == "free-dump" ? "dump" : diff --git a/src/os/kstore/KStore.cc b/src/os/kstore/KStore.cc index 7158486ca38..a069d429155 100644 --- a/src/os/kstore/KStore.cc +++ b/src/os/kstore/KStore.cc @@ -1651,6 +1651,13 @@ bufferlist KStore::OmapIteratorImpl::value() return it->value(); } +std::string_view KStore::OmapIteratorImpl::value_as_sv() +{ + std::shared_lock l{c->lock}; + ceph_assert(it->valid()); + return it->value_as_sv(); +} + int KStore::omap_get( CollectionHandle& ch, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -1866,6 +1873,71 @@ ObjectMap::ObjectMapIterator KStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(c, o, it)); } +int KStore::omap_iterate( + CollectionHandle &ch, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f) +{ + dout(10) << __func__ << " " << ch->cid << " " << oid << dendl; + Collection *c = static_cast<Collection*>(ch.get()); + { + std::shared_lock l{c->lock}; + + OnodeRef o = c->get_onode(oid, false); + if (!o || !o->exists) { + dout(10) << __func__ << " " << oid << "doesn't exist" <<dendl; + return -ENOENT; + } + o->flush(); + dout(10) << __func__ << " header = " << o->onode.omap_head <<dendl; + + KeyValueDB::Iterator it = db->get_iterator(PREFIX_OMAP); + std::string tail; + std::string seek_key; + if (o->onode.omap_head) { + return 0; // nothing to do + } + + // acquire data depedencies for seek & iterate + get_omap_key(o->onode.omap_head, start_from.seek_position, &seek_key); + get_omap_tail(o->onode.omap_head, &tail); + + // acquire the iterator + { + it = db->get_iterator(PREFIX_OMAP); + } + + // seek the iterator + { + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it->lower_bound(seek_key); + } else { + it->upper_bound(seek_key); + } + } + + // iterate! + while (it->valid()) { + std::string user_key; + if (const auto& db_key = it->raw_key().second; db_key >= tail) { + break; + } else { + decode_omap_key(db_key, &user_key); + } + omap_iter_ret_t ret = f(user_key, it->value_as_sv()); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + it->next(); + } else { + ceph_abort(); + } + } + } + return 0; +} + // ----------------- // write helpers diff --git a/src/os/kstore/KStore.h b/src/os/kstore/KStore.h index 9a9d413c66a..06115d3cab7 100644 --- a/src/os/kstore/KStore.h +++ b/src/os/kstore/KStore.h @@ -180,6 +180,7 @@ public: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; int status() override { return 0; } @@ -553,6 +554,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override { fsid = u; } diff --git a/src/os/memstore/MemStore.cc b/src/os/memstore/MemStore.cc index 89cb09361cf..f9d3bf0d8a2 100644 --- a/src/os/memstore/MemStore.cc +++ b/src/os/memstore/MemStore.cc @@ -537,30 +537,6 @@ int MemStore::omap_get_values( return 0; } -#ifdef WITH_SEASTAR -int MemStore::omap_get_values( - CollectionHandle& ch, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) -{ - dout(10) << __func__ << " " << ch->cid << " " << oid << dendl; - Collection *c = static_cast<Collection*>(ch.get()); - ObjectRef o = c->get_object(oid); - if (!o) - return -ENOENT; - assert(start_after); - std::lock_guard lock{o->omap_mutex}; - for (auto it = o->omap.upper_bound(*start_after); - it != std::end(o->omap); - ++it) { - out->insert(*it); - } - return 0; -} -#endif - int MemStore::omap_check_keys( CollectionHandle& ch, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -622,6 +598,10 @@ public: std::lock_guard lock{o->omap_mutex}; return it->second; } + std::string_view value_as_sv() override { + std::lock_guard lock{o->omap_mutex}; + return std::string_view{it->second.c_str(), it->second.length()}; + } int status() override { return 0; } @@ -639,6 +619,48 @@ ObjectMap::ObjectMapIterator MemStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(c, o)); } +int MemStore::omap_iterate( + CollectionHandle &ch, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f) +{ + Collection *c = static_cast<Collection*>(ch.get()); + ObjectRef o = c->get_object(oid); + if (!o) { + return -ENOENT; + } + + { + std::lock_guard lock{o->omap_mutex}; + + // obtain seek the iterator + decltype(o->omap)::iterator it; + { + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it = o->omap.lower_bound(start_from.seek_position); + } else { + it = o->omap.upper_bound(start_from.seek_position); + } + } + + // iterate! + while (it != o->omap.end()) { + // potentially rectifying memcpy but who cares for memstore? + omap_iter_ret_t ret = + f(it->first, std::string_view{it->second.c_str(), it->second.length()}); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + ++it; + } else { + ceph_abort(); + } + } + } + return 0; +} + // --------------- // write operations diff --git a/src/os/memstore/MemStore.h b/src/os/memstore/MemStore.h index 2abe552891f..9621773598f 100644 --- a/src/os/memstore/MemStore.h +++ b/src/os/memstore/MemStore.h @@ -363,14 +363,6 @@ public: const std::set<std::string> &keys, ///< [in] Keys to get std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) override; -#ifdef WITH_SEASTAR - int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) override; -#endif using ObjectStore::omap_check_keys; /// Filters keys into out which are defined on oid @@ -387,6 +379,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override; uuid_d get_fsid() override; diff --git a/src/osd/ECBackend.cc b/src/osd/ECBackend.cc index fa2570aba42..8630b038812 100644 --- a/src/osd/ECBackend.cc +++ b/src/osd/ECBackend.cc @@ -945,6 +945,10 @@ void ECBackend::handle_sub_write( } trace.event("handle_sub_write"); + if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_write_error3(op.soid)) { + ceph_abort_msg("Error inject - OSD down"); + } if (!get_parent()->pgb_is_primary()) get_parent()->update_stats(op.stats); ObjectStore::Transaction localt; @@ -1191,6 +1195,15 @@ void ECBackend::handle_sub_write_reply( i->second->on_all_commit = 0; i->second->trace.event("ec write all committed"); } + if (cct->_conf->bluestore_debug_inject_read_err && + (i->second->pending_commit.size() == 1) && + ec_inject_test_write_error2(i->second->hoid)) { + std::string cmd = + "{ \"prefix\": \"osd down\", \"ids\": [\"" + std::to_string( get_parent()->whoami() ) + "\"] }"; + vector<std::string> vcmd{cmd}; + dout(0) << __func__ << " Error inject - marking OSD down" << dendl; + get_parent()->start_mon_command(vcmd, {}, nullptr, nullptr, nullptr); + } rmw_pipeline.check_ops(); } @@ -1208,6 +1221,19 @@ void ECBackend::handle_sub_read_reply( return; } ReadOp &rop = iter->second; + if (cct->_conf->bluestore_debug_inject_read_err) { + for (auto i = op.buffers_read.begin(); + i != op.buffers_read.end(); + ++i) { + if (ec_inject_test_read_error0(ghobject_t(i->first, ghobject_t::NO_GEN, op.from.shard))) { + dout(0) << __func__ << " Error inject - EIO error for shard " << op.from.shard << dendl; + op.buffers_read.erase(i->first); + op.attrs_read.erase(i->first); + op.errors[i->first] = -EIO; + } + + } + } for (auto i = op.buffers_read.begin(); i != op.buffers_read.end(); ++i) { diff --git a/src/osd/ECCommon.cc b/src/osd/ECCommon.cc index 609ac3141ae..59077547fcb 100644 --- a/src/osd/ECCommon.cc +++ b/src/osd/ECCommon.cc @@ -226,8 +226,14 @@ void ECCommon::ReadPipeline::get_all_avail_shards( ++i) { dout(10) << __func__ << ": checking acting " << *i << dendl; const pg_missing_t &missing = get_parent()->get_shard_missing(*i); - if (error_shards.find(*i) != error_shards.end()) + if (error_shards.contains(*i)) { continue; + } + if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_read_error1(ghobject_t(hoid, ghobject_t::NO_GEN, i->shard))) { + dout(0) << __func__ << " Error inject - Missing shard " << i->shard << dendl; + continue; + } if (!missing.is_missing(hoid)) { ceph_assert(!have.count(i->shard)); have.insert(i->shard); @@ -912,6 +918,11 @@ bool ECCommon::RMWPipeline::try_reads_to_commit() if (*i == get_parent()->whoami_shard()) { should_write_local = true; local_write_op.claim(sop); + } else if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_write_error1(ghobject_t(op->hoid, + ghobject_t::NO_GEN, i->shard))) { + dout(0) << " Error inject - Dropping write message to shard " << + i->shard << dendl; } else { MOSDECSubOpWrite *r = new MOSDECSubOpWrite(sop); r->pgid = spg_t(get_parent()->primary_spg_t().pgid, i->shard); @@ -1090,3 +1101,305 @@ ECUtil::HashInfoRef ECCommon::UnstableHashInfoRegistry::get_hash_info( } return ref; } + +// Error inject interfaces +static ceph::recursive_mutex ec_inject_lock = + ceph::make_recursive_mutex("ECCommon::ec_inject_lock"); +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_read_failures0; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_read_failures1; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures0; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures1; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures2; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures3; +static std::map<ghobject_t,shard_id_t> ec_inject_write_failures0_shard; +static std::set<osd_reqid_t> ec_inject_write_failures0_reqid; + +/** + * Configure a read error inject that typically forces additional reads of + * shards in an EC pool to recover data using the redundancy. With multiple + * errors it is possible to force client reads to fail. + * + * Type 0 - Simulate a medium error. Fail a read with -EIO to force + * additional reads and a decode + * + * Type 1 - Simulate a missing OSD. Dont even try to read a shard + * + * @brief Set up a read error inject for an object in an EC pool. + * @param o Target object for the error inject. + * @param when Error inject starts after this many object store reads. + * @param duration Error inject affects this many object store reads. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Result of configuring the error inject. + */ +std::string ec_inject_read_error(const ghobject_t& o, + const int64_t type, + const int64_t when, + const int64_t duration) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + ghobject_t os = o; + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + switch (type) { + case 0: + ec_inject_read_failures0[os] = std::pair(when, duration); + return "ok - read returns EIO"; + case 1: + ec_inject_read_failures1[os] = std::pair(when, duration); + return "ok - read pretends shard is missing"; + default: + break; + } + return "unrecognized error inject type"; +} + +/** + * Configure a write error inject that either fails an OSD or causes a + * client write operation to be rolled back. + * + * Type 0 - Tests rollback. Drop a write I/O to a shard, then simulate an OSD + * down to force rollback to occur, lastly fail the retried write from the + * client so the results of the rollback can be inspected. + * + * Type 1 - Drop a write I/O to a shard. Used on its own this will hang a + * write I/O. + * + * Type 2 - Simulate an OSD down (ceph osd down) to force a new epoch. Usually + * used together with type 1 to force a rollback + * + * Type 3 - Abort when an OSD processes a write I/O to a shard. Typically the + * client write will be commited while the OSD is absent which will result in + * recovery or backfill later when the OSD returns. + * + * @brief Set up a write error inject for an object in an EC pool. + * @param o Target object for the error inject. + * @param when Error inject starts after this many object store reads. + * @param duration Error inject affects this many object store reads. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Result of configuring the error inect. + */ +std::string ec_inject_write_error(const ghobject_t& o, + const int64_t type, + const int64_t when, + const int64_t duration) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + bool no_shard = true; + std::string result; + switch (type) { + case 0: + failures = &ec_inject_write_failures0; + result = "ok - drop write, sim OSD down and fail client retry with EINVAL"; + break; + case 1: + failures = &ec_inject_write_failures1; + no_shard = false; + result = "ok - drop write to shard"; + break; + case 2: + failures = &ec_inject_write_failures2; + result = "ok - inject OSD down"; + break; + case 3: + if (duration != 1) { + return "duration must be 1"; + } + failures = &ec_inject_write_failures3; + result = "ok - write abort OSDs"; + break; + default: + return "unrecognized error inject type"; + } + if (no_shard) { + os.set_shard(shard_id_t::NO_SHARD); + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + (*failures)[os] = std::pair(when, duration); + if (type == 0) { + ec_inject_write_failures0_shard[os] = o.shard_id; + } + return result; +} + +/** + * @brief Clear a previously configured read error inject. + * @param o Target object for the error inject. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Indication of how many errors were cleared. + */ +std::string ec_inject_clear_read_error(const ghobject_t& o, + const int64_t type) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + int64_t remaining = 0; + switch (type) { + case 0: + failures = &ec_inject_read_failures0; + break; + case 1: + failures = &ec_inject_read_failures1; + break; + default: + return "unrecognized error inject type"; + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + auto it = failures->find(os); + if (it != failures->end()) { + remaining = it->second.second; + failures->erase(it); + } + if (remaining == 0) { + return "no outstanding error injects"; + } else if (remaining == 1) { + return "ok - 1 inject cleared"; + } + return "ok - " + std::to_string(remaining) + " injects cleared"; +} + +/** + * @brief Clear a previously configured write error inject. + * @param o Target object for the error inject. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Indication of how many errors were cleared. + */ +std::string ec_inject_clear_write_error(const ghobject_t& o, + const int64_t type) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + bool no_shard = true; + int64_t remaining = 0; + switch (type) { + case 0: + failures = &ec_inject_write_failures0; + break; + case 1: + failures = &ec_inject_write_failures1; + no_shard = false; + break; + case 2: + failures = &ec_inject_write_failures2; + break; + case 3: + failures = &ec_inject_write_failures3; + break; + default: + return "unrecognized error inject type"; + } + if (no_shard) { + os.set_shard(shard_id_t::NO_SHARD); + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + auto it = failures->find(os); + if (it != failures->end()) { + remaining = it->second.second; + failures->erase(it); + if (type == 0) { + ec_inject_write_failures0_shard.erase(os); + } + } + if (remaining == 0) { + return "no outstanding error injects"; + } else if (remaining == 1) { + return "ok - 1 inject cleared"; + } + return "ok - " + std::to_string(remaining) + " injects cleared"; +} + +static bool ec_inject_test_error(const ghobject_t& o, + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures) +{ + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + auto it = failures->find(o); + if (it == failures->end()) { + ghobject_t os = o; + os.hobj.oid.name = "*"; + os.hobj.set_hash(0); + it = failures->find(os); + } + if (it != failures->end()) { + auto && [when,duration] = it->second; + if (when > 0) { + when--; + return false; + } + if (--duration <= 0) { + failures->erase(it); + } + return true; + } + return false; +} + +bool ec_inject_test_read_error0(const ghobject_t& o) +{ + return ec_inject_test_error(o, &ec_inject_read_failures0); +} + +bool ec_inject_test_read_error1(const ghobject_t& o) +{ + return ec_inject_test_error(o, &ec_inject_read_failures1); +} + +bool ec_inject_test_write_error0(const hobject_t& o, + const osd_reqid_t& reqid) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + ghobject_t os = ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD); + if (ec_inject_write_failures0_reqid.count(reqid)) { + // Matched reqid of retried write - flag for failure + ec_inject_write_failures0_reqid.erase(reqid); + return true; + } + auto it = ec_inject_write_failures0.find(os); + if (it == ec_inject_write_failures0.end()) { + os.hobj.oid.name = "*"; + os.hobj.set_hash(0); + it = ec_inject_write_failures0.find(os); + } + if (it != ec_inject_write_failures0.end()) { + auto && [when, duration] = it->second; + auto shard = ec_inject_write_failures0_shard.find(os)->second; + if (when > 0) { + when--; + } else { + if (--duration <= 0) { + ec_inject_write_failures0.erase(it); + ec_inject_write_failures0_shard.erase(os); + } + // Error inject triggered - save reqid + ec_inject_write_failures0_reqid.insert(reqid); + // Set up error inject to drop message to primary + ec_inject_write_error(ghobject_t(o, ghobject_t::NO_GEN, shard), 1, 0, 1); + } + } + return false; +} + +bool ec_inject_test_write_error1(const ghobject_t& o) { + bool rc = ec_inject_test_error(o, &ec_inject_write_failures1); + if (rc) { + // Set up error inject to generate OSD down + ec_inject_write_error(o, 2, 0, 1); + } + return rc; +} + +bool ec_inject_test_write_error2(const hobject_t& o) { + return ec_inject_test_error( + ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD), + &ec_inject_write_failures2); +} + +bool ec_inject_test_write_error3(const hobject_t& o) { + return ec_inject_test_error( + ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD), + &ec_inject_write_failures3); +} diff --git a/src/osd/ECCommon.h b/src/osd/ECCommon.h index 7ff9cae7646..de4c11ad50f 100644 --- a/src/osd/ECCommon.h +++ b/src/osd/ECCommon.h @@ -493,6 +493,7 @@ struct ECCommon { ); ///< @return error code, 0 on success void schedule_recovery_work(); + }; /** @@ -843,3 +844,15 @@ void ECCommon::ReadPipeline::filter_read_op( on_schedule_recovery(op); } } + +// Error inject interfaces +std::string ec_inject_read_error(const ghobject_t& o, const int64_t type, const int64_t when, const int64_t duration); +std::string ec_inject_write_error(const ghobject_t& o, const int64_t type, const int64_t when, const int64_t duration); +std::string ec_inject_clear_read_error(const ghobject_t& o, const int64_t type); +std::string ec_inject_clear_write_error(const ghobject_t& o, const int64_t type); +bool ec_inject_test_read_error0(const ghobject_t& o); +bool ec_inject_test_read_error1(const ghobject_t& o); +bool ec_inject_test_write_error0(const hobject_t& o,const osd_reqid_t& reqid); +bool ec_inject_test_write_error1(const ghobject_t& o); +bool ec_inject_test_write_error2(const hobject_t& o); +bool ec_inject_test_write_error3(const hobject_t& o); diff --git a/src/osd/ExtentCache.h b/src/osd/ExtentCache.h index 972228cd077..7dc1d4f7263 100644 --- a/src/osd/ExtentCache.h +++ b/src/osd/ExtentCache.h @@ -363,7 +363,7 @@ private: extent, boost::intrusive::list_member_hook<>, &extent::pin_list_member>; - using list = boost::intrusive::list<extent, list_member_options>; + using list = boost::intrusive::list<extent, boost::intrusive::constant_time_size<false>, list_member_options>; list pin_list; ~pin_state() { ceph_assert(pin_list.empty()); diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc index 97fefc5e54a..9c9e540cf61 100644 --- a/src/osd/OSD.cc +++ b/src/osd/OSD.cc @@ -37,6 +37,7 @@ #include "osd/PG.h" #include "osd/scrubber/scrub_machine.h" #include "osd/scrubber/pg_scrubber.h" +#include "osd/ECCommon.h" #include "include/types.h" #include "include/compat.h" @@ -4348,6 +4349,46 @@ void OSD::final_init() "inject metadata error to an object"); ceph_assert(r == 0); r = admin_socket->register_command( + "injectecreaderr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false " \ + "name=when,type=CephInt,req=false " \ + "name=duration,type=CephInt,req=false", + test_ops_hook, + "inject error for read of object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecclearreaderr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false", + test_ops_hook, + "clear read error injects for object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecwriteerr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false " \ + "name=when,type=CephInt,req=false " \ + "name=duration,type=CephInt,req=false", + test_ops_hook, + "inject error for write of object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecclearwriteerr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false", + test_ops_hook, + "clear write error inject for object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( "set_recovery_delay " \ "name=utime,type=CephInt,req=false", test_ops_hook, @@ -6487,8 +6528,10 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, //directly request the osd make a change. if (command == "setomapval" || command == "rmomapkey" || command == "setomapheader" || command == "getomap" || - command == "truncobj" || command == "injectmdataerr" || - command == "injectdataerr" + command == "truncobj" || + command == "injectmdataerr" || command == "injectdataerr" || + command == "injectecreaderr" || command == "injectecclearreaderr" || + command == "injectecwriteerr" || command == "injectecclearwriteerr" ) { pg_t rawpg; int64_t pool; @@ -6527,8 +6570,21 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, ghobject_t gobj(obj, ghobject_t::NO_GEN, shard_id_t(uint8_t(shardid))); spg_t pgid(curmap->raw_pg_to_pg(rawpg), shard_id_t(shardid)); if (curmap->pg_is_ec(rawpg)) { - if ((command != "injectdataerr") && (command != "injectmdataerr")) { - ss << "Must not call on ec pool, except injectdataerr or injectmdataerr"; + if ((command != "injectdataerr") && + (command != "injectmdataerr") && + (command != "injectecreaderr") && + (command != "injectecclearreaderr") && + (command != "injectecwriteerr") && + (command != "injectecclearwriteerr")) { + ss << "Must not call on ec pool"; + return; + } + } else { + if ((command == "injectecreaderr") || + (command == "injecteclearreaderr") || + (command == "injectecwriteerr") || + (command == "injecteclearwriteerr")) { + ss << "Only supported on ec pool"; return; } } @@ -6607,6 +6663,38 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, } else if (command == "injectmdataerr") { store->inject_mdata_error(gobj); ss << "ok"; + } else if (command == "injectecreaderr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + int64_t when = cmd_getval_or<int64_t>(cmdmap, "when", 0); + int64_t duration = cmd_getval_or<int64_t>(cmdmap, "duration", 1); + ss << ec_inject_read_error(gobj, type, when, duration); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecclearreaderr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + ss << ec_inject_clear_read_error(gobj, type); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecwriteerr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + int64_t when = cmd_getval_or<int64_t>(cmdmap, "when", 0); + int64_t duration = cmd_getval_or<int64_t>(cmdmap, "duration", 1); + ss << ec_inject_write_error(gobj, type, when, duration); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecclearwriteerr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + ss << ec_inject_clear_write_error(gobj, type); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } } return; } @@ -9958,7 +10046,8 @@ const char** OSD::get_tracked_conf_keys() const "osd_scrub_max_interval", "osd_op_thread_timeout", "osd_op_thread_suicide_timeout", - NULL + "osd_max_scrubs", + nullptr }; return KEYS; } @@ -10002,6 +10091,10 @@ void OSD::handle_conf_change(const ConfigProxy& conf, service.snap_reserver.set_max(cct->_conf->osd_max_trimming_pgs); } if (changed.count("osd_max_scrubs")) { + dout(0) << fmt::format( + "{}: scrub concurrency max changed to {}", + __func__, cct->_conf->osd_max_scrubs) + << dendl; service.scrub_reserver.set_max(cct->_conf->osd_max_scrubs); } if (changed.count("osd_op_complaint_time") || diff --git a/src/osd/OSDMap.cc b/src/osd/OSDMap.cc index b87484c1a9d..9b3593d54e5 100644 --- a/src/osd/OSDMap.cc +++ b/src/osd/OSDMap.cc @@ -1642,12 +1642,10 @@ void OSDMap::get_out_of_subnet_osd_counts(CephContext *cct, for (int i = 0; i < max_osd; i++) { if (exists(i) && is_up(i)) { if (const auto& addrs = get_addrs(i).v; addrs.size() >= 2) { - auto v1_addr = addrs[0].ip_only_to_str(); - if (!is_addr_in_subnet(cct, public_network, v1_addr)) { + if (!is_addr_in_subnet(cct, public_network, addrs[0])) { unreachable->emplace(i); } - auto v2_addr = addrs[1].ip_only_to_str(); - if (!is_addr_in_subnet(cct, public_network, v2_addr)) { + if (!is_addr_in_subnet(cct, public_network, addrs[1])) { unreachable->emplace(i); } } diff --git a/src/osd/PG.cc b/src/osd/PG.cc index 307651fd627..cd7cad777bc 100644 --- a/src/osd/PG.cc +++ b/src/osd/PG.cc @@ -1278,7 +1278,6 @@ Scrub::schedule_result_t PG::start_scrubbing( pg_cond.allow_deep = !(get_osdmap()->test_flag(CEPH_OSDMAP_NODEEP_SCRUB) || pool.info.has_flag(pg_pool_t::FLAG_NODEEP_SCRUB)); - pg_cond.has_deep_errors = (info.stats.stats.sum.num_deep_scrub_errors > 0); pg_cond.can_autorepair = (cct->_conf->osd_scrub_auto_repair && get_pgbackend()->auto_repair_supported()); diff --git a/src/osd/PGBackend.h b/src/osd/PGBackend.h index b87aa1da677..f5eb9ea951e 100644 --- a/src/osd/PGBackend.h +++ b/src/osd/PGBackend.h @@ -290,6 +290,10 @@ typedef std::shared_ptr<const OSDMap> OSDMapRef; MessageRef, Connection *con) = 0; virtual void send_message_osd_cluster( Message *m, const ConnectionRef& con) = 0; + virtual void start_mon_command( + const std::vector<std::string>& cmd, const bufferlist& inbl, + bufferlist *outbl, std::string *outs, + Context *onfinish) = 0; virtual ConnectionRef get_con_osd_cluster(int peer, epoch_t from_epoch) = 0; virtual entity_name_t get_cluster_msgr_name() = 0; diff --git a/src/osd/PrimaryLogPG.cc b/src/osd/PrimaryLogPG.cc index 44f8e85b5ef..3324ba9dc91 100644 --- a/src/osd/PrimaryLogPG.cc +++ b/src/osd/PrimaryLogPG.cc @@ -2286,6 +2286,16 @@ void PrimaryLogPG::do_op(OpRequestRef& op) } } + if (cct->_conf->bluestore_debug_inject_read_err && + op->may_write() && + pool.info.is_erasure() && + ec_inject_test_write_error0(m->get_hobj(), m->get_reqid())) { + // Fail retried write with error + dout(0) << __func__ << " Error inject - Fail retried write with EINVAL" << dendl; + osd->reply_op_error(op, -EINVAL); + return; + } + ObjectContextRef obc; bool can_create = op->may_write(); hobject_t missing_oid; @@ -5798,10 +5808,19 @@ int PrimaryLogPG::do_extent_cmp(OpContext *ctx, OSDOp& osd_op) int PrimaryLogPG::finish_extent_cmp(OSDOp& osd_op, const bufferlist &read_bl) { - for (uint64_t idx = 0; idx < osd_op.indata.length(); ++idx) { - char read_byte = (idx < read_bl.length() ? read_bl[idx] : 0); - if (osd_op.indata[idx] != read_byte) { - return (-MAX_ERRNO - idx); + auto input_iter = osd_op.indata.begin(); + auto read_iter = read_bl.begin(); + uint64_t idx = 0; + + while (input_iter != osd_op.indata.end()) { + char read_byte = (read_iter != read_bl.end() ? *read_iter : 0); + if (*input_iter != read_byte) { + return (-MAX_ERRNO - idx); + } + ++idx; + ++input_iter; + if (read_iter != read_bl.end()) { + ++read_iter; } } @@ -7767,27 +7786,34 @@ int PrimaryLogPG::do_osd_ops(OpContext *ctx, vector<OSDOp>& ops) bool truncated = false; bufferlist bl; if (oi.is_omap()) { - ObjectMap::ObjectMapIterator iter = osd->store->get_omap_iterator( - ch, ghobject_t(soid) - ); - if (!iter) { - result = -ENOENT; - goto fail; - } - iter->upper_bound(start_after); - if (filter_prefix > start_after) iter->lower_bound(filter_prefix); - for (num = 0; - iter->valid() && - iter->key().substr(0, filter_prefix.size()) == filter_prefix; - ++num, iter->next()) { - dout(20) << "Found key " << iter->key() << dendl; - if (num >= max_return || - bl.length() >= cct->_conf->osd_max_omap_bytes_per_request) { - truncated = true; - break; - } - encode(iter->key(), bl); - encode(iter->value(), bl); + using omap_iter_seek_t = ObjectStore::omap_iter_seek_t; + result = osd->store->omap_iterate( + ch, ghobject_t(soid), + // try to seek as many keys-at-once as possible for the sake of performance. + // note complexity should be logarithmic, so seek(n/2) + seek(n/2) is worse + // than just seek(n). + ObjectStore::omap_iter_seek_t{ + .seek_position = std::max(start_after, filter_prefix), + .seek_type = filter_prefix > start_after ? omap_iter_seek_t::LOWER_BOUND + : omap_iter_seek_t::UPPER_BOUND + }, + [&bl, &truncated, &filter_prefix, &num, max_return, + max_bytes=cct->_conf->osd_max_omap_bytes_per_request] + (std::string_view key, std::string_view value) mutable { + if (key.substr(0, filter_prefix.size()) != filter_prefix) { + return ObjectStore::omap_iter_ret_t::STOP; + } + if (num >= max_return || bl.length() >= max_bytes) { + truncated = true; + return ObjectStore::omap_iter_ret_t::STOP; + } + encode(key, bl); + encode(value, bl); + ++num; + return ObjectStore::omap_iter_ret_t::NEXT; + }); + if (result < 0) { + goto fail; } } // else return empty out_set encode(num, osd_op.outdata); diff --git a/src/osd/PrimaryLogPG.h b/src/osd/PrimaryLogPG.h index f66b5c6e16a..bf55d539821 100644 --- a/src/osd/PrimaryLogPG.h +++ b/src/osd/PrimaryLogPG.h @@ -622,6 +622,12 @@ public: Message *m, const ConnectionRef& con) override { osd->send_message_osd_cluster(m, con); } + void start_mon_command( + const std::vector<std::string>& cmd, const bufferlist& inbl, + bufferlist *outbl, std::string *outs, + Context *onfinish) override { + osd->monc->start_mon_command(cmd, inbl, outbl, outs, onfinish); + } ConnectionRef get_con_osd_cluster(int peer, epoch_t from_epoch) override; entity_name_t get_cluster_msgr_name() override { return osd->get_cluster_msgr_name(); @@ -1993,6 +1999,7 @@ public: private: DynamicPerfStats m_dynamic_perf_stats; + }; inline ostream& operator<<(ostream& out, const PrimaryLogPG::RepGather& repop) @@ -2021,5 +2028,4 @@ inline ostream& operator<<(ostream& out, void intrusive_ptr_add_ref(PrimaryLogPG::RepGather *repop); void intrusive_ptr_release(PrimaryLogPG::RepGather *repop); - #endif diff --git a/src/osd/Session.h b/src/osd/Session.h index 9fa9c655456..05a0119d31e 100644 --- a/src/osd/Session.h +++ b/src/osd/Session.h @@ -136,7 +136,7 @@ struct Session : public RefCountedObject { ceph::mutex session_dispatch_lock = ceph::make_mutex("Session::session_dispatch_lock"); - boost::intrusive::list<OpRequest> waiting_on_map; + boost::intrusive::list<OpRequest, boost::intrusive::constant_time_size<false>> waiting_on_map; ceph::spinlock projected_epoch_lock; epoch_t projected_epoch = 0; diff --git a/src/osd/osd_types.cc b/src/osd/osd_types.cc index 5c2cf8b16b0..048f5aa0009 100644 --- a/src/osd/osd_types.cc +++ b/src/osd/osd_types.cc @@ -2942,6 +2942,14 @@ std::string pg_stat_t::dump_scrub_schedule() const return fmt::format( "Blocked! locked objects (for {}s)", scrub_sched_status.m_duration_seconds); + } else if (scrub_sched_status.m_num_to_reserve != 0) { + // we are waiting for some replicas to respond + return fmt::format( + "Reserving. Waiting {}s for OSD.{} ({}/{})", + scrub_sched_status.m_duration_seconds, + scrub_sched_status.m_osd_to_respond, + scrub_sched_status.m_ordinal_of_requested_replica, + scrub_sched_status.m_num_to_reserve); } else { return fmt::format( "{}scrubbing for {}s", @@ -2964,7 +2972,7 @@ std::string pg_stat_t::dump_scrub_schedule() const case pg_scrub_sched_status_t::queued: return fmt::format( "queued for {}scrub", - ((scrub_sched_status.m_is_deep == scrub_level_t::deep) ? "deep " : "")); + (scrub_sched_status.m_is_deep == scrub_level_t::deep) ? "deep " : ""); default: // a bug! return "SCRUB STATE MISMATCH!"s; @@ -2979,12 +2987,15 @@ bool operator==(const pg_scrubbing_status_t& l, const pg_scrubbing_status_t& r) l.m_duration_seconds == r.m_duration_seconds && l.m_is_active == r.m_is_active && l.m_is_deep == r.m_is_deep && - l.m_is_periodic == r.m_is_periodic; + l.m_is_periodic == r.m_is_periodic && + l.m_osd_to_respond == r.m_osd_to_respond && + l.m_ordinal_of_requested_replica == r.m_ordinal_of_requested_replica && + l.m_num_to_reserve == r.m_num_to_reserve; } void pg_stat_t::encode(ceph::buffer::list &bl) const { - ENCODE_START(29, 22, bl); + ENCODE_START(30, 22, bl); encode(version, bl); encode(reported_seq, bl); encode(reported_epoch, bl); @@ -3044,6 +3055,9 @@ void pg_stat_t::encode(ceph::buffer::list &bl) const encode(objects_trimmed, bl); encode(snaptrim_duration, bl); encode(log_dups_size, bl); + encode(scrub_sched_status.m_osd_to_respond, bl); + encode(scrub_sched_status.m_ordinal_of_requested_replica, bl); + encode(scrub_sched_status.m_num_to_reserve, bl); ENCODE_FINISH(bl); } @@ -3052,7 +3066,7 @@ void pg_stat_t::decode(ceph::buffer::list::const_iterator &bl) { bool tmp; uint32_t old_state; - DECODE_START(29, bl); + DECODE_START(30, bl); decode(version, bl); decode(reported_seq, bl); decode(reported_epoch, bl); @@ -3142,6 +3156,18 @@ void pg_stat_t::decode(ceph::buffer::list::const_iterator &bl) if (struct_v >= 29) { decode(log_dups_size, bl); } + if (struct_v >= 30) { + uint16_t osd_to_respond; + decode(osd_to_respond, bl); + scrub_sched_status.m_osd_to_respond = osd_to_respond; + uint8_t tmp8; + decode(tmp8, bl); + scrub_sched_status.m_ordinal_of_requested_replica = tmp8; + decode(tmp8, bl); + scrub_sched_status.m_num_to_reserve = tmp8; + } else { + scrub_sched_status.m_num_to_reserve = 0; + } } DECODE_FINISH(bl); } diff --git a/src/osd/osd_types.h b/src/osd/osd_types.h index b6f5335a0f5..485fddead7a 100644 --- a/src/osd/osd_types.h +++ b/src/osd/osd_types.h @@ -1151,9 +1151,8 @@ public: bool is_set(key_t key) const; template<typename T> - void set(key_t key, const T &val) { - value_t value = val; - opts[key] = value; + void set(key_t key, T &&val) { + opts.insert_or_assign(key, std::forward<T>(val)); } template<typename T> @@ -2223,6 +2222,13 @@ struct pg_scrubbing_status_t { bool m_is_active{false}; scrub_level_t m_is_deep{scrub_level_t::shallow}; bool m_is_periodic{true}; + // the following are only relevant when we are reserving replicas: + uint16_t m_osd_to_respond{0}; + /// this is the n'th replica we are reserving (out of m_num_to_reserve) + uint8_t m_ordinal_of_requested_replica{0}; + /// the number of replicas we are reserving for scrubbing. 0 means we are not + /// in the process of reserving replicas. + uint8_t m_num_to_reserve{0}; }; bool operator==(const pg_scrubbing_status_t& l, const pg_scrubbing_status_t& r); diff --git a/src/osd/scrubber/pg_scrubber.cc b/src/osd/scrubber/pg_scrubber.cc index aa53df5ae8a..ba83f6ac600 100644 --- a/src/osd/scrubber/pg_scrubber.cc +++ b/src/osd/scrubber/pg_scrubber.cc @@ -588,6 +588,10 @@ scrub_level_t PgScrubber::scrub_requested( return scrub_level_t::shallow; } + // abort an ongoing scrub, if it's of the lowest priority + // and stuck in replica reservations. + m_fsm->process_event(AbortIfReserving{}); + // update the relevant SchedTarget (either shallow or deep). Set its urgency // to either operator_requested or must_repair. Push it into the queue auto& trgt = m_scrub_job->get_target(scrub_level); @@ -2073,6 +2077,7 @@ void PgScrubber::scrub_finish() } cleanup_on_finish(); + m_active_target.reset(); if (do_auto_scrub) { request_rescrubbing(); } @@ -2299,26 +2304,6 @@ Scrub::schedule_result_t PgScrubber::start_scrub_session( } } - // restricting shallow scrubs of PGs that have deep errors: - if (pg_cond.has_deep_errors && trgt.is_shallow()) { - if (trgt.urgency() < urgency_t::operator_requested) { - // if there are deep errors, we should have scheduled a deep scrub first. - // If we are here trying to perform a shallow scrub, it means that for some - // reason that deep scrub failed to be initiated. We will not try a shallow - // scrub until this is solved. - dout(10) << __func__ << ": Regular scrub skipped due to deep-scrub errors" - << dendl; - requeue_penalized( - s_or_d, delay_both_targets_t::no, delay_cause_t::pg_state, clock_now); - return schedule_result_t::target_specific_failure; - } else { - // we will honor the request anyway, but will report the issue - m_osds->clog->error() << fmt::format( - "osd.{} pg {} Regular scrub request, deep-scrub details will be lost", - m_osds->whoami, m_pg_id); - } - } - // if only explicitly requested repairing is allowed - skip other types // of scrubbing if (osd_restrictions.allow_requested_repair_only && @@ -2433,6 +2418,16 @@ void PgScrubber::dump_active_scrubber(ceph::Formatter* f) const } else { f->dump_string("schedule", "scrubbing"); } + const auto maybe_register = m_fsm->get_reservation_status(); + if (maybe_register && maybe_register->m_num_to_reserve != 0) { + f->dump_bool("is_reserving_replicas", true); + f->dump_int("osd_to_respond", maybe_register->m_osd_to_respond); + f->dump_int("duration_seconds", maybe_register->m_duration_seconds); + f->dump_int("requested_in_order", maybe_register->m_ordinal_of_requested_replica); + f->dump_int("num_to_reserve", maybe_register->m_num_to_reserve); + } else { + f->dump_bool("is_reserving_replicas", false); + } } pg_scrubbing_status_t PgScrubber::get_schedule() const @@ -2461,7 +2456,7 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const pg_scrub_sched_status_t::blocked, true, // active (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), - false}; + (m_active_target->urgency() == urgency_t::periodic_regular)}; } else { int32_t dur_seconds = @@ -2472,9 +2467,11 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const pg_scrub_sched_status_t::active, true, // active (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), - false /* is periodic? unknown, actually */}; + (m_active_target->urgency() == urgency_t::periodic_regular)}; } } + + // not registered to be scrubbed? if (!m_scrub_job->is_registered()) { return pg_scrubbing_status_t{ utime_t{}, @@ -2485,8 +2482,34 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const false}; } - // not taking 'no-*scrub' flags into account here. + // in session, but still reserving replicas? + const auto maybe_register = m_fsm->get_reservation_status(); + if (maybe_register) { + // note that if we are here, we are scrubbing (even though + // m_active is false). The 'maybe_register' attests to being in + // ReservingReplicas state, and m_active wasn't set yet. + dout(20) << fmt::format( + "{}:maybe_register: osd:{} {}s ({} of {})", __func__, + maybe_register->m_osd_to_respond, + maybe_register->m_duration_seconds, + maybe_register->m_ordinal_of_requested_replica, + maybe_register->m_num_to_reserve) + << dendl; + return pg_scrubbing_status_t{ + utime_t{}, + maybe_register->m_duration_seconds, + pg_scrub_sched_status_t::active, + true, // active + (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), + (m_active_target->urgency() == urgency_t::periodic_regular), + maybe_register->m_osd_to_respond, + maybe_register->m_ordinal_of_requested_replica, + maybe_register->m_num_to_reserve}; + } + const auto first_ready = m_scrub_job->earliest_eligible(now_is); + // eligible for scrubbing, but not yet selected to be scrubbed? + // (not taking 'no-*scrub' flags into account here.) if (first_ready) { const auto& targ = first_ready->get(); return pg_scrubbing_status_t{ diff --git a/src/osd/scrubber/scrub_machine.cc b/src/osd/scrubber/scrub_machine.cc index da9466758f4..10866ce580a 100644 --- a/src/osd/scrubber/scrub_machine.cc +++ b/src/osd/scrubber/scrub_machine.cc @@ -106,6 +106,25 @@ ceph::timespan ScrubMachine::get_time_scrubbing() const return ceph::timespan{}; } +std::optional<pg_scrubbing_status_t> ScrubMachine::get_reservation_status() + const +{ + const auto resv_state = state_cast<const ReservingReplicas*>(); + if (!resv_state) { + return std::nullopt; + } + const auto session = state_cast<const Session*>(); + dout(30) << fmt::format( + "{}: we are reserving {:p}-{:p}", __func__, (void*)session, + (void*)resv_state) + << dendl; + if (!session || !session->m_reservations) { + dout(20) << fmt::format("{}: no reservations data", __func__) << dendl; + return std::nullopt; + } + return session->get_reservation_status(); +} + // ////////////// the actual actions // ----------------------- NotActive ----------------------------------------- @@ -203,6 +222,23 @@ sc::result Session::react(const IntervalChanged&) return transit<NotActive>(); } +std::optional<pg_scrubbing_status_t> Session::get_reservation_status() const +{ + if (!m_reservations) { + return std::nullopt; + } + DECLARE_LOCALS; // 'scrbr' & 'pg_id' aliases + const auto req = m_reservations->get_last_sent(); + pg_scrubbing_status_t s; + s.m_osd_to_respond = req ? req->osd : 0; + s.m_ordinal_of_requested_replica = m_reservations->active_requests_cnt(); + s.m_num_to_reserve = scrbr->get_pg()->get_actingset().size() - 1; + s.m_duration_seconds = + duration_cast<seconds>(context<ScrubMachine>().get_time_scrubbing()) + .count(); + return s; +} + // ----------------------- ReservingReplicas --------------------------------- diff --git a/src/osd/scrubber/scrub_machine.h b/src/osd/scrubber/scrub_machine.h index ad0d3bfba38..f7f739692bf 100644 --- a/src/osd/scrubber/scrub_machine.h +++ b/src/osd/scrubber/scrub_machine.h @@ -2,6 +2,7 @@ // vim: ts=8 sw=2 smarttab #pragma once +#include <optional> #include <string> #include <boost/statechart/custom_reaction.hpp> @@ -160,6 +161,11 @@ VALUE_EVENT(ReserverGranted, AsyncScrubResData); /// all replicas have granted our reserve request MEV(RemotesReserved) +/// abort the scrub session, if in ReservingReplicas state +/// (used when the operator issues a scrub request, and we no longer +/// need the reservations) +MEV(AbortIfReserving) + /// initiate a new scrubbing session (relevant if we are a Primary) MEV(StartScrub) @@ -289,9 +295,12 @@ class ScrubMachine : public sc::state_machine<ScrubMachine, NotActive> { [[nodiscard]] bool is_accepting_updates() const; [[nodiscard]] bool is_primary_idle() const; - // elapsed time for the currently active scrub.session + /// elapsed time for the currently active scrub.session ceph::timespan get_time_scrubbing() const; + /// replica reservation process status + std::optional<pg_scrubbing_status_t> get_reservation_status() const; + // ///////////////// aux declarations & functions //////////////////////// // @@ -555,6 +564,9 @@ struct Session : sc::state<Session, PrimaryActive, ReservingReplicas>, /// abort reason - if known. Determines the delay time imposed on the /// failed scrub target. std::optional<Scrub::delay_cause_t> m_abort_reason{std::nullopt}; + + /// when reserving replicas: fetch the reservation status + std::optional<pg_scrubbing_status_t> get_reservation_status() const; }; struct ReservingReplicas : sc::state<ReservingReplicas, Session>, NamedSimply { @@ -563,6 +575,7 @@ struct ReservingReplicas : sc::state<ReservingReplicas, Session>, NamedSimply { using reactions = mpl::list< sc::custom_reaction<ReplicaGrant>, sc::custom_reaction<ReplicaReject>, + sc::transition<AbortIfReserving, PrimaryIdle>, sc::transition<RemotesReserved, ActiveScrubbing>>; ScrubTimePoint entered_at = ScrubClock::now(); diff --git a/src/osd/scrubber/scrub_reservations.h b/src/osd/scrubber/scrub_reservations.h index 173b23d7db5..f5eca48b888 100644 --- a/src/osd/scrubber/scrub_reservations.h +++ b/src/osd/scrubber/scrub_reservations.h @@ -157,13 +157,13 @@ class ReplicaReservations { // note: 'public', as accessed via the 'standard' dout_prefix() macro std::ostream& gen_prefix(std::ostream& out, std::string fn) const; + /// The number of requests that have been sent (and not rejected) so far. + size_t active_requests_cnt() const; + private: /// send 'release' messages to all replicas we have managed to reserve void release_all(); - /// The number of requests that have been sent (and not rejected) so far. - size_t active_requests_cnt() const; - /** * Send a reservation request to the next replica. * - if there are no more replicas to send requests to, return true diff --git a/src/osd/scrubber_common.h b/src/osd/scrubber_common.h index 809107e593b..2ab5570a715 100644 --- a/src/osd/scrubber_common.h +++ b/src/osd/scrubber_common.h @@ -109,7 +109,6 @@ static_assert(sizeof(Scrub::OSDRestrictions) <= sizeof(uint32_t)); struct ScrubPGPreconds { bool allow_shallow{true}; bool allow_deep{true}; - bool has_deep_errors{false}; bool can_autorepair{false}; }; static_assert(sizeof(Scrub::ScrubPGPreconds) <= sizeof(uint32_t)); @@ -181,9 +180,8 @@ struct formatter<Scrub::ScrubPGPreconds> { auto format(const Scrub::ScrubPGPreconds& conds, FormatContext& ctx) const { return fmt::format_to( - ctx.out(), "allowed(shallow/deep):{:1}/{:1},deep-err:{:1},can-autorepair:{:1}", - conds.allow_shallow, conds.allow_deep, conds.has_deep_errors, - conds.can_autorepair); + ctx.out(), "allowed(shallow/deep):{:1}/{:1},can-autorepair:{:1}", + conds.allow_shallow, conds.allow_deep, conds.can_autorepair); } }; diff --git a/src/osdc/CMakeLists.txt b/src/osdc/CMakeLists.txt index 205ad3d4f42..637ce327555 100644 --- a/src/osdc/CMakeLists.txt +++ b/src/osdc/CMakeLists.txt @@ -1,9 +1,8 @@ set(osdc_files Filer.cc ObjectCacher.cc - Objecter.cc - error_code.cc - Striper.cc) + error_code.cc) +# Objecter.cc and Striper.cc are part of libcommon add_library(osdc STATIC ${osdc_files}) target_link_libraries(osdc ceph-common) if(WITH_EVENTTRACE) diff --git a/src/osdc/Objecter.cc b/src/osdc/Objecter.cc index 087b623333b..82d43bb3dde 100644 --- a/src/osdc/Objecter.cc +++ b/src/osdc/Objecter.cc @@ -1393,7 +1393,7 @@ void Objecter::handle_osd_map(MOSDMap *m) for (auto& [c, ec] : p->second) { asio::post(service.get_executor(), asio::append(std::move(c), ec)); } - waiting_for_map.erase(p++); + p = waiting_for_map.erase(p); } monc->sub_got("osdmap", osdmap->get_epoch()); diff --git a/src/pybind/mgr/cephadm/inventory.py b/src/pybind/mgr/cephadm/inventory.py index f1c56d75378..550604fc55b 100644 --- a/src/pybind/mgr/cephadm/inventory.py +++ b/src/pybind/mgr/cephadm/inventory.py @@ -2036,8 +2036,8 @@ class CertKeyStore(): var = service_name if entity in self.service_name_cert else host j = {} self.known_certs[entity][var] = cert_obj - for service_name in self.known_certs[entity].keys(): - j[var] = Cert.to_json(self.known_certs[entity][var]) + for cert_key in self.known_certs[entity]: + j[cert_key] = Cert.to_json(self.known_certs[entity][cert_key]) else: self.known_certs[entity] = cert_obj j = Cert.to_json(cert_obj) diff --git a/src/pybind/mgr/cephadm/module.py b/src/pybind/mgr/cephadm/module.py index bf14f8d1715..6690153d435 100644 --- a/src/pybind/mgr/cephadm/module.py +++ b/src/pybind/mgr/cephadm/module.py @@ -2460,7 +2460,7 @@ Then run the following: @handle_orch_error def service_action(self, action: str, service_name: str) -> List[str]: - if service_name not in self.spec_store.all_specs.keys(): + if service_name not in self.spec_store.all_specs.keys() and service_name != 'osd': raise OrchestratorError(f'Invalid service name "{service_name}".' + ' View currently running services using "ceph orch ls"') dds: List[DaemonDescription] = self.cache.get_daemons_by_service(service_name) @@ -3925,6 +3925,50 @@ Then run the following: return self.to_remove_osds.all_osds() @handle_orch_error + def set_osd_spec(self, service_name: str, osd_ids: List[str]) -> str: + """ + Update unit.meta file for osd with service name + """ + if service_name not in self.spec_store: + raise OrchestratorError(f"Cannot find service '{service_name}' in the inventory. " + "Please try again after applying an OSD service that matches " + "the service name to which you want to attach OSDs.") + + daemons: List[orchestrator.DaemonDescription] = self.cache.get_daemons_by_type('osd') + update_osd = defaultdict(list) + for daemon in daemons: + if daemon.daemon_id in osd_ids and daemon.hostname: + update_osd[daemon.hostname].append(daemon.daemon_id) + + if not update_osd: + raise OrchestratorError(f"Unable to find OSDs: {osd_ids}") + + failed_osds = [] + success_osds = [] + for host in update_osd: + osds = ",".join(update_osd[host]) + # run cephadm command with all host osds on specific host, + # if it fails, continue with other hosts + try: + with self.async_timeout_handler(host): + outs, errs, _code = self.wait_async( + CephadmServe(self)._run_cephadm(host, + cephadmNoImage, + 'update-osd-service', + ['--service-name', service_name, '--osd-ids', osds])) + if _code: + self.log.error(f"Failed to update service for {osds} osd. Cephadm error: {errs}") + failed_osds.extend(update_osd[host]) + else: + success_osds.extend(update_osd[host]) + except Exception: + self.log.exception(f"Failed to set service name for {osds}") + failed_osds.extend(update_osd[host]) + self.cache.invalidate_host_daemons(host) + self._kick_serve_loop() + return f"Updated service for osd {','.join(success_osds)}" + (f" and failed for {','.join(failed_osds)}" if failed_osds else "") + + @handle_orch_error @host_exists() def drain_host(self, hostname: str, force: bool = False, keep_conf_keyring: bool = False, zap_osd_devices: bool = False) -> str: """ diff --git a/src/pybind/mgr/cephadm/schedule.py b/src/pybind/mgr/cephadm/schedule.py index 98d2fe99897..04d3712c50a 100644 --- a/src/pybind/mgr/cephadm/schedule.py +++ b/src/pybind/mgr/cephadm/schedule.py @@ -385,6 +385,8 @@ class HostAssignment(object): def find_ip_on_host(self, hostname: str, subnets: List[str]) -> Optional[str]: for subnet in subnets: + # to normalize subnet + subnet = str(ipaddress.ip_network(subnet)) ips: List[str] = [] # following is to allow loopback interfaces for both ipv4 and ipv6. Since we # only have the subnet (and no IP) we assume default loopback IP address. diff --git a/src/pybind/mgr/cephadm/services/cephadmservice.py b/src/pybind/mgr/cephadm/services/cephadmservice.py index 04f5af28a9b..4f83d7bb0fb 100644 --- a/src/pybind/mgr/cephadm/services/cephadmservice.py +++ b/src/pybind/mgr/cephadm/services/cephadmservice.py @@ -1157,6 +1157,14 @@ class RgwService(CephService): 'value': str(spec.rgw_bucket_counters_cache_size), }) + if getattr(spec, 'disable_multisite_sync_traffic', None) is not None: + ret, out, err = self.mgr.check_mon_command({ + 'prefix': 'config set', + 'who': daemon_name, + 'name': 'rgw_run_sync_thread', + 'value': 'false' if spec.disable_multisite_sync_traffic else 'true', + }) + daemon_spec.keyring = keyring daemon_spec.final_config, daemon_spec.deps = self.generate_config(daemon_spec) diff --git a/src/pybind/mgr/cephadm/services/monitoring.py b/src/pybind/mgr/cephadm/services/monitoring.py index 1b9cf618570..9c5b5a112f3 100644 --- a/src/pybind/mgr/cephadm/services/monitoring.py +++ b/src/pybind/mgr/cephadm/services/monitoring.py @@ -3,6 +3,7 @@ import logging import os import socket from typing import List, Any, Tuple, Dict, Optional, cast +import ipaddress from mgr_module import HandleCommandResult @@ -57,6 +58,8 @@ class GrafanaService(CephadmService): if ip_to_bind_to: daemon_spec.port_ips = {str(grafana_port): ip_to_bind_to} grafana_ip = ip_to_bind_to + if ipaddress.ip_network(grafana_ip).version == 6: + grafana_ip = f"[{grafana_ip}]" domain = self.mgr.get_fqdn(daemon_spec.host) mgmt_gw_ips = [] @@ -354,6 +357,13 @@ class AlertmanagerService(CephadmService): addr = self.mgr.get_fqdn(dd.hostname) peers.append(build_url(host=addr, port=port).lstrip('/')) + ip_to_bind_to = '' + if spec.only_bind_port_on_networks and spec.networks: + assert daemon_spec.host is not None + ip_to_bind_to = self.mgr.get_first_matching_network_ip(daemon_spec.host, spec) or '' + if ip_to_bind_to: + daemon_spec.port_ips = {str(port): ip_to_bind_to} + deps.append(f'secure_monitoring_stack:{self.mgr.secure_monitoring_stack}') if security_enabled: alertmanager_user, alertmanager_password = self.mgr._get_alertmanager_credentials() @@ -376,7 +386,8 @@ class AlertmanagerService(CephadmService): }, 'peers': peers, 'web_config': '/etc/alertmanager/web.yml', - 'use_url_prefix': mgmt_gw_enabled + 'use_url_prefix': mgmt_gw_enabled, + 'ip_to_bind_to': ip_to_bind_to }, sorted(deps) else: return { @@ -384,7 +395,8 @@ class AlertmanagerService(CephadmService): "alertmanager.yml": yml }, "peers": peers, - 'use_url_prefix': mgmt_gw_enabled + 'use_url_prefix': mgmt_gw_enabled, + 'ip_to_bind_to': ip_to_bind_to }, sorted(deps) def get_active_daemon(self, daemon_descrs: List[DaemonDescription]) -> DaemonDescription: diff --git a/src/pybind/mgr/cephadm/services/nvmeof.py b/src/pybind/mgr/cephadm/services/nvmeof.py index 0c9dc244c57..b3fd526815e 100644 --- a/src/pybind/mgr/cephadm/services/nvmeof.py +++ b/src/pybind/mgr/cephadm/services/nvmeof.py @@ -38,6 +38,8 @@ class NvmeofService(CephService): spec = cast(NvmeofServiceSpec, self.mgr.spec_store[daemon_spec.service_name].spec) nvmeof_gw_id = daemon_spec.daemon_id host_ip = self.mgr.inventory.get_addr(daemon_spec.host) + map_addr = spec.addr_map.get(daemon_spec.host) if spec.addr_map else None + map_discovery_addr = spec.discovery_addr_map.get(daemon_spec.host) if spec.discovery_addr_map else None keyring = self.get_keyring_with_caps(self.get_auth_entity(nvmeof_gw_id), ['mon', 'profile rbd', @@ -47,8 +49,14 @@ class NvmeofService(CephService): transport_tcp_options = json.dumps(spec.transport_tcp_options) if spec.transport_tcp_options else None name = '{}.{}'.format(utils.name_to_config_section('nvmeof'), nvmeof_gw_id) rados_id = name[len('client.'):] if name.startswith('client.') else name - addr = spec.addr or host_ip - discovery_addr = spec.discovery_addr or host_ip + + # The address is first searched in the per node address map, + # then in the spec address configuration. + # If neither is defined, the host IP is used as a fallback. + addr = map_addr or spec.addr or host_ip + self.mgr.log.info(f"gateway address: {addr} from {map_addr=} {spec.addr=} {host_ip=}") + discovery_addr = map_discovery_addr or spec.discovery_addr or host_ip + self.mgr.log.info(f"discovery address: {discovery_addr} from {map_discovery_addr=} {spec.discovery_addr=} {host_ip=}") context = { 'spec': spec, 'name': name, diff --git a/src/pybind/mgr/cephadm/ssh.py b/src/pybind/mgr/cephadm/ssh.py index 1622cb001ab..acb5a77c51b 100644 --- a/src/pybind/mgr/cephadm/ssh.py +++ b/src/pybind/mgr/cephadm/ssh.py @@ -358,7 +358,7 @@ class SSHManager: await self._check_execute_command(host, chown, addr=addr) chmod = RemoteCommand(Executables.CHMOD, [oct(mode)[2:], tmp_path]) await self._check_execute_command(host, chmod, addr=addr) - mv = RemoteCommand(Executables.MV, [tmp_path, path]) + mv = RemoteCommand(Executables.MV, ['-Z', tmp_path, path]) await self._check_execute_command(host, mv, addr=addr) except Exception as e: msg = f"Unable to write {host}:{path}: {e}" diff --git a/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 b/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 index de993cb6ce3..b6955caf616 100644 --- a/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 +++ b/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 @@ -8,6 +8,8 @@ global: tls_config: {% if security_enabled %} ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key {% else %} insecure_skip_verify: true {% endif %} diff --git a/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 b/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 index b9773ceeeb3..14af0fd48ca 100644 --- a/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 +++ b/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 @@ -9,6 +9,7 @@ events { http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; diff --git a/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 b/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 index acc93507424..37f2db52732 100644 --- a/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 +++ b/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 @@ -10,10 +10,15 @@ state_update_interval_sec = {{ spec.state_update_interval_sec }} enable_spdk_discovery_controller = {{ spec.enable_spdk_discovery_controller }} enable_key_encryption = {{ spec.enable_key_encryption }} encryption_key = /encryption.key +rebalance_period_sec = {{ spec.rebalance_period_sec }} +max_gws_in_grp = {{ spec.max_gws_in_grp }} +max_ns_to_change_lb_grp = {{ spec.max_ns_to_change_lb_grp }} enable_prometheus_exporter = {{ spec.enable_prometheus_exporter }} prometheus_exporter_ssl = False -prometheus_port = 10008 +prometheus_port = {{ spec.prometheus_port }} +prometheus_stats_interval = {{ spec.prometheus_stats_interval }} verify_nqns = {{ spec.verify_nqns }} +verify_keys = {{ spec.verify_keys }} omap_file_lock_duration = {{ spec.omap_file_lock_duration }} omap_file_lock_retries = {{ spec.omap_file_lock_retries }} omap_file_lock_retry_sleep_interval = {{ spec.omap_file_lock_retry_sleep_interval }} diff --git a/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 b/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 index ecfd899af71..961da145dac 100644 --- a/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 +++ b/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 @@ -28,6 +28,8 @@ alerting: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} - scheme: http http_sd_configs: @@ -56,6 +58,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} honor_labels: true http_sd_configs: @@ -81,6 +85,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ node_exporter_sd_url }} @@ -104,6 +110,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ haproxy_sd_url }} @@ -128,6 +136,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} honor_labels: true http_sd_configs: @@ -149,6 +159,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ nvmeof_sd_url }} @@ -169,6 +181,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ nfs_sd_url }} @@ -189,6 +203,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ smb_sd_url }} diff --git a/src/pybind/mgr/cephadm/tests/test_cephadm.py b/src/pybind/mgr/cephadm/tests/test_cephadm.py index b81510504d9..22bd26def91 100644 --- a/src/pybind/mgr/cephadm/tests/test_cephadm.py +++ b/src/pybind/mgr/cephadm/tests/test_cephadm.py @@ -1741,16 +1741,23 @@ class TestCephadm(object): nvmeof_client_cert = 'fake-nvmeof-client-cert' nvmeof_server_cert = 'fake-nvmeof-server-cert' nvmeof_root_ca_cert = 'fake-nvmeof-root-ca-cert' + grafana_cert_host_1 = 'grafana-cert-host-1' + grafana_cert_host_2 = 'grafana-cert-host-2' cephadm_module.cert_key_store.save_cert('rgw_frontend_ssl_cert', rgw_frontend_rgw_foo_host2_cert, service_name='rgw.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_server_cert', nvmeof_server_cert, service_name='nvmeof.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_client_cert', nvmeof_client_cert, service_name='nvmeof.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_root_ca_cert', nvmeof_root_ca_cert, service_name='nvmeof.foo', user_made=True) + cephadm_module.cert_key_store.save_cert('grafana_cert', grafana_cert_host_1, host='host-1', user_made=True) + cephadm_module.cert_key_store.save_cert('grafana_cert', grafana_cert_host_2, host='host-2', user_made=True) expected_calls = [ mock.call(f'{CERT_STORE_CERT_PREFIX}rgw_frontend_ssl_cert', json.dumps({'rgw.foo': Cert(rgw_frontend_rgw_foo_host2_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_server_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_server_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_client_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_client_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_root_ca_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_root_ca_cert, True).to_json()})), + mock.call(f'{CERT_STORE_CERT_PREFIX}grafana_cert', json.dumps({'host-1': Cert(grafana_cert_host_1, True).to_json()})), + mock.call(f'{CERT_STORE_CERT_PREFIX}grafana_cert', json.dumps({'host-1': Cert(grafana_cert_host_1, True).to_json(), + 'host-2': Cert(grafana_cert_host_2, True).to_json()})) ] _set_store.assert_has_calls(expected_calls) @@ -1795,17 +1802,20 @@ class TestCephadm(object): cephadm_module.cert_key_store._init_known_cert_key_dicts() grafana_host1_key = 'fake-grafana-host1-key' + grafana_host2_key = 'fake-grafana-host2-key' nvmeof_client_key = 'nvmeof-client-key' nvmeof_server_key = 'nvmeof-server-key' nvmeof_encryption_key = 'nvmeof-encryption-key' - grafana_host1_key = 'fake-grafana-host1-cert' cephadm_module.cert_key_store.save_key('grafana_key', grafana_host1_key, host='host1') + cephadm_module.cert_key_store.save_key('grafana_key', grafana_host2_key, host='host2') cephadm_module.cert_key_store.save_key('nvmeof_client_key', nvmeof_client_key, service_name='nvmeof.foo') cephadm_module.cert_key_store.save_key('nvmeof_server_key', nvmeof_server_key, service_name='nvmeof.foo') cephadm_module.cert_key_store.save_key('nvmeof_encryption_key', nvmeof_encryption_key, service_name='nvmeof.foo') expected_calls = [ mock.call(f'{CERT_STORE_KEY_PREFIX}grafana_key', json.dumps({'host1': PrivKey(grafana_host1_key).to_json()})), + mock.call(f'{CERT_STORE_KEY_PREFIX}grafana_key', json.dumps({'host1': PrivKey(grafana_host1_key).to_json(), + 'host2': PrivKey(grafana_host2_key).to_json()})), mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_client_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_client_key).to_json()})), mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_server_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_server_key).to_json()})), mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_encryption_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_encryption_key).to_json()})), diff --git a/src/pybind/mgr/cephadm/tests/test_services.py b/src/pybind/mgr/cephadm/tests/test_services.py index 1b3c9bdb899..d872219df80 100644 --- a/src/pybind/mgr/cephadm/tests/test_services.py +++ b/src/pybind/mgr/cephadm/tests/test_services.py @@ -400,10 +400,15 @@ state_update_interval_sec = 5 enable_spdk_discovery_controller = False enable_key_encryption = True encryption_key = /encryption.key +rebalance_period_sec = 7 +max_gws_in_grp = 16 +max_ns_to_change_lb_grp = 8 enable_prometheus_exporter = True prometheus_exporter_ssl = False prometheus_port = 10008 +prometheus_stats_interval = 10 verify_nqns = True +verify_keys = True omap_file_lock_duration = 20 omap_file_lock_retries = 30 omap_file_lock_retry_sleep_interval = 1.0 @@ -576,7 +581,14 @@ class TestMonitoring: mock_getfqdn.return_value = purl.hostname with with_host(cephadm_module, "test"): - with with_service(cephadm_module, AlertManagerSpec()): + cephadm_module.cache.update_host_networks('test', { + '1.2.3.0/24': { + 'if0': ['1.2.3.1'] + }, + }) + with with_service(cephadm_module, AlertManagerSpec('alertmanager', + networks=['1.2.3.0/24'], + only_bind_port_on_networks=True)): y = dedent(self._get_config(expected_yaml_url)).lstrip() _run_cephadm.assert_called_with( 'test', @@ -590,11 +602,12 @@ class TestMonitoring: "deploy_arguments": [], "params": { 'tcp_ports': [9093, 9094], + 'port_ips': {"9094": "1.2.3.1"}, }, "meta": { 'service_name': 'alertmanager', 'ports': [9093, 9094], - 'ip': None, + 'ip': '1.2.3.1', 'deployed_by': [], 'rank': None, 'rank_generation': None, @@ -607,6 +620,7 @@ class TestMonitoring: }, "peers": [], "use_url_prefix": False, + "ip_to_bind_to": "1.2.3.1", } }), error_ok=True, @@ -629,8 +643,16 @@ class TestMonitoring: cephadm_module.secure_monitoring_stack = True cephadm_module.set_store(AlertmanagerService.USER_CFG_KEY, 'alertmanager_user') cephadm_module.set_store(AlertmanagerService.PASS_CFG_KEY, 'alertmanager_plain_password') + + cephadm_module.cache.update_host_networks('test', { + 'fd12:3456:789a::/64': { + 'if0': ['fd12:3456:789a::10'] + }, + }) with with_service(cephadm_module, MgmtGatewaySpec("mgmt-gateway")) as _, \ - with_service(cephadm_module, AlertManagerSpec()): + with_service(cephadm_module, AlertManagerSpec('alertmanager', + networks=['fd12:3456:789a::/64'], + only_bind_port_on_networks=True)): y = dedent(""" # This file is generated by cephadm. @@ -641,6 +663,8 @@ class TestMonitoring: http_config: tls_config: ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key route: receiver: 'default' @@ -681,11 +705,12 @@ class TestMonitoring: "deploy_arguments": [], "params": { 'tcp_ports': [9093, 9094], + 'port_ips': {"9094": "fd12:3456:789a::10"} }, "meta": { 'service_name': 'alertmanager', 'ports': [9093, 9094], - 'ip': None, + 'ip': 'fd12:3456:789a::10', 'deployed_by': [], 'rank': None, 'rank_generation': None, @@ -703,6 +728,7 @@ class TestMonitoring: 'peers': [], 'web_config': '/etc/alertmanager/web.yml', "use_url_prefix": True, + "ip_to_bind_to": "fd12:3456:789a::10", } }), error_ok=True, @@ -736,6 +762,8 @@ class TestMonitoring: http_config: tls_config: ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key route: receiver: 'default' @@ -796,6 +824,7 @@ class TestMonitoring: 'peers': [], 'web_config': '/etc/alertmanager/web.yml', "use_url_prefix": False, + "ip_to_bind_to": "", } }), error_ok=True, @@ -1165,6 +1194,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key scrape_configs: - job_name: 'ceph' @@ -1186,6 +1217,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'node' relabel_configs: @@ -1204,6 +1237,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'haproxy' relabel_configs: @@ -1220,6 +1255,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'ceph-exporter' relabel_configs: @@ -1237,6 +1274,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'nvmeof' honor_labels: true @@ -1250,6 +1289,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'nfs' honor_labels: true @@ -1263,6 +1304,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'smb' honor_labels: true @@ -1276,6 +1319,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key """).lstrip() @@ -2066,6 +2111,26 @@ class TestRGWService: }) assert f == expected + @pytest.mark.parametrize( + "disable_sync_traffic", + [ + (True), + (False), + ] + ) + @patch("cephadm.serve.CephadmServe._run_cephadm", _run_cephadm('{}')) + def test_rgw_disable_sync_traffic(self, disable_sync_traffic, cephadm_module: CephadmOrchestrator): + with with_host(cephadm_module, 'host1'): + s = RGWSpec(service_id="foo", + disable_multisite_sync_traffic=disable_sync_traffic) + with with_service(cephadm_module, s) as dds: + _, f, _ = cephadm_module.check_mon_command({ + 'prefix': 'config get', + 'who': f'client.{dds[0]}', + 'key': 'rgw_run_sync_thread', + }) + assert f == ('false' if disable_sync_traffic else 'true') + class TestMonService: @@ -3869,6 +3934,7 @@ class TestMgmtGateway: http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; @@ -4116,6 +4182,7 @@ class TestMgmtGateway: http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; diff --git a/src/pybind/mgr/dashboard/controllers/cluster_configuration.py b/src/pybind/mgr/dashboard/controllers/cluster_configuration.py index da5be2cc81d..292f381d79f 100644 --- a/src/pybind/mgr/dashboard/controllers/cluster_configuration.py +++ b/src/pybind/mgr/dashboard/controllers/cluster_configuration.py @@ -1,12 +1,14 @@ # -*- coding: utf-8 -*- +from typing import Optional + import cherrypy from .. import mgr from ..exceptions import DashboardException from ..security import Scope from ..services.ceph_service import CephService -from . import APIDoc, APIRouter, EndpointDoc, RESTController +from . import APIDoc, APIRouter, EndpointDoc, Param, RESTController FILTER_SCHEMA = [{ "name": (str, 'Name of the config option'), @@ -80,22 +82,33 @@ class ClusterConfiguration(RESTController): return config_options - def create(self, name, value): + @EndpointDoc("Create/Update Cluster Configuration", + parameters={ + 'name': Param(str, 'Config option name'), + 'value': ( + [ + { + 'section': Param( + str, 'Section/Client where config needs to be updated' + ), + 'value': Param(str, 'Value of the config option') + } + ], 'Section and Value of the config option' + ), + 'force_update': Param(bool, 'Force update the config option', False, None) + } + ) + def create(self, name, value, force_update: Optional[bool] = None): # Check if config option is updateable at runtime - self._updateable_at_runtime([name]) + self._updateable_at_runtime([name], force_update) - # Update config option - avail_sections = ['global', 'mon', 'mgr', 'osd', 'mds', 'client'] + for entry in value: + section = entry['section'] + entry_value = entry['value'] - for section in avail_sections: - for entry in value: - if entry['value'] is None: - break - - if entry['section'] == section: - CephService.send_command('mon', 'config set', who=section, name=name, - value=str(entry['value'])) - break + if entry_value not in (None, ''): + CephService.send_command('mon', 'config set', who=section, name=name, + value=str(entry_value)) else: CephService.send_command('mon', 'config rm', who=section, name=name) @@ -116,11 +129,24 @@ class ClusterConfiguration(RESTController): raise cherrypy.HTTPError(404) - def _updateable_at_runtime(self, config_option_names): + def _updateable_at_runtime(self, config_option_names, force_update=False): not_updateable = [] for name in config_option_names: config_option = self._get_config_option(name) + + # making rgw configuration to be editable by bypassing 'can_update_at_runtime' + # as the same can be done via CLI. + if force_update and 'rgw' in name and not config_option['can_update_at_runtime']: + break + + if force_update and 'rgw' not in name and not config_option['can_update_at_runtime']: + raise DashboardException( + msg=f'Only the configuration containing "rgw" can be edited at runtime with' + f' force_update flag, hence not able to update "{name}"', + code='config_option_not_updatable_at_runtime', + component='cluster_configuration' + ) if not config_option['can_update_at_runtime']: not_updateable.append(name) diff --git a/src/pybind/mgr/dashboard/controllers/rgw.py b/src/pybind/mgr/dashboard/controllers/rgw.py index 9d257674794..d48542a7590 100755 --- a/src/pybind/mgr/dashboard/controllers/rgw.py +++ b/src/pybind/mgr/dashboard/controllers/rgw.py @@ -106,13 +106,11 @@ class RgwMultisiteStatus(RESTController): @allow_empty_body # pylint: disable=W0102,W0613 def migrate(self, daemon_name=None, realm_name=None, zonegroup_name=None, zone_name=None, - zonegroup_endpoints=None, zone_endpoints=None, access_key=None, - secret_key=None): + zonegroup_endpoints=None, zone_endpoints=None, username=None): multisite_instance = RgwMultisite() result = multisite_instance.migrate_to_multisite(realm_name, zonegroup_name, zone_name, zonegroup_endpoints, - zone_endpoints, access_key, - secret_key) + zone_endpoints, username) return result @RESTController.Collection(method='POST', path='/multisite-replications') @@ -773,6 +771,9 @@ class RgwUser(RgwRESTController): return users def get(self, uid, daemon_name=None, stats=True) -> dict: + return self._get(uid, daemon_name=daemon_name, stats=stats) + + def _get(self, uid, daemon_name=None, stats=True) -> dict: query_params = '?stats' if stats else '' result = self.proxy(daemon_name, 'GET', 'user{}'.format(query_params), {'uid': uid, 'stats': stats}) @@ -788,7 +789,7 @@ class RgwUser(RgwRESTController): # type: (Optional[str]) -> List[str] emails = [] for uid in json.loads(self.list(daemon_name)): # type: ignore - user = json.loads(self.get(uid, daemon_name)) # type: ignore + user = self._get(uid, daemon_name) # type: ignore if user["email"]: emails.append(user["email"]) return emails @@ -910,7 +911,7 @@ class RgwUser(RgwRESTController): secret_key=None, daemon_name=None): # pylint: disable=R1705 subusr_array = [] - user = json.loads(self.get(uid, daemon_name)) # type: ignore + user = self._get(uid, daemon_name) # type: ignore subusers = user["subusers"] for sub_usr in subusers: subusr_array.append(sub_usr["id"]) diff --git a/src/pybind/mgr/dashboard/controllers/rgw_iam.py b/src/pybind/mgr/dashboard/controllers/rgw_iam.py new file mode 100644 index 00000000000..458bbbb7321 --- /dev/null +++ b/src/pybind/mgr/dashboard/controllers/rgw_iam.py @@ -0,0 +1,52 @@ +from typing import Optional + +from ..security import Scope +from ..services.rgw_iam import RgwAccounts +from ..tools import str_to_bool +from . import APIDoc, APIRouter, EndpointDoc, RESTController, allow_empty_body + + +@APIRouter('rgw/accounts', Scope.RGW) +@APIDoc("RGW User Accounts API", "RgwUserAccounts") +class RgwUserAccountsController(RESTController): + + @allow_empty_body + def create(self, account_name: Optional[str] = None, + account_id: Optional[str] = None, email: Optional[str] = None): + return RgwAccounts.create_account(account_name, account_id, email) + + def list(self, detailed: bool = False): + detailed = str_to_bool(detailed) + return RgwAccounts.get_accounts(detailed) + + @EndpointDoc("Get RGW Account by id", + parameters={'account_id': (str, 'Account id')}) + def get(self, account_id: str): + return RgwAccounts.get_account(account_id) + + @EndpointDoc("Delete RGW Account", + parameters={'account_id': (str, 'Account id')}) + def delete(self, account_id): + return RgwAccounts.delete_account(account_id) + + @EndpointDoc("Update RGW account info", + parameters={'account_id': (str, 'Account id')}) + @allow_empty_body + def set(self, account_id: str, account_name: Optional[str] = None, + email: Optional[str] = None): + return RgwAccounts.modify_account(account_id, account_name, email) + + @EndpointDoc("Set RGW Account/Bucket quota", + parameters={'account_id': (str, 'Account id'), + 'max_size': (str, 'Max size')}) + @RESTController.Resource(method='PUT', path='/quota') + @allow_empty_body + def set_quota(self, quota_type: str, account_id: str, max_size: str, max_objects: str): + return RgwAccounts.set_quota(quota_type, account_id, max_size, max_objects) + + @EndpointDoc("Enable/Disable RGW Account/Bucket quota", + parameters={'account_id': (str, 'Account id')}) + @RESTController.Resource(method='PUT', path='/quota/status') + @allow_empty_body + def set_quota_status(self, quota_type: str, account_id: str, quota_status: str): + return RgwAccounts.set_quota_status(quota_type, account_id, quota_status) diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts index 983140a44c4..b71719c4396 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts @@ -30,7 +30,6 @@ describe('Configuration page', () => { beforeEach(() => { configuration.clearTableSearchInput(); - configuration.getTableCount('found').as('configFound'); }); after(() => { @@ -50,6 +49,8 @@ describe('Configuration page', () => { }); it('should verify modified filter is applied properly', () => { + configuration.clearFilter(); + configuration.getTableCount('found').as('configFound'); configuration.filterTable('Modified', 'no'); configuration.getTableCount('found').as('unmodifiedConfigs'); diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts index 82e79a676ec..4132387d0f1 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts @@ -12,7 +12,6 @@ export class ConfigurationPageHelper extends PageHelper { configClear(name: string) { this.navigateTo(); const valList = ['global', 'mon', 'mgr', 'osd', 'mds', 'client']; // Editable values - this.getFirstTableCell(name).click(); cy.contains('button', 'Edit').click(); // Waits for the data to load @@ -26,6 +25,8 @@ export class ConfigurationPageHelper extends PageHelper { cy.wait(3 * 1000); + this.clearFilter(); + // Enter config setting name into filter box this.searchTable(name, 100); @@ -49,6 +50,7 @@ export class ConfigurationPageHelper extends PageHelper { * Ex: [global, '2'] is the global value with an input of 2 */ edit(name: string, ...values: [string, string][]) { + this.clearFilter(); this.getFirstTableCell(name).click(); cy.contains('button', 'Edit').click(); @@ -78,4 +80,12 @@ export class ConfigurationPageHelper extends PageHelper { cy.contains('[data-testid=config-details-table]', `${value[0]}\: ${value[1]}`); }); } + + clearFilter() { + cy.get('div.filter-tags') // Find the div with class filter-tags + .find('button.cds--btn.cds--btn--ghost') // Find the button with specific classes + .contains('Clear filters') // Ensure the button contains the text "Clear filters" + .should('be.visible') // Assert that the button is visible + .click(); + } } diff --git a/src/pybind/mgr/dashboard/frontend/package-lock.json b/src/pybind/mgr/dashboard/frontend/package-lock.json index a091d3577ec..7b3e9beeb9f 100644 --- a/src/pybind/mgr/dashboard/frontend/package-lock.json +++ b/src/pybind/mgr/dashboard/frontend/package-lock.json @@ -29,7 +29,7 @@ "@types/file-saver": "2.0.1", "async-mutex": "0.2.4", "bootstrap": "5.2.3", - "carbon-components-angular": "5.48.0", + "carbon-components-angular": "5.56.2", "chart.js": "4.4.0", "chartjs-adapter-moment": "1.0.1", "detect-browser": "5.2.0", @@ -11208,9 +11208,9 @@ ] }, "node_modules/carbon-components-angular": { - "version": "5.48.0", - "resolved": "https://registry.npmjs.org/carbon-components-angular/-/carbon-components-angular-5.48.0.tgz", - "integrity": "sha512-NZwpKBKgkgaR51S0Pm16MvashO4g8B+dzGeNE8l/RYRWXpVDLShR6YnBc80t2iMtTolxGiPwHmzpyMieVtIGLg==", + "version": "5.56.2", + "resolved": "https://registry.npmjs.org/carbon-components-angular/-/carbon-components-angular-5.56.2.tgz", + "integrity": "sha512-mU5Nep4HwwRQIrToasLwfR8cB1ph6Hn3jnLjDw0kWN+NJj5HEwPizOhbTPQbQxvBo0stro2fMWW0x3ge519Hgg==", "hasInstallScript": true, "dependencies": { "@carbon/icon-helpers": "10.37.0", diff --git a/src/pybind/mgr/dashboard/frontend/package.json b/src/pybind/mgr/dashboard/frontend/package.json index 3348c5ff097..aa0035dcb78 100644 --- a/src/pybind/mgr/dashboard/frontend/package.json +++ b/src/pybind/mgr/dashboard/frontend/package.json @@ -63,7 +63,7 @@ "@types/file-saver": "2.0.1", "async-mutex": "0.2.4", "bootstrap": "5.2.3", - "carbon-components-angular": "5.48.0", + "carbon-components-angular": "5.56.2", "chart.js": "4.4.0", "chartjs-adapter-moment": "1.0.1", "detect-browser": "5.2.0", diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts index bca65a887c5..b6d23e35c9d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts @@ -1,4 +1,5 @@ export class ConfigFormCreateRequestModel { name: string; value: Array<any> = []; + force_update: boolean = false; } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html index 741c18d52a6..a6775dcee17 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html @@ -150,7 +150,7 @@ </div> <!-- Footer --> <div class="card-footer"> - <cd-form-button-panel (submitActionEvent)="submit()" + <cd-form-button-panel (submitActionEvent)="forceUpdate ? openCriticalConfirmModal() : submit()" [form]="configForm" [submitText]="actionLabels.UPDATE" wrappingClass="text-right"></cd-form-button-panel> @@ -158,3 +158,4 @@ </div> </form> </div> + diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts index b6e9e700be4..118cb18430a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts @@ -13,6 +13,10 @@ import { CdForm } from '~/app/shared/forms/cd-form'; import { CdFormGroup } from '~/app/shared/forms/cd-form-group'; import { NotificationService } from '~/app/shared/services/notification.service'; import { ConfigFormCreateRequestModel } from './configuration-form-create-request.model'; +import { CriticalConfirmationModalComponent } from '~/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component'; +import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; + +const RGW = 'rgw'; @Component({ selector: 'cd-configuration-form', @@ -29,13 +33,15 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { maxValue: number; patternHelpText: string; availSections = ['global', 'mon', 'mgr', 'osd', 'mds', 'client']; + forceUpdate: boolean; constructor( public actionLabels: ActionLabelsI18n, private route: ActivatedRoute, private router: Router, private configService: ConfigurationService, - private notificationService: NotificationService + private notificationService: NotificationService, + private modalService: ModalCdsService ) { super(); this.createForm(); @@ -95,7 +101,6 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { setResponse(response: ConfigFormModel) { this.response = response; const validators = this.getValidators(response); - this.configForm.get('name').setValue(response.name); this.configForm.get('desc').setValue(response.desc); this.configForm.get('long_desc').setValue(response.long_desc); @@ -118,7 +123,7 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { this.configForm.get('values').get(value.section).setValue(sectionValue); }); } - + this.forceUpdate = !this.response.can_update_at_runtime && response.name.includes(RGW); this.availSections.forEach((section) => { this.configForm.get('values').get(section).setValidators(validators); }); @@ -134,7 +139,7 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { this.availSections.forEach((section) => { const sectionValue = this.configForm.getValue(section); - if (sectionValue !== null && sectionValue !== '') { + if (sectionValue !== null) { values.push({ section: section, value: sectionValue }); } }); @@ -143,12 +148,28 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { const request = new ConfigFormCreateRequestModel(); request.name = this.configForm.getValue('name'); request.value = values; + if (this.forceUpdate) { + request.force_update = this.forceUpdate; + } return request; } return null; } + openCriticalConfirmModal() { + this.modalService.show(CriticalConfirmationModalComponent, { + buttonText: $localize`Force Edit`, + actionDescription: $localize`force edit`, + itemDescription: $localize`configuration`, + infoMessage: 'Updating this configuration might require restarting the client', + submitAction: () => { + this.modalService.dismissAll(); + this.submit(); + } + }); + } + submit() { const request = this.createRequest(); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts index a57603d4c8a..7a446ce808c 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts @@ -12,6 +12,8 @@ import { CdTableSelection } from '~/app/shared/models/cd-table-selection'; import { Permission } from '~/app/shared/models/permissions'; import { AuthStorageService } from '~/app/shared/services/auth-storage.service'; +const RGW = 'rgw'; + @Component({ selector: 'cd-configuration', templateUrl: './configuration.component.html', @@ -26,10 +28,26 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { selection = new CdTableSelection(); filters: CdTableColumn[] = [ { + name: $localize`Modified`, + prop: 'modified', + filterOptions: [$localize`yes`, $localize`no`], + filterInitValue: $localize`yes`, + filterPredicate: (row, value) => { + if (value === 'yes' && row.hasOwnProperty('value')) { + return true; + } + + if (value === 'no' && !row.hasOwnProperty('value')) { + return true; + } + + return false; + } + }, + { name: $localize`Level`, prop: 'level', filterOptions: ['basic', 'advanced', 'dev'], - filterInitValue: 'basic', filterPredicate: (row, value) => { enum Level { basic = 0, @@ -60,22 +78,6 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { } return row.source.includes(value); } - }, - { - name: $localize`Modified`, - prop: 'modified', - filterOptions: ['yes', 'no'], - filterPredicate: (row, value) => { - if (value === 'yes' && row.hasOwnProperty('value')) { - return true; - } - - if (value === 'no' && !row.hasOwnProperty('value')) { - return true; - } - - return false; - } } ]; @@ -143,7 +145,9 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { if (selection.selected.length !== 1) { return false; } - - return selection.selected[0].can_update_at_runtime; + if ((this.selection.selected[0].name as string).includes(RGW)) { + return true; + } + return this.selection.selected[0].can_update_at_runtime; } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html index b05d07fb31b..79cd60c970f 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html @@ -77,7 +77,7 @@ [class.icon-danger-color]="row.remainingDays < 2" [class.icon-warning-color]="row.remainingDays < 8" class="{{ icons.warning }}"></i> - <span title="{{ value | cdDate }}">{{ row.remainingTimeWithoutSeconds / 1000 | duration }}</span> + <span title="{{ row.expiryDate }}">{{ row.remainingTimeWithoutSeconds / 1000 | duration }}</span> </span> <span *ngIf="row.remainingTimeWithoutSeconds <= 0 && row.remainingDays <=0 && row.cluster_alias !== 'local-cluster'"> <i i18n-title diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts index 78b4c9c1859..cfdc2e1720e 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts @@ -115,6 +115,7 @@ export class MultiClusterListComponent extends ListWithDetails implements OnInit cluster['ttl'] ); cluster['remainingDays'] = this.getRemainingDays(cluster['ttl']); + cluster['expiryDate'] = new Date(Date.now() + cluster['ttl']).toLocaleString(); } }); } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts index 40c2e95d1e0..a07dcfcdd35 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts @@ -29,6 +29,7 @@ import { PlacementPipe } from './placement.pipe'; import { ServiceFormComponent } from './service-form/service-form.component'; import { SettingsService } from '~/app/shared/api/settings.service'; import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; +import { CellTemplate } from '~/app/shared/enum/cell-template.enum'; const BASE_URL = 'services'; @@ -176,6 +177,16 @@ export class ServicesComponent extends ListWithDetails implements OnChanges, OnI prop: 'status.last_refresh', pipe: this.relativeDatePipe, flexGrow: 1 + }, + { + name: $localize`Ports`, + prop: 'status.ports', + flexGrow: 1, + cellTransformation: CellTemplate.map, + customTemplateConfig: { + undefined: '-', + '': '-' + } } ]; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts index 3e146ef7ad5..f96ec0ccffa 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts @@ -56,7 +56,7 @@ export class RgwMultisiteZonegroupDeletionFormComponent implements OnInit, After .subscribe(() => { this.notificationService.show( NotificationType.success, - $localize`Zone: '${this.zonegroup.name}' deleted successfully` + $localize`Zonegroup: '${this.zonegroup.name}' deleted successfully` ); this.activeModal.close(); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts index 4496b056734..b7400219120 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts @@ -100,6 +100,11 @@ export class RgwBucketListComponent extends ListWithDetails implements OnInit, O prop: 'object_usage', cellTemplate: this.bucketObjectTpl, flexGrow: 0.8 + }, + { + name: $localize`Number of Shards`, + prop: 'num_shards', + flexGrow: 0.8 } ]; const getBucketUri = () => diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html index 51f72dd7f89..9117e71c34b 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html @@ -111,36 +111,21 @@ </div> <div class="form-group row"> <label class="cd-col-form-label required" - for="access_key" - i18n>S3 access key - <cd-helper> - <span>To see or copy your S3 access key, go to <b>Object Gateway > Users</b> and click on your user name. In <b>Keys</b>, click <b>Show</b>. View the access key by clicking Show and copy the key by clicking <b>Copy to Clipboard</b>.</span> - </cd-helper> - </label> + for="username" + i18n>Username</label> <div class="cd-col-form-input"> <input class="form-control" type="text" - placeholder="e.g." - id="access_key" - name="access_key" - formControlName="access_key"> - </div> - </div> - <div class="form-group row"> - <label class="cd-col-form-label required" - for="access_key" - i18n>S3 secret key - <cd-helper> - <span>To see or copy your S3 access key, go to <b>Object Gateway > Users</b> and click on your user name. In <b>Keys</b>, click <b>Show</b>. View the secret key by clicking Show and copy the key by clicking <b>Copy to Clipboard</b>.</span> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="e.g." - id="secret_key" - name="secret_key" - formControlName="secret_key"> + placeholder="username" + id="username" + name="username" + formControlName="username"> + <cd-help-text> + <span i18n>Specify the username for the system user. This user will be created automatically as part of the process.</span> + </cd-help-text> + <span class="invalid-feedback" + *ngIf="multisiteMigrateForm.showError('username', formDir, 'required')" + i18n>This field is required.</span> </div> </div> </div> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts index 1073dee429a..d9ad56a5bf4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts @@ -11,7 +11,7 @@ import { NotificationType } from '~/app/shared/enum/notification-type.enum'; import { CdFormGroup } from '~/app/shared/forms/cd-form-group'; import { CdValidators } from '~/app/shared/forms/cd-validators'; import { NotificationService } from '~/app/shared/services/notification.service'; -import { RgwRealm, RgwZone, RgwZonegroup, SystemKey } from '../models/rgw-multisite'; +import { RgwRealm, RgwZone, RgwZonegroup } from '../models/rgw-multisite'; import { ModalService } from '~/app/shared/services/modal.service'; import { RgwDaemonService } from '~/app/shared/api/rgw-daemon.service'; @@ -135,8 +135,9 @@ export class RgwMultisiteMigrateComponent implements OnInit { Validators.required ] ), - access_key: new UntypedFormControl(null), - secret_key: new UntypedFormControl(null) + username: new UntypedFormControl(null, { + validators: [Validators.required] + }) }); } @@ -174,21 +175,21 @@ export class RgwMultisiteMigrateComponent implements OnInit { this.zone = new RgwZone(); this.zone.name = values['zoneName']; this.zone.endpoints = values['zone_endpoints']; - this.zone.system_key = new SystemKey(); - this.zone.system_key.access_key = values['access_key']; - this.zone.system_key.secret_key = values['secret_key']; - this.rgwMultisiteService.migrate(this.realm, this.zonegroup, this.zone).subscribe( - () => { - this.notificationService.show( - NotificationType.success, - $localize`Migration done successfully` - ); - this.submitAction.emit(); - this.activeModal.close(); - }, - () => { - this.notificationService.show(NotificationType.error, $localize`Migration failed`); - } - ); + this.rgwMultisiteService + .migrate(this.realm, this.zonegroup, this.zone, values['username']) + .subscribe( + () => { + this.rgwMultisiteService.setRestartGatewayMessage(false); + this.notificationService.show( + NotificationType.success, + $localize`Migration done successfully` + ); + this.submitAction.emit(); + this.activeModal.close(); + }, + () => { + this.notificationService.show(NotificationType.error, $localize`Migration failed`); + } + ); } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html index 3a9ce12df9d..16963b06920 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html @@ -60,6 +60,7 @@ </cd-dashboard-area-chart> <cd-dashboard-area-chart chartTitle="Latency" dataUnits="ms" + decimals="2" [labelsArray]="['GET', 'PUT']" [dataArray]="[queriesResults.AVG_GET_LATENCY, queriesResults.AVG_PUT_LATENCY]"> </cd-dashboard-area-chart> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts index c0e0517896c..f1f04f7c2f0 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts @@ -17,13 +17,48 @@ import { UserFormComponent } from './user-form/user-form.component'; import { UserListComponent } from './user-list/user-list.component'; import { UserPasswordFormComponent } from './user-password-form/user-password-form.component'; import { UserTabsComponent } from './user-tabs/user-tabs.component'; -import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-components-angular'; + +import { + ButtonModule, + CheckboxModule, + DatePickerModule, + GridModule, + IconModule, + IconService, + InputModule, + ModalModule, + NumberModule, + RadioModule, + SelectModule, + UIShellModule, + TimePickerModule, + ComboBoxModule +} from 'carbon-components-angular'; +// Icons +import ChevronDown from '@carbon/icons/es/chevron--down/16'; +import Close from '@carbon/icons/es/close/32'; +import AddFilled from '@carbon/icons/es/add--filled/32'; +import SubtractFilled from '@carbon/icons/es/subtract--filled/32'; +import Reset from '@carbon/icons/es/reset/32'; +import EyeIcon from '@carbon/icons/es/view/16'; @NgModule({ imports: [ CommonModule, FormsModule, ReactiveFormsModule, SharedModule, + UIShellModule, + InputModule, + GridModule, + ButtonModule, + IconModule, + CheckboxModule, + RadioModule, + SelectModule, + NumberModule, + ModalModule, + DatePickerModule, + TimePickerModule, NgbNavModule, NgbPopoverModule, NgxPipeFunctionModule, @@ -31,8 +66,8 @@ import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-compon NgbModule, IconModule, GridModule, - ButtonModule, - InputModule + InputModule, + ComboBoxModule ], declarations: [ LoginComponent, @@ -46,7 +81,11 @@ import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-compon UserPasswordFormComponent ] }) -export class AuthModule {} +export class AuthModule { + constructor(private iconService: IconService) { + this.iconService.registerAll([ChevronDown, Close, AddFilled, SubtractFilled, Reset, EyeIcon]); + } +} const routes: Routes = [ { path: '', redirectTo: 'users', pathMatch: 'full' }, diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts index 2d323b04ea5..abf529196f6 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts @@ -4,11 +4,12 @@ export class UserFormRoleModel implements SelectOption { name: string; description: string; selected = false; - scopes_permissions: object; - enabled = true; - - constructor(name: string, description: string) { + scopes_permissions?: object; + enabled: boolean; + content: string; + constructor(name: string, description: string, content: string) { this.name = name; this.description = description; + this.content = content; } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html index 4169d54c39f..d2e52158473 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html @@ -1,249 +1,205 @@ -<div class="cd-col-form" - *cdFormLoading="loading"> - <form name="userForm" - #formDir="ngForm" - [formGroup]="userForm" - novalidate> - <div class="card"> +<div cdsCol + [columnNumbers]="{md: 4}"> + <ng-container *cdFormLoading="loading"> + <form #frm="ngForm" + #formDir="ngForm" + [formGroup]="userForm" + novalidate> <div i18n="form title" - class="card-header">{{ action | titlecase }} {{ resource | upperFirst }}</div> - <div class="card-body"> - - <!-- Username --> - <div class="form-group row"> - <label class="cd-col-form-label" - [ngClass]="{'required': mode !== userFormMode.editing}" - for="username" - i18n>Username</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="Username..." - id="username" - name="username" - formControlName="username" - autocomplete="off" - autofocus - ngbTooltip="White spaces at the beginning and end will be trimmed" - i18n-ngbTooltip - cdTrim> - <span class="invalid-feedback" - *ngIf="userForm.showError('username', formDir, 'required')" - i18n>This field is required.</span> - <span class="invalid-feedback" - *ngIf="userForm.showError('username', formDir, 'notUnique')" - i18n>The username already exists.</span> - </div> - </div> - - <!-- Password --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label class="cd-col-form-label" - for="password"> - <ng-container i18n>Password</ng-container> - <cd-helper *ngIf="passwordPolicyHelpText.length > 0" - class="text-pre-wrap" - html="{{ passwordPolicyHelpText }}"> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - type="password" - placeholder="Password..." - id="password" - name="password" - autocomplete="new-password" - formControlName="password"> - <button type="button" - class="btn btn-light" - cdPasswordButton="password"> - </button> - </div> - <div class="password-strength-level"> - <div class="{{ passwordStrengthLevelClass }}" - data-toggle="tooltip" - title="{{ passwordValuation }}"> - </div> - </div> - <span class="invalid-feedback" - *ngIf="userForm.showError('password', formDir, 'required')" - i18n>This field is required.</span> - <span class="invalid-feedback" - *ngIf="userForm.showError('password', formDir, 'passwordPolicy')"> - {{ passwordValuation }} - </span> - </div> - </div> - - <!-- Confirm password --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label i18n - class="cd-col-form-label" - for="confirmpassword">Confirm password</label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - type="password" - placeholder="Confirm password..." - id="confirmpassword" - name="confirmpassword" - autocomplete="new-password" - formControlName="confirmpassword"> - <button type="button" - class="btn btn-light" - cdPasswordButton="confirmpassword"> - </button> - <span class="invalid-feedback" - *ngIf="userForm.showError('confirmpassword', formDir, 'match')" - i18n>Password confirmation doesn't match the password.</span> - </div> - <span class="invalid-feedback" - *ngIf="userForm.showError('confirmpassword', formDir, 'required')" - i18n>This field is required.</span> - </div> - </div> + class="form-header">{{ action | titlecase }} {{ resource | upperFirst }} + </div> + <!-- UserName --> + <div class="form-item"> + <cds-text-label labelInputID="username" + cdRequiredField="Username" + [invalid]="!userForm.controls.username.valid && userForm.controls.username.dirty" + [invalidText]="usernameError" + i18n>Username + <input cdsText + placeholder="Username..." + i18n-placeholder + id="username" + formControlName="username" + [invalid]="!userForm.controls.username.valid && userForm.controls.username.dirty" + autofocus + ngbTooltip="White spaces at the beginning and end will be trimmed" + i18n-ngbTooltip + cdTrim> + </cds-text-label> + <ng-template #usernameError> + <span *ngIf="userForm.showError('username', formDir, 'required')"> + <ng-container i18n> + This field is required. + </ng-container> + </span> + <span *ngIf="userForm.showError('username', formDir, 'notUnique')"> + <ng-container i18n> + The username already exists. + </ng-container> + </span> + </ng-template> + </div> + <!-- Password --> + <div class="form-item"> + <cds-password-label labelInputID="password" + label="Password..." + [invalid]="!userForm.controls.password.valid && userForm.controls.password.dirty" + [invalidText]="passwordError" + i18n>Password + <cd-helper *ngIf="passwordPolicyHelpText.length > 0" + class="text-pre-wrap" + html="{{ passwordPolicyHelpText }}"> + </cd-helper> + <input cdsPassword + type="password" + placeholder="Password..." + id="password" + autocomplete="new-password" + formControlName="password" + > + </cds-password-label> + <ng-template #passwordError> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'match')" + i18n>Password confirmation doesn't match the password. + </span> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'required')" + i18n>This field is required.</span> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'passwordPolicy')"> + {{ passwordValuation }} + </span> + </ng-template> + </div> - <!-- Password expiration date --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label class="cd-col-form-label" - [ngClass]="{'required': pwdExpirationSettings.pwdExpirationSpan > 0}" - for="pwdExpirationDate"> - <ng-container i18n>Password expiration date</ng-container> - <cd-helper class="text-pre-wrap" - *ngIf="pwdExpirationSettings.pwdExpirationSpan == 0"> - <p> + <!-- Confirm password --> + <div class="form-item"> + <cds-password-label labelInputID="confirmpassword" + label="Confirm password..." + [invalid]="!userForm.controls.confirmpassword.valid && userForm.controls.confirmpassword.dirty" + [invalidText]="confirmpasswordError" + i18n> Confirm password + <input cdsPassword + type="password" + placeholder="Confirm password..." + id="confirmpassword" + formControlName="confirmpassword"> + </cds-password-label> + <ng-template #confirmpasswordError> + <span class="invalid-feedback" + *ngIf="userForm.showError('confirmpassword', formDir, 'match')" + i18n>Password confirmation doesn't match the password.</span> + <span class="invalid-feedback" + *ngIf="userForm.showError('confirmpassword', formDir, 'required')" + i18n>This field is required.</span> + </ng-template> + </div> + <!-- Password expiration date --> + <div class="form-item" + *ngIf="!authStorageService.isSSO()"> + <cds-text-label [ngClass]="{'required': pwdExpirationSettings.pwdExpirationSpan > 0}">{{'Password Expiration Date'}} + <cd-helper class="text-pre-wrap" + *ngIf="pwdExpirationSettings.pwdExpirationSpan == 0"> + <span> The Dashboard setting defining the expiration interval of passwords is currently set to <strong>0</strong>. This means if a date is set, the user password will only expire once. - </p> - <p> - Consider configuring the Dashboard setting - <a routerLink="/mgr-modules/edit/dashboard" - class="alert-link">USER_PWD_EXPIRATION_SPAN</a> - in order to let passwords expire periodically. - </p> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - i18n-placeholder - placeholder="Password expiration date..." - id="pwdExpirationDate" - name="pwdExpirationDate" - formControlName="pwdExpirationDate" - [ngbPopover]="popContent" - triggers="manual" - #p="ngbPopover" - (click)="p.open()" - (keypress)="p.close()"> - <button type="button" - class="btn btn-light" - (click)="clearExpirationDate()"> - <i class="icon-prepend {{ icons.destroy }}"></i> - </button> - <span class="invalid-feedback" - *ngIf="userForm.showError('pwdExpirationDate', formDir, 'required')" - i18n>This field is required.</span> - </div> - </div> - </div> - - <!-- Name --> - <div class="form-group row"> - <label i18n - class="cd-col-form-label" - for="name">Full name</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="Full name..." - id="name" - name="name" - formControlName="name"> - </div> - </div> - - <!-- Email --> - <div class="form-group row"> - <label i18n - class="cd-col-form-label" - for="email">Email</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="email" - placeholder="Email..." - id="email" - name="email" - formControlName="email"> - - <span class="invalid-feedback" - *ngIf="userForm.showError('email', formDir, 'email')" - i18n>Invalid email.</span> - </div> - </div> - - <!-- Roles --> - <div class="form-group row"> - <label class="cd-col-form-label" - i18n>Roles</label> - <div class="cd-col-form-input"> - <span class="no-border full-height" - *ngIf="allRoles"> - <cd-select-badges [data]="userForm.controls.roles.value" - [options]="allRoles" - [messages]="messages"></cd-select-badges> </span> - </div> - </div> - - <!-- Enabled --> - <div class="form-group row" - *ngIf="!isCurrentUser()"> - <div class="cd-col-form-offset"> - <div class="custom-control custom-checkbox"> - <input type="checkbox" - class="custom-control-input" - id="enabled" - name="enabled" - formControlName="enabled"> - <label class="custom-control-label" - for="enabled" - i18n>Enabled</label> - </div> - </div> - </div> - - <!-- Force change password --> - <div class="form-group row" - *ngIf="!isCurrentUser() && !authStorageService.isSSO()"> - <div class="cd-col-form-offset"> - <div class="custom-control custom-checkbox"> - <input type="checkbox" - class="custom-control-input" - id="pwdUpdateRequired" - name="pwdUpdateRequired" - formControlName="pwdUpdateRequired"> - <label class="custom-control-label" - for="pwdUpdateRequired" - i18n>User must change password at next logon</label> - </div> - </div> - </div> - + <span>Consider configuring the Dashboard setting + <a routerLink="/mgr-modules/edit/dashboard" + class="alert-link">USER_PWD_EXPIRATION_SPAN</a> + in order to let passwords expire periodically. + </span> + </cd-helper> + <cd-date-time-picker [control]="userForm.get('pwdExpirationDate')" + placeHolder="Password expiration date" + [hasTime]="false" + [defaultDate]="passwordexp" + i18n-name + > + </cd-date-time-picker> + </cds-text-label> + <span class="invalid-feedback" + *ngIf="userForm.showError('pwdExpirationDate', formDir, 'required')" + i18n>This field is required. + </span> + </div> + <!--Full Name--> + <div class="form-item"> + <cds-text-label for="name" + i18n> Full Name + <input cdsText + type="text" + placeholder="Full Name..." + id="name" + formControlName="name"> + </cds-text-label> + </div> + <!-- Email --> + <div class="form-item"> + <cds-text-label for="email" + [invalid]="!userForm.controls.email.valid && userForm.controls.email.dirty" + [invalidText]="emailError" + i18n> + Email + <input cdsText + type="email" + placeholder="Email..." + id="email" + formControlName="email"> + </cds-text-label> + <ng-template #emailError> + <span class="invalid-feedback" + *ngIf="userForm.showError('email', formDir, 'email')" + i18n>Invalid email. + </span> + </ng-template> + </div> + <!-- Roles --> + <div class="form-item" + *ngIf="allRoles"> + <cds-combo-box label="Roles" + type="multi" + selectionFeedback="top-after-reopen" + for="roles" + formControlName="roles" + id="roles" + placeholder="Select Roles..." + i18n-placeholder + [appendInline]="true" + [items]="allRoles" + itemValueKey="name" + i18n> + <cds-dropdown-list></cds-dropdown-list> + </cds-combo-box> + </div> + <!-- Enabled --> + <div class="form-item" + *ngIf="!isCurrentUser()"> + <cds-checkbox id="enabled" + formControlName="enabled" + name="enabled" + i18n>Enabled + </cds-checkbox> </div> - <div class="card-footer"> - <cd-form-button-panel (submitActionEvent)="submit()" - [form]="userForm" - [submitText]="(action | titlecase) + ' ' + (resource | upperFirst)" - wrappingClass="text-right"></cd-form-button-panel> + <!-- Force change password --> + <div class="form-item" + *ngIf="!isCurrentUser() && !authStorageService.isSSO()"> + <cds-checkbox id="pwdUpdateRequired" + formControlName="pwdUpdateRequired" + name="pwdUpdateRequired" + i18n>User must change password at next logon + </cds-checkbox> </div> - </div> - </form> + <!--Submit Button--> + <cd-form-button-panel (submitActionEvent)="submit()" + [form]="userForm" + [submitText]="(action | titlecase) + ' ' + (resource | upperFirst)" + wrappingClass="text-right"> + </cd-form-button-panel> + </form> + </ng-container> </div> <ng-template #removeSelfUserReadUpdatePermissionTpl> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts index 7c02b86eae0..009d4c193e4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts @@ -55,7 +55,8 @@ export class UserFormComponent extends CdForm implements OnInit { icons = Icons; pwdExpirationSettings: CdPwdExpirationSettings; pwdExpirationFormat = 'YYYY-MM-DD'; - + selectedRole: string[]; + passwordexp: boolean = false; constructor( private authService: AuthService, private authStorageService: AuthStorageService, @@ -91,6 +92,7 @@ export class UserFormComponent extends CdForm implements OnInit { password: [ '', [], + [ CdValidators.passwordPolicy( this.userService, @@ -105,7 +107,7 @@ export class UserFormComponent extends CdForm implements OnInit { ] ], confirmpassword: [''], - pwdExpirationDate: [undefined], + pwdExpirationDate: [''], email: ['', [CdValidators.email]], roles: [[]], enabled: [true, [Validators.required]], @@ -121,8 +123,10 @@ export class UserFormComponent extends CdForm implements OnInit { if (this.router.url.startsWith('/user-management/users/edit')) { this.mode = this.userFormMode.editing; this.action = this.actionLabels.EDIT; + this.passwordexp = false; } else { this.action = this.actionLabels.CREATE; + this.passwordexp = true; } const observables = [this.roleService.list(), this.settingsService.getStandardSettings()]; @@ -130,6 +134,7 @@ export class UserFormComponent extends CdForm implements OnInit { (result: [UserFormRoleModel[], CdPwdExpirationSettings]) => { this.allRoles = _.map(result[0], (role) => { role.enabled = true; + role.content = `${role.name}, ${role.description}`; return role; }); this.pwdExpirationSettings = new CdPwdExpirationSettings(result[1]); @@ -158,7 +163,6 @@ export class UserFormComponent extends CdForm implements OnInit { this.userService.get(username).subscribe((userFormModel: UserFormModel) => { this.response = _.cloneDeep(userFormModel); this.setResponse(userFormModel); - this.loadingReady(); }); }); @@ -173,20 +177,28 @@ export class UserFormComponent extends CdForm implements OnInit { this.userForm.get(key).setValue(response[key]) ); const expirationDate = response['pwdExpirationDate']; + if (expirationDate) { + this.passwordexp = false; this.userForm .get('pwdExpirationDate') .setValue(moment(expirationDate * 1000).format(this.pwdExpirationFormat)); + } else { + this.passwordexp = true; } } getRequest(): UserFormModel { const userFormModel = new UserFormModel(); + ['username', 'password', 'name', 'email', 'roles', 'enabled', 'pwdUpdateRequired'].forEach( - (key) => (userFormModel[key] = this.userForm.get(key).value) + (key) => { + userFormModel[key] = this.userForm.get(key).value; + } ); const expirationDate = this.userForm.get('pwdExpirationDate').value; if (expirationDate) { + this.passwordexp = false; const mom = moment(expirationDate, this.pwdExpirationFormat); if ( this.mode !== this.userFormMode.editing || diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts index 317293be07c..fedc7b8de0f 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts @@ -163,11 +163,9 @@ export class PrometheusService { checkNan ) { queriesResults[queryName].forEach((valueArray: any[]) => { - valueArray.forEach((val, index) => { - if (isNaN(parseFloat(val[1]))) { - valueArray[index][1] = '0'; - } - }); + if (isNaN(parseFloat(valueArray[1]))) { + valueArray[1] = '0'; + } }); } }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts index 3dc886e172f..8a39dc8a284 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts @@ -16,7 +16,7 @@ export class RgwMultisiteService { constructor(private http: HttpClient, public rgwDaemonService: RgwDaemonService) {} - migrate(realm: RgwRealm, zonegroup: RgwZonegroup, zone: RgwZone) { + migrate(realm: RgwRealm, zonegroup: RgwZonegroup, zone: RgwZone, username: string) { return this.rgwDaemonService.request((params: HttpParams) => { params = params.appendAll({ realm_name: realm.name, @@ -24,8 +24,7 @@ export class RgwMultisiteService { zone_name: zone.name, zonegroup_endpoints: zonegroup.endpoints, zone_endpoints: zone.endpoints, - access_key: zone.system_key.access_key, - secret_key: zone.system_key.secret_key + username: username }); return this.http.put(`${this.uiUrl}/migrate`, null, { params: params }); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts index d3ebc5f37c6..0e1c0906f4a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts @@ -9,4 +9,5 @@ export class ConfigFormModel { min: any; max: any; services: Array<string>; + can_update_at_runtime: boolean; } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html index 4b973187dbb..90e3a1d2be8 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html @@ -49,7 +49,7 @@ [form]="deletionForm" [submitText]="(actionDescription | titlecase) + ' ' + itemDescription" [modalForm]="true" - [submitBtnType]="actionDescription === 'delete' || 'remove' ? 'danger' : 'primary'"></cd-form-button-panel> + [submitBtnType]="(actionDescription === 'delete' || actionDescription === 'remove') ? 'danger' : 'primary'"></cd-form-button-panel> </cds-modal> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html index 328e72cc595..ccdb70e39e4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html @@ -1,22 +1,23 @@ -<div cdsCol - class="form-item"> - <div cdsRow> -<cds-date-picker [label]="name" - i18n-label - placeholder="NOT PROTECTED" - formControlname="expiresAt" - dateFormat="Y/m/d" - [value]="date" - (valueChange)="onModelChange($event)" - [helperText]="helperText" - [disabled]="disabled" - cdsTheme="theme"></cds-date-picker> -<cds-timepicker (valueChange)="onModelChange($event)" - [(ngModel)]="time" - label="Select a time" - [disabled]="disabled" - pattern="(1[012]|[0-9]):[0-5][0-9]" - *ngIf="hasTime"> +<div cdsCol> + <div cdsRow + class="form-item-append"> + <cds-text-label>{{name}} + <cds-date-picker i18n-label + [placeholder]="placeHolder" + formControlname="expiresAt" + dateFormat="Y/m/d" + [value]="date" + (valueChange)="onModelChange($event)" + [helperText]="helperText" + [disabled]="disabled" + cdsTheme="theme"></cds-date-picker> + </cds-text-label> + <cds-text-label *ngIf="hasTime">Select a time + <cds-timepicker (valueChange)="onModelChange($event)" + [(ngModel)]="time" + [disabled]="disabled" + pattern="(1[012]|[0-9]):[0-5][0-9]" + *ngIf="hasTime"> <cds-timepicker-select [(ngModel)]="ampm" [disabled]="disabled" (valueChange)="onModelChange($event)"> @@ -24,5 +25,7 @@ value="AM">AM</option> <option value="PM">PM</option> </cds-timepicker-select> -</cds-timepicker></div> +</cds-timepicker> +</cds-text-label> +</div> </div> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss index e69de29bb2d..39f2a7115a1 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss @@ -0,0 +1,3 @@ +.form-item-append { + margin-top: 1rem; +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts index 4841d2ed92d..3458d9171a7 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts @@ -25,7 +25,8 @@ export class DateTimePickerComponent implements OnInit { @Input() helperText = ''; - + @Input() + placeHolder = ''; @Input() disabled = false; @@ -39,9 +40,8 @@ export class DateTimePickerComponent implements OnInit { date: { [key: number]: string }[] = []; time: string; ampm: string; - sub: Subscription; - + @Input() defaultDate: boolean = false; constructor(private calendar: NgbCalendar) {} ngOnInit() { @@ -59,8 +59,12 @@ export class DateTimePickerComponent implements OnInit { if (!mom.isValid() || mom.isBefore(moment())) { mom = moment(); } + if (this.defaultDate) { + this.date.push([]); + } else { + this.date.push(mom.format('YYYY-MM-DD')); + } - this.date.push(mom.format('YYYY-MM-DD')); const time = mom.format('HH:mm:ss'); this.time = mom.format('hh:mm'); this.ampm = mom.hour() >= 12 ? 'PM' : 'AM'; @@ -76,7 +80,9 @@ export class DateTimePickerComponent implements OnInit { onModelChange(event?: any) { if (event) { - if (Array.isArray(event)) { + if (event.length === 0) { + this.datetime.date = { date: null, time: null, ampm: null }; + } else if (Array.isArray(event)) { this.datetime.date = moment(event[0]).format('YYYY-MM-DD'); } else if (event && ['AM', 'PM'].includes(event)) { const initialMoment = moment(this.datetime.time, 'hh:mm:ss A'); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html index da1a4800f7f..81ad90914b6 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html @@ -5,7 +5,8 @@ <ng-content></ng-content> </ng-template> -<cds-tooltip [description]="popoverTpl"> +<cds-tooltip [description]="popoverTpl" + [autoAlign]="true"> <svg cdsIcon="information" size="16" title="info"></svg> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts index 361a404a11b..f1bbebed51d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts @@ -11,8 +11,8 @@ export enum Promqls { export enum RgwPromqls { RGW_REQUEST_PER_SECOND = 'sum(rate(ceph_rgw_req[1m]))', - AVG_GET_LATENCY = 'sum(rate(ceph_rgw_op_get_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_get_obj_lat_count[1m]))', - AVG_PUT_LATENCY = 'sum(rate(ceph_rgw_op_put_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_put_obj_lat_count[1m]))', + AVG_GET_LATENCY = '(sum(rate(ceph_rgw_op_get_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_get_obj_lat_count[1m]))) * 1000', + AVG_PUT_LATENCY = '(sum(rate(ceph_rgw_op_put_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_put_obj_lat_count[1m]))) * 1000', GET_BANDWIDTH = 'sum(rate(ceph_rgw_op_get_obj_bytes[1m]))', PUT_BANDWIDTH = 'sum(rate(ceph_rgw_op_put_obj_bytes[1m]))' } diff --git a/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss b/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss index 37f89aba17f..0725b63dbfd 100644 --- a/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss +++ b/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss @@ -33,6 +33,7 @@ $content-theme: map-merge( text-primary: vv.$dark, text-secondary: vv.$dark, text-disabled: vv.$gray-500, + icon-secondary: vv.$gray-800, field-01: colors.$gray-10, interactive: vv.$primary ) diff --git a/src/pybind/mgr/dashboard/openapi.yaml b/src/pybind/mgr/dashboard/openapi.yaml index de836298064..de1b3e8b60e 100644 --- a/src/pybind/mgr/dashboard/openapi.yaml +++ b/src/pybind/mgr/dashboard/openapi.yaml @@ -3977,10 +3977,27 @@ paths: application/json: schema: properties: + force_update: + description: Force update the config option + type: boolean name: + description: Config option name type: string value: - type: string + description: Section and Value of the config option + items: + properties: + section: + description: Section/Client where config needs to be updated + type: string + value: + description: Value of the config option + type: string + required: + - section + - value + type: object + type: array required: - name - value @@ -4007,6 +4024,7 @@ paths: trace. security: - jwt: [] + summary: Create/Update Cluster Configuration tags: - ClusterConfiguration put: @@ -10790,6 +10808,274 @@ paths: - jwt: [] tags: - Prometheus + /api/rgw/accounts: + get: + parameters: + - default: false + in: query + name: detailed + schema: + type: boolean + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + tags: + - RgwUserAccounts + post: + parameters: [] + requestBody: + content: + application/json: + schema: + properties: + account_id: + type: integer + account_name: + type: integer + email: + type: string + type: object + responses: + '201': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource created. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}: + delete: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + responses: + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '204': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource deleted. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Delete RGW Account + tags: + - RgwUserAccounts + get: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Get RGW Account by id + tags: + - RgwUserAccounts + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + account_name: + type: integer + email: + type: string + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Update RGW account info + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}/quota: + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + max_objects: + type: string + max_size: + description: Max size + type: string + quota_type: + type: string + required: + - quota_type + - max_size + - max_objects + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Set RGW Account/Bucket quota + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}/quota/status: + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + quota_status: + type: string + quota_type: + type: string + required: + - quota_type + - quota_status + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Enable/Disable RGW Account/Bucket quota + tags: + - RgwUserAccounts /api/rgw/bucket: get: parameters: @@ -15966,6 +16252,8 @@ tags: name: RgwSite - description: RGW User Management API name: RgwUser +- description: RGW User Accounts API + name: RgwUserAccounts - description: '*No description available*' name: RgwZone - description: '*No description available*' diff --git a/src/pybind/mgr/dashboard/services/rgw_client.py b/src/pybind/mgr/dashboard/services/rgw_client.py index 2fe09821694..9fa249acf44 100755 --- a/src/pybind/mgr/dashboard/services/rgw_client.py +++ b/src/pybind/mgr/dashboard/services/rgw_client.py @@ -1348,8 +1348,7 @@ class RgwMultisiteAutomation: class RgwMultisite: def migrate_to_multisite(self, realm_name: str, zonegroup_name: str, zone_name: str, - zonegroup_endpoints: str, zone_endpoints: str, access_key: str, - secret_key: str): + zonegroup_endpoints: str, zone_endpoints: str, username: str): rgw_realm_create_cmd = ['realm', 'create', '--rgw-realm', realm_name, '--default'] try: exit_code, _, err = mgr.send_rgwadmin_command(rgw_realm_create_cmd, False) @@ -1411,18 +1410,34 @@ class RgwMultisite: http_status_code=500, component='rgw') except SubprocessError as error: raise DashboardException(error, http_status_code=500, component='rgw') + self.update_period() - if access_key and secret_key: - rgw_zone_modify_cmd = ['zone', 'modify', '--rgw-zone', zone_name, - '--access-key', access_key, '--secret', secret_key] - try: - exit_code, _, err = mgr.send_rgwadmin_command(rgw_zone_modify_cmd) - if exit_code > 0: - raise DashboardException(e=err, msg='Unable to modify zone', + try: + user_details = self.create_system_user(username, zone_name) + if user_details: + keys = user_details['keys'][0] + access_key = keys['access_key'] + secret_key = keys['secret_key'] + if access_key and secret_key: + self.modify_zone(zone_name=zone_name, + zonegroup_name=zonegroup_name, + default='true', master='true', + endpoints=zone_endpoints, + access_key=keys['access_key'], + secret_key=keys['secret_key']) + else: + raise DashboardException(msg='Access key or secret key is missing', http_status_code=500, component='rgw') - except SubprocessError as error: - raise DashboardException(error, http_status_code=500, component='rgw') - self.update_period() + except Exception as e: + raise DashboardException(msg='Failed to modify zone or create system user: %s' % e, + http_status_code=500, component='rgw') + + try: + rgw_service_manager = RgwServiceManager() + rgw_service_manager.restart_rgw_daemons_and_set_credentials() + except Exception as e: + raise DashboardException(msg='Failed to restart RGW daemon: %s' % e, + http_status_code=500, component='rgw') def create_realm(self, realm_name: str, default: bool): rgw_realm_create_cmd = ['realm', 'create'] diff --git a/src/pybind/mgr/dashboard/services/rgw_iam.py b/src/pybind/mgr/dashboard/services/rgw_iam.py index dbf00df25e0..5f490323441 100644 --- a/src/pybind/mgr/dashboard/services/rgw_iam.py +++ b/src/pybind/mgr/dashboard/services/rgw_iam.py @@ -1,12 +1,13 @@ from subprocess import SubprocessError -from typing import List +from typing import List, Optional from .. import mgr from ..exceptions import DashboardException class RgwAccounts: - def send_rgw_cmd(self, command: List[str]): + @classmethod + def send_rgw_cmd(cls, command: List[str]): try: exit_code, out, err = mgr.send_rgwadmin_command(command) @@ -19,6 +20,78 @@ class RgwAccounts: except SubprocessError as e: raise DashboardException(e, component='rgw') - def get_accounts(self): + @classmethod + def get_accounts(cls, detailed: bool = False): + """ + Query account Id's, optionally returning full details. + + :param detailed: Boolean to indicate if full account details are required. + """ get_accounts_cmd = ['account', 'list'] - return self.send_rgw_cmd(get_accounts_cmd) + account_list = cls.send_rgw_cmd(get_accounts_cmd) + detailed_account_list = [] + if detailed: + for account in account_list: + detailed_account_list.append(cls.get_account(account)) + return detailed_account_list + return account_list + + @classmethod + def get_account(cls, account_id: str): + get_account_cmd = ['account', 'get', '--account-id', account_id] + return cls.send_rgw_cmd(get_account_cmd) + + @classmethod + def create_account(cls, account_name: Optional[str] = None, + account_id: Optional[str] = None, email: Optional[str] = None): + create_accounts_cmd = ['account', 'create'] + + if account_name: + create_accounts_cmd += ['--account-name', account_name] + + if account_id: + create_accounts_cmd += ['--account_id', account_id] + + if email: + create_accounts_cmd += ['--email', email] + + return cls.send_rgw_cmd(create_accounts_cmd) + + @classmethod + def modify_account(cls, account_id: str, account_name: Optional[str] = None, + email: Optional[str] = None): + modify_accounts_cmd = ['account', 'modify', '--account-id', account_id] + + if account_name: + modify_accounts_cmd += ['--account-name', account_name] + + if email: + modify_accounts_cmd += ['--email', email] + + return cls.send_rgw_cmd(modify_accounts_cmd) + + @classmethod + def delete_account(cls, account_id: str): + modify_accounts_cmd = ['account', 'rm', '--account-id', account_id] + + return cls.send_rgw_cmd(modify_accounts_cmd) + + @classmethod + def get_account_stats(cls, account_id: str): + account_stats_cmd = ['account', 'stats', '--account-id', account_id] + + return cls.send_rgw_cmd(account_stats_cmd) + + @classmethod + def set_quota(cls, quota_type: str, account_id: str, max_size: str, max_objects: str): + set_quota_cmd = ['quota', 'set', '--quota-scope', quota_type, '--account-id', account_id, + '--max-size', max_size, '--max-objects', max_objects] + + return cls.send_rgw_cmd(set_quota_cmd) + + @classmethod + def set_quota_status(cls, quota_type: str, account_id: str, quota_status: str): + set_quota_status_cmd = ['quota', quota_status, '--quota-scope', quota_type, + '--account-id', account_id] + + return cls.send_rgw_cmd(set_quota_status_cmd) diff --git a/src/pybind/mgr/dashboard/tests/test_rgw_iam.py b/src/pybind/mgr/dashboard/tests/test_rgw_iam.py new file mode 100644 index 00000000000..133b5a0d390 --- /dev/null +++ b/src/pybind/mgr/dashboard/tests/test_rgw_iam.py @@ -0,0 +1,292 @@ +from unittest import TestCase +from unittest.mock import patch + +from ..controllers.rgw_iam import RgwUserAccountsController +from ..services.rgw_iam import RgwAccounts + + +class TestRgwUserAccountsController(TestCase): + + @patch.object(RgwAccounts, 'create_account') + def test_create_account(self, mock_create_account): + mockReturnVal = { + "id": "RGW18661471562806836", + "tenant": "", + "name": "", + "email": "", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + # Mock the return value of the create_account method + mock_create_account.return_value = mockReturnVal + + controller = RgwUserAccountsController() + result = controller.create(account_name='test_account', account_id='RGW18661471562806836', + email='test@example.com') + + # Check if the account creation method was called with the correct parameters + mock_create_account.assert_called_with('test_account', 'RGW18661471562806836', + 'test@example.com') + # Check the returned result + self.assertEqual(result, mockReturnVal) + + @patch.object(RgwAccounts, 'get_accounts') + def test_list_accounts(self, mock_get_accounts): + mock_return_value = [ + "RGW22222222222222222", + "RGW59378973811515857", + "RGW11111111111111111" + ] + + mock_get_accounts.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.list(detailed=False) + + mock_get_accounts.assert_called_with(False) + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'get_accounts') + def test_list_accounts_with_details(self, mock_get_accounts): + mock_return_value = [ + { + "id": "RGW22222222222222222", + "tenant": "", + "name": "Account2", + "email": "account2@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + }, + { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + ] + + mock_get_accounts.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.list(detailed=True) + + mock_get_accounts.assert_called_with(True) + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'get_account') + def test_get_account(self, mock_get_account): + mock_return_value = { + "id": "RGW22222222222222222", + "tenant": "", + "name": "Account2", + "email": "account2@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + mock_get_account.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.get(account_id='RGW22222222222222222') + + mock_get_account.assert_called_with('RGW22222222222222222') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'delete_account') + def test_delete_account(self, mock_delete_account): + mock_delete_account.return_value = None + + controller = RgwUserAccountsController() + result = controller.delete(account_id='RGW59378973811515857') + + mock_delete_account.assert_called_with('RGW59378973811515857') + + self.assertEqual(result, None) + + @patch.object(RgwAccounts, 'modify_account') + def test_set_account_name(self, mock_modify_account): + mock_return_value = mock_return_value = { + "id": "RGW59378973811515857", + "tenant": "", + "name": "new_account_name", + "email": "new_email@example.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + mock_modify_account.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set(account_id='RGW59378973811515857', account_name='new_account_name', + email='new_email@example.com') + + mock_modify_account.assert_called_with('RGW59378973811515857', 'new_account_name', + 'new_email@example.com') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'set_quota') + def test_set_quota(self, mock_set_quota): + mock_return_value = { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": 10737418240, + "max_size_kb": 10485760, + "max_objects": 1000000 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": 1000000 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + mock_set_quota.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set_quota(quota_type='account', account_id='RGW11111111111111111', + max_size='10GB', max_objects='1000') + + mock_set_quota.assert_called_with('account', 'RGW11111111111111111', '10GB', '1000') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'set_quota_status') + def test_set_quota_status(self, mock_set_quota_status): + mock_return_value = { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": True, + "check_on_raw": False, + "max_size": 10737418240, + "max_size_kb": 10485760, + "max_objects": 1000000 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": 1000000 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + mock_set_quota_status.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set_quota_status(quota_type='account', + account_id='RGW11111111111111111', + quota_status='enabled') + + mock_set_quota_status.assert_called_with('account', 'RGW11111111111111111', 'enabled') + + self.assertEqual(result, mock_return_value) diff --git a/src/pybind/mgr/orchestrator/_interface.py b/src/pybind/mgr/orchestrator/_interface.py index a505801eea5..4fbc975ae9f 100644 --- a/src/pybind/mgr/orchestrator/_interface.py +++ b/src/pybind/mgr/orchestrator/_interface.py @@ -747,6 +747,10 @@ class Orchestrator(object): """ raise NotImplementedError() + def set_osd_spec(self, service_name: str, osd_ids: List[str]) -> OrchResult: + """ set service of osd """ + raise NotImplementedError() + def blink_device_light(self, ident_fault: str, on: bool, locations: List['DeviceLightLoc']) -> OrchResult[List[str]]: """ Instructs the orchestrator to enable or disable either the ident or the fault LED. diff --git a/src/pybind/mgr/orchestrator/module.py b/src/pybind/mgr/orchestrator/module.py index 332bc75d862..d5a1bb3da2b 100644 --- a/src/pybind/mgr/orchestrator/module.py +++ b/src/pybind/mgr/orchestrator/module.py @@ -1472,6 +1472,14 @@ Usage: return HandleCommandResult(stdout=out) + @_cli_write_command('orch osd set-spec-affinity') + def _osd_set_spec(self, service_name: str, osd_id: List[str]) -> HandleCommandResult: + """Set service spec affinity for osd""" + completion = self.set_osd_spec(service_name, osd_id) + res = raise_if_exception(completion) + + return HandleCommandResult(stdout=res) + @_cli_write_command('orch daemon add') def daemon_add_misc(self, daemon_type: Optional[ServiceType] = None, @@ -1666,7 +1674,13 @@ Usage: specs: List[Union[ServiceSpec, HostSpec]] = [] # YAML '---' document separator with no content generates # None entries in the output. Let's skip them silently. - content = [o for o in yaml_objs if o is not None] + try: + content = [o for o in yaml_objs if o is not None] + except yaml.scanner.ScannerError as e: + msg = f"Invalid YAML received : {str(e)}" + self.log.exception(msg) + return HandleCommandResult(-errno.EINVAL, stderr=msg) + for s in content: try: spec = json_to_generic_spec(s) @@ -2191,7 +2205,13 @@ Usage: specs: List[TunedProfileSpec] = [] # YAML '---' document separator with no content generates # None entries in the output. Let's skip them silently. - content = [o for o in yaml_objs if o is not None] + try: + content = [o for o in yaml_objs if o is not None] + except yaml.scanner.ScannerError as e: + msg = f"Invalid YAML received : {str(e)}" + self.log.exception(msg) + return HandleCommandResult(-errno.EINVAL, stderr=msg) + for spec in content: specs.append(TunedProfileSpec.from_json(spec)) else: diff --git a/src/pybind/mgr/snap_schedule/fs/schedule_client.py b/src/pybind/mgr/snap_schedule/fs/schedule_client.py index b58f20f1275..12e5e980737 100644 --- a/src/pybind/mgr/snap_schedule/fs/schedule_client.py +++ b/src/pybind/mgr/snap_schedule/fs/schedule_client.py @@ -163,6 +163,7 @@ class SnapSchedClient(CephfsClient): self.sqlite_connections: Dict[str, DBInfo] = {} self.active_timers: Dict[Tuple[str, str], List[Timer]] = {} self.conn_lock: Lock = Lock() # lock to protect add/lookup db connections + self.timers_lock: Lock = Lock() # restart old schedules for fs_name in self.get_all_filesystems(): @@ -273,6 +274,27 @@ class SnapSchedClient(CephfsClient): if self._is_allowed_repeat(r, path)][0:1] return rows + def delete_references_to_unavailable_fs(self, available_fs_names: Set[str]) -> None: + fs_to_remove: Set[str] = set() + self.timers_lock.acquire() + for fs, path in list(self.active_timers.keys()): # each key is a tuple + if fs not in available_fs_names: + fs_to_remove.add(fs) + log.debug(f'Cancelled timers for "{fs}:{path}"') + for t in self.active_timers[(fs, path)]: + t.cancel() + log.debug(f'Removed timer instance for "{fs}"') + del self.active_timers[(fs, path)] + self.timers_lock.release() + + self.conn_lock.acquire() + for fs in fs_to_remove: + log.debug(f'Closed DB connection to "{fs}"') + self.sqlite_connections[fs].db.close() + log.debug(f'Removed DB connection to "{fs}"') + del self.sqlite_connections[fs] + self.conn_lock.release() + def refresh_snap_timers(self, fs: str, path: str, olddb: Optional[sqlite3.Connection] = None) -> None: try: log.debug((f'SnapDB on {fs} changed for {path}, ' @@ -286,6 +308,7 @@ class SnapSchedClient(CephfsClient): with self.get_schedule_db(fs) as conn_mgr: db = conn_mgr.dbinfo.db rows = self.fetch_schedules(db, path) + self.timers_lock.acquire() timers = self.active_timers.get((fs, path), []) for timer in timers: timer.cancel() @@ -299,6 +322,7 @@ class SnapSchedClient(CephfsClient): timers.append(t) log.debug(f'Will snapshot {path} in fs {fs} in {row[1]}s') self.active_timers[(fs, path)] = timers + self.timers_lock.release() except Exception: self._log_exception('refresh_snap_timers') diff --git a/src/pybind/mgr/snap_schedule/module.py b/src/pybind/mgr/snap_schedule/module.py index d8f04a62b94..adf982448b1 100644 --- a/src/pybind/mgr/snap_schedule/module.py +++ b/src/pybind/mgr/snap_schedule/module.py @@ -8,12 +8,14 @@ import json import sqlite3 from typing import Any, Dict, Optional, Tuple, Union from .fs.schedule_client import SnapSchedClient -from mgr_module import MgrModule, CLIReadCommand, CLIWriteCommand, Option +from mgr_module import MgrModule, CLIReadCommand, CLIWriteCommand, Option, NotifyType from mgr_util import CephfsConnectionException from threading import Event class Module(MgrModule): + NOTIFY_TYPES = [NotifyType.fs_map] + MODULE_OPTIONS = [ Option( 'allow_m_granularity', @@ -37,6 +39,21 @@ class Module(MgrModule): self._initialized = Event() self.client = SnapSchedClient(self) + def notify(self, notify_type: NotifyType, notify_id: str) -> None: + if notify_type != NotifyType.fs_map: + return + fs_map = self.get('fs_map') + if not fs_map: + return + + # we don't know for which fs config has been changed + fs_names = set() + for fs in fs_map['filesystems']: + fs_name = fs['mdsmap']['fs_name'] + fs_names.add(fs_name) + + self.client.delete_references_to_unavailable_fs(fs_names) + def _subvolume_exist(self, fs: str, subvol: Union[str, None], group: Union[str, None]) -> bool: rc, subvolumes, err = self.remote('volumes', 'subvolume_ls', fs, group) if rc == 0: diff --git a/src/pybind/mgr/volumes/fs/operations/volume.py b/src/pybind/mgr/volumes/fs/operations/volume.py index b2574fd76d5..93844bce119 100644 --- a/src/pybind/mgr/volumes/fs/operations/volume.py +++ b/src/pybind/mgr/volumes/fs/operations/volume.py @@ -133,7 +133,11 @@ def delete_volume(mgr, volname, metadata_pool, data_pools): r, outb, outs = remove_pool(mgr, data_pool) if r != 0: return r, outb, outs - result_str = "metadata pool: {0} data pool: {1} removed".format(metadata_pool, str(data_pools)) + result_str = f"metadata pool: {metadata_pool} data pool: {str(data_pools)} removed.\n" + result_str += "If there are active snapshot schedules associated with this " + result_str += "volume, you might see EIO errors in the mgr logs or at the " + result_str += "snap-schedule command-line due to the missing volume. " + result_str += "However, these errors are transient and will get auto-resolved." return r, result_str, "" def rename_volume(mgr, volname: str, newvolname: str) -> Tuple[int, str, str]: diff --git a/src/python-common/ceph/deployment/service_spec.py b/src/python-common/ceph/deployment/service_spec.py index ebe66989cc0..1ac9fa49e32 100644 --- a/src/python-common/ceph/deployment/service_spec.py +++ b/src/python-common/ceph/deployment/service_spec.py @@ -25,7 +25,9 @@ from typing import ( import yaml from ceph.deployment.hostspec import HostSpec, SpecValidationError, assert_valid_host -from ceph.deployment.utils import unwrap_ipv6, valid_addr +from ceph.deployment.utils import unwrap_ipv6, valid_addr, verify_non_negative_int +from ceph.deployment.utils import verify_positive_int, verify_non_negative_number +from ceph.deployment.utils import verify_boolean, verify_enum from ceph.utils import is_hex ServiceSpecT = TypeVar('ServiceSpecT', bound='ServiceSpec') @@ -1229,6 +1231,7 @@ class RGWSpec(ServiceSpec): rgw_bucket_counters_cache: Optional[bool] = False, rgw_bucket_counters_cache_size: Optional[int] = None, generate_cert: bool = False, + disable_multisite_sync_traffic: Optional[bool] = None, ): assert service_type == 'rgw', service_type @@ -1281,6 +1284,8 @@ class RGWSpec(ServiceSpec): self.rgw_bucket_counters_cache_size = rgw_bucket_counters_cache_size #: Whether we should generate a cert/key for the user if not provided self.generate_cert = generate_cert + #: Used to make RGW not do multisite replication so it can dedicate to IO + self.disable_multisite_sync_traffic = disable_multisite_sync_traffic def get_port_start(self) -> List[int]: return [self.get_port()] @@ -1328,6 +1333,7 @@ class NvmeofServiceSpec(ServiceSpec): name: Optional[str] = None, group: Optional[str] = None, addr: Optional[str] = None, + addr_map: Optional[Dict[str, str]] = None, port: Optional[int] = None, pool: Optional[str] = None, enable_auth: bool = False, @@ -1336,13 +1342,19 @@ class NvmeofServiceSpec(ServiceSpec): enable_spdk_discovery_controller: Optional[bool] = False, enable_key_encryption: Optional[bool] = True, encryption_key: Optional[str] = None, + rebalance_period_sec: Optional[int] = 7, + max_gws_in_grp: Optional[int] = 16, + max_ns_to_change_lb_grp: Optional[int] = 8, omap_file_lock_duration: Optional[int] = 20, omap_file_lock_retries: Optional[int] = 30, omap_file_lock_retry_sleep_interval: Optional[float] = 1.0, omap_file_update_reloads: Optional[int] = 10, enable_prometheus_exporter: Optional[bool] = True, + prometheus_port: Optional[int] = 10008, + prometheus_stats_interval: Optional[int] = 10, bdevs_per_cluster: Optional[int] = 32, verify_nqns: Optional[bool] = True, + verify_keys: Optional[bool] = True, allowed_consecutive_spdk_ping_failures: Optional[int] = 1, spdk_ping_interval_in_seconds: Optional[float] = 2.0, ping_spdk_under_lock: Optional[bool] = False, @@ -1373,6 +1385,7 @@ class NvmeofServiceSpec(ServiceSpec): {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, tgt_cmd_extra_args: Optional[str] = None, discovery_addr: Optional[str] = None, + discovery_addr_map: Optional[Dict[str, str]] = None, discovery_port: Optional[int] = None, log_level: Optional[str] = 'INFO', log_files_enabled: Optional[bool] = True, @@ -1407,6 +1420,8 @@ class NvmeofServiceSpec(ServiceSpec): self.pool = pool #: ``addr`` address of the nvmeof gateway self.addr = addr + #: ``addr_map`` per node address map of the nvmeof gateways + self.addr_map = addr_map #: ``port`` port of the nvmeof gateway self.port = port or 5500 #: ``name`` name of the nvmeof gateway @@ -1425,10 +1440,22 @@ class NvmeofServiceSpec(ServiceSpec): self.enable_key_encryption = enable_key_encryption #: ``encryption_key`` gateway encryption key self.encryption_key = encryption_key + #: ``rebalance_period_sec`` number of seconds between cycles of auto namesapce rebalancing + self.rebalance_period_sec = rebalance_period_sec + #: ``max_gws_in_grp`` max number of gateways in one group + self.max_gws_in_grp = max_gws_in_grp + #: ``max_ns_to_change_lb_grp`` max number of namespaces before switching to a new lb group + self.max_ns_to_change_lb_grp = max_ns_to_change_lb_grp #: ``enable_prometheus_exporter`` enables Prometheus exporter self.enable_prometheus_exporter = enable_prometheus_exporter + #: ``prometheus_port`` Prometheus port + self.prometheus_port = prometheus_port or 10008 + #: ``prometheus_stats_interval`` Prometheus get stats interval + self.prometheus_stats_interval = prometheus_stats_interval #: ``verify_nqns`` enables verification of subsystem and host NQNs for validity self.verify_nqns = verify_nqns + #: ``verify_keys`` enables verification of PSJ and DHCHAP keys in the gateway + self.verify_keys = verify_keys #: ``omap_file_lock_duration`` number of seconds before automatically unlock OMAP file lock self.omap_file_lock_duration = omap_file_lock_duration #: ``omap_file_lock_retries`` number of retries to lock OMAP file before giving up @@ -1495,6 +1522,8 @@ class NvmeofServiceSpec(ServiceSpec): self.tgt_cmd_extra_args = tgt_cmd_extra_args #: ``discovery_addr`` address of the discovery service self.discovery_addr = discovery_addr + #: ``discovery_addr_map`` per node address map of the discovery service + self.discovery_addr_map = discovery_addr_map #: ``discovery_port`` port of the discovery service self.discovery_port = discovery_port or 8009 #: ``log_level`` the nvmeof gateway log level @@ -1521,7 +1550,7 @@ class NvmeofServiceSpec(ServiceSpec): self.monitor_client_log_file_dir = monitor_client_log_file_dir def get_port_start(self) -> List[int]: - return [5500, 4420, 8009] + return [self.port, 4420, self.discovery_port] def validate(self) -> None: # TODO: what other parameters should be validated as part of this function? @@ -1530,6 +1559,7 @@ class NvmeofServiceSpec(ServiceSpec): if not self.pool: raise SpecValidationError('Cannot add NVMEOF: No Pool specified') + verify_boolean(self.enable_auth, "Enable authentication") if self.enable_auth: if not all([self.server_key, self.server_cert, self.client_key, self.client_cert, self.root_ca_cert]): @@ -1544,142 +1574,65 @@ class NvmeofServiceSpec(ServiceSpec): if self.transports not in ['tcp']: raise SpecValidationError('Invalid transport. Valid values are tcp') - if self.log_level: - if self.log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'critical']: - raise SpecValidationError( - 'Invalid log level. Valid values are: debug, info, warning, error, critial') - - if self.spdk_log_level: - if self.spdk_log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'notice']: - raise SpecValidationError( - 'Invalid SPDK log level. Valid values are: ' - 'DEBUG, INFO, WARNING, ERROR, NOTICE') - - if self.spdk_protocol_log_level: - if self.spdk_protocol_log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'notice']: - raise SpecValidationError( - 'Invalid SPDK protocol log level. Valid values are: ' - 'DEBUG, INFO, WARNING, ERROR, NOTICE') + verify_enum(self.log_level, "log level", ['debug', 'info', 'warning', 'error', 'critical']) + verify_enum(self.spdk_log_level, "SPDK log level", + ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'NOTICE']) + verify_enum(self.spdk_protocol_log_level, "SPDK protocol log level", + ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'NOTICE']) + verify_positive_int(self.bdevs_per_cluster, "Bdevs per cluster") + if self.bdevs_per_cluster is not None and self.bdevs_per_cluster < 1: + raise SpecValidationError("Bdevs per cluster should be at least 1") + verify_non_negative_number(self.spdk_ping_interval_in_seconds, "SPDK ping interval") if ( - self.spdk_ping_interval_in_seconds + self.spdk_ping_interval_in_seconds is not None and self.spdk_ping_interval_in_seconds < 1.0 ): raise SpecValidationError("SPDK ping interval should be at least 1 second") + verify_non_negative_int(self.allowed_consecutive_spdk_ping_failures, + "Allowed consecutive SPDK ping failures") if ( - self.allowed_consecutive_spdk_ping_failures + self.allowed_consecutive_spdk_ping_failures is not None and self.allowed_consecutive_spdk_ping_failures < 1 ): raise SpecValidationError("Allowed consecutive SPDK ping failures should be at least 1") - if ( - self.state_update_interval_sec - and self.state_update_interval_sec < 0 - ): - raise SpecValidationError("State update interval can't be negative") - - if ( - self.omap_file_lock_duration - and self.omap_file_lock_duration < 0 - ): - raise SpecValidationError("OMAP file lock duration can't be negative") - - if ( - self.omap_file_lock_retries - and self.omap_file_lock_retries < 0 - ): - raise SpecValidationError("OMAP file lock retries can't be negative") - - if ( - self.omap_file_update_reloads - and self.omap_file_update_reloads < 0 - ): - raise SpecValidationError("OMAP file reloads can't be negative") - - if ( - self.spdk_timeout - and self.spdk_timeout < 0.0 - ): - raise SpecValidationError("SPDK timeout can't be negative") - - if ( - self.conn_retries - and self.conn_retries < 0 - ): - raise SpecValidationError("Connection retries can't be negative") - - if ( - self.max_log_file_size_in_mb - and self.max_log_file_size_in_mb < 0 - ): - raise SpecValidationError("Log file size can't be negative") - - if ( - self.max_log_files_count - and self.max_log_files_count < 0 - ): - raise SpecValidationError("Log files count can't be negative") - - if ( - self.max_log_directory_backups - and self.max_log_directory_backups < 0 - ): - raise SpecValidationError("Log file directory backups can't be negative") - - if (self.max_hosts_per_namespace and self.max_hosts_per_namespace < 0): - raise SpecValidationError("Max hosts per namespace can't be negative") - - if (self.max_namespaces_with_netmask and self.max_namespaces_with_netmask < 0): - raise SpecValidationError("Max namespaces with netmask can't be negative") - - if not isinstance(self.max_subsystems, int): - raise SpecValidationError("Max subsystems must be an integer") - - if self.max_subsystems <= 0: - raise SpecValidationError("Max subsystems must be greater than zero") - - if not isinstance(self.max_namespaces, int): - raise SpecValidationError("Max namespaces must be an integer") - - if self.max_namespaces <= 0: - raise SpecValidationError("Max namespaces must be greater than zero") - - if not isinstance(self.max_namespaces_per_subsystem, int): - raise SpecValidationError("Max namespaces per subsystem must be an integer") - - if self.max_namespaces_per_subsystem <= 0: - raise SpecValidationError("Max namespaces per subsystem must be greater than zero") - - if not isinstance(self.max_hosts_per_subsystem, int): - raise SpecValidationError("Max hosts per subsystem must be an integer") - - if self.max_hosts_per_subsystem <= 0: - raise SpecValidationError("Max hosts per subsystem must be greater than zero") - - if ( - self.monitor_timeout - and self.monitor_timeout < 0.0 - ): - raise SpecValidationError("Monitor timeout can't be negative") - - if self.port and self.port < 0: - raise SpecValidationError("Port can't be negative") - - if self.discovery_port and self.discovery_port < 0: - raise SpecValidationError("Discovery port can't be negative") + verify_non_negative_int(self.state_update_interval_sec, "State update interval") + verify_non_negative_int(self.rebalance_period_sec, "Rebalance period") + verify_non_negative_int(self.max_gws_in_grp, "Max gateways in group") + verify_non_negative_int(self.max_ns_to_change_lb_grp, + "Max namespaces to change load balancing group") + verify_non_negative_int(self.omap_file_lock_duration, "OMAP file lock duration") + verify_non_negative_number(self.omap_file_lock_retry_sleep_interval, + "OMAP file lock sleep interval") + verify_non_negative_int(self.omap_file_lock_retries, "OMAP file lock retries") + verify_non_negative_int(self.omap_file_update_reloads, "OMAP file reloads") + verify_non_negative_number(self.spdk_timeout, "SPDK timeout") + verify_non_negative_int(self.max_log_file_size_in_mb, "Log file size") + verify_non_negative_int(self.max_log_files_count, "Log files count") + verify_non_negative_int(self.max_log_directory_backups, "Log file directory backups") + verify_non_negative_int(self.max_hosts_per_namespace, "Max hosts per namespace") + verify_non_negative_int(self.max_namespaces_with_netmask, "Max namespaces with netmask") + verify_positive_int(self.max_subsystems, "Max subsystems") + verify_positive_int(self.max_namespaces, "Max namespaces") + verify_positive_int(self.max_namespaces_per_subsystem, "Max namespaces per subsystem") + verify_positive_int(self.max_hosts_per_subsystem, "Max hosts per subsystem") + verify_non_negative_number(self.monitor_timeout, "Monitor timeout") + verify_non_negative_int(self.port, "Port") + verify_non_negative_int(self.discovery_port, "Discovery port") + verify_non_negative_int(self.prometheus_port, "Prometheus port") + verify_non_negative_int(self.prometheus_stats_interval, "Prometheus stats interval") + verify_boolean(self.state_update_notify, "State update notify") + verify_boolean(self.enable_spdk_discovery_controller, "Enable SPDK discovery controller") + verify_boolean(self.enable_key_encryption, "Enable key encryption") + verify_boolean(self.enable_prometheus_exporter, "Enable Prometheus exporter") + verify_boolean(self.verify_nqns, "Verify NQNs") + verify_boolean(self.verify_keys, "Verify Keys") + verify_boolean(self.log_files_enabled, "Log files enabled") + verify_boolean(self.log_files_rotation_enabled, "Log files rotation enabled") + verify_boolean(self.verbose_log_messages, "Verbose log messages") + verify_boolean(self.enable_monitor_client, "Enable monitor client") yaml.add_representer(NvmeofServiceSpec, ServiceSpec.yaml_representer) @@ -2378,6 +2331,7 @@ class AlertManagerSpec(MonitoringSpec): user_data: Optional[Dict[str, Any]] = None, config: Optional[Dict[str, str]] = None, networks: Optional[List[str]] = None, + only_bind_port_on_networks: bool = False, port: Optional[int] = None, secure: bool = False, extra_container_args: Optional[GeneralArgList] = None, @@ -2408,6 +2362,7 @@ class AlertManagerSpec(MonitoringSpec): # <webhook_configs> configuration. self.user_data = user_data or {} self.secure = secure + self.only_bind_port_on_networks = only_bind_port_on_networks def get_port_start(self) -> List[int]: return [self.get_port(), 9094] @@ -2454,7 +2409,7 @@ class GrafanaSpec(MonitoringSpec): self.protocol = protocol # whether ports daemons for this service bind to should - # bind to only hte networks listed in networks param, or + # bind to only the networks listed in networks param, or # to all networks. Defaults to false which is saying to bind # on all networks. self.only_bind_port_on_networks = only_bind_port_on_networks diff --git a/src/python-common/ceph/deployment/utils.py b/src/python-common/ceph/deployment/utils.py index f800e373897..758eddc9412 100644 --- a/src/python-common/ceph/deployment/utils.py +++ b/src/python-common/ceph/deployment/utils.py @@ -1,7 +1,9 @@ import ipaddress import socket -from typing import Tuple, Optional +from typing import Tuple, Optional, Any from urllib.parse import urlparse +from ceph.deployment.hostspec import SpecValidationError +from numbers import Number def unwrap_ipv6(address): @@ -100,3 +102,50 @@ def valid_addr(addr: str) -> Tuple[bool, str]: if addr[0].isalpha() and '.' in addr: return _dns_lookup(addr, port) return _ip_lookup(addr, port) + + +def verify_numeric(field: Any, field_name: str) -> None: + if field is not None: + if not isinstance(field, Number) or isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be a number") + + +def verify_non_negative_int(field: Any, field_name: str) -> None: + verify_numeric(field, field_name) + if field is not None: + if not isinstance(field, int) or isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be an integer") + if field < 0: + raise SpecValidationError(f"{field_name} can't be negative") + + +def verify_positive_int(field: Any, field_name: str) -> None: + verify_non_negative_int(field, field_name) + if field is not None and field <= 0: + raise SpecValidationError(f"{field_name} must be greater than zero") + + +def verify_non_negative_number(field: Any, field_name: str) -> None: + verify_numeric(field, field_name) + if field is not None: + if field < 0.0: + raise SpecValidationError(f"{field_name} can't be negative") + + +def verify_boolean(field: Any, field_name: str) -> None: + if field is not None: + if not isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be a boolean") + + +def verify_enum(field: Any, field_name: str, allowed: list) -> None: + if field: + allowed_lower = [] + if not isinstance(field, str): + raise SpecValidationError(f"{field_name} must be a string") + for val in allowed: + assert isinstance(val, str) + allowed_lower.append(val.lower()) + if field.lower() not in allowed_lower: + raise SpecValidationError( + f'Invalid {field_name}. Valid values are: {", ".join(allowed)}') diff --git a/src/rgw/CMakeLists.txt b/src/rgw/CMakeLists.txt index 96f3237b896..41e473e23f0 100644 --- a/src/rgw/CMakeLists.txt +++ b/src/rgw/CMakeLists.txt @@ -90,6 +90,7 @@ set(librgw_common_srcs rgw_notify_event_type.cc rgw_period_history.cc rgw_period_puller.cc + rgw_s3_filter.cc rgw_pubsub.cc rgw_coroutine.cc rgw_cr_rest.cc @@ -486,9 +487,9 @@ target_link_libraries(radosgw PRIVATE install(TARGETS radosgw DESTINATION bin) set(radosgw_admin_srcs - rgw_admin.cc - rgw_sync_checkpoint.cc - rgw_orphan.cc) + radosgw-admin/radosgw-admin.cc + radosgw-admin/sync_checkpoint.cc + radosgw-admin/orphan.cc) # this is unsatisfying and hopefully temporary; ARROW should not be # part of radosgw_admin diff --git a/src/rgw/driver/daos/rgw_sal_daos.cc b/src/rgw/driver/daos/rgw_sal_daos.cc index a87d88c4b85..92dd7afe2fb 100644 --- a/src/rgw/driver/daos/rgw_sal_daos.cc +++ b/src/rgw/driver/daos/rgw_sal_daos.cc @@ -858,8 +858,6 @@ bool DaosZone::is_writeable() { return true; } bool DaosZone::get_redirect_endpoint(std::string* endpoint) { return false; } -bool DaosZone::has_zonegroup_api(const std::string& api) const { return false; } - const std::string& DaosZone::get_current_period_id() { return current_period->get_id(); } diff --git a/src/rgw/driver/daos/rgw_sal_daos.h b/src/rgw/driver/daos/rgw_sal_daos.h index e382fdb04ae..5515579a441 100644 --- a/src/rgw/driver/daos/rgw_sal_daos.h +++ b/src/rgw/driver/daos/rgw_sal_daos.h @@ -484,7 +484,6 @@ class DaosZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() { return zone_params->system_key; diff --git a/src/rgw/driver/motr/rgw_sal_motr.cc b/src/rgw/driver/motr/rgw_sal_motr.cc index b999673ac18..463ea8c5b11 100644 --- a/src/rgw/driver/motr/rgw_sal_motr.cc +++ b/src/rgw/driver/motr/rgw_sal_motr.cc @@ -1111,11 +1111,6 @@ bool MotrZone::get_redirect_endpoint(std::string* endpoint) return false; } -bool MotrZone::has_zonegroup_api(const std::string& api) const -{ - return (zonegroup.group.api_name == api); -} - const std::string& MotrZone::get_current_period_id() { return current_period->get_id(); diff --git a/src/rgw/driver/motr/rgw_sal_motr.h b/src/rgw/driver/motr/rgw_sal_motr.h index f92074b9d94..0f99ae48e86 100644 --- a/src/rgw/driver/motr/rgw_sal_motr.h +++ b/src/rgw/driver/motr/rgw_sal_motr.h @@ -525,7 +525,6 @@ class MotrZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() { return zone_params->system_key; } virtual const std::string& get_realm_name() { return realm->get_name(); } diff --git a/src/rgw/driver/posix/notify.h b/src/rgw/driver/posix/notify.h index 9f6088a893a..4463abc57c2 100644 --- a/src/rgw/driver/posix/notify.h +++ b/src/rgw/driver/posix/notify.h @@ -212,7 +212,7 @@ namespace file::listing { void signal_shutdown() { uint64_t msg{sig_shutdown}; - (void) write(efd, &msg, sizeof(uint64_t)); + std::ignore = write(efd, &msg, sizeof(uint64_t)); } friend class Notify; diff --git a/src/rgw/driver/posix/rgw_sal_posix.cc b/src/rgw/driver/posix/rgw_sal_posix.cc index 1345468210f..9d76462baa0 100644 --- a/src/rgw/driver/posix/rgw_sal_posix.cc +++ b/src/rgw/driver/posix/rgw_sal_posix.cc @@ -2893,6 +2893,14 @@ int POSIXObject::copy_object(const ACLOwner& owner, return dobj->set_obj_attrs(dpp, &attrs, nullptr, y, rgw::sal::FLAG_LOG_OP); } +int POSIXObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + return -EOPNOTSUPP; +} + int POSIXObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { int ret = stat(dpp); diff --git a/src/rgw/driver/posix/rgw_sal_posix.h b/src/rgw/driver/posix/rgw_sal_posix.h index 8ec72bbc1bc..bf3478ad6ab 100644 --- a/src/rgw/driver/posix/rgw_sal_posix.h +++ b/src/rgw/driver/posix/rgw_sal_posix.h @@ -653,6 +653,13 @@ public: const DoutPrefixProvider* dpp, optional_yield y) override; virtual RGWAccessControlPolicy& get_acl(void) override { return acls; } virtual int set_acl(const RGWAccessControlPolicy& acl) override { acls = acl; return 0; } + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; diff --git a/src/rgw/driver/rados/rgw_data_sync.cc b/src/rgw/driver/rados/rgw_data_sync.cc index 792671579b7..c0a9059a251 100644 --- a/src/rgw/driver/rados/rgw_data_sync.cc +++ b/src/rgw/driver/rados/rgw_data_sync.cc @@ -2617,6 +2617,7 @@ class RGWUserPermHandler { rgw::IAM::Environment env; std::unique_ptr<rgw::auth::Identity> identity; RGWAccessControlPolicy user_acl; + std::vector<rgw::IAM::Policy> user_policies; }; std::shared_ptr<_info> info; @@ -2644,7 +2645,7 @@ class RGWUserPermHandler { } auto result = rgw::auth::transform_old_authinfo( - sync_env->dpp, null_yield, sync_env->driver, user.get()); + sync_env->dpp, null_yield, sync_env->driver, user.get(), &info->user_policies); if (!result) { return result.error(); } @@ -2679,6 +2680,7 @@ public: std::shared_ptr<_info> info; RGWAccessControlPolicy bucket_acl; std::optional<perm_state> ps; + boost::optional<rgw::IAM::Policy> bucket_policy; public: Bucket() {} @@ -2686,9 +2688,7 @@ public: const RGWBucketInfo& bucket_info, const map<string, bufferlist>& bucket_attrs); - bool verify_bucket_permission(int perm); - bool verify_object_permission(const map<string, bufferlist>& obj_attrs, - int perm); + bool verify_bucket_permission(const rgw_obj_key& obj_key, const uint64_t op); }; static int policy_from_attrs(CephContext *cct, @@ -2728,6 +2728,14 @@ int RGWUserPermHandler::Bucket::init(RGWUserPermHandler *handler, return r; } + // load bucket policy + try { + bucket_policy = get_iam_policy_from_attr(sync_env->cct, bucket_attrs, bucket_info.bucket.tenant); + } catch (const std::exception& e) { + ldpp_dout(sync_env->dpp, 0) << "ERROR: reading IAM Policy: " << e.what() << dendl; + return -EACCES; + } + ps.emplace(sync_env->cct, info->env, info->identity.get(), @@ -2740,36 +2748,40 @@ int RGWUserPermHandler::Bucket::init(RGWUserPermHandler *handler, return 0; } -bool RGWUserPermHandler::Bucket::verify_bucket_permission(int perm) -{ - return verify_bucket_permission_no_policy(sync_env->dpp, - &(*ps), - info->user_acl, - bucket_acl, - perm); -} - -bool RGWUserPermHandler::Bucket::verify_object_permission(const map<string, bufferlist>& obj_attrs, - int perm) +bool RGWUserPermHandler::Bucket::verify_bucket_permission(const rgw_obj_key& obj_key, const uint64_t op) { - RGWAccessControlPolicy obj_acl; - - int r = policy_from_attrs(sync_env->cct, obj_attrs, &obj_acl); - if (r < 0) { - return r; - } - - return verify_bucket_permission_no_policy(sync_env->dpp, - &(*ps), - bucket_acl, - obj_acl, - perm); + const rgw_obj obj(ps->bucket_info.bucket, obj_key); + const auto arn = rgw::ARN(obj); + + if (ps->identity->get_account()) { + const bool account_root = (ps->identity->get_identity_type() == TYPE_ROOT); + if (!ps->identity->is_owner_of(bucket_acl.get_owner().id)) { + ldpp_dout(sync_env->dpp, 4) << "cross-account request for bucket owner " + << bucket_acl.get_owner().id << " != " << ps->identity->get_aclowner().id << dendl; + // cross-account requests evaluate the identity-based policies separately + // from the resource-based policies and require Allow from both + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, {}, {}, {}, + info->user_policies, {}, op) + && ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, false, info->user_acl, + bucket_acl, bucket_policy, {}, {}, op); + } else { + // don't consult acls for same-account access. require an Allow from + // either identity- or resource-based policy + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, {}, {}, + bucket_policy, info->user_policies, + {}, op); + } + } + constexpr bool account_root = false; + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, + info->user_acl, bucket_acl, + bucket_policy, info->user_policies, + {}, op); } class RGWFetchObjFilter_Sync : public RGWFetchObjFilter_Default { rgw_bucket_sync_pipe sync_pipe; - std::shared_ptr<RGWUserPermHandler::Bucket> bucket_perms; std::optional<rgw_sync_pipe_dest_params> verify_dest_params; std::optional<ceph::real_time> mtime; @@ -2782,10 +2794,8 @@ class RGWFetchObjFilter_Sync : public RGWFetchObjFilter_Default { public: RGWFetchObjFilter_Sync(rgw_bucket_sync_pipe& _sync_pipe, - std::shared_ptr<RGWUserPermHandler::Bucket>& _bucket_perms, std::optional<rgw_sync_pipe_dest_params>&& _verify_dest_params, std::shared_ptr<bool>& _need_retry) : sync_pipe(_sync_pipe), - bucket_perms(_bucket_perms), verify_dest_params(std::move(_verify_dest_params)), need_retry(_need_retry) { *need_retry = false; @@ -2852,12 +2862,6 @@ int RGWFetchObjFilter_Sync::filter(CephContext *cct, *poverride_owner = acl_translation_owner; } } - if (params.mode == rgw_sync_pipe_params::MODE_USER) { - if (!bucket_perms->verify_object_permission(obj_attrs, RGW_PERM_READ)) { - ldout(cct, 0) << "ERROR: " << __func__ << ": permission check failed: user not allowed to fetch object" << dendl; - return -EPERM; - } - } if (!dest_placement_rule && params.dest.storage_class) { @@ -2900,7 +2904,6 @@ class RGWObjFetchCR : public RGWCoroutine { rgw_sync_pipe_params::Mode param_mode; std::optional<RGWUserPermHandler> user_perms; - std::shared_ptr<RGWUserPermHandler::Bucket> source_bucket_perms; RGWUserPermHandler::Bucket dest_bucket_perms; std::optional<rgw_sync_pipe_dest_params> dest_params; @@ -3016,20 +3019,10 @@ public: return set_cr_error(retcode); } - if (!dest_bucket_perms.verify_bucket_permission(RGW_PERM_WRITE)) { + if (!dest_bucket_perms.verify_bucket_permission(dest_key.value_or(key), rgw::IAM::s3PutObject)) { ldout(cct, 0) << "ERROR: " << __func__ << ": permission check failed: user not allowed to write into bucket (bucket=" << sync_pipe.info.dest_bucket.get_key() << ")" << dendl; return -EPERM; } - - /* init source bucket permission structure */ - source_bucket_perms = make_shared<RGWUserPermHandler::Bucket>(); - r = user_perms->init_bucket(sync_pipe.source_bucket_info, - sync_pipe.source_bucket_attrs, - source_bucket_perms.get()); - if (r < 0) { - ldout(cct, 20) << "ERROR: " << __func__ << ": failed to init bucket perms manager for uid=" << *param_user << " bucket=" << sync_pipe.source_bucket_info.bucket.get_key() << dendl; - return set_cr_error(retcode); - } } yield { @@ -3037,12 +3030,11 @@ public: need_retry = make_shared<bool>(); } auto filter = make_shared<RGWFetchObjFilter_Sync>(sync_pipe, - source_bucket_perms, std::move(dest_params), need_retry); call(new RGWFetchRemoteObjCR(sync_env->async_rados, sync_env->driver, sc->source_zone, - nullopt, + param_user, sync_pipe.source_bucket_info.bucket, std::nullopt, sync_pipe.dest_bucket_info, key, dest_key, versioned_epoch, @@ -4528,7 +4520,7 @@ public: } tn->set_resource_name(SSTR(bucket_str_noinstance(bs.bucket) << "/" << key)); } - if (retcode == -ERR_PRECONDITION_FAILED) { + if (retcode == -ERR_PRECONDITION_FAILED || retcode == -EPERM) { pretty_print(sc->env, "Skipping object s3://{}/{} in sync from zone {}\n", bs.bucket.name, key, zone_name); set_status("Skipping object sync: precondition failed (object contains newer change or policy doesn't allow sync)"); diff --git a/src/rgw/driver/rados/rgw_datalog.cc b/src/rgw/driver/rados/rgw_datalog.cc index 4c9503071ef..d7e57d7e1c1 100644 --- a/src/rgw/driver/rados/rgw_datalog.cc +++ b/src/rgw/driver/rados/rgw_datalog.cc @@ -576,7 +576,7 @@ int RGWDataChangesLog::renew_entries(const DoutPrefixProvider *dpp) if (ret < 0) { /* we don't really need to have a special handling for failed cases here, * as this is just an optimization. */ - ldpp_dout(dpp, -1) << "ERROR: svc.cls->timelog.add() returned " << ret << dendl; + ldpp_dout(dpp, -1) << "ERROR: be->push() returned " << ret << dendl; return ret; } diff --git a/src/rgw/driver/rados/rgw_period.cc b/src/rgw/driver/rados/rgw_period.cc index f18e8e46bc5..aacb9b6a09a 100644 --- a/src/rgw/driver/rados/rgw_period.cc +++ b/src/rgw/driver/rados/rgw_period.cc @@ -68,20 +68,6 @@ int RGWPeriod::delete_obj(const DoutPrefixProvider *dpp, optional_yield y) return ret; } -int RGWPeriod::add_zonegroup(const DoutPrefixProvider *dpp, const RGWZoneGroup& zonegroup, optional_yield y) -{ - if (zonegroup.realm_id != realm_id) { - return 0; - } - int ret = period_map.update(zonegroup, cct); - if (ret < 0) { - ldpp_dout(dpp, 0) << "ERROR: updating period map: " << cpp_strerror(-ret) << dendl; - return ret; - } - - return store_info(dpp, false, y); -} - int RGWPeriod::update(const DoutPrefixProvider *dpp, optional_yield y) { auto zone_svc = sysobj_svc->get_zone_svc(); diff --git a/src/rgw/driver/rados/rgw_pubsub_push.cc b/src/rgw/driver/rados/rgw_pubsub_push.cc index 07d65fa1028..d22c61e9b08 100644 --- a/src/rgw/driver/rados/rgw_pubsub_push.cc +++ b/src/rgw/driver/rados/rgw_pubsub_push.cc @@ -281,7 +281,7 @@ public: conn_id, _endpoint, get_bool(args, "use-ssl", false), get_bool(args, "verify-ssl", true), args.get_optional("ca-location"), args.get_optional("mechanism"), args.get_optional("user-name"), - args.get_optional("password"))) { + args.get_optional("password"), args.get_optional("kafka-brokers"))) { throw configuration_error("Kafka: failed to create connection to: " + _endpoint); } @@ -434,4 +434,3 @@ void RGWPubSubEndpoint::shutdown_all() { #endif shutdown_http_manager(); } - diff --git a/src/rgw/driver/rados/rgw_rados.cc b/src/rgw/driver/rados/rgw_rados.cc index 2ba3559c006..69075c506f1 100644 --- a/src/rgw/driver/rados/rgw_rados.cc +++ b/src/rgw/driver/rados/rgw_rados.cc @@ -5485,7 +5485,7 @@ int RGWRados::delete_bucket(RGWBucketInfo& bucket_info, RGWObjVersionTracker& ob } /* if the bucket is not synced we can remove the meta file */ - if (!svc.zone->is_syncing_bucket_meta(bucket)) { + if (!svc.zone->is_syncing_bucket_meta()) { RGWObjVersionTracker objv_tracker; r = ctl.bucket->remove_bucket_instance_info(bucket, bucket_info, y, dpp); if (r < 0) { @@ -6962,13 +6962,13 @@ int RGWRados::set_attrs(const DoutPrefixProvider *dpp, RGWObjectCtx* octx, RGWBu } return 0; -} +} /* RGWRados::set_attrs() */ -static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, - RGWRados* store, RGWBucketInfo& bucket_info, - RGWObjectCtx* rctx, RGWObjManifest* manifest, - int part_num, int* parts_count, bool prefetch, - RGWObjState** pstate, RGWObjManifest** pmanifest) +int RGWRados::get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, + RGWRados* store, RGWBucketInfo& bucket_info, + RGWObjectCtx* rctx, RGWObjManifest* manifest, + int part_num, int* parts_count, bool prefetch, + RGWObjState** pstate, RGWObjManifest** pmanifest) { if (!manifest) { return -ERR_INVALID_PART; @@ -7047,6 +7047,9 @@ static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, // update the object size sm->state.size = part_manifest.get_obj_size(); + if (!sm->state.attrset.count(RGW_ATTR_COMPRESSION)) { + sm->state.accounted_size = sm->state.size; + } *pmanifest = &part_manifest; return 0; diff --git a/src/rgw/driver/rados/rgw_rados.h b/src/rgw/driver/rados/rgw_rados.h index b24823b60dc..fe79916392f 100644 --- a/src/rgw/driver/rados/rgw_rados.h +++ b/src/rgw/driver/rados/rgw_rados.h @@ -1071,6 +1071,12 @@ public: }; // class RGWRados::Bucket::List }; // class RGWRados::Bucket + static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, + RGWRados* store, RGWBucketInfo& bucket_info, + RGWObjectCtx* rctx, RGWObjManifest* manifest, + int part_num, int* parts_count, bool prefetch, + RGWObjState** pstate, RGWObjManifest** pmanifest); + int on_last_entry_in_listing(const DoutPrefixProvider *dpp, RGWBucketInfo& bucket_info, const std::string& obj_prefix, diff --git a/src/rgw/driver/rados/rgw_sal_rados.cc b/src/rgw/driver/rados/rgw_sal_rados.cc index 1eb4eecceea..4c67d0ee71a 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.cc +++ b/src/rgw/driver/rados/rgw_sal_rados.cc @@ -723,7 +723,7 @@ int RadosBucket::merge_and_store_attrs(const DoutPrefixProvider* dpp, Attrs& new attrs[it.first] = it.second; } return store->ctl()->bucket->set_bucket_instance_attrs(get_info(), - new_attrs, &get_info().objv_tracker, y, dpp); + attrs, &get_info().objv_tracker, y, dpp); } int RadosBucket::try_refresh_info(const DoutPrefixProvider* dpp, ceph::real_time* pmtime, optional_yield y) @@ -2471,7 +2471,108 @@ bool RadosObject::is_sync_completed(const DoutPrefixProvider* dpp, const rgw_bi_log_entry& earliest_marker = entries.front(); return earliest_marker.timestamp > obj_mtime; -} +} /* is_sync_completed */ + +int RadosObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + int ret{0}; + + /* require an object with a manifest, so call to get_obj_state() must precede this */ + if (! manifest) { + return -EINVAL; + } + + RGWObjManifest::obj_iterator end = manifest->obj_end(dpp); + if (end.get_cur_part_id() == 0) { // not multipart + ldpp_dout(dpp, 20) << __func__ << " object does not have a multipart manifest" + << dendl; + return 0; + } + + auto end_part_id = end.get_cur_part_id(); + auto parts_count = (end_part_id == 1) ? 1 : end_part_id - 1; + if (marker > (parts_count - 1)) { + return 0; + } + + RGWObjManifest::obj_iterator part_iter = manifest->obj_begin(dpp); + + if (marker != 0) { + ldpp_dout_fmt(dpp, 20, + "{} seeking to part #{} in the object manifest", + __func__, marker); + + part_iter = manifest->obj_find_part(dpp, marker + 1); + + if (part_iter == end) { + ldpp_dout_fmt(dpp, 5, + "{} failed to find part #{} in the object manifest", + __func__, marker + 1); + return 0; + } + } + + RGWObjectCtx& obj_ctx = get_ctx(); + RGWBucketInfo& bucket_info = get_bucket()->get_info(); + + Object::Part obj_part{}; + for (; part_iter != manifest->obj_end(dpp); ++part_iter) { + + /* we're only interested in the first object in each logical part */ + auto cur_part_id = part_iter.get_cur_part_id(); + if (cur_part_id == obj_part.part_number) { + continue; + } + + if (max_parts < 1) { + *truncated = true; + break; + } + + /* get_part_obj_state alters the passed manifest** to point to a part + * manifest, which we don't want to leak out here */ + RGWObjManifest* obj_m = manifest; + RGWObjState* astate; + bool part_prefetch = false; + ret = RGWRados::get_part_obj_state(dpp, y, store->getRados(), bucket_info, &obj_ctx, + obj_m, cur_part_id, &parts_count, + part_prefetch, &astate, &obj_m); + + if (ret < 0) { + ldpp_dout_fmt(dpp, 4, + "{} get_part_obj_state() failed ret={}", + __func__, ret); + break; + } + + obj_part.part_number = part_iter.get_cur_part_id(); + obj_part.part_size = astate->accounted_size; + + if (auto iter = astate->attrset.find(RGW_ATTR_CKSUM); + iter != astate->attrset.end()) { + try { + rgw::cksum::Cksum part_cksum; + auto ck_iter = iter->second.cbegin(); + part_cksum.decode(ck_iter); + obj_part.cksum = std::move(part_cksum); + } catch (buffer::error& err) { + ldpp_dout_fmt(dpp, 4, + "WARN: {} could not decode stored cksum, " + "caught buffer::error", + __func__); + } + } + + each_func(obj_part); + *next_marker = ++marker; + --max_parts; + } /* each part */ + + return ret; +} /* RadosObject::list_parts */ int RadosObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { @@ -4500,11 +4601,6 @@ bool RadosZone::get_redirect_endpoint(std::string* endpoint) return true; } -bool RadosZone::has_zonegroup_api(const std::string& api) const -{ - return store->svc()->zone->has_zonegroup_api(api); -} - const std::string& RadosZone::get_current_period_id() { return store->svc()->zone->get_current_period_id(); diff --git a/src/rgw/driver/rados/rgw_sal_rados.h b/src/rgw/driver/rados/rgw_sal_rados.h index 23d81a934b0..85ea247e345 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.h +++ b/src/rgw/driver/rados/rgw_sal_rados.h @@ -107,7 +107,6 @@ class RadosZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() override; virtual const std::string& get_realm_name() override; @@ -593,12 +592,18 @@ class RadosObject : public StoreObject { StoreObject::set_compressed(); } - virtual bool is_sync_completed(const DoutPrefixProvider* dpp, const ceph::real_time& obj_mtime) override; /* For rgw_admin.cc */ RGWObjState& get_state() { return state; } virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; virtual int get_obj_attrs(optional_yield y, const DoutPrefixProvider* dpp, rgw_obj* target_obj = NULL) override; virtual int modify_obj_attrs(const char* attr_name, bufferlist& attr_val, optional_yield y, const DoutPrefixProvider* dpp) override; diff --git a/src/rgw/driver/rados/rgw_tools.cc b/src/rgw/driver/rados/rgw_tools.cc index 79d2be0bcfa..bf7a309e864 100644 --- a/src/rgw/driver/rados/rgw_tools.cc +++ b/src/rgw/driver/rados/rgw_tools.cc @@ -339,21 +339,35 @@ int rgw_list_pool(const DoutPrefixProvider *dpp, ldpp_dout(dpp, 10) << "failed to parse cursor: " << marker << dendl; return -EINVAL; } - - auto iter = ioctx.nobjects_begin(oc); + librados::NObjectIterator iter; + try { + iter = ioctx.nobjects_begin(oc); + } catch (const std::system_error& e) { + ldpp_dout(dpp, 1) << "rgw_list_pool: Failed to begin iteration of pool " + << ioctx.get_pool_name() << " with error " + << e.what() << dendl; + return ceph::from_error_code(e.code()); + } /// Pool_iterate if (iter == ioctx.nobjects_end()) return -ENOENT; - for (; oids->size() < max && iter != ioctx.nobjects_end(); ++iter) { - string oid = iter->get_oid(); - ldpp_dout(dpp, 20) << "RGWRados::pool_iterate: got " << oid << dendl; + try { + for (; oids->size() < max && iter != ioctx.nobjects_end(); ++iter) { + string oid = iter->get_oid(); + ldpp_dout(dpp, 20) << "RGWRados::pool_iterate: got " << oid << dendl; - // fill it in with initial values; we may correct later - if (filter && !filter(oid, oid)) - continue; + // fill it in with initial values; we may correct later + if (filter && !filter(oid, oid)) + continue; - oids->push_back(oid); + oids->push_back(oid); + } + } catch (const std::system_error& e) { + ldpp_dout(dpp, 1) << "rgw_list_pool: Failed iterating pool " + << ioctx.get_pool_name() << " with error " + << e.what() << dendl; + return ceph::from_error_code(e.code()); } marker = iter.get_cursor().to_str(); diff --git a/src/rgw/driver/rados/rgw_user.h b/src/rgw/driver/rados/rgw_user.h index ab157f38e39..4ae7d13eff7 100644 --- a/src/rgw/driver/rados/rgw_user.h +++ b/src/rgw/driver/rados/rgw_user.h @@ -19,11 +19,11 @@ #define RGW_USER_ANON_ID "anonymous" -#define SECRET_KEY_LEN 40 -#define PUBLIC_ID_LEN 20 -#define RAND_SUBUSER_LEN 5 +constexpr auto SECRET_KEY_LEN=40; +constexpr auto PUBLIC_ID_LEN=20; +constexpr auto RAND_SUBUSER_LEN=5; -#define XMLNS_AWS_S3 "http://s3.amazonaws.com/doc/2006-03-01/" +constexpr auto XMLNS_AWS_S3 = "http://s3.amazonaws.com/doc/2006-03-01/"; class RGWUserCtl; class RGWBucketCtl; diff --git a/src/rgw/driver/rados/rgw_zone.h b/src/rgw/driver/rados/rgw_zone.h index c542abc76d6..5fb2b4b8096 100644 --- a/src/rgw/driver/rados/rgw_zone.h +++ b/src/rgw/driver/rados/rgw_zone.h @@ -769,7 +769,6 @@ public: int create(const DoutPrefixProvider *dpp, optional_yield y, bool exclusive = true); int delete_obj(const DoutPrefixProvider *dpp, optional_yield y); int store_info(const DoutPrefixProvider *dpp, bool exclusive, optional_yield y); - int add_zonegroup(const DoutPrefixProvider *dpp, const RGWZoneGroup& zonegroup, optional_yield y); void fork(); int update(const DoutPrefixProvider *dpp, optional_yield y); diff --git a/src/rgw/rgw_orphan.cc b/src/rgw/radosgw-admin/orphan.cc index b7dc562c721..9fca3b99a7c 100644 --- a/src/rgw/rgw_orphan.cc +++ b/src/rgw/radosgw-admin/orphan.cc @@ -1,6 +1,12 @@ + +/* + * Copyright (C) 2024 IBM +*/ + // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab ft=cpp +#include "radosgw-admin/orphan.h" #include <string> @@ -10,7 +16,6 @@ #include "rgw_op.h" #include "rgw_multi.h" -#include "rgw_orphan.h" #include "rgw_zone.h" #include "rgw_bucket.h" #include "rgw_sal_rados.h" diff --git a/src/rgw/rgw_orphan.h b/src/rgw/radosgw-admin/orphan.h index db811d31d9a..db811d31d9a 100644 --- a/src/rgw/rgw_orphan.h +++ b/src/rgw/radosgw-admin/orphan.h diff --git a/src/rgw/rgw_admin.cc b/src/rgw/radosgw-admin/radosgw-admin.cc index a73db542014..182e42b8e31 100644 --- a/src/rgw/rgw_admin.cc +++ b/src/rgw/radosgw-admin/radosgw-admin.cc @@ -1,12 +1,15 @@ +/* + * Copyright (C) 2025 IBM +*/ + // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab ft=cpp -#include <errno.h> -#include <iostream> -#include <sstream> +#include <cerrno> #include <string> - -#include <boost/optional.hpp> +#include <sstream> +#include <optional> +#include <iostream> extern "C" { #include <liboath/oath.h> @@ -38,6 +41,9 @@ extern "C" { #include "include/utime.h" #include "include/str_list.h" +#include "radosgw-admin/orphan.h" +#include "radosgw-admin/sync_checkpoint.h" + #include "rgw_user.h" #include "rgw_otp.h" #include "rgw_rados.h" @@ -48,7 +54,6 @@ extern "C" { #include "rgw_log.h" #include "rgw_formats.h" #include "rgw_usage.h" -#include "rgw_orphan.h" #include "rgw_sync.h" #include "rgw_trim_bilog.h" #include "rgw_trim_datalog.h" @@ -62,7 +67,6 @@ extern "C" { #include "rgw_zone.h" #include "rgw_pubsub.h" #include "rgw_bucket_sync.h" -#include "rgw_sync_checkpoint.h" #include "rgw_lua.h" #include "rgw_sal.h" #include "rgw_sal_config.h" @@ -82,11 +86,6 @@ extern "C" { #define dout_context g_ceph_context -#define SECRET_KEY_LEN 40 -#define PUBLIC_ID_LEN 20 - -using namespace std; - static rgw::sal::Driver* driver = NULL; static constexpr auto dout_subsys = ceph_subsys_rgw; @@ -117,19 +116,13 @@ static const DoutPrefixProvider* dpp() { } \ } while (0) -static inline int posix_errortrans(int r) +using namespace std; + +inline int posix_errortrans(int r) { - switch(r) { - case ERR_NO_SUCH_BUCKET: - r = ENOENT; - break; - default: - break; - } - return r; + return ERR_NO_SUCH_BUCKET == r ? ENOENT : r; } - static const std::string LUA_CONTEXT_LIST("prerequest, postrequest, background, getdata, putdata"); void usage() @@ -361,6 +354,7 @@ void usage() cout << " --secret/--secret-key=<key> specify secret key\n"; cout << " --gen-access-key generate random access key (for S3)\n"; cout << " --gen-secret generate random secret key\n"; + cout << " --generate-key create user with or without credentials\n"; cout << " --key-type=<type> key type, options are: swift, s3\n"; cout << " --key-active=<bool> activate or deactivate a key\n"; cout << " --temp-url-key[-2]=<key> temp url key\n"; @@ -1271,7 +1265,7 @@ static int read_input(const string& infile, bufferlist& bl) } } -#define READ_CHUNK 8196 + constexpr auto READ_CHUNK=8196; int r; int err; @@ -2553,35 +2547,104 @@ std::ostream& operator<<(std::ostream& out, const indented& h) { return out << std::setw(h.w) << h.header << std::setw(1) << ' '; } -static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::RadosStore* driver, const RGWZone& zone, +struct bucket_source_sync_info { + const RGWZone& _source; + std::string_view error; + std::map<int,std::string> shards_behind; + int total_shards; + std::string_view status; + rgw_bucket bucket_source; + + bucket_source_sync_info(const RGWZone& source): _source(source) {} + + void _print_plaintext(std::ostream& out, int width) const { + out << indented{width, "source zone"} << _source.id << " (" << _source.name << ")" << std::endl; + if (!error.empty()) { + out << indented{width} << error << std::endl; + return; + } + out << indented{width, "source bucket"} << bucket_source << std::endl; + if (!status.empty()) { + out << indented{width} << status << std::endl; + return; + } + out << indented{width} << "incremental sync on " << total_shards << " shards\n"; + if (!shards_behind.empty()) { + out << indented{width} << "bucket is behind on " << shards_behind.size() << " shards\n"; + set<int> shard_ids; + for (auto const& [shard_id, _] : shards_behind) { + shard_ids.insert(shard_id); + } + out << indented{width} << "behind shards: [" << shard_ids << "]\n"; + } else { + out << indented{width} << "bucket is caught up with source\n"; + } + } + + void _print_formatter(std::ostream& out, Formatter* formatter) const { + formatter->open_object_section("source"); + formatter->dump_string("source_zone", _source.id); + formatter->dump_string("source_name", _source.name); + + if (!error.empty()) { + formatter->dump_string("error", error); + formatter->close_section(); + formatter->flush(out); + return; + } + + formatter->dump_string("source_bucket", bucket_source.name); + formatter->dump_string("source_bucket_id", bucket_source.bucket_id); + + if (!status.empty()) { + formatter->dump_string("status", status); + formatter->close_section(); + formatter->flush(out); + return; + } + + formatter->dump_int("total_shards", total_shards); + formatter->open_array_section("behind_shards"); + for (auto const& [id, marker] : shards_behind) { + formatter->open_object_section("shard"); + formatter->dump_int("shard_id", id); + formatter->dump_string("shard_marker", marker); + formatter->close_section(); + } + formatter->close_section(); + formatter->close_section(); + formatter->flush(out); + } +}; + +static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::RadosStore* driver, + const RGWZone& zone, const RGWZone& source, RGWRESTConn *conn, const RGWBucketInfo& bucket_info, rgw_sync_bucket_pipe pipe, - int width, std::ostream& out) + bucket_source_sync_info& source_sync_info) { - out << indented{width, "source zone"} << source.id << " (" << source.name << ")" << std::endl; - // syncing from this zone? if (!driver->svc()->zone->zone_syncs_from(zone, source)) { - out << indented{width} << "does not sync from zone\n"; + source_sync_info.error = "does not sync from zone"; return 0; } if (!pipe.source.bucket) { - ldpp_dout(dpp, -1) << __func__ << "(): missing source bucket" << dendl; + source_sync_info.error = fmt::format("{} (): missing source bucket", __func__); return -EINVAL; } std::unique_ptr<rgw::sal::Bucket> source_bucket; int r = init_bucket(*pipe.source.bucket, &source_bucket); if (r < 0) { - ldpp_dout(dpp, -1) << "failed to read source bucket info: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read source bucket info: {}", cpp_strerror(r)); return r; } - out << indented{width, "source bucket"} << source_bucket->get_key() << std::endl; - pipe.source.bucket = source_bucket->get_key(); + source_sync_info.bucket_source = source_bucket->get_key(); + pipe.source.bucket = source_bucket->get_key(); pipe.dest.bucket = bucket_info.bucket; uint64_t gen = 0; @@ -2592,15 +2655,15 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra r = rgw_read_bucket_full_sync_status(dpp, driver, pipe, &full_status, null_yield); if (r >= 0) { if (full_status.state == BucketSyncState::Init) { - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } if (full_status.state == BucketSyncState::Stopped) { - out << indented{width} << "stopped: bucket sync is disabled\n"; + source_sync_info.status = "stopped: bucket sync is disabled"; return 0; } if (full_status.state == BucketSyncState::Full) { - out << indented{width} << "full sync: " << full_status.full.count << " objects completed\n"; + source_sync_info.status = fmt::format("full sync: {} objects completed", full_status.full.count); return 0; } gen = full_status.incremental_gen; @@ -2609,46 +2672,45 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra // no full status, but there may be per-shard status from before upgrade const auto& logs = source_bucket->get_info().layout.logs; if (logs.empty()) { - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } const auto& log = logs.front(); if (log.gen > 0) { // this isn't the backward-compatible case, so we just haven't started yet - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } if (log.layout.type != rgw::BucketLogType::InIndex) { - ldpp_dout(dpp, -1) << "unrecognized log layout type " << log.layout.type << dendl; + source_sync_info.error = fmt::format("unrecognized log layout type {}", to_string(log.layout.type)); return -EINVAL; } // use shard count from our log gen=0 shard_status.resize(rgw::num_shards(log.layout.in_index)); } else { - lderr(driver->ctx()) << "failed to read bucket full sync status: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read bucket full sync status: {}", cpp_strerror(r)); return r; } r = rgw_read_bucket_inc_sync_status(dpp, driver, pipe, gen, &shard_status); if (r < 0) { - lderr(driver->ctx()) << "failed to read bucket incremental sync status: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read bucket incremental sync status: {}", cpp_strerror(r)); return r; } const int total_shards = shard_status.size(); - - out << indented{width} << "incremental sync on " << total_shards << " shards\n"; + source_sync_info.total_shards = total_shards; rgw_bucket_index_marker_info remote_info; BucketIndexShardsManager remote_markers; r = rgw_read_remote_bilog_info(dpp, conn, source_bucket->get_key(), remote_info, remote_markers, null_yield); if (r < 0) { - ldpp_dout(dpp, -1) << "failed to read remote log: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read remote log: {}", cpp_strerror(r)); return r; } - std::set<int> shards_behind; + std::map<int, std::string> shards_behind; for (const auto& r : remote_markers.get()) { auto shard_id = r.first; if (r.second.empty()) { @@ -2656,21 +2718,17 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra } if (shard_id >= total_shards) { // unexpected shard id. we don't have status for it, so we're behind - shards_behind.insert(shard_id); + shards_behind[shard_id] = r.second; continue; } auto& m = shard_status[shard_id]; const auto pos = BucketIndexShardsManager::get_shard_marker(m.inc_marker.position); if (pos < r.second) { - shards_behind.insert(shard_id); + shards_behind[shard_id] = r.second; } } - if (!shards_behind.empty()) { - out << indented{width} << "bucket is behind on " << shards_behind.size() << " shards\n"; - out << indented{width} << "behind shards: [" << shards_behind << "]\n"; - } else { - out << indented{width} << "bucket is caught up with source\n"; - } + + source_sync_info.shards_behind = std::move(shards_behind); return 0; } @@ -2881,25 +2939,82 @@ static int bucket_sync_info(rgw::sal::Driver* driver, const RGWBucketInfo& info, return 0; } +struct bucket_sync_status_info { + std::vector<bucket_source_sync_info> source_status_info; + rgw::sal::Zone* _zone; + const rgw::sal::ZoneGroup* _zonegroup; + const RGWBucketInfo& _bucket_info; + const int width = 15; + std::string error; + + bucket_sync_status_info(const RGWBucketInfo& bucket_info): _bucket_info(bucket_info) {} + + void print(std::ostream& out, bool use_formatter, Formatter* formatter) { + if (use_formatter) { + _print_formatter(out, formatter); + } else { + _print_plaintext(out); + } + } + + void _print_plaintext(std::ostream& out) { + out << indented{width, "realm"} << _zone->get_realm_id() << " (" << _zone->get_realm_name() << ")" << std::endl; + out << indented{width, "zonegroup"} << _zonegroup->get_id() << " (" << _zonegroup->get_name() << ")" << std::endl; + out << indented{width, "zone"} << _zone->get_id() << " (" << _zone->get_name() << ")" << std::endl; + out << indented{width, "bucket"} << _bucket_info.bucket << std::endl; + out << indented{width, "current time"} + << to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms) << "\n\n"; + + if (!error.empty()){ + out << error << std::endl; + } + + for (const auto &info : source_status_info) { + info._print_plaintext(out, width); + } + } + + void _print_formatter(std::ostream& out, Formatter* formatter) { + formatter->open_object_section("test"); + formatter->dump_string("realm", _zone->get_realm_id()); + formatter->dump_string("realm_name", _zone->get_realm_name()); + formatter->dump_string("zonegroup", _zonegroup->get_id()); + formatter->dump_string("zonegroup_name", _zonegroup->get_name()); + formatter->dump_string("zone", _zone->get_id()); + formatter->dump_string("zone_name", _zone->get_name()); + formatter->dump_string("bucket", _bucket_info.bucket.name); + formatter->dump_string("bucket_instance_id", _bucket_info.bucket.bucket_id); + formatter->dump_string("current_time", to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms)); + + if (!error.empty()) { + formatter->dump_string("error", error); + } + + formatter->open_array_section("sources"); + for (const auto &info : source_status_info) { + info._print_formatter(out, formatter); + } + formatter->close_section(); + + formatter->close_section(); + formatter->flush(out); + } + +}; + static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& info, const rgw_zone_id& source_zone_id, std::optional<rgw_bucket>& opt_source_bucket, - std::ostream& out) + bucket_sync_status_info& bucket_sync_info) { const rgw::sal::ZoneGroup& zonegroup = driver->get_zone()->get_zonegroup(); rgw::sal::Zone* zone = driver->get_zone(); - constexpr int width = 15; - - out << indented{width, "realm"} << zone->get_realm_id() << " (" << zone->get_realm_name() << ")\n"; - out << indented{width, "zonegroup"} << zonegroup.get_id() << " (" << zonegroup.get_name() << ")\n"; - out << indented{width, "zone"} << zone->get_id() << " (" << zone->get_name() << ")\n"; - out << indented{width, "bucket"} << info.bucket << "\n"; - out << indented{width, "current time"} - << to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms) << "\n\n"; + bucket_sync_info._zone = zone; + bucket_sync_info._zonegroup = &zonegroup; if (!static_cast<rgw::sal::RadosStore*>(driver)->ctl()->bucket->bucket_imports_data(info.bucket, null_yield, dpp())) { - out << "Sync is disabled for bucket " << info.bucket.name << " or bucket has no sync sources" << std::endl; + bucket_sync_info.error = fmt::format("Sync is disabled for bucket {} or bucket has no sync sources", info.bucket.name); return 0; } @@ -2907,7 +3022,7 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf int r = driver->get_sync_policy_handler(dpp(), std::nullopt, info.bucket, &handler, null_yield); if (r < 0) { - ldpp_dout(dpp(), -1) << "ERROR: failed to get policy handler for bucket (" << info.bucket << "): r=" << r << ": " << cpp_strerror(-r) << dendl; + bucket_sync_info.error = fmt::format("ERROR: failed to get policy handler for bucket ({}): r={}: {}", info.bucket.name, r, cpp_strerror(-r)); return r; } @@ -2920,13 +3035,12 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf std::unique_ptr<rgw::sal::Zone> zone; int ret = driver->get_zone()->get_zonegroup().get_zone_by_id(source_zone_id.id, &zone); if (ret < 0) { - ldpp_dout(dpp(), -1) << "Source zone not found in zonegroup " - << zonegroup.get_name() << dendl; + bucket_sync_info.error = fmt::format("Source zone not found in zonegroup {}", zonegroup.get_name()); return -EINVAL; } auto c = zone_conn_map.find(source_zone_id); if (c == zone_conn_map.end()) { - ldpp_dout(dpp(), -1) << "No connection to zone " << zone->get_name() << dendl; + bucket_sync_info.error = fmt::format("No connection to zone {}", zone->get_name()); return -EINVAL; } zone_ids.insert(source_zone_id); @@ -2957,10 +3071,15 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf continue; } if (pipe.source.zone.value_or(rgw_zone_id()) == z->second.id) { - bucket_source_sync_status(dpp(), static_cast<rgw::sal::RadosStore*>(driver), static_cast<rgw::sal::RadosStore*>(driver)->svc()->zone->get_zone(), z->second, + bucket_source_sync_info source_sync_info(z->second); + auto ret = bucket_source_sync_status(dpp(), static_cast<rgw::sal::RadosStore*>(driver), static_cast<rgw::sal::RadosStore*>(driver)->svc()->zone->get_zone(), z->second, c->second, info, pipe, - width, out); + source_sync_info); + + if (ret == 0) { + bucket_sync_info.source_status_info.emplace_back(std::move(source_sync_info)); + } } } } @@ -3431,6 +3550,13 @@ int main(int argc, const char **argv) OPT opt_cmd = OPT::NO_CMD; int gen_access_key = 0; int gen_secret_key = 0; + enum generate_key_enum { + OPTION_SET_FALSE = 0, + OPTION_SET_TRUE = 1, + OPTION_NOT_SET = 2, + }; + + generate_key_enum generate_key = OPTION_NOT_SET; bool set_perm = false; bool set_temp_url_key = false; map<int, string> temp_url_keys; @@ -3488,6 +3614,7 @@ int main(int argc, const char **argv) list<string> tags_rm; int placement_inline_data = true; bool placement_inline_data_specified = false; + bool format_arg_passed = false; int64_t max_objects = -1; int64_t max_size = -1; @@ -3711,6 +3838,17 @@ int main(int argc, const char **argv) cerr << "bad key type: " << key_type_str << std::endl; exit(1); } + } else if (ceph_argparse_witharg(args, i, &val, "--generate-key", (char*)NULL)) { + key_type_str = val; + if (key_type_str.compare("true") == 0) { + generate_key = OPTION_SET_TRUE; + } else if(key_type_str.compare("false") == 0) { + generate_key = OPTION_SET_FALSE; + } else { + cerr << "wrong value for --generate-key: " << key_type_str << " please specify either true or false" << std::endl; + exit(1); + } + // do nothing } else if (ceph_argparse_binary_flag(args, i, &key_active, NULL, "--key-active", (char*)NULL)) { key_active_specified = true; } else if (ceph_argparse_witharg(args, i, &val, "--job-id", (char*)NULL)) { @@ -3867,6 +4005,7 @@ int main(int argc, const char **argv) new_bucket_name = val; } else if (ceph_argparse_witharg(args, i, &val, "--format", (char*)NULL)) { format = val; + format_arg_passed = true; } else if (ceph_argparse_witharg(args, i, &val, "--categories", (char*)NULL)) { string cat_str = val; list<string> cat_list; @@ -4473,14 +4612,21 @@ int main(int argc, const char **argv) } /* check key parameter conflict */ - if ((!access_key.empty()) && gen_access_key) { - cerr << "ERROR: key parameter conflict, --access-key & --gen-access-key" << std::endl; + if ((!access_key.empty()) && (gen_access_key || generate_key == OPTION_SET_TRUE)) { + cerr << "ERROR: key parameter conflict, --access-key & --gen-access-key/generate-key" << std::endl; + return EINVAL; + } + if ((!secret_key.empty()) && (gen_secret_key || generate_key == OPTION_SET_TRUE)) { + cerr << "ERROR: key parameter conflict, --secret & --gen-secret/generate-key" << std::endl; return EINVAL; } - if ((!secret_key.empty()) && gen_secret_key) { - cerr << "ERROR: key parameter conflict, --secret & --gen-secret" << std::endl; + if (generate_key == OPTION_SET_FALSE) { + if ((!access_key.empty()) || gen_access_key || (!secret_key.empty()) || gen_secret_key) { + cerr << "ERROR: key parameter conflict, if --generate-key is not set so no other key parameters can be set" << std::endl; return EINVAL; + } } + } // default to pretty json @@ -6645,7 +6791,7 @@ int main(int argc, const char **argv) } break; case OPT::USER_CREATE: - if (!user_op.has_existing_user()) { + if (!user_op.has_existing_user() && (generate_key != OPTION_SET_FALSE)) { user_op.set_generate_key(); // generate a new key by default } ret = ruser.add(dpp(), user_op, null_yield, &err_msg); @@ -7599,7 +7745,7 @@ int main(int argc, const char **argv) << "' to target bucket '" << configuration.target_bucket << "'" << std::endl; return -ret; } - cerr << "flushed pending logging object '" << obj_name + cout << "flushed pending logging object '" << obj_name << "' to target bucket '" << configuration.target_bucket << "'" << std::endl; return 0; } @@ -9901,7 +10047,18 @@ next: if (ret < 0) { return -ret; } - bucket_sync_status(driver, bucket->get_info(), source_zone, opt_source_bucket, std::cout); + + auto bucket_info = bucket->get_info(); + bucket_sync_status_info bucket_sync_info(bucket_info); + + ret = bucket_sync_status(driver, bucket_info, source_zone, + opt_source_bucket, bucket_sync_info); + + if (ret == 0) { + bucket_sync_info.print(std::cout, format_arg_passed, formatter.get()); + } else { + cerr << "failed to get bucket sync status. see logs for more info" << std::endl; + } } if (opt_cmd == OPT::BUCKET_SYNC_MARKERS) { diff --git a/src/rgw/rgw_sync_checkpoint.cc b/src/rgw/radosgw-admin/sync_checkpoint.cc index 1172e79a48f..0303ed6c747 100644 --- a/src/rgw/rgw_sync_checkpoint.cc +++ b/src/rgw/radosgw-admin/sync_checkpoint.cc @@ -5,6 +5,7 @@ * Ceph - scalable distributed file system * * Copyright (C) 2020 Red Hat, Inc. + * Copyright (C) 2024 IBM * * This is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -13,9 +14,12 @@ * */ +#include "radosgw-admin/sync_checkpoint.h" + #include <fmt/format.h> + #include "common/errno.h" -#include "rgw_sync_checkpoint.h" + #include "rgw_sal_rados.h" #include "rgw_bucket_sync.h" #include "rgw_data_sync.h" diff --git a/src/rgw/rgw_sync_checkpoint.h b/src/rgw/radosgw-admin/sync_checkpoint.h index 28df68d8860..28df68d8860 100644 --- a/src/rgw/rgw_sync_checkpoint.h +++ b/src/rgw/radosgw-admin/sync_checkpoint.h diff --git a/src/rgw/rgw_asio_frontend.cc b/src/rgw/rgw_asio_frontend.cc index 30e1e77fd15..ebe42d96de9 100644 --- a/src/rgw/rgw_asio_frontend.cc +++ b/src/rgw/rgw_asio_frontend.cc @@ -1194,8 +1194,11 @@ void AsioFrontend::pause() l.signal.emit(boost::asio::cancellation_type::terminal); } - // close all connections so outstanding requests fail quickly - connections.close(ec); + const bool graceful_stop{ g_ceph_context->_conf->rgw_graceful_stop }; + if (!graceful_stop) { + // close all connections so outstanding requests fail quickly + connections.close(ec); + } // pause and wait until outstanding requests complete pause_mutex.lock(ec); diff --git a/src/rgw/rgw_auth.cc b/src/rgw/rgw_auth.cc index ec2a2079622..a0b494eb9c5 100644 --- a/src/rgw/rgw_auth.cc +++ b/src/rgw/rgw_auth.cc @@ -188,7 +188,8 @@ int load_account_and_policies(const DoutPrefixProvider* dpp, static auto transform_old_authinfo(const RGWUserInfo& user, std::optional<RGWAccountInfo> account, - std::vector<IAM::Policy> policies) + std::vector<IAM::Policy> policies, + sal::Driver* driver) -> std::unique_ptr<rgw::auth::Identity> { /* This class is not intended for public use. Should be removed altogether @@ -198,6 +199,7 @@ static auto transform_old_authinfo(const RGWUserInfo& user, /* For this particular case it's OK to use rgw_user structure to convey * the identity info as this was the policy for doing that before the * new auth. */ + sal::Driver* driver; const rgw_user id; const std::string display_name; const std::string path; @@ -208,8 +210,10 @@ static auto transform_old_authinfo(const RGWUserInfo& user, public: DummyIdentityApplier(const RGWUserInfo& user, std::optional<RGWAccountInfo> account, - std::vector<IAM::Policy> policies) - : id(user.user_id), + std::vector<IAM::Policy> policies, + sal::Driver* driver) + : driver(driver), + id(user.user_id), display_name(user.display_name), path(user.path), is_admin(user.admin), @@ -294,9 +298,9 @@ static auto transform_old_authinfo(const RGWUserInfo& user, << ", is_admin=" << is_admin << ")"; } - void load_acct_info(const DoutPrefixProvider* dpp, - RGWUserInfo& user_info) const override { + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override { // noop, this user info was passed in on construction + return driver->get_user(id); } void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const { @@ -307,13 +311,14 @@ static auto transform_old_authinfo(const RGWUserInfo& user, }; return std::make_unique<DummyIdentityApplier>( - user, std::move(account), std::move(policies)); + user, std::move(account), std::move(policies), driver); } auto transform_old_authinfo(const DoutPrefixProvider* dpp, optional_yield y, sal::Driver* driver, - sal::User* user) + sal::User* user, + std::vector<IAM::Policy>* policies_) -> tl::expected<std::unique_ptr<Identity>, int> { const RGWUserInfo& info = user->get_info(); @@ -328,7 +333,10 @@ auto transform_old_authinfo(const DoutPrefixProvider* dpp, return tl::unexpected(r); } - return transform_old_authinfo(info, std::move(account), std::move(policies)); + if (policies_) { // return policies to caller if requested + *policies_ = policies; + } + return transform_old_authinfo(info, std::move(account), std::move(policies), driver); } } /* namespace auth */ @@ -523,7 +531,7 @@ rgw::auth::Strategy::apply(const DoutPrefixProvider *dpp, const rgw::auth::Strat /* Account used by a given RGWOp is decoupled from identity employed * in the authorization phase (RGWOp::verify_permissions). */ - applier->load_acct_info(dpp, s->user->get_info()); + s->user = applier->load_acct_info(dpp); s->perm_mask = applier->get_perm_mask(); /* This is the single place where we pass req_state as a pointer @@ -631,36 +639,36 @@ void rgw::auth::WebIdentityApplier::create_account(const DoutPrefixProvider* dpp user_info = user->get_info(); } -void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const { +auto rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { rgw_user federated_user; federated_user.id = this->sub; federated_user.tenant = role_tenant; federated_user.ns = "oidc"; + std::unique_ptr<rgw::sal::User> user = driver->get_user(federated_user); if (account) { // we don't need shadow users for account roles because bucket ownership, // quota, and stats are tracked by the account instead of the user - user_info.user_id = std::move(federated_user); + RGWUserInfo& user_info = user->get_info(); user_info.display_name = user_name; user_info.type = TYPE_WEB; - return; + // the user_info.user_id is initialized by driver->get_user(...) + return user; } - std::unique_ptr<rgw::sal::User> user = driver->get_user(federated_user); - //Check in oidc namespace if (user->load_user(dpp, null_yield) >= 0) { /* Succeeded. */ - user_info = user->get_info(); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } user->clear_ns(); //Check for old users which wouldn't have been created in oidc namespace if (user->load_user(dpp, null_yield) >= 0) { /* Succeeded. */ - user_info = user->get_info(); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } //Check if user_id.buckets already exists, may have been from the time, when shadow users didnt exist @@ -671,7 +679,7 @@ void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp last_synced, last_updated); if (ret < 0 && ret != -ENOENT) { ldpp_dout(dpp, 0) << "ERROR: reading stats for the user returned error " << ret << dendl; - return; + return user; } if (ret == -ENOENT) { /* in case of ENOENT, which means user doesnt have buckets */ //In this case user will be created in oidc namespace @@ -684,7 +692,8 @@ void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp } ldpp_dout(dpp, 0) << "NOTICE: couldn't map oidc federated user " << federated_user << dendl; - create_account(dpp, federated_user, this->user_name, user_info); + create_account(dpp, federated_user, this->user_name, user->get_info()); + return user; } void rgw::auth::WebIdentityApplier::modify_request_state(const DoutPrefixProvider *dpp, req_state* s) const @@ -936,7 +945,7 @@ void rgw::auth::RemoteApplier::write_ops_log_entry(rgw_log_entry& entry) const } /* TODO(rzarzynski): we need to handle display_name changes. */ -void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* It's supposed that RGWRemoteAuthApplier tries to load account info * that belongs to the authenticated identity. Another policy may be @@ -975,9 +984,9 @@ void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGW (void) load_account_and_policies(dpp, null_yield, driver, user->get_info(), user->get_attrs(), account, policies); - user_info = std::move(user->get_info()); owner_acct_user = std::move(tenanted_uid); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } } @@ -990,15 +999,16 @@ void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGW (void) load_account_and_policies(dpp, null_yield, driver, user->get_info(), user->get_attrs(), account, policies); - user_info = std::move(user->get_info()); owner_acct_user = acct_user; - return; + // the user_info in user is initialized by user->load_user(...) + return user; } ldpp_dout(dpp, 0) << "NOTICE: couldn't map swift user " << acct_user << dendl; - create_account(dpp, acct_user, implicit_tenant, user_info); + create_account(dpp, acct_user, implicit_tenant, user->get_info()); /* Succeeded if we are here (create_account() hasn't throwed). */ + return user; } void rgw::auth::RemoteApplier::modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const @@ -1098,11 +1108,11 @@ uint32_t rgw::auth::LocalApplier::get_perm_mask(const std::string& subuser_name, } } -void rgw::auth::LocalApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::LocalApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* Load the account that belongs to the authenticated identity. An extra call * to RADOS may be safely skipped in this case. */ - user_info = this->user_info; + return std::unique_ptr<rgw::sal::User>(user.release()); } void rgw::auth::LocalApplier::modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const @@ -1121,6 +1131,22 @@ void rgw::auth::LocalApplier::write_ops_log_entry(rgw_log_entry& entry) const } } +rgw::auth::LocalApplier::LocalApplier(CephContext* const cct, + std::unique_ptr<rgw::sal::User> user, + std::optional<RGWAccountInfo> account, + std::vector<IAM::Policy> policies, + std::string subuser, + const std::optional<uint32_t>& perm_mask, + const std::string access_key_id) + : user_info(user->get_info()), + user(std::move(user)), + account(std::move(account)), + policies(std::move(policies)), + subuser(std::move(subuser)), + perm_mask(perm_mask.value_or(RGW_PERM_INVALID)), + access_key_id(access_key_id) { +} + ACLOwner rgw::auth::RoleApplier::get_aclowner() const { ACLOwner owner; @@ -1183,10 +1209,11 @@ bool rgw::auth::RoleApplier::is_identity(const Principal& p) const { return false; } -void rgw::auth::RoleApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::RoleApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* Load the user id */ - user_info.user_id = this->token_attrs.user_id; + std::unique_ptr<rgw::sal::User> user = driver->get_user(this->token_attrs.user_id); + return user; } void rgw::auth::RoleApplier::write_ops_log_entry(rgw_log_entry& entry) const @@ -1267,9 +1294,10 @@ rgw::auth::AnonymousEngine::authenticate(const DoutPrefixProvider* dpp, const re } else { RGWUserInfo user_info; rgw_get_anon_user(user_info); - + std::unique_ptr<rgw::sal::User> user = s->user->clone(); + user->get_info() = user_info; auto apl = \ - apl_factory->create_apl_local(cct, s, user_info, std::nullopt, {}, + apl_factory->create_apl_local(cct, s, std::move(user), std::nullopt, {}, rgw::auth::LocalApplier::NO_SUBUSER, std::nullopt, rgw::auth::LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); diff --git a/src/rgw/rgw_auth.h b/src/rgw/rgw_auth.h index f3edbbab845..22b0816bac9 100644 --- a/src/rgw/rgw_auth.h +++ b/src/rgw/rgw_auth.h @@ -105,7 +105,8 @@ inline std::ostream& operator<<(std::ostream& out, auto transform_old_authinfo(const DoutPrefixProvider* dpp, optional_yield y, sal::Driver* driver, - sal::User* user) + sal::User* user, + std::vector<IAM::Policy>* policies_ = nullptr) -> tl::expected<std::unique_ptr<Identity>, int>; // Load the user account and all user/group policies. May throw @@ -139,7 +140,7 @@ public: * * XXX: be aware that the "account" term refers to rgw_user. The naming * is legacy. */ - virtual void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const = 0; /* out */ + virtual auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> = 0; /* out */ /* Apply any changes to request state. This method will be most useful for * TempURL of Swift API. */ @@ -484,7 +485,7 @@ public: bool is_identity(const Principal& p) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; uint32_t get_identity_type() const override { return TYPE_WEB; @@ -656,7 +657,7 @@ public: uint32_t get_perm_mask() const override { return info.perm_mask; } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; void write_ops_log_entry(rgw_log_entry& entry) const override; uint32_t get_identity_type() const override { return info.acct_type; } @@ -683,7 +684,7 @@ public: /* rgw::auth::LocalApplier targets those auth engines that base on the data - * enclosed in the RGWUserInfo control structure. As a side effect of doing + * enclosed in the rgw::sal::User->RGWUserInfo control structure. As a side effect of doing * the authentication process, they must have it loaded. Leveraging this is * a way to avoid unnecessary calls to underlying RADOS store. */ class LocalApplier : public IdentityApplier { @@ -691,6 +692,7 @@ class LocalApplier : public IdentityApplier { protected: const RGWUserInfo user_info; + mutable std::unique_ptr<rgw::sal::User> user; const std::optional<RGWAccountInfo> account; const std::vector<IAM::Policy> policies; const std::string subuser; @@ -705,19 +707,12 @@ public: static const std::string NO_ACCESS_KEY; LocalApplier(CephContext* const cct, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, std::string subuser, const std::optional<uint32_t>& perm_mask, - const std::string access_key_id) - : user_info(user_info), - account(std::move(account)), - policies(std::move(policies)), - subuser(std::move(subuser)), - perm_mask(perm_mask.value_or(RGW_PERM_INVALID)), - access_key_id(access_key_id) { - } + const std::string access_key_id); ACLOwner get_aclowner() const override; uint32_t get_perms_from_aclspec(const DoutPrefixProvider* dpp, const aclspec_t& aclspec) const override; @@ -732,7 +727,7 @@ public: } } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; uint32_t get_identity_type() const override { return user_info.type; } std::string get_acct_name() const override { return {}; } @@ -750,7 +745,7 @@ public: virtual ~Factory() {} virtual aplptr_t create_apl_local(CephContext* cct, const req_state* s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, @@ -779,15 +774,20 @@ public: std::vector<std::pair<std::string, std::string>> principal_tags; }; protected: + CephContext* const cct; + rgw::sal::Driver* driver; Role role; TokenAttrs token_attrs; public: RoleApplier(CephContext* const cct, + rgw::sal::Driver* driver, const Role& role, const TokenAttrs& token_attrs) - : role(role), + : cct(cct), + driver(driver), + role(role), token_attrs(token_attrs) {} ACLOwner get_aclowner() const override; @@ -803,7 +803,7 @@ public: return RGW_PERM_NONE; } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ uint32_t get_identity_type() const override { return TYPE_ROLE; } std::string get_acct_name() const override { return {}; } std::string get_subuser() const override { return {}; } diff --git a/src/rgw/rgw_auth_filters.h b/src/rgw/rgw_auth_filters.h index a93641e8b8e..7d264197c52 100644 --- a/src/rgw/rgw_auth_filters.h +++ b/src/rgw/rgw_auth_filters.h @@ -117,8 +117,8 @@ public: return get_decoratee().get_account(); } - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override { /* out */ - return get_decoratee().load_acct_info(dpp, user_info); + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override { /* out */ + return get_decoratee().load_acct_info(dpp); } void modify_request_state(const DoutPrefixProvider* dpp, req_state * s) const override { /* in/out */ @@ -152,7 +152,7 @@ public: } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ }; /* static declaration: UNKNOWN_ACCT will be an empty rgw_user that is a result @@ -169,23 +169,25 @@ void ThirdPartyAccountApplier<T>::to_str(std::ostream& out) const } template <typename T> -void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const +auto ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { + std::unique_ptr<rgw::sal::User> luser; if (UNKNOWN_ACCT == acct_user_override) { /* There is no override specified by the upper layer. This means that we'll * load the account owned by the authenticated identity (aka auth_user). */ - DecoratedApplier<T>::load_acct_info(dpp, user_info); + luser = DecoratedApplier<T>::load_acct_info(dpp); } else if (DecoratedApplier<T>::is_owner_of(acct_user_override)) { /* The override has been specified but the account belongs to the authenticated * identity. We may safely forward the call to a next stage. */ - DecoratedApplier<T>::load_acct_info(dpp, user_info); + luser = DecoratedApplier<T>::load_acct_info(dpp); } else if (this->is_anonymous()) { /* If the user was authed by the anonymous engine then scope the ANON user * to the correct tenant */ + luser = driver->get_user(rgw_user(RGW_USER_ANON_ID)); if (acct_user_override.tenant.empty()) - user_info.user_id = rgw_user(acct_user_override.id, RGW_USER_ANON_ID); + luser->get_info().user_id = rgw_user(acct_user_override.id, RGW_USER_ANON_ID); else - user_info.user_id = rgw_user(acct_user_override.tenant, RGW_USER_ANON_ID); + luser->get_info().user_id = rgw_user(acct_user_override.tenant, RGW_USER_ANON_ID); } else { /* Compatibility mechanism for multi-tenancy. For more details refer to * load_acct_info method of rgw::auth::RemoteApplier. */ @@ -196,9 +198,10 @@ void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, user = driver->get_user(tenanted_uid); if (user->load_user(dpp, null_yield) >= 0) { - user_info = user->get_info(); + // the user_info in luser is initialized by user->load_user(...) + luser = user->clone(); /* Succeeded. */ - return; + return luser; } } @@ -213,8 +216,10 @@ void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, throw ret; } } - user_info = user->get_info(); + // the user_info in luser is initialized by user->load_user(...) + luser = user->clone(); } + return luser; } template <typename T> static inline @@ -248,7 +253,7 @@ public: } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; /* in/out */ ACLOwner get_aclowner() const override { @@ -271,10 +276,10 @@ void SysReqApplier<T>::to_str(std::ostream& out) const } template <typename T> -void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const +auto SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { - DecoratedApplier<T>::load_acct_info(dpp, user_info); - is_system = user_info.system; + std::unique_ptr<rgw::sal::User> user = DecoratedApplier<T>::load_acct_info(dpp); + is_system = user->get_info().system; if (is_system) { //ldpp_dout(dpp, 20) << "system request" << dendl; @@ -285,7 +290,7 @@ void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo effective_owner->id = parse_owner(str); if (const auto* uid = std::get_if<rgw_user>(&effective_owner->id); uid) { - std::unique_ptr<rgw::sal::User> user = driver->get_user(*uid); + user = driver->get_user(*uid); if (user->load_user(dpp, null_yield) < 0) { //ldpp_dout(dpp, 0) << "User lookup failed!" << dendl; throw -EACCES; @@ -294,14 +299,14 @@ void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo } } } + return user; } template <typename T> void SysReqApplier<T>::modify_request_state(const DoutPrefixProvider* dpp, req_state* const s) const { if (boost::logic::indeterminate(is_system)) { - RGWUserInfo unused_info; - load_acct_info(dpp, unused_info); + std::unique_ptr<rgw::sal::User> unused_user{ load_acct_info(dpp) }; } if (is_system) { diff --git a/src/rgw/rgw_auth_s3.h b/src/rgw/rgw_auth_s3.h index 2f7fd2d7598..5815a520e02 100644 --- a/src/rgw/rgw_auth_s3.h +++ b/src/rgw/rgw_auth_s3.h @@ -55,14 +55,14 @@ class STSAuthStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, const std::optional<uint32_t>& perm_mask, const std::string& access_key_id) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id)); return aplptr_t(new decltype(apl)(std::move(apl))); } @@ -72,7 +72,7 @@ class STSAuthStrategy : public rgw::auth::Strategy, RoleApplier::Role role, RoleApplier::TokenAttrs token_attrs) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - rgw::auth::RoleApplier(cct, std::move(role), std::move(token_attrs))); + rgw::auth::RoleApplier(cct, driver, std::move(role), std::move(token_attrs))); return aplptr_t(new decltype(apl)(std::move(apl))); } @@ -176,14 +176,14 @@ class AWSAuthStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, const std::optional<uint32_t>& perm_mask, const std::string& access_key_id) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id)); /* TODO(rzarzynski): replace with static_ptr. */ return aplptr_t(new decltype(apl)(std::move(apl))); diff --git a/src/rgw/rgw_bucket_layout.cc b/src/rgw/rgw_bucket_layout.cc index f8c485d89c3..1f8db396a0d 100644 --- a/src/rgw/rgw_bucket_layout.cc +++ b/src/rgw/rgw_bucket_layout.cc @@ -376,9 +376,9 @@ void encode_json_impl(const char *name, const BucketLayout& l, ceph::Formatter * for (const auto& log : l.logs) { encode_json("log", log, f); } + f->close_section(); // logs[] utime_t jt(l.judge_reshard_lock_time); encode_json("judge_reshard_lock_time", jt, f); - f->close_section(); // logs[] f->close_section(); } void decode_json_obj(BucketLayout& l, JSONObj *obj) diff --git a/src/rgw/rgw_bucket_logging.cc b/src/rgw/rgw_bucket_logging.cc index 87a242d9952..d24a53024f1 100644 --- a/src/rgw/rgw_bucket_logging.cc +++ b/src/rgw/rgw_bucket_logging.cc @@ -31,6 +31,9 @@ bool configuration::decode_xml(XMLObj* obj) { logging_type = LoggingType::Standard; } else if (type == "Journal") { logging_type = LoggingType::Journal; + if (iter = o->find("Filter"); XMLObj* const filter_o = iter.get_next()) { + RGWXMLDecoder::decode_xml("S3Key", key_filter, filter_o); + } } else { // we don't allow for type "Any" in the configuration throw RGWXMLDecoder::err("invalid bucket logging record type: '" + type + "'"); @@ -73,6 +76,11 @@ void configuration::dump_xml(Formatter *f) const { break; case LoggingType::Journal: ::encode_xml("LoggingType", "Journal", f); + if (key_filter.has_content()) { + f->open_object_section("Filter"); + ::encode_xml("S3Key", key_filter, f); + f->close_section(); // Filter + } break; case LoggingType::Any: ::encode_xml("LoggingType", "", f); @@ -118,6 +126,10 @@ void configuration::dump(Formatter *f) const { break; case LoggingType::Journal: encode_json("loggingType", "Journal", f); + if (key_filter.has_content()) { + Formatter::ObjectSection s(*f, "Filter"); + encode_json("S3Key", key_filter, f); + } break; case LoggingType::Any: encode_json("loggingType", "", f); @@ -526,6 +538,11 @@ int log_record(rgw::sal::Driver* driver, if (type != LoggingType::Any && configuration.logging_type != type) { return 0; } + if (configuration.key_filter.has_content()) { + if (!match(configuration.key_filter, obj->get_name())) { + return 0; + } + } ldpp_dout(dpp, 20) << "INFO: found matching logging configuration of bucket '" << s->bucket->get_name() << "' configuration: " << configuration.to_json_str() << dendl; if (auto ret = log_record(driver, obj, s, op_name, etag, size, configuration, dpp, y, async_completion, log_source_bucket); ret < 0) { diff --git a/src/rgw/rgw_bucket_logging.h b/src/rgw/rgw_bucket_logging.h index d380cf0ef09..d4877bafb0f 100644 --- a/src/rgw/rgw_bucket_logging.h +++ b/src/rgw/rgw_bucket_logging.h @@ -10,6 +10,7 @@ #include "include/buffer.h" #include "include/encoding.h" #include "common/async/yield_context.h" +#include "rgw_s3_filter.h" class XMLObj; namespace ceph { class Formatter; } @@ -48,6 +49,14 @@ namespace rgw::bucketlogging { <LoggingType>Standard|Journal</LoggingType> <!-- Ceph extension --> <ObjectRollTime>integer</ObjectRollTime> <!-- Ceph extension --> <RecordsBatchSize>integer</RecordsBatchSize> <!-- Ceph extension --> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> </LoggingEnabled> </BucketLoggingStatus> */ @@ -78,6 +87,7 @@ struct configuration { PartitionDateSource date_source = PartitionDateSource::DeliveryTime; // EventTime: use only year, month, and day. The hour, minutes and seconds are set to 00 in the key // DeliveryTime: the time the log object was created + rgw_s3_key_filter key_filter; bool decode_xml(XMLObj *obj); void dump_xml(Formatter *f) const; void dump(Formatter *f) const; // json @@ -92,6 +102,9 @@ struct configuration { encode(static_cast<int>(logging_type), bl); encode(records_batch_size, bl); encode(static_cast<int>(date_source), bl); + if (logging_type == LoggingType::Journal) { + encode(key_filter, bl); + } ENCODE_FINISH(bl); } @@ -108,6 +121,9 @@ struct configuration { decode(records_batch_size, bl); decode(type, bl); date_source = static_cast<PartitionDateSource>(type); + if (logging_type == LoggingType::Journal) { + decode(key_filter, bl); + } DECODE_FINISH(bl); } }; diff --git a/src/rgw/rgw_cksum_pipe.cc b/src/rgw/rgw_cksum_pipe.cc index e06957e2715..0bec8d341af 100644 --- a/src/rgw/rgw_cksum_pipe.cc +++ b/src/rgw/rgw_cksum_pipe.cc @@ -18,6 +18,7 @@ #include <string> #include <fmt/format.h> #include <boost/algorithm/string.hpp> +#include "rgw_cksum.h" #include "rgw_common.h" #include "common/dout.h" #include "rgw_client_io.h" @@ -34,7 +35,8 @@ namespace rgw::putobj { {} std::unique_ptr<RGWPutObj_Cksum> RGWPutObj_Cksum::Factory( - rgw::sal::DataProcessor* next, const RGWEnv& env) + rgw::sal::DataProcessor* next, const RGWEnv& env, + rgw::cksum::Type override_type) { /* look for matching headers */ auto algo_header = cksum_algorithm_hdr(env); @@ -49,6 +51,13 @@ namespace rgw::putobj { throw rgw::io::Exception(EINVAL, std::system_category()); } /* no checksum header */ + if (override_type != rgw::cksum::Type::none) { + /* XXXX safe? do we need to fixup env as well? */ + auto algo_header = cksum_algorithm_hdr(override_type); + return + std::make_unique<RGWPutObj_Cksum>( + next, override_type, std::move(algo_header)); + } return std::unique_ptr<RGWPutObj_Cksum>(); } diff --git a/src/rgw/rgw_cksum_pipe.h b/src/rgw/rgw_cksum_pipe.h index fddcd283c84..c459d156335 100644 --- a/src/rgw/rgw_cksum_pipe.h +++ b/src/rgw/rgw_cksum_pipe.h @@ -20,6 +20,7 @@ #include <tuple> #include <cstring> #include <boost/algorithm/string/case_conv.hpp> +#include "rgw_cksum.h" #include "rgw_cksum_digest.h" #include "rgw_common.h" #include "rgw_putobj.h" @@ -29,6 +30,38 @@ namespace rgw::putobj { namespace cksum = rgw::cksum; using cksum_hdr_t = std::pair<const char*, const char*>; + static inline const cksum_hdr_t cksum_algorithm_hdr(rgw::cksum::Type t) { + static constexpr std::string_view hdr = + "HTTP_X_AMZ_SDK_CHECKSUM_ALGORITHM"; + using rgw::cksum::Type; + switch (t) { + case Type::sha256: + return cksum_hdr_t(hdr.data(), "SHA256"); + break; + case Type::crc32: + return cksum_hdr_t(hdr.data(), "CRC32"); + break; + case Type::crc32c: + return cksum_hdr_t(hdr.data(), "CRC32C"); + break; + case Type::xxh3: + return cksum_hdr_t(hdr.data(), "XX3"); + break; + case Type::sha1: + return cksum_hdr_t(hdr.data(), "SHA1"); + break; + case Type::sha512: + return cksum_hdr_t(hdr.data(), "SHA512"); + break; + case Type::blake3: + return cksum_hdr_t(hdr.data(), "BLAKE3"); + break; + default: + break; + }; + return cksum_hdr_t(nullptr, nullptr);; + } + static inline const cksum_hdr_t cksum_algorithm_hdr(const RGWEnv& env) { /* If the individual checksum value you provide through x-amz-checksum-algorithm doesn't match the checksum algorithm @@ -102,7 +135,8 @@ namespace rgw::putobj { using VerifyResult = std::tuple<bool, const cksum::Cksum&>; static std::unique_ptr<RGWPutObj_Cksum> Factory( - rgw::sal::DataProcessor* next, const RGWEnv&); + rgw::sal::DataProcessor* next, const RGWEnv&, + rgw::cksum::Type override_type); RGWPutObj_Cksum(rgw::sal::DataProcessor* next, rgw::cksum::Type _type, cksum_hdr_t&& _hdr); diff --git a/src/rgw/rgw_common.cc b/src/rgw/rgw_common.cc index 1a59ba02999..6610538542c 100644 --- a/src/rgw/rgw_common.cc +++ b/src/rgw/rgw_common.cc @@ -63,6 +63,7 @@ rgw_http_errors rgw_http_s3_errors({ { ERR_INVALID_DIGEST, {400, "InvalidDigest" }}, { ERR_BAD_DIGEST, {400, "BadDigest" }}, { ERR_INVALID_LOCATION_CONSTRAINT, {400, "InvalidLocationConstraint" }}, + { ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION, {400, "IllegalLocationConstraintException" }}, { ERR_ZONEGROUP_DEFAULT_PLACEMENT_MISCONFIGURATION, {400, "ZonegroupDefaultPlacementMisconfiguration" }}, { ERR_INVALID_BUCKET_NAME, {400, "InvalidBucketName" }}, { ERR_INVALID_OBJECT_NAME, {400, "InvalidObjectName" }}, @@ -3206,3 +3207,14 @@ void RGWObjVersionTracker::generate_new_write_ver(CephContext *cct) append_rand_alpha(cct, write_version.tag, write_version.tag, TAG_LEN); } +boost::optional<rgw::IAM::Policy> +get_iam_policy_from_attr(CephContext* cct, + const std::map<std::string, bufferlist>& attrs, + const std::string& tenant) +{ + if (auto i = attrs.find(RGW_ATTR_IAM_POLICY); i != attrs.end()) { + return Policy(cct, &tenant, i->second.to_str(), false); + } else { + return boost::none; + } +} diff --git a/src/rgw/rgw_common.h b/src/rgw/rgw_common.h index 8dafe147509..d2917838f36 100644 --- a/src/rgw/rgw_common.h +++ b/src/rgw/rgw_common.h @@ -337,6 +337,7 @@ inline constexpr const char* RGW_REST_STS_XMLNS = #define ERR_PRESIGNED_URL_EXPIRED 2223 #define ERR_PRESIGNED_URL_DISABLED 2224 #define ERR_AUTHORIZATION 2225 // SNS 403 AuthorizationError +#define ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION 2226 #define ERR_BUSY_RESHARDING 2300 // also in cls_rgw_types.h, don't change! #define ERR_NO_SUCH_ENTITY 2301 @@ -1747,24 +1748,22 @@ rgw::IAM::Effect evaluate_iam_policies( bool verify_user_permission(const DoutPrefixProvider* dpp, req_state * const s, - const RGWAccessControlPolicy& user_acl, - const std::vector<rgw::IAM::Policy>& user_policies, - const std::vector<rgw::IAM::Policy>& session_policies, - const rgw::ARN& res, - const uint64_t op, - bool mandatory_policy=true); -bool verify_user_permission_no_policy(const DoutPrefixProvider* dpp, - req_state * const s, - const RGWAccessControlPolicy& user_acl, - const int perm); -bool verify_user_permission(const DoutPrefixProvider* dpp, - req_state * const s, const rgw::ARN& res, const uint64_t op, bool mandatory_policy=true); bool verify_user_permission_no_policy(const DoutPrefixProvider* dpp, req_state * const s, int perm); +bool verify_bucket_permission(const DoutPrefixProvider* dpp, + struct perm_state_base * const s, + const rgw::ARN& arn, + bool account_root, + const RGWAccessControlPolicy& user_acl, + const RGWAccessControlPolicy& bucket_acl, + const boost::optional<rgw::IAM::Policy>& bucket_policy, + const std::vector<rgw::IAM::Policy>& identity_policies, + const std::vector<rgw::IAM::Policy>& session_policies, + const uint64_t op); bool verify_bucket_permission( const DoutPrefixProvider* dpp, req_state * const s, @@ -2012,3 +2011,8 @@ struct AioCompletionDeleter { void operator()(librados::AioCompletion* c) { c->release(); } }; using aio_completion_ptr = std::unique_ptr<librados::AioCompletion, AioCompletionDeleter>; + +extern boost::optional<rgw::IAM::Policy> +get_iam_policy_from_attr(CephContext* cct, + const std::map<std::string, bufferlist>& attrs, + const std::string& tenant); diff --git a/src/rgw/rgw_file_int.h b/src/rgw/rgw_file_int.h index 0a1db645207..84eff1e252e 100644 --- a/src/rgw/rgw_file_int.h +++ b/src/rgw/rgw_file_int.h @@ -2298,6 +2298,8 @@ public: std::string uri; std::map<std::string, buffer::list> attrs; RGWLibFS::BucketStats& bs; + real_time ctime; + bool name_matched = false; RGWStatBucketRequest(CephContext* _cct, std::unique_ptr<rgw::sal::User> _user, const std::string& _path, @@ -2312,9 +2314,7 @@ public: return (iter != attrs.end()) ? &(iter->second) : nullptr; } - real_time get_ctime() const { - return bucket->get_creation_time(); - } + real_time get_ctime() { return ctime; } bool only_bucket() override { return false; } @@ -2342,22 +2342,26 @@ public: return 0; } - virtual int get_params() { - return 0; + int get_params(optional_yield) override { return 0; } + + void complete() override { + // get_state() will no longer be there after execute_req() + // so save what we need from get_state()->bucket + ctime = get_state()->bucket->get_creation_time(); + name_matched = get_state()->bucket->get_name().length() > 0; + + RGWOp::complete(); } void send_response() override { - bucket->get_creation_time() = get_state()->bucket->get_info().creation_time; bs.size = stats.size; bs.size_rounded = stats.size_rounded; - bs.creation_time = bucket->get_creation_time(); + bs.creation_time = get_state()->bucket->get_info().creation_time; bs.num_entries = stats.num_objects; std::swap(attrs, get_state()->bucket_attrs); } - bool matched() { - return (bucket->get_name().length() > 0); - } + bool matched() { return name_matched; } }; /* RGWStatBucketRequest */ diff --git a/src/rgw/rgw_iam_policy.cc b/src/rgw/rgw_iam_policy.cc index ce76ed4c3c3..ef6761d4222 100644 --- a/src/rgw/rgw_iam_policy.cc +++ b/src/rgw/rgw_iam_policy.cc @@ -94,6 +94,8 @@ static const actpair actpairs[] = { "s3:GetPublicAccessBlock", s3GetPublicAccessBlock }, { "s3:GetObjectAcl", s3GetObjectAcl }, { "s3:GetObject", s3GetObject }, + { "s3:GetObjectAttributes", s3GetObjectAttributes }, + { "s3:GetObjectVersionAttributes", s3GetObjectVersionAttributes }, { "s3:GetObjectTorrent", s3GetObjectTorrent }, { "s3:GetObjectVersionAcl", s3GetObjectVersionAcl }, { "s3:GetObjectVersion", s3GetObjectVersion }, @@ -113,6 +115,7 @@ static const actpair actpairs[] = { "s3:PutBucketCORS", s3PutBucketCORS }, { "s3:PutBucketEncryption", s3PutBucketEncryption }, { "s3:PutBucketLogging", s3PutBucketLogging }, + { "s3:PostBucketLogging", s3PostBucketLogging }, { "s3:PutBucketNotification", s3PutBucketNotification }, { "s3:PutBucketOwnershipControls", s3PutBucketOwnershipControls }, { "s3:PutBucketPolicy", s3PutBucketPolicy }, @@ -1334,6 +1337,7 @@ const char* action_bit_string(uint64_t action) { case s3ListBucketVersions: return "s3:ListBucketVersions"; + case s3ListAllMyBuckets: return "s3:ListAllMyBuckets"; @@ -1406,6 +1410,9 @@ const char* action_bit_string(uint64_t action) { case s3PutBucketLogging: return "s3:PutBucketLogging"; + case s3PostBucketLogging: + return "s3:PostBucketLogging"; + case s3GetBucketTagging: return "s3:GetBucketTagging"; @@ -1475,6 +1482,12 @@ const char* action_bit_string(uint64_t action) { case s3BypassGovernanceRetention: return "s3:BypassGovernanceRetention"; + case s3GetObjectAttributes: + return "s3:GetObjectAttributes"; + + case s3GetObjectVersionAttributes: + return "s3:GetObjectVersionAttributes"; + case s3DescribeJob: return "s3:DescribeJob"; diff --git a/src/rgw/rgw_iam_policy.h b/src/rgw/rgw_iam_policy.h index 1494cbf0b81..dd323ee4b9c 100644 --- a/src/rgw/rgw_iam_policy.h +++ b/src/rgw/rgw_iam_policy.h @@ -81,6 +81,7 @@ enum { s3PutBucketNotification, s3GetBucketLogging, s3PutBucketLogging, + s3PostBucketLogging, s3GetBucketTagging, s3PutBucketTagging, s3GetBucketWebsite, @@ -114,6 +115,8 @@ enum { s3GetBucketEncryption, s3PutBucketEncryption, s3DescribeJob, + s3GetObjectAttributes, + s3GetObjectVersionAttributes, s3All, s3objectlambdaGetObject, @@ -246,6 +249,8 @@ inline int op_to_perm(std::uint64_t op) { case s3GetObjectVersionTagging: case s3GetObjectRetention: case s3GetObjectLegalHold: + case s3GetObjectAttributes: + case s3GetObjectVersionAttributes: case s3ListAllMyBuckets: case s3ListBucket: case s3ListBucketMultipartUploads: @@ -298,6 +303,7 @@ inline int op_to_perm(std::uint64_t op) { case s3PutBucketCORS: case s3PutBucketEncryption: case s3PutBucketLogging: + case s3PostBucketLogging: case s3PutBucketNotification: case s3PutBucketPolicy: case s3PutBucketRequestPayment: diff --git a/src/rgw/rgw_kafka.cc b/src/rgw/rgw_kafka.cc index 0807993338d..b38b1a78ec4 100644 --- a/src/rgw/rgw_kafka.cc +++ b/src/rgw/rgw_kafka.cc @@ -13,6 +13,7 @@ #include <thread> #include <atomic> #include <mutex> +#include <boost/algorithm/string.hpp> #include <boost/functional/hash.hpp> #include <boost/lockfree/queue.hpp> #include "common/dout.h" @@ -595,7 +596,8 @@ public: boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> topic_user_name, - boost::optional<const std::string&> topic_password) { + boost::optional<const std::string&> topic_password, + boost::optional<const std::string&> brokers) { if (stopped) { ldout(cct, 1) << "Kafka connect: manager is stopped" << dendl; return false; @@ -603,8 +605,8 @@ public: std::string user; std::string password; - std::string broker; - if (!parse_url_authority(url, broker, user, password)) { + std::string broker_list; + if (!parse_url_authority(url, broker_list, user, password)) { // TODO: increment counter ldout(cct, 1) << "Kafka connect: URL parsing failed" << dendl; return false; @@ -632,7 +634,13 @@ public: ldout(cct, 1) << "Kafka connect: user/password are only allowed over secure connection" << dendl; return false; } - connection_id_t tmp_id(broker, user, password, ca_location, mechanism, + + if (brokers.has_value()) { + broker_list.append(","); + broker_list.append(brokers.get()); + } + + connection_id_t tmp_id(broker_list, user, password, ca_location, mechanism, use_ssl); std::lock_guard lock(connections_lock); const auto it = connections.find(tmp_id); @@ -652,7 +660,7 @@ public: return false; } - auto conn = std::make_unique<connection_t>(cct, broker, use_ssl, verify_ssl, ca_location, user, password, mechanism); + auto conn = std::make_unique<connection_t>(cct, broker_list, use_ssl, verify_ssl, ca_location, user, password, mechanism); if (!new_producer(conn.get())) { ldout(cct, 10) << "Kafka connect: producer creation failed in new connection" << dendl; return false; @@ -770,11 +778,12 @@ bool connect(connection_id_t& conn_id, boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> user_name, - boost::optional<const std::string&> password) { + boost::optional<const std::string&> password, + boost::optional<const std::string&> brokers) { std::shared_lock lock(s_manager_mutex); if (!s_manager) return false; return s_manager->connect(conn_id, url, use_ssl, verify_ssl, ca_location, - mechanism, user_name, password); + mechanism, user_name, password, brokers); } int publish(const connection_id_t& conn_id, diff --git a/src/rgw/rgw_kafka.h b/src/rgw/rgw_kafka.h index b7aa0d15759..858b185219f 100644 --- a/src/rgw/rgw_kafka.h +++ b/src/rgw/rgw_kafka.h @@ -48,7 +48,8 @@ bool connect(connection_id_t& conn_id, boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> user_name, - boost::optional<const std::string&> password); + boost::optional<const std::string&> password, + boost::optional<const std::string&> brokers); // publish a message over a connection that was already created int publish(const connection_id_t& conn_id, diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc index 8378d9ef22e..7b0ca3134a3 100644 --- a/src/rgw/rgw_op.cc +++ b/src/rgw/rgw_op.cc @@ -25,8 +25,10 @@ #include "common/ceph_json.h" #include "common/static_ptr.h" #include "common/perf_counters_key.h" +#include "rgw_cksum.h" #include "rgw_cksum_digest.h" #include "rgw_common.h" +#include "common/split.h" #include "rgw_tracer.h" #include "rgw_rados.h" @@ -331,19 +333,6 @@ static int get_obj_policy_from_attr(const DoutPrefixProvider *dpp, return ret; } - -static boost::optional<Policy> -get_iam_policy_from_attr(CephContext* cct, - const map<string, bufferlist>& attrs, - const string& tenant) -{ - if (auto i = attrs.find(RGW_ATTR_IAM_POLICY); i != attrs.end()) { - return Policy(cct, &tenant, i->second.to_str(), false); - } else { - return none; - } -} - static boost::optional<PublicAccessBlockConfiguration> get_public_access_conf_from_attr(const map<string, bufferlist>& attrs) { @@ -1350,9 +1339,9 @@ void RGWDeleteBucketTags::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this, y] { - rgw::sal::Attrs attrs = s->bucket->get_attrs(); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_TAGS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, y); + op_ret = s->bucket->put_info(this, false, real_time(), y); if (op_ret < 0) { ldpp_dout(this, 0) << "RGWDeleteBucketTags() failed to remove RGW_ATTR_TAGS on bucket=" << s->bucket->get_name() @@ -3135,17 +3124,19 @@ static int load_bucket_stats(const DoutPrefixProvider* dpp, optional_yield y, void RGWStatBucket::execute(optional_yield y) { - if (!s->bucket_exists) { - op_ret = -ERR_NO_SUCH_BUCKET; + op_ret = get_params(y); + if (op_ret < 0) { return; } - op_ret = driver->load_bucket(this, s->bucket->get_key(), &bucket, y); - if (op_ret) { + if (!s->bucket_exists) { + op_ret = -ERR_NO_SUCH_BUCKET; return; } - op_ret = load_bucket_stats(this, y, *s->bucket, stats); + if (report_stats) { + op_ret = load_bucket_stats(this, y, *s->bucket, stats); + } } int RGWListBucket::verify_permission(optional_yield y) @@ -3569,54 +3560,62 @@ void RGWCreateBucket::execute(optional_yield y) const rgw::SiteConfig& site = *s->penv.site; const std::optional<RGWPeriod>& period = site.get_period(); const RGWZoneGroup& my_zonegroup = site.get_zonegroup(); - - if (s->system_request) { - // allow system requests to override the target zonegroup. for forwarded - // requests, we'll create the bucket for the originating zonegroup - createparams.zonegroup_id = s->info.args.get(RGW_SYS_PARAM_PREFIX "zonegroup"); - } - + const std::string rgwx_zonegroup = s->info.args.get(RGW_SYS_PARAM_PREFIX "zonegroup"); const RGWZoneGroup* bucket_zonegroup = &my_zonegroup; - if (createparams.zonegroup_id.empty()) { - // default to the local zonegroup - createparams.zonegroup_id = my_zonegroup.id; - } else if (period) { - auto z = period->period_map.zonegroups.find(createparams.zonegroup_id); - if (z == period->period_map.zonegroups.end()) { - ldpp_dout(this, 0) << "could not find zonegroup " - << createparams.zonegroup_id << " in current period" << dendl; - op_ret = -ENOENT; - return; - } - bucket_zonegroup = &z->second; - } else if (createparams.zonegroup_id != my_zonegroup.id) { - ldpp_dout(this, 0) << "zonegroup does not match current zonegroup " - << createparams.zonegroup_id << dendl; - op_ret = -ENOENT; - return; - } - // validate the LocationConstraint + // Validate LocationConstraint if it's provided and enforcement is strict if (!location_constraint.empty() && !relaxed_region_enforcement) { - // on the master zonegroup, allow any valid api_name. otherwise it has to - // match the bucket's zonegroup - if (period && my_zonegroup.is_master) { - if (!period->period_map.zonegroups_by_api.count(location_constraint)) { + if (period) { + auto location_iter = period->period_map.zonegroups_by_api.find(location_constraint); + if (location_iter == period->period_map.zonegroups_by_api.end()) { ldpp_dout(this, 0) << "location constraint (" << location_constraint << ") can't be found." << dendl; op_ret = -ERR_INVALID_LOCATION_CONSTRAINT; - s->err.message = "The specified location-constraint is not valid"; + s->err.message = fmt::format("The {} location constraint is not valid.", + location_constraint); return; } - } else if (bucket_zonegroup->api_name != location_constraint) { + bucket_zonegroup = &location_iter->second; + } else if (location_constraint != my_zonegroup.api_name) { // if we don't have a period, we can only use the current zonegroup - so check if the location matches by api name here ldpp_dout(this, 0) << "location constraint (" << location_constraint - << ") doesn't match zonegroup (" << bucket_zonegroup->api_name - << ')' << dendl; - op_ret = -ERR_INVALID_LOCATION_CONSTRAINT; - s->err.message = "The specified location-constraint is not valid"; + << ") doesn't match zonegroup (" << my_zonegroup.api_name << ")" << dendl; + op_ret = -ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION; + s->err.message = fmt::format("The {} location constraint is incompatible " + "for the region specific endpoint this request was sent to.", + location_constraint); return; } } + // If it's a system request, use the provided zonegroup if available + else if (s->system_request && !rgwx_zonegroup.empty()) { + if (period) { + auto zonegroup_iter = period->period_map.zonegroups.find(rgwx_zonegroup); + if (zonegroup_iter == period->period_map.zonegroups.end()) { + ldpp_dout(this, 0) << "could not find zonegroup " << rgwx_zonegroup + << " in current period" << dendl; + op_ret = -ENOENT; + return; + } + bucket_zonegroup = &zonegroup_iter->second; + } + } + + const bool enforce_location_match = + !period || // No period: no multisite, so no need to enforce location match. + !s->system_request || // All user requests are enforced to match zonegroup's location. + !my_zonegroup.is_master; // but if it's a system request (forwarded) only allow remote creation on master zonegroup. + if (enforce_location_match && !my_zonegroup.equals(bucket_zonegroup->get_id())) { + ldpp_dout(this, 0) << "location constraint (" << bucket_zonegroup->api_name + << ") doesn't match zonegroup (" << my_zonegroup.api_name << ")" << dendl; + op_ret = -ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION; + s->err.message = fmt::format("The {} location constraint is incompatible " + "for the region specific endpoint this request was sent to.", + bucket_zonegroup->api_name); + return; + } + + // Set the final zonegroup ID + createparams.zonegroup_id = bucket_zonegroup->id; // select and validate the placement target op_ret = select_bucket_placement(this, *bucket_zonegroup, s->user->get_info(), @@ -3625,7 +3624,7 @@ void RGWCreateBucket::execute(optional_yield y) return; } - if (bucket_zonegroup == &my_zonegroup) { + if (my_zonegroup.equals(bucket_zonegroup->get_id())) { // look up the zone placement pool createparams.zone_placement = rgw::find_zone_placement( this, site.get_zone_params(), createparams.placement_rule); @@ -3714,7 +3713,6 @@ void RGWCreateBucket::execute(optional_yield y) if (!driver->is_meta_master()) { // apply bucket creation on the master zone first - bufferlist in_data; JSONParser jp; op_ret = rgw_forward_request_to_master(this, *s->penv.site, s->owner.id, &in_data, &jp, s->info, y); @@ -3791,7 +3789,10 @@ void RGWCreateBucket::execute(optional_yield y) s->bucket->get_info().has_website = !s->bucket->get_info().website_conf.is_empty(); /* This will also set the quota on the bucket. */ - op_ret = s->bucket->merge_and_store_attrs(this, createparams.attrs, y); + s->bucket->set_attrs(std::move(createparams.attrs)); + constexpr bool exclusive = false; // overwrite + constexpr ceph::real_time no_set_mtime{}; + op_ret = s->bucket->put_info(this, exclusive, no_set_mtime, y); } while (op_ret == -ECANCELED && tries++ < 20); /* Restore the proper return code. */ @@ -4342,6 +4343,9 @@ void RGWPutObj::execute(optional_yield y) } return; } + + multipart_cksum_type = upload->cksum_type; + /* upload will go out of scope, so copy the dest placement for later use */ s->dest_placement = *pdest_placement; pdest_placement = &s->dest_placement; @@ -4472,11 +4476,12 @@ void RGWPutObj::execute(optional_yield y) /* optional streaming checksum */ try { cksum_filter = - rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env); + rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env, multipart_cksum_type); } catch (const rgw::io::Exception& e) { op_ret = -e.code().value(); return; } + if (cksum_filter) { filter = &*cksum_filter; } @@ -4623,10 +4628,12 @@ void RGWPutObj::execute(optional_yield y) if (cksum_filter) { const auto& hdr = cksum_filter->header(); + auto expected_ck = cksum_filter->expected(*s->info.env); auto cksum_verify = cksum_filter->verify(*s->info.env); // valid or no supplied cksum cksum = get<1>(cksum_verify); - if (std::get<0>(cksum_verify)) { + if ((!expected_ck) || + std::get<0>(cksum_verify)) { buffer::list cksum_bl; ldpp_dout_fmt(this, 16, @@ -4634,14 +4641,13 @@ void RGWPutObj::execute(optional_yield y) "\n\tcomputed={} == \n\texpected={}", hdr.second, cksum->to_armor(), - cksum_filter->expected(*s->info.env)); + (!!expected_ck) ? expected_ck : "(checksum unavailable)"); cksum->encode(cksum_bl); emplace_attr(RGW_ATTR_CKSUM, std::move(cksum_bl)); } else { /* content checksum mismatch */ auto computed_ck = cksum->to_armor(); - auto expected_ck = cksum_filter->expected(*s->info.env); ldpp_dout_fmt(this, 4, "{} content checksum mismatch" @@ -4844,7 +4850,8 @@ void RGWPostObj::execute(optional_yield y) /* optional streaming checksum */ try { cksum_filter = - rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env); + rgw::putobj::RGWPutObj_Cksum::Factory( + filter, *s->info.env, rgw::cksum::Type::none /* no override */); } catch (const rgw::io::Exception& e) { op_ret = -e.code().value(); return; @@ -5192,7 +5199,10 @@ void RGWPutMetadataBucket::execute(optional_yield y) /* Setting attributes also stores the provided bucket info. Due * to this fact, the new quota settings can be serialized with * the same call. */ - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + s->bucket->set_attrs(attrs); + constexpr bool exclusive = false; // overwrite + constexpr ceph::real_time no_set_mtime{}; + op_ret = s->bucket->put_info(this, exclusive, no_set_mtime, s->yield); return op_ret; }, y); } @@ -5980,8 +5990,6 @@ void RGWGetACLs::execute(optional_yield y) acls = ss.str(); } - - int RGWPutACLs::verify_permission(optional_yield y) { bool perm; @@ -6003,6 +6011,74 @@ int RGWPutACLs::verify_permission(optional_yield y) return 0; } +uint16_t RGWGetObjAttrs::recognize_attrs(const std::string& hdr, uint16_t deflt) +{ + auto attrs{deflt}; + auto sa = ceph::split(hdr, ","); + for (auto& k : sa) { + if (boost::iequals(k, "etag")) { + attrs |= as_flag(ReqAttributes::Etag); + } + if (boost::iequals(k, "checksum")) { + attrs |= as_flag(ReqAttributes::Checksum); + } + if (boost::iequals(k, "objectparts")) { + attrs |= as_flag(ReqAttributes::ObjectParts); + } + if (boost::iequals(k, "objectsize")) { + attrs |= as_flag(ReqAttributes::ObjectSize); + } + if (boost::iequals(k, "storageclass")) { + attrs |= as_flag(ReqAttributes::StorageClass); + } + } + return attrs; +} /* RGWGetObjAttrs::recognize_attrs */ + +int RGWGetObjAttrs::verify_permission(optional_yield y) +{ + bool perm = false; + auto [has_s3_existing_tag, has_s3_resource_tag] = + rgw_check_policy_condition(this, s); + + if (! rgw::sal::Object::empty(s->object.get())) { + + auto iam_action1 = s->object->get_instance().empty() ? + rgw::IAM::s3GetObject : + rgw::IAM::s3GetObjectVersion; + + auto iam_action2 = s->object->get_instance().empty() ? + rgw::IAM::s3GetObjectAttributes : + rgw::IAM::s3GetObjectVersionAttributes; + + if (has_s3_existing_tag || has_s3_resource_tag) { + rgw_iam_add_objtags(this, s, has_s3_existing_tag, has_s3_resource_tag); + } + + /* XXXX the following conjunction should be &&--but iam_action2 is currently not + * hooked up and always fails (but should succeed if the requestor has READ + * acess to the object) */ + perm = (verify_object_permission(this, s, iam_action1) || /* && */ + verify_object_permission(this, s, iam_action2)); + } + + if (! perm) { + return -EACCES; + } + + return 0; +} + +void RGWGetObjAttrs::pre_exec() +{ + rgw_bucket_object_pre_exec(s); +} + +void RGWGetObjAttrs::execute(optional_yield y) +{ + RGWGetObj::execute(y); +} /* RGWGetObjAttrs::execute */ + int RGWGetLC::verify_permission(optional_yield y) { auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); @@ -6384,9 +6460,9 @@ void RGWDeleteCORS::execute(optional_yield y) return op_ret; } - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_CORS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); if (op_ret < 0) { ldpp_dout(this, 0) << "RGWLC::RGWDeleteCORS() failed to set attrs on bucket=" << s->bucket->get_name() << " returned err=" << op_ret << dendl; @@ -6670,6 +6746,14 @@ try_sum_part_cksums(const DoutPrefixProvider *dpp, ++parts_ix; auto& part_cksum = part.second->get_cksum(); + if (! part_cksum) { + ldpp_dout_fmt(dpp, 0, + "ERROR: multipart part checksum not present (ix=={})", + parts_ix); + op_ret = -ERR_INVALID_REQUEST; + return op_ret; + } + ldpp_dout_fmt(dpp, 16, "INFO: {} iterate part: {} {} {}", __func__, parts_ix, part_cksum->type_string(), @@ -7040,17 +7124,30 @@ void RGWAbortMultipart::execute(optional_yield y) return; upload = s->bucket->get_multipart_upload(s->object->get_name(), upload_id); + meta_obj = upload->get_meta_obj(); + meta_obj->set_in_extra_data(true); + meta_obj->get_obj_attrs(s->yield, this); + jspan_context trace_ctx(false, false); if (tracing::rgw::tracer.is_enabled()) { // read meta object attributes for trace info - meta_obj = upload->get_meta_obj(); - meta_obj->set_in_extra_data(true); - meta_obj->get_obj_attrs(s->yield, this); extract_span_context(meta_obj->get_attrs(), trace_ctx); } multipart_trace = tracing::rgw::tracer.add_span(name(), trace_ctx); + int max_lock_secs_mp = + s->cct->_conf.get_val<int64_t>("rgw_mp_lock_max_time"); + utime_t dur(max_lock_secs_mp, 0); + auto serializer = meta_obj->get_serializer(this, "RGWCompleteMultipart"); + op_ret = serializer->try_lock(this, dur, y); + if (op_ret < 0) { + if (op_ret == -ENOENT) { + op_ret = -ERR_NO_SUCH_UPLOAD; + } + return; + } op_ret = upload->abort(this, s->cct, y); + serializer->unlock(); } int RGWListMultipart::verify_permission(optional_yield y) @@ -7354,6 +7451,12 @@ void RGWDeleteMultiObj::execute(optional_yield y) return; } + if (multi_delete->objects.empty()) { + s->err.message = "Missing required element Object"; + op_ret = -ERR_MALFORMED_XML; + return; + } + constexpr int DEFAULT_MAX_NUM = 1000; int max_num = s->cct->_conf->rgw_delete_multi_obj_max_num; if (max_num < 0) { @@ -8509,9 +8612,9 @@ void RGWDeleteBucketPolicy::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this] { - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_IAM_POLICY); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); return op_ret; }, y); } @@ -9029,9 +9132,9 @@ void RGWDeleteBucketPublicAccessBlock::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this] { - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_PUBLIC_ACCESS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); return op_ret; }, y); } @@ -9140,10 +9243,10 @@ void RGWDeleteBucketEncryption::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this, y] { - rgw::sal::Attrs attrs = s->bucket->get_attrs(); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_BUCKET_ENCRYPTION_POLICY); attrs.erase(RGW_ATTR_BUCKET_ENCRYPTION_KEY_ID); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, y); + op_ret = s->bucket->put_info(this, false, real_time(), y); return op_ret; }, y); } diff --git a/src/rgw/rgw_op.h b/src/rgw/rgw_op.h index 0098b67e032..dcf64c31572 100644 --- a/src/rgw/rgw_op.h +++ b/src/rgw/rgw_op.h @@ -12,6 +12,7 @@ #pragma once +#include <cstdint> #include <limits.h> #include <array> @@ -1087,14 +1088,15 @@ public: class RGWStatBucket : public RGWOp { protected: - std::unique_ptr<rgw::sal::Bucket> bucket; RGWStorageStats stats; + bool report_stats{true}; public: int verify_permission(optional_yield y) override; void pre_exec() override; void execute(optional_yield y) override; + virtual int get_params(optional_yield y) = 0; void send_response() override = 0; const char* name() const override { return "stat_bucket"; } RGWOpType get_type() override { return RGW_OP_STAT_BUCKET; } @@ -1110,6 +1112,7 @@ class RGWCreateBucket : public RGWOp { bool relaxed_region_enforcement = false; RGWCORSConfiguration cors_config; std::set<std::string> rmattr_names; + bufferlist in_data; virtual bool need_metadata_upload() const { return false; } @@ -1236,6 +1239,7 @@ protected: std::string multipart_upload_id; std::string multipart_part_str; int multipart_part_num = 0; + rgw::cksum::Type multipart_cksum_type{rgw::cksum::Type::none}; jspan_ptr multipart_trace; boost::optional<ceph::real_time> delete_at; @@ -1643,6 +1647,50 @@ public: uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; } }; +class RGWGetObjAttrs : public RGWGetObj { +protected: + std::string version_id; + std::string expected_bucket_owner; + std::optional<int> marker; + std::optional<int> max_parts; + uint16_t requested_attributes{0}; +#if 0 + /* used to decrypt attributes for objects stored with SSE-C */ + x-amz-server-side-encryption-customer-algorithm + x-amz-server-side-encryption-customer-key + x-amz-server-side-encryption-customer-key-MD5 +#endif +public: + + enum class ReqAttributes : uint16_t { + None = 0, + Etag, + Checksum, + ObjectParts, + StorageClass, + ObjectSize + }; + + static uint16_t as_flag(ReqAttributes attr) { + return 1 << (uint16_t(attr) ? uint16_t(attr) - 1 : 0); + } + + static uint16_t recognize_attrs(const std::string& hdr, uint16_t deflt = 0); + + RGWGetObjAttrs() : RGWGetObj() + { + RGWGetObj::get_data = false; // it's extra false + } + + int verify_permission(optional_yield y) override; + void pre_exec() override; + void execute(optional_yield y) override; + void send_response() override = 0; + const char* name() const override { return "get_obj_attrs"; } + RGWOpType get_type() override { return RGW_OP_GET_OBJ_ATTRS; } + uint32_t op_mask() override { return RGW_OP_TYPE_READ; } +}; /* RGWGetObjAttrs */ + class RGWGetLC : public RGWOp { protected: diff --git a/src/rgw/rgw_op_type.h b/src/rgw/rgw_op_type.h index a36274add40..2c8225d289e 100644 --- a/src/rgw/rgw_op_type.h +++ b/src/rgw/rgw_op_type.h @@ -30,6 +30,7 @@ enum RGWOpType { RGW_OP_COPY_OBJ, RGW_OP_GET_ACLS, RGW_OP_PUT_ACLS, + RGW_OP_GET_OBJ_ATTRS, RGW_OP_GET_CORS, RGW_OP_PUT_CORS, RGW_OP_DELETE_CORS, @@ -117,6 +118,7 @@ enum RGWOpType { RGW_OP_DETACH_GROUP_POLICY, RGW_OP_LIST_ATTACHED_GROUP_POLICIES, RGW_OP_PUT_BUCKET_LOGGING, + RGW_OP_POST_BUCKET_LOGGING, /* rgw specific */ RGW_OP_ADMIN_SET_METADATA, RGW_OP_GET_OBJ_LAYOUT, diff --git a/src/rgw/rgw_pubsub.cc b/src/rgw/rgw_pubsub.cc index cb68d72d7da..87a46bd61a6 100644 --- a/src/rgw/rgw_pubsub.cc +++ b/src/rgw/rgw_pubsub.cc @@ -62,214 +62,6 @@ void set_event_id(std::string& id, const std::string& hash, const utime_t& ts) { } } -void rgw_s3_key_filter::dump(Formatter *f) const { - if (!has_content()) { - return; - } - f->open_array_section("FilterRules"); - if (!prefix_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "prefix", f); - ::encode_json("Value", prefix_rule, f); - f->close_section(); - } - if (!suffix_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "suffix", f); - ::encode_json("Value", suffix_rule, f); - f->close_section(); - } - if (!regex_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "regex", f); - ::encode_json("Value", regex_rule, f); - f->close_section(); - } - f->close_section(); -} - -bool rgw_s3_key_filter::decode_xml(XMLObj* obj) { - XMLObjIter iter = obj->find("FilterRule"); - XMLObj *o; - - const auto throw_if_missing = true; - auto prefix_not_set = true; - auto suffix_not_set = true; - auto regex_not_set = true; - std::string name; - - while ((o = iter.get_next())) { - RGWXMLDecoder::decode_xml("Name", name, o, throw_if_missing); - if (name == "prefix" && prefix_not_set) { - prefix_not_set = false; - RGWXMLDecoder::decode_xml("Value", prefix_rule, o, throw_if_missing); - } else if (name == "suffix" && suffix_not_set) { - suffix_not_set = false; - RGWXMLDecoder::decode_xml("Value", suffix_rule, o, throw_if_missing); - } else if (name == "regex" && regex_not_set) { - regex_not_set = false; - RGWXMLDecoder::decode_xml("Value", regex_rule, o, throw_if_missing); - } else { - throw RGWXMLDecoder::err("invalid/duplicate S3Key filter rule name: '" + name + "'"); - } - } - return true; -} - -void rgw_s3_key_filter::dump_xml(Formatter *f) const { - if (!prefix_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "prefix", f); - ::encode_xml("Value", prefix_rule, f); - f->close_section(); - } - if (!suffix_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "suffix", f); - ::encode_xml("Value", suffix_rule, f); - f->close_section(); - } - if (!regex_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "regex", f); - ::encode_xml("Value", regex_rule, f); - f->close_section(); - } -} - -bool rgw_s3_key_filter::has_content() const { - return !(prefix_rule.empty() && suffix_rule.empty() && regex_rule.empty()); -} - -void rgw_s3_key_value_filter::dump(Formatter *f) const { - if (!has_content()) { - return; - } - f->open_array_section("FilterRules"); - for (const auto& key_value : kv) { - f->open_object_section(""); - ::encode_json("Name", key_value.first, f); - ::encode_json("Value", key_value.second, f); - f->close_section(); - } - f->close_section(); -} - -bool rgw_s3_key_value_filter::decode_xml(XMLObj* obj) { - kv.clear(); - XMLObjIter iter = obj->find("FilterRule"); - XMLObj *o; - - const auto throw_if_missing = true; - - std::string key; - std::string value; - - while ((o = iter.get_next())) { - RGWXMLDecoder::decode_xml("Name", key, o, throw_if_missing); - RGWXMLDecoder::decode_xml("Value", value, o, throw_if_missing); - kv.emplace(key, value); - } - return true; -} - -void rgw_s3_key_value_filter::dump_xml(Formatter *f) const { - for (const auto& key_value : kv) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", key_value.first, f); - ::encode_xml("Value", key_value.second, f); - f->close_section(); - } -} - -bool rgw_s3_key_value_filter::has_content() const { - return !kv.empty(); -} - -void rgw_s3_filter::dump(Formatter *f) const { - encode_json("S3Key", key_filter, f); - encode_json("S3Metadata", metadata_filter, f); - encode_json("S3Tags", tag_filter, f); -} - -bool rgw_s3_filter::decode_xml(XMLObj* obj) { - RGWXMLDecoder::decode_xml("S3Key", key_filter, obj); - RGWXMLDecoder::decode_xml("S3Metadata", metadata_filter, obj); - RGWXMLDecoder::decode_xml("S3Tags", tag_filter, obj); - return true; -} - -void rgw_s3_filter::dump_xml(Formatter *f) const { - if (key_filter.has_content()) { - ::encode_xml("S3Key", key_filter, f); - } - if (metadata_filter.has_content()) { - ::encode_xml("S3Metadata", metadata_filter, f); - } - if (tag_filter.has_content()) { - ::encode_xml("S3Tags", tag_filter, f); - } -} - -bool rgw_s3_filter::has_content() const { - return key_filter.has_content() || - metadata_filter.has_content() || - tag_filter.has_content(); -} - -bool match(const rgw_s3_key_filter& filter, const std::string& key) { - const auto key_size = key.size(); - const auto prefix_size = filter.prefix_rule.size(); - if (prefix_size != 0) { - // prefix rule exists - if (prefix_size > key_size) { - // if prefix is longer than key, we fail - return false; - } - if (!std::equal(filter.prefix_rule.begin(), filter.prefix_rule.end(), key.begin())) { - return false; - } - } - const auto suffix_size = filter.suffix_rule.size(); - if (suffix_size != 0) { - // suffix rule exists - if (suffix_size > key_size) { - // if suffix is longer than key, we fail - return false; - } - if (!std::equal(filter.suffix_rule.begin(), filter.suffix_rule.end(), (key.end() - suffix_size))) { - return false; - } - } - if (!filter.regex_rule.empty()) { - // TODO add regex chaching in the filter - const std::regex base_regex(filter.regex_rule); - if (!std::regex_match(key, base_regex)) { - return false; - } - } - return true; -} - -bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv) { - // all filter pairs must exist with the same value in the object's metadata/tags - // object metadata/tags may include items not in the filter - return std::includes(kv.begin(), kv.end(), filter.kv.begin(), filter.kv.end()); -} - -bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv) { - // all filter pairs must exist with the same value in the object's metadata/tags - // object metadata/tags may include items not in the filter - for (auto& filter : filter.kv) { - auto result = kv.equal_range(filter.first); - if (std::any_of(result.first, result.second, [&filter](const std::pair<std::string, std::string>& p) { return p.second == filter.second;})) - continue; - else - return false; - } - return true; -} - bool match(const rgw::notify::EventTypeList& events, rgw::notify::EventType event) { // if event list exists, and none of the events in the list matches the event type, filter the message if (!events.empty() && std::find(events.begin(), events.end(), event) == events.end()) { diff --git a/src/rgw/rgw_pubsub.h b/src/rgw/rgw_pubsub.h index 8a6b290cb85..176ada95204 100644 --- a/src/rgw/rgw_pubsub.h +++ b/src/rgw/rgw_pubsub.h @@ -9,94 +9,10 @@ #include "rgw_zone.h" #include "rgw_notify_event_type.h" #include <boost/container/flat_map.hpp> +#include "rgw_s3_filter.h" class XMLObj; -struct rgw_s3_key_filter { - std::string prefix_rule; - std::string suffix_rule; - std::string regex_rule; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(1, 1, bl); - encode(prefix_rule, bl); - encode(suffix_rule, bl); - encode(regex_rule, bl); - ENCODE_FINISH(bl); - } - - void decode(bufferlist::const_iterator& bl) { - DECODE_START(1, bl); - decode(prefix_rule, bl); - decode(suffix_rule, bl); - decode(regex_rule, bl); - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_key_filter) - -using KeyValueMap = boost::container::flat_map<std::string, std::string>; -using KeyMultiValueMap = std::multimap<std::string, std::string>; - -struct rgw_s3_key_value_filter { - KeyValueMap kv; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(1, 1, bl); - encode(kv, bl); - ENCODE_FINISH(bl); - } - void decode(bufferlist::const_iterator& bl) { - DECODE_START(1, bl); - decode(kv, bl); - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_key_value_filter) - -struct rgw_s3_filter { - rgw_s3_key_filter key_filter; - rgw_s3_key_value_filter metadata_filter; - rgw_s3_key_value_filter tag_filter; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(2, 1, bl); - encode(key_filter, bl); - encode(metadata_filter, bl); - encode(tag_filter, bl); - ENCODE_FINISH(bl); - } - - void decode(bufferlist::const_iterator& bl) { - DECODE_START(2, bl); - decode(key_filter, bl); - decode(metadata_filter, bl); - if (struct_v >= 2) { - decode(tag_filter, bl); - } - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_filter) - using OptionalFilter = std::optional<rgw_s3_filter>; struct rgw_pubsub_topic_filter; diff --git a/src/rgw/rgw_rest.cc b/src/rgw/rgw_rest.cc index 03241b7369a..ac5e65c0dd6 100644 --- a/src/rgw/rgw_rest.cc +++ b/src/rgw/rgw_rest.cc @@ -666,8 +666,10 @@ static void build_redirect_url(req_state *s, const string& redirect_base, string dest_uri = dest_uri.substr(0, dest_uri.size() - 1); } dest_uri += s->info.request_uri; - dest_uri += "?"; - dest_uri += s->info.request_params; + if (!s->info.request_params.empty()) { + dest_uri += "?"; + dest_uri += s->info.request_params; + } } void abort_early(req_state *s, RGWOp* op, int err_no, @@ -1668,7 +1670,6 @@ int RGWDeleteMultiObj_ObjStore::get_params(optional_yield y) return op_ret; } - void RGWRESTOp::send_response() { if (!flusher.did_start()) { diff --git a/src/rgw/rgw_rest.h b/src/rgw/rgw_rest.h index aa33080af56..9111696453e 100644 --- a/src/rgw/rgw_rest.h +++ b/src/rgw/rgw_rest.h @@ -403,6 +403,17 @@ public: virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); } }; +class RGWGetObjAttrs_ObjStore : public RGWGetObjAttrs { +public: + RGWGetObjAttrs_ObjStore() {} + ~RGWGetObjAttrs_ObjStore() override {} + + int get_params(optional_yield y) = 0; + /* not actually used */ + int send_response_data_error(optional_yield y) override { return 0; }; + int send_response_data(bufferlist& bl, off_t ofs, off_t len) override { return 0; }; +}; + class RGWGetLC_ObjStore : public RGWGetLC { public: RGWGetLC_ObjStore() {} diff --git a/src/rgw/rgw_rest_bucket_logging.cc b/src/rgw/rgw_rest_bucket_logging.cc index 8d17114d804..ed12ce855a9 100644 --- a/src/rgw/rgw_rest_bucket_logging.cc +++ b/src/rgw/rgw_rest_bucket_logging.cc @@ -209,6 +209,84 @@ class RGWPutBucketLoggingOp : public RGWDefaultResponseOp { } }; +// Post /<bucket name>/?logging +// actual configuration is XML encoded in the body of the message +class RGWPostBucketLoggingOp : public RGWDefaultResponseOp { + int verify_permission(optional_yield y) override { + auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); + if (has_s3_resource_tag) + rgw_iam_add_buckettags(this, s); + + if (!verify_bucket_permission(this, s, rgw::IAM::s3PostBucketLogging)) { + return -EACCES; + } + + return 0; + } + + const char* name() const override { return "post_bucket_logging"; } + RGWOpType get_type() override { return RGW_OP_POST_BUCKET_LOGGING; } + uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; } + + void execute(optional_yield y) override { + op_ret = verify_bucket_logging_params(this, s); + if (op_ret < 0) { + return; + } + + std::unique_ptr<rgw::sal::Bucket> bucket; + op_ret = driver->load_bucket(this, rgw_bucket(s->bucket_tenant, s->bucket_name), + &bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get bucket '" << s->bucket_name << "', ret = " << op_ret << dendl; + return; + } + const auto& bucket_attrs = bucket->get_attrs(); + auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (iter == bucket_attrs.end()) { + ldpp_dout(this, 1) << "WARNING: no logging configured on bucket" << dendl; + return; + } + rgw::bucketlogging::configuration configuration; + try { + configuration.enabled = true; + decode(configuration, iter->second); + } catch (buffer::error& err) { + ldpp_dout(this, 1) << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "'. error: " << err.what() << dendl; + op_ret = -EINVAL; + return; + } + + std::unique_ptr<rgw::sal::Bucket> target_bucket; + op_ret = driver->load_bucket(this, rgw_bucket(s->bucket_tenant, configuration.target_bucket), + &target_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get target bucket '" << configuration.target_bucket << "', ret = " << op_ret << dendl; + return; + } + std::string obj_name; + RGWObjVersionTracker objv_tracker; + op_ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, this, &objv_tracker); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get pending logging object name from target bucket '" << configuration.target_bucket << "'" << dendl; + return; + } + op_ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, this, null_yield, true, &objv_tracker); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to flush pending logging object '" << obj_name + << "' to target bucket '" << configuration.target_bucket << "'" << dendl; + return; + } + ldpp_dout(this, 20) << "flushed pending logging object '" << obj_name + << "' to target bucket '" << configuration.target_bucket << "'" << dendl; + } +}; + +RGWOp* RGWHandler_REST_BucketLogging_S3::create_post_op() { + return new RGWPostBucketLoggingOp(); +} + RGWOp* RGWHandler_REST_BucketLogging_S3::create_put_op() { return new RGWPutBucketLoggingOp(); } diff --git a/src/rgw/rgw_rest_bucket_logging.h b/src/rgw/rgw_rest_bucket_logging.h index ca2220084bb..0b31d88dad8 100644 --- a/src/rgw/rgw_rest_bucket_logging.h +++ b/src/rgw/rgw_rest_bucket_logging.h @@ -14,5 +14,6 @@ public: virtual ~RGWHandler_REST_BucketLogging_S3() = default; static RGWOp* create_get_op(); static RGWOp* create_put_op(); + static RGWOp* create_post_op(); }; diff --git a/src/rgw/rgw_rest_pubsub.cc b/src/rgw/rgw_rest_pubsub.cc index adfc86d87cb..f1ffe09cf25 100644 --- a/src/rgw/rgw_rest_pubsub.cc +++ b/src/rgw/rgw_rest_pubsub.cc @@ -234,7 +234,13 @@ bool verify_topic_permission(const DoutPrefixProvider* dpp, req_state* s, return verify_topic_permission(dpp, s, topic.owner, arn, policy, op); } -// command (AWS compliant): +bool should_forward_request_to_master(req_state* s, rgw::sal::Driver* driver) { + return (!driver->is_meta_master() && + rgw::all_zonegroups_support(*s->penv.site, + rgw::zone_features::notification_v2)); +} + +// command (AWS compliant): // POST // Action=CreateTopic&Name=<topic-name>[&OpaqueData=data][&push-endpoint=<endpoint>[&persistent][&<arg1>=<value1>]] class RGWPSCreateTopicOp : public RGWOp { @@ -273,7 +279,7 @@ class RGWPSCreateTopicOp : public RGWOp { // Remove the args that are parsed, so the push_endpoint_args only contains // necessary one's which is parsed after this if. but only if master zone, // else we do not remove as request is forwarded to master. - if (driver->is_meta_master()) { + if (!should_forward_request_to_master(s, driver)) { s->info.args.remove("OpaqueData"); s->info.args.remove("push-endpoint"); s->info.args.remove("persistent"); @@ -396,7 +402,7 @@ class RGWPSCreateTopicOp : public RGWOp { void RGWPSCreateTopicOp::execute(optional_yield y) { // master request will replicate the topic creation. - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); if (op_ret < 0) { @@ -863,7 +869,7 @@ class RGWPSSetTopicAttributesOp : public RGWOp { }; void RGWPSSetTopicAttributesOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); if (op_ret < 0) { @@ -1008,9 +1014,10 @@ class RGWPSDeleteTopicOp : public RGWOp { }; void RGWPSDeleteTopicOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); + if (op_ret < 0) { ldpp_dout(this, 1) << "DeleteTopic forward_request_to_master returned ret = " << op_ret @@ -1260,7 +1267,7 @@ int RGWPSCreateNotifOp::verify_permission(optional_yield y) { } void RGWPSCreateNotifOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &data, nullptr, s->info, y); if (op_ret < 0) { @@ -1462,7 +1469,7 @@ int RGWPSDeleteNotifOp::verify_permission(optional_yield y) { } void RGWPSDeleteNotifOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { bufferlist indata; op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &indata, nullptr, s->info, y); diff --git a/src/rgw/rgw_rest_s3.cc b/src/rgw/rgw_rest_s3.cc index b7046118ef3..885991244a6 100644 --- a/src/rgw/rgw_rest_s3.cc +++ b/src/rgw/rgw_rest_s3.cc @@ -9,6 +9,7 @@ #include <string_view> #include "common/ceph_crypto.h" +#include "common/dout.h" #include "common/split.h" #include "common/Formatter.h" #include "common/utf8.h" @@ -807,7 +808,6 @@ void RGWGetObjTags_ObjStore_S3::send_response_data(bufferlist& bl) } } - int RGWPutObjTags_ObjStore_S3::get_params(optional_yield y) { RGWXMLParser parser; @@ -2401,34 +2401,41 @@ void RGWGetBucketWebsite_ObjStore_S3::send_response() rgw_flush_formatter_and_reset(s, s->formatter); } -static void dump_bucket_metadata(req_state *s, rgw::sal::Bucket* bucket, +static void dump_bucket_metadata(req_state *s, RGWStorageStats& stats) { dump_header(s, "X-RGW-Object-Count", static_cast<long long>(stats.num_objects)); dump_header(s, "X-RGW-Bytes-Used", static_cast<long long>(stats.size)); +} - // only bucket's owner is allowed to get the quota settings of the account - if (s->auth.identity->is_owner_of(bucket->get_owner())) { - const auto& user_info = s->user->get_info(); - const auto& bucket_quota = s->bucket->get_info().quota; // bucket quota - dump_header(s, "X-RGW-Quota-Max-Buckets", static_cast<long long>(user_info.max_buckets)); - - if (user_info.quota.user_quota.enabled){ - dump_header(s, "X-RGW-Quota-User-Size", static_cast<long long>(user_info.quota.user_quota.max_size)); - dump_header(s, "X-RGW-Quota-User-Objects", static_cast<long long>(user_info.quota.user_quota.max_objects)); - } +int RGWStatBucket_ObjStore_S3::get_params(optional_yield y) +{ + report_stats = s->info.args.exists("read-stats"); - if (bucket_quota.enabled){ - dump_header(s, "X-RGW-Quota-Bucket-Size", static_cast<long long>(bucket_quota.max_size)); - dump_header(s, "X-RGW-Quota-Bucket-Objects", static_cast<long long>(bucket_quota.max_objects)); - } - } + return 0; } void RGWStatBucket_ObjStore_S3::send_response() { if (op_ret >= 0) { - dump_bucket_metadata(s, bucket.get(), stats); + if (report_stats) { + dump_bucket_metadata(s, stats); + } + // only bucket's owner is allowed to get the quota settings of the account + if (s->auth.identity->is_owner_of(s->bucket->get_owner())) { + const auto& user_info = s->user->get_info(); + const auto& bucket_quota = s->bucket->get_info().quota; // bucket quota + + dump_header(s, "X-RGW-Quota-Max-Buckets", static_cast<long long>(user_info.max_buckets)); + if (user_info.quota.user_quota.enabled) { + dump_header(s, "X-RGW-Quota-User-Size", static_cast<long long>(user_info.quota.user_quota.max_size)); + dump_header(s, "X-RGW-Quota-User-Objects", static_cast<long long>(user_info.quota.user_quota.max_objects)); + } + if (bucket_quota.enabled) { + dump_header(s, "X-RGW-Quota-Bucket-Size", static_cast<long long>(bucket_quota.max_size)); + dump_header(s, "X-RGW-Quota-Bucket-Objects", static_cast<long long>(bucket_quota.max_objects)); + } + } } set_req_state_err(s, op_ret); @@ -2526,6 +2533,10 @@ int RGWCreateBucket_ObjStore_S3::get_params(optional_yield y) if ((op_ret < 0) && (op_ret != -ERR_LENGTH_REQUIRED)) return op_ret; + if (!driver->is_meta_master()) { + in_data.append(data); + } + if (data.length()) { RGWCreateBucketParser parser; @@ -3808,6 +3819,196 @@ void RGWPutACLs_ObjStore_S3::send_response() dump_start(s); } +int RGWGetObjAttrs_ObjStore_S3::get_params(optional_yield y) +{ + string err; + auto& env = s->info.env; + version_id = s->info.args.get("versionId"); + + auto hdr = env->get_optional("HTTP_X_AMZ_EXPECTED_BUCKET_OWNER"); + if (hdr) { + expected_bucket_owner = *hdr; + } + + hdr = env->get_optional("HTTP_X_AMZ_MAX_PARTS"); + if (hdr) { + max_parts = strict_strtol(hdr->c_str(), 10, &err); + if (!err.empty()) { + s->err.message = "Invalid value for MaxParts: " + err; + ldpp_dout(s, 10) << "Invalid value for MaxParts " << *hdr << ": " + << err << dendl; + return -ERR_INVALID_PART; + } + max_parts = std::min(*max_parts, 1000); + } + + hdr = env->get_optional("HTTP_X_AMZ_PART_NUMBER_MARKER"); + if (hdr) { + marker = strict_strtol(hdr->c_str(), 10, &err); + if (!err.empty()) { + s->err.message = "Invalid value for PartNumberMarker: " + err; + ldpp_dout(s, 10) << "Invalid value for PartNumberMarker " << *hdr << ": " + << err << dendl; + return -ERR_INVALID_PART; + } + } + + hdr = env->get_optional("HTTP_X_AMZ_OBJECT_ATTRIBUTES"); + if (hdr) { + requested_attributes = recognize_attrs(*hdr); + } + + /* XXX skipping SSE-C params for now */ + + return 0; +} /* RGWGetObjAttrs_ObjStore_S3::get_params(...) */ + +int RGWGetObjAttrs_ObjStore_S3::get_decrypt_filter( + std::unique_ptr<RGWGetObj_Filter> *filter, + RGWGetObj_Filter* cb, bufferlist* manifest_bl) +{ + // we aren't actually decrypting the data, but for objects encrypted with + // SSE-C we do need to verify that required headers are present and valid + // + // in the SSE-KMS and SSE-S3 cases, this unfortunately causes us to fetch + // decryption keys which we don't need :( + std::unique_ptr<BlockCrypt> block_crypt; // ignored + std::map<std::string, std::string> crypt_http_responses; // ignored + return rgw_s3_prepare_decrypt(s, s->yield, attrs, &block_crypt, + crypt_http_responses); +} + +void RGWGetObjAttrs_ObjStore_S3::send_response() +{ + if (op_ret) + set_req_state_err(s, op_ret); + dump_errno(s); + + if (op_ret == 0) { + version_id = s->object->get_instance(); + + // x-amz-delete-marker: DeleteMarker // not sure we can plausibly do this? + dump_last_modified(s, lastmod); + dump_header_if_nonempty(s, "x-amz-version-id", version_id); + // x-amz-request-charged: RequestCharged + } + + end_header(s, this, to_mime_type(s->format)); + dump_start(s); + + if (op_ret == 0) { + s->formatter->open_object_section("GetObjectAttributes"); + if (requested_attributes & as_flag(ReqAttributes::Etag)) { + if (lo_etag.empty()) { + auto iter = attrs.find(RGW_ATTR_ETAG); + if (iter != attrs.end()) { + lo_etag = iter->second.to_str(); + } + } + s->formatter->dump_string("ETag", lo_etag); + } + + if (requested_attributes & as_flag(ReqAttributes::Checksum)) { + s->formatter->open_object_section("Checksum"); + auto iter = attrs.find(RGW_ATTR_CKSUM); + if (iter != attrs.end()) { + try { + rgw::cksum::Cksum cksum; + auto bliter = iter->second.cbegin(); + cksum.decode(bliter); + if (multipart_parts_count && multipart_parts_count > 0) { + s->formatter->dump_string(cksum.element_name(), + fmt::format("{}-{}", cksum.to_armor(), *multipart_parts_count)); + } else { + s->formatter->dump_string(cksum.element_name(), cksum.to_armor()); + } + } catch (buffer::error& err) { + ldpp_dout(this, 0) + << "ERROR: could not decode stored cksum, caught buffer::error" << dendl; + } + } + s->formatter->close_section(); /* Checksum */ + } /* Checksum */ + + if (requested_attributes & as_flag(ReqAttributes::ObjectParts)) { + if (multipart_parts_count && multipart_parts_count > 0) { + + /* XXX the following was needed to see a manifest at list_parts()! */ + op_ret = s->object->load_obj_state(s, s->yield); + if (op_ret < 0) { + ldpp_dout_fmt(this, 0, + "ERROR: {} load_obj_state() failed ret={}", __func__, + op_ret); + } + + ldpp_dout_fmt(this, 16, + "{} attr flags={} parts_count={}", + __func__, requested_attributes, *multipart_parts_count); + + s->formatter->open_object_section("ObjectParts"); + + bool truncated = false; + int next_marker; + + using namespace rgw::sal; + + int ret = + s->object->list_parts( + this, s->cct, + max_parts ? *max_parts : 1000, + marker ? *marker : 0, + &next_marker, &truncated, + [&](const Object::Part& part) -> int { + s->formatter->open_object_section("Part"); + s->formatter->dump_int("PartNumber", part.part_number); + s->formatter->dump_unsigned("Size", part.part_size); + if (part.cksum.type != rgw::cksum::Type::none) { + s->formatter->dump_string(part.cksum.element_name(), part.cksum.to_armor()); + } + s->formatter->close_section(); /* Part */ + return 0; + }, s->yield); + + if (ret < 0) { + ldpp_dout_fmt(this, 0, + "ERROR: {} list-parts failed for {}", + __func__, s->object->get_name()); + } + /* AWS docs disagree on the name of this element */ + s->formatter->dump_int("PartsCount", *multipart_parts_count); + s->formatter->dump_int("TotalPartsCount", *multipart_parts_count); + s->formatter->dump_bool("IsTruncated", truncated); + if (max_parts) { + s->formatter->dump_int("MaxParts", *max_parts); + } + if(truncated) { + s->formatter->dump_int("NextPartNumberMarker", next_marker); + } + if (marker) { + s->formatter->dump_int("PartNumberMarker", *marker); + } + s->formatter->close_section(); + } /* multipart_parts_count positive */ + } /* ObjectParts */ + + if (requested_attributes & as_flag(ReqAttributes::ObjectSize)) { + s->formatter->dump_int("ObjectSize", s->obj_size); + } + + if (requested_attributes & as_flag(ReqAttributes::StorageClass)) { + auto iter = attrs.find(RGW_ATTR_STORAGE_CLASS); + if (iter != attrs.end()) { + s->formatter->dump_string("StorageClass", iter->second.to_str()); + } else { + s->formatter->dump_string("StorageClass", "STANDARD"); + } + } + s->formatter->close_section(); + } /* op_ret == 0 */ + + rgw_flush_formatter_and_reset(s, s->formatter); +} /* RGWGetObjAttrs_ObjStore_S3::send_response */ + void RGWGetLC_ObjStore_S3::execute(optional_yield y) { config.set_ctx(s->cct); @@ -4787,6 +4988,7 @@ RGWOp *RGWHandler_REST_Bucket_S3::get_obj_op(bool get_data) const RGWOp *RGWHandler_REST_Bucket_S3::op_get() { + /* XXX maybe we could replace this with an indexing operation */ if (s->info.args.sub_resource_exists("encryption")) return nullptr; @@ -4947,6 +5149,10 @@ RGWOp *RGWHandler_REST_Bucket_S3::op_post() return new RGWDeleteMultiObj_ObjStore_S3; } + if (s->info.args.exists("logging")) { + return RGWHandler_REST_BucketLogging_S3::create_post_op(); + } + if (s->info.args.exists("mdsearch")) { if (!s->cct->_conf->rgw_enable_mdsearch) { return NULL; @@ -4979,6 +5185,8 @@ RGWOp *RGWHandler_REST_Obj_S3::op_get() return new RGWGetObjLayout_ObjStore_S3; } else if (is_tagging_op()) { return new RGWGetObjTags_ObjStore_S3; + } else if (is_attributes_op()) { + return new RGWGetObjAttrs_ObjStore_S3; } else if (is_obj_retention_op()) { return new RGWGetObjRetention_ObjStore_S3; } else if (is_obj_legal_hold_op()) { @@ -6104,7 +6312,8 @@ AWSGeneralAbstractor::get_auth_data_v4(const req_state* const s, case RGW_OP_GET_BUCKET_PUBLIC_ACCESS_BLOCK: case RGW_OP_DELETE_BUCKET_PUBLIC_ACCESS_BLOCK: case RGW_OP_GET_OBJ://s3select its post-method(payload contain the query) , the request is get-object - case RGW_OP_PUT_BUCKET_LOGGING: + case RGW_OP_PUT_BUCKET_LOGGING: + case RGW_OP_POST_BUCKET_LOGGING: case RGW_OP_GET_BUCKET_LOGGING: break; default: @@ -6523,7 +6732,7 @@ rgw::auth::s3::LocalEngine::authenticate( /* Ignore signature for HTTP OPTIONS */ if (s->op_type == RGW_OP_OPTIONS_CORS) { auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), k.subuser, std::nullopt, access_key_id); return result_t::grant(std::move(apl), completer_factory(k.key)); } @@ -6544,7 +6753,7 @@ rgw::auth::s3::LocalEngine::authenticate( } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), k.subuser, std::nullopt, access_key_id); return result_t::grant(std::move(apl), completer_factory(k.key)); } @@ -6753,7 +6962,7 @@ rgw::auth::s3::STSEngine::authenticate( string subuser; auto apl = local_apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), subuser, token.perm_mask, std::string(_access_key_id)); return result_t::grant(std::move(apl), completer_factory(token.secret_access_key)); } diff --git a/src/rgw/rgw_rest_s3.h b/src/rgw/rgw_rest_s3.h index c47edb38855..e8fdc69751c 100644 --- a/src/rgw/rgw_rest_s3.h +++ b/src/rgw/rgw_rest_s3.h @@ -242,6 +242,7 @@ public: ~RGWStatBucket_ObjStore_S3() override {} void send_response() override; + int get_params(optional_yield y) override; }; class RGWCreateBucket_ObjStore_S3 : public RGWCreateBucket_ObjStore { @@ -373,6 +374,18 @@ public: int get_params(optional_yield y) override; }; +class RGWGetObjAttrs_ObjStore_S3 : public RGWGetObjAttrs_ObjStore { +public: + RGWGetObjAttrs_ObjStore_S3() {} + ~RGWGetObjAttrs_ObjStore_S3() override {} + + int get_params(optional_yield y) override; + int get_decrypt_filter(std::unique_ptr<RGWGetObj_Filter>* filter, + RGWGetObj_Filter* cb, + bufferlist* manifest_bl) override; + void send_response() override; +}; + class RGWGetLC_ObjStore_S3 : public RGWGetLC_ObjStore { protected: RGWLifecycleConfiguration_S3 config; @@ -700,6 +713,9 @@ protected: bool is_acl_op() const { return s->info.args.exists("acl"); } + bool is_attributes_op() const { + return s->info.args.exists("attributes"); + } bool is_cors_op() const { return s->info.args.exists("cors"); } @@ -758,6 +774,9 @@ protected: bool is_acl_op() const { return s->info.args.exists("acl"); } + bool is_attributes_op() const { + return s->info.args.exists("attributes"); + } bool is_tagging_op() const { return s->info.args.exists("tagging"); } diff --git a/src/rgw/rgw_rest_sts.cc b/src/rgw/rgw_rest_sts.cc index f2bd9429a55..1101da0af3c 100644 --- a/src/rgw/rgw_rest_sts.cc +++ b/src/rgw/rgw_rest_sts.cc @@ -436,6 +436,9 @@ WebTokenEngine::validate_signature(const DoutPrefixProvider* dpp, const jwt::dec .allow_algorithm(jwt::algorithm::ps512{cert}); verifier.verify(decoded); + } else { + ldpp_dout(dpp, 0) << "Unsupported algorithm: " << algorithm << dendl; + throw -EINVAL; } } catch (std::runtime_error& e) { ldpp_dout(dpp, 0) << "Signature validation failed: " << e.what() << dendl; diff --git a/src/rgw/rgw_rest_swift.cc b/src/rgw/rgw_rest_swift.cc index 35c36d1ae1a..b8ff3ca2fe8 100644 --- a/src/rgw/rgw_rest_swift.cc +++ b/src/rgw/rgw_rest_swift.cc @@ -447,7 +447,6 @@ int RGWListBucket_ObjStore_SWIFT::get_params(optional_yield y) } static void dump_container_metadata(req_state *, - const rgw::sal::Bucket*, const std::optional<RGWStorageStats>& stats, const RGWQuotaInfo&, const RGWBucketWebsiteConf&); @@ -458,7 +457,7 @@ void RGWListBucket_ObjStore_SWIFT::send_response() map<string, bool>::iterator pref_iter = common_prefixes.begin(); dump_start(s); - dump_container_metadata(s, s->bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); s->formatter->open_array_section_with_attrs("container", @@ -558,7 +557,6 @@ next: } // RGWListBucket_ObjStore_SWIFT::send_response static void dump_container_metadata(req_state *s, - const rgw::sal::Bucket* bucket, const std::optional<RGWStorageStats>& stats, const RGWQuotaInfo& quota, const RGWBucketWebsiteConf& ws_conf) @@ -683,7 +681,7 @@ void RGWStatBucket_ObjStore_SWIFT::send_response() { if (op_ret >= 0) { op_ret = STATUS_NO_CONTENT; - dump_container_metadata(s, bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); } @@ -2640,7 +2638,7 @@ RGWOp* RGWSwiftWebsiteHandler::get_ws_listing_op() /* Generate the header now. */ set_req_state_err(s, op_ret); dump_errno(s); - dump_container_metadata(s, s->bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); end_header(s, this, "text/html"); if (op_ret < 0) { diff --git a/src/rgw/rgw_rest_swift.h b/src/rgw/rgw_rest_swift.h index eb1c4422e34..ec206a5160f 100644 --- a/src/rgw/rgw_rest_swift.h +++ b/src/rgw/rgw_rest_swift.h @@ -86,6 +86,7 @@ public: RGWStatBucket_ObjStore_SWIFT() {} ~RGWStatBucket_ObjStore_SWIFT() override {} + int get_params(optional_yield y) override { return 0; } void send_response() override; }; diff --git a/src/rgw/rgw_s3_filter.cc b/src/rgw/rgw_s3_filter.cc new file mode 100644 index 00000000000..05a7c4a7293 --- /dev/null +++ b/src/rgw/rgw_s3_filter.cc @@ -0,0 +1,269 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#include "rgw_pubsub.h" +#include "rgw_tools.h" +#include "rgw_xml.h" +#include "rgw_s3_filter.h" +#include "common/errno.h" +#include "rgw_sal.h" +#include <regex> +#include <algorithm> + +void rgw_s3_key_filter::dump(Formatter *f) const { + if (!has_content()) { + return; + } + f->open_array_section("FilterRules"); + if (!prefix_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "prefix", f); + ::encode_json("Value", prefix_rule, f); + f->close_section(); + } + if (!suffix_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "suffix", f); + ::encode_json("Value", suffix_rule, f); + f->close_section(); + } + if (!regex_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "regex", f); + ::encode_json("Value", regex_rule, f); + f->close_section(); + } + f->close_section(); +} + +bool rgw_s3_key_filter::decode_xml(XMLObj* obj) { + XMLObjIter iter = obj->find("FilterRule"); + XMLObj *o; + + const auto throw_if_missing = true; + auto prefix_not_set = true; + auto suffix_not_set = true; + auto regex_not_set = true; + std::string name; + + while ((o = iter.get_next())) { + RGWXMLDecoder::decode_xml("Name", name, o, throw_if_missing); + if (name == "prefix" && prefix_not_set) { + prefix_not_set = false; + RGWXMLDecoder::decode_xml("Value", prefix_rule, o, throw_if_missing); + } else if (name == "suffix" && suffix_not_set) { + suffix_not_set = false; + RGWXMLDecoder::decode_xml("Value", suffix_rule, o, throw_if_missing); + } else if (name == "regex" && regex_not_set) { + regex_not_set = false; + RGWXMLDecoder::decode_xml("Value", regex_rule, o, throw_if_missing); + } else { + throw RGWXMLDecoder::err("invalid/duplicate S3Key filter rule name: '" + name + "'"); + } + } + return true; +} + +void rgw_s3_key_filter::dump_xml(Formatter *f) const { + if (!prefix_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "prefix", f); + ::encode_xml("Value", prefix_rule, f); + f->close_section(); + } + if (!suffix_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "suffix", f); + ::encode_xml("Value", suffix_rule, f); + f->close_section(); + } + if (!regex_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "regex", f); + ::encode_xml("Value", regex_rule, f); + f->close_section(); + } +} + +bool rgw_s3_key_filter::has_content() const { + return !(prefix_rule.empty() && suffix_rule.empty() && regex_rule.empty()); +} + +void rgw_s3_key_value_filter::dump(Formatter *f) const { + if (!has_content()) { + return; + } + f->open_array_section("FilterRules"); + for (const auto& key_value : kv) { + f->open_object_section(""); + ::encode_json("Name", key_value.first, f); + ::encode_json("Value", key_value.second, f); + f->close_section(); + } + f->close_section(); +} + +bool rgw_s3_key_value_filter::decode_xml(XMLObj* obj) { + kv.clear(); + XMLObjIter iter = obj->find("FilterRule"); + XMLObj *o; + + const auto throw_if_missing = true; + + std::string key; + std::string value; + + while ((o = iter.get_next())) { + RGWXMLDecoder::decode_xml("Name", key, o, throw_if_missing); + RGWXMLDecoder::decode_xml("Value", value, o, throw_if_missing); + kv.emplace(key, value); + } + return true; +} + +void rgw_s3_key_value_filter::dump_xml(Formatter *f) const { + for (const auto& key_value : kv) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", key_value.first, f); + ::encode_xml("Value", key_value.second, f); + f->close_section(); + } +} + +bool rgw_s3_key_value_filter::has_content() const { + return !kv.empty(); +} + +void rgw_s3_filter::dump(Formatter *f) const { + encode_json("S3Key", key_filter, f); + encode_json("S3Metadata", metadata_filter, f); + encode_json("S3Tags", tag_filter, f); +} + +bool rgw_s3_filter::decode_xml(XMLObj* obj) { + RGWXMLDecoder::decode_xml("S3Key", key_filter, obj); + RGWXMLDecoder::decode_xml("S3Metadata", metadata_filter, obj); + RGWXMLDecoder::decode_xml("S3Tags", tag_filter, obj); + return true; +} + +void rgw_s3_filter::dump_xml(Formatter *f) const { + if (key_filter.has_content()) { + ::encode_xml("S3Key", key_filter, f); + } + if (metadata_filter.has_content()) { + ::encode_xml("S3Metadata", metadata_filter, f); + } + if (tag_filter.has_content()) { + ::encode_xml("S3Tags", tag_filter, f); + } +} + +bool rgw_s3_filter::has_content() const { + return key_filter.has_content() || + metadata_filter.has_content() || + tag_filter.has_content(); +} + +bool match(const rgw_s3_key_filter& filter, const std::string& key) { + const auto key_size = key.size(); + const auto prefix_size = filter.prefix_rule.size(); + if (prefix_size != 0) { + // prefix rule exists + if (prefix_size > key_size) { + // if prefix is longer than key, we fail + return false; + } + if (!std::equal(filter.prefix_rule.begin(), filter.prefix_rule.end(), key.begin())) { + return false; + } + } + const auto suffix_size = filter.suffix_rule.size(); + if (suffix_size != 0) { + // suffix rule exists + if (suffix_size > key_size) { + // if suffix is longer than key, we fail + return false; + } + if (!std::equal(filter.suffix_rule.begin(), filter.suffix_rule.end(), (key.end() - suffix_size))) { + return false; + } + } + if (!filter.regex_rule.empty()) { + // TODO add regex caching in the filter + const std::regex base_regex(filter.regex_rule); + if (!std::regex_match(key, base_regex)) { + return false; + } + } + return true; +} + +bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv) { + // all filter pairs must exist with the same value in the object's metadata/tags + // object metadata/tags may include items not in the filter + return std::includes(kv.begin(), kv.end(), filter.kv.begin(), filter.kv.end()); +} + +bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv) { + // all filter pairs must exist with the same value in the object's metadata/tags + // object metadata/tags may include items not in the filter + for (auto& filter : filter.kv) { + auto result = kv.equal_range(filter.first); + if (std::any_of(result.first, result.second, [&filter](const std::pair<std::string, std::string>& p) { return p.second == filter.second;})) + continue; + else + return false; + } + return true; +} + +bool match(const rgw_s3_filter& s3_filter, const rgw::sal::Object* obj) { + if (obj == nullptr) { + return false; + } + + if (match(s3_filter.key_filter, obj->get_name())) { + return true; + } + + const auto &attrs = obj->get_attrs(); + if (!s3_filter.metadata_filter.kv.empty()) { + KeyValueMap attrs_map; + for (auto& attr : attrs) { + if (boost::algorithm::starts_with(attr.first, RGW_ATTR_META_PREFIX)) { + std::string_view key(attr.first); + key.remove_prefix(sizeof(RGW_ATTR_PREFIX)-1); + // we want to pass a null terminated version + // of the bufferlist, hence "to_str().c_str()" + attrs_map.emplace(key, attr.second.to_str().c_str()); + } + } + if (match(s3_filter.metadata_filter, attrs_map)) { + return true; + } + } + + if (!s3_filter.tag_filter.kv.empty()) { + // tag filter exists + // try to fetch tags from the attributes + KeyMultiValueMap tags; + const auto attr_iter = attrs.find(RGW_ATTR_TAGS); + if (attr_iter != attrs.end()) { + auto bliter = attr_iter->second.cbegin(); + RGWObjTags obj_tags; + try { + ::decode(obj_tags, bliter); + } catch (buffer::error &) { + // not able to decode tags + return false; + } + tags = std::move(obj_tags.get_tags()); + } + if (match(s3_filter.tag_filter, tags)) { + return true; + } + } + + return false; +} diff --git a/src/rgw/rgw_s3_filter.h b/src/rgw/rgw_s3_filter.h new file mode 100644 index 00000000000..9bbc4ef0088 --- /dev/null +++ b/src/rgw/rgw_s3_filter.h @@ -0,0 +1,102 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#pragma once + +#include "rgw_tools.h" +#include <boost/container/flat_map.hpp> + +class XMLObj; + +struct rgw_s3_key_filter { + std::string prefix_rule; + std::string suffix_rule; + std::string regex_rule; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(1, 1, bl); + encode(prefix_rule, bl); + encode(suffix_rule, bl); + encode(regex_rule, bl); + ENCODE_FINISH(bl); + } + + void decode(bufferlist::const_iterator& bl) { + DECODE_START(1, bl); + decode(prefix_rule, bl); + decode(suffix_rule, bl); + decode(regex_rule, bl); + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_key_filter) + +using KeyValueMap = boost::container::flat_map<std::string, std::string>; +using KeyMultiValueMap = std::multimap<std::string, std::string>; + +struct rgw_s3_key_value_filter { + KeyValueMap kv; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(1, 1, bl); + encode(kv, bl); + ENCODE_FINISH(bl); + } + void decode(bufferlist::const_iterator& bl) { + DECODE_START(1, bl); + decode(kv, bl); + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_key_value_filter) + +struct rgw_s3_filter { + rgw_s3_key_filter key_filter; + rgw_s3_key_value_filter metadata_filter; + rgw_s3_key_value_filter tag_filter; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(2, 1, bl); + encode(key_filter, bl); + encode(metadata_filter, bl); + encode(tag_filter, bl); + ENCODE_FINISH(bl); + } + + void decode(bufferlist::const_iterator& bl) { + DECODE_START(2, bl); + decode(key_filter, bl); + decode(metadata_filter, bl); + if (struct_v >= 2) { + decode(tag_filter, bl); + } + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_filter) + +bool match(const rgw_s3_key_filter& filter, const std::string& key); + +bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv); + +bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv); + +bool match(const rgw_s3_filter& filter, const rgw::sal::Object* obj); diff --git a/src/rgw/rgw_sal.h b/src/rgw/rgw_sal.h index e098c4decf7..4b94f74b851 100644 --- a/src/rgw/rgw_sal.h +++ b/src/rgw/rgw_sal.h @@ -15,6 +15,7 @@ #pragma once +#include <cstdint> #include <optional> #include <boost/intrusive_ptr.hpp> #include <boost/smart_ptr/intrusive_ref_counter.hpp> @@ -26,6 +27,7 @@ #include "rgw_notify_event_type.h" #include "rgw_req_context.h" #include "include/random.h" +#include "include/function2.hpp" // FIXME: following subclass dependencies #include "driver/rados/rgw_user.h" @@ -1169,6 +1171,9 @@ class Object { std::string* version_id, std::string* tag, std::string* etag, void (*progress_cb)(off_t, void *), void* progress_data, const DoutPrefixProvider* dpp, optional_yield y) = 0; + + /** return logging subsystem */ + virtual unsigned get_subsys() { return ceph_subsys_rgw; }; /** Get the ACL for this object */ virtual RGWAccessControlPolicy& get_acl(void) = 0; /** Set the ACL for this object */ @@ -1249,6 +1254,28 @@ class Object { /** Dump driver-specific object layout info in JSON */ virtual int dump_obj_layout(const DoutPrefixProvider *dpp, optional_yield y, Formatter* f) = 0; + /* A transfer data type describing metadata specific to one part of a + * completed multipart upload object, following the GetObjectAttributes + * response syntax for Object::Parts here: + * https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html */ + class Part + { + public: + int part_number; + uint32_t part_size; + rgw::cksum::Cksum cksum; + }; /* Part */ + + /* callback function/object used by list_parts */ + using list_parts_each_t = + const fu2::unique_function<int(const Part&) const>; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) = 0; + /** Get the cached attributes for this object */ virtual Attrs& get_attrs(void) = 0; /** Get the (const) cached attributes for this object */ @@ -1447,7 +1474,7 @@ public: virtual int init(const DoutPrefixProvider* dpp, optional_yield y, ACLOwner& owner, rgw_placement_rule& dest_placement, rgw::sal::Attrs& attrs) = 0; /** List all the parts of this upload, filling the parts cache */ virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, - int num_parts, int marker, + int max_parts, int marker, int* next_marker, bool* truncated, optional_yield y, bool assume_unsorted = false) = 0; /** Abort this upload */ @@ -1751,8 +1778,6 @@ class Zone { virtual bool is_writeable() = 0; /** Get the URL for the endpoint for redirecting to this zone */ virtual bool get_redirect_endpoint(std::string* endpoint) = 0; - /** Check to see if the given API is supported in this zone */ - virtual bool has_zonegroup_api(const std::string& api) const = 0; /** Get the current period ID for this zone */ virtual const std::string& get_current_period_id() = 0; /** Get thes system access key for this zone */ diff --git a/src/rgw/rgw_sal_dbstore.cc b/src/rgw/rgw_sal_dbstore.cc index d3af42cf2ec..02fd7a49cda 100644 --- a/src/rgw/rgw_sal_dbstore.cc +++ b/src/rgw/rgw_sal_dbstore.cc @@ -271,7 +271,7 @@ namespace rgw::sal { /* XXX: handle has_instance_obj like in set_bucket_instance_attrs() */ - ret = store->getDB()->update_bucket(dpp, "attrs", info, false, nullptr, &new_attrs, nullptr, &get_info().objv_tracker); + ret = store->getDB()->update_bucket(dpp, "attrs", info, false, nullptr, &attrs, nullptr, &get_info().objv_tracker); return ret; } @@ -458,14 +458,6 @@ namespace rgw::sal { return false; } - bool DBZone::has_zonegroup_api(const std::string& api) const - { - if (api == "default") - return true; - - return false; - } - const std::string& DBZone::get_current_period_id() { return current_period->get_id(); @@ -496,6 +488,14 @@ namespace rgw::sal { return std::make_unique<DBLuaManager>(this); } + int DBObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) + { + return -EOPNOTSUPP; + } + int DBObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { RGWObjState* astate; diff --git a/src/rgw/rgw_sal_dbstore.h b/src/rgw/rgw_sal_dbstore.h index 107ba735a63..4df10d1dce1 100644 --- a/src/rgw/rgw_sal_dbstore.h +++ b/src/rgw/rgw_sal_dbstore.h @@ -268,22 +268,22 @@ protected: class DBZone : public StoreZone { protected: DBStore* store; - RGWRealm *realm{nullptr}; - DBZoneGroup *zonegroup{nullptr}; - RGWZone *zone_public_config{nullptr}; /* external zone params, e.g., entrypoints, log flags, etc. */ - RGWZoneParams *zone_params{nullptr}; /* internal zone params, e.g., rados pools */ - RGWPeriod *current_period{nullptr}; + std::unique_ptr<RGWRealm> realm; + std::unique_ptr<DBZoneGroup> zonegroup; + std::unique_ptr<RGWZone> zone_public_config; /* external zone params, e.g., entrypoints, log flags, etc. */ + std::unique_ptr<RGWZoneParams> zone_params; /* internal zone params, e.g., rados pools */ + std::unique_ptr<RGWPeriod> current_period; public: DBZone(DBStore* _store) : store(_store) { - realm = new RGWRealm(); + realm = std::make_unique<RGWRealm>(); std::unique_ptr<RGWZoneGroup> rzg = std::make_unique<RGWZoneGroup>("default", "default"); rzg->api_name = "default"; rzg->is_master = true; - zonegroup = new DBZoneGroup(store, std::move(rzg)); - zone_public_config = new RGWZone(); - zone_params = new RGWZoneParams(); - current_period = new RGWPeriod(); + zonegroup = std::make_unique<DBZoneGroup>(store, std::move(rzg)); + zone_public_config = std::make_unique<RGWZone>(); + zone_params = std::make_unique<RGWZoneParams>(); + current_period = std::make_unique<RGWPeriod>(); // XXX: only default and STANDARD supported for now RGWZonePlacementInfo info; @@ -292,13 +292,7 @@ protected: info.storage_classes = sc; zone_params->placement_pools["default"] = info; } - ~DBZone() { - delete realm; - delete zonegroup; - delete zone_public_config; - delete zone_params; - delete current_period; - } + ~DBZone() = default; virtual std::unique_ptr<Zone> clone() override { return std::make_unique<DBZone>(store); @@ -309,7 +303,6 @@ protected: virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() override; virtual const std::string& get_realm_name() override; @@ -535,6 +528,7 @@ protected: DBObject(DBObject& _o) = default; + virtual unsigned get_subsys() { return ceph_subsys_rgw_dbstore; }; virtual int delete_object(const DoutPrefixProvider* dpp, optional_yield y, uint32_t flags, @@ -560,6 +554,13 @@ protected: virtual int set_acl(const RGWAccessControlPolicy& acl) override { acls = acl; return 0; } virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; virtual int get_obj_attrs(optional_yield y, const DoutPrefixProvider* dpp, rgw_obj* target_obj = NULL) override; virtual int modify_obj_attrs(const char* attr_name, bufferlist& attr_val, optional_yield y, const DoutPrefixProvider* dpp) override; diff --git a/src/rgw/rgw_sal_filter.cc b/src/rgw/rgw_sal_filter.cc index 733bfa39ee2..15da580988e 100644 --- a/src/rgw/rgw_sal_filter.cc +++ b/src/rgw/rgw_sal_filter.cc @@ -1046,6 +1046,17 @@ RGWAccessControlPolicy& FilterObject::get_acl() return next->get_acl(); } +int FilterObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + return next->list_parts(dpp, cct, max_parts, marker, next_marker, + truncated, + sal::Object::list_parts_each_t(each_func), + y); +} + int FilterObject::load_obj_state(const DoutPrefixProvider *dpp, optional_yield y, bool follow_olh) { return next->load_obj_state(dpp, y, follow_olh); diff --git a/src/rgw/rgw_sal_filter.h b/src/rgw/rgw_sal_filter.h index 43a440e8b10..947ce9d4bf5 100644 --- a/src/rgw/rgw_sal_filter.h +++ b/src/rgw/rgw_sal_filter.h @@ -108,9 +108,6 @@ public: virtual bool get_redirect_endpoint(std::string* endpoint) override { return next->get_redirect_endpoint(endpoint); } - virtual bool has_zonegroup_api(const std::string& api) const override { - return next->has_zonegroup_api(api); - } virtual const std::string& get_current_period_id() override { return next->get_current_period_id(); } @@ -781,6 +778,12 @@ public: virtual bool empty() const override { return next->empty(); } virtual const std::string &get_name() const override { return next->get_name(); } + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider *dpp, optional_yield y, bool follow_olh = true) override; virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, diff --git a/src/rgw/rgw_swift_auth.cc b/src/rgw/rgw_swift_auth.cc index 032b3734bf9..937f74601b3 100644 --- a/src/rgw/rgw_swift_auth.cc +++ b/src/rgw/rgw_swift_auth.cc @@ -522,7 +522,7 @@ ExternalTokenEngine::authenticate(const DoutPrefixProvider* dpp, } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), + cct, s, std::move(user), std::move(account), std::move(policies), extract_swift_subuser(swift_user), std::nullopt, LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); @@ -685,7 +685,7 @@ SignedTokenEngine::authenticate(const DoutPrefixProvider* dpp, } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), + cct, s, std::move(user), std::move(account), std::move(policies), extract_swift_subuser(swift_user), std::nullopt, LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); diff --git a/src/rgw/rgw_swift_auth.h b/src/rgw/rgw_swift_auth.h index 9049c54f5ca..c27a24a2619 100644 --- a/src/rgw/rgw_swift_auth.h +++ b/src/rgw/rgw_swift_auth.h @@ -23,8 +23,8 @@ namespace swift { class TempURLApplier : public rgw::auth::LocalApplier { public: TempURLApplier(CephContext* const cct, - const RGWUserInfo& user_info) - : LocalApplier(cct, user_info, std::nullopt, {}, LocalApplier::NO_SUBUSER, + std::unique_ptr<rgw::sal::User> user) + : LocalApplier(cct, std::move(user), std::nullopt, {}, LocalApplier::NO_SUBUSER, std::nullopt, LocalApplier::NO_ACCESS_KEY) {} @@ -155,8 +155,8 @@ public: class SwiftAnonymousApplier : public rgw::auth::LocalApplier { public: SwiftAnonymousApplier(CephContext* const cct, - const RGWUserInfo& user_info) - : LocalApplier(cct, user_info, std::nullopt, {}, LocalApplier::NO_SUBUSER, + std::unique_ptr<rgw::sal::User> user) + : LocalApplier(cct, std::move(user), std::nullopt, {}, LocalApplier::NO_SUBUSER, std::nullopt, LocalApplier::NO_ACCESS_KEY) { } bool is_admin_of(const rgw_owner& o) const {return false;} @@ -238,7 +238,7 @@ class DefaultStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, @@ -247,7 +247,7 @@ class DefaultStrategy : public rgw::auth::Strategy, auto apl = \ rgw::auth::add_3rdparty(driver, rgw_user(s->account_name), rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id))); /* TODO(rzarzynski): replace with static_ptr. */ return aplptr_t(new decltype(apl)(std::move(apl))); @@ -259,7 +259,9 @@ class DefaultStrategy : public rgw::auth::Strategy, /* TempURL doesn't need any user account override. It's a Swift-specific * mechanism that requires account name internally, so there is no * business with delegating the responsibility outside. */ - return aplptr_t(new rgw::auth::swift::TempURLApplier(cct, user_info)); + std::unique_ptr<rgw::sal::User> user = s->user->clone(); + user->get_info() = user_info; + return aplptr_t(new rgw::auth::swift::TempURLApplier(cct, std::move(user))); } public: diff --git a/src/rgw/services/svc_zone.cc b/src/rgw/services/svc_zone.cc index 70cf40eb6cb..97d81550058 100644 --- a/src/rgw/services/svc_zone.cc +++ b/src/rgw/services/svc_zone.cc @@ -657,18 +657,6 @@ const string& RGWSI_Zone::get_current_period_id() const return current_period->get_id(); } -bool RGWSI_Zone::has_zonegroup_api(const std::string& api) const -{ - if (!current_period->get_id().empty()) { - const auto& zonegroups_by_api = current_period->get_map().zonegroups_by_api; - if (zonegroups_by_api.find(api) != zonegroups_by_api.end()) - return true; - } else if (zonegroup->api_name == api) { - return true; - } - return false; -} - bool RGWSI_Zone::zone_is_writeable() { return writeable_zone && !get_zone().is_read_only(); @@ -743,8 +731,7 @@ bool RGWSI_Zone::is_meta_master() const bool RGWSI_Zone::need_to_log_metadata() const { - return is_meta_master() && - (zonegroup->zones.size() > 1 || current_period->is_multi_zonegroups_with_zones()); + return is_meta_master() && is_syncing_bucket_meta(); } bool RGWSI_Zone::can_reshard() const @@ -761,33 +748,16 @@ bool RGWSI_Zone::can_reshard() const /** * Check to see if the bucket metadata could be synced - * bucket: the bucket to check * Returns false is the bucket is not synced */ -bool RGWSI_Zone::is_syncing_bucket_meta(const rgw_bucket& bucket) +bool RGWSI_Zone::is_syncing_bucket_meta() const { - /* no current period */ if (current_period->get_id().empty()) { return false; } - /* zonegroup is not master zonegroup */ - if (!zonegroup->is_master_zonegroup()) { - return false; - } - - /* single zonegroup and a single zone */ - if (current_period->is_single_zonegroup() && zonegroup->zones.size() == 1) { - return false; - } - - /* zone is not master */ - if (zonegroup->master_zone != zone_public_config->id) { - return false; - } - - return true; + return zonegroup->zones.size() > 1 || current_period->is_multi_zonegroups_with_zones(); } diff --git a/src/rgw/services/svc_zone.h b/src/rgw/services/svc_zone.h index c4a3a28f0d7..719546eb8db 100644 --- a/src/rgw/services/svc_zone.h +++ b/src/rgw/services/svc_zone.h @@ -96,7 +96,6 @@ public: uint32_t get_zone_short_id() const; const std::string& get_current_period_id() const; - bool has_zonegroup_api(const std::string& api) const; bool zone_is_writeable(); bool zone_syncs_from(const RGWZone& target_zone, const RGWZone& source_zone) const; @@ -146,7 +145,7 @@ public: bool need_to_log_data() const; bool need_to_log_metadata() const; bool can_reshard() const; - bool is_syncing_bucket_meta(const rgw_bucket& bucket); + bool is_syncing_bucket_meta() const; int list_zonegroups(const DoutPrefixProvider *dpp, std::list<std::string>& zonegroups); int list_regions(const DoutPrefixProvider *dpp, std::list<std::string>& regions); diff --git a/src/script/ceph-backport.sh b/src/script/ceph-backport.sh index a56509e3d3a..c216ed32d9b 100755 --- a/src/script/ceph-backport.sh +++ b/src/script/ceph-backport.sh @@ -779,7 +779,7 @@ function maybe_deduce_remote { else assert_fail "bad remote_type ->$remote_type<- in maybe_deduce_remote" fi - remote=$(git remote -v | grep --extended-regexp --ignore-case '(://|@)github.com(/|:|:/)'${url_component}'/ceph(\s|\.|\/)' | head -n1 | cut -f 1) + remote=$(git remote -v | grep --extended-regexp --ignore-case '(://|@)github.com(/|:|:/)'${url_component}'/ceph(\s|\.|\/|-)' | head -n1 | cut -f 1) echo "$remote" } diff --git a/src/script/run-make.sh b/src/script/run-make.sh index 52d43d3a171..23724028fe6 100755 --- a/src/script/run-make.sh +++ b/src/script/run-make.sh @@ -29,6 +29,7 @@ function clean_up_after_myself() { function detect_ceph_dev_pkgs() { local boost_root=/opt/ceph + local cmake_opts="" if test -f $boost_root/include/boost/config.hpp; then cmake_opts+=" -DWITH_SYSTEM_BOOST=ON -DBOOST_ROOT=$boost_root" else diff --git a/src/test/CMakeLists.txt b/src/test/CMakeLists.txt index 0ea0bb29347..82816fb07c8 100644 --- a/src/test/CMakeLists.txt +++ b/src/test/CMakeLists.txt @@ -591,6 +591,7 @@ if(NOT WIN32) ceph_snappy cls_lock ceph_test_objectstore + ceph_test_bluefs ceph_erasure_code_non_regression cython_modules crushtool diff --git a/src/test/ObjectMap/KeyValueDBMemory.cc b/src/test/ObjectMap/KeyValueDBMemory.cc index 234e963397e..cfe25930d6a 100644 --- a/src/test/ObjectMap/KeyValueDBMemory.cc +++ b/src/test/ObjectMap/KeyValueDBMemory.cc @@ -132,12 +132,26 @@ public: return ""; } + string_view key_as_sv() override { + if (valid()) + return (*it).first.second; + else + return ""; + } + pair<string,string> raw_key() override { if (valid()) return (*it).first; else return make_pair("", ""); } + + pair<string_view,string_view> raw_key_as_sv() override { + if (valid()) + return (*it).first; + else + return make_pair("", ""); + } bool raw_key_is_prefixed(const string &prefix) override { return prefix == (*it).first.first; @@ -150,6 +164,13 @@ public: return bufferlist(); } + std::string_view value_as_sv() override { + if (valid()) + return std::string_view{it->second.c_str(), it->second.length()}; + else + return std::string_view(); + } + int status() override { return 0; } diff --git a/src/test/cli/radosgw-admin/help.t b/src/test/cli/radosgw-admin/help.t index cb45a9883c3..8b7c6d6e3db 100644 --- a/src/test/cli/radosgw-admin/help.t +++ b/src/test/cli/radosgw-admin/help.t @@ -226,6 +226,7 @@ --secret/--secret-key=<key> specify secret key --gen-access-key generate random access key (for S3) --gen-secret generate random secret key + --generate-key create user with or without credentials --key-type=<type> key type, options are: swift, s3 --key-active=<bool> activate or deactivate a key --temp-url-key[-2]=<key> temp url key diff --git a/src/test/cli/rbd/help.t b/src/test/cli/rbd/help.t index 984175a97b9..5f304258358 100644 --- a/src/test/cli/rbd/help.t +++ b/src/test/cli/rbd/help.t @@ -916,7 +916,7 @@ [--group-namespace <group-namespace>] [--group <group>] [--image-pool <image-pool>] [--image-namespace <image-namespace>] - [--image <image>] [--pool <pool>] + [--image <image>] <group-spec> <image-spec> Add an image to a group. @@ -934,7 +934,6 @@ --image-pool arg image pool name --image-namespace arg image namespace name --image arg image name - -p [ --pool ] arg pool name unless overridden rbd help group image list usage: rbd group image list [--format <format>] [--pretty-format] @@ -960,8 +959,7 @@ [--group-namespace <group-namespace>] [--group <group>] [--image-pool <image-pool>] [--image-namespace <image-namespace>] - [--image <image>] [--pool <pool>] - [--image-id <image-id>] + [--image <image>] [--image-id <image-id>] <group-spec> <image-spec> Remove an image from a group. @@ -979,7 +977,6 @@ --image-pool arg image pool name --image-namespace arg image namespace name --image arg image name - -p [ --pool ] arg pool name unless overridden --image-id arg image id rbd help group info diff --git a/src/test/cls_log/test_cls_log.cc b/src/test/cls_log/test_cls_log.cc index f8c1a32494a..91e38844dec 100644 --- a/src/test/cls_log/test_cls_log.cc +++ b/src/test/cls_log/test_cls_log.cc @@ -332,7 +332,6 @@ TEST_F(cls_log, trim_by_marker) utime_t start_time = ceph_clock_now(); generate_log(ioctx, oid, 10, start_time, true); - utime_t zero_time; std::vector<cls_log_entry> log1; { list<cls_log_entry> entries; diff --git a/src/test/common/CMakeLists.txt b/src/test/common/CMakeLists.txt index 33ff38b932d..3c9cf0003aa 100644 --- a/src/test/common/CMakeLists.txt +++ b/src/test/common/CMakeLists.txt @@ -229,6 +229,12 @@ add_executable(unittest_xmlformatter add_ceph_unittest(unittest_xmlformatter) target_link_libraries(unittest_xmlformatter ceph-common) +add_executable(unittest_htmlformatter + test_htmlformatter.cc + ) +add_ceph_unittest(unittest_htmlformatter) +target_link_libraries(unittest_htmlformatter ceph-common) + # unittest_bit_vector add_executable(unittest_bit_vector test_bit_vector.cc diff --git a/src/test/common/test_htmlformatter.cc b/src/test/common/test_htmlformatter.cc new file mode 100644 index 00000000000..0a8d827b53a --- /dev/null +++ b/src/test/common/test_htmlformatter.cc @@ -0,0 +1,26 @@ +#include "gtest/gtest.h" + +#include "common/HTMLFormatter.h" +#include <sstream> +#include <string> + +using namespace ceph; + +TEST(htmlformatter, dump_format_large_item) +{ + std::stringstream sout; + HTMLFormatter formatter(false); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.flush(sout); + + std::string uri = base_url + "/" + bucket_name + "/" + object_key; + std::string expected_output = "<li>Location: " + uri + "</li>"; + + EXPECT_EQ(expected_output, sout.str()); +}
\ No newline at end of file diff --git a/src/test/common/test_intrusive_lru.cc b/src/test/common/test_intrusive_lru.cc index af8edb8e2bf..1410b73fcaa 100644 --- a/src/test/common/test_intrusive_lru.cc +++ b/src/test/common/test_intrusive_lru.cc @@ -177,9 +177,9 @@ TEST(LRU, clear_range) { auto [live_ref2, existed2] = cache.add(5, 4); ASSERT_FALSE(existed2); - cache.clear_range(0,4); + cache.clear_range(0, 4, [](auto&){}); - // Should not exists (Unreferenced): + // Should not exist { auto [ref, existed] = cache.add(1, 4); ASSERT_FALSE(existed); @@ -192,21 +192,27 @@ TEST(LRU, clear_range) { auto [ref, existed] = cache.add(3, 4); ASSERT_FALSE(existed); } - // Should exist (Still being referenced): { auto [ref, existed] = cache.add(4, 4); - ASSERT_TRUE(existed); + ASSERT_FALSE(existed); } - // Should exists (Still being referenced and wasn't removed) + ASSERT_TRUE(live_ref1->is_invalidated()); + // Should exist, wasn't removed) { auto [ref, existed] = cache.add(5, 4); ASSERT_TRUE(existed); } - // Test out of bound deletion: + ASSERT_FALSE(live_ref2->is_invalidated()); + // Test clear_range with right bound past last entry + cache.clear_range(3, 8, [](auto&){}); + ASSERT_TRUE(live_ref2->is_invalidated()); { - cache.clear_range(3,8); auto [ref, existed] = cache.add(4, 4); - ASSERT_TRUE(existed); + ASSERT_FALSE(existed); + } + { + auto [ref, existed] = cache.add(5, 4); + ASSERT_FALSE(existed); } { auto [ref, existed] = cache.add(3, 4); diff --git a/src/test/common/test_json_formatter.cc b/src/test/common/test_json_formatter.cc index 9cc19b24ad1..d0ddd262c0a 100644 --- a/src/test/common/test_json_formatter.cc +++ b/src/test/common/test_json_formatter.cc @@ -102,3 +102,27 @@ TEST(formatter, dump_inf_or_nan) EXPECT_EQ(parser.find_obj("nan_val")->get_data(), "null"); EXPECT_EQ(parser.find_obj("nan_val_alt")->get_data(), "null"); } + +TEST(formatter, dump_large_item) { + JSONFormatter formatter; + formatter.open_object_section("large_item"); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + std::string full_url = base_url + "/" + bucket_name + "/" + object_key; + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.close_section(); + bufferlist bl; + formatter.flush(bl); + + // std::cout << std::string(bl.c_str(), bl.length()) << std::endl; + + JSONParser parser; + parser.parse(bl.c_str(), bl.length()); + + EXPECT_TRUE(parser.parse(bl.c_str(), bl.length())); + EXPECT_EQ(parser.find_obj("Location")->get_data(), full_url); +} diff --git a/src/test/common/test_tableformatter.cc b/src/test/common/test_tableformatter.cc index b152014a2b5..90de133d315 100644 --- a/src/test/common/test_tableformatter.cc +++ b/src/test/common/test_tableformatter.cc @@ -250,6 +250,23 @@ TEST(tableformatter, multiline_keyval) EXPECT_EQ(cmp, sout.str()); } +TEST(tableformatter, dump_large_item) { + std::stringstream sout; + TableFormatter* formatter = (TableFormatter*) Formatter::create("table-kv"); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + std::string full_url = base_url + "/" + bucket_name + "/" + object_key; + formatter->dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + formatter->flush(sout); + delete formatter; + + std::string cmp = "key::Location=\"" + full_url + "\" \n"; + EXPECT_EQ(cmp, sout.str()); +} + /* * Local Variables: * compile-command: "cd ../.. ; make -j4 && diff --git a/src/test/common/test_xmlformatter.cc b/src/test/common/test_xmlformatter.cc index 9ac6dde456e..abbe9e4e25e 100644 --- a/src/test/common/test_xmlformatter.cc +++ b/src/test/common/test_xmlformatter.cc @@ -163,3 +163,25 @@ TEST(xmlformatter, pretty_lowercased_underscored) "<string_item>String</string_item>\n\n"; EXPECT_EQ(cmp, sout.str()); } + +TEST(xmlformatter, dump_format_large_item) +{ + std::stringstream sout; + XMLFormatter formatter( + true, // pretty + false, // lowercased + false); // underscored + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.flush(sout); + + std::string uri = base_url + "/" + bucket_name + "/" + object_key; + std::string expected_output = "<Location>" + uri + "</Location>\n\n"; + + EXPECT_EQ(expected_output, sout.str()); +}
\ No newline at end of file diff --git a/src/test/crimson/seastore/test_block.h b/src/test/crimson/seastore/test_block.h index e1fe8e06f8a..546f357dea0 100644 --- a/src/test/crimson/seastore/test_block.h +++ b/src/test/crimson/seastore/test_block.h @@ -39,8 +39,8 @@ struct test_block_delta_t { inline std::ostream &operator<<( std::ostream &lhs, const test_extent_desc_t &rhs) { - return lhs << "test_extent_desc_t(len=" << rhs.len - << ", checksum=" << rhs.checksum << ")"; + return lhs << "test_extent_desc_t(len=0x" << std::hex << rhs.len + << ", checksum=0x" << rhs.checksum << std::dec << ")"; } struct TestBlock : crimson::os::seastore::LogicalCachedExtent { diff --git a/src/test/crimson/seastore/test_btree_lba_manager.cc b/src/test/crimson/seastore/test_btree_lba_manager.cc index 9988df3a124..7874411e0ff 100644 --- a/src/test/crimson/seastore/test_btree_lba_manager.cc +++ b/src/test/crimson/seastore/test_btree_lba_manager.cc @@ -112,14 +112,22 @@ struct btree_test_base : seastar::future<> submit_transaction(TransactionRef t) { auto record = cache->prepare_record(*t, JOURNAL_SEQ_NULL, JOURNAL_SEQ_NULL); - return journal->submit_record(std::move(record), t->get_handle()).safe_then( - [this, t=std::move(t)](auto submit_result) mutable { - cache->complete_commit( - *t, + return seastar::do_with( + std::move(t), [this, record=std::move(record)](auto& _t) mutable { + auto& t = *_t; + return journal->submit_record( + std::move(record), + t.get_handle(), + t.get_src(), + [this, &t](auto submit_result) { + cache->complete_commit( + t, submit_result.record_block_base, submit_result.write_result.start_seq); - complete_commit(*t); - }).handle_error(crimson::ct_error::assert_all{}); + complete_commit(t); + } + ).handle_error(crimson::ct_error::assert_all{}); + }); } virtual LBAManager::mkfs_ret test_structure_setup(Transaction &t) = 0; @@ -149,7 +157,10 @@ struct btree_test_base : }).safe_then([this] { return seastar::do_with( cache->create_transaction( - Transaction::src_t::MUTATE, "test_set_up_fut", false), + Transaction::src_t::MUTATE, + "test_set_up_fut", + CACHE_HINT_TOUCH, + false), [this](auto &ref_t) { return with_trans_intr(*ref_t, [&](auto &t) { cache->init(); @@ -228,7 +239,10 @@ struct lba_btree_test : btree_test_base { template <typename F> auto lba_btree_update(F &&f) { auto tref = cache->create_transaction( - Transaction::src_t::MUTATE, "test_btree_update", false); + Transaction::src_t::MUTATE, + "test_btree_update", + CACHE_HINT_TOUCH, + false); auto &t = *tref; with_trans_intr( t, @@ -273,7 +287,10 @@ struct lba_btree_test : btree_test_base { template <typename F> auto lba_btree_read(F &&f) { auto t = cache->create_transaction( - Transaction::src_t::READ, "test_btree_read", false); + Transaction::src_t::READ, + "test_btree_read", + CACHE_HINT_TOUCH, + false); return with_trans_intr( *t, [this, f=std::forward<F>(f)](auto &t) mutable { @@ -421,7 +438,10 @@ struct btree_lba_manager_test : btree_test_base { auto create_transaction(bool create_fake_extent=true) { auto t = test_transaction_t{ cache->create_transaction( - Transaction::src_t::MUTATE, "test_mutate_lba", false), + Transaction::src_t::MUTATE, + "test_mutate_lba", + CACHE_HINT_TOUCH, + false), test_lba_mappings }; if (create_fake_extent) { @@ -437,7 +457,10 @@ struct btree_lba_manager_test : btree_test_base { auto create_weak_transaction() { auto t = test_transaction_t{ cache->create_transaction( - Transaction::src_t::READ, "test_read_weak", true), + Transaction::src_t::READ, + "test_read_weak", + CACHE_HINT_TOUCH, + true), test_lba_mappings }; return t; diff --git a/src/test/crimson/seastore/test_cbjournal.cc b/src/test/crimson/seastore/test_cbjournal.cc index d00a0f42729..47a08d68cbb 100644 --- a/src/test/crimson/seastore/test_cbjournal.cc +++ b/src/test/crimson/seastore/test_cbjournal.cc @@ -181,15 +181,20 @@ struct cbjournal_test_t : public seastar_test_suite_t, JournalTrimmer auto submit_record(record_t&& record) { entries.push_back(record); + entry_validator_t& back = entries.back(); OrderingHandle handle = get_dummy_ordering_handle(); - auto [addr, w_result] = cbj->submit_record( - std::move(record), - handle).unsafe_get(); - entries.back().seq = w_result.start_seq; - entries.back().entries = 1; - entries.back().magic = cbj->get_cjs().get_cbj_header().magic; - logger().debug("submit entry to addr {}", entries.back().seq); - return convert_paddr_to_abs_addr(entries.back().seq.offset); + cbj->submit_record( + std::move(record), + handle, + transaction_type_t::MUTATE, + [this, &back](auto locator) { + back.seq = locator.write_result.start_seq; + back.entries = 1; + back.magic = cbj->get_cjs().get_cbj_header().magic; + logger().debug("submit entry to addr {}", back.seq); + } + ).unsafe_get(); + return convert_paddr_to_abs_addr(back.seq.offset); } seastar::future<> tear_down_fut() final { diff --git a/src/test/crimson/seastore/test_seastore_cache.cc b/src/test/crimson/seastore/test_seastore_cache.cc index 6e24f436b98..fa774886139 100644 --- a/src/test/crimson/seastore/test_seastore_cache.cc +++ b/src/test/crimson/seastore/test_seastore_cache.cc @@ -87,7 +87,10 @@ struct cache_test_t : public seastar_test_suite_t { auto get_transaction() { return cache->create_transaction( - Transaction::src_t::MUTATE, "test_cache", false); + Transaction::src_t::MUTATE, + "test_cache", + CACHE_HINT_TOUCH, + false); } template <typename T, typename... Args> diff --git a/src/test/crimson/seastore/test_seastore_journal.cc b/src/test/crimson/seastore/test_seastore_journal.cc index 2eb791b1d46..04a99319b11 100644 --- a/src/test/crimson/seastore/test_seastore_journal.cc +++ b/src/test/crimson/seastore/test_seastore_journal.cc @@ -233,12 +233,17 @@ struct journal_test_t : seastar_test_suite_t, SegmentProvider, JournalTrimmer { auto submit_record(T&&... _record) { auto record{std::forward<T>(_record)...}; records.push_back(record); + record_validator_t& back = records.back(); OrderingHandle handle = get_dummy_ordering_handle(); - auto [addr, _] = journal->submit_record( + journal->submit_record( std::move(record), - handle).unsafe_get(); - records.back().record_final_offset = addr; - return addr; + handle, + transaction_type_t::MUTATE, + [&back](auto locator) { + back.record_final_offset = locator.record_block_base; + } + ).unsafe_get(); + return back.record_final_offset; } extent_t generate_extent(size_t blocks) { diff --git a/src/test/crimson/test_backfill.cc b/src/test/crimson/test_backfill.cc index 7e058c80ed6..e0fc5821d08 100644 --- a/src/test/crimson/test_backfill.cc +++ b/src/test/crimson/test_backfill.cc @@ -119,6 +119,11 @@ class BackfillFixture : public crimson::osd::BackfillState::BackfillListener { events_to_dispatch.emplace_back(event.intrusive_from_this()); } + template <class EventT> + void schedule_event_immediate(const EventT& event) { + events_to_dispatch.emplace_front(event.intrusive_from_this()); + } + // BackfillListener { void request_replica_scan( const pg_shard_t& target, @@ -188,12 +193,11 @@ public: struct PGFacade; void cancel() { - events_to_dispatch.clear(); - schedule_event(crimson::osd::BackfillState::CancelBackfill{}); + schedule_event_immediate(crimson::osd::BackfillState::CancelBackfill{}); } void resume() { - schedule_event(crimson::osd::BackfillState::Triggered{}); + schedule_event_immediate(crimson::osd::BackfillState::Triggered{}); } }; @@ -274,6 +278,9 @@ struct BackfillFixture::PGFacade : public crimson::osd::BackfillState::PGFacade return backfill_source.projected_log; } + std::ostream &print(std::ostream &out) const override { + return out << "FakePGFacade"; + } }; BackfillFixture::BackfillFixture( @@ -452,7 +459,69 @@ TEST(backfill, two_empty_replicas) EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); } -TEST(backfill, cancel_resume) +TEST(backfill, cancel_resume_middle_of_primaryscan) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_replicascan1) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_replicascan2) { const auto reference_store = FakeStore{ { { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, @@ -469,12 +538,43 @@ TEST(backfill, cancel_resume) EXPECT_CALL(cluster_fixture, backfilled); cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.cancel(); cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.resume(); cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_push1) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); @@ -483,7 +583,7 @@ TEST(backfill, cancel_resume) EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); } -TEST(backfill, cancel_resume_middle_of_scan) +TEST(backfill, cancel_resume_middle_of_push2) { const auto reference_store = FakeStore{ { { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, @@ -501,14 +601,46 @@ TEST(backfill, cancel_resume_middle_of_scan) EXPECT_CALL(cluster_fixture, backfilled); cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.cancel(); cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.resume(); cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_push3) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::RequestDone>(); cluster_fixture.next_till_done(); EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); diff --git a/src/test/fio/fio_librgw.cc b/src/test/fio/fio_librgw.cc index bac4ff2daac..89c09647b61 100644 --- a/src/test/fio/fio_librgw.cc +++ b/src/test/fio/fio_librgw.cc @@ -300,8 +300,6 @@ namespace { */ static void fio_librgw_cleanup(struct thread_data *td) { - int r = 0; - dprint(FD_IO, "fio_librgw_cleanup\n"); /* cleanup specific data */ @@ -312,9 +310,9 @@ namespace { data->release_handles(); if (data->bucket_fh) { - r = rgw_fh_rele(data->fs, data->bucket_fh, 0 /* flags */); + rgw_fh_rele(data->fs, data->bucket_fh, 0 /* flags */); } - r = rgw_umount(data->fs, RGW_UMOUNT_FLAG_NONE); + rgw_umount(data->fs, RGW_UMOUNT_FLAG_NONE); librgw_shutdown(data->rgw_h); td->io_ops_data = nullptr; delete data; diff --git a/src/test/libcephfs/test.cc b/src/test/libcephfs/test.cc index 6f10d2bbd4e..8760f594ae6 100644 --- a/src/test/libcephfs/test.cc +++ b/src/test/libcephfs/test.cc @@ -2706,6 +2706,19 @@ TEST(LibCephFS, Statxat) { ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFDIR); ASSERT_EQ(ceph_statxat(cmount, fd, rel_file_name_2, &stx, 0, 0), 0); ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + // test relative to root with empty relpath +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int dir_fd = ceph_openat(cmount, fd, dir_name, O_DIRECTORY | O_RDONLY, 0); + ASSERT_LE(0, dir_fd); + ASSERT_EQ(ceph_statxat(cmount, dir_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFDIR); + ASSERT_EQ(0, ceph_close(cmount, dir_fd)); + int file_fd = ceph_openat(cmount, fd, rel_file_name_2, O_RDONLY, 0); + ASSERT_LE(0, file_fd); + ASSERT_EQ(ceph_statxat(cmount, file_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + ASSERT_EQ(0, ceph_close(cmount, file_fd)); +#endif ASSERT_EQ(0, ceph_close(cmount, fd)); // test relative to dir @@ -2713,6 +2726,14 @@ TEST(LibCephFS, Statxat) { ASSERT_LE(0, fd); ASSERT_EQ(ceph_statxat(cmount, fd, rel_file_name_1, &stx, 0, 0), 0); ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + // test relative to dir with empty relpath +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int rel_file_fd = ceph_openat(cmount, fd, rel_file_name_1, O_RDONLY, 0); + ASSERT_LE(0, rel_file_fd); + ASSERT_EQ(ceph_statxat(cmount, rel_file_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + ASSERT_EQ(0, ceph_close(cmount, rel_file_fd)); +#endif // delete the dirtree, recreate and verify ASSERT_EQ(0, ceph_unlink(cmount, file_path)); @@ -3265,6 +3286,13 @@ TEST(LibCephFS, Chownat) { // change ownership to nobody -- we assume nobody exists and id is always 65534 ASSERT_EQ(ceph_conf_set(cmount, "client_permissions", "0"), 0); ASSERT_EQ(ceph_chownat(cmount, fd, rel_file_path, 65534, 65534, 0), 0); + // change relative fd ownership with AT_EMPTY_PATH +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int file_fd = ceph_openat(cmount, fd, rel_file_path, O_RDONLY, 0); + ASSERT_LE(0, file_fd); + ASSERT_EQ(ceph_chownat(cmount, file_fd, "", 65534, 65534, AT_EMPTY_PATH), 0); + ceph_close(cmount, file_fd); +#endif ASSERT_EQ(ceph_conf_set(cmount, "client_permissions", "1"), 0); ceph_close(cmount, fd); diff --git a/src/test/librados/aio.cc b/src/test/librados/aio.cc index 68587fe87d1..7fb90bdd38e 100644 --- a/src/test/librados/aio.cc +++ b/src/test/librados/aio.cc @@ -1722,3 +1722,59 @@ TEST(LibRadosAioEC, MultiWrite) { rados_aio_release(my_completion2); rados_aio_release(my_completion3); } + +TEST(LibRadosAio, CancelBeforeSubmit) { + AioTestData test_data; + ASSERT_EQ("", test_data.init()); + + rados_completion_t completion; + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + rados_aio_release(completion); +} + +TEST(LibRadosAio, CancelBeforeComplete) { + AioTestData test_data; + ASSERT_EQ("", test_data.init()); + + // cancellation tests are racy, so retry if completion beats the cancellation + int ret = 0; + int tries = 10; + do { + rados_completion_t completion; + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + char buf[128]; + ASSERT_EQ(0, rados_aio_read(test_data.m_ioctx, "nonexistent", + completion, buf, sizeof(buf), 0)); + + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + { + TestAlarm alarm; + ASSERT_EQ(0, rados_aio_wait_for_complete(completion)); + } + ret = rados_aio_get_return_value(completion); + rados_aio_release(completion); + } while (ret == -ENOENT && --tries); + + ASSERT_EQ(-ECANCELED, ret); +} + +TEST(LibRadosAio, CancelAfterComplete) { + AioTestData test_data; + rados_completion_t completion; + ASSERT_EQ("", test_data.init()); + + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + char buf[128]; + ASSERT_EQ(0, rados_aio_read(test_data.m_ioctx, "nonexistent", + completion, buf, sizeof(buf), 0)); + + { + TestAlarm alarm; + ASSERT_EQ(0, rados_aio_wait_for_complete(completion)); + } + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + ASSERT_EQ(-ENOENT, rados_aio_get_return_value(completion)); + rados_aio_release(completion); +} diff --git a/src/test/librados/aio_cxx.cc b/src/test/librados/aio_cxx.cc index a70af050d70..5e35869b5c2 100644 --- a/src/test/librados/aio_cxx.cc +++ b/src/test/librados/aio_cxx.cc @@ -2467,3 +2467,92 @@ TEST(LibRadosAio, MultiReads) { ASSERT_EQ(0, memcmp(buf, bl.c_str(), sizeof(buf))); } } + +// cancellation test fixture for global setup/teardown +// parameterized to test both IoCtx::aio_cancel() and AioCompletion::cancel() +class Cancel : public ::testing::TestWithParam<bool> { + static constexpr auto pool_prefix = "ceph_test_rados_api_pp"; + static Rados rados; + static std::string pool_name; + protected: + static IoCtx ioctx; + public: + static void SetUpTestCase() { + pool_name = get_temp_pool_name(pool_prefix); + ASSERT_EQ("", create_one_pool_pp(pool_name, rados)); + ASSERT_EQ(0, rados.ioctx_create(pool_name.c_str(), ioctx)); + } + static void TearDownTestCase() { + destroy_one_pool_pp(pool_name, rados); + } +}; +Rados Cancel::rados; +std::string Cancel::pool_name; +IoCtx Cancel::ioctx; + +TEST_P(Cancel, BeforeSubmit) +{ + const bool use_completion = GetParam(); + + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + if (use_completion) { + ASSERT_EQ(0, c->cancel()); + } else { + ASSERT_EQ(0, ioctx.aio_cancel(c.get())); + } +} + +TEST_P(Cancel, BeforeComplete) +{ + const bool use_completion = GetParam(); + + // cancellation tests are racy, so retry if completion beats the cancellation + int ret = 0; + int tries = 10; + do { + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + ObjectReadOperation op; + op.assert_exists(); + ioctx.aio_operate("nonexistent", c.get(), &op, nullptr); + + if (use_completion) { + EXPECT_EQ(0, c->cancel()); + } else { + EXPECT_EQ(0, ioctx.aio_cancel(c.get())); + } + { + TestAlarm alarm; + ASSERT_EQ(0, c->wait_for_complete()); + } + ret = c->get_return_value(); + } while (ret == -ENOENT && --tries); + + EXPECT_EQ(-ECANCELED, ret); +} + +TEST_P(Cancel, AfterComplete) +{ + const bool use_completion = GetParam(); + + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + ObjectReadOperation op; + op.assert_exists(); + ioctx.aio_operate("nonexistent", c.get(), &op, nullptr); + { + TestAlarm alarm; + ASSERT_EQ(0, c->wait_for_complete()); + } + if (use_completion) { + EXPECT_EQ(0, c->cancel()); + } else { + EXPECT_EQ(0, ioctx.aio_cancel(c.get())); + } + EXPECT_EQ(-ENOENT, c->get_return_value()); +} + +std::string cancel_test_name(const testing::TestParamInfo<Cancel::ParamType>& info) +{ + return info.param ? "cancel" : "aio_cancel"; +} + +INSTANTIATE_TEST_SUITE_P(LibRadosAio, Cancel, testing::Bool(), cancel_test_name); diff --git a/src/test/librados/asio.cc b/src/test/librados/asio.cc index 01ebb957150..500f36508a7 100644 --- a/src/test/librados/asio.cc +++ b/src/test/librados/asio.cc @@ -21,10 +21,14 @@ #include <boost/range/begin.hpp> #include <boost/range/end.hpp> +#include <boost/asio/bind_cancellation_slot.hpp> +#include <boost/asio/cancellation_signal.hpp> #include <boost/asio/io_context.hpp> #include <boost/asio/spawn.hpp> #include <boost/asio/use_future.hpp> +#include <optional> + #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context @@ -78,6 +82,15 @@ void rethrow(std::exception_ptr eptr) { if (eptr) std::rethrow_exception(eptr); } +auto capture(std::optional<error_code>& out) { + return [&out] (error_code ec, ...) { out = ec; }; +} + +auto capture(boost::asio::cancellation_signal& signal, + std::optional<error_code>& out) { + return boost::asio::bind_cancellation_slot(signal.slot(), capture(out)); +} + TEST_F(AsioRados, AsyncReadCallback) { boost::asio::io_context service; @@ -385,6 +398,130 @@ TEST_F(AsioRados, AsyncWriteOperationYield) service.run(); } +// FIXME: this crashes on windows with: +// Thread 1 received signal SIGILL, Illegal instruction. +#ifndef _WIN32 + +TEST_F(AsioRados, AsyncReadOperationCancelTerminal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectReadOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::terminal); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncReadOperationCancelTotal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectReadOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::total); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncWriteOperationCancelTerminal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectWriteOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::terminal); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncWriteOperationCancelTotal) +{ + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> ec; + + librados::ObjectWriteOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, ec)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(ec); + + // noop, write only supports terminal + signal.emit(boost::asio::cancellation_type::total); + + service.run(); + ASSERT_TRUE(ec); + EXPECT_EQ(ec, std::errc::no_such_file_or_directory); + + signal.emit(boost::asio::cancellation_type::all); // noop +} + +#endif // not _WIN32 + int main(int argc, char **argv) { auto args = argv_to_vec(argc, argv); diff --git a/src/test/librbd/migration/test_mock_HttpClient.cc b/src/test/librbd/migration/test_mock_HttpClient.cc index f3888755c79..901c4231dd0 100644 --- a/src/test/librbd/migration/test_mock_HttpClient.cc +++ b/src/test/librbd/migration/test_mock_HttpClient.cc @@ -307,7 +307,7 @@ TEST_F(TestMockMigrationHttpClient, OpenCloseHttps) { boost::asio::ssl::context ssl_context{boost::asio::ssl::context::tlsv12}; load_server_certificate(ssl_context); - boost::beast::ssl_stream<boost::beast::tcp_stream> ssl_stream{ + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_stream{ std::move(socket), ssl_context}; C_SaferCond on_ssl_handshake_ctx; @@ -341,7 +341,7 @@ TEST_F(TestMockMigrationHttpClient, OpenHttpsHandshakeFail) { boost::asio::ssl::context ssl_context{boost::asio::ssl::context::tlsv12}; load_server_certificate(ssl_context); - boost::beast::ssl_stream<boost::beast::tcp_stream> ssl_stream{ + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_stream{ std::move(socket), ssl_context}; C_SaferCond on_ssl_handshake_ctx; diff --git a/src/test/librbd/test_internal.cc b/src/test/librbd/test_internal.cc index 008fdcfa7be..8f6cbb9e807 100644 --- a/src/test/librbd/test_internal.cc +++ b/src/test/librbd/test_internal.cc @@ -1571,6 +1571,83 @@ TEST_F(TestInternal, FlattenNoEmptyObjects) rados_ioctx_destroy(d_ioctx); } +TEST_F(TestInternal, FlattenInconsistentObjectMap) +{ + REQUIRE_FEATURE(RBD_FEATURE_LAYERING | RBD_FEATURE_OBJECT_MAP); + REQUIRE(!is_feature_enabled(RBD_FEATURE_STRIPINGV2)); + + librbd::ImageCtx* ictx; + ASSERT_EQ(0, open_image(m_image_name, &ictx)); + + librbd::NoOpProgressContext no_op; + ASSERT_EQ(0, ictx->operations->resize((1 << ictx->order) * 5, true, no_op)); + + bufferlist bl; + bl.append(std::string(256, '1')); + for (int i = 1; i < 5; i++) { + ASSERT_EQ(256, api::Io<>::write(*ictx, (1 << ictx->order) * i, 256, + bufferlist{bl}, 0)); + } + + ASSERT_EQ(0, snap_create(*ictx, "snap")); + ASSERT_EQ(0, snap_protect(*ictx, "snap")); + + uint64_t features; + ASSERT_EQ(0, librbd::get_features(ictx, &features)); + + std::string clone_name = get_temp_image_name(); + int order = ictx->order; + ASSERT_EQ(0, librbd::clone(m_ioctx, m_image_name.c_str(), "snap", m_ioctx, + clone_name.c_str(), features, &order, 0, 0)); + + close_image(ictx); + ASSERT_EQ(0, open_image(clone_name, &ictx)); + + C_SaferCond lock_ctx; + { + std::shared_lock owner_locker{ictx->owner_lock}; + ictx->exclusive_lock->try_acquire_lock(&lock_ctx); + } + ASSERT_EQ(0, lock_ctx.wait()); + ASSERT_TRUE(ictx->exclusive_lock->is_lock_owner()); + + ceph::BitVector<2> inconsistent_object_map; + inconsistent_object_map.resize(5); + inconsistent_object_map[0] = OBJECT_NONEXISTENT; + inconsistent_object_map[1] = OBJECT_NONEXISTENT; + inconsistent_object_map[2] = OBJECT_EXISTS; + inconsistent_object_map[3] = OBJECT_EXISTS_CLEAN; + // OBJECT_PENDING shouldn't happen within parent overlap, but test + // anyway + inconsistent_object_map[4] = OBJECT_PENDING; + + auto object_map = new librbd::ObjectMap<>(*ictx, CEPH_NOSNAP); + C_SaferCond save_ctx; + { + std::shared_lock owner_locker{ictx->owner_lock}; + std::unique_lock image_locker{ictx->image_lock}; + object_map->set_object_map(inconsistent_object_map); + object_map->aio_save(&save_ctx); + } + ASSERT_EQ(0, save_ctx.wait()); + object_map->put(); + + close_image(ictx); + ASSERT_EQ(0, open_image(clone_name, &ictx)); + ASSERT_EQ(0, ictx->operations->flatten(no_op)); + + bufferptr read_ptr(256); + bufferlist read_bl; + read_bl.push_back(read_ptr); + + librbd::io::ReadResult read_result{&read_bl}; + for (int i = 1; i < 5; i++) { + ASSERT_EQ(256, api::Io<>::read(*ictx, (1 << ictx->order) * i, 256, + librbd::io::ReadResult{read_result}, 0)); + EXPECT_TRUE(bl.contents_equal(read_bl)); + } +} + TEST_F(TestInternal, PoolMetadataConfApply) { REQUIRE_FORMAT_V2(); diff --git a/src/test/objectstore/CMakeLists.txt b/src/test/objectstore/CMakeLists.txt index bddff3f6727..08388640043 100644 --- a/src/test/objectstore/CMakeLists.txt +++ b/src/test/objectstore/CMakeLists.txt @@ -48,6 +48,18 @@ add_executable(unittest_rocksdb_option add_ceph_unittest(unittest_rocksdb_option) target_link_libraries(unittest_rocksdb_option global os ${BLKID_LIBRARIES}) +# ceph_test_bluefs (a clone of unittest_bluefs) +add_executable(ceph_test_bluefs + test_bluefs.cc + ) +target_link_libraries(ceph_test_bluefs + os + global + ${UNITTEST_LIBS} + ) +install(TARGETS ceph_test_bluefs + DESTINATION ${CMAKE_INSTALL_BINDIR}) + if(WITH_EVENTTRACE) add_dependencies(os eventtrace_tp) endif() diff --git a/src/test/objectstore/ObjectStoreImitator.h b/src/test/objectstore/ObjectStoreImitator.h index d71d7f2fe58..875f9041b83 100644 --- a/src/test/objectstore/ObjectStoreImitator.h +++ b/src/test/objectstore/ObjectStoreImitator.h @@ -347,6 +347,16 @@ public: ) override { return {}; } + + int omap_iterate(CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + /// [in] where the iterator should point to at the beginning + omap_iter_seek_t start_from, + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override { + return 0; + } + void set_fsid(uuid_d u) override {} uuid_d get_fsid() override { return {}; } uint64_t estimate_objects_overhead(uint64_t num_objects) override { diff --git a/src/test/objectstore/allocsim/ops_replayer.cc b/src/test/objectstore/allocsim/ops_replayer.cc index fd947f5c454..c5908d9f576 100644 --- a/src/test/objectstore/allocsim/ops_replayer.cc +++ b/src/test/objectstore/allocsim/ops_replayer.cc @@ -1,4 +1,5 @@ #include <algorithm> +#include <functional> #include <boost/program_options/value_semantic.hpp> #include <cassert> #include <cctype> @@ -13,26 +14,46 @@ #include <fstream> #include <filesystem> #include <mutex> -#include "include/rados/buffer_fwd.h" -#include "include/rados/librados.hpp" #include <atomic> -#include <fmt/format.h> #include <map> #include <memory> #include <random> #include <string> #include <iostream> #include <vector> +#include <format> + +#include <fmt/format.h> #include <boost/program_options/variables_map.hpp> #include <boost/program_options/parsers.hpp> +#include "include/rados/buffer_fwd.h" +#include "include/rados/librados.hpp" + namespace po = boost::program_options; using namespace std; using namespace ceph; +namespace settings { + +// Returns a function which restricts a value to a specified range by throwing if it is not in range: +// (Note: std::clamp() does not throw.) +auto clamp_or_throw(auto min, auto max) +{ + return [=](auto& x) { + if(std::less<>{}(x, min) or std::greater<>{}(x, max)) { + throw std::out_of_range(fmt::format("value expected between {} and {}, but got {}", min, max, x)); + } + + return x; + }; +} + +} // namespace settings + // compare shared_ptr<string> struct StringPtrCompare { @@ -338,8 +359,8 @@ int main(int argc, char** argv) { // options uint64_t io_depth = 8; - uint64_t nparser_threads = 16; - uint64_t nworker_threads = 16; + int nparser_threads = 16; + int nworker_threads = 16; string file("input.txt"); string ceph_conf_path("./ceph.conf"); string pool("test_pool"); @@ -351,8 +372,8 @@ int main(int argc, char** argv) { ("input-files,i", po::value<vector<string>>()->multitoken(), "List of input files (output of op_scraper.py). Multiple files will be merged and sorted by time order") ("ceph-conf", po::value<string>(&ceph_conf_path)->default_value("ceph.conf"), "Path to ceph conf") ("io-depth", po::value<uint64_t>(&io_depth)->default_value(64), "I/O depth") - ("parser-threads", po::value<uint64_t>(&nparser_threads)->default_value(16), "Number of parser threads") - ("worker-threads", po::value<uint64_t>(&nworker_threads)->default_value(16), "Number of I/O worker threads") + ("parser-threads", po::value<int>(&nparser_threads)->default_value(16)->notifier(settings::clamp_or_throw(1, 256)), "Number of parser threads") + ("worker-threads", po::value<int>(&nworker_threads)->default_value(16)->notifier(settings::clamp_or_throw(1, 256)), "Number of I/O worker threads") ("pool", po::value<string>(&pool)->default_value("test_pool"), "Pool to use for I/O") ("skip-do-ops", po::bool_switch(&skip_do_ops)->default_value(false), "Skip doing operations") ; diff --git a/src/test/objectstore/test_bluefs.cc b/src/test/objectstore/test_bluefs.cc index e2f95dd6c80..60147b5397c 100644 --- a/src/test/objectstore/test_bluefs.cc +++ b/src/test/objectstore/test_bluefs.cc @@ -23,6 +23,11 @@ using namespace std; +// some test should not be executed on jenkins make check +#define SKIP_JENKINS() \ + if (getenv("JENKINS_HOME") != nullptr) GTEST_SKIP_("test disabled on jenkins"); + + std::unique_ptr<char[]> gen_buffer(uint64_t size) { std::unique_ptr<char[]> buffer = std::make_unique<char[]>(size); @@ -174,6 +179,7 @@ TEST(BlueFS, small_appends) { } TEST(BlueFS, very_large_write) { + SKIP_JENKINS(); // we'll write a ~5G file, so allocate more than that for the whole fs uint64_t size = 1048576 * 1024 * 6ull; TempBdev bdev{size}; @@ -248,6 +254,7 @@ TEST(BlueFS, very_large_write) { } TEST(BlueFS, very_large_write2) { + SKIP_JENKINS(); // we'll write a ~5G file, so allocate more than that for the whole fs uint64_t size_full = 1048576 * 1024 * 6ull; uint64_t size = 1048576 * 1024 * 5ull; @@ -1601,6 +1608,91 @@ TEST(BlueFS, test_log_runway_advance_seq) { fs.compact_log(); } +TEST(BlueFS, test_69481_truncate_corrupts_log) { + uint64_t size = 1048576 * 128; + TempBdev bdev{size}; + BlueFS fs(g_ceph_context); + ASSERT_EQ(0, fs.add_block_device(BlueFS::BDEV_DB, bdev.path, false)); + uuid_d fsid; + ASSERT_EQ(0, fs.mkfs(fsid, { BlueFS::BDEV_DB, false, false })); + ASSERT_EQ(0, fs.mount()); + ASSERT_EQ(0, fs.maybe_verify_layout({ BlueFS::BDEV_DB, false, false })); + + BlueFS::FileWriter *f = nullptr; + BlueFS::FileWriter *a = nullptr; + ASSERT_EQ(0, fs.mkdir("dir")); + ASSERT_EQ(0, fs.open_for_write("dir", "test-file", &f, false)); + ASSERT_EQ(0, fs.open_for_write("dir", "just-allocate", &a, false)); + + // create 4 distinct extents in file f + // a is here only to prevent f from merging extents together + fs.preallocate(f->file, 0, 0x10000); + fs.preallocate(a->file, 0, 0x10000); + fs.preallocate(f->file, 0, 0x20000); + fs.preallocate(a->file, 0, 0x20000); + fs.preallocate(f->file, 0, 0x30000); + fs.preallocate(a->file, 0, 0x30000); + fs.preallocate(f->file, 0, 0x40000); + fs.preallocate(a->file, 0, 0x40000); + fs.close_writer(a); + + fs.truncate(f, 0); + fs.fsync(f); + + bufferlist bl; + bl.append(std::string(" ", 0x15678)); + f->append(bl); + fs.truncate(f, 0x15678); + fs.fsync(f); + fs.close_writer(f); + + fs.umount(); + // remount to verify + ASSERT_EQ(0, fs.mount()); + fs.umount(); +} + +TEST(BlueFS, test_69481_truncate_asserts) { + uint64_t size = 1048576 * 128; + TempBdev bdev{size}; + BlueFS fs(g_ceph_context); + ASSERT_EQ(0, fs.add_block_device(BlueFS::BDEV_DB, bdev.path, false)); + uuid_d fsid; + ASSERT_EQ(0, fs.mkfs(fsid, { BlueFS::BDEV_DB, false, false })); + ASSERT_EQ(0, fs.mount()); + ASSERT_EQ(0, fs.maybe_verify_layout({ BlueFS::BDEV_DB, false, false })); + + BlueFS::FileWriter *f = nullptr; + BlueFS::FileWriter *a = nullptr; + ASSERT_EQ(0, fs.mkdir("dir")); + ASSERT_EQ(0, fs.open_for_write("dir", "test-file", &f, false)); + ASSERT_EQ(0, fs.open_for_write("dir", "just-allocate", &a, false)); + + // create 4 distinct extents in file f + // a is here only to prevent f from merging extents together + fs.preallocate(f->file, 0, 0x10000); + fs.preallocate(a->file, 0, 0x10000); + fs.preallocate(f->file, 0, 0x20000); + fs.preallocate(a->file, 0, 0x20000); + fs.preallocate(f->file, 0, 0x30000); + fs.preallocate(a->file, 0, 0x30000); + fs.preallocate(f->file, 0, 0x40000); + fs.preallocate(a->file, 0, 0x40000); + fs.close_writer(a); + + fs.truncate(f, 0); + fs.fsync(f); + + bufferlist bl; + bl.append(std::string(" ", 0x35678)); + f->append(bl); + fs.truncate(f, 0x35678); + fs.fsync(f); + fs.close_writer(f); + + fs.umount(); +} + int main(int argc, char **argv) { auto args = argv_to_vec(argc, argv); map<string,string> defaults = { diff --git a/src/test/osd/CMakeLists.txt b/src/test/osd/CMakeLists.txt index f2d1471e22e..798558ebbe0 100644 --- a/src/test/osd/CMakeLists.txt +++ b/src/test/osd/CMakeLists.txt @@ -22,7 +22,7 @@ install(TARGETS add_executable(ceph_test_rados_io_sequence ${CMAKE_CURRENT_SOURCE_DIR}/ceph_test_rados_io_sequence.cc) target_link_libraries(ceph_test_rados_io_sequence - librados global object_io_exerciser) + librados global object_io_exerciser json_structures) install(TARGETS ceph_test_rados_io_sequence DESTINATION ${CMAKE_INSTALL_BINDIR}) diff --git a/src/test/osd/ceph_test_rados_io_sequence.cc b/src/test/osd/ceph_test_rados_io_sequence.cc index 4a768a016e2..96808ea37e5 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.cc +++ b/src/test/osd/ceph_test_rados_io_sequence.cc @@ -1,83 +1,104 @@ #include "ceph_test_rados_io_sequence.h" +#include <boost/asio/io_context.hpp> #include <iostream> #include <vector> -#include <boost/asio/io_context.hpp> - -#include "include/random.h" - -#include "librados/librados_asio.h" -#include "common/ceph_argparse.h" -#include "include/interval_set.h" -#include "global/global_init.h" -#include "global/global_context.h" +#include "common/Formatter.h" #include "common/Thread.h" +#include "common/ceph_argparse.h" +#include "common/ceph_json.h" #include "common/debug.h" #include "common/dout.h" #include "common/split.h" #include "common/strtol.h" // for strict_iecstrtoll() +#include "common/ceph_json.h" +#include "common/Formatter.h" #include "common/io_exerciser/DataGenerator.h" +#include "common/io_exerciser/EcIoSequence.h" +#include "common/io_exerciser/IoOp.h" +#include "common/io_exerciser/IoSequence.h" #include "common/io_exerciser/Model.h" #include "common/io_exerciser/ObjectModel.h" #include "common/io_exerciser/RadosIo.h" -#include "common/io_exerciser/IoOp.h" -#include "common/io_exerciser/IoSequence.h" +#include "common/json/BalancerStructures.h" +#include "common/json/ConfigStructures.h" +#include "common/json/OSDStructures.h" +#include "fmt/format.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/interval_set.h" +#include "include/random.h" +#include "json_spirit/json_spirit.h" +#include "librados/librados_asio.h" #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context +using OpType = ceph::io_exerciser::OpType; + +using DoneOp = ceph::io_exerciser::DoneOp; +using BarrierOp = ceph::io_exerciser::BarrierOp; +using CreateOp = ceph::io_exerciser::CreateOp; +using RemoveOp = ceph::io_exerciser::RemoveOp; +using SingleReadOp = ceph::io_exerciser::SingleReadOp; +using DoubleReadOp = ceph::io_exerciser::DoubleReadOp; +using TripleReadOp = ceph::io_exerciser::TripleReadOp; +using SingleWriteOp = ceph::io_exerciser::SingleWriteOp; +using DoubleWriteOp = ceph::io_exerciser::DoubleWriteOp; +using TripleWriteOp = ceph::io_exerciser::TripleWriteOp; +using SingleFailedWriteOp = ceph::io_exerciser::SingleFailedWriteOp; +using DoubleFailedWriteOp = ceph::io_exerciser::DoubleFailedWriteOp; +using TripleFailedWriteOp = ceph::io_exerciser::TripleFailedWriteOp; + namespace { - struct Size {}; - void validate(boost::any& v, const std::vector<std::string>& values, - Size *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - - std::string parse_error; - uint64_t size = strict_iecstrtoll(s, &parse_error); - if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); - } - v = boost::any(size); - } - - struct Pair {}; - void validate(boost::any& v, const std::vector<std::string>& values, - Pair *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - auto part = ceph::split(s).begin(); - std::string parse_error; - int first = strict_iecstrtoll(*part++, &parse_error); - int second = strict_iecstrtoll(*part, &parse_error); - if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); - } - v = boost::any(std::pair<int,int>{first,second}); - } - - struct PluginString {}; - void validate(boost::any& v, const std::vector<std::string>& values, - PluginString *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - - const std::string_view* pluginIt = std::find( - ceph::io_sequence::tester::pluginChoices.begin(), - ceph::io_sequence::tester::pluginChoices.end(), - s - ); - if(ceph::io_sequence::tester::pluginChoices.end() == pluginIt) - { - throw po::validation_error(po::validation_error::invalid_option_value); - } +struct Size {}; +void validate(boost::any& v, const std::vector<std::string>& values, + Size* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); - v = boost::any(*pluginIt); + std::string parse_error; + uint64_t size = strict_iecstrtoll(s, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); } + v = boost::any(size); +} + +struct Pair {}; +void validate(boost::any& v, const std::vector<std::string>& values, + Pair* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); + auto part = ceph::split(s).begin(); + std::string parse_error; + int first = strict_iecstrtoll(*part++, &parse_error); + int second = strict_iecstrtoll(*part, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + v = boost::any(std::pair<int, int>{first, second}); +} + +struct PluginString {}; +void validate(boost::any& v, const std::vector<std::string>& values, + PluginString* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); + + const std::string_view* pluginIt = + std::find(ceph::io_sequence::tester::pluginChoices.begin(), + ceph::io_sequence::tester::pluginChoices.end(), s); + if (ceph::io_sequence::tester::pluginChoices.end() == pluginIt) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + + v = boost::any(*pluginIt); +} - constexpr std::string_view usage[] = { +constexpr std::string_view usage[] = { "Basic usage:", "", "ceph_test_rados_io_sequence", @@ -119,103 +140,99 @@ namespace { "\t are specified with unit of blocksize. Supported commands:", "\t\t create <len>", "\t\t remove", - "\t\t read|write <off> <len>", - "\t\t read2|write2 <off> <len> <off> <len>", - "\t\t read3|write3 <off> <len> <off> <len> <off> <len>", - "\t\t done" - }; - - po::options_description get_options_description() - { - po::options_description desc("ceph_test_rados_io options"); - desc.add_options() - ("help,h", - "show help message") - ("listsequence,l", - "show list of sequences") - ("dryrun,d", - "test sequence, do not issue any I/O") - ("verbose", - "more verbose output during test") - ("sequence,s", po::value<int>(), - "test specified sequence") - ("seed", po::value<int>(), - "seed for whole test") - ("seqseed", po::value<int>(), - "seed for sequence") - ("blocksize,b", po::value<Size>(), - "block size (default 2048)") - ("chunksize,c", po::value<Size>(), - "chunk size (default 4096)") - ("pool,p", po::value<std::string>(), - "pool name") - ("object,o", po::value<std::string>()->default_value("test"), - "object name") - ("km", po::value<Pair>(), - "k,m EC pool profile (default 2,2)") - ("plugin", po::value<PluginString>(), - "EC plugin (isa or jerasure)") - ("objectsize", po::value<Pair>(), - "min,max object size in blocks (default 1,32)") - ("threads,t", po::value<int>(), - "number of threads of I/O per object (default 1)") - ("parallel,p", po::value<int>()->default_value(1), - "number of objects to exercise in parallel") - ("interactive", - "interactive mode, execute IO commands from stdin"); - - return desc; - } - - int parse_io_seq_options( - po::variables_map& vm, - int argc, - char** argv) - { - std::vector<std::string> unrecognized_options; - try { - po::options_description desc = get_options_description(); - - auto parsed = po::command_line_parser(argc, argv) - .options(desc) - .allow_unregistered() - .run(); - po::store(parsed, vm); - po::notify(vm); - unrecognized_options = po::collect_unrecognized(parsed.options, - po::include_positional); - - if (!unrecognized_options.empty()) - { - std::stringstream ss; - ss << "Unrecognised command options supplied: "; - while (unrecognized_options.size() > 1) - { - ss << unrecognized_options.back().c_str() << ", "; - unrecognized_options.pop_back(); - } - ss << unrecognized_options.back(); - dout(0) << ss.str() << dendl; - return 1; + "\t\t read|write|failedwrite <off> <len>", + "\t\t read2|write2|failedwrite2 <off> <len> <off> <len>", + "\t\t read3|write3|failedwrite3 <off> <len> <off> <len> <off> <len>", + "\t\t injecterror <type> <shard> <good_count> <fail_count>", + "\t\t clearinject <type> <shard>", + "\t\t done"}; + +po::options_description get_options_description() { + po::options_description desc("ceph_test_rados_io options"); + desc.add_options()("help,h", "show help message")("listsequence,l", + "show list of sequences")( + "dryrun,d", "test sequence, do not issue any I/O")( + "verbose", "more verbose output during test")( + "sequence,s", po::value<int>(), "test specified sequence")( + "seed", po::value<int>(), "seed for whole test")( + "seqseed", po::value<int>(), "seed for sequence")( + "blocksize,b", po::value<Size>(), "block size (default 2048)")( + "chunksize,c", po::value<Size>(), "chunk size (default 4096)")( + "pool,p", po::value<std::string>(), "pool name")( + "object,o", po::value<std::string>()->default_value("test"), + "object name")("km", po::value<Pair>(), + "k,m EC pool profile (default 2,2)")( + "plugin", po::value<PluginString>(), "EC plugin (isa or jerasure)")( + "objectsize", po::value<Pair>(), + "min,max object size in blocks (default 1,32)")( + "threads,t", po::value<int>(), + "number of threads of I/O per object (default 1)")( + "parallel,p", po::value<int>()->default_value(1), + "number of objects to exercise in parallel")( + "testrecovery", + "Inject errors during sequences to test recovery processes of OSDs")( + "interactive", "interactive mode, execute IO commands from stdin")( + "allow_pool_autoscaling", + "Allows pool autoscaling. Disabled by default.")( + "allow_pool_balancer", "Enables pool balancing. Disabled by default.")( + "allow_pool_deep_scrubbing", + "Enables pool deep scrub. Disabled by default.")( + "allow_pool_scrubbing", "Enables pool scrubbing. Disabled by default."); + + return desc; +} + +int parse_io_seq_options(po::variables_map& vm, int argc, char** argv) { + std::vector<std::string> unrecognized_options; + try { + po::options_description desc = get_options_description(); + + auto parsed = po::command_line_parser(argc, argv) + .options(desc) + .allow_unregistered() + .run(); + po::store(parsed, vm); + po::notify(vm); + unrecognized_options = + po::collect_unrecognized(parsed.options, po::include_positional); + + if (!unrecognized_options.empty()) { + std::stringstream ss; + ss << "Unrecognised command options supplied: "; + while (unrecognized_options.size() > 1) { + ss << unrecognized_options.back().c_str() << ", "; + unrecognized_options.pop_back(); } - } catch(const po::error& e) { - std::cerr << "error: " << e.what() << std::endl; + ss << unrecognized_options.back(); + dout(0) << ss.str() << dendl; return 1; } - - return 0; + } catch (const po::error& e) { + std::cerr << "error: " << e.what() << std::endl; + return 1; } + + return 0; } +template <typename S> +int send_mon_command(S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, Formatter* f) { + std::ostringstream oss; + encode_json(name, s, f); + f->flush(oss); + int rc = rados.mon_command(oss.str(), inbl, outbl, nullptr); + return rc; +} + +} // namespace + template <typename T, int N, const std::array<T, N>& Ts> -ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts> - ::ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - const std::string& option_name, - bool set_forced, - bool select_first) - : rng(rng), - option_name(option_name) { +ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>:: + ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, const std::string& option_name, + bool set_forced, bool select_first) + : rng(rng), option_name(option_name) { if (set_forced && vm.count(option_name)) { force_value = vm[option_name].as<T>(); } @@ -226,76 +243,54 @@ ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts> } template <typename T, int N, const std::array<T, N>& Ts> -bool ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::isForced() -{ +bool ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::isForced() { return force_value.has_value(); } template <typename T, int N, const std::array<T, N>& Ts> -const T ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::choose() -{ +const T ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::choose() { if (force_value.has_value()) { return *force_value; } else if (first_value.has_value()) { return *std::exchange(first_value, std::nullopt); } else { - return choices[rng(N-1)]; + return choices[rng(N - 1)]; } } - - ceph::io_sequence::tester::SelectObjectSize::SelectObjectSize( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "objectsize", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "objectsize", true, true) {} ceph::io_sequence::tester::SelectBlockSize::SelectBlockSize( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "blocksize", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "blocksize", true, true) {} ceph::io_sequence::tester::SelectNumThreads::SelectNumThreads( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "threads", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "threads", true, true) {} ceph::io_sequence::tester::SelectSeqRange::SelectSeqRange( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "sequence", false, false) -{ + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "sequence", false, false) { if (vm.count(option_name)) { ceph::io_exerciser::Sequence s = - static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>()); + static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>()); if (s < ceph::io_exerciser::Sequence::SEQUENCE_BEGIN || s >= ceph::io_exerciser::Sequence::SEQUENCE_END) { dout(0) << "Sequence argument out of range" << dendl; throw po::validation_error(po::validation_error::invalid_option_value); } ceph::io_exerciser::Sequence e = s; - force_value = std::make_optional<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>>( - std::make_pair(s, ++e)); + force_value = std::make_optional< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>>( + std::make_pair(s, ++e)); } } -const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence> - ceph::io_sequence::tester::SelectSeqRange::choose() { - if (force_value.has_value()) - { +const std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> +ceph::io_sequence::tester::SelectSeqRange::choose() { + if (force_value.has_value()) { return *force_value; } else { return std::make_pair(ceph::io_exerciser::Sequence::SEQUENCE_BEGIN, @@ -303,45 +298,34 @@ const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence> } } - - ceph::io_sequence::tester::SelectErasureKM::SelectErasureKM( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "km", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "km", true, true) {} ceph::io_sequence::tester::SelectErasurePlugin::SelectErasurePlugin( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "plugin", true, false) -{ -} - - - -ceph::io_sequence::tester::SelectErasureChunkSize::SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm) - : ProgramOptionSelector(rng, vm, "stripe_unit", true, false) -{ -} - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "plugin", true, false) {} +ceph::io_sequence::tester::SelectErasureChunkSize::SelectErasureChunkSize( + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "chunksize", true, true) {} ceph::io_sequence::tester::SelectECPool::SelectECPool( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - librados::Rados& rados, - bool dry_run) - : ProgramOptionSelector(rng, vm, "pool", false, false), - rados(rados), - dry_run(dry_run), - skm(SelectErasureKM(rng, vm)), - spl(SelectErasurePlugin(rng, vm)), - scs(SelectErasureChunkSize(rng, vm)) -{ + ceph::util::random_number_generator<int>& rng, po::variables_map vm, + librados::Rados& rados, bool dry_run, bool allow_pool_autoscaling, + bool allow_pool_balancer, bool allow_pool_deep_scrubbing, + bool allow_pool_scrubbing, bool test_recovery) + : ProgramOptionSelector(rng, vm, "pool", false, false), + rados(rados), + dry_run(dry_run), + allow_pool_autoscaling(allow_pool_autoscaling), + allow_pool_balancer(allow_pool_balancer), + allow_pool_deep_scrubbing(allow_pool_deep_scrubbing), + allow_pool_scrubbing(allow_pool_scrubbing), + test_recovery(test_recovery), + skm(SelectErasureKM(rng, vm)), + spl(SelectErasurePlugin(rng, vm)), + scs(SelectErasureChunkSize(rng, vm)) { if (!skm.isForced()) { if (vm.count("pool")) { force_value = vm["pool"].as<std::string>(); @@ -349,147 +333,239 @@ ceph::io_sequence::tester::SelectECPool::SelectECPool( } } -const std::string ceph::io_sequence::tester::SelectECPool::choose() -{ - std::pair<int,int> value; +const std::string ceph::io_sequence::tester::SelectECPool::choose() { + std::pair<int, int> value; if (!skm.isForced() && force_value.has_value()) { + int rc; + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDPoolGetRequest osdPoolGetRequest{*force_value}; + rc = send_mon_command(osdPoolGetRequest, rados, "OSDPoolGetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDPoolGetReply osdPoolGetReply; + osdPoolGetReply.decode_json(&p); + + ceph::messaging::osd::OSDECProfileGetRequest osdECProfileGetRequest{ + osdPoolGetReply.erasure_code_profile}; + rc = send_mon_command(osdECProfileGetRequest, rados, + "OSDECProfileGetRequest", inbl, &outbl, + formatter.get()); + ceph_assert(rc == 0); + + success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDECProfileGetReply reply; + reply.decode_json(&p); + k = reply.k; + m = reply.m; return *force_value; } else { value = skm.choose(); } - int k = value.first; - int m = value.second; + k = value.first; + m = value.second; const std::string plugin = std::string(spl.choose()); const uint64_t chunk_size = scs.choose(); - std::string pool_name = "ec_" + plugin + - "_cs" + std::to_string(chunk_size) + - "_k" + std::to_string(k) + - "_m" + std::to_string(m); - if (!dry_run) - { + std::string pool_name = "ec_" + plugin + "_cs" + std::to_string(chunk_size) + + "_k" + std::to_string(k) + "_m" + std::to_string(m); + if (!dry_run) { create_pool(rados, pool_name, plugin, chunk_size, k, m); } return pool_name; } void ceph::io_sequence::tester::SelectECPool::create_pool( - librados::Rados& rados, - const std::string& pool_name, - const std::string& plugin, - uint64_t chunk_size, - int k, int m) -{ + librados::Rados& rados, const std::string& pool_name, + const std::string& plugin, uint64_t chunk_size, int k, int m) { int rc; bufferlist inbl, outbl; - std::string profile_create = - "{\"prefix\": \"osd erasure-code-profile set\", \ - \"name\": \"testprofile-" + pool_name + "\", \ - \"profile\": [ \"plugin=" + plugin + "\", \ - \"k=" + std::to_string(k) + "\", \ - \"m=" + std::to_string(m) + "\", \ - \"stripe_unit=" + std::to_string(chunk_size) + "\", \ - \"crush-failure-domain=osd\"]}"; - rc = rados.mon_command(profile_create, inbl, &outbl, nullptr); + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDECProfileSetRequest ecProfileSetRequest{ + fmt::format("testprofile-{}", pool_name), + {fmt::format("plugin={}", plugin), fmt::format("k={}", k), + fmt::format("m={}", m), fmt::format("stripe_unit={}", chunk_size), + fmt::format("crush-failure-domain=osd")}}; + rc = send_mon_command(ecProfileSetRequest, rados, "OSDECProfileSetRequest", + inbl, &outbl, formatter.get()); ceph_assert(rc == 0); - std::string cmdstr = - "{\"prefix\": \"osd pool create\", \ - \"pool\": \"" + pool_name + "\", \ - \"pool_type\": \"erasure\", \ - \"pg_num\": 8, \ - \"pgp_num\": 8, \ - \"erasure_code_profile\": \"testprofile-" + pool_name + "\"}"; - rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr); + + ceph::messaging::osd::OSDECPoolCreateRequest poolCreateRequest{ + pool_name, "erasure", 8, 8, fmt::format("testprofile-{}", pool_name)}; + rc = send_mon_command(poolCreateRequest, rados, "OSDECPoolCreateRequest", + inbl, &outbl, formatter.get()); ceph_assert(rc == 0); -} + if (allow_pool_autoscaling) { + ceph::messaging::osd::OSDSetRequest setNoAutoscaleRequest{"noautoscale", + std::nullopt}; + rc = send_mon_command(setNoAutoscaleRequest, rados, "OSDSetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (allow_pool_balancer) { + ceph::messaging::balancer::BalancerOffRequest balancerOffRequest{}; + rc = send_mon_command(balancerOffRequest, rados, "BalancerOffRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + ceph::messaging::balancer::BalancerStatusRequest balancerStatusRequest{}; + rc = send_mon_command(balancerStatusRequest, rados, "BalancerStatusRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::balancer::BalancerStatusReply reply; + reply.decode_json(&p); + ceph_assert(!reply.active); + } + if (allow_pool_deep_scrubbing) { + ceph::messaging::osd::OSDSetRequest setNoDeepScrubRequest{"nodeep-scrub", + std::nullopt}; + rc = send_mon_command(setNoDeepScrubRequest, rados, "setNoDeepScrubRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (allow_pool_scrubbing) { + ceph::messaging::osd::OSDSetRequest setNoScrubRequest{"noscrub", + std::nullopt}; + rc = send_mon_command(setNoScrubRequest, rados, "OSDSetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (test_recovery) { + ceph::messaging::config::ConfigSetRequest configSetBluestoreDebugRequest{ + "global", "bluestore_debug_inject_read_err", "true", std::nullopt}; + rc = send_mon_command(configSetBluestoreDebugRequest, rados, + "ConfigSetRequest", inbl, &outbl, + formatter.get()); + ceph_assert(rc == 0); + + ceph::messaging::config::ConfigSetRequest configSetMaxMarkdownRequest{ + "global", "osd_max_markdown_count", "99999999", std::nullopt}; + rc = + send_mon_command(configSetMaxMarkdownRequest, rados, "ConfigSetRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + } +} -ceph::io_sequence::tester::TestObject::TestObject( const std::string oid, - librados::Rados& rados, - boost::asio::io_context& asio, - SelectBlockSize& sbs, - SelectECPool& spo, - SelectObjectSize& sos, - SelectNumThreads& snt, - SelectSeqRange& ssr, - ceph::util::random_number_generator<int>& rng, - ceph::mutex& lock, - ceph::condition_variable& cond, - bool dryrun, - bool verbose, - std::optional<int> seqseed) : - rng(rng), verbose(verbose), seqseed(seqseed) -{ +ceph::io_sequence::tester::TestObject::TestObject( + const std::string oid, librados::Rados& rados, + boost::asio::io_context& asio, SelectBlockSize& sbs, SelectECPool& spo, + SelectObjectSize& sos, SelectNumThreads& snt, SelectSeqRange& ssr, + ceph::util::random_number_generator<int>& rng, ceph::mutex& lock, + ceph::condition_variable& cond, bool dryrun, bool verbose, + std::optional<int> seqseed, bool testrecovery) + : rng(rng), verbose(verbose), seqseed(seqseed), testrecovery(testrecovery) { if (dryrun) { - verbose = true; - exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>(oid, - sbs.choose(), - rng()); + exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>( + oid, sbs.choose(), rng()); } else { const std::string pool = spo.choose(); + poolK = spo.getChosenK(); + poolM = spo.getChosenM(); + int threads = snt.choose(); - exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, - asio, - pool, - oid, - sbs.choose(), - rng(), - threads, - lock, - cond); - dout(0) << "= " << oid << " pool=" << pool - << " threads=" << threads - << " blocksize=" << exerciser_model->get_block_size() - << " =" << dendl; + + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + std::optional<std::vector<int>> cached_shard_order = std::nullopt; + + if (!spo.get_allow_pool_autoscaling() && !spo.get_allow_pool_balancer() && + !spo.get_allow_pool_deep_scrubbing() && + !spo.get_allow_pool_scrubbing()) { + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, oid, ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply{}; + reply.decode_json(&p); + cached_shard_order = reply.acting; + } + + exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>( + rados, asio, pool, oid, cached_shard_order, sbs.choose(), rng(), + threads, lock, cond); + dout(0) << "= " << oid << " pool=" << pool << " threads=" << threads + << " blocksize=" << exerciser_model->get_block_size() << " =" + << dendl; } obj_size_range = sos.choose(); seq_range = ssr.choose(); curseq = seq_range.first; - seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, - obj_size_range, - seqseed.value_or(rng())); + + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + curseq, obj_size_range, poolK, poolM, seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + curseq, obj_size_range, seqseed.value_or(rng())); + } + op = seq->next(); done = false; - dout(0) << "== " << exerciser_model->get_oid() << " " - << curseq << " " - << seq->get_name() - << " ==" <<dendl; + dout(0) << "== " << exerciser_model->get_oid() << " " << curseq << " " + << seq->get_name_with_seqseed() << " ==" << dendl; } -bool ceph::io_sequence::tester::TestObject::readyForIo() -{ +bool ceph::io_sequence::tester::TestObject::readyForIo() { return exerciser_model->readyForIoOp(*op); } -bool ceph::io_sequence::tester::TestObject::next() -{ +bool ceph::io_sequence::tester::TestObject::next() { if (!done) { if (verbose) { - dout(0) << exerciser_model->get_oid() - << " Step " << seq->get_step() << ": " - << op->to_string(exerciser_model->get_block_size()) << dendl; + dout(0) << exerciser_model->get_oid() << " Step " << seq->get_step() + << ": " << op->to_string(exerciser_model->get_block_size()) + << dendl; } else { - dout(5) << exerciser_model->get_oid() - << " Step " << seq->get_step() << ": " - << op->to_string(exerciser_model->get_block_size()) << dendl; + dout(5) << exerciser_model->get_oid() << " Step " << seq->get_step() + << ": " << op->to_string(exerciser_model->get_block_size()) + << dendl; } exerciser_model->applyIoOp(*op); - if (op->done()) { - ++curseq; - if (curseq == seq_range.second) { + if (op->getOpType() == ceph::io_exerciser::OpType::Done) { + curseq = seq->getNextSupportedSequenceId(); + if (curseq >= seq_range.second) { done = true; dout(0) << exerciser_model->get_oid() << " Number of IOs = " << exerciser_model->get_num_io() << dendl; } else { - seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, - obj_size_range, - seqseed.value_or(rng())); - dout(0) << "== " << exerciser_model->get_oid() << " " - << curseq << " " << seq->get_name() - << " ==" <<dendl; + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + curseq, obj_size_range, poolK, poolM, seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + curseq, obj_size_range, seqseed.value_or(rng())); + } + + dout(0) << "== " << exerciser_model->get_oid() << " " << curseq << " " + << seq->get_name_with_seqseed() << " ==" << dendl; op = seq->next(); } } else { @@ -499,27 +575,30 @@ bool ceph::io_sequence::tester::TestObject::next() return done; } -bool ceph::io_sequence::tester::TestObject::finished() -{ - return done; -} +bool ceph::io_sequence::tester::TestObject::finished() { return done; } -int ceph::io_sequence::tester::TestObject::get_num_io() -{ +int ceph::io_sequence::tester::TestObject::get_num_io() { return exerciser_model->get_num_io(); } ceph::io_sequence::tester::TestRunner::TestRunner(po::variables_map& vm, - librados::Rados& rados) : - rados(rados), - seed(vm.contains("seed") ? vm["seed"].as<int>() : time(nullptr)), - rng(ceph::util::random_number_generator<int>(seed)), - sbs{rng, vm}, - sos{rng, vm}, - spo{rng, vm, rados, vm.contains("dryrun")}, - snt{rng, vm}, - ssr{rng, vm} -{ + librados::Rados& rados) + : rados(rados), + seed(vm.contains("seed") ? vm["seed"].as<int>() : time(nullptr)), + rng(ceph::util::random_number_generator<int>(seed)), + sbs{rng, vm}, + sos{rng, vm}, + spo{rng, + vm, + rados, + vm.contains("dryrun"), + vm.contains("allow_pool_autoscaling"), + vm.contains("allow_pool_balancer"), + vm.contains("allow_pool_deep_scrubbing"), + vm.contains("allow_pool_scrubbing"), + vm.contains("test_recovery")}, + snt{rng, vm}, + ssr{rng, vm} { dout(0) << "Test using seed " << seed << dendl; verbose = vm.contains("verbose"); @@ -532,19 +611,23 @@ ceph::io_sequence::tester::TestRunner::TestRunner(po::variables_map& vm, num_objects = vm["parallel"].as<int>(); object_name = vm["object"].as<std::string>(); interactive = vm.contains("interactive"); + testrecovery = vm.contains("testrecovery"); + + allow_pool_autoscaling = vm.contains("allow_pool_autoscaling"); + allow_pool_balancer = vm.contains("allow_pool_balancer"); + allow_pool_deep_scrubbing = vm.contains("allow_pool_deep_scrubbing"); + allow_pool_scrubbing = vm.contains("allow_pool_scrubbing"); - if (!dryrun) - { + if (!dryrun) { guard.emplace(boost::asio::make_work_guard(asio)); - thread = make_named_thread("io_thread",[&asio = asio] { asio.run(); }); + thread = make_named_thread("io_thread", [&asio = asio] { asio.run(); }); } show_help = vm.contains("help"); show_sequence = vm.contains("listsequence"); } -ceph::io_sequence::tester::TestRunner::~TestRunner() -{ +ceph::io_sequence::tester::TestRunner::~TestRunner() { if (!dryrun) { guard = std::nullopt; asio.stop(); @@ -553,34 +636,38 @@ ceph::io_sequence::tester::TestRunner::~TestRunner() } } -void ceph::io_sequence::tester::TestRunner::help() -{ +void ceph::io_sequence::tester::TestRunner::help() { std::cout << get_options_description() << std::endl; for (auto line : usage) { std::cout << line << std::endl; } } -void ceph::io_sequence::tester::TestRunner::list_sequence() -{ +void ceph::io_sequence::tester::TestRunner::list_sequence(bool testrecovery) { // List seqeunces - std::pair<int,int> obj_size_range = sos.choose(); - for (ceph::io_exerciser::Sequence s - = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; - s < ceph::io_exerciser::Sequence::SEQUENCE_END; ++s) { - std::unique_ptr<ceph::io_exerciser::IoSequence> seq = - ceph::io_exerciser::IoSequence::generate_sequence(s, - obj_size_range, - seqseed.value_or(rng())); - dout(0) << s << " " << seq->get_name() << dendl; + std::pair<int, int> obj_size_range = sos.choose(); + ceph::io_exerciser::Sequence s = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; + std::unique_ptr<ceph::io_exerciser::IoSequence> seq; + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + s, obj_size_range, spo.getChosenK(), spo.getChosenM(), + seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + s, obj_size_range, seqseed.value_or(rng())); } + + do { + dout(0) << s << " " << seq->get_name_with_seqseed() << dendl; + s = seq->getNextSupportedSequenceId(); + } while (s != ceph::io_exerciser::Sequence::SEQUENCE_END); } -std::string ceph::io_sequence::tester::TestRunner::get_token() -{ - static std::string line; - static ceph::split split = ceph::split(""); - static ceph::spliterator tokens; +void ceph::io_sequence::tester::TestRunner::clear_tokens() { + tokens = split.end(); +} + +std::string ceph::io_sequence::tester::TestRunner::get_token() { while (line.empty() || tokens == split.end()) { if (!std::getline(std::cin, line)) { throw std::runtime_error("End of input"); @@ -591,127 +678,211 @@ std::string ceph::io_sequence::tester::TestRunner::get_token() return std::string(*tokens++); } -uint64_t ceph::io_sequence::tester::TestRunner::get_numeric_token() -{ +std::optional<std::string> +ceph::io_sequence::tester::TestRunner ::get_optional_token() { + std::optional<std::string> ret = std::nullopt; + if (tokens != split.end()) { + ret = std::string(*tokens++); + } + return ret; +} + +uint64_t ceph::io_sequence::tester::TestRunner::get_numeric_token() { std::string parse_error; std::string token = get_token(); uint64_t num = strict_iecstrtoll(token, &parse_error); if (!parse_error.empty()) { - throw std::runtime_error("Invalid number "+token); + throw std::runtime_error("Invalid number " + token); } return num; } -bool ceph::io_sequence::tester::TestRunner::run_test() -{ - if (show_help) - { +std::optional<uint64_t> +ceph::io_sequence::tester::TestRunner ::get_optional_numeric_token() { + std::string parse_error; + std::optional<std::string> token = get_optional_token(); + if (token) { + uint64_t num = strict_iecstrtoll(*token, &parse_error); + if (!parse_error.empty()) { + throw std::runtime_error("Invalid number " + *token); + } + return num; + } + + return std::optional<uint64_t>(std::nullopt); +} + +bool ceph::io_sequence::tester::TestRunner::run_test() { + if (show_help) { help(); return true; - } - else if (show_sequence) - { - list_sequence(); + } else if (show_sequence) { + list_sequence(testrecovery); return true; - } - else if (interactive) - { + } else if (interactive) { return run_interactive_test(); - } - else - { + } else { return run_automated_test(); } } -bool ceph::io_sequence::tester::TestRunner::run_interactive_test() -{ +bool ceph::io_sequence::tester::TestRunner::run_interactive_test() { bool done = false; std::unique_ptr<ceph::io_exerciser::IoOp> ioop; std::unique_ptr<ceph::io_exerciser::Model> model; if (dryrun) { - model = std::make_unique<ceph::io_exerciser::ObjectModel>(object_name, - sbs.choose(), - rng()); + model = std::make_unique<ceph::io_exerciser::ObjectModel>( + object_name, sbs.choose(), rng()); } else { const std::string pool = spo.choose(); - model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, asio, pool, - object_name, sbs.choose(), - rng(), 1, // 1 thread - lock, cond); + + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, object_name, ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply{}; + reply.decode_json(&p); + + model = std::make_unique<ceph::io_exerciser::RadosIo>( + rados, asio, pool, object_name, reply.acting, sbs.choose(), rng(), + 1, // 1 thread + lock, cond); } while (!done) { const std::string op = get_token(); - if (!op.compare("done") || !op.compare("q") || !op.compare("quit")) { - ioop = ceph::io_exerciser::IoOp::generate_done(); - } else if (!op.compare("create")) { - ioop = ceph::io_exerciser::IoOp::generate_create(get_numeric_token()); - } else if (!op.compare("remove") || !op.compare("delete")) { - ioop = ceph::io_exerciser::IoOp::generate_remove(); - } else if (!op.compare("read")) { + if (op == "done" || op == "q" || op == "quit") { + ioop = ceph::io_exerciser::DoneOp::generate(); + } else if (op == "create") { + ioop = ceph::io_exerciser::CreateOp::generate(get_numeric_token()); + } else if (op == "remove" || op == "delete") { + ioop = ceph::io_exerciser::RemoveOp::generate(); + } else if (op == "read") { uint64_t offset = get_numeric_token(); uint64_t length = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read(offset, length); - } else if (!op.compare("read2")) { + ioop = ceph::io_exerciser::SingleReadOp::generate(offset, length); + } else if (op == "read2") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read2(offset1, length1, - offset2, length2); - } else if (!op.compare("read3")) { + ioop = DoubleReadOp::generate(offset1, length1, offset2, length2); + } else if (op == "read3") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); uint64_t offset3 = get_numeric_token(); uint64_t length3 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read3(offset1, length1, - offset2, length2, - offset3, length3); - } else if (!op.compare("write")) { + ioop = TripleReadOp::generate(offset1, length1, offset2, length2, offset3, + length3); + } else if (op == "write") { uint64_t offset = get_numeric_token(); uint64_t length = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write(offset, length); - } else if (!op.compare("write2")) { + ioop = SingleWriteOp::generate(offset, length); + } else if (op == "write2") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write2(offset1, length1, - offset2, length2); - } else if (!op.compare("write3")) { + ioop = DoubleWriteOp::generate(offset1, length1, offset2, length2); + } else if (op == "write3") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); uint64_t offset3 = get_numeric_token(); uint64_t length3 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write3(offset1, length1, - offset2, length2, - offset3, length3); + ioop = TripleWriteOp::generate(offset1, length1, offset2, length2, + offset3, length3); + } else if (op == "failedwrite") { + uint64_t offset = get_numeric_token(); + uint64_t length = get_numeric_token(); + ioop = SingleFailedWriteOp::generate(offset, length); + } else if (op == "failedwrite2") { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + ioop = DoubleFailedWriteOp::generate(offset1, length1, offset2, length2); + } else if (op == "failedwrite3") { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + uint64_t offset3 = get_numeric_token(); + uint64_t length3 = get_numeric_token(); + ioop = TripleFailedWriteOp::generate(offset1, length1, offset2, length2, + offset3, length3); + } else if (op == "injecterror") { + std::string inject_type = get_token(); + int shard = get_numeric_token(); + std::optional<int> type = get_optional_numeric_token(); + std::optional<int> when = get_optional_numeric_token(); + std::optional<int> duration = get_optional_numeric_token(); + if (inject_type == "read") { + ioop = ceph::io_exerciser::InjectReadErrorOp::generate(shard, type, + when, duration); + } else if (inject_type == "write") { + ioop = ceph::io_exerciser::InjectWriteErrorOp::generate(shard, type, + when, duration); + } else { + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid error inject {}. No action performed.", + inject_type) + << dendl; + } + } else if (op == "clearinject") { + std::string inject_type = get_token(); + int shard = get_numeric_token(); + std::optional<int> type = get_optional_numeric_token(); + if (inject_type == "read") { + ioop = + ceph::io_exerciser::ClearReadErrorInjectOp::generate(shard, type); + } else if (inject_type == "write") { + ioop = + ceph::io_exerciser::ClearWriteErrorInjectOp::generate(shard, type); + } else { + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid error inject {}. No action performed.", + inject_type) + << dendl; + } } else { - throw std::runtime_error("Invalid operation "+op); + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid op {}. No action performed.", op) + << dendl; } - dout(0) << ioop->to_string(model->get_block_size()) << dendl; - model->applyIoOp(*ioop); - done = ioop->done(); - if (!done) { - ioop = ceph::io_exerciser::IoOp::generate_barrier(); + if (ioop) { + dout(0) << ioop->to_string(model->get_block_size()) << dendl; model->applyIoOp(*ioop); + done = ioop->getOpType() == ceph::io_exerciser::OpType::Done; + if (!done) { + ioop = ceph::io_exerciser::BarrierOp::generate(); + model->applyIoOp(*ioop); + } } } return true; } -bool ceph::io_sequence::tester::TestRunner::run_automated_test() -{ +bool ceph::io_sequence::tester::TestRunner::run_automated_test() { // Create a test for each object - std::vector<std::shared_ptr< - ceph::io_sequence::tester::TestObject>> test_objects; + std::vector<std::shared_ptr<ceph::io_sequence::tester::TestObject>> + test_objects; for (int obj = 0; obj < num_objects; obj++) { std::string name; @@ -721,15 +892,9 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() name = object_name + std::to_string(obj); } test_objects.push_back( - std::make_shared<ceph::io_sequence::tester::TestObject>( - name, - rados, asio, - sbs, spo, sos, snt, ssr, - rng, lock, cond, - dryrun, verbose, - seqseed - ) - ); + std::make_shared<ceph::io_sequence::tester::TestObject>( + name, rados, asio, sbs, spo, sos, snt, ssr, rng, lock, cond, dryrun, + verbose, seqseed, testrecovery)); } if (!dryrun) { rados.wait_for_latest_osdmap(); @@ -748,16 +913,15 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() for (auto obj = test_objects.begin(); obj != test_objects.end(); ++obj) { std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj; if (!to->finished()) { - lock.lock(); - bool ready = to->readyForIo(); - lock.unlock(); - if (ready) - { - to->next(); - started_io = true; - } else { - need_wait = true; - } + lock.lock(); + bool ready = to->readyForIo(); + lock.unlock(); + if (ready) { + to->next(); + started_io = true; + } else { + need_wait = true; + } } } if (!started_io && need_wait) { @@ -767,8 +931,7 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj; if (!to->finished()) { need_wait = !to->readyForIo(); - if (!need_wait) - { + if (!need_wait) { break; } } @@ -788,18 +951,16 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() return true; } -int main(int argc, char **argv) -{ +int main(int argc, char** argv) { auto args = argv_to_vec(argc, argv); env_to_vec(args); auto cct = global_init(NULL, args, CEPH_ENTITY_TYPE_CLIENT, - CODE_ENVIRONMENT_UTILITY, 0); + CODE_ENVIRONMENT_UTILITY, 0); common_init_finish(cct.get()); po::variables_map vm; int rc = parse_io_seq_options(vm, argc, argv); - if (rc != 0) - { + if (rc != 0) { return rc; } @@ -814,7 +975,7 @@ int main(int argc, char **argv) std::unique_ptr<ceph::io_sequence::tester::TestRunner> runner; try { runner = std::make_unique<ceph::io_sequence::tester::TestRunner>(vm, rados); - } catch(const po::error& e) { + } catch (const po::error& e) { return 1; } runner->run_test(); diff --git a/src/test/osd/ceph_test_rados_io_sequence.h b/src/test/osd/ceph_test_rados_io_sequence.h index 4e21d025700..9af5f706b2f 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.h +++ b/src/test/osd/ceph_test_rados_io_sequence.h @@ -1,34 +1,36 @@ +#include <boost/program_options.hpp> +#include <optional> #include <utility> -#include "include/random.h" - -#include "global/global_init.h" -#include "global/global_context.h" - #include "common/io_exerciser/IoOp.h" #include "common/io_exerciser/IoSequence.h" #include "common/io_exerciser/Model.h" - +#include "common/split.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/random.h" #include "librados/librados_asio.h" #include <boost/asio/io_context.hpp> #include <boost/program_options.hpp> +#include <optional> + /* Overview * * class ProgramOptionSelector - * Base class for selector objects below with common code for + * Base class for selector objects below with common code for * selecting options - * + * * class SelectObjectSize * Selects min and max object sizes for a test * * class SelectErasureKM * Selects an EC k and m value for a test - * + * * class SelectErasurePlugin * Selects an plugin for a test - * + * * class SelectECPool * Selects an EC pool (plugin,k and m) for a test. Also creates the * pool as well. @@ -58,287 +60,279 @@ namespace po = boost::program_options; -namespace ceph -{ - namespace io_sequence::tester - { - // Choices for min and max object size - inline constexpr size_t objectSizeSize = 10; - inline constexpr std::array<std::pair<int,int>,objectSizeSize> - objectSizeChoices = {{ - {1,32}, // Default - best for boundary checking - {12,14}, - {28,30}, - {36,38}, - {42,44}, - {52,54}, - {66,68}, - {72,74}, - {83,83}, - {97,97} - }}; - - // Choices for block size - inline constexpr int blockSizeSize = 5; - inline constexpr std::array<uint64_t, blockSizeSize> blockSizeChoices = {{ - 2048, // Default - test boundaries for EC 4K chunk size - 512, - 3767, - 4096, - 32768 - }}; - - // Choices for number of threads - inline constexpr int threadArraySize = 4; - inline constexpr std::array<int, threadArraySize> threadCountChoices = {{ - 1, // Default - 2, - 4, - 8 - }}; - - // Choices for EC k+m profile - inline constexpr int kmSize = 6; - inline constexpr std::array<std::pair<int,int>, kmSize> kmChoices = {{ - {2,2}, // Default - reasonable coverage - {2,1}, - {2,3}, - {3,2}, - {4,2}, - {5,1} - }}; - - // Choices for EC chunk size - inline constexpr int chunkSizeSize = 3; - inline constexpr std::array<uint64_t, chunkSizeSize> chunkSizeChoices = {{ - 4*1024, - 64*1024, - 256*1024 - }}; - - // Choices for plugin - inline constexpr int pluginListSize = 2; - inline constexpr std::array<std::string_view, - pluginListSize> pluginChoices = {{ - "jerasure", - "isa" - }}; - - inline constexpr std::array<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>, - 0> sequencePairs = {{}}; - - inline constexpr std::array<std::string, 0> poolChoices = {{}}; - - template <typename T, int N, const std::array<T, N>& Ts> - class ProgramOptionSelector - { - public: - ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - const std::string& option_name, - bool set_forced, - bool select_first - ); - virtual ~ProgramOptionSelector() = default; - bool isForced(); - virtual const T choose(); - - protected: - ceph::util::random_number_generator<int>& rng; - static constexpr std::array<T, N> choices = Ts; - - std::optional<T> force_value; - std::optional<T> first_value; - - std::string option_name; - }; - - class SelectObjectSize - : public ProgramOptionSelector<std::pair<int, int>, - io_sequence::tester::objectSizeSize, - io_sequence::tester::objectSizeChoices> - { - public: - SelectObjectSize(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectBlockSize - : public ProgramOptionSelector<uint64_t, - io_sequence::tester::blockSizeSize, - io_sequence::tester::blockSizeChoices> - { - public: - SelectBlockSize(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectNumThreads - : public ProgramOptionSelector<int, - io_sequence::tester::threadArraySize, - io_sequence::tester::threadCountChoices> - { - public: - SelectNumThreads(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectSeqRange - : public ProgramOptionSelector<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>, - 0, io_sequence::tester::sequencePairs> - { - public: - SelectSeqRange(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - - const std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence> choose() override; - }; - - class SelectErasureKM - : public ProgramOptionSelector<std::pair<int,int>, - io_sequence::tester::kmSize, - io_sequence::tester::kmChoices> - { - public: - SelectErasureKM(ceph::util::random_number_generator<int>& rng, +namespace ceph { +namespace io_sequence::tester { +// Choices for min and max object size +inline constexpr size_t objectSizeSize = 10; +inline constexpr std::array<std::pair<int, int>, objectSizeSize> + objectSizeChoices = {{{1, 32}, // Default - best for boundary checking + {12, 14}, + {28, 30}, + {36, 38}, + {42, 44}, + {52, 54}, + {66, 68}, + {72, 74}, + {83, 83}, + {97, 97}}}; + +// Choices for block size +inline constexpr int blockSizeSize = 5; +inline constexpr std::array<uint64_t, blockSizeSize> blockSizeChoices = { + {2048, // Default - test boundaries for EC 4K chunk size + 512, 3767, 4096, 32768}}; + +// Choices for number of threads +inline constexpr int threadArraySize = 4; +inline constexpr std::array<int, threadArraySize> threadCountChoices = { + {1, // Default + 2, 4, 8}}; + +// Choices for EC k+m profile +inline constexpr int kmSize = 6; +inline constexpr std::array<std::pair<int, int>, kmSize> kmChoices = { + {{2, 2}, // Default - reasonable coverage + {2, 1}, + {2, 3}, + {3, 2}, + {4, 2}, + {5, 1}}}; + +// Choices for EC chunk size +inline constexpr int chunkSizeSize = 3; +inline constexpr std::array<uint64_t, chunkSizeSize> chunkSizeChoices = { + {4 * 1024, 64 * 1024, 256 * 1024}}; + +// Choices for plugin +inline constexpr int pluginListSize = 2; +inline constexpr std::array<std::string_view, pluginListSize> pluginChoices = { + {"jerasure", "isa"}}; + +inline constexpr std::array< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>, 0> + sequencePairs = {{}}; + +inline constexpr std::array<std::string, 0> poolChoices = {{}}; + +template <typename T, int N, const std::array<T, N>& Ts> +class ProgramOptionSelector { + public: + ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, const std::string& option_name, + bool set_forced, bool select_first); + virtual ~ProgramOptionSelector() = default; + bool isForced(); + virtual const T choose(); + + protected: + ceph::util::random_number_generator<int>& rng; + static constexpr std::array<T, N> choices = Ts; + + std::optional<T> force_value; + std::optional<T> first_value; + + std::string option_name; +}; + +class SelectObjectSize + : public ProgramOptionSelector<std::pair<int, int>, + io_sequence::tester::objectSizeSize, + io_sequence::tester::objectSizeChoices> { + public: + SelectObjectSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectBlockSize + : public ProgramOptionSelector<uint64_t, io_sequence::tester::blockSizeSize, + io_sequence::tester::blockSizeChoices> { + public: + SelectBlockSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectNumThreads + : public ProgramOptionSelector<int, io_sequence::tester::threadArraySize, + io_sequence::tester::threadCountChoices> { + public: + SelectNumThreads(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectSeqRange + : public ProgramOptionSelector< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>, + 0, io_sequence::tester::sequencePairs> { + public: + SelectSeqRange(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); + + const std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> + choose() override; +}; + +class SelectErasureKM + : public ProgramOptionSelector<std::pair<int, int>, + io_sequence::tester::kmSize, + io_sequence::tester::kmChoices> { + public: + SelectErasureKM(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectErasurePlugin + : public ProgramOptionSelector<std::string_view, + io_sequence::tester::pluginListSize, + io_sequence::tester::pluginChoices> { + public: + SelectErasurePlugin(ceph::util::random_number_generator<int>& rng, po::variables_map vm); - }; - - class SelectErasurePlugin - : public ProgramOptionSelector<std::string_view, - io_sequence::tester::pluginListSize, - io_sequence::tester::pluginChoices> - { - public: - SelectErasurePlugin(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectErasureChunkSize - : public ProgramOptionSelector<uint64_t, - io_sequence::tester::chunkSizeSize, - io_sequence::tester::chunkSizeChoices> - { - public: - SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm); - }; - - class SelectECPool - : public ProgramOptionSelector<std::string, - 0, - io_sequence::tester::poolChoices> - { - public: - SelectECPool(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - librados::Rados& rados, - bool dry_run); - const std::string choose() override; - - private: - void create_pool(librados::Rados& rados, - const std::string& pool_name, - const std::string& plugin, - uint64_t chunk_size, - int k, int m); - - protected: - librados::Rados& rados; - bool dry_run; - - SelectErasureKM skm; - SelectErasurePlugin spl; - SelectErasureChunkSize scs; - }; - - class TestObject - { - public: - TestObject( const std::string oid, - librados::Rados& rados, - boost::asio::io_context& asio, - ceph::io_sequence::tester::SelectBlockSize& sbs, - ceph::io_sequence::tester::SelectECPool& spl, - ceph::io_sequence::tester::SelectObjectSize& sos, - ceph::io_sequence::tester::SelectNumThreads& snt, - ceph::io_sequence::tester::SelectSeqRange& ssr, - ceph::util::random_number_generator<int>& rng, - ceph::mutex& lock, - ceph::condition_variable& cond, - bool dryrun, - bool verbose, - std::optional<int> seqseed); - - int get_num_io(); - bool readyForIo(); - bool next(); - bool finished(); - - protected: - std::unique_ptr<ceph::io_exerciser::Model> exerciser_model; - std::pair<int,int> obj_size_range; - std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence> seq_range; - ceph::io_exerciser::Sequence curseq; - std::unique_ptr<ceph::io_exerciser::IoSequence> seq; - std::unique_ptr<ceph::io_exerciser::IoOp> op; - bool done; - ceph::util::random_number_generator<int>& rng; - bool verbose; - std::optional<int> seqseed; - }; - - class TestRunner - { - public: - TestRunner(po::variables_map& vm, librados::Rados& rados); - ~TestRunner(); - - bool run_test(); - - private: - librados::Rados& rados; - int seed; - ceph::util::random_number_generator<int> rng; - - ceph::io_sequence::tester::SelectBlockSize sbs; - ceph::io_sequence::tester::SelectObjectSize sos; - ceph::io_sequence::tester::SelectECPool spo; - ceph::io_sequence::tester::SelectNumThreads snt; - ceph::io_sequence::tester::SelectSeqRange ssr; - - boost::asio::io_context asio; - std::thread thread; - std::optional<boost::asio::executor_work_guard< - boost::asio::io_context::executor_type>> guard; - ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); - ceph::condition_variable cond; - - bool input_valid; - - bool verbose; - bool dryrun; - std::optional<int> seqseed; - bool interactive; - - bool show_sequence; - bool show_help; - - int num_objects; - std::string object_name; - - std::string get_token(); - uint64_t get_numeric_token(); - - bool run_automated_test(); - - bool run_interactive_test(); - - void help(); - void list_sequence(); - }; - } -} +}; + +class SelectErasureChunkSize + : public ProgramOptionSelector<uint64_t, io_sequence::tester::chunkSizeSize, + io_sequence::tester::chunkSizeChoices> { + public: + SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectECPool + : public ProgramOptionSelector<std::string, 0, + io_sequence::tester::poolChoices> { + public: + SelectECPool(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, librados::Rados& rados, bool dry_run, + bool allow_pool_autoscaling, bool allow_pool_balancer, + bool allow_pool_deep_scrubbing, bool allow_pool_scrubbing, + bool test_recovery); + const std::string choose() override; + + bool get_allow_pool_autoscaling() { return allow_pool_autoscaling; } + bool get_allow_pool_balancer() { return allow_pool_balancer; } + bool get_allow_pool_deep_scrubbing() { return allow_pool_deep_scrubbing; } + bool get_allow_pool_scrubbing() { return allow_pool_scrubbing; } + int getChosenK() const { return k; } + int getChosenM() const { return m; } + + private: + void create_pool(librados::Rados& rados, const std::string& pool_name, + const std::string& plugin, uint64_t chunk_size, int k, + int m); + + protected: + librados::Rados& rados; + bool dry_run; + bool allow_pool_autoscaling; + bool allow_pool_balancer; + bool allow_pool_deep_scrubbing; + bool allow_pool_scrubbing; + bool test_recovery; + int k; + int m; + + SelectErasureKM skm; + SelectErasurePlugin spl; + SelectErasureChunkSize scs; +}; + +class TestObject { + public: + TestObject(const std::string oid, librados::Rados& rados, + boost::asio::io_context& asio, + ceph::io_sequence::tester::SelectBlockSize& sbs, + ceph::io_sequence::tester::SelectECPool& spl, + ceph::io_sequence::tester::SelectObjectSize& sos, + ceph::io_sequence::tester::SelectNumThreads& snt, + ceph::io_sequence::tester::SelectSeqRange& ssr, + ceph::util::random_number_generator<int>& rng, ceph::mutex& lock, + ceph::condition_variable& cond, bool dryrun, bool verbose, + std::optional<int> seqseed, bool testRecovery); + + int get_num_io(); + bool readyForIo(); + bool next(); + bool finished(); + + protected: + std::unique_ptr<ceph::io_exerciser::Model> exerciser_model; + std::pair<int, int> obj_size_range; + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> + seq_range; + ceph::io_exerciser::Sequence curseq; + std::unique_ptr<ceph::io_exerciser::IoSequence> seq; + std::unique_ptr<ceph::io_exerciser::IoOp> op; + bool done; + ceph::util::random_number_generator<int>& rng; + bool verbose; + std::optional<int> seqseed; + int poolK; + int poolM; + bool testrecovery; +}; + +class TestRunner { + public: + TestRunner(po::variables_map& vm, librados::Rados& rados); + ~TestRunner(); + + bool run_test(); + + private: + librados::Rados& rados; + int seed; + ceph::util::random_number_generator<int> rng; + + ceph::io_sequence::tester::SelectBlockSize sbs; + ceph::io_sequence::tester::SelectObjectSize sos; + ceph::io_sequence::tester::SelectECPool spo; + ceph::io_sequence::tester::SelectNumThreads snt; + ceph::io_sequence::tester::SelectSeqRange ssr; + + boost::asio::io_context asio; + std::thread thread; + std::optional< + boost::asio::executor_work_guard<boost::asio::io_context::executor_type>> + guard; + ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); + ceph::condition_variable cond; + + bool input_valid; + + bool verbose; + bool dryrun; + std::optional<int> seqseed; + bool interactive; + + bool testrecovery; + + bool allow_pool_autoscaling; + bool allow_pool_balancer; + bool allow_pool_deep_scrubbing; + bool allow_pool_scrubbing; + + bool show_sequence; + bool show_help; + + int num_objects; + std::string object_name; + + std::string line; + ceph::split split = ceph::split(""); + ceph::spliterator tokens; + + void clear_tokens(); + std::string get_token(); + std::optional<std::string> get_optional_token(); + uint64_t get_numeric_token(); + std::optional<uint64_t> get_optional_numeric_token(); + + bool run_automated_test(); + + bool run_interactive_test(); + + void help(); + void list_sequence(bool testrecovery); +}; +} // namespace io_sequence::tester +} // namespace ceph diff --git a/src/test/pybind/pytest.ini b/src/test/pybind/pytest.ini index dccf2a346dc..97569e88299 100644 --- a/src/test/pybind/pytest.ini +++ b/src/test/pybind/pytest.ini @@ -7,3 +7,4 @@ markers = stats tier watch + wait diff --git a/src/test/pybind/test_rados.py b/src/test/pybind/test_rados.py index cb2a4f96101..25423bd8dcb 100644 --- a/src/test/pybind/test_rados.py +++ b/src/test/pybind/test_rados.py @@ -207,7 +207,7 @@ class TestRados(object): def test_get_fsid(self): fsid = self.rados.get_fsid() - assert re.match('[0-9a-f\-]{36}', fsid, re.I) + assert re.match(r'[0-9a-f\-]{36}', fsid, re.I) def test_blocklist_add(self): self.rados.blocklist_add("1.2.3.4/123", 1) diff --git a/src/test/rgw/bucket_notification/api.py b/src/test/rgw/bucket_notification/api.py index e7ec31f1711..e84aa16edc7 100644 --- a/src/test/rgw/bucket_notification/api.py +++ b/src/test/rgw/bucket_notification/api.py @@ -247,12 +247,16 @@ def delete_all_topics(conn, tenant, cluster): if tenant == '': topics_result = admin(['topic', 'list'], cluster) topics_json = json.loads(topics_result[0]) + if 'topics' not in topics_json: + topics_json = topics_json.get('result',{}) for topic in topics_json['topics']: rm_result = admin(['topic', 'rm', '--topic', topic['name']], cluster) print(rm_result) else: topics_result = admin(['topic', 'list', '--tenant', tenant], cluster) topics_json = json.loads(topics_result[0]) + if 'topics' not in topics_json: + topics_json = topics_json.get('result',{}) for topic in topics_json['topics']: rm_result = admin(['topic', 'rm', '--tenant', tenant, '--topic', topic['name']], cluster) print(rm_result) diff --git a/src/test/rgw/bucket_notification/requirements.txt b/src/test/rgw/bucket_notification/requirements.txt index a3cff2bedab..bb74eceedc3 100644 --- a/src/test/rgw/bucket_notification/requirements.txt +++ b/src/test/rgw/bucket_notification/requirements.txt @@ -1,4 +1,4 @@ -nose >=1.0.0 +nose-py3 >=1.0.0 boto >=2.6.0 boto3 >=1.0.0 configparser >=5.0.0 diff --git a/src/test/rgw/bucket_notification/test_bn.py b/src/test/rgw/bucket_notification/test_bn.py index 359990b3531..665fbca7494 100644 --- a/src/test/rgw/bucket_notification/test_bn.py +++ b/src/test/rgw/bucket_notification/test_bn.py @@ -410,17 +410,25 @@ kafka_server = 'localhost' class KafkaReceiver(object): """class for receiving and storing messages on a topic from the kafka broker""" - def __init__(self, topic, security_type): + def __init__(self, topic, security_type, kafka_server='localhost'): from kafka import KafkaConsumer remaining_retries = 10 port = 9092 if security_type != 'PLAINTEXT': security_type = 'SSL' port = 9093 + + if kafka_server is None: + endpoint = "localhost" + ":" + str(port) + elif ":" not in kafka_server: + endpoint = kafka_server + ":" + str(port) + else: + endpoint = kafka_server + while remaining_retries > 0: try: self.consumer = KafkaConsumer(topic, - bootstrap_servers = kafka_server+':'+str(port), + bootstrap_servers=endpoint, security_protocol=security_type, consumer_timeout_ms=16000, auto_offset_reset='earliest') @@ -468,9 +476,9 @@ def kafka_receiver_thread_runner(receiver): print('Kafka receiver ended unexpectedly: ' + str(error)) -def create_kafka_receiver_thread(topic, security_type='PLAINTEXT'): +def create_kafka_receiver_thread(topic, security_type='PLAINTEXT', kafka_brokers=None): """create kafka receiver and thread""" - receiver = KafkaReceiver(topic, security_type) + receiver = KafkaReceiver(topic, security_type, kafka_server=kafka_brokers) task = threading.Thread(target=kafka_receiver_thread_runner, args=(receiver,)) task.daemon = True return task, receiver @@ -1304,7 +1312,7 @@ def test_ps_s3_notification_errors_on_master(): conn.delete_bucket(bucket_name) -def notification_push(endpoint_type, conn, account=None, cloudevents=False): +def notification_push(endpoint_type, conn, account=None, cloudevents=False, kafka_brokers=None): """ test pushinging notification """ zonegroup = get_config_zonegroup() # create bucket @@ -1359,11 +1367,13 @@ def notification_push(endpoint_type, conn, account=None, cloudevents=False): assert_equal(status/100, 2) elif endpoint_type == 'kafka': # start amqp receiver - task, receiver = create_kafka_receiver_thread(topic_name) + task, receiver = create_kafka_receiver_thread(topic_name, kafka_brokers=kafka_brokers) task.start() endpoint_address = 'kafka://' + kafka_server # without acks from broker endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker' + if kafka_brokers is not None: + endpoint_args += '&kafka-brokers=' + kafka_brokers # create s3 topic topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() @@ -1581,6 +1591,20 @@ def test_notification_push_kafka(): notification_push('kafka', conn) +@attr('kafka_failover') +def test_notification_push_kafka_multiple_brokers_override(): + """ test pushing kafka s3 notification on master """ + conn = connection() + notification_push('kafka', conn, kafka_brokers='localhost:9092,localhost:19092') + + +@attr('kafka_failover') +def test_notification_push_kafka_multiple_brokers_append(): + """ test pushing kafka s3 notification on master """ + conn = connection() + notification_push('kafka', conn, kafka_brokers='localhost:19092') + + @attr('http_test') def test_ps_s3_notification_multi_delete_on_master(): """ test deletion of multiple keys on master """ @@ -2981,7 +3005,6 @@ def wait_for_queue_to_drain(topic_name, tenant=None, account=None, http_port=Non log.info('waited for %ds for queue %s to drain', time_diff, topic_name) -@attr('kafka_test') def persistent_topic_stats(conn, endpoint_type): zonegroup = get_config_zonegroup() @@ -2993,12 +3016,13 @@ def persistent_topic_stats(conn, endpoint_type): host = get_ip() task = None port = None + wrong_port = 1234 + endpoint_address = endpoint_type+'://'+host+':'+str(wrong_port) if endpoint_type == 'http': # create random port for the http server port = random.randint(10000, 20000) # start an http server in a separate thread receiver = HTTPServerWithEvents((host, port)) - endpoint_address = 'http://'+host+':'+str(port) endpoint_args = 'push-endpoint='+endpoint_address+'&persistent=true'+ \ '&retry_sleep_duration=1' elif endpoint_type == 'amqp': @@ -3006,23 +3030,18 @@ def persistent_topic_stats(conn, endpoint_type): exchange = 'ex1' task, receiver = create_amqp_receiver_thread(exchange, topic_name) task.start() - endpoint_address = 'amqp://' + host endpoint_args = 'push-endpoint='+endpoint_address+'&amqp-exchange='+exchange+'&amqp-ack-level=broker&persistent=true'+ \ '&retry_sleep_duration=1' elif endpoint_type == 'kafka': # start kafka receiver task, receiver = create_kafka_receiver_thread(topic_name) task.start() - endpoint_address = 'kafka://' + host endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ '&retry_sleep_duration=1' else: return SkipTest('Unknown endpoint type: ' + endpoint_type) # create s3 topic - endpoint_address = 'kafka://' + host + ':1234' # wrong port - endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ - '&retry_sleep_duration=1' topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() # create s3 notification @@ -3070,9 +3089,19 @@ def persistent_topic_stats(conn, endpoint_type): get_stats_persistent_topic(topic_name, 2 * number_of_objects) # change the endpoint port - endpoint_address = 'kafka://' + host - endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ - '&retry_sleep_duration=1' + if endpoint_type == 'http': + endpoint_address = endpoint_type+'://'+host+':'+str(port) + endpoint_args = 'push-endpoint='+endpoint_address+'&persistent=true'+ \ + '&retry_sleep_duration=1' + elif endpoint_type == 'amqp': + endpoint_address = endpoint_type+'://'+host + endpoint_args = 'push-endpoint='+endpoint_address+'&amqp-exchange='+exchange+'&amqp-ack-level=broker&persistent=true'+ \ + '&retry_sleep_duration=1' + elif endpoint_type == 'kafka': + endpoint_address = endpoint_type+'://'+host + endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ + '&retry_sleep_duration=1' + topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() @@ -3087,19 +3116,26 @@ def persistent_topic_stats(conn, endpoint_type): @attr('http_test') -def persistent_topic_stats_http(): +def test_persistent_topic_stats_http(): """ test persistent topic stats, http endpoint """ conn = connection() persistent_topic_stats(conn, 'http') @attr('kafka_test') -def persistent_topic_stats_kafka(): +def test_persistent_topic_stats_kafka(): """ test persistent topic stats, kafka endpoint """ conn = connection() persistent_topic_stats(conn, 'kafka') +@attr('amqp_test') +def test_persistent_topic_stats_amqp(): + """ test persistent topic stats, amqp endpoint """ + conn = connection() + persistent_topic_stats(conn, 'amqp') + + @attr('kafka_test') def test_persistent_topic_dump(): """ test persistent topic dump """ @@ -4359,6 +4395,242 @@ def test_ps_s3_multiple_topics_notification(): http_server.close() +@attr('data_path_v2_test') +def test_ps_s3_list_topics_migration(): + """ test list topics on migration""" + if get_config_cluster() == 'noname': + return SkipTest('realm is needed for migration test') + + # Initialize connections and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topics = [f"{bucket_name}{TOPIC_SUFFIX}{i}" for i in range(1, 7)] + tenant_topics = [f"{tenant}_{topic}" for topic in topics] + + # Define topic names with version + topic_versions = { + "topic1_v2": f"{topics[0]}_v2", + "topic2_v2": f"{topics[1]}_v2", + "topic3_v1": f"{topics[2]}_v1", + "topic4_v1": f"{topics[3]}_v1", + "topic5_v1": f"{topics[4]}_v1", + "topic6_v1": f"{topics[5]}_v1", + "tenant_topic1_v2": f"{tenant_topics[0]}_v2", + "tenant_topic2_v1": f"{tenant_topics[1]}_v1", + "tenant_topic3_v1": f"{tenant_topics[2]}_v1" + } + + # Get necessary configurations + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + conf_cluster = get_config_cluster() + + # Make sure there are no leftover topics on v2 + zonegroup_modify_feature(enable=True, feature_name=zonegroup_feature_notification_v2) + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Start v1 notification + # Make sure there are no leftover topics on v1 + zonegroup_modify_feature(enable=False, feature_name=zonegroup_feature_notification_v2) + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Create s3 - v1 topics + topic_conf = PSTopicS3(conn1, topic_versions['topic3_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic4_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn4 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic5_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn5 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic6_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn6 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic2_v1'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic3_v1'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn3 = tenant_topic_conf.set_config() + + # Start v2 notification + zonegroup_modify_feature(enable=True, feature_name=zonegroup_feature_notification_v2) + + # Create s3 - v2 topics + topic_conf = PSTopicS3(conn1, topic_versions['topic1_v2'], zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic2_v2'], zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic1_v2'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 6) + + # Verify tenant topics + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 3) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + topic_conf.del_config(topic_arn4) + topic_conf.del_config(topic_arn5) + topic_conf.del_config(topic_arn6) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + tenant_topic_conf.del_config(tenant_topic_arn3) + + +@attr('basic_test') +def test_ps_s3_list_topics(): + """ test list topics""" + + # Initialize connections, topic names and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topic_name1 = bucket_name + TOPIC_SUFFIX + '1' + topic_name2 = bucket_name + TOPIC_SUFFIX + '2' + topic_name3 = bucket_name + TOPIC_SUFFIX + '3' + tenant_topic_name1 = tenant + "_" + topic_name1 + tenant_topic_name2 = tenant + "_" + topic_name2 + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + + # Make sure there are no leftover topics + delete_all_topics(conn1, '', get_config_cluster()) + delete_all_topics(conn2, tenant, get_config_cluster()) + + # Create s3 - v2 topics + topic_conf = PSTopicS3(conn1, topic_name1, zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name2, zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name3, zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name1, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name2, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] # version 2 + assert_equal(len(member), 3) + + # Verify topics for tenant + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 2) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + +@attr('data_path_v2_test') +def test_ps_s3_list_topics_v1(): + """ test list topics on v1""" + if get_config_cluster() == 'noname': + return SkipTest('realm is needed') + + # Initialize connections and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topic_name1 = bucket_name + TOPIC_SUFFIX + '1' + topic_name2 = bucket_name + TOPIC_SUFFIX + '2' + topic_name3 = bucket_name + TOPIC_SUFFIX + '3' + tenant_topic_name1 = tenant + "_" + topic_name1 + tenant_topic_name2 = tenant + "_" + topic_name2 + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + conf_cluster = get_config_cluster() + + # Make sure there are no leftover topics + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Make sure that we disable v2 + zonegroup_modify_feature(enable=False, feature_name=zonegroup_feature_notification_v2) + + # Create s3 - v1 topics + topic_conf = PSTopicS3(conn1, topic_name1, zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name2, zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name3, zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name1, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name2, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 3) + + # Verify tenant topics + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 2) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + + @attr('basic_test') def test_ps_s3_topic_permissions(): """ test s3 topic set/get/delete permissions """ diff --git a/src/test/rgw/rgw_multi/tests.py b/src/test/rgw/rgw_multi/tests.py index d95feb5aa95..433cd034fe0 100644 --- a/src/test/rgw/rgw_multi/tests.py +++ b/src/test/rgw/rgw_multi/tests.py @@ -15,6 +15,7 @@ import boto import boto.s3.connection from boto.s3.website import WebsiteConfiguration from boto.s3.cors import CORSConfiguration +from botocore.exceptions import ClientError from nose.tools import eq_ as eq from nose.tools import assert_not_equal, assert_equal, assert_true, assert_false @@ -3638,4 +3639,23 @@ def test_copy_object_different_bucket(): CopySource = source_bucket.name + '/' + objname) zonegroup_bucket_checkpoint(zonegroup_conns, dest_bucket.name) - + +def test_bucket_create_location_constraint(): + for zonegroup in realm.current_period.zonegroups: + zonegroup_conns = ZonegroupConns(zonegroup) + for zg in realm.current_period.zonegroups: + z = zonegroup_conns.rw_zones[0] + bucket_name = gen_bucket_name() + if zg.name == zonegroup.name: + # my zonegroup should pass + z.s3_client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': zg.name}) + # check bucket location + response = z.s3_client.get_bucket_location(Bucket=bucket_name) + assert_equal(response['LocationConstraint'], zg.name) + else: + # other zonegroup should fail with 400 + e = assert_raises(ClientError, + z.s3_client.create_bucket, + Bucket=bucket_name, + CreateBucketConfiguration={'LocationConstraint': zg.name}) + assert e.response['ResponseMetadata']['HTTPStatusCode'] == 400 diff --git a/src/test/rgw/test_rgw_iam_policy.cc b/src/test/rgw/test_rgw_iam_policy.cc index 7dadb7812ff..1d13c2aa013 100644 --- a/src/test/rgw/test_rgw_iam_policy.cc +++ b/src/test/rgw/test_rgw_iam_policy.cc @@ -75,6 +75,8 @@ using rgw::IAM::s3GetObjectTagging; using rgw::IAM::s3GetObjectVersion; using rgw::IAM::s3GetObjectVersionTagging; using rgw::IAM::s3GetObjectVersionTorrent; +using rgw::IAM::s3GetObjectAttributes; +using rgw::IAM::s3GetObjectVersionAttributes; using rgw::IAM::s3GetPublicAccessBlock; using rgw::IAM::s3GetReplicationConfiguration; using rgw::IAM::s3ListAllMyBuckets; @@ -419,6 +421,8 @@ TEST_F(PolicyTest, Parse3) { act2[s3GetObjectVersionAcl] = 1; act2[s3GetObjectTorrent] = 1; act2[s3GetObjectVersionTorrent] = 1; + act2[s3GetObjectAttributes] = 1; + act2[s3GetObjectVersionAttributes] = 1; act2[s3GetAccelerateConfiguration] = 1; act2[s3GetBucketAcl] = 1; act2[s3GetBucketOwnershipControls] = 1; @@ -487,6 +491,8 @@ TEST_F(PolicyTest, Eval3) { s3allow[s3GetObjectVersion] = 1; s3allow[s3GetObjectAcl] = 1; s3allow[s3GetObjectVersionAcl] = 1; + s3allow[s3GetObjectAttributes] = 1; + s3allow[s3GetObjectVersionAttributes] = 1; s3allow[s3GetObjectTorrent] = 1; s3allow[s3GetObjectVersionTorrent] = 1; s3allow[s3GetAccelerateConfiguration] = 1; @@ -883,6 +889,8 @@ TEST_F(ManagedPolicyTest, AmazonS3ReadOnlyAccess) act[s3GetObjectVersionAcl] = 1; act[s3GetObjectTorrent] = 1; act[s3GetObjectVersionTorrent] = 1; + act[s3GetObjectAttributes] = 1; + act[s3GetObjectVersionAttributes] = 1; act[s3GetAccelerateConfiguration] = 1; act[s3GetBucketAcl] = 1; act[s3GetBucketOwnershipControls] = 1; diff --git a/src/test/test_ipaddr.cc b/src/test/test_ipaddr.cc index 49038815318..21df1d4056b 100644 --- a/src/test/test_ipaddr.cc +++ b/src/test/test_ipaddr.cc @@ -995,3 +995,158 @@ TEST(pick_address, ipv4_ipv6_enabled2) ASSERT_EQ(-1, r); } } + +// Test for IPv4 address +TEST(is_addr_in_subnet, ipv4) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("10.1.1.2", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(true, r); +} + +// Test for IPv6 address +TEST(is_addr_in_subnet, ipv6) +{ + std::string public_network = "2001:db8::/64"; + entity_addr_t addr; + addr.parse("2001:db8::1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv6", "true"); + cct->_conf.set_val("ms_bind_ipv4", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(true, r); +} + +// Test for invalid address +TEST(is_addr_in_subnet, invalid_address) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("192.168.1.1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(false, r); +} + +// Test for malformed address +TEST(is_addr_in_subnet, malformed_address) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("invalid_address", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + // Test with a malformed address + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(false, r); +} + +TEST(is_addr_in_subnet, boundary_ipv4) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr_low; + addr_low.parse("10.1.1.0", nullptr); + entity_addr_t addr_high; + addr_high.parse("10.1.1.255", nullptr); + entity_addr_t addr_out; + addr_out.parse("10.1.2.0", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_low)); + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_high)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network, addr_out)); +} + +TEST(is_addr_in_subnet, boundary_ipv6) +{ + std::string public_network = "2001:db8::/64"; + entity_addr_t addr_low; + addr_low.parse("2001:db8::", nullptr); + entity_addr_t addr_high; + addr_high.parse("2001:db8:0:0:ffff:ffff:ffff:ffff", nullptr); + entity_addr_t addr_out; + addr_out.parse("2001:db9::", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv6", "true"); + cct->_conf.set_val("ms_bind_ipv4", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_low)); + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_high)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network, addr_out)); +} + +TEST(is_addr_in_subnet, overlapping_subnets) +{ + std::string public_network_1 = "10.1.1.0/24"; + std::string public_network_2 = "10.1.2.0/24"; + entity_addr_t addr; + addr.parse("10.1.1.5", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network_1, addr)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr)); +} + +TEST(is_addr_in_subnet, mismatched_family) +{ + std::string public_network_1 = "2001:db8::/64"; + entity_addr_t addr_1; + addr_1.parse("10.1.1.5", nullptr); + + std::string public_network_2 = "10.1.1.0/24"; + entity_addr_t addr_2; + addr_2.parse("2001:db8::1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "true"); + + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_1, addr_1)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr_2)); +} + +TEST(is_addr_in_subnet, invalid_subnets) +{ + std::string public_network_1 = "10.1.1.0/33"; + std::string public_network_2 = "25.0.0.99/10"; + entity_addr_t addr; + addr.parse("10.1.1.2", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_1, addr)); // Invalid prefix + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr)); // Invalid subnet string +} + diff --git a/src/tools/cephfs_mirror/PeerReplayer.cc b/src/tools/cephfs_mirror/PeerReplayer.cc index 91117cf5f2b..77e93ef6a99 100644 --- a/src/tools/cephfs_mirror/PeerReplayer.cc +++ b/src/tools/cephfs_mirror/PeerReplayer.cc @@ -120,7 +120,9 @@ int opendirat(MountRef mnt, int dirfd, const std::string &relpath, int flags, int fd = r; r = ceph_fdopendir(mnt, fd, dirp); - ceph_close(mnt, fd); + if (r < 0) { + ceph_close(mnt, fd); + } return r; } @@ -1222,15 +1224,6 @@ int PeerReplayer::sync_perms(const std::string& path) { return 0; } -void PeerReplayer::post_sync_close_handles(const FHandles &fh) { - dout(20) << dendl; - - // @FHandles.r_fd_dir_root is closed in @unregister_directory since - // its used to acquire an exclusive lock on remote dir_root. - ceph_close(m_local_mount, fh.c_fd); - ceph_close(fh.p_mnt, fh.p_fd); -} - int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot ¤t) { dout(20) << ": dir_root=" << dir_root << ", current=" << current << dendl; FHandles fh; @@ -1240,10 +1233,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu return r; } - BOOST_SCOPE_EXIT_ALL( (this)(&fh) ) { - post_sync_close_handles(fh); - }; - // record that we are going to "dirty" the data under this // directory root auto snap_id_str{stringify(current.second)}; @@ -1252,6 +1241,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": error setting \"ceph.mirror.dirty_snap_id\" on dir_root=" << dir_root << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1263,6 +1254,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to stat snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1271,8 +1264,12 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to open local snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } + // starting from this point we shouldn't care about manual closing of fh.c_fd, + // it will be closed automatically when bound tdirp is closed. std::stack<SyncEntry> sync_stack; sync_stack.emplace(SyncEntry(".", tdirp, tstx)); @@ -1282,12 +1279,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu break; } - r = pre_sync_check_and_open_handles(dir_root, current, boost::none, &fh); - if (r < 0) { - dout(5) << ": cannot proceed with sync: " << cpp_strerror(r) << dendl; - return r; - } - dout(20) << ": " << sync_stack.size() << " entries in stack" << dendl; std::string e_name; auto &entry = sync_stack.top(); @@ -1390,6 +1381,18 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu sync_stack.pop(); } + dout(20) << " cur:" << fh.c_fd + << " prev:" << fh.p_fd + << " ret = " << r + << dendl; + + // @FHandles.r_fd_dir_root is closed in @unregister_directory since + // its used to acquire an exclusive lock on remote dir_root. + + // c_fd has been used in ceph_fdopendir call so + // there is no need to close this fd manually. + ceph_close(fh.p_mnt, fh.p_fd); + return r; } @@ -1409,9 +1412,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu return r; } - BOOST_SCOPE_EXIT_ALL( (this)(&fh) ) { - post_sync_close_handles(fh); - }; // record that we are going to "dirty" the data under this directory root auto snap_id_str{stringify(current.second)}; @@ -1420,6 +1420,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": error setting \"ceph.mirror.dirty_snap_id\" on dir_root=" << dir_root << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1431,6 +1433,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to stat snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1450,11 +1454,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu dout(0) << ": backing off r=" << r << dendl; break; } - r = pre_sync_check_and_open_handles(dir_root, current, prev, &fh); - if (r < 0) { - dout(5) << ": cannot proceed with sync: " << cpp_strerror(r) << dendl; - return r; - } dout(20) << ": " << sync_queue.size() << " entries in queue" << dendl; const auto &queue_entry = sync_queue.front(); @@ -1464,12 +1463,16 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu stringify((*prev).first).c_str(), current.first.c_str(), &sd_info); if (r != 0) { derr << ": failed to open snapdiff, r=" << r << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } while (0 < (r = ceph_readdir_snapdiff(&sd_info, &sd_entry))) { if (r < 0) { derr << ": failed to read directory=" << epath << dendl; ceph_close_snapdiff(&sd_info); + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1561,6 +1564,17 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu } sync_queue.pop(); } + + dout(20) << " current:" << fh.c_fd + << " prev:" << fh.p_fd + << " ret = " << r + << dendl; + + // @FHandles.r_fd_dir_root is closed in @unregister_directory since + // its used to acquire an exclusive lock on remote dir_root. + + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } diff --git a/src/tools/cephfs_mirror/PeerReplayer.h b/src/tools/cephfs_mirror/PeerReplayer.h index 933cb182635..32c71301f00 100644 --- a/src/tools/cephfs_mirror/PeerReplayer.h +++ b/src/tools/cephfs_mirror/PeerReplayer.h @@ -307,7 +307,6 @@ private: int open_dir(MountRef mnt, const std::string &dir_path, boost::optional<uint64_t> snap_id); int pre_sync_check_and_open_handles(const std::string &dir_root, const Snapshot ¤t, boost::optional<Snapshot> prev, FHandles *fh); - void post_sync_close_handles(const FHandles &fh); int do_synchronize(const std::string &dir_root, const Snapshot ¤t, boost::optional<Snapshot> prev); diff --git a/src/tools/monmaptool.cc b/src/tools/monmaptool.cc index f1b86e00362..dc882a006a2 100644 --- a/src/tools/monmaptool.cc +++ b/src/tools/monmaptool.cc @@ -375,6 +375,10 @@ int main(int argc, const char **argv) return r; } + if (handle_features(features, monmap)) { + modified = true; + } + if (min_mon_release != ceph_release_t::unknown) { monmap.min_mon_release = min_mon_release; cout << "setting min_mon_release = " << min_mon_release << std::endl; @@ -459,10 +463,6 @@ int main(int argc, const char **argv) monmap.remove(p); } - if (handle_features(features, monmap)) { - modified = true; - } - if (!print && !modified && !show_features) { cerr << "no action specified" << std::endl; helpful_exit(); diff --git a/src/tools/rbd/Utils.cc b/src/tools/rbd/Utils.cc index 95c8725aa33..b20dca05bc6 100644 --- a/src/tools/rbd/Utils.cc +++ b/src/tools/rbd/Utils.cc @@ -337,11 +337,14 @@ int get_pool_image_snapshot_names(const po::variables_map &vm, SpecValidation spec_validation) { std::string pool_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_POOL_NAME : at::POOL_NAME); + std::string namespace_key = (mod == at::ARGUMENT_MODIFIER_DEST ? + at::DEST_NAMESPACE_NAME : at::NAMESPACE_NAME); std::string image_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_IMAGE_NAME : at::IMAGE_NAME); + return get_pool_generic_snapshot_names(vm, mod, spec_arg_index, pool_key, - pool_name, namespace_name, image_key, - "image", image_name, snap_name, + pool_name, namespace_key, namespace_name, + image_key, "image", image_name, snap_name, image_name_required, snapshot_presence, spec_validation); } @@ -351,6 +354,7 @@ int get_pool_generic_snapshot_names(const po::variables_map &vm, size_t *spec_arg_index, const std::string& pool_key, std::string *pool_name, + const std::string& namespace_key, std::string *namespace_name, const std::string& generic_key, const std::string& generic_key_desc, @@ -359,8 +363,6 @@ int get_pool_generic_snapshot_names(const po::variables_map &vm, bool generic_name_required, SnapshotPresence snapshot_presence, SpecValidation spec_validation) { - std::string namespace_key = (mod == at::ARGUMENT_MODIFIER_DEST ? - at::DEST_NAMESPACE_NAME : at::NAMESPACE_NAME); std::string snap_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_SNAPSHOT_NAME : at::SNAPSHOT_NAME); diff --git a/src/tools/rbd/Utils.h b/src/tools/rbd/Utils.h index 5076fd7fe9c..6aa0f2fdbdf 100644 --- a/src/tools/rbd/Utils.h +++ b/src/tools/rbd/Utils.h @@ -163,10 +163,11 @@ int get_pool_generic_snapshot_names( const boost::program_options::variables_map &vm, argument_types::ArgumentModifier mod, size_t *spec_arg_index, const std::string& pool_key, std::string *pool_name, - std::string *namespace_name, const std::string& generic_key, - const std::string& generic_key_desc, std::string *generic_name, - std::string *snap_name, bool generic_name_required, - SnapshotPresence snapshot_presence, SpecValidation spec_validation); + const std::string& namespace_key, std::string *namespace_name, + const std::string& generic_key, const std::string& generic_key_desc, + std::string *generic_name, std::string *snap_name, + bool generic_name_required, SnapshotPresence snapshot_presence, + SpecValidation spec_validation); int get_pool_image_id(const boost::program_options::variables_map &vm, size_t *spec_arg_index, diff --git a/src/tools/rbd/action/Group.cc b/src/tools/rbd/action/Group.cc index d97e120d438..100bdc19496 100644 --- a/src/tools/rbd/action/Group.cc +++ b/src/tools/rbd/action/Group.cc @@ -28,6 +28,9 @@ static const std::string DEST_GROUP_NAME("dest-group"); static const std::string GROUP_POOL_NAME("group-" + at::POOL_NAME); static const std::string IMAGE_POOL_NAME("image-" + at::POOL_NAME); +static const std::string GROUP_NAMESPACE_NAME("group-" + at::NAMESPACE_NAME); +static const std::string IMAGE_NAMESPACE_NAME("image-" + at::NAMESPACE_NAME); + void add_group_option(po::options_description *opt, at::ArgumentModifier modifier) { std::string name = GROUP_NAME; @@ -107,8 +110,8 @@ int execute_create(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -187,8 +190,8 @@ int execute_remove(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -221,8 +224,8 @@ int execute_rename(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -233,9 +236,9 @@ int execute_rename(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_DEST, &arg_index, at::DEST_POOL_NAME, - &dest_pool_name, &dest_namespace_name, DEST_GROUP_NAME, "group", - &dest_group_name, nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &dest_pool_name, at::DEST_NAMESPACE_NAME, &dest_namespace_name, + DEST_GROUP_NAME, "group", &dest_group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -283,8 +286,8 @@ int execute_info(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -335,8 +338,9 @@ int execute_add(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, GROUP_POOL_NAME, - &group_pool_name, &group_namespace_name, GROUP_NAME, "group", &group_name, - nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + &group_pool_name, GROUP_NAMESPACE_NAME, &group_namespace_name, + GROUP_NAME, "group", &group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -347,9 +351,9 @@ int execute_add(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, IMAGE_POOL_NAME, - &image_pool_name, &image_namespace_name, at::IMAGE_NAME, "image", - &image_name, nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &image_pool_name, IMAGE_NAMESPACE_NAME, &image_namespace_name, + at::IMAGE_NAME, "image", &image_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -393,8 +397,9 @@ int execute_remove_image(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, GROUP_POOL_NAME, - &group_pool_name, &group_namespace_name, GROUP_NAME, "group", &group_name, - nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + &group_pool_name, GROUP_NAMESPACE_NAME, &group_namespace_name, + GROUP_NAME, "group", &group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -410,9 +415,9 @@ int execute_remove_image(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, IMAGE_POOL_NAME, - &image_pool_name, &image_namespace_name, at::IMAGE_NAME, "image", - &image_name, nullptr, image_id.empty(), utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &image_pool_name, IMAGE_NAMESPACE_NAME, &image_namespace_name, + at::IMAGE_NAME, "image", &image_name, nullptr, image_id.empty(), + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -464,8 +469,8 @@ int execute_list_images(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -563,8 +568,9 @@ int execute_group_snap_create(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -604,8 +610,9 @@ int execute_group_snap_remove(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -640,8 +647,9 @@ int execute_group_snap_rename(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &source_snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &source_snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -696,8 +704,8 @@ int execute_group_snap_list(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -764,8 +772,9 @@ int execute_group_snap_info(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &group_snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &group_snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -872,8 +881,9 @@ int execute_group_snap_rollback(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -954,9 +964,6 @@ void get_add_arguments(po::options_description *positional, add_prefixed_pool_option(options, "image"); add_prefixed_namespace_option(options, "image"); at::add_image_option(options, at::ARGUMENT_MODIFIER_NONE); - - at::add_pool_option(options, at::ARGUMENT_MODIFIER_NONE, - " unless overridden"); } void get_remove_image_arguments(po::options_description *positional, @@ -979,8 +986,6 @@ void get_remove_image_arguments(po::options_description *positional, add_prefixed_namespace_option(options, "image"); at::add_image_option(options, at::ARGUMENT_MODIFIER_NONE); - at::add_pool_option(options, at::ARGUMENT_MODIFIER_NONE, - " unless overridden"); at::add_image_id_option(options); } diff --git a/src/tools/rbd/action/MirrorPool.cc b/src/tools/rbd/action/MirrorPool.cc index 58e2d4dc329..6a546c3f73a 100644 --- a/src/tools/rbd/action/MirrorPool.cc +++ b/src/tools/rbd/action/MirrorPool.cc @@ -355,6 +355,10 @@ protected: virtual ~ImageRequestBase() { } + virtual bool open_read_only() const { + return false; + } + virtual bool skip_get_info() const { return false; } @@ -429,8 +433,13 @@ private: librbd::RBD rbd; auto aio_completion = utils::create_aio_completion< ImageRequestBase, &ImageRequestBase::handle_open_image>(this); - rbd.aio_open(m_io_ctx, m_image, m_image_name.c_str(), nullptr, - aio_completion); + if (open_read_only()) { + rbd.aio_open_read_only(m_io_ctx, m_image, m_image_name.c_str(), nullptr, + aio_completion); + } else { + rbd.aio_open(m_io_ctx, m_image, m_image_name.c_str(), nullptr, + aio_completion); + } } void handle_open_image(int r) { @@ -604,6 +613,10 @@ public: } protected: + bool open_read_only() const override { + return true; + } + bool skip_get_info() const override { return true; } diff --git a/src/vstart.sh b/src/vstart.sh index 41a8310891b..a992f33c856 100755 --- a/src/vstart.sh +++ b/src/vstart.sh @@ -159,6 +159,7 @@ smallmds=0 short=0 crimson=0 ec=0 +cephexporter=0 cephadm=0 parallel=true restart=1 @@ -233,6 +234,7 @@ options: -G disable Kerberos/GSSApi authentication --hitset <pool> <hit_set_type>: enable hitset tracking -e : create an erasure pool + --cephexporter: start the ceph-exporter daemon -o config add extra config parameters to all sections --rgw_port specify ceph rgw http listen port --rgw_frontend specify the rgw frontend configuration @@ -372,6 +374,9 @@ case $1 in -e) ec=1 ;; + --cephexporter) + cephexporter=1 + ;; --new | -n) new=1 ;; @@ -963,7 +968,17 @@ $BLUESTORE_OPTS ; kstore kstore fsck on mount = true +EOF + if [ "$crimson" -eq 1 ]; then + wconf <<EOF + crimson osd objectstore = $objectstore +EOF + else + wconf <<EOF osd objectstore = $objectstore +EOF + fi + wconf <<EOF $SEASTORE_OPTS $COSDSHORT $(format_conf "${extra_conf}") @@ -1130,6 +1145,17 @@ EOF fi } +start_cephexporter() { + debug echo "Starting Ceph exporter daemon..." + + # Define socket directory for the exporter + # Start the exporter daemon + prunb ceph-exporter \ + -c "$conf_fn" \ + --sock-dir "$CEPH_ASOK_DIR" \ + --addrs "$IP" +} + start_osd() { if [ $inc_osd_num -gt 0 ]; then old_maxosd=$($CEPH_BIN/ceph osd getmaxosd | sed -e 's/max_osd = //' -e 's/ in epoch.*//') @@ -1676,28 +1702,30 @@ if [ "$ceph_osd" == "crimson-osd" ]; then if [ "$trace" -ne 0 ]; then extra_seastar_args=" --trace" fi - if [ "$(expr $(nproc) - 1)" -gt "$(($CEPH_NUM_OSD * crimson_smp))" ]; then - if [ $crimson_alien_num_cores -gt 0 ]; then - alien_bottom_cpu=$(($CEPH_NUM_OSD * crimson_smp)) - alien_top_cpu=$(( alien_bottom_cpu + crimson_alien_num_cores - 1 )) - # Ensure top value within range: - if [ "$(($alien_top_cpu))" -gt "$(expr $(nproc) - 1)" ]; then - alien_top_cpu=$(expr $(nproc) - 1) + if [ "$objectstore" == "bluestore" ]; then + if [ "$(expr $(nproc) - 1)" -gt "$(($CEPH_NUM_OSD * crimson_smp))" ]; then + if [ $crimson_alien_num_cores -gt 0 ]; then + alien_bottom_cpu=$(($CEPH_NUM_OSD * crimson_smp)) + alien_top_cpu=$(( alien_bottom_cpu + crimson_alien_num_cores - 1 )) + # Ensure top value within range: + if [ "$(($alien_top_cpu))" -gt "$(expr $(nproc) - 1)" ]; then + alien_top_cpu=$(expr $(nproc) - 1) + fi + echo "crimson_alien_thread_cpu_cores: $alien_bottom_cpu-$alien_top_cpu" + # This is a (logical) processor id range, it could be refined to encompass only physical processor ids + # (equivalently, ignore hyperthreading sibling processor ids) + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores "$alien_bottom_cpu-$alien_top_cpu" + else + echo "crimson_alien_thread_cpu_cores:" $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" + fi + if [ $crimson_alien_num_threads -gt 0 ]; then + echo "$CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads $crimson_alien_num_threads" + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads "$crimson_alien_num_threads" fi - echo "crimson_alien_thread_cpu_cores: $alien_bottom_cpu-$alien_top_cpu" - # This is a (logical) processor id range, it could be refined to encompass only physical processor ids - # (equivalently, ignore hyperthreading sibling processor ids) - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores "$alien_bottom_cpu-$alien_top_cpu" else - echo "crimson_alien_thread_cpu_cores:" $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" - fi - if [ $crimson_alien_num_threads -gt 0 ]; then - echo "$CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads $crimson_alien_num_threads" - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads "$crimson_alien_num_threads" + echo "No alien thread cpu core isolation" fi - else - echo "No alien thread cpu core isolation" fi fi @@ -1726,6 +1754,10 @@ if [ $CEPH_NUM_MDS -gt 0 ]; then ceph_adm fs authorize \* "client.fs" / rwp >> "$keyring_fn" fi +if [ "$cephexporter" -eq 1 ]; then + start_cephexporter +fi + # Don't set max_mds until all the daemons are started, otherwise # the intended standbys might end up in active roles. if [ "$CEPH_MAX_MDS" -gt 1 ]; then |
