diff options
757 files changed, 26624 insertions, 7328 deletions
diff --git a/.github/labeler.yml b/.github/labeler.yml index cc32be38501..3a3a348dcfe 100644 --- a/.github/labeler.yml +++ b/.github/labeler.yml @@ -6,6 +6,7 @@ build/ops: - admin/** - ceph.spec.in - cmake/** + - container/** - debian/** - do_cmake.sh - do_freebsd.sh diff --git a/.githubmap b/.githubmap index 68c711aa587..5265fa59bed 100644 --- a/.githubmap +++ b/.githubmap @@ -188,3 +188,4 @@ robbat2 Robin H. Johnson <robbat2@orbis-terrarum.net> leonid-s-usov Leonid Usov <leonid.usov@ibm.com> ffilz Frank S. Filz <ffilzlnx@mindspring.com> Jayaprakash-ibm Jaya Prakash Madaka <jayaprakash@ibm.com> +spuiuk Sachin Prabhu <sp@spui.uk> diff --git a/PendingReleaseNotes b/PendingReleaseNotes index 146cab64d6f..b4824a65584 100644 --- a/PendingReleaseNotes +++ b/PendingReleaseNotes @@ -1,5 +1,16 @@ >=20.0.0 +* RGW: The User Account feature introduced in Squid provides first-class support for + IAM APIs and policy. Our preliminary STS support was instead based on tenants, and + exposed some IAM APIs to admins only. This tenant-level IAM functionality is now + deprecated in favor of accounts. While we'll continue to support the tenant feature + itself for namespace isolation, the following features will be removed no sooner + than the V release: + * tenant-level IAM APIs like CreateRole, PutRolePolicy and PutUserPolicy, + * use of tenant names instead of accounts in IAM policy documents, + * interpretation of IAM policy without cross-account policy evaluation, + * S3 API support for cross-tenant names such as `Bucket='tenant:bucketname'` + * RBD: All Python APIs that produce timestamps now return "aware" `datetime` objects instead of "naive" ones (i.e. those including time zone information instead of those not including it). All timestamps remain to be in UTC but @@ -34,6 +45,8 @@ (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. +* RADOS: Added convenience function `librados::AioCompletion::cancel()` with + the same behavior as `librados::IoCtx::aio_cancel()`. * mgr/restful, mgr/zabbix: both modules, already deprecated since 2020, have been finally removed. They have not been actively maintenance in the last years, @@ -48,6 +61,10 @@ CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. +* The HeadBucket API now reports the `X-RGW-Bytes-Used` and `X-RGW-Object-Count` + headers only when the `read-stats` querystring is explicitly included in the + API request. + >=19.2.1 * CephFS: Command `fs subvolume create` now allows tagging subvolumes through option @@ -57,6 +74,9 @@ `ceph fs subvolume earmark set`, `ceph fs subvolume earmark get` and `ceph fs subvolume earmark rm` have been added to set, get and remove earmark from a given subvolume. +* RADOS: A performance botteneck in the balancer mgr module has been fixed. + Related Tracker: https://tracker.ceph.com/issues/68657 + >=19.0.0 * cephx: key rotation is now possible using `ceph auth rotate`. Previously, @@ -324,6 +344,8 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config * NFS: The export create/apply of CephFS based exports will now have a additional parameter `cmount_path` under the FSAL block, which specifies the path within the CephFS to mount this export on. If this and the other `EXPORT { FSAL {} }` options are the same between multiple exports, those exports will share a single CephFS client. If not specified, the default is `/`. +* CephFS: MDS emits a warning with estimated replay completion time when replay + runs for more than 30 seconds. >=18.0.0 diff --git a/README.md b/README.md index 56257697e9a..f8fcf35e8b7 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ To build Ceph, follow this procedure: ninja -j3 - > [IMPORTANT] + > [!IMPORTANT] > > [Ninja](https://ninja-build.org/) is the build system used by the Ceph > project to build test builds. The number of jobs used by `ninja` is @@ -126,6 +126,9 @@ To build Ceph, follow this procedure: 5. Install the vstart cluster: ninja install + + + ### CMake Options @@ -177,6 +180,36 @@ The diagnostic colors will be visible when the following command is run: Other available values for `DIAGNOSTICS_COLOR` are `auto` (default) and `never`. +## Tips and Tricks + + * Use "debug builds" only when needed. Debugging builds are helpful for + development, but they can slow down performance. Use + `-DCMAKE_BUILD_TYPE=Release` when debugging isn't necessary. + * Enable Selective Daemons when testing specific components. Don't start + unnecessary daemons. + * Preserve Existing Data skip cluster reinitialization between tests by + using the `-n` flag. + * To manage a vstart cluster, stop daemons using `./stop.sh` and start them + with `./vstart.sh --daemon osd.${ID} [--nodaemonize]`. + * Restart the sockets by stopping and restarting the daemons associated with + them. This ensures that there are no stale sockets in the cluster. + * To track RocksDB performance, set `export ROCKSDB_PERF=true` and start + the cluster by using the command `./vstart.sh -n -d -x --bluestore`. + * Build with `vstart-base` using debug flags in cmake, compile, and deploy + via `./vstart.sh -d -n --bluestore`. + * To containerize, generate configurations with `vstart.sh`, and deploy with + Docker, mapping directories and configuring the network. + * Manage containers using `docker run`, `stop`, and `rm`. For detailed + setups, consult the Ceph-Container repository. + +## Troubleshooting + + * Cluster Fails to Start: Look for errors in the logs under the `out/` + directory. + * OSD Crashes: Check the OSD logs for errors. + * Cluster in a `Health Error` State: Run the `ceph status` command to + identify the issue. + * RocksDB Errors: Look for RocksDB-related errors in the OSD logs. ## Building a source tarball diff --git a/ceph.spec.in b/ceph.spec.in index ece1ebf2ec8..1e4c1f0850c 100644 --- a/ceph.spec.in +++ b/ceph.spec.in @@ -1186,18 +1186,41 @@ Obsoletes: libcephfs1 < %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-libs < %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-libcephfs < %{_epoch_prefix}%{version}-%{release} %endif +Recommends: libcephfs-proxy2 = %{_epoch_prefix}%{version}-%{release} +Requires: libcephfs-daemon %description -n libcephfs2 Ceph is a distributed network file system designed to provide excellent performance, reliability, and scalability. This is a shared library allowing applications to access a Ceph distributed file system via a POSIX-like interface. +%package -n libcephfs-proxy2 +Summary: Proxy for libcephfs +%if 0%{?suse_version} +Group: System/Libraries +%endif +Recommends: libcephfs-daemon = %{_epoch_prefix}%{version}-%{release} +%description -n libcephfs-proxy2 +This package contains the libcephfs_proxy.so library that allow applications +to share cephfs mounts to reduce resource consumption. + +%package -n libcephfs-daemon +Summary: Deamon for the libcephfs proxy +%if 0%{?suse_version} +Group: System/Filesystems +%endif +Requires: libcephfs2 = %{_epoch_prefix}%{version}-%{release} +%description -n libcephfs-daemon +This package contains the libcephfsd daemon that allows applications to share +cephfs mounts to reduce resource consumption. + %package -n libcephfs-devel Summary: Ceph distributed file system headers %if 0%{?suse_version} Group: Development/Libraries/C and C++ %endif Requires: libcephfs2 = %{_epoch_prefix}%{version}-%{release} +Requires: libcephfs-proxy2 = %{_epoch_prefix}%{version}-%{release} Requires: librados-devel = %{_epoch_prefix}%{version}-%{release} Obsoletes: ceph-devel < %{_epoch_prefix}%{version}-%{release} Provides: libcephfs2-devel = %{_epoch_prefix}%{version}-%{release} @@ -2524,6 +2547,16 @@ fi %postun -n libcephfs2 -p /sbin/ldconfig +%files -n libcephfs-proxy2 +%{_libdir}/libcephfs_proxy.so.* + +%post -n libcephfs-proxy2 -p /sbin/ldconfig + +%postun -n libcephfs-proxy2 -p /sbin/ldconfig + +%files -n libcephfs-daemon +%{_sbindir}/libcephfsd + %files -n libcephfs-devel %dir %{_includedir}/cephfs %{_includedir}/cephfs/libcephfs.h @@ -2532,6 +2565,8 @@ fi %dir %{_includedir}/cephfs/metrics %{_includedir}/cephfs/metrics/Types.h %{_libdir}/libcephfs.so +%{_libdir}/libcephfs_proxy.so +%{_libdir}/pkgconfig/cephfs.pc %files -n python%{python3_pkgversion}-cephfs %{python3_sitearch}/cephfs.cpython*.so diff --git a/container/Containerfile b/container/Containerfile index 2f75c8c6ce6..9a5a88e76a1 100644 --- a/container/Containerfile +++ b/container/Containerfile @@ -22,6 +22,7 @@ ARG OSD_FLAVOR="default" # (optional) Should be 'true' for CI builds (pull from shaman, etc.) ARG CI_CONTAINER="true" + RUN /bin/echo -e "\ FROM_IMAGE: ${FROM_IMAGE}\n\ CEPH_REF: ${CEPH_REF}\n\ @@ -61,30 +62,36 @@ RUN \ echo "enabled=1" >> /etc/yum.repos.d/ganesha.repo # ISCSI repo -RUN set -x && \ +RUN set -ex && \ curl -s -L https://shaman.ceph.com/api/repos/tcmu-runner/main/latest/centos/9/repo?arch=$(arch) -o /etc/yum.repos.d/tcmu-runner.repo && \ case "${CEPH_REF}" in \ quincy|reef) \ - curl -s -L https://download.ceph.com/ceph-iscsi/3/rpm/el9/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ;\ + curl -fs -L https://download.ceph.com/ceph-iscsi/3/rpm/el9/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ;\ ;;\ main|*) \ - curl -s -L https://shaman.ceph.com/api/repos/ceph-iscsi/main/latest/centos/9/repo -o /etc/yum.repos.d/ceph-iscsi.repo ;\ + curl -fs -L https://shaman.ceph.com/api/repos/ceph-iscsi/main/latest/centos/9/repo -o /etc/yum.repos.d/ceph-iscsi.repo ;\ ;;\ esac # Ceph repo -RUN set -x && \ +RUN --mount=type=secret,id=prerelease_creds set -ex && \ rpm --import 'https://download.ceph.com/keys/release.asc' && \ ARCH=$(arch); if [ "${ARCH}" == "aarch64" ]; then ARCH="arm64"; fi ;\ IS_RELEASE=0 ;\ if [[ "${CI_CONTAINER}" == "true" ]] ; then \ # TODO: this can return different ceph builds (SHA1) for x86 vs. arm runs. is it important to fix? - REPO_URL=$(curl -s "https://shaman.ceph.com/api/search/?project=ceph&distros=centos/9/${ARCH}&flavor=${OSD_FLAVOR}&ref=${CEPH_REF}&sha1=latest" | jq -r .[0].url) ;\ + REPO_URL=$(curl -fs "https://shaman.ceph.com/api/search/?project=ceph&distros=centos/9/${ARCH}&flavor=${OSD_FLAVOR}&ref=${CEPH_REF}&sha1=latest" | jq -r .[0].url) ;\ else \ IS_RELEASE=1 ;\ - REPO_URL="http://download.ceph.com/rpm-${CEPH_REF}/el9/" ;\ + source /run/secrets/prerelease_creds; \ + REPO_URL="https://${PRERELEASE_USERNAME}:${PRERELEASE_PASSWORD}@download.ceph.com/prerelease/ceph/rpm-${CEPH_REF}/el9/" ;\ fi && \ - rpm -Uvh "$REPO_URL/noarch/ceph-release-1-${IS_RELEASE}.el9.noarch.rpm" + rpm -Uvh "$REPO_URL/noarch/ceph-release-1-${IS_RELEASE}.el9.noarch.rpm" ; \ + if [[ "$IS_RELEASE" == 1 ]] ; then \ + sed -i "s;http://download.ceph.com/;https://${PRERELEASE_USERNAME}:${PRERELEASE_PASSWORD}@download.ceph.com/prerelease/ceph/;" /etc/yum.repos.d/ceph.repo ; \ + dnf clean expire-cache ; \ + fi + # Copr repos # scikit for mgr-diskprediction-local @@ -186,7 +193,7 @@ RUN \ grep -sqo "obtain_device_list_from_udev = 0" /etc/lvm/lvm.conf # CLEAN UP! -RUN set -x && \ +RUN set -ex && \ dnf clean all && \ rm -rf /var/cache/dnf/* && \ rm -rf /var/lib/dnf/* && \ @@ -194,7 +201,8 @@ RUN set -x && \ # remove unnecessary files with big impact rm -rf /etc/selinux /usr/share/{doc,man,selinux} && \ # don't keep compiled python binaries - find / -xdev \( -name "*.pyc" -o -name "*.pyo" \) -delete + find / -xdev \( -name "*.pyc" -o -name "*.pyo" \) -delete && \ + rm -f /etc/yum.repos.d/{ceph,ganesha,tcmu-runner,ceph-iscsi}.repo # Verify that the packages installed haven't been accidentally cleaned, then # clean the package list and re-clean unnecessary RPM database files @@ -204,6 +212,7 @@ RUN rpm -q $(cat packages.txt) && rm -f /var/lib/rpm/__db* && rm -f *packages.tx # Set some envs in the container for quickly inspecting details about the build at runtime ENV CEPH_IS_DEVEL="${CI_CONTAINER}" \ CEPH_REF="${CEPH_REF}" \ + CEPH_VERSION="${CEPH_REF}" \ CEPH_OSD_FLAVOR="${OSD_FLAVOR}" \ FROM_IMAGE="${FROM_IMAGE}" diff --git a/container/build.sh b/container/build.sh index 5edf469d2d2..d7712524e4d 100755 --- a/container/build.sh +++ b/container/build.sh @@ -22,6 +22,8 @@ CONTAINER_REPO_HOSTNAME (quay.ceph.io, for CI, for instance) CONTAINER_REPO_ORGANIZATION (ceph-ci, for CI, for instance) CONTAINER_REPO_USERNAME CONTAINER_REPO_PASSWORD +PRERELEASE_USERNAME for download.ceph.com:/prerelease/ceph +PRERELEASE_PASSWORD For a release build: (from ceph.git, built and pushed to download.ceph.com) CI_CONTAINER: must be 'false' @@ -41,12 +43,17 @@ CEPH_SHA1=${CEPH_SHA1:-$(git rev-parse HEAD)} # default: build host arch ARCH=${ARCH:-$(arch)} if [[ "${ARCH}" == "aarch64" ]] ; then ARCH=arm64; fi +REPO_ARCH=amd64 +if [[ "${ARCH}" = arm64 ]] ; then + REPO_ARCH=arm64 +fi + if [[ ${CI_CONTAINER} == "true" ]] ; then CONTAINER_REPO_HOSTNAME=${CONTAINER_REPO_HOSTNAME:-quay.ceph.io} - CONTAINER_REPO_ORGANIZATION=${CONTAINER_REPO_ORGANIZATION:-ceph/ceph-${ARCH}} + CONTAINER_REPO_ORGANIZATION=${CONTAINER_REPO_ORGANIZATION:-ceph-ci/ceph} else - CONTAINER_REPO_HOSTNAME=${CONTAINER_REPO_HOSTNAME:-quay.io} - CONTAINER_REPO_ORGANIZATION=${CONTAINER_REPO_ORGANIZATION:-ceph/ceph} + CONTAINER_REPO_HOSTNAME=${CONTAINER_REPO_HOSTNAME:-quay.ceph.io} + CONTAINER_REPO_ORGANIZATION=${CONTAINER_REPO_ORGANIZATION:-ceph/prerelease-${REPO_ARCH}} # default: most-recent annotated tag VERSION=${VERSION:-$(git describe --abbrev=0)} fi @@ -61,7 +68,7 @@ fi : "${CONTAINER_REPO_ORGANIZATION:?}" : "${CONTAINER_REPO_USERNAME:?}" : "${CONTAINER_REPO_PASSWORD:?}" -if [[ ${CI_CONTAINER} != "true" ]] ; then ${VERSION:?}; fi +if [[ ${CI_CONTAINER} != "true" ]] ; then : "${VERSION:?}"; fi # check for valid repo auth (if pushing) ORGURL=${CONTAINER_REPO_HOSTNAME}/${CONTAINER_REPO_ORGANIZATION} @@ -87,6 +94,14 @@ fi # BRANCH will be, say, origin/main. remove <remote>/ BRANCH=${BRANCH##*/} +# podman build only supports secret files. +# This must be removed after podman build +touch prerelease.secret.txt +chmod 600 prerelease.secret.txt +echo -e "\ + PRERELEASE_USERNAME=${PRERELEASE_USERNAME}\n + PRERELEASE_PASSWORD=${PRERELEASE_PASSWORD}\n " > prerelease.secret.txt + podman build --pull=newer --squash -f $CFILE -t build.sh.output \ --build-arg FROM_IMAGE=${FROM_IMAGE:-quay.io/centos/centos:stream9} \ --build-arg CEPH_SHA1=${CEPH_SHA1} \ @@ -94,8 +109,11 @@ podman build --pull=newer --squash -f $CFILE -t build.sh.output \ --build-arg CEPH_REF=${BRANCH:-main} \ --build-arg OSD_FLAVOR=${FLAVOR:-default} \ --build-arg CI_CONTAINER=${CI_CONTAINER:-default} \ + --secret=id=prerelease_creds,src=./prerelease.secret.txt \ 2>&1 +rm ./prerelease.secret.txt + image_id=$(podman image ls localhost/build.sh.output --format '{{.ID}}') # grab useful image attributes for building the tag @@ -124,7 +142,7 @@ eval ${vars} fromtag=${CEPH_CONTAINER_FROM_IMAGE##*/} # translate : to - fromtag=${fromtag/:/-} -builddate=$(date +%Y%m%d) +builddate=$(date -u +%Y%m%d) local_tag=${fromtag}-${CEPH_CONTAINER_CEPH_REF}-${CEPH_CONTAINER_ARCH}-${builddate} repopath=${CONTAINER_REPO_HOSTNAME}/${CONTAINER_REPO_ORGANIZATION} @@ -162,13 +180,13 @@ if [[ ${CI_CONTAINER} == "true" ]] ; then else # # non-CI build. Tags are like v19.1.0-20240701 - # push to quay.ceph.io/ceph/prerelease + # push to quay.ceph.io/ceph/prerelease-$REPO_ARCH # - version_tag=${repopath}/prerelease/ceph-${ARCH}:${VERSION}-${builddate} + version_tag=${repopath}/prerelease-${REPO_ARCH}:v${VERSION}-${builddate} podman tag ${image_id} ${version_tag} if [[ -z "${NO_PUSH}" ]] ; then - podman push ${image_id} ${version_tag} + podman push ${version_tag} fi fi diff --git a/container/make-manifest-list.py b/container/make-manifest-list.py index 010dcaed2b7..27b00cc4777 100755 --- a/container/make-manifest-list.py +++ b/container/make-manifest-list.py @@ -1,12 +1,34 @@ #!/usr/bin/python3 # +# in default mode: # make a combined "manifest-list" container out of two arch-specific containers # searches for latest tags on HOST/{AMD,ARM}64_REPO, makes sure they refer # to the same Ceph SHA1, and creates a manifest-list ("fat") image on -# MANIFEST_HOST/MANIFEST_REPO with the 'standard' set of tags. +# MANIFEST_HOST/MANIFEST_REPO with the 'standard' set of tags: +# v<major> +# v<major>.<minor> +# v<major>.<minor>.<micro> +# v<major>.<minor>.<micro>-<YYYYMMDD> # -# uses scratch local manifest LOCALMANIFEST, will be destroyed if present +# uses scratch local manifest LOCALMANIFEST, defined here; will be destroyed if present +# +# in promote mode (by adding the --promote argument): +# instead of building the manifest-list container, copy it +# (and all of its tags) from the prerelease repo to the release repo +# +# Assumes valid logins to the necessary hosts/repos with permission to write images +# +# Environment variables to set: +# ARCH_SPECIFIC_HOST (default 'quay.ceph.io'): host of prerelease repos +# AMD64_REPO (default 'ceph/prerelease-amd64') prerelease amd64 repo +# ARM64_REPO (default 'ceph/prerelease-arm64') prerelease arm64 repo +# MANIFEST_HOST (default 'quay.ceph.io') prerelease manifest-list host +# MANIFEST_REPO (default 'ceph/prerelease') prerelease manifest-list repo +# RELEASE_MANIFEST_HOST (default 'quay.io') release host +# RELEASE_MANIFEST_REPO (default 'ceph/ceph') release repo + +import argparse from datetime import datetime import functools import json @@ -15,16 +37,6 @@ import re import subprocess import sys -# optional env vars (will default if not set) - -OPTIONAL_VARS = ( - 'HOST', - 'AMD64_REPO', - 'ARM64_REPO', - 'MANIFEST_HOST', - 'MANIFEST_REPO', -) - # Manifest image. Will be destroyed if already present. LOCALMANIFEST = 'localhost/m' @@ -47,10 +59,7 @@ def run_command(args): return True, result.stdout, result.stderr except subprocess.CalledProcessError as e: - print(f"Command '{e.cmd}' returned {e.returncode}") - print("Error output:") - print(e.stderr) - return False, result.stdout, result.stderr + return False, e.output, e.stderr def get_command_output(args): @@ -68,10 +77,16 @@ def run_command_show_failure(args): @functools.lru_cache +def get_tags(path): + cmdout = get_command_output(f'skopeo list-tags docker://{path}') + return json.loads(cmdout)['Tags'] + + def get_latest_tag(path): - latest_tag = json.loads( - get_command_output(f'skopeo list-tags docker://{path}') - )['Tags'][-1] + try: + latest_tag = get_tags(path)[-1] + except IndexError: + return None return latest_tag @@ -84,27 +99,53 @@ def get_image_inspect(path): def get_sha1(info): - return info['Labels']['GIT_COMMIT'] + labels = info.get('Labels', None) + if not labels: + return None + return labels.get('CEPH_SHA1', None) -def main(): - host = os.environ.get('HOST', 'quay.io') - amd64_repo = os.environ.get('AMD64_REPO', 'ceph/ceph-amd64') - arm64_repo = os.environ.get('ARM64_REPO', 'ceph/ceph-arm64') - manifest_host = os.environ.get('MANIFEST_HOST', host) - manifest_repo = os.environ.get('MANIFEST_REPO', 'ceph/ceph') +@functools.lru_cache +def get_all_matching_digest_tags(path, tag): + + matching_tags = list() + digest = get_image_inspect(f'{path}:{tag}')['Digest'] + + for t in get_tags(path): + this_digest = get_image_inspect(f'{path}:{t}')['Digest'] + if this_digest == digest: + matching_tags.append(t) + + return matching_tags + + +def parse_args(): + ap = argparse.ArgumentParser() + ap.add_argument('-n', '--dry-run', action='store_true', help='do all local manipulations but do not push final containers to MANIFEST_HOST, or in --promote, calculate but do not copy images to release host') + ap.add_argument('-P', '--promote', action='store_true', help='promote newest prerelease manifest container to released (move from MANIFEST_HOST to RELEASE_MANIFEST_HOST') + args = ap.parse_args() + return args + +def build_prerelease(sysargs): + global args + + arch_specific_host = os.environ.get('ARCH_SPECIFIC_HOST', 'quay.ceph.io') + amd64_repo = os.environ.get('AMD64_REPO', 'ceph/prerelease-amd64') + arm64_repo = os.environ.get('ARM64_REPO', 'ceph/prerelease-arm64') + manifest_host = os.environ.get('MANIFEST_HOST', 'quay.ceph.io') + manifest_repo = os.environ.get('MANIFEST_REPO', 'ceph/prerelease') + dump_vars( - ('host', + ('arch_specific_host', 'amd64_repo', 'arm64_repo', 'manifest_host', 'manifest_repo', ), locals()) - repopaths = ( - f'{host}/{amd64_repo}', - f'{host}/{arm64_repo}', + f'{arch_specific_host}/{amd64_repo}', + f'{arch_specific_host}/{arm64_repo}', ) tags = [get_latest_tag(p) for p in repopaths] print(f'latest tags: amd64:{tags[0]} arm64:{tags[1]}') @@ -145,8 +186,8 @@ def main(): # create manifest list image with the standard list of tags # ignore failure on manifest rm - run_command(f'podman manifest rm localhost/m') - run_command_show_failure(f'podman manifest create localhost/m') + run_command(f'podman manifest rm {LOCALMANIFEST}') + run_command_show_failure(f'podman manifest create {LOCALMANIFEST}') for p in paths_with_tags: run_command_show_failure(f'podman manifest add m {p}') base = f'{manifest_host}/{manifest_repo}' @@ -156,8 +197,55 @@ def main(): f'v{major}.{minor}.{micro}', f'v{major}.{minor}.{micro}-{datetime.today().strftime("%Y%m%d")}', ): - run_command_show_failure( - f'podman manifest push localhost/m {base}:{t}') + if sysargs.dry_run: + print(f'skipping podman manifest push {LOCALMANIFEST} {base}:{t}') + else: + run_command_show_failure( + f'podman manifest push {LOCALMANIFEST} {base}:{t}') + +def promote(sysargs): + manifest_host = os.environ.get('MANIFEST_HOST', 'quay.ceph.io') + manifest_repo = os.environ.get('MANIFEST_REPO', 'ceph/prerelease') + release_manifest_host = os.environ.get('RELEASE_MANIFEST_HOST', 'quay.io') + release_manifest_repo = os.environ.get('RELEASE_MANIFEST_REPO', 'ceph/ceph') + dump_vars( + ('manifest_host', + 'manifest_repo', + 'release_manifest_host', + 'release_manifest_repo', + ), + locals()) + + manifest_path = f'{manifest_host}/{manifest_repo}' + release_path = f'{release_manifest_host}/{release_manifest_repo}' + latest_tag = get_latest_tag(manifest_path) + all_tags = get_all_matching_digest_tags(manifest_path, latest_tag) + + copypaths = list() + for t in all_tags: + from_path = f'{manifest_path}:{t}' + to_path = f'{release_path}:{t}' + copypaths.append((from_path, to_path)) + + if sysargs.dry_run: + for f, t in copypaths: + print(f'dry-run: Would copy: {f} -> {t}') + return(0) + + for f, t in copypaths: + print(f'Will copy: {f} -> {t}') + + for f, t in copypaths: + run_command_show_failure(f'skopeo copy --multi-arch=all docker://{f} docker://{t}') + + +def main(): + args = parse_args() + + if args.promote: + promote(args) + else: + build_prerelease(args) if (__name__ == '__main__'): diff --git a/debian/.gitignore b/debian/.gitignore index 32ca866d753..1d6ef3a34b5 100644 --- a/debian/.gitignore +++ b/debian/.gitignore @@ -38,4 +38,6 @@ /python-cephfs /libcephfs-java /libcephfs-jni +/libcephfs-proxy0-dbg +/libcephfs-proxy0 /tmp diff --git a/debian/control b/debian/control index d31a82bbc75..a7d2dbb4c3a 100644 --- a/debian/control +++ b/debian/control @@ -891,6 +891,7 @@ Conflicts: libceph, Replaces: libceph, libceph1, libcephfs, +Recommends: libcephfs-proxy2 (= ${binary:Version}) Architecture: linux-any Section: libs Depends: ${misc:Depends}, @@ -919,10 +920,61 @@ Description: debugging symbols for libcephfs2 . This package contains debugging symbols for libcephfs2. +Package: libcephfs-proxy2 +Architecture: linux-any +Section: libs +Depends: ${misc:Depends}, + ${shlibs:Depends}, +Recommends: libcephfs-daemon (= ${binary:Version}) +Description: Libcephfs proxy library + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + +Package: libcephfs-proxy2-dbg +Architecture: linux-any +Section: debug +Priority: extra +Depends: libcephfs-proxy2 (= ${binary:Version}), + ${misc:Depends}, +Description: debugging symbols for libcephfs-proxy2 + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + . + This package contains debugging symbols for libcephfs-proxy2. + +Package: libcephfs-daemon +Architecture: linux-any +Depends: libcephfs2 (= ${binary:Version}), + ${misc:Depends}, +Description: Libcephfs proxy daemon + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + +Package: libcephfs-daemon-dbg +Architecture: linux-any +Section: debug +Priority: extra +Depends: libcephfs-daemon (= ${binary:Version}), + ${misc:Depends}, +Description: debugging symbols for libcephfs-daemon + Ceph is a massively scalable, open-source, distributed + storage system that runs on commodity hardware and delivers object, + block and file system storage. This allows applications to share + libcephfs' CephFS mounts to reduce resource consumption. + . + This package contains debugging symbols for libcephfs-proxy2. + Package: libcephfs-dev Architecture: linux-any Section: libdevel Depends: libcephfs2 (= ${binary:Version}), + libcephfs-proxy2 (= ${binary:Version}), ${misc:Depends}, Conflicts: libceph-dev, libceph1-dev, @@ -944,10 +996,10 @@ Package: librgw2 Architecture: linux-any Section: libs Depends: librados2 (= ${binary:Version}), + liblua5.3-0, ${misc:Depends}, ${shlibs:Depends}, - liblua5.3-dev, - luarocks, +Suggests: luarocks, Description: RADOS Gateway client library RADOS is a distributed object store used by the Ceph distributed storage system. This package provides a REST gateway to the diff --git a/debian/libcephfs-daemon.install b/debian/libcephfs-daemon.install new file mode 100644 index 00000000000..454de46d2d7 --- /dev/null +++ b/debian/libcephfs-daemon.install @@ -0,0 +1 @@ +usr/sbin/libcephfsd diff --git a/debian/libcephfs-dev.install b/debian/libcephfs-dev.install index cf22dce62d4..47431dd5526 100644 --- a/debian/libcephfs-dev.install +++ b/debian/libcephfs-dev.install @@ -3,3 +3,5 @@ usr/include/cephfs/libcephfs.h usr/include/cephfs/types.h usr/include/cephfs/metrics/Types.h usr/lib/libcephfs.so +usr/lib/libcephfs_proxy.so +usr/lib/pkgconfig/cephfs.pc diff --git a/debian/libcephfs-proxy2.install b/debian/libcephfs-proxy2.install new file mode 100644 index 00000000000..fc363125bc2 --- /dev/null +++ b/debian/libcephfs-proxy2.install @@ -0,0 +1 @@ +usr/lib/libcephfs_proxy.so.* diff --git a/debian/rules b/debian/rules index 3fbed3f3a2e..6c0ab5e12c6 100755 --- a/debian/rules +++ b/debian/rules @@ -121,6 +121,8 @@ override_dh_strip: dh_strip -plibradosstriper1 --dbg-package=libradosstriper1-dbg dh_strip -plibrbd1 --dbg-package=librbd1-dbg dh_strip -plibcephfs2 --dbg-package=libcephfs2-dbg + dh_strip -plibcephfs-proxy2 --dbg-package=libcephfs-proxy2-dbg + dh_strip -plibcephfs-daemon --dbg-package=libcephfs-daemon-dbg dh_strip -plibrgw2 --dbg-package=librgw2-dbg dh_strip -pradosgw --dbg-package=radosgw-dbg dh_strip -pceph-test --dbg-package=ceph-test-dbg diff --git a/doc/_ext/ceph_commands.py b/doc/_ext/ceph_commands.py index 0697c71f0e1..d96eab08853 100644 --- a/doc/_ext/ceph_commands.py +++ b/doc/_ext/ceph_commands.py @@ -94,7 +94,7 @@ class CmdParam(object): self.goodchars = goodchars self.positional = positional != 'false' - assert who == None + assert who is None def help(self): advanced = [] diff --git a/doc/cephadm/services/mgmt-gateway.rst b/doc/cephadm/services/mgmt-gateway.rst index 2b88d55952e..55c024817ae 100644 --- a/doc/cephadm/services/mgmt-gateway.rst +++ b/doc/cephadm/services/mgmt-gateway.rst @@ -183,7 +183,7 @@ The `mgmt-gateway` service internally makes use of nginx reverse proxy. The foll :: - DEFAULT_NGINX_IMAGE = 'quay.io/ceph/nginx:1.26.1' + mgr/cephadm/container_image_nginx = 'quay.io/ceph/nginx:sclorg-nginx-126' Admins can specify the image to be used by changing the `container_image_nginx` cephadm module option. If there were already running daemon(s) you must redeploy the daemon(s) in order to have them actually use the new image. diff --git a/doc/cephadm/services/mon.rst b/doc/cephadm/services/mon.rst index 389dc450e95..86cc121c9d5 100644 --- a/doc/cephadm/services/mon.rst +++ b/doc/cephadm/services/mon.rst @@ -23,8 +23,8 @@ cluster to a particular subnet. ``cephadm`` designates that subnet as the default subnet of the cluster. New monitor daemons will be assigned by default to that subnet unless cephadm is instructed to do otherwise. -If all of the ceph monitor daemons in your cluster are in the same subnet, -manual administration of the ceph monitor daemons is not necessary. +If all of the Ceph monitor daemons in your cluster are in the same subnet, +manual administration of the Ceph monitor daemons is not necessary. ``cephadm`` will automatically add up to five monitors to the subnet, as needed, as new hosts are added to the cluster. @@ -35,7 +35,7 @@ the placement of daemons. Designating a Particular Subnet for Monitors -------------------------------------------- -To designate a particular IP subnet for use by ceph monitor daemons, use a +To designate a particular IP subnet for use by Ceph monitor daemons, use a command of the following form, including the subnet's address in `CIDR`_ format (e.g., ``10.1.2.0/24``): diff --git a/doc/cephadm/services/monitoring.rst b/doc/cephadm/services/monitoring.rst index 0e2c62105fa..ef987fd7bd3 100644 --- a/doc/cephadm/services/monitoring.rst +++ b/doc/cephadm/services/monitoring.rst @@ -173,24 +173,22 @@ the [ceph-users] mailing list in April of 2024. The thread can be viewed here: ``var/lib/ceph/{FSID}/cephadm.{DIGEST}``, where ``{DIGEST}`` is an alphanumeric string representing the currently-running version of Ceph. -To see the default container images, run a command of the following form: +To see the default container images, run below command: .. prompt:: bash # - grep -E "DEFAULT*IMAGE" /var/lib/ceph/{FSID}/cephadm.{DIGEST} + cephadm list-images -:: - - DEFAULT_PROMETHEUS_IMAGE = 'quay.io/prometheus/prometheus:v2.51.0' - DEFAULT_LOKI_IMAGE = 'docker.io/grafana/loki:2.9.5' - DEFAULT_PROMTAIL_IMAGE = 'docker.io/grafana/promtail:2.9.5' - DEFAULT_NODE_EXPORTER_IMAGE = 'quay.io/prometheus/node-exporter:v1.7.0' - DEFAULT_ALERT_MANAGER_IMAGE = 'quay.io/prometheus/alertmanager:v0.27.0' - DEFAULT_GRAFANA_IMAGE = 'quay.io/ceph/grafana:10.4.0' Default monitoring images are specified in -``/src/cephadm/cephadmlib/constants.py`` and in -``/src/pybind/mgr/cephadm/module.py``. +``/src/python-common/ceph/cephadm/images.py``. + + +.. autoclass:: ceph.cephadm.images.DefaultImages + :members: + :undoc-members: + :exclude-members: desc, image_ref, key + Using custom images ~~~~~~~~~~~~~~~~~~~ diff --git a/doc/cephadm/services/osd.rst b/doc/cephadm/services/osd.rst index 831bd238c79..90ebd86f897 100644 --- a/doc/cephadm/services/osd.rst +++ b/doc/cephadm/services/osd.rst @@ -198,6 +198,18 @@ There are a few ways to create new OSDs: .. warning:: When deploying new OSDs with ``cephadm``, ensure that the ``ceph-osd`` package is not already installed on the target host. If it is installed, conflicts may arise in the management and control of the OSD that may lead to errors or unexpected behavior. +* OSDs created via ``ceph orch daemon add`` are by default not added to the orchestrator's OSD service, they get added to 'osd' service. To attach an OSD to a different, existing OSD service, issue a command of the following form: + + .. prompt:: bash * + + ceph orch osd set-spec-affinity <service_name> <osd_id(s)> + + For example: + + .. prompt:: bash # + + ceph orch osd set-spec-affinity osd.default_drive_group 0 1 + Dry Run ------- diff --git a/doc/cephadm/services/rgw.rst b/doc/cephadm/services/rgw.rst index ed0b149365a..3df8ed2fc56 100644 --- a/doc/cephadm/services/rgw.rst +++ b/doc/cephadm/services/rgw.rst @@ -173,6 +173,32 @@ Then apply this yaml document: Note the value of ``rgw_frontend_ssl_certificate`` is a literal string as indicated by a ``|`` character preserving newline characters. +Disabling multisite sync traffic +-------------------------------- + +There is an RGW config option called ``rgw_run_sync_thread`` that tells the +RGW daemon to not transmit multisite replication data. This is useful if you want +that RGW daemon to be dedicated to I/O rather than multisite sync operations. +The RGW spec file includes a setting ``disable_multisite_sync_traffic`` that when +set to "True" will tell cephadm to set ``rgw_run_sync_thread`` to false for all +RGW daemons deployed for that RGW service. For example + +.. code-block:: yaml + + service_type: rgw + service_id: foo + placement: + label: rgw + spec: + rgw_realm: myrealm + rgw_zone: myzone + rgw_zonegroup: myzg + disable_multisite_sync_traffic: True + +.. note:: This will only stop the RGW daemon(s) from sending replication data. + The daemon can still receive replication data unless it has been removed + from the zonegroup and zone replication endpoints. + Service specification --------------------- diff --git a/doc/cephfs/health-messages.rst b/doc/cephfs/health-messages.rst index 0f171c6ccc9..7aa1f2e44ee 100644 --- a/doc/cephfs/health-messages.rst +++ b/doc/cephfs/health-messages.rst @@ -269,3 +269,11 @@ other daemons, please see :ref:`health-checks`. To evict and permanently block broken clients from connecting to the cluster, set the ``required_client_feature`` bit ``client_mds_auth_caps``. + +``MDS_ESTIMATED_REPLAY_TIME`` +----------------------------- + Message + "HEALTH_WARN Replay: x% complete. Estimated time remaining *x* seconds + + Description + When an MDS journal replay takes more than 30 seconds, this message indicates the estimated time to completion. diff --git a/doc/cephfs/index.rst b/doc/cephfs/index.rst index 57ea336c00b..630d29f1956 100644 --- a/doc/cephfs/index.rst +++ b/doc/cephfs/index.rst @@ -93,6 +93,7 @@ Administration CephFS Top Utility <cephfs-top> Scheduled Snapshots <snap-schedule> CephFS Snapshot Mirroring <cephfs-mirroring> + Purge Queue <purge-queue> .. raw:: html @@ -147,6 +148,7 @@ CephFS Concepts LazyIO <lazyio> Directory fragmentation <dirfrags> Multiple active MDS daemons <multimds> + Snapshots <snapshots> .. raw:: html diff --git a/doc/cephfs/mount-using-kernel-driver.rst b/doc/cephfs/mount-using-kernel-driver.rst index 22ede055d0b..bf229099a19 100644 --- a/doc/cephfs/mount-using-kernel-driver.rst +++ b/doc/cephfs/mount-using-kernel-driver.rst @@ -4,32 +4,32 @@ Mount CephFS using Kernel Driver ================================= -The CephFS kernel driver is part of the Linux kernel. It allows mounting -CephFS as a regular file system with native kernel performance. It is the -client of choice for most use-cases. +The CephFS kernel driver is part of the Linux kernel. It makes possible the +mounting of CephFS as a regular file system with native kernel performance. It +is the client of choice for most use-cases. -.. note:: CephFS mount device string now uses a new (v2) syntax. The mount - helper (and the kernel) is backward compatible with the old syntax. - This means that the old syntax can still be used for mounting with - newer mount helpers and kernel. However, it is recommended to use - the new syntax whenever possible. +.. note:: The CephFS mount device string now uses a new syntax ("v2"). The + mount helper is backward compatible with the old syntax. The kernel is + backward-compatible with the old syntax. This means that the old syntax can + still be used for mounting with newer mount helpers and with the kernel. Prerequisites ============= Complete General Prerequisites ------------------------------ -Go through the prerequisites required by both, kernel as well as FUSE mounts, -in `Mount CephFS: Prerequisites`_ page. +Go through the prerequisites required by both kernel and FUSE mounts, +as described on the `Mount CephFS: Prerequisites`_ page. Is mount helper present? ------------------------ -``mount.ceph`` helper is installed by Ceph packages. The helper passes the +The ``mount.ceph`` helper is installed by Ceph packages. The helper passes the monitor address(es) and CephX user keyrings, saving the Ceph admin the effort of passing these details explicitly while mounting CephFS. If the helper is not present on the client machine, CephFS can still be mounted using the kernel -driver, but only by passing these details explicitly to the ``mount`` command. -To check whether ``mount.ceph`` is present on your system, run the following command: +driver but only by passing these details explicitly to the ``mount`` command. +To check whether ``mount.ceph`` is present on your system, run the following +command: .. prompt:: bash # @@ -38,70 +38,88 @@ To check whether ``mount.ceph`` is present on your system, run the following com Which Kernel Version? --------------------- -Because the kernel client is distributed as part of the linux kernel (not -as part of packaged ceph releases), you will need to consider which kernel +Because the kernel client is distributed as part of the Linux kernel (and not +as part of the packaged Ceph releases), you will need to consider which kernel version to use on your client nodes. Older kernels are known to include buggy -ceph clients, and may not support features that more recent Ceph clusters +Ceph clients and may not support features that more recent Ceph clusters support. -Remember that the "latest" kernel in a stable linux distribution is likely -to be years behind the latest upstream linux kernel where Ceph development +Remember that the "latest" kernel in a stable Linux distribution is likely +to be years behind the latest upstream Linux kernel where Ceph development takes place (including bug fixes). As a rough guide, as of Ceph 10.x (Jewel), you should be using a least a 4.x kernel. If you absolutely have to use an older kernel, you should use the fuse client instead of the kernel client. -This advice does not apply if you are using a linux distribution that -includes CephFS support, as in this case the distributor will be responsible -for backporting fixes to their stable kernel: check with your vendor. +This advice does not apply if you are using a Linux distribution that includes +CephFS support. In that case, the distributor is responsible for backporting +fixes to their stable kernel. Check with your vendor. Synopsis ======== -In general, the command to mount CephFS via kernel driver looks like this:: +This is the general form of the command for mounting CephFS via the kernel driver: - mount -t ceph {device-string}={path-to-mounted} {mount-point} -o {key-value-args} {other-args} +.. prompt:: bash # + + mount -t ceph {device-string}={path-to-mounted} {mount-point} -o {key-value-args} {other-args} Mounting CephFS =============== -On Ceph clusters, CephX is enabled by default. Use ``mount`` command to -mount CephFS with the kernel driver:: +CephX authentication is enabled by default in Ceph clusters. Use the ``mount`` +command to use the kernel driver to mount CephFS: + +.. prompt:: bash # - mkdir /mnt/mycephfs - mount -t ceph <name>@<fsid>.<fs_name>=/ /mnt/mycephfs + mkdir /mnt/mycephfs + mount -t ceph <name>@<fsid>.<fs_name>=/ /mnt/mycephfs -``name`` is the username of the CephX user we are using to mount CephFS. -``fsid`` is the FSID of the ceph cluster which can be found using -``ceph fsid`` command. ``fs_name`` is the file system to mount. The kernel -driver requires MON's socket and the secret key for the CephX user, e.g.:: +#. ``name`` is the username of the CephX user we are using to mount CephFS. +#. ``fsid`` is the FSID of the Ceph cluster, which can be found using the + ``ceph fsid`` command. ``fs_name`` is the file system to mount. The kernel + driver requires a ceph Monitor's address and the secret key of the CephX + user. For example: - mount -t ceph cephuser@b3acfc0d-575f-41d3-9c91-0e7ed3dbb3fa.cephfs=/ -o mon_addr=192.168.0.1:6789,secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== + .. prompt:: bash # -When using the mount helper, monitor hosts and FSID are optional. ``mount.ceph`` -helper figures out these details automatically by finding and reading ceph conf -file, .e.g:: + mount -t ceph cephuser@b3acfc0d-575f-41d3-9c91-0e7ed3dbb3fa.cephfs=/ -o mon_addr=192.168.0.1:6789,secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== - mount -t ceph cephuser@.cephfs=/ -o secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== +When using the mount helper, monitor hosts and FSID are optional. The +``mount.ceph`` helper discovers these details by finding and reading the ceph +conf file. For example: -.. note:: Note that the dot (``.``) still needs to be a part of the device string. +.. prompt:: bash # -A potential problem with the above command is that the secret key is left in your -shell's command history. To prevent that you can copy the secret key inside a file -and pass the file by using the option ``secretfile`` instead of ``secret``:: + mount -t ceph cephuser@.cephfs=/ -o secret=AQATSKdNGBnwLhAAnNDKnH65FmVKpXZJVasUeQ== - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o secretfile=/etc/ceph/cephuser.secret +.. note:: Note that the dot (``.`` in the string ``cephuser@.cephfs``) must be + a part of the device string. -Ensure the permissions on the secret key file are appropriate (preferably, ``600``). +A weakness of this method is that it will leave the secret key in your shell's +command history. To avoid this, copy the secret key inside a file and pass the +file by using the option ``secretfile`` instead of ``secret``. For example: -Multiple monitor hosts can be passed by separating each address with a ``/``:: +.. prompt:: bash # - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o mon_addr=192.168.0.1:6789/192.168.0.2:6789,secretfile=/etc/ceph/cephuser.secret + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o secretfile=/etc/ceph/cephuser.secret -In case CephX is disabled, you can omit any credential related options:: +Ensure that the permissions on the secret key file are appropriate (preferably, +``600``). - mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs +Multiple monitor hosts can be passed by separating addresses with a ``/``: -.. note:: The ceph user name still needs to be passed as part of the device string. +.. prompt:: bash # + + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs -o + mon_addr=192.168.0.1:6789/192.168.0.2:6789,secretfile=/etc/ceph/cephuser.secret + +If CephX is disabled, omit any credential-related options. For example: + +.. prompt:: bash # + + mount -t ceph cephuser@.cephfs=/ /mnt/mycephfs + +.. note:: The Ceph user name must be passed as part of the device string. To mount a subtree of the CephFS root, append the path to the device string:: @@ -111,29 +129,40 @@ Backward Compatibility ====================== The old syntax is supported for backward compatibility. -To mount CephFS with the kernel driver:: +To mount CephFS with the kernel driver, run the following commands: + +.. prompt:: bash # + + mkdir /mnt/mycephfs + mount -t ceph :/ /mnt/mycephfs -o name=admin - mkdir /mnt/mycephfs - mount -t ceph :/ /mnt/mycephfs -o name=admin +The key-value argument right after the option ``-o`` is the CephX credential. +``name`` is the username of the CephX user that is mounting CephFS. -The key-value argument right after option ``-o`` is CephX credential; -``name`` is the username of the CephX user we are using to mount CephFS. +To mount a non-default FS (in this example, ``cephfs2``), run commands of the following form. These commands are to be used in cases in which the cluster +has multiple file systems: -To mount a non-default FS ``cephfs2``, in case the cluster has multiple FSs:: +.. prompt:: bash # + + mount -t ceph :/ /mnt/mycephfs -o name=admin,fs=cephfs2 - mount -t ceph :/ /mnt/mycephfs -o name=admin,fs=cephfs2 +or - or +.. prompt:: bash # - mount -t ceph :/ /mnt/mycephfs -o name=admin,mds_namespace=cephfs2 + mount -t ceph :/ /mnt/mycephfs -o name=admin,mds_namespace=cephfs2 -.. note:: The option ``mds_namespace`` is deprecated. Use ``fs=`` instead when using the old syntax for mounting. +.. note:: The option ``mds_namespace`` is deprecated. Use ``fs=`` instead when + using the old syntax for mounting. Unmounting CephFS ================= -To unmount the Ceph file system, use the ``umount`` command as usual:: +To unmount the Ceph file system, use the ``umount`` command, as in this +example: + +.. prompt:: bash # - umount /mnt/mycephfs + umount /mnt/mycephfs .. tip:: Ensure that you are not within the file system directories before executing this command. @@ -150,11 +179,12 @@ For example:: cephuser@.cephfs=/ /mnt/ceph ceph mon_addr=192.168.0.1:6789,noatime,_netdev 0 0 -If the ``secret`` or ``secretfile`` options are not specified then the mount helper -will attempt to find a secret for the given ``name`` in one of the configured keyrings. +If the ``secret`` or ``secretfile`` options are not specified, the mount +helper will attempt to find a secret for the given ``name`` in one of the +configured keyrings. -See `User Management`_ for details on CephX user management and mount.ceph_ -manual for more options it can take. For troubleshooting, see +See `User Management`_ for details on CephX user management and the mount.ceph_ +manual for a list of the options it recognizes. For troubleshooting, see :ref:`kernel_mount_debugging`. .. _fstab: ../fstab/#kernel-driver diff --git a/doc/cephfs/purge-queue.rst b/doc/cephfs/purge-queue.rst new file mode 100644 index 00000000000..d7a68e7fa55 --- /dev/null +++ b/doc/cephfs/purge-queue.rst @@ -0,0 +1,106 @@ +============ +Purge Queue +============ + +MDS maintains a data structure known as **Purge Queue** which is responsible +for managing and executing the parallel deletion of files. +There is a purge queue for every MDS rank. Purge queues consist of purge items +which contain nominal information from the inodes such as size and the layout +(i.e. all other un-needed metadata information is discarded making it +independent of all metadata structures). + +Deletion process +================ + +When a client requests deletion of a directory (say ``rm -rf``): + +- MDS queues the files and subdirectories (purge items) from pq (purge queue) + journal in the purge queue. +- Processes the deletion of inodes in background in small and manageable + chunks. +- MDS instructs underlying OSDs to clean up the associated objects in data + pool. +- Updates the journal. + +.. note:: If the users delete the files more quickly than the + purge queue can process then the data pool usage might increase + substantially over time. In extreme scenarios, the purge queue + backlog can become so huge that it can slacken the capacity reclaim + and the linux ``du`` command for CephFS might report inconsistent + data compared to the CephFS Data pool. + +There are a few tunable configs that MDS uses internally to throttle purge +queue processing: + +.. confval:: filer_max_purge_ops +.. confval:: mds_max_purge_files +.. confval:: mds_max_purge_ops +.. confval:: mds_max_purge_ops_per_pg + +Generally, the defaults are adequate for most clusters. However, in +case of pretty huge clusters, if the need arises like ``pq_item_in_journal`` +(counter of things pending deletion) reaching gigantic figure then the configs +can be tuned to 4-5 times of the default value as a starting point and +further increments are subject to more requirements. + +Start from the most trivial config ``filer_max_purge_ops``, which should help +reclaim the space more quickly:: + + $ ceph config set mds filer_max_purge_ops 40 + +Incrementing ``filer_max_purge_ops`` should just work for most +clusters but if it doesn't then move ahead with tuning other configs:: + + $ ceph config set mds mds_max_purge_files 256 + $ ceph config set mds mds_max_purge_ops 32768 + $ ceph config set mds mds_max_purge_ops_per_pg 2 + +.. note:: Setting these values won't immediately break anything except + inasmuch as they control how many delete ops we issue to the + underlying RADOS cluster, but might eat up some cluster performance + if the values set are staggeringly high. + +.. note:: The purge queue is not an auto-tuning system in terms of its work + limits as compared to what is going on. So it is advised to make + a conscious decision while tuning the configs based on the cluster + size and workload. + +Examining purge queue perf counters +=================================== + +When analysing MDS perf dumps, the purge queue statistics look like:: + + "purge_queue": { + "pq_executing_ops": 56655, + "pq_executing_ops_high_water": 65350, + "pq_executing": 1, + "pq_executing_high_water": 3, + "pq_executed": 25, + "pq_item_in_journal": 6567004 + } + +Let us understand what each of these means: + +.. list-table:: + :widths: 50 50 + :header-rows: 1 + + * - Name + - Description + * - pq_executing_ops + - Purge queue operations in flight + * - pq_executing_ops_high_water + - Maximum number of executing purge operations recorded + * - pq_executing + - Purge queue files being deleted + * - pq_executing_high_water + - Maximum number of executing file purges + * - pq_executed + - Purge queue files deleted + * - pq_item_in_journal + - Purge items (files) left in journal + +.. note:: ``pq_executing`` and ``pq_executing_ops`` might look similar but + there is a small nuance. ``pq_executing`` tracks number of files + in the purge queue while ``pq_executing_ops`` is the count of RADOS + objects from all the files in purge queue. diff --git a/doc/cephfs/snap-schedule.rst b/doc/cephfs/snap-schedule.rst index a94d938040f..48e79047864 100644 --- a/doc/cephfs/snap-schedule.rst +++ b/doc/cephfs/snap-schedule.rst @@ -197,6 +197,15 @@ this happens, the next snapshot will be schedule as if the previous one was not delayed, i.e. one or more delayed snapshots will not cause drift in the overall schedule. +If a volume is deleted while snapshot schedules are active on the volume, then +there might be cases when Python Tracebacks are seen in the log file or on the +command-line when commands are executed on such volumes. Although measures have +been taken to take note of the fs_map changes and delete active timers and +close database connections to avoid Python Tracebacks, it is not possible to +completely mute the tracebacks due to the inherent nature of problem. In the +event that such tracebacks are seen, the only solution to get the system to a +stable state is the disable and re-enable the snap_schedule Manager Module. + In order to somewhat limit the overall number of snapshots in a file system, the module will only keep a maximum of 50 snapshots per directory. If the retention policy results in more then 50 retained snapshots, the retention list will be diff --git a/doc/cephfs/snapshots.rst b/doc/cephfs/snapshots.rst new file mode 100644 index 00000000000..a60be96ed53 --- /dev/null +++ b/doc/cephfs/snapshots.rst @@ -0,0 +1,85 @@ +================ +CephFS Snapshots +================ + +CephFS snapshots create an immutable view of the file system at the point +in time they are taken. CephFS support snapshots which is managed in a +special hidden subdirectory named ``.snap`` .Snapshots are created using +``mkdir`` inside this directory. + +Snapshots can be exposed with a different name by changing the following client configurations. + +- ``snapdirname`` which is a mount option for kernel clients +- ``client_snapdir`` which is a mount option for ceph-fuse. + +Snapshot Creation +================== + +CephFS snapshot feature is enabled by default on new file systems. To enable +it on existing file systems, use the command below. + +.. code-block:: bash + + $ ceph fs set <fs_name> allow_new_snaps true + +When snapshots are enabled, all directories in CephFS will have a special ``.snap`` +directory. (You may configure a different name with the client snapdir setting if +you wish.) +To create a CephFS snapshot, create a subdirectory under ``.snap`` with a name of +your choice. +For example, to create a snapshot on directory ``/file1/``, invoke ``mkdir /file1/.snap/snapshot-name`` + +.. code-block:: bash + + $ touch file1 + $ cd .snap + $ mkdir my_snapshot + +Using snapshot to recover data +=============================== + +Snapshots can also be used to recover some deleted files. + +- ``create a file1 and create snapshot snap1`` + +.. code-block:: bash + + $ touch /mnt/cephfs/file1 + $ cd .snap + $ mkdir snap1 + +- ``create a file2 and create snapshot snap2`` + +.. code-block:: bash + + $ touch /mnt/cephfs/file2 + $ cd .snap + $ mkdir snap2 + +- ``delete file1 and create a new snapshot snap3`` + +.. code-block:: bash + + $ rm /mnt/cephfs/file1 + $ cd .snap + $ mkdir snap3 + +- ``recover file1 using snapshot snap2 using cp command`` + +.. code-block:: bash + + $ cd .snap + $ cd snap2 + $ cp file1 /mnt/cephfs/ + +Snapshot Deletion +================== + +Snapshots are deleted by invoking ``rmdir`` on the ``.snap`` directory they are +rooted in. (Attempts to delete a directory which roots the snapshots will fail; +you must delete the snapshots first.) + +.. code-block:: bash + + $ cd .snap + $ rmdir my_snapshot diff --git a/doc/conf.py b/doc/conf.py index 4fdc9a53b75..5293ff1b212 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -76,7 +76,7 @@ html_show_sphinx = False html_static_path = ["_static"] html_sidebars = { '**': ['smarttoc.html', 'searchbox.html'] - } +} html_css_files = ['css/custom.css'] @@ -133,13 +133,23 @@ extensions = [ 'sphinxcontrib.mermaid', 'sphinxcontrib.openapi', 'sphinxcontrib.seqdiag', - ] +] ditaa = shutil.which("ditaa") if ditaa is not None: # in case we don't have binfmt_misc enabled or jar is not registered - ditaa_args = ['-jar', ditaa] - ditaa = 'java' + _jar_paths = [ + '/usr/share/ditaa/lib/ditaa.jar', # Gentoo + '/usr/share/ditaa/ditaa.jar', # deb + '/usr/share/java/ditaa.jar', # rpm + ] + _jar_paths = [p for p in _jar_paths if os.path.exists(p)] + if _jar_paths: + ditaa = 'java' + ditaa_args = ['-jar', _jar_paths[0]] + else: + # keep ditaa from shutil.which + ditaa_args = [] extensions += ['sphinxcontrib.ditaa'] else: extensions += ['plantweb.directive'] diff --git a/doc/dev/crimson/pipeline.rst b/doc/dev/crimson/pipeline.rst index e9115c6d7c3..6e47b79d80b 100644 --- a/doc/dev/crimson/pipeline.rst +++ b/doc/dev/crimson/pipeline.rst @@ -2,96 +2,34 @@ The ``ClientRequest`` pipeline ============================== -In crimson, exactly like in the classical OSD, a client request has data and -ordering dependencies which must be satisfied before processing (actually -a particular phase of) can begin. As one of the goals behind crimson is to -preserve the compatibility with the existing OSD incarnation, the same semantic -must be assured. An obvious example of such data dependency is the fact that -an OSD needs to have a version of OSDMap that matches the one used by the client -(``Message::get_min_epoch()``). - -If a dependency is not satisfied, the processing stops. It is crucial to note -the same must happen to all other requests that are sequenced-after (due to -their ordering requirements). - -There are a few cases when the blocking of a client request can happen. - - - ``ClientRequest::ConnectionPipeline::await_map`` - wait for particular OSDMap version is available at the OSD level - ``ClientRequest::ConnectionPipeline::get_pg`` - wait a particular PG becomes available on OSD - ``ClientRequest::PGPipeline::await_map`` - wait on a PG being advanced to particular epoch - ``ClientRequest::PGPipeline::wait_for_active`` - wait for a PG to become *active* (i.e. have ``is_active()`` asserted) - ``ClientRequest::PGPipeline::recover_missing`` - wait on an object to be recovered (i.e. leaving the ``missing`` set) - ``ClientRequest::PGPipeline::get_obc`` - wait on an object to be available for locking. The ``obc`` will be locked - before this operation is allowed to continue - ``ClientRequest::PGPipeline::process`` - wait if any other ``MOSDOp`` message is handled against this PG - -At any moment, a ``ClientRequest`` being served should be in one and only one -of the phases described above. Similarly, an object denoting particular phase -can host not more than a single ``ClientRequest`` the same time. At low-level -this is achieved with a combination of a barrier and an exclusive lock. -They implement the semantic of a semaphore with a single slot for these exclusive -phases. - -As the execution advances, request enters next phase and leaves the current one -freeing it for another ``ClientRequest`` instance. All these phases form a pipeline -which assures the order is preserved. - -These pipeline phases are divided into two ordering domains: ``ConnectionPipeline`` -and ``PGPipeline``. The former ensures order across a client connection while -the latter does that across a PG. That is, requests originating from the same -connection are executed in the same order as they were sent by the client. -The same applies to the PG domain: when requests from multiple connections reach -a PG, they are executed in the same order as they entered a first blocking phase -of the ``PGPipeline``. - -Comparison with the classical OSD ----------------------------------- -As the audience of this document are Ceph Developers, it seems reasonable to -match the phases of crimson's ``ClientRequest`` pipeline with the blocking -stages in the classical OSD. The names in the right column are names of -containers (lists and maps) used to implement these stages. They are also -already documented in the ``PG.h`` header. - -+----------------------------------------+--------------------------------------+ -| crimson | ceph-osd waiting list | -+========================================+======================================+ -|``ConnectionPipeline::await_map`` | ``OSDShardPGSlot::waiting`` and | -|``ConnectionPipeline::get_pg`` | ``OSDShardPGSlot::waiting_peering`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::await_map`` | ``PG::waiting_for_map`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::wait_for_active`` | ``PG::waiting_for_peered`` | -| +--------------------------------------+ -| | ``PG::waiting_for_flush`` | -| +--------------------------------------+ -| | ``PG::waiting_for_active`` | -+----------------------------------------+--------------------------------------+ -|To be done (``PG_STATE_LAGGY``) | ``PG::waiting_for_readable`` | -+----------------------------------------+--------------------------------------+ -|To be done | ``PG::waiting_for_scrub`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::recover_missing`` | ``PG::waiting_for_unreadable_object``| -| +--------------------------------------+ -| | ``PG::waiting_for_degraded_object`` | -+----------------------------------------+--------------------------------------+ -|To be done (proxying) | ``PG::waiting_for_blocked_object`` | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::get_obc`` | *obc rwlocks* | -+----------------------------------------+--------------------------------------+ -|``PGPipeline::process`` | ``PG::lock`` (roughly) | -+----------------------------------------+--------------------------------------+ - - -As the last word it might be worth to emphasize that the ordering implementations -in both classical OSD and in crimson are stricter than a theoretical minimum one -required by the RADOS protocol. For instance, we could parallelize read operations -targeting the same object at the price of extra complexity but we don't -- the -simplicity has won. +RADOS requires writes on each object to be ordered. If a client +submits a sequence of concurrent writes (doesn't want for the prior to +complete before submitting the next), that client may rely on the +writes being completed in the order in which they are submitted. + +As a result, the client->osd communication and queueing mechanisms on +both sides must take care to ensure that writes on a (connection, +object) pair remain ordered for the entire process. + +crimson-osd enforces this ordering via Pipelines and Stages +(crimson/osd/osd_operation.h). Upon arrival at the OSD, messages +enter the ConnectionPipeline::AwaitActive stage and proceed +through a sequence of pipeline stages: + +* ConnectionPipeline: per-connection stages representing the message handling + path prior to being handed off to the target PG +* PerShardPipeline: intermediate Pipeline representing the hand off from the + receiving shard to the shard with the target PG. +* CommonPGPipeline: represents processing on the target PG prior to obtaining + the ObjectContext for the target of the operation. +* CommonOBCPipeline: represents the actual processing of the IO on the target + object + +Because CommonOBCPipeline is per-object rather than per-connection or +per-pg, multiple requests on different objects may be in the same +CommonOBCPipeline stage concurrently. This allows us to serve +multiple reads in the same PG concurrently. We can also process +writes on multiple objects concurrently up to the point at which the +write is actually submitted. + +See crimson/osd/osd_operations/client_request.(h|cc) for details. diff --git a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst index 34dfd521eaa..6964012ef31 100644 --- a/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst +++ b/doc/dev/developer_guide/testing_integration_tests/tests-integration-testing-teuthology-workflow.rst @@ -6,7 +6,8 @@ Integration Tests using Teuthology Workflow Infrastructure -------------- -Components: +Components +********** 1. `ceph-ci`_: Clone of the main Ceph repository, used for triggering Jenkins Ceph builds for development. @@ -44,7 +45,27 @@ Components: Each Teuthology test *run* contains multiple test *jobs*. Each job runs in an environment isolated from other jobs, on a different collection of test nodes. -To test a change in Ceph, follow these steps: +Workflow Overview +***************** + +.. image:: workflow.png + + +To test a change in Ceph, start by pushing a branch with your changes to the +`ceph-ci`_ repository. This will automatically trigger the Jenkins process +to build Ceph binaries - the status of the build can be observed on `Shaman`_. +These built packages will be uploaded on `Chacra`_. + +To schedule a Teuthology integration test against this new build, you will +need access to the Sepia lab. Once you have access, log into the Teuthology +machine and complete the one-time initial Teuthology setup required to run +Teuthology commands. After the setup, use the ``teuthology-suite`` command to schedule +a Teuthology run. In this command, use the ``-c <ceph-ci branch name>`` option to +specify your build. The results of your test can be observed on `Pulpito`_. +Log into a `developer playground machine`_ to review the Teuthology run's archive logs. + + +The rest of the document will explain these steps in detail: 1. Getting binaries - Build Ceph. 2. Scheduling Test Run: @@ -98,6 +119,31 @@ Ceph binaries must be built for your branch before you can use teuthology to run .. _the Chacra site: https://shaman.ceph.com/api/search/?status=ready&project=ceph +Pushing to the ceph-ci repository +********************************* + +Follow these steps to push to the ceph-ci repository. After pushing, a new build will +automatically be scheduled. + +1. Add the ceph-ci repository as a remote to your local clone of the Ceph repository: + + .. prompt:: bash $ + + git remote add ceph-ci git@github.com:ceph/ceph-ci.git + + $ git remote -v + origin git@github.com:ceph/ceph.git (fetch) + origin git@github.com:ceph/ceph.git (push) + ceph-ci git@github.com:ceph/ceph-ci.git (fetch) + ceph-ci git@github.com:ceph/ceph-ci.git (push) + +2. Push your branch upstream by running a command of the following form: + + .. prompt:: bash $ + + $ git push ceph-ci wip-yourname-feature-x + + Naming the ceph-ci branch ************************* Prepend your branch with your name before you push it to ceph-ci. For example, @@ -110,15 +156,14 @@ the name of that stable branch in your ceph-ci branch name. For example, the ``feature-x`` PR branch should be named ``wip-feature-x-nautilus``. *This is not just a convention. This ensures that your branch is built in the correct environment.* -You can choose to only trigger a CentOS 9.Stream build (excluding other distro like ubuntu) -by adding "centos9-only" at the end of the ceph-ci branch name. For example, -``wip-$yourname-feature-centos9-only``. This helps to get quicker builds and save resources -when you don't require binaries for other distros. - Delete the branch from ceph-ci when you no longer need it. If you are logged in to GitHub, all your branches on ceph-ci can be found here: https://github.com/ceph/ceph-ci/branches. +.. note:: You can choose to only trigger a CentOS 9.Stream build (excluding other + distro like ubuntu) by adding "centos9-only" at the end of the ceph-ci branch name. + For example, ``wip-$yourname-feature-centos9-only``. This helps to get quicker builds + and save resources when you don't require binaries for other distros. Scheduling Test Run ------------------- diff --git a/doc/dev/developer_guide/testing_integration_tests/workflow.png b/doc/dev/developer_guide/testing_integration_tests/workflow.png Binary files differnew file mode 100644 index 00000000000..610baf683bc --- /dev/null +++ b/doc/dev/developer_guide/testing_integration_tests/workflow.png diff --git a/doc/dev/development-workflow.rst b/doc/dev/development-workflow.rst index dfcab929daa..530944bf580 100644 --- a/doc/dev/development-workflow.rst +++ b/doc/dev/development-workflow.rst @@ -108,12 +108,12 @@ status of an open issue can be: * ``Pending Backport``: the fix needs to be backported to the stable releases listed in the backport field -For each ``Pending Backport`` issue, there exists at least one issue -in the ``Backport`` tracker to record the work done to cherry pick the -necessary commits from the master branch to the target stable branch. -See `the backporter manual -<http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO>`_ for more -information. +For each ``Pending Backport`` issue, there exists at least one issue in the +``Backport`` tracker to record the work done to cherry pick the necessary +commits from the master branch to the target stable branch. See `the backporter +manual +<https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst>`_ for +more information. Running and interpreting teuthology integration tests ===================================================== @@ -243,6 +243,10 @@ differences: * All commits are cherry-picked with ``git cherry-pick -x`` to reference the original commit +.. note: If a backport is appropriate, the submitter is responsible for + determining appropriate target stable branches to which backports must be + made. + See `the backporter manual -<http://tracker.ceph.com/projects/ceph-releases/wiki/HOWTO>`_ for more -information. +<https://github.com/ceph/ceph/blob/main/SubmittingPatches-backports.rst>`_ for +more information. diff --git a/doc/dev/libcephfs_proxy.rst b/doc/dev/libcephfs_proxy.rst new file mode 100644 index 00000000000..baa96f765c9 --- /dev/null +++ b/doc/dev/libcephfs_proxy.rst @@ -0,0 +1,289 @@ +Design of the libcephfs proxy +============================= + +Description of the problem +-------------------------- + +When an application connects to a Ceph volume through the *libcephfs.so* +library, a cache is created locally inside the process. The *libcephfs.so* +implementation already deals with memory usage of the cache and adjusts it so +that it doesn't consume all the available memory. However, if multiple +processes connect to CephFS through different instances of the library, each +one of them will keep a private cache. In this case memory management is not +effective because, even configuring memory limits, the number of libcephfs +instances that can be created is unbounded and they can't work in a coordinated +way to correctly control ressource usage. Due to this, it's relatively easy to +consume all memory when all processes are using data cache intensively. This +causes the OOM killer to terminate those processes. + +Proposed solution +----------------- + +High level approach +^^^^^^^^^^^^^^^^^^^ + +The main idea is to create a *libcephfs_proxy.so* library that will provide the +same API as the original *libcephfs.so*, but won't cache any data. This library +can be used by any application currently using *libcephfs.so* in a transparent +way (i.e. no code modification required) just by linking against +*libcephfs_proxy.so* instead of *libcephfs.so*, or even using *LD_PRELOAD*. + +A new *libcephfsd* daemon will also be created. This daemon will link against +the real *libcephfs.so* library and will listen for incoming connections on a +UNIX socket. + +When an application starts and initiates CephFS requests through the +*libcephfs_proxy.so* library, it will connect to the *libcephfsd* daemon +through the UNIX socket and it will forward all CephFS requests to it. The +daemon will use the real *libcephfs.so* to execute those requests and the +answers will be returned back to the application, potentially caching data in +the *libcephfsd* process itself. All this will happen transparently, without +any knowledge from the application. + +The daemon will share low level *libcephfs.so* mounts between different +applications to avoid creating an instance for each application, which would +have the same effect on memory as linking each application directly to the +*libcephfs.so* library. This will be done only if the configuration defined by +the applications is identical. Otherwise new independent instances will still +be created. + +Some *libcephfs.so* functions will need to be implemented in an special way +inside the *libcephfsd* daemon to hide the differences caused by sharing the +same mount instance with more than one client (for example chdir/getcwd cannot +rely directly on the ``ceph_chdir()``/``ceph_getcwd()`` of *libcephfs.so*). + +Initially, only the subset of the low-level interface functions of +*libcephfs.so* that are used by the Samba's VFS CephFS module will be provided. + +Design of common components +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Network protocol +"""""""""""""""" + +Since the connection through the UNIX socket is to another process that runs on +the same machine and the data we need to pass is quite simple, we'll avoid all +the overhead of generic XDR encoding/decoding and RPC transmissions by using a +very simple serialization implemented in the code itself. For the future we may +consider using cap'n proto (https://capnproto.org), which claims to have zero +overhead for encoding and decoding, and would provide an easy way to support +backward compatibility if the network protocol needs to be modified in the +future. + +Design of the *libcephfs_proxy.so* library +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +This library will basically connect to the UNIX socket where the *libcephfsd* +daemon is listening, wait for requests coming from the application, serialize +all function arguments and send them to the daemon. Once the daemon responds it +will deserialize the answer and return the result to the application. + +Local caching +""""""""""""" + +While the main purpose of this library is to avoid independent caches on each +process, some preliminary testing has shown a big performance drop for +workloads based on metadata operations and/or small files when all requests go +through the proxy daemon. To minimize this, metadata caching should be +implemented. Metadata cache is much smaller than data cache and will provide a +good trade-off between memory usage and performance. + +To implement caching in a safe way, it's required to correctly invalidate data +before it becomes stale. Currently libcephfs.so provides invalidation +notifications that can be used to implement this, but its semantics are not +fully understood yet, so the cache in the libcephfs_proxy.so library will be +designed and implemented in a future version. + + +Design of the *libcephfsd* daemon +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The daemon will be a regular process that will centralize libcephfs requests +coming from other processes on the same machine. + +Process maintenance +""""""""""""""""""" + +Since the process will work as a standalone daemon, a simple systemd unit file +will be provided to manage it as a regular system service. Most probably this +will be integrated inside cephadm in the future. + +In case the *libcephfsd* daemon crashes, we'll rely on systemd to restart it. + + +Special functions +^^^^^^^^^^^^^^^^^ + +Some functions will need to be handled in a special way inside the *libcephfsd* +daemon to provide correct functionality since forwarding them directly to +*libcephfs.so* could return incorrect results due to the sharing of low-level +mounts. + +**Sharing of underlying struct ceph_mount_info** + +The main purpose of the proxy is to avoid creating a new mount for each process +when they are accessing the same data. To be able to provide this we need to +"virtualize" the mount points and let the application believe it's using its +own mount when, in fact, it could be using a shared mount. + +The daemon will track the Ceph account used to connect to the volume, the +configuration file and any specific configuration changes done before mounting +the volume. Only if all settings are exactly the same as another already +mounted instance, then the mount will be shared. The daemon won't understand +CephFS settings nor any potential dependencies between settings. For this +reason, a very strict comparison will be done: the configuration file needs to +be identical and any other changes made afterwards need to be set to the exact +same value and in the same order so that two configurations can be considered +identical. + +The check to determine whether two configurations are identical or not will be +done just before mounting the volume (i.e. ``ceph_mount()``). This means that +during the configuration phase, we may have many simultaneous mounts allocated +but not yet mounted. However only one of them will become a real mount. The +others will remain unmounted and will be eventually destroyed once users +unmount and release them. + +The following functions will be affected: + +* **ceph_create** + + This one will allocate a new ceph_mount_info structure, and the provided id + will be recorded for future comparison of potentially matching mounts. + +* **ceph_release** + + This one will release an unmounted ceph_mount_info structure. Unmounted + structures won't be shared with anyone else. + +* **ceph_conf_read_file** + + This one will read the configuration file, compute a checksum and make a + copy. The copy will make sure that there are no changes in the configuration + file since the time the checksum was computed, and the checksum will be + recorded for future comparison of potentially matching mounts. + +* **ceph_conf_get** + + This one will obtain the requested setting, recording it for future + comparison of potentially matching mounts. + + Even though this may seem unnecessary, since the daemon is considering the + configuration as a black box, it could be possible to have some dynamic + setting that could return different values depending on external factors, so + the daemon also requires that any requested setting returns the same value to + consider two configurations identical. + +* **ceph_conf_set** + + This one will record the modified value for future comparison of potentially + matching mounts. + + In normal circumstances, some settings may be set even after having mounted + the volume. The proxy won't allow that to avoid potential interferences with + other clients sharing the same mount. + +* **ceph_init** + + This one will be a no-op. Calling this function triggers the allocation of + several resources and starts some threads. This is just a waste of resources + if this *ceph_mount_info* structure is not finally mounted because it matches + with an already existing mount. + + Only if at the time of mount (i.e. ``ceph_mount()``) there's no match with + already existing mounts, then the mount will be initialized and mounted at + the same time. + +* **ceph_select_filesystem** + + This one will record the selected file system for future comparison of + potentially matching mounts. + +* **ceph_mount** + + This one will try to find an active mount that matches with all the + configurations defined for this *ceph_mount_info* structure. If none is + found, it will be mounted. Otherwise, the already existing mount will be + shared with this client. + + The unmounted *ceph_mount_info* structures will be kept around associated + with the mounted one. + + All "real" mounts will be made against the absolute root of the volume + (i.e. "/") to make sure they can be shared with other clients later, + regardless of whether they use the same mount point or not. This means that + just after mounting, the daemon will need to resolve and store the root inode + of the "virtual" mount point. + + The CWD (Current Working Directory) will also be initialized to the same + inode. + +* **ceph_unmount** + + This one will detach the client from the mounted *ceph_mount_info* structure + and reattach it to one of the associated unmounted structures. If this was + the last user of the mount, it's finally unmounted instead. + + After calling this function, the client continues using a private + *ceph_mount_info* structure that is used exclusively by itself, so other + configuration changes and operations can be done safely. + +**Confine accesses to the intended mount point** + +Since the effective mount point may not match the real mount point, some +functions could be able to return inodes outside of the effective mount point +if not handled with care. To avoid it and provide the result that the user +application expects, we will need to simulate some of them inside the +*libcephfsd* daemon. + +There are three special cases to consider: + +1. Handling of paths starting with "/" +2. Handling of paths containing ".." (i.e. parent directory) +3. Handling of paths containing symbolic links + +When these special paths are found, they need to be handled in a special way to +make sure that the returned inodes are what the client expects. + +The following functions will be affected: + +* **ceph_ll_lookup** + + Lookup accepts ".." as the name to resolve. If the parent directory is the + root of the "virtual" mount point (which may not be the same as the real + mount point), we'll need to return the inode corresponding to the "virtual" + mount point stored at the time of mount, instead of the real parent. + +* **ceph_ll_lookup_root** + + This one needs to return the root inode stored at the time of mount. + +* **ceph_ll_walk** + + This one will be completely reimplemented inside the daemon to be able to + correctly parse each path component and symbolic link, and handle "/" and + ".." in the correct way. + +* **ceph_chdir** + + This one will resolve the passed path and store it along the corresponding + inode inside the current "virtual" mount. The real ``ceph_chdir()`` won't be + called. + +* **ceph_getcwd** + + This one will just return the path stored in the "virtual" mount from + previous ``ceph_chdir()`` calls. + +**Handle AT_FDCWD** + +Any function that receives a file descriptor could also receive the special +*AT_FDCWD* value. These functions need to check for that value and use the +"virtual" CWD instead. + +Testing +------- + +The proxy should be transparent to any application already using +*libcephfs.so*. This also applies to testing scripts and applications. So any +existing test against the regular *libcephfs.so* library can also be used to +test the proxy. diff --git a/doc/dev/release-process.rst b/doc/dev/release-process.rst index 67f867fecba..a4939cc8e25 100644 --- a/doc/dev/release-process.rst +++ b/doc/dev/release-process.rst @@ -136,7 +136,9 @@ See `the Ceph Tracker wiki page that explains how to write the release notes <ht #. Obtain the sha1 of the version commit from the `build job <https://jenkins.ceph.com/view/all/job/ceph>`_ or the ``sha1`` file created by the `ceph-setup <https://jenkins.ceph.com/job/ceph-setup/>`_ job. -#. Download the packages from chacra.ceph.com to the signing virtual machine. These packages get downloaded to ``/opt/repos`` where the `Sepia Lab Long Running (Ceph) Cluster <https://wiki.sepia.ceph.com/doku.php?id=services:longrunningcluster>`_ is mounted. +#. Download the packages from chacra.ceph.com to the signing virtual machine. These packages get downloaded to ``/opt/repos`` where the `Sepia Lab Long Running (Ceph) Cluster <https://wiki.sepia.ceph.com/doku.php?id=services:longrunningcluster>`_ is mounted. Note: this step will also run a command to transfer the +source tarballs from chacra.ceph.com to download.ceph.com directly, by +ssh'ing to download.ceph.com and running /home/signer/bin/get-tarballs.sh. .. prompt:: bash $ @@ -210,27 +212,63 @@ See `the Ceph Tracker wiki page that explains how to write the release notes <ht sync-push ceph octopus -This leaves the packages in a password-protected prerelease area -at https://download.ceph.com/prerelease/ceph. Verify them from there. -When done and ready for release, mv the directories to the release -directory (that is, "mv <whatever you're promoting> ../..". +This leaves the packages, and the tarball, in a password-protected +prerelease area at https://download.ceph.com/prerelease/ceph. Verify them +from there. When done and ready for release, log into download.ceph.com and +mv the directories and the tarballs from the prerelease home +(/data/download.ceph.com/www/prerelease/ceph) to the release directory +(/data/download.ceph.com/www). 5. Build Containers =================== -Prerelease containers (x86_64 only) are built by -https://2.jenkins.ceph.com/job/ceph-container-prerelease-build; run it -with appropriate parameters. Test container images will appear on -quay.ceph.io in the ceph/prerelease repo, built from the prerelease area -on download.ceph.com. When satisfied with them, and after you have promoted -the prerelease packages to released status as above, start the following two jobs: +Architecture-specific containers are built during the ceph build and +pushed to quay.ceph.io/ceph/prerelease-{amd64,arm64}, containing the +packages built in that ceph build. The prerelease 'fat' container, +or manifest-list container, that refers to both arch-specific containers, +is built by hand using the command "make-manifest-list.py" in +ceph.git:src/container/make-manifest-list.py. Note that you must +be logged into the appropriate container repos for any of these +manipulations: quay.ceph.io for fetching prerelease arch-specific +containers and pushing the prerelease manifest-list container, and +quay.io for promoting the prerelease containers to released containers. -#. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs/ -#. https://2.jenkins.ceph.com/job/ceph-container-build-ceph-base-push-imgs-arm64/ + .. prompt:: bash + + cd <ceph-checkout>/src/container + ./make-manifest-list.py + +Reasonable defaults are set for all inputs, but environment variables +can be used to override: + + * ARCH_SPECIFIC_HOST (default 'quay.ceph.io'): host of prerelease repos + * AMD64_REPO (default 'ceph/prerelease-amd64') prerelease amd64 repo + * ARM64_REPO (default 'ceph/prerelease-arm64') prerelease arm64 repo + +(prerelease arch-specific containers will be copied from here) + + * MANIFEST_HOST (default 'quay.ceph.io') prerelease manifest-list host + * MANIFEST_REPO (default 'ceph/prerelease') prerelease manifest-list repo + +(prerelease manifest-list containers will be placed here) + +Finally, when all appropriate testing/ verification is done on the +container images, you can use make-manifest-list.py to promote them to +their final release location on quay.io/ceph/ceph: + + .. prompt:: bash + + cd <ceph-checkout>/src/container + ./make-manifest-list.py --promote + +Two more environment variables can override the default destination for +promotion (the source of the prerelease container to be promoted is +as above, in MANIFEST_HOST/REPO): + + * RELEASE_MANIFEST_HOST (default 'quay.io') release host + * RELEASE_MANIFEST_REPO (default 'ceph/ceph') release repo -which will rebuild and publish both architectures using the released packages -on download.ceph.com (into a multiarchitecture container image). 6. Announce the Release ======================= diff --git a/doc/glossary.rst b/doc/glossary.rst index 1ab962dcee9..5ecee57d21d 100644 --- a/doc/glossary.rst +++ b/doc/glossary.rst @@ -261,6 +261,14 @@ Another name for :term:`Dashboard`. Dashboard Plugin + The dashboard plugin was a Mimic-era web application that + visualized information and statistics about the Ceph cluster + using a web server hosted by the :ref:`Ceph + Manager<ceph-manager-daemon>`. + + See `the Mimic-era Dashboard Plugin documentation + <https://docs.ceph.com/en/mimic/mgr/dashboard/>`_. + DC **D**\ata **C**\enter. diff --git a/doc/man/8/cephadm.rst b/doc/man/8/cephadm.rst index 0847066b66d..3c23a9867f7 100644 --- a/doc/man/8/cephadm.rst +++ b/doc/man/8/cephadm.rst @@ -13,7 +13,7 @@ Synopsis | [--log-dir LOG_DIR] [--logrotate-dir LOGROTATE_DIR] | [--unit-dir UNIT_DIR] [--verbose] [--timeout TIMEOUT] | [--retry RETRY] [--no-container-init] -| {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install} +| {version,pull,inspect-image,ls,list-networks,adopt,rm-daemon,rm-cluster,run,shell,enter,ceph-volume,unit,logs,bootstrap,deploy,check-host,prepare-host,add-repo,rm-repo,install,list-images,update-osd-service} | ... @@ -104,6 +104,9 @@ Synopsis | [--registry-password REGISTRY_PASSWORD] | [--registry-json REGISTRY_JSON] [--fsid FSID] +| **cephadm** **list-images** + +| **cephadm** **update-osd-service** [-h] [--fsid FSID] --osd-ids OSD_IDS --service-name SERVICE_NAME Description @@ -527,6 +530,24 @@ Arguments: * [--name NAME, -n NAME] daemon name (type.id) +list-images +----------- + +List the default container images for all services in ini format. The output can be modified with custom images and passed to --config flag during bootstrap. + + +update-osd-service +------------------ + +Update the OSD service for specific OSDs + +Arguments: + +* [--fsid FSID] cluster FSID +* --osd-ids OSD_IDS Comma-separated OSD IDs +* --service-name SERVICE_NAME OSD service name + + Availability ============ diff --git a/doc/man/8/radosgw-admin.rst b/doc/man/8/radosgw-admin.rst index c7750c348ad..3cd4338a5ec 100644 --- a/doc/man/8/radosgw-admin.rst +++ b/doc/man/8/radosgw-admin.rst @@ -541,6 +541,13 @@ Options Generate random secret key. +.. option:: --generate-key + + create user with or without credentials. + If this option set to false, then user cannot set --gen-access-key/--gen-secret/--secret-key/--access-key. + If this option set to true, then user cannot set --secret-key/--access-key and bypass options for --gen-secret/--gen-access-key. + Default is true. + .. option:: --key-type=<type> Key type, options are: swift, s3. diff --git a/doc/man/8/rbd.rst b/doc/man/8/rbd.rst index 492dad652d2..6956b9e334b 100644 --- a/doc/man/8/rbd.rst +++ b/doc/man/8/rbd.rst @@ -576,7 +576,11 @@ Commands details for every mirror-enabled image in the pool or namespace. :command:`mirror snapshot schedule add` [-p | --pool *pool*] [--namespace *namespace*] [--image *image*] *interval* [*start-time*] - Add mirror snapshot schedule. + Add mirror snapshot schedule. The ``interval`` can be specified in + days, hours, or minutes using the d, h, m suffix respectively. + The ``start-time`` is a time string in ISO 8601 format. Not providing the + ``--pool``, ``--namespace`` and ``--image`` options creates a global + schedule which applies to all mirror-enabled images in the cluster. :command:`mirror snapshot schedule list` [-R | --recursive] [--format *format*] [--pretty-format] [-p | --pool *pool*] [--namespace *namespace*] [--image *image*] List mirror snapshot schedule. @@ -1031,6 +1035,9 @@ To restore an image from trash and rename it:: rbd trash restore mypool/myimage-id --image mynewimage +To create a mirror snapshot schedule for an image:: + + rbd mirror snapshot schedule add --pool mypool --image myimage 12h 14:00:00-05:00 Availability ============ diff --git a/doc/mgr/dashboard.rst b/doc/mgr/dashboard.rst index b0448bd0eef..32824fab4b5 100644 --- a/doc/mgr/dashboard.rst +++ b/doc/mgr/dashboard.rst @@ -1310,9 +1310,9 @@ redirection on standby nodes. mode tcp option httpchk GET / http-check expect status 200 - server x <HOST>:<PORT> ssl check verify none - server y <HOST>:<PORT> ssl check verify none - server z <HOST>:<PORT> ssl check verify none + server x <HOST>:<PORT> check check-ssl verify none + server y <HOST>:<PORT> check check-ssl verify none + server z <HOST>:<PORT> check check-ssl verify none .. _dashboard-auditing: diff --git a/doc/rados/operations/erasure-code.rst b/doc/rados/operations/erasure-code.rst index e53f348cdf4..aa79890c3a9 100644 --- a/doc/rados/operations/erasure-code.rst +++ b/doc/rados/operations/erasure-code.rst @@ -224,7 +224,7 @@ failures overlap. - m=2 - m=3 - m=4 - - m=4 + - m=5 - m=6 - m=7 - m=8 diff --git a/doc/rados/operations/health-checks.rst b/doc/rados/operations/health-checks.rst index 640fa9b7fdb..a1498a09fd0 100644 --- a/doc/rados/operations/health-checks.rst +++ b/doc/rados/operations/health-checks.rst @@ -29,58 +29,57 @@ Monitor DAEMON_OLD_VERSION __________________ -Warn if one or more Ceph daemons are running an old Ceph release. A health -check is raised if multiple versions are detected. This condition must exist -for a period of time greater than ``mon_warn_older_version_delay`` (set to one -week by default) in order for the health check to be raised. This allows most +One or more Ceph daemons are running an old Ceph release. A health check is +raised if multiple versions are detected. This condition must exist for a +period of time greater than ``mon_warn_older_version_delay`` (set to one week +by default) in order for the health check to be raised. This allows most upgrades to proceed without raising a warning that is both expected and -ephemeral. If the upgrade -is paused for an extended time, ``health mute`` can be used by running -``ceph health mute DAEMON_OLD_VERSION --sticky``. Be sure, however, to run -``ceph health unmute DAEMON_OLD_VERSION`` after the upgrade has finished so -that any future, unexpected instances are not masked. +ephemeral. If the upgrade is paused for an extended time, ``health mute`` can +be used by running ``ceph health mute DAEMON_OLD_VERSION --sticky``. Be sure, +however, to run ``ceph health unmute DAEMON_OLD_VERSION`` after the upgrade has +finished so that any future, unexpected instances are not masked. MON_DOWN ________ One or more Ceph Monitor daemons are down. The cluster requires a majority -(more than one-half) of the provsioned monitors to be available. When one or more monitors -are down, clients may have a harder time forming their initial connection to -the cluster, as they may need to try additional IP addresses before they reach an -operating monitor. +(more than one-half) of the provsioned monitors to be available. When one or +more monitors are down, clients may have a harder time forming their initial +connection to the cluster, as they may need to try additional IP addresses +before they reach an operating monitor. -Down monitor daemons should be restored or restarted as soon as possible to reduce the -risk that an additional monitor failure may cause a service outage. +Down monitor daemons should be restored or restarted as soon as possible to +reduce the risk that an additional monitor failure may cause a service outage. MON_CLOCK_SKEW ______________ -The clocks on hosts running Ceph Monitor daemons are not -well-synchronized. This health check is raised if the cluster detects a clock -skew greater than ``mon_clock_drift_allowed``. +The clocks on hosts running Ceph Monitor daemons are not well-synchronized. +This health check is raised if the cluster detects a clock skew greater than +``mon_clock_drift_allowed``. This issue is best resolved by synchronizing the clocks by using a tool like -the legacy ``ntpd`` or the newer ``chrony``. It is ideal to configure -NTP daemons to sync against multiple internal and external sources for resilience; +the legacy ``ntpd`` or the newer ``chrony``. It is ideal to configure NTP +daemons to sync against multiple internal and external sources for resilience; the protocol will adaptively determine the best available source. It is also -beneficial to have the NTP daemons on Ceph Monitor hosts sync against each other, -as it is even more important that Monitors be synchronized with each other than it -is for them to be _correct_ with respect to reference time. +beneficial to have the NTP daemons on Ceph Monitor hosts sync against each +other, as it is even more important that Monitors be synchronized with each +other than it is for them to be _correct_ with respect to reference time. If it is impractical to keep the clocks closely synchronized, the -``mon_clock_drift_allowed`` threshold can be increased. However, this -value must stay significantly below the ``mon_lease`` interval in order for the +``mon_clock_drift_allowed`` threshold can be increased. However, this value +must stay significantly below the ``mon_lease`` interval in order for the monitor cluster to function properly. It is not difficult with a quality NTP -or PTP configuration to have sub-millisecond synchronization, so there are very, very -few occasions when it is appropriate to change this value. +or PTP configuration to have sub-millisecond synchronization, so there are +very, very few occasions when it is appropriate to change this value. MON_MSGR2_NOT_ENABLED _____________________ -The :confval:`ms_bind_msgr2` option is enabled but one or more monitors are -not configured in the cluster's monmap to bind to a v2 port. This -means that features specific to the msgr2 protocol (for example, encryption) -are unavailable on some or all connections. +The :confval:`ms_bind_msgr2` option is enabled but one or more monitors are not +configured in the cluster's monmap to bind to a v2 port. This means that +features specific to the msgr2 protocol (for example, encryption) are +unavailable on some or all connections. In most cases this can be corrected by running the following command: @@ -100,32 +99,32 @@ manually. MON_DISK_LOW ____________ -One or more monitors are low on storage space. This health check is raised if the -percentage of available space on the file system used by the monitor database -(normally ``/var/lib/ceph/mon``) drops below the percentage value +One or more monitors are low on storage space. This health check is raised if +the percentage of available space on the file system used by the monitor +database (normally ``/var/lib/ceph/mon``) drops below the percentage value ``mon_data_avail_warn`` (default: 30%). This alert might indicate that some other process or user on the system is -filling up the file system used by the monitor. It might also -indicate that the monitor database is too large (see ``MON_DISK_BIG`` -below). Another common scenario is that Ceph logging subsystem levels have -been raised for troubleshooting purposes without subsequent return to default -levels. Ongoing verbose logging can easily fill up the files system containing -``/var/log``. If you trim logs that are currently open, remember to restart or -instruct your syslog or other daemon to re-open the log file. +filling up the file system used by the monitor. It might also indicate that the +monitor database is too large (see ``MON_DISK_BIG`` below). Another common +scenario is that Ceph logging subsystem levels have been raised for +troubleshooting purposes without subsequent return to default levels. Ongoing +verbose logging can easily fill up the files system containing ``/var/log``. If +you trim logs that are currently open, remember to restart or instruct your +syslog or other daemon to re-open the log file. -If space cannot be freed, the monitor's data directory might need to be -moved to another storage device or file system (this relocation process must be carried out while the monitor -daemon is not running). +If space cannot be freed, the monitor's data directory might need to be moved +to another storage device or file system (this relocation process must be +carried out while the monitor daemon is not running). MON_DISK_CRIT _____________ -One or more monitors are critically low on storage space. This health check is raised if the -percentage of available space on the file system used by the monitor database -(normally ``/var/lib/ceph/mon``) drops below the percentage value -``mon_data_avail_crit`` (default: 5%). See ``MON_DISK_LOW``, above. +One or more monitors are critically low on storage space. This health check is +raised if the percentage of available space on the file system used by the +monitor database (normally ``/var/lib/ceph/mon``) drops below the percentage +value ``mon_data_avail_crit`` (default: 5%). See ``MON_DISK_LOW``, above. MON_DISK_BIG ____________ @@ -235,8 +234,8 @@ this alert can be temporarily silenced by running the following command: ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 1w # 1 week -Although we do NOT recommend doing so, you can also disable this alert indefinitely -by running the following command: +Although we do NOT recommend doing so, you can also disable this alert +indefinitely by running the following command: .. prompt:: bash $ @@ -258,8 +257,8 @@ However, the cluster will still be able to perform client I/O operations and recover from failures. The down manager daemon(s) should be restarted as soon as possible to ensure -that the cluster can be monitored (for example, so that ``ceph -s`` -information is available and up to date, and so that metrics can be scraped by Prometheus). +that the cluster can be monitored (for example, so that ``ceph -s`` information +is available and up to date, and so that metrics can be scraped by Prometheus). MGR_MODULE_DEPENDENCY @@ -300,9 +299,8 @@ ________ One or more OSDs are marked ``down``. The ceph-osd daemon(s) or their host(s) may have crashed or been stopped, or peer OSDs might be unable to reach the OSD -over the public or private network. -Common causes include a stopped or crashed daemon, a "down" host, or a network -failure. +over the public or private network. Common causes include a stopped or crashed +daemon, a "down" host, or a network failure. Verify that the host is healthy, the daemon is started, and the network is functioning. If the daemon has crashed, the daemon log file @@ -513,9 +511,9 @@ or newer to start. To safely set the flag, run the following command: OSD_FILESTORE __________________ -Warn if OSDs are running the old Filestore back end. The Filestore OSD back end is -deprecated; the BlueStore back end has been the default object store since the -Ceph Luminous release. +Warn if OSDs are running the old Filestore back end. The Filestore OSD back end +is deprecated; the BlueStore back end has been the default object store since +the Ceph Luminous release. The 'mclock_scheduler' is not supported for Filestore OSDs. For this reason, the default 'osd_op_queue' is set to 'wpq' for Filestore OSDs and is enforced @@ -545,9 +543,9 @@ of any update to Reef or to later releases. OSD_UNREACHABLE _______________ -Registered v1/v2 public address of one or more OSD(s) is/are out of the -defined `public_network` subnet, which will prevent these unreachable OSDs -from communicating with ceph clients properly. +The registered v1/v2 public address or addresses of one or more OSD(s) is or +are out of the defined `public_network` subnet, which prevents these +unreachable OSDs from communicating with ceph clients properly. Even though these unreachable OSDs are in up state, rados clients will hang till TCP timeout before erroring out due to this inconsistency. @@ -555,7 +553,7 @@ will hang till TCP timeout before erroring out due to this inconsistency. POOL_FULL _________ -One or more pools have reached their quota and are no longer allowing writes. +One or more pools have reached quota and no longer allow writes. To see pool quotas and utilization, run the following command: @@ -641,9 +639,10 @@ command: BLUESTORE_FRAGMENTATION _______________________ -As BlueStore operates, the free space on the underlying storage will become -fragmented. This is normal and unavoidable, but excessive fragmentation causes -slowdown. To inspect BlueStore fragmentation, run the following command: +``BLUESTORE_FRAGMENTATION`` indicates that the free space that underlies +BlueStore has become fragmented. This is normal and unavoidable, but excessive +fragmentation causes slowdown. To inspect BlueStore fragmentation, run the +following command: .. prompt:: bash $ @@ -682,11 +681,9 @@ One or more OSDs have BlueStore volumes that were created prior to the Nautilus release. (In Nautilus, BlueStore tracks its internal usage statistics on a granular, per-pool basis.) -If *all* OSDs -are older than Nautilus, this means that the per-pool metrics are -simply unavailable. But if there is a mixture of pre-Nautilus and -post-Nautilus OSDs, the cluster usage statistics reported by ``ceph -df`` will be inaccurate. +If *all* OSDs are older than Nautilus, this means that the per-pool metrics are +simply unavailable. But if there is a mixture of pre-Nautilus and post-Nautilus +OSDs, the cluster usage statistics reported by ``ceph df`` will be inaccurate. The old OSDs can be updated to use the new usage-tracking scheme by stopping each OSD, running a repair operation, and then restarting the OSD. For example, @@ -798,7 +795,7 @@ about the source of the problem. BLUESTORE_SPURIOUS_READ_ERRORS ______________________________ -One or more BlueStore OSDs detect read errors on the main device. +One (or more) BlueStore OSDs detects read errors on the main device. BlueStore has recovered from these errors by retrying disk reads. This alert might indicate issues with underlying hardware, issues with the I/O subsystem, or something similar. Such issues can cause permanent data @@ -824,25 +821,27 @@ Or, to disable this alert on a specific OSD, run the following command: BLOCK_DEVICE_STALLED_READ_ALERT _______________________________ -There are certain BlueStore log messages that surface storage drive issues +There are BlueStore log messages that reveal storage drive issues that can cause performance degradation and potentially data unavailability or -loss. +loss. These may indicate a storage drive that is failing and should be +evaluated and possibly removed and replaced. ``read stalled read 0x29f40370000~100000 (buffered) since 63410177.290546s, timeout is 5.000000s`` -However, this is difficult to spot as there's no discernible warning (a +However, this is difficult to spot because there no discernible warning (a health warning or info in ``ceph health detail`` for example). More observations can be found here: https://tracker.ceph.com/issues/62500 -As there can be false positive ``stalled read`` instances, a mechanism -has been added for more reliability. If in last ``bdev_stalled_read_warn_lifetime`` -duration the number of ``stalled read`` indications are found to be more than or equal to +Also because there can be false positive ``stalled read`` instances, a mechanism +has been added to increase accuracy. If in the last ``bdev_stalled_read_warn_lifetime`` +seconds the number of ``stalled read`` events is found to be greater than or equal to ``bdev_stalled_read_warn_threshold`` for a given BlueStore block device, this -warning will be reported in ``ceph health detail``. +warning will be reported in ``ceph health detail``. The warning state will be +removed when the condition clears. -By default value of ``bdev_stalled_read_warn_lifetime = 86400s`` and -``bdev_stalled_read_warn_threshold = 1``. But user can configure it for -individual OSDs. +The defaults for :confval:`bdev_stalled_read_warn_lifetime` +and :confval:`bdev_stalled_read_warn_threshold` may be overridden globally or for +specific OSDs. To change this, run the following command: @@ -851,7 +850,8 @@ To change this, run the following command: ceph config set global bdev_stalled_read_warn_lifetime 10 ceph config set global bdev_stalled_read_warn_threshold 5 -this may be done surgically for individual OSDs or a given mask +This may be done for specific OSDs or a given mask. For example, +to apply only to SSD OSDs: .. prompt:: bash $ @@ -863,40 +863,43 @@ this may be done surgically for individual OSDs or a given mask WAL_DEVICE_STALLED_READ_ALERT _____________________________ -A similar warning like ``BLOCK_DEVICE_STALLED_READ_ALERT`` will be raised to -identify ``stalled read`` instances on a given BlueStore OSD's ``WAL_DEVICE``. -This warning can be configured via ``bdev_stalled_read_warn_lifetime`` and -``bdev_stalled_read_warn_threshold`` parameters similarly described in the -``BLOCK_DEVICE_STALLED_READ_ALERT`` warning section. +The warning state ``WAL_DEVICE_STALLED_READ_ALERT`` is raised to indicate +``stalled read`` instances on a given BlueStore OSD's ``WAL_DEVICE``. This +warning can be configured via the :confval:`bdev_stalled_read_warn_lifetime` +and :confval:`bdev_stalled_read_warn_threshold` options with commands similar +to those described in the ``BLOCK_DEVICE_STALLED_READ_ALERT`` warning section. DB_DEVICE_STALLED_READ_ALERT ____________________________ -A similar warning like ``BLOCK_DEVICE_STALLED_READ_ALERT`` will be raised to -identify ``stalled read`` instances on a given BlueStore OSD's ``WAL_DEVICE``. -This warning can be configured via ``bdev_stalled_read_warn_lifetime`` and -``bdev_stalled_read_warn_threshold`` parameters similarly described in the -``BLOCK_DEVICE_STALLED_READ_ALERT`` warning section. +The warning state ``DB_DEVICE_STALLED_READ_ALERT`` is raised to indicate +``stalled read`` instances on a given BlueStore OSD's ``DB_DEVICE``. This +warning can be configured via the :confval:`bdev_stalled_read_warn_lifetime` +and :confval:`bdev_stalled_read_warn_threshold` options with commands similar +to those described in the ``BLOCK_DEVICE_STALLED_READ_ALERT`` warning section. BLUESTORE_SLOW_OP_ALERT _______________________ -There are certain BlueStore log messages that surface storage drive issues -that can lead to performance degradation and data unavailability or loss. +There are BlueStore log messages that reveal storage drive issues that can lead +to performance degradation and data unavailability or loss. These indicate +that the storage drive may be failing and should be investigated and +potentially replaced. ``log_latency_fn slow operation observed for _txc_committed_kv, latency = 12.028621219s, txc = 0x55a107c30f00`` ``log_latency_fn slow operation observed for upper_bound, latency = 6.25955s`` ``log_latency slow operation observed for submit_transaction..`` As there can be false positive ``slow ops`` instances, a mechanism has -been added for more reliability. If in last ``bluestore_slow_ops_warn_lifetime`` -duration ``slow ops`` indications are found more than or equal to -``bluestore_slow_ops_warn_threshold`` for a given BlueStore OSD, this warning -will be reported in ``ceph health detail``. +been added for more reliability. If in the last ``bluestore_slow_ops_warn_lifetime`` +seconds the number of ``slow ops`` indications are found greater than or equal to +:confval:`bluestore_slow_ops_warn_threshold` for a given BlueStore OSD, this +warning will be reported in ``ceph health detail``. The warning state is +cleared when the condition clears. -By default value of ``bluestore_slow_ops_warn_lifetime = 86400s`` and -``bluestore_slow_ops_warn_threshold = 1``. But user can configure it for -individual OSDs. +The defaults for :confval:`bluestore_slow_ops_warn_lifetime` and +:confval:`bluestore_slow_ops_warn_threshold` may be overidden globally or for +specific OSDs. To change this, run the following command: @@ -905,7 +908,7 @@ To change this, run the following command: ceph config set global bluestore_slow_ops_warn_lifetime 10 ceph config set global bluestore_slow_ops_warn_threshold 5 -this may be done surgically for individual OSDs or a given mask +this may be done for specific OSDs or a given mask, for example: .. prompt:: bash $ @@ -931,8 +934,9 @@ the system. Note that this marking ``out`` is normally done automatically if ``mgr/devicehealth/mark_out_threshold``). If an OSD device is compromised but the OSD(s) on that device are still ``up``, recovery can be degraded. In such cases it may be advantageous to forcibly stop the OSD daemon(s) in question so -that recovery can proceed from surviving healthly OSDs. This should only be -done with extreme care so that data availability is not compromised. +that recovery can proceed from surviving healthly OSDs. This must be +done with extreme care and attention to failure domains so that data availability +is not compromised. To check device health, run the following command: @@ -940,8 +944,8 @@ To check device health, run the following command: ceph device info <device-id> -Device life expectancy is set either by a prediction model that the Manager -runs or by an external tool that is activated by running the following command: +Device life expectancy is set either by a prediction model that the Ceph Manager +runs or by an external tool that runs a command the following form: .. prompt:: bash $ @@ -1095,7 +1099,7 @@ ____________________ The count of read repairs has exceeded the config value threshold ``mon_osd_warn_num_repaired`` (default: ``10``). Because scrub handles errors only for data at rest, and because any read error that occurs when another -replica is available will be repaired immediately so that the client can get +replica is available is repaired immediately so that the client can get the object data, there might exist failing disks that are not registering any scrub errors. This repair count is maintained as a way of identifying any such failing disks. @@ -1112,8 +1116,8 @@ LARGE_OMAP_OBJECTS __________________ One or more pools contain large omap objects, as determined by -``osd_deep_scrub_large_omap_object_key_threshold`` (threshold for the number of -keys to determine what is considered a large omap object) or +``osd_deep_scrub_large_omap_object_key_threshold`` (the threshold for the +number of keys to determine what is considered a large omap object) or ``osd_deep_scrub_large_omap_object_value_sum_threshold`` (the threshold for the summed size in bytes of all key values to determine what is considered a large omap object) or both. To find more information on object name, key count, and @@ -1133,7 +1137,7 @@ CACHE_POOL_NEAR_FULL ____________________ A cache-tier pool is nearly full, as determined by the ``target_max_bytes`` and -``target_max_objects`` properties of the cache pool. Once the pool reaches the +``target_max_objects`` properties of the cache pool. When the pool reaches the target threshold, write requests to the pool might block while data is flushed and evicted from the cache. This state normally leads to very high latencies and poor performance. @@ -1279,10 +1283,10 @@ For more information, see :ref:`choosing-number-of-placement-groups` and POOL_TARGET_SIZE_BYTES_OVERCOMMITTED ____________________________________ -One or more pools have a ``target_size_bytes`` property that is set in order to -estimate the expected size of the pool, but the value(s) of this property are -greater than the total available storage (either by themselves or in -combination with other pools). +One or more pools does have a ``target_size_bytes`` property that is set in +order to estimate the expected size of the pool, but the value or values of +this property are greater than the total available storage (either by +themselves or in combination with other pools). This alert is usually an indication that the ``target_size_bytes`` value for the pool is too large and should be reduced or set to zero. To reduce the @@ -1354,7 +1358,7 @@ data have too many PGs. See *TOO_MANY_PGS* above. To silence the health check, raise the threshold by adjusting the ``mon_pg_warn_max_object_skew`` config option on the managers. -The health check will be silenced for a specific pool only if +The health check is silenced for a specific pool only if ``pg_autoscale_mode`` is set to ``on``. POOL_APP_NOT_ENABLED @@ -1421,8 +1425,8 @@ resolution, see :ref:`storage-capacity` and :ref:`no-free-drive-space`. OBJECT_MISPLACED ________________ -One or more objects in the cluster are not stored on the node that CRUSH would -prefer that they be stored on. This alert is an indication that data migration +One or more objects in the cluster are not stored on the node that CRUSH +prefers that they be stored on. This alert is an indication that data migration due to a recent cluster change has not yet completed. Misplaced data is not a dangerous condition in and of itself; data consistency @@ -1489,7 +1493,7 @@ percentage (determined by ``mon_warn_pg_not_scrubbed_ratio``) of the interval has elapsed after the time the scrub was scheduled and no scrub has been performed. -PGs will be scrubbed only if they are flagged as ``clean`` (which means that +PGs are scrubbed only if they are flagged as ``clean`` (which means that they are to be cleaned, and not that they have been examined and found to be clean). Misplaced or degraded PGs will not be flagged as ``clean`` (see *PG_AVAILABILITY* and *PG_DEGRADED* above). @@ -1621,9 +1625,10 @@ Stretch Mode INCORRECT_NUM_BUCKETS_STRETCH_MODE __________________________________ -Stretch mode currently only support 2 dividing buckets with OSDs, this warning suggests -that the number of dividing buckets is not equal to 2 after stretch mode is enabled. -You can expect unpredictable failures and MON assertions until the condition is fixed. +Stretch mode currently only support 2 dividing buckets with OSDs, this warning +suggests that the number of dividing buckets is not equal to 2 after stretch +mode is enabled. You can expect unpredictable failures and MON assertions +until the condition is fixed. We encourage you to fix this by removing additional dividing buckets or bump the number of dividing buckets to 2. @@ -1646,18 +1651,28 @@ NVMeoF Gateway NVMEOF_SINGLE_GATEWAY _____________________ -One of the gateway group has only one gateway. This is not ideal because it makes -high availability (HA) impossible with a single gatway in a group. This can lead to -problems with failover and failback operations for the NVMeoF gateway. +One of the gateway group has only one gateway. This is not ideal because it +makes high availability (HA) impossible with a single gatway in a group. This +can lead to problems with failover and failback operations for the NVMeoF +gateway. It's recommended to have multiple NVMeoF gateways in a group. NVMEOF_GATEWAY_DOWN ___________________ -Some of the gateways are in the GW_UNAVAILABLE state. If a NVMeoF daemon has crashed, -the daemon log file (found at ``/var/log/ceph/``) may contain troubleshooting information. +Some of the gateways are in the GW_UNAVAILABLE state. If a NVMeoF daemon has +crashed, the daemon log file (found at ``/var/log/ceph/``) may contain +troubleshooting information. + +NVMEOF_GATEWAY_DELETING +_______________________ +Some of the gateways are in the GW_DELETING state. They will stay in this +state until all the namespaces under the gateway's load balancing group are +moved to another load balancing group ID. This is done automatically by the +load balancing process. If this alert persist for a long time, there might +be an issue with that process. Miscellaneous ------------- diff --git a/doc/rados/operations/stretch-mode.rst b/doc/rados/operations/stretch-mode.rst index ffb94e52943..7a4fa46117d 100644 --- a/doc/rados/operations/stretch-mode.rst +++ b/doc/rados/operations/stretch-mode.rst @@ -94,15 +94,54 @@ configuration across the entire cluster. Conversely, opt for a ``stretch pool`` when you need a particular pool to be replicated across ``more than two data centers``, providing a more granular level of control and a larger cluster size. +Limitations +----------- + +Individual Stretch Pools do not support I/O operations during a netsplit +scenario between two or more zones. While the cluster remains accessible for +basic Ceph commands, I/O usage remains unavailable until the netsplit is +resolved. This is different from ``stretch mode``, where the tiebreaker monitor +can isolate one zone of the cluster and continue I/O operations in degraded +mode during a netsplit. See :ref:`stretch_mode1` + +Ceph is designed to tolerate multiple host failures. However, if more than 25% of +the OSDs in the cluster go down, Ceph may stop marking OSDs as out which will prevent rebalancing +and some PGs might go inactive. This behavior is controlled by the ``mon_osd_min_in_ratio`` parameter. +By default, mon_osd_min_in_ratio is set to 0.75, meaning that at least 75% of the OSDs +in the cluster must remain ``active`` before any additional OSDs can be marked out. +This setting prevents too many OSDs from being marked out as this might lead to significant +data movement. The data movement can cause high client I/O impact and long recovery times when +the OSDs are returned to service. If Ceph stops marking OSDs as out, some PGs may fail to +rebalance to surviving OSDs, potentially leading to ``inactive`` PGs. +See https://tracker.ceph.com/issues/68338 for more information. + +.. _stretch_mode1: + Stretch Mode ============ -Stretch mode is designed to handle deployments in which you cannot guarantee the -replication of data across two data centers. This kind of situation can arise -when the cluster's CRUSH rule specifies that three copies are to be made, but -then a copy is placed in each data center with a ``min_size`` of 2. Under such -conditions, a placement group can become active with two copies in the first -data center and no copies in the second data center. +Stretch mode is designed to handle netsplit scenarios between two data zones as well +as the loss of one data zone. It handles the netsplit scenario by choosing the surviving zone +that has the better connection to the ``tiebreaker monitor``. It handles the loss of one zone by +reducing the ``size`` to ``2`` and ``min_size`` to ``1``, allowing the cluster to continue operating +with the remaining zone. When the lost zone comes back, the cluster will recover the lost data +and return to normal operation. + +Connectivity Monitor Election Strategy +--------------------------------------- +When using stretch mode, the monitor election strategy must be set to ``connectivity``. +This strategy tracks network connectivity between the monitors and is +used to determine which zone should be favored when the cluster is in a netsplit scenario. + +See `Changing Monitor Elections`_ + +Stretch Peering Rule +-------------------- +One critical behavior of stretch mode is its ability to prevent a PG from going active if the acting set +contains only replicas from a single zone. This safeguard is crucial for mitigating the risk of data +loss during site failures because if a PG were allowed to go active with replicas only in a single site, +writes could be acknowledged despite a lack of redundancy. In the event of a site failure, all data in the +affected PG would be lost. Entering Stretch Mode --------------------- @@ -248,7 +287,7 @@ possible, if needed). .. _Changing Monitor elections: ../change-mon-elections Exiting Stretch Mode -===================== +-------------------- To exit stretch mode, run the following command: .. prompt:: bash $ diff --git a/doc/radosgw/account.rst b/doc/radosgw/account.rst index 6dab997d93e..0e4ede5a50a 100644 --- a/doc/radosgw/account.rst +++ b/doc/radosgw/account.rst @@ -174,6 +174,11 @@ An existing user can be adopted into an account with ``user modify``:: .. note:: Account membership is permanent. Once added, users cannot be removed from their account. +.. note:: The IAM User API imposes additional requirements on the format + of ``UserName``, which is enforced when migrating users into an account. + If migration fails with "UserName contains invalid characters", the + ``--display-name`` should be modified to match ``[\w+=,.@-]+``. + .. warning:: Ownership of the user's notification topics will not be transferred to the account. Notifications will continue to work, but the topics will no longer be visible to SNS Topic APIs. Topics and diff --git a/doc/radosgw/admin.rst b/doc/radosgw/admin.rst index 7c7d9d6df14..8dbf8c10b04 100644 --- a/doc/radosgw/admin.rst +++ b/doc/radosgw/admin.rst @@ -262,6 +262,7 @@ include: - ``--secret-key=<key>`` manually specifies a S3 secret key or a Swift secret key. - ``--gen-access-key`` automatically generates a random S3 access key. - ``--gen-secret`` automatically generates a random S3 secret key or a random Swift secret key. +- ``--generate-key`` create user with or without credentials. If sets to false, then user cannot set ``gen-secret/gen-access-key/access-key/secret-key`` Adding S3 keys ~~~~~~~~~~~~~~ diff --git a/doc/radosgw/bucket_logging.rst b/doc/radosgw/bucket_logging.rst new file mode 100644 index 00000000000..f3e790f5705 --- /dev/null +++ b/doc/radosgw/bucket_logging.rst @@ -0,0 +1,157 @@ +==================== +Bucket Logging +==================== + +.. versionadded:: T + +.. contents:: + +Bucket logging provides a mechanism for logging all access to a bucket. The +log data can be used to monitor bucket activity, detect unauthorized +access, get insights into the bucket usage and use the logs as a journal for bucket changes. +The log records are stored in objects in a separate bucket and can be analyzed later. +Logging configuration is done at the bucket level and can be enabled or disabled at any time. +The log bucket can accumulate logs from multiple buckets. It is recommended to configured +a different "prefix" for each bucket, so that the logs of different buckets will be stored +in different objects in the log bucket. + +.. note:: + + - The log bucket must be created before enabling logging on a bucket + - The log bucket cannot be the same as the bucket being logged + - The log bucket cannot have logging enabled on it + + +.. toctree:: + :maxdepth: 1 + +Logging Reliability +------------------- +For performance reasons, even though the log records are written to persistent storage, the log object will +appear in the log bucket only after some configurable amount of time (or if the maximum object size of 128MB is reached). +This time (in seconds) could be set per source bucket via a Ceph extension to the REST API, +or globally via the `rgw_bucket_logging_obj_roll_time` configuration option. If not set, the default time is 5 minutes. +Adding a log object to the log bucket is done "lazily", meaning, that if no more records are written to the object, it may +remain outside of the log bucket even after the configured time has passed. +To counter that, you can flush all logging objects on a given source bucket to log them, +regardless if enough time passed or if no more records are written to the object. +Flushing will happen automatically when logging is disabled on a bucket, its logging configuration is changed, or the bucket is deleted. + +Standard +```````` +If logging type is set to "Standard" (the default) the log records are written to the log bucket after the bucket operation is completed. +This means that there are the logging operation may fail, with no indication to he client. + +Journal +``````` +If logging type is set to "Journal", the records are written to the log bucket before the bucket operation is completed. +This means that if the logging action fails, the operation will not be executed, and an error will be returned to the client. +An exception to the above are "multi/delete" log records: if writing these log records fail, the operation continues and may still be successful. +Journal mode supports filtering out records based on matches of the prefixes and suffixes of the logged object keys. Regular-expression matching can also be used on these to create filters. +Note that it may happen that the log records were successfully written, but the bucket operation failed, since the logs are written. + + +Bucket Logging REST API +----------------------- +Detailed under: `Bucket Operations`_. + + +Log Objects Key Format +---------------------- + +Simple +`````` +has the following format: + +:: + + <prefix><year-month-day-hour-minute-second>-<16 bytes unique-id> + +For example: + +:: + + fish/2024-08-06-09-40-09-TI9ROKN05DD4HPQF + +Partitioned +``````````` +has the following format: + +:: + + <prefix><bucket owner>/<source region>/[tenant:]<bucket name>/<year>/<month>/<day>/<year-month-day-hour-minute-second>-<16 bytes unique-id> + +For example: + +:: + + fish/testid//all-log/2024/08/06/2024-08-06-10-11-18-1HMU3UMWOJKNQJ0X + +Log Records +~~~~~~~~~~~ + +The log records are space separated string columns and have the following possible formats: + +Journal +``````` +minimum amount of data used for journaling bucket changes (this is a Ceph extension). + + - bucket owner (or dash if empty) + - bucket name (or dash if empty). in the format: ``[tenant:]<bucket name>`` + - time in the following format: ``[day/month/year:hour:minute:second timezone]`` + - object key (or dash if empty) + - operation in the following format: ``WEBSITE/REST.<HTTP method>.<resource>`` + - object size (or dash if empty) + - version id (dash if empty or question mark if unknown) + - eTag + +For example: + +:: + + testid fish [06/Aug/2024:09:40:09 +0000] myfile - REST.PUT.OBJECT 4cfdfc1f58e762d3e116787cb92fac60 + testid fish [06/Aug/2024:09:40:28 +0000] myfile REST.DELETE.OBJECT 4cfdfc1f58e762d3e116787cb92fac60 + + +Standard +```````` +based on `AWS Logging Record Format`_. + + - bucket owner (or dash if empty) + - bucket name (or dash if empty). in the format: ``[tenant:]<bucket name>`` + - time + - remote IP (not supported, always a dash) + - user or account (or dash if empty) + - request ID + - operation in the following format: ``WEBSITE/REST.<HTTP method>.<resource>`` + - object key (or dash if empty) + - request URI in the following format: ``"<HTTP method> <URI> <HTTP version>"`` + - HTTP status (or dash if zero). Note that in most cases log is written before the status is known + - error code (or dash if empty) + - bytes sent (or dash if zero) + - object size (or dash if zero) + - total time (not supported, always a dash) + - turnaround time (not supported, always a dash) + - referrer (not supported, always a dash) + - user agent (not supported, always a dash) + - version id (or dash if empty) + - host id taken from "x-amz-id-2" (or dash if empty) + - signature version (not supported, always a dash) + - cipher suite (not supported, always a dash) + - authentication type (not supported, always a dash) + - host header (or dash if empty) + - TLS version (not supported, always a dash) + - access point ARN (not supported, always a dash) + - ACL flag ("Yes" if the request is an ACL operation, otherwise dash) + +For example: + +:: + + testid fish [06/Aug/2024:09:30:25 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.15085270386962380710 REST.PUT.OBJECT myfile "PUT /fish/myfile HTTP/1.1" 200 - 512 512 - - - - - - - - - localhost - - + testid fish [06/Aug/2024:09:30:51 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.7046073853138417766 REST.GET.OBJECT myfile "GET /fish/myfile HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - - + testid fish [06/Aug/2024:09:30:56 +0000] - testid 9e369a15-5f43-4f07-b638-de920b22f91b.4179.10723158448701085570 REST.DELETE.OBJECT myfile "DELETE /fish/myfile1 HTTP/1.1" 200 - - 512 - - - - - - - - - localhost - - + + +.. _AWS Logging Record Format: https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html +.. _Bucket Operations: ../s3/bucketops diff --git a/doc/radosgw/config-ref.rst b/doc/radosgw/config-ref.rst index edc6a90b0f9..b4aa56fff54 100644 --- a/doc/radosgw/config-ref.rst +++ b/doc/radosgw/config-ref.rst @@ -78,7 +78,7 @@ These values can be tuned based upon your specific workload to further increase aggressiveness of lifecycle processing. For a workload with a larger number of buckets (thousands) you would look at increasing the :confval:`rgw_lc_max_worker` value from the default value of 3 whereas for a workload with a smaller number of buckets but higher number of objects (hundreds of thousands) -per bucket you would consider decreasing :confval:`rgw_lc_max_wp_worker` from the default value of 3. +per bucket you would consider increasing :confval:`rgw_lc_max_wp_worker` from the default value of 3. .. note:: When looking to tune either of these specific values please validate the current Cluster performance and Ceph Object Gateway utilization before increasing. diff --git a/doc/radosgw/index.rst b/doc/radosgw/index.rst index 3085e1a528f..bfb44082632 100644 --- a/doc/radosgw/index.rst +++ b/doc/radosgw/index.rst @@ -89,3 +89,5 @@ Cluster with one API and then retrieve that data with the other API. Cloud Transition <cloud-transition> Metrics <metrics> UADK Acceleration for Compression <uadk-accel> + Bucket Logging <bucket_logging> + diff --git a/doc/radosgw/notifications.rst b/doc/radosgw/notifications.rst index 05653956be1..897c280facf 100644 --- a/doc/radosgw/notifications.rst +++ b/doc/radosgw/notifications.rst @@ -188,6 +188,7 @@ updating, use the name of an existing topic and different endpoint values). [&Attributes.entry.15.key=Policy&Attributes.entry.15.value=<policy-JSON-string>] [&Attributes.entry.16.key=user-name&Attributes.entry.16.value=<user-name-string>] [&Attributes.entry.17.key=password&Attributes.entry.17.value=<password-string>] + [&Attributes.entry.18.key=kafka-brokers&Attributes.entry.18.value=<kafka-broker-list>] Request parameters: @@ -296,6 +297,8 @@ Request parameters: - "broker": Messages are considered "delivered" if acked by the broker. (This is the default.) + - kafka-brokers: A command-separated list of host:port of kafka brokers. These brokers (may contain a broker which is defined in kafka uri) will be added to kafka uri to support sending notifcations to a kafka cluster. + .. note:: - The key-value pair of a specific parameter need not reside in the same @@ -571,6 +574,7 @@ Valid AttributeName that can be passed: - mechanism: may be provided together with user/password (default: ``PLAIN``). - kafka-ack-level: No end2end acknowledgement is required. Messages may persist in the broker before being delivered to their final destinations. + - kafka-brokers: Set endpoint with broker(s) as a comma-separated list of host or host:port (default port 9092). Notifications ~~~~~~~~~~~~~ diff --git a/doc/radosgw/s3.rst b/doc/radosgw/s3.rst index cb5eb3adbdb..7acfb84f88c 100644 --- a/doc/radosgw/s3.rst +++ b/doc/radosgw/s3.rst @@ -82,6 +82,8 @@ The following table describes the support status for current Amazon S3 functiona +---------------------------------+-----------------+----------------------------------------+ | **Storage Class** | Supported | See :ref:`storage_classes` | +---------------------------------+-----------------+----------------------------------------+ +| **Bucket Logging** | Supported | | ++---------------------------------+-----------------+----------------------------------------+ Unsupported Header Fields ------------------------- diff --git a/doc/radosgw/s3/bucketops.rst b/doc/radosgw/s3/bucketops.rst index 984733fff75..c33a8c0f410 100644 --- a/doc/radosgw/s3/bucketops.rst +++ b/doc/radosgw/s3/bucketops.rst @@ -705,3 +705,232 @@ HTTP Response +---------------+-----------------------+----------------------------------------------------------+ .. _S3 Notification Compatibility: ../../s3-notification-compatibility + +Enable Bucket Logging +--------------------- + +Enable logging for a bucket. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?logging HTTP/1.1 + + +Request Entities +~~~~~~~~~~~~~~~~ + +Parameters are XML encoded in the body of the request, in the following format: + +:: + + <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <LoggingEnabled> + <TargetBucket>string</TargetBucket> + <TargetGrants> + <Grant> + <Grantee> + <DisplayName>string</DisplayName> + <EmailAddress>string</EmailAddress> + <ID>string</ID> + <xsi:type>string</xsi:type> + <URI>string</URI> + </Grantee> + <Permission>string</Permission> + </Grant> + </TargetGrants> + <TargetObjectKeyFormat> + <PartitionedPrefix> + <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource> + </PartitionedPrefix> + <SimplePrefix> + </SimplePrefix> + </TargetObjectKeyFormat> + <TargetPrefix>string</TargetPrefix> + <LoggingType>Standard|Journal</LoggingType> + <ObjectRollTime>integer</ObjectRollTime> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> + </LoggingEnabled> + </BucketLoggingStatus> + ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| Name | Type | Description | Required | ++===============================+===========+======================================================================================+==========+ +| ``BucketLoggingStatus`` | Container | Enabling/Disabling logging configuration for the bucket. | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``LoggingEnabled`` | Container | Holding the logging configuration for the bucket. | Yes | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TargetBucket`` | String | The bucket where the logs are stored. The log bucket cannot have bucket logging | Yes | +| | | enabled. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TargetGrants`` | Container | Not supported. The owner of the log bucket is the owner of the log objects. | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TargetObjectKeyFormat`` | Container | The format of the log object key. Contains either ``PartitionedPrefix`` or | No | +| | | ``SimplePrefix`` entities. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``PartitionedPrefix`` | Container | Indicates a partitioned log object key format. Note that ``PartitionDateSource`` | No | +| | | is ignored and hardcoded as ``DeliveryTime`` | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``SimplePrefix`` | Container | Indicates a simple log object key format (default format) | No | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``TargetPrefix`` | String | The prefix for the log objects. Used in both formats. May be used to distinguish | No | +| | | between different source buckets writing log records to the same log bucket. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``LoggingType`` | String | The type of logging. Valid values are: | No | +| | | ``Standard`` (default) all bucket operations are logged after being perfomed. | | +| | | The log record will contain all fields. | | +| | | ``Journal`` only PUT, COPY, MULTI/DELETE and MPU operations are logged. | | +| | | Will record the minimum subset of fields in the log record that is needed | | +| | | for journaling. | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ +| ``ObjectRollTime`` | Integer | The time in seconds after which a new log object is created, and the previous log | No | +| | | object added to the log bucket. Default is 3600 seconds (1 hour). | | ++-------------------------------+-----------+--------------------------------------------------------------------------------------+----------+ + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``400`` | MalformedXML | The XML is not well-formed | ++---------------+-----------------------+----------------------------------------------------------+ +| ``400`` | InvalidArgument | Missing mandatory value or invalid value | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + + +Disable Bucket Logging +---------------------- + +Disable bucket logging from a bucket. + +Syntax +~~~~~~ + +:: + + PUT /{bucket}?logging HTTP/1.1 + + +Request Entities +~~~~~~~~~~~~~~~~ + +Parameters are XML encoded in the body of the request, in the following format: + +:: + + <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + </BucketLoggingStatus> + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + +Get Bucket Logging +------------------ + +Get logging configured on a bucket. + +Syntax +~~~~~~ + +:: + + GET /{bucket}?logging HTTP/1.1 + + +Response Entities +~~~~~~~~~~~~~~~~~ + +Response is XML encoded in the body of the request, in the following format: + +:: + + <BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <LoggingEnabled> + <TargetBucket>string</TargetBucket> + <TargetGrants> + <Grant> + <Grantee> + <DisplayName>string</DisplayName> + <EmailAddress>string</EmailAddress> + <ID>string</ID> + <xsi:type>string</xsi:type> + <URI>string</URI> + </Grantee> + <Permission>string</Permission> + </Grant> + </TargetGrants> + <TargetObjectKeyFormat> + <PartitionedPrefix> + <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource> + </PartitionedPrefix> + <SimplePrefix> + </SimplePrefix> + </TargetObjectKeyFormat> + <TargetPrefix>string</TargetPrefix> + <LoggingType>Standard|Journal</LoggingType> + <ObjectRollTime>integer</ObjectRollTime> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> + </LoggingEnabled> + </BucketLoggingStatus> + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + +Flush Bucket Logging +-------------------- + +Flushes all logging objects for a given source bucket (logging bucket are written lazily). + +Syntax +~~~~~~ + +:: + + POST /{bucket}?logging HTTP/1.1 + + +HTTP Response +~~~~~~~~~~~~~ + ++---------------+-----------------------+----------------------------------------------------------+ +| HTTP Status | Status Code | Description | ++===============+=======================+==========================================================+ +| ``201`` | Created | Flushed all logging objects successfully | ++---------------+-----------------------+----------------------------------------------------------+ +| ``404`` | NoSuchBucket | The bucket does not exist | ++---------------+-----------------------+----------------------------------------------------------+ + diff --git a/doc/radosgw/s3/objectops.rst b/doc/radosgw/s3/objectops.rst index 2ac52607fe3..ddc5fb910c4 100644 --- a/doc/radosgw/s3/objectops.rst +++ b/doc/radosgw/s3/objectops.rst @@ -115,7 +115,7 @@ Request Headers +---------------------------+------------------------------------------------+--------------------------------+------------+ | **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ -| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +| **if-none-match** | Gets only if object ETag doesn't match. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ Response Headers @@ -155,7 +155,7 @@ Request Headers +---------------------------+------------------------------------------------+--------------------------------+------------+ | **if-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ -| **if-none-match** | Gets only if object ETag matches ETag. | Entity Tag | No | +| **if-none-match** | Gets only if object ETag doesn't match | Entity Tag | No | +---------------------------+------------------------------------------------+--------------------------------+------------+ Get Object ACL diff --git a/doc/radosgw/uadk-accel.rst b/doc/radosgw/uadk-accel.rst index fdf99f891f0..aaafe1c21df 100644 --- a/doc/radosgw/uadk-accel.rst +++ b/doc/radosgw/uadk-accel.rst @@ -2,9 +2,9 @@ UADK Acceleration for Compression =============================================== -UADK is a framework for applications to access hardware accelerators in a -unified, secure, and efficient way. UADK is comprised of UACCE, libwd and many -other algorithm libraries. +UADK is a framework that makes it possible for applications to access hardware +accelerators in a unified, secure, and efficient way. UADK is comprised of +UACCE, libwd, and many other algorithm libraries. See `Compressor UADK Support`_. @@ -12,31 +12,31 @@ See `Compressor UADK Support`_. UADK in the Software Stack ========================== -UADK is a general-purpose user space accelerator framework that uses shared -virtual addressing (SVA) to provide a unified programming interface for hardware -acceleration of cryptographic and compression algorithms. +UADK is a general-purpose user space accelerator framework that uses Shared +Virtual Addressing (SVA) to provide a unified programming interface for +hardware acceleration of cryptographic and compression algorithms. UADK includes Unified/User-space-access-intended Accelerator Framework (UACCE), which enables hardware accelerators that support SVA to adapt to UADK. Currently, HiSilicon Kunpeng hardware accelerators have been registered with UACCE. Through the UADK framework, users can run cryptographic and compression -algorithms using hardware accelerators instead of CPUs, freeing up CPU computing -power and improving computing performance. +algorithms using hardware accelerators instead of CPUs, which frees up CPU +computing power and improves computing performance. -A user can access the hardware accelerators by performing user-mode operations on -the character devices, or the use of UADK can be done via frameworks that have -been enabled by others including UADK support (for example, OpenSSL* libcrypto*, -DPDK, and the Linux* Kernel Crypto Framework). +Users can access the hardware accelerators by performing user-mode operations +on the character devices, or the use of UADK can be achieved via frameworks +that have been enabled by others including UADK support (for example, OpenSSL* +libcrypto*, DPDK, and the Linux* Kernel Crypto Framework). See `OpenSSL UADK Engine`_. UADK Environment Setup ====================== -UADK consists of UACCE, vendors’ drivers, and an algorithm layer. UADK requires the -hardware accelerator to support SVA, and the operating system to support IOMMU and -SVA. Hardware accelerators from different vendors are registered as different character -devices with UACCE by using kernel-mode drivers of the vendors. +UADK consists of UACCE, vendor drivers, and an algorithm layer. UADK requires +the hardware accelerator to support SVA, and the operating system to support +IOMMU and SVA. Hardware accelerators are registered as different character +devices with UACCE by kernel-mode drivers. :: @@ -77,11 +77,12 @@ Configuration #. Kernel Requirement -User needs to make sure that UACCE is already supported in Linux kernel. The kernel version -should be at least v5.9 with SVA (Shared Virtual Addressing) enabled. +Users must ensure that UACCE is supported by the Linux kernel release in use, +which should be 5.9 or later with SVA (Shared Virtual Addressing) enabled. -UACCE may be built as a module or built into the kernel. Here's an example to build UACCE -with hardware accelerators for the HiSilicon Kunpeng platform. +UACCE may be built as a loadable module or built into the kernel. Here's an +example to build UACCE with hardware accelerators for the HiSilicon Kunpeng +platform. .. prompt:: bash $ @@ -97,13 +98,17 @@ with hardware accelerators for the HiSilicon Kunpeng platform. Make sure all these above kernel configurations are selected. #. UADK enablement -If the architecture is aarch64, it will automatically download the UADK source code to build -the static library. If it runs on other architecture, user can enable it with build parameters -`-DWITH_UADK=true` - -#. Manual Build UADK -As the above paragraph shows, the UADK is enabled automatically, no need to build manually. -For developer who is interested in UADK, you can refer to the below steps for building. +If the architecture is ``aarch64``, it will automatically download the UADK +source code to build the static library. When building on other CPU +architectures, the user may enable UADK by adding ``-DWITH_UADK=true`` to the +compilation command line options. Note that UADK may not be compatible with all +architectures. + +#. Manually Building UADK +As implied in the above paragraph, if the architecture is ``aarch64``, the UADK +is enabled automatically and there is no need to build it manually. However, +below we provide the procedure for manually building UADK so that developers +can study how it is built. .. prompt:: bash $ @@ -115,9 +120,9 @@ For developer who is interested in UADK, you can refer to the below steps for bu make make install - .. note:: Without –prefix, UADK will be installed to /usr/local/lib by - default. If get error:"cannot find -lnuma", please install - the `libnuma-dev`. + .. note:: Without ``--prefix``, UADK will be installed under + ``/usr/local/lib`` by default. If you get the error: + ``cannot find -lnuma``, install the ``libnuma-dev`` package. #. Configure @@ -126,7 +131,8 @@ For developer who is interested in UADK, you can refer to the below steps for bu uadk_compressor_enabled=true - The default value in `global.yaml.in` for `uadk_compressor_enabled` is false. + The default value in `global.yaml.in` for `uadk_compressor_enabled` is + ``false``. .. _Compressor UADK Support: https://github.com/ceph/ceph/pull/58336 .. _OpenSSL UADK Engine: https://github.com/Linaro/uadk_engine diff --git a/doc/releases/index.rst b/doc/releases/index.rst index a8015c65465..1393770878f 100644 --- a/doc/releases/index.rst +++ b/doc/releases/index.rst @@ -23,7 +23,6 @@ security fixes. Squid (v19.2.*) <squid> Reef (v18.2.*) <reef> - Quincy (v17.2.*) <quincy> .. ceph_releases:: releases.yml current @@ -40,6 +39,7 @@ receive bug fixes or backports). :maxdepth: 1 :hidden: + Quincy (v17.2.*) <quincy> Pacific (v16.2.*) <pacific> Octopus (v15.2.*) <octopus> Nautilus (v14.2.*) <nautilus> diff --git a/doc/releases/quincy.rst b/doc/releases/quincy.rst index 6ba46552bd0..76ad268adf8 100644 --- a/doc/releases/quincy.rst +++ b/doc/releases/quincy.rst @@ -26,10 +26,10 @@ from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo f point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 These CERN packages come with no warranty and have not been tested. The software in them has been -tested by Ceph according to [platforms](https://docs.ceph.com/en/latest/start/os-recommendations/#platforms). -The repository for el8 builds is hosted by CERN on [Linux @ CERN](https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/). -The public part of the GPG key used to sign the packages is available at -[RPM-GPG-KEY-Ceph-Community](https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community). +tested by Ceph according to `platforms <https://docs.ceph.com/en/latest/start/os-recommendations/#platforms>`_ . +The repository for el8 builds is hosted by CERN on `Linux@CERN <https://linuxsoft.cern.ch/repos/ceph-ext-quincy8el-stable/>`_ . +The public part of the GPG key used to sign the +packages is available at `RPM-GPG-KEY-Ceph-Community <https://linuxsoft.cern.ch/repos/RPM-GPG-KEY-Ceph-Community>`_ . Notable Changes --------------- diff --git a/doc/releases/releases.yml b/doc/releases/releases.yml index 948f9eab278..6a76cc7c92c 100644 --- a/doc/releases/releases.yml +++ b/doc/releases/releases.yml @@ -32,6 +32,7 @@ releases: quincy: target_eol: 2024-06-01 + actual_eol: 2025-01-13 releases: - version: 17.2.8 released: 2024-11-25 diff --git a/doc/start/hardware-recommendations.rst b/doc/start/hardware-recommendations.rst index 3c3c781a815..3d5e44d8e02 100644 --- a/doc/start/hardware-recommendations.rst +++ b/doc/start/hardware-recommendations.rst @@ -311,7 +311,7 @@ media cost. Moreover, when using NVMe SSDs, you do not need *any* HBA. This additionally reduces the HDD vs SSD cost gap when the system as a whole is considered. The initial cost of a fancy RAID HBA plus onboard cache plus battery backup (BBU or supercapacitor) can easily exceed more than 1000 US -dollars even after discounts - a sum that goes a log way toward SSD cost parity. +dollars even after discounts - a sum that goes a long way toward SSD cost parity. An HBA-free system may also cost hundreds of US dollars less every year if one purchases an annual maintenance contract or extended warranty. diff --git a/examples/rgw/boto3/bucket_logging.py b/examples/rgw/boto3/bucket_logging.py new file mode 100644 index 00000000000..7a972dac8bc --- /dev/null +++ b/examples/rgw/boto3/bucket_logging.py @@ -0,0 +1,61 @@ +#!/usr/bin/python + +import boto3 +import sys + +if len(sys.argv) != 3: + print('Usage: ' + sys.argv[0] + ' <bucket> <target bucket>') + sys.exit(1) + +# bucket name as first argument +bucket = sys.argv[1] +# target bucket name as the 2nd argument +target_bucket = sys.argv[2] + +# endpoint and keys from vstart +endpoint = 'http://127.0.0.1:8000' +access_key='0555b35654ad1656d804' +secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' # notsecret + +client = boto3.client('s3', + endpoint_url=endpoint, + aws_access_key_id=access_key, + aws_secret_access_key=secret_key) + + +# create the source bucket +response = client.create_bucket(Bucket=bucket) +print(response) + +# create the target bucket +response = client.create_bucket(Bucket=target_bucket) +print(response) + +bucket_logging_conf = {'LoggingEnabled': { + 'TargetBucket': target_bucket, + 'TargetPrefix': 'log/', + 'TargetObjectKeyFormat': { + 'SimplePrefix': {} + }, + 'ObjectRollTime': 60, + 'LoggingType': 'Journal', + "Filter": { + "Key": { + "FilterRules": + [ + { + "Name": "prefix", + "Value": "myfile" + } + ] + } + } + } +} + +response = client.put_bucket_logging(Bucket=bucket, BucketLoggingStatus=bucket_logging_conf) +print(response) + +response = client.get_bucket_logging(Bucket=bucket) +print(response) + diff --git a/examples/rgw/boto3/head_bucket_stats.py b/examples/rgw/boto3/head_bucket_stats.py new file mode 100755 index 00000000000..1de40d63f4a --- /dev/null +++ b/examples/rgw/boto3/head_bucket_stats.py @@ -0,0 +1,27 @@ +#!/usr/bin/python + +import boto3 +import sys + +if len(sys.argv) != 2: + print('Usage: ' + sys.argv[0] + ' <bucket>') + sys.exit(1) + +# bucket name as first argument +bucketname = sys.argv[1] + +# endpoint and keys from vstart +endpoint = 'http://127.0.0.1:8000' +access_key='0555b35654ad1656d804' +secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' + +client = boto3.client('s3', + endpoint_url=endpoint, + aws_access_key_id=access_key, + aws_secret_access_key=secret_key) + +# reading bucket stats via HeadBucket + +response = client.head_bucket(Bucket=bucketname, ReadStats=True) + +print('Objects:', response['ObjectCount'], 'Bytes:', response['BytesUsed']) diff --git a/examples/rgw/boto3/post_bucket_logging.py b/examples/rgw/boto3/post_bucket_logging.py new file mode 100644 index 00000000000..130fc53b50a --- /dev/null +++ b/examples/rgw/boto3/post_bucket_logging.py @@ -0,0 +1,23 @@ +import boto3 +import sys + + +if len(sys.argv) == 2: + # bucket name as first argument + bucketname = sys.argv[1] +else: + print('Usage: ' + sys.argv[0] + ' <bucket>') + sys.exit(1) + +# endpoint and keys from vstart +endpoint = 'http://127.0.0.1:8000/'+bucketname +access_key='0555b35654ad1656d804' +secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q==' + +client = boto3.client('s3', + endpoint_url=endpoint, + aws_access_key_id=access_key, + aws_secret_access_key=secret_key) + +# flushing the logs for bucket logging +print(client.post_bucket_logging(Bucket=bucketname)) diff --git a/examples/rgw/boto3/service-2.sdk-extras.json b/examples/rgw/boto3/service-2.sdk-extras.json index 46fef1abdbb..4618543d61b 100644 --- a/examples/rgw/boto3/service-2.sdk-extras.json +++ b/examples/rgw/boto3/service-2.sdk-extras.json @@ -13,6 +13,17 @@ "documentationUrl":"https://docs.ceph.com/docs/master/radosgw/s3/bucketops/#delete-notification", "documentation":"<p>Deletes the notification configuration from the bucket.</p>" }, + "PostBucketLogging":{ + "name":"PostBucketLogging", + "http":{ + "method":"POST", + "requestUri":"/{Bucket}?logging", + "responseCode":201 + }, + "input":{"shape":"PostBucketLoggingRequest"}, + "documentationUrl":"https://docs.ceph.com/docs/master/radosgw/s3/bucketops/#post-bucket-logging", + "documentation":"<p>Flushes the logging objects of the buckets.</p>" + }, "GetUsageStats":{ "name":"GetUsageStats", "http":{ @@ -146,6 +157,18 @@ } } }, + "PostBucketLoggingRequest":{ + "type":"structure", + "required":["Bucket"], + "members":{ + "Bucket":{ + "shape":"BucketName", + "documentation":"<p>Name of the bucket to flush its logging objects.</p>", + "location":"uri", + "locationName":"Bucket" + } + } + }, "FilterRule":{ "type":"structure", "members":{ @@ -235,24 +258,157 @@ "UsageStatsSummary": { "type": "structure", "members": { - "QuotaMaxBytes":{"shape":"QuotaMaxBytes"}, - "QuotaMaxBuckets":{"shape": "QuotaMaxBuckets"}, - "QuotaMaxObjCount":{"shape":"QuotaMaxObjCount"}, - "QuotaMaxBytesPerBucket":{"shape":"QuotaMaxBytesPerBucket"}, + "QuotaMaxBytes":{"shape":"QuotaMaxBytes"}, + "QuotaMaxBuckets":{"shape": "QuotaMaxBuckets"}, + "QuotaMaxObjCount":{"shape":"QuotaMaxObjCount"}, + "QuotaMaxBytesPerBucket":{"shape":"QuotaMaxBytesPerBucket"}, "QuotaMaxObjCountPerBucket":{"shape":"QuotaMaxObjCountPerBucket"}, - "TotalBytes":{"shape":"TotalBytes"}, + "TotalBytes":{"shape":"TotalBytes"}, "TotalBytesRounded":{"shape":"TotalBytesRounded"}, "TotalEntries":{"shape":"TotalEntries"} } }, "QuotaMaxBytes":{"type":"integer"}, - "QuotaMaxBuckets":{"type": "integer"}, - "QuotaMaxObjCount":{"type":"integer"}, - "QuotaMaxBytesPerBucket":{"type":"integer"}, - "QuotaMaxObjCountPerBucket":{"type":"integer"}, - "TotalBytesRounded":{"type":"integer"}, + "QuotaMaxBuckets":{"type": "integer"}, + "QuotaMaxObjCount":{"type":"integer"}, + "QuotaMaxBytesPerBucket":{"type":"integer"}, + "QuotaMaxObjCountPerBucket":{"type":"integer"}, + "TotalBytesRounded":{"type":"integer"}, "TotalBytes":{"type":"integer"}, - "TotalEntries":{"type":"integer"} + "TotalEntries":{"type":"integer"}, + "LoggingEnabled":{ + "type":"structure", + "required":[ + "TargetBucket", + "TargetPrefix" + ], + "members":{ + "TargetBucket":{ + "shape":"TargetBucket", + "documentation":"<p>Specifies the bucket where you want to store server access logs. You can have your logs delivered to any bucket that you own. You can also configure multiple buckets to deliver their logs to the same target bucket. In this case, you should choose a different <code>TargetPrefix</code> for each source bucket so that the delivered log files can be distinguished by key.</p>" + }, + "TargetGrants":{ + "shape":"TargetGrants", + "documentation":"<p>Container for granting information.</p> <p>Should be used when the write permissions to the tagert bucket should eb different than the permissions of the user performing the operation thta needs to be logged. This is usually used in cased of batched logging. see: <code>RecordBatchSize</code>.</p>" + }, + "TargetPrefix":{ + "shape":"TargetPrefix", + "documentation":"<p>A prefix for all log object keys. If you store log files from multiple buckets in a single bucket, you can use a prefix to distinguish which log files came from which bucket.</p>" + }, + "TargetObjectKeyFormat":{ + "shape":"TargetObjectKeyFormat", + "documentation":"<p>key format for log objects.</p>" + }, + "ObjectRollTime":{ + "shape":"ObjectRollTime", + "documentation":"<p>time in seconds to move the log object to the target bucket and start another log object.</p>" + }, + "LoggingType":{ + "shape":"LoggingType", + "documentation":"<p>use Standard log type to log all bucket operations i nthe standard format. use Journal log type to log only creations and deletion of objects in more compact format.</p>" + }, + "RecordsBatchSize":{ + "shape":"RecordsBatchSize", + "documentation":"indicates how many records to batch in memory before writing to the object. if set to zero, records are written syncronously to the object. if <code>ObjectRollTime</code>e is reached, the batch of records will be written to the object regardless of the number of records. </p>" + }, + "Filter":{ + "shape":"LoggingConfigurationFilter", + "documentation":"<p>A filter for all log object. Filter for the object by its key (prefix, suffix and regex).</p>" + } + }, + "documentation":"<p>Describes where logs are stored the prefix assigned to all log object keys for a bucket, and their format. also, the level the delivery guarantee of the records.</p>" + }, + "TargetObjectKeyFormat":{ + "type":"structure", + "members":{ + "SimplePrefix":{ + "shape":"SimplePrefix", + "documentation":"<p>To use the simple format for S3 keys for log objects. To specify SimplePrefix format, set SimplePrefix to {}.</p>", + "locationName":"SimplePrefix" + }, + "PartitionedPrefix":{ + "shape":"PartitionedPrefix", + "documentation":"<p>Partitioned S3 key for log objects.</p>", + "locationName":"PartitionedPrefix" + } + }, + "documentation":"<p>Key format for log objects. Only one format, PartitionedPrefix or SimplePrefix, is allowed.</p>" + }, + "SimplePrefix":{ + "type":"structure", + "members":{ + }, + "documentation":"<p>To use simple format for S3 keys for log objects, set SimplePrefix to an empty object.</p> <p> <code>[DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]</code> </p>", + "locationName":"SimplePrefix" + }, + "PartitionDateSource":{ + "type":"string", + "enum":[ + "EventTime", + "DeliveryTime" + ] + }, + "PartitionedPrefix":{ + "type":"structure", + "members":{ + "PartitionDateSource":{ + "shape":"PartitionDateSource", + "documentation":"<p>Specifies the partition date source for the partitioned prefix. PartitionDateSource can be EventTime or DeliveryTime.</p>" + } + }, + "documentation":"<p>Amazon S3 keys for log objects are partitioned in the following format:</p> <p> <code>[DestinationPrefix][SourceAccountId]/[SourceRegion]/[SourceBucket]/[YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString]</code> </p> <p>PartitionedPrefix defaults to EventTime delivery when server access logs are delivered.</p>", + "locationName":"PartitionedPrefix" + }, + "ObjectRollTime":{"type":"integer"}, + "RecordsBatchSize":{"type":"integer"}, + "LoggingType":{ + "type":"string", + "enum": [ + "Standard", + "Journal" + ] + }, + "LoggingConfigurationFilter":{ + "type":"structure", + "members":{ + "Key":{ + "shape":"S3KeyFilter", + "documentation":"<p/>", + "locationName":"S3Key" + } + }, + "documentation":"<p>A filter for all log object. Filter for the object by its key (prefix, suffix and regex).</p>", + "locationName":"Filter" + }, + "HeadBucketRequest": { + "members": { + "ReadStats":{ + "shape":"ReadStats", + "documentation":"<p>Read additional usage statistics for <code>ObjectCount</code> and <code>BytesUsed</code> in the response.</p> <note> <p>This request parameter is a Ceph RGW extension.</p> </note>", + "location":"querystring", + "locationName":"read-stats" + } + } + }, + "HeadBucketOutput":{ + "members":{ + "ObjectCount":{ + "shape":"ObjectCount", + "documentation": "<p>Total number of objects/versions in the bucket.</p>", + "location": "header", + "locationName": "x-rgw-object-count" + }, + "BytesUsed":{ + "shape":"BytesUsed", + "documentation": "<p>Total size in bytes of all objects/versions in the bucket.</p>", + "location": "header", + "locationName": "x-rgw-bytes-used" + } + } + }, + "ReadStats":{"type":"boolean"}, + "ObjectCount":{"type":"integer"}, + "BytesUsed":{"type":"integer"} }, "documentation":"<p/>" } diff --git a/monitoring/ceph-mixin/config.libsonnet b/monitoring/ceph-mixin/config.libsonnet index a15b88422fc..e917b4c2dac 100644 --- a/monitoring/ceph-mixin/config.libsonnet +++ b/monitoring/ceph-mixin/config.libsonnet @@ -9,12 +9,12 @@ CephNodeNetworkPacketDropsPerSec: 10, CephRBDMirrorImageTransferBandwidthThreshold: 0.8, CephRBDMirrorImagesPerDaemonThreshold: 100, - NVMeoFMaxGatewaysPerGroup: 4, - NVMeoFMaxGatewaysPerCluster: 4, + NVMeoFMaxGatewaysPerGroup: 8, + NVMeoFMaxGatewaysPerCluster: 32, NVMeoFHighGatewayCPU: 80, NVMeoFMaxSubsystemsPerGateway: 128, - NVMeoFMaxNamespaces: 1024, - NVMeoFHighClientCount: 32, + NVMeoFMaxNamespaces: 2048, + NVMeoFHighClientCount: 128, NVMeoFHighHostCPU: 80, // // Read/Write latency is defined in ms diff --git a/monitoring/ceph-mixin/dashboards/rgw.libsonnet b/monitoring/ceph-mixin/dashboards/rgw.libsonnet index 79a4b7a14eb..c0c548b79c8 100644 --- a/monitoring/ceph-mixin/dashboards/rgw.libsonnet +++ b/monitoring/ceph-mixin/dashboards/rgw.libsonnet @@ -298,7 +298,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_get_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_get_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -314,7 +314,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_put_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_put_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -331,7 +331,7 @@ local g = import 'grafonnet/grafana.libsonnet'; sum by (rgw_host) ( label_replace( rate(ceph_rgw_req{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ) @@ -351,7 +351,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_get_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_get_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -385,7 +385,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace(sum by (instance_id) ( rate(ceph_rgw_op_get_obj_bytes{%(matchers)s}[$__rate_interval]) + rate(ceph_rgw_op_put_obj_bytes{%(matchers)s}[$__rate_interval])) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), @@ -404,7 +404,7 @@ local g = import 'grafonnet/grafana.libsonnet'; label_replace( rate(ceph_rgw_op_put_obj_lat_sum{%(matchers)s}[$__rate_interval]) / rate(ceph_rgw_op_put_obj_lat_count{%(matchers)s}[$__rate_interval]) * - on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{%(matchers)s}, + on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~"$rgw_servers", %(matchers)s}, "rgw_host", "$1", "ceph_daemon", "rgw.(.*)" ) ||| % $.matchers(), diff --git a/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json b/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json index 5e185b63b7f..5bf8279c27c 100644 --- a/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json +++ b/monitoring/ceph-mixin/dashboards_out/radosgw-overview.json @@ -108,14 +108,14 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "GET {{rgw_host}}", "refId": "A" }, { - "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "PUT {{rgw_host}}", @@ -210,7 +210,7 @@ "steppedLine": false, "targets": [ { - "expr": "sum by (rgw_host) (\n label_replace(\n rate(ceph_rgw_req{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n )\n)\n", + "expr": "sum by (rgw_host) (\n label_replace(\n rate(ceph_rgw_req{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n )\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -305,7 +305,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_get_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_get_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -502,7 +502,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(sum by (instance_id) (\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval]) +\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval])) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(sum by (instance_id) (\n rate(ceph_rgw_op_get_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval]) +\n rate(ceph_rgw_op_put_obj_bytes{cluster=~\"$cluster\", }[$__rate_interval])) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", @@ -597,7 +597,7 @@ "steppedLine": false, "targets": [ { - "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", + "expr": "label_replace(\n rate(ceph_rgw_op_put_obj_lat_sum{cluster=~\"$cluster\", }[$__rate_interval]) /\n rate(ceph_rgw_op_put_obj_lat_count{cluster=~\"$cluster\", }[$__rate_interval]) *\n on (instance_id) group_left (ceph_daemon) ceph_rgw_metadata{ceph_daemon=~\"$rgw_servers\", cluster=~\"$cluster\", },\n \"rgw_host\", \"$1\", \"ceph_daemon\", \"rgw.(.*)\"\n)\n", "format": "time_series", "intervalFactor": 1, "legendFormat": "{{rgw_host}}", diff --git a/monitoring/ceph-mixin/prometheus_alerts.libsonnet b/monitoring/ceph-mixin/prometheus_alerts.libsonnet index 143e65f20e7..5d1ab49b533 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.libsonnet +++ b/monitoring/ceph-mixin/prometheus_alerts.libsonnet @@ -856,6 +856,16 @@ }, }, { + alert: 'NVMeoFMultipleNamespacesOfRBDImage', + 'for': '1m', + expr: 'count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1', + labels: { severity: 'warning', type: 'ceph_default' }, + annotations: { + summary: 'RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace ', + description: 'Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups.', + }, + }, + { alert: 'NVMeoFTooManyGateways', 'for': '1m', expr: 'count(ceph_nvmeof_gateway_info) by (cluster) > %.2f' % [$._config.NVMeoFMaxGatewaysPerCluster], diff --git a/monitoring/ceph-mixin/prometheus_alerts.yml b/monitoring/ceph-mixin/prometheus_alerts.yml index 3eb8a8db4fa..7c0da4d51a4 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.yml +++ b/monitoring/ceph-mixin/prometheus_alerts.yml @@ -765,20 +765,29 @@ groups: labels: severity: "warning" type: "ceph_default" + - alert: "NVMeoFMultipleNamespacesOfRBDImage" + annotations: + description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups." + summary: "RBD image {{ $labels.pool_name }}/{{ $labels.rbd_name }} cannot be reused for multiple NVMeoF namespace " + expr: "count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1" + for: "1m" + labels: + severity: "warning" + type: "ceph_default" - alert: "NVMeoFTooManyGateways" annotations: - description: "You may create many gateways, but 4 is the tested limit" + description: "You may create many gateways, but 32 is the tested limit" summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}" - expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 4.00" + expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 32.00" for: "1m" labels: severity: "warning" type: "ceph_default" - alert: "NVMeoFMaxGatewayGroupSize" annotations: - description: "You may create many gateways in a gateway group, but 4 is the tested limit" + description: "You may create many gateways in a gateway group, but 8 is the tested limit" summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}" - expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 4.00" + expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 8.00" for: "1m" labels: severity: "warning" @@ -823,7 +832,7 @@ groups: annotations: description: "Although you may continue to create namespaces in {{ $labels.gateway_host }}, the configuration may not be supported" summary: "The number of namespaces defined to the gateway exceeds supported values on cluster {{ $labels.cluster }}" - expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\"gateway_host\",\"$1\",\"instance\",\"(.*?)(?::.*)?\")) > 1024.00" + expr: "sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,\"gateway_host\",\"$1\",\"instance\",\"(.*?)(?::.*)?\")) > 2048.00" for: "1m" labels: severity: "warning" @@ -839,9 +848,9 @@ groups: type: "ceph_default" - alert: "NVMeoFHighClientCount" annotations: - description: "The supported limit for clients connecting to a subsystem is 32" + description: "The supported limit for clients connecting to a subsystem is 128" summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}" - expr: "ceph_nvmeof_subsystem_host_count > 32.00" + expr: "ceph_nvmeof_subsystem_host_count > 128.00" for: "1m" labels: severity: "warning" diff --git a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml index a4e63bbcf73..83b4ff80375 100644 --- a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml +++ b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml @@ -2270,6 +2270,54 @@ tests: summary: "wah subsystem has reached its maximum number of namespaces on cluster mycluster" description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to wah" +# NVMeoFMultipleNamespacesOfRBDImage + - interval: 1m + input_series: + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev1", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev1", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev2", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage2"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev2", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage2"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev3", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev3", instance="ceph-nvme-vm2", pool_name="mypool", rbd_name="myimage1"}' + values: '1x10' + - series: 'ceph_nvmeof_bdev_metadata{bdev_name="bdev4", instance="ceph-nvme-vm1", pool_name="mypool", rbd_name="myimage1"}' # bdev with no ns + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="1", bdev_name="bdev1", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="1", bdev_name="bdev1", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="2", bdev_name="bdev2", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn1", nsid="2", bdev_name="bdev2", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn2", nsid="1", bdev_name="bdev3", instance="ceph-nvme-vm1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="nqn2", nsid="1", bdev_name="bdev3", instance="ceph-nvme-vm2", cluster="mycluster"}' + values: '1x10' + promql_expr_test: + - expr: count by(pool_name, rbd_name) (count by(bdev_name, pool_name, rbd_name) (ceph_nvmeof_bdev_metadata and on (bdev_name) ceph_nvmeof_subsystem_namespace_metadata)) > 1 + eval_time: 1m + exp_samples: + - labels: '{pool_name="mypool", rbd_name="myimage1"}' + value: 2 + alert_rule_test: + - eval_time: 5m + alertname: NVMeoFMultipleNamespacesOfRBDImage + exp_alerts: + - exp_labels: + pool_name: mypool + rbd_name: myimage1 + severity: warning + type: ceph_default + exp_annotations: + summary: "RBD image mypool/myimage1 cannot be reused for multiple NVMeoF namespace " + description: "Each NVMeoF namespace must have a unique RBD pool and image, across all different gateway groups." + # NVMeoFTooManyGateways - interval: 1m input_series: @@ -2283,12 +2331,69 @@ tests: values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.5",cluster="mycluster"}' values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.6",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.7",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.8",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.9",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.10",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.11",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.12",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.13",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.14",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.15",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.16",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.17",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.18",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.19",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.20",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.21",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.22",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.23",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.24",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.25",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.26",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.27",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.28",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.29",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.30",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.31",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.32",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.33",cluster="mycluster"}' + values: '1+0x20' + promql_expr_test: - - expr: count(ceph_nvmeof_gateway_info) by (cluster) > 4.00 + - expr: count(ceph_nvmeof_gateway_info) by (cluster) > 32.00 eval_time: 1m exp_samples: - labels: '{cluster="mycluster"}' - value: 5 + value: 33 alert_rule_test: - eval_time: 5m alertname: NVMeoFTooManyGateways @@ -2299,7 +2404,7 @@ tests: type: ceph_default exp_annotations: summary: "Max supported gateways exceeded on cluster mycluster" - description: "You may create many gateways, but 4 is the tested limit" + description: "You may create many gateways, but 32 is the tested limit" # NVMeoFMaxGatewayGroupSize - interval: 1m @@ -2314,16 +2419,24 @@ tests: values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.12",cluster="mycluster"}' values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.10",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.14",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.11",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.13",cluster="mycluster"}' + values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4",cluster="mycluster"}' values: '1+0x20' - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5",cluster="mycluster"}' values: '1+0x20' promql_expr_test: - - expr: count(ceph_nvmeof_gateway_info) by (cluster, group) > 4.00 + - expr: count(ceph_nvmeof_gateway_info) by (cluster, group) > 8.00 eval_time: 1m exp_samples: - labels: '{cluster="mycluster",group="group-1"}' - value: 5 + value: 9 alert_rule_test: - eval_time: 5m alertname: NVMeoFMaxGatewayGroupSize @@ -2335,7 +2448,7 @@ tests: type: ceph_default exp_annotations: summary: "Max gateways within a gateway group (group-1) exceeded on cluster mycluster" - description: "You may create many gateways in a gateway group, but 4 is the tested limit" + description: "You may create many gateways in a gateway group, but 8 is the tested limit" # NVMeoFSingleGatewayGroup - interval: 1m @@ -2719,12 +2832,14 @@ tests: values: '200+0x10' - series: 'ceph_nvmeof_subsystem_namespace_count{instance="node-1:10008",nqn="nqn10",cluster="mycluster"}' values: '200+0x10' + - series: 'ceph_nvmeof_subsystem_namespace_count{instance="node-1:10008",nqn="nqn11",cluster="mycluster"}' + values: '200+0x10' promql_expr_test: - - expr: sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,"gateway_host","$1","instance","(.*):.*")) > 1024 + - expr: sum by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_namespace_count,"gateway_host","$1","instance","(.*):.*")) > 2048 eval_time: 1m exp_samples: - labels: '{gateway_host="node-1", cluster="mycluster"}' - value: 2000 + value: 2200 alert_rule_test: - eval_time: 5m alertname: NVMeoFTooManyNamespaces @@ -2767,15 +2882,15 @@ tests: - interval: 1m input_series: - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn1",cluster="mycluster"}' - values: '2 2 2 4 4 8 8 8 10 10 20 20 32 34 34 38 38 40 44 44' + values: '2 4 8 10 20 30 40 50 62 74 80 95 100 110 130 130 130 130 130 130' - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn2",cluster="mycluster"}' - values: '2 2 2 8 8 8 16 16 16 16 16 16 16 16 16 16 16 16 16 16' + values: '2 8 16 16 16 16 16 16 16 16 20 20 32 34 34 36 37 37 37 37' promql_expr_test: - - expr: ceph_nvmeof_subsystem_host_count > 32.00 + - expr: ceph_nvmeof_subsystem_host_count > 128.00 eval_time: 15m exp_samples: - labels: '{__name__="ceph_nvmeof_subsystem_host_count",nqn="nqn1",cluster="mycluster"}' - value: 38 + value: 130 alert_rule_test: - eval_time: 20m alertname: NVMeoFHighClientCount @@ -2787,7 +2902,7 @@ tests: type: ceph_default exp_annotations: summary: "The number of clients connected to nqn1 is too high on cluster mycluster" - description: "The supported limit for clients connecting to a subsystem is 32" + description: "The supported limit for clients connecting to a subsystem is 128" # NVMeoFMissingListener - interval: 1m diff --git a/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature b/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature index 8d96dcdd610..a34d5759437 100644 --- a/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature +++ b/monitoring/ceph-mixin/tests_dashboards/features/radosgw_overview.feature @@ -7,6 +7,7 @@ Scenario: "Test Average GET Latencies" | ceph_rgw_op_get_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 20 60 80 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `Average GET/PUT Latencies by RGW Instance` with legend `GET {{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1.5 | @@ -18,6 +19,7 @@ Scenario: "Test Average PUT Latencies" | ceph_rgw_op_put_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 10 30 50 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `Average GET/PUT Latencies by RGW Instance` with legend `PUT {{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1 | @@ -28,6 +30,7 @@ Scenario: "Test Total Requests/sec by RGW Instance" | ceph_rgw_req{instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 10 50 100 | | ceph_rgw_metadata{ceph_daemon="rgw.1", hostname="localhost", instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.1` Then Grafana panel `Total Requests/sec by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {rgw_host="1"} | 1.5 | @@ -39,6 +42,7 @@ Scenario: "Test GET Latencies by RGW Instance" | ceph_rgw_op_get_obj_lat_count{instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 20 60 80 | | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `GET Latencies by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1.5 | @@ -71,6 +75,7 @@ Scenario: "Test Bandwidth by RGW Instance" | ceph_rgw_metadata{ceph_daemon="rgw.1", hostname="localhost", instance="127.0.0.1", instance_id="92806566", job="ceph", cluster="mycluster"} | 1 1 1 | When evaluation time is `1m` And interval is `30s` + And variable `rgw_servers` is `rgw.1` Then Grafana panel `Bandwidth by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.1", instance_id="92806566", rgw_host="1"} | 2.25 | @@ -83,6 +88,7 @@ Scenario: "Test PUT Latencies by RGW Instance" | ceph_rgw_metadata{ceph_daemon="rgw.foo", hostname="localhost", instance="127.0.0.1", instance_id="58892247", job="ceph", cluster="mycluster"} | 1 1 1 | When evaluation time is `1m` And interval is `30s` + And variable `rgw_servers` is `rgw.foo` Then Grafana panel `PUT Latencies by RGW Instance` with legend `{{rgw_host}}` shows: | metrics | values | | {ceph_daemon="rgw.foo", instance="127.0.0.1", instance_id="58892247", job="ceph", rgw_host="foo", cluster="mycluster"} | 1 | diff --git a/qa/config/crimson_bluestore.yaml b/qa/config/crimson_bluestore.yaml new file mode 100644 index 00000000000..d5ba487b9bf --- /dev/null +++ b/qa/config/crimson_bluestore.yaml @@ -0,0 +1,25 @@ +overrides: + ceph: + fs: xfs + conf: + osd: + # crimson's osd objectstore option + crimson osd objectstore: bluestore + debug alienstore: 20 + bluestore block size: 96636764160 + debug bluestore: 20 + debug bluefs: 20 + debug rocksdb: 10 + bluestore compression mode: aggressive + bluestore fsck on mount: true + bluestore compression algorithm: snappy + # lower the full ratios since we can fill up a 100gb osd so quickly + mon osd full ratio: .9 + mon osd backfillfull_ratio: .85 + mon osd nearfull ratio: .8 + osd failsafe full ratio: .95 + bluestore rocksdb cf: false + log to stderr: true + err to stderr: true + log flush on exit: true + log to file: false diff --git a/qa/config/crimson_qa_overrides.yaml b/qa/config/crimson_qa_overrides.yaml index fa8f49a4986..a10c59d77cc 100644 --- a/qa/config/crimson_qa_overrides.yaml +++ b/qa/config/crimson_qa_overrides.yaml @@ -9,6 +9,7 @@ overrides: osd pool default crimson: true osd: crimson osd obc lru size: 10 + debug ms: 20 flavor: crimson workunit: env: diff --git a/qa/config/seastore.yaml b/qa/config/crimson_seastore.yaml index 6158563eedf..d1919456ab1 100644 --- a/qa/config/seastore.yaml +++ b/qa/config/crimson_seastore.yaml @@ -1,13 +1,13 @@ overrides: ceph: - fs: xfs conf: osd: - osd objectstore: seastore + # crimson's osd objectstore option + crimson osd objectstore: seastore debug seastore: 20 debug seastore onode: 20 debug seastore odata: 20 - debug seastore ompap: 20 + debug seastore omap: 20 debug seastore tm: 20 debug seastore t: 20 debug seastore cleaner: 20 diff --git a/qa/crontab/teuthology-cronjobs b/qa/crontab/teuthology-cronjobs index c979e5b105f..c558a1382ef 100644 --- a/qa/crontab/teuthology-cronjobs +++ b/qa/crontab/teuthology-cronjobs @@ -52,7 +52,6 @@ TEUTHOLOGY_SUITE_ARGS="--non-interactive --newest=100 --ceph-repo=https://git.ce 00 05 * * 0,2,4 $CW $SS 1 --ceph main --suite smoke -p 100 --force-priority 08 05 * * 0 $CW $SS 1 --ceph squid --suite smoke -p 100 --force-priority 16 05 * * 0 $CW $SS 1 --ceph reef --suite smoke -p 100 --force-priority -24 05 * * 0 $CW $SS 1 --ceph quincy --suite smoke -p 100 --force-priority ## ********** windows tests on main branch - weekly # 00 03 * * 1 CEPH_BRANCH=main; MACHINE_NAME=smithi; $CW teuthology-suite -v -c $CEPH_BRANCH -n 100 -m $MACHINE_NAME -s windows -k distro -e $CEPH_QA_EMAIL @@ -122,7 +121,6 @@ TEUTHOLOGY_SUITE_ARGS="--non-interactive --newest=100 --ceph-repo=https://git.ce 16 00 * * 1 $CW $SS 1 --ceph quincy --suite upgrade-clients/client-upgrade-pacific-quincy --suite-branch pacific -p 820 24 00 * * 1 $CW $SS 120000 --ceph quincy --suite upgrade:octopus-x -p 820 32 00 * * 1 $CW $SS 120000 --ceph quincy --suite upgrade:pacific-x -p 820 -40 00 * * 1 $CW $SS 1 --ceph quincy --suite upgrade/quincy-p2p -p 820 ### upgrade runs for reef release ###### on smithi diff --git a/qa/rgw/s3tests-branch.yaml b/qa/rgw/s3tests-branch.yaml index ef6819c87e0..8710ce35893 100644 --- a/qa/rgw/s3tests-branch.yaml +++ b/qa/rgw/s3tests-branch.yaml @@ -1,4 +1,4 @@ overrides: s3tests: - force-branch: ceph-master - # git_remote: https://github.com/ceph/ + force-branch: ceph-master + # git_remote: https://github.com/ceph/ diff --git a/qa/standalone/scrub/osd-recovery-scrub.sh b/qa/standalone/scrub/osd-recovery-scrub.sh index 843e9b9901b..7b77a60f35b 100755 --- a/qa/standalone/scrub/osd-recovery-scrub.sh +++ b/qa/standalone/scrub/osd-recovery-scrub.sh @@ -163,7 +163,7 @@ function wait_for_scrub_mod() { fi sleep 1 # are we still the primary? - local current_primary=`bin/ceph pg $pgid query | jq '.acting[0]' ` + local current_primary=`./bin/ceph pg $pgid query | jq '.acting[0]' ` if [ $orig_primary != $current_primary ]; then echo $orig_primary no longer primary for $pgid return 0 @@ -194,7 +194,7 @@ function pg_scrub_mod() { local last_scrub=$(get_last_scrub_stamp $pgid) # locate the primary - local my_primary=`bin/ceph pg $pgid query | jq '.acting[0]' ` + local my_primary=`./bin/ceph pg $pgid query | jq '.acting[0]' ` local recovery=false ceph pg scrub $pgid #ceph --format json pg dump pgs | jq ".pg_stats | .[] | select(.pgid == \"$pgid\") | .state" diff --git a/qa/standalone/scrub/osd-scrub-repair.sh b/qa/standalone/scrub/osd-scrub-repair.sh index 491e46603f7..6dd5b10ae8f 100755 --- a/qa/standalone/scrub/osd-scrub-repair.sh +++ b/qa/standalone/scrub/osd-scrub-repair.sh @@ -5833,7 +5833,7 @@ function TEST_periodic_scrub_replicated() { flush_pg_stats # Request a regular scrub and it will be done - pg_schedule_scrub $pg + pg_scrub $pg grep -q "Regular scrub request, deep-scrub details will be lost" $dir/osd.${primary}.log || return 1 # deep-scrub error is no longer present diff --git a/qa/standalone/scrub/osd-scrub-test.sh b/qa/standalone/scrub/osd-scrub-test.sh index 8015e023bdd..385479258f2 100755 --- a/qa/standalone/scrub/osd-scrub-test.sh +++ b/qa/standalone/scrub/osd-scrub-test.sh @@ -544,6 +544,9 @@ function TEST_dump_scrub_schedule() { --osd_op_queue=wpq \ --osd_stats_update_period_not_scrubbing=1 \ --osd_stats_update_period_scrubbing=1 \ + --osd_scrub_retry_after_noscrub=1 \ + --osd_scrub_retry_pg_state=2 \ + --osd_scrub_retry_delay=2 \ --osd_scrub_sleep=0.2" for osd in $(seq 0 $(expr $OSDS - 1)) @@ -600,17 +603,16 @@ function TEST_dump_scrub_schedule() { declare -A expct_dmp_duration=( ['dmp_last_duration']="0" ['dmp_last_duration_neg']="not0" ) wait_any_cond $pgid 10 $saved_last_stamp expct_dmp_duration "WaitingAfterScrub_dmp " sched_data || return 1 - sleep 2 - # # step 2: set noscrub and request a "periodic scrub". Watch for the change in the 'is the scrub # scheduled for the future' value # - ceph tell osd.* config set osd_shallow_scrub_chunk_max "3" || return 1 - ceph tell osd.* config set osd_scrub_sleep "2.0" || return 1 ceph osd set noscrub || return 1 sleep 2 + ceph tell osd.* config set osd_shallow_scrub_chunk_max "3" || return 1 + ceph tell osd.* config set osd_scrub_sleep "2.0" || return 1 + sleep 8 saved_last_stamp=${sched_data['query_last_stamp']} ceph tell $pgid schedule-scrub @@ -683,6 +685,234 @@ function TEST_pg_dump_objects_scrubbed() { teardown $dir || return 1 } +function wait_initial_scrubs() { + local -n pg_to_prim_dict=$1 + local extr_dbg=1 # note: 3 and above leave some temp files around + + # set a long schedule for the periodic scrubs. Wait for the + # initial 'no previous scrub is known' scrubs to finish for all PGs. + ceph tell osd.* config set osd_scrub_min_interval 7200 + ceph tell osd.* config set osd_deep_scrub_interval 14400 + ceph tell osd.* config set osd_max_scrubs 32 + ceph tell osd.* config set osd_scrub_sleep 0 + ceph tell osd.* config set osd_shallow_scrub_chunk_max 10 + ceph tell osd.* config set osd_scrub_chunk_max 10 + + for pg in "${!pg_to_prim_dict[@]}"; do + (( extr_dbg >= 1 )) && echo "Scheduling initial scrub for $pg" + ceph tell $pg scrub || return 1 + done + + sleep 1 + (( extr_dbg >= 1 )) && ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' + + tout=20 + while [ $tout -gt 0 ] ; do + sleep 0.5 + (( extr_dbg >= 2 )) && ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' + not_done=$(ceph pg dump pgs --format=json-pretty | \ + jq '.pg_stats | map(select(.last_scrub_duration == 0)) | map({pgid: .pgid, last_scrub_duration: .last_scrub_duration})' | wc -l ) + # note that we should ignore a header line + if [ "$not_done" -le 1 ]; then + break + fi + not_done=$(( (not_done - 2) / 4 )) + echo "Still waiting for $not_done PGs to finish initial scrubs (timeout $tout)" + tout=$((tout - 1)) + done + (( tout == 0 )) && return 1 + return 0 +} + + +# Whenever a PG is being scrubbed at a regular, periodic, urgency, and is queued +# for its replicas: +# if the operator is requesting a scrub of the same PG, the operator's request +# should trigger an abort of the ongoing scrub. +# +# The test process: +# - a periodic scrub is initiated of a PG. That scrub is set to be a very slow one. +# - a second PG, which shares some of its replicas, is intrcuted to be scrubbed. That one +# should be stuck in replica reservation. We will verify that. +# - now - the operator is requesting that second PG to be scrubbed. The original (pending) +# scrub should be aborted. We would check for: +# - the new, operator's scrub to be scheduled +# - the replicas' reservers to be released +function TEST_abort_periodic_for_operator() { + local dir=$1 + local -A cluster_conf=( + ['osds_num']="5" + ['pgs_in_pool']="16" + ['pool_name']="test" + ) + local extr_dbg=1 # note: 3 and above leave some temp files around + + standard_scrub_wpq_cluster "$dir" cluster_conf 3 || return 1 + local poolid=${cluster_conf['pool_id']} + local poolname=${cluster_conf['pool_name']} + echo "Pool: $poolname : $poolid" + + #turn off '-x' (but remember previous state) + local saved_echo_flag=${-//[^x]/} + set +x + + # fill the pool with some data + TESTDATA="testdata.$$" + dd if=/dev/urandom of=$TESTDATA bs=320 count=1 + for i in $( seq 1 256 ) + do + rados -p "$poolname" put "obj${i}" $TESTDATA 2>/dev/null 1>/dev/null + done + rm -f $TESTDATA + if [[ -n "$saved_echo_flag" ]]; then set -x; fi + + # create the dictionary of the PGs in the pool + declare -A pg_pr + declare -A pg_ac + declare -A pg_po + build_pg_dicts "$dir" pg_pr pg_ac pg_po "-" + (( extr_dbg >= 2 )) && echo "PGs table:" + for pg in "${!pg_pr[@]}"; do + (( extr_dbg >= 2 )) && echo "Got: $pg: ${pg_pr[$pg]} ( ${pg_ac[$pg]} ) ${pg_po[$pg]}" + done + + wait_initial_scrubs pg_pr || return 1 + + # limit all OSDs to one scrub at a time + ceph tell osd.* config set osd_max_scrubs 1 + ceph tell osd.* config set osd_stats_update_period_not_scrubbing 1 + + # configure for slow scrubs + ceph tell osd.* config set osd_scrub_sleep 3 + ceph tell osd.* config set osd_shallow_scrub_chunk_max 2 + ceph tell osd.* config set osd_scrub_chunk_max 2 + (( extr_dbg >= 2 )) && ceph tell osd.2 dump_scrub_reservations --format=json-pretty + + # the first PG to work with: + local pg1="1.0" + # and another one, that shares its primary, and at least one more active set member + local pg2="" + for pg in "${!pg_pr[@]}"; do + if [[ "${pg_pr[$pg]}" == "${pg_pr[$pg1]}" ]]; then + local -i common=0 + count_common_active $pg $pg1 pg_ac common + if [[ $common -gt 1 ]]; then + pg2=$pg + break + fi + fi + done + if [[ -z "$pg2" ]]; then + # \todo handle the case when no such PG is found + echo "No PG found with the same primary as $pg1" + return 1 + fi + + # the common primary is allowed two concurrent scrubs + ceph tell osd."${pg_pr[$pg1]}" config set osd_max_scrubs 2 + echo "The two PGs to manipulate are $pg1 and $pg2" + + set_query_debug "$pg1" + # wait till the information published by pg1 is updated to show it as + # not being scrubbed + local is_act + for i in $( seq 1 3 ) + do + is_act=$(ceph pg "$pg1" query | jq '.scrubber.active') + if [[ "$is_act" = "false" ]]; then + break + fi + echo "Still waiting for pg $pg1 to finish scrubbing" + sleep 0.7 + done + ceph pg dump pgs + if [[ "$is_act" != "false" ]]; then + ceph pg "$pg1" query + echo "PG $pg1 appears to be still scrubbing" + return 1 + fi + sleep 0.5 + + echo "Initiating a periodic scrub of $pg1" + (( extr_dbg >= 2 )) && ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + ceph tell $pg1 schedule-deep-scrub || return 1 + sleep 1 + (( extr_dbg >= 2 )) && ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + for i in $( seq 1 14 ) + do + sleep 0.5 + stt=$(ceph pg "$pg1" query | jq '.scrubber') + is_active=$(echo $stt | jq '.active') + is_reserving_replicas=$(echo $stt | jq '.is_reserving_replicas') + if [[ "$is_active" = "true" && "$is_reserving_replicas" = "false" ]]; then + break + fi + echo "Still waiting for pg $pg1 to start scrubbing: $stt" + done + if [[ "$is_active" != "true" || "$is_reserving_replicas" != "false" ]]; then + ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + echo "The scrub is not active or is reserving replicas" + return 1 + fi + (( extr_dbg >= 2 )) && ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + + # PG 1 is scrubbing, and has reserved the replicas - soem of which are shared + # by PG 2. As the max-scrubs was set to 1, that should prevent PG 2 from + # reserving its replicas. + + (( extr_dbg >= 1 )) && ceph tell osd.* dump_scrub_reservations --format=json-pretty + + # now - the 2'nd scrub - which should be blocked on reserving + set_query_debug "$pg2" + ceph tell "$pg2" schedule-deep-scrub + sleep 0.5 + (( extr_dbg >= 2 )) && echo "====================================================================================" + (( extr_dbg >= 2 )) && ceph pg "$pg2" query -f json-pretty | jq '.scrubber' + (( extr_dbg >= 2 )) && ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + sleep 1 + (( extr_dbg >= 2 )) && echo "====================================================================================" + (( extr_dbg >= 2 )) && ceph pg "$pg2" query -f json-pretty | jq '.scrubber' + (( extr_dbg >= 2 )) && ceph pg "$pg1" query -f json-pretty | jq '.scrubber' + + # make sure pg2 scrub is stuck in the reserving state + local stt2=$(ceph pg "$pg2" query | jq '.scrubber') + local pg2_is_reserving + pg2_is_reserving=$(echo $stt2 | jq '.is_reserving_replicas') + if [[ "$pg2_is_reserving" != "true" ]]; then + echo "The scheduled scrub for $pg2 should have been stuck" + ceph pg dump pgs + return 1 + fi + + # now - issue an operator-initiated scrub on pg2. + # The periodic scrub should be aborted, and the operator-initiated scrub should start. + echo "Instructing $pg2 to perform a high-priority scrub" + ceph tell "$pg2" scrub + for i in $( seq 1 10 ) + do + sleep 0.5 + stt2=$(ceph pg "$pg2" query | jq '.scrubber') + pg2_is_active=$(echo $stt2 | jq '.active') + pg2_is_reserving=$(echo $stt2 | jq '.is_reserving_replicas') + if [[ "$pg2_is_active" = "true" && "$pg2_is_reserving" != "true" ]]; then + break + fi + echo "Still waiting: $stt2" + done + + if [[ "$pg2_is_active" != "true" || "$pg2_is_reserving" = "true" ]]; then + echo "The high-priority scrub for $pg2 is not active or is reserving replicas" + return 1 + fi + echo "Done" +} + + + main osd-scrub-test "$@" # Local Variables: diff --git a/qa/standalone/scrub/scrub-helpers.sh b/qa/standalone/scrub/scrub-helpers.sh index 49b8346b8d2..dd37b643e08 100644 --- a/qa/standalone/scrub/scrub-helpers.sh +++ b/qa/standalone/scrub/scrub-helpers.sh @@ -240,8 +240,8 @@ function standard_scrub_cluster() { local saved_echo_flag=${-//[^x]/} set +x - run_mon $dir a --osd_pool_default_size=$OSDS || return 1 - run_mgr $dir x || return 1 + run_mon $dir a --osd_pool_default_size=3 || return 1 + run_mgr $dir x --mgr_stats_period=1 || return 1 local ceph_osd_args="--osd_deep_scrub_randomize_ratio=0 \ --osd_scrub_interval_randomize_ratio=0 \ @@ -249,9 +249,12 @@ function standard_scrub_cluster() { --osd_pool_default_pg_autoscale_mode=off \ --osd_pg_stat_report_interval_max_seconds=1 \ --osd_pg_stat_report_interval_max_epochs=1 \ + --osd_stats_update_period_not_scrubbing=3 \ + --osd_stats_update_period_scrubbing=1 \ --osd_scrub_retry_after_noscrub=5 \ --osd_scrub_retry_pg_state=5 \ --osd_scrub_retry_delay=3 \ + --osd_pool_default_size=3 \ $extra_pars" for osd in $(seq 0 $(expr $OSDS - 1)) @@ -297,6 +300,107 @@ function standard_scrub_wpq_cluster() { } +# Parse the output of a 'pg dump pgs_brief' command and build a set of dictionaries: +# - pg_primary_dict: a dictionary of pgid -> acting_primary +# - pg_acting_dict: a dictionary of pgid -> acting set +# - pg_pool_dict: a dictionary of pgid -> pool +# If the input file is '-', the function will fetch the dump directly from the ceph cluster. +function build_pg_dicts { + local dir=$1 + local -n pg_primary_dict=$2 + local -n pg_acting_dict=$3 + local -n pg_pool_dict=$4 + local infile=$5 + + local extr_dbg=0 # note: 3 and above leave some temp files around + + #turn off '-x' (but remember previous state) + local saved_echo_flag=${-//[^x]/} + set +x + + # if the infile name is '-', fetch the dump directly from the ceph cluster + if [[ $infile == "-" ]]; then + local -r ceph_cmd="ceph pg dump pgs_brief -f=json-pretty" + local -r ceph_cmd_out=$(eval $ceph_cmd) + local -r ceph_cmd_rc=$? + if [[ $ceph_cmd_rc -ne 0 ]]; then + echo "Error: the command '$ceph_cmd' failed with return code $ceph_cmd_rc" + fi + (( extr_dbg >= 3 )) && echo "$ceph_cmd_out" > /tmp/e2 + l0=`echo "$ceph_cmd_out" | jq '[.pg_stats | group_by(.pg_stats)[0] | map({pgid: .pgid, pool: (.pgid | split(".")[0]), acting: .acting, acting_primary: .acting_primary})] | .[]' ` + else + l0=`jq '[.pg_stats | group_by(.pg_stats)[0] | map({pgid: .pgid, pool: (.pgid | split(".")[0]), acting: .acting, acting_primary: .acting_primary})] | .[]' $infile ` + fi + (( extr_dbg >= 2 )) && echo "L0: $l0" + + mapfile -t l1 < <(echo "$l0" | jq -c '.[]') + (( extr_dbg >= 2 )) && echo "L1: ${#l1[@]}" + + for item in "${l1[@]}"; do + pgid=$(echo "$item" | jq -r '.pgid') + acting=$(echo "$item" | jq -r '.acting | @sh') + pg_acting_dict["$pgid"]=$acting + acting_primary=$(echo "$item" | jq -r '.acting_primary') + pg_primary_dict["$pgid"]=$acting_primary + pool=$(echo "$item" | jq -r '.pool') + pg_pool_dict["$pgid"]=$pool + done + + if [[ -n "$saved_echo_flag" ]]; then set -x; fi +} + + +# a function that counts the number of common active-set elements between two PGs +# 1 - the first PG +# 2 - the second PG +# 3 - the dictionary of active sets +function count_common_active { + local pg1=$1 + local pg2=$2 + local -n pg_acting_dict=$3 + local -n res=$4 + + local -a a1=(${pg_acting_dict[$pg1]}) + local -a a2=(${pg_acting_dict[$pg2]}) + + local -i cnt=0 + for i in "${a1[@]}"; do + for j in "${a2[@]}"; do + if [[ $i -eq $j ]]; then + cnt=$((cnt+1)) + fi + done + done + + res=$cnt +} + + +# given a PG, find another one with a disjoint active set +# - but allow a possible common Primary +# 1 - the PG +# 2 - the dictionary of active sets +# 3 - [out] - the PG with a disjoint active set +function find_disjoint_but_primary { + local pg=$1 + local -n ac_dict=$2 + local -n p_dict=$3 + local -n res=$4 + + for cand in "${!ac_dict[@]}"; do + if [[ "$cand" != "$pg" ]]; then + local -i common=0 + count_common_active "$pg" "$cand" ac_dict common + if [[ $common -eq 0 || ( $common -eq 1 && "${p_dict[$pg]}" == "${p_dict[$cand]}" )]]; then + res=$cand + return + fi + fi + done +} + + + # A debug flag is set for the PG specified, causing the 'pg query' command to display # an additional 'scrub sessions counter' field. # diff --git a/qa/suites/crimson-rados-experimental/.qa b/qa/suites/crimson-rados-experimental/.qa index fea2489fdf6..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/.qa +++ b/qa/suites/crimson-rados-experimental/.qa @@ -1 +1 @@ -../.qa
\ No newline at end of file +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml deleted file mode 120000 index bd9854e7029..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/centos_latest.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/distros/supported/centos_latest.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml deleted file mode 100644 index d8e5898b99f..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-1.yaml +++ /dev/null @@ -1,14 +0,0 @@ -overrides: - ceph-deploy: - conf: - global: - osd pool default size: 2 - osd crush chooseleaf type: 0 - osd pool default pg num: 128 - osd pool default pgp num: 128 - ceph: - conf: - osd: - osd shutdown pgref assert: true -roles: -- [mon.a, mgr.x, osd.0, osd.1, osd.2, client.0] diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml deleted file mode 100644 index c22f08eecf8..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/ceph.yaml +++ /dev/null @@ -1,18 +0,0 @@ -overrides: - install: - ceph: - flavor: crimson -tasks: -- install: -- ceph: - conf: - osd: - debug monc: 20 - mon: - mon min osdmap epochs: 50 - paxos service trim min: 10 - # prune full osdmaps regularly - mon osdmap full prune min: 15 - mon osdmap full prune interval: 2 - mon osdmap full prune txsize: 2 - flavor: crimson diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml deleted file mode 120000 index 6a70c381709..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/config/seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml deleted file mode 100644 index ad8c921425b..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/rados_api_tests.yaml +++ /dev/null @@ -1,28 +0,0 @@ -overrides: - ceph: - log-ignorelist: - - reached quota - - but it is still running - - overall HEALTH_ - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(CACHE_POOL_NO_HIT_SET\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(PG_DEGRADED\) - conf: - client: - debug ms: 1 - mon: - mon warn on pool no app: false - osd: - osd class load list: "*" - osd class default list: "*" - osd blocked scrub grace period: 3600 -tasks: -- workunit: - clients: - client.0: - - rados/test.sh - - rados/test_pool_quota.sh diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml deleted file mode 100644 index 25efcdac83d..00000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/readwrite.yaml +++ /dev/null @@ -1,18 +0,0 @@ -overrides: - ceph: - crush_tunables: optimal - conf: - mon: - mon osd initial require min compat client: luminous - osd: - osd_discard_disconnected_ops: false -tasks: -- rados: - clients: [client.0] - ops: 4000 - objects: 500 - max_attr_len: 8192 - op_weights: - read: 45 - write: 45 - delete: 10 diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/% b/qa/suites/crimson-rados-experimental/thrash/% index e69de29bb2d..e69de29bb2d 100644 --- a/qa/suites/crimson-rados-experimental/seastore/basic/% +++ b/qa/suites/crimson-rados-experimental/thrash/% diff --git a/qa/suites/crimson-rados-experimental/seastore/.qa b/qa/suites/crimson-rados-experimental/thrash/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/.qa diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/.qa b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled new file mode 120000 index 00000000000..5393a75548a --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/2-size-2-min-size.yaml.disabled @@ -0,0 +1 @@ +.qa/overrides/2-size-2-min-size.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml new file mode 120000 index 00000000000..5ff70eadf75 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/0-size-min-size-overrides/3-size-2-min-size.yaml @@ -0,0 +1 @@ +.qa/overrides/3-size-2-min-size.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/.qa b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/normal_pg_log.yaml diff --git a/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/short_pg_log.yaml index abd86d7d986..abd86d7d986 120000 --- a/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml.disabled +++ b/qa/suites/crimson-rados-experimental/thrash/1-pg-log-overrides/short_pg_log.yaml diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/$ diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/.qa b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/deploy/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/default.yaml diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled new file mode 120000 index 00000000000..47afd70202d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-active-recovery.yaml.disabled @@ -0,0 +1 @@ +.qa/overrides/more-active-recovery.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled new file mode 100644 index 00000000000..0bbc72db754 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-partial-recovery.yaml.disabled @@ -0,0 +1,6 @@ +overrides: + ceph: + conf: + global: + osd_async_recovery_min_cost: 1 + osd_object_clean_region_max_num_intervals: 1000 diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled new file mode 100644 index 00000000000..4aed086bcc3 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-async-recovery.yaml.disabled @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + global: + osd_async_recovery_min_cost: 1 diff --git a/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled new file mode 100644 index 00000000000..88f15f2f691 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/2-recovery-overrides/more-partial-recovery.yaml.disabled @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + global: + osd_object_clean_region_max_num_intervals: 1000 diff --git a/qa/suites/crimson-rados-experimental/thrash/clusters/+ b/qa/suites/crimson-rados-experimental/thrash/clusters/+ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/+ diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/.qa b/qa/suites/crimson-rados-experimental/thrash/clusters/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/.qa diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-2.yaml b/qa/suites/crimson-rados-experimental/thrash/clusters/fixed-2.yaml index 9774de6887b..79641f695ab 100644 --- a/qa/suites/crimson-rados-experimental/seastore/basic/clusters/fixed-2.yaml +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/fixed-2.yaml @@ -6,6 +6,15 @@ overrides: conf: osd: osd shutdown pgref assert: true + crimson alien thread cpu cores: 6-7 + osd.0: + crimson seastar cpu cores: 0-2 + osd.1: + crimson seastar cpu cores: 3-5 + osd.2: + crimson seastar cpu cores: 0-2 + osd.3: + crimson seastar cpu cores: 3-5 global: ms cluster mode: crc ms service mode: crc diff --git a/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled new file mode 100644 index 00000000000..e559d9126e8 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/clusters/openstack.yaml.disabled @@ -0,0 +1,4 @@ +openstack: + - volumes: # attached to each instance + count: 4 + size: 10 # GB diff --git a/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro b/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro new file mode 120000 index 00000000000..a5b729b9efa --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/crimson-supported-all-distro @@ -0,0 +1 @@ +.qa/distros/crimson-supported-all-distro/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/crimson_qa_overrides.yaml b/qa/suites/crimson-rados-experimental/thrash/crimson_qa_overrides.yaml index 2bf67af1b18..2bf67af1b18 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/crimson_qa_overrides.yaml +++ b/qa/suites/crimson-rados-experimental/thrash/crimson_qa_overrides.yaml diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/.qa b/qa/suites/crimson-rados-experimental/thrash/deploy/.qa index a602a0353e7..a602a0353e7 120000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/tasks/.qa +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/.qa diff --git a/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml b/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml new file mode 100644 index 00000000000..ecad09cfe3a --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/ceph.yaml @@ -0,0 +1,11 @@ +overrides: + install: + ceph: + flavor: crimson +tasks: +- install: +- ceph: + conf: + osd: + debug monc: 20 + flavor: crimson diff --git a/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled b/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled new file mode 100644 index 00000000000..0c2062240ee --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/deploy/cephadm.yaml.disabled @@ -0,0 +1,16 @@ +# no need to verify os + flavor + sha1 +verify_ceph_hash: false +tasks: +- cephadm: + conf: + mgr: + debug ms: 1 + debug mgr: 20 + debug osd: 10 +- cephadm.shell: + mon.a: + - ceph orch status + - ceph orch ps + - ceph orch ls + - ceph orch host ls + - ceph orch device ls diff --git a/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa b/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/objectstore/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml b/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa b/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashers/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml b/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml new file mode 100644 index 00000000000..aa44b6101ff --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashers/default.yaml @@ -0,0 +1,34 @@ +overrides: + ceph: + log-ignorelist: + - but it is still running + - objects unfound and apparently lost + conf: + osd: + osd debug reject backfill probability: .3 + osd scrub min interval: 60 + osd scrub max interval: 120 + osd max backfills: 3 + osd snap trim sleep: 2 + osd delete sleep: 1 + mon: + mon min osdmap epochs: 50 + paxos service trim min: 10 + # prune full osdmaps regularly + mon osdmap full prune min: 15 + mon osdmap full prune interval: 2 + mon osdmap full prune txsize: 2 +tasks: +- thrashosds: + timeout: 2400 + dump_ops_enable: false + sighup_delay: 0 + min_in: 3 + noscrub_toggle_delay: 0 + chance_thrash_pg_upmap: 0 + reweight_osd: 0 + thrash_primary_affinity: false + ceph_objectstore_tool: false + chance_inject_pause_short: 0 + chance_thrash_cluster_full: 0 + chance_reset_purged_snaps_last: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml b/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml new file mode 120000 index 00000000000..9124eb1aa29 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/thrashosds-health.yaml @@ -0,0 +1 @@ +.qa/tasks/thrashosds-health.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/.qa b/qa/suites/crimson-rados-experimental/thrash/workloads/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml new file mode 100644 index 00000000000..8c9764ade84 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/admin_socket_objecter_requests.yaml @@ -0,0 +1,13 @@ +overrides: + ceph: + conf: + client.0: + admin socket: /var/run/ceph/ceph-$name.asok +tasks: +- radosbench: + clients: [client.0] + time: 150 +- admin_socket: + client.0: + objecter_requests: + test: "http://git.ceph.com/?p={repo};a=blob_plain;f=src/test/admin_socket/objecter_requests;hb={branch}" diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml new file mode 100644 index 00000000000..d35e8421ab4 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/pool-snaps-few-objects.yaml @@ -0,0 +1,20 @@ +overrides: + conf: + osd: + osd deep scrub update digest min age: 0 +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + pool_snaps: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml new file mode 100644 index 00000000000..902c4b56a1e --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench-high-concurrency.yaml @@ -0,0 +1,49 @@ +overrides: + ceph: + conf: + client.0: + debug ms: 1 + debug objecter: 20 + debug rados: 20 +tasks: +- full_sequential: + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 + - radosbench: + clients: [client.0] + concurrency: 128 + size: 8192 + time: 90 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml new file mode 100644 index 00000000000..071f55e3928 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/radosbench.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + conf: + client.0: + debug ms: 1 + debug objecter: 20 + debug rados: 20 +tasks: +- full_sequential: + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 + - radosbench: + clients: [client.0] + time: 90 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml new file mode 100644 index 00000000000..afe04229898 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-balanced.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + balance_reads: true + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml new file mode 100644 index 00000000000..445b582ea42 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects-localized.yaml @@ -0,0 +1,24 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + localize_reads: true + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml new file mode 100644 index 00000000000..e7e8070fd76 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/small-objects.yaml @@ -0,0 +1,23 @@ +overrides: + ceph: + crush_tunables: jewel +tasks: +- rados: + clients: [client.0] + ops: 400000 + max_seconds: 600 + max_in_flight: 64 + objects: 1024 + size: 16384 + max_attr_len: 8192 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 + setattr: 25 + rmattr: 25 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml new file mode 100644 index 00000000000..1161c3cc253 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-balanced.yaml @@ -0,0 +1,15 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + balance_reads: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml new file mode 100644 index 00000000000..80af0def0e4 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects-localized.yaml @@ -0,0 +1,15 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + localize_reads: true + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml new file mode 100644 index 00000000000..0694ffcd0d6 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/snaps-few-objects.yaml @@ -0,0 +1,14 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 50 + op_weights: + read: 100 + write: 100 + delete: 50 + snap_create: 50 + snap_remove: 50 + rollback: 0 + # TODO: CEPH_OSD_OP_COPY_FROM + copy_from: 0 diff --git a/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml b/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml new file mode 100644 index 00000000000..606dcae6922 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/thrash/workloads/write_fadvise_dontneed.yaml @@ -0,0 +1,8 @@ +tasks: +- rados: + clients: [client.0] + ops: 4000 + objects: 500 + write_fadvise_dontneed: true + op_weights: + write: 100 diff --git a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore b/qa/suites/crimson-rados/singleton/objectstore deleted file mode 120000 index dbccf5ad928..00000000000 --- a/qa/suites/crimson-rados/singleton/objectstore +++ /dev/null @@ -1 +0,0 @@ -../thrash/objectstore
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/.qa b/qa/suites/crimson-rados/singleton/objectstore/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml new file mode 120000 index 00000000000..481e393be4a --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml.disabled b/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml index abd86d7d986..abd86d7d986 120000 --- a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml.disabled +++ b/qa/suites/crimson-rados/thrash/1-pg-log-overrides/short_pg_log.yaml diff --git a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled new file mode 120000 index 00000000000..61e26e7acf8 --- /dev/null +++ b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml.disabled @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml b/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml new file mode 120000 index 00000000000..abd86d7d986 --- /dev/null +++ b/qa/suites/crimson-rados/thrash_simple/1-pg-log-overrides/short_pg_log.yaml @@ -0,0 +1 @@ +.qa/overrides/short_pg_log.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml index e84f396e4b2..481e393be4a 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml
\ No newline at end of file +.qa/config/crimson_bluestore.yaml
\ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml index 6a70c381709..61e26e7acf8 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml
\ No newline at end of file +.qa/config/crimson_seastore.yaml
\ No newline at end of file diff --git a/qa/suites/fs/multifs/tasks/failover.yaml b/qa/suites/fs/multifs/tasks/failover.yaml index 55dde639c23..b7a0338566c 100644 --- a/qa/suites/fs/multifs/tasks/failover.yaml +++ b/qa/suites/fs/multifs/tasks/failover.yaml @@ -8,6 +8,7 @@ overrides: - \(MDS_DAMAGE\) - \(FS_DEGRADED\) - \(MDS_CACHE_OVERSIZED\) + - \(MDS_ESTIMATED_REPLAY_TIME\) ceph-fuse: disabled: true tasks: diff --git a/qa/suites/fs/nfs/tasks/nfs.yaml b/qa/suites/fs/nfs/tasks/nfs.yaml index aa966bff214..2dd668c9f88 100644 --- a/qa/suites/fs/nfs/tasks/nfs.yaml +++ b/qa/suites/fs/nfs/tasks/nfs.yaml @@ -1,3 +1,10 @@ +overrides: + install: + extra_system_packages: + rpm: + - fio + deb: + - fio tasks: - cephfs_test_runner: modules: diff --git a/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml b/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml index 602d3416263..aa327b0cdf5 100644 --- a/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml +++ b/qa/suites/fs/workload/tasks/6-workunit/kernel_untar_build.yaml @@ -5,6 +5,7 @@ overrides: - "mds.dir_split" tasks: - workunit: + timeout: 5h clients: all: - kernel_untar_build.sh diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml index 7c97edae552..0416ae2ea4e 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_initiator.yaml @@ -1,7 +1,8 @@ +# runs on default nvmeof image (i.e. DEFAULT_NVMEOF_IMAGE) tasks: - nvmeof: installer: host.a - gw_image: quay.io/ceph/nvmeof:latest # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" + gw_image: default # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" rbd: pool_name: mypool image_name_prefix: myimage diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml index 9ef37004427..dfe31380bb6 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_namespaces.yaml @@ -18,6 +18,7 @@ tasks: clients: client.0: - nvmeof/setup_subsystem.sh + - nvmeof/basic_tests.sh env: RBD_POOL: mypool RBD_IMAGE_PREFIX: myimage @@ -27,7 +28,6 @@ tasks: timeout: 30m clients: client.0: - - nvmeof/basic_tests.sh - nvmeof/fio_test.sh --rbd_iostat client.1: - nvmeof/basic_tests.sh diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml index 12cb50b408d..d66b6fc8093 100644 --- a/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_scalability.yaml @@ -31,8 +31,11 @@ tasks: no_coverage_and_limits: true timeout: 30m clients: - client.0: + client.3: - nvmeof/scalability_test.sh nvmeof.a,nvmeof.b - nvmeof/scalability_test.sh nvmeof.b,nvmeof.c,nvmeof.d + - nvmeof/scalability_test.sh nvmeof.b,nvmeof.c env: SCALING_DELAYS: '50' + RBD_POOL: mypool + NVMEOF_GROUP: mygroup0 diff --git a/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml b/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml new file mode 100644 index 00000000000..83d54cdf5c3 --- /dev/null +++ b/qa/suites/nvmeof/thrash/gateway-initiator-setup/10-subsys-90-namespace-no_huge_pages.yaml @@ -0,0 +1,37 @@ +tasks: +- nvmeof: + installer: host.a + gw_image: quay.io/ceph/nvmeof:latest # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" + rbd: + pool_name: mypool + image_name_prefix: myimage + gateway_config: + subsystems_count: 10 + namespaces_count: 90 # each subsystem + cli_image: quay.io/ceph/nvmeof-cli:latest + +- cephadm.wait_for_service: + service: nvmeof.mypool.mygroup0 + +- cephadm.exec: + host.a: + - ceph orch ls nvmeof --export > /tmp/nvmeof-orig.yaml + - cp /tmp/nvmeof-orig.yaml /tmp/nvmeof-no-huge-page.yaml + - "sed -i '/ pool: mypool/a\\ spdk_mem_size: 4096' /tmp/nvmeof-no-huge-page.yaml" + - cat /tmp/nvmeof-no-huge-page.yaml + - ceph orch ls --refresh + - ceph orch apply -i /tmp/nvmeof-no-huge-page.yaml + - ceph orch redeploy nvmeof.mypool.mygroup0 + +- cephadm.wait_for_service: + service: nvmeof.mypool.mygroup0 + +- workunit: + no_coverage_and_limits: true + clients: + client.0: + - nvmeof/setup_subsystem.sh + - nvmeof/basic_tests.sh + env: + RBD_POOL: mypool + RBD_IMAGE_PREFIX: myimage diff --git a/qa/suites/nvmeof/thrash/gateway-initiator-setup/3-subsys-60-namespace.yaml b/qa/suites/nvmeof/thrash/gateway-initiator-setup/120-subsys-8-namespace.yaml index b4755a6433b..0f7ac011a60 100644 --- a/qa/suites/nvmeof/thrash/gateway-initiator-setup/3-subsys-60-namespace.yaml +++ b/qa/suites/nvmeof/thrash/gateway-initiator-setup/120-subsys-8-namespace.yaml @@ -6,8 +6,8 @@ tasks: pool_name: mypool image_name_prefix: myimage gateway_config: - subsystems_count: 3 - namespaces_count: 20 # each subsystem + subsystems_count: 120 + namespaces_count: 8 # each subsystem cli_image: quay.io/ceph/nvmeof-cli:latest - cephadm.wait_for_service: diff --git a/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml b/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml index 19fa2ec605d..46037784d31 100644 --- a/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml +++ b/qa/suites/nvmeof/thrash/thrashers/nvmeof_mon_thrash.yaml @@ -11,6 +11,7 @@ overrides: - NVMEOF_SINGLE_GATEWAY - NVMEOF_GATEWAY_DOWN - are in unavailable state + - is unavailable - is in error state - failed cephadm daemon diff --git a/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml b/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml index 80bf0527715..b58dc14d87b 100644 --- a/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml +++ b/qa/suites/nvmeof/thrash/thrashers/nvmeof_thrash.yaml @@ -6,9 +6,11 @@ overrides: - NVMEOF_SINGLE_GATEWAY - NVMEOF_GATEWAY_DOWN - are in unavailable state + - is unavailable - is in error state - failed cephadm daemon tasks: - nvmeof.thrash: checker_host: 'client.0' + randomize: False diff --git a/qa/suites/nvmeof/thrash/workloads/fio.yaml b/qa/suites/nvmeof/thrash/workloads/fio.yaml index b042b92d6ae..f9a0d0ebde5 100644 --- a/qa/suites/nvmeof/thrash/workloads/fio.yaml +++ b/qa/suites/nvmeof/thrash/workloads/fio.yaml @@ -1,11 +1,11 @@ tasks: - workunit: no_coverage_and_limits: true - timeout: 30m + timeout: 60m clients: client.0: - - nvmeof/fio_test.sh --rbd_iostat + - nvmeof/fio_test.sh --random_devices 200 env: RBD_POOL: mypool IOSTAT_INTERVAL: '10' - RUNTIME: '600' + RUNTIME: '1800' diff --git a/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml b/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml index 372bf2561fa..8b3c4c11ac6 100644 --- a/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml +++ b/qa/suites/rados/monthrash/workloads/rados_mon_osdmap_prune.yaml @@ -15,6 +15,7 @@ overrides: # causing tests to fail due to health warns, even if # the tests themselves are successful. - \(OSDMAP_FLAGS\) + - \(PG_DEGRADED\) tasks: - workunit: clients: diff --git a/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml b/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml new file mode 100644 index 00000000000..7cd47898544 --- /dev/null +++ b/qa/suites/rados/objectstore/backends/ceph_test_bluefs.yaml @@ -0,0 +1,8 @@ +roles: +- [mon.a, mgr.x, osd.0, osd.1, client.0] +tasks: +- install: +- exec: + client.0: + - mkdir $TESTDIR/ceph_test_bluefs && cd $TESTDIR/ceph_test_bluefs && ceph_test_bluefs --log-file $TESTDIR/archive/ceph_test_bluefs.log --debug-bluefs 5/20 --gtest_catch_exceptions=0 + - rm -rf $TESTDIR/ceph_test_bluefs diff --git a/qa/suites/rados/singleton/all/stretch-mode-5-mons-8-osds.yaml b/qa/suites/rados/singleton/all/stretch-mode-5-mons-8-osds.yaml index d7b10c50a94..69a54b0f1b7 100644 --- a/qa/suites/rados/singleton/all/stretch-mode-5-mons-8-osds.yaml +++ b/qa/suites/rados/singleton/all/stretch-mode-5-mons-8-osds.yaml @@ -26,7 +26,6 @@ overrides: ceph: conf: global: - mon election default strategy: 3 osd pool default size: 3 osd pool default min size: 2 mon: @@ -55,4 +54,4 @@ tasks: - mon/mon-stretch-mode-5-mons-8-osds.sh - cephfs_test_runner: modules: - - tasks.stretch_mode_disable_enable
\ No newline at end of file + - tasks.stretch_mode_disable_enable diff --git a/qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml b/qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml index a8bbbafece0..b916bed1475 100644 --- a/qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml +++ b/qa/suites/rados/thrash-old-clients/workloads/rbd_cls.yaml @@ -2,6 +2,9 @@ meta: - desc: | rbd object class functional tests tasks: -- exec: - client.2: - - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror +- workunit: + clients: + client.2: + - cls/test_cls_rbd.sh + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' diff --git a/qa/suites/rados/verify/validater/valgrind.yaml b/qa/suites/rados/verify/validater/valgrind.yaml index e2dc29b5f7e..17cf141b0cd 100644 --- a/qa/suites/rados/verify/validater/valgrind.yaml +++ b/qa/suites/rados/verify/validater/valgrind.yaml @@ -27,6 +27,7 @@ overrides: - \(SLOW_OPS\) - slow request - OSD bench result + - OSD_DOWN valgrind: mon: [--tool=memcheck, --leak-check=full, --show-reachable=yes] osd: [--tool=memcheck] diff --git a/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml b/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml new file mode 100644 index 00000000000..d2072c41a68 --- /dev/null +++ b/qa/suites/rbd/migration/6-prepare/qcow2-https.yaml @@ -0,0 +1,8 @@ +tasks: + - exec: + client.0: + - mkdir /home/ubuntu/cephtest/migration + - qemu-img create -f qcow2 /home/ubuntu/cephtest/migration/empty.qcow2 1G + - echo '{"type":"qcow","stream":{"type":"http","url":"https://download.ceph.com/qa/ubuntu-12.04.qcow2"}}' | rbd migration prepare --import-only --source-spec-path - client.0.0 + - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.1 + - rbd migration prepare --import-only --source-spec '{"type":"qcow","stream":{"type":"file","file_path":"/home/ubuntu/cephtest/migration/empty.qcow2"}}' client.0.2 diff --git a/qa/suites/rgw/bucket-logging/% b/qa/suites/rgw/bucket-logging/% new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/rgw/bucket-logging/% diff --git a/qa/suites/rgw/bucket-logging/.qa b/qa/suites/rgw/bucket-logging/.qa new file mode 120000 index 00000000000..a602a0353e7 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/.qa @@ -0,0 +1 @@ +../.qa/
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/0-install.yaml b/qa/suites/rgw/bucket-logging/0-install.yaml new file mode 100644 index 00000000000..6cf82f57476 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/0-install.yaml @@ -0,0 +1,13 @@ +tasks: +- install: +- ceph: +- openssl_keys: +- rgw: [client.0] +- tox: [client.0] + +overrides: + ceph: + conf: + global: + osd_min_pg_log_entries: 10 + osd_max_pg_log_entries: 10 diff --git a/qa/suites/rgw/bucket-logging/beast.yaml b/qa/suites/rgw/bucket-logging/beast.yaml new file mode 120000 index 00000000000..09ced62c42a --- /dev/null +++ b/qa/suites/rgw/bucket-logging/beast.yaml @@ -0,0 +1 @@ +.qa/rgw_frontend/beast.yaml
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/fixed-1.yaml b/qa/suites/rgw/bucket-logging/fixed-1.yaml new file mode 120000 index 00000000000..02df5dd0cd0 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/fixed-1.yaml @@ -0,0 +1 @@ +.qa/clusters/fixed-1.yaml
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/ignore-pg-availability.yaml b/qa/suites/rgw/bucket-logging/ignore-pg-availability.yaml new file mode 120000 index 00000000000..32340b1fa8b --- /dev/null +++ b/qa/suites/rgw/bucket-logging/ignore-pg-availability.yaml @@ -0,0 +1 @@ +.qa/rgw/ignore-pg-availability.yaml
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/overrides.yaml b/qa/suites/rgw/bucket-logging/overrides.yaml new file mode 100644 index 00000000000..a448a323d36 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/overrides.yaml @@ -0,0 +1,10 @@ +overrides: + ceph: + conf: + client: + setuser: ceph + setgroup: ceph + debug rgw: 20 + rgw bucket logging obj roll time: 5 + rgw: + storage classes: LUKEWARM, FROZEN diff --git a/qa/suites/rgw/bucket-logging/s3tests-branch.yaml b/qa/suites/rgw/bucket-logging/s3tests-branch.yaml new file mode 120000 index 00000000000..bdcaca48ae0 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/s3tests-branch.yaml @@ -0,0 +1 @@ +.qa/rgw/s3tests-branch.yaml
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/supported-distros b/qa/suites/rgw/bucket-logging/supported-distros new file mode 120000 index 00000000000..78f2991b407 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/supported-distros @@ -0,0 +1 @@ +.qa/distros/supported-random-distro$/
\ No newline at end of file diff --git a/qa/suites/rgw/bucket-logging/tasks/+ b/qa/suites/rgw/bucket-logging/tasks/+ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/rgw/bucket-logging/tasks/+ diff --git a/qa/suites/rgw/bucket-logging/tasks/s3tests.yaml b/qa/suites/rgw/bucket-logging/tasks/s3tests.yaml new file mode 100644 index 00000000000..c1d3b7192e1 --- /dev/null +++ b/qa/suites/rgw/bucket-logging/tasks/s3tests.yaml @@ -0,0 +1,6 @@ +tasks: +- s3tests: + client.0: + boto3_extensions: True + rgw_server: client.0 + extra_attrs: ["bucket_logging"] diff --git a/qa/suites/rgw/crypt/2-kms/barbican.yaml b/qa/suites/rgw/crypt/2-kms/barbican.yaml index 9bf5fb81131..e3f78810416 100644 --- a/qa/suites/rgw/crypt/2-kms/barbican.yaml +++ b/qa/suites/rgw/crypt/2-kms/barbican.yaml @@ -27,7 +27,7 @@ tasks: - tox: [ client.0 ] - keystone: client.0: - force-branch: stable/2023.1 + force-branch: stable/2024.1 services: - name: swift type: object-store @@ -68,7 +68,7 @@ tasks: project: s3 - barbican: client.0: - force-branch: stable/2023.1 + force-branch: stable/2024.1 use-keystone-role: client.0 keystone_authtoken: auth_plugin: password diff --git a/qa/suites/rgw/multisite/realms/two-zonegroup.yaml.disabled b/qa/suites/rgw/multisite/realms/two-zonegroup.yaml index 5e4234236a9..ac2104cdd05 100644 --- a/qa/suites/rgw/multisite/realms/two-zonegroup.yaml.disabled +++ b/qa/suites/rgw/multisite/realms/two-zonegroup.yaml @@ -28,4 +28,4 @@ overrides: - name: b2 endpoints: [c2.client.1] rgw-multisite-tests: - args: [tests.py] + args: [tests.py, -a, '!fails_with_rgw'] diff --git a/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml b/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml index 462570e7727..303f98d540e 100644 --- a/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml +++ b/qa/suites/rgw/notifications/tasks/kafka/test_kafka.yaml @@ -1,7 +1,7 @@ tasks: - kafka: client.0: - kafka_version: 2.6.0 + kafka_version: 3.8.1 - notification-tests: client.0: extra_attr: ["kafka_test", "data_path_v2_kafka_test"] diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/+ b/qa/suites/rgw/notifications/tasks/kafka_failover/+ new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/+ diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml b/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml new file mode 100644 index 00000000000..5c83d5c0d23 --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/0-install.yaml @@ -0,0 +1,20 @@ +tasks: +- install: +- ceph: +- openssl_keys: +- rgw: + client.0: + +overrides: + install: + ceph: + extra_system_packages: + rpm: + - java + deb: + - default-jre + ceph: + conf: + global: + osd_min_pg_log_entries: 10 + osd_max_pg_log_entries: 10 diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros b/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros new file mode 120000 index 00000000000..46280a42a96 --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/supported-distros @@ -0,0 +1 @@ +../../.qa/distros/supported-random-distro$/
\ No newline at end of file diff --git a/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml b/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml new file mode 100644 index 00000000000..01d6fc637de --- /dev/null +++ b/qa/suites/rgw/notifications/tasks/kafka_failover/test_kafka.yaml @@ -0,0 +1,8 @@ +tasks: +- kafka-failover: + client.0: + kafka_version: 3.8.1 +- notification-tests: + client.0: + extra_attr: ["kafka_failover"] + rgw_server: client.0 diff --git a/qa/suites/rgw/sts/auth-order/.qa b/qa/suites/rgw/sts/auth-order/.qa new file mode 120000 index 00000000000..fea2489fdf6 --- /dev/null +++ b/qa/suites/rgw/sts/auth-order/.qa @@ -0,0 +1 @@ +../.qa
\ No newline at end of file diff --git a/qa/suites/rgw/sts/auth-order/local-sts.yaml b/qa/suites/rgw/sts/auth-order/local-sts.yaml new file mode 100644 index 00000000000..2f7dcc6b128 --- /dev/null +++ b/qa/suites/rgw/sts/auth-order/local-sts.yaml @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + client: + rgw s3 auth order: local, sts, external diff --git a/qa/suites/rgw/sts/auth-order/sts-local.yaml b/qa/suites/rgw/sts/auth-order/sts-local.yaml new file mode 100644 index 00000000000..a7b00d00f0b --- /dev/null +++ b/qa/suites/rgw/sts/auth-order/sts-local.yaml @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + client: + rgw s3 auth order: sts, local, external diff --git a/qa/suites/rgw/tempest/0-install.yaml b/qa/suites/rgw/tempest/0-install.yaml index f968db20c2b..b6ef17de4ee 100644 --- a/qa/suites/rgw/tempest/0-install.yaml +++ b/qa/suites/rgw/tempest/0-install.yaml @@ -4,7 +4,7 @@ tasks: - tox: [ client.0 ] - keystone: client.0: - force-branch: stable/2023.1 + force-branch: stable/2024.1 services: - name: swift type: object-store diff --git a/qa/suites/rgw/tempest/tasks/s3/% b/qa/suites/rgw/tempest/tasks/s3/% new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/qa/suites/rgw/tempest/tasks/s3/% diff --git a/qa/suites/rgw/tempest/tasks/s3/.qa b/qa/suites/rgw/tempest/tasks/s3/.qa new file mode 120000 index 00000000000..fea2489fdf6 --- /dev/null +++ b/qa/suites/rgw/tempest/tasks/s3/.qa @@ -0,0 +1 @@ +../.qa
\ No newline at end of file diff --git a/qa/suites/rgw/tempest/tasks/s3/auth-order/.qa b/qa/suites/rgw/tempest/tasks/s3/auth-order/.qa new file mode 120000 index 00000000000..fea2489fdf6 --- /dev/null +++ b/qa/suites/rgw/tempest/tasks/s3/auth-order/.qa @@ -0,0 +1 @@ +../.qa
\ No newline at end of file diff --git a/qa/suites/rgw/tempest/tasks/s3/auth-order/external-local.yaml b/qa/suites/rgw/tempest/tasks/s3/auth-order/external-local.yaml new file mode 100644 index 00000000000..c46a51e0958 --- /dev/null +++ b/qa/suites/rgw/tempest/tasks/s3/auth-order/external-local.yaml @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + client: + rgw s3 auth order: sts, external, local diff --git a/qa/suites/rgw/tempest/tasks/s3/auth-order/local-external.yaml b/qa/suites/rgw/tempest/tasks/s3/auth-order/local-external.yaml new file mode 100644 index 00000000000..a7b00d00f0b --- /dev/null +++ b/qa/suites/rgw/tempest/tasks/s3/auth-order/local-external.yaml @@ -0,0 +1,5 @@ +overrides: + ceph: + conf: + client: + rgw s3 auth order: sts, local, external diff --git a/qa/suites/rgw/tempest/tasks/s3tests.yaml b/qa/suites/rgw/tempest/tasks/s3/s3tests.yaml index 4efb579fa83..4efb579fa83 100644 --- a/qa/suites/rgw/tempest/tasks/s3tests.yaml +++ b/qa/suites/rgw/tempest/tasks/s3/s3tests.yaml diff --git a/qa/suites/upgrade/quincy-x/parallel/0-start.yaml b/qa/suites/upgrade/quincy-x/parallel/0-start.yaml index 40fbcefe728..62fb6427f72 100644 --- a/qa/suites/upgrade/quincy-x/parallel/0-start.yaml +++ b/qa/suites/upgrade/quincy-x/parallel/0-start.yaml @@ -32,13 +32,22 @@ overrides: osd: osd shutdown pgref assert: true log-ignorelist: - - \(POOL_APP_NOT_ENABLED\) + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum + - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED diff --git a/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml b/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml index e27c7c0f092..f7167975aa9 100644 --- a/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml +++ b/qa/suites/upgrade/quincy-x/parallel/1-tasks.yaml @@ -1,11 +1,8 @@ overrides: ceph: log-ignorelist: - - mons down - - mon down - - MON_DOWN - - out of quorum - - PG_AVAILABILITY + - Telemetry requires re-opt-in + - telemetry module includes new collections tasks: - install: branch: quincy diff --git a/qa/suites/upgrade/quincy-x/parallel/workload/rados_api.yaml b/qa/suites/upgrade/quincy-x/parallel/workload/rados_api.yaml index 9c2ff9da185..9a0585cc074 100644 --- a/qa/suites/upgrade/quincy-x/parallel/workload/rados_api.yaml +++ b/qa/suites/upgrade/quincy-x/parallel/workload/rados_api.yaml @@ -9,4 +9,6 @@ workload: clients: client.0: - cls + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done end rados_api.yaml" diff --git a/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml b/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml index 005514292ce..5641471629e 100644 --- a/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml +++ b/qa/suites/upgrade/quincy-x/stress-split/1-start.yaml @@ -1,17 +1,25 @@ overrides: ceph: log-ignorelist: - - \(POOL_APP_NOT_ENABLED\) + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum + - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED - OSDMAP_FLAGS - - PG_AVAILABILITY + - OSD_UPGRADE_FINISHED tasks: - install: branch: quincy diff --git a/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-cls.yaml b/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-cls.yaml index b722f187361..a55dddf46f7 100644 --- a/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-cls.yaml +++ b/qa/suites/upgrade/quincy-x/stress-split/2-first-half-tasks/rbd-cls.yaml @@ -7,4 +7,6 @@ first-half-tasks: clients: client.0: - cls/test_cls_rbd.sh + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done cls/test_cls_rbd.sh 5-workload" diff --git a/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-cls.yaml b/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-cls.yaml index 649b024a476..d54ba8039d0 100644 --- a/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-cls.yaml +++ b/qa/suites/upgrade/quincy-x/stress-split/3-stress-tasks/rbd-cls.yaml @@ -7,4 +7,6 @@ stress-tasks: clients: client.0: - cls/test_cls_rbd.sh + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done cls/test_cls_rbd.sh 5-workload" diff --git a/qa/suites/upgrade/reef-x/parallel/0-start.yaml b/qa/suites/upgrade/reef-x/parallel/0-start.yaml index 146bd57960d..62fb6427f72 100644 --- a/qa/suites/upgrade/reef-x/parallel/0-start.yaml +++ b/qa/suites/upgrade/reef-x/parallel/0-start.yaml @@ -32,4 +32,22 @@ overrides: osd: osd shutdown pgref assert: true log-ignorelist: - - PG_DEGRADED + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down + - OSD_DOWN + - mons down + - mon down + - MON_DOWN + - out of quorum + - PG_AVAILABILITY + - PG_DEGRADED + - Reduced data availability + - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED + - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED diff --git a/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml b/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml index ce4e0cc228b..b5160c2dd00 100644 --- a/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml +++ b/qa/suites/upgrade/reef-x/parallel/1-tasks.yaml @@ -1,12 +1,8 @@ overrides: ceph: log-ignorelist: - - mons down - - mon down - - MON_DOWN - - out of quorum - - PG_AVAILABILITY - - PG_DEGRADED + - Telemetry requires re-opt-in + - telemetry module includes new collections tasks: - install: branch: reef diff --git a/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml b/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml index 5e995da7d2c..fa93b2f2ece 100644 --- a/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml +++ b/qa/suites/upgrade/reef-x/parallel/overrides/ignorelist_health.yaml @@ -1,20 +1,19 @@ overrides: ceph: log-ignorelist: - - \(MDS_ALL_DOWN\) - - \(MDS_UP_LESS_THAN_MAX\) - - \(OSD_SLOW_PING_TIME + - MDS_ALL_DOWN + - MDS_UP_LESS_THAN_MAX + - OSD_SLOW_PING_TIME - reached quota + - running out of quota - overall HEALTH_ - - \(CACHE_POOL_NO_HIT_SET\) - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(SLOW_OPS\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(OBJECT_MISPLACED\) + - CACHE_POOL_NO_HIT_SET + - pool\(s\) full + - POOL_FULL + - SMALLER_PGP_NUM + - SLOW_OPS + - CACHE_POOL_NEAR_FULL + - OBJECT_MISPLACED - slow request - - \(MON_DOWN\) - noscrub - nodeep-scrub diff --git a/qa/suites/upgrade/reef-x/parallel/workload/rados_api.yaml b/qa/suites/upgrade/reef-x/parallel/workload/rados_api.yaml index a46e34db5dd..79cf1a96601 100644 --- a/qa/suites/upgrade/reef-x/parallel/workload/rados_api.yaml +++ b/qa/suites/upgrade/reef-x/parallel/workload/rados_api.yaml @@ -9,4 +9,6 @@ workload: clients: client.0: - cls + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done end rados_api.yaml" diff --git a/qa/suites/upgrade/reef-x/stress-split/1-start.yaml b/qa/suites/upgrade/reef-x/stress-split/1-start.yaml index 992f9e1bc36..59ccfe2cd02 100644 --- a/qa/suites/upgrade/reef-x/stress-split/1-start.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/1-start.yaml @@ -1,11 +1,25 @@ overrides: ceph: log-ignorelist: + - do not have an application enabled + - application not enabled + - or freeform for custom applications + - POOL_APP_NOT_ENABLED + - is down + - OSD_DOWN - mons down - mon down - MON_DOWN - out of quorum - PG_AVAILABILITY + - PG_DEGRADED + - Reduced data availability + - Degraded data redundancy + - pg .* is stuck inactive + - pg .* is .*degraded + - FS_DEGRADED + - OSDMAP_FLAGS + - OSD_UPGRADE_FINISHED tasks: - install: branch: reef diff --git a/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-cls.yaml b/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-cls.yaml index f092096f444..79ad2af8ea1 100644 --- a/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-cls.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/2-first-half-tasks/rbd-cls.yaml @@ -7,4 +7,6 @@ first-half-tasks: clients: client.0: - cls/test_cls_rbd.sh + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done cls/test_cls_rbd.sh 5-workload" diff --git a/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-cls.yaml b/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-cls.yaml index 05bb672b3ac..166327a58f9 100644 --- a/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-cls.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/3-stress-tasks/rbd-cls.yaml @@ -7,4 +7,6 @@ stress-tasks: clients: client.0: - cls/test_cls_rbd.sh + env: + CLS_RBD_GTEST_FILTER: '-TestClsRbd.group_snap_set:TestClsRbd.group_snap_remove' - print: "**** done cls/test_cls_rbd.sh 5-workload" diff --git a/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml b/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml index 5e995da7d2c..fa93b2f2ece 100644 --- a/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml +++ b/qa/suites/upgrade/reef-x/stress-split/overrides/ignorelist_health.yaml @@ -1,20 +1,19 @@ overrides: ceph: log-ignorelist: - - \(MDS_ALL_DOWN\) - - \(MDS_UP_LESS_THAN_MAX\) - - \(OSD_SLOW_PING_TIME + - MDS_ALL_DOWN + - MDS_UP_LESS_THAN_MAX + - OSD_SLOW_PING_TIME - reached quota + - running out of quota - overall HEALTH_ - - \(CACHE_POOL_NO_HIT_SET\) - - \(POOL_FULL\) - - \(SMALLER_PGP_NUM\) - - \(SLOW_OPS\) - - \(CACHE_POOL_NEAR_FULL\) - - \(POOL_APP_NOT_ENABLED\) - - \(PG_AVAILABILITY\) - - \(OBJECT_MISPLACED\) + - CACHE_POOL_NO_HIT_SET + - pool\(s\) full + - POOL_FULL + - SMALLER_PGP_NUM + - SLOW_OPS + - CACHE_POOL_NEAR_FULL + - OBJECT_MISPLACED - slow request - - \(MON_DOWN\) - noscrub - nodeep-scrub diff --git a/qa/tasks/cephfs/test_exports.py b/qa/tasks/cephfs/test_exports.py index 346f139874b..468378fce3d 100644 --- a/qa/tasks/cephfs/test_exports.py +++ b/qa/tasks/cephfs/test_exports.py @@ -153,6 +153,8 @@ class TestExportPin(CephFSTestCase): # vstart.sh sets mds_debug_subtrees to True. That causes a ESubtreeMap # to be written out every event. Yuck! self.config_set('mds', 'mds_debug_subtrees', False) + # make sure ESubtreeMap is written frequently enough: + self.config_set('mds', 'mds_log_minor_segments_per_major_segment', '4') self.config_rm('mds', 'mds bal split size') # don't split /top self.mount_a.run_shell_payload("rm -rf 1") diff --git a/qa/tasks/cephfs/test_failover.py b/qa/tasks/cephfs/test_failover.py index 29af1e76a4f..46139163ddd 100644 --- a/qa/tasks/cephfs/test_failover.py +++ b/qa/tasks/cephfs/test_failover.py @@ -1,3 +1,4 @@ +import re import time import signal import logging @@ -342,6 +343,60 @@ class TestClusterResize(CephFSTestCase): self.fs.wait_for_daemons(timeout=90) +class TestFailoverBeaconHealth(CephFSTestCase): + CLIENTS_REQUIRED = 1 + MDSS_REQUIRED = 1 + + def initiate_journal_replay(self, num_files=100): + """ Initiate journal replay by creating files and restarting mds server.""" + + self.config_set("mds", "mds_delay_journal_replay_for_testing", "5000") + self.mounts[0].test_files = [str(x) for x in range(num_files)] + self.mounts[0].create_files() + self.fs.fail() + self.fs.set_joinable() + + def test_replay_beacon_estimated_time(self): + """ + That beacon emits warning message with estimated time to complete replay + """ + self.initiate_journal_replay() + self.wait_for_health("MDS_ESTIMATED_REPLAY_TIME", 60) + # remove the config so that replay finishes and the cluster + # is HEALTH_OK + self.config_rm("mds", "mds_delay_journal_replay_for_testing") + self.wait_for_health_clear(timeout=60) + + def test_replay_estimated_time_accuracy(self): + self.initiate_journal_replay(250) + def replay_complete(): + health = self.ceph_cluster.mon_manager.get_mon_health(debug=False, detail=True) + codes = [s for s in health['checks']] + return 'MDS_ESTIMATED_REPLAY_TIME' not in codes + + def get_estimated_time(): + completion_percentage = 0.0 + time_duration = pending_duration = 0 + with safe_while(sleep=5, tries=360) as proceed: + while proceed(): + health = self.ceph_cluster.mon_manager.get_mon_health(debug=False, detail=True) + codes = [s for s in health['checks']] + if 'MDS_ESTIMATED_REPLAY_TIME' in codes: + message = health['checks']['MDS_ESTIMATED_REPLAY_TIME']['detail'][0]['message'] + ### sample warning string: "mds.a(mds.0): replay: 50.0446% complete - elapsed time: 582s, estimated time remaining: 581s" + m = re.match(".* replay: (\d+(\.\d+)?)% complete - elapsed time: (\d+)s, estimated time remaining: (\d+)s", message) + if not m: + continue + completion_percentage = float(m.group(1)) + time_duration = int(m.group(3)) + pending_duration = int(m.group(4)) + log.debug(f"MDS_ESTIMATED_REPLAY_TIME is present in health: {message}, duration: {time_duration}, completion_percentage: {completion_percentage}") + if completion_percentage >= 50: + return (completion_percentage, time_duration, pending_duration) + _, _, pending_duration = get_estimated_time() + # wait for 25% more time to avoid false negative failures + self.wait_until_true(replay_complete, timeout=pending_duration * 1.25) + class TestFailover(CephFSTestCase): CLIENTS_REQUIRED = 1 MDSS_REQUIRED = 2 diff --git a/qa/tasks/cephfs/test_nfs.py b/qa/tasks/cephfs/test_nfs.py index faa35be6926..0a1c07dce04 100644 --- a/qa/tasks/cephfs/test_nfs.py +++ b/qa/tasks/cephfs/test_nfs.py @@ -369,6 +369,45 @@ class TestNFS(MgrTestCase): except CommandFailedError as e: self.fail(f"expected read/write of a file to be successful but failed with {e.exitstatus}") + def _mnt_nfs(self, pseudo_path, port, ip): + ''' + Mount created export + :param pseudo_path: It is the pseudo root name + :param port: Port of deployed nfs cluster + :param ip: IP of deployed nfs cluster + ''' + tries = 3 + while True: + try: + self.ctx.cluster.run( + args=['sudo', 'mount', '-t', 'nfs', '-o', f'port={port}', + f'{ip}:{pseudo_path}', '/mnt']) + break + except CommandFailedError: + if tries: + tries -= 1 + time.sleep(2) + continue + raise + + self.ctx.cluster.run(args=['sudo', 'chmod', '1777', '/mnt']) + + def _test_fio(self, pseudo_path, port, ip): + ''' + run fio with libaio on /mnt/fio + :param mnt_path: nfs mount point + ''' + try: + self._mnt_nfs(pseudo_path, port, ip) + self.ctx.cluster.run(args=['mkdir', '/mnt/fio']) + fio_cmd=['sudo', 'fio', '--ioengine=libaio', '-directory=/mnt/fio', '--filename=fio.randrw.test', '--name=job', '--bs=16k', '--direct=1', '--group_reporting', '--iodepth=128', '--randrepeat=0', '--norandommap=1', '--thread=2', '--ramp_time=20s', '--offset_increment=5%', '--size=5G', '--time_based', '--runtime=300', '--ramp_time=1s', '--percentage_random=0', '--rw=randrw', '--rwmixread=50'] + self.ctx.cluster.run(args=fio_cmd) + except CommandFailedError as e: + self.fail(f"expected fio to be successful but failed with {e.exitstatus}") + finally: + self.ctx.cluster.run(args=['sudo', 'rm', '-rf', '/mnt/fio']) + self.ctx.cluster.run(args=['sudo', 'umount', '/mnt']) + def _write_to_read_only_export(self, pseudo_path, port, ip): ''' Check if write to read only export fails @@ -627,6 +666,18 @@ class TestNFS(MgrTestCase): self._test_data_read_write(self.pseudo_path, port, ip) self._test_delete_cluster() + def test_async_io_fio(self): + ''' + Test async io using fio. Expect completion without hang or crash + ''' + self._test_create_cluster() + self._create_export(export_id='1', create_fs=True, + extra_cmd=['--pseudo-path', self.pseudo_path]) + port, ip = self._get_port_ip_info() + self._check_nfs_cluster_status('running', 'NFS Ganesha cluster restart failed') + self._test_fio(self.pseudo_path, port, ip) + self._test_delete_cluster() + def test_cluster_info(self): ''' Test cluster info outputs correct ip and hostname diff --git a/qa/tasks/check_counter.py b/qa/tasks/check_counter.py index 40818f3f475..1f63b6a0bd4 100644 --- a/qa/tasks/check_counter.py +++ b/qa/tasks/check_counter.py @@ -1,11 +1,14 @@ import logging import json +import errno from teuthology.task import Task from teuthology import misc from tasks import ceph_manager +from tasks.cephfs.filesystem import MDSCluster +from teuthology.exceptions import CommandFailedError log = logging.getLogger(__name__) @@ -61,6 +64,9 @@ class CheckCounter(Task): mon_manager = ceph_manager.CephManager(self.admin_remote, ctx=self.ctx, logger=log.getChild('ceph_manager')) active_mgr = json.loads(mon_manager.raw_cluster_cmd("mgr", "dump", "--format=json-pretty"))["active_name"] + mds_cluster = MDSCluster(self.ctx) + status = mds_cluster.status() + for daemon_type, counters in targets.items(): # List of 'a', 'b', 'c'... daemon_ids = list(misc.all_roles_of_type(self.ctx.cluster, daemon_type)) @@ -80,13 +86,31 @@ class CheckCounter(Task): else: log.debug("Getting stats from {0}".format(daemon_id)) - manager = self.ctx.managers[cluster_name] - proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) - response_data = proc.stdout.getvalue().strip() + if daemon_type == 'mds': + mds_info = status.get_mds(daemon_id) + if not mds_info: + continue + mds = f"mds.{mds_info['gid']}" + if mds_info['state'] != "up:active": + log.debug(f"skipping {mds}") + continue + log.debug(f"Getting stats from {mds}") + try: + proc = mon_manager.raw_cluster_cmd("tell", mds, "perf", "dump", + "--format=json-pretty") + response_data = proc.strip() + except CommandFailedError as e: + if e.exitstatus == errno.ENOENT: + log.debug(f"Failed to do 'perf dump' on {mds}") + continue + else: + manager = self.ctx.managers[cluster_name] + proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) + response_data = proc.stdout.getvalue().strip() if response_data: perf_dump = json.loads(response_data) else: - log.warning("No admin socket response from {0}, skipping".format(daemon_id)) + log.warning("No response from {0}, skipping".format(daemon_id)) continue minval = '' diff --git a/qa/tasks/kafka.py b/qa/tasks/kafka.py index 5e6c208ca30..833f03babf6 100644 --- a/qa/tasks/kafka.py +++ b/qa/tasks/kafka.py @@ -4,6 +4,7 @@ Deploy and configure Kafka for Teuthology import contextlib import logging import time +import os from teuthology import misc as teuthology from teuthology import contextutil @@ -33,6 +34,13 @@ def install_kafka(ctx, config): assert isinstance(config, dict) log.info('Installing Kafka...') + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + for (client, _) in config.items(): (remote,) = ctx.cluster.only(client).remotes.keys() test_dir=teuthology.get_testdir(ctx) @@ -40,7 +48,8 @@ def install_kafka(ctx, config): kafka_file = kafka_prefix + current_version + '.tgz' - link1 = 'https://archive.apache.org/dist/kafka/' + current_version + '/' + kafka_file + link1 = '{apache_mirror_url_front}/kafka/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + current_version + '/' + kafka_file ctx.cluster.only(client).run( args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'wget', link1], ) diff --git a/qa/tasks/kafka_failover.py b/qa/tasks/kafka_failover.py new file mode 100644 index 00000000000..3ca60ab84fc --- /dev/null +++ b/qa/tasks/kafka_failover.py @@ -0,0 +1,244 @@ +""" +Deploy and configure Kafka for Teuthology +""" +import contextlib +import logging +import time +import os + +from teuthology import misc as teuthology +from teuthology import contextutil +from teuthology.orchestra import run + +log = logging.getLogger(__name__) + +def get_kafka_version(config): + for client, client_config in config.items(): + if 'kafka_version' in client_config: + kafka_version = client_config.get('kafka_version') + return kafka_version + +kafka_prefix = 'kafka_2.13-' + +def get_kafka_dir(ctx, config): + kafka_version = get_kafka_version(config) + current_version = kafka_prefix + kafka_version + return '{tdir}/{ver}'.format(tdir=teuthology.get_testdir(ctx),ver=current_version) + + +@contextlib.contextmanager +def install_kafka(ctx, config): + """ + Downloading the kafka tar file. + """ + assert isinstance(config, dict) + log.info('Installing Kafka...') + + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + + for (client, _) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + test_dir=teuthology.get_testdir(ctx) + current_version = get_kafka_version(config) + + kafka_file = kafka_prefix + current_version + '.tgz' + + link1 = '{apache_mirror_url_front}/kafka/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + current_version + '/' + kafka_file + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'wget', link1], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=test_dir), run.Raw('&&'), 'tar', '-xvzf', kafka_file], + ) + + kafka_dir = get_kafka_dir(ctx, config) + # create config for second broker + second_broker_config_name = "server2.properties" + second_broker_data = "{tdir}/data/broker02".format(tdir=kafka_dir) + second_broker_data_logs_escaped = "{}/logs".format(second_broker_data).replace("/", "\/") + + ctx.cluster.only(client).run( + args=['cd', '{tdir}'.format(tdir=kafka_dir), run.Raw('&&'), + 'cp', '{tdir}/config/server.properties'.format(tdir=kafka_dir), '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'mkdir', '-p', '{tdir}/data'.format(tdir=kafka_dir) + ], + ) + + # edit config + ctx.cluster.only(client).run( + args=['sed', '-i', 's/broker.id=0/broker.id=1/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/#listeners=PLAINTEXT:\/\/:9092/listeners=PLAINTEXT:\/\/localhost:19092/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/#advertised.listeners=PLAINTEXT:\/\/your.host.name:9092/advertised.listeners=PLAINTEXT:\/\/localhost:19092/g', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'sed', '-i', 's/log.dirs=\/tmp\/kafka-logs/log.dirs={}/g'.format(second_broker_data_logs_escaped), '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name), run.Raw('&&'), + 'cat', '{tdir}/config/{second_broker_config_name}'.format(tdir=kafka_dir, second_broker_config_name=second_broker_config_name) + ] + ) + + try: + yield + finally: + log.info('Removing packaged dependencies of Kafka...') + test_dir=get_kafka_dir(ctx, config) + current_version = get_kafka_version(config) + for (client,_) in config.items(): + ctx.cluster.only(client).run( + args=['rm', '-rf', '{tdir}/logs'.format(tdir=test_dir)], + ) + + ctx.cluster.only(client).run( + args=['rm', '-rf', test_dir], + ) + + ctx.cluster.only(client).run( + args=['rm', '-rf', '{tdir}/{doc}'.format(tdir=teuthology.get_testdir(ctx),doc=kafka_file)], + ) + + +@contextlib.contextmanager +def run_kafka(ctx,config): + """ + This includes two parts: + 1. Starting Zookeeper service + 2. Starting Kafka service + """ + assert isinstance(config, dict) + log.info('Bringing up Zookeeper and Kafka services...') + for (client,_) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + kafka_dir = get_kafka_dir(ctx, config) + + second_broker_data = "{tdir}/data/broker02".format(tdir=kafka_dir) + second_broker_java_log_dir = "{}/java_logs".format(second_broker_data) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + './zookeeper-server-start.sh', + '{tir}/config/zookeeper.properties'.format(tir=kafka_dir), + run.Raw('&'), 'exit' + ], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + './kafka-server-start.sh', + '{tir}/config/server.properties'.format(tir=get_kafka_dir(ctx, config)), + run.Raw('&'), 'exit' + ], + ) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=kafka_dir), run.Raw('&&'), + run.Raw('LOG_DIR={second_broker_java_log_dir}'.format(second_broker_java_log_dir=second_broker_java_log_dir)), + './kafka-server-start.sh', '{tdir}/config/server2.properties'.format(tdir=kafka_dir), + run.Raw('&'), 'exit' + ], + ) + + try: + yield + finally: + log.info('Stopping Zookeeper and Kafka Services...') + + for (client, _) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-server-stop.sh', + '{tir}/config/kafka.properties'.format(tir=get_kafka_dir(ctx, config)), + ], + ) + + time.sleep(5) + + ctx.cluster.only(client).run( + args=['cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './zookeeper-server-stop.sh', + '{tir}/config/zookeeper.properties'.format(tir=get_kafka_dir(ctx, config)), + ], + ) + + time.sleep(5) + + ctx.cluster.only(client).run(args=['killall', '-9', 'java']) + + +@contextlib.contextmanager +def run_admin_cmds(ctx,config): + """ + Running Kafka Admin commands in order to check the working of producer anf consumer and creation of topic. + """ + assert isinstance(config, dict) + log.info('Checking kafka server through producer/consumer commands...') + for (client,_) in config.items(): + (remote,) = ctx.cluster.only(client).remotes.keys() + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-topics.sh', '--create', '--topic', 'quickstart-events', + '--bootstrap-server', 'localhost:9092' + ], + ) + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + 'echo', "First", run.Raw('|'), + './kafka-console-producer.sh', '--topic', 'quickstart-events', + '--bootstrap-server', 'localhost:9092' + ], + ) + + ctx.cluster.only(client).run( + args=[ + 'cd', '{tdir}/bin'.format(tdir=get_kafka_dir(ctx, config)), run.Raw('&&'), + './kafka-console-consumer.sh', '--topic', 'quickstart-events', + '--from-beginning', + '--bootstrap-server', 'localhost:9092', + run.Raw('&'), 'exit' + ], + ) + + try: + yield + finally: + pass + + +@contextlib.contextmanager +def task(ctx,config): + """ + Following is the way how to run kafka:: + tasks: + - kafka: + client.0: + kafka_version: 2.6.0 + """ + assert config is None or isinstance(config, list) \ + or isinstance(config, dict), \ + "task kafka only supports a list or dictionary for configuration" + + all_clients = ['client.{id}'.format(id=id_) + for id_ in teuthology.all_roles_of_type(ctx.cluster, 'client')] + if config is None: + config = all_clients + if isinstance(config, list): + config = dict.fromkeys(config) + + log.debug('Kafka config is %s', config) + + with contextutil.nested( + lambda: install_kafka(ctx=ctx, config=config), + lambda: run_kafka(ctx=ctx, config=config), + lambda: run_admin_cmds(ctx=ctx, config=config), + ): + yield + diff --git a/qa/tasks/mgr/dashboard/test_rbd.py b/qa/tasks/mgr/dashboard/test_rbd.py index a872645e33e..83b3bf520c2 100644 --- a/qa/tasks/mgr/dashboard/test_rbd.py +++ b/qa/tasks/mgr/dashboard/test_rbd.py @@ -869,7 +869,19 @@ class RbdTest(DashboardTestCase): self.assertEqual(clone_format_version, 2) self.assertStatus(200) + # if empty list is sent, then the config will remain as it is value = [] + res = [{'section': "global", 'value': "2"}] + self._post('/api/cluster_conf', { + 'name': config_name, + 'value': value + }) + self.wait_until_equal( + lambda: _get_config_by_name(config_name), + res, + timeout=60) + + value = [{'section': "global", 'value': ""}] self._post('/api/cluster_conf', { 'name': config_name, 'value': value diff --git a/qa/tasks/mgr/dashboard/test_rgw.py b/qa/tasks/mgr/dashboard/test_rgw.py index 5c7b0329675..a9071bc2a3a 100644 --- a/qa/tasks/mgr/dashboard/test_rgw.py +++ b/qa/tasks/mgr/dashboard/test_rgw.py @@ -785,7 +785,7 @@ class RgwUserSubuserTest(RgwTestCase): 'access': 'readwrite', 'key_type': 'swift' }) - self.assertStatus(200) + self.assertStatus(201) data = self.jsonBody() subuser = self.find_object_in_list('id', 'teuth-test-user:tux', data) self.assertIsInstance(subuser, object) @@ -808,7 +808,7 @@ class RgwUserSubuserTest(RgwTestCase): 'access_key': 'yyy', 'secret_key': 'xxx' }) - self.assertStatus(200) + self.assertStatus(201) data = self.jsonBody() subuser = self.find_object_in_list('id', 'teuth-test-user:hugo', data) self.assertIsInstance(subuser, object) diff --git a/qa/tasks/notification_tests.py b/qa/tasks/notification_tests.py index b4697a6f797..f1eae3c89c4 100644 --- a/qa/tasks/notification_tests.py +++ b/qa/tasks/notification_tests.py @@ -220,7 +220,7 @@ def run_tests(ctx, config): for client, client_config in config.items(): (remote,) = ctx.cluster.only(client).remotes.keys() - attr = ["!kafka_test", "!data_path_v2_kafka_test", "!amqp_test", "!amqp_ssl_test", "!kafka_security_test", "!modification_required", "!manual_test", "!http_test"] + attr = ["!kafka_test", "!data_path_v2_kafka_test", "!kafka_failover", "!amqp_test", "!amqp_ssl_test", "!kafka_security_test", "!modification_required", "!manual_test", "!http_test"] if 'extra_attr' in client_config: attr = client_config.get('extra_attr') diff --git a/qa/tasks/nvme_loop.py b/qa/tasks/nvme_loop.py index fef270ea085..fdec467a16d 100644 --- a/qa/tasks/nvme_loop.py +++ b/qa/tasks/nvme_loop.py @@ -70,7 +70,7 @@ def task(ctx, config): remote.run(args=['lsblk'], stdout=StringIO()) p = remote.run(args=['sudo', 'nvme', 'list', '-o', 'json'], stdout=StringIO()) new_devs = [] - # `nvme list -o json` will return the following output: + # `nvme list -o json` will return one of the following output: '''{ "Devices" : [ { @@ -91,13 +91,112 @@ def task(ctx, config): } ] }''' + '''{ + "Devices":[ + { + "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4", + "HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c", + "Subsystems":[ + { + "Subsystem":"nvme-subsys0", + "SubsystemNQN":"nqn.2014.08.org.nvmexpress:80868086CVFT623300LN400BGN INTEL SSDPEDMD400G4", + "Controllers":[ + { + "Controller":"nvme0", + "Cntlid":"0", + "SerialNumber":"CVFT623300LN400BGN", + "ModelNumber":"INTEL SSDPEDMD400G4", + "Firmware":"8DV101H0", + "Transport":"pcie", + "Address":"0000:02:00.0", + "Slot":"2", + "Namespaces":[ + { + "NameSpace":"nvme0n1", + "Generic":"ng0n1", + "NSID":1, + "UsedBytes":400088457216, + "MaximumLBA":781422768, + "PhysicalSize":400088457216, + "SectorSize":512 + } + ], + "Paths":[ + ] + } + ], + "Namespaces":[ + ] + } + ] + } + ] + } + ''' + '''{ + "Devices":[ + { + "HostNQN":"nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-0cc47ada6ba4", + "HostID":"898a0e10-da2d-4a42-8017-d9c445089d0c", + "Subsystems":[ + { + "Subsystem":"nvme-subsys0", + "SubsystemNQN":"nqn.2014.08.org.nvmexpress:80868086CVFT534400C2400BGN INTEL SSDPEDMD400G4", + "Controllers":[ + { + "Controller":"nvme0", + "Cntlid":"0", + "SerialNumber":"CVFT534400C2400BGN", + "ModelNumber":"INTEL SSDPEDMD400G4", + "Firmware":"8DV101H0", + "Transport":"pcie", + "Address":"0000:02:00.0", + "Slot":"2", + "Namespaces":[ + { + "NameSpace":"nvme0n1", + "Generic":"ng0n1", + "NSID":1, + "UsedBytes":400088457216, + "MaximumLBA":781422768, + "PhysicalSize":400088457216, + "SectorSize":512 + } + ], + "Paths":[ + ] + } + ], + "Namespaces":[ + ] + } + ] + } + ] + } + ''' nvme_list = json.loads(p.stdout.getvalue()) for device in nvme_list['Devices']: - dev = device['DevicePath'] - vendor = device['ModelNumber'] - if dev.startswith('/dev/') and vendor == 'Linux': - new_devs.append(dev) - bluestore_zap(remote, dev) + try: + # first try format 1 / older format + dev = device['DevicePath'] + vendor = device['ModelNumber'] + if dev.startswith('/dev/') and vendor == 'Linux': + new_devs.append(dev) + bluestore_zap(remote, dev) + except KeyError: + for subsystem in device['Subsystems']: + # format 2 + if 'Namespaces' in subsystem and subsystem['Namespaces']: + dev = '/dev/' + subsystem['Namespaces'][0]['NameSpace'] + # try format 3 last + else: + dev = '/dev/' + subsystem['Controllers'][0]['Namespaces'][0]['NameSpace'] + # vendor is the same for format 2 and 3 + vendor = subsystem['Controllers'][0]['ModelNumber'] + if vendor == 'Linux': + new_devs.append(dev) + bluestore_zap(remote, dev) log.info(f'new_devs {new_devs}') assert len(new_devs) <= len(devs) if len(new_devs) == len(devs): diff --git a/qa/tasks/nvmeof.py b/qa/tasks/nvmeof.py index 42e357294d9..691a6f7dd86 100644 --- a/qa/tasks/nvmeof.py +++ b/qa/tasks/nvmeof.py @@ -128,12 +128,11 @@ class Nvmeof(Task): total_images = int(self.namespaces_count) * int(self.subsystems_count) log.info(f'[nvmeof]: creating {total_images} images') + rbd_create_cmd = [] for i in range(1, total_images + 1): imagename = self.image_name_prefix + str(i) - log.info(f'[nvmeof]: rbd create {poolname}/{imagename} --size {self.rbd_size}') - _shell(self.ctx, self.cluster_name, self.remote, [ - 'rbd', 'create', f'{poolname}/{imagename}', '--size', f'{self.rbd_size}' - ]) + rbd_create_cmd += ['rbd', 'create', f'{poolname}/{imagename}', '--size', f'{self.rbd_size}', run.Raw(';')] + _shell(self.ctx, self.cluster_name, self.remote, rbd_create_cmd) for role, i in daemons.items(): remote, id_ = i @@ -251,9 +250,9 @@ class NvmeofThrasher(Thrasher, Greenlet): daemon_max_thrash_times: For now, NVMeoF daemons have limitation that each daemon can - be thrashed only 3 times in span of 30 mins. This option + be thrashed only 5 times in span of 30 mins. This option allows to set the amount of times it could be thrashed in a period - of time. (default: 3) + of time. (default: 5) daemon_max_thrash_period: This option goes with the above option. It sets the period of time over which each daemons can be thrashed for daemon_max_thrash_times @@ -306,17 +305,17 @@ class NvmeofThrasher(Thrasher, Greenlet): self.max_thrash_daemons = int(self.config.get('max_thrash', len(self.daemons) - 1)) # Limits on thrashing each daemon - self.daemon_max_thrash_times = int(self.config.get('daemon_max_thrash_times', 3)) + self.daemon_max_thrash_times = int(self.config.get('daemon_max_thrash_times', 5)) self.daemon_max_thrash_period = int(self.config.get('daemon_max_thrash_period', 30 * 60)) # seconds self.min_thrash_delay = int(self.config.get('min_thrash_delay', 60)) self.max_thrash_delay = int(self.config.get('max_thrash_delay', self.min_thrash_delay + 30)) - self.min_revive_delay = int(self.config.get('min_revive_delay', 100)) + self.min_revive_delay = int(self.config.get('min_revive_delay', 60)) self.max_revive_delay = int(self.config.get('max_revive_delay', self.min_revive_delay + 30)) def _get_devices(self, remote): GET_DEVICE_CMD = "sudo nvme list --output-format=json | " \ - "jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == \"Ceph bdev Controller\") | .DevicePath'" + "jq -r '.Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == \"Ceph bdev Controller\")) | .Namespaces | sort_by(.NSID) | .[] | .NameSpace'" devices = remote.sh(GET_DEVICE_CMD).split() return devices @@ -347,6 +346,7 @@ class NvmeofThrasher(Thrasher, Greenlet): run.Raw('&&'), 'ceph', 'orch', 'ps', '--daemon-type', 'nvmeof', run.Raw('&&'), 'ceph', 'health', 'detail', run.Raw('&&'), 'ceph', '-s', + run.Raw('&&'), 'sudo', 'nvme', 'list', ] for dev in self.devices: check_cmd += [ @@ -421,13 +421,11 @@ class NvmeofThrasher(Thrasher, Greenlet): while not self.stopping.is_set(): killed_daemons = defaultdict(list) - weight = 1.0 / len(self.daemons) - count = 0 + thrash_daemon_num = self.rng.randint(1, self.max_thrash_daemons) + selected_daemons = self.rng.sample(self.daemons, thrash_daemon_num) for daemon in self.daemons: - skip = self.rng.uniform(0.0, 1.0) - if weight <= skip: - self.log('skipping daemon {label} with skip ({skip}) > weight ({weight})'.format( - label=daemon.id_, skip=skip, weight=weight)) + if daemon not in selected_daemons: + self.log(f'skipping daemon {daemon.id_} ...') continue # For now, nvmeof daemons can only be thrashed 3 times in last 30mins. @@ -445,17 +443,11 @@ class NvmeofThrasher(Thrasher, Greenlet): continue self.log('kill {label}'.format(label=daemon.id_)) - # daemon.stop() kill_method = self.kill_daemon(daemon) killed_daemons[kill_method].append(daemon) daemons_thrash_history[daemon.id_] += [datetime.now()] - # only thrash max_thrash_daemons amount of daemons - count += 1 - if count >= self.max_thrash_daemons: - break - if killed_daemons: iteration_summary = "thrashed- " for kill_method in killed_daemons: @@ -468,7 +460,7 @@ class NvmeofThrasher(Thrasher, Greenlet): self.log(f'waiting for {revive_delay} secs before reviving') time.sleep(revive_delay) # blocking wait - self.log('done waiting before reviving') + self.log(f'done waiting before reviving - iteration #{len(summary)}: {iteration_summary}') self.do_checks() self.switch_task() @@ -487,7 +479,7 @@ class NvmeofThrasher(Thrasher, Greenlet): if thrash_delay > 0.0: self.log(f'waiting for {thrash_delay} secs before thrashing') time.sleep(thrash_delay) # blocking - self.log('done waiting before thrashing') + self.log('done waiting before thrashing - everything should be up now') self.do_checks() self.switch_task() diff --git a/qa/tasks/radosgw_admin.py b/qa/tasks/radosgw_admin.py index 3b98702acca..fb82378761b 100644 --- a/qa/tasks/radosgw_admin.py +++ b/qa/tasks/radosgw_admin.py @@ -16,6 +16,7 @@ import logging import time import datetime import sys +import errno from io import StringIO from queue import Queue @@ -725,6 +726,40 @@ def task(ctx, config): (err, out) = rgwadmin(ctx, client, ['user', 'rm', '--tenant', tenant_name, '--uid', 'tenanteduser'], check_status=True) + account_id = 'RGW12312312312312312' + account_name = 'testacct' + rgwadmin(ctx, client, [ + 'account', 'create', + '--account-id', account_id, + '--account-name', account_name, + ], check_status=True) + rgwadmin(ctx, client, [ + 'user', 'create', + '--account-id', account_id, + '--uid', 'testacctuser', + '--display-name', 'accountuser', + '--gen-access-key', + '--gen-secret', + ], check_status=True) + + # TESTCASE 'bucket link', 'bucket', 'account user', 'fails' + (err, out) = rgwadmin(ctx, client, ['bucket', 'link', '--bucket', bucket_name, '--uid', 'testacctuser']) + assert err == errno.EINVAL + + rgwadmin(ctx, client, ['user', 'rm', '--uid', 'testacctuser'], check_status=True) + + # TESTCASE 'bucket link', 'bucket', 'account', 'succeeds' + rgwadmin(ctx, client, + ['bucket', 'link', '--bucket', bucket_name, '--account-id', account_id], + check_status=True) + + # relink the bucket to the first user and delete the account + rgwadmin(ctx, client, + ['bucket', 'link', '--bucket', bucket_name, '--uid', user1], + check_status=True) + rgwadmin(ctx, client, ['account', 'rm', '--account-id', account_id], + check_status=True) + # TESTCASE 'object-rm', 'object', 'rm', 'remove object', 'succeeds, object is removed' # upload an object diff --git a/qa/tasks/rgw_multisite.py b/qa/tasks/rgw_multisite.py index e83a54efc2b..f93ca017fa2 100644 --- a/qa/tasks/rgw_multisite.py +++ b/qa/tasks/rgw_multisite.py @@ -361,6 +361,8 @@ def create_zonegroup(cluster, gateways, period, config): if endpoints: # replace client names with their gateway endpoints config['endpoints'] = extract_gateway_endpoints(gateways, endpoints) + if not config.get('api_name'): # otherwise it will be set to an empty string + config['api_name'] = config['name'] zonegroup = multisite.ZoneGroup(config['name'], period) # `zonegroup set` needs --default on command line, and 'is_master' in json args = is_default_arg(config) diff --git a/qa/tasks/s3a_hadoop.py b/qa/tasks/s3a_hadoop.py index 7b77359fcf2..4518a6f397c 100644 --- a/qa/tasks/s3a_hadoop.py +++ b/qa/tasks/s3a_hadoop.py @@ -1,5 +1,6 @@ import contextlib import logging +import os from teuthology import misc from teuthology.orchestra import run @@ -40,7 +41,7 @@ def task(ctx, config): # get versions maven_major = config.get('maven-major', 'maven-3') - maven_version = config.get('maven-version', '3.6.3') + maven_version = config.get('maven-version', '3.9.9') hadoop_ver = config.get('hadoop-version', '2.9.2') bucket_name = config.get('bucket-name', 's3atest') access_key = config.get('access-key', 'EGAQRD2ULOIFKFSKCT4F') @@ -48,11 +49,19 @@ def task(ctx, config): 'secret-key', 'zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb') + # programmatically find a nearby mirror so as not to hammer archive.apache.org + apache_mirror_cmd="curl 'https://www.apache.org/dyn/closer.cgi' 2>/dev/null | " \ + "grep -o '<strong>[^<]*</strong>' | sed 's/<[^>]*>//g' | head -n 1" + log.info("determining apache mirror by running: " + apache_mirror_cmd) + apache_mirror_url_front = os.popen(apache_mirror_cmd).read().rstrip() # note: includes trailing slash (/) + log.info("chosen apache mirror is " + apache_mirror_url_front) + # set versions for cloning the repo apache_maven = 'apache-maven-{maven_version}-bin.tar.gz'.format( maven_version=maven_version) - maven_link = 'http://archive.apache.org/dist/maven/' + \ - '{maven_major}/{maven_version}/binaries/'.format(maven_major=maven_major, maven_version=maven_version) + apache_maven + maven_link = '{apache_mirror_url_front}/maven/'.format(apache_mirror_url_front=apache_mirror_url_front) + \ + '{maven_major}/{maven_version}/binaries/'.format(maven_major=maven_major, maven_version=maven_version) + \ + apache_maven hadoop_git = 'https://github.com/apache/hadoop' hadoop_rel = 'hadoop-{ver} rel/release-{ver}'.format(ver=hadoop_ver) if hadoop_ver == 'trunk': @@ -204,6 +213,7 @@ def run_s3atest(client, maven_version, testdir, test_options): run.Raw('&&'), run.Raw(rm_test), run.Raw('&&'), + run.Raw('JAVA_HOME=$(alternatives --list | grep jre_1.8.0 | head -n 1 | awk \'{print $3}\')'), run.Raw(run_test), run.Raw(test_options) ] diff --git a/qa/tasks/s3tests.py b/qa/tasks/s3tests.py index 6d7b39d5892..85ab97d23cd 100644 --- a/qa/tasks/s3tests.py +++ b/qa/tasks/s3tests.py @@ -57,6 +57,17 @@ def download(ctx, config): 'git', 'reset', '--hard', sha1, ], ) + if client_config.get('boto3_extensions'): + ctx.cluster.only(client).run( + args=['mkdir', + '-p', + '/home/ubuntu/.aws/models/s3/2006-03-01/'] + ) + (remote,) = ctx.cluster.only(client).remotes.keys() + remote_file = '/home/ubuntu/.aws/models/s3/2006-03-01/service-2.sdk-extras.json' + local_file = '{qadir}/../examples/rgw/boto3/service-2.sdk-extras.json'.format(qadir=ctx.config.get('suite_path')) + remote.put_file(local_file, remote_file) + try: yield finally: @@ -70,6 +81,17 @@ def download(ctx, config): '{tdir}/s3-tests-{client}'.format(tdir=testdir, client=client), ], ) + if client_config.get('boto3_extensions'): + ctx.cluster.only(client).run( + args=[ + 'rm', '-rf', '/home/ubuntu/.aws/models/s3/2006-03-01/service-2.sdk-extras.json', + ], + ) + ctx.cluster.only(client).run( + args=[ + 'cd', '/home/ubuntu/', run.Raw('&&'), 'rmdir', '-p', '.aws/models/s3/2006-03-01/', + ], + ) def _config_user(s3tests_conf, section, user, email): @@ -444,8 +466,10 @@ def run_tests(ctx, config): attrs += ['not fails_with_subdomain'] if not client_config.get('with-sse-s3'): attrs += ['not sse_s3'] - + attrs += client_config.get('extra_attrs', []) + if 'bucket_logging' not in attrs: + attrs += ['not bucket_logging'] if 'unit_test_scan' in client_config and client_config['unit_test_scan']: xmlfile_id = datetime.datetime.now().strftime("%Y-%m-%d-%H:%M:%S--") + str(uuid.uuid4()) xmlpath= f'{testdir}/archive/s3test-{xmlfile_id}.xml' diff --git a/qa/tasks/s3tests_java.py b/qa/tasks/s3tests_java.py index 3e20e10d06c..a58aa6cf0b4 100644 --- a/qa/tasks/s3tests_java.py +++ b/qa/tasks/s3tests_java.py @@ -284,6 +284,7 @@ class S3tests_java(Task): args = ['cd', '{tdir}/s3-tests-java'.format(tdir=testdir), run.Raw('&&'), + run.Raw('JAVA_HOME=$(alternatives --list | grep jre_1.8.0 | head -n 1 | awk \'{print $3}\')'), '/opt/gradle/gradle/bin/gradle', 'clean', 'test', '--rerun-tasks', '--no-build-cache', ] diff --git a/qa/workunits/mon/mon-stretch-mode-5-mons-8-osds.sh b/qa/workunits/mon/mon-stretch-mode-5-mons-8-osds.sh index ded13854160..827fb0a0b13 100755 --- a/qa/workunits/mon/mon-stretch-mode-5-mons-8-osds.sh +++ b/qa/workunits/mon/mon-stretch-mode-5-mons-8-osds.sh @@ -9,6 +9,10 @@ if [ $NUM_OSDS_UP -lt 8 ]; then exit 1 fi +# ensure election strategy is set to "connectivity" +# See https://tracker.ceph.com/issues/69107 +ceph mon set election_strategy connectivity + for dc in dc1 dc2 do ceph osd crush add-bucket $dc datacenter @@ -65,4 +69,4 @@ EOF crushtool --compile crushmap_modified.txt -o crushmap.bin ceph osd setcrushmap -i crushmap.bin -ceph mon enable_stretch_mode e stretch_rule datacenter
\ No newline at end of file +ceph mon enable_stretch_mode e stretch_rule datacenter diff --git a/qa/workunits/nvmeof/basic_tests.sh b/qa/workunits/nvmeof/basic_tests.sh index dc6fd1669da..9e7a1f5134e 100755 --- a/qa/workunits/nvmeof/basic_tests.sh +++ b/qa/workunits/nvmeof/basic_tests.sh @@ -38,8 +38,10 @@ disconnect_all() { connect_all() { sudo nvme connect-all --traddr=$NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --transport=tcp -l 3600 sleep 5 - output=$(sudo nvme list --output-format=json) - if ! echo "$output" | grep -q "$SPDK_CONTROLLER"; then + expected_devices_count=$1 + actual_devices=$(sudo nvme list --output-format=json | jq -r ".Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == \"$SPDK_CONTROLLER\")) | .Namespaces[].NameSpace" | wc -l) + if [ "$actual_devices" -ne "$expected_devices_count" ]; then + sudo nvme list --output-format=json return 1 fi } @@ -72,11 +74,13 @@ test_run connect test_run list_subsys 1 test_run disconnect_all test_run list_subsys 0 -test_run connect_all +devices_count=$(( $NVMEOF_NAMESPACES_COUNT * $NVMEOF_SUBSYSTEMS_COUNT )) +test_run connect_all $devices_count gateways_count=$(( $(echo "$NVMEOF_GATEWAY_IP_ADDRESSES" | tr -cd ',' | wc -c) + 1 )) multipath_count=$(( $gateways_count * $NVMEOF_SUBSYSTEMS_COUNT)) test_run list_subsys $multipath_count + echo "-------------Test Summary-------------" echo "[nvmeof] All nvmeof basic tests passed!" diff --git a/qa/workunits/nvmeof/fio_test.sh b/qa/workunits/nvmeof/fio_test.sh index 57d355a6318..f7f783afc67 100755 --- a/qa/workunits/nvmeof/fio_test.sh +++ b/qa/workunits/nvmeof/fio_test.sh @@ -5,6 +5,7 @@ sudo yum -y install sysstat namespace_range_start= namespace_range_end= +random_devices_count= rbd_iostat=false while [[ $# -gt 0 ]]; do @@ -17,6 +18,10 @@ while [[ $# -gt 0 ]]; do namespace_range_end=$2 shift 2 ;; + --random_devices) + random_devices_count=$2 + shift 2 + ;; --rbd_iostat) rbd_iostat=true shift @@ -29,7 +34,7 @@ done fio_file=$(mktemp -t nvmeof-fio-XXXX) all_drives_list=$(sudo nvme list --output-format=json | - jq -r '.Devices | sort_by(.NameSpace) | .[] | select(.ModelNumber == "Ceph bdev Controller") | .DevicePath') + jq -r '.Devices[].Subsystems[] | select(.Controllers | all(.ModelNumber == "Ceph bdev Controller")) | .Namespaces | sort_by(.NSID) | .[] | .NameSpace') # When the script is passed --start_ns and --end_ns (example: `nvmeof_fio_test.sh --start_ns 1 --end_ns 3`), # then fio runs on namespaces only in the defined range (which is 1 to 3 here). @@ -37,6 +42,8 @@ all_drives_list=$(sudo nvme list --output-format=json | # run on first 3 namespaces here. if [ "$namespace_range_start" ] || [ "$namespace_range_end" ]; then selected_drives=$(echo "${all_drives_list[@]}" | sed -n "${namespace_range_start},${namespace_range_end}p") +elif [ "$random_devices_count" ]; then + selected_drives=$(echo "${all_drives_list[@]}" | shuf -n $random_devices_count) else selected_drives="${all_drives_list[@]}" fi diff --git a/qa/workunits/nvmeof/scalability_test.sh b/qa/workunits/nvmeof/scalability_test.sh index 5a26b6284f7..8ede4b7eda2 100755 --- a/qa/workunits/nvmeof/scalability_test.sh +++ b/qa/workunits/nvmeof/scalability_test.sh @@ -3,37 +3,64 @@ GATEWAYS=$1 # exmaple "nvmeof.a,nvmeof.b" DELAY="${SCALING_DELAYS:-50}" +POOL="${RBD_POOL:-mypool}" +GROUP="${NVMEOF_GROUP:-mygroup0}" +source /etc/ceph/nvmeof.env if [ -z "$GATEWAYS" ]; then echo "At least one gateway needs to be defined for scalability test" exit 1 fi -pip3 install yq - status_checks() { - ceph nvme-gw show mypool '' - ceph orch ls - ceph orch ps - ceph -s + expected_count=$1 + + output=$(ceph nvme-gw show $POOL $GROUP) + nvme_show=$(echo $output | grep -o '"AVAILABLE"' | wc -l) + if [ "$nvme_show" -ne "$expected_count" ]; then + return 1 + fi + + orch_ls=$(ceph orch ls) + if ! echo "$orch_ls" | grep -q "$expected_count/$expected_count"; then + return 1 + fi + + output=$(ceph orch ps --service-name nvmeof.$POOL.$GROUP) + orch_ps=$(echo $output | grep -o 'running' | wc -l) + if [ "$orch_ps" -ne "$expected_count" ]; then + return 1 + fi + + ceph_status=$(ceph -s) + if ! echo "$ceph_status" | grep -q "HEALTH_OK"; then + return 1 + fi } +total_gateways_count=$(( $(echo "$NVMEOF_GATEWAY_IP_ADDRESSES" | tr -cd ',' | wc -c) + 1 )) +scaled_down_gateways_count=$(( total_gateways_count - $(echo "$GATEWAYS" | tr -cd ',' | wc -c) - 1 )) + echo "[nvmeof.scale] Setting up config to remove gateways ${GATEWAYS}" +ceph orch ls --service-name nvmeof.$POOL.$GROUP --export > /tmp/nvmeof-gw.yaml ceph orch ls nvmeof --export > /tmp/nvmeof-gw.yaml cat /tmp/nvmeof-gw.yaml -yq "del(.placement.hosts[] | select(. | test(\".*($(echo $GATEWAYS | sed 's/,/|/g'))\")))" /tmp/nvmeof-gw.yaml > /tmp/nvmeof-gw-new.yaml + +pattern=$(echo $GATEWAYS | sed 's/,/\\|/g') +sed "/$pattern/d" /tmp/nvmeof-gw.yaml > /tmp/nvmeof-gw-new.yaml cat /tmp/nvmeof-gw-new.yaml echo "[nvmeof.scale] Starting scale testing by removing ${GATEWAYS}" -status_checks -ceph orch rm nvmeof.mypool && sleep 20 # temp workaround +status_checks $total_gateways_count ceph orch apply -i /tmp/nvmeof-gw-new.yaml # downscale +ceph orch redeploy nvmeof.$POOL.$GROUP sleep $DELAY -status_checks -ceph orch rm nvmeof.mypool && sleep 20 # temp workaround +status_checks $scaled_down_gateways_count +echo "[nvmeof.scale] Downscale complete - removed gateways (${GATEWAYS}); now scaling back up" ceph orch apply -i /tmp/nvmeof-gw.yaml #upscale +ceph orch redeploy nvmeof.$POOL.$GROUP sleep $DELAY -status_checks +status_checks $total_gateways_count echo "[nvmeof.scale] Scale testing passed for ${GATEWAYS}" diff --git a/qa/workunits/nvmeof/setup_subsystem.sh b/qa/workunits/nvmeof/setup_subsystem.sh index cc4024323eb..b573647b1e3 100755 --- a/qa/workunits/nvmeof/setup_subsystem.sh +++ b/qa/workunits/nvmeof/setup_subsystem.sh @@ -26,14 +26,21 @@ list_subsystems () { done } +list_namespaces () { + for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do + subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" + sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT --format plain namespace list --subsystem $subsystem_nqn + done +} + +echo "[nvmeof] Starting subsystem setup..." + # add all subsystems for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT subsystem add --subsystem $subsystem_nqn --no-group-append done -list_subsystems - # add all gateway listeners for i in "${!gateway_ips[@]}" do @@ -65,11 +72,5 @@ done list_subsystems -# list namespaces -for i in $(seq 1 $NVMEOF_SUBSYSTEMS_COUNT); do - subsystem_nqn="${NVMEOF_SUBSYSTEMS_PREFIX}${i}" - sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $NVMEOF_DEFAULT_GATEWAY_IP_ADDRESS --server-port $NVMEOF_SRPORT --format plain namespace list --subsystem $subsystem_nqn -done - echo "[nvmeof] Subsystem setup done" diff --git a/qa/workunits/rbd/cli_generic.sh b/qa/workunits/rbd/cli_generic.sh index 2aa27d3d655..0ceb9ff54cf 100755 --- a/qa/workunits/rbd/cli_generic.sh +++ b/qa/workunits/rbd/cli_generic.sh @@ -914,6 +914,11 @@ test_namespace() { rbd group create rbd/test1/group1 rbd group image add rbd/test1/group1 rbd/test1/image1 + rbd group image add --group-pool rbd --group-namespace test1 --group group1 \ + --image-pool rbd --image-namespace test1 --image image2 + rbd group image rm --group-pool rbd --group-namespace test1 --group group1 \ + --image-pool rbd --image-namespace test1 --image image1 + rbd group image rm rbd/test1/group1 rbd/test1/image2 rbd group rm rbd/test1/group1 rbd trash move rbd/test1/image1 diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 43bab75680d..9cbe350b388 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -16,6 +16,7 @@ endif() set(bindir ${CMAKE_INSTALL_FULL_BINDIR}) set(sbindir ${CMAKE_INSTALL_FULL_SBINDIR}) set(libdir ${CMAKE_INSTALL_FULL_LIBDIR}) +set(includedir ${CMAKE_INSTALL_FULL_INCLUDEDIR}) set(sysconfdir ${CMAKE_INSTALL_FULL_SYSCONFDIR}) set(libexecdir ${CMAKE_INSTALL_FULL_LIBEXECDIR}) set(pkgdatadir ${CMAKE_INSTALL_FULL_DATADIR}) @@ -31,6 +32,12 @@ configure_file(ceph-post-file.in configure_file(ceph-crash.in ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/ceph-crash @ONLY) +if(WITH_LIBCEPHFS) + configure_file( + ${CMAKE_SOURCE_DIR}/src/cephfs.pc.in + ${CMAKE_BINARY_DIR}/src/cephfs.pc @ONLY) +endif(WITH_LIBCEPHFS) + # the src/.git_version file may be written out by make-dist; otherwise # we pull the git version from .git option(ENABLE_GIT_VERSION "build Ceph with git version string" ON) @@ -832,10 +839,12 @@ if(WITH_LIBCEPHFS) target_link_libraries(cephfs PRIVATE client ceph-common ${CRYPTO_LIBS} ${EXTRALIBS}) if(ENABLE_SHARED) + set(libcephfs_version 2.0.0) + set(libcephfs_soversion 2) set_target_properties(cephfs PROPERTIES OUTPUT_NAME cephfs - VERSION 2.0.0 - SOVERSION 2) + VERSION ${libcephfs_version} + SOVERSION ${libcephfs_soversion}) if(NOT APPLE AND NOT (WIN32 AND CMAKE_CXX_COMPILER_ID STREQUAL Clang)) foreach(name ceph-common client osdc) @@ -848,6 +857,9 @@ if(WITH_LIBCEPHFS) install(DIRECTORY "include/cephfs" DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}) + install(FILES + ${CMAKE_BINARY_DIR}/src/cephfs.pc + DESTINATION ${CMAKE_INSTALL_LIBDIR}/pkgconfig) set(ceph_syn_srcs ceph_syn.cc client/SyntheticClient.cc) @@ -857,6 +869,7 @@ if(WITH_LIBCEPHFS) if(LINUX) add_subdirectory(mount) endif() + add_subdirectory(libcephfs_proxy) endif(WITH_LIBCEPHFS) if(WITH_LIBCEPHSQLITE) diff --git a/src/bash_completion/radosgw-admin b/src/bash_completion/radosgw-admin index 023a83f87e4..d9e36d8ef29 100644 --- a/src/bash_completion/radosgw-admin +++ b/src/bash_completion/radosgw-admin @@ -19,7 +19,7 @@ _radosgw_admin() if [[ ${cur} == -* ]] ; then COMPREPLY=( $(compgen -W "--uid --subuser --access-key --os-user --email --auth_uid --secret --os-secret --gen-access-key --gen-secret \ - --access --display-name --bucket --object --date --conf --name --id --version -s -w" -- ${cur}) ) + --access --display-name --bucket --object --date --conf --name --id --version -s -w --generate-key" -- ${cur}) ) return 0 fi diff --git a/src/ceph-volume/ceph_volume/devices/lvm/zap.py b/src/ceph-volume/ceph_volume/devices/lvm/zap.py index c278de43eb0..a6d82c7f0fa 100644 --- a/src/ceph-volume/ceph_volume/devices/lvm/zap.py +++ b/src/ceph-volume/ceph_volume/devices/lvm/zap.py @@ -119,13 +119,12 @@ class Zap: osd_uuid = details.get('osd_uuid') break - for osd_uuid, details in raw_report.items(): + for _, details in raw_report.items(): device: str = details.get('device') if details.get('osd_uuid') == osd_uuid: raw_devices.add(device) return list(raw_devices) - def find_associated_devices(self) -> List[api.Volume]: """From an ``osd_id`` and/or an ``osd_fsid``, filter out all the Logical Volumes (LVs) in the diff --git a/src/ceph-volume/ceph_volume/main.py b/src/ceph-volume/ceph_volume/main.py index f8eca65ec49..4f27f429e89 100644 --- a/src/ceph-volume/ceph_volume/main.py +++ b/src/ceph-volume/ceph_volume/main.py @@ -11,8 +11,16 @@ try: from importlib.metadata import entry_points def get_entry_points(group: str): # type: ignore - return entry_points().get(group, []) # type: ignore + eps = entry_points() + if hasattr(eps, 'select'): + # New importlib.metadata uses .select() + return eps.select(group=group) + else: + # Fallback to older EntryPoints that returns dicts + return eps.get(group, []) # type: ignore + except ImportError: + # Fallback to `pkg_resources` for older versions from pkg_resources import iter_entry_points as entry_points # type: ignore def get_entry_points(group: str): # type: ignore diff --git a/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py b/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py index cca64e83ab0..c971b7776ef 100644 --- a/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py +++ b/src/ceph-volume/ceph_volume/tests/devices/lvm/data_zap.py @@ -22,7 +22,7 @@ ceph_bluestore_tool_output = ''' "whoami": "0" }, "/dev/vdx": { - "osd_uuid": "d5a496bc-dcb9-4ad0-a12c-393d3200d2b6", + "osd_uuid": "d5a496bc-dcb9-4ad0-a12c-393d3200d2b7", "size": 214748364800, "btime": "2024-10-16T10:51:05.955279+0000", "description": "main", diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 77b55314f66..921e61a4534 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -347,12 +347,21 @@ def lsblk_all(device: str = '', return result -def is_device(dev): +def is_device(dev: str) -> bool: """ - Boolean to determine if a given device is a block device (**not** - a partition!) + Determines whether the given path corresponds to a block device (not a partition). - For example: /dev/sda would return True, but not /dev/sdc1 + This function checks whether the provided device path represents a valid block device, + such as a physical disk (/dev/sda) or an allowed loop device, but excludes partitions + (/dev/sdc1). It performs several validation steps, including file existence, path format, + device type, and additional checks for loop devices if allowed. + + Args: + dev (str): The path to the device (e.g., "/dev/sda"). + + Returns: + bool: True if the path corresponds to a valid block device (not a partition), + otherwise False. """ if not os.path.exists(dev): return False @@ -364,7 +373,7 @@ def is_device(dev): TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'loop'] # fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev) diff --git a/src/ceph-volume/ceph_volume/util/prepare.py b/src/ceph-volume/ceph_volume/util/prepare.py index 9c863b83d93..ff7fc023fc4 100644 --- a/src/ceph-volume/ceph_volume/util/prepare.py +++ b/src/ceph-volume/ceph_volume/util/prepare.py @@ -9,6 +9,7 @@ import logging import json from ceph_volume import process, conf, terminal from ceph_volume.util import system, constants, str_to_int, disk +from typing import Optional logger = logging.getLogger(__name__) mlogger = terminal.MultiLogger(__name__) @@ -121,7 +122,7 @@ def get_block_wal_size(lv_format=True): return wal_size -def create_id(fsid, json_secrets, osd_id=None): +def create_id(fsid: str, json_secrets: str, osd_id: Optional[str]=None) -> str: """ :param fsid: The osd fsid to create, always required :param json_secrets: a json-ready object with whatever secrets are wanted diff --git a/src/cephadm/cephadm.py b/src/cephadm/cephadm.py index d2ddf564116..a8616980e4d 100755 --- a/src/cephadm/cephadm.py +++ b/src/cephadm/cephadm.py @@ -111,6 +111,7 @@ from cephadmlib.file_utils import ( unlink_file, write_new, write_tmp, + update_meta_file, ) from cephadmlib.net_utils import ( build_addrv_params, @@ -3453,6 +3454,7 @@ def list_daemons( detail: bool = True, legacy_dir: Optional[str] = None, daemon_name: Optional[str] = None, + type_of_daemon: Optional[str] = None, ) -> List[Dict[str, str]]: host_version: Optional[str] = None ls = [] @@ -3489,6 +3491,8 @@ def list_daemons( if os.path.exists(data_dir): for i in os.listdir(data_dir): if i in ['mon', 'osd', 'mds', 'mgr', 'rgw']: + if type_of_daemon and type_of_daemon != i: + continue daemon_type = i for j in os.listdir(os.path.join(data_dir, i)): if '-' not in j: @@ -3525,6 +3529,8 @@ def list_daemons( if daemon_name and name != daemon_name: continue (daemon_type, daemon_id) = j.split('.', 1) + if type_of_daemon and type_of_daemon != daemon_type: + continue unit_name = get_unit_name(fsid, daemon_type, daemon_id) @@ -4705,6 +4711,34 @@ def command_list_images(ctx: CephadmContext) -> None: # print default images cp_obj.write(sys.stdout) + +def update_service_for_daemon(ctx: CephadmContext, + available_daemons: list, + update_daemons: list) -> None: + """ Update the unit.meta file of daemon with required service name for valid daemons""" + + data = {'service_name': ctx.service_name} + # check if all the daemon names are valid + if not set(update_daemons).issubset(set(available_daemons)): + raise Error(f'Error EINVAL: one or more daemons of {update_daemons} does not exist on this host') + for name in update_daemons: + path = os.path.join(ctx.data_dir, ctx.fsid, name, 'unit.meta') + update_meta_file(path, data) + print(f'Successfully updated daemon {name} with service {ctx.service_name}') + + +@infer_fsid +def command_update_osd_service(ctx: CephadmContext) -> int: + """update service for provided daemon""" + update_daemons = [f'osd.{osd_id}' for osd_id in ctx.osd_ids.split(',')] + daemons = list_daemons(ctx, detail=False, type_of_daemon='osd') + if not daemons: + raise Error(f'Daemon {ctx.osd_ids} does not exists on this host') + available_daemons = [d['name'] for d in daemons] + update_service_for_daemon(ctx, available_daemons, update_daemons) + return 0 + + ################################## @@ -5571,6 +5605,14 @@ def _get_parser(): parser_list_images = subparsers.add_parser( 'list-images', help='list all the default images') parser_list_images.set_defaults(func=command_list_images) + + parser_update_service = subparsers.add_parser( + 'update-osd-service', help='update service for provided daemon') + parser_update_service.set_defaults(func=command_update_osd_service) + parser_update_service.add_argument('--fsid', help='cluster FSID') + parser_update_service.add_argument('--osd-ids', required=True, help='Comma-separated OSD IDs') + parser_update_service.add_argument('--service-name', required=True, help='OSD service name') + return parser diff --git a/src/cephadm/cephadmlib/container_types.py b/src/cephadm/cephadmlib/container_types.py index f1e829cbdf7..52a68888e78 100644 --- a/src/cephadm/cephadmlib/container_types.py +++ b/src/cephadm/cephadmlib/container_types.py @@ -9,7 +9,7 @@ from typing import Dict, List, Optional, Any, Union, Tuple, Iterable, cast from .call_wrappers import call, call_throws, CallVerbosity from .constants import DEFAULT_TIMEOUT -import ceph.cephadm.images as default_images +from ceph.cephadm.images import DefaultImages from .container_engines import Docker, Podman from .context import CephadmContext from .daemon_identity import DaemonIdentity, DaemonSubIdentity @@ -665,14 +665,8 @@ def enable_shared_namespaces( def get_mgr_images() -> dict: """Return dict of default mgr images""" - mgr_prefix = 'mgr/cephadm/container_image_' - mgr_images = {} - images = vars(default_images) - for key, value in images.items(): - if key.startswith('DEFAULT_') and key.endswith('_IMAGE'): - # flake8 and black disagree about spaces around ":" hence the noqa comment - suffix = key[ - len('DEFAULT_') : -len('_IMAGE') # noqa: E203 - ].lower() - mgr_images[mgr_prefix + suffix] = value + mgr_prefix = 'mgr/cephadm/' + mgr_images = { + f'{mgr_prefix}{image.key}': image.image_ref for image in DefaultImages + } return mgr_images diff --git a/src/cephadm/cephadmlib/daemons/ingress.py b/src/cephadm/cephadmlib/daemons/ingress.py index c88e39ac025..645654b59c8 100644 --- a/src/cephadm/cephadmlib/daemons/ingress.py +++ b/src/cephadm/cephadmlib/daemons/ingress.py @@ -2,10 +2,7 @@ import os from typing import Dict, List, Optional, Tuple, Union -from ceph.cephadm.images import ( - DEFAULT_HAPROXY_IMAGE, - DEFAULT_KEEPALIVED_IMAGE, -) +from ceph.cephadm.images import DefaultImages from ..constants import ( DATA_DIR_MODE, ) @@ -27,7 +24,7 @@ class HAproxy(ContainerDaemonForm): daemon_type = 'haproxy' required_files = ['haproxy.cfg'] - default_image = DEFAULT_HAPROXY_IMAGE + default_image = DefaultImages.HAPROXY.image_ref @classmethod def for_daemon_type(cls, daemon_type: str) -> bool: @@ -155,7 +152,7 @@ class Keepalived(ContainerDaemonForm): daemon_type = 'keepalived' required_files = ['keepalived.conf'] - default_image = DEFAULT_KEEPALIVED_IMAGE + default_image = DefaultImages.KEEPALIVED.image_ref @classmethod def for_daemon_type(cls, daemon_type: str) -> bool: diff --git a/src/cephadm/cephadmlib/daemons/mgmt_gateway.py b/src/cephadm/cephadmlib/daemons/mgmt_gateway.py index 85f72495909..2be18809aa3 100644 --- a/src/cephadm/cephadmlib/daemons/mgmt_gateway.py +++ b/src/cephadm/cephadmlib/daemons/mgmt_gateway.py @@ -11,7 +11,7 @@ from ..context_getters import fetch_configs from ..daemon_form import register as register_daemon_form from ..daemon_identity import DaemonIdentity from ..deployment_utils import to_deployment_container -from ceph.cephadm.images import DEFAULT_NGINX_IMAGE +from ceph.cephadm.images import DefaultImages from ..data_utils import dict_get, is_fsid from ..file_utils import populate_files, makedirs, recursive_chown from ..exceptions import Error @@ -32,7 +32,7 @@ class MgmtGateway(ContainerDaemonForm): 'nginx_internal.key', ] - default_image = DEFAULT_NGINX_IMAGE + default_image = DefaultImages.NGINX.image_ref @classmethod def for_daemon_type(cls, daemon_type: str) -> bool: @@ -44,7 +44,7 @@ class MgmtGateway(ContainerDaemonForm): fsid: str, daemon_id: str, config_json: Dict, - image: str = DEFAULT_NGINX_IMAGE, + image: str = DefaultImages.NGINX.image_ref, ): self.ctx = ctx self.fsid = fsid diff --git a/src/cephadm/cephadmlib/daemons/monitoring.py b/src/cephadm/cephadmlib/daemons/monitoring.py index 710093f0f46..4ba00daaefb 100644 --- a/src/cephadm/cephadmlib/daemons/monitoring.py +++ b/src/cephadm/cephadmlib/daemons/monitoring.py @@ -3,14 +3,7 @@ import os from typing import Dict, List, Tuple from ..call_wrappers import call, CallVerbosity -from ceph.cephadm.images import ( - DEFAULT_ALERTMANAGER_IMAGE, - DEFAULT_GRAFANA_IMAGE, - DEFAULT_LOKI_IMAGE, - DEFAULT_NODE_EXPORTER_IMAGE, - DEFAULT_PROMETHEUS_IMAGE, - DEFAULT_PROMTAIL_IMAGE, -) +from ceph.cephadm.images import DefaultImages from ..constants import ( UID_NOBODY, GID_NOGROUP, @@ -23,7 +16,13 @@ from ..daemon_form import register as register_daemon_form from ..daemon_identity import DaemonIdentity from ..deployment_utils import to_deployment_container from ..exceptions import Error -from ..net_utils import get_fqdn, get_hostname, get_ip_addresses, wrap_ipv6 +from ..net_utils import ( + get_fqdn, + get_hostname, + get_ip_addresses, + wrap_ipv6, + EndPoint, +) @register_daemon_form @@ -43,7 +42,7 @@ class Monitoring(ContainerDaemonForm): components = { 'prometheus': { - 'image': DEFAULT_PROMETHEUS_IMAGE, + 'image': DefaultImages.PROMETHEUS.image_ref, 'cpus': '2', 'memory': '4GB', 'args': [ @@ -55,7 +54,7 @@ class Monitoring(ContainerDaemonForm): ], }, 'loki': { - 'image': DEFAULT_LOKI_IMAGE, + 'image': DefaultImages.LOKI.image_ref, 'cpus': '1', 'memory': '1GB', 'args': [ @@ -64,7 +63,7 @@ class Monitoring(ContainerDaemonForm): 'config-json-files': ['loki.yml'], }, 'promtail': { - 'image': DEFAULT_PROMTAIL_IMAGE, + 'image': DefaultImages.PROMTAIL.image_ref, 'cpus': '1', 'memory': '1GB', 'args': [ @@ -75,13 +74,13 @@ class Monitoring(ContainerDaemonForm): ], }, 'node-exporter': { - 'image': DEFAULT_NODE_EXPORTER_IMAGE, + 'image': DefaultImages.NODE_EXPORTER.image_ref, 'cpus': '1', 'memory': '1GB', 'args': ['--no-collector.timex'], }, 'grafana': { - 'image': DEFAULT_GRAFANA_IMAGE, + 'image': DefaultImages.GRAFANA.image_ref, 'cpus': '2', 'memory': '4GB', 'args': [], @@ -93,14 +92,9 @@ class Monitoring(ContainerDaemonForm): ], }, 'alertmanager': { - 'image': DEFAULT_ALERTMANAGER_IMAGE, + 'image': DefaultImages.ALERTMANAGER.image_ref, 'cpus': '2', 'memory': '2GB', - 'args': [ - '--cluster.listen-address=:{}'.format( - port_map['alertmanager'][1] - ), - ], 'config-json-files': [ 'alertmanager.yml', ], @@ -255,11 +249,14 @@ class Monitoring(ContainerDaemonForm): ip = meta['ip'] if 'ports' in meta and meta['ports']: port = meta['ports'][0] - if daemon_type == 'prometheus': - config = fetch_configs(ctx) + config = fetch_configs(ctx) + if daemon_type in ['prometheus', 'alertmanager']: ip_to_bind_to = config.get('ip_to_bind_to', '') if ip_to_bind_to: ip = ip_to_bind_to + web_listen_addr = str(EndPoint(ip, port)) + r += [f'--web.listen-address={web_listen_addr}'] + if daemon_type == 'prometheus': retention_time = config.get('retention_time', '15d') retention_size = config.get( 'retention_size', '0' @@ -283,9 +280,11 @@ class Monitoring(ContainerDaemonForm): r += ['--web.route-prefix=/prometheus/'] else: r += [f'--web.external-url={scheme}://{host}:{port}'] - r += [f'--web.listen-address={ip}:{port}'] if daemon_type == 'alertmanager': - config = fetch_configs(ctx) + clus_listen_addr = str( + EndPoint(ip, self.port_map[daemon_type][1]) + ) + r += [f'--cluster.listen-address={clus_listen_addr}'] use_url_prefix = config.get('use_url_prefix', False) peers = config.get('peers', list()) # type: ignore for peer in peers: @@ -301,13 +300,11 @@ class Monitoring(ContainerDaemonForm): if daemon_type == 'promtail': r += ['--config.expand-env'] if daemon_type == 'prometheus': - config = fetch_configs(ctx) try: r += [f'--web.config.file={config["web_config"]}'] except KeyError: pass if daemon_type == 'node-exporter': - config = fetch_configs(ctx) try: r += [f'--web.config.file={config["web_config"]}'] except KeyError: diff --git a/src/cephadm/cephadmlib/daemons/nvmeof.py b/src/cephadm/cephadmlib/daemons/nvmeof.py index d916c7e6391..51b085df2a7 100644 --- a/src/cephadm/cephadmlib/daemons/nvmeof.py +++ b/src/cephadm/cephadmlib/daemons/nvmeof.py @@ -8,7 +8,7 @@ from ..container_types import CephContainer from ..context_getters import fetch_configs, get_config_and_keyring from ..daemon_form import register as register_daemon_form from ..daemon_identity import DaemonIdentity -from ceph.cephadm.images import DEFAULT_NVMEOF_IMAGE +from ceph.cephadm.images import DefaultImages from ..context import CephadmContext from ..data_utils import dict_get, is_fsid from ..deployment_utils import to_deployment_container @@ -26,7 +26,7 @@ class CephNvmeof(ContainerDaemonForm): daemon_type = 'nvmeof' required_files = ['ceph-nvmeof.conf'] - default_image = DEFAULT_NVMEOF_IMAGE + default_image = DefaultImages.NVMEOF.image_ref @classmethod def for_daemon_type(cls, daemon_type: str) -> bool: @@ -38,7 +38,7 @@ class CephNvmeof(ContainerDaemonForm): fsid: str, daemon_id: Union[int, str], config_json: Dict, - image: str = DEFAULT_NVMEOF_IMAGE, + image: str = DefaultImages.NVMEOF.image_ref, ) -> None: self.ctx = ctx self.fsid = fsid @@ -100,6 +100,7 @@ class CephNvmeof(ContainerDaemonForm): 'client_cert', 'client_key', 'root_ca_cert', + 'encryption_key', ]: if fn in files: mounts[ diff --git a/src/cephadm/cephadmlib/daemons/oauth2_proxy.py b/src/cephadm/cephadmlib/daemons/oauth2_proxy.py index 14202111c14..c4f4ec5562f 100644 --- a/src/cephadm/cephadmlib/daemons/oauth2_proxy.py +++ b/src/cephadm/cephadmlib/daemons/oauth2_proxy.py @@ -11,7 +11,7 @@ from ..context_getters import fetch_configs from ..daemon_form import register as register_daemon_form from ..daemon_identity import DaemonIdentity from ..deployment_utils import to_deployment_container -from ceph.cephadm.images import DEFAULT_OAUTH2_PROXY_IMAGE +from ceph.cephadm.images import DefaultImages from ..constants import UID_NOBODY, GID_NOGROUP from ..data_utils import dict_get, is_fsid from ..file_utils import populate_files, makedirs, recursive_chown @@ -25,7 +25,7 @@ logger = logging.getLogger() class OAuth2Proxy(ContainerDaemonForm): """Define the configs for the jaeger tracing containers""" - default_image = DEFAULT_OAUTH2_PROXY_IMAGE + default_image = DefaultImages.OAUTH2_PROXY.image_ref daemon_type = 'oauth2-proxy' required_files = [ 'oauth2-proxy.conf', @@ -43,7 +43,7 @@ class OAuth2Proxy(ContainerDaemonForm): fsid: str, daemon_id: str, config_json: Dict, - image: str = DEFAULT_OAUTH2_PROXY_IMAGE, + image: str = DefaultImages.OAUTH2_PROXY.image_ref, ): self.ctx = ctx self.fsid = fsid diff --git a/src/cephadm/cephadmlib/daemons/smb.py b/src/cephadm/cephadmlib/daemons/smb.py index 33d43cbe6ce..0efde198812 100644 --- a/src/cephadm/cephadmlib/daemons/smb.py +++ b/src/cephadm/cephadmlib/daemons/smb.py @@ -14,7 +14,7 @@ from .. import data_utils from .. import deployment_utils from .. import file_utils from ..call_wrappers import call, CallVerbosity -from ceph.cephadm.images import DEFAULT_SAMBA_IMAGE +from ceph.cephadm.images import DefaultImages from ..container_daemon_form import ContainerDaemonForm, daemon_to_container from ..container_engines import Podman from ..container_types import ( @@ -368,7 +368,7 @@ class SMB(ContainerDaemonForm): daemon_type = 'smb' daemon_base = '/usr/sbin/smbd' - default_image = DEFAULT_SAMBA_IMAGE + default_image = DefaultImages.SAMBA.image_ref @classmethod def for_daemon_type(cls, daemon_type: str) -> bool: diff --git a/src/cephadm/cephadmlib/daemons/snmp.py b/src/cephadm/cephadmlib/daemons/snmp.py index ab84a302f2c..0557a2ef972 100644 --- a/src/cephadm/cephadmlib/daemons/snmp.py +++ b/src/cephadm/cephadmlib/daemons/snmp.py @@ -5,7 +5,7 @@ from typing import Any, Dict, List, Optional, Tuple, Union from urllib.error import HTTPError, URLError from urllib.request import urlopen -from ceph.cephadm.images import DEFAULT_SNMP_GATEWAY_IMAGE +from ceph.cephadm.images import DefaultImages from ..container_daemon_form import ContainerDaemonForm, daemon_to_container from ..container_types import CephContainer from ..context import CephadmContext @@ -24,7 +24,7 @@ class SNMPGateway(ContainerDaemonForm): daemon_type = 'snmp-gateway' SUPPORTED_VERSIONS = ['V2c', 'V3'] - default_image = DEFAULT_SNMP_GATEWAY_IMAGE + default_image = DefaultImages.SNMP_GATEWAY.image_ref DEFAULT_PORT = 9464 env_filename = 'snmp-gateway.conf' diff --git a/src/cephadm/cephadmlib/daemons/tracing.py b/src/cephadm/cephadmlib/daemons/tracing.py index 4cf74339455..44548a61d14 100644 --- a/src/cephadm/cephadmlib/daemons/tracing.py +++ b/src/cephadm/cephadmlib/daemons/tracing.py @@ -2,12 +2,7 @@ import logging from typing import Any, Dict, List, Tuple -from ceph.cephadm.images import ( - DEFAULT_ELASTICSEARCH_IMAGE, - DEFAULT_JAEGER_AGENT_IMAGE, - DEFAULT_JAEGER_COLLECTOR_IMAGE, - DEFAULT_JAEGER_QUERY_IMAGE, -) +from ceph.cephadm.images import DefaultImages from ..container_daemon_form import ContainerDaemonForm, daemon_to_container from ..container_types import CephContainer from ..context import CephadmContext @@ -27,17 +22,17 @@ class Tracing(ContainerDaemonForm): components: Dict[str, Dict[str, Any]] = { 'elasticsearch': { - 'image': DEFAULT_ELASTICSEARCH_IMAGE, + 'image': DefaultImages.ELASTICSEARCH.image_ref, 'envs': ['discovery.type=single-node'], }, 'jaeger-agent': { - 'image': DEFAULT_JAEGER_AGENT_IMAGE, + 'image': DefaultImages.JAEGER_AGENT.image_ref, }, 'jaeger-collector': { - 'image': DEFAULT_JAEGER_COLLECTOR_IMAGE, + 'image': DefaultImages.JAEGER_COLLECTOR.image_ref, }, 'jaeger-query': { - 'image': DEFAULT_JAEGER_QUERY_IMAGE, + 'image': DefaultImages.JAEGER_QUERY.image_ref, }, } # type: ignore diff --git a/src/cephadm/cephadmlib/file_utils.py b/src/cephadm/cephadmlib/file_utils.py index 27e70e31756..4dd88cc3671 100644 --- a/src/cephadm/cephadmlib/file_utils.py +++ b/src/cephadm/cephadmlib/file_utils.py @@ -5,6 +5,7 @@ import datetime import logging import os import tempfile +import json from contextlib import contextmanager from pathlib import Path @@ -157,3 +158,26 @@ def unlink_file( except Exception: if not ignore_errors: raise + + +def update_meta_file(file_path: str, update_key_val: dict) -> None: + """Update key in the file with provided value""" + try: + with open(file_path, 'r') as fh: + data = json.load(fh) + file_stat = os.stat(file_path) + except FileNotFoundError: + raise + except Exception: + logger.exception(f'Failed to update {file_path}') + raise + data.update( + {key: value for key, value in update_key_val.items() if key in data} + ) + + with write_new( + file_path, + owner=(file_stat.st_uid, file_stat.st_gid), + perms=(file_stat.st_mode & 0o777), + ) as fh: + fh.write(json.dumps(data, indent=4) + '\n') diff --git a/src/cephadm/cephadmlib/net_utils.py b/src/cephadm/cephadmlib/net_utils.py index 9a7f138b1c6..bfa61d933ef 100644 --- a/src/cephadm/cephadmlib/net_utils.py +++ b/src/cephadm/cephadmlib/net_utils.py @@ -24,12 +24,22 @@ class EndPoint: def __init__(self, ip: str, port: int) -> None: self.ip = ip self.port = port + self.is_ipv4 = True + try: + if ip and ipaddress.ip_network(ip).version == 6: + self.is_ipv4 = False + except Exception: + logger.exception('Failed to check ip address version') def __str__(self) -> str: - return f'{self.ip}:{self.port}' + if self.is_ipv4: + return f'{self.ip}:{self.port}' + return f'[{self.ip}]:{self.port}' def __repr__(self) -> str: - return f'{self.ip}:{self.port}' + if self.is_ipv4: + return f'{self.ip}:{self.port}' + return f'[{self.ip}]:{self.port}' def attempt_bind(ctx, s, address, port): diff --git a/src/cephadm/tests/test_deploy.py b/src/cephadm/tests/test_deploy.py index c5094db335f..1736639ed55 100644 --- a/src/cephadm/tests/test_deploy.py +++ b/src/cephadm/tests/test_deploy.py @@ -316,7 +316,7 @@ def test_deploy_a_monitoring_container(cephadm_fs, funkypatch): runfile_lines = f.read().splitlines() assert 'podman' in runfile_lines[-1] assert runfile_lines[-1].endswith( - 'quay.io/titans/prometheus:latest --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://10.10.10.10:9095 --web.listen-address=1.2.3.4:9095' + 'quay.io/titans/prometheus:latest --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/prometheus --web.listen-address=1.2.3.4:9095 --storage.tsdb.retention.time=15d --storage.tsdb.retention.size=0 --web.external-url=http://10.10.10.10:9095' ) assert '--user 8765' in runfile_lines[-1] assert f'-v /var/lib/ceph/{fsid}/prometheus.fire/etc/prometheus:/etc/prometheus:Z' in runfile_lines[-1] diff --git a/src/cephfs.pc.in b/src/cephfs.pc.in new file mode 100644 index 00000000000..3c5761495bb --- /dev/null +++ b/src/cephfs.pc.in @@ -0,0 +1,10 @@ +prefix=@prefix@ +exec_prefix=${prefix} +libdir=@libdir@ +includedir=@includedir@ + +Name: libcephfs +Description: Ceph distributed file system client library +Version: @libcephfs_version@ +Cflags: -I${includedir}/cephfs -D_FILE_OFFSET_BITS=64 +Libs: -L${libdir} -lcephfs diff --git a/src/client/Client.cc b/src/client/Client.cc index c404057b929..00b85a8e746 100644 --- a/src/client/Client.cc +++ b/src/client/Client.cc @@ -8907,7 +8907,6 @@ int Client::chownat(int dirfd, const char *relpath, uid_t new_uid, gid_t new_gid tout(cct) << new_gid << std::endl; tout(cct) << flags << std::endl; - filepath path(relpath); InodeRef in; InodeRef dirinode; @@ -8917,10 +8916,24 @@ int Client::chownat(int dirfd, const char *relpath, uid_t new_uid, gid_t new_gid return r; } - r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), 0, dirinode); - if (r < 0) { - return r; + if (!strcmp(relpath, "")) { +#if defined(__linux__) && defined(AT_EMPTY_PATH) + if (flags & AT_EMPTY_PATH) { + in = dirinode; + goto out; + } +#endif + return -CEPHFS_ENOENT; + } else { + filepath path(relpath); + r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), 0, dirinode); + if (r < 0) { + return r; + } } + +out: + struct stat attr; attr.st_uid = new_uid; attr.st_gid = new_gid; @@ -11740,8 +11753,12 @@ int64_t Client::_write(Fh *f, int64_t offset, uint64_t size, const char *buf, cond_iofinish = new C_SaferCond(); filer_iofinish.reset(cond_iofinish); } else { - //Register a wrapper callback for the C_Write_Finisher which takes 'client_lock' - filer_iofinish.reset(new C_Lock_Client_Finisher(this, iofinish.get())); + //Register a wrapper callback C_Lock_Client_Finisher for the C_Write_Finisher which takes 'client_lock'. + //Use C_OnFinisher for callbacks. The op_cancel_writes has to be called without 'client_lock' held because + //the callback registered here needs to take it. This would cause incorrect lock order i.e., objecter->rwlock + //taken by objecter's op_cancel and then 'client_lock' taken by callback. To fix the lock order, queue + //the callback using the finisher + filer_iofinish.reset(new C_OnFinisher(new C_Lock_Client_Finisher(this, iofinish.get()), &objecter_finisher)); } get_cap_ref(in, CEPH_CAP_FILE_BUFFER); @@ -12230,6 +12247,7 @@ int Client::statxat(int dirfd, const char *relpath, unsigned mask = statx_to_mask(flags, want); + InodeRef in; InodeRef dirinode; std::scoped_lock lock(client_lock); int r = get_fd_inode(dirfd, &dirinode); @@ -12237,12 +12255,24 @@ int Client::statxat(int dirfd, const char *relpath, return r; } - InodeRef in; - filepath path(relpath); - r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), mask, dirinode); - if (r < 0) { - return r; + if (!strcmp(relpath, "")) { +#if defined(__linux__) && defined(AT_EMPTY_PATH) + if (flags & AT_EMPTY_PATH) { + in = dirinode; + goto out; + } +#endif + return -CEPHFS_ENOENT; + } else { + filepath path(relpath); + r = path_walk(path, &in, perms, !(flags & AT_SYMLINK_NOFOLLOW), mask, dirinode); + if (r < 0) { + return r; + } } + +out: + r = _getattr(in, mask, perms); if (r < 0) { ldout(cct, 3) << __func__ << " exit on error!" << dendl; diff --git a/src/client/Client.h b/src/client/Client.h index f8c39e2fdd6..d5108c12262 100644 --- a/src/client/Client.h +++ b/src/client/Client.h @@ -219,6 +219,7 @@ struct dir_result_t { ordered_count = 0; cache_index = 0; buffer.clear(); + fd = -1; } InodeRef inode; diff --git a/src/cls/rbd/cls_rbd_client.cc b/src/cls/rbd/cls_rbd_client.cc index 559ac221f89..fca2d0ea18b 100644 --- a/src/cls/rbd/cls_rbd_client.cc +++ b/src/cls/rbd/cls_rbd_client.cc @@ -2538,7 +2538,7 @@ int mirror_image_map_list( mirror_image_map_list_start(&op, start_after, max_read); bufferlist out_bl; - int r = ioctx->operate(RBD_MIRRORING, &op, &out_bl); + int r = ioctx->operate(RBD_MIRROR_LEADER, &op, &out_bl); if (r < 0) { return r; } diff --git a/src/cls/rgw/cls_rgw_types.cc b/src/cls/rgw/cls_rgw_types.cc index d5f6ba4bdee..9fd60aaff3f 100644 --- a/src/cls/rgw/cls_rgw_types.cc +++ b/src/cls/rgw/cls_rgw_types.cc @@ -194,7 +194,9 @@ void rgw_bucket_dir_entry_meta::dump(Formatter *f) const utime_t ut(mtime); encode_json("mtime", ut, f); encode_json("etag", etag, f); - encode_json("storage_class", storage_class, f); + encode_json("storage_class", + rgw_placement_rule::get_canonical_storage_class(storage_class), + f); encode_json("owner", owner, f); encode_json("owner_display_name", owner_display_name, f); encode_json("content_type", content_type, f); diff --git a/src/common/CMakeLists.txt b/src/common/CMakeLists.txt index ea3cce16609..c607839a8d2 100644 --- a/src/common/CMakeLists.txt +++ b/src/common/CMakeLists.txt @@ -13,6 +13,7 @@ if(WIN32) endif() add_subdirectory(io_exerciser) +add_subdirectory(json) add_subdirectory(options) set(common_srcs diff --git a/src/common/Formatter.cc b/src/common/Formatter.cc index fd3b2be0221..cd12e4f9885 100644 --- a/src/common/Formatter.cc +++ b/src/common/Formatter.cc @@ -23,6 +23,8 @@ #include <algorithm> #include <set> #include <limits> +#include <utility> +#include <boost/container/small_vector.hpp> // ----------------------- namespace ceph { @@ -365,10 +367,21 @@ std::ostream& JSONFormatter::dump_stream(std::string_view name) void JSONFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; - add_value(name, buf, quoted); + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } + + add_value(name, buf.data(), quoted); } int JSONFormatter::get_len() const @@ -550,15 +563,27 @@ std::ostream& XMLFormatter::dump_stream(std::string_view name) void XMLFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - size_t len = vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } + auto e = get_xml_name(name); print_spaces(); if (ns) { - m_ss << "<" << e << " xmlns=\"" << ns << "\">" << xml_stream_escaper(std::string_view(buf, len)) << "</" << e << ">"; + m_ss << "<" << e << " xmlns=\"" << ns << "\">" << xml_stream_escaper(std::string_view(buf.data(), len)) << "</" << e << ">"; } else { - m_ss << "<" << e << ">" << xml_stream_escaper(std::string_view(buf, len)) << "</" << e << ">"; + m_ss << "<" << e << ">" << xml_stream_escaper(std::string_view(buf.data(), len)) << "</" << e << ">"; } if (m_pretty) @@ -927,14 +952,26 @@ void TableFormatter::dump_format_va(std::string_view name, const char *fmt, va_list ap) { finish_pending_string(); - char buf[LARGE_SIZE]; - vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + int len = vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + va_end(ap_copy); + + if (std::cmp_greater_equal(len, buf.size())) { + // output was truncated, allocate a buffer large enough + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap); + } size_t i = m_vec_index(name); if (ns) { - m_ss << ns << "." << buf; - } else - m_ss << buf; + m_ss << ns << "." << buf.data(); + } else { + m_ss << buf.data(); + } m_vec[i].push_back(std::make_pair(get_section_name(name), m_ss.str())); m_ss.clear(); diff --git a/src/common/HTMLFormatter.cc b/src/common/HTMLFormatter.cc index e7e985531d8..1bc8d864cb6 100644 --- a/src/common/HTMLFormatter.cc +++ b/src/common/HTMLFormatter.cc @@ -23,6 +23,8 @@ #include <stdlib.h> #include <string> #include <string.h> // for strdup +#include <boost/container/small_vector.hpp> +#include <utility> // for std::cmp_greater_equal #include "common/escape.h" @@ -138,17 +140,27 @@ std::ostream& HTMLFormatter::dump_stream(std::string_view name) void HTMLFormatter::dump_format_va(std::string_view name, const char *ns, bool quoted, const char *fmt, va_list ap) { - char buf[LARGE_SIZE]; - size_t len = vsnprintf(buf, LARGE_SIZE, fmt, ap); + auto buf = boost::container::small_vector<char, LARGE_SIZE>{ + LARGE_SIZE, boost::container::default_init}; + + va_list ap_copy; + va_copy(ap_copy, ap); + size_t len = vsnprintf(buf.data(), buf.size(), fmt, ap); + va_end(ap_copy); + + if(std::cmp_greater_equal(len, buf.size())){ + buf.resize(len + 1, boost::container::default_init); + vsnprintf(buf.data(), buf.size(), fmt, ap_copy); + } std::string e(name); print_spaces(); if (ns) { m_ss << "<li xmlns=\"" << ns << "\">" << e << ": " - << xml_stream_escaper(std::string_view(buf, len)) << "</li>"; + << xml_stream_escaper(std::string_view(buf.data(), len)) << "</li>"; } else { m_ss << "<li>" << e << ": " - << xml_stream_escaper(std::string_view(buf, len)) << "</li>"; + << xml_stream_escaper(std::string_view(buf.data(), len)) << "</li>"; } if (m_pretty) diff --git a/src/common/async/bind_handler.h b/src/common/async/bind_handler.h index 69128501a07..4cc9a2a113d 100644 --- a/src/common/async/bind_handler.h +++ b/src/common/async/bind_handler.h @@ -16,8 +16,7 @@ #define CEPH_ASYNC_BIND_HANDLER_H #include <tuple> -#include <boost/asio/associated_allocator.hpp> -#include <boost/asio/associated_executor.hpp> +#include <boost/asio/associator.hpp> namespace ceph::async { @@ -52,25 +51,25 @@ struct CompletionHandler { void operator()() && { std::apply(std::move(handler), std::move(args)); } - - using allocator_type = boost::asio::associated_allocator_t<Handler>; - allocator_type get_allocator() const noexcept { - return boost::asio::get_associated_allocator(handler); - } }; } // namespace ceph::async namespace boost::asio { -// specialize boost::asio::associated_executor<> for CompletionHandler -template <typename Handler, typename Tuple, typename Executor> -struct associated_executor<ceph::async::CompletionHandler<Handler, Tuple>, Executor> { - using type = boost::asio::associated_executor_t<Handler, Executor>; - - static type get(const ceph::async::CompletionHandler<Handler, Tuple>& handler, - const Executor& ex = Executor()) noexcept { - return boost::asio::get_associated_executor(handler.handler, ex); +// forward the handler's associated executor, allocator, cancellation slot, etc +template <template <typename, typename> class Associator, + typename Handler, typename Tuple, typename DefaultCandidate> +struct associator<Associator, + ceph::async::CompletionHandler<Handler, Tuple>, DefaultCandidate> + : Associator<Handler, DefaultCandidate> +{ + static auto get(const ceph::async::CompletionHandler<Handler, Tuple>& h) noexcept { + return Associator<Handler, DefaultCandidate>::get(h.handler); + } + static auto get(const ceph::async::CompletionHandler<Handler, Tuple>& h, + const DefaultCandidate& c) noexcept { + return Associator<Handler, DefaultCandidate>::get(h.handler, c); } }; diff --git a/src/common/async/bind_like.h b/src/common/async/bind_like.h deleted file mode 100644 index c360eac0aad..00000000000 --- a/src/common/async/bind_like.h +++ /dev/null @@ -1,39 +0,0 @@ -// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- -// vim: ts=8 sw=2 smarttab -/* - * Ceph - scalable distributed file system - * - * Copyright (C) 2020 Red Hat <contact@redhat.com> - * Author: Adam C. Emerson - * - * This is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License version 2.1, as published by the Free Software - * Foundation. See file COPYING. - * - */ - -#include <boost/asio/associated_allocator.hpp> -#include <boost/asio/associated_executor.hpp> -#include <boost/asio/bind_allocator.hpp> -#include <boost/asio/bind_executor.hpp> - -namespace ceph::async { -template<typename Executor, typename Allocator, typename Completion> -auto bind_ea(const Executor& executor, const Allocator& allocator, - Completion&& completion) { - return bind_allocator(allocator, - boost::asio::bind_executor( - executor, - std::forward<Completion>(completion))); -} - - -// Bind `Completion` to the executor and allocator of `Proto` -template<typename Proto, typename Completion> -auto bind_like(const Proto& proto, Completion&& completion) { - return bind_ea(boost::asio::get_associated_executor(proto), - boost::asio::get_associated_allocator(proto), - std::forward<Completion>(completion)); -} -} diff --git a/src/common/async/completion.h b/src/common/async/completion.h index d8065934e01..6cdfaaa63b7 100644 --- a/src/common/async/completion.h +++ b/src/common/async/completion.h @@ -21,6 +21,7 @@ #include <boost/asio/defer.hpp> #include <boost/asio/dispatch.hpp> #include <boost/asio/executor_work_guard.hpp> +#include <boost/asio/recycling_allocator.hpp> #include <boost/asio/post.hpp> #include "bind_handler.h" @@ -173,7 +174,8 @@ class CompletionImpl final : public Completion<void(Args...), T> { Handler handler; // use Handler's associated allocator - using Alloc2 = boost::asio::associated_allocator_t<Handler>; + using DefaultAlloc = boost::asio::recycling_allocator<void>; + using Alloc2 = boost::asio::associated_allocator_t<Handler, DefaultAlloc>; using Traits2 = std::allocator_traits<Alloc2>; using RebindAlloc2 = typename Traits2::template rebind_alloc<CompletionImpl>; using RebindTraits2 = std::allocator_traits<RebindAlloc2>; @@ -196,16 +198,16 @@ class CompletionImpl final : public Completion<void(Args...), T> { void destroy_defer(std::tuple<Args...>&& args) override { auto w = std::move(work); auto ex2 = w.second.get_executor(); - RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler); + RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler, DefaultAlloc{}); auto f = bind_and_forward(ex2, std::move(handler), std::move(args)); RebindTraits2::destroy(alloc2, this); RebindTraits2::deallocate(alloc2, this, 1); - boost::asio::defer(boost::asio::bind_executor(ex2, std::move(f))); + boost::asio::defer(std::move(f)); } void destroy_dispatch(std::tuple<Args...>&& args) override { auto w = std::move(work); auto ex2 = w.second.get_executor(); - RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler); + RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler, DefaultAlloc{}); auto f = bind_and_forward(ex2, std::move(handler), std::move(args)); RebindTraits2::destroy(alloc2, this); RebindTraits2::deallocate(alloc2, this, 1); @@ -214,14 +216,14 @@ class CompletionImpl final : public Completion<void(Args...), T> { void destroy_post(std::tuple<Args...>&& args) override { auto w = std::move(work); auto ex2 = w.second.get_executor(); - RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler); + RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler, DefaultAlloc{}); auto f = bind_and_forward(ex2, std::move(handler), std::move(args)); RebindTraits2::destroy(alloc2, this); RebindTraits2::deallocate(alloc2, this, 1); boost::asio::post(std::move(f)); } void destroy() override { - RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler); + RebindAlloc2 alloc2 = boost::asio::get_associated_allocator(handler, DefaultAlloc{}); RebindTraits2::destroy(alloc2, this); RebindTraits2::deallocate(alloc2, this, 1); } @@ -238,7 +240,7 @@ class CompletionImpl final : public Completion<void(Args...), T> { public: template <typename ...TArgs> static auto create(const Executor1& ex, Handler&& handler, TArgs&& ...args) { - auto alloc2 = boost::asio::get_associated_allocator(handler); + auto alloc2 = boost::asio::get_associated_allocator(handler, DefaultAlloc{}); using Ptr = std::unique_ptr<CompletionImpl>; return Ptr{new (alloc2) CompletionImpl(ex, std::move(handler), std::forward<TArgs>(args)...)}; diff --git a/src/common/async/detail/shared_mutex.h b/src/common/async/detail/shared_mutex.h index 6eae25b430d..db7890049ab 100644 --- a/src/common/async/detail/shared_mutex.h +++ b/src/common/async/detail/shared_mutex.h @@ -19,6 +19,7 @@ #include <optional> #include <shared_mutex> // for std::shared_lock +#include <boost/asio/append.hpp> #include <boost/smart_ptr/intrusive_ref_counter.hpp> #include <boost/intrusive_ptr.hpp> #include <boost/intrusive/list.hpp> @@ -134,10 +135,8 @@ auto SharedMutexImpl::async_lock(Mutex& mtx, CompletionToken&& token) state = Exclusive; // post a successful completion - auto ex2 = boost::asio::get_associated_executor(handler, ex1); - auto h = boost::asio::bind_executor(ex2, std::move(handler)); - boost::asio::post(bind_handler(std::move(h), ec, - std::unique_lock{mtx, std::adopt_lock})); + boost::asio::post(ex1, boost::asio::append(std::move(handler), + ec, std::unique_lock{mtx, std::adopt_lock})); } else { // create a request and add it to the exclusive list using LockCompletion = typename Request::LockCompletion; @@ -224,10 +223,8 @@ auto SharedMutexImpl::async_lock_shared(Mutex& mtx, CompletionToken&& token) if (exclusive_queue.empty() && state < MaxShared) { state++; - auto ex2 = boost::asio::get_associated_executor(handler, ex1); - auto h = boost::asio::bind_executor(ex2, std::move(handler)); - boost::asio::post(bind_handler(std::move(h), ec, - std::shared_lock{mtx, std::adopt_lock})); + boost::asio::post(ex1, boost::asio::append(std::move(handler), + ec, std::shared_lock{mtx, std::adopt_lock})); } else { using LockCompletion = typename Request::LockCompletion; auto request = LockCompletion::create(ex1, std::move(handler), mtx); diff --git a/src/common/async/forward_handler.h b/src/common/async/forward_handler.h index 1491ef6085d..e204ca9862c 100644 --- a/src/common/async/forward_handler.h +++ b/src/common/async/forward_handler.h @@ -15,8 +15,7 @@ #ifndef CEPH_ASYNC_FORWARD_HANDLER_H #define CEPH_ASYNC_FORWARD_HANDLER_H -#include <boost/asio/associated_allocator.hpp> -#include <boost/asio/associated_executor.hpp> +#include <boost/asio/associator.hpp> namespace ceph::async { @@ -47,25 +46,25 @@ struct ForwardingHandler { void operator()(Args&& ...args) { std::move(handler)(std::forward<Args>(args)...); } - - using allocator_type = boost::asio::associated_allocator_t<Handler>; - allocator_type get_allocator() const noexcept { - return boost::asio::get_associated_allocator(handler); - } }; } // namespace ceph::async namespace boost::asio { -// specialize boost::asio::associated_executor<> for ForwardingHandler -template <typename Handler, typename Executor> -struct associated_executor<ceph::async::ForwardingHandler<Handler>, Executor> { - using type = boost::asio::associated_executor_t<Handler, Executor>; - - static type get(const ceph::async::ForwardingHandler<Handler>& handler, - const Executor& ex = Executor()) noexcept { - return boost::asio::get_associated_executor(handler.handler, ex); +// forward the handler's associated executor, allocator, cancellation slot, etc +template <template <typename, typename> class Associator, + typename Handler, typename DefaultCandidate> +struct associator<Associator, + ceph::async::ForwardingHandler<Handler>, DefaultCandidate> + : Associator<Handler, DefaultCandidate> +{ + static auto get(const ceph::async::ForwardingHandler<Handler>& h) noexcept { + return Associator<Handler, DefaultCandidate>::get(h.handler); + } + static auto get(const ceph::async::ForwardingHandler<Handler>& h, + const DefaultCandidate& c) noexcept { + return Associator<Handler, DefaultCandidate>::get(h.handler, c); } }; diff --git a/src/common/bit_vector.hpp b/src/common/bit_vector.hpp index 961d9a0192e..c5fd491ed29 100644 --- a/src/common/bit_vector.hpp +++ b/src/common/bit_vector.hpp @@ -29,8 +29,8 @@ private: static const uint8_t MASK = static_cast<uint8_t>((1 << _bit_count) - 1); // must be power of 2 - BOOST_STATIC_ASSERT((_bit_count != 0) && !(_bit_count & (_bit_count - 1))); - BOOST_STATIC_ASSERT(_bit_count <= BITS_PER_BYTE); + static_assert((_bit_count != 0) && !(_bit_count & (_bit_count - 1))); + static_assert(_bit_count <= BITS_PER_BYTE); template <typename DataIterator> class ReferenceImpl { diff --git a/src/common/ceph_time.h b/src/common/ceph_time.h index 01feff4c063..0b05be5372e 100644 --- a/src/common/ceph_time.h +++ b/src/common/ceph_time.h @@ -342,6 +342,23 @@ public: } }; +// Please note time_guard is not thread safety -- multiple threads +// updating same diff_accumulator can corrupt it. +template <class ClockT = mono_clock> +class time_guard { + const typename ClockT::time_point start; + timespan& diff_accumulator; + +public: + time_guard(timespan& diff_accumulator) + : start(ClockT::now()), + diff_accumulator(diff_accumulator) { + } + ~time_guard() { + diff_accumulator += ClockT::now() - start; + } +}; + namespace time_detail { // So that our subtractions produce negative spans rather than // arithmetic underflow. diff --git a/src/common/config_cacher.h b/src/common/config_cacher.h index 91b8152dde1..f23195955a1 100644 --- a/src/common/config_cacher.h +++ b/src/common/config_cacher.h @@ -18,21 +18,30 @@ #include "common/config_obs.h" #include "common/config.h" +/** + * A simple class to cache a single configuration value. + * Points to note: + * - as get_tracked_conf_keys() must return a pointer to a null-terminated + * array of C-strings, 'keys' - an array - is used to hold the sole key + * that this observer is interested in. + * - the const cast should be removed once we change the + * get_tracked_conf_keys() to return const char* const * (or something + * similar). + */ template <typename ValueT> class md_config_cacher_t : public md_config_obs_t { ConfigProxy& conf; - const char* const option_name; + const char* keys[2]; std::atomic<ValueT> value_cache; const char** get_tracked_conf_keys() const override { - const static char* keys[] = { option_name, nullptr }; - return keys; + return const_cast<const char**>(keys); } void handle_conf_change(const ConfigProxy& conf, const std::set<std::string>& changed) override { - if (changed.count(option_name)) { - value_cache.store(conf.get_val<ValueT>(option_name)); + if (changed.contains(keys[0])) { + value_cache.store(conf.get_val<ValueT>(keys[0])); } } @@ -40,10 +49,10 @@ public: md_config_cacher_t(ConfigProxy& conf, const char* const option_name) : conf(conf), - option_name(option_name) { + keys{option_name, nullptr} { conf.add_observer(this); std::atomic_init(&value_cache, - conf.get_val<ValueT>(option_name)); + conf.get_val<ValueT>(keys[0])); } ~md_config_cacher_t() { diff --git a/src/common/intrusive_lru.h b/src/common/intrusive_lru.h index 3ed3625d8a0..f9132773e4d 100644 --- a/src/common/intrusive_lru.h +++ b/src/common/intrusive_lru.h @@ -94,6 +94,8 @@ public: friend void intrusive_ptr_add_ref<>(intrusive_lru_base<Config> *); friend void intrusive_ptr_release<>(intrusive_lru_base<Config> *); + unsigned get_use_count() const { return use_count; } + virtual ~intrusive_lru_base() {} }; @@ -176,6 +178,25 @@ class intrusive_lru { evict(lru_target_size); } + /// clear [from, to) invoking f upon and invalidating any live references + template <typename F> + void clear_range( + typename lru_set_t::iterator from, typename lru_set_t::iterator to, + F &&f) { + while (from != to) { + if (!(*from).lru) { + unreferenced_list.erase(lru_list_t::s_iterator_to(*from)); + from = lru_set.erase_and_dispose(from, [](auto *p) { delete p; } ); + } else { + std::invoke(f, static_cast<T&>(*from)); + from->lru = nullptr; + assert(from->is_invalidated()); + from = lru_set.erase_and_dispose( + from, [](auto *p) { assert(p->use_count > 0); }); + } + } + } + public: /** * Returns the TRef corresponding to k if it exists or @@ -198,38 +219,28 @@ public: } } - /* - * Clears unreferenced elements from the lru set [from, to] + /** + * clear_range + * + * Clears elements from the lru set in [from, to] invoking F upon and + * invalidating any with outstanding references */ - void clear_range( - const K& from, - const K& to) { - auto from_iter = lru_set.lower_bound(from); - auto to_iter = lru_set.upper_bound(to); - for (auto i = from_iter; i != to_iter; ) { - if (!(*i).lru) { - unreferenced_list.erase(lru_list_t::s_iterator_to(*i)); - i = lru_set.erase_and_dispose(i, [](auto *p) - { delete p; } ); - } else { - i++; - } - } + template <typename F> + void clear_range(const K& from, const K& to, F &&f) { + auto from_iter = lru_set.lower_bound(from); + auto to_iter = lru_set.upper_bound(to); + clear_range(from_iter, to_iter, std::forward<F>(f)); } - /// drop all elements from lru, invoke f on any with outstanding references + /** + * clear + * + * Clears all elements from the lru set invoking F upon and + * invalidating any with outstanding references + */ template <typename F> void clear(F &&f) { - evict(0); - assert(unreferenced_list.empty()); - for (auto &i: lru_set) { - std::invoke(f, static_cast<T&>(i)); - i.lru = nullptr; - assert(i.is_invalidated()); - } - lru_set.clear_and_dispose([](auto *i){ - assert(i->use_count > 0); /* don't delete, still has a ref count */ - }); + clear_range(lru_set.begin(), lru_set.end(), std::forward<F>(f)); } template <class F> diff --git a/src/common/io_exerciser/CMakeLists.txt b/src/common/io_exerciser/CMakeLists.txt index 07091df86e1..ab2e64fc222 100644 --- a/src/common/io_exerciser/CMakeLists.txt +++ b/src/common/io_exerciser/CMakeLists.txt @@ -5,9 +5,11 @@ add_library(object_io_exerciser STATIC Model.cc ObjectModel.cc RadosIo.cc + EcIoSequence.cc ) target_link_libraries(object_io_exerciser - librados + librados global + json_structures )
\ No newline at end of file diff --git a/src/common/io_exerciser/DataGenerator.cc b/src/common/io_exerciser/DataGenerator.cc index 9aa77eeb6e9..701c32fa9ec 100644 --- a/src/common/io_exerciser/DataGenerator.cc +++ b/src/common/io_exerciser/DataGenerator.cc @@ -2,32 +2,28 @@ // vim: ts=8 sw=2 smarttab #include "DataGenerator.h" -#include "ObjectModel.h" +#include <chrono> +#include <iostream> +#include <stdexcept> +#include "ObjectModel.h" #include "common/debug.h" #include "common/dout.h" - #include "fmt/format.h" #include "fmt/ranges.h" -#include <chrono> -#include <iostream> -#include <stdexcept> - #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context using DataGenerator = ceph::io_exerciser::data_generation::DataGenerator; -using SeededRandomGenerator = ceph::io_exerciser::data_generation - ::SeededRandomGenerator; -using HeaderedSeededRandomGenerator = ceph::io_exerciser::data_generation - ::HeaderedSeededRandomGenerator; +using SeededRandomGenerator = + ceph::io_exerciser::data_generation ::SeededRandomGenerator; +using HeaderedSeededRandomGenerator = + ceph::io_exerciser::data_generation ::HeaderedSeededRandomGenerator; std::unique_ptr<DataGenerator> DataGenerator::create_generator( - GenerationType generationType, const ObjectModel& model) -{ - switch(generationType) - { + GenerationType generationType, const ObjectModel& model) { + switch (generationType) { case GenerationType::SeededRandom: return std::make_unique<SeededRandomGenerator>(model); case GenerationType::HeaderedSeededRandom: @@ -39,28 +35,25 @@ std::unique_ptr<DataGenerator> DataGenerator::create_generator( return nullptr; } -bufferlist DataGenerator::generate_wrong_data(uint64_t offset, uint64_t length) -{ +bufferlist DataGenerator::generate_wrong_data(uint64_t offset, + uint64_t length) { bufferlist retlist; uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; - for (uint64_t block_offset = offset; - block_offset < offset + length; - block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { std::memset(buffer, 0, block_size); retlist.append(ceph::bufferptr(buffer, block_size)); } return retlist; } -bool DataGenerator::validate(bufferlist& bufferlist, uint64_t offset, uint64_t length) -{ +bool DataGenerator::validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length) { return bufferlist.contents_equal(generate_data(offset, length)); } -ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) -{ +ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) { uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; @@ -70,29 +63,26 @@ ceph::bufferptr SeededRandomGenerator::generate_block(uint64_t block_offset) constexpr size_t generation_length = sizeof(uint64_t); - for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) - { + for (uint64_t i = 0; i < block_size; + i += (2 * generation_length), rand1++, rand2--) { std::memcpy(buffer + i, &rand1, generation_length); std::memcpy(buffer + i + generation_length, &rand2, generation_length); } size_t remainingBytes = block_size % (generation_length * 2); - if (remainingBytes > generation_length) - { + if (remainingBytes > generation_length) { size_t remainingBytes2 = remainingBytes - generation_length; std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); - } - else if (remainingBytes > 0) - { + } else if (remainingBytes > 0) { std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); } return ceph::bufferptr(buffer, block_size); } -ceph::bufferptr SeededRandomGenerator::generate_wrong_block(uint64_t block_offset) -{ +ceph::bufferptr SeededRandomGenerator::generate_wrong_block( + uint64_t block_offset) { uint64_t block_size = m_model.get_block_size(); char buffer[block_size]; @@ -102,141 +92,134 @@ ceph::bufferptr SeededRandomGenerator::generate_wrong_block(uint64_t block_offse constexpr size_t generation_length = sizeof(uint64_t); - for (uint64_t i = 0; i < block_size; i+=(2*generation_length), rand1++, rand2--) - { + for (uint64_t i = 0; i < block_size; + i += (2 * generation_length), rand1++, rand2--) { std::memcpy(buffer + i, &rand1, generation_length); std::memcpy(buffer + i + generation_length, &rand2, generation_length); } size_t remainingBytes = block_size % (generation_length * 2); - if (remainingBytes > generation_length) - { + if (remainingBytes > generation_length) { size_t remainingBytes2 = remainingBytes - generation_length; std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); std::memcpy(buffer + block_size - remainingBytes2, &rand2, remainingBytes2); - } - else if (remainingBytes > 0) - { + } else if (remainingBytes > 0) { std::memcpy(buffer + block_size - remainingBytes, &rand1, remainingBytes); } return ceph::bufferptr(buffer, block_size); } -bufferlist SeededRandomGenerator::generate_data(uint64_t offset, uint64_t length) -{ +bufferlist SeededRandomGenerator::generate_data(uint64_t offset, + uint64_t length) { bufferlist retlist; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { retlist.append(generate_block(block_offset)); } return retlist; } -bufferlist SeededRandomGenerator::generate_wrong_data(uint64_t offset, uint64_t length) -{ +bufferlist SeededRandomGenerator::generate_wrong_data(uint64_t offset, + uint64_t length) { bufferlist retlist; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { retlist.append(generate_wrong_block(block_offset)); } return retlist; } -HeaderedSeededRandomGenerator - ::HeaderedSeededRandomGenerator(const ObjectModel& model, - std::optional<uint64_t> unique_run_id) : - SeededRandomGenerator(model), - unique_run_id(unique_run_id.value_or(generate_unique_run_id())) -{ - -} +HeaderedSeededRandomGenerator ::HeaderedSeededRandomGenerator( + const ObjectModel& model, std::optional<uint64_t> unique_run_id) + : SeededRandomGenerator(model), + unique_run_id(unique_run_id.value_or(generate_unique_run_id())) {} -uint64_t HeaderedSeededRandomGenerator::generate_unique_run_id() -{ +uint64_t HeaderedSeededRandomGenerator::generate_unique_run_id() { std::mt19937_64 random_generator = - std::mt19937_64(duration_cast<std::chrono::milliseconds>( - std::chrono::system_clock::now().time_since_epoch()).count()); + std::mt19937_64(duration_cast<std::chrono::milliseconds>( + std::chrono::system_clock::now().time_since_epoch()) + .count()); - return random_generator(); + return random_generator(); } -ceph::bufferptr HeaderedSeededRandomGenerator::generate_block(uint64_t block_offset) -{ +ceph::bufferptr HeaderedSeededRandomGenerator::generate_block( + uint64_t block_offset) { SeedBytes seed = m_model.get_seed(block_offset); - TimeBytes current_time = duration_cast<std::chrono::milliseconds>( - std::chrono::system_clock::now().time_since_epoch()).count(); + TimeBytes current_time = + duration_cast<std::chrono::milliseconds>( + std::chrono::system_clock::now().time_since_epoch()) + .count(); - ceph::bufferptr bufferptr = SeededRandomGenerator::generate_block(block_offset); + ceph::bufferptr bufferptr = + SeededRandomGenerator::generate_block(block_offset); - std::memcpy(bufferptr.c_str() + uniqueIdStart(), &unique_run_id, uniqueIdLength()); + std::memcpy(bufferptr.c_str() + uniqueIdStart(), &unique_run_id, + uniqueIdLength()); std::memcpy(bufferptr.c_str() + seedStart(), &seed, seedLength()); std::memcpy(bufferptr.c_str() + timeStart(), ¤t_time, timeLength()); return bufferptr; } -ceph::bufferptr HeaderedSeededRandomGenerator::generate_wrong_block(uint64_t block_offset) -{ +ceph::bufferptr HeaderedSeededRandomGenerator::generate_wrong_block( + uint64_t block_offset) { return HeaderedSeededRandomGenerator::generate_block(block_offset % 8); } const HeaderedSeededRandomGenerator::UniqueIdBytes - HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist) { UniqueIdBytes read_unique_run_id = 0; - std::memcpy(&read_unique_run_id, - &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()], - uniqueIdLength()); + std::memcpy( + &read_unique_run_id, + &bufferlist[(block_offset * m_model.get_block_size()) + uniqueIdStart()], + uniqueIdLength()); return read_unique_run_id; } const HeaderedSeededRandomGenerator::SeedBytes - HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readSeed(uint64_t block_offset, + const bufferlist& bufferlist) { SeedBytes read_seed = 0; - std::memcpy(&read_seed, - &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()], - seedLength()); + std::memcpy( + &read_seed, + &bufferlist[(block_offset * m_model.get_block_size()) + seedStart()], + seedLength()); return read_seed; } const HeaderedSeededRandomGenerator::TimeBytes - HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset, - const bufferlist& bufferlist) -{ +HeaderedSeededRandomGenerator::readDateTime(uint64_t block_offset, + const bufferlist& bufferlist) { TimeBytes read_time = 0; - std::memcpy(&read_time, - &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()], - timeLength()); + std::memcpy( + &read_time, + &bufferlist[(block_offset * m_model.get_block_size()) + timeStart()], + timeLength()); return read_time; } bool HeaderedSeededRandomGenerator::validate(bufferlist& bufferlist, - uint64_t offset, uint64_t length) -{ + uint64_t offset, uint64_t length) { std::vector<uint64_t> invalid_block_offsets; - for (uint64_t block_offset = offset; block_offset < offset + length; block_offset++) - { - bool valid_block - = validate_block(block_offset, - (bufferlist.c_str() + ((block_offset - offset) * - m_model.get_block_size()))); - if (!valid_block) - { + for (uint64_t block_offset = offset; block_offset < offset + length; + block_offset++) { + bool valid_block = validate_block( + block_offset, (bufferlist.c_str() + + ((block_offset - offset) * m_model.get_block_size()))); + if (!valid_block) { invalid_block_offsets.push_back(block_offset); } } - if (!invalid_block_offsets.empty()) - { + if (!invalid_block_offsets.empty()) { printDebugInformationForOffsets(offset, invalid_block_offsets, bufferlist); } @@ -244,59 +227,51 @@ bool HeaderedSeededRandomGenerator::validate(bufferlist& bufferlist, } bool HeaderedSeededRandomGenerator::validate_block(uint64_t block_offset, - const char* buffer_start) -{ + const char* buffer_start) { // We validate the block matches what we generate byte for byte // however we ignore the time section of the header ceph::bufferptr bufferptr = generate_block(block_offset); bool valid = strncmp(bufferptr.c_str(), buffer_start, timeStart()) == 0; - valid = valid ? strncmp(bufferptr.c_str() + timeEnd(), - buffer_start + timeEnd(), - m_model.get_block_size() - timeEnd()) == 0 : valid; + valid = valid + ? strncmp(bufferptr.c_str() + timeEnd(), buffer_start + timeEnd(), + m_model.get_block_size() - timeEnd()) == 0 + : valid; return valid; } const HeaderedSeededRandomGenerator::ErrorType - HeaderedSeededRandomGenerator::getErrorTypeForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist) -{ - try - { - UniqueIdBytes read_unique_run_id = readUniqueRunId(block_offset - read_offset, - bufferlist); - if (unique_run_id != read_unique_run_id) - { +HeaderedSeededRandomGenerator::getErrorTypeForBlock( + uint64_t read_offset, uint64_t block_offset, const bufferlist& bufferlist) { + try { + UniqueIdBytes read_unique_run_id = + readUniqueRunId(block_offset - read_offset, bufferlist); + if (unique_run_id != read_unique_run_id) { return ErrorType::RUN_ID_MISMATCH; } SeedBytes read_seed = readSeed(block_offset - read_offset, bufferlist); - if (m_model.get_seed(block_offset) != read_seed) - { + if (m_model.get_seed(block_offset) != read_seed) { return ErrorType::SEED_MISMATCH; } if (std::strncmp(&bufferlist[((block_offset - read_offset) * - m_model.get_block_size()) + bodyStart()], + m_model.get_block_size()) + + bodyStart()], generate_block(block_offset).c_str() + bodyStart(), - m_model.get_block_size() - bodyStart()) != 0) - { + m_model.get_block_size() - bodyStart()) != 0) { return ErrorType::DATA_MISMATCH; } - } - catch(const std::exception& e) - { + } catch (const std::exception& e) { return ErrorType::DATA_NOT_FOUND; } return ErrorType::UNKNOWN; } -void HeaderedSeededRandomGenerator - ::printDebugInformationForBlock(uint64_t read_offset, uint64_t block_offset, - const bufferlist& bufferlist) -{ - ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, bufferlist); +void HeaderedSeededRandomGenerator ::printDebugInformationForBlock( + uint64_t read_offset, uint64_t block_offset, const bufferlist& bufferlist) { + ErrorType blockError = + getErrorTypeForBlock(read_offset, block_offset, bufferlist); TimeBytes read_time = 0; std::time_t ttp; @@ -304,433 +279,361 @@ void HeaderedSeededRandomGenerator char read_bytes[m_model.get_block_size()]; char generated_bytes[m_model.get_block_size()]; - if (blockError == ErrorType::DATA_MISMATCH || blockError == ErrorType::UNKNOWN) - { + if (blockError == ErrorType::DATA_MISMATCH || + blockError == ErrorType::UNKNOWN) { read_time = readDateTime(block_offset - read_offset, bufferlist); - std::chrono::system_clock::time_point time_point{std::chrono::milliseconds{read_time}}; + std::chrono::system_clock::time_point time_point{ + std::chrono::milliseconds{read_time}}; ttp = std::chrono::system_clock::to_time_t(time_point); - std::memcpy(&read_bytes, - &bufferlist[((block_offset - read_offset) * m_model.get_block_size())], - m_model.get_block_size() - bodyStart()); - std::memcpy(&generated_bytes, - generate_block(block_offset).c_str(), + std::memcpy( + &read_bytes, + &bufferlist[((block_offset - read_offset) * m_model.get_block_size())], + m_model.get_block_size() - bodyStart()); + std::memcpy(&generated_bytes, generate_block(block_offset).c_str(), m_model.get_block_size() - bodyStart()); } std::string error_string; - switch(blockError) - { - case ErrorType::RUN_ID_MISMATCH: - { - UniqueIdBytes read_unique_run_id = readUniqueRunId((block_offset - read_offset), - bufferlist); - error_string = fmt::format("Header (Run ID) mismatch detected at block {} " - "(byte offset {}) Header expected run id {} but found id {}. " - "Block data corrupt or not written from this instance of this application.", - block_offset, - block_offset * m_model.get_block_size(), - unique_run_id, - read_unique_run_id); - } - break; - - case ErrorType::SEED_MISMATCH: - { + switch (blockError) { + case ErrorType::RUN_ID_MISMATCH: { + UniqueIdBytes read_unique_run_id = + readUniqueRunId((block_offset - read_offset), bufferlist); + error_string = fmt::format( + "Header (Run ID) mismatch detected at block {} " + "(byte offset {}) Header expected run id {} but found id {}. " + "Block data corrupt or not written from this instance of this " + "application.", + block_offset, block_offset * m_model.get_block_size(), unique_run_id, + read_unique_run_id); + } break; + + case ErrorType::SEED_MISMATCH: { SeedBytes read_seed = readSeed((block_offset - read_offset), bufferlist); - if (m_model.get_seed_offsets(read_seed).size() == 0) - { - error_string = fmt::format("Data (Seed) mismatch detected at block {}" - " (byte offset {}). Header expected seed {} but found seed {}. " - "Read data was not from any other recognised block in the object.", - block_offset, - block_offset * m_model.get_block_size(), - m_model.get_seed(block_offset), - read_seed); - } - else - { + if (m_model.get_seed_offsets(read_seed).size() == 0) { + error_string = fmt::format( + "Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}. " + "Read data was not from any other recognised block in the object.", + block_offset, block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), read_seed); + } else { std::vector<int> seed_offsets = m_model.get_seed_offsets(read_seed); - error_string = fmt::format("Data (Seed) mismatch detected at block {}" - " (byte offset {}). Header expected seed {} but found seed {}." - " Read data was from a different block(s): {}", - block_offset, - block_offset * m_model.get_block_size(), - m_model.get_seed(block_offset), - read_seed, + error_string = fmt::format( + "Data (Seed) mismatch detected at block {}" + " (byte offset {}). Header expected seed {} but found seed {}." + " Read data was from a different block(s): {}", + block_offset, block_offset * m_model.get_block_size(), + m_model.get_seed(block_offset), read_seed, fmt::join(seed_offsets.begin(), seed_offsets.end(), "")); } - } - break; - - case ErrorType::DATA_MISMATCH: - { - error_string = fmt::format("Data (Body) mismatch detected at block {}" - " (byte offset {}). Header data matches, data body does not." - " Data written at {}\nExpected data: \n{:02x}\nRead data:{:02x}", - block_offset, - block_offset * m_model.get_block_size(), + } break; + + case ErrorType::DATA_MISMATCH: { + error_string = fmt::format( + "Data (Body) mismatch detected at block {}" + " (byte offset {}). Header data matches, data body does not." + " Data written at {}\nExpected data: \n{:02x}\nRead data:{:02x}", + block_offset, block_offset * m_model.get_block_size(), std::ctime(&ttp), - fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), + ""), fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); - } - break; + } break; - case ErrorType::DATA_NOT_FOUND: - { + case ErrorType::DATA_NOT_FOUND: { uint64_t bufferlist_length = bufferlist.to_str().size(); - error_string = fmt::format("Data (Body) could not be read at block {}" - " (byte offset {}) offset in bufferlist returned from read: {}" - " ({} bytes). Returned bufferlist length: {}.", - block_offset, - block_offset * m_model.get_block_size(), + error_string = fmt::format( + "Data (Body) could not be read at block {}" + " (byte offset {}) offset in bufferlist returned from read: {}" + " ({} bytes). Returned bufferlist length: {}.", + block_offset, block_offset * m_model.get_block_size(), (block_offset - read_offset), (block_offset - read_offset) * m_model.get_block_size(), bufferlist_length); - } - break; + } break; case ErrorType::UNKNOWN: - [[ fallthrough ]]; - - default: - { - error_string = fmt::format("Data mismatch detected at block {}" - " (byte offset {}).\nExpected data:\n{:02x}\nRead data:\n{:02x}", - block_offset, - block_offset * m_model.get_block_size(), - fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), ""), + [[fallthrough]]; + + default: { + error_string = fmt::format( + "Data mismatch detected at block {}" + " (byte offset {}).\nExpected data:\n{:02x}\nRead data:\n{:02x}", + block_offset, block_offset * m_model.get_block_size(), + fmt::join(generated_bytes, generated_bytes + m_model.get_block_size(), + ""), fmt::join(read_bytes, read_bytes + m_model.get_block_size(), "")); - } - break; + } break; } dout(0) << error_string << dendl; } -void HeaderedSeededRandomGenerator - ::printDebugInformationForRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - ErrorType rangeError, - const bufferlist& bufferlist) -{ - switch(rangeError) - { - case ErrorType::RUN_ID_MISMATCH: - printDebugInformationForRunIdMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::SEED_MISMATCH: - printDebugInformationForSeedMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::DATA_MISMATCH: - printDebugInformationDataBodyMismatchRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::DATA_NOT_FOUND: - printDebugInformationDataNotFoundRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; - case ErrorType::UNKNOWN: - [[ fallthrough ]]; - default: - printDebugInformationCorruptRange(read_offset, start_block_offset, - range_length_in_blocks, bufferlist); - break; +void HeaderedSeededRandomGenerator ::printDebugInformationForRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, ErrorType rangeError, + const bufferlist& bufferlist) { + switch (rangeError) { + case ErrorType::RUN_ID_MISMATCH: + printDebugInformationForRunIdMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::SEED_MISMATCH: + printDebugInformationForSeedMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_MISMATCH: + printDebugInformationDataBodyMismatchRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::DATA_NOT_FOUND: + printDebugInformationDataNotFoundRange( + read_offset, start_block_offset, range_length_in_blocks, bufferlist); + break; + case ErrorType::UNKNOWN: + [[fallthrough]]; + default: + printDebugInformationCorruptRange(read_offset, start_block_offset, + range_length_in_blocks, bufferlist); + break; } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForRunIdMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForRunIdMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { uint64_t range_start = start_block_offset; uint64_t range_length = 0; - UniqueIdBytes initial_read_unique_run_id = readUniqueRunId(start_block_offset - read_offset, - bufferlist); + UniqueIdBytes initial_read_unique_run_id = + readUniqueRunId(start_block_offset - read_offset, bufferlist); for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { - ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) - == ErrorType::RUN_ID_MISMATCH); + i < start_block_offset + range_length_in_blocks; i++) { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) == + ErrorType::RUN_ID_MISMATCH); - UniqueIdBytes read_unique_run_id = readUniqueRunId(i - read_offset, bufferlist); + UniqueIdBytes read_unique_run_id = + readUniqueRunId(i - read_offset, bufferlist); if (initial_read_unique_run_id != read_unique_run_id || - i == (start_block_offset + range_length_in_blocks - 1)) - { - if (range_length == 1) - { + i == (start_block_offset + range_length_in_blocks - 1)) { + if (range_length == 1) { printDebugInformationForBlock(read_offset, i, bufferlist); - } - else if (range_length > 1) - { - dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {} ({} bytes)" - " and spanning a range of {} blocks ({} bytes). " - "Expected run id {} for range but found id {}" - " for all blocks in range. " - "Block data corrupt or not written from this instance of this application.", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - unique_run_id, - initial_read_unique_run_id) << dendl; + } else if (range_length > 1) { + dout(0) + << fmt::format( + "Data (Run ID) Mismatch detected from block {} ({} bytes)" + " and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id {}" + " for all blocks in range. " + "Block data corrupt or not written from this instance of " + "this application.", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + unique_run_id, initial_read_unique_run_id) + << dendl; } range_start = i; range_length = 1; initial_read_unique_run_id = read_unique_run_id; - } - else - { + } else { range_length++; } } - if (range_length == 1) - { - printDebugInformationForBlock(read_offset, - start_block_offset + range_length_in_blocks - 1, - bufferlist); - } - else if (range_length > 1) - { - dout(0) << fmt::format("Data (Run ID) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Expected run id {} for range but found id for all blocks in range. " - "Block data corrupt or not written from this instance of this application.", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - unique_run_id, - initial_read_unique_run_id) + if (range_length == 1) { + printDebugInformationForBlock( + read_offset, start_block_offset + range_length_in_blocks - 1, + bufferlist); + } else if (range_length > 1) { + dout(0) << fmt::format( + "Data (Run ID) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Expected run id {} for range but found id for all blocks " + "in range. " + "Block data corrupt or not written from this instance of " + "this application.", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + unique_run_id, initial_read_unique_run_id) << dendl; } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForSeedMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForSeedMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { uint64_t range_start = start_block_offset; uint64_t range_length = 0; // Assert here if needed, as we can't support values // that can't be converted to a signed integer. - ceph_assert(m_model.get_block_size() < (std::numeric_limits<uint64_t>::max() / 2)); + ceph_assert(m_model.get_block_size() < + (std::numeric_limits<uint64_t>::max() / 2)); std::optional<int64_t> range_offset = 0; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { - ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) - == ErrorType::SEED_MISMATCH); + i < start_block_offset + range_length_in_blocks; i++) { + ceph_assert(getErrorTypeForBlock(read_offset, i, bufferlist) == + ErrorType::SEED_MISMATCH); SeedBytes read_seed = readSeed(i - read_offset, bufferlist); std::vector<int> seed_found_offsets = m_model.get_seed_offsets(read_seed); if ((seed_found_offsets.size() == 1 && - (static_cast<int64_t>(seed_found_offsets.front() - i) == range_offset)) || - range_length == 0) - { - if (range_length == 0) - { + (static_cast<int64_t>(seed_found_offsets.front() - i) == + range_offset)) || + range_length == 0) { + if (range_length == 0) { range_start = i; - if (seed_found_offsets.size() > 0) - { + if (seed_found_offsets.size() > 0) { range_offset = seed_found_offsets.front() - i; - } - else - { + } else { range_offset = std::nullopt; } } range_length++; - } - else - { - if (range_length == 1) - { + } else { + if (range_length == 1) { printDebugInformationForBlock(read_offset, i - 1, bufferlist); - } - else if (range_length > 1 && range_offset.has_value()) - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Returned data located starting from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, range_length * m_model.get_block_size(), - static_cast<uint64_t>(*range_offset) + range_start, - (static_cast<uint64_t>(*range_offset) + range_start) - * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) - << dendl; - } - else - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {}" - " ({} bytes) and spanning a range of {} blocks ({} bytes). " - "Data seed mismatch spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, range_length * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) - << dendl; + } else if (range_length > 1 && range_offset.has_value()) { + dout(0) + << fmt::format( + "Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + static_cast<uint64_t>(*range_offset) + range_start, + (static_cast<uint64_t>(*range_offset) + range_start) * + m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) + << dendl; + } else { + dout(0) + << fmt::format( + "Data (Seed) Mismatch detected from block {}" + " ({} bytes) and spanning a range of {} blocks ({} bytes). " + "Data seed mismatch spanning a range of {} blocks ({} " + "bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) + << dendl; } range_length = 1; range_start = i; - if (seed_found_offsets.size() > 0) - { + if (seed_found_offsets.size() > 0) { range_offset = seed_found_offsets.front() - i; - } - else - { + } else { range_offset = std::nullopt; } } } - if (range_length == 1) - { - printDebugInformationForBlock(read_offset, - start_block_offset + range_length_in_blocks - 1, - bufferlist); - } - else if (range_length > 1 && range_offset.has_value()) - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes). " - "Returned data located starting from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - *range_offset + range_start, - (*range_offset + range_start) * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) + if (range_length == 1) { + printDebugInformationForBlock( + read_offset, start_block_offset + range_length_in_blocks - 1, + bufferlist); + } else if (range_length > 1 && range_offset.has_value()) { + dout(0) << fmt::format( + "Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "Returned data located starting from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + *range_offset + range_start, + (*range_offset + range_start) * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) << dendl; - } - else - { - dout(0) << fmt::format("Data (Seed) Mismatch detected from block {} ({} bytes) " - "and spanning a range of {} blocks ({} bytes). " - "and spanning a range of {} blocks ({} bytes).", - range_start, - range_start * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size(), - range_length, - range_length * m_model.get_block_size()) + } else { + dout(0) << fmt::format( + "Data (Seed) Mismatch detected from block {} ({} bytes) " + "and spanning a range of {} blocks ({} bytes). " + "and spanning a range of {} blocks ({} bytes).", + range_start, range_start * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size(), + range_length, range_length * m_model.get_block_size()) << dendl; } } -void HeaderedSeededRandomGenerator -::printDebugInformationDataBodyMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " - "Headers look as expected for range, " - "but generated data body does not match. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationDataBodyMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationCorruptRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data Mismatch detected in blocks from {} to {}. " - "Headers look as expected for range, " - "but generated data body does not match. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationCorruptRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data Mismatch detected in blocks from {} to {}. " + "Headers look as expected for range, " + "but generated data body does not match. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; for (uint64_t i = start_block_offset; - i < start_block_offset + range_length_in_blocks; i++) - { + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationDataNotFoundRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist) -{ - dout(0) << fmt::format("Data not found for blocks from {} to {}. " - "More information given for individual blocks below.", - start_block_offset, - start_block_offset + range_length_in_blocks - 1) +void HeaderedSeededRandomGenerator ::printDebugInformationDataNotFoundRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist) { + dout(0) << fmt::format( + "Data not found for blocks from {} to {}. " + "More information given for individual blocks below.", + start_block_offset, + start_block_offset + range_length_in_blocks - 1) << dendl; - for (uint64_t i = start_block_offset; i < start_block_offset + range_length_in_blocks; i++) - { + for (uint64_t i = start_block_offset; + i < start_block_offset + range_length_in_blocks; i++) { printDebugInformationForBlock(read_offset, i, bufferlist); } } -void HeaderedSeededRandomGenerator - ::printDebugInformationForOffsets(uint64_t read_offset, - std::vector<uint64_t> offsets, - const bufferlist& bufferlist) -{ +void HeaderedSeededRandomGenerator ::printDebugInformationForOffsets( + uint64_t read_offset, std::vector<uint64_t> offsets, + const bufferlist& bufferlist) { uint64_t range_start = 0; uint64_t range_length = 0; ErrorType rangeError = ErrorType::UNKNOWN; - for (const uint64_t& block_offset : offsets) - { - ErrorType blockError = getErrorTypeForBlock(read_offset, block_offset, - bufferlist); + for (const uint64_t& block_offset : offsets) { + ErrorType blockError = + getErrorTypeForBlock(read_offset, block_offset, bufferlist); - if (range_start == 0 && range_length == 0) - { + if (range_start == 0 && range_length == 0) { range_start = block_offset; range_length = 1; rangeError = blockError; - } - else if (blockError == rangeError && - range_start + range_length == block_offset) -{ + } else if (blockError == rangeError && + range_start + range_length == block_offset) { range_length++; - } - else - { - if (range_length == 1) - { + } else { + if (range_length == 1) { printDebugInformationForBlock(read_offset, range_start, bufferlist); - } - else if (range_length > 1) - { + } else if (range_length > 1) { printDebugInformationForRange(read_offset, range_start, range_length, rangeError, bufferlist); } @@ -741,12 +644,9 @@ void HeaderedSeededRandomGenerator } } - if (range_length == 1) - { + if (range_length == 1) { printDebugInformationForBlock(read_offset, range_start, bufferlist); - } - else if (range_length > 1) - { + } else if (range_length > 1) { printDebugInformationForRange(read_offset, range_start, range_length, rangeError, bufferlist); } diff --git a/src/common/io_exerciser/DataGenerator.h b/src/common/io_exerciser/DataGenerator.h index 1e5784a54cc..c497c78ed61 100644 --- a/src/common/io_exerciser/DataGenerator.h +++ b/src/common/io_exerciser/DataGenerator.h @@ -3,8 +3,8 @@ #include <memory> #include <random> -#include "include/buffer.h" #include "ObjectModel.h" +#include "include/buffer.h" /* Overview * @@ -23,149 +23,139 @@ * * class HeaderedSeededRandomGenerator * Inherits from SeededDataGenerator. Generates entirely random patterns - * based on the seed retrieved by the model, however also appends a + * based on the seed retrieved by the model, however also appends a * header to the start of each block. This generator also provides * a range of verbose debug options to help disagnose a miscompare * whenever it detects unexpected data. */ namespace ceph { - namespace io_exerciser { - namespace data_generation { - enum class GenerationType { - SeededRandom, - HeaderedSeededRandom - // CompressedGenerator - // MixedGenerator - }; - - class DataGenerator { - public: - virtual ~DataGenerator() = default; - static std::unique_ptr<DataGenerator> - create_generator(GenerationType generatorType, - const ObjectModel& model); - virtual bufferlist generate_data(uint64_t length, uint64_t offset)=0; - virtual bool validate(bufferlist& bufferlist, uint64_t offset, - uint64_t length); - - // Used for testing debug outputs from data generation - virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length); - - protected: - const ObjectModel& m_model; - - DataGenerator(const ObjectModel& model) : m_model(model) {} - }; - - class SeededRandomGenerator : public DataGenerator - { - public: - SeededRandomGenerator(const ObjectModel& model) - : DataGenerator(model) {} - - virtual bufferptr generate_block(uint64_t offset); - virtual bufferlist generate_data(uint64_t length, uint64_t offset); - virtual bufferptr generate_wrong_block(uint64_t offset); - virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length) override; - }; - - class HeaderedSeededRandomGenerator : public SeededRandomGenerator - { - public: - HeaderedSeededRandomGenerator(const ObjectModel& model, - std::optional<uint64_t> unique_run_id = std::nullopt); - - bufferptr generate_block(uint64_t offset) override; - bufferptr generate_wrong_block(uint64_t offset) override; - bool validate(bufferlist& bufferlist, uint64_t offset, - uint64_t length) override; - - private: - using UniqueIdBytes = uint64_t; - using SeedBytes = int; - using TimeBytes = uint64_t; - - enum class ErrorType { - RUN_ID_MISMATCH, - SEED_MISMATCH, - DATA_MISMATCH, - DATA_NOT_FOUND, - UNKNOWN - }; - - constexpr uint8_t headerStart() const - { return 0; }; - constexpr uint8_t uniqueIdStart() const - { return headerStart(); }; - constexpr uint8_t uniqueIdLength() const - { return sizeof(UniqueIdBytes); }; - constexpr uint8_t seedStart() const - { return uniqueIdStart() + uniqueIdLength(); }; - constexpr uint8_t seedLength() const - { return sizeof(SeedBytes); }; - constexpr uint8_t timeStart() const - { return seedStart() + seedLength(); }; - constexpr uint8_t timeLength() const - { return sizeof(TimeBytes); }; - constexpr uint8_t timeEnd() const - { return timeStart() + timeLength(); }; - constexpr uint8_t headerLength() const - { return uniqueIdLength() + seedLength() + timeLength(); }; - constexpr uint8_t bodyStart() const - { return headerStart() + headerLength(); }; - - const UniqueIdBytes readUniqueRunId(uint64_t block_offset, - const bufferlist& bufferlist); - const SeedBytes readSeed(uint64_t block_offset, - const bufferlist& bufferlist); - const TimeBytes readDateTime(uint64_t block_offset, +namespace io_exerciser { +namespace data_generation { +enum class GenerationType { + SeededRandom, + HeaderedSeededRandom + // CompressedGenerator + // MixedGenerator +}; + +class DataGenerator { + public: + virtual ~DataGenerator() = default; + static std::unique_ptr<DataGenerator> create_generator( + GenerationType generatorType, const ObjectModel& model); + virtual bufferlist generate_data(uint64_t length, uint64_t offset) = 0; + virtual bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length); + + // Used for testing debug outputs from data generation + virtual bufferlist generate_wrong_data(uint64_t offset, uint64_t length); + + protected: + const ObjectModel& m_model; + + DataGenerator(const ObjectModel& model) : m_model(model) {} +}; + +class SeededRandomGenerator : public DataGenerator { + public: + SeededRandomGenerator(const ObjectModel& model) : DataGenerator(model) {} + + virtual bufferptr generate_block(uint64_t offset); + bufferlist generate_data(uint64_t length, uint64_t offset) override; + virtual bufferptr generate_wrong_block(uint64_t offset); + bufferlist generate_wrong_data(uint64_t offset, + uint64_t length) override; +}; + +class HeaderedSeededRandomGenerator : public SeededRandomGenerator { + public: + HeaderedSeededRandomGenerator( + const ObjectModel& model, + std::optional<uint64_t> unique_run_id = std::nullopt); + + bufferptr generate_block(uint64_t offset) override; + bufferptr generate_wrong_block(uint64_t offset) override; + bool validate(bufferlist& bufferlist, uint64_t offset, + uint64_t length) override; + + private: + using UniqueIdBytes = uint64_t; + using SeedBytes = int; + using TimeBytes = uint64_t; + + enum class ErrorType { + RUN_ID_MISMATCH, + SEED_MISMATCH, + DATA_MISMATCH, + DATA_NOT_FOUND, + UNKNOWN + }; + + constexpr uint8_t headerStart() const { return 0; }; + constexpr uint8_t uniqueIdStart() const { return headerStart(); }; + constexpr uint8_t uniqueIdLength() const { return sizeof(UniqueIdBytes); }; + constexpr uint8_t seedStart() const { + return uniqueIdStart() + uniqueIdLength(); + }; + constexpr uint8_t seedLength() const { return sizeof(SeedBytes); }; + constexpr uint8_t timeStart() const { return seedStart() + seedLength(); }; + constexpr uint8_t timeLength() const { return sizeof(TimeBytes); }; + constexpr uint8_t timeEnd() const { return timeStart() + timeLength(); }; + constexpr uint8_t headerLength() const { + return uniqueIdLength() + seedLength() + timeLength(); + }; + constexpr uint8_t bodyStart() const { + return headerStart() + headerLength(); + }; + + const UniqueIdBytes readUniqueRunId(uint64_t block_offset, + const bufferlist& bufferlist); + const SeedBytes readSeed(uint64_t block_offset, const bufferlist& bufferlist); + const TimeBytes readDateTime(uint64_t block_offset, + const bufferlist& bufferlist); + + const UniqueIdBytes unique_run_id; + + uint64_t generate_unique_run_id(); + + bool validate_block(uint64_t block_offset, const char* buffer_start); + + const ErrorType getErrorTypeForBlock(uint64_t read_offset, + uint64_t block_offset, const bufferlist& bufferlist); - const UniqueIdBytes unique_run_id; - - uint64_t generate_unique_run_id(); - - bool validate_block(uint64_t block_offset, const char* buffer_start); - - const ErrorType getErrorTypeForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist); - - void printDebugInformationForBlock(uint64_t read_offset, - uint64_t block_offset, - const bufferlist& bufferlist); - void printDebugInformationForRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - ErrorType rangeError, - const bufferlist& bufferlist); - - void printDebugInformationForRunIdMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationForSeedMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationDataBodyMismatchRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationDataNotFoundRange(uint64_t ßread_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - void printDebugInformationCorruptRange(uint64_t read_offset, - uint64_t start_block_offset, - uint64_t range_length_in_blocks, - const bufferlist& bufferlist); - - void printDebugInformationForOffsets(uint64_t read_offset, - std::vector<uint64_t> offsets, - const bufferlist& bufferlist); - }; - } - } -} + void printDebugInformationForBlock(uint64_t read_offset, + uint64_t block_offset, + const bufferlist& bufferlist); + void printDebugInformationForRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + ErrorType rangeError, + const bufferlist& bufferlist); + + void printDebugInformationForRunIdMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationForSeedMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationDataBodyMismatchRange( + uint64_t read_offset, uint64_t start_block_offset, + uint64_t range_length_in_blocks, const bufferlist& bufferlist); + void printDebugInformationDataNotFoundRange(uint64_t ßread_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + void printDebugInformationCorruptRange(uint64_t read_offset, + uint64_t start_block_offset, + uint64_t range_length_in_blocks, + const bufferlist& bufferlist); + + void printDebugInformationForOffsets(uint64_t read_offset, + std::vector<uint64_t> offsets, + const bufferlist& bufferlist); +}; +} // namespace data_generation +} // namespace io_exerciser +} // namespace ceph diff --git a/src/common/io_exerciser/EcIoSequence.cc b/src/common/io_exerciser/EcIoSequence.cc new file mode 100644 index 00000000000..611920c96e0 --- /dev/null +++ b/src/common/io_exerciser/EcIoSequence.cc @@ -0,0 +1,267 @@ +#include "EcIoSequence.h" + +#include <memory> + +using IoOp = ceph::io_exerciser::IoOp; +using Sequence = ceph::io_exerciser::Sequence; +using IoSequence = ceph::io_exerciser::IoSequence; +using EcIoSequence = ceph::io_exerciser::EcIoSequence; +using ReadInjectSequence = ceph::io_exerciser::ReadInjectSequence; + +bool EcIoSequence::is_supported(Sequence sequence) const { return true; } + +std::unique_ptr<IoSequence> EcIoSequence::generate_sequence( + Sequence sequence, std::pair<int, int> obj_size_range, int k, int m, + int seed) { + switch (sequence) { + case Sequence::SEQUENCE_SEQ0: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ1: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ2: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ3: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ4: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ5: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ6: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ7: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ8: + [[fallthrough]]; + case Sequence::SEQUENCE_SEQ9: + return std::make_unique<ReadInjectSequence>(obj_size_range, seed, + sequence, k, m); + case Sequence::SEQUENCE_SEQ10: + return std::make_unique<Seq10>(obj_size_range, seed, k, m); + default: + ceph_abort_msg("Unrecognised sequence"); + } +} + +EcIoSequence::EcIoSequence(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + setup_inject(false), + clear_inject(false), + shard_to_inject(std::nullopt) {} + +void EcIoSequence::select_random_data_shard_to_inject_read_error(int k, int m) { + shard_to_inject = rng(k - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_data_shard_to_inject_write_error(int k, + int m) { + // Write errors do not support injecting to the primary OSD + shard_to_inject = rng(1, k - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_shard_to_inject_read_error(int k, int m) { + shard_to_inject = rng(k + m - 1); + setup_inject = true; +} + +void EcIoSequence::select_random_shard_to_inject_write_error(int k, int m) { + // Write errors do not support injecting to the primary OSD + shard_to_inject = rng(1, k + m - 1); + setup_inject = true; +} + +void EcIoSequence::generate_random_read_inject_type() { + inject_op_type = static_cast<InjectOpType>( + rng(static_cast<int>(InjectOpType::ReadEIO), + static_cast<int>(InjectOpType::ReadMissingShard))); +} + +void EcIoSequence::generate_random_write_inject_type() { + inject_op_type = static_cast<InjectOpType>( + rng(static_cast<int>(InjectOpType::WriteFailAndRollback), + static_cast<int>(InjectOpType::WriteOSDAbort))); +} + +ceph::io_exerciser::ReadInjectSequence::ReadInjectSequence( + std::pair<int, int> obj_size_range, int seed, Sequence s, int k, int m) + : EcIoSequence(obj_size_range, seed) { + child_sequence = IoSequence::generate_sequence(s, obj_size_range, seed); + select_random_data_shard_to_inject_read_error(k, m); + generate_random_read_inject_type(); +} + +Sequence ceph::io_exerciser::ReadInjectSequence::get_id() const { + return child_sequence->get_id(); +} + +std::string ceph::io_exerciser::ReadInjectSequence::get_name() const { + return child_sequence->get_name() + + " running with read errors injected on shard " + + std::to_string(*shard_to_inject); +} + +std::unique_ptr<IoOp> ReadInjectSequence::next() { + step++; + + if (nextOp) { + std::unique_ptr<IoOp> retOp = nullptr; + nextOp.swap(retOp); + return retOp; + } + + std::unique_ptr<IoOp> childOp = child_sequence->next(); + + switch (childOp->getOpType()) { + case OpType::Remove: + nextOp.swap(childOp); + switch (inject_op_type) { + case InjectOpType::ReadEIO: + return ClearReadErrorInjectOp::generate(*shard_to_inject, 0); + case InjectOpType::ReadMissingShard: + return ClearReadErrorInjectOp::generate(*shard_to_inject, 1); + case InjectOpType::WriteFailAndRollback: + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 0); + case InjectOpType::WriteOSDAbort: + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 3); + case InjectOpType::None: + [[fallthrough]]; + default: + ceph_abort_msg("Unsupported operation"); + } + break; + case OpType::Create: + switch (inject_op_type) { + case InjectOpType::ReadEIO: + nextOp = InjectReadErrorOp::generate( + *shard_to_inject, 0, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::ReadMissingShard: + nextOp = InjectReadErrorOp::generate( + *shard_to_inject, 1, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::WriteFailAndRollback: + nextOp = InjectWriteErrorOp::generate( + *shard_to_inject, 0, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::WriteOSDAbort: + nextOp = InjectWriteErrorOp::generate( + *shard_to_inject, 3, 0, std::numeric_limits<uint64_t>::max()); + break; + case InjectOpType::None: + [[fallthrough]]; + default: + ceph_abort_msg("Unsupported operation"); + } + break; + default: + // Do nothing in default case + break; + } + + return childOp; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> +ceph::io_exerciser::ReadInjectSequence::_next() { + ceph_abort_msg( + "Should not reach this point, " + "this sequence should only consume complete sequences"); + + return DoneOp::generate(); +} + +ceph::io_exerciser::Seq10::Seq10(std::pair<int, int> obj_size_range, int seed, + int k, int m) + : EcIoSequence(obj_size_range, seed), + offset(0), + length(1), + inject_error_done(false), + failed_write_done(false), + read_done(false), + successful_write_done(false), + test_all_lengths(false), // Only test length(1) due to time constraints + test_all_sizes( + false) // Only test obj_size(rand()) due to time constraints +{ + select_random_shard_to_inject_write_error(k, m); + // We will inject specifically as part of our sequence in this sequence + setup_inject = false; + if (!test_all_sizes) { + select_random_object_size(); + } +} + +Sequence ceph::io_exerciser::Seq10::get_id() const { + return Sequence::SEQUENCE_SEQ10; +} + +std::string ceph::io_exerciser::Seq10::get_name() const { + return "Sequential writes of length " + std::to_string(length) + + " with queue depth 1" + " first injecting a failed write and read it to ensure it rolls back, " + "then" + " successfully writing the data and reading the write the ensure it " + "is applied"; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq10::_next() { + if (!inject_error_done) { + inject_error_done = true; + return InjectWriteErrorOp::generate(*shard_to_inject, 0, 0, + std::numeric_limits<uint64_t>::max()); + } else if (!failed_write_done) { + failed_write_done = true; + read_done = false; + barrier = true; + return SingleFailedWriteOp::generate(offset, length); + } else if (failed_write_done && !read_done) { + read_done = true; + barrier = true; + return SingleReadOp::generate(offset, length); + } else if (!clear_inject_done) { + clear_inject_done = true; + return ClearWriteErrorInjectOp::generate(*shard_to_inject, 0); + } else if (!successful_write_done) { + successful_write_done = true; + read_done = false; + barrier = true; + return SingleWriteOp::generate(offset, length); + } else if (successful_write_done && !read_done) { + read_done = true; + return SingleReadOp::generate(offset, length); + } else if (successful_write_done && read_done) { + offset++; + inject_error_done = false; + failed_write_done = false; + read_done = false; + clear_inject_done = false; + successful_write_done = false; + + if (offset + length >= obj_size) { + if (!test_all_lengths) { + remove = true; + done = true; + return BarrierOp::generate(); + } + + offset = 0; + length++; + if (length > obj_size) { + if (!test_all_sizes) { + remove = true; + done = true; + return BarrierOp::generate(); + } + + length = 1; + return increment_object_size(); + } + } + + return BarrierOp::generate(); + } else { + ceph_abort_msg("Sequence in undefined state. Aborting"); + return DoneOp::generate(); + } +}
\ No newline at end of file diff --git a/src/common/io_exerciser/EcIoSequence.h b/src/common/io_exerciser/EcIoSequence.h new file mode 100644 index 00000000000..37283b3906b --- /dev/null +++ b/src/common/io_exerciser/EcIoSequence.h @@ -0,0 +1,65 @@ +#include "IoSequence.h" + +namespace ceph { +namespace io_exerciser { +class EcIoSequence : public IoSequence { + public: + virtual bool is_supported(Sequence sequence) const override; + static std::unique_ptr<IoSequence> generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int k, int m, int seed); + + protected: + bool setup_inject; + bool clear_inject; + std::optional<uint64_t> shard_to_inject; + InjectOpType inject_op_type; + + EcIoSequence(std::pair<int, int> obj_size_range, int seed); + + // Writes cannot be sent to injected on shard zero, so selections seperated + // out + void select_random_data_shard_to_inject_read_error(int k, int m); + void select_random_data_shard_to_inject_write_error(int k, int m); + void select_random_shard_to_inject_read_error(int k, int m); + void select_random_shard_to_inject_write_error(int k, int m); + void generate_random_read_inject_type(); + void generate_random_write_inject_type(); +}; + +class ReadInjectSequence : public EcIoSequence { + public: + ReadInjectSequence(std::pair<int, int> obj_size_range, int seed, Sequence s, + int k, int m); + + Sequence get_id() const override; + std::string get_name() const override; + virtual std::unique_ptr<IoOp> next() override; + std::unique_ptr<IoOp> _next() override; + + private: + std::unique_ptr<IoSequence> child_sequence; + std::unique_ptr<IoOp> nextOp; +}; + +class Seq10 : public EcIoSequence { + public: + Seq10(std::pair<int, int> obj_size_range, int seed, int k, int m); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + + bool inject_error_done; + bool failed_write_done; + bool read_done; + bool clear_inject_done; + bool successful_write_done; + bool test_all_lengths; + bool test_all_sizes; +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/IoOp.cc b/src/common/io_exerciser/IoOp.cc index cd855ba6fff..493d1f435b4 100644 --- a/src/common/io_exerciser/IoOp.cc +++ b/src/common/io_exerciser/IoOp.cc @@ -1,188 +1,316 @@ #include "IoOp.h" -using IoOp = ceph::io_exerciser::IoOp; +#include "fmt/format.h" +#include "include/ceph_assert.h" -IoOp::IoOp( OpType op, - uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) : - op(op), - offset1(offset1), length1(length1), - offset2(offset2), length2(length2), - offset3(offset3), length3(length3) -{ +using IoOp = ceph::io_exerciser::IoOp; +using OpType = ceph::io_exerciser::OpType; -} +using DoneOp = ceph::io_exerciser::DoneOp; +using BarrierOp = ceph::io_exerciser::BarrierOp; +using CreateOp = ceph::io_exerciser::CreateOp; +using RemoveOp = ceph::io_exerciser::RemoveOp; +using SingleReadOp = ceph::io_exerciser::SingleReadOp; +using DoubleReadOp = ceph::io_exerciser::DoubleReadOp; +using TripleReadOp = ceph::io_exerciser::TripleReadOp; +using SingleWriteOp = ceph::io_exerciser::SingleWriteOp; +using DoubleWriteOp = ceph::io_exerciser::DoubleWriteOp; +using TripleWriteOp = ceph::io_exerciser::TripleWriteOp; +using SingleFailedWriteOp = ceph::io_exerciser::SingleFailedWriteOp; +using DoubleFailedWriteOp = ceph::io_exerciser::DoubleFailedWriteOp; +using TripleFailedWriteOp = ceph::io_exerciser::TripleFailedWriteOp; -std::string IoOp::value_to_string(uint64_t v) const -{ +namespace { +std::string value_to_string(uint64_t v) { if (v < 1024 || (v % 1024) != 0) { return std::to_string(v); - }else if (v < 1024*1024 || (v % (1024 * 1024)) != 0 ) { + } else if (v < 1024 * 1024 || (v % (1024 * 1024)) != 0) { return std::to_string(v / 1024) + "K"; - }else{ + } else { return std::to_string(v / 1024 / 1024) + "M"; } } +} // namespace -std::unique_ptr<IoOp> IoOp - ::generate_done() { +IoOp::IoOp() {} - return std::make_unique<IoOp>(OpType::Done); -} +template <OpType opType> +ceph::io_exerciser::TestOp<opType>::TestOp() : IoOp() {} + +DoneOp::DoneOp() : TestOp<OpType::Done>() {} -std::unique_ptr<IoOp> IoOp - ::generate_barrier() { +std::string DoneOp::to_string(uint64_t block_size) const { return "Done"; } - return std::make_unique<IoOp>(OpType::BARRIER); +std::unique_ptr<DoneOp> DoneOp::generate() { + return std::make_unique<DoneOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_create(uint64_t size) { +BarrierOp::BarrierOp() : TestOp<OpType::Barrier>() {} - return std::make_unique<IoOp>(OpType::CREATE,0,size); +std::unique_ptr<BarrierOp> BarrierOp::generate() { + return std::make_unique<BarrierOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_remove() { - - return std::make_unique<IoOp>(OpType::REMOVE); +std::string BarrierOp::to_string(uint64_t block_size) const { + return "Barrier"; } -std::unique_ptr<IoOp> IoOp - ::generate_read(uint64_t offset, uint64_t length) { +CreateOp::CreateOp(uint64_t size) : TestOp<OpType::Create>(), size(size) {} - return std::make_unique<IoOp>(OpType::READ, offset, length); +std::unique_ptr<CreateOp> CreateOp::generate(uint64_t size) { + return std::make_unique<CreateOp>(size); } -std::unique_ptr<IoOp> IoOp - ::generate_read2(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2) { +std::string CreateOp::to_string(uint64_t block_size) const { + return "Create (size=" + value_to_string(size * block_size) + ")"; +} - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } +RemoveOp::RemoveOp() : TestOp<OpType::Remove>() {} - return std::make_unique<IoOp>(OpType::READ2, - offset1, length1, - offset2, length2); +std::unique_ptr<RemoveOp> RemoveOp::generate() { + return std::make_unique<RemoveOp>(); } -std::unique_ptr<IoOp> IoOp - ::generate_read3(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) { +std::string RemoveOp::to_string(uint64_t block_size) const { return "Remove"; } - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); +template <OpType opType, int numIOs> +ceph::io_exerciser::ReadWriteOp<opType, numIOs>::ReadWriteOp( + std::array<uint64_t, numIOs>&& offset, + std::array<uint64_t, numIOs>&& length) + : TestOp<opType>(), offset(offset), length(length) { + auto compare = [](uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2) { + if (offset1 < offset2) { + ceph_assert(offset1 + length1 <= offset2); + } else { + ceph_assert(offset2 + length2 <= offset1); + } + }; + + if (numIOs > 1) { + for (int i = 0; i < numIOs - 1; i++) { + for (int j = i + 1; j < numIOs; j++) { + compare(offset[i], length[i], offset[j], length[j]); + } + } } - if (offset1 < offset3) { - ceph_assert( offset1 + length1 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset1 ); +} + +template <OpType opType, int numIOs> +std::string ceph::io_exerciser::ReadWriteOp<opType, numIOs>::to_string( + uint64_t block_size) const { + std::string offset_length_desc; + if (numIOs > 0) { + offset_length_desc += fmt::format( + "offset1={}", value_to_string(this->offset[0] * block_size)); + offset_length_desc += fmt::format( + ",length1={}", value_to_string(this->length[0] * block_size)); + for (int i = 1; i < numIOs; i++) { + offset_length_desc += fmt::format( + ",offset{}={}", i + 1, value_to_string(this->offset[i] * block_size)); + offset_length_desc += fmt::format( + ",length{}={}", i + 1, value_to_string(this->length[i] * block_size)); + } } - if (offset2 < offset3) { - ceph_assert( offset2 + length2 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset2 ); + switch (opType) { + case OpType::Read: + [[fallthrough]]; + case OpType::Read2: + [[fallthrough]]; + case OpType::Read3: + return fmt::format("Read{} ({})", numIOs, offset_length_desc); + case OpType::Write: + [[fallthrough]]; + case OpType::Write2: + [[fallthrough]]; + case OpType::Write3: + return fmt::format("Write{} ({})", numIOs, offset_length_desc); + case OpType::FailedWrite: + [[fallthrough]]; + case OpType::FailedWrite2: + [[fallthrough]]; + case OpType::FailedWrite3: + return fmt::format("FailedWrite{} ({})", numIOs, offset_length_desc); + default: + ceph_abort_msg( + fmt::format("Unsupported op type by ReadWriteOp ({})", opType)); } - return std::make_unique<IoOp>(OpType::READ3, - offset1, length1, - offset2, length2, - offset3, length3); } -std::unique_ptr<IoOp> IoOp::generate_write(uint64_t offset, uint64_t length) { - return std::make_unique<IoOp>(OpType::WRITE, offset, length); +SingleReadOp::SingleReadOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::Read, 1>({offset}, {length}) {} + +std::unique_ptr<SingleReadOp> SingleReadOp::generate(uint64_t offset, + uint64_t length) { + return std::make_unique<SingleReadOp>(offset, length); } -std::unique_ptr<IoOp> IoOp::generate_write2(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2) { - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } - return std::make_unique<IoOp>(OpType::WRITE2, - offset1, length1, - offset2, length2); +DoubleReadOp::DoubleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2) + : ReadWriteOp<OpType::Read2, 2>({offset1, offset2}, {length1, length2}) {} + +std::unique_ptr<DoubleReadOp> DoubleReadOp::generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2) { + return std::make_unique<DoubleReadOp>(offset1, length1, offset2, length2); } -std::unique_ptr<IoOp> IoOp::generate_write3(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3) { - if (offset1 < offset2) { - ceph_assert( offset1 + length1 <= offset2 ); - } else { - ceph_assert( offset2 + length2 <= offset1 ); - } - if (offset1 < offset3) { - ceph_assert( offset1 + length1 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset1 ); - } - if (offset2 < offset3) { - ceph_assert( offset2 + length2 <= offset3 ); - } else { - ceph_assert( offset3 + length3 <= offset2 ); - } - return std::make_unique<IoOp>(OpType::WRITE3, - offset1, length1, - offset2, length2, - offset3, length3); -} - -bool IoOp::done() { - return (op == OpType::Done); -} - -std::string IoOp::to_string(uint64_t block_size) const -{ - switch (op) { - case OpType::Done: - return "Done"; - case OpType::BARRIER: - return "Barrier"; - case OpType::CREATE: - return "Create (size=" + value_to_string(length1 * block_size) + ")"; - case OpType::REMOVE: - return "Remove"; - case OpType::READ: - return "Read (offset=" + value_to_string(offset1 * block_size) + - ",length=" + value_to_string(length1 * block_size) + ")"; - case OpType::READ2: - return "Read2 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + ")"; - case OpType::READ3: - return "Read3 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + - ",offset3=" + value_to_string(offset3 * block_size) + - ",length3=" + value_to_string(length3 * block_size) + ")"; - case OpType::WRITE: - return "Write (offset=" + value_to_string(offset1 * block_size) + - ",length=" + value_to_string(length1 * block_size) + ")"; - case OpType::WRITE2: - return "Write2 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + ")"; - case OpType::WRITE3: - return "Write3 (offset1=" + value_to_string(offset1 * block_size) + - ",length1=" + value_to_string(length1 * block_size) + - ",offset2=" + value_to_string(offset2 * block_size) + - ",length2=" + value_to_string(length2 * block_size) + - ",offset3=" + value_to_string(offset3 * block_size) + - ",length3=" + value_to_string(length3 * block_size) + ")"; - default: - break; - } - return "Unknown"; +TripleReadOp::TripleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::Read3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleReadOp> TripleReadOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleReadOp>(offset1, length1, offset2, length2, + offset3, length3); +} + +SingleWriteOp::SingleWriteOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::Write, 1>({offset}, {length}) {} + +std::unique_ptr<SingleWriteOp> SingleWriteOp::generate(uint64_t offset, + uint64_t length) { + return std::make_unique<SingleWriteOp>(offset, length); +} + +DoubleWriteOp::DoubleWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) + : ReadWriteOp<OpType::Write2, 2>({offset1, offset2}, {length1, length2}) {} + +std::unique_ptr<DoubleWriteOp> DoubleWriteOp::generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2) { + return std::make_unique<DoubleWriteOp>(offset1, length1, offset2, length2); +} + +TripleWriteOp::TripleWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::Write3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleWriteOp> TripleWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleWriteOp>(offset1, length1, offset2, length2, + offset3, length3); +} + +SingleFailedWriteOp::SingleFailedWriteOp(uint64_t offset, uint64_t length) + : ReadWriteOp<OpType::FailedWrite, 1>({offset}, {length}) {} + +std::unique_ptr<SingleFailedWriteOp> SingleFailedWriteOp::generate( + uint64_t offset, uint64_t length) { + return std::make_unique<SingleFailedWriteOp>(offset, length); +} + +DoubleFailedWriteOp::DoubleFailedWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2) + : ReadWriteOp<OpType::FailedWrite2, 2>({offset1, offset2}, + {length1, length2}) {} + +std::unique_ptr<DoubleFailedWriteOp> DoubleFailedWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2) { + return std::make_unique<DoubleFailedWriteOp>(offset1, length1, offset2, + length2); +} + +TripleFailedWriteOp::TripleFailedWriteOp(uint64_t offset1, uint64_t length1, + uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) + : ReadWriteOp<OpType::FailedWrite3, 3>({offset1, offset2, offset3}, + {length1, length2, length3}) {} + +std::unique_ptr<TripleFailedWriteOp> TripleFailedWriteOp::generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3) { + return std::make_unique<TripleFailedWriteOp>(offset1, length1, offset2, + length2, offset3, length3); +} + +template <ceph::io_exerciser::OpType opType> +ceph::io_exerciser::InjectErrorOp<opType>::InjectErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : TestOp<opType>(), + shard(shard), + type(type), + when(when), + duration(duration) {} + +template <ceph::io_exerciser::OpType opType> +std::string ceph::io_exerciser::InjectErrorOp<opType>::to_string( + uint64_t blocksize) const { + std::string_view inject_type = get_inject_type_string(); + return fmt::format( + "Inject {} error on shard {} of type {}" + " after {} successful inject(s) lasting {} inject(s)", + inject_type, shard, type.value_or(0), when.value_or(0), + duration.value_or(1)); +} + +ceph::io_exerciser::InjectReadErrorOp::InjectReadErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : InjectErrorOp<OpType::InjectReadError>(shard, type, when, duration) {} + +std::unique_ptr<ceph::io_exerciser::InjectReadErrorOp> +ceph::io_exerciser ::InjectReadErrorOp::generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) { + return std::make_unique<InjectReadErrorOp>(shard, type, when, duration); +} + +ceph::io_exerciser::InjectWriteErrorOp::InjectWriteErrorOp( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) + : InjectErrorOp<OpType::InjectWriteError>(shard, type, when, duration) {} + +std::unique_ptr<ceph::io_exerciser::InjectWriteErrorOp> +ceph::io_exerciser ::InjectWriteErrorOp::generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration) { + return std::make_unique<InjectWriteErrorOp>(shard, type, when, duration); +} + +template <ceph::io_exerciser::OpType opType> +ceph::io_exerciser::ClearErrorInjectOp<opType>::ClearErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : TestOp<opType>(), shard(shard), type(type) {} + +template <ceph::io_exerciser::OpType opType> +std::string ceph::io_exerciser::ClearErrorInjectOp<opType>::to_string( + uint64_t blocksize) const { + std::string_view inject_type = get_inject_type_string(); + return fmt::format("Clear {} injects on shard {} of type {}", inject_type, + shard, type.value_or(0)); +} + +ceph::io_exerciser::ClearReadErrorInjectOp::ClearReadErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : ClearErrorInjectOp<OpType::ClearReadErrorInject>(shard, type) {} + +std::unique_ptr<ceph::io_exerciser::ClearReadErrorInjectOp> +ceph::io_exerciser ::ClearReadErrorInjectOp::generate( + int shard, const std::optional<uint64_t>& type) { + return std::make_unique<ClearReadErrorInjectOp>(shard, type); +} + +ceph::io_exerciser::ClearWriteErrorInjectOp::ClearWriteErrorInjectOp( + int shard, const std::optional<uint64_t>& type) + : ClearErrorInjectOp<OpType::ClearWriteErrorInject>(shard, type) {} + +std::unique_ptr<ceph::io_exerciser::ClearWriteErrorInjectOp> +ceph::io_exerciser ::ClearWriteErrorInjectOp::generate( + int shard, const std::optional<uint64_t>& type) { + return std::make_unique<ClearWriteErrorInjectOp>(shard, type); }
\ No newline at end of file diff --git a/src/common/io_exerciser/IoOp.h b/src/common/io_exerciser/IoOp.h index 60c02a93d4e..1887eafcc1f 100644 --- a/src/common/io_exerciser/IoOp.h +++ b/src/common/io_exerciser/IoOp.h @@ -1,94 +1,248 @@ #pragma once -#include <string> +#include <array> #include <memory> -#include "include/ceph_assert.h" +#include <optional> +#include <string> + +#include "OpType.h" /* Overview * - * enum OpType - * Enumeration of different types of I/O operation - * * class IoOp * Stores details for an I/O operation. Generated by IoSequences * and applied by IoExerciser's */ namespace ceph { - namespace io_exerciser { - - enum class OpType { - Done, // End of I/O sequence - BARRIER, // Barrier - all prior I/Os must complete - CREATE, // Create object and pattern with data - REMOVE, // Remove object - READ, // Read - READ2, // 2 Reads in one op - READ3, // 3 Reads in one op - WRITE, // Write - WRITE2, // 2 Writes in one op - WRITE3 // 3 Writes in one op - }; - - class IoOp { - protected: - std::string value_to_string(uint64_t v) const; - - public: - OpType op; - uint64_t offset1; - uint64_t length1; - uint64_t offset2; - uint64_t length2; - uint64_t offset3; - uint64_t length3; - - IoOp( OpType op, - uint64_t offset1 = 0, uint64_t length1 = 0, - uint64_t offset2 = 0, uint64_t length2 = 0, - uint64_t offset3 = 0, uint64_t length3 = 0 ); - - static std::unique_ptr<IoOp> generate_done(); - - static std::unique_ptr<IoOp> generate_barrier(); - - static std::unique_ptr<IoOp> generate_create(uint64_t size); - - static std::unique_ptr<IoOp> generate_remove(); - - static std::unique_ptr<IoOp> generate_read(uint64_t offset, +namespace io_exerciser { + +class IoOp { + public: + IoOp(); + virtual ~IoOp() = default; + virtual std::string to_string(uint64_t block_size) const = 0; + virtual constexpr OpType getOpType() const = 0; +}; + +template <OpType opType> +class TestOp : public IoOp { + public: + TestOp(); + constexpr OpType getOpType() const override { return opType; } +}; + +class DoneOp : public TestOp<OpType::Done> { + public: + DoneOp(); + static std::unique_ptr<DoneOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +class BarrierOp : public TestOp<OpType::Barrier> { + public: + BarrierOp(); + static std::unique_ptr<BarrierOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +class CreateOp : public TestOp<OpType::Create> { + public: + CreateOp(uint64_t size); + static std::unique_ptr<CreateOp> generate(uint64_t size); + std::string to_string(uint64_t block_size) const override; + uint64_t size; +}; + +class RemoveOp : public TestOp<OpType::Remove> { + public: + RemoveOp(); + static std::unique_ptr<RemoveOp> generate(); + std::string to_string(uint64_t block_size) const override; +}; + +template <OpType opType, int numIOs> +class ReadWriteOp : public TestOp<opType> { + public: + std::array<uint64_t, numIOs> offset; + std::array<uint64_t, numIOs> length; + + protected: + ReadWriteOp(std::array<uint64_t, numIOs>&& offset, + std::array<uint64_t, numIOs>&& length); + std::string to_string(uint64_t block_size) const override; +}; + +class SingleReadOp : public ReadWriteOp<OpType::Read, 1> { + public: + SingleReadOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleReadOp> generate(uint64_t offset, + uint64_t length); +}; + +class DoubleReadOp : public ReadWriteOp<OpType::Read2, 2> { + public: + DoubleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleReadOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleReadOp : public ReadWriteOp<OpType::Read3, 3> { + public: + TripleReadOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleReadOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +class SingleWriteOp : public ReadWriteOp<OpType::Write, 1> { + public: + SingleWriteOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleWriteOp> generate(uint64_t offset, uint64_t length); +}; + +class DoubleWriteOp : public ReadWriteOp<OpType::Write2, 2> { + public: + DoubleWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleWriteOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleWriteOp : public ReadWriteOp<OpType::Write3, 3> { + public: + TripleWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleWriteOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +class SingleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite, 1> { + public: + SingleFailedWriteOp(uint64_t offset, uint64_t length); + static std::unique_ptr<SingleFailedWriteOp> generate(uint64_t offset, + uint64_t length); +}; + +class DoubleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite2, 2> { + public: + DoubleFailedWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2); + static std::unique_ptr<DoubleFailedWriteOp> generate(uint64_t offset1, + uint64_t length1, + uint64_t offset2, + uint64_t length2); +}; + +class TripleFailedWriteOp : public ReadWriteOp<OpType::FailedWrite3, 3> { + public: + TripleFailedWriteOp(uint64_t offset1, uint64_t length1, uint64_t offset2, + uint64_t length2, uint64_t offset3, uint64_t length3); + static std::unique_ptr<TripleFailedWriteOp> generate( + uint64_t offset1, uint64_t length1, uint64_t offset2, uint64_t length2, + uint64_t offset3, uint64_t length3); +}; + +template <ceph::io_exerciser::OpType opType> +class InjectErrorOp : public TestOp<opType> { + public: + InjectErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + std::string to_string(uint64_t block_size) const override; + + int shard; + std::optional<uint64_t> type; + std::optional<uint64_t> when; + std::optional<uint64_t> duration; + + protected: + virtual inline constexpr std::string_view get_inject_type_string() const = 0; +}; + +class InjectReadErrorOp : public InjectErrorOp<OpType::InjectReadError> { + public: + InjectReadErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + static std::unique_ptr<InjectReadErrorOp> generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "read"; + } +}; + +class InjectWriteErrorOp : public InjectErrorOp<OpType::InjectWriteError> { + public: + InjectWriteErrorOp(int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + static std::unique_ptr<InjectWriteErrorOp> generate( + int shard, const std::optional<uint64_t>& type, + const std::optional<uint64_t>& when, + const std::optional<uint64_t>& duration); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "write"; + } +}; + +template <ceph::io_exerciser::OpType opType> +class ClearErrorInjectOp : public TestOp<opType> { + public: + ClearErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + std::string to_string(uint64_t block_size) const override; + + int shard; + std::optional<uint64_t> type; + + protected: + virtual inline constexpr std::string_view get_inject_type_string() const = 0; +}; + +class ClearReadErrorInjectOp + : public ClearErrorInjectOp<OpType::ClearReadErrorInject> { + public: + ClearReadErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + static std::unique_ptr<ClearReadErrorInjectOp> generate( + int shard, const std::optional<uint64_t>& type); + + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "read"; + } +}; + +class ClearWriteErrorInjectOp + : public ClearErrorInjectOp<OpType::ClearWriteErrorInject> { + public: + ClearWriteErrorInjectOp(int shard, const std::optional<uint64_t>& type); + + static std::unique_ptr<ClearWriteErrorInjectOp> generate( + int shard, const std::optional<uint64_t>& type); - static std::unique_ptr<IoOp> generate_read2(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2); - - static std::unique_ptr<IoOp> generate_read3(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2, - uint64_t offset3, - uint64_t length3); - - static std::unique_ptr<IoOp> generate_write(uint64_t offset, - uint64_t length); - - static std::unique_ptr<IoOp> generate_write2(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2); - - static std::unique_ptr<IoOp> generate_write3(uint64_t offset1, - uint64_t length1, - uint64_t offset2, - uint64_t length2, - uint64_t offset3, - uint64_t length3); - - bool done(); - - std::string to_string(uint64_t block_size) const; - }; + protected: + inline constexpr std::string_view get_inject_type_string() const override { + return "write"; } -}
\ No newline at end of file +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.cc b/src/common/io_exerciser/IoSequence.cc index 4a7ca0593d1..83f1cc595a5 100644 --- a/src/common/io_exerciser/IoSequence.cc +++ b/src/common/io_exerciser/IoSequence.cc @@ -1,12 +1,12 @@ #include "IoSequence.h" +using IoOp = ceph::io_exerciser::IoOp; using Sequence = ceph::io_exerciser::Sequence; using IoSequence = ceph::io_exerciser::IoSequence; -std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& seq) -{ - switch (seq) - { +std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, + const Sequence& seq) { + switch (seq) { case Sequence::SEQUENCE_SEQ0: os << "SEQUENCE_SEQ0"; break; @@ -37,6 +37,9 @@ std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& s case Sequence::SEQUENCE_SEQ9: os << "SEQUENCE_SEQ9"; break; + case Sequence::SEQUENCE_SEQ10: + os << "SEQUENCE_SEQ10"; + break; case Sequence::SEQUENCE_END: os << "SEQUENCE_END"; break; @@ -44,19 +47,12 @@ std::ostream& ceph::io_exerciser::operator<<(std::ostream& os, const Sequence& s return os; } -IoSequence::IoSequence(std::pair<int,int> obj_size_range, - int seed) : - min_obj_size(obj_size_range.first), max_obj_size(obj_size_range.second), - create(true), barrier(false), done(false), remove(false), - obj_size(min_obj_size), step(-1), seed(seed) -{ - rng.seed(seed); +bool IoSequence::is_supported(Sequence sequence) const { + return sequence != Sequence::SEQUENCE_SEQ10; } -std::unique_ptr<IoSequence> IoSequence::generate_sequence(Sequence s, - std::pair<int,int> obj_size_range, - int seed) -{ +std::unique_ptr<IoSequence> IoSequence::generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int seed) { switch (s) { case Sequence::SEQUENCE_SEQ0: return std::make_unique<Seq0>(obj_size_range, seed); @@ -78,24 +74,39 @@ std::unique_ptr<IoSequence> IoSequence::generate_sequence(Sequence s, return std::make_unique<Seq8>(obj_size_range, seed); case Sequence::SEQUENCE_SEQ9: return std::make_unique<Seq9>(obj_size_range, seed); + case Sequence::SEQUENCE_SEQ10: + ceph_abort_msg( + "Sequence 10 only supported for erasure coded pools " + "through the EcIoSequence interface"); + return nullptr; default: break; } return nullptr; } -int IoSequence::get_step() const -{ - return step; +IoSequence::IoSequence(std::pair<int, int> obj_size_range, int seed) + : min_obj_size(obj_size_range.first), + max_obj_size(obj_size_range.second), + create(true), + barrier(false), + done(false), + remove(false), + obj_size(min_obj_size), + step(-1), + seed(seed) { + rng.seed(seed); } -int IoSequence::get_seed() const -{ - return seed; +std::string ceph::io_exerciser::IoSequence::get_name_with_seqseed() const { + return get_name() + " (seqseed " + std::to_string(get_seed()) + ")"; } -void IoSequence::set_min_object_size(uint64_t size) -{ +int IoSequence::get_step() const { return step; } + +int IoSequence::get_seed() const { return seed; } + +void IoSequence::set_min_object_size(uint64_t size) { min_obj_size = size; if (obj_size < size) { obj_size = size; @@ -105,23 +116,20 @@ void IoSequence::set_min_object_size(uint64_t size) } } -void IoSequence::set_max_object_size(uint64_t size) -{ +void IoSequence::set_max_object_size(uint64_t size) { max_obj_size = size; if (obj_size > size) { done = true; } } -void IoSequence::select_random_object_size() -{ +void IoSequence::select_random_object_size() { if (max_obj_size != min_obj_size) { obj_size = min_obj_size + rng(max_obj_size - min_obj_size); } } -std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::increment_object_size() -{ +std::unique_ptr<IoOp> IoSequence::increment_object_size() { obj_size++; if (obj_size > max_obj_size) { done = true; @@ -129,106 +137,118 @@ std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::increment_object_size() create = true; barrier = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } -std::unique_ptr<ceph::io_exerciser::IoOp> IoSequence::next() -{ +Sequence IoSequence::getNextSupportedSequenceId() const { + Sequence sequence = get_id(); + ++sequence; + for (; sequence < Sequence::SEQUENCE_END; ++sequence) { + if (is_supported(sequence)) { + return sequence; + } + } + + return Sequence::SEQUENCE_END; +} + +std::unique_ptr<IoOp> IoSequence::next() { step++; if (remove) { remove = false; - return IoOp::generate_remove(); + return RemoveOp::generate(); } if (barrier) { barrier = false; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } if (done) { - return IoOp::generate_done(); + return DoneOp::generate(); } if (create) { create = false; barrier = true; - return IoOp::generate_create(obj_size); + return CreateOp::generate(obj_size); } return _next(); } - - -ceph::io_exerciser::Seq0::Seq0(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0) -{ +ceph::io_exerciser::Seq0::Seq0(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0) { select_random_object_size(); length = 1 + rng(obj_size - 1); } -std::string ceph::io_exerciser::Seq0::get_name() const -{ +Sequence ceph::io_exerciser::Seq0::get_id() const { + return Sequence::SEQUENCE_SEQ0; +} + +std::string ceph::io_exerciser::Seq0::get_name() const { return "Sequential reads of length " + std::to_string(length) + - " with queue depth 1 (seqseed " + std::to_string(get_seed()) + ")"; + " with queue depth 1"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq0::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq0::_next() { std::unique_ptr<IoOp> r; if (offset >= obj_size) { done = true; barrier = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } if (offset + length > obj_size) { - r = IoOp::generate_read(offset, obj_size - offset); + r = SingleReadOp::generate(offset, obj_size - offset); } else { - r = IoOp::generate_read(offset, length); + r = SingleReadOp::generate(offset, length); } offset += length; return r; } - - -ceph::io_exerciser::Seq1::Seq1(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed) -{ +ceph::io_exerciser::Seq1::Seq1(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed) { select_random_object_size(); count = 3 * obj_size; } -std::string ceph::io_exerciser::Seq1::get_name() const -{ - return "Random offset, random length read/write I/O with queue depth 1 (seqseed " - + std::to_string(get_seed()) + ")"; +Sequence ceph::io_exerciser::Seq1::get_id() const { + return Sequence::SEQUENCE_SEQ1; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq1::_next() -{ +std::string ceph::io_exerciser::Seq1::get_name() const { + return "Random offset, random length read/write I/O with queue depth 1"; +} + +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq1::_next() { barrier = true; if (count-- == 0) { done = true; remove = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } uint64_t offset = rng(obj_size - 1); uint64_t length = 1 + rng(obj_size - 1 - offset); - return (rng(2) != 0) ? IoOp::generate_write(offset, length) : - IoOp::generate_read(offset, length); -} + if (rng(2) != 0) { + return SingleWriteOp::generate(offset, length); + } else { + return SingleReadOp::generate(offset, length); + } +} +ceph::io_exerciser::Seq2::Seq2(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0), length(0) {} -ceph::io_exerciser::Seq2::Seq2(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(0) {} +Sequence ceph::io_exerciser::Seq2::get_id() const { + return Sequence::SEQUENCE_SEQ2; +} -std::string ceph::io_exerciser::Seq2::get_name() const -{ +std::string ceph::io_exerciser::Seq2::get_name() const { return "Permutations of offset and length read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() { length++; if (length > obj_size - offset) { length = 1; @@ -239,24 +259,23 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq2::_next() return increment_object_size(); } } - return IoOp::generate_read(offset, length); + return SingleReadOp::generate(offset, length); } - - -ceph::io_exerciser::Seq3::Seq3(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(0) -{ +ceph::io_exerciser::Seq3::Seq3(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(0) { set_min_object_size(2); } -std::string ceph::io_exerciser::Seq3::get_name() const -{ +Sequence ceph::io_exerciser::Seq3::get_id() const { + return Sequence::SEQUENCE_SEQ3; +} + +std::string ceph::io_exerciser::Seq3::get_name() const { return "Permutations of offset 2-region 1-block read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() { offset2++; if (offset2 >= obj_size - offset1) { offset2 = 1; @@ -267,24 +286,23 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq3::_next() return increment_object_size(); } } - return IoOp::generate_read2(offset1, 1, offset1 + offset2, 1); + return DoubleReadOp::generate(offset1, 1, offset1 + offset2, 1); } - - -ceph::io_exerciser::Seq4::Seq4(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(1) -{ +ceph::io_exerciser::Seq4::Seq4(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(1) { set_min_object_size(3); } -std::string ceph::io_exerciser::Seq4::get_name() const -{ +Sequence ceph::io_exerciser::Seq4::get_id() const { + return Sequence::SEQUENCE_SEQ4; +} + +std::string ceph::io_exerciser::Seq4::get_name() const { return "Permutations of offset 3-region 1-block read I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() { offset2++; if (offset2 >= obj_size - offset1) { offset2 = 2; @@ -295,33 +313,35 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq4::_next() return increment_object_size(); } } - return IoOp::generate_read3(offset1, 1, - offset1 + offset2, 1, - (offset1 * 2 + offset2)/2, 1); + return TripleReadOp::generate(offset1, 1, (offset1 + offset2), 1, + (offset1 * 2 + offset2) / 2, 1); } +ceph::io_exerciser::Seq5::Seq5(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + offset(0), + length(1), + doneread(false), + donebarrier(false) {} +Sequence ceph::io_exerciser::Seq5::get_id() const { + return Sequence::SEQUENCE_SEQ5; +} -ceph::io_exerciser::Seq5::Seq5(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(1), - doneread(false), donebarrier(false) {} - -std::string ceph::io_exerciser::Seq5::get_name() const -{ +std::string ceph::io_exerciser::Seq5::get_name() const { return "Permutation of length sequential writes"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() { if (offset >= obj_size) { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } doneread = false; donebarrier = false; @@ -333,33 +353,36 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq5::_next() } } uint64_t io_len = (offset + length > obj_size) ? (obj_size - offset) : length; - std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len); + std::unique_ptr<IoOp> r = SingleWriteOp::generate(offset, io_len); offset += io_len; return r; } +ceph::io_exerciser::Seq6::Seq6(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), + offset(0), + length(1), + doneread(false), + donebarrier(false) {} +Sequence ceph::io_exerciser::Seq6::get_id() const { + return Sequence::SEQUENCE_SEQ6; +} -ceph::io_exerciser::Seq6::Seq6(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(1), - doneread(false), donebarrier(false) {} - -std::string ceph::io_exerciser::Seq6::get_name() const -{ +std::string ceph::io_exerciser::Seq6::get_name() const { return "Permutation of length sequential writes, different alignment"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() { if (offset >= obj_size) { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } doneread = false; donebarrier = false; @@ -374,74 +397,72 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq6::_next() if (io_len == 0) { io_len = length; } - std::unique_ptr<IoOp> r = IoOp::generate_write(offset, io_len); + std::unique_ptr<IoOp> r = SingleWriteOp::generate(offset, io_len); offset += io_len; return r; } - - -ceph::io_exerciser::Seq7::Seq7(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed) -{ +ceph::io_exerciser::Seq7::Seq7(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed) { set_min_object_size(2); offset = obj_size; } -std::string ceph::io_exerciser::Seq7::get_name() const -{ +Sequence ceph::io_exerciser::Seq7::get_id() const { + return Sequence::SEQUENCE_SEQ7; +} + +std::string ceph::io_exerciser::Seq7::get_name() const { return "Permutations of offset 2-region 1-block writes"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq7::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq7::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } if (offset == 0) { doneread = false; donebarrier = false; - offset = obj_size+1; + offset = obj_size + 1; return increment_object_size(); } offset--; - if (offset == obj_size/2) { + if (offset == obj_size / 2) { return _next(); } doneread = false; donebarrier = false; - return IoOp::generate_write2(offset, 1, obj_size/2, 1); + return DoubleReadOp::generate(offset, 1, obj_size / 2, 1); } - - -ceph::io_exerciser::Seq8::Seq8(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset1(0), offset2(1) -{ +ceph::io_exerciser::Seq8::Seq8(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset1(0), offset2(1) { set_min_object_size(3); } -std::string ceph::io_exerciser::Seq8::get_name() const -{ +Sequence ceph::io_exerciser::Seq8::get_id() const { + return Sequence::SEQUENCE_SEQ8; +} + +std::string ceph::io_exerciser::Seq8::get_name() const { return "Permutations of offset 3-region 1-block write I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } offset2++; if (offset2 >= obj_size - offset1) { @@ -455,34 +476,30 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq8::_next() } doneread = false; donebarrier = false; - return IoOp::generate_write3(offset1, 1, - offset1 + offset2, 1, - (offset1 * 2 + offset2)/2, 1); + return TripleWriteOp::generate(offset1, 1, offset1 + offset2, 1, + (offset1 * 2 + offset2) / 2, 1); } +ceph::io_exerciser::Seq9::Seq9(std::pair<int, int> obj_size_range, int seed) + : IoSequence(obj_size_range, seed), offset(0), length(0) {} - -ceph::io_exerciser::Seq9::Seq9(std::pair<int,int> obj_size_range, int seed) : - IoSequence(obj_size_range, seed), offset(0), length(0) -{ - +Sequence ceph::io_exerciser::Seq9::get_id() const { + return Sequence::SEQUENCE_SEQ9; } -std::string ceph::io_exerciser::Seq9::get_name() const -{ +std::string ceph::io_exerciser::Seq9::get_name() const { return "Permutations of offset and length write I/O"; } -std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() -{ +std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() { if (!doneread) { if (!donebarrier) { donebarrier = true; - return IoOp::generate_barrier(); + return BarrierOp::generate(); } doneread = true; barrier = true; - return IoOp::generate_read(0, obj_size); + return SingleReadOp::generate(0, obj_size); } length++; if (length > obj_size - offset) { @@ -496,5 +513,5 @@ std::unique_ptr<ceph::io_exerciser::IoOp> ceph::io_exerciser::Seq9::_next() } doneread = false; donebarrier = false; - return IoOp::generate_write(offset, length); + return SingleWriteOp::generate(offset, length); }
\ No newline at end of file diff --git a/src/common/io_exerciser/IoSequence.h b/src/common/io_exerciser/IoSequence.h index 114ff76303f..b6c254cf096 100644 --- a/src/common/io_exerciser/IoSequence.h +++ b/src/common/io_exerciser/IoSequence.h @@ -3,7 +3,6 @@ #pragma once #include "IoOp.h" - #include "include/random.h" /* Overview @@ -29,195 +28,209 @@ */ namespace ceph { - namespace io_exerciser { - - enum class Sequence { - SEQUENCE_SEQ0, - SEQUENCE_SEQ1, - SEQUENCE_SEQ2, - SEQUENCE_SEQ3, - SEQUENCE_SEQ4, - SEQUENCE_SEQ5, - SEQUENCE_SEQ6, - SEQUENCE_SEQ7, - SEQUENCE_SEQ8, - SEQUENCE_SEQ9, - // - SEQUENCE_END, - SEQUENCE_BEGIN = SEQUENCE_SEQ0 - }; - - inline Sequence operator++( Sequence& s ) - { - return s = (Sequence)(((int)(s) + 1)); - } - - std::ostream& operator<<(std::ostream& os, const Sequence& seq); - - /* I/O Sequences */ - - class IoSequence { - public: - virtual ~IoSequence() = default; - - virtual std::string get_name() const = 0; - int get_step() const; - int get_seed() const; - - std::unique_ptr<IoOp> next(); - - static std::unique_ptr<IoSequence> - generate_sequence(Sequence s, std::pair<int,int> obj_size_range, int seed ); - - protected: - uint64_t min_obj_size; - uint64_t max_obj_size; - bool create; - bool barrier; - bool done; - bool remove; - uint64_t obj_size; - int step; - int seed; - ceph::util::random_number_generator<int> rng = - ceph::util::random_number_generator<int>(); - - IoSequence(std::pair<int,int> obj_size_range, int seed); - - virtual std::unique_ptr<IoOp> _next() = 0; - - void set_min_object_size(uint64_t size); - void set_max_object_size(uint64_t size); - void select_random_object_size(); - std::unique_ptr<IoOp> increment_object_size(); - - }; - - class Seq0: public IoSequence { - public: - Seq0(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - }; - - class Seq1: public IoSequence { - public: - Seq1(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next(); - - private: - int count; - }; - - class Seq2: public IoSequence { - public: - Seq2(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - }; - - class Seq3: public IoSequence { - public: - Seq3(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - private: - uint64_t offset1; - uint64_t offset2; - }; - - class Seq4: public IoSequence { - public: - Seq4(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset1; - uint64_t offset2; - }; - - class Seq5: public IoSequence { - public: - Seq5(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - bool doneread; - bool donebarrier; - }; - - class Seq6: public IoSequence { - public: - Seq6(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - uint64_t length; - bool doneread; - bool donebarrier; - }; - - class Seq7: public IoSequence { - public: - Seq7(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - - private: - uint64_t offset; - bool doneread = true; - bool donebarrier = false; - }; - - class Seq8: public IoSequence { - public: - Seq8(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - std::unique_ptr<IoOp> _next() override; - private: - uint64_t offset1; - uint64_t offset2; - bool doneread = true; - bool donebarrier = false; - }; - - class Seq9: public IoSequence { - private: - uint64_t offset; - uint64_t length; - bool doneread = true; - bool donebarrier = false; - - public: - Seq9(std::pair<int,int> obj_size_range, int seed); - - std::string get_name() const override; - - std::unique_ptr<IoOp> _next() override; - }; - } -}
\ No newline at end of file +namespace io_exerciser { + +enum class Sequence { + SEQUENCE_SEQ0, + SEQUENCE_SEQ1, + SEQUENCE_SEQ2, + SEQUENCE_SEQ3, + SEQUENCE_SEQ4, + SEQUENCE_SEQ5, + SEQUENCE_SEQ6, + SEQUENCE_SEQ7, + SEQUENCE_SEQ8, + SEQUENCE_SEQ9, + SEQUENCE_SEQ10, + + SEQUENCE_END, + SEQUENCE_BEGIN = SEQUENCE_SEQ0 +}; + +inline Sequence operator++(Sequence& s) { + return s = (Sequence)(((int)(s) + 1)); +} + +std::ostream& operator<<(std::ostream& os, const Sequence& seq); + +/* I/O Sequences */ + +class IoSequence { + public: + virtual ~IoSequence() = default; + + virtual Sequence get_id() const = 0; + virtual std::string get_name_with_seqseed() const; + virtual std::string get_name() const = 0; + int get_step() const; + int get_seed() const; + + virtual Sequence getNextSupportedSequenceId() const; + virtual std::unique_ptr<IoOp> next(); + + virtual bool is_supported(Sequence sequence) const; + static std::unique_ptr<IoSequence> generate_sequence( + Sequence s, std::pair<int, int> obj_size_range, int seed); + + protected: + uint64_t min_obj_size; + uint64_t max_obj_size; + bool create; + bool barrier; + bool done; + bool remove; + uint64_t obj_size; + int step; + int seed; + ceph::util::random_number_generator<int> rng = + ceph::util::random_number_generator<int>(); + + IoSequence(std::pair<int, int> obj_size_range, int seed); + + virtual std::unique_ptr<IoOp> _next() = 0; + + void set_min_object_size(uint64_t size); + void set_max_object_size(uint64_t size); + void select_random_object_size(); + std::unique_ptr<IoOp> increment_object_size(); +}; + +class Seq0 : public IoSequence { + public: + Seq0(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; +}; + +class Seq1 : public IoSequence { + public: + Seq1(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + int count; +}; + +class Seq2 : public IoSequence { + public: + Seq2(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; +}; + +class Seq3 : public IoSequence { + public: + Seq3(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; +}; + +class Seq4 : public IoSequence { + public: + Seq4(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; +}; + +class Seq5 : public IoSequence { + public: + Seq5(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; +}; + +class Seq6 : public IoSequence { + public: + Seq6(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + uint64_t length; + bool doneread; + bool donebarrier; +}; + +class Seq7 : public IoSequence { + public: + Seq7(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset; + bool doneread = true; + bool donebarrier = false; +}; + +class Seq8 : public IoSequence { + public: + Seq8(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; + + private: + uint64_t offset1; + uint64_t offset2; + bool doneread = true; + bool donebarrier = false; +}; + +class Seq9 : public IoSequence { + private: + uint64_t offset; + uint64_t length; + bool doneread = true; + bool donebarrier = false; + + public: + Seq9(std::pair<int, int> obj_size_range, int seed); + + Sequence get_id() const override; + std::string get_name() const override; + std::unique_ptr<IoOp> _next() override; +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/Model.cc b/src/common/io_exerciser/Model.cc index 50812ecbb15..6548e1eda7a 100644 --- a/src/common/io_exerciser/Model.cc +++ b/src/common/io_exerciser/Model.cc @@ -4,25 +4,11 @@ using Model = ceph::io_exerciser::Model; -Model::Model(const std::string& oid, uint64_t block_size) : -num_io(0), -oid(oid), -block_size(block_size) -{ +Model::Model(const std::string& oid, uint64_t block_size) + : num_io(0), oid(oid), block_size(block_size) {} -} +const uint64_t Model::get_block_size() const { return block_size; } -const uint64_t Model::get_block_size() const -{ - return block_size; -} +const std::string Model::get_oid() const { return oid; } -const std::string Model::get_oid() const -{ - return oid; -} - -int Model::get_num_io() const -{ - return num_io; -}
\ No newline at end of file +int Model::get_num_io() const { return num_io; }
\ No newline at end of file diff --git a/src/common/io_exerciser/Model.h b/src/common/io_exerciser/Model.h index 58d107409a6..9e421e79a78 100644 --- a/src/common/io_exerciser/Model.h +++ b/src/common/io_exerciser/Model.h @@ -1,15 +1,13 @@ #pragma once -#include "IoOp.h" - #include <boost/asio/io_context.hpp> -#include "librados/librados_asio.h" - -#include "include/interval_set.h" -#include "global/global_init.h" -#include "global/global_context.h" +#include "IoOp.h" #include "common/Thread.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/interval_set.h" +#include "librados/librados_asio.h" /* Overview * @@ -21,29 +19,27 @@ */ namespace ceph { - namespace io_exerciser { - - class Model - { - protected: - int num_io{0}; - std::string oid; - uint64_t block_size; - - public: - Model(const std::string& oid, uint64_t block_size); - virtual ~Model() = default; - - virtual bool readyForIoOp(IoOp& op) = 0; - virtual void applyIoOp(IoOp& op) = 0; - - const std::string get_oid() const; - const uint64_t get_block_size() const; - int get_num_io() const; - }; - - /* Simple RADOS I/O generator */ - - - } -}
\ No newline at end of file +namespace io_exerciser { + +class Model { + protected: + int num_io{0}; + std::string oid; + uint64_t block_size; + + public: + Model(const std::string& oid, uint64_t block_size); + virtual ~Model() = default; + + virtual bool readyForIoOp(IoOp& op) = 0; + virtual void applyIoOp(IoOp& op) = 0; + + const std::string get_oid() const; + const uint64_t get_block_size() const; + int get_num_io() const; +}; + +/* Simple RADOS I/O generator */ + +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/ObjectModel.cc b/src/common/io_exerciser/ObjectModel.cc index 589f6434282..454d7254cf2 100644 --- a/src/common/io_exerciser/ObjectModel.cc +++ b/src/common/io_exerciser/ObjectModel.cc @@ -6,25 +6,20 @@ using ObjectModel = ceph::io_exerciser::ObjectModel; -ObjectModel::ObjectModel(const std::string& oid, uint64_t block_size, int seed) : - Model(oid, block_size), created(false) -{ +ObjectModel::ObjectModel(const std::string& oid, uint64_t block_size, int seed) + : Model(oid, block_size), created(false) { rng.seed(seed); } -int ObjectModel::get_seed(uint64_t offset) const -{ +int ObjectModel::get_seed(uint64_t offset) const { ceph_assert(offset < contents.size()); return contents[offset]; } -std::vector<int> ObjectModel::get_seed_offsets(int seed) const -{ +std::vector<int> ObjectModel::get_seed_offsets(int seed) const { std::vector<int> offsets; - for (size_t i = 0; i < contents.size(); i++) - { - if (contents[i] == seed) - { + for (size_t i = 0; i < contents.size(); i++) { + if (contents[i] == seed) { offsets.push_back(i); } } @@ -32,8 +27,7 @@ std::vector<int> ObjectModel::get_seed_offsets(int seed) const return offsets; } -std::string ObjectModel::to_string(int mask) const -{ +std::string ObjectModel::to_string(int mask) const { if (!created) { return "Object does not exist"; } @@ -48,107 +42,127 @@ std::string ObjectModel::to_string(int mask) const return result; } -bool ObjectModel::readyForIoOp(IoOp& op) -{ - return true; -} - -void ObjectModel::applyIoOp(IoOp& op) -{ - auto generate_random = [&rng = rng]() { - return rng(); - }; - - switch (op.op) { - case OpType::BARRIER: - reads.clear(); - writes.clear(); - break; - - case OpType::CREATE: - ceph_assert(!created); - ceph_assert(reads.empty()); - ceph_assert(writes.empty()); - created = true; - contents.resize(op.length1); - std::generate(std::execution::seq, contents.begin(), contents.end(), - generate_random); - break; - - case OpType::REMOVE: - ceph_assert(created); - ceph_assert(reads.empty()); - ceph_assert(writes.empty()); - created = false; - contents.resize(0); - break; - - case OpType::READ3: - ceph_assert(created); - ceph_assert(op.offset3 + op.length3 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset3, op.length3)); - reads.union_insert(op.offset3, op.length3); - [[fallthrough]]; - - case OpType::READ2: - ceph_assert(created); - ceph_assert(op.offset2 + op.length2 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset2, op.length2)); - reads.union_insert(op.offset2, op.length2); - [[fallthrough]]; - - case OpType::READ: - ceph_assert(created); - ceph_assert(op.offset1 + op.length1 <= contents.size()); - // Not allowed: read overlapping with parallel write - ceph_assert(!writes.intersects(op.offset1, op.length1)); - reads.union_insert(op.offset1, op.length1); - num_io++; - break; - - case OpType::WRITE3: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset3, op.length3)); - ceph_assert(!writes.intersects(op.offset3, op.length3)); - writes.union_insert(op.offset3, op.length3); - ceph_assert(op.offset3 + op.length3 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset3), - std::next(contents.begin(), op.offset3 + op.length3), - generate_random); - [[fallthrough]]; - - case OpType::WRITE2: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset2, op.length2)); - ceph_assert(!writes.intersects(op.offset2, op.length2)); - writes.union_insert(op.offset2, op.length2); - ceph_assert(op.offset2 + op.length2 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset2), - std::next(contents.begin(), op.offset2 + op.length2), - generate_random); - [[fallthrough]]; - - case OpType::WRITE: - ceph_assert(created); - // Not allowed: write overlapping with parallel read or write - ceph_assert(!reads.intersects(op.offset1, op.length1)); - ceph_assert(!writes.intersects(op.offset1, op.length1)); - writes.union_insert(op.offset1, op.length1); - ceph_assert(op.offset1 + op.length1 <= contents.size()); - std::generate(std::execution::seq, - std::next(contents.begin(), op.offset1), - std::next(contents.begin(), op.offset1 + op.length1), - generate_random); - num_io++; - break; - default: - break; +bool ObjectModel::readyForIoOp(IoOp& op) { return true; } + +void ObjectModel::applyIoOp(IoOp& op) { + auto generate_random = [&rng = rng]() { return rng(); }; + + auto verify_and_record_read_op = + [&contents = contents, &created = created, &num_io = num_io, + &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N>& readOp) { + ceph_assert(created); + for (int i = 0; i < N; i++) { + ceph_assert(readOp.offset[i] + readOp.length[i] <= contents.size()); + // Not allowed: read overlapping with parallel write + ceph_assert(!writes.intersects(readOp.offset[i], readOp.length[i])); + reads.union_insert(readOp.offset[i], readOp.length[i]); + } + num_io++; + }; + + auto verify_write_and_record_and_generate_seed = + [&generate_random, &contents = contents, &created = created, + &num_io = num_io, &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N> writeOp) { + ceph_assert(created); + for (int i = 0; i < N; i++) { + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(writeOp.offset[i], writeOp.length[i])); + ceph_assert(!writes.intersects(writeOp.offset[i], writeOp.length[i])); + writes.union_insert(writeOp.offset[i], writeOp.length[i]); + ceph_assert(writeOp.offset[i] + writeOp.length[i] <= contents.size()); + std::generate(std::execution::seq, + std::next(contents.begin(), writeOp.offset[i]), + std::next(contents.begin(), + writeOp.offset[i] + writeOp.length[i]), + generate_random); + } + num_io++; + }; + + auto verify_failed_write_and_record = + [&contents = contents, &created = created, &num_io = num_io, + &reads = reads, + &writes = writes]<OpType opType, int N>(ReadWriteOp<opType, N> writeOp) { + // Ensure write should still be valid, even though we are expecting OSD + // failure + ceph_assert(created); + for (int i = 0; i < N; i++) { + // Not allowed: write overlapping with parallel read or write + ceph_assert(!reads.intersects(writeOp.offset[i], writeOp.length[i])); + ceph_assert(!writes.intersects(writeOp.offset[i], writeOp.length[i])); + writes.union_insert(writeOp.offset[i], writeOp.length[i]); + ceph_assert(writeOp.offset[i] + writeOp.length[i] <= contents.size()); + } + num_io++; + }; + + switch (op.getOpType()) { + case OpType::Barrier: + reads.clear(); + writes.clear(); + break; + + case OpType::Create: + ceph_assert(!created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = true; + contents.resize(static_cast<CreateOp&>(op).size); + std::generate(std::execution::seq, contents.begin(), contents.end(), + generate_random); + break; + + case OpType::Remove: + ceph_assert(created); + ceph_assert(reads.empty()); + ceph_assert(writes.empty()); + created = false; + contents.resize(0); + break; + + case OpType::Read: { + SingleReadOp& readOp = static_cast<SingleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + case OpType::Read2: { + DoubleReadOp& readOp = static_cast<DoubleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + case OpType::Read3: { + TripleReadOp& readOp = static_cast<TripleReadOp&>(op); + verify_and_record_read_op(readOp); + } break; + + case OpType::Write: { + ceph_assert(created); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::Write2: { + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::Write3: { + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + verify_write_and_record_and_generate_seed(writeOp); + } break; + case OpType::FailedWrite: { + ceph_assert(created); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + case OpType::FailedWrite2: { + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + case OpType::FailedWrite3: { + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + verify_failed_write_and_record(writeOp); + } break; + default: + break; } } diff --git a/src/common/io_exerciser/ObjectModel.h b/src/common/io_exerciser/ObjectModel.h index 93c70f41429..cad1307b84e 100644 --- a/src/common/io_exerciser/ObjectModel.h +++ b/src/common/io_exerciser/ObjectModel.h @@ -14,40 +14,41 @@ */ namespace ceph { - namespace io_exerciser { - /* Model of an object to track its data contents */ - - class ObjectModel : public Model { - private: - bool created; - std::vector<int> contents; - ceph::util::random_number_generator<int> rng = - ceph::util::random_number_generator<int>(); - - // Track read and write I/Os that can be submitted in - // parallel to detect violations: - // - // * Read may not overlap with a parallel write - // * Write may not overlap with a parallel read or write - // * Create / remove may not be in parallel with read or write - // - // Fix broken test cases by adding barrier ops to restrict - // I/O exercisers from issuing conflicting ops in parallel - interval_set<uint64_t> reads; - interval_set<uint64_t> writes; - public: - ObjectModel(const std::string& oid, uint64_t block_size, int seed); - - int get_seed(uint64_t offset) const; - std::vector<int> get_seed_offsets(int seed) const; - - std::string to_string(int mask = -1) const; - - bool readyForIoOp(IoOp& op); - void applyIoOp(IoOp& op); - - void encode(ceph::buffer::list& bl) const; - void decode(ceph::buffer::list::const_iterator& bl); - }; - } -}
\ No newline at end of file +namespace io_exerciser { +/* Model of an object to track its data contents */ + +class ObjectModel : public Model { + private: + bool created; + std::vector<int> contents; + ceph::util::random_number_generator<int> rng = + ceph::util::random_number_generator<int>(); + + // Track read and write I/Os that can be submitted in + // parallel to detect violations: + // + // * Read may not overlap with a parallel write + // * Write may not overlap with a parallel read or write + // * Create / remove may not be in parallel with read or write + // + // Fix broken test cases by adding barrier ops to restrict + // I/O exercisers from issuing conflicting ops in parallel + interval_set<uint64_t> reads; + interval_set<uint64_t> writes; + + public: + ObjectModel(const std::string& oid, uint64_t block_size, int seed); + + int get_seed(uint64_t offset) const; + std::vector<int> get_seed_offsets(int seed) const; + + std::string to_string(int mask = -1) const; + + bool readyForIoOp(IoOp& op); + void applyIoOp(IoOp& op); + + void encode(ceph::buffer::list& bl) const; + void decode(ceph::buffer::list::const_iterator& bl); +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/io_exerciser/OpType.h b/src/common/io_exerciser/OpType.h new file mode 100644 index 00000000000..7cddb805e45 --- /dev/null +++ b/src/common/io_exerciser/OpType.h @@ -0,0 +1,91 @@ +#pragma once + +#include <fmt/format.h> +#include <include/ceph_assert.h> + +/* Overview + * + * enum OpType + * Enumeration of different types of I/O operation + * + */ + +namespace ceph { +namespace io_exerciser { +enum class OpType { + Done, // End of I/O sequence + Barrier, // Barrier - all prior I/Os must complete + Create, // Create object and pattern with data + Remove, // Remove object + Read, // Read + Read2, // Two reads in a single op + Read3, // Three reads in a single op + Write, // Write + Write2, // Two writes in a single op + Write3, // Three writes in a single op + FailedWrite, // A write which should fail + FailedWrite2, // Two writes in one op which should fail + FailedWrite3, // Three writes in one op which should fail + InjectReadError, // Op to tell OSD to inject read errors + InjectWriteError, // Op to tell OSD to inject write errors + ClearReadErrorInject, // Op to tell OSD to clear read error injects + ClearWriteErrorInject // Op to tell OSD to clear write error injects +}; + +enum class InjectOpType { + None, + ReadEIO, + ReadMissingShard, + WriteFailAndRollback, + WriteOSDAbort +}; +} // namespace io_exerciser +} // namespace ceph + +template <> +struct fmt::formatter<ceph::io_exerciser::OpType> { + constexpr auto parse(format_parse_context& ctx) { return ctx.begin(); } + + auto format(ceph::io_exerciser::OpType opType, + fmt::format_context& ctx) const -> fmt::format_context::iterator { + switch (opType) { + case ceph::io_exerciser::OpType::Done: + return fmt::format_to(ctx.out(), "Done"); + case ceph::io_exerciser::OpType::Barrier: + return fmt::format_to(ctx.out(), "Barrier"); + case ceph::io_exerciser::OpType::Create: + return fmt::format_to(ctx.out(), "Create"); + case ceph::io_exerciser::OpType::Remove: + return fmt::format_to(ctx.out(), "Remove"); + case ceph::io_exerciser::OpType::Read: + return fmt::format_to(ctx.out(), "Read"); + case ceph::io_exerciser::OpType::Read2: + return fmt::format_to(ctx.out(), "Read2"); + case ceph::io_exerciser::OpType::Read3: + return fmt::format_to(ctx.out(), "Read3"); + case ceph::io_exerciser::OpType::Write: + return fmt::format_to(ctx.out(), "Write"); + case ceph::io_exerciser::OpType::Write2: + return fmt::format_to(ctx.out(), "Write2"); + case ceph::io_exerciser::OpType::Write3: + return fmt::format_to(ctx.out(), "Write3"); + case ceph::io_exerciser::OpType::FailedWrite: + return fmt::format_to(ctx.out(), "FailedWrite"); + case ceph::io_exerciser::OpType::FailedWrite2: + return fmt::format_to(ctx.out(), "FailedWrite2"); + case ceph::io_exerciser::OpType::FailedWrite3: + return fmt::format_to(ctx.out(), "FailedWrite3"); + case ceph::io_exerciser::OpType::InjectReadError: + return fmt::format_to(ctx.out(), "InjectReadError"); + case ceph::io_exerciser::OpType::InjectWriteError: + return fmt::format_to(ctx.out(), "InjectWriteError"); + case ceph::io_exerciser::OpType::ClearReadErrorInject: + return fmt::format_to(ctx.out(), "ClearReadErrorInject"); + case ceph::io_exerciser::OpType::ClearWriteErrorInject: + return fmt::format_to(ctx.out(), "ClearWriteErrorInject"); + default: + ceph_abort_msg("Unknown OpType"); + return fmt::format_to(ctx.out(), "Unknown OpType"); + } + } +};
\ No newline at end of file diff --git a/src/common/io_exerciser/RadosIo.cc b/src/common/io_exerciser/RadosIo.cc index 44b82260263..a78c074228b 100644 --- a/src/common/io_exerciser/RadosIo.cc +++ b/src/common/io_exerciser/RadosIo.cc @@ -1,300 +1,453 @@ #include "RadosIo.h" +#include <fmt/format.h> +#include <json_spirit/json_spirit.h> + +#include <ranges> + #include "DataGenerator.h" +#include "common/ceph_json.h" +#include "common/json/OSDStructures.h" using RadosIo = ceph::io_exerciser::RadosIo; -RadosIo::RadosIo(librados::Rados& rados, - boost::asio::io_context& asio, - const std::string& pool, - const std::string& oid, - uint64_t block_size, - int seed, - int threads, - ceph::mutex& lock, - ceph::condition_variable& cond) : - Model(oid, block_size), - rados(rados), - asio(asio), - om(std::make_unique<ObjectModel>(oid, block_size, seed)), - db(data_generation::DataGenerator::create_generator( - data_generation::GenerationType::HeaderedSeededRandom, *om)), - pool(pool), - threads(threads), - lock(lock), - cond(cond), - outstanding_io(0) -{ +namespace { +template <typename S> +int send_osd_command(int osd, S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, + Formatter* f) { + encode_json(name, s, f); + + std::ostringstream oss; + f->flush(oss); + int rc = rados.osd_command(osd, oss.str(), inbl, outbl, nullptr); + return rc; +} + +template <typename S> +int send_mon_command(S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, + Formatter* f) { + encode_json(name, s, f); + + std::ostringstream oss; + f->flush(oss); + int rc = rados.mon_command(oss.str(), inbl, outbl, nullptr); + return rc; +} +} // namespace + +RadosIo::RadosIo(librados::Rados& rados, boost::asio::io_context& asio, + const std::string& pool, const std::string& oid, + const std::optional<std::vector<int>>& cached_shard_order, + uint64_t block_size, int seed, int threads, ceph::mutex& lock, + ceph::condition_variable& cond) + : Model(oid, block_size), + rados(rados), + asio(asio), + om(std::make_unique<ObjectModel>(oid, block_size, seed)), + db(data_generation::DataGenerator::create_generator( + data_generation::GenerationType::HeaderedSeededRandom, *om)), + pool(pool), + cached_shard_order(cached_shard_order), + threads(threads), + lock(lock), + cond(cond), + outstanding_io(0) { int rc; rc = rados.ioctx_create(pool.c_str(), io); ceph_assert(rc == 0); allow_ec_overwrites(true); } -RadosIo::~RadosIo() -{ -} +RadosIo::~RadosIo() {} -void RadosIo::start_io() -{ +void RadosIo::start_io() { std::lock_guard l(lock); outstanding_io++; } -void RadosIo::finish_io() -{ +void RadosIo::finish_io() { std::lock_guard l(lock); ceph_assert(outstanding_io > 0); outstanding_io--; cond.notify_all(); } -void RadosIo::wait_for_io(int count) -{ +void RadosIo::wait_for_io(int count) { std::unique_lock l(lock); while (outstanding_io > count) { cond.wait(l); } } -void RadosIo::allow_ec_overwrites(bool allow) -{ +void RadosIo::allow_ec_overwrites(bool allow) { int rc; bufferlist inbl, outbl; - std::string cmdstr = - "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + "\", \ + std::string cmdstr = "{\"prefix\": \"osd pool set\", \"pool\": \"" + pool + + "\", \ \"var\": \"allow_ec_overwrites\", \"val\": \"" + - (allow ? "true" : "false") + "\"}"; + (allow ? "true" : "false") + "\"}"; rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr); ceph_assert(rc == 0); } -RadosIo::AsyncOpInfo::AsyncOpInfo(uint64_t offset1, uint64_t length1, - uint64_t offset2, uint64_t length2, - uint64_t offset3, uint64_t length3 ) : - offset1(offset1), length1(length1), - offset2(offset2), length2(length2), - offset3(offset3), length3(length3) -{ +template <int N> +RadosIo::AsyncOpInfo<N>::AsyncOpInfo(const std::array<uint64_t, N>& offset, + const std::array<uint64_t, N>& length) + : offset(offset), length(length) {} -} - -bool RadosIo::readyForIoOp(IoOp &op) -{ - ceph_assert(ceph_mutex_is_locked_by_me(lock)); //Must be called with lock held +bool RadosIo::readyForIoOp(IoOp& op) { + ceph_assert( + ceph_mutex_is_locked_by_me(lock)); // Must be called with lock held if (!om->readyForIoOp(op)) { return false; } - switch (op.op) { - case OpType::Done: - case OpType::BARRIER: - return outstanding_io == 0; - default: - return outstanding_io < threads; + + switch (op.getOpType()) { + case OpType::Done: + case OpType::Barrier: + return outstanding_io == 0; + default: + return outstanding_io < threads; } } -void RadosIo::applyIoOp(IoOp &op) -{ - std::shared_ptr<AsyncOpInfo> op_info; - +void RadosIo::applyIoOp(IoOp& op) { om->applyIoOp(op); // If there are thread concurrent I/Os in flight then wait for // at least one I/O to complete - wait_for_io(threads-1); - - switch (op.op) { - case OpType::Done: - [[ fallthrough ]]; - case OpType::BARRIER: - // Wait for all outstanding I/O to complete - wait_for_io(0); - break; - - case OpType::CREATE: - { + wait_for_io(threads - 1); + + switch (op.getOpType()) { + case OpType::Done: + [[fallthrough]]; + case OpType::Barrier: + // Wait for all outstanding I/O to complete + wait_for_io(0); + break; + + case OpType::Create: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(0, op.length1); - op_info->bl1 = db->generate_data(0, op.length1); - op_info->wop.write_full(op_info->bl1); - auto create_cb = [this] (boost::system::error_code ec, - version_t ver) { + uint64_t opSize = static_cast<CreateOp&>(op).size; + std::shared_ptr<AsyncOpInfo<1>> op_info = + std::make_shared<AsyncOpInfo<1>>(std::array<uint64_t, 1>{0}, + std::array<uint64_t, 1>{opSize}); + op_info->bufferlist[0] = db->generate_data(0, opSize); + op_info->wop.write_full(op_info->bufferlist[0]); + auto create_cb = [this](boost::system::error_code ec, version_t ver) { ceph_assert(ec == boost::system::errc::success); finish_io(); }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, create_cb); + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, + create_cb); + break; } - break; - case OpType::REMOVE: - { + case OpType::Remove: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(); + auto op_info = std::make_shared<AsyncOpInfo<0>>(); op_info->wop.remove(); - auto remove_cb = [this] (boost::system::error_code ec, - version_t ver) { + auto remove_cb = [this](boost::system::error_code ec, version_t ver) { ceph_assert(ec == boost::system::errc::success); finish_io(); }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, remove_cb); + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, + remove_cb); + break; } - break; + case OpType::Read: + [[fallthrough]]; + case OpType::Read2: + [[fallthrough]]; + case OpType::Read3: + [[fallthrough]]; + case OpType::Write: + [[fallthrough]]; + case OpType::Write2: + [[fallthrough]]; + case OpType::Write3: + [[fallthrough]]; + case OpType::FailedWrite: + [[fallthrough]]; + case OpType::FailedWrite2: + [[fallthrough]]; + case OpType::FailedWrite3: + applyReadWriteOp(op); + break; + case OpType::InjectReadError: + [[fallthrough]]; + case OpType::InjectWriteError: + [[fallthrough]]; + case OpType::ClearReadErrorInject: + [[fallthrough]]; + case OpType::ClearWriteErrorInject: + applyInjectOp(op); + break; + default: + ceph_abort_msg("Unrecognised Op"); + break; + } +} - case OpType::READ: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1); - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - auto read_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read_cb); - num_io++; +void RadosIo::applyReadWriteOp(IoOp& op) { + auto applyReadOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> readOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(readOp.offset, readOp.length); + + for (int i = 0; i < N; i++) { + op_info->rop.read(readOp.offset[i] * block_size, + readOp.length[i] * block_size, &op_info->bufferlist[i], + nullptr); } - break; + auto read_cb = [this, op_info](boost::system::error_code ec, version_t ver, + bufferlist bl) { + ceph_assert(ec == boost::system::errc::success); + for (int i = 0; i < N; i++) { + ceph_assert(db->validate(op_info->bufferlist[i], op_info->offset[i], + op_info->length[i])); + } + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->rop, 0, nullptr, read_cb); + num_io++; + }; - case OpType::READ2: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, - op.length1, - op.offset2, - op.length2); - - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - op_info->rop.read(op.offset2 * block_size, - op.length2 * block_size, - &op_info->bl2, nullptr); - auto read2_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - ceph_assert(db->validate(op_info->bl2, - op_info->offset2, - op_info->length2)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read2_cb); - num_io++; + auto applyWriteOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> writeOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(writeOp.offset, writeOp.length); + for (int i = 0; i < N; i++) { + op_info->bufferlist[i] = + db->generate_data(writeOp.offset[i], writeOp.length[i]); + op_info->wop.write(writeOp.offset[i] * block_size, + op_info->bufferlist[i]); } - break; + auto write_cb = [this](boost::system::error_code ec, version_t ver) { + ceph_assert(ec == boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, write_cb); + num_io++; + }; - case OpType::READ3: - { - start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2, - op.offset3, op.length3); - op_info->rop.read(op.offset1 * block_size, - op.length1 * block_size, - &op_info->bl1, nullptr); - op_info->rop.read(op.offset2 * block_size, - op.length2 * block_size, - &op_info->bl2, nullptr); - op_info->rop.read(op.offset3 * block_size, - op.length3 * block_size, - &op_info->bl3, nullptr); - auto read3_cb = [this, op_info] (boost::system::error_code ec, - version_t ver, - bufferlist bl) { - ceph_assert(ec == boost::system::errc::success); - ceph_assert(db->validate(op_info->bl1, - op_info->offset1, - op_info->length1)); - ceph_assert(db->validate(op_info->bl2, - op_info->offset2, - op_info->length2)); - ceph_assert(db->validate(op_info->bl3, - op_info->offset3, - op_info->length3)); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->rop, 0, nullptr, read3_cb); - num_io++; + auto applyFailedWriteOp = [this]<OpType opType, int N>( + ReadWriteOp<opType, N> writeOp) { + auto op_info = + std::make_shared<AsyncOpInfo<N>>(writeOp.offset, writeOp.length); + for (int i = 0; i < N; i++) { + op_info->bufferlist[i] = + db->generate_data(writeOp.offset[i], writeOp.length[i]); + op_info->wop.write(writeOp.offset[i] * block_size, + op_info->bufferlist[i]); } - break; + auto write_cb = [this, writeOp](boost::system::error_code ec, + version_t ver) { + ceph_assert(ec != boost::system::errc::success); + finish_io(); + }; + librados::async_operate(asio, io, oid, &op_info->wop, 0, nullptr, write_cb); + num_io++; + }; - case OpType::WRITE: - { + switch (op.getOpType()) { + case OpType::Read: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - auto write_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write_cb); - num_io++; + SingleReadOp& readOp = static_cast<SingleReadOp&>(op); + applyReadOp(readOp); + break; } - break; - - case OpType::WRITE2: - { + case OpType::Read2: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - op_info->bl2 = db->generate_data(op.offset2, op.length2); - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - op_info->wop.write(op.offset2 * block_size, op_info->bl2); - auto write2_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write2_cb); - num_io++; + DoubleReadOp& readOp = static_cast<DoubleReadOp&>(op); + applyReadOp(readOp); + break; + } + case OpType::Read3: { + start_io(); + TripleReadOp& readOp = static_cast<TripleReadOp&>(op); + applyReadOp(readOp); + break; + } + case OpType::Write: { + start_io(); + SingleWriteOp& writeOp = static_cast<SingleWriteOp&>(op); + applyWriteOp(writeOp); + break; + } + case OpType::Write2: { + start_io(); + DoubleWriteOp& writeOp = static_cast<DoubleWriteOp&>(op); + applyWriteOp(writeOp); + break; + } + case OpType::Write3: { + start_io(); + TripleWriteOp& writeOp = static_cast<TripleWriteOp&>(op); + applyWriteOp(writeOp); + break; } - break; - case OpType::WRITE3: - { + case OpType::FailedWrite: { start_io(); - op_info = std::make_shared<AsyncOpInfo>(op.offset1, op.length1, - op.offset2, op.length2, - op.offset3, op.length3); - op_info->bl1 = db->generate_data(op.offset1, op.length1); - op_info->bl2 = db->generate_data(op.offset2, op.length2); - op_info->bl3 = db->generate_data(op.offset3, op.length3); - op_info->wop.write(op.offset1 * block_size, op_info->bl1); - op_info->wop.write(op.offset2 * block_size, op_info->bl2); - op_info->wop.write(op.offset3 * block_size, op_info->bl3); - auto write3_cb = [this] (boost::system::error_code ec, - version_t ver) { - ceph_assert(ec == boost::system::errc::success); - finish_io(); - }; - librados::async_operate(asio, io, oid, - &op_info->wop, 0, nullptr, write3_cb); - num_io++; + SingleFailedWriteOp& writeOp = static_cast<SingleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; + } + case OpType::FailedWrite2: { + start_io(); + DoubleFailedWriteOp& writeOp = static_cast<DoubleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; + } + case OpType::FailedWrite3: { + start_io(); + TripleFailedWriteOp& writeOp = static_cast<TripleFailedWriteOp&>(op); + applyFailedWriteOp(writeOp); + break; } - break; - default: - break; + default: + ceph_abort_msg( + fmt::format("Unsupported Read/Write operation ({})", op.getOpType())); + break; } } + +void RadosIo::applyInjectOp(IoOp& op) { + bufferlist osdmap_inbl, inject_inbl, osdmap_outbl, inject_outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + int osd = -1; + std::vector<int> shard_order; + + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, get_oid(), ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", osdmap_inbl, + &osdmap_outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(osdmap_outbl.c_str(), osdmap_outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply; + reply.decode_json(&p); + + osd = reply.acting_primary; + shard_order = reply.acting; + + switch (op.getOpType()) { + case OpType::InjectReadError: { + InjectReadErrorOp& errorOp = static_cast<InjectReadErrorOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECErrorRequest<InjectOpType::ReadEIO> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 1) { + ceph::messaging::osd::InjectECErrorRequest< + InjectOpType::ReadMissingShard> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort_msg("Unsupported inject type"); + } + break; + } + case OpType::InjectWriteError: { + InjectWriteErrorOp& errorOp = static_cast<InjectWriteErrorOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECErrorRequest< + InjectOpType::WriteFailAndRollback> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 3) { + ceph::messaging::osd::InjectECErrorRequest<InjectOpType::WriteOSDAbort> + injectErrorRequest{pool, oid, errorOp.shard, + errorOp.type, errorOp.when, errorOp.duration}; + int rc = send_osd_command(osd, injectErrorRequest, rados, + "InjectECErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + + // This inject is sent directly to the shard we want to inject the error + // on + osd = shard_order[errorOp.shard]; + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + case OpType::ClearReadErrorInject: { + ClearReadErrorInjectOp& errorOp = + static_cast<ClearReadErrorInjectOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECClearErrorRequest<InjectOpType::ReadEIO> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 1) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::ReadMissingShard> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + case OpType::ClearWriteErrorInject: { + ClearReadErrorInjectOp& errorOp = + static_cast<ClearReadErrorInjectOp&>(op); + + if (errorOp.type == 0) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::WriteFailAndRollback> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else if (errorOp.type == 3) { + ceph::messaging::osd::InjectECClearErrorRequest< + InjectOpType::WriteOSDAbort> + clearErrorInject{pool, oid, errorOp.shard, errorOp.type}; + int rc = send_osd_command(osd, clearErrorInject, rados, + "InjectECClearErrorRequest", inject_inbl, + &inject_outbl, formatter.get()); + ceph_assert(rc == 0); + } else { + ceph_abort("Unsupported inject type"); + } + + break; + } + default: + ceph_abort_msg( + fmt::format("Unsupported inject operation ({})", op.getOpType())); + break; + } +}
\ No newline at end of file diff --git a/src/common/io_exerciser/RadosIo.h b/src/common/io_exerciser/RadosIo.h index 179c5bba3ae..a5c66ad4768 100644 --- a/src/common/io_exerciser/RadosIo.h +++ b/src/common/io_exerciser/RadosIo.h @@ -10,71 +10,65 @@ * in the object. Uses DataBuffer to create and validate * data buffers. When there are not barrier I/Os this may * issue multiple async I/Os in parallel. - * + * */ namespace ceph { - namespace io_exerciser { - namespace data_generation { - class DataGenerator; - } - - class RadosIo: public Model { - protected: - librados::Rados& rados; - boost::asio::io_context& asio; - std::unique_ptr<ObjectModel> om; - std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator> db; - std::string pool; - int threads; - ceph::mutex& lock; - ceph::condition_variable& cond; - librados::IoCtx io; - int outstanding_io; +namespace io_exerciser { +namespace data_generation { +class DataGenerator; +} + +class RadosIo : public Model { + protected: + librados::Rados& rados; + boost::asio::io_context& asio; + std::unique_ptr<ObjectModel> om; + std::unique_ptr<ceph::io_exerciser::data_generation::DataGenerator> db; + std::string pool; + std::optional<std::vector<int>> cached_shard_order; + int threads; + ceph::mutex& lock; + ceph::condition_variable& cond; + librados::IoCtx io; + int outstanding_io; + + void start_io(); + void finish_io(); + void wait_for_io(int count); + + public: + RadosIo(librados::Rados& rados, boost::asio::io_context& asio, + const std::string& pool, const std::string& oid, + const std::optional<std::vector<int>>& cached_shard_order, + uint64_t block_size, int seed, int threads, ceph::mutex& lock, + ceph::condition_variable& cond); - void start_io(); - void finish_io(); - void wait_for_io(int count); - - public: - RadosIo(librados::Rados& rados, - boost::asio::io_context& asio, - const std::string& pool, - const std::string& oid, - uint64_t block_size, - int seed, - int threads, - ceph::mutex& lock, - ceph::condition_variable& cond); + ~RadosIo(); - ~RadosIo(); + void allow_ec_overwrites(bool allow); - void allow_ec_overwrites(bool allow); + template <int N> + class AsyncOpInfo { + public: + librados::ObjectReadOperation rop; + librados::ObjectWriteOperation wop; + std::array<ceph::bufferlist, N> bufferlist; + std::array<uint64_t, N> offset; + std::array<uint64_t, N> length; - class AsyncOpInfo { - public: - librados::ObjectReadOperation rop; - librados::ObjectWriteOperation wop; - ceph::buffer::list bl1; - ceph::buffer::list bl2; - ceph::buffer::list bl3; - uint64_t offset1; - uint64_t length1; - uint64_t offset2; - uint64_t length2; - uint64_t offset3; - uint64_t length3; + AsyncOpInfo(const std::array<uint64_t, N>& offset = {}, + const std::array<uint64_t, N>& length = {}); + ~AsyncOpInfo() = default; + }; - AsyncOpInfo(uint64_t offset1 = 0, uint64_t length1 = 0, - uint64_t offset2 = 0, uint64_t length2 = 0, - uint64_t offset3 = 0, uint64_t length3 = 0 ); - ~AsyncOpInfo() = default; - }; + // Must be called with lock held + bool readyForIoOp(IoOp& op); + void applyIoOp(IoOp& op); - // Must be called with lock held - bool readyForIoOp(IoOp& op); - - void applyIoOp(IoOp& op); - }; - } -}
\ No newline at end of file + private: + void applyReadWriteOp(IoOp& op); + void applyInjectOp(IoOp& op); +}; +} // namespace io_exerciser +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/BalancerStructures.cc b/src/common/json/BalancerStructures.cc new file mode 100644 index 00000000000..48dfb843761 --- /dev/null +++ b/src/common/json/BalancerStructures.cc @@ -0,0 +1,38 @@ +#include "BalancerStructures.h" + +#include "common/ceph_json.h" + +using namespace ceph::messaging::balancer; + +void BalancerOffRequest::dump(Formatter* f) const { + encode_json("prefix", "balancer off", f); +} + +void BalancerOffRequest::decode_json(JSONObj* obj) {} + +void BalancerStatusRequest::dump(Formatter* f) const { + encode_json("prefix", "balancer status", f); +} + +void BalancerStatusRequest::decode_json(JSONObj* obj) {} + +void BalancerStatusReply::dump(Formatter* f) const { + encode_json("active", active, f); + encode_json("last_optimization_duration", last_optimization_duration, f); + encode_json("last_optimization_started", last_optimization_started, f); + encode_json("mode", mode, f); + encode_json("no_optimization_needed", no_optimization_needed, f); + encode_json("optimize_result", optimize_result, f); +} + +void BalancerStatusReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("active", active, obj); + JSONDecoder::decode_json("last_optimization_duration", + last_optimization_duration, obj); + JSONDecoder::decode_json("last_optimization_started", + last_optimization_started, obj); + JSONDecoder::decode_json("mode", mode, obj); + JSONDecoder::decode_json("no_optimization_needed", no_optimization_needed, + obj); + JSONDecoder::decode_json("optimize_result", optimize_result, obj); +}
\ No newline at end of file diff --git a/src/common/json/BalancerStructures.h b/src/common/json/BalancerStructures.h new file mode 100644 index 00000000000..bbf5c748eb3 --- /dev/null +++ b/src/common/json/BalancerStructures.h @@ -0,0 +1,35 @@ +#pragma once + +#include <string> + +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace balancer { +struct BalancerOffRequest { + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct BalancerStatusRequest { + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct BalancerStatusReply { + bool active; + std::string last_optimization_duration; + std::string last_optimization_started; + std::string mode; + bool no_optimization_needed; + std::string optimize_result; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; +} // namespace balancer +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/CMakeLists.txt b/src/common/json/CMakeLists.txt new file mode 100644 index 00000000000..1497daf93db --- /dev/null +++ b/src/common/json/CMakeLists.txt @@ -0,0 +1,4 @@ +add_library(json_structures STATIC + BalancerStructures.cc ConfigStructures.cc OSDStructures.cc) + + target_link_libraries(json_structures global)
\ No newline at end of file diff --git a/src/common/json/ConfigStructures.cc b/src/common/json/ConfigStructures.cc new file mode 100644 index 00000000000..651278d002a --- /dev/null +++ b/src/common/json/ConfigStructures.cc @@ -0,0 +1,20 @@ +#include "ConfigStructures.h" + +#include "common/ceph_json.h" + +using namespace ceph::messaging::config; + +void ConfigSetRequest::dump(Formatter* f) const { + encode_json("prefix", "config set", f); + encode_json("who", who, f); + encode_json("name", name, f); + encode_json("value", value, f); + encode_json("force", force, f); +} + +void ConfigSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("who", who, obj); + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("value", value, obj); + JSONDecoder::decode_json("force", force, obj); +}
\ No newline at end of file diff --git a/src/common/json/ConfigStructures.h b/src/common/json/ConfigStructures.h new file mode 100644 index 00000000000..554229d75f4 --- /dev/null +++ b/src/common/json/ConfigStructures.h @@ -0,0 +1,24 @@ +#pragma once + +#include <optional> +#include <string> + +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace config { +struct ConfigSetRequest { + std::string who; + std::string name; + std::string value; + std::optional<bool> force; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; +} // namespace config +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/json/OSDStructures.cc b/src/common/json/OSDStructures.cc new file mode 100644 index 00000000000..aaac5f6e169 --- /dev/null +++ b/src/common/json/OSDStructures.cc @@ -0,0 +1,150 @@ +#include "OSDStructures.h" + +#include "common/ceph_json.h" +#include "common/io_exerciser/OpType.h" + +using namespace ceph::messaging::osd; + +void OSDMapRequest::dump(Formatter* f) const { + encode_json("prefix", "osd map", f); + encode_json("pool", pool, f); + encode_json("object", object, f); + encode_json("nspace", nspace, f); + encode_json("format", format, f); +} + +void OSDMapRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("object", object, obj); + JSONDecoder::decode_json("nspace", nspace, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDMapReply::dump(Formatter* f) const { + encode_json("epoch", epoch, f); + encode_json("pool", pool, f); + encode_json("pool_id", pool_id, f); + encode_json("objname", objname, f); + encode_json("raw_pgid", raw_pgid, f); + encode_json("pgid", pgid, f); + encode_json("up", up, f); + encode_json("up_primary", up_primary, f); + encode_json("acting", acting, f); + encode_json("acting_primary", acting_primary, f); +} + +void OSDMapReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("epoch", epoch, obj); + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("pool_id", pool_id, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("raw_pgid", raw_pgid, obj); + JSONDecoder::decode_json("pgid", pgid, obj); + JSONDecoder::decode_json("up", up, obj); + JSONDecoder::decode_json("up_primary", up_primary, obj); + JSONDecoder::decode_json("acting", acting, obj); + JSONDecoder::decode_json("acting_primary", acting_primary, obj); +} + +void OSDPoolGetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool get", f); + encode_json("pool", pool, f); + encode_json("var", var, f); + encode_json("format", format, f); +} + +void OSDPoolGetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("var", var, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDPoolGetReply::dump(Formatter* f) const { + encode_json("erasure_code_profile", erasure_code_profile, f); +} + +void OSDPoolGetReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("erasure_code_profile", erasure_code_profile, obj); +} + +void OSDECProfileGetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool get", f); + encode_json("name", name, f); + encode_json("format", format, f); +} + +void OSDECProfileGetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("format", format, obj); +} + +void OSDECProfileGetReply::dump(Formatter* f) const { + encode_json("crush-device-class", crush_device_class, f); + encode_json("crush-failure-domain", crush_failure_domain, f); + encode_json("crush-num-failure-domains", crush_num_failure_domains, f); + encode_json("crush-osds-per-failure-domain", crush_osds_per_failure_domain, + f); + encode_json("crush-root", crush_root, f); + encode_json("jerasure-per-chunk-alignment", jerasure_per_chunk_alignment, f); + encode_json("k", k, f); + encode_json("m", m, f); + encode_json("plugin", plugin, f); + encode_json("technique", technique, f); + encode_json("w", w, f); +} + +void OSDECProfileGetReply::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("crush-device-class", crush_device_class, obj); + JSONDecoder::decode_json("crush-failure-domain", crush_failure_domain, obj); + JSONDecoder::decode_json("crush-num-failure-domains", + crush_num_failure_domains, obj); + JSONDecoder::decode_json("crush-osds-per-failure-domain", + crush_osds_per_failure_domain, obj); + JSONDecoder::decode_json("crush-root", crush_root, obj); + JSONDecoder::decode_json("jerasure-per-chunk-alignment", + jerasure_per_chunk_alignment, obj); + JSONDecoder::decode_json("k", k, obj); + JSONDecoder::decode_json("m", m, obj); + JSONDecoder::decode_json("plugin", plugin, obj); + JSONDecoder::decode_json("technique", technique, obj); + JSONDecoder::decode_json("w", w, obj); +} + +void OSDECProfileSetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd erasure-code-profile set", f); + encode_json("name", name, f); + encode_json("profile", profile, f); +} + +void OSDECProfileSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("name", name, obj); + JSONDecoder::decode_json("profile", profile, obj); +} + +void OSDECPoolCreateRequest::dump(Formatter* f) const { + encode_json("prefix", "osd pool create", f); + encode_json("pool", pool, f); + encode_json("pool_type", pool_type, f); + encode_json("pg_num", pg_num, f); + encode_json("pgp_num", pgp_num, f); + encode_json("erasure_code_profile", erasure_code_profile, f); +} + +void OSDECPoolCreateRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("pool_type", pool_type, obj); + JSONDecoder::decode_json("pg_num", pg_num, obj); + JSONDecoder::decode_json("pgp_num", pgp_num, obj); + JSONDecoder::decode_json("erasure_code_profile", erasure_code_profile, obj); +} + +void OSDSetRequest::dump(Formatter* f) const { + encode_json("prefix", "osd set", f); + encode_json("key", key, f); + encode_json("yes_i_really_mean_it", yes_i_really_mean_it, f); +} + +void OSDSetRequest::decode_json(JSONObj* obj) { + JSONDecoder::decode_json("key", key, obj); + JSONDecoder::decode_json("yes_i_really_mean_it", yes_i_really_mean_it, obj); +}
\ No newline at end of file diff --git a/src/common/json/OSDStructures.h b/src/common/json/OSDStructures.h new file mode 100644 index 00000000000..3e4528a099f --- /dev/null +++ b/src/common/json/OSDStructures.h @@ -0,0 +1,189 @@ +#pragma once + +#include <memory> +#include <string> +#include <vector> + +#include "common/ceph_json.h" +#include "common/io_exerciser/OpType.h" +#include "include/types.h" + +class JSONObj; + +namespace ceph { +namespace messaging { +namespace osd { +struct OSDMapRequest { + std::string pool; + std::string object; + std::string nspace; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDMapReply { + epoch_t epoch; + std::string pool; + uint64_t pool_id; + std::string objname; + std::string raw_pgid; + std::string pgid; + std::vector<int> up; + int up_primary; + std::vector<int> acting; + int acting_primary; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDPoolGetRequest { + std::string pool; + std::string var = "erasure_code_profile"; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDPoolGetReply { + std::string erasure_code_profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileGetRequest { + std::string name; + std::string format = "json"; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileGetReply { + std::string crush_device_class; + std::string crush_failure_domain; + int crush_num_failure_domains; + int crush_osds_per_failure_domain; + std::string crush_root; + bool jerasure_per_chunk_alignment; + int k; + int m; + std::string plugin; + std::string technique; + std::string w; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECProfileSetRequest { + std::string name; + std::vector<std::string> profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDECPoolCreateRequest { + std::string pool; + std::string pool_type; + int pg_num; + int pgp_num; + std::string erasure_code_profile; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +struct OSDSetRequest { + std::string key; + std::optional<bool> yes_i_really_mean_it = std::nullopt; + + void dump(Formatter* f) const; + void decode_json(JSONObj* obj); +}; + +// These structures are sent directly to the relevant OSD +// rather than the monitor +template <io_exerciser::InjectOpType op_type> +struct InjectECErrorRequest { + std::string pool; + std::string objname; + int shardid; + std::optional<uint64_t> type; + std::optional<uint64_t> when; + std::optional<uint64_t> duration; + + void dump(Formatter* f) const { + switch (op_type) { + case io_exerciser::InjectOpType::ReadEIO: + [[fallthrough]]; + case io_exerciser::InjectOpType::ReadMissingShard: + ::encode_json("prefix", "injectecreaderr", f); + break; + case io_exerciser::InjectOpType::WriteFailAndRollback: + [[fallthrough]]; + case io_exerciser::InjectOpType::WriteOSDAbort: + ::encode_json("prefix", "injectecwriteerr", f); + break; + default: + ceph_abort_msg("Unsupported Inject Type"); + } + ::encode_json("pool", pool, f); + ::encode_json("objname", objname, f); + ::encode_json("shardid", shardid, f); + ::encode_json("type", type, f); + ::encode_json("when", when, f); + ::encode_json("duration", duration, f); + } + void decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("shardid", shardid, obj); + JSONDecoder::decode_json("type", type, obj); + JSONDecoder::decode_json("when", when, obj); + JSONDecoder::decode_json("duration", duration, obj); + } +}; + +template <io_exerciser::InjectOpType op_type> +struct InjectECClearErrorRequest { + std::string pool; + std::string objname; + int shardid; + std::optional<uint64_t> type; + + void dump(Formatter* f) const { + switch (op_type) { + case io_exerciser::InjectOpType::ReadEIO: + [[fallthrough]]; + case io_exerciser::InjectOpType::ReadMissingShard: + ::encode_json("prefix", "injectecclearreaderr", f); + break; + case io_exerciser::InjectOpType::WriteFailAndRollback: + [[fallthrough]]; + case io_exerciser::InjectOpType::WriteOSDAbort: + ::encode_json("prefix", "injectecclearwriteerr", f); + break; + default: + ceph_abort_msg("Unsupported Inject Type"); + } + ::encode_json("pool", pool, f); + ::encode_json("objname", objname, f); + ::encode_json("shardid", shardid, f); + ::encode_json("type", type, f); + } + void decode_json(JSONObj* obj) { + JSONDecoder::decode_json("pool", pool, obj); + JSONDecoder::decode_json("objname", objname, obj); + JSONDecoder::decode_json("shardid", shardid, obj); + JSONDecoder::decode_json("type", type, obj); + } +}; +} // namespace osd +} // namespace messaging +} // namespace ceph
\ No newline at end of file diff --git a/src/common/options/crimson.yaml.in b/src/common/options/crimson.yaml.in index 69b3a615576..132a4a09e89 100644 --- a/src/common/options/crimson.yaml.in +++ b/src/common/options/crimson.yaml.in @@ -2,6 +2,17 @@ --- options: +- name: crimson_osd_objectstore + type: str + level: advanced + desc: backend type for a Crimson OSD (e.g seastore or bluestore) + default: bluestore + enum_values: + - bluestore + - seastore + - cyanstore + flags: + - create - name: crimson_osd_obc_lru_size type: uint level: advanced diff --git a/src/common/options/global.yaml.in b/src/common/options/global.yaml.in index b331601baf6..8fb77c956cf 100644 --- a/src/common/options/global.yaml.in +++ b/src/common/options/global.yaml.in @@ -5485,15 +5485,21 @@ options: - name: bluestore_slow_ops_warn_lifetime type: uint level: advanced - desc: A configurable duration for slow ops warning to be appeared if number of occurence pass `bluestore_slow_ops_warn_threshold` in `bluestore_slow_ops_warn_lifetime` seconds + desc: Set the time period during which a BlueStore slow ops warning will be raised when the `bluestore_slow_ops_warn_threshold` is exceeded. This is not the same as `osd_op_complaint_time`, which is about RADOS ops at the OSD level. default: 86400 with_legacy: true + see_also: + - bluestore_slow_ops_warn_threshold + - osd_op_complaint_time - name: bluestore_slow_ops_warn_threshold type: uint level: advanced - desc: A configurable number for slow ops warning to be appeared if number of occurence pass `bluestore_slow_ops_warn_threshold` in `bluestore_slow_ops_warn_lifetime` seconds + desc: Set the minimum number of BluesStore slow ops before raising a health warning state default: 1 with_legacy: true + see_also: + - bluestore_slow_ops_warn_lifetime + - osd_op_complaint_time - name: bluestore_fsck_error_on_no_per_pool_omap type: bool level: advanced @@ -5566,7 +5572,7 @@ options: level: dev desc: Sets threshold at which shrinking max free chunk size triggers enabling best-fit mode. - long_desc: 'AVL allocator works in two modes: near-fit and best-fit. By default, + long_desc: 'The AVL allocator works in two modes: near-fit and best-fit. By default, it uses very fast near-fit mode, in which it tries to fit a new block near the last allocated block of similar size. The second mode is much slower best-fit mode, in which it tries to find an exact match for the requested allocation. This @@ -5586,7 +5592,7 @@ options: last allocated block of similar size. The second mode is much slower best-fit mode, in which it tries to find an exact match for the requested allocation. This mode is used when either the device gets fragmented or when it is low on free - space. When free space is smaller than ''bluestore_avl_alloc_bf_free_pct'', best-fit + space. When free space is smaller than `bluestore_avl_alloc_bf_free_pct`, best-fit mode is used.' default: 4 see_also: diff --git a/src/common/options/mds.yaml.in b/src/common/options/mds.yaml.in index 94824faef6b..03a53cd7cea 100644 --- a/src/common/options/mds.yaml.in +++ b/src/common/options/mds.yaml.in @@ -1713,6 +1713,12 @@ options: default: 1000 services: - mds +- name: mds_delay_journal_replay_for_testing + type: millisecs + level: dev + desc: Delay the journal replay to verify the replay time estimate + long_desc: Jorunal replay warning is activated if the mds has been in replay state for more than 30 seconds. This config delays replay for validating the replay warning in tests. + default: 0 flags: - runtime - name: mds_server_dispatch_killpoint_random @@ -1739,4 +1745,4 @@ options: default: 16 services: - mds - min: 8 + min: 4 diff --git a/src/common/options/mon.yaml.in b/src/common/options/mon.yaml.in index ab1634bc154..1307030e3fb 100644 --- a/src/common/options/mon.yaml.in +++ b/src/common/options/mon.yaml.in @@ -91,6 +91,13 @@ options: default: 1000 services: - mon +- name: mon_nvmeofgw_delete_grace + type: secs + level: advanced + desc: Issue NVMEOF_GATEWAY_DELETING health warning after this amount of time has elapsed + default: 15_min + services: + - mon - name: mon_mgr_inactive_grace type: int level: advanced diff --git a/src/common/options/rgw.yaml.in b/src/common/options/rgw.yaml.in index 0ce5bc332fd..0bcf1656d78 100644 --- a/src/common/options/rgw.yaml.in +++ b/src/common/options/rgw.yaml.in @@ -4250,3 +4250,13 @@ options: flags: - startup with_legacy: true +- name: rgw_bucket_logging_obj_roll_time + type: uint + level: advanced + desc: Default time in seconds for the bucket logging object to roll + long_desc: Object roll time can be provided in the bucket logging configuration. + If not provided, this value will be used. + default: 300 + services: + - rgw + with_legacy: true diff --git a/src/common/pick_address.cc b/src/common/pick_address.cc index d125d7171e0..a0629a15686 100644 --- a/src/common/pick_address.cc +++ b/src/common/pick_address.cc @@ -642,17 +642,24 @@ int get_iface_numa_node( bool is_addr_in_subnet( CephContext *cct, const std::string &networks, - const std::string &addr) + const entity_addr_t &addr) { const auto nets = get_str_list(networks); ceph_assert(!nets.empty()); - unsigned ipv = CEPH_PICK_ADDRESS_IPV4; - struct sockaddr_in public_addr; - public_addr.sin_family = AF_INET; - - if(inet_pton(AF_INET, addr.c_str(), &public_addr.sin_addr) != 1) { - lderr(cct) << "unable to convert chosen address to string: " << addr << dendl; + struct sockaddr_in6 public_addr6; + struct sockaddr_in public_addr4; + + if (addr.is_ipv4() && + inet_pton(AF_INET, addr.ip_only_to_str().c_str(), &public_addr4.sin_addr) == 1) { + public_addr4.sin_family = AF_INET; + } else if (addr.is_ipv6() && + inet_pton(AF_INET6, addr.ip_only_to_str().c_str(), &public_addr6.sin6_addr) == 1) { + public_addr6.sin6_family = AF_INET6; + ipv = CEPH_PICK_ADDRESS_IPV6; + } else { + std::string_view addr_type = addr.is_ipv4() ? "IPv4" : "IPv6"; + lderr(cct) << "IP address " << addr << " is not parseable as " << addr_type << dendl; return false; } @@ -660,10 +667,16 @@ bool is_addr_in_subnet( struct ifaddrs ifa; memset(&ifa, 0, sizeof(ifa)); ifa.ifa_next = nullptr; - ifa.ifa_addr = (struct sockaddr*)&public_addr; + if (addr.is_ipv4()) { + ifa.ifa_addr = (struct sockaddr*)&public_addr4; + } else if (addr.is_ipv6()) { + ifa.ifa_addr = (struct sockaddr*)&public_addr6; + } + if(matches_with_net(cct, ifa, net, ipv)) { return true; } } + lderr(cct) << "address " << addr << " is not in networks '" << networks << "'" << dendl; return false; } diff --git a/src/common/pick_address.h b/src/common/pick_address.h index 40575d7d155..c28a6037ded 100644 --- a/src/common/pick_address.h +++ b/src/common/pick_address.h @@ -98,6 +98,6 @@ int get_iface_numa_node( bool is_addr_in_subnet( CephContext *cct, const std::string &networks, - const std::string &addr); + const entity_addr_t &addr); #endif diff --git a/src/common/sstring.hh b/src/common/sstring.hh index b0fcd9b5c47..7e0d6c6b1e2 100644 --- a/src/common/sstring.hh +++ b/src/common/sstring.hh @@ -44,6 +44,7 @@ template <typename char_type, typename Size, Size max_size> class basic_sstring { static_assert( (std::is_same<char_type, char>::value + || std::is_same<char_type, char8_t>::value || std::is_same<char_type, signed char>::value || std::is_same<char_type, unsigned char>::value), "basic_sstring only supports single byte char types"); diff --git a/src/compressor/lz4/LZ4Compressor.cc b/src/compressor/lz4/LZ4Compressor.cc index a209a5ac149..1504a2fe65d 100644 --- a/src/compressor/lz4/LZ4Compressor.cc +++ b/src/compressor/lz4/LZ4Compressor.cc @@ -121,16 +121,12 @@ int LZ4Compressor::decompress(ceph::buffer::list::const_iterator &p, LZ4_streamDecode_t lz4_stream_decode; LZ4_setStreamDecode(&lz4_stream_decode, nullptr, 0); - ceph::buffer::ptr cur_ptr = p.get_current_ptr(); - ceph::buffer::ptr *ptr = &cur_ptr; - std::optional<ceph::buffer::ptr> data_holder; - if (compressed_len != cur_ptr.length()) { - data_holder.emplace(compressed_len); - p.copy_deep(compressed_len, *data_holder); - ptr = &*data_holder; - } - - char *c_in = ptr->c_str(); + ceph::buffer::list indata; + // this does a shallow copy + p.copy(compressed_len, indata); + // if the input isn't fragmented, c_str() costs almost nothing. + // otherwise rectifying copy will be taken + const char* c_in = indata.c_str(); char *c_out = dstptr.c_str(); for (unsigned i = 0; i < count; ++i) { int r = LZ4_decompress_safe_continue( diff --git a/src/crimson/admin/osd_admin.cc b/src/crimson/admin/osd_admin.cc index de9626a2f2d..41da72c9fde 100644 --- a/src/crimson/admin/osd_admin.cc +++ b/src/crimson/admin/osd_admin.cc @@ -14,6 +14,7 @@ #include "common/config.h" #include "crimson/admin/admin_socket.h" #include "crimson/common/log.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/osd/exceptions.h" #include "crimson/osd/osd.h" #include "crimson/osd/pg.h" diff --git a/src/crimson/common/logclient.cc b/src/crimson/common/logclient.cc index d402ecd1901..a3c30227bc7 100644 --- a/src/crimson/common/logclient.cc +++ b/src/crimson/common/logclient.cc @@ -7,6 +7,7 @@ #include "crimson/net/Messenger.h" #include "crimson/mon/MonClient.h" #include "mon/MonMap.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/Graylog.h" using std::map; diff --git a/src/crimson/common/shared_lru.h b/src/crimson/common/shared_lru.h index 92d99d332c4..0d73658e709 100644 --- a/src/crimson/common/shared_lru.h +++ b/src/crimson/common/shared_lru.h @@ -25,12 +25,17 @@ class SharedLRU { SimpleLRU<K, shared_ptr_t, false> cache; std::map<K, std::pair<weak_ptr_t, V*>> weak_refs; + // Once all of the shared pointers are destoryed, + // erase the tracked object from the weak_ref map + // before actually destorying it struct Deleter { - SharedLRU<K,V>* cache; + SharedLRU<K,V>* shared_lru_ptr; const K key; - void operator()(V* ptr) { - cache->_erase_weak(key); - delete ptr; + void operator()(V* value_ptr) { + if (shared_lru_ptr) { + shared_lru_ptr->_erase_weak(key); + } + delete value_ptr; } }; void _erase_weak(const K& key) { @@ -42,9 +47,19 @@ public: {} ~SharedLRU() { cache.clear(); + // initially, we were assuming that no pointer obtained from SharedLRU // can outlive the lru itself. However, since going with the interruption // concept for handling shutdowns, this is no longer valid. + // Moreover, before clearing weak_refs, invalidate each deleter + // cache pointer as this SharedLRU is being destoryed. + for (const auto& [key, value] : weak_refs) { + shared_ptr_t val; + val = value.first.lock(); + auto this_deleter = get_deleter<Deleter>(val); + this_deleter->shared_lru_ptr = nullptr; + } + weak_refs.clear(); } /** diff --git a/src/crimson/common/tmap_helpers.cc b/src/crimson/common/tmap_helpers.cc index 9c14ebc450e..58c4fc7e218 100644 --- a/src/crimson/common/tmap_helpers.cc +++ b/src/crimson/common/tmap_helpers.cc @@ -7,6 +7,8 @@ #include "include/encoding.h" #include "include/rados.h" +#include <map> + namespace detail { #define decode_or_return(v, bp) \ diff --git a/src/crimson/mon/MonClient.cc b/src/crimson/mon/MonClient.cc index 4919f0bf21f..4c076cf43c6 100644 --- a/src/crimson/mon/MonClient.cc +++ b/src/crimson/mon/MonClient.cc @@ -13,6 +13,7 @@ #include "auth/AuthClientHandler.h" #include "auth/RotatingKeyRing.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/hostname.h" #include "include/utime_fmt.h" diff --git a/src/crimson/net/Socket.cc b/src/crimson/net/Socket.cc index 2c729f4e8c2..3a7aeaf9651 100644 --- a/src/crimson/net/Socket.cc +++ b/src/crimson/net/Socket.cc @@ -8,6 +8,7 @@ #include <seastar/net/packet.hh> #include "crimson/common/log.h" +#include "include/random.h" // for ceph::util::generate_random_number() #include "Errors.h" using crimson::common::local_conf; diff --git a/src/crimson/net/io_handler.cc b/src/crimson/net/io_handler.cc index b93124f3c12..bc5e9bf404c 100644 --- a/src/crimson/net/io_handler.cc +++ b/src/crimson/net/io_handler.cc @@ -347,7 +347,7 @@ void IOHandler::do_set_io_state( { ceph_assert_always(seastar::this_shard_id() == get_shard_id()); auto prv_state = get_io_state(); - logger().debug("{} got {}do_set_io_state(): prv_state={}, new_state={}, " + logger().debug("{} got {} do_set_io_state(): prv_state={}, new_state={}, " "fa={}, set_notify_out={}, at {}", conn, cc_seq.has_value() ? fmt::format("{} ", *cc_seq) : "", @@ -984,7 +984,7 @@ void IOHandler::notify_out_dispatch() }); }); } - if (shard_states->try_enter_out_dispatching()) { + if (shard_states->try_enter_out_dispatching(conn)) { shard_states->dispatch_in_background( "do_out_dispatch", conn, [this] { return do_out_dispatch(*shard_states); diff --git a/src/crimson/net/io_handler.h b/src/crimson/net/io_handler.h index 5986fcb16ac..41c76ab925b 100644 --- a/src/crimson/net/io_handler.h +++ b/src/crimson/net/io_handler.h @@ -309,7 +309,7 @@ public: in_exit_dispatching = std::nullopt; } - bool try_enter_out_dispatching() { + bool try_enter_out_dispatching(SocketConnection &conn) { assert(seastar::this_shard_id() == sid); if (out_dispatching) { // already dispatching out @@ -327,6 +327,9 @@ public: // do not dispatch out return false; default: + crimson::get_logger(ceph_subsys_ms).error( + "{} try_enter_out_dispatching() got wrong io_state {}", + conn, io_state); ceph_abort("impossible"); } } @@ -574,6 +577,8 @@ struct fmt::formatter<crimson::net::IOHandler::io_state_t> case switched: name = "switched"; break; + default: + name = "undefined"; } return formatter<string_view>::format(name, ctx); } diff --git a/src/crimson/os/alienstore/alien_store.cc b/src/crimson/os/alienstore/alien_store.cc index f390823a8a0..db6decd84f9 100644 --- a/src/crimson/os/alienstore/alien_store.cc +++ b/src/crimson/os/alienstore/alien_store.cc @@ -141,7 +141,8 @@ seastar::future<> AlienStore::stop() AlienStore::base_errorator::future<bool> AlienStore::exists( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { return op_gates.simple_dispatch("exists", [=, this] { return tp->submit(ch->get_cid().hash_to_shard(tp->size()), [=, this] { @@ -212,7 +213,8 @@ seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> AlienStore::list_objects(CollectionRef ch, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const + uint64_t limit, + uint32_t op_flags) const { logger().debug("{}", __func__); assert(tp); @@ -348,7 +350,8 @@ AlienStore::readv(CollectionRef ch, AlienStore::get_attr_errorator::future<ceph::bufferlist> AlienStore::get_attr(CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { logger().debug("{}", __func__); assert(tp); @@ -376,7 +379,8 @@ AlienStore::get_attr(CollectionRef ch, AlienStore::get_attrs_ertr::future<AlienStore::attrs_t> AlienStore::get_attrs(CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { logger().debug("{}", __func__); assert(tp); @@ -397,7 +401,8 @@ AlienStore::get_attrs(CollectionRef ch, auto AlienStore::omap_get_values(CollectionRef ch, const ghobject_t& oid, - const set<string>& keys) + const set<string>& keys, + uint32_t op_flags) -> read_errorator::future<omap_values_t> { logger().debug("{}", __func__); @@ -421,7 +426,8 @@ auto AlienStore::omap_get_values(CollectionRef ch, auto AlienStore::omap_get_values(CollectionRef ch, const ghobject_t &oid, - const std::optional<string> &start) + const std::optional<string> &start, + uint32_t op_flags) -> read_errorator::future<std::tuple<bool, omap_values_t>> { logger().debug("{} with_start", __func__); @@ -429,8 +435,21 @@ auto AlienStore::omap_get_values(CollectionRef ch, return do_with_op_gate(omap_values_t{}, [=, this] (auto &values) { return tp->submit(ch->get_cid().hash_to_shard(tp->size()), [=, this, &values] { auto c = static_cast<AlienCollection*>(ch.get()); - return store->omap_get_values(c->collection, oid, start, - reinterpret_cast<map<string, bufferlist>*>(&values)); + return store->omap_iterate( + c->collection, oid, + ObjectStore::omap_iter_seek_t{ + .seek_position = start.value_or(std::string{}), + // FIXME: classical OSDs begins iteration from LOWER_BOUND + // (or UPPER_BOUND if filter_prefix > start). However, these + // bits are not implemented yet + .seek_type = ObjectStore::omap_iter_seek_t::UPPER_BOUND + }, + [&values] + (std::string_view key, std::string_view value) mutable { + values[std::string{key}].append(value); + // FIXME: there is limit on number of entries yet + return ObjectStore::omap_iter_ret_t::NEXT; + }); }).then([&values] (int r) -> read_errorator::future<std::tuple<bool, omap_values_t>> { if (r == -ENOENT) { @@ -578,7 +597,8 @@ unsigned AlienStore::get_max_attr_name_length() const seastar::future<struct stat> AlienStore::stat( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { assert(tp); return do_with_op_gate((struct stat){}, [this, ch, oid](auto& st) { @@ -604,7 +624,8 @@ seastar::future<std::string> AlienStore::get_default_device_class() } auto AlienStore::omap_get_header(CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) -> get_attr_errorator::future<ceph::bufferlist> { assert(tp); @@ -630,7 +651,8 @@ AlienStore::read_errorator::future<std::map<uint64_t, uint64_t>> AlienStore::fie CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { assert(tp); return do_with_op_gate(std::map<uint64_t, uint64_t>(), [=, this](auto& destmap) { diff --git a/src/crimson/os/alienstore/alien_store.h b/src/crimson/os/alienstore/alien_store.h index 853585dac9c..1d39411450e 100644 --- a/src/crimson/os/alienstore/alien_store.h +++ b/src/crimson/os/alienstore/alien_store.h @@ -36,7 +36,8 @@ public: base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; mkfs_ertr::future<> mkfs(uuid_d new_osd_fsid) final; read_errorator::future<ceph::bufferlist> read(CollectionRef c, const ghobject_t& oid, @@ -49,29 +50,36 @@ public: uint32_t op_flags = 0) final; - get_attr_errorator::future<ceph::bufferlist> get_attr(CollectionRef c, - const ghobject_t& oid, - std::string_view name) const final; - get_attrs_ertr::future<attrs_t> get_attrs(CollectionRef c, - const ghobject_t& oid) final; + get_attr_errorator::future<ceph::bufferlist> get_attr( + CollectionRef c, + const ghobject_t& oid, + std::string_view name, + uint32_t op_flags = 0) const final; + get_attrs_ertr::future<attrs_t> get_attrs( + CollectionRef c, + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; /// Retrieves paged set of values > start (if present) read_errorator::future<std::tuple<bool, omap_values_t>> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; ///< @return <done, values> values.empty() iff done seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; seastar::future<CollectionRef> open_collection(const coll_t& cid) final; @@ -97,16 +105,19 @@ public: unsigned get_max_attr_name_length() const final; seastar::future<struct stat> stat( CollectionRef, - const ghobject_t&) final; + const ghobject_t&, + uint32_t op_flags = 0) final; seastar::future<std::string> get_default_device_class() final; get_attr_errorator::future<ceph::bufferlist> omap_get_header( CollectionRef, - const ghobject_t&) final; + const ghobject_t&, + uint32_t) final; read_errorator::future<std::map<uint64_t, uint64_t>> fiemap( CollectionRef, const ghobject_t&, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags) final; FuturizedStore::Shard& get_sharded_store() final { return *this; diff --git a/src/crimson/os/alienstore/thread_pool.cc b/src/crimson/os/alienstore/thread_pool.cc index 277055ec51e..2d208548b32 100644 --- a/src/crimson/os/alienstore/thread_pool.cc +++ b/src/crimson/os/alienstore/thread_pool.cc @@ -7,6 +7,7 @@ #include <pthread.h> #include "include/ceph_assert.h" +#include "include/intarith.h" // for round_up_to() #include "crimson/common/config_proxy.h" using crimson::common::local_conf; diff --git a/src/crimson/os/cyanstore/cyan_store.cc b/src/crimson/os/cyanstore/cyan_store.cc index 3f861a9271f..41819fb5eb6 100644 --- a/src/crimson/os/cyanstore/cyan_store.cc +++ b/src/crimson/os/cyanstore/cyan_store.cc @@ -12,6 +12,7 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" +#include "crimson/common/perf_counters_collection.h" #include "cyan_collection.h" #include "cyan_object.h" @@ -207,7 +208,8 @@ CyanStore::Shard::list_objects( CollectionRef ch, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const + uint64_t limit, + uint32_t op_flags) const { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {} {} {}", @@ -256,7 +258,8 @@ CyanStore::Shard::list_collections() CyanStore::Shard::base_errorator::future<bool> CyanStore::Shard::exists( CollectionRef ch, - const ghobject_t &oid) + const ghobject_t &oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); if (!c->exists) { @@ -332,7 +335,8 @@ CyanStore::Shard::get_attr_errorator::future<ceph::bufferlist> CyanStore::Shard::get_attr( CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {}", @@ -351,7 +355,8 @@ CyanStore::Shard::get_attr( CyanStore::Shard::get_attrs_ertr::future<CyanStore::Shard::attrs_t> CyanStore::Shard::get_attrs( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); logger().debug("{} {} {}", @@ -366,7 +371,8 @@ CyanStore::Shard::get_attrs( auto CyanStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t& oid, - const omap_keys_t& keys) + const omap_keys_t& keys, + uint32_t op_flags) -> read_errorator::future<omap_values_t> { auto c = static_cast<Collection*>(ch.get()); @@ -387,7 +393,8 @@ auto CyanStore::Shard::omap_get_values( auto CyanStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const std::optional<string> &start) + const std::optional<string> &start, + uint32_t op_flags) -> CyanStore::Shard::read_errorator::future<std::tuple<bool, omap_values_t>> { auto c = static_cast<Collection*>(ch.get()); @@ -408,7 +415,8 @@ auto CyanStore::Shard::omap_get_values( auto CyanStore::Shard::omap_get_header( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) -> CyanStore::Shard::get_attr_errorator::future<ceph::bufferlist> { auto c = static_cast<Collection*>(ch.get()); @@ -976,7 +984,8 @@ CyanStore::Shard::fiemap( CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); @@ -991,7 +1000,8 @@ CyanStore::Shard::fiemap( seastar::future<struct stat> CyanStore::Shard::stat( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { auto c = static_cast<Collection*>(ch.get()); auto o = c->get_object(oid); diff --git a/src/crimson/os/cyanstore/cyan_store.h b/src/crimson/os/cyanstore/cyan_store.h index e9394991bc2..1d481ef5829 100644 --- a/src/crimson/os/cyanstore/cyan_store.h +++ b/src/crimson/os/cyanstore/cyan_store.h @@ -34,11 +34,13 @@ public: seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; base_errorator::future<bool> exists( CollectionRef ch, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<ceph::bufferlist> read( CollectionRef c, @@ -56,33 +58,39 @@ public: get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const final; + std::string_view name, + uint32_t op_flags = 0) const final; get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; read_errorator::future<std::tuple<bool, omap_values_t>> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; get_attr_errorator::future<ceph::bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; @@ -101,7 +109,8 @@ public: CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags) final; unsigned get_max_attr_name_length() const final; diff --git a/src/crimson/os/futurized_store.h b/src/crimson/os/futurized_store.h index 51ef2331014..e7d4c8546de 100644 --- a/src/crimson/os/futurized_store.h +++ b/src/crimson/os/futurized_store.h @@ -54,7 +54,8 @@ public: virtual base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; using get_attr_errorator = crimson::errorator< crimson::ct_error::enoent, @@ -62,42 +63,49 @@ public: virtual get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const = 0; + std::string_view name, + uint32_t op_flags = 0) const = 0; using get_attrs_ertr = crimson::errorator< crimson::ct_error::enoent>; using attrs_t = std::map<std::string, ceph::bufferlist, std::less<>>; virtual get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; virtual seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; using omap_values_t = attrs_t; using omap_keys_t = std::set<std::string>; virtual read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) = 0; + const omap_keys_t& keys, + uint32_t op_flags = 0) = 0; using omap_values_paged_t = std::tuple<bool, omap_values_t>; virtual read_errorator::future<omap_values_paged_t> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) = 0; ///< @return <done, values> values.empty() only if done virtual get_attr_errorator::future<bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) = 0; + const ghobject_t& oid, + uint32_t op_flags = 0) = 0; virtual seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> list_objects( CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const = 0; + uint64_t limit, + uint32_t op_flags = 0) const = 0; virtual seastar::future<CollectionRef> create_new_collection(const coll_t& cid) = 0; @@ -153,7 +161,8 @@ public: CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) = 0; + uint64_t len, + uint32_t op_flags = 0) = 0; virtual unsigned get_max_attr_name_length() const = 0; }; diff --git a/src/crimson/os/seastore/CMakeLists.txt b/src/crimson/os/seastore/CMakeLists.txt index e5b8960c38c..3da5e65ceec 100644 --- a/src/crimson/os/seastore/CMakeLists.txt +++ b/src/crimson/os/seastore/CMakeLists.txt @@ -1,5 +1,6 @@ set(crimson_seastore_srcs cached_extent.cc + lba_mapping.cc seastore_types.cc segment_manager.cc segment_manager/ephemeral.cc @@ -19,7 +20,6 @@ set(crimson_seastore_srcs omap_manager.cc omap_manager/btree/btree_omap_manager.cc omap_manager/btree/omap_btree_node_impl.cc - btree/btree_range_pin.cc btree/fixed_kv_node.cc onode.cc onode_manager/staged-fltree/node.cc diff --git a/src/crimson/os/seastore/async_cleaner.cc b/src/crimson/os/seastore/async_cleaner.cc index 5046980eae5..64e6749562e 100644 --- a/src/crimson/os/seastore/async_cleaner.cc +++ b/src/crimson/os/seastore/async_cleaner.cc @@ -131,7 +131,7 @@ void segments_info_t::add_segment_manager( auto ssize = segment_manager.get_segment_size(); auto nsegments = segment_manager.get_num_segments(); auto sm_size = segment_manager.get_available_size(); - INFO("adding segment manager {}, size={}, ssize={}, segments={}", + INFO("adding segment manager {}, size=0x{:x}, segment size=0x{:x}, segments={}", device_id_printer_t{d_id}, sm_size, ssize, nsegments); ceph_assert(ssize > 0); ceph_assert(nsegments > 0); @@ -329,9 +329,9 @@ std::ostream &operator<<(std::ostream &os, const segments_info_t &infos) << ", closed=" << infos.get_num_closed() << ", type_journal=" << infos.get_num_type_journal() << ", type_ool=" << infos.get_num_type_ool() - << ", total=" << infos.get_total_bytes() << "B" - << ", available=" << infos.get_available_bytes() << "B" - << ", unavailable=" << infos.get_unavailable_bytes() << "B" + << ", total=0x" << std::hex << infos.get_total_bytes() << "B" + << ", available=0x" << infos.get_available_bytes() << "B" + << ", unavailable=0x" << infos.get_unavailable_bytes() << "B" << std::dec << ", available_ratio=" << infos.get_available_ratio() << ", submitted_head=" << infos.get_submitted_journal_head() << ", time_bound=" << sea_time_point_printer_t{infos.get_time_bound()} @@ -609,6 +609,7 @@ JournalTrimmerImpl::trim_alloc() return extent_callback->with_transaction_intr( Transaction::src_t::TRIM_ALLOC, "trim_alloc", + CACHE_HINT_NOCACHE, [this, FNAME](auto &t) { auto target = get_alloc_tail_target(); @@ -653,6 +654,7 @@ JournalTrimmerImpl::trim_dirty() return extent_callback->with_transaction_intr( Transaction::src_t::TRIM_DIRTY, "trim_dirty", + CACHE_HINT_NOCACHE, [this, FNAME](auto &t) { auto target = get_dirty_tail_target(); @@ -765,10 +767,10 @@ int64_t SpaceTrackerDetailed::SegmentMap::allocate( for (auto i = b; i < e; ++i) { if (bitmap[i]) { if (!error) { - ERROR("found allocated in {}, {} ~ {}", segment, offset, len); + ERROR("found allocated in {}, 0x{:x}~0x{:x}", segment, offset, len); error = true; } - DEBUG("block {} allocated", i * block_size); + DEBUG("block 0x{:x}B allocated", i * block_size); } bitmap[i] = true; } @@ -792,10 +794,10 @@ int64_t SpaceTrackerDetailed::SegmentMap::release( for (auto i = b; i < e; ++i) { if (!bitmap[i]) { if (!error) { - ERROR("found unallocated in {}, {} ~ {}", segment, offset, len); + ERROR("found unallocated in {}, 0x{:x}~0x{:x}", segment, offset, len); error = true; } - DEBUG("block {} unallocated", i * block_size); + DEBUG("block 0x{:x}B unallocated", i * block_size); } bitmap[i] = false; } @@ -831,7 +833,7 @@ void SpaceTrackerDetailed::SegmentMap::dump_usage(extent_len_t block_size) const INFO("dump start"); for (unsigned i = 0; i < bitmap.size(); ++i) { if (bitmap[i]) { - LOCAL_LOGGER.info(" {} still live", i * block_size); + LOCAL_LOGGER.info(" 0x{:x}B still live", i * block_size); } } } @@ -847,7 +849,7 @@ void SpaceTrackerDetailed::dump_usage(segment_id_t id) const void SpaceTrackerSimple::dump_usage(segment_id_t id) const { LOG_PREFIX(SpaceTrackerSimple::dump_usage); - INFO("id: {}, live_bytes: {}", + INFO("id: {}, live_bytes: 0x{:x}", id, live_bytes_by_segment[id].live_bytes); } @@ -1125,6 +1127,7 @@ SegmentCleaner::do_reclaim_space( return extent_callback->with_transaction_intr( src, "clean_reclaim_space", + CACHE_HINT_NOCACHE, [this, &backref_extents, &pin_list, &reclaimed](auto &t) { return seastar::do_with( @@ -1142,8 +1145,7 @@ SegmentCleaner::do_reclaim_space( pin->get_key(), pin->get_val(), pin->get_length(), - pin->get_type(), - JOURNAL_SEQ_NULL); + pin->get_type()); } for (auto &cached_backref : cached_backref_entries) { if (cached_backref.laddr == L_ADDR_NULL) { @@ -1165,7 +1167,7 @@ SegmentCleaner::do_reclaim_space( [this, &extents, &t](auto &ent) { LOG_PREFIX(SegmentCleaner::do_reclaim_space); - TRACET("getting extent of type {} at {}~{}", + TRACET("getting extent of type {} at {}~0x{:x}", t, ent.type, ent.paddr, @@ -1241,6 +1243,7 @@ SegmentCleaner::clean_space_ret SegmentCleaner::clean_space() return extent_callback->with_transaction_intr( Transaction::src_t::READ, "retrieve_from_backref_tree", + CACHE_HINT_NOCACHE, [this, &weak_read_ret](auto &t) { return backref_manager.get_mappings( t, @@ -1507,6 +1510,7 @@ bool SegmentCleaner::check_usage() SpaceTrackerIRef tracker(space_tracker->make_empty()); extent_callback->with_transaction_weak( "check_usage", + CACHE_HINT_NOCACHE, [this, &tracker](auto &t) { return backref_manager.scan_mapped_space( t, @@ -1568,7 +1572,7 @@ void SegmentCleaner::mark_space_used( background_callback->maybe_wake_background(); assert(ret > 0); - DEBUG("segment {} new len: {}~{}, live_bytes: {}", + DEBUG("segment {} new len: {}~0x{:x}, live_bytes: 0x{:x}", seg_addr.get_segment_id(), addr, len, @@ -1591,7 +1595,7 @@ void SegmentCleaner::mark_space_free( stats.used_bytes -= len; auto& seg_addr = addr.as_seg_paddr(); - DEBUG("segment {} free len: {}~{}", + DEBUG("segment {} free len: {}~0x{:x}", seg_addr.get_segment_id(), addr, len); auto old_usage = calc_utilization(seg_addr.get_segment_id()); [[maybe_unused]] auto ret = space_tracker->release( @@ -1602,7 +1606,7 @@ void SegmentCleaner::mark_space_free( adjust_segment_util(old_usage, new_usage); background_callback->maybe_wake_blocked_io(); assert(ret >= 0); - DEBUG("segment {} free len: {}~{}, live_bytes: {}", + DEBUG("segment {} free len: {}~0x{:x}, live_bytes: 0x{:x}", seg_addr.get_segment_id(), addr, len, @@ -1687,11 +1691,11 @@ void SegmentCleaner::print(std::ostream &os, bool is_detailed) const << ", reclaim_ratio=" << get_reclaim_ratio() << ", alive_ratio=" << get_alive_ratio(); if (is_detailed) { - os << ", unavailable_unreclaimable=" + os << ", unavailable_unreclaimable=0x" << std::hex << get_unavailable_unreclaimable_bytes() << "B" - << ", unavailable_reclaimble=" + << ", unavailable_reclaimble=0x" << get_unavailable_reclaimable_bytes() << "B" - << ", alive=" << stats.used_bytes << "B" + << ", alive=0x" << stats.used_bytes << "B" << std::dec << ", " << segments; } os << ")"; @@ -1722,7 +1726,7 @@ void RBMCleaner::mark_space_used( for (auto rbm : rbms) { if (addr.get_device_id() == rbm->get_device_id()) { if (rbm->get_start() <= addr) { - DEBUG("allocate addr: {} len: {}", addr, len); + DEBUG("allocate addr: {} len: 0x{:x}", addr, len); stats.used_bytes += len; rbm->mark_space_used(addr, len); } @@ -1741,7 +1745,7 @@ void RBMCleaner::mark_space_free( for (auto rbm : rbms) { if (addr.get_device_id() == rbm->get_device_id()) { if (rbm->get_start() <= addr) { - DEBUG("free addr: {} len: {}", addr, len); + DEBUG("free addr: {} len: 0x{:x}", addr, len); ceph_assert(stats.used_bytes >= len); stats.used_bytes -= len; rbm->mark_space_free(addr, len); @@ -1813,6 +1817,7 @@ bool RBMCleaner::check_usage() RBMSpaceTracker tracker(rbms); extent_callback->with_transaction_weak( "check_usage", + CACHE_HINT_NOCACHE, [this, &tracker, &rbms](auto &t) { return backref_manager.scan_mapped_space( t, diff --git a/src/crimson/os/seastore/async_cleaner.h b/src/crimson/os/seastore/async_cleaner.h index 424247c5bdc..1cef771aeb8 100644 --- a/src/crimson/os/seastore/async_cleaner.h +++ b/src/crimson/os/seastore/async_cleaner.h @@ -17,6 +17,7 @@ #include "crimson/os/seastore/randomblock_manager_group.h" #include "crimson/os/seastore/transaction.h" #include "crimson/os/seastore/segment_seq_allocator.h" +#include "crimson/os/seastore/backref_mapping.h" namespace crimson::os::seastore { @@ -299,24 +300,29 @@ public: /// Creates empty transaction /// weak transaction should be type READ virtual TransactionRef create_transaction( - Transaction::src_t, const char *name, bool is_weak=false) = 0; + Transaction::src_t, + const char *name, + cache_hint_t cache_hint = CACHE_HINT_TOUCH, + bool is_weak=false) = 0; /// Creates empty transaction with interruptible context template <typename Func> auto with_transaction_intr( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, Func &&f) { return do_with_transaction_intr<Func, false>( - src, name, std::forward<Func>(f)); + src, name, cache_hint, std::forward<Func>(f)); } template <typename Func> auto with_transaction_weak( const char* name, + cache_hint_t cache_hint, Func &&f) { return do_with_transaction_intr<Func, true>( - Transaction::src_t::READ, name, std::forward<Func>(f) + Transaction::src_t::READ, name, cache_hint, std::forward<Func>(f) ).handle_error( crimson::ct_error::eagain::assert_failure{"unexpected eagain"}, crimson::ct_error::pass_further_all{} @@ -385,9 +391,10 @@ private: auto do_with_transaction_intr( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, Func &&f) { return seastar::do_with( - create_transaction(src, name, IsWeak), + create_transaction(src, name, cache_hint, IsWeak), [f=std::forward<Func>(f)](auto &ref_t) mutable { return with_trans_intr( *ref_t, diff --git a/src/crimson/os/seastore/backref/btree_backref_manager.h b/src/crimson/os/seastore/backref/btree_backref_manager.h index 38084bb00e6..24897dd55da 100644 --- a/src/crimson/os/seastore/backref/btree_backref_manager.h +++ b/src/crimson/os/seastore/backref/btree_backref_manager.h @@ -9,44 +9,28 @@ namespace crimson::os::seastore::backref { -constexpr size_t BACKREF_BLOCK_SIZE = 4096; - -class BtreeBackrefMapping : public BtreeNodeMapping<paddr_t, laddr_t> { - extent_types_t type; +class BtreeBackrefMapping : public BackrefMapping { public: BtreeBackrefMapping(op_context_t<paddr_t> ctx) - : BtreeNodeMapping(ctx) {} + : BackrefMapping(ctx) {} BtreeBackrefMapping( op_context_t<paddr_t> ctx, CachedExtentRef parent, uint16_t pos, backref_map_val_t &val, backref_node_meta_t &&meta) - : BtreeNodeMapping( + : BackrefMapping( + val.type, ctx, parent, pos, val.laddr, val.len, - std::forward<backref_node_meta_t>(meta)), - type(val.type) - {} - extent_types_t get_type() const final { - return type; - } - - bool is_clone() const final { - return false; - } - -protected: - std::unique_ptr<BtreeNodeMapping<paddr_t, laddr_t>> _duplicate( - op_context_t<paddr_t> ctx) const final { - return std::unique_ptr<BtreeNodeMapping<paddr_t, laddr_t>>( - new BtreeBackrefMapping(ctx)); - } + std::forward<backref_node_meta_t>(meta)) {} }; +constexpr size_t BACKREF_BLOCK_SIZE = 4096; + using BackrefBtree = FixedKVBtree< paddr_t, backref_map_val_t, BackrefInternalNode, BackrefLeafNode, BtreeBackrefMapping, BACKREF_BLOCK_SIZE, false>; diff --git a/src/crimson/os/seastore/backref_entry.h b/src/crimson/os/seastore/backref_entry.h new file mode 100644 index 00000000000..5f9becc9565 --- /dev/null +++ b/src/crimson/os/seastore/backref_entry.h @@ -0,0 +1,127 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include <memory> +#include <iostream> + +#if FMT_VERSION >= 90000 +#include <fmt/ostream.h> +#endif + +#include <boost/intrusive/set.hpp> + +#include "crimson/os/seastore/seastore_types.h" + +namespace crimson::os::seastore { + +struct backref_entry_t { + using ref_t = std::unique_ptr<backref_entry_t>; + + backref_entry_t( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) + : paddr(paddr), + laddr(laddr), + len(len), + type(type) { + assert(len > 0); + } + paddr_t paddr = P_ADDR_NULL; + laddr_t laddr = L_ADDR_NULL; + extent_len_t len = 0; + extent_types_t type = extent_types_t::NONE; + friend bool operator< ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr < r.paddr; + } + friend bool operator> ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr > r.paddr; + } + friend bool operator== ( + const backref_entry_t &l, + const backref_entry_t &r) { + return l.paddr == r.paddr; + } + + using set_hook_t = + boost::intrusive::set_member_hook< + boost::intrusive::link_mode< + boost::intrusive::auto_unlink>>; + set_hook_t backref_set_hook; + using backref_set_member_options = boost::intrusive::member_hook< + backref_entry_t, + set_hook_t, + &backref_entry_t::backref_set_hook>; + using multiset_t = boost::intrusive::multiset< + backref_entry_t, + backref_set_member_options, + boost::intrusive::constant_time_size<false>>; + + struct cmp_t { + using is_transparent = paddr_t; + bool operator()( + const backref_entry_t &l, + const backref_entry_t &r) const { + return l.paddr < r.paddr; + } + bool operator()(const paddr_t l, const backref_entry_t &r) const { + return l < r.paddr; + } + bool operator()(const backref_entry_t &l, const paddr_t r) const { + return l.paddr < r; + } + }; + + static ref_t create_alloc( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type)); + assert(laddr != L_ADDR_NULL); + return std::make_unique<backref_entry_t>( + paddr, laddr, len, type); + } + + static ref_t create_retire( + const paddr_t& paddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type) || + is_retired_placeholder_type(type)); + return std::make_unique<backref_entry_t>( + paddr, L_ADDR_NULL, len, type); + } + + static ref_t create(const alloc_blk_t& delta) { + return std::make_unique<backref_entry_t>( + delta.paddr, delta.laddr, delta.len, delta.type); + } +}; + +inline std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent) { + return out << "backref_entry_t{" + << ent.paddr << "~0x" << std::hex << ent.len << std::dec << ", " + << "laddr: " << ent.laddr << ", " + << "type: " << ent.type + << "}"; +} + +using backref_entry_ref = backref_entry_t::ref_t; +using backref_entry_mset_t = backref_entry_t::multiset_t; +using backref_entry_refs_t = std::vector<backref_entry_ref>; +using backref_entryrefs_by_seq_t = std::map<journal_seq_t, backref_entry_refs_t>; +using backref_entry_query_set_t = std::set<backref_entry_t, backref_entry_t::cmp_t>; + +} // namespace crimson::os::seastore + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::os::seastore::backref_entry_t> : fmt::ostream_formatter {}; +#endif diff --git a/src/crimson/os/seastore/backref_manager.h b/src/crimson/os/seastore/backref_manager.h index 3feedb997b4..8c746b571b2 100644 --- a/src/crimson/os/seastore/backref_manager.h +++ b/src/crimson/os/seastore/backref_manager.h @@ -6,6 +6,7 @@ #include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/transaction.h" +#include "crimson/os/seastore/backref_mapping.h" namespace crimson::os::seastore { diff --git a/src/crimson/os/seastore/backref_mapping.h b/src/crimson/os/seastore/backref_mapping.h new file mode 100644 index 00000000000..d0a6a0ea6ff --- /dev/null +++ b/src/crimson/os/seastore/backref_mapping.h @@ -0,0 +1,27 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include "crimson/os/seastore/btree/btree_range_pin.h" + +namespace crimson::os::seastore { + +class BackrefMapping : public BtreeNodeMapping<paddr_t, laddr_t> { + extent_types_t type; +public: + BackrefMapping(op_context_t<paddr_t> ctx) + : BtreeNodeMapping(ctx) {} + template <typename... T> + BackrefMapping(extent_types_t type, T&&... t) + : BtreeNodeMapping(std::forward<T>(t)...), + type(type) {} + extent_types_t get_type() const { + return type; + } +}; + +using BackrefMappingRef = std::unique_ptr<BackrefMapping>; +using backref_pin_list_t = std::list<BackrefMappingRef>; + +} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/btree/btree_range_pin.cc b/src/crimson/os/seastore/btree/btree_range_pin.cc deleted file mode 100644 index f0d507a24c4..00000000000 --- a/src/crimson/os/seastore/btree/btree_range_pin.cc +++ /dev/null @@ -1,54 +0,0 @@ -// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- -// vim: ts=8 sw=2 smarttab - -#include "crimson/os/seastore/btree/btree_range_pin.h" -#include "crimson/os/seastore/btree/fixed_kv_node.h" - -namespace crimson::os::seastore { - -template <typename key_t, typename val_t> -get_child_ret_t<LogicalCachedExtent> -BtreeNodeMapping<key_t, val_t>::get_logical_extent( - Transaction &t) -{ - ceph_assert(is_parent_viewable()); - assert(pos != std::numeric_limits<uint16_t>::max()); - ceph_assert(t.get_trans_id() == ctx.trans.get_trans_id()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - auto v = p.template get_child<LogicalCachedExtent>(ctx, pos, k); - if (!v.has_child()) { - this->child_pos = v.get_child_pos(); - } - return v; -} - -template <typename key_t, typename val_t> -bool BtreeNodeMapping<key_t, val_t>::is_stable() const -{ - assert(!this->parent_modified()); - assert(pos != std::numeric_limits<uint16_t>::max()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - return p.is_child_stable(ctx, pos, k); -} - -template <typename key_t, typename val_t> -bool BtreeNodeMapping<key_t, val_t>::is_data_stable() const -{ - assert(!this->parent_modified()); - assert(pos != std::numeric_limits<uint16_t>::max()); - auto &p = (FixedKVNode<key_t>&)*parent; - auto k = this->is_indirect() - ? this->get_intermediate_base() - : get_key(); - return p.is_child_data_stable(ctx, pos, k); -} - -template class BtreeNodeMapping<laddr_t, paddr_t>; -template class BtreeNodeMapping<paddr_t, laddr_t>; -} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/btree/btree_range_pin.h b/src/crimson/os/seastore/btree/btree_range_pin.h index 91751801e5d..bfd350a8bed 100644 --- a/src/crimson/os/seastore/btree/btree_range_pin.h +++ b/src/crimson/os/seastore/btree/btree_range_pin.h @@ -7,11 +7,12 @@ #include "crimson/common/log.h" -#include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/seastore_types.h" +#include "crimson/os/seastore/transaction.h" namespace crimson::os::seastore { +class Cache; template <typename node_key_t> struct op_context_t { @@ -116,8 +117,6 @@ protected: extent_len_t len = 0; fixed_kv_node_meta_t<key_t> range; uint16_t pos = std::numeric_limits<uint16_t>::max(); - - virtual std::unique_ptr<BtreeNodeMapping> _duplicate(op_context_t<key_t>) const = 0; fixed_kv_node_meta_t<key_t> _get_pin_range() const { return range; } @@ -139,11 +138,7 @@ public: len(len), range(meta), pos(pos) - { - if (!parent->is_pending()) { - this->child_pos = {parent, pos}; - } - } + {} CachedExtentRef get_parent() const final { return parent; @@ -162,11 +157,6 @@ public: return len; } - extent_types_t get_type() const override { - ceph_abort("should never happen"); - return extent_types_t::ROOT; - } - val_t get_val() const final { if constexpr (std::is_same_v<val_t, paddr_t>) { return value.get_paddr(); @@ -180,16 +170,6 @@ public: return range.begin; } - PhysicalNodeMappingRef<key_t, val_t> duplicate() const final { - auto ret = _duplicate(ctx); - ret->range = range; - ret->value = value; - ret->parent = parent; - ret->len = len; - ret->pos = pos; - return ret; - } - bool has_been_invalidated() const final { return parent->has_been_invalidated(); } @@ -215,9 +195,6 @@ public: return unviewable; } - get_child_ret_t<LogicalCachedExtent> get_logical_extent(Transaction&) final; - bool is_stable() const final; - bool is_data_stable() const final; bool is_parent_viewable() const final { ceph_assert(parent); if (!parent->is_valid()) { diff --git a/src/crimson/os/seastore/cache.cc b/src/crimson/os/seastore/cache.cc index 254e3eb0da4..86f816e1648 100644 --- a/src/crimson/os/seastore/cache.cc +++ b/src/crimson/os/seastore/cache.cc @@ -28,15 +28,6 @@ SET_SUBSYS(seastore_cache); namespace crimson::os::seastore { -std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent) { - return out << "backref_entry_t{" - << ent.paddr << "~" << ent.len << ", " - << "laddr: " << ent.laddr << ", " - << "type: " << ent.type << ", " - << "seq: " << ent.seq << ", " - << "}"; -} - Cache::Cache( ExtentPlacementManager &epm) : epm(epm), @@ -44,7 +35,7 @@ Cache::Cache( "seastore_cache_lru_size")) { LOG_PREFIX(Cache::Cache); - INFO("created, lru_capacity={}B", lru.get_capacity_bytes()); + INFO("created, lru_capacity=0x{:x}B", lru.get_capacity_bytes()); register_metrics(); segment_providers_by_device_id.resize(DEVICE_ID_MAX, nullptr); } @@ -1348,21 +1339,39 @@ record_t Cache::prepare_record( io_stat_t retire_stat; std::vector<alloc_delta_t> alloc_deltas; alloc_delta_t rel_delta; + backref_entry_refs_t backref_entries; rel_delta.op = alloc_delta_t::op_types_t::CLEAR; for (auto &i: t.retired_set) { auto &extent = i.extent; get_by_ext(efforts.retire_by_ext, extent->get_type()).increment(extent->get_length()); retire_stat.increment(extent->get_length()); - DEBUGT("retired and remove extent -- {}", t, *extent); + DEBUGT("retired and remove extent {}~0x{:x} -- {}", + t, extent->get_paddr(), extent->get_length(), *extent); commit_retire_extent(t, extent); - if (is_backref_mapped_extent_node(extent) || - is_retired_placeholder_type(extent->get_type())) { + + // Note: commit extents and backref allocations in the same place + if (is_backref_mapped_type(extent->get_type()) || + is_retired_placeholder_type(extent->get_type())) { + DEBUGT("backref_entry free {}~0x{:x}", + t, + extent->get_paddr(), + extent->get_length()); rel_delta.alloc_blk_ranges.emplace_back( - extent->get_paddr(), - L_ADDR_NULL, - extent->get_length(), - extent->get_type()); + alloc_blk_t::create_retire( + extent->get_paddr(), + extent->get_length(), + extent->get_type())); + backref_entries.emplace_back( + backref_entry_t::create_retire( + extent->get_paddr(), + extent->get_length(), + extent->get_type())); + } else if (is_backref_node(extent->get_type())) { + remove_backref_extent(extent->get_paddr()); + } else { + ERRORT("Got unexpected extent type: {}", t, *extent); + ceph_abort("imposible"); } } alloc_deltas.emplace_back(std::move(rel_delta)); @@ -1399,27 +1408,40 @@ record_t Cache::prepare_record( if (modify_time == NULL_TIME) { modify_time = commit_time; } + laddr_t fresh_laddr; + if (i->is_logical()) { + fresh_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + fresh_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + fresh_laddr = L_ADDR_NULL; + } record.push_back(extent_t{ i->get_type(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), + fresh_laddr, std::move(bl) }, modify_time); - if (i->is_valid() - && is_backref_mapped_extent_node(i)) { + + if (!i->is_valid()) { + continue; + } + if (is_backref_mapped_type(i->get_type())) { + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + assert(i->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL); + alloc_laddr = L_ADDR_MIN; + } alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), - i->get_length(), - i->get_type()); + alloc_blk_t::create_alloc( + i->get_paddr(), + alloc_laddr, + i->get_length(), + i->get_type())); } } @@ -1430,14 +1452,20 @@ record_t Cache::prepare_record( get_by_ext(efforts.fresh_ool_by_ext, i->get_type()).increment(i->get_length()); i->prepare_commit(); - if (is_backref_mapped_extent_node(i)) { + if (is_backref_mapped_type(i->get_type())) { + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else { + assert(is_lba_node(i->get_type())); + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin, - i->get_length(), - i->get_type()); + alloc_blk_t::create_alloc( + i->get_paddr(), + alloc_laddr, + i->get_length(), + i->get_type())); } } @@ -1455,19 +1483,57 @@ record_t Cache::prepare_record( i->state = CachedExtent::extent_state_t::CLEAN; assert(i->is_logical()); i->clear_modified_region(); - touch_extent(*i, &trans_src); + touch_extent(*i, &trans_src, t.get_cache_hint()); DEBUGT("inplace rewrite ool block is commmitted -- {}", t, *i); } + auto existing_stats = t.get_existing_block_stats(); + DEBUGT("total existing blocks num: {}, exist clean num: {}, " + "exist mutation pending num: {}", + t, + existing_stats.valid_num, + existing_stats.clean_num, + existing_stats.mutated_num); for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - alloc_delta.alloc_blk_ranges.emplace_back( - i->get_paddr(), + assert(is_logical_type(i->get_type())); + if (!i->is_valid()) { + continue; + } + + if (i->is_exist_clean()) { + i->state = CachedExtent::extent_state_t::CLEAN; + } else { + assert(i->is_exist_mutation_pending()); + // i->state must become DIRTY in complete_commit() + } + + // exist mutation pending extents must be in t.mutated_block_list + add_extent(i); + const auto t_src = t.get_src(); + if (i->is_dirty()) { + add_to_dirty(i, &t_src); + } else { + touch_extent(*i, &t_src, t.get_cache_hint()); + } + + alloc_delta.alloc_blk_ranges.emplace_back( + alloc_blk_t::create_alloc( + i->get_paddr(), i->cast<LogicalCachedExtent>()->get_laddr(), i->get_length(), - i->get_type()); - } + i->get_type())); + + // Note: commit extents and backref allocations in the same place + // Note: remapping is split into 2 steps, retire and alloc, they must be + // committed atomically together + backref_entries.emplace_back( + backref_entry_t::create_alloc( + i->get_paddr(), + i->cast<LogicalCachedExtent>()->get_laddr(), + i->get_length(), + i->get_type())); } + alloc_deltas.emplace_back(std::move(alloc_delta)); for (auto b : alloc_deltas) { @@ -1521,6 +1587,9 @@ record_t Cache::prepare_record( record.push_back(std::move(delta)); } + apply_backref_mset(backref_entries); + t.set_backref_entries(std::move(backref_entries)); + ceph_assert(t.get_fresh_block_stats().num == t.inline_block_list.size() + t.ool_block_list.size() + @@ -1620,26 +1689,35 @@ record_t Cache::prepare_record( return record; } -void Cache::backref_batch_update( - std::vector<backref_entry_ref> &&list, - const journal_seq_t &seq) +void Cache::apply_backref_byseq( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq) { - LOG_PREFIX(Cache::backref_batch_update); - DEBUG("inserting {} entries at {}", list.size(), seq); - ceph_assert(seq != JOURNAL_SEQ_NULL); - - for (auto &ent : list) { - backref_entry_mset.insert(*ent); + LOG_PREFIX(Cache::apply_backref_byseq); + DEBUG("backref_entry apply {} entries at {}", + backref_entries.size(), seq); + assert(seq != JOURNAL_SEQ_NULL); + if (backref_entries.empty()) { + return; } - - auto iter = backref_entryrefs_by_seq.find(seq); - if (iter == backref_entryrefs_by_seq.end()) { - backref_entryrefs_by_seq.emplace(seq, std::move(list)); + if (backref_entryrefs_by_seq.empty()) { + backref_entryrefs_by_seq.insert( + backref_entryrefs_by_seq.end(), + {seq, std::move(backref_entries)}); + return; + } + auto last = backref_entryrefs_by_seq.rbegin(); + assert(last->first <= seq); + if (last->first == seq) { + last->second.insert( + last->second.end(), + std::make_move_iterator(backref_entries.begin()), + std::make_move_iterator(backref_entries.end())); } else { - iter->second.insert( - iter->second.end(), - std::make_move_iterator(list.begin()), - std::make_move_iterator(list.end())); + assert(last->first < seq); + backref_entryrefs_by_seq.insert( + backref_entryrefs_by_seq.end(), + {seq, std::move(backref_entries)}); } } @@ -1652,7 +1730,7 @@ void Cache::complete_commit( SUBTRACET(seastore_t, "final_block_start={}, start_seq={}", t, final_block_start, start_seq); - std::vector<backref_entry_ref> backref_list; + backref_entry_refs_t backref_entries; t.for_each_finalized_fresh_block([&](const CachedExtentRef &i) { if (!i->is_valid()) { return; @@ -1681,24 +1759,30 @@ void Cache::complete_commit( add_extent(i); assert(!i->is_dirty()); const auto t_src = t.get_src(); - touch_extent(*i, &t_src); + touch_extent(*i, &t_src, t.get_cache_hint()); epm.commit_space_used(i->get_paddr(), i->get_length()); - if (is_backref_mapped_extent_node(i)) { - DEBUGT("backref_list new {} len 0x{:x}", + + // Note: commit extents and backref allocations in the same place + if (is_backref_mapped_type(i->get_type())) { + DEBUGT("backref_entry alloc {}~0x{:x}", t, i->get_paddr(), i->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( + laddr_t alloc_laddr; + if (i->is_logical()) { + alloc_laddr = i->cast<LogicalCachedExtent>()->get_laddr(); + } else if (is_lba_node(i->get_type())) { + alloc_laddr = i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin; + } else { + assert(i->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL); + alloc_laddr = L_ADDR_MIN; + } + backref_entries.emplace_back( + backref_entry_t::create_alloc( i->get_paddr(), - i->is_logical() - ? i->cast<LogicalCachedExtent>()->get_laddr() - : (is_lba_node(i->get_type()) - ? i->cast<lba_manager::btree::LBANode>()->get_node_meta().begin - : L_ADDR_NULL), + alloc_laddr, i->get_length(), - i->get_type(), - start_seq)); + i->get_type())); } else if (is_backref_node(i->get_type())) { add_backref_extent( i->get_paddr(), @@ -1735,9 +1819,10 @@ void Cache::complete_commit( epm.mark_space_free(extent->get_paddr(), extent->get_length()); } for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - epm.mark_space_used(i->get_paddr(), i->get_length()); + if (!i->is_valid()) { + continue; } + epm.mark_space_used(i->get_paddr(), i->get_length()); } for (auto &i: t.mutated_block_list) { @@ -1751,64 +1836,10 @@ void Cache::complete_commit( for (auto &i: t.retired_set) { auto &extent = i.extent; extent->dirty_from_or_retired_at = start_seq; - if (is_backref_mapped_extent_node(extent) || - is_retired_placeholder_type(extent->get_type())) { - DEBUGT("backref_list free {} len 0x{:x}", - t, - extent->get_paddr(), - extent->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - extent->get_paddr(), - L_ADDR_NULL, - extent->get_length(), - extent->get_type(), - start_seq)); - } else if (is_backref_node(extent->get_type())) { - remove_backref_extent(extent->get_paddr()); - } else { - ERRORT("{}", t, *extent); - ceph_abort("not possible"); - } } - auto existing_stats = t.get_existing_block_stats(); - DEBUGT("total existing blocks num: {}, exist clean num: {}, " - "exist mutation pending num: {}", - t, - existing_stats.valid_num, - existing_stats.clean_num, - existing_stats.mutated_num); - for (auto &i: t.existing_block_list) { - if (i->is_valid()) { - if (i->is_exist_clean()) { - i->state = CachedExtent::extent_state_t::CLEAN; - } else { - assert(i->state == CachedExtent::extent_state_t::DIRTY); - } - DEBUGT("backref_list new existing {} len 0x{:x}", - t, - i->get_paddr(), - i->get_length()); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - i->get_paddr(), - i->cast<LogicalCachedExtent>()->get_laddr(), - i->get_length(), - i->get_type(), - start_seq)); - add_extent(i); - const auto t_src = t.get_src(); - if (i->is_dirty()) { - add_to_dirty(i, &t_src); - } else { - touch_extent(*i, &t_src); - } - } - } - if (!backref_list.empty()) { - backref_batch_update(std::move(backref_list), start_seq); - } + apply_backref_byseq(t.move_backref_entries(), start_seq); + commit_backref_entries(std::move(backref_entries), start_seq); for (auto &i: t.pre_alloc_list) { if (!i->is_valid()) { @@ -1931,7 +1962,7 @@ Cache::replay_delta( alloc_delta_t alloc_delta; decode(alloc_delta, delta.bl); - std::vector<backref_entry_ref> backref_list; + backref_entry_refs_t backref_entries; for (auto &alloc_blk : alloc_delta.alloc_blk_ranges) { if (alloc_blk.paddr.is_relative()) { assert(alloc_blk.paddr.is_record_relative()); @@ -1939,17 +1970,10 @@ Cache::replay_delta( } DEBUG("replay alloc_blk {}~0x{:x} {}, journal_seq: {}", alloc_blk.paddr, alloc_blk.len, alloc_blk.laddr, journal_seq); - backref_list.emplace_back( - std::make_unique<backref_entry_t>( - alloc_blk.paddr, - alloc_blk.laddr, - alloc_blk.len, - alloc_blk.type, - journal_seq)); - } - if (!backref_list.empty()) { - backref_batch_update(std::move(backref_list), journal_seq); + backref_entries.emplace_back( + backref_entry_t::create(alloc_blk)); } + commit_backref_entries(std::move(backref_entries), journal_seq); return replay_delta_ertr::make_ready_future<std::pair<bool, CachedExtentRef>>( std::make_pair(true, nullptr)); } @@ -2002,7 +2026,7 @@ Cache::replay_delta( [](CachedExtent &) {}, [this](CachedExtent &ext) { // replay is not included by the cache hit metrics - touch_extent(ext, nullptr); + touch_extent(ext, nullptr, CACHE_HINT_TOUCH); }, nullptr) : _get_extent_if_cached( diff --git a/src/crimson/os/seastore/cache.h b/src/crimson/os/seastore/cache.h index 520df473f2f..a239b861726 100644 --- a/src/crimson/os/seastore/cache.h +++ b/src/crimson/os/seastore/cache.h @@ -3,14 +3,13 @@ #pragma once -#include <iostream> - #include "seastar/core/shared_future.hh" #include "include/buffer.h" #include "crimson/common/errorator.h" #include "crimson/common/errorator-loop.h" +#include "crimson/os/seastore/backref_entry.h" #include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/extent_placement_manager.h" #include "crimson/os/seastore/logging.h" @@ -38,86 +37,6 @@ class FixedKVBtree; class BackrefManager; class SegmentProvider; -struct backref_entry_t { - backref_entry_t( - const paddr_t paddr, - const laddr_t laddr, - const extent_len_t len, - const extent_types_t type, - const journal_seq_t seq) - : paddr(paddr), - laddr(laddr), - len(len), - type(type), - seq(seq) - {} - backref_entry_t(alloc_blk_t alloc_blk) - : paddr(alloc_blk.paddr), - laddr(alloc_blk.laddr), - len(alloc_blk.len), - type(alloc_blk.type) - {} - paddr_t paddr = P_ADDR_NULL; - laddr_t laddr = L_ADDR_NULL; - extent_len_t len = 0; - extent_types_t type = - extent_types_t::ROOT; - journal_seq_t seq; - friend bool operator< ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr < r.paddr; - } - friend bool operator> ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr > r.paddr; - } - friend bool operator== ( - const backref_entry_t &l, - const backref_entry_t &r) { - return l.paddr == r.paddr; - } - - using set_hook_t = - boost::intrusive::set_member_hook< - boost::intrusive::link_mode< - boost::intrusive::auto_unlink>>; - set_hook_t backref_set_hook; - using backref_set_member_options = boost::intrusive::member_hook< - backref_entry_t, - set_hook_t, - &backref_entry_t::backref_set_hook>; - using multiset_t = boost::intrusive::multiset< - backref_entry_t, - backref_set_member_options, - boost::intrusive::constant_time_size<false>>; - - struct cmp_t { - using is_transparent = paddr_t; - bool operator()( - const backref_entry_t &l, - const backref_entry_t &r) const { - return l.paddr < r.paddr; - } - bool operator()(const paddr_t l, const backref_entry_t &r) const { - return l < r.paddr; - } - bool operator()(const backref_entry_t &l, const paddr_t r) const { - return l.paddr < r; - } - }; -}; - -std::ostream &operator<<(std::ostream &out, const backref_entry_t &ent); - -using backref_entry_ref = std::unique_ptr<backref_entry_t>; -using backref_entry_mset_t = backref_entry_t::multiset_t; -using backref_entry_refs_t = std::vector<backref_entry_ref>; -using backref_entryrefs_by_seq_t = std::map<journal_seq_t, backref_entry_refs_t>; -using backref_entry_query_set_t = std::set< - backref_entry_t, backref_entry_t::cmp_t>; - /** * Cache * @@ -205,6 +124,7 @@ public: TransactionRef create_transaction( Transaction::src_t src, const char* name, + cache_hint_t cache_hint, bool is_weak) { LOG_PREFIX(Cache::create_transaction); @@ -218,7 +138,8 @@ public: [this](Transaction& t) { return on_transaction_destruct(t); }, - ++next_id + ++next_id, + cache_hint ); SUBDEBUGT(seastore_t, "created name={}, source={}, is_weak={}", *ret, name, src, is_weak); @@ -365,7 +286,7 @@ public: SUBDEBUGT(seastore_cache, "{} {} is present in cache -- {}", t, type, offset, *ret); t.add_to_read_set(ret); - touch_extent(*ret, &t_src); + touch_extent(*ret, &t_src, t.get_cache_hint()); return ret->wait_io().then([ret] { return get_extent_if_cached_iertr::make_ready_future< CachedExtentRef>(ret); @@ -422,7 +343,7 @@ public: t, T::TYPE, offset, length); auto f = [&t, this, t_src](CachedExtent &ext) { t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent<T>( @@ -470,7 +391,7 @@ public: ++stats.access.s.load_absent; t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent<T>( @@ -568,7 +489,7 @@ public: ++access_stats.cache_lru; ++stats.access.s.cache_lru; } - touch_extent(*p_extent, &t_src); + touch_extent(*p_extent, &t_src, t.get_cache_hint()); } else { if (p_extent->is_dirty()) { ++access_stats.trans_dirty; @@ -915,7 +836,7 @@ private: t, type, offset, length, laddr); auto f = [&t, this, t_src](CachedExtent &ext) { t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent_by_type( @@ -957,7 +878,7 @@ private: ++stats.access.s.load_absent; t.add_to_read_set(CachedExtentRef(&ext)); - touch_extent(ext, &t_src); + touch_extent(ext, &t_src, t.get_cache_hint()); }; return trans_intr::make_interruptible( do_get_caching_extent_by_type( @@ -984,7 +905,7 @@ private: for (auto it = start_iter; it != end_iter; it++) { - res.emplace(it->paddr, it->laddr, it->len, it->type, it->seq); + res.emplace(it->paddr, it->laddr, it->len, it->type); } return res; } @@ -1336,7 +1257,7 @@ public: { LOG_PREFIX(Cache::init_cached_extents); SUBINFOT(seastore_cache, - "start with {}({}B) extents, {} dirty, dirty_from={}, alloc_from={}", + "start with {}(0x{:x}B) extents, {} dirty, dirty_from={}, alloc_from={}", t, extents_index.size(), extents_index.get_bytes(), @@ -1379,7 +1300,7 @@ public: } ).si_then([this, FNAME, &t] { SUBINFOT(seastore_cache, - "finish with {}({}B) extents, {} dirty, dirty_from={}, alloc_from={}", + "finish with {}(0x{:x}B) extents, {} dirty, dirty_from={}, alloc_from={}", t, extents_index.size(), extents_index.get_bytes(), @@ -1553,11 +1474,10 @@ private: /// Update lru for access to ref void touch_extent( CachedExtent &ext, - const Transaction::src_t* p_src) + const Transaction::src_t* p_src, + cache_hint_t hint) { - if (p_src && - is_background_transaction(*p_src) && - is_logical_type(ext.get_type())) { + if (hint == CACHE_HINT_NOCACHE && is_logical_type(ext.get_type())) { return; } if (ext.is_stable_clean() && !ext.is_placeholder()) { @@ -1907,9 +1827,23 @@ private: seastar::metrics::metric_group metrics; void register_metrics(); - void backref_batch_update( - std::vector<backref_entry_ref> &&, - const journal_seq_t &); + void apply_backref_mset( + backref_entry_refs_t& backref_entries) { + for (auto& entry : backref_entries) { + backref_entry_mset.insert(*entry); + } + } + + void apply_backref_byseq( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq); + + void commit_backref_entries( + backref_entry_refs_t&& backref_entries, + const journal_seq_t& seq) { + apply_backref_mset(backref_entries); + apply_backref_byseq(std::move(backref_entries), seq); + } /// Add extent to extents handling dirty and refcounting /// diff --git a/src/crimson/os/seastore/cached_extent.cc b/src/crimson/os/seastore/cached_extent.cc index 085a519cb68..49fede1d9a8 100644 --- a/src/crimson/os/seastore/cached_extent.cc +++ b/src/crimson/os/seastore/cached_extent.cc @@ -7,6 +7,7 @@ #include "crimson/common/log.h" #include "crimson/os/seastore/btree/fixed_kv_node.h" +#include "crimson/os/seastore/lba_mapping.h" namespace { [[maybe_unused]] seastar::logger& logger() { @@ -38,12 +39,6 @@ void intrusive_ptr_release(CachedExtent *ptr) #endif -bool is_backref_mapped_extent_node(const CachedExtentRef &extent) { - return extent->is_logical() - || is_lba_node(extent->get_type()) - || extent->get_type() == extent_types_t::TEST_BLOCK_PHYSICAL; -} - std::ostream &operator<<(std::ostream &out, CachedExtent::extent_state_t state) { switch (state) { @@ -148,6 +143,12 @@ void LogicalCachedExtent::on_replace_prior() { parent->children[off] = this; } +void LogicalCachedExtent::maybe_set_intermediate_laddr(LBAMapping &mapping) { + laddr = mapping.is_indirect() + ? mapping.get_intermediate_base() + : mapping.get_key(); +} + parent_tracker_t::~parent_tracker_t() { // this is parent's tracker, reset it auto &p = (FixedKVNode<laddr_t>&)*parent; @@ -156,32 +157,6 @@ parent_tracker_t::~parent_tracker_t() { } } -std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs) -{ - out << "LBAMapping(" << rhs.get_key() - << "~0x" << std::hex << rhs.get_length() << std::dec - << "->" << rhs.get_val(); - if (rhs.is_indirect()) { - out << ",indirect(" << rhs.get_intermediate_base() - << "~0x" << std::hex << rhs.get_intermediate_length() - << "@0x" << rhs.get_intermediate_offset() << std::dec - << ")"; - } - out << ")"; - return out; -} - -std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs) -{ - bool first = true; - out << '['; - for (const auto &i: rhs) { - out << (first ? "" : ",") << *i; - first = false; - } - return out << ']'; -} - bool BufferSpace::is_range_loaded(extent_len_t offset, extent_len_t length) const { assert(length > 0); diff --git a/src/crimson/os/seastore/cached_extent.h b/src/crimson/os/seastore/cached_extent.h index 2a54e8ded77..9dc60d719eb 100644 --- a/src/crimson/os/seastore/cached_extent.h +++ b/src/crimson/os/seastore/cached_extent.h @@ -6,6 +6,7 @@ #include <iostream> #include <boost/intrusive/list.hpp> +#include <boost/intrusive/set.hpp> #include <boost/intrusive_ptr.hpp> #include <boost/smart_ptr/intrusive_ref_counter.hpp> @@ -1096,8 +1097,6 @@ protected: std::ostream &operator<<(std::ostream &, CachedExtent::extent_state_t); std::ostream &operator<<(std::ostream &, const CachedExtent&); -bool is_backref_mapped_extent_node(const CachedExtentRef &extent); - /// Compare extents by paddr struct paddr_cmp { bool operator()(paddr_t lhs, const CachedExtent &rhs) const { @@ -1280,7 +1279,6 @@ private: }; class ChildableCachedExtent; -class LogicalCachedExtent; class child_pos_t { public: @@ -1338,48 +1336,18 @@ using PhysicalNodeMappingRef = std::unique_ptr<PhysicalNodeMapping<key_t, val_t> template <typename key_t, typename val_t> class PhysicalNodeMapping { public: + PhysicalNodeMapping() = default; + PhysicalNodeMapping(const PhysicalNodeMapping&) = delete; virtual extent_len_t get_length() const = 0; - virtual extent_types_t get_type() const = 0; virtual val_t get_val() const = 0; virtual key_t get_key() const = 0; - virtual PhysicalNodeMappingRef<key_t, val_t> duplicate() const = 0; - virtual PhysicalNodeMappingRef<key_t, val_t> refresh_with_pending_parent() { - ceph_abort("impossible"); - return {}; - } virtual bool has_been_invalidated() const = 0; virtual CachedExtentRef get_parent() const = 0; virtual uint16_t get_pos() const = 0; - // An lba pin may be indirect, see comments in lba_manager/btree/btree_lba_manager.h - virtual bool is_indirect() const { return false; } - virtual key_t get_intermediate_key() const { return min_max_t<key_t>::null; } - virtual key_t get_intermediate_base() const { return min_max_t<key_t>::null; } - virtual extent_len_t get_intermediate_length() const { return 0; } virtual uint32_t get_checksum() const { ceph_abort("impossible"); return 0; } - // The start offset of the pin, must be 0 if the pin is not indirect - virtual extent_len_t get_intermediate_offset() const { - return std::numeric_limits<extent_len_t>::max(); - } - - virtual get_child_ret_t<LogicalCachedExtent> - get_logical_extent(Transaction &t) = 0; - - void link_child(ChildableCachedExtent *c) { - ceph_assert(child_pos); - child_pos->link_child(c); - } - - // For reserved mappings, the return values are - // undefined although it won't crash - virtual bool is_stable() const = 0; - virtual bool is_data_stable() const = 0; - virtual bool is_clone() const = 0; - bool is_zero_reserved() const { - return !get_val().is_real(); - } virtual bool is_parent_viewable() const = 0; virtual bool is_parent_valid() const = 0; virtual bool parent_modified() const { @@ -1392,24 +1360,8 @@ public: } virtual ~PhysicalNodeMapping() {} -protected: - std::optional<child_pos_t> child_pos = std::nullopt; }; -using LBAMapping = PhysicalNodeMapping<laddr_t, paddr_t>; -using LBAMappingRef = PhysicalNodeMappingRef<laddr_t, paddr_t>; - -std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs); - -using lba_pin_list_t = std::list<LBAMappingRef>; - -std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs); - -using BackrefMapping = PhysicalNodeMapping<paddr_t, laddr_t>; -using BackrefMappingRef = PhysicalNodeMappingRef<paddr_t, laddr_t>; - -using backref_pin_list_t = std::list<BackrefMappingRef>; - /** * RetiredExtentPlaceholder * @@ -1523,6 +1475,8 @@ private: return out; } }; + +class LBAMapping; /** * LogicalCachedExtent * @@ -1557,11 +1511,7 @@ public: laddr = nladdr; } - void maybe_set_intermediate_laddr(LBAMapping &mapping) { - laddr = mapping.is_indirect() - ? mapping.get_intermediate_base() - : mapping.get_key(); - } + void maybe_set_intermediate_laddr(LBAMapping &mapping); void apply_delta_and_adjust_crc( paddr_t base, const ceph::bufferlist &bl) final { @@ -1661,8 +1611,6 @@ using lextent_list_t = addr_extent_list_base_t< } #if FMT_VERSION >= 90000 -template <> struct fmt::formatter<crimson::os::seastore::lba_pin_list_t> : fmt::ostream_formatter {}; template <> struct fmt::formatter<crimson::os::seastore::CachedExtent> : fmt::ostream_formatter {}; template <> struct fmt::formatter<crimson::os::seastore::LogicalCachedExtent> : fmt::ostream_formatter {}; -template <> struct fmt::formatter<crimson::os::seastore::LBAMapping> : fmt::ostream_formatter {}; #endif diff --git a/src/crimson/os/seastore/device.cc b/src/crimson/os/seastore/device.cc index c3bda82a7f6..cc83eb54826 100644 --- a/src/crimson/os/seastore/device.cc +++ b/src/crimson/os/seastore/device.cc @@ -12,7 +12,7 @@ namespace crimson::os::seastore { std::ostream& operator<<(std::ostream& out, const device_spec_t& ds) { return out << "device_spec(" - << "magic=" << ds.magic + << "magic=0x" << std::hex << ds.magic << std::dec << ", dtype=" << ds.dtype << ", " << device_id_printer_t{ds.id} << ")"; diff --git a/src/crimson/os/seastore/extent_placement_manager.cc b/src/crimson/os/seastore/extent_placement_manager.cc index 0458fbfed74..fd19eeb7e58 100644 --- a/src/crimson/os/seastore/extent_placement_manager.cc +++ b/src/crimson/os/seastore/extent_placement_manager.cc @@ -1069,8 +1069,8 @@ RandomBlockOolWriter::do_write( w_info.bp = bp; writes.push_back(w_info); } - TRACE("current extent: base off {} len {},\ - maybe-merged current extent: base off {} len {}", + TRACE("current extent: {}~0x{:x},\ + maybe-merged current extent: {}~0x{:x}", paddr, ex->get_length(), writes.back().offset, writes.back().bp.length()); } diff --git a/src/crimson/os/seastore/extent_placement_manager.h b/src/crimson/os/seastore/extent_placement_manager.h index 47fe71ae464..4ff9729c5f4 100644 --- a/src/crimson/os/seastore/extent_placement_manager.h +++ b/src/crimson/os/seastore/extent_placement_manager.h @@ -236,9 +236,9 @@ struct io_usage_t { cleaner_usage_t cleaner_usage; friend std::ostream &operator<<(std::ostream &out, const io_usage_t &usage) { return out << "io_usage_t(" - << "inline_usage=" << usage.inline_usage - << ", main_cleaner_usage=" << usage.cleaner_usage.main_usage - << ", cold_cleaner_usage=" << usage.cleaner_usage.cold_ool_usage + << "inline_usage=0x" << std::hex << usage.inline_usage + << ", main_cleaner_usage=0x" << usage.cleaner_usage.main_usage + << ", cold_cleaner_usage=0x" << usage.cleaner_usage.cold_ool_usage << std::dec << ")"; } }; @@ -424,7 +424,7 @@ public: : ext.start + (ext.len - left); allocs.emplace_back(alloc_result_t{start, std::move(bp), gen}); SUBDEBUGT(seastore_epm, - "allocated {} {}B extent at {}, hint={}, gen={}", + "allocated {} 0x{:x}B extent at {}, hint={}, gen={}", t, type, len, start, hint, gen); left -= len; } diff --git a/src/crimson/os/seastore/extentmap_manager.cc b/src/crimson/os/seastore/extentmap_manager.cc deleted file mode 100644 index b0dc1b8c8a8..00000000000 --- a/src/crimson/os/seastore/extentmap_manager.cc +++ /dev/null @@ -1,33 +0,0 @@ -// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- -// vim: ts=8 sw=2 smarttab -#include <experimental/iterator> -#include <iostream> - -#include "crimson/os/seastore/transaction_manager.h" -#include "crimson/os/seastore/extentmap_manager.h" -#include "crimson/os/seastore/extentmap_manager/btree/btree_extentmap_manager.h" -namespace crimson::os::seastore::extentmap_manager { - -ExtentMapManagerRef create_extentmap_manager( - TransactionManager &trans_manager) { - return ExtentMapManagerRef(new BtreeExtentMapManager(trans_manager)); -} - -} - -namespace crimson::os::seastore { - -std::ostream &operator<<(std::ostream &out, const extent_mapping_t &rhs) -{ - return out << "extent_mapping_t (" << rhs.logical_offset << "~" << rhs.length - << "->" << rhs.laddr << ")"; -} - -std::ostream &operator<<(std::ostream &out, const extent_map_list_t &rhs) -{ - out << '['; - std::copy(std::begin(rhs), std::end(rhs), std::experimental::make_ostream_joiner(out, ", ")); - return out << ']'; -} - -} diff --git a/src/crimson/os/seastore/journal.h b/src/crimson/os/seastore/journal.h index a5c9029c43c..298935bd22e 100644 --- a/src/crimson/os/seastore/journal.h +++ b/src/crimson/os/seastore/journal.h @@ -59,13 +59,13 @@ public: crimson::ct_error::erange, crimson::ct_error::input_output_error >; - using submit_record_ret = submit_record_ertr::future< - record_locator_t - >; - virtual submit_record_ret submit_record( + using on_submission_func_t = std::function< + void(record_locator_t)>; + virtual submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle - ) = 0; + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) = 0; /** * flush @@ -101,9 +101,6 @@ public: virtual replay_ret replay( delta_handler_t &&delta_handler) = 0; - virtual seastar::future<> finish_commit( - transaction_type_t type) = 0; - virtual ~Journal() {} virtual backend_type_t get_type() = 0; diff --git a/src/crimson/os/seastore/journal/circular_bounded_journal.cc b/src/crimson/os/seastore/journal/circular_bounded_journal.cc index 9ee8b1b997f..41ff8318aba 100644 --- a/src/crimson/os/seastore/journal/circular_bounded_journal.cc +++ b/src/crimson/os/seastore/journal/circular_bounded_journal.cc @@ -58,35 +58,52 @@ CircularBoundedJournal::close_ertr::future<> CircularBoundedJournal::close() return record_submitter.close(); } -CircularBoundedJournal::submit_record_ret +CircularBoundedJournal::submit_record_ertr::future<> CircularBoundedJournal::submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) { LOG_PREFIX(CircularBoundedJournal::submit_record); DEBUG("H{} {} start ...", (void*)&handle, record); assert(write_pipeline); - return do_submit_record(std::move(record), handle); + return do_submit_record( + std::move(record), handle, std::move(on_submission) + ).safe_then([this, t_src] { + if (is_trim_transaction(t_src)) { + return update_journal_tail( + trimmer.get_dirty_tail(), + trimmer.get_alloc_tail()); + } else { + return seastar::now(); + } + }); } -CircularBoundedJournal::submit_record_ret +CircularBoundedJournal::submit_record_ertr::future<> CircularBoundedJournal::do_submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + on_submission_func_t &&on_submission) { LOG_PREFIX(CircularBoundedJournal::do_submit_record); if (!record_submitter.is_available()) { DEBUG("H{} wait ...", (void*)&handle); return record_submitter.wait_available( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } auto action = record_submitter.check_action(record.size); if (action == RecordSubmitter::action_t::ROLL) { return record_submitter.roll_segment( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } @@ -99,13 +116,16 @@ CircularBoundedJournal::do_submit_record( return handle.enter(write_pipeline->device_submission ).then([submit_fut=std::move(submit_ret.future)]() mutable { return std::move(submit_fut); - }).safe_then([FNAME, this, &handle](record_locator_t result) { + }).safe_then([FNAME, this, &handle, on_submission=std::move(on_submission) + ](record_locator_t result) mutable { return handle.enter(write_pipeline->finalize - ).then([FNAME, this, result, &handle] { + ).then([FNAME, this, result, &handle, + on_submission=std::move(on_submission)] { DEBUG("H{} finish with {}", (void*)&handle, result); auto new_committed_to = result.write_result.get_end_seq(); record_submitter.update_committed_to(new_committed_to); - return result; + std::invoke(on_submission, result); + return seastar::now(); }); }); } @@ -392,13 +412,4 @@ Journal::replay_ret CircularBoundedJournal::replay( }); } -seastar::future<> CircularBoundedJournal::finish_commit(transaction_type_t type) { - if (is_trim_transaction(type)) { - return update_journal_tail( - trimmer.get_dirty_tail(), - trimmer.get_alloc_tail()); - } - return seastar::now(); -} - } diff --git a/src/crimson/os/seastore/journal/circular_bounded_journal.h b/src/crimson/os/seastore/journal/circular_bounded_journal.h index 874bd8dc086..16278df6cfe 100644 --- a/src/crimson/os/seastore/journal/circular_bounded_journal.h +++ b/src/crimson/os/seastore/journal/circular_bounded_journal.h @@ -80,9 +80,11 @@ public: return backend_type_t::RANDOM_BLOCK; } - submit_record_ret submit_record( + submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission ) final; seastar::future<> flush( @@ -148,8 +150,6 @@ public: return cjs.get_records_start(); } - seastar::future<> finish_commit(transaction_type_t type) final; - using cbj_delta_handler_t = std::function< replay_ertr::future<bool>( const record_locator_t&, @@ -160,7 +160,10 @@ public: cbj_delta_handler_t &&delta_handler, journal_seq_t tail); - submit_record_ret do_submit_record(record_t &&record, OrderingHandle &handle); + submit_record_ertr::future<> do_submit_record( + record_t &&record, + OrderingHandle &handle, + on_submission_func_t &&on_submission); void try_read_rolled_header(scan_valid_records_cursor &cursor) { paddr_t addr = convert_abs_addr_to_paddr( diff --git a/src/crimson/os/seastore/journal/record_submitter.cc b/src/crimson/os/seastore/journal/record_submitter.cc index adf8251b8a7..4976eee96e7 100644 --- a/src/crimson/os/seastore/journal/record_submitter.cc +++ b/src/crimson/os/seastore/journal/record_submitter.cc @@ -24,7 +24,7 @@ RecordBatch::add_pending( LOG_PREFIX(RecordBatch::add_pending); auto new_size = get_encoded_length_after(record, block_size); auto dlength_offset = pending.size.dlength; - TRACE("{} batches={}, write_size={}, dlength_offset={} ...", + TRACE("{} batches={}, write_size=0x{:x}, dlength_offset=0x{:x} ...", name, pending.get_size() + 1, new_size.get_encoded_length(), @@ -144,7 +144,7 @@ RecordSubmitter::RecordSubmitter( batches(new RecordBatch[io_depth + 1]) { LOG_PREFIX(RecordSubmitter); - INFO("{} io_depth_limit={}, batch_capacity={}, batch_flush_size={}, " + INFO("{} io_depth_limit={}, batch_capacity={}, batch_flush_size=0x{:x}, " "preferred_fullness={}", get_name(), io_depth, batch_capacity, batch_flush_size, preferred_fullness); diff --git a/src/crimson/os/seastore/journal/segment_allocator.cc b/src/crimson/os/seastore/journal/segment_allocator.cc index 11f3cc8fd31..5405662b91e 100644 --- a/src/crimson/os/seastore/journal/segment_allocator.cc +++ b/src/crimson/os/seastore/journal/segment_allocator.cc @@ -189,7 +189,7 @@ SegmentAllocator::write(ceph::bufferlist&& to_write) auto write_length = to_write.length(); auto write_start_offset = written_to; if (unlikely(LOCAL_LOGGER.is_enabled(seastar::log_level::trace))) { - TRACE("{} {}~{}", print_name, get_written_to(), write_length); + TRACE("{} {}~0x{:x}", print_name, get_written_to(), write_length); } assert(write_length > 0); assert((write_length % get_block_size()) == 0); @@ -250,7 +250,7 @@ SegmentAllocator::close_segment() close_seg_info.num_extents}; ceph::bufferlist bl; encode(tail, bl); - INFO("{} close segment {}, written_to={}", + INFO("{} close segment {}, written_to=0x{:x}", print_name, tail, written_to); diff --git a/src/crimson/os/seastore/journal/segmented_journal.cc b/src/crimson/os/seastore/journal/segmented_journal.cc index eca45f113c2..67c0b3fb8ac 100644 --- a/src/crimson/os/seastore/journal/segmented_journal.cc +++ b/src/crimson/os/seastore/journal/segmented_journal.cc @@ -368,25 +368,30 @@ seastar::future<> SegmentedJournal::flush(OrderingHandle &handle) }); } -SegmentedJournal::submit_record_ret +SegmentedJournal::submit_record_ertr::future<> SegmentedJournal::do_submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + on_submission_func_t &&on_submission) { LOG_PREFIX(SegmentedJournal::do_submit_record); if (!record_submitter.is_available()) { DEBUG("H{} wait ...", (void*)&handle); return record_submitter.wait_available( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } auto action = record_submitter.check_action(record.size); if (action == RecordSubmitter::action_t::ROLL) { DEBUG("H{} roll, unavailable ...", (void*)&handle); return record_submitter.roll_segment( - ).safe_then([this, record=std::move(record), &handle]() mutable { - return do_submit_record(std::move(record), handle); + ).safe_then([this, record=std::move(record), &handle, + on_submission=std::move(on_submission)]() mutable { + return do_submit_record( + std::move(record), handle, std::move(on_submission)); }); } else { // SUBMIT_FULL/NOT_FULL DEBUG("H{} submit {} ...", @@ -398,22 +403,27 @@ SegmentedJournal::do_submit_record( return handle.enter(write_pipeline->device_submission ).then([submit_fut=std::move(submit_ret.future)]() mutable { return std::move(submit_fut); - }).safe_then([FNAME, this, &handle](record_locator_t result) { + }).safe_then([FNAME, this, &handle, on_submission=std::move(on_submission) + ](record_locator_t result) mutable { return handle.enter(write_pipeline->finalize - ).then([FNAME, this, result, &handle] { + ).then([FNAME, this, result, &handle, + on_submission=std::move(on_submission)] { DEBUG("H{} finish with {}", (void*)&handle, result); auto new_committed_to = result.write_result.get_end_seq(); record_submitter.update_committed_to(new_committed_to); - return result; + std::invoke(on_submission, result); + return seastar::now(); }); }); } } -SegmentedJournal::submit_record_ret +SegmentedJournal::submit_record_ertr::future<> SegmentedJournal::submit_record( record_t &&record, - OrderingHandle &handle) + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) { LOG_PREFIX(SegmentedJournal::submit_record); DEBUG("H{} {} start ...", (void*)&handle, record); @@ -424,12 +434,13 @@ SegmentedJournal::submit_record( ).get_encoded_length(); auto max_record_length = journal_segment_allocator.get_max_write_length(); if (expected_size > max_record_length) { - ERROR("H{} {} exceeds max record size {}", + ERROR("H{} {} exceeds max record size 0x{:x}", (void*)&handle, record, max_record_length); return crimson::ct_error::erange::make(); } - return do_submit_record(std::move(record), handle); + return do_submit_record( + std::move(record), handle, std::move(on_submission)); } } diff --git a/src/crimson/os/seastore/journal/segmented_journal.h b/src/crimson/os/seastore/journal/segmented_journal.h index 891de7ec306..3f51de70fb3 100644 --- a/src/crimson/os/seastore/journal/segmented_journal.h +++ b/src/crimson/os/seastore/journal/segmented_journal.h @@ -44,9 +44,11 @@ public: close_ertr::future<> close() final; - submit_record_ret submit_record( + submit_record_ertr::future<> submit_record( record_t &&record, - OrderingHandle &handle) final; + OrderingHandle &handle, + transaction_type_t t_src, + on_submission_func_t &&on_submission) final; seastar::future<> flush(OrderingHandle &handle) final; @@ -59,9 +61,6 @@ public: backend_type_t get_type() final { return backend_type_t::SEGMENTED; } - seastar::future<> finish_commit(transaction_type_t type) { - return seastar::now(); - } bool is_checksum_needed() final { // segmented journal always requires checksum @@ -69,10 +68,10 @@ public: } private: - submit_record_ret do_submit_record( + submit_record_ertr::future<> do_submit_record( record_t &&record, - OrderingHandle &handle - ); + OrderingHandle &handle, + on_submission_func_t &&on_submission); SegmentSeqAllocatorRef segment_seq_allocator; SegmentAllocator journal_segment_allocator; diff --git a/src/crimson/os/seastore/lba_manager.h b/src/crimson/os/seastore/lba_manager.h index a050b2cdf47..9a34bf56157 100644 --- a/src/crimson/os/seastore/lba_manager.h +++ b/src/crimson/os/seastore/lba_manager.h @@ -19,6 +19,7 @@ #include "crimson/os/seastore/cache.h" #include "crimson/os/seastore/seastore_types.h" +#include "crimson/os/seastore/lba_mapping.h" namespace crimson::os::seastore { diff --git a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc index 007737ff450..888d3c359ac 100644 --- a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc +++ b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.cc @@ -94,6 +94,45 @@ void unlink_phy_tree_root_node<laddr_t>(RootBlockRef &root_block) { namespace crimson::os::seastore::lba_manager::btree { +get_child_ret_t<LogicalCachedExtent> +BtreeLBAMapping::get_logical_extent(Transaction &t) +{ + ceph_assert(is_parent_viewable()); + assert(pos != std::numeric_limits<uint16_t>::max()); + ceph_assert(t.get_trans_id() == ctx.trans.get_trans_id()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + auto v = p.template get_child<LogicalCachedExtent>(ctx, pos, k); + if (!v.has_child()) { + this->child_pos = v.get_child_pos(); + } + return v; +} + +bool BtreeLBAMapping::is_stable() const +{ + assert(!this->parent_modified()); + assert(pos != std::numeric_limits<uint16_t>::max()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + return p.is_child_stable(ctx, pos, k); +} + +bool BtreeLBAMapping::is_data_stable() const +{ + assert(!this->parent_modified()); + assert(pos != std::numeric_limits<uint16_t>::max()); + auto &p = static_cast<LBALeafNode&>(*parent); + auto k = this->is_indirect() + ? this->get_intermediate_base() + : get_key(); + return p.is_child_data_stable(ctx, pos, k); +} + BtreeLBAManager::mkfs_ret BtreeLBAManager::mkfs( Transaction &t) diff --git a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h index ef10ff9623b..e0902053d0e 100644 --- a/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h +++ b/src/crimson/os/seastore/lba_manager/btree/btree_lba_manager.h @@ -23,11 +23,15 @@ #include "crimson/os/seastore/lba_manager/btree/lba_btree_node.h" #include "crimson/os/seastore/btree/btree_range_pin.h" +namespace crimson::os::seastore { +class LogicalCachedExtent; +} + namespace crimson::os::seastore::lba_manager::btree { struct LBALeafNode; -class BtreeLBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { +class BtreeLBAMapping : public LBAMapping { // To support cloning, there are two kinds of lba mappings: // 1. physical lba mapping: the pladdr in the value of which is the paddr of // the corresponding extent; @@ -61,14 +65,14 @@ class BtreeLBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { // their keys. public: BtreeLBAMapping(op_context_t<laddr_t> ctx) - : BtreeNodeMapping(ctx) {} + : LBAMapping(ctx) {} BtreeLBAMapping( op_context_t<laddr_t> c, LBALeafNodeRef parent, uint16_t pos, lba_map_val_t &val, lba_node_meta_t meta) - : BtreeNodeMapping( + : LBAMapping( c, parent, pos, @@ -190,8 +194,12 @@ public: SUBDEBUGT(seastore_lba, "new pin {}", ctx.trans, static_cast<LBAMapping&>(*new_pin)); return new_pin; } + bool is_stable() const final; + bool is_data_stable() const final; + get_child_ret_t<LogicalCachedExtent> get_logical_extent(Transaction &t); + protected: - std::unique_ptr<BtreeNodeMapping<laddr_t, paddr_t>> _duplicate( + LBAMappingRef _duplicate( op_context_t<laddr_t> ctx) const final { auto pin = std::unique_ptr<BtreeLBAMapping>(new BtreeLBAMapping(ctx)); pin->key = key; diff --git a/src/crimson/os/seastore/lba_mapping.cc b/src/crimson/os/seastore/lba_mapping.cc new file mode 100644 index 00000000000..90fae09ce21 --- /dev/null +++ b/src/crimson/os/seastore/lba_mapping.cc @@ -0,0 +1,44 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#include "lba_mapping.h" + +namespace crimson::os::seastore { + +std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs) +{ + out << "LBAMapping(" << rhs.get_key() + << "~0x" << std::hex << rhs.get_length() << std::dec + << "->" << rhs.get_val(); + if (rhs.is_indirect()) { + out << ",indirect(" << rhs.get_intermediate_base() + << "~0x" << std::hex << rhs.get_intermediate_length() + << "@0x" << rhs.get_intermediate_offset() << std::dec + << ")"; + } + out << ")"; + return out; +} + +std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs) +{ + bool first = true; + out << '['; + for (const auto &i: rhs) { + out << (first ? "" : ",") << *i; + first = false; + } + return out << ']'; +} + +LBAMappingRef LBAMapping::duplicate() const { + auto ret = _duplicate(ctx); + ret->range = range; + ret->value = value; + ret->parent = parent; + ret->len = len; + ret->pos = pos; + return ret; +} + +} // namespace crimson::os::seastore diff --git a/src/crimson/os/seastore/lba_mapping.h b/src/crimson/os/seastore/lba_mapping.h new file mode 100644 index 00000000000..338d4d53f55 --- /dev/null +++ b/src/crimson/os/seastore/lba_mapping.h @@ -0,0 +1,73 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#pragma once + +#include "crimson/os/seastore/cached_extent.h" +#include "crimson/os/seastore/btree/btree_range_pin.h" + +namespace crimson::os::seastore { + +class LBAMapping; +using LBAMappingRef = std::unique_ptr<LBAMapping>; + +class LogicalCachedExtent; + +class LBAMapping : public BtreeNodeMapping<laddr_t, paddr_t> { +public: + LBAMapping(op_context_t<laddr_t> ctx) + : BtreeNodeMapping<laddr_t, paddr_t>(ctx) {} + template <typename... T> + LBAMapping(T&&... t) + : BtreeNodeMapping<laddr_t, paddr_t>(std::forward<T>(t)...) + { + if (!parent->is_pending()) { + this->child_pos = {parent, pos}; + } + } + + // An lba pin may be indirect, see comments in lba_manager/btree/btree_lba_manager.h + virtual bool is_indirect() const = 0; + virtual laddr_t get_intermediate_key() const = 0; + virtual laddr_t get_intermediate_base() const = 0; + virtual extent_len_t get_intermediate_length() const = 0; + // The start offset of the pin, must be 0 if the pin is not indirect + virtual extent_len_t get_intermediate_offset() const = 0; + + virtual get_child_ret_t<LogicalCachedExtent> + get_logical_extent(Transaction &t) = 0; + + void link_child(ChildableCachedExtent *c) { + ceph_assert(child_pos); + child_pos->link_child(c); + } + virtual LBAMappingRef refresh_with_pending_parent() = 0; + + // For reserved mappings, the return values are + // undefined although it won't crash + virtual bool is_stable() const = 0; + virtual bool is_data_stable() const = 0; + virtual bool is_clone() const = 0; + bool is_zero_reserved() const { + return !get_val().is_real(); + } + + LBAMappingRef duplicate() const; + + virtual ~LBAMapping() {} +protected: + virtual LBAMappingRef _duplicate(op_context_t<laddr_t>) const = 0; + std::optional<child_pos_t> child_pos = std::nullopt; +}; + +std::ostream &operator<<(std::ostream &out, const LBAMapping &rhs); +using lba_pin_list_t = std::list<LBAMappingRef>; + +std::ostream &operator<<(std::ostream &out, const lba_pin_list_t &rhs); + +} // namespace crimson::os::seastore + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::os::seastore::LBAMapping> : fmt::ostream_formatter {}; +template <> struct fmt::formatter<crimson::os::seastore::lba_pin_list_t> : fmt::ostream_formatter {}; +#endif diff --git a/src/crimson/os/seastore/object_data_handler.cc b/src/crimson/os/seastore/object_data_handler.cc index 36413f1206a..8f817a521cf 100644 --- a/src/crimson/os/seastore/object_data_handler.cc +++ b/src/crimson/os/seastore/object_data_handler.cc @@ -525,7 +525,7 @@ ObjectDataHandler::write_ret do_insertions( if (region.is_data()) { assert_aligned(region.len); ceph_assert(region.len == region.bl->length()); - DEBUGT("allocating extent: {}~{}", + DEBUGT("allocating extent: {}~0x{:x}", ctx.t, region.addr, region.len); @@ -556,7 +556,7 @@ ObjectDataHandler::write_ret do_insertions( ObjectDataHandler::write_iertr::pass_further{} ); } else if (region.is_zero()) { - DEBUGT("reserving: {}~{}", + DEBUGT("reserving: {}~0x{:x}", ctx.t, region.addr, region.len); @@ -698,7 +698,7 @@ public: << ", aligned_data_end=" << overwrite_plan.aligned_data_end << ", left_operation=" << overwrite_plan.left_operation << ", right_operation=" << overwrite_plan.right_operation - << ", block_size=" << overwrite_plan.block_size + << ", block_size=0x" << std::hex << overwrite_plan.block_size << std::dec << ", is_left_fresh=" << overwrite_plan.is_left_fresh << ", is_right_fresh=" << overwrite_plan.is_right_fresh << ")"; @@ -1059,13 +1059,13 @@ ObjectDataHandler::write_ret ObjectDataHandler::prepare_data_reservation( ceph_assert(size <= max_object_size); if (!object_data.is_null()) { ceph_assert(object_data.get_reserved_data_len() == max_object_size); - DEBUGT("reservation present: {}~{}", + DEBUGT("reservation present: {}~0x{:x}", ctx.t, object_data.get_reserved_data_base(), object_data.get_reserved_data_len()); return write_iertr::now(); } else { - DEBUGT("reserving: {}~{}", + DEBUGT("reserving: {}~0x{:x}", ctx.t, ctx.onode.get_data_hint(), max_object_size); @@ -1098,7 +1098,7 @@ ObjectDataHandler::clear_ret ObjectDataHandler::trim_data_reservation( LOG_PREFIX(ObjectDataHandler::trim_data_reservation); auto data_base = object_data.get_reserved_data_base(); auto data_len = object_data.get_reserved_data_len(); - DEBUGT("object_data: {}~{}", ctx.t, data_base, data_len); + DEBUGT("object_data: {}~0x{:x}", ctx.t, data_base, data_len); laddr_t aligned_start = (data_base + size).get_aligned_laddr(); loffset_t aligned_length = data_len - aligned_start.get_byte_distance<loffset_t>(data_base); @@ -1134,7 +1134,7 @@ ObjectDataHandler::clear_ret ObjectDataHandler::trim_data_reservation( if (append_len == 0) { LOG_PREFIX(ObjectDataHandler::trim_data_reservation); TRACET("First pin overlaps the boundary and has aligned data" - "create existing at addr:{}, len:{}", + "create existing at addr:{}, len:0x{:x}", ctx.t, pin.get_key(), size - pin_offset); to_write.push_back(extent_to_write_t::create_existing( pin.duplicate(), @@ -1156,7 +1156,7 @@ ObjectDataHandler::clear_ret ObjectDataHandler::trim_data_reservation( write_bl.append_zero(append_len); LOG_PREFIX(ObjectDataHandler::trim_data_reservation); TRACET("First pin overlaps the boundary and has unaligned data" - "create data at addr:{}, len:{}", + "create data at addr:{}, len:0x{:x}", ctx.t, pin.get_key(), write_bl.length()); to_write.push_back(extent_to_write_t::create_data( pin.get_key(), @@ -1303,7 +1303,7 @@ ObjectDataHandler::write_ret ObjectDataHandler::overwrite( (auto &pins, auto &to_write) mutable { LOG_PREFIX(ObjectDataHandler::overwrite); - DEBUGT("overwrite: {}~{}", + DEBUGT("overwrite: 0x{:x}~0x{:x}", ctx.t, offset, len); @@ -1393,7 +1393,7 @@ ObjectDataHandler::zero_ret ObjectDataHandler::zero( ctx, [this, ctx, offset, len](auto &object_data) { LOG_PREFIX(ObjectDataHandler::zero); - DEBUGT("zero to {}~{}, object_data: {}~{}, is_null {}", + DEBUGT("zero to 0x{:x}~0x{:x}, object_data: {}~0x{:x}, is_null {}", ctx.t, offset, len, @@ -1434,7 +1434,7 @@ ObjectDataHandler::write_ret ObjectDataHandler::write( ctx, [this, ctx, offset, &bl](auto &object_data) { LOG_PREFIX(ObjectDataHandler::write); - DEBUGT("writing to {}~{}, object_data: {}~{}, is_null {}", + DEBUGT("writing to 0x{:x}~0x{:x}, object_data: {}~0x{:x}, is_null {}", ctx.t, offset, bl.length(), @@ -1479,7 +1479,7 @@ ObjectDataHandler::read_ret ObjectDataHandler::read( ctx, [ctx, obj_offset, len, &ret](const auto &object_data) { LOG_PREFIX(ObjectDataHandler::read); - DEBUGT("reading {}~{}", + DEBUGT("reading {}~0x{:x}", ctx.t, object_data.get_reserved_data_base(), object_data.get_reserved_data_len()); @@ -1536,7 +1536,7 @@ ObjectDataHandler::read_ret ObjectDataHandler::read( l_current_end.get_byte_distance<extent_len_t>(l_current); if (pin->get_val().is_zero()) { - DEBUGT("got {}~{} from zero-pin {}~{}", + DEBUGT("got {}~0x{:x} from zero-pin {}~0x{:x}", ctx.t, l_current, read_len, @@ -1553,7 +1553,7 @@ ObjectDataHandler::read_ret ObjectDataHandler::read( l_current_end_aligned.get_byte_distance<extent_len_t>(pin_start); read_len_aligned -= read_start_aligned; extent_len_t unalign_start_offset = read_start - read_start_aligned; - DEBUGT("reading {}~{} from pin {}~{}", + DEBUGT("reading {}~0x{:x} from pin {}~0x{:x}", ctx.t, l_current, read_len, @@ -1609,7 +1609,7 @@ ObjectDataHandler::fiemap_ret ObjectDataHandler::fiemap( [ctx, obj_offset, len, &ret](const auto &object_data) { LOG_PREFIX(ObjectDataHandler::fiemap); DEBUGT( - "{}~{}, reservation {}~{}", + "0x{:x}~0x{:x}, reservation {}~0x{:x}", ctx.t, obj_offset, len, @@ -1664,7 +1664,7 @@ ObjectDataHandler::truncate_ret ObjectDataHandler::truncate( ctx, [this, ctx, offset](auto &object_data) { LOG_PREFIX(ObjectDataHandler::truncate); - DEBUGT("truncating {}~{} offset: {}", + DEBUGT("truncating {}~0x{:x} offset: 0x{:x}", ctx.t, object_data.get_reserved_data_base(), object_data.get_reserved_data_len(), @@ -1707,7 +1707,7 @@ ObjectDataHandler::clone_ret ObjectDataHandler::clone_extents( laddr_t data_base) { LOG_PREFIX(ObjectDataHandler::clone_extents); - TRACET(" object_data: {}~{}, data_base: {}", + TRACET("object_data: {}~0x{:x}, data_base: 0x{:x}", ctx.t, object_data.get_reserved_data_base(), object_data.get_reserved_data_len(), @@ -1792,7 +1792,7 @@ ObjectDataHandler::clone_ret ObjectDataHandler::clone( auto len = object_data.get_reserved_data_len(); object_data.clear(); LOG_PREFIX(ObjectDataHandler::clone); - DEBUGT("cloned obj reserve_data_base: {}, len {}", + DEBUGT("cloned obj reserve_data_base: {}, len 0x{:x}", ctx.t, d_object_data.get_reserved_data_base(), d_object_data.get_reserved_data_len()); @@ -1802,7 +1802,7 @@ ObjectDataHandler::clone_ret ObjectDataHandler::clone( d_object_data.get_reserved_data_len() ).si_then([&d_object_data, ctx, &object_data, base, len, this] { LOG_PREFIX("ObjectDataHandler::clone"); - DEBUGT("head obj reserve_data_base: {}, len {}", + DEBUGT("head obj reserve_data_base: {}, len 0x{:x}", ctx.t, object_data.get_reserved_data_base(), object_data.get_reserved_data_len()); diff --git a/src/crimson/os/seastore/onode.cc b/src/crimson/os/seastore/onode.cc index f3fd6eb18a5..dc8f6e87c8e 100644 --- a/src/crimson/os/seastore/onode.cc +++ b/src/crimson/os/seastore/onode.cc @@ -11,7 +11,7 @@ std::ostream& operator<<(std::ostream &out, const Onode &rhs) auto &layout = rhs.get_layout(); return out << "Onode(" << "hobj=" << rhs.hobj << ", " - << "size=" << static_cast<uint32_t>(layout.size) + << "size=0x" << std::hex << static_cast<uint32_t>(layout.size) << std::dec << ")"; } diff --git a/src/crimson/os/seastore/record_scanner.cc b/src/crimson/os/seastore/record_scanner.cc index 5fab11505ce..172ba77577e 100644 --- a/src/crimson/os/seastore/record_scanner.cc +++ b/src/crimson/os/seastore/record_scanner.cc @@ -18,7 +18,7 @@ RecordScanner::scan_valid_records( { LOG_PREFIX(RecordScanner::scan_valid_records); initialize_cursor(cursor); - DEBUG("starting at {}, budget={}", cursor, budget); + DEBUG("starting at {}, budget=0x{:x}", cursor, budget); auto retref = std::make_unique<size_t>(0); auto &budget_used = *retref; return crimson::repeat( @@ -91,7 +91,7 @@ RecordScanner::scan_valid_records( } }().safe_then([=, &budget_used, &cursor] { if (cursor.is_complete() || budget_used >= budget) { - DEBUG("finish at {}, budget_used={}, budget={}", + DEBUG("finish at {}, budget_used=0x{:x}, budget=0x{:x}", cursor, budget_used, budget); return seastar::stop_iteration::yes; } else { @@ -112,13 +112,13 @@ RecordScanner::read_validate_record_metadata( paddr_t start = cursor.seq.offset; auto block_size = cursor.get_block_size(); if (get_segment_off(cursor.seq.offset) + block_size > get_segment_end_offset(cursor.seq.offset)) { - DEBUG("failed -- record group header block {}~4096 > segment_size {}", - start, get_segment_end_offset(cursor.seq.offset)); + DEBUG("failed -- record group header block {}~0x{:x} > segment_size 0x{:x}", + start, block_size, get_segment_end_offset(cursor.seq.offset)); return read_validate_record_metadata_ret( read_validate_record_metadata_ertr::ready_future_marker{}, std::nullopt); } - TRACE("reading record group header block {}~4096", start); + TRACE("reading record group header block {}~0x{:x}", start, block_size); return read(start, block_size ).safe_then([this, FNAME, nonce, block_size, &cursor](bufferptr bptr) -> read_validate_record_metadata_ret { @@ -159,7 +159,7 @@ RecordScanner::read_validate_record_metadata( paddr_t rest_start = cursor.seq.offset.add_offset(block_size); auto rest_len = header.mdlength - block_size; - TRACE("reading record group header rest {}~{}", rest_start, rest_len); + TRACE("reading record group header rest {}~0x{:x}", rest_start, rest_len); return read(rest_start, rest_len ).safe_then([header=std::move(header), bl=std::move(bl) ](auto&& bptail) mutable { @@ -189,7 +189,7 @@ RecordScanner::read_validate_data_ret RecordScanner::read_validate_data( { LOG_PREFIX(RecordScanner::read_validate_data); auto data_addr = record_base.add_offset(header.mdlength); - TRACE("reading record group data blocks {}~{}", data_addr, header.dlength); + TRACE("reading record group data blocks {}~0x{:x}", data_addr, header.dlength); return read( data_addr, header.dlength @@ -220,7 +220,7 @@ RecordScanner::consume_next_records( total_length } }; - DEBUG("processing {} at {}, budget_used={}", + DEBUG("processing {} at {}, budget_used=0x{:x}", next.header, locator, budget_used); return handler( locator, diff --git a/src/crimson/os/seastore/seastore.cc b/src/crimson/os/seastore/seastore.cc index 2bf55d5cde7..6a866cb1f9b 100644 --- a/src/crimson/os/seastore/seastore.cc +++ b/src/crimson/os/seastore/seastore.cc @@ -408,6 +408,7 @@ SeaStore::Shard::mkfs_managers() return transaction_manager->with_transaction_intr( Transaction::src_t::MUTATE, "mkfs_seastore", + CACHE_HINT_TOUCH, [this](auto& t) { LOG_PREFIX(SeaStoreS::mkfs_managers); @@ -897,9 +898,10 @@ get_ranges(CollectionRef ch, seastar::future<std::tuple<std::vector<ghobject_t>, ghobject_t>> SeaStore::Shard::list_objects(CollectionRef ch, - const ghobject_t& start, - const ghobject_t& end, - uint64_t limit) const + const ghobject_t& start, + const ghobject_t& end, + uint64_t limit, + uint32_t op_flags) const { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -910,13 +912,14 @@ SeaStore::Shard::list_objects(CollectionRef ch, return seastar::do_with( RetType(std::vector<ghobject_t>(), start), std::move(limit), - [this, ch, start, end](auto& ret, auto& limit) { - return repeat_eagain([this, ch, start, end, &limit, &ret] { + [this, ch, start, end, op_flags](auto& ret, auto& limit) { + return repeat_eagain([this, ch, start, end, &limit, &ret, op_flags] { ++(shard_stats.repeat_read_num); return transaction_manager->with_transaction_intr( Transaction::src_t::READ, "list_objects", + op_flags, [this, ch, start, end, &limit, &ret](auto &t) { LOG_PREFIX(SeaStoreS::list_objects); @@ -1054,6 +1057,7 @@ SeaStore::Shard::list_collections() return transaction_manager->with_transaction_intr( Transaction::src_t::READ, "list_collections", + CACHE_HINT_TOUCH, [this, &ret](auto& t) { LOG_PREFIX(SeaStoreS::list_collections); @@ -1137,6 +1141,7 @@ SeaStore::Shard::read( Transaction::src_t::READ, "read", op_type_t::READ, + op_flags, [this, offset, len, op_flags](auto &t, auto &onode) { return _read(t, onode, offset, len, op_flags); }).finally([this] { @@ -1148,7 +1153,8 @@ SeaStore::Shard::read( SeaStore::Shard::base_errorator::future<bool> SeaStore::Shard::exists( CollectionRef c, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { LOG_PREFIX(SeaStoreS::exists); ++(shard_stats.read_num); @@ -1160,6 +1166,7 @@ SeaStore::Shard::exists( Transaction::src_t::READ, "exists", op_type_t::READ, + op_flags, [FNAME](auto& t, auto&) { DEBUGT("exists", t); return seastar::make_ready_future<bool>(true); @@ -1240,7 +1247,8 @@ SeaStore::Shard::get_attr_errorator::future<ceph::bufferlist> SeaStore::Shard::get_attr( CollectionRef ch, const ghobject_t& oid, - std::string_view name) const + std::string_view name, + uint32_t op_flags) const { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1251,6 +1259,7 @@ SeaStore::Shard::get_attr( Transaction::src_t::READ, "get_attr", op_type_t::GET_ATTR, + op_flags, [this, name](auto &t, auto& onode) { return _get_attr(t, onode, name); }).handle_error( @@ -1296,7 +1305,8 @@ SeaStore::Shard::_get_attrs( SeaStore::Shard::get_attrs_ertr::future<SeaStore::Shard::attrs_t> SeaStore::Shard::get_attrs( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1307,6 +1317,7 @@ SeaStore::Shard::get_attrs( Transaction::src_t::READ, "get_attrs", op_type_t::GET_ATTRS, + op_flags, [this](auto &t, auto& onode) { return _get_attrs(t, onode); }).handle_error( @@ -1331,14 +1342,15 @@ seastar::future<struct stat> SeaStore::Shard::_stat( st.st_blksize = device->get_block_size(); st.st_blocks = (st.st_size + st.st_blksize - 1) / st.st_blksize; st.st_nlink = 1; - DEBUGT("oid={}, size={}, blksize={}", + DEBUGT("oid={}, size=0x{:x}, blksize=0x{:x}", t, oid, st.st_size, st.st_blksize); return seastar::make_ready_future<struct stat>(st); } seastar::future<struct stat> SeaStore::Shard::stat( CollectionRef c, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1349,6 +1361,7 @@ seastar::future<struct stat> SeaStore::Shard::stat( Transaction::src_t::READ, "stat", op_type_t::STAT, + op_flags, [this, oid](auto &t, auto &onode) { return _stat(t, onode, oid); }).handle_error( @@ -1364,9 +1377,10 @@ seastar::future<struct stat> SeaStore::Shard::stat( SeaStore::Shard::get_attr_errorator::future<ceph::bufferlist> SeaStore::Shard::omap_get_header( CollectionRef ch, - const ghobject_t& oid) + const ghobject_t& oid, + uint32_t op_flags) { - return get_attr(ch, oid, OMAP_HEADER_XATTR_KEY); + return get_attr(ch, oid, OMAP_HEADER_XATTR_KEY, op_flags); } SeaStore::base_iertr::future<SeaStore::Shard::omap_values_t> @@ -1389,7 +1403,8 @@ SeaStore::Shard::read_errorator::future<SeaStore::Shard::omap_values_t> SeaStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const omap_keys_t &keys) + const omap_keys_t &keys, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1400,6 +1415,7 @@ SeaStore::Shard::omap_get_values( Transaction::src_t::READ, "omap_get_values", op_type_t::OMAP_GET_VALUES, + op_flags, [this, keys](auto &t, auto &onode) { return do_omap_get_values(t, onode, keys); }).finally([this] { @@ -1529,7 +1545,8 @@ SeaStore::Shard::read_errorator::future<SeaStore::Shard::omap_values_paged_t> SeaStore::Shard::omap_get_values( CollectionRef ch, const ghobject_t &oid, - const std::optional<std::string> &start) + const std::optional<std::string> &start, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1540,6 +1557,7 @@ SeaStore::Shard::omap_get_values( Transaction::src_t::READ, "omap_get_values2", op_type_t::OMAP_GET_VALUES2, + op_flags, [this, start](auto &t, auto &onode) { return do_omap_get_values(t, onode, start); }).finally([this] { @@ -1589,7 +1607,8 @@ SeaStore::Shard::fiemap( CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { ++(shard_stats.read_num); ++(shard_stats.pending_read_num); @@ -1600,6 +1619,7 @@ SeaStore::Shard::fiemap( Transaction::src_t::READ, "fiemap", op_type_t::READ, + op_flags, [this, off, len](auto &t, auto &onode) { return _fiemap(t, onode, off, len); }).finally([this] { @@ -1640,7 +1660,7 @@ seastar::future<> SeaStore::Shard::do_transaction_no_callbacks( [this, num_bytes](auto &ctx) { LOG_PREFIX(SeaStoreS::do_transaction_no_callbacks); return with_trans_intr(*ctx.transaction, [&ctx, this, FNAME, num_bytes](auto &t) { - DEBUGT("cid={}, {} operations, {} bytes, {} colls, {} objects ...", + DEBUGT("cid={}, {} operations, 0x{:x} bytes, {} colls, {} objects ...", t, ctx.ch->get_cid(), ctx.ext_transaction.get_num_ops(), num_bytes, @@ -2677,6 +2697,7 @@ seastar::future<> SeaStore::Shard::write_meta( return transaction_manager->with_transaction_intr( Transaction::src_t::MUTATE, "write_meta", + CACHE_HINT_NOCACHE, [this, &key, &value](auto& t) { LOG_PREFIX(SeaStoreS::write_meta); diff --git a/src/crimson/os/seastore/seastore.h b/src/crimson/os/seastore/seastore.h index fd7e177da63..e2a993b9e20 100644 --- a/src/crimson/os/seastore/seastore.h +++ b/src/crimson/os/seastore/seastore.h @@ -101,7 +101,8 @@ public: seastar::future<struct stat> stat( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<ceph::bufferlist> read( CollectionRef c, @@ -118,32 +119,38 @@ public: base_errorator::future<bool> exists( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; get_attr_errorator::future<ceph::bufferlist> get_attr( CollectionRef c, const ghobject_t& oid, - std::string_view name) const final; + std::string_view name, + uint32_t op_flags = 0) const final; get_attrs_ertr::future<attrs_t> get_attrs( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; read_errorator::future<omap_values_t> omap_get_values( CollectionRef c, const ghobject_t& oid, - const omap_keys_t& keys) final; + const omap_keys_t& keys, + uint32_t op_flags = 0) final; /// Retrieves paged set of values > start (if present) read_errorator::future<omap_values_paged_t> omap_get_values( CollectionRef c, ///< [in] collection const ghobject_t &oid, ///< [in] oid - const std::optional<std::string> &start ///< [in] start, empty for begin + const std::optional<std::string> &start, ///< [in] start, empty for begin + uint32_t op_flags = 0 ) final; ///< @return <done, values> values.empty() iff done get_attr_errorator::future<bufferlist> omap_get_header( CollectionRef c, - const ghobject_t& oid) final; + const ghobject_t& oid, + uint32_t op_flags = 0) final; /// std::get<1>(ret) returns end if and only if the listing has listed all /// the items within the range, otherwise it returns the next key to be listed. @@ -151,7 +158,8 @@ public: CollectionRef c, const ghobject_t& start, const ghobject_t& end, - uint64_t limit) const final; + uint64_t limit, + uint32_t op_flags = 0) const final; seastar::future<CollectionRef> create_new_collection(const coll_t& cid) final; seastar::future<CollectionRef> open_collection(const coll_t& cid) final; @@ -170,7 +178,8 @@ public: CollectionRef ch, const ghobject_t& oid, uint64_t off, - uint64_t len) final; + uint64_t len, + uint32_t op_flags = 0) final; unsigned get_max_attr_name_length() const final { return 256; @@ -251,7 +260,8 @@ public: return seastar::do_with( internal_context_t( ch, std::move(t), - transaction_manager->create_transaction(src, tname)), + transaction_manager->create_transaction( + src, tname, t.get_fadvise_flags())), std::forward<F>(f), [this, op_type](auto &ctx, auto &f) { assert(shard_stats.starting_io_num); @@ -298,20 +308,22 @@ public: Transaction::src_t src, const char* tname, op_type_t op_type, + cache_hint_t cache_hint_flags, F &&f) const { auto begin_time = std::chrono::steady_clock::now(); return seastar::do_with( oid, Ret{}, std::forward<F>(f), - [this, ch, src, op_type, begin_time, tname + [this, ch, src, op_type, begin_time, tname, cache_hint_flags ](auto &oid, auto &ret, auto &f) { - return repeat_eagain([&, this, ch, src, tname] { + return repeat_eagain([&, this, ch, src, tname, cache_hint_flags] { assert(src == Transaction::src_t::READ); ++(shard_stats.repeat_read_num); return transaction_manager->with_transaction_intr( src, tname, + cache_hint_flags, [&, this, ch, tname](auto& t) { LOG_PREFIX(SeaStoreS::repeat_with_onode); diff --git a/src/crimson/os/seastore/seastore_types.cc b/src/crimson/os/seastore/seastore_types.cc index 450118e5e75..a57f56d4ab4 100644 --- a/src/crimson/os/seastore/seastore_types.cc +++ b/src/crimson/os/seastore/seastore_types.cc @@ -351,11 +351,11 @@ std::ostream &operator<<(std::ostream &out, const delta_info_t &delta) << "type: " << delta.type << ", paddr: " << delta.paddr << ", laddr: " << delta.laddr - << ", prev_crc: " << delta.prev_crc - << ", final_crc: " << delta.final_crc - << ", length: " << delta.length + << ", prev_crc: 0x" << std::hex << delta.prev_crc + << ", final_crc: 0x" << delta.final_crc + << ", length: 0x" << delta.length << std::dec << ", pversion: " << delta.pversion - << ", ext_seq: " << delta.ext_seq + << ", ext_seq: " << segment_seq_printer_t{delta.ext_seq} << ", seg_type: " << delta.seg_type << ")"; } @@ -373,7 +373,7 @@ std::ostream &operator<<(std::ostream &out, const extent_info_t &info) return out << "extent_info_t(" << "type: " << info.type << ", addr: " << info.addr - << ", len: " << info.len + << ", len: 0x" << std::hex << info.len << std::dec << ")"; } @@ -387,7 +387,7 @@ std::ostream &operator<<(std::ostream &out, const segment_header_t &header) << " " << rewrite_gen_printer_t{header.generation} << ", dirty_tail=" << header.dirty_tail << ", alloc_tail=" << header.alloc_tail - << ", segment_nonce=" << header.segment_nonce + << ", segment_nonce=0x" << std::hex << header.segment_nonce << std::dec << ", modify_time=" << mod_time_point_printer_t{header.modify_time} << ")"; } @@ -398,7 +398,7 @@ std::ostream &operator<<(std::ostream &out, const segment_tail_t &tail) << tail.physical_segment_id << " " << tail.type << " " << segment_seq_printer_t{tail.segment_seq} - << ", segment_nonce=" << tail.segment_nonce + << ", segment_nonce=0x" << std::hex << tail.segment_nonce << std::dec << ", modify_time=" << mod_time_point_printer_t{tail.modify_time} << ", num_extents=" << tail.num_extents << ")"; @@ -464,8 +464,8 @@ std::ostream &operator<<(std::ostream& out, const record_size_t& rsize) { return out << "record_size_t(" << "record_type=" << rsize.record_type - << "raw_md=" << rsize.get_raw_mdlength() - << ", data=" << rsize.dlength + << "raw_md=0x" << std::hex << rsize.get_raw_mdlength() + << ", data=0x" << rsize.dlength << std::dec << ")"; } @@ -509,11 +509,11 @@ std::ostream& operator<<(std::ostream& out, const record_group_header_t& h) { return out << "record_group_header_t(" << "num_records=" << h.records - << ", mdlength=" << h.mdlength - << ", dlength=" << h.dlength - << ", nonce=" << h.segment_nonce + << ", mdlength=0x" << std::hex << h.mdlength + << ", dlength=0x" << h.dlength + << ", segment_nonce=0x" << h.segment_nonce << std::dec << ", committed_to=" << h.committed_to - << ", data_crc=" << h.data_crc + << ", data_crc=0x" << std::hex << h.data_crc << std::dec << ")"; } @@ -556,9 +556,9 @@ std::ostream& operator<<(std::ostream& out, const record_group_size_t& size) { return out << "record_group_size_t(" << "record_type=" << size.record_type - << "raw_md=" << size.get_raw_mdlength() - << ", data=" << size.dlength - << ", block_size=" << size.block_size + << "raw_md=0x" << std::hex << size.get_raw_mdlength() + << ", data=0x" << size.dlength + << ", block_size=0x" << size.block_size << std::dec << ", fullness=" << size.get_fullness() << ")"; } @@ -913,7 +913,7 @@ std::ostream& operator<<(std::ostream& out, const write_result_t& w) { return out << "write_result_t(" << "start=" << w.start_seq - << ", length=" << w.length + << ", length=0x" << std::hex << w.length << std::dec << ")"; } diff --git a/src/crimson/os/seastore/seastore_types.h b/src/crimson/os/seastore/seastore_types.h index 65cad878fba..5930469ca07 100644 --- a/src/crimson/os/seastore/seastore_types.h +++ b/src/crimson/os/seastore/seastore_types.h @@ -3,6 +3,7 @@ #pragma once +#include <deque> #include <limits> #include <numeric> #include <optional> @@ -14,13 +15,47 @@ #include "include/byteorder.h" #include "include/denc.h" +#include "include/encoding.h" #include "include/buffer.h" #include "include/intarith.h" #include "include/interval_set.h" #include "include/uuid.h" +#include "include/rados.h" namespace crimson::os::seastore { +class cache_hint_t { + enum hint_t { + TOUCH, + NOCACHE + }; +public: + static constexpr cache_hint_t get_touch() { + return hint_t::TOUCH; + } + static constexpr cache_hint_t get_nocache() { + return hint_t::NOCACHE; + } + cache_hint_t(uint32_t flags) { + if (unlikely(flags & CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) || + unlikely(flags & CEPH_OSD_OP_FLAG_FADVISE_NOCACHE)) { + hint = NOCACHE; + } + } + bool operator==(const cache_hint_t &other) const { + return hint == other.hint; + } + bool operator!=(const cache_hint_t &other) const { + return hint != other.hint; + } +private: + constexpr cache_hint_t(hint_t hint) : hint(hint) {} + hint_t hint = hint_t::TOUCH; +}; + +inline constexpr cache_hint_t CACHE_HINT_TOUCH = cache_hint_t::get_touch(); +inline constexpr cache_hint_t CACHE_HINT_NOCACHE = cache_hint_t::get_nocache(); + /* using a special xattr key "omap_header" to store omap header */ const std::string OMAP_HEADER_XATTR_KEY = "omap_header"; @@ -1226,7 +1261,6 @@ constexpr laddr_t L_ADDR_MAX = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX); constexpr laddr_t L_ADDR_MIN = laddr_t::from_raw_uint(0); constexpr laddr_t L_ADDR_NULL = L_ADDR_MAX; constexpr laddr_t L_ADDR_ROOT = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX - 1); -constexpr laddr_t L_ADDR_LBAT = laddr_t::from_raw_uint(laddr_t::RAW_VALUE_MAX - 2); struct __attribute__((packed)) laddr_le_t { ceph_le64 laddr; @@ -1467,6 +1501,23 @@ constexpr bool is_physical_type(extent_types_t type) { } } +constexpr bool is_backref_mapped_type(extent_types_t type) { + if ((type >= extent_types_t::LADDR_INTERNAL && + type <= extent_types_t::OBJECT_DATA_BLOCK) || + type == extent_types_t::TEST_BLOCK || + type == extent_types_t::TEST_BLOCK_PHYSICAL) { + assert(is_logical_type(type) || + is_lba_node(type) || + type == extent_types_t::TEST_BLOCK_PHYSICAL); + return true; + } else { + assert(!is_logical_type(type) && + !is_lba_node(type) && + type != extent_types_t::TEST_BLOCK_PHYSICAL); + return false; + } +} + constexpr bool is_real_type(extent_types_t type) { if (type <= extent_types_t::OBJECT_DATA_BLOCK || (type >= extent_types_t::TEST_BLOCK && @@ -1618,8 +1669,8 @@ struct delta_info_t { extent_types_t type = extent_types_t::NONE; ///< delta type paddr_t paddr; ///< physical address laddr_t laddr = L_ADDR_NULL; ///< logical address - uint32_t prev_crc = 0; - uint32_t final_crc = 0; + checksum_t prev_crc = 0; + checksum_t final_crc = 0; extent_len_t length = 0; ///< extent length extent_version_t pversion; ///< prior version segment_seq_t ext_seq; ///< seq of the extent's segment @@ -1943,12 +1994,13 @@ struct __attribute__((packed)) root_t { struct alloc_blk_t { alloc_blk_t( - paddr_t paddr, - laddr_t laddr, + const paddr_t& paddr, + const laddr_t& laddr, extent_len_t len, extent_types_t type) - : paddr(paddr), laddr(laddr), len(len), type(type) - {} + : paddr(paddr), laddr(laddr), len(len), type(type) { + assert(len > 0); + } explicit alloc_blk_t() = default; @@ -1964,6 +2016,25 @@ struct alloc_blk_t { denc(v.type, p); DENC_FINISH(p); } + + static alloc_blk_t create_alloc( + const paddr_t& paddr, + const laddr_t& laddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type)); + assert(laddr != L_ADDR_NULL); + return alloc_blk_t(paddr, laddr, len, type); + } + + static alloc_blk_t create_retire( + const paddr_t& paddr, + extent_len_t len, + extent_types_t type) { + assert(is_backref_mapped_type(type) || + is_retired_placeholder_type(type)); + return alloc_blk_t(paddr, L_ADDR_NULL, len, type); + } }; // use absolute address diff --git a/src/crimson/os/seastore/segment_manager.cc b/src/crimson/os/seastore/segment_manager.cc index 1be9cce5f6b..3eced41081e 100644 --- a/src/crimson/os/seastore/segment_manager.cc +++ b/src/crimson/os/seastore/segment_manager.cc @@ -16,10 +16,10 @@ namespace crimson::os::seastore { std::ostream& operator<<(std::ostream& out, const block_shard_info_t& sf) { out << "(" - << "size=" << sf.size - << ", segments=" <<sf.segments - << ", tracker_offset=" <<sf.tracker_offset - << ", first_segment_offset=" <<sf.first_segment_offset + << "size=0x" << std::hex << sf.size << std::dec + << ", segments=" << sf.segments + << ", tracker_offset=0x" << std::hex << sf.tracker_offset + << ", first_segment_offset=0x" << sf.first_segment_offset << std::dec <<")"; return out; } @@ -28,8 +28,8 @@ std::ostream& operator<<(std::ostream& out, const block_sm_superblock_t& sb) { out << "superblock(" << "shard_num=" << sb.shard_num - << ", segment_size=" << sb.segment_size - << ", block_size=" << sb.block_size + << ", segment_size=0x" << std::hex << sb.segment_size + << ", block_size=0x" << sb.block_size << std::dec << ", shard_info:"; for (auto &sf : sb.shard_infos) { out << sf diff --git a/src/crimson/os/seastore/segment_manager/block.cc b/src/crimson/os/seastore/segment_manager/block.cc index 0500271f81a..7077aad7407 100644 --- a/src/crimson/os/seastore/segment_manager/block.cc +++ b/src/crimson/os/seastore/segment_manager/block.cc @@ -60,7 +60,7 @@ static write_ertr::future<> do_write( { LOG_PREFIX(block_do_write); auto len = bptr.length(); - TRACE("{} poffset={}~{} ...", + TRACE("{} poffset=0x{:x}~0x{:x} ...", device_id_printer_t{device_id}, offset, len); return device.dma_write( offset, @@ -68,16 +68,16 @@ static write_ertr::future<> do_write( len ).handle_exception( [FNAME, device_id, offset, len](auto e) -> write_ertr::future<size_t> { - ERROR("{} poffset={}~{} got error -- {}", + ERROR("{} poffset=0x{:x}~0x{:x} got error -- {}", device_id_printer_t{device_id}, offset, len, e); return crimson::ct_error::input_output_error::make(); }).then([FNAME, device_id, offset, len](auto result) -> write_ertr::future<> { if (result != len) { - ERROR("{} poffset={}~{} write len={} inconsistent", + ERROR("{} poffset=0x{:x}~0x{:x} write len=0x{:x} inconsistent", device_id_printer_t{device_id}, offset, len, result); return crimson::ct_error::input_output_error::make(); } - TRACE("{} poffset={}~{} done", device_id_printer_t{device_id}, offset, len); + TRACE("{} poffset=0x{:x}~0x{:x} done", device_id_printer_t{device_id}, offset, len); return write_ertr::now(); }); } @@ -90,7 +90,7 @@ static write_ertr::future<> do_writev( size_t block_size) { LOG_PREFIX(block_do_writev); - TRACE("{} poffset={}~{}, {} buffers", + TRACE("{} poffset=0x{:x}~0x{:x}, {} buffers", device_id_printer_t{device_id}, offset, bl.length(), bl.get_num_buffers()); // writev requires each buffer to be aligned to the disks' block @@ -109,22 +109,22 @@ static write_ertr::future<> do_writev( auto off = offset + p.offset; auto len = p.length; auto& iov = p.iov; - TRACE("{} poffset={}~{} dma_write ...", + TRACE("{} poffset=0x{:x}~0x{:x} dma_write ...", device_id_printer_t{device_id}, off, len); return device.dma_write(off, std::move(iov) ).handle_exception( [FNAME, device_id, off, len](auto e) -> write_ertr::future<size_t> { - ERROR("{} poffset={}~{} dma_write got error -- {}", + ERROR("{} poffset=0x{:x}~0x{:x} dma_write got error -- {}", device_id_printer_t{device_id}, off, len, e); return crimson::ct_error::input_output_error::make(); }).then([FNAME, device_id, off, len](size_t written) -> write_ertr::future<> { if (written != len) { - ERROR("{} poffset={}~{} dma_write len={} inconsistent", + ERROR("{} poffset=0x{:x}~0x{:x} dma_write len=0x{:x} inconsistent", device_id_printer_t{device_id}, off, len, written); return crimson::ct_error::input_output_error::make(); } - TRACE("{} poffset={}~{} dma_write done", + TRACE("{} poffset=0x{:x}~0x{:x} dma_write done", device_id_printer_t{device_id}, off, len); return write_ertr::now(); }); @@ -140,7 +140,7 @@ static read_ertr::future<> do_read( bufferptr &bptr) { LOG_PREFIX(block_do_read); - TRACE("{} poffset={}~{} ...", device_id_printer_t{device_id}, offset, len); + TRACE("{} poffset=0x{:x}~0x{:x} ...", device_id_printer_t{device_id}, offset, len); assert(len <= bptr.length()); return device.dma_read( offset, @@ -153,16 +153,16 @@ static read_ertr::future<> do_read( // once seastar::future<T>::handle_exception() returns seastar::futurize_t<T> [FNAME, device_id, offset, len](auto e) -> read_ertr::future<size_t> { - ERROR("{} poffset={}~{} got error -- {}", + ERROR("{} poffset=0x{:x}~0x{:x} got error -- {}", device_id_printer_t{device_id}, offset, len, e); return crimson::ct_error::input_output_error::make(); }).then([FNAME, device_id, offset, len](auto result) -> read_ertr::future<> { if (result != len) { - ERROR("{} poffset={}~{} read len={} inconsistent", + ERROR("{} poffset=0x{:x}~0x{:x} read len=0x{:x} inconsistent", device_id_printer_t{device_id}, offset, len, result); return crimson::ct_error::input_output_error::make(); } - TRACE("{} poffset={}~{} done", device_id_printer_t{device_id}, offset, len); + TRACE("{} poffset=0x{:x}~0x{:x} done", device_id_printer_t{device_id}, offset, len); return read_ertr::now(); }); } @@ -174,7 +174,7 @@ SegmentStateTracker::write_out( uint64_t offset) { LOG_PREFIX(SegmentStateTracker::write_out); - DEBUG("{} poffset={}~{}", + DEBUG("{} poffset=0x{:x}~0x{:x}", device_id_printer_t{device_id}, offset, bptr.length()); return do_write(device_id, device, offset, bptr); } @@ -186,7 +186,7 @@ SegmentStateTracker::read_in( uint64_t offset) { LOG_PREFIX(SegmentStateTracker::read_in); - DEBUG("{} poffset={}~{}", + DEBUG("{} poffset=0x{:x}~0x{:x}", device_id_printer_t{device_id}, offset, bptr.length()); return do_read( device_id, @@ -230,7 +230,7 @@ block_sm_superblock_t make_superblock( + i * segments_per_shard * config_segment_size; } - INFO("{} disk_size={}, segment_size={}, block_size={}", + INFO("{} disk_size=0x{:x}, segment_size=0x{:x}, block_size=0x{:x}", device_id_printer_t{device_id}, size, uint64_t(config_segment_size), @@ -255,7 +255,7 @@ static check_create_device_ret check_create_device( size_t size) { LOG_PREFIX(block_check_create_device); - INFO("path={}, size={}", path, size); + INFO("path={}, size=0x{:x}", path, size); return seastar::open_file_dma( path, seastar::open_flags::exclusive | @@ -266,7 +266,7 @@ static check_create_device_ret check_create_device( file, [size, FNAME, &path](auto &f) -> seastar::future<> { - DEBUG("path={} created, truncating to {}", path, size); + DEBUG("path={} created, truncating to 0x{:x}", path, size); ceph_assert(f); return f.truncate( size @@ -318,8 +318,8 @@ open_device_ret open_device( ).then([stat, &path, FNAME](auto file) mutable { return file.size().then([stat, file, &path, FNAME](auto size) mutable { stat.size = size; - INFO("path={} successful, size={}, block_size={}", - path, stat.size, stat.block_size); + INFO("path={} successful, size=0x{:x}, block_size=0x{:x}", + path, stat.size, stat.block_size); return std::make_pair(file, stat); }); }); @@ -410,19 +410,19 @@ Segment::write_ertr::future<> BlockSegment::write( { LOG_PREFIX(BlockSegment::write); auto paddr = paddr_t::make_seg_paddr(id, offset); - DEBUG("{} offset={}~{} poffset={} ...", + DEBUG("{} offset=0x{:x}~0x{:x} poffset=0x{:x} ...", id, offset, bl.length(), manager.get_offset(paddr)); if (offset < write_pointer || offset % manager.superblock.block_size != 0 || bl.length() % manager.superblock.block_size != 0) { - ERROR("{} offset={}~{} poffset={} invalid write", + ERROR("{} offset=0x{:x}~0x{:x} poffset=0x{:x} invalid write", id, offset, bl.length(), manager.get_offset(paddr)); return crimson::ct_error::invarg::make(); } if (offset + bl.length() > manager.superblock.segment_size) { - ERROR("{} offset={}~{} poffset={} write out of the range {}", + ERROR("{} offset=0x{:x}~0x{:x} poffset=0x{:x} write out of the range 0x{:x}", id, offset, bl.length(), manager.get_offset(paddr), manager.superblock.segment_size); return crimson::ct_error::enospc::make(); @@ -443,7 +443,7 @@ Segment::close_ertr::future<> BlockSegmentManager::segment_close( LOG_PREFIX(BlockSegmentManager::segment_close); auto s_id = id.device_segment_id(); int unused_bytes = get_segment_size() - write_pointer; - INFO("{} unused_bytes={} ...", id, unused_bytes); + INFO("{} unused_bytes=0x{:x} ...", id, unused_bytes); assert(unused_bytes >= 0); assert(id.device_id() == get_device_id()); @@ -693,24 +693,24 @@ SegmentManager::read_ertr::future<> BlockSegmentManager::read( auto s_id = id.device_segment_id(); auto s_off = seg_addr.get_segment_off(); auto p_off = get_offset(addr); - DEBUG("{} offset={}~{} poffset={} ...", id, s_off, len, p_off); + DEBUG("{} offset=0x{:x}~0x{:x} poffset=0x{:x} ...", id, s_off, len, p_off); assert(addr.get_device_id() == get_device_id()); if (s_off % superblock.block_size != 0 || len % superblock.block_size != 0) { - ERROR("{} offset={}~{} poffset={} invalid read", id, s_off, len, p_off); + ERROR("{} offset=0x{:x}~0x{:x} poffset=0x{:x} invalid read", id, s_off, len, p_off); return crimson::ct_error::invarg::make(); } if (s_id >= get_num_segments()) { - ERROR("{} offset={}~{} poffset={} segment-id out of range {}", + ERROR("{} offset=0x{:x}~0x{:x} poffset=0x{:x} segment-id out of range {}", id, s_off, len, p_off, get_num_segments()); return crimson::ct_error::invarg::make(); } if (s_off + len > superblock.segment_size) { - ERROR("{} offset={}~{} poffset={} read out of range {}", + ERROR("{} offset=0x{:x}~0x{:x} poffset=0x{:x} read out of range 0x{:x}", id, s_off, len, p_off, superblock.segment_size); return crimson::ct_error::invarg::make(); } @@ -718,7 +718,7 @@ SegmentManager::read_ertr::future<> BlockSegmentManager::read( if (tracker->get(s_id) == segment_state_t::EMPTY) { // XXX: not an error during scanning, // might need refactor to increase the log level - DEBUG("{} offset={}~{} poffset={} invalid state {}", + DEBUG("{} offset=0x{:x}~0x{:x} poffset=0x{:x} invalid state {}", id, s_off, len, p_off, tracker->get(s_id)); return crimson::ct_error::enoent::make(); } diff --git a/src/crimson/os/seastore/segment_manager/ephemeral.cc b/src/crimson/os/seastore/segment_manager/ephemeral.cc index 4a4873afb94..bdd97e88733 100644 --- a/src/crimson/os/seastore/segment_manager/ephemeral.cc +++ b/src/crimson/os/seastore/segment_manager/ephemeral.cc @@ -20,8 +20,11 @@ namespace { namespace crimson::os::seastore::segment_manager { std::ostream &operator<<(std::ostream &lhs, const ephemeral_config_t &c) { - return lhs << "ephemeral_config_t(size=" << c.size << ", block_size=" << c.block_size - << ", segment_size=" << c.segment_size << ")"; + return lhs << "ephemeral_config_t(size=0x" + << std::hex << c.size + << ", block_size=0x" << c.block_size + << ", segment_size=0x" << c.segment_size + << std::dec << ")"; } EphemeralSegmentManagerRef create_test_ephemeral() { @@ -141,7 +144,8 @@ Segment::write_ertr::future<> EphemeralSegmentManager::segment_write( { auto& seg_addr = addr.as_seg_paddr(); logger().debug( - "segment_write to segment {} at offset {}, physical offset {}, len {}, crc {}", + "segment_write to segment {} at offset 0x{:x}, " + "physical offset 0x{:x}, len 0x{:x}, crc 0x{:x}", seg_addr.get_segment_id(), seg_addr.get_segment_off(), get_offset(addr), @@ -268,7 +272,7 @@ SegmentManager::read_ertr::future<> EphemeralSegmentManager::read( if (seg_addr.get_segment_off() + len > config.segment_size) { logger().error( - "EphemeralSegmentManager::read: invalid offset {}~{}!", + "EphemeralSegmentManager::read: invalid offset {}~0x{:x}!", addr, len); return crimson::ct_error::invarg::make(); @@ -279,7 +283,8 @@ SegmentManager::read_ertr::future<> EphemeralSegmentManager::read( bufferlist bl; bl.push_back(out); logger().debug( - "segment_read to segment {} at offset {}, physical offset {}, length {}, crc {}", + "segment_read to segment {} at offset 0x{:x}, " + "physical offset 0x{:x}, length 0x{:x}, crc 0x{:x}", seg_addr.get_segment_id().device_segment_id(), seg_addr.get_segment_off(), get_offset(addr), diff --git a/src/crimson/os/seastore/segment_manager/zbd.cc b/src/crimson/os/seastore/segment_manager/zbd.cc index 88521a947f8..22efbed5940 100644 --- a/src/crimson/os/seastore/segment_manager/zbd.cc +++ b/src/crimson/os/seastore/segment_manager/zbd.cc @@ -56,7 +56,7 @@ static open_device_ret open_device( path, seastar::follow_symlink::yes ).then([FNAME, mode, &path](auto stat) mutable { return seastar::open_file_dma(path, mode).then([=](auto file) { - DEBUG("open of device {} successful, size {}", + DEBUG("open of device {} successful, size 0x{:x}", path, stat.size); return std::make_pair(file, stat); @@ -100,11 +100,12 @@ static zbd_sm_metadata_t make_metadata( WARN("Ignoring configuration values for device and segment size"); INFO( - "device size: {}, available size: {}, block size: {}, allocated size: {}," - " total zones {}, zone size: {}, zone capacity: {}," - " total segments: {}, zones per segment: {}, segment size: {}" + "device size: 0x{:x}, available size: 0x{:x}," + " block size: 0x{:x}, allocated size: 0x{:x}," + " total zones {}, zone size: 0x{:x}, zone capacity: 0x{:x}," + " total segments: {}, zones per segment: {}, segment size: 0x{:x}" " conv zones: {}, swr zones: {}, per shard segments: {}" - " per shard available size: {}", + " per shard available size: 0x{:x}", total_size, available_size, data.block_size, @@ -126,8 +127,8 @@ static zbd_sm_metadata_t make_metadata( shard_infos[i].segments = per_shard_segments; shard_infos[i].first_segment_offset = zone_size * skipped_zones + i * segment_size * per_shard_segments; - INFO("First segment offset for shard {} is: {}", - i, shard_infos[i].first_segment_offset); + INFO("First segment offset for shard {} is: 0x{:x}", + i, shard_infos[i].first_segment_offset); } zbd_sm_metadata_t ret = zbd_sm_metadata_t{ @@ -248,7 +249,7 @@ static write_ertr::future<> do_write( bufferptr &bptr) { LOG_PREFIX(ZBDSegmentManager::do_write); - DEBUG("offset {} len {}", + DEBUG("offset 0x{:x} len 0x{:x}", offset, bptr.length()); return device.dma_write( @@ -277,7 +278,7 @@ static write_ertr::future<> do_writev( size_t block_size) { LOG_PREFIX(ZBDSegmentManager::do_writev); - DEBUG("{} offset {} len {}", + DEBUG("{} offset 0x{:x} len 0x{:x}", device_id_printer_t{device_id}, offset, bl.length()); // writev requires each buffer to be aligned to the disks' block // size, we need to rebuild here @@ -295,23 +296,23 @@ static write_ertr::future<> do_writev( auto off = offset + p.offset; auto len = p.length; auto& iov = p.iov; - DEBUG("{} poffset={}~{} dma_write ...", + DEBUG("{} poffset=0x{:x}~0x{:x} dma_write ...", device_id_printer_t{device_id}, off, len); return device.dma_write(off, std::move(iov) ).handle_exception( [FNAME, device_id, off, len](auto e) -> write_ertr::future<size_t> { - ERROR("{} poffset={}~{} dma_write got error -- {}", + ERROR("{} poffset=0x{:x}~0x{:x} dma_write got error -- {}", device_id_printer_t{device_id}, off, len, e); return crimson::ct_error::input_output_error::make(); }).then([FNAME, device_id, off, len](size_t written) -> write_ertr::future<> { if (written != len) { - ERROR("{} poffset={}~{} dma_write len={} inconsistent", + ERROR("{} poffset=0x{:x}~0x{:x} dma_write len=0x{:x} inconsistent", device_id_printer_t{device_id}, off, len, written); return crimson::ct_error::input_output_error::make(); } - DEBUG("{} poffset={}~{} dma_write done", + DEBUG("{} poffset=0x{:x}~0x{:x} dma_write done", device_id_printer_t{device_id}, off, len); return write_ertr::now(); @@ -329,12 +330,12 @@ write_metadata(seastar::file &device, zbd_sm_metadata_t sb) bufferptr(ceph::buffer::create_page_aligned(sb.block_size)), [=, &device](auto &bp) { LOG_PREFIX(ZBDSegmentManager::write_metadata); - DEBUG("block_size {}", sb.block_size); + DEBUG("block_size 0x{:x}", sb.block_size); bufferlist bl; encode(sb, bl); auto iter = bl.begin(); assert(bl.length() < sb.block_size); - DEBUG("buffer length {}", bl.length()); + DEBUG("buffer length 0x{:x}", bl.length()); iter.copy(bl.length(), bp.c_str()); DEBUG("doing writeout"); return do_write(device, 0, bp); @@ -349,7 +350,7 @@ static read_ertr::future<> do_read( { LOG_PREFIX(ZBDSegmentManager::do_read); assert(len <= bptr.length()); - DEBUG("offset {} len {}", + DEBUG("offset 0x{:x} len 0x{:x}", offset, len); return device.dma_read( @@ -659,7 +660,7 @@ SegmentManager::read_ertr::future<> ZBDSegmentManager::read( } if (seg_addr.get_segment_off() + len > metadata.segment_capacity) { - ERROR("invalid read offset {}, len {}", + ERROR("invalid read offset {}, len 0x{:x}", addr, len); return crimson::ct_error::invarg::make(); @@ -703,7 +704,7 @@ Segment::write_ertr::future<> ZBDSegmentManager::segment_write( assert(addr.get_device_id() == get_device_id()); assert((bl.length() % metadata.block_size) == 0); auto& seg_addr = addr.as_seg_paddr(); - DEBUG("write to segment {} at offset {}, physical offset {}, len {}", + DEBUG("write to segment {} at offset 0x{:x}, physical offset 0x{:x}, len 0x{:x}", seg_addr.get_segment_id(), seg_addr.get_segment_off(), get_offset(addr), @@ -756,7 +757,7 @@ Segment::write_ertr::future<> ZBDSegment::write( LOG_PREFIX(ZBDSegment::write); if (offset != write_pointer || offset % manager.metadata.block_size != 0) { ERROR("Segment offset and zone write pointer mismatch. " - "segment {} segment-offset {} write pointer {}", + "segment {} segment-offset 0x{:x} write pointer 0x{:x}", id, offset, write_pointer); return crimson::ct_error::invarg::make(); } @@ -772,7 +773,7 @@ Segment::write_ertr::future<> ZBDSegment::write_padding_bytes( size_t padding_bytes) { LOG_PREFIX(ZBDSegment::write_padding_bytes); - DEBUG("Writing {} padding bytes to segment {} at wp {}", + DEBUG("Writing 0x{:x} padding bytes to segment {} at wp 0x{:x}", padding_bytes, id, write_pointer); return crimson::repeat([FNAME, padding_bytes, this] () mutable { @@ -804,7 +805,7 @@ Segment::write_ertr::future<> ZBDSegment::advance_wp( { LOG_PREFIX(ZBDSegment::advance_wp); - DEBUG("Advancing write pointer from {} to {}", write_pointer, offset); + DEBUG("Advancing write pointer from 0x{:x} to 0x{:x}", write_pointer, offset); if (offset < write_pointer) { return crimson::ct_error::invarg::make(); } diff --git a/src/crimson/os/seastore/segment_manager_group.cc b/src/crimson/os/seastore/segment_manager_group.cc index 332b794b70e..f4822c9a18c 100644 --- a/src/crimson/os/seastore/segment_manager_group.cc +++ b/src/crimson/os/seastore/segment_manager_group.cc @@ -26,13 +26,13 @@ SegmentManagerGroup::read_segment_tail(segment_id_t segment) } ).safe_then([=, &segment_manager](bufferptr bptr) -> read_segment_tail_ret { LOG_PREFIX(SegmentManagerGroup::read_segment_tail); - DEBUG("segment {} bptr size {}", segment, bptr.length()); + DEBUG("segment {} bptr size 0x{:x}", segment, bptr.length()); segment_tail_t tail; bufferlist bl; bl.push_back(bptr); - DEBUG("segment {} block crc {}", + DEBUG("segment {} block crc 0x{:x}", segment, bl.begin().crc32c(segment_manager.get_block_size(), 0)); @@ -66,13 +66,13 @@ SegmentManagerGroup::read_segment_header(segment_id_t segment) } ).safe_then([=, &segment_manager](bufferptr bptr) -> read_segment_header_ret { LOG_PREFIX(SegmentManagerGroup::read_segment_header); - DEBUG("segment {} bptr size {}", segment, bptr.length()); + DEBUG("segment {} bptr size 0x{:x}", segment, bptr.length()); segment_header_t header; bufferlist bl; bl.push_back(bptr); - DEBUG("segment {} block crc {}", + DEBUG("segment {} block crc 0x{:x}", segment, bl.begin().crc32c(segment_manager.get_block_size(), 0)); @@ -111,7 +111,7 @@ SegmentManagerGroup::read(paddr_t start, size_t len) LOG_PREFIX(SegmentManagerGroup::read); assert(has_device(start.get_device_id())); auto& segment_manager = *segment_managers[start.get_device_id()]; - TRACE("reading data {}~{}", start, len); + TRACE("reading data {}~0x{:x}", start, len); return segment_manager.read( start, len diff --git a/src/crimson/os/seastore/transaction.h b/src/crimson/os/seastore/transaction.h index 9b95161a404..cd8c333c69f 100644 --- a/src/crimson/os/seastore/transaction.h +++ b/src/crimson/os/seastore/transaction.h @@ -8,11 +8,12 @@ #include <boost/intrusive/list.hpp> #include "crimson/common/log.h" +#include "crimson/os/seastore/backref_entry.h" +#include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/logging.h" #include "crimson/os/seastore/ordering_handle.h" -#include "crimson/os/seastore/seastore_types.h" -#include "crimson/os/seastore/cached_extent.h" #include "crimson/os/seastore/root_block.h" +#include "crimson/os/seastore/seastore_types.h" #include "crimson/os/seastore/transaction_interruptor.h" namespace crimson::os::seastore { @@ -408,12 +409,14 @@ public: src_t src, journal_seq_t initiated_after, on_destruct_func_t&& f, - transaction_id_t trans_id + transaction_id_t trans_id, + cache_hint_t cache_hint ) : weak(weak), handle(std::move(handle)), on_destruct(std::move(f)), src(src), - trans_id(trans_id) + trans_id(trans_id), + cache_hint(cache_hint) {} void invalidate_clear_write_set() { @@ -460,6 +463,7 @@ public: ool_write_stats = {}; rewrite_stats = {}; conflicted = false; + assert(backref_entries.empty()); if (!has_reset) { has_reset = true; } @@ -571,10 +575,23 @@ public: return pre_alloc_list; } + cache_hint_t get_cache_hint() const { + return cache_hint; + } + private: friend class Cache; friend Ref make_test_transaction(); + void set_backref_entries(backref_entry_refs_t&& entries) { + assert(backref_entries.empty()); + backref_entries = std::move(entries); + } + + backref_entry_refs_t move_backref_entries() { + return std::move(backref_entries); + } + /** * If set, *this may not be used to perform writes and will not provide * consistentency allowing operations using to avoid maintaining a read_set. @@ -669,6 +686,10 @@ private: transaction_id_t trans_id = TRANS_ID_NULL; seastar::lw_shared_ptr<rbm_pending_ool_t> pending_ool; + + backref_entry_refs_t backref_entries; + + cache_hint_t cache_hint = CACHE_HINT_TOUCH; }; using TransactionRef = Transaction::Ref; @@ -681,7 +702,8 @@ inline TransactionRef make_test_transaction() { Transaction::src_t::MUTATE, JOURNAL_SEQ_NULL, [](Transaction&) {}, - ++next_id + ++next_id, + CACHE_HINT_TOUCH ); } diff --git a/src/crimson/os/seastore/transaction_manager.cc b/src/crimson/os/seastore/transaction_manager.cc index 745df70235b..807d88b2cbc 100644 --- a/src/crimson/os/seastore/transaction_manager.cc +++ b/src/crimson/os/seastore/transaction_manager.cc @@ -66,6 +66,7 @@ TransactionManager::mkfs_ertr::future<> TransactionManager::mkfs() return with_transaction_intr( Transaction::src_t::MUTATE, "mkfs_tm", + CACHE_HINT_TOUCH, [this, FNAME](auto& t) { cache->init(); @@ -131,6 +132,7 @@ TransactionManager::mount() journal->get_trimmer().set_journal_head(start_seq); return with_transaction_weak( "mount", + CACHE_HINT_TOUCH, [this](auto &t) { return cache->init_cached_extents(t, [this](auto &t, auto &e) { @@ -221,7 +223,7 @@ TransactionManager::ref_ret TransactionManager::inc_ref( TRACET("{}", t, offset); return lba_manager->incref_extent(t, offset ).si_then([FNAME, offset, &t](auto result) { - DEBUGT("extent refcount is incremented to {} -- {}~{}, {}", + DEBUGT("extent refcount is incremented to {} -- {}~0x{:x}, {}", t, result.refcount, offset, result.length, result.addr); return result.refcount; }); @@ -461,8 +463,12 @@ TransactionManager::do_submit_transaction( } SUBTRACET(seastore_t, "submitting record", tref); - return journal->submit_record(std::move(record), tref.get_handle() - ).safe_then([this, FNAME, &tref](auto submit_result) mutable { + return journal->submit_record( + std::move(record), + tref.get_handle(), + tref.get_src(), + [this, FNAME, &tref](record_locator_t submit_result) + { SUBDEBUGT(seastore_t, "committed with {}", tref, submit_result); auto start_seq = submit_result.write_result.start_seq; journal->get_trimmer().set_journal_head(start_seq); @@ -473,10 +479,8 @@ TransactionManager::do_submit_transaction( journal->get_trimmer().update_journal_tails( cache->get_oldest_dirty_from().value_or(start_seq), cache->get_oldest_backref_dirty_from().value_or(start_seq)); - return journal->finish_commit(tref.get_src() - ).then([&tref] { - return tref.get_handle().complete(); - }); + }).safe_then([&tref] { + return tref.get_handle().complete(); }).handle_error( submit_transaction_iertr::pass_further{}, crimson::ct_error::assert_all{"Hit error submitting to journal"} @@ -508,7 +512,7 @@ TransactionManager::get_next_dirty_extents( size_t max_bytes) { LOG_PREFIX(TransactionManager::get_next_dirty_extents); - DEBUGT("max_bytes={}B, seq={}", t, max_bytes, seq); + DEBUGT("max_bytes=0x{:x}B, seq={}", t, max_bytes, seq); return cache->get_next_dirty_extents(t, seq, max_bytes); } diff --git a/src/crimson/os/seastore/transaction_manager.h b/src/crimson/os/seastore/transaction_manager.h index 0317d3b4bb8..e574460894a 100644 --- a/src/crimson/os/seastore/transaction_manager.h +++ b/src/crimson/os/seastore/transaction_manager.h @@ -741,8 +741,9 @@ public: TransactionRef create_transaction( Transaction::src_t src, const char* name, + cache_hint_t cache_hint = CACHE_HINT_TOUCH, bool is_weak=false) final { - return cache->create_transaction(src, name, is_weak); + return cache->create_transaction(src, name, cache_hint, is_weak); } using ExtentCallbackInterface::submit_transaction_direct_ret; @@ -1066,7 +1067,7 @@ private: auto crc = ref->calc_crc32c(); SUBTRACET( seastore_tm, - "got extent -- {}, chksum in the lba tree: {}, actual chksum: {}", + "got extent -- {}, chksum in the lba tree: 0x{:x}, actual chksum: 0x{:x}", t, *ref, pin->get_checksum(), @@ -1080,7 +1081,7 @@ private: } if (unlikely(inconsistent)) { SUBERRORT(seastore_tm, - "extent checksum inconsistent, recorded: {}, actual: {}, {}", + "extent checksum inconsistent, recorded: 0x{:x}, actual: 0x{:x}, {}", t, pin->get_checksum(), crc, @@ -1142,7 +1143,7 @@ private: auto crc = ref->calc_crc32c(); SUBTRACET( seastore_tm, - "got extent -- {}, chksum in the lba tree: {}, actual chksum: {}", + "got extent -- {}, chksum in the lba tree: 0x{:x}, actual chksum: 0x{:x}", t, *ref, pin->get_checksum(), @@ -1157,7 +1158,7 @@ private: } if (unlikely(inconsistent)) { SUBERRORT(seastore_tm, - "extent checksum inconsistent, recorded: {}, actual: {}, {}", + "extent checksum inconsistent, recorded: 0x{:x}, actual: 0x{:x}, {}", t, pin->get_checksum(), crc, diff --git a/src/crimson/osd/backfill_facades.h b/src/crimson/osd/backfill_facades.h index 64544d4c870..ce649303d4f 100644 --- a/src/crimson/osd/backfill_facades.h +++ b/src/crimson/osd/backfill_facades.h @@ -82,6 +82,9 @@ struct PGFacade final : BackfillState::PGFacade { } PGFacade(PG& pg) : pg(pg) {} + std::ostream &print(std::ostream &out) const override { + return out << pg; + } }; } // namespace crimson::osd diff --git a/src/crimson/osd/backfill_state.cc b/src/crimson/osd/backfill_state.cc index 837fd2eb2af..f957f072c93 100644 --- a/src/crimson/osd/backfill_state.cc +++ b/src/crimson/osd/backfill_state.cc @@ -8,11 +8,7 @@ #include "crimson/osd/backfill_state.h" #include "osd/osd_types_fmt.h" -namespace { - seastar::logger& logger() { - return crimson::get_logger(ceph_subsys_osd); - } -} +SET_SUBSYS(osd); namespace crimson::osd { @@ -27,22 +23,23 @@ BackfillState::BackfillState( progress_tracker( std::make_unique<BackfillState::ProgressTracker>(backfill_machine)) { - logger().debug("{}:{}", __func__, __LINE__); + LOG_PREFIX(BackfillState::BackfillState); + DEBUGDPP("", *backfill_machine.pg); backfill_machine.initiate(); } template <class S> BackfillState::StateHelper<S>::StateHelper() { - logger().debug("enter {}", - boost::typeindex::type_id<S>().pretty_name()); + LOG_PREFIX(BackfillState::StateHelper); + DEBUGDPP("enter {}", pg(), boost::typeindex::type_id<S>().pretty_name()); } template <class S> BackfillState::StateHelper<S>::~StateHelper() { - logger().debug("exit {}", - boost::typeindex::type_id<S>().pretty_name()); + LOG_PREFIX(BackfillState::StateHelper); + DEBUG("exit {}", boost::typeindex::type_id<S>().pretty_name()); } BackfillState::~BackfillState() = default; @@ -63,13 +60,16 @@ BackfillState::BackfillMachine::~BackfillMachine() = default; BackfillState::Initial::Initial(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::Initial::Initial); backfill_state().last_backfill_started = peering_state().earliest_backfill(); - logger().debug("{}: bft={} from {}", - __func__, peering_state().get_backfill_targets(), - backfill_state().last_backfill_started); + DEBUGDPP("{}: bft={} from {}", + pg(), + __func__, + peering_state().get_backfill_targets(), + backfill_state().last_backfill_started); for (const auto& bt : peering_state().get_backfill_targets()) { - logger().debug("{}: target shard {} from {}", - __func__, bt, peering_state().get_peer_last_backfill(bt)); + DEBUGDPP("{}: target shard {} from {}", + pg(), __func__, bt, peering_state().get_peer_last_backfill(bt)); } ceph_assert(peering_state().get_backfill_targets().size()); ceph_assert(!backfill_state().last_backfill_started.is_max()); @@ -80,7 +80,8 @@ BackfillState::Initial::Initial(my_context ctx) boost::statechart::result BackfillState::Initial::react(const BackfillState::Triggered& evt) { - logger().debug("{}: backfill triggered", __func__); + LOG_PREFIX(BackfillState::Initial::react::Triggered); + DEBUGDPP("", pg()); ceph_assert(backfill_state().last_backfill_started == \ peering_state().earliest_backfill()); ceph_assert(peering_state().is_backfilling()); @@ -93,26 +94,10 @@ BackfillState::Initial::react(const BackfillState::Triggered& evt) if (Enqueuing::all_enqueued(peering_state(), backfill_state().backfill_info, backfill_state().peer_backfill_info)) { - logger().debug("{}: switching to Done state", __func__); + DEBUGDPP("switching to Done state", pg()); return transit<BackfillState::Done>(); } else { - logger().debug("{}: switching to Enqueuing state", __func__); - return transit<BackfillState::Enqueuing>(); - } -} - -boost::statechart::result -BackfillState::Cancelled::react(const BackfillState::Triggered& evt) -{ - logger().debug("{}: backfill re-triggered", __func__); - ceph_assert(peering_state().is_backfilling()); - if (Enqueuing::all_enqueued(peering_state(), - backfill_state().backfill_info, - backfill_state().peer_backfill_info)) { - logger().debug("{}: switching to Done state", __func__); - return transit<BackfillState::Done>(); - } else { - logger().debug("{}: switching to Enqueuing state", __func__); + DEBUGDPP("switching to Enqueuing state", pg()); return transit<BackfillState::Enqueuing>(); } } @@ -120,9 +105,10 @@ BackfillState::Cancelled::react(const BackfillState::Triggered& evt) // -- Enqueuing void BackfillState::Enqueuing::maybe_update_range() { + LOG_PREFIX(BackfillState::Enqueuing::maybe_update_range); if (auto& primary_bi = backfill_state().backfill_info; primary_bi.version >= pg().get_projected_last_update()) { - logger().info("{}: bi is current", __func__); + INFODPP("bi is current", pg()); ceph_assert(primary_bi.version == pg().get_projected_last_update()); } else if (primary_bi.version >= peering_state().get_log_tail()) { if (peering_state().get_pg_log().get_log().empty() && @@ -136,31 +122,31 @@ void BackfillState::Enqueuing::maybe_update_range() ceph_assert(primary_bi.version == eversion_t()); return; } - logger().debug("{}: bi is old, ({}) can be updated with log to {}", - __func__, - primary_bi.version, - pg().get_projected_last_update()); + DEBUGDPP("{}: bi is old, ({}) can be updated with log to {}", + pg(), + primary_bi.version, + pg().get_projected_last_update()); auto func = [&](const pg_log_entry_t& e) { - logger().debug("maybe_update_range(lambda): updating from version {}", - e.version); + DEBUGDPP("maybe_update_range(lambda): updating from version {}", + pg(), e.version); if (e.soid >= primary_bi.begin && e.soid < primary_bi.end) { if (e.is_update()) { - logger().debug("maybe_update_range(lambda): {} updated to ver {}", - e.soid, e.version); + DEBUGDPP("maybe_update_range(lambda): {} updated to ver {}", + pg(), e.soid, e.version); primary_bi.objects.erase(e.soid); primary_bi.objects.insert(std::make_pair(e.soid, e.version)); } else if (e.is_delete()) { - logger().debug("maybe_update_range(lambda): {} removed", - e.soid); + DEBUGDPP("maybe_update_range(lambda): {} removed", + pg(), e.soid); primary_bi.objects.erase(e.soid); } } }; - logger().debug("{}: scanning pg log first", __func__); + DEBUGDPP("{}: scanning pg log first", pg()); peering_state().scan_log_after(primary_bi.version, func); - logger().debug("{}: scanning projected log", __func__); + DEBUGDPP("{}: scanning projected log", pg()); pg().get_projected_log().scan_log_after(primary_bi.version, func); primary_bi.version = pg().get_projected_last_update(); } else { @@ -244,6 +230,7 @@ void BackfillState::Enqueuing::trim_backfilled_object_from_intervals( BackfillState::Enqueuing::result_t BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) { + LOG_PREFIX(BackfillState::Enqueuing::remove_on_peers); // set `new_last_backfill_started` to `check` result_t result { {}, check }; for (const auto& bt : peering_state().get_backfill_targets()) { @@ -255,8 +242,8 @@ BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) backfill_listener().enqueue_drop(bt, pbi.begin, version); } } - logger().debug("{}: BACKFILL removing {} from peers {}", - __func__, check, result.pbi_targets); + DEBUGDPP("BACKFILL removing {} from peers {}", + pg(), check, result.pbi_targets); ceph_assert(!result.pbi_targets.empty()); return result; } @@ -264,7 +251,8 @@ BackfillState::Enqueuing::remove_on_peers(const hobject_t& check) BackfillState::Enqueuing::result_t BackfillState::Enqueuing::update_on_peers(const hobject_t& check) { - logger().debug("{}: check={}", __func__, check); + LOG_PREFIX(BackfillState::Enqueuing::update_on_peers); + DEBUGDPP("check={}", pg(), check); const auto& primary_bi = backfill_state().backfill_info; result_t result { {}, primary_bi.begin }; std::map<hobject_t, std::pair<eversion_t, std::vector<pg_shard_t>>> backfills; @@ -325,6 +313,7 @@ bool BackfillState::Enqueuing::Enqueuing::all_emptied( BackfillState::Enqueuing::Enqueuing(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::Enqueuing::Enqueuing); auto& primary_bi = backfill_state().backfill_info; // update our local interval to cope with recent changes @@ -334,8 +323,7 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) // that backfill will be spinning here over and over. For the sake // of performance and complexity we don't synchronize with entire PG. // similar can happen in classical OSD. - logger().warn("{}: bi is old, rescanning of local backfill_info", - __func__); + WARNDPP("bi is old, rescanning of local backfill_info", pg()); post_event(RequestPrimaryScanning{}); return; } else { @@ -347,13 +335,14 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) primary_bi)) { // need to grab one another chunk of the object namespace and restart // the queueing. - logger().debug("{}: reached end for current local chunk", __func__); + DEBUGDPP("reached end for current local chunk", pg()); post_event(RequestPrimaryScanning{}); return; } do { if (!backfill_listener().budget_available()) { + DEBUGDPP("throttle failed, turning to Waiting", pg()); post_event(RequestWaiting{}); return; } else if (should_rescan_replicas(backfill_state().peer_backfill_info, @@ -392,16 +381,25 @@ BackfillState::Enqueuing::Enqueuing(my_context ctx) } } while (!all_emptied(primary_bi, backfill_state().peer_backfill_info)); - if (backfill_state().progress_tracker->tracked_objects_completed() - && Enqueuing::all_enqueued(peering_state(), - backfill_state().backfill_info, - backfill_state().peer_backfill_info)) { - backfill_state().last_backfill_started = hobject_t::get_max(); - backfill_listener().update_peers_last_backfill(hobject_t::get_max()); + if (should_rescan_primary(backfill_state().peer_backfill_info, + primary_bi)) { + // need to grab one another chunk of the object namespace and restart + // the queueing. + DEBUGDPP("reached end for current local chunk", pg()); + post_event(RequestPrimaryScanning{}); + return; + } else { + if (backfill_state().progress_tracker->tracked_objects_completed() + && Enqueuing::all_enqueued(peering_state(), + backfill_state().backfill_info, + backfill_state().peer_backfill_info)) { + backfill_state().last_backfill_started = hobject_t::get_max(); + backfill_listener().update_peers_last_backfill(hobject_t::get_max()); + } + DEBUGDPP("reached end for both local and all peers " + "but still has in-flight operations", pg()); + post_event(RequestWaiting{}); } - logger().debug("{}: reached end for both local and all peers " - "but still has in-flight operations", __func__); - post_event(RequestWaiting{}); } // -- PrimaryScanning @@ -416,16 +414,45 @@ BackfillState::PrimaryScanning::PrimaryScanning(my_context ctx) boost::statechart::result BackfillState::PrimaryScanning::react(PrimaryScanned evt) { - logger().debug("{}", __func__); + LOG_PREFIX(BackfillState::PrimaryScanning::react::PrimaryScanned); + DEBUGDPP("", pg()); backfill_state().backfill_info = std::move(evt.result); - return transit<Enqueuing>(); + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } + return discard_event(); +} + +boost::statechart::result +BackfillState::PrimaryScanning::react(CancelBackfill evt) +{ + LOG_PREFIX(BackfillState::PrimaryScanning::react::SuspendBackfill); + DEBUGDPP("suspended within PrimaryScanning", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::PrimaryScanning::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::PrimaryScanning::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } boost::statechart::result BackfillState::PrimaryScanning::react(ObjectPushed evt) { - logger().debug("PrimaryScanning::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::PrimaryScanning::react::ObjectPushed); + DEBUGDPP("PrimaryScanning::react() on ObjectPushed; evt.object={}", + pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, true); return discard_event(); } @@ -443,11 +470,11 @@ bool BackfillState::ReplicasScanning::replica_needs_scan( BackfillState::ReplicasScanning::ReplicasScanning(my_context ctx) : my_base(ctx) { + LOG_PREFIX(BackfillState::ReplicasScanning::ReplicasScanning); for (const auto& bt : peering_state().get_backfill_targets()) { if (const auto& pbi = backfill_state().peer_backfill_info.at(bt); replica_needs_scan(pbi, backfill_state().backfill_info)) { - logger().debug("{}: scanning peer osd.{} from {}", - __func__, bt, pbi.end); + DEBUGDPP("scanning peer osd.{} from {}", pg(), bt, pbi.end); backfill_listener().request_replica_scan(bt, pbi.end, hobject_t{}); ceph_assert(waiting_on_backfill.find(bt) == \ @@ -469,8 +496,9 @@ BackfillState::ReplicasScanning::~ReplicasScanning() boost::statechart::result BackfillState::ReplicasScanning::react(ReplicaScanned evt) { - logger().debug("{}: got scan result from osd={}, result={}", - __func__, evt.from, evt.result); + LOG_PREFIX(BackfillState::ReplicasScanning::react::ReplicaScanned); + DEBUGDPP("got scan result from osd={}, result={}", + pg(), evt.from, evt.result); // TODO: maybe we'll be able to move waiting_on_backfill from // the machine to the state. ceph_assert(peering_state().is_backfill_target(evt.from)); @@ -479,12 +507,17 @@ BackfillState::ReplicasScanning::react(ReplicaScanned evt) if (waiting_on_backfill.empty()) { ceph_assert(backfill_state().peer_backfill_info.size() == \ peering_state().get_backfill_targets().size()); - return transit<Enqueuing>(); + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } } } else { - // we canceled backfill for a while due to a too full, and this + // we suspended backfill for a while due to a too full, and this // is an extra response from a non-too-full peer - logger().debug("{}: canceled backfill (too full?)", __func__); + DEBUGDPP("suspended backfill (too full?)", pg()); } return discard_event(); } @@ -492,17 +525,30 @@ BackfillState::ReplicasScanning::react(ReplicaScanned evt) boost::statechart::result BackfillState::ReplicasScanning::react(CancelBackfill evt) { - logger().debug("{}: cancelled within ReplicasScanning", - __func__); - waiting_on_backfill.clear(); - return transit<Cancelled>(); + LOG_PREFIX(BackfillState::ReplicasScanning::react::SuspendBackfill); + DEBUGDPP("suspended within ReplicasScanning", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::ReplicasScanning::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::ReplicasScanning::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } boost::statechart::result BackfillState::ReplicasScanning::react(ObjectPushed evt) { - logger().debug("ReplicasScanning::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::ReplicasScanning::react::ObjectPushed); + DEBUGDPP("ReplicasScanning::react() on ObjectPushed; evt.object={}", + pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, true); return discard_event(); } @@ -517,17 +563,45 @@ BackfillState::Waiting::Waiting(my_context ctx) boost::statechart::result BackfillState::Waiting::react(ObjectPushed evt) { - logger().debug("Waiting::react() on ObjectPushed; evt.object={}", - evt.object); + LOG_PREFIX(BackfillState::Waiting::react::ObjectPushed); + DEBUGDPP("Waiting::react() on ObjectPushed; evt.object={}", pg(), evt.object); backfill_state().progress_tracker->complete_to(evt.object, evt.stat, false); - return transit<Enqueuing>();; + if (!backfill_state().is_suspended()) { + return transit<Enqueuing>(); + } else { + DEBUGDPP("backfill suspended, not going Enqueuing", pg()); + backfill_state().go_enqueuing_on_resume(); + } + return discard_event(); +} + +boost::statechart::result +BackfillState::Waiting::react(CancelBackfill evt) +{ + LOG_PREFIX(BackfillState::Waiting::react::SuspendBackfill); + DEBUGDPP("suspended within Waiting", pg()); + backfill_state().on_suspended(); + return discard_event(); +} + +boost::statechart::result +BackfillState::Waiting::react(Triggered evt) +{ + LOG_PREFIX(BackfillState::Waiting::react::Triggered); + ceph_assert(backfill_state().is_suspended()); + if (backfill_state().on_resumed()) { + DEBUGDPP("Backfill resumed, going Enqueuing", pg()); + return transit<Enqueuing>(); + } + return discard_event(); } // -- Done BackfillState::Done::Done(my_context ctx) : my_base(ctx) { - logger().info("{}: backfill is done", __func__); + LOG_PREFIX(BackfillState::Done::Done); + INFODPP("backfill is done", pg()); backfill_listener().backfilled(); } @@ -537,13 +611,6 @@ BackfillState::Crashed::Crashed() ceph_abort_msg("{}: this should not happen"); } -// -- Cancelled -BackfillState::Cancelled::Cancelled(my_context ctx) - : my_base(ctx) -{ - ceph_assert(peering_state().get_backfill_targets().size()); -} - // ProgressTracker is an intermediary between the BackfillListener and // BackfillMachine + its states. All requests to push or drop an object // are directed through it. The same happens with notifications about @@ -577,8 +644,8 @@ void BackfillState::ProgressTracker::complete_to( const pg_stat_t& stats, bool may_push_to_max) { - logger().debug("{}: obj={}", - __func__, obj); + LOG_PREFIX(BackfillState::ProgressTracker::complete_to); + DEBUGDPP("obj={}", pg(), obj); if (auto completion_iter = registry.find(obj); completion_iter != std::end(registry)) { completion_iter->second = \ @@ -619,4 +686,19 @@ void BackfillState::enqueue_standalone_push( backfill_machine.backfill_listener.enqueue_push(obj, v, peers); } +void BackfillState::enqueue_standalone_delete( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers) +{ + progress_tracker->enqueue_drop(obj); + for (auto bt : peers) { + backfill_machine.backfill_listener.enqueue_drop(bt, obj, v); + } +} + +std::ostream &operator<<(std::ostream &out, const BackfillState::PGFacade &pg) { + return pg.print(out); +} + } // namespace crimson::osd diff --git a/src/crimson/osd/backfill_state.h b/src/crimson/osd/backfill_state.h index 072c91e079d..517a02ea4df 100644 --- a/src/crimson/osd/backfill_state.h +++ b/src/crimson/osd/backfill_state.h @@ -62,6 +62,8 @@ struct BackfillState { struct CancelBackfill : sc::event<CancelBackfill> { }; + struct ThrottleAcquired : sc::event<ThrottleAcquired> { + }; private: // internal events struct RequestPrimaryScanning : sc::event<RequestPrimaryScanning> { @@ -136,34 +138,10 @@ public: explicit Crashed(); }; - struct Cancelled : sc::state<Cancelled, BackfillMachine>, - StateHelper<Cancelled> { - using reactions = boost::mpl::list< - sc::custom_reaction<Triggered>, - sc::custom_reaction<PrimaryScanned>, - sc::custom_reaction<ReplicaScanned>, - sc::custom_reaction<ObjectPushed>, - sc::transition<sc::event_base, Crashed>>; - explicit Cancelled(my_context); - // resume after triggering backfill by on_activate_complete(). - // transit to Enqueuing. - sc::result react(const Triggered&); - sc::result react(const PrimaryScanned&) { - return discard_event(); - } - sc::result react(const ReplicaScanned&) { - return discard_event(); - } - sc::result react(const ObjectPushed&) { - return discard_event(); - } - }; - struct Initial : sc::state<Initial, BackfillMachine>, StateHelper<Initial> { using reactions = boost::mpl::list< sc::custom_reaction<Triggered>, - sc::transition<CancelBackfill, Cancelled>, sc::transition<sc::event_base, Crashed>>; explicit Initial(my_context); // initialize after triggering backfill by on_activate_complete(). @@ -174,12 +152,9 @@ public: struct Enqueuing : sc::state<Enqueuing, BackfillMachine>, StateHelper<Enqueuing> { using reactions = boost::mpl::list< - sc::transition<CancelBackfill, Cancelled>, sc::transition<RequestPrimaryScanning, PrimaryScanning>, sc::transition<RequestReplicasScanning, ReplicasScanning>, sc::transition<RequestWaiting, Waiting>, - sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, sc::transition<sc::event_base, Crashed>>; explicit Enqueuing(my_context); @@ -237,12 +212,15 @@ public: sc::custom_reaction<ObjectPushed>, sc::custom_reaction<PrimaryScanned>, sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, + sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, sc::transition<sc::event_base, Crashed>>; explicit PrimaryScanning(my_context); sc::result react(ObjectPushed); // collect scanning result and transit to Enqueuing. sc::result react(PrimaryScanned); + sc::result react(CancelBackfill); + sc::result react(Triggered); }; struct ReplicasScanning : sc::state<ReplicasScanning, BackfillMachine>, @@ -251,6 +229,7 @@ public: sc::custom_reaction<ObjectPushed>, sc::custom_reaction<ReplicaScanned>, sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, sc::transition<RequestDone, Done>, sc::transition<sc::event_base, Crashed>>; explicit ReplicasScanning(my_context); @@ -259,6 +238,7 @@ public: sc::result react(ObjectPushed); sc::result react(ReplicaScanned); sc::result react(CancelBackfill); + sc::result react(Triggered); // indicate whether a particular peer should be scanned to retrieve // BackfillInterval for new range of hobject_t namespace. @@ -277,10 +257,14 @@ public: using reactions = boost::mpl::list< sc::custom_reaction<ObjectPushed>, sc::transition<RequestDone, Done>, - sc::transition<CancelBackfill, Cancelled>, + sc::custom_reaction<CancelBackfill>, + sc::custom_reaction<Triggered>, + sc::transition<ThrottleAcquired, Enqueuing>, sc::transition<sc::event_base, Crashed>>; explicit Waiting(my_context); sc::result react(ObjectPushed); + sc::result react(CancelBackfill); + sc::result react(Triggered); }; struct Done : sc::state<Done, BackfillMachine>, @@ -308,6 +292,11 @@ public: const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers); + void enqueue_standalone_delete( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers); + bool is_triggered() const { return backfill_machine.triggering_event() != nullptr; @@ -325,6 +314,26 @@ public: } } private: + struct backfill_suspend_state_t { + bool suspended = false; + bool should_go_enqueuing = false; + } backfill_suspend_state; + bool is_suspended() const { + return backfill_suspend_state.suspended; + } + void on_suspended() { + ceph_assert(!is_suspended()); + backfill_suspend_state = {true, false}; + } + bool on_resumed() { + auto go_enqueuing = backfill_suspend_state.should_go_enqueuing; + backfill_suspend_state = {false, false}; + return go_enqueuing; + } + void go_enqueuing_on_resume() { + ceph_assert(is_suspended()); + backfill_suspend_state.should_go_enqueuing = true; + } hobject_t last_backfill_started; BackfillInterval backfill_info; std::map<pg_shard_t, BackfillInterval> peer_backfill_info; @@ -405,8 +414,10 @@ struct BackfillState::PGFacade { virtual const eversion_t& get_projected_last_update() const = 0; virtual const PGLog::IndexedLog& get_projected_log() const = 0; + virtual std::ostream &print(std::ostream &out) const = 0; virtual ~PGFacade() {} }; +std::ostream &operator<<(std::ostream &out, const BackfillState::PGFacade &pg); class BackfillState::ProgressTracker { // TODO: apply_stat, @@ -433,6 +444,9 @@ class BackfillState::ProgressTracker { BackfillListener& backfill_listener() { return backfill_machine.backfill_listener; } + PGFacade& pg() { + return *backfill_machine.pg; + } public: ProgressTracker(BackfillMachine& backfill_machine) @@ -447,3 +461,9 @@ public: }; } // namespace crimson::osd + +#if FMT_VERSION >= 90000 +template <> struct fmt::formatter<crimson::osd::BackfillState::PGFacade> + : fmt::ostream_formatter {}; +#endif + diff --git a/src/crimson/osd/heartbeat.cc b/src/crimson/osd/heartbeat.cc index 03986952b4f..5902fc8c14f 100644 --- a/src/crimson/osd/heartbeat.cc +++ b/src/crimson/osd/heartbeat.cc @@ -9,6 +9,7 @@ #include "messages/MOSDPing.h" #include "messages/MOSDFailure.h" +#include "msg/msg_types.h" #include "crimson/common/config_proxy.h" #include "crimson/common/formatter.h" diff --git a/src/crimson/osd/main.cc b/src/crimson/osd/main.cc index fa387804dcd..0bfd3e2266b 100644 --- a/src/crimson/osd/main.cc +++ b/src/crimson/osd/main.cc @@ -24,6 +24,7 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" #include "crimson/common/fatal_signal.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/mon/MonClient.h" #include "crimson/net/Messenger.h" #include "crimson/osd/stop_signal.h" @@ -201,7 +202,7 @@ int main(int argc, const char* argv[]) true); } auto store = crimson::os::FuturizedStore::create( - local_conf().get_val<std::string>("osd_objectstore"), + local_conf().get_val<std::string>("crimson_osd_objectstore"), local_conf().get_val<std::string>("osd_data"), local_conf().get_config_values()); diff --git a/src/crimson/osd/main_config_bootstrap_helpers.cc b/src/crimson/osd/main_config_bootstrap_helpers.cc index 3596929527f..e4920eb870f 100644 --- a/src/crimson/osd/main_config_bootstrap_helpers.cc +++ b/src/crimson/osd/main_config_bootstrap_helpers.cc @@ -17,10 +17,13 @@ #include "crimson/common/buffer_io.h" #include "crimson/common/config_proxy.h" #include "crimson/common/fatal_signal.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/mon/MonClient.h" #include "crimson/net/Messenger.h" #include "crimson/osd/main_config_bootstrap_helpers.h" +#include <sys/wait.h> // for waitpid() + using namespace std::literals; using crimson::common::local_conf; using crimson::common::sharded_conf; diff --git a/src/crimson/osd/object_context.h b/src/crimson/osd/object_context.h index 6f51045931d..4195e5dc597 100644 --- a/src/crimson/osd/object_context.h +++ b/src/crimson/osd/object_context.h @@ -9,6 +9,7 @@ #include <seastar/core/shared_future.hh> #include <seastar/core/shared_ptr.hh> +#include "common/fmt_common.h" #include "common/intrusive_lru.h" #include "osd/object_state.h" #include "crimson/common/exception.h" @@ -73,6 +74,8 @@ public: using watch_key_t = std::pair<uint64_t, entity_name_t>; std::map<watch_key_t, seastar::shared_ptr<crimson::osd::Watch>> watchers; + CommonOBCPipeline obc_pipeline; + ObjectContext(hobject_t hoid) : lock(hoid), obs(std::move(hoid)) {} @@ -128,14 +131,49 @@ public: } bool is_valid() const { - return !invalidated_by_interval_change; + return !invalidated; } private: boost::intrusive::list_member_hook<> obc_accessing_hook; uint64_t list_link_cnt = 0; + + /** + * loading_started + * + * ObjectContext instances may be used for pipeline stages + * prior to actually being loaded. + * + * ObjectContextLoader::load_and_lock* use loading_started + * to determine whether to initiate loading or simply take + * the desired lock directly. + * + * If loading_started is not set, the task must set it and + * (syncronously) take an exclusive lock. That exclusive lock + * must be held until the loading completes, at which point the + * lock may be relaxed or released. + * + * If loading_started is set, it is safe to directly take + * the desired lock, once the lock is obtained loading may + * be assumed to be complete. + * + * loading_started, once set, remains set for the lifetime + * of the object. + */ + bool loading_started = false; + + /// true once set_*_state has been called, used for debugging bool fully_loaded = false; - bool invalidated_by_interval_change = false; + + /** + * invalidated + * + * Set to true upon eviction from cache. This happens to all + * cached obc's upon interval change and to the target of + * a repop received on a replica to ensure that the cached + * state is refreshed upon subsequent replica read. + */ + bool invalidated = false; friend class ObjectContextRegistry; friend class ObjectContextLoader; @@ -156,6 +194,15 @@ public: } } + template <typename FormatContext> + auto fmt_print_ctx(FormatContext & ctx) const { + return fmt::format_to( + ctx.out(), "ObjectContext({}, oid={}, refcount={})", + (void*)this, + get_oid(), + get_use_count()); + } + using obc_accessing_option_t = boost::intrusive::member_hook< ObjectContext, boost::intrusive::list_member_hook<>, @@ -186,12 +233,14 @@ public: void clear_range(const hobject_t &from, const hobject_t &to) { - obc_lru.clear_range(from, to); + obc_lru.clear_range(from, to, [](auto &obc) { + obc.invalidated = true; + }); } void invalidate_on_interval_change() { obc_lru.clear([](auto &obc) { - obc.invalidated_by_interval_change = true; + obc.invalidated = true; }); } diff --git a/src/crimson/osd/object_context_loader.cc b/src/crimson/osd/object_context_loader.cc index 869ca91504c..483251a23b5 100644 --- a/src/crimson/osd/object_context_loader.cc +++ b/src/crimson/osd/object_context_loader.cc @@ -16,19 +16,20 @@ ObjectContextLoader::load_and_lock_head(Manager &manager, RWState::State lock_ty LOG_PREFIX(ObjectContextLoader::load_and_lock_head); DEBUGDPP("{} {}", dpp, manager.target, lock_type); auto releaser = manager.get_releaser(); - // no users pre-populate head_state on this path, so we don't bother to - // handle it ceph_assert(manager.target.is_head()); - ceph_assert(manager.head_state.is_empty()); + + if (manager.head_state.is_empty()) { + auto [obc, _] = obc_registry.get_cached_obc(manager.target); + manager.set_state_obc(manager.head_state, obc); + } ceph_assert(manager.target_state.is_empty()); - auto [obc, existed] = obc_registry.get_cached_obc(manager.target); - manager.set_state_obc(manager.target_state, obc); - manager.set_state_obc(manager.head_state, obc); + manager.set_state_obc(manager.target_state, manager.head_state.obc); - if (existed) { + if (manager.target_state.obc->loading_started) { co_await manager.target_state.lock_to(lock_type); } else { manager.target_state.lock_excl_sync(); + manager.target_state.obc->loading_started = true; co_await load_obc(manager.target_state.obc); manager.target_state.demote_excl_to(lock_type); } @@ -36,7 +37,8 @@ ObjectContextLoader::load_and_lock_head(Manager &manager, RWState::State lock_ty } ObjectContextLoader::load_and_lock_fut -ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_type) +ObjectContextLoader::load_and_lock_clone( + Manager &manager, RWState::State lock_type, bool lock_head) { LOG_PREFIX(ObjectContextLoader::load_and_lock_clone); DEBUGDPP("{} {}", dpp, manager.target, lock_type); @@ -46,16 +48,20 @@ ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_t ceph_assert(manager.target_state.is_empty()); if (manager.head_state.is_empty()) { - auto [obc, existed] = obc_registry.get_cached_obc(manager.target.get_head()); + auto [obc, _] = obc_registry.get_cached_obc(manager.target.get_head()); manager.set_state_obc(manager.head_state, obc); + } - if (existed) { - co_await manager.head_state.lock_to(RWState::RWREAD); - } else { - manager.head_state.lock_excl_sync(); - co_await load_obc(manager.head_state.obc); - manager.head_state.demote_excl_to(RWState::RWREAD); - } + if (!manager.head_state.obc->loading_started) { + // caller is responsible for pre-populating a loaded obc if lock_head is + // false + ceph_assert(lock_head); + manager.head_state.lock_excl_sync(); + manager.head_state.obc->loading_started = true; + co_await load_obc(manager.head_state.obc); + manager.head_state.demote_excl_to(RWState::RWREAD); + } else if (lock_head) { + co_await manager.head_state.lock_to(RWState::RWREAD); } if (manager.options.resolve_clone) { @@ -95,13 +101,14 @@ ObjectContextLoader::load_and_lock_clone(Manager &manager, RWState::State lock_t manager.head_state.state = RWState::RWNONE; } } else { - auto [obc, existed] = obc_registry.get_cached_obc(manager.target); + auto [obc, _] = obc_registry.get_cached_obc(manager.target); manager.set_state_obc(manager.target_state, obc); - if (existed) { + if (manager.target_state.obc->loading_started) { co_await manager.target_state.lock_to(RWState::RWREAD); } else { manager.target_state.lock_excl_sync(); + manager.target_state.obc->loading_started = true; co_await load_obc(manager.target_state.obc); manager.target_state.obc->set_clone_ssc(manager.head_state.obc->ssc); manager.target_state.demote_excl_to(RWState::RWREAD); diff --git a/src/crimson/osd/object_context_loader.h b/src/crimson/osd/object_context_loader.h index 6d007d65176..49f8f1572bf 100644 --- a/src/crimson/osd/object_context_loader.h +++ b/src/crimson/osd/object_context_loader.h @@ -2,9 +2,11 @@ #include <seastar/core/future.hh> #include <seastar/util/defer.hh> +#include "crimson/common/coroutine.h" #include "crimson/common/errorator.h" #include "crimson/common/log.h" #include "crimson/osd/object_context.h" +#include "crimson/osd/osd_operation.h" #include "crimson/osd/pg_backend.h" #include "osd/object_state_fmt.h" @@ -143,9 +145,7 @@ public: if (s.is_empty()) return; s.release_lock(); - SUBDEBUGDPP( - osd, "released object {}, {}", - loader.dpp, s.obc->get_oid(), s.obc->obs); + SUBDEBUGDPP(osd, "releasing obc {}, {}", loader.dpp, *(s.obc), s.obc->obs); s.obc->remove_from(loader.obc_set_accessing); s = state_t(); } @@ -165,11 +165,13 @@ public: ObjectContextRef &get_obc() { ceph_assert(!target_state.is_empty()); + ceph_assert(target_state.obc->is_loaded()); return target_state.obc; } ObjectContextRef &get_head_obc() { ceph_assert(!head_state.is_empty()); + ceph_assert(head_state.obc->is_loaded()); return head_state.obc; } @@ -188,19 +190,49 @@ public: release(); } }; - Manager get_obc_manager(hobject_t oid, bool resolve_clone = true) { + + class Orderer { + friend ObjectContextLoader; + ObjectContextRef orderer_obc; + public: + CommonOBCPipeline &obc_pp() { + ceph_assert(orderer_obc); + return orderer_obc->obc_pipeline; + } + + ~Orderer() { + LOG_PREFIX(ObjectContextLoader::~Orderer); + SUBDEBUG(osd, "releasing obc {}, {}", *(orderer_obc)); + } + }; + + Orderer get_obc_orderer(const hobject_t &oid) { + Orderer ret; + std::tie(ret.orderer_obc, std::ignore) = + obc_registry.get_cached_obc(oid.get_head()); + return ret; + } + + Manager get_obc_manager(const hobject_t &oid, bool resolve_clone = true) { Manager ret(*this, oid); ret.options.resolve_clone = resolve_clone; return ret; } + Manager get_obc_manager( + Orderer &orderer, const hobject_t &oid, bool resolve_clone = true) { + Manager ret = get_obc_manager(oid, resolve_clone); + ret.set_state_obc(ret.head_state, orderer.orderer_obc); + return ret; + } + using load_and_lock_ertr = load_obc_ertr; using load_and_lock_iertr = interruptible::interruptible_errorator< IOInterruptCondition, load_and_lock_ertr>; using load_and_lock_fut = load_and_lock_iertr::future<>; private: load_and_lock_fut load_and_lock_head(Manager &, RWState::State); - load_and_lock_fut load_and_lock_clone(Manager &, RWState::State); + load_and_lock_fut load_and_lock_clone(Manager &, RWState::State, bool lock_head=true); public: load_and_lock_fut load_and_lock(Manager &, RWState::State); @@ -244,7 +276,7 @@ public: // locks on head as part of this call. manager.head_state.obc = head; manager.head_state.obc->append_to(obc_set_accessing); - co_await load_and_lock(manager, State); + co_await load_and_lock_clone(manager, State, false); co_await std::invoke(func, head, manager.get_obc()); } diff --git a/src/crimson/osd/ops_executer.cc b/src/crimson/osd/ops_executer.cc index 97b241fdce4..cbc35c21a04 100644 --- a/src/crimson/osd/ops_executer.cc +++ b/src/crimson/osd/ops_executer.cc @@ -15,12 +15,15 @@ #include <seastar/core/thread.hh> +#include "crimson/common/log.h" #include "crimson/osd/exceptions.h" #include "crimson/osd/pg.h" #include "crimson/osd/watch.h" #include "osd/ClassHandler.h" #include "osd/SnapMapper.h" +SET_SUBSYS(osd); + namespace { seastar::logger& logger() { return crimson::get_logger(ceph_subsys_osd); @@ -464,10 +467,7 @@ auto OpsExecuter::do_const_op(Func&& f) { template <class Func> auto OpsExecuter::do_write_op(Func&& f, OpsExecuter::modified_by m) { ++num_write; - if (!osd_op_params) { - osd_op_params.emplace(); - fill_op_params(m); - } + check_init_op_params(m); return std::forward<Func>(f)(pg->get_backend(), obc->obs, txn); } OpsExecuter::call_errorator::future<> OpsExecuter::do_assert_ver( @@ -822,25 +822,100 @@ OpsExecuter::do_execute_op(OSDOp& osd_op) } } -void OpsExecuter::fill_op_params(OpsExecuter::modified_by m) +OpsExecuter::rep_op_fut_t +OpsExecuter::flush_changes_and_submit( + const std::vector<OSDOp>& ops, + SnapMapper& snap_mapper, + OSDriver& osdriver) { - osd_op_params->req_id = msg->get_reqid(); - osd_op_params->mtime = msg->get_mtime(); - osd_op_params->at_version = pg->get_next_version(); - osd_op_params->pg_trim_to = pg->get_pg_trim_to(); - osd_op_params->pg_committed_to = pg->get_pg_committed_to(); - osd_op_params->last_complete = pg->get_info().last_complete; - osd_op_params->user_modify = (m == modified_by::user); + const bool want_mutate = !txn.empty(); + // osd_op_params are instantiated by every wr-like operation. + assert(osd_op_params || !want_mutate); + assert(obc); + + auto submitted = interruptor::now(); + auto all_completed = interruptor::now(); + + if (cloning_ctx) { + ceph_assert(want_mutate); + } + + apply_stats(); + if (want_mutate) { + osd_op_params->at_version = pg->get_next_version(); + osd_op_params->pg_trim_to = pg->get_pg_trim_to(); + osd_op_params->pg_committed_to = pg->get_pg_committed_to(); + osd_op_params->last_complete = pg->get_info().last_complete; + + std::vector<pg_log_entry_t> log_entries; + + if (cloning_ctx) { + log_entries.emplace_back(complete_cloning_ctx()); + } + + log_entries.emplace_back(prepare_head_update(ops, txn)); + + if (auto log_rit = log_entries.rbegin(); log_rit != log_entries.rend()) { + ceph_assert(log_rit->version == osd_op_params->at_version); + } + + /* + * This works around the gcc bug causing the generated code to incorrectly + * execute unconditionally before the predicate. + * + * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101244 + */ + auto clone_obc = cloning_ctx + ? std::move(cloning_ctx->clone_obc) + : nullptr; + auto [_submitted, _all_completed] = co_await pg->submit_transaction( + std::move(obc), + std::move(clone_obc), + std::move(txn), + std::move(*osd_op_params), + std::move(log_entries) + ); + + submitted = std::move(_submitted); + all_completed = std::move(_all_completed); + } + + if (op_effects.size()) [[unlikely]] { + // need extra ref pg due to apply_stats() which can be executed after + // informing snap mapper + all_completed = + std::move(all_completed).then_interruptible([this, pg=this->pg] { + // let's do the cleaning of `op_effects` in destructor + return interruptor::do_for_each(op_effects, + [pg=std::move(pg)](auto& op_effect) { + return op_effect->execute(pg); + }); + }); + } + + co_return std::make_tuple( + std::move(submitted), + std::move(all_completed)); } -std::vector<pg_log_entry_t> OpsExecuter::prepare_transaction( - const std::vector<OSDOp>& ops) +pg_log_entry_t OpsExecuter::prepare_head_update( + const std::vector<OSDOp>& ops, + ceph::os::Transaction &txn) { - // let's ensure we don't need to inform SnapMapper about this particular - // entry. + LOG_PREFIX(OpsExecuter::prepare_head_update); assert(obc->obs.oi.soid.snap >= CEPH_MAXSNAP); - std::vector<pg_log_entry_t> log_entries; - log_entries.emplace_back( + + update_clone_overlap(); + if (cloning_ctx) { + obc->ssc->snapset = std::move(cloning_ctx->new_snapset); + } + if (snapc.seq > obc->ssc->snapset.seq) { + // update snapset with latest snap context + obc->ssc->snapset.seq = snapc.seq; + obc->ssc->snapset.snaps.clear(); + } + + pg_log_entry_t ret{ obc->obs.exists ? pg_log_entry_t::MODIFY : pg_log_entry_t::DELETE, obc->obs.oi.soid, @@ -849,15 +924,38 @@ std::vector<pg_log_entry_t> OpsExecuter::prepare_transaction( osd_op_params->user_modify ? osd_op_params->at_version.version : 0, osd_op_params->req_id, osd_op_params->mtime, - op_info.allows_returnvec() && !ops.empty() ? ops.back().rval.code : 0); + op_info.allows_returnvec() && !ops.empty() ? ops.back().rval.code : 0}; + if (op_info.allows_returnvec()) { // also the per-op values are recorded in the pg log - log_entries.back().set_op_returns(ops); - logger().debug("{} op_returns: {}", - __func__, log_entries.back().op_returns); + ret.set_op_returns(ops); + DEBUGDPP("op returns: {}", *pg, ret.op_returns); + } + ret.clean_regions = std::move(osd_op_params->clean_regions); + + + if (obc->obs.exists) { + obc->obs.oi.prior_version = obc->obs.oi.version; + obc->obs.oi.version = osd_op_params->at_version; + if (osd_op_params->user_modify) + obc->obs.oi.user_version = osd_op_params->at_version.version; + obc->obs.oi.last_reqid = osd_op_params->req_id; + obc->obs.oi.mtime = osd_op_params->mtime; + obc->obs.oi.local_mtime = ceph_clock_now(); + + obc->ssc->exists = true; + pg->get_backend().set_metadata( + obc->obs.oi.soid, + obc->obs.oi, + obc->obs.oi.soid.is_head() ? &(obc->ssc->snapset) : nullptr, + txn); + } else { + // reset cached ObjectState without enforcing eviction + obc->obs.oi = object_info_t(obc->obs.oi.soid); } - log_entries.back().clean_regions = std::move(osd_op_params->clean_regions); - return log_entries; + + DEBUGDPP("entry: {}", *pg, ret); + return ret; } // Defined here because there is a circular dependency between OpsExecuter and PG @@ -871,25 +969,26 @@ version_t OpsExecuter::get_last_user_version() const return pg->get_last_user_version(); } -std::unique_ptr<OpsExecuter::CloningContext> OpsExecuter::execute_clone( +void OpsExecuter::prepare_cloning_ctx( const SnapContext& snapc, const ObjectState& initial_obs, const SnapSet& initial_snapset, PGBackend& backend, ceph::os::Transaction& txn) { + LOG_PREFIX(OpsExecuter::prepare_cloning_ctx); const hobject_t& soid = initial_obs.oi.soid; logger().debug("{} {} snapset={} snapc={}", __func__, soid, initial_snapset, snapc); - auto cloning_ctx = std::make_unique<CloningContext>(); + cloning_ctx = std::make_unique<CloningContext>(); cloning_ctx->new_snapset = initial_snapset; // clone object, the snap field is set to the seq of the SnapContext // at its creation. - hobject_t coid = soid; - coid.snap = snapc.seq; + cloning_ctx->coid = soid; + cloning_ctx->coid.snap = snapc.seq; // existing snaps are stored in descending order in snapc, // cloned_snaps vector will hold all the snaps stored until snapset.seq @@ -900,49 +999,63 @@ std::unique_ptr<OpsExecuter::CloningContext> OpsExecuter::execute_clone( return std::vector<snapid_t>{std::begin(snapc.snaps), last}; }(); - auto clone_obc = prepare_clone(coid, osd_op_params->at_version); - osd_op_params->at_version.version++; + // make clone here, but populate in metadata in complete_cloning_ctx + backend.clone_for_write(soid, cloning_ctx->coid, txn); - // make clone - backend.clone(clone_obc->obs.oi, initial_obs, clone_obc->obs, txn); + cloning_ctx->clone_obc = prepare_clone(cloning_ctx->coid, initial_obs); delta_stats.num_objects++; - if (clone_obc->obs.oi.is_omap()) { + if (cloning_ctx->clone_obc->obs.oi.is_omap()) { delta_stats.num_objects_omap++; } delta_stats.num_object_clones++; // newsnapset is obc's ssc - cloning_ctx->new_snapset.clones.push_back(coid.snap); - cloning_ctx->new_snapset.clone_size[coid.snap] = initial_obs.oi.size; - cloning_ctx->new_snapset.clone_snaps[coid.snap] = cloned_snaps; + cloning_ctx->new_snapset.clones.push_back(cloning_ctx->coid.snap); + cloning_ctx->new_snapset.clone_size[cloning_ctx->coid.snap] = initial_obs.oi.size; + cloning_ctx->new_snapset.clone_snaps[cloning_ctx->coid.snap] = cloned_snaps; // clone_overlap should contain an entry for each clone // (an empty interval_set if there is no overlap) - auto &overlap = cloning_ctx->new_snapset.clone_overlap[coid.snap]; + auto &overlap = cloning_ctx->new_snapset.clone_overlap[cloning_ctx->coid.snap]; if (initial_obs.oi.size) { overlap.insert(0, initial_obs.oi.size); } // log clone - logger().debug("cloning v {} to {} v {} snaps={} snapset={}", - initial_obs.oi.version, coid, - osd_op_params->at_version, cloned_snaps, cloning_ctx->new_snapset); + DEBUGDPP("cloning v {} to {} v {} snaps={} snapset={}", *pg, + initial_obs.oi.version, cloning_ctx->coid, + osd_op_params->at_version, cloned_snaps, cloning_ctx->new_snapset); +} - cloning_ctx->log_entry = { +pg_log_entry_t OpsExecuter::complete_cloning_ctx() +{ + ceph_assert(cloning_ctx); + const auto &coid = cloning_ctx->coid; + cloning_ctx->clone_obc->obs.oi.version = osd_op_params->at_version; + + osd_op_params->at_version.version++; + + pg->get_backend().set_metadata( + cloning_ctx->coid, + cloning_ctx->clone_obc->obs.oi, + nullptr /* snapset */, + txn); + + pg_log_entry_t ret{ pg_log_entry_t::CLONE, coid, - clone_obc->obs.oi.version, - clone_obc->obs.oi.prior_version, - clone_obc->obs.oi.user_version, + cloning_ctx->clone_obc->obs.oi.version, + cloning_ctx->clone_obc->obs.oi.prior_version, + cloning_ctx->clone_obc->obs.oi.user_version, osd_reqid_t(), - clone_obc->obs.oi.mtime, // will be replaced in `apply_to()` + cloning_ctx->clone_obc->obs.oi.mtime, // will be replaced in `apply_to()` 0 }; - encode(cloned_snaps, cloning_ctx->log_entry.snaps); - cloning_ctx->log_entry.clean_regions.mark_data_region_dirty(0, initial_obs.oi.size); - cloning_ctx->clone_obc = clone_obc; - - return cloning_ctx; + ceph_assert(cloning_ctx->new_snapset.clone_snaps.count(coid.snap)); + encode(cloning_ctx->new_snapset.clone_snaps[coid.snap], ret.snaps); + ret.clean_regions.mark_data_region_dirty(0, cloning_ctx->clone_obc->obs.oi.size); + ret.mtime = cloning_ctx->clone_obc->obs.oi.mtime; + return ret; } void OpsExecuter::update_clone_overlap() { @@ -965,47 +1078,16 @@ void OpsExecuter::update_clone_overlap() { delta_stats.num_bytes += osd_op_params->modified_ranges.size(); } -void OpsExecuter::CloningContext::apply_to( - std::vector<pg_log_entry_t>& log_entries, - ObjectContext& processed_obc) -{ - log_entry.mtime = processed_obc.obs.oi.mtime; - log_entries.insert(log_entries.begin(), std::move(log_entry)); - processed_obc.ssc->snapset = std::move(new_snapset); -} - -std::vector<pg_log_entry_t> -OpsExecuter::flush_clone_metadata( - std::vector<pg_log_entry_t>&& log_entries, - SnapMapper& snap_mapper, - OSDriver& osdriver, - ceph::os::Transaction& txn) -{ - assert(!txn.empty()); - update_clone_overlap(); - if (cloning_ctx) { - cloning_ctx->apply_to(log_entries, *obc); - } - if (snapc.seq > obc->ssc->snapset.seq) { - // update snapset with latest snap context - obc->ssc->snapset.seq = snapc.seq; - obc->ssc->snapset.snaps.clear(); - } - logger().debug("{} done, initial snapset={}, new snapset={}", - __func__, obc->obs.oi.soid, obc->ssc->snapset); - return std::move(log_entries); -} - ObjectContextRef OpsExecuter::prepare_clone( const hobject_t& coid, - eversion_t version) + const ObjectState& initial_obs) { ceph_assert(pg->is_primary()); ObjectState clone_obs{coid}; clone_obs.exists = true; - clone_obs.oi.version = version; - clone_obs.oi.prior_version = obc->obs.oi.version; - clone_obs.oi.copy_user_bits(obc->obs.oi); + // clone_obs.oi.version will be populated in complete_cloning_ctx + clone_obs.oi.prior_version = initial_obs.oi.version; + clone_obs.oi.copy_user_bits(initial_obs.oi); clone_obs.oi.clear_flag(object_info_t::FLAG_WHITEOUT); auto [clone_obc, existed] = pg->obc_registry.get_cached_obc(std::move(coid)); @@ -1036,11 +1118,12 @@ OpsExecuter::OpsExecuter(Ref<PG> pg, { if (op_info.may_write() && should_clone(*obc, snapc)) { do_write_op([this](auto& backend, auto& os, auto& txn) { - cloning_ctx = execute_clone(std::as_const(snapc), - std::as_const(obc->obs), - std::as_const(obc->ssc->snapset), - backend, - txn); + prepare_cloning_ctx( + std::as_const(snapc), + std::as_const(obc->obs), + std::as_const(obc->ssc->snapset), + backend, + txn); }); } } diff --git a/src/crimson/osd/ops_executer.h b/src/crimson/osd/ops_executer.h index 94b64ccebb1..f5554bd6919 100644 --- a/src/crimson/osd/ops_executer.h +++ b/src/crimson/osd/ops_executer.h @@ -195,26 +195,26 @@ private: SnapContext snapc; // writer snap context struct CloningContext { + /// id of new clone, populated in prepare_cloning_ctx + hobject_t coid; + /// new snapset, populated in prepare_cloning_ctx SnapSet new_snapset; - pg_log_entry_t log_entry; + /// populated in complete_cloning_ctx ObjectContextRef clone_obc; - - void apply_to( - std::vector<pg_log_entry_t>& log_entries, - ObjectContext& processed_obc); }; std::unique_ptr<CloningContext> cloning_ctx; - /** - * execute_clone + * prepare_cloning_ctx * * If snapc contains a snap which occurred logically after the last write * seen by this object (see OpsExecuter::should_clone()), we first need - * make a clone of the object at its current state. execute_clone primes - * txn with that clone operation and returns an - * OpsExecuter::CloningContext which will allow us to fill in the corresponding - * metadata and log_entries once the operations have been processed. + * make a clone of the object at its current state. prepare_cloning_ctx + * primes txn with that clone operation and populates cloning_ctx with + * an obc for the clone and a new snapset reflecting the clone. + * + * complete_cloning_ctx later uses the information from cloning_ctx to + * generate a log entry and object_info versions for the clone. * * Note that this strategy differs from classic, which instead performs this * work at the end and reorders the transaction. See @@ -227,13 +227,15 @@ private: * @param backend [in,out] interface for generating mutations * @param txn [out] transaction for the operation */ - std::unique_ptr<CloningContext> execute_clone( + void prepare_cloning_ctx( const SnapContext& snapc, const ObjectState& initial_obs, const SnapSet& initial_snapset, PGBackend& backend, ceph::os::Transaction& txn); + /// complete clone, populate clone_obc, return log entry + pg_log_entry_t complete_cloning_ctx(); /** * should_clone @@ -264,12 +266,6 @@ private: */ void update_clone_overlap(); - std::vector<pg_log_entry_t> flush_clone_metadata( - std::vector<pg_log_entry_t>&& log_entries, - SnapMapper& snap_mapper, - OSDriver& osdriver, - ceph::os::Transaction& txn); - private: // this gizmo could be wrapped in std::optional for the sake of lazy // initialization. we don't need it for ops that doesn't have effect @@ -400,15 +396,22 @@ public: std::tuple<interruptible_future<>, interruptible_future<>>; using rep_op_fut_t = interruptible_future<rep_op_fut_tuple>; - template <typename MutFunc> - rep_op_fut_t flush_changes_n_do_ops_effects( + rep_op_fut_t flush_changes_and_submit( const std::vector<OSDOp>& ops, SnapMapper& snap_mapper, - OSDriver& osdriver, - MutFunc mut_func) &&; - std::vector<pg_log_entry_t> prepare_transaction( - const std::vector<OSDOp>& ops); - void fill_op_params(modified_by m); + OSDriver& osdriver); + pg_log_entry_t prepare_head_update( + const std::vector<OSDOp>& ops, + ceph::os::Transaction &txn); + + void check_init_op_params(OpsExecuter::modified_by m) { + if (!osd_op_params) { + osd_op_params.emplace(); + osd_op_params->req_id = msg->get_reqid(); + osd_op_params->mtime = msg->get_mtime(); + osd_op_params->user_modify = (m == modified_by::user); + } + } ObjectContextRef get_obc() const { return obc; @@ -443,7 +446,7 @@ public: ObjectContextRef prepare_clone( const hobject_t& coid, - eversion_t version); + const ObjectState& initial_obs); void apply_stats(); }; @@ -485,69 +488,6 @@ auto OpsExecuter::with_effect_on_obc( return std::forward<MainFunc>(main_func)(ctx_ref); } -template <typename MutFunc> -OpsExecuter::rep_op_fut_t -OpsExecuter::flush_changes_n_do_ops_effects( - const std::vector<OSDOp>& ops, - SnapMapper& snap_mapper, - OSDriver& osdriver, - MutFunc mut_func) && -{ - const bool want_mutate = !txn.empty(); - // osd_op_params are instantiated by every wr-like operation. - assert(osd_op_params || !want_mutate); - assert(obc); - - auto submitted = interruptor::now(); - auto all_completed = interruptor::now(); - - if (cloning_ctx) { - ceph_assert(want_mutate); - } - - apply_stats(); - if (want_mutate) { - auto log_entries = flush_clone_metadata( - prepare_transaction(ops), - snap_mapper, - osdriver, - txn); - - if (auto log_rit = log_entries.rbegin(); log_rit != log_entries.rend()) { - ceph_assert(log_rit->version == osd_op_params->at_version); - } - - auto [_submitted, _all_completed] = co_await mut_func( - std::move(txn), - std::move(obc), - std::move(*osd_op_params), - std::move(log_entries), - cloning_ctx - ? std::move(cloning_ctx->clone_obc) - : nullptr); - - submitted = std::move(_submitted); - all_completed = std::move(_all_completed); - } - - if (op_effects.size()) [[unlikely]] { - // need extra ref pg due to apply_stats() which can be executed after - // informing snap mapper - all_completed = - std::move(all_completed).then_interruptible([this, pg=this->pg] { - // let's do the cleaning of `op_effects` in destructor - return interruptor::do_for_each(op_effects, - [pg=std::move(pg)](auto& op_effect) { - return op_effect->execute(pg); - }); - }); - } - - co_return std::make_tuple( - std::move(submitted), - std::move(all_completed)); -} - template <class Func> struct OpsExecuter::RollbackHelper { void rollback_obc_if_modified(); diff --git a/src/crimson/osd/osd_operation.h b/src/crimson/osd/osd_operation.h index fd8b049c0bf..394375c1129 100644 --- a/src/crimson/osd/osd_operation.h +++ b/src/crimson/osd/osd_operation.h @@ -50,24 +50,36 @@ struct PGPeeringPipeline { }; struct CommonPGPipeline { - struct WaitForActive : OrderedExclusivePhaseT<WaitForActive> { - static constexpr auto type_name = "CommonPGPipeline:::wait_for_active"; - } wait_for_active; - struct RecoverMissing : OrderedConcurrentPhaseT<RecoverMissing> { - static constexpr auto type_name = "CommonPGPipeline::recover_missing"; - } recover_missing; - struct CheckAlreadyCompleteGetObc : OrderedExclusivePhaseT<CheckAlreadyCompleteGetObc> { - static constexpr auto type_name = "CommonPGPipeline::check_already_complete_get_obc"; - } check_already_complete_get_obc; - struct LockOBC : OrderedConcurrentPhaseT<LockOBC> { - static constexpr auto type_name = "CommonPGPipeline::lock_obc"; - } lock_obc; + struct WaitPGReady : OrderedConcurrentPhaseT<WaitPGReady> { + static constexpr auto type_name = "CommonPGPipeline:::wait_pg_ready"; + } wait_pg_ready; + struct GetOBC : OrderedExclusivePhaseT<GetOBC> { + static constexpr auto type_name = "CommonPGPipeline:::get_obc"; + } get_obc; +}; + +struct PGRepopPipeline { + struct Process : OrderedExclusivePhaseT<Process> { + static constexpr auto type_name = "PGRepopPipeline::process"; + } process; + struct WaitCommit : OrderedConcurrentPhaseT<WaitCommit> { + static constexpr auto type_name = "PGRepopPipeline::wait_repop"; + } wait_commit; + struct SendReply : OrderedExclusivePhaseT<SendReply> { + static constexpr auto type_name = "PGRepopPipeline::send_reply"; + } send_reply; +}; + +struct CommonOBCPipeline { struct Process : OrderedExclusivePhaseT<Process> { - static constexpr auto type_name = "CommonPGPipeline::process"; + static constexpr auto type_name = "CommonOBCPipeline::process"; } process; struct WaitRepop : OrderedConcurrentPhaseT<WaitRepop> { - static constexpr auto type_name = "ClientRequest::PGPipeline::wait_repop"; + static constexpr auto type_name = "CommonOBCPipeline::wait_repop"; } wait_repop; + struct SendReply : OrderedExclusivePhaseT<SendReply> { + static constexpr auto type_name = "CommonOBCPipeline::send_reply"; + } send_reply; }; @@ -205,6 +217,9 @@ protected: public: static constexpr bool is_trackable = true; + virtual bool requires_pg() const { + return true; + } }; template <class T> @@ -326,6 +341,18 @@ public: with_throttle_while(std::forward<Args>(args)...), *this); } + // Returns std::nullopt if the throttle is acquired immediately, + // returns the future for the acquiring otherwise + std::optional<seastar::future<>> + try_acquire_throttle_now(crimson::osd::scheduler::params_t params) { + if (!max_in_progress || in_progress < max_in_progress) { + ++in_progress; + --pending; + return std::nullopt; + } + return acquire_throttle(params); + } + private: void dump_detail(Formatter *f) const final; diff --git a/src/crimson/osd/osd_operation_external_tracking.h b/src/crimson/osd/osd_operation_external_tracking.h index d2786a95e4d..6a2d7e3ccbd 100644 --- a/src/crimson/osd/osd_operation_external_tracking.h +++ b/src/crimson/osd/osd_operation_external_tracking.h @@ -25,24 +25,23 @@ struct LttngBackend ConnectionPipeline::AwaitMap::BlockingEvent::Backend, ConnectionPipeline::GetPGMapping::BlockingEvent::Backend, PerShardPipeline::CreateOrWaitPG::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent::Backend, + CommonPGPipeline::GetOBC::BlockingEvent::Backend, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, PGMap::PGCreationBlockingEvent::Backend, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent::Backend, PG_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitForActive::BlockingEvent::Backend, PGActivationBlocker::BlockingEvent::Backend, scrub::PGScrubber::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing:: - BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::Process::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::SendReply::BlockingEvent::Backend, - ClientRequest::CompletionEvent::Backend + ClientRequest::CompletionEvent::Backend, + CommonOBCPipeline::Process::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, + CommonOBCPipeline::SendReply::BlockingEvent::Backend, + PGRepopPipeline::Process::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent::Backend, + PGRepopPipeline::SendReply::BlockingEvent::Backend { void handle(ClientRequest::StartEvent&, const Operation&) override {} @@ -72,24 +71,28 @@ struct LttngBackend const PerShardPipeline::CreateOrWaitPG& blocker) override { } - void handle(PGMap::PGCreationBlockingEvent&, - const Operation&, - const PGMap::PGCreationBlocker&) override { + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent& ev, + const Operation& op, + const CommonPGPipeline::WaitPGReady& blocker) override { } - void handle(ClientRequest::PGPipeline::AwaitMap::BlockingEvent& ev, + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { + } + + void handle(CommonPGPipeline::GetOBC::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::AwaitMap& blocker) override { + const CommonPGPipeline::GetOBC& blocker) override { } - void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + void handle(PGMap::PGCreationBlockingEvent&, const Operation&, - const PG_OSDMapGate::OSDMapBlocker&) override { + const PGMap::PGCreationBlocker&) override { } - void handle(ClientRequest::PGPipeline::WaitForActive::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::WaitForActive& blocker) override { + void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + const Operation&, + const PG_OSDMapGate::OSDMapBlocker&) override { } void handle(PGActivationBlocker::BlockingEvent& ev, @@ -102,51 +105,47 @@ struct LttngBackend const scrub::PGScrubber& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent& ev, + void handle(CommonOBCPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::RecoverMissing& blocker) override { + const CommonOBCPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { - } - - void handle(ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc& blocker) override { + const CommonOBCPipeline::WaitRepop& blocker) override { } - - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::LockOBC& blocker) override { + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { + void handle(CommonOBCPipeline::SendReply::BlockingEvent& ev, + const Operation& op, + const CommonOBCPipeline::SendReply& blocker) override { } - void handle(ClientRequest::PGPipeline::Process::BlockingEvent& ev, + void handle(PGRepopPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::Process& blocker) override { + const PGRepopPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::WaitRepop& blocker) override { + const PGRepopPipeline::WaitCommit& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::SendReply::BlockingEvent& ev, + void handle(PGRepopPipeline::SendReply::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::SendReply& blocker) override { + const PGRepopPipeline::SendReply& blocker) override { } void handle(ClientRequest::CompletionEvent&, const Operation&) override {} + }; struct HistoricBackend @@ -155,24 +154,23 @@ struct HistoricBackend ConnectionPipeline::AwaitMap::BlockingEvent::Backend, ConnectionPipeline::GetPGMapping::BlockingEvent::Backend, PerShardPipeline::CreateOrWaitPG::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::Backend, + CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent::Backend, + CommonPGPipeline::GetOBC::BlockingEvent::Backend, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, PGMap::PGCreationBlockingEvent::Backend, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent::Backend, PG_OSDMapGate::OSDMapBlocker::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitForActive::BlockingEvent::Backend, PGActivationBlocker::BlockingEvent::Backend, scrub::PGScrubber::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::Backend, - ClientRequest::PGPipeline::RecoverMissing:: - BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::Backend, - ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::Process::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::Backend, - ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, - ClientRequest::PGPipeline::SendReply::BlockingEvent::Backend, - ClientRequest::CompletionEvent::Backend + ClientRequest::CompletionEvent::Backend, + CommonOBCPipeline::Process::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::Backend, + CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent::Backend, + CommonOBCPipeline::SendReply::BlockingEvent::Backend, + PGRepopPipeline::Process::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::Backend, + PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent::Backend, + PGRepopPipeline::SendReply::BlockingEvent::Backend { void handle(ClientRequest::StartEvent&, const Operation&) override {} @@ -202,24 +200,28 @@ struct HistoricBackend const PerShardPipeline::CreateOrWaitPG& blocker) override { } - void handle(PGMap::PGCreationBlockingEvent&, - const Operation&, - const PGMap::PGCreationBlocker&) override { + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent& ev, + const Operation& op, + const CommonPGPipeline::WaitPGReady& blocker) override { + } + + void handle(CommonPGPipeline::WaitPGReady::BlockingEvent::ExitBarrierEvent& ev, + const Operation& op) override { } - void handle(ClientRequest::PGPipeline::AwaitMap::BlockingEvent& ev, + void handle(CommonPGPipeline::GetOBC::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::AwaitMap& blocker) override { + const CommonPGPipeline::GetOBC& blocker) override { } - void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + void handle(PGMap::PGCreationBlockingEvent&, const Operation&, - const PG_OSDMapGate::OSDMapBlocker&) override { + const PGMap::PGCreationBlocker&) override { } - void handle(ClientRequest::PGPipeline::WaitForActive::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::WaitForActive& blocker) override { + void handle(PG_OSDMapGate::OSDMapBlocker::BlockingEvent&, + const Operation&, + const PG_OSDMapGate::OSDMapBlocker&) override { } void handle(PGActivationBlocker::BlockingEvent& ev, @@ -232,55 +234,52 @@ struct HistoricBackend const scrub::PGScrubber& blocker) override { } - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent& ev, - const Operation& op, - const ClientRequest::PGPipeline::RecoverMissing& blocker) override { - } - - void handle(ClientRequest::PGPipeline::RecoverMissing::BlockingEvent::ExitBarrierEvent& ev, - const Operation& op) override { + static const ClientRequest& to_client_request(const Operation& op) { +#ifdef NDEBUG + return static_cast<const ClientRequest&>(op); +#else + return dynamic_cast<const ClientRequest&>(op); +#endif } - void handle(ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent& ev, + void handle(CommonOBCPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::CheckAlreadyCompleteGetObc& blocker) override { + const CommonOBCPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::LockOBC& blocker) override { + const CommonOBCPipeline::WaitRepop& blocker) override { } - void handle(ClientRequest::PGPipeline::LockOBC::BlockingEvent::ExitBarrierEvent& ev, + void handle(CommonOBCPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::Process::BlockingEvent& ev, + void handle(CommonOBCPipeline::SendReply::BlockingEvent& ev, + const Operation& op, + const CommonOBCPipeline::SendReply& blocker) override { + } + + void handle(PGRepopPipeline::Process::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::Process& blocker) override { + const PGRepopPipeline::Process& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::WaitRepop& blocker) override { + const PGRepopPipeline::WaitCommit& blocker) override { } - void handle(ClientRequest::PGPipeline::WaitRepop::BlockingEvent::ExitBarrierEvent& ev, + void handle(PGRepopPipeline::WaitCommit::BlockingEvent::ExitBarrierEvent& ev, const Operation& op) override { } - void handle(ClientRequest::PGPipeline::SendReply::BlockingEvent& ev, + void handle(PGRepopPipeline::SendReply::BlockingEvent& ev, const Operation& op, - const ClientRequest::PGPipeline::SendReply& blocker) override { + const PGRepopPipeline::SendReply& blocker) override { } - static const ClientRequest& to_client_request(const Operation& op) { -#ifdef NDEBUG - return static_cast<const ClientRequest&>(op); -#else - return dynamic_cast<const ClientRequest&>(op); -#endif - } void handle(ClientRequest::CompletionEvent&, const Operation& op) override { if (crimson::common::local_conf()->osd_op_history_size) { diff --git a/src/crimson/osd/osd_operations/client_request.cc b/src/crimson/osd/osd_operations/client_request.cc index 61a56600a57..fcd0f318db2 100644 --- a/src/crimson/osd/osd_operations/client_request.cc +++ b/src/crimson/osd/osd_operations/client_request.cc @@ -101,7 +101,7 @@ PerShardPipeline &ClientRequest::get_pershard_pipeline( return shard_services.get_client_request_pipeline(); } -ClientRequest::PGPipeline &ClientRequest::client_pp(PG &pg) +CommonPGPipeline &ClientRequest::client_pp(PG &pg) { return pg.request_pg_pipeline; } @@ -140,6 +140,15 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{} start", *pgref, *this); PG &pg = *pgref; + + DEBUGDPP("{}.{}: entering wait_pg_ready stage", + *pgref, *this, this_instance_id); + // The prior stage is OrderedExclusive (PerShardPipeline::create_or_wait_pg) + // and wait_pg_ready is OrderedConcurrent. This transition, therefore, cannot + // block and using enter_stage_sync is legal and more efficient than + // enter_stage. + ihref.enter_stage_sync(client_pp(pg).wait_pg_ready, *this); + if (!m->get_hobj().get_key().empty()) { // There are no users of locator. It was used to ensure that multipart-upload // parts would end up in the same PG so that they could be clone_range'd into @@ -156,28 +165,22 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{}: discarding {}", *pgref, *this, this_instance_id); co_return; } - DEBUGDPP("{}.{}: entering await_map stage", - *pgref, *this, this_instance_id); - co_await ihref.enter_stage<interruptor>(client_pp(pg).await_map, *this); - DEBUGDPP("{}.{}: entered await_map stage, waiting for map", - pg, *this, this_instance_id); + auto map_epoch = co_await interruptor::make_interruptible( ihref.enter_blocker( *this, pg.osdmap_gate, &decltype(pg.osdmap_gate)::wait_for_map, m->get_min_epoch(), nullptr)); - DEBUGDPP("{}.{}: map epoch got {}, entering wait_for_active", + DEBUGDPP("{}.{}: waited for epoch {}, waiting for active", pg, *this, this_instance_id, map_epoch); - co_await ihref.enter_stage<interruptor>(client_pp(pg).wait_for_active, *this); - - DEBUGDPP("{}.{}: entered wait_for_active stage, waiting for active", - pg, *this, this_instance_id); co_await interruptor::make_interruptible( ihref.enter_blocker( *this, pg.wait_for_active_blocker, &decltype(pg.wait_for_active_blocker)::wait)); + co_await ihref.enter_stage<interruptor>(client_pp(pg).get_obc, *this); + if (int res = op_info.set_from_op(&*m, *pg.get_osdmap()); res != 0) { co_await reply_op_error(pgref, res); @@ -238,10 +241,6 @@ ClientRequest::interruptible_future<> ClientRequest::with_pg_process_interruptib DEBUGDPP("{}.{}: process[_pg]_op complete, completing handle", *pgref, *this, this_instance_id); co_await interruptor::make_interruptible(ihref.handle.complete()); - - DEBUGDPP("{}.{}: process[_pg]_op complete," - "removing request from orderer", - *pgref, *this, this_instance_id); } seastar::future<> ClientRequest::with_pg_process( @@ -257,10 +256,13 @@ seastar::future<> ClientRequest::with_pg_process( auto instance_handle = get_instance_handle(); auto &ihref = *instance_handle; return interruptor::with_interruption( - [this, pgref, this_instance_id, &ihref]() mutable { + [FNAME, this, pgref, this_instance_id, &ihref]() mutable { return with_pg_process_interruptible( pgref, this_instance_id, ihref - ).then_interruptible([this, pgref] { + ).then_interruptible([FNAME, this, this_instance_id, pgref] { + DEBUGDPP("{}.{}: with_pg_process_interruptible complete," + " completing request", + *pgref, *this, this_instance_id); complete_request(*pgref); }); }, [FNAME, this, this_instance_id, pgref](std::exception_ptr eptr) { @@ -268,9 +270,10 @@ seastar::future<> ClientRequest::with_pg_process( *pgref, *this, this_instance_id, eptr); }, pgref, pgref->get_osdmap_epoch()).finally( [this, FNAME, opref=std::move(opref), pgref, - this_instance_id, instance_handle=std::move(instance_handle), &ihref] { + this_instance_id, instance_handle=std::move(instance_handle), &ihref]() mutable { DEBUGDPP("{}.{}: exit", *pgref, *this, this_instance_id); - ihref.handle.exit(); + return ihref.handle.complete( + ).finally([instance_handle=std::move(instance_handle)] {}); }); } @@ -345,7 +348,13 @@ ClientRequest::process_op( instance_handle_t &ihref, Ref<PG> pg, unsigned this_instance_id) { LOG_PREFIX(ClientRequest::process_op); - ihref.enter_stage_sync(client_pp(*pg).recover_missing, *this); + ihref.obc_orderer = pg->obc_loader.get_obc_orderer(m->get_hobj()); + auto obc_manager = pg->obc_loader.get_obc_manager( + *(ihref.obc_orderer), + m->get_hobj()); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().process, *this); + if (!pg->is_primary()) { DEBUGDPP( "Skipping recover_missings on non primary pg for soid {}", @@ -365,16 +374,6 @@ ClientRequest::process_op( } } - /** - * The previous stage of recover_missing is a concurrent phase. - * Checking for already_complete requests must done exclusively. - * Since get_obc is also an exclusive stage, we can merge both stages into - * a single stage and avoid stage switching overhead. - */ - DEBUGDPP("{}.{}: entering check_already_complete_get_obc", - *pg, *this, this_instance_id); - co_await ihref.enter_stage<interruptor>( - client_pp(*pg).check_already_complete_get_obc, *this); DEBUGDPP("{}.{}: checking already_complete", *pg, *this, this_instance_id); auto completed = co_await pg->already_complete(m->get_reqid()); @@ -402,12 +401,6 @@ ClientRequest::process_op( DEBUGDPP("{}.{}: past scrub blocker, getting obc", *pg, *this, this_instance_id); - auto obc_manager = pg->obc_loader.get_obc_manager(m->get_hobj()); - - // initiate load_and_lock in order, but wait concurrently - ihref.enter_stage_sync( - client_pp(*pg).lock_obc, *this); - int load_err = co_await pg->obc_loader.load_and_lock( obc_manager, pg->get_lock_type(op_info) ).si_then([]() -> int { @@ -425,13 +418,8 @@ ClientRequest::process_op( co_return; } - DEBUGDPP("{}.{}: got obc {}, entering process stage", + DEBUGDPP("{}.{}: obc {} loaded and locked, calling do_process", *pg, *this, this_instance_id, obc_manager.get_obc()->obs); - co_await ihref.enter_stage<interruptor>( - client_pp(*pg).process, *this); - - DEBUGDPP("{}.{}: in process stage, calling do_process", - *pg, *this, this_instance_id); co_await do_process( ihref, pg, obc_manager.get_obc(), this_instance_id ); @@ -553,12 +541,14 @@ ClientRequest::do_process( std::move(ox), m->ops); co_await std::move(submitted); } - co_await ihref.enter_stage<interruptor>(client_pp(*pg).wait_repop, *this); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().wait_repop, *this); co_await std::move(all_completed); } - co_await ihref.enter_stage<interruptor>(client_pp(*pg).send_reply, *this); + co_await ihref.enter_stage<interruptor>( + ihref.obc_orderer->obc_pp().send_reply, *this); if (ret) { int err = -ret->value(); diff --git a/src/crimson/osd/osd_operations/client_request.h b/src/crimson/osd/osd_operations/client_request.h index 6d1043e2783..91a6728fd4b 100644 --- a/src/crimson/osd/osd_operations/client_request.h +++ b/src/crimson/osd/osd_operations/client_request.h @@ -11,6 +11,7 @@ #include "osd/osd_op_util.h" #include "crimson/net/Connection.h" #include "crimson/osd/object_context.h" +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osdmap_gate.h" #include "crimson/osd/osd_operation.h" #include "crimson/osd/osd_operations/client_request_common.h" @@ -41,21 +42,9 @@ class ClientRequest final : public PhasedOperationT<ClientRequest>, unsigned instance_id = 0; public: - class PGPipeline : public CommonPGPipeline { - public: - struct AwaitMap : OrderedExclusivePhaseT<AwaitMap> { - static constexpr auto type_name = "ClientRequest::PGPipeline::await_map"; - } await_map; - struct SendReply : OrderedExclusivePhaseT<SendReply> { - static constexpr auto type_name = "ClientRequest::PGPipeline::send_reply"; - } send_reply; - friend class ClientRequest; - friend class LttngBackend; - friend class HistoricBackend; - friend class ReqRequest; - friend class LogMissingRequest; - friend class LogMissingRequestReply; - }; + epoch_t get_epoch_sent_at() const { + return m->get_map_epoch(); + } /** * instance_handle_t @@ -93,20 +82,18 @@ public: // don't leave any references on the source core, so we just bypass it by using // intrusive_ptr instead. using ref_t = boost::intrusive_ptr<instance_handle_t>; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; std::tuple< - PGPipeline::AwaitMap::BlockingEvent, + CommonPGPipeline::WaitPGReady::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, - PGPipeline::WaitForActive::BlockingEvent, PGActivationBlocker::BlockingEvent, - PGPipeline::RecoverMissing::BlockingEvent, + CommonPGPipeline::GetOBC::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, scrub::PGScrubber::BlockingEvent, - PGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - PGPipeline::LockOBC::BlockingEvent, - PGPipeline::Process::BlockingEvent, - PGPipeline::WaitRepop::BlockingEvent, - PGPipeline::SendReply::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, + CommonOBCPipeline::SendReply::BlockingEvent, CompletionEvent > pg_tracking_events; @@ -293,7 +280,7 @@ private: unsigned this_instance_id); bool is_pg_op() const; - PGPipeline &client_pp(PG &pg); + CommonPGPipeline &client_pp(PG &pg); template <typename Errorator> using interruptible_errorator = diff --git a/src/crimson/osd/osd_operations/internal_client_request.cc b/src/crimson/osd/osd_operations/internal_client_request.cc index 4790025065a..b8f7646bc74 100644 --- a/src/crimson/osd/osd_operations/internal_client_request.cc +++ b/src/crimson/osd/osd_operations/internal_client_request.cc @@ -52,39 +52,17 @@ CommonPGPipeline& InternalClientRequest::client_pp() } InternalClientRequest::interruptible_future<> -InternalClientRequest::do_process( - crimson::osd::ObjectContextRef obc, - std::vector<OSDOp> &osd_ops) -{ - LOG_PREFIX(InternalClientRequest::do_process); - auto params = get_do_osd_ops_params(); - OpsExecuter ox( - pg, obc, op_info, params, params.get_connection(), SnapContext{}); - co_await pg->run_executer( - ox, obc, op_info, osd_ops - ).handle_error_interruptible( - crimson::ct_error::all_same_way( - [this, FNAME](auto e) { - ERRORDPPI("{}: got unexpected error {}", *pg, *this, e); - ceph_assert(0 == "should not return an error"); - return interruptor::now(); - }) - ); - - auto [submitted, completed] = co_await pg->submit_executer( - std::move(ox), osd_ops); - - co_await std::move(submitted); - co_await std::move(completed); -} - -InternalClientRequest::interruptible_future<> InternalClientRequest::with_interruption() { LOG_PREFIX(InternalClientRequest::with_interruption); assert(pg->is_active()); - co_await enter_stage<interruptor>(client_pp().recover_missing); + obc_orderer = pg->obc_loader.get_obc_orderer(get_target_oid()); + auto obc_manager = pg->obc_loader.get_obc_manager( + *obc_orderer, + get_target_oid()); + + co_await enter_stage<interruptor>(obc_orderer->obc_pp().process); bool unfound = co_await do_recover_missing( pg, get_target_oid(), osd_reqid_t()); @@ -94,10 +72,8 @@ InternalClientRequest::with_interruption() std::make_error_code(std::errc::operation_canceled), fmt::format("{} is unfound, drop it!", get_target_oid())); } - co_await enter_stage<interruptor>( - client_pp().check_already_complete_get_obc); - DEBUGI("{}: getting obc lock", *this); + DEBUGI("{}: generating ops", *this); auto osd_ops = create_osd_ops(); @@ -107,26 +83,37 @@ InternalClientRequest::with_interruption() std::as_const(osd_ops), pg->get_pgid().pgid, *pg->get_osdmap()); assert(ret == 0); - auto obc_manager = pg->obc_loader.get_obc_manager(get_target_oid()); - - // initiate load_and_lock in order, but wait concurrently - enter_stage_sync(client_pp().lock_obc); - co_await pg->obc_loader.load_and_lock( obc_manager, pg->get_lock_type(op_info) ).handle_error_interruptible( crimson::ct_error::assert_all("unexpected error") ); - DEBUGDPP("{}: got obc {}, entering process stage", - *pg, *this, obc_manager.get_obc()->obs); - co_await enter_stage<interruptor>(client_pp().process); + auto params = get_do_osd_ops_params(); + OpsExecuter ox( + pg, obc_manager.get_obc(), op_info, params, params.get_connection(), + SnapContext{}); + co_await pg->run_executer( + ox, obc_manager.get_obc(), op_info, osd_ops + ).handle_error_interruptible( + crimson::ct_error::all_same_way( + [this, FNAME](auto e) { + ERRORDPPI("{}: got unexpected error {}", *pg, *this, e); + ceph_assert(0 == "should not return an error"); + return interruptor::now(); + }) + ); + + auto [submitted, completed] = co_await pg->submit_executer( + std::move(ox), osd_ops); + + co_await std::move(submitted); - DEBUGDPP("{}: in process stage, calling do_process", - *pg, *this); - co_await do_process(obc_manager.get_obc(), osd_ops); + co_await enter_stage<interruptor>(obc_orderer->obc_pp().wait_repop); + + co_await std::move(completed); - logger().debug("{}: complete", *this); + DEBUGDPP("{}: complete", *pg, *this); co_await interruptor::make_interruptible(handle.complete()); co_return; } @@ -148,7 +135,7 @@ seastar::future<> InternalClientRequest::start() return seastar::now(); }).finally([this] { logger().debug("{}: exit", *this); - handle.exit(); + return handle.complete(); }); } diff --git a/src/crimson/osd/osd_operations/internal_client_request.h b/src/crimson/osd/osd_operations/internal_client_request.h index 6023db0a8db..1cfde4ab080 100644 --- a/src/crimson/osd/osd_operations/internal_client_request.h +++ b/src/crimson/osd/osd_operations/internal_client_request.h @@ -4,6 +4,7 @@ #pragma once #include "crimson/common/type_helpers.h" +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osd_operation.h" #include "crimson/osd/osd_operations/client_request_common.h" #include "crimson/osd/pg.h" @@ -45,11 +46,10 @@ private: crimson::osd::ObjectContextRef obc, std::vector<OSDOp> &osd_ops); - seastar::future<> do_process(); - Ref<PG> pg; epoch_t start_epoch; OpInfo op_info; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; public: @@ -57,12 +57,8 @@ public: std::tuple< StartEvent, - CommonPGPipeline::WaitForActive::BlockingEvent, - PGActivationBlocker::BlockingEvent, - CommonPGPipeline::RecoverMissing::BlockingEvent, - CommonPGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - CommonPGPipeline::LockOBC::BlockingEvent, - CommonPGPipeline::Process::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, CompletionEvent > tracking_events; }; diff --git a/src/crimson/osd/osd_operations/logmissing_request.cc b/src/crimson/osd/osd_operations/logmissing_request.cc index 8147c969260..274744cdd92 100644 --- a/src/crimson/osd/osd_operations/logmissing_request.cc +++ b/src/crimson/osd/osd_operations/logmissing_request.cc @@ -58,9 +58,9 @@ PerShardPipeline &LogMissingRequest::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &LogMissingRequest::client_pp(PG &pg) +PGRepopPipeline &LogMissingRequest::repop_pipeline(PG &pg) { - return pg.request_pg_pipeline; + return pg.repop_pipeline; } seastar::future<> LogMissingRequest::with_pg( @@ -73,7 +73,7 @@ seastar::future<> LogMissingRequest::with_pg( return interruptor::with_interruption([this, pg] { LOG_PREFIX(LogMissingRequest::with_pg); DEBUGI("{}: pg present", *this); - return this->template enter_stage<interruptor>(client_pp(*pg).await_map + return this->template enter_stage<interruptor>(repop_pipeline(*pg).process ).then_interruptible([this, pg] { return this->template with_blocking_event< PG_OSDMapGate::OSDMapBlocker::BlockingEvent diff --git a/src/crimson/osd/osd_operations/logmissing_request.h b/src/crimson/osd/osd_operations/logmissing_request.h index 51c9d540cb5..fe4761c4ab4 100644 --- a/src/crimson/osd/osd_operations/logmissing_request.h +++ b/src/crimson/osd/osd_operations/logmissing_request.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -77,14 +80,14 @@ public: ConnectionPipeline::AwaitMap::BlockingEvent, ConnectionPipeline::GetPGMapping::BlockingEvent, PerShardPipeline::CreateOrWaitPG::BlockingEvent, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent, + PGRepopPipeline::Process::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, PGMap::PGCreationBlockingEvent, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); + PGRepopPipeline &repop_pipeline(PG &pg); crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/logmissing_request_reply.cc b/src/crimson/osd/osd_operations/logmissing_request_reply.cc index fb122a95cd1..5640610bd01 100644 --- a/src/crimson/osd/osd_operations/logmissing_request_reply.cc +++ b/src/crimson/osd/osd_operations/logmissing_request_reply.cc @@ -56,11 +56,6 @@ PerShardPipeline &LogMissingRequestReply::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &LogMissingRequestReply::client_pp(PG &pg) -{ - return pg.request_pg_pipeline; -} - seastar::future<> LogMissingRequestReply::with_pg( ShardServices &shard_services, Ref<PG> pg) { diff --git a/src/crimson/osd/osd_operations/logmissing_request_reply.h b/src/crimson/osd/osd_operations/logmissing_request_reply.h index c741b41bd0f..bdb6c2ac6ac 100644 --- a/src/crimson/osd/osd_operations/logmissing_request_reply.h +++ b/src/crimson/osd/osd_operations/logmissing_request_reply.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -82,8 +85,6 @@ public: > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); - crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/peering_event.h b/src/crimson/osd/osd_operations/peering_event.h index 85de5c711d6..aa6b8a95a94 100644 --- a/src/crimson/osd/osd_operations/peering_event.h +++ b/src/crimson/osd/osd_operations/peering_event.h @@ -44,6 +44,10 @@ protected: float delay = 0; PGPeeringEvent evt; + epoch_t get_epoch_sent_at() const { + return evt.get_epoch_sent(); + } + const pg_shard_t get_from() const { return from; } @@ -84,6 +88,10 @@ public: evt(std::forward<Args>(args)...) {} + bool requires_pg() const final { + return evt.requires_pg; + } + void print(std::ostream &) const final; void dump_detail(ceph::Formatter* f) const final; seastar::future<> with_pg( diff --git a/src/crimson/osd/osd_operations/pg_advance_map.h b/src/crimson/osd/osd_operations/pg_advance_map.h index 43be7319545..21702f6ff4f 100644 --- a/src/crimson/osd/osd_operations/pg_advance_map.h +++ b/src/crimson/osd/osd_operations/pg_advance_map.h @@ -50,6 +50,10 @@ public: PGPeeringPipeline::Process::BlockingEvent > tracking_events; + epoch_t get_epoch_sent_at() const { + return to; + } + private: PGPeeringPipeline &peering_pp(PG &pg); }; diff --git a/src/crimson/osd/osd_operations/recovery_subrequest.h b/src/crimson/osd/osd_operations/recovery_subrequest.h index 17c2faf97ea..2fe8ff372b3 100644 --- a/src/crimson/osd/osd_operations/recovery_subrequest.h +++ b/src/crimson/osd/osd_operations/recovery_subrequest.h @@ -39,6 +39,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return m->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return m->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); diff --git a/src/crimson/osd/osd_operations/replicated_request.cc b/src/crimson/osd/osd_operations/replicated_request.cc index 5ca11e5dd15..ec607758c55 100644 --- a/src/crimson/osd/osd_operations/replicated_request.cc +++ b/src/crimson/osd/osd_operations/replicated_request.cc @@ -5,6 +5,7 @@ #include "common/Formatter.h" +#include "crimson/common/coroutine.h" #include "crimson/osd/osd.h" #include "crimson/osd/osd_connection_priv.h" #include "crimson/osd/osd_operation_external_tracking.h" @@ -58,39 +59,57 @@ PerShardPipeline &RepRequest::get_pershard_pipeline( return shard_services.get_replicated_request_pipeline(); } -ClientRequest::PGPipeline &RepRequest::client_pp(PG &pg) +PGRepopPipeline &RepRequest::repop_pipeline(PG &pg) { - return pg.request_pg_pipeline; + return pg.repop_pipeline; +} + +RepRequest::interruptible_future<> RepRequest::with_pg_interruptible( + Ref<PG> pg) +{ + LOG_PREFIX(RepRequest::with_pg_interruptible); + DEBUGI("{}", *this); + co_await this->template enter_stage<interruptor>(repop_pipeline(*pg).process); + co_await interruptor::make_interruptible(this->template with_blocking_event< + PG_OSDMapGate::OSDMapBlocker::BlockingEvent + >([this, pg](auto &&trigger) { + return pg->osdmap_gate.wait_for_map( + std::move(trigger), req->min_epoch); + })); + + if (pg->can_discard_replica_op(*req)) { + co_return; + } + + auto [commit_fut, reply] = co_await pg->handle_rep_op(req); + + // Transitions from OrderedExclusive->OrderedConcurrent cannot block + this->template enter_stage_sync(repop_pipeline(*pg).wait_commit); + + co_await std::move(commit_fut); + + co_await this->template enter_stage<interruptor>( + repop_pipeline(*pg).send_reply); + + co_await interruptor::make_interruptible( + pg->shard_services.send_to_osd( + req->from.osd, std::move(reply), pg->get_osdmap_epoch()) + ); } seastar::future<> RepRequest::with_pg( ShardServices &shard_services, Ref<PG> pg) { LOG_PREFIX(RepRequest::with_pg); - DEBUGI("{}: RepRequest::with_pg", *this); + DEBUGI("{}", *this); IRef ref = this; return interruptor::with_interruption([this, pg] { - LOG_PREFIX(RepRequest::with_pg); - DEBUGI("{}: pg present", *this); - return this->template enter_stage<interruptor>(client_pp(*pg).await_map - ).then_interruptible([this, pg] { - return this->template with_blocking_event< - PG_OSDMapGate::OSDMapBlocker::BlockingEvent - >([this, pg](auto &&trigger) { - return pg->osdmap_gate.wait_for_map( - std::move(trigger), req->min_epoch); - }); - }).then_interruptible([this, pg] (auto) { - return pg->handle_rep_op(req); - }).then_interruptible([this] { - logger().debug("{}: complete", *this); - return handle.complete(); - }); + return with_pg_interruptible(pg); }, [](std::exception_ptr) { return seastar::now(); }, pg, pg->get_osdmap_epoch()).finally([this, ref=std::move(ref)] { logger().debug("{}: exit", *this); - handle.exit(); + return handle.complete(); }); } diff --git a/src/crimson/osd/osd_operations/replicated_request.h b/src/crimson/osd/osd_operations/replicated_request.h index ff5dea6d6db..c2494b3715f 100644 --- a/src/crimson/osd/osd_operations/replicated_request.h +++ b/src/crimson/osd/osd_operations/replicated_request.h @@ -36,6 +36,9 @@ public: } PipelineHandle &get_handle() { return handle; } epoch_t get_epoch() const { return req->get_min_epoch(); } + epoch_t get_epoch_sent_at() const { + return req->get_map_epoch(); + } ConnectionPipeline &get_connection_pipeline(); @@ -68,6 +71,9 @@ public: r_conn = make_local_shared_foreign(std::move(conn)); } + interruptible_future<> with_pg_interruptible( + Ref<PG> pg); + seastar::future<> with_pg( ShardServices &shard_services, Ref<PG> pg); @@ -77,14 +83,16 @@ public: ConnectionPipeline::AwaitMap::BlockingEvent, ConnectionPipeline::GetPGMapping::BlockingEvent, PerShardPipeline::CreateOrWaitPG::BlockingEvent, - ClientRequest::PGPipeline::AwaitMap::BlockingEvent, + PGRepopPipeline::Process::BlockingEvent, + PGRepopPipeline::WaitCommit::BlockingEvent, + PGRepopPipeline::SendReply::BlockingEvent, PG_OSDMapGate::OSDMapBlocker::BlockingEvent, PGMap::PGCreationBlockingEvent, OSD_OSDMapGate::OSDMapBlocker::BlockingEvent > tracking_events; private: - ClientRequest::PGPipeline &client_pp(PG &pg); + PGRepopPipeline &repop_pipeline(PG &pg); crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; diff --git a/src/crimson/osd/osd_operations/scrub_events.h b/src/crimson/osd/osd_operations/scrub_events.h index 02a5d852bb7..8bed90e4c14 100644 --- a/src/crimson/osd/osd_operations/scrub_events.h +++ b/src/crimson/osd/osd_operations/scrub_events.h @@ -27,11 +27,11 @@ class RemoteScrubEventBaseT : public PhasedOperationT<T> { crimson::net::ConnectionRef l_conn; crimson::net::ConnectionXcoreRef r_conn; - epoch_t epoch; spg_t pgid; protected: using interruptor = InterruptibleOperation::interruptor; + epoch_t epoch; template <typename U=void> using ifut = InterruptibleOperation::interruptible_future<U>; @@ -40,7 +40,7 @@ protected: public: RemoteScrubEventBaseT( crimson::net::ConnectionRef conn, epoch_t epoch, spg_t pgid) - : l_conn(std::move(conn)), epoch(epoch), pgid(pgid) {} + : l_conn(std::move(conn)), pgid(pgid), epoch(epoch) {} PGPeeringPipeline &get_peering_pipeline(PG &pg); @@ -117,6 +117,10 @@ public: : RemoteScrubEventBaseT<ScrubRequested>(std::forward<Args>(base_args)...), deep(deep) {} + epoch_t get_epoch_sent_at() const { + return epoch; + } + void print(std::ostream &out) const final { out << "(deep=" << deep << ")"; } @@ -141,6 +145,10 @@ public: ceph_assert(scrub::PGScrubber::is_scrub_message(*m)); } + epoch_t get_epoch_sent_at() const { + return epoch; + } + void print(std::ostream &out) const final { out << "(m=" << *m << ")"; } diff --git a/src/crimson/osd/osd_operations/snaptrim_event.cc b/src/crimson/osd/osd_operations/snaptrim_event.cc index 8cab6125682..f8fb7aef6f2 100644 --- a/src/crimson/osd/osd_operations/snaptrim_event.cc +++ b/src/crimson/osd/osd_operations/snaptrim_event.cc @@ -388,20 +388,24 @@ SnapTrimObjSubEvent::remove_or_update( SnapTrimObjSubEvent::snap_trim_obj_subevent_ret_t SnapTrimObjSubEvent::start() { + obc_orderer = pg->obc_loader.get_obc_orderer( + coid); + ceph_assert(pg->is_active_clean()); - auto exit_handle = seastar::defer([this] { - logger().debug("{}: exit", *this); - handle.exit(); + auto exit_handle = seastar::defer([this, opref = IRef(this)] { + logger().debug("{}: exit", *opref); + std::ignore = handle.complete().then([opref = std::move(opref)] {}); }); co_await enter_stage<interruptor>( - client_pp().check_already_complete_get_obc); + obc_orderer->obc_pp().process); logger().debug("{}: getting obc for {}", *this, coid); auto obc_manager = pg->obc_loader.get_obc_manager( + *obc_orderer, coid, false /* resolve_oid */); co_await pg->obc_loader.load_and_lock( @@ -411,43 +415,39 @@ SnapTrimObjSubEvent::start() crimson::ct_error::assert_all{"unexpected error in SnapTrimObjSubEvent"} ); - co_await process_and_submit( - obc_manager.get_head_obc(), obc_manager.get_obc() - ).handle_error_interruptible( - remove_or_update_iertr::pass_further{}, - crimson::ct_error::assert_all{"unexpected error in SnapTrimObjSubEvent"} - ); - - logger().debug("{}: completed", *this); - co_await interruptor::make_interruptible(handle.complete()); -} - -ObjectContextLoader::load_obc_iertr::future<> -SnapTrimObjSubEvent::process_and_submit(ObjectContextRef head_obc, - ObjectContextRef clone_obc) { - logger().debug("{}: got clone_obc={}", *this, clone_obc->get_oid()); + logger().debug("{}: got obc={}", *this, obc_manager.get_obc()->get_oid()); - co_await enter_stage<interruptor>(client_pp().process); - - logger().debug("{}: processing clone_obc={}", *this, clone_obc->get_oid()); - - auto txn = co_await remove_or_update(clone_obc, head_obc); - - auto [submitted, all_completed] = co_await pg->submit_transaction( - std::move(clone_obc), - nullptr, - std::move(txn), - std::move(osd_op_p), - std::move(log_entries) - ); + auto all_completed = interruptor::now(); + { + // as with PG::submit_executer, we need to build the pg log entries + // and submit the transaction atomically + co_await interruptor::make_interruptible(pg->submit_lock.lock()); + auto unlocker = seastar::defer([this] { + pg->submit_lock.unlock(); + }); - co_await std::move(submitted); + logger().debug("{}: calling remove_or_update obc={}", + *this, obc_manager.get_obc()->get_oid()); + + auto txn = co_await remove_or_update( + obc_manager.get_obc(), obc_manager.get_head_obc()); + + auto submitted = interruptor::now(); + std::tie(submitted, all_completed) = co_await pg->submit_transaction( + ObjectContextRef(obc_manager.get_obc()), + nullptr, + std::move(txn), + std::move(osd_op_p), + std::move(log_entries) + ); + co_await std::move(submitted); + } - co_await enter_stage<interruptor>(client_pp().wait_repop); + co_await enter_stage<interruptor>(obc_orderer->obc_pp().wait_repop); co_await std::move(all_completed); - co_return; + logger().debug("{}: completed", *this); } void SnapTrimObjSubEvent::print(std::ostream &lhs) const diff --git a/src/crimson/osd/osd_operations/snaptrim_event.h b/src/crimson/osd/osd_operations/snaptrim_event.h index 1164b3169d2..a2b4d357568 100644 --- a/src/crimson/osd/osd_operations/snaptrim_event.h +++ b/src/crimson/osd/osd_operations/snaptrim_event.h @@ -6,6 +6,7 @@ #include <iostream> #include <seastar/core/future.hh> +#include "crimson/osd/object_context_loader.h" #include "crimson/osd/osdmap_gate.h" #include "crimson/osd/osd_operation.h" #include "crimson/common/subop_blocker.h" @@ -112,10 +113,6 @@ public: private: object_stat_sum_t delta_stats; - ObjectContextLoader::load_obc_iertr::future<> process_and_submit( - ObjectContextRef head_obc, - ObjectContextRef clone_obc); - snap_trim_obj_subevent_ret_t remove_clone( ObjectContextRef obc, ObjectContextRef head_obc, @@ -158,6 +155,7 @@ private: } Ref<PG> pg; + std::optional<ObjectContextLoader::Orderer> obc_orderer; PipelineHandle handle; osd_op_params_t osd_op_p; const hobject_t coid; @@ -169,9 +167,8 @@ public: std::tuple< StartEvent, - CommonPGPipeline::CheckAlreadyCompleteGetObc::BlockingEvent, - CommonPGPipeline::Process::BlockingEvent, - CommonPGPipeline::WaitRepop::BlockingEvent, + CommonOBCPipeline::Process::BlockingEvent, + CommonOBCPipeline::WaitRepop::BlockingEvent, CompletionEvent > tracking_events; }; diff --git a/src/crimson/osd/pg.cc b/src/crimson/osd/pg.cc index 1e2988efbbe..2746e730f2b 100644 --- a/src/crimson/osd/pg.cc +++ b/src/crimson/osd/pg.cc @@ -868,46 +868,18 @@ std::ostream& operator<<(std::ostream& os, const PG& pg) return os; } -void PG::mutate_object( - ObjectContextRef& obc, - ceph::os::Transaction& txn, - osd_op_params_t& osd_op_p) +void PG::enqueue_push_for_backfill( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers) { - if (obc->obs.exists) { - obc->obs.oi.prior_version = obc->obs.oi.version; - obc->obs.oi.version = osd_op_p.at_version; - if (osd_op_p.user_modify) - obc->obs.oi.user_version = osd_op_p.at_version.version; - obc->obs.oi.last_reqid = osd_op_p.req_id; - obc->obs.oi.mtime = osd_op_p.mtime; - obc->obs.oi.local_mtime = ceph_clock_now(); - - // object_info_t - { - ceph::bufferlist osv; - obc->obs.oi.encode_no_oid(osv, CEPH_FEATURES_ALL); - // TODO: get_osdmap()->get_features(CEPH_ENTITY_TYPE_OSD, nullptr)); - txn.setattr(coll_ref->get_cid(), ghobject_t{obc->obs.oi.soid}, OI_ATTR, osv); - } - - // snapset - if (obc->obs.oi.soid.snap == CEPH_NOSNAP) { - logger().debug("final snapset {} in {}", - obc->ssc->snapset, obc->obs.oi.soid); - ceph::bufferlist bss; - encode(obc->ssc->snapset, bss); - txn.setattr(coll_ref->get_cid(), ghobject_t{obc->obs.oi.soid}, SS_ATTR, bss); - obc->ssc->exists = true; - } else { - logger().debug("no snapset (this is a clone)"); - } - } else { - // reset cached ObjectState without enforcing eviction - obc->obs.oi = object_info_t(obc->obs.oi.soid); - } + assert(recovery_handler); + assert(recovery_handler->backfill_state); + auto backfill_state = recovery_handler->backfill_state.get(); + backfill_state->enqueue_standalone_push(obj, v, peers); } -void PG::enqueue_push_for_backfill( +void PG::enqueue_delete_for_backfill( const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers) @@ -915,7 +887,7 @@ void PG::enqueue_push_for_backfill( assert(recovery_handler); assert(recovery_handler->backfill_state); auto backfill_state = recovery_handler->backfill_state.get(); - backfill_state->enqueue_standalone_push(obj, v, peers); + backfill_state->enqueue_standalone_delete(obj, v, peers); } PG::interruptible_future< @@ -1039,8 +1011,15 @@ PG::interruptible_future<eversion_t> PG::submit_error_log( const std::error_code e, ceph_tid_t rep_tid) { - logger().debug("{}: {} rep_tid: {} error: {}", - __func__, *m, rep_tid, e); + // as with submit_executer, need to ensure that log numbering and submission + // are atomic + co_await interruptor::make_interruptible(submit_lock.lock()); + auto unlocker = seastar::defer([this] { + submit_lock.unlock(); + }); + LOG_PREFIX(PG::submit_error_log); + DEBUGDPP("{} rep_tid: {} error: {}", + *this, *m, rep_tid, e); const osd_reqid_t &reqid = m->get_reqid(); mempool::osd_pglog::list<pg_log_entry_t> log_entries; log_entries.push_back(pg_log_entry_t(pg_log_entry_t::ERROR, @@ -1061,47 +1040,45 @@ PG::interruptible_future<eversion_t> PG::submit_error_log( log_entries, t, peering_state.get_pg_trim_to(), peering_state.get_pg_committed_to()); - return seastar::do_with(log_entries, set<pg_shard_t>{}, - [this, t=std::move(t), rep_tid](auto& log_entries, auto& waiting_on) mutable { - return interruptor::do_for_each(get_acting_recovery_backfill(), - [this, log_entries, waiting_on, rep_tid] - (auto& i) mutable { - pg_shard_t peer(i); - if (peer == pg_whoami) { - return seastar::now(); - } - ceph_assert(peering_state.get_peer_missing().count(peer)); - ceph_assert(peering_state.has_peer_info(peer)); - auto log_m = crimson::make_message<MOSDPGUpdateLogMissing>( - log_entries, - spg_t(peering_state.get_info().pgid.pgid, i.shard), - pg_whoami.shard, - get_osdmap_epoch(), - get_last_peering_reset(), - rep_tid, - peering_state.get_pg_trim_to(), - peering_state.get_pg_committed_to()); - waiting_on.insert(peer); - logger().debug("submit_error_log: sending log" - "missing_request (rep_tid: {} entries: {})" - " to osd {}", rep_tid, log_entries, peer.osd); - return shard_services.send_to_osd(peer.osd, - std::move(log_m), - get_osdmap_epoch()); - }).then_interruptible([this, waiting_on, t=std::move(t), rep_tid] () mutable { - waiting_on.insert(pg_whoami); - logger().debug("submit_error_log: inserting rep_tid {}", rep_tid); - log_entry_update_waiting_on.insert( - std::make_pair(rep_tid, - log_update_t{std::move(waiting_on)})); - return shard_services.get_store().do_transaction( - get_collection_ref(), std::move(t) - ).then([this] { - peering_state.update_trim_to(); - return seastar::make_ready_future<eversion_t>(projected_last_update); - }); - }); - }); + + set<pg_shard_t> waiting_on; + for (const auto &peer: get_acting_recovery_backfill()) { + if (peer == pg_whoami) { + continue; + } + ceph_assert(peering_state.get_peer_missing().count(peer)); + ceph_assert(peering_state.has_peer_info(peer)); + auto log_m = crimson::make_message<MOSDPGUpdateLogMissing>( + log_entries, + spg_t(peering_state.get_info().pgid.pgid, peer.shard), + pg_whoami.shard, + get_osdmap_epoch(), + get_last_peering_reset(), + rep_tid, + peering_state.get_pg_trim_to(), + peering_state.get_pg_committed_to()); + waiting_on.insert(peer); + + DEBUGDPP("sending log missing_request (rep_tid: {} entries: {}) to osd {}", + *this, rep_tid, log_entries, peer.osd); + co_await interruptor::make_interruptible( + shard_services.send_to_osd( + peer.osd, + std::move(log_m), + get_osdmap_epoch())); + } + waiting_on.insert(pg_whoami); + DEBUGDPP("inserting rep_tid {}", *this, rep_tid); + log_entry_update_waiting_on.insert( + std::make_pair(rep_tid, + log_update_t{std::move(waiting_on)})); + co_await interruptor::make_interruptible( + shard_services.get_store().do_transaction( + get_collection_ref(), std::move(t) + )); + + peering_state.update_trim_to(); + co_return projected_last_update; } PG::run_executer_fut PG::run_executer( @@ -1157,27 +1134,25 @@ PG::submit_executer_fut PG::submit_executer( OpsExecuter &&ox, const std::vector<OSDOp>& ops) { LOG_PREFIX(PG::submit_executer); - // transaction must commit at this point - return std::move( + DEBUGDPP("", *this); + + // we need to build the pg log entries and submit the transaction + // atomically to ensure log ordering + co_await interruptor::make_interruptible(submit_lock.lock()); + auto unlocker = seastar::defer([this] { + submit_lock.unlock(); + }); + + auto [submitted, completed] = co_await std::move( ox - ).flush_changes_n_do_ops_effects( + ).flush_changes_and_submit( ops, snap_mapper, - osdriver, - [FNAME, this](auto&& txn, - auto&& obc, - auto&& osd_op_p, - auto&& log_entries, - auto&& new_clone) { - DEBUGDPP("object {} submitting txn", *this, obc->get_oid()); - mutate_object(obc, txn, osd_op_p); - return submit_transaction( - std::move(obc), - std::move(new_clone), - std::move(txn), - std::move(osd_op_p), - std::move(log_entries)); - }); + osdriver + ); + co_return std::make_tuple( + std::move(submitted).then_interruptible([unlocker=std::move(unlocker)] {}), + std::move(completed)); } PG::interruptible_future<MURef<MOSDOpReply>> PG::do_pg_ops(Ref<MOSDOp> m) @@ -1251,13 +1226,10 @@ void PG::update_stats(const pg_stat_t &stat) { ); } -PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) +PG::handle_rep_op_fut PG::handle_rep_op(Ref<MOSDRepOp> req) { LOG_PREFIX(PG::handle_rep_op); DEBUGDPP("{}", *this, *req); - if (can_discard_replica_op(*req)) { - co_return; - } ceph::os::Transaction txn; auto encoded_txn = req->get_data().cbegin(); @@ -1279,7 +1251,8 @@ PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) txn, false); DEBUGDPP("{} do_transaction", *this, *req); - co_await interruptor::make_interruptible( + + auto commit_fut = interruptor::make_interruptible( shard_services.get_store().do_transaction(coll_ref, std::move(txn)) ); @@ -1290,10 +1263,7 @@ PG::interruptible_future<> PG::handle_rep_op(Ref<MOSDRepOp> req) req.get(), pg_whoami, 0, map_epoch, req->get_min_epoch(), CEPH_OSD_FLAG_ONDISK); reply->set_last_complete_ondisk(lcod); - co_await interruptor::make_interruptible( - shard_services.send_to_osd(req->from.osd, std::move(reply), map_epoch) - ); - co_return; + co_return handle_rep_op_ret(std::move(commit_fut), std::move(reply)); } PG::interruptible_future<> PG::update_snap_map( @@ -1324,20 +1294,21 @@ void PG::log_operation( bool transaction_applied, ObjectStore::Transaction &txn, bool async) { - logger().debug("{}", __func__); + LOG_PREFIX(PG::log_operation); + DEBUGDPP("", *this); if (is_primary()) { ceph_assert(trim_to <= peering_state.get_pg_committed_to()); } auto last = logv.rbegin(); if (is_primary() && last != logv.rend()) { - logger().debug("{} on primary, trimming projected log", - __func__); + DEBUGDPP("on primary, trimming projected log", *this); projected_log.skip_can_rollback_to_to_head(); projected_log.trim(shard_services.get_cct(), last->version, nullptr, nullptr, nullptr); } if (!is_primary()) { // && !is_ec_pg() + DEBUGDPP("on replica, clearing obc", *this); replica_clear_repop_obc(logv); } if (!logv.empty()) { @@ -1354,13 +1325,13 @@ void PG::log_operation( void PG::replica_clear_repop_obc( const std::vector<pg_log_entry_t> &logv) { - logger().debug("{} clearing {} entries", __func__, logv.size()); - for (auto &&e: logv) { - logger().debug(" {} get_object_boundary(from): {} " - " head version(to): {}", - e.soid, - e.soid.get_object_boundary(), - e.soid.get_head()); + LOG_PREFIX(PG::replica_clear_repop_obc); + DEBUGDPP("clearing obc for {} log entries", logv.size()); + for (auto &&e: logv) { + DEBUGDPP("clearing entry for {} from: {} to: {}", + e.soid, + e.soid.get_object_boundary(), + e.soid.get_head()); /* Have to blast all clones, they share a snapset */ obc_registry.clear_range( e.soid.get_object_boundary(), e.soid.get_head()); diff --git a/src/crimson/osd/pg.h b/src/crimson/osd/pg.h index 15aeec0e4f3..06038c0aa00 100644 --- a/src/crimson/osd/pg.h +++ b/src/crimson/osd/pg.h @@ -10,6 +10,7 @@ #include <seastar/core/shared_future.hh> #include "common/dout.h" +#include "common/ostream_temp.h" #include "include/interval_set.h" #include "crimson/net/Fwd.h" #include "messages/MOSDRepOpReply.h" @@ -77,7 +78,8 @@ class PG : public boost::intrusive_ref_counter< using ec_profile_t = std::map<std::string,std::string>; using cached_map_t = OSDMapService::cached_map_t; - ClientRequest::PGPipeline request_pg_pipeline; + CommonPGPipeline request_pg_pipeline; + PGRepopPipeline repop_pipeline; PGPeeringPipeline peering_request_pg_pipeline; ClientRequest::Orderer client_request_orderer; @@ -594,7 +596,13 @@ public: using with_obc_func_t = std::function<load_obc_iertr::future<> (ObjectContextRef, ObjectContextRef)>; - interruptible_future<> handle_rep_op(Ref<MOSDRepOp> m); + using handle_rep_op_ret = std::tuple< + interruptible_future<>, // resolves upon commit + MURef<MOSDRepOpReply> // reply message + >; + // outer future resolves upon submission + using handle_rep_op_fut = interruptible_future<handle_rep_op_ret>; + handle_rep_op_fut handle_rep_op(Ref<MOSDRepOp> m); void update_stats(const pg_stat_t &stat); interruptible_future<> update_snap_map( const std::vector<pg_log_entry_t> &log_entries, @@ -663,6 +671,7 @@ private: const OpInfo &op_info, std::vector<OSDOp>& ops); + seastar::shared_mutex submit_lock; using submit_executer_ret = std::tuple< interruptible_future<>, interruptible_future<>>; @@ -675,6 +684,8 @@ private: struct do_osd_ops_params_t; interruptible_future<MURef<MOSDOpReply>> do_pg_ops(Ref<MOSDOp> m); + +public: interruptible_future< std::tuple<interruptible_future<>, interruptible_future<>>> submit_transaction( @@ -683,6 +694,8 @@ private: ceph::os::Transaction&& txn, osd_op_params_t&& oop, std::vector<pg_log_entry_t>&& log_entries); + +private: interruptible_future<> repair_object( const hobject_t& oid, eversion_t& v); @@ -891,10 +904,11 @@ private: const hobject_t &obj, const eversion_t &v, const std::vector<pg_shard_t> &peers); - void mutate_object( - ObjectContextRef& obc, - ceph::os::Transaction& txn, - osd_op_params_t& osd_op_p); + void enqueue_delete_for_backfill( + const hobject_t &obj, + const eversion_t &v, + const std::vector<pg_shard_t> &peers); + bool can_discard_replica_op(const Message& m, epoch_t m_map_epoch) const; bool can_discard_op(const MOSDOp& m) const; void context_registry_on_change(); diff --git a/src/crimson/osd/pg_backend.cc b/src/crimson/osd/pg_backend.cc index 24a381b4cf7..79895de06de 100644 --- a/src/crimson/osd/pg_backend.cc +++ b/src/crimson/osd/pg_backend.cc @@ -1283,22 +1283,6 @@ PGBackend::rm_xattr( return rm_xattr_iertr::now(); } -void PGBackend::clone( - /* const */object_info_t& snap_oi, - const ObjectState& os, - const ObjectState& d_os, - ceph::os::Transaction& txn) -{ - // See OpsExecuter::execute_clone documentation - txn.clone(coll->get_cid(), ghobject_t{os.oi.soid}, ghobject_t{d_os.oi.soid}); - { - ceph::bufferlist bv; - snap_oi.encode_no_oid(bv, CEPH_FEATURES_ALL); - txn.setattr(coll->get_cid(), ghobject_t{d_os.oi.soid}, OI_ATTR, bv); - } - txn.rmattr(coll->get_cid(), ghobject_t{d_os.oi.soid}, SS_ATTR); -} - using get_omap_ertr = crimson::os::FuturizedStore::Shard::read_errorator::extend< crimson::ct_error::enodata>; @@ -1341,9 +1325,10 @@ maybe_get_omap_vals( PGBackend::ll_read_ierrorator::future<ceph::bufferlist> PGBackend::omap_get_header( const crimson::os::CollectionRef& c, - const ghobject_t& oid) const + const ghobject_t& oid, + uint32_t op_flags) const { - return store->omap_get_header(c, oid) + return store->omap_get_header(c, oid, op_flags) .handle_error( crimson::ct_error::enodata::handle([] { return seastar::make_ready_future<bufferlist>(); @@ -1356,10 +1341,13 @@ PGBackend::ll_read_ierrorator::future<> PGBackend::omap_get_header( const ObjectState& os, OSDOp& osd_op, - object_stat_sum_t& delta_stats) const + object_stat_sum_t& delta_stats, + uint32_t op_flags) const { if (os.oi.is_omap()) { - return omap_get_header(coll, ghobject_t{os.oi.soid}).safe_then_interruptible( + return omap_get_header( + coll, ghobject_t{os.oi.soid}, CEPH_OSD_OP_FLAG_FADVISE_DONTNEED + ).safe_then_interruptible( [&delta_stats, &osd_op] (ceph::bufferlist&& header) { osd_op.outdata = std::move(header); delta_stats.num_rd_kb += shift_round_up(osd_op.outdata.length(), 10); @@ -1723,7 +1711,8 @@ PGBackend::fiemap( CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len) + uint64_t len, + uint32_t op_flags) { return store->fiemap(c, oid, off, len); } @@ -1835,3 +1824,32 @@ PGBackend::read_ierrorator::future<> PGBackend::tmapget( read_errorator::pass_further{}); } +void PGBackend::set_metadata( + const hobject_t &obj, + object_info_t &oi, + const SnapSet *ss /* non-null iff head */, + ceph::os::Transaction& txn) +{ + ceph_assert((obj.is_head() && ss) || (!obj.is_head() && !ss)); + { + ceph::bufferlist bv; + oi.encode_no_oid(bv, CEPH_FEATURES_ALL); + txn.setattr(coll->get_cid(), ghobject_t{obj}, OI_ATTR, bv); + } + if (ss) { + ceph::bufferlist bss; + encode(*ss, bss); + txn.setattr(coll->get_cid(), ghobject_t{obj}, SS_ATTR, bss); + } +} + +void PGBackend::clone_for_write( + const hobject_t &from, + const hobject_t &to, + ceph::os::Transaction &txn) +{ + // See OpsExecuter::execute_clone documentation + txn.clone(coll->get_cid(), ghobject_t{from}, ghobject_t{to}); + txn.rmattr(coll->get_cid(), ghobject_t{to}, SS_ATTR); +} + diff --git a/src/crimson/osd/pg_backend.h b/src/crimson/osd/pg_backend.h index 813218983fd..9c2230375b0 100644 --- a/src/crimson/osd/pg_backend.h +++ b/src/crimson/osd/pg_backend.h @@ -308,11 +308,6 @@ public: ObjectState& os, const OSDOp& osd_op, ceph::os::Transaction& trans); - void clone( - /* const */object_info_t& snap_oi, - const ObjectState& os, - const ObjectState& d_os, - ceph::os::Transaction& trans); interruptible_future<struct stat> stat( CollectionRef c, const ghobject_t& oid) const; @@ -320,7 +315,8 @@ public: CollectionRef c, const ghobject_t& oid, uint64_t off, - uint64_t len); + uint64_t len, + uint32_t op_flags = 0); write_iertr::future<> tmapput( ObjectState& os, @@ -380,11 +376,13 @@ public: object_stat_sum_t& delta_stats); ll_read_ierrorator::future<ceph::bufferlist> omap_get_header( const crimson::os::CollectionRef& c, - const ghobject_t& oid) const; + const ghobject_t& oid, + uint32_t op_flags = 0) const; ll_read_ierrorator::future<> omap_get_header( const ObjectState& os, OSDOp& osd_op, - object_stat_sum_t& delta_stats) const; + object_stat_sum_t& delta_stats, + uint32_t op_flags = 0) const; interruptible_future<> omap_set_header( ObjectState& os, const OSDOp& osd_op, @@ -411,6 +409,20 @@ public: ceph::os::Transaction& trans, osd_op_params_t& osd_op_params, object_stat_sum_t& delta_stats); + + /// sets oi and (for head) ss attrs + void set_metadata( + const hobject_t &obj, + object_info_t &oi, + const SnapSet *ss /* non-null iff head */, + ceph::os::Transaction& trans); + + /// clone from->to and clear ss attribute on to + void clone_for_write( + const hobject_t &from, + const hobject_t &to, + ceph::os::Transaction& trans); + virtual rep_op_fut_t submit_transaction(const std::set<pg_shard_t> &pg_shards, const hobject_t& hoid, diff --git a/src/crimson/osd/pg_recovery.cc b/src/crimson/osd/pg_recovery.cc index ec3af0d2b00..5eef584c776 100644 --- a/src/crimson/osd/pg_recovery.cc +++ b/src/crimson/osd/pg_recovery.cc @@ -67,8 +67,6 @@ PGRecovery::start_recovery_ops( if (max_to_start > 0) { max_to_start -= start_replica_recovery_ops(trigger, max_to_start, &started); } - using interruptor = - crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>; return interruptor::parallel_for_each(started, [] (auto&& ifut) { return std::move(ifut); @@ -609,8 +607,21 @@ void PGRecovery::update_peers_last_backfill( bool PGRecovery::budget_available() const { - // TODO: the limits! - return true; + crimson::osd::scheduler::params_t params = + {1, 0, crimson::osd::scheduler::scheduler_class_t::background_best_effort}; + auto &ss = pg->get_shard_services(); + auto futopt = ss.try_acquire_throttle_now(std::move(params)); + if (!futopt) { + return true; + } + std::ignore = interruptor::make_interruptible(std::move(*futopt) + ).then_interruptible([this] { + assert(!backfill_state->is_triggered()); + using BackfillState = crimson::osd::BackfillState; + backfill_state->process_event( + BackfillState::ThrottleAcquired{}.intrusive_from_this()); + }); + return false; } void PGRecovery::on_pg_clean() diff --git a/src/crimson/osd/pg_recovery.h b/src/crimson/osd/pg_recovery.h index 657e6d3e888..5c7b5c5ef2b 100644 --- a/src/crimson/osd/pg_recovery.h +++ b/src/crimson/osd/pg_recovery.h @@ -25,6 +25,8 @@ class PGBackend; class PGRecovery : public crimson::osd::BackfillState::BackfillListener { public: + using interruptor = + crimson::interruptible::interruptor<crimson::osd::IOInterruptCondition>; template <typename T = void> using interruptible_future = RecoveryBackend::interruptible_future<T>; PGRecovery(PGRecoveryListener* pg) : pg(pg) {} diff --git a/src/crimson/osd/pg_shard_manager.h b/src/crimson/osd/pg_shard_manager.h index b9879c8c9dd..f7bd7a6c08e 100644 --- a/src/crimson/osd/pg_shard_manager.h +++ b/src/crimson/osd/pg_shard_manager.h @@ -256,18 +256,40 @@ public: auto &opref = *op; return opref.template with_blocking_event< PGMap::PGCreationBlockingEvent - >([&target_shard_services, &opref](auto &&trigger) { - return target_shard_services.wait_for_pg( - std::move(trigger), opref.get_pgid()); - }).safe_then([&logger, &target_shard_services, &opref](Ref<PG> pgref) { - logger.debug("{}: have_pg", opref); - return opref.with_pg(target_shard_services, pgref); - }).handle_error( - crimson::ct_error::ecanceled::handle([&logger, &opref](auto) { - logger.debug("{}: pg creation canceled, dropping", opref); - return seastar::now(); - }) - ).then([op=std::move(op)] {}); + >([&target_shard_services, &opref, &logger](auto &&trigger) mutable { + auto pg = target_shard_services.get_pg(opref.get_pgid()); + auto fut = ShardServices::wait_for_pg_ertr::make_ready_future<Ref<PG>>(pg); + if (!pg) { + if (opref.requires_pg()) { + auto osdmap = target_shard_services.get_map(); + if (!osdmap->is_up_acting_osd_shard( + opref.get_pgid(), target_shard_services.local_state.whoami)) { + logger.debug( + "pg {} for {} is no longer here, discarding", + opref.get_pgid(), opref); + opref.get_handle().exit(); + auto _fut = seastar::now(); + if (osdmap->get_epoch() > opref.get_epoch_sent_at()) { + _fut = target_shard_services.send_incremental_map( + std::ref(opref.get_foreign_connection()), + opref.get_epoch_sent_at() + 1); + } + return _fut; + } + } + fut = target_shard_services.wait_for_pg( + std::move(trigger), opref.get_pgid()); + } + return fut.safe_then([&logger, &target_shard_services, &opref](Ref<PG> pgref) { + logger.debug("{}: have_pg", opref); + return opref.with_pg(target_shard_services, pgref); + }).handle_error( + crimson::ct_error::ecanceled::handle([&logger, &opref](auto) { + logger.debug("{}: pg creation canceled, dropping", opref); + return seastar::now(); + }) + ); + }).then([op=std::move(op)] {}); } seastar::future<> load_pgs(crimson::os::FuturizedStore& store); diff --git a/src/crimson/osd/replicated_backend.cc b/src/crimson/osd/replicated_backend.cc index f09cd147ea9..6c8abecffaf 100644 --- a/src/crimson/osd/replicated_backend.cc +++ b/src/crimson/osd/replicated_backend.cc @@ -96,11 +96,18 @@ ReplicatedBackend::submit_transaction( bufferlist encoded_txn; encode(txn, encoded_txn); + bool is_delete = false; for (auto &le : log_entries) { le.mark_unrollbackable(); + if (le.is_delete()) { + is_delete = true; + } } + co_await pg.update_snap_map(log_entries, txn); + std::vector<pg_shard_t> to_push_clone; + std::vector<pg_shard_t> to_push_delete; auto sends = std::make_unique<std::vector<seastar::future<>>>(); for (auto &pg_shard : pg_shards) { if (pg_shard == whoami) { @@ -115,12 +122,17 @@ ReplicatedBackend::submit_transaction( m = new_repop_msg( pg_shard, hoid, encoded_txn, osd_op_p, min_epoch, map_epoch, log_entries, false, tid); - if (_new_clone && pg.is_missing_on_peer(pg_shard, hoid)) { - // The head is in the push queue but hasn't been pushed yet. - // We need to ensure that the newly created clone will be - // pushed as well, otherwise we might skip it. - // See: https://tracker.ceph.com/issues/68808 - to_push_clone.push_back(pg_shard); + if (pg.is_missing_on_peer(pg_shard, hoid)) { + if (_new_clone) { + // The head is in the push queue but hasn't been pushed yet. + // We need to ensure that the newly created clone will be + // pushed as well, otherwise we might skip it. + // See: https://tracker.ceph.com/issues/68808 + to_push_clone.push_back(pg_shard); + } + if (is_delete) { + to_push_delete.push_back(pg_shard); + } } } pending_txn->second.acked_peers.push_back({pg_shard, eversion_t{}}); @@ -130,8 +142,6 @@ ReplicatedBackend::submit_transaction( pg_shard.osd, std::move(m), map_epoch)); } - co_await pg.update_snap_map(log_entries, txn); - pg.log_operation( std::move(log_entries), osd_op_p.pg_trim_to, @@ -157,7 +167,8 @@ ReplicatedBackend::submit_transaction( return seastar::now(); } return peers->all_committed.get_shared_future(); - }).then_interruptible([pending_txn, this, _new_clone, + }).then_interruptible([pending_txn, this, _new_clone, &hoid, + to_push_delete=std::move(to_push_delete), to_push_clone=std::move(to_push_clone)] { auto acked_peers = std::move(pending_txn->second.acked_peers); pending_trans.erase(pending_txn); @@ -167,6 +178,9 @@ ReplicatedBackend::submit_transaction( _new_clone->obs.oi.version, to_push_clone); } + if (!to_push_delete.empty()) { + pg.enqueue_delete_for_backfill(hoid, {}, to_push_delete); + } return seastar::make_ready_future< crimson::osd::acked_peers_t>(std::move(acked_peers)); }); diff --git a/src/crimson/osd/replicated_recovery_backend.cc b/src/crimson/osd/replicated_recovery_backend.cc index 76f24196b51..0d6c9d38236 100644 --- a/src/crimson/osd/replicated_recovery_backend.cc +++ b/src/crimson/osd/replicated_recovery_backend.cc @@ -35,6 +35,15 @@ ReplicatedRecoveryBackend::recover_object( logger().debug("recover_object: loading obc: {}", soid); return pg.obc_loader.with_obc<RWState::RWREAD>(soid, [this, soid, need](auto head, auto obc) { + if (!obc->obs.exists) { + // XXX: this recovery must be triggered by backfills and the corresponding + // object must have been deleted by some client request after the object + // is enqueued for push but before the lock is acquired by the recovery. + // + // Abort the recovery in this case, a "recover_delete" must have been + // added for this object by the client request that deleted it. + return interruptor::now(); + } logger().debug("recover_object: loaded obc: {}", obc->obs.oi.soid); auto& recovery_waiter = get_recovering(soid); recovery_waiter.obc = obc; @@ -306,7 +315,10 @@ ReplicatedRecoveryBackend::recover_delete( } return seastar::make_ready_future<>(); }).then_interruptible([this, soid, &stat_diff] { - pg.get_recovery_handler()->on_global_recover(soid, stat_diff, true); + const auto &missing = pg.get_peering_state().get_pg_log().get_missing(); + if (!missing.is_missing(soid)) { + pg.get_recovery_handler()->on_global_recover(soid, stat_diff, true); + } return seastar::make_ready_future<>(); }); }); @@ -568,14 +580,17 @@ ReplicatedRecoveryBackend::read_metadata_for_push_op( return seastar::make_ready_future<eversion_t>(ver); } return interruptor::make_interruptible(interruptor::when_all_succeed( - backend->omap_get_header(coll, ghobject_t(oid)).handle_error_interruptible<false>( + backend->omap_get_header( + coll, ghobject_t(oid), CEPH_OSD_OP_FLAG_FADVISE_DONTNEED + ).handle_error_interruptible<false>( crimson::os::FuturizedStore::Shard::read_errorator::all_same_way( [oid] (const std::error_code& e) { logger().debug("read_metadata_for_push_op, error {} when getting omap header: {}", e, oid); return seastar::make_ready_future<bufferlist>(); })), - interruptor::make_interruptible(store->get_attrs(coll, ghobject_t(oid))) - .handle_error_interruptible<false>( + interruptor::make_interruptible( + store->get_attrs(coll, ghobject_t(oid), CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) + ).handle_error_interruptible<false>( crimson::os::FuturizedStore::Shard::get_attrs_ertr::all_same_way( [oid] (const std::error_code& e) { logger().debug("read_metadata_for_push_op, error {} when getting attrs: {}", e, oid); @@ -613,8 +628,14 @@ ReplicatedRecoveryBackend::read_object_for_push_op( return seastar::make_ready_future<uint64_t>(offset); } // 1. get the extents in the interested range - return interruptor::make_interruptible(backend->fiemap(coll, ghobject_t{oid}, - 0, copy_subset.range_end())).safe_then_interruptible( + return interruptor::make_interruptible( + backend->fiemap( + coll, + ghobject_t{oid}, + 0, + copy_subset.range_end(), + CEPH_OSD_OP_FLAG_FADVISE_DONTNEED) + ).safe_then_interruptible( [=, this](auto&& fiemap_included) mutable { interval_set<uint64_t> extents; try { @@ -630,8 +651,12 @@ ReplicatedRecoveryBackend::read_object_for_push_op( push_op->data_included.span_of(extents, offset, max_len); // 3. read the truncated extents // TODO: check if the returned extents are pruned - return interruptor::make_interruptible(store->readv(coll, ghobject_t{oid}, - push_op->data_included, 0)); + return interruptor::make_interruptible( + store->readv( + coll, + ghobject_t{oid}, + push_op->data_included, + CEPH_OSD_OP_FLAG_FADVISE_DONTNEED)); }).safe_then_interruptible([push_op, range_end=copy_subset.range_end()](auto &&bl) { push_op->data.claim_append(std::move(bl)); uint64_t recovered_to = 0; diff --git a/src/crimson/osd/shard_services.cc b/src/crimson/osd/shard_services.cc index c2340898929..e1acb34636f 100644 --- a/src/crimson/osd/shard_services.cc +++ b/src/crimson/osd/shard_services.cc @@ -783,6 +783,11 @@ seastar::future<> ShardServices::dispatch_context_transaction( co_return; } +Ref<PG> ShardServices::get_pg(spg_t pgid) +{ + return local_state.get_pg(pgid); +} + seastar::future<> ShardServices::dispatch_context_messages( BufferedRecoveryMessages &&ctx) { diff --git a/src/crimson/osd/shard_services.h b/src/crimson/osd/shard_services.h index fb86418aba2..f1ed9b8d911 100644 --- a/src/crimson/osd/shard_services.h +++ b/src/crimson/osd/shard_services.h @@ -10,6 +10,7 @@ #include "include/common_fwd.h" #include "osd_operation.h" +#include "osd/osd_types_fmt.h" #include "msg/MessageRef.h" #include "crimson/common/exception.h" #include "crimson/common/shared_lru.h" @@ -482,6 +483,8 @@ public: return pg_to_shard_mapping.remove_pg_mapping(pgid); } + Ref<PG> get_pg(spg_t pgid); + crimson::common::CephContext *get_cct() { return &(local_state.cct); } @@ -588,6 +591,7 @@ public: FORWARD_TO_OSD_SINGLETON(get_pool_info) FORWARD(with_throttle_while, with_throttle_while, local_state.throttler) + FORWARD(try_acquire_throttle_now, try_acquire_throttle_now, local_state.throttler) FORWARD_TO_OSD_SINGLETON(build_incremental_map_msg) FORWARD_TO_OSD_SINGLETON(send_incremental_map) diff --git a/src/crimson/tools/perf_crimson_msgr.cc b/src/crimson/tools/perf_crimson_msgr.cc index e5f56361fff..5623438f821 100644 --- a/src/crimson/tools/perf_crimson_msgr.cc +++ b/src/crimson/tools/perf_crimson_msgr.cc @@ -1,6 +1,7 @@ // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab +#include <iomanip> #include <map> #include <boost/program_options.hpp> #include <boost/iterator/counting_iterator.hpp> diff --git a/src/crimson/tools/store_nbd/tm_driver.cc b/src/crimson/tools/store_nbd/tm_driver.cc index 389ecd78afc..870809c5153 100644 --- a/src/crimson/tools/store_nbd/tm_driver.cc +++ b/src/crimson/tools/store_nbd/tm_driver.cc @@ -25,6 +25,7 @@ seastar::future<> TMDriver::write( return tm->with_transaction_intr( Transaction::src_t::MUTATE, "write", + CACHE_HINT_TOUCH, [this, offset, &ptr](auto& t) { return tm->remove(t, laddr_t::from_byte_offset(offset) @@ -112,6 +113,7 @@ seastar::future<bufferlist> TMDriver::read( return tm->with_transaction_intr( Transaction::src_t::READ, "read", + CACHE_HINT_TOUCH, [=, &blret, this](auto& t) { return read_extents(t, laddr_t::from_byte_offset(offset), size diff --git a/src/erasure-code/isa/ErasureCodeIsa.cc b/src/erasure-code/isa/ErasureCodeIsa.cc index 1548139756b..7c28bbb0a6a 100644 --- a/src/erasure-code/isa/ErasureCodeIsa.cc +++ b/src/erasure-code/isa/ErasureCodeIsa.cc @@ -117,16 +117,48 @@ int ErasureCodeIsa::decode_chunks(const set<int> &want_to_read, // ----------------------------------------------------------------------------- void +ErasureCodeIsa::isa_xor(char **data, char **coding, int blocksize) +{ + // If addresses are aligned to 32 bytes, then we can use xor_gen() + // Otherwise, use byte_xor() + int i; + bool src_aligned = true; + + for (i = 0; i < k; i++) { + src_aligned &= is_aligned(data[i], EC_ISA_ADDRESS_ALIGNMENT); + } + + if (src_aligned && is_aligned(coding[0], EC_ISA_ADDRESS_ALIGNMENT)) { + xor_gen(k+1, blocksize, (void**) data); + } + else { + memcpy(coding[0], data[0], blocksize); + for (i = 1; i < k; i++) { + byte_xor(data[i], coding[0], data[i]+blocksize); + } + } +} + +void +ErasureCodeIsa::byte_xor(char *data, char *coding, char *data_end) +{ + while (data < data_end) + *coding++ ^= *data++; +} + +// ----------------------------------------------------------------------------- + +void ErasureCodeIsaDefault::isa_encode(char **data, char **coding, int blocksize) { - if (m == 1) - // single parity stripe - xor_gen(k+m, blocksize, (void**) data); - else + if (m == 1) { + isa_xor(data, coding, blocksize); + } else { ec_encode_data(blocksize, k, m, encode_tbls, (unsigned char**) data, (unsigned char**) coding); + } } // ----------------------------------------------------------------------------- @@ -158,7 +190,7 @@ ErasureCodeIsaDefault::isa_decode(int *erasures, unsigned char *recover_source[k]; unsigned char *recover_target[m]; - unsigned char *recover_buf[k+1]; + char *recover_buf[k+1]; // count the errors for (int l = 0; erasures[l] != -1; l++) { @@ -181,18 +213,18 @@ ErasureCodeIsaDefault::isa_decode(int *erasures, for (i = 0; i < (k + 1); i++) { if (erasure_contains(erasures, i)) { if (i < k) { - recover_buf[i] = (unsigned char*) coding[0]; - recover_buf[k] = (unsigned char*) data[i]; + recover_buf[i] = coding[0]; + recover_buf[k] = data[i]; parity_set = true; } else { - recover_buf[i] = (unsigned char*) coding[0]; + recover_buf[i] = coding[0]; } } else { if (i < k) { - recover_buf[i] = (unsigned char*) data[i]; + recover_buf[i] = data[i]; } else { if (!parity_set) { - recover_buf[i] = (unsigned char*) coding[0]; + recover_buf[i] = coding[0]; } } } @@ -230,7 +262,7 @@ ErasureCodeIsaDefault::isa_decode(int *erasures, ((matrixtype == kVandermonde) && (nerrs == 1) && (erasures[0] < (k + 1)))) { // single parity decoding dout(20) << "isa_decode: reconstruct using xor_gen [" << erasures[0] << "]" << dendl; - xor_gen(k+1, blocksize, (void **) recover_buf); + isa_xor(recover_buf, &recover_buf[k], blocksize); return 0; } diff --git a/src/erasure-code/isa/ErasureCodeIsa.h b/src/erasure-code/isa/ErasureCodeIsa.h index 85f1cd9cb46..4d338c8b418 100644 --- a/src/erasure-code/isa/ErasureCodeIsa.h +++ b/src/erasure-code/isa/ErasureCodeIsa.h @@ -32,6 +32,9 @@ #define EC_ISA_ADDRESS_ALIGNMENT 32u +#define is_aligned(POINTER, BYTE_COUNT) \ + (((uintptr_t)(const void *)(POINTER)) % (BYTE_COUNT) == 0) + class ErasureCodeIsa : public ceph::ErasureCode { public: @@ -84,6 +87,10 @@ public: int init(ceph::ErasureCodeProfile &profile, std::ostream *ss) override; + void isa_xor(char **data, char **coding, int blocksize); + + void byte_xor(char *data, char *coding, char *data_end); + virtual void isa_encode(char **data, char **coding, int blocksize) = 0; diff --git a/src/exporter/ceph_exporter.cc b/src/exporter/ceph_exporter.cc index 44b67c7e615..2232851c094 100644 --- a/src/exporter/ceph_exporter.cc +++ b/src/exporter/ceph_exporter.cc @@ -30,13 +30,13 @@ static void handle_signal(int signum) static void usage() { std::cout << "usage: ceph-exporter [options]\n" << "options:\n" - " --sock-dir: The path to ceph daemons socket files dir\n" - " --addrs: Host ip address where exporter is deployed\n" - " --port: Port to deploy exporter on. Default is 9926\n" - " --cert-file: Path to the certificate file to use https\n" - " --key-file: Path to the certificate key file to use https\n" + " --sock-dir: The path to Ceph daemon sockets (*.asok)\n" + " --addrs: Host IP address on which the exporter is to listen\n" + " --port: TCP Port on which the exporter is to listen. Default is 9926\n" + " --cert-file: Path to the certificate file when using HTTPS\n" + " --key-file: Path to the certificate key file when using HTTPS\n" " --prio-limit: Only perf counters greater than or equal to prio-limit are fetched. Default: 5\n" - " --stats-period: Time to wait before sending requests again to exporter server (seconds). Default: 5s" + " --stats-period: Interval between daemon scrapes (seconds). Default: 5s" << std::endl; generic_server_usage(); } diff --git a/src/include/cephfs/ceph_ll_client.h b/src/include/cephfs/ceph_ll_client.h index ac5b7c22471..458edce590b 100644 --- a/src/include/cephfs/ceph_ll_client.h +++ b/src/include/cephfs/ceph_ll_client.h @@ -110,6 +110,9 @@ struct ceph_statx { * others in the future, we disallow setting any that aren't recognized. */ #define CEPH_REQ_FLAG_MASK (AT_SYMLINK_NOFOLLOW|AT_STATX_DONT_SYNC) +#if defined(__linux__) && defined(AT_EMPTY_PATH) +#define CEPH_AT_EMPTY_PATH (CEPH_REQ_FLAG_MASK|AT_EMPTY_PATH) +#endif /* fallocate mode flags */ #ifndef FALLOC_FL_KEEP_SIZE diff --git a/src/include/cephfs/libcephfs.h b/src/include/cephfs/libcephfs.h index ba0b76e072b..f26eabfbd5c 100644 --- a/src/include/cephfs/libcephfs.h +++ b/src/include/cephfs/libcephfs.h @@ -937,7 +937,7 @@ int ceph_fstatx(struct ceph_mount_info *cmount, int fd, struct ceph_statx *stx, * @param relpath to the file/directory to get statistics of * @param stx the ceph_statx struct that will be filled in with the file's statistics. * @param want bitfield of CEPH_STATX_* flags showing designed attributes - * @param flags bitfield that can be used to set AT_* modifier flags (AT_STATX_SYNC_AS_STAT, AT_STATX_FORCE_SYNC, AT_STATX_DONT_SYNC and AT_SYMLINK_NOFOLLOW) + * @param flags bitfield that can be used to set AT_* modifier flags (AT_STATX_DONT_SYNC, AT_SYMLINK_NOFOLLOW and AT_EMPTY_PATH) * @returns 0 on success or negative error code on failure. */ int ceph_statxat(struct ceph_mount_info *cmount, int dirfd, const char *relpath, @@ -1104,7 +1104,7 @@ int ceph_lchown(struct ceph_mount_info *cmount, const char *path, int uid, int g * @param relpath the relpath of the file/directory to change the ownership of. * @param uid the user id to set on the file/directory. * @param gid the group id to set on the file/directory. - * @param flags bitfield that can be used to set AT_* modifier flags (AT_SYMLINK_NOFOLLOW) + * @param flags bitfield that can be used to set AT_* modifier flags (AT_SYMLINK_NOFOLLOW and AT_EMPTY_PATH) * @returns 0 on success or negative error code on failure. */ int ceph_chownat(struct ceph_mount_info *cmount, int dirfd, const char *relpath, diff --git a/src/include/rados/librados.hpp b/src/include/rados/librados.hpp index 4a7ac3ea6e0..0f5a9036eff 100644 --- a/src/include/rados/librados.hpp +++ b/src/include/rados/librados.hpp @@ -202,6 +202,8 @@ inline namespace v14_2_0 { int set_complete_callback(void *cb_arg, callback_t cb); int set_safe_callback(void *cb_arg, callback_t cb) __attribute__ ((deprecated)); + /// Request immediate cancellation as if by IoCtx::aio_cancel(). + int cancel(); int wait_for_complete(); int wait_for_safe() __attribute__ ((deprecated)); int wait_for_complete_and_cb(); @@ -772,17 +774,30 @@ inline namespace v14_2_0 { void tier_evict(); }; - /* IoCtx : This is a context in which we can perform I/O. - * It includes a Pool, + /** + * @brief A handle to a RADOS pool used to perform I/O operations. * * Typical use (error checking omitted): - * + * @code * IoCtx p; * rados.ioctx_create("my_pool", p); - * p->stat(&stats); - * ... etc ... + * p.stat("my_object", &size, &mtime); + * @endcode + * + * IoCtx holds a pointer to its underlying implementation. The dup() + * method performs a deep copy of this implementation, but the copy + * construction and assignment operations perform shallow copies by + * sharing that pointer. + * + * Function names starting with aio_ are asynchronous operations that + * return immediately after submitting a request, and whose completions + * are managed by the given AioCompletion pointer. The IoCtx's underlying + * implementation is involved in the delivery of these completions, so + * the caller must guarantee that its lifetime is preserved until then - + * if not by preserving the IoCtx instance that submitted the request, + * then by a copied/moved instance that shares the same implementation. * - * NOTE: be sure to call watch_flush() prior to destroying any IoCtx + * @note Be sure to call watch_flush() prior to destroying any IoCtx * that is used for watch events to ensure that racing callbacks * have completed. */ @@ -791,9 +806,13 @@ inline namespace v14_2_0 { public: IoCtx(); static void from_rados_ioctx_t(rados_ioctx_t p, IoCtx &pool); + /// Construct a shallow copy of rhs, sharing its underlying implementation. IoCtx(const IoCtx& rhs); + /// Assign a shallow copy of rhs, sharing its underlying implementation. IoCtx& operator=(const IoCtx& rhs); + /// Move construct from rhs, transferring its underlying implementation. IoCtx(IoCtx&& rhs) noexcept; + /// Move assign from rhs, transferring its underlying implementation. IoCtx& operator=(IoCtx&& rhs) noexcept; ~IoCtx(); @@ -1150,7 +1169,8 @@ inline namespace v14_2_0 { int aio_stat2(const std::string& oid, AioCompletion *c, uint64_t *psize, struct timespec *pts); /** - * Cancel aio operation + * Request immediate cancellation with error code -ECANCELED + * if the operation hasn't already completed. * * @param c completion handle * @returns 0 on success, negative error code on failure diff --git a/src/include/random.h b/src/include/random.h index f2e3e37bcd7..6b7c9405efd 100644 --- a/src/include/random.h +++ b/src/include/random.h @@ -16,9 +16,9 @@ #define CEPH_RANDOM_H 1 #include <mutex> +#include <optional> #include <random> #include <type_traits> -#include <boost/optional.hpp> // Workaround for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=85494 #ifdef __MINGW32__ @@ -123,7 +123,7 @@ void randomize_rng() template <typename EngineT> EngineT& engine() { - thread_local boost::optional<EngineT> rng_engine; + thread_local std::optional<EngineT> rng_engine; if (!rng_engine) { rng_engine.emplace(EngineT()); diff --git a/src/json_spirit/CMakeLists.txt b/src/json_spirit/CMakeLists.txt index b3b5ce2e6f2..681ac909e63 100644 --- a/src/json_spirit/CMakeLists.txt +++ b/src/json_spirit/CMakeLists.txt @@ -1,4 +1,4 @@ add_library(json_spirit STATIC json_spirit_reader.cpp json_spirit_writer.cpp) -target_link_libraries(json_spirit common_utf8) +target_link_libraries(json_spirit common_utf8 Boost::thread) diff --git a/src/kv/KeyValueDB.h b/src/kv/KeyValueDB.h index 858742d511e..d926840180e 100644 --- a/src/kv/KeyValueDB.h +++ b/src/kv/KeyValueDB.h @@ -9,6 +9,7 @@ #include <map> #include <optional> #include <string> +#include <string_view> #include <boost/scoped_ptr.hpp> #include "include/encoding.h" #include "common/Formatter.h" @@ -211,6 +212,10 @@ public: return ""; } virtual ceph::buffer::list value() = 0; + // When valid() returns true, value returned as string-view + // is guaranteed to be valid until iterator is moved to another + // position; that is until call to next() / seek_to_first() / etc. + virtual std::string_view value_as_sv() = 0; virtual int status() = 0; virtual ~SimplestIteratorImpl() {} }; @@ -220,7 +225,12 @@ public: virtual ~IteratorImpl() {} virtual int seek_to_last() = 0; virtual int prev() = 0; + // When valid() returns true, key returned as string-view + // is guaranteed to be valid until iterator is moved to another + // position; that is until call to next() / seek_to_first() / etc. + virtual std::string_view key_as_sv() = 0; virtual std::pair<std::string, std::string> raw_key() = 0; + virtual std::pair<std::string_view, std::string_view> raw_key_as_sv() = 0; virtual ceph::buffer::ptr value_as_ptr() { ceph::buffer::list bl = value(); if (bl.length() == 1) { @@ -247,7 +257,9 @@ public: virtual int next() = 0; virtual int prev() = 0; virtual std::string key() = 0; + virtual std::string_view key_as_sv() = 0; virtual std::pair<std::string,std::string> raw_key() = 0; + virtual std::pair<std::string_view, std::string_view> raw_key_as_sv() = 0; virtual bool raw_key_is_prefixed(const std::string &prefix) = 0; virtual ceph::buffer::list value() = 0; virtual ceph::buffer::ptr value_as_ptr() { @@ -258,6 +270,7 @@ public: return ceph::buffer::ptr(); } } + virtual std::string_view value_as_sv() = 0; virtual int status() = 0; virtual size_t key_size() { return 0; @@ -315,15 +328,24 @@ private: std::string key() override { return generic_iter->key(); } + std::string_view key_as_sv() override { + return generic_iter->key_as_sv(); + } std::pair<std::string, std::string> raw_key() override { return generic_iter->raw_key(); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return generic_iter->raw_key_as_sv(); + } ceph::buffer::list value() override { return generic_iter->value(); } ceph::buffer::ptr value_as_ptr() override { return generic_iter->value_as_ptr(); } + std::string_view value_as_sv() override { + return generic_iter->value_as_sv(); + } int status() override { return generic_iter->status(); } diff --git a/src/kv/RocksDBStore.cc b/src/kv/RocksDBStore.cc index ca63ea06484..51d224b67c0 100644 --- a/src/kv/RocksDBStore.cc +++ b/src/kv/RocksDBStore.cc @@ -6,6 +6,7 @@ #include <memory> #include <set> #include <string> +#include <string_view> #include <errno.h> #include <unistd.h> #include <sys/types.h> @@ -47,6 +48,7 @@ using std::ostream; using std::pair; using std::set; using std::string; +using std::string_view; using std::unique_ptr; using std::vector; @@ -1992,7 +1994,7 @@ int RocksDBStore::split_key(rocksdb::Slice in, string *prefix, string *key) // Find separator inside Slice char* separator = (char*) memchr(in.data(), 0, in.size()); - if (separator == NULL) + if (separator == nullptr) return -EINVAL; prefix_len = size_t(separator - in.data()); if (prefix_len >= in.size()) @@ -2006,6 +2008,27 @@ int RocksDBStore::split_key(rocksdb::Slice in, string *prefix, string *key) return 0; } +// TODO: deduplicate the code, preferrably by removing the string variant +int RocksDBStore::split_key(rocksdb::Slice in, string_view *prefix, string_view *key) +{ + size_t prefix_len = 0; + + // Find separator inside Slice + char* separator = (char*) memchr(in.data(), 0, in.size()); + if (separator == nullptr) + return -EINVAL; + prefix_len = size_t(separator - in.data()); + if (prefix_len >= in.size()) + return -EINVAL; + + // Fetch prefix and/or key directly from Slice + if (prefix) + *prefix = string_view(in.data(), prefix_len); + if (key) + *key = string_view(separator + 1, in.size() - prefix_len - 1); + return 0; +} + void RocksDBStore::compact() { dout(2) << __func__ << " starting" << dendl; @@ -2226,7 +2249,13 @@ int RocksDBStore::RocksDBWholeSpaceIteratorImpl::prev() string RocksDBStore::RocksDBWholeSpaceIteratorImpl::key() { string out_key; - split_key(dbiter->key(), 0, &out_key); + split_key(dbiter->key(), nullptr, &out_key); + return out_key; +} +string_view RocksDBStore::RocksDBWholeSpaceIteratorImpl::key_as_sv() +{ + string_view out_key; + split_key(dbiter->key(), nullptr, &out_key); return out_key; } pair<string,string> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key() @@ -2235,6 +2264,12 @@ pair<string,string> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key() split_key(dbiter->key(), &prefix, &key); return make_pair(prefix, key); } +pair<string_view,string_view> RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key_as_sv() +{ + string_view prefix, key; + split_key(dbiter->key(), &prefix, &key); + return make_pair(prefix, key); +} bool RocksDBStore::RocksDBWholeSpaceIteratorImpl::raw_key_is_prefixed(const string &prefix) { // Look for "prefix\0" right in rocksb::Slice @@ -2267,6 +2302,12 @@ bufferptr RocksDBStore::RocksDBWholeSpaceIteratorImpl::value_as_ptr() return bufferptr(val.data(), val.size()); } +std::string_view RocksDBStore::RocksDBWholeSpaceIteratorImpl::value_as_sv() +{ + rocksdb::Slice val = dbiter->value(); + return std::string_view{val.data(), val.size()}; +} + int RocksDBStore::RocksDBWholeSpaceIteratorImpl::status() { return dbiter->status().ok() ? 0 : -1; @@ -2348,9 +2389,15 @@ public: string key() override { return dbiter->key().ToString(); } + string_view key_as_sv() override { + return dbiter->key().ToStringView(); + } std::pair<std::string, std::string> raw_key() override { return make_pair(prefix, key()); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return make_pair(prefix, dbiter->key().ToStringView()); + } bufferlist value() override { return to_bufferlist(dbiter->value()); } @@ -2358,6 +2405,10 @@ public: rocksdb::Slice val = dbiter->value(); return bufferptr(val.data(), val.size()); } + std::string_view value_as_sv() override { + rocksdb::Slice val = dbiter->value(); + return std::string_view{val.data(), val.size()}; + } int status() override { return dbiter->status().ok() ? 0 : -1; } @@ -2668,6 +2719,15 @@ public: } } + std::string_view key_as_sv() override + { + if (smaller == on_main) { + return main->key_as_sv(); + } else { + return current_shard->second->key_as_sv(); + } + } + std::pair<std::string,std::string> raw_key() override { if (smaller == on_main) { @@ -2677,6 +2737,15 @@ public: } } + std::pair<std::string_view,std::string_view> raw_key_as_sv() override + { + if (smaller == on_main) { + return main->raw_key_as_sv(); + } else { + return { current_shard->first, current_shard->second->key_as_sv() }; + } + } + bool raw_key_is_prefixed(const std::string &prefix) override { if (smaller == on_main) { @@ -2695,6 +2764,15 @@ public: } } + std::string_view value_as_sv() override + { + if (smaller == on_main) { + return main->value_as_sv(); + } else { + return current_shard->second->value_as_sv(); + } + } + int status() override { //because we already had to inspect key, it must be ok @@ -3017,9 +3095,15 @@ public: string key() override { return iters[0]->key().ToString(); } + string_view key_as_sv() override { + return iters[0]->key().ToStringView(); + } std::pair<std::string, std::string> raw_key() override { return make_pair(prefix, key()); } + std::pair<std::string_view, std::string_view> raw_key_as_sv() override { + return make_pair(prefix, iters[0]->key().ToStringView()); + } bufferlist value() override { return to_bufferlist(iters[0]->value()); } @@ -3027,6 +3111,10 @@ public: rocksdb::Slice val = iters[0]->value(); return bufferptr(val.data(), val.size()); } + std::string_view value_as_sv() override { + rocksdb::Slice val = iters[0]->value(); + return std::string_view{val.data(), val.size()}; + } int status() override { return iters[0]->status().ok() ? 0 : -1; } diff --git a/src/kv/RocksDBStore.h b/src/kv/RocksDBStore.h index 477b209854c..50b91be2bf6 100644 --- a/src/kv/RocksDBStore.h +++ b/src/kv/RocksDBStore.h @@ -386,10 +386,13 @@ public: int next() override; int prev() override; std::string key() override; + std::string_view key_as_sv() override; std::pair<std::string,std::string> raw_key() override; + std::pair<std::string_view,std::string_view> raw_key_as_sv() override; bool raw_key_is_prefixed(const std::string &prefix) override; ceph::bufferlist value() override; ceph::bufferptr value_as_ptr() override; + std::string_view value_as_sv() override; int status() override; size_t key_size() override; size_t value_size() override; @@ -419,6 +422,7 @@ public: } static int split_key(rocksdb::Slice in, std::string *prefix, std::string *key); + static int split_key(rocksdb::Slice in, std::string_view *prefix, std::string_view *key); static std::string past_prefix(const std::string &prefix); diff --git a/src/libcephfs.cc b/src/libcephfs.cc index 7eea6665f61..60da6145787 100644 --- a/src/libcephfs.cc +++ b/src/libcephfs.cc @@ -982,7 +982,11 @@ extern "C" int ceph_statxat(struct ceph_mount_info *cmount, int dirfd, const cha { if (!cmount->is_mounted()) return -CEPHFS_ENOTCONN; +#ifdef CEPH_AT_EMPTY_PATH + if (flags & ~CEPH_AT_EMPTY_PATH) +#else if (flags & ~CEPH_REQ_FLAG_MASK) +#endif return -CEPHFS_EINVAL; return cmount->get_client()->statxat(dirfd, relpath, stx, cmount->default_perms, want, flags); diff --git a/src/libcephfs_proxy/CMakeLists.txt b/src/libcephfs_proxy/CMakeLists.txt new file mode 100644 index 00000000000..e19841241e7 --- /dev/null +++ b/src/libcephfs_proxy/CMakeLists.txt @@ -0,0 +1,18 @@ +set(proxy_common_srcs proxy_link.c proxy_log.c) +set(libcephfsd_srcs libcephfsd.c proxy_manager.c proxy_mount.c proxy_helpers.c ${proxy_common_srcs}) +set(libcephfs_proxy_srcs libcephfs_proxy.c ${proxy_common_srcs}) + +add_executable(libcephfsd ${libcephfsd_srcs}) +add_library(cephfs_proxy ${CEPH_SHARED} ${libcephfs_proxy_srcs}) + +target_link_libraries(libcephfsd cephfs ${CRYPTO_LIBS}) + +if(ENABLE_SHARED) + set_target_properties(cephfs_proxy PROPERTIES + OUTPUT_NAME cephfs_proxy + VERSION 2.0.0 + SOVERSION 2) +endif(ENABLE_SHARED) + +install(TARGETS libcephfsd DESTINATION ${CMAKE_INSTALL_SBINDIR}) +install(TARGETS cephfs_proxy DESTINATION ${CMAKE_INSTALL_LIBDIR}) diff --git a/src/libcephfs_proxy/libcephfs_proxy.c b/src/libcephfs_proxy/libcephfs_proxy.c new file mode 100644 index 00000000000..149fae123f7 --- /dev/null +++ b/src/libcephfs_proxy/libcephfs_proxy.c @@ -0,0 +1,869 @@ + +#include <stdlib.h> + +#include "include/cephfs/libcephfs.h" + +#include "proxy_log.h" +#include "proxy_helpers.h" +#include "proxy_requests.h" + +/* We override the definition of the ceph_mount_info structure to contain + * internal proxy information. This is already a black box for libcephfs users, + * so this won't be noticed. */ +struct ceph_mount_info { + proxy_link_t link; + uint64_t cmount; +}; + +/* The global_cmount is used to stablish an initial connection to serve requests + * not related to a real cmount, like ceph_version or ceph_userperm_new. */ +static struct ceph_mount_info global_cmount = { PROXY_LINK_DISCONNECTED, 0 }; + +static bool client_stop(proxy_link_t *link) +{ + return false; +} + +static int32_t proxy_connect(proxy_link_t *link) +{ + CEPH_REQ(hello, req, 0, ans, 0); + char *path, *env; + int32_t sd, err; + + path = PROXY_SOCKET; + env = getenv(PROXY_SOCKET_ENV); + if (env != NULL) { + path = env; + } + + sd = proxy_link_client(link, path, client_stop); + if (sd < 0) { + return sd; + } + + req.id = LIBCEPHFS_LIB_CLIENT; + err = proxy_link_send(sd, req_iov, 1); + if (err < 0) { + goto failed; + } + err = proxy_link_recv(sd, ans_iov, 1); + if (err < 0) { + goto failed; + } + + proxy_log(LOG_INFO, 0, "Connected to libcephfsd version %d.%d", + ans.major, ans.minor); + + if ((ans.major != LIBCEPHFSD_MAJOR) || + (ans.minor != LIBCEPHFSD_MINOR)) { + err = proxy_log(LOG_ERR, ENOTSUP, "Version not supported"); + goto failed; + } + + return sd; + +failed: + proxy_link_close(link); + + return err; +} + +static void proxy_disconnect(proxy_link_t *link) +{ + proxy_link_close(link); +} + +static int32_t proxy_global_connect(void) +{ + int32_t err; + + err = 0; + + if (!proxy_link_is_connected(&global_cmount.link)) { + err = proxy_connect(&global_cmount.link); + } + + return err; +} + +static int32_t proxy_check(struct ceph_mount_info *cmount, int32_t err, + int32_t result) +{ + if (err < 0) { + proxy_disconnect(&cmount->link); + proxy_log(LOG_ERR, err, "Disconnected from libcephfsd"); + + return err; + } + + return result; +} + +/* Macros to simplify communication with the server. */ +#define CEPH_RUN(_cmount, _op, _req, _ans) \ + ({ \ + int32_t __err = \ + CEPH_CALL((_cmount)->link.sd, _op, _req, _ans); \ + __err = proxy_check(_cmount, __err, (_ans).header.result); \ + __err; \ + }) + +#define CEPH_PROCESS(_cmount, _op, _req, _ans) \ + ({ \ + int32_t __err = -ENOTCONN; \ + if (proxy_link_is_connected(&(_cmount)->link)) { \ + (_req).cmount = (_cmount)->cmount; \ + __err = CEPH_RUN(_cmount, _op, _req, _ans); \ + } \ + __err; \ + }) + +__public int ceph_chdir(struct ceph_mount_info *cmount, const char *path) +{ + CEPH_REQ(ceph_chdir, req, 1, ans, 0); + + CEPH_STR_ADD(req, path, path); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CHDIR, req, ans); +} + +__public int ceph_conf_get(struct ceph_mount_info *cmount, const char *option, + char *buf, size_t len) +{ + CEPH_REQ(ceph_conf_get, req, 1, ans, 1); + + req.size = len; + + CEPH_STR_ADD(req, option, option); + CEPH_BUFF_ADD(ans, buf, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_GET, req, ans); +} + +__public int ceph_conf_read_file(struct ceph_mount_info *cmount, + const char *path_list) +{ + CEPH_REQ(ceph_conf_read_file, req, 1, ans, 0); + + CEPH_STR_ADD(req, path, path_list); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_READ_FILE, req, ans); +} + +__public int ceph_conf_set(struct ceph_mount_info *cmount, const char *option, + const char *value) +{ + CEPH_REQ(ceph_conf_set, req, 2, ans, 0); + + CEPH_STR_ADD(req, option, option); + CEPH_STR_ADD(req, value, value); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_CONF_SET, req, ans); +} + +__public int ceph_create(struct ceph_mount_info **cmount, const char *const id) +{ + CEPH_REQ(ceph_create, req, 1, ans, 0); + struct ceph_mount_info *ceph_mount; + int32_t sd, err; + + ceph_mount = proxy_malloc(sizeof(struct ceph_mount_info)); + if (ceph_mount == NULL) { + return -ENOMEM; + } + + err = proxy_connect(&ceph_mount->link); + if (err < 0) { + goto failed; + } + sd = err; + + CEPH_STR_ADD(req, id, id); + + err = CEPH_CALL(sd, LIBCEPHFSD_OP_CREATE, req, ans); + if ((err < 0) || ((err = ans.header.result) < 0)) { + goto failed_link; + } + + ceph_mount->cmount = ans.cmount; + + *cmount = ceph_mount; + + return 0; + +failed_link: + proxy_disconnect(&ceph_mount->link); + +failed: + proxy_free(ceph_mount); + + return err; +} + +__public const char *ceph_getcwd(struct ceph_mount_info *cmount) +{ + static char cwd[PATH_MAX]; + int32_t err; + + CEPH_REQ(ceph_getcwd, req, 0, ans, 1); + + CEPH_BUFF_ADD(ans, cwd, sizeof(cwd)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_GETCWD, req, ans); + if (err >= 0) { + return cwd; + } + + errno = -err; + + return NULL; +} + +__public int ceph_init(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_init, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_INIT, req, ans); +} + +__public int ceph_ll_close(struct ceph_mount_info *cmount, + struct Fh *filehandle) +{ + CEPH_REQ(ceph_ll_close, req, 0, ans, 0); + + req.fh = ptr_value(filehandle); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_CLOSE, req, ans); +} + +__public int ceph_ll_create(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, int oflags, + Inode **outp, Fh **fhp, struct ceph_statx *stx, + unsigned want, unsigned lflags, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_create, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.oflags = oflags; + req.want = want; + req.flags = lflags; + + CEPH_STR_ADD(req, name, name); + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_CREATE, req, ans); + if (err >= 0) { + *outp = value_ptr(ans.inode); + *fhp = value_ptr(ans.fh); + } + + return err; +} + +__public int ceph_ll_fallocate(struct ceph_mount_info *cmount, struct Fh *fh, + int mode, int64_t offset, int64_t length) +{ + CEPH_REQ(ceph_ll_fallocate, req, 0, ans, 0); + + req.fh = ptr_value(fh); + req.mode = mode; + req.offset = offset; + req.length = length; + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_FALLOCATE, req, ans); +} + +__public int ceph_ll_fsync(struct ceph_mount_info *cmount, struct Fh *fh, + int syncdataonly) +{ + CEPH_REQ(ceph_ll_fsync, req, 0, ans, 0); + + req.fh = ptr_value(fh); + req.dataonly = syncdataonly; + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_FSYNC, req, ans); +} + +__public int ceph_ll_getattr(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_statx *stx, unsigned int want, + unsigned int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_getattr, req, 0, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.want = want; + req.flags = flags; + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_GETATTR, req, ans); +} + +__public int ceph_ll_getxattr(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, void *value, size_t size, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_getxattr, req, 1, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = size; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, value, size); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_GETXATTR, req, ans); +} + +__public int ceph_ll_link(struct ceph_mount_info *cmount, struct Inode *in, + struct Inode *newparent, const char *name, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_link, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.parent = ptr_value(newparent); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LINK, req, ans); +} + +__public int ceph_ll_listxattr(struct ceph_mount_info *cmount, struct Inode *in, + char *list, size_t buf_size, size_t *list_size, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_listxattr, req, 0, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = buf_size; + + CEPH_BUFF_ADD(ans, list, buf_size); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LISTXATTR, req, ans); + if (err >= 0) { + *list_size = ans.size; + } + + return err; +} + +__public int ceph_ll_lookup(struct ceph_mount_info *cmount, Inode *parent, + const char *name, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_lookup, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_lookup_inode(struct ceph_mount_info *cmount, + struct inodeno_t ino, Inode **inode) +{ + CEPH_REQ(ceph_ll_lookup_inode, req, 0, ans, 0); + int32_t err; + + req.ino = ino; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP_INODE, req, ans); + if (err >= 0) { + *inode = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_lookup_root(struct ceph_mount_info *cmount, Inode **parent) +{ + CEPH_REQ(ceph_ll_lookup_root, req, 0, ans, 0); + int32_t err; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LOOKUP_ROOT, req, ans); + if (err >= 0) { + *parent = value_ptr(ans.inode); + } + + return err; +} + +__public off_t ceph_ll_lseek(struct ceph_mount_info *cmount, + struct Fh *filehandle, off_t offset, int whence) +{ + CEPH_REQ(ceph_ll_lseek, req, 0, ans, 0); + int32_t err; + + req.fh = ptr_value(filehandle); + req.offset = offset; + req.whence = whence; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_LSEEK, req, ans); + if (err >= 0) { + return ans.offset; + } + + return err; +} + +__public int ceph_ll_mkdir(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_mkdir, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_MKDIR, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_mknod(struct ceph_mount_info *cmount, Inode *parent, + const char *name, mode_t mode, dev_t rdev, + Inode **out, struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_mknod, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(parent); + req.mode = mode; + req.rdev = rdev; + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_MKNOD, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_open(struct ceph_mount_info *cmount, struct Inode *in, + int flags, struct Fh **fh, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_open, req, 0, ans, 0); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.flags = flags; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_OPEN, req, ans); + if (err >= 0) { + *fh = value_ptr(ans.fh); + } + + return err; +} + +__public int ceph_ll_opendir(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_dir_result **dirpp, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_opendir, req, 0, ans, 0); + int32_t err; + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_OPENDIR, req, ans); + if (err >= 0) { + *dirpp = value_ptr(ans.dir); + } + + return err; +} + +__public int ceph_ll_put(struct ceph_mount_info *cmount, struct Inode *in) +{ + CEPH_REQ(ceph_ll_put, req, 0, ans, 0); + + req.inode = ptr_value(in); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_PUT, req, ans); +} + +__public int ceph_ll_read(struct ceph_mount_info *cmount, struct Fh *filehandle, + int64_t off, uint64_t len, char *buf) +{ + CEPH_REQ(ceph_ll_read, req, 0, ans, 1); + + req.fh = ptr_value(filehandle); + req.offset = off; + req.len = len; + + CEPH_BUFF_ADD(ans, buf, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_READ, req, ans); +} + +__public int ceph_ll_readlink(struct ceph_mount_info *cmount, struct Inode *in, + char *buf, size_t bufsize, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_readlink, req, 0, ans, 1); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = bufsize; + + CEPH_BUFF_ADD(ans, buf, bufsize); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_READLINK, req, ans); +} + +__public int ceph_ll_releasedir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dir) +{ + CEPH_REQ(ceph_ll_releasedir, req, 0, ans, 0); + + req.dir = ptr_value(dir); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RELEASEDIR, req, ans); +} + +__public int ceph_ll_removexattr(struct ceph_mount_info *cmount, + struct Inode *in, const char *name, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_removexattr, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_REMOVEXATTR, req, ans); +} + +__public int ceph_ll_rename(struct ceph_mount_info *cmount, + struct Inode *parent, const char *name, + struct Inode *newparent, const char *newname, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_rename, req, 2, ans, 0); + + req.userperm = ptr_value(perms); + req.old_parent = ptr_value(parent); + req.new_parent = ptr_value(newparent); + CEPH_STR_ADD(req, old_name, name); + CEPH_STR_ADD(req, new_name, newname); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RENAME, req, ans); +} + +__public void ceph_rewinddir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dirp) +{ + CEPH_REQ(ceph_rewinddir, req, 0, ans, 0); + + req.dir = ptr_value(dirp); + + CEPH_PROCESS(cmount, LIBCEPHFSD_OP_REWINDDIR, req, ans); +} + +__public int ceph_ll_rmdir(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_rmdir, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_RMDIR, req, ans); +} + +__public int ceph_ll_setattr(struct ceph_mount_info *cmount, struct Inode *in, + struct ceph_statx *stx, int mask, + const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_setattr, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.mask = mask; + CEPH_BUFF_ADD(req, stx, sizeof(*stx)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SETATTR, req, ans); +} + +__public int ceph_ll_setxattr(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const void *value, size_t size, + int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_setxattr, req, 2, ans, 0); + + req.userperm = ptr_value(perms); + req.inode = ptr_value(in); + req.size = size; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + CEPH_BUFF_ADD(req, value, size); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SETXATTR, req, ans); +} + +__public int ceph_ll_statfs(struct ceph_mount_info *cmount, struct Inode *in, + struct statvfs *stbuf) +{ + CEPH_REQ(ceph_ll_statfs, req, 0, ans, 1); + + req.inode = ptr_value(in); + + CEPH_BUFF_ADD(ans, stbuf, sizeof(*stbuf)); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_STATFS, req, ans); +} + +__public int ceph_ll_symlink(struct ceph_mount_info *cmount, Inode *in, + const char *name, const char *value, Inode **out, + struct ceph_statx *stx, unsigned want, + unsigned flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_symlink, req, 2, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, name, name); + CEPH_STR_ADD(req, target, value); + + CEPH_BUFF_ADD(req, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_SYMLINK, req, ans); + if (err >= 0) { + *out = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_unlink(struct ceph_mount_info *cmount, struct Inode *in, + const char *name, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_unlink, req, 1, ans, 0); + + req.userperm = ptr_value(perms); + req.parent = ptr_value(in); + CEPH_STR_ADD(req, name, name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_UNLINK, req, ans); +} + +__public int ceph_ll_walk(struct ceph_mount_info *cmount, const char *name, + Inode **i, struct ceph_statx *stx, unsigned int want, + unsigned int flags, const UserPerm *perms) +{ + CEPH_REQ(ceph_ll_walk, req, 1, ans, 1); + int32_t err; + + req.userperm = ptr_value(perms); + req.want = want; + req.flags = flags; + CEPH_STR_ADD(req, path, name); + + CEPH_BUFF_ADD(ans, stx, sizeof(*stx)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_WALK, req, ans); + if (err >= 0) { + *i = value_ptr(ans.inode); + } + + return err; +} + +__public int ceph_ll_write(struct ceph_mount_info *cmount, + struct Fh *filehandle, int64_t off, uint64_t len, + const char *data) +{ + CEPH_REQ(ceph_ll_write, req, 1, ans, 0); + + req.fh = ptr_value(filehandle); + req.offset = off; + req.len = len; + CEPH_BUFF_ADD(req, data, len); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_LL_WRITE, req, ans); +} + +__public int ceph_mount(struct ceph_mount_info *cmount, const char *root) +{ + CEPH_REQ(ceph_mount, req, 1, ans, 0); + + CEPH_STR_ADD(req, root, root); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_MOUNT, req, ans); +} + +__public struct dirent *ceph_readdir(struct ceph_mount_info *cmount, + struct ceph_dir_result *dirp) +{ + static struct dirent de; + int32_t err; + + CEPH_REQ(ceph_readdir, req, 0, ans, 1); + + req.dir = ptr_value(dirp); + + CEPH_BUFF_ADD(ans, &de, sizeof(de)); + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_READDIR, req, ans); + if (err < 0) { + errno = -err; + return NULL; + } + if (ans.eod) { + return NULL; + } + + return &de; +} + +__public int ceph_release(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_release, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_RELEASE, req, ans); +} + +__public int ceph_select_filesystem(struct ceph_mount_info *cmount, + const char *fs_name) +{ + CEPH_REQ(ceph_select_filesystem, req, 1, ans, 0); + + CEPH_STR_ADD(req, fs, fs_name); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_SELECT_FILESYSTEM, req, ans); +} + +__public int ceph_unmount(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_unmount, req, 0, ans, 0); + + return CEPH_PROCESS(cmount, LIBCEPHFSD_OP_UNMOUNT, req, ans); +} + +__public void ceph_userperm_destroy(UserPerm *perms) +{ + CEPH_REQ(ceph_userperm_destroy, req, 0, ans, 0); + + req.userperm = ptr_value(perms); + + CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_USERPERM_DESTROY, req, ans); +} + +__public UserPerm *ceph_userperm_new(uid_t uid, gid_t gid, int ngids, + gid_t *gidlist) +{ + CEPH_REQ(ceph_userperm_new, req, 1, ans, 0); + int32_t err; + + req.uid = uid; + req.gid = gid; + req.groups = ngids; + CEPH_BUFF_ADD(req, gidlist, sizeof(gid_t) * ngids); + + err = proxy_global_connect(); + if (err >= 0) { + err = CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_USERPERM_NEW, req, + ans); + } + if (err >= 0) { + return value_ptr(ans.userperm); + } + + errno = -err; + + return NULL; +} + +__public const char *ceph_version(int *major, int *minor, int *patch) +{ + static char cached_version[128]; + static int32_t cached_major = -1, cached_minor, cached_patch; + + if (cached_major < 0) { + CEPH_REQ(ceph_version, req, 0, ans, 1); + int32_t err; + + CEPH_BUFF_ADD(ans, cached_version, sizeof(cached_version)); + + err = proxy_global_connect(); + if (err >= 0) { + err = CEPH_RUN(&global_cmount, LIBCEPHFSD_OP_VERSION, + req, ans); + } + + if (err < 0) { + *major = 0; + *minor = 0; + *patch = 0; + + return "Unknown"; + } + + cached_major = ans.major; + cached_minor = ans.minor; + cached_patch = ans.patch; + } + + *major = cached_major; + *minor = cached_minor; + *patch = cached_patch; + + return cached_version; +} + +__public UserPerm *ceph_mount_perms(struct ceph_mount_info *cmount) +{ + CEPH_REQ(ceph_mount_perms, req, 0, ans, 0); + int32_t err; + + err = CEPH_PROCESS(cmount, LIBCEPHFSD_OP_MOUNT_PERMS, req, ans); + if (err < 0) { + errno = -err; + return NULL; + } + + return value_ptr(ans.userperm); +} diff --git a/src/libcephfs_proxy/libcephfsd.c b/src/libcephfs_proxy/libcephfsd.c new file mode 100644 index 00000000000..ee2d99a0aae --- /dev/null +++ b/src/libcephfs_proxy/libcephfsd.c @@ -0,0 +1,1823 @@ + +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> +#include <getopt.h> +#include <endian.h> + +#include "include/cephfs/libcephfs.h" + +#include "proxy_manager.h" +#include "proxy_link.h" +#include "proxy_helpers.h" +#include "proxy_log.h" +#include "proxy_requests.h" +#include "proxy_mount.h" + +typedef struct _proxy_server { + proxy_link_t link; + proxy_manager_t *manager; +} proxy_server_t; + +typedef struct _proxy_client { + proxy_worker_t worker; + proxy_link_t *link; + proxy_random_t random; + void *buffer; + uint32_t buffer_size; + int32_t sd; +} proxy_client_t; + +typedef struct _proxy { + proxy_manager_t manager; + proxy_log_handler_t log_handler; + const char *socket_path; +} proxy_t; + +typedef int32_t (*proxy_handler_t)(proxy_client_t *, proxy_req_t *, + const void *data, int32_t data_size); + +/* This is used for requests that are not associated with a cmount. */ +static proxy_random_t global_random; + +static int32_t send_error(proxy_client_t *client, int32_t error) +{ + proxy_link_ans_t ans; + struct iovec iov[1]; + + iov[0].iov_base = &ans; + iov[0].iov_len = sizeof(ans); + + return proxy_link_ans_send(client->sd, error, iov, 1); +} + +static uint64_t uint64_checksum(uint64_t value) +{ + value = (value & 0xff00ff00ff00ffULL) + + ((value >> 8) & 0xff00ff00ff00ffULL); + value += value >> 16; + value += value >> 32; + + return value & 0xff; +} + +static uint64_t ptr_checksum(proxy_random_t *rnd, void *ptr) +{ + uint64_t value; + + if (ptr == NULL) { + return 0; + } + + value = (uint64_t)(uintptr_t)ptr; + /* Many current processors don't use the full 64-bits for the virtual + * address space, and Linux assigns the lower 128 TiB (47 bits) for + * user-space applications on most architectures, so the highest 8 bits + * of all valid addressess are always 0. + * + * We use this to encode a checksum in the high byte of the address to + * be able to do a verification before dereferencing the pointer, + * avoiding crashes if the client passes an invalid or corrupted pointer + * value. + * + * Alternatives like using indexes in a table or registering valid + * pointers require access to a shared data structure that will require + * thread synchronization, making it slower. */ + if ((value & 0xff00000000000007ULL) != 0) { + proxy_log(LOG_ERR, EINVAL, + "Unexpected pointer value"); + abort(); + } + + value -= uint64_checksum(value) << 56; + + return random_scramble(rnd, value); +} + +static int32_t ptr_check(proxy_random_t *rnd, uint64_t value, void **pptr) +{ + if (value == 0) { + *pptr = NULL; + return 0; + } + + value = random_unscramble(rnd, value); + + if ((uint64_checksum(value) != 0) || ((value & 7) != 0)) { + proxy_log(LOG_ERR, EFAULT, "Unexpected pointer value"); + return -EFAULT; + } + + *pptr = (void *)(uintptr_t)(value & 0xffffffffffffffULL); + + return 0; +} + +/* Macro to simplify request handling. */ +#define CEPH_COMPLETE(_client, _err, _ans) \ + ({ \ + int32_t __err = (_err); \ + if (__err < 0) { \ + __err = send_error(_client, __err); \ + } else { \ + __err = CEPH_RET(_client->sd, __err, _ans); \ + } \ + __err; \ + }) + +#ifdef PROXY_TRACE +#define TRACE(_fmt, _args...) printf(_fmt "\n", ##_args) +#else +#define TRACE(_fmt, _args...) do { } while (0) +#endif + +static int32_t libcephfsd_version(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_version, ans, 1); + const char *text; + int32_t major, minor, patch; + + text = ceph_version(&major, &minor, &patch); + TRACE("ceph_version(%d, %d, %d) -> %s", major, minor, patch, text); + + ans.major = major; + ans.minor = minor; + ans.patch = patch; + + CEPH_STR_ADD(ans, text, text); + + return CEPH_RET(client->sd, 0, ans); +} + +static int32_t libcephfsd_userperm_new(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_userperm_new, ans, 0); + UserPerm *userperm; + int32_t err; + + userperm = ceph_userperm_new(req->userperm_new.uid, + req->userperm_new.gid, + req->userperm_new.groups, (gid_t *)data); + TRACE("ceph_userperm_new(%u, %u, %u) -> %p", req->userperm_new.uid, + req->userperm_new.gid, req->userperm_new.groups, userperm); + + err = -ENOMEM; + if (userperm != NULL) { + ans.userperm = ptr_checksum(&global_random, userperm); + err = 0; + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_userperm_destroy(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_userperm_destroy, ans, 0); + UserPerm *perms; + int32_t err; + + err = ptr_check(&global_random, req->userperm_destroy.userperm, + (void **)&perms); + + if (err >= 0) { + ceph_userperm_destroy(perms); + TRACE("ceph_userperm_destroy(%p)", perms); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_create(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_create, ans, 0); + proxy_mount_t *mount; + const char *id; + int32_t err; + + id = CEPH_STR_GET(req->create, id, data); + + err = proxy_mount_create(&mount, id); + TRACE("ceph_create(%p, '%s') -> %d", mount, id, err); + + if (err >= 0) { + ans.cmount = ptr_checksum(&client->random, mount); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_release(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_release, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->release.cmount, (void **)&mount); + if (err >= 0) { + err = proxy_mount_release(mount); + TRACE("ceph_release(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_read_file(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_conf_read_file, ans, 0); + proxy_mount_t *mount; + const char *path; + int32_t err; + + err = ptr_check(&client->random, req->conf_read_file.cmount, + (void **)&mount); + if (err >= 0) { + path = CEPH_STR_GET(req->conf_read_file, path, data); + + err = proxy_mount_config(mount, path); + TRACE("ceph_conf_read_file(%p, '%s') ->%d", mount, path, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_get(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_conf_get, ans, 1); + proxy_mount_t *mount; + const char *option; + void *buffer; + uint32_t size; + int32_t err; + + buffer = client->buffer; + size = client->buffer_size; + if (req->conf_get.size < size) { + size = req->conf_get.size; + } + err = ptr_check(&client->random, req->conf_get.cmount, (void **)&mount); + if (err >= 0) { + option = CEPH_STR_GET(req->conf_get, option, data); + + err = proxy_mount_get(mount, option, buffer, size); + TRACE("ceph_conf_get(%p, '%s', '%s') -> %d", mount, option, + (char *)buffer, err); + + if (err >= 0) { + CEPH_DATA_ADD(ans, value, buffer, strlen(buffer) + 1); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_conf_set(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_conf_set, ans, 0); + proxy_mount_t *mount; + const char *option, *value; + int32_t err; + + err = ptr_check(&client->random, req->conf_set.cmount, (void **)&mount); + if (err >= 0) { + option = CEPH_STR_GET(req->conf_set, option, data); + value = CEPH_STR_GET(req->conf_set, value, + data + req->conf_set.option); + + err = proxy_mount_set(mount, option, value); + TRACE("ceph_conf_set(%p, '%s', '%s') -> %d", mount, option, + value, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_init(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_init, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->init.cmount, (void **)&mount); + if (err >= 0) { + err = proxy_mount_init(mount); + TRACE("ceph_init(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_select_filesystem(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_select_filesystem, ans, 0); + proxy_mount_t *mount; + const char *fs; + int32_t err; + + err = ptr_check(&client->random, req->select_filesystem.cmount, + (void **)&mount); + if (err >= 0) { + fs = CEPH_STR_GET(req->select_filesystem, fs, data); + + err = proxy_mount_select(mount, fs); + TRACE("ceph_select_filesystem(%p, '%s') -> %d", mount, fs, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_mount(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_mount, ans, 0); + proxy_mount_t *mount; + const char *root; + int32_t err; + + err = ptr_check(&client->random, req->mount.cmount, (void **)&mount); + if (err >= 0) { + root = CEPH_STR_GET(req->mount, root, data); + + err = proxy_mount_mount(mount, root); + TRACE("ceph_mount(%p, '%s') -> %d", mount, root, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_unmount(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_unmount, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->unmount.cmount, (void **)&mount); + + if (err >= 0) { + err = proxy_mount_unmount(mount); + TRACE("ceph_unmount(%p) -> %d", mount, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_statfs(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_statfs, ans, 1); + struct statvfs st; + proxy_mount_t *mount; + struct Inode *inode; + int32_t err; + + err = ptr_check(&client->random, req->ll_statfs.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_statfs.inode, + (void **)&inode); + } + + if (err >= 0) { + CEPH_BUFF_ADD(ans, &st, sizeof(st)); + + err = ceph_ll_statfs(proxy_cmount(mount), inode, &st); + TRACE("ceph_ll_statfs(%p, %p) -> %d", mount, inode, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *out; + const char *name; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_lookup.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_lookup.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_lookup.want; + flags = req->ll_lookup.flags; + name = CEPH_STR_GET(req->ll_lookup, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + if (name == NULL) { + err = proxy_log(LOG_ERR, EINVAL, + "NULL name passed to ceph_ll_lookup()"); + } else { + // Forbid going outside of the root mount point + if ((parent == mount->root) && + (strcmp(name, "..") == 0)) { + name = "."; + } + + err = ceph_ll_lookup(proxy_cmount(mount), parent, name, + &out, &stx, want, flags, perms); + } + + TRACE("ceph_ll_lookup(%p, %p, '%s', %p, %x, %x, %p) -> %d", + mount, parent, name, out, want, flags, perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, out); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup_inode(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup_inode, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + struct inodeno_t ino; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup_inode.cmount, + (void **)&mount); + if (err >= 0) { + ino = req->ll_lookup_inode.ino; + + err = ceph_ll_lookup_inode(proxy_cmount(mount), ino, &inode); + TRACE("ceph_ll_lookup_inode(%p, %lu, %p) -> %d", mount, ino.val, + inode, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lookup_root(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_lookup_root, ans, 0); + proxy_mount_t *mount; + int32_t err; + + err = ptr_check(&client->random, req->ll_lookup_root.cmount, + (void **)&mount); + if (err >= 0) { + /* The libcephfs view of the root of the mount could be + * different than ours, so we can't rely on + * ceph_ll_lookup_root(). We fake it by returning the cached + * root inode at the time of mount. */ + err = proxy_inode_ref(mount, mount->root_ino); + TRACE("ceph_ll_lookup_root(%p, %p) -> %d", mount, mount->root, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, mount->root); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_put(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_put, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + int32_t err; + + err = ptr_check(&client->random, req->ll_put.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_put.inode, + (void **)&inode); + } + + if (err >= 0) { + err = ceph_ll_put(proxy_cmount(mount), inode); + TRACE("ceph_ll_put(%p, %p) -> %d", mount, inode, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_walk(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_walk, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + const char *path; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_walk.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&global_random, req->ll_walk.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_walk.want; + flags = req->ll_walk.flags; + path = CEPH_STR_GET(req->ll_walk, path, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = proxy_path_resolve(mount, path, &inode, &stx, want, flags, + perms, NULL); + TRACE("ceph_ll_walk(%p, '%s', %p, %x, %x, %p) -> %d", mount, + path, inode, want, flags, perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_chdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_chdir, ans, 0); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + const char *path; + char *realpath; + int32_t err; + + err = ptr_check(&client->random, req->chdir.cmount, (void **)&mount); + if (err >= 0) { + path = CEPH_STR_GET(req->chdir, path, data); + + /* Since the libcephfs mount may be shared, we can't really + * change the current directory to avoid interferences with + * other users, so we just lookup the new directory and keep an + * internal reference. */ + err = proxy_path_resolve(mount, path, &inode, &stx, + CEPH_STATX_INO, 0, mount->perms, + &realpath); + TRACE("ceph_chdir(%p, '%s') -> %d", mount, path, err); + if (err >= 0) { + ceph_ll_put(proxy_cmount(mount), mount->cwd); + mount->cwd = inode; + mount->cwd_ino = stx.stx_ino; + + /* TODO: This path may become outdated if the parent + * directories are moved, however this seems the + * best we can do for now. */ + proxy_free(mount->cwd_path); + mount->cwd_path = realpath; + mount->cwd_path_len = strlen(realpath); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_getcwd(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_getcwd, ans, 1); + proxy_mount_t *mount; + const char *path; + int32_t err; + + err = ptr_check(&client->random, req->getcwd.cmount, (void **)&mount); + + if (err >= 0) { + /* We just return the cached name from the last chdir(). */ + path = mount->cwd_path; + TRACE("ceph_getcwd(%p) -> '%s'", mount, path); + CEPH_STR_ADD(ans, path, path); + err = 0; + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_readdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_readdir, ans, 1); + struct dirent de; + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->readdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->readdir.dir, + (void **)&dirp); + } + + if (err >= 0) { + err = ceph_readdir_r(proxy_cmount(mount), dirp, &de); + TRACE("ceph_readdir_r(%p, %p, %p) -> %d", mount, dirp, &de, + err); + ans.eod = true; + if (err > 0) { + ans.eod = false; + CEPH_BUFF_ADD(ans, &de, + offset_of(struct dirent, d_name) + + strlen(de.d_name) + 1); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_rewinddir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_rewinddir, ans, 0); + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->rewinddir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->rewinddir.dir, + (void **)&dirp); + } + + if (err >= 0) { + ceph_rewinddir(proxy_cmount(mount), dirp); + TRACE("ceph_rewinddir(%p, %p)", mount, dirp); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_open(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_open, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + struct Fh *fh; + int32_t flags, err; + + err = ptr_check(&client->random, req->ll_open.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_open.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_open.userperm, + (void **)&perms); + } + if (err >= 0) { + flags = req->ll_open.flags; + + err = ceph_ll_open(proxy_cmount(mount), inode, flags, &fh, + perms); + TRACE("ceph_ll_open(%p, %p, %x, %p, %p) -> %d", mount, inode, + flags, fh, perms, err); + + if (err >= 0) { + ans.fh = ptr_checksum(&client->random, fh); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_create(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_create, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + struct Fh *fh; + const char *name; + UserPerm *perms; + mode_t mode; + uint32_t want, flags; + int32_t oflags, err; + + err = ptr_check(&client->random, req->ll_create.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_create.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_create.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_create.mode; + oflags = req->ll_create.oflags; + want = req->ll_create.want; + flags = req->ll_create.flags; + name = CEPH_STR_GET(req->ll_create, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_create(proxy_cmount(mount), parent, name, mode, + oflags, &inode, &fh, &stx, want, flags, + perms); + TRACE("ceph_ll_create(%p, %p, '%s', %o, %x, %p, %p, %x, %x, " + "%p) -> %d", + mount, parent, name, mode, oflags, inode, fh, want, flags, + perms, err); + + if (err >= 0) { + ans.fh = ptr_checksum(&client->random, fh); + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_mknod(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_mknod, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + dev_t rdev; + mode_t mode; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_mknod.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_mknod.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_mknod.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_mknod.mode; + rdev = req->ll_mknod.rdev; + want = req->ll_mknod.want; + flags = req->ll_mknod.flags; + name = CEPH_STR_GET(req->ll_mknod, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_mknod(proxy_cmount(mount), parent, name, mode, + rdev, &inode, &stx, want, flags, perms); + TRACE("ceph_ll_mknod(%p, %p, '%s', %o, %lx, %p, %x, %x, %p) -> " + "%d", + mount, parent, name, mode, rdev, inode, want, flags, + perms, err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_close(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_close, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int32_t err; + + err = ptr_check(&client->random, req->ll_close.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_close.fh, + (void **)&fh); + } + + if (err >= 0) { + err = ceph_ll_close(proxy_cmount(mount), fh); + TRACE("ceph_ll_close(%p, %p) -> %d", mount, fh, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_rename(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_rename, ans, 0); + proxy_mount_t *mount; + struct Inode *old_parent, *new_parent; + const char *old_name, *new_name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_rename.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rename.old_parent, + (void **)&old_parent); + } + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rename.new_parent, + (void **)&new_parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_rename.userperm, + (void **)&perms); + } + if (err >= 0) { + old_name = CEPH_STR_GET(req->ll_rename, old_name, data); + new_name = CEPH_STR_GET(req->ll_rename, new_name, + data + req->ll_rename.old_name); + + err = ceph_ll_rename(proxy_cmount(mount), old_parent, old_name, + new_parent, new_name, perms); + TRACE("ceph_ll_rename(%p, %p, '%s', %p, '%s', %p) -> %d", mount, + old_parent, old_name, new_parent, new_name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_lseek(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_lseek, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + off_t offset, pos; + int32_t whence, err; + + err = ptr_check(&client->random, req->ll_lseek.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_lseek.fh, + (void **)&fh); + } + if (err >= 0) { + offset = req->ll_lseek.offset; + whence = req->ll_lseek.whence; + + pos = ceph_ll_lseek(proxy_cmount(mount), fh, offset, whence); + err = -errno; + TRACE("ceph_ll_lseek(%p, %p, %ld, %d) -> %ld (%d)", mount, fh, + offset, whence, pos, -err); + + if (pos >= 0) { + ans.offset = pos; + err = 0; + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_read(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_read, ans, 1); + proxy_mount_t *mount; + struct Fh *fh; + void *buffer; + uint64_t len; + int64_t offset; + uint32_t size; + int32_t err; + + buffer = client->buffer; + + err = ptr_check(&client->random, req->ll_read.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_read.fh, (void **)&fh); + } + if (err >= 0) { + offset = req->ll_read.offset; + len = req->ll_read.len; + + size = client->buffer_size; + if (len > size) { + buffer = proxy_malloc(len); + if (buffer == NULL) { + err = -ENOMEM; + } + } + if (err >= 0) { + err = ceph_ll_read(proxy_cmount(mount), fh, offset, len, + buffer); + TRACE("ceph_ll_read(%p, %p, %ld, %lu) -> %d", mount, fh, + offset, len, err); + + if (err >= 0) { + CEPH_BUFF_ADD(ans, buffer, err); + } + } + } + + err = CEPH_COMPLETE(client, err, ans); + + if (buffer != client->buffer) { + proxy_free(buffer); + } + + return err; +} + +static int32_t libcephfsd_ll_write(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_write, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + uint64_t len; + int64_t offset; + int32_t err; + + err = ptr_check(&client->random, req->ll_write.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_write.fh, + (void **)&fh); + } + if (err >= 0) { + offset = req->ll_write.offset; + len = req->ll_write.len; + + err = ceph_ll_write(proxy_cmount(mount), fh, offset, len, data); + TRACE("ceph_ll_write(%p, %p, %ld, %lu) -> %d", mount, fh, + offset, len, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_link(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_link, ans, 0); + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_link.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_link.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&client->random, req->ll_link.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_link.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_link, name, data); + + err = ceph_ll_link(proxy_cmount(mount), inode, parent, name, + perms); + TRACE("ceph_ll_link(%p, %p, %p, '%s', %p) -> %d", mount, inode, + parent, name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_unlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_unlink, ans, 0); + proxy_mount_t *mount; + struct Inode *parent; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_unlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_unlink.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_unlink.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_unlink, name, data); + + err = ceph_ll_unlink(proxy_cmount(mount), parent, name, perms); + TRACE("ceph_ll_unlink(%p, %p, '%s', %p) -> %d", mount, parent, + name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_getattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_getattr, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_getattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_getattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_getattr.userperm, + (void **)&perms); + } + if (err >= 0) { + want = req->ll_getattr.want; + flags = req->ll_getattr.flags; + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_getattr(proxy_cmount(mount), inode, &stx, want, + flags, perms); + TRACE("ceph_ll_getattr(%p, %p, %x, %x, %p) -> %d", mount, inode, + want, flags, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_setattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_setattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + int32_t mask, err; + + err = ptr_check(&client->random, req->ll_setattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_setattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_setattr.userperm, + (void **)&perms); + } + if (err >= 0) { + mask = req->ll_setattr.mask; + + err = ceph_ll_setattr(proxy_cmount(mount), inode, (void *)data, + mask, perms); + TRACE("ceph_ll_setattr(%p, %p, %x, %p) -> %d", mount, inode, + mask, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_fallocate(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_fallocate, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int64_t offset, len; + mode_t mode; + int32_t err; + + err = ptr_check(&client->random, req->ll_fallocate.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_fallocate.fh, + (void **)&fh); + } + if (err >= 0) { + mode = req->ll_fallocate.mode; + offset = req->ll_fallocate.offset; + len = req->ll_fallocate.length; + + err = ceph_ll_fallocate(proxy_cmount(mount), fh, mode, offset, + len); + TRACE("ceph_ll_fallocate(%p, %p, %o, %ld, %lu) -> %d", mount, + fh, mode, offset, len, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_fsync(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_fsync, ans, 0); + proxy_mount_t *mount; + struct Fh *fh; + int32_t dataonly, err; + + err = ptr_check(&client->random, req->ll_fsync.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_fsync.fh, + (void **)&fh); + } + if (err >= 0) { + dataonly = req->ll_fsync.dataonly; + + err = ceph_ll_fsync(proxy_cmount(mount), fh, dataonly); + TRACE("ceph_ll_fsync(%p, %p, %d) -> %d", mount, fh, dataonly, + err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_listxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_listxattr, ans, 1); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_listxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_listxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_listxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_listxattr.size; + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_listxattr(proxy_cmount(mount), inode, + client->buffer, size, &size, perms); + TRACE("ceph_ll_listxattr(%p, %p, %lu, %p) -> %d", mount, inode, + size, perms, err); + + if (err >= 0) { + ans.size = size; + CEPH_BUFF_ADD(ans, client->buffer, size); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_getxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_getxattr, ans, 1); + proxy_mount_t *mount; + struct Inode *inode; + const char *name; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_getxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_getxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_getxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_getxattr.size; + name = CEPH_STR_GET(req->ll_getxattr, name, data); + + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_getxattr(proxy_cmount(mount), inode, name, + client->buffer, size, perms); + TRACE("ceph_ll_getxattr(%p, %p, '%s', %p) -> %d", mount, inode, + name, perms, err); + + if (err >= 0) { + CEPH_BUFF_ADD(ans, client->buffer, err); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_setxattr(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_setxattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + const char *name, *value; + UserPerm *perms; + size_t size; + int32_t flags, err; + + err = ptr_check(&client->random, req->ll_setxattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_setxattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_setxattr.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_setxattr, name, data); + value = data + req->ll_setxattr.name; + size = req->ll_setxattr.size; + flags = req->ll_setxattr.flags; + + err = ceph_ll_setxattr(proxy_cmount(mount), inode, name, value, + size, flags, perms); + TRACE("ceph_ll_setxattr(%p, %p, '%s', %p, %x, %p) -> %d", mount, + inode, name, value, flags, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_removexattr(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_removexattr, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_removexattr.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_removexattr.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_removexattr.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_removexattr, name, data); + + err = ceph_ll_removexattr(proxy_cmount(mount), inode, name, + perms); + TRACE("ceph_ll_removexattr(%p, %p, '%s', %p) -> %d", mount, + inode, name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_readlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_readlink, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + UserPerm *perms; + size_t size; + int32_t err; + + err = ptr_check(&client->random, req->ll_readlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_readlink.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_readlink.userperm, + (void **)&perms); + } + if (err >= 0) { + size = req->ll_readlink.size; + + if (size > client->buffer_size) { + size = client->buffer_size; + } + err = ceph_ll_readlink(proxy_cmount(mount), inode, + client->buffer, size, perms); + TRACE("ceph_ll_readlink(%p, %p, %p) -> %d", mount, inode, perms, + err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_symlink(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_symlink, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + UserPerm *perms; + const char *name, *value; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_symlink.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_symlink.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_symlink.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_symlink, name, data); + value = CEPH_STR_GET(req->ll_symlink, target, + data + req->ll_symlink.name); + want = req->ll_symlink.want; + flags = req->ll_symlink.flags; + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_symlink(proxy_cmount(mount), parent, name, value, + &inode, &stx, want, flags, perms); + TRACE("ceph_ll_symlink(%p, %p, '%s', '%s', %p, %x, %x, %p) -> " + "%d", + mount, parent, name, value, inode, want, flags, perms, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_opendir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_opendir, ans, 0); + proxy_mount_t *mount; + struct Inode *inode; + struct ceph_dir_result *dirp; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_opendir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_opendir.inode, + (void **)&inode); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_opendir.userperm, + (void **)&perms); + } + + if (err >= 0) { + err = ceph_ll_opendir(proxy_cmount(mount), inode, &dirp, perms); + TRACE("ceph_ll_opendir(%p, %p, %p, %p) -> %d", mount, inode, + dirp, perms, err); + + if (err >= 0) { + ans.dir = ptr_checksum(&client->random, dirp); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_mkdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_mkdir, ans, 1); + struct ceph_statx stx; + proxy_mount_t *mount; + struct Inode *parent, *inode; + const char *name; + UserPerm *perms; + mode_t mode; + uint32_t want, flags; + int32_t err; + + err = ptr_check(&client->random, req->ll_mkdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_mkdir.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_mkdir.userperm, + (void **)&perms); + } + if (err >= 0) { + mode = req->ll_mkdir.mode; + want = req->ll_mkdir.want; + flags = req->ll_mkdir.flags; + name = CEPH_STR_GET(req->ll_mkdir, name, data); + + CEPH_BUFF_ADD(ans, &stx, sizeof(stx)); + + err = ceph_ll_mkdir(proxy_cmount(mount), parent, name, mode, + &inode, &stx, want, flags, perms); + TRACE("ceph_ll_mkdir(%p, %p, '%s', %o, %p, %x, %x, %p) -> %d", + mount, parent, name, mode, inode, want, flags, perms, + err); + + if (err >= 0) { + ans.inode = ptr_checksum(&client->random, inode); + } + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_rmdir(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_ll_rmdir, ans, 0); + proxy_mount_t *mount; + struct Inode *parent; + const char *name; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->ll_rmdir.cmount, (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_rmdir.parent, + (void **)&parent); + } + if (err >= 0) { + err = ptr_check(&global_random, req->ll_rmdir.userperm, + (void **)&perms); + } + if (err >= 0) { + name = CEPH_STR_GET(req->ll_rmdir, name, data); + + err = ceph_ll_rmdir(proxy_cmount(mount), parent, name, perms); + TRACE("ceph_ll_rmdir(%p, %p, '%s', %p) -> %d", mount, parent, + name, perms, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_ll_releasedir(proxy_client_t *client, + proxy_req_t *req, const void *data, + int32_t data_size) +{ + CEPH_DATA(ceph_ll_releasedir, ans, 0); + proxy_mount_t *mount; + struct ceph_dir_result *dirp; + int32_t err; + + err = ptr_check(&client->random, req->ll_releasedir.cmount, + (void **)&mount); + if (err >= 0) { + err = ptr_check(&client->random, req->ll_releasedir.dir, + (void **)&dirp); + } + + if (err >= 0) { + err = ceph_ll_releasedir(proxy_cmount(mount), dirp); + TRACE("ceph_ll_releasedir(%p, %p) -> %d", mount, dirp, err); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static int32_t libcephfsd_mount_perms(proxy_client_t *client, proxy_req_t *req, + const void *data, int32_t data_size) +{ + CEPH_DATA(ceph_mount_perms, ans, 0); + proxy_mount_t *mount; + UserPerm *perms; + int32_t err; + + err = ptr_check(&client->random, req->mount_perms.cmount, + (void **)&mount); + if (err >= 0) { + perms = ceph_mount_perms(proxy_cmount(mount)); + TRACE("ceph_mount_perms(%p) -> %p", mount, perms); + + ans.userperm = ptr_checksum(&global_random, perms); + } + + return CEPH_COMPLETE(client, err, ans); +} + +static proxy_handler_t libcephfsd_handlers[LIBCEPHFSD_OP_TOTAL_OPS] = { + [LIBCEPHFSD_OP_VERSION] = libcephfsd_version, + [LIBCEPHFSD_OP_USERPERM_NEW] = libcephfsd_userperm_new, + [LIBCEPHFSD_OP_USERPERM_DESTROY] = libcephfsd_userperm_destroy, + [LIBCEPHFSD_OP_CREATE] = libcephfsd_create, + [LIBCEPHFSD_OP_RELEASE] = libcephfsd_release, + [LIBCEPHFSD_OP_CONF_READ_FILE] = libcephfsd_conf_read_file, + [LIBCEPHFSD_OP_CONF_GET] = libcephfsd_conf_get, + [LIBCEPHFSD_OP_CONF_SET] = libcephfsd_conf_set, + [LIBCEPHFSD_OP_INIT] = libcephfsd_init, + [LIBCEPHFSD_OP_SELECT_FILESYSTEM] = libcephfsd_select_filesystem, + [LIBCEPHFSD_OP_MOUNT] = libcephfsd_mount, + [LIBCEPHFSD_OP_UNMOUNT] = libcephfsd_unmount, + [LIBCEPHFSD_OP_LL_STATFS] = libcephfsd_ll_statfs, + [LIBCEPHFSD_OP_LL_LOOKUP] = libcephfsd_ll_lookup, + [LIBCEPHFSD_OP_LL_LOOKUP_INODE] = libcephfsd_ll_lookup_inode, + [LIBCEPHFSD_OP_LL_LOOKUP_ROOT] = libcephfsd_ll_lookup_root, + [LIBCEPHFSD_OP_LL_PUT] = libcephfsd_ll_put, + [LIBCEPHFSD_OP_LL_WALK] = libcephfsd_ll_walk, + [LIBCEPHFSD_OP_CHDIR] = libcephfsd_chdir, + [LIBCEPHFSD_OP_GETCWD] = libcephfsd_getcwd, + [LIBCEPHFSD_OP_READDIR] = libcephfsd_readdir, + [LIBCEPHFSD_OP_REWINDDIR] = libcephfsd_rewinddir, + [LIBCEPHFSD_OP_LL_OPEN] = libcephfsd_ll_open, + [LIBCEPHFSD_OP_LL_CREATE] = libcephfsd_ll_create, + [LIBCEPHFSD_OP_LL_MKNOD] = libcephfsd_ll_mknod, + [LIBCEPHFSD_OP_LL_CLOSE] = libcephfsd_ll_close, + [LIBCEPHFSD_OP_LL_RENAME] = libcephfsd_ll_rename, + [LIBCEPHFSD_OP_LL_LSEEK] = libcephfsd_ll_lseek, + [LIBCEPHFSD_OP_LL_READ] = libcephfsd_ll_read, + [LIBCEPHFSD_OP_LL_WRITE] = libcephfsd_ll_write, + [LIBCEPHFSD_OP_LL_LINK] = libcephfsd_ll_link, + [LIBCEPHFSD_OP_LL_UNLINK] = libcephfsd_ll_unlink, + [LIBCEPHFSD_OP_LL_GETATTR] = libcephfsd_ll_getattr, + [LIBCEPHFSD_OP_LL_SETATTR] = libcephfsd_ll_setattr, + [LIBCEPHFSD_OP_LL_FALLOCATE] = libcephfsd_ll_fallocate, + [LIBCEPHFSD_OP_LL_FSYNC] = libcephfsd_ll_fsync, + [LIBCEPHFSD_OP_LL_LISTXATTR] = libcephfsd_ll_listxattr, + [LIBCEPHFSD_OP_LL_GETXATTR] = libcephfsd_ll_getxattr, + [LIBCEPHFSD_OP_LL_SETXATTR] = libcephfsd_ll_setxattr, + [LIBCEPHFSD_OP_LL_REMOVEXATTR] = libcephfsd_ll_removexattr, + [LIBCEPHFSD_OP_LL_READLINK] = libcephfsd_ll_readlink, + [LIBCEPHFSD_OP_LL_SYMLINK] = libcephfsd_ll_symlink, + [LIBCEPHFSD_OP_LL_OPENDIR] = libcephfsd_ll_opendir, + [LIBCEPHFSD_OP_LL_MKDIR] = libcephfsd_ll_mkdir, + [LIBCEPHFSD_OP_LL_RMDIR] = libcephfsd_ll_rmdir, + [LIBCEPHFSD_OP_LL_RELEASEDIR] = libcephfsd_ll_releasedir, + [LIBCEPHFSD_OP_MOUNT_PERMS] = libcephfsd_mount_perms, +}; + +static void serve_binary(proxy_client_t *client) +{ + proxy_req_t req; + CEPH_DATA(hello, ans, 0); + struct iovec req_iov[2]; + void *buffer; + uint32_t size; + int32_t err; + + /* This buffer will be used by most of the requests. For requests that + * require more space (probably just some writes), a new temporary + * buffer will be allocated by proxy_link_req_recv() code. */ + size = 65536; + buffer = proxy_malloc(size); + if (buffer == NULL) { + return; + } + + ans.major = LIBCEPHFSD_MAJOR; + ans.minor = LIBCEPHFSD_MINOR; + err = proxy_link_send(client->sd, ans_iov, ans_count); + if (err < 0) { + proxy_free(buffer); + return; + } + + while (true) { + req_iov[0].iov_base = &req; + req_iov[0].iov_len = sizeof(req); + req_iov[1].iov_base = buffer; + req_iov[1].iov_len = size; + + err = proxy_link_req_recv(client->sd, req_iov, 2); + if (err > 0) { + if (req.header.op >= LIBCEPHFSD_OP_TOTAL_OPS) { + err = send_error(client, -ENOSYS); + } else if (libcephfsd_handlers[req.header.op] == NULL) { + err = send_error(client, -EOPNOTSUPP); + } else { + err = libcephfsd_handlers[req.header.op]( + client, &req, req_iov[1].iov_base, + req.header.data_len); + } + } + + if (req_iov[1].iov_base != buffer) { + /* Free the buffer if it was temporarily allocated. */ + proxy_free(req_iov[1].iov_base); + } + + if (err < 0) { + break; + } + } + + proxy_free(buffer); +} + +static void serve_connection(proxy_worker_t *worker) +{ + CEPH_DATA(hello, req, 0); + proxy_client_t *client; + int32_t err; + + client = container_of(worker, proxy_client_t, worker); + + err = proxy_link_recv(client->sd, req_iov, req_count); + if (err >= 0) { + if (req.id == LIBCEPHFS_LIB_CLIENT) { + serve_binary(client); + } else { + proxy_log(LOG_ERR, EINVAL, + "Invalid client initial message"); + } + } + + close(client->sd); +} + +static void destroy_connection(proxy_worker_t *worker) +{ + proxy_client_t *client; + + client = container_of(worker, proxy_client_t, worker); + + proxy_free(client->buffer); + proxy_free(client); +} + +static int32_t accept_connection(proxy_link_t *link, int32_t sd) +{ + proxy_server_t *server; + proxy_client_t *client; + int32_t err; + + server = container_of(link, proxy_server_t, link); + + client = proxy_malloc(sizeof(proxy_client_t)); + if (client == NULL) { + err = -ENOMEM; + goto failed_close; + } + + client->buffer_size = 65536; + client->buffer = proxy_malloc(client->buffer_size); + if (client->buffer == NULL) { + err = -ENOMEM; + goto failed_client; + } + + random_init(&client->random); + client->sd = sd; + client->link = link; + + /* TODO: Make request management asynchronous and avoid creating a + * thread for each connection. */ + err = proxy_manager_launch(server->manager, &client->worker, + serve_connection, destroy_connection); + if (err < 0) { + goto failed_buffer; + } + + return 0; + +failed_buffer: + proxy_free(client->buffer); + +failed_client: + proxy_free(client); + +failed_close: + close(sd); + + return err; +} + +static bool check_stop(proxy_link_t *link) +{ + proxy_server_t *server; + + server = container_of(link, proxy_server_t, link); + + return proxy_manager_stop(server->manager); +} + +static int32_t server_start(proxy_manager_t *manager) +{ + proxy_server_t server; + proxy_t *proxy; + + proxy = container_of(manager, proxy_t, manager); + + server.manager = manager; + + return proxy_link_server(&server.link, proxy->socket_path, + accept_connection, check_stop); +} + +static void log_print(proxy_log_handler_t *handler, int32_t level, int32_t err, + const char *msg) +{ + printf("[%d] %s\n", level, msg); +} + +static struct option main_opts[] = { + {"socket", required_argument, NULL, 's'}, + {} +}; + +int32_t main(int32_t argc, char *argv[]) +{ + struct timespec now; + proxy_t proxy; + char *env; + int32_t err, val; + + clock_gettime(CLOCK_MONOTONIC, &now); + srand(now.tv_nsec); + + random_init(&global_random); + + proxy_log_register(&proxy.log_handler, log_print); + + proxy.socket_path = PROXY_SOCKET; + + env = getenv(PROXY_SOCKET_ENV); + if (env != NULL) { + proxy.socket_path = env; + } + + while ((val = getopt_long(argc, argv, ":s:", main_opts, NULL)) >= 0) { + if (val == 's') { + proxy.socket_path = optarg; + } else if (val == ':') { + proxy_log(LOG_ERR, ENODATA, + "Argument missing for '%s'\n", optopt); + return 1; + } else if (val == '?') { + proxy_log(LOG_ERR, EINVAL, + "Unknown option '%s'\n", optopt); + return 1; + } else { + proxy_log(LOG_ERR, EINVAL, + "Unexpected error parsing the options\n"); + return 1; + } + } + if (optind < argc) { + proxy_log(LOG_ERR, EINVAL, + "Unexpected arguments in command line"); + return 1; + } + + err = proxy_manager_run(&proxy.manager, server_start); + + proxy_log_deregister(&proxy.log_handler); + + return err < 0 ? 1 : 0; +} diff --git a/src/libcephfs_proxy/proxy.h b/src/libcephfs_proxy/proxy.h new file mode 100644 index 00000000000..cfb69072f19 --- /dev/null +++ b/src/libcephfs_proxy/proxy.h @@ -0,0 +1,67 @@ + +#ifndef __LIBCEPHFSD_PROXY_H__ +#define __LIBCEPHFSD_PROXY_H__ + +#include <string.h> +#include <errno.h> +#include <stdarg.h> +#include <stdint.h> +#include <stdbool.h> + +#define LIBCEPHFSD_MAJOR 0 +#define LIBCEPHFSD_MINOR 2 + +#define LIBCEPHFS_LIB_CLIENT 0xe3e5f0e8 // 'ceph' xor 0x80808080 + +#define PROXY_SOCKET "/run/libcephfsd.sock" +#define PROXY_SOCKET_ENV "LIBCEPHFSD_SOCKET" + +#define offset_of(_type, _field) ((uintptr_t) & ((_type *)0)->_field) + +#define container_of(_ptr, _type, _field) \ + ((_type *)((uintptr_t)(_ptr) - offset_of(_type, _field))) + +struct _list; +typedef struct _list list_t; + +struct _proxy_buffer_ops; +typedef struct _proxy_buffer_ops proxy_buffer_ops_t; + +struct _proxy_buffer; +typedef struct _proxy_buffer proxy_buffer_t; + +struct _proxy_output; +typedef struct _proxy_output proxy_output_t; + +struct _proxy_log_handler; +typedef struct _proxy_log_handler proxy_log_handler_t; + +struct _proxy_worker; +typedef struct _proxy_worker proxy_worker_t; + +struct _proxy_manager; +typedef struct _proxy_manager proxy_manager_t; + +struct _proxy_link; +typedef struct _proxy_link proxy_link_t; + +typedef int32_t (*proxy_output_write_t)(proxy_output_t *); +typedef int32_t (*proxy_output_full_t)(proxy_output_t *); + +typedef void (*proxy_log_callback_t)(proxy_log_handler_t *, int32_t, int32_t, + const char *); + +typedef void (*proxy_worker_start_t)(proxy_worker_t *); +typedef void (*proxy_worker_destroy_t)(proxy_worker_t *); + +typedef int32_t (*proxy_manager_start_t)(proxy_manager_t *); + +typedef int32_t (*proxy_link_start_t)(proxy_link_t *, int32_t); +typedef bool (*proxy_link_stop_t)(proxy_link_t *); + +struct _list { + list_t *next; + list_t *prev; +}; + +#endif diff --git a/src/libcephfs_proxy/proxy_helpers.c b/src/libcephfs_proxy/proxy_helpers.c new file mode 100644 index 00000000000..149d84d34bb --- /dev/null +++ b/src/libcephfs_proxy/proxy_helpers.c @@ -0,0 +1,81 @@ + +#include "proxy_helpers.h" + +#include <openssl/evp.h> + +static const char hex_digits[] = "0123456789abcdef"; + +int32_t proxy_hash(uint8_t *hash, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data) +{ + EVP_MD_CTX *ctx; + void *ptr; + uint32_t bytes; + int32_t i, err, len; + + if (size < 32) { + return proxy_log(LOG_ERR, ENOBUFS, + "Digest buffer is too small"); + } + + ctx = EVP_MD_CTX_new(); + if (ctx == NULL) { + return proxy_log(LOG_ERR, ENOMEM, "EVP_MD_CTX_new() failed"); + } + + if (!EVP_DigestInit_ex2(ctx, EVP_sha256(), NULL)) { + err = proxy_log(LOG_ERR, ENOMEM, "EVP_DigestInit_ex2() failed"); + goto done; + } + + i = 0; + while ((len = feed(&ptr, data, i)) > 0) { + if (!EVP_DigestUpdate(ctx, ptr, len)) { + err = proxy_log(LOG_ERR, ENOMEM, + "EVP_DigestUpdate() failed"); + goto done; + } + i++; + } + if (len < 0) { + err = len; + goto done; + } + + if (!EVP_DigestFinal_ex(ctx, hash, &bytes)) { + err = proxy_log(LOG_ERR, ENOMEM, "EVP_DigestFinal_ex() failed"); + goto done; + } + + err = 0; + +done: + EVP_MD_CTX_free(ctx); + + return err; +} + +int32_t proxy_hash_hex(char *digest, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data) +{ + uint8_t hash[32]; + int32_t i, err; + + if (size < 65) { + return proxy_log(LOG_ERR, ENOBUFS, + "Digest buffer is too small"); + } + + err = proxy_hash(hash, sizeof(hash), feed, data); + if (err < 0) { + return err; + } + + for (i = 0; i < 32; i++) { + *digest++ = hex_digits[hash[i] >> 4]; + *digest++ = hex_digits[hash[i] & 15]; + } + *digest = 0; + + return 0; +} diff --git a/src/libcephfs_proxy/proxy_helpers.h b/src/libcephfs_proxy/proxy_helpers.h new file mode 100644 index 00000000000..b4f58e7e3b3 --- /dev/null +++ b/src/libcephfs_proxy/proxy_helpers.h @@ -0,0 +1,311 @@ + +#ifndef __LIBCEPHFS_PROXY_HELPERS_H__ +#define __LIBCEPHFS_PROXY_HELPERS_H__ + +#include <stdlib.h> +#include <signal.h> +#include <pthread.h> +#include <stdint.h> +#include <stdbool.h> + +#include "proxy_log.h" + +#define __public __attribute__((__visibility__("default"))) + +#define ptr_value(_ptr) ((uint64_t)(uintptr_t)(_ptr)) +#define value_ptr(_val) ((void *)(uintptr_t)(_val)) + +typedef struct _proxy_random { + uint64_t mask; + uint64_t factor; + uint64_t factor_inv; + uint64_t shift; +} proxy_random_t; + +/* Generate a 64-bits random number different than 0. */ +static inline uint64_t random_u64(void) +{ + uint64_t value; + int32_t i; + + do { + value = 0; + for (i = 0; i < 4; i++) { + value <<= 16; + value ^= (random() >> 8) & 0xffff; + } + } while (value == 0); + + return value; +} + +/* Randomly initialize the data used to scramble pointers. */ +static inline void random_init(proxy_random_t *rnd) +{ + uint64_t inv; + + rnd->mask = random_u64(); + + /* Generate an odd multiplicative factor different than 1. */ + do { + rnd->factor = random_u64() | 1; + } while (rnd->factor == 1); + + /* Compute the inverse of 'factor' modulo 2^64. */ + inv = rnd->factor & 0x3; + inv *= 0x000000012 - rnd->factor * inv; + inv *= 0x000000102 - rnd->factor * inv; + inv *= 0x000010002 - rnd->factor * inv; + inv *= 0x100000002 - rnd->factor * inv; + rnd->factor_inv = inv * (2 - rnd->factor * inv); + + rnd->shift = random_u64(); +} + +/* Obfuscate a pointer. */ +static inline uint64_t random_scramble(proxy_random_t *rnd, uint64_t value) +{ + uint32_t bits; + + bits = __builtin_popcountll(value); + + /* rnd->shift is rotated by the amount of bits set to 1 in the original + * value, and the lowest 6 bits are extracted. This generates a + * pseudo-random number that depends on the number of bits of the + * value. */ + bits = ((rnd->shift >> bits) | (rnd->shift << (64 - bits))) & 0x3f; + + /* The value is rotated by the amount just computed. */ + value = (value << bits) | (value >> (64 - bits)); + + /* The final result is masked with a random number. */ + value ^= rnd->mask; + + /* And multiplied by a random factor modulo 2^64. */ + return value * rnd->factor; +} + +/* Recover a pointer. */ +static inline uint64_t random_unscramble(proxy_random_t *rnd, uint64_t value) +{ + uint32_t bits; + + /* Divide by the random factor (i.e. multiply by the inverse of the + * factor). */ + value *= rnd->factor_inv; + + /* Remove the mask. */ + value ^= rnd->mask; + + /* Get the number of bits the pointer was rotated. */ + bits = __builtin_popcountll(value); + bits = ((rnd->shift >> bits) | (rnd->shift << (64 - bits))) & 0x3f; + + /* Undo the rotation to recover the original value. */ + return (value >> bits) | (value << (64 - bits)); +} + +static inline void *proxy_malloc(size_t size) +{ + void *ptr; + + ptr = malloc(size); + if (ptr == NULL) { + proxy_log(LOG_ERR, errno, "Failed to allocate memory"); + } + + return ptr; +} + +static inline int32_t proxy_realloc(void **pptr, size_t size) +{ + void *ptr; + + ptr = realloc(*pptr, size); + if (ptr == NULL) { + return proxy_log(LOG_ERR, errno, "Failed to reallocate memory"); + } + + *pptr = ptr; + + return 0; +} + +static inline void proxy_free(void *ptr) +{ + free(ptr); +} + +static inline char *proxy_strdup(const char *str) +{ + char *ptr; + + ptr = strdup(str); + if (ptr == NULL) { + proxy_log(LOG_ERR, errno, "Failed to copy a string"); + return NULL; + } + + return ptr; +} + +static inline int32_t proxy_mutex_init(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_init(mutex, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, "Failed to initialize a mutex"); + } + + return 0; +} + +static inline void proxy_mutex_lock(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_lock(mutex); + if (err != 0) { + proxy_abort(err, "Mutex cannot be acquired"); + } +} + +static inline void proxy_mutex_unlock(pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_mutex_unlock(mutex); + if (err != 0) { + proxy_abort(err, "Mutex cannot be released"); + } +} + +static inline int32_t proxy_rwmutex_init(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_init(mutex, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, + "Failed to initialize a rwmutex"); + } + + return 0; +} + +static inline void proxy_rwmutex_rdlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_rdlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be acquired for read"); + } +} + +static inline void proxy_rwmutex_wrlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_wrlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be acquired for write"); + } +} + +static inline void proxy_rwmutex_unlock(pthread_rwlock_t *mutex) +{ + int32_t err; + + err = pthread_rwlock_unlock(mutex); + if (err != 0) { + proxy_abort(err, "RWMutex cannot be released"); + } +} + +static inline int32_t proxy_condition_init(pthread_cond_t *condition) +{ + int32_t err; + + err = pthread_cond_init(condition, NULL); + if (err != 0) { + return proxy_log(LOG_ERR, err, + "Failed to initialize a condition variable"); + } + + return 0; +} + +static inline void proxy_condition_signal(pthread_cond_t *condition) +{ + int32_t err; + + err = pthread_cond_signal(condition); + if (err != 0) { + proxy_abort(err, "Condition variable cannot be signaled"); + } +} + +static inline void proxy_condition_wait(pthread_cond_t *condition, + pthread_mutex_t *mutex) +{ + int32_t err; + + err = pthread_cond_wait(condition, mutex); + if (err != 0) { + proxy_abort(err, "Condition variable cannot be waited"); + } +} + +static inline int32_t proxy_thread_create(pthread_t *tid, + void *(*start)(void *), void *arg) +{ + int32_t err; + + err = pthread_create(tid, NULL, start, arg); + if (err != 0) { + proxy_log(LOG_ERR, err, "Failed to create a thread"); + } + + return err; +} + +static inline void proxy_thread_kill(pthread_t tid, int32_t signum) +{ + int32_t err; + + err = pthread_kill(tid, signum); + if (err != 0) { + proxy_abort(err, "Failed to send a signal to a thread"); + } +} + +static inline void proxy_thread_join(pthread_t tid) +{ + int32_t err; + + err = pthread_join(tid, NULL); + if (err != 0) { + proxy_log(LOG_ERR, err, "Unable to join a thread"); + } +} + +static inline int32_t proxy_signal_set(int32_t signum, struct sigaction *action, + struct sigaction *old) +{ + if (sigaction(signum, action, old) < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to configure a signal"); + } + + return 0; +} + +int32_t proxy_hash(uint8_t *hash, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data); + +int32_t proxy_hash_hex(char *digest, size_t size, + int32_t (*feed)(void **, void *, int32_t), void *data); + +#endif diff --git a/src/libcephfs_proxy/proxy_link.c b/src/libcephfs_proxy/proxy_link.c new file mode 100644 index 00000000000..20d9086ffa9 --- /dev/null +++ b/src/libcephfs_proxy/proxy_link.c @@ -0,0 +1,421 @@ + +#include <stdio.h> +#include <unistd.h> +#include <sys/uio.h> + +#include "proxy_link.h" +#include "proxy_manager.h" +#include "proxy_helpers.h" +#include "proxy_log.h" + +static int32_t iov_length(struct iovec *iov, int32_t count) +{ + int32_t len; + + len = 0; + while (count > 0) { + len += iov->iov_len; + iov++; + count--; + } + + return len; +} + +static int32_t proxy_link_prepare(struct sockaddr_un *addr, const char *path) +{ + struct sigaction action; + int32_t sd, len, err; + + memset(&action, 0, sizeof(action)); + action.sa_handler = SIG_IGN; + err = proxy_signal_set(SIGPIPE, &action, NULL); + if (err < 0) { + return err; + } + + memset(addr, 0, sizeof(*addr)); + addr->sun_family = AF_UNIX; + len = snprintf(addr->sun_path, sizeof(addr->sun_path), "%s", path); + if (len < 0) { + return proxy_log(LOG_ERR, EINVAL, + "Failed to copy Unix socket path"); + } + if (len >= sizeof(addr->sun_path)) { + return proxy_log(LOG_ERR, ENAMETOOLONG, + "Unix socket path too long"); + } + + sd = socket(AF_UNIX, SOCK_STREAM, 0); + if (sd < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to create a Unix socket"); + } + + return sd; +} + +int32_t proxy_link_client(proxy_link_t *link, const char *path, + proxy_link_stop_t stop) +{ + struct sockaddr_un addr; + int32_t sd, err; + + link->stop = stop; + link->sd = -1; + + sd = proxy_link_prepare(&addr, path); + if (sd < 0) { + return sd; + } + + err = 0; + while (err >= 0) { + if (connect(sd, (struct sockaddr *)&addr, sizeof(addr)) < 0) { + if (errno == EINTR) { + continue; + } + + err = proxy_log(LOG_ERR, errno, + "Failed to connect to libcephfsd"); + } else { + link->sd = sd; + return sd; + } + } + + close(sd); + + return err; +} + +void proxy_link_close(proxy_link_t *link) +{ + close(link->sd); + link->sd = -1; +} + +int32_t proxy_link_server(proxy_link_t *link, const char *path, + proxy_link_start_t start, proxy_link_stop_t stop) +{ + struct sockaddr_un addr; + socklen_t len; + int32_t cd, err; + + link->stop = stop; + link->sd = -1; + + err = proxy_link_prepare(&addr, path); + if (err < 0) { + return err; + } + link->sd = err; + + if ((unlink(path) < 0) && (errno != ENOENT) && (errno != ENOTDIR)) { + err = proxy_log(LOG_ERR, errno, + "Failed to remove existing socket"); + goto done; + } + + if (bind(link->sd, (struct sockaddr *)&addr, sizeof(addr)) < 0) { + err = proxy_log(LOG_ERR, errno, "Failed to bind Unix socket"); + goto done; + } + + if (listen(link->sd, SOMAXCONN) < 0) { + err = proxy_log(LOG_ERR, errno, + "Failed to listen from Unix socket"); + goto done; + } + + while (!stop(link)) { + len = sizeof(addr); + cd = accept(link->sd, (struct sockaddr *)&addr, &len); + if (cd < 0) { + if (errno != EINTR) { + proxy_log(LOG_ERR, errno, + "Failed to accept a connection"); + } + } else { + start(link, cd); + } + } + + err = 0; + +done: + close(link->sd); + + return err; +} + +int32_t proxy_link_read(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size) +{ + ssize_t len; + + do { + len = read(sd, buffer, size); + if (len < 0) { + if (errno == EINTR) { + if (link->stop(link)) { + return -EINTR; + } + continue; + } + return proxy_log(LOG_ERR, errno, + "Failed to read from socket"); + } + } while (len < 0); + + return len; +} + +int32_t proxy_link_write(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size) +{ + ssize_t len; + int32_t total; + + total = size; + while (total > 0) { + len = write(sd, buffer, total); + if (len < 0) { + if (errno == EINTR) { + if (link->stop(link)) { + return -EINTR; + } + continue; + } + return proxy_log(LOG_ERR, errno, + "Failed to write to socket"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENOBUFS, + "No data written to socket"); + } + + buffer += len; + total -= len; + } + + return size; +} + +int32_t proxy_link_send(int32_t sd, struct iovec *iov, int32_t count) +{ + struct iovec iov_copy[count]; + ssize_t len; + int32_t total; + + memcpy(iov_copy, iov, sizeof(struct iovec) * count); + iov = iov_copy; + + total = 0; + while (count > 0) { + len = writev(sd, iov, count); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "Failed to send data"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENOBUFS, "Partial write"); + } + total += len; + + while ((count > 0) && (iov->iov_len <= len)) { + len -= iov->iov_len; + iov++; + count--; + } + + if (count > 0) { + iov->iov_base += len; + iov->iov_len -= len; + } + } + + return total; +} + +int32_t proxy_link_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + struct iovec iov_copy[count]; + ssize_t len; + int32_t total; + + memcpy(iov_copy, iov, sizeof(struct iovec) * count); + iov = iov_copy; + + total = 0; + while (count > 0) { + len = readv(sd, iov, count); + if (len < 0) { + return proxy_log(LOG_ERR, errno, + "Failed to receive data"); + } + if (len == 0) { + return proxy_log(LOG_ERR, ENODATA, "Partial read"); + } + total += len; + + while ((count > 0) && (iov->iov_len <= len)) { + len -= iov->iov_len; + iov++; + count--; + } + + if (count > 0) { + iov->iov_base += len; + iov->iov_len -= len; + } + } + + return total; +} + +int32_t proxy_link_req_send(int32_t sd, int32_t op, struct iovec *iov, + int32_t count) +{ + proxy_link_req_t *req; + + req = iov[0].iov_base; + + req->header_len = iov[0].iov_len; + req->op = op; + req->data_len = iov_length(iov + 1, count - 1); + + return proxy_link_send(sd, iov, count); +} + +int32_t proxy_link_req_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + proxy_link_req_t *req; + void *buffer; + int32_t err, len, total; + + len = iov->iov_len; + iov->iov_len = sizeof(proxy_link_req_t); + err = proxy_link_recv(sd, iov, 1); + if (err < 0) { + return err; + } + total = err; + + req = iov->iov_base; + + if (req->data_len > 0) { + if (count == 1) { + return proxy_log(LOG_ERR, ENOBUFS, + "Request data is too long"); + } + if (iov[1].iov_len < req->data_len) { + buffer = proxy_malloc(req->data_len); + if (buffer == NULL) { + return -ENOMEM; + } + iov[1].iov_base = buffer; + } + iov[1].iov_len = req->data_len; + } else { + count = 1; + } + + if (req->header_len > sizeof(proxy_link_req_t)) { + if (len < req->header_len) { + return proxy_log(LOG_ERR, ENOBUFS, + "Request is too long"); + } + iov->iov_base += sizeof(proxy_link_req_t); + iov->iov_len = req->header_len - sizeof(proxy_link_req_t); + } else { + iov++; + count--; + if (count == 0) { + return total; + } + } + + err = proxy_link_recv(sd, iov, count); + if (err < 0) { + return err; + } + + return total + err; +} + +int32_t proxy_link_ans_send(int32_t sd, int32_t result, struct iovec *iov, + int32_t count) +{ + proxy_link_ans_t *ans; + + ans = iov->iov_base; + + ans->header_len = iov->iov_len; + ans->flags = 0; + ans->result = result; + ans->data_len = iov_length(iov + 1, count - 1); + + return proxy_link_send(sd, iov, count); +} + +int32_t proxy_link_ans_recv(int32_t sd, struct iovec *iov, int32_t count) +{ + proxy_link_ans_t *ans; + int32_t err, len, total; + + len = iov->iov_len; + iov->iov_len = sizeof(proxy_link_ans_t); + err = proxy_link_recv(sd, iov, 1); + if (err < 0) { + return err; + } + total = err; + + ans = iov->iov_base; + + if (ans->data_len > 0) { + if ((count == 1) || (iov[1].iov_len < ans->data_len)) { + return proxy_log(LOG_ERR, ENOBUFS, + "Answer data is too long"); + } + iov[1].iov_len = ans->data_len; + } else { + count = 1; + } + + if (ans->header_len > sizeof(proxy_link_ans_t)) { + if (len < ans->header_len) { + return proxy_log(LOG_ERR, ENOBUFS, + "Answer is too long"); + } + iov->iov_base += sizeof(proxy_link_ans_t); + iov->iov_len = ans->header_len - sizeof(proxy_link_ans_t); + } else { + iov++; + count--; + if (count == 0) { + return total; + } + } + + err = proxy_link_recv(sd, iov, count); + if (err < 0) { + return err; + } + + return total + err; +} + +int32_t proxy_link_request(int32_t sd, int32_t op, struct iovec *req_iov, + int32_t req_count, struct iovec *ans_iov, + int32_t ans_count) +{ + int32_t err; + + err = proxy_link_req_send(sd, op, req_iov, req_count); + if (err < 0) { + return err; + } + + return proxy_link_ans_recv(sd, ans_iov, ans_count); +} diff --git a/src/libcephfs_proxy/proxy_link.h b/src/libcephfs_proxy/proxy_link.h new file mode 100644 index 00000000000..01a32d94377 --- /dev/null +++ b/src/libcephfs_proxy/proxy_link.h @@ -0,0 +1,67 @@ + +#ifndef __LIBCEPHFS_PROXY_LINK_H__ +#define __LIBCEPHFS_PROXY_LINK_H__ + +#include <sys/socket.h> +#include <sys/un.h> + +#include "proxy.h" + +#define PROXY_LINK_DISCONNECTED { NULL, -1 } + +struct _proxy_link { + proxy_link_stop_t stop; + int32_t sd; +}; + +typedef struct _proxy_link_req { + uint16_t header_len; + uint16_t op; + uint32_t data_len; +} proxy_link_req_t; + +typedef struct _proxy_link_ans { + uint16_t header_len; + uint16_t flags; + int32_t result; + uint32_t data_len; +} proxy_link_ans_t; + +static inline bool proxy_link_is_connected(proxy_link_t *link) +{ + return link->sd >= 0; +} + +int32_t proxy_link_client(proxy_link_t *link, const char *path, + proxy_link_stop_t stop); + +void proxy_link_close(proxy_link_t *link); + +int32_t proxy_link_server(proxy_link_t *link, const char *path, + proxy_link_start_t start, proxy_link_stop_t stop); + +int32_t proxy_link_read(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size); + +int32_t proxy_link_write(proxy_link_t *link, int32_t sd, void *buffer, + int32_t size); + +int32_t proxy_link_send(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_req_send(int32_t sd, int32_t op, struct iovec *iov, + int32_t count); + +int32_t proxy_link_req_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_ans_send(int32_t sd, int32_t result, struct iovec *iov, + int32_t count); + +int32_t proxy_link_ans_recv(int32_t sd, struct iovec *iov, int32_t count); + +int32_t proxy_link_request(int32_t sd, int32_t op, struct iovec *req_iov, + int32_t req_count, struct iovec *ans_iov, + int32_t ans_count); + +#endif diff --git a/src/libcephfs_proxy/proxy_list.h b/src/libcephfs_proxy/proxy_list.h new file mode 100644 index 00000000000..3dcb2ff9791 --- /dev/null +++ b/src/libcephfs_proxy/proxy_list.h @@ -0,0 +1,121 @@ + +#ifndef __LIBCEPHFS_PROXY_LIST_H__ +#define __LIBCEPHFS_PROXY_LIST_H__ + +#include "proxy.h" + +#define LIST_INIT(_list) { _list, _list } + +#define list_entry(_ptr, _type, _field) container_of(_ptr, _type, _field) + +#define list_first_entry(_list, _type, _field) \ + list_entry((_list)->next, _type, _field) + +#define list_last_entry(_list, _type, _field) \ + list_entry((_list)->prev, _type, _field) + +#define list_next_entry(_ptr, _field) \ + list_first_entry(&_ptr->_field, __typeof(*_ptr), _field) + +#define list_for_each_entry(_ptr, _list, _field) \ + for (_ptr = list_first_entry(_list, __typeof(*_ptr), _field); \ + &_ptr->_field != _list; _ptr = list_next_entry(_ptr, _field)) + +static inline void list_init(list_t *list) +{ + list->next = list; + list->prev = list; +} + +static inline bool list_empty(list_t *list) +{ + return list->next == list; +} + +static inline void list_add_between(list_t *item, list_t *prev, list_t *next) +{ + item->next = next; + item->prev = prev; + prev->next = item; + next->prev = item; +} + +static inline void list_add(list_t *item, list_t *list) +{ + list_add_between(item, list, list->next); +} + +static inline void list_add_tail(list_t *item, list_t *list) +{ + list_add_between(item, list->prev, list); +} + +static inline void list_del(list_t *list) +{ + list->next->prev = list->prev; + list->prev->next = list->next; +} + +static inline void list_del_init(list_t *list) +{ + list_del(list); + list_init(list); +} + +static inline void list_move(list_t *item, list_t *list) +{ + list_del(item); + list_add(item, list); +} + +static inline void list_move_tail(list_t *item, list_t *list) +{ + list_del(item); + list_add_tail(item, list); +} + +static inline void list_splice_between(list_t *src, list_t *prev, list_t *next) +{ + list_t *first, *last; + + first = src->next; + last = src->prev; + + first->prev = prev; + prev->next = first; + + last->next = next; + next->prev = last; +} + +static inline void list_splice(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst, dst->next); + } +} + +static inline void list_splice_tail(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst->prev, dst); + } +} + +static inline void list_splice_init(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst, dst->next); + list_init(src); + } +} + +static inline void list_splice_tail_init(list_t *src, list_t *dst) +{ + if (!list_empty(src)) { + list_splice_between(src, dst->prev, dst); + list_init(src); + } +} + +#endif diff --git a/src/libcephfs_proxy/proxy_log.c b/src/libcephfs_proxy/proxy_log.c new file mode 100644 index 00000000000..dc1afed63de --- /dev/null +++ b/src/libcephfs_proxy/proxy_log.c @@ -0,0 +1,110 @@ + +#include <stdio.h> +#include <stdarg.h> + +#include "proxy_log.h" +#include "proxy_helpers.h" +#include "proxy_list.h" + +#define PROXY_LOG_BUFFER_SIZE 4096 + +static __thread char proxy_log_buffer[PROXY_LOG_BUFFER_SIZE]; + +static pthread_rwlock_t proxy_log_mutex = PTHREAD_RWLOCK_INITIALIZER; +static list_t proxy_log_handlers = LIST_INIT(&proxy_log_handlers); + +static void proxy_log_write(int32_t level, int32_t err, const char *msg) +{ + proxy_log_handler_t *handler; + + proxy_rwmutex_rdlock(&proxy_log_mutex); + + list_for_each_entry(handler, &proxy_log_handlers, list) { + handler->callback(handler, level, err, msg); + } + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +__public void proxy_log_register(proxy_log_handler_t *handler, + proxy_log_callback_t callback) +{ + handler->callback = callback; + + proxy_rwmutex_wrlock(&proxy_log_mutex); + + list_add_tail(&handler->list, &proxy_log_handlers); + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +__public void proxy_log_deregister(proxy_log_handler_t *handler) +{ + proxy_rwmutex_wrlock(&proxy_log_mutex); + + list_del_init(&handler->list); + + proxy_rwmutex_unlock(&proxy_log_mutex); +} + +static void proxy_log_msg(char *buffer, const char *text) +{ + int32_t len; + + len = strlen(text) + 1; + + memcpy(buffer, text, len); +} + +int32_t proxy_log_args(int32_t level, int32_t err, const char *fmt, + va_list args) +{ + static __thread bool busy = false; + int32_t len; + + if (busy) { + return -err; + } + busy = true; + + len = vsnprintf(proxy_log_buffer, sizeof(proxy_log_buffer), fmt, args); + if (len < 0) { + proxy_log_msg(proxy_log_buffer, + "<log message formatting failed>"); + } else if (len >= sizeof(proxy_log_buffer)) { + proxy_log_msg(proxy_log_buffer + sizeof(proxy_log_buffer) - 6, + "[...]"); + } + + proxy_log_write(level, err, proxy_log_buffer); + + busy = false; + + return -err; +} + +int32_t proxy_log(int32_t level, int32_t err, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + err = proxy_log_args(level, err, fmt, args); + va_end(args); + + return err; +} + +void proxy_abort_args(int32_t err, const char *fmt, va_list args) +{ + proxy_log_args(LOG_CRIT, err, fmt, args); + abort(); +} + +void proxy_abort(int32_t err, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + proxy_abort_args(err, fmt, args); + va_end(args); +} diff --git a/src/libcephfs_proxy/proxy_log.h b/src/libcephfs_proxy/proxy_log.h new file mode 100644 index 00000000000..02f45f9b110 --- /dev/null +++ b/src/libcephfs_proxy/proxy_log.h @@ -0,0 +1,28 @@ + +#ifndef __LIBCEPHFSD_PROXY_LOG_H__ +#define __LIBCEPHFSD_PROXY_LOG_H__ + +#include "proxy.h" + +enum { LOG_CRIT, LOG_ERR, LOG_WARN, LOG_INFO, LOG_DBG }; + +struct _proxy_log_handler { + list_t list; + proxy_log_callback_t callback; +}; + +int32_t proxy_log_args(int32_t level, int32_t err, const char *fmt, + va_list args); + +int32_t proxy_log(int32_t level, int32_t err, const char *fmt, ...); + +void proxy_abort_args(int32_t err, const char *fmt, va_list args); + +void proxy_abort(int32_t err, const char *fmt, ...); + +void proxy_log_register(proxy_log_handler_t *handler, + proxy_log_callback_t callback); + +void proxy_log_deregister(proxy_log_handler_t *handler); + +#endif diff --git a/src/libcephfs_proxy/proxy_manager.c b/src/libcephfs_proxy/proxy_manager.c new file mode 100644 index 00000000000..ea57083e700 --- /dev/null +++ b/src/libcephfs_proxy/proxy_manager.c @@ -0,0 +1,247 @@ + +#include <signal.h> + +#include "proxy_manager.h" +#include "proxy_helpers.h" +#include "proxy_list.h" +#include "proxy_log.h" + +static void proxy_manager_signal_handler(int32_t signum, siginfo_t *info, + void *ctx) +{ +} + +static void proxy_worker_register(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + list_add_tail(&worker->list, &manager->workers); + + proxy_mutex_unlock(&manager->mutex); +} + +static void proxy_worker_deregister(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + list_del_init(&worker->list); + if (list_empty(&manager->workers)) { + proxy_condition_signal(&manager->condition); + } + + proxy_mutex_unlock(&manager->mutex); +} + +static void proxy_worker_finished(proxy_worker_t *worker) +{ + proxy_manager_t *manager; + + manager = worker->manager; + + proxy_mutex_lock(&manager->mutex); + + if (list_empty(&manager->finished)) { + proxy_condition_signal(&manager->condition); + } + + list_move_tail(&worker->list, &manager->finished); + + proxy_mutex_unlock(&manager->mutex); +} + +static void *proxy_worker_start(void *arg) +{ + proxy_worker_t *worker; + + worker = arg; + + worker->start(worker); + + proxy_worker_finished(worker); + + return NULL; +} + +static void *proxy_manager_start(void *arg) +{ + proxy_manager_t *manager; + proxy_worker_t *worker; + + manager = arg; + + proxy_mutex_lock(&manager->mutex); + + while (true) { + while (!list_empty(&manager->finished)) { + worker = list_first_entry(&manager->finished, + proxy_worker_t, list); + list_del_init(&worker->list); + + proxy_mutex_unlock(&manager->mutex); + + proxy_thread_join(worker->tid); + + if (worker->destroy != NULL) { + worker->destroy(worker); + } + + proxy_mutex_lock(&manager->mutex); + } + + if (manager->stop && list_empty(&manager->workers)) { + break; + } + + proxy_condition_wait(&manager->condition, &manager->mutex); + } + + manager->done = true; + proxy_condition_signal(&manager->condition); + + proxy_mutex_unlock(&manager->mutex); + + return NULL; +} + +static int32_t proxy_manager_init(proxy_manager_t *manager) +{ + int32_t err; + + list_init(&manager->workers); + list_init(&manager->finished); + + manager->stop = false; + manager->done = false; + + manager->main_tid = pthread_self(); + + err = proxy_mutex_init(&manager->mutex); + if (err < 0) { + return err; + } + + err = proxy_condition_init(&manager->condition); + if (err < 0) { + pthread_mutex_destroy(&manager->mutex); + } + + return err; +} + +static void proxy_manager_destroy(proxy_manager_t *manager) +{ + pthread_cond_destroy(&manager->condition); + pthread_mutex_destroy(&manager->mutex); +} + +static int32_t proxy_manager_setup_signals(struct sigaction *old) +{ + struct sigaction action; + + /* The CONT signal will be used to wake threads blocked in I/O. */ + memset(&action, 0, sizeof(action)); + action.sa_flags = SA_SIGINFO; + action.sa_sigaction = proxy_manager_signal_handler; + + return proxy_signal_set(SIGCONT, &action, old); +} + +static void proxy_manager_restore_signals(struct sigaction *action) +{ + proxy_signal_set(SIGCONT, action, NULL); +} + +static void proxy_manager_terminate(proxy_manager_t *manager) +{ + proxy_worker_t *worker; + + proxy_mutex_lock(&manager->mutex); + + list_for_each_entry(worker, &manager->workers, list) { + worker->stop = true; + proxy_thread_kill(worker->tid, SIGCONT); + } + + while (!manager->done) { + proxy_condition_wait(&manager->condition, &manager->mutex); + } + + proxy_mutex_unlock(&manager->mutex); + + proxy_thread_join(manager->tid); +} + +int32_t proxy_manager_run(proxy_manager_t *manager, proxy_manager_start_t start) +{ + struct sigaction old_action; + int32_t err; + + err = proxy_manager_init(manager); + if (err < 0) { + return err; + } + + err = proxy_manager_setup_signals(&old_action); + if (err < 0) { + goto done_destroy; + } + + err = proxy_thread_create(&manager->tid, proxy_manager_start, manager); + if (err < 0) { + goto done_signal; + } + + err = start(manager); + + proxy_manager_terminate(manager); + +done_signal: + proxy_manager_restore_signals(&old_action); + +done_destroy: + proxy_manager_destroy(manager); + + return err; +} + +void proxy_manager_shutdown(proxy_manager_t *manager) +{ + proxy_mutex_lock(&manager->mutex); + + manager->stop = true; + proxy_condition_signal(&manager->condition); + + proxy_mutex_unlock(&manager->mutex); + + /* Wake the thread if it was blocked in an I/O operation. */ + proxy_thread_kill(manager->main_tid, SIGCONT); +} + +int32_t proxy_manager_launch(proxy_manager_t *manager, proxy_worker_t *worker, + proxy_worker_start_t start, + proxy_worker_destroy_t destroy) +{ + int32_t err; + + worker->manager = manager; + worker->start = start; + worker->destroy = destroy; + worker->stop = false; + + proxy_worker_register(worker); + + err = proxy_thread_create(&worker->tid, proxy_worker_start, worker); + if (err < 0) { + proxy_worker_deregister(worker); + } + + return err; +} diff --git a/src/libcephfs_proxy/proxy_manager.h b/src/libcephfs_proxy/proxy_manager.h new file mode 100644 index 00000000000..6a539be8d5b --- /dev/null +++ b/src/libcephfs_proxy/proxy_manager.h @@ -0,0 +1,43 @@ + +#ifndef __LIBCEPHFSD_PROXY_MANAGER_H__ +#define __LIBCEPHFSD_PROXY_MANAGER_H__ + +#include <pthread.h> + +#include "proxy.h" + +struct _proxy_worker { + list_t list; + pthread_t tid; + proxy_manager_t *manager; + proxy_worker_start_t start; + proxy_worker_destroy_t destroy; + bool stop; +}; + +struct _proxy_manager { + list_t workers; + list_t finished; + pthread_t main_tid; + pthread_t tid; + pthread_mutex_t mutex; + pthread_cond_t condition; + bool stop; + bool done; +}; + +int32_t proxy_manager_run(proxy_manager_t *manager, + proxy_manager_start_t start); + +void proxy_manager_shutdown(proxy_manager_t *manager); + +int32_t proxy_manager_launch(proxy_manager_t *manager, proxy_worker_t *worker, + proxy_worker_start_t start, + proxy_worker_destroy_t destroy); + +static inline bool proxy_manager_stop(proxy_manager_t *manager) +{ + return manager->stop; +} + +#endif diff --git a/src/libcephfs_proxy/proxy_mount.c b/src/libcephfs_proxy/proxy_mount.c new file mode 100644 index 00000000000..abfef1232c2 --- /dev/null +++ b/src/libcephfs_proxy/proxy_mount.c @@ -0,0 +1,1246 @@ + +#include "proxy_mount.h" +#include "proxy_helpers.h" + +#include <stdio.h> +#include <unistd.h> +#include <sys/stat.h> +#include <fcntl.h> + +/* Maximum number of symlinks to visit while resolving a path before returning + * ELOOP. */ +#define PROXY_MAX_SYMLINKS 16 + +struct _proxy_linked_str; +typedef struct _proxy_linked_str proxy_linked_str_t; + +/* This structure is used to handle symlinks found during the walk of a path. + * + * We'll start with an initial string representing a path. If one of the + * components is found to be a symlink, a new proxy_linked_str_t will be + * created with the content of the symlink. Then the new string will point + * to the old string, which may still contain some additional path components. + * The new string will be traversed resolving symlinks as they are found in the + * same way. Once it finished, the old string is recovered and traversal + * continues from the point it was left. */ +struct _proxy_linked_str { + proxy_linked_str_t *next; + char *remaining; + char data[]; +}; + +/* This structure is used to traverse a path while resolving any symlink + * found. At the end, it will contain the realpath of the entry and its + * inode. */ +typedef struct _proxy_path_iterator { + struct ceph_statx stx; + struct ceph_mount_info *cmount; + proxy_linked_str_t *lstr; + UserPerm *perms; + struct Inode *root; + struct Inode *base; + char *realpath; + uint64_t root_ino; + uint64_t base_ino; + uint32_t realpath_size; + uint32_t realpath_len; + uint32_t symlinks; + bool release; + bool follow; +} proxy_path_iterator_t; + +typedef struct _proxy_config { + int32_t src; + int32_t dst; + int32_t size; + int32_t total; + void *buffer; +} proxy_config_t; + +typedef struct _proxy_change { + list_t list; + uint32_t size; + char data[]; +} proxy_change_t; + +typedef struct _proxy_iter { + proxy_instance_t *instance; + list_t *item; +} proxy_iter_t; + +typedef struct _proxy_instance_pool { + pthread_mutex_t mutex; + list_t hash[256]; +} proxy_mount_pool_t; + +static proxy_mount_pool_t instance_pool = { + .mutex = PTHREAD_MUTEX_INITIALIZER, +}; + +/* Ceph client instance sharing + * + * The main purpose of the libcephfs proxy is to avoid the multiple independent + * data caches that are created when libcephfs is used from different processes. + * However the cache is not created per process but per client instance, so each + * call to `ceph_create()` creates its own private data cache instance. Just + * forwarding the libcephfs API calls to a single proxy process is not enough to + * solve the problem. + * + * The proxy will try to reuse existing client instances to reduce the number of + * independent caches. However it's not always possible to map all proxy clients + * to a single libcephfs instance. When different settings are used, separate + * Ceph instances are required to avoid unwanted behaviors. + * + * Even though it's possible that some Ceph options may be compatible even if + * they have different values, the proxy won't try to handle these cases. It + * will consider the configuration as a black box, and only 100% equal + * configurations will share the Ceph client instance. + */ + +/* Ceph configuration file management + * + * We won't try to parse Ceph configuration files. The proxy only wants to know + * if a configuration is equal or not. To do so, when a configuration file is + * passed to the proxy, it will create a private copy and compute an SHA256 + * hash. If the hash doesn't match, the configuration is considered different, + * even if it's not a real difference (like additional empty lines or the order + * of the options). + * + * The private copy is necessary to enforce that the settings are not changed + * concurrently, which could make us believe that two configurations are equal + * when they are not. + * + * Besides a configuration file, the user can also make manual configuration + * changes by using `ceph_conf_set()`. These changes are also tracked and + * compared to be sure that the active configuration matches. Only if the + * configuration file is exactly equal and all the applied changes are the same, + * and in the same order, the Ceph client instance will be shared. + */ + +int32_t proxy_inode_ref(proxy_mount_t *mount, uint64_t inode) +{ + inodeno_t ino; + struct Inode *tmp; + int32_t err; + + /* There's no way to tell libcephfs to increase the reference counter of + * an inode, so we do a full lookup for now. */ + + ino.val = inode; + + err = ceph_ll_lookup_inode(proxy_cmount(mount), ino, &tmp); + if (err < 0) { + proxy_log(LOG_ERR, -err, "ceph_ll_loolkup_inode() failed"); + } + + return err; +} + +static proxy_linked_str_t *proxy_linked_str_create(const char *str, + proxy_linked_str_t *next) +{ + proxy_linked_str_t *lstr; + uint32_t len; + + len = strlen(str) + 1; + lstr = proxy_malloc(sizeof(proxy_linked_str_t) + len); + if (lstr != NULL) { + lstr->next = next; + if (len > 1) { + lstr->remaining = lstr->data; + memcpy(lstr->data, str, len); + } else { + lstr->remaining = NULL; + } + } + + return lstr; +} + +static proxy_linked_str_t *proxy_linked_str_next(proxy_linked_str_t *lstr) +{ + proxy_linked_str_t *next; + + next = lstr->next; + proxy_free(lstr); + + return next; +} + +static void proxy_linked_str_destroy(proxy_linked_str_t *lstr) +{ + while (lstr != NULL) { + lstr = proxy_linked_str_next(lstr); + } +} + +static bool proxy_linked_str_empty(proxy_linked_str_t *lstr) +{ + return lstr->remaining == NULL; +} + +static char *proxy_linked_str_scan(proxy_linked_str_t *lstr, char ch) +{ + char *current; + + current = lstr->remaining; + lstr->remaining = strchr(lstr->remaining, ch); + if (lstr->remaining != NULL) { + *lstr->remaining++ = 0; + } + + return current; +} + +static int32_t proxy_path_iterator_init(proxy_path_iterator_t *iter, + proxy_mount_t *mount, const char *path, + UserPerm *perms, bool realpath, + bool follow) +{ + uint32_t len; + char ch; + + if (path == NULL) { + return proxy_log(LOG_ERR, EINVAL, "NULL path received"); + } + + memset(&iter->stx, 0, sizeof(iter->stx)); + iter->cmount = proxy_cmount(mount); + iter->perms = perms; + iter->root = mount->root; + iter->root_ino = mount->root_ino; + iter->base = mount->cwd; + iter->base_ino = mount->cwd_ino; + iter->symlinks = 0; + iter->release = false; + iter->follow = follow; + + len = strlen(path) + 1; + + ch = *path; + if (ch == '/') { + iter->base = mount->root; + iter->base_ino = mount->root_ino; + path++; + } + + iter->realpath = NULL; + iter->realpath_len = 0; + iter->realpath_size = 0; + + if (realpath) { + if (ch != '/') { + len += mount->cwd_path_len; + } + len = (len + 63) & ~63; + iter->realpath_size = len; + + iter->realpath = proxy_malloc(len); + if (iter->realpath == NULL) { + return -ENOMEM; + } + if (ch != '/') { + memcpy(iter->realpath, mount->cwd_path, + mount->cwd_path_len + 1); + iter->realpath_len = mount->cwd_path_len; + } else { + iter->realpath[0] = '/'; + iter->realpath[1] = 0; + iter->realpath_len = 1; + } + } + + iter->lstr = proxy_linked_str_create(path, NULL); + if (iter->lstr == NULL) { + proxy_free(iter->realpath); + return -ENOMEM; + } + + return 0; +} + +static char *proxy_path_iterator_next(proxy_path_iterator_t *iter) +{ + while (proxy_linked_str_empty(iter->lstr)) { + iter->lstr = proxy_linked_str_next(iter->lstr); + if (iter->lstr == NULL) { + return NULL; + } + } + + return proxy_linked_str_scan(iter->lstr, '/'); +} + +static bool proxy_path_iterator_is_last(proxy_path_iterator_t *iter) +{ + proxy_linked_str_t *lstr; + + lstr = iter->lstr; + while (proxy_linked_str_empty(iter->lstr)) { + lstr = lstr->next; + if (lstr == NULL) { + return true; + } + } + + return false; +} + +static void proxy_path_iterator_destroy(proxy_path_iterator_t *iter) +{ + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + + proxy_free(iter->realpath); + proxy_linked_str_destroy(iter->lstr); +} + +static int32_t proxy_path_iterator_resolve(proxy_path_iterator_t *iter) +{ + static __thread char path[PATH_MAX]; + proxy_linked_str_t *lstr; + char *ptr; + int32_t err; + + if (++iter->symlinks > PROXY_MAX_SYMLINKS) { + return proxy_log(LOG_ERR, ELOOP, "Too many symbolic links"); + } + + err = ceph_ll_readlink(iter->cmount, iter->base, path, sizeof(path), + iter->perms); + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_ll_readlink() failed"); + } + + ptr = path; + if (*ptr == '/') { + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + iter->base = iter->root; + iter->base_ino = iter->root_ino; + iter->release = false; + if (iter->realpath != NULL) { + iter->realpath[1] = 0; + iter->realpath_len = 1; + } + + ptr++; + } + + lstr = proxy_linked_str_create(ptr, iter->lstr); + if (lstr == NULL) { + return -ENOMEM; + } + iter->lstr = lstr; + + return 0; +} + +static int32_t proxy_path_iterator_append(proxy_path_iterator_t *iter, + const char *name) +{ + uint32_t len, size; + int32_t err; + + len = strlen(name) + 1; + size = iter->realpath_size; + if (iter->realpath_len + len >= size) { + do { + size <<= 1; + } while (iter->realpath_len + len >= size); + err = proxy_realloc((void **)&iter->realpath, size); + if (err < 0) { + return err; + } + iter->realpath_size = size; + } + + if (iter->realpath_len > 1) { + iter->realpath[iter->realpath_len++] = '/'; + } + memcpy(iter->realpath + iter->realpath_len, name, len); + iter->realpath_len += len - 1; + + return 0; +} + +static void proxy_path_iterator_remove(proxy_path_iterator_t *iter) +{ + while ((iter->realpath_len > 0) && + (iter->realpath[--iter->realpath_len] != '/')) { + } +} + +static int32_t proxy_path_lookup(struct ceph_mount_info *cmount, + struct Inode *parent, const char *name, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms) +{ + int32_t err; + + err = ceph_ll_lookup(cmount, parent, name, inode, stx, want, flags, + perms); + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_ll_lookup() failed"); + } + + return err; +} + +static int32_t proxy_path_iterator_lookup(proxy_path_iterator_t *iter, + const char *name) +{ + struct Inode *inode; + int32_t err; + + if (S_ISLNK(iter->stx.stx_mode)) { + return proxy_path_iterator_resolve(iter); + } + + err = proxy_path_lookup(iter->cmount, iter->base, name, &inode, + &iter->stx, CEPH_STATX_INO | CEPH_STATX_MODE, + AT_SYMLINK_NOFOLLOW, iter->perms); + if (err < 0) { + return err; + } + + if (iter->realpath != NULL) { + if ((name[0] == '.') && (name[1] == '.') && (name[2] == 0)) { + proxy_path_iterator_remove(iter); + } else { + err = proxy_path_iterator_append(iter, name); + if (err < 0) { + ceph_ll_put(iter->cmount, inode); + return err; + } + } + } + + if (iter->release) { + ceph_ll_put(iter->cmount, iter->base); + } + iter->base = inode; + iter->base_ino = iter->stx.stx_ino; + iter->release = true; + + if (iter->follow && S_ISLNK(iter->stx.stx_mode) && + proxy_path_iterator_is_last(iter)) { + return proxy_path_iterator_resolve(iter); + } + + return 0; +} + +/* Implements a path walk ensuring that it's not possible to go higher than the + * root mount point used in ceph_mount(). This means that it handles absolute + * paths and ".." entries in a special way, including paths found in symbolic + * links. */ +int32_t proxy_path_resolve(proxy_mount_t *mount, const char *path, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms, + char **realpath) +{ + proxy_path_iterator_t iter; + char *name, c; + int32_t err; + + err = proxy_path_iterator_init(&iter, mount, path, perms, + realpath != NULL, + (flags & AT_SYMLINK_NOFOLLOW) == 0); + if (err < 0) { + return err; + } + + while ((err >= 0) && + ((name = proxy_path_iterator_next(&iter)) != NULL)) { + c = *name; + if (c == '.') { + c = name[1]; + if ((c == '.') && (iter.base == mount->root)) { + c = name[2]; + } + } + if (c == 0) { + continue; + } + + err = proxy_path_iterator_lookup(&iter, name); + } + + if (err >= 0) { + err = proxy_path_lookup(proxy_cmount(mount), iter.base, ".", + inode, stx, want, flags, iter.perms); + } + + if ((err >= 0) && (realpath != NULL)) { + *realpath = iter.realpath; + iter.realpath = NULL; + } + + proxy_path_iterator_destroy(&iter); + + return err; +} + +static int32_t proxy_config_source_prepare(const char *config, struct stat *st) +{ + int32_t fd, err; + + fd = open(config, O_RDONLY); + if (fd < 0) { + return proxy_log(LOG_ERR, errno, "open() failed"); + } + + if (fstat(fd, st) < 0) { + err = proxy_log(LOG_ERR, errno, "fstat() failed"); + goto failed; + } + + if (!S_ISREG(st->st_mode)) { + err = proxy_log(LOG_ERR, EINVAL, + "Configuration file is not a regular file"); + goto failed; + } + + return fd; + +failed: + close(fd); + + return err; +} + +static void proxy_config_source_close(int32_t fd) +{ + close(fd); +} + +static int32_t proxy_config_source_read(int32_t fd, void *buffer, size_t size) +{ + ssize_t len; + + len = read(fd, buffer, size); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "read() failed"); + } + + return len; +} + +static int32_t proxy_config_source_validate(int32_t fd, struct stat *before, + int32_t size) +{ + struct stat after; + + if (fstat(fd, &after) < 0) { + return proxy_log(LOG_ERR, errno, "fstat() failed"); + } + + if ((before->st_size != size) || (before->st_size != after.st_size) || + (before->st_blocks != after.st_blocks) || + (before->st_ctim.tv_sec != after.st_ctim.tv_sec) || + (before->st_ctim.tv_nsec != after.st_ctim.tv_nsec) || + (before->st_mtim.tv_sec != after.st_mtim.tv_sec) || + (before->st_mtim.tv_nsec != after.st_mtim.tv_nsec)) { + proxy_log(LOG_WARN, 0, + "Configuration file has been modified while " + "reading it"); + + return 0; + } + + return 1; +} + +static int32_t proxy_config_destination_prepare(void) +{ + int32_t fd; + + fd = openat(AT_FDCWD, ".", O_TMPFILE | O_WRONLY, 0600); + if (fd < 0) { + return proxy_log(LOG_ERR, errno, "openat() failed"); + } + + return fd; +} + +static void proxy_config_destination_close(int32_t fd) +{ + close(fd); +} + +static int32_t proxy_config_destination_write(int32_t fd, void *data, + size_t size) +{ + ssize_t len; + + len = write(fd, data, size); + if (len < 0) { + return proxy_log(LOG_ERR, errno, "write() failed"); + } + if (len != size) { + return proxy_log(LOG_ERR, ENOSPC, "Partial write"); + } + + return size; +} + +static int32_t proxy_config_destination_commit(int32_t fd, const char *name) +{ + char path[32]; + + if (fsync(fd) < 0) { + return proxy_log(LOG_ERR, errno, "fsync() failed"); + } + + if (linkat(fd, "", AT_FDCWD, name, AT_EMPTY_PATH) < 0) { + if (errno == EEXIST) { + return 0; + } + + /* This may fail if the user doesn't have CAP_DAC_READ_SEARCH. + * In this case we attempt to link it using the /proc + * filesystem. */ + } + + snprintf(path, sizeof(path), "/proc/self/fd/%d", fd); + if (linkat(AT_FDCWD, path, AT_FDCWD, name, AT_SYMLINK_FOLLOW) < 0) { + if (errno != EEXIST) { + return proxy_log(LOG_ERR, errno, "linkat() failed"); + } + } + + return 0; +} + +static int32_t proxy_config_transfer(void **ptr, void *data, int32_t idx) +{ + proxy_config_t *cfg; + int32_t len, err; + + cfg = data; + + len = proxy_config_source_read(cfg->src, cfg->buffer, cfg->size); + if (len <= 0) { + return len; + } + + err = proxy_config_destination_write(cfg->dst, cfg->buffer, len); + if (err < 0) { + return err; + } + + cfg->total += len; + + *ptr = cfg->buffer; + + return len; +} + +/* Copies and checksums a given configuration to a file and makes sure that it + * has not been modified. */ +static int32_t proxy_config_prepare(const char *config, char *path, + int32_t size) +{ + char hash[65]; + proxy_config_t cfg; + struct stat before; + int32_t err; + + cfg.size = 4096; + cfg.buffer = proxy_malloc(cfg.size); + if (cfg.buffer == NULL) { + return -ENOMEM; + } + cfg.total = 0; + + cfg.src = proxy_config_source_prepare(config, &before); + if (cfg.src < 0) { + err = cfg.src; + goto done_mem; + } + + cfg.dst = proxy_config_destination_prepare(); + if (cfg.dst < 0) { + err = cfg.dst; + goto done_src; + } + + err = proxy_hash_hex(hash, sizeof(hash), proxy_config_transfer, &cfg); + if (err < 0) { + goto done_dst; + } + + err = proxy_config_source_validate(cfg.src, &before, cfg.total); + if (err < 0) { + goto done_dst; + } + + err = snprintf(path, size, "ceph-%s.conf", hash); + if (err < 0) { + err = proxy_log(LOG_ERR, errno, "snprintf() failed"); + goto done_dst; + } + if (err >= size) { + err = proxy_log(LOG_ERR, ENOBUFS, + "Insufficient space to store the name"); + goto done_dst; + } + + err = proxy_config_destination_commit(cfg.dst, path); + +done_dst: + proxy_config_destination_close(cfg.dst); + +done_src: + proxy_config_source_close(cfg.src); + +done_mem: + proxy_free(cfg.buffer); + + return err; +} + +/* Record changes to the configuration. */ +static int32_t proxy_instance_change_add(proxy_instance_t *instance, + const char *arg1, const char *arg2, + const char *arg3) +{ + proxy_change_t *change; + int32_t len[3], total; + + len[0] = strlen(arg1) + 1; + if (arg2 == NULL) { + arg2 = "<null>"; + } + len[1] = strlen(arg2) + 1; + len[2] = 0; + if (arg3 != NULL) { + len[2] = strlen(arg3) + 1; + } + + total = len[0] + len[1] + len[2]; + + change = proxy_malloc(sizeof(proxy_change_t) + total); + if (change == NULL) { + return -ENOMEM; + } + change->size = total; + + memcpy(change->data, arg1, len[0]); + memcpy(change->data + len[0], arg2, len[1]); + if (arg3 != NULL) { + memcpy(change->data + len[0] + len[1], arg3, len[2]); + } + + list_add_tail(&change->list, &instance->changes); + + return 0; +} + +static void proxy_instance_change_del(proxy_instance_t *instance) +{ + proxy_change_t *change; + + change = list_last_entry(&instance->changes, proxy_change_t, list); + list_del(&change->list); + + proxy_free(change); +} + +/* Destroy a Ceph client instance */ +static void proxy_instance_destroy(proxy_instance_t *instance) +{ + if (instance->mounted) { + ceph_unmount(instance->cmount); + } + + if (instance->cmount != NULL) { + ceph_release(instance->cmount); + } + + while (!list_empty(&instance->changes)) { + proxy_instance_change_del(instance); + } + + proxy_free(instance); +} + +/* Create a new Ceph client instance with the provided id */ +static int32_t proxy_instance_create(proxy_instance_t **pinstance, + const char *id) +{ + struct ceph_mount_info *cmount; + proxy_instance_t *instance; + int32_t err; + + instance = proxy_malloc(sizeof(proxy_instance_t)); + if (instance == NULL) { + return -ENOMEM; + } + + list_init(&instance->siblings); + list_init(&instance->changes); + instance->cmount = NULL; + instance->inited = false; + instance->mounted = false; + + err = proxy_instance_change_add(instance, "id", id, NULL); + if (err < 0) { + goto failed; + } + + err = ceph_create(&cmount, id); + if (err < 0) { + proxy_log(LOG_ERR, -err, "ceph_create() failed"); + goto failed; + } + + instance->cmount = cmount; + + *pinstance = instance; + + return 0; + +failed: + proxy_instance_destroy(instance); + + return err; +} + +static int32_t proxy_instance_release(proxy_instance_t *instance) +{ + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot release an active connection"); + } + + proxy_instance_destroy(instance); + + return 0; +} + +/* Assign a configuration file to the instance. */ +static int32_t proxy_instance_config(proxy_instance_t *instance, + const char *config) +{ + char path[128], *ppath; + int32_t err; + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot configure a mounted instance"); + } + + ppath = NULL; + if (config != NULL) { + err = proxy_config_prepare(config, path, sizeof(path)); + if (err < 0) { + return err; + } + ppath = path; + } + + err = proxy_instance_change_add(instance, "conf", ppath, NULL); + if (err < 0) { + return err; + } + + err = ceph_conf_read_file(instance->cmount, ppath); + if (err < 0) { + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_option_get(proxy_instance_t *instance, + const char *name, char *value, + size_t size) +{ + int32_t err, res; + + if (name == NULL) { + return proxy_log(LOG_ERR, EINVAL, "NULL option name"); + } + + res = ceph_conf_get(instance->cmount, name, value, size); + if (res < 0) { + return proxy_log( + LOG_ERR, -res, + "Failed to get configuration from a client instance"); + } + + err = proxy_instance_change_add(instance, "get", name, value); + if (err < 0) { + return err; + } + + return res; +} + +static int32_t proxy_instance_option_set(proxy_instance_t *instance, + const char *name, const char *value) +{ + int32_t err; + + if ((name == NULL) || (value == NULL)) { + return proxy_log(LOG_ERR, EINVAL, "NULL value or option name"); + } + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot configure a mounted instance"); + } + + err = proxy_instance_change_add(instance, "set", name, value); + if (err < 0) { + return err; + } + + err = ceph_conf_set(instance->cmount, name, value); + if (err < 0) { + proxy_log(LOG_ERR, -err, + "Failed to configure a client instance"); + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_select(proxy_instance_t *instance, const char *fs) +{ + int32_t err; + + if (instance->mounted) { + return proxy_log( + LOG_ERR, EISCONN, + "Cannot select a filesystem on a mounted instance"); + } + + err = proxy_instance_change_add(instance, "fs", fs, NULL); + if (err < 0) { + return err; + } + + err = ceph_select_filesystem(instance->cmount, fs); + if (err < 0) { + proxy_log(LOG_ERR, -err, + "Failed to select a filesystem on a client instance"); + proxy_instance_change_del(instance); + } + + return err; +} + +static int32_t proxy_instance_init(proxy_instance_t *instance) +{ + if (instance->mounted || instance->inited) { + return 0; + } + + /* ceph_init() does start several internal threads. However, an instance + * may not end up being mounted if the configuration matches with + * another mounted instance. Since ceph_mount() also calls ceph_init() + * if not already done, we avoid initializing it here to reduce resource + * consumption. */ + + instance->inited = true; + + return 0; +} + +static int32_t proxy_instance_hash(void **ptr, void *data, int32_t idx) +{ + proxy_iter_t *iter; + proxy_change_t *change; + + iter = data; + + if (iter->item == &iter->instance->changes) { + return 0; + } + + change = list_entry(iter->item, proxy_change_t, list); + iter->item = iter->item->next; + + *ptr = change->data; + + return change->size; +} + +/* Check if an existing instance matches the configuration used for the current + * one. If so, share the mount. Otherwise, create a new mount. */ +static int32_t proxy_instance_mount(proxy_instance_t **pinstance) +{ + proxy_instance_t *instance, *existing; + proxy_iter_t iter; + list_t *list; + int32_t err; + + instance = *pinstance; + + if (instance->mounted) { + return proxy_log(LOG_ERR, EISCONN, + "Cannot mount and already mounted instance"); + } + + iter.instance = instance; + iter.item = instance->changes.next; + + /* Create a hash that includes all settings. */ + err = proxy_hash(instance->hash, sizeof(instance->hash), + proxy_instance_hash, &iter); + if (err < 0) { + return err; + } + + list = &instance_pool.hash[instance->hash[0]]; + + proxy_mutex_lock(&instance_pool.mutex); + + if (list->next == NULL) { + list_init(list); + } else { + list_for_each_entry(existing, list, list) { + if (memcmp(existing->hash, instance->hash, 32) == 0) { + /* A match has been found. Instead of destroying + * the current instance, it's stored as a + * sibling of the one found. It will be + * reassigned to an instance when someone + * unmounts. */ + list_add(&instance->list, &existing->siblings); + goto found; + } + } + } + + /* No matching instance has been found. Just create a new one. The root + * is always "/". Each virtual mount point will locally store its root + * path. */ + err = ceph_mount(instance->cmount, "/"); + if (err >= 0) { + err = ceph_ll_lookup_root(instance->cmount, &instance->root); + if (err >= 0) { + instance->inited = true; + instance->mounted = true; + list_add(&instance->list, list); + } else { + ceph_unmount(instance->cmount); + } + } + + existing = NULL; + +found: + proxy_mutex_unlock(&instance_pool.mutex); + + if (err < 0) { + return proxy_log(LOG_ERR, -err, "ceph_mount() failed"); + } + + if (existing != NULL) { + proxy_log(LOG_INFO, 0, "Shared a client instance (%p)", + existing); + *pinstance = existing; + } else { + proxy_log(LOG_INFO, 0, "Created a new client instance (%p)", + instance); + } + + return 0; +} + +static int32_t proxy_instance_unmount(proxy_instance_t **pinstance) +{ + proxy_instance_t *instance, *sibling; + int32_t err; + + instance = *pinstance; + + if (!instance->mounted) { + return proxy_log(LOG_ERR, ENOTCONN, + "Cannot unmount an already unmount instance"); + } + + sibling = NULL; + + proxy_mutex_lock(&instance_pool.mutex); + + if (list_empty(&instance->siblings)) { + /* This is the last mount using this instance. We unmount it. */ + list_del(&instance->list); + instance->mounted = false; + } else { + /* There are other mounts sharing this instance. Take one of the + * saved siblings, which share the exact same configuration but + * are not mounted, to assign it to the current mount. */ + sibling = list_first_entry(&instance->siblings, + proxy_instance_t, list); + list_del_init(&sibling->list); + } + + proxy_mutex_unlock(&instance_pool.mutex); + + if (sibling == NULL) { + ceph_ll_put(instance->cmount, instance->root); + + err = ceph_unmount(instance->cmount); + if (err < 0) { + return proxy_log(LOG_ERR, -err, + "ceph_unmount() failed"); + } + } else { + *pinstance = sibling; + } + + return 0; +} + +int32_t proxy_mount_create(proxy_mount_t **pmount, const char *id) +{ + proxy_mount_t *mount; + int32_t err; + + mount = proxy_malloc(sizeof(proxy_mount_t)); + if (mount == NULL) { + return -ENOMEM; + } + mount->root = NULL; + + err = proxy_instance_create(&mount->instance, id); + if (err < 0) { + proxy_free(mount); + return err; + } + + *pmount = mount; + + return 0; +} + +int32_t proxy_mount_config(proxy_mount_t *mount, const char *config) +{ + return proxy_instance_config(mount->instance, config); +} + +int32_t proxy_mount_set(proxy_mount_t *mount, const char *name, + const char *value) +{ + return proxy_instance_option_set(mount->instance, name, value); +} + +int32_t proxy_mount_get(proxy_mount_t *mount, const char *name, char *value, + size_t size) +{ + return proxy_instance_option_get(mount->instance, name, value, size); +} + +int32_t proxy_mount_select(proxy_mount_t *mount, const char *fs) +{ + return proxy_instance_select(mount->instance, fs); +} + +int32_t proxy_mount_init(proxy_mount_t *mount) +{ + return proxy_instance_init(mount->instance); +} + +int32_t proxy_mount_mount(proxy_mount_t *mount, const char *root) +{ + struct ceph_statx stx; + struct ceph_mount_info *cmount; + int32_t err; + + err = proxy_instance_mount(&mount->instance); + if (err < 0) { + return err; + } + + cmount = proxy_cmount(mount); + + mount->perms = ceph_mount_perms(cmount); + + if (root == NULL) { + root = "/"; + } + + /* Temporarily set the root and cwd inodes to make proxy_path_resolve() + * to work correctly. */ + mount->root = mount->instance->root; + mount->root_ino = CEPH_INO_ROOT; + + mount->cwd = mount->instance->root; + mount->cwd_ino = CEPH_INO_ROOT; + + /* Resolve the desired root directory. */ + err = proxy_path_resolve(mount, root, &mount->root, &stx, + CEPH_STATX_ALL_STATS, 0, mount->perms, NULL); + if (err < 0) { + goto failed; + } + if (!S_ISDIR(stx.stx_mode)) { + err = proxy_log(LOG_ERR, ENOTDIR, + "The root path is not a directory"); + goto failed_root; + } + + mount->cwd_path = proxy_strdup("/"); + if (mount->cwd_path == NULL) { + err = -ENOMEM; + goto failed_root; + } + mount->cwd_path_len = 1; + + mount->root_ino = stx.stx_ino; + + err = proxy_inode_ref(mount, stx.stx_ino); + if (err < 0) { + goto failed_path; + } + + mount->cwd = mount->root; + mount->cwd_ino = stx.stx_ino; + + return 0; + +failed_path: + proxy_free(mount->cwd_path); + +failed_root: + ceph_ll_put(proxy_cmount(mount), mount->root); + +failed: + proxy_instance_unmount(&mount->instance); + + return err; +} + +int32_t proxy_mount_unmount(proxy_mount_t *mount) +{ + ceph_ll_put(proxy_cmount(mount), mount->root); + mount->root = NULL; + mount->root_ino = 0; + + ceph_ll_put(proxy_cmount(mount), mount->cwd); + mount->cwd = NULL; + mount->cwd_ino = 0; + + proxy_free(mount->cwd_path); + + return proxy_instance_unmount(&mount->instance); +} + +int32_t proxy_mount_release(proxy_mount_t *mount) +{ + int32_t err; + + err = proxy_instance_release(mount->instance); + if (err >= 0) { + proxy_free(mount); + } + + return err; +} diff --git a/src/libcephfs_proxy/proxy_mount.h b/src/libcephfs_proxy/proxy_mount.h new file mode 100644 index 00000000000..14bd58fabb2 --- /dev/null +++ b/src/libcephfs_proxy/proxy_mount.h @@ -0,0 +1,64 @@ + +#ifndef __LIBCEPHFSD_PROXY_MOUNT_H__ +#define __LIBCEPHFSD_PROXY_MOUNT_H__ + +#include "proxy.h" +#include "proxy_list.h" + +#include "include/cephfs/libcephfs.h" + +typedef struct _proxy_instance { + uint8_t hash[32]; + list_t list; + list_t siblings; + list_t changes; + struct ceph_mount_info *cmount; + struct Inode *root; + bool inited; + bool mounted; +} proxy_instance_t; + +typedef struct _proxy_mount { + proxy_instance_t *instance; + UserPerm *perms; + struct Inode *root; + struct Inode *cwd; + char *cwd_path; + uint64_t root_ino; + uint64_t cwd_ino; + uint32_t cwd_path_len; +} proxy_mount_t; + +static inline struct ceph_mount_info *proxy_cmount(proxy_mount_t *mount) +{ + return mount->instance->cmount; +} + +int32_t proxy_inode_ref(proxy_mount_t *mount, uint64_t inode); + +int32_t proxy_mount_create(proxy_mount_t **pmount, const char *id); + +int32_t proxy_mount_config(proxy_mount_t *mount, const char *config); + +int32_t proxy_mount_set(proxy_mount_t *mount, const char *name, + const char *value); + +int32_t proxy_mount_get(proxy_mount_t *mount, const char *name, char *value, + size_t size); + +int32_t proxy_mount_select(proxy_mount_t *mount, const char *fs); + +int32_t proxy_mount_init(proxy_mount_t *mount); + +int32_t proxy_mount_mount(proxy_mount_t *mount, const char *root); + +int32_t proxy_mount_unmount(proxy_mount_t *mount); + +int32_t proxy_mount_release(proxy_mount_t *mount); + +int32_t proxy_path_resolve(proxy_mount_t *mount, const char *path, + struct Inode **inode, struct ceph_statx *stx, + uint32_t want, uint32_t flags, UserPerm *perms, + char **realpath); + +#endif diff --git a/src/libcephfs_proxy/proxy_requests.h b/src/libcephfs_proxy/proxy_requests.h new file mode 100644 index 00000000000..4e3739276bb --- /dev/null +++ b/src/libcephfs_proxy/proxy_requests.h @@ -0,0 +1,343 @@ + +#ifndef __LIBCEPHFSD_PROXY_REQUESTS_H__ +#define __LIBCEPHFSD_PROXY_REQUESTS_H__ + +#include "proxy.h" +#include "proxy_link.h" + +/* Macros to add and get data from communication buffers. */ + +#define CEPH_BUFF_ADD(_data, _ptr, _size) \ + do { \ + _data##_iov[_data##_count].iov_base = (void *)(_ptr); \ + _data##_iov[_data##_count].iov_len = (_size); \ + _data##_count++; \ + } while (0) + +#define CEPH_DATA_ADD(_data, _field, _ptr, _size) \ + do { \ + (_data)._field = (_size); \ + CEPH_BUFF_ADD(_data, _ptr, (_data)._field); \ + } while (0) + +#define CEPH_STR_ADD(_data, _field, _str) \ + do { \ + if ((_str) != NULL) { \ + CEPH_DATA_ADD(_data, _field, _str, strlen(_str) + 1); \ + } else { \ + (_data)._field = 0; \ + } \ + } while (0) + +#define CEPH_STR_GET(_data, _field, _ptr) \ + ({ \ + const void *__ptr = (_ptr); \ + if ((_data)._field == 0) { \ + __ptr = NULL; \ + } \ + __ptr; \ + }) + +#define CEPH_DATA(_name, _data, _data_count) \ + proxy_##_name##_##_data##_t _data; \ + struct iovec _data##_iov[_data_count + 1]; \ + int32_t _data##_count = 0; \ + CEPH_BUFF_ADD(_data, &_data, sizeof(_data)) + +#define CEPH_REQ(_name, _req, _req_count, _ans, _ans_count) \ + CEPH_DATA(_name, _req, _req_count); \ + CEPH_DATA(_name, _ans, _ans_count) + +#define CEPH_CALL(_sd, _op, _req, _ans) \ + proxy_link_request((_sd), _op, _req##_iov, _req##_count, _ans##_iov, \ + _ans##_count) + +#define CEPH_RET(_sd, _res, _ans) \ + proxy_link_ans_send((_sd), (_res), _ans##_iov, _ans##_count) + +enum { + LIBCEPHFSD_OP_NULL = 0, + + LIBCEPHFSD_OP_VERSION, + LIBCEPHFSD_OP_USERPERM_NEW, + LIBCEPHFSD_OP_USERPERM_DESTROY, + LIBCEPHFSD_OP_CREATE, + LIBCEPHFSD_OP_RELEASE, + LIBCEPHFSD_OP_CONF_READ_FILE, + LIBCEPHFSD_OP_CONF_GET, + LIBCEPHFSD_OP_CONF_SET, + LIBCEPHFSD_OP_INIT, + LIBCEPHFSD_OP_SELECT_FILESYSTEM, + LIBCEPHFSD_OP_MOUNT, + LIBCEPHFSD_OP_UNMOUNT, + LIBCEPHFSD_OP_LL_STATFS, + LIBCEPHFSD_OP_LL_LOOKUP, + LIBCEPHFSD_OP_LL_LOOKUP_INODE, + LIBCEPHFSD_OP_LL_LOOKUP_ROOT, + LIBCEPHFSD_OP_LL_PUT, + LIBCEPHFSD_OP_LL_WALK, + LIBCEPHFSD_OP_CHDIR, + LIBCEPHFSD_OP_GETCWD, + LIBCEPHFSD_OP_READDIR, + LIBCEPHFSD_OP_REWINDDIR, + LIBCEPHFSD_OP_LL_OPEN, + LIBCEPHFSD_OP_LL_CREATE, + LIBCEPHFSD_OP_LL_MKNOD, + LIBCEPHFSD_OP_LL_CLOSE, + LIBCEPHFSD_OP_LL_RENAME, + LIBCEPHFSD_OP_LL_LSEEK, + LIBCEPHFSD_OP_LL_READ, + LIBCEPHFSD_OP_LL_WRITE, + LIBCEPHFSD_OP_LL_LINK, + LIBCEPHFSD_OP_LL_UNLINK, + LIBCEPHFSD_OP_LL_GETATTR, + LIBCEPHFSD_OP_LL_SETATTR, + LIBCEPHFSD_OP_LL_FALLOCATE, + LIBCEPHFSD_OP_LL_FSYNC, + LIBCEPHFSD_OP_LL_LISTXATTR, + LIBCEPHFSD_OP_LL_GETXATTR, + LIBCEPHFSD_OP_LL_SETXATTR, + LIBCEPHFSD_OP_LL_REMOVEXATTR, + LIBCEPHFSD_OP_LL_READLINK, + LIBCEPHFSD_OP_LL_SYMLINK, + LIBCEPHFSD_OP_LL_OPENDIR, + LIBCEPHFSD_OP_LL_MKDIR, + LIBCEPHFSD_OP_LL_RMDIR, + LIBCEPHFSD_OP_LL_RELEASEDIR, + LIBCEPHFSD_OP_MOUNT_PERMS, + + LIBCEPHFSD_OP_TOTAL_OPS +}; + +#define CEPH_TYPE_REQ(_name, _fields...) \ + struct _proxy_##_name##_req; \ + typedef struct _proxy_##_name##_req proxy_##_name##_req_t; \ + struct _proxy_##_name##_req { \ + _fields \ + } + +#define CEPH_TYPE_ANS(_name, _fields...) \ + struct _proxy_##_name##_ans; \ + typedef struct _proxy_##_name##_ans proxy_##_name##_ans_t; \ + struct _proxy_##_name##_ans { \ + _fields \ + } + +#define FIELDS(_fields...) _fields +#define REQ(_fields...) FIELDS(proxy_link_req_t header; _fields) +#define REQ_CMOUNT(_fields...) REQ(uint64_t cmount; _fields) +#define ANS(_fields...) FIELDS(proxy_link_ans_t header; _fields) +#define ANS_CMOUNT(_fields...) ANS(uint64_t cmount; _fields) + +#define CEPH_TYPE(_name, _req, _ans) \ + CEPH_TYPE_REQ(_name, _req); \ + CEPH_TYPE_ANS(_name, _ans) + +/* Declaration of types used to transder requests and answers. */ + +CEPH_TYPE(hello, FIELDS(uint32_t id;), FIELDS(int16_t major; int16_t minor;)); + +CEPH_TYPE(ceph_version, REQ(), + ANS(int32_t major; int32_t minor; int32_t patch; int16_t text;)); + +CEPH_TYPE(ceph_userperm_new, REQ(uint32_t uid; uint32_t gid; uint32_t groups;), + ANS(uint64_t userperm;)); + +CEPH_TYPE(ceph_userperm_destroy, REQ(uint64_t userperm;), ANS()); + +CEPH_TYPE(ceph_create, REQ(int16_t id;), ANS_CMOUNT()); + +CEPH_TYPE(ceph_release, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_conf_read_file, REQ_CMOUNT(uint16_t path;), ANS()); + +CEPH_TYPE(ceph_conf_get, REQ_CMOUNT(uint32_t size; uint16_t option;), + ANS(uint16_t value;)); + +CEPH_TYPE(ceph_conf_set, REQ_CMOUNT(uint16_t option; uint16_t value;), ANS()); + +CEPH_TYPE(ceph_init, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_select_filesystem, REQ_CMOUNT(uint16_t fs;), ANS()); + +CEPH_TYPE(ceph_mount, REQ_CMOUNT(uint16_t root;), ANS()); + +CEPH_TYPE(ceph_unmount, REQ_CMOUNT(), ANS()); + +CEPH_TYPE(ceph_ll_statfs, REQ_CMOUNT(uint64_t inode;), ANS()); + +CEPH_TYPE(ceph_ll_lookup, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint32_t want; + uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_lookup_inode, REQ_CMOUNT(struct inodeno_t ino;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_lookup_root, REQ_CMOUNT(), ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_put, REQ_CMOUNT(uint64_t inode;), ANS()); + +CEPH_TYPE(ceph_ll_walk, + REQ_CMOUNT(uint64_t userperm; uint32_t want; uint32_t flags; + uint16_t path;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_chdir, REQ_CMOUNT(uint16_t path;), ANS()); + +CEPH_TYPE(ceph_getcwd, REQ_CMOUNT(), ANS(uint16_t path;)); + +CEPH_TYPE(ceph_readdir, REQ_CMOUNT(uint64_t dir;), ANS(bool eod;)); + +CEPH_TYPE(ceph_rewinddir, REQ_CMOUNT(uint64_t dir;), ANS()); + +CEPH_TYPE(ceph_ll_open, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; int32_t flags;), + ANS(uint64_t fh;)); + +CEPH_TYPE(ceph_ll_create, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + int32_t oflags; uint32_t want; uint32_t flags; + uint16_t name;), + ANS(uint64_t inode; uint64_t fh;)); + +CEPH_TYPE(ceph_ll_mknod, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + dev_t rdev; uint32_t want; uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_close, REQ_CMOUNT(uint64_t fh;), ANS()); + +CEPH_TYPE(ceph_ll_rename, + REQ_CMOUNT(uint64_t userperm; uint64_t old_parent; + uint64_t new_parent; uint16_t old_name; + uint16_t new_name;), + ANS()); + +CEPH_TYPE(ceph_ll_lseek, REQ_CMOUNT(uint64_t fh; off_t offset; int32_t whence;), + ANS(off_t offset;)); + +CEPH_TYPE(ceph_ll_read, REQ_CMOUNT(uint64_t fh; int64_t offset; uint64_t len;), + ANS()); + +CEPH_TYPE(ceph_ll_write, REQ_CMOUNT(uint64_t fh; int64_t offset; uint64_t len;), + ANS()); + +CEPH_TYPE(ceph_ll_link, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint64_t parent; + uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_unlink, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_getattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint32_t want; + uint32_t flags;), + ANS()); + +CEPH_TYPE(ceph_ll_setattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; int32_t mask;), ANS()); + +CEPH_TYPE(ceph_ll_fallocate, + REQ_CMOUNT(uint64_t fh; int64_t offset; int64_t length; + int32_t mode;), + ANS()); + +CEPH_TYPE(ceph_ll_fsync, REQ_CMOUNT(uint64_t fh; int32_t dataonly;), ANS()); + +CEPH_TYPE(ceph_ll_listxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size;), + ANS(size_t size;)); + +CEPH_TYPE(ceph_ll_getxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size; + uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_setxattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size; + int32_t flags; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_removexattr, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; uint16_t name;), ANS()); + +CEPH_TYPE(ceph_ll_readlink, + REQ_CMOUNT(uint64_t userperm; uint64_t inode; size_t size;), ANS()); + +CEPH_TYPE(ceph_ll_symlink, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint32_t want; + uint32_t flags; uint16_t name; uint16_t target;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_opendir, REQ_CMOUNT(uint64_t userperm; uint64_t inode;), + ANS(uint64_t dir;)); + +CEPH_TYPE(ceph_ll_mkdir, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; mode_t mode; + uint32_t want; uint32_t flags; uint16_t name;), + ANS(uint64_t inode;)); + +CEPH_TYPE(ceph_ll_rmdir, + REQ_CMOUNT(uint64_t userperm; uint64_t parent; uint16_t name;), + ANS()); + +CEPH_TYPE(ceph_ll_releasedir, REQ_CMOUNT(uint64_t dir;), ANS()); + +CEPH_TYPE(ceph_mount_perms, REQ_CMOUNT(), ANS(uint64_t userperm;)); + +typedef union _proxy_req { + proxy_link_req_t header; + + proxy_ceph_version_req_t version; + proxy_ceph_userperm_new_req_t userperm_new; + proxy_ceph_userperm_destroy_req_t userperm_destroy; + proxy_ceph_create_req_t create; + proxy_ceph_release_req_t release; + proxy_ceph_conf_read_file_req_t conf_read_file; + proxy_ceph_conf_get_req_t conf_get; + proxy_ceph_conf_set_req_t conf_set; + proxy_ceph_init_req_t init; + proxy_ceph_select_filesystem_req_t select_filesystem; + proxy_ceph_mount_req_t mount; + proxy_ceph_unmount_req_t unmount; + proxy_ceph_ll_statfs_req_t ll_statfs; + proxy_ceph_ll_lookup_req_t ll_lookup; + proxy_ceph_ll_lookup_inode_req_t ll_lookup_inode; + proxy_ceph_ll_lookup_root_req_t ll_lookup_root; + proxy_ceph_ll_put_req_t ll_put; + proxy_ceph_ll_walk_req_t ll_walk; + proxy_ceph_chdir_req_t chdir; + proxy_ceph_getcwd_req_t getcwd; + proxy_ceph_readdir_req_t readdir; + proxy_ceph_rewinddir_req_t rewinddir; + proxy_ceph_ll_open_req_t ll_open; + proxy_ceph_ll_create_req_t ll_create; + proxy_ceph_ll_mknod_req_t ll_mknod; + proxy_ceph_ll_close_req_t ll_close; + proxy_ceph_ll_rename_req_t ll_rename; + proxy_ceph_ll_lseek_req_t ll_lseek; + proxy_ceph_ll_read_req_t ll_read; + proxy_ceph_ll_write_req_t ll_write; + proxy_ceph_ll_link_req_t ll_link; + proxy_ceph_ll_unlink_req_t ll_unlink; + proxy_ceph_ll_getattr_req_t ll_getattr; + proxy_ceph_ll_setattr_req_t ll_setattr; + proxy_ceph_ll_fallocate_req_t ll_fallocate; + proxy_ceph_ll_fsync_req_t ll_fsync; + proxy_ceph_ll_listxattr_req_t ll_listxattr; + proxy_ceph_ll_getxattr_req_t ll_getxattr; + proxy_ceph_ll_setxattr_req_t ll_setxattr; + proxy_ceph_ll_removexattr_req_t ll_removexattr; + proxy_ceph_ll_readlink_req_t ll_readlink; + proxy_ceph_ll_symlink_req_t ll_symlink; + proxy_ceph_ll_opendir_req_t ll_opendir; + proxy_ceph_ll_mkdir_req_t ll_mkdir; + proxy_ceph_ll_rmdir_req_t ll_rmdir; + proxy_ceph_ll_releasedir_req_t ll_releasedir; + proxy_ceph_mount_perms_req_t mount_perms; +} proxy_req_t; + +#endif diff --git a/src/librados/librados_asio.h b/src/librados/librados_asio.h index 0aedc376575..3e5b7c57c6f 100644 --- a/src/librados/librados_asio.h +++ b/src/librados/librados_asio.h @@ -14,6 +14,9 @@ #ifndef LIBRADOS_ASIO_H #define LIBRADOS_ASIO_H +#include <boost/asio/associated_cancellation_slot.hpp> +#include <boost/asio/cancellation_type.hpp> + #include "include/rados/librados.hpp" #include "common/async/completion.h" #include "librados/AioCompletionImpl.h" @@ -74,6 +77,7 @@ struct Invoker<void> { template <typename Result> struct AsyncOp : Invoker<Result> { unique_aio_completion_ptr aio_completion; + boost::asio::cancellation_slot slot; using Signature = typename Invoker<Result>::Signature; using Completion = ceph::async::Completion<Signature, AsyncOp<Result>>; @@ -83,6 +87,7 @@ struct AsyncOp : Invoker<Result> { auto p = std::unique_ptr<Completion>{static_cast<Completion*>(arg)}; // move result out of Completion memory being freed auto op = std::move(p->user_data); + op.slot.clear(); // clear our cancellation handler // access AioCompletionImpl directly to avoid locking const librados::AioCompletionImpl* pc = op.aio_completion->pc; const int ret = pc->rval; @@ -94,11 +99,46 @@ struct AsyncOp : Invoker<Result> { op.dispatch(std::move(p), ec, ver); } + struct op_cancellation { + AioCompletion* completion = nullptr; + bool is_read = false; + + void operator()(boost::asio::cancellation_type type) { + if (completion == nullptr) { + return; // no AioCompletion attached + } else if (type == boost::asio::cancellation_type::none) { + return; // no cancellation requested + } else if (is_read) { + // read operations produce no side effects, so can satisfy the + // requirements of 'total' cancellation. the weaker requirements + // of 'partial' and 'terminal' are also satisfied + completion->cancel(); + } else if (type == boost::asio::cancellation_type::terminal) { + // write operations only support 'terminal' cancellation because we + // can't guarantee that no osd has succeeded (or will succeed) in + // applying the write + completion->cancel(); + } + } + }; + template <typename Executor1, typename CompletionHandler> - static auto create(const Executor1& ex1, CompletionHandler&& handler) { + static auto create(const Executor1& ex1, bool is_read, + CompletionHandler&& handler) { + op_cancellation* cancel_handler = nullptr; + auto slot = boost::asio::get_associated_cancellation_slot(handler); + if (slot.is_connected()) { + cancel_handler = &slot.template emplace<op_cancellation>(); + } + auto p = Completion::create(ex1, std::move(handler)); p->user_data.aio_completion.reset( Rados::aio_create_completion(p.get(), aio_dispatch)); + if (cancel_handler) { + cancel_handler->completion = p->user_data.aio_completion.get(); + cancel_handler->is_read = is_read; + p->user_data.slot = std::move(slot); + } return p; } }; @@ -108,6 +148,9 @@ struct AsyncOp : Invoker<Result> { /// Calls IoCtx::aio_read() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, size_t len, uint64_t off, CompletionToken&& token) @@ -117,7 +160,8 @@ auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, size_t len, uint64_t off) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = true; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_read(oid, op.aio_completion.get(), &op.result, len, off); @@ -132,6 +176,9 @@ auto async_read(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_write() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, const bufferlist &bl, size_t len, uint64_t off, @@ -142,7 +189,8 @@ auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, const bufferlist &bl, size_t len, uint64_t off) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_write(oid, op.aio_completion.get(), bl, len, off); @@ -157,6 +205,9 @@ auto async_write(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_operate() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, ObjectReadOperation *read_op, int flags, @@ -167,7 +218,8 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, ObjectReadOperation *read_op, int flags) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = true; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_operate(oid, op.aio_completion.get(), read_op, @@ -183,6 +235,9 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_operate() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, ObjectWriteOperation *write_op, int flags, @@ -194,7 +249,8 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, [] (auto handler, auto ex, IoCtx& io, const std::string& oid, ObjectWriteOperation *write_op, int flags, const jspan_context* trace_ctx) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_operate(oid, op.aio_completion.get(), write_op, flags, trace_ctx); @@ -209,6 +265,9 @@ auto async_operate(ExecutionContext& ctx, IoCtx& io, const std::string& oid, /// Calls IoCtx::aio_notify() and arranges for the AioCompletion to call a /// given handler with signature (error_code, version_t, bufferlist). +/// +/// The given IoCtx reference is not required to remain valid, but some IoCtx +/// instance must preserve its underlying implementation until completion. template <typename ExecutionContext, typename CompletionToken> auto async_notify(ExecutionContext& ctx, IoCtx& io, const std::string& oid, bufferlist& bl, uint64_t timeout_ms, CompletionToken &&token) @@ -218,7 +277,8 @@ auto async_notify(ExecutionContext& ctx, IoCtx& io, const std::string& oid, return boost::asio::async_initiate<CompletionToken, Signature>( [] (auto handler, auto ex, IoCtx& io, const std::string& oid, bufferlist& bl, uint64_t timeout_ms) { - auto p = Op::create(ex, std::move(handler)); + constexpr bool is_read = false; + auto p = Op::create(ex, is_read, std::move(handler)); auto& op = p->user_data; int ret = io.aio_notify(oid, op.aio_completion.get(), diff --git a/src/librados/librados_cxx.cc b/src/librados/librados_cxx.cc index 2167eeade3c..60217b99b41 100644 --- a/src/librados/librados_cxx.cc +++ b/src/librados/librados_cxx.cc @@ -1103,6 +1103,14 @@ void librados::AioCompletion::release() delete this; } +int librados::AioCompletion::cancel() +{ + if (!pc->io) { + return 0; // no operation was started + } + return pc->io->aio_cancel(pc); +} + ///////////////////////////// IoCtx ////////////////////////////// librados::IoCtx::IoCtx() : io_ctx_impl(NULL) { diff --git a/src/librbd/ObjectMap.cc b/src/librbd/ObjectMap.cc index 65e3fc4a4c2..160bb4dcf9e 100644 --- a/src/librbd/ObjectMap.cc +++ b/src/librbd/ObjectMap.cc @@ -107,32 +107,6 @@ bool ObjectMap<I>::object_may_exist(uint64_t object_no) const } template <typename I> -bool ObjectMap<I>::object_may_not_exist(uint64_t object_no) const -{ - ceph_assert(ceph_mutex_is_locked(m_image_ctx.image_lock)); - - // Fall back to default logic if object map is disabled or invalid - if (!m_image_ctx.test_features(RBD_FEATURE_OBJECT_MAP, - m_image_ctx.image_lock)) { - return true; - } - - bool flags_set; - int r = m_image_ctx.test_flags(m_image_ctx.snap_id, - RBD_FLAG_OBJECT_MAP_INVALID, - m_image_ctx.image_lock, &flags_set); - if (r < 0 || flags_set) { - return true; - } - - uint8_t state = (*this)[object_no]; - bool nonexistent = (state != OBJECT_EXISTS && state != OBJECT_EXISTS_CLEAN); - ldout(m_image_ctx.cct, 20) << "object_no=" << object_no << " r=" - << nonexistent << dendl; - return nonexistent; -} - -template <typename I> bool ObjectMap<I>::update_required(const ceph::BitVector<2>::Iterator& it, uint8_t new_state) { ceph_assert(ceph_mutex_is_locked(m_lock)); diff --git a/src/librbd/ObjectMap.h b/src/librbd/ObjectMap.h index 35ea4cb88f9..5e7fcbbe9dd 100644 --- a/src/librbd/ObjectMap.h +++ b/src/librbd/ObjectMap.h @@ -65,7 +65,6 @@ public: void close(Context *on_finish); bool set_object_map(ceph::BitVector<2> &target_object_map); bool object_may_exist(uint64_t object_no) const; - bool object_may_not_exist(uint64_t object_no) const; void aio_save(Context *on_finish); void aio_resize(uint64_t new_size, uint8_t default_object_state, diff --git a/src/librbd/migration/HttpClient.cc b/src/librbd/migration/HttpClient.cc index 6a504d3a9ac..d212981a917 100644 --- a/src/librbd/migration/HttpClient.cc +++ b/src/librbd/migration/HttpClient.cc @@ -63,14 +63,13 @@ public: m_on_shutdown = on_finish; auto current_state = m_state; + m_state = STATE_SHUTTING_DOWN; + if (current_state == STATE_UNINITIALIZED) { // never initialized or resolve/connect failed on_finish->complete(0); return; - } - - m_state = STATE_SHUTTING_DOWN; - if (current_state != STATE_READY) { + } else if (current_state != STATE_READY) { // delay shutdown until current state transition completes return; } @@ -118,7 +117,7 @@ public: ceph_assert(m_http_client->m_strand.running_in_this_thread()); auto cct = m_http_client->m_cct; - ldout(cct, 20) << "work=" << work.get() << ", r=" << -ec.value() << dendl; + ldout(cct, 20) << "work=" << work.get() << ", ec=" << ec.what() << dendl; ceph_assert(m_in_flight_requests > 0); --m_in_flight_requests; @@ -187,13 +186,14 @@ protected: virtual void connect(boost::asio::ip::tcp::resolver::results_type results, Context* on_finish) = 0; virtual void disconnect(Context* on_finish) = 0; + virtual void reset_stream() = 0; void close_socket() { auto cct = m_http_client->m_cct; ldout(cct, 15) << dendl; boost::system::error_code ec; - boost::beast::get_lowest_layer(derived().stream()).socket().close(ec); + derived().stream().lowest_layer().close(ec); } private: @@ -229,7 +229,6 @@ private: auto cct = m_http_client->m_cct; ldout(cct, 15) << dendl; - shutdown_socket(); m_resolver.async_resolve( m_http_client->m_url_spec.host, m_http_client->m_url_spec.port, [this, on_finish](boost::system::error_code ec, auto results) { @@ -358,8 +357,7 @@ private: } int shutdown_socket() { - if (!boost::beast::get_lowest_layer( - derived().stream()).socket().is_open()) { + if (!derived().stream().lowest_layer().is_open()) { return 0; } @@ -367,7 +365,7 @@ private: ldout(cct, 15) << dendl; boost::system::error_code ec; - boost::beast::get_lowest_layer(derived().stream()).socket().shutdown( + derived().stream().lowest_layer().shutdown( boost::asio::ip::tcp::socket::shutdown_both, ec); if (ec && ec != boost::beast::errc::not_connected) { @@ -414,7 +412,7 @@ private: void handle_receive(boost::system::error_code ec, std::shared_ptr<Work>&& work) { auto cct = m_http_client->m_cct; - ldout(cct, 15) << "work=" << work.get() << ", r=" << -ec.value() << dendl; + ldout(cct, 15) << "work=" << work.get() << ", ec=" << ec.what() << dendl; ceph_assert(m_in_flight_requests > 0); --m_in_flight_requests; @@ -445,10 +443,10 @@ private: ldout(cct, 5) << "remote peer stream closed, retrying request" << dendl; m_receive_queue.push_front(work); } else if (ec == boost::beast::error::timeout) { - lderr(cct) << "timed-out while issuing request" << dendl; + lderr(cct) << "timed-out while receiving response" << dendl; work->complete(-ETIMEDOUT, {}); } else { - lderr(cct) << "failed to issue request: " << ec.message() << dendl; + lderr(cct) << "failed to receive response: " << ec.message() << dendl; work->complete(-ec.value(), {}); } @@ -473,7 +471,7 @@ private: r = -EACCES; } else if (boost::beast::http::to_status_class(result) != boost::beast::http::status_class::successful) { - lderr(cct) << "failed to retrieve size: HTTP " << result << dendl; + lderr(cct) << "failed to retrieve resource: HTTP " << result << dendl; r = -EIO; } @@ -501,7 +499,10 @@ private: << "next_state=" << next_state << ", " << "r=" << r << dendl; - m_state = next_state; + if (current_state != STATE_SHUTTING_DOWN) { + m_state = next_state; + } + if (current_state == STATE_CONNECTING) { if (next_state == STATE_UNINITIALIZED) { shutdown_socket(); @@ -512,14 +513,17 @@ private: return; } } else if (current_state == STATE_SHUTTING_DOWN) { + ceph_assert(m_on_shutdown != nullptr); if (next_state == STATE_READY) { // shut down requested while connecting/resetting disconnect(new LambdaContext([this](int r) { handle_shut_down(r); })); return; } else if (next_state == STATE_UNINITIALIZED || - next_state == STATE_SHUTDOWN || next_state == STATE_RESET_CONNECTING) { - ceph_assert(m_on_shutdown != nullptr); + shutdown_socket(); + m_on_shutdown->complete(r); + return; + } else if (next_state == STATE_SHUTDOWN) { m_on_shutdown->complete(r); return; } @@ -528,6 +532,7 @@ private: ceph_assert(next_state == STATE_RESET_CONNECTING); ceph_assert(on_finish == nullptr); shutdown_socket(); + reset_stream(); resolve_host(nullptr); return; } else if (current_state == STATE_RESET_CONNECTING) { @@ -589,7 +594,7 @@ public: this->close_socket(); } - inline boost::beast::tcp_stream& + inline boost::asio::ip::tcp::socket& stream() { return m_stream; } @@ -601,20 +606,25 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - m_stream.async_connect( - results, - [on_finish](boost::system::error_code ec, const auto& endpoint) { - on_finish->complete(-ec.value()); - }); + ceph_assert(!m_stream.is_open()); + boost::asio::async_connect(m_stream, + results, + [on_finish](boost::system::error_code ec, + const auto& endpoint) { + on_finish->complete(-ec.value()); + }); } void disconnect(Context* on_finish) override { on_finish->complete(0); } -private: - boost::beast::tcp_stream m_stream; + void reset_stream() override { + // no-op -- tcp_stream object can be reused after shut down + } +private: + boost::asio::ip::tcp::socket m_stream; }; #undef dout_prefix @@ -633,7 +643,7 @@ public: this->close_socket(); } - inline boost::beast::ssl_stream<boost::beast::tcp_stream>& + inline boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream() { return m_stream; } @@ -645,7 +655,9 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - boost::beast::get_lowest_layer(m_stream).async_connect( + ceph_assert(!m_stream.lowest_layer().is_open()); + async_connect( + m_stream.lowest_layer(), results, [this, on_finish](boost::system::error_code ec, const auto& endpoint) { handle_connect(-ec.value(), on_finish); @@ -657,19 +669,25 @@ protected: auto cct = http_client->m_cct; ldout(cct, 15) << dendl; - if (!m_ssl_enabled) { - on_finish->complete(0); - return; - } - m_stream.async_shutdown( - asio::util::get_callback_adapter([this, on_finish](int r) { - shutdown(r, on_finish); })); + [this, on_finish](boost::system::error_code ec) { + handle_disconnect(ec, on_finish); + }); + } + + void reset_stream() override { + auto http_client = this->m_http_client; + auto cct = http_client->m_cct; + ldout(cct, 15) << dendl; + + // ssl_stream object can't be reused after shut down -- move-in + // a freshly constructed instance + m_stream = boost::asio::ssl::stream<boost::asio::ip::tcp::socket>( + http_client->m_strand, http_client->m_ssl_context); } private: - boost::beast::ssl_stream<boost::beast::tcp_stream> m_stream; - bool m_ssl_enabled = false; + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> m_stream; void handle_connect(int r, Context* on_finish) { auto http_client = this->m_http_client; @@ -728,33 +746,38 @@ private: // Perform the SSL/TLS handshake m_stream.async_handshake( boost::asio::ssl::stream_base::client, - asio::util::get_callback_adapter( - [this, on_finish](int r) { handle_handshake(r, on_finish); })); + [this, on_finish](boost::system::error_code ec) { + handle_handshake(ec, on_finish); + }); } - void handle_handshake(int r, Context* on_finish) { + void handle_handshake(boost::system::error_code ec, Context* on_finish) { auto http_client = this->m_http_client; auto cct = http_client->m_cct; - ldout(cct, 15) << "r=" << r << dendl; + ldout(cct, 15) << "ec=" << ec.what() << dendl; - if (r < 0) { - lderr(cct) << "failed to complete handshake: " << cpp_strerror(r) + if (ec) { + lderr(cct) << "failed to complete SSL handshake: " << ec.message() << dendl; - disconnect(new LambdaContext([r, on_finish](int) { - on_finish->complete(r); })); + on_finish->complete(-ec.value()); return; } - m_ssl_enabled = true; on_finish->complete(0); } - void shutdown(int r, Context* on_finish) { + void handle_disconnect(boost::system::error_code ec, Context* on_finish) { auto http_client = this->m_http_client; auto cct = http_client->m_cct; - ldout(cct, 15) << "r=" << r << dendl; + ldout(cct, 15) << "ec=" << ec.what() << dendl; - on_finish->complete(r); + if (ec && ec != boost::asio::ssl::error::stream_truncated) { + lderr(cct) << "failed to shut down SSL: " << ec.message() << dendl; + on_finish->complete(-ec.value()); + return; + } + + on_finish->complete(0); } }; diff --git a/src/librbd/migration/HttpClient.h b/src/librbd/migration/HttpClient.h index 3997e6159e7..5844f918693 100644 --- a/src/librbd/migration/HttpClient.h +++ b/src/librbd/migration/HttpClient.h @@ -13,13 +13,12 @@ #include <boost/asio/strand.hpp> #include <boost/asio/ip/tcp.hpp> #include <boost/asio/ssl/context.hpp> +#include <boost/asio/ssl/stream.hpp> #include <boost/beast/version.hpp> -#include <boost/beast/core/tcp_stream.hpp> #include <boost/beast/http/empty_body.hpp> #include <boost/beast/http/message.hpp> #include <boost/beast/http/string_body.hpp> #include <boost/beast/http/write.hpp> -#include <boost/beast/ssl/ssl_stream.hpp> #include <functional> #include <memory> #include <string> @@ -97,7 +96,7 @@ public: completion(r, std::move(response)); } - void operator()(boost::beast::tcp_stream& stream) override { + void operator()(boost::asio::ip::tcp::socket& stream) override { preprocess_request(); boost::beast::http::async_write( @@ -110,7 +109,7 @@ public: } void operator()( - boost::beast::ssl_stream<boost::beast::tcp_stream>& stream) override { + boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream) override { preprocess_request(); boost::beast::http::async_write( @@ -152,9 +151,9 @@ private: virtual bool need_eof() const = 0; virtual bool header_only() const = 0; virtual void complete(int r, Response&&) = 0; - virtual void operator()(boost::beast::tcp_stream& stream) = 0; + virtual void operator()(boost::asio::ip::tcp::socket& stream) = 0; virtual void operator()( - boost::beast::ssl_stream<boost::beast::tcp_stream>& stream) = 0; + boost::asio::ssl::stream<boost::asio::ip::tcp::socket>& stream) = 0; }; template <typename D> struct HttpSession; diff --git a/src/librbd/operation/FlattenRequest.cc b/src/librbd/operation/FlattenRequest.cc index 7bc34681924..8034637e8e6 100644 --- a/src/librbd/operation/FlattenRequest.cc +++ b/src/librbd/operation/FlattenRequest.cc @@ -49,15 +49,6 @@ public: return -ERESTART; } - { - std::shared_lock image_lock{image_ctx.image_lock}; - if (image_ctx.object_map != nullptr && - !image_ctx.object_map->object_may_not_exist(m_object_no)) { - // can skip because the object already exists - return 1; - } - } - if (!io::util::trigger_copyup( &image_ctx, m_object_no, m_io_context, this)) { // stop early if the parent went away - it just means diff --git a/src/mds/Beacon.cc b/src/mds/Beacon.cc index 6fbfc79d416..1c1eeb4ecf8 100644 --- a/src/mds/Beacon.cc +++ b/src/mds/Beacon.cc @@ -26,6 +26,7 @@ #include "mds/MDSRank.h" #include "mds/MDSMap.h" #include "mds/Locker.h" +#include "mds/mdstypes.h" #include "Beacon.h" @@ -550,6 +551,19 @@ void Beacon::notify_health(MDSRank const *mds) } } } + if (mds->is_replay()) { + CachedStackStringStream css; + auto estimate = mds->mdlog->get_estimated_replay_finish_time(); + // this probably should be configurable, however, its fine to report + // if replay is running for more than 30 seconds. + if (estimate.elapsed_time > std::chrono::seconds(30)) { + *css << "replay: " << estimate.percent_complete << "% complete - elapsed time: " + << estimate.elapsed_time << ", estimated time remaining: " + << estimate.estimated_time; + MDSHealthMetric m(MDS_HEALTH_ESTIMATED_REPLAY_TIME, HEALTH_WARN, css->strv()); + health.metrics.push_back(m); + } + } } MDSMap::DaemonState Beacon::get_want_state() const diff --git a/src/mds/MDLog.cc b/src/mds/MDLog.cc index 4bbf2a1a141..d041e3b2fc8 100644 --- a/src/mds/MDLog.cc +++ b/src/mds/MDLog.cc @@ -221,7 +221,46 @@ uint64_t MDLog::get_safe_pos() const return journaler->get_write_safe_pos(); } +// estimate the replay completion time based on mdlog journal pointers +EstimatedReplayTime MDLog::get_estimated_replay_finish_time() { + ceph_assert(mds->is_replay()); + EstimatedReplayTime estimated_time{0, std::chrono::seconds::zero(), std::chrono::seconds::zero()}; + if (!journaler) { + return estimated_time; + } + + auto read_pos = journaler->get_read_pos(); + auto write_pos = journaler->get_write_pos(); + auto trimmed_pos = journaler->get_trimmed_pos(); + + dout(20) << __func__ << ": read_pos=" << read_pos << ", write_pos=" + << write_pos << ", trimmed_pos=" << trimmed_pos << dendl; + + if (read_pos == trimmed_pos || write_pos == trimmed_pos) { + return estimated_time; + } + + auto total_bytes = write_pos - trimmed_pos; + double percent_complete = ((double)(read_pos - trimmed_pos)) / (double)total_bytes; + auto elapsed_time = std::chrono::duration_cast<std::chrono::seconds> + (ceph::coarse_mono_clock::now() - replay_start_time); + auto time = ((1 - percent_complete) / percent_complete) * elapsed_time; + + dout(20) << __func__ << "percent_complete=" << percent_complete + << ", elapsed_time=" << elapsed_time + << ", estimated_time=" << std::chrono::round<std::chrono::seconds>(time) + << dendl; + + estimated_time.percent_complete = percent_complete * 100; + estimated_time.elapsed_time = elapsed_time; + estimated_time.estimated_time = std::chrono::round<std::chrono::seconds>(time); + dout(20) << __func__ << "estimated_time.percent_complete=" << estimated_time.percent_complete + << ", estimated_time.elapsed_time=" << estimated_time.elapsed_time + << ", estimated_time.estimated_time=" << estimated_time.estimated_time + << dendl; + return estimated_time; +} void MDLog::create(MDSContext *c) { @@ -1137,6 +1176,7 @@ void MDLog::_recovery_thread(MDSContext *completion) { std::lock_guard l(mds->mds_lock); journaler = front_journal; + replay_start_time = ceph::coarse_mono_clock::now(); } C_SaferCond recover_wait; @@ -1374,11 +1414,17 @@ void MDLog::_reformat_journal(JournalPointer const &jp_in, Journaler *old_journa // i am a separate thread void MDLog::_replay_thread() { - dout(10) << "_replay_thread start" << dendl; + dout(10) << __func__ << ": start time: " << replay_start_time << ", now: " + << ceph::coarse_mono_clock::now() << dendl; // loop int r = 0; while (1) { + auto sleep_time = g_conf().get_val<std::chrono::milliseconds>("mds_delay_journal_replay_for_testing"); + if (unlikely(sleep_time > 0ms)) { + dout(10) << __func__ << ": sleeping for " << sleep_time << "ms" << dendl; + std::this_thread::sleep_for(sleep_time); + } // wait for read? journaler->check_isreadable(); if (journaler->get_error()) { diff --git a/src/mds/MDLog.h b/src/mds/MDLog.h index a858b40fa03..180a34c9d82 100644 --- a/src/mds/MDLog.h +++ b/src/mds/MDLog.h @@ -53,6 +53,7 @@ enum { #include "LogSegment.h" #include "MDSMap.h" #include "SegmentBoundary.h" +#include "mdstypes.h" #include <list> #include <map> @@ -162,6 +163,7 @@ public: void reopen(MDSContext *onopen); void append(); void replay(MDSContext *onfinish); + EstimatedReplayTime get_estimated_replay_finish_time(); void standby_trim_segments(); @@ -328,5 +330,7 @@ private: std::atomic<bool> upkeep_log_trim_shutdown{false}; std::map<uint64_t, std::vector<Context*>> waiting_for_expire; // protected by mds_lock + + ceph::coarse_mono_time replay_start_time = ceph::coarse_mono_clock::zero(); }; #endif diff --git a/src/mds/MetricsHandler.cc b/src/mds/MetricsHandler.cc index 9fc4c6122a4..d9c09e06b27 100644 --- a/src/mds/MetricsHandler.cc +++ b/src/mds/MetricsHandler.cc @@ -20,15 +20,6 @@ MetricsHandler::MetricsHandler(CephContext *cct, MDSRank *mds) mds(mds) { } -bool MetricsHandler::ms_can_fast_dispatch2(const cref_t<Message> &m) const { - return m->get_type() == CEPH_MSG_CLIENT_METRICS || m->get_type() == MSG_MDS_PING; -} - -void MetricsHandler::ms_fast_dispatch2(const ref_t<Message> &m) { - bool handled = ms_dispatch2(m); - ceph_assert(handled); -} - bool MetricsHandler::ms_dispatch2(const ref_t<Message> &m) { if (m->get_type() == CEPH_MSG_CLIENT_METRICS && m->get_connection()->get_peer_type() == CEPH_ENTITY_TYPE_CLIENT) { diff --git a/src/mds/MetricsHandler.h b/src/mds/MetricsHandler.h index 0b75b024860..25ee208aa95 100644 --- a/src/mds/MetricsHandler.h +++ b/src/mds/MetricsHandler.h @@ -25,11 +25,6 @@ class MetricsHandler : public Dispatcher { public: MetricsHandler(CephContext *cct, MDSRank *mds); - bool ms_can_fast_dispatch_any() const override { - return true; - } - bool ms_can_fast_dispatch2(const cref_t<Message> &m) const override; - void ms_fast_dispatch2(const ref_t<Message> &m) override; bool ms_dispatch2(const ref_t<Message> &m) override; void ms_handle_connect(Connection *c) override { diff --git a/src/mds/mdstypes.cc b/src/mds/mdstypes.cc index 680218e62e3..f9424eed6dc 100644 --- a/src/mds/mdstypes.cc +++ b/src/mds/mdstypes.cc @@ -1042,3 +1042,8 @@ void snaprealm_reconnect_t::generate_test_instances(std::list<snaprealm_reconnec ls.back()->realm.seq = 2; ls.back()->realm.parent = 1; } + +void EstimatedReplayTime::print(std::ostream& out) { + out << "replay: " << percent_complete << "% complete - elapsed time: " + << elapsed_time << ", estimated time remaining: " << estimated_time; +} diff --git a/src/mds/mdstypes.h b/src/mds/mdstypes.h index 3b8269006cb..742d7b23432 100644 --- a/src/mds/mdstypes.h +++ b/src/mds/mdstypes.h @@ -1044,4 +1044,12 @@ inline bool operator==(const MDSCacheObjectInfo& l, const MDSCacheObjectInfo& r) } WRITE_CLASS_ENCODER(MDSCacheObjectInfo) +struct EstimatedReplayTime { + double percent_complete; + std::chrono::seconds estimated_time; + std::chrono::seconds elapsed_time; + + void print(std::ostream& out); +}; + #endif diff --git a/src/messages/MMDSBeacon.h b/src/messages/MMDSBeacon.h index c157c33e758..526285aae8c 100644 --- a/src/messages/MMDSBeacon.h +++ b/src/messages/MMDSBeacon.h @@ -48,6 +48,7 @@ enum mds_metric_t { MDS_HEALTH_CLIENTS_LAGGY, MDS_HEALTH_CLIENTS_LAGGY_MANY, MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH, + MDS_HEALTH_ESTIMATED_REPLAY_TIME, MDS_HEALTH_DUMMY, // not a real health warning, for testing }; @@ -69,6 +70,7 @@ inline const char *mds_metric_name(mds_metric_t m) case MDS_HEALTH_CLIENTS_LAGGY: return "MDS_CLIENTS_LAGGY"; case MDS_HEALTH_CLIENTS_LAGGY_MANY: return "MDS_CLIENTS_LAGGY_MANY"; case MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH: return "MDS_CLIENTS_BROKEN_ROOTSQUASH"; + case MDS_HEALTH_ESTIMATED_REPLAY_TIME: return "MDS_ESTIMATED_REPLAY_TIME"; case MDS_HEALTH_DUMMY: return "MDS_DUMMY"; default: return "???"; @@ -107,6 +109,8 @@ inline const char *mds_metric_summary(mds_metric_t m) return "%num% client(s) laggy due to laggy OSDs"; case MDS_HEALTH_CLIENTS_BROKEN_ROOTSQUASH: return "%num% MDS report clients with broken root_squash implementation"; + case MDS_HEALTH_ESTIMATED_REPLAY_TIME: + return "%num% estimated journal replay time"; default: return "???"; } diff --git a/src/mgr/PyModule.cc b/src/mgr/PyModule.cc index cff63ef4a6b..4f996489ba0 100644 --- a/src/mgr/PyModule.cc +++ b/src/mgr/PyModule.cc @@ -38,6 +38,18 @@ std::string PyModule::mgr_store_prefix = "mgr/"; #define BOOST_BIND_GLOBAL_PLACEHOLDERS // Boost apparently can't be bothered to fix its own usage of its own // deprecated features. + +// Fix instances of "'BOOST_PP_ITERATION_02' was not declared in this scope; did +// you mean 'BOOST_PP_ITERATION_05'" and related macro error bullshit that spans +// 300 lines of errors +// +// Apparently you can't include boost/python stuff _and_ have this header +// defined +// +// Thanks to the ceph-aur folks for the fix at: +// https://github.com/bazaah/aur-ceph/commit/8c5cc7d8deec002f7596b6d0860859a0a718f12b +#undef BOOST_MPL_CFG_NO_PREPROCESSED_HEADERS + #include <boost/python/extract.hpp> #include <boost/python/import.hpp> #include <boost/python/object.hpp> diff --git a/src/mgr/PyModule.h b/src/mgr/PyModule.h index 177447c2cb3..a47db3a47ef 100644 --- a/src/mgr/PyModule.h +++ b/src/mgr/PyModule.h @@ -161,9 +161,9 @@ public: } const std::string &get_name() const { - std::lock_guard l(lock) ; return module_name; + return module_name; } - const std::string &get_error_string() const { + std::string get_error_string() const { std::lock_guard l(lock) ; return error_string; } bool get_can_run() const { diff --git a/src/mon/FSCommands.cc b/src/mon/FSCommands.cc index 6220a357ff0..cc53d2869f7 100644 --- a/src/mon/FSCommands.cc +++ b/src/mon/FSCommands.cc @@ -1211,6 +1211,11 @@ class RemoveFilesystemHandler : public FileSystemCommandHandler fsmap.erase_filesystem(fsp->get_fscid()); + ss << "If there are active snapshot schedules associated with this " + << "file-system, you might see EIO errors in the mgr logs or at the " + << "snap-schedule command-line due to the missing file-system. " + << "However, these errors are transient and will get auto-resolved."; + return 0; } }; diff --git a/src/mon/Monitor.cc b/src/mon/Monitor.cc index 03826e3dd48..7332ec3edb1 100644 --- a/src/mon/Monitor.cc +++ b/src/mon/Monitor.cc @@ -5675,10 +5675,13 @@ void Monitor::handle_scrub(MonOpRequestRef op) if (scrub_result.size() == quorum.size()) { scrub_check_results(); scrub_result.clear(); - if (scrub_state->finished) + if (scrub_state->finished) { + const utime_t lat = ceph_clock_now() - scrub_state->start; + dout(10) << __func__ << " mon scrub latency: " << lat << dendl; scrub_finish(); - else + } else { scrub(); + } } } break; diff --git a/src/mon/Monitor.h b/src/mon/Monitor.h index 8c152fe108f..557edbf2eb4 100644 --- a/src/mon/Monitor.h +++ b/src/mon/Monitor.h @@ -342,8 +342,10 @@ private: struct ScrubState { std::pair<std::string,std::string> last_key; ///< last scrubbed key bool finished; + const utime_t start; - ScrubState() : finished(false) { } + ScrubState() : finished(false), + start(ceph_clock_now()) { } virtual ~ScrubState() { } }; std::shared_ptr<ScrubState> scrub_state; ///< keeps track of current scrub diff --git a/src/mon/NVMeofGwMap.cc b/src/mon/NVMeofGwMap.cc index 719403925ad..2d2735f1e7c 100755 --- a/src/mon/NVMeofGwMap.cc +++ b/src/mon/NVMeofGwMap.cc @@ -171,6 +171,8 @@ int NVMeofGwMap::cfg_delete_gw( << state.availability << " Resulting GW availability: " << state.availability << dendl; state.subsystems.clear();//ignore subsystems of this GW + utime_t now = ceph_clock_now(); + mon->nvmegwmon()->gws_deleting_time[group_key][gw_id] = now; return 0; } } @@ -895,10 +897,12 @@ struct CMonRequestProposal : public Context { } }; -void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const +void NVMeofGwMap::get_health_checks(health_check_map_t *checks) { list<string> singleGatewayDetail; list<string> gatewayDownDetail; + list<string> gatewayInDeletingDetail; + int deleting_gateways = 0; for (const auto& created_map_pair: created_gws) { const auto& group_key = created_map_pair.first; auto& group = group_key.second; @@ -915,9 +919,37 @@ void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const ostringstream ss; ss << "NVMeoF Gateway '" << gw_id << "' is unavailable." ; gatewayDownDetail.push_back(ss.str()); + } else if (gw_created.availability == gw_availability_t::GW_DELETING) { + deleting_gateways++; + utime_t now = ceph_clock_now(); + bool found_deleting_time = false; + auto gws_deleting_time = mon->nvmegwmon()->gws_deleting_time; + auto group_it = gws_deleting_time.find(group_key); + if (group_it != gws_deleting_time.end()) { + auto& gw_map = group_it->second; + auto gw_it = gw_map.find(gw_id); + if (gw_it != gw_map.end()) { + found_deleting_time = true; + utime_t delete_time = gw_it->second; + if ((now - delete_time) > g_conf().get_val<std::chrono::seconds>("mon_nvmeofgw_delete_grace").count()) { + ostringstream ss; + ss << "NVMeoF Gateway '" << gw_id << "' is in deleting state."; + gatewayInDeletingDetail.push_back(ss.str()); + } + } + } + if (!found_deleting_time) { + // DELETING gateway not found in gws_deleting_time, set timeout now + mon->nvmegwmon()->gws_deleting_time[group_key][gw_id] = now; + } } } } + if (deleting_gateways == 0) { + // no gateway in GW_DELETING state currently, flush old gws_deleting_time + mon->nvmegwmon()->gws_deleting_time.clear(); + } + if (!singleGatewayDetail.empty()) { ostringstream ss; ss << singleGatewayDetail.size() << " group(s) have only 1 nvmeof gateway" @@ -934,6 +966,15 @@ void NVMeofGwMap::get_health_checks(health_check_map_t *checks) const ss.str(), gatewayDownDetail.size()); d.detail.swap(gatewayDownDetail); } + if (!gatewayInDeletingDetail.empty()) { + ostringstream ss; + ss << gatewayInDeletingDetail.size() << " gateway(s) are in deleting state" + << "; namespaces are automatically balanced across remaining gateways, " + << "this should take a few minutes."; + auto& d = checks->add("NVMEOF_GATEWAY_DELETING", HEALTH_WARN, + ss.str(), gatewayInDeletingDetail.size()); + d.detail.swap(gatewayInDeletingDetail); + } } int NVMeofGwMap::blocklist_gw( diff --git a/src/mon/NVMeofGwMap.h b/src/mon/NVMeofGwMap.h index 5f657733012..85fd62b3a07 100755 --- a/src/mon/NVMeofGwMap.h +++ b/src/mon/NVMeofGwMap.h @@ -144,7 +144,7 @@ public: DECODE_FINISH(bl); } - void get_health_checks(health_check_map_t *checks) const; + void get_health_checks(health_check_map_t *checks); }; #include "NVMeofGwSerialize.h" diff --git a/src/mon/NVMeofGwMon.cc b/src/mon/NVMeofGwMon.cc index 0fe5c3e655f..c9a6b789b89 100644 --- a/src/mon/NVMeofGwMon.cc +++ b/src/mon/NVMeofGwMon.cc @@ -66,11 +66,6 @@ void NVMeofGwMon::on_shutdown() void NVMeofGwMon::tick() { - if (++tick_ratio == 10) { - global_rebalance_index++; - dout(20) << "rebalance index " << global_rebalance_index << dendl; - tick_ratio = 0; - } if (!is_active() || !mon.is_leader()) { dout(10) << "NVMeofGwMon leader : " << mon.is_leader() << "active : " << is_active() << dendl; @@ -329,8 +324,9 @@ bool NVMeofGwMon::preprocess_command(MonOpRequestRef op) if (HAVE_FEATURE(mon.get_quorum_con_features(), NVMEOFHA)) { f->dump_string("features", "LB"); if (map.created_gws[group_key].size()) { - uint32_t index = (global_rebalance_index % - map.created_gws[group_key].size()) + 1; + time_t seconds_since_1970 = time(NULL); + uint32_t index = ((seconds_since_1970/60) % + map.created_gws[group_key].size()) + 1; f->dump_unsigned("rebalance_ana_group", index); } } @@ -625,15 +621,15 @@ bool NVMeofGwMon::prepare_beacon(MonOpRequestRef op) avail = gw_availability_t::GW_CREATED; dout(20) << "No-subsystems condition detected for GW " << gw_id <<dendl; } else { - bool listener_found = true; + bool listener_found = false; for (auto &subs: sub) { - if (subs.listeners.size() == 0) { - listener_found = false; - dout(10) << "No-listeners condition detected for GW " << gw_id << " for nqn " << subs.nqn << dendl; + if (subs.listeners.size()) { + listener_found = true; break; } } if (!listener_found) { + dout(10) << "No-listeners condition detected for GW " << gw_id << dendl; avail = gw_availability_t::GW_CREATED; } }// for HA no-subsystems and no-listeners are same usecases diff --git a/src/mon/NVMeofGwMon.h b/src/mon/NVMeofGwMon.h index 2d13e153bd2..d7f5fd89cde 100644 --- a/src/mon/NVMeofGwMon.h +++ b/src/mon/NVMeofGwMon.h @@ -82,10 +82,9 @@ public: void check_subs(bool type); void check_sub(Subscription *sub); + std::map<NvmeGroupKey, std::map<NvmeGwId, utime_t>> gws_deleting_time; + private: - // used for calculate pool & group GW responsible for rebalance - uint32_t global_rebalance_index = 1; - uint8_t tick_ratio = 0; void synchronize_last_beacon(); void process_gw_down(const NvmeGwId &gw_id, const NvmeGroupKey& group_key, bool &propose_pending, diff --git a/src/msg/async/AsyncMessenger.cc b/src/msg/async/AsyncMessenger.cc index 2e38bd434a8..6b3a8c3f6dc 100644 --- a/src/msg/async/AsyncMessenger.cc +++ b/src/msg/async/AsyncMessenger.cc @@ -207,22 +207,22 @@ void Processor::accept() } else if (r == -EAGAIN) { break; } else if (r == -EMFILE || r == -ENFILE) { - lderr(msgr->cct) << __func__ << " open file descriptions limit reached sd = " << listen_socket.fd() + lderr(msgr->cct) << __func__ << " open file descriptors limit reached fd = " << listen_socket.fd() << " errno " << r << " " << cpp_strerror(r) << dendl; if (++accept_error_num > msgr->cct->_conf->ms_max_accept_failures) { - lderr(msgr->cct) << "Proccessor accept has encountered enough error numbers, just do ceph_abort()." << dendl; + lderr(msgr->cct) << "Proccessor accept has encountered too many errors, just do ceph_abort()." << dendl; ceph_abort(); } continue; } else if (r == -ECONNABORTED) { - ldout(msgr->cct, 0) << __func__ << " it was closed because of rst arrived sd = " << listen_socket.fd() + ldout(msgr->cct, 0) << __func__ << " closed because of rst arrival fd = " << listen_socket.fd() << " errno " << r << " " << cpp_strerror(r) << dendl; continue; } else { lderr(msgr->cct) << __func__ << " no incoming connection?" << " errno " << r << " " << cpp_strerror(r) << dendl; if (++accept_error_num > msgr->cct->_conf->ms_max_accept_failures) { - lderr(msgr->cct) << "Proccessor accept has encountered enough error numbers, just do ceph_abort()." << dendl; + lderr(msgr->cct) << "Proccessor accept has encountered too many errors, just do ceph_abort()." << dendl; ceph_abort(); } continue; diff --git a/src/msg/async/Event.cc b/src/msg/async/Event.cc index 08e117ea54a..16abb1368b0 100644 --- a/src/msg/async/Event.cc +++ b/src/msg/async/Event.cc @@ -347,7 +347,7 @@ void EventCenter::wakeup() return ; ldout(cct, 20) << __func__ << dendl; - char buf = 'c'; + static constexpr char buf = 'c'; // wake up "event_wait" #ifdef _WIN32 int n = send(notify_send_fd, &buf, sizeof(buf), 0); diff --git a/src/nvmeof/NVMeofGwMonitorClient.cc b/src/nvmeof/NVMeofGwMonitorClient.cc index ce3328aec51..1b128055e08 100644 --- a/src/nvmeof/NVMeofGwMonitorClient.cc +++ b/src/nvmeof/NVMeofGwMonitorClient.cc @@ -42,7 +42,6 @@ NVMeofGwMonitorClient::NVMeofGwMonitorClient(int argc, const char **argv) : monc{g_ceph_context, poolctx}, client_messenger(Messenger::create(g_ceph_context, "async", entity_name_t::CLIENT(-1), "client", getpid())), objecter{g_ceph_context, client_messenger.get(), &monc, poolctx}, - client{client_messenger.get(), &monc, &objecter}, timer(g_ceph_context, beacon_lock), orig_argc(argc), orig_argv(argv) @@ -134,7 +133,6 @@ int NVMeofGwMonitorClient::init() // Initialize Messenger client_messenger->add_dispatcher_tail(this); client_messenger->add_dispatcher_head(&objecter); - client_messenger->add_dispatcher_tail(&client); client_messenger->start(); poolctx.start(2); @@ -190,7 +188,6 @@ int NVMeofGwMonitorClient::init() objecter.init(); objecter.enable_blocklist_events(); objecter.start(); - client.init(); timer.init(); { @@ -302,8 +299,7 @@ void NVMeofGwMonitorClient::shutdown() std::lock_guard bl(beacon_lock); timer.shutdown(); } - // client uses monc and objecter - client.shutdown(); + // Stop asio threads, so leftover events won't call into shut down // monclient/objecter. poolctx.finish(); diff --git a/src/nvmeof/NVMeofGwMonitorClient.h b/src/nvmeof/NVMeofGwMonitorClient.h index 6dd167e4e58..e01c823afb5 100644 --- a/src/nvmeof/NVMeofGwMonitorClient.h +++ b/src/nvmeof/NVMeofGwMonitorClient.h @@ -21,7 +21,6 @@ #include "common/Timer.h" #include "common/LogClient.h" -#include "client/Client.h" #include "mon/MonClient.h" #include "osdc/Objecter.h" #include "messages/MNVMeofGwMap.h" @@ -58,7 +57,6 @@ protected: MonClient monc; std::unique_ptr<Messenger> client_messenger; Objecter objecter; - Client client; std::map<NvmeGroupKey, NvmeGwMonClientStates> map; ceph::mutex lock = ceph::make_mutex("NVMeofGw::lock"); // allow beacons to be sent independently of handle_nvmeof_gw_map diff --git a/src/os/DBObjectMap.cc b/src/os/DBObjectMap.cc index 7da9a67be62..65627b5f818 100644 --- a/src/os/DBObjectMap.cc +++ b/src/os/DBObjectMap.cc @@ -519,6 +519,11 @@ bufferlist DBObjectMap::DBObjectMapIteratorImpl::value() return cur_iter->value(); } +std::string_view DBObjectMap::DBObjectMapIteratorImpl::value_as_sv() +{ + return cur_iter->value_as_sv(); +} + int DBObjectMap::DBObjectMapIteratorImpl::status() { return r; diff --git a/src/os/DBObjectMap.h b/src/os/DBObjectMap.h index 444f21eb815..1e1452010e7 100644 --- a/src/os/DBObjectMap.h +++ b/src/os/DBObjectMap.h @@ -393,6 +393,7 @@ private: int next() override { ceph_abort(); return 0; } std::string key() override { ceph_abort(); return ""; } ceph::buffer::list value() override { ceph_abort(); return ceph::buffer::list(); } + std::string_view value_as_sv() override { ceph_abort(); return std::string_view(); } int status() override { return 0; } }; @@ -431,6 +432,7 @@ private: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; int status() override; bool on_parent() { diff --git a/src/os/ObjectStore.h b/src/os/ObjectStore.h index 521435b6c31..df3ae920a2f 100644 --- a/src/os/ObjectStore.h +++ b/src/os/ObjectStore.h @@ -29,6 +29,7 @@ #include <errno.h> #include <sys/stat.h> +#include <functional> #include <map> #include <memory> #include <vector> @@ -735,15 +736,6 @@ public: std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) = 0; -#ifdef WITH_SEASTAR - virtual int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) = 0; -#endif - /// Filters keys into out which are defined on oid virtual int omap_check_keys( CollectionHandle &c, ///< [in] Collection containing oid @@ -766,6 +758,48 @@ public: const ghobject_t &oid ///< [in] object ) = 0; + struct omap_iter_seek_t { + std::string seek_position; + enum { + // start with provided key (seek_position), if it exists + LOWER_BOUND, + // skip provided key (seek_position) even if it exists + UPPER_BOUND + } seek_type = LOWER_BOUND; + static omap_iter_seek_t min_lower_bound() { return {}; } + }; + enum class omap_iter_ret_t { + STOP, + NEXT + }; + /** + * Iterate over object map with user-provided callable + * + * Warning! The callable is executed under lock on bluestore + * operations in c. Do not use bluestore methods on c while + * iterating. (Filling in a transaction is no problem). + * + * @param c collection + * @param oid object + * @param start_from where the iterator should point to at + * the beginning + * @param visitor callable that takes OMAP key and corresponding + * value as string_views and controls iteration + * by the return. It is executed for every object's + * OMAP entry from `start_from` till end of the + * object's OMAP or till the iteration is stopped + * by `STOP`. Please note that if there is no such + * entry, `visitor` will be called 0 times. + * @return error code, zero on success + */ + virtual int omap_iterate( + CollectionHandle &c, + const ghobject_t &oid, + omap_iter_seek_t start_from, + std::function<omap_iter_ret_t(std::string_view, + std::string_view)> visitor + ) = 0; + virtual int flush_journal() { return -EOPNOTSUPP; } virtual int dump_journal(std::ostream& out) { return -EOPNOTSUPP; } diff --git a/src/os/bluestore/BlueFS.cc b/src/os/bluestore/BlueFS.cc index 5f4f1a4d48a..2f88acdc93b 100644 --- a/src/os/bluestore/BlueFS.cc +++ b/src/os/bluestore/BlueFS.cc @@ -5,6 +5,7 @@ #include "bluestore_common.h" #include "BlueFS.h" +#include "common/Clock.h" // for ceph_clock_now() #include "common/debug.h" #include "common/errno.h" #include "common/perf_counters.h" @@ -12,6 +13,12 @@ #include "include/ceph_assert.h" #include "common/admin_socket.h" +#if defined(WITH_SEASTAR) && !defined(WITH_ALIEN) +#include "crimson/common/perf_counters_collection.h" +#else +#include "common/perf_counters_collection.h" +#endif + #define dout_context cct #define dout_subsys ceph_subsys_bluefs #undef dout_prefix @@ -1699,7 +1706,8 @@ int BlueFS::_replay(bool noop, bool to_stdout) << " fnode=" << fnode << " delta=" << delta << dendl; - ceph_assert(delta.offset == fnode.allocated); + // be leanient, if there is no extents just produce error message + ceph_assert(delta.offset == fnode.allocated || delta.extents.empty()); } if (cct->_conf->bluefs_log_replay_check_allocations) { int r = _check_allocations(fnode, @@ -3823,6 +3831,7 @@ int BlueFS::truncate(FileWriter *h, uint64_t offset)/*_WF_L*/ fnode.size = offset; fnode.allocated = new_allocated; fnode.reset_delta(); + fnode.recalc_allocated(); log.t.op_file_update(fnode); // sad, but is_dirty must be set to signal flushing of the log h->file->is_dirty = true; diff --git a/src/os/bluestore/BlueStore.cc b/src/os/bluestore/BlueStore.cc index a024a0c2105..25e6c4fe596 100644 --- a/src/os/bluestore/BlueStore.cc +++ b/src/os/bluestore/BlueStore.cc @@ -4830,7 +4830,7 @@ void BlueStore::Onode::rewrite_omap_key(const string& old, string *out) out->append(old.c_str() + out->length(), old.size() - out->length()); } -void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) +size_t BlueStore::Onode::calc_userkey_offset_in_omap_key() const { size_t pos = sizeof(uint64_t) + 1; if (!onode.is_pgmeta_omap()) { @@ -4840,9 +4840,15 @@ void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) pos += sizeof(uint64_t); } } - *user_key = key.substr(pos); + return pos; } +void BlueStore::Onode::decode_omap_key(const string& key, string *user_key) +{ + *user_key = key.substr(calc_userkey_offset_in_omap_key()); +} + + void BlueStore::Onode::finish_write(TransContext* txc, uint32_t offset, uint32_t length) { while (true) { @@ -5519,7 +5525,13 @@ BlueStore::OmapIteratorImpl::OmapIteratorImpl( if (o->onode.has_omap()) { o->get_omap_key(string(), &head); o->get_omap_tail(&tail); + auto start1 = mono_clock::now(); it->lower_bound(head); + c->store->log_latency( + __func__, + l_bluestore_omap_seek_to_first_lat, + mono_clock::now() - start1, + c->store->cct->_conf->bluestore_log_omap_iterator_age); } } BlueStore::OmapIteratorImpl::~OmapIteratorImpl() @@ -5654,6 +5666,13 @@ bufferlist BlueStore::OmapIteratorImpl::value() return it->value(); } +std::string_view BlueStore::OmapIteratorImpl::value_as_sv() +{ + std::shared_lock l(c->lock); + ceph_assert(it->valid()); + return it->value_as_sv(); +} + // ===================================== @@ -13601,52 +13620,6 @@ int BlueStore::omap_get_values( return r; } -#ifdef WITH_SEASTAR -int BlueStore::omap_get_values( - CollectionHandle &c_, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<string> &start_after, ///< [in] Keys to get - map<string, bufferlist> *output ///< [out] Returned keys and values - ) -{ - Collection *c = static_cast<Collection *>(c_.get()); - dout(15) << __func__ << " " << c->get_cid() << " oid " << oid << dendl; - if (!c->exists) - return -ENOENT; - std::shared_lock l(c->lock); - int r = 0; - OnodeRef o = c->get_onode(oid, false); - if (!o || !o->exists) { - r = -ENOENT; - goto out; - } - if (!o->onode.has_omap()) { - goto out; - } - o->flush(); - { - ObjectMap::ObjectMapIterator iter = get_omap_iterator(c_, oid); - if (!iter) { - r = -ENOENT; - goto out; - } - if (start_after) { - iter->upper_bound(*start_after); - } else { - iter->seek_to_first(); - } - for (; iter->valid(); iter->next()) { - output->insert(make_pair(iter->key(), iter->value())); - } - } - -out: - dout(10) << __func__ << " " << c->get_cid() << " oid " << oid << " = " << r - << dendl; - return r; -} -#endif - int BlueStore::omap_check_keys( CollectionHandle &c_, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -13724,6 +13697,94 @@ ObjectMap::ObjectMapIterator BlueStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(logger,c, o, it)); } +int BlueStore::omap_iterate( + CollectionHandle &c_, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) +{ + Collection *c = static_cast<Collection *>(c_.get()); + dout(10) << __func__ << " " << c->get_cid() << " " << oid << dendl; + if (!c->exists) { + return -ENOENT; + } + std::shared_lock l(c->lock); + OnodeRef o = c->get_onode(oid, false); + if (!o || !o->exists) { + dout(10) << __func__ << " " << oid << "doesn't exist" <<dendl; + return -ENOENT; + } + o->flush(); + dout(10) << __func__ << " has_omap = " << (int)o->onode.has_omap() <<dendl; + if (!o->onode.has_omap()) { + // nothing to do + return 0; + } + + KeyValueDB::Iterator it; + { + auto bounds = KeyValueDB::IteratorBounds(); + std::string lower_bound, upper_bound; + o->get_omap_key(string(), &lower_bound); + o->get_omap_tail(&upper_bound); + bounds.lower_bound = std::move(lower_bound); + bounds.upper_bound = std::move(upper_bound); + it = db->get_iterator(o->get_omap_prefix(), 0, std::move(bounds)); + } + + // seek the iterator + { + std::string key; + o->get_omap_key(start_from.seek_position, &key); + auto start = ceph::mono_clock::now(); + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it->lower_bound(key); + c->store->log_latency( + __func__, + l_bluestore_omap_lower_bound_lat, + ceph::mono_clock::now() - start, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + } else { + it->upper_bound(key); + c->store->log_latency( + __func__, + l_bluestore_omap_upper_bound_lat, + ceph::mono_clock::now() - start, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + } + } + + // iterate! + std::string tail; + o->get_omap_tail(&tail); + const std::string_view::size_type userkey_offset_in_dbkey = + o->calc_userkey_offset_in_omap_key(); + ceph::timespan next_lat_acc{0}; + while (it->valid()) { + const auto& db_key = it->raw_key_as_sv().second; + if (db_key >= tail) { + break; + } + std::string_view user_key = db_key.substr(userkey_offset_in_dbkey); + omap_iter_ret_t ret = f(user_key, it->value_as_sv()); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + ceph::time_guard<ceph::mono_clock>{next_lat_acc}; + it->next(); + } else { + ceph_abort(); + } + } + c->store->log_latency( + __func__, + l_bluestore_omap_next_lat, + next_lat_acc, + c->store->cct->_conf->bluestore_log_omap_iterator_age); + return 0; +} + // ----------------- // write helpers diff --git a/src/os/bluestore/BlueStore.h b/src/os/bluestore/BlueStore.h index 99f8d057cf0..5549f97ffea 100644 --- a/src/os/bluestore/BlueStore.h +++ b/src/os/bluestore/BlueStore.h @@ -1457,6 +1457,7 @@ public: } void rewrite_omap_key(const std::string& old, std::string *out); + size_t calc_userkey_offset_in_omap_key() const; void decode_omap_key(const std::string& key, std::string *user_key); void finish_write(TransContext* txc, uint32_t offset, uint32_t length); @@ -1753,6 +1754,7 @@ public: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; std::string tail_key() override { return tail; } @@ -3416,15 +3418,6 @@ public: std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) override; -#ifdef WITH_SEASTAR - int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) override; -#endif - /// Filters keys into out which are defined on oid int omap_check_keys( CollectionHandle &c, ///< [in] Collection containing oid @@ -3438,6 +3431,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override { fsid = u; } diff --git a/src/os/bluestore/bluefs_types.cc b/src/os/bluestore/bluefs_types.cc index e18dd490140..fe77f7f74d8 100644 --- a/src/os/bluestore/bluefs_types.cc +++ b/src/os/bluestore/bluefs_types.cc @@ -154,7 +154,9 @@ mempool::bluefs::vector<bluefs_extent_t>::iterator bluefs_fnode_t::seek( assert(it != extents_index.begin()); --it; assert(offset >= *it); - p += it - extents_index.begin(); + uint32_t skip = it - extents_index.begin(); + ceph_assert(skip <= extents.size()); + p += skip; offset -= *it; } diff --git a/src/os/bluestore/bluefs_types.h b/src/os/bluestore/bluefs_types.h index 627118c12f8..08b3ca0cf41 100644 --- a/src/os/bluestore/bluefs_types.h +++ b/src/os/bluestore/bluefs_types.h @@ -89,6 +89,7 @@ struct bluefs_fnode_t { void recalc_allocated() { allocated = 0; extents_index.reserve(extents.size()); + extents_index.clear(); for (auto& p : extents) { extents_index.emplace_back(allocated); allocated += p.length; diff --git a/src/os/bluestore/bluestore_tool.cc b/src/os/bluestore/bluestore_tool.cc index d62721b4366..16f1e6434e0 100644 --- a/src/os/bluestore/bluestore_tool.cc +++ b/src/os/bluestore/bluestore_tool.cc @@ -1136,7 +1136,7 @@ int main(int argc, char **argv) } return r; } - } else if (action == "free-dump" || action == "free-score" || action == "fragmentation") { + } else if (action == "free-dump" || action == "free-score" || action == "free-fragmentation") { AdminSocket *admin_socket = g_ceph_context->get_admin_socket(); ceph_assert(admin_socket); std::string action_name = action == "free-dump" ? "dump" : diff --git a/src/os/kstore/KStore.cc b/src/os/kstore/KStore.cc index 7158486ca38..a069d429155 100644 --- a/src/os/kstore/KStore.cc +++ b/src/os/kstore/KStore.cc @@ -1651,6 +1651,13 @@ bufferlist KStore::OmapIteratorImpl::value() return it->value(); } +std::string_view KStore::OmapIteratorImpl::value_as_sv() +{ + std::shared_lock l{c->lock}; + ceph_assert(it->valid()); + return it->value_as_sv(); +} + int KStore::omap_get( CollectionHandle& ch, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -1866,6 +1873,71 @@ ObjectMap::ObjectMapIterator KStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(c, o, it)); } +int KStore::omap_iterate( + CollectionHandle &ch, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f) +{ + dout(10) << __func__ << " " << ch->cid << " " << oid << dendl; + Collection *c = static_cast<Collection*>(ch.get()); + { + std::shared_lock l{c->lock}; + + OnodeRef o = c->get_onode(oid, false); + if (!o || !o->exists) { + dout(10) << __func__ << " " << oid << "doesn't exist" <<dendl; + return -ENOENT; + } + o->flush(); + dout(10) << __func__ << " header = " << o->onode.omap_head <<dendl; + + KeyValueDB::Iterator it = db->get_iterator(PREFIX_OMAP); + std::string tail; + std::string seek_key; + if (o->onode.omap_head) { + return 0; // nothing to do + } + + // acquire data depedencies for seek & iterate + get_omap_key(o->onode.omap_head, start_from.seek_position, &seek_key); + get_omap_tail(o->onode.omap_head, &tail); + + // acquire the iterator + { + it = db->get_iterator(PREFIX_OMAP); + } + + // seek the iterator + { + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it->lower_bound(seek_key); + } else { + it->upper_bound(seek_key); + } + } + + // iterate! + while (it->valid()) { + std::string user_key; + if (const auto& db_key = it->raw_key().second; db_key >= tail) { + break; + } else { + decode_omap_key(db_key, &user_key); + } + omap_iter_ret_t ret = f(user_key, it->value_as_sv()); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + it->next(); + } else { + ceph_abort(); + } + } + } + return 0; +} + // ----------------- // write helpers diff --git a/src/os/kstore/KStore.h b/src/os/kstore/KStore.h index 9a9d413c66a..06115d3cab7 100644 --- a/src/os/kstore/KStore.h +++ b/src/os/kstore/KStore.h @@ -180,6 +180,7 @@ public: int next() override; std::string key() override; ceph::buffer::list value() override; + std::string_view value_as_sv() override; int status() override { return 0; } @@ -553,6 +554,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override { fsid = u; } diff --git a/src/os/memstore/MemStore.cc b/src/os/memstore/MemStore.cc index 89cb09361cf..f9d3bf0d8a2 100644 --- a/src/os/memstore/MemStore.cc +++ b/src/os/memstore/MemStore.cc @@ -537,30 +537,6 @@ int MemStore::omap_get_values( return 0; } -#ifdef WITH_SEASTAR -int MemStore::omap_get_values( - CollectionHandle& ch, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) -{ - dout(10) << __func__ << " " << ch->cid << " " << oid << dendl; - Collection *c = static_cast<Collection*>(ch.get()); - ObjectRef o = c->get_object(oid); - if (!o) - return -ENOENT; - assert(start_after); - std::lock_guard lock{o->omap_mutex}; - for (auto it = o->omap.upper_bound(*start_after); - it != std::end(o->omap); - ++it) { - out->insert(*it); - } - return 0; -} -#endif - int MemStore::omap_check_keys( CollectionHandle& ch, ///< [in] Collection containing oid const ghobject_t &oid, ///< [in] Object containing omap @@ -622,6 +598,10 @@ public: std::lock_guard lock{o->omap_mutex}; return it->second; } + std::string_view value_as_sv() override { + std::lock_guard lock{o->omap_mutex}; + return std::string_view{it->second.c_str(), it->second.length()}; + } int status() override { return 0; } @@ -639,6 +619,48 @@ ObjectMap::ObjectMapIterator MemStore::get_omap_iterator( return ObjectMap::ObjectMapIterator(new OmapIteratorImpl(c, o)); } +int MemStore::omap_iterate( + CollectionHandle &ch, ///< [in] collection + const ghobject_t &oid, ///< [in] object + ObjectStore::omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f) +{ + Collection *c = static_cast<Collection*>(ch.get()); + ObjectRef o = c->get_object(oid); + if (!o) { + return -ENOENT; + } + + { + std::lock_guard lock{o->omap_mutex}; + + // obtain seek the iterator + decltype(o->omap)::iterator it; + { + if (start_from.seek_type == omap_iter_seek_t::LOWER_BOUND) { + it = o->omap.lower_bound(start_from.seek_position); + } else { + it = o->omap.upper_bound(start_from.seek_position); + } + } + + // iterate! + while (it != o->omap.end()) { + // potentially rectifying memcpy but who cares for memstore? + omap_iter_ret_t ret = + f(it->first, std::string_view{it->second.c_str(), it->second.length()}); + if (ret == omap_iter_ret_t::STOP) { + break; + } else if (ret == omap_iter_ret_t::NEXT) { + ++it; + } else { + ceph_abort(); + } + } + } + return 0; +} + // --------------- // write operations diff --git a/src/os/memstore/MemStore.h b/src/os/memstore/MemStore.h index 2abe552891f..9621773598f 100644 --- a/src/os/memstore/MemStore.h +++ b/src/os/memstore/MemStore.h @@ -363,14 +363,6 @@ public: const std::set<std::string> &keys, ///< [in] Keys to get std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values ) override; -#ifdef WITH_SEASTAR - int omap_get_values( - CollectionHandle &c, ///< [in] Collection containing oid - const ghobject_t &oid, ///< [in] Object containing omap - const std::optional<std::string> &start_after, ///< [in] Keys to get - std::map<std::string, ceph::buffer::list> *out ///< [out] Returned keys and values - ) override; -#endif using ObjectStore::omap_check_keys; /// Filters keys into out which are defined on oid @@ -387,6 +379,13 @@ public: const ghobject_t &oid ///< [in] object ) override; + int omap_iterate( + CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + omap_iter_seek_t start_from, ///< [in] where the iterator should point to at the beginning + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override; + void set_fsid(uuid_d u) override; uuid_d get_fsid() override; diff --git a/src/osd/ECBackend.cc b/src/osd/ECBackend.cc index fa2570aba42..8630b038812 100644 --- a/src/osd/ECBackend.cc +++ b/src/osd/ECBackend.cc @@ -945,6 +945,10 @@ void ECBackend::handle_sub_write( } trace.event("handle_sub_write"); + if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_write_error3(op.soid)) { + ceph_abort_msg("Error inject - OSD down"); + } if (!get_parent()->pgb_is_primary()) get_parent()->update_stats(op.stats); ObjectStore::Transaction localt; @@ -1191,6 +1195,15 @@ void ECBackend::handle_sub_write_reply( i->second->on_all_commit = 0; i->second->trace.event("ec write all committed"); } + if (cct->_conf->bluestore_debug_inject_read_err && + (i->second->pending_commit.size() == 1) && + ec_inject_test_write_error2(i->second->hoid)) { + std::string cmd = + "{ \"prefix\": \"osd down\", \"ids\": [\"" + std::to_string( get_parent()->whoami() ) + "\"] }"; + vector<std::string> vcmd{cmd}; + dout(0) << __func__ << " Error inject - marking OSD down" << dendl; + get_parent()->start_mon_command(vcmd, {}, nullptr, nullptr, nullptr); + } rmw_pipeline.check_ops(); } @@ -1208,6 +1221,19 @@ void ECBackend::handle_sub_read_reply( return; } ReadOp &rop = iter->second; + if (cct->_conf->bluestore_debug_inject_read_err) { + for (auto i = op.buffers_read.begin(); + i != op.buffers_read.end(); + ++i) { + if (ec_inject_test_read_error0(ghobject_t(i->first, ghobject_t::NO_GEN, op.from.shard))) { + dout(0) << __func__ << " Error inject - EIO error for shard " << op.from.shard << dendl; + op.buffers_read.erase(i->first); + op.attrs_read.erase(i->first); + op.errors[i->first] = -EIO; + } + + } + } for (auto i = op.buffers_read.begin(); i != op.buffers_read.end(); ++i) { diff --git a/src/osd/ECCommon.cc b/src/osd/ECCommon.cc index 609ac3141ae..59077547fcb 100644 --- a/src/osd/ECCommon.cc +++ b/src/osd/ECCommon.cc @@ -226,8 +226,14 @@ void ECCommon::ReadPipeline::get_all_avail_shards( ++i) { dout(10) << __func__ << ": checking acting " << *i << dendl; const pg_missing_t &missing = get_parent()->get_shard_missing(*i); - if (error_shards.find(*i) != error_shards.end()) + if (error_shards.contains(*i)) { continue; + } + if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_read_error1(ghobject_t(hoid, ghobject_t::NO_GEN, i->shard))) { + dout(0) << __func__ << " Error inject - Missing shard " << i->shard << dendl; + continue; + } if (!missing.is_missing(hoid)) { ceph_assert(!have.count(i->shard)); have.insert(i->shard); @@ -912,6 +918,11 @@ bool ECCommon::RMWPipeline::try_reads_to_commit() if (*i == get_parent()->whoami_shard()) { should_write_local = true; local_write_op.claim(sop); + } else if (cct->_conf->bluestore_debug_inject_read_err && + ec_inject_test_write_error1(ghobject_t(op->hoid, + ghobject_t::NO_GEN, i->shard))) { + dout(0) << " Error inject - Dropping write message to shard " << + i->shard << dendl; } else { MOSDECSubOpWrite *r = new MOSDECSubOpWrite(sop); r->pgid = spg_t(get_parent()->primary_spg_t().pgid, i->shard); @@ -1090,3 +1101,305 @@ ECUtil::HashInfoRef ECCommon::UnstableHashInfoRegistry::get_hash_info( } return ref; } + +// Error inject interfaces +static ceph::recursive_mutex ec_inject_lock = + ceph::make_recursive_mutex("ECCommon::ec_inject_lock"); +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_read_failures0; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_read_failures1; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures0; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures1; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures2; +static std::map<ghobject_t,std::pair<int64_t,int64_t>> ec_inject_write_failures3; +static std::map<ghobject_t,shard_id_t> ec_inject_write_failures0_shard; +static std::set<osd_reqid_t> ec_inject_write_failures0_reqid; + +/** + * Configure a read error inject that typically forces additional reads of + * shards in an EC pool to recover data using the redundancy. With multiple + * errors it is possible to force client reads to fail. + * + * Type 0 - Simulate a medium error. Fail a read with -EIO to force + * additional reads and a decode + * + * Type 1 - Simulate a missing OSD. Dont even try to read a shard + * + * @brief Set up a read error inject for an object in an EC pool. + * @param o Target object for the error inject. + * @param when Error inject starts after this many object store reads. + * @param duration Error inject affects this many object store reads. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Result of configuring the error inject. + */ +std::string ec_inject_read_error(const ghobject_t& o, + const int64_t type, + const int64_t when, + const int64_t duration) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + ghobject_t os = o; + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + switch (type) { + case 0: + ec_inject_read_failures0[os] = std::pair(when, duration); + return "ok - read returns EIO"; + case 1: + ec_inject_read_failures1[os] = std::pair(when, duration); + return "ok - read pretends shard is missing"; + default: + break; + } + return "unrecognized error inject type"; +} + +/** + * Configure a write error inject that either fails an OSD or causes a + * client write operation to be rolled back. + * + * Type 0 - Tests rollback. Drop a write I/O to a shard, then simulate an OSD + * down to force rollback to occur, lastly fail the retried write from the + * client so the results of the rollback can be inspected. + * + * Type 1 - Drop a write I/O to a shard. Used on its own this will hang a + * write I/O. + * + * Type 2 - Simulate an OSD down (ceph osd down) to force a new epoch. Usually + * used together with type 1 to force a rollback + * + * Type 3 - Abort when an OSD processes a write I/O to a shard. Typically the + * client write will be commited while the OSD is absent which will result in + * recovery or backfill later when the OSD returns. + * + * @brief Set up a write error inject for an object in an EC pool. + * @param o Target object for the error inject. + * @param when Error inject starts after this many object store reads. + * @param duration Error inject affects this many object store reads. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Result of configuring the error inect. + */ +std::string ec_inject_write_error(const ghobject_t& o, + const int64_t type, + const int64_t when, + const int64_t duration) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + bool no_shard = true; + std::string result; + switch (type) { + case 0: + failures = &ec_inject_write_failures0; + result = "ok - drop write, sim OSD down and fail client retry with EINVAL"; + break; + case 1: + failures = &ec_inject_write_failures1; + no_shard = false; + result = "ok - drop write to shard"; + break; + case 2: + failures = &ec_inject_write_failures2; + result = "ok - inject OSD down"; + break; + case 3: + if (duration != 1) { + return "duration must be 1"; + } + failures = &ec_inject_write_failures3; + result = "ok - write abort OSDs"; + break; + default: + return "unrecognized error inject type"; + } + if (no_shard) { + os.set_shard(shard_id_t::NO_SHARD); + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + (*failures)[os] = std::pair(when, duration); + if (type == 0) { + ec_inject_write_failures0_shard[os] = o.shard_id; + } + return result; +} + +/** + * @brief Clear a previously configured read error inject. + * @param o Target object for the error inject. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Indication of how many errors were cleared. + */ +std::string ec_inject_clear_read_error(const ghobject_t& o, + const int64_t type) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + int64_t remaining = 0; + switch (type) { + case 0: + failures = &ec_inject_read_failures0; + break; + case 1: + failures = &ec_inject_read_failures1; + break; + default: + return "unrecognized error inject type"; + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + auto it = failures->find(os); + if (it != failures->end()) { + remaining = it->second.second; + failures->erase(it); + } + if (remaining == 0) { + return "no outstanding error injects"; + } else if (remaining == 1) { + return "ok - 1 inject cleared"; + } + return "ok - " + std::to_string(remaining) + " injects cleared"; +} + +/** + * @brief Clear a previously configured write error inject. + * @param o Target object for the error inject. + * @param type Type of error inject 0 = EIO, 1 = missing shard. + * @return string Indication of how many errors were cleared. + */ +std::string ec_inject_clear_write_error(const ghobject_t& o, + const int64_t type) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures; + ghobject_t os = o; + bool no_shard = true; + int64_t remaining = 0; + switch (type) { + case 0: + failures = &ec_inject_write_failures0; + break; + case 1: + failures = &ec_inject_write_failures1; + no_shard = false; + break; + case 2: + failures = &ec_inject_write_failures2; + break; + case 3: + failures = &ec_inject_write_failures3; + break; + default: + return "unrecognized error inject type"; + } + if (no_shard) { + os.set_shard(shard_id_t::NO_SHARD); + } + if (os.hobj.oid.name == "*") { + os.hobj.set_hash(0); + } + auto it = failures->find(os); + if (it != failures->end()) { + remaining = it->second.second; + failures->erase(it); + if (type == 0) { + ec_inject_write_failures0_shard.erase(os); + } + } + if (remaining == 0) { + return "no outstanding error injects"; + } else if (remaining == 1) { + return "ok - 1 inject cleared"; + } + return "ok - " + std::to_string(remaining) + " injects cleared"; +} + +static bool ec_inject_test_error(const ghobject_t& o, + std::map<ghobject_t,std::pair<int64_t,int64_t>> *failures) +{ + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + auto it = failures->find(o); + if (it == failures->end()) { + ghobject_t os = o; + os.hobj.oid.name = "*"; + os.hobj.set_hash(0); + it = failures->find(os); + } + if (it != failures->end()) { + auto && [when,duration] = it->second; + if (when > 0) { + when--; + return false; + } + if (--duration <= 0) { + failures->erase(it); + } + return true; + } + return false; +} + +bool ec_inject_test_read_error0(const ghobject_t& o) +{ + return ec_inject_test_error(o, &ec_inject_read_failures0); +} + +bool ec_inject_test_read_error1(const ghobject_t& o) +{ + return ec_inject_test_error(o, &ec_inject_read_failures1); +} + +bool ec_inject_test_write_error0(const hobject_t& o, + const osd_reqid_t& reqid) { + std::lock_guard<ceph::recursive_mutex> l(ec_inject_lock); + ghobject_t os = ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD); + if (ec_inject_write_failures0_reqid.count(reqid)) { + // Matched reqid of retried write - flag for failure + ec_inject_write_failures0_reqid.erase(reqid); + return true; + } + auto it = ec_inject_write_failures0.find(os); + if (it == ec_inject_write_failures0.end()) { + os.hobj.oid.name = "*"; + os.hobj.set_hash(0); + it = ec_inject_write_failures0.find(os); + } + if (it != ec_inject_write_failures0.end()) { + auto && [when, duration] = it->second; + auto shard = ec_inject_write_failures0_shard.find(os)->second; + if (when > 0) { + when--; + } else { + if (--duration <= 0) { + ec_inject_write_failures0.erase(it); + ec_inject_write_failures0_shard.erase(os); + } + // Error inject triggered - save reqid + ec_inject_write_failures0_reqid.insert(reqid); + // Set up error inject to drop message to primary + ec_inject_write_error(ghobject_t(o, ghobject_t::NO_GEN, shard), 1, 0, 1); + } + } + return false; +} + +bool ec_inject_test_write_error1(const ghobject_t& o) { + bool rc = ec_inject_test_error(o, &ec_inject_write_failures1); + if (rc) { + // Set up error inject to generate OSD down + ec_inject_write_error(o, 2, 0, 1); + } + return rc; +} + +bool ec_inject_test_write_error2(const hobject_t& o) { + return ec_inject_test_error( + ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD), + &ec_inject_write_failures2); +} + +bool ec_inject_test_write_error3(const hobject_t& o) { + return ec_inject_test_error( + ghobject_t(o, ghobject_t::NO_GEN, shard_id_t::NO_SHARD), + &ec_inject_write_failures3); +} diff --git a/src/osd/ECCommon.h b/src/osd/ECCommon.h index 7ff9cae7646..de4c11ad50f 100644 --- a/src/osd/ECCommon.h +++ b/src/osd/ECCommon.h @@ -493,6 +493,7 @@ struct ECCommon { ); ///< @return error code, 0 on success void schedule_recovery_work(); + }; /** @@ -843,3 +844,15 @@ void ECCommon::ReadPipeline::filter_read_op( on_schedule_recovery(op); } } + +// Error inject interfaces +std::string ec_inject_read_error(const ghobject_t& o, const int64_t type, const int64_t when, const int64_t duration); +std::string ec_inject_write_error(const ghobject_t& o, const int64_t type, const int64_t when, const int64_t duration); +std::string ec_inject_clear_read_error(const ghobject_t& o, const int64_t type); +std::string ec_inject_clear_write_error(const ghobject_t& o, const int64_t type); +bool ec_inject_test_read_error0(const ghobject_t& o); +bool ec_inject_test_read_error1(const ghobject_t& o); +bool ec_inject_test_write_error0(const hobject_t& o,const osd_reqid_t& reqid); +bool ec_inject_test_write_error1(const ghobject_t& o); +bool ec_inject_test_write_error2(const hobject_t& o); +bool ec_inject_test_write_error3(const hobject_t& o); diff --git a/src/osd/ExtentCache.h b/src/osd/ExtentCache.h index 972228cd077..7dc1d4f7263 100644 --- a/src/osd/ExtentCache.h +++ b/src/osd/ExtentCache.h @@ -363,7 +363,7 @@ private: extent, boost::intrusive::list_member_hook<>, &extent::pin_list_member>; - using list = boost::intrusive::list<extent, list_member_options>; + using list = boost::intrusive::list<extent, boost::intrusive::constant_time_size<false>, list_member_options>; list pin_list; ~pin_state() { ceph_assert(pin_list.empty()); diff --git a/src/osd/OSD.cc b/src/osd/OSD.cc index 97fefc5e54a..9c9e540cf61 100644 --- a/src/osd/OSD.cc +++ b/src/osd/OSD.cc @@ -37,6 +37,7 @@ #include "osd/PG.h" #include "osd/scrubber/scrub_machine.h" #include "osd/scrubber/pg_scrubber.h" +#include "osd/ECCommon.h" #include "include/types.h" #include "include/compat.h" @@ -4348,6 +4349,46 @@ void OSD::final_init() "inject metadata error to an object"); ceph_assert(r == 0); r = admin_socket->register_command( + "injectecreaderr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false " \ + "name=when,type=CephInt,req=false " \ + "name=duration,type=CephInt,req=false", + test_ops_hook, + "inject error for read of object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecclearreaderr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false", + test_ops_hook, + "clear read error injects for object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecwriteerr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false " \ + "name=when,type=CephInt,req=false " \ + "name=duration,type=CephInt,req=false", + test_ops_hook, + "inject error for write of object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( + "injectecclearwriteerr " \ + "name=pool,type=CephString " \ + "name=objname,type=CephObjectname " \ + "name=shardid,type=CephInt,req=true,range=0|255 " \ + "name=type,type=CephInt,req=false", + test_ops_hook, + "clear write error inject for object in an EC pool"); + ceph_assert(r == 0); + r = admin_socket->register_command( "set_recovery_delay " \ "name=utime,type=CephInt,req=false", test_ops_hook, @@ -6487,8 +6528,10 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, //directly request the osd make a change. if (command == "setomapval" || command == "rmomapkey" || command == "setomapheader" || command == "getomap" || - command == "truncobj" || command == "injectmdataerr" || - command == "injectdataerr" + command == "truncobj" || + command == "injectmdataerr" || command == "injectdataerr" || + command == "injectecreaderr" || command == "injectecclearreaderr" || + command == "injectecwriteerr" || command == "injectecclearwriteerr" ) { pg_t rawpg; int64_t pool; @@ -6527,8 +6570,21 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, ghobject_t gobj(obj, ghobject_t::NO_GEN, shard_id_t(uint8_t(shardid))); spg_t pgid(curmap->raw_pg_to_pg(rawpg), shard_id_t(shardid)); if (curmap->pg_is_ec(rawpg)) { - if ((command != "injectdataerr") && (command != "injectmdataerr")) { - ss << "Must not call on ec pool, except injectdataerr or injectmdataerr"; + if ((command != "injectdataerr") && + (command != "injectmdataerr") && + (command != "injectecreaderr") && + (command != "injectecclearreaderr") && + (command != "injectecwriteerr") && + (command != "injectecclearwriteerr")) { + ss << "Must not call on ec pool"; + return; + } + } else { + if ((command == "injectecreaderr") || + (command == "injecteclearreaderr") || + (command == "injectecwriteerr") || + (command == "injecteclearwriteerr")) { + ss << "Only supported on ec pool"; return; } } @@ -6607,6 +6663,38 @@ void TestOpsSocketHook::test_ops(OSDService *service, ObjectStore *store, } else if (command == "injectmdataerr") { store->inject_mdata_error(gobj); ss << "ok"; + } else if (command == "injectecreaderr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + int64_t when = cmd_getval_or<int64_t>(cmdmap, "when", 0); + int64_t duration = cmd_getval_or<int64_t>(cmdmap, "duration", 1); + ss << ec_inject_read_error(gobj, type, when, duration); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecclearreaderr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + ss << ec_inject_clear_read_error(gobj, type); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecwriteerr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + int64_t when = cmd_getval_or<int64_t>(cmdmap, "when", 0); + int64_t duration = cmd_getval_or<int64_t>(cmdmap, "duration", 1); + ss << ec_inject_write_error(gobj, type, when, duration); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } + } else if (command == "injectecclearwriteerr") { + if (service->cct->_conf->bluestore_debug_inject_read_err) { + int64_t type = cmd_getval_or<int64_t>(cmdmap, "type", 0); + ss << ec_inject_clear_write_error(gobj, type); + } else { + ss << "bluestore_debug_inject_read_err not enabled"; + } } return; } @@ -9958,7 +10046,8 @@ const char** OSD::get_tracked_conf_keys() const "osd_scrub_max_interval", "osd_op_thread_timeout", "osd_op_thread_suicide_timeout", - NULL + "osd_max_scrubs", + nullptr }; return KEYS; } @@ -10002,6 +10091,10 @@ void OSD::handle_conf_change(const ConfigProxy& conf, service.snap_reserver.set_max(cct->_conf->osd_max_trimming_pgs); } if (changed.count("osd_max_scrubs")) { + dout(0) << fmt::format( + "{}: scrub concurrency max changed to {}", + __func__, cct->_conf->osd_max_scrubs) + << dendl; service.scrub_reserver.set_max(cct->_conf->osd_max_scrubs); } if (changed.count("osd_op_complaint_time") || diff --git a/src/osd/OSDMap.cc b/src/osd/OSDMap.cc index b87484c1a9d..9b3593d54e5 100644 --- a/src/osd/OSDMap.cc +++ b/src/osd/OSDMap.cc @@ -1642,12 +1642,10 @@ void OSDMap::get_out_of_subnet_osd_counts(CephContext *cct, for (int i = 0; i < max_osd; i++) { if (exists(i) && is_up(i)) { if (const auto& addrs = get_addrs(i).v; addrs.size() >= 2) { - auto v1_addr = addrs[0].ip_only_to_str(); - if (!is_addr_in_subnet(cct, public_network, v1_addr)) { + if (!is_addr_in_subnet(cct, public_network, addrs[0])) { unreachable->emplace(i); } - auto v2_addr = addrs[1].ip_only_to_str(); - if (!is_addr_in_subnet(cct, public_network, v2_addr)) { + if (!is_addr_in_subnet(cct, public_network, addrs[1])) { unreachable->emplace(i); } } diff --git a/src/osd/PG.cc b/src/osd/PG.cc index 307651fd627..cd7cad777bc 100644 --- a/src/osd/PG.cc +++ b/src/osd/PG.cc @@ -1278,7 +1278,6 @@ Scrub::schedule_result_t PG::start_scrubbing( pg_cond.allow_deep = !(get_osdmap()->test_flag(CEPH_OSDMAP_NODEEP_SCRUB) || pool.info.has_flag(pg_pool_t::FLAG_NODEEP_SCRUB)); - pg_cond.has_deep_errors = (info.stats.stats.sum.num_deep_scrub_errors > 0); pg_cond.can_autorepair = (cct->_conf->osd_scrub_auto_repair && get_pgbackend()->auto_repair_supported()); diff --git a/src/osd/PGBackend.h b/src/osd/PGBackend.h index b87aa1da677..f5eb9ea951e 100644 --- a/src/osd/PGBackend.h +++ b/src/osd/PGBackend.h @@ -290,6 +290,10 @@ typedef std::shared_ptr<const OSDMap> OSDMapRef; MessageRef, Connection *con) = 0; virtual void send_message_osd_cluster( Message *m, const ConnectionRef& con) = 0; + virtual void start_mon_command( + const std::vector<std::string>& cmd, const bufferlist& inbl, + bufferlist *outbl, std::string *outs, + Context *onfinish) = 0; virtual ConnectionRef get_con_osd_cluster(int peer, epoch_t from_epoch) = 0; virtual entity_name_t get_cluster_msgr_name() = 0; diff --git a/src/osd/PrimaryLogPG.cc b/src/osd/PrimaryLogPG.cc index 44f8e85b5ef..3324ba9dc91 100644 --- a/src/osd/PrimaryLogPG.cc +++ b/src/osd/PrimaryLogPG.cc @@ -2286,6 +2286,16 @@ void PrimaryLogPG::do_op(OpRequestRef& op) } } + if (cct->_conf->bluestore_debug_inject_read_err && + op->may_write() && + pool.info.is_erasure() && + ec_inject_test_write_error0(m->get_hobj(), m->get_reqid())) { + // Fail retried write with error + dout(0) << __func__ << " Error inject - Fail retried write with EINVAL" << dendl; + osd->reply_op_error(op, -EINVAL); + return; + } + ObjectContextRef obc; bool can_create = op->may_write(); hobject_t missing_oid; @@ -5798,10 +5808,19 @@ int PrimaryLogPG::do_extent_cmp(OpContext *ctx, OSDOp& osd_op) int PrimaryLogPG::finish_extent_cmp(OSDOp& osd_op, const bufferlist &read_bl) { - for (uint64_t idx = 0; idx < osd_op.indata.length(); ++idx) { - char read_byte = (idx < read_bl.length() ? read_bl[idx] : 0); - if (osd_op.indata[idx] != read_byte) { - return (-MAX_ERRNO - idx); + auto input_iter = osd_op.indata.begin(); + auto read_iter = read_bl.begin(); + uint64_t idx = 0; + + while (input_iter != osd_op.indata.end()) { + char read_byte = (read_iter != read_bl.end() ? *read_iter : 0); + if (*input_iter != read_byte) { + return (-MAX_ERRNO - idx); + } + ++idx; + ++input_iter; + if (read_iter != read_bl.end()) { + ++read_iter; } } @@ -7767,27 +7786,34 @@ int PrimaryLogPG::do_osd_ops(OpContext *ctx, vector<OSDOp>& ops) bool truncated = false; bufferlist bl; if (oi.is_omap()) { - ObjectMap::ObjectMapIterator iter = osd->store->get_omap_iterator( - ch, ghobject_t(soid) - ); - if (!iter) { - result = -ENOENT; - goto fail; - } - iter->upper_bound(start_after); - if (filter_prefix > start_after) iter->lower_bound(filter_prefix); - for (num = 0; - iter->valid() && - iter->key().substr(0, filter_prefix.size()) == filter_prefix; - ++num, iter->next()) { - dout(20) << "Found key " << iter->key() << dendl; - if (num >= max_return || - bl.length() >= cct->_conf->osd_max_omap_bytes_per_request) { - truncated = true; - break; - } - encode(iter->key(), bl); - encode(iter->value(), bl); + using omap_iter_seek_t = ObjectStore::omap_iter_seek_t; + result = osd->store->omap_iterate( + ch, ghobject_t(soid), + // try to seek as many keys-at-once as possible for the sake of performance. + // note complexity should be logarithmic, so seek(n/2) + seek(n/2) is worse + // than just seek(n). + ObjectStore::omap_iter_seek_t{ + .seek_position = std::max(start_after, filter_prefix), + .seek_type = filter_prefix > start_after ? omap_iter_seek_t::LOWER_BOUND + : omap_iter_seek_t::UPPER_BOUND + }, + [&bl, &truncated, &filter_prefix, &num, max_return, + max_bytes=cct->_conf->osd_max_omap_bytes_per_request] + (std::string_view key, std::string_view value) mutable { + if (key.substr(0, filter_prefix.size()) != filter_prefix) { + return ObjectStore::omap_iter_ret_t::STOP; + } + if (num >= max_return || bl.length() >= max_bytes) { + truncated = true; + return ObjectStore::omap_iter_ret_t::STOP; + } + encode(key, bl); + encode(value, bl); + ++num; + return ObjectStore::omap_iter_ret_t::NEXT; + }); + if (result < 0) { + goto fail; } } // else return empty out_set encode(num, osd_op.outdata); diff --git a/src/osd/PrimaryLogPG.h b/src/osd/PrimaryLogPG.h index f66b5c6e16a..bf55d539821 100644 --- a/src/osd/PrimaryLogPG.h +++ b/src/osd/PrimaryLogPG.h @@ -622,6 +622,12 @@ public: Message *m, const ConnectionRef& con) override { osd->send_message_osd_cluster(m, con); } + void start_mon_command( + const std::vector<std::string>& cmd, const bufferlist& inbl, + bufferlist *outbl, std::string *outs, + Context *onfinish) override { + osd->monc->start_mon_command(cmd, inbl, outbl, outs, onfinish); + } ConnectionRef get_con_osd_cluster(int peer, epoch_t from_epoch) override; entity_name_t get_cluster_msgr_name() override { return osd->get_cluster_msgr_name(); @@ -1993,6 +1999,7 @@ public: private: DynamicPerfStats m_dynamic_perf_stats; + }; inline ostream& operator<<(ostream& out, const PrimaryLogPG::RepGather& repop) @@ -2021,5 +2028,4 @@ inline ostream& operator<<(ostream& out, void intrusive_ptr_add_ref(PrimaryLogPG::RepGather *repop); void intrusive_ptr_release(PrimaryLogPG::RepGather *repop); - #endif diff --git a/src/osd/Session.h b/src/osd/Session.h index 9fa9c655456..05a0119d31e 100644 --- a/src/osd/Session.h +++ b/src/osd/Session.h @@ -136,7 +136,7 @@ struct Session : public RefCountedObject { ceph::mutex session_dispatch_lock = ceph::make_mutex("Session::session_dispatch_lock"); - boost::intrusive::list<OpRequest> waiting_on_map; + boost::intrusive::list<OpRequest, boost::intrusive::constant_time_size<false>> waiting_on_map; ceph::spinlock projected_epoch_lock; epoch_t projected_epoch = 0; diff --git a/src/osd/osd_types.cc b/src/osd/osd_types.cc index 5c2cf8b16b0..048f5aa0009 100644 --- a/src/osd/osd_types.cc +++ b/src/osd/osd_types.cc @@ -2942,6 +2942,14 @@ std::string pg_stat_t::dump_scrub_schedule() const return fmt::format( "Blocked! locked objects (for {}s)", scrub_sched_status.m_duration_seconds); + } else if (scrub_sched_status.m_num_to_reserve != 0) { + // we are waiting for some replicas to respond + return fmt::format( + "Reserving. Waiting {}s for OSD.{} ({}/{})", + scrub_sched_status.m_duration_seconds, + scrub_sched_status.m_osd_to_respond, + scrub_sched_status.m_ordinal_of_requested_replica, + scrub_sched_status.m_num_to_reserve); } else { return fmt::format( "{}scrubbing for {}s", @@ -2964,7 +2972,7 @@ std::string pg_stat_t::dump_scrub_schedule() const case pg_scrub_sched_status_t::queued: return fmt::format( "queued for {}scrub", - ((scrub_sched_status.m_is_deep == scrub_level_t::deep) ? "deep " : "")); + (scrub_sched_status.m_is_deep == scrub_level_t::deep) ? "deep " : ""); default: // a bug! return "SCRUB STATE MISMATCH!"s; @@ -2979,12 +2987,15 @@ bool operator==(const pg_scrubbing_status_t& l, const pg_scrubbing_status_t& r) l.m_duration_seconds == r.m_duration_seconds && l.m_is_active == r.m_is_active && l.m_is_deep == r.m_is_deep && - l.m_is_periodic == r.m_is_periodic; + l.m_is_periodic == r.m_is_periodic && + l.m_osd_to_respond == r.m_osd_to_respond && + l.m_ordinal_of_requested_replica == r.m_ordinal_of_requested_replica && + l.m_num_to_reserve == r.m_num_to_reserve; } void pg_stat_t::encode(ceph::buffer::list &bl) const { - ENCODE_START(29, 22, bl); + ENCODE_START(30, 22, bl); encode(version, bl); encode(reported_seq, bl); encode(reported_epoch, bl); @@ -3044,6 +3055,9 @@ void pg_stat_t::encode(ceph::buffer::list &bl) const encode(objects_trimmed, bl); encode(snaptrim_duration, bl); encode(log_dups_size, bl); + encode(scrub_sched_status.m_osd_to_respond, bl); + encode(scrub_sched_status.m_ordinal_of_requested_replica, bl); + encode(scrub_sched_status.m_num_to_reserve, bl); ENCODE_FINISH(bl); } @@ -3052,7 +3066,7 @@ void pg_stat_t::decode(ceph::buffer::list::const_iterator &bl) { bool tmp; uint32_t old_state; - DECODE_START(29, bl); + DECODE_START(30, bl); decode(version, bl); decode(reported_seq, bl); decode(reported_epoch, bl); @@ -3142,6 +3156,18 @@ void pg_stat_t::decode(ceph::buffer::list::const_iterator &bl) if (struct_v >= 29) { decode(log_dups_size, bl); } + if (struct_v >= 30) { + uint16_t osd_to_respond; + decode(osd_to_respond, bl); + scrub_sched_status.m_osd_to_respond = osd_to_respond; + uint8_t tmp8; + decode(tmp8, bl); + scrub_sched_status.m_ordinal_of_requested_replica = tmp8; + decode(tmp8, bl); + scrub_sched_status.m_num_to_reserve = tmp8; + } else { + scrub_sched_status.m_num_to_reserve = 0; + } } DECODE_FINISH(bl); } diff --git a/src/osd/osd_types.h b/src/osd/osd_types.h index b6f5335a0f5..485fddead7a 100644 --- a/src/osd/osd_types.h +++ b/src/osd/osd_types.h @@ -1151,9 +1151,8 @@ public: bool is_set(key_t key) const; template<typename T> - void set(key_t key, const T &val) { - value_t value = val; - opts[key] = value; + void set(key_t key, T &&val) { + opts.insert_or_assign(key, std::forward<T>(val)); } template<typename T> @@ -2223,6 +2222,13 @@ struct pg_scrubbing_status_t { bool m_is_active{false}; scrub_level_t m_is_deep{scrub_level_t::shallow}; bool m_is_periodic{true}; + // the following are only relevant when we are reserving replicas: + uint16_t m_osd_to_respond{0}; + /// this is the n'th replica we are reserving (out of m_num_to_reserve) + uint8_t m_ordinal_of_requested_replica{0}; + /// the number of replicas we are reserving for scrubbing. 0 means we are not + /// in the process of reserving replicas. + uint8_t m_num_to_reserve{0}; }; bool operator==(const pg_scrubbing_status_t& l, const pg_scrubbing_status_t& r); diff --git a/src/osd/scrubber/pg_scrubber.cc b/src/osd/scrubber/pg_scrubber.cc index aa53df5ae8a..ba83f6ac600 100644 --- a/src/osd/scrubber/pg_scrubber.cc +++ b/src/osd/scrubber/pg_scrubber.cc @@ -588,6 +588,10 @@ scrub_level_t PgScrubber::scrub_requested( return scrub_level_t::shallow; } + // abort an ongoing scrub, if it's of the lowest priority + // and stuck in replica reservations. + m_fsm->process_event(AbortIfReserving{}); + // update the relevant SchedTarget (either shallow or deep). Set its urgency // to either operator_requested or must_repair. Push it into the queue auto& trgt = m_scrub_job->get_target(scrub_level); @@ -2073,6 +2077,7 @@ void PgScrubber::scrub_finish() } cleanup_on_finish(); + m_active_target.reset(); if (do_auto_scrub) { request_rescrubbing(); } @@ -2299,26 +2304,6 @@ Scrub::schedule_result_t PgScrubber::start_scrub_session( } } - // restricting shallow scrubs of PGs that have deep errors: - if (pg_cond.has_deep_errors && trgt.is_shallow()) { - if (trgt.urgency() < urgency_t::operator_requested) { - // if there are deep errors, we should have scheduled a deep scrub first. - // If we are here trying to perform a shallow scrub, it means that for some - // reason that deep scrub failed to be initiated. We will not try a shallow - // scrub until this is solved. - dout(10) << __func__ << ": Regular scrub skipped due to deep-scrub errors" - << dendl; - requeue_penalized( - s_or_d, delay_both_targets_t::no, delay_cause_t::pg_state, clock_now); - return schedule_result_t::target_specific_failure; - } else { - // we will honor the request anyway, but will report the issue - m_osds->clog->error() << fmt::format( - "osd.{} pg {} Regular scrub request, deep-scrub details will be lost", - m_osds->whoami, m_pg_id); - } - } - // if only explicitly requested repairing is allowed - skip other types // of scrubbing if (osd_restrictions.allow_requested_repair_only && @@ -2433,6 +2418,16 @@ void PgScrubber::dump_active_scrubber(ceph::Formatter* f) const } else { f->dump_string("schedule", "scrubbing"); } + const auto maybe_register = m_fsm->get_reservation_status(); + if (maybe_register && maybe_register->m_num_to_reserve != 0) { + f->dump_bool("is_reserving_replicas", true); + f->dump_int("osd_to_respond", maybe_register->m_osd_to_respond); + f->dump_int("duration_seconds", maybe_register->m_duration_seconds); + f->dump_int("requested_in_order", maybe_register->m_ordinal_of_requested_replica); + f->dump_int("num_to_reserve", maybe_register->m_num_to_reserve); + } else { + f->dump_bool("is_reserving_replicas", false); + } } pg_scrubbing_status_t PgScrubber::get_schedule() const @@ -2461,7 +2456,7 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const pg_scrub_sched_status_t::blocked, true, // active (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), - false}; + (m_active_target->urgency() == urgency_t::periodic_regular)}; } else { int32_t dur_seconds = @@ -2472,9 +2467,11 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const pg_scrub_sched_status_t::active, true, // active (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), - false /* is periodic? unknown, actually */}; + (m_active_target->urgency() == urgency_t::periodic_regular)}; } } + + // not registered to be scrubbed? if (!m_scrub_job->is_registered()) { return pg_scrubbing_status_t{ utime_t{}, @@ -2485,8 +2482,34 @@ pg_scrubbing_status_t PgScrubber::get_schedule() const false}; } - // not taking 'no-*scrub' flags into account here. + // in session, but still reserving replicas? + const auto maybe_register = m_fsm->get_reservation_status(); + if (maybe_register) { + // note that if we are here, we are scrubbing (even though + // m_active is false). The 'maybe_register' attests to being in + // ReservingReplicas state, and m_active wasn't set yet. + dout(20) << fmt::format( + "{}:maybe_register: osd:{} {}s ({} of {})", __func__, + maybe_register->m_osd_to_respond, + maybe_register->m_duration_seconds, + maybe_register->m_ordinal_of_requested_replica, + maybe_register->m_num_to_reserve) + << dendl; + return pg_scrubbing_status_t{ + utime_t{}, + maybe_register->m_duration_seconds, + pg_scrub_sched_status_t::active, + true, // active + (m_is_deep ? scrub_level_t::deep : scrub_level_t::shallow), + (m_active_target->urgency() == urgency_t::periodic_regular), + maybe_register->m_osd_to_respond, + maybe_register->m_ordinal_of_requested_replica, + maybe_register->m_num_to_reserve}; + } + const auto first_ready = m_scrub_job->earliest_eligible(now_is); + // eligible for scrubbing, but not yet selected to be scrubbed? + // (not taking 'no-*scrub' flags into account here.) if (first_ready) { const auto& targ = first_ready->get(); return pg_scrubbing_status_t{ diff --git a/src/osd/scrubber/scrub_job.cc b/src/osd/scrubber/scrub_job.cc index 9e0d03b6ea4..a02ebe8fd44 100644 --- a/src/osd/scrubber/scrub_job.cc +++ b/src/osd/scrubber/scrub_job.cc @@ -14,17 +14,6 @@ using ScrubJob = Scrub::ScrubJob; using delay_ready_t = Scrub::delay_ready_t; using namespace std::chrono; -namespace { -utime_t add_double(utime_t t, double d) -{ - double int_part; - double frac_as_ns = 1'000'000'000 * std::modf(d, &int_part); - return utime_t{ - t.sec() + static_cast<int>(int_part), - static_cast<int>(t.nsec() + frac_as_ns)}; -} -} // namespace - using SchedEntry = Scrub::SchedEntry; // ////////////////////////////////////////////////////////////////////////// // diff --git a/src/osd/scrubber/scrub_machine.cc b/src/osd/scrubber/scrub_machine.cc index da9466758f4..10866ce580a 100644 --- a/src/osd/scrubber/scrub_machine.cc +++ b/src/osd/scrubber/scrub_machine.cc @@ -106,6 +106,25 @@ ceph::timespan ScrubMachine::get_time_scrubbing() const return ceph::timespan{}; } +std::optional<pg_scrubbing_status_t> ScrubMachine::get_reservation_status() + const +{ + const auto resv_state = state_cast<const ReservingReplicas*>(); + if (!resv_state) { + return std::nullopt; + } + const auto session = state_cast<const Session*>(); + dout(30) << fmt::format( + "{}: we are reserving {:p}-{:p}", __func__, (void*)session, + (void*)resv_state) + << dendl; + if (!session || !session->m_reservations) { + dout(20) << fmt::format("{}: no reservations data", __func__) << dendl; + return std::nullopt; + } + return session->get_reservation_status(); +} + // ////////////// the actual actions // ----------------------- NotActive ----------------------------------------- @@ -203,6 +222,23 @@ sc::result Session::react(const IntervalChanged&) return transit<NotActive>(); } +std::optional<pg_scrubbing_status_t> Session::get_reservation_status() const +{ + if (!m_reservations) { + return std::nullopt; + } + DECLARE_LOCALS; // 'scrbr' & 'pg_id' aliases + const auto req = m_reservations->get_last_sent(); + pg_scrubbing_status_t s; + s.m_osd_to_respond = req ? req->osd : 0; + s.m_ordinal_of_requested_replica = m_reservations->active_requests_cnt(); + s.m_num_to_reserve = scrbr->get_pg()->get_actingset().size() - 1; + s.m_duration_seconds = + duration_cast<seconds>(context<ScrubMachine>().get_time_scrubbing()) + .count(); + return s; +} + // ----------------------- ReservingReplicas --------------------------------- diff --git a/src/osd/scrubber/scrub_machine.h b/src/osd/scrubber/scrub_machine.h index ad0d3bfba38..f7f739692bf 100644 --- a/src/osd/scrubber/scrub_machine.h +++ b/src/osd/scrubber/scrub_machine.h @@ -2,6 +2,7 @@ // vim: ts=8 sw=2 smarttab #pragma once +#include <optional> #include <string> #include <boost/statechart/custom_reaction.hpp> @@ -160,6 +161,11 @@ VALUE_EVENT(ReserverGranted, AsyncScrubResData); /// all replicas have granted our reserve request MEV(RemotesReserved) +/// abort the scrub session, if in ReservingReplicas state +/// (used when the operator issues a scrub request, and we no longer +/// need the reservations) +MEV(AbortIfReserving) + /// initiate a new scrubbing session (relevant if we are a Primary) MEV(StartScrub) @@ -289,9 +295,12 @@ class ScrubMachine : public sc::state_machine<ScrubMachine, NotActive> { [[nodiscard]] bool is_accepting_updates() const; [[nodiscard]] bool is_primary_idle() const; - // elapsed time for the currently active scrub.session + /// elapsed time for the currently active scrub.session ceph::timespan get_time_scrubbing() const; + /// replica reservation process status + std::optional<pg_scrubbing_status_t> get_reservation_status() const; + // ///////////////// aux declarations & functions //////////////////////// // @@ -555,6 +564,9 @@ struct Session : sc::state<Session, PrimaryActive, ReservingReplicas>, /// abort reason - if known. Determines the delay time imposed on the /// failed scrub target. std::optional<Scrub::delay_cause_t> m_abort_reason{std::nullopt}; + + /// when reserving replicas: fetch the reservation status + std::optional<pg_scrubbing_status_t> get_reservation_status() const; }; struct ReservingReplicas : sc::state<ReservingReplicas, Session>, NamedSimply { @@ -563,6 +575,7 @@ struct ReservingReplicas : sc::state<ReservingReplicas, Session>, NamedSimply { using reactions = mpl::list< sc::custom_reaction<ReplicaGrant>, sc::custom_reaction<ReplicaReject>, + sc::transition<AbortIfReserving, PrimaryIdle>, sc::transition<RemotesReserved, ActiveScrubbing>>; ScrubTimePoint entered_at = ScrubClock::now(); diff --git a/src/osd/scrubber/scrub_reservations.h b/src/osd/scrubber/scrub_reservations.h index 173b23d7db5..f5eca48b888 100644 --- a/src/osd/scrubber/scrub_reservations.h +++ b/src/osd/scrubber/scrub_reservations.h @@ -157,13 +157,13 @@ class ReplicaReservations { // note: 'public', as accessed via the 'standard' dout_prefix() macro std::ostream& gen_prefix(std::ostream& out, std::string fn) const; + /// The number of requests that have been sent (and not rejected) so far. + size_t active_requests_cnt() const; + private: /// send 'release' messages to all replicas we have managed to reserve void release_all(); - /// The number of requests that have been sent (and not rejected) so far. - size_t active_requests_cnt() const; - /** * Send a reservation request to the next replica. * - if there are no more replicas to send requests to, return true diff --git a/src/osd/scrubber_common.h b/src/osd/scrubber_common.h index 809107e593b..2ab5570a715 100644 --- a/src/osd/scrubber_common.h +++ b/src/osd/scrubber_common.h @@ -109,7 +109,6 @@ static_assert(sizeof(Scrub::OSDRestrictions) <= sizeof(uint32_t)); struct ScrubPGPreconds { bool allow_shallow{true}; bool allow_deep{true}; - bool has_deep_errors{false}; bool can_autorepair{false}; }; static_assert(sizeof(Scrub::ScrubPGPreconds) <= sizeof(uint32_t)); @@ -181,9 +180,8 @@ struct formatter<Scrub::ScrubPGPreconds> { auto format(const Scrub::ScrubPGPreconds& conds, FormatContext& ctx) const { return fmt::format_to( - ctx.out(), "allowed(shallow/deep):{:1}/{:1},deep-err:{:1},can-autorepair:{:1}", - conds.allow_shallow, conds.allow_deep, conds.has_deep_errors, - conds.can_autorepair); + ctx.out(), "allowed(shallow/deep):{:1}/{:1},can-autorepair:{:1}", + conds.allow_shallow, conds.allow_deep, conds.can_autorepair); } }; diff --git a/src/osdc/CMakeLists.txt b/src/osdc/CMakeLists.txt index 205ad3d4f42..637ce327555 100644 --- a/src/osdc/CMakeLists.txt +++ b/src/osdc/CMakeLists.txt @@ -1,9 +1,8 @@ set(osdc_files Filer.cc ObjectCacher.cc - Objecter.cc - error_code.cc - Striper.cc) + error_code.cc) +# Objecter.cc and Striper.cc are part of libcommon add_library(osdc STATIC ${osdc_files}) target_link_libraries(osdc ceph-common) if(WITH_EVENTTRACE) diff --git a/src/osdc/Objecter.cc b/src/osdc/Objecter.cc index 087b623333b..82d43bb3dde 100644 --- a/src/osdc/Objecter.cc +++ b/src/osdc/Objecter.cc @@ -1393,7 +1393,7 @@ void Objecter::handle_osd_map(MOSDMap *m) for (auto& [c, ec] : p->second) { asio::post(service.get_executor(), asio::append(std::move(c), ec)); } - waiting_for_map.erase(p++); + p = waiting_for_map.erase(p); } monc->sub_got("osdmap", osdmap->get_epoch()); diff --git a/src/pybind/mgr/cephadm/inventory.py b/src/pybind/mgr/cephadm/inventory.py index d3681cbe60a..550604fc55b 100644 --- a/src/pybind/mgr/cephadm/inventory.py +++ b/src/pybind/mgr/cephadm/inventory.py @@ -438,6 +438,7 @@ class SpecStore(): for key_attr in [ 'server_key', 'client_key', + 'encryption_key', ]: key = getattr(nvmeof_spec, key_attr, None) if key: @@ -490,6 +491,7 @@ class SpecStore(): self.mgr.cert_key_store.rm_cert('nvmeof_root_ca_cert', service_name=spec.service_name()) self.mgr.cert_key_store.rm_key('nvmeof_server_key', service_name=spec.service_name()) self.mgr.cert_key_store.rm_key('nvmeof_client_key', service_name=spec.service_name()) + self.mgr.cert_key_store.rm_key('nvmeof_encryption_key', service_name=spec.service_name()) def get_created(self, spec: ServiceSpec) -> Optional[datetime.datetime]: return self.spec_created.get(spec.service_name()) @@ -1969,6 +1971,7 @@ class CertKeyStore(): 'ingress_ssl_key', 'nvmeof_server_key', 'nvmeof_client_key', + 'nvmeof_encryption_key', ] known_certs: Dict[str, Any] = {} @@ -2005,6 +2008,7 @@ class CertKeyStore(): 'ingress_ssl_key': {}, # service-name -> key 'nvmeof_server_key': {}, # service-name -> key 'nvmeof_client_key': {}, # service-name -> key + 'nvmeof_encryption_key': {}, # service-name -> key } def get_cert(self, entity: str, service_name: str = '', host: str = '') -> str: @@ -2032,8 +2036,8 @@ class CertKeyStore(): var = service_name if entity in self.service_name_cert else host j = {} self.known_certs[entity][var] = cert_obj - for service_name in self.known_certs[entity].keys(): - j[var] = Cert.to_json(self.known_certs[entity][var]) + for cert_key in self.known_certs[entity]: + j[cert_key] = Cert.to_json(self.known_certs[entity][cert_key]) else: self.known_certs[entity] = cert_obj j = Cert.to_json(cert_obj) diff --git a/src/pybind/mgr/cephadm/module.py b/src/pybind/mgr/cephadm/module.py index e851f1ee3fc..6690153d435 100644 --- a/src/pybind/mgr/cephadm/module.py +++ b/src/pybind/mgr/cephadm/module.py @@ -30,7 +30,7 @@ import multiprocessing.pool import subprocess from prettytable import PrettyTable -import ceph.cephadm.images as default_images +from ceph.cephadm.images import DefaultImages from ceph.deployment import inventory from ceph.deployment.drive_group import DriveGroupSpec from ceph.deployment.service_spec import \ @@ -218,96 +218,6 @@ class CephadmOrchestrator(orchestrator.Orchestrator, MgrModule, runtime=True, ), Option( - 'container_image_prometheus', - default=default_images.DEFAULT_PROMETHEUS_IMAGE, - desc='Prometheus container image', - ), - Option( - 'container_image_nvmeof', - default=default_images.DEFAULT_NVMEOF_IMAGE, - desc='Nvme-of container image', - ), - Option( - 'container_image_grafana', - default=default_images.DEFAULT_GRAFANA_IMAGE, - desc='Prometheus container image', - ), - Option( - 'container_image_alertmanager', - default=default_images.DEFAULT_ALERTMANAGER_IMAGE, - desc='Prometheus container image', - ), - Option( - 'container_image_node_exporter', - default=default_images.DEFAULT_NODE_EXPORTER_IMAGE, - desc='Prometheus container image', - ), - Option( - 'container_image_loki', - default=default_images.DEFAULT_LOKI_IMAGE, - desc='Loki container image', - ), - Option( - 'container_image_promtail', - default=default_images.DEFAULT_PROMTAIL_IMAGE, - desc='Promtail container image', - ), - Option( - 'container_image_haproxy', - default=default_images.DEFAULT_HAPROXY_IMAGE, - desc='HAproxy container image', - ), - Option( - 'container_image_keepalived', - default=default_images.DEFAULT_KEEPALIVED_IMAGE, - desc='Keepalived container image', - ), - Option( - 'container_image_snmp_gateway', - default=default_images.DEFAULT_SNMP_GATEWAY_IMAGE, - desc='SNMP Gateway container image', - ), - Option( - 'container_image_nginx', - default=default_images.DEFAULT_NGINX_IMAGE, - desc='Nginx container image', - ), - Option( - 'container_image_oauth2_proxy', - default=default_images.DEFAULT_OAUTH2_PROXY_IMAGE, - desc='oauth2-proxy container image', - ), - Option( - 'container_image_elasticsearch', - default=default_images.DEFAULT_ELASTICSEARCH_IMAGE, - desc='elasticsearch container image', - ), - Option( - 'container_image_jaeger_agent', - default=default_images.DEFAULT_JAEGER_AGENT_IMAGE, - desc='Jaeger agent container image', - ), - Option( - 'container_image_jaeger_collector', - default=default_images.DEFAULT_JAEGER_COLLECTOR_IMAGE, - desc='Jaeger collector container image', - ), - Option( - 'container_image_jaeger_query', - default=default_images.DEFAULT_JAEGER_QUERY_IMAGE, - desc='Jaeger query container image', - ), - Option( - 'container_image_samba', - default=default_images.DEFAULT_SAMBA_IMAGE, - desc='Samba/SMB container image', - ), - Option( - 'container_image_samba_metrics', - default=default_images.DEFAULT_SAMBA_METRICS_IMAGE, - desc='Samba/SMB metrics exporter container image', - ), - Option( 'warn_on_stray_hosts', type='bool', default=True, @@ -543,6 +453,8 @@ class CephadmOrchestrator(orchestrator.Orchestrator, MgrModule, desc="Default address for RedFish API (oob management)." ), ] + for image in DefaultImages: + MODULE_OPTIONS.append(Option(image.key, default=image.image_ref, desc=image.desc)) def __init__(self, *args: Any, **kwargs: Any): super(CephadmOrchestrator, self).__init__(*args, **kwargs) @@ -2548,7 +2460,7 @@ Then run the following: @handle_orch_error def service_action(self, action: str, service_name: str) -> List[str]: - if service_name not in self.spec_store.all_specs.keys(): + if service_name not in self.spec_store.all_specs.keys() and service_name != 'osd': raise OrchestratorError(f'Invalid service name "{service_name}".' + ' View currently running services using "ceph orch ls"') dds: List[DaemonDescription] = self.cache.get_daemons_by_service(service_name) @@ -4013,6 +3925,50 @@ Then run the following: return self.to_remove_osds.all_osds() @handle_orch_error + def set_osd_spec(self, service_name: str, osd_ids: List[str]) -> str: + """ + Update unit.meta file for osd with service name + """ + if service_name not in self.spec_store: + raise OrchestratorError(f"Cannot find service '{service_name}' in the inventory. " + "Please try again after applying an OSD service that matches " + "the service name to which you want to attach OSDs.") + + daemons: List[orchestrator.DaemonDescription] = self.cache.get_daemons_by_type('osd') + update_osd = defaultdict(list) + for daemon in daemons: + if daemon.daemon_id in osd_ids and daemon.hostname: + update_osd[daemon.hostname].append(daemon.daemon_id) + + if not update_osd: + raise OrchestratorError(f"Unable to find OSDs: {osd_ids}") + + failed_osds = [] + success_osds = [] + for host in update_osd: + osds = ",".join(update_osd[host]) + # run cephadm command with all host osds on specific host, + # if it fails, continue with other hosts + try: + with self.async_timeout_handler(host): + outs, errs, _code = self.wait_async( + CephadmServe(self)._run_cephadm(host, + cephadmNoImage, + 'update-osd-service', + ['--service-name', service_name, '--osd-ids', osds])) + if _code: + self.log.error(f"Failed to update service for {osds} osd. Cephadm error: {errs}") + failed_osds.extend(update_osd[host]) + else: + success_osds.extend(update_osd[host]) + except Exception: + self.log.exception(f"Failed to set service name for {osds}") + failed_osds.extend(update_osd[host]) + self.cache.invalidate_host_daemons(host) + self._kick_serve_loop() + return f"Updated service for osd {','.join(success_osds)}" + (f" and failed for {','.join(failed_osds)}" if failed_osds else "") + + @handle_orch_error @host_exists() def drain_host(self, hostname: str, force: bool = False, keep_conf_keyring: bool = False, zap_osd_devices: bool = False) -> str: """ diff --git a/src/pybind/mgr/cephadm/schedule.py b/src/pybind/mgr/cephadm/schedule.py index 98d2fe99897..04d3712c50a 100644 --- a/src/pybind/mgr/cephadm/schedule.py +++ b/src/pybind/mgr/cephadm/schedule.py @@ -385,6 +385,8 @@ class HostAssignment(object): def find_ip_on_host(self, hostname: str, subnets: List[str]) -> Optional[str]: for subnet in subnets: + # to normalize subnet + subnet = str(ipaddress.ip_network(subnet)) ips: List[str] = [] # following is to allow loopback interfaces for both ipv4 and ipv6. Since we # only have the subnet (and no IP) we assume default loopback IP address. diff --git a/src/pybind/mgr/cephadm/services/cephadmservice.py b/src/pybind/mgr/cephadm/services/cephadmservice.py index 04f5af28a9b..4f83d7bb0fb 100644 --- a/src/pybind/mgr/cephadm/services/cephadmservice.py +++ b/src/pybind/mgr/cephadm/services/cephadmservice.py @@ -1157,6 +1157,14 @@ class RgwService(CephService): 'value': str(spec.rgw_bucket_counters_cache_size), }) + if getattr(spec, 'disable_multisite_sync_traffic', None) is not None: + ret, out, err = self.mgr.check_mon_command({ + 'prefix': 'config set', + 'who': daemon_name, + 'name': 'rgw_run_sync_thread', + 'value': 'false' if spec.disable_multisite_sync_traffic else 'true', + }) + daemon_spec.keyring = keyring daemon_spec.final_config, daemon_spec.deps = self.generate_config(daemon_spec) diff --git a/src/pybind/mgr/cephadm/services/monitoring.py b/src/pybind/mgr/cephadm/services/monitoring.py index 1b9cf618570..9c5b5a112f3 100644 --- a/src/pybind/mgr/cephadm/services/monitoring.py +++ b/src/pybind/mgr/cephadm/services/monitoring.py @@ -3,6 +3,7 @@ import logging import os import socket from typing import List, Any, Tuple, Dict, Optional, cast +import ipaddress from mgr_module import HandleCommandResult @@ -57,6 +58,8 @@ class GrafanaService(CephadmService): if ip_to_bind_to: daemon_spec.port_ips = {str(grafana_port): ip_to_bind_to} grafana_ip = ip_to_bind_to + if ipaddress.ip_network(grafana_ip).version == 6: + grafana_ip = f"[{grafana_ip}]" domain = self.mgr.get_fqdn(daemon_spec.host) mgmt_gw_ips = [] @@ -354,6 +357,13 @@ class AlertmanagerService(CephadmService): addr = self.mgr.get_fqdn(dd.hostname) peers.append(build_url(host=addr, port=port).lstrip('/')) + ip_to_bind_to = '' + if spec.only_bind_port_on_networks and spec.networks: + assert daemon_spec.host is not None + ip_to_bind_to = self.mgr.get_first_matching_network_ip(daemon_spec.host, spec) or '' + if ip_to_bind_to: + daemon_spec.port_ips = {str(port): ip_to_bind_to} + deps.append(f'secure_monitoring_stack:{self.mgr.secure_monitoring_stack}') if security_enabled: alertmanager_user, alertmanager_password = self.mgr._get_alertmanager_credentials() @@ -376,7 +386,8 @@ class AlertmanagerService(CephadmService): }, 'peers': peers, 'web_config': '/etc/alertmanager/web.yml', - 'use_url_prefix': mgmt_gw_enabled + 'use_url_prefix': mgmt_gw_enabled, + 'ip_to_bind_to': ip_to_bind_to }, sorted(deps) else: return { @@ -384,7 +395,8 @@ class AlertmanagerService(CephadmService): "alertmanager.yml": yml }, "peers": peers, - 'use_url_prefix': mgmt_gw_enabled + 'use_url_prefix': mgmt_gw_enabled, + 'ip_to_bind_to': ip_to_bind_to }, sorted(deps) def get_active_daemon(self, daemon_descrs: List[DaemonDescription]) -> DaemonDescription: diff --git a/src/pybind/mgr/cephadm/services/nvmeof.py b/src/pybind/mgr/cephadm/services/nvmeof.py index 418be93b6af..b3fd526815e 100644 --- a/src/pybind/mgr/cephadm/services/nvmeof.py +++ b/src/pybind/mgr/cephadm/services/nvmeof.py @@ -38,6 +38,8 @@ class NvmeofService(CephService): spec = cast(NvmeofServiceSpec, self.mgr.spec_store[daemon_spec.service_name].spec) nvmeof_gw_id = daemon_spec.daemon_id host_ip = self.mgr.inventory.get_addr(daemon_spec.host) + map_addr = spec.addr_map.get(daemon_spec.host) if spec.addr_map else None + map_discovery_addr = spec.discovery_addr_map.get(daemon_spec.host) if spec.discovery_addr_map else None keyring = self.get_keyring_with_caps(self.get_auth_entity(nvmeof_gw_id), ['mon', 'profile rbd', @@ -47,8 +49,14 @@ class NvmeofService(CephService): transport_tcp_options = json.dumps(spec.transport_tcp_options) if spec.transport_tcp_options else None name = '{}.{}'.format(utils.name_to_config_section('nvmeof'), nvmeof_gw_id) rados_id = name[len('client.'):] if name.startswith('client.') else name - addr = spec.addr or host_ip - discovery_addr = spec.discovery_addr or host_ip + + # The address is first searched in the per node address map, + # then in the spec address configuration. + # If neither is defined, the host IP is used as a fallback. + addr = map_addr or spec.addr or host_ip + self.mgr.log.info(f"gateway address: {addr} from {map_addr=} {spec.addr=} {host_ip=}") + discovery_addr = map_discovery_addr or spec.discovery_addr or host_ip + self.mgr.log.info(f"discovery address: {discovery_addr} from {map_discovery_addr=} {spec.discovery_addr=} {host_ip=}") context = { 'spec': spec, 'name': name, @@ -91,6 +99,9 @@ class NvmeofService(CephService): daemon_spec.extra_files['client_key'] = spec.client_key daemon_spec.extra_files['root_ca_cert'] = spec.root_ca_cert + if spec.encryption_key: + daemon_spec.extra_files['encryption_key'] = spec.encryption_key + daemon_spec.final_config, daemon_spec.deps = self.generate_config(daemon_spec) daemon_spec.deps = [] return daemon_spec diff --git a/src/pybind/mgr/cephadm/ssh.py b/src/pybind/mgr/cephadm/ssh.py index 1622cb001ab..acb5a77c51b 100644 --- a/src/pybind/mgr/cephadm/ssh.py +++ b/src/pybind/mgr/cephadm/ssh.py @@ -358,7 +358,7 @@ class SSHManager: await self._check_execute_command(host, chown, addr=addr) chmod = RemoteCommand(Executables.CHMOD, [oct(mode)[2:], tmp_path]) await self._check_execute_command(host, chmod, addr=addr) - mv = RemoteCommand(Executables.MV, [tmp_path, path]) + mv = RemoteCommand(Executables.MV, ['-Z', tmp_path, path]) await self._check_execute_command(host, mv, addr=addr) except Exception as e: msg = f"Unable to write {host}:{path}: {e}" diff --git a/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 b/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 index de993cb6ce3..b6955caf616 100644 --- a/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 +++ b/src/pybind/mgr/cephadm/templates/services/alertmanager/alertmanager.yml.j2 @@ -8,6 +8,8 @@ global: tls_config: {% if security_enabled %} ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key {% else %} insecure_skip_verify: true {% endif %} diff --git a/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 b/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 index b9773ceeeb3..14af0fd48ca 100644 --- a/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 +++ b/src/pybind/mgr/cephadm/templates/services/mgmt-gateway/nginx.conf.j2 @@ -9,6 +9,7 @@ events { http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; diff --git a/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 b/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 index 73f36952ee7..37f2db52732 100644 --- a/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 +++ b/src/pybind/mgr/cephadm/templates/services/nvmeof/ceph-nvmeof.conf.j2 @@ -9,10 +9,16 @@ state_update_notify = {{ spec.state_update_notify }} state_update_interval_sec = {{ spec.state_update_interval_sec }} enable_spdk_discovery_controller = {{ spec.enable_spdk_discovery_controller }} enable_key_encryption = {{ spec.enable_key_encryption }} +encryption_key = /encryption.key +rebalance_period_sec = {{ spec.rebalance_period_sec }} +max_gws_in_grp = {{ spec.max_gws_in_grp }} +max_ns_to_change_lb_grp = {{ spec.max_ns_to_change_lb_grp }} enable_prometheus_exporter = {{ spec.enable_prometheus_exporter }} prometheus_exporter_ssl = False -prometheus_port = 10008 +prometheus_port = {{ spec.prometheus_port }} +prometheus_stats_interval = {{ spec.prometheus_stats_interval }} verify_nqns = {{ spec.verify_nqns }} +verify_keys = {{ spec.verify_keys }} omap_file_lock_duration = {{ spec.omap_file_lock_duration }} omap_file_lock_retries = {{ spec.omap_file_lock_retries }} omap_file_lock_retry_sleep_interval = {{ spec.omap_file_lock_retry_sleep_interval }} diff --git a/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 b/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 index ecfd899af71..961da145dac 100644 --- a/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 +++ b/src/pybind/mgr/cephadm/templates/services/prometheus/prometheus.yml.j2 @@ -28,6 +28,8 @@ alerting: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} - scheme: http http_sd_configs: @@ -56,6 +58,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} honor_labels: true http_sd_configs: @@ -81,6 +85,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ node_exporter_sd_url }} @@ -104,6 +110,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ haproxy_sd_url }} @@ -128,6 +136,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} honor_labels: true http_sd_configs: @@ -149,6 +159,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ nvmeof_sd_url }} @@ -169,6 +181,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ nfs_sd_url }} @@ -189,6 +203,8 @@ scrape_configs: password: {{ service_discovery_password }} tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key {% else %} http_sd_configs: - url: {{ smb_sd_url }} diff --git a/src/pybind/mgr/cephadm/tests/test_cephadm.py b/src/pybind/mgr/cephadm/tests/test_cephadm.py index 3c647476e44..22bd26def91 100644 --- a/src/pybind/mgr/cephadm/tests/test_cephadm.py +++ b/src/pybind/mgr/cephadm/tests/test_cephadm.py @@ -1741,16 +1741,23 @@ class TestCephadm(object): nvmeof_client_cert = 'fake-nvmeof-client-cert' nvmeof_server_cert = 'fake-nvmeof-server-cert' nvmeof_root_ca_cert = 'fake-nvmeof-root-ca-cert' + grafana_cert_host_1 = 'grafana-cert-host-1' + grafana_cert_host_2 = 'grafana-cert-host-2' cephadm_module.cert_key_store.save_cert('rgw_frontend_ssl_cert', rgw_frontend_rgw_foo_host2_cert, service_name='rgw.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_server_cert', nvmeof_server_cert, service_name='nvmeof.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_client_cert', nvmeof_client_cert, service_name='nvmeof.foo', user_made=True) cephadm_module.cert_key_store.save_cert('nvmeof_root_ca_cert', nvmeof_root_ca_cert, service_name='nvmeof.foo', user_made=True) + cephadm_module.cert_key_store.save_cert('grafana_cert', grafana_cert_host_1, host='host-1', user_made=True) + cephadm_module.cert_key_store.save_cert('grafana_cert', grafana_cert_host_2, host='host-2', user_made=True) expected_calls = [ mock.call(f'{CERT_STORE_CERT_PREFIX}rgw_frontend_ssl_cert', json.dumps({'rgw.foo': Cert(rgw_frontend_rgw_foo_host2_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_server_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_server_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_client_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_client_cert, True).to_json()})), mock.call(f'{CERT_STORE_CERT_PREFIX}nvmeof_root_ca_cert', json.dumps({'nvmeof.foo': Cert(nvmeof_root_ca_cert, True).to_json()})), + mock.call(f'{CERT_STORE_CERT_PREFIX}grafana_cert', json.dumps({'host-1': Cert(grafana_cert_host_1, True).to_json()})), + mock.call(f'{CERT_STORE_CERT_PREFIX}grafana_cert', json.dumps({'host-1': Cert(grafana_cert_host_1, True).to_json(), + 'host-2': Cert(grafana_cert_host_2, True).to_json()})) ] _set_store.assert_has_calls(expected_calls) @@ -1795,17 +1802,23 @@ class TestCephadm(object): cephadm_module.cert_key_store._init_known_cert_key_dicts() grafana_host1_key = 'fake-grafana-host1-key' + grafana_host2_key = 'fake-grafana-host2-key' nvmeof_client_key = 'nvmeof-client-key' nvmeof_server_key = 'nvmeof-server-key' - grafana_host1_key = 'fake-grafana-host1-cert' + nvmeof_encryption_key = 'nvmeof-encryption-key' cephadm_module.cert_key_store.save_key('grafana_key', grafana_host1_key, host='host1') + cephadm_module.cert_key_store.save_key('grafana_key', grafana_host2_key, host='host2') cephadm_module.cert_key_store.save_key('nvmeof_client_key', nvmeof_client_key, service_name='nvmeof.foo') cephadm_module.cert_key_store.save_key('nvmeof_server_key', nvmeof_server_key, service_name='nvmeof.foo') + cephadm_module.cert_key_store.save_key('nvmeof_encryption_key', nvmeof_encryption_key, service_name='nvmeof.foo') expected_calls = [ mock.call(f'{CERT_STORE_KEY_PREFIX}grafana_key', json.dumps({'host1': PrivKey(grafana_host1_key).to_json()})), + mock.call(f'{CERT_STORE_KEY_PREFIX}grafana_key', json.dumps({'host1': PrivKey(grafana_host1_key).to_json(), + 'host2': PrivKey(grafana_host2_key).to_json()})), mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_client_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_client_key).to_json()})), mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_server_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_server_key).to_json()})), + mock.call(f'{CERT_STORE_KEY_PREFIX}nvmeof_encryption_key', json.dumps({'nvmeof.foo': PrivKey(nvmeof_encryption_key).to_json()})), ] _set_store.assert_has_calls(expected_calls) @@ -1822,15 +1835,19 @@ class TestCephadm(object): 'ingress_ssl_key': False, 'nvmeof_client_key': False, 'nvmeof_server_key': False, + 'nvmeof_encryption_key': False, } assert cephadm_module.cert_key_store.key_ls() == expected_ls cephadm_module.cert_key_store.save_key('nvmeof_client_key', 'xxx', service_name='nvmeof.foo') cephadm_module.cert_key_store.save_key('nvmeof_server_key', 'xxx', service_name='nvmeof.foo') + cephadm_module.cert_key_store.save_key('nvmeof_encryption_key', 'xxx', service_name='nvmeof.foo') expected_ls['nvmeof_server_key'] = {} expected_ls['nvmeof_server_key']['nvmeof.foo'] = True expected_ls['nvmeof_client_key'] = {} expected_ls['nvmeof_client_key']['nvmeof.foo'] = True + expected_ls['nvmeof_encryption_key'] = {} + expected_ls['nvmeof_encryption_key']['nvmeof.foo'] = True assert cephadm_module.cert_key_store.key_ls() == expected_ls @mock.patch("cephadm.module.CephadmOrchestrator.get_store_prefix") @@ -1844,6 +1861,7 @@ class TestCephadm(object): nvmeof_root_ca_cert = 'nvmeof-root-ca-cert' nvmeof_server_key = 'nvmeof-server-key' nvmeof_client_key = 'nvmeof-client-key' + nvmeof_encryption_key = 'nvmeof-encryption-key' def _fake_prefix_store(key): if key == 'cert_store.cert.': @@ -1858,6 +1876,7 @@ class TestCephadm(object): f'{CERT_STORE_KEY_PREFIX}grafana_key': json.dumps({'host1': PrivKey(grafana_host1_key).to_json()}), f'{CERT_STORE_KEY_PREFIX}nvmeof_server_key': json.dumps({'nvmeof.foo': PrivKey(nvmeof_server_key).to_json()}), f'{CERT_STORE_KEY_PREFIX}nvmeof_client_key': json.dumps({'nvmeof.foo': PrivKey(nvmeof_client_key).to_json()}), + f'{CERT_STORE_KEY_PREFIX}nvmeof_encryption_key': json.dumps({'nvmeof.foo': PrivKey(nvmeof_encryption_key).to_json()}), } else: raise Exception(f'Get store with unexpected value {key}') @@ -1871,6 +1890,7 @@ class TestCephadm(object): assert cephadm_module.cert_key_store.known_keys['grafana_key']['host1'] == PrivKey(grafana_host1_key) assert cephadm_module.cert_key_store.known_keys['nvmeof_server_key']['nvmeof.foo'] == PrivKey(nvmeof_server_key) assert cephadm_module.cert_key_store.known_keys['nvmeof_client_key']['nvmeof.foo'] == PrivKey(nvmeof_client_key) + assert cephadm_module.cert_key_store.known_keys['nvmeof_encryption_key']['nvmeof.foo'] == PrivKey(nvmeof_encryption_key) def test_cert_store_get_cert_key(self, cephadm_module: CephadmOrchestrator): cephadm_module.cert_key_store._init_known_cert_key_dicts() @@ -1898,13 +1918,16 @@ class TestCephadm(object): grafana_host1_key = 'fake-grafana-host1-cert' nvmeof_server_key = 'nvmeof-server-key' + nvmeof_encryption_key = 'nvmeof-encryption-key' cephadm_module.cert_key_store.save_key('grafana_key', grafana_host1_key, host='host1') cephadm_module.cert_key_store.save_key('grafana_key', grafana_host1_key, host='host1') cephadm_module.cert_key_store.save_key('nvmeof_server_key', nvmeof_server_key, service_name='nvmeof.foo') + cephadm_module.cert_key_store.save_key('nvmeof_encryption_key', nvmeof_encryption_key, service_name='nvmeof.foo') assert cephadm_module.cert_key_store.get_key('grafana_key', host='host1') == grafana_host1_key assert cephadm_module.cert_key_store.get_key('nvmeof_server_key', service_name='nvmeof.foo') == nvmeof_server_key assert cephadm_module.cert_key_store.get_key('nvmeof_client_key', service_name='nvmeof.foo') == '' + assert cephadm_module.cert_key_store.get_key('nvmeof_encryption_key', service_name='nvmeof.foo') == nvmeof_encryption_key with pytest.raises(OrchestratorError, match='Attempted to access priv key for unknown entity'): cephadm_module.cert_key_store.get_key('unknown_entity') diff --git a/src/pybind/mgr/cephadm/tests/test_services.py b/src/pybind/mgr/cephadm/tests/test_services.py index 75c7c3c5bf7..d872219df80 100644 --- a/src/pybind/mgr/cephadm/tests/test_services.py +++ b/src/pybind/mgr/cephadm/tests/test_services.py @@ -399,10 +399,16 @@ state_update_notify = True state_update_interval_sec = 5 enable_spdk_discovery_controller = False enable_key_encryption = True +encryption_key = /encryption.key +rebalance_period_sec = 7 +max_gws_in_grp = 16 +max_ns_to_change_lb_grp = 8 enable_prometheus_exporter = True prometheus_exporter_ssl = False prometheus_port = 10008 +prometheus_stats_interval = 10 verify_nqns = True +verify_keys = True omap_file_lock_duration = 20 omap_file_lock_retries = 30 omap_file_lock_retry_sleep_interval = 1.0 @@ -575,7 +581,14 @@ class TestMonitoring: mock_getfqdn.return_value = purl.hostname with with_host(cephadm_module, "test"): - with with_service(cephadm_module, AlertManagerSpec()): + cephadm_module.cache.update_host_networks('test', { + '1.2.3.0/24': { + 'if0': ['1.2.3.1'] + }, + }) + with with_service(cephadm_module, AlertManagerSpec('alertmanager', + networks=['1.2.3.0/24'], + only_bind_port_on_networks=True)): y = dedent(self._get_config(expected_yaml_url)).lstrip() _run_cephadm.assert_called_with( 'test', @@ -589,11 +602,12 @@ class TestMonitoring: "deploy_arguments": [], "params": { 'tcp_ports': [9093, 9094], + 'port_ips': {"9094": "1.2.3.1"}, }, "meta": { 'service_name': 'alertmanager', 'ports': [9093, 9094], - 'ip': None, + 'ip': '1.2.3.1', 'deployed_by': [], 'rank': None, 'rank_generation': None, @@ -606,6 +620,7 @@ class TestMonitoring: }, "peers": [], "use_url_prefix": False, + "ip_to_bind_to": "1.2.3.1", } }), error_ok=True, @@ -628,8 +643,16 @@ class TestMonitoring: cephadm_module.secure_monitoring_stack = True cephadm_module.set_store(AlertmanagerService.USER_CFG_KEY, 'alertmanager_user') cephadm_module.set_store(AlertmanagerService.PASS_CFG_KEY, 'alertmanager_plain_password') + + cephadm_module.cache.update_host_networks('test', { + 'fd12:3456:789a::/64': { + 'if0': ['fd12:3456:789a::10'] + }, + }) with with_service(cephadm_module, MgmtGatewaySpec("mgmt-gateway")) as _, \ - with_service(cephadm_module, AlertManagerSpec()): + with_service(cephadm_module, AlertManagerSpec('alertmanager', + networks=['fd12:3456:789a::/64'], + only_bind_port_on_networks=True)): y = dedent(""" # This file is generated by cephadm. @@ -640,6 +663,8 @@ class TestMonitoring: http_config: tls_config: ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key route: receiver: 'default' @@ -680,11 +705,12 @@ class TestMonitoring: "deploy_arguments": [], "params": { 'tcp_ports': [9093, 9094], + 'port_ips': {"9094": "fd12:3456:789a::10"} }, "meta": { 'service_name': 'alertmanager', 'ports': [9093, 9094], - 'ip': None, + 'ip': 'fd12:3456:789a::10', 'deployed_by': [], 'rank': None, 'rank_generation': None, @@ -702,6 +728,7 @@ class TestMonitoring: 'peers': [], 'web_config': '/etc/alertmanager/web.yml', "use_url_prefix": True, + "ip_to_bind_to": "fd12:3456:789a::10", } }), error_ok=True, @@ -735,6 +762,8 @@ class TestMonitoring: http_config: tls_config: ca_file: root_cert.pem + cert_file: alertmanager.crt + key_file: alertmanager.key route: receiver: 'default' @@ -795,6 +824,7 @@ class TestMonitoring: 'peers': [], 'web_config': '/etc/alertmanager/web.yml', "use_url_prefix": False, + "ip_to_bind_to": "", } }), error_ok=True, @@ -1164,6 +1194,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key scrape_configs: - job_name: 'ceph' @@ -1185,6 +1217,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'node' relabel_configs: @@ -1203,6 +1237,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'haproxy' relabel_configs: @@ -1219,6 +1255,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'ceph-exporter' relabel_configs: @@ -1236,6 +1274,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'nvmeof' honor_labels: true @@ -1249,6 +1289,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'nfs' honor_labels: true @@ -1262,6 +1304,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key - job_name: 'smb' honor_labels: true @@ -1275,6 +1319,8 @@ class TestMonitoring: password: sd_password tls_config: ca_file: root_cert.pem + cert_file: prometheus.crt + key_file: prometheus.key """).lstrip() @@ -2065,6 +2111,26 @@ class TestRGWService: }) assert f == expected + @pytest.mark.parametrize( + "disable_sync_traffic", + [ + (True), + (False), + ] + ) + @patch("cephadm.serve.CephadmServe._run_cephadm", _run_cephadm('{}')) + def test_rgw_disable_sync_traffic(self, disable_sync_traffic, cephadm_module: CephadmOrchestrator): + with with_host(cephadm_module, 'host1'): + s = RGWSpec(service_id="foo", + disable_multisite_sync_traffic=disable_sync_traffic) + with with_service(cephadm_module, s) as dds: + _, f, _ = cephadm_module.check_mon_command({ + 'prefix': 'config get', + 'who': f'client.{dds[0]}', + 'key': 'rgw_run_sync_thread', + }) + assert f == ('false' if disable_sync_traffic else 'true') + class TestMonService: @@ -3868,6 +3934,7 @@ class TestMgmtGateway: http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; @@ -4115,6 +4182,7 @@ class TestMgmtGateway: http { #access_log /dev/stdout; + error_log /dev/stderr info; client_header_buffer_size 32K; large_client_header_buffers 4 32k; proxy_busy_buffers_size 512k; diff --git a/src/pybind/mgr/dashboard/HACKING.rst b/src/pybind/mgr/dashboard/HACKING.rst index 39c3d6744b9..6da428a0d5f 100644 --- a/src/pybind/mgr/dashboard/HACKING.rst +++ b/src/pybind/mgr/dashboard/HACKING.rst @@ -4,7 +4,7 @@ Ceph Dashboard Developer Documentation Note: The content of this file has been moved into the Ceph Developer Guide. If you're interested in helping with the development of the dashboard, please -see ``/doc/dev/developer_guide/dash_devel.rst`` or the `online version +see ``/doc/dev/developer_guide/dash-devel.rst`` or the `online version <https://ceph.readthedocs.io/en/latest/dev/developer_guide/dash-devel/>`_ for details on how to set up a development environment and other development-related topics. diff --git a/src/pybind/mgr/dashboard/controllers/cephfs.py b/src/pybind/mgr/dashboard/controllers/cephfs.py index 9f9b7501f44..d05b7551365 100644 --- a/src/pybind/mgr/dashboard/controllers/cephfs.py +++ b/src/pybind/mgr/dashboard/controllers/cephfs.py @@ -2,7 +2,6 @@ # pylint: disable=too-many-lines import errno import json -import logging import os from collections import defaultdict from typing import Any, Dict, List @@ -30,8 +29,6 @@ GET_STATFS_SCHEMA = { 'subdirs': (int, '') } -logger = logging.getLogger("controllers.rgw") - # pylint: disable=R0904 @APIRouter('/cephfs', Scope.CEPHFS) diff --git a/src/pybind/mgr/dashboard/controllers/cluster_configuration.py b/src/pybind/mgr/dashboard/controllers/cluster_configuration.py index da5be2cc81d..292f381d79f 100644 --- a/src/pybind/mgr/dashboard/controllers/cluster_configuration.py +++ b/src/pybind/mgr/dashboard/controllers/cluster_configuration.py @@ -1,12 +1,14 @@ # -*- coding: utf-8 -*- +from typing import Optional + import cherrypy from .. import mgr from ..exceptions import DashboardException from ..security import Scope from ..services.ceph_service import CephService -from . import APIDoc, APIRouter, EndpointDoc, RESTController +from . import APIDoc, APIRouter, EndpointDoc, Param, RESTController FILTER_SCHEMA = [{ "name": (str, 'Name of the config option'), @@ -80,22 +82,33 @@ class ClusterConfiguration(RESTController): return config_options - def create(self, name, value): + @EndpointDoc("Create/Update Cluster Configuration", + parameters={ + 'name': Param(str, 'Config option name'), + 'value': ( + [ + { + 'section': Param( + str, 'Section/Client where config needs to be updated' + ), + 'value': Param(str, 'Value of the config option') + } + ], 'Section and Value of the config option' + ), + 'force_update': Param(bool, 'Force update the config option', False, None) + } + ) + def create(self, name, value, force_update: Optional[bool] = None): # Check if config option is updateable at runtime - self._updateable_at_runtime([name]) + self._updateable_at_runtime([name], force_update) - # Update config option - avail_sections = ['global', 'mon', 'mgr', 'osd', 'mds', 'client'] + for entry in value: + section = entry['section'] + entry_value = entry['value'] - for section in avail_sections: - for entry in value: - if entry['value'] is None: - break - - if entry['section'] == section: - CephService.send_command('mon', 'config set', who=section, name=name, - value=str(entry['value'])) - break + if entry_value not in (None, ''): + CephService.send_command('mon', 'config set', who=section, name=name, + value=str(entry_value)) else: CephService.send_command('mon', 'config rm', who=section, name=name) @@ -116,11 +129,24 @@ class ClusterConfiguration(RESTController): raise cherrypy.HTTPError(404) - def _updateable_at_runtime(self, config_option_names): + def _updateable_at_runtime(self, config_option_names, force_update=False): not_updateable = [] for name in config_option_names: config_option = self._get_config_option(name) + + # making rgw configuration to be editable by bypassing 'can_update_at_runtime' + # as the same can be done via CLI. + if force_update and 'rgw' in name and not config_option['can_update_at_runtime']: + break + + if force_update and 'rgw' not in name and not config_option['can_update_at_runtime']: + raise DashboardException( + msg=f'Only the configuration containing "rgw" can be edited at runtime with' + f' force_update flag, hence not able to update "{name}"', + code='config_option_not_updatable_at_runtime', + component='cluster_configuration' + ) if not config_option['can_update_at_runtime']: not_updateable.append(name) diff --git a/src/pybind/mgr/dashboard/controllers/rgw.py b/src/pybind/mgr/dashboard/controllers/rgw.py index 9d257674794..d48542a7590 100755 --- a/src/pybind/mgr/dashboard/controllers/rgw.py +++ b/src/pybind/mgr/dashboard/controllers/rgw.py @@ -106,13 +106,11 @@ class RgwMultisiteStatus(RESTController): @allow_empty_body # pylint: disable=W0102,W0613 def migrate(self, daemon_name=None, realm_name=None, zonegroup_name=None, zone_name=None, - zonegroup_endpoints=None, zone_endpoints=None, access_key=None, - secret_key=None): + zonegroup_endpoints=None, zone_endpoints=None, username=None): multisite_instance = RgwMultisite() result = multisite_instance.migrate_to_multisite(realm_name, zonegroup_name, zone_name, zonegroup_endpoints, - zone_endpoints, access_key, - secret_key) + zone_endpoints, username) return result @RESTController.Collection(method='POST', path='/multisite-replications') @@ -773,6 +771,9 @@ class RgwUser(RgwRESTController): return users def get(self, uid, daemon_name=None, stats=True) -> dict: + return self._get(uid, daemon_name=daemon_name, stats=stats) + + def _get(self, uid, daemon_name=None, stats=True) -> dict: query_params = '?stats' if stats else '' result = self.proxy(daemon_name, 'GET', 'user{}'.format(query_params), {'uid': uid, 'stats': stats}) @@ -788,7 +789,7 @@ class RgwUser(RgwRESTController): # type: (Optional[str]) -> List[str] emails = [] for uid in json.loads(self.list(daemon_name)): # type: ignore - user = json.loads(self.get(uid, daemon_name)) # type: ignore + user = self._get(uid, daemon_name) # type: ignore if user["email"]: emails.append(user["email"]) return emails @@ -910,7 +911,7 @@ class RgwUser(RgwRESTController): secret_key=None, daemon_name=None): # pylint: disable=R1705 subusr_array = [] - user = json.loads(self.get(uid, daemon_name)) # type: ignore + user = self._get(uid, daemon_name) # type: ignore subusers = user["subusers"] for sub_usr in subusers: subusr_array.append(sub_usr["id"]) diff --git a/src/pybind/mgr/dashboard/controllers/rgw_iam.py b/src/pybind/mgr/dashboard/controllers/rgw_iam.py new file mode 100644 index 00000000000..458bbbb7321 --- /dev/null +++ b/src/pybind/mgr/dashboard/controllers/rgw_iam.py @@ -0,0 +1,52 @@ +from typing import Optional + +from ..security import Scope +from ..services.rgw_iam import RgwAccounts +from ..tools import str_to_bool +from . import APIDoc, APIRouter, EndpointDoc, RESTController, allow_empty_body + + +@APIRouter('rgw/accounts', Scope.RGW) +@APIDoc("RGW User Accounts API", "RgwUserAccounts") +class RgwUserAccountsController(RESTController): + + @allow_empty_body + def create(self, account_name: Optional[str] = None, + account_id: Optional[str] = None, email: Optional[str] = None): + return RgwAccounts.create_account(account_name, account_id, email) + + def list(self, detailed: bool = False): + detailed = str_to_bool(detailed) + return RgwAccounts.get_accounts(detailed) + + @EndpointDoc("Get RGW Account by id", + parameters={'account_id': (str, 'Account id')}) + def get(self, account_id: str): + return RgwAccounts.get_account(account_id) + + @EndpointDoc("Delete RGW Account", + parameters={'account_id': (str, 'Account id')}) + def delete(self, account_id): + return RgwAccounts.delete_account(account_id) + + @EndpointDoc("Update RGW account info", + parameters={'account_id': (str, 'Account id')}) + @allow_empty_body + def set(self, account_id: str, account_name: Optional[str] = None, + email: Optional[str] = None): + return RgwAccounts.modify_account(account_id, account_name, email) + + @EndpointDoc("Set RGW Account/Bucket quota", + parameters={'account_id': (str, 'Account id'), + 'max_size': (str, 'Max size')}) + @RESTController.Resource(method='PUT', path='/quota') + @allow_empty_body + def set_quota(self, quota_type: str, account_id: str, max_size: str, max_objects: str): + return RgwAccounts.set_quota(quota_type, account_id, max_size, max_objects) + + @EndpointDoc("Enable/Disable RGW Account/Bucket quota", + parameters={'account_id': (str, 'Account id')}) + @RESTController.Resource(method='PUT', path='/quota/status') + @allow_empty_body + def set_quota_status(self, quota_type: str, account_id: str, quota_status: str): + return RgwAccounts.set_quota_status(quota_type, account_id, quota_status) diff --git a/src/pybind/mgr/dashboard/controllers/smb.py b/src/pybind/mgr/dashboard/controllers/smb.py new file mode 100644 index 00000000000..97eff8c3dfe --- /dev/null +++ b/src/pybind/mgr/dashboard/controllers/smb.py @@ -0,0 +1,186 @@ + +# -*- coding: utf-8 -*- + +import json +import logging +from typing import List + +from smb.enums import Intent +from smb.proto import Simplified +from smb.resources import Cluster, Share + +from dashboard.controllers._docs import EndpointDoc +from dashboard.controllers._permissions import CreatePermission, DeletePermission +from dashboard.exceptions import DashboardException + +from .. import mgr +from ..security import Scope +from . import APIDoc, APIRouter, ReadPermission, RESTController + +logger = logging.getLogger('controllers.smb') + +CLUSTER_SCHEMA = { + "resource_type": (str, "ceph.smb.cluster"), + "cluster_id": (str, "Unique identifier for the cluster"), + "auth_mode": (str, "Either 'active-directory' or 'user'"), + "intent": (str, "Desired state of the resource, e.g., 'present' or 'removed'"), + "domain_settings": ({ + "realm": (str, "Domain realm, e.g., 'DOMAIN1.SINK.TEST'"), + "join_sources": ([{ + "source_type": (str, "resource"), + "ref": (str, "Reference identifier for the join auth resource") + }], "List of join auth sources for domain settings") + }, "Domain-specific settings for active-directory auth mode"), + "user_group_settings": ([{ + "source_type": (str, "resource"), + "ref": (str, "Reference identifier for the user group resource") + }], "User group settings for user auth mode"), + "custom_dns": ([str], "List of custom DNS server addresses"), + "placement": ({ + "count": (int, "Number of instances to place") + }, "Placement configuration for the resource") +} + +CLUSTER_SCHEMA_RESULTS = { + "results": ([{ + "resource": ({ + "resource_type": (str, "ceph.smb.cluster"), + "cluster_id": (str, "Unique identifier for the cluster"), + "auth_mode": (str, "Either 'active-directory' or 'user'"), + "intent": (str, "Desired state of the resource, e.g., 'present' or 'removed'"), + "domain_settings": ({ + "realm": (str, "Domain realm, e.g., 'DOMAIN1.SINK.TEST'"), + "join_sources": ([{ + "source_type": (str, "resource"), + "ref": (str, "Reference identifier for the join auth resource") + }], "List of join auth sources for domain settings") + }, "Domain-specific settings for active-directory auth mode"), + "user_group_settings": ([{ + "source_type": (str, "resource"), + "ref": (str, "Reference identifier for the user group resource") + }], "User group settings for user auth mode (optional)"), + "custom_dns": ([str], "List of custom DNS server addresses (optional)"), + "placement": ({ + "count": (int, "Number of instances to place") + }, "Placement configuration for the resource (optional)"), + }, "Resource details"), + "state": (str, "State of the resource"), + "success": (bool, "Indicates whether the operation was successful") + }], "List of results with resource details"), + "success": (bool, "Overall success status of the operation") +} + +LIST_CLUSTER_SCHEMA = [CLUSTER_SCHEMA] + +SHARE_SCHEMA = { + "resource_type": (str, "ceph.smb.share"), + "cluster_id": (str, "Unique identifier for the cluster"), + "share_id": (str, "Unique identifier for the share"), + "intent": (str, "Desired state of the resource, e.g., 'present' or 'removed'"), + "name": (str, "Name of the share"), + "readonly": (bool, "Indicates if the share is read-only"), + "browseable": (bool, "Indicates if the share is browseable"), + "cephfs": ({ + "volume": (str, "Name of the CephFS file system"), + "path": (str, "Path within the CephFS file system"), + "provider": (str, "Provider of the CephFS share, e.g., 'samba-vfs'") + }, "Configuration for the CephFS share") +} + + +@APIRouter('/smb/cluster', Scope.SMB) +@APIDoc("SMB Cluster Management API", "SMB") +class SMBCluster(RESTController): + _resource: str = 'ceph.smb.cluster' + + @ReadPermission + @EndpointDoc("List smb clusters", + responses={200: LIST_CLUSTER_SCHEMA}) + def list(self) -> List[Cluster]: + """ + List smb clusters + """ + res = mgr.remote('smb', 'show', [self._resource]) + return res['resources'] if 'resources' in res else [res] + + @ReadPermission + @EndpointDoc("Get an smb cluster", + parameters={ + 'cluster_id': (str, 'Unique identifier for the cluster') + }, + responses={200: CLUSTER_SCHEMA}) + def get(self, cluster_id: str) -> Cluster: + """ + Get an smb cluster by cluster id + """ + return mgr.remote('smb', 'show', [f'{self._resource}.{cluster_id}']) + + @CreatePermission + @EndpointDoc("Create smb cluster", + parameters={ + 'cluster_resource': (str, 'cluster_resource') + }, + responses={201: CLUSTER_SCHEMA_RESULTS}) + def create(self, cluster_resource: Cluster) -> Simplified: + """ + Create an smb cluster + + :param cluster_resource: Dict cluster data + :return: Returns cluster resource. + :rtype: Dict[str, Any] + """ + try: + return mgr.remote( + 'smb', + 'apply_resources', + json.dumps(cluster_resource)).to_simplified() + except RuntimeError as e: + raise DashboardException(e, component='smb') + + +@APIRouter('/smb/share', Scope.SMB) +@APIDoc("SMB Share Management API", "SMB") +class SMBShare(RESTController): + _resource: str = 'ceph.smb.share' + + @ReadPermission + @EndpointDoc("List smb shares", + parameters={ + 'cluster_id': (str, 'Unique identifier for the cluster') + }, + responses={200: SHARE_SCHEMA}) + def list(self, cluster_id: str = '') -> List[Share]: + """ + List all smb shares or all shares for a given cluster + + :param cluster_id: Dict containing cluster information + :return: Returns list of shares. + :rtype: List[Dict] + """ + res = mgr.remote( + 'smb', + 'show', + [f'{self._resource}.{cluster_id}' if cluster_id else self._resource]) + return res['resources'] if 'resources' in res else res + + @DeletePermission + @EndpointDoc("Remove smb shares", + parameters={ + 'cluster_id': (str, 'Unique identifier for the cluster'), + 'share_id': (str, 'Unique identifier for the share') + }, + responses={204: None}) + def delete(self, cluster_id: str, share_id: str): + """ + Remove an smb share from a given cluster + + :param cluster_id: Cluster identifier + :param share_id: Share identifier + :return: None. + """ + resource = {} + resource['resource_type'] = self._resource + resource['cluster_id'] = cluster_id + resource['share_id'] = share_id + resource['intent'] = Intent.REMOVED + return mgr.remote('smb', 'apply_resources', json.dumps(resource)).one().to_simplified() diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts index 983140a44c4..b71719c4396 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.e2e-spec.ts @@ -30,7 +30,6 @@ describe('Configuration page', () => { beforeEach(() => { configuration.clearTableSearchInput(); - configuration.getTableCount('found').as('configFound'); }); after(() => { @@ -50,6 +49,8 @@ describe('Configuration page', () => { }); it('should verify modified filter is applied properly', () => { + configuration.clearFilter(); + configuration.getTableCount('found').as('configFound'); configuration.filterTable('Modified', 'no'); configuration.getTableCount('found').as('unmodifiedConfigs'); diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts index 82e79a676ec..4132387d0f1 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/cluster/configuration.po.ts @@ -12,7 +12,6 @@ export class ConfigurationPageHelper extends PageHelper { configClear(name: string) { this.navigateTo(); const valList = ['global', 'mon', 'mgr', 'osd', 'mds', 'client']; // Editable values - this.getFirstTableCell(name).click(); cy.contains('button', 'Edit').click(); // Waits for the data to load @@ -26,6 +25,8 @@ export class ConfigurationPageHelper extends PageHelper { cy.wait(3 * 1000); + this.clearFilter(); + // Enter config setting name into filter box this.searchTable(name, 100); @@ -49,6 +50,7 @@ export class ConfigurationPageHelper extends PageHelper { * Ex: [global, '2'] is the global value with an input of 2 */ edit(name: string, ...values: [string, string][]) { + this.clearFilter(); this.getFirstTableCell(name).click(); cy.contains('button', 'Edit').click(); @@ -78,4 +80,12 @@ export class ConfigurationPageHelper extends PageHelper { cy.contains('[data-testid=config-details-table]', `${value[0]}\: ${value[1]}`); }); } + + clearFilter() { + cy.get('div.filter-tags') // Find the div with class filter-tags + .find('button.cds--btn.cds--btn--ghost') // Find the button with specific classes + .contains('Clear filters') // Ensure the button contains the text "Clear filters" + .should('be.visible') // Assert that the button is visible + .click(); + } } diff --git a/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.po.ts b/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.po.ts index 89c4c7394d9..ae0e5b64f25 100644 --- a/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.po.ts +++ b/src/pybind/mgr/dashboard/frontend/cypress/e2e/ui/navigation.po.ts @@ -48,7 +48,8 @@ export class NavigationPageHelper extends PageHelper { menu: 'File', submenus: [ { menu: 'File Systems', component: 'cd-cephfs-list' }, - { menu: 'NFS', component: 'cd-error' } + { menu: 'NFS', component: 'cd-error' }, + { menu: 'SMB', component: 'cd-smb-cluster-list' } ] }, { diff --git a/src/pybind/mgr/dashboard/frontend/package-lock.json b/src/pybind/mgr/dashboard/frontend/package-lock.json index a091d3577ec..7b3e9beeb9f 100644 --- a/src/pybind/mgr/dashboard/frontend/package-lock.json +++ b/src/pybind/mgr/dashboard/frontend/package-lock.json @@ -29,7 +29,7 @@ "@types/file-saver": "2.0.1", "async-mutex": "0.2.4", "bootstrap": "5.2.3", - "carbon-components-angular": "5.48.0", + "carbon-components-angular": "5.56.2", "chart.js": "4.4.0", "chartjs-adapter-moment": "1.0.1", "detect-browser": "5.2.0", @@ -11208,9 +11208,9 @@ ] }, "node_modules/carbon-components-angular": { - "version": "5.48.0", - "resolved": "https://registry.npmjs.org/carbon-components-angular/-/carbon-components-angular-5.48.0.tgz", - "integrity": "sha512-NZwpKBKgkgaR51S0Pm16MvashO4g8B+dzGeNE8l/RYRWXpVDLShR6YnBc80t2iMtTolxGiPwHmzpyMieVtIGLg==", + "version": "5.56.2", + "resolved": "https://registry.npmjs.org/carbon-components-angular/-/carbon-components-angular-5.56.2.tgz", + "integrity": "sha512-mU5Nep4HwwRQIrToasLwfR8cB1ph6Hn3jnLjDw0kWN+NJj5HEwPizOhbTPQbQxvBo0stro2fMWW0x3ge519Hgg==", "hasInstallScript": true, "dependencies": { "@carbon/icon-helpers": "10.37.0", diff --git a/src/pybind/mgr/dashboard/frontend/package.json b/src/pybind/mgr/dashboard/frontend/package.json index 3348c5ff097..aa0035dcb78 100644 --- a/src/pybind/mgr/dashboard/frontend/package.json +++ b/src/pybind/mgr/dashboard/frontend/package.json @@ -63,7 +63,7 @@ "@types/file-saver": "2.0.1", "async-mutex": "0.2.4", "bootstrap": "5.2.3", - "carbon-components-angular": "5.48.0", + "carbon-components-angular": "5.56.2", "chart.js": "4.4.0", "chartjs-adapter-moment": "1.0.1", "detect-browser": "5.2.0", diff --git a/src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts b/src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts index 99d7bd0e2d8..f389b64a454 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/app-routing.module.ts @@ -51,6 +51,7 @@ import { UpgradeProgressComponent } from './ceph/cluster/upgrade/upgrade-progres import { MultiClusterComponent } from './ceph/cluster/multi-cluster/multi-cluster.component'; import { MultiClusterListComponent } from './ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component'; import { MultiClusterDetailsComponent } from './ceph/cluster/multi-cluster/multi-cluster-details/multi-cluster-details.component'; +import { SmbClusterListComponent } from './ceph/smb/smb-cluster-list/smb-cluster-list.component'; @Injectable() export class PerformanceCounterBreadcrumbsResolver extends BreadcrumbsResolver { @@ -429,6 +430,13 @@ const routes: Routes = [ data: { breadcrumbs: ActionLabels.EDIT } } ] + }, + { + path: 'smb', + data: { + breadcrumbs: 'File/SMB' + }, + children: [{ path: '', component: SmbClusterListComponent }] } ] }, diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/ceph.module.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/ceph.module.ts index 47772304b50..d269b6aa912 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/ceph.module.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/ceph.module.ts @@ -7,6 +7,7 @@ import { ClusterModule } from './cluster/cluster.module'; import { DashboardModule } from './dashboard/dashboard.module'; import { NfsModule } from './nfs/nfs.module'; import { PerformanceCounterModule } from './performance-counter/performance-counter.module'; +import { SmbModule } from './smb/smb.module'; @NgModule({ imports: [ @@ -16,6 +17,7 @@ import { PerformanceCounterModule } from './performance-counter/performance-coun PerformanceCounterModule, CephfsModule, NfsModule, + SmbModule, SharedModule ], declarations: [] diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts index bca65a887c5..b6d23e35c9d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form-create-request.model.ts @@ -1,4 +1,5 @@ export class ConfigFormCreateRequestModel { name: string; value: Array<any> = []; + force_update: boolean = false; } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html index 741c18d52a6..a6775dcee17 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html @@ -150,7 +150,7 @@ </div> <!-- Footer --> <div class="card-footer"> - <cd-form-button-panel (submitActionEvent)="submit()" + <cd-form-button-panel (submitActionEvent)="forceUpdate ? openCriticalConfirmModal() : submit()" [form]="configForm" [submitText]="actionLabels.UPDATE" wrappingClass="text-right"></cd-form-button-panel> @@ -158,3 +158,4 @@ </div> </form> </div> + diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts index b6e9e700be4..118cb18430a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.ts @@ -13,6 +13,10 @@ import { CdForm } from '~/app/shared/forms/cd-form'; import { CdFormGroup } from '~/app/shared/forms/cd-form-group'; import { NotificationService } from '~/app/shared/services/notification.service'; import { ConfigFormCreateRequestModel } from './configuration-form-create-request.model'; +import { CriticalConfirmationModalComponent } from '~/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component'; +import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; + +const RGW = 'rgw'; @Component({ selector: 'cd-configuration-form', @@ -29,13 +33,15 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { maxValue: number; patternHelpText: string; availSections = ['global', 'mon', 'mgr', 'osd', 'mds', 'client']; + forceUpdate: boolean; constructor( public actionLabels: ActionLabelsI18n, private route: ActivatedRoute, private router: Router, private configService: ConfigurationService, - private notificationService: NotificationService + private notificationService: NotificationService, + private modalService: ModalCdsService ) { super(); this.createForm(); @@ -95,7 +101,6 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { setResponse(response: ConfigFormModel) { this.response = response; const validators = this.getValidators(response); - this.configForm.get('name').setValue(response.name); this.configForm.get('desc').setValue(response.desc); this.configForm.get('long_desc').setValue(response.long_desc); @@ -118,7 +123,7 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { this.configForm.get('values').get(value.section).setValue(sectionValue); }); } - + this.forceUpdate = !this.response.can_update_at_runtime && response.name.includes(RGW); this.availSections.forEach((section) => { this.configForm.get('values').get(section).setValidators(validators); }); @@ -134,7 +139,7 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { this.availSections.forEach((section) => { const sectionValue = this.configForm.getValue(section); - if (sectionValue !== null && sectionValue !== '') { + if (sectionValue !== null) { values.push({ section: section, value: sectionValue }); } }); @@ -143,12 +148,28 @@ export class ConfigurationFormComponent extends CdForm implements OnInit { const request = new ConfigFormCreateRequestModel(); request.name = this.configForm.getValue('name'); request.value = values; + if (this.forceUpdate) { + request.force_update = this.forceUpdate; + } return request; } return null; } + openCriticalConfirmModal() { + this.modalService.show(CriticalConfirmationModalComponent, { + buttonText: $localize`Force Edit`, + actionDescription: $localize`force edit`, + itemDescription: $localize`configuration`, + infoMessage: 'Updating this configuration might require restarting the client', + submitAction: () => { + this.modalService.dismissAll(); + this.submit(); + } + }); + } + submit() { const request = this.createRequest(); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts index a57603d4c8a..7a446ce808c 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.ts @@ -12,6 +12,8 @@ import { CdTableSelection } from '~/app/shared/models/cd-table-selection'; import { Permission } from '~/app/shared/models/permissions'; import { AuthStorageService } from '~/app/shared/services/auth-storage.service'; +const RGW = 'rgw'; + @Component({ selector: 'cd-configuration', templateUrl: './configuration.component.html', @@ -26,10 +28,26 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { selection = new CdTableSelection(); filters: CdTableColumn[] = [ { + name: $localize`Modified`, + prop: 'modified', + filterOptions: [$localize`yes`, $localize`no`], + filterInitValue: $localize`yes`, + filterPredicate: (row, value) => { + if (value === 'yes' && row.hasOwnProperty('value')) { + return true; + } + + if (value === 'no' && !row.hasOwnProperty('value')) { + return true; + } + + return false; + } + }, + { name: $localize`Level`, prop: 'level', filterOptions: ['basic', 'advanced', 'dev'], - filterInitValue: 'basic', filterPredicate: (row, value) => { enum Level { basic = 0, @@ -60,22 +78,6 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { } return row.source.includes(value); } - }, - { - name: $localize`Modified`, - prop: 'modified', - filterOptions: ['yes', 'no'], - filterPredicate: (row, value) => { - if (value === 'yes' && row.hasOwnProperty('value')) { - return true; - } - - if (value === 'no' && !row.hasOwnProperty('value')) { - return true; - } - - return false; - } } ]; @@ -143,7 +145,9 @@ export class ConfigurationComponent extends ListWithDetails implements OnInit { if (selection.selected.length !== 1) { return false; } - - return selection.selected[0].can_update_at_runtime; + if ((this.selection.selected[0].name as string).includes(RGW)) { + return true; + } + return this.selection.selected[0].can_update_at_runtime; } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html index b05d07fb31b..79cd60c970f 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.html @@ -77,7 +77,7 @@ [class.icon-danger-color]="row.remainingDays < 2" [class.icon-warning-color]="row.remainingDays < 8" class="{{ icons.warning }}"></i> - <span title="{{ value | cdDate }}">{{ row.remainingTimeWithoutSeconds / 1000 | duration }}</span> + <span title="{{ row.expiryDate }}">{{ row.remainingTimeWithoutSeconds / 1000 | duration }}</span> </span> <span *ngIf="row.remainingTimeWithoutSeconds <= 0 && row.remainingDays <=0 && row.cluster_alias !== 'local-cluster'"> <i i18n-title diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts index 78b4c9c1859..cfdc2e1720e 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/multi-cluster/multi-cluster-list/multi-cluster-list.component.ts @@ -115,6 +115,7 @@ export class MultiClusterListComponent extends ListWithDetails implements OnInit cluster['ttl'] ); cluster['remainingDays'] = this.getRemainingDays(cluster['ttl']); + cluster['expiryDate'] = new Date(Date.now() + cluster['ttl']).toLocaleString(); } }); } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts index 40c2e95d1e0..a07dcfcdd35 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/services/services.component.ts @@ -29,6 +29,7 @@ import { PlacementPipe } from './placement.pipe'; import { ServiceFormComponent } from './service-form/service-form.component'; import { SettingsService } from '~/app/shared/api/settings.service'; import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; +import { CellTemplate } from '~/app/shared/enum/cell-template.enum'; const BASE_URL = 'services'; @@ -176,6 +177,16 @@ export class ServicesComponent extends ListWithDetails implements OnChanges, OnI prop: 'status.last_refresh', pipe: this.relativeDatePipe, flexGrow: 1 + }, + { + name: $localize`Ports`, + prop: 'status.ports', + flexGrow: 1, + cellTransformation: CellTemplate.map, + customTemplateConfig: { + undefined: '-', + '': '-' + } } ]; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket.ts new file mode 100644 index 00000000000..96553c20e91 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-bucket.ts @@ -0,0 +1,37 @@ +export interface Bucket { + bucket: string; + tenant: string; + versioning: string; + zonegroup: string; + placement_rule: string; + explicit_placement: { + data_pool: string; + data_extra_pool: string; + index_pool: string; + }; + id: string; + marker: string; + index_type: string; + index_generation: number; + num_shards: number; + reshard_status: string; + judge_reshard_lock_time: string; + object_lock_enabled: boolean; + mfa_enabled: boolean; + owner: string; + ver: string; + master_ver: string; + mtime: string; + creation_time: string; + max_marker: string; + usage: Record<string, any>; + bucket_quota: { + enabled: boolean; + check_on_raw: boolean; + max_size: number; + max_size_kb: number; + max_objects: number; + }; + read_tracker: number; + bid: string; +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts index 3e146ef7ad5..f96ec0ccffa 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/models/rgw-multisite-zonegroup-deletion-form/rgw-multisite-zonegroup-deletion-form.component.ts @@ -56,7 +56,7 @@ export class RgwMultisiteZonegroupDeletionFormComponent implements OnInit, After .subscribe(() => { this.notificationService.show( NotificationType.success, - $localize`Zone: '${this.zonegroup.name}' deleted successfully` + $localize`Zonegroup: '${this.zonegroup.name}' deleted successfully` ); this.activeModal.close(); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts index 9cb8b52ee0e..b7400219120 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-bucket-list/rgw-bucket-list.component.ts @@ -1,7 +1,8 @@ -import { Component, NgZone, OnInit, TemplateRef, ViewChild } from '@angular/core'; +import { Component, NgZone, OnDestroy, OnInit, TemplateRef, ViewChild } from '@angular/core'; import _ from 'lodash'; -import { forkJoin as observableForkJoin, Observable, Subscriber } from 'rxjs'; +import { forkJoin as observableForkJoin, Observable, Subscriber, Subscription } from 'rxjs'; +import { switchMap } from 'rxjs/operators'; import { RgwBucketService } from '~/app/shared/api/rgw-bucket.service'; import { ListWithDetails } from '~/app/shared/classes/list-with-details.class'; @@ -21,6 +22,7 @@ import { AuthStorageService } from '~/app/shared/services/auth-storage.service'; import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; import { TaskWrapperService } from '~/app/shared/services/task-wrapper.service'; import { URLBuilderService } from '~/app/shared/services/url-builder.service'; +import { Bucket } from '../models/rgw-bucket'; const BASE_URL = 'rgw/bucket'; @@ -30,7 +32,7 @@ const BASE_URL = 'rgw/bucket'; styleUrls: ['./rgw-bucket-list.component.scss'], providers: [{ provide: URLBuilderService, useValue: new URLBuilderService(BASE_URL) }] }) -export class RgwBucketListComponent extends ListWithDetails implements OnInit { +export class RgwBucketListComponent extends ListWithDetails implements OnInit, OnDestroy { @ViewChild(TableComponent, { static: true }) table: TableComponent; @ViewChild('bucketSizeTpl', { static: true }) @@ -43,9 +45,10 @@ export class RgwBucketListComponent extends ListWithDetails implements OnInit { permission: Permission; tableActions: CdTableAction[]; columns: CdTableColumn[] = []; - buckets: object[] = []; + buckets: Bucket[] = []; selection: CdTableSelection = new CdTableSelection(); declare staleTimeout: number; + private subs: Subscription = new Subscription(); constructor( private authStorageService: AuthStorageService, @@ -97,6 +100,11 @@ export class RgwBucketListComponent extends ListWithDetails implements OnInit { prop: 'object_usage', cellTemplate: this.bucketObjectTpl, flexGrow: 0.8 + }, + { + name: $localize`Number of Shards`, + prop: 'num_shards', + flexGrow: 0.8 } ]; const getBucketUri = () => @@ -126,33 +134,18 @@ export class RgwBucketListComponent extends ListWithDetails implements OnInit { this.setTableRefreshTimeout(); } - transformBucketData() { - _.forEach(this.buckets, (bucketKey) => { - const maxBucketSize = bucketKey['bucket_quota']['max_size']; - const maxBucketObjects = bucketKey['bucket_quota']['max_objects']; - bucketKey['bucket_size'] = 0; - bucketKey['num_objects'] = 0; - if (!_.isEmpty(bucketKey['usage'])) { - bucketKey['bucket_size'] = bucketKey['usage']['rgw.main']['size_actual']; - bucketKey['num_objects'] = bucketKey['usage']['rgw.main']['num_objects']; - } - bucketKey['size_usage'] = - maxBucketSize > 0 ? bucketKey['bucket_size'] / maxBucketSize : undefined; - bucketKey['object_usage'] = - maxBucketObjects > 0 ? bucketKey['num_objects'] / maxBucketObjects : undefined; - }); - } - getBucketList(context: CdTableFetchDataContext) { this.setTableRefreshTimeout(); - this.rgwBucketService.list(true).subscribe( - (resp: object[]) => { - this.buckets = resp; - this.transformBucketData(); - }, - () => { - context.error(); - } + this.subs.add( + this.rgwBucketService + .fetchAndTransformBuckets() + .pipe(switchMap(() => this.rgwBucketService.buckets$)) + .subscribe({ + next: (buckets) => { + this.buckets = buckets; + }, + error: () => context.error() + }) ); } @@ -198,4 +191,8 @@ export class RgwBucketListComponent extends ListWithDetails implements OnInit { } }); } + + ngOnDestroy() { + this.subs.unsubscribe(); + } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html index 51f72dd7f89..9117e71c34b 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.html @@ -111,36 +111,21 @@ </div> <div class="form-group row"> <label class="cd-col-form-label required" - for="access_key" - i18n>S3 access key - <cd-helper> - <span>To see or copy your S3 access key, go to <b>Object Gateway > Users</b> and click on your user name. In <b>Keys</b>, click <b>Show</b>. View the access key by clicking Show and copy the key by clicking <b>Copy to Clipboard</b>.</span> - </cd-helper> - </label> + for="username" + i18n>Username</label> <div class="cd-col-form-input"> <input class="form-control" type="text" - placeholder="e.g." - id="access_key" - name="access_key" - formControlName="access_key"> - </div> - </div> - <div class="form-group row"> - <label class="cd-col-form-label required" - for="access_key" - i18n>S3 secret key - <cd-helper> - <span>To see or copy your S3 access key, go to <b>Object Gateway > Users</b> and click on your user name. In <b>Keys</b>, click <b>Show</b>. View the secret key by clicking Show and copy the key by clicking <b>Copy to Clipboard</b>.</span> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="e.g." - id="secret_key" - name="secret_key" - formControlName="secret_key"> + placeholder="username" + id="username" + name="username" + formControlName="username"> + <cd-help-text> + <span i18n>Specify the username for the system user. This user will be created automatically as part of the process.</span> + </cd-help-text> + <span class="invalid-feedback" + *ngIf="multisiteMigrateForm.showError('username', formDir, 'required')" + i18n>This field is required.</span> </div> </div> </div> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts index 1073dee429a..d9ad56a5bf4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-migrate/rgw-multisite-migrate.component.ts @@ -11,7 +11,7 @@ import { NotificationType } from '~/app/shared/enum/notification-type.enum'; import { CdFormGroup } from '~/app/shared/forms/cd-form-group'; import { CdValidators } from '~/app/shared/forms/cd-validators'; import { NotificationService } from '~/app/shared/services/notification.service'; -import { RgwRealm, RgwZone, RgwZonegroup, SystemKey } from '../models/rgw-multisite'; +import { RgwRealm, RgwZone, RgwZonegroup } from '../models/rgw-multisite'; import { ModalService } from '~/app/shared/services/modal.service'; import { RgwDaemonService } from '~/app/shared/api/rgw-daemon.service'; @@ -135,8 +135,9 @@ export class RgwMultisiteMigrateComponent implements OnInit { Validators.required ] ), - access_key: new UntypedFormControl(null), - secret_key: new UntypedFormControl(null) + username: new UntypedFormControl(null, { + validators: [Validators.required] + }) }); } @@ -174,21 +175,21 @@ export class RgwMultisiteMigrateComponent implements OnInit { this.zone = new RgwZone(); this.zone.name = values['zoneName']; this.zone.endpoints = values['zone_endpoints']; - this.zone.system_key = new SystemKey(); - this.zone.system_key.access_key = values['access_key']; - this.zone.system_key.secret_key = values['secret_key']; - this.rgwMultisiteService.migrate(this.realm, this.zonegroup, this.zone).subscribe( - () => { - this.notificationService.show( - NotificationType.success, - $localize`Migration done successfully` - ); - this.submitAction.emit(); - this.activeModal.close(); - }, - () => { - this.notificationService.show(NotificationType.error, $localize`Migration failed`); - } - ); + this.rgwMultisiteService + .migrate(this.realm, this.zonegroup, this.zone, values['username']) + .subscribe( + () => { + this.rgwMultisiteService.setRestartGatewayMessage(false); + this.notificationService.show( + NotificationType.success, + $localize`Migration done successfully` + ); + this.submitAction.emit(); + this.activeModal.close(); + }, + () => { + this.notificationService.show(NotificationType.error, $localize`Migration failed`); + } + ); } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-sync-policy-form/rgw-multisite-sync-policy-form.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-sync-policy-form/rgw-multisite-sync-policy-form.component.html index b9c6b2c9651..511eaaa6526 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-sync-policy-form/rgw-multisite-sync-policy-form.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-multisite-sync-policy-form/rgw-multisite-sync-policy-form.component.html @@ -86,6 +86,7 @@ <div class="text-right"> <cd-form-button-panel (submitActionEvent)="submit()" [form]="syncPolicyForm" + [disabled]="syncPolicyForm.pending || syncPolicyForm.pristine || syncPolicyForm.invalid" [submitText]="(action | titlecase) + ' ' + (resource | upperFirst)"></cd-form-button-panel> </div> </div> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html index 3a9ce12df9d..16963b06920 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.html @@ -60,6 +60,7 @@ </cd-dashboard-area-chart> <cd-dashboard-area-chart chartTitle="Latency" dataUnits="ms" + decimals="2" [labelsArray]="['GET', 'PUT']" [dataArray]="[queriesResults.AVG_GET_LATENCY, queriesResults.AVG_PUT_LATENCY]"> </cd-dashboard-area-chart> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.spec.ts index 36cafa855a3..c7aaddcd08f 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.spec.ts @@ -1,24 +1,33 @@ -import { ComponentFixture, TestBed } from '@angular/core/testing'; - +import { ComponentFixture, TestBed, fakeAsync, tick } from '@angular/core/testing'; +import { of, BehaviorSubject, combineLatest } from 'rxjs'; import { RgwOverviewDashboardComponent } from './rgw-overview-dashboard.component'; -import { of } from 'rxjs'; +import { HttpClientTestingModule } from '@angular/common/http/testing'; +import { RgwBucketService } from '~/app/shared/api/rgw-bucket.service'; import { RgwDaemonService } from '~/app/shared/api/rgw-daemon.service'; import { RgwDaemon } from '../models/rgw-daemon'; -import { HttpClientTestingModule } from '@angular/common/http/testing'; +import { CardComponent } from '~/app/shared/components/card/card.component'; +import { CardRowComponent } from '~/app/shared/components/card-row/card-row.component'; import { DimlessBinaryPipe } from '~/app/shared/pipes/dimless-binary.pipe'; +import { NO_ERRORS_SCHEMA } from '@angular/core'; import { RgwRealmService } from '~/app/shared/api/rgw-realm.service'; -import { RgwZonegroupService } from '~/app/shared/api/rgw-zonegroup.service'; import { RgwZoneService } from '~/app/shared/api/rgw-zone.service'; -import { RgwBucketService } from '~/app/shared/api/rgw-bucket.service'; -import { HealthService } from '~/app/shared/api/health.service'; -import { CardRowComponent } from '~/app/shared/components/card-row/card-row.component'; -import { CardComponent } from '~/app/shared/components/card/card.component'; -import { NO_ERRORS_SCHEMA } from '@angular/core'; -import { configureTestBed } from '~/testing/unit-test-helper'; +import { RgwZonegroupService } from '~/app/shared/api/rgw-zonegroup.service'; describe('RgwOverviewDashboardComponent', () => { let component: RgwOverviewDashboardComponent; let fixture: ComponentFixture<RgwOverviewDashboardComponent>; + let listDaemonsSpy: jest.SpyInstance; + let listRealmsSpy: jest.SpyInstance; + let listZonegroupsSpy: jest.SpyInstance; + let listZonesSpy: jest.SpyInstance; + let fetchAndTransformBucketsSpy: jest.SpyInstance; + let totalBucketsAndUsersSpy: jest.SpyInstance; + + const totalNumObjectsSubject = new BehaviorSubject<number>(290); + const totalUsedCapacitySubject = new BehaviorSubject<number>(9338880); + const averageObjectSizeSubject = new BehaviorSubject<number>(1280); + const bucketsCount = 2; + const usersCount = 5; const daemon: RgwDaemon = { id: '8000', service_map_id: '4803', @@ -47,38 +56,44 @@ describe('RgwOverviewDashboardComponent', () => { zones: ['zone4', 'zone5', 'zone6', 'zone7'] }; - const bucketAndUserList = { - buckets_count: 2, - users_count: 2 - }; - - const healthData = { - total_objects: '290', - total_pool_bytes_used: 9338880 - }; - - let listDaemonsSpy: jest.SpyInstance; - let listZonesSpy: jest.SpyInstance; - let listZonegroupsSpy: jest.SpyInstance; - let listRealmsSpy: jest.SpyInstance; - let listBucketsSpy: jest.SpyInstance; - let healthDataSpy: jest.SpyInstance; - - configureTestBed({ - declarations: [ - RgwOverviewDashboardComponent, - CardComponent, - CardRowComponent, - DimlessBinaryPipe - ], - schemas: [NO_ERRORS_SCHEMA], - imports: [HttpClientTestingModule] - }); - beforeEach(() => { + TestBed.configureTestingModule({ + declarations: [ + RgwOverviewDashboardComponent, + CardComponent, + CardRowComponent, + DimlessBinaryPipe + ], + schemas: [NO_ERRORS_SCHEMA], + providers: [ + { provide: RgwDaemonService, useValue: { list: jest.fn() } }, + { provide: RgwRealmService, useValue: { list: jest.fn() } }, + { provide: RgwZonegroupService, useValue: { list: jest.fn() } }, + { provide: RgwZoneService, useValue: { list: jest.fn() } }, + { + provide: RgwBucketService, + useValue: { + fetchAndTransformBuckets: jest.fn(), + totalNumObjects$: totalNumObjectsSubject.asObservable(), + totalUsedCapacity$: totalUsedCapacitySubject.asObservable(), + averageObjectSize$: averageObjectSizeSubject.asObservable(), + getTotalBucketsAndUsersLength: jest.fn() + } + } + ], + imports: [HttpClientTestingModule] + }).compileComponents(); + fixture = TestBed.createComponent(RgwOverviewDashboardComponent); + component = fixture.componentInstance; listDaemonsSpy = jest .spyOn(TestBed.inject(RgwDaemonService), 'list') .mockReturnValue(of([daemon])); + fetchAndTransformBucketsSpy = jest + .spyOn(TestBed.inject(RgwBucketService), 'fetchAndTransformBuckets') + .mockReturnValue(of(null)); + totalBucketsAndUsersSpy = jest + .spyOn(TestBed.inject(RgwBucketService), 'getTotalBucketsAndUsersLength') + .mockReturnValue(of({ buckets_count: bucketsCount, users_count: usersCount })); listRealmsSpy = jest .spyOn(TestBed.inject(RgwRealmService), 'list') .mockReturnValue(of(realmList)); @@ -86,56 +101,60 @@ describe('RgwOverviewDashboardComponent', () => { .spyOn(TestBed.inject(RgwZonegroupService), 'list') .mockReturnValue(of(zonegroupList)); listZonesSpy = jest.spyOn(TestBed.inject(RgwZoneService), 'list').mockReturnValue(of(zoneList)); - listBucketsSpy = jest - .spyOn(TestBed.inject(RgwBucketService), 'getTotalBucketsAndUsersLength') - .mockReturnValue(of(bucketAndUserList)); - healthDataSpy = jest - .spyOn(TestBed.inject(HealthService), 'getClusterCapacity') - .mockReturnValue(of(healthData)); - fixture = TestBed.createComponent(RgwOverviewDashboardComponent); - component = fixture.componentInstance; fixture.detectChanges(); }); - it('should create', () => { + it('should create the component', () => { expect(component).toBeTruthy(); }); it('should render all cards', () => { - fixture.detectChanges(); const dashboardCards = fixture.debugElement.nativeElement.querySelectorAll('cd-card'); expect(dashboardCards.length).toBe(5); }); - it('should get corresponding data into Daemons', () => { - expect(listDaemonsSpy).toHaveBeenCalled(); - expect(component.rgwDaemonCount).toEqual(1); - }); - - it('should get corresponding data into Realms', () => { + it('should get data for Realms', () => { expect(listRealmsSpy).toHaveBeenCalled(); expect(component.rgwRealmCount).toEqual(2); }); - it('should get corresponding data into Zonegroups', () => { + it('should get data for Zonegroups', () => { expect(listZonegroupsSpy).toHaveBeenCalled(); expect(component.rgwZonegroupCount).toEqual(3); }); - it('should get corresponding data into Zones', () => { + it('should get data for Zones', () => { expect(listZonesSpy).toHaveBeenCalled(); expect(component.rgwZoneCount).toEqual(4); }); - it('should get corresponding data into Buckets', () => { - expect(listBucketsSpy).toHaveBeenCalled(); - expect(component.rgwBucketCount).toEqual(2); - expect(component.UserCount).toEqual(2); - }); - - it('should get corresponding data into Objects and capacity', () => { - expect(healthDataSpy).toHaveBeenCalled(); - expect(component.objectCount).toEqual('290'); + it('should set component properties from services using combineLatest', fakeAsync(() => { + component.interval = of(null).subscribe(() => { + component.fetchDataSub = combineLatest([ + TestBed.inject(RgwDaemonService).list(), + TestBed.inject(RgwBucketService).fetchAndTransformBuckets(), + totalNumObjectsSubject.asObservable(), + totalUsedCapacitySubject.asObservable(), + averageObjectSizeSubject.asObservable(), + TestBed.inject(RgwBucketService).getTotalBucketsAndUsersLength() + ]).subscribe(([daemonData, _, objectCount, usedCapacity, averageSize, bucketData]) => { + component.rgwDaemonCount = daemonData.length; + component.objectCount = objectCount; + component.totalPoolUsedBytes = usedCapacity; + component.averageObjectSize = averageSize; + component.rgwBucketCount = bucketData.buckets_count; + component.UserCount = bucketData.users_count; + }); + }); + tick(); + expect(listDaemonsSpy).toHaveBeenCalled(); + expect(fetchAndTransformBucketsSpy).toHaveBeenCalled(); + expect(totalBucketsAndUsersSpy).toHaveBeenCalled(); + expect(component.rgwDaemonCount).toEqual(1); + expect(component.objectCount).toEqual(290); expect(component.totalPoolUsedBytes).toEqual(9338880); - }); + expect(component.averageObjectSize).toEqual(1280); + expect(component.rgwBucketCount).toEqual(bucketsCount); + expect(component.UserCount).toEqual(usersCount); + })); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.ts index 00037a7235b..f3a99505e2c 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-overview-dashboard/rgw-overview-dashboard.component.ts @@ -1,7 +1,7 @@ import { Component, OnDestroy, OnInit } from '@angular/core'; import _ from 'lodash'; -import { Observable, ReplaySubject, Subscription, of } from 'rxjs'; +import { Observable, ReplaySubject, Subscription, combineLatest, of } from 'rxjs'; import { Permissions } from '~/app/shared/models/permissions'; import { AuthStorageService } from '~/app/shared/services/auth-storage.service'; @@ -14,7 +14,6 @@ import { RgwBucketService } from '~/app/shared/api/rgw-bucket.service'; import { PrometheusService } from '~/app/shared/api/prometheus.service'; import { RgwPromqls as queries } from '~/app/shared/enum/dashboard-promqls.enum'; -import { HealthService } from '~/app/shared/api/health.service'; import { Icons } from '~/app/shared/enum/icons.enum'; import { RgwMultisiteService } from '~/app/shared/api/rgw-multisite.service'; import { catchError, shareReplay, switchMap, tap } from 'rxjs/operators'; @@ -39,13 +38,10 @@ export class RgwOverviewDashboardComponent implements OnInit, OnDestroy { totalPoolUsedBytes = 0; averageObjectSize = 0; realmData: any; - daemonSub: Subscription; realmSub: Subscription; multisiteInfo: object[] = []; ZonegroupSub: Subscription; ZoneSUb: Subscription; - HealthSub: Subscription; - BucketSub: Subscription; queriesResults: { [key: string]: [] } = { RGW_REQUEST_PER_SECOND: [], BANDWIDTH: [], @@ -65,10 +61,10 @@ export class RgwOverviewDashboardComponent implements OnInit, OnDestroy { multisiteSyncStatus$: Observable<any>; subject = new ReplaySubject<any>(); syncCardLoading = true; + fetchDataSub: Subscription; constructor( private authStorageService: AuthStorageService, - private healthService: HealthService, private refreshIntervalService: RefreshIntervalService, private rgwDaemonService: RgwDaemonService, private rgwRealmService: RgwRealmService, @@ -83,24 +79,23 @@ export class RgwOverviewDashboardComponent implements OnInit, OnDestroy { ngOnInit() { this.interval = this.refreshIntervalService.intervalData$.subscribe(() => { - this.daemonSub = this.rgwDaemonService.list().subscribe((data: any) => { - this.rgwDaemonCount = data.length; - }); - this.HealthSub = this.healthService.getClusterCapacity().subscribe((data: any) => { - this.objectCount = data['total_objects']; - this.totalPoolUsedBytes = data['total_pool_bytes_used']; - this.averageObjectSize = data['average_object_size']; - }); - setTimeout(() => { + this.fetchDataSub = combineLatest([ + this.rgwDaemonService.list(), + this.rgwBucketService.fetchAndTransformBuckets(), + this.rgwBucketService.totalNumObjects$, + this.rgwBucketService.totalUsedCapacity$, + this.rgwBucketService.averageObjectSize$, + this.rgwBucketService.getTotalBucketsAndUsersLength() + ]).subscribe(([daemonData, _, objectCount, usedCapacity, averageSize, bucketData]) => { + this.rgwDaemonCount = daemonData.length; + this.objectCount = objectCount; + this.totalPoolUsedBytes = usedCapacity; + this.averageObjectSize = averageSize; + this.rgwBucketCount = bucketData.buckets_count; + this.UserCount = bucketData.users_count; this.getSyncStatus(); }); }); - this.BucketSub = this.rgwBucketService - .getTotalBucketsAndUsersLength() - .subscribe((data: any) => { - this.rgwBucketCount = data['buckets_count']; - this.UserCount = data['users_count']; - }); this.realmSub = this.rgwRealmService.list().subscribe((data: any) => { this.rgwRealmCount = data['realms'].length; }); @@ -139,14 +134,12 @@ export class RgwOverviewDashboardComponent implements OnInit, OnDestroy { } ngOnDestroy() { - this.interval.unsubscribe(); - this.daemonSub.unsubscribe(); - this.realmSub.unsubscribe(); - this.ZonegroupSub.unsubscribe(); - this.ZoneSUb.unsubscribe(); - this.BucketSub.unsubscribe(); - this.HealthSub.unsubscribe(); - this.prometheusService.unsubscribe(); + this.interval?.unsubscribe(); + this.realmSub?.unsubscribe(); + this.ZonegroupSub?.unsubscribe(); + this.ZoneSUb?.unsubscribe(); + this.fetchDataSub?.unsubscribe(); + this.prometheusService?.unsubscribe(); } getPrometheusData(selectedTime: any) { diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.html b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.html new file mode 100644 index 00000000000..1a73e58cdd2 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.html @@ -0,0 +1,15 @@ +<ng-container *ngIf="smbClusters$ | async as smbClusters"> + <cd-table + #table + [data]="smbClusters" + columnMode="flex" + [columns]="columns" + identifier="id" + forceIdentifier="true" + selectionType="single" + [hasDetails]="false" + (setExpandedRow)="setExpandedRow($event)" + (fetchData)="loadSMBCluster($event)" + > + </cd-table> +</ng-container> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.scss b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.scss new file mode 100644 index 00000000000..e69de29bb2d --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.scss diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.spec.ts new file mode 100644 index 00000000000..d1c24d1dbe1 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.spec.ts @@ -0,0 +1,35 @@ +import { ComponentFixture, TestBed } from '@angular/core/testing'; + +import { SmbClusterListComponent } from './smb-cluster-list.component'; +import { BrowserAnimationsModule } from '@angular/platform-browser/animations'; +import { SharedModule } from '~/app/shared/shared.module'; +import { HttpClientTestingModule } from '@angular/common/http/testing'; +import { RouterTestingModule } from '@angular/router/testing'; + +import { ToastrModule } from 'ngx-toastr'; + +describe('SmbClusterListComponent', () => { + let component: SmbClusterListComponent; + let fixture: ComponentFixture<SmbClusterListComponent>; + + beforeEach(async () => { + await TestBed.configureTestingModule({ + imports: [ + BrowserAnimationsModule, + SharedModule, + HttpClientTestingModule, + ToastrModule.forRoot(), + RouterTestingModule + ], + declarations: [SmbClusterListComponent] + }).compileComponents(); + + fixture = TestBed.createComponent(SmbClusterListComponent); + component = fixture.componentInstance; + fixture.detectChanges(); + }); + + it('should create', () => { + expect(component).toBeTruthy(); + }); +}); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.ts new file mode 100644 index 00000000000..bf61643a0cc --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb-cluster-list/smb-cluster-list.component.ts @@ -0,0 +1,73 @@ +import { Component, OnInit, ViewChild } from '@angular/core'; +import { catchError, switchMap } from 'rxjs/operators'; +import { BehaviorSubject, Observable, of } from 'rxjs'; + +import _ from 'lodash'; + +import { ActionLabelsI18n } from '~/app/shared/constants/app.constants'; +import { TableComponent } from '~/app/shared/datatable/table/table.component'; +import { CdTableAction } from '~/app/shared/models/cd-table-action'; +import { CdTableColumn } from '~/app/shared/models/cd-table-column'; +import { CdTableFetchDataContext } from '~/app/shared/models/cd-table-fetch-data-context'; +import { ListWithDetails } from '~/app/shared/classes/list-with-details.class'; +import { Permission } from '~/app/shared/models/permissions'; + +import { AuthStorageService } from '~/app/shared/services/auth-storage.service'; +import { SmbService } from '~/app/shared/api/smb.service'; +import { SMBCluster } from '../smb.model'; + +@Component({ + selector: 'cd-smb-cluster-list', + templateUrl: './smb-cluster-list.component.html', + styleUrls: ['./smb-cluster-list.component.scss'] +}) +export class SmbClusterListComponent extends ListWithDetails implements OnInit { + @ViewChild('table', { static: true }) + table: TableComponent; + columns: CdTableColumn[]; + permission: Permission; + tableActions: CdTableAction[]; + context: CdTableFetchDataContext; + + smbClusters$: Observable<SMBCluster[]>; + subject$ = new BehaviorSubject<SMBCluster[]>([]); + + constructor( + private authStorageService: AuthStorageService, + public actionLabels: ActionLabelsI18n, + private smbService: SmbService + ) { + super(); + this.permission = this.authStorageService.getPermissions().smb; + } + + ngOnInit() { + this.columns = [ + { + name: $localize`Name`, + prop: 'cluster_id', + flexGrow: 2 + }, + { + name: $localize`Authentication Mode`, + prop: 'auth_mode', + flexGrow: 2 + } + ]; + + this.smbClusters$ = this.subject$.pipe( + switchMap(() => + this.smbService.listClusters().pipe( + catchError(() => { + this.context.error(); + return of(null); + }) + ) + ) + ); + } + + loadSMBCluster() { + this.subject$.next([]); + } +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.model.ts new file mode 100644 index 00000000000..3796d924565 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.model.ts @@ -0,0 +1,28 @@ +import { CephServicePlacement } from '~/app/shared/models/service.interface'; + +export interface SMBCluster { + cluster_id: string; + auth_mode: AuthMode; + intent: string; + domain_settings?: DomainSettings; + user_group_settings?: string[]; + custom_dns?: string[]; + placement?: CephServicePlacement; + clustering?: string; + public_addrs?: PublicAddress; +} + +export interface DomainSettings { + realm?: string; + join_sources_ref?: string[]; +} + +export interface PublicAddress { + address: string; + destination: string; +} + +export interface AuthMode { + user: 'User'; + activeDirectory: 'active-directory'; +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.module.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.module.ts new file mode 100644 index 00000000000..7cd237dd8e0 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/smb/smb.module.ts @@ -0,0 +1,44 @@ +import { CommonModule } from '@angular/common'; +import { NgModule } from '@angular/core'; +import { ReactiveFormsModule } from '@angular/forms'; +import { RouterModule } from '@angular/router'; + +import { NgbNavModule, NgbTooltipModule, NgbTypeaheadModule } from '@ng-bootstrap/ng-bootstrap'; + +import { SharedModule } from '~/app/shared/shared.module'; + +import { + ButtonModule, + GridModule, + IconModule, + IconService, + InputModule, + SelectModule +} from 'carbon-components-angular'; + +import Close from '@carbon/icons/es/close/32'; +import { SmbClusterListComponent } from './smb-cluster-list/smb-cluster-list.component'; + +@NgModule({ + imports: [ + ReactiveFormsModule, + RouterModule, + SharedModule, + NgbNavModule, + CommonModule, + NgbTypeaheadModule, + NgbTooltipModule, + GridModule, + SelectModule, + InputModule, + ButtonModule, + IconModule + ], + exports: [SmbClusterListComponent], + declarations: [SmbClusterListComponent] +}) +export class SmbModule { + constructor(private iconService: IconService) { + this.iconService.registerAll([Close]); + } +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts index c0e0517896c..f1f04f7c2f0 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/auth.module.ts @@ -17,13 +17,48 @@ import { UserFormComponent } from './user-form/user-form.component'; import { UserListComponent } from './user-list/user-list.component'; import { UserPasswordFormComponent } from './user-password-form/user-password-form.component'; import { UserTabsComponent } from './user-tabs/user-tabs.component'; -import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-components-angular'; + +import { + ButtonModule, + CheckboxModule, + DatePickerModule, + GridModule, + IconModule, + IconService, + InputModule, + ModalModule, + NumberModule, + RadioModule, + SelectModule, + UIShellModule, + TimePickerModule, + ComboBoxModule +} from 'carbon-components-angular'; +// Icons +import ChevronDown from '@carbon/icons/es/chevron--down/16'; +import Close from '@carbon/icons/es/close/32'; +import AddFilled from '@carbon/icons/es/add--filled/32'; +import SubtractFilled from '@carbon/icons/es/subtract--filled/32'; +import Reset from '@carbon/icons/es/reset/32'; +import EyeIcon from '@carbon/icons/es/view/16'; @NgModule({ imports: [ CommonModule, FormsModule, ReactiveFormsModule, SharedModule, + UIShellModule, + InputModule, + GridModule, + ButtonModule, + IconModule, + CheckboxModule, + RadioModule, + SelectModule, + NumberModule, + ModalModule, + DatePickerModule, + TimePickerModule, NgbNavModule, NgbPopoverModule, NgxPipeFunctionModule, @@ -31,8 +66,8 @@ import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-compon NgbModule, IconModule, GridModule, - ButtonModule, - InputModule + InputModule, + ComboBoxModule ], declarations: [ LoginComponent, @@ -46,7 +81,11 @@ import { ButtonModule, GridModule, IconModule, InputModule } from 'carbon-compon UserPasswordFormComponent ] }) -export class AuthModule {} +export class AuthModule { + constructor(private iconService: IconService) { + this.iconService.registerAll([ChevronDown, Close, AddFilled, SubtractFilled, Reset, EyeIcon]); + } +} const routes: Routes = [ { path: '', redirectTo: 'users', pathMatch: 'full' }, diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts index 2d323b04ea5..abf529196f6 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form-role.model.ts @@ -4,11 +4,12 @@ export class UserFormRoleModel implements SelectOption { name: string; description: string; selected = false; - scopes_permissions: object; - enabled = true; - - constructor(name: string, description: string) { + scopes_permissions?: object; + enabled: boolean; + content: string; + constructor(name: string, description: string, content: string) { this.name = name; this.description = description; + this.content = content; } } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html index 4169d54c39f..d2e52158473 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html @@ -1,249 +1,205 @@ -<div class="cd-col-form" - *cdFormLoading="loading"> - <form name="userForm" - #formDir="ngForm" - [formGroup]="userForm" - novalidate> - <div class="card"> +<div cdsCol + [columnNumbers]="{md: 4}"> + <ng-container *cdFormLoading="loading"> + <form #frm="ngForm" + #formDir="ngForm" + [formGroup]="userForm" + novalidate> <div i18n="form title" - class="card-header">{{ action | titlecase }} {{ resource | upperFirst }}</div> - <div class="card-body"> - - <!-- Username --> - <div class="form-group row"> - <label class="cd-col-form-label" - [ngClass]="{'required': mode !== userFormMode.editing}" - for="username" - i18n>Username</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="Username..." - id="username" - name="username" - formControlName="username" - autocomplete="off" - autofocus - ngbTooltip="White spaces at the beginning and end will be trimmed" - i18n-ngbTooltip - cdTrim> - <span class="invalid-feedback" - *ngIf="userForm.showError('username', formDir, 'required')" - i18n>This field is required.</span> - <span class="invalid-feedback" - *ngIf="userForm.showError('username', formDir, 'notUnique')" - i18n>The username already exists.</span> - </div> - </div> - - <!-- Password --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label class="cd-col-form-label" - for="password"> - <ng-container i18n>Password</ng-container> - <cd-helper *ngIf="passwordPolicyHelpText.length > 0" - class="text-pre-wrap" - html="{{ passwordPolicyHelpText }}"> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - type="password" - placeholder="Password..." - id="password" - name="password" - autocomplete="new-password" - formControlName="password"> - <button type="button" - class="btn btn-light" - cdPasswordButton="password"> - </button> - </div> - <div class="password-strength-level"> - <div class="{{ passwordStrengthLevelClass }}" - data-toggle="tooltip" - title="{{ passwordValuation }}"> - </div> - </div> - <span class="invalid-feedback" - *ngIf="userForm.showError('password', formDir, 'required')" - i18n>This field is required.</span> - <span class="invalid-feedback" - *ngIf="userForm.showError('password', formDir, 'passwordPolicy')"> - {{ passwordValuation }} - </span> - </div> - </div> - - <!-- Confirm password --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label i18n - class="cd-col-form-label" - for="confirmpassword">Confirm password</label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - type="password" - placeholder="Confirm password..." - id="confirmpassword" - name="confirmpassword" - autocomplete="new-password" - formControlName="confirmpassword"> - <button type="button" - class="btn btn-light" - cdPasswordButton="confirmpassword"> - </button> - <span class="invalid-feedback" - *ngIf="userForm.showError('confirmpassword', formDir, 'match')" - i18n>Password confirmation doesn't match the password.</span> - </div> - <span class="invalid-feedback" - *ngIf="userForm.showError('confirmpassword', formDir, 'required')" - i18n>This field is required.</span> - </div> - </div> + class="form-header">{{ action | titlecase }} {{ resource | upperFirst }} + </div> + <!-- UserName --> + <div class="form-item"> + <cds-text-label labelInputID="username" + cdRequiredField="Username" + [invalid]="!userForm.controls.username.valid && userForm.controls.username.dirty" + [invalidText]="usernameError" + i18n>Username + <input cdsText + placeholder="Username..." + i18n-placeholder + id="username" + formControlName="username" + [invalid]="!userForm.controls.username.valid && userForm.controls.username.dirty" + autofocus + ngbTooltip="White spaces at the beginning and end will be trimmed" + i18n-ngbTooltip + cdTrim> + </cds-text-label> + <ng-template #usernameError> + <span *ngIf="userForm.showError('username', formDir, 'required')"> + <ng-container i18n> + This field is required. + </ng-container> + </span> + <span *ngIf="userForm.showError('username', formDir, 'notUnique')"> + <ng-container i18n> + The username already exists. + </ng-container> + </span> + </ng-template> + </div> + <!-- Password --> + <div class="form-item"> + <cds-password-label labelInputID="password" + label="Password..." + [invalid]="!userForm.controls.password.valid && userForm.controls.password.dirty" + [invalidText]="passwordError" + i18n>Password + <cd-helper *ngIf="passwordPolicyHelpText.length > 0" + class="text-pre-wrap" + html="{{ passwordPolicyHelpText }}"> + </cd-helper> + <input cdsPassword + type="password" + placeholder="Password..." + id="password" + autocomplete="new-password" + formControlName="password" + > + </cds-password-label> + <ng-template #passwordError> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'match')" + i18n>Password confirmation doesn't match the password. + </span> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'required')" + i18n>This field is required.</span> + <span class="invalid-feedback" + *ngIf="userForm.showError('password', formDir, 'passwordPolicy')"> + {{ passwordValuation }} + </span> + </ng-template> + </div> - <!-- Password expiration date --> - <div class="form-group row" - *ngIf="!authStorageService.isSSO()"> - <label class="cd-col-form-label" - [ngClass]="{'required': pwdExpirationSettings.pwdExpirationSpan > 0}" - for="pwdExpirationDate"> - <ng-container i18n>Password expiration date</ng-container> - <cd-helper class="text-pre-wrap" - *ngIf="pwdExpirationSettings.pwdExpirationSpan == 0"> - <p> + <!-- Confirm password --> + <div class="form-item"> + <cds-password-label labelInputID="confirmpassword" + label="Confirm password..." + [invalid]="!userForm.controls.confirmpassword.valid && userForm.controls.confirmpassword.dirty" + [invalidText]="confirmpasswordError" + i18n> Confirm password + <input cdsPassword + type="password" + placeholder="Confirm password..." + id="confirmpassword" + formControlName="confirmpassword"> + </cds-password-label> + <ng-template #confirmpasswordError> + <span class="invalid-feedback" + *ngIf="userForm.showError('confirmpassword', formDir, 'match')" + i18n>Password confirmation doesn't match the password.</span> + <span class="invalid-feedback" + *ngIf="userForm.showError('confirmpassword', formDir, 'required')" + i18n>This field is required.</span> + </ng-template> + </div> + <!-- Password expiration date --> + <div class="form-item" + *ngIf="!authStorageService.isSSO()"> + <cds-text-label [ngClass]="{'required': pwdExpirationSettings.pwdExpirationSpan > 0}">{{'Password Expiration Date'}} + <cd-helper class="text-pre-wrap" + *ngIf="pwdExpirationSettings.pwdExpirationSpan == 0"> + <span> The Dashboard setting defining the expiration interval of passwords is currently set to <strong>0</strong>. This means if a date is set, the user password will only expire once. - </p> - <p> - Consider configuring the Dashboard setting - <a routerLink="/mgr-modules/edit/dashboard" - class="alert-link">USER_PWD_EXPIRATION_SPAN</a> - in order to let passwords expire periodically. - </p> - </cd-helper> - </label> - <div class="cd-col-form-input"> - <div class="input-group"> - <input class="form-control" - i18n-placeholder - placeholder="Password expiration date..." - id="pwdExpirationDate" - name="pwdExpirationDate" - formControlName="pwdExpirationDate" - [ngbPopover]="popContent" - triggers="manual" - #p="ngbPopover" - (click)="p.open()" - (keypress)="p.close()"> - <button type="button" - class="btn btn-light" - (click)="clearExpirationDate()"> - <i class="icon-prepend {{ icons.destroy }}"></i> - </button> - <span class="invalid-feedback" - *ngIf="userForm.showError('pwdExpirationDate', formDir, 'required')" - i18n>This field is required.</span> - </div> - </div> - </div> - - <!-- Name --> - <div class="form-group row"> - <label i18n - class="cd-col-form-label" - for="name">Full name</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="text" - placeholder="Full name..." - id="name" - name="name" - formControlName="name"> - </div> - </div> - - <!-- Email --> - <div class="form-group row"> - <label i18n - class="cd-col-form-label" - for="email">Email</label> - <div class="cd-col-form-input"> - <input class="form-control" - type="email" - placeholder="Email..." - id="email" - name="email" - formControlName="email"> - - <span class="invalid-feedback" - *ngIf="userForm.showError('email', formDir, 'email')" - i18n>Invalid email.</span> - </div> - </div> - - <!-- Roles --> - <div class="form-group row"> - <label class="cd-col-form-label" - i18n>Roles</label> - <div class="cd-col-form-input"> - <span class="no-border full-height" - *ngIf="allRoles"> - <cd-select-badges [data]="userForm.controls.roles.value" - [options]="allRoles" - [messages]="messages"></cd-select-badges> </span> - </div> - </div> - - <!-- Enabled --> - <div class="form-group row" - *ngIf="!isCurrentUser()"> - <div class="cd-col-form-offset"> - <div class="custom-control custom-checkbox"> - <input type="checkbox" - class="custom-control-input" - id="enabled" - name="enabled" - formControlName="enabled"> - <label class="custom-control-label" - for="enabled" - i18n>Enabled</label> - </div> - </div> - </div> - - <!-- Force change password --> - <div class="form-group row" - *ngIf="!isCurrentUser() && !authStorageService.isSSO()"> - <div class="cd-col-form-offset"> - <div class="custom-control custom-checkbox"> - <input type="checkbox" - class="custom-control-input" - id="pwdUpdateRequired" - name="pwdUpdateRequired" - formControlName="pwdUpdateRequired"> - <label class="custom-control-label" - for="pwdUpdateRequired" - i18n>User must change password at next logon</label> - </div> - </div> - </div> - + <span>Consider configuring the Dashboard setting + <a routerLink="/mgr-modules/edit/dashboard" + class="alert-link">USER_PWD_EXPIRATION_SPAN</a> + in order to let passwords expire periodically. + </span> + </cd-helper> + <cd-date-time-picker [control]="userForm.get('pwdExpirationDate')" + placeHolder="Password expiration date" + [hasTime]="false" + [defaultDate]="passwordexp" + i18n-name + > + </cd-date-time-picker> + </cds-text-label> + <span class="invalid-feedback" + *ngIf="userForm.showError('pwdExpirationDate', formDir, 'required')" + i18n>This field is required. + </span> + </div> + <!--Full Name--> + <div class="form-item"> + <cds-text-label for="name" + i18n> Full Name + <input cdsText + type="text" + placeholder="Full Name..." + id="name" + formControlName="name"> + </cds-text-label> + </div> + <!-- Email --> + <div class="form-item"> + <cds-text-label for="email" + [invalid]="!userForm.controls.email.valid && userForm.controls.email.dirty" + [invalidText]="emailError" + i18n> + Email + <input cdsText + type="email" + placeholder="Email..." + id="email" + formControlName="email"> + </cds-text-label> + <ng-template #emailError> + <span class="invalid-feedback" + *ngIf="userForm.showError('email', formDir, 'email')" + i18n>Invalid email. + </span> + </ng-template> + </div> + <!-- Roles --> + <div class="form-item" + *ngIf="allRoles"> + <cds-combo-box label="Roles" + type="multi" + selectionFeedback="top-after-reopen" + for="roles" + formControlName="roles" + id="roles" + placeholder="Select Roles..." + i18n-placeholder + [appendInline]="true" + [items]="allRoles" + itemValueKey="name" + i18n> + <cds-dropdown-list></cds-dropdown-list> + </cds-combo-box> + </div> + <!-- Enabled --> + <div class="form-item" + *ngIf="!isCurrentUser()"> + <cds-checkbox id="enabled" + formControlName="enabled" + name="enabled" + i18n>Enabled + </cds-checkbox> </div> - <div class="card-footer"> - <cd-form-button-panel (submitActionEvent)="submit()" - [form]="userForm" - [submitText]="(action | titlecase) + ' ' + (resource | upperFirst)" - wrappingClass="text-right"></cd-form-button-panel> + <!-- Force change password --> + <div class="form-item" + *ngIf="!isCurrentUser() && !authStorageService.isSSO()"> + <cds-checkbox id="pwdUpdateRequired" + formControlName="pwdUpdateRequired" + name="pwdUpdateRequired" + i18n>User must change password at next logon + </cds-checkbox> </div> - </div> - </form> + <!--Submit Button--> + <cd-form-button-panel (submitActionEvent)="submit()" + [form]="userForm" + [submitText]="(action | titlecase) + ' ' + (resource | upperFirst)" + wrappingClass="text-right"> + </cd-form-button-panel> + </form> + </ng-container> </div> <ng-template #removeSelfUserReadUpdatePermissionTpl> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts index 7c02b86eae0..009d4c193e4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.ts @@ -55,7 +55,8 @@ export class UserFormComponent extends CdForm implements OnInit { icons = Icons; pwdExpirationSettings: CdPwdExpirationSettings; pwdExpirationFormat = 'YYYY-MM-DD'; - + selectedRole: string[]; + passwordexp: boolean = false; constructor( private authService: AuthService, private authStorageService: AuthStorageService, @@ -91,6 +92,7 @@ export class UserFormComponent extends CdForm implements OnInit { password: [ '', [], + [ CdValidators.passwordPolicy( this.userService, @@ -105,7 +107,7 @@ export class UserFormComponent extends CdForm implements OnInit { ] ], confirmpassword: [''], - pwdExpirationDate: [undefined], + pwdExpirationDate: [''], email: ['', [CdValidators.email]], roles: [[]], enabled: [true, [Validators.required]], @@ -121,8 +123,10 @@ export class UserFormComponent extends CdForm implements OnInit { if (this.router.url.startsWith('/user-management/users/edit')) { this.mode = this.userFormMode.editing; this.action = this.actionLabels.EDIT; + this.passwordexp = false; } else { this.action = this.actionLabels.CREATE; + this.passwordexp = true; } const observables = [this.roleService.list(), this.settingsService.getStandardSettings()]; @@ -130,6 +134,7 @@ export class UserFormComponent extends CdForm implements OnInit { (result: [UserFormRoleModel[], CdPwdExpirationSettings]) => { this.allRoles = _.map(result[0], (role) => { role.enabled = true; + role.content = `${role.name}, ${role.description}`; return role; }); this.pwdExpirationSettings = new CdPwdExpirationSettings(result[1]); @@ -158,7 +163,6 @@ export class UserFormComponent extends CdForm implements OnInit { this.userService.get(username).subscribe((userFormModel: UserFormModel) => { this.response = _.cloneDeep(userFormModel); this.setResponse(userFormModel); - this.loadingReady(); }); }); @@ -173,20 +177,28 @@ export class UserFormComponent extends CdForm implements OnInit { this.userForm.get(key).setValue(response[key]) ); const expirationDate = response['pwdExpirationDate']; + if (expirationDate) { + this.passwordexp = false; this.userForm .get('pwdExpirationDate') .setValue(moment(expirationDate * 1000).format(this.pwdExpirationFormat)); + } else { + this.passwordexp = true; } } getRequest(): UserFormModel { const userFormModel = new UserFormModel(); + ['username', 'password', 'name', 'email', 'roles', 'enabled', 'pwdUpdateRequired'].forEach( - (key) => (userFormModel[key] = this.userForm.get(key).value) + (key) => { + userFormModel[key] = this.userForm.get(key).value; + } ); const expirationDate = this.userForm.get('pwdExpirationDate').value; if (expirationDate) { + this.passwordexp = false; const mom = moment(expirationDate, this.pwdExpirationFormat); if ( this.mode !== this.userFormMode.editing || diff --git a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html index 0150896e883..0bcb5278f91 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html @@ -240,7 +240,7 @@ <!-- Filesystem --> <cds-sidenav-menu title="File" i18n-title - *ngIf="permissions.nfs.read && enabledFeature.nfs || permissions.cephfs.read && enabledFeature.cephfs" + *ngIf="permissions.nfs.read && enabledFeature.nfs || permissions.cephfs.read && enabledFeature.cephfs || permissions.smb.read" class="tc_menuitem_file"> <svg cdsIcon="file-storage" icon @@ -257,6 +257,12 @@ i18n-title *ngIf="permissions.nfs.read && enabledFeature.nfs" class="tc_submenuitem tc_submenuitem_file_nfs"><span i18n>NFS</span></cds-sidenav-item> + <cds-sidenav-item route="/cephfs/smb" + [useRouter]="true" + title="SMB" + i18n-title + *ngIf="permissions.smb.read" + class="tc_submenuitem tc_submenuitem_file_smb"><span i18n>SMB</span></cds-sidenav-item> </cds-sidenav-menu> <!-- Observability --> <cds-sidenav-menu title="Observability" diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts index 317293be07c..fedc7b8de0f 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/prometheus.service.ts @@ -163,11 +163,9 @@ export class PrometheusService { checkNan ) { queriesResults[queryName].forEach((valueArray: any[]) => { - valueArray.forEach((val, index) => { - if (isNaN(parseFloat(val[1]))) { - valueArray[index][1] = '0'; - } - }); + if (isNaN(parseFloat(valueArray[1]))) { + valueArray[1] = '0'; + } }); } }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.ts index 595b02ec276..ed3134f5cae 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-bucket.service.ts @@ -2,8 +2,9 @@ import { HttpClient, HttpParams } from '@angular/common/http'; import { Injectable } from '@angular/core'; import _ from 'lodash'; -import { of as observableOf } from 'rxjs'; -import { catchError, mapTo } from 'rxjs/operators'; +import { BehaviorSubject, of as observableOf } from 'rxjs'; +import { catchError, map, mapTo } from 'rxjs/operators'; +import { Bucket } from '~/app/ceph/rgw/models/rgw-bucket'; import { ApiClient } from '~/app/shared/api/api-client'; import { RgwDaemonService } from '~/app/shared/api/rgw-daemon.service'; @@ -15,11 +16,65 @@ import { cdEncode } from '~/app/shared/decorators/cd-encode'; }) export class RgwBucketService extends ApiClient { private url = 'api/rgw/bucket'; + private bucketsSubject = new BehaviorSubject<Bucket[]>([]); + private totalNumObjectsSubject = new BehaviorSubject<number>(0); + private totalUsedCapacitySubject = new BehaviorSubject<number>(0); + private averageObjectSizeSubject = new BehaviorSubject<number>(0); + buckets$ = this.bucketsSubject.asObservable(); + totalNumObjects$ = this.totalNumObjectsSubject.asObservable(); + totalUsedCapacity$ = this.totalUsedCapacitySubject.asObservable(); + averageObjectSize$ = this.averageObjectSizeSubject.asObservable(); constructor(private http: HttpClient, private rgwDaemonService: RgwDaemonService) { super(); } + fetchAndTransformBuckets() { + return this.list(true).pipe( + map((buckets: Bucket[]) => { + let totalNumObjects = 0; + let totalUsedCapacity = 0; + let averageObjectSize = 0; + const transformedBuckets = buckets.map((bucket) => this.transformBucket(bucket)); + transformedBuckets.forEach((bucket) => { + totalNumObjects += bucket?.num_objects || 0; + totalUsedCapacity += bucket?.bucket_size || 0; + }); + averageObjectSize = this.calculateAverageObjectSize(totalNumObjects, totalUsedCapacity); + this.bucketsSubject.next(transformedBuckets); + this.totalNumObjectsSubject.next(totalNumObjects); + this.totalUsedCapacitySubject.next(totalUsedCapacity); + this.averageObjectSizeSubject.next(averageObjectSize); + }) + ); + } + + transformBucket(bucket: Bucket) { + const maxBucketSize = bucket?.bucket_quota?.max_size ?? 0; + const maxBucketObjects = bucket?.bucket_quota?.max_objects ?? 0; + const bucket_size = bucket['usage']?.['rgw.main']?.['size_actual'] || 0; + const num_objects = bucket['usage']?.['rgw.main']?.['num_objects'] || 0; + return { + ...bucket, + bucket_size, + num_objects, + size_usage: this.calculateSizeUsage(bucket_size, maxBucketSize), + object_usage: this.calculateObjectUsage(num_objects, maxBucketObjects) + }; + } + + calculateSizeUsage(bucket_size: number, maxBucketSize: number) { + return maxBucketSize > 0 ? bucket_size / maxBucketSize : undefined; + } + + calculateObjectUsage(num_objects: number, maxBucketObjects: number) { + return maxBucketObjects > 0 ? num_objects / maxBucketObjects : undefined; + } + + calculateAverageObjectSize(totalNumObjects: number, totalUsedCapacity: number) { + return totalNumObjects > 0 ? totalUsedCapacity / totalNumObjects : 0; + } + /** * Get the list of buckets. * @return Observable<Object[]> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts index 3dc886e172f..8a39dc8a284 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/rgw-multisite.service.ts @@ -16,7 +16,7 @@ export class RgwMultisiteService { constructor(private http: HttpClient, public rgwDaemonService: RgwDaemonService) {} - migrate(realm: RgwRealm, zonegroup: RgwZonegroup, zone: RgwZone) { + migrate(realm: RgwRealm, zonegroup: RgwZonegroup, zone: RgwZone, username: string) { return this.rgwDaemonService.request((params: HttpParams) => { params = params.appendAll({ realm_name: realm.name, @@ -24,8 +24,7 @@ export class RgwMultisiteService { zone_name: zone.name, zonegroup_endpoints: zonegroup.endpoints, zone_endpoints: zone.endpoints, - access_key: zone.system_key.access_key, - secret_key: zone.system_key.secret_key + username: username }); return this.http.put(`${this.uiUrl}/migrate`, null, { params: params }); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.spec.ts new file mode 100644 index 00000000000..2dcdbdb6402 --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.spec.ts @@ -0,0 +1,31 @@ +import { TestBed } from '@angular/core/testing'; +import { HttpClientTestingModule, HttpTestingController } from '@angular/common/http/testing'; + +import { SmbService } from './smb.service'; +import { configureTestBed } from '~/testing/unit-test-helper'; + +describe('SmbService', () => { + let service: SmbService; + let httpTesting: HttpTestingController; + + configureTestBed({ + providers: [SmbService], + imports: [HttpClientTestingModule] + }); + + beforeEach(() => { + TestBed.configureTestingModule({}); + service = TestBed.inject(SmbService); + httpTesting = TestBed.inject(HttpTestingController); + }); + + it('should be created', () => { + expect(service).toBeTruthy(); + }); + + it('should call list', () => { + service.listClusters().subscribe(); + const req = httpTesting.expectOne('api/smb/cluster'); + expect(req.request.method).toBe('GET'); + }); +}); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.ts new file mode 100644 index 00000000000..4f4ebcb423c --- /dev/null +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/api/smb.service.ts @@ -0,0 +1,18 @@ +import { HttpClient } from '@angular/common/http'; +import { Injectable } from '@angular/core'; +import { Observable } from 'rxjs'; + +import { SMBCluster } from '~/app/ceph/smb/smb.model'; + +@Injectable({ + providedIn: 'root' +}) +export class SmbService { + baseURL = 'api/smb'; + + constructor(private http: HttpClient) {} + + listClusters(): Observable<SMBCluster[]> { + return this.http.get<SMBCluster[]>(`${this.baseURL}/cluster`); + } +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts index d3ebc5f37c6..0e1c0906f4a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/config-option/config-option.model.ts @@ -9,4 +9,5 @@ export class ConfigFormModel { min: any; max: any; services: Array<string>; + can_update_at_runtime: boolean; } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html index 4b973187dbb..90e3a1d2be8 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html @@ -49,7 +49,7 @@ [form]="deletionForm" [submitText]="(actionDescription | titlecase) + ' ' + itemDescription" [modalForm]="true" - [submitBtnType]="actionDescription === 'delete' || 'remove' ? 'danger' : 'primary'"></cd-form-button-panel> + [submitBtnType]="(actionDescription === 'delete' || actionDescription === 'remove') ? 'danger' : 'primary'"></cd-form-button-panel> </cds-modal> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html index 328e72cc595..ccdb70e39e4 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.html @@ -1,22 +1,23 @@ -<div cdsCol - class="form-item"> - <div cdsRow> -<cds-date-picker [label]="name" - i18n-label - placeholder="NOT PROTECTED" - formControlname="expiresAt" - dateFormat="Y/m/d" - [value]="date" - (valueChange)="onModelChange($event)" - [helperText]="helperText" - [disabled]="disabled" - cdsTheme="theme"></cds-date-picker> -<cds-timepicker (valueChange)="onModelChange($event)" - [(ngModel)]="time" - label="Select a time" - [disabled]="disabled" - pattern="(1[012]|[0-9]):[0-5][0-9]" - *ngIf="hasTime"> +<div cdsCol> + <div cdsRow + class="form-item-append"> + <cds-text-label>{{name}} + <cds-date-picker i18n-label + [placeholder]="placeHolder" + formControlname="expiresAt" + dateFormat="Y/m/d" + [value]="date" + (valueChange)="onModelChange($event)" + [helperText]="helperText" + [disabled]="disabled" + cdsTheme="theme"></cds-date-picker> + </cds-text-label> + <cds-text-label *ngIf="hasTime">Select a time + <cds-timepicker (valueChange)="onModelChange($event)" + [(ngModel)]="time" + [disabled]="disabled" + pattern="(1[012]|[0-9]):[0-5][0-9]" + *ngIf="hasTime"> <cds-timepicker-select [(ngModel)]="ampm" [disabled]="disabled" (valueChange)="onModelChange($event)"> @@ -24,5 +25,7 @@ value="AM">AM</option> <option value="PM">PM</option> </cds-timepicker-select> -</cds-timepicker></div> +</cds-timepicker> +</cds-text-label> +</div> </div> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss index e69de29bb2d..39f2a7115a1 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.scss @@ -0,0 +1,3 @@ +.form-item-append { + margin-top: 1rem; +} diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts index 4841d2ed92d..3458d9171a7 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/date-time-picker/date-time-picker.component.ts @@ -25,7 +25,8 @@ export class DateTimePickerComponent implements OnInit { @Input() helperText = ''; - + @Input() + placeHolder = ''; @Input() disabled = false; @@ -39,9 +40,8 @@ export class DateTimePickerComponent implements OnInit { date: { [key: number]: string }[] = []; time: string; ampm: string; - sub: Subscription; - + @Input() defaultDate: boolean = false; constructor(private calendar: NgbCalendar) {} ngOnInit() { @@ -59,8 +59,12 @@ export class DateTimePickerComponent implements OnInit { if (!mom.isValid() || mom.isBefore(moment())) { mom = moment(); } + if (this.defaultDate) { + this.date.push([]); + } else { + this.date.push(mom.format('YYYY-MM-DD')); + } - this.date.push(mom.format('YYYY-MM-DD')); const time = mom.format('HH:mm:ss'); this.time = mom.format('hh:mm'); this.ampm = mom.hour() >= 12 ? 'PM' : 'AM'; @@ -76,7 +80,9 @@ export class DateTimePickerComponent implements OnInit { onModelChange(event?: any) { if (event) { - if (Array.isArray(event)) { + if (event.length === 0) { + this.datetime.date = { date: null, time: null, ampm: null }; + } else if (Array.isArray(event)) { this.datetime.date = moment(event[0]).format('YYYY-MM-DD'); } else if (event && ['AM', 'PM'].includes(event)) { const initialMoment = moment(this.datetime.time, 'hh:mm:ss A'); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html index da1a4800f7f..81ad90914b6 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/components/helper/helper.component.html @@ -5,7 +5,8 @@ <ng-content></ng-content> </ng-template> -<cds-tooltip [description]="popoverTpl"> +<cds-tooltip [description]="popoverTpl" + [autoAlign]="true"> <svg cdsIcon="information" size="16" title="info"></svg> diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts index 361a404a11b..f1bbebed51d 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/enum/dashboard-promqls.enum.ts @@ -11,8 +11,8 @@ export enum Promqls { export enum RgwPromqls { RGW_REQUEST_PER_SECOND = 'sum(rate(ceph_rgw_req[1m]))', - AVG_GET_LATENCY = 'sum(rate(ceph_rgw_op_get_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_get_obj_lat_count[1m]))', - AVG_PUT_LATENCY = 'sum(rate(ceph_rgw_op_put_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_put_obj_lat_count[1m]))', + AVG_GET_LATENCY = '(sum(rate(ceph_rgw_op_get_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_get_obj_lat_count[1m]))) * 1000', + AVG_PUT_LATENCY = '(sum(rate(ceph_rgw_op_put_obj_lat_sum[1m])) / sum(rate(ceph_rgw_op_put_obj_lat_count[1m]))) * 1000', GET_BANDWIDTH = 'sum(rate(ceph_rgw_op_get_obj_bytes[1m]))', PUT_BANDWIDTH = 'sum(rate(ceph_rgw_op_put_obj_bytes[1m]))' } diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permission.spec.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permission.spec.ts index 213fb416ea5..d1010a3408a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permission.spec.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permission.spec.ts @@ -19,7 +19,8 @@ describe('cd-notification classes', () => { rbdImage: { create: false, delete: false, read: false, update: false }, rbdMirroring: { create: false, delete: false, read: false, update: false }, rgw: { create: false, delete: false, read: false, update: false }, - user: { create: false, delete: false, read: false, update: false } + user: { create: false, delete: false, read: false, update: false }, + smb: { create: false, delete: false, read: false, update: false } }); }); @@ -40,7 +41,8 @@ describe('cd-notification classes', () => { 'rbd-image': ['create', 'read', 'update', 'delete'], 'rbd-mirroring': ['create', 'read', 'update', 'delete'], rgw: ['create', 'read', 'update', 'delete'], - user: ['create', 'read', 'update', 'delete'] + user: ['create', 'read', 'update', 'delete'], + smb: ['create', 'read', 'update', 'delete'] }; expect(new Permissions(fullyGranted)).toEqual({ cephfs: { create: true, delete: true, read: true, update: true }, @@ -59,7 +61,8 @@ describe('cd-notification classes', () => { rbdImage: { create: true, delete: true, read: true, update: true }, rbdMirroring: { create: true, delete: true, read: true, update: true }, rgw: { create: true, delete: true, read: true, update: true }, - user: { create: true, delete: true, read: true, update: true } + user: { create: true, delete: true, read: true, update: true }, + smb: { create: true, delete: true, read: true, update: true } }); }); }); diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permissions.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permissions.ts index 5e9fe4aae47..838385d840a 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permissions.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/shared/models/permissions.ts @@ -29,6 +29,7 @@ export class Permissions { grafana: Permission; prometheus: Permission; nfs: Permission; + smb: Permission; constructor(serverPermissions: any) { this.hosts = new Permission(serverPermissions['hosts']); @@ -48,5 +49,6 @@ export class Permissions { this.grafana = new Permission(serverPermissions['grafana']); this.prometheus = new Permission(serverPermissions['prometheus']); this.nfs = new Permission(serverPermissions['nfs-ganesha']); + this.smb = new Permission(serverPermissions['smb']); } } diff --git a/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss b/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss index 37f89aba17f..0725b63dbfd 100644 --- a/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss +++ b/src/pybind/mgr/dashboard/frontend/src/styles/themes/_content.scss @@ -33,6 +33,7 @@ $content-theme: map-merge( text-primary: vv.$dark, text-secondary: vv.$dark, text-disabled: vv.$gray-500, + icon-secondary: vv.$gray-800, field-01: colors.$gray-10, interactive: vv.$primary ) diff --git a/src/pybind/mgr/dashboard/openapi.yaml b/src/pybind/mgr/dashboard/openapi.yaml index b464344e27a..de1b3e8b60e 100644 --- a/src/pybind/mgr/dashboard/openapi.yaml +++ b/src/pybind/mgr/dashboard/openapi.yaml @@ -3977,10 +3977,27 @@ paths: application/json: schema: properties: + force_update: + description: Force update the config option + type: boolean name: + description: Config option name type: string value: - type: string + description: Section and Value of the config option + items: + properties: + section: + description: Section/Client where config needs to be updated + type: string + value: + description: Value of the config option + type: string + required: + - section + - value + type: object + type: array required: - name - value @@ -4007,6 +4024,7 @@ paths: trace. security: - jwt: [] + summary: Create/Update Cluster Configuration tags: - ClusterConfiguration put: @@ -10790,6 +10808,274 @@ paths: - jwt: [] tags: - Prometheus + /api/rgw/accounts: + get: + parameters: + - default: false + in: query + name: detailed + schema: + type: boolean + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + tags: + - RgwUserAccounts + post: + parameters: [] + requestBody: + content: + application/json: + schema: + properties: + account_id: + type: integer + account_name: + type: integer + email: + type: string + type: object + responses: + '201': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource created. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}: + delete: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + responses: + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '204': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource deleted. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Delete RGW Account + tags: + - RgwUserAccounts + get: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Get RGW Account by id + tags: + - RgwUserAccounts + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + account_name: + type: integer + email: + type: string + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Update RGW account info + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}/quota: + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + max_objects: + type: string + max_size: + description: Max size + type: string + quota_type: + type: string + required: + - quota_type + - max_size + - max_objects + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Set RGW Account/Bucket quota + tags: + - RgwUserAccounts + /api/rgw/accounts/{account_id}/quota/status: + put: + parameters: + - description: Account id + in: path + name: account_id + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + properties: + quota_status: + type: string + quota_type: + type: string + required: + - quota_type + - quota_status + type: object + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Resource updated. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Enable/Disable RGW Account/Bucket quota + tags: + - RgwUserAccounts /api/rgw/bucket: get: parameters: @@ -14055,6 +14341,500 @@ paths: - jwt: [] tags: - Settings + /api/smb/cluster: + get: + description: "\n List smb clusters\n " + parameters: [] + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + schema: + items: + properties: + auth_mode: + description: Either 'active-directory' or 'user' + type: string + cluster_id: + description: Unique identifier for the cluster + type: string + custom_dns: + description: List of custom DNS server addresses + items: + type: string + type: array + domain_settings: + description: Domain-specific settings for active-directory auth + mode + properties: + join_sources: + description: List of join auth sources for domain settings + items: + properties: + ref: + description: Reference identifier for the join auth + resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + realm: + description: Domain realm, e.g., 'DOMAIN1.SINK.TEST' + type: string + required: + - realm + - join_sources + type: object + intent: + description: Desired state of the resource, e.g., 'present' + or 'removed' + type: string + placement: + description: Placement configuration for the resource + properties: + count: + description: Number of instances to place + type: integer + required: + - count + type: object + resource_type: + description: ceph.smb.cluster + type: string + user_group_settings: + description: User group settings for user auth mode + items: + properties: + ref: + description: Reference identifier for the user group resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + type: object + required: + - resource_type + - cluster_id + - auth_mode + - intent + - domain_settings + - user_group_settings + - custom_dns + - placement + type: array + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: List smb clusters + tags: + - SMB + post: + description: "\n Create an smb cluster\n\n :param cluster_resource:\ + \ Dict cluster data\n :return: Returns cluster resource.\n :rtype:\ + \ Dict[str, Any]\n " + parameters: [] + requestBody: + content: + application/json: + schema: + properties: + cluster_resource: + description: cluster_resource + type: string + required: + - cluster_resource + type: object + responses: + '201': + content: + application/vnd.ceph.api.v1.0+json: + schema: + properties: + results: + description: List of results with resource details + items: + properties: + resource: + description: Resource details + properties: + auth_mode: + description: Either 'active-directory' or 'user' + type: string + cluster_id: + description: Unique identifier for the cluster + type: string + custom_dns: + description: List of custom DNS server addresses (optional) + items: + type: string + type: array + domain_settings: + description: Domain-specific settings for active-directory + auth mode + properties: + join_sources: + description: List of join auth sources for domain + settings + items: + properties: + ref: + description: Reference identifier for the + join auth resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + realm: + description: Domain realm, e.g., 'DOMAIN1.SINK.TEST' + type: string + required: + - realm + - join_sources + type: object + intent: + description: Desired state of the resource, e.g., 'present' + or 'removed' + type: string + placement: + description: Placement configuration for the resource + (optional) + properties: + count: + description: Number of instances to place + type: integer + required: + - count + type: object + resource_type: + description: ceph.smb.cluster + type: string + user_group_settings: + description: User group settings for user auth mode + (optional) + items: + properties: + ref: + description: Reference identifier for the user + group resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + required: + - resource_type + - cluster_id + - auth_mode + - intent + - domain_settings + - user_group_settings + - custom_dns + - placement + type: object + state: + description: State of the resource + type: string + success: + description: Indicates whether the operation was successful + type: boolean + required: + - resource + - state + - success + type: object + type: array + success: + description: Overall success status of the operation + type: boolean + required: + - results + - success + type: object + description: Resource created. + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Create smb cluster + tags: + - SMB + /api/smb/cluster/{cluster_id}: + get: + description: "\n Get an smb cluster by cluster id\n " + parameters: + - description: Unique identifier for the cluster + in: path + name: cluster_id + required: true + schema: + type: string + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + schema: + properties: + auth_mode: + description: Either 'active-directory' or 'user' + type: string + cluster_id: + description: Unique identifier for the cluster + type: string + custom_dns: + description: List of custom DNS server addresses + items: + type: string + type: array + domain_settings: + description: Domain-specific settings for active-directory auth + mode + properties: + join_sources: + description: List of join auth sources for domain settings + items: + properties: + ref: + description: Reference identifier for the join auth + resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + realm: + description: Domain realm, e.g., 'DOMAIN1.SINK.TEST' + type: string + required: + - realm + - join_sources + type: object + intent: + description: Desired state of the resource, e.g., 'present' or + 'removed' + type: string + placement: + description: Placement configuration for the resource + properties: + count: + description: Number of instances to place + type: integer + required: + - count + type: object + resource_type: + description: ceph.smb.cluster + type: string + user_group_settings: + description: User group settings for user auth mode + items: + properties: + ref: + description: Reference identifier for the user group resource + type: string + source_type: + description: resource + type: string + required: + - source_type + - ref + type: object + type: array + required: + - resource_type + - cluster_id + - auth_mode + - intent + - domain_settings + - user_group_settings + - custom_dns + - placement + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Get an smb cluster + tags: + - SMB + /api/smb/share: + get: + description: "\n List all smb shares or all shares for a given cluster\n\ + \n :param cluster_id: Dict containing cluster information\n \ + \ :return: Returns list of shares.\n :rtype: List[Dict]\n " + parameters: + - default: '' + description: Unique identifier for the cluster + in: query + name: cluster_id + schema: + type: string + responses: + '200': + content: + application/vnd.ceph.api.v1.0+json: + schema: + properties: + browseable: + description: Indicates if the share is browseable + type: boolean + cephfs: + description: Configuration for the CephFS share + properties: + path: + description: Path within the CephFS file system + type: string + provider: + description: Provider of the CephFS share, e.g., 'samba-vfs' + type: string + volume: + description: Name of the CephFS file system + type: string + required: + - volume + - path + - provider + type: object + cluster_id: + description: Unique identifier for the cluster + type: string + intent: + description: Desired state of the resource, e.g., 'present' or + 'removed' + type: string + name: + description: Name of the share + type: string + readonly: + description: Indicates if the share is read-only + type: boolean + resource_type: + description: ceph.smb.share + type: string + share_id: + description: Unique identifier for the share + type: string + required: + - resource_type + - cluster_id + - share_id + - intent + - name + - readonly + - browseable + - cephfs + type: object + description: OK + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: List smb shares + tags: + - SMB + /api/smb/share/{cluster_id}/{share_id}: + delete: + description: "\n Remove an smb share from a given cluster\n\n \ + \ :param cluster_id: Cluster identifier\n :param share_id: Share identifier\n\ + \ :return: None.\n " + parameters: + - description: Unique identifier for the cluster + in: path + name: cluster_id + required: true + schema: + type: string + - description: Unique identifier for the share + in: path + name: share_id + required: true + schema: + type: string + responses: + '202': + content: + application/vnd.ceph.api.v1.0+json: + type: object + description: Operation is still executing. Please check the task queue. + '204': + content: + application/vnd.ceph.api.v1.0+json: + schema: + properties: {} + type: object + description: Resource deleted. + '400': + description: Operation exception. Please check the response body for details. + '401': + description: Unauthenticated access. Please login first. + '403': + description: Unauthorized access. Please check your permissions. + '500': + description: Unexpected error. Please check the response body for the stack + trace. + security: + - jwt: [] + summary: Remove smb shares + tags: + - SMB /api/summary: get: parameters: [] @@ -15472,12 +16252,16 @@ tags: name: RgwSite - description: RGW User Management API name: RgwUser +- description: RGW User Accounts API + name: RgwUserAccounts - description: '*No description available*' name: RgwZone - description: '*No description available*' name: RgwZonegroup - description: Role Management API name: Role +- description: SMB Cluster Management API + name: SMB - description: Service Management API name: Service - description: Settings Management API diff --git a/src/pybind/mgr/dashboard/plugins/feature_toggles.py b/src/pybind/mgr/dashboard/plugins/feature_toggles.py index f16b2e68c4d..63b1d762c34 100644 --- a/src/pybind/mgr/dashboard/plugins/feature_toggles.py +++ b/src/pybind/mgr/dashboard/plugins/feature_toggles.py @@ -1,10 +1,11 @@ # -*- coding: utf-8 -*- from enum import Enum -from typing import List, Optional, Set, no_type_check +from typing import Dict, List, Optional, Set, no_type_check import cherrypy from mgr_module import CLICommand, Option +from mgr_util import CLIWarning from ..controllers.cephfs import CephFS from ..controllers.iscsi import Iscsi, IscsiTarget @@ -27,6 +28,14 @@ class Features(Enum): NFS = 'nfs' DASHBOARD = 'dashboard' + # if we want to add any custom warning message when enabling a feature + # we can add it here as key-value pair in warn_msg. + # eg: Features.ISCSI.value: 'iscsi warning message' + @property + def warning(self): + warn_msg: Dict[str, str] = {} + return warn_msg.get(self.value, None) + PREDISABLED_FEATURES = set() # type: Set[str] @@ -91,6 +100,8 @@ class FeatureToggles(I.CanMgr, I.Setupable, I.HasOptions, mgr.set_module_option( self.OPTION_FMT.format(feature), action == Actions.ENABLE) + if action == Actions.ENABLE and feature.warning: + msg += [CLIWarning(feature.warning)] msg += ["Feature '{.value}': {}".format( feature, 'enabled' if action == Actions.ENABLE else diff --git a/src/pybind/mgr/dashboard/security.py b/src/pybind/mgr/dashboard/security.py index 2b624aabcc7..c329d24e1b3 100644 --- a/src/pybind/mgr/dashboard/security.py +++ b/src/pybind/mgr/dashboard/security.py @@ -27,6 +27,7 @@ class Scope(object): DASHBOARD_SETTINGS = "dashboard-settings" NFS_GANESHA = "nfs-ganesha" NVME_OF = "nvme-of" + SMB = "smb" @classmethod def all_scopes(cls): diff --git a/src/pybind/mgr/dashboard/services/access_control.py b/src/pybind/mgr/dashboard/services/access_control.py index 21c1a9572bb..6319802b6cc 100644 --- a/src/pybind/mgr/dashboard/services/access_control.py +++ b/src/pybind/mgr/dashboard/services/access_control.py @@ -278,6 +278,16 @@ GANESHA_MGR_ROLE = Role( Scope.CEPHFS: [_P.READ, _P.CREATE, _P.UPDATE, _P.DELETE], Scope.RGW: [_P.READ, _P.CREATE, _P.UPDATE, _P.DELETE], Scope.GRAFANA: [_P.READ], + Scope.SMB: [_P.READ] + }) + +SMB_MGR_ROLE = Role( + 'smb-manager', 'allows full permissions for the smb scope', { + Scope.SMB: [_P.READ, _P.CREATE, _P.UPDATE, _P.DELETE], + Scope.CEPHFS: [_P.READ, _P.CREATE, _P.UPDATE, _P.DELETE], + Scope.RGW: [_P.READ, _P.CREATE, _P.UPDATE, _P.DELETE], + Scope.GRAFANA: [_P.READ], + Scope.NFS_GANESHA: [_P.READ] }) @@ -290,6 +300,7 @@ SYSTEM_ROLES = { POOL_MGR_ROLE.name: POOL_MGR_ROLE, CEPHFS_MGR_ROLE.name: CEPHFS_MGR_ROLE, GANESHA_MGR_ROLE.name: GANESHA_MGR_ROLE, + SMB_MGR_ROLE.name: SMB_MGR_ROLE, } # static name-like roles list for role mapping diff --git a/src/pybind/mgr/dashboard/services/cluster.py b/src/pybind/mgr/dashboard/services/cluster.py index 9caaf196336..3d7c21ac9ae 100644 --- a/src/pybind/mgr/dashboard/services/cluster.py +++ b/src/pybind/mgr/dashboard/services/cluster.py @@ -9,9 +9,6 @@ class ClusterCapacity(NamedTuple): total_avail_bytes: int total_bytes: int total_used_raw_bytes: int - total_objects: int - total_pool_bytes_used: int - average_object_size: int class ClusterModel: @@ -47,33 +44,9 @@ class ClusterModel: @classmethod def get_capacity(cls) -> ClusterCapacity: df = mgr.get('df') - total_pool_bytes_used = 0 - average_object_size = 0 - total_data_pool_objects = 0 - total_data_pool_bytes_used = 0 - rgw_pools_data = cls.get_rgw_pools() - - for pool in df['pools']: - pool_name = str(pool['name']) - if pool_name in rgw_pools_data: - if pool_name.endswith('.data'): - objects = pool['stats']['objects'] - pool_bytes_used = pool['stats']['bytes_used'] - total_pool_bytes_used += pool_bytes_used - total_data_pool_objects += objects - replica = rgw_pools_data[pool_name] - total_data_pool_bytes_used += pool_bytes_used / replica - - average_object_size = total_data_pool_bytes_used / total_data_pool_objects if total_data_pool_objects != 0 else 0 # noqa E501 #pylint: disable=line-too-long - - return ClusterCapacity( - total_avail_bytes=df['stats']['total_avail_bytes'], - total_bytes=df['stats']['total_bytes'], - total_used_raw_bytes=df['stats']['total_used_raw_bytes'], - total_objects=total_data_pool_objects, - total_pool_bytes_used=total_pool_bytes_used, - average_object_size=average_object_size - )._asdict() + return ClusterCapacity(total_avail_bytes=df['stats']['total_avail_bytes'], + total_bytes=df['stats']['total_bytes'], + total_used_raw_bytes=df['stats']['total_used_raw_bytes'])._asdict() @classmethod def get_rgw_pools(cls): diff --git a/src/pybind/mgr/dashboard/services/rgw_client.py b/src/pybind/mgr/dashboard/services/rgw_client.py index 2fe09821694..9fa249acf44 100755 --- a/src/pybind/mgr/dashboard/services/rgw_client.py +++ b/src/pybind/mgr/dashboard/services/rgw_client.py @@ -1348,8 +1348,7 @@ class RgwMultisiteAutomation: class RgwMultisite: def migrate_to_multisite(self, realm_name: str, zonegroup_name: str, zone_name: str, - zonegroup_endpoints: str, zone_endpoints: str, access_key: str, - secret_key: str): + zonegroup_endpoints: str, zone_endpoints: str, username: str): rgw_realm_create_cmd = ['realm', 'create', '--rgw-realm', realm_name, '--default'] try: exit_code, _, err = mgr.send_rgwadmin_command(rgw_realm_create_cmd, False) @@ -1411,18 +1410,34 @@ class RgwMultisite: http_status_code=500, component='rgw') except SubprocessError as error: raise DashboardException(error, http_status_code=500, component='rgw') + self.update_period() - if access_key and secret_key: - rgw_zone_modify_cmd = ['zone', 'modify', '--rgw-zone', zone_name, - '--access-key', access_key, '--secret', secret_key] - try: - exit_code, _, err = mgr.send_rgwadmin_command(rgw_zone_modify_cmd) - if exit_code > 0: - raise DashboardException(e=err, msg='Unable to modify zone', + try: + user_details = self.create_system_user(username, zone_name) + if user_details: + keys = user_details['keys'][0] + access_key = keys['access_key'] + secret_key = keys['secret_key'] + if access_key and secret_key: + self.modify_zone(zone_name=zone_name, + zonegroup_name=zonegroup_name, + default='true', master='true', + endpoints=zone_endpoints, + access_key=keys['access_key'], + secret_key=keys['secret_key']) + else: + raise DashboardException(msg='Access key or secret key is missing', http_status_code=500, component='rgw') - except SubprocessError as error: - raise DashboardException(error, http_status_code=500, component='rgw') - self.update_period() + except Exception as e: + raise DashboardException(msg='Failed to modify zone or create system user: %s' % e, + http_status_code=500, component='rgw') + + try: + rgw_service_manager = RgwServiceManager() + rgw_service_manager.restart_rgw_daemons_and_set_credentials() + except Exception as e: + raise DashboardException(msg='Failed to restart RGW daemon: %s' % e, + http_status_code=500, component='rgw') def create_realm(self, realm_name: str, default: bool): rgw_realm_create_cmd = ['realm', 'create'] diff --git a/src/pybind/mgr/dashboard/services/rgw_iam.py b/src/pybind/mgr/dashboard/services/rgw_iam.py index dbf00df25e0..5f490323441 100644 --- a/src/pybind/mgr/dashboard/services/rgw_iam.py +++ b/src/pybind/mgr/dashboard/services/rgw_iam.py @@ -1,12 +1,13 @@ from subprocess import SubprocessError -from typing import List +from typing import List, Optional from .. import mgr from ..exceptions import DashboardException class RgwAccounts: - def send_rgw_cmd(self, command: List[str]): + @classmethod + def send_rgw_cmd(cls, command: List[str]): try: exit_code, out, err = mgr.send_rgwadmin_command(command) @@ -19,6 +20,78 @@ class RgwAccounts: except SubprocessError as e: raise DashboardException(e, component='rgw') - def get_accounts(self): + @classmethod + def get_accounts(cls, detailed: bool = False): + """ + Query account Id's, optionally returning full details. + + :param detailed: Boolean to indicate if full account details are required. + """ get_accounts_cmd = ['account', 'list'] - return self.send_rgw_cmd(get_accounts_cmd) + account_list = cls.send_rgw_cmd(get_accounts_cmd) + detailed_account_list = [] + if detailed: + for account in account_list: + detailed_account_list.append(cls.get_account(account)) + return detailed_account_list + return account_list + + @classmethod + def get_account(cls, account_id: str): + get_account_cmd = ['account', 'get', '--account-id', account_id] + return cls.send_rgw_cmd(get_account_cmd) + + @classmethod + def create_account(cls, account_name: Optional[str] = None, + account_id: Optional[str] = None, email: Optional[str] = None): + create_accounts_cmd = ['account', 'create'] + + if account_name: + create_accounts_cmd += ['--account-name', account_name] + + if account_id: + create_accounts_cmd += ['--account_id', account_id] + + if email: + create_accounts_cmd += ['--email', email] + + return cls.send_rgw_cmd(create_accounts_cmd) + + @classmethod + def modify_account(cls, account_id: str, account_name: Optional[str] = None, + email: Optional[str] = None): + modify_accounts_cmd = ['account', 'modify', '--account-id', account_id] + + if account_name: + modify_accounts_cmd += ['--account-name', account_name] + + if email: + modify_accounts_cmd += ['--email', email] + + return cls.send_rgw_cmd(modify_accounts_cmd) + + @classmethod + def delete_account(cls, account_id: str): + modify_accounts_cmd = ['account', 'rm', '--account-id', account_id] + + return cls.send_rgw_cmd(modify_accounts_cmd) + + @classmethod + def get_account_stats(cls, account_id: str): + account_stats_cmd = ['account', 'stats', '--account-id', account_id] + + return cls.send_rgw_cmd(account_stats_cmd) + + @classmethod + def set_quota(cls, quota_type: str, account_id: str, max_size: str, max_objects: str): + set_quota_cmd = ['quota', 'set', '--quota-scope', quota_type, '--account-id', account_id, + '--max-size', max_size, '--max-objects', max_objects] + + return cls.send_rgw_cmd(set_quota_cmd) + + @classmethod + def set_quota_status(cls, quota_type: str, account_id: str, quota_status: str): + set_quota_status_cmd = ['quota', quota_status, '--quota-scope', quota_type, + '--account-id', account_id] + + return cls.send_rgw_cmd(set_quota_status_cmd) diff --git a/src/pybind/mgr/dashboard/tests/test_rgw_iam.py b/src/pybind/mgr/dashboard/tests/test_rgw_iam.py new file mode 100644 index 00000000000..133b5a0d390 --- /dev/null +++ b/src/pybind/mgr/dashboard/tests/test_rgw_iam.py @@ -0,0 +1,292 @@ +from unittest import TestCase +from unittest.mock import patch + +from ..controllers.rgw_iam import RgwUserAccountsController +from ..services.rgw_iam import RgwAccounts + + +class TestRgwUserAccountsController(TestCase): + + @patch.object(RgwAccounts, 'create_account') + def test_create_account(self, mock_create_account): + mockReturnVal = { + "id": "RGW18661471562806836", + "tenant": "", + "name": "", + "email": "", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + # Mock the return value of the create_account method + mock_create_account.return_value = mockReturnVal + + controller = RgwUserAccountsController() + result = controller.create(account_name='test_account', account_id='RGW18661471562806836', + email='test@example.com') + + # Check if the account creation method was called with the correct parameters + mock_create_account.assert_called_with('test_account', 'RGW18661471562806836', + 'test@example.com') + # Check the returned result + self.assertEqual(result, mockReturnVal) + + @patch.object(RgwAccounts, 'get_accounts') + def test_list_accounts(self, mock_get_accounts): + mock_return_value = [ + "RGW22222222222222222", + "RGW59378973811515857", + "RGW11111111111111111" + ] + + mock_get_accounts.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.list(detailed=False) + + mock_get_accounts.assert_called_with(False) + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'get_accounts') + def test_list_accounts_with_details(self, mock_get_accounts): + mock_return_value = [ + { + "id": "RGW22222222222222222", + "tenant": "", + "name": "Account2", + "email": "account2@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + }, + { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + ] + + mock_get_accounts.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.list(detailed=True) + + mock_get_accounts.assert_called_with(True) + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'get_account') + def test_get_account(self, mock_get_account): + mock_return_value = { + "id": "RGW22222222222222222", + "tenant": "", + "name": "Account2", + "email": "account2@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + mock_get_account.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.get(account_id='RGW22222222222222222') + + mock_get_account.assert_called_with('RGW22222222222222222') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'delete_account') + def test_delete_account(self, mock_delete_account): + mock_delete_account.return_value = None + + controller = RgwUserAccountsController() + result = controller.delete(account_id='RGW59378973811515857') + + mock_delete_account.assert_called_with('RGW59378973811515857') + + self.assertEqual(result, None) + + @patch.object(RgwAccounts, 'modify_account') + def test_set_account_name(self, mock_modify_account): + mock_return_value = mock_return_value = { + "id": "RGW59378973811515857", + "tenant": "", + "name": "new_account_name", + "email": "new_email@example.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": -1 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + mock_modify_account.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set(account_id='RGW59378973811515857', account_name='new_account_name', + email='new_email@example.com') + + mock_modify_account.assert_called_with('RGW59378973811515857', 'new_account_name', + 'new_email@example.com') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'set_quota') + def test_set_quota(self, mock_set_quota): + mock_return_value = { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": False, + "check_on_raw": False, + "max_size": 10737418240, + "max_size_kb": 10485760, + "max_objects": 1000000 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": 1000000 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + mock_set_quota.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set_quota(quota_type='account', account_id='RGW11111111111111111', + max_size='10GB', max_objects='1000') + + mock_set_quota.assert_called_with('account', 'RGW11111111111111111', '10GB', '1000') + + self.assertEqual(result, mock_return_value) + + @patch.object(RgwAccounts, 'set_quota_status') + def test_set_quota_status(self, mock_set_quota_status): + mock_return_value = { + "id": "RGW11111111111111111", + "tenant": "", + "name": "Account1", + "email": "account1@ceph.com", + "quota": { + "enabled": True, + "check_on_raw": False, + "max_size": 10737418240, + "max_size_kb": 10485760, + "max_objects": 1000000 + }, + "bucket_quota": { + "enabled": False, + "check_on_raw": False, + "max_size": -1, + "max_size_kb": 0, + "max_objects": 1000000 + }, + "max_users": 1000, + "max_roles": 1000, + "max_groups": 1000, + "max_buckets": 1000, + "max_access_keys": 4 + } + + mock_set_quota_status.return_value = mock_return_value + + controller = RgwUserAccountsController() + result = controller.set_quota_status(quota_type='account', + account_id='RGW11111111111111111', + quota_status='enabled') + + mock_set_quota_status.assert_called_with('account', 'RGW11111111111111111', 'enabled') + + self.assertEqual(result, mock_return_value) diff --git a/src/pybind/mgr/dashboard/tests/test_smb.py b/src/pybind/mgr/dashboard/tests/test_smb.py new file mode 100644 index 00000000000..754df482add --- /dev/null +++ b/src/pybind/mgr/dashboard/tests/test_smb.py @@ -0,0 +1,197 @@ +import json +from unittest.mock import Mock + +from dashboard.controllers.smb import SMBCluster, SMBShare + +from .. import mgr +from ..tests import ControllerTestCase + + +class SMBClusterTest(ControllerTestCase): + _endpoint = '/api/smb/cluster' + + _clusters = { + "resources": [{ + "resource_type": "ceph.smb.cluster", + "cluster_id": "clusterADTest", + "auth_mode": "active-directory", + "intent": "present", + "domain_settings": { + "realm": "DOMAIN1.SINK.TEST", + "join_sources": [ + { + "source_type": "resource", + "ref": "join1-admin" + }] + }, + "custom_dns": [ + "192.168.76.204" + ], + "placement": { + "count": 1 + } + }, + { + "resource_type": "ceph.smb.cluster", + "cluster_id": "clusterUserTest", + "auth_mode": "user", + "intent": "present", + "user_group_settings": [ + { + "source_type": "resource", + "ref": "ug1" + } + ] + }] + } + + @classmethod + def setup_server(cls): + cls.setup_controllers([SMBCluster]) + + def test_list_one_cluster(self): + mgr.remote = Mock(return_value=self._clusters['resources'][0]) + + self._get(self._endpoint) + self.assertStatus(200) + self.assertJsonBody([self._clusters['resources'][0]]) + + def test_list_multiple_clusters(self): + mgr.remote = Mock(return_value=self._clusters) + + self._get(self._endpoint) + self.assertStatus(200) + self.assertJsonBody(self._clusters['resources']) + + def test_get_cluster(self): + mgr.remote = Mock(return_value=self._clusters['resources'][0]) + cluster_id = self._clusters['resources'][0]['cluster_id'] + self._get(f'{self._endpoint}/{cluster_id}') + self.assertStatus(200) + self.assertJsonBody(self._clusters['resources'][0]) + mgr.remote.assert_called_once_with('smb', 'show', [f'ceph.smb.cluster.{cluster_id}']) + + def test_create_ad(self): + mock_simplified = Mock() + mock_simplified.to_simplified.return_value = json.dumps(self._clusters['resources'][0]) + mgr.remote = Mock(return_value=mock_simplified) + + _cluster_data = """ + { "cluster_resource": { + "resource_type": "ceph.smb.cluster", + "cluster_id": "clusterADTest", + "auth_mode": "active-directory", + "domain_settings": { + "realm": "DOMAIN1.SINK.TEST", + "join_sources": [ + { + "source_type": "resource", + "ref": "join1-admin" + } + ] + } + } + } + """ + + self._post(self._endpoint, _cluster_data) + self.assertStatus(201) + self.assertInJsonBody(json.dumps(self._clusters['resources'][0])) + + def test_create_user(self): + mock_simplified = Mock() + mock_simplified.to_simplified.return_value = json.dumps(self._clusters['resources'][1]) + mgr.remote = Mock(return_value=mock_simplified) + + _cluster_data = """ + { "cluster_resource": { + "resource_type": "ceph.smb.cluster", + "cluster_id": "clusterUser123Test", + "auth_mode": "user", + "user_group_settings": [ + { + "source_type": "resource", + "ref": "ug1" + } + ] + } + } + """ + self._post(self._endpoint, _cluster_data) + self.assertStatus(201) + self.assertInJsonBody(json.dumps(self._clusters['resources'][1])) + + +class SMBShareTest(ControllerTestCase): + _endpoint = '/api/smb/share' + + _shares = [{ + "resource_type": "ceph.smb.share", + "cluster_id": "clusterUserTest", + "share_id": "share1", + "intent": "present", + "name": "share1name", + "readonly": "false", + "browseable": "true", + "cephfs": { + "volume": "fs1", + "path": "/", + "provider": "samba-vfs" + } + }, + { + "resource_type": "ceph.smb.share", + "cluster_id": "clusterADTest", + "share_id": "share2", + "intent": "present", + "name": "share2name", + "readonly": "false", + "browseable": "true", + "cephfs": { + "volume": "fs2", + "path": "/", + "provider": "samba-vfs" + } + } + ] + + @classmethod + def setup_server(cls): + cls.setup_controllers([SMBShare]) + + def test_list_all(self): + mgr.remote = Mock(return_value=self._shares) + + self._get(self._endpoint) + self.assertStatus(200) + self.assertJsonBody(self._shares) + + def test_list_from_cluster(self): + mgr.remote = Mock(return_value=self._shares[0]) + + self._get(self._endpoint) + self.assertStatus(200) + self.assertJsonBody(self._shares[0]) + + def test_delete(self): + _res = { + "resource": { + "resource_type": "ceph.smb.share", + "cluster_id": "smbCluster1", + "share_id": "share1", + "intent": "removed" + }, + "state": "removed", + "success": "true" + } + _res_simplified = { + "resource_type": "ceph.smb.share", + "cluster_id": "smbCluster1", + "share_id": "share1", + "intent": "removed" + } + mgr.remote = Mock(return_value=Mock(return_value=_res)) + mgr.remote.return_value.one.return_value.to_simplified = Mock(return_value=_res_simplified) + self._delete(f'{self._endpoint}/smbCluster1/share1') + self.assertStatus(204) + mgr.remote.assert_called_once_with('smb', 'apply_resources', json.dumps(_res_simplified)) diff --git a/src/pybind/mgr/mgr_util.py b/src/pybind/mgr/mgr_util.py index 5d37d478de7..162946f998d 100644 --- a/src/pybind/mgr/mgr_util.py +++ b/src/pybind/mgr/mgr_util.py @@ -67,6 +67,13 @@ class PortAlreadyInUse(Exception): pass +# helper function for showing a warning text in +# the terminal +class CLIWarning(str): + def __new__(cls, content: str) -> "CLIWarning": + return super().__new__(cls, f"WARNING: {content}") + + class CephfsConnectionException(Exception): def __init__(self, error_code: int, error_message: str): self.errno = error_code diff --git a/src/pybind/mgr/orchestrator/_interface.py b/src/pybind/mgr/orchestrator/_interface.py index a505801eea5..4fbc975ae9f 100644 --- a/src/pybind/mgr/orchestrator/_interface.py +++ b/src/pybind/mgr/orchestrator/_interface.py @@ -747,6 +747,10 @@ class Orchestrator(object): """ raise NotImplementedError() + def set_osd_spec(self, service_name: str, osd_ids: List[str]) -> OrchResult: + """ set service of osd """ + raise NotImplementedError() + def blink_device_light(self, ident_fault: str, on: bool, locations: List['DeviceLightLoc']) -> OrchResult[List[str]]: """ Instructs the orchestrator to enable or disable either the ident or the fault LED. diff --git a/src/pybind/mgr/orchestrator/module.py b/src/pybind/mgr/orchestrator/module.py index 332bc75d862..d5a1bb3da2b 100644 --- a/src/pybind/mgr/orchestrator/module.py +++ b/src/pybind/mgr/orchestrator/module.py @@ -1472,6 +1472,14 @@ Usage: return HandleCommandResult(stdout=out) + @_cli_write_command('orch osd set-spec-affinity') + def _osd_set_spec(self, service_name: str, osd_id: List[str]) -> HandleCommandResult: + """Set service spec affinity for osd""" + completion = self.set_osd_spec(service_name, osd_id) + res = raise_if_exception(completion) + + return HandleCommandResult(stdout=res) + @_cli_write_command('orch daemon add') def daemon_add_misc(self, daemon_type: Optional[ServiceType] = None, @@ -1666,7 +1674,13 @@ Usage: specs: List[Union[ServiceSpec, HostSpec]] = [] # YAML '---' document separator with no content generates # None entries in the output. Let's skip them silently. - content = [o for o in yaml_objs if o is not None] + try: + content = [o for o in yaml_objs if o is not None] + except yaml.scanner.ScannerError as e: + msg = f"Invalid YAML received : {str(e)}" + self.log.exception(msg) + return HandleCommandResult(-errno.EINVAL, stderr=msg) + for s in content: try: spec = json_to_generic_spec(s) @@ -2191,7 +2205,13 @@ Usage: specs: List[TunedProfileSpec] = [] # YAML '---' document separator with no content generates # None entries in the output. Let's skip them silently. - content = [o for o in yaml_objs if o is not None] + try: + content = [o for o in yaml_objs if o is not None] + except yaml.scanner.ScannerError as e: + msg = f"Invalid YAML received : {str(e)}" + self.log.exception(msg) + return HandleCommandResult(-errno.EINVAL, stderr=msg) + for spec in content: specs.append(TunedProfileSpec.from_json(spec)) else: diff --git a/src/pybind/mgr/snap_schedule/fs/schedule_client.py b/src/pybind/mgr/snap_schedule/fs/schedule_client.py index b58f20f1275..12e5e980737 100644 --- a/src/pybind/mgr/snap_schedule/fs/schedule_client.py +++ b/src/pybind/mgr/snap_schedule/fs/schedule_client.py @@ -163,6 +163,7 @@ class SnapSchedClient(CephfsClient): self.sqlite_connections: Dict[str, DBInfo] = {} self.active_timers: Dict[Tuple[str, str], List[Timer]] = {} self.conn_lock: Lock = Lock() # lock to protect add/lookup db connections + self.timers_lock: Lock = Lock() # restart old schedules for fs_name in self.get_all_filesystems(): @@ -273,6 +274,27 @@ class SnapSchedClient(CephfsClient): if self._is_allowed_repeat(r, path)][0:1] return rows + def delete_references_to_unavailable_fs(self, available_fs_names: Set[str]) -> None: + fs_to_remove: Set[str] = set() + self.timers_lock.acquire() + for fs, path in list(self.active_timers.keys()): # each key is a tuple + if fs not in available_fs_names: + fs_to_remove.add(fs) + log.debug(f'Cancelled timers for "{fs}:{path}"') + for t in self.active_timers[(fs, path)]: + t.cancel() + log.debug(f'Removed timer instance for "{fs}"') + del self.active_timers[(fs, path)] + self.timers_lock.release() + + self.conn_lock.acquire() + for fs in fs_to_remove: + log.debug(f'Closed DB connection to "{fs}"') + self.sqlite_connections[fs].db.close() + log.debug(f'Removed DB connection to "{fs}"') + del self.sqlite_connections[fs] + self.conn_lock.release() + def refresh_snap_timers(self, fs: str, path: str, olddb: Optional[sqlite3.Connection] = None) -> None: try: log.debug((f'SnapDB on {fs} changed for {path}, ' @@ -286,6 +308,7 @@ class SnapSchedClient(CephfsClient): with self.get_schedule_db(fs) as conn_mgr: db = conn_mgr.dbinfo.db rows = self.fetch_schedules(db, path) + self.timers_lock.acquire() timers = self.active_timers.get((fs, path), []) for timer in timers: timer.cancel() @@ -299,6 +322,7 @@ class SnapSchedClient(CephfsClient): timers.append(t) log.debug(f'Will snapshot {path} in fs {fs} in {row[1]}s') self.active_timers[(fs, path)] = timers + self.timers_lock.release() except Exception: self._log_exception('refresh_snap_timers') diff --git a/src/pybind/mgr/snap_schedule/module.py b/src/pybind/mgr/snap_schedule/module.py index d8f04a62b94..adf982448b1 100644 --- a/src/pybind/mgr/snap_schedule/module.py +++ b/src/pybind/mgr/snap_schedule/module.py @@ -8,12 +8,14 @@ import json import sqlite3 from typing import Any, Dict, Optional, Tuple, Union from .fs.schedule_client import SnapSchedClient -from mgr_module import MgrModule, CLIReadCommand, CLIWriteCommand, Option +from mgr_module import MgrModule, CLIReadCommand, CLIWriteCommand, Option, NotifyType from mgr_util import CephfsConnectionException from threading import Event class Module(MgrModule): + NOTIFY_TYPES = [NotifyType.fs_map] + MODULE_OPTIONS = [ Option( 'allow_m_granularity', @@ -37,6 +39,21 @@ class Module(MgrModule): self._initialized = Event() self.client = SnapSchedClient(self) + def notify(self, notify_type: NotifyType, notify_id: str) -> None: + if notify_type != NotifyType.fs_map: + return + fs_map = self.get('fs_map') + if not fs_map: + return + + # we don't know for which fs config has been changed + fs_names = set() + for fs in fs_map['filesystems']: + fs_name = fs['mdsmap']['fs_name'] + fs_names.add(fs_name) + + self.client.delete_references_to_unavailable_fs(fs_names) + def _subvolume_exist(self, fs: str, subvol: Union[str, None], group: Union[str, None]) -> bool: rc, subvolumes, err = self.remote('volumes', 'subvolume_ls', fs, group) if rc == 0: diff --git a/src/pybind/mgr/volumes/fs/operations/volume.py b/src/pybind/mgr/volumes/fs/operations/volume.py index b2574fd76d5..93844bce119 100644 --- a/src/pybind/mgr/volumes/fs/operations/volume.py +++ b/src/pybind/mgr/volumes/fs/operations/volume.py @@ -133,7 +133,11 @@ def delete_volume(mgr, volname, metadata_pool, data_pools): r, outb, outs = remove_pool(mgr, data_pool) if r != 0: return r, outb, outs - result_str = "metadata pool: {0} data pool: {1} removed".format(metadata_pool, str(data_pools)) + result_str = f"metadata pool: {metadata_pool} data pool: {str(data_pools)} removed.\n" + result_str += "If there are active snapshot schedules associated with this " + result_str += "volume, you might see EIO errors in the mgr logs or at the " + result_str += "snap-schedule command-line due to the missing volume. " + result_str += "However, these errors are transient and will get auto-resolved." return r, result_str, "" def rename_volume(mgr, volname: str, newvolname: str) -> Tuple[int, str, str]: diff --git a/src/pybind/rbd/rbd.pyx b/src/pybind/rbd/rbd.pyx index 786c9ef0fec..bcae8cc289c 100644 --- a/src/pybind/rbd/rbd.pyx +++ b/src/pybind/rbd/rbd.pyx @@ -645,7 +645,7 @@ class RBD(object): :param p_name: the parent image name :type name: str :param p_snapshot: the parent image snapshot name or id - :type name: str + :type name: str or int :param c_ioctx: the child context that represents the new clone :type ioctx: :class:`rados.Ioctx` :param c_name: the clone (child) name diff --git a/src/python-common/ceph/cephadm/images.py b/src/python-common/ceph/cephadm/images.py index 2399cdb6dc9..5b3c7421205 100644 --- a/src/python-common/ceph/cephadm/images.py +++ b/src/python-common/ceph/cephadm/images.py @@ -1,19 +1,57 @@ # Default container images ----------------------------------------------------- -DEFAULT_PROMETHEUS_IMAGE = 'quay.io/prometheus/prometheus:v2.51.0' -DEFAULT_NODE_EXPORTER_IMAGE = 'quay.io/prometheus/node-exporter:v1.7.0' -DEFAULT_NVMEOF_IMAGE = 'quay.io/ceph/nvmeof:1.3.3' -DEFAULT_LOKI_IMAGE = 'quay.io/ceph/loki:3.0.0' -DEFAULT_PROMTAIL_IMAGE = 'quay.io/ceph/promtail:3.0.0' -DEFAULT_ALERTMANAGER_IMAGE = 'quay.io/prometheus/alertmanager:v0.27.0' -DEFAULT_GRAFANA_IMAGE = 'quay.io/ceph/grafana:10.4.8' -DEFAULT_HAPROXY_IMAGE = 'quay.io/ceph/haproxy:2.3' -DEFAULT_KEEPALIVED_IMAGE = 'quay.io/ceph/keepalived:2.2.4' -DEFAULT_SNMP_GATEWAY_IMAGE = 'quay.io/ceph/snmp-notifier:v1.2.1' -DEFAULT_ELASTICSEARCH_IMAGE = 'quay.io/omrizeneva/elasticsearch:6.8.23' -DEFAULT_JAEGER_COLLECTOR_IMAGE = 'quay.io/jaegertracing/jaeger-collector:1.29' -DEFAULT_JAEGER_AGENT_IMAGE = 'quay.io/jaegertracing/jaeger-agent:1.29' -DEFAULT_NGINX_IMAGE = 'quay.io/ceph/nginx:sclorg-nginx-126' -DEFAULT_OAUTH2_PROXY_IMAGE = 'quay.io/oauth2-proxy/oauth2-proxy:v7.6.0' -DEFAULT_JAEGER_QUERY_IMAGE = 'quay.io/jaegertracing/jaeger-query:1.29' -DEFAULT_SAMBA_IMAGE = 'quay.io/samba.org/samba-server:devbuilds-centos-amd64' -DEFAULT_SAMBA_METRICS_IMAGE = 'quay.io/samba.org/samba-metrics:latest' + +from typing import NamedTuple +from enum import Enum + + +class ContainerImage(NamedTuple): + image_ref: str # reference to default container image + key: str # image key + desc: str # description of image + + def __repr__(self) -> str: + return self.image_ref + + +def _create_image(image_ref: str, key: str) -> ContainerImage: + _img_prefix = 'container_image_' + description = key.replace('_', ' ').capitalize() + return ContainerImage( + image_ref, + f'{_img_prefix}{key}', + f'{description} container image' + ) + + +class DefaultImages(Enum): + PROMETHEUS = _create_image('quay.io/prometheus/prometheus:v2.51.0', 'prometheus') + LOKI = _create_image('docker.io/grafana/loki:3.0.0', 'loki') + PROMTAIL = _create_image('docker.io/grafana/promtail:3.0.0', 'promtail') + NODE_EXPORTER = _create_image('quay.io/prometheus/node-exporter:v1.7.0', 'node_exporter') + ALERTMANAGER = _create_image('quay.io/prometheus/alertmanager:v0.27.0', 'alertmanager') + GRAFANA = _create_image('quay.io/ceph/grafana:10.4.8', 'grafana') + HAPROXY = _create_image('quay.io/ceph/haproxy:2.3', 'haproxy') + KEEPALIVED = _create_image('quay.io/ceph/keepalived:2.2.4', 'keepalived') + NVMEOF = _create_image('quay.io/ceph/nvmeof:1.4', 'nvmeof') + SNMP_GATEWAY = _create_image('docker.io/maxwo/snmp-notifier:v1.2.1', 'snmp_gateway') + ELASTICSEARCH = _create_image('quay.io/omrizeneva/elasticsearch:6.8.23', 'elasticsearch') + JAEGER_COLLECTOR = _create_image('quay.io/jaegertracing/jaeger-collector:1.29', + 'jaeger_collector') + JAEGER_AGENT = _create_image('quay.io/jaegertracing/jaeger-agent:1.29', 'jaeger_agent') + JAEGER_QUERY = _create_image('quay.io/jaegertracing/jaeger-query:1.29', 'jaeger_query') + SAMBA = _create_image('quay.io/samba.org/samba-server:devbuilds-centos-amd64', 'samba') + SAMBA_METRICS = _create_image('quay.io/samba.org/samba-metrics:latest', 'samba_metrics') + NGINX = _create_image('quay.io/ceph/nginx:sclorg-nginx-126', 'nginx') + OAUTH2_PROXY = _create_image('quay.io/oauth2-proxy/oauth2-proxy:v7.6.0', 'oauth2_proxy') + + @property + def image_ref(self) -> str: + return self.value.image_ref + + @property + def key(self) -> str: + return self.value.key + + @property + def desc(self) -> str: + return self.value.desc diff --git a/src/python-common/ceph/deployment/service_spec.py b/src/python-common/ceph/deployment/service_spec.py index 10e9f267940..1ac9fa49e32 100644 --- a/src/python-common/ceph/deployment/service_spec.py +++ b/src/python-common/ceph/deployment/service_spec.py @@ -25,7 +25,9 @@ from typing import ( import yaml from ceph.deployment.hostspec import HostSpec, SpecValidationError, assert_valid_host -from ceph.deployment.utils import unwrap_ipv6, valid_addr +from ceph.deployment.utils import unwrap_ipv6, valid_addr, verify_non_negative_int +from ceph.deployment.utils import verify_positive_int, verify_non_negative_number +from ceph.deployment.utils import verify_boolean, verify_enum from ceph.utils import is_hex ServiceSpecT = TypeVar('ServiceSpecT', bound='ServiceSpec') @@ -1229,6 +1231,7 @@ class RGWSpec(ServiceSpec): rgw_bucket_counters_cache: Optional[bool] = False, rgw_bucket_counters_cache_size: Optional[int] = None, generate_cert: bool = False, + disable_multisite_sync_traffic: Optional[bool] = None, ): assert service_type == 'rgw', service_type @@ -1281,6 +1284,8 @@ class RGWSpec(ServiceSpec): self.rgw_bucket_counters_cache_size = rgw_bucket_counters_cache_size #: Whether we should generate a cert/key for the user if not provided self.generate_cert = generate_cert + #: Used to make RGW not do multisite replication so it can dedicate to IO + self.disable_multisite_sync_traffic = disable_multisite_sync_traffic def get_port_start(self) -> List[int]: return [self.get_port()] @@ -1328,6 +1333,7 @@ class NvmeofServiceSpec(ServiceSpec): name: Optional[str] = None, group: Optional[str] = None, addr: Optional[str] = None, + addr_map: Optional[Dict[str, str]] = None, port: Optional[int] = None, pool: Optional[str] = None, enable_auth: bool = False, @@ -1335,13 +1341,20 @@ class NvmeofServiceSpec(ServiceSpec): state_update_interval_sec: Optional[int] = 5, enable_spdk_discovery_controller: Optional[bool] = False, enable_key_encryption: Optional[bool] = True, + encryption_key: Optional[str] = None, + rebalance_period_sec: Optional[int] = 7, + max_gws_in_grp: Optional[int] = 16, + max_ns_to_change_lb_grp: Optional[int] = 8, omap_file_lock_duration: Optional[int] = 20, omap_file_lock_retries: Optional[int] = 30, omap_file_lock_retry_sleep_interval: Optional[float] = 1.0, omap_file_update_reloads: Optional[int] = 10, enable_prometheus_exporter: Optional[bool] = True, + prometheus_port: Optional[int] = 10008, + prometheus_stats_interval: Optional[int] = 10, bdevs_per_cluster: Optional[int] = 32, verify_nqns: Optional[bool] = True, + verify_keys: Optional[bool] = True, allowed_consecutive_spdk_ping_failures: Optional[int] = 1, spdk_ping_interval_in_seconds: Optional[float] = 2.0, ping_spdk_under_lock: Optional[bool] = False, @@ -1372,6 +1385,7 @@ class NvmeofServiceSpec(ServiceSpec): {"in_capsule_data_size": 8192, "max_io_qpairs_per_ctrlr": 7}, tgt_cmd_extra_args: Optional[str] = None, discovery_addr: Optional[str] = None, + discovery_addr_map: Optional[Dict[str, str]] = None, discovery_port: Optional[int] = None, log_level: Optional[str] = 'INFO', log_files_enabled: Optional[bool] = True, @@ -1406,6 +1420,8 @@ class NvmeofServiceSpec(ServiceSpec): self.pool = pool #: ``addr`` address of the nvmeof gateway self.addr = addr + #: ``addr_map`` per node address map of the nvmeof gateways + self.addr_map = addr_map #: ``port`` port of the nvmeof gateway self.port = port or 5500 #: ``name`` name of the nvmeof gateway @@ -1422,10 +1438,24 @@ class NvmeofServiceSpec(ServiceSpec): self.enable_spdk_discovery_controller = enable_spdk_discovery_controller #: ``enable_key_encryption`` encrypt DHCHAP and PSK keys before saving in OMAP self.enable_key_encryption = enable_key_encryption + #: ``encryption_key`` gateway encryption key + self.encryption_key = encryption_key + #: ``rebalance_period_sec`` number of seconds between cycles of auto namesapce rebalancing + self.rebalance_period_sec = rebalance_period_sec + #: ``max_gws_in_grp`` max number of gateways in one group + self.max_gws_in_grp = max_gws_in_grp + #: ``max_ns_to_change_lb_grp`` max number of namespaces before switching to a new lb group + self.max_ns_to_change_lb_grp = max_ns_to_change_lb_grp #: ``enable_prometheus_exporter`` enables Prometheus exporter self.enable_prometheus_exporter = enable_prometheus_exporter + #: ``prometheus_port`` Prometheus port + self.prometheus_port = prometheus_port or 10008 + #: ``prometheus_stats_interval`` Prometheus get stats interval + self.prometheus_stats_interval = prometheus_stats_interval #: ``verify_nqns`` enables verification of subsystem and host NQNs for validity self.verify_nqns = verify_nqns + #: ``verify_keys`` enables verification of PSJ and DHCHAP keys in the gateway + self.verify_keys = verify_keys #: ``omap_file_lock_duration`` number of seconds before automatically unlock OMAP file lock self.omap_file_lock_duration = omap_file_lock_duration #: ``omap_file_lock_retries`` number of retries to lock OMAP file before giving up @@ -1492,6 +1522,8 @@ class NvmeofServiceSpec(ServiceSpec): self.tgt_cmd_extra_args = tgt_cmd_extra_args #: ``discovery_addr`` address of the discovery service self.discovery_addr = discovery_addr + #: ``discovery_addr_map`` per node address map of the discovery service + self.discovery_addr_map = discovery_addr_map #: ``discovery_port`` port of the discovery service self.discovery_port = discovery_port or 8009 #: ``log_level`` the nvmeof gateway log level @@ -1518,7 +1550,7 @@ class NvmeofServiceSpec(ServiceSpec): self.monitor_client_log_file_dir = monitor_client_log_file_dir def get_port_start(self) -> List[int]: - return [5500, 4420, 8009] + return [self.port, 4420, self.discovery_port] def validate(self) -> None: # TODO: what other parameters should be validated as part of this function? @@ -1527,6 +1559,7 @@ class NvmeofServiceSpec(ServiceSpec): if not self.pool: raise SpecValidationError('Cannot add NVMEOF: No Pool specified') + verify_boolean(self.enable_auth, "Enable authentication") if self.enable_auth: if not all([self.server_key, self.server_cert, self.client_key, self.client_cert, self.root_ca_cert]): @@ -1541,142 +1574,65 @@ class NvmeofServiceSpec(ServiceSpec): if self.transports not in ['tcp']: raise SpecValidationError('Invalid transport. Valid values are tcp') - if self.log_level: - if self.log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'critical']: - raise SpecValidationError( - 'Invalid log level. Valid values are: debug, info, warning, error, critial') - - if self.spdk_log_level: - if self.spdk_log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'notice']: - raise SpecValidationError( - 'Invalid SPDK log level. Valid values are: ' - 'DEBUG, INFO, WARNING, ERROR, NOTICE') - - if self.spdk_protocol_log_level: - if self.spdk_protocol_log_level.lower() not in ['debug', - 'info', - 'warning', - 'error', - 'notice']: - raise SpecValidationError( - 'Invalid SPDK protocol log level. Valid values are: ' - 'DEBUG, INFO, WARNING, ERROR, NOTICE') + verify_enum(self.log_level, "log level", ['debug', 'info', 'warning', 'error', 'critical']) + verify_enum(self.spdk_log_level, "SPDK log level", + ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'NOTICE']) + verify_enum(self.spdk_protocol_log_level, "SPDK protocol log level", + ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'NOTICE']) + verify_positive_int(self.bdevs_per_cluster, "Bdevs per cluster") + if self.bdevs_per_cluster is not None and self.bdevs_per_cluster < 1: + raise SpecValidationError("Bdevs per cluster should be at least 1") + verify_non_negative_number(self.spdk_ping_interval_in_seconds, "SPDK ping interval") if ( - self.spdk_ping_interval_in_seconds + self.spdk_ping_interval_in_seconds is not None and self.spdk_ping_interval_in_seconds < 1.0 ): raise SpecValidationError("SPDK ping interval should be at least 1 second") + verify_non_negative_int(self.allowed_consecutive_spdk_ping_failures, + "Allowed consecutive SPDK ping failures") if ( - self.allowed_consecutive_spdk_ping_failures + self.allowed_consecutive_spdk_ping_failures is not None and self.allowed_consecutive_spdk_ping_failures < 1 ): raise SpecValidationError("Allowed consecutive SPDK ping failures should be at least 1") - if ( - self.state_update_interval_sec - and self.state_update_interval_sec < 0 - ): - raise SpecValidationError("State update interval can't be negative") - - if ( - self.omap_file_lock_duration - and self.omap_file_lock_duration < 0 - ): - raise SpecValidationError("OMAP file lock duration can't be negative") - - if ( - self.omap_file_lock_retries - and self.omap_file_lock_retries < 0 - ): - raise SpecValidationError("OMAP file lock retries can't be negative") - - if ( - self.omap_file_update_reloads - and self.omap_file_update_reloads < 0 - ): - raise SpecValidationError("OMAP file reloads can't be negative") - - if ( - self.spdk_timeout - and self.spdk_timeout < 0.0 - ): - raise SpecValidationError("SPDK timeout can't be negative") - - if ( - self.conn_retries - and self.conn_retries < 0 - ): - raise SpecValidationError("Connection retries can't be negative") - - if ( - self.max_log_file_size_in_mb - and self.max_log_file_size_in_mb < 0 - ): - raise SpecValidationError("Log file size can't be negative") - - if ( - self.max_log_files_count - and self.max_log_files_count < 0 - ): - raise SpecValidationError("Log files count can't be negative") - - if ( - self.max_log_directory_backups - and self.max_log_directory_backups < 0 - ): - raise SpecValidationError("Log file directory backups can't be negative") - - if (self.max_hosts_per_namespace and self.max_hosts_per_namespace < 0): - raise SpecValidationError("Max hosts per namespace can't be negative") - - if (self.max_namespaces_with_netmask and self.max_namespaces_with_netmask < 0): - raise SpecValidationError("Max namespaces with netmask can't be negative") - - if not isinstance(self.max_subsystems, int): - raise SpecValidationError("Max subsystems must be an integer") - - if self.max_subsystems <= 0: - raise SpecValidationError("Max subsystems must be greater than zero") - - if not isinstance(self.max_namespaces, int): - raise SpecValidationError("Max namespaces must be an integer") - - if self.max_namespaces <= 0: - raise SpecValidationError("Max namespaces must be greater than zero") - - if not isinstance(self.max_namespaces_per_subsystem, int): - raise SpecValidationError("Max namespaces per subsystem must be an integer") - - if self.max_namespaces_per_subsystem <= 0: - raise SpecValidationError("Max namespaces per subsystem must be greater than zero") - - if not isinstance(self.max_hosts_per_subsystem, int): - raise SpecValidationError("Max hosts per subsystem must be an integer") - - if self.max_hosts_per_subsystem <= 0: - raise SpecValidationError("Max hosts per subsystem must be greater than zero") - - if ( - self.monitor_timeout - and self.monitor_timeout < 0.0 - ): - raise SpecValidationError("Monitor timeout can't be negative") - - if self.port and self.port < 0: - raise SpecValidationError("Port can't be negative") - - if self.discovery_port and self.discovery_port < 0: - raise SpecValidationError("Discovery port can't be negative") + verify_non_negative_int(self.state_update_interval_sec, "State update interval") + verify_non_negative_int(self.rebalance_period_sec, "Rebalance period") + verify_non_negative_int(self.max_gws_in_grp, "Max gateways in group") + verify_non_negative_int(self.max_ns_to_change_lb_grp, + "Max namespaces to change load balancing group") + verify_non_negative_int(self.omap_file_lock_duration, "OMAP file lock duration") + verify_non_negative_number(self.omap_file_lock_retry_sleep_interval, + "OMAP file lock sleep interval") + verify_non_negative_int(self.omap_file_lock_retries, "OMAP file lock retries") + verify_non_negative_int(self.omap_file_update_reloads, "OMAP file reloads") + verify_non_negative_number(self.spdk_timeout, "SPDK timeout") + verify_non_negative_int(self.max_log_file_size_in_mb, "Log file size") + verify_non_negative_int(self.max_log_files_count, "Log files count") + verify_non_negative_int(self.max_log_directory_backups, "Log file directory backups") + verify_non_negative_int(self.max_hosts_per_namespace, "Max hosts per namespace") + verify_non_negative_int(self.max_namespaces_with_netmask, "Max namespaces with netmask") + verify_positive_int(self.max_subsystems, "Max subsystems") + verify_positive_int(self.max_namespaces, "Max namespaces") + verify_positive_int(self.max_namespaces_per_subsystem, "Max namespaces per subsystem") + verify_positive_int(self.max_hosts_per_subsystem, "Max hosts per subsystem") + verify_non_negative_number(self.monitor_timeout, "Monitor timeout") + verify_non_negative_int(self.port, "Port") + verify_non_negative_int(self.discovery_port, "Discovery port") + verify_non_negative_int(self.prometheus_port, "Prometheus port") + verify_non_negative_int(self.prometheus_stats_interval, "Prometheus stats interval") + verify_boolean(self.state_update_notify, "State update notify") + verify_boolean(self.enable_spdk_discovery_controller, "Enable SPDK discovery controller") + verify_boolean(self.enable_key_encryption, "Enable key encryption") + verify_boolean(self.enable_prometheus_exporter, "Enable Prometheus exporter") + verify_boolean(self.verify_nqns, "Verify NQNs") + verify_boolean(self.verify_keys, "Verify Keys") + verify_boolean(self.log_files_enabled, "Log files enabled") + verify_boolean(self.log_files_rotation_enabled, "Log files rotation enabled") + verify_boolean(self.verbose_log_messages, "Verbose log messages") + verify_boolean(self.enable_monitor_client, "Enable monitor client") yaml.add_representer(NvmeofServiceSpec, ServiceSpec.yaml_representer) @@ -2375,6 +2331,7 @@ class AlertManagerSpec(MonitoringSpec): user_data: Optional[Dict[str, Any]] = None, config: Optional[Dict[str, str]] = None, networks: Optional[List[str]] = None, + only_bind_port_on_networks: bool = False, port: Optional[int] = None, secure: bool = False, extra_container_args: Optional[GeneralArgList] = None, @@ -2405,6 +2362,7 @@ class AlertManagerSpec(MonitoringSpec): # <webhook_configs> configuration. self.user_data = user_data or {} self.secure = secure + self.only_bind_port_on_networks = only_bind_port_on_networks def get_port_start(self) -> List[int]: return [self.get_port(), 9094] @@ -2451,7 +2409,7 @@ class GrafanaSpec(MonitoringSpec): self.protocol = protocol # whether ports daemons for this service bind to should - # bind to only hte networks listed in networks param, or + # bind to only the networks listed in networks param, or # to all networks. Defaults to false which is saying to bind # on all networks. self.only_bind_port_on_networks = only_bind_port_on_networks diff --git a/src/python-common/ceph/deployment/utils.py b/src/python-common/ceph/deployment/utils.py index f800e373897..758eddc9412 100644 --- a/src/python-common/ceph/deployment/utils.py +++ b/src/python-common/ceph/deployment/utils.py @@ -1,7 +1,9 @@ import ipaddress import socket -from typing import Tuple, Optional +from typing import Tuple, Optional, Any from urllib.parse import urlparse +from ceph.deployment.hostspec import SpecValidationError +from numbers import Number def unwrap_ipv6(address): @@ -100,3 +102,50 @@ def valid_addr(addr: str) -> Tuple[bool, str]: if addr[0].isalpha() and '.' in addr: return _dns_lookup(addr, port) return _ip_lookup(addr, port) + + +def verify_numeric(field: Any, field_name: str) -> None: + if field is not None: + if not isinstance(field, Number) or isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be a number") + + +def verify_non_negative_int(field: Any, field_name: str) -> None: + verify_numeric(field, field_name) + if field is not None: + if not isinstance(field, int) or isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be an integer") + if field < 0: + raise SpecValidationError(f"{field_name} can't be negative") + + +def verify_positive_int(field: Any, field_name: str) -> None: + verify_non_negative_int(field, field_name) + if field is not None and field <= 0: + raise SpecValidationError(f"{field_name} must be greater than zero") + + +def verify_non_negative_number(field: Any, field_name: str) -> None: + verify_numeric(field, field_name) + if field is not None: + if field < 0.0: + raise SpecValidationError(f"{field_name} can't be negative") + + +def verify_boolean(field: Any, field_name: str) -> None: + if field is not None: + if not isinstance(field, bool): + raise SpecValidationError(f"{field_name} must be a boolean") + + +def verify_enum(field: Any, field_name: str, allowed: list) -> None: + if field: + allowed_lower = [] + if not isinstance(field, str): + raise SpecValidationError(f"{field_name} must be a string") + for val in allowed: + assert isinstance(val, str) + allowed_lower.append(val.lower()) + if field.lower() not in allowed_lower: + raise SpecValidationError( + f'Invalid {field_name}. Valid values are: {", ".join(allowed)}') diff --git a/src/rgw/CMakeLists.txt b/src/rgw/CMakeLists.txt index 329b01d2cac..41e473e23f0 100644 --- a/src/rgw/CMakeLists.txt +++ b/src/rgw/CMakeLists.txt @@ -90,6 +90,7 @@ set(librgw_common_srcs rgw_notify_event_type.cc rgw_period_history.cc rgw_period_puller.cc + rgw_s3_filter.cc rgw_pubsub.cc rgw_coroutine.cc rgw_cr_rest.cc @@ -151,6 +152,8 @@ set(librgw_common_srcs rgw_data_access.cc driver/rados/account.cc driver/rados/buckets.cc + rgw_bucket_logging.cc + rgw_rest_bucket_logging.cc driver/rados/cls_fifo_legacy.cc driver/rados/group.cc driver/rados/groups.cc @@ -484,9 +487,9 @@ target_link_libraries(radosgw PRIVATE install(TARGETS radosgw DESTINATION bin) set(radosgw_admin_srcs - rgw_admin.cc - rgw_sync_checkpoint.cc - rgw_orphan.cc) + radosgw-admin/radosgw-admin.cc + radosgw-admin/sync_checkpoint.cc + radosgw-admin/orphan.cc) # this is unsatisfying and hopefully temporary; ARROW should not be # part of radosgw_admin diff --git a/src/rgw/driver/daos/rgw_sal_daos.cc b/src/rgw/driver/daos/rgw_sal_daos.cc index a87d88c4b85..92dd7afe2fb 100644 --- a/src/rgw/driver/daos/rgw_sal_daos.cc +++ b/src/rgw/driver/daos/rgw_sal_daos.cc @@ -858,8 +858,6 @@ bool DaosZone::is_writeable() { return true; } bool DaosZone::get_redirect_endpoint(std::string* endpoint) { return false; } -bool DaosZone::has_zonegroup_api(const std::string& api) const { return false; } - const std::string& DaosZone::get_current_period_id() { return current_period->get_id(); } diff --git a/src/rgw/driver/daos/rgw_sal_daos.h b/src/rgw/driver/daos/rgw_sal_daos.h index e382fdb04ae..5515579a441 100644 --- a/src/rgw/driver/daos/rgw_sal_daos.h +++ b/src/rgw/driver/daos/rgw_sal_daos.h @@ -484,7 +484,6 @@ class DaosZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() { return zone_params->system_key; diff --git a/src/rgw/driver/dbstore/README.md b/src/rgw/driver/dbstore/README.md index 2bf3391c8c4..bcde79a2891 100644 --- a/src/rgw/driver/dbstore/README.md +++ b/src/rgw/driver/dbstore/README.md @@ -5,7 +5,7 @@ Standalone Rados Gateway (RGW) on DBStore (Experimental) ## CMake Option Add below cmake option (enabled by default) - -DWITH_RADOSGW_DBSTORE=ON + -DWITH_RADOSGW_DBSTORE=ON ## Build @@ -40,8 +40,8 @@ The location and prefix for the database files can be configured using the follo To execute DBStore unit test cases (using Gtest framework), from build directory ninja unittest_dbstore_tests - ./bin/unittest_dbstore_tests [logfile] [loglevel] - (default logfile: rgw_dbstore_tests.log, loglevel: 20) + ./bin/unittest_dbstore_tests [logfile] [loglevel] [tenantname] + (default logfile: rgw_dbstore_tests.log, loglevel: 20, default_ns_<timestamp_at_time_of_run>) ninja unittest_dbstore_mgr_tests ./bin/unittest_dbstore_mgr_tests @@ -50,4 +50,3 @@ To execute Sample test file ninja src/rgw/driver/dbstore/install ./bin/dbstore-bin [logfile] [loglevel] (default logfile: rgw_dbstore_bin.log, loglevel: 20) - diff --git a/src/rgw/driver/dbstore/tests/dbstore_tests.cc b/src/rgw/driver/dbstore/tests/dbstore_tests.cc index 2ceed7218d8..554c4d29382 100644 --- a/src/rgw/driver/dbstore/tests/dbstore_tests.cc +++ b/src/rgw/driver/dbstore/tests/dbstore_tests.cc @@ -21,7 +21,7 @@ namespace gtest { Environment(): tenant("default_ns"), db(nullptr), db_type("SQLite"), ret(-1) {} - Environment(string tenantname, string db_typename): + Environment(string tenantname, string db_typename): tenant(tenantname), db(nullptr), db_type(db_typename), ret(-1) {} @@ -153,8 +153,8 @@ TEST_F(DBStoreTest, InsertUser) { RGWAccessKey k2("id2", "key2"); params.op.user.uinfo.access_keys["id1"] = k1; params.op.user.uinfo.access_keys["id2"] = k2; - params.op.user.user_version.ver = 1; - params.op.user.user_version.tag = "UserTAG"; + params.op.user.user_version.ver = 1; + params.op.user.user_version.tag = "UserTAG"; ret = db->ProcessOp(dpp, "InsertUser", ¶ms); ASSERT_EQ(ret, 0); @@ -841,7 +841,7 @@ TEST_F(DBStoreTest, IterateObject) { TEST_F(DBStoreTest, ListBucketObjects) { struct DBOpParams params = GlobalParams; int ret = -1; - + int max = 2; bool is_truncated = false; rgw_obj_key marker1; @@ -1032,7 +1032,7 @@ TEST_F(DBStoreTest, DeleteVersionedObject) { true, &s); ASSERT_EQ(ret, -ENOENT); - /* Delete delete marker..should be able to read object now */ + /* Delete delete marker..should be able to read object now */ params.op.obj.state.obj.key.instance = dm_instance; DB::Object op_target3(db, params.op.bucket.info, params.op.obj.state.obj); DB::Object::Delete delete_op2(&op_target3); @@ -1307,13 +1307,13 @@ TEST_F(DBStoreTest, LCEntry) { ASSERT_EQ(ret, 0); // get entry index1, entry1 - ret = db->get_entry(index1, ents[0], entry); + ret = db->get_entry(index1, ents[0], entry); ASSERT_EQ(ret, 0); ASSERT_EQ(entry.status, lc_uninitial); ASSERT_EQ(entry.start_time, lc_time); // get next entry index1, entry2 - ret = db->get_next_entry(index1, ents[1], entry); + ret = db->get_next_entry(index1, ents[1], entry); ASSERT_EQ(ret, 0); ASSERT_EQ(entry.bucket, ents[2]); ASSERT_EQ(entry.status, lc_uninitial); @@ -1323,7 +1323,7 @@ TEST_F(DBStoreTest, LCEntry) { entry4.status = lc_complete; ret = db->set_entry(index2, entry4); ASSERT_EQ(ret, 0); - ret = db->get_entry(index2, ents[3], entry); + ret = db->get_entry(index2, ents[3], entry); ASSERT_EQ(ret, 0); ASSERT_EQ(entry.status, lc_complete); @@ -1337,7 +1337,7 @@ TEST_F(DBStoreTest, LCEntry) { } // remove index1, entry3 - ret = db->rm_entry(index1, entry3); + ret = db->rm_entry(index1, entry3); ASSERT_EQ(ret, 0); // get next entry index1, entry2.. should be empty @@ -1373,8 +1373,8 @@ TEST_F(DBStoreTest, InsertTestIDUser) { params.op.user.uinfo.user_email = "tester@ceph.com"; RGWAccessKey k1("0555b35654ad1656d804", "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="); params.op.user.uinfo.access_keys["0555b35654ad1656d804"] = k1; - params.op.user.user_version.ver = 1; - params.op.user.user_version.tag = "UserTAG"; + params.op.user.user_version.ver = 1; + params.op.user.user_version.tag = "UserTAG"; ret = db->ProcessOp(dpp, "InsertUser", ¶ms); ASSERT_EQ(ret, 0); @@ -1385,12 +1385,14 @@ int main(int argc, char **argv) int ret = -1; string c_logfile = "rgw_dbstore_tests.log"; int c_loglevel = 20; + string c_tenant = "default_ns_" + std::to_string(time(NULL)); - // format: ./dbstore-tests logfile loglevel - if (argc == 3) { + // format: ./dbstore-tests logfile loglevel tenantname + if (argc == 4) { c_logfile = argv[1]; c_loglevel = (atoi)(argv[2]); - cout << "logfile:" << c_logfile << ", loglevel set to " << c_loglevel << "\n"; + c_tenant = argv[3]; + cout << "logfile:" << c_logfile << ", loglevel set to " << c_loglevel << ", db is " << c_tenant << "\n"; } ::testing::InitGoogleTest(&argc, argv); @@ -1398,6 +1400,7 @@ int main(int argc, char **argv) gtest::env = new gtest::Environment(); gtest::env->logfile = c_logfile; gtest::env->loglevel = c_loglevel; + gtest::env->tenant = c_tenant; ::testing::AddGlobalTestEnvironment(gtest::env); ret = RUN_ALL_TESTS(); diff --git a/src/rgw/driver/motr/rgw_sal_motr.cc b/src/rgw/driver/motr/rgw_sal_motr.cc index b999673ac18..463ea8c5b11 100644 --- a/src/rgw/driver/motr/rgw_sal_motr.cc +++ b/src/rgw/driver/motr/rgw_sal_motr.cc @@ -1111,11 +1111,6 @@ bool MotrZone::get_redirect_endpoint(std::string* endpoint) return false; } -bool MotrZone::has_zonegroup_api(const std::string& api) const -{ - return (zonegroup.group.api_name == api); -} - const std::string& MotrZone::get_current_period_id() { return current_period->get_id(); diff --git a/src/rgw/driver/motr/rgw_sal_motr.h b/src/rgw/driver/motr/rgw_sal_motr.h index f92074b9d94..0f99ae48e86 100644 --- a/src/rgw/driver/motr/rgw_sal_motr.h +++ b/src/rgw/driver/motr/rgw_sal_motr.h @@ -525,7 +525,6 @@ class MotrZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() { return zone_params->system_key; } virtual const std::string& get_realm_name() { return realm->get_name(); } diff --git a/src/rgw/driver/posix/notify.h b/src/rgw/driver/posix/notify.h index 9f6088a893a..4463abc57c2 100644 --- a/src/rgw/driver/posix/notify.h +++ b/src/rgw/driver/posix/notify.h @@ -212,7 +212,7 @@ namespace file::listing { void signal_shutdown() { uint64_t msg{sig_shutdown}; - (void) write(efd, &msg, sizeof(uint64_t)); + std::ignore = write(efd, &msg, sizeof(uint64_t)); } friend class Notify; diff --git a/src/rgw/driver/posix/rgw_sal_posix.cc b/src/rgw/driver/posix/rgw_sal_posix.cc index 1345468210f..9d76462baa0 100644 --- a/src/rgw/driver/posix/rgw_sal_posix.cc +++ b/src/rgw/driver/posix/rgw_sal_posix.cc @@ -2893,6 +2893,14 @@ int POSIXObject::copy_object(const ACLOwner& owner, return dobj->set_obj_attrs(dpp, &attrs, nullptr, y, rgw::sal::FLAG_LOG_OP); } +int POSIXObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + return -EOPNOTSUPP; +} + int POSIXObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { int ret = stat(dpp); diff --git a/src/rgw/driver/posix/rgw_sal_posix.h b/src/rgw/driver/posix/rgw_sal_posix.h index 8ec72bbc1bc..bf3478ad6ab 100644 --- a/src/rgw/driver/posix/rgw_sal_posix.h +++ b/src/rgw/driver/posix/rgw_sal_posix.h @@ -653,6 +653,13 @@ public: const DoutPrefixProvider* dpp, optional_yield y) override; virtual RGWAccessControlPolicy& get_acl(void) override { return acls; } virtual int set_acl(const RGWAccessControlPolicy& acl) override { acls = acl; return 0; } + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; diff --git a/src/rgw/driver/rados/rgw_bucket.cc b/src/rgw/driver/rados/rgw_bucket.cc index 21d238d3341..d043aea0783 100644 --- a/src/rgw/driver/rados/rgw_bucket.cc +++ b/src/rgw/driver/rados/rgw_bucket.cc @@ -169,7 +169,8 @@ int RGWBucket::init(rgw::sal::Driver* _driver, RGWBucketAdminOpState& op_state, driver = _driver; - std::string bucket_name = op_state.get_bucket_name(); + auto bucket_name = op_state.get_bucket_name(); + auto bucket_id = op_state.get_bucket_id(); if (bucket_name.empty() && op_state.get_user_id().empty()) return -EINVAL; @@ -184,7 +185,7 @@ int RGWBucket::init(rgw::sal::Driver* _driver, RGWBucketAdminOpState& op_state, bucket_name = bucket_name.substr(pos + 1); } - int r = driver->load_bucket(dpp, rgw_bucket(tenant, bucket_name), + int r = driver->load_bucket(dpp, rgw_bucket(tenant, bucket_name, bucket_id), &bucket, y); if (r < 0) { set_err_msg(err_msg, "failed to fetch bucket info for bucket=" + bucket_name); @@ -1140,6 +1141,16 @@ int RGWBucketAdminOp::dump_s3_policy(rgw::sal::Driver* driver, RGWBucketAdminOpS int RGWBucketAdminOp::unlink(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_state, const DoutPrefixProvider *dpp, optional_yield y, string *err) { + rgw_owner owner; + if (op_state.is_account_op()) { + owner = op_state.get_account_id(); + } else if (op_state.is_user_op()) { + owner = op_state.get_user_id(); + } else { + set_err_msg(err, "requires user or account id"); + return -EINVAL; + } + auto radosdriver = dynamic_cast<rgw::sal::RadosStore*>(driver); if (!radosdriver) { set_err_msg(err, "rados store only"); @@ -1152,13 +1163,18 @@ int RGWBucketAdminOp::unlink(rgw::sal::Driver* driver, RGWBucketAdminOpState& op return ret; auto* rados = radosdriver->getRados()->get_rados_handle(); - return radosdriver->ctl()->bucket->unlink_bucket(*rados, op_state.get_user_id(), op_state.get_bucket()->get_info().bucket, y, dpp, true); + return radosdriver->ctl()->bucket->unlink_bucket(*rados, owner, op_state.get_bucket()->get_info().bucket, y, dpp, true); } int RGWBucketAdminOp::link(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_state, const DoutPrefixProvider *dpp, optional_yield y, string *err) { - if (!op_state.is_user_op()) { - set_err_msg(err, "empty user id"); + rgw_owner new_owner; + if (op_state.is_account_op()) { + new_owner = op_state.get_account_id(); + } else if (op_state.is_user_op()) { + new_owner = op_state.get_user_id(); + } else { + set_err_msg(err, "requires user or account id"); return -EINVAL; } auto radosdriver = dynamic_cast<rgw::sal::RadosStore*>(driver); @@ -1172,8 +1188,26 @@ int RGWBucketAdminOp::link(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_s if (ret < 0) return ret; + std::string display_name; + if (op_state.is_account_op()) { + RGWAccountInfo info; + rgw::sal::Attrs attrs; + RGWObjVersionTracker objv; + ret = driver->load_account_by_id(dpp, y, op_state.get_account_id(), + info, attrs, objv); + if (ret < 0) { + set_err_msg(err, "failed to load account"); + return ret; + } + display_name = std::move(info.name); + } else if (!bucket.get_user()->get_info().account_id.empty()) { + set_err_msg(err, "account users cannot own buckets. use --account-id instead"); + return -EINVAL; + } else { + display_name = bucket.get_user()->get_display_name(); + } + string bucket_id = op_state.get_bucket_id(); - std::string display_name = op_state.get_user_display_name(); std::unique_ptr<rgw::sal::Bucket> loc_bucket; std::unique_ptr<rgw::sal::Bucket> old_bucket; @@ -1187,7 +1221,7 @@ int RGWBucketAdminOp::link(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_s old_bucket = loc_bucket->clone(); - loc_bucket->get_key().tenant = op_state.get_user_id().tenant; + loc_bucket->get_key().tenant = op_state.get_tenant(); if (!op_state.new_bucket_name.empty()) { auto pos = op_state.new_bucket_name.find('/'); @@ -1236,14 +1270,14 @@ int RGWBucketAdminOp::link(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_s } RGWAccessControlPolicy policy_instance; - policy_instance.create_default(op_state.get_user_id(), display_name); + policy_instance.create_default(new_owner, display_name); owner = policy_instance.get_owner(); aclbl.clear(); policy_instance.encode(aclbl); bool exclusive = false; - loc_bucket->get_info().owner = op_state.get_user_id(); + loc_bucket->get_info().owner = new_owner; if (*loc_bucket != *old_bucket) { loc_bucket->get_info().bucket = loc_bucket->get_key(); loc_bucket->get_info().objv_tracker.version_for_read()->ver = 0; @@ -1259,13 +1293,13 @@ int RGWBucketAdminOp::link(rgw::sal::Driver* driver, RGWBucketAdminOpState& op_s /* link to user */ RGWBucketEntryPoint ep; ep.bucket = loc_bucket->get_info().bucket; - ep.owner = op_state.get_user_id(); + ep.owner = new_owner; ep.creation_time = loc_bucket->get_info().creation_time; ep.linked = true; rgw::sal::Attrs ep_attrs; rgw_ep_info ep_data{ep, ep_attrs}; - r = radosdriver->ctl()->bucket->link_bucket(*rados, op_state.get_user_id(), loc_bucket->get_info().bucket, loc_bucket->get_info().creation_time, y, dpp, true, &ep_data); + r = radosdriver->ctl()->bucket->link_bucket(*rados, new_owner, loc_bucket->get_info().bucket, loc_bucket->get_info().creation_time, y, dpp, true, &ep_data); if (r < 0) { set_err_msg(err, "failed to relink bucket"); return r; diff --git a/src/rgw/driver/rados/rgw_bucket.h b/src/rgw/driver/rados/rgw_bucket.h index 85434ba7299..9ee31c8814e 100644 --- a/src/rgw/driver/rados/rgw_bucket.h +++ b/src/rgw/driver/rados/rgw_bucket.h @@ -361,6 +361,7 @@ public: void clear_failure() { failure = false; } const RGWBucketInfo& get_bucket_info() const { return bucket->get_info(); } + rgw::sal::User* get_user() { return user.get(); } }; class RGWBucketAdminOp { diff --git a/src/rgw/driver/rados/rgw_data_sync.cc b/src/rgw/driver/rados/rgw_data_sync.cc index d5437f548c1..c0a9059a251 100644 --- a/src/rgw/driver/rados/rgw_data_sync.cc +++ b/src/rgw/driver/rados/rgw_data_sync.cc @@ -2617,6 +2617,7 @@ class RGWUserPermHandler { rgw::IAM::Environment env; std::unique_ptr<rgw::auth::Identity> identity; RGWAccessControlPolicy user_acl; + std::vector<rgw::IAM::Policy> user_policies; }; std::shared_ptr<_info> info; @@ -2644,7 +2645,7 @@ class RGWUserPermHandler { } auto result = rgw::auth::transform_old_authinfo( - sync_env->dpp, null_yield, sync_env->driver, user.get()); + sync_env->dpp, null_yield, sync_env->driver, user.get(), &info->user_policies); if (!result) { return result.error(); } @@ -2679,6 +2680,7 @@ public: std::shared_ptr<_info> info; RGWAccessControlPolicy bucket_acl; std::optional<perm_state> ps; + boost::optional<rgw::IAM::Policy> bucket_policy; public: Bucket() {} @@ -2686,9 +2688,7 @@ public: const RGWBucketInfo& bucket_info, const map<string, bufferlist>& bucket_attrs); - bool verify_bucket_permission(int perm); - bool verify_object_permission(const map<string, bufferlist>& obj_attrs, - int perm); + bool verify_bucket_permission(const rgw_obj_key& obj_key, const uint64_t op); }; static int policy_from_attrs(CephContext *cct, @@ -2728,6 +2728,14 @@ int RGWUserPermHandler::Bucket::init(RGWUserPermHandler *handler, return r; } + // load bucket policy + try { + bucket_policy = get_iam_policy_from_attr(sync_env->cct, bucket_attrs, bucket_info.bucket.tenant); + } catch (const std::exception& e) { + ldpp_dout(sync_env->dpp, 0) << "ERROR: reading IAM Policy: " << e.what() << dendl; + return -EACCES; + } + ps.emplace(sync_env->cct, info->env, info->identity.get(), @@ -2740,36 +2748,40 @@ int RGWUserPermHandler::Bucket::init(RGWUserPermHandler *handler, return 0; } -bool RGWUserPermHandler::Bucket::verify_bucket_permission(int perm) -{ - return verify_bucket_permission_no_policy(sync_env->dpp, - &(*ps), - info->user_acl, - bucket_acl, - perm); -} - -bool RGWUserPermHandler::Bucket::verify_object_permission(const map<string, bufferlist>& obj_attrs, - int perm) +bool RGWUserPermHandler::Bucket::verify_bucket_permission(const rgw_obj_key& obj_key, const uint64_t op) { - RGWAccessControlPolicy obj_acl; - - int r = policy_from_attrs(sync_env->cct, obj_attrs, &obj_acl); - if (r < 0) { - return r; - } - - return verify_bucket_permission_no_policy(sync_env->dpp, - &(*ps), - bucket_acl, - obj_acl, - perm); + const rgw_obj obj(ps->bucket_info.bucket, obj_key); + const auto arn = rgw::ARN(obj); + + if (ps->identity->get_account()) { + const bool account_root = (ps->identity->get_identity_type() == TYPE_ROOT); + if (!ps->identity->is_owner_of(bucket_acl.get_owner().id)) { + ldpp_dout(sync_env->dpp, 4) << "cross-account request for bucket owner " + << bucket_acl.get_owner().id << " != " << ps->identity->get_aclowner().id << dendl; + // cross-account requests evaluate the identity-based policies separately + // from the resource-based policies and require Allow from both + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, {}, {}, {}, + info->user_policies, {}, op) + && ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, false, info->user_acl, + bucket_acl, bucket_policy, {}, {}, op); + } else { + // don't consult acls for same-account access. require an Allow from + // either identity- or resource-based policy + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, {}, {}, + bucket_policy, info->user_policies, + {}, op); + } + } + constexpr bool account_root = false; + return ::verify_bucket_permission(sync_env->dpp, &(*ps), arn, account_root, + info->user_acl, bucket_acl, + bucket_policy, info->user_policies, + {}, op); } class RGWFetchObjFilter_Sync : public RGWFetchObjFilter_Default { rgw_bucket_sync_pipe sync_pipe; - std::shared_ptr<RGWUserPermHandler::Bucket> bucket_perms; std::optional<rgw_sync_pipe_dest_params> verify_dest_params; std::optional<ceph::real_time> mtime; @@ -2782,10 +2794,8 @@ class RGWFetchObjFilter_Sync : public RGWFetchObjFilter_Default { public: RGWFetchObjFilter_Sync(rgw_bucket_sync_pipe& _sync_pipe, - std::shared_ptr<RGWUserPermHandler::Bucket>& _bucket_perms, std::optional<rgw_sync_pipe_dest_params>&& _verify_dest_params, std::shared_ptr<bool>& _need_retry) : sync_pipe(_sync_pipe), - bucket_perms(_bucket_perms), verify_dest_params(std::move(_verify_dest_params)), need_retry(_need_retry) { *need_retry = false; @@ -2852,12 +2862,6 @@ int RGWFetchObjFilter_Sync::filter(CephContext *cct, *poverride_owner = acl_translation_owner; } } - if (params.mode == rgw_sync_pipe_params::MODE_USER) { - if (!bucket_perms->verify_object_permission(obj_attrs, RGW_PERM_READ)) { - ldout(cct, 0) << "ERROR: " << __func__ << ": permission check failed: user not allowed to fetch object" << dendl; - return -EPERM; - } - } if (!dest_placement_rule && params.dest.storage_class) { @@ -2900,7 +2904,6 @@ class RGWObjFetchCR : public RGWCoroutine { rgw_sync_pipe_params::Mode param_mode; std::optional<RGWUserPermHandler> user_perms; - std::shared_ptr<RGWUserPermHandler::Bucket> source_bucket_perms; RGWUserPermHandler::Bucket dest_bucket_perms; std::optional<rgw_sync_pipe_dest_params> dest_params; @@ -3016,20 +3019,10 @@ public: return set_cr_error(retcode); } - if (!dest_bucket_perms.verify_bucket_permission(RGW_PERM_WRITE)) { + if (!dest_bucket_perms.verify_bucket_permission(dest_key.value_or(key), rgw::IAM::s3PutObject)) { ldout(cct, 0) << "ERROR: " << __func__ << ": permission check failed: user not allowed to write into bucket (bucket=" << sync_pipe.info.dest_bucket.get_key() << ")" << dendl; return -EPERM; } - - /* init source bucket permission structure */ - source_bucket_perms = make_shared<RGWUserPermHandler::Bucket>(); - r = user_perms->init_bucket(sync_pipe.source_bucket_info, - sync_pipe.source_bucket_attrs, - source_bucket_perms.get()); - if (r < 0) { - ldout(cct, 20) << "ERROR: " << __func__ << ": failed to init bucket perms manager for uid=" << *param_user << " bucket=" << sync_pipe.source_bucket_info.bucket.get_key() << dendl; - return set_cr_error(retcode); - } } yield { @@ -3037,12 +3030,11 @@ public: need_retry = make_shared<bool>(); } auto filter = make_shared<RGWFetchObjFilter_Sync>(sync_pipe, - source_bucket_perms, std::move(dest_params), need_retry); call(new RGWFetchRemoteObjCR(sync_env->async_rados, sync_env->driver, sc->source_zone, - nullopt, + param_user, sync_pipe.source_bucket_info.bucket, std::nullopt, sync_pipe.dest_bucket_info, key, dest_key, versioned_epoch, @@ -4528,7 +4520,7 @@ public: } tn->set_resource_name(SSTR(bucket_str_noinstance(bs.bucket) << "/" << key)); } - if (retcode == -ERR_PRECONDITION_FAILED) { + if (retcode == -ERR_PRECONDITION_FAILED || retcode == -EPERM) { pretty_print(sc->env, "Skipping object s3://{}/{} in sync from zone {}\n", bs.bucket.name, key, zone_name); set_status("Skipping object sync: precondition failed (object contains newer change or policy doesn't allow sync)"); @@ -6052,13 +6044,12 @@ int RGWSyncBucketCR::operate(const DoutPrefixProvider *dpp) } else { tn->log(20, SSTR("logged prev gen entry (bucket=" << source_bs.bucket << ", shard_id=" << source_bs.shard_id << ", gen=" << current_gen << " in error repo: retcode=" << retcode)); } - } else { + } retcode = -EAGAIN; tn->log(10, SSTR("ERROR: requested sync of future generation " << *gen << " > " << current_gen << ", returning " << retcode << " for later retry")); return set_cr_error(retcode); - } } else if (*gen < current_gen) { tn->log(10, SSTR("WARNING: requested sync of past generation " << *gen << " < " << current_gen diff --git a/src/rgw/driver/rados/rgw_datalog.cc b/src/rgw/driver/rados/rgw_datalog.cc index 4c9503071ef..d7e57d7e1c1 100644 --- a/src/rgw/driver/rados/rgw_datalog.cc +++ b/src/rgw/driver/rados/rgw_datalog.cc @@ -576,7 +576,7 @@ int RGWDataChangesLog::renew_entries(const DoutPrefixProvider *dpp) if (ret < 0) { /* we don't really need to have a special handling for failed cases here, * as this is just an optimization. */ - ldpp_dout(dpp, -1) << "ERROR: svc.cls->timelog.add() returned " << ret << dendl; + ldpp_dout(dpp, -1) << "ERROR: be->push() returned " << ret << dendl; return ret; } diff --git a/src/rgw/driver/rados/rgw_period.cc b/src/rgw/driver/rados/rgw_period.cc index f18e8e46bc5..aacb9b6a09a 100644 --- a/src/rgw/driver/rados/rgw_period.cc +++ b/src/rgw/driver/rados/rgw_period.cc @@ -68,20 +68,6 @@ int RGWPeriod::delete_obj(const DoutPrefixProvider *dpp, optional_yield y) return ret; } -int RGWPeriod::add_zonegroup(const DoutPrefixProvider *dpp, const RGWZoneGroup& zonegroup, optional_yield y) -{ - if (zonegroup.realm_id != realm_id) { - return 0; - } - int ret = period_map.update(zonegroup, cct); - if (ret < 0) { - ldpp_dout(dpp, 0) << "ERROR: updating period map: " << cpp_strerror(-ret) << dendl; - return ret; - } - - return store_info(dpp, false, y); -} - int RGWPeriod::update(const DoutPrefixProvider *dpp, optional_yield y) { auto zone_svc = sysobj_svc->get_zone_svc(); diff --git a/src/rgw/driver/rados/rgw_pubsub_push.cc b/src/rgw/driver/rados/rgw_pubsub_push.cc index 07d65fa1028..d22c61e9b08 100644 --- a/src/rgw/driver/rados/rgw_pubsub_push.cc +++ b/src/rgw/driver/rados/rgw_pubsub_push.cc @@ -281,7 +281,7 @@ public: conn_id, _endpoint, get_bool(args, "use-ssl", false), get_bool(args, "verify-ssl", true), args.get_optional("ca-location"), args.get_optional("mechanism"), args.get_optional("user-name"), - args.get_optional("password"))) { + args.get_optional("password"), args.get_optional("kafka-brokers"))) { throw configuration_error("Kafka: failed to create connection to: " + _endpoint); } @@ -434,4 +434,3 @@ void RGWPubSubEndpoint::shutdown_all() { #endif shutdown_http_manager(); } - diff --git a/src/rgw/driver/rados/rgw_putobj_processor.cc b/src/rgw/driver/rados/rgw_putobj_processor.cc index f04ed1db8d4..9e27c5adbc9 100644 --- a/src/rgw/driver/rados/rgw_putobj_processor.cc +++ b/src/rgw/driver/rados/rgw_putobj_processor.cc @@ -597,6 +597,11 @@ int MultipartObjectProcessor::complete( } if (r < 0) { + if (r == -ETIMEDOUT) { + // The meta_obj_ref write may eventually succeed, clear the set of objects for deletion. if it + // doesn't ever succeed, we'll orphan any tail objects as if we'd crashed before that write + writer.clear_written(); + } return r == -ENOENT ? -ERR_NO_SUCH_UPLOAD : r; } @@ -783,6 +788,11 @@ int AppendObjectProcessor::complete( attrs, rctx, writer.get_trace(), flags & rgw::sal::FLAG_LOG_OP); if (r < 0) { + if (r == -ETIMEDOUT) { + // The head object write may eventually succeed, clear the set of objects for deletion. if it + // doesn't ever succeed, we'll orphan any tail objects as if we'd crashed before that write + writer.clear_written(); + } return r; } if (!obj_op.meta.canceled) { diff --git a/src/rgw/driver/rados/rgw_rados.cc b/src/rgw/driver/rados/rgw_rados.cc index a133b54dc59..69075c506f1 100644 --- a/src/rgw/driver/rados/rgw_rados.cc +++ b/src/rgw/driver/rados/rgw_rados.cc @@ -1930,11 +1930,58 @@ int RGWRados::Bucket::List::list_objects_ordered( ": finished due to getting past requested namespace \"" << params.ns << "\"" << dendl; goto done; - } + } else if (!obj.ns.empty()) { + // We're in the namespace range and we're enforcing an empty + // namespace, therefore we can skip past a congtiguous chunk + // of namespaced entries. Namespaces are demarcated in the + // index key by underscores before and after the namespace + // name (e.g., "_somenamespace_somekey"). Also, regular + // entries might begin with an underscore, in which case + // they're escaped with another underscore (e.g., "_foobar" + // is encoded as "__foobar"). We also have to account for + // the fact that in lexical ordering there are characters + // both before underscore (e.g., uppercase letters) and + // after (e.g., lowercase letters). So that means there can + // be five distinct and meaningful regions in the lexical + // ordering of entries, which we'll use examples to help + // illustrate: + + // 1. FOOBAR (regular pre-underscore) + // 2. _BAZ_foobar (namespaced, with namespace pre-underscore) + // 3. __foobar (regular with escaped underscore) + // 4. _baz_foobar (namespaced, with namespace post-underscore) + // 5. foobar (regular, post-underscore) + + // So if we're skipping namespaces and recognize we're in + // region 2, we must skip to region 3. And if we recognize + // we're in region 4, we skip to region 5. + rgw_obj_index_key potential_marker; + if (obj.ns[0] < '_') { + // We're in region 2, so need to skip to region 3. The + // caret (^) is the ASCII character that preceeds + // underscore, so we'll set the marker to the + // caret/circumflex followed by 0xFF, so the key after can + // be in the double underscore range. + potential_marker = rgw_obj_index_key("_^\xFF"); + } else { + // we're passed the escaped underscore region (i.e., + // starting with two underscores), so we can skip past the + // underscore region + potential_marker = rgw_obj_index_key("_\xFF"); + } + + if (cur_marker < potential_marker) { + ldpp_dout(dpp, 20) << __func__ << + ": skipping past region of namespaced entries, starting with \"" << + entry.key << "\"" << dendl; + cur_marker = potential_marker; + break; // leave inner loop (for) and allow another cls call + } + } - /* we're skipping past namespaced objects */ + // we're skipping past namespaced objects ldpp_dout(dpp, 20) << __func__ << - ": skipping past namespaced objects, including \"" << entry.key << + ": skipping past individual namespaced entry \"" << entry.key << "\"" << dendl; continue; } @@ -1955,7 +2002,7 @@ int RGWRados::Bucket::List::list_objects_ordered( if (params.access_list_filter && !params.access_list_filter(obj.name, index_key.name)) { ldpp_dout(dpp, 20) << __func__ << - ": skipping past namespaced objects, including \"" << entry.key << + ": skipping past filtered out entry \"" << entry.key << "\"" << dendl; continue; } @@ -3343,12 +3390,17 @@ int RGWRados::Object::Write::_do_write_meta(uint64_t size, uint64_t accounted_si return 0; done_cancel: - int ret = index_op->cancel(rctx.dpp, meta.remove_objs, rctx.y, log_op); - if (ret < 0) { - ldpp_dout(rctx.dpp, 0) << "ERROR: index_op.cancel() returned ret=" << ret << dendl; - } + // if r == -ETIMEDOUT, rgw can't determine whether or not the rados op succeeded + // we shouldn't be calling index_op->cancel() in this case + // Instead, we should leave that pending entry in the index so than bucket listing can recover with check_disk_state() and cls_rgw_suggest_changes() + if (r != -ETIMEDOUT) { + int ret = index_op->cancel(rctx.dpp, meta.remove_objs, rctx.y, log_op); + if (ret < 0) { + ldpp_dout(rctx.dpp, 0) << "ERROR: index_op.cancel() returned ret=" << ret << dendl; + } - meta.canceled = true; + meta.canceled = true; + } /* we lost in a race. There are a few options: * - existing object was rewritten (ECANCELED) @@ -5252,13 +5304,7 @@ int RGWRados::restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx, ceph::real_time restore_time = real_clock::now(); { - char buf[32]; - utime_t ut(restore_time); - snprintf(buf, sizeof(buf), "%lld.%09lld", - (long long)ut.sec(), - (long long)ut.nsec()); bufferlist bl; - bl.append(buf, 32); encode(restore_time, bl); attrs[RGW_ATTR_RESTORE_TIME] = std::move(bl); } @@ -5278,13 +5324,7 @@ int RGWRados::restore_obj_from_cloud(RGWLCCloudTierCtx& tier_ctx, delete_at = expiration_date; { - char buf[32]; - utime_t ut(expiration_date); - snprintf(buf, sizeof(buf), "%lld.%09lld", - (long long)ut.sec(), - (long long)ut.nsec()); bufferlist bl; - bl.append(buf, 32); encode(expiration_date, bl); attrs[RGW_ATTR_RESTORE_EXPIRY_DATE] = std::move(bl); } @@ -5445,7 +5485,7 @@ int RGWRados::delete_bucket(RGWBucketInfo& bucket_info, RGWObjVersionTracker& ob } /* if the bucket is not synced we can remove the meta file */ - if (!svc.zone->is_syncing_bucket_meta(bucket)) { + if (!svc.zone->is_syncing_bucket_meta()) { RGWObjVersionTracker objv_tracker; r = ctl.bucket->remove_bucket_instance_info(bucket, bucket_info, y, dpp); if (r < 0) { @@ -6105,7 +6145,11 @@ int RGWRados::Object::Delete::delete_obj(optional_yield y, const DoutPrefixProvi const bool need_invalidate = (r == -ECANCELED); int64_t poolid = ioctx.get_id(); - if (r >= 0) { + if (r == -ETIMEDOUT) { + // rgw can't determine whether or not the delete succeeded, shouldn't be calling either of complete_del() or cancel() + // leaving that pending entry in the index so that bucket listing can recover with check_disk_state() and cls_rgw_suggest_changes() + ldpp_dout(dpp, 0) << "ERROR: rgw_rados_operate returned r=" << r << dendl; + } else if (r >= 0 || r == -ENOENT) { tombstone_cache_t *obj_tombstone_cache = store->get_tombstone_cache(); if (obj_tombstone_cache) { tombstone_entry entry{*state}; @@ -6918,13 +6962,13 @@ int RGWRados::set_attrs(const DoutPrefixProvider *dpp, RGWObjectCtx* octx, RGWBu } return 0; -} +} /* RGWRados::set_attrs() */ -static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, - RGWRados* store, RGWBucketInfo& bucket_info, - RGWObjectCtx* rctx, RGWObjManifest* manifest, - int part_num, int* parts_count, bool prefetch, - RGWObjState** pstate, RGWObjManifest** pmanifest) +int RGWRados::get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, + RGWRados* store, RGWBucketInfo& bucket_info, + RGWObjectCtx* rctx, RGWObjManifest* manifest, + int part_num, int* parts_count, bool prefetch, + RGWObjState** pstate, RGWObjManifest** pmanifest) { if (!manifest) { return -ERR_INVALID_PART; @@ -7003,6 +7047,9 @@ static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, // update the object size sm->state.size = part_manifest.get_obj_size(); + if (!sm->state.attrset.count(RGW_ATTR_COMPRESSION)) { + sm->state.accounted_size = sm->state.size; + } *pmanifest = &part_manifest; return 0; diff --git a/src/rgw/driver/rados/rgw_rados.h b/src/rgw/driver/rados/rgw_rados.h index b24823b60dc..fe79916392f 100644 --- a/src/rgw/driver/rados/rgw_rados.h +++ b/src/rgw/driver/rados/rgw_rados.h @@ -1071,6 +1071,12 @@ public: }; // class RGWRados::Bucket::List }; // class RGWRados::Bucket + static int get_part_obj_state(const DoutPrefixProvider* dpp, optional_yield y, + RGWRados* store, RGWBucketInfo& bucket_info, + RGWObjectCtx* rctx, RGWObjManifest* manifest, + int part_num, int* parts_count, bool prefetch, + RGWObjState** pstate, RGWObjManifest** pmanifest); + int on_last_entry_in_listing(const DoutPrefixProvider *dpp, RGWBucketInfo& bucket_info, const std::string& obj_prefix, diff --git a/src/rgw/driver/rados/rgw_rest_bucket.cc b/src/rgw/driver/rados/rgw_rest_bucket.cc index dc71e40335f..0c3f7029604 100644 --- a/src/rgw/driver/rados/rgw_rest_bucket.cc +++ b/src/rgw/driver/rados/rgw_rest_bucket.cc @@ -141,6 +141,7 @@ void RGWOp_Bucket_Link::execute(optional_yield y) RGWBucketAdminOpState op_state; RESTArgs::get_string(s, "uid", uid_str, &uid_str); + RESTArgs::get_string(s, "account-id", op_state.account_id, &op_state.account_id); RESTArgs::get_string(s, "bucket", bucket, &bucket); RESTArgs::get_string(s, "bucket-id", bucket_id, &bucket_id); RESTArgs::get_string(s, "new-bucket-name", new_bucket_name, &new_bucket_name); @@ -184,6 +185,7 @@ void RGWOp_Bucket_Unlink::execute(optional_yield y) RESTArgs::get_string(s, "uid", uid_str, &uid_str); rgw_user uid(uid_str); + RESTArgs::get_string(s, "account-id", op_state.account_id, &op_state.account_id); RESTArgs::get_string(s, "bucket", bucket, &bucket); op_state.set_user_id(uid); diff --git a/src/rgw/driver/rados/rgw_sal_rados.cc b/src/rgw/driver/rados/rgw_sal_rados.cc index 11b86a25841..4c05421653b 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.cc +++ b/src/rgw/driver/rados/rgw_sal_rados.cc @@ -13,8 +13,11 @@ * */ +#include <asm-generic/errno-base.h> #include <errno.h> +#include <fmt/core.h> #include <stdlib.h> +#include <string> #include <system_error> #include <filesystem> #include <unistd.h> @@ -26,9 +29,12 @@ #include "include/function2.hpp" #include "common/Clock.h" +#include "common/ceph_time.h" #include "common/errno.h" #include "role.h" +#include "rgw_obj_types.h" +#include "rgw_rados.h" #include "rgw_sal.h" #include "rgw_sal_rados.h" #include "rgw_bucket.h" @@ -56,6 +62,7 @@ #include "rgw_rest_realm.h" #include "rgw_rest_user.h" #include "rgw_lc_tier.h" +#include "rgw_bucket_logging.h" #include "services/svc_sys_obj.h" #include "services/svc_mdlog.h" #include "services/svc_cls.h" @@ -422,6 +429,10 @@ int RadosBucket::remove(const DoutPrefixProvider* dpp, ldpp_dout(dpp, -1) << "ERROR: unable to remove notifications from bucket. ret=" << ps_ret << dendl; } + if (ret = rgw::bucketlogging::bucket_deletion_cleanup(dpp, store, this, y); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: could not cleanup bucket logging configuration and pending objects, ret = " << ret << dendl; + } + ret = store->ctl()->bucket->unlink_bucket(rados, info.owner, info.bucket, y, dpp, false); if (ret < 0) { @@ -716,7 +727,7 @@ int RadosBucket::merge_and_store_attrs(const DoutPrefixProvider* dpp, Attrs& new attrs[it.first] = it.second; } return store->ctl()->bucket->set_bucket_instance_attrs(get_info(), - new_attrs, &get_info().objv_tracker, y, dpp); + attrs, &get_info().objv_tracker, y, dpp); } int RadosBucket::try_refresh_info(const DoutPrefixProvider* dpp, ceph::real_time* pmtime, optional_yield y) @@ -1017,6 +1028,281 @@ int RadosBucket::remove_topics(RGWObjVersionTracker* objv_tracker, objv_tracker, y); } +int RadosBucket::get_logging_object_name(std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) { + rgw_pool data_pool; + const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix); + if (!store->getRados()->get_obj_data_pool(get_placement_rule(), rgw_obj{get_key(), obj_name_oid}, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when getting logging object name" << dendl; + return -EIO; + } + bufferlist bl; + const int ret = rgw_get_system_obj(store->svc()->sysobj, + data_pool, + obj_name_oid, + bl, + objv_tracker, + nullptr, + y, + dpp, + nullptr, + nullptr); + if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get logging object name from '" << obj_name_oid << "'. ret = " << ret << dendl; + return ret; + } + obj_name = bl.to_str(); + return 0; +} + +int RadosBucket::set_logging_object_name(const std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + bool new_obj, + RGWObjVersionTracker* objv_tracker) { + rgw_pool data_pool; + const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix); + if (!store->getRados()->get_obj_data_pool(get_placement_rule(), rgw_obj{get_key(), obj_name_oid}, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when setting logging object name" << dendl; + return -EIO; + } + bufferlist bl; + bl.append(obj_name); + const int ret = rgw_put_system_obj(dpp, store->svc()->sysobj, + data_pool, + obj_name_oid, + bl, + new_obj, + objv_tracker, + ceph::real_time::clock::now(), + y, + nullptr); + if (ret == -EEXIST) { + ldpp_dout(dpp, 20) << "INFO: race detected in initializing '" << obj_name_oid << "' with logging object name:'" << obj_name << "'. ret = " << ret << dendl; + } else if (ret == -ECANCELED) { + ldpp_dout(dpp, 20) << "INFO: race detected in updating logging object name '" << obj_name << "' at '" << obj_name_oid << "'. ret = " << ret << dendl; + } else if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to set logging object name '" << obj_name << "' at '" << obj_name_oid << "'. ret = " << ret << dendl; + } + return ret; +} + +int RadosBucket::remove_logging_object_name(const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) { + rgw_pool data_pool; + const auto obj_name_oid = bucketlogging::object_name_oid(this, prefix); + if (!store->getRados()->get_obj_data_pool(get_placement_rule(), rgw_obj{get_key(), obj_name_oid}, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when setting logging object name" << dendl; + return -EIO; + } + return rgw_delete_system_obj(dpp, store->svc()->sysobj, + data_pool, + obj_name_oid, + objv_tracker, + y); +} + +std::string to_temp_object_name(const rgw::sal::Bucket* bucket, const std::string& obj_name) { + return fmt::format("{}__shadow_{}0", + bucket->get_bucket_id(), + obj_name); +} + +int RadosBucket::remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) { + rgw_pool data_pool; + const rgw_obj head_obj{get_key(), obj_name}; + const auto placement_rule = get_placement_rule(); + + if (!store->getRados()->get_obj_data_pool(placement_rule, head_obj, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when deleting logging object" << dendl; + return -EIO; + } + + const auto temp_obj_name = to_temp_object_name(this, obj_name); + return rgw_delete_system_obj(dpp, store->svc()->sysobj, + data_pool, + temp_obj_name, + nullptr, + y); +} + +int RadosBucket::commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) { + rgw_pool data_pool; + const rgw_obj head_obj{get_key(), obj_name}; + const auto placement_rule = get_placement_rule(); + + if (!store->getRados()->get_obj_data_pool(placement_rule, head_obj, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when comitting logging object" << dendl; + return -EIO; + } + + const auto temp_obj_name = to_temp_object_name(this, obj_name); + std::map<string, bufferlist> obj_attrs; + ceph::real_time mtime; + bufferlist bl_data; + if (const auto ret = rgw_get_system_obj(store->svc()->sysobj, + data_pool, + temp_obj_name, + bl_data, + nullptr, + &mtime, + y, + dpp, + &obj_attrs, + nullptr); ret < 0 && ret != -ENOENT) { + ldpp_dout(dpp, 1) << "ERROR: failed to read logging data when comitting object '" << temp_obj_name + << ". error: " << ret << dendl; + return ret; + } else if (ret == -ENOENT) { + ldpp_dout(dpp, 1) << "WARNING: temporary logging object '" << temp_obj_name << "' does not exists" << dendl; + return 0; + } + + uint64_t size = bl_data.length(); + const uint64_t max_obj_size = store->ctx()->_conf->osd_max_object_size; + RGWObjManifest manifest; + manifest.set_prefix(obj_name); + manifest.set_trivial_rule(0, max_obj_size); + RGWObjManifest::generator manifest_gen; + if (const auto ret = manifest_gen.create_begin(store->ctx(), &manifest, + placement_rule, + nullptr, // no special placment for tail + get_key(), + head_obj); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to create manifest when comitting logging object. error: " << + ret << dendl; + return ret; + } + + if (const auto ret = manifest_gen.create_next(size); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to add object to manifest when comitting logging object. error: " << + ret << dendl; + return ret; + } + + if (const auto expected_temp_obj = manifest_gen.get_cur_obj(store->getRados()); + temp_obj_name != expected_temp_obj.oid) { + // TODO: cleanup temporary object, commit would never succeed + ldpp_dout(dpp, 1) << "ERROR: temporary logging object name mismatch: '" << + temp_obj_name << "' != '" << expected_temp_obj.oid << "'" << dendl; + return -EINVAL; + } + + RGWObjectCtx obj_ctx(store); + obj_ctx.set_atomic(head_obj); + const auto& bucket_info = get_info(); + RGWRados::Object rgw_head_obj(store->getRados(), + bucket_info, + obj_ctx, + head_obj); + // disable versioning on the logging objects + rgw_head_obj.set_versioning_disabled(true); + RGWRados::Object::Write head_obj_wop(&rgw_head_obj); + head_obj_wop.meta.manifest = &manifest; + head_obj_wop.meta.bucket_owner = bucket_info.owner; + head_obj_wop.meta.flags = PUT_OBJ_CREATE; + head_obj_wop.meta.mtime = &mtime; + // TODO: head_obj_wop.meta.ptag + // the owner of the logging object is the bucket owner + // not the user that wrote the log that triggered the commit + const ACLOwner owner{bucket_info.owner, ""}; // TODO: missing display name + head_obj_wop.meta.owner = owner; + const auto etag = TOPNSPC::crypto::digest<TOPNSPC::crypto::MD5>(bl_data).to_str(); + bufferlist bl_etag; + bl_etag.append(etag.c_str()); + obj_attrs.emplace(RGW_ATTR_ETAG, std::move(bl_etag)); + const req_context rctx{dpp, y, nullptr}; + jspan_context trace{false, false}; + if (const auto ret = head_obj_wop.write_meta(0, size, obj_attrs, rctx, trace); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to commit logging object '" << temp_obj_name << + "' to bucket id '" << get_info().bucket <<"'. error: " << ret << dendl; + return ret; + } + ldpp_dout(dpp, 20) << "INFO: committed logging object '" << temp_obj_name << + "' with size of " << size << " bytes, to bucket '" << get_key() << "' as '" << + obj_name << "'" << dendl; + return 0; +} + +struct BucketLoggingCompleteArg { + BucketLoggingCompleteArg(const std::string& _obj_name, size_t _size, CephContext* _cct) + : obj_name{_obj_name}, size{_size}, cct{_cct} {} + const std::string obj_name; + const size_t size; + CephContext* cct; +}; + +void bucket_logging_completion(rados_completion_t completion, void* args) { + auto* aio_comp = reinterpret_cast<librados::AioCompletionImpl*>(completion); + std::unique_ptr<BucketLoggingCompleteArg> logging_args(reinterpret_cast<BucketLoggingCompleteArg*>(args)); + if (aio_comp->get_return_value() < 0) { + ldout(logging_args->cct, 1) << "ERROR: failed to complete append to logging object '" << logging_args->obj_name << + "'. ret = " << aio_comp->get_return_value() << dendl; + } else { + ldout(logging_args->cct, 20) << "INFO: wrote " << logging_args->size << " bytes to logging object '" << + logging_args->obj_name << "'" << dendl; + } +} + +int RadosBucket::write_logging_object(const std::string& obj_name, + const std::string& record, + optional_yield y, + const DoutPrefixProvider *dpp, + bool async_completion) { + const auto temp_obj_name = to_temp_object_name(this, obj_name); + rgw_pool data_pool; + rgw_obj obj{get_key(), obj_name}; + if (!store->getRados()->get_obj_data_pool(get_placement_rule(), obj, &data_pool)) { + ldpp_dout(dpp, 1) << "ERROR: failed to get data pool for bucket '" << get_name() << + "' when writing logging object" << dendl; + return -EIO; + } + librados::IoCtx io_ctx; + if (const auto ret = rgw_init_ioctx(dpp, store->getRados()->get_rados_handle(), data_pool, io_ctx); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get IO context for logging object from data pool:" << data_pool.to_str() << dendl; + return -EIO; + } + bufferlist bl; + bl.append(record); + bl.append("\n"); + // append the record to the temporary object + // if this is the first record, the object will be created + librados::ObjectWriteOperation op; + op.append(bl); + if (async_completion) { + aio_completion_ptr completion{librados::Rados::aio_create_completion()}; + auto arg = std::make_unique<BucketLoggingCompleteArg>(temp_obj_name, record.length(), store->ctx()); + completion->set_complete_callback(arg.get(), bucket_logging_completion); + if (const auto ret = io_ctx.aio_operate(temp_obj_name, completion.get(), &op); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to append to logging object '" << temp_obj_name << + "'. ret = " << ret << dendl; + return ret; + } + std::ignore = arg.release(); + std::ignore = completion.release(); + return 0; + } + if (const auto ret = rgw_rados_operate(dpp, io_ctx, temp_obj_name, &op, y); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to append to logging object '" << temp_obj_name << + "'. ret = " << ret << dendl; + return ret; + } + ldpp_dout(dpp, 20) << "INFO: wrote " << record.length() << " bytes to logging object '" << + temp_obj_name << "'" << dendl; + return 0; +} + std::unique_ptr<User> RadosStore::get_user(const rgw_user &u) { return std::make_unique<RadosUser>(this, u); @@ -1652,7 +1938,7 @@ int RadosStore::read_topics(const std::string& tenant, rgw_pubsub_topics& topics } int RadosStore::stat_topics_v1(const std::string& tenant, optional_yield y, const DoutPrefixProvider *dpp) { - return rgw_stat_system_obj(dpp, svc()->sysobj, svc()->zone->get_zone_params().log_pool, topics_oid(tenant), nullptr, nullptr, y, nullptr); + return rgw_stat_system_obj(dpp, svc()->sysobj, svc()->zone->get_zone_params().log_pool, topics_oid(tenant), nullptr, nullptr, nullptr, y, nullptr); } int RadosStore::write_topics(const std::string& tenant, const rgw_pubsub_topics& topics, RGWObjVersionTracker* objv_tracker, @@ -2228,7 +2514,108 @@ bool RadosObject::is_sync_completed(const DoutPrefixProvider* dpp, const rgw_bi_log_entry& earliest_marker = entries.front(); return earliest_marker.timestamp > obj_mtime; -} +} /* is_sync_completed */ + +int RadosObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + int ret{0}; + + /* require an object with a manifest, so call to get_obj_state() must precede this */ + if (! manifest) { + return -EINVAL; + } + + RGWObjManifest::obj_iterator end = manifest->obj_end(dpp); + if (end.get_cur_part_id() == 0) { // not multipart + ldpp_dout(dpp, 20) << __func__ << " object does not have a multipart manifest" + << dendl; + return 0; + } + + auto end_part_id = end.get_cur_part_id(); + auto parts_count = (end_part_id == 1) ? 1 : end_part_id - 1; + if (marker > (parts_count - 1)) { + return 0; + } + + RGWObjManifest::obj_iterator part_iter = manifest->obj_begin(dpp); + + if (marker != 0) { + ldpp_dout_fmt(dpp, 20, + "{} seeking to part #{} in the object manifest", + __func__, marker); + + part_iter = manifest->obj_find_part(dpp, marker + 1); + + if (part_iter == end) { + ldpp_dout_fmt(dpp, 5, + "{} failed to find part #{} in the object manifest", + __func__, marker + 1); + return 0; + } + } + + RGWObjectCtx& obj_ctx = get_ctx(); + RGWBucketInfo& bucket_info = get_bucket()->get_info(); + + Object::Part obj_part{}; + for (; part_iter != manifest->obj_end(dpp); ++part_iter) { + + /* we're only interested in the first object in each logical part */ + auto cur_part_id = part_iter.get_cur_part_id(); + if (cur_part_id == obj_part.part_number) { + continue; + } + + if (max_parts < 1) { + *truncated = true; + break; + } + + /* get_part_obj_state alters the passed manifest** to point to a part + * manifest, which we don't want to leak out here */ + RGWObjManifest* obj_m = manifest; + RGWObjState* astate; + bool part_prefetch = false; + ret = RGWRados::get_part_obj_state(dpp, y, store->getRados(), bucket_info, &obj_ctx, + obj_m, cur_part_id, &parts_count, + part_prefetch, &astate, &obj_m); + + if (ret < 0) { + ldpp_dout_fmt(dpp, 4, + "{} get_part_obj_state() failed ret={}", + __func__, ret); + break; + } + + obj_part.part_number = part_iter.get_cur_part_id(); + obj_part.part_size = astate->accounted_size; + + if (auto iter = astate->attrset.find(RGW_ATTR_CKSUM); + iter != astate->attrset.end()) { + try { + rgw::cksum::Cksum part_cksum; + auto ck_iter = iter->second.cbegin(); + part_cksum.decode(ck_iter); + obj_part.cksum = std::move(part_cksum); + } catch (buffer::error& err) { + ldpp_dout_fmt(dpp, 4, + "WARN: {} could not decode stored cksum, " + "caught buffer::error", + __func__); + } + } + + each_func(obj_part); + *next_marker = ++marker; + --max_parts; + } /* each part */ + + return ret; +} /* RadosObject::list_parts */ int RadosObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { @@ -3412,7 +3799,7 @@ int RadosMultipartUpload::init(const DoutPrefixProvider *dpp, optional_yield y, multipart_upload_info upload_info; upload_info.dest_placement = dest_placement; upload_info.cksum_type = cksum_type; - + if (obj_legal_hold) { upload_info.obj_legal_hold_exist = true; upload_info.obj_legal_hold = (*obj_legal_hold); @@ -4257,11 +4644,6 @@ bool RadosZone::get_redirect_endpoint(std::string* endpoint) return true; } -bool RadosZone::has_zonegroup_api(const std::string& api) const -{ - return store->svc()->zone->has_zonegroup_api(api); -} - const std::string& RadosZone::get_current_period_id() { return store->svc()->zone->get_current_period_id(); @@ -4508,8 +4890,8 @@ void RadosLuaManager::handle_reload_notify(const DoutPrefixProvider* dpp, option #ifdef WITH_RADOSGW_LUA_PACKAGES rgw::lua::packages_t failed_packages; std::string install_dir; - auto r = rgw::lua::install_packages(dpp, store, - y, store->ctx()->_conf.get_val<std::string>("rgw_luarocks_location"), + auto r = rgw::lua::install_packages(dpp, store, + y, store->ctx()->_conf.get_val<std::string>("rgw_luarocks_location"), failed_packages, install_dir); if (r < 0) { ldpp_dout(dpp, 1) << "WARNING: failed to install Lua packages from allowlist. error code: " << r @@ -4520,9 +4902,9 @@ void RadosLuaManager::handle_reload_notify(const DoutPrefixProvider* dpp, option ldpp_dout(dpp, 5) << "WARNING: failed to install Lua package: " << p << " from allowlist" << dendl; } -#else +#else const int r = 0; -#endif +#endif ack_reload(dpp, notify_id, cookie, r); } @@ -4544,7 +4926,7 @@ int RadosLuaManager::reload_packages(const DoutPrefixProvider *dpp, optional_yie << ". error: " << cpp_strerror(r) << dendl; return r; } - + std::vector<librados::notify_ack_t> acks; std::vector<librados::notify_timeout_t> timeouts; ioctx.decode_notify_response(reply_bl, &acks, &timeouts); @@ -4558,7 +4940,7 @@ int RadosLuaManager::reload_packages(const DoutPrefixProvider *dpp, optional_yie auto iter = ack.payload_bl.cbegin(); ceph::decode(r, iter); } catch (buffer::error& err) { - ldpp_dout(dpp, 1) << "ERROR: couldn't decode Lua packages reload status. error: " << + ldpp_dout(dpp, 1) << "ERROR: couldn't decode Lua packages reload status. error: " << err.what() << dendl; return -EINVAL; } diff --git a/src/rgw/driver/rados/rgw_sal_rados.h b/src/rgw/driver/rados/rgw_sal_rados.h index be681c9f975..e65c3c0050e 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.h +++ b/src/rgw/driver/rados/rgw_sal_rados.h @@ -107,7 +107,6 @@ class RadosZone : public StoreZone { virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() override; virtual const std::string& get_realm_name() override; @@ -593,12 +592,18 @@ class RadosObject : public StoreObject { StoreObject::set_compressed(); } - virtual bool is_sync_completed(const DoutPrefixProvider* dpp, const ceph::real_time& obj_mtime) override; /* For rgw_admin.cc */ RGWObjState& get_state() { return state; } virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; virtual int get_obj_attrs(optional_yield y, const DoutPrefixProvider* dpp, rgw_obj* target_obj = NULL) override; virtual int modify_obj_attrs(const char* attr_name, bufferlist& attr_val, optional_yield y, const DoutPrefixProvider* dpp) override; @@ -775,6 +780,24 @@ class RadosBucket : public StoreBucket { optional_yield y, const DoutPrefixProvider *dpp) override; int remove_topics(RGWObjVersionTracker* objv_tracker, optional_yield y, const DoutPrefixProvider *dpp) override; + int get_logging_object_name(std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override; + int set_logging_object_name(const std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + bool new_obj, + RGWObjVersionTracker* objv_tracker) override; + int remove_logging_object_name(const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override; + int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override; + int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override; + int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override; private: int link(const DoutPrefixProvider* dpp, const rgw_owner& new_owner, optional_yield y, bool update_entrypoint = true, RGWObjVersionTracker* objv = nullptr); diff --git a/src/rgw/driver/rados/rgw_tools.cc b/src/rgw/driver/rados/rgw_tools.cc index f5cd193d815..bf7a309e864 100644 --- a/src/rgw/driver/rados/rgw_tools.cc +++ b/src/rgw/driver/rados/rgw_tools.cc @@ -155,7 +155,7 @@ int rgw_put_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj, int rgw_stat_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj, const rgw_pool& pool, const std::string& key, RGWObjVersionTracker *objv_tracker, - real_time *pmtime, optional_yield y, + real_time *pmtime, uint64_t *psize, optional_yield y, std::map<std::string, bufferlist> *pattrs) { rgw_raw_obj obj(pool, key); @@ -163,6 +163,7 @@ int rgw_stat_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj, return sysobj.rop() .set_attrs(pattrs) .set_last_mod(pmtime) + .set_obj_size(psize) .stat(y, dpp); } @@ -185,7 +186,7 @@ int rgw_get_system_obj(RGWSI_SysObj* svc_sysobj, const rgw_pool& pool, const str .read(dpp, &bl, y); } -int rgw_delete_system_obj(const DoutPrefixProvider *dpp, +int rgw_delete_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj *sysobj_svc, const rgw_pool& pool, const string& oid, RGWObjVersionTracker *objv_tracker, optional_yield y) { @@ -338,21 +339,35 @@ int rgw_list_pool(const DoutPrefixProvider *dpp, ldpp_dout(dpp, 10) << "failed to parse cursor: " << marker << dendl; return -EINVAL; } - - auto iter = ioctx.nobjects_begin(oc); + librados::NObjectIterator iter; + try { + iter = ioctx.nobjects_begin(oc); + } catch (const std::system_error& e) { + ldpp_dout(dpp, 1) << "rgw_list_pool: Failed to begin iteration of pool " + << ioctx.get_pool_name() << " with error " + << e.what() << dendl; + return ceph::from_error_code(e.code()); + } /// Pool_iterate if (iter == ioctx.nobjects_end()) return -ENOENT; - for (; oids->size() < max && iter != ioctx.nobjects_end(); ++iter) { - string oid = iter->get_oid(); - ldpp_dout(dpp, 20) << "RGWRados::pool_iterate: got " << oid << dendl; + try { + for (; oids->size() < max && iter != ioctx.nobjects_end(); ++iter) { + string oid = iter->get_oid(); + ldpp_dout(dpp, 20) << "RGWRados::pool_iterate: got " << oid << dendl; - // fill it in with initial values; we may correct later - if (filter && !filter(oid, oid)) - continue; + // fill it in with initial values; we may correct later + if (filter && !filter(oid, oid)) + continue; - oids->push_back(oid); + oids->push_back(oid); + } + } catch (const std::system_error& e) { + ldpp_dout(dpp, 1) << "rgw_list_pool: Failed iterating pool " + << ioctx.get_pool_name() << " with error " + << e.what() << dendl; + return ceph::from_error_code(e.code()); } marker = iter.get_cursor().to_str(); diff --git a/src/rgw/driver/rados/rgw_tools.h b/src/rgw/driver/rados/rgw_tools.h index 016da256263..b86d280a4a3 100644 --- a/src/rgw/driver/rados/rgw_tools.h +++ b/src/rgw/driver/rados/rgw_tools.h @@ -76,13 +76,13 @@ int rgw_get_system_obj(RGWSI_SysObj* svc_sysobj, const rgw_pool& pool, rgw_cache_entry_info *cache_info = nullptr, boost::optional<obj_version> refresh_version = boost::none, bool raw_attrs=false); -int rgw_delete_system_obj(const DoutPrefixProvider *dpp, +int rgw_delete_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj *sysobj_svc, const rgw_pool& pool, const std::string& oid, RGWObjVersionTracker *objv_tracker, optional_yield y); int rgw_stat_system_obj(const DoutPrefixProvider *dpp, RGWSI_SysObj* svc_sysobj, const rgw_pool& pool, const std::string& key, RGWObjVersionTracker *objv_tracker, - real_time *pmtime, optional_yield y, + real_time *pmtime, uint64_t *psize, optional_yield y, std::map<std::string, bufferlist> *pattrs = nullptr); const char *rgw_find_mime_by_ext(std::string& ext); diff --git a/src/rgw/driver/rados/rgw_user.cc b/src/rgw/driver/rados/rgw_user.cc index 94a18ffcbab..894d8e40950 100644 --- a/src/rgw/driver/rados/rgw_user.cc +++ b/src/rgw/driver/rados/rgw_user.cc @@ -1755,7 +1755,11 @@ int RGWUser::execute_add(const DoutPrefixProvider *dpp, RGWUserAdminOpState& op_ user_info.display_name = display_name; user_info.type = TYPE_RGW; - // tenant must not look like a valid account id + // user/tenant must not look like a valid account id + if (rgw::account::validate_id(uid.id)) { + set_err_msg(err_msg, "uid must not be formatted as an account id"); + return -EINVAL; + } if (rgw::account::validate_id(uid.tenant)) { set_err_msg(err_msg, "tenant must not be formatted as an account id"); return -EINVAL; diff --git a/src/rgw/driver/rados/rgw_user.h b/src/rgw/driver/rados/rgw_user.h index ab157f38e39..4ae7d13eff7 100644 --- a/src/rgw/driver/rados/rgw_user.h +++ b/src/rgw/driver/rados/rgw_user.h @@ -19,11 +19,11 @@ #define RGW_USER_ANON_ID "anonymous" -#define SECRET_KEY_LEN 40 -#define PUBLIC_ID_LEN 20 -#define RAND_SUBUSER_LEN 5 +constexpr auto SECRET_KEY_LEN=40; +constexpr auto PUBLIC_ID_LEN=20; +constexpr auto RAND_SUBUSER_LEN=5; -#define XMLNS_AWS_S3 "http://s3.amazonaws.com/doc/2006-03-01/" +constexpr auto XMLNS_AWS_S3 = "http://s3.amazonaws.com/doc/2006-03-01/"; class RGWUserCtl; class RGWBucketCtl; diff --git a/src/rgw/driver/rados/rgw_zone.h b/src/rgw/driver/rados/rgw_zone.h index c542abc76d6..5fb2b4b8096 100644 --- a/src/rgw/driver/rados/rgw_zone.h +++ b/src/rgw/driver/rados/rgw_zone.h @@ -769,7 +769,6 @@ public: int create(const DoutPrefixProvider *dpp, optional_yield y, bool exclusive = true); int delete_obj(const DoutPrefixProvider *dpp, optional_yield y); int store_info(const DoutPrefixProvider *dpp, bool exclusive, optional_yield y); - int add_zonegroup(const DoutPrefixProvider *dpp, const RGWZoneGroup& zonegroup, optional_yield y); void fork(); int update(const DoutPrefixProvider *dpp, optional_yield y); diff --git a/src/rgw/rgw_orphan.cc b/src/rgw/radosgw-admin/orphan.cc index b7dc562c721..9fca3b99a7c 100644 --- a/src/rgw/rgw_orphan.cc +++ b/src/rgw/radosgw-admin/orphan.cc @@ -1,6 +1,12 @@ + +/* + * Copyright (C) 2024 IBM +*/ + // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab ft=cpp +#include "radosgw-admin/orphan.h" #include <string> @@ -10,7 +16,6 @@ #include "rgw_op.h" #include "rgw_multi.h" -#include "rgw_orphan.h" #include "rgw_zone.h" #include "rgw_bucket.h" #include "rgw_sal_rados.h" diff --git a/src/rgw/rgw_orphan.h b/src/rgw/radosgw-admin/orphan.h index db811d31d9a..db811d31d9a 100644 --- a/src/rgw/rgw_orphan.h +++ b/src/rgw/radosgw-admin/orphan.h diff --git a/src/rgw/rgw_admin.cc b/src/rgw/radosgw-admin/radosgw-admin.cc index b00dfaa1ec5..47b68d3f902 100644 --- a/src/rgw/rgw_admin.cc +++ b/src/rgw/radosgw-admin/radosgw-admin.cc @@ -1,12 +1,15 @@ +/* + * Copyright (C) 2025 IBM +*/ + // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab ft=cpp -#include <errno.h> -#include <iostream> -#include <sstream> +#include <cerrno> #include <string> - -#include <boost/optional.hpp> +#include <sstream> +#include <optional> +#include <iostream> extern "C" { #include <liboath/oath.h> @@ -38,6 +41,9 @@ extern "C" { #include "include/utime.h" #include "include/str_list.h" +#include "radosgw-admin/orphan.h" +#include "radosgw-admin/sync_checkpoint.h" + #include "rgw_user.h" #include "rgw_otp.h" #include "rgw_rados.h" @@ -48,7 +54,6 @@ extern "C" { #include "rgw_log.h" #include "rgw_formats.h" #include "rgw_usage.h" -#include "rgw_orphan.h" #include "rgw_sync.h" #include "rgw_trim_bilog.h" #include "rgw_trim_datalog.h" @@ -62,12 +67,12 @@ extern "C" { #include "rgw_zone.h" #include "rgw_pubsub.h" #include "rgw_bucket_sync.h" -#include "rgw_sync_checkpoint.h" #include "rgw_lua.h" #include "rgw_sal.h" #include "rgw_sal_config.h" #include "rgw_data_access.h" #include "rgw_account.h" +#include "rgw_bucket_logging.h" #include "services/svc_sync_modules.h" #include "services/svc_cls.h" @@ -81,11 +86,6 @@ extern "C" { #define dout_context g_ceph_context -#define SECRET_KEY_LEN 40 -#define PUBLIC_ID_LEN 20 - -using namespace std; - static rgw::sal::Driver* driver = NULL; static constexpr auto dout_subsys = ceph_subsys_rgw; @@ -116,19 +116,13 @@ static const DoutPrefixProvider* dpp() { } \ } while (0) -static inline int posix_errortrans(int r) +using namespace std; + +inline int posix_errortrans(int r) { - switch(r) { - case ERR_NO_SUCH_BUCKET: - r = ENOENT; - break; - default: - break; - } - return r; + return ERR_NO_SUCH_BUCKET == r ? ENOENT : r; } - static const std::string LUA_CONTEXT_LIST("prerequest, postrequest, background, getdata, putdata"); void usage() @@ -177,6 +171,8 @@ void usage() cout << " bucket sync disable disable bucket sync\n"; cout << " bucket sync enable enable bucket sync\n"; cout << " bucket radoslist list rados objects backing bucket's objects\n"; + cout << " bucket logging flush flush pending log records object of source bucket to the log bucket\n"; + cout << " bucket logging info get info on bucket logging configuration on source bucket or list of sources in log bucket\n"; cout << " bi get retrieve bucket index object entries\n"; cout << " bi put store bucket index object entries\n"; cout << " bi list list raw bucket index entries\n"; @@ -359,6 +355,7 @@ void usage() cout << " --secret/--secret-key=<key> specify secret key\n"; cout << " --gen-access-key generate random access key (for S3)\n"; cout << " --gen-secret generate random secret key\n"; + cout << " --generate-key create user with or without credentials\n"; cout << " --key-type=<type> key type, options are: swift, s3\n"; cout << " --key-active=<bool> activate or deactivate a key\n"; cout << " --temp-url-key[-2]=<key> temp url key\n"; @@ -704,6 +701,8 @@ enum class OPT { BUCKET_SHARD_OBJECTS, BUCKET_OBJECT_SHARD, BUCKET_RESYNC_ENCRYPTED_MULTIPART, + BUCKET_LOGGING_FLUSH, + BUCKET_LOGGING_INFO, POLICY, LOG_LIST, LOG_SHOW, @@ -942,6 +941,8 @@ static SimpleCmd::Commands all_cmds = { { "bucket shard object", OPT::BUCKET_SHARD_OBJECTS }, { "bucket object shard", OPT::BUCKET_OBJECT_SHARD }, { "bucket resync encrypted multipart", OPT::BUCKET_RESYNC_ENCRYPTED_MULTIPART }, + { "bucket logging flush", OPT::BUCKET_LOGGING_FLUSH }, + { "bucket logging info", OPT::BUCKET_LOGGING_INFO }, { "policy", OPT::POLICY }, { "log list", OPT::LOG_LIST }, { "log show", OPT::LOG_SHOW }, @@ -1267,7 +1268,7 @@ static int read_input(const string& infile, bufferlist& bl) } } -#define READ_CHUNK 8196 + constexpr auto READ_CHUNK=8196; int r; int err; @@ -2549,35 +2550,104 @@ std::ostream& operator<<(std::ostream& out, const indented& h) { return out << std::setw(h.w) << h.header << std::setw(1) << ' '; } -static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::RadosStore* driver, const RGWZone& zone, +struct bucket_source_sync_info { + const RGWZone& _source; + std::string_view error; + std::map<int,std::string> shards_behind; + int total_shards; + std::string_view status; + rgw_bucket bucket_source; + + bucket_source_sync_info(const RGWZone& source): _source(source) {} + + void _print_plaintext(std::ostream& out, int width) const { + out << indented{width, "source zone"} << _source.id << " (" << _source.name << ")" << std::endl; + if (!error.empty()) { + out << indented{width} << error << std::endl; + return; + } + out << indented{width, "source bucket"} << bucket_source << std::endl; + if (!status.empty()) { + out << indented{width} << status << std::endl; + return; + } + out << indented{width} << "incremental sync on " << total_shards << " shards\n"; + if (!shards_behind.empty()) { + out << indented{width} << "bucket is behind on " << shards_behind.size() << " shards\n"; + set<int> shard_ids; + for (auto const& [shard_id, _] : shards_behind) { + shard_ids.insert(shard_id); + } + out << indented{width} << "behind shards: [" << shard_ids << "]\n"; + } else { + out << indented{width} << "bucket is caught up with source\n"; + } + } + + void _print_formatter(std::ostream& out, Formatter* formatter) const { + formatter->open_object_section("source"); + formatter->dump_string("source_zone", _source.id); + formatter->dump_string("source_name", _source.name); + + if (!error.empty()) { + formatter->dump_string("error", error); + formatter->close_section(); + formatter->flush(out); + return; + } + + formatter->dump_string("source_bucket", bucket_source.name); + formatter->dump_string("source_bucket_id", bucket_source.bucket_id); + + if (!status.empty()) { + formatter->dump_string("status", status); + formatter->close_section(); + formatter->flush(out); + return; + } + + formatter->dump_int("total_shards", total_shards); + formatter->open_array_section("behind_shards"); + for (auto const& [id, marker] : shards_behind) { + formatter->open_object_section("shard"); + formatter->dump_int("shard_id", id); + formatter->dump_string("shard_marker", marker); + formatter->close_section(); + } + formatter->close_section(); + formatter->close_section(); + formatter->flush(out); + } +}; + +static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::RadosStore* driver, + const RGWZone& zone, const RGWZone& source, RGWRESTConn *conn, const RGWBucketInfo& bucket_info, rgw_sync_bucket_pipe pipe, - int width, std::ostream& out) + bucket_source_sync_info& source_sync_info) { - out << indented{width, "source zone"} << source.id << " (" << source.name << ")" << std::endl; - // syncing from this zone? if (!driver->svc()->zone->zone_syncs_from(zone, source)) { - out << indented{width} << "does not sync from zone\n"; + source_sync_info.error = "does not sync from zone"; return 0; } if (!pipe.source.bucket) { - ldpp_dout(dpp, -1) << __func__ << "(): missing source bucket" << dendl; + source_sync_info.error = fmt::format("{} (): missing source bucket", __func__); return -EINVAL; } std::unique_ptr<rgw::sal::Bucket> source_bucket; int r = init_bucket(*pipe.source.bucket, &source_bucket); if (r < 0) { - ldpp_dout(dpp, -1) << "failed to read source bucket info: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read source bucket info: {}", cpp_strerror(r)); return r; } - out << indented{width, "source bucket"} << source_bucket->get_key() << std::endl; - pipe.source.bucket = source_bucket->get_key(); + source_sync_info.bucket_source = source_bucket->get_key(); + pipe.source.bucket = source_bucket->get_key(); pipe.dest.bucket = bucket_info.bucket; uint64_t gen = 0; @@ -2588,15 +2658,15 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra r = rgw_read_bucket_full_sync_status(dpp, driver, pipe, &full_status, null_yield); if (r >= 0) { if (full_status.state == BucketSyncState::Init) { - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } if (full_status.state == BucketSyncState::Stopped) { - out << indented{width} << "stopped: bucket sync is disabled\n"; + source_sync_info.status = "stopped: bucket sync is disabled"; return 0; } if (full_status.state == BucketSyncState::Full) { - out << indented{width} << "full sync: " << full_status.full.count << " objects completed\n"; + source_sync_info.status = fmt::format("full sync: {} objects completed", full_status.full.count); return 0; } gen = full_status.incremental_gen; @@ -2605,46 +2675,45 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra // no full status, but there may be per-shard status from before upgrade const auto& logs = source_bucket->get_info().layout.logs; if (logs.empty()) { - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } const auto& log = logs.front(); if (log.gen > 0) { // this isn't the backward-compatible case, so we just haven't started yet - out << indented{width} << "init: bucket sync has not started\n"; + source_sync_info.status = "init: bucket sync has not started"; return 0; } if (log.layout.type != rgw::BucketLogType::InIndex) { - ldpp_dout(dpp, -1) << "unrecognized log layout type " << log.layout.type << dendl; + source_sync_info.error = fmt::format("unrecognized log layout type {}", to_string(log.layout.type)); return -EINVAL; } // use shard count from our log gen=0 shard_status.resize(rgw::num_shards(log.layout.in_index)); } else { - lderr(driver->ctx()) << "failed to read bucket full sync status: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read bucket full sync status: {}", cpp_strerror(r)); return r; } r = rgw_read_bucket_inc_sync_status(dpp, driver, pipe, gen, &shard_status); if (r < 0) { - lderr(driver->ctx()) << "failed to read bucket incremental sync status: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read bucket incremental sync status: {}", cpp_strerror(r)); return r; } const int total_shards = shard_status.size(); - - out << indented{width} << "incremental sync on " << total_shards << " shards\n"; + source_sync_info.total_shards = total_shards; rgw_bucket_index_marker_info remote_info; BucketIndexShardsManager remote_markers; r = rgw_read_remote_bilog_info(dpp, conn, source_bucket->get_key(), remote_info, remote_markers, null_yield); if (r < 0) { - ldpp_dout(dpp, -1) << "failed to read remote log: " << cpp_strerror(r) << dendl; + source_sync_info.error = fmt::format("failed to read remote log: {}", cpp_strerror(r)); return r; } - std::set<int> shards_behind; + std::map<int, std::string> shards_behind; for (const auto& r : remote_markers.get()) { auto shard_id = r.first; if (r.second.empty()) { @@ -2652,21 +2721,17 @@ static int bucket_source_sync_status(const DoutPrefixProvider *dpp, rgw::sal::Ra } if (shard_id >= total_shards) { // unexpected shard id. we don't have status for it, so we're behind - shards_behind.insert(shard_id); + shards_behind[shard_id] = r.second; continue; } auto& m = shard_status[shard_id]; const auto pos = BucketIndexShardsManager::get_shard_marker(m.inc_marker.position); if (pos < r.second) { - shards_behind.insert(shard_id); + shards_behind[shard_id] = r.second; } } - if (!shards_behind.empty()) { - out << indented{width} << "bucket is behind on " << shards_behind.size() << " shards\n"; - out << indented{width} << "behind shards: [" << shards_behind << "]\n"; - } else { - out << indented{width} << "bucket is caught up with source\n"; - } + + source_sync_info.shards_behind = std::move(shards_behind); return 0; } @@ -2877,25 +2942,82 @@ static int bucket_sync_info(rgw::sal::Driver* driver, const RGWBucketInfo& info, return 0; } +struct bucket_sync_status_info { + std::vector<bucket_source_sync_info> source_status_info; + rgw::sal::Zone* _zone; + const rgw::sal::ZoneGroup* _zonegroup; + const RGWBucketInfo& _bucket_info; + const int width = 15; + std::string error; + + bucket_sync_status_info(const RGWBucketInfo& bucket_info): _bucket_info(bucket_info) {} + + void print(std::ostream& out, bool use_formatter, Formatter* formatter) { + if (use_formatter) { + _print_formatter(out, formatter); + } else { + _print_plaintext(out); + } + } + + void _print_plaintext(std::ostream& out) { + out << indented{width, "realm"} << _zone->get_realm_id() << " (" << _zone->get_realm_name() << ")" << std::endl; + out << indented{width, "zonegroup"} << _zonegroup->get_id() << " (" << _zonegroup->get_name() << ")" << std::endl; + out << indented{width, "zone"} << _zone->get_id() << " (" << _zone->get_name() << ")" << std::endl; + out << indented{width, "bucket"} << _bucket_info.bucket << std::endl; + out << indented{width, "current time"} + << to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms) << "\n\n"; + + if (!error.empty()){ + out << error << std::endl; + } + + for (const auto &info : source_status_info) { + info._print_plaintext(out, width); + } + } + + void _print_formatter(std::ostream& out, Formatter* formatter) { + formatter->open_object_section("test"); + formatter->dump_string("realm", _zone->get_realm_id()); + formatter->dump_string("realm_name", _zone->get_realm_name()); + formatter->dump_string("zonegroup", _zonegroup->get_id()); + formatter->dump_string("zonegroup_name", _zonegroup->get_name()); + formatter->dump_string("zone", _zone->get_id()); + formatter->dump_string("zone_name", _zone->get_name()); + formatter->dump_string("bucket", _bucket_info.bucket.name); + formatter->dump_string("bucket_instance_id", _bucket_info.bucket.bucket_id); + formatter->dump_string("current_time", to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms)); + + if (!error.empty()) { + formatter->dump_string("error", error); + } + + formatter->open_array_section("sources"); + for (const auto &info : source_status_info) { + info._print_formatter(out, formatter); + } + formatter->close_section(); + + formatter->close_section(); + formatter->flush(out); + } + +}; + static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& info, const rgw_zone_id& source_zone_id, std::optional<rgw_bucket>& opt_source_bucket, - std::ostream& out) + bucket_sync_status_info& bucket_sync_info) { const rgw::sal::ZoneGroup& zonegroup = driver->get_zone()->get_zonegroup(); rgw::sal::Zone* zone = driver->get_zone(); - constexpr int width = 15; - - out << indented{width, "realm"} << zone->get_realm_id() << " (" << zone->get_realm_name() << ")\n"; - out << indented{width, "zonegroup"} << zonegroup.get_id() << " (" << zonegroup.get_name() << ")\n"; - out << indented{width, "zone"} << zone->get_id() << " (" << zone->get_name() << ")\n"; - out << indented{width, "bucket"} << info.bucket << "\n"; - out << indented{width, "current time"} - << to_iso_8601(ceph::real_clock::now(), iso_8601_format::YMDhms) << "\n\n"; + bucket_sync_info._zone = zone; + bucket_sync_info._zonegroup = &zonegroup; if (!static_cast<rgw::sal::RadosStore*>(driver)->ctl()->bucket->bucket_imports_data(info.bucket, null_yield, dpp())) { - out << "Sync is disabled for bucket " << info.bucket.name << " or bucket has no sync sources" << std::endl; + bucket_sync_info.error = fmt::format("Sync is disabled for bucket {} or bucket has no sync sources", info.bucket.name); return 0; } @@ -2903,7 +3025,7 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf int r = driver->get_sync_policy_handler(dpp(), std::nullopt, info.bucket, &handler, null_yield); if (r < 0) { - ldpp_dout(dpp(), -1) << "ERROR: failed to get policy handler for bucket (" << info.bucket << "): r=" << r << ": " << cpp_strerror(-r) << dendl; + bucket_sync_info.error = fmt::format("ERROR: failed to get policy handler for bucket ({}): r={}: {}", info.bucket.name, r, cpp_strerror(-r)); return r; } @@ -2916,13 +3038,12 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf std::unique_ptr<rgw::sal::Zone> zone; int ret = driver->get_zone()->get_zonegroup().get_zone_by_id(source_zone_id.id, &zone); if (ret < 0) { - ldpp_dout(dpp(), -1) << "Source zone not found in zonegroup " - << zonegroup.get_name() << dendl; + bucket_sync_info.error = fmt::format("Source zone not found in zonegroup {}", zonegroup.get_name()); return -EINVAL; } auto c = zone_conn_map.find(source_zone_id); if (c == zone_conn_map.end()) { - ldpp_dout(dpp(), -1) << "No connection to zone " << zone->get_name() << dendl; + bucket_sync_info.error = fmt::format("No connection to zone {}", zone->get_name()); return -EINVAL; } zone_ids.insert(source_zone_id); @@ -2953,10 +3074,15 @@ static int bucket_sync_status(rgw::sal::Driver* driver, const RGWBucketInfo& inf continue; } if (pipe.source.zone.value_or(rgw_zone_id()) == z->second.id) { - bucket_source_sync_status(dpp(), static_cast<rgw::sal::RadosStore*>(driver), static_cast<rgw::sal::RadosStore*>(driver)->svc()->zone->get_zone(), z->second, + bucket_source_sync_info source_sync_info(z->second); + auto ret = bucket_source_sync_status(dpp(), static_cast<rgw::sal::RadosStore*>(driver), static_cast<rgw::sal::RadosStore*>(driver)->svc()->zone->get_zone(), z->second, c->second, info, pipe, - width, out); + source_sync_info); + + if (ret == 0) { + bucket_sync_info.source_status_info.emplace_back(std::move(source_sync_info)); + } } } } @@ -3427,6 +3553,13 @@ int main(int argc, const char **argv) OPT opt_cmd = OPT::NO_CMD; int gen_access_key = 0; int gen_secret_key = 0; + enum generate_key_enum { + OPTION_SET_FALSE = 0, + OPTION_SET_TRUE = 1, + OPTION_NOT_SET = 2, + }; + + generate_key_enum generate_key = OPTION_NOT_SET; bool set_perm = false; bool set_temp_url_key = false; map<int, string> temp_url_keys; @@ -3484,6 +3617,7 @@ int main(int argc, const char **argv) list<string> tags_rm; int placement_inline_data = true; bool placement_inline_data_specified = false; + bool format_arg_passed = false; int64_t max_objects = -1; int64_t max_size = -1; @@ -3707,6 +3841,17 @@ int main(int argc, const char **argv) cerr << "bad key type: " << key_type_str << std::endl; exit(1); } + } else if (ceph_argparse_witharg(args, i, &val, "--generate-key", (char*)NULL)) { + key_type_str = val; + if (key_type_str.compare("true") == 0) { + generate_key = OPTION_SET_TRUE; + } else if(key_type_str.compare("false") == 0) { + generate_key = OPTION_SET_FALSE; + } else { + cerr << "wrong value for --generate-key: " << key_type_str << " please specify either true or false" << std::endl; + exit(1); + } + // do nothing } else if (ceph_argparse_binary_flag(args, i, &key_active, NULL, "--key-active", (char*)NULL)) { key_active_specified = true; } else if (ceph_argparse_witharg(args, i, &val, "--job-id", (char*)NULL)) { @@ -3863,6 +4008,7 @@ int main(int argc, const char **argv) new_bucket_name = val; } else if (ceph_argparse_witharg(args, i, &val, "--format", (char*)NULL)) { format = val; + format_arg_passed = true; } else if (ceph_argparse_witharg(args, i, &val, "--categories", (char*)NULL)) { string cat_str = val; list<string> cat_list; @@ -4469,14 +4615,21 @@ int main(int argc, const char **argv) } /* check key parameter conflict */ - if ((!access_key.empty()) && gen_access_key) { - cerr << "ERROR: key parameter conflict, --access-key & --gen-access-key" << std::endl; + if ((!access_key.empty()) && (gen_access_key || generate_key == OPTION_SET_TRUE)) { + cerr << "ERROR: key parameter conflict, --access-key & --gen-access-key/generate-key" << std::endl; + return EINVAL; + } + if ((!secret_key.empty()) && (gen_secret_key || generate_key == OPTION_SET_TRUE)) { + cerr << "ERROR: key parameter conflict, --secret & --gen-secret/generate-key" << std::endl; return EINVAL; } - if ((!secret_key.empty()) && gen_secret_key) { - cerr << "ERROR: key parameter conflict, --secret & --gen-secret" << std::endl; + if (generate_key == OPTION_SET_FALSE) { + if ((!access_key.empty()) || gen_access_key || (!secret_key.empty()) || gen_secret_key) { + cerr << "ERROR: key parameter conflict, if --generate-key is not set so no other key parameters can be set" << std::endl; return EINVAL; + } } + } // default to pretty json @@ -6641,7 +6794,7 @@ int main(int argc, const char **argv) } break; case OPT::USER_CREATE: - if (!user_op.has_existing_user()) { + if (!user_op.has_existing_user() && (generate_key != OPTION_SET_FALSE)) { user_op.set_generate_key(); // generate a new key by default } ret = ruser.add(dpp(), user_op, null_yield, &err_msg); @@ -7552,6 +7705,95 @@ int main(int argc, const char **argv) } } + if (opt_cmd == OPT::BUCKET_LOGGING_FLUSH) { + if (bucket_name.empty()) { + cerr << "ERROR: bucket not specified" << std::endl; + return EINVAL; + } + int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket); + if (ret < 0) { + return -ret; + } + const auto& bucket_attrs = bucket->get_attrs(); + auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (iter == bucket_attrs.end()) { + cerr << "WARNING: no logging configured on bucket" << std::endl; + return 0; + } + rgw::bucketlogging::configuration configuration; + try { + configuration.enabled = true; + decode(configuration, iter->second); + } catch (buffer::error& err) { + cerr << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "'. error: " << err.what() << std::endl; + return EINVAL; + } + std::unique_ptr<rgw::sal::Bucket> target_bucket; + ret = init_bucket(tenant, configuration.target_bucket, "", &target_bucket); + if (ret < 0) { + cerr << "ERROR: failed to get target logging bucket '" << configuration.target_bucket << "'" << std::endl; + return -ret; + } + std::string obj_name; + RGWObjVersionTracker objv_tracker; + ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, dpp(), &objv_tracker); + if (ret < 0) { + cerr << "ERROR: failed to get pending logging object name from target bucket '" << configuration.target_bucket << "'" << std::endl; + return -ret; + } + ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, dpp(), null_yield, true, &objv_tracker); + if (ret < 0) { + cerr << "ERROR: failed to flush pending logging object '" << obj_name + << "' to target bucket '" << configuration.target_bucket << "'" << std::endl; + return -ret; + } + cout << "flushed pending logging object '" << obj_name + << "' to target bucket '" << configuration.target_bucket << "'" << std::endl; + return 0; + } + + if (opt_cmd == OPT::BUCKET_LOGGING_INFO) { + if (bucket_name.empty()) { + cerr << "ERROR: bucket not specified" << std::endl; + return EINVAL; + } + int ret = init_bucket(tenant, bucket_name, bucket_id, &bucket); + if (ret < 0) { + return -ret; + } + const auto& bucket_attrs = bucket->get_attrs(); + auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (iter != bucket_attrs.end()) { + rgw::bucketlogging::configuration configuration; + try { + configuration.enabled = true; + decode(configuration, iter->second); + } catch (buffer::error& err) { + cerr << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "'. error: " << err.what() << std::endl; + return EINVAL; + } + encode_json("logging", configuration, formatter.get()); + formatter->flush(cout); + } + iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING_SOURCES); + if (iter != bucket_attrs.end()) { + rgw::bucketlogging::source_buckets sources; + try { + decode(sources, iter->second); + } catch (buffer::error& err) { + cerr << "ERROR: failed to decode logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES + << "'. error: " << err.what() << std::endl; + return EINVAL; + } + encode_json("logging_sources", sources, formatter.get()); + formatter->flush(cout); + } + + return 0; + } + if (opt_cmd == OPT::LOG_LIST) { // filter by date? if (date.size() && date.size() != 10) { @@ -8623,6 +8865,10 @@ next: handled = decode_dump<uint64_t>("pg_ver", bl, formatter.get()); } else if (iter->first == RGW_ATTR_SOURCE_ZONE) { handled = decode_dump<uint32_t>("source_zone", bl, formatter.get()); + } else if (iter->first == RGW_ATTR_RESTORE_EXPIRY_DATE) { + handled = decode_dump<utime_t>("restore_expiry_date", bl, formatter.get()); + } else if (iter->first == RGW_ATTR_RESTORE_TIME) { + handled = decode_dump<utime_t>("restore_time", bl, formatter.get()); } if (!handled) @@ -9845,7 +10091,18 @@ next: if (ret < 0) { return -ret; } - bucket_sync_status(driver, bucket->get_info(), source_zone, opt_source_bucket, std::cout); + + auto bucket_info = bucket->get_info(); + bucket_sync_status_info bucket_sync_info(bucket_info); + + ret = bucket_sync_status(driver, bucket_info, source_zone, + opt_source_bucket, bucket_sync_info); + + if (ret == 0) { + bucket_sync_info.print(std::cout, format_arg_passed, formatter.get()); + } else { + cerr << "failed to get bucket sync status. see logs for more info" << std::endl; + } } if (opt_cmd == OPT::BUCKET_SYNC_MARKERS) { @@ -10335,7 +10592,8 @@ next: if (!rgw::sal::User::empty(user)) { pipe->params.user = user->get_id(); - } else if (pipe->params.mode == rgw_sync_pipe_params::MODE_USER) { + } else if (pipe->params.mode == rgw_sync_pipe_params::MODE_USER && + pipe->params.user.empty()) { cerr << "ERROR: missing --uid for --mode=user" << std::endl; return EINVAL; } diff --git a/src/rgw/rgw_sync_checkpoint.cc b/src/rgw/radosgw-admin/sync_checkpoint.cc index 1172e79a48f..0303ed6c747 100644 --- a/src/rgw/rgw_sync_checkpoint.cc +++ b/src/rgw/radosgw-admin/sync_checkpoint.cc @@ -5,6 +5,7 @@ * Ceph - scalable distributed file system * * Copyright (C) 2020 Red Hat, Inc. + * Copyright (C) 2024 IBM * * This is free software; you can redistribute it and/or * modify it under the terms of the GNU Lesser General Public @@ -13,9 +14,12 @@ * */ +#include "radosgw-admin/sync_checkpoint.h" + #include <fmt/format.h> + #include "common/errno.h" -#include "rgw_sync_checkpoint.h" + #include "rgw_sal_rados.h" #include "rgw_bucket_sync.h" #include "rgw_data_sync.h" diff --git a/src/rgw/rgw_sync_checkpoint.h b/src/rgw/radosgw-admin/sync_checkpoint.h index 28df68d8860..28df68d8860 100644 --- a/src/rgw/rgw_sync_checkpoint.h +++ b/src/rgw/radosgw-admin/sync_checkpoint.h diff --git a/src/rgw/rgw_asio_frontend.cc b/src/rgw/rgw_asio_frontend.cc index 30e1e77fd15..ebe42d96de9 100644 --- a/src/rgw/rgw_asio_frontend.cc +++ b/src/rgw/rgw_asio_frontend.cc @@ -1194,8 +1194,11 @@ void AsioFrontend::pause() l.signal.emit(boost::asio::cancellation_type::terminal); } - // close all connections so outstanding requests fail quickly - connections.close(ec); + const bool graceful_stop{ g_ceph_context->_conf->rgw_graceful_stop }; + if (!graceful_stop) { + // close all connections so outstanding requests fail quickly + connections.close(ec); + } // pause and wait until outstanding requests complete pause_mutex.lock(ec); diff --git a/src/rgw/rgw_auth.cc b/src/rgw/rgw_auth.cc index ac1ed8b75d6..a0b494eb9c5 100644 --- a/src/rgw/rgw_auth.cc +++ b/src/rgw/rgw_auth.cc @@ -188,7 +188,8 @@ int load_account_and_policies(const DoutPrefixProvider* dpp, static auto transform_old_authinfo(const RGWUserInfo& user, std::optional<RGWAccountInfo> account, - std::vector<IAM::Policy> policies) + std::vector<IAM::Policy> policies, + sal::Driver* driver) -> std::unique_ptr<rgw::auth::Identity> { /* This class is not intended for public use. Should be removed altogether @@ -198,6 +199,7 @@ static auto transform_old_authinfo(const RGWUserInfo& user, /* For this particular case it's OK to use rgw_user structure to convey * the identity info as this was the policy for doing that before the * new auth. */ + sal::Driver* driver; const rgw_user id; const std::string display_name; const std::string path; @@ -208,8 +210,10 @@ static auto transform_old_authinfo(const RGWUserInfo& user, public: DummyIdentityApplier(const RGWUserInfo& user, std::optional<RGWAccountInfo> account, - std::vector<IAM::Policy> policies) - : id(user.user_id), + std::vector<IAM::Policy> policies, + sal::Driver* driver) + : driver(driver), + id(user.user_id), display_name(user.display_name), path(user.path), is_admin(user.admin), @@ -294,9 +298,9 @@ static auto transform_old_authinfo(const RGWUserInfo& user, << ", is_admin=" << is_admin << ")"; } - void load_acct_info(const DoutPrefixProvider* dpp, - RGWUserInfo& user_info) const override { + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override { // noop, this user info was passed in on construction + return driver->get_user(id); } void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const { @@ -307,13 +311,14 @@ static auto transform_old_authinfo(const RGWUserInfo& user, }; return std::make_unique<DummyIdentityApplier>( - user, std::move(account), std::move(policies)); + user, std::move(account), std::move(policies), driver); } auto transform_old_authinfo(const DoutPrefixProvider* dpp, optional_yield y, sal::Driver* driver, - sal::User* user) + sal::User* user, + std::vector<IAM::Policy>* policies_) -> tl::expected<std::unique_ptr<Identity>, int> { const RGWUserInfo& info = user->get_info(); @@ -328,7 +333,10 @@ auto transform_old_authinfo(const DoutPrefixProvider* dpp, return tl::unexpected(r); } - return transform_old_authinfo(info, std::move(account), std::move(policies)); + if (policies_) { // return policies to caller if requested + *policies_ = policies; + } + return transform_old_authinfo(info, std::move(account), std::move(policies), driver); } } /* namespace auth */ @@ -377,7 +385,7 @@ strategy_handle_rejected(rgw::auth::Engine::result_t&& engine_result, case Control::FALLBACK: /* Don't try next. */ - return std::make_pair(false, std::move(engine_result)); + return std::make_pair(false, std::move(strategy_result)); default: /* Huh, memory corruption? */ @@ -523,7 +531,7 @@ rgw::auth::Strategy::apply(const DoutPrefixProvider *dpp, const rgw::auth::Strat /* Account used by a given RGWOp is decoupled from identity employed * in the authorization phase (RGWOp::verify_permissions). */ - applier->load_acct_info(dpp, s->user->get_info()); + s->user = applier->load_acct_info(dpp); s->perm_mask = applier->get_perm_mask(); /* This is the single place where we pass req_state as a pointer @@ -631,36 +639,36 @@ void rgw::auth::WebIdentityApplier::create_account(const DoutPrefixProvider* dpp user_info = user->get_info(); } -void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const { +auto rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { rgw_user federated_user; federated_user.id = this->sub; federated_user.tenant = role_tenant; federated_user.ns = "oidc"; + std::unique_ptr<rgw::sal::User> user = driver->get_user(federated_user); if (account) { // we don't need shadow users for account roles because bucket ownership, // quota, and stats are tracked by the account instead of the user - user_info.user_id = std::move(federated_user); + RGWUserInfo& user_info = user->get_info(); user_info.display_name = user_name; user_info.type = TYPE_WEB; - return; + // the user_info.user_id is initialized by driver->get_user(...) + return user; } - std::unique_ptr<rgw::sal::User> user = driver->get_user(federated_user); - //Check in oidc namespace if (user->load_user(dpp, null_yield) >= 0) { /* Succeeded. */ - user_info = user->get_info(); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } user->clear_ns(); //Check for old users which wouldn't have been created in oidc namespace if (user->load_user(dpp, null_yield) >= 0) { /* Succeeded. */ - user_info = user->get_info(); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } //Check if user_id.buckets already exists, may have been from the time, when shadow users didnt exist @@ -671,7 +679,7 @@ void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp last_synced, last_updated); if (ret < 0 && ret != -ENOENT) { ldpp_dout(dpp, 0) << "ERROR: reading stats for the user returned error " << ret << dendl; - return; + return user; } if (ret == -ENOENT) { /* in case of ENOENT, which means user doesnt have buckets */ //In this case user will be created in oidc namespace @@ -684,7 +692,8 @@ void rgw::auth::WebIdentityApplier::load_acct_info(const DoutPrefixProvider* dpp } ldpp_dout(dpp, 0) << "NOTICE: couldn't map oidc federated user " << federated_user << dendl; - create_account(dpp, federated_user, this->user_name, user_info); + create_account(dpp, federated_user, this->user_name, user->get_info()); + return user; } void rgw::auth::WebIdentityApplier::modify_request_state(const DoutPrefixProvider *dpp, req_state* s) const @@ -936,7 +945,7 @@ void rgw::auth::RemoteApplier::write_ops_log_entry(rgw_log_entry& entry) const } /* TODO(rzarzynski): we need to handle display_name changes. */ -void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* It's supposed that RGWRemoteAuthApplier tries to load account info * that belongs to the authenticated identity. Another policy may be @@ -975,9 +984,9 @@ void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGW (void) load_account_and_policies(dpp, null_yield, driver, user->get_info(), user->get_attrs(), account, policies); - user_info = std::move(user->get_info()); owner_acct_user = std::move(tenanted_uid); - return; + // the user_info in user is initialized by user->load_user(...) + return user; } } @@ -990,15 +999,16 @@ void rgw::auth::RemoteApplier::load_acct_info(const DoutPrefixProvider* dpp, RGW (void) load_account_and_policies(dpp, null_yield, driver, user->get_info(), user->get_attrs(), account, policies); - user_info = std::move(user->get_info()); owner_acct_user = acct_user; - return; + // the user_info in user is initialized by user->load_user(...) + return user; } ldpp_dout(dpp, 0) << "NOTICE: couldn't map swift user " << acct_user << dendl; - create_account(dpp, acct_user, implicit_tenant, user_info); + create_account(dpp, acct_user, implicit_tenant, user->get_info()); /* Succeeded if we are here (create_account() hasn't throwed). */ + return user; } void rgw::auth::RemoteApplier::modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const @@ -1098,11 +1108,11 @@ uint32_t rgw::auth::LocalApplier::get_perm_mask(const std::string& subuser_name, } } -void rgw::auth::LocalApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::LocalApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* Load the account that belongs to the authenticated identity. An extra call * to RADOS may be safely skipped in this case. */ - user_info = this->user_info; + return std::unique_ptr<rgw::sal::User>(user.release()); } void rgw::auth::LocalApplier::modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const @@ -1121,6 +1131,22 @@ void rgw::auth::LocalApplier::write_ops_log_entry(rgw_log_entry& entry) const } } +rgw::auth::LocalApplier::LocalApplier(CephContext* const cct, + std::unique_ptr<rgw::sal::User> user, + std::optional<RGWAccountInfo> account, + std::vector<IAM::Policy> policies, + std::string subuser, + const std::optional<uint32_t>& perm_mask, + const std::string access_key_id) + : user_info(user->get_info()), + user(std::move(user)), + account(std::move(account)), + policies(std::move(policies)), + subuser(std::move(subuser)), + perm_mask(perm_mask.value_or(RGW_PERM_INVALID)), + access_key_id(access_key_id) { +} + ACLOwner rgw::auth::RoleApplier::get_aclowner() const { ACLOwner owner; @@ -1183,10 +1209,11 @@ bool rgw::auth::RoleApplier::is_identity(const Principal& p) const { return false; } -void rgw::auth::RoleApplier::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const /* out */ +auto rgw::auth::RoleApplier::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> /* out */ { /* Load the user id */ - user_info.user_id = this->token_attrs.user_id; + std::unique_ptr<rgw::sal::User> user = driver->get_user(this->token_attrs.user_id); + return user; } void rgw::auth::RoleApplier::write_ops_log_entry(rgw_log_entry& entry) const @@ -1267,9 +1294,10 @@ rgw::auth::AnonymousEngine::authenticate(const DoutPrefixProvider* dpp, const re } else { RGWUserInfo user_info; rgw_get_anon_user(user_info); - + std::unique_ptr<rgw::sal::User> user = s->user->clone(); + user->get_info() = user_info; auto apl = \ - apl_factory->create_apl_local(cct, s, user_info, std::nullopt, {}, + apl_factory->create_apl_local(cct, s, std::move(user), std::nullopt, {}, rgw::auth::LocalApplier::NO_SUBUSER, std::nullopt, rgw::auth::LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); diff --git a/src/rgw/rgw_auth.h b/src/rgw/rgw_auth.h index f3edbbab845..22b0816bac9 100644 --- a/src/rgw/rgw_auth.h +++ b/src/rgw/rgw_auth.h @@ -105,7 +105,8 @@ inline std::ostream& operator<<(std::ostream& out, auto transform_old_authinfo(const DoutPrefixProvider* dpp, optional_yield y, sal::Driver* driver, - sal::User* user) + sal::User* user, + std::vector<IAM::Policy>* policies_ = nullptr) -> tl::expected<std::unique_ptr<Identity>, int>; // Load the user account and all user/group policies. May throw @@ -139,7 +140,7 @@ public: * * XXX: be aware that the "account" term refers to rgw_user. The naming * is legacy. */ - virtual void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const = 0; /* out */ + virtual auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> = 0; /* out */ /* Apply any changes to request state. This method will be most useful for * TempURL of Swift API. */ @@ -484,7 +485,7 @@ public: bool is_identity(const Principal& p) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; uint32_t get_identity_type() const override { return TYPE_WEB; @@ -656,7 +657,7 @@ public: uint32_t get_perm_mask() const override { return info.perm_mask; } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; void write_ops_log_entry(rgw_log_entry& entry) const override; uint32_t get_identity_type() const override { return info.acct_type; } @@ -683,7 +684,7 @@ public: /* rgw::auth::LocalApplier targets those auth engines that base on the data - * enclosed in the RGWUserInfo control structure. As a side effect of doing + * enclosed in the rgw::sal::User->RGWUserInfo control structure. As a side effect of doing * the authentication process, they must have it loaded. Leveraging this is * a way to avoid unnecessary calls to underlying RADOS store. */ class LocalApplier : public IdentityApplier { @@ -691,6 +692,7 @@ class LocalApplier : public IdentityApplier { protected: const RGWUserInfo user_info; + mutable std::unique_ptr<rgw::sal::User> user; const std::optional<RGWAccountInfo> account; const std::vector<IAM::Policy> policies; const std::string subuser; @@ -705,19 +707,12 @@ public: static const std::string NO_ACCESS_KEY; LocalApplier(CephContext* const cct, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, std::string subuser, const std::optional<uint32_t>& perm_mask, - const std::string access_key_id) - : user_info(user_info), - account(std::move(account)), - policies(std::move(policies)), - subuser(std::move(subuser)), - perm_mask(perm_mask.value_or(RGW_PERM_INVALID)), - access_key_id(access_key_id) { - } + const std::string access_key_id); ACLOwner get_aclowner() const override; uint32_t get_perms_from_aclspec(const DoutPrefixProvider* dpp, const aclspec_t& aclspec) const override; @@ -732,7 +727,7 @@ public: } } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; uint32_t get_identity_type() const override { return user_info.type; } std::string get_acct_name() const override { return {}; } @@ -750,7 +745,7 @@ public: virtual ~Factory() {} virtual aplptr_t create_apl_local(CephContext* cct, const req_state* s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, @@ -779,15 +774,20 @@ public: std::vector<std::pair<std::string, std::string>> principal_tags; }; protected: + CephContext* const cct; + rgw::sal::Driver* driver; Role role; TokenAttrs token_attrs; public: RoleApplier(CephContext* const cct, + rgw::sal::Driver* driver, const Role& role, const TokenAttrs& token_attrs) - : role(role), + : cct(cct), + driver(driver), + role(role), token_attrs(token_attrs) {} ACLOwner get_aclowner() const override; @@ -803,7 +803,7 @@ public: return RGW_PERM_NONE; } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ uint32_t get_identity_type() const override { return TYPE_ROLE; } std::string get_acct_name() const override { return {}; } std::string get_subuser() const override { return {}; } diff --git a/src/rgw/rgw_auth_filters.h b/src/rgw/rgw_auth_filters.h index a93641e8b8e..7d264197c52 100644 --- a/src/rgw/rgw_auth_filters.h +++ b/src/rgw/rgw_auth_filters.h @@ -117,8 +117,8 @@ public: return get_decoratee().get_account(); } - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override { /* out */ - return get_decoratee().load_acct_info(dpp, user_info); + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override { /* out */ + return get_decoratee().load_acct_info(dpp); } void modify_request_state(const DoutPrefixProvider* dpp, req_state * s) const override { /* in/out */ @@ -152,7 +152,7 @@ public: } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ }; /* static declaration: UNKNOWN_ACCT will be an empty rgw_user that is a result @@ -169,23 +169,25 @@ void ThirdPartyAccountApplier<T>::to_str(std::ostream& out) const } template <typename T> -void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const +auto ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { + std::unique_ptr<rgw::sal::User> luser; if (UNKNOWN_ACCT == acct_user_override) { /* There is no override specified by the upper layer. This means that we'll * load the account owned by the authenticated identity (aka auth_user). */ - DecoratedApplier<T>::load_acct_info(dpp, user_info); + luser = DecoratedApplier<T>::load_acct_info(dpp); } else if (DecoratedApplier<T>::is_owner_of(acct_user_override)) { /* The override has been specified but the account belongs to the authenticated * identity. We may safely forward the call to a next stage. */ - DecoratedApplier<T>::load_acct_info(dpp, user_info); + luser = DecoratedApplier<T>::load_acct_info(dpp); } else if (this->is_anonymous()) { /* If the user was authed by the anonymous engine then scope the ANON user * to the correct tenant */ + luser = driver->get_user(rgw_user(RGW_USER_ANON_ID)); if (acct_user_override.tenant.empty()) - user_info.user_id = rgw_user(acct_user_override.id, RGW_USER_ANON_ID); + luser->get_info().user_id = rgw_user(acct_user_override.id, RGW_USER_ANON_ID); else - user_info.user_id = rgw_user(acct_user_override.tenant, RGW_USER_ANON_ID); + luser->get_info().user_id = rgw_user(acct_user_override.tenant, RGW_USER_ANON_ID); } else { /* Compatibility mechanism for multi-tenancy. For more details refer to * load_acct_info method of rgw::auth::RemoteApplier. */ @@ -196,9 +198,10 @@ void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, user = driver->get_user(tenanted_uid); if (user->load_user(dpp, null_yield) >= 0) { - user_info = user->get_info(); + // the user_info in luser is initialized by user->load_user(...) + luser = user->clone(); /* Succeeded. */ - return; + return luser; } } @@ -213,8 +216,10 @@ void ThirdPartyAccountApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, throw ret; } } - user_info = user->get_info(); + // the user_info in luser is initialized by user->load_user(...) + luser = user->clone(); } + return luser; } template <typename T> static inline @@ -248,7 +253,7 @@ public: } void to_str(std::ostream& out) const override; - void load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const override; /* out */ + auto load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> override; /* out */ void modify_request_state(const DoutPrefixProvider* dpp, req_state* s) const override; /* in/out */ ACLOwner get_aclowner() const override { @@ -271,10 +276,10 @@ void SysReqApplier<T>::to_str(std::ostream& out) const } template <typename T> -void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo& user_info) const +auto SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp) const -> std::unique_ptr<rgw::sal::User> { - DecoratedApplier<T>::load_acct_info(dpp, user_info); - is_system = user_info.system; + std::unique_ptr<rgw::sal::User> user = DecoratedApplier<T>::load_acct_info(dpp); + is_system = user->get_info().system; if (is_system) { //ldpp_dout(dpp, 20) << "system request" << dendl; @@ -285,7 +290,7 @@ void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo effective_owner->id = parse_owner(str); if (const auto* uid = std::get_if<rgw_user>(&effective_owner->id); uid) { - std::unique_ptr<rgw::sal::User> user = driver->get_user(*uid); + user = driver->get_user(*uid); if (user->load_user(dpp, null_yield) < 0) { //ldpp_dout(dpp, 0) << "User lookup failed!" << dendl; throw -EACCES; @@ -294,14 +299,14 @@ void SysReqApplier<T>::load_acct_info(const DoutPrefixProvider* dpp, RGWUserInfo } } } + return user; } template <typename T> void SysReqApplier<T>::modify_request_state(const DoutPrefixProvider* dpp, req_state* const s) const { if (boost::logic::indeterminate(is_system)) { - RGWUserInfo unused_info; - load_acct_info(dpp, unused_info); + std::unique_ptr<rgw::sal::User> unused_user{ load_acct_info(dpp) }; } if (is_system) { diff --git a/src/rgw/rgw_auth_s3.h b/src/rgw/rgw_auth_s3.h index 2f7fd2d7598..5815a520e02 100644 --- a/src/rgw/rgw_auth_s3.h +++ b/src/rgw/rgw_auth_s3.h @@ -55,14 +55,14 @@ class STSAuthStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, const std::optional<uint32_t>& perm_mask, const std::string& access_key_id) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id)); return aplptr_t(new decltype(apl)(std::move(apl))); } @@ -72,7 +72,7 @@ class STSAuthStrategy : public rgw::auth::Strategy, RoleApplier::Role role, RoleApplier::TokenAttrs token_attrs) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - rgw::auth::RoleApplier(cct, std::move(role), std::move(token_attrs))); + rgw::auth::RoleApplier(cct, driver, std::move(role), std::move(token_attrs))); return aplptr_t(new decltype(apl)(std::move(apl))); } @@ -176,14 +176,14 @@ class AWSAuthStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, const std::optional<uint32_t>& perm_mask, const std::string& access_key_id) const override { auto apl = rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id)); /* TODO(rzarzynski): replace with static_ptr. */ return aplptr_t(new decltype(apl)(std::move(apl))); diff --git a/src/rgw/rgw_bucket_layout.cc b/src/rgw/rgw_bucket_layout.cc index f8c485d89c3..1f8db396a0d 100644 --- a/src/rgw/rgw_bucket_layout.cc +++ b/src/rgw/rgw_bucket_layout.cc @@ -376,9 +376,9 @@ void encode_json_impl(const char *name, const BucketLayout& l, ceph::Formatter * for (const auto& log : l.logs) { encode_json("log", log, f); } + f->close_section(); // logs[] utime_t jt(l.judge_reshard_lock_time); encode_json("judge_reshard_lock_time", jt, f); - f->close_section(); // logs[] f->close_section(); } void decode_json_obj(BucketLayout& l, JSONObj *obj) diff --git a/src/rgw/rgw_bucket_logging.cc b/src/rgw/rgw_bucket_logging.cc new file mode 100644 index 00000000000..dd407f26e8c --- /dev/null +++ b/src/rgw/rgw_bucket_logging.cc @@ -0,0 +1,799 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#include <time.h> +#include <random> +#include "common/ceph_time.h" +#include "rgw_bucket_logging.h" +#include "rgw_xml.h" +#include "rgw_sal.h" +#include "rgw_op.h" + +#define dout_subsys ceph_subsys_rgw + +namespace rgw::bucketlogging { + +bool configuration::decode_xml(XMLObj* obj) { + const auto throw_if_missing = true; + enabled = false; + XMLObjIter iter = obj->find("LoggingEnabled"); + XMLObj* const o = iter.get_next(); + if (o) { + enabled = true; + RGWXMLDecoder::decode_xml("TargetBucket", target_bucket, o, throw_if_missing); + RGWXMLDecoder::decode_xml("TargetPrefix", target_prefix, o); + // TODO: decode grant + RGWXMLDecoder::decode_xml("ObjectRollTime", obj_roll_time, default_obj_roll_time, o); + std::string default_type{"Standard"}; + std::string type; + RGWXMLDecoder::decode_xml("LoggingType", type, default_type, o); + if (type == "Standard") { + logging_type = LoggingType::Standard; + } else if (type == "Journal") { + logging_type = LoggingType::Journal; + if (iter = o->find("Filter"); XMLObj* const filter_o = iter.get_next()) { + RGWXMLDecoder::decode_xml("S3Key", key_filter, filter_o); + } + } else { + // we don't allow for type "Any" in the configuration + throw RGWXMLDecoder::err("invalid bucket logging record type: '" + type + "'"); + } + RGWXMLDecoder::decode_xml("RecordsBatchSize", records_batch_size, o); + if (iter = o->find("TargetObjectKeyFormat"); XMLObj* const oo = iter.get_next()) { + if (iter = oo->find("PartitionedPrefix"); XMLObj* const ooo = iter.get_next()) { + obj_key_format = KeyFormat::Partitioned; + default_type = "DeliveryTime"; + RGWXMLDecoder::decode_xml("PartitionDateSource", type, default_type, ooo); + if (type == "DeliveryTime") { + date_source = PartitionDateSource::DeliveryTime; + } else if (type == "EventTime") { + date_source = PartitionDateSource::EventTime; + } else { + throw RGWXMLDecoder::err("invalid bucket logging partition date source: '" + type + "'"); + } + } else if (iter = oo->find("SimplePrefix"); iter.get_next()) { + obj_key_format = KeyFormat::Simple; + } else { + throw RGWXMLDecoder::err("TargetObjectKeyFormat must contain a format tag"); + } + } + } + + return true; +} + +void configuration::dump_xml(Formatter *f) const { + if (!enabled) { + return; + } + f->open_object_section("LoggingEnabled"); + ::encode_xml("TargetBucket", target_bucket, f); + ::encode_xml("TargetPrefix", target_prefix, f); + ::encode_xml("ObjectRollTime", obj_roll_time, f); + switch (logging_type) { + case LoggingType::Standard: + ::encode_xml("LoggingType", "Standard", f); + break; + case LoggingType::Journal: + ::encode_xml("LoggingType", "Journal", f); + if (key_filter.has_content()) { + f->open_object_section("Filter"); + ::encode_xml("S3Key", key_filter, f); + f->close_section(); // Filter + } + break; + case LoggingType::Any: + ::encode_xml("LoggingType", "", f); + break; + } + ::encode_xml("RecordsBatchSize", records_batch_size, f); + f->open_object_section("TargetObjectKeyFormat"); + switch (obj_key_format) { + case KeyFormat::Partitioned: + f->open_object_section("PartitionedPrefix"); + switch (date_source) { + case PartitionDateSource::DeliveryTime: + ::encode_xml("PartitionDateSource", "DeliveryTime", f); + break; + case PartitionDateSource::EventTime: + ::encode_xml("PartitionDateSource", "EventTime", f); + break; + } + f->close_section(); // PartitionedPrefix + break; + case KeyFormat::Simple: + f->open_object_section("SimplePrefix"); // empty section + f->close_section(); + break; + } + f->close_section(); // TargetObjectKeyFormat + f->close_section(); // LoggingEnabled +} + +void configuration::dump(Formatter *f) const { + Formatter::ObjectSection s(*f, "bucketLoggingStatus"); + if (!enabled) { + return; + } + { + Formatter::ObjectSection s(*f, "loggingEnabled"); + encode_json("targetBucket", target_bucket, f); + encode_json("targetPrefix", target_prefix, f); + encode_json("objectRollTime", obj_roll_time, f); + switch (logging_type) { + case LoggingType::Standard: + encode_json("loggingType", "Standard", f); + break; + case LoggingType::Journal: + encode_json("loggingType", "Journal", f); + if (key_filter.has_content()) { + Formatter::ObjectSection s(*f, "Filter"); + encode_json("S3Key", key_filter, f); + } + break; + case LoggingType::Any: + encode_json("loggingType", "", f); + break; + } + encode_json("recordsBatchSize", records_batch_size, f); + { + Formatter::ObjectSection s(*f, "targetObjectKeyFormat"); + switch (obj_key_format) { + case KeyFormat::Partitioned: + { + Formatter::ObjectSection s(*f, "partitionedPrefix"); + switch (date_source) { + case PartitionDateSource::DeliveryTime: + encode_json("partitionDateSource", "DeliveryTime", f); + break; + case PartitionDateSource::EventTime: + encode_json("partitionDateSource", "EventTime", f); + break; + } + } + break; + case KeyFormat::Simple: + { + Formatter::ObjectSection s(*f, "simplePrefix"); + } + break; + } + } + } +} + +std::string configuration::to_json_str() const { + JSONFormatter f; + dump(&f); + std::stringstream ss; + f.flush(ss); + return ss.str(); +} + +template<size_t N> +std::string unique_string() { + static const std::string possible_characters{"0123456789ABCDEFGHIJKLMNOPQRSTUVWXY"}; + static const auto max_possible_value = possible_characters.length() - 1; + std::random_device rd; + std::mt19937 engine(rd()); + std::uniform_int_distribution<> dist(0, max_possible_value); + std::string str(N, '\0'); + std::generate_n(str.begin(), N, [&](){return possible_characters[dist(engine)];}); + return str; +} + +constexpr size_t UniqueStringLength = 16; + +ceph::coarse_real_time time_from_name(const std::string& obj_name, const DoutPrefixProvider *dpp) { + static const auto time_format_length = std::string{"YYYY-MM-DD-hh-mm-ss"}.length(); + const auto obj_name_length = obj_name.length(); + ceph::coarse_real_time extracted_time; + if (obj_name_length < time_format_length + UniqueStringLength + 1) { + ldpp_dout(dpp, 1) << "ERROR: logging object name too short: " << obj_name << dendl; + return extracted_time; + } + const auto time_start_pos = obj_name_length - (time_format_length + UniqueStringLength + 1); + // note: +1 is for the dash between the timestamp and the unique string + std::string time_str = obj_name.substr(time_start_pos, time_format_length); + + std::tm t = {}; + if (const auto ret = strptime(time_str.c_str(), "%Y-%m-%d-%H-%M-%S", &t); ret == nullptr || *ret != '\0') { + ldpp_dout(dpp, 1) << "ERROR: invalid time format: '" << time_str << "' in logging object name: " << obj_name << dendl; + return extracted_time; + } + extracted_time = ceph::coarse_real_time::clock::from_time_t(mktime(&t)); + ldpp_dout(dpp, 20) << "INFO: time '" << extracted_time << "' extracted from logging object name: " << obj_name << dendl; + return extracted_time; +} + +std::string full_bucket_name(const std::unique_ptr<rgw::sal::Bucket>& bucket) { + if (bucket->get_tenant().empty()) { + return bucket->get_name(); + } + return fmt::format("{}:{}", bucket->get_tenant(), bucket->get_name()); +} + +int new_logging_object(const configuration& conf, + const std::unique_ptr<rgw::sal::Bucket>& bucket, + std::string& obj_name, + const DoutPrefixProvider *dpp, + optional_yield y, + bool init_obj, + RGWObjVersionTracker* objv_tracker) { + const auto tt = ceph::coarse_real_time::clock::to_time_t(ceph::coarse_real_time::clock::now()); + std::tm t{}; + localtime_r(&tt, &t); + + const auto unique = unique_string<UniqueStringLength>(); + const auto old_name = obj_name; + + switch (conf.obj_key_format) { + case KeyFormat::Simple: + obj_name = fmt::format("{}{:%Y-%m-%d-%H-%M-%S}-{}", + conf.target_prefix, + t, + unique); + break; + case KeyFormat::Partitioned: + { + // TODO: use date_source + const auto source_region = ""; // TODO + obj_name = fmt::format("{}{}/{}/{}/{:%Y/%m/%d}/{:%Y-%m-%d-%H-%M-%S}-{}", + conf.target_prefix, + to_string(bucket->get_owner()), + source_region, + full_bucket_name(bucket), + t, + t, + unique); + } + break; + } + int ret = bucket->set_logging_object_name(obj_name, conf.target_prefix, y, dpp, init_obj, objv_tracker); + if (ret == -EEXIST || ret == -ECANCELED) { + if (ret = bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, nullptr); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" << + conf.target_bucket << "' and prefix '" << conf.target_prefix << "', ret = " << ret << dendl; + return ret; + } + ldpp_dout(dpp, 20) << "INFO: name already set. got name of logging object '" << obj_name << "' of bucket '" << + conf.target_bucket << "' and prefix '" << conf.target_prefix << "'" << dendl; + return -ECANCELED; + } else if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to write name of logging object '" << obj_name << "' of bucket '" << + conf.target_bucket << "'. ret = " << ret << dendl; + return ret; + } + ldpp_dout(dpp, 20) << "INFO: wrote name of logging object '" << obj_name << "' of bucket '" << + conf.target_bucket << "'" << dendl; + return 0; +} + +int commit_logging_object(const configuration& conf, + const DoutPrefixProvider *dpp, + rgw::sal::Driver* driver, + const std::string& tenant_name, + optional_yield y) { + std::string target_bucket_name; + std::string target_tenant_name; + auto ret = rgw_parse_url_bucket(conf.target_bucket, tenant_name, target_tenant_name, target_bucket_name); + if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to parse target bucket '" << conf.target_bucket << "' when commiting logging object, ret = " + << ret << dendl; + return ret; + } + const rgw_bucket target_bucket_id(target_tenant_name, target_bucket_name); + std::unique_ptr<rgw::sal::Bucket> target_bucket; + ret = driver->load_bucket(dpp, target_bucket_id, + &target_bucket, y); + if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get target logging bucket '" << target_bucket_id << "' when commiting logging object, ret = " + << ret << dendl; + return ret; + } + return commit_logging_object(conf, target_bucket, dpp, y); +} + +int commit_logging_object(const configuration& conf, + const std::unique_ptr<rgw::sal::Bucket>& target_bucket, + const DoutPrefixProvider *dpp, + optional_yield y) { + std::string obj_name; + if (const auto ret = target_bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, nullptr); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" << + target_bucket->get_info().bucket << "'. ret = " << ret << dendl; + return ret; + } + return target_bucket->commit_logging_object(obj_name, y, dpp); +} + +int rollover_logging_object(const configuration& conf, + const std::unique_ptr<rgw::sal::Bucket>& bucket, + std::string& obj_name, + const DoutPrefixProvider *dpp, + optional_yield y, + bool must_commit, + RGWObjVersionTracker* objv_tracker) { + std::string target_bucket_name; + std::string target_tenant_name; + std::ignore = rgw_parse_url_bucket(conf.target_bucket, bucket->get_tenant(), target_tenant_name, target_bucket_name); + if (target_bucket_name != bucket->get_name() || target_tenant_name != bucket->get_tenant()) { + ldpp_dout(dpp, 1) << "ERROR: bucket name mismatch. conf= '" << conf.target_bucket << + "', bucket= '" << bucket->get_info().bucket << "'" << dendl; + return -EINVAL; + } + const auto old_obj = obj_name; + const auto ret = new_logging_object(conf, bucket, obj_name, dpp, y, false, objv_tracker); + if (ret == -ECANCELED) { + ldpp_dout(dpp, 20) << "INFO: rollover already performed for '" << old_obj << "' to bucket '" << + conf.target_bucket << "'. ret = " << ret << dendl; + return 0; + } else if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to rollover logging object '" << old_obj << "' to bucket '" << + conf.target_bucket << "'. ret = " << ret << dendl; + return ret; + } + if (const auto ret = bucket->commit_logging_object(old_obj, y, dpp); ret < 0) { + if (must_commit) { + return ret; + } + ldpp_dout(dpp, 5) << "WARNING: failed to commit logging object '" << old_obj << "' to bucket '" << + conf.target_bucket << "'. ret = " << ret << dendl; + // we still want to write the new records to the new object even if commit failed + // will try to commit again next time + } + return 0; +} + +#define dash_if_empty(S) (S).empty() ? "-" : S +#define dash_if_empty_or_null(P, S) (((P) == nullptr) || (S).empty()) ? "-" : S +#define dash_if_zero(I) (I) == 0 ? "-" : std::to_string(I) +#define dash_if_zero_or_null(P, I) (((P) == nullptr) || ((I) == 0)) ? "-" : std::to_string(I) + +/* S3 bucket standard log record + * based on: https://docs.aws.amazon.com/AmazonS3/latest/userguide/LogFormat.html + - bucket owner + - bucket name + - The time at which the request was received at UTC time. The format, as follows: [%d/%b/%Y:%H:%M:%S %z] + - The apparent IP address of the requester + - The canonical user ID of the requester, or a - for unauthenticated requests + - Request ID + - REST.HTTP_method.resource_type or S3.action.resource_type for Lifecycle and logging + - The key (object name) part of the request (source key in case of copy) + - The Request-URI part of the HTTP request message + - The numeric HTTP status code of the response + - The S3 Error code, or - if no error occurred + - The number of response bytes sent, excluding HTTP protocol overhead, or - if zero + - Object Size + - Total time: milliseconds including network transmission time. from first byte received to last byte transmitted + - turn around time: milliseconds exluding networks transmission time. from last byte received to first byte transmitted + - The value of the HTTP Referer header, if present, or - if not + - User Agent + - The version ID in the request, or - if the operation doesn't take a versionId parameter + - Host ID: x-amz-id-2 + - SigV2 or SigV4, that was used to authenticate the request or a - for unauthenticated requests + - SSL cipher that was negotiated for an HTTPS request or a - for HTTP + - The type of request authentication used: AuthHeader, QueryString or a - for unauthenticated requests + - Host Header: The RGW endpoint fqdn + - TLS version negotiated by the client: TLSv1.1, TLSv1.2, TLSv1.3, or - if TLS wasn't used + - ARN of the access point of the request. If the access point ARN is malformed or not used, the field will contain a - + - A string that indicates whether the request required an (ACL) for authorization. If ACL is required, the string is Yes. If no ACLs were required, the string is - + +S3 bucket short (ceph) log record + - bucket owner + - bucket name + - The time at which the request was received at UTC time. The format, as follows: [%d/%b/%Y:%H:%M:%S %z] + - REST.HTTP_method.resource_type or S3.action.resource_type for Lifecycle and logging + - The key (object name) part of the request (source key in case of copy) + - Object version in case of versioned bucket + - Object Size + - eTag +};*/ + +int log_record(rgw::sal::Driver* driver, + const sal::Object* obj, + const req_state* s, + const std::string& op_name, + const std::string& etag, + size_t size, + const configuration& conf, + const DoutPrefixProvider *dpp, + optional_yield y, + bool async_completion, + bool log_source_bucket) { + if (!s->bucket) { + ldpp_dout(dpp, 1) << "ERROR: only bucket operations are logged" << dendl; + return -EINVAL; + } + std::string target_bucket_name; + std::string target_tenant_name; + auto ret = rgw_parse_url_bucket(conf.target_bucket, s->bucket_tenant, target_tenant_name, target_bucket_name); + if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to parse target bucket '" << conf.target_bucket << "', ret = " << ret << dendl; + return ret; + } + const rgw_bucket target_bucket_id(target_tenant_name, target_bucket_name); + std::unique_ptr<rgw::sal::Bucket> target_bucket; + ret = driver->load_bucket(dpp, target_bucket_id, + &target_bucket, y); + if (ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to get target logging bucket '" << target_bucket_id << "'. ret = " << ret << dendl; + return ret; + } + std::string obj_name; + RGWObjVersionTracker objv_tracker; + ret = target_bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, &objv_tracker); + if (ret == 0) { + const auto time_to_commit = time_from_name(obj_name, dpp) + std::chrono::seconds(conf.obj_roll_time); + if (ceph::coarse_real_time::clock::now() > time_to_commit) { + ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' exceeded its time, will be committed to bucket '" << + conf.target_bucket << "'" << dendl; + if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, y, false, &objv_tracker); ret < 0) { + return ret; + } + } else { + ldpp_dout(dpp, 20) << "INFO: record will be written to current logging object '" << obj_name << "'. will be comitted at: " << time_to_commit << dendl; + } + } else if (ret == -ENOENT) { + // try to create the temporary log object for the first time + ret = new_logging_object(conf, target_bucket, obj_name, dpp, y, true, nullptr); + if (ret == 0) { + ldpp_dout(dpp, 20) << "INFO: first time logging for bucket '" << conf.target_bucket << "' and prefix '" << + conf.target_prefix << "'" << dendl; + } else if (ret == -ECANCELED) { + ldpp_dout(dpp, 20) << "INFO: logging object '" << obj_name << "' already exists for bucket '" << conf.target_bucket << "' and prefix" << + conf.target_prefix << "'" << dendl; + } else { + ldpp_dout(dpp, 1) << "ERROR: failed to create logging object of bucket '" << + conf.target_bucket << "' and prefix '" << conf.target_prefix << "' for the first time. ret = " << ret << dendl; + return ret; + } + } else { + ldpp_dout(dpp, 1) << "ERROR: failed to get name of logging object of bucket '" << + conf.target_bucket << "'. ret = " << ret << dendl; + return ret; + } + + std::string record; + const auto tt = ceph::coarse_real_time::clock::to_time_t(s->time); + std::tm t{}; + localtime_r(&tt, &t); + auto user_or_account = s->account_name; + if (user_or_account.empty()) { + s->user->get_id().to_str(user_or_account); + } + auto fqdn = s->info.host; + if (!s->info.domain.empty() && !fqdn.empty()) { + fqdn.append(".").append(s->info.domain); + } + + std::string bucket_owner; + std::string bucket_name; + if (log_source_bucket) { + if (!s->src_object || !s->src_object->get_bucket()) { + ldpp_dout(dpp, 1) << "ERROR: source object or bucket is missing when logging source bucket" << dendl; + return -EINVAL; + } + bucket_owner = to_string(s->src_object->get_bucket()->get_owner()); + bucket_name = s->src_bucket_name; + } else { + bucket_owner = to_string( s->bucket->get_owner()); + bucket_name = full_bucket_name(s->bucket); + } + + switch (conf.logging_type) { + case LoggingType::Standard: + record = fmt::format("{} {} [{:%d/%b/%Y:%H:%M:%S %z}] {} {} {} {} {} \"{} {}{}{} HTTP/1.1\" {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {} {}", + dash_if_empty(bucket_owner), + dash_if_empty(bucket_name), + t, + "-", // no requester IP + dash_if_empty(user_or_account), + dash_if_empty(s->req_id), + op_name, + dash_if_empty_or_null(obj, obj->get_name()), + s->info.method, + s->info.request_uri, + s->info.request_params.empty() ? "" : "?", + s->info.request_params, + dash_if_zero(s->err.http_ret), + dash_if_empty(s->err.err_code), + dash_if_zero(s->content_length), + dash_if_zero(size), + "-", // no total time when logging record + std::chrono::duration_cast<std::chrono::milliseconds>(s->time_elapsed()), + "-", // TODO: referer + "-", // TODO: user agent + dash_if_empty_or_null(obj, obj->get_instance()), + s->info.x_meta_map.contains("x-amz-id-2") ? s->info.x_meta_map.at("x-amz-id-2") : "-", + "-", // TODO: Signature Version (SigV2 or SigV4) + "-", // TODO: SSL cipher. e.g. "ECDHE-RSA-AES128-GCM-SHA256" + "-", // TODO: Auth type. e.g. "AuthHeader" + dash_if_empty(fqdn), + "-", // TODO: TLS version. e.g. "TLSv1.2" or "TLSv1.3" + "-", // no access point ARN + (s->has_acl_header) ? "Yes" : "-"); + break; + case LoggingType::Journal: + record = fmt::format("{} {} [{:%d/%b/%Y:%H:%M:%S %z}] {} {} {} {} {}", + dash_if_empty(to_string(s->bucket->get_owner())), + dash_if_empty(full_bucket_name(s->bucket)), + t, + op_name, + dash_if_empty_or_null(obj, obj->get_name()), + dash_if_zero(size), + dash_if_empty_or_null(obj, obj->get_instance()), + dash_if_empty(etag)); + break; + case LoggingType::Any: + ldpp_dout(dpp, 1) << "ERROR: failed to format record when writing to logging object '" << + obj_name << "' due to unsupported logging type" << dendl; + return -EINVAL; + } + + if (ret = target_bucket->write_logging_object(obj_name, + record, + y, + dpp, + async_completion); ret < 0 && ret != -EFBIG) { + ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" << + obj_name << "'. ret = " << ret << dendl; + return ret; + } + if (ret == -EFBIG) { + ldpp_dout(dpp, 20) << "WARNING: logging object '" << obj_name << "' is full, will be committed to bucket '" << + conf.target_bucket << "'" << dendl; + if (ret = rollover_logging_object(conf, target_bucket, obj_name, dpp, y, true, nullptr); ret < 0 ) { + return ret; + } + if (ret = target_bucket->write_logging_object(obj_name, + record, + y, + dpp, + async_completion); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to write record to logging object '" << + obj_name << "'. ret = " << ret << dendl; + return ret; + } + } + + ldpp_dout(dpp, 20) << "INFO: wrote logging record: '" << record + << "' to '" << obj_name << "'" << dendl; + return 0; +} + +std::string object_name_oid(const rgw::sal::Bucket* bucket, const std::string& prefix) { + // TODO: do i need bucket marker in the name? + return fmt::format("logging.{}.bucket.{}/{}", bucket->get_tenant(), bucket->get_bucket_id(), prefix); +} + +int log_record(rgw::sal::Driver* driver, + LoggingType type, + const sal::Object* obj, + const req_state* s, + const std::string& op_name, + const std::string& etag, + size_t size, + const DoutPrefixProvider *dpp, + optional_yield y, + bool async_completion, + bool log_source_bucket) { + if (!s->bucket) { + // logging only bucket operations + return 0; + } + // check if bucket logging is needed + const auto& bucket_attrs = s->bucket->get_attrs(); + auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (iter == bucket_attrs.end()) { + return 0; + } + configuration configuration; + try { + configuration.enabled = true; + auto bl_iter = iter->second.cbegin(); + decode(configuration, bl_iter); + if (type != LoggingType::Any && configuration.logging_type != type) { + return 0; + } + if (configuration.key_filter.has_content()) { + if (!match(configuration.key_filter, obj->get_name())) { + return 0; + } + } + ldpp_dout(dpp, 20) << "INFO: found matching logging configuration of bucket '" << s->bucket->get_info().bucket << + "' configuration: " << configuration.to_json_str() << dendl; + if (auto ret = log_record(driver, obj, s, op_name, etag, size, configuration, dpp, y, async_completion, log_source_bucket); ret < 0) { + ldpp_dout(dpp, 1) << "ERROR: failed to perform logging for bucket '" << s->bucket->get_info().bucket << + "'. ret=" << ret << dendl; + return ret; + } + } catch (buffer::error& err) { + ldpp_dout(dpp, 1) << "ERROR: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "'. error: " << err.what() << dendl; + return -EINVAL; + } + return 0; +} + +int get_bucket_id(const std::string& bucket_name, const std::string& tenant_name, rgw_bucket& bucket_id) { + std::string parsed_bucket_name; + std::string parsed_tenant_name; + if (const auto ret = rgw_parse_url_bucket(bucket_name, tenant_name, parsed_tenant_name, parsed_bucket_name); ret < 0) { + return ret; + } + bucket_id = rgw_bucket{parsed_tenant_name, parsed_bucket_name}; + return 0; +} + +int update_bucket_logging_sources(const DoutPrefixProvider* dpp, rgw::sal::Driver* driver, const rgw_bucket& target_bucket_id, const rgw_bucket& src_bucket_id, bool add, optional_yield y) { + std::unique_ptr<rgw::sal::Bucket> target_bucket; + const auto ret = driver->load_bucket(dpp, target_bucket_id, &target_bucket, y); + if (ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to get target bucket '" << target_bucket_id << "', ret = " << ret << dendl; + return ret; + } + return update_bucket_logging_sources(dpp, target_bucket, src_bucket_id, add, y); +} + +int update_bucket_logging_sources(const DoutPrefixProvider* dpp, std::unique_ptr<rgw::sal::Bucket>& bucket, const rgw_bucket& src_bucket_id, bool add, optional_yield y) { + return retry_raced_bucket_write(dpp, bucket.get(), [dpp, &bucket, &src_bucket_id, add, y] { + auto& attrs = bucket->get_attrs(); + auto iter = attrs.find(RGW_ATTR_BUCKET_LOGGING_SOURCES); + if (iter == attrs.end()) { + if (!add) { + ldpp_dout(dpp, 20) << "INFO: no logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES + << "' for bucket '" << bucket->get_info().bucket << "', nothing to remove" << dendl; + return 0; + } + source_buckets sources{src_bucket_id}; + bufferlist bl; + ceph::encode(sources, bl); + attrs.insert(std::make_pair(RGW_ATTR_BUCKET_LOGGING_SOURCES, std::move(bl))); + return bucket->merge_and_store_attrs(dpp, attrs, y); + } + try { + source_buckets sources; + ceph::decode(sources, iter->second); + if ((add && sources.insert(src_bucket_id).second) || + (!add && sources.erase(src_bucket_id) > 0)) { + bufferlist bl; + ceph::encode(sources, bl); + iter->second = std::move(bl); + return bucket->merge_and_store_attrs(dpp, attrs, y); + } + } catch (buffer::error& err) { + ldpp_dout(dpp, 1) << "WARNING: failed to decode logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES + << "' for bucket '" << bucket->get_info().bucket << "', error: " << err.what() << dendl; + } + ldpp_dout(dpp, 20) << "INFO: logging source '" << src_bucket_id << "' already " << + (add ? "added to" : "removed from") << " bucket '" << bucket->get_info().bucket << "'" << dendl; + return 0; + }, y); +} + + +int bucket_deletion_cleanup(const DoutPrefixProvider* dpp, + sal::Driver* driver, + sal::Bucket* bucket, + optional_yield y) { + // if the bucket is used a log bucket, we should delete all pending log objects + // and also delete the object holding the pending object name + auto& attrs = bucket->get_attrs(); + if (const auto iter = attrs.find(RGW_ATTR_BUCKET_LOGGING_SOURCES); iter != attrs.end()) { + try { + source_buckets sources; + ceph::decode(sources, iter->second); + for (const auto& source : sources) { + std::unique_ptr<rgw::sal::Bucket> src_bucket; + if (const auto ret = driver->load_bucket(dpp, source, &src_bucket, y); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to get logging source bucket '" << source << "' for log bucket '" << + bucket->get_info().bucket << "', ret = " << ret << dendl; + continue; + } + auto& src_attrs = src_bucket->get_attrs(); + if (const auto iter = src_attrs.find(RGW_ATTR_BUCKET_LOGGING); iter != src_attrs.end()) { + configuration conf; + try { + auto bl_iter = iter->second.cbegin(); + decode(conf, bl_iter); + std::string obj_name; + RGWObjVersionTracker objv; + if (const auto ret = bucket->get_logging_object_name(obj_name, conf.target_prefix, y, dpp, &objv); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to get logging object name for log bucket '" << bucket->get_info().bucket << + "', ret = " << ret << dendl; + continue; + } + if (const auto ret = bucket->remove_logging_object(obj_name, y, dpp); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to delete pending logging object '" << obj_name << "' for log bucket '" << + bucket->get_info().bucket << "', ret = " << ret << dendl; + continue; + } + ldpp_dout(dpp, 20) << "INFO: successfully deleted pending logging object '" << obj_name << "' from deleted log bucket '" << + bucket->get_info().bucket << "'" << dendl; + if (const auto ret = bucket->remove_logging_object_name(conf.target_prefix, y, dpp, &objv); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to delete object holding bucket logging object name for log bucket '" << + bucket->get_info().bucket << "', ret = " << ret << dendl; + continue; + } + ldpp_dout(dpp, 20) << "INFO: successfully deleted object holding bucket logging object name from deleted log bucket '" << + bucket->get_info().bucket << "'" << dendl; + } catch (buffer::error& err) { + ldpp_dout(dpp, 1) << "WARNING: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "' of bucket '" << src_bucket->get_info().bucket << "', error: " << err.what() << dendl; + } + } + } + } catch (buffer::error& err) { + ldpp_dout(dpp, 1) << "WARNING: failed to decode logging sources attribute '" << RGW_ATTR_BUCKET_LOGGING_SOURCES + << "' for bucket '" << bucket->get_info().bucket << "', error: " << err.what() << dendl; + return -EIO; + } + } + + return source_bucket_cleanup(dpp, driver, bucket, false, y); +} + +int source_bucket_cleanup(const DoutPrefixProvider* dpp, + sal::Driver* driver, + sal::Bucket* bucket, + bool remove_attr, + optional_yield y) { + std::optional<configuration> conf; + const auto& info = bucket->get_info(); + if (const auto ret = retry_raced_bucket_write(dpp, bucket, [dpp, bucket, &conf, &info, remove_attr, y] { + auto& attrs = bucket->get_attrs(); + if (auto iter = attrs.find(RGW_ATTR_BUCKET_LOGGING); iter != attrs.end()) { + try { + auto bl_iter = iter->second.cbegin(); + configuration tmp_conf; + tmp_conf.enabled = true; + decode(tmp_conf, bl_iter); + conf = std::move(tmp_conf); + } catch (buffer::error& err) { + ldpp_dout(dpp, 1) << "WARNING: failed to decode existing logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "' of bucket '" << info.bucket << "', error: " << err.what() << dendl; + return -EIO; + } + if (remove_attr) { + attrs.erase(iter); + return bucket->merge_and_store_attrs(dpp, attrs, y); + } + } + // nothing to remove or no need to remove + return 0; + }, y); ret < 0) { + if (remove_attr) { + ldpp_dout(dpp, 1) << "ERROR: failed to remove logging attribute '" << RGW_ATTR_BUCKET_LOGGING << "' from bucket '" << + info.bucket << "', ret = " << ret << dendl; + } + return ret; + } + if (!conf) { + // no logging attribute found + return 0; + } + if (const auto ret = commit_logging_object(*conf, dpp, driver, info.bucket.tenant, y); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: could not commit pending logging object of bucket '" << + info.bucket << "', ret = " << ret << dendl; + } else { + ldpp_dout(dpp, 20) << "INFO: successfully committed pending logging object of bucket '" << info.bucket << "'" << dendl; + } + rgw_bucket target_bucket_id; + rgw_bucket src_bucket_id{info.bucket.tenant, info.bucket.name}; + if (const auto ret = get_bucket_id(conf->target_bucket, info.bucket.tenant, target_bucket_id); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: failed to parse target bucket '" << conf->target_bucket << "', ret = " << ret << dendl; + return 0; + } + if (const auto ret = update_bucket_logging_sources(dpp, driver, target_bucket_id, src_bucket_id, false, y); ret < 0) { + ldpp_dout(dpp, 1) << "WARNING: could not update bucket logging source '" << + info.bucket << "', ret = " << ret << dendl; + return 0; + } + ldpp_dout(dpp, 20) << "INFO: successfully updated bucket logging source '" << + info.bucket << "'"<< dendl; + return 0; +} + +} // namespace rgw::bucketlogging + diff --git a/src/rgw/rgw_bucket_logging.h b/src/rgw/rgw_bucket_logging.h new file mode 100644 index 00000000000..cbdb8b55f88 --- /dev/null +++ b/src/rgw/rgw_bucket_logging.h @@ -0,0 +1,250 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#pragma once + +#include <string> +#include <cstdint> +#include "rgw_sal_fwd.h" +#include "include/buffer.h" +#include "include/encoding.h" +#include "common/async/yield_context.h" +#include "rgw_s3_filter.h" + +class XMLObj; +namespace ceph { class Formatter; } +class DoutPrefixProvider; +struct req_state; +struct RGWObjVersionTracker; +class RGWOp; + +namespace rgw::bucketlogging { +/* S3 bucket logging configuration + * based on: https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLogging.html + * with ceph extensions +<BucketLoggingStatus xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> + <LoggingEnabled> + <TargetBucket>string</TargetBucket> + <TargetGrants> + <Grant> + <Grantee> + <DisplayName>string</DisplayName> + <EmailAddress>string</EmailAddress> + <ID>string</ID> + <xsi:type>string</xsi:type> + <URI>string</URI> + </Grantee> + <Permission>string</Permission> + </Grant> + </TargetGrants> + <TargetObjectKeyFormat> + <PartitionedPrefix> + <PartitionDateSource>DeliveryTime|EventTime</PartitionDateSource> + </PartitionedPrefix> + <SimplePrefix> + </SimplePrefix> + </TargetObjectKeyFormat> + <TargetPrefix>string</TargetPrefix> + <LoggingType>Standard|Journal</LoggingType> <!-- Ceph extension --> + <ObjectRollTime>integer</ObjectRollTime> <!-- Ceph extension --> + <RecordsBatchSize>integer</RecordsBatchSize> <!-- Ceph extension --> + <Filter> + <S3Key> + <FilterRule> + <Name>suffix/prefix/regex</Name> + <Value></Value> + </FilterRule> + </S3Key> + </Filter> + </LoggingEnabled> +</BucketLoggingStatus> +*/ + +enum class KeyFormat {Partitioned, Simple}; +enum class LoggingType {Standard, Journal, Any}; +enum class PartitionDateSource {DeliveryTime, EventTime}; + +struct configuration { + bool operator==(const configuration& rhs) const { + return enabled == rhs.enabled && + target_bucket == rhs.target_bucket && + obj_key_format == rhs.obj_key_format && + target_prefix == rhs.target_prefix && + obj_roll_time == rhs.obj_roll_time && + logging_type == rhs.logging_type && + records_batch_size == rhs.records_batch_size && + date_source == rhs.date_source && + key_filter == rhs.key_filter; + } + uint32_t default_obj_roll_time = 300; + bool enabled = false; + std::string target_bucket; + KeyFormat obj_key_format = KeyFormat::Simple; + // target object key formats: + // Partitioned: [DestinationPrefix][SourceAccountId]/[SourceRegion]/[SourceBucket]/[YYYY]/[MM]/[DD]/[YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString] + // Simple: [DestinationPrefix][YYYY]-[MM]-[DD]-[hh]-[mm]-[ss]-[UniqueString] + std::string target_prefix; // a prefix for all log object keys. + // useful when multiple bucket log to the same target + // or when the target bucket is used for other things than logs + uint32_t obj_roll_time; // time in seconds to move object to bucket and start another object + LoggingType logging_type = LoggingType::Standard; + // in case of "Standard: logging type, all bucket operations are logged + // in case of "Journal" logging type only the following operations are logged: PUT, COPY, MULTI/DELETE, Complete MPU + uint32_t records_batch_size = 0; // how many records to batch in memory before writing to the object + // if set to zero, records are written syncronously to the object. + // if obj_roll_time is reached, the batch of records will be written to the object + // regardless of the number of records + PartitionDateSource date_source = PartitionDateSource::DeliveryTime; + // EventTime: use only year, month, and day. The hour, minutes and seconds are set to 00 in the key + // DeliveryTime: the time the log object was created + rgw_s3_key_filter key_filter; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + void dump(Formatter *f) const; // json + std::string to_json_str() const; + + void encode(ceph::bufferlist& bl) const { + ENCODE_START(1, 1, bl); + encode(target_bucket, bl); + encode(static_cast<int>(obj_key_format), bl); + encode(target_prefix, bl); + encode(obj_roll_time, bl); + encode(static_cast<int>(logging_type), bl); + encode(records_batch_size, bl); + encode(static_cast<int>(date_source), bl); + if (logging_type == LoggingType::Journal) { + encode(key_filter, bl); + } + ENCODE_FINISH(bl); + } + + void decode(bufferlist::const_iterator& bl) { + DECODE_START(1, bl); + decode(target_bucket, bl); + int type; + decode(type, bl); + obj_key_format = static_cast<KeyFormat>(type); + decode(target_prefix, bl); + decode(obj_roll_time, bl); + decode(type, bl); + logging_type = static_cast<LoggingType>(type); + decode(records_batch_size, bl); + decode(type, bl); + date_source = static_cast<PartitionDateSource>(type); + if (logging_type == LoggingType::Journal) { + decode(key_filter, bl); + } + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(configuration) + +using source_buckets = std::set<rgw_bucket>; + +constexpr unsigned MAX_BUCKET_LOGGING_BUFFER = 1000; + +using bucket_logging_records = std::array<std::string, MAX_BUCKET_LOGGING_BUFFER>; + +template <typename Records> +inline std::string to_string(const Records& records) { + std::string str_records; + for (const auto& record : records) { + str_records.append(to_string(record)).append("\n"); + } + return str_records; +} + +// log a bucket logging record according to the configuration +int log_record(rgw::sal::Driver* driver, + const sal::Object* obj, + const req_state* s, + const std::string& op_name, + const std::string& etag, + size_t size, + const configuration& conf, + const DoutPrefixProvider *dpp, + optional_yield y, + bool async_completion, + bool log_source_bucket); + +// commit the pending log objec to the log bucket +// and create a new pending log object +// if "must_commit" is "false" the function will return success even if the pending log object was not committed +int rollover_logging_object(const configuration& conf, + const std::unique_ptr<rgw::sal::Bucket>& bucket, + std::string& obj_name, + const DoutPrefixProvider *dpp, + optional_yield y, + bool must_commit, + RGWObjVersionTracker* objv_tracker); + +// commit the pending log object to the log bucket +// use this for cleanup, when new pending object is not needed +// and target bucket is known +int commit_logging_object(const configuration& conf, + const std::unique_ptr<rgw::sal::Bucket>& target_bucket, + const DoutPrefixProvider *dpp, + optional_yield y); + +// commit the pending log object to the log bucket +// use this for cleanup, when new pending object is not needed +// and target bucket shoud be loaded based on the configuration +int commit_logging_object(const configuration& conf, + const DoutPrefixProvider *dpp, + rgw::sal::Driver* driver, + const std::string& tenant_name, + optional_yield y); + +// return the oid of the object holding the name of the temporary logging object +// bucket - log bucket +// prefix - logging prefix from configuration. should be used when multiple buckets log into the same log bucket +std::string object_name_oid(const rgw::sal::Bucket* bucket, const std::string& prefix); + +// log a bucket logging record according to type +// configuration is fetched from bucket attributes +// if no configuration exists, or if type does not match the function return zero (success) +int log_record(rgw::sal::Driver* driver, + LoggingType type, + const sal::Object* obj, + const req_state* s, + const std::string& op_name, + const std::string& etag, + size_t size, + const DoutPrefixProvider *dpp, + optional_yield y, + bool async_completion, + bool log_source_bucket); + +// return (by ref) an rgw_bucket object with the bucket name and tenant name +// fails if the bucket name is not in the format: [tenant name:]<bucket name> +int get_bucket_id(const std::string& bucket_name, const std::string& tenant_name, rgw_bucket& bucket_id); + +// update (add or remove) a source bucket from the list of source buckets in the target bucket +// use this function when the target bucket is already loaded +int update_bucket_logging_sources(const DoutPrefixProvider* dpp, std::unique_ptr<rgw::sal::Bucket>& bucket, + const rgw_bucket& src_bucket, bool add, optional_yield y); + +// update (add or remove) a source bucket from the list of source buckets in the target bucket +// use this function when the target bucket is not known and needs to be loaded +int update_bucket_logging_sources(const DoutPrefixProvider* dpp, rgw::sal::Driver* driver, const rgw_bucket& target_bucket_id, + const rgw_bucket& src_bucket_id, bool add, optional_yield y); + +// when source bucket is deleted, all pending log objects should be comitted to the log bucket +// when the target bucket is deleted, all pending log objects should be deleted, as well as the object holding the pending log object name +int bucket_deletion_cleanup(const DoutPrefixProvider* dpp, + sal::Driver* driver, + sal::Bucket* bucket, + optional_yield y); + +// if bucket has bucket logging configuration associated with it then: +// if "remove_attr" is true, the bucket logging configuration should be removed from the bucket +// in addition: +// any pending log objects should be comitted to the log bucket +// and the log bucket should be updated to remove the bucket as a source +int source_bucket_cleanup(const DoutPrefixProvider* dpp, + sal::Driver* driver, + sal::Bucket* bucket, + bool remove_attr, + optional_yield y); +} // namespace rgw::bucketlogging + diff --git a/src/rgw/rgw_cksum_pipe.cc b/src/rgw/rgw_cksum_pipe.cc index e06957e2715..0bec8d341af 100644 --- a/src/rgw/rgw_cksum_pipe.cc +++ b/src/rgw/rgw_cksum_pipe.cc @@ -18,6 +18,7 @@ #include <string> #include <fmt/format.h> #include <boost/algorithm/string.hpp> +#include "rgw_cksum.h" #include "rgw_common.h" #include "common/dout.h" #include "rgw_client_io.h" @@ -34,7 +35,8 @@ namespace rgw::putobj { {} std::unique_ptr<RGWPutObj_Cksum> RGWPutObj_Cksum::Factory( - rgw::sal::DataProcessor* next, const RGWEnv& env) + rgw::sal::DataProcessor* next, const RGWEnv& env, + rgw::cksum::Type override_type) { /* look for matching headers */ auto algo_header = cksum_algorithm_hdr(env); @@ -49,6 +51,13 @@ namespace rgw::putobj { throw rgw::io::Exception(EINVAL, std::system_category()); } /* no checksum header */ + if (override_type != rgw::cksum::Type::none) { + /* XXXX safe? do we need to fixup env as well? */ + auto algo_header = cksum_algorithm_hdr(override_type); + return + std::make_unique<RGWPutObj_Cksum>( + next, override_type, std::move(algo_header)); + } return std::unique_ptr<RGWPutObj_Cksum>(); } diff --git a/src/rgw/rgw_cksum_pipe.h b/src/rgw/rgw_cksum_pipe.h index fddcd283c84..c459d156335 100644 --- a/src/rgw/rgw_cksum_pipe.h +++ b/src/rgw/rgw_cksum_pipe.h @@ -20,6 +20,7 @@ #include <tuple> #include <cstring> #include <boost/algorithm/string/case_conv.hpp> +#include "rgw_cksum.h" #include "rgw_cksum_digest.h" #include "rgw_common.h" #include "rgw_putobj.h" @@ -29,6 +30,38 @@ namespace rgw::putobj { namespace cksum = rgw::cksum; using cksum_hdr_t = std::pair<const char*, const char*>; + static inline const cksum_hdr_t cksum_algorithm_hdr(rgw::cksum::Type t) { + static constexpr std::string_view hdr = + "HTTP_X_AMZ_SDK_CHECKSUM_ALGORITHM"; + using rgw::cksum::Type; + switch (t) { + case Type::sha256: + return cksum_hdr_t(hdr.data(), "SHA256"); + break; + case Type::crc32: + return cksum_hdr_t(hdr.data(), "CRC32"); + break; + case Type::crc32c: + return cksum_hdr_t(hdr.data(), "CRC32C"); + break; + case Type::xxh3: + return cksum_hdr_t(hdr.data(), "XX3"); + break; + case Type::sha1: + return cksum_hdr_t(hdr.data(), "SHA1"); + break; + case Type::sha512: + return cksum_hdr_t(hdr.data(), "SHA512"); + break; + case Type::blake3: + return cksum_hdr_t(hdr.data(), "BLAKE3"); + break; + default: + break; + }; + return cksum_hdr_t(nullptr, nullptr);; + } + static inline const cksum_hdr_t cksum_algorithm_hdr(const RGWEnv& env) { /* If the individual checksum value you provide through x-amz-checksum-algorithm doesn't match the checksum algorithm @@ -102,7 +135,8 @@ namespace rgw::putobj { using VerifyResult = std::tuple<bool, const cksum::Cksum&>; static std::unique_ptr<RGWPutObj_Cksum> Factory( - rgw::sal::DataProcessor* next, const RGWEnv&); + rgw::sal::DataProcessor* next, const RGWEnv&, + rgw::cksum::Type override_type); RGWPutObj_Cksum(rgw::sal::DataProcessor* next, rgw::cksum::Type _type, cksum_hdr_t&& _hdr); diff --git a/src/rgw/rgw_common.cc b/src/rgw/rgw_common.cc index 1a59ba02999..6610538542c 100644 --- a/src/rgw/rgw_common.cc +++ b/src/rgw/rgw_common.cc @@ -63,6 +63,7 @@ rgw_http_errors rgw_http_s3_errors({ { ERR_INVALID_DIGEST, {400, "InvalidDigest" }}, { ERR_BAD_DIGEST, {400, "BadDigest" }}, { ERR_INVALID_LOCATION_CONSTRAINT, {400, "InvalidLocationConstraint" }}, + { ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION, {400, "IllegalLocationConstraintException" }}, { ERR_ZONEGROUP_DEFAULT_PLACEMENT_MISCONFIGURATION, {400, "ZonegroupDefaultPlacementMisconfiguration" }}, { ERR_INVALID_BUCKET_NAME, {400, "InvalidBucketName" }}, { ERR_INVALID_OBJECT_NAME, {400, "InvalidObjectName" }}, @@ -3206,3 +3207,14 @@ void RGWObjVersionTracker::generate_new_write_ver(CephContext *cct) append_rand_alpha(cct, write_version.tag, write_version.tag, TAG_LEN); } +boost::optional<rgw::IAM::Policy> +get_iam_policy_from_attr(CephContext* cct, + const std::map<std::string, bufferlist>& attrs, + const std::string& tenant) +{ + if (auto i = attrs.find(RGW_ATTR_IAM_POLICY); i != attrs.end()) { + return Policy(cct, &tenant, i->second.to_str(), false); + } else { + return boost::none; + } +} diff --git a/src/rgw/rgw_common.h b/src/rgw/rgw_common.h index a8f6a1107a9..99f7db4f569 100644 --- a/src/rgw/rgw_common.h +++ b/src/rgw/rgw_common.h @@ -107,6 +107,8 @@ using ceph::crypto::MD5; #define RGW_ATTR_SLO_UINDICATOR RGW_ATTR_META_PREFIX "static-large-object" #define RGW_ATTR_X_ROBOTS_TAG RGW_ATTR_PREFIX "x-robots-tag" #define RGW_ATTR_STORAGE_CLASS RGW_ATTR_PREFIX "storage_class" +#define RGW_ATTR_BUCKET_LOGGING RGW_ATTR_PREFIX "logging" +#define RGW_ATTR_BUCKET_LOGGING_SOURCES RGW_ATTR_PREFIX "logging-sources" /* S3 Object Lock*/ #define RGW_ATTR_OBJECT_LOCK RGW_ATTR_PREFIX "object-lock" @@ -336,6 +338,7 @@ inline constexpr const char* RGW_REST_STS_XMLNS = #define ERR_PRESIGNED_URL_EXPIRED 2223 #define ERR_PRESIGNED_URL_DISABLED 2224 #define ERR_AUTHORIZATION 2225 // SNS 403 AuthorizationError +#define ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION 2226 #define ERR_BUSY_RESHARDING 2300 // also in cls_rgw_types.h, don't change! #define ERR_NO_SUCH_ENTITY 2301 @@ -1746,24 +1749,22 @@ rgw::IAM::Effect evaluate_iam_policies( bool verify_user_permission(const DoutPrefixProvider* dpp, req_state * const s, - const RGWAccessControlPolicy& user_acl, - const std::vector<rgw::IAM::Policy>& user_policies, - const std::vector<rgw::IAM::Policy>& session_policies, - const rgw::ARN& res, - const uint64_t op, - bool mandatory_policy=true); -bool verify_user_permission_no_policy(const DoutPrefixProvider* dpp, - req_state * const s, - const RGWAccessControlPolicy& user_acl, - const int perm); -bool verify_user_permission(const DoutPrefixProvider* dpp, - req_state * const s, const rgw::ARN& res, const uint64_t op, bool mandatory_policy=true); bool verify_user_permission_no_policy(const DoutPrefixProvider* dpp, req_state * const s, int perm); +bool verify_bucket_permission(const DoutPrefixProvider* dpp, + struct perm_state_base * const s, + const rgw::ARN& arn, + bool account_root, + const RGWAccessControlPolicy& user_acl, + const RGWAccessControlPolicy& bucket_acl, + const boost::optional<rgw::IAM::Policy>& bucket_policy, + const std::vector<rgw::IAM::Policy>& identity_policies, + const std::vector<rgw::IAM::Policy>& session_policies, + const uint64_t op); bool verify_bucket_permission( const DoutPrefixProvider* dpp, req_state * const s, @@ -2011,3 +2012,8 @@ struct AioCompletionDeleter { void operator()(librados::AioCompletion* c) { c->release(); } }; using aio_completion_ptr = std::unique_ptr<librados::AioCompletion, AioCompletionDeleter>; + +extern boost::optional<rgw::IAM::Policy> +get_iam_policy_from_attr(CephContext* cct, + const std::map<std::string, bufferlist>& attrs, + const std::string& tenant); diff --git a/src/rgw/rgw_file_int.h b/src/rgw/rgw_file_int.h index 0a1db645207..84eff1e252e 100644 --- a/src/rgw/rgw_file_int.h +++ b/src/rgw/rgw_file_int.h @@ -2298,6 +2298,8 @@ public: std::string uri; std::map<std::string, buffer::list> attrs; RGWLibFS::BucketStats& bs; + real_time ctime; + bool name_matched = false; RGWStatBucketRequest(CephContext* _cct, std::unique_ptr<rgw::sal::User> _user, const std::string& _path, @@ -2312,9 +2314,7 @@ public: return (iter != attrs.end()) ? &(iter->second) : nullptr; } - real_time get_ctime() const { - return bucket->get_creation_time(); - } + real_time get_ctime() { return ctime; } bool only_bucket() override { return false; } @@ -2342,22 +2342,26 @@ public: return 0; } - virtual int get_params() { - return 0; + int get_params(optional_yield) override { return 0; } + + void complete() override { + // get_state() will no longer be there after execute_req() + // so save what we need from get_state()->bucket + ctime = get_state()->bucket->get_creation_time(); + name_matched = get_state()->bucket->get_name().length() > 0; + + RGWOp::complete(); } void send_response() override { - bucket->get_creation_time() = get_state()->bucket->get_info().creation_time; bs.size = stats.size; bs.size_rounded = stats.size_rounded; - bs.creation_time = bucket->get_creation_time(); + bs.creation_time = get_state()->bucket->get_info().creation_time; bs.num_entries = stats.num_objects; std::swap(attrs, get_state()->bucket_attrs); } - bool matched() { - return (bucket->get_name().length() > 0); - } + bool matched() { return name_matched; } }; /* RGWStatBucketRequest */ diff --git a/src/rgw/rgw_iam_policy.cc b/src/rgw/rgw_iam_policy.cc index ce76ed4c3c3..ef6761d4222 100644 --- a/src/rgw/rgw_iam_policy.cc +++ b/src/rgw/rgw_iam_policy.cc @@ -94,6 +94,8 @@ static const actpair actpairs[] = { "s3:GetPublicAccessBlock", s3GetPublicAccessBlock }, { "s3:GetObjectAcl", s3GetObjectAcl }, { "s3:GetObject", s3GetObject }, + { "s3:GetObjectAttributes", s3GetObjectAttributes }, + { "s3:GetObjectVersionAttributes", s3GetObjectVersionAttributes }, { "s3:GetObjectTorrent", s3GetObjectTorrent }, { "s3:GetObjectVersionAcl", s3GetObjectVersionAcl }, { "s3:GetObjectVersion", s3GetObjectVersion }, @@ -113,6 +115,7 @@ static const actpair actpairs[] = { "s3:PutBucketCORS", s3PutBucketCORS }, { "s3:PutBucketEncryption", s3PutBucketEncryption }, { "s3:PutBucketLogging", s3PutBucketLogging }, + { "s3:PostBucketLogging", s3PostBucketLogging }, { "s3:PutBucketNotification", s3PutBucketNotification }, { "s3:PutBucketOwnershipControls", s3PutBucketOwnershipControls }, { "s3:PutBucketPolicy", s3PutBucketPolicy }, @@ -1334,6 +1337,7 @@ const char* action_bit_string(uint64_t action) { case s3ListBucketVersions: return "s3:ListBucketVersions"; + case s3ListAllMyBuckets: return "s3:ListAllMyBuckets"; @@ -1406,6 +1410,9 @@ const char* action_bit_string(uint64_t action) { case s3PutBucketLogging: return "s3:PutBucketLogging"; + case s3PostBucketLogging: + return "s3:PostBucketLogging"; + case s3GetBucketTagging: return "s3:GetBucketTagging"; @@ -1475,6 +1482,12 @@ const char* action_bit_string(uint64_t action) { case s3BypassGovernanceRetention: return "s3:BypassGovernanceRetention"; + case s3GetObjectAttributes: + return "s3:GetObjectAttributes"; + + case s3GetObjectVersionAttributes: + return "s3:GetObjectVersionAttributes"; + case s3DescribeJob: return "s3:DescribeJob"; diff --git a/src/rgw/rgw_iam_policy.h b/src/rgw/rgw_iam_policy.h index 1494cbf0b81..dd323ee4b9c 100644 --- a/src/rgw/rgw_iam_policy.h +++ b/src/rgw/rgw_iam_policy.h @@ -81,6 +81,7 @@ enum { s3PutBucketNotification, s3GetBucketLogging, s3PutBucketLogging, + s3PostBucketLogging, s3GetBucketTagging, s3PutBucketTagging, s3GetBucketWebsite, @@ -114,6 +115,8 @@ enum { s3GetBucketEncryption, s3PutBucketEncryption, s3DescribeJob, + s3GetObjectAttributes, + s3GetObjectVersionAttributes, s3All, s3objectlambdaGetObject, @@ -246,6 +249,8 @@ inline int op_to_perm(std::uint64_t op) { case s3GetObjectVersionTagging: case s3GetObjectRetention: case s3GetObjectLegalHold: + case s3GetObjectAttributes: + case s3GetObjectVersionAttributes: case s3ListAllMyBuckets: case s3ListBucket: case s3ListBucketMultipartUploads: @@ -298,6 +303,7 @@ inline int op_to_perm(std::uint64_t op) { case s3PutBucketCORS: case s3PutBucketEncryption: case s3PutBucketLogging: + case s3PostBucketLogging: case s3PutBucketNotification: case s3PutBucketPolicy: case s3PutBucketRequestPayment: diff --git a/src/rgw/rgw_kafka.cc b/src/rgw/rgw_kafka.cc index 0807993338d..b38b1a78ec4 100644 --- a/src/rgw/rgw_kafka.cc +++ b/src/rgw/rgw_kafka.cc @@ -13,6 +13,7 @@ #include <thread> #include <atomic> #include <mutex> +#include <boost/algorithm/string.hpp> #include <boost/functional/hash.hpp> #include <boost/lockfree/queue.hpp> #include "common/dout.h" @@ -595,7 +596,8 @@ public: boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> topic_user_name, - boost::optional<const std::string&> topic_password) { + boost::optional<const std::string&> topic_password, + boost::optional<const std::string&> brokers) { if (stopped) { ldout(cct, 1) << "Kafka connect: manager is stopped" << dendl; return false; @@ -603,8 +605,8 @@ public: std::string user; std::string password; - std::string broker; - if (!parse_url_authority(url, broker, user, password)) { + std::string broker_list; + if (!parse_url_authority(url, broker_list, user, password)) { // TODO: increment counter ldout(cct, 1) << "Kafka connect: URL parsing failed" << dendl; return false; @@ -632,7 +634,13 @@ public: ldout(cct, 1) << "Kafka connect: user/password are only allowed over secure connection" << dendl; return false; } - connection_id_t tmp_id(broker, user, password, ca_location, mechanism, + + if (brokers.has_value()) { + broker_list.append(","); + broker_list.append(brokers.get()); + } + + connection_id_t tmp_id(broker_list, user, password, ca_location, mechanism, use_ssl); std::lock_guard lock(connections_lock); const auto it = connections.find(tmp_id); @@ -652,7 +660,7 @@ public: return false; } - auto conn = std::make_unique<connection_t>(cct, broker, use_ssl, verify_ssl, ca_location, user, password, mechanism); + auto conn = std::make_unique<connection_t>(cct, broker_list, use_ssl, verify_ssl, ca_location, user, password, mechanism); if (!new_producer(conn.get())) { ldout(cct, 10) << "Kafka connect: producer creation failed in new connection" << dendl; return false; @@ -770,11 +778,12 @@ bool connect(connection_id_t& conn_id, boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> user_name, - boost::optional<const std::string&> password) { + boost::optional<const std::string&> password, + boost::optional<const std::string&> brokers) { std::shared_lock lock(s_manager_mutex); if (!s_manager) return false; return s_manager->connect(conn_id, url, use_ssl, verify_ssl, ca_location, - mechanism, user_name, password); + mechanism, user_name, password, brokers); } int publish(const connection_id_t& conn_id, diff --git a/src/rgw/rgw_kafka.h b/src/rgw/rgw_kafka.h index b7aa0d15759..858b185219f 100644 --- a/src/rgw/rgw_kafka.h +++ b/src/rgw/rgw_kafka.h @@ -48,7 +48,8 @@ bool connect(connection_id_t& conn_id, boost::optional<const std::string&> ca_location, boost::optional<const std::string&> mechanism, boost::optional<const std::string&> user_name, - boost::optional<const std::string&> password); + boost::optional<const std::string&> password, + boost::optional<const std::string&> brokers); // publish a message over a connection that was already created int publish(const connection_id_t& conn_id, diff --git a/src/rgw/rgw_lc.cc b/src/rgw/rgw_lc.cc index a7f2ceabad3..c9fb4765d59 100644 --- a/src/rgw/rgw_lc.cc +++ b/src/rgw/rgw_lc.cc @@ -1183,7 +1183,7 @@ public: << " " << oc.wq->thr_name() << dendl; } else { /* ! o.is_delete_marker() */ - r = remove_expired_obj(oc.dpp, oc, !oc.bucket->versioned(), + r = remove_expired_obj(oc.dpp, oc, !oc.bucket->versioning_enabled(), {rgw::notify::ObjectExpirationCurrent, rgw::notify::LifecycleExpirationDelete}); if (r < 0) { diff --git a/src/rgw/rgw_op.cc b/src/rgw/rgw_op.cc index 0dcf1e0f7d5..7b0ca3134a3 100644 --- a/src/rgw/rgw_op.cc +++ b/src/rgw/rgw_op.cc @@ -25,8 +25,10 @@ #include "common/ceph_json.h" #include "common/static_ptr.h" #include "common/perf_counters_key.h" +#include "rgw_cksum.h" #include "rgw_cksum_digest.h" #include "rgw_common.h" +#include "common/split.h" #include "rgw_tracer.h" #include "rgw_rados.h" @@ -64,6 +66,7 @@ #include "rgw_lua.h" #include "rgw_iam_managed_policy.h" #include "rgw_bucket_sync.h" +#include "rgw_bucket_logging.h" #include "services/svc_zone.h" #include "services/svc_quota.h" @@ -148,7 +151,7 @@ int rgw_forward_request_to_master(const DoutPrefixProvider* dpp, // use the master zone's endpoints auto conn = RGWRESTConn{dpp->get_cct(), z->second.id, z->second.endpoints, - creds, zg->second.id, zg->second.api_name}; + creds, site.get_zonegroup().id, zg->second.api_name}; bufferlist outdata; constexpr size_t max_response_size = 128 * 1024; // we expect a very small response int ret = conn.forward(dpp, effective_owner, req, nullptr, @@ -330,19 +333,6 @@ static int get_obj_policy_from_attr(const DoutPrefixProvider *dpp, return ret; } - -static boost::optional<Policy> -get_iam_policy_from_attr(CephContext* cct, - const map<string, bufferlist>& attrs, - const string& tenant) -{ - if (auto i = attrs.find(RGW_ATTR_IAM_POLICY); i != attrs.end()) { - return Policy(cct, &tenant, i->second.to_str(), false); - } else { - return none; - } -} - static boost::optional<PublicAccessBlockConfiguration> get_public_access_conf_from_attr(const map<string, bufferlist>& attrs) { @@ -757,7 +747,7 @@ static int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s, r return 0; } -static int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s) { +int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s) { return rgw_iam_add_buckettags(dpp, s, s->bucket.get()); } @@ -830,7 +820,7 @@ static std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvide return make_tuple(has_existing_obj_tag, has_resource_tag); } -static std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag=true) { +std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag) { return rgw_check_policy_condition(dpp, s->iam_policy, s->iam_identity_policies, s->session_policies, check_obj_exist_tag); } @@ -943,6 +933,17 @@ void handle_replication_status_header( /* * GET on CloudTiered objects either it will synced to other zones. * In all other cases, it will try to fetch the object from remote cloud endpoint. + * + * @return: + * Note - return status may differ based on whether it is RESTORE op or + * READTHROUGH/GET op. + * for e.g, ERR_INVALID_OBJECT_STATE is sent for non cloud-transitioned + * incase of restore op and ERR_REQUEST_TIMEOUT is applicable only for + * read-through etc. + * `<0` : failed to process; s->err.message & op_ret set accrodingly + * `0` : restore request initiated + * `1` : restore is already in progress + * `2` : already restored */ int handle_cloudtier_obj(req_state* s, const DoutPrefixProvider *dpp, rgw::sal::Driver* driver, rgw::sal::Attrs& attrs, bool sync_cloudtiered, std::optional<uint64_t> days, @@ -1051,12 +1052,17 @@ int handle_cloudtier_obj(req_state* s, const DoutPrefixProvider *dpp, rgw::sal:: s->err.message = "restore is still in progress"; } return op_ret; - } else if ((!restore_op) && (restore_status == rgw::sal::RGWRestoreStatus::RestoreAlreadyInProgress)) { - op_ret = -ERR_REQUEST_TIMEOUT; - ldpp_dout(dpp, 5) << "restore is still in progress, please check restore status and retry" << dendl; - s->err.message = "restore is still in progress"; - } else { // CloudRestored..return success - return 0; + } else if (restore_status == rgw::sal::RGWRestoreStatus::RestoreAlreadyInProgress) { + if (!restore_op) { + op_ret = -ERR_REQUEST_TIMEOUT; + ldpp_dout(dpp, 5) << "restore is still in progress, please check restore status and retry" << dendl; + s->err.message = "restore is still in progress"; + return op_ret; + } else { + return 1; // for restore-op, corresponds to RESTORE_ALREADY_IN_PROGRESS + } + } else { + return 2; // corresponds to CLOUD_RESTORED } } catch (const buffer::end_of_buffer&) { //empty manifest; it's not cloud-tiered @@ -1333,9 +1339,9 @@ void RGWDeleteBucketTags::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this, y] { - rgw::sal::Attrs attrs = s->bucket->get_attrs(); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_TAGS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, y); + op_ret = s->bucket->put_info(this, false, real_time(), y); if (op_ret < 0) { ldpp_dout(this, 0) << "RGWDeleteBucketTags() failed to remove RGW_ATTR_TAGS on bucket=" << s->bucket->get_name() @@ -2338,6 +2344,7 @@ void RGWGetObj::execute(optional_yield y) rgw::op_counters::inc(counters, l_rgw_op_get_obj, 1); std::unique_ptr<rgw::sal::Object::ReadOp> read_op(s->object->get_read_op()); + std::string etag; op_ret = get_params(y); if (op_ret < 0) @@ -3117,17 +3124,19 @@ static int load_bucket_stats(const DoutPrefixProvider* dpp, optional_yield y, void RGWStatBucket::execute(optional_yield y) { - if (!s->bucket_exists) { - op_ret = -ERR_NO_SUCH_BUCKET; + op_ret = get_params(y); + if (op_ret < 0) { return; } - op_ret = driver->load_bucket(this, s->bucket->get_key(), &bucket, y); - if (op_ret) { + if (!s->bucket_exists) { + op_ret = -ERR_NO_SUCH_BUCKET; return; } - op_ret = load_bucket_stats(this, y, *s->bucket, stats); + if (report_stats) { + op_ret = load_bucket_stats(this, y, *s->bucket, stats); + } } int RGWListBucket::verify_permission(optional_yield y) @@ -3220,19 +3229,6 @@ void RGWListBucket::execute(optional_yield y) rgw::op_counters::tinc(counters, l_rgw_op_list_obj_lat, s->time_elapsed()); } -int RGWGetBucketLogging::verify_permission(optional_yield y) -{ - auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); - if (has_s3_resource_tag) - rgw_iam_add_buckettags(this, s); - - if (!verify_bucket_permission(this, s, rgw::IAM::s3GetBucketLogging)) { - return -EACCES; - } - - return 0; -} - int RGWGetBucketLocation::verify_permission(optional_yield y) { auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); @@ -3564,54 +3560,62 @@ void RGWCreateBucket::execute(optional_yield y) const rgw::SiteConfig& site = *s->penv.site; const std::optional<RGWPeriod>& period = site.get_period(); const RGWZoneGroup& my_zonegroup = site.get_zonegroup(); - - if (s->system_request) { - // allow system requests to override the target zonegroup. for forwarded - // requests, we'll create the bucket for the originating zonegroup - createparams.zonegroup_id = s->info.args.get(RGW_SYS_PARAM_PREFIX "zonegroup"); - } - + const std::string rgwx_zonegroup = s->info.args.get(RGW_SYS_PARAM_PREFIX "zonegroup"); const RGWZoneGroup* bucket_zonegroup = &my_zonegroup; - if (createparams.zonegroup_id.empty()) { - // default to the local zonegroup - createparams.zonegroup_id = my_zonegroup.id; - } else if (period) { - auto z = period->period_map.zonegroups.find(createparams.zonegroup_id); - if (z == period->period_map.zonegroups.end()) { - ldpp_dout(this, 0) << "could not find zonegroup " - << createparams.zonegroup_id << " in current period" << dendl; - op_ret = -ENOENT; - return; - } - bucket_zonegroup = &z->second; - } else if (createparams.zonegroup_id != my_zonegroup.id) { - ldpp_dout(this, 0) << "zonegroup does not match current zonegroup " - << createparams.zonegroup_id << dendl; - op_ret = -ENOENT; - return; - } - // validate the LocationConstraint + // Validate LocationConstraint if it's provided and enforcement is strict if (!location_constraint.empty() && !relaxed_region_enforcement) { - // on the master zonegroup, allow any valid api_name. otherwise it has to - // match the bucket's zonegroup - if (period && my_zonegroup.is_master) { - if (!period->period_map.zonegroups_by_api.count(location_constraint)) { + if (period) { + auto location_iter = period->period_map.zonegroups_by_api.find(location_constraint); + if (location_iter == period->period_map.zonegroups_by_api.end()) { ldpp_dout(this, 0) << "location constraint (" << location_constraint << ") can't be found." << dendl; op_ret = -ERR_INVALID_LOCATION_CONSTRAINT; - s->err.message = "The specified location-constraint is not valid"; + s->err.message = fmt::format("The {} location constraint is not valid.", + location_constraint); return; } - } else if (bucket_zonegroup->api_name != location_constraint) { + bucket_zonegroup = &location_iter->second; + } else if (location_constraint != my_zonegroup.api_name) { // if we don't have a period, we can only use the current zonegroup - so check if the location matches by api name here ldpp_dout(this, 0) << "location constraint (" << location_constraint - << ") doesn't match zonegroup (" << bucket_zonegroup->api_name - << ')' << dendl; - op_ret = -ERR_INVALID_LOCATION_CONSTRAINT; - s->err.message = "The specified location-constraint is not valid"; + << ") doesn't match zonegroup (" << my_zonegroup.api_name << ")" << dendl; + op_ret = -ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION; + s->err.message = fmt::format("The {} location constraint is incompatible " + "for the region specific endpoint this request was sent to.", + location_constraint); return; } } + // If it's a system request, use the provided zonegroup if available + else if (s->system_request && !rgwx_zonegroup.empty()) { + if (period) { + auto zonegroup_iter = period->period_map.zonegroups.find(rgwx_zonegroup); + if (zonegroup_iter == period->period_map.zonegroups.end()) { + ldpp_dout(this, 0) << "could not find zonegroup " << rgwx_zonegroup + << " in current period" << dendl; + op_ret = -ENOENT; + return; + } + bucket_zonegroup = &zonegroup_iter->second; + } + } + + const bool enforce_location_match = + !period || // No period: no multisite, so no need to enforce location match. + !s->system_request || // All user requests are enforced to match zonegroup's location. + !my_zonegroup.is_master; // but if it's a system request (forwarded) only allow remote creation on master zonegroup. + if (enforce_location_match && !my_zonegroup.equals(bucket_zonegroup->get_id())) { + ldpp_dout(this, 0) << "location constraint (" << bucket_zonegroup->api_name + << ") doesn't match zonegroup (" << my_zonegroup.api_name << ")" << dendl; + op_ret = -ERR_ILLEGAL_LOCATION_CONSTRAINT_EXCEPTION; + s->err.message = fmt::format("The {} location constraint is incompatible " + "for the region specific endpoint this request was sent to.", + bucket_zonegroup->api_name); + return; + } + + // Set the final zonegroup ID + createparams.zonegroup_id = bucket_zonegroup->id; // select and validate the placement target op_ret = select_bucket_placement(this, *bucket_zonegroup, s->user->get_info(), @@ -3620,7 +3624,7 @@ void RGWCreateBucket::execute(optional_yield y) return; } - if (bucket_zonegroup == &my_zonegroup) { + if (my_zonegroup.equals(bucket_zonegroup->get_id())) { // look up the zone placement pool createparams.zone_placement = rgw::find_zone_placement( this, site.get_zone_params(), createparams.placement_rule); @@ -3709,7 +3713,6 @@ void RGWCreateBucket::execute(optional_yield y) if (!driver->is_meta_master()) { // apply bucket creation on the master zone first - bufferlist in_data; JSONParser jp; op_ret = rgw_forward_request_to_master(this, *s->penv.site, s->owner.id, &in_data, &jp, s->info, y); @@ -3786,7 +3789,10 @@ void RGWCreateBucket::execute(optional_yield y) s->bucket->get_info().has_website = !s->bucket->get_info().website_conf.is_empty(); /* This will also set the quota on the bucket. */ - op_ret = s->bucket->merge_and_store_attrs(this, createparams.attrs, y); + s->bucket->set_attrs(std::move(createparams.attrs)); + constexpr bool exclusive = false; // overwrite + constexpr ceph::real_time no_set_mtime{}; + op_ret = s->bucket->put_info(this, exclusive, no_set_mtime, y); } while (op_ret == -ECANCELED && tries++ < 20); /* Restore the proper return code. */ @@ -4337,6 +4343,9 @@ void RGWPutObj::execute(optional_yield y) } return; } + + multipart_cksum_type = upload->cksum_type; + /* upload will go out of scope, so copy the dest placement for later use */ s->dest_placement = *pdest_placement; pdest_placement = &s->dest_placement; @@ -4467,11 +4476,12 @@ void RGWPutObj::execute(optional_yield y) /* optional streaming checksum */ try { cksum_filter = - rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env); + rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env, multipart_cksum_type); } catch (const rgw::io::Exception& e) { op_ret = -e.code().value(); return; } + if (cksum_filter) { filter = &*cksum_filter; } @@ -4618,10 +4628,12 @@ void RGWPutObj::execute(optional_yield y) if (cksum_filter) { const auto& hdr = cksum_filter->header(); + auto expected_ck = cksum_filter->expected(*s->info.env); auto cksum_verify = cksum_filter->verify(*s->info.env); // valid or no supplied cksum cksum = get<1>(cksum_verify); - if (std::get<0>(cksum_verify)) { + if ((!expected_ck) || + std::get<0>(cksum_verify)) { buffer::list cksum_bl; ldpp_dout_fmt(this, 16, @@ -4629,14 +4641,13 @@ void RGWPutObj::execute(optional_yield y) "\n\tcomputed={} == \n\texpected={}", hdr.second, cksum->to_armor(), - cksum_filter->expected(*s->info.env)); + (!!expected_ck) ? expected_ck : "(checksum unavailable)"); cksum->encode(cksum_bl); emplace_attr(RGW_ATTR_CKSUM, std::move(cksum_bl)); } else { /* content checksum mismatch */ auto computed_ck = cksum->to_armor(); - auto expected_ck = cksum_filter->expected(*s->info.env); ldpp_dout_fmt(this, 4, "{} content checksum mismatch" @@ -4677,6 +4688,13 @@ void RGWPutObj::execute(optional_yield y) obj_retention->encode(obj_retention_bl); emplace_attr(RGW_ATTR_OBJECT_RETENTION, std::move(obj_retention_bl)); } + + if (!multipart) { + op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, s->object->get_size(), this, y, false, false); + if (op_ret < 0) { + return; + } + } // don't track the individual parts of multipart uploads. they replicate in // full after CompleteMultipart @@ -4832,7 +4850,8 @@ void RGWPostObj::execute(optional_yield y) /* optional streaming checksum */ try { cksum_filter = - rgw::putobj::RGWPutObj_Cksum::Factory(filter, *s->info.env); + rgw::putobj::RGWPutObj_Cksum::Factory( + filter, *s->info.env, rgw::cksum::Type::none /* no override */); } catch (const rgw::io::Exception& e) { op_ret = -e.code().value(); return; @@ -5180,7 +5199,10 @@ void RGWPutMetadataBucket::execute(optional_yield y) /* Setting attributes also stores the provided bucket info. Due * to this fact, the new quota settings can be serialized with * the same call. */ - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + s->bucket->set_attrs(attrs); + constexpr bool exclusive = false; // overwrite + constexpr ceph::real_time no_set_mtime{}; + op_ret = s->bucket->put_info(this, exclusive, no_set_mtime, s->yield); return op_ret; }, y); } @@ -5282,33 +5304,14 @@ void RGWRestoreObj::execute(optional_yield y) int op_ret = s->object->get_obj_attrs(y, this); if (op_ret < 0) { ldpp_dout(this, 1) << "failed to fetch get_obj_attrs op ret = " << op_ret << dendl; + restore_ret = op_ret; return; } - rgw::sal::Attrs attrs = s->object->get_attrs(); - auto attr_iter = attrs.find(RGW_ATTR_MANIFEST); - if (attr_iter != attrs.end()) { - RGWObjManifest m; - decode(m, attr_iter->second); - RGWObjTier tier_config; - m.get_tier_config(&tier_config); - if (m.get_tier_type() == "cloud-s3") { - ldpp_dout(this, 20) << "execute: expiry days" << expiry_days <<dendl; - op_ret = handle_cloudtier_obj(s, this, driver, attrs, false, expiry_days, true, y); - if (op_ret < 0) { - ldpp_dout(this, 4) << "Cannot get cloud tiered object: " << *s->object - <<". Failing with " << op_ret << dendl; - if (op_ret == -ERR_INVALID_OBJECT_STATE) { - s->err.message = "This object was transitioned to cloud-s3"; - } - } - } else { - ldpp_dout(this, 20) << "not cloud tier object erroring" << dendl; - op_ret = -ERR_INVALID_OBJECT_STATE; - } - } else { - ldpp_dout(this, 20) << " manifest not found" << dendl; - } - ldpp_dout(this, 20) << "completed restore" << dendl; + rgw::sal::Attrs attrs; + attrs = s->object->get_attrs(); + op_ret = handle_cloudtier_obj(s, this, driver, attrs, false, expiry_days, true, y); + restore_ret = op_ret; + ldpp_dout(this, 20) << "Restore completed of object: " << *s->object << "with op ret: " << restore_ret <<dendl; return; } @@ -5539,6 +5542,13 @@ void RGWDeleteObj::execute(optional_yield y) } } + if (op_ret == 0) { + if (auto ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, obj_size, this, y, false, false); ret < 0) { + // don't reply with an error in case of failed delete logging + ldpp_dout(this, 5) << "WARNING: DELETE operation ignores bucket logging failure: " << ret << dendl; + } + } + if (op_ret == -ECANCELED) { op_ret = 0; } @@ -5883,6 +5893,12 @@ void RGWCopyObj::execute(optional_yield y) return; } + etag = s->src_object->get_attrs()[RGW_ATTR_ETAG].to_str(); + op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), etag, obj_size, this, y, false, false); + if (op_ret < 0) { + return; + } + op_ret = s->src_object->copy_object(s->owner, s->user->get_id(), &s->info, @@ -5911,12 +5927,17 @@ void RGWCopyObj::execute(optional_yield y) this, s->yield); + int ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Standard, s->src_object.get(), s, "REST.COPY.OBJECT_GET", etag, obj_size, this, y, true, true); + if (ret < 0) { + ldpp_dout(this, 5) << "WARNING: COPY operation ignores bucket logging failure of the GET part: " << ret << dendl; + } + if (op_ret < 0) { return; } // send request to notification manager - int ret = res->publish_commit(this, obj_size, mtime, etag, s->object->get_instance()); + ret = res->publish_commit(this, obj_size, mtime, etag, s->object->get_instance()); if (ret < 0) { ldpp_dout(this, 1) << "ERROR: publishing notification failed, with error: " << ret << dendl; // too late to rollback operation, hence op_ret is not set here @@ -5969,8 +5990,6 @@ void RGWGetACLs::execute(optional_yield y) acls = ss.str(); } - - int RGWPutACLs::verify_permission(optional_yield y) { bool perm; @@ -5992,6 +6011,74 @@ int RGWPutACLs::verify_permission(optional_yield y) return 0; } +uint16_t RGWGetObjAttrs::recognize_attrs(const std::string& hdr, uint16_t deflt) +{ + auto attrs{deflt}; + auto sa = ceph::split(hdr, ","); + for (auto& k : sa) { + if (boost::iequals(k, "etag")) { + attrs |= as_flag(ReqAttributes::Etag); + } + if (boost::iequals(k, "checksum")) { + attrs |= as_flag(ReqAttributes::Checksum); + } + if (boost::iequals(k, "objectparts")) { + attrs |= as_flag(ReqAttributes::ObjectParts); + } + if (boost::iequals(k, "objectsize")) { + attrs |= as_flag(ReqAttributes::ObjectSize); + } + if (boost::iequals(k, "storageclass")) { + attrs |= as_flag(ReqAttributes::StorageClass); + } + } + return attrs; +} /* RGWGetObjAttrs::recognize_attrs */ + +int RGWGetObjAttrs::verify_permission(optional_yield y) +{ + bool perm = false; + auto [has_s3_existing_tag, has_s3_resource_tag] = + rgw_check_policy_condition(this, s); + + if (! rgw::sal::Object::empty(s->object.get())) { + + auto iam_action1 = s->object->get_instance().empty() ? + rgw::IAM::s3GetObject : + rgw::IAM::s3GetObjectVersion; + + auto iam_action2 = s->object->get_instance().empty() ? + rgw::IAM::s3GetObjectAttributes : + rgw::IAM::s3GetObjectVersionAttributes; + + if (has_s3_existing_tag || has_s3_resource_tag) { + rgw_iam_add_objtags(this, s, has_s3_existing_tag, has_s3_resource_tag); + } + + /* XXXX the following conjunction should be &&--but iam_action2 is currently not + * hooked up and always fails (but should succeed if the requestor has READ + * acess to the object) */ + perm = (verify_object_permission(this, s, iam_action1) || /* && */ + verify_object_permission(this, s, iam_action2)); + } + + if (! perm) { + return -EACCES; + } + + return 0; +} + +void RGWGetObjAttrs::pre_exec() +{ + rgw_bucket_object_pre_exec(s); +} + +void RGWGetObjAttrs::execute(optional_yield y) +{ + RGWGetObj::execute(y); +} /* RGWGetObjAttrs::execute */ + int RGWGetLC::verify_permission(optional_yield y) { auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); @@ -6373,9 +6460,9 @@ void RGWDeleteCORS::execute(optional_yield y) return op_ret; } - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_CORS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); if (op_ret < 0) { ldpp_dout(this, 0) << "RGWLC::RGWDeleteCORS() failed to set attrs on bucket=" << s->bucket->get_name() << " returned err=" << op_ret << dendl; @@ -6659,6 +6746,14 @@ try_sum_part_cksums(const DoutPrefixProvider *dpp, ++parts_ix; auto& part_cksum = part.second->get_cksum(); + if (! part_cksum) { + ldpp_dout_fmt(dpp, 0, + "ERROR: multipart part checksum not present (ix=={})", + parts_ix); + op_ret = -ERR_INVALID_REQUEST; + return op_ret; + } + ldpp_dout_fmt(dpp, 16, "INFO: {} iterate part: {} {} {}", __func__, parts_ix, part_cksum->type_string(), @@ -6876,7 +6971,13 @@ void RGWCompleteMultipart::execute(optional_yield y) RGWObjVersionTracker& objv_tracker = meta_obj->get_version_tracker(); using prefix_map_t = rgw::sal::MultipartUpload::prefix_map_t; - prefix_map_t processed_prefixes; + prefix_map_t processed_prefixes; + + // no etag and size before completion + op_ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Journal, s->object.get(), s, canonical_name(), "", 0, this, y, false, false); + if (op_ret < 0) { + return; + } op_ret = upload->complete(this, y, s->cct, parts->parts, remove_objs, accounted_size, @@ -7023,17 +7124,30 @@ void RGWAbortMultipart::execute(optional_yield y) return; upload = s->bucket->get_multipart_upload(s->object->get_name(), upload_id); + meta_obj = upload->get_meta_obj(); + meta_obj->set_in_extra_data(true); + meta_obj->get_obj_attrs(s->yield, this); + jspan_context trace_ctx(false, false); if (tracing::rgw::tracer.is_enabled()) { // read meta object attributes for trace info - meta_obj = upload->get_meta_obj(); - meta_obj->set_in_extra_data(true); - meta_obj->get_obj_attrs(s->yield, this); extract_span_context(meta_obj->get_attrs(), trace_ctx); } multipart_trace = tracing::rgw::tracer.add_span(name(), trace_ctx); + int max_lock_secs_mp = + s->cct->_conf.get_val<int64_t>("rgw_mp_lock_max_time"); + utime_t dur(max_lock_secs_mp, 0); + auto serializer = meta_obj->get_serializer(this, "RGWCompleteMultipart"); + op_ret = serializer->try_lock(this, dur, y); + if (op_ret < 0) { + if (op_ret == -ENOENT) { + op_ret = -ERR_NO_SUCH_UPLOAD; + } + return; + } op_ret = upload->abort(this, s->cct, y); + serializer->unlock(); } int RGWListMultipart::verify_permission(optional_yield y) @@ -7292,6 +7406,12 @@ void RGWDeleteMultiObj::handle_individual_object(const rgw_obj_key& o, optional_ if (op_ret == -ENOENT) { op_ret = 0; } + + if (auto ret = rgw::bucketlogging::log_record(driver, rgw::bucketlogging::LoggingType::Any, obj.get(), s, canonical_name(), etag, obj_size, this, y, true, false); ret < 0) { + // don't reply with an error in case of failed delete logging + ldpp_dout(this, 5) << "WARNING: multi DELETE operation ignores bucket logging failure: " << ret << dendl; + } + if (op_ret == 0) { // send request to notification manager int ret = res->publish_commit(dpp, obj_size, ceph::real_clock::now(), etag, version_id); @@ -7331,6 +7451,12 @@ void RGWDeleteMultiObj::execute(optional_yield y) return; } + if (multi_delete->objects.empty()) { + s->err.message = "Missing required element Object"; + op_ret = -ERR_MALFORMED_XML; + return; + } + constexpr int DEFAULT_MAX_NUM = 1000; int max_num = s->cct->_conf->rgw_delete_multi_obj_max_num; if (max_num < 0) { @@ -8486,9 +8612,9 @@ void RGWDeleteBucketPolicy::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this] { - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_IAM_POLICY); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); return op_ret; }, y); } @@ -9006,9 +9132,9 @@ void RGWDeleteBucketPublicAccessBlock::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this] { - rgw::sal::Attrs attrs(s->bucket_attrs); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_PUBLIC_ACCESS); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, s->yield); + op_ret = s->bucket->put_info(this, false, real_time(), s->yield); return op_ret; }, y); } @@ -9117,10 +9243,10 @@ void RGWDeleteBucketEncryption::execute(optional_yield y) } op_ret = retry_raced_bucket_write(this, s->bucket.get(), [this, y] { - rgw::sal::Attrs attrs = s->bucket->get_attrs(); + rgw::sal::Attrs& attrs = s->bucket->get_attrs(); attrs.erase(RGW_ATTR_BUCKET_ENCRYPTION_POLICY); attrs.erase(RGW_ATTR_BUCKET_ENCRYPTION_KEY_ID); - op_ret = s->bucket->merge_and_store_attrs(this, attrs, y); + op_ret = s->bucket->put_info(this, false, real_time(), y); return op_ret; }, y); } diff --git a/src/rgw/rgw_op.h b/src/rgw/rgw_op.h index df05500a437..dcf64c31572 100644 --- a/src/rgw/rgw_op.h +++ b/src/rgw/rgw_op.h @@ -12,6 +12,7 @@ #pragma once +#include <cstdint> #include <limits.h> #include <array> @@ -83,6 +84,10 @@ int rgw_op_get_bucket_policy_from_attr(const DoutPrefixProvider *dpp, RGWAccessControlPolicy& policy, optional_yield y); +std::tuple<bool, bool> rgw_check_policy_condition(const DoutPrefixProvider *dpp, req_state* s, bool check_obj_exist_tag=true); + +int rgw_iam_add_buckettags(const DoutPrefixProvider *dpp, req_state* s); + class RGWHandler { protected: rgw::sal::Driver* driver{nullptr}; @@ -296,6 +301,7 @@ public: } virtual const char* name() const = 0; virtual RGWOpType get_type() { return RGW_OP_UNKNOWN; } + virtual std::string canonical_name() const { return fmt::format("REST.{}.{}", s->info.method, name()); } virtual uint32_t op_mask() { return 0; } @@ -974,18 +980,6 @@ public: virtual bool need_container_stats() { return false; } }; -class RGWGetBucketLogging : public RGWOp { -public: - RGWGetBucketLogging() {} - int verify_permission(optional_yield y) override; - void execute(optional_yield) override { } - - void send_response() override = 0; - const char* name() const override { return "get_bucket_logging"; } - RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOGGING; } - uint32_t op_mask() override { return RGW_OP_TYPE_READ; } -}; - class RGWGetBucketLocation : public RGWOp { public: RGWGetBucketLocation() {} @@ -1094,14 +1088,15 @@ public: class RGWStatBucket : public RGWOp { protected: - std::unique_ptr<rgw::sal::Bucket> bucket; RGWStorageStats stats; + bool report_stats{true}; public: int verify_permission(optional_yield y) override; void pre_exec() override; void execute(optional_yield y) override; + virtual int get_params(optional_yield y) = 0; void send_response() override = 0; const char* name() const override { return "stat_bucket"; } RGWOpType get_type() override { return RGW_OP_STAT_BUCKET; } @@ -1117,6 +1112,7 @@ class RGWCreateBucket : public RGWOp { bool relaxed_region_enforcement = false; RGWCORSConfiguration cors_config; std::set<std::string> rmattr_names; + bufferlist in_data; virtual bool need_metadata_upload() const { return false; } @@ -1243,6 +1239,7 @@ protected: std::string multipart_upload_id; std::string multipart_part_str; int multipart_part_num = 0; + rgw::cksum::Type multipart_cksum_type{rgw::cksum::Type::none}; jspan_ptr multipart_trace; boost::optional<ceph::real_time> delete_at; @@ -1464,6 +1461,7 @@ public: class RGWRestoreObj : public RGWOp { protected: std::optional<uint64_t> expiry_days; + int restore_ret; public: RGWRestoreObj() {} @@ -1649,6 +1647,50 @@ public: uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; } }; +class RGWGetObjAttrs : public RGWGetObj { +protected: + std::string version_id; + std::string expected_bucket_owner; + std::optional<int> marker; + std::optional<int> max_parts; + uint16_t requested_attributes{0}; +#if 0 + /* used to decrypt attributes for objects stored with SSE-C */ + x-amz-server-side-encryption-customer-algorithm + x-amz-server-side-encryption-customer-key + x-amz-server-side-encryption-customer-key-MD5 +#endif +public: + + enum class ReqAttributes : uint16_t { + None = 0, + Etag, + Checksum, + ObjectParts, + StorageClass, + ObjectSize + }; + + static uint16_t as_flag(ReqAttributes attr) { + return 1 << (uint16_t(attr) ? uint16_t(attr) - 1 : 0); + } + + static uint16_t recognize_attrs(const std::string& hdr, uint16_t deflt = 0); + + RGWGetObjAttrs() : RGWGetObj() + { + RGWGetObj::get_data = false; // it's extra false + } + + int verify_permission(optional_yield y) override; + void pre_exec() override; + void execute(optional_yield y) override; + void send_response() override = 0; + const char* name() const override { return "get_obj_attrs"; } + RGWOpType get_type() override { return RGW_OP_GET_OBJ_ATTRS; } + uint32_t op_mask() override { return RGW_OP_TYPE_READ; } +}; /* RGWGetObjAttrs */ + class RGWGetLC : public RGWOp { protected: diff --git a/src/rgw/rgw_op_type.h b/src/rgw/rgw_op_type.h index f0c3b072e47..2c8225d289e 100644 --- a/src/rgw/rgw_op_type.h +++ b/src/rgw/rgw_op_type.h @@ -30,6 +30,7 @@ enum RGWOpType { RGW_OP_COPY_OBJ, RGW_OP_GET_ACLS, RGW_OP_PUT_ACLS, + RGW_OP_GET_OBJ_ATTRS, RGW_OP_GET_CORS, RGW_OP_PUT_CORS, RGW_OP_DELETE_CORS, @@ -116,6 +117,8 @@ enum RGWOpType { RGW_OP_ATTACH_GROUP_POLICY, RGW_OP_DETACH_GROUP_POLICY, RGW_OP_LIST_ATTACHED_GROUP_POLICIES, + RGW_OP_PUT_BUCKET_LOGGING, + RGW_OP_POST_BUCKET_LOGGING, /* rgw specific */ RGW_OP_ADMIN_SET_METADATA, RGW_OP_GET_OBJ_LAYOUT, diff --git a/src/rgw/rgw_process.cc b/src/rgw/rgw_process.cc index 8be7be79069..9ad599b3252 100644 --- a/src/rgw/rgw_process.cc +++ b/src/rgw/rgw_process.cc @@ -21,6 +21,7 @@ #include "rgw_lua_request.h" #include "rgw_tracer.h" #include "rgw_ratelimit.h" +#include "rgw_bucket_logging.h" #include "services/svc_zone_utils.h" @@ -444,6 +445,20 @@ done: rgw_log_op(rest, s, op, penv.olog); } + if (op) { + std::ignore = rgw::bucketlogging::log_record(driver, + rgw::bucketlogging::LoggingType::Standard, + s->object.get(), + s, + op->canonical_name(), + "", + (s->src_object ? s->src_object->get_size() : (s->object ? s->object->get_size() : 0)), + op, + yield, + true, + false); + } + if (http_ret != nullptr) { *http_ret = s->err.http_ret; } diff --git a/src/rgw/rgw_pubsub.cc b/src/rgw/rgw_pubsub.cc index cb68d72d7da..87a46bd61a6 100644 --- a/src/rgw/rgw_pubsub.cc +++ b/src/rgw/rgw_pubsub.cc @@ -62,214 +62,6 @@ void set_event_id(std::string& id, const std::string& hash, const utime_t& ts) { } } -void rgw_s3_key_filter::dump(Formatter *f) const { - if (!has_content()) { - return; - } - f->open_array_section("FilterRules"); - if (!prefix_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "prefix", f); - ::encode_json("Value", prefix_rule, f); - f->close_section(); - } - if (!suffix_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "suffix", f); - ::encode_json("Value", suffix_rule, f); - f->close_section(); - } - if (!regex_rule.empty()) { - f->open_object_section(""); - ::encode_json("Name", "regex", f); - ::encode_json("Value", regex_rule, f); - f->close_section(); - } - f->close_section(); -} - -bool rgw_s3_key_filter::decode_xml(XMLObj* obj) { - XMLObjIter iter = obj->find("FilterRule"); - XMLObj *o; - - const auto throw_if_missing = true; - auto prefix_not_set = true; - auto suffix_not_set = true; - auto regex_not_set = true; - std::string name; - - while ((o = iter.get_next())) { - RGWXMLDecoder::decode_xml("Name", name, o, throw_if_missing); - if (name == "prefix" && prefix_not_set) { - prefix_not_set = false; - RGWXMLDecoder::decode_xml("Value", prefix_rule, o, throw_if_missing); - } else if (name == "suffix" && suffix_not_set) { - suffix_not_set = false; - RGWXMLDecoder::decode_xml("Value", suffix_rule, o, throw_if_missing); - } else if (name == "regex" && regex_not_set) { - regex_not_set = false; - RGWXMLDecoder::decode_xml("Value", regex_rule, o, throw_if_missing); - } else { - throw RGWXMLDecoder::err("invalid/duplicate S3Key filter rule name: '" + name + "'"); - } - } - return true; -} - -void rgw_s3_key_filter::dump_xml(Formatter *f) const { - if (!prefix_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "prefix", f); - ::encode_xml("Value", prefix_rule, f); - f->close_section(); - } - if (!suffix_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "suffix", f); - ::encode_xml("Value", suffix_rule, f); - f->close_section(); - } - if (!regex_rule.empty()) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", "regex", f); - ::encode_xml("Value", regex_rule, f); - f->close_section(); - } -} - -bool rgw_s3_key_filter::has_content() const { - return !(prefix_rule.empty() && suffix_rule.empty() && regex_rule.empty()); -} - -void rgw_s3_key_value_filter::dump(Formatter *f) const { - if (!has_content()) { - return; - } - f->open_array_section("FilterRules"); - for (const auto& key_value : kv) { - f->open_object_section(""); - ::encode_json("Name", key_value.first, f); - ::encode_json("Value", key_value.second, f); - f->close_section(); - } - f->close_section(); -} - -bool rgw_s3_key_value_filter::decode_xml(XMLObj* obj) { - kv.clear(); - XMLObjIter iter = obj->find("FilterRule"); - XMLObj *o; - - const auto throw_if_missing = true; - - std::string key; - std::string value; - - while ((o = iter.get_next())) { - RGWXMLDecoder::decode_xml("Name", key, o, throw_if_missing); - RGWXMLDecoder::decode_xml("Value", value, o, throw_if_missing); - kv.emplace(key, value); - } - return true; -} - -void rgw_s3_key_value_filter::dump_xml(Formatter *f) const { - for (const auto& key_value : kv) { - f->open_object_section("FilterRule"); - ::encode_xml("Name", key_value.first, f); - ::encode_xml("Value", key_value.second, f); - f->close_section(); - } -} - -bool rgw_s3_key_value_filter::has_content() const { - return !kv.empty(); -} - -void rgw_s3_filter::dump(Formatter *f) const { - encode_json("S3Key", key_filter, f); - encode_json("S3Metadata", metadata_filter, f); - encode_json("S3Tags", tag_filter, f); -} - -bool rgw_s3_filter::decode_xml(XMLObj* obj) { - RGWXMLDecoder::decode_xml("S3Key", key_filter, obj); - RGWXMLDecoder::decode_xml("S3Metadata", metadata_filter, obj); - RGWXMLDecoder::decode_xml("S3Tags", tag_filter, obj); - return true; -} - -void rgw_s3_filter::dump_xml(Formatter *f) const { - if (key_filter.has_content()) { - ::encode_xml("S3Key", key_filter, f); - } - if (metadata_filter.has_content()) { - ::encode_xml("S3Metadata", metadata_filter, f); - } - if (tag_filter.has_content()) { - ::encode_xml("S3Tags", tag_filter, f); - } -} - -bool rgw_s3_filter::has_content() const { - return key_filter.has_content() || - metadata_filter.has_content() || - tag_filter.has_content(); -} - -bool match(const rgw_s3_key_filter& filter, const std::string& key) { - const auto key_size = key.size(); - const auto prefix_size = filter.prefix_rule.size(); - if (prefix_size != 0) { - // prefix rule exists - if (prefix_size > key_size) { - // if prefix is longer than key, we fail - return false; - } - if (!std::equal(filter.prefix_rule.begin(), filter.prefix_rule.end(), key.begin())) { - return false; - } - } - const auto suffix_size = filter.suffix_rule.size(); - if (suffix_size != 0) { - // suffix rule exists - if (suffix_size > key_size) { - // if suffix is longer than key, we fail - return false; - } - if (!std::equal(filter.suffix_rule.begin(), filter.suffix_rule.end(), (key.end() - suffix_size))) { - return false; - } - } - if (!filter.regex_rule.empty()) { - // TODO add regex chaching in the filter - const std::regex base_regex(filter.regex_rule); - if (!std::regex_match(key, base_regex)) { - return false; - } - } - return true; -} - -bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv) { - // all filter pairs must exist with the same value in the object's metadata/tags - // object metadata/tags may include items not in the filter - return std::includes(kv.begin(), kv.end(), filter.kv.begin(), filter.kv.end()); -} - -bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv) { - // all filter pairs must exist with the same value in the object's metadata/tags - // object metadata/tags may include items not in the filter - for (auto& filter : filter.kv) { - auto result = kv.equal_range(filter.first); - if (std::any_of(result.first, result.second, [&filter](const std::pair<std::string, std::string>& p) { return p.second == filter.second;})) - continue; - else - return false; - } - return true; -} - bool match(const rgw::notify::EventTypeList& events, rgw::notify::EventType event) { // if event list exists, and none of the events in the list matches the event type, filter the message if (!events.empty() && std::find(events.begin(), events.end(), event) == events.end()) { diff --git a/src/rgw/rgw_pubsub.h b/src/rgw/rgw_pubsub.h index 8a6b290cb85..176ada95204 100644 --- a/src/rgw/rgw_pubsub.h +++ b/src/rgw/rgw_pubsub.h @@ -9,94 +9,10 @@ #include "rgw_zone.h" #include "rgw_notify_event_type.h" #include <boost/container/flat_map.hpp> +#include "rgw_s3_filter.h" class XMLObj; -struct rgw_s3_key_filter { - std::string prefix_rule; - std::string suffix_rule; - std::string regex_rule; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(1, 1, bl); - encode(prefix_rule, bl); - encode(suffix_rule, bl); - encode(regex_rule, bl); - ENCODE_FINISH(bl); - } - - void decode(bufferlist::const_iterator& bl) { - DECODE_START(1, bl); - decode(prefix_rule, bl); - decode(suffix_rule, bl); - decode(regex_rule, bl); - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_key_filter) - -using KeyValueMap = boost::container::flat_map<std::string, std::string>; -using KeyMultiValueMap = std::multimap<std::string, std::string>; - -struct rgw_s3_key_value_filter { - KeyValueMap kv; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(1, 1, bl); - encode(kv, bl); - ENCODE_FINISH(bl); - } - void decode(bufferlist::const_iterator& bl) { - DECODE_START(1, bl); - decode(kv, bl); - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_key_value_filter) - -struct rgw_s3_filter { - rgw_s3_key_filter key_filter; - rgw_s3_key_value_filter metadata_filter; - rgw_s3_key_value_filter tag_filter; - - bool has_content() const; - - void dump(Formatter *f) const; - bool decode_xml(XMLObj *obj); - void dump_xml(Formatter *f) const; - - void encode(bufferlist& bl) const { - ENCODE_START(2, 1, bl); - encode(key_filter, bl); - encode(metadata_filter, bl); - encode(tag_filter, bl); - ENCODE_FINISH(bl); - } - - void decode(bufferlist::const_iterator& bl) { - DECODE_START(2, bl); - decode(key_filter, bl); - decode(metadata_filter, bl); - if (struct_v >= 2) { - decode(tag_filter, bl); - } - DECODE_FINISH(bl); - } -}; -WRITE_CLASS_ENCODER(rgw_s3_filter) - using OptionalFilter = std::optional<rgw_s3_filter>; struct rgw_pubsub_topic_filter; diff --git a/src/rgw/rgw_rest.cc b/src/rgw/rgw_rest.cc index a202d5acf4e..ac5e65c0dd6 100644 --- a/src/rgw/rgw_rest.cc +++ b/src/rgw/rgw_rest.cc @@ -666,8 +666,10 @@ static void build_redirect_url(req_state *s, const string& redirect_base, string dest_uri = dest_uri.substr(0, dest_uri.size() - 1); } dest_uri += s->info.request_uri; - dest_uri += "?"; - dest_uri += s->info.request_params; + if (!s->info.request_params.empty()) { + dest_uri += "?"; + dest_uri += s->info.request_params; + } } void abort_early(req_state *s, RGWOp* op, int err_no, @@ -1467,7 +1469,7 @@ int RGWPutACLs_ObjStore::get_params(optional_yield y) { const auto max_size = s->cct->_conf->rgw_max_put_param_size; std::tie(op_ret, data) = read_all_input(s, max_size, false); - ldpp_dout(s, 0) << "RGWPutACLs_ObjStore::get_params read data is: " << data.c_str() << dendl; + ldpp_dout(s, 20) << "RGWPutACLs_ObjStore::get_params read data is: " << data.c_str() << dendl; return op_ret; } @@ -1668,7 +1670,6 @@ int RGWDeleteMultiObj_ObjStore::get_params(optional_yield y) return op_ret; } - void RGWRESTOp::send_response() { if (!flusher.did_start()) { diff --git a/src/rgw/rgw_rest.h b/src/rgw/rgw_rest.h index 3abba0124a6..9111696453e 100644 --- a/src/rgw/rgw_rest.h +++ b/src/rgw/rgw_rest.h @@ -129,30 +129,39 @@ public: } int get_params(optional_yield y) override; + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; class RGWGetObjTags_ObjStore : public RGWGetObjTags { public: RGWGetObjTags_ObjStore() {}; ~RGWGetObjTags_ObjStore() {}; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); } }; class RGWPutObjTags_ObjStore: public RGWPutObjTags { public: RGWPutObjTags_ObjStore() {}; ~RGWPutObjTags_ObjStore() {}; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_TAGGING", s->info.method); } }; class RGWGetBucketTags_ObjStore : public RGWGetBucketTags { public: RGWGetBucketTags_ObjStore() = default; virtual ~RGWGetBucketTags_ObjStore() = default; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); } }; class RGWPutBucketTags_ObjStore: public RGWPutBucketTags { public: RGWPutBucketTags_ObjStore() = default; virtual ~RGWPutBucketTags_ObjStore() = default; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_TAGGING", s->info.method); } }; class RGWGetBucketReplication_ObjStore : public RGWGetBucketReplication { @@ -177,42 +186,56 @@ class RGWListBuckets_ObjStore : public RGWListBuckets { public: RGWListBuckets_ObjStore() {} ~RGWListBuckets_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKETS", s->info.method); } }; class RGWGetUsage_ObjStore : public RGWGetUsage { public: RGWGetUsage_ObjStore() {} ~RGWGetUsage_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.USER_USAGE", s->info.method); } }; class RGWListBucket_ObjStore : public RGWListBucket { public: RGWListBucket_ObjStore() {} ~RGWListBucket_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); } }; class RGWStatAccount_ObjStore : public RGWStatAccount { public: RGWStatAccount_ObjStore() {} ~RGWStatAccount_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_STATUS", s->info.method); } }; class RGWStatBucket_ObjStore : public RGWStatBucket { public: RGWStatBucket_ObjStore() {} ~RGWStatBucket_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_STATUS", s->info.method); } }; class RGWCreateBucket_ObjStore : public RGWCreateBucket { public: RGWCreateBucket_ObjStore() {} ~RGWCreateBucket_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); } }; class RGWDeleteBucket_ObjStore : public RGWDeleteBucket { public: RGWDeleteBucket_ObjStore() {} ~RGWDeleteBucket_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET", s->info.method); } }; class RGWPutObj_ObjStore : public RGWPutObj @@ -224,6 +247,8 @@ public: int verify_params() override; int get_params(optional_yield y) override; int get_data(bufferlist& bl) override; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; class RGWPostObj_ObjStore : public RGWPostObj @@ -294,6 +319,7 @@ public: ~RGWPostObj_ObjStore() override {} int verify_params() override; + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; @@ -302,6 +328,8 @@ class RGWPutMetadataAccount_ObjStore : public RGWPutMetadataAccount public: RGWPutMetadataAccount_ObjStore() {} ~RGWPutMetadataAccount_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACCOUNT_METADATA", s->info.method); } }; class RGWPutMetadataBucket_ObjStore : public RGWPutMetadataBucket @@ -309,6 +337,8 @@ class RGWPutMetadataBucket_ObjStore : public RGWPutMetadataBucket public: RGWPutMetadataBucket_ObjStore() {} ~RGWPutMetadataBucket_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_METADATA", s->info.method); } }; class RGWPutMetadataObject_ObjStore : public RGWPutMetadataObject @@ -316,18 +346,24 @@ class RGWPutMetadataObject_ObjStore : public RGWPutMetadataObject public: RGWPutMetadataObject_ObjStore() {} ~RGWPutMetadataObject_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT_METADATA", s->info.method); } }; class RGWRestoreObj_ObjStore : public RGWRestoreObj { public: RGWRestoreObj_ObjStore() {} ~RGWRestoreObj_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; class RGWDeleteObj_ObjStore : public RGWDeleteObj { public: RGWDeleteObj_ObjStore() {} ~RGWDeleteObj_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; class RGWGetCrossDomainPolicy_ObjStore : public RGWGetCrossDomainPolicy { @@ -346,12 +382,16 @@ class RGWCopyObj_ObjStore : public RGWCopyObj { public: RGWCopyObj_ObjStore() {} ~RGWCopyObj_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.OBJECT", s->info.method); } }; class RGWGetACLs_ObjStore : public RGWGetACLs { public: RGWGetACLs_ObjStore() {} ~RGWGetACLs_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); } }; class RGWPutACLs_ObjStore : public RGWPutACLs { @@ -360,12 +400,26 @@ public: ~RGWPutACLs_ObjStore() override {} int get_params(optional_yield y) override; + virtual std::string canonical_name() const override { return fmt::format("REST.{}.ACL", s->info.method); } +}; + +class RGWGetObjAttrs_ObjStore : public RGWGetObjAttrs { +public: + RGWGetObjAttrs_ObjStore() {} + ~RGWGetObjAttrs_ObjStore() override {} + + int get_params(optional_yield y) = 0; + /* not actually used */ + int send_response_data_error(optional_yield y) override { return 0; }; + int send_response_data(bufferlist& bl, off_t ofs, off_t len) override { return 0; }; }; class RGWGetLC_ObjStore : public RGWGetLC { public: RGWGetLC_ObjStore() {} ~RGWGetLC_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); } }; class RGWPutLC_ObjStore : public RGWPutLC { @@ -374,6 +428,7 @@ public: ~RGWPutLC_ObjStore() override {} int get_params(optional_yield y) override; + virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); } }; class RGWDeleteLC_ObjStore : public RGWDeleteLC { @@ -381,30 +436,39 @@ public: RGWDeleteLC_ObjStore() {} ~RGWDeleteLC_ObjStore() override {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.LIFECYCLE", s->info.method); } }; class RGWGetCORS_ObjStore : public RGWGetCORS { public: RGWGetCORS_ObjStore() {} ~RGWGetCORS_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); } }; class RGWPutCORS_ObjStore : public RGWPutCORS { public: RGWPutCORS_ObjStore() {} ~RGWPutCORS_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); } }; class RGWDeleteCORS_ObjStore : public RGWDeleteCORS { public: RGWDeleteCORS_ObjStore() {} ~RGWDeleteCORS_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); } }; class RGWOptionsCORS_ObjStore : public RGWOptionsCORS { public: RGWOptionsCORS_ObjStore() {} ~RGWOptionsCORS_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.CORS", s->info.method); } }; class RGWGetBucketEncryption_ObjStore : public RGWGetBucketEncryption { @@ -429,6 +493,8 @@ class RGWInitMultipart_ObjStore : public RGWInitMultipart { public: RGWInitMultipart_ObjStore() {} ~RGWInitMultipart_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); } }; class RGWCompleteMultipart_ObjStore : public RGWCompleteMultipart { @@ -436,6 +502,7 @@ public: RGWCompleteMultipart_ObjStore() {} ~RGWCompleteMultipart_ObjStore() override {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); } int get_params(optional_yield y) override; }; @@ -443,6 +510,8 @@ class RGWAbortMultipart_ObjStore : public RGWAbortMultipart { public: RGWAbortMultipart_ObjStore() {} ~RGWAbortMultipart_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); } }; class RGWListMultipart_ObjStore : public RGWListMultipart { @@ -450,6 +519,7 @@ public: RGWListMultipart_ObjStore() {} ~RGWListMultipart_ObjStore() override {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOAD", s->info.method); } int get_params(optional_yield y) override; }; @@ -458,6 +528,7 @@ public: RGWListBucketMultiparts_ObjStore() {} ~RGWListBucketMultiparts_ObjStore() override {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.UPLOADS", s->info.method); } int get_params(optional_yield y) override; }; @@ -465,12 +536,16 @@ class RGWBulkDelete_ObjStore : public RGWBulkDelete { public: RGWBulkDelete_ObjStore() {} ~RGWBulkDelete_ObjStore() override {} + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_DELETE", s->info.method); } }; class RGWBulkUploadOp_ObjStore : public RGWBulkUploadOp { public: RGWBulkUploadOp_ObjStore() = default; ~RGWBulkUploadOp_ObjStore() = default; + + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BULK_UPLOAD", s->info.method); } }; class RGWDeleteMultiObj_ObjStore : public RGWDeleteMultiObj { @@ -479,6 +554,7 @@ public: ~RGWDeleteMultiObj_ObjStore() override {} int get_params(optional_yield y) override; + virtual std::string canonical_name() const override { return fmt::format("REST.{}.DELETE_MULTI_OBJECT", s->info.method); } }; class RGWInfo_ObjStore : public RGWInfo { diff --git a/src/rgw/rgw_rest_bucket_logging.cc b/src/rgw/rgw_rest_bucket_logging.cc new file mode 100644 index 00000000000..afd79b0a548 --- /dev/null +++ b/src/rgw/rgw_rest_bucket_logging.cc @@ -0,0 +1,369 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab + +#include "common/dout.h" +#include "rgw_op.h" +#include "rgw_rest.h" +#include "rgw_rest_s3.h" +#include "rgw_arn.h" +#include "rgw_auth_s3.h" +#include "rgw_url.h" +#include "rgw_bucket_logging.h" +#include "rgw_rest_bucket_logging.h" + +#define dout_context g_ceph_context +#define dout_subsys ceph_subsys_rgw + +namespace { + int verify_bucket_logging_params(const DoutPrefixProvider* dpp, const req_state* s) { + bool exists; + const auto no_value = s->info.args.get("logging", &exists); + if (!exists) { + ldpp_dout(dpp, 1) << "ERROR: missing required param 'logging'" << dendl; + return -EINVAL; + } + if (no_value.length() > 0) { + ldpp_dout(dpp, 1) << "ERROR: param 'logging' should not have any value" << dendl; + return -EINVAL; + } + if (s->bucket_name.empty()) { + ldpp_dout(dpp, 1) << "ERROR: logging request must be on a bucket" << dendl; + return -EINVAL; + } + return 0; + } +} + +// GET /<bucket name>/?logging +// reply is XML encoded +class RGWGetBucketLoggingOp : public RGWOp { + rgw::bucketlogging::configuration configuration; + +public: + int verify_permission(optional_yield y) override { + auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); + if (has_s3_resource_tag) + rgw_iam_add_buckettags(this, s); + + if (!verify_bucket_permission(this, s, rgw::IAM::s3GetBucketLogging)) { + return -EACCES; + } + + return 0; + } + + void execute(optional_yield y) override { + op_ret = verify_bucket_logging_params(this, s); + if (op_ret < 0) { + return; + } + + const rgw_bucket src_bucket_id(s->bucket_tenant, s->bucket_name); + std::unique_ptr<rgw::sal::Bucket> src_bucket; + op_ret = driver->load_bucket(this, src_bucket_id, + &src_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get bucket '" << src_bucket_id << "', ret = " << op_ret << dendl; + return; + } + if (auto iter = src_bucket->get_attrs().find(RGW_ATTR_BUCKET_LOGGING); iter != src_bucket->get_attrs().end()) { + try { + configuration.enabled = true; + decode(configuration, iter->second); + } catch (buffer::error& err) { + ldpp_dout(this, 1) << "WARNING: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "' for bucket '" << src_bucket_id << "', error: " << err.what() << dendl; + op_ret = -EIO; + return; + } + } else { + ldpp_dout(this, 5) << "WARNING: no logging configuration on bucket '" << src_bucket_id << "'" << dendl; + return; + } + ldpp_dout(this, 20) << "INFO: found logging configuration on bucket '" << src_bucket_id << "'" + << "'. configuration: " << configuration.to_json_str() << dendl; + } + + void send_response() override { + dump_errno(s); + end_header(s, this, to_mime_type(s->format)); + dump_start(s); + + s->formatter->open_object_section_in_ns("BucketLoggingStatus", XMLNS_AWS_S3); + configuration.dump_xml(s->formatter); + s->formatter->close_section(); + rgw_flush_formatter_and_reset(s, s->formatter); + } + const char* name() const override { return "get_bucket_logging"; } + RGWOpType get_type() override { return RGW_OP_GET_BUCKET_LOGGING; } + uint32_t op_mask() override { return RGW_OP_TYPE_READ; } +}; + +// PUT /<bucket name>/?logging +// actual configuration is XML encoded in the body of the message +class RGWPutBucketLoggingOp : public RGWDefaultResponseOp { + int verify_permission(optional_yield y) override { + auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); + if (has_s3_resource_tag) + rgw_iam_add_buckettags(this, s); + + if (!verify_bucket_permission(this, s, rgw::IAM::s3PutBucketLogging)) { + return -EACCES; + } + + return 0; + } + + const char* name() const override { return "put_bucket_logging"; } + RGWOpType get_type() override { return RGW_OP_PUT_BUCKET_LOGGING; } + uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; } + + void execute(optional_yield y) override { + op_ret = verify_bucket_logging_params(this, s); + if (op_ret < 0) { + return; + } + + const auto max_size = s->cct->_conf->rgw_max_put_param_size; + bufferlist data; + std::tie(op_ret, data) = read_all_input(s, max_size, false); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to read XML logging payload, ret = " << op_ret << dendl; + return; + } + if (data.length() == 0) { + ldpp_dout(this, 1) << "ERROR: XML logging payload missing" << dendl; + op_ret = -EINVAL; + return; + } + + RGWXMLDecoder::XMLParser parser; + if (!parser.init()){ + ldpp_dout(this, 1) << "ERROR: failed to initialize XML parser" << dendl; + op_ret = -EINVAL; + return; + } + if (!parser.parse(data.c_str(), data.length(), 1)) { + ldpp_dout(this, 1) << "ERROR: failed to parse XML logging payload" << dendl; + op_ret = -ERR_MALFORMED_XML; + return; + } + rgw::bucketlogging::configuration configuration; + configuration.default_obj_roll_time = get_cct()->_conf->rgw_bucket_logging_obj_roll_time; + try { + RGWXMLDecoder::decode_xml("BucketLoggingStatus", configuration, &parser, true); + } catch (RGWXMLDecoder::err& err) { + ldpp_dout(this, 1) << "ERROR: failed to parse XML logging payload. error: " << err << dendl; + op_ret = -ERR_MALFORMED_XML; + return; + } + + const rgw_bucket src_bucket_id(s->bucket_tenant, s->bucket_name); + std::unique_ptr<rgw::sal::Bucket> src_bucket; + op_ret = driver->load_bucket(this, src_bucket_id, + &src_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get bucket '" << src_bucket_id << "', ret = " << op_ret << dendl; + return; + } + + if (!configuration.enabled) { + op_ret = rgw::bucketlogging::source_bucket_cleanup(this, driver, src_bucket.get(), true, y); + return; + } + + // set logging configuration + rgw_bucket target_bucket_id; + if (op_ret = rgw::bucketlogging::get_bucket_id(configuration.target_bucket, s->bucket_tenant, target_bucket_id); op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to parse target bucket '" << configuration.target_bucket << "', ret = " << op_ret << dendl; + return; + } + + if (target_bucket_id == src_bucket_id) { + ldpp_dout(this, 1) << "ERROR: target bucket '" << target_bucket_id << "' must be different from source bucket" << dendl; + op_ret = -EINVAL; + return; + } + std::unique_ptr<rgw::sal::Bucket> target_bucket; + op_ret = driver->load_bucket(this, target_bucket_id, + &target_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get target bucket '" << target_bucket_id << "', ret = " << op_ret << dendl; + return; + } + auto& target_attrs = target_bucket->get_attrs(); + if (target_attrs.find(RGW_ATTR_BUCKET_LOGGING) != target_attrs.end()) { + // target bucket must not have logging set on it + ldpp_dout(this, 1) << "ERROR: logging target bucket '" << target_bucket_id << "', is configured with bucket logging" << dendl; + op_ret = -EINVAL; + return; + } + // verify target bucket does not have encryption + if (target_attrs.find(RGW_ATTR_BUCKET_ENCRYPTION_POLICY) != target_attrs.end()) { + ldpp_dout(this, 1) << "ERROR: logging target bucket '" << target_bucket_id << "', is configured with encryption" << dendl; + op_ret = -EINVAL; + return; + } + std::optional<rgw::bucketlogging::configuration> old_conf; + bufferlist conf_bl; + encode(configuration, conf_bl); + op_ret = retry_raced_bucket_write(this, src_bucket.get(), [this, &conf_bl, &src_bucket, &old_conf, &configuration, y] { + auto& attrs = src_bucket->get_attrs(); + auto it = attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (it != attrs.end()) { + try { + rgw::bucketlogging::configuration tmp_conf; + tmp_conf.enabled = true; + decode(tmp_conf, it->second); + old_conf = std::move(tmp_conf); + } catch (buffer::error& err) { + ldpp_dout(this, 1) << "WARNING: failed to decode existing logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "' for bucket '" << src_bucket->get_info().bucket << "', error: " << err.what() << dendl; + } + if (!old_conf || (old_conf && *old_conf != configuration)) { + // conf changed (or was unknown) - update + it->second = conf_bl; + return src_bucket->merge_and_store_attrs(this, attrs, y); + } + // nothing to update + return 0; + } + // conf was added + attrs.insert(std::make_pair(RGW_ATTR_BUCKET_LOGGING, conf_bl)); + return src_bucket->merge_and_store_attrs(this, attrs, y); + }, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to set logging attribute '" << RGW_ATTR_BUCKET_LOGGING << "' to bucket '" << + src_bucket_id << "', ret = " << op_ret << dendl; + return; + } + if (!old_conf) { + ldpp_dout(this, 20) << "INFO: new logging configuration added to bucket '" << src_bucket_id << "'. configuration: " << + configuration.to_json_str() << dendl; + if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, target_bucket, src_bucket_id, true, y); ret < 0) { + ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of target bucket '" << + target_bucket_id << "', ret = " << ret << dendl; + } + } else if (*old_conf != configuration) { + // conf changed - do cleanup + if (const auto ret = commit_logging_object(*old_conf, target_bucket, this, y); ret < 0) { + ldpp_dout(this, 1) << "WARNING: could not commit pending logging object when updating logging configuration of bucket '" << + src_bucket->get_info().bucket << "', ret = " << ret << dendl; + } else { + ldpp_dout(this, 20) << "INFO: committed pending logging object when updating logging configuration of bucket '" << + src_bucket->get_info().bucket << "'" << dendl; + } + if (old_conf->target_bucket != configuration.target_bucket) { + rgw_bucket old_target_bucket_id; + if (const auto ret = rgw::bucketlogging::get_bucket_id(old_conf->target_bucket, s->bucket_tenant, old_target_bucket_id); ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to parse target bucket '" << old_conf->target_bucket << "', ret = " << ret << dendl; + return; + } + if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, driver, old_target_bucket_id, src_bucket_id, false, y); ret < 0) { + ldpp_dout(this, 1) << "WARNING: failed to remove source bucket '" << src_bucket_id << "' from logging sources of original target bucket '" << + old_target_bucket_id << "', ret = " << ret << dendl; + } + if (const auto ret = rgw::bucketlogging::update_bucket_logging_sources(this, target_bucket, src_bucket_id, true, y); ret < 0) { + ldpp_dout(this, 1) << "WARNING: failed to add source bucket '" << src_bucket_id << "' to logging sources of target bucket '" << + target_bucket_id << "', ret = " << ret << dendl; + } + } + ldpp_dout(this, 20) << "INFO: wrote logging configuration to bucket '" << src_bucket_id << "'. configuration: " << + configuration.to_json_str() << dendl; + } else { + ldpp_dout(this, 20) << "INFO: logging configuration of bucket '" << src_bucket_id << "' did not change" << dendl; + } + } +}; + +// Post /<bucket name>/?logging +class RGWPostBucketLoggingOp : public RGWDefaultResponseOp { + int verify_permission(optional_yield y) override { + auto [has_s3_existing_tag, has_s3_resource_tag] = rgw_check_policy_condition(this, s, false); + if (has_s3_resource_tag) + rgw_iam_add_buckettags(this, s); + + if (!verify_bucket_permission(this, s, rgw::IAM::s3PostBucketLogging)) { + return -EACCES; + } + + return 0; + } + + const char* name() const override { return "post_bucket_logging"; } + RGWOpType get_type() override { return RGW_OP_POST_BUCKET_LOGGING; } + uint32_t op_mask() override { return RGW_OP_TYPE_WRITE; } + + void execute(optional_yield y) override { + op_ret = verify_bucket_logging_params(this, s); + if (op_ret < 0) { + return; + } + + const rgw_bucket src_bucket_id(s->bucket_tenant, s->bucket_name); + std::unique_ptr<rgw::sal::Bucket> src_bucket; + op_ret = driver->load_bucket(this, src_bucket_id, + &src_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get bucket '" << src_bucket_id << "', ret = " << op_ret << dendl; + return; + } + const auto& bucket_attrs = src_bucket->get_attrs(); + auto iter = bucket_attrs.find(RGW_ATTR_BUCKET_LOGGING); + if (iter == bucket_attrs.end()) { + ldpp_dout(this, 1) << "WARNING: no logging configured on bucket '" << src_bucket_id << "'" << dendl; + return; + } + rgw::bucketlogging::configuration configuration; + try { + configuration.enabled = true; + decode(configuration, iter->second); + } catch (buffer::error& err) { + ldpp_dout(this, 1) << "WARNING: failed to decode logging attribute '" << RGW_ATTR_BUCKET_LOGGING + << "' for bucket '" << src_bucket_id << "', error: " << err.what() << dendl; + op_ret = -EINVAL; + return; + } + + rgw_bucket target_bucket_id; + if (op_ret = rgw::bucketlogging::get_bucket_id(configuration.target_bucket, s->bucket_tenant, target_bucket_id); op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to parse target bucket '" << configuration.target_bucket << "', ret = " << op_ret << dendl; + return; + } + std::unique_ptr<rgw::sal::Bucket> target_bucket; + op_ret = driver->load_bucket(this, target_bucket_id, + &target_bucket, y); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get target bucket '" << target_bucket_id << "', ret = " << op_ret << dendl; + return; + } + std::string obj_name; + RGWObjVersionTracker objv_tracker; + op_ret = target_bucket->get_logging_object_name(obj_name, configuration.target_prefix, null_yield, this, &objv_tracker); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to get pending logging object name from target bucket '" << target_bucket_id << "'" << dendl; + return; + } + op_ret = rgw::bucketlogging::rollover_logging_object(configuration, target_bucket, obj_name, this, null_yield, true, &objv_tracker); + if (op_ret < 0) { + ldpp_dout(this, 1) << "ERROR: failed to flush pending logging object '" << obj_name + << "' to target bucket '" << target_bucket_id << "'" << dendl; + return; + } + ldpp_dout(this, 20) << "INFO: flushed pending logging object '" << obj_name + << "' to target bucket '" << configuration.target_bucket << "'" << dendl; + } +}; + +RGWOp* RGWHandler_REST_BucketLogging_S3::create_post_op() { + return new RGWPostBucketLoggingOp(); +} + +RGWOp* RGWHandler_REST_BucketLogging_S3::create_put_op() { + return new RGWPutBucketLoggingOp(); +} + +RGWOp* RGWHandler_REST_BucketLogging_S3::create_get_op() { + return new RGWGetBucketLoggingOp(); +} + diff --git a/src/rgw/rgw_rest_bucket_logging.h b/src/rgw/rgw_rest_bucket_logging.h new file mode 100644 index 00000000000..0b31d88dad8 --- /dev/null +++ b/src/rgw/rgw_rest_bucket_logging.h @@ -0,0 +1,19 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab +#pragma once + +#include "rgw_rest_s3.h" + +// s3 compliant bucket logging handler factory +class RGWHandler_REST_BucketLogging_S3 : public RGWHandler_REST_S3 { +protected: + int init_permissions(RGWOp* op, optional_yield y) override {return 0;} + int read_permissions(RGWOp* op, optional_yield y) override {return 0;} + bool supports_quota() override {return false;} +public: + virtual ~RGWHandler_REST_BucketLogging_S3() = default; + static RGWOp* create_get_op(); + static RGWOp* create_put_op(); + static RGWOp* create_post_op(); +}; + diff --git a/src/rgw/rgw_rest_pubsub.cc b/src/rgw/rgw_rest_pubsub.cc index adfc86d87cb..f1ffe09cf25 100644 --- a/src/rgw/rgw_rest_pubsub.cc +++ b/src/rgw/rgw_rest_pubsub.cc @@ -234,7 +234,13 @@ bool verify_topic_permission(const DoutPrefixProvider* dpp, req_state* s, return verify_topic_permission(dpp, s, topic.owner, arn, policy, op); } -// command (AWS compliant): +bool should_forward_request_to_master(req_state* s, rgw::sal::Driver* driver) { + return (!driver->is_meta_master() && + rgw::all_zonegroups_support(*s->penv.site, + rgw::zone_features::notification_v2)); +} + +// command (AWS compliant): // POST // Action=CreateTopic&Name=<topic-name>[&OpaqueData=data][&push-endpoint=<endpoint>[&persistent][&<arg1>=<value1>]] class RGWPSCreateTopicOp : public RGWOp { @@ -273,7 +279,7 @@ class RGWPSCreateTopicOp : public RGWOp { // Remove the args that are parsed, so the push_endpoint_args only contains // necessary one's which is parsed after this if. but only if master zone, // else we do not remove as request is forwarded to master. - if (driver->is_meta_master()) { + if (!should_forward_request_to_master(s, driver)) { s->info.args.remove("OpaqueData"); s->info.args.remove("push-endpoint"); s->info.args.remove("persistent"); @@ -396,7 +402,7 @@ class RGWPSCreateTopicOp : public RGWOp { void RGWPSCreateTopicOp::execute(optional_yield y) { // master request will replicate the topic creation. - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); if (op_ret < 0) { @@ -863,7 +869,7 @@ class RGWPSSetTopicAttributesOp : public RGWOp { }; void RGWPSSetTopicAttributesOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); if (op_ret < 0) { @@ -1008,9 +1014,10 @@ class RGWPSDeleteTopicOp : public RGWOp { }; void RGWPSDeleteTopicOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &bl_post_body, nullptr, s->info, y); + if (op_ret < 0) { ldpp_dout(this, 1) << "DeleteTopic forward_request_to_master returned ret = " << op_ret @@ -1260,7 +1267,7 @@ int RGWPSCreateNotifOp::verify_permission(optional_yield y) { } void RGWPSCreateNotifOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &data, nullptr, s->info, y); if (op_ret < 0) { @@ -1462,7 +1469,7 @@ int RGWPSDeleteNotifOp::verify_permission(optional_yield y) { } void RGWPSDeleteNotifOp::execute(optional_yield y) { - if (!driver->is_meta_master()) { + if (should_forward_request_to_master(s, driver)) { bufferlist indata; op_ret = rgw_forward_request_to_master( this, *s->penv.site, s->owner.id, &indata, nullptr, s->info, y); diff --git a/src/rgw/rgw_rest_s3.cc b/src/rgw/rgw_rest_s3.cc index 68223405cf4..885991244a6 100644 --- a/src/rgw/rgw_rest_s3.cc +++ b/src/rgw/rgw_rest_s3.cc @@ -9,6 +9,7 @@ #include <string_view> #include "common/ceph_crypto.h" +#include "common/dout.h" #include "common/split.h" #include "common/Formatter.h" #include "common/utf8.h" @@ -69,6 +70,7 @@ #include "rgw_role.h" #include "rgw_rest_sts.h" #include "rgw_rest_iam.h" +#include "rgw_rest_bucket_logging.h" #include "rgw_sts.h" #include "rgw_sal_rados.h" #include "rgw_cksum_pipe.h" @@ -449,8 +451,7 @@ int RGWGetObj_ObjStore_S3::send_response_data(bufferlist& bl, off_t bl_ofs, dump_content_length(s, total_len); dump_last_modified(s, lastmod); dump_header_if_nonempty(s, "x-amz-version-id", version_id); - dump_header_if_nonempty(s, "x-amz-expiration", expires); - + dump_header_if_nonempty(s, "x-amz-expiration", expires); if (attrs.find(RGW_ATTR_APPEND_PART_NUM) != attrs.end()) { dump_header(s, "x-rgw-object-type", "Appendable"); dump_header(s, "x-rgw-next-append-position", s->obj_size); @@ -526,7 +527,29 @@ int RGWGetObj_ObjStore_S3::send_response_data(bufferlist& bl, off_t bl_ofs, auto iter = bl.cbegin(); decode(rt, iter); + rgw::sal::RGWRestoreStatus restore_status; + attr_iter = attrs.find(RGW_ATTR_RESTORE_STATUS); + if (attr_iter != attrs.end()) { + bufferlist bl = attr_iter->second; + auto iter = bl.cbegin(); + decode(restore_status, iter); + } + + //restore status + if (restore_status == rgw::sal::RGWRestoreStatus::RestoreAlreadyInProgress) { + dump_header(s, "x-amz-restore", "ongoing-request=\"true\""); + } if (rt == rgw::sal::RGWRestoreType::Temporary) { + auto expire_iter = attrs.find(RGW_ATTR_RESTORE_EXPIRY_DATE); + ceph::real_time expiration_date; + + if (expire_iter != attrs.end()) { + bufferlist bl = expire_iter->second; + auto iter = bl.cbegin(); + decode(expiration_date, iter); + } + //restore status + dump_header_if_nonempty(s, "x-amz-restore", "ongoing-request=\"false\", expiry-date=\""+ dump_time_to_str(expiration_date) +"\""); // temporary restore; set storage-class to cloudtier storage class auto c_iter = attrs.find(RGW_ATTR_CLOUDTIER_STORAGE_CLASS); @@ -785,7 +808,6 @@ void RGWGetObjTags_ObjStore_S3::send_response_data(bufferlist& bl) } } - int RGWPutObjTags_ObjStore_S3::get_params(optional_yield y) { RGWXMLParser parser; @@ -2128,16 +2150,6 @@ void RGWListBucket_ObjStore_S3v2::send_response() rgw_flush_formatter_and_reset(s, s->formatter); } -void RGWGetBucketLogging_ObjStore_S3::send_response() -{ - dump_errno(s); - end_header(s, this, to_mime_type(s->format)); - dump_start(s); - - s->formatter->open_object_section_in_ns("BucketLoggingStatus", XMLNS_AWS_S3); - s->formatter->close_section(); - rgw_flush_formatter_and_reset(s, s->formatter); -} void RGWGetBucketLocation_ObjStore_S3::send_response() { @@ -2389,34 +2401,41 @@ void RGWGetBucketWebsite_ObjStore_S3::send_response() rgw_flush_formatter_and_reset(s, s->formatter); } -static void dump_bucket_metadata(req_state *s, rgw::sal::Bucket* bucket, +static void dump_bucket_metadata(req_state *s, RGWStorageStats& stats) { dump_header(s, "X-RGW-Object-Count", static_cast<long long>(stats.num_objects)); dump_header(s, "X-RGW-Bytes-Used", static_cast<long long>(stats.size)); +} - // only bucket's owner is allowed to get the quota settings of the account - if (s->auth.identity->is_owner_of(bucket->get_owner())) { - const auto& user_info = s->user->get_info(); - const auto& bucket_quota = s->bucket->get_info().quota; // bucket quota - dump_header(s, "X-RGW-Quota-Max-Buckets", static_cast<long long>(user_info.max_buckets)); - - if (user_info.quota.user_quota.enabled){ - dump_header(s, "X-RGW-Quota-User-Size", static_cast<long long>(user_info.quota.user_quota.max_size)); - dump_header(s, "X-RGW-Quota-User-Objects", static_cast<long long>(user_info.quota.user_quota.max_objects)); - } +int RGWStatBucket_ObjStore_S3::get_params(optional_yield y) +{ + report_stats = s->info.args.exists("read-stats"); - if (bucket_quota.enabled){ - dump_header(s, "X-RGW-Quota-Bucket-Size", static_cast<long long>(bucket_quota.max_size)); - dump_header(s, "X-RGW-Quota-Bucket-Objects", static_cast<long long>(bucket_quota.max_objects)); - } - } + return 0; } void RGWStatBucket_ObjStore_S3::send_response() { if (op_ret >= 0) { - dump_bucket_metadata(s, bucket.get(), stats); + if (report_stats) { + dump_bucket_metadata(s, stats); + } + // only bucket's owner is allowed to get the quota settings of the account + if (s->auth.identity->is_owner_of(s->bucket->get_owner())) { + const auto& user_info = s->user->get_info(); + const auto& bucket_quota = s->bucket->get_info().quota; // bucket quota + + dump_header(s, "X-RGW-Quota-Max-Buckets", static_cast<long long>(user_info.max_buckets)); + if (user_info.quota.user_quota.enabled) { + dump_header(s, "X-RGW-Quota-User-Size", static_cast<long long>(user_info.quota.user_quota.max_size)); + dump_header(s, "X-RGW-Quota-User-Objects", static_cast<long long>(user_info.quota.user_quota.max_objects)); + } + if (bucket_quota.enabled) { + dump_header(s, "X-RGW-Quota-Bucket-Size", static_cast<long long>(bucket_quota.max_size)); + dump_header(s, "X-RGW-Quota-Bucket-Objects", static_cast<long long>(bucket_quota.max_objects)); + } + } } set_req_state_err(s, op_ret); @@ -2514,6 +2533,10 @@ int RGWCreateBucket_ObjStore_S3::get_params(optional_yield y) if ((op_ret < 0) && (op_ret != -ERR_LENGTH_REQUIRED)) return op_ret; + if (!driver->is_meta_master()) { + in_data.append(data); + } + if (data.length()) { RGWCreateBucketParser parser; @@ -3519,38 +3542,46 @@ int RGWRestoreObj_ObjStore_S3::get_params(optional_yield y) void RGWRestoreObj_ObjStore_S3::send_response() { - if (op_ret < 0) - { - set_req_state_err(s, op_ret); + if (restore_ret < 0) { + set_req_state_err(s, restore_ret); dump_errno(s); end_header(s, this); dump_start(s); return; } - rgw::sal::Attrs attrs = s->object->get_attrs(); - auto attr_iter = attrs.find(RGW_ATTR_RESTORE_STATUS); - rgw::sal::RGWRestoreStatus restore_status; - if (attr_iter != attrs.end()) { - bufferlist bl = attr_iter->second; - auto iter = bl.cbegin(); - decode(restore_status, iter); - } - ldpp_dout(this, 10) << "restore_status=" << restore_status << dendl; - - if (attr_iter == attrs.end() || restore_status != rgw::sal::RGWRestoreStatus::None) { - s->err.http_ret = 202; //Accepted - dump_header(s, "x-amz-restore", rgw_bl_str(restore_status)); - } else if (restore_status != rgw::sal::RGWRestoreStatus::RestoreAlreadyInProgress) { + if (restore_ret == 0) { + s->err.http_ret = 202; // OK + } else if (restore_ret == 1) { s->err.http_ret = 409; // Conflict - dump_header_if_nonempty(s, "x-amz-restore", rgw_bl_str(restore_status)); - } else if (restore_status != rgw::sal::RGWRestoreStatus::CloudRestored) { - s->err.http_ret = 200; // OK - dump_header_if_nonempty(s, "x-amz-restore", rgw_bl_str(restore_status)); - } else { - s->err.http_ret = 202; // Accepted - dump_header_if_nonempty(s, "x-amz-restore", rgw_bl_str(restore_status)); - } + dump_header(s, "x-amz-restore", "on-going-request=\"true\""); + } else if (restore_ret == 2) { + rgw::sal::Attrs attrs; + ceph::real_time expiration_date; + rgw::sal::RGWRestoreType rt; + attrs = s->object->get_attrs(); + auto expire_iter = attrs.find(RGW_ATTR_RESTORE_EXPIRY_DATE); + auto type_iter = attrs.find(RGW_ATTR_RESTORE_TYPE); + + if (expire_iter != attrs.end()) { + bufferlist bl = expire_iter->second; + auto iter = bl.cbegin(); + decode(expiration_date, iter); + } + + if (type_iter != attrs.end()) { + bufferlist bl = type_iter->second; + auto iter = bl.cbegin(); + decode(rt, iter); + } + if (rt == rgw::sal::RGWRestoreType::Temporary) { + s->err.http_ret = 200; // OK + dump_header(s, "x-amz-restore", "ongoing-request=\"false\", expiry-date=\""+ dump_time_to_str(expiration_date) +"\""); + } else { + s->err.http_ret = 200; + dump_header(s, "x-amz-restore", "ongoing-request=\"false\""); + } + } dump_errno(s); end_header(s, this); @@ -3788,6 +3819,196 @@ void RGWPutACLs_ObjStore_S3::send_response() dump_start(s); } +int RGWGetObjAttrs_ObjStore_S3::get_params(optional_yield y) +{ + string err; + auto& env = s->info.env; + version_id = s->info.args.get("versionId"); + + auto hdr = env->get_optional("HTTP_X_AMZ_EXPECTED_BUCKET_OWNER"); + if (hdr) { + expected_bucket_owner = *hdr; + } + + hdr = env->get_optional("HTTP_X_AMZ_MAX_PARTS"); + if (hdr) { + max_parts = strict_strtol(hdr->c_str(), 10, &err); + if (!err.empty()) { + s->err.message = "Invalid value for MaxParts: " + err; + ldpp_dout(s, 10) << "Invalid value for MaxParts " << *hdr << ": " + << err << dendl; + return -ERR_INVALID_PART; + } + max_parts = std::min(*max_parts, 1000); + } + + hdr = env->get_optional("HTTP_X_AMZ_PART_NUMBER_MARKER"); + if (hdr) { + marker = strict_strtol(hdr->c_str(), 10, &err); + if (!err.empty()) { + s->err.message = "Invalid value for PartNumberMarker: " + err; + ldpp_dout(s, 10) << "Invalid value for PartNumberMarker " << *hdr << ": " + << err << dendl; + return -ERR_INVALID_PART; + } + } + + hdr = env->get_optional("HTTP_X_AMZ_OBJECT_ATTRIBUTES"); + if (hdr) { + requested_attributes = recognize_attrs(*hdr); + } + + /* XXX skipping SSE-C params for now */ + + return 0; +} /* RGWGetObjAttrs_ObjStore_S3::get_params(...) */ + +int RGWGetObjAttrs_ObjStore_S3::get_decrypt_filter( + std::unique_ptr<RGWGetObj_Filter> *filter, + RGWGetObj_Filter* cb, bufferlist* manifest_bl) +{ + // we aren't actually decrypting the data, but for objects encrypted with + // SSE-C we do need to verify that required headers are present and valid + // + // in the SSE-KMS and SSE-S3 cases, this unfortunately causes us to fetch + // decryption keys which we don't need :( + std::unique_ptr<BlockCrypt> block_crypt; // ignored + std::map<std::string, std::string> crypt_http_responses; // ignored + return rgw_s3_prepare_decrypt(s, s->yield, attrs, &block_crypt, + crypt_http_responses); +} + +void RGWGetObjAttrs_ObjStore_S3::send_response() +{ + if (op_ret) + set_req_state_err(s, op_ret); + dump_errno(s); + + if (op_ret == 0) { + version_id = s->object->get_instance(); + + // x-amz-delete-marker: DeleteMarker // not sure we can plausibly do this? + dump_last_modified(s, lastmod); + dump_header_if_nonempty(s, "x-amz-version-id", version_id); + // x-amz-request-charged: RequestCharged + } + + end_header(s, this, to_mime_type(s->format)); + dump_start(s); + + if (op_ret == 0) { + s->formatter->open_object_section("GetObjectAttributes"); + if (requested_attributes & as_flag(ReqAttributes::Etag)) { + if (lo_etag.empty()) { + auto iter = attrs.find(RGW_ATTR_ETAG); + if (iter != attrs.end()) { + lo_etag = iter->second.to_str(); + } + } + s->formatter->dump_string("ETag", lo_etag); + } + + if (requested_attributes & as_flag(ReqAttributes::Checksum)) { + s->formatter->open_object_section("Checksum"); + auto iter = attrs.find(RGW_ATTR_CKSUM); + if (iter != attrs.end()) { + try { + rgw::cksum::Cksum cksum; + auto bliter = iter->second.cbegin(); + cksum.decode(bliter); + if (multipart_parts_count && multipart_parts_count > 0) { + s->formatter->dump_string(cksum.element_name(), + fmt::format("{}-{}", cksum.to_armor(), *multipart_parts_count)); + } else { + s->formatter->dump_string(cksum.element_name(), cksum.to_armor()); + } + } catch (buffer::error& err) { + ldpp_dout(this, 0) + << "ERROR: could not decode stored cksum, caught buffer::error" << dendl; + } + } + s->formatter->close_section(); /* Checksum */ + } /* Checksum */ + + if (requested_attributes & as_flag(ReqAttributes::ObjectParts)) { + if (multipart_parts_count && multipart_parts_count > 0) { + + /* XXX the following was needed to see a manifest at list_parts()! */ + op_ret = s->object->load_obj_state(s, s->yield); + if (op_ret < 0) { + ldpp_dout_fmt(this, 0, + "ERROR: {} load_obj_state() failed ret={}", __func__, + op_ret); + } + + ldpp_dout_fmt(this, 16, + "{} attr flags={} parts_count={}", + __func__, requested_attributes, *multipart_parts_count); + + s->formatter->open_object_section("ObjectParts"); + + bool truncated = false; + int next_marker; + + using namespace rgw::sal; + + int ret = + s->object->list_parts( + this, s->cct, + max_parts ? *max_parts : 1000, + marker ? *marker : 0, + &next_marker, &truncated, + [&](const Object::Part& part) -> int { + s->formatter->open_object_section("Part"); + s->formatter->dump_int("PartNumber", part.part_number); + s->formatter->dump_unsigned("Size", part.part_size); + if (part.cksum.type != rgw::cksum::Type::none) { + s->formatter->dump_string(part.cksum.element_name(), part.cksum.to_armor()); + } + s->formatter->close_section(); /* Part */ + return 0; + }, s->yield); + + if (ret < 0) { + ldpp_dout_fmt(this, 0, + "ERROR: {} list-parts failed for {}", + __func__, s->object->get_name()); + } + /* AWS docs disagree on the name of this element */ + s->formatter->dump_int("PartsCount", *multipart_parts_count); + s->formatter->dump_int("TotalPartsCount", *multipart_parts_count); + s->formatter->dump_bool("IsTruncated", truncated); + if (max_parts) { + s->formatter->dump_int("MaxParts", *max_parts); + } + if(truncated) { + s->formatter->dump_int("NextPartNumberMarker", next_marker); + } + if (marker) { + s->formatter->dump_int("PartNumberMarker", *marker); + } + s->formatter->close_section(); + } /* multipart_parts_count positive */ + } /* ObjectParts */ + + if (requested_attributes & as_flag(ReqAttributes::ObjectSize)) { + s->formatter->dump_int("ObjectSize", s->obj_size); + } + + if (requested_attributes & as_flag(ReqAttributes::StorageClass)) { + auto iter = attrs.find(RGW_ATTR_STORAGE_CLASS); + if (iter != attrs.end()) { + s->formatter->dump_string("StorageClass", iter->second.to_str()); + } else { + s->formatter->dump_string("StorageClass", "STANDARD"); + } + } + s->formatter->close_section(); + } /* op_ret == 0 */ + + rgw_flush_formatter_and_reset(s, s->formatter); +} /* RGWGetObjAttrs_ObjStore_S3::send_response */ + void RGWGetLC_ObjStore_S3::execute(optional_yield y) { config.set_ctx(s->cct); @@ -4767,11 +4988,12 @@ RGWOp *RGWHandler_REST_Bucket_S3::get_obj_op(bool get_data) const RGWOp *RGWHandler_REST_Bucket_S3::op_get() { + /* XXX maybe we could replace this with an indexing operation */ if (s->info.args.sub_resource_exists("encryption")) return nullptr; if (s->info.args.sub_resource_exists("logging")) - return new RGWGetBucketLogging_ObjStore_S3; + return RGWHandler_REST_BucketLogging_S3::create_get_op(); if (s->info.args.sub_resource_exists("location")) return new RGWGetBucketLocation_ObjStore_S3; @@ -4835,9 +5057,10 @@ RGWOp *RGWHandler_REST_Bucket_S3::op_head() RGWOp *RGWHandler_REST_Bucket_S3::op_put() { - if (s->info.args.sub_resource_exists("logging") || - s->info.args.sub_resource_exists("encryption")) + if (s->info.args.sub_resource_exists("encryption")) return nullptr; + if (s->info.args.sub_resource_exists("logging")) + return RGWHandler_REST_BucketLogging_S3::create_put_op(); if (s->info.args.sub_resource_exists("versioning")) return new RGWSetBucketVersioning_ObjStore_S3; if (s->info.args.sub_resource_exists("website")) { @@ -4882,8 +5105,7 @@ RGWOp *RGWHandler_REST_Bucket_S3::op_put() RGWOp *RGWHandler_REST_Bucket_S3::op_delete() { - if (s->info.args.sub_resource_exists("logging") || - s->info.args.sub_resource_exists("encryption")) + if (s->info.args.sub_resource_exists("encryption")) return nullptr; if (is_tagging_op()) { @@ -4927,6 +5149,10 @@ RGWOp *RGWHandler_REST_Bucket_S3::op_post() return new RGWDeleteMultiObj_ObjStore_S3; } + if (s->info.args.exists("logging")) { + return RGWHandler_REST_BucketLogging_S3::create_post_op(); + } + if (s->info.args.exists("mdsearch")) { if (!s->cct->_conf->rgw_enable_mdsearch) { return NULL; @@ -4959,6 +5185,8 @@ RGWOp *RGWHandler_REST_Obj_S3::op_get() return new RGWGetObjLayout_ObjStore_S3; } else if (is_tagging_op()) { return new RGWGetObjTags_ObjStore_S3; + } else if (is_attributes_op()) { + return new RGWGetObjAttrs_ObjStore_S3; } else if (is_obj_retention_op()) { return new RGWGetObjRetention_ObjStore_S3; } else if (is_obj_legal_hold_op()) { @@ -6084,6 +6312,9 @@ AWSGeneralAbstractor::get_auth_data_v4(const req_state* const s, case RGW_OP_GET_BUCKET_PUBLIC_ACCESS_BLOCK: case RGW_OP_DELETE_BUCKET_PUBLIC_ACCESS_BLOCK: case RGW_OP_GET_OBJ://s3select its post-method(payload contain the query) , the request is get-object + case RGW_OP_PUT_BUCKET_LOGGING: + case RGW_OP_POST_BUCKET_LOGGING: + case RGW_OP_GET_BUCKET_LOGGING: break; default: ldpp_dout(s, 10) << "ERROR: AWS4 completion for operation: " << s->op_type << ", NOT IMPLEMENTED" << dendl; @@ -6472,7 +6703,7 @@ rgw::auth::s3::LocalEngine::authenticate( if (driver->get_user_by_access_key(dpp, access_key_id, y, &user) < 0) { ldpp_dout(dpp, 5) << "error reading user info, uid=" << access_key_id << " can't authenticate" << dendl; - return result_t::reject(-ERR_INVALID_ACCESS_KEY); + return result_t::deny(-ERR_INVALID_ACCESS_KEY); } //TODO: Uncomment, when we have a migration plan in place. /*else { @@ -6494,14 +6725,14 @@ rgw::auth::s3::LocalEngine::authenticate( const auto iter = user->get_info().access_keys.find(access_key_id); if (iter == std::end(user->get_info().access_keys)) { ldpp_dout(dpp, 0) << "ERROR: access key not encoded in user info" << dendl; - return result_t::reject(-EPERM); + return result_t::deny(-EPERM); } const RGWAccessKey& k = iter->second; /* Ignore signature for HTTP OPTIONS */ if (s->op_type == RGW_OP_OPTIONS_CORS) { auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), k.subuser, std::nullopt, access_key_id); return result_t::grant(std::move(apl), completer_factory(k.key)); } @@ -6518,11 +6749,11 @@ rgw::auth::s3::LocalEngine::authenticate( ldpp_dout(dpp, 15) << "compare=" << compare << dendl; if (compare != 0) { - return result_t::reject(-ERR_SIGNATURE_NO_MATCH); + return result_t::deny(-ERR_SIGNATURE_NO_MATCH); } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), k.subuser, std::nullopt, access_key_id); return result_t::grant(std::move(apl), completer_factory(k.key)); } @@ -6731,7 +6962,7 @@ rgw::auth::s3::STSEngine::authenticate( string subuser; auto apl = local_apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), std::move(policies), + cct, s, std::move(user), std::move(account), std::move(policies), subuser, token.perm_mask, std::string(_access_key_id)); return result_t::grant(std::move(apl), completer_factory(token.secret_access_key)); } diff --git a/src/rgw/rgw_rest_s3.h b/src/rgw/rgw_rest_s3.h index 63909f57036..e8fdc69751c 100644 --- a/src/rgw/rgw_rest_s3.h +++ b/src/rgw/rgw_rest_s3.h @@ -183,14 +183,6 @@ public: void send_versioned_response(); }; -class RGWGetBucketLogging_ObjStore_S3 : public RGWGetBucketLogging { -public: - RGWGetBucketLogging_ObjStore_S3() {} - ~RGWGetBucketLogging_ObjStore_S3() override {} - - void send_response() override; -}; - class RGWGetBucketLocation_ObjStore_S3 : public RGWGetBucketLocation { public: RGWGetBucketLocation_ObjStore_S3() {} @@ -222,6 +214,7 @@ public: ~RGWGetBucketWebsite_ObjStore_S3() override {} void send_response() override; + virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); } }; class RGWSetBucketWebsite_ObjStore_S3 : public RGWSetBucketWebsite { @@ -231,6 +224,7 @@ public: int get_params(optional_yield y) override; void send_response() override; + virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); } }; class RGWDeleteBucketWebsite_ObjStore_S3 : public RGWDeleteBucketWebsite { @@ -239,6 +233,7 @@ public: ~RGWDeleteBucketWebsite_ObjStore_S3() override {} void send_response() override; + virtual std::string canonical_name() const override { return fmt::format("WEBSITE.{}.BUCKET_WEBSITE", s->info.method); } }; class RGWStatBucket_ObjStore_S3 : public RGWStatBucket_ObjStore { @@ -247,6 +242,7 @@ public: ~RGWStatBucket_ObjStore_S3() override {} void send_response() override; + int get_params(optional_yield y) override; }; class RGWCreateBucket_ObjStore_S3 : public RGWCreateBucket_ObjStore { @@ -378,6 +374,18 @@ public: int get_params(optional_yield y) override; }; +class RGWGetObjAttrs_ObjStore_S3 : public RGWGetObjAttrs_ObjStore { +public: + RGWGetObjAttrs_ObjStore_S3() {} + ~RGWGetObjAttrs_ObjStore_S3() override {} + + int get_params(optional_yield y) override; + int get_decrypt_filter(std::unique_ptr<RGWGetObj_Filter>* filter, + RGWGetObj_Filter* cb, + bufferlist* manifest_bl) override; + void send_response() override; +}; + class RGWGetLC_ObjStore_S3 : public RGWGetLC_ObjStore { protected: RGWLifecycleConfiguration_S3 config; @@ -595,6 +603,7 @@ class RGWConfigBucketMetaSearch_ObjStore_S3 : public RGWConfigBucketMetaSearch { public: RGWConfigBucketMetaSearch_ObjStore_S3() {} ~RGWConfigBucketMetaSearch_ObjStore_S3() {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); } int get_params(optional_yield y) override; void send_response() override; @@ -612,6 +621,7 @@ class RGWDelBucketMetaSearch_ObjStore_S3 : public RGWDelBucketMetaSearch { public: RGWDelBucketMetaSearch_ObjStore_S3() {} ~RGWDelBucketMetaSearch_ObjStore_S3() {} + virtual std::string canonical_name() const override { return fmt::format("REST.{}.BUCKET_MDSEARCH", s->info.method); } void send_response() override; }; @@ -703,6 +713,9 @@ protected: bool is_acl_op() const { return s->info.args.exists("acl"); } + bool is_attributes_op() const { + return s->info.args.exists("attributes"); + } bool is_cors_op() const { return s->info.args.exists("cors"); } @@ -761,6 +774,9 @@ protected: bool is_acl_op() const { return s->info.args.exists("acl"); } + bool is_attributes_op() const { + return s->info.args.exists("attributes"); + } bool is_tagging_op() const { return s->info.args.exists("tagging"); } diff --git a/src/rgw/rgw_rest_sts.cc b/src/rgw/rgw_rest_sts.cc index f2bd9429a55..1101da0af3c 100644 --- a/src/rgw/rgw_rest_sts.cc +++ b/src/rgw/rgw_rest_sts.cc @@ -436,6 +436,9 @@ WebTokenEngine::validate_signature(const DoutPrefixProvider* dpp, const jwt::dec .allow_algorithm(jwt::algorithm::ps512{cert}); verifier.verify(decoded); + } else { + ldpp_dout(dpp, 0) << "Unsupported algorithm: " << algorithm << dendl; + throw -EINVAL; } } catch (std::runtime_error& e) { ldpp_dout(dpp, 0) << "Signature validation failed: " << e.what() << dendl; diff --git a/src/rgw/rgw_rest_swift.cc b/src/rgw/rgw_rest_swift.cc index 35c36d1ae1a..b8ff3ca2fe8 100644 --- a/src/rgw/rgw_rest_swift.cc +++ b/src/rgw/rgw_rest_swift.cc @@ -447,7 +447,6 @@ int RGWListBucket_ObjStore_SWIFT::get_params(optional_yield y) } static void dump_container_metadata(req_state *, - const rgw::sal::Bucket*, const std::optional<RGWStorageStats>& stats, const RGWQuotaInfo&, const RGWBucketWebsiteConf&); @@ -458,7 +457,7 @@ void RGWListBucket_ObjStore_SWIFT::send_response() map<string, bool>::iterator pref_iter = common_prefixes.begin(); dump_start(s); - dump_container_metadata(s, s->bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); s->formatter->open_array_section_with_attrs("container", @@ -558,7 +557,6 @@ next: } // RGWListBucket_ObjStore_SWIFT::send_response static void dump_container_metadata(req_state *s, - const rgw::sal::Bucket* bucket, const std::optional<RGWStorageStats>& stats, const RGWQuotaInfo& quota, const RGWBucketWebsiteConf& ws_conf) @@ -683,7 +681,7 @@ void RGWStatBucket_ObjStore_SWIFT::send_response() { if (op_ret >= 0) { op_ret = STATUS_NO_CONTENT; - dump_container_metadata(s, bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); } @@ -2640,7 +2638,7 @@ RGWOp* RGWSwiftWebsiteHandler::get_ws_listing_op() /* Generate the header now. */ set_req_state_err(s, op_ret); dump_errno(s); - dump_container_metadata(s, s->bucket.get(), stats, quota.bucket_quota, + dump_container_metadata(s, stats, quota.bucket_quota, s->bucket->get_info().website_conf); end_header(s, this, "text/html"); if (op_ret < 0) { diff --git a/src/rgw/rgw_rest_swift.h b/src/rgw/rgw_rest_swift.h index eb1c4422e34..ec206a5160f 100644 --- a/src/rgw/rgw_rest_swift.h +++ b/src/rgw/rgw_rest_swift.h @@ -86,6 +86,7 @@ public: RGWStatBucket_ObjStore_SWIFT() {} ~RGWStatBucket_ObjStore_SWIFT() override {} + int get_params(optional_yield y) override { return 0; } void send_response() override; }; diff --git a/src/rgw/rgw_s3_filter.cc b/src/rgw/rgw_s3_filter.cc new file mode 100644 index 00000000000..05a7c4a7293 --- /dev/null +++ b/src/rgw/rgw_s3_filter.cc @@ -0,0 +1,269 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#include "rgw_pubsub.h" +#include "rgw_tools.h" +#include "rgw_xml.h" +#include "rgw_s3_filter.h" +#include "common/errno.h" +#include "rgw_sal.h" +#include <regex> +#include <algorithm> + +void rgw_s3_key_filter::dump(Formatter *f) const { + if (!has_content()) { + return; + } + f->open_array_section("FilterRules"); + if (!prefix_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "prefix", f); + ::encode_json("Value", prefix_rule, f); + f->close_section(); + } + if (!suffix_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "suffix", f); + ::encode_json("Value", suffix_rule, f); + f->close_section(); + } + if (!regex_rule.empty()) { + f->open_object_section(""); + ::encode_json("Name", "regex", f); + ::encode_json("Value", regex_rule, f); + f->close_section(); + } + f->close_section(); +} + +bool rgw_s3_key_filter::decode_xml(XMLObj* obj) { + XMLObjIter iter = obj->find("FilterRule"); + XMLObj *o; + + const auto throw_if_missing = true; + auto prefix_not_set = true; + auto suffix_not_set = true; + auto regex_not_set = true; + std::string name; + + while ((o = iter.get_next())) { + RGWXMLDecoder::decode_xml("Name", name, o, throw_if_missing); + if (name == "prefix" && prefix_not_set) { + prefix_not_set = false; + RGWXMLDecoder::decode_xml("Value", prefix_rule, o, throw_if_missing); + } else if (name == "suffix" && suffix_not_set) { + suffix_not_set = false; + RGWXMLDecoder::decode_xml("Value", suffix_rule, o, throw_if_missing); + } else if (name == "regex" && regex_not_set) { + regex_not_set = false; + RGWXMLDecoder::decode_xml("Value", regex_rule, o, throw_if_missing); + } else { + throw RGWXMLDecoder::err("invalid/duplicate S3Key filter rule name: '" + name + "'"); + } + } + return true; +} + +void rgw_s3_key_filter::dump_xml(Formatter *f) const { + if (!prefix_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "prefix", f); + ::encode_xml("Value", prefix_rule, f); + f->close_section(); + } + if (!suffix_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "suffix", f); + ::encode_xml("Value", suffix_rule, f); + f->close_section(); + } + if (!regex_rule.empty()) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", "regex", f); + ::encode_xml("Value", regex_rule, f); + f->close_section(); + } +} + +bool rgw_s3_key_filter::has_content() const { + return !(prefix_rule.empty() && suffix_rule.empty() && regex_rule.empty()); +} + +void rgw_s3_key_value_filter::dump(Formatter *f) const { + if (!has_content()) { + return; + } + f->open_array_section("FilterRules"); + for (const auto& key_value : kv) { + f->open_object_section(""); + ::encode_json("Name", key_value.first, f); + ::encode_json("Value", key_value.second, f); + f->close_section(); + } + f->close_section(); +} + +bool rgw_s3_key_value_filter::decode_xml(XMLObj* obj) { + kv.clear(); + XMLObjIter iter = obj->find("FilterRule"); + XMLObj *o; + + const auto throw_if_missing = true; + + std::string key; + std::string value; + + while ((o = iter.get_next())) { + RGWXMLDecoder::decode_xml("Name", key, o, throw_if_missing); + RGWXMLDecoder::decode_xml("Value", value, o, throw_if_missing); + kv.emplace(key, value); + } + return true; +} + +void rgw_s3_key_value_filter::dump_xml(Formatter *f) const { + for (const auto& key_value : kv) { + f->open_object_section("FilterRule"); + ::encode_xml("Name", key_value.first, f); + ::encode_xml("Value", key_value.second, f); + f->close_section(); + } +} + +bool rgw_s3_key_value_filter::has_content() const { + return !kv.empty(); +} + +void rgw_s3_filter::dump(Formatter *f) const { + encode_json("S3Key", key_filter, f); + encode_json("S3Metadata", metadata_filter, f); + encode_json("S3Tags", tag_filter, f); +} + +bool rgw_s3_filter::decode_xml(XMLObj* obj) { + RGWXMLDecoder::decode_xml("S3Key", key_filter, obj); + RGWXMLDecoder::decode_xml("S3Metadata", metadata_filter, obj); + RGWXMLDecoder::decode_xml("S3Tags", tag_filter, obj); + return true; +} + +void rgw_s3_filter::dump_xml(Formatter *f) const { + if (key_filter.has_content()) { + ::encode_xml("S3Key", key_filter, f); + } + if (metadata_filter.has_content()) { + ::encode_xml("S3Metadata", metadata_filter, f); + } + if (tag_filter.has_content()) { + ::encode_xml("S3Tags", tag_filter, f); + } +} + +bool rgw_s3_filter::has_content() const { + return key_filter.has_content() || + metadata_filter.has_content() || + tag_filter.has_content(); +} + +bool match(const rgw_s3_key_filter& filter, const std::string& key) { + const auto key_size = key.size(); + const auto prefix_size = filter.prefix_rule.size(); + if (prefix_size != 0) { + // prefix rule exists + if (prefix_size > key_size) { + // if prefix is longer than key, we fail + return false; + } + if (!std::equal(filter.prefix_rule.begin(), filter.prefix_rule.end(), key.begin())) { + return false; + } + } + const auto suffix_size = filter.suffix_rule.size(); + if (suffix_size != 0) { + // suffix rule exists + if (suffix_size > key_size) { + // if suffix is longer than key, we fail + return false; + } + if (!std::equal(filter.suffix_rule.begin(), filter.suffix_rule.end(), (key.end() - suffix_size))) { + return false; + } + } + if (!filter.regex_rule.empty()) { + // TODO add regex caching in the filter + const std::regex base_regex(filter.regex_rule); + if (!std::regex_match(key, base_regex)) { + return false; + } + } + return true; +} + +bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv) { + // all filter pairs must exist with the same value in the object's metadata/tags + // object metadata/tags may include items not in the filter + return std::includes(kv.begin(), kv.end(), filter.kv.begin(), filter.kv.end()); +} + +bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv) { + // all filter pairs must exist with the same value in the object's metadata/tags + // object metadata/tags may include items not in the filter + for (auto& filter : filter.kv) { + auto result = kv.equal_range(filter.first); + if (std::any_of(result.first, result.second, [&filter](const std::pair<std::string, std::string>& p) { return p.second == filter.second;})) + continue; + else + return false; + } + return true; +} + +bool match(const rgw_s3_filter& s3_filter, const rgw::sal::Object* obj) { + if (obj == nullptr) { + return false; + } + + if (match(s3_filter.key_filter, obj->get_name())) { + return true; + } + + const auto &attrs = obj->get_attrs(); + if (!s3_filter.metadata_filter.kv.empty()) { + KeyValueMap attrs_map; + for (auto& attr : attrs) { + if (boost::algorithm::starts_with(attr.first, RGW_ATTR_META_PREFIX)) { + std::string_view key(attr.first); + key.remove_prefix(sizeof(RGW_ATTR_PREFIX)-1); + // we want to pass a null terminated version + // of the bufferlist, hence "to_str().c_str()" + attrs_map.emplace(key, attr.second.to_str().c_str()); + } + } + if (match(s3_filter.metadata_filter, attrs_map)) { + return true; + } + } + + if (!s3_filter.tag_filter.kv.empty()) { + // tag filter exists + // try to fetch tags from the attributes + KeyMultiValueMap tags; + const auto attr_iter = attrs.find(RGW_ATTR_TAGS); + if (attr_iter != attrs.end()) { + auto bliter = attr_iter->second.cbegin(); + RGWObjTags obj_tags; + try { + ::decode(obj_tags, bliter); + } catch (buffer::error &) { + // not able to decode tags + return false; + } + tags = std::move(obj_tags.get_tags()); + } + if (match(s3_filter.tag_filter, tags)) { + return true; + } + } + + return false; +} diff --git a/src/rgw/rgw_s3_filter.h b/src/rgw/rgw_s3_filter.h new file mode 100644 index 00000000000..0273da9a364 --- /dev/null +++ b/src/rgw/rgw_s3_filter.h @@ -0,0 +1,103 @@ +// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- +// vim: ts=8 sw=2 smarttab ft=cpp + +#pragma once + +#include "rgw_tools.h" +#include <boost/container/flat_map.hpp> + +class XMLObj; + +struct rgw_s3_key_filter { + bool operator==(const rgw_s3_key_filter& rhs) const = default; + std::string prefix_rule; + std::string suffix_rule; + std::string regex_rule; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(1, 1, bl); + encode(prefix_rule, bl); + encode(suffix_rule, bl); + encode(regex_rule, bl); + ENCODE_FINISH(bl); + } + + void decode(bufferlist::const_iterator& bl) { + DECODE_START(1, bl); + decode(prefix_rule, bl); + decode(suffix_rule, bl); + decode(regex_rule, bl); + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_key_filter) + +using KeyValueMap = boost::container::flat_map<std::string, std::string>; +using KeyMultiValueMap = std::multimap<std::string, std::string>; + +struct rgw_s3_key_value_filter { + KeyValueMap kv; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(1, 1, bl); + encode(kv, bl); + ENCODE_FINISH(bl); + } + void decode(bufferlist::const_iterator& bl) { + DECODE_START(1, bl); + decode(kv, bl); + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_key_value_filter) + +struct rgw_s3_filter { + rgw_s3_key_filter key_filter; + rgw_s3_key_value_filter metadata_filter; + rgw_s3_key_value_filter tag_filter; + + bool has_content() const; + + void dump(Formatter *f) const; + bool decode_xml(XMLObj *obj); + void dump_xml(Formatter *f) const; + + void encode(bufferlist& bl) const { + ENCODE_START(2, 1, bl); + encode(key_filter, bl); + encode(metadata_filter, bl); + encode(tag_filter, bl); + ENCODE_FINISH(bl); + } + + void decode(bufferlist::const_iterator& bl) { + DECODE_START(2, bl); + decode(key_filter, bl); + decode(metadata_filter, bl); + if (struct_v >= 2) { + decode(tag_filter, bl); + } + DECODE_FINISH(bl); + } +}; +WRITE_CLASS_ENCODER(rgw_s3_filter) + +bool match(const rgw_s3_key_filter& filter, const std::string& key); + +bool match(const rgw_s3_key_value_filter& filter, const KeyValueMap& kv); + +bool match(const rgw_s3_key_value_filter& filter, const KeyMultiValueMap& kv); + +bool match(const rgw_s3_filter& filter, const rgw::sal::Object* obj); diff --git a/src/rgw/rgw_sal.h b/src/rgw/rgw_sal.h index 769d7435442..97e25179fc9 100644 --- a/src/rgw/rgw_sal.h +++ b/src/rgw/rgw_sal.h @@ -15,6 +15,7 @@ #pragma once +#include <cstdint> #include <optional> #include <boost/intrusive_ptr.hpp> #include <boost/smart_ptr/intrusive_ref_counter.hpp> @@ -26,6 +27,7 @@ #include "rgw_notify_event_type.h" #include "rgw_req_context.h" #include "include/random.h" +#include "include/function2.hpp" // FIXME: following subclass dependencies #include "driver/rados/rgw_user.h" @@ -874,7 +876,7 @@ class Bucket { std::string zonegroup_id; rgw_placement_rule placement_rule; // zone placement is optional on buckets created for another zonegroup - const RGWZonePlacementInfo* zone_placement; + const RGWZonePlacementInfo* zone_placement = nullptr; RGWAccessControlPolicy policy; Attrs attrs; bool obj_lock_enabled = false; @@ -1003,6 +1005,31 @@ class Bucket { virtual int remove_topics(RGWObjVersionTracker* objv_tracker, optional_yield y, const DoutPrefixProvider *dpp) = 0; + /** Read the name of the pending bucket logging object name */ + virtual int get_logging_object_name(std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) = 0; + /** Update the name of the pending bucket logging object name */ + virtual int set_logging_object_name(const std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + bool new_obj, + RGWObjVersionTracker* objv_tracker) = 0; + /** Remove the object holding the name of the pending bucket logging object */ + virtual int remove_logging_object_name(const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) = 0; + /** Move the pending bucket logging object into the bucket */ + virtual int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) = 0; + //** Remove the pending bucket logging object */ + virtual int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) = 0; + /** Write a record to the pending bucket logging object */ + virtual int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) = 0; + /* dang - This is temporary, until the API is completed */ virtual rgw_bucket& get_key() = 0; virtual RGWBucketInfo& get_info() = 0; @@ -1151,6 +1178,9 @@ class Object { std::string* version_id, std::string* tag, std::string* etag, void (*progress_cb)(off_t, void *), void* progress_data, const DoutPrefixProvider* dpp, optional_yield y) = 0; + + /** return logging subsystem */ + virtual unsigned get_subsys() { return ceph_subsys_rgw; }; /** Get the ACL for this object */ virtual RGWAccessControlPolicy& get_acl(void) = 0; /** Set the ACL for this object */ @@ -1231,6 +1261,28 @@ class Object { /** Dump driver-specific object layout info in JSON */ virtual int dump_obj_layout(const DoutPrefixProvider *dpp, optional_yield y, Formatter* f) = 0; + /* A transfer data type describing metadata specific to one part of a + * completed multipart upload object, following the GetObjectAttributes + * response syntax for Object::Parts here: + * https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObjectAttributes.html */ + class Part + { + public: + int part_number; + uint32_t part_size; + rgw::cksum::Cksum cksum; + }; /* Part */ + + /* callback function/object used by list_parts */ + using list_parts_each_t = + const fu2::unique_function<int(const Part&) const>; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) = 0; + /** Get the cached attributes for this object */ virtual Attrs& get_attrs(void) = 0; /** Get the (const) cached attributes for this object */ @@ -1429,7 +1481,7 @@ public: virtual int init(const DoutPrefixProvider* dpp, optional_yield y, ACLOwner& owner, rgw_placement_rule& dest_placement, rgw::sal::Attrs& attrs) = 0; /** List all the parts of this upload, filling the parts cache */ virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, - int num_parts, int marker, + int max_parts, int marker, int* next_marker, bool* truncated, optional_yield y, bool assume_unsorted = false) = 0; /** Abort this upload */ @@ -1733,8 +1785,6 @@ class Zone { virtual bool is_writeable() = 0; /** Get the URL for the endpoint for redirecting to this zone */ virtual bool get_redirect_endpoint(std::string* endpoint) = 0; - /** Check to see if the given API is supported in this zone */ - virtual bool has_zonegroup_api(const std::string& api) const = 0; /** Get the current period ID for this zone */ virtual const std::string& get_current_period_id() = 0; /** Get thes system access key for this zone */ diff --git a/src/rgw/rgw_sal_dbstore.cc b/src/rgw/rgw_sal_dbstore.cc index d3af42cf2ec..02fd7a49cda 100644 --- a/src/rgw/rgw_sal_dbstore.cc +++ b/src/rgw/rgw_sal_dbstore.cc @@ -271,7 +271,7 @@ namespace rgw::sal { /* XXX: handle has_instance_obj like in set_bucket_instance_attrs() */ - ret = store->getDB()->update_bucket(dpp, "attrs", info, false, nullptr, &new_attrs, nullptr, &get_info().objv_tracker); + ret = store->getDB()->update_bucket(dpp, "attrs", info, false, nullptr, &attrs, nullptr, &get_info().objv_tracker); return ret; } @@ -458,14 +458,6 @@ namespace rgw::sal { return false; } - bool DBZone::has_zonegroup_api(const std::string& api) const - { - if (api == "default") - return true; - - return false; - } - const std::string& DBZone::get_current_period_id() { return current_period->get_id(); @@ -496,6 +488,14 @@ namespace rgw::sal { return std::make_unique<DBLuaManager>(this); } + int DBObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) + { + return -EOPNOTSUPP; + } + int DBObject::load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh) { RGWObjState* astate; diff --git a/src/rgw/rgw_sal_dbstore.h b/src/rgw/rgw_sal_dbstore.h index 107ba735a63..4df10d1dce1 100644 --- a/src/rgw/rgw_sal_dbstore.h +++ b/src/rgw/rgw_sal_dbstore.h @@ -268,22 +268,22 @@ protected: class DBZone : public StoreZone { protected: DBStore* store; - RGWRealm *realm{nullptr}; - DBZoneGroup *zonegroup{nullptr}; - RGWZone *zone_public_config{nullptr}; /* external zone params, e.g., entrypoints, log flags, etc. */ - RGWZoneParams *zone_params{nullptr}; /* internal zone params, e.g., rados pools */ - RGWPeriod *current_period{nullptr}; + std::unique_ptr<RGWRealm> realm; + std::unique_ptr<DBZoneGroup> zonegroup; + std::unique_ptr<RGWZone> zone_public_config; /* external zone params, e.g., entrypoints, log flags, etc. */ + std::unique_ptr<RGWZoneParams> zone_params; /* internal zone params, e.g., rados pools */ + std::unique_ptr<RGWPeriod> current_period; public: DBZone(DBStore* _store) : store(_store) { - realm = new RGWRealm(); + realm = std::make_unique<RGWRealm>(); std::unique_ptr<RGWZoneGroup> rzg = std::make_unique<RGWZoneGroup>("default", "default"); rzg->api_name = "default"; rzg->is_master = true; - zonegroup = new DBZoneGroup(store, std::move(rzg)); - zone_public_config = new RGWZone(); - zone_params = new RGWZoneParams(); - current_period = new RGWPeriod(); + zonegroup = std::make_unique<DBZoneGroup>(store, std::move(rzg)); + zone_public_config = std::make_unique<RGWZone>(); + zone_params = std::make_unique<RGWZoneParams>(); + current_period = std::make_unique<RGWPeriod>(); // XXX: only default and STANDARD supported for now RGWZonePlacementInfo info; @@ -292,13 +292,7 @@ protected: info.storage_classes = sc; zone_params->placement_pools["default"] = info; } - ~DBZone() { - delete realm; - delete zonegroup; - delete zone_public_config; - delete zone_params; - delete current_period; - } + ~DBZone() = default; virtual std::unique_ptr<Zone> clone() override { return std::make_unique<DBZone>(store); @@ -309,7 +303,6 @@ protected: virtual const std::string& get_name() const override; virtual bool is_writeable() override; virtual bool get_redirect_endpoint(std::string* endpoint) override; - virtual bool has_zonegroup_api(const std::string& api) const override; virtual const std::string& get_current_period_id() override; virtual const RGWAccessKey& get_system_key() override; virtual const std::string& get_realm_name() override; @@ -535,6 +528,7 @@ protected: DBObject(DBObject& _o) = default; + virtual unsigned get_subsys() { return ceph_subsys_rgw_dbstore; }; virtual int delete_object(const DoutPrefixProvider* dpp, optional_yield y, uint32_t flags, @@ -560,6 +554,13 @@ protected: virtual int set_acl(const RGWAccessControlPolicy& acl) override { acls = acl; return 0; } virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, Attrs* delattrs, optional_yield y, uint32_t flags) override; + + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider* dpp, optional_yield y, bool follow_olh = true) override; virtual int get_obj_attrs(optional_yield y, const DoutPrefixProvider* dpp, rgw_obj* target_obj = NULL) override; virtual int modify_obj_attrs(const char* attr_name, bufferlist& attr_val, optional_yield y, const DoutPrefixProvider* dpp) override; diff --git a/src/rgw/rgw_sal_filter.cc b/src/rgw/rgw_sal_filter.cc index 733bfa39ee2..15da580988e 100644 --- a/src/rgw/rgw_sal_filter.cc +++ b/src/rgw/rgw_sal_filter.cc @@ -1046,6 +1046,17 @@ RGWAccessControlPolicy& FilterObject::get_acl() return next->get_acl(); } +int FilterObject::list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) +{ + return next->list_parts(dpp, cct, max_parts, marker, next_marker, + truncated, + sal::Object::list_parts_each_t(each_func), + y); +} + int FilterObject::load_obj_state(const DoutPrefixProvider *dpp, optional_yield y, bool follow_olh) { return next->load_obj_state(dpp, y, follow_olh); diff --git a/src/rgw/rgw_sal_filter.h b/src/rgw/rgw_sal_filter.h index 17b102f7619..b6b6ed42b8f 100644 --- a/src/rgw/rgw_sal_filter.h +++ b/src/rgw/rgw_sal_filter.h @@ -108,9 +108,6 @@ public: virtual bool get_redirect_endpoint(std::string* endpoint) override { return next->get_redirect_endpoint(endpoint); } - virtual bool has_zonegroup_api(const std::string& api) const override { - return next->has_zonegroup_api(api); - } virtual const std::string& get_current_period_id() override { return next->get_current_period_id(); } @@ -669,6 +666,36 @@ public: optional_yield y, const DoutPrefixProvider *dpp) override { return next->remove_topics(objv_tracker, y, dpp); } + int get_logging_object_name(std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override { + return next->get_logging_object_name(obj_name, prefix, y, dpp, objv_tracker); + } + int set_logging_object_name(const std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + bool new_obj, + RGWObjVersionTracker* objv_track) override { + return next->set_logging_object_name(obj_name, prefix, y, dpp, new_obj, objv_track); + } + int remove_logging_object_name(const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override { + return next->remove_logging_object_name(prefix, y, dpp, objv_tracker); + } + int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp)override { + return next->commit_logging_object(obj_name, y, dpp); + } + int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override { + return next->remove_logging_object(obj_name, y, dpp); + } + int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override { + return next->write_logging_object(obj_name, record, y, dpp, async_completion); + } virtual rgw_bucket& get_key() override { return next->get_key(); } virtual RGWBucketInfo& get_info() override { return next->get_info(); } @@ -760,6 +787,12 @@ public: virtual bool empty() const override { return next->empty(); } virtual const std::string &get_name() const override { return next->get_name(); } + /** If multipart, enumerate (a range [marker..marker+[min(max_parts, parts_count-1)] of) parts of the object */ + virtual int list_parts(const DoutPrefixProvider* dpp, CephContext* cct, + int max_parts, int marker, int* next_marker, + bool* truncated, list_parts_each_t each_func, + optional_yield y) override; + virtual int load_obj_state(const DoutPrefixProvider *dpp, optional_yield y, bool follow_olh = true) override; virtual int set_obj_attrs(const DoutPrefixProvider* dpp, Attrs* setattrs, diff --git a/src/rgw/rgw_sal_store.h b/src/rgw/rgw_sal_store.h index 47d031fbfc6..99b90564997 100644 --- a/src/rgw/rgw_sal_store.h +++ b/src/rgw/rgw_sal_store.h @@ -253,6 +253,26 @@ class StoreBucket : public Bucket { optional_yield y, const DoutPrefixProvider *dpp) override {return 0;} int remove_topics(RGWObjVersionTracker* objv_tracker, optional_yield y, const DoutPrefixProvider *dpp) override {return 0;} + int get_logging_object_name(std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override { return 0; } + int set_logging_object_name(const std::string& obj_name, + const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + bool new_obj, + RGWObjVersionTracker* objv_tracker) override { return 0; } + int remove_logging_object_name(const std::string& prefix, + optional_yield y, + const DoutPrefixProvider *dpp, + RGWObjVersionTracker* objv_tracker) override { return 0; } + int commit_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override { return 0; } + int remove_logging_object(const std::string& obj_name, optional_yield y, const DoutPrefixProvider *dpp) override { return 0; } + int write_logging_object(const std::string& obj_name, const std::string& record, optional_yield y, const DoutPrefixProvider *dpp, bool async_completion) override { + return 0; + } friend class BucketList; }; diff --git a/src/rgw/rgw_swift_auth.cc b/src/rgw/rgw_swift_auth.cc index 032b3734bf9..937f74601b3 100644 --- a/src/rgw/rgw_swift_auth.cc +++ b/src/rgw/rgw_swift_auth.cc @@ -522,7 +522,7 @@ ExternalTokenEngine::authenticate(const DoutPrefixProvider* dpp, } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), + cct, s, std::move(user), std::move(account), std::move(policies), extract_swift_subuser(swift_user), std::nullopt, LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); @@ -685,7 +685,7 @@ SignedTokenEngine::authenticate(const DoutPrefixProvider* dpp, } auto apl = apl_factory->create_apl_local( - cct, s, user->get_info(), std::move(account), + cct, s, std::move(user), std::move(account), std::move(policies), extract_swift_subuser(swift_user), std::nullopt, LocalApplier::NO_ACCESS_KEY); return result_t::grant(std::move(apl)); diff --git a/src/rgw/rgw_swift_auth.h b/src/rgw/rgw_swift_auth.h index 9049c54f5ca..c27a24a2619 100644 --- a/src/rgw/rgw_swift_auth.h +++ b/src/rgw/rgw_swift_auth.h @@ -23,8 +23,8 @@ namespace swift { class TempURLApplier : public rgw::auth::LocalApplier { public: TempURLApplier(CephContext* const cct, - const RGWUserInfo& user_info) - : LocalApplier(cct, user_info, std::nullopt, {}, LocalApplier::NO_SUBUSER, + std::unique_ptr<rgw::sal::User> user) + : LocalApplier(cct, std::move(user), std::nullopt, {}, LocalApplier::NO_SUBUSER, std::nullopt, LocalApplier::NO_ACCESS_KEY) {} @@ -155,8 +155,8 @@ public: class SwiftAnonymousApplier : public rgw::auth::LocalApplier { public: SwiftAnonymousApplier(CephContext* const cct, - const RGWUserInfo& user_info) - : LocalApplier(cct, user_info, std::nullopt, {}, LocalApplier::NO_SUBUSER, + std::unique_ptr<rgw::sal::User> user) + : LocalApplier(cct, std::move(user), std::nullopt, {}, LocalApplier::NO_SUBUSER, std::nullopt, LocalApplier::NO_ACCESS_KEY) { } bool is_admin_of(const rgw_owner& o) const {return false;} @@ -238,7 +238,7 @@ class DefaultStrategy : public rgw::auth::Strategy, aplptr_t create_apl_local(CephContext* const cct, const req_state* const s, - const RGWUserInfo& user_info, + std::unique_ptr<rgw::sal::User> user, std::optional<RGWAccountInfo> account, std::vector<IAM::Policy> policies, const std::string& subuser, @@ -247,7 +247,7 @@ class DefaultStrategy : public rgw::auth::Strategy, auto apl = \ rgw::auth::add_3rdparty(driver, rgw_user(s->account_name), rgw::auth::add_sysreq(cct, driver, s, - LocalApplier(cct, user_info, std::move(account), std::move(policies), + LocalApplier(cct, std::move(user), std::move(account), std::move(policies), subuser, perm_mask, access_key_id))); /* TODO(rzarzynski): replace with static_ptr. */ return aplptr_t(new decltype(apl)(std::move(apl))); @@ -259,7 +259,9 @@ class DefaultStrategy : public rgw::auth::Strategy, /* TempURL doesn't need any user account override. It's a Swift-specific * mechanism that requires account name internally, so there is no * business with delegating the responsibility outside. */ - return aplptr_t(new rgw::auth::swift::TempURLApplier(cct, user_info)); + std::unique_ptr<rgw::sal::User> user = s->user->clone(); + user->get_info() = user_info; + return aplptr_t(new rgw::auth::swift::TempURLApplier(cct, std::move(user))); } public: diff --git a/src/rgw/services/svc_zone.cc b/src/rgw/services/svc_zone.cc index 70cf40eb6cb..97d81550058 100644 --- a/src/rgw/services/svc_zone.cc +++ b/src/rgw/services/svc_zone.cc @@ -657,18 +657,6 @@ const string& RGWSI_Zone::get_current_period_id() const return current_period->get_id(); } -bool RGWSI_Zone::has_zonegroup_api(const std::string& api) const -{ - if (!current_period->get_id().empty()) { - const auto& zonegroups_by_api = current_period->get_map().zonegroups_by_api; - if (zonegroups_by_api.find(api) != zonegroups_by_api.end()) - return true; - } else if (zonegroup->api_name == api) { - return true; - } - return false; -} - bool RGWSI_Zone::zone_is_writeable() { return writeable_zone && !get_zone().is_read_only(); @@ -743,8 +731,7 @@ bool RGWSI_Zone::is_meta_master() const bool RGWSI_Zone::need_to_log_metadata() const { - return is_meta_master() && - (zonegroup->zones.size() > 1 || current_period->is_multi_zonegroups_with_zones()); + return is_meta_master() && is_syncing_bucket_meta(); } bool RGWSI_Zone::can_reshard() const @@ -761,33 +748,16 @@ bool RGWSI_Zone::can_reshard() const /** * Check to see if the bucket metadata could be synced - * bucket: the bucket to check * Returns false is the bucket is not synced */ -bool RGWSI_Zone::is_syncing_bucket_meta(const rgw_bucket& bucket) +bool RGWSI_Zone::is_syncing_bucket_meta() const { - /* no current period */ if (current_period->get_id().empty()) { return false; } - /* zonegroup is not master zonegroup */ - if (!zonegroup->is_master_zonegroup()) { - return false; - } - - /* single zonegroup and a single zone */ - if (current_period->is_single_zonegroup() && zonegroup->zones.size() == 1) { - return false; - } - - /* zone is not master */ - if (zonegroup->master_zone != zone_public_config->id) { - return false; - } - - return true; + return zonegroup->zones.size() > 1 || current_period->is_multi_zonegroups_with_zones(); } diff --git a/src/rgw/services/svc_zone.h b/src/rgw/services/svc_zone.h index c4a3a28f0d7..719546eb8db 100644 --- a/src/rgw/services/svc_zone.h +++ b/src/rgw/services/svc_zone.h @@ -96,7 +96,6 @@ public: uint32_t get_zone_short_id() const; const std::string& get_current_period_id() const; - bool has_zonegroup_api(const std::string& api) const; bool zone_is_writeable(); bool zone_syncs_from(const RGWZone& target_zone, const RGWZone& source_zone) const; @@ -146,7 +145,7 @@ public: bool need_to_log_data() const; bool need_to_log_metadata() const; bool can_reshard() const; - bool is_syncing_bucket_meta(const rgw_bucket& bucket); + bool is_syncing_bucket_meta() const; int list_zonegroups(const DoutPrefixProvider *dpp, std::list<std::string>& zonegroups); int list_regions(const DoutPrefixProvider *dpp, std::list<std::string>& regions); diff --git a/src/script/ceph-backport.sh b/src/script/ceph-backport.sh index a56509e3d3a..c216ed32d9b 100755 --- a/src/script/ceph-backport.sh +++ b/src/script/ceph-backport.sh @@ -779,7 +779,7 @@ function maybe_deduce_remote { else assert_fail "bad remote_type ->$remote_type<- in maybe_deduce_remote" fi - remote=$(git remote -v | grep --extended-regexp --ignore-case '(://|@)github.com(/|:|:/)'${url_component}'/ceph(\s|\.|\/)' | head -n1 | cut -f 1) + remote=$(git remote -v | grep --extended-regexp --ignore-case '(://|@)github.com(/|:|:/)'${url_component}'/ceph(\s|\.|\/|-)' | head -n1 | cut -f 1) echo "$remote" } diff --git a/src/script/run-make.sh b/src/script/run-make.sh index 52d43d3a171..23724028fe6 100755 --- a/src/script/run-make.sh +++ b/src/script/run-make.sh @@ -29,6 +29,7 @@ function clean_up_after_myself() { function detect_ceph_dev_pkgs() { local boost_root=/opt/ceph + local cmake_opts="" if test -f $boost_root/include/boost/config.hpp; then cmake_opts+=" -DWITH_SYSTEM_BOOST=ON -DBOOST_ROOT=$boost_root" else diff --git a/src/test/CMakeLists.txt b/src/test/CMakeLists.txt index 0ea0bb29347..82816fb07c8 100644 --- a/src/test/CMakeLists.txt +++ b/src/test/CMakeLists.txt @@ -591,6 +591,7 @@ if(NOT WIN32) ceph_snappy cls_lock ceph_test_objectstore + ceph_test_bluefs ceph_erasure_code_non_regression cython_modules crushtool diff --git a/src/test/ObjectMap/KeyValueDBMemory.cc b/src/test/ObjectMap/KeyValueDBMemory.cc index 234e963397e..cfe25930d6a 100644 --- a/src/test/ObjectMap/KeyValueDBMemory.cc +++ b/src/test/ObjectMap/KeyValueDBMemory.cc @@ -132,12 +132,26 @@ public: return ""; } + string_view key_as_sv() override { + if (valid()) + return (*it).first.second; + else + return ""; + } + pair<string,string> raw_key() override { if (valid()) return (*it).first; else return make_pair("", ""); } + + pair<string_view,string_view> raw_key_as_sv() override { + if (valid()) + return (*it).first; + else + return make_pair("", ""); + } bool raw_key_is_prefixed(const string &prefix) override { return prefix == (*it).first.first; @@ -150,6 +164,13 @@ public: return bufferlist(); } + std::string_view value_as_sv() override { + if (valid()) + return std::string_view{it->second.c_str(), it->second.length()}; + else + return std::string_view(); + } + int status() override { return 0; } diff --git a/src/test/bench_log.cc b/src/test/bench_log.cc index 9e7c02afc41..2408c5dffb6 100644 --- a/src/test/bench_log.cc +++ b/src/test/bench_log.cc @@ -10,6 +10,7 @@ #include "common/config.h" #include "common/ceph_argparse.h" #include "global/global_init.h" +#include "log/Log.h" #define dout_context g_ceph_context diff --git a/src/test/cli/radosgw-admin/help.t b/src/test/cli/radosgw-admin/help.t index 0b937a3f988..c1675d11a80 100644 --- a/src/test/cli/radosgw-admin/help.t +++ b/src/test/cli/radosgw-admin/help.t @@ -43,6 +43,8 @@ bucket sync disable disable bucket sync bucket sync enable enable bucket sync bucket radoslist list rados objects backing bucket's objects + bucket logging flush flush pending log records object of source bucket to the log bucket + bucket logging info get info on bucket logging configuration on source bucket or list of sources in log bucket bi get retrieve bucket index object entries bi put store bucket index object entries bi list list raw bucket index entries @@ -225,6 +227,7 @@ --secret/--secret-key=<key> specify secret key --gen-access-key generate random access key (for S3) --gen-secret generate random secret key + --generate-key create user with or without credentials --key-type=<type> key type, options are: swift, s3 --key-active=<bool> activate or deactivate a key --temp-url-key[-2]=<key> temp url key diff --git a/src/test/cli/rbd/help.t b/src/test/cli/rbd/help.t index 984175a97b9..5f304258358 100644 --- a/src/test/cli/rbd/help.t +++ b/src/test/cli/rbd/help.t @@ -916,7 +916,7 @@ [--group-namespace <group-namespace>] [--group <group>] [--image-pool <image-pool>] [--image-namespace <image-namespace>] - [--image <image>] [--pool <pool>] + [--image <image>] <group-spec> <image-spec> Add an image to a group. @@ -934,7 +934,6 @@ --image-pool arg image pool name --image-namespace arg image namespace name --image arg image name - -p [ --pool ] arg pool name unless overridden rbd help group image list usage: rbd group image list [--format <format>] [--pretty-format] @@ -960,8 +959,7 @@ [--group-namespace <group-namespace>] [--group <group>] [--image-pool <image-pool>] [--image-namespace <image-namespace>] - [--image <image>] [--pool <pool>] - [--image-id <image-id>] + [--image <image>] [--image-id <image-id>] <group-spec> <image-spec> Remove an image from a group. @@ -979,7 +977,6 @@ --image-pool arg image pool name --image-namespace arg image namespace name --image arg image name - -p [ --pool ] arg pool name unless overridden --image-id arg image id rbd help group info diff --git a/src/test/client/TestClient.h b/src/test/client/TestClient.h index bf3b274af60..d4f3364ad5e 100644 --- a/src/test/client/TestClient.h +++ b/src/test/client/TestClient.h @@ -20,6 +20,7 @@ #include "msg/Messenger.h" #include "mon/MonClient.h" #include "osdc/ObjectCacher.h" +#include "osdc/Objecter.h" #include "client/MetaRequest.h" #include "client/Client.h" #include "messages/MClientReclaim.h" diff --git a/src/test/cls_log/test_cls_log.cc b/src/test/cls_log/test_cls_log.cc index f8c1a32494a..91e38844dec 100644 --- a/src/test/cls_log/test_cls_log.cc +++ b/src/test/cls_log/test_cls_log.cc @@ -332,7 +332,6 @@ TEST_F(cls_log, trim_by_marker) utime_t start_time = ceph_clock_now(); generate_log(ioctx, oid, 10, start_time, true); - utime_t zero_time; std::vector<cls_log_entry> log1; { list<cls_log_entry> entries; diff --git a/src/test/cls_rbd/test_cls_rbd.cc b/src/test/cls_rbd/test_cls_rbd.cc index 7eb03cc42f5..4f39a8e1eab 100644 --- a/src/test/cls_rbd/test_cls_rbd.cc +++ b/src/test/cls_rbd/test_cls_rbd.cc @@ -2156,7 +2156,7 @@ TEST_F(TestClsRbd, mirror_image_map) { librados::IoCtx ioctx; ASSERT_EQ(0, _rados.ioctx_create(_pool_name.c_str(), ioctx)); - ioctx.remove(RBD_MIRRORING); + ioctx.remove(RBD_MIRROR_LEADER); std::map<std::string, cls::rbd::MirrorImageMap> image_mapping; ASSERT_EQ(-ENOENT, mirror_image_map_list(&ioctx, "", 0, &image_mapping)); @@ -2177,7 +2177,7 @@ TEST_F(TestClsRbd, mirror_image_map) mirror_image_map_update(&op, global_image_id, mirror_image_map); } - ASSERT_EQ(0, ioctx.operate(RBD_MIRRORING, &op)); + ASSERT_EQ(0, ioctx.operate(RBD_MIRROR_LEADER, &op)); } ASSERT_EQ(0, mirror_image_map_list(&ioctx, "", 1000, &image_mapping)); @@ -2203,7 +2203,7 @@ TEST_F(TestClsRbd, mirror_image_map) librados::ObjectWriteOperation op; mirror_image_map_remove(&op, "1"); mirror_image_map_update(&op, "10", expected_mirror_image_map); - ASSERT_EQ(0, ioctx.operate(RBD_MIRRORING, &op)); + ASSERT_EQ(0, ioctx.operate(RBD_MIRROR_LEADER, &op)); ASSERT_EQ(0, mirror_image_map_list(&ioctx, "0", 1, &image_mapping)); ASSERT_EQ(1U, image_mapping.size()); diff --git a/src/test/cls_rgw/test_cls_rgw_stats.cc b/src/test/cls_rgw/test_cls_rgw_stats.cc index 80fa88fabf5..90bf4bd25b6 100644 --- a/src/test/cls_rgw/test_cls_rgw_stats.cc +++ b/src/test/cls_rgw/test_cls_rgw_stats.cc @@ -10,6 +10,7 @@ #include "common/dout.h" #include "common/errno.h" #include "common/random_string.h" +#include "include/random.h" // for ceph::util::generate_random_number() #include "global/global_context.h" #include "test/librados/test_cxx.h" diff --git a/src/test/common/CMakeLists.txt b/src/test/common/CMakeLists.txt index 33ff38b932d..3c9cf0003aa 100644 --- a/src/test/common/CMakeLists.txt +++ b/src/test/common/CMakeLists.txt @@ -229,6 +229,12 @@ add_executable(unittest_xmlformatter add_ceph_unittest(unittest_xmlformatter) target_link_libraries(unittest_xmlformatter ceph-common) +add_executable(unittest_htmlformatter + test_htmlformatter.cc + ) +add_ceph_unittest(unittest_htmlformatter) +target_link_libraries(unittest_htmlformatter ceph-common) + # unittest_bit_vector add_executable(unittest_bit_vector test_bit_vector.cc diff --git a/src/test/common/test_async_shared_mutex.cc b/src/test/common/test_async_shared_mutex.cc index ed3a55a70ed..ebd9b937ac8 100644 --- a/src/test/common/test_async_shared_mutex.cc +++ b/src/test/common/test_async_shared_mutex.cc @@ -15,6 +15,7 @@ #include "common/async/shared_mutex.h" #include <future> #include <optional> +#include <shared_mutex> // for std::shared_lock #include <boost/asio/bind_executor.hpp> #include <boost/asio/io_context.hpp> #include <gtest/gtest.h> diff --git a/src/test/common/test_htmlformatter.cc b/src/test/common/test_htmlformatter.cc new file mode 100644 index 00000000000..0a8d827b53a --- /dev/null +++ b/src/test/common/test_htmlformatter.cc @@ -0,0 +1,26 @@ +#include "gtest/gtest.h" + +#include "common/HTMLFormatter.h" +#include <sstream> +#include <string> + +using namespace ceph; + +TEST(htmlformatter, dump_format_large_item) +{ + std::stringstream sout; + HTMLFormatter formatter(false); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.flush(sout); + + std::string uri = base_url + "/" + bucket_name + "/" + object_key; + std::string expected_output = "<li>Location: " + uri + "</li>"; + + EXPECT_EQ(expected_output, sout.str()); +}
\ No newline at end of file diff --git a/src/test/common/test_intrusive_lru.cc b/src/test/common/test_intrusive_lru.cc index af8edb8e2bf..1410b73fcaa 100644 --- a/src/test/common/test_intrusive_lru.cc +++ b/src/test/common/test_intrusive_lru.cc @@ -177,9 +177,9 @@ TEST(LRU, clear_range) { auto [live_ref2, existed2] = cache.add(5, 4); ASSERT_FALSE(existed2); - cache.clear_range(0,4); + cache.clear_range(0, 4, [](auto&){}); - // Should not exists (Unreferenced): + // Should not exist { auto [ref, existed] = cache.add(1, 4); ASSERT_FALSE(existed); @@ -192,21 +192,27 @@ TEST(LRU, clear_range) { auto [ref, existed] = cache.add(3, 4); ASSERT_FALSE(existed); } - // Should exist (Still being referenced): { auto [ref, existed] = cache.add(4, 4); - ASSERT_TRUE(existed); + ASSERT_FALSE(existed); } - // Should exists (Still being referenced and wasn't removed) + ASSERT_TRUE(live_ref1->is_invalidated()); + // Should exist, wasn't removed) { auto [ref, existed] = cache.add(5, 4); ASSERT_TRUE(existed); } - // Test out of bound deletion: + ASSERT_FALSE(live_ref2->is_invalidated()); + // Test clear_range with right bound past last entry + cache.clear_range(3, 8, [](auto&){}); + ASSERT_TRUE(live_ref2->is_invalidated()); { - cache.clear_range(3,8); auto [ref, existed] = cache.add(4, 4); - ASSERT_TRUE(existed); + ASSERT_FALSE(existed); + } + { + auto [ref, existed] = cache.add(5, 4); + ASSERT_FALSE(existed); } { auto [ref, existed] = cache.add(3, 4); diff --git a/src/test/common/test_json_formatter.cc b/src/test/common/test_json_formatter.cc index 9cc19b24ad1..d0ddd262c0a 100644 --- a/src/test/common/test_json_formatter.cc +++ b/src/test/common/test_json_formatter.cc @@ -102,3 +102,27 @@ TEST(formatter, dump_inf_or_nan) EXPECT_EQ(parser.find_obj("nan_val")->get_data(), "null"); EXPECT_EQ(parser.find_obj("nan_val_alt")->get_data(), "null"); } + +TEST(formatter, dump_large_item) { + JSONFormatter formatter; + formatter.open_object_section("large_item"); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + std::string full_url = base_url + "/" + bucket_name + "/" + object_key; + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.close_section(); + bufferlist bl; + formatter.flush(bl); + + // std::cout << std::string(bl.c_str(), bl.length()) << std::endl; + + JSONParser parser; + parser.parse(bl.c_str(), bl.length()); + + EXPECT_TRUE(parser.parse(bl.c_str(), bl.length())); + EXPECT_EQ(parser.find_obj("Location")->get_data(), full_url); +} diff --git a/src/test/common/test_tableformatter.cc b/src/test/common/test_tableformatter.cc index b152014a2b5..90de133d315 100644 --- a/src/test/common/test_tableformatter.cc +++ b/src/test/common/test_tableformatter.cc @@ -250,6 +250,23 @@ TEST(tableformatter, multiline_keyval) EXPECT_EQ(cmp, sout.str()); } +TEST(tableformatter, dump_large_item) { + std::stringstream sout; + TableFormatter* formatter = (TableFormatter*) Formatter::create("table-kv"); + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + std::string full_url = base_url + "/" + bucket_name + "/" + object_key; + formatter->dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + formatter->flush(sout); + delete formatter; + + std::string cmp = "key::Location=\"" + full_url + "\" \n"; + EXPECT_EQ(cmp, sout.str()); +} + /* * Local Variables: * compile-command: "cd ../.. ; make -j4 && diff --git a/src/test/common/test_time.cc b/src/test/common/test_time.cc index bc19ba573d1..80af6fad805 100644 --- a/src/test/common/test_time.cc +++ b/src/test/common/test_time.cc @@ -28,6 +28,8 @@ using ceph::real_time; using ceph::real_clock; using ceph::real_time; +using ceph::mono_clock; + using ceph::coarse_real_clock; using ceph::coarse_mono_clock; diff --git a/src/test/common/test_xmlformatter.cc b/src/test/common/test_xmlformatter.cc index 9ac6dde456e..abbe9e4e25e 100644 --- a/src/test/common/test_xmlformatter.cc +++ b/src/test/common/test_xmlformatter.cc @@ -163,3 +163,25 @@ TEST(xmlformatter, pretty_lowercased_underscored) "<string_item>String</string_item>\n\n"; EXPECT_EQ(cmp, sout.str()); } + +TEST(xmlformatter, dump_format_large_item) +{ + std::stringstream sout; + XMLFormatter formatter( + true, // pretty + false, // lowercased + false); // underscored + + std::string base_url("http://example.com"); + std::string bucket_name("bucket"); + std::string object_key(1024, 'a'); + + formatter.dump_format("Location", "%s/%s/%s", base_url.c_str(), bucket_name.c_str(), object_key.c_str()); + + formatter.flush(sout); + + std::string uri = base_url + "/" + bucket_name + "/" + object_key; + std::string expected_output = "<Location>" + uri + "</Location>\n\n"; + + EXPECT_EQ(expected_output, sout.str()); +}
\ No newline at end of file diff --git a/src/test/crimson/seastore/test_block.h b/src/test/crimson/seastore/test_block.h index e1fe8e06f8a..546f357dea0 100644 --- a/src/test/crimson/seastore/test_block.h +++ b/src/test/crimson/seastore/test_block.h @@ -39,8 +39,8 @@ struct test_block_delta_t { inline std::ostream &operator<<( std::ostream &lhs, const test_extent_desc_t &rhs) { - return lhs << "test_extent_desc_t(len=" << rhs.len - << ", checksum=" << rhs.checksum << ")"; + return lhs << "test_extent_desc_t(len=0x" << std::hex << rhs.len + << ", checksum=0x" << rhs.checksum << std::dec << ")"; } struct TestBlock : crimson::os::seastore::LogicalCachedExtent { diff --git a/src/test/crimson/seastore/test_btree_lba_manager.cc b/src/test/crimson/seastore/test_btree_lba_manager.cc index 9988df3a124..7874411e0ff 100644 --- a/src/test/crimson/seastore/test_btree_lba_manager.cc +++ b/src/test/crimson/seastore/test_btree_lba_manager.cc @@ -112,14 +112,22 @@ struct btree_test_base : seastar::future<> submit_transaction(TransactionRef t) { auto record = cache->prepare_record(*t, JOURNAL_SEQ_NULL, JOURNAL_SEQ_NULL); - return journal->submit_record(std::move(record), t->get_handle()).safe_then( - [this, t=std::move(t)](auto submit_result) mutable { - cache->complete_commit( - *t, + return seastar::do_with( + std::move(t), [this, record=std::move(record)](auto& _t) mutable { + auto& t = *_t; + return journal->submit_record( + std::move(record), + t.get_handle(), + t.get_src(), + [this, &t](auto submit_result) { + cache->complete_commit( + t, submit_result.record_block_base, submit_result.write_result.start_seq); - complete_commit(*t); - }).handle_error(crimson::ct_error::assert_all{}); + complete_commit(t); + } + ).handle_error(crimson::ct_error::assert_all{}); + }); } virtual LBAManager::mkfs_ret test_structure_setup(Transaction &t) = 0; @@ -149,7 +157,10 @@ struct btree_test_base : }).safe_then([this] { return seastar::do_with( cache->create_transaction( - Transaction::src_t::MUTATE, "test_set_up_fut", false), + Transaction::src_t::MUTATE, + "test_set_up_fut", + CACHE_HINT_TOUCH, + false), [this](auto &ref_t) { return with_trans_intr(*ref_t, [&](auto &t) { cache->init(); @@ -228,7 +239,10 @@ struct lba_btree_test : btree_test_base { template <typename F> auto lba_btree_update(F &&f) { auto tref = cache->create_transaction( - Transaction::src_t::MUTATE, "test_btree_update", false); + Transaction::src_t::MUTATE, + "test_btree_update", + CACHE_HINT_TOUCH, + false); auto &t = *tref; with_trans_intr( t, @@ -273,7 +287,10 @@ struct lba_btree_test : btree_test_base { template <typename F> auto lba_btree_read(F &&f) { auto t = cache->create_transaction( - Transaction::src_t::READ, "test_btree_read", false); + Transaction::src_t::READ, + "test_btree_read", + CACHE_HINT_TOUCH, + false); return with_trans_intr( *t, [this, f=std::forward<F>(f)](auto &t) mutable { @@ -421,7 +438,10 @@ struct btree_lba_manager_test : btree_test_base { auto create_transaction(bool create_fake_extent=true) { auto t = test_transaction_t{ cache->create_transaction( - Transaction::src_t::MUTATE, "test_mutate_lba", false), + Transaction::src_t::MUTATE, + "test_mutate_lba", + CACHE_HINT_TOUCH, + false), test_lba_mappings }; if (create_fake_extent) { @@ -437,7 +457,10 @@ struct btree_lba_manager_test : btree_test_base { auto create_weak_transaction() { auto t = test_transaction_t{ cache->create_transaction( - Transaction::src_t::READ, "test_read_weak", true), + Transaction::src_t::READ, + "test_read_weak", + CACHE_HINT_TOUCH, + true), test_lba_mappings }; return t; diff --git a/src/test/crimson/seastore/test_cbjournal.cc b/src/test/crimson/seastore/test_cbjournal.cc index d00a0f42729..47a08d68cbb 100644 --- a/src/test/crimson/seastore/test_cbjournal.cc +++ b/src/test/crimson/seastore/test_cbjournal.cc @@ -181,15 +181,20 @@ struct cbjournal_test_t : public seastar_test_suite_t, JournalTrimmer auto submit_record(record_t&& record) { entries.push_back(record); + entry_validator_t& back = entries.back(); OrderingHandle handle = get_dummy_ordering_handle(); - auto [addr, w_result] = cbj->submit_record( - std::move(record), - handle).unsafe_get(); - entries.back().seq = w_result.start_seq; - entries.back().entries = 1; - entries.back().magic = cbj->get_cjs().get_cbj_header().magic; - logger().debug("submit entry to addr {}", entries.back().seq); - return convert_paddr_to_abs_addr(entries.back().seq.offset); + cbj->submit_record( + std::move(record), + handle, + transaction_type_t::MUTATE, + [this, &back](auto locator) { + back.seq = locator.write_result.start_seq; + back.entries = 1; + back.magic = cbj->get_cjs().get_cbj_header().magic; + logger().debug("submit entry to addr {}", back.seq); + } + ).unsafe_get(); + return convert_paddr_to_abs_addr(back.seq.offset); } seastar::future<> tear_down_fut() final { diff --git a/src/test/crimson/seastore/test_seastore_cache.cc b/src/test/crimson/seastore/test_seastore_cache.cc index 6e24f436b98..fa774886139 100644 --- a/src/test/crimson/seastore/test_seastore_cache.cc +++ b/src/test/crimson/seastore/test_seastore_cache.cc @@ -87,7 +87,10 @@ struct cache_test_t : public seastar_test_suite_t { auto get_transaction() { return cache->create_transaction( - Transaction::src_t::MUTATE, "test_cache", false); + Transaction::src_t::MUTATE, + "test_cache", + CACHE_HINT_TOUCH, + false); } template <typename T, typename... Args> diff --git a/src/test/crimson/seastore/test_seastore_journal.cc b/src/test/crimson/seastore/test_seastore_journal.cc index 2eb791b1d46..04a99319b11 100644 --- a/src/test/crimson/seastore/test_seastore_journal.cc +++ b/src/test/crimson/seastore/test_seastore_journal.cc @@ -233,12 +233,17 @@ struct journal_test_t : seastar_test_suite_t, SegmentProvider, JournalTrimmer { auto submit_record(T&&... _record) { auto record{std::forward<T>(_record)...}; records.push_back(record); + record_validator_t& back = records.back(); OrderingHandle handle = get_dummy_ordering_handle(); - auto [addr, _] = journal->submit_record( + journal->submit_record( std::move(record), - handle).unsafe_get(); - records.back().record_final_offset = addr; - return addr; + handle, + transaction_type_t::MUTATE, + [&back](auto locator) { + back.record_final_offset = locator.record_block_base; + } + ).unsafe_get(); + return back.record_final_offset; } extent_t generate_extent(size_t blocks) { diff --git a/src/test/crimson/test_backfill.cc b/src/test/crimson/test_backfill.cc index 7e058c80ed6..e0fc5821d08 100644 --- a/src/test/crimson/test_backfill.cc +++ b/src/test/crimson/test_backfill.cc @@ -119,6 +119,11 @@ class BackfillFixture : public crimson::osd::BackfillState::BackfillListener { events_to_dispatch.emplace_back(event.intrusive_from_this()); } + template <class EventT> + void schedule_event_immediate(const EventT& event) { + events_to_dispatch.emplace_front(event.intrusive_from_this()); + } + // BackfillListener { void request_replica_scan( const pg_shard_t& target, @@ -188,12 +193,11 @@ public: struct PGFacade; void cancel() { - events_to_dispatch.clear(); - schedule_event(crimson::osd::BackfillState::CancelBackfill{}); + schedule_event_immediate(crimson::osd::BackfillState::CancelBackfill{}); } void resume() { - schedule_event(crimson::osd::BackfillState::Triggered{}); + schedule_event_immediate(crimson::osd::BackfillState::Triggered{}); } }; @@ -274,6 +278,9 @@ struct BackfillFixture::PGFacade : public crimson::osd::BackfillState::PGFacade return backfill_source.projected_log; } + std::ostream &print(std::ostream &out) const override { + return out << "FakePGFacade"; + } }; BackfillFixture::BackfillFixture( @@ -452,7 +459,69 @@ TEST(backfill, two_empty_replicas) EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); } -TEST(backfill, cancel_resume) +TEST(backfill, cancel_resume_middle_of_primaryscan) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_replicascan1) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_replicascan2) { const auto reference_store = FakeStore{ { { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, @@ -469,12 +538,43 @@ TEST(backfill, cancel_resume) EXPECT_CALL(cluster_fixture, backfilled); cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.cancel(); cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.resume(); cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_push1) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); @@ -483,7 +583,7 @@ TEST(backfill, cancel_resume) EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); } -TEST(backfill, cancel_resume_middle_of_scan) +TEST(backfill, cancel_resume_middle_of_push2) { const auto reference_store = FakeStore{ { { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, @@ -501,14 +601,46 @@ TEST(backfill, cancel_resume_middle_of_scan) EXPECT_CALL(cluster_fixture, backfilled); cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.cancel(); cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.resume(); cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.next_till_done(); + + EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); +} + +TEST(backfill, cancel_resume_middle_of_push3) +{ + const auto reference_store = FakeStore{ { + { "1:00058bcc:::rbd_data.1018ac3e755.00000000000000d5:head", {10, 234} }, + { "1:00ed7f8e:::rbd_data.1018ac3e755.00000000000000af:head", {10, 196} }, + { "1:01483aea:::rbd_data.1018ac3e755.0000000000000095:head", {10, 169} }, + }}; + auto cluster_fixture = BackfillFixtureBuilder::add_source( + reference_store.objs + ).add_target( + { /* nothing 1 */ } + ).add_target( + { /* nothing 2 */ } + ).get_result(); + + EXPECT_CALL(cluster_fixture, backfilled); + cluster_fixture.next_round2<crimson::osd::BackfillState::PrimaryScanned>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ReplicaScanned>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.cancel(); + cluster_fixture.next_round2<crimson::osd::BackfillState::CancelBackfill>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); cluster_fixture.next_round2<crimson::osd::BackfillState::ObjectPushed>(); + cluster_fixture.resume(); + cluster_fixture.next_round2<crimson::osd::BackfillState::Triggered>(); + cluster_fixture.next_round2<crimson::osd::BackfillState::RequestDone>(); cluster_fixture.next_till_done(); EXPECT_TRUE(cluster_fixture.all_stores_look_like(reference_store)); diff --git a/src/test/crimson/test_messenger_thrash.cc b/src/test/crimson/test_messenger_thrash.cc index 72e3f221755..246613f4446 100644 --- a/src/test/crimson/test_messenger_thrash.cc +++ b/src/test/crimson/test_messenger_thrash.cc @@ -22,6 +22,8 @@ #include "crimson/net/Messenger.h" #include "test/crimson/ctest_utils.h" +#include <boost/random/uniform_int.hpp> + using namespace std::chrono_literals; namespace bpo = boost::program_options; using crimson::common::local_conf; diff --git a/src/test/crimson/test_monc.cc b/src/test/crimson/test_monc.cc index c30098fe87f..7152110d646 100644 --- a/src/test/crimson/test_monc.cc +++ b/src/test/crimson/test_monc.cc @@ -2,6 +2,7 @@ #include "common/ceph_argparse.h" #include "crimson/common/auth_handler.h" #include "crimson/common/config_proxy.h" +#include "crimson/common/perf_counters_collection.h" #include "crimson/mon/MonClient.h" #include "crimson/net/Connection.h" #include "crimson/net/Messenger.h" diff --git a/src/test/fio/fio_ceph_objectstore.cc b/src/test/fio/fio_ceph_objectstore.cc index ade043f0cd1..f5fa9ceca73 100644 --- a/src/test/fio/fio_ceph_objectstore.cc +++ b/src/test/fio/fio_ceph_objectstore.cc @@ -15,6 +15,7 @@ #include "os/ObjectStore.h" #include "global/global_init.h" +#include "common/debug.h" #include "common/errno.h" #include "include/intarith.h" #include "include/stringify.h" @@ -29,6 +30,12 @@ #include "include/ceph_assert.h" // fio.h clobbers our assert.h #include <algorithm> +#if defined(WITH_SEASTAR) && !defined(WITH_ALIEN) +#include "crimson/common/perf_counters_collection.h" +#else +#include "common/perf_counters_collection.h" +#endif + #define dout_context g_ceph_context #define dout_subsys ceph_subsys_ diff --git a/src/test/fio/fio_librgw.cc b/src/test/fio/fio_librgw.cc index bac4ff2daac..89c09647b61 100644 --- a/src/test/fio/fio_librgw.cc +++ b/src/test/fio/fio_librgw.cc @@ -300,8 +300,6 @@ namespace { */ static void fio_librgw_cleanup(struct thread_data *td) { - int r = 0; - dprint(FD_IO, "fio_librgw_cleanup\n"); /* cleanup specific data */ @@ -312,9 +310,9 @@ namespace { data->release_handles(); if (data->bucket_fh) { - r = rgw_fh_rele(data->fs, data->bucket_fh, 0 /* flags */); + rgw_fh_rele(data->fs, data->bucket_fh, 0 /* flags */); } - r = rgw_umount(data->fs, RGW_UMOUNT_FLAG_NONE); + rgw_umount(data->fs, RGW_UMOUNT_FLAG_NONE); librgw_shutdown(data->rgw_h); td->io_ops_data = nullptr; delete data; diff --git a/src/test/immutable_object_cache/test_DomainSocket.cc b/src/test/immutable_object_cache/test_DomainSocket.cc index 31d1b9adc20..98ef1722071 100644 --- a/src/test/immutable_object_cache/test_DomainSocket.cc +++ b/src/test/immutable_object_cache/test_DomainSocket.cc @@ -6,6 +6,7 @@ #include "gtest/gtest.h" #include "include/Context.h" +#include "include/unordered_set.h" #include "global/global_init.h" #include "global/global_context.h" @@ -13,6 +14,7 @@ #include "tools/immutable_object_cache/CacheClient.h" #include "tools/immutable_object_cache/CacheServer.h" +using ceph::unordered_set; using namespace ceph::immutable_obj_cache; class TestCommunication :public ::testing::Test { diff --git a/src/test/libcephfs/test.cc b/src/test/libcephfs/test.cc index 6f10d2bbd4e..8760f594ae6 100644 --- a/src/test/libcephfs/test.cc +++ b/src/test/libcephfs/test.cc @@ -2706,6 +2706,19 @@ TEST(LibCephFS, Statxat) { ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFDIR); ASSERT_EQ(ceph_statxat(cmount, fd, rel_file_name_2, &stx, 0, 0), 0); ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + // test relative to root with empty relpath +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int dir_fd = ceph_openat(cmount, fd, dir_name, O_DIRECTORY | O_RDONLY, 0); + ASSERT_LE(0, dir_fd); + ASSERT_EQ(ceph_statxat(cmount, dir_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFDIR); + ASSERT_EQ(0, ceph_close(cmount, dir_fd)); + int file_fd = ceph_openat(cmount, fd, rel_file_name_2, O_RDONLY, 0); + ASSERT_LE(0, file_fd); + ASSERT_EQ(ceph_statxat(cmount, file_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + ASSERT_EQ(0, ceph_close(cmount, file_fd)); +#endif ASSERT_EQ(0, ceph_close(cmount, fd)); // test relative to dir @@ -2713,6 +2726,14 @@ TEST(LibCephFS, Statxat) { ASSERT_LE(0, fd); ASSERT_EQ(ceph_statxat(cmount, fd, rel_file_name_1, &stx, 0, 0), 0); ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + // test relative to dir with empty relpath +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int rel_file_fd = ceph_openat(cmount, fd, rel_file_name_1, O_RDONLY, 0); + ASSERT_LE(0, rel_file_fd); + ASSERT_EQ(ceph_statxat(cmount, rel_file_fd, "", &stx, 0, AT_EMPTY_PATH), 0); + ASSERT_EQ(stx.stx_mode & S_IFMT, S_IFREG); + ASSERT_EQ(0, ceph_close(cmount, rel_file_fd)); +#endif // delete the dirtree, recreate and verify ASSERT_EQ(0, ceph_unlink(cmount, file_path)); @@ -3265,6 +3286,13 @@ TEST(LibCephFS, Chownat) { // change ownership to nobody -- we assume nobody exists and id is always 65534 ASSERT_EQ(ceph_conf_set(cmount, "client_permissions", "0"), 0); ASSERT_EQ(ceph_chownat(cmount, fd, rel_file_path, 65534, 65534, 0), 0); + // change relative fd ownership with AT_EMPTY_PATH +#if defined(__linux__) && defined(AT_EMPTY_PATH) + int file_fd = ceph_openat(cmount, fd, rel_file_path, O_RDONLY, 0); + ASSERT_LE(0, file_fd); + ASSERT_EQ(ceph_chownat(cmount, file_fd, "", 65534, 65534, AT_EMPTY_PATH), 0); + ceph_close(cmount, file_fd); +#endif ASSERT_EQ(ceph_conf_set(cmount, "client_permissions", "1"), 0); ceph_close(cmount, fd); diff --git a/src/test/librados/aio.cc b/src/test/librados/aio.cc index 68587fe87d1..7fb90bdd38e 100644 --- a/src/test/librados/aio.cc +++ b/src/test/librados/aio.cc @@ -1722,3 +1722,59 @@ TEST(LibRadosAioEC, MultiWrite) { rados_aio_release(my_completion2); rados_aio_release(my_completion3); } + +TEST(LibRadosAio, CancelBeforeSubmit) { + AioTestData test_data; + ASSERT_EQ("", test_data.init()); + + rados_completion_t completion; + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + rados_aio_release(completion); +} + +TEST(LibRadosAio, CancelBeforeComplete) { + AioTestData test_data; + ASSERT_EQ("", test_data.init()); + + // cancellation tests are racy, so retry if completion beats the cancellation + int ret = 0; + int tries = 10; + do { + rados_completion_t completion; + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + char buf[128]; + ASSERT_EQ(0, rados_aio_read(test_data.m_ioctx, "nonexistent", + completion, buf, sizeof(buf), 0)); + + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + { + TestAlarm alarm; + ASSERT_EQ(0, rados_aio_wait_for_complete(completion)); + } + ret = rados_aio_get_return_value(completion); + rados_aio_release(completion); + } while (ret == -ENOENT && --tries); + + ASSERT_EQ(-ECANCELED, ret); +} + +TEST(LibRadosAio, CancelAfterComplete) { + AioTestData test_data; + rados_completion_t completion; + ASSERT_EQ("", test_data.init()); + + ASSERT_EQ(0, rados_aio_create_completion2(nullptr, nullptr, &completion)); + char buf[128]; + ASSERT_EQ(0, rados_aio_read(test_data.m_ioctx, "nonexistent", + completion, buf, sizeof(buf), 0)); + + { + TestAlarm alarm; + ASSERT_EQ(0, rados_aio_wait_for_complete(completion)); + } + ASSERT_EQ(0, rados_aio_cancel(test_data.m_ioctx, completion)); + ASSERT_EQ(-ENOENT, rados_aio_get_return_value(completion)); + rados_aio_release(completion); +} diff --git a/src/test/librados/aio_cxx.cc b/src/test/librados/aio_cxx.cc index 92326e4dbc0..5e35869b5c2 100644 --- a/src/test/librados/aio_cxx.cc +++ b/src/test/librados/aio_cxx.cc @@ -1,5 +1,6 @@ #include <errno.h> #include <fcntl.h> +#include <deque> #include <sstream> #include <string> #include <utility> @@ -2466,3 +2467,92 @@ TEST(LibRadosAio, MultiReads) { ASSERT_EQ(0, memcmp(buf, bl.c_str(), sizeof(buf))); } } + +// cancellation test fixture for global setup/teardown +// parameterized to test both IoCtx::aio_cancel() and AioCompletion::cancel() +class Cancel : public ::testing::TestWithParam<bool> { + static constexpr auto pool_prefix = "ceph_test_rados_api_pp"; + static Rados rados; + static std::string pool_name; + protected: + static IoCtx ioctx; + public: + static void SetUpTestCase() { + pool_name = get_temp_pool_name(pool_prefix); + ASSERT_EQ("", create_one_pool_pp(pool_name, rados)); + ASSERT_EQ(0, rados.ioctx_create(pool_name.c_str(), ioctx)); + } + static void TearDownTestCase() { + destroy_one_pool_pp(pool_name, rados); + } +}; +Rados Cancel::rados; +std::string Cancel::pool_name; +IoCtx Cancel::ioctx; + +TEST_P(Cancel, BeforeSubmit) +{ + const bool use_completion = GetParam(); + + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + if (use_completion) { + ASSERT_EQ(0, c->cancel()); + } else { + ASSERT_EQ(0, ioctx.aio_cancel(c.get())); + } +} + +TEST_P(Cancel, BeforeComplete) +{ + const bool use_completion = GetParam(); + + // cancellation tests are racy, so retry if completion beats the cancellation + int ret = 0; + int tries = 10; + do { + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + ObjectReadOperation op; + op.assert_exists(); + ioctx.aio_operate("nonexistent", c.get(), &op, nullptr); + + if (use_completion) { + EXPECT_EQ(0, c->cancel()); + } else { + EXPECT_EQ(0, ioctx.aio_cancel(c.get())); + } + { + TestAlarm alarm; + ASSERT_EQ(0, c->wait_for_complete()); + } + ret = c->get_return_value(); + } while (ret == -ENOENT && --tries); + + EXPECT_EQ(-ECANCELED, ret); +} + +TEST_P(Cancel, AfterComplete) +{ + const bool use_completion = GetParam(); + + auto c = std::unique_ptr<AioCompletion>{Rados::aio_create_completion()}; + ObjectReadOperation op; + op.assert_exists(); + ioctx.aio_operate("nonexistent", c.get(), &op, nullptr); + { + TestAlarm alarm; + ASSERT_EQ(0, c->wait_for_complete()); + } + if (use_completion) { + EXPECT_EQ(0, c->cancel()); + } else { + EXPECT_EQ(0, ioctx.aio_cancel(c.get())); + } + EXPECT_EQ(-ENOENT, c->get_return_value()); +} + +std::string cancel_test_name(const testing::TestParamInfo<Cancel::ParamType>& info) +{ + return info.param ? "cancel" : "aio_cancel"; +} + +INSTANTIATE_TEST_SUITE_P(LibRadosAio, Cancel, testing::Bool(), cancel_test_name); diff --git a/src/test/librados/asio.cc b/src/test/librados/asio.cc index 01ebb957150..500f36508a7 100644 --- a/src/test/librados/asio.cc +++ b/src/test/librados/asio.cc @@ -21,10 +21,14 @@ #include <boost/range/begin.hpp> #include <boost/range/end.hpp> +#include <boost/asio/bind_cancellation_slot.hpp> +#include <boost/asio/cancellation_signal.hpp> #include <boost/asio/io_context.hpp> #include <boost/asio/spawn.hpp> #include <boost/asio/use_future.hpp> +#include <optional> + #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context @@ -78,6 +82,15 @@ void rethrow(std::exception_ptr eptr) { if (eptr) std::rethrow_exception(eptr); } +auto capture(std::optional<error_code>& out) { + return [&out] (error_code ec, ...) { out = ec; }; +} + +auto capture(boost::asio::cancellation_signal& signal, + std::optional<error_code>& out) { + return boost::asio::bind_cancellation_slot(signal.slot(), capture(out)); +} + TEST_F(AsioRados, AsyncReadCallback) { boost::asio::io_context service; @@ -385,6 +398,130 @@ TEST_F(AsioRados, AsyncWriteOperationYield) service.run(); } +// FIXME: this crashes on windows with: +// Thread 1 received signal SIGILL, Illegal instruction. +#ifndef _WIN32 + +TEST_F(AsioRados, AsyncReadOperationCancelTerminal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectReadOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::terminal); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncReadOperationCancelTotal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectReadOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::total); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncWriteOperationCancelTerminal) +{ + // cancellation tests are racy, so retry if completion beats the cancellation + boost::system::error_code ec; + int tries = 10; + do { + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> result; + + librados::ObjectWriteOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, result)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(result); + + signal.emit(boost::asio::cancellation_type::terminal); + + service.run(); + ASSERT_TRUE(result); + ec = *result; + + signal.emit(boost::asio::cancellation_type::all); // noop + } while (ec == std::errc::no_such_file_or_directory && --tries); + + EXPECT_EQ(ec, boost::asio::error::operation_aborted); +} + +TEST_F(AsioRados, AsyncWriteOperationCancelTotal) +{ + boost::asio::io_context service; + boost::asio::cancellation_signal signal; + std::optional<error_code> ec; + + librados::ObjectWriteOperation op; + op.assert_exists(); + librados::async_operate(service, io, "noexist", &op, 0, nullptr, + capture(signal, ec)); + + service.poll(); + EXPECT_FALSE(service.stopped()); + EXPECT_FALSE(ec); + + // noop, write only supports terminal + signal.emit(boost::asio::cancellation_type::total); + + service.run(); + ASSERT_TRUE(ec); + EXPECT_EQ(ec, std::errc::no_such_file_or_directory); + + signal.emit(boost::asio::cancellation_type::all); // noop +} + +#endif // not _WIN32 + int main(int argc, char **argv) { auto args = argv_to_vec(argc, argv); diff --git a/src/test/librados/misc.cc b/src/test/librados/misc.cc index 6425d3aac02..ced9fb5f2ee 100644 --- a/src/test/librados/misc.cc +++ b/src/test/librados/misc.cc @@ -12,6 +12,8 @@ #include "include/scope_guard.h" #include "include/stringify.h" #include "common/Checksummer.h" +#include "common/Clock.h" // for ceph_clock_now() +#include "common/config_proxy.h" // for class ConfigProxy #include "global/global_context.h" #include "test/librados/test.h" #include "test/librados/TestCase.h" diff --git a/src/test/librados/misc_cxx.cc b/src/test/librados/misc_cxx.cc index 1f8c212beaf..46bda84a638 100644 --- a/src/test/librados/misc_cxx.cc +++ b/src/test/librados/misc_cxx.cc @@ -16,6 +16,7 @@ #include "include/scope_guard.h" #include "include/stringify.h" #include "common/Checksummer.h" +#include "common/config_proxy.h" // for class ConfigProxy #include "mds/mdstypes.h" #include "global/global_context.h" #include "test/librados/testcase_cxx.h" diff --git a/src/test/librados/test_common.cc b/src/test/librados/test_common.cc index 647a9ff4858..e6e6c21bdf1 100644 --- a/src/test/librados/test_common.cc +++ b/src/test/librados/test_common.cc @@ -2,6 +2,7 @@ // vim: ts=8 sw=2 smarttab #include "common/Formatter.h" +#include "include/ceph_assert.h" #include "include/stringify.h" #include "json_spirit/json_spirit.h" #include "test_common.h" diff --git a/src/test/librados_test_stub/TestMemIoCtxImpl.cc b/src/test/librados_test_stub/TestMemIoCtxImpl.cc index 248fd5b8feb..363bc9b62fc 100644 --- a/src/test/librados_test_stub/TestMemIoCtxImpl.cc +++ b/src/test/librados_test_stub/TestMemIoCtxImpl.cc @@ -6,6 +6,7 @@ #include "common/Clock.h" #include "include/err.h" #include <functional> +#include <shared_mutex> // for std::shared_lock #include <boost/algorithm/string/predicate.hpp> #include <errno.h> #include <include/compat.h> diff --git a/src/test/librados_test_stub/TestMemRadosClient.cc b/src/test/librados_test_stub/TestMemRadosClient.cc index 37d45327c30..09cd20c465e 100644 --- a/src/test/librados_test_stub/TestMemRadosClient.cc +++ b/src/test/librados_test_stub/TestMemRadosClient.cc @@ -5,6 +5,7 @@ #include "test/librados_test_stub/TestMemCluster.h" #include "test/librados_test_stub/TestMemIoCtxImpl.h" #include <errno.h> +#include <shared_mutex> // for std::shared_lock #include <sstream> namespace librados { diff --git a/src/test/librbd/fsx.cc b/src/test/librbd/fsx.cc index 4ba00ad1555..27029ed6330 100644 --- a/src/test/librbd/fsx.cc +++ b/src/test/librbd/fsx.cc @@ -60,6 +60,7 @@ #include "include/rados/librados.hpp" #include "include/rbd/librbd.h" #include "include/rbd/librbd.hpp" +#include "include/rbd_types.h" // for RBD_DATA_PREFIX #include "common/Cond.h" #include "common/SubProcess.h" #include "common/safe_io.h" diff --git a/src/test/librbd/io/test_mock_ImageRequest.cc b/src/test/librbd/io/test_mock_ImageRequest.cc index 6ee67fe5f1c..b68009304d5 100644 --- a/src/test/librbd/io/test_mock_ImageRequest.cc +++ b/src/test/librbd/io/test_mock_ImageRequest.cc @@ -10,6 +10,8 @@ #include "librbd/io/ObjectDispatchSpec.h" #include "librbd/io/Utils.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/journal/test_Replay.cc b/src/test/librbd/journal/test_Replay.cc index 9b4580e6472..1fb3f6225ad 100644 --- a/src/test/librbd/journal/test_Replay.cc +++ b/src/test/librbd/journal/test_Replay.cc @@ -22,6 +22,8 @@ #include "librbd/io/ReadResult.h" #include "librbd/journal/Types.h" +#include <shared_mutex> // for std::shared_lock + void register_test_journal_replay() { } diff --git a/src/test/librbd/migration/test_mock_HttpClient.cc b/src/test/librbd/migration/test_mock_HttpClient.cc index f3888755c79..901c4231dd0 100644 --- a/src/test/librbd/migration/test_mock_HttpClient.cc +++ b/src/test/librbd/migration/test_mock_HttpClient.cc @@ -307,7 +307,7 @@ TEST_F(TestMockMigrationHttpClient, OpenCloseHttps) { boost::asio::ssl::context ssl_context{boost::asio::ssl::context::tlsv12}; load_server_certificate(ssl_context); - boost::beast::ssl_stream<boost::beast::tcp_stream> ssl_stream{ + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_stream{ std::move(socket), ssl_context}; C_SaferCond on_ssl_handshake_ctx; @@ -341,7 +341,7 @@ TEST_F(TestMockMigrationHttpClient, OpenHttpsHandshakeFail) { boost::asio::ssl::context ssl_context{boost::asio::ssl::context::tlsv12}; load_server_certificate(ssl_context); - boost::beast::ssl_stream<boost::beast::tcp_stream> ssl_stream{ + boost::asio::ssl::stream<boost::asio::ip::tcp::socket> ssl_stream{ std::move(socket), ssl_context}; C_SaferCond on_ssl_handshake_ctx; diff --git a/src/test/librbd/mock/MockObjectMap.h b/src/test/librbd/mock/MockObjectMap.h index 5e3235cf023..427d064e150 100644 --- a/src/test/librbd/mock/MockObjectMap.h +++ b/src/test/librbd/mock/MockObjectMap.h @@ -8,6 +8,8 @@ #include "librbd/Utils.h" #include "gmock/gmock.h" +#include <boost/optional/optional_io.hpp> + namespace librbd { struct MockObjectMap { diff --git a/src/test/librbd/object_map/test_mock_InvalidateRequest.cc b/src/test/librbd/object_map/test_mock_InvalidateRequest.cc index 5ea40c03d69..eabce9cca5c 100644 --- a/src/test/librbd/object_map/test_mock_InvalidateRequest.cc +++ b/src/test/librbd/object_map/test_mock_InvalidateRequest.cc @@ -10,6 +10,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace object_map { diff --git a/src/test/librbd/object_map/test_mock_SnapshotCreateRequest.cc b/src/test/librbd/object_map/test_mock_SnapshotCreateRequest.cc index 7f77aaf839f..0c78d9fdaf5 100644 --- a/src/test/librbd/object_map/test_mock_SnapshotCreateRequest.cc +++ b/src/test/librbd/object_map/test_mock_SnapshotCreateRequest.cc @@ -12,6 +12,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace object_map { diff --git a/src/test/librbd/object_map/test_mock_SnapshotRemoveRequest.cc b/src/test/librbd/object_map/test_mock_SnapshotRemoveRequest.cc index 20318743d30..8d3cb2a2b81 100644 --- a/src/test/librbd/object_map/test_mock_SnapshotRemoveRequest.cc +++ b/src/test/librbd/object_map/test_mock_SnapshotRemoveRequest.cc @@ -12,6 +12,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace object_map { diff --git a/src/test/librbd/object_map/test_mock_SnapshotRollbackRequest.cc b/src/test/librbd/object_map/test_mock_SnapshotRollbackRequest.cc index 7b89a0996c1..bfae75ce680 100644 --- a/src/test/librbd/object_map/test_mock_SnapshotRollbackRequest.cc +++ b/src/test/librbd/object_map/test_mock_SnapshotRollbackRequest.cc @@ -11,6 +11,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace object_map { diff --git a/src/test/librbd/object_map/test_mock_UpdateRequest.cc b/src/test/librbd/object_map/test_mock_UpdateRequest.cc index c240dec0004..5c4934d4ad5 100644 --- a/src/test/librbd/object_map/test_mock_UpdateRequest.cc +++ b/src/test/librbd/object_map/test_mock_UpdateRequest.cc @@ -14,6 +14,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace object_map { diff --git a/src/test/librbd/operation/test_mock_DisableFeaturesRequest.cc b/src/test/librbd/operation/test_mock_DisableFeaturesRequest.cc index 171ac41a71a..db6b21d024d 100644 --- a/src/test/librbd/operation/test_mock_DisableFeaturesRequest.cc +++ b/src/test/librbd/operation/test_mock_DisableFeaturesRequest.cc @@ -20,6 +20,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/operation/test_mock_EnableFeaturesRequest.cc b/src/test/librbd/operation/test_mock_EnableFeaturesRequest.cc index b7bf7d1781d..615e471d114 100644 --- a/src/test/librbd/operation/test_mock_EnableFeaturesRequest.cc +++ b/src/test/librbd/operation/test_mock_EnableFeaturesRequest.cc @@ -19,6 +19,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/operation/test_mock_Request.cc b/src/test/librbd/operation/test_mock_Request.cc index 5c5e7a37535..f23a3386c7f 100644 --- a/src/test/librbd/operation/test_mock_Request.cc +++ b/src/test/librbd/operation/test_mock_Request.cc @@ -8,6 +8,8 @@ #include "librbd/AsyncRequest.h" #include "librbd/operation/Request.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/operation/test_mock_ResizeRequest.cc b/src/test/librbd/operation/test_mock_ResizeRequest.cc index 552ba5c9756..b80ef20f0a4 100644 --- a/src/test/librbd/operation/test_mock_ResizeRequest.cc +++ b/src/test/librbd/operation/test_mock_ResizeRequest.cc @@ -14,6 +14,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace util { diff --git a/src/test/librbd/operation/test_mock_SnapshotCreateRequest.cc b/src/test/librbd/operation/test_mock_SnapshotCreateRequest.cc index 218fc6b0417..2756a616090 100644 --- a/src/test/librbd/operation/test_mock_SnapshotCreateRequest.cc +++ b/src/test/librbd/operation/test_mock_SnapshotCreateRequest.cc @@ -13,6 +13,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace mirror { namespace snapshot { diff --git a/src/test/librbd/operation/test_mock_SnapshotProtectRequest.cc b/src/test/librbd/operation/test_mock_SnapshotProtectRequest.cc index aa8c1e78dc7..bd488fc511d 100644 --- a/src/test/librbd/operation/test_mock_SnapshotProtectRequest.cc +++ b/src/test/librbd/operation/test_mock_SnapshotProtectRequest.cc @@ -15,6 +15,8 @@ // template definitions #include "librbd/operation/SnapshotProtectRequest.cc" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace operation { diff --git a/src/test/librbd/operation/test_mock_SnapshotRemoveRequest.cc b/src/test/librbd/operation/test_mock_SnapshotRemoveRequest.cc index 4469cb80dde..e55c6e8687c 100644 --- a/src/test/librbd/operation/test_mock_SnapshotRemoveRequest.cc +++ b/src/test/librbd/operation/test_mock_SnapshotRemoveRequest.cc @@ -15,6 +15,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace image { diff --git a/src/test/librbd/operation/test_mock_SnapshotRollbackRequest.cc b/src/test/librbd/operation/test_mock_SnapshotRollbackRequest.cc index 65eac7a6d6c..d11378ec4de 100644 --- a/src/test/librbd/operation/test_mock_SnapshotRollbackRequest.cc +++ b/src/test/librbd/operation/test_mock_SnapshotRollbackRequest.cc @@ -14,6 +14,8 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/operation/test_mock_SnapshotUnprotectRequest.cc b/src/test/librbd/operation/test_mock_SnapshotUnprotectRequest.cc index 26b1be2066a..9b48e0c0009 100644 --- a/src/test/librbd/operation/test_mock_SnapshotUnprotectRequest.cc +++ b/src/test/librbd/operation/test_mock_SnapshotUnprotectRequest.cc @@ -17,6 +17,8 @@ // template definitions #include "librbd/operation/SnapshotUnprotectRequest.cc" +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace operation { diff --git a/src/test/librbd/operation/test_mock_TrimRequest.cc b/src/test/librbd/operation/test_mock_TrimRequest.cc index 1771e741377..aebd2117dae 100644 --- a/src/test/librbd/operation/test_mock_TrimRequest.cc +++ b/src/test/librbd/operation/test_mock_TrimRequest.cc @@ -16,6 +16,8 @@ #include "gtest/gtest.h" #include <boost/variant.hpp> +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/librbd/test_DeepCopy.cc b/src/test/librbd/test_DeepCopy.cc index fcce0c642e2..457f958e68a 100644 --- a/src/test/librbd/test_DeepCopy.cc +++ b/src/test/librbd/test_DeepCopy.cc @@ -11,6 +11,8 @@ #include "librbd/io/ReadResult.h" #include "test/librados/crimson_utils.h" +#include <shared_mutex> // for std::shared_lock + void register_test_deep_copy() { } diff --git a/src/test/librbd/test_ImageWatcher.cc b/src/test/librbd/test_ImageWatcher.cc index 780ce7c0e3a..1adc5b4149d 100644 --- a/src/test/librbd/test_ImageWatcher.cc +++ b/src/test/librbd/test_ImageWatcher.cc @@ -25,6 +25,7 @@ #include <iostream> #include <map> #include <set> +#include <shared_mutex> // for std::shared_lock #include <sstream> #include <vector> diff --git a/src/test/librbd/test_Migration.cc b/src/test/librbd/test_Migration.cc index 8c0f4b61b21..b017fbba7e6 100644 --- a/src/test/librbd/test_Migration.cc +++ b/src/test/librbd/test_Migration.cc @@ -20,6 +20,8 @@ #include "common/Cond.h" #include <boost/scope_exit.hpp> +#include <shared_mutex> // for std::shared_lock + void register_test_migration() { } diff --git a/src/test/librbd/test_ObjectMap.cc b/src/test/librbd/test_ObjectMap.cc index 32d223a1d27..15d5db37cde 100644 --- a/src/test/librbd/test_ObjectMap.cc +++ b/src/test/librbd/test_ObjectMap.cc @@ -17,6 +17,8 @@ #include <boost/accumulators/statistics/stats.hpp> #include <boost/accumulators/statistics/rolling_sum.hpp> +#include <shared_mutex> // for std::shared_lock + void register_test_object_map() { } diff --git a/src/test/librbd/test_fixture.cc b/src/test/librbd/test_fixture.cc index 9ddebec482e..d2a3d469ece 100644 --- a/src/test/librbd/test_fixture.cc +++ b/src/test/librbd/test_fixture.cc @@ -17,6 +17,7 @@ #include "test/librados/test.h" #include "test/librados/test_cxx.h" #include <iostream> +#include <shared_mutex> // for std::shared_lock #include <sstream> #include <stdlib.h> diff --git a/src/test/librbd/test_internal.cc b/src/test/librbd/test_internal.cc index 37930cb26bb..8f6cbb9e807 100644 --- a/src/test/librbd/test_internal.cc +++ b/src/test/librbd/test_internal.cc @@ -27,6 +27,7 @@ #include <boost/scope_exit.hpp> #include <boost/algorithm/string/predicate.hpp> #include <boost/assign/list_of.hpp> +#include <shared_mutex> // for std::shared_lock #include <utility> #include <vector> #include "test/librados/crimson_utils.h" @@ -1570,6 +1571,83 @@ TEST_F(TestInternal, FlattenNoEmptyObjects) rados_ioctx_destroy(d_ioctx); } +TEST_F(TestInternal, FlattenInconsistentObjectMap) +{ + REQUIRE_FEATURE(RBD_FEATURE_LAYERING | RBD_FEATURE_OBJECT_MAP); + REQUIRE(!is_feature_enabled(RBD_FEATURE_STRIPINGV2)); + + librbd::ImageCtx* ictx; + ASSERT_EQ(0, open_image(m_image_name, &ictx)); + + librbd::NoOpProgressContext no_op; + ASSERT_EQ(0, ictx->operations->resize((1 << ictx->order) * 5, true, no_op)); + + bufferlist bl; + bl.append(std::string(256, '1')); + for (int i = 1; i < 5; i++) { + ASSERT_EQ(256, api::Io<>::write(*ictx, (1 << ictx->order) * i, 256, + bufferlist{bl}, 0)); + } + + ASSERT_EQ(0, snap_create(*ictx, "snap")); + ASSERT_EQ(0, snap_protect(*ictx, "snap")); + + uint64_t features; + ASSERT_EQ(0, librbd::get_features(ictx, &features)); + + std::string clone_name = get_temp_image_name(); + int order = ictx->order; + ASSERT_EQ(0, librbd::clone(m_ioctx, m_image_name.c_str(), "snap", m_ioctx, + clone_name.c_str(), features, &order, 0, 0)); + + close_image(ictx); + ASSERT_EQ(0, open_image(clone_name, &ictx)); + + C_SaferCond lock_ctx; + { + std::shared_lock owner_locker{ictx->owner_lock}; + ictx->exclusive_lock->try_acquire_lock(&lock_ctx); + } + ASSERT_EQ(0, lock_ctx.wait()); + ASSERT_TRUE(ictx->exclusive_lock->is_lock_owner()); + + ceph::BitVector<2> inconsistent_object_map; + inconsistent_object_map.resize(5); + inconsistent_object_map[0] = OBJECT_NONEXISTENT; + inconsistent_object_map[1] = OBJECT_NONEXISTENT; + inconsistent_object_map[2] = OBJECT_EXISTS; + inconsistent_object_map[3] = OBJECT_EXISTS_CLEAN; + // OBJECT_PENDING shouldn't happen within parent overlap, but test + // anyway + inconsistent_object_map[4] = OBJECT_PENDING; + + auto object_map = new librbd::ObjectMap<>(*ictx, CEPH_NOSNAP); + C_SaferCond save_ctx; + { + std::shared_lock owner_locker{ictx->owner_lock}; + std::unique_lock image_locker{ictx->image_lock}; + object_map->set_object_map(inconsistent_object_map); + object_map->aio_save(&save_ctx); + } + ASSERT_EQ(0, save_ctx.wait()); + object_map->put(); + + close_image(ictx); + ASSERT_EQ(0, open_image(clone_name, &ictx)); + ASSERT_EQ(0, ictx->operations->flatten(no_op)); + + bufferptr read_ptr(256); + bufferlist read_bl; + read_bl.push_back(read_ptr); + + librbd::io::ReadResult read_result{&read_bl}; + for (int i = 1; i < 5; i++) { + ASSERT_EQ(256, api::Io<>::read(*ictx, (1 << ictx->order) * i, 256, + librbd::io::ReadResult{read_result}, 0)); + EXPECT_TRUE(bl.contents_equal(read_bl)); + } +} + TEST_F(TestInternal, PoolMetadataConfApply) { REQUIRE_FORMAT_V2(); diff --git a/src/test/librbd/test_mock_ExclusiveLock.cc b/src/test/librbd/test_mock_ExclusiveLock.cc index 6feb54ec661..259adab9d0b 100644 --- a/src/test/librbd/test_mock_ExclusiveLock.cc +++ b/src/test/librbd/test_mock_ExclusiveLock.cc @@ -15,6 +15,7 @@ #include "gmock/gmock.h" #include "gtest/gtest.h" #include <list> +#include <shared_mutex> // for std::shared_lock #include <boost/scope_exit.hpp> namespace librbd { diff --git a/src/test/librbd/test_mock_Journal.cc b/src/test/librbd/test_mock_Journal.cc index 589695c50b3..d39f9c75c02 100644 --- a/src/test/librbd/test_mock_Journal.cc +++ b/src/test/librbd/test_mock_Journal.cc @@ -9,6 +9,7 @@ #include "test/librbd/mock/io/MockObjectDispatch.h" #include "common/Cond.h" #include "common/ceph_mutex.h" +#include "common/debug.h" #include "common/WorkQueue.h" #include "cls/journal/cls_journal_types.h" #include "journal/Journaler.h" @@ -28,6 +29,7 @@ #include "gtest/gtest.h" #include <functional> #include <list> +#include <shared_mutex> // for std::shared_lock #include <boost/scope_exit.hpp> #define dout_context g_ceph_context diff --git a/src/test/librbd/test_mock_ObjectMap.cc b/src/test/librbd/test_mock_ObjectMap.cc index 39e29172c58..e2085d093cb 100644 --- a/src/test/librbd/test_mock_ObjectMap.cc +++ b/src/test/librbd/test_mock_ObjectMap.cc @@ -10,6 +10,8 @@ #include "librbd/object_map/UpdateRequest.h" #include <boost/scope_exit.hpp> +#include <shared_mutex> // for std::shared_lock + namespace librbd { namespace { diff --git a/src/test/mon/PGMap.cc b/src/test/mon/PGMap.cc index 43d6de4c783..c8f167a4594 100644 --- a/src/test/mon/PGMap.cc +++ b/src/test/mon/PGMap.cc @@ -14,6 +14,7 @@ #include "mon/PGMap.h" #include "gtest/gtest.h" +#include "common/TextTable.h" #include "include/stringify.h" using namespace std; diff --git a/src/test/neorados/read_operations.cc b/src/test/neorados/read_operations.cc index adf5f34ae5e..d5df84585b8 100644 --- a/src/test/neorados/read_operations.cc +++ b/src/test/neorados/read_operations.cc @@ -15,6 +15,7 @@ #include <initializer_list> #include <memory> #include <string_view> +#include <unordered_set> #include <utility> #include <boost/asio/use_awaitable.hpp> @@ -27,6 +28,7 @@ #include <xxHash/xxhash.h> #include "include/neorados/RADOS.hpp" +#include "include/rbd/features.h" // for RBD_FEATURES_ALL #include "osd/error_code.h" diff --git a/src/test/objectstore/Allocator_bench.cc b/src/test/objectstore/Allocator_bench.cc index 1758d8c338e..0c577f4fe1b 100644 --- a/src/test/objectstore/Allocator_bench.cc +++ b/src/test/objectstore/Allocator_bench.cc @@ -14,6 +14,7 @@ #include "include/Context.h" #include "os/bluestore/Allocator.h" +#include <boost/random/mersenne_twister.hpp> #include <boost/random/uniform_int.hpp> typedef boost::mt11213b gen_type; diff --git a/src/test/objectstore/Allocator_test.cc b/src/test/objectstore/Allocator_test.cc index 47d29e8590a..1a66303add3 100644 --- a/src/test/objectstore/Allocator_test.cc +++ b/src/test/objectstore/Allocator_test.cc @@ -5,6 +5,7 @@ * Author: Ramesh Chander, Ramesh.Chander@sandisk.com */ #include <iostream> +#include <boost/random/mersenne_twister.hpp> // for boost::mt11213b #include <boost/scoped_ptr.hpp> #include <gtest/gtest.h> diff --git a/src/test/objectstore/CMakeLists.txt b/src/test/objectstore/CMakeLists.txt index bddff3f6727..08388640043 100644 --- a/src/test/objectstore/CMakeLists.txt +++ b/src/test/objectstore/CMakeLists.txt @@ -48,6 +48,18 @@ add_executable(unittest_rocksdb_option add_ceph_unittest(unittest_rocksdb_option) target_link_libraries(unittest_rocksdb_option global os ${BLKID_LIBRARIES}) +# ceph_test_bluefs (a clone of unittest_bluefs) +add_executable(ceph_test_bluefs + test_bluefs.cc + ) +target_link_libraries(ceph_test_bluefs + os + global + ${UNITTEST_LIBS} + ) +install(TARGETS ceph_test_bluefs + DESTINATION ${CMAKE_INSTALL_BINDIR}) + if(WITH_EVENTTRACE) add_dependencies(os eventtrace_tp) endif() diff --git a/src/test/objectstore/Fragmentation_simulator.cc b/src/test/objectstore/Fragmentation_simulator.cc index 02a2991cd0c..1dbaaa8e58c 100644 --- a/src/test/objectstore/Fragmentation_simulator.cc +++ b/src/test/objectstore/Fragmentation_simulator.cc @@ -7,6 +7,7 @@ #include "common/ceph_argparse.h" #include "common/ceph_mutex.h" #include "common/common_init.h" +#include "common/debug.h" #include "common/hobject.h" #include "global/global_context.h" @@ -18,6 +19,7 @@ #include "os/ObjectStore.h" #include "test/objectstore/ObjectStoreImitator.h" #include <fstream> +#include <boost/random/mersenne_twister.hpp> // for boost::mt11213b #include <boost/random/uniform_int.hpp> #include <fmt/core.h> #include <mutex> diff --git a/src/test/objectstore/ObjectStoreImitator.cc b/src/test/objectstore/ObjectStoreImitator.cc index 6b4e7d9eaa8..14cc20c1622 100644 --- a/src/test/objectstore/ObjectStoreImitator.cc +++ b/src/test/objectstore/ObjectStoreImitator.cc @@ -6,6 +6,7 @@ */ #include "test/objectstore/ObjectStoreImitator.h" #include "common/Clock.h" +#include "common/debug.h" #include "common/Finisher.h" #include "common/errno.h" #include "include/ceph_assert.h" @@ -13,6 +14,7 @@ #include "os/bluestore/bluestore_types.h" #include <algorithm> #include <cmath> +#include <shared_mutex> // for std::shared_lock #define dout_context g_ceph_context #define dout_subsys ceph_subsys_test diff --git a/src/test/objectstore/ObjectStoreImitator.h b/src/test/objectstore/ObjectStoreImitator.h index d71d7f2fe58..875f9041b83 100644 --- a/src/test/objectstore/ObjectStoreImitator.h +++ b/src/test/objectstore/ObjectStoreImitator.h @@ -347,6 +347,16 @@ public: ) override { return {}; } + + int omap_iterate(CollectionHandle &c, ///< [in] collection + const ghobject_t &oid, ///< [in] object + /// [in] where the iterator should point to at the beginning + omap_iter_seek_t start_from, + std::function<omap_iter_ret_t(std::string_view, std::string_view)> f + ) override { + return 0; + } + void set_fsid(uuid_d u) override {} uuid_d get_fsid() override { return {}; } uint64_t estimate_objects_overhead(uint64_t num_objects) override { diff --git a/src/test/objectstore/allocsim/ops_replayer.cc b/src/test/objectstore/allocsim/ops_replayer.cc index fd947f5c454..c5908d9f576 100644 --- a/src/test/objectstore/allocsim/ops_replayer.cc +++ b/src/test/objectstore/allocsim/ops_replayer.cc @@ -1,4 +1,5 @@ #include <algorithm> +#include <functional> #include <boost/program_options/value_semantic.hpp> #include <cassert> #include <cctype> @@ -13,26 +14,46 @@ #include <fstream> #include <filesystem> #include <mutex> -#include "include/rados/buffer_fwd.h" -#include "include/rados/librados.hpp" #include <atomic> -#include <fmt/format.h> #include <map> #include <memory> #include <random> #include <string> #include <iostream> #include <vector> +#include <format> + +#include <fmt/format.h> #include <boost/program_options/variables_map.hpp> #include <boost/program_options/parsers.hpp> +#include "include/rados/buffer_fwd.h" +#include "include/rados/librados.hpp" + namespace po = boost::program_options; using namespace std; using namespace ceph; +namespace settings { + +// Returns a function which restricts a value to a specified range by throwing if it is not in range: +// (Note: std::clamp() does not throw.) +auto clamp_or_throw(auto min, auto max) +{ + return [=](auto& x) { + if(std::less<>{}(x, min) or std::greater<>{}(x, max)) { + throw std::out_of_range(fmt::format("value expected between {} and {}, but got {}", min, max, x)); + } + + return x; + }; +} + +} // namespace settings + // compare shared_ptr<string> struct StringPtrCompare { @@ -338,8 +359,8 @@ int main(int argc, char** argv) { // options uint64_t io_depth = 8; - uint64_t nparser_threads = 16; - uint64_t nworker_threads = 16; + int nparser_threads = 16; + int nworker_threads = 16; string file("input.txt"); string ceph_conf_path("./ceph.conf"); string pool("test_pool"); @@ -351,8 +372,8 @@ int main(int argc, char** argv) { ("input-files,i", po::value<vector<string>>()->multitoken(), "List of input files (output of op_scraper.py). Multiple files will be merged and sorted by time order") ("ceph-conf", po::value<string>(&ceph_conf_path)->default_value("ceph.conf"), "Path to ceph conf") ("io-depth", po::value<uint64_t>(&io_depth)->default_value(64), "I/O depth") - ("parser-threads", po::value<uint64_t>(&nparser_threads)->default_value(16), "Number of parser threads") - ("worker-threads", po::value<uint64_t>(&nworker_threads)->default_value(16), "Number of I/O worker threads") + ("parser-threads", po::value<int>(&nparser_threads)->default_value(16)->notifier(settings::clamp_or_throw(1, 256)), "Number of parser threads") + ("worker-threads", po::value<int>(&nworker_threads)->default_value(16)->notifier(settings::clamp_or_throw(1, 256)), "Number of I/O worker threads") ("pool", po::value<string>(&pool)->default_value("test_pool"), "Pool to use for I/O") ("skip-do-ops", po::bool_switch(&skip_do_ops)->default_value(false), "Skip doing operations") ; diff --git a/src/test/objectstore/store_test.cc b/src/test/objectstore/store_test.cc index 38b62f3ea6d..a75857e5e97 100644 --- a/src/test/objectstore/store_test.cc +++ b/src/test/objectstore/store_test.cc @@ -39,6 +39,7 @@ #include "global/global_init.h" #include "common/ceph_mutex.h" #include "common/Cond.h" +#include "common/debug.h" #include "common/errno.h" #include "common/options.h" // for the size literals #include "common/pretty_binary.h" diff --git a/src/test/objectstore/test_bluefs.cc b/src/test/objectstore/test_bluefs.cc index e2f95dd6c80..60147b5397c 100644 --- a/src/test/objectstore/test_bluefs.cc +++ b/src/test/objectstore/test_bluefs.cc @@ -23,6 +23,11 @@ using namespace std; +// some test should not be executed on jenkins make check +#define SKIP_JENKINS() \ + if (getenv("JENKINS_HOME") != nullptr) GTEST_SKIP_("test disabled on jenkins"); + + std::unique_ptr<char[]> gen_buffer(uint64_t size) { std::unique_ptr<char[]> buffer = std::make_unique<char[]>(size); @@ -174,6 +179,7 @@ TEST(BlueFS, small_appends) { } TEST(BlueFS, very_large_write) { + SKIP_JENKINS(); // we'll write a ~5G file, so allocate more than that for the whole fs uint64_t size = 1048576 * 1024 * 6ull; TempBdev bdev{size}; @@ -248,6 +254,7 @@ TEST(BlueFS, very_large_write) { } TEST(BlueFS, very_large_write2) { + SKIP_JENKINS(); // we'll write a ~5G file, so allocate more than that for the whole fs uint64_t size_full = 1048576 * 1024 * 6ull; uint64_t size = 1048576 * 1024 * 5ull; @@ -1601,6 +1608,91 @@ TEST(BlueFS, test_log_runway_advance_seq) { fs.compact_log(); } +TEST(BlueFS, test_69481_truncate_corrupts_log) { + uint64_t size = 1048576 * 128; + TempBdev bdev{size}; + BlueFS fs(g_ceph_context); + ASSERT_EQ(0, fs.add_block_device(BlueFS::BDEV_DB, bdev.path, false)); + uuid_d fsid; + ASSERT_EQ(0, fs.mkfs(fsid, { BlueFS::BDEV_DB, false, false })); + ASSERT_EQ(0, fs.mount()); + ASSERT_EQ(0, fs.maybe_verify_layout({ BlueFS::BDEV_DB, false, false })); + + BlueFS::FileWriter *f = nullptr; + BlueFS::FileWriter *a = nullptr; + ASSERT_EQ(0, fs.mkdir("dir")); + ASSERT_EQ(0, fs.open_for_write("dir", "test-file", &f, false)); + ASSERT_EQ(0, fs.open_for_write("dir", "just-allocate", &a, false)); + + // create 4 distinct extents in file f + // a is here only to prevent f from merging extents together + fs.preallocate(f->file, 0, 0x10000); + fs.preallocate(a->file, 0, 0x10000); + fs.preallocate(f->file, 0, 0x20000); + fs.preallocate(a->file, 0, 0x20000); + fs.preallocate(f->file, 0, 0x30000); + fs.preallocate(a->file, 0, 0x30000); + fs.preallocate(f->file, 0, 0x40000); + fs.preallocate(a->file, 0, 0x40000); + fs.close_writer(a); + + fs.truncate(f, 0); + fs.fsync(f); + + bufferlist bl; + bl.append(std::string(" ", 0x15678)); + f->append(bl); + fs.truncate(f, 0x15678); + fs.fsync(f); + fs.close_writer(f); + + fs.umount(); + // remount to verify + ASSERT_EQ(0, fs.mount()); + fs.umount(); +} + +TEST(BlueFS, test_69481_truncate_asserts) { + uint64_t size = 1048576 * 128; + TempBdev bdev{size}; + BlueFS fs(g_ceph_context); + ASSERT_EQ(0, fs.add_block_device(BlueFS::BDEV_DB, bdev.path, false)); + uuid_d fsid; + ASSERT_EQ(0, fs.mkfs(fsid, { BlueFS::BDEV_DB, false, false })); + ASSERT_EQ(0, fs.mount()); + ASSERT_EQ(0, fs.maybe_verify_layout({ BlueFS::BDEV_DB, false, false })); + + BlueFS::FileWriter *f = nullptr; + BlueFS::FileWriter *a = nullptr; + ASSERT_EQ(0, fs.mkdir("dir")); + ASSERT_EQ(0, fs.open_for_write("dir", "test-file", &f, false)); + ASSERT_EQ(0, fs.open_for_write("dir", "just-allocate", &a, false)); + + // create 4 distinct extents in file f + // a is here only to prevent f from merging extents together + fs.preallocate(f->file, 0, 0x10000); + fs.preallocate(a->file, 0, 0x10000); + fs.preallocate(f->file, 0, 0x20000); + fs.preallocate(a->file, 0, 0x20000); + fs.preallocate(f->file, 0, 0x30000); + fs.preallocate(a->file, 0, 0x30000); + fs.preallocate(f->file, 0, 0x40000); + fs.preallocate(a->file, 0, 0x40000); + fs.close_writer(a); + + fs.truncate(f, 0); + fs.fsync(f); + + bufferlist bl; + bl.append(std::string(" ", 0x35678)); + f->append(bl); + fs.truncate(f, 0x35678); + fs.fsync(f); + fs.close_writer(f); + + fs.umount(); +} + int main(int argc, char **argv) { auto args = argv_to_vec(argc, argv); map<string,string> defaults = { diff --git a/src/test/objectstore/test_deferred.cc b/src/test/objectstore/test_deferred.cc index 1b5608101c8..4c2790eaac7 100644 --- a/src/test/objectstore/test_deferred.cc +++ b/src/test/objectstore/test_deferred.cc @@ -2,6 +2,7 @@ // vim: ts=8 sw=2 smarttab #include <stdio.h> +#include <stdlib.h> #include <string.h> #include <iostream> #include <memory> @@ -44,10 +45,11 @@ void create_deferred_and_terminate() { coll_t cid; ghobject_t hoid; ObjectStore::CollectionHandle ch; - ceph_assert(::mkdir("bluestore.test_temp_dir", 0777) == 0); + std::string const db_store_dir = "bluestore.test_temp_dir_" + std::to_string(time(NULL)); + ceph_assert(::mkdir(db_store_dir.c_str(), 0777) == 0); store = ObjectStore::create(g_ceph_context, "bluestore", - "bluestore.test_temp_dir", + db_store_dir.c_str(), "store_test_temp_journal"); ceph_assert(store->mkfs() == 0); ceph_assert(store->mount() == 0); diff --git a/src/test/objectstore/test_memstore_clone.cc b/src/test/objectstore/test_memstore_clone.cc index 507f74d22d2..b4ef66933e1 100644 --- a/src/test/objectstore/test_memstore_clone.cc +++ b/src/test/objectstore/test_memstore_clone.cc @@ -17,6 +17,7 @@ #include "os/ObjectStore.h" #include <gtest/gtest.h> #include "include/ceph_assert.h" +#include "common/debug.h" #include "common/errno.h" #include "store_test_fixture.h" diff --git a/src/test/objectstore_bench.cc b/src/test/objectstore_bench.cc index 65a2987d08d..ef5392efb45 100644 --- a/src/test/objectstore_bench.cc +++ b/src/test/objectstore_bench.cc @@ -12,6 +12,7 @@ #include "global/global_init.h" +#include "common/debug.h" #include "common/strtol.h" #include "common/ceph_argparse.h" diff --git a/src/test/osd/CMakeLists.txt b/src/test/osd/CMakeLists.txt index f2d1471e22e..798558ebbe0 100644 --- a/src/test/osd/CMakeLists.txt +++ b/src/test/osd/CMakeLists.txt @@ -22,7 +22,7 @@ install(TARGETS add_executable(ceph_test_rados_io_sequence ${CMAKE_CURRENT_SOURCE_DIR}/ceph_test_rados_io_sequence.cc) target_link_libraries(ceph_test_rados_io_sequence - librados global object_io_exerciser) + librados global object_io_exerciser json_structures) install(TARGETS ceph_test_rados_io_sequence DESTINATION ${CMAKE_INSTALL_BINDIR}) diff --git a/src/test/osd/ceph_test_rados_io_sequence.cc b/src/test/osd/ceph_test_rados_io_sequence.cc index 5e340c5c9c5..96808ea37e5 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.cc +++ b/src/test/osd/ceph_test_rados_io_sequence.cc @@ -1,82 +1,104 @@ #include "ceph_test_rados_io_sequence.h" +#include <boost/asio/io_context.hpp> #include <iostream> #include <vector> -#include <boost/asio/io_context.hpp> - -#include "include/random.h" - -#include "librados/librados_asio.h" -#include "common/ceph_argparse.h" -#include "include/interval_set.h" -#include "global/global_init.h" -#include "global/global_context.h" +#include "common/Formatter.h" #include "common/Thread.h" +#include "common/ceph_argparse.h" +#include "common/ceph_json.h" #include "common/debug.h" #include "common/dout.h" #include "common/split.h" +#include "common/strtol.h" // for strict_iecstrtoll() +#include "common/ceph_json.h" +#include "common/Formatter.h" #include "common/io_exerciser/DataGenerator.h" +#include "common/io_exerciser/EcIoSequence.h" +#include "common/io_exerciser/IoOp.h" +#include "common/io_exerciser/IoSequence.h" #include "common/io_exerciser/Model.h" #include "common/io_exerciser/ObjectModel.h" #include "common/io_exerciser/RadosIo.h" -#include "common/io_exerciser/IoOp.h" -#include "common/io_exerciser/IoSequence.h" +#include "common/json/BalancerStructures.h" +#include "common/json/ConfigStructures.h" +#include "common/json/OSDStructures.h" +#include "fmt/format.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/interval_set.h" +#include "include/random.h" +#include "json_spirit/json_spirit.h" +#include "librados/librados_asio.h" #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context +using OpType = ceph::io_exerciser::OpType; + +using DoneOp = ceph::io_exerciser::DoneOp; +using BarrierOp = ceph::io_exerciser::BarrierOp; +using CreateOp = ceph::io_exerciser::CreateOp; +using RemoveOp = ceph::io_exerciser::RemoveOp; +using SingleReadOp = ceph::io_exerciser::SingleReadOp; +using DoubleReadOp = ceph::io_exerciser::DoubleReadOp; +using TripleReadOp = ceph::io_exerciser::TripleReadOp; +using SingleWriteOp = ceph::io_exerciser::SingleWriteOp; +using DoubleWriteOp = ceph::io_exerciser::DoubleWriteOp; +using TripleWriteOp = ceph::io_exerciser::TripleWriteOp; +using SingleFailedWriteOp = ceph::io_exerciser::SingleFailedWriteOp; +using DoubleFailedWriteOp = ceph::io_exerciser::DoubleFailedWriteOp; +using TripleFailedWriteOp = ceph::io_exerciser::TripleFailedWriteOp; + namespace { - struct Size {}; - void validate(boost::any& v, const std::vector<std::string>& values, - Size *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - - std::string parse_error; - uint64_t size = strict_iecstrtoll(s, &parse_error); - if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); - } - v = boost::any(size); - } - - struct Pair {}; - void validate(boost::any& v, const std::vector<std::string>& values, - Pair *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - auto part = ceph::split(s).begin(); - std::string parse_error; - int first = strict_iecstrtoll(*part++, &parse_error); - int second = strict_iecstrtoll(*part, &parse_error); - if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); - } - v = boost::any(std::pair<int,int>{first,second}); - } - - struct PluginString {}; - void validate(boost::any& v, const std::vector<std::string>& values, - PluginString *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - - const std::string_view* pluginIt = std::find( - ceph::io_sequence::tester::pluginChoices.begin(), - ceph::io_sequence::tester::pluginChoices.end(), - s - ); - if(ceph::io_sequence::tester::pluginChoices.end() == pluginIt) - { - throw po::validation_error(po::validation_error::invalid_option_value); - } +struct Size {}; +void validate(boost::any& v, const std::vector<std::string>& values, + Size* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); - v = boost::any(*pluginIt); + std::string parse_error; + uint64_t size = strict_iecstrtoll(s, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); } + v = boost::any(size); +} + +struct Pair {}; +void validate(boost::any& v, const std::vector<std::string>& values, + Pair* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); + auto part = ceph::split(s).begin(); + std::string parse_error; + int first = strict_iecstrtoll(*part++, &parse_error); + int second = strict_iecstrtoll(*part, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + v = boost::any(std::pair<int, int>{first, second}); +} + +struct PluginString {}; +void validate(boost::any& v, const std::vector<std::string>& values, + PluginString* target_type, int) { + po::validators::check_first_occurrence(v); + const std::string& s = po::validators::get_single_string(values); + + const std::string_view* pluginIt = + std::find(ceph::io_sequence::tester::pluginChoices.begin(), + ceph::io_sequence::tester::pluginChoices.end(), s); + if (ceph::io_sequence::tester::pluginChoices.end() == pluginIt) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + + v = boost::any(*pluginIt); +} - constexpr std::string_view usage[] = { +constexpr std::string_view usage[] = { "Basic usage:", "", "ceph_test_rados_io_sequence", @@ -118,103 +140,99 @@ namespace { "\t are specified with unit of blocksize. Supported commands:", "\t\t create <len>", "\t\t remove", - "\t\t read|write <off> <len>", - "\t\t read2|write2 <off> <len> <off> <len>", - "\t\t read3|write3 <off> <len> <off> <len> <off> <len>", - "\t\t done" - }; - - po::options_description get_options_description() - { - po::options_description desc("ceph_test_rados_io options"); - desc.add_options() - ("help,h", - "show help message") - ("listsequence,l", - "show list of sequences") - ("dryrun,d", - "test sequence, do not issue any I/O") - ("verbose", - "more verbose output during test") - ("sequence,s", po::value<int>(), - "test specified sequence") - ("seed", po::value<int>(), - "seed for whole test") - ("seqseed", po::value<int>(), - "seed for sequence") - ("blocksize,b", po::value<Size>(), - "block size (default 2048)") - ("chunksize,c", po::value<Size>(), - "chunk size (default 4096)") - ("pool,p", po::value<std::string>(), - "pool name") - ("object,o", po::value<std::string>()->default_value("test"), - "object name") - ("km", po::value<Pair>(), - "k,m EC pool profile (default 2,2)") - ("plugin", po::value<PluginString>(), - "EC plugin (isa or jerasure)") - ("objectsize", po::value<Pair>(), - "min,max object size in blocks (default 1,32)") - ("threads,t", po::value<int>(), - "number of threads of I/O per object (default 1)") - ("parallel,p", po::value<int>()->default_value(1), - "number of objects to exercise in parallel") - ("interactive", - "interactive mode, execute IO commands from stdin"); - - return desc; - } - - int parse_io_seq_options( - po::variables_map& vm, - int argc, - char** argv) - { - std::vector<std::string> unrecognized_options; - try { - po::options_description desc = get_options_description(); - - auto parsed = po::command_line_parser(argc, argv) - .options(desc) - .allow_unregistered() - .run(); - po::store(parsed, vm); - po::notify(vm); - unrecognized_options = po::collect_unrecognized(parsed.options, - po::include_positional); - - if (!unrecognized_options.empty()) - { - std::stringstream ss; - ss << "Unrecognised command options supplied: "; - while (unrecognized_options.size() > 1) - { - ss << unrecognized_options.back().c_str() << ", "; - unrecognized_options.pop_back(); - } - ss << unrecognized_options.back(); - dout(0) << ss.str() << dendl; - return 1; + "\t\t read|write|failedwrite <off> <len>", + "\t\t read2|write2|failedwrite2 <off> <len> <off> <len>", + "\t\t read3|write3|failedwrite3 <off> <len> <off> <len> <off> <len>", + "\t\t injecterror <type> <shard> <good_count> <fail_count>", + "\t\t clearinject <type> <shard>", + "\t\t done"}; + +po::options_description get_options_description() { + po::options_description desc("ceph_test_rados_io options"); + desc.add_options()("help,h", "show help message")("listsequence,l", + "show list of sequences")( + "dryrun,d", "test sequence, do not issue any I/O")( + "verbose", "more verbose output during test")( + "sequence,s", po::value<int>(), "test specified sequence")( + "seed", po::value<int>(), "seed for whole test")( + "seqseed", po::value<int>(), "seed for sequence")( + "blocksize,b", po::value<Size>(), "block size (default 2048)")( + "chunksize,c", po::value<Size>(), "chunk size (default 4096)")( + "pool,p", po::value<std::string>(), "pool name")( + "object,o", po::value<std::string>()->default_value("test"), + "object name")("km", po::value<Pair>(), + "k,m EC pool profile (default 2,2)")( + "plugin", po::value<PluginString>(), "EC plugin (isa or jerasure)")( + "objectsize", po::value<Pair>(), + "min,max object size in blocks (default 1,32)")( + "threads,t", po::value<int>(), + "number of threads of I/O per object (default 1)")( + "parallel,p", po::value<int>()->default_value(1), + "number of objects to exercise in parallel")( + "testrecovery", + "Inject errors during sequences to test recovery processes of OSDs")( + "interactive", "interactive mode, execute IO commands from stdin")( + "allow_pool_autoscaling", + "Allows pool autoscaling. Disabled by default.")( + "allow_pool_balancer", "Enables pool balancing. Disabled by default.")( + "allow_pool_deep_scrubbing", + "Enables pool deep scrub. Disabled by default.")( + "allow_pool_scrubbing", "Enables pool scrubbing. Disabled by default."); + + return desc; +} + +int parse_io_seq_options(po::variables_map& vm, int argc, char** argv) { + std::vector<std::string> unrecognized_options; + try { + po::options_description desc = get_options_description(); + + auto parsed = po::command_line_parser(argc, argv) + .options(desc) + .allow_unregistered() + .run(); + po::store(parsed, vm); + po::notify(vm); + unrecognized_options = + po::collect_unrecognized(parsed.options, po::include_positional); + + if (!unrecognized_options.empty()) { + std::stringstream ss; + ss << "Unrecognised command options supplied: "; + while (unrecognized_options.size() > 1) { + ss << unrecognized_options.back().c_str() << ", "; + unrecognized_options.pop_back(); } - } catch(const po::error& e) { - std::cerr << "error: " << e.what() << std::endl; + ss << unrecognized_options.back(); + dout(0) << ss.str() << dendl; return 1; } - - return 0; + } catch (const po::error& e) { + std::cerr << "error: " << e.what() << std::endl; + return 1; } + + return 0; } +template <typename S> +int send_mon_command(S& s, librados::Rados& rados, const char* name, + ceph::buffer::list& inbl, ceph::buffer::list* outbl, Formatter* f) { + std::ostringstream oss; + encode_json(name, s, f); + f->flush(oss); + int rc = rados.mon_command(oss.str(), inbl, outbl, nullptr); + return rc; +} + +} // namespace + template <typename T, int N, const std::array<T, N>& Ts> -ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts> - ::ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - const std::string& option_name, - bool set_forced, - bool select_first) - : rng(rng), - option_name(option_name) { +ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>:: + ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, const std::string& option_name, + bool set_forced, bool select_first) + : rng(rng), option_name(option_name) { if (set_forced && vm.count(option_name)) { force_value = vm[option_name].as<T>(); } @@ -225,76 +243,54 @@ ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts> } template <typename T, int N, const std::array<T, N>& Ts> -bool ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::isForced() -{ +bool ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::isForced() { return force_value.has_value(); } template <typename T, int N, const std::array<T, N>& Ts> -const T ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::choose() -{ +const T ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts>::choose() { if (force_value.has_value()) { return *force_value; } else if (first_value.has_value()) { return *std::exchange(first_value, std::nullopt); } else { - return choices[rng(N-1)]; + return choices[rng(N - 1)]; } } - - ceph::io_sequence::tester::SelectObjectSize::SelectObjectSize( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "objectsize", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "objectsize", true, true) {} ceph::io_sequence::tester::SelectBlockSize::SelectBlockSize( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "blocksize", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "blocksize", true, true) {} ceph::io_sequence::tester::SelectNumThreads::SelectNumThreads( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "threads", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "threads", true, true) {} ceph::io_sequence::tester::SelectSeqRange::SelectSeqRange( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "sequence", false, false) -{ + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "sequence", false, false) { if (vm.count(option_name)) { ceph::io_exerciser::Sequence s = - static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>()); + static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>()); if (s < ceph::io_exerciser::Sequence::SEQUENCE_BEGIN || s >= ceph::io_exerciser::Sequence::SEQUENCE_END) { dout(0) << "Sequence argument out of range" << dendl; throw po::validation_error(po::validation_error::invalid_option_value); } ceph::io_exerciser::Sequence e = s; - force_value = std::make_optional<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>>( - std::make_pair(s, ++e)); + force_value = std::make_optional< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>>( + std::make_pair(s, ++e)); } } -const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence> - ceph::io_sequence::tester::SelectSeqRange::choose() { - if (force_value.has_value()) - { +const std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> +ceph::io_sequence::tester::SelectSeqRange::choose() { + if (force_value.has_value()) { return *force_value; } else { return std::make_pair(ceph::io_exerciser::Sequence::SEQUENCE_BEGIN, @@ -302,45 +298,34 @@ const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence> } } - - ceph::io_sequence::tester::SelectErasureKM::SelectErasureKM( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "km", true, true) -{ -} - - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "km", true, true) {} ceph::io_sequence::tester::SelectErasurePlugin::SelectErasurePlugin( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm) - : ProgramOptionSelector(rng, vm, "plugin", true, false) -{ -} - - - -ceph::io_sequence::tester::SelectErasureChunkSize::SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm) - : ProgramOptionSelector(rng, vm, "stripe_unit", true, false) -{ -} - + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "plugin", true, false) {} +ceph::io_sequence::tester::SelectErasureChunkSize::SelectErasureChunkSize( + ceph::util::random_number_generator<int>& rng, po::variables_map vm) + : ProgramOptionSelector(rng, vm, "chunksize", true, true) {} ceph::io_sequence::tester::SelectECPool::SelectECPool( - ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - librados::Rados& rados, - bool dry_run) - : ProgramOptionSelector(rng, vm, "pool", false, false), - rados(rados), - dry_run(dry_run), - skm(SelectErasureKM(rng, vm)), - spl(SelectErasurePlugin(rng, vm)), - scs(SelectErasureChunkSize(rng, vm)) -{ + ceph::util::random_number_generator<int>& rng, po::variables_map vm, + librados::Rados& rados, bool dry_run, bool allow_pool_autoscaling, + bool allow_pool_balancer, bool allow_pool_deep_scrubbing, + bool allow_pool_scrubbing, bool test_recovery) + : ProgramOptionSelector(rng, vm, "pool", false, false), + rados(rados), + dry_run(dry_run), + allow_pool_autoscaling(allow_pool_autoscaling), + allow_pool_balancer(allow_pool_balancer), + allow_pool_deep_scrubbing(allow_pool_deep_scrubbing), + allow_pool_scrubbing(allow_pool_scrubbing), + test_recovery(test_recovery), + skm(SelectErasureKM(rng, vm)), + spl(SelectErasurePlugin(rng, vm)), + scs(SelectErasureChunkSize(rng, vm)) { if (!skm.isForced()) { if (vm.count("pool")) { force_value = vm["pool"].as<std::string>(); @@ -348,147 +333,239 @@ ceph::io_sequence::tester::SelectECPool::SelectECPool( } } -const std::string ceph::io_sequence::tester::SelectECPool::choose() -{ - std::pair<int,int> value; +const std::string ceph::io_sequence::tester::SelectECPool::choose() { + std::pair<int, int> value; if (!skm.isForced() && force_value.has_value()) { + int rc; + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDPoolGetRequest osdPoolGetRequest{*force_value}; + rc = send_mon_command(osdPoolGetRequest, rados, "OSDPoolGetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDPoolGetReply osdPoolGetReply; + osdPoolGetReply.decode_json(&p); + + ceph::messaging::osd::OSDECProfileGetRequest osdECProfileGetRequest{ + osdPoolGetReply.erasure_code_profile}; + rc = send_mon_command(osdECProfileGetRequest, rados, + "OSDECProfileGetRequest", inbl, &outbl, + formatter.get()); + ceph_assert(rc == 0); + + success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDECProfileGetReply reply; + reply.decode_json(&p); + k = reply.k; + m = reply.m; return *force_value; } else { value = skm.choose(); } - int k = value.first; - int m = value.second; + k = value.first; + m = value.second; const std::string plugin = std::string(spl.choose()); const uint64_t chunk_size = scs.choose(); - std::string pool_name = "ec_" + plugin + - "_cs" + std::to_string(chunk_size) + - "_k" + std::to_string(k) + - "_m" + std::to_string(m); - if (!dry_run) - { + std::string pool_name = "ec_" + plugin + "_cs" + std::to_string(chunk_size) + + "_k" + std::to_string(k) + "_m" + std::to_string(m); + if (!dry_run) { create_pool(rados, pool_name, plugin, chunk_size, k, m); } return pool_name; } void ceph::io_sequence::tester::SelectECPool::create_pool( - librados::Rados& rados, - const std::string& pool_name, - const std::string& plugin, - uint64_t chunk_size, - int k, int m) -{ + librados::Rados& rados, const std::string& pool_name, + const std::string& plugin, uint64_t chunk_size, int k, int m) { int rc; bufferlist inbl, outbl; - std::string profile_create = - "{\"prefix\": \"osd erasure-code-profile set\", \ - \"name\": \"testprofile-" + pool_name + "\", \ - \"profile\": [ \"plugin=" + plugin + "\", \ - \"k=" + std::to_string(k) + "\", \ - \"m=" + std::to_string(m) + "\", \ - \"stripe_unit=" + std::to_string(chunk_size) + "\", \ - \"crush-failure-domain=osd\"]}"; - rc = rados.mon_command(profile_create, inbl, &outbl, nullptr); + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDECProfileSetRequest ecProfileSetRequest{ + fmt::format("testprofile-{}", pool_name), + {fmt::format("plugin={}", plugin), fmt::format("k={}", k), + fmt::format("m={}", m), fmt::format("stripe_unit={}", chunk_size), + fmt::format("crush-failure-domain=osd")}}; + rc = send_mon_command(ecProfileSetRequest, rados, "OSDECProfileSetRequest", + inbl, &outbl, formatter.get()); ceph_assert(rc == 0); - std::string cmdstr = - "{\"prefix\": \"osd pool create\", \ - \"pool\": \"" + pool_name + "\", \ - \"pool_type\": \"erasure\", \ - \"pg_num\": 8, \ - \"pgp_num\": 8, \ - \"erasure_code_profile\": \"testprofile-" + pool_name + "\"}"; - rc = rados.mon_command(cmdstr, inbl, &outbl, nullptr); + + ceph::messaging::osd::OSDECPoolCreateRequest poolCreateRequest{ + pool_name, "erasure", 8, 8, fmt::format("testprofile-{}", pool_name)}; + rc = send_mon_command(poolCreateRequest, rados, "OSDECPoolCreateRequest", + inbl, &outbl, formatter.get()); ceph_assert(rc == 0); -} + if (allow_pool_autoscaling) { + ceph::messaging::osd::OSDSetRequest setNoAutoscaleRequest{"noautoscale", + std::nullopt}; + rc = send_mon_command(setNoAutoscaleRequest, rados, "OSDSetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (allow_pool_balancer) { + ceph::messaging::balancer::BalancerOffRequest balancerOffRequest{}; + rc = send_mon_command(balancerOffRequest, rados, "BalancerOffRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + ceph::messaging::balancer::BalancerStatusRequest balancerStatusRequest{}; + rc = send_mon_command(balancerStatusRequest, rados, "BalancerStatusRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::balancer::BalancerStatusReply reply; + reply.decode_json(&p); + ceph_assert(!reply.active); + } + if (allow_pool_deep_scrubbing) { + ceph::messaging::osd::OSDSetRequest setNoDeepScrubRequest{"nodeep-scrub", + std::nullopt}; + rc = send_mon_command(setNoDeepScrubRequest, rados, "setNoDeepScrubRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (allow_pool_scrubbing) { + ceph::messaging::osd::OSDSetRequest setNoScrubRequest{"noscrub", + std::nullopt}; + rc = send_mon_command(setNoScrubRequest, rados, "OSDSetRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + } + + if (test_recovery) { + ceph::messaging::config::ConfigSetRequest configSetBluestoreDebugRequest{ + "global", "bluestore_debug_inject_read_err", "true", std::nullopt}; + rc = send_mon_command(configSetBluestoreDebugRequest, rados, + "ConfigSetRequest", inbl, &outbl, + formatter.get()); + ceph_assert(rc == 0); + + ceph::messaging::config::ConfigSetRequest configSetMaxMarkdownRequest{ + "global", "osd_max_markdown_count", "99999999", std::nullopt}; + rc = + send_mon_command(configSetMaxMarkdownRequest, rados, "ConfigSetRequest", + inbl, &outbl, formatter.get()); + ceph_assert(rc == 0); + } +} -ceph::io_sequence::tester::TestObject::TestObject( const std::string oid, - librados::Rados& rados, - boost::asio::io_context& asio, - SelectBlockSize& sbs, - SelectECPool& spo, - SelectObjectSize& sos, - SelectNumThreads& snt, - SelectSeqRange& ssr, - ceph::util::random_number_generator<int>& rng, - ceph::mutex& lock, - ceph::condition_variable& cond, - bool dryrun, - bool verbose, - std::optional<int> seqseed) : - rng(rng), verbose(verbose), seqseed(seqseed) -{ +ceph::io_sequence::tester::TestObject::TestObject( + const std::string oid, librados::Rados& rados, + boost::asio::io_context& asio, SelectBlockSize& sbs, SelectECPool& spo, + SelectObjectSize& sos, SelectNumThreads& snt, SelectSeqRange& ssr, + ceph::util::random_number_generator<int>& rng, ceph::mutex& lock, + ceph::condition_variable& cond, bool dryrun, bool verbose, + std::optional<int> seqseed, bool testrecovery) + : rng(rng), verbose(verbose), seqseed(seqseed), testrecovery(testrecovery) { if (dryrun) { - verbose = true; - exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>(oid, - sbs.choose(), - rng()); + exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>( + oid, sbs.choose(), rng()); } else { const std::string pool = spo.choose(); + poolK = spo.getChosenK(); + poolM = spo.getChosenM(); + int threads = snt.choose(); - exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, - asio, - pool, - oid, - sbs.choose(), - rng(), - threads, - lock, - cond); - dout(0) << "= " << oid << " pool=" << pool - << " threads=" << threads - << " blocksize=" << exerciser_model->get_block_size() - << " =" << dendl; + + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + std::optional<std::vector<int>> cached_shard_order = std::nullopt; + + if (!spo.get_allow_pool_autoscaling() && !spo.get_allow_pool_balancer() && + !spo.get_allow_pool_deep_scrubbing() && + !spo.get_allow_pool_scrubbing()) { + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, oid, ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply{}; + reply.decode_json(&p); + cached_shard_order = reply.acting; + } + + exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>( + rados, asio, pool, oid, cached_shard_order, sbs.choose(), rng(), + threads, lock, cond); + dout(0) << "= " << oid << " pool=" << pool << " threads=" << threads + << " blocksize=" << exerciser_model->get_block_size() << " =" + << dendl; } obj_size_range = sos.choose(); seq_range = ssr.choose(); curseq = seq_range.first; - seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, - obj_size_range, - seqseed.value_or(rng())); + + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + curseq, obj_size_range, poolK, poolM, seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + curseq, obj_size_range, seqseed.value_or(rng())); + } + op = seq->next(); done = false; - dout(0) << "== " << exerciser_model->get_oid() << " " - << curseq << " " - << seq->get_name() - << " ==" <<dendl; + dout(0) << "== " << exerciser_model->get_oid() << " " << curseq << " " + << seq->get_name_with_seqseed() << " ==" << dendl; } -bool ceph::io_sequence::tester::TestObject::readyForIo() -{ +bool ceph::io_sequence::tester::TestObject::readyForIo() { return exerciser_model->readyForIoOp(*op); } -bool ceph::io_sequence::tester::TestObject::next() -{ +bool ceph::io_sequence::tester::TestObject::next() { if (!done) { if (verbose) { - dout(0) << exerciser_model->get_oid() - << " Step " << seq->get_step() << ": " - << op->to_string(exerciser_model->get_block_size()) << dendl; + dout(0) << exerciser_model->get_oid() << " Step " << seq->get_step() + << ": " << op->to_string(exerciser_model->get_block_size()) + << dendl; } else { - dout(5) << exerciser_model->get_oid() - << " Step " << seq->get_step() << ": " - << op->to_string(exerciser_model->get_block_size()) << dendl; + dout(5) << exerciser_model->get_oid() << " Step " << seq->get_step() + << ": " << op->to_string(exerciser_model->get_block_size()) + << dendl; } exerciser_model->applyIoOp(*op); - if (op->done()) { - ++curseq; - if (curseq == seq_range.second) { + if (op->getOpType() == ceph::io_exerciser::OpType::Done) { + curseq = seq->getNextSupportedSequenceId(); + if (curseq >= seq_range.second) { done = true; dout(0) << exerciser_model->get_oid() << " Number of IOs = " << exerciser_model->get_num_io() << dendl; } else { - seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, - obj_size_range, - seqseed.value_or(rng())); - dout(0) << "== " << exerciser_model->get_oid() << " " - << curseq << " " << seq->get_name() - << " ==" <<dendl; + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + curseq, obj_size_range, poolK, poolM, seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + curseq, obj_size_range, seqseed.value_or(rng())); + } + + dout(0) << "== " << exerciser_model->get_oid() << " " << curseq << " " + << seq->get_name_with_seqseed() << " ==" << dendl; op = seq->next(); } } else { @@ -498,27 +575,30 @@ bool ceph::io_sequence::tester::TestObject::next() return done; } -bool ceph::io_sequence::tester::TestObject::finished() -{ - return done; -} +bool ceph::io_sequence::tester::TestObject::finished() { return done; } -int ceph::io_sequence::tester::TestObject::get_num_io() -{ +int ceph::io_sequence::tester::TestObject::get_num_io() { return exerciser_model->get_num_io(); } ceph::io_sequence::tester::TestRunner::TestRunner(po::variables_map& vm, - librados::Rados& rados) : - rados(rados), - seed(vm.contains("seed") ? vm["seed"].as<int>() : time(nullptr)), - rng(ceph::util::random_number_generator<int>(seed)), - sbs{rng, vm}, - sos{rng, vm}, - spo{rng, vm, rados, vm.contains("dryrun")}, - snt{rng, vm}, - ssr{rng, vm} -{ + librados::Rados& rados) + : rados(rados), + seed(vm.contains("seed") ? vm["seed"].as<int>() : time(nullptr)), + rng(ceph::util::random_number_generator<int>(seed)), + sbs{rng, vm}, + sos{rng, vm}, + spo{rng, + vm, + rados, + vm.contains("dryrun"), + vm.contains("allow_pool_autoscaling"), + vm.contains("allow_pool_balancer"), + vm.contains("allow_pool_deep_scrubbing"), + vm.contains("allow_pool_scrubbing"), + vm.contains("test_recovery")}, + snt{rng, vm}, + ssr{rng, vm} { dout(0) << "Test using seed " << seed << dendl; verbose = vm.contains("verbose"); @@ -531,19 +611,23 @@ ceph::io_sequence::tester::TestRunner::TestRunner(po::variables_map& vm, num_objects = vm["parallel"].as<int>(); object_name = vm["object"].as<std::string>(); interactive = vm.contains("interactive"); + testrecovery = vm.contains("testrecovery"); + + allow_pool_autoscaling = vm.contains("allow_pool_autoscaling"); + allow_pool_balancer = vm.contains("allow_pool_balancer"); + allow_pool_deep_scrubbing = vm.contains("allow_pool_deep_scrubbing"); + allow_pool_scrubbing = vm.contains("allow_pool_scrubbing"); - if (!dryrun) - { + if (!dryrun) { guard.emplace(boost::asio::make_work_guard(asio)); - thread = make_named_thread("io_thread",[&asio = asio] { asio.run(); }); + thread = make_named_thread("io_thread", [&asio = asio] { asio.run(); }); } show_help = vm.contains("help"); show_sequence = vm.contains("listsequence"); } -ceph::io_sequence::tester::TestRunner::~TestRunner() -{ +ceph::io_sequence::tester::TestRunner::~TestRunner() { if (!dryrun) { guard = std::nullopt; asio.stop(); @@ -552,34 +636,38 @@ ceph::io_sequence::tester::TestRunner::~TestRunner() } } -void ceph::io_sequence::tester::TestRunner::help() -{ +void ceph::io_sequence::tester::TestRunner::help() { std::cout << get_options_description() << std::endl; for (auto line : usage) { std::cout << line << std::endl; } } -void ceph::io_sequence::tester::TestRunner::list_sequence() -{ +void ceph::io_sequence::tester::TestRunner::list_sequence(bool testrecovery) { // List seqeunces - std::pair<int,int> obj_size_range = sos.choose(); - for (ceph::io_exerciser::Sequence s - = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; - s < ceph::io_exerciser::Sequence::SEQUENCE_END; ++s) { - std::unique_ptr<ceph::io_exerciser::IoSequence> seq = - ceph::io_exerciser::IoSequence::generate_sequence(s, - obj_size_range, - seqseed.value_or(rng())); - dout(0) << s << " " << seq->get_name() << dendl; + std::pair<int, int> obj_size_range = sos.choose(); + ceph::io_exerciser::Sequence s = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; + std::unique_ptr<ceph::io_exerciser::IoSequence> seq; + if (testrecovery) { + seq = ceph::io_exerciser::EcIoSequence::generate_sequence( + s, obj_size_range, spo.getChosenK(), spo.getChosenM(), + seqseed.value_or(rng())); + } else { + seq = ceph::io_exerciser::IoSequence::generate_sequence( + s, obj_size_range, seqseed.value_or(rng())); } + + do { + dout(0) << s << " " << seq->get_name_with_seqseed() << dendl; + s = seq->getNextSupportedSequenceId(); + } while (s != ceph::io_exerciser::Sequence::SEQUENCE_END); } -std::string ceph::io_sequence::tester::TestRunner::get_token() -{ - static std::string line; - static ceph::split split = ceph::split(""); - static ceph::spliterator tokens; +void ceph::io_sequence::tester::TestRunner::clear_tokens() { + tokens = split.end(); +} + +std::string ceph::io_sequence::tester::TestRunner::get_token() { while (line.empty() || tokens == split.end()) { if (!std::getline(std::cin, line)) { throw std::runtime_error("End of input"); @@ -590,127 +678,211 @@ std::string ceph::io_sequence::tester::TestRunner::get_token() return std::string(*tokens++); } -uint64_t ceph::io_sequence::tester::TestRunner::get_numeric_token() -{ +std::optional<std::string> +ceph::io_sequence::tester::TestRunner ::get_optional_token() { + std::optional<std::string> ret = std::nullopt; + if (tokens != split.end()) { + ret = std::string(*tokens++); + } + return ret; +} + +uint64_t ceph::io_sequence::tester::TestRunner::get_numeric_token() { std::string parse_error; std::string token = get_token(); uint64_t num = strict_iecstrtoll(token, &parse_error); if (!parse_error.empty()) { - throw std::runtime_error("Invalid number "+token); + throw std::runtime_error("Invalid number " + token); } return num; } -bool ceph::io_sequence::tester::TestRunner::run_test() -{ - if (show_help) - { +std::optional<uint64_t> +ceph::io_sequence::tester::TestRunner ::get_optional_numeric_token() { + std::string parse_error; + std::optional<std::string> token = get_optional_token(); + if (token) { + uint64_t num = strict_iecstrtoll(*token, &parse_error); + if (!parse_error.empty()) { + throw std::runtime_error("Invalid number " + *token); + } + return num; + } + + return std::optional<uint64_t>(std::nullopt); +} + +bool ceph::io_sequence::tester::TestRunner::run_test() { + if (show_help) { help(); return true; - } - else if (show_sequence) - { - list_sequence(); + } else if (show_sequence) { + list_sequence(testrecovery); return true; - } - else if (interactive) - { + } else if (interactive) { return run_interactive_test(); - } - else - { + } else { return run_automated_test(); } } -bool ceph::io_sequence::tester::TestRunner::run_interactive_test() -{ +bool ceph::io_sequence::tester::TestRunner::run_interactive_test() { bool done = false; std::unique_ptr<ceph::io_exerciser::IoOp> ioop; std::unique_ptr<ceph::io_exerciser::Model> model; if (dryrun) { - model = std::make_unique<ceph::io_exerciser::ObjectModel>(object_name, - sbs.choose(), - rng()); + model = std::make_unique<ceph::io_exerciser::ObjectModel>( + object_name, sbs.choose(), rng()); } else { const std::string pool = spo.choose(); - model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, asio, pool, - object_name, sbs.choose(), - rng(), 1, // 1 thread - lock, cond); + + bufferlist inbl, outbl; + auto formatter = std::make_unique<JSONFormatter>(false); + + ceph::messaging::osd::OSDMapRequest osdMapRequest{pool, object_name, ""}; + int rc = send_mon_command(osdMapRequest, rados, "OSDMapRequest", inbl, + &outbl, formatter.get()); + ceph_assert(rc == 0); + + JSONParser p; + bool success = p.parse(outbl.c_str(), outbl.length()); + ceph_assert(success); + + ceph::messaging::osd::OSDMapReply reply{}; + reply.decode_json(&p); + + model = std::make_unique<ceph::io_exerciser::RadosIo>( + rados, asio, pool, object_name, reply.acting, sbs.choose(), rng(), + 1, // 1 thread + lock, cond); } while (!done) { const std::string op = get_token(); - if (!op.compare("done") || !op.compare("q") || !op.compare("quit")) { - ioop = ceph::io_exerciser::IoOp::generate_done(); - } else if (!op.compare("create")) { - ioop = ceph::io_exerciser::IoOp::generate_create(get_numeric_token()); - } else if (!op.compare("remove") || !op.compare("delete")) { - ioop = ceph::io_exerciser::IoOp::generate_remove(); - } else if (!op.compare("read")) { + if (op == "done" || op == "q" || op == "quit") { + ioop = ceph::io_exerciser::DoneOp::generate(); + } else if (op == "create") { + ioop = ceph::io_exerciser::CreateOp::generate(get_numeric_token()); + } else if (op == "remove" || op == "delete") { + ioop = ceph::io_exerciser::RemoveOp::generate(); + } else if (op == "read") { uint64_t offset = get_numeric_token(); uint64_t length = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read(offset, length); - } else if (!op.compare("read2")) { + ioop = ceph::io_exerciser::SingleReadOp::generate(offset, length); + } else if (op == "read2") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read2(offset1, length1, - offset2, length2); - } else if (!op.compare("read3")) { + ioop = DoubleReadOp::generate(offset1, length1, offset2, length2); + } else if (op == "read3") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); uint64_t offset3 = get_numeric_token(); uint64_t length3 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_read3(offset1, length1, - offset2, length2, - offset3, length3); - } else if (!op.compare("write")) { + ioop = TripleReadOp::generate(offset1, length1, offset2, length2, offset3, + length3); + } else if (op == "write") { uint64_t offset = get_numeric_token(); uint64_t length = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write(offset, length); - } else if (!op.compare("write2")) { + ioop = SingleWriteOp::generate(offset, length); + } else if (op == "write2") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write2(offset1, length1, - offset2, length2); - } else if (!op.compare("write3")) { + ioop = DoubleWriteOp::generate(offset1, length1, offset2, length2); + } else if (op == "write3") { uint64_t offset1 = get_numeric_token(); uint64_t length1 = get_numeric_token(); uint64_t offset2 = get_numeric_token(); uint64_t length2 = get_numeric_token(); uint64_t offset3 = get_numeric_token(); uint64_t length3 = get_numeric_token(); - ioop = ceph::io_exerciser::IoOp::generate_write3(offset1, length1, - offset2, length2, - offset3, length3); + ioop = TripleWriteOp::generate(offset1, length1, offset2, length2, + offset3, length3); + } else if (op == "failedwrite") { + uint64_t offset = get_numeric_token(); + uint64_t length = get_numeric_token(); + ioop = SingleFailedWriteOp::generate(offset, length); + } else if (op == "failedwrite2") { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + ioop = DoubleFailedWriteOp::generate(offset1, length1, offset2, length2); + } else if (op == "failedwrite3") { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + uint64_t offset3 = get_numeric_token(); + uint64_t length3 = get_numeric_token(); + ioop = TripleFailedWriteOp::generate(offset1, length1, offset2, length2, + offset3, length3); + } else if (op == "injecterror") { + std::string inject_type = get_token(); + int shard = get_numeric_token(); + std::optional<int> type = get_optional_numeric_token(); + std::optional<int> when = get_optional_numeric_token(); + std::optional<int> duration = get_optional_numeric_token(); + if (inject_type == "read") { + ioop = ceph::io_exerciser::InjectReadErrorOp::generate(shard, type, + when, duration); + } else if (inject_type == "write") { + ioop = ceph::io_exerciser::InjectWriteErrorOp::generate(shard, type, + when, duration); + } else { + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid error inject {}. No action performed.", + inject_type) + << dendl; + } + } else if (op == "clearinject") { + std::string inject_type = get_token(); + int shard = get_numeric_token(); + std::optional<int> type = get_optional_numeric_token(); + if (inject_type == "read") { + ioop = + ceph::io_exerciser::ClearReadErrorInjectOp::generate(shard, type); + } else if (inject_type == "write") { + ioop = + ceph::io_exerciser::ClearWriteErrorInjectOp::generate(shard, type); + } else { + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid error inject {}. No action performed.", + inject_type) + << dendl; + } } else { - throw std::runtime_error("Invalid operation "+op); + clear_tokens(); + ioop.reset(); + dout(0) << fmt::format("Invalid op {}. No action performed.", op) + << dendl; } - dout(0) << ioop->to_string(model->get_block_size()) << dendl; - model->applyIoOp(*ioop); - done = ioop->done(); - if (!done) { - ioop = ceph::io_exerciser::IoOp::generate_barrier(); + if (ioop) { + dout(0) << ioop->to_string(model->get_block_size()) << dendl; model->applyIoOp(*ioop); + done = ioop->getOpType() == ceph::io_exerciser::OpType::Done; + if (!done) { + ioop = ceph::io_exerciser::BarrierOp::generate(); + model->applyIoOp(*ioop); + } } } return true; } -bool ceph::io_sequence::tester::TestRunner::run_automated_test() -{ +bool ceph::io_sequence::tester::TestRunner::run_automated_test() { // Create a test for each object - std::vector<std::shared_ptr< - ceph::io_sequence::tester::TestObject>> test_objects; + std::vector<std::shared_ptr<ceph::io_sequence::tester::TestObject>> + test_objects; for (int obj = 0; obj < num_objects; obj++) { std::string name; @@ -720,15 +892,9 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() name = object_name + std::to_string(obj); } test_objects.push_back( - std::make_shared<ceph::io_sequence::tester::TestObject>( - name, - rados, asio, - sbs, spo, sos, snt, ssr, - rng, lock, cond, - dryrun, verbose, - seqseed - ) - ); + std::make_shared<ceph::io_sequence::tester::TestObject>( + name, rados, asio, sbs, spo, sos, snt, ssr, rng, lock, cond, dryrun, + verbose, seqseed, testrecovery)); } if (!dryrun) { rados.wait_for_latest_osdmap(); @@ -747,16 +913,15 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() for (auto obj = test_objects.begin(); obj != test_objects.end(); ++obj) { std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj; if (!to->finished()) { - lock.lock(); - bool ready = to->readyForIo(); - lock.unlock(); - if (ready) - { - to->next(); - started_io = true; - } else { - need_wait = true; - } + lock.lock(); + bool ready = to->readyForIo(); + lock.unlock(); + if (ready) { + to->next(); + started_io = true; + } else { + need_wait = true; + } } } if (!started_io && need_wait) { @@ -766,8 +931,7 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() std::shared_ptr<ceph::io_sequence::tester::TestObject> to = *obj; if (!to->finished()) { need_wait = !to->readyForIo(); - if (!need_wait) - { + if (!need_wait) { break; } } @@ -787,18 +951,16 @@ bool ceph::io_sequence::tester::TestRunner::run_automated_test() return true; } -int main(int argc, char **argv) -{ +int main(int argc, char** argv) { auto args = argv_to_vec(argc, argv); env_to_vec(args); auto cct = global_init(NULL, args, CEPH_ENTITY_TYPE_CLIENT, - CODE_ENVIRONMENT_UTILITY, 0); + CODE_ENVIRONMENT_UTILITY, 0); common_init_finish(cct.get()); po::variables_map vm; int rc = parse_io_seq_options(vm, argc, argv); - if (rc != 0) - { + if (rc != 0) { return rc; } @@ -813,7 +975,7 @@ int main(int argc, char **argv) std::unique_ptr<ceph::io_sequence::tester::TestRunner> runner; try { runner = std::make_unique<ceph::io_sequence::tester::TestRunner>(vm, rados); - } catch(const po::error& e) { + } catch (const po::error& e) { return 1; } runner->run_test(); diff --git a/src/test/osd/ceph_test_rados_io_sequence.h b/src/test/osd/ceph_test_rados_io_sequence.h index 4f77c940274..9af5f706b2f 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.h +++ b/src/test/osd/ceph_test_rados_io_sequence.h @@ -1,33 +1,36 @@ +#include <boost/program_options.hpp> +#include <optional> #include <utility> -#include "include/random.h" - -#include "global/global_init.h" -#include "global/global_context.h" - #include "common/io_exerciser/IoOp.h" #include "common/io_exerciser/IoSequence.h" #include "common/io_exerciser/Model.h" - +#include "common/split.h" +#include "global/global_context.h" +#include "global/global_init.h" +#include "include/random.h" #include "librados/librados_asio.h" +#include <boost/asio/io_context.hpp> #include <boost/program_options.hpp> +#include <optional> + /* Overview * * class ProgramOptionSelector - * Base class for selector objects below with common code for + * Base class for selector objects below with common code for * selecting options - * + * * class SelectObjectSize * Selects min and max object sizes for a test * * class SelectErasureKM * Selects an EC k and m value for a test - * + * * class SelectErasurePlugin * Selects an plugin for a test - * + * * class SelectECPool * Selects an EC pool (plugin,k and m) for a test. Also creates the * pool as well. @@ -57,287 +60,279 @@ namespace po = boost::program_options; -namespace ceph -{ - namespace io_sequence::tester - { - // Choices for min and max object size - inline constexpr size_t objectSizeSize = 10; - inline constexpr std::array<std::pair<int,int>,objectSizeSize> - objectSizeChoices = {{ - {1,32}, // Default - best for boundary checking - {12,14}, - {28,30}, - {36,38}, - {42,44}, - {52,54}, - {66,68}, - {72,74}, - {83,83}, - {97,97} - }}; - - // Choices for block size - inline constexpr int blockSizeSize = 5; - inline constexpr std::array<uint64_t, blockSizeSize> blockSizeChoices = {{ - 2048, // Default - test boundaries for EC 4K chunk size - 512, - 3767, - 4096, - 32768 - }}; - - // Choices for number of threads - inline constexpr int threadArraySize = 4; - inline constexpr std::array<int, threadArraySize> threadCountChoices = {{ - 1, // Default - 2, - 4, - 8 - }}; - - // Choices for EC k+m profile - inline constexpr int kmSize = 6; - inline constexpr std::array<std::pair<int,int>, kmSize> kmChoices = {{ - {2,2}, // Default - reasonable coverage - {2,1}, - {2,3}, - {3,2}, - {4,2}, - {5,1} - }}; - - // Choices for EC chunk size - inline constexpr int chunkSizeSize = 3; - inline constexpr std::array<uint64_t, chunkSizeSize> chunkSizeChoices = {{ - 4*1024, - 64*1024, - 256*1024 - }}; - - // Choices for plugin - inline constexpr int pluginListSize = 2; - inline constexpr std::array<std::string_view, - pluginListSize> pluginChoices = {{ - "jerasure", - "isa" - }}; - - inline constexpr std::array<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>, - 0> sequencePairs = {{}}; - - inline constexpr std::array<std::string, 0> poolChoices = {{}}; - - template <typename T, int N, const std::array<T, N>& Ts> - class ProgramOptionSelector - { - public: - ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - const std::string& option_name, - bool set_forced, - bool select_first - ); - virtual ~ProgramOptionSelector() = default; - bool isForced(); - virtual const T choose(); - - protected: - ceph::util::random_number_generator<int>& rng; - static constexpr std::array<T, N> choices = Ts; - - std::optional<T> force_value; - std::optional<T> first_value; - - std::string option_name; - }; - - class SelectObjectSize - : public ProgramOptionSelector<std::pair<int, int>, - io_sequence::tester::objectSizeSize, - io_sequence::tester::objectSizeChoices> - { - public: - SelectObjectSize(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectBlockSize - : public ProgramOptionSelector<uint64_t, - io_sequence::tester::blockSizeSize, - io_sequence::tester::blockSizeChoices> - { - public: - SelectBlockSize(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectNumThreads - : public ProgramOptionSelector<int, - io_sequence::tester::threadArraySize, - io_sequence::tester::threadCountChoices> - { - public: - SelectNumThreads(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectSeqRange - : public ProgramOptionSelector<std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence>, - 0, io_sequence::tester::sequencePairs> - { - public: - SelectSeqRange(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - - const std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence> choose() override; - }; - - class SelectErasureKM - : public ProgramOptionSelector<std::pair<int,int>, - io_sequence::tester::kmSize, - io_sequence::tester::kmChoices> - { - public: - SelectErasureKM(ceph::util::random_number_generator<int>& rng, +namespace ceph { +namespace io_sequence::tester { +// Choices for min and max object size +inline constexpr size_t objectSizeSize = 10; +inline constexpr std::array<std::pair<int, int>, objectSizeSize> + objectSizeChoices = {{{1, 32}, // Default - best for boundary checking + {12, 14}, + {28, 30}, + {36, 38}, + {42, 44}, + {52, 54}, + {66, 68}, + {72, 74}, + {83, 83}, + {97, 97}}}; + +// Choices for block size +inline constexpr int blockSizeSize = 5; +inline constexpr std::array<uint64_t, blockSizeSize> blockSizeChoices = { + {2048, // Default - test boundaries for EC 4K chunk size + 512, 3767, 4096, 32768}}; + +// Choices for number of threads +inline constexpr int threadArraySize = 4; +inline constexpr std::array<int, threadArraySize> threadCountChoices = { + {1, // Default + 2, 4, 8}}; + +// Choices for EC k+m profile +inline constexpr int kmSize = 6; +inline constexpr std::array<std::pair<int, int>, kmSize> kmChoices = { + {{2, 2}, // Default - reasonable coverage + {2, 1}, + {2, 3}, + {3, 2}, + {4, 2}, + {5, 1}}}; + +// Choices for EC chunk size +inline constexpr int chunkSizeSize = 3; +inline constexpr std::array<uint64_t, chunkSizeSize> chunkSizeChoices = { + {4 * 1024, 64 * 1024, 256 * 1024}}; + +// Choices for plugin +inline constexpr int pluginListSize = 2; +inline constexpr std::array<std::string_view, pluginListSize> pluginChoices = { + {"jerasure", "isa"}}; + +inline constexpr std::array< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>, 0> + sequencePairs = {{}}; + +inline constexpr std::array<std::string, 0> poolChoices = {{}}; + +template <typename T, int N, const std::array<T, N>& Ts> +class ProgramOptionSelector { + public: + ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, const std::string& option_name, + bool set_forced, bool select_first); + virtual ~ProgramOptionSelector() = default; + bool isForced(); + virtual const T choose(); + + protected: + ceph::util::random_number_generator<int>& rng; + static constexpr std::array<T, N> choices = Ts; + + std::optional<T> force_value; + std::optional<T> first_value; + + std::string option_name; +}; + +class SelectObjectSize + : public ProgramOptionSelector<std::pair<int, int>, + io_sequence::tester::objectSizeSize, + io_sequence::tester::objectSizeChoices> { + public: + SelectObjectSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectBlockSize + : public ProgramOptionSelector<uint64_t, io_sequence::tester::blockSizeSize, + io_sequence::tester::blockSizeChoices> { + public: + SelectBlockSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectNumThreads + : public ProgramOptionSelector<int, io_sequence::tester::threadArraySize, + io_sequence::tester::threadCountChoices> { + public: + SelectNumThreads(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectSeqRange + : public ProgramOptionSelector< + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence>, + 0, io_sequence::tester::sequencePairs> { + public: + SelectSeqRange(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); + + const std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> + choose() override; +}; + +class SelectErasureKM + : public ProgramOptionSelector<std::pair<int, int>, + io_sequence::tester::kmSize, + io_sequence::tester::kmChoices> { + public: + SelectErasureKM(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectErasurePlugin + : public ProgramOptionSelector<std::string_view, + io_sequence::tester::pluginListSize, + io_sequence::tester::pluginChoices> { + public: + SelectErasurePlugin(ceph::util::random_number_generator<int>& rng, po::variables_map vm); - }; - - class SelectErasurePlugin - : public ProgramOptionSelector<std::string_view, - io_sequence::tester::pluginListSize, - io_sequence::tester::pluginChoices> - { - public: - SelectErasurePlugin(ceph::util::random_number_generator<int>& rng, - po::variables_map vm); - }; - - class SelectErasureChunkSize - : public ProgramOptionSelector<uint64_t, - io_sequence::tester::chunkSizeSize, - io_sequence::tester::chunkSizeChoices> - { - public: - SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, po::variables_map vm); - }; - - class SelectECPool - : public ProgramOptionSelector<std::string, - 0, - io_sequence::tester::poolChoices> - { - public: - SelectECPool(ceph::util::random_number_generator<int>& rng, - po::variables_map vm, - librados::Rados& rados, - bool dry_run); - const std::string choose() override; - - private: - void create_pool(librados::Rados& rados, - const std::string& pool_name, - const std::string& plugin, - uint64_t chunk_size, - int k, int m); - - protected: - librados::Rados& rados; - bool dry_run; - - SelectErasureKM skm; - SelectErasurePlugin spl; - SelectErasureChunkSize scs; - }; - - class TestObject - { - public: - TestObject( const std::string oid, - librados::Rados& rados, - boost::asio::io_context& asio, - ceph::io_sequence::tester::SelectBlockSize& sbs, - ceph::io_sequence::tester::SelectECPool& spl, - ceph::io_sequence::tester::SelectObjectSize& sos, - ceph::io_sequence::tester::SelectNumThreads& snt, - ceph::io_sequence::tester::SelectSeqRange& ssr, - ceph::util::random_number_generator<int>& rng, - ceph::mutex& lock, - ceph::condition_variable& cond, - bool dryrun, - bool verbose, - std::optional<int> seqseed); - - int get_num_io(); - bool readyForIo(); - bool next(); - bool finished(); - - protected: - std::unique_ptr<ceph::io_exerciser::Model> exerciser_model; - std::pair<int,int> obj_size_range; - std::pair<ceph::io_exerciser::Sequence, - ceph::io_exerciser::Sequence> seq_range; - ceph::io_exerciser::Sequence curseq; - std::unique_ptr<ceph::io_exerciser::IoSequence> seq; - std::unique_ptr<ceph::io_exerciser::IoOp> op; - bool done; - ceph::util::random_number_generator<int>& rng; - bool verbose; - std::optional<int> seqseed; - }; - - class TestRunner - { - public: - TestRunner(po::variables_map& vm, librados::Rados& rados); - ~TestRunner(); - - bool run_test(); - - private: - librados::Rados& rados; - int seed; - ceph::util::random_number_generator<int> rng; - - ceph::io_sequence::tester::SelectBlockSize sbs; - ceph::io_sequence::tester::SelectObjectSize sos; - ceph::io_sequence::tester::SelectECPool spo; - ceph::io_sequence::tester::SelectNumThreads snt; - ceph::io_sequence::tester::SelectSeqRange ssr; - - boost::asio::io_context asio; - std::thread thread; - std::optional<boost::asio::executor_work_guard< - boost::asio::io_context::executor_type>> guard; - ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); - ceph::condition_variable cond; - - bool input_valid; - - bool verbose; - bool dryrun; - std::optional<int> seqseed; - bool interactive; - - bool show_sequence; - bool show_help; - - int num_objects; - std::string object_name; - - std::string get_token(); - uint64_t get_numeric_token(); - - bool run_automated_test(); - - bool run_interactive_test(); - - void help(); - void list_sequence(); - }; - } -}
\ No newline at end of file +}; + +class SelectErasureChunkSize + : public ProgramOptionSelector<uint64_t, io_sequence::tester::chunkSizeSize, + io_sequence::tester::chunkSizeChoices> { + public: + SelectErasureChunkSize(ceph::util::random_number_generator<int>& rng, + po::variables_map vm); +}; + +class SelectECPool + : public ProgramOptionSelector<std::string, 0, + io_sequence::tester::poolChoices> { + public: + SelectECPool(ceph::util::random_number_generator<int>& rng, + po::variables_map vm, librados::Rados& rados, bool dry_run, + bool allow_pool_autoscaling, bool allow_pool_balancer, + bool allow_pool_deep_scrubbing, bool allow_pool_scrubbing, + bool test_recovery); + const std::string choose() override; + + bool get_allow_pool_autoscaling() { return allow_pool_autoscaling; } + bool get_allow_pool_balancer() { return allow_pool_balancer; } + bool get_allow_pool_deep_scrubbing() { return allow_pool_deep_scrubbing; } + bool get_allow_pool_scrubbing() { return allow_pool_scrubbing; } + int getChosenK() const { return k; } + int getChosenM() const { return m; } + + private: + void create_pool(librados::Rados& rados, const std::string& pool_name, + const std::string& plugin, uint64_t chunk_size, int k, + int m); + + protected: + librados::Rados& rados; + bool dry_run; + bool allow_pool_autoscaling; + bool allow_pool_balancer; + bool allow_pool_deep_scrubbing; + bool allow_pool_scrubbing; + bool test_recovery; + int k; + int m; + + SelectErasureKM skm; + SelectErasurePlugin spl; + SelectErasureChunkSize scs; +}; + +class TestObject { + public: + TestObject(const std::string oid, librados::Rados& rados, + boost::asio::io_context& asio, + ceph::io_sequence::tester::SelectBlockSize& sbs, + ceph::io_sequence::tester::SelectECPool& spl, + ceph::io_sequence::tester::SelectObjectSize& sos, + ceph::io_sequence::tester::SelectNumThreads& snt, + ceph::io_sequence::tester::SelectSeqRange& ssr, + ceph::util::random_number_generator<int>& rng, ceph::mutex& lock, + ceph::condition_variable& cond, bool dryrun, bool verbose, + std::optional<int> seqseed, bool testRecovery); + + int get_num_io(); + bool readyForIo(); + bool next(); + bool finished(); + + protected: + std::unique_ptr<ceph::io_exerciser::Model> exerciser_model; + std::pair<int, int> obj_size_range; + std::pair<ceph::io_exerciser::Sequence, ceph::io_exerciser::Sequence> + seq_range; + ceph::io_exerciser::Sequence curseq; + std::unique_ptr<ceph::io_exerciser::IoSequence> seq; + std::unique_ptr<ceph::io_exerciser::IoOp> op; + bool done; + ceph::util::random_number_generator<int>& rng; + bool verbose; + std::optional<int> seqseed; + int poolK; + int poolM; + bool testrecovery; +}; + +class TestRunner { + public: + TestRunner(po::variables_map& vm, librados::Rados& rados); + ~TestRunner(); + + bool run_test(); + + private: + librados::Rados& rados; + int seed; + ceph::util::random_number_generator<int> rng; + + ceph::io_sequence::tester::SelectBlockSize sbs; + ceph::io_sequence::tester::SelectObjectSize sos; + ceph::io_sequence::tester::SelectECPool spo; + ceph::io_sequence::tester::SelectNumThreads snt; + ceph::io_sequence::tester::SelectSeqRange ssr; + + boost::asio::io_context asio; + std::thread thread; + std::optional< + boost::asio::executor_work_guard<boost::asio::io_context::executor_type>> + guard; + ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); + ceph::condition_variable cond; + + bool input_valid; + + bool verbose; + bool dryrun; + std::optional<int> seqseed; + bool interactive; + + bool testrecovery; + + bool allow_pool_autoscaling; + bool allow_pool_balancer; + bool allow_pool_deep_scrubbing; + bool allow_pool_scrubbing; + + bool show_sequence; + bool show_help; + + int num_objects; + std::string object_name; + + std::string line; + ceph::split split = ceph::split(""); + ceph::spliterator tokens; + + void clear_tokens(); + std::string get_token(); + std::optional<std::string> get_optional_token(); + uint64_t get_numeric_token(); + std::optional<uint64_t> get_optional_numeric_token(); + + bool run_automated_test(); + + bool run_interactive_test(); + + void help(); + void list_sequence(bool testrecovery); +}; +} // namespace io_sequence::tester +} // namespace ceph diff --git a/src/test/osd/test_ec_transaction.cc b/src/test/osd/test_ec_transaction.cc index c17df4802ed..64397ce88bf 100644 --- a/src/test/osd/test_ec_transaction.cc +++ b/src/test/osd/test_ec_transaction.cc @@ -15,6 +15,7 @@ #include <gtest/gtest.h> #include "osd/PGTransaction.h" #include "osd/ECTransaction.h" +#include "common/debug.h" #include "test/unit.cc" diff --git a/src/test/osdc/MemWriteback.cc b/src/test/osdc/MemWriteback.cc index 4cb11291a98..bcc828e064b 100644 --- a/src/test/osdc/MemWriteback.cc +++ b/src/test/osdc/MemWriteback.cc @@ -1,6 +1,8 @@ // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab +#include "MemWriteback.h" + #include <errno.h> #include <time.h> @@ -11,8 +13,7 @@ #include "common/ceph_mutex.h" #include "include/ceph_assert.h" #include "common/ceph_time.h" - -#include "MemWriteback.h" +#include "common/snap_types.h" // for class SnapContext #define dout_context g_ceph_context #define dout_subsys ceph_subsys_objectcacher diff --git a/src/test/pybind/pytest.ini b/src/test/pybind/pytest.ini index dccf2a346dc..97569e88299 100644 --- a/src/test/pybind/pytest.ini +++ b/src/test/pybind/pytest.ini @@ -7,3 +7,4 @@ markers = stats tier watch + wait diff --git a/src/test/pybind/test_ceph_argparse.py b/src/test/pybind/test_ceph_argparse.py index 3039223abdf..630e6046b24 100755 --- a/src/test/pybind/test_ceph_argparse.py +++ b/src/test/pybind/test_ceph_argparse.py @@ -217,7 +217,7 @@ class TestPG(TestArgparse): def test_pg_missing_args_output(self): ret, _, stderr = self._capture_output(['pg'], stderr=True) self.assertEqual({}, ret) - self.assertRegexpMatches(stderr, re.compile('no valid command found.* closest matches')) + self.assertRegex(stderr, re.compile('no valid command found.* closest matches')) def test_pg_wrong_arg_output(self): ret, _, stderr = self._capture_output(['pg', 'map', 'bad-pgid'], @@ -416,10 +416,10 @@ class TestMDS(TestArgparse): class TestFS(TestArgparse): - + def test_dump(self): self.check_0_or_1_natural_arg('fs', 'dump') - + def test_fs_new(self): self._assert_valid_command(['fs', 'new', 'default', 'metadata', 'data']) @@ -912,7 +912,7 @@ class TestOSD(TestArgparse): '1.2.3.4/567', '600.40']) self._assert_valid_command(['osd', 'blocklist', action, '1.2.3.4', '600.40']) - + self._assert_valid_command(['osd', 'blocklist', action, 'v1:1.2.3.4', '600.40']) self._assert_valid_command(['osd', 'blocklist', action, @@ -925,7 +925,7 @@ class TestOSD(TestArgparse): 'v2:[2607:f298:4:2243::5522]:0/0', '600.40']) self._assert_valid_command(['osd', 'blocklist', action, '[2001:0db8::85a3:0000:8a2e:0370:7334]:0/0', '600.40']) - + self.assertEqual({}, validate_command(sigdict, ['osd', 'blocklist', action, 'invalid', diff --git a/src/test/pybind/test_rados.py b/src/test/pybind/test_rados.py index cb2a4f96101..25423bd8dcb 100644 --- a/src/test/pybind/test_rados.py +++ b/src/test/pybind/test_rados.py @@ -207,7 +207,7 @@ class TestRados(object): def test_get_fsid(self): fsid = self.rados.get_fsid() - assert re.match('[0-9a-f\-]{36}', fsid, re.I) + assert re.match(r'[0-9a-f\-]{36}', fsid, re.I) def test_blocklist_add(self): self.rados.blocklist_add("1.2.3.4/123", 1) diff --git a/src/test/rbd_mirror/test_ImageReplayer.cc b/src/test/rbd_mirror/test_ImageReplayer.cc index abe163cfd69..360c85a76b6 100644 --- a/src/test/rbd_mirror/test_ImageReplayer.cc +++ b/src/test/rbd_mirror/test_ImageReplayer.cc @@ -47,6 +47,8 @@ #include "test/librados/test_cxx.h" #include "gtest/gtest.h" +#include <shared_mutex> // for std::shared_lock + void register_test_rbd_mirror() { } diff --git a/src/test/rbd_mirror/test_ImageSync.cc b/src/test/rbd_mirror/test_ImageSync.cc index 93349ca1163..9418496bce2 100644 --- a/src/test/rbd_mirror/test_ImageSync.cc +++ b/src/test/rbd_mirror/test_ImageSync.cc @@ -24,6 +24,8 @@ #include "tools/rbd_mirror/Throttler.h" #include "tools/rbd_mirror/image_replayer/journal/StateBuilder.h" +#include <shared_mutex> // for std::shared_lock + void register_test_image_sync() { } diff --git a/src/test/rgw/bucket_notification/api.py b/src/test/rgw/bucket_notification/api.py index e7ec31f1711..e84aa16edc7 100644 --- a/src/test/rgw/bucket_notification/api.py +++ b/src/test/rgw/bucket_notification/api.py @@ -247,12 +247,16 @@ def delete_all_topics(conn, tenant, cluster): if tenant == '': topics_result = admin(['topic', 'list'], cluster) topics_json = json.loads(topics_result[0]) + if 'topics' not in topics_json: + topics_json = topics_json.get('result',{}) for topic in topics_json['topics']: rm_result = admin(['topic', 'rm', '--topic', topic['name']], cluster) print(rm_result) else: topics_result = admin(['topic', 'list', '--tenant', tenant], cluster) topics_json = json.loads(topics_result[0]) + if 'topics' not in topics_json: + topics_json = topics_json.get('result',{}) for topic in topics_json['topics']: rm_result = admin(['topic', 'rm', '--tenant', tenant, '--topic', topic['name']], cluster) print(rm_result) diff --git a/src/test/rgw/bucket_notification/requirements.txt b/src/test/rgw/bucket_notification/requirements.txt index a3cff2bedab..bb74eceedc3 100644 --- a/src/test/rgw/bucket_notification/requirements.txt +++ b/src/test/rgw/bucket_notification/requirements.txt @@ -1,4 +1,4 @@ -nose >=1.0.0 +nose-py3 >=1.0.0 boto >=2.6.0 boto3 >=1.0.0 configparser >=5.0.0 diff --git a/src/test/rgw/bucket_notification/test_bn.py b/src/test/rgw/bucket_notification/test_bn.py index 359990b3531..665fbca7494 100644 --- a/src/test/rgw/bucket_notification/test_bn.py +++ b/src/test/rgw/bucket_notification/test_bn.py @@ -410,17 +410,25 @@ kafka_server = 'localhost' class KafkaReceiver(object): """class for receiving and storing messages on a topic from the kafka broker""" - def __init__(self, topic, security_type): + def __init__(self, topic, security_type, kafka_server='localhost'): from kafka import KafkaConsumer remaining_retries = 10 port = 9092 if security_type != 'PLAINTEXT': security_type = 'SSL' port = 9093 + + if kafka_server is None: + endpoint = "localhost" + ":" + str(port) + elif ":" not in kafka_server: + endpoint = kafka_server + ":" + str(port) + else: + endpoint = kafka_server + while remaining_retries > 0: try: self.consumer = KafkaConsumer(topic, - bootstrap_servers = kafka_server+':'+str(port), + bootstrap_servers=endpoint, security_protocol=security_type, consumer_timeout_ms=16000, auto_offset_reset='earliest') @@ -468,9 +476,9 @@ def kafka_receiver_thread_runner(receiver): print('Kafka receiver ended unexpectedly: ' + str(error)) -def create_kafka_receiver_thread(topic, security_type='PLAINTEXT'): +def create_kafka_receiver_thread(topic, security_type='PLAINTEXT', kafka_brokers=None): """create kafka receiver and thread""" - receiver = KafkaReceiver(topic, security_type) + receiver = KafkaReceiver(topic, security_type, kafka_server=kafka_brokers) task = threading.Thread(target=kafka_receiver_thread_runner, args=(receiver,)) task.daemon = True return task, receiver @@ -1304,7 +1312,7 @@ def test_ps_s3_notification_errors_on_master(): conn.delete_bucket(bucket_name) -def notification_push(endpoint_type, conn, account=None, cloudevents=False): +def notification_push(endpoint_type, conn, account=None, cloudevents=False, kafka_brokers=None): """ test pushinging notification """ zonegroup = get_config_zonegroup() # create bucket @@ -1359,11 +1367,13 @@ def notification_push(endpoint_type, conn, account=None, cloudevents=False): assert_equal(status/100, 2) elif endpoint_type == 'kafka': # start amqp receiver - task, receiver = create_kafka_receiver_thread(topic_name) + task, receiver = create_kafka_receiver_thread(topic_name, kafka_brokers=kafka_brokers) task.start() endpoint_address = 'kafka://' + kafka_server # without acks from broker endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker' + if kafka_brokers is not None: + endpoint_args += '&kafka-brokers=' + kafka_brokers # create s3 topic topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() @@ -1581,6 +1591,20 @@ def test_notification_push_kafka(): notification_push('kafka', conn) +@attr('kafka_failover') +def test_notification_push_kafka_multiple_brokers_override(): + """ test pushing kafka s3 notification on master """ + conn = connection() + notification_push('kafka', conn, kafka_brokers='localhost:9092,localhost:19092') + + +@attr('kafka_failover') +def test_notification_push_kafka_multiple_brokers_append(): + """ test pushing kafka s3 notification on master """ + conn = connection() + notification_push('kafka', conn, kafka_brokers='localhost:19092') + + @attr('http_test') def test_ps_s3_notification_multi_delete_on_master(): """ test deletion of multiple keys on master """ @@ -2981,7 +3005,6 @@ def wait_for_queue_to_drain(topic_name, tenant=None, account=None, http_port=Non log.info('waited for %ds for queue %s to drain', time_diff, topic_name) -@attr('kafka_test') def persistent_topic_stats(conn, endpoint_type): zonegroup = get_config_zonegroup() @@ -2993,12 +3016,13 @@ def persistent_topic_stats(conn, endpoint_type): host = get_ip() task = None port = None + wrong_port = 1234 + endpoint_address = endpoint_type+'://'+host+':'+str(wrong_port) if endpoint_type == 'http': # create random port for the http server port = random.randint(10000, 20000) # start an http server in a separate thread receiver = HTTPServerWithEvents((host, port)) - endpoint_address = 'http://'+host+':'+str(port) endpoint_args = 'push-endpoint='+endpoint_address+'&persistent=true'+ \ '&retry_sleep_duration=1' elif endpoint_type == 'amqp': @@ -3006,23 +3030,18 @@ def persistent_topic_stats(conn, endpoint_type): exchange = 'ex1' task, receiver = create_amqp_receiver_thread(exchange, topic_name) task.start() - endpoint_address = 'amqp://' + host endpoint_args = 'push-endpoint='+endpoint_address+'&amqp-exchange='+exchange+'&amqp-ack-level=broker&persistent=true'+ \ '&retry_sleep_duration=1' elif endpoint_type == 'kafka': # start kafka receiver task, receiver = create_kafka_receiver_thread(topic_name) task.start() - endpoint_address = 'kafka://' + host endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ '&retry_sleep_duration=1' else: return SkipTest('Unknown endpoint type: ' + endpoint_type) # create s3 topic - endpoint_address = 'kafka://' + host + ':1234' # wrong port - endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ - '&retry_sleep_duration=1' topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() # create s3 notification @@ -3070,9 +3089,19 @@ def persistent_topic_stats(conn, endpoint_type): get_stats_persistent_topic(topic_name, 2 * number_of_objects) # change the endpoint port - endpoint_address = 'kafka://' + host - endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ - '&retry_sleep_duration=1' + if endpoint_type == 'http': + endpoint_address = endpoint_type+'://'+host+':'+str(port) + endpoint_args = 'push-endpoint='+endpoint_address+'&persistent=true'+ \ + '&retry_sleep_duration=1' + elif endpoint_type == 'amqp': + endpoint_address = endpoint_type+'://'+host + endpoint_args = 'push-endpoint='+endpoint_address+'&amqp-exchange='+exchange+'&amqp-ack-level=broker&persistent=true'+ \ + '&retry_sleep_duration=1' + elif endpoint_type == 'kafka': + endpoint_address = endpoint_type+'://'+host + endpoint_args = 'push-endpoint='+endpoint_address+'&kafka-ack-level=broker&persistent=true'+ \ + '&retry_sleep_duration=1' + topic_conf = PSTopicS3(conn, topic_name, zonegroup, endpoint_args=endpoint_args) topic_arn = topic_conf.set_config() @@ -3087,19 +3116,26 @@ def persistent_topic_stats(conn, endpoint_type): @attr('http_test') -def persistent_topic_stats_http(): +def test_persistent_topic_stats_http(): """ test persistent topic stats, http endpoint """ conn = connection() persistent_topic_stats(conn, 'http') @attr('kafka_test') -def persistent_topic_stats_kafka(): +def test_persistent_topic_stats_kafka(): """ test persistent topic stats, kafka endpoint """ conn = connection() persistent_topic_stats(conn, 'kafka') +@attr('amqp_test') +def test_persistent_topic_stats_amqp(): + """ test persistent topic stats, amqp endpoint """ + conn = connection() + persistent_topic_stats(conn, 'amqp') + + @attr('kafka_test') def test_persistent_topic_dump(): """ test persistent topic dump """ @@ -4359,6 +4395,242 @@ def test_ps_s3_multiple_topics_notification(): http_server.close() +@attr('data_path_v2_test') +def test_ps_s3_list_topics_migration(): + """ test list topics on migration""" + if get_config_cluster() == 'noname': + return SkipTest('realm is needed for migration test') + + # Initialize connections and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topics = [f"{bucket_name}{TOPIC_SUFFIX}{i}" for i in range(1, 7)] + tenant_topics = [f"{tenant}_{topic}" for topic in topics] + + # Define topic names with version + topic_versions = { + "topic1_v2": f"{topics[0]}_v2", + "topic2_v2": f"{topics[1]}_v2", + "topic3_v1": f"{topics[2]}_v1", + "topic4_v1": f"{topics[3]}_v1", + "topic5_v1": f"{topics[4]}_v1", + "topic6_v1": f"{topics[5]}_v1", + "tenant_topic1_v2": f"{tenant_topics[0]}_v2", + "tenant_topic2_v1": f"{tenant_topics[1]}_v1", + "tenant_topic3_v1": f"{tenant_topics[2]}_v1" + } + + # Get necessary configurations + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + conf_cluster = get_config_cluster() + + # Make sure there are no leftover topics on v2 + zonegroup_modify_feature(enable=True, feature_name=zonegroup_feature_notification_v2) + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Start v1 notification + # Make sure there are no leftover topics on v1 + zonegroup_modify_feature(enable=False, feature_name=zonegroup_feature_notification_v2) + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Create s3 - v1 topics + topic_conf = PSTopicS3(conn1, topic_versions['topic3_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic4_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn4 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic5_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn5 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic6_v1'], zonegroup, endpoint_args=endpoint_args) + topic_arn6 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic2_v1'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic3_v1'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn3 = tenant_topic_conf.set_config() + + # Start v2 notification + zonegroup_modify_feature(enable=True, feature_name=zonegroup_feature_notification_v2) + + # Create s3 - v2 topics + topic_conf = PSTopicS3(conn1, topic_versions['topic1_v2'], zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_versions['topic2_v2'], zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, topic_versions['tenant_topic1_v2'], zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 6) + + # Verify tenant topics + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 3) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + topic_conf.del_config(topic_arn4) + topic_conf.del_config(topic_arn5) + topic_conf.del_config(topic_arn6) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + tenant_topic_conf.del_config(tenant_topic_arn3) + + +@attr('basic_test') +def test_ps_s3_list_topics(): + """ test list topics""" + + # Initialize connections, topic names and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topic_name1 = bucket_name + TOPIC_SUFFIX + '1' + topic_name2 = bucket_name + TOPIC_SUFFIX + '2' + topic_name3 = bucket_name + TOPIC_SUFFIX + '3' + tenant_topic_name1 = tenant + "_" + topic_name1 + tenant_topic_name2 = tenant + "_" + topic_name2 + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + + # Make sure there are no leftover topics + delete_all_topics(conn1, '', get_config_cluster()) + delete_all_topics(conn2, tenant, get_config_cluster()) + + # Create s3 - v2 topics + topic_conf = PSTopicS3(conn1, topic_name1, zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name2, zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name3, zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name1, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name2, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] # version 2 + assert_equal(len(member), 3) + + # Verify topics for tenant + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 2) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + +@attr('data_path_v2_test') +def test_ps_s3_list_topics_v1(): + """ test list topics on v1""" + if get_config_cluster() == 'noname': + return SkipTest('realm is needed') + + # Initialize connections and configurations + conn1 = connection() + tenant = 'kaboom1' + conn2 = connect_random_user(tenant) + bucket_name = gen_bucket_name() + topic_name1 = bucket_name + TOPIC_SUFFIX + '1' + topic_name2 = bucket_name + TOPIC_SUFFIX + '2' + topic_name3 = bucket_name + TOPIC_SUFFIX + '3' + tenant_topic_name1 = tenant + "_" + topic_name1 + tenant_topic_name2 = tenant + "_" + topic_name2 + host = get_ip() + http_port = random.randint(10000, 20000) + endpoint_address = 'http://' + host + ':' + str(http_port) + endpoint_args = 'push-endpoint=' + endpoint_address + '&persistent=true' + zonegroup = get_config_zonegroup() + conf_cluster = get_config_cluster() + + # Make sure there are no leftover topics + delete_all_topics(conn1, '', conf_cluster) + delete_all_topics(conn2, tenant, conf_cluster) + + # Make sure that we disable v2 + zonegroup_modify_feature(enable=False, feature_name=zonegroup_feature_notification_v2) + + # Create s3 - v1 topics + topic_conf = PSTopicS3(conn1, topic_name1, zonegroup, endpoint_args=endpoint_args) + topic_arn1 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name2, zonegroup, endpoint_args=endpoint_args) + topic_arn2 = topic_conf.set_config() + topic_conf = PSTopicS3(conn1, topic_name3, zonegroup, endpoint_args=endpoint_args) + topic_arn3 = topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name1, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn1 = tenant_topic_conf.set_config() + tenant_topic_conf = PSTopicS3(conn2, tenant_topic_name2, zonegroup, endpoint_args=endpoint_args) + tenant_topic_arn2 = tenant_topic_conf.set_config() + + # Verify topics list + try: + # Verify no tenant topics + res, status = topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 3) + + # Verify tenant topics + res, status = tenant_topic_conf.get_list() + assert_equal(status // 100, 2) + listTopicsResponse = res.get('ListTopicsResponse', {}) + listTopicsResult = listTopicsResponse.get('ListTopicsResult', {}) + topics = listTopicsResult.get('Topics', {}) + member = topics['member'] if topics else [] + assert_equal(len(member), 2) + finally: + # Cleanup created topics + topic_conf.del_config(topic_arn1) + topic_conf.del_config(topic_arn2) + topic_conf.del_config(topic_arn3) + tenant_topic_conf.del_config(tenant_topic_arn1) + tenant_topic_conf.del_config(tenant_topic_arn2) + + @attr('basic_test') def test_ps_s3_topic_permissions(): """ test s3 topic set/get/delete permissions """ diff --git a/src/test/rgw/rgw_multi/tests.py b/src/test/rgw/rgw_multi/tests.py index 2d49c7a0ce0..433cd034fe0 100644 --- a/src/test/rgw/rgw_multi/tests.py +++ b/src/test/rgw/rgw_multi/tests.py @@ -15,6 +15,7 @@ import boto import boto.s3.connection from boto.s3.website import WebsiteConfiguration from boto.s3.cors import CORSConfiguration +from botocore.exceptions import ClientError from nose.tools import eq_ as eq from nose.tools import assert_not_equal, assert_equal, assert_true, assert_false @@ -573,6 +574,7 @@ def create_bucket_per_zone_in_realm(): b, z = create_bucket_per_zone(zg_conn) buckets.extend(b) zone_bucket.extend(z) + realm_meta_checkpoint(realm) return buckets, zone_bucket def test_bucket_create(): @@ -1212,6 +1214,9 @@ def test_datalog_autotrim(): # wait for metadata and data sync to catch up zonegroup_meta_checkpoint(zonegroup) zonegroup_data_checkpoint(zonegroup_conns) + zonegroup_bucket_checkpoint(zonegroup_conns, bucket.name) + time.sleep(config.checkpoint_delay) + zonegroup_data_checkpoint(zonegroup_conns) # trim each datalog for zone, _ in zone_bucket: @@ -3634,4 +3639,23 @@ def test_copy_object_different_bucket(): CopySource = source_bucket.name + '/' + objname) zonegroup_bucket_checkpoint(zonegroup_conns, dest_bucket.name) - + +def test_bucket_create_location_constraint(): + for zonegroup in realm.current_period.zonegroups: + zonegroup_conns = ZonegroupConns(zonegroup) + for zg in realm.current_period.zonegroups: + z = zonegroup_conns.rw_zones[0] + bucket_name = gen_bucket_name() + if zg.name == zonegroup.name: + # my zonegroup should pass + z.s3_client.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': zg.name}) + # check bucket location + response = z.s3_client.get_bucket_location(Bucket=bucket_name) + assert_equal(response['LocationConstraint'], zg.name) + else: + # other zonegroup should fail with 400 + e = assert_raises(ClientError, + z.s3_client.create_bucket, + Bucket=bucket_name, + CreateBucketConfiguration={'LocationConstraint': zg.name}) + assert e.response['ResponseMetadata']['HTTPStatusCode'] == 400 diff --git a/src/test/rgw/test-rgw-common.sh b/src/test/rgw/test-rgw-common.sh index 9129092898e..6798a15ba31 100644 --- a/src/test/rgw/test-rgw-common.sh +++ b/src/test/rgw/test-rgw-common.sh @@ -103,7 +103,7 @@ function init_first_zone { # create zonegroup, zone x $(rgw_admin $cid) zonegroup create --rgw-zonegroup=$zg --master --default - x $(rgw_admin $cid) zone create --rgw-zonegroup=$zg --rgw-zone=$zone --access-key=${access_key} --secret=${secret} --endpoints=$endpoints --default + x $(rgw_admin $cid) zone create --rgw-zonegroup=$zg --rgw-zone=$zone --access-key=${access_key} --secret=${secret} --endpoints=$endpoints --master --default x $(rgw_admin $cid) user create --uid=zone.user --display-name=ZoneUser --access-key=${access_key} --secret=${secret} --system x $(rgw_admin $cid) period update --commit @@ -128,7 +128,7 @@ function init_zone_in_existing_zg { x $(rgw_admin $cid) period update --commit } -function init_first_zone_in_slave_zg { +function init_first_zone_in_peer_zg { [ $# -ne 8 ] && echo "init_first_zone_in_slave_zg() needs 8 params" && exit 1 cid=$1 diff --git a/src/test/rgw/test-rgw-multisite.sh b/src/test/rgw/test-rgw-multisite.sh index a005b19e3da..d3a1b265ca6 100755 --- a/src/test/rgw/test-rgw-multisite.sh +++ b/src/test/rgw/test-rgw-multisite.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash -[ $# -lt 1 ] && echo "usage: $0 <num-clusters> [rgw parameters...]" && exit 1 +[ $# -lt 1 ] && echo "usage: $0 <num-zones> <num-zonegroups>[rgw parameters...]" && exit 1 -num_clusters=$1 +num_zones=$1 +num_zonegroups=$2 shift -[ $num_clusters -lt 1 ] && echo "clusters num must be at least 1" && exit 1 +[ $num_zones -lt 1 ] && echo "clusters num must be at least 1" && exit 1 . "`dirname $0`/test-rgw-common.sh" . "`dirname $0`/test-rgw-meta-sync.sh" @@ -53,7 +54,7 @@ echo realm_status=$output endpoints="" i=2 -while [ $i -le $num_clusters ]; do +while [ $i -le $num_zones ]; do x $(start_ceph_cluster c$i) -n $(get_mstart_parameters $i) j=1 endpoints="" @@ -74,10 +75,53 @@ while [ $i -le $num_clusters ]; do i=$((i+1)) done -i=2 -while [ $i -le $num_clusters ]; do - wait_for_meta_sync c1 c$i $realm_name +endpoints="" +k=2 +while [ $k -le $num_zonegroups ]; do + x $(start_ceph_cluster c$i) -n $(get_mstart_parameters $i) + j=1 + endpoints="" + while [ $j -le $rgws ]; do + port=$((8000+i*100+j)) + endpoints="$endpoints""$url:$port," + j=$((j+1)) + done + # create new zone, start rgw + init_first_zone_in_peer_zg c$i $realm_name zg${k} zg${k}-${i} 8101 $endpoints $system_access_key $system_secret + j=1 + while [ $j -le $rgws ]; do + port=$((8000+i*100+j)) + x $(rgw c$i "$port" "$@") + j="$((j+1))" + done +# bring up next clusters in zonegroup k i=$((i+1)) + + endpoints="" + l=2 + while [ $l -le $num_zones ]; do + x $(start_ceph_cluster c$i) -n $(get_mstart_parameters $i) + j=1 + endpoints="" + while [ $j -le $rgws ]; do + port=$((8000+i*100+j)) + endpoints="$endpoints""$url:$port," + j=$((j+1)) + done + + # create new zone, start rgw + init_zone_in_existing_zg c$i $realm_name zg${k} zg${k}-${i} 8101 $endpoints $zone_port $system_access_key $system_secret + j=1 + while [ $j -le $rgws ]; do + port=$((8000+i*100+j)) + x $(rgw c$i "$port" "$@") + j="$((j+1))" + done + l=$((l+1)) + i=$((i+1)) + done + + k=$((k+1)) done diff --git a/src/test/rgw/test_log_backing.cc b/src/test/rgw/test_log_backing.cc index e4109d535d1..a6de690af0f 100644 --- a/src/test/rgw/test_log_backing.cc +++ b/src/test/rgw/test_log_backing.cc @@ -20,6 +20,7 @@ #include <fmt/format.h> +#include "common/Clock.h" // for ceph_clock_now() #include "include/types.h" #include "include/rados/librados.hpp" diff --git a/src/test/rgw/test_rgw_iam_policy.cc b/src/test/rgw/test_rgw_iam_policy.cc index 7dadb7812ff..1d13c2aa013 100644 --- a/src/test/rgw/test_rgw_iam_policy.cc +++ b/src/test/rgw/test_rgw_iam_policy.cc @@ -75,6 +75,8 @@ using rgw::IAM::s3GetObjectTagging; using rgw::IAM::s3GetObjectVersion; using rgw::IAM::s3GetObjectVersionTagging; using rgw::IAM::s3GetObjectVersionTorrent; +using rgw::IAM::s3GetObjectAttributes; +using rgw::IAM::s3GetObjectVersionAttributes; using rgw::IAM::s3GetPublicAccessBlock; using rgw::IAM::s3GetReplicationConfiguration; using rgw::IAM::s3ListAllMyBuckets; @@ -419,6 +421,8 @@ TEST_F(PolicyTest, Parse3) { act2[s3GetObjectVersionAcl] = 1; act2[s3GetObjectTorrent] = 1; act2[s3GetObjectVersionTorrent] = 1; + act2[s3GetObjectAttributes] = 1; + act2[s3GetObjectVersionAttributes] = 1; act2[s3GetAccelerateConfiguration] = 1; act2[s3GetBucketAcl] = 1; act2[s3GetBucketOwnershipControls] = 1; @@ -487,6 +491,8 @@ TEST_F(PolicyTest, Eval3) { s3allow[s3GetObjectVersion] = 1; s3allow[s3GetObjectAcl] = 1; s3allow[s3GetObjectVersionAcl] = 1; + s3allow[s3GetObjectAttributes] = 1; + s3allow[s3GetObjectVersionAttributes] = 1; s3allow[s3GetObjectTorrent] = 1; s3allow[s3GetObjectVersionTorrent] = 1; s3allow[s3GetAccelerateConfiguration] = 1; @@ -883,6 +889,8 @@ TEST_F(ManagedPolicyTest, AmazonS3ReadOnlyAccess) act[s3GetObjectVersionAcl] = 1; act[s3GetObjectTorrent] = 1; act[s3GetObjectVersionTorrent] = 1; + act[s3GetObjectAttributes] = 1; + act[s3GetObjectVersionAttributes] = 1; act[s3GetAccelerateConfiguration] = 1; act[s3GetBucketAcl] = 1; act[s3GetBucketOwnershipControls] = 1; diff --git a/src/test/signals.cc b/src/test/signals.cc index dc24900a8c4..513b176b780 100644 --- a/src/test/signals.cc +++ b/src/test/signals.cc @@ -3,6 +3,7 @@ #include "global/signal_handler.h" #include "common/debug.h" #include "include/coredumpctl.h" +#include "log/Log.h" #include "gtest/gtest.h" diff --git a/src/test/test_denc.cc b/src/test/test_denc.cc index c9144bd05bf..db742b5e2cf 100644 --- a/src/test/test_denc.cc +++ b/src/test/test_denc.cc @@ -26,6 +26,10 @@ #include "include/denc.h" +#include <boost/container/flat_map.hpp> +#include <boost/container/flat_set.hpp> +#include <boost/optional.hpp> + using namespace std; // test helpers diff --git a/src/test/test_ipaddr.cc b/src/test/test_ipaddr.cc index 49038815318..21df1d4056b 100644 --- a/src/test/test_ipaddr.cc +++ b/src/test/test_ipaddr.cc @@ -995,3 +995,158 @@ TEST(pick_address, ipv4_ipv6_enabled2) ASSERT_EQ(-1, r); } } + +// Test for IPv4 address +TEST(is_addr_in_subnet, ipv4) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("10.1.1.2", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(true, r); +} + +// Test for IPv6 address +TEST(is_addr_in_subnet, ipv6) +{ + std::string public_network = "2001:db8::/64"; + entity_addr_t addr; + addr.parse("2001:db8::1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv6", "true"); + cct->_conf.set_val("ms_bind_ipv4", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(true, r); +} + +// Test for invalid address +TEST(is_addr_in_subnet, invalid_address) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("192.168.1.1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(false, r); +} + +// Test for malformed address +TEST(is_addr_in_subnet, malformed_address) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr; + addr.parse("invalid_address", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + // Test with a malformed address + bool r = is_addr_in_subnet(cct.get(), public_network, addr); + ASSERT_EQ(false, r); +} + +TEST(is_addr_in_subnet, boundary_ipv4) +{ + std::string public_network = "10.1.1.0/24"; + entity_addr_t addr_low; + addr_low.parse("10.1.1.0", nullptr); + entity_addr_t addr_high; + addr_high.parse("10.1.1.255", nullptr); + entity_addr_t addr_out; + addr_out.parse("10.1.2.0", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_low)); + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_high)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network, addr_out)); +} + +TEST(is_addr_in_subnet, boundary_ipv6) +{ + std::string public_network = "2001:db8::/64"; + entity_addr_t addr_low; + addr_low.parse("2001:db8::", nullptr); + entity_addr_t addr_high; + addr_high.parse("2001:db8:0:0:ffff:ffff:ffff:ffff", nullptr); + entity_addr_t addr_out; + addr_out.parse("2001:db9::", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv6", "true"); + cct->_conf.set_val("ms_bind_ipv4", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_low)); + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network, addr_high)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network, addr_out)); +} + +TEST(is_addr_in_subnet, overlapping_subnets) +{ + std::string public_network_1 = "10.1.1.0/24"; + std::string public_network_2 = "10.1.2.0/24"; + entity_addr_t addr; + addr.parse("10.1.1.5", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "false"); + + ASSERT_TRUE(is_addr_in_subnet(cct.get(), public_network_1, addr)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr)); +} + +TEST(is_addr_in_subnet, mismatched_family) +{ + std::string public_network_1 = "2001:db8::/64"; + entity_addr_t addr_1; + addr_1.parse("10.1.1.5", nullptr); + + std::string public_network_2 = "10.1.1.0/24"; + entity_addr_t addr_2; + addr_2.parse("2001:db8::1", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + cct->_conf.set_val("ms_bind_ipv4", "true"); + cct->_conf.set_val("ms_bind_ipv6", "true"); + + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_1, addr_1)); + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr_2)); +} + +TEST(is_addr_in_subnet, invalid_subnets) +{ + std::string public_network_1 = "10.1.1.0/33"; + std::string public_network_2 = "25.0.0.99/10"; + entity_addr_t addr; + addr.parse("10.1.1.2", nullptr); + + boost::intrusive_ptr<CephContext> cct(new CephContext(CEPH_ENTITY_TYPE_OSD), false); + cct->_conf._clear_safe_to_start_threads(); + + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_1, addr)); // Invalid prefix + ASSERT_FALSE(is_addr_in_subnet(cct.get(), public_network_2, addr)); // Invalid subnet string +} + diff --git a/src/tools/ceph-dencoder/sstring.h b/src/tools/ceph-dencoder/sstring.h index c2493c10efa..829a0eb307f 100644 --- a/src/tools/ceph-dencoder/sstring.h +++ b/src/tools/ceph-dencoder/sstring.h @@ -7,7 +7,7 @@ class sstring_wrapper { using sstring16 = basic_sstring<char, uint32_t, 16>; sstring16 s1; - using sstring24 = basic_sstring<unsigned char, uint16_t, 24>; + using sstring24 = basic_sstring<char8_t, uint16_t, 24>; sstring24 s2; public: sstring_wrapper() = default; diff --git a/src/tools/cephfs/top/cephfs-top b/src/tools/cephfs/top/cephfs-top index 9ecc47fc2d5..45900f9a025 100755 --- a/src/tools/cephfs/top/cephfs-top +++ b/src/tools/cephfs/top/cephfs-top @@ -148,7 +148,7 @@ def wrap(s, sl): """return a '+' suffixed wrapped string""" if len(s) < sl: return s - return f'{s[0:sl-1]}+' + return f'{s[0:sl - 1]}+' class FSTopBase(object): diff --git a/src/tools/cephfs_mirror/PeerReplayer.cc b/src/tools/cephfs_mirror/PeerReplayer.cc index 91117cf5f2b..77e93ef6a99 100644 --- a/src/tools/cephfs_mirror/PeerReplayer.cc +++ b/src/tools/cephfs_mirror/PeerReplayer.cc @@ -120,7 +120,9 @@ int opendirat(MountRef mnt, int dirfd, const std::string &relpath, int flags, int fd = r; r = ceph_fdopendir(mnt, fd, dirp); - ceph_close(mnt, fd); + if (r < 0) { + ceph_close(mnt, fd); + } return r; } @@ -1222,15 +1224,6 @@ int PeerReplayer::sync_perms(const std::string& path) { return 0; } -void PeerReplayer::post_sync_close_handles(const FHandles &fh) { - dout(20) << dendl; - - // @FHandles.r_fd_dir_root is closed in @unregister_directory since - // its used to acquire an exclusive lock on remote dir_root. - ceph_close(m_local_mount, fh.c_fd); - ceph_close(fh.p_mnt, fh.p_fd); -} - int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot ¤t) { dout(20) << ": dir_root=" << dir_root << ", current=" << current << dendl; FHandles fh; @@ -1240,10 +1233,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu return r; } - BOOST_SCOPE_EXIT_ALL( (this)(&fh) ) { - post_sync_close_handles(fh); - }; - // record that we are going to "dirty" the data under this // directory root auto snap_id_str{stringify(current.second)}; @@ -1252,6 +1241,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": error setting \"ceph.mirror.dirty_snap_id\" on dir_root=" << dir_root << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1263,6 +1254,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to stat snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1271,8 +1264,12 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to open local snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } + // starting from this point we shouldn't care about manual closing of fh.c_fd, + // it will be closed automatically when bound tdirp is closed. std::stack<SyncEntry> sync_stack; sync_stack.emplace(SyncEntry(".", tdirp, tstx)); @@ -1282,12 +1279,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu break; } - r = pre_sync_check_and_open_handles(dir_root, current, boost::none, &fh); - if (r < 0) { - dout(5) << ": cannot proceed with sync: " << cpp_strerror(r) << dendl; - return r; - } - dout(20) << ": " << sync_stack.size() << " entries in stack" << dendl; std::string e_name; auto &entry = sync_stack.top(); @@ -1390,6 +1381,18 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu sync_stack.pop(); } + dout(20) << " cur:" << fh.c_fd + << " prev:" << fh.p_fd + << " ret = " << r + << dendl; + + // @FHandles.r_fd_dir_root is closed in @unregister_directory since + // its used to acquire an exclusive lock on remote dir_root. + + // c_fd has been used in ceph_fdopendir call so + // there is no need to close this fd manually. + ceph_close(fh.p_mnt, fh.p_fd); + return r; } @@ -1409,9 +1412,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu return r; } - BOOST_SCOPE_EXIT_ALL( (this)(&fh) ) { - post_sync_close_handles(fh); - }; // record that we are going to "dirty" the data under this directory root auto snap_id_str{stringify(current.second)}; @@ -1420,6 +1420,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": error setting \"ceph.mirror.dirty_snap_id\" on dir_root=" << dir_root << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1431,6 +1433,8 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu if (r < 0) { derr << ": failed to stat snap=" << current.first << ": " << cpp_strerror(r) << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1450,11 +1454,6 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu dout(0) << ": backing off r=" << r << dendl; break; } - r = pre_sync_check_and_open_handles(dir_root, current, prev, &fh); - if (r < 0) { - dout(5) << ": cannot proceed with sync: " << cpp_strerror(r) << dendl; - return r; - } dout(20) << ": " << sync_queue.size() << " entries in queue" << dendl; const auto &queue_entry = sync_queue.front(); @@ -1464,12 +1463,16 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu stringify((*prev).first).c_str(), current.first.c_str(), &sd_info); if (r != 0) { derr << ": failed to open snapdiff, r=" << r << dendl; + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } while (0 < (r = ceph_readdir_snapdiff(&sd_info, &sd_entry))) { if (r < 0) { derr << ": failed to read directory=" << epath << dendl; ceph_close_snapdiff(&sd_info); + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } @@ -1561,6 +1564,17 @@ int PeerReplayer::do_synchronize(const std::string &dir_root, const Snapshot &cu } sync_queue.pop(); } + + dout(20) << " current:" << fh.c_fd + << " prev:" << fh.p_fd + << " ret = " << r + << dendl; + + // @FHandles.r_fd_dir_root is closed in @unregister_directory since + // its used to acquire an exclusive lock on remote dir_root. + + ceph_close(m_local_mount, fh.c_fd); + ceph_close(fh.p_mnt, fh.p_fd); return r; } diff --git a/src/tools/cephfs_mirror/PeerReplayer.h b/src/tools/cephfs_mirror/PeerReplayer.h index 933cb182635..32c71301f00 100644 --- a/src/tools/cephfs_mirror/PeerReplayer.h +++ b/src/tools/cephfs_mirror/PeerReplayer.h @@ -307,7 +307,6 @@ private: int open_dir(MountRef mnt, const std::string &dir_path, boost::optional<uint64_t> snap_id); int pre_sync_check_and_open_handles(const std::string &dir_root, const Snapshot ¤t, boost::optional<Snapshot> prev, FHandles *fh); - void post_sync_close_handles(const FHandles &fh); int do_synchronize(const std::string &dir_root, const Snapshot ¤t, boost::optional<Snapshot> prev); diff --git a/src/tools/monmaptool.cc b/src/tools/monmaptool.cc index f1b86e00362..dc882a006a2 100644 --- a/src/tools/monmaptool.cc +++ b/src/tools/monmaptool.cc @@ -375,6 +375,10 @@ int main(int argc, const char **argv) return r; } + if (handle_features(features, monmap)) { + modified = true; + } + if (min_mon_release != ceph_release_t::unknown) { monmap.min_mon_release = min_mon_release; cout << "setting min_mon_release = " << min_mon_release << std::endl; @@ -459,10 +463,6 @@ int main(int argc, const char **argv) monmap.remove(p); } - if (handle_features(features, monmap)) { - modified = true; - } - if (!print && !modified && !show_features) { cerr << "no action specified" << std::endl; helpful_exit(); diff --git a/src/tools/rbd/Utils.cc b/src/tools/rbd/Utils.cc index 95c8725aa33..b20dca05bc6 100644 --- a/src/tools/rbd/Utils.cc +++ b/src/tools/rbd/Utils.cc @@ -337,11 +337,14 @@ int get_pool_image_snapshot_names(const po::variables_map &vm, SpecValidation spec_validation) { std::string pool_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_POOL_NAME : at::POOL_NAME); + std::string namespace_key = (mod == at::ARGUMENT_MODIFIER_DEST ? + at::DEST_NAMESPACE_NAME : at::NAMESPACE_NAME); std::string image_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_IMAGE_NAME : at::IMAGE_NAME); + return get_pool_generic_snapshot_names(vm, mod, spec_arg_index, pool_key, - pool_name, namespace_name, image_key, - "image", image_name, snap_name, + pool_name, namespace_key, namespace_name, + image_key, "image", image_name, snap_name, image_name_required, snapshot_presence, spec_validation); } @@ -351,6 +354,7 @@ int get_pool_generic_snapshot_names(const po::variables_map &vm, size_t *spec_arg_index, const std::string& pool_key, std::string *pool_name, + const std::string& namespace_key, std::string *namespace_name, const std::string& generic_key, const std::string& generic_key_desc, @@ -359,8 +363,6 @@ int get_pool_generic_snapshot_names(const po::variables_map &vm, bool generic_name_required, SnapshotPresence snapshot_presence, SpecValidation spec_validation) { - std::string namespace_key = (mod == at::ARGUMENT_MODIFIER_DEST ? - at::DEST_NAMESPACE_NAME : at::NAMESPACE_NAME); std::string snap_key = (mod == at::ARGUMENT_MODIFIER_DEST ? at::DEST_SNAPSHOT_NAME : at::SNAPSHOT_NAME); diff --git a/src/tools/rbd/Utils.h b/src/tools/rbd/Utils.h index 5076fd7fe9c..6aa0f2fdbdf 100644 --- a/src/tools/rbd/Utils.h +++ b/src/tools/rbd/Utils.h @@ -163,10 +163,11 @@ int get_pool_generic_snapshot_names( const boost::program_options::variables_map &vm, argument_types::ArgumentModifier mod, size_t *spec_arg_index, const std::string& pool_key, std::string *pool_name, - std::string *namespace_name, const std::string& generic_key, - const std::string& generic_key_desc, std::string *generic_name, - std::string *snap_name, bool generic_name_required, - SnapshotPresence snapshot_presence, SpecValidation spec_validation); + const std::string& namespace_key, std::string *namespace_name, + const std::string& generic_key, const std::string& generic_key_desc, + std::string *generic_name, std::string *snap_name, + bool generic_name_required, SnapshotPresence snapshot_presence, + SpecValidation spec_validation); int get_pool_image_id(const boost::program_options::variables_map &vm, size_t *spec_arg_index, diff --git a/src/tools/rbd/action/Group.cc b/src/tools/rbd/action/Group.cc index d97e120d438..100bdc19496 100644 --- a/src/tools/rbd/action/Group.cc +++ b/src/tools/rbd/action/Group.cc @@ -28,6 +28,9 @@ static const std::string DEST_GROUP_NAME("dest-group"); static const std::string GROUP_POOL_NAME("group-" + at::POOL_NAME); static const std::string IMAGE_POOL_NAME("image-" + at::POOL_NAME); +static const std::string GROUP_NAMESPACE_NAME("group-" + at::NAMESPACE_NAME); +static const std::string IMAGE_NAMESPACE_NAME("image-" + at::NAMESPACE_NAME); + void add_group_option(po::options_description *opt, at::ArgumentModifier modifier) { std::string name = GROUP_NAME; @@ -107,8 +110,8 @@ int execute_create(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -187,8 +190,8 @@ int execute_remove(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -221,8 +224,8 @@ int execute_rename(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -233,9 +236,9 @@ int execute_rename(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_DEST, &arg_index, at::DEST_POOL_NAME, - &dest_pool_name, &dest_namespace_name, DEST_GROUP_NAME, "group", - &dest_group_name, nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &dest_pool_name, at::DEST_NAMESPACE_NAME, &dest_namespace_name, + DEST_GROUP_NAME, "group", &dest_group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -283,8 +286,8 @@ int execute_info(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -335,8 +338,9 @@ int execute_add(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, GROUP_POOL_NAME, - &group_pool_name, &group_namespace_name, GROUP_NAME, "group", &group_name, - nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + &group_pool_name, GROUP_NAMESPACE_NAME, &group_namespace_name, + GROUP_NAME, "group", &group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -347,9 +351,9 @@ int execute_add(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, IMAGE_POOL_NAME, - &image_pool_name, &image_namespace_name, at::IMAGE_NAME, "image", - &image_name, nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &image_pool_name, IMAGE_NAMESPACE_NAME, &image_namespace_name, + at::IMAGE_NAME, "image", &image_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -393,8 +397,9 @@ int execute_remove_image(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, GROUP_POOL_NAME, - &group_pool_name, &group_namespace_name, GROUP_NAME, "group", &group_name, - nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + &group_pool_name, GROUP_NAMESPACE_NAME, &group_namespace_name, + GROUP_NAME, "group", &group_name, nullptr, true, + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -410,9 +415,9 @@ int execute_remove_image(const po::variables_map &vm, r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, IMAGE_POOL_NAME, - &image_pool_name, &image_namespace_name, at::IMAGE_NAME, "image", - &image_name, nullptr, image_id.empty(), utils::SNAPSHOT_PRESENCE_NONE, - utils::SPEC_VALIDATION_FULL); + &image_pool_name, IMAGE_NAMESPACE_NAME, &image_namespace_name, + at::IMAGE_NAME, "image", &image_name, nullptr, image_id.empty(), + utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -464,8 +469,8 @@ int execute_list_images(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -563,8 +568,9 @@ int execute_group_snap_create(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -604,8 +610,9 @@ int execute_group_snap_remove(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -640,8 +647,9 @@ int execute_group_snap_rename(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &source_snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &source_snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -696,8 +704,8 @@ int execute_group_snap_list(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, nullptr, true, - utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + nullptr, true, utils::SNAPSHOT_PRESENCE_NONE, utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -764,8 +772,9 @@ int execute_group_snap_info(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &group_snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &group_snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -872,8 +881,9 @@ int execute_group_snap_rollback(const po::variables_map &vm, int r = utils::get_pool_generic_snapshot_names( vm, at::ARGUMENT_MODIFIER_NONE, &arg_index, at::POOL_NAME, &pool_name, - &namespace_name, GROUP_NAME, "group", &group_name, &snap_name, true, - utils::SNAPSHOT_PRESENCE_REQUIRED, utils::SPEC_VALIDATION_FULL); + at::NAMESPACE_NAME, &namespace_name, GROUP_NAME, "group", &group_name, + &snap_name, true, utils::SNAPSHOT_PRESENCE_REQUIRED, + utils::SPEC_VALIDATION_FULL); if (r < 0) { return r; } @@ -954,9 +964,6 @@ void get_add_arguments(po::options_description *positional, add_prefixed_pool_option(options, "image"); add_prefixed_namespace_option(options, "image"); at::add_image_option(options, at::ARGUMENT_MODIFIER_NONE); - - at::add_pool_option(options, at::ARGUMENT_MODIFIER_NONE, - " unless overridden"); } void get_remove_image_arguments(po::options_description *positional, @@ -979,8 +986,6 @@ void get_remove_image_arguments(po::options_description *positional, add_prefixed_namespace_option(options, "image"); at::add_image_option(options, at::ARGUMENT_MODIFIER_NONE); - at::add_pool_option(options, at::ARGUMENT_MODIFIER_NONE, - " unless overridden"); at::add_image_id_option(options); } diff --git a/src/tools/rbd/action/MirrorPool.cc b/src/tools/rbd/action/MirrorPool.cc index 58e2d4dc329..6a546c3f73a 100644 --- a/src/tools/rbd/action/MirrorPool.cc +++ b/src/tools/rbd/action/MirrorPool.cc @@ -355,6 +355,10 @@ protected: virtual ~ImageRequestBase() { } + virtual bool open_read_only() const { + return false; + } + virtual bool skip_get_info() const { return false; } @@ -429,8 +433,13 @@ private: librbd::RBD rbd; auto aio_completion = utils::create_aio_completion< ImageRequestBase, &ImageRequestBase::handle_open_image>(this); - rbd.aio_open(m_io_ctx, m_image, m_image_name.c_str(), nullptr, - aio_completion); + if (open_read_only()) { + rbd.aio_open_read_only(m_io_ctx, m_image, m_image_name.c_str(), nullptr, + aio_completion); + } else { + rbd.aio_open(m_io_ctx, m_image, m_image_name.c_str(), nullptr, + aio_completion); + } } void handle_open_image(int r) { @@ -604,6 +613,10 @@ public: } protected: + bool open_read_only() const override { + return true; + } + bool skip_get_info() const override { return true; } diff --git a/src/vstart.sh b/src/vstart.sh index 41a8310891b..a992f33c856 100755 --- a/src/vstart.sh +++ b/src/vstart.sh @@ -159,6 +159,7 @@ smallmds=0 short=0 crimson=0 ec=0 +cephexporter=0 cephadm=0 parallel=true restart=1 @@ -233,6 +234,7 @@ options: -G disable Kerberos/GSSApi authentication --hitset <pool> <hit_set_type>: enable hitset tracking -e : create an erasure pool + --cephexporter: start the ceph-exporter daemon -o config add extra config parameters to all sections --rgw_port specify ceph rgw http listen port --rgw_frontend specify the rgw frontend configuration @@ -372,6 +374,9 @@ case $1 in -e) ec=1 ;; + --cephexporter) + cephexporter=1 + ;; --new | -n) new=1 ;; @@ -963,7 +968,17 @@ $BLUESTORE_OPTS ; kstore kstore fsck on mount = true +EOF + if [ "$crimson" -eq 1 ]; then + wconf <<EOF + crimson osd objectstore = $objectstore +EOF + else + wconf <<EOF osd objectstore = $objectstore +EOF + fi + wconf <<EOF $SEASTORE_OPTS $COSDSHORT $(format_conf "${extra_conf}") @@ -1130,6 +1145,17 @@ EOF fi } +start_cephexporter() { + debug echo "Starting Ceph exporter daemon..." + + # Define socket directory for the exporter + # Start the exporter daemon + prunb ceph-exporter \ + -c "$conf_fn" \ + --sock-dir "$CEPH_ASOK_DIR" \ + --addrs "$IP" +} + start_osd() { if [ $inc_osd_num -gt 0 ]; then old_maxosd=$($CEPH_BIN/ceph osd getmaxosd | sed -e 's/max_osd = //' -e 's/ in epoch.*//') @@ -1676,28 +1702,30 @@ if [ "$ceph_osd" == "crimson-osd" ]; then if [ "$trace" -ne 0 ]; then extra_seastar_args=" --trace" fi - if [ "$(expr $(nproc) - 1)" -gt "$(($CEPH_NUM_OSD * crimson_smp))" ]; then - if [ $crimson_alien_num_cores -gt 0 ]; then - alien_bottom_cpu=$(($CEPH_NUM_OSD * crimson_smp)) - alien_top_cpu=$(( alien_bottom_cpu + crimson_alien_num_cores - 1 )) - # Ensure top value within range: - if [ "$(($alien_top_cpu))" -gt "$(expr $(nproc) - 1)" ]; then - alien_top_cpu=$(expr $(nproc) - 1) + if [ "$objectstore" == "bluestore" ]; then + if [ "$(expr $(nproc) - 1)" -gt "$(($CEPH_NUM_OSD * crimson_smp))" ]; then + if [ $crimson_alien_num_cores -gt 0 ]; then + alien_bottom_cpu=$(($CEPH_NUM_OSD * crimson_smp)) + alien_top_cpu=$(( alien_bottom_cpu + crimson_alien_num_cores - 1 )) + # Ensure top value within range: + if [ "$(($alien_top_cpu))" -gt "$(expr $(nproc) - 1)" ]; then + alien_top_cpu=$(expr $(nproc) - 1) + fi + echo "crimson_alien_thread_cpu_cores: $alien_bottom_cpu-$alien_top_cpu" + # This is a (logical) processor id range, it could be refined to encompass only physical processor ids + # (equivalently, ignore hyperthreading sibling processor ids) + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores "$alien_bottom_cpu-$alien_top_cpu" + else + echo "crimson_alien_thread_cpu_cores:" $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" + fi + if [ $crimson_alien_num_threads -gt 0 ]; then + echo "$CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads $crimson_alien_num_threads" + $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads "$crimson_alien_num_threads" fi - echo "crimson_alien_thread_cpu_cores: $alien_bottom_cpu-$alien_top_cpu" - # This is a (logical) processor id range, it could be refined to encompass only physical processor ids - # (equivalently, ignore hyperthreading sibling processor ids) - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores "$alien_bottom_cpu-$alien_top_cpu" else - echo "crimson_alien_thread_cpu_cores:" $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_thread_cpu_cores $(($CEPH_NUM_OSD * crimson_smp))-"$(expr $(nproc) - 1)" - fi - if [ $crimson_alien_num_threads -gt 0 ]; then - echo "$CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads $crimson_alien_num_threads" - $CEPH_BIN/ceph -c $conf_fn config set osd crimson_alien_op_num_threads "$crimson_alien_num_threads" + echo "No alien thread cpu core isolation" fi - else - echo "No alien thread cpu core isolation" fi fi @@ -1726,6 +1754,10 @@ if [ $CEPH_NUM_MDS -gt 0 ]; then ceph_adm fs authorize \* "client.fs" / rwp >> "$keyring_fn" fi +if [ "$cephexporter" -eq 1 ]; then + start_cephexporter +fi + # Don't set max_mds until all the daemons are started, otherwise # the intended standbys might end up in active roles. if [ "$CEPH_MAX_MDS" -gt 1 ]; then |
