diff options
58 files changed, 2294 insertions, 1443 deletions
diff --git a/doc/cephadm/services/smb.rst b/doc/cephadm/services/smb.rst index 7b6a31f1c87..cc36a61b9d5 100644 --- a/doc/cephadm/services/smb.rst +++ b/doc/cephadm/services/smb.rst @@ -24,7 +24,7 @@ Samba Containers with the following command: .. prompt:: bash # - orch apply smb <cluster_id> <config_uri> [--features ...] [--placement ...] ... + ceph orch apply smb <cluster_id> <config_uri> [--features ...] [--placement ...] ... There are a number of additional parameters that the command accepts. See the Service Specification for a description of these options. diff --git a/doc/cephfs/fs-volumes.rst b/doc/cephfs/fs-volumes.rst index 34cb7917d1b..3e7dd7815ad 100644 --- a/doc/cephfs/fs-volumes.rst +++ b/doc/cephfs/fs-volumes.rst @@ -14,12 +14,12 @@ abstractions: * FS volumes, an abstraction for CephFS file systems -* FS subvolumes, an abstraction for independent CephFS directory trees - * FS subvolume groups, an abstraction for a directory level higher than FS subvolumes. Used to effect policies (e.g., :doc:`/cephfs/file-layouts`) across a set of subvolumes +* FS subvolumes, an abstraction for independent CephFS directory trees + Possible use-cases for the export abstractions: * FS subvolumes used as Manila shares or CSI volumes diff --git a/doc/dev/kclient.rst b/doc/dev/kclient.rst new file mode 100644 index 00000000000..fd4903ac1ab --- /dev/null +++ b/doc/dev/kclient.rst @@ -0,0 +1,478 @@ +Testing changes to the Linux Kernel CephFS driver +================================================= + +This walkthrough will explain one (opinionated) way to do testing of the Linux +kernel client against a development cluster. We will try to mimimize any +assumptions about pre-existing knowledge of how to do kernel builds or any +related best-practices. + +.. note:: There are many completely valid ways to do kernel development for + Ceph. This guide is a walkthrough of the author's own environment. + You may decide to do things very differently. + +Step One: build the kernel +========================== + +Clone the kernel: + +.. code-block:: bash + + git init linux && cd linux + git remote add torvalds git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git + git remote add ceph https://github.com/ceph/ceph-client.git + git fetch && git checkout torvalds/master + + +Configure the kernel: + +.. code-block:: bash + + make defconfig + +.. note:: You can alternatively use the `Ceph Kernel QA Config`_ for building the kernel. + +We now have a kernel config with reasonable defaults for the architecture you're +building on. The next thing to do is to enable configs which will build Ceph and/or +provide functionality we need to do testing. + +.. code-block:: bash + + cat > ~/.ceph.config <<EOF + CONFIG_CEPH_FS=y + CONFIG_CEPH_FSCACHE=y + CONFIG_CEPH_FS_POSIX_ACL=y + CONFIG_CEPH_FS_SECURITY_LABEL=y + CONFIG_CEPH_LIB_PRETTYDEBUG=y + CONFIG_DYNAMIC_DEBUG=y + CONFIG_DYNAMIC_DEBUG_CORE=y + CONFIG_FRAME_POINTER=y + CONFIG_FSCACHE + CONFIG_FSCACHE_STATS + CONFIG_FS_ENCRYPTION=y + CONFIG_FS_ENCRYPTION_ALGS=y + CONFIG_KGDB=y + CONFIG_KGDB_SERIAL_CONSOLE=y + CONFIG_XFS_FS=y + EOF + +Beyond enabling Ceph-related configs, we are also enabling some useful +debug configs and XFS (as an alternative to ext4 if needed for our root file +system). + +.. note:: It is a good idea to not build anything as a kernel module. Otherwise, you would need to ``make modules_install`` on the root drive of the VM. + +Now, merge the configs. + + +.. code-block:: bash + + + scripts/kconfig/merge_config.sh .config ~/.ceph.config + + +Finally, build the kernel: + +.. code-block:: bash + + make -j + + +.. note:: This document does not discuss how to get relevant utilities for your + distribution to actually build the kernel, like gcc. Please use your search + engine of choice to learn how to do that. + + +Step Two: create a VM +===================== + +A virtual machine is a good choice for testing the kernel client for a few reasons: + +* You can more easily monitor and configure networking for the VM. +* You can very rapidly test a change to the kernel (build -> mount in less than 10 seconds). +* A fault in the kernel won't crash your machine. +* You have a suite of tools available for analysis on the running kernel. + +The main decision for you to make is what Linux distribution you want to use. +This document uses Arch Linux due to the author's familiarity. We also use LVM +to create a volume. You may use partitions or whatever mechanism you like to +create a block device. In general, this block device will be used repeatedly in +testing. You may want to use snapshots to avoid a VM somehow corrupting your +root disk and forcing you to start over. + + +.. code-block:: bash + + # create a volume + VOLUME_GROUP=foo + sudo lvcreate -L 256G "$VOLUME_GROUP" -n $(whoami)-vm-0 + DEV="/dev/${VOLUME_GROUP}/$(whoami)-vm-0" + sudo mkfs.xfs "$DEV" + sudo mount "$DEV" /mnt + sudo pacstrap /mnt base base-devel vim less jq + sudo arch-chroot /mnt + # # delete root's password for ease of login + # passwd -d root + # mkdir -p /root/.ssh && echo "$YOUR_SSH_KEY_PUBKEY" >> /root/.ssh/authorized_keys + # exit + sudo umount /mnt + +Once that's done, we should be able to run a VM: + + +.. code-block:: bash + + qemu-system-x86_64 -enable-kvm -kernel $(pwd)/arch/x86/boot/bzImage -drive file="$DEV",if=virtio,format=raw -append 'root=/dev/vda rw' + +You should see output like: + +:: + + VNC server running on ::1:5900 + +You could view that console using: + + +.. code-block:: bash + + vncviewer 127.0.0.1:5900 + +Congratulations, you have a VM running the kernel that you just built. + + +Step Three: Networking the VM +============================= + +This is the "hard part" and requires the most customization depending on what +you want to do. For this author, I currently have a development setup like: + + +:: + + sepian netns + ______________ + | | + | kernel VM | sepia-bounce VM vossi04.front.sepia.ceph.com + | ------- | | ------ ------- + | | | | | 192.168.20.1 | | | | + | | |--|--|- <- wireguard -> | | <-- sepia vpn -> | | + | |_____| | | 192.168.20.2 |____| |_____| + | br0 | + |______________| + + +The sepia-bounce VM is used as a bounce box to the sepia lab. It can proxy ssh +connections, route any sepia-bound traffic, or serve as a DNS proxy. The use of +a sepia-bounce VM is optional but can be useful, especially if you want to +create numerous kernel VMs for testing. + +I like to use the vossi04 `developer playground`_ to build Ceph and setup a +vstart cluster. It has sufficient resources to make building Ceph very fast +(~5 minutes cold build) and local disk resources to run a decent vstart +cluster. + +To avoid overcomplicating this document with the details of the sepia-bounce +VM, I will note the following main configurations used for the purpose of +testing the kernel: + +- setup a wireguard tunnel between the machine creating kernel VMs and the sepia-bounce VM +- use ``systemd-resolved`` as a DNS resolver and listen on 192.168.20.2 (instead of just localhost) +- connect to the sepia `VPN`_ and use `systemd resolved update script`_ to configure ``systemd-resolved`` to use the DNS servers acquired via DHCP from the sepia VPN +- configure ``firewalld`` to allow wireguard traffic and to masquerade and forward traffic to the sepia vpn + +The next task is to connect the kernel VM to the sepia-bounce VM. A network +namespace can be useful for this purpose to isolate traffic / routing rules for +the VMs. For me, I orchestrate this using a custom systemd one-shot unit that +looks like: + +:: + + # create the net namespace + ExecStart=/usr/bin/ip netns add sepian + # bring lo up + ExecStart=/usr/bin/ip netns exec sepian ip link set dev lo up + # setup wireguard to sepia-bounce + ExecStart=/usr/bin/ip link add wg-sepian type wireguard + ExecStart=/usr/bin/wg setconf wg-sepian /etc/wireguard/wg-sepian.conf + # move the wireguard interface to the sepian nents + ExecStart=/usr/bin/ip link set wg-sepian netns sepian + # configure the static ip and bring it up + ExecStart=/usr/bin/ip netns exec sepian ip addr add 192.168.20.1/24 dev wg-sepian + ExecStart=/usr/bin/ip netns exec sepian ip link set wg-sepian up + # logging info + ExecStart=/usr/bin/ip netns exec sepian ip addr + ExecStart=/usr/bin/ip netns exec sepian ip route + # make wireguard the default route + ExecStart=/usr/bin/ip netns exec sepian ip route add default via 192.168.20.2 dev wg-sepian + # more logging + ExecStart=/usr/bin/ip netns exec sepian ip route + # add a bridge interface for VMs + ExecStart=/usr/bin/ip netns exec sepian ip link add name br0 type bridge + # configure the addresses and bring it up + ExecStart=/usr/bin/ip netns exec sepian ip addr add 192.168.0.1/24 dev br0 + ExecStart=/usr/bin/ip netns exec sepian ip link set br0 up + # masquerade/forward traffic to sepia-bounce + ExecStart=/usr/bin/ip netns exec sepian iptables -t nat -A POSTROUTING -o wg-sepian -j MASQUERADE + + +When using the network namespace, we will use ``ip netns exec``. There is a +handy feature to automatically bind mount files into the ``/etc`` namespace for +commands run via that command: + +:: + + # cat /etc/netns/sepian/resolv.conf + nameserver 192.168.20.2 + +That file will configure the libc name resolution stack to route DNS requests +for applications to the ``systemd-resolved`` daemon running on sepia-bounce. +Consequently, any application running in that netns will be able to resolve +sepia hostnames: + +:: + + $ sudo ip netns exec sepian host vossi04.front.sepia.ceph.com + vossi04.front.sepia.ceph.com has address 172.21.10.4 + + +Okay, great. We have a network namespace that forwards traffic to the sepia +VPN. The next mental step is to connect virtual machines running a kernel to +the bridge we have configured. The straightforward way to do that is to create +a "tap" device which connects to the bridge: + +.. code-block:: bash + + sudo ip netns exec sepian qemu-system-x86_64 \ + -enable-kvm \ + -kernel $(pwd)/arch/x86/boot/bzImage \ + -drive file="$DEV",if=virtio,format=raw \ + -netdev tap,id=net0,ifname=tap0,script="$HOME/bin/qemu-br0",downscript=no \ + -device virtio-net-pci,netdev=net0 \ + -append 'root=/dev/vda rw' + +The new relevant bits here are (a) executing the VM in the netns we have +constructed; (b) a ``-netdev`` command to configure a tap device; (c) a +virtual network card for the VM. There is also a script ``$HOME/bin/qemu-br0`` +run by qemu to configure the tap device it creates for the VM: + +:: + + #!/bin/bash + tap=$1 + ip link set "$tap" master br0 + ip link set dev "$tap" up + +That simply plugs the new tap device into the bridge. + +This is all well and good but we are now missing one last crucial step. What is +the IP address of the VM? There are two options: + +1. configure a static IP but the VM's root device networking stack + configuration must be modified +2. use DHCP and configure the root device for VMs to always use dhcp to + configure their ethernet device addresses + +The second option is more complicated to setup, since you must run a DHCP +server now, but provides the greatest flexibility for adding more VMs as needed +when testing. + +The modified (or "hacked") standard dhcpd systemd service looks like: + +:: + + # cat sepian-dhcpd.service + [Unit] + Description=IPv4 DHCP server + After=network.target network-online.target sepian-netns.service + Wants=network-online.target + Requires=sepian-netns.service + + [Service] + ExecStartPre=/usr/bin/touch /tmp/dhcpd.leases + ExecStartPre=/usr/bin/cat /etc/netns/sepian/dhcpd.conf + ExecStart=/usr/bin/dhcpd -f -4 -q -cf /etc/netns/sepian/dhcpd.conf -lf /tmp/dhcpd.leases + NetworkNamespacePath=/var/run/netns/sepian + RuntimeDirectory=dhcpd4 + User=dhcp + AmbientCapabilities=CAP_NET_BIND_SERVICE CAP_NET_RAW + ProtectSystem=full + ProtectHome=on + KillSignal=SIGINT + # We pull in network-online.target for a configured network connection. + # However this is not guaranteed to be the network connection our + # networks are configured for. So try to restart on failure with a delay + # of two seconds. Rate limiting kicks in after 12 seconds. + RestartSec=2s + Restart=on-failure + StartLimitInterval=12s + + [Install] + WantedBy=multi-user.target + +Similarly, the referenced dhcpd.conf: + +:: + + # cat /etc/netns/sepian/dhcpd.conf + option domain-name-servers 192.168.20.2; + option subnet-mask 255.255.255.0; + option routers 192.168.0.1; + subnet 192.168.0.0 netmask 255.255.255.0 { + range 192.168.0.100 192.168.0.199; + } + +Importantly, this tells the VM to route traffic to 192.168.0.1 (the IP of the +bridge in the netns) and DNS can be provided by 192.168.20.2 (via +``systemd-resolved`` on the sepia-bounce VM). + +In the VM, the networking looks like: + +:: + + [root@archlinux ~]# ip link + 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000 + link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff + 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 + link/sit 0.0.0.0 brd 0.0.0.0 + [root@archlinux ~]# ip addr + 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + inet6 ::1/128 scope host noprefixroute + valid_lft forever preferred_lft forever + 2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 + link/ether 52:54:00:12:34:56 brd ff:ff:ff:ff:ff:ff + inet 192.168.0.100/24 metric 1024 brd 192.168.0.255 scope global dynamic enp0s3 + valid_lft 28435sec preferred_lft 28435sec + inet6 fe80::5054:ff:fe12:3456/64 scope link proto kernel_ll + valid_lft forever preferred_lft forever + 3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000 + link/sit 0.0.0.0 brd 0.0.0.0 + [root@archlinux ~]# systemd-resolve --status + Global + Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported + resolv.conf mode: stub + Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google + + Link 2 (enp0s3) + Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 + Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported + Current DNS Server: 192.168.20.2 + DNS Servers: 192.168.20.2 + + Link 3 (sit0) + Current Scopes: none + Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported + + +Finally, some other networking configurations to consider: + +* Run the VM on your machine with full access to the host networking stack. If you have the sepia vpn, this will probably work without too much configuration. +* Run the VM in a netns as above but also setup the sepia vpn in the same netns. This can help to avoid using a sepia-bounce VM. You'll still need to configure routing between the bridge and the sepia VPN. +* Run the VM in a netns as above but only use a local vstart cluster (possibly in another VM) in the same netns. + + +Step Four: mounting a CephFS file system in your VM +--------------------------------------------------- + +This guide uses a vstart cluster on a machine in the sepia lab. Because the mon +addresses will change with any new vstart cluster, it will invalidate any +static configuration we may setup for our VM mounting the CephFS via the kernel +driver. So, we should create a script to fetch the configuration for our +vstart cluster prior to mounting: + +.. code-block:: bash + + #!/bin/bash + # kmount.sh -- mount a vstart Ceph cluster on a remote machine + + # the cephx client credential, vstart creates "client.fs" by default + NAME=fs + # static fs name, vstart creates an "a" file system by default + FS=a + # where to mount on the VM + MOUNTPOINT=/mnt + # cephfs mount point (root by default) + CEPHFS_MOUNTPOINT=/ + + function run { + printf '%s\n' "$*" >&2 + "$@" + } + + function mssh { + run ssh vossi04.front.sepia.ceph.com "cd ceph/build && (source vstart_environment.sh; $1)" + } + + # create the minimum config (including mon addresses) and store it in the VM's ceph.conf. This is not used for mounting; we're storing it for potential use with `ceph` commands. + mssh "ceph config generate-minimal-conf" > /etc/ceph/ceph.conf + # get the vstart cluster's fsid + FSID=$(mssh "ceph fsid") + # get the auth key associated with client.fs + KEY=$(mssh "ceph auth get-key client.$NAME") + # dump the v2 mon addresses and format for the -o mon_addr mount option + MONS=$(mssh "ceph mon dump --format=json" | jq -r '.mons[] | .public_addrs.addrvec[] | select(.type == "v2") | .addr' | paste -s -d/) + + # turn on kernel debugging (and any other debugging you'd like) + echo "module ceph +p" | tee /sys/kernel/debug/dynamic_debug/control + # do the mount! we use the new device syntax for this mount + run mount -t ceph "${NAME}@${FSID}.${FS}=${CEPHFS_MOUNTPOINT}" -o "mon_addr=${MONS},ms_mode=crc,name=${NAME},secret=${KEY},norequire_active_mds,noshare" "$MOUNTPOINT" + +That would be run like: + +.. code-block:: bash + + $ sudo ip netns exec sepian ssh root@192.168.0.100 ./kmount.sh + ... + mount -t ceph fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a=/ -o mon_addr=172.21.10.4:40762/172.21.10.4:40764/172.21.10.4:40766,ms_mode=crc,name=fs,secret=AQD0jgln43pBCxAA7cJlZ4Px7J0UmiK4A4j3rA==,norequire_active_mds,noshare /mnt + $ sudo ip netns exec sepian ssh root@192.168.0.100 df -h /mnt + Filesystem Size Used Avail Use% Mounted on + fs@c9653bca-110b-4f70-9f84-5a195b205e9a.a=/ 169G 0 169G 0% /mnt + + +If you run into difficulties, it may be: + +* The firewall on the node running the vstart cluster is blocking your connections. +* Some misconfiguration in your networking stack. +* An incorrect configuration for the mount. + + +Step Five: testing kernel changes in teuthology +----------------------------------------------- + +There 3 static branches in the `ceph kernel git repository`_ managed by the Ceph team: + +* `for-linus <https://github.com/ceph/ceph-client/tree/for-linus>`_: A branch managed by the primary Ceph maintainer to share changes with Linus Torvalds (upstream). Do not push to this branch. +* `master <https://github.com/ceph/ceph-client/tree/master>`_: A staging ground for patches planned to be sent to Linus. Do not push to this branch. +* `testing <https://github.com/ceph/ceph-client/tree/testing>`_ A staging ground for miscellaneous patches that need wider QA testing (via nightlies or regular Ceph QA testing). Push patches you believe to be nearly ready for upstream acceptance. + +You may also push a ``wip-$feature`` branch to the ``ceph-client.git`` +repository which will be built by Jenkins. Then view the results of the build +in `Shaman <https://shaman.ceph.com/builds/kernel/>`_. + +Once a kernel branch is built, you can test it via the ``fs`` CephFS QA suite: + +.. code-block:: bash + + $ teuthology-suite ... --suite fs --kernel wip-$feature --filter k-testing + + +The ``k-testing`` filter is looking for the fragment which normally sets +``testing`` branch of the kernel for routine QA. That is, the ``fs`` suite +regularly runs tests against whatever is in the ``testing`` branch of the +kernel. We are overriding that choice of kernel branch via the ``--kernel +wip-$featuree`` switch. + +.. note:: Without filtering for ``k-testing``, the ``fs`` suite will also run jobs using ceph-fuse or stock kernel, libcephfs tests, and other tests that may not be of interest to you when evaluating changes to the kernel. + +The actual override is controlled using Lua merge scripts in the +``k-testing.yaml`` fragment. See that file for more details. + + +.. _VPN: https://wiki.sepia.ceph.com/doku.php?id=vpnaccess +.. _systemd resolved update script: systemd-resolved: https://wiki.archlinux.org/title/Systemd-resolved +.. _Ceph Kernel QA Config: https://github.com/ceph/ceph-build/tree/899d0848a0f487f7e4cee773556aaf9529b8db26/kernel/build +.. _developer playground: https://wiki.sepia.ceph.com/doku.php?id=devplayground#developer_playgrounds +.. _ceph kernel git repository: https://github.com/ceph/ceph-client diff --git a/monitoring/ceph-mixin/prometheus_alerts.libsonnet b/monitoring/ceph-mixin/prometheus_alerts.libsonnet index fa2899b22c1..cde1a736f8c 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.libsonnet +++ b/monitoring/ceph-mixin/prometheus_alerts.libsonnet @@ -45,8 +45,8 @@ 'for': '30s', expr: ||| ( - (ceph_health_detail{name="MON_DOWN"} == 1) * on() ( - count(ceph_mon_quorum_status == 1) == bool (floor(count(ceph_mon_metadata) / 2) + 1) + (ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) ( + count(ceph_mon_quorum_status == 1) by(cluster)== bool (floor(count(ceph_mon_metadata) by(cluster) / 2) + 1) ) ) == 1 |||, @@ -54,22 +54,20 @@ annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down', summary: 'Monitor quorum is at risk%(cluster)s' % $.MultiClusterSummary(), - description: '{{ $min := query "floor(count(ceph_mon_metadata) / 2) + 1" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', + description: '{{ $min := printf "floor(count(ceph_mon_metadata{cluster=\'%s\'}) / 2) + 1" .Labels.cluster | query | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range printf "(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', }, }, { alert: 'CephMonDown', 'for': '30s', expr: ||| - count(ceph_mon_quorum_status == 0) <= (count(ceph_mon_metadata) - floor(count(ceph_mon_metadata) / 2) + 1) + (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1))) |||, labels: { severity: 'warning', type: 'ceph_default' }, annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down', summary: 'One or more monitors down%(cluster)s' % $.MultiClusterSummary(), - description: ||| - {{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }} - |||, + description: '{{ $down := printf "count(ceph_mon_quorum_status{cluster=\'%s\'} == 0)" .Labels.cluster | query | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range printf "(ceph_mon_quorum_status{cluster=\'%s\'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', }, }, { @@ -112,11 +110,11 @@ rules: [ { alert: 'CephOSDDownHigh', - expr: 'count(ceph_osd_up == 0) / count(ceph_osd_up) * 100 >= 10', + expr: 'count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.4.1' }, annotations: { summary: 'More than 10%% of OSDs are down%(cluster)s' % $.MultiClusterSummary(), - description: '{{ $value | humanize }}% or {{ with query "count(ceph_osd_up == 0)" }}{{ . | first | value }}{{ end }} of {{ with query "count(ceph_osd_up)" }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', + description: '{{ $value | humanize }}% or {{ with printf "count (ceph_osd_up{cluster=\'%s\'} == 0)" .Labels.cluster | query }}{{ . | first | value }}{{ end }} of {{ with printf "count (ceph_osd_up{cluster=\'%s\'})" .Labels.cluster | query }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range printf "(ceph_osd_up{cluster=\'%s\'} * on(cluster, ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', }, }, { @@ -126,7 +124,7 @@ labels: { severity: 'warning', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.4.8' }, annotations: { summary: 'An OSD host is offline%(cluster)s' % $.MultiClusterSummary(), - description: 'The following OSDs are down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0" }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}', + description: 'The following OSDs are down: {{- range printf "(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0" .Labels.cluster | query }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}', }, }, { @@ -137,9 +135,7 @@ annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down', summary: 'An OSD has been marked down%(cluster)s' % $.MultiClusterSummary(), - description: ||| - {{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s "" }}is{{ else }}are{{ end }} down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0"}} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }} - |||, + description: '{{ $num := printf "count(ceph_osd_up{cluster=\'%s\'} == 0) " .Labels.cluster | query | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s "" }}is{{ else }}are{{ end }} down: {{- range printf "(ceph_osd_up{cluster=\'%s\'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}', }, }, { @@ -235,7 +231,7 @@ labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.4.7' }, annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany', - summary: 'Too many devices are predicted to fail, unable to resolve%(cluster)s' % $.MultiClusterSummary(), + summary: 'Too many devices are predicted to fail%(cluster)s, unable to resolve' % $.MultiClusterSummary(), description: 'The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated.', }, }, @@ -298,7 +294,7 @@ labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.5.1' }, annotations: { documentation: 'https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages', - summary: 'CephFS filesystem is damaged%(cluster)s.' % $.MultiClusterSummary(), + summary: 'CephFS filesystem is damaged%(cluster)s' % $.MultiClusterSummary(), description: 'Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support.', }, }, @@ -390,7 +386,7 @@ expr: 'up{job="ceph"} == 0', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.6.2' }, annotations: { - summary: 'The mgr/prometheus module is not available%(cluster)s' % $.MultiClusterSummary(), + summary: 'The mgr/prometheus module is not available', description: "The mgr/prometheus module at {{ $labels.instance }} is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use 'ceph -s' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with 'ceph mgr module ls'. If it is not listed as enabled, enable it with 'ceph mgr module enable prometheus'.", }, }, @@ -507,7 +503,7 @@ expr: 'node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100 < 5', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.8.1' }, annotations: { - summary: 'Root filesystem is dangerously full%(cluster)s' % $.MultiClusterSummary(), + summary: 'Root filesystem is dangerously full', description: 'Root volume is dangerously full: {{ $value | humanize }}% free.', }, }, @@ -527,7 +523,7 @@ ||| % $._config, labels: { severity: 'warning', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.8.2' }, annotations: { - summary: 'One or more NICs reports packet drops%(cluster)s' % $.MultiClusterSummary(), + summary: 'One or more NICs reports packet drops', description: 'Node {{ $labels.instance }} experiences packet drop > %(CephNodeNetworkPacketDropsThreshold)s%% or > %(CephNodeNetworkPacketDropsPerSec)s packets/s on interface {{ $labels.device }}.' % { CephNodeNetworkPacketDropsThreshold: $._config.CephNodeNetworkPacketDropsThreshold * 100, CephNodeNetworkPacketDropsPerSec: $._config.CephNodeNetworkPacketDropsPerSec }, }, }, @@ -564,7 +560,7 @@ }, { alert: 'CephNodeDiskspaceWarning', - expr: 'predict_linear(node_filesystem_free_bytes{device=~"/.*"}[2d], 3600 * 24 * 5) *on(instance) group_left(nodename) node_uname_info < 0', + expr: 'predict_linear(node_filesystem_free_bytes{device=~"/.*"}[2d], 3600 * 24 * 5) * on(cluster, instance) group_left(nodename) node_uname_info < 0', labels: { severity: 'warning', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.8.4' }, annotations: { summary: 'Host filesystem free space is getting low%(cluster)s' % $.MultiClusterSummary(), @@ -573,7 +569,7 @@ }, { alert: 'CephNodeInconsistentMTU', - expr: 'node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( max by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) )or node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( min by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) )', + expr: 'node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( max by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) )or node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( min by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) )', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { summary: 'MTU settings across Ceph hosts are inconsistent%(cluster)s' % $.MultiClusterSummary(), @@ -611,7 +607,7 @@ annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full', summary: 'Pool is full - writes are blocked%(cluster)s' % $.MultiClusterSummary(), - description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range query \"topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))\" }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)", + description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range printf \"topk(5, sort_desc(ceph_pool_percent_used{cluster='%s'} * on(cluster,pool_id) group_right ceph_pool_metadata))\" .Labels.cluster | query }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)", }, }, { @@ -647,7 +643,7 @@ labels: { severity: 'warning', type: 'ceph_default' }, annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops', - summary: '{{ $labels.ceph_daemon }} operations are slow to complete', + summary: '{{ $labels.ceph_daemon }} operations are slow to complete%(cluster)s' % $.MultiClusterSummary(), description: '{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)', }, }, @@ -763,7 +759,7 @@ expr: 'absent(up{job="ceph"})', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.12.1' }, annotations: { - summary: 'The scrape job for Ceph is missing from Prometheus%(cluster)s' % $.MultiClusterSummary(), + summary: 'The scrape job for Ceph is missing from Prometheus', description: "The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you'll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance.", }, }, @@ -775,7 +771,7 @@ { alert: 'CephObjectMissing', 'for': '30s', - expr: '(ceph_health_detail{name="OBJECT_UNFOUND"} == 1) * on() (count(ceph_osd_up == 1) == bool count(ceph_osd_metadata)) == 1', + expr: '(ceph_health_detail{name="OBJECT_UNFOUND"} == 1) * on() group_right(cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by(cluster)) == 1', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.10.1' }, annotations: { documentation: 'https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound', @@ -807,31 +803,31 @@ { alert: 'CephRBDMirrorImagesPerDaemonHigh', 'for': '1m', - expr: 'sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > %(CephRBDMirrorImagesPerDaemonThreshold)s' % $._config, + expr: 'sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > %(CephRBDMirrorImagesPerDaemonThreshold)s' % $._config, labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.10.2' }, annotations: { - summary: 'Number of image replications are now above %(CephRBDMirrorImagesPerDaemonThreshold)s' % $._config, - description: 'Number of image replications per daemon is not suppossed to go beyond threshold %(CephRBDMirrorImagesPerDaemonThreshold)s' % $._config, + summary: 'Number of image replications are now above %(CephRBDMirrorImagesPerDaemonThreshold)s%(cluster)s' % [$._config.CephRBDMirrorImagesPerDaemonThreshold, $.MultiClusterSummary()], + description: 'Number of image replications per daemon is not supposed to go beyond threshold %(CephRBDMirrorImagesPerDaemonThreshold)s' % $._config, }, }, { alert: 'CephRBDMirrorImagesNotInSync', 'for': '1m', - expr: 'sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0', + expr: 'sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.10.3' }, annotations: { - summary: 'Some of the RBD mirror images are not in sync with the remote counter parts.', + summary: 'Some of the RBD mirror images are not in sync with the remote counter parts%(cluster)s' % $.MultiClusterSummary(), description: 'Both local and remote RBD mirror images should be in sync.', }, }, { alert: 'CephRBDMirrorImagesNotInSyncVeryHigh', 'for': '1m', - expr: 'count by (ceph_daemon) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon) (ceph_rbd_mirror_snapshot_snapshots)*.1)', + expr: 'count by (ceph_daemon, cluster) ((topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots)*.1)', labels: { severity: 'critical', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.10.4' }, annotations: { - summary: 'Number of unsynchronized images are very high.', - description: 'More than 10% of the images have synchronization problems', + summary: 'Number of unsynchronized images are very high%(cluster)s' % $.MultiClusterSummary(), + description: 'More than 10% of the images have synchronization problems.', }, }, { @@ -840,7 +836,7 @@ expr: 'rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > %.2f' % [$._config.CephRBDMirrorImageTransferBandwidthThreshold], labels: { severity: 'warning', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.10.5' }, annotations: { - summary: 'The replication network usage has been increased over %d%s in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes' % [$._config.CephRBDMirrorImageTransferBandwidthThreshold * 100, '%'], + summary: 'The replication network usage%(cluster)s has been increased over %d%s in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes' % [$.MultiClusterSummary(), $._config.CephRBDMirrorImageTransferBandwidthThreshold * 100, '%'], description: 'Detected a heavy increase in bandwidth for rbd replications (over %d%s) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously' % [$._config.CephRBDMirrorImageTransferBandwidthThreshold * 100, '%'], }, }, @@ -852,50 +848,50 @@ { alert: 'NVMeoFSubsystemNamespaceLimit', 'for': '1m', - expr: '(count by(nqn) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit', + expr: '(count by(nqn, cluster) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: '{{ $labels.nqn }} subsystem has reached its maximum number of namespaces %(cluster)s' % $.MultiClusterSummary(), + summary: '{{ $labels.nqn }} subsystem has reached its maximum number of namespaces%(cluster)s' % $.MultiClusterSummary(), description: 'Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}', }, }, { alert: 'NVMeoFTooManyGateways', 'for': '1m', - expr: 'count(ceph_nvmeof_gateway_info) > %.2f' % [$._config.NVMeoFMaxGatewaysPerCluster], + expr: 'count(ceph_nvmeof_gateway_info) by (cluster) > %.2f' % [$._config.NVMeoFMaxGatewaysPerCluster], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'Max supported gateways exceeded %(cluster)s' % $.MultiClusterSummary(), + summary: 'Max supported gateways exceeded%(cluster)s' % $.MultiClusterSummary(), description: 'You may create many gateways, but %(NVMeoFMaxGatewaysPerCluster)d is the tested limit' % $._config, }, }, { alert: 'NVMeoFMaxGatewayGroupSize', 'for': '1m', - expr: 'count by(group) (ceph_nvmeof_gateway_info) > %.2f' % [$._config.NVMeoFMaxGatewaysPerGroup], + expr: 'count(ceph_nvmeof_gateway_info) by (cluster,group) > %.2f' % [$._config.NVMeoFMaxGatewaysPerGroup], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'Max gateways within a gateway group ({{ $labels.group }}) exceeded %(cluster)s' % $.MultiClusterSummary(), + summary: 'Max gateways within a gateway group ({{ $labels.group }}) exceeded%(cluster)s' % $.MultiClusterSummary(), description: 'You may create many gateways in a gateway group, but %(NVMeoFMaxGatewaysPerGroup)d is the tested limit' % $._config, }, }, { alert: 'NVMeoFSingleGatewayGroup', 'for': '5m', - expr: 'count by(group) (ceph_nvmeof_gateway_info) == 1', + expr: 'count(ceph_nvmeof_gateway_info) by(cluster,group) == 1', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible %(cluster)s' % $.MultiClusterSummary(), + summary: 'The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible%(cluster)s' % $.MultiClusterSummary(), description: 'Although a single member gateway group is valid, it should only be used for test purposes', }, }, { alert: 'NVMeoFHighGatewayCPU', 'for': '10m', - expr: 'label_replace(avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode="busy"}[1m])),"instance","$1","instance","(.*):.*") > %.2f' % [$._config.NVMeoFHighGatewayCPU], + expr: 'label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode="busy"}[1m])),"instance","$1","instance","(.*):.*") > %.2f' % [$._config.NVMeoFHighGatewayCPU], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'CPU used by {{ $labels.instance }} NVMe-oF Gateway is high %(cluster)s' % $.MultiClusterSummary(), + summary: 'CPU used by {{ $labels.instance }} NVMe-oF Gateway is high%(cluster)s' % $.MultiClusterSummary(), description: 'Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores', }, }, @@ -905,27 +901,27 @@ expr: 'ceph_nvmeof_subsystem_metadata{allow_any_host="yes"}', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'Subsystem {{ $labels.nqn }} has been defined without host level security %(cluster)s' % $.MultiClusterSummary(), + summary: 'Subsystem {{ $labels.nqn }} has been defined without host level security%(cluster)s' % $.MultiClusterSummary(), description: 'It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss', }, }, { alert: 'NVMeoFTooManySubsystems', 'for': '1m', - expr: 'count by(gateway_host) (label_replace(ceph_nvmeof_subsystem_metadata,"gateway_host","$1","instance","(.*):.*")) > %.2f' % [$._config.NVMeoFMaxSubsystemsPerGateway], + expr: 'count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,"gateway_host","$1","instance","(.*):.*")) > %.2f' % [$._config.NVMeoFMaxSubsystemsPerGateway], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'The number of subsystems defined to the gateway exceeds supported values %(cluster)s' % $.MultiClusterSummary(), + summary: 'The number of subsystems defined to the gateway exceeds supported values%(cluster)s' % $.MultiClusterSummary(), description: 'Although you may continue to create subsystems in {{ $labels.gateway_host }}, the configuration may not be supported', }, }, { alert: 'NVMeoFVersionMismatch', 'for': '1h', - expr: 'count(count by(version) (ceph_nvmeof_gateway_info)) > 1', + expr: 'count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'The cluster has different NVMe-oF gateway releases active %(cluster)s' % $.MultiClusterSummary(), + summary: 'Too many different NVMe-oF gateway releases active%(cluster)s' % $.MultiClusterSummary(), description: 'This may indicate an issue with deployment. Check cephadm logs', }, }, @@ -935,17 +931,17 @@ expr: 'ceph_nvmeof_subsystem_host_count > %.2f' % [$._config.NVMeoFHighClientCount], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'The number of clients connected to {{ $labels.nqn }} is too high %(cluster)s' % $.MultiClusterSummary(), + summary: 'The number of clients connected to {{ $labels.nqn }} is too high%(cluster)s' % $.MultiClusterSummary(), description: 'The supported limit for clients connecting to a subsystem is %(NVMeoFHighClientCount)d' % $._config, }, }, { alert: 'NVMeoFHighHostCPU', 'for': '10m', - expr: '100-((100*(avg by(host) (label_replace(rate(node_cpu_seconds_total{mode="idle"}[5m]),"host","$1","instance","(.*):.*")) * on(host) group_right label_replace(ceph_nvmeof_gateway_info,"host","$1","instance","(.*):.*")))) >= %.2f' % [$._config.NVMeoFHighHostCPU], + expr: '100-((100*(avg by(cluster,host) (label_replace(rate(node_cpu_seconds_total{mode="idle"}[5m]),"host","$1","instance","(.*):.*")) * on(cluster, host) group_right label_replace(ceph_nvmeof_gateway_info,"host","$1","instance","(.*):.*")))) >= %.2f' % [$._config.NVMeoFHighHostCPU], labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'The CPU is high ({{ $value }}%%) on NVMeoF Gateway host ({{ $labels.host }}) %(cluster)s' % $.MultiClusterSummary(), + summary: 'The CPU is high ({{ $value }}%%) on NVMeoF Gateway host ({{ $labels.host }})%(cluster)s' % $.MultiClusterSummary(), description: 'High CPU on a gateway host can lead to CPU contention and performance degradation', }, }, @@ -955,7 +951,7 @@ expr: 'ceph_nvmeof_subsystem_listener_iface_info{operstate="down"}', labels: { severity: 'warning', type: 'ceph_default', oid: '1.3.6.1.4.1.50495.1.2.1.14.1' }, annotations: { - summary: 'Network interface {{ $labels.device }} is down %(cluster)s' % $.MultiClusterSummary(), + summary: 'Network interface {{ $labels.device }} is down%(cluster)s' % $.MultiClusterSummary(), description: 'A NIC used by one or more subsystems is in a down state', }, }, @@ -965,7 +961,7 @@ expr: 'ceph_nvmeof_subsystem_listener_iface_info{duplex!="full"}', labels: { severity: 'warning', type: 'ceph_default' }, annotations: { - summary: 'Network interface {{ $labels.device }} is not running in full duplex mode %(cluster)s' % $.MultiClusterSummary(), + summary: 'Network interface {{ $labels.device }} is not running in full duplex mode%(cluster)s' % $.MultiClusterSummary(), description: 'Until this is resolved, performance from the gateway will be degraded', }, }, diff --git a/monitoring/ceph-mixin/prometheus_alerts.yml b/monitoring/ceph-mixin/prometheus_alerts.yml index 84452e5845a..ba6a6ded0a3 100644 --- a/monitoring/ceph-mixin/prometheus_alerts.yml +++ b/monitoring/ceph-mixin/prometheus_alerts.yml @@ -3,8 +3,8 @@ groups: rules: - alert: "CephHealthError" annotations: - description: "The cluster state has been HEALTH_ERROR for more than 5 minutes. Please check 'ceph health detail' for more information." - summary: "Ceph is in the ERROR state" + description: "The cluster state has been HEALTH_ERROR for more than 5 minutes on cluster {{ $labels.cluster }}. Please check 'ceph health detail' for more information." + summary: "Ceph is in the ERROR state on cluster {{ $labels.cluster }}" expr: "ceph_health_status == 2" for: "5m" labels: @@ -13,8 +13,8 @@ groups: type: "ceph_default" - alert: "CephHealthWarning" annotations: - description: "The cluster state has been HEALTH_WARN for more than 15 minutes. Please check 'ceph health detail' for more information." - summary: "Ceph is in the WARNING state" + description: "The cluster state has been HEALTH_WARN for more than 15 minutes on cluster {{ $labels.cluster }}. Please check 'ceph health detail' for more information." + summary: "Ceph is in the WARNING state on cluster {{ $labels.cluster }}" expr: "ceph_health_status == 1" for: "15m" labels: @@ -24,13 +24,13 @@ groups: rules: - alert: "CephMonDownQuorumAtRisk" annotations: - description: "{{ $min := query \"floor(count(ceph_mon_metadata) / 2) + 1\" | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range query \"(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" + description: "{{ $min := printf \"floor(count(ceph_mon_metadata{cluster='%s'}) / 2) + 1\" .Labels.cluster | query | first | value }}Quorum requires a majority of monitors (x {{ $min }}) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: {{- range printf \"(ceph_mon_quorum_status{cluster='%s'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down" - summary: "Monitor quorum is at risk" + summary: "Monitor quorum is at risk on cluster {{ $labels.cluster }}" expr: | ( - (ceph_health_detail{name="MON_DOWN"} == 1) * on() ( - count(ceph_mon_quorum_status == 1) == bool (floor(count(ceph_mon_metadata) / 2) + 1) + (ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) ( + count(ceph_mon_quorum_status == 1) by(cluster)== bool (floor(count(ceph_mon_metadata) by(cluster) / 2) + 1) ) ) == 1 for: "30s" @@ -40,12 +40,11 @@ groups: type: "ceph_default" - alert: "CephMonDown" annotations: - description: | - {{ $down := query "count(ceph_mon_quorum_status == 0)" | first | value }}{{ $s := "" }}{{ if gt $down 1.0 }}{{ $s = "s" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range query "(ceph_mon_quorum_status == 0) + on(ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }} + description: "{{ $down := printf \"count(ceph_mon_quorum_status{cluster='%s'} == 0)\" .Labels.cluster | query | first | value }}{{ $s := \"\" }}{{ if gt $down 1.0 }}{{ $s = \"s\" }}{{ end }}You have {{ $down }} monitor{{ $s }} down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: {{- range printf \"(ceph_mon_quorum_status{cluster='%s'} == 0) + on(cluster,ceph_daemon) group_left(hostname) (ceph_mon_metadata * 0)\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down" - summary: "One or more monitors down" + summary: "One or more monitors down on cluster {{ $labels.cluster }}" expr: | - count(ceph_mon_quorum_status == 0) <= (count(ceph_mon_metadata) - floor(count(ceph_mon_metadata) / 2) + 1) + (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1))) for: "30s" labels: severity: "warning" @@ -54,7 +53,7 @@ groups: annotations: description: "The free space available to a monitor's store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \"ceph_mon_metadata\"}} - {{ .Labels.hostname }} {{- end }}" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit" - summary: "Filesystem space on at least one monitor is critically low" + summary: "Filesystem space on at least one monitor is critically low on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MON_DISK_CRIT\"} == 1" for: "1m" labels: @@ -65,7 +64,7 @@ groups: annotations: description: "The space available to a monitor's store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; {{- range query \"ceph_mon_metadata\"}} - {{ .Labels.hostname }} {{- end }}" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low" - summary: "Drive space on at least one monitor is approaching full" + summary: "Drive space on at least one monitor is approaching full on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MON_DISK_LOW\"} == 1" for: "5m" labels: @@ -75,7 +74,7 @@ groups: annotations: description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with 'ceph time-sync-status' and the state and peers of your ntpd or chrony daemon." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew" - summary: "Clock skew detected among monitors" + summary: "Clock skew detected among monitors on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MON_CLOCK_SKEW\"} == 1" for: "1m" labels: @@ -85,17 +84,17 @@ groups: rules: - alert: "CephOSDDownHigh" annotations: - description: "{{ $value | humanize }}% or {{ with query \"count(ceph_osd_up == 0)\" }}{{ . | first | value }}{{ end }} of {{ with query \"count(ceph_osd_up)\" }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range query \"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\" }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" - summary: "More than 10% of OSDs are down" - expr: "count(ceph_osd_up == 0) / count(ceph_osd_up) * 100 >= 10" + description: "{{ $value | humanize }}% or {{ with printf \"count (ceph_osd_up{cluster='%s'} == 0)\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} of {{ with printf \"count (ceph_osd_up{cluster='%s'})\" .Labels.cluster | query }}{{ . | first | value }}{{ end }} OSDs are down (>= 10%). The following OSDs are down: {{- range printf \"(ceph_osd_up{cluster='%s'} * on(cluster, ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" + summary: "More than 10% of OSDs are down on cluster {{ $labels.cluster }}" + expr: "count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10" labels: oid: "1.3.6.1.4.1.50495.1.2.1.4.1" severity: "critical" type: "ceph_default" - alert: "CephOSDHostDown" annotations: - description: "The following OSDs are down: {{- range query \"(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\" }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}" - summary: "An OSD host is offline" + description: "The following OSDs are down: {{- range printf \"(ceph_osd_up{cluster='%s'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\" .Labels.cluster | query }} - {{ .Labels.hostname }} : {{ .Labels.ceph_daemon }} {{- end }}" + summary: "An OSD host is offline on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_HOST_DOWN\"} == 1" for: "5m" labels: @@ -104,10 +103,9 @@ groups: type: "ceph_default" - alert: "CephOSDDown" annotations: - description: | - {{ $num := query "count(ceph_osd_up == 0)" | first | value }}{{ $s := "" }}{{ if gt $num 1.0 }}{{ $s = "s" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s "" }}is{{ else }}are{{ end }} down: {{- range query "(ceph_osd_up * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0"}} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }} + description: "{{ $num := printf \"count(ceph_osd_up{cluster='%s'} == 0) \" .Labels.cluster | query | first | value }}{{ $s := \"\" }}{{ if gt $num 1.0 }}{{ $s = \"s\" }}{{ end }}{{ $num }} OSD{{ $s }} down for over 5mins. The following OSD{{ $s }} {{ if eq $s \"\" }}is{{ else }}are{{ end }} down: {{- range printf \"(ceph_osd_up{cluster='%s'} * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) == 0\" .Labels.cluster | query }} - {{ .Labels.ceph_daemon }} on {{ .Labels.hostname }} {{- end }}" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down" - summary: "An OSD has been marked down" + summary: "An OSD has been marked down on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_DOWN\"} == 1" for: "5m" labels: @@ -118,7 +116,7 @@ groups: annotations: description: "One or more OSDs have reached the NEARFULL threshold. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull" - summary: "OSD(s) running low on free space (NEARFULL)" + summary: "OSD(s) running low on free space (NEARFULL) on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_NEARFULL\"} == 1" for: "5m" labels: @@ -129,7 +127,7 @@ groups: annotations: description: "An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full" - summary: "OSD full, writes blocked" + summary: "OSD full, writes blocked on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_FULL\"} > 0" for: "1m" labels: @@ -140,7 +138,7 @@ groups: annotations: description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull" - summary: "OSD(s) too full for backfill operations" + summary: "OSD(s) too full for backfill operations on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_BACKFILLFULL\"} > 0" for: "1m" labels: @@ -150,7 +148,7 @@ groups: annotations: description: "Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs" - summary: "OSD reports a high number of read errors" + summary: "OSD reports a high number of read errors on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_TOO_MANY_REPAIRS\"} == 1" for: "30s" labels: @@ -159,7 +157,7 @@ groups: - alert: "CephOSDTimeoutsPublicNetwork" annotations: description: "OSD heartbeats on the cluster's 'public' network (frontend) are running slow. Investigate the network for latency or loss issues. Use 'ceph health detail' to show the affected OSDs." - summary: "Network issues delaying OSD heartbeats (public network)" + summary: "Network issues delaying OSD heartbeats (public network) on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_SLOW_PING_TIME_FRONT\"} == 1" for: "1m" labels: @@ -168,7 +166,7 @@ groups: - alert: "CephOSDTimeoutsClusterNetwork" annotations: description: "OSD heartbeats on the cluster's 'cluster' network (backend) are slow. Investigate the network for latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs." - summary: "Network issues delaying OSD heartbeats (cluster network)" + summary: "Network issues delaying OSD heartbeats (cluster network) on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"OSD_SLOW_PING_TIME_BACK\"} == 1" for: "1m" labels: @@ -178,7 +176,7 @@ groups: annotations: description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch" - summary: "OSD size inconsistency error" + summary: "OSD size inconsistency error on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"BLUESTORE_DISK_SIZE_MISMATCH\"} == 1" for: "1m" labels: @@ -188,7 +186,7 @@ groups: annotations: description: "The device health module has determined that one or more devices will fail soon. To review device status use 'ceph device ls'. To show a specific device use 'ceph device info <dev id>'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#id2" - summary: "Device(s) predicted to fail soon" + summary: "Device(s) predicted to fail soon on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"DEVICE_HEALTH\"} == 1" for: "1m" labels: @@ -198,7 +196,7 @@ groups: annotations: description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany" - summary: "Too many devices are predicted to fail, unable to resolve" + summary: "Too many devices are predicted to fail on cluster {{ $labels.cluster }}, unable to resolve" expr: "ceph_health_detail{name=\"DEVICE_HEALTH_TOOMANY\"} == 1" for: "1m" labels: @@ -209,7 +207,7 @@ groups: annotations: description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use" - summary: "Device failure is predicted, but unable to relocate data" + summary: "Device failure is predicted, but unable to relocate data on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"DEVICE_HEALTH_IN_USE\"} == 1" for: "1m" labels: @@ -219,8 +217,8 @@ groups: annotations: description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up {{ $value | humanize }} times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)." documentation: "https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds" - summary: "Network issues are causing OSDs to flap (mark each other down)" - expr: "(rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1" + summary: "Network issues are causing OSDs to flap (mark each other down) on cluster {{ $labels.cluster }}" + expr: "(rate(ceph_osd_up[5m]) * on(cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata) * 60 > 1" labels: oid: "1.3.6.1.4.1.50495.1.2.1.4.4" severity: "warning" @@ -229,7 +227,7 @@ groups: annotations: description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors" - summary: "Device read errors detected" + summary: "Device read errors detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"BLUESTORE_SPURIOUS_READ_ERRORS\"} == 1" for: "30s" labels: @@ -238,12 +236,12 @@ groups: - alert: "CephPGImbalance" annotations: description: "OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count." - summary: "PGs are not balanced across OSDs" + summary: "PGs are not balanced across OSDs on cluster {{ $labels.cluster }}" expr: | abs( - ((ceph_osd_numpg > 0) - on (job) group_left avg(ceph_osd_numpg > 0) by (job)) / + ((ceph_osd_numpg > 0) - on (cluster,job) group_left avg(ceph_osd_numpg > 0) by (cluster,job)) / on (job) group_left avg(ceph_osd_numpg > 0) by (job) - ) * on (ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30 + ) * on (cluster,ceph_daemon) group_left(hostname) ceph_osd_metadata > 0.30 for: "5m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.4.5" @@ -255,7 +253,7 @@ groups: annotations: description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages" - summary: "CephFS filesystem is damaged." + summary: "CephFS filesystem is damaged on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MDS_DAMAGE\"} > 0" for: "1m" labels: @@ -266,7 +264,7 @@ groups: annotations: description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down" - summary: "CephFS filesystem is offline" + summary: "CephFS filesystem is offline on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MDS_ALL_DOWN\"} > 0" for: "1m" labels: @@ -277,7 +275,7 @@ groups: annotations: description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded" - summary: "CephFS filesystem is degraded" + summary: "CephFS filesystem is degraded on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"FS_DEGRADED\"} > 0" for: "1m" labels: @@ -288,7 +286,7 @@ groups: annotations: description: "The filesystem's 'max_mds' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max" - summary: "Ceph MDS daemon count is lower than configured" + summary: "Ceph MDS daemon count is lower than configured on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MDS_UP_LESS_THAN_MAX\"} > 0" for: "1m" labels: @@ -298,7 +296,7 @@ groups: annotations: description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby" - summary: "Ceph filesystem standby daemons too few" + summary: "Ceph filesystem standby daemons too few on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MDS_INSUFFICIENT_STANDBY\"} > 0" for: "1m" labels: @@ -308,7 +306,7 @@ groups: annotations: description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds" - summary: "MDS daemon failed, no further standby available" + summary: "MDS daemon failed, no further standby available on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"FS_WITH_FAILED_MDS\"} > 0" for: "1m" labels: @@ -319,7 +317,7 @@ groups: annotations: description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support." documentation: "https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages" - summary: "CephFS filesystem in read only mode due to write error(s)" + summary: "CephFS filesystem in read only mode due to write error(s) on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"MDS_HEALTH_READ_ONLY\"} > 0" for: "1m" labels: @@ -332,7 +330,7 @@ groups: annotations: description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the 'ceph crash' command to determine which module has failed, and archive it to acknowledge the failure." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash" - summary: "A manager module has recently crashed" + summary: "A manager module has recently crashed on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"RECENT_MGR_MODULE_CRASH\"} == 1" for: "5m" labels: @@ -354,8 +352,8 @@ groups: - alert: "CephPGsInactive" annotations: description: "{{ $value }} PGs have been inactive for more than 5 minutes in pool {{ $labels.name }}. Inactive placement groups are not able to serve read/write requests." - summary: "One or more placement groups are inactive" - expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0" + summary: "One or more placement groups are inactive on cluster {{ $labels.cluster }}" + expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0" for: "5m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.7.1" @@ -364,8 +362,8 @@ groups: - alert: "CephPGsUnclean" annotations: description: "{{ $value }} PGs have been unclean for more than 15 minutes in pool {{ $labels.name }}. Unclean PGs have not recovered from a previous failure." - summary: "One or more placement groups are marked unclean" - expr: "ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0" + summary: "One or more placement groups are marked unclean on cluster {{ $labels.cluster }}" + expr: "ceph_pool_metadata * on(cluster,pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0" for: "15m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.7.2" @@ -375,7 +373,7 @@ groups: annotations: description: "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use 'rados list-inconsistent-pg <pool>'. To repair PGs use the 'ceph pg repair <pg_num>' command." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged" - summary: "Placement group damaged, manual intervention needed" + summary: "Placement group damaged, manual intervention needed on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=~\"PG_DAMAGED|OSD_SCRUB_ERRORS\"} == 1" for: "5m" labels: @@ -386,7 +384,7 @@ groups: annotations: description: "Data redundancy is at risk since one or more OSDs are at or above the 'full' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full" - summary: "OSDs are too full for recovery" + summary: "OSDs are too full for recovery on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"PG_RECOVERY_FULL\"} == 1" for: "1m" labels: @@ -397,7 +395,7 @@ groups: annotations: description: "Data availability is reduced, impacting the cluster's ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability" - summary: "PG is unavailable, blocking I/O" + summary: "PG is unavailable on cluster {{ $labels.cluster }}, blocking I/O" expr: "((ceph_health_detail{name=\"PG_AVAILABILITY\"} == 1) - scalar(ceph_health_detail{name=\"OSD_DOWN\"})) == 1" for: "1m" labels: @@ -408,7 +406,7 @@ groups: annotations: description: "Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the 'backfillfull' threshold. Add more capacity, or delete unwanted data." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full" - summary: "Backfill operations are blocked due to lack of free space" + summary: "Backfill operations are blocked due to lack of free space on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"PG_BACKFILL_FULL\"} == 1" for: "1m" labels: @@ -419,7 +417,7 @@ groups: annotations: description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a 'clean' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub <pgid>" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed" - summary: "Placement group(s) have not been scrubbed" + summary: "Placement group(s) have not been scrubbed on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"PG_NOT_SCRUBBED\"} == 1" for: "5m" labels: @@ -429,7 +427,7 @@ groups: annotations: description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\n Check that the pg_autoscaler has not been disabled for any pools with 'ceph osd pool autoscale-status', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool ('ceph osd pool set cephfs.cephfs.meta target_size_ratio .1') or set the pg_autoscaler mode to 'warn' and adjust pg_num appropriately for one or more pools." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs" - summary: "Placement groups per OSD is too high" + summary: "Placement groups per OSD is too high on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"TOO_MANY_PGS\"} == 1" for: "1m" labels: @@ -439,7 +437,7 @@ groups: annotations: description: "One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a 'clean' state during the deep-scrub window." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed" - summary: "Placement group(s) have not been deep scrubbed" + summary: "Placement group(s) have not been deep scrubbed on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"PG_NOT_DEEP_SCRUBBED\"} == 1" for: "5m" labels: @@ -479,7 +477,7 @@ groups: - alert: "CephNodeNetworkPacketErrors" annotations: description: "Node {{ $labels.instance }} experiences packet errors > 0.01% or > 10 packets/s on interface {{ $labels.device }}." - summary: "One or more NICs reports packet errors" + summary: "One or more NICs reports packet errors on cluster {{ $labels.cluster }}" expr: | ( rate(node_network_receive_errs_total{device!="lo"}[1m]) + @@ -498,7 +496,7 @@ groups: - alert: "CephNodeNetworkBondDegraded" annotations: description: "Bond {{ $labels.master }} is degraded on Node {{ $labels.instance }}." - summary: "Degraded Bond on Node {{ $labels.instance }}" + summary: "Degraded Bond on Node {{ $labels.instance }} on cluster {{ $labels.cluster }}" expr: | node_bonding_slaves - node_bonding_active != 0 labels: @@ -507,8 +505,8 @@ groups: - alert: "CephNodeDiskspaceWarning" annotations: description: "Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will be full in less than 5 days based on the 48 hour trailing fill rate." - summary: "Host filesystem free space is getting low" - expr: "predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5) *on(instance) group_left(nodename) node_uname_info < 0" + summary: "Host filesystem free space is getting low on cluster {{ $labels.cluster }}" + expr: "predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5) * on(cluster, instance) group_left(nodename) node_uname_info < 0" labels: oid: "1.3.6.1.4.1.50495.1.2.1.8.4" severity: "warning" @@ -516,8 +514,8 @@ groups: - alert: "CephNodeInconsistentMTU" annotations: description: "Node {{ $labels.instance }} has a different MTU size ({{ $value }}) than the median of devices named {{ $labels.device }}." - summary: "MTU settings across Ceph hosts are inconsistent" - expr: "node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0) == scalar( max by (device) (node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) != quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) )or node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0) == scalar( min by (device) (node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) != quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) )" + summary: "MTU settings across Ceph hosts are inconsistent on cluster {{ $labels.cluster }}" + expr: "node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0) == scalar( max by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) != quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) )or node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0) == scalar( min by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) != quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!=\"lo\"} > 0)) )" labels: severity: "warning" type: "ceph_default" @@ -526,8 +524,8 @@ groups: - alert: "CephPoolGrowthWarning" annotations: description: "Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours." - summary: "Pool growth rate may soon exceed capacity" - expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id, instance) group_right() ceph_pool_metadata) >= 95" + summary: "Pool growth rate may soon exceed capacity on cluster {{ $labels.cluster }}" + expr: "(predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(cluster,pool_id, instance) group_right() ceph_pool_metadata) >= 95" labels: oid: "1.3.6.1.4.1.50495.1.2.1.9.2" severity: "warning" @@ -535,16 +533,16 @@ groups: - alert: "CephPoolBackfillFull" annotations: description: "A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity." - summary: "Free space in a pool is too low for recovery/backfill" + summary: "Free space in a pool is too low for recovery/backfill on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"POOL_BACKFILLFULL\"} > 0" labels: severity: "warning" type: "ceph_default" - alert: "CephPoolFull" annotations: - description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range query \"topk(5, sort_desc(ceph_pool_percent_used * on(pool_id) group_right ceph_pool_metadata))\" }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)" + description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) {{- range printf \"topk(5, sort_desc(ceph_pool_percent_used{cluster='%s'} * on(cluster,pool_id) group_right ceph_pool_metadata))\" .Labels.cluster | query }} - {{ .Labels.name }} at {{ .Value }}% {{- end }} Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full" - summary: "Pool is full - writes are blocked" + summary: "Pool is full - writes are blocked on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"POOL_FULL\"} > 0" for: "1m" labels: @@ -554,7 +552,7 @@ groups: - alert: "CephPoolNearFull" annotations: description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn't made available. Determine the affected pool with 'ceph df detail', looking at QUOTA BYTES and STORED. Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>). Also ensure that the balancer is active." - summary: "One or more Ceph pools are nearly full" + summary: "One or more Ceph pools are nearly full on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"POOL_NEAR_FULL\"} > 0" for: "5m" labels: @@ -566,7 +564,7 @@ groups: annotations: description: "{{ $value }} OSD requests are taking too long to process (osd_op_complaint_time exceeded)" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops" - summary: "OSD operations are slow to complete" + summary: "OSD operations are slow to complete on cluster {{ $labels.cluster }}" expr: "ceph_healthcheck_slow_ops > 0" for: "30s" labels: @@ -576,7 +574,7 @@ groups: annotations: description: "{{ $labels.ceph_daemon }} operations are taking too long to process (complaint time exceeded)" documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops" - summary: "{{ $labels.ceph_daemon }} operations are slow to complete" + summary: "{{ $labels.ceph_daemon }} operations are slow to complete on cluster {{ $labels.cluster }}" expr: "ceph_daemon_health_metrics{type=\"SLOW_OPS\"} > 0" for: "30s" labels: @@ -587,7 +585,7 @@ groups: - alert: "CephadmUpgradeFailed" annotations: description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue" - summary: "Ceph version upgrade has failed" + summary: "Ceph version upgrade has failed on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"UPGRADE_EXCEPTION\"} > 0" for: "30s" labels: @@ -597,7 +595,7 @@ groups: - alert: "CephadmDaemonFailed" annotations: description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with 'ceph health detail'. you may start daemons with the 'ceph orch daemon start <daemon_id>'" - summary: "A ceph daemon managed by cephadm is down" + summary: "A ceph daemon managed by cephadm is down on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"CEPHADM_FAILED_DAEMON\"} > 0" for: "30s" labels: @@ -608,7 +606,7 @@ groups: annotations: description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with 'ceph orch resume'" documentation: "https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused" - summary: "Orchestration tasks via cephadm are PAUSED" + summary: "Orchestration tasks via cephadm are PAUSED on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"CEPHADM_PAUSED\"} > 0" for: "1m" labels: @@ -619,7 +617,7 @@ groups: - alert: "HardwareStorageError" annotations: description: "Some storage devices are in error. Check `ceph health detail`." - summary: "Storage devices error(s) detected" + summary: "Storage devices error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_STORAGE\"} > 0" for: "30s" labels: @@ -629,7 +627,7 @@ groups: - alert: "HardwareMemoryError" annotations: description: "DIMM error(s) detected. Check `ceph health detail`." - summary: "DIMM error(s) detected" + summary: "DIMM error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_MEMORY\"} > 0" for: "30s" labels: @@ -639,7 +637,7 @@ groups: - alert: "HardwareProcessorError" annotations: description: "Processor error(s) detected. Check `ceph health detail`." - summary: "Processor error(s) detected" + summary: "Processor error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_PROCESSOR\"} > 0" for: "30s" labels: @@ -649,7 +647,7 @@ groups: - alert: "HardwareNetworkError" annotations: description: "Network error(s) detected. Check `ceph health detail`." - summary: "Network error(s) detected" + summary: "Network error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_NETWORK\"} > 0" for: "30s" labels: @@ -659,7 +657,7 @@ groups: - alert: "HardwarePowerError" annotations: description: "Power supply error(s) detected. Check `ceph health detail`." - summary: "Power supply error(s) detected" + summary: "Power supply error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_POWER\"} > 0" for: "30s" labels: @@ -669,7 +667,7 @@ groups: - alert: "HardwareFanError" annotations: description: "Fan error(s) detected. Check `ceph health detail`." - summary: "Fan error(s) detected" + summary: "Fan error(s) detected on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"HARDWARE_FANS\"} > 0" for: "30s" labels: @@ -694,8 +692,8 @@ groups: annotations: description: "The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound" - summary: "Object(s) marked UNFOUND" - expr: "(ceph_health_detail{name=\"OBJECT_UNFOUND\"} == 1) * on() (count(ceph_osd_up == 1) == bool count(ceph_osd_metadata)) == 1" + summary: "Object(s) marked UNFOUND on cluster {{ $labels.cluster }}" + expr: "(ceph_health_detail{name=\"OBJECT_UNFOUND\"} == 1) * on() group_right(cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by(cluster)) == 1" for: "30s" labels: oid: "1.3.6.1.4.1.50495.1.2.1.10.1" @@ -707,7 +705,7 @@ groups: annotations: description: "One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the 'ceph crash archive <id>' command." documentation: "https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash" - summary: "One or more Ceph daemons have crashed, and are pending acknowledgement" + summary: "One or more Ceph daemons have crashed, and are pending acknowledgement on cluster {{ $labels.cluster }}" expr: "ceph_health_detail{name=\"RECENT_CRASH\"} == 1" for: "1m" labels: @@ -718,9 +716,9 @@ groups: rules: - alert: "CephRBDMirrorImagesPerDaemonHigh" annotations: - description: "Number of image replications per daemon is not suppossed to go beyond threshold 100" - summary: "Number of image replications are now above 100" - expr: "sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100" + description: "Number of image replications per daemon is not supposed to go beyond threshold 100" + summary: "Number of image replications are now above 100 on cluster {{ $labels.cluster }}" + expr: "sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100" for: "1m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.10.2" @@ -729,8 +727,8 @@ groups: - alert: "CephRBDMirrorImagesNotInSync" annotations: description: "Both local and remote RBD mirror images should be in sync." - summary: "Some of the RBD mirror images are not in sync with the remote counter parts." - expr: "sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0" + summary: "Some of the RBD mirror images are not in sync with the remote counter parts on cluster {{ $labels.cluster }}" + expr: "sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0" for: "1m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.10.3" @@ -738,9 +736,9 @@ groups: type: "ceph_default" - alert: "CephRBDMirrorImagesNotInSyncVeryHigh" annotations: - description: "More than 10% of the images have synchronization problems" - summary: "Number of unsynchronized images are very high." - expr: "count by (ceph_daemon) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon) (ceph_rbd_mirror_snapshot_snapshots)*.1)" + description: "More than 10% of the images have synchronization problems." + summary: "Number of unsynchronized images are very high on cluster {{ $labels.cluster }}" + expr: "count by (ceph_daemon, cluster) ((topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) > (sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots)*.1)" for: "1m" labels: oid: "1.3.6.1.4.1.50495.1.2.1.10.4" @@ -749,7 +747,7 @@ groups: - alert: "CephRBDMirrorImageTransferBandwidthHigh" annotations: description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously" - summary: "The replication network usage has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes" + summary: "The replication network usage on cluster {{ $labels.cluster }} has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes" expr: "rate(ceph_rbd_mirror_journal_replay_bytes[30m]) > 0.80" for: "1m" labels: @@ -761,8 +759,8 @@ groups: - alert: "NVMeoFSubsystemNamespaceLimit" annotations: description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to {{ $labels.nqn }}" - summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces " - expr: "(count by(nqn) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit" + summary: "{{ $labels.nqn }} subsystem has reached its maximum number of namespaces on cluster {{ $labels.cluster }}" + expr: "(count by(nqn, cluster) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit" for: "1m" labels: severity: "warning" @@ -770,17 +768,17 @@ groups: - alert: "NVMeoFTooManyGateways" annotations: description: "You may create many gateways, but 4 is the tested limit" - summary: "Max supported gateways exceeded " - expr: "count(ceph_nvmeof_gateway_info) > 4.00" + summary: "Max supported gateways exceeded on cluster {{ $labels.cluster }}" + expr: "count(ceph_nvmeof_gateway_info) by (cluster) > 4.00" for: "1m" labels: severity: "warning" type: "ceph_default" - alert: "NVMeoFMaxGatewayGroupSize" annotations: - description: "You may create many gateways in a gateway group, but 2 is the tested limit" - summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded " - expr: "count by(group) (ceph_nvmeof_gateway_info) > 2.00" + description: "You may create many gateways in a gateway group, but 4 is the tested limit" + summary: "Max gateways within a gateway group ({{ $labels.group }}) exceeded on cluster {{ $labels.cluster }}" + expr: "count(ceph_nvmeof_gateway_info) by (cluster,group) > 4.00" for: "1m" labels: severity: "warning" @@ -788,8 +786,8 @@ groups: - alert: "NVMeoFSingleGatewayGroup" annotations: description: "Although a single member gateway group is valid, it should only be used for test purposes" - summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible " - expr: "count by(group) (ceph_nvmeof_gateway_info) == 1" + summary: "The gateway group {{ $labels.group }} consists of a single gateway - HA is not possible on cluster {{ $labels.cluster }}" + expr: "count(ceph_nvmeof_gateway_info) by(cluster,group) == 1" for: "5m" labels: severity: "warning" @@ -797,8 +795,8 @@ groups: - alert: "NVMeoFHighGatewayCPU" annotations: description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores" - summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high " - expr: "label_replace(avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode=\"busy\"}[1m])),\"instance\",\"$1\",\"instance\",\"(.*):.*\") > 80.00" + summary: "CPU used by {{ $labels.instance }} NVMe-oF Gateway is high on cluster {{ $labels.cluster }}" + expr: "label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode=\"busy\"}[1m])),\"instance\",\"$1\",\"instance\",\"(.*):.*\") > 80.00" for: "10m" labels: severity: "warning" @@ -806,7 +804,7 @@ groups: - alert: "NVMeoFGatewayOpenSecurity" annotations: description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss" - summary: "Subsystem {{ $labels.nqn }} has been defined without host level security " + summary: "Subsystem {{ $labels.nqn }} has been defined without host level security on cluster {{ $labels.cluster }}" expr: "ceph_nvmeof_subsystem_metadata{allow_any_host=\"yes\"}" for: "5m" labels: @@ -815,8 +813,8 @@ groups: - alert: "NVMeoFTooManySubsystems" annotations: description: "Although you may continue to create subsystems in {{ $labels.gateway_host }}, the configuration may not be supported" - summary: "The number of subsystems defined to the gateway exceeds supported values " - expr: "count by(gateway_host) (label_replace(ceph_nvmeof_subsystem_metadata,\"gateway_host\",\"$1\",\"instance\",\"(.*):.*\")) > 16.00" + summary: "The number of subsystems defined to the gateway exceeds supported values on cluster {{ $labels.cluster }}" + expr: "count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,\"gateway_host\",\"$1\",\"instance\",\"(.*):.*\")) > 16.00" for: "1m" labels: severity: "warning" @@ -824,8 +822,8 @@ groups: - alert: "NVMeoFVersionMismatch" annotations: description: "This may indicate an issue with deployment. Check cephadm logs" - summary: "The cluster has different NVMe-oF gateway releases active " - expr: "count(count by(version) (ceph_nvmeof_gateway_info)) > 1" + summary: "Too many different NVMe-oF gateway releases active on cluster {{ $labels.cluster }}" + expr: "count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1" for: "1h" labels: severity: "warning" @@ -833,7 +831,7 @@ groups: - alert: "NVMeoFHighClientCount" annotations: description: "The supported limit for clients connecting to a subsystem is 32" - summary: "The number of clients connected to {{ $labels.nqn }} is too high " + summary: "The number of clients connected to {{ $labels.nqn }} is too high on cluster {{ $labels.cluster }}" expr: "ceph_nvmeof_subsystem_host_count > 32.00" for: "1m" labels: @@ -842,8 +840,8 @@ groups: - alert: "NVMeoFHighHostCPU" annotations: description: "High CPU on a gateway host can lead to CPU contention and performance degradation" - summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) " - expr: "100-((100*(avg by(host) (label_replace(rate(node_cpu_seconds_total{mode=\"idle\"}[5m]),\"host\",\"$1\",\"instance\",\"(.*):.*\")) * on(host) group_right label_replace(ceph_nvmeof_gateway_info,\"host\",\"$1\",\"instance\",\"(.*):.*\")))) >= 80.00" + summary: "The CPU is high ({{ $value }}%) on NVMeoF Gateway host ({{ $labels.host }}) on cluster {{ $labels.cluster }}" + expr: "100-((100*(avg by(cluster,host) (label_replace(rate(node_cpu_seconds_total{mode=\"idle\"}[5m]),\"host\",\"$1\",\"instance\",\"(.*):.*\")) * on(cluster, host) group_right label_replace(ceph_nvmeof_gateway_info,\"host\",\"$1\",\"instance\",\"(.*):.*\")))) >= 80.00" for: "10m" labels: severity: "warning" @@ -851,7 +849,7 @@ groups: - alert: "NVMeoFInterfaceDown" annotations: description: "A NIC used by one or more subsystems is in a down state" - summary: "Network interface {{ $labels.device }} is down " + summary: "Network interface {{ $labels.device }} is down on cluster {{ $labels.cluster }}" expr: "ceph_nvmeof_subsystem_listener_iface_info{operstate=\"down\"}" for: "30s" labels: @@ -861,7 +859,7 @@ groups: - alert: "NVMeoFInterfaceDuplex" annotations: description: "Until this is resolved, performance from the gateway will be degraded" - summary: "Network interface {{ $labels.device }} is not running in full duplex mode " + summary: "Network interface {{ $labels.device }} is not running in full duplex mode on cluster {{ $labels.cluster }}" expr: "ceph_nvmeof_subsystem_listener_iface_info{duplex!=\"full\"}" for: "30s" labels: diff --git a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml index 40d6f4d0983..a269ff74227 100644 --- a/monitoring/ceph-mixin/tests_alerts/test_alerts.yml +++ b/monitoring/ceph-mixin/tests_alerts/test_alerts.yml @@ -5,13 +5,13 @@ tests: # health error - interval: 5m input_series: - - series: 'ceph_health_status{instance="ceph:9283",job="ceph"}' + - series: 'ceph_health_status{instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '2 2 2 2 2 2 2' promql_expr_test: - expr: ceph_health_status == 2 eval_time: 5m exp_samples: - - labels: 'ceph_health_status{instance="ceph:9283",job="ceph"}' + - labels: 'ceph_health_status{instance="ceph:9283",job="ceph",cluster="mycluster"}' value: 2 alert_rule_test: - eval_time: 1m @@ -25,20 +25,21 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.2.1 type: ceph_default severity: critical + cluster: mycluster exp_annotations: - summary: Ceph is in the ERROR state - description: The cluster state has been HEALTH_ERROR for more than 5 minutes. Please check 'ceph health detail' for more information. + summary: Ceph is in the ERROR state on cluster mycluster + description: The cluster state has been HEALTH_ERROR for more than 5 minutes on cluster mycluster. Please check 'ceph health detail' for more information. # health warning - interval: 5m input_series: - - series: 'ceph_health_status{instance="ceph:9283",job="ceph"}' + - series: 'ceph_health_status{instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_status == 1 eval_time: 15m exp_samples: - - labels: 'ceph_health_status{instance="ceph:9283",job="ceph"}' + - labels: 'ceph_health_status{instance="ceph:9283",job="ceph",cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 10m @@ -51,45 +52,46 @@ tests: job: ceph type: ceph_default severity: warning + cluster: mycluster exp_annotations: - summary: Ceph is in the WARNING state - description: The cluster state has been HEALTH_WARN for more than 15 minutes. Please check 'ceph health detail' for more information. + summary: Ceph is in the WARNING state on cluster mycluster + description: The cluster state has been HEALTH_WARN for more than 15 minutes on cluster mycluster. Please check 'ceph health detail' for more information. # 10% OSDs down - interval: 1m input_series: - - series: 'ceph_osd_up{ceph_daemon="osd.0",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.0",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_osd_up{ceph_daemon="osd.1",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.1",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '0 0 0 0 0' - - series: 'ceph_osd_up{ceph_daemon="osd.2",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.2",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1 1 1 1 1' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.0", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",cluster="mycluster",objectstore="bluestore", + public_addr="172.20.0.2"}' values: '1 1 1 1 1' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.1", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",cluster="mycluster",objectstore="bluestore", + public_addr="172.20.0.2"}' values: '1 1 1 1 1' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.2", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",cluster="mycluster",objectstore="bluestore", + public_addr="172.20.0.2"}' values: '1 1 1 1 1' promql_expr_test: - - expr: count(ceph_osd_up == 0) / count(ceph_osd_up) * 100 >= 10 + - expr: count by (cluster) (ceph_osd_up == 0) / count by (cluster) (ceph_osd_up) * 100 >= 10 eval_time: 1m exp_samples: - - labels: '{}' + - labels: '{cluster="mycluster" }' value: 3.333333333333333E+01 alert_rule_test: - eval_time: 1m @@ -99,39 +101,40 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.4.1 type: ceph_default severity: critical + cluster: mycluster exp_annotations: - summary: More than 10% of OSDs are down + summary: More than 10% of OSDs are down on cluster mycluster description: "33.33% or 1 of 3 OSDs are down (>= 10%). The following OSDs are down: - osd.1 on ceph" # flapping OSD - interval: 1s input_series: - - series: 'ceph_osd_up{ceph_daemon="osd.0",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.0",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1+1x100' - - series: 'ceph_osd_up{ceph_daemon="osd.1",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.1",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1+0x100' - - series: 'ceph_osd_up{ceph_daemon="osd.2",instance="ceph:9283",job="ceph"}' + - series: 'ceph_osd_up{ceph_daemon="osd.2",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1+0x100' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.0", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", + public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.1", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", + public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.2", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", + cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", + public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' promql_expr_test: - expr: | @@ -142,7 +145,7 @@ tests: eval_time: 1m exp_samples: - labels: '{ceph_daemon="osd.0", hostname="ceph", instance="ceph:9283", - job="ceph"}' + job="ceph",cluster="mycluster"}' value: 1.2200000000000001E+01 alert_rule_test: - eval_time: 5m @@ -155,54 +158,39 @@ tests: job: ceph oid: 1.3.6.1.4.1.50495.1.2.1.4.4 severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds - summary: Network issues are causing OSDs to flap (mark each other down) + summary: Network issues are causing OSDs to flap (mark each other down) on cluster mycluster description: "OSD osd.0 on ceph was marked down and back up 20.1 times once a minute for 5 minutes. This may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster network, or the public network if no cluster network is deployed. Check the network stats on the listed host(s)." # high pg count deviation - interval: 1m input_series: - - series: 'ceph_osd_numpg{ceph_daemon="osd.0",instance="ceph:9283", - job="ceph"}' + - series: 'ceph_osd_numpg{ceph_daemon="osd.0",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '100 100 100 100 100 160' - - series: 'ceph_osd_numpg{ceph_daemon="osd.1",instance="ceph:9283", - job="ceph"}' + - series: 'ceph_osd_numpg{ceph_daemon="osd.1",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '100 100 100 100 100 320' - - series: 'ceph_osd_numpg{ceph_daemon="osd.2",instance="ceph:9283", - job="ceph"}' + - series: 'ceph_osd_numpg{ceph_daemon="osd.2",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '100 100 100 100 100 160' - - series: 'ceph_osd_numpg{ceph_daemon="osd.3",instance="ceph:9283", - job="ceph"}' + - series: 'ceph_osd_numpg{ceph_daemon="osd.3",instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '100 100 100 100 100 160' - - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.0", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.0",ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)",cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore",public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' - - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.1", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.1",ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)",cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore",public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' - - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.2", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.2",ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)",cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore",public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' - - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.3", - ceph_version="ceph version 17.0.0-189-g3558fd72 - (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)", - cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", - hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore", - public_addr="172.20.0.2"}' + - series: 'ceph_osd_metadata{back_iface="eth0",ceph_daemon="osd.3",ceph_version="ceph version 17.0.0-189-g3558fd72 + (3558fd7291855971aa6481a2ade468ad61fbb346) pacific (dev)",cluster_addr="172.20.0.2",device_class="hdd",front_iface="eth0", + hostname="ceph",instance="ceph:9283",job="ceph",objectstore="bluestore",public_addr="172.20.0.2",cluster="mycluster"}' values: '1 1 1 1 1 1' promql_expr_test: - expr: | @@ -215,8 +203,7 @@ tests: eval_time: 5m exp_samples: - - labels: '{ceph_daemon="osd.1", hostname="ceph", instance="ceph:9283", - job="ceph"}' + - labels: '{ceph_daemon="osd.1", hostname="ceph", instance="ceph:9283",job="ceph",cluster="mycluster"}' value: 6E-01 alert_rule_test: - eval_time: 10m @@ -229,44 +216,40 @@ tests: job: ceph oid: 1.3.6.1.4.1.50495.1.2.1.4.5 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: PGs are not balanced across OSDs + summary: PGs are not balanced across OSDs on cluster mycluster description: "OSD osd.1 on ceph deviates by more than 30% from average PG count." # pgs inactive - interval: 1m input_series: - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="1"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="1"}' values: '1 1 1 1 1 1 1 1' - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="2"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="2"}' values: '1 1 1 1 1 1 1 1' - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="3"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="3"}' values: '1 1 1 1 1 1 1 1' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="1"}' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="1"}' values: '1 1 1 1 1 1 1 1' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="2"}' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="2"}' values: '32 32 32 32 32 32 32 32' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="3"}' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="3"}' values: '33 32 32 32 32 33 33 32' - - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",pool_id="1"}' + - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="1"}' values: '1 1 1 1 1 1 1 1 1' - - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",pool_id="2"}' + - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="2"}' values: '32 32 32 32 32 32 32 32' - - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",pool_id="3"}' + - series: 'ceph_pg_active{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="3"}' values: '32 32 32 32 32 32 32 32' promql_expr_test: - expr: ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_active) > 0 eval_time: 5m exp_samples: - - labels: '{instance="ceph:9283", job="ceph", - name="device_health_metrics", - pool_id="3"}' - value: 1 + - labels: '{instance="ceph:9283", job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="3"}' + value: 1 alert_rule_test: - eval_time: 5m alertname: CephPGsInactive @@ -278,46 +261,39 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.7.1 pool_id: 3 severity: critical + cluster: mycluster type: ceph_default exp_annotations: - summary: One or more placement groups are inactive + summary: One or more placement groups are inactive on cluster mycluster description: "1 PGs have been inactive for more than 5 minutes in pool device_health_metrics. Inactive placement groups are not able to serve read/write requests." #pgs unclean - interval: 1m input_series: - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="1"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="1"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="2"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="2"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="device_health_metrics",pool_id="3"}' + - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph",cluster="mycluster",name="device_health_metrics",pool_id="3"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="1"}' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="1"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="2"}' - values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 - 32 32 32' - - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",pool_id="3"}' - values: '33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 - 33 33' - - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",pool_id="1"}' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="2"}' + values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32' + - series: 'ceph_pg_total{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="3"}' + values: '33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33 33' + - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="1"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",pool_id="2"}' - values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 - 32 32' - - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",pool_id="3"}' - values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 - 32 32' + - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="2"}' + values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32' + - series: 'ceph_pg_clean{instance="ceph:9283",job="ceph",cluster="mycluster",pool_id="3"}' + values: '32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32 32' promql_expr_test: - expr: ceph_pool_metadata * on(pool_id,instance) group_left() (ceph_pg_total - ceph_pg_clean) > 0 eval_time: 15m exp_samples: - - labels: '{instance="ceph:9283", job="ceph", - name="device_health_metrics", pool_id="3"}' + - labels: '{instance="ceph:9283", job="ceph",cluster="mycluster",name="device_health_metrics", pool_id="3"}' value: 1 alert_rule_test: - eval_time: 16m @@ -330,32 +306,33 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.7.2 pool_id: 3 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: One or more placement groups are marked unclean + summary: One or more placement groups are marked unclean on cluster mycluster description: "1 PGs have been unclean for more than 15 minutes in pool device_health_metrics. Unclean PGs have not recovered from a previous failure." # root volume full - interval: 1m input_series: - series: 'node_filesystem_avail_bytes{device="/dev/mapper/fedora_localhost - --live-home",fstype="ext4",instance="node-exporter",job="node-exporter", - mountpoint="/"}' + --live-home",fstype="ext4",instance="node-exporter",job="node-exporter", + mountpoint="/"}' values: '35336400896 35336400896 35336400896 35336400896 35336400896 - 3525385519.104 3533640089' + 3525385519.104 3533640089' - series: 'node_filesystem_size_bytes{device="/dev/mapper/fedora_localhost - --live-home",fstype="ext4",instance="node-exporter",job="node-exporter", - mountpoint="/"}' + --live-home",fstype="ext4",instance="node-exporter",job="node-exporter", + mountpoint="/"}' values: '73445531648 73445531648 73445531648 73445531648 73445531648 - 73445531648 73445531648' + 73445531648 73445531648' promql_expr_test: - expr: node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100 < 5 eval_time: 5m exp_samples: - labels: '{device="/dev/mapper/fedora_localhost --live-home", - fstype="ext4", instance="node-exporter", job="node-exporter", - mountpoint="/"}' + fstype="ext4", instance="node-exporter", job="node-exporter", + mountpoint="/"}' value: 4.8E+00 alert_rule_test: - eval_time: 10m @@ -377,17 +354,13 @@ tests: # network packets dropped - interval: 1m input_series: - - series: 'node_network_receive_drop_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_receive_drop_total{device="eth0",instance="node-exporter",job="node-exporter"}' values: '0+600x10' - - series: 'node_network_transmit_drop_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_transmit_drop_total{device="eth0",instance="node-exporter",job="node-exporter"}' values: '0+600x10' - - series: 'node_network_receive_packets_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_receive_packets_total{device="eth0",instance="node-exporter",job="node-exporter"}' values: '0+750x10' - - series: 'node_network_transmit_packets_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_transmit_packets_total{device="eth0",instance="node-exporter",job="node-exporter"}' values: '0+750x10' promql_expr_test: - expr: | @@ -404,8 +377,7 @@ tests: eval_time: 5m exp_samples: - - labels: '{device="eth0", instance="node-exporter", - job="node-exporter"}' + - labels: '{device="eth0", instance="node-exporter",job="node-exporter"}' value: 8E-1 alert_rule_test: - eval_time: 5m @@ -425,17 +397,13 @@ tests: # network packets errors - interval: 1m input_series: - - series: 'node_network_receive_errs_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_receive_errs_total{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0+600x10' - - series: 'node_network_transmit_errs_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_transmit_errs_total{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0+600x10' - - series: 'node_network_transmit_packets_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_transmit_packets_total{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0+750x10' - - series: 'node_network_receive_packets_total{device="eth0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_network_receive_packets_total{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0+750x10' promql_expr_test: - expr: | @@ -452,8 +420,7 @@ tests: eval_time: 5m exp_samples: - - labels: '{device="eth0", instance="node-exporter", - job="node-exporter"}' + - labels: '{device="eth0", instance="node-exporter",job="node-exporter",cluster="mycluster"}' value: 8E-01 alert_rule_test: - eval_time: 5m @@ -466,26 +433,24 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.8.3 severity: warning type: ceph_default - exp_annotations: - summary: One or more NICs reports packet errors + cluster: mycluster + exp_annotations: + summary: One or more NICs reports packet errors on cluster mycluster description: "Node node-exporter experiences packet errors > 0.01% or > 10 packets/s on interface eth0." # Bond is missing a peer - interval: 1m input_series: - - series: 'node_bonding_active{master="bond0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_bonding_active{master="bond0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '3' - - series: 'node_bonding_slaves{master="bond0", - instance="node-exporter",job="node-exporter"}' + - series: 'node_bonding_slaves{master="bond0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '4' promql_expr_test: - expr: | node_bonding_slaves - node_bonding_active != 0 eval_time: 5m exp_samples: - - labels: '{master="bond0", instance="node-exporter", - job="node-exporter"}' + - labels: '{master="bond0", instance="node-exporter",job="node-exporter",cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 5m @@ -497,23 +462,22 @@ tests: job: node-exporter severity: warning type: ceph_default + cluster: mycluster exp_annotations: - summary: Degraded Bond on Node node-exporter + summary: Degraded Bond on Node node-exporter on cluster mycluster description: "Bond bond0 is degraded on Node node-exporter." # Node Storage disk space filling up - interval: 1m # 20GB = 21474836480, 256MB = 268435456 input_series: - - series: 'node_filesystem_free_bytes{device="/dev/mapper/vg-root", - fstype="xfs",instance="node-1",mountpoint="/rootfs"}' + - series: 'node_filesystem_free_bytes{device="/dev/mapper/vg-root",fstype="xfs",instance="node-1",mountpoint="/rootfs",cluster="mycluster"}' values: '21474836480-268435456x48' - - series: 'node_filesystem_free_bytes{device="/dev/mapper/vg-root", - fstype="xfs",instance="node-2",mountpoint="/rootfs"}' + - series: 'node_filesystem_free_bytes{device="/dev/mapper/vg-root",fstype="xfs",instance="node-2",mountpoint="/rootfs",cluster="mycluster"}' values: '21474836480+0x48' - - series: 'node_uname_info{instance="node-1", nodename="node-1.unittests.com"}' + - series: 'node_uname_info{instance="node-1", nodename="node-1.unittests.com",cluster="mycluster"}' values: 1+0x48 - - series: 'node_uname_info{instance="node-2", nodename="node-2.unittests.com"}' + - series: 'node_uname_info{instance="node-2", nodename="node-2.unittests.com",cluster="mycluster"}' values: 1+0x48 promql_expr_test: - expr: | @@ -521,8 +485,7 @@ tests: on(instance) group_left(nodename) node_uname_info < 0 eval_time: 5m exp_samples: - - labels: '{device="/dev/mapper/vg-root",instance="node-1",fstype="xfs", - mountpoint="/rootfs",nodename="node-1.unittests.com"}' + - labels: '{device="/dev/mapper/vg-root",instance="node-1",fstype="xfs",mountpoint="/rootfs",nodename="node-1.unittests.com",cluster="mycluster"}' value: -1.912602624E+12 alert_rule_test: - eval_time: 5m @@ -537,72 +500,60 @@ tests: instance: node-1 mountpoint: /rootfs nodename: node-1.unittests.com + cluster: mycluster exp_annotations: - summary: Host filesystem free space is getting low + summary: Host filesystem free space is getting low on cluster mycluster description: "Mountpoint /rootfs on node-1.unittests.com will be full in less than 5 days based on the 48 hour trailing fill rate." + # MTU Mismatch - interval: 1m input_series: - - series: 'node_network_mtu_bytes{device="eth0",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1500 1500 1500 1500 1500' - - series: 'node_network_mtu_bytes{device="eth1",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth1",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1500 1500 1500 1500 1500' - - series: 'node_network_mtu_bytes{device="eth2",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth2",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1500 1500 1500 1500 1500' - - series: 'node_network_mtu_bytes{device="eth3",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth3",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1500 1500 1500 1500 1500' - - series: 'node_network_mtu_bytes{device="eth4",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth4",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '9000 9000 9000 9000 9000' - - series: 'node_network_mtu_bytes{device="eth4",instance="hostname1", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth4",instance="hostname1",job="node-exporter",cluster="mycluster"}' values: '2200 2200 2200 2200 2200' - - series: 'node_network_mtu_bytes{device="eth4",instance="hostname2", - job="node-exporter"}' + - series: 'node_network_mtu_bytes{device="eth4",instance="hostname2",job="node-exporter",cluster="mycluster"}' values: '2400 2400 2400 2400 2400' - - series: 'node_network_up{device="eth0",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_up{device="eth0",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0 0 0 0 0' - - series: 'node_network_up{device="eth1",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_up{device="eth1",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '0 0 0 0 0' - - series: 'node_network_up{device="eth2",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_up{device="eth2",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'node_network_up{device="eth3",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_up{device="eth3",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'node_network_up{device="eth4",instance="node-exporter", - job="node-exporter"}' + - series: 'node_network_up{device="eth4",instance="node-exporter",job="node-exporter",cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'node_network_up{device="eth4",instance="hostname1", - job="node-exporter"}' + - series: 'node_network_up{device="eth4",instance="hostname1",job="node-exporter",cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'node_network_up{device="eth4",instance="hostname2", - job="node-exporter"}' + - series: 'node_network_up{device="eth4",instance="hostname2",job="node-exporter",cluster="mycluster"}' values: '0 0 0 0 0' promql_expr_test: - expr: | node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( - max by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != - quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) + max by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != + quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) ) or node_network_mtu_bytes * (node_network_up{device!="lo"} > 0) == scalar( - min by (device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != - quantile by (device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) + min by (cluster,device) (node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) != + quantile by (cluster,device) (.5, node_network_mtu_bytes * (node_network_up{device!="lo"} > 0)) ) eval_time: 1m exp_samples: - - labels: '{device="eth4", instance="node-exporter", job="node-exporter"}' + - labels: '{device="eth4", instance="node-exporter", job="node-exporter", cluster="mycluster"}' value: 9000 - - labels: '{device="eth4", instance="hostname1", job="node-exporter"}' + - labels: '{device="eth4", instance="hostname1", job="node-exporter", cluster="mycluster"}' value: 2200 alert_rule_test: - eval_time: 1m @@ -614,8 +565,9 @@ tests: job: node-exporter severity: warning type: ceph_default + cluster: "mycluster" exp_annotations: - summary: MTU settings across Ceph hosts are inconsistent + summary: MTU settings across Ceph hosts are inconsistent on cluster mycluster description: "Node hostname1 has a different MTU size (2200) than the median of devices named eth4." - exp_labels: device: eth4 @@ -623,51 +575,52 @@ tests: job: node-exporter severity: warning type: ceph_default + cluster: "mycluster" exp_annotations: - summary: MTU settings across Ceph hosts are inconsistent + summary: MTU settings across Ceph hosts are inconsistent on cluster mycluster description: "Node node-exporter has a different MTU size (9000) than the median of devices named eth4." # pool full, data series has 6 but using topk(5) so to ensure the # results are working as expected - interval: 1m input_series: - - series: 'ceph_health_detail{name="POOL_FULL"}' + - series: 'ceph_health_detail{name="POOL_FULL", cluster="mycluster"}' values: '0 0 0 1 1 1 1 1 1 1 1' - - series: 'ceph_pool_percent_used{pool_id="1"}' + - series: 'ceph_pool_percent_used{pool_id="1", cluster="mycluster"}' values: '32+0x10' - - series: 'ceph_pool_percent_used{pool_id="2"}' + - series: 'ceph_pool_percent_used{pool_id="2", cluster="mycluster"}' values: '96+0x10' - - series: 'ceph_pool_percent_used{pool_id="3"}' + - series: 'ceph_pool_percent_used{pool_id="3", cluster="mycluster"}' values: '90+0x10' - - series: 'ceph_pool_percent_used{pool_id="4"}' + - series: 'ceph_pool_percent_used{pool_id="4", cluster="mycluster"}' values: '72+0x10' - - series: 'ceph_pool_percent_used{pool_id="5"}' + - series: 'ceph_pool_percent_used{pool_id="5", cluster="mycluster"}' values: '19+0x10' - - series: 'ceph_pool_percent_used{pool_id="6"}' + - series: 'ceph_pool_percent_used{pool_id="6", cluster="mycluster"}' values: '10+0x10' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="cephfs_data",pool_id="1"}' + name="cephfs_data",pool_id="1", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="rbd",pool_id="2"}' + name="rbd",pool_id="2", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="iscsi",pool_id="3"}' + name="iscsi",pool_id="3", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="default.rgw.index",pool_id="4"}' + name="default.rgw.index",pool_id="4", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="default.rgw.log",pool_id="5"}' + name="default.rgw.log",pool_id="5", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' - series: 'ceph_pool_metadata{instance="ceph:9283",job="ceph", - name="dummy",pool_id="6"}' + name="dummy",pool_id="6", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="POOL_FULL"} > 0 eval_time: 5m exp_samples: - - labels: '{__name__="ceph_health_detail", name="POOL_FULL"}' + - labels: '{__name__="ceph_health_detail", name="POOL_FULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -678,23 +631,24 @@ tests: - exp_labels: name: POOL_FULL severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.9.1 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pool-full - summary: Pool is full - writes are blocked + summary: Pool is full - writes are blocked on cluster mycluster description: "A pool has reached its MAX quota, or OSDs supporting the pool have reached the FULL threshold. Until this is resolved, writes to the pool will be blocked. Pool Breakdown (top 5) - rbd at 96% - iscsi at 90% - default.rgw.index at 72% - cephfs_data at 32% - default.rgw.log at 19% Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>)" + # slow OSD ops - interval : 1m input_series: - - series: 'ceph_healthcheck_slow_ops{instance="ceph:9283",job="ceph"}' + - series: 'ceph_healthcheck_slow_ops{instance="ceph:9283",job="ceph",cluster="mycluster"}' values: '1+0x120' promql_expr_test: - expr: ceph_healthcheck_slow_ops > 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_healthcheck_slow_ops", instance="ceph:9283", - job="ceph"}' + - labels: '{__name__="ceph_healthcheck_slow_ops", instance="ceph:9283",job="ceph",cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 20m @@ -704,23 +658,23 @@ tests: instance: ceph:9283 job: ceph severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops - summary: OSD operations are slow to complete + summary: OSD operations are slow to complete on cluster mycluster description: "1 OSD requests are taking too long to process (osd_op_complaint_time exceeded)" # slow daemon ops - interval : 1m input_series: - - series: 'ceph_daemon_health_metrics{ceph_daemon="osd.1", instance="ceph:9283",job="ceph", type="SLOW_OPS"}' + - series: 'ceph_daemon_health_metrics{ceph_daemon="osd.1", instance="ceph:9283", job="ceph", type="SLOW_OPS", cluster="mycluster"}' values: '1+0x120' promql_expr_test: - expr: 'ceph_daemon_health_metrics{type="SLOW_OPS"} > 0' eval_time: 1m exp_samples: - - labels: '{__name__="ceph_daemon_health_metrics", ceph_daemon="osd.1",instance="ceph:9283", - job="ceph", type="SLOW_OPS"}' + - labels: '{__name__="ceph_daemon_health_metrics", ceph_daemon="osd.1", instance="ceph:9283", cluster="mycluster",job="ceph", type="SLOW_OPS"}' value: 1 alert_rule_test: - eval_time: 20m @@ -731,22 +685,23 @@ tests: ceph_daemon: "osd.1" job: ceph severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#slow-ops - summary: osd.1 operations are slow to complete + summary: osd.1 operations are slow to complete on cluster mycluster description: "osd.1 operations are taking too long to process (complaint time exceeded)" # CEPHADM orchestrator alert triggers - interval: 30s input_series: - - series: 'ceph_health_detail{name="UPGRADE_EXCEPTION"}' + - series: 'ceph_health_detail{name="UPGRADE_EXCEPTION", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="UPGRADE_EXCEPTION"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="UPGRADE_EXCEPTION"}' + - labels: '{__name__="ceph_health_detail", name="UPGRADE_EXCEPTION", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -757,20 +712,21 @@ tests: - exp_labels: name: UPGRADE_EXCEPTION severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.11.2 exp_annotations: - summary: Ceph version upgrade has failed + summary: Ceph version upgrade has failed on cluster mycluster description: "The cephadm cluster upgrade process has failed. The cluster remains in an undetermined state. Please review the cephadm logs, to understand the nature of the issue" - interval: 30s input_series: - - series: 'ceph_health_detail{name="CEPHADM_FAILED_DAEMON"}' + - series: 'ceph_health_detail{name="CEPHADM_FAILED_DAEMON", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="CEPHADM_FAILED_DAEMON"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="CEPHADM_FAILED_DAEMON"}' + - labels: '{__name__="ceph_health_detail", name="CEPHADM_FAILED_DAEMON", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -781,20 +737,21 @@ tests: - exp_labels: name: CEPHADM_FAILED_DAEMON severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.11.1 exp_annotations: - summary: A ceph daemon managed by cephadm is down + summary: A ceph daemon managed by cephadm is down on cluster mycluster description: "A daemon managed by cephadm is no longer active. Determine, which daemon is down with 'ceph health detail'. you may start daemons with the 'ceph orch daemon start <daemon_id>'" - interval: 1m input_series: - - series: 'ceph_health_detail{name="CEPHADM_PAUSED"}' + - series: 'ceph_health_detail{name="CEPHADM_PAUSED", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="CEPHADM_PAUSED"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="CEPHADM_PAUSED"}' + - labels: '{__name__="ceph_health_detail", name="CEPHADM_PAUSED", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -805,21 +762,23 @@ tests: - exp_labels: name: CEPHADM_PAUSED severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/cephadm/operations#cephadm-paused - summary: Orchestration tasks via cephadm are PAUSED + summary: Orchestration tasks via cephadm are PAUSED on cluster mycluster description: "Cluster management has been paused manually. This will prevent the orchestrator from service management and reconciliation. If this is not intentional, resume cephadm operations with 'ceph orch resume'" + # MDS - interval: 1m input_series: - - series: 'ceph_health_detail{name="MDS_DAMAGE"}' + - series: 'ceph_health_detail{name="MDS_DAMAGE", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="MDS_DAMAGE"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MDS_DAMAGE"}' + - labels: '{__name__="ceph_health_detail", name="MDS_DAMAGE", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -830,21 +789,22 @@ tests: - exp_labels: name: MDS_DAMAGE severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.5.1 exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages - summary: CephFS filesystem is damaged. + summary: CephFS filesystem is damaged on cluster mycluster description: "Filesystem metadata has been corrupted. Data may be inaccessible. Analyze metrics from the MDS daemon admin socket, or escalate to support." - interval: 1m input_series: - - series: 'ceph_health_detail{name="MDS_HEALTH_READ_ONLY"}' + - series: 'ceph_health_detail{name="MDS_HEALTH_READ_ONLY", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="MDS_HEALTH_READ_ONLY"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MDS_HEALTH_READ_ONLY"}' + - labels: '{__name__="ceph_health_detail", name="MDS_HEALTH_READ_ONLY", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -855,21 +815,22 @@ tests: - exp_labels: name: MDS_HEALTH_READ_ONLY severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.5.2 exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages#cephfs-health-messages - summary: CephFS filesystem in read only mode due to write error(s) + summary: CephFS filesystem in read only mode due to write error(s) on cluster mycluster description: "The filesystem has switched to READ ONLY due to an unexpected error when writing to the metadata pool. Either analyze the output from the MDS daemon admin socket, or escalate to support." - interval: 1m input_series: - - series: 'ceph_health_detail{name="MDS_ALL_DOWN"}' + - series: 'ceph_health_detail{name="MDS_ALL_DOWN", cluster="mycluster"}' values: '0 0 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="MDS_ALL_DOWN"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MDS_ALL_DOWN"}' + - labels: '{__name__="ceph_health_detail", name="MDS_ALL_DOWN", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -880,21 +841,22 @@ tests: - exp_labels: name: MDS_ALL_DOWN severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.5.3 exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-all-down - summary: CephFS filesystem is offline + summary: CephFS filesystem is offline on cluster mycluster description: "All MDS ranks are unavailable. The MDS daemons managing metadata are down, rendering the filesystem offline." - interval: 1m input_series: - - series: 'ceph_health_detail{name="FS_DEGRADED"}' + - series: 'ceph_health_detail{name="FS_DEGRADED", cluster="mycluster"}' values: '0 0 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="FS_DEGRADED"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="FS_DEGRADED"}' + - labels: '{__name__="ceph_health_detail", name="FS_DEGRADED", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -905,21 +867,22 @@ tests: - exp_labels: name: FS_DEGRADED severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.5.4 exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-degraded - summary: CephFS filesystem is degraded + summary: CephFS filesystem is degraded on cluster mycluster description: "One or more metadata daemons (MDS ranks) are failed or in a damaged state. At best the filesystem is partially available, at worst the filesystem is completely unusable." - interval: 1m input_series: - - series: 'ceph_health_detail{name="MDS_INSUFFICIENT_STANDBY"}' + - series: 'ceph_health_detail{name="MDS_INSUFFICIENT_STANDBY", cluster="mycluster"}' values: '0 0 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="MDS_INSUFFICIENT_STANDBY"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MDS_INSUFFICIENT_STANDBY"}' + - labels: '{__name__="ceph_health_detail", name="MDS_INSUFFICIENT_STANDBY", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -930,20 +893,21 @@ tests: - exp_labels: name: MDS_INSUFFICIENT_STANDBY severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-insufficient-standby - summary: Ceph filesystem standby daemons too few + summary: Ceph filesystem standby daemons too few on cluster mycluster description: "The minimum number of standby daemons required by standby_count_wanted is less than the current number of standby daemons. Adjust the standby count or increase the number of MDS daemons." - interval: 1m input_series: - - series: 'ceph_health_detail{name="FS_WITH_FAILED_MDS"}' + - series: 'ceph_health_detail{name="FS_WITH_FAILED_MDS", cluster="mycluster"}' values: '0 0 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="FS_WITH_FAILED_MDS"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="FS_WITH_FAILED_MDS"}' + - labels: '{__name__="ceph_health_detail", name="FS_WITH_FAILED_MDS", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -954,21 +918,22 @@ tests: - exp_labels: name: FS_WITH_FAILED_MDS severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.5.5 exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#fs-with-failed-mds - summary: MDS daemon failed, no further standby available + summary: MDS daemon failed, no further standby available on cluster mycluster description: "An MDS daemon has failed, leaving only one active rank and no available standby. Investigate the cause of the failure or add a standby MDS." - interval: 1m input_series: - - series: 'ceph_health_detail{name="MDS_UP_LESS_THAN_MAX"}' + - series: 'ceph_health_detail{name="MDS_UP_LESS_THAN_MAX", cluster="mycluster"}' values: '0 0 1 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="MDS_UP_LESS_THAN_MAX"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MDS_UP_LESS_THAN_MAX"}' + - labels: '{__name__="ceph_health_detail", name="MDS_UP_LESS_THAN_MAX", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -979,10 +944,11 @@ tests: - exp_labels: name: MDS_UP_LESS_THAN_MAX severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/cephfs/health-messages/#mds-up-less-than-max - summary: Ceph MDS daemon count is lower than configured + summary: Ceph MDS daemon count is lower than configured on cluster mycluster description: "The filesystem's 'max_mds' setting defines the number of MDS ranks in the filesystem. The current number of active MDS daemons is less than this value." # MGR - interval: 1m @@ -1012,13 +978,13 @@ tests: description: "The mgr/prometheus module at ceph-mgr:9283 is unreachable. This could mean that the module has been disabled or the mgr daemon itself is down. Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to an admin node or toolbox pod and use 'ceph -s' to to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can determine module status with 'ceph mgr module ls'. If it is not listed as enabled, enable it with 'ceph mgr module enable prometheus'." - interval: 1m input_series: - - series: 'ceph_health_detail{name="RECENT_MGR_MODULE_CRASH"}' + - series: 'ceph_health_detail{name="RECENT_MGR_MODULE_CRASH", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="RECENT_MGR_MODULE_CRASH"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="RECENT_MGR_MODULE_CRASH"}' + - labels: '{__name__="ceph_health_detail", name="RECENT_MGR_MODULE_CRASH", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1029,24 +995,26 @@ tests: - exp_labels: name: RECENT_MGR_MODULE_CRASH severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.6.1 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#recent-mgr-module-crash - summary: A manager module has recently crashed + summary: A manager module has recently crashed on cluster mycluster description: "One or more mgr modules have crashed and have yet to be acknowledged by an administrator. A crashed module may impact functionality within the cluster. Use the 'ceph crash' command to determine which module has failed, and archive it to acknowledge the failure." + # MON - interval: 1m input_series: - - series: 'ceph_health_detail{name="MON_DISK_CRIT"}' + - series: 'ceph_health_detail{name="MON_DISK_CRIT", cluster="mycluster"}' values: '0+0x2 1+0x10' - - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-a"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-a", cluster="mycluster"}' values: '1+0x13' promql_expr_test: - expr: ceph_health_detail{name="MON_DISK_CRIT"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MON_DISK_CRIT"}' + - labels: '{__name__="ceph_health_detail", name="MON_DISK_CRIT", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1057,23 +1025,24 @@ tests: - exp_labels: name: "MON_DISK_CRIT" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.3.2 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-crit - summary: Filesystem space on at least one monitor is critically low + summary: Filesystem space on at least one monitor is critically low on cluster mycluster description: "The free space available to a monitor's store is critically low. You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; - ceph-mon-a" - interval: 1m input_series: - - series: 'ceph_health_detail{name="MON_DISK_LOW"}' + - series: 'ceph_health_detail{name="MON_DISK_LOW", cluster="mycluster"}' values: '0+0x2 1+0x10' - - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-a"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-a", cluster="mycluster"}' values: '1+0x13' promql_expr_test: - expr: ceph_health_detail{name="MON_DISK_LOW"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MON_DISK_LOW"}' + - labels: '{__name__="ceph_health_detail", name="MON_DISK_LOW", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1084,20 +1053,21 @@ tests: - exp_labels: name: "MON_DISK_LOW" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-disk-low - summary: Drive space on at least one monitor is approaching full + summary: Drive space on at least one monitor is approaching full on cluster mycluster description: "The space available to a monitor's store is approaching full (>70% is the default). You should increase the space available to the monitor(s). The default directory is /var/lib/ceph/mon-*/data/store.db on traditional deployments, and /var/lib/rook/mon-*/data/store.db on the mon pod's worker node for Rook. Look for old, rotated versions of *.log and MANIFEST*. Do NOT touch any *.sst files. Also check any other directories under /var/lib/rook and other directories on the same filesystem, often /var/log and /var/tmp are culprits. Your monitor hosts are; - ceph-mon-a" - interval: 1m input_series: - - series: 'ceph_health_detail{name="MON_CLOCK_SKEW"}' + - series: 'ceph_health_detail{name="MON_CLOCK_SKEW", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="MON_CLOCK_SKEW"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="MON_CLOCK_SKEW"}' + - labels: '{__name__="ceph_health_detail", name="MON_CLOCK_SKEW", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1108,34 +1078,35 @@ tests: - exp_labels: name: "MON_CLOCK_SKEW" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-clock-skew - summary: Clock skew detected among monitors + summary: Clock skew detected among monitors on cluster mycluster description: "Ceph monitors rely on closely synchronized time to maintain quorum and cluster consistency. This event indicates that the time on at least one mon has drifted too far from the lead mon. Review cluster status with ceph -s. This will show which monitors are affected. Check the time sync status on each monitor host with 'ceph time-sync-status' and the state and peers of your ntpd or chrony daemon." # Check 3 mons one down, quorum at risk - interval: 1m input_series: - - series: 'ceph_health_detail{name="MON_DOWN"}' + - series: 'ceph_health_detail{name="MON_DOWN", cluster="mycluster"}' values: '0+0x2 1+0x12' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.a"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.a", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.b"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.b", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.c"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.c", cluster="mycluster"}' values: '1+0x2 0+0x12' - - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-1"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-1", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.b", hostname="ceph-mon-2"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.b", hostname="ceph-mon-2", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.c", hostname="ceph-mon-3"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.c", hostname="ceph-mon-3", cluster="mycluster"}' values: '1+0x14' promql_expr_test: - - expr: ((ceph_health_detail{name="MON_DOWN"} == 1) * on() (count(ceph_mon_quorum_status == 1) == bool (floor(count(ceph_mon_metadata) / 2) + 1))) == 1 + - expr: ((ceph_health_detail{name="MON_DOWN"} == 1) * on() group_right(cluster) (count(ceph_mon_quorum_status == 1) by (cluster) == bool (floor(count(ceph_mon_metadata) by (cluster) / 2) + 1))) == 1 eval_time: 3m exp_samples: - - labels: '{}' + - labels: '{cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1146,40 +1117,41 @@ tests: exp_alerts: - exp_labels: severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.3.1 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down - summary: Monitor quorum is at risk + summary: Monitor quorum is at risk on cluster mycluster description: "Quorum requires a majority of monitors (x 2) to be active. Without quorum the cluster will become inoperable, affecting all services and connected clients. The following monitors are down: - mon.c on ceph-mon-3" # check 5 mons, 1 down - warning only - interval: 1m input_series: - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.a"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.a", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.b"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.b", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.c"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.c", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.d"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.d", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_quorum_status{ceph_daemon="mon.e"}' + - series: 'ceph_mon_quorum_status{ceph_daemon="mon.e", cluster="mycluster"}' values: '1+0x2 0+0x12' - - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-1"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.a", hostname="ceph-mon-1", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.b", hostname="ceph-mon-2"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.b", hostname="ceph-mon-2", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.c", hostname="ceph-mon-3"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.c", hostname="ceph-mon-3", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.d", hostname="ceph-mon-4"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.d", hostname="ceph-mon-4", cluster="mycluster"}' values: '1+0x14' - - series: 'ceph_mon_metadata{ceph_daemon="mon.e", hostname="ceph-mon-5"}' + - series: 'ceph_mon_metadata{ceph_daemon="mon.e", hostname="ceph-mon-5", cluster="mycluster"}' values: '1+0x14' promql_expr_test: - - expr: (count(ceph_mon_quorum_status == 0) <= (count(ceph_mon_metadata) - floor(count(ceph_mon_metadata) / 2) + 1)) + - expr: (count by (cluster) (ceph_mon_quorum_status == 0)) <= (count by (cluster) (ceph_mon_metadata) - floor((count by (cluster) (ceph_mon_metadata) / 2 + 1))) eval_time: 3m exp_samples: - - labels: '{}' + - labels: '{cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1189,21 +1161,23 @@ tests: exp_alerts: - exp_labels: severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#mon-down - summary: One or more monitors down - description: "You have 1 monitor down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: - mon.e on ceph-mon-5\n" + summary: One or more monitors down on cluster mycluster + description: "You have 1 monitor down. Quorum is still intact, but the loss of an additional monitor will make your cluster inoperable. The following monitors are down: - mon.e on ceph-mon-5" + # Device Health - interval: 1m input_series: - - series: 'ceph_health_detail{name="DEVICE_HEALTH"}' + - series: 'ceph_health_detail{name="DEVICE_HEALTH", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="DEVICE_HEALTH"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH"}' + - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1214,20 +1188,21 @@ tests: - exp_labels: name: "DEVICE_HEALTH" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#id2 - summary: Device(s) predicted to fail soon + summary: Device(s) predicted to fail soon on cluster mycluster description: "The device health module has determined that one or more devices will fail soon. To review device status use 'ceph device ls'. To show a specific device use 'ceph device info <dev id>'. Mark the OSD out so that data may migrate to other OSDs. Once the OSD has drained, destroy the OSD, replace the device, and redeploy the OSD." - interval: 1m input_series: - - series: 'ceph_health_detail{name="DEVICE_HEALTH_TOOMANY"}' + - series: 'ceph_health_detail{name="DEVICE_HEALTH_TOOMANY", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="DEVICE_HEALTH_TOOMANY"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH_TOOMANY"}' + - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH_TOOMANY", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1238,21 +1213,22 @@ tests: - exp_labels: name: "DEVICE_HEALTH_TOOMANY" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.4.7 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-toomany - summary: Too many devices are predicted to fail, unable to resolve + summary: Too many devices are predicted to fail on cluster mycluster, unable to resolve description: "The device health module has determined that devices predicted to fail can not be remediated automatically, since too many OSDs would be removed from the cluster to ensure performance and availability. Prevent data integrity issues by adding new OSDs so that data may be relocated." - interval: 1m input_series: - - series: 'ceph_health_detail{name="DEVICE_HEALTH_IN_USE"}' + - series: 'ceph_health_detail{name="DEVICE_HEALTH_IN_USE", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="DEVICE_HEALTH_IN_USE"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH_IN_USE"}' + - labels: '{__name__="ceph_health_detail", name="DEVICE_HEALTH_IN_USE", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1263,25 +1239,27 @@ tests: - exp_labels: name: "DEVICE_HEALTH_IN_USE" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#device-health-in-use - summary: Device failure is predicted, but unable to relocate data + summary: Device failure is predicted, but unable to relocate data on cluster mycluster description: "The device health module has determined that one or more devices will fail soon, but the normal process of relocating the data on the device to other OSDs in the cluster is blocked. \nEnsure that the cluster has available free space. It may be necessary to add capacity to the cluster to allow data from the failing device to successfully migrate, or to enable the balancer." + # OSD - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_HOST_DOWN"}' + - series: 'ceph_health_detail{name="OSD_HOST_DOWN", cluster="mycluster"}' values: '0+0x2 1+0x10' - - series: 'ceph_osd_up{ceph_daemon="osd.0"}' + - series: 'ceph_osd_up{ceph_daemon="osd.0", cluster="mycluster"}' values: '1+0x2 0+0x10' - - series: 'ceph_osd_metadata{ceph_daemon="osd.0", hostname="ceph-osd-1"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.0", hostname="ceph-osd-1", cluster="mycluster"}' values: '1+0x12' promql_expr_test: - expr: ceph_health_detail{name="OSD_HOST_DOWN"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_HOST_DOWN"}' + - labels: '{__name__="ceph_health_detail", name="OSD_HOST_DOWN", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1292,20 +1270,21 @@ tests: - exp_labels: name: "OSD_HOST_DOWN" severity: warning + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.4.8 exp_annotations: - summary: An OSD host is offline + summary: An OSD host is offline on cluster mycluster description: "The following OSDs are down: - ceph-osd-1 : osd.0" - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_SLOW_PING_TIME_FRONT"}' + - series: 'ceph_health_detail{name="OSD_SLOW_PING_TIME_FRONT", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="OSD_SLOW_PING_TIME_FRONT"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_SLOW_PING_TIME_FRONT"}' + - labels: '{__name__="ceph_health_detail", name="OSD_SLOW_PING_TIME_FRONT", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1316,19 +1295,20 @@ tests: - exp_labels: name: "OSD_SLOW_PING_TIME_FRONT" severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: Network issues delaying OSD heartbeats (public network) + summary: Network issues delaying OSD heartbeats (public network) on cluster mycluster description: "OSD heartbeats on the cluster's 'public' network (frontend) are running slow. Investigate the network for latency or loss issues. Use 'ceph health detail' to show the affected OSDs." - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_SLOW_PING_TIME_BACK"}' + - series: 'ceph_health_detail{name="OSD_SLOW_PING_TIME_BACK", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="OSD_SLOW_PING_TIME_BACK"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_SLOW_PING_TIME_BACK"}' + - labels: '{__name__="ceph_health_detail", name="OSD_SLOW_PING_TIME_BACK", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1339,19 +1319,20 @@ tests: - exp_labels: name: "OSD_SLOW_PING_TIME_BACK" severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: Network issues delaying OSD heartbeats (cluster network) + summary: Network issues delaying OSD heartbeats (cluster network) on cluster mycluster description: "OSD heartbeats on the cluster's 'cluster' network (backend) are slow. Investigate the network for latency issues on this subnet. Use 'ceph health detail' to show the affected OSDs." - interval: 1m input_series: - - series: 'ceph_health_detail{name="BLUESTORE_DISK_SIZE_MISMATCH"}' + - series: 'ceph_health_detail{name="BLUESTORE_DISK_SIZE_MISMATCH", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="BLUESTORE_DISK_SIZE_MISMATCH"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="BLUESTORE_DISK_SIZE_MISMATCH"}' + - labels: '{__name__="ceph_health_detail", name="BLUESTORE_DISK_SIZE_MISMATCH", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1362,20 +1343,21 @@ tests: - exp_labels: name: "BLUESTORE_DISK_SIZE_MISMATCH" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-disk-size-mismatch - summary: OSD size inconsistency error + summary: OSD size inconsistency error on cluster mycluster description: "One or more OSDs have an internal inconsistency between metadata and the size of the device. This could lead to the OSD(s) crashing in future. You should redeploy the affected OSDs." - interval: 30s input_series: - - series: 'ceph_health_detail{name="BLUESTORE_SPURIOUS_READ_ERRORS"}' + - series: 'ceph_health_detail{name="BLUESTORE_SPURIOUS_READ_ERRORS", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="BLUESTORE_SPURIOUS_READ_ERRORS"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="BLUESTORE_SPURIOUS_READ_ERRORS"}' + - labels: '{__name__="ceph_health_detail", name="BLUESTORE_SPURIOUS_READ_ERRORS", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1386,32 +1368,33 @@ tests: - exp_labels: name: "BLUESTORE_SPURIOUS_READ_ERRORS" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#bluestore-spurious-read-errors - summary: Device read errors detected + summary: Device read errors detected on cluster mycluster description: "An OSD has encountered read errors, but the OSD has recovered by retrying the reads. This may indicate an issue with hardware or the kernel." - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_DOWN"}' + - series: 'ceph_health_detail{name="OSD_DOWN", cluster="mycluster"}' values: '0+0x2 1+0x10' - - series: 'ceph_osd_up{ceph_daemon="osd.0"}' + - series: 'ceph_osd_up{ceph_daemon="osd.0", cluster="mycluster"}' values: '1+0x12' - - series: 'ceph_osd_up{ceph_daemon="osd.1"}' + - series: 'ceph_osd_up{ceph_daemon="osd.1", cluster="mycluster"}' values: '1+0x2 0+0x10' - - series: 'ceph_osd_up{ceph_daemon="osd.2"}' + - series: 'ceph_osd_up{ceph_daemon="osd.2", cluster="mycluster"}' values: '1+0x12' - - series: 'ceph_osd_metadata{ceph_daemon="osd.0", hostname="ceph-osd-1"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.0", hostname="ceph-osd-1", cluster="mycluster"}' values: '1+0x12' - - series: 'ceph_osd_metadata{ceph_daemon="osd.1", hostname="ceph-osd-2"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.1", hostname="ceph-osd-2", cluster="mycluster"}' values: '1+0x12' - - series: 'ceph_osd_metadata{ceph_daemon="osd.2", hostname="ceph-osd-3"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.2", hostname="ceph-osd-3", cluster="mycluster"}' values: '1+0x12' promql_expr_test: - expr: ceph_health_detail{name="OSD_DOWN"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_DOWN"}' + - labels: '{__name__="ceph_health_detail", name="OSD_DOWN", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1422,21 +1405,22 @@ tests: - exp_labels: name: "OSD_DOWN" severity: warning + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.4.2 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-down - summary: An OSD has been marked down - description: "1 OSD down for over 5mins. The following OSD is down: - osd.1 on ceph-osd-2\n" + summary: An OSD has been marked down on cluster mycluster + description: "1 OSD down for over 5mins. The following OSD is down: - osd.1 on ceph-osd-2" - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_NEARFULL"}' + - series: 'ceph_health_detail{name="OSD_NEARFULL", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="OSD_NEARFULL"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_NEARFULL"}' + - labels: '{__name__="ceph_health_detail", name="OSD_NEARFULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1447,21 +1431,22 @@ tests: - exp_labels: name: "OSD_NEARFULL" severity: warning + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.4.3 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-nearfull - summary: OSD(s) running low on free space (NEARFULL) + summary: OSD(s) running low on free space (NEARFULL) on cluster mycluster description: One or more OSDs have reached the NEARFULL threshold. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data. - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_FULL"}' + - series: 'ceph_health_detail{name="OSD_FULL", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="OSD_FULL"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_FULL"}' + - labels: '{__name__="ceph_health_detail", name="OSD_FULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1472,21 +1457,22 @@ tests: - exp_labels: name: "OSD_FULL" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.4.6 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-full - summary: OSD full, writes blocked + summary: OSD full, writes blocked on cluster mycluster description: An OSD has reached the FULL threshold. Writes to pools that share the affected OSD will be blocked. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data. - interval: 1m input_series: - - series: 'ceph_health_detail{name="OSD_BACKFILLFULL"}' + - series: 'ceph_health_detail{name="OSD_BACKFILLFULL", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="OSD_BACKFILLFULL"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_BACKFILLFULL"}' + - labels: '{__name__="ceph_health_detail", name="OSD_BACKFILLFULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1497,20 +1483,21 @@ tests: - exp_labels: name: "OSD_BACKFILLFULL" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-backfillfull - summary: OSD(s) too full for backfill operations + summary: OSD(s) too full for backfill operations on cluster mycluster description: "An OSD has reached the BACKFILL FULL threshold. This will prevent rebalance operations from completing. Use 'ceph health detail' and 'ceph osd df' to identify the problem. To resolve, add capacity to the affected OSD's failure domain, restore down/out OSDs, or delete unwanted data." - interval: 30s input_series: - - series: 'ceph_health_detail{name="OSD_TOO_MANY_REPAIRS"}' + - series: 'ceph_health_detail{name="OSD_TOO_MANY_REPAIRS", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="OSD_TOO_MANY_REPAIRS"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="OSD_TOO_MANY_REPAIRS"}' + - labels: '{__name__="ceph_health_detail", name="OSD_TOO_MANY_REPAIRS", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1521,38 +1508,40 @@ tests: - exp_labels: name: "OSD_TOO_MANY_REPAIRS" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#osd-too-many-repairs - summary: OSD reports a high number of read errors + summary: OSD reports a high number of read errors on cluster mycluster description: Reads from an OSD have used a secondary PG to return data to the client, indicating a potential failing drive. + # Pools # trigger percent full prediction on pools 1 and 2 only - interval: 12h input_series: - - series: 'ceph_pool_percent_used{pool_id="1", instance="9090"}' + - series: 'ceph_pool_percent_used{pool_id="1", instance="9090", cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_pool_percent_used{pool_id="1", instance="8090"}' + - series: 'ceph_pool_percent_used{pool_id="1", instance="8090", cluster="mycluster"}' values: '78 89 79 98 78' - - series: 'ceph_pool_percent_used{pool_id="2", instance="9090"}' + - series: 'ceph_pool_percent_used{pool_id="2", instance="9090", cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_pool_percent_used{pool_id="2", instance="8090"}' + - series: 'ceph_pool_percent_used{pool_id="2", instance="8090", cluster="mycluster"}' values: '22 22 23 23 24' - - series: 'ceph_pool_metadata{pool_id="1" , instance="9090" ,name="rbd",type="replicated"}' + - series: 'ceph_pool_metadata{pool_id="1" , instance="9090" ,name="rbd",type="replicated", cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_pool_metadata{pool_id="1", instance="8090",name="default.rgw.index",type="replicated"}' + - series: 'ceph_pool_metadata{pool_id="1", instance="8090",name="default.rgw.index",type="replicated", cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_pool_metadata{pool_id="2" , instance="9090" ,name="rbd",type="replicated"}' + - series: 'ceph_pool_metadata{pool_id="2" , instance="9090" ,name="rbd",type="replicated", cluster="mycluster"}' values: '1 1 1 1 1' - - series: 'ceph_pool_metadata{pool_id="2", instance="8090",name="default.rgw.index",type="replicated"}' + - series: 'ceph_pool_metadata{pool_id="2", instance="8090",name="default.rgw.index",type="replicated", cluster="mycluster"}' values: '1 1 1 1 1' promql_expr_test: - expr: | - (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id, instance) + (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(cluster, pool_id, instance) group_right() ceph_pool_metadata) >= 95 eval_time: 36h exp_samples: - - labels: '{instance="8090",name="default.rgw.index",pool_id="1",type="replicated"}' + - labels: '{instance="8090",name="default.rgw.index",pool_id="1",type="replicated", cluster="mycluster"}' value: 1.435E+02 # 142% alert_rule_test: - eval_time: 48h @@ -1563,20 +1552,21 @@ tests: name: default.rgw.index pool_id: 1 severity: warning + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.9.2 exp_annotations: - summary: Pool growth rate may soon exceed capacity + summary: Pool growth rate may soon exceed capacity on cluster mycluster description: Pool 'default.rgw.index' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours. - interval: 1m input_series: - - series: 'ceph_health_detail{name="POOL_BACKFILLFULL"}' + - series: 'ceph_health_detail{name="POOL_BACKFILLFULL", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="POOL_BACKFILLFULL"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="POOL_BACKFILLFULL"}' + - labels: '{__name__="ceph_health_detail", name="POOL_BACKFILLFULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1587,20 +1577,21 @@ tests: - exp_labels: name: "POOL_BACKFILLFULL" severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: Free space in a pool is too low for recovery/backfill + summary: Free space in a pool is too low for recovery/backfill on cluster mycluster description: A pool is approaching the near full threshold, which will prevent recovery/backfill operations from completing. Consider adding more capacity. - interval: 1m input_series: - - series: 'ceph_health_detail{name="POOL_NEAR_FULL"}' + - series: 'ceph_health_detail{name="POOL_NEAR_FULL", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="POOL_NEAR_FULL"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="POOL_NEAR_FULL"}' + - labels: '{__name__="ceph_health_detail", name="POOL_NEAR_FULL", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1611,21 +1602,22 @@ tests: - exp_labels: name: "POOL_NEAR_FULL" severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: One or more Ceph pools are nearly full + summary: One or more Ceph pools are nearly full on cluster mycluster description: "A pool has exceeded the warning (percent full) threshold, or OSDs supporting the pool have reached the NEARFULL threshold. Writes may continue, but you are at risk of the pool going read-only if more capacity isn't made available. Determine the affected pool with 'ceph df detail', looking at QUOTA BYTES and STORED. Increase the pool's quota, or add capacity to the cluster first then increase the pool's quota (e.g. ceph osd pool set quota <pool_name> max_bytes <bytes>). Also ensure that the balancer is active." # PGs - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_NOT_SCRUBBED"}' + - series: 'ceph_health_detail{name="PG_NOT_SCRUBBED",cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="PG_NOT_SCRUBBED"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="PG_NOT_SCRUBBED"}' + - labels: '{__name__="ceph_health_detail", name="PG_NOT_SCRUBBED", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1636,20 +1628,21 @@ tests: - exp_labels: name: "PG_NOT_SCRUBBED" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-scrubbed - summary: Placement group(s) have not been scrubbed + summary: Placement group(s) have not been scrubbed on cluster mycluster description: "One or more PGs have not been scrubbed recently. Scrubs check metadata integrity, protecting against bit-rot. They check that metadata is consistent across data replicas. When PGs miss their scrub interval, it may indicate that the scrub window is too small, or PGs were not in a 'clean' state during the scrub window. You can manually initiate a scrub with: ceph pg scrub <pgid>" - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_DAMAGED"}' + - series: 'ceph_health_detail{name="PG_DAMAGED",cluster="mycluster"}' values: '0+0x4 1+0x20' promql_expr_test: - expr: ceph_health_detail{name=~"PG_DAMAGED|OSD_SCRUB_ERRORS"} == 1 eval_time: 5m exp_samples: - - labels: '{__name__="ceph_health_detail", name="PG_DAMAGED"}' + - labels: '{__name__="ceph_health_detail", name="PG_DAMAGED", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1660,21 +1653,22 @@ tests: - exp_labels: name: "PG_DAMAGED" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.7.4 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-damaged - summary: Placement group damaged, manual intervention needed + summary: Placement group damaged, manual intervention needed on cluster mycluster description: During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent. Check to see which PG is affected, and attempt a manual repair if necessary. To list problematic placement groups, use 'rados list-inconsistent-pg <pool>'. To repair PGs use the 'ceph pg repair <pg_num>' command. - interval: 1m input_series: - - series: 'ceph_health_detail{name="TOO_MANY_PGS"}' + - series: 'ceph_health_detail{name="TOO_MANY_PGS",cluster="mycluster"}' values: '0+0x4 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="TOO_MANY_PGS"} == 1 eval_time: 5m exp_samples: - - labels: '{__name__="ceph_health_detail", name="TOO_MANY_PGS"}' + - labels: '{__name__="ceph_health_detail", name="TOO_MANY_PGS", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1685,20 +1679,21 @@ tests: - exp_labels: name: "TOO_MANY_PGS" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks/#too-many-pgs - summary: Placement groups per OSD is too high + summary: Placement groups per OSD is too high on cluster mycluster description: "The number of placement groups per OSD is too high (exceeds the mon_max_pg_per_osd setting).\n Check that the pg_autoscaler has not been disabled for any pools with 'ceph osd pool autoscale-status', and that the profile selected is appropriate. You may also adjust the target_size_ratio of a pool to guide the autoscaler based on the expected relative size of the pool ('ceph osd pool set cephfs.cephfs.meta target_size_ratio .1') or set the pg_autoscaler mode to 'warn' and adjust pg_num appropriately for one or more pools." - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_RECOVERY_FULL"}' + - series: 'ceph_health_detail{name="PG_RECOVERY_FULL", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="PG_RECOVERY_FULL"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="PG_RECOVERY_FULL"}' + - labels: '{__name__="ceph_health_detail", name="PG_RECOVERY_FULL", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1709,21 +1704,22 @@ tests: - exp_labels: name: "PG_RECOVERY_FULL" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.7.5 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-recovery-full - summary: OSDs are too full for recovery + summary: OSDs are too full for recovery on cluster mycluster description: Data redundancy is at risk since one or more OSDs are at or above the 'full' threshold. Add more capacity to the cluster, restore down/out OSDs, or delete unwanted data. - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_BACKFILL_FULL"}' + - series: 'ceph_health_detail{name="PG_BACKFILL_FULL", cluster="mycluster"}' values: '0+0x2 1+0x20' promql_expr_test: - expr: ceph_health_detail{name="PG_BACKFILL_FULL"} == 0 eval_time: 1m exp_samples: - - labels: '{__name__="ceph_health_detail", name="PG_BACKFILL_FULL"}' + - labels: '{__name__="ceph_health_detail", name="PG_BACKFILL_FULL", cluster="mycluster"}' value: 0 alert_rule_test: - eval_time: 1m @@ -1734,17 +1730,18 @@ tests: - exp_labels: name: "PG_BACKFILL_FULL" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.7.6 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-backfill-full - summary: Backfill operations are blocked due to lack of free space + summary: Backfill operations are blocked due to lack of free space on cluster mycluster description: Data redundancy may be at risk due to lack of free space within the cluster. One or more OSDs have reached the 'backfillfull' threshold. Add more capacity, or delete unwanted data. - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_AVAILABILITY"}' + - series: 'ceph_health_detail{name="PG_AVAILABILITY", cluster="mycluster"}' values: '0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_health_detail{name="OSD_DOWN"}' + - series: 'ceph_health_detail{name="OSD_DOWN", cluster="mycluster"}' values: '0 0 0 1 1 1 1 1 1 0 0 0 0 0 0 0' promql_expr_test: - expr: ((ceph_health_detail{name="PG_AVAILABILITY"} == 1) - scalar(ceph_health_detail{name="OSD_DOWN"})) @@ -1767,21 +1764,22 @@ tests: - exp_labels: name: "PG_AVAILABILITY" severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.7.3 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-availability - summary: PG is unavailable, blocking I/O + summary: PG is unavailable on cluster mycluster, blocking I/O description: Data availability is reduced, impacting the cluster's ability to service I/O. One or more placement groups (PGs) are in a state that blocks I/O. - interval: 1m input_series: - - series: 'ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"}' + - series: 'ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED", cluster="mycluster"}' values: '0+0x2 1+0x10' promql_expr_test: - expr: ceph_health_detail{name="PG_NOT_DEEP_SCRUBBED"} == 1 eval_time: 3m exp_samples: - - labels: '{__name__="ceph_health_detail", name="PG_NOT_DEEP_SCRUBBED"}' + - labels: '{__name__="ceph_health_detail", name="PG_NOT_DEEP_SCRUBBED", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -1792,10 +1790,11 @@ tests: - exp_labels: name: "PG_NOT_DEEP_SCRUBBED" severity: warning + cluster: mycluster type: ceph_default exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#pg-not-deep-scrubbed - summary: Placement group(s) have not been deep scrubbed + summary: Placement group(s) have not been deep scrubbed on cluster mycluster description: One or more PGs have not been deep scrubbed recently. Deep scrubs protect against bit-rot. They compare data replicas to ensure consistency. When PGs miss their deep scrub interval, it may indicate that the window is too small or PGs were not in a 'clean' state during the deep-scrub window. # Prometheus @@ -1821,25 +1820,26 @@ tests: exp_annotations: summary: The scrape job for Ceph is missing from Prometheus description: The prometheus job that scrapes from Ceph is no longer defined, this will effectively mean you'll have no metrics or alerts for the cluster. Please review the job definitions in the prometheus.yml file of the prometheus instance. + # RADOS - interval: 1m input_series: - - series: 'ceph_health_detail{name="OBJECT_UNFOUND"}' + - series: 'ceph_health_detail{name="OBJECT_UNFOUND", cluster="mycluster"}' values: '0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_up{ceph_daemon="osd.0"}' + - series: 'ceph_osd_up{ceph_daemon="osd.0", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_up{ceph_daemon="osd.1"}' + - series: 'ceph_osd_up{ceph_daemon="osd.1", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_up{ceph_daemon="osd.2"}' + - series: 'ceph_osd_up{ceph_daemon="osd.2", cluster="mycluster"}' values: '1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_metadata{ceph_daemon="osd.0"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.0", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_metadata{ceph_daemon="osd.1"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.1", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' - - series: 'ceph_osd_metadata{ceph_daemon="osd.2"}' + - series: 'ceph_osd_metadata{ceph_daemon="osd.2", cluster="mycluster"}' values: '1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1' promql_expr_test: - - expr: (ceph_health_detail{name="OBJECT_UNFOUND"} == 1) * on() (count(ceph_osd_up == 1) == bool count(ceph_osd_metadata)) == 1 + - expr: (ceph_health_detail{name="OBJECT_UNFOUND"} == 1) * on() group_right (cluster) (count(ceph_osd_up == 1) by (cluster) == bool count(ceph_osd_metadata) by (cluster)) == 1 eval_time: 1m exp_samples: alert_rule_test: @@ -1853,16 +1853,18 @@ tests: exp_alerts: - exp_labels: severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.10.1 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks#object-unfound - summary: Object(s) marked UNFOUND + summary: Object(s) marked UNFOUND on cluster mycluster description: The latest version of a RADOS object can not be found, even though all OSDs are up. I/O requests for this object from clients will block (hang). Resolving this issue may require the object to be rolled back to a prior version manually, and manually verified. + # Generic Alerts - interval: 1m input_series: - - series: 'ceph_health_detail{name="RECENT_CRASH"}' + - series: 'ceph_health_detail{name="RECENT_CRASH", cluster="mycluster"}' values: '0 0 0 1 1 1 1 1 1 1 1' promql_expr_test: - expr: ceph_health_detail{name="RECENT_CRASH"} == 1 @@ -1880,11 +1882,12 @@ tests: - exp_labels: name: RECENT_CRASH severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.1.2 exp_annotations: documentation: https://docs.ceph.com/en/latest/rados/operations/health-checks/#recent-crash - summary: One or more Ceph daemons have crashed, and are pending acknowledgement + summary: One or more Ceph daemons have crashed, and are pending acknowledgement on cluster mycluster description: One or more daemons have crashed recently, and need to be acknowledged. This notification ensures that software crashes do not go unseen. To acknowledge a crash, use the 'ceph crash archive <id>' command. # new rbdmirror alerts tests @@ -1892,21 +1895,21 @@ tests: # alert: CephRBDMirrorImagesPerDaemonHigh - interval: 1m input_series: - - series: 'ceph_rbd_mirror_snapshot_image_snapshots{ceph_daemon="client.admin.40628", image="image1", namespace="default", pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_snapshots{ceph_daemon="client.admin.40628", image="image1", namespace="default", pool="data", cluster="mycluster"}' values: '0+0x20 1+1x130' - - series: 'ceph_rbd_mirror_snapshot_image_snapshots{ceph_daemon="client.admin.40628", image="image2", namespace="default", pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_snapshots{ceph_daemon="client.admin.40628", image="image2", namespace="default", pool="data", cluster="mycluster"}' values: '1+1x130 131+0x20' # prometheus query test promql_expr_test: # negative test where there are no samples - - expr: sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100 + - expr: sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100 eval_time: 50m exp_samples: # second positive test - - expr: sum by (ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100 + - expr: sum by (cluster, ceph_daemon, namespace) (ceph_rbd_mirror_snapshot_image_snapshots) > 100 eval_time: 70m exp_samples: - - labels: '{ceph_daemon="client.admin.40628", namespace="default"}' + - labels: '{ceph_daemon="client.admin.40628", namespace="default", cluster="mycluster"}' value: 121 # prometheus alert test alert_rule_test: @@ -1921,31 +1924,32 @@ tests: - exp_labels: oid: "1.3.6.1.4.1.50495.1.2.1.10.2" severity: "critical" + cluster: mycluster type: "ceph_default" ceph_daemon: "client.admin.40628" namespace: "default" exp_annotations: - description: "Number of image replications per daemon is not suppossed to go beyond threshold 100" - summary: "Number of image replications are now above 100" + summary: "Number of image replications are now above 100 on cluster mycluster" + description: "Number of image replications per daemon is not supposed to go beyond threshold 100" # alert: CephRBDMirrorImagesNotInSync - interval: 1m input_series: - - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data",cluster="mycluster"}' values: '1.678+0x20 2.03+0x20 3.21+0x20' - - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data",cluster="mycluster"}' values: '1.678+0x20 2.03+0x20 2.03+0x20' # prometheus query test promql_expr_test: # negative test where there are no samples - - expr: sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0 + - expr: sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0 eval_time: 30m exp_samples: # second positive test - - expr: sum by (ceph_daemon, image, namespace, pool) (topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0 + - expr: sum by (cluster, ceph_daemon, image, namespace, pool) (topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (cluster, ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0 eval_time: 45m exp_samples: - - labels: '{ceph_daemon="client.admin.40628", image="image1", namespace="default", pool="data"}' + - labels: '{ceph_daemon="client.admin.40628", image="image1", namespace="default", pool="data", cluster="mycluster"}' value: 1.1800000000000002 # prometheus alert test alert_rule_test: @@ -1962,48 +1966,49 @@ tests: pool: "data" oid: "1.3.6.1.4.1.50495.1.2.1.10.3" severity: "critical" + cluster: mycluster type: "ceph_default" ceph_daemon: "client.admin.40628" namespace: "default" exp_annotations: + summary: "Some of the RBD mirror images are not in sync with the remote counter parts on cluster mycluster" description: "Both local and remote RBD mirror images should be in sync." - summary: "Some of the RBD mirror images are not in sync with the remote counter parts." # alert: CephRBDMirrorImagesNotInSyncVeryHigh - interval: 1m input_series: - - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data",cluster="mycluster"}' values: '1.678+0x20 2.03+0x20 3.21+0x20' - - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image1",namespace="default",pool="data",cluster="mycluster"}' values: '1.678+0x20 2.03+0x20 2.03+0x20' - - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image2",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image2",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x20 3.301+0x14 3.301+0x26' - - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image2",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image2",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x20 3.301+0x14 7.13+0x26' - - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image3",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image3",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x20 3.301+0x14 3.301+0x26' - - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image3",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image3",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x20 3.301+0x14 7.13+0x26' - - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image4",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_local_timestamp{ceph_daemon="client.admin.40628",image="image4",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x65' - - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image4",namespace="default",pool="data"}' + - series: 'ceph_rbd_mirror_snapshot_image_remote_timestamp{ceph_daemon="client.admin.40628",image="image4",namespace="default",pool="data",cluster="mycluster"}' values: '2.189+0x65' - - series: 'ceph_rbd_mirror_snapshot_snapshots{ceph_daemon="client.admin.40628"}' + - series: 'ceph_rbd_mirror_snapshot_snapshots{ceph_daemon="client.admin.40628",cluster="mycluster"}' values: '1+0x20 2+0x45' # prometheus query test promql_expr_test: # test each query individually # query 1 - - expr: count by (ceph_daemon) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) + - expr: count by (ceph_daemon, cluster) ((topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_local_timestamp) - topk by (ceph_daemon, image, namespace, pool) (1, ceph_rbd_mirror_snapshot_image_remote_timestamp)) != 0) eval_time: 45m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628",cluster="mycluster"}' value: 3 # query 2 - - expr: sum by (ceph_daemon) (ceph_rbd_mirror_snapshot_snapshots) * .1 + - expr: sum by (ceph_daemon, cluster) (ceph_rbd_mirror_snapshot_snapshots) * .1 eval_time: 45m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628",cluster="mycluster"}' value: 0.2 # prometheus alert test alert_rule_test: @@ -2019,15 +2024,16 @@ tests: ceph_daemon: "client.admin.40628" oid: "1.3.6.1.4.1.50495.1.2.1.10.4" severity: "critical" + cluster: mycluster type: "ceph_default" exp_annotations: - description: "More than 10% of the images have synchronization problems" - summary: "Number of unsynchronized images are very high." + summary: "Number of unsynchronized images are very high on cluster mycluster" + description: "More than 10% of the images have synchronization problems." # alert: "CephRBDMirrorImageTransferBandwidthHigh" - interval: 1m input_series: - - series: 'ceph_rbd_mirror_journal_replay_bytes{ceph_daemon="client.admin.40628"}' + - series: 'ceph_rbd_mirror_journal_replay_bytes{ceph_daemon="client.admin.40628", cluster="mycluster"}' values: '0+0x10 1+0x5 10+30x25 736+200x30' # prometheus query test promql_expr_test: @@ -2036,25 +2042,25 @@ tests: - expr: rate(ceph_rbd_mirror_journal_replay_bytes[5m]) eval_time: 5m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628", cluster="mycluster"}' value: 0.0 # rate 2 - expr: rate(ceph_rbd_mirror_journal_replay_bytes[5m]) eval_time: 20m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628", cluster="mycluster"}' value: 0.33 # rate 3 - expr: rate(ceph_rbd_mirror_journal_replay_bytes[5m]) eval_time: 40m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628", cluster="mycluster"}' value: 0.5 # rate 4 - expr: rate(ceph_rbd_mirror_journal_replay_bytes[5m]) eval_time: 50m exp_samples: - - labels: '{ceph_daemon="client.admin.40628"}' + - labels: '{ceph_daemon="client.admin.40628", cluster="mycluster"}' value: 3.3333333333333335 # prometheus alert test alert_rule_test: @@ -2070,20 +2076,21 @@ tests: ceph_daemon: "client.admin.40628" oid: "1.3.6.1.4.1.50495.1.2.1.10.5" severity: "warning" + cluster: mycluster type: "ceph_default" exp_annotations: + summary: "The replication network usage on cluster mycluster has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes" description: "Detected a heavy increase in bandwidth for rbd replications (over 80%) in the last 30 min. This might not be a problem, but it is good to review the number of images being replicated simultaneously" - summary: "The replication network usage has been increased over 80% in the last 30 minutes. Review the number of images being replicated. This alert will be cleaned automatically after 30 minutes" - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_STORAGE"}' + - series: 'ceph_health_detail{name="HARDWARE_STORAGE", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_STORAGE"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_STORAGE"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_STORAGE", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2094,20 +2101,21 @@ tests: - exp_labels: name: HARDWARE_STORAGE severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.1 exp_annotations: - summary: Storage devices error(s) detected + summary: Storage devices error(s) detected on cluster mycluster description: "Some storage devices are in error. Check `ceph health detail`." - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_MEMORY"}' + - series: 'ceph_health_detail{name="HARDWARE_MEMORY", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_MEMORY"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_MEMORY"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_MEMORY", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2118,20 +2126,21 @@ tests: - exp_labels: name: HARDWARE_MEMORY severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.2 exp_annotations: - summary: DIMM error(s) detected + summary: DIMM error(s) detected on cluster mycluster description: "DIMM error(s) detected. Check `ceph health detail`." - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_PROCESSOR"}' + - series: 'ceph_health_detail{name="HARDWARE_PROCESSOR", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_PROCESSOR"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_PROCESSOR"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_PROCESSOR", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2142,20 +2151,21 @@ tests: - exp_labels: name: HARDWARE_PROCESSOR severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.3 exp_annotations: - summary: Processor error(s) detected + summary: Processor error(s) detected on cluster mycluster description: "Processor error(s) detected. Check `ceph health detail`." - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_NETWORK"}' + - series: 'ceph_health_detail{name="HARDWARE_NETWORK", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_NETWORK"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_NETWORK"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_NETWORK", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2166,20 +2176,21 @@ tests: - exp_labels: name: HARDWARE_NETWORK severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.4 exp_annotations: - summary: Network error(s) detected + summary: Network error(s) detected on cluster mycluster description: "Network error(s) detected. Check `ceph health detail`." - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_POWER"}' + - series: 'ceph_health_detail{name="HARDWARE_POWER", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_POWER"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_POWER"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_POWER", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2190,20 +2201,21 @@ tests: - exp_labels: name: HARDWARE_POWER severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.5 exp_annotations: - summary: Power supply error(s) detected + summary: Power supply error(s) detected on cluster mycluster description: "Power supply error(s) detected. Check `ceph health detail`." - interval: 30s input_series: - - series: 'ceph_health_detail{name="HARDWARE_FANS"}' + - series: 'ceph_health_detail{name="HARDWARE_FANS", cluster="mycluster"}' values: '1+0x40' promql_expr_test: - expr: ceph_health_detail{name="HARDWARE_FANS"} > 0 eval_time: 2m exp_samples: - - labels: '{__name__="ceph_health_detail", name="HARDWARE_FANS"}' + - labels: '{__name__="ceph_health_detail", name="HARDWARE_FANS", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 1m @@ -2214,35 +2226,36 @@ tests: - exp_labels: name: HARDWARE_FANS severity: critical + cluster: mycluster type: ceph_default oid: 1.3.6.1.4.1.50495.1.2.1.13.6 exp_annotations: - summary: Fan error(s) detected + summary: Fan error(s) detected on cluster mycluster description: "Fan error(s) detected. Check `ceph health detail`." # nvmeof Tests # NVMeoFSubsystemNamespaceLimit - interval: 1m input_series: - - series: 'ceph_nvmeof_subsystem_namespace_limit{nqn="wah"}' + - series: 'ceph_nvmeof_subsystem_namespace_limit{nqn="wah", cluster="mycluster"}' values: '5x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk1"}' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk1", cluster="mycluster"}' + values: '1x10' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk2", cluster="mycluster"}' values: '1x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk2"}' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk3", cluster="mycluster"}' values: '1x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk3"}' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk4", cluster="mycluster"}' values: '1x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk4"}' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk5", cluster="mycluster"}' values: '1x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk5"}' + - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk6", cluster="mycluster"}' values: '1x10' - - series: 'ceph_nvmeof_subsystem_namespace_metadata{nqn="wah", bdev_name="disk6"}' - values: '1x10' promql_expr_test: - - expr: (count by(nqn) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit + - expr: (count by(nqn, cluster) (ceph_nvmeof_subsystem_namespace_metadata)) >= ceph_nvmeof_subsystem_namespace_limit eval_time: 1m exp_samples: - - labels: '{nqn="wah"}' + - labels: '{nqn="wah",cluster="mycluster"}' value: 6 alert_rule_test: - eval_time: 5m @@ -2251,29 +2264,30 @@ tests: - exp_labels: nqn: wah severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "wah subsystem has reached its maximum number of namespaces " + summary: "wah subsystem has reached its maximum number of namespaces on cluster mycluster" description: "Subsystems have a max namespace limit defined at creation time. This alert means that no more namespaces can be added to wah" # NVMeoFTooManyGateways - interval: 1m input_series: - - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.1"}' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.1",cluster="mycluster"}' values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.2"}' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.2",cluster="mycluster"}' values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.3"}' - values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.4"}' - values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.5"}' - values: '1+0x20' - promql_expr_test: - - expr: count(ceph_nvmeof_gateway_info) > 4.00 + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.3",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.4",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{addr="1.1.1.5",cluster="mycluster"}' + values: '1+0x20' + promql_expr_test: + - expr: count(ceph_nvmeof_gateway_info) by (cluster) > 4.00 eval_time: 1m exp_samples: - - labels: '{}' + - labels: '{cluster="mycluster"}' value: 5 alert_rule_test: - eval_time: 5m @@ -2281,30 +2295,35 @@ tests: exp_alerts: - exp_labels: severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "Max supported gateways exceeded " + summary: "Max supported gateways exceeded on cluster mycluster" description: "You may create many gateways, but 4 is the tested limit" # NVMeoFMaxGatewayGroupSize - interval: 1m input_series: - - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.1"}' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.1",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.2",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.3",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.9",cluster="mycluster"}' values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.2"}' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.12",cluster="mycluster"}' values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.3"}' - values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4"}' - values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5"}' - values: '1+0x20' - promql_expr_test: - - expr: count by(group) (ceph_nvmeof_gateway_info) > 2.00 + - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5",cluster="mycluster"}' + values: '1+0x20' + promql_expr_test: + - expr: count(ceph_nvmeof_gateway_info) by (cluster, group) > 4.00 eval_time: 1m exp_samples: - - labels: '{group="group-1"}' - value: 3 + - labels: '{cluster="mycluster",group="group-1"}' + value: 5 alert_rule_test: - eval_time: 5m alertname: NVMeoFMaxGatewayGroupSize @@ -2312,25 +2331,26 @@ tests: - exp_labels: group: group-1 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "Max gateways within a gateway group (group-1) exceeded " - description: "You may create many gateways in a gateway group, but 2 is the tested limit" + summary: "Max gateways within a gateway group (group-1) exceeded on cluster mycluster" + description: "You may create many gateways in a gateway group, but 4 is the tested limit" # NVMeoFSingleGatewayGroup - interval: 1m input_series: - - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.2"}' + - series: 'ceph_nvmeof_gateway_info{group="group-1",addr="1.1.1.2",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4",cluster="mycluster"}' + values: '1+0x20' + - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5",cluster="mycluster"}' values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.4"}' - values: '1+0x20' - - series: 'ceph_nvmeof_gateway_info{group="group-2",addr="1.1.1.5"}' - values: '1+0x20' promql_expr_test: - - expr: count by(group) (ceph_nvmeof_gateway_info) == 1 + - expr: count by(group, cluster) (ceph_nvmeof_gateway_info) == 1 eval_time: 1m exp_samples: - - labels: '{group="group-1"}' + - labels: '{group="group-1", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 5m @@ -2339,21 +2359,22 @@ tests: - exp_labels: group: group-1 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "The gateway group group-1 consists of a single gateway - HA is not possible " - description: "Although a single member gateway group is valid, it should only be used for test purposes" + summary: "The gateway group group-1 consists of a single gateway - HA is not possible on cluster mycluster" + description: "Although a single member gateway group is valid, it should only be used for test purposes" # NVMeoFHighGatewayCPU - interval: 1m input_series: - - series: 'ceph_nvmeof_reactor_seconds_total{mode="busy",name="nvmf_tgt_poll_group_0",instance="node-1:10008"}' + - series: 'ceph_nvmeof_reactor_seconds_total{mode="busy",name="nvmf_tgt_poll_group_0",instance="node-1:10008",cluster="mycluster"}' values: '880+5080x20' promql_expr_test: - - expr: label_replace(avg by(instance) (rate(ceph_nvmeof_reactor_seconds_total{mode="busy"}[1m])),"instance","$1","instance","(.*):.*") > 80 + - expr: label_replace(avg by(instance, cluster) (rate(ceph_nvmeof_reactor_seconds_total{mode="busy"}[1m])),"instance","$1","instance","(.*):.*") > 80 eval_time: 5m exp_samples: - - labels: '{instance="node-1"}' + - labels: '{instance="node-1", cluster="mycluster"}' value: 8.466666666666667E+01 alert_rule_test: - eval_time: 15m @@ -2362,23 +2383,24 @@ tests: - exp_labels: instance: node-1 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "CPU used by node-1 NVMe-oF Gateway is high " - description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores" - + summary: "CPU used by node-1 NVMe-oF Gateway is high on cluster mycluster" + description: "Typically, high CPU may indicate degraded performance. Consider increasing the number of reactor cores" + # NVMeoFGatewayOpenSecurity - interval: 1m input_series: - - series: 'ceph_nvmeof_subsystem_metadata{nqn="nqn.good", allow_any_host="no"}' + - series: 'ceph_nvmeof_subsystem_metadata{nqn="nqn.good", allow_any_host="no", cluster="mycluster"}' values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{nqn="nqn.bad", allow_any_host="yes"}' + - series: 'ceph_nvmeof_subsystem_metadata{nqn="nqn.bad", allow_any_host="yes", cluster="mycluster"}' values: '1+0x10' promql_expr_test: - expr: ceph_nvmeof_subsystem_metadata{allow_any_host="yes"} eval_time: 1m exp_samples: - - labels: '{__name__="ceph_nvmeof_subsystem_metadata",nqn="nqn.bad",allow_any_host="yes"}' + - labels: '{__name__="ceph_nvmeof_subsystem_metadata",nqn="nqn.bad",allow_any_host="yes", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 5m @@ -2388,53 +2410,54 @@ tests: allow_any_host: yes nqn: nqn.bad severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "Subsystem nqn.bad has been defined without host level security " - description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss" + summary: "Subsystem nqn.bad has been defined without host level security on cluster mycluster" + description: "It is good practice to ensure subsystems use host security to reduce the risk of unexpected data loss" # NVMeoFTooManySubsystems - interval: 1m input_series: - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn1"}' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn1",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn2",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn3",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn4",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn5",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn6",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn7",cluster="mycluster"}' values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn2"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn3"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn4"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn5"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn6"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn7"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn8"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn9"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn10"}' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn8",cluster="mycluster"}' values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn11"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn12"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn13"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn14"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn15"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn16"}' - values: '1+0x10' - - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn17"}' - values: '1+0x10' - promql_expr_test: - - expr: count by(gateway_host) (label_replace(ceph_nvmeof_subsystem_metadata,"gateway_host","$1","instance","(.*):.*")) > 16 + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn9",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn10",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn11",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn12",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn13",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn14",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn15",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn16",cluster="mycluster"}' + values: '1+0x10' + - series: 'ceph_nvmeof_subsystem_metadata{instance="node-1:10008",nqn="nqn17",cluster="mycluster"}' + values: '1+0x10' + promql_expr_test: + - expr: count by(gateway_host, cluster) (label_replace(ceph_nvmeof_subsystem_metadata,"gateway_host","$1","instance","(.*):.*")) > 16 eval_time: 1m exp_samples: - - labels: '{gateway_host="node-1"}' + - labels: '{gateway_host="node-1", cluster="mycluster"}' value: 17 alert_rule_test: - eval_time: 5m @@ -2443,23 +2466,24 @@ tests: - exp_labels: gateway_host: node-1 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "The number of subsystems defined to the gateway exceeds supported values " - description: "Although you may continue to create subsystems in node-1, the configuration may not be supported" + summary: "The number of subsystems defined to the gateway exceeds supported values on cluster mycluster" + description: "Although you may continue to create subsystems in node-1, the configuration may not be supported" # NVMeoFVersionMismatch - interval: 1m input_series: - - series: 'ceph_nvmeof_gateway_info{version="0.0.7"}' + - series: 'ceph_nvmeof_gateway_info{version="0.0.7",cluster="mycluster"}' values: '1+0x80' - - series: 'ceph_nvmeof_gateway_info{version="1.0.0"}' + - series: 'ceph_nvmeof_gateway_info{version="1.0.0",cluster="mycluster"}' values: '1+0x80' promql_expr_test: - - expr: count(count by(version) (ceph_nvmeof_gateway_info)) > 1 + - expr: count(count(ceph_nvmeof_gateway_info) by (cluster, version)) by (cluster) > 1 eval_time: 1m exp_samples: - - labels: '{}' + - labels: '{cluster="mycluster"}' value: 2 alert_rule_test: - eval_time: 1h @@ -2467,23 +2491,24 @@ tests: exp_alerts: - exp_labels: severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "The cluster has different NVMe-oF gateway releases active " - description: "This may indicate an issue with deployment. Check cephadm logs" + summary: "Too many different NVMe-oF gateway releases active on cluster mycluster" + description: "This may indicate an issue with deployment. Check cephadm logs" # NVMeoFHighClientCount - interval: 1m input_series: - - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn1"}' + - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn1",cluster="mycluster"}' values: '2 2 2 4 4 8 8 8 10 10 20 20 32 34 34 38 38 40 44 44' - - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn2"}' + - series: 'ceph_nvmeof_subsystem_host_count{nqn="nqn2",cluster="mycluster"}' values: '2 2 2 8 8 8 16 16 16 16 16 16 16 16 16 16 16 16 16 16' promql_expr_test: - expr: ceph_nvmeof_subsystem_host_count > 32.00 eval_time: 15m exp_samples: - - labels: '{__name__="ceph_nvmeof_subsystem_host_count",nqn="nqn1"}' + - labels: '{__name__="ceph_nvmeof_subsystem_host_count",nqn="nqn1",cluster="mycluster"}' value: 38 alert_rule_test: - eval_time: 20m @@ -2492,55 +2517,57 @@ tests: - exp_labels: nqn: nqn1 severity: warning + cluster: mycluster type: ceph_default exp_annotations: - summary: "The number of clients connected to nqn1 is too high " - description: "The supported limit for clients connecting to a subsystem is 32" - + summary: "The number of clients connected to nqn1 is too high on cluster mycluster" + description: "The supported limit for clients connecting to a subsystem is 32" + # NVMeoFHighHostCPU - interval: 1m input_series: - - series: 'node_cpu_seconds_total{mode="idle",instance="node-1:9100",cpu="0"}' + - series: 'node_cpu_seconds_total{mode="idle",instance="node-1:9100",cpu="0",cluster="mycluster"}' values: '0+18x10 180+9x20' - - series: 'node_cpu_seconds_total{mode="idle",instance="node-1:9100",cpu="1"}' + - series: 'node_cpu_seconds_total{mode="idle",instance="node-1:9100",cpu="1",cluster="mycluster"}' values: '0+18x10 180+9x20' - - series: 'ceph_nvmeof_gateway_info{instance="node-1:10008"}' + - series: 'ceph_nvmeof_gateway_info{instance="node-1:10008",cluster="mycluster"}' values: '1.00+0x20' promql_expr_test: - expr: 100-((100*(avg by(host) (label_replace(rate(node_cpu_seconds_total{mode="idle"}[5m]),"host","$1","instance","(.*):.*")) * on(host) group_right label_replace(ceph_nvmeof_gateway_info,"host","$1","instance","(.*):.*")))) >= 80 eval_time: 16m exp_samples: - - labels: '{host="node-1",instance="node-1:10008"}' + - labels: '{host="node-1",instance="node-1:10008",cluster="mycluster"}' value: 85 alert_rule_test: # negative match at 15m - eval_time: 15m alertname: NVMeoFHighHostCPU - # positive match at 25m + # positive match at 25m - eval_time: 25m alertname: NVMeoFHighHostCPU exp_alerts: - exp_labels: instance: node-1:10008 host: node-1 + cluster: mycluster severity: warning type: ceph_default exp_annotations: - summary: "The CPU is high (85%) on NVMeoF Gateway host (node-1) " - description: "High CPU on a gateway host can lead to CPU contention and performance degradation" + summary: "The CPU is high (85%) on NVMeoF Gateway host (node-1) on cluster mycluster" + description: "High CPU on a gateway host can lead to CPU contention and performance degradation" # NVMeoFInterfaceDown - triggered on eth0 only - interval: 30s input_series: - - series: 'ceph_nvmeof_subsystem_listener_iface_info{operstate="down", device="eth0"}' + - series: 'ceph_nvmeof_subsystem_listener_iface_info{operstate="down", device="eth0", cluster="mycluster"}' + values: '1+0x30' + - series: 'ceph_nvmeof_subsystem_listener_iface_info{operstate="up", device="eth1", cluster="mycluster"}' values: '1+0x30' - - series: 'ceph_nvmeof_subsystem_listener_iface_info{operstate="up", device="eth1"}' - values: '1+0x30' promql_expr_test: - expr: ceph_nvmeof_subsystem_listener_iface_info{operstate="down"} eval_time: 1m exp_samples: - - labels: '{__name__="ceph_nvmeof_subsystem_listener_iface_info", device="eth0", operstate="down"}' + - labels: '{__name__="ceph_nvmeof_subsystem_listener_iface_info", device="eth0", operstate="down", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 5m @@ -2550,24 +2577,25 @@ tests: oid: 1.3.6.1.4.1.50495.1.2.1.14.1 operstate: down device: eth0 + cluster: mycluster severity: warning type: ceph_default exp_annotations: - summary: "Network interface eth0 is down " - description: "A NIC used by one or more subsystems is in a down state" + summary: "Network interface eth0 is down on cluster mycluster" + description: "A NIC used by one or more subsystems is in a down state" # NVMeoFInterfaceDuplex - triggered on eth1 only - interval: 30s input_series: - - series: 'ceph_nvmeof_subsystem_listener_iface_info{duplex="full", device="eth0"}' - values: '1+0x30' - - series: 'ceph_nvmeof_subsystem_listener_iface_info{duplex="half", device="eth1"}' + - series: 'ceph_nvmeof_subsystem_listener_iface_info{duplex="full", device="eth0", cluster="mycluster"}' + values: '1+0x30' + - series: 'ceph_nvmeof_subsystem_listener_iface_info{duplex="half", device="eth1", cluster="mycluster"}' values: '1+0x30' promql_expr_test: - expr: ceph_nvmeof_subsystem_listener_iface_info{duplex!="full"} eval_time: 30s exp_samples: - - labels: '{__name__="ceph_nvmeof_subsystem_listener_iface_info", device="eth1", duplex="half"}' + - labels: '{__name__="ceph_nvmeof_subsystem_listener_iface_info", device="eth1", duplex="half", cluster="mycluster"}' value: 1 alert_rule_test: - eval_time: 5m @@ -2576,18 +2604,19 @@ tests: - exp_labels: duplex: half device: eth1 + cluster: mycluster severity: warning type: ceph_default exp_annotations: - summary: "Network interface eth1 is not running in full duplex mode " - description: "Until this is resolved, performance from the gateway will be degraded" + summary: "Network interface eth1 is not running in full duplex mode on cluster mycluster" + description: "Until this is resolved, performance from the gateway will be degraded" - # NVMeoFHighReadLatency + # NVMeoFHighReadLatency - interval: 30s input_series: - - series: 'ceph_nvmeof_bdev_read_seconds_total{instance="node-1:10008",bdev_name="disk1"}' - values: '0+1680x10 19800+3000x20' - - series: 'ceph_nvmeof_bdev_reads_completed_total{instance="node-1:10008",bdev_name="disk1"}' + - series: 'ceph_nvmeof_bdev_read_seconds_total{instance="node-1:10008",bdev_name="disk1",cluster="mycluster"}' + values: '0+1680x10 19800+3000x20' + - series: 'ceph_nvmeof_bdev_reads_completed_total{instance="node-1:10008",bdev_name="disk1",cluster="mycluster"}' values: '0+286000x10 2980000+120000x20' promql_expr_test: - expr: label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_read_seconds_total[1m]) / rate(ceph_nvmeof_bdev_reads_completed_total[1m])))),"gateway","$1","instance","(.*):.*") > 0.02 @@ -2610,14 +2639,14 @@ tests: type: ceph_default exp_annotations: summary: "The average read latency over the last 5 mins has reached 10 ms or more on node-1" - description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate" + description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate" - # NVMeoFHighWriteLatency + # NVMeoFHighWriteLatency - interval: 30s input_series: - - series: 'ceph_nvmeof_bdev_write_seconds_total{instance="node-1:10008",bdev_name="disk1"}' - values: '0+1680x10 19800+3000x20' - - series: 'ceph_nvmeof_bdev_writes_completed_total{instance="node-1:10008",bdev_name="disk1"}' + - series: 'ceph_nvmeof_bdev_write_seconds_total{instance="node-1:10008",bdev_name="disk1",cluster="mycluster"}' + values: '0+1680x10 19800+3000x20' + - series: 'ceph_nvmeof_bdev_writes_completed_total{instance="node-1:10008",bdev_name="disk1",cluster="mycluster"}' values: '0+286000x10 2980000+120000x20' promql_expr_test: - expr: label_replace((avg by(instance) ((rate(ceph_nvmeof_bdev_write_seconds_total[1m]) / rate(ceph_nvmeof_bdev_writes_completed_total[1m])))),"gateway","$1","instance","(.*):.*") > 0.02 @@ -2641,4 +2670,3 @@ tests: exp_annotations: summary: "The average write latency over the last 5 mins has reached 20 ms or more on node-1" description: "High latencies may indicate a constraint within the cluster e.g. CPU, network. Please investigate" -
\ No newline at end of file diff --git a/monitoring/ceph-mixin/tox.ini b/monitoring/ceph-mixin/tox.ini index 90ac311edba..8d1ec872e7f 100644 --- a/monitoring/ceph-mixin/tox.ini +++ b/monitoring/ceph-mixin/tox.ini @@ -26,6 +26,7 @@ allowlist_externals = jsonnet jsonnetfmt sh + ./lint-jsonnet.sh description = check: Ensure that auto-generated files matches the current version fix: Update generated files from jsonnet file with latest changes diff --git a/qa/suites/nvmeof/basic/workloads/nvmeof_mtls.yaml b/qa/suites/nvmeof/basic/workloads/nvmeof_mtls.yaml new file mode 100644 index 00000000000..8eb4f6dc63c --- /dev/null +++ b/qa/suites/nvmeof/basic/workloads/nvmeof_mtls.yaml @@ -0,0 +1,36 @@ +tasks: +- nvmeof: + installer: host.a + gw_image: quay.io/ceph/nvmeof:latest # "default" is the image cephadm defaults to; change to test specific nvmeof images, example "latest" + rbd: + pool_name: mypool + image_name_prefix: myimage + gateway_config: + subsystems_count: 3 + namespaces_count: 20 + cli_image: quay.io/ceph/nvmeof-cli:latest + create_mtls_secrets: true + +- cephadm.wait_for_service: + service: nvmeof.mypool.mygroup0 + +- workunit: + no_coverage_and_limits: true + timeout: 30m + clients: + client.0: + - nvmeof/setup_subsystem.sh + - nvmeof/basic_tests.sh + - nvmeof/fio_test.sh --rbd_iostat + env: + RBD_POOL: mypool + RBD_IMAGE_PREFIX: myimage + IOSTAT_INTERVAL: '10' + RUNTIME: '60' + +- workunit: + no_coverage_and_limits: true + timeout: 30m + clients: + client.0: + - nvmeof/mtls_test.sh diff --git a/qa/suites/orch/cephadm/upgrade/3-upgrade/staggered.yaml b/qa/suites/orch/cephadm/upgrade/3-upgrade/staggered.yaml index 0080d3bf730..c6bec082843 100644 --- a/qa/suites/orch/cephadm/upgrade/3-upgrade/staggered.yaml +++ b/qa/suites/orch/cephadm/upgrade/3-upgrade/staggered.yaml @@ -131,8 +131,10 @@ tasks: - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps + - ceph versions # verify all rgw daemons on same version and version hash matches what we are upgrading to - - ceph versions | jq -e '.rgw | length == 1' + # `ceph versions` might not get updated immediately for rgw so retry this + - time timeout 60 bash -c "until ceph versions | jq -e '.rgw | length == 1'; do sleep 2; done" - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail diff --git a/qa/tasks/nvmeof.py b/qa/tasks/nvmeof.py index b89f123c97e..42e357294d9 100644 --- a/qa/tasks/nvmeof.py +++ b/qa/tasks/nvmeof.py @@ -32,6 +32,7 @@ class Nvmeof(Task): gateway_config: namespaces_count: 10 cli_version: latest + create_mtls_secrets: False """ @@ -69,6 +70,7 @@ class Nvmeof(Task): self.serial = gateway_config.get('serial', 'SPDK00000000000001') self.port = gateway_config.get('port', '4420') self.srport = gateway_config.get('srport', '5500') + self.create_mtls_secrets = gateway_config.get('create_mtls_secrets', False) def deploy_nvmeof(self): """ @@ -147,7 +149,38 @@ class Nvmeof(Task): started=True, ) log.info("[nvmeof]: executed deploy_nvmeof successfully!") - + + def write_mtls_config(self, gateway_ips): + log.info("[nvmeof]: writing mtls config...") + allowed_ips = "" + for ip in gateway_ips: + allowed_ips += ("IP:" + ip + ",") + self.remote.run( + args=[ + "sudo", "openssl", "req", "-x509", "-newkey", "rsa:4096", "-nodes", "-keyout", "/etc/ceph/server.key", + "-out", "/etc/ceph/server.crt", "-days", "3650", "-subj", "/CN=my.server", "-addext", f"subjectAltName={allowed_ips[:-1]}" + ] + ) + self.remote.run( + args=[ + "sudo", "openssl", "req", "-x509", "-newkey", "rsa:4096", "-nodes", "-keyout", "/etc/ceph/client.key", + "-out", "/etc/ceph/client.crt", "-days", "3650", "-subj", "/CN=client1" + ] + ) + secrets_files = {"/etc/ceph/server.key": None, + "/etc/ceph/server.crt": None, + "/etc/ceph/client.key": None, + "/etc/ceph/client.crt": None, + } + for file in secrets_files.keys(): + secrets_files[file] = self.remote.read_file(path=file, sudo=True) + + for remote in self.ctx.cluster.remotes.keys(): + for remote_file in secrets_files.keys(): + data = secrets_files[remote_file] + remote.sudo_write_file(path=remote_file, data=data, mode='0644') + log.info("[nvmeof]: written mtls config!") + def set_gateway_cfg(self): log.info('[nvmeof]: running set_gateway_cfg...') ip_address = self.remote.ip_address @@ -174,6 +207,8 @@ class Nvmeof(Task): data=conf_data, sudo=True ) + if self.create_mtls_secrets: + self.write_mtls_config(gateway_ips) log.info("[nvmeof]: executed set_gateway_cfg successfully!") diff --git a/qa/workunits/nvmeof/mtls_test.sh b/qa/workunits/nvmeof/mtls_test.sh new file mode 100755 index 00000000000..e13ca530e8d --- /dev/null +++ b/qa/workunits/nvmeof/mtls_test.sh @@ -0,0 +1,76 @@ +#!/bin/bash + +set -ex +source /etc/ceph/nvmeof.env + +# install yq +wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /tmp/yq && chmod +x /tmp/yq + +subjectAltName=$(echo "$NVMEOF_GATEWAY_IP_ADDRESSES" | sed 's/,/,IP:/g') + +# create mtls spec files +ceph orch ls nvmeof --export > /tmp/gw-conf-original.yaml +sudo /tmp/yq ".spec.enable_auth=true | \ + .spec.root_ca_cert=\"mountcert\" | \ + .spec.client_cert = load_str(\"/etc/ceph/client.crt\") | \ + .spec.client_key = load_str(\"/etc/ceph/client.key\") | \ + .spec.server_cert = load_str(\"/etc/ceph/server.crt\") | \ + .spec.server_key = load_str(\"/etc/ceph/server.key\")" /tmp/gw-conf-original.yaml > /tmp/gw-conf-with-mtls.yaml +cp /tmp/gw-conf-original.yaml /tmp/gw-conf-without-mtls.yaml +sudo /tmp/yq '.spec.enable_auth=false' -i /tmp/gw-conf-without-mtls.yaml + +wait_for_service() { + MAX_RETRIES=30 + for ((RETRY_COUNT=1; RETRY_COUNT<=MAX_RETRIES; RETRY_COUNT++)); do + + if ceph orch ls --refresh | grep -q "nvmeof"; then + echo "Found nvmeof in the output!" + break + fi + if [ $RETRY_COUNT -eq $MAX_RETRIES ]; then + echo "Reached maximum retries ($MAX_RETRIES). Exiting." + break + fi + sleep 5 + done + ceph orch ps + ceph orch ls --refresh +} + +# deploy mtls +cat /tmp/gw-conf-with-mtls.yaml +ceph orch apply -i /tmp/gw-conf-with-mtls.yaml +ceph orch redeploy nvmeof.mypool.mygroup0 +sleep 100 +wait_for_service + + +# test +IFS=',' read -ra gateway_ips <<< "$NVMEOF_GATEWAY_IP_ADDRESSES" +for i in "${!gateway_ips[@]}" +do + ip="${gateway_ips[i]}" + sudo podman run -v /etc/ceph/server.crt:/server.crt:z -v /etc/ceph/client.crt:/client.crt:z \ + -v /etc/ceph/client.key:/client.key:z \ + -it $NVMEOF_CLI_IMAGE --server-address $ip --server-port $NVMEOF_SRPORT \ + --client-key /client.key --client-cert /client.crt --server-cert /server.crt --format json subsystem list +done + + +# remove mtls +cat /tmp/gw-conf-without-mtls.yaml +ceph orch apply -i /tmp/gw-conf-without-mtls.yaml +ceph orch redeploy nvmeof.mypool.mygroup0 +sleep 100 +wait_for_service + + +# test +IFS=',' read -ra gateway_ips <<< "$NVMEOF_GATEWAY_IP_ADDRESSES" +for i in "${!gateway_ips[@]}" +do + ip="${gateway_ips[i]}" + sudo podman run -it $NVMEOF_CLI_IMAGE --server-address $ip --server-port $NVMEOF_SRPORT \ + --format json subsystem list +done + diff --git a/src/common/Finisher.cc b/src/common/Finisher.cc index ff931faffc1..43550f35197 100644 --- a/src/common/Finisher.cc +++ b/src/common/Finisher.cc @@ -2,11 +2,40 @@ // vim: ts=8 sw=2 smarttab #include "Finisher.h" +#include "common/perf_counters.h" + +#include <fmt/core.h> #define dout_subsys ceph_subsys_finisher #undef dout_prefix #define dout_prefix *_dout << "finisher(" << this << ") " +Finisher::Finisher(CephContext *cct_) : + cct(cct_), finisher_lock(ceph::make_mutex("Finisher::finisher_lock")), + thread_name("fn_anonymous"), + finisher_thread(this) {} + +Finisher::Finisher(CephContext *cct_, std::string_view name, std::string &&tn) : + cct(cct_), finisher_lock(ceph::make_mutex(fmt::format("Finisher::{}", name))), + thread_name(std::move(tn)), + finisher_thread(this) { + PerfCountersBuilder b(cct, fmt::format("finisher-{}", name), + l_finisher_first, l_finisher_last); + b.add_u64(l_finisher_queue_len, "queue_len"); + b.add_time_avg(l_finisher_complete_lat, "complete_latency"); + logger = b.create_perf_counters(); + cct->get_perfcounters_collection()->add(logger); + logger->set(l_finisher_queue_len, 0); + logger->set(l_finisher_complete_lat, 0); +} + +Finisher::~Finisher() { + if (logger && cct) { + cct->get_perfcounters_collection()->remove(logger); + delete logger; + } +} + void Finisher::start() { ldout(cct, 10) << __func__ << dendl; @@ -20,7 +49,7 @@ void Finisher::stop() finisher_stop = true; // we don't have any new work to do, but we want the worker to wake up anyway // to process the stop condition. - finisher_cond.notify_all(); + finisher_cond.notify_one(); finisher_lock.unlock(); finisher_thread.join(); // wait until the worker exits completely ldout(cct, 10) << __func__ << " finish" << dendl; @@ -40,7 +69,7 @@ void Finisher::wait_for_empty() bool Finisher::is_empty() { - std::unique_lock ul(finisher_lock); + const std::lock_guard l{finisher_lock}; return finisher_queue.empty(); } diff --git a/src/common/Finisher.h b/src/common/Finisher.h index 9091d0b892a..acee6594ca4 100644 --- a/src/common/Finisher.h +++ b/src/common/Finisher.h @@ -19,10 +19,8 @@ #include "include/common_fwd.h" #include "common/Thread.h" #include "common/ceph_mutex.h" -#include "common/perf_counters.h" #include "common/Cond.h" - /// Finisher queue length performance counter ID. enum { l_finisher_first = 997082, @@ -37,23 +35,23 @@ enum { * contexts to complete is thread-safe. */ class Finisher { - CephContext *cct; + CephContext *const cct; ceph::mutex finisher_lock; ///< Protects access to queues and finisher_running. ceph::condition_variable finisher_cond; ///< Signaled when there is something to process. ceph::condition_variable finisher_empty_cond; ///< Signaled when the finisher has nothing more to process. - bool finisher_stop; ///< Set when the finisher should stop. - bool finisher_running; ///< True when the finisher is currently executing contexts. - bool finisher_empty_wait; ///< True mean someone wait finisher empty. + bool finisher_stop = false; ///< Set when the finisher should stop. + bool finisher_running = false; ///< True when the finisher is currently executing contexts. + bool finisher_empty_wait = false; ///< True mean someone wait finisher empty. /// Queue for contexts for which complete(0) will be called. std::vector<std::pair<Context*,int>> finisher_queue; std::vector<std::pair<Context*,int>> in_progress_queue; - std::string thread_name; + const std::string thread_name; /// Performance counter for the finisher's queue length. /// Only active for named finishers. - PerfCounters *logger; + PerfCounters *logger = nullptr; void *finisher_thread_entry(); @@ -66,56 +64,34 @@ class Finisher { public: /// Add a context to complete, optionally specifying a parameter for the complete function. void queue(Context *c, int r = 0) { - std::unique_lock ul(finisher_lock); - bool was_empty = finisher_queue.empty(); - finisher_queue.push_back(std::make_pair(c, r)); - if (was_empty) { - finisher_cond.notify_one(); + { + const std::lock_guard l{finisher_lock}; + const bool should_notify = finisher_queue.empty() && !finisher_running; + finisher_queue.push_back(std::make_pair(c, r)); + if (should_notify) { + finisher_cond.notify_one(); + } } + if (logger) logger->inc(l_finisher_queue_len); } - void queue(std::list<Context*>& ls) { + // TODO use C++20 concept checks instead of SFINAE + template<typename T> + auto queue(T &ls) -> decltype(std::distance(ls.begin(), ls.end()), void()) { { - std::unique_lock ul(finisher_lock); - if (finisher_queue.empty()) { - finisher_cond.notify_all(); - } - for (auto i : ls) { - finisher_queue.push_back(std::make_pair(i, 0)); - } - if (logger) - logger->inc(l_finisher_queue_len, ls.size()); - } - ls.clear(); - } - void queue(std::deque<Context*>& ls) { - { - std::unique_lock ul(finisher_lock); - if (finisher_queue.empty()) { - finisher_cond.notify_all(); - } - for (auto i : ls) { + const std::lock_guard l{finisher_lock}; + const bool should_notify = finisher_queue.empty() && !finisher_running; + for (Context *i : ls) { finisher_queue.push_back(std::make_pair(i, 0)); } - if (logger) - logger->inc(l_finisher_queue_len, ls.size()); - } - ls.clear(); - } - void queue(std::vector<Context*>& ls) { - { - std::unique_lock ul(finisher_lock); - if (finisher_queue.empty()) { - finisher_cond.notify_all(); + if (should_notify) { + finisher_cond.notify_one(); } - for (auto i : ls) { - finisher_queue.push_back(std::make_pair(i, 0)); - } - if (logger) - logger->inc(l_finisher_queue_len, ls.size()); } + if (logger) + logger->inc(l_finisher_queue_len, ls.size()); ls.clear(); } @@ -137,36 +113,17 @@ class Finisher { bool is_empty(); + std::string_view get_thread_name() const noexcept { + return thread_name; + } + /// Construct an anonymous Finisher. /// Anonymous finishers do not log their queue length. - explicit Finisher(CephContext *cct_) : - cct(cct_), finisher_lock(ceph::make_mutex("Finisher::finisher_lock")), - finisher_stop(false), finisher_running(false), finisher_empty_wait(false), - thread_name("fn_anonymous"), logger(0), - finisher_thread(this) {} + explicit Finisher(CephContext *cct_); /// Construct a named Finisher that logs its queue length. - Finisher(CephContext *cct_, std::string name, std::string tn) : - cct(cct_), finisher_lock(ceph::make_mutex("Finisher::" + name)), - finisher_stop(false), finisher_running(false), finisher_empty_wait(false), - thread_name(tn), logger(0), - finisher_thread(this) { - PerfCountersBuilder b(cct, std::string("finisher-") + name, - l_finisher_first, l_finisher_last); - b.add_u64(l_finisher_queue_len, "queue_len"); - b.add_time_avg(l_finisher_complete_lat, "complete_latency"); - logger = b.create_perf_counters(); - cct->get_perfcounters_collection()->add(logger); - logger->set(l_finisher_queue_len, 0); - logger->set(l_finisher_complete_lat, 0); - } - - ~Finisher() { - if (logger && cct) { - cct->get_perfcounters_collection()->remove(logger); - delete logger; - } - } + Finisher(CephContext *cct_, std::string_view name, std::string &&tn); + ~Finisher(); }; /// Context that is completed asynchronously on the supplied finisher. diff --git a/src/common/HeartbeatMap.cc b/src/common/HeartbeatMap.cc index 54442709229..246cec9460b 100644 --- a/src/common/HeartbeatMap.cc +++ b/src/common/HeartbeatMap.cc @@ -43,11 +43,11 @@ HeartbeatMap::~HeartbeatMap() ceph_assert(m_workers.empty()); } -heartbeat_handle_d *HeartbeatMap::add_worker(const string& name, pthread_t thread_id) +heartbeat_handle_d *HeartbeatMap::add_worker(string&& name, pthread_t thread_id) { std::unique_lock locker{m_rwlock}; ldout(m_cct, 10) << "add_worker '" << name << "'" << dendl; - heartbeat_handle_d *h = new heartbeat_handle_d(name); + heartbeat_handle_d *h = new heartbeat_handle_d(std::move(name)); ANNOTATE_BENIGN_RACE_SIZED(&h->timeout, sizeof(h->timeout), "heartbeat_handle_d timeout"); ANNOTATE_BENIGN_RACE_SIZED(&h->suicide_timeout, sizeof(h->suicide_timeout), diff --git a/src/common/HeartbeatMap.h b/src/common/HeartbeatMap.h index 6f486b21ca8..401042cc271 100644 --- a/src/common/HeartbeatMap.h +++ b/src/common/HeartbeatMap.h @@ -48,15 +48,15 @@ struct heartbeat_handle_d { ceph::timespan suicide_grace = ceph::timespan::zero(); std::list<heartbeat_handle_d*>::iterator list_item; - explicit heartbeat_handle_d(const std::string& n) - : name(n) + explicit heartbeat_handle_d(std::string&& n) + : name(std::move(n)) { } }; class HeartbeatMap { public: // register/unregister - heartbeat_handle_d *add_worker(const std::string& name, pthread_t thread_id); + heartbeat_handle_d *add_worker(std::string&& name, pthread_t thread_id); void remove_worker(const heartbeat_handle_d *h); // reset the timeout so that it expects another touch within grace amount of time diff --git a/src/common/LRUSet.h b/src/common/LRUSet.h index b62956ba460..c8c66e85458 100644 --- a/src/common/LRUSet.h +++ b/src/common/LRUSet.h @@ -43,6 +43,7 @@ class LRUSet { // lru boost::intrusive::list< Node, + boost::intrusive::constant_time_size<false>, boost::intrusive::member_hook<Node, boost::intrusive::list_member_hook<>, &Node::lru_item> diff --git a/src/common/SloppyCRCMap.cc b/src/common/SloppyCRCMap.cc index ec9cbdf53a6..f82a70701d2 100644 --- a/src/common/SloppyCRCMap.cc +++ b/src/common/SloppyCRCMap.cc @@ -73,7 +73,7 @@ void SloppyCRCMap::truncate(uint64_t offset) offset -= offset % block_size; std::map<uint64_t,uint32_t>::iterator p = crc_map.lower_bound(offset); while (p != crc_map.end()) - crc_map.erase(p++); + p = crc_map.erase(p); } void SloppyCRCMap::zero(uint64_t offset, uint64_t len) diff --git a/src/common/TrackedOp.h b/src/common/TrackedOp.h index f4cf8902dac..57d73038364 100644 --- a/src/common/TrackedOp.h +++ b/src/common/TrackedOp.h @@ -243,6 +243,7 @@ private: public: typedef boost::intrusive::list< TrackedOp, + boost::intrusive::constant_time_size<false>, boost::intrusive::member_hook< TrackedOp, boost::intrusive::list_member_hook<>, diff --git a/src/common/ceph_context.h b/src/common/ceph_context.h index f1877647877..6a02d5c5bf1 100644 --- a/src/common/ceph_context.h +++ b/src/common/ceph_context.h @@ -282,10 +282,20 @@ public: void set_mon_addrs(const MonMap& mm); void set_mon_addrs(const std::vector<entity_addrvec_t>& in) { auto ptr = std::make_shared<std::vector<entity_addrvec_t>>(in); +#if defined(__GNUC__) && __GNUC__ < 12 + // workaround for GCC 11 bug atomic_store_explicit(&_mon_addrs, std::move(ptr), std::memory_order_relaxed); +#else + _mon_addrs.store(std::move(ptr), std::memory_order_relaxed); +#endif } std::shared_ptr<std::vector<entity_addrvec_t>> get_mon_addrs() const { +#if defined(__GNUC__) && __GNUC__ < 12 + // workaround for GCC 11 bug auto ptr = atomic_load_explicit(&_mon_addrs, std::memory_order_relaxed); +#else + auto ptr = _mon_addrs.load(std::memory_order_relaxed); +#endif return ptr; } @@ -306,7 +316,12 @@ private: int _crypto_inited; +#if defined(__GNUC__) && __GNUC__ < 12 + // workaround for GCC 11 bug std::shared_ptr<std::vector<entity_addrvec_t>> _mon_addrs; +#else + std::atomic<std::shared_ptr<std::vector<entity_addrvec_t>>> _mon_addrs; +#endif /* libcommon service thread. * SIGHUP wakes this thread, which then reopens logfiles */ diff --git a/src/common/config.cc b/src/common/config.cc index e151e94bb90..3a5ee91c347 100644 --- a/src/common/config.cc +++ b/src/common/config.cc @@ -24,6 +24,8 @@ #include "common/hostname.h" #include "common/dout.h" +#include <fmt/core.h> + /* Don't use standard Ceph logging in this file. * We can't use logging until it's initialized, and a lot of the necessary * initialization happens here. @@ -131,14 +133,11 @@ md_config_t::md_config_t(ConfigValues& values, // Define the debug_* options as well. subsys_options.reserve(values.subsys.get_num()); for (unsigned i = 0; i < values.subsys.get_num(); ++i) { - string name = string("debug_") + values.subsys.get_name(i); - subsys_options.push_back( - Option(name, Option::TYPE_STR, Option::LEVEL_ADVANCED)); + subsys_options.emplace_back( + fmt::format("debug_{}", values.subsys.get_name(i)), Option::TYPE_STR, Option::LEVEL_ADVANCED); Option& opt = subsys_options.back(); - opt.set_default(stringify(values.subsys.get_log_level(i)) + "/" + - stringify(values.subsys.get_gather_level(i))); - string desc = string("Debug level for ") + values.subsys.get_name(i); - opt.set_description(desc.c_str()); + opt.set_default(fmt::format("{}/{}", values.subsys.get_log_level(i), values.subsys.get_gather_level(i))); + opt.set_description(fmt::format("Debug level for {}", values.subsys.get_name(i)).c_str()); opt.set_flag(Option::FLAG_RUNTIME); opt.set_long_description("The value takes the form 'N' or 'N/M' where N and M are values between 0 and 99. N is the debug level to log (all values below this are included), and M is the level to gather and buffer in memory. In the event of a crash, the most recent items <= M are dumped to the log file."); opt.set_subsys(i); @@ -158,7 +157,7 @@ md_config_t::md_config_t(ConfigValues& values, } else { // normalize to M/N n = m; - *value = stringify(m) + "/" + stringify(n); + *value = fmt::format("{}/{}", m, n); } } else { *error_message = "value must take the form N or N/M, where N and M are integers"; @@ -775,7 +774,7 @@ int md_config_t::parse_option(ConfigValues& values, option_name = opt.name; if (ceph_argparse_witharg( args, i, &val, err, - string(string("--default-") + opt.name).c_str(), (char*)NULL)) { + fmt::format("--default-{}", opt.name).c_str(), (char*)NULL)) { if (!err.str().empty()) { error_message = err.str(); ret = -EINVAL; @@ -1268,7 +1267,7 @@ Option::value_t md_config_t::_expand_meta( << Option::to_str(*i->second) << "\n"; } } - return Option::value_t(std::string("$") + o->name); + return Option::value_t(fmt::format("${}", o->name)); } else { // recursively evaluate! string n; diff --git a/src/common/config_obs_mgr.h b/src/common/config_obs_mgr.h index 759930df92d..5336538e438 100644 --- a/src/common/config_obs_mgr.h +++ b/src/common/config_obs_mgr.h @@ -75,7 +75,7 @@ typename ObserverMgr<ConfigObs>::config_obs_wptr ObserverMgr<ConfigObs>::remove_ for (auto o = observers.begin(); o != observers.end(); ) { if (*o->second == observer) { ptr = std::move(o->second); - observers.erase(o++); + o = observers.erase(o); found_obs = true; } else { ++o; diff --git a/src/common/intrusive_lru.h b/src/common/intrusive_lru.h index 564cceef1cc..3ed3625d8a0 100644 --- a/src/common/intrusive_lru.h +++ b/src/common/intrusive_lru.h @@ -125,6 +125,7 @@ class intrusive_lru { using lru_list_t = boost::intrusive::list< base_t, + boost::intrusive::constant_time_size<false>, boost::intrusive::member_hook< base_t, boost::intrusive::list_member_hook<>, diff --git a/src/common/io_exerciser/RadosIo.cc b/src/common/io_exerciser/RadosIo.cc index a28a1e2f488..44b82260263 100644 --- a/src/common/io_exerciser/RadosIo.cc +++ b/src/common/io_exerciser/RadosIo.cc @@ -154,7 +154,9 @@ void RadosIo::applyIoOp(IoOp &op) version_t ver, bufferlist bl) { ceph_assert(ec == boost::system::errc::success); - db->validate(op_info->bl1, op_info->offset1, op_info->length1); + ceph_assert(db->validate(op_info->bl1, + op_info->offset1, + op_info->length1)); finish_io(); }; librados::async_operate(asio, io, oid, @@ -181,8 +183,12 @@ void RadosIo::applyIoOp(IoOp &op) version_t ver, bufferlist bl) { ceph_assert(ec == boost::system::errc::success); - db->validate(op_info->bl1, op_info->offset1, op_info->length1); - db->validate(op_info->bl2, op_info->offset2, op_info->length2); + ceph_assert(db->validate(op_info->bl1, + op_info->offset1, + op_info->length1)); + ceph_assert(db->validate(op_info->bl2, + op_info->offset2, + op_info->length2)); finish_io(); }; librados::async_operate(asio, io, oid, @@ -210,9 +216,15 @@ void RadosIo::applyIoOp(IoOp &op) version_t ver, bufferlist bl) { ceph_assert(ec == boost::system::errc::success); - db->validate(op_info->bl1, op_info->offset1, op_info->length1); - db->validate(op_info->bl2, op_info->offset2, op_info->length2); - db->validate(op_info->bl3, op_info->offset3, op_info->length3); + ceph_assert(db->validate(op_info->bl1, + op_info->offset1, + op_info->length1)); + ceph_assert(db->validate(op_info->bl2, + op_info->offset2, + op_info->length2)); + ceph_assert(db->validate(op_info->bl3, + op_info->offset3, + op_info->length3)); finish_io(); }; librados::async_operate(asio, io, oid, diff --git a/src/common/options.h b/src/common/options.h index ad39936d43a..abded4cc0dd 100644 --- a/src/common/options.h +++ b/src/common/options.h @@ -207,8 +207,8 @@ struct Option { typedef std::function<int(std::string *, std::string *)> validator_fn_t; validator_fn_t validator; - Option(std::string const &name, type_t t, level_t l) - : name(name), type(t), level(l) + Option(std::string &&name, type_t t, level_t l) + : name(std::move(name)), type(t), level(l) { // While value_t is nullable (via std::monostate), we don't ever // want it set that way in an Option instance: within an instance, diff --git a/src/mds/Beacon.cc b/src/mds/Beacon.cc index 059b540feb0..642d3428a27 100644 --- a/src/mds/Beacon.cc +++ b/src/mds/Beacon.cc @@ -17,6 +17,7 @@ #include "common/likely.h" #include "common/HeartbeatMap.h" +#include "include/compat.h" // for ceph_pthread_setname() #include "include/stringify.h" #include "include/util.h" @@ -73,6 +74,7 @@ void Beacon::init(const MDSMap &mdsmap) _notify_mdsmap(mdsmap); sender = std::thread([this]() { + ceph_pthread_setname(pthread_self(), "beacon"); std::unique_lock<std::mutex> lock(mutex); bool sent; while (!finished) { diff --git a/src/mds/CDir.cc b/src/mds/CDir.cc index f000da7928a..76e9fee68f8 100644 --- a/src/mds/CDir.cc +++ b/src/mds/CDir.cc @@ -1319,7 +1319,7 @@ void CDir::take_dentry_waiting(std::string_view dname, snapid_t first, snapid_t << it->first.snapid << " on " << *this << dendl; std::copy(it->second.begin(), it->second.end(), std::back_inserter(ls)); - waiting_on_dentry.erase(it++); + it = waiting_on_dentry.erase(it); } if (waiting_on_dentry.empty()) @@ -2823,8 +2823,6 @@ void CDir::_committed(int r, version_t v) auto it = waiting_for_commit.begin(); while (it != waiting_for_commit.end()) { - auto _it = it; - ++_it; if (it->first > committed_version) { dout(10) << " there are waiters for " << it->first << ", committing again" << dendl; _commit(it->first, -1); @@ -2834,8 +2832,7 @@ void CDir::_committed(int r, version_t v) for (const auto &waiter : it->second) t.push_back(waiter); mdcache->mds->queue_waiters(t); - waiting_for_commit.erase(it); - it = _it; + it = waiting_for_commit.erase(it); if (!(++count % mdcache->mds->heartbeat_reset_grace())) mdcache->mds->heartbeat_reset(); diff --git a/src/mds/MDCache.cc b/src/mds/MDCache.cc index c433c77b453..eb2b529dcfa 100644 --- a/src/mds/MDCache.cc +++ b/src/mds/MDCache.cc @@ -960,17 +960,15 @@ void MDCache::adjust_subtree_auth(CDir *dir, mds_authority_t auth, bool adjust_p dir->set_dir_auth(auth); // move items nested beneath me, under me. - set<CDir*>::iterator p = subtrees[root].begin(); + auto p = subtrees[root].begin(); while (p != subtrees[root].end()) { - set<CDir*>::iterator next = p; - ++next; if (get_subtree_root((*p)->get_parent_dir()) == dir) { // move under me dout(10) << " claiming child bound " << **p << dendl; subtrees[dir].insert(*p); - subtrees[root].erase(p); - } - p = next; + p = subtrees[root].erase(p); + } else + ++p; } // i am a bound of the parent subtree. @@ -1113,17 +1111,15 @@ void MDCache::adjust_bounded_subtree_auth(CDir *dir, const set<CDir*>& bounds, m dir->set_dir_auth(auth); // move items nested beneath me, under me. - set<CDir*>::iterator p = subtrees[root].begin(); + auto p = subtrees[root].begin(); while (p != subtrees[root].end()) { - set<CDir*>::iterator next = p; - ++next; if (get_subtree_root((*p)->get_parent_dir()) == dir) { // move under me dout(10) << " claiming child bound " << **p << dendl; subtrees[dir].insert(*p); - subtrees[root].erase(p); - } - p = next; + p = subtrees[root].erase(p); + } else + ++p; } // i am a bound of the parent subtree. @@ -1172,8 +1168,8 @@ void MDCache::adjust_bounded_subtree_auth(CDir *dir, const set<CDir*>& bounds, m } // merge stray bounds? while (!subtrees[dir].empty()) { - set<CDir*> copy = subtrees[dir]; - for (set<CDir*>::iterator p = copy.begin(); p != copy.end(); ++p) { + const auto copy = subtrees[dir]; + for (auto p = copy.begin(); p != copy.end(); ++p) { if (bounds.count(*p) == 0) { CDir *stray = *p; dout(10) << " swallowing extra subtree at " << *stray << dendl; @@ -1214,7 +1210,7 @@ void MDCache::get_force_dirfrag_bound_set(const vector<dirfrag_t>& dfs, set<CDir } dout(10) << " by ino: " << byino << dendl; - for (map<inodeno_t,fragset_t>::iterator p = byino.begin(); p != byino.end(); ++p) { + for (auto p = byino.begin(); p != byino.end(); ++p) { p->second.simplify(); CInode *diri = get_inode(p->first); if (!diri) @@ -1222,7 +1218,7 @@ void MDCache::get_force_dirfrag_bound_set(const vector<dirfrag_t>& dfs, set<CDir dout(10) << " checking fragset " << p->second.get() << " on " << *diri << dendl; fragtree_t tmpdft; - for (set<frag_t>::iterator q = p->second.begin(); q != p->second.end(); ++q) + for (auto q = p->second.begin(); q != p->second.end(); ++q) tmpdft.force_to_leaf(g_ceph_context, *q); for (const auto& fg : p->second) { @@ -1267,7 +1263,7 @@ void MDCache::map_dirfrag_set(const list<dirfrag_t>& dfs, set<CDir*>& result) ino_fragset[df.ino].insert_raw(df.frag); } // get frags - for (map<inodeno_t, fragset_t>::iterator p = ino_fragset.begin(); + for (auto p = ino_fragset.begin(); p != ino_fragset.end(); ++p) { p->second.simplify(); @@ -1347,7 +1343,7 @@ void MDCache::get_wouldbe_subtree_bounds(CDir *dir, set<CDir*>& bounds) } else { // find them CDir *root = get_subtree_root(dir); - for (set<CDir*>::iterator p = subtrees[root].begin(); + for (auto p = subtrees[root].begin(); p != subtrees[root].end(); ++p) { CDir *t = *p; @@ -1415,7 +1411,7 @@ void MDCache::adjust_subtree_after_rename(CInode *diri, CDir *olddir, bool pop) CDir *newdir = diri->get_parent_dir(); if (pop) { - map<CInode*,list<pair<CDir*,CDir*> > >::iterator p = projected_subtree_renames.find(diri); + auto p = projected_subtree_renames.find(diri); ceph_assert(p != projected_subtree_renames.end()); ceph_assert(!p->second.empty()); ceph_assert(p->second.front().first == olddir); @@ -1815,7 +1811,7 @@ void MDCache::project_rstat_inode_to_frag(const MutationRef& mut, if (cur->last != CEPH_NOSNAP) { ceph_assert(cur->dirty_old_rstats.empty()); - set<snapid_t>::const_iterator q = snaps.lower_bound(std::max(first, floor)); + auto q = snaps.lower_bound(std::max(first, floor)); if (q == snaps.end() || *q > cur->last) return; } @@ -2487,7 +2483,7 @@ void MDCache::logged_leader_update(metareqid_t reqid) */ void MDCache::finish_committed_leaders() { - for (map<metareqid_t, uleader>::iterator p = uncommitted_leaders.begin(); + for (auto p = uncommitted_leaders.begin(); p != uncommitted_leaders.end(); ++p) { p->second.recovering = false; @@ -2536,16 +2532,16 @@ void MDCache::_move_subtree_map_bound(dirfrag_t df, dirfrag_t oldparent, dirfrag map<dirfrag_t,vector<dirfrag_t> >& subtrees) { if (subtrees.count(oldparent)) { - vector<dirfrag_t>& v = subtrees[oldparent]; + auto& v = subtrees[oldparent]; dout(10) << " removing " << df << " from " << oldparent << " bounds " << v << dendl; - for (vector<dirfrag_t>::iterator it = v.begin(); it != v.end(); ++it) + for (auto it = v.begin(); it != v.end(); ++it) if (*it == df) { v.erase(it); break; } } if (subtrees.count(newparent)) { - vector<dirfrag_t>& v = subtrees[newparent]; + auto& v = subtrees[newparent]; dout(10) << " adding " << df << " to " << newparent << " bounds " << v << dendl; v.push_back(df); } @@ -2766,7 +2762,7 @@ void MDCache::send_peer_resolves() map<mds_rank_t, ref_t<MMDSResolve>> resolves; if (mds->is_resolve()) { - for (map<metareqid_t, upeer>::iterator p = uncommitted_peers.begin(); + for (auto p = uncommitted_peers.begin(); p != uncommitted_peers.end(); ++p) { mds_rank_t leader = p->second.leader; @@ -2777,7 +2773,7 @@ void MDCache::send_peer_resolves() } else { set<mds_rank_t> resolve_set; mds->mdsmap->get_mds_set(resolve_set, MDSMap::STATE_RESOLVE); - for (ceph::unordered_map<metareqid_t, MDRequestRef>::iterator p = active_requests.begin(); + for (auto p = active_requests.begin(); p != active_requests.end(); ++p) { MDRequestRef& mdr = p->second; @@ -2828,7 +2824,7 @@ void MDCache::send_subtree_resolves() } map<mds_rank_t, ref_t<MMDSResolve>> resolves; - for (set<mds_rank_t>::iterator p = recovery_set.begin(); + for (auto p = recovery_set.begin(); p != recovery_set.end(); ++p) { if (*p == mds->get_nodeid()) @@ -2841,7 +2837,7 @@ void MDCache::send_subtree_resolves() map<dirfrag_t, vector<dirfrag_t> > my_ambig_imports; // known - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { CDir *dir = p->first; @@ -2858,7 +2854,7 @@ void MDCache::send_subtree_resolves() set<CDir*> bounds; get_subtree_bounds(dir, bounds); vector<dirfrag_t> dfls; - for (set<CDir*>::iterator q = bounds.begin(); q != bounds.end(); ++q) + for (auto q = bounds.begin(); q != bounds.end(); ++q) dfls.push_back((*q)->dirfrag()); my_ambig_imports[dir->dirfrag()] = dfls; @@ -2870,7 +2866,7 @@ void MDCache::send_subtree_resolves() } // bounds too vector<dirfrag_t> dfls; - for (set<CDir*>::iterator q = subtrees[dir].begin(); + for (auto q = subtrees[dir].begin(); q != subtrees[dir].end(); ++q) { CDir *bound = *q; @@ -2883,7 +2879,7 @@ void MDCache::send_subtree_resolves() } // ambiguous - for (map<dirfrag_t, vector<dirfrag_t> >::iterator p = my_ambiguous_imports.begin(); + for (auto p = my_ambiguous_imports.begin(); p != my_ambiguous_imports.end(); ++p) { my_ambig_imports[p->first] = p->second; @@ -2896,9 +2892,9 @@ void MDCache::send_subtree_resolves() while (i < p->second.size()) { dirfrag_t b = p->second[i]; if (my_subtrees.count(b)) { - vector<dirfrag_t>& bb = my_subtrees[b]; + auto& bb = my_subtrees[b]; dout(10) << " simplify: " << p->first << " swallowing " << b << " with bounds " << bb << dendl; - for (vector<dirfrag_t>::iterator r = bb.begin(); r != bb.end(); ++r) + for (auto r = bb.begin(); r != bb.end(); ++r) p->second.push_back(*r); my_subtrees.erase(b); p->second.erase(p->second.begin() + i); @@ -2963,7 +2959,7 @@ void MDCache::handle_mds_failure(mds_rank_t who) // clean up any requests peer to/from this node list<MDRequestRef> finish; - for (ceph::unordered_map<metareqid_t, MDRequestRef>::iterator p = active_requests.begin(); + for (auto p = active_requests.begin(); p != active_requests.end(); ++p) { MDRequestRef& mdr = p->second; @@ -3061,7 +3057,7 @@ void MDCache::handle_mds_failure(mds_rank_t who) } } - for (map<metareqid_t, uleader>::iterator p = uncommitted_leaders.begin(); + for (auto p = uncommitted_leaders.begin(); p != uncommitted_leaders.end(); ++p) { // The failed MDS may have already committed the peer update @@ -3080,7 +3076,7 @@ void MDCache::handle_mds_failure(mds_rank_t who) kick_find_ino_peers(who); kick_open_ino_peers(who); - for (map<dirfrag_t,fragment_info_t>::iterator p = fragments.begin(); + for (auto p = fragments.begin(); p != fragments.end(); ) { dirfrag_t df = p->first; fragment_info_t& info = p->second; @@ -3089,18 +3085,17 @@ void MDCache::handle_mds_failure(mds_rank_t who) if (info.notify_ack_waiting.erase(who) && info.notify_ack_waiting.empty()) { fragment_drop_locks(info); - fragment_maybe_finish(p++); + p = fragment_maybe_finish(p); } else { ++p; } continue; } - ++p; dout(10) << "cancelling fragment " << df << " bit " << info.bits << dendl; std::vector<CDir*> dirs; info.dirs.swap(dirs); - fragments.erase(df); + p = fragments.erase(p); fragment_unmark_unfreeze_dirs(dirs); } @@ -3126,7 +3121,7 @@ void MDCache::handle_mds_recovery(mds_rank_t who) MDSContext::vec waiters; // wake up any waiters in their subtrees - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { CDir *dir = p->first; @@ -3241,7 +3236,7 @@ void MDCache::handle_resolve(const cref_t<MMDSResolve> &m) map<client_t,Capability::Export> cap_exports = inode_caps.cap_exports; ceph_assert(get_inode(ino)); - for (map<client_t,Capability::Export>::iterator q = cap_exports.begin(); + for (auto q = cap_exports.begin(); q != cap_exports.end(); ++q) { Capability::Import& im = rejoin_imported_caps[from][ino][q->first]; @@ -3283,10 +3278,8 @@ void MDCache::handle_resolve(const cref_t<MMDSResolve> &m) if (mds->is_clientreplay() || mds->is_active() || mds->is_stopping()) { survivor = true; // check for any import success/failure (from this node) - map<dirfrag_t, vector<dirfrag_t> >::iterator p = my_ambiguous_imports.begin(); + auto p = my_ambiguous_imports.begin(); while (p != my_ambiguous_imports.end()) { - map<dirfrag_t, vector<dirfrag_t> >::iterator next = p; - ++next; CDir *dir = get_dirfrag(p->first); ceph_assert(dir); dout(10) << "checking ambiguous import " << *dir << dendl; @@ -3305,7 +3298,7 @@ void MDCache::handle_resolve(const cref_t<MMDSResolve> &m) bool inside = true; set<CDir*> bounds; get_force_dirfrag_bound_set(q.second, bounds); - for (set<CDir*>::iterator p = bounds.begin(); p != bounds.end(); ++p) { + for (auto p = bounds.begin(); p != bounds.end(); ++p) { CDir *bound = *p; if (bound->contains(dir)) { inside = false; // nope, bound is dir or parent of dir, not inside. @@ -3316,7 +3309,7 @@ void MDCache::handle_resolve(const cref_t<MMDSResolve> &m) claimed_by_sender = true; } - my_ambiguous_imports.erase(p); // no longer ambiguous. + p = my_ambiguous_imports.erase(p); // no longer ambiguous. if (claimed_by_sender) { dout(7) << "ambiguous import failed on " << *dir << dendl; migrator->import_reverse(dir); @@ -3324,8 +3317,8 @@ void MDCache::handle_resolve(const cref_t<MMDSResolve> &m) dout(7) << "ambiguous import succeeded on " << *dir << dendl; migrator->import_finish(dir, true); } - } - p = next; + } else + ++p; } } @@ -3507,9 +3500,9 @@ void MDCache::add_uncommitted_peer(metareqid_t reqid, LogSegment *ls, mds_rank_t if (su == nullptr) { return; } - for(set<CInode*>::iterator p = su->olddirs.begin(); p != su->olddirs.end(); ++p) + for(auto p = su->olddirs.begin(); p != su->olddirs.end(); ++p) uncommitted_peer_rename_olddir[*p]++; - for(set<CInode*>::iterator p = su->unlinked.begin(); p != su->unlinked.end(); ++p) + for(auto p = su->unlinked.begin(); p != su->unlinked.end(); ++p) uncommitted_peer_unlink[*p]++; } @@ -3533,9 +3526,9 @@ void MDCache::finish_uncommitted_peer(metareqid_t reqid, bool assert_exist) return; } // discard the non-auth subtree we renamed out of - for(set<CInode*>::iterator p = su->olddirs.begin(); p != su->olddirs.end(); ++p) { + for(auto p = su->olddirs.begin(); p != su->olddirs.end(); ++p) { CInode *diri = *p; - map<CInode*, int>::iterator it = uncommitted_peer_rename_olddir.find(diri); + auto it = uncommitted_peer_rename_olddir.find(diri); ceph_assert(it != uncommitted_peer_rename_olddir.end()); it->second--; if (it->second == 0) { @@ -3553,9 +3546,9 @@ void MDCache::finish_uncommitted_peer(metareqid_t reqid, bool assert_exist) ceph_assert(it->second > 0); } // removed the inodes that were unlinked by peer update - for(set<CInode*>::iterator p = su->unlinked.begin(); p != su->unlinked.end(); ++p) { + for(auto p = su->unlinked.begin(); p != su->unlinked.end(); ++p) { CInode *in = *p; - map<CInode*, int>::iterator it = uncommitted_peer_unlink.find(in); + auto it = uncommitted_peer_unlink.find(in); ceph_assert(it != uncommitted_peer_unlink.end()); it->second--; if (it->second == 0) { @@ -3598,13 +3591,13 @@ void MDCache::disambiguate_other_imports() bool recovering = !(mds->is_clientreplay() || mds->is_active() || mds->is_stopping()); // other nodes' ambiguous imports - for (map<mds_rank_t, map<dirfrag_t, vector<dirfrag_t> > >::iterator p = other_ambiguous_imports.begin(); + for (auto p = other_ambiguous_imports.begin(); p != other_ambiguous_imports.end(); ++p) { mds_rank_t who = p->first; dout(10) << "ambiguous imports for mds." << who << dendl; - for (map<dirfrag_t, vector<dirfrag_t> >::iterator q = p->second.begin(); + for (auto q = p->second.begin(); q != p->second.end(); ++q) { dout(10) << " ambiguous import " << q->first << " bounds " << q->second << dendl; @@ -3639,7 +3632,7 @@ void MDCache::disambiguate_my_imports() // my ambiguous imports mds_authority_t me_ambig(mds->get_nodeid(), mds->get_nodeid()); while (!my_ambiguous_imports.empty()) { - map<dirfrag_t, vector<dirfrag_t> >::iterator q = my_ambiguous_imports.begin(); + auto q = my_ambiguous_imports.begin(); CDir *dir = get_dirfrag(q->first); ceph_assert(dir); @@ -3667,7 +3660,7 @@ void MDCache::disambiguate_my_imports() mds->mdlog->flush(); // verify all my subtrees are unambiguous! - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { CDir *dir = p->first; @@ -3692,7 +3685,7 @@ void MDCache::add_ambiguous_import(CDir *base, const set<CDir*>& bounds) { // make a list vector<dirfrag_t> binos; - for (set<CDir*>::iterator p = bounds.begin(); + for (auto p = bounds.begin(); p != bounds.end(); ++p) binos.push_back((*p)->dirfrag()); @@ -3849,14 +3842,14 @@ void MDCache::recalc_auth_bits(bool replay) } set<CInode*> subtree_inodes; - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { if (p->first->dir_auth.first == mds->get_nodeid()) subtree_inodes.insert(p->first->inode); } - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { if (p->first->inode->is_mdsdir()) { @@ -4079,7 +4072,7 @@ void MDCache::rejoin_send_rejoins() ++q; } else { // remove reconnect with no session - p.second.second.erase(q++); + q = p.second.second.erase(q); } } rejoins[target]->cap_exports[p.first] = p.second.second; @@ -4096,7 +4089,7 @@ void MDCache::rejoin_send_rejoins() // check all subtrees - for (map<CDir*, set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { CDir *dir = p->first; @@ -4166,7 +4159,7 @@ void MDCache::rejoin_send_rejoins() if (!mds->is_rejoin()) { // i am survivor. send strong rejoin. // note request remote_auth_pins, xlocks - for (ceph::unordered_map<metareqid_t, MDRequestRef>::iterator p = active_requests.begin(); + for (auto p = active_requests.begin(); p != active_requests.end(); ++p) { MDRequestRef& mdr = p->second; @@ -4582,7 +4575,7 @@ void MDCache::handle_cache_rejoin_weak(const cref_t<MMDSCacheRejoin> &weak) } // weak base inodes? (root, stray, etc.) - for (set<vinodeno_t>::iterator p = weak->weak_inodes.begin(); + for (auto p = weak->weak_inodes.begin(); p != weak->weak_inodes.end(); ++p) { CInode *in = get_inode(*p); @@ -4616,7 +4609,7 @@ void MDCache::handle_cache_rejoin_weak(const cref_t<MMDSCacheRejoin> &weak) rejoin_scour_survivor_replicas(from, ack, acked_inodes, gather_locks); mds->send_message(ack, weak->get_connection()); - for (set<SimpleLock*>::iterator p = gather_locks.begin(); p != gather_locks.end(); ++p) { + for (auto p = gather_locks.begin(); p != gather_locks.end(); ++p) { if (!(*p)->is_stable()) mds->locker->eval_gather(*p); } @@ -5184,12 +5177,12 @@ void MDCache::handle_cache_rejoin_ack(const cref_t<MMDSCacheRejoin> &ack) auto bp = ack->imported_caps.cbegin(); decode(peer_imported, bp); - for (map<inodeno_t,map<client_t,Capability::Import> >::iterator p = peer_imported.begin(); + for (auto p = peer_imported.begin(); p != peer_imported.end(); ++p) { auto& ex = cap_exports.at(p->first); ceph_assert(ex.first == from); - for (map<client_t,Capability::Import>::iterator q = p->second.begin(); + for (auto q = p->second.begin(); q != p->second.end(); ++q) { auto r = ex.second.find(q->first); @@ -5271,7 +5264,7 @@ void MDCache::rejoin_trim_undef_inodes() dout(10) << "rejoin_trim_undef_inodes" << dendl; while (!rejoin_undef_inodes.empty()) { - set<CInode*>::iterator p = rejoin_undef_inodes.begin(); + auto p = rejoin_undef_inodes.begin(); CInode *in = *p; rejoin_undef_inodes.erase(p); @@ -5496,12 +5489,12 @@ bool MDCache::process_imported_caps() } // process caps that were exported by peer rename - for (map<inodeno_t,pair<mds_rank_t,map<client_t,Capability::Export> > >::iterator p = rejoin_peer_exports.begin(); + for (auto p = rejoin_peer_exports.begin(); p != rejoin_peer_exports.end(); ++p) { CInode *in = get_inode(p->first); ceph_assert(in); - for (map<client_t,Capability::Export>::iterator q = p->second.second.begin(); + for (auto q = p->second.second.begin(); q != p->second.second.end(); ++q) { auto r = rejoin_session_map.find(q->first); @@ -5568,7 +5561,7 @@ bool MDCache::process_imported_caps() } } } - cap_imports.erase(p++); // remove and move on + p = cap_imports.erase(p); // remove and move on } } else { trim_non_auth(); @@ -5690,7 +5683,7 @@ void MDCache::prepare_realm_merge(SnapRealm *realm, SnapRealm *parent_realm, for (auto p = realm->inodes_with_caps.begin(); !p.end(); ++p) split_inos.push_back((*p)->ino()); - for (set<SnapRealm*>::iterator p = realm->open_children.begin(); + for (auto p = realm->open_children.begin(); p != realm->open_children.end(); ++p) split_realms.push_back((*p)->inode->ino()); @@ -5737,12 +5730,12 @@ void MDCache::clean_open_file_lists() { dout(10) << "clean_open_file_lists" << dendl; - for (map<uint64_t,LogSegment*>::iterator p = mds->mdlog->segments.begin(); + for (auto p = mds->mdlog->segments.begin(); p != mds->mdlog->segments.end(); ++p) { LogSegment *ls = p->second; - elist<CInode*>::iterator q = ls->open_files.begin(member_offset(CInode, item_open_file)); + auto q = ls->open_files.begin(member_offset(CInode, item_open_file)); while (!q.end()) { CInode *in = *q; ++q; @@ -5828,7 +5821,7 @@ void MDCache::export_remaining_imported_caps() mds->heartbeat_reset(); } - for (map<inodeno_t, MDSContext::vec >::iterator p = cap_reconnect_waiters.begin(); + for (auto p = cap_reconnect_waiters.begin(); p != cap_reconnect_waiters.end(); ++p) mds->queue_waiters(p->second); @@ -5869,7 +5862,7 @@ Capability* MDCache::try_reconnect_cap(CInode *in, Session *session) dout(15) << " chose lock states on " << *in << dendl; } - map<inodeno_t, MDSContext::vec >::iterator it = + auto it = cap_reconnect_waiters.find(in->ino()); if (it != cap_reconnect_waiters.end()) { mds->queue_waiters(it->second); @@ -5956,7 +5949,7 @@ void MDCache::open_snaprealms() } } - rejoin_pending_snaprealms.erase(it++); + it = rejoin_pending_snaprealms.erase(it); in->put(CInode::PIN_OPENINGSNAPPARENTS); send_snaps(splits); @@ -6094,10 +6087,10 @@ void MDCache::rejoin_send_acks() dout(7) << "rejoin_send_acks" << dendl; // replicate stray - for (map<mds_rank_t, set<CInode*> >::iterator p = rejoin_unlinked_inodes.begin(); + for (auto p = rejoin_unlinked_inodes.begin(); p != rejoin_unlinked_inodes.end(); ++p) { - for (set<CInode*>::iterator q = p->second.begin(); + for (auto q = p->second.begin(); q != p->second.end(); ++q) { CInode *in = *q; @@ -6127,7 +6120,7 @@ void MDCache::rejoin_send_acks() // send acks to everyone in the recovery set map<mds_rank_t,ref_t<MMDSCacheRejoin>> acks; - for (set<mds_rank_t>::iterator p = recovery_set.begin(); + for (auto p = recovery_set.begin(); p != recovery_set.end(); ++p) { if (rejoin_ack_sent.count(*p)) @@ -6138,7 +6131,7 @@ void MDCache::rejoin_send_acks() rejoin_ack_sent = recovery_set; // walk subtrees - for (map<CDir*,set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { CDir *dir = p->first; @@ -6236,7 +6229,7 @@ void MDCache::rejoin_send_acks() } // include inode base for any inodes whose scatterlocks may have updated - for (set<CInode*>::iterator p = rejoin_potential_updated_scatterlocks.begin(); + for (auto p = rejoin_potential_updated_scatterlocks.begin(); p != rejoin_potential_updated_scatterlocks.end(); ++p) { CInode *in = *p; @@ -6663,7 +6656,7 @@ void MDCache::truncate_inode_finish(CInode *in, LogSegment *ls) { dout(10) << "truncate_inode_finish " << *in << dendl; - set<CInode*>::iterator p = ls->truncating_inodes.find(in); + auto p = ls->truncating_inodes.find(in); ceph_assert(p != ls->truncating_inodes.end()); ls->truncating_inodes.erase(p); @@ -6719,7 +6712,7 @@ void MDCache::remove_recovered_truncate(CInode *in, LogSegment *ls) dout(20) << "remove_recovered_truncate " << *in << " in log segment " << ls->seq << "/" << ls->offset << dendl; // if we have the logseg the truncate started in, it must be in our list. - set<CInode*>::iterator p = ls->truncating_inodes.find(in); + auto p = ls->truncating_inodes.find(in); ceph_assert(p != ls->truncating_inodes.end()); ls->truncating_inodes.erase(p); in->put(CInode::PIN_TRUNCATING); @@ -6728,11 +6721,11 @@ void MDCache::remove_recovered_truncate(CInode *in, LogSegment *ls) void MDCache::start_recovered_truncates() { dout(10) << "start_recovered_truncates" << dendl; - for (map<uint64_t,LogSegment*>::iterator p = mds->mdlog->segments.begin(); + for (auto p = mds->mdlog->segments.begin(); p != mds->mdlog->segments.end(); ++p) { LogSegment *ls = p->second; - for (set<CInode*>::iterator q = ls->truncating_inodes.begin(); + for (auto q = ls->truncating_inodes.begin(); q != ls->truncating_inodes.end(); ++q) { CInode *in = *q; @@ -7006,7 +6999,7 @@ std::pair<bool, uint64_t> MDCache::trim(uint64_t count) // Other rank's base inodes (when I'm stopping) if (mds->is_stopping()) { - for (set<CInode*>::iterator p = base_inodes.begin(); + for (auto p = base_inodes.begin(); p != base_inodes.end();) { CInode *base_in = *p; ++p; @@ -7278,7 +7271,7 @@ void MDCache::trim_non_auth() dout(7) << "trim_non_auth" << dendl; // temporarily pin all subtree roots - for (map<CDir*, set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) p->first->get(CDir::PIN_SUBTREETEMP); @@ -7349,7 +7342,7 @@ void MDCache::trim_non_auth() lru.lru_touch_entire_pintail(); // unpin all subtrees - for (map<CDir*, set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) p->first->put(CDir::PIN_SUBTREETEMP); @@ -7461,7 +7454,7 @@ void MDCache::try_trim_non_auth_subtree(CDir *dir) // can we now trim child subtrees? set<CDir*> bounds; get_subtree_bounds(dir, bounds); - for (set<CDir*>::iterator p = bounds.begin(); p != bounds.end(); ++p) { + for (auto p = bounds.begin(); p != bounds.end(); ++p) { CDir *bd = *p; if (bd->get_dir_auth().first != mds->get_nodeid() && // we are not auth bd->get_num_any() == 0 && // and empty @@ -7746,7 +7739,7 @@ void MDCache::handle_cache_expire(const cref_t<MCacheExpire> &m) } } - for (set<SimpleLock*>::iterator p = gather_locks.begin(); p != gather_locks.end(); ++p) { + for (auto p = gather_locks.begin(); p != gather_locks.end(); ++p) { if (!(*p)->is_stable()) mds->locker->eval_gather(*p); } @@ -9292,7 +9285,7 @@ void MDCache::do_open_ino_peer(inodeno_t ino, open_ino_info_t& info) info.auth_hint = MDS_RANK_NONE; } } else { - for (set<mds_rank_t>::iterator p = active.begin(); p != active.end(); ++p) + for (auto p = active.begin(); p != active.end(); ++p) if (*p != whoami && info.checked.count(*p) == 0) { peer = *p; break; @@ -9405,7 +9398,7 @@ void MDCache::kick_open_ino_peers(mds_rank_t who) { dout(10) << "kick_open_ino_peers mds." << who << dendl; - for (map<inodeno_t, open_ino_info_t>::iterator p = opening_inodes.begin(); + for (auto p = opening_inodes.begin(); p != opening_inodes.end(); ++p) { open_ino_info_t& info = p->second; @@ -9546,7 +9539,7 @@ void MDCache::_do_find_ino_peer(find_ino_peer_info_t& fip) m = fip.hint; fip.hint = MDS_RANK_NONE; } else { - for (set<mds_rank_t>::iterator p = active.begin(); p != active.end(); ++p) + for (auto p = active.begin(); p != active.end(); ++p) if (*p != mds->get_nodeid() && fip.checked.count(*p) == 0) { m = *p; @@ -9645,7 +9638,7 @@ void MDCache::handle_find_ino_reply(const cref_t<MMDSFindInoReply> &m) void MDCache::kick_find_ino_peers(mds_rank_t who) { // find_ino_peers requests we should move on from - for (map<ceph_tid_t,find_ino_peer_info_t>::iterator p = find_ino_peer.begin(); + for (auto p = find_ino_peer.begin(); p != find_ino_peer.end(); ++p) { find_ino_peer_info_t& fip = p->second; @@ -9665,7 +9658,7 @@ void MDCache::kick_find_ino_peers(mds_rank_t who) int MDCache::get_num_client_requests() { int count = 0; - for (ceph::unordered_map<metareqid_t, MDRequestRef>::iterator p = active_requests.begin(); + for (auto p = active_requests.begin(); p != active_requests.end(); ++p) { MDRequestRef& mdr = p->second; @@ -9766,7 +9759,7 @@ MDRequestRef MDCache::request_start_internal(int op) MDRequestRef MDCache::request_get(metareqid_t rid) { - ceph::unordered_map<metareqid_t, MDRequestRef>::iterator p = active_requests.find(rid); + auto p = active_requests.find(rid); ceph_assert(p != active_requests.end()); dout(7) << "request_get " << rid << " " << *p->second << dendl; return p->second; @@ -10435,7 +10428,7 @@ void MDCache::discover_path(CDir *base, void MDCache::kick_discovers(mds_rank_t who) { - for (map<ceph_tid_t,discover_info_t>::iterator p = discovers.begin(); + for (auto p = discovers.begin(); p != discovers.end(); ++p) { if (p->second.mds != who) @@ -10772,7 +10765,7 @@ void MDCache::handle_discover_reply(const cref_t<MDiscoverReply> &m) // decrement discover counters if (m->get_tid()) { - map<ceph_tid_t,discover_info_t>::iterator p = discovers.find(m->get_tid()); + auto p = discovers.find(m->get_tid()); if (p != discovers.end()) { dout(10) << " found tid " << m->get_tid() << dendl; discovers.erase(p); @@ -11178,7 +11171,7 @@ int MDCache::send_dir_updates(CDir *dir, bool bcast) } mds_rank_t whoami = mds->get_nodeid(); - for (set<mds_rank_t>::iterator it = who.begin(); + for (auto it = who.begin(); it != who.end(); ++it) { if (*it == whoami) continue; @@ -11351,7 +11344,7 @@ void MDCache::send_dentry_unlink(CDentry *dn, CDentry *straydn, const MDRequestR CInode *strayin = straydn->get_linkage()->get_inode(); strayin->encode_snap_blob(snapbl); } - for (set<mds_rank_t>::iterator it = replicas.begin(); + for (auto it = replicas.begin(); it != replicas.end(); ++it) { // don't tell (rmdir) witnesses; they already know @@ -11588,7 +11581,7 @@ void MDCache::adjust_dir_fragments(CInode *diri, set<CDir*> bounds; bounds.swap(subtrees[dir]); subtrees.erase(dir); - for (set<CDir*>::iterator p = bounds.begin(); + for (auto p = bounds.begin(); p != bounds.end(); ++p) { CDir *frag = get_subtree_root((*p)->get_parent_dir()); @@ -11627,11 +11620,11 @@ void MDCache::adjust_dir_fragments(CInode *diri, for (const auto& dir : srcfrags) { ceph_assert(dir->is_subtree_root()); dout(10) << " taking srcfrag subtree bounds from " << *dir << dendl; - map<CDir*, set<CDir*> >::iterator q = subtrees.find(dir); - set<CDir*>::iterator r = q->second.begin(); + auto q = subtrees.find(dir); + auto r = q->second.begin(); while (r != subtrees[dir].end()) { new_bounds.insert(*r); - subtrees[dir].erase(r++); + r = subtrees[dir].erase(r); } subtrees.erase(q); @@ -11835,7 +11828,7 @@ public: void MDCache::fragment_mark_and_complete(const MDRequestRef& mdr) { dirfrag_t basedirfrag = mdr->more()->fragment_base; - map<dirfrag_t,fragment_info_t>::iterator it = fragments.find(basedirfrag); + auto it = fragments.find(basedirfrag); if (it == fragments.end() || it->second.mdr != mdr) { dout(7) << "fragment_mark_and_complete " << basedirfrag << " must have aborted" << dendl; request_finish(mdr); @@ -11938,8 +11931,7 @@ void MDCache::fragment_unmark_unfreeze_dirs(const std::vector<CDir*>& dirs) bool MDCache::fragment_are_all_frozen(CDir *dir) { ceph_assert(dir->is_frozen_dir()); - map<dirfrag_t,fragment_info_t>::iterator p; - for (p = fragments.lower_bound(dirfrag_t(dir->ino(), 0)); + for (auto p = fragments.lower_bound(dirfrag_t(dir->ino(), 0)); p != fragments.end() && p->first.ino == dir->ino(); ++p) { if (p->first.frag.contains(dir->get_frag())) @@ -11951,8 +11943,7 @@ bool MDCache::fragment_are_all_frozen(CDir *dir) void MDCache::fragment_freeze_inc_num_waiters(CDir *dir) { - map<dirfrag_t,fragment_info_t>::iterator p; - for (p = fragments.lower_bound(dirfrag_t(dir->ino(), 0)); + for (auto p = fragments.lower_bound(dirfrag_t(dir->ino(), 0)); p != fragments.end() && p->first.ino == dir->ino(); ++p) { if (p->first.frag.contains(dir->get_frag())) { @@ -11971,7 +11962,7 @@ void MDCache::find_stale_fragment_freeze() utime_t cutoff = now; cutoff -= g_conf()->mds_freeze_tree_timeout; - for (map<dirfrag_t,fragment_info_t>::iterator p = fragments.begin(); + for (auto p = fragments.begin(); p != fragments.end(); ) { dirfrag_t df = p->first; fragment_info_t& info = p->second; @@ -12060,7 +12051,7 @@ public: void MDCache::fragment_frozen(const MDRequestRef& mdr, int r) { dirfrag_t basedirfrag = mdr->more()->fragment_base; - map<dirfrag_t,fragment_info_t>::iterator it = fragments.find(basedirfrag); + auto it = fragments.find(basedirfrag); if (it == fragments.end() || it->second.mdr != mdr || r < 0) { dout(7) << "fragment_frozen " << basedirfrag << " must have aborted; rc=" << r << dendl; request_finish(mdr); @@ -12079,7 +12070,7 @@ void MDCache::fragment_frozen(const MDRequestRef& mdr, int r) void MDCache::dispatch_fragment_dir(const MDRequestRef& mdr, bool abort_if_freezing) { dirfrag_t basedirfrag = mdr->more()->fragment_base; - map<dirfrag_t,fragment_info_t>::iterator it = fragments.find(basedirfrag); + auto it = fragments.find(basedirfrag); if (it == fragments.end() || it->second.mdr != mdr) { dout(7) << __func__ << ": " << basedirfrag << " must have aborted" << dendl; request_finish(mdr); @@ -12402,12 +12393,12 @@ void MDCache::fragment_drop_locks(fragment_info_t& info) //info.mdr.reset(); } -void MDCache::fragment_maybe_finish(const fragment_info_iterator& it) +MDCache::fragment_info_iterator MDCache::fragment_maybe_finish(const fragment_info_iterator it) { ceph_assert(kill_dirfrag_at != dirfrag_killpoint::FRAGMENT_MAYBE_FINISH); if (!it->second.finishing) - return; + return it; // unmark & auth_unpin for (const auto &dir : it->second.resultfrags) { @@ -12421,7 +12412,7 @@ void MDCache::fragment_maybe_finish(const fragment_info_iterator& it) mds->balancer->maybe_fragment(dir, false); } - fragments.erase(it); + return fragments.erase(it); } @@ -12522,7 +12513,7 @@ void MDCache::finish_uncommitted_fragment(dirfrag_t basedirfrag, int op) { dout(10) << "finish_uncommitted_fragments: base dirfrag " << basedirfrag << " op " << EFragment::op_name(op) << dendl; - map<dirfrag_t, ufragment>::iterator it = uncommitted_fragments.find(basedirfrag); + auto it = uncommitted_fragments.find(basedirfrag); if (it != uncommitted_fragments.end()) { ufragment& uf = it->second; if (op != EFragment::OP_FINISH && !uf.old_frags.empty()) { @@ -12539,7 +12530,7 @@ void MDCache::rollback_uncommitted_fragment(dirfrag_t basedirfrag, frag_vec_t&& { dout(10) << "rollback_uncommitted_fragment: base dirfrag " << basedirfrag << " old_frags (" << old_frags << ")" << dendl; - map<dirfrag_t, ufragment>::iterator it = uncommitted_fragments.find(basedirfrag); + auto it = uncommitted_fragments.find(basedirfrag); if (it != uncommitted_fragments.end()) { ufragment& uf = it->second; if (!uf.old_frags.empty()) { @@ -12575,7 +12566,7 @@ struct C_MDC_FragmentRollback : public MDCacheLogContext { void MDCache::rollback_uncommitted_fragments() { dout(10) << "rollback_uncommitted_fragments: " << uncommitted_fragments.size() << " pending" << dendl; - for (map<dirfrag_t, ufragment>::iterator p = uncommitted_fragments.begin(); + for (auto p = uncommitted_fragments.begin(); p != uncommitted_fragments.end(); ++p) { ufragment &uf = p->second; @@ -12722,7 +12713,7 @@ void MDCache::show_subtrees(int dbl, bool force_print) // root frags std::vector<CDir*> basefrags; - for (set<CInode*>::iterator p = base_inodes.begin(); + for (auto p = base_inodes.begin(); p != base_inodes.end(); ++p) (*p)->get_dirfrags(basefrags); @@ -12760,13 +12751,11 @@ void MDCache::show_subtrees(int dbl, bool force_print) seen.insert(dir); // nested items? - if (!subtrees[dir].empty()) { - for (set<CDir*>::iterator p = subtrees[dir].begin(); - p != subtrees[dir].end(); - ++p) { - //dout(25) << " saw sub " << **p << dendl; - q.push_front(pair<CDir*,int>(*p, d+1)); - } + for (auto p = subtrees[dir].begin(); + p != subtrees[dir].end(); + ++p) { + //dout(25) << " saw sub " << **p << dendl; + q.push_front(pair<CDir*,int>(*p, d+1)); } } @@ -12831,7 +12820,7 @@ void MDCache::show_subtrees(int dbl, bool force_print) else indent += " "; - for (set<CDir*>::iterator p = subtrees[dir].begin(); + for (auto p = subtrees[dir].begin(); p != subtrees[dir].end(); ++p) q.push_front(pair<CDir*,int>(*p, d+2)); @@ -12840,7 +12829,7 @@ void MDCache::show_subtrees(int dbl, bool force_print) // verify there isn't stray crap in subtree map int lost = 0; - for (map<CDir*, set<CDir*> >::iterator p = subtrees.begin(); + for (auto p = subtrees.begin(); p != subtrees.end(); ++p) { if (subtrees_seen.count(p->first)) continue; diff --git a/src/mds/MDCache.h b/src/mds/MDCache.h index 0b01c9ab859..3c5d7e5e4f4 100644 --- a/src/mds/MDCache.h +++ b/src/mds/MDCache.h @@ -1485,7 +1485,7 @@ private: void fragment_frozen(const MDRequestRef& mdr, int r); void fragment_unmark_unfreeze_dirs(const std::vector<CDir*>& dirs); void fragment_drop_locks(fragment_info_t &info); - void fragment_maybe_finish(const fragment_info_iterator& it); + fragment_info_iterator fragment_maybe_finish(const fragment_info_iterator it); void dispatch_fragment_dir(const MDRequestRef& mdr, bool abort_if_freezing=false); void _fragment_logged(const MDRequestRef& mdr); void _fragment_stored(const MDRequestRef& mdr); diff --git a/src/mgr/ActivePyModule.h b/src/mgr/ActivePyModule.h index 187fb68f846..8538f6e236a 100644 --- a/src/mgr/ActivePyModule.h +++ b/src/mgr/ActivePyModule.h @@ -27,6 +27,8 @@ #include "PyModuleRunner.h" #include "PyModule.h" +#include <fmt/core.h> + #include <vector> #include <string> @@ -46,7 +48,6 @@ private: std::string m_command_perms; const MgrSession* m_session = nullptr; - std::string fin_thread_name; public: Finisher finisher; // per active module finisher to execute commands @@ -54,8 +55,7 @@ public: ActivePyModule(const PyModuleRef &py_module_, LogChannelRef clog_) : PyModuleRunner(py_module_, clog_), - fin_thread_name(std::string("m-fin-" + py_module->get_name()).substr(0,15)), - finisher(g_ceph_context, thread_name, fin_thread_name) + finisher(g_ceph_context, thread_name, fmt::format("m-fin-{}", py_module->get_name()).substr(0,15)) { } @@ -97,14 +97,14 @@ public: uri = str; } - std::string get_uri() const + std::string_view get_uri() const { return uri; } - std::string get_fin_thread_name() const + std::string_view get_fin_thread_name() const { - return fin_thread_name; + return finisher.get_thread_name(); } bool is_authorized(const std::map<std::string, std::string>& arguments) const; diff --git a/src/mgr/ActivePyModules.cc b/src/mgr/ActivePyModules.cc index 17bb3951142..aebbb5d8c9a 100644 --- a/src/mgr/ActivePyModules.cc +++ b/src/mgr/ActivePyModules.cc @@ -770,9 +770,9 @@ std::map<std::string, std::string> ActivePyModules::get_services() const std::map<std::string, std::string> result; std::lock_guard l(lock); for (const auto& [name, module] : modules) { - std::string svc_str = module->get_uri(); + const std::string_view svc_str = module->get_uri(); if (!svc_str.empty()) { - result[name] = svc_str; + result.emplace(name, svc_str); } } diff --git a/src/mon/ConfigMap.cc b/src/mon/ConfigMap.cc index 86528c1dedf..1444103f460 100644 --- a/src/mon/ConfigMap.cc +++ b/src/mon/ConfigMap.cc @@ -266,7 +266,7 @@ int ConfigMap::add_option( ldout(cct, 10) << __func__ << " unrecognized option '" << name << "'" << dendl; stray_options.push_back( std::unique_ptr<Option>( - new Option(name, Option::TYPE_STR, Option::LEVEL_UNKNOWN))); + new Option(std::string{name}, Option::TYPE_STR, Option::LEVEL_UNKNOWN))); opt = stray_options.back().get(); } diff --git a/src/mon/MgrMonitor.cc b/src/mon/MgrMonitor.cc index 8f7612540d0..b89878dddb7 100644 --- a/src/mon/MgrMonitor.cc +++ b/src/mon/MgrMonitor.cc @@ -217,7 +217,7 @@ void MgrMonitor::update_from_paxos(bool *need_bootstrap) string name = string("mgr/") + i.name + "/" + j.second.name; auto p = mgr_module_options.emplace( name, - Option(name, static_cast<Option::type_t>(j.second.type), + Option(std::string{name}, static_cast<Option::type_t>(j.second.type), static_cast<Option::level_t>(j.second.level))); Option& opt = p.first->second; opt.set_flags(static_cast<Option::flag_t>(j.second.flags)); diff --git a/src/msg/Message.h b/src/msg/Message.h index 3e5c58ec376..bb67ff3eef5 100644 --- a/src/msg/Message.h +++ b/src/msg/Message.h @@ -262,8 +262,8 @@ public: #endif protected: - ceph_msg_header header; // headerelope - ceph_msg_footer footer; + ceph_msg_header header{}; // headerelope + ceph_msg_footer footer{}; ceph::buffer::list payload; // "front" unaligned blob ceph::buffer::list middle; // "middle" unaligned blob ceph::buffer::list data; // data payload (page-alignment will be preserved where possible) @@ -332,16 +332,11 @@ protected: friend class Messenger; public: - Message() { - memset(&header, 0, sizeof(header)); - memset(&footer, 0, sizeof(footer)); - } + Message() = default; Message(int t, int version=1, int compat_version=0) { - memset(&header, 0, sizeof(header)); header.type = t; header.version = version; header.compat_version = compat_version; - memset(&footer, 0, sizeof(footer)); } Message *get() { diff --git a/src/msg/async/AsyncConnection.cc b/src/msg/async/AsyncConnection.cc index 683be086efa..ab3d454748e 100644 --- a/src/msg/async/AsyncConnection.cc +++ b/src/msg/async/AsyncConnection.cc @@ -310,7 +310,7 @@ ssize_t AsyncConnection::write(ceph::buffer::list &bl, outgoing_bl.claim_append(bl); ssize_t r = _try_send(more); if (r > 0) { - writeCallback = callback; + writeCallback = std::move(callback); } return r; } @@ -621,7 +621,7 @@ void AsyncConnection::fault() } void AsyncConnection::_stop() { - writeCallback.reset(); + writeCallback = {}; dispatch_queue->discard_queue(conn_id); async_msgr->unregister_conn(this); worker->release_worker(); @@ -737,8 +737,7 @@ void AsyncConnection::handle_write_callback() { recv_start_time = ceph::mono_clock::now(); write_lock.lock(); if (writeCallback) { - auto callback = *writeCallback; - writeCallback.reset(); + auto callback = std::move(writeCallback); write_lock.unlock(); callback(0); return; diff --git a/src/msg/async/AsyncConnection.h b/src/msg/async/AsyncConnection.h index 78a590f8ca3..a4f18e2c4fb 100644 --- a/src/msg/async/AsyncConnection.h +++ b/src/msg/async/AsyncConnection.h @@ -223,7 +223,7 @@ private: std::unique_ptr<Protocol> protocol; - std::optional<std::function<void(ssize_t)>> writeCallback; + std::function<void(ssize_t)> writeCallback; std::function<void(char *, ssize_t)> readCallback; std::optional<unsigned> pendingReadLen; char *read_buffer; diff --git a/src/msg/async/Event.h b/src/msg/async/Event.h index a595667e447..6acd6275738 100644 --- a/src/msg/async/Event.h +++ b/src/msg/async/Event.h @@ -97,11 +97,7 @@ class EventCenter { using clock_type = ceph::coarse_mono_clock; struct AssociatedCenters { - EventCenter *centers[MAX_EVENTCENTER]; - AssociatedCenters() { - // FIPS zeroization audit 20191115: this memset is not security related. - memset(centers, 0, MAX_EVENTCENTER * sizeof(EventCenter*)); - } + EventCenter *centers[MAX_EVENTCENTER]{}; }; struct FileEvent { diff --git a/src/msg/async/ProtocolV1.cc b/src/msg/async/ProtocolV1.cc index b14de7b1e56..a53f6389c31 100644 --- a/src/msg/async/ProtocolV1.cc +++ b/src/msg/async/ProtocolV1.cc @@ -90,9 +90,8 @@ void ProtocolV1::connect() { // reset connect state variables authorizer_buf.clear(); - // FIPS zeroization audit 20191115: these memsets are not security related. - memset(&connect_msg, 0, sizeof(connect_msg)); - memset(&connect_reply, 0, sizeof(connect_reply)); + connect_msg = {}; + connect_reply = {}; global_seq = messenger->get_global_seq(); } @@ -820,7 +819,7 @@ CtPtr ProtocolV1::read_message_data_prepare() { #if 0 // rx_buffers is broken by design... see // http://tracker.ceph.com/issues/22480 - map<ceph_tid_t, pair<ceph::buffer::list, int> >::iterator p = + const auto p = connection->rx_buffers.find(current_header.tid); if (p != connection->rx_buffers.end()) { ldout(cct, 10) << __func__ << " seleting rx buffer v " << p->second.second @@ -1205,7 +1204,7 @@ void ProtocolV1::requeue_sent() { return; } - list<out_q_entry_t> &rq = out_q[CEPH_MSG_PRIO_HIGHEST]; + auto &rq = out_q[CEPH_MSG_PRIO_HIGHEST]; out_seq -= sent.size(); while (!sent.empty()) { Message *m = sent.back(); @@ -1220,10 +1219,11 @@ void ProtocolV1::requeue_sent() { uint64_t ProtocolV1::discard_requeued_up_to(uint64_t out_seq, uint64_t seq) { ldout(cct, 10) << __func__ << " " << seq << dendl; std::lock_guard<std::mutex> l(connection->write_lock); - if (out_q.count(CEPH_MSG_PRIO_HIGHEST) == 0) { + const auto it = out_q.find(CEPH_MSG_PRIO_HIGHEST); + if (it == out_q.end()) { return seq; } - list<out_q_entry_t> &rq = out_q[CEPH_MSG_PRIO_HIGHEST]; + auto &rq = it->second; uint64_t count = out_seq; while (!rq.empty()) { Message* const m = rq.front().m; @@ -1235,7 +1235,7 @@ uint64_t ProtocolV1::discard_requeued_up_to(uint64_t out_seq, uint64_t seq) { rq.pop_front(); count++; } - if (rq.empty()) out_q.erase(CEPH_MSG_PRIO_HIGHEST); + if (rq.empty()) out_q.erase(it); return count; } @@ -1246,18 +1246,16 @@ uint64_t ProtocolV1::discard_requeued_up_to(uint64_t out_seq, uint64_t seq) { void ProtocolV1::discard_out_queue() { ldout(cct, 10) << __func__ << " started" << dendl; - for (list<Message *>::iterator p = sent.begin(); p != sent.end(); ++p) { - ldout(cct, 20) << __func__ << " discard " << *p << dendl; - (*p)->put(); + for (Message *msg : sent) { + ldout(cct, 20) << __func__ << " discard " << msg << dendl; + msg->put(); } sent.clear(); - for (map<int, list<out_q_entry_t>>::iterator p = - out_q.begin(); - p != out_q.end(); ++p) { - for (list<out_q_entry_t>::iterator r = p->second.begin(); - r != p->second.end(); ++r) { - ldout(cct, 20) << __func__ << " discard " << r->m << dendl; - r->m->put(); + for (auto& [ prio, entries ] : out_q) { + static_cast<void>(prio); + for (auto& entry : entries) { + ldout(cct, 20) << __func__ << " discard " << entry.m << dendl; + entry.m->put(); } } out_q.clear(); @@ -1296,7 +1294,7 @@ void ProtocolV1::reset_recv_state() // clean read and write callbacks connection->pendingReadLen.reset(); - connection->writeCallback.reset(); + connection->writeCallback = {}; if (state > THROTTLE_MESSAGE && state <= READ_FOOTER_AND_DISPATCH && connection->policy.throttler_messages) { @@ -1328,14 +1326,12 @@ void ProtocolV1::reset_recv_state() ProtocolV1::out_q_entry_t ProtocolV1::_get_next_outgoing() { out_q_entry_t out_entry; - if (!out_q.empty()) { - map<int, list<out_q_entry_t>>::reverse_iterator it = - out_q.rbegin(); + if (const auto it = out_q.begin(); it != out_q.end()) { ceph_assert(!it->second.empty()); - list<out_q_entry_t>::iterator p = it->second.begin(); + const auto p = it->second.begin(); out_entry = *p; it->second.erase(p); - if (it->second.empty()) out_q.erase(it->first); + if (it->second.empty()) out_q.erase(it); } return out_entry; } @@ -1572,8 +1568,7 @@ CtPtr ProtocolV1::handle_connect_message_write(int r) { CtPtr ProtocolV1::wait_connect_reply() { ldout(cct, 20) << __func__ << dendl; - // FIPS zeroization audit 20191115: this memset is not security related. - memset(&connect_reply, 0, sizeof(connect_reply)); + connect_reply = {}; return READ(sizeof(connect_reply), handle_connect_reply_1); } @@ -1923,8 +1918,7 @@ CtPtr ProtocolV1::handle_client_banner(char *buffer, int r) { CtPtr ProtocolV1::wait_connect_message() { ldout(cct, 20) << __func__ << dendl; - // FIPS zeroization audit 20191115: this memset is not security related. - memset(&connect_msg, 0, sizeof(connect_msg)); + connect_msg = {}; return READ(sizeof(connect_msg), handle_connect_message_1); } @@ -1988,8 +1982,7 @@ CtPtr ProtocolV1::handle_connect_message_2() { ceph_msg_connect_reply reply; ceph::buffer::list authorizer_reply; - // FIPS zeroization audit 20191115: this memset is not security related. - memset(&reply, 0, sizeof(reply)); + reply = {}; reply.protocol_version = messenger->get_proto_version(connection->peer_type, false); @@ -2616,8 +2609,7 @@ CtPtr ProtocolV1::server_ready() { << dendl; ldout(cct, 20) << __func__ << " accept done" << dendl; - // FIPS zeroization audit 20191115: this memset is not security related. - memset(&connect_msg, 0, sizeof(connect_msg)); + connect_msg = {}; if (connection->delay_state) { ceph_assert(connection->delay_state->ready()); diff --git a/src/msg/async/ProtocolV1.h b/src/msg/async/ProtocolV1.h index 1b7c1d2b5f8..63bc1cd0946 100644 --- a/src/msg/async/ProtocolV1.h +++ b/src/msg/async/ProtocolV1.h @@ -112,7 +112,12 @@ protected: bool is_prepared {false}; }; // priority queue for outbound msgs - std::map<int, std::list<out_q_entry_t>> out_q; + + /** + * A queue for each priority value, highest priority first. + */ + std::map<int, std::list<out_q_entry_t>, std::greater<int>> out_q; + bool keepalive; bool write_in_progress = false; diff --git a/src/msg/async/ProtocolV2.cc b/src/msg/async/ProtocolV2.cc index 6d44d6c783f..ed6f93cdd48 100644 --- a/src/msg/async/ProtocolV2.cc +++ b/src/msg/async/ProtocolV2.cc @@ -127,9 +127,9 @@ bool ProtocolV2::is_connected() { return can_write; } void ProtocolV2::discard_out_queue() { ldout(cct, 10) << __func__ << " started" << dendl; - for (auto p = sent.begin(); p != sent.end(); ++p) { - ldout(cct, 20) << __func__ << " discard " << *p << dendl; - (*p)->put(); + for (Message *msg : sent) { + ldout(cct, 20) << __func__ << " discard " << msg << dendl; + msg->put(); } sent.clear(); for (auto& [ prio, entries ] : out_queue) { @@ -211,10 +211,11 @@ void ProtocolV2::requeue_sent() { uint64_t ProtocolV2::discard_requeued_up_to(uint64_t out_seq, uint64_t seq) { ldout(cct, 10) << __func__ << " " << seq << dendl; std::lock_guard<std::mutex> l(connection->write_lock); - if (out_queue.count(CEPH_MSG_PRIO_HIGHEST) == 0) { + const auto it = out_queue.find(CEPH_MSG_PRIO_HIGHEST); + if (it == out_queue.end()) { return seq; } - auto& rq = out_queue[CEPH_MSG_PRIO_HIGHEST]; + auto& rq = it->second; uint64_t count = out_seq; while (!rq.empty()) { Message* const m = rq.front().m; @@ -226,7 +227,7 @@ uint64_t ProtocolV2::discard_requeued_up_to(uint64_t out_seq, uint64_t seq) { rq.pop_front(); count++; } - if (rq.empty()) out_queue.erase(CEPH_MSG_PRIO_HIGHEST); + if (rq.empty()) out_queue.erase(it); return count; } @@ -265,7 +266,7 @@ void ProtocolV2::reset_recv_state() { // clean read and write callbacks connection->pendingReadLen.reset(); - connection->writeCallback.reset(); + connection->writeCallback = {}; next_tag = static_cast<Tag>(0); @@ -507,14 +508,13 @@ void ProtocolV2::read_event() { ProtocolV2::out_queue_entry_t ProtocolV2::_get_next_outgoing() { out_queue_entry_t out_entry; - if (!out_queue.empty()) { - auto it = out_queue.rbegin(); + if (const auto it = out_queue.begin(); it != out_queue.end()) { auto& entries = it->second; ceph_assert(!entries.empty()); out_entry = entries.front(); entries.pop_front(); if (entries.empty()) { - out_queue.erase(it->first); + out_queue.erase(it); } } return out_entry; @@ -796,7 +796,7 @@ CtPtr ProtocolV2::read(CONTINUATION_RXBPTR_TYPE<ProtocolV2> &next, } template <class F> -CtPtr ProtocolV2::write(const std::string &desc, +CtPtr ProtocolV2::write(std::string_view desc, CONTINUATION_TYPE<ProtocolV2> &next, F &frame) { ceph::bufferlist bl; @@ -812,7 +812,7 @@ CtPtr ProtocolV2::write(const std::string &desc, return write(desc, next, bl); } -CtPtr ProtocolV2::write(const std::string &desc, +CtPtr ProtocolV2::write(std::string_view desc, CONTINUATION_TYPE<ProtocolV2> &next, ceph::bufferlist &buffer) { if (unlikely(pre_auth.enabled)) { diff --git a/src/msg/async/ProtocolV2.h b/src/msg/async/ProtocolV2.h index 6441866fea4..1ee258c4975 100644 --- a/src/msg/async/ProtocolV2.h +++ b/src/msg/async/ProtocolV2.h @@ -93,7 +93,12 @@ private: bool is_prepared {false}; Message* m {nullptr}; }; - std::map<int, std::list<out_queue_entry_t>> out_queue; + + /** + * A queue for each priority value, highest priority first. + */ + std::map<int, std::list<out_queue_entry_t>, std::greater<int>> out_queue; + std::list<Message *> sent; std::atomic<uint64_t> out_seq{0}; std::atomic<uint64_t> in_seq{0}; @@ -130,10 +135,10 @@ private: Ct<ProtocolV2> *read(CONTINUATION_RXBPTR_TYPE<ProtocolV2> &next, rx_buffer_t&& buffer); template <class F> - Ct<ProtocolV2> *write(const std::string &desc, + Ct<ProtocolV2> *write(std::string_view desc, CONTINUATION_TYPE<ProtocolV2> &next, F &frame); - Ct<ProtocolV2> *write(const std::string &desc, + Ct<ProtocolV2> *write(std::string_view desc, CONTINUATION_TYPE<ProtocolV2> &next, ceph::bufferlist &buffer); diff --git a/src/msg/async/frames_v2.cc b/src/msg/async/frames_v2.cc index ef4a6ddabfb..a9b03c74d4d 100644 --- a/src/msg/async/frames_v2.cc +++ b/src/msg/async/frames_v2.cc @@ -63,9 +63,7 @@ static bool check_epilogue_late_status(__u8 late_status) { void FrameAssembler::fill_preamble(Tag tag, preamble_block_t& preamble) const { - // FIPS zeroization audit 20191115: this memset is not security related. - ::memset(&preamble, 0, sizeof(preamble)); - + preamble = {}; preamble.tag = static_cast<__u8>(tag); for (size_t i = 0; i < m_descs.size(); i++) { preamble.segments[i].length = m_descs[i].logical_len; @@ -100,9 +98,7 @@ uint64_t FrameAssembler::get_frame_onwire_len() const { bufferlist FrameAssembler::asm_crc_rev0(const preamble_block_t& preamble, bufferlist segment_bls[]) const { - epilogue_crc_rev0_block_t epilogue; - // FIPS zeroization audit 20191115: this memset is not security related. - ::memset(&epilogue, 0, sizeof(epilogue)); + epilogue_crc_rev0_block_t epilogue{}; bufferlist frame_bl(sizeof(preamble) + sizeof(epilogue)); frame_bl.append(reinterpret_cast<const char*>(&preamble), sizeof(preamble)); @@ -123,9 +119,7 @@ bufferlist FrameAssembler::asm_secure_rev0(const preamble_block_t& preamble, preamble_bl.append(reinterpret_cast<const char*>(&preamble), sizeof(preamble)); - epilogue_secure_rev0_block_t epilogue; - // FIPS zeroization audit 20191115: this memset is not security related. - ::memset(&epilogue, 0, sizeof(epilogue)); + epilogue_secure_rev0_block_t epilogue{}; bufferlist epilogue_bl(sizeof(epilogue)); epilogue_bl.append(reinterpret_cast<const char*>(&epilogue), sizeof(epilogue)); @@ -151,9 +145,7 @@ bufferlist FrameAssembler::asm_secure_rev0(const preamble_block_t& preamble, bufferlist FrameAssembler::asm_crc_rev1(const preamble_block_t& preamble, bufferlist segment_bls[]) const { - epilogue_crc_rev1_block_t epilogue; - // FIPS zeroization audit 20191115: this memset is not security related. - ::memset(&epilogue, 0, sizeof(epilogue)); + epilogue_crc_rev1_block_t epilogue{}; epilogue.late_status |= FRAME_LATE_STATUS_COMPLETE; bufferlist frame_bl(sizeof(preamble) + FRAME_CRC_SIZE + sizeof(epilogue)); @@ -215,9 +207,7 @@ bufferlist FrameAssembler::asm_secure_rev1(const preamble_block_t& preamble, return frame_bl; // no epilogue if only one segment } - epilogue_secure_rev1_block_t epilogue; - // FIPS zeroization audit 20191115: this memset is not security related. - ::memset(&epilogue, 0, sizeof(epilogue)); + epilogue_secure_rev1_block_t epilogue{}; epilogue.late_status |= FRAME_LATE_STATUS_COMPLETE; bufferlist epilogue_bl(sizeof(epilogue)); epilogue_bl.append(reinterpret_cast<const char*>(&epilogue), diff --git a/src/os/bluestore/bluestore_tool.cc b/src/os/bluestore/bluestore_tool.cc index c24bf2161aa..d62721b4366 100644 --- a/src/os/bluestore/bluestore_tool.cc +++ b/src/os/bluestore/bluestore_tool.cc @@ -746,20 +746,25 @@ int main(int argc, char **argv) else if (action == "show-label") { JSONFormatter jf(true); jf.open_object_section("devices"); + bool any_success = false; for (auto& i : devs) { + jf.open_object_section(i.c_str()); bluestore_bdev_label_t label; int r = BlueStore::read_bdev_label(cct.get(), i, &label); if (r < 0) { - cerr << "unable to read label for " << i << ": " - << cpp_strerror(r) << std::endl; - exit(EXIT_FAILURE); + cerr << "unable to read label for " << i << ": " + << cpp_strerror(r) << std::endl; + } else { + any_success = true; + label.dump(&jf); } - jf.open_object_section(i.c_str()); - label.dump(&jf); jf.close_section(); } jf.close_section(); jf.flush(cout); + if (!any_success) { + exit(EXIT_FAILURE); + } } else if (action == "set-label-key") { bluestore_bdev_label_t label; diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts index da1a3f355c7..d79a4a6ccad 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-snapshotschedule-form/cephfs-snapshotschedule-form.component.ts @@ -19,6 +19,7 @@ import { CephfsSnapshotScheduleService } from '~/app/shared/api/cephfs-snapshot- import { CephfsSubvolumeService } from '~/app/shared/api/cephfs-subvolume.service'; import { DirectoryStoreService } from '~/app/shared/api/directory-store.service'; import { ActionLabelsI18n, URLVerbs } from '~/app/shared/constants/app.constants'; +import { DEFAULT_SUBVOLUME_GROUP } from '~/app/shared/constants/cephfs.constant'; import { Icons } from '~/app/shared/enum/icons.enum'; import { RepeatFrequency } from '~/app/shared/enum/repeat-frequency.enum'; import { RetentionFrequency } from '~/app/shared/enum/retention-frequency.enum'; @@ -35,7 +36,6 @@ import { TaskWrapperService } from '~/app/shared/services/task-wrapper.service'; const VALIDATON_TIMER = 300; const DEBOUNCE_TIMER = 300; -const DEFAULT_SUBVOLUME_GROUP = '_nogroup'; @Component({ selector: 'cd-cephfs-snapshotschedule-form', diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.ts index b5eb7a88681..be7b81940df 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-list/cephfs-subvolume-list.component.ts @@ -34,8 +34,7 @@ import { CephfsMountDetailsComponent } from '../cephfs-mount-details/cephfs-moun import { HealthService } from '~/app/shared/api/health.service'; import _ from 'lodash'; import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; - -const DEFAULT_SUBVOLUME_GROUP = '_nogroup'; +import { DEFAULT_SUBVOLUME_GROUP } from '~/app/shared/constants/cephfs.constant'; @Component({ selector: 'cd-cephfs-subvolume-list', diff --git a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.ts b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.ts index 92f337f85a6..0087ffd66cd 100644 --- a/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.ts +++ b/src/pybind/mgr/dashboard/frontend/src/app/ceph/cephfs/cephfs-subvolume-snapshots-list/cephfs-subvolume-snapshots-list.component.ts @@ -26,7 +26,7 @@ import moment from 'moment'; import { Validators } from '@angular/forms'; import { CdValidators } from '~/app/shared/forms/cd-validators'; import { ModalCdsService } from '~/app/shared/services/modal-cds.service'; -import { DEFAULT_SUBVOLUME_GROUP } from '~/app/shared/constants/cephfs'; +import { DEFAULT_SUBVOLUME_GROUP } from '~/app/shared/constants/cephfs.constant'; @Component({ selector: 'cd-cephfs-subvolume-snapshots-list', diff --git a/src/pybind/mgr/dashboard/frontend/src/app/shared/constants/cephfs.ts b/src/pybind/mgr/dashboard/frontend/src/app/shared/constants/cephfs.ts deleted file mode 100644 index 56890ff7214..00000000000 --- a/src/pybind/mgr/dashboard/frontend/src/app/shared/constants/cephfs.ts +++ /dev/null @@ -1 +0,0 @@ -export const DEFAULT_SUBVOLUME_GROUP = '_nogroup'; diff --git a/src/rgw/driver/rados/rgw_lc_tier.cc b/src/rgw/driver/rados/rgw_lc_tier.cc index e932c997621..b153a7b4a42 100644 --- a/src/rgw/driver/rados/rgw_lc_tier.cc +++ b/src/rgw/driver/rados/rgw_lc_tier.cc @@ -77,8 +77,9 @@ WRITE_CLASS_ENCODER(rgw_lc_multipart_upload_info) static inline string get_key_instance(const rgw_obj_key& key) { - if (!key.instance.empty() && - !key.have_null_instance()) { + // if non-current entry, add versionID to the + // transitioned object name including "null". + if (!key.instance.empty()) { return "-" + key.instance; } return ""; diff --git a/src/rgw/driver/rados/rgw_rados.cc b/src/rgw/driver/rados/rgw_rados.cc index d154082994e..a133b54dc59 100644 --- a/src/rgw/driver/rados/rgw_rados.cc +++ b/src/rgw/driver/rados/rgw_rados.cc @@ -5089,7 +5089,7 @@ int RGWRados::copy_obj_data(RGWObjectCtx& obj_ctx, int RGWRados::transition_obj(RGWObjectCtx& obj_ctx, RGWBucketInfo& bucket_info, - const rgw_obj& obj, + rgw_obj obj, const rgw_placement_rule& placement_rule, const real_time& mtime, uint64_t olh_epoch, @@ -5120,6 +5120,11 @@ int RGWRados::transition_obj(RGWObjectCtx& obj_ctx, return -ECANCELED; } + // bi expects empty instance for the entries created when bucket versioning + // is not enabled or suspended. + if (obj.key.instance == "null") { + obj.key.instance.clear(); + } attrs.erase(RGW_ATTR_ID_TAG); attrs.erase(RGW_ATTR_TAIL_TAG); diff --git a/src/rgw/driver/rados/rgw_rados.h b/src/rgw/driver/rados/rgw_rados.h index 9a2ba0af0e2..b24823b60dc 100644 --- a/src/rgw/driver/rados/rgw_rados.h +++ b/src/rgw/driver/rados/rgw_rados.h @@ -1234,7 +1234,7 @@ public: int transition_obj(RGWObjectCtx& obj_ctx, RGWBucketInfo& bucket_info, - const rgw_obj& obj, + rgw_obj obj, const rgw_placement_rule& placement_rule, const real_time& mtime, uint64_t olh_epoch, diff --git a/src/rgw/driver/rados/rgw_sal_rados.cc b/src/rgw/driver/rados/rgw_sal_rados.cc index bb416b0c2c3..11b86a25841 100644 --- a/src/rgw/driver/rados/rgw_sal_rados.cc +++ b/src/rgw/driver/rados/rgw_sal_rados.cc @@ -2792,6 +2792,13 @@ int RadosObject::write_cloud_tier(const DoutPrefixProvider* dpp, { rgw::sal::RadosPlacementTier* rtier = static_cast<rgw::sal::RadosPlacementTier*>(tier); map<string, bufferlist> attrs = get_attrs(); + rgw_obj_key& obj_key = get_key(); + // bi expects empty instance for the entries created when bucket versioning + // is not enabled or suspended. + if (obj_key.instance == "null") { + obj_key.instance.clear(); + } + RGWRados::Object op_target(store->getRados(), bucket->get_info(), *rados_ctx, get_obj()); RGWRados::Object::Write obj_op(&op_target); diff --git a/src/rgw/rgw_admin.cc b/src/rgw/rgw_admin.cc index a8874195217..b00dfaa1ec5 100644 --- a/src/rgw/rgw_admin.cc +++ b/src/rgw/rgw_admin.cc @@ -11187,22 +11187,22 @@ next: } formatter->open_object_section("result"); - formatter->open_array_section("topics"); - do { - rgw_pubsub_topics result; - int ret = ps.get_topics(dpp(), next_token, max_entries, - result, next_token, null_yield); - if (ret < 0 && ret != -ENOENT) { - cerr << "ERROR: could not get topics: " << cpp_strerror(-ret) << std::endl; - return -ret; - } - for (const auto& [_, topic] : result.topics) { - if (owner && *owner != topic.owner) { - continue; + rgw_pubsub_topics result; + if (rgw::all_zonegroups_support(*site, rgw::zone_features::notification_v2) && + driver->stat_topics_v1(tenant, null_yield, dpp()) == -ENOENT) { + formatter->open_array_section("topics"); + do { + int ret = ps.get_topics_v2(dpp(), next_token, max_entries, + result, next_token, null_yield); + if (ret < 0 && ret != -ENOENT) { + cerr << "ERROR: could not get topics: " << cpp_strerror(-ret) << std::endl; + return -ret; } - std::set<std::string> subscribed_buckets; - if (rgw::all_zonegroups_support(*site, rgw::zone_features::notification_v2) && - driver->stat_topics_v1(tenant, null_yield, dpp()) == -ENOENT) { + for (const auto& [_, topic] : result.topics) { + if (owner && *owner != topic.owner) { + continue; + } + std::set<std::string> subscribed_buckets; ret = driver->get_bucket_topic_mapping(topic, subscribed_buckets, null_yield, dpp()); if (ret < 0) { @@ -11210,15 +11210,21 @@ next: << topic.name << ", ret=" << ret << std::endl; } show_topics_info_v2(topic, subscribed_buckets, formatter.get()); - } else { - encode_json("result", result, formatter.get()); - } - if (max_entries_specified) { - --max_entries; + if (max_entries_specified) { + --max_entries; + } } + result.topics.clear(); + } while (!next_token.empty() && max_entries > 0); + formatter->close_section(); // topics + } else { // v1, list all topics + int ret = ps.get_topics_v1(dpp(), result, null_yield); + if (ret < 0 && ret != -ENOENT) { + cerr << "ERROR: could not get topics: " << cpp_strerror(-ret) << std::endl; + return -ret; } - } while (!next_token.empty() && max_entries > 0); - formatter->close_section(); // topics + encode_json("result", result, formatter.get()); + } if (max_entries_specified) { encode_json("truncated", !next_token.empty(), formatter.get()); if (!next_token.empty()) { diff --git a/src/rgw/rgw_lc.cc b/src/rgw/rgw_lc.cc index 78807888dfd..a7f2ceabad3 100644 --- a/src/rgw/rgw_lc.cc +++ b/src/rgw/rgw_lc.cc @@ -495,6 +495,14 @@ struct lc_op_ctx { octx(env.driver), dpp(dpp), wq(wq) { obj = bucket->get_object(o.key); + /* once bucket versioning is enabled, the non-current entries with + * instance empty should have instance set to "null" to be able + * to correctly read its olh version entry. + */ + if (o.key.instance.empty() && bucket->versioned() && !o.is_current()) { + rgw_obj_key& obj_key = obj->get_key(); + obj_key.instance = "null"; + } } bool next_has_same_name(const std::string& key_name) { @@ -1355,9 +1363,9 @@ public: int delete_tier_obj(lc_op_ctx& oc) { int ret = 0; - /* If bucket is versioned, create delete_marker for current version + /* If bucket has versioning enabled, create delete_marker for current version */ - if (! oc.bucket->versioned()) { + if (! oc.bucket->versioning_enabled()) { ret = remove_expired_obj(oc.dpp, oc, true, {rgw::notify::ObjectTransition}); ldpp_dout(oc.dpp, 20) << "delete_tier_obj Object(key:" << oc.o.key @@ -1387,9 +1395,10 @@ public: int transition_obj_to_cloud(lc_op_ctx& oc) { int ret{0}; - /* If CurrentVersion object, remove it & create delete marker */ + /* If CurrentVersion object & bucket has versioning enabled, remove it & + * create delete marker */ bool delete_object = (!oc.tier->retain_head_object() || - (oc.o.is_current() && oc.bucket->versioned())); + (oc.o.is_current() && oc.bucket->versioning_enabled())); /* notifications */ auto& bucket = oc.bucket; diff --git a/src/rgw/rgw_pubsub.cc b/src/rgw/rgw_pubsub.cc index 92b65b0ebba..cb68d72d7da 100644 --- a/src/rgw/rgw_pubsub.cc +++ b/src/rgw/rgw_pubsub.cc @@ -570,22 +570,16 @@ RGWPubSub::RGWPubSub(rgw::sal::Driver* _driver, { } -int RGWPubSub::get_topics(const DoutPrefixProvider* dpp, - const std::string& start_marker, int max_items, - rgw_pubsub_topics& result, std::string& next_marker, - optional_yield y) const +int RGWPubSub::get_topics_v2(const DoutPrefixProvider* dpp, + const std::string& start_marker, int max_items, + rgw_pubsub_topics& result, std::string& next_marker, + optional_yield y) const { if (rgw::account::validate_id(tenant)) { // if our tenant is an account, return the account listing return list_account_topics(dpp, start_marker, max_items, result, next_marker, y); } - - if (!use_notification_v2 || driver->stat_topics_v1(tenant, y, dpp) != -ENOENT) { - // in case of v1 or during migration we use v1 topics - // v1 returns all topics, ignoring marker/max_items - return read_topics_v1(dpp, result, nullptr, y); - } // TODO: prefix filter on 'tenant:' void* handle = NULL; @@ -629,6 +623,13 @@ int RGWPubSub::get_topics(const DoutPrefixProvider* dpp, return ret; } +int RGWPubSub::get_topics_v1(const DoutPrefixProvider* dpp, + rgw_pubsub_topics& result, + optional_yield y) const +{ + return read_topics_v1(dpp, result, nullptr, y); +} + int RGWPubSub::list_account_topics(const DoutPrefixProvider* dpp, const std::string& start_marker, int max_items, rgw_pubsub_topics& result, diff --git a/src/rgw/rgw_pubsub.h b/src/rgw/rgw_pubsub.h index b7ce443af03..8a6b290cb85 100644 --- a/src/rgw/rgw_pubsub.h +++ b/src/rgw/rgw_pubsub.h @@ -643,9 +643,14 @@ public: // get a paginated list of topics // return 0 on success, error code otherwise - int get_topics(const DoutPrefixProvider* dpp, - const std::string& start_marker, int max_items, - rgw_pubsub_topics& result, std::string& next_marker, + int get_topics_v2(const DoutPrefixProvider* dpp, + const std::string& start_marker, int max_items, + rgw_pubsub_topics& result, std::string& next_marker, + optional_yield y) const; + + // return 0 on success, error code otherwise + int get_topics_v1(const DoutPrefixProvider* dpp, + rgw_pubsub_topics& result, optional_yield y) const; // get a topic with by its name and populate it into "result" diff --git a/src/rgw/rgw_rest_pubsub.cc b/src/rgw/rgw_rest_pubsub.cc index a3784ca95b0..c0345a4f88a 100644 --- a/src/rgw/rgw_rest_pubsub.cc +++ b/src/rgw/rgw_rest_pubsub.cc @@ -493,8 +493,13 @@ void RGWPSListTopicsOp::execute(optional_yield y) { const std::string start_token = s->info.args.get("NextToken"); const RGWPubSub ps(driver, get_account_or_tenant(s->owner.id), *s->penv.site); - constexpr int max_items = 100; - op_ret = ps.get_topics(this, start_token, max_items, result, next_token, y); + if (rgw::all_zonegroups_support(*s->penv.site, rgw::zone_features::notification_v2) && + driver->stat_topics_v1(s->bucket->get_tenant(), null_yield, this) == -ENOENT) { + op_ret = ps.get_topics_v1(this, result, y); + } else { + constexpr int max_items = 100; + op_ret = ps.get_topics_v2(this, start_token, max_items, result, next_token, y); + } // if there are no topics it is not considered an error op_ret = op_ret == -ENOENT ? 0 : op_ret; if (op_ret < 0) { diff --git a/src/test/osd/ceph_test_rados_io_sequence.cc b/src/test/osd/ceph_test_rados_io_sequence.cc index dfc0304d00b..5e340c5c9c5 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.cc +++ b/src/test/osd/ceph_test_rados_io_sequence.cc @@ -27,15 +27,193 @@ #define dout_subsys ceph_subsys_rados #define dout_context g_ceph_context +namespace { + struct Size {}; + void validate(boost::any& v, const std::vector<std::string>& values, + Size *target_type, int) { + po::validators::check_first_occurrence(v); + const std::string &s = po::validators::get_single_string(values); + + std::string parse_error; + uint64_t size = strict_iecstrtoll(s, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + v = boost::any(size); + } + + struct Pair {}; + void validate(boost::any& v, const std::vector<std::string>& values, + Pair *target_type, int) { + po::validators::check_first_occurrence(v); + const std::string &s = po::validators::get_single_string(values); + auto part = ceph::split(s).begin(); + std::string parse_error; + int first = strict_iecstrtoll(*part++, &parse_error); + int second = strict_iecstrtoll(*part, &parse_error); + if (!parse_error.empty()) { + throw po::validation_error(po::validation_error::invalid_option_value); + } + v = boost::any(std::pair<int,int>{first,second}); + } + + struct PluginString {}; + void validate(boost::any& v, const std::vector<std::string>& values, + PluginString *target_type, int) { + po::validators::check_first_occurrence(v); + const std::string &s = po::validators::get_single_string(values); + + const std::string_view* pluginIt = std::find( + ceph::io_sequence::tester::pluginChoices.begin(), + ceph::io_sequence::tester::pluginChoices.end(), + s + ); + if(ceph::io_sequence::tester::pluginChoices.end() == pluginIt) + { + throw po::validation_error(po::validation_error::invalid_option_value); + } + + v = boost::any(*pluginIt); + } + + constexpr std::string_view usage[] = { + "Basic usage:", + "", + "ceph_test_rados_io_sequence", + "\t Test I/O to a single object using default settings. Good for", + "\t testing boundary conditions", + "", + "ceph_test_rados_io_sequence --parallel <n>", + "\t Run parallel test to multiple objects. First object is tested with", + "\t default settings, other objects are tested with random settings", + "", + "Advanced usage:", + "", + "ceph_test_rados_io_sequence --blocksize <b> --km <k,m> --plugin <p>", + " --objectsize <min,max> --threads <t>", + "ceph_test_rados_io_sequence --blocksize <b> --pool <p> --object <oid>", + " --objectsize <min,max> --threads <t>", + "\tCustomize the test, if a pool is specified then it defines the", + "\tReplica/EC configuration", + "", + "ceph_test_rados_io_sequence --listsequence", + "\t Display list of supported I/O sequences", + "", + "ceph_test_rados_io_sequence --dryrun --sequence <n>", + "\t Show I/O that will be generated for a sequence, validate", + "\t seqeunce has correct I/O barriers to restrict concurrency", + "", + "ceph_test_rados_io_sequence --seed <seed>", + "\t Repeat a previous test with the same random numbers (seed is", + "\t displayed at start of test), if threads = 1 then this will produce", + "\t the exact same sequence of I/O, if threads > 1 then I/Os are issued", + "\t in parallel so ordering might be slightly different", + "", + "ceph_test_rados_io_sequence --sequence <n> --seqseed <n>", + "\t Repeat a sequence from a previous test with the same random", + "\t numbers (seqseed is displayed at start of sequence)", + "", + "ceph_test_rados_io_sequence --pool <p> --object <oid> --interactive", + "\t Execute sequence of I/O commands from stdin. Offset and length", + "\t are specified with unit of blocksize. Supported commands:", + "\t\t create <len>", + "\t\t remove", + "\t\t read|write <off> <len>", + "\t\t read2|write2 <off> <len> <off> <len>", + "\t\t read3|write3 <off> <len> <off> <len> <off> <len>", + "\t\t done" + }; + + po::options_description get_options_description() + { + po::options_description desc("ceph_test_rados_io options"); + desc.add_options() + ("help,h", + "show help message") + ("listsequence,l", + "show list of sequences") + ("dryrun,d", + "test sequence, do not issue any I/O") + ("verbose", + "more verbose output during test") + ("sequence,s", po::value<int>(), + "test specified sequence") + ("seed", po::value<int>(), + "seed for whole test") + ("seqseed", po::value<int>(), + "seed for sequence") + ("blocksize,b", po::value<Size>(), + "block size (default 2048)") + ("chunksize,c", po::value<Size>(), + "chunk size (default 4096)") + ("pool,p", po::value<std::string>(), + "pool name") + ("object,o", po::value<std::string>()->default_value("test"), + "object name") + ("km", po::value<Pair>(), + "k,m EC pool profile (default 2,2)") + ("plugin", po::value<PluginString>(), + "EC plugin (isa or jerasure)") + ("objectsize", po::value<Pair>(), + "min,max object size in blocks (default 1,32)") + ("threads,t", po::value<int>(), + "number of threads of I/O per object (default 1)") + ("parallel,p", po::value<int>()->default_value(1), + "number of objects to exercise in parallel") + ("interactive", + "interactive mode, execute IO commands from stdin"); + + return desc; + } + + int parse_io_seq_options( + po::variables_map& vm, + int argc, + char** argv) + { + std::vector<std::string> unrecognized_options; + try { + po::options_description desc = get_options_description(); + + auto parsed = po::command_line_parser(argc, argv) + .options(desc) + .allow_unregistered() + .run(); + po::store(parsed, vm); + po::notify(vm); + unrecognized_options = po::collect_unrecognized(parsed.options, + po::include_positional); + + if (!unrecognized_options.empty()) + { + std::stringstream ss; + ss << "Unrecognised command options supplied: "; + while (unrecognized_options.size() > 1) + { + ss << unrecognized_options.back().c_str() << ", "; + unrecognized_options.pop_back(); + } + ss << unrecognized_options.back(); + dout(0) << ss.str() << dendl; + return 1; + } + } catch(const po::error& e) { + std::cerr << "error: " << e.what() << std::endl; + return 1; + } + + return 0; + } +} + template <typename T, int N, const std::array<T, N>& Ts> ceph::io_sequence::tester::ProgramOptionSelector<T, N, Ts> ::ProgramOptionSelector(ceph::util::random_number_generator<int>& rng, po::variables_map vm, const std::string& option_name, bool set_forced, - bool select_first) + bool select_first) : rng(rng), - // choices(choices), option_name(option_name) { if (set_forced && vm.count(option_name)) { force_value = vm[option_name].as<T>(); @@ -86,7 +264,7 @@ ceph::io_sequence::tester::SelectBlockSize::SelectBlockSize( ceph::io_sequence::tester::SelectNumThreads::SelectNumThreads( ceph::util::random_number_generator<int>& rng, - po::variables_map vm) + po::variables_map vm) : ProgramOptionSelector(rng, vm, "threads", true, true) { } @@ -99,9 +277,9 @@ ceph::io_sequence::tester::SelectSeqRange::SelectSeqRange( : ProgramOptionSelector(rng, vm, "sequence", false, false) { if (vm.count(option_name)) { - ceph::io_exerciser::Sequence s = + ceph::io_exerciser::Sequence s = static_cast<ceph::io_exerciser::Sequence>(vm["sequence"].as<int>()); - if (s < ceph::io_exerciser::Sequence::SEQUENCE_BEGIN || + if (s < ceph::io_exerciser::Sequence::SEQUENCE_BEGIN || s >= ceph::io_exerciser::Sequence::SEQUENCE_END) { dout(0) << "Sequence argument out of range" << dendl; throw po::validation_error(po::validation_error::invalid_option_value); @@ -127,7 +305,7 @@ const std::pair<ceph::io_exerciser::Sequence,ceph::io_exerciser::Sequence> ceph::io_sequence::tester::SelectErasureKM::SelectErasureKM( - ceph::util::random_number_generator<int>& rng, + ceph::util::random_number_generator<int>& rng, po::variables_map vm) : ProgramOptionSelector(rng, vm, "km", true, true) { @@ -180,11 +358,11 @@ const std::string ceph::io_sequence::tester::SelectECPool::choose() } int k = value.first; int m = value.second; - + const std::string plugin = std::string(spl.choose()); const uint64_t chunk_size = scs.choose(); - std::string pool_name = "ec_" + plugin + + std::string pool_name = "ec_" + plugin + "_cs" + std::to_string(chunk_size) + "_k" + std::to_string(k) + "_m" + std::to_string(m); @@ -234,33 +412,32 @@ ceph::io_sequence::tester::TestObject::TestObject( const std::string oid, SelectECPool& spo, SelectObjectSize& sos, SelectNumThreads& snt, - SelectSeqRange & ssr, + SelectSeqRange& ssr, ceph::util::random_number_generator<int>& rng, ceph::mutex& lock, ceph::condition_variable& cond, bool dryrun, bool verbose, - bool has_seqseed, - int seqseed) : - rng(rng), verbose(verbose), has_seqseed(has_seqseed), seqseed(seqseed) + std::optional<int> seqseed) : + rng(rng), verbose(verbose), seqseed(seqseed) { if (dryrun) { verbose = true; exerciser_model = std::make_unique<ceph::io_exerciser::ObjectModel>(oid, - sbs.choose(), - rng()); + sbs.choose(), + rng()); } else { const std::string pool = spo.choose(); int threads = snt.choose(); exerciser_model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, - asio, - pool, - oid, - sbs.choose(), - rng(), - threads, - lock, - cond); + asio, + pool, + oid, + sbs.choose(), + rng(), + threads, + lock, + cond); dout(0) << "= " << oid << " pool=" << pool << " threads=" << threads << " blocksize=" << exerciser_model->get_block_size() @@ -271,14 +448,12 @@ ceph::io_sequence::tester::TestObject::TestObject( const std::string oid, curseq = seq_range.first; seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, obj_size_range, - has_seqseed ? - seqseed : - rng()); + seqseed.value_or(rng())); op = seq->next(); done = false; - dout(0) << "== " << exerciser_model->get_oid() << " " - << curseq << " " - << seq->get_name() + dout(0) << "== " << exerciser_model->get_oid() << " " + << curseq << " " + << seq->get_name() << " ==" <<dendl; } @@ -291,12 +466,12 @@ bool ceph::io_sequence::tester::TestObject::next() { if (!done) { if (verbose) { - dout(0) << exerciser_model->get_oid() + dout(0) << exerciser_model->get_oid() << " Step " << seq->get_step() << ": " << op->to_string(exerciser_model->get_block_size()) << dendl; } else { - dout(5) << exerciser_model->get_oid() - << " Step " << seq->get_step() << ": " + dout(5) << exerciser_model->get_oid() + << " Step " << seq->get_step() << ": " << op->to_string(exerciser_model->get_block_size()) << dendl; } exerciser_model->applyIoOp(*op); @@ -310,10 +485,8 @@ bool ceph::io_sequence::tester::TestObject::next() } else { seq = ceph::io_exerciser::IoSequence::generate_sequence(curseq, obj_size_range, - has_seqseed ? - seqseed : - rng()); - dout(0) << "== " << exerciser_model->get_oid() << " " + seqseed.value_or(rng())); + dout(0) << "== " << exerciser_model->get_oid() << " " << curseq << " " << seq->get_name() << " ==" <<dendl; op = seq->next(); @@ -335,97 +508,232 @@ int ceph::io_sequence::tester::TestObject::get_num_io() return exerciser_model->get_num_io(); } -struct Size {}; -void validate(boost::any& v, const std::vector<std::string>& values, - Size *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); +ceph::io_sequence::tester::TestRunner::TestRunner(po::variables_map& vm, + librados::Rados& rados) : + rados(rados), + seed(vm.contains("seed") ? vm["seed"].as<int>() : time(nullptr)), + rng(ceph::util::random_number_generator<int>(seed)), + sbs{rng, vm}, + sos{rng, vm}, + spo{rng, vm, rados, vm.contains("dryrun")}, + snt{rng, vm}, + ssr{rng, vm} +{ + dout(0) << "Test using seed " << seed << dendl; - std::string parse_error; - uint64_t size = strict_iecstrtoll(s, &parse_error); - if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); + verbose = vm.contains("verbose"); + dryrun = vm.contains("dryrun"); + + seqseed = std::nullopt; + if (vm.contains("seqseed")) { + seqseed = vm["seqseed"].as<int>(); + } + num_objects = vm["parallel"].as<int>(); + object_name = vm["object"].as<std::string>(); + interactive = vm.contains("interactive"); + + if (!dryrun) + { + guard.emplace(boost::asio::make_work_guard(asio)); + thread = make_named_thread("io_thread",[&asio = asio] { asio.run(); }); } - v = boost::any(size); + + show_help = vm.contains("help"); + show_sequence = vm.contains("listsequence"); } -struct Pair {}; -void validate(boost::any& v, const std::vector<std::string>& values, - Pair *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - auto part = ceph::split(s).begin(); +ceph::io_sequence::tester::TestRunner::~TestRunner() +{ + if (!dryrun) { + guard = std::nullopt; + asio.stop(); + thread.join(); + rados.shutdown(); + } +} + +void ceph::io_sequence::tester::TestRunner::help() +{ + std::cout << get_options_description() << std::endl; + for (auto line : usage) { + std::cout << line << std::endl; + } +} + +void ceph::io_sequence::tester::TestRunner::list_sequence() +{ + // List seqeunces + std::pair<int,int> obj_size_range = sos.choose(); + for (ceph::io_exerciser::Sequence s + = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; + s < ceph::io_exerciser::Sequence::SEQUENCE_END; ++s) { + std::unique_ptr<ceph::io_exerciser::IoSequence> seq = + ceph::io_exerciser::IoSequence::generate_sequence(s, + obj_size_range, + seqseed.value_or(rng())); + dout(0) << s << " " << seq->get_name() << dendl; + } +} + +std::string ceph::io_sequence::tester::TestRunner::get_token() +{ + static std::string line; + static ceph::split split = ceph::split(""); + static ceph::spliterator tokens; + while (line.empty() || tokens == split.end()) { + if (!std::getline(std::cin, line)) { + throw std::runtime_error("End of input"); + } + split = ceph::split(line); + tokens = split.begin(); + } + return std::string(*tokens++); +} + +uint64_t ceph::io_sequence::tester::TestRunner::get_numeric_token() +{ std::string parse_error; - int first = strict_iecstrtoll(*part++, &parse_error); - int second = strict_iecstrtoll(*part, &parse_error); + std::string token = get_token(); + uint64_t num = strict_iecstrtoll(token, &parse_error); if (!parse_error.empty()) { - throw po::validation_error(po::validation_error::invalid_option_value); + throw std::runtime_error("Invalid number "+token); } - v = boost::any(std::pair<int,int>{first,second}); + return num; } -struct PluginString {}; -void validate(boost::any& v, const std::vector<std::string>& values, - PluginString *target_type, int) { - po::validators::check_first_occurrence(v); - const std::string &s = po::validators::get_single_string(values); - - const std::string_view* pluginIt = std::find( - ceph::io_sequence::tester::pluginChoices.begin(), - ceph::io_sequence::tester::pluginChoices.end(), - s - ); - if(ceph::io_sequence::tester::pluginChoices.end() == pluginIt) +bool ceph::io_sequence::tester::TestRunner::run_test() +{ + if (show_help) { - throw po::validation_error(po::validation_error::invalid_option_value); + help(); + return true; + } + else if (show_sequence) + { + list_sequence(); + return true; + } + else if (interactive) + { + return run_interactive_test(); + } + else + { + return run_automated_test(); } - - v = boost::any(*pluginIt); } -int parse_io_seq_options( - po::variables_map& vm, - const po::options_description& desc, - int argc, - char** argv) -{ - std::vector<std::string> unrecognized_options; - try { - auto parsed = po::command_line_parser(argc, argv) - .options(desc) - .allow_unregistered() - .run(); - po::store(parsed, vm); - po::notify(vm); - unrecognized_options = po::collect_unrecognized(parsed.options, - po::include_positional); - - if (!unrecognized_options.empty()) - { - std::stringstream ss; - ss << "Unrecognised command options supplied: "; - while (unrecognized_options.size() > 1) - { - ss << unrecognized_options.back().c_str() << ", "; - unrecognized_options.pop_back(); - } - ss << unrecognized_options.back(); - dout(0) << ss.str() << dendl; - return 1; +bool ceph::io_sequence::tester::TestRunner::run_interactive_test() +{ + bool done = false; + std::unique_ptr<ceph::io_exerciser::IoOp> ioop; + std::unique_ptr<ceph::io_exerciser::Model> model; + + if (dryrun) { + model = std::make_unique<ceph::io_exerciser::ObjectModel>(object_name, + sbs.choose(), + rng()); + } else { + const std::string pool = spo.choose(); + model = std::make_unique<ceph::io_exerciser::RadosIo>(rados, asio, pool, + object_name, sbs.choose(), + rng(), 1, // 1 thread + lock, cond); + } + + while (!done) { + const std::string op = get_token(); + if (!op.compare("done") || !op.compare("q") || !op.compare("quit")) { + ioop = ceph::io_exerciser::IoOp::generate_done(); + } else if (!op.compare("create")) { + ioop = ceph::io_exerciser::IoOp::generate_create(get_numeric_token()); + } else if (!op.compare("remove") || !op.compare("delete")) { + ioop = ceph::io_exerciser::IoOp::generate_remove(); + } else if (!op.compare("read")) { + uint64_t offset = get_numeric_token(); + uint64_t length = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_read(offset, length); + } else if (!op.compare("read2")) { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_read2(offset1, length1, + offset2, length2); + } else if (!op.compare("read3")) { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + uint64_t offset3 = get_numeric_token(); + uint64_t length3 = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_read3(offset1, length1, + offset2, length2, + offset3, length3); + } else if (!op.compare("write")) { + uint64_t offset = get_numeric_token(); + uint64_t length = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_write(offset, length); + } else if (!op.compare("write2")) { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_write2(offset1, length1, + offset2, length2); + } else if (!op.compare("write3")) { + uint64_t offset1 = get_numeric_token(); + uint64_t length1 = get_numeric_token(); + uint64_t offset2 = get_numeric_token(); + uint64_t length2 = get_numeric_token(); + uint64_t offset3 = get_numeric_token(); + uint64_t length3 = get_numeric_token(); + ioop = ceph::io_exerciser::IoOp::generate_write3(offset1, length1, + offset2, length2, + offset3, length3); + } else { + throw std::runtime_error("Invalid operation "+op); + } + dout(0) << ioop->to_string(model->get_block_size()) << dendl; + model->applyIoOp(*ioop); + done = ioop->done(); + if (!done) { + ioop = ceph::io_exerciser::IoOp::generate_barrier(); + model->applyIoOp(*ioop); } - } catch(const po::error& e) { - std::cerr << "error: " << e.what() << std::endl; - return 1; } - return 0; + return true; } -void run_test(const std::vector< - std::shared_ptr<ceph::io_sequence::tester::TestObject> - >& test_objects, - ceph::mutex& lock) +bool ceph::io_sequence::tester::TestRunner::run_automated_test() { + // Create a test for each object + std::vector<std::shared_ptr< + ceph::io_sequence::tester::TestObject>> test_objects; + + for (int obj = 0; obj < num_objects; obj++) { + std::string name; + if (obj == 0) { + name = object_name; + } else { + name = object_name + std::to_string(obj); + } + test_objects.push_back( + std::make_shared<ceph::io_sequence::tester::TestObject>( + name, + rados, asio, + sbs, spo, sos, snt, ssr, + rng, lock, cond, + dryrun, verbose, + seqseed + ) + ); + } + if (!dryrun) { + rados.wait_for_latest_osdmap(); + } + // Main loop of test - while not all test objects have finished // check to see if any are able to start a new I/O. If all test // objects are waiting for I/O to complete then wait on a cond @@ -475,56 +783,8 @@ void run_test(const std::vector< ceph_assert(to->finished()); } dout(0) << "Total number of IOs = " << total_io << dendl; -} -namespace { - constexpr std::string_view usage[] = { - "Basic usage:", - "", - "ceph_test_rados_io_sequence", - "\t Test I/O to a single object using default settings. Good for", - "\t testing boundary conditions", - "", - "ceph_test_rados_io_sequence --parallel <n>", - "\t Run parallel test to multiple objects. First object is tested with", - "\t default settings, other objects are tested with random settings", - "", - "Advanced usage:", - "", - "ceph_test_rados_io_sequence --blocksize <b> --km <k,m> --plugin <p>", - " --objectsize <min,max> --threads <t>", - "ceph_test_rados_io_sequence --blocksize <b> --pool <p> --object <oid>", - " --objectsize <min,max> --threads <t>", - "\tCustomize the test, if a pool is specified then it defines the", - "\tReplica/EC configuration", - "", - "ceph_test_rados_io_sequence --listsequence", - "\t Display list of supported I/O sequences", - "", - "ceph_test_rados_io_sequence --dryrun --sequence <n>", - "\t Show I/O that will be generated for a sequence, validate", - "\t seqeunce has correct I/O barriers to restrict concurrency", - "", - "ceph_test_rados_io_sequence --seed <seed>", - "\t Repeat a previous test with the same random numbers (seed is", - "\t displayed at start of test), if threads = 1 then this will produce", - "\t the exact same sequence of I/O, if threads > 1 then I/Os are issued", - "\t in parallel so ordering might be slightly different", - "", - "ceph_test_rados_io_sequence --sequence <n> --seqseed <n>", - "\t Repeat a sequence from a previous test with the same random", - "\t numbers (seqseed is displayed at start of sequence)", - "", - "ceph_test_rados_io_sequence --pool <p> --object <oid> --interactive", - "\t Execute sequence of I/O commands from stdin. Offset and length", - "\t are specified with unit of blocksize. Supported commands:", - "\t\t create <len>", - "\t\t remove", - "\t\t read|write <off> <len>", - "\t\t read2|write2 <off> <len> <off> <len>", - "\t\t read3|write3 <off> <len> <off> <len> <off> <len>", - "\t\t done" - }; + return true; } int main(int argc, char **argv) @@ -535,161 +795,28 @@ int main(int argc, char **argv) CODE_ENVIRONMENT_UTILITY, 0); common_init_finish(cct.get()); - librados::Rados rados; - boost::asio::io_context asio; - std::thread thread; - std::optional<boost::asio::executor_work_guard< - boost::asio::io_context::executor_type>> guard; - ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); - ceph::condition_variable cond; - - po::options_description desc("ceph_test_rados_io options"); - - desc.add_options() - ("help,h", - "show help message") - ("listsequence,l", - "show list of sequences") - ("dryrun,d", - "test sequence, do not issue any I/O") - ("verbose", - "more verbose output during test") - ("sequence,s", po::value<int>(), - "test specified sequence") - ("seed", po::value<int>(), - "seed for whole test") - ("seqseed", po::value<int>(), - "seed for sequence") - ("blocksize,b", po::value<Size>(), - "block size (default 2048)") - ("chunksize,c", po::value<Size>(), - "chunk size (default 4096)") - ("pool,p", po::value<std::string>(), - "pool name") - ("km", po::value<Pair>(), - "k,m EC pool profile (default 2,2)") - ("plugin", po::value<PluginString>(), - "EC plugin (isa or jerasure)") - ("objectsize", po::value<Pair>(), - "min,max object size in blocks (default 1,32)") - ("threads,t", po::value<int>(), - "number of threads of I/O per object (default 1)") - ("objects,o", po::value<int>()->default_value(1), - "number of objects to exercise in parallel"); - po::variables_map vm; - int rc = parse_io_seq_options(vm, desc, argc, argv); + int rc = parse_io_seq_options(vm, argc, argv); if (rc != 0) { return rc; } - if (vm.count("help")) { - std::cout << desc << std::endl; - for (auto line : usage) { - std::cout << line << std::endl; - } - return 0; - } - - // Seed - int seed = time(nullptr); - if (vm.count("seed")) { - seed = vm["seed"].as<int>(); - } - dout(0) << "Test using seed " << seed << dendl; - auto rng = ceph::util::random_number_generator<int>(seed); - - bool verbose = vm.count("verbose"); - bool dryrun = vm.count("dryrun"); - bool has_seqseed = vm.count("seqseed"); - int seqseed = 0; - if (has_seqseed) { - seqseed = vm["seqseed"].as<int>(); - } - int num_objects = vm["objects"].as<int>(); - - if (!dryrun) { + librados::Rados rados; + if (!vm.contains("dryrun")) { rc = rados.init_with_context(g_ceph_context); ceph_assert(rc == 0); rc = rados.connect(); ceph_assert(rc == 0); + } - guard.emplace(boost::asio::make_work_guard(asio)); - thread = make_named_thread("io_thread",[&asio] { asio.run(); }); - } - - // Select block size - std::unique_ptr<ceph::io_sequence::tester::SelectBlockSize> sbs - = std::make_unique<ceph::io_sequence::tester::SelectBlockSize>(rng, vm); - - // Select pool - std::unique_ptr<ceph::io_sequence::tester::SelectECPool> spo - = std::make_unique<ceph::io_sequence::tester::SelectECPool>(rng, vm, - rados, - dryrun); - - // Select object size range - std::unique_ptr<ceph::io_sequence::tester::SelectObjectSize> sos - = std::make_unique<ceph::io_sequence::tester::SelectObjectSize>(rng, - vm); - - // Select number of threads - std::unique_ptr<ceph::io_sequence::tester::SelectNumThreads> snt = - std::make_unique<ceph::io_sequence::tester::SelectNumThreads>(rng, vm); - - // Select range of sequences - std::unique_ptr<ceph::io_sequence::tester::SelectSeqRange> ssr; + std::unique_ptr<ceph::io_sequence::tester::TestRunner> runner; try { - ssr = std::make_unique<ceph::io_sequence::tester::SelectSeqRange>(rng, vm); + runner = std::make_unique<ceph::io_sequence::tester::TestRunner>(vm, rados); } catch(const po::error& e) { return 1; } + runner->run_test(); - // List seqeunces - if (vm.count("listsequence")) { - std::pair<int,int> obj_size_range = sos->choose(); - for (ceph::io_exerciser::Sequence s - = ceph::io_exerciser::Sequence::SEQUENCE_BEGIN; - s < ceph::io_exerciser::Sequence::SEQUENCE_END; ++s) { - std::unique_ptr<ceph::io_exerciser::IoSequence> seq = - ceph::io_exerciser::IoSequence::generate_sequence(s, - obj_size_range, - has_seqseed ? - seqseed : - rng()); - dout(0) << s << " " << seq->get_name() << dendl; - } - return 0; - } - - // Create a test for each object - std::vector<std::shared_ptr< - ceph::io_sequence::tester::TestObject>> test_objects; - - for (int obj = 0; obj < num_objects; obj++) { - test_objects.push_back( - std::make_shared<ceph::io_sequence::tester::TestObject>( - "test" + std::to_string(obj), - rados, asio, - *sbs, *spo, *sos, *snt, *ssr, - rng, lock, cond, - dryrun, verbose, - has_seqseed, seqseed - ) - ); - } - if (!dryrun) { - rados.wait_for_latest_osdmap(); - } - - run_test(test_objects, lock); - - if (!dryrun) { - guard = std::nullopt; - asio.stop(); - thread.join(); - rados.shutdown(); - } return 0; } diff --git a/src/test/osd/ceph_test_rados_io_sequence.h b/src/test/osd/ceph_test_rados_io_sequence.h index 3a84b7bc824..4f77c940274 100644 --- a/src/test/osd/ceph_test_rados_io_sequence.h +++ b/src/test/osd/ceph_test_rados_io_sequence.h @@ -1,5 +1,7 @@ #include <utility> +#include "include/random.h" + #include "global/global_init.h" #include "global/global_context.h" @@ -267,8 +269,7 @@ namespace ceph ceph::condition_variable& cond, bool dryrun, bool verbose, - bool has_seqseed, - int seqseed); + std::optional<int> seqseed); int get_num_io(); bool readyForIo(); @@ -286,8 +287,57 @@ namespace ceph bool done; ceph::util::random_number_generator<int>& rng; bool verbose; - bool has_seqseed; - int seqseed; + std::optional<int> seqseed; + }; + + class TestRunner + { + public: + TestRunner(po::variables_map& vm, librados::Rados& rados); + ~TestRunner(); + + bool run_test(); + + private: + librados::Rados& rados; + int seed; + ceph::util::random_number_generator<int> rng; + + ceph::io_sequence::tester::SelectBlockSize sbs; + ceph::io_sequence::tester::SelectObjectSize sos; + ceph::io_sequence::tester::SelectECPool spo; + ceph::io_sequence::tester::SelectNumThreads snt; + ceph::io_sequence::tester::SelectSeqRange ssr; + + boost::asio::io_context asio; + std::thread thread; + std::optional<boost::asio::executor_work_guard< + boost::asio::io_context::executor_type>> guard; + ceph::mutex lock = ceph::make_mutex("RadosIo::lock"); + ceph::condition_variable cond; + + bool input_valid; + + bool verbose; + bool dryrun; + std::optional<int> seqseed; + bool interactive; + + bool show_sequence; + bool show_help; + + int num_objects; + std::string object_name; + + std::string get_token(); + uint64_t get_numeric_token(); + + bool run_automated_test(); + + bool run_interactive_test(); + + void help(); + void list_sequence(); }; } }
\ No newline at end of file diff --git a/src/test/rgw/bucket_notification/test_bn.py b/src/test/rgw/bucket_notification/test_bn.py index 642ab6955a4..359990b3531 100644 --- a/src/test/rgw/bucket_notification/test_bn.py +++ b/src/test/rgw/bucket_notification/test_bn.py @@ -711,19 +711,16 @@ def test_ps_s3_topic_on_master(): assert_equal(status, 404) # get the remaining 2 topics - result, status = topic_conf1.get_list() - assert_equal(status, 200) - assert_equal(len(result['ListTopicsResponse']['ListTopicsResult']['Topics']['member']), 2) + list_topics(2, tenant) # delete topics - result = topic_conf2.del_config() + status = topic_conf2.del_config() assert_equal(status, 200) - result = topic_conf3.del_config() + status = topic_conf3.del_config() assert_equal(status, 200) # get topic list, make sure it is empty - result, status = topic_conf1.get_list() - assert_equal(result['ListTopicsResponse']['ListTopicsResult']['Topics'], None) + list_topics(0, tenant) @attr('basic_test') |