diff options
| author | zhangjianwei <zhangjianwei2_yewu@cmss.chinamobile.com> | 2023-09-15 07:50:57 +0200 |
|---|---|---|
| committer | zhangjianwei2 <zhangjianwei2_yewu@cmss.chinamobile.com> | 2024-04-19 10:55:02 +0200 |
| commit | 94d175908a705171fbc6d3ae3332008e144055d3 (patch) | |
| tree | ef83721ed57e66167121c6eb159fc59c818b3e7e /src/msg/async/ProtocolV1.cc | |
| parent | Merge PR #56755 into main (diff) | |
| download | ceph-94d175908a705171fbc6d3ae3332008e144055d3.tar.xz ceph-94d175908a705171fbc6d3ae3332008e144055d3.zip | |
src/msg: fix high CPU consumption of msgr worker thread
problem analysis:
- std::multimap<clock_type::time_point, TimeEvent> time_events
- time precision is nanoseconds
- in EventCenter::process_events function
- end_time > now : Nanosecond comparison
- std::chrono::microseconds>(end_time - now) :
- but converted to microseconds difference
- so timeout_microseconds = 0
- epoll_wait(..., 0) not sleep
- rados bench count : 6000
- Proportion of 0 events processed
- 41898337 / 44796903 = 93.52%
- osd single msgr worker thread cpu high to 100%
solution:
- due to epoll_wait is milliseconds
- add ms_time_events_min_wait_interval
to control how long time_events should wait at least
- so default value aligned to 1000 microseconds
- rados bench count : 6000
- Proportion of 0 events processed
- 424466 / 4489181 = 9.45%
- osd single msgr worker thread cpu high to 30~40%
issue: https://tracker.ceph.com/issues/62512
co-author: yanghonggang <yanghonggang_yewu@cmss.chinamobile.com>
Signed-off-by: zhangjianwei <zhangjianwei2_yewu@cmss.chinamobile.com>
Diffstat (limited to 'src/msg/async/ProtocolV1.cc')
| -rw-r--r-- | src/msg/async/ProtocolV1.cc | 7 |
1 files changed, 4 insertions, 3 deletions
diff --git a/src/msg/async/ProtocolV1.cc b/src/msg/async/ProtocolV1.cc index b45ad8ca515..0ddd267926d 100644 --- a/src/msg/async/ProtocolV1.cc +++ b/src/msg/async/ProtocolV1.cc @@ -677,7 +677,7 @@ CtPtr ProtocolV1::throttle_message() { // short time, so we can wait a ms. if (connection->register_time_events.empty()) { connection->register_time_events.insert( - connection->center->create_time_event(1000, + connection->center->create_time_event(cct->_conf->ms_client_throttle_retry_time_interval, connection->wakeup_handler)); } return nullptr; @@ -710,7 +710,8 @@ CtPtr ProtocolV1::throttle_bytes() { if (connection->register_time_events.empty()) { connection->register_time_events.insert( connection->center->create_time_event( - 1000, connection->wakeup_handler)); + cct->_conf->ms_client_throttle_retry_time_interval, + connection->wakeup_handler)); } return nullptr; } @@ -737,7 +738,7 @@ CtPtr ProtocolV1::throttle_dispatch_queue() { // short time, so we can wait a ms. if (connection->register_time_events.empty()) { connection->register_time_events.insert( - connection->center->create_time_event(1000, + connection->center->create_time_event(cct->_conf->ms_client_throttle_retry_time_interval, connection->wakeup_handler)); } return nullptr; |