[Wed Aug 28 22:48:36 2019] INFO: task kworker/8:2:1251897 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/8:2 D ffff99bb2b3ad140 0 1251897 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c579c>] ? pagevec_lru_move_fn+0x10c/0x130 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0d09f7>] ? finish_task_switch+0x57/0x1c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/35:1:1251924 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/35:1 D ffff99bab32e5140 0 1251924 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc047c508>] ? mlx5e_sq_xmit+0x3e8/0xd00 [mlx5_core] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0dccbe>] ? account_entity_dequeue+0xae/0xd0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0e084c>] ? dequeue_entity+0x11c/0x5e0 [Wed Aug 28 22:48:36 2019] [<ffffffffac61a417>] ? kernel_sendmsg+0x37/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/44:2:1252306 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/44:2 D ffff99bb10ff30c0 0 1252306 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc047c508>] ? mlx5e_sq_xmit+0x3e8/0xd00 [mlx5_core] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc0444028>] ? mlx5_core_modify_cq_moderation+0x78/0xa0 [mlx5_core] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/42:0:1252315 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/42:0 D ffff998cd1b78000 0 1252315 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0dccbe>] ? account_entity_dequeue+0xae/0xd0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0e084c>] ? dequeue_entity+0x11c/0x5e0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/47:3:1252413 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/47:3 D ffff99bb308e1040 0 1252413 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c579c>] ? pagevec_lru_move_fn+0x10c/0x130 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0dccbe>] ? account_entity_dequeue+0xae/0xd0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0e084c>] ? dequeue_entity+0x11c/0x5e0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/39:0:1252462 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/39:0 D ffff99bb1d6be180 0 1252462 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0dccbe>] ? account_entity_dequeue+0xae/0xd0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0e084c>] ? dequeue_entity+0x11c/0x5e0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/8:1:1252669 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/8:1 D ffff99bb3faec100 0 1252669 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc047c508>] ? mlx5e_sq_xmit+0x3e8/0xd00 [mlx5_core] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0dccbe>] ? account_entity_dequeue+0xae/0xd0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0e084c>] ? dequeue_entity+0x11c/0x5e0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/43:2:1252751 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/43:2 D ffff999e9baca080 0 1252751 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a417>] ? kernel_sendmsg+0x37/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc4019>] ? ceph_tcp_sendmsg+0x59/0x80 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] INFO: task kworker/36:1:1252769 blocked for more than 120 seconds. [Wed Aug 28 22:48:36 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [Wed Aug 28 22:48:36 2019] kworker/36:1 D ffff998cf3e7c100 0 1252769 2 0x00000080 [Wed Aug 28 22:48:36 2019] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Wed Aug 28 22:48:36 2019] Call Trace: [Wed Aug 28 22:48:36 2019] [<ffffffffac769df9>] schedule_preempt_disabled+0x29/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac767d77>] __mutex_lock_slowpath+0xc7/0x1d0 [Wed Aug 28 22:48:36 2019] [<ffffffffac76715f>] mutex_lock+0x1f/0x2f [Wed Aug 28 22:48:36 2019] [<ffffffffc1e28000>] ceph_check_caps+0x440/0xc10 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac23e82a>] ? __check_object_size+0x1ca/0x250 [Wed Aug 28 22:48:36 2019] [<ffffffffac38c6fc>] ? memcpy_toiovec+0x5c/0xb0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e2a75f>] ceph_put_wrbuffer_cap_refs+0x18f/0x350 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac1c378d>] ? test_clear_page_writeback+0x5d/0x1b0 [Wed Aug 28 22:48:36 2019] [<ffffffffc1e1fecc>] writepages_finish+0x29c/0x3f0 [ceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dcdebe>] __complete_request+0x2e/0x70 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd6d1d>] handle_reply+0x77d/0xac0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac326b74>] ? skcipher_encrypt_blkcipher+0x54/0x70 [Wed Aug 28 22:48:36 2019] [<ffffffffac380202>] ? memzero_explicit+0x12/0x20 [Wed Aug 28 22:48:36 2019] [<ffffffffc1de3ff4>] ? ceph_crypt+0x3b4/0x560 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1de4697>] ? calc_signature+0xe7/0x110 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dd8ae3>] dispatch+0x363/0xca0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac61a56a>] ? kernel_recvmsg+0x3a/0x50 [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc6fb4>] try_read+0x514/0x12c0 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffc1dc7f64>] ceph_con_workfn+0xe4/0x1530 [libceph] [Wed Aug 28 22:48:36 2019] [<ffffffffac768a7a>] ? __schedule+0x42a/0x860 [Wed Aug 28 22:48:36 2019] [<ffffffffac0b9ebf>] process_one_work+0x17f/0x440 [Wed Aug 28 22:48:36 2019] [<ffffffffac0baf56>] worker_thread+0x126/0x3c0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0bae30>] ? manage_workers.isra.25+0x2a0/0x2a0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1da1>] kthread+0xd1/0xe0 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40 [Wed Aug 28 22:48:36 2019] [<ffffffffac775c1d>] ret_from_fork_nospec_begin+0x7/0x21 [Wed Aug 28 22:48:36 2019] [<ffffffffac0c1cd0>] ? insert_kthread_work+0x40/0x40