iSCSI targets are blocked and do not recover until system reboot

[kjavheri]

we are running targetlci v2.1.53 on CentOS Stream release 8 with kernel v4.18.0-394. we are getting below messages in /var/log/messages every 3-4 seconds.

kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
kernel: iSCSI Login negotiation failed.

After this message, if we try to restart the target service, it goes into hang state and doesn't restart for very long time. we can't even kill the process. Final workaround is to restart the target VM to make the service normal again.

we also found some call traces in the /var/log/messages as below.

 
Dec 28 05:49:07 kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
Dec 28 05:49:07 kernel: iSCSI Login negotiation failed.
Dec 28 05:49:23 kernel: iSCSI Login timeout on Network Portal 10.74.67.11:3260
Dec 28 05:49:23 kernel: tx_data returned -104, expecting 48.
Dec 28 05:49:23 kernel: iSCSI Login negotiation failed.
Dec 28 05:49:26 kernel: INFO: task kworker/0:2:110640 blocked for more than 120 seconds.
Dec 28 05:49:26 kernel:      Not tainted 4.18.0-394.el8.x86_64 #1
Dec 28 05:49:26 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 28 05:49:26 kernel: task:kworker/0:2     state:D stack:    0 pid:110640 ppid:     2 flags:0x80004080
Dec 28 05:49:26 kernel: Workqueue: events target_tmr_work [target_core_mod]
Dec 28 05:49:26 kernel: Call Trace:
Dec 28 05:49:26 kernel: __schedule+0x2d1/0x830
Dec 28 05:49:26 kernel: ? __switch_to_asm+0x35/0x70
Dec 28 05:49:26 kernel: ? __switch_to_asm+0x41/0x70
Dec 28 05:49:26 kernel: schedule+0x35/0xa0
Dec 28 05:49:26 kernel: schedule_timeout+0x274/0x300
Dec 28 05:49:26 kernel: ? __schedule+0x2d9/0x830
Dec 28 05:49:26 kernel: ? __irq_work_queue_local+0x4b/0x50
Dec 28 05:49:26 kernel: ? irq_work_queue+0x16/0x20
Dec 28 05:49:26 kernel: wait_for_completion+0x96/0x100
Dec 28 05:49:26 kernel: target_put_cmd_and_wait+0x6e/0x90 [target_core_mod]
Dec 28 05:49:26 kernel: core_tmr_abort_task.cold.7+0x197/0x212 [target_core_mod]
Dec 28 05:49:26 kernel: target_tmr_work+0x96/0x110 [target_core_mod]
Dec 28 05:49:26 kernel: process_one_work+0x1a7/0x360
Dec 28 05:49:26 kernel: worker_thread+0x30/0x390
Dec 28 05:49:26 kernel: ? create_worker+0x1a0/0x1a0
Dec 28 05:49:26 kernel: kthread+0x10a/0x120
Dec 28 05:49:26 kernel: ? set_kthread_struct+0x40/0x40
Dec 28 05:49:26 kernel: ret_from_fork+0x35/0x40

This issue is reproducible after simple test of turning off network between initiator and target VM.