Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hard lockup detected when vm starts with ftrace on #22

Closed
gowrishankarm opened this issue Oct 31, 2017 · 6 comments
Closed

hard lockup detected when vm starts with ftrace on #22

gowrishankarm opened this issue Oct 31, 2017 · 6 comments

Comments

@gowrishankarm
Copy link

gowrishankarm commented Oct 31, 2017 •
edited by cdeadmin
Loading

cde:info Mirrored with LTC bug https://bugzilla.linux.ibm.com/show_bug.cgi?id=160778 </cde:info>

When I try to start a vm on host (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le) with ftrace on (using function tracer), I see watchdog kickoff for cpu hard lockup as in below trace.

[ 3230.222246] Watchdog CPU:96 detected Hard LOCKUP other CPUS:24
[ 3276.051870] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3276.051979] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=2985 
[ 3276.052082] 	(detected by 56, t=6002 jiffies, g=17311, c=17310, q=5714)
[ 3276.052209] Sending NMI from CPU 56 to CPUs 24:
[ 3286.061806] NETDEV WATCHDOG: net0 (tg3): transmit queue 0 timed out
[ 3286.061936] ------------[ cut here ]------------
[ 3286.062008] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:320 dev_watchdog+0x34c/0x360
[ 3286.062103] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3286.062979] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3286.063089] task: c00000000136e000 task.stack: c0000000013fc000
[ 3286.063166] NIP:  c0000000009a8fac LR: c0000000009a8fa8 CTR: c0000000001b1800
[ 3286.063258] REGS: c0000000013ff4b0 TRAP: 0700   Not tainted  (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3286.063380] MSR:  9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 22002822  XER: 20000000
[ 3286.063502] CFAR: c000000000176f28 SOFTE: 1 
GPR00: c0000000009a8fa8 c0000000013ff730 c000000001403900 0000000000000037 
GPR04: c0000007ff50abd0 c0000007ff521410 9000000000009033 0000000000000020 
GPR08: 0000000000000000 c000000000f6126c 00000007fe5b0000 9000000000001003 
GPR12: 0000000000002200 c00000000fd60000 0000000000000000 0000000000000000 
GPR16: 0000000000200102 0000000100048e71 c0000000013fc000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: 0000000000000000 ffffffffffffffff 0000000000000000 0000000000000000 
GPR28: 0000000000000004 c000000001433b00 c000000ff108e000 0000000000000000 
[ 3286.064360] NIP [c0000000009a8fac] dev_watchdog+0x34c/0x360
[ 3286.064427] LR [c0000000009a8fa8] dev_watchdog+0x348/0x360
[ 3286.064491] Call Trace:
[ 3286.064528] [c0000000013ff730] [c0000000009a8fa8] dev_watchdog+0x348/0x360 (unreliable)
[ 3286.064631] [c0000000013ff7d0] [c000000000198c30] call_timer_fn+0x60/0x1d0
[ 3286.064715] [c0000000013ff860] [c000000000198f20] expire_timers+0x140/0x1e0
[ 3286.064799] [c0000000013ff8d0] [c000000000199098] run_timer_softirq+0xd8/0x240
[ 3286.064897] [c0000000013ff960] [c000000000b0f6fc] __do_softirq+0x15c/0x3a4
[ 3286.064981] [c0000000013ffa50] [c0000000001064e8] irq_exit+0x118/0x130
[ 3286.065104] [c0000000013ffa70] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 3286.065241] [c0000000013ffaa0] [c000000000009208] decrementer_common+0x158/0x160
[ 3286.065402] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 3286.065613] [c0000000013ffd90] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 3286.065779] [c0000000013ffdb0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 3286.065937] [c0000000013ffe10] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 3286.066074] [c0000000013ffe30] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 3286.066207] [c0000000013ffea0] [c00000000015fe88] cpu_startup_entry+0x38/0x50
[ 3286.066367] [c0000000013ffed0] [c00000000000d918] rest_init+0xe8/0x100
[ 3286.066502] [c0000000013fff00] [c000000000e542d0] start_kernel+0x54c/0x568
[ 3286.066638] [c0000000013fff90] [c00000000000b27c] start_here_common+0x1c/0x520
[ 3286.066795] Instruction dump:
[ 3286.066880] 3d02fff3 7fc3f378 99282246 4bfc4ce1 60000000 7fc4f378 7fe6fb78 7c651b78 
[ 3286.067056] 3c62ff9e 3863d898 4b7cdf3d 60000000 <0fe00000> 4bffff84 60000000 60000000 
[ 3286.067233] ---[ end trace 7d5fcf569e0e59ec ]---
[ 3286.067344] tg3 0005:05:00.0 net0: transmit timed out, resetting
[ 3287.176015] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=56
[ 3287.176185] rcu_sched       I    0     9      2 0x00000800
[ 3287.176254] Call Trace:
[ 3287.176294] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3287.176396] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3287.176480] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3287.176564] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3287.176647] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3287.176746] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3287.176844] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3287.176929] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3287.177052] Watchdog CPU:56 Hard LOCKUP
[ 3287.177054] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3287.177133] CPU: 56 PID: 0 Comm: swapper/56 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3287.177135] task: c000000ffb967e00 task.stack: c000000ffba40000
[ 3287.177137] NIP:  c0000000000165c8 LR: c0000000000165a4 CTR: c000000000182050
[ 3287.177139] REGS: c00000003fd5fd80 TRAP: 0900   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3287.177141] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28002042  XER: 20000000
[ 3287.177153] CFAR: c000000000090558 SOFTE: 0 
GPR00: c0000000000165a4 c000001fffdcbf20 c000000001403900 0000000000000010 
GPR04: c000001fffdcbe60 0000000000000002 0000000000000000 00000000000001c0 
GPR08: 0000000000000000 b000000000009033 0000000000000000 0000000000000000 
GPR12: c0000000000903f0 c00000000fd84c00 c000000ffba43f90 0000000000000000 
GPR16: 0000000000200042 0000000100048ee0 c000000ffba40000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: c000000ffba40000 c000000ffba40000 c000000001432200 c000000ffba43b30 
GPR28: 0000000000000000 c000000000f74ee0 c000000ffba436e0 c000001fffdc8000 
[ 3287.177200] NIP [c0000000000165c8] __do_irq+0x88/0x200
[ 3287.177203] LR [c0000000000165a4] __do_irq+0x64/0x200
[ 3287.177205] Call Trace:
[ 3287.177208] [c000001fffdcbf20] [c0000000000165a4] __do_irq+0x64/0x200 (unreliable)
[ 3287.177213] [c000001fffdcbf90] [c000000000029fa4] call_do_irq+0x14/0x24
[ 3287.177217] [c000000ffba43620] [c0000000000167dc] do_IRQ+0x9c/0x110
[ 3287.177222] [c000000ffba43670] [c000000000008c58] hardware_interrupt_common+0x158/0x160
[ 3287.177228] --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
    LR = arch_local_irq_restore+0x40/0x90
[ 3287.177232] [c000000ffba43960] [c000000000141904] vtime_account_irq_enter+0x64/0x80 (unreliable)
[ 3287.177238] [c000000ffba43980] [c000000000b0f690] __do_softirq+0xf0/0x3a4
[ 3287.177242] [c000000ffba43a70] [c0000000001064e8] irq_exit+0x118/0x130
[ 3287.177247] [c000000ffba43a90] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 3287.177251] [c000000ffba43ac0] [c000000000009208] decrementer_common+0x158/0x160
[ 3287.177258] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 3287.177261] [c000000ffba43db0] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 3287.177268] [c000000ffba43dd0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 3287.177272] [c000000ffba43e30] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 3287.177276] [c000000ffba43e50] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 3287.177281] [c000000ffba43ec0] [c00000000015fe88] cpu_startup_entry+0x38/0x50
[ 3287.177285] [c000000ffba43ef0] [c000000000048a74] start_secondary+0x4e4/0x530
[ 3287.177290] [c000000ffba43f90] [c00000000000b16c] start_secondary_prolog+0x10/0x14
[ 3287.177292] Instruction dump:
[ 3287.177295] 7d2903a6 7d2c4b78 4e800421 e8410018 894d028b 7948f7e3 554a003c 994d028b 
[ 3287.177306] 40820010 e92d0020 61298000 7d210164 <2fa30000> 41de012c 48161569 60000000 
[ 3287.181946] Watchdog CPU:56 became unstuck
[ 3287.327533] tg3 0005:05:00.0 net0: 0x00000000: 0x165714e4, 0x00100546, 0x02000001, 0x00800000
[ 3287.327647] tg3 0005:05:00.0 net0: 0x00000010: 0x0000000c, 0x00002501, 0x0001000c, 0x00002501
[ 3287.327756] tg3 0005:05:00.0 net0: 0x00000020: 0x0002000c, 0x00002501, 0x00000000, 0x04201014
[ 3287.327865] tg3 0005:05:00.0 net0: 0x00000030: 0x00000000, 0x00000048, 0x00000000, 0x00000100
[ 3287.327974] tg3 0005:05:00.0 net0: 0x00000040: 0x00000000, 0x03000000, 0xc8035001, 0x64002008
[ 3287.328083] tg3 0005:05:00.0 net0: 0x00000050: 0x00005803, 0x00000000, 0x0086a005, 0x00000000
[ 3287.328191] tg3 0005:05:00.0 net0: 0x00000060: 0x00000000, 0x00000000, 0xf0010298, 0x00380080
[ 3287.328300] tg3 0005:05:00.0 net0: 0x00000070: 0x000710b0, 0xf4f6fffe, 0x00000000, 0x00000000
[ 3287.328409] tg3 0005:05:00.0 net0: 0x00000080: 0x165714e4, 0x0000001e, 0x00000000, 0x00000628
[ 3287.328517] tg3 0005:05:00.0 net0: 0x00000090: 0x00000000, 0x00000000, 0x00000000, 0x00000361
[ 3287.328626] tg3 0005:05:00.0 net0: 0x000000a0: 0x8010ac11, 0x00000004, 0x00000124, 0x00020010
[ 3287.328734] tg3 0005:05:00.0 net0: 0x000000b0: 0x10648d81, 0x0010242e, 0x00055c41, 0x10410000
[ 3287.328843] tg3 0005:05:00.0 net0: 0x000000d0: 0x0000001f, 0x00000010, 0x00000000, 0x00010001
[ 3287.328952] tg3 0005:05:00.0 net0: 0x000000f0: 0x00000000, 0x05719001, 0x00000000, 0xffffffff
[ 3287.329060] tg3 0005:05:00.0 net0: 0x00000100: 0x13c10001, 0x00000000, 0x00000000, 0x00062030
[ 3287.329168] tg3 0005:05:00.0 net0: 0x00000110: 0x00000000, 0x00002000, 0x000001e0, 0x00000000
[ 3287.329277] tg3 0005:05:00.0 net0: 0x00000130: 0x00000000, 0x00000000, 0x00000000, 0x15010003
[ 3287.329385] tg3 0005:05:00.0 net0: 0x00000140: 0x94028f64, 0x000098be, 0x00000000, 0x00000000
[ 3287.329494] tg3 0005:05:00.0 net0: 0x00000150: 0x16010004, 0x00000000, 0x0007812c, 0x00000000
[ 3287.329601] tg3 0005:05:00.0 net0: 0x00000160: 0x23010002, 0x00000000, 0x00000000, 0x00000000
[ 3287.329710] tg3 0005:05:00.0 net0: 0x00000170: 0x00000000, 0x800000ff, 0x00000000, 0x00000000
[ 3287.329818] tg3 0005:05:00.0 net0: 0x00000200: 0x00000000, 0x03000000, 0x00000000, 0xf8000000
[ 3287.329926] tg3 0005:05:00.0 net0: 0x00000210: 0x00000000, 0x7c000000, 0x00000000, 0x14000000
[ 3287.330035] tg3 0005:05:00.0 net0: 0x00000220: 0x00000000, 0xd3000000, 0x00000000, 0x00000000
[ 3287.330144] tg3 0005:05:00.0 net0: 0x00000260: 0x00000000, 0x00000000, 0x00000000, 0x00000361
[ 3287.330252] tg3 0005:05:00.0 net0: 0x00000280: 0x00000000, 0x00000628, 0x00000000, 0x0000007c
[ 3287.330361] tg3 0005:05:00.0 net0: 0x00000290: 0x00000000, 0x00000321, 0x00000000, 0x000000d4
[ 3287.330469] tg3 0005:05:00.0 net0: 0x00000300: 0x00000000, 0x0000005c, 0x00000000, 0x00000000
[ 3287.330578] tg3 0005:05:00.0 net0: 0x00000400: 0x18e04804, 0x00400000, 0x00001000, 0x00000900
[ 3287.330687] tg3 0005:05:00.0 net0: 0x00000410: 0x000098be, 0x94028f64, 0x000098be, 0x94028f64
[ 3287.330795] tg3 0005:05:00.0 net0: 0x00000420: 0x000098be, 0x94028f64, 0x000098be, 0x94028f64
[ 3287.330904] tg3 0005:05:00.0 net0: 0x00000430: 0x00000400, 0x00000000, 0x000000e7, 0x000005f2
[ 3287.331012] tg3 0005:05:00.0 net0: 0x00000440: 0x00000000, 0x00000000, 0x00000000, 0x04384400
[ 3287.331121] tg3 0005:05:00.0 net0: 0x00000450: 0x00000001, 0x00008000, 0x00000000, 0x00000112
[ 3287.331229] tg3 0005:05:00.0 net0: 0x00000460: 0x00000008, 0x00002620, 0x01ff0006, 0x00000000
[ 3287.331337] tg3 0005:05:00.0 net0: 0x00000470: 0x80000000, 0x00000000, 0x00080000, 0x40000000
[ 3287.331445] tg3 0005:05:00.0 net0: 0x00000480: 0x42000000, 0x7fffffff, 0x06000004, 0x7fffffff
[ 3287.331553] tg3 0005:05:00.0 net0: 0x00000500: 0x00000008, 0x00000002, 0x00000000, 0x00000000
[ 3287.331661] tg3 0005:05:00.0 net0: 0x00000590: 0x00e00000, 0x00000000, 0x00000000, 0x00000000
[ 3287.331770] tg3 0005:05:00.0 net0: 0x000005b0: 0x00000000, 0xc0000000, 0x00000000, 0x00000000
[ 3287.331890] tg3 0005:05:00.0 net0: 0x000005c0: 0xd384af98, 0x86411db1, 0x00000000, 0x00000000
[ 3287.331999] tg3 0005:05:00.0 net0: 0x00000600: 0xffffffff, 0x00f800d1, 0x00000000, 0x00001f04
[ 3287.332108] tg3 0005:05:00.0 net0: 0x00000610: 0xffffffff, 0x00000000, 0x07c00344, 0x00000000
[ 3287.332216] tg3 0005:05:00.0 net0: 0x00000620: 0x00000040, 0x00000000, 0x00000000, 0x00000000
[ 3287.332325] tg3 0005:05:00.0 net0: 0x00000630: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332432] tg3 0005:05:00.0 net0: 0x00000640: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332541] tg3 0005:05:00.0 net0: 0x00000650: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332650] tg3 0005:05:00.0 net0: 0x00000660: 0x01230123, 0x01230123, 0x01230123, 0x01230123
[ 3287.332758] tg3 0005:05:00.0 net0: 0x00000670: 0x88ec6860, 0xe62ee10c, 0x337b4659, 0x664e2a50
[ 3287.332867] tg3 0005:05:00.0 net0: 0x00000680: 0x83c13961, 0x3e5dd26e, 0x96e38c98, 0x9629f348
[ 3287.332975] tg3 0005:05:00.0 net0: 0x00000690: 0xab3402e3, 0xb7282180, 0x00000000, 0x00000000
[ 3287.333083] tg3 0005:05:00.0 net0: 0x000006c0: 0x00000000, 0x00000000, 0x04000000, 0x00000000
[ 3287.333192] tg3 0005:05:00.0 net0: 0x00000800: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.333299] tg3 0005:05:00.0 net0: 0x00000810: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.333408] tg3 0005:05:00.0 net0: 0x00000820: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 3287.333516] tg3 0005:05:00.0 net0: 0x00000830: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333624] tg3 0005:05:00.0 net0: 0x00000840: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333733] tg3 0005:05:00.0 net0: 0x00000850: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.333842] tg3 0005:05:00.0 net0: 0x00000860: 0xffffffff, 0xffffffff, 0xffffffff, 0x00000000
[ 3287.333950] tg3 0005:05:00.0 net0: 0x00000880: 0x00000fca, 0x0000d0f7, 0x00000000, 0x00000001
[ 3287.334059] tg3 0005:05:00.0 net0: 0x00000890: 0x00000010, 0x00000014, 0x00000000, 0x00000000
[ 3287.334167] tg3 0005:05:00.0 net0: 0x000008f0: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 3287.334276] tg3 0005:05:00.0 net0: 0x00000900: 0x00021e0e, 0xffffffff, 0x00000000, 0x00000000
[ 3287.334384] tg3 0005:05:00.0 net0: 0x00000910: 0x00000000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.334492] tg3 0005:05:00.0 net0: 0x00000920: 0x00000000, 0x00000000, 0xffffffff, 0x00000000
[ 3287.334601] tg3 0005:05:00.0 net0: 0x00000930: 0x00000000, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334710] tg3 0005:05:00.0 net0: 0x00000940: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334819] tg3 0005:05:00.0 net0: 0x00000950: 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff
[ 3287.334928] tg3 0005:05:00.0 net0: 0x00000960: 0xffffffff, 0xffffffff, 0xffffffff, 0x0000033c
[ 3287.335036] tg3 0005:05:00.0 net0: 0x00000970: 0x00000008, 0x00000005, 0x00000000, 0x00000000
[ 3287.335145] tg3 0005:05:00.0 net0: 0x00000980: 0x007a9d77, 0x0000d0f7, 0x00000000, 0x00000411
[ 3287.335253] tg3 0005:05:00.0 net0: 0x00000990: 0x0000d2bd, 0x00007623, 0x00000000, 0x00000000
[ 3287.335361] tg3 0005:05:00.0 net0: 0x00000c00: 0x0000000a, 0x00000000, 0x00000003, 0x00000001
[ 3287.335470] tg3 0005:05:00.0 net0: 0x00000c10: 0x00000000, 0x00000000, 0x00000000, 0x00280000
[ 3287.335578] tg3 0005:05:00.0 net0: 0x00000c80: 0x00000349, 0x00000000, 0x00000000, 0x00000000
[ 3287.335688] tg3 0005:05:00.0 net0: 0x00000ce0: 0xec720310, 0x08000007, 0x00000028, 0x00041028
[ 3287.335796] tg3 0005:05:00.0 net0: 0x00000cf0: 0x00000000, 0x5000005c, 0x00000000, 0x00000000
[ 3287.335904] tg3 0005:05:00.0 net0: 0x00001000: 0x00000002, 0x00000000, 0xa0004f50, 0x00000000
[ 3287.336013] tg3 0005:05:00.0 net0: 0x00001010: 0x005c05c1, 0x00004f50, 0x00000000, 0x00000000
[ 3287.336122] tg3 0005:05:00.0 net0: 0x00001400: 0x00000006, 0x00000000, 0x00000000, 0x00000000
[ 3287.336231] tg3 0005:05:00.0 net0: 0x00001440: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336340] tg3 0005:05:00.0 net0: 0x00001450: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336448] tg3 0005:05:00.0 net0: 0x00001460: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336557] tg3 0005:05:00.0 net0: 0x00001470: 0x0000005c, 0x0000005c, 0x0000005c, 0x0000005c
[ 3287.336665] tg3 0005:05:00.0 net0: 0x00001480: 0x00001111, 0x00000000, 0x00000000, 0x00000000
[ 3287.336774] tg3 0005:05:00.0 net0: 0x00001800: 0x00000016, 0x00000000, 0x0000005c, 0x00000000
[ 3287.336883] tg3 0005:05:00.0 net0: 0x00001830: 0x00000000, 0x00000000, 0x00000000, 0xbbec0000
[ 3287.336991] tg3 0005:05:00.0 net0: 0x00001840: 0xbbec0000, 0x0800000f, 0x00000201, 0xc0000000
[ 3287.337099] tg3 0005:05:00.0 net0: 0x00001850: 0x0000001f, 0x00000000, 0x000041a0, 0x005c005c
[ 3287.337208] tg3 0005:05:00.0 net0: 0x00001860: 0x02000000, 0x00000000, 0xbbec05a0, 0x0800000f
[ 3287.337317] tg3 0005:05:00.0 net0: 0x00001c00: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 3287.337427] tg3 0005:05:00.0 net0: 0x00002000: 0x00000002, 0x00000000, 0x00000000, 0x00000000
[ 3287.337535] tg3 0005:05:00.0 net0: 0x00002010: 0x00000181, 0x00000001, 0x00780003, 0x00000000
[ 3287.337644] tg3 0005:05:00.0 net0: 0x00002100: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337753] tg3 0005:05:00.0 net0: 0x00002110: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337862] tg3 0005:05:00.0 net0: 0x00002120: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.337970] tg3 0005:05:00.0 net0: 0x00002130: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338078] tg3 0005:05:00.0 net0: 0x00002140: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338187] tg3 0005:05:00.0 net0: 0x00002150: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338297] tg3 0005:05:00.0 net0: 0x00002160: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338406] tg3 0005:05:00.0 net0: 0x00002170: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338515] tg3 0005:05:00.0 net0: 0x00002180: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338623] tg3 0005:05:00.0 net0: 0x00002190: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338732] tg3 0005:05:00.0 net0: 0x000021a0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338841] tg3 0005:05:00.0 net0: 0x000021b0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.338949] tg3 0005:05:00.0 net0: 0x000021c0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339058] tg3 0005:05:00.0 net0: 0x000021d0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339168] tg3 0005:05:00.0 net0: 0x000021e0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339276] tg3 0005:05:00.0 net0: 0x000021f0: 0x0009bcde, 0x0009bcde, 0x00000000, 0x00000000
[ 3287.339385] tg3 0005:05:00.0 net0: 0x00002200: 0x00014e97, 0x00000000, 0x00000000, 0x00000000
[ 3287.339493] tg3 0005:05:00.0 net0: 0x00002400: 0x00010012, 0x00000000, 0x00400001, 0x00000000
[ 3287.339602] tg3 0005:05:00.0 net0: 0x00002410: 0x0000000f, 0x00005d00, 0x00000000, 0x00000000
[ 3287.339711] tg3 0005:05:00.0 net0: 0x00002440: 0x00000000, 0x00000000, 0x00000002, 0x00044400
[ 3287.339821] tg3 0005:05:00.0 net0: 0x00002450: 0x0800000f, 0xf33d0000, 0x08001800, 0x00040000
[ 3287.339929] tg3 0005:05:00.0 net0: 0x00002470: 0x00000000, 0x00000299, 0x00000000, 0x00000000
[ 3287.340038] tg3 0005:05:00.0 net0: 0x00002500: 0x00000000, 0x00000000, 0x00000002, 0x00044800
[ 3287.340146] tg3 0005:05:00.0 net0: 0x00002510: 0x00000000, 0x00000000, 0x00000002, 0x00040400
[ 3287.340255] tg3 0005:05:00.0 net0: 0x00002520: 0x00000000, 0x00000000, 0x00000002, 0x00044c00
[ 3287.340363] tg3 0005:05:00.0 net0: 0x00002530: 0x00000000, 0x00000000, 0x00000002, 0x00040800
[ 3287.340472] tg3 0005:05:00.0 net0: 0x00002540: 0x00000000, 0x00000000, 0x00000002, 0x00045000
[ 3287.340581] tg3 0005:05:00.0 net0: 0x00002550: 0x00000000, 0x00000000, 0x00000002, 0x00040c00
[ 3287.340688] tg3 0005:05:00.0 net0: 0x00002560: 0x00000000, 0x00000000, 0x00000002, 0x00045400
[ 3287.340797] tg3 0005:05:00.0 net0: 0x00002570: 0x00000000, 0x00000000, 0x00000002, 0x00041000
[ 3287.340905] tg3 0005:05:00.0 net0: 0x00002580: 0x00000000, 0x00000000, 0x00000002, 0x00045800
[ 3287.341013] tg3 0005:05:00.0 net0: 0x00002590: 0x00000000, 0x00000000, 0x00000002, 0x00041400
[ 3287.341122] tg3 0005:05:00.0 net0: 0x000025a0: 0x00000000, 0x00000000, 0x00000002, 0x00045c00
[ 3287.341229] tg3 0005:05:00.0 net0: 0x000025b0: 0x00000000, 0x00000000, 0x00000002, 0x00041800
[ 3287.341338] tg3 0005:05:00.0 net0: 0x000025c0: 0x00000000, 0x00000000, 0x00000002, 0x00046000
[ 3287.341446] tg3 0005:05:00.0 net0: 0x000025d0: 0x00000000, 0x00000000, 0x00000002, 0x00041c00
000000, 0x00000000
[ 3287.346116] tg3 0005:05:00.0 net0: 0x00003c20: 0x00000000, 0x00000005, 0x00000000, 0x00000000
[ 3287.346225] tg3 0005:05:00.0 net0: 0x00003c30: 0x00000000, 0x00000000, 0x0800000f, 0xe3af0000
[ 3287.346333] tg3 0005:05:00.0 net0: 0x00003c40: 0x00000000, 0x00000b00, 0x00000000, 0x00000000
[ 3287.346441] tg3 0005:05:00.0 net0: 0x00003c50: 0x00000000, 0x0000029c, 0x00000000, 0x00000000
[ 3287.346549] tg3 0005:05:00.0 net0: 0x00003c80: 0x0000062b, 0x0000007c, 0x00000321, 0x000000d4
[ 3287.346657] tg3 0005:05:00.0 net0: 0x00003cc0: 0x0000005c, 0x00000000, 0x00000000, 0x00000000
[ 3287.346766] tg3 0005:05:00.0 net0: 0x00003cd0: 0x00000000, 0x0000000f, 0x00000000, 0x00000000
[ 3287.346874] tg3 0005:05:00.0 net0: 0x00003d00: 0x0800000f, 0xc2120000, 0x0800000f, 0xf23e0000
[ 3287.346982] tg3 0005:05:00.0 net0: 0x00003d10: 0x0800000f, 0xe6a70000, 0x0800000f, 0xe54b0000
[ 3287.347090] tg3 0005:05:00.0 net0: 0x00003d80: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 3287.347199] tg3 0005:05:00.0 net0: 0x00003d90: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 3287.347307] tg3 0005:05:00.0 net0: 0x00003da0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 3287.347415] tg3 0005:05:00.0 net0: 0x00003db0: 0x00000014, 0x00000000, 0x00000005, 0x00000000
[ 3287.347523] tg3 0005:05:00.0 net0: 0x00003dc0: 0x00000005, 0x00000000, 0x00000014, 0x00000000
[ 3287.347632] tg3 0005:05:00.0 net0: 0x00003dd0: 0x00000005, 0x00000000, 0x00000005, 0x00000000
[ 3287.347740] tg3 0005:05:00.0 net0: 0x00003fc0: 0x0000786f, 0x00000000, 0x00000000, 0x00000000
[ 3287.347848] tg3 0005:05:00.0 net0: 0x00004000: 0x00000002, 0x00000000, 0x001692bd, 0x000d625e
[ 3287.347956] tg3 0005:05:00.0 net0: 0x00004010: 0x00000000, 0x002e7012, 0x00000480, 0x00847042
[ 3287.348064] tg3 0005:05:00.0 net0: 0x00004020: 0x00000000, 0x00000000, 0x00000010, 0x00000000
[ 3287.348173] tg3 0005:05:00.0 net0: 0x00004030: 0x00000010, 0x00000050, 0x00000000, 0x00000000
[ 3287.348281] tg3 0005:05:00.0 net0: 0x00004040: 0x00000000, 0x00000000, 0x01083620, 0x00000000
[ 3287.348388] tg3 0005:05:00.0 net0: 0x00004050: 0x00000000, 0x00000000, 0x002e7010, 0x0048f002
[ 3287.348497] tg3 0005:05:00.0 net0: 0x00004060: 0x00400000, 0x00000000, 0x00000000, 0x00000000
[ 3287.348605] tg3 0005:05:00.0 net0: 0x00004400: 0x80000006, 0x00000000, 0x00010000, 0x0000a000
[ 3287.348713] tg3 0005:05:00.0 net0: 0x00004410: 0x00000000, 0x0000002a, 0x000000a0, 0x00000000
[ 3287.348822] tg3 0005:05:00.0 net0: 0x00004420: 0x0000003d, 0x00000000, 0x00000000, 0x00000000
[ 3287.348929] tg3 0005:05:00.0 net0: 0x00004440: 0x00000000, 0x00000000, 0x00000000, 0x04f14052
[ 3287.349039] tg3 0005:05:00.0 net0: 0x00004450: 0x0002033f, 0x00e800e9, 0x00000000, 0x00000000
[ 3287.349147] tg3 0005:05:00.0 net0: 0x00004800: 0x180303fe, 0x00000000, 0x00000000, 0x00000100
[ 3287.349254] tg3 0005:05:00.0 net0: 0x00004810: 0x00000000, 0x00000008, 0x05929c80, 0x00000000
[ 3287.349363] tg3 0005:05:00.0 net0: 0x00004820: 0x0000009a, 0x00000000, 0x00080000, 0x00000000
[ 3287.349472] tg3 0005:05:00.0 net0: 0x00004840: 0x00000000, 0x00000000, 0x000e2200, 0x0042f446
[ 3287.349580] tg3 0005:05:00.0 net0: 0x00004850: 0xfe1f915e, 0x804f054a, 0x8f8e8f8e, 0x00000000
[ 3287.349689] tg3 0005:05:00.0 net0: 0x00004860: 0x0000009a, 0x113e0007, 0x00000800, 0x6a000000
[ 3287.349797] tg3 0005:05:00.0 net0: 0x00004870: 0x00000080, 0x00000000, 0x00000000, 0x00000000
[ 3287.349907] tg3 0005:05:00.0 net0: 0x00004900: 0x00090404, 0x00305407, 0x00000000, 0x00000000
[ 3287.350015] tg3 0005:05:00.0 net0: 0x00004910: 0x000f001c, 0x00000000, 0x00000000, 0x00000000
[ 3287.350123] tg3 0005:05:00.0 net0: 0x00004a00: 0x180303fe, 0x00200000, 0x00200020, 0x7a9d0000
[ 3287.350232] tg3 0005:05:00.0 net0: 0x00004a10: 0xf33d6320, 0x008c0904, 0x00200010, 0x00000000
[ 3287.350340] tg3 0005:05:00.0 net0: 0x00004a20: 0x00000000, 0x00000000, 0xf02c0000, 0xf33d6380
[ 3287.350449] tg3 0005:05:00.0 net0: 0x00004a30: 0x00000000, 0x00000128, 0x00000128, 0x00000000
[ 3287.350557] tg3 0005:05:00.0 net0: 0x00004a40: 0xf33d62e0, 0xf33d6300, 0xf33d6320, 0xf33d62c0
[ 3287.350666] tg3 0005:05:00.0 net0: 0x00004a50: 0x00200020, 0x00200020, 0x00200020, 0x00200020
[ 3287.350776] tg3 0005:05:00.0 net0: 0x00004a70: 0x00090404, 0x00305407, 0x000f001c, 0x00000000
[ 3287.350884] tg3 0005:05:00.0 net0: 0x00004b00: 0x180303fe, 0x00420003, 0x30000000, 0x00280120
[ 3287.350992] tg3 0005:05:00.0 net0: 0x00004b10: 0x00420040, 0x00280002, 0x0042dc90, 0x00000000
[ 3287.351101] tg3 0005:05:00.0 net0: 0x00004b20: 0x00000051, 0x02278000, 0xac3904e9, 0x0f420048
[ 3287.351210] tg3 0005:05:00.0 net0: 0x00004b30: 0xec7202e9, 0x0f280028, 0xac3924e9, 0x0f420048
[ 3287.351319] tg3 0005:05:00.0 net0: 0x00004b40: 0xec720311, 0x0f280028, 0xcecebebe, 0xfafa9595
[ 3287.351427] tg3 0005:05:00.0 net0: 0x00004b50: 0xf03a0000, 0xac390530, 0xef050000, 0x514f0050
[ 3287.351535] tg3 0005:05:00.0 net0: 0x00004b60: 0xf0390000, 0xec720310, 0x8f030000, 0x4000005c
[ 3287.351644] tg3 0005:05:00.0 net0: 0x00004b70: 0xf03a0000, 0xac392530, 0xef050000, 0x000000ff
[ 3287.351753] tg3 0005:05:00.0 net0: 0x00004b80: 0x00000051, 0x113e0007, 0x00000800, 0x6a000000
[ 3287.351872] tg3 0005:05:00.0 net0: 0x00004b90: 0x00000080, 0x00090404, 0x00305407, 0x000f001c
[ 3287.351980] tg3 0005:05:00.0 net0: 0x00004ba0: 0x00f00024, 0x00000000, 0x00000000, 0x00000000
[ 3287.352089] tg3 0005:05:00.0 net0: 0x00004bb0: 0xf12cd0e9, 0xac3908e9, 0xec720169, 0xf12cd8e9
[ 3287.352199] tg3 0005:05:00.0 net0: 0x00004bc0: 0xac3904ee, 0xec7202e8, 0xac3924ee, 0xec720310
[ 3287.352307] tg3 0005:05:00.0 net0: 0x00004bd0: 0xf12cd0ee, 0xac3908ee, 0xec720168, 0xf12cd8ee
[ 3287.352416] tg3 0005:05:00.0 net0: 0x00004be0: 0x00280042, 0x00280042, 0x0042006a, 0x006a0180
[ 3287.352524] tg3 0005:05:00.0 net0: 0x00004bf0: 0xf0390000, 0xec720338, 0x8f030000, 0x00005bad
[ 3287.352633] tg3 0005:05:00.0 net0: 0x00004c00: 0x200003fe, 0x00000000, 0x00000000, 0x00000000
[ 3287.352742] tg3 0005:05:00.0 net0: 0x00004c10: 0x0000003f, 0x00000000, 0x00000006, 0x00000000
[ 3287.352850] tg3 0005:05:00.0 net0: 0x00004c20: 0x00000000, 0x00000000, 0x00000000, 0x00000006
[ 3287.352958] tg3 0005:05:00.0 net0: 0x00004c30: 0x00000000, 0x001f0000, 0x00000089, 0x00000089
[ 3287.353068] tg3 0005:05:00.0 net0: 0x00004c40: 0x00000000, 0xbb60c540, 0x001d0020, 0x00140020
[ 3287.353177] tg3 0005:05:00.0 net0: 0x00004c50: 0x2007b62a, 0x0062a0d3, 0xd432107c, 0x001c1111
[ 3287.353286] tg3 0005:05:00.0 net0: 0x00004c60: 0x00000020, 0x00000000, 0x00000000, 0x00000000
[ 3287.353394] tg3 0005:05:00.0 net0: 0x00005000: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.353503] tg3 0005:05:00.0 net0: 0x00005010: 0x00000000, 0x00000000, 0x00000000, 0x08001d54
[ 3287.353612] tg3 0005:05:00.0 net0: 0x00005020: 0x30632000, 0x00000000, 0x00000000, 0x40000020
[ 3287.353720] tg3 0005:05:00.0 net0: 0x00005030: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.353828] tg3 0005:05:00.0 net0: 0x00005040: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.353938] tg3 0005:05:00.0 net0: 0x00005080: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.354046] tg3 0005:05:00.0 net0: 0x00005090: 0x00000000, 0x00000000, 0x00000000, 0x08001800
[ 3287.354155] tg3 0005:05:00.0 net0: 0x000050a0: 0x00641824, 0x00000000, 0x00000000, 0x40000020
[ 3287.354263] tg3 0005:05:00.0 net0: 0x000050b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.354372] tg3 0005:05:00.0 net0: 0x000050c0: 0x00000000, 0x00000000, 0x080024a0, 0x00000000
[ 3287.354481] tg3 0005:05:00.0 net0: 0x00005100: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 3287.354590] tg3 0005:05:00.0 net0: 0x00005110: 0x00000000, 0x00000000, 0x00000000, 0x08002988
[ 3287.354699] tg3 0005:05:00.0 net0: 0x00005120: 0x0a0005f9, 0x00000000, 0x00000000, 0x40000020
[ 3287.354808] tg3 0005:05:00.0 net0: 0x00005130: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.354916] tg3 0005:05:00.0 net0: 0x00005140: 0x00000000, 0x00000000, 0x08001d66, 0x00000000
[ 3287.355025] tg3 0005:05:00.0 net0: 0x00005180: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.355134] tg3 0005:05:00.0 net0: 0x00005190: 0x00000000, 0x00000000, 0x00000000, 0x08001850
[ 3287.355243] tg3 0005:05:00.0 net0: 0x000051a0: 0x30632000, 0x00000000, 0x00000000, 0x40000020
[ 3287.355352] tg3 0005:05:00.0 net0: 0x000051b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.355460] tg3 0005:05:00.0 net0: 0x000051c0: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.355569] tg3 0005:05:00.0 net0: 0x00005200: 0x00000000, 0x14600011, 0x00000000, 0xb49a89ab
[ 3287.355678] tg3 0005:05:00.0 net0: 0x00005210: 0x1460019f, 0x08006f98, 0xc0000000, 0x00000005
[ 3287.355786] tg3 0005:05:00.0 net0: 0x00005220: 0x00000000, 0xffb7fbfb, 0x00000000, 0x08006f80
[ 3287.355895] tg3 0005:05:00.0 net0: 0x00005230: 0x00000005, 0x00000000, 0xffb7fbfb, 0x00000000
[ 3287.356004] tg3 0005:05:00.0 net0: 0x00005240: 0x08006f80, 0x00000005, 0x00000000, 0xffb7fbfb
[ 3287.356113] tg3 0005:05:00.0 net0: 0x00005250: 0x00000000, 0x08006f80, 0x00000005, 0x00000000
[ 3287.356222] tg3 0005:05:00.0 net0: 0x00005260: 0xffb7fbfb, 0x00000000, 0x08006f80, 0x00000005
[ 3287.356331] tg3 0005:05:00.0 net0: 0x00005270: 0x00000000, 0xffb7fbfb, 0x00000000, 0x08006f80
[ 3287.356440] tg3 0005:05:00.0 net0: 0x00005280: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.356548] tg3 0005:05:00.0 net0: 0x00005290: 0x00000000, 0x00000000, 0x00000000, 0x08001850
[ 3287.356657] tg3 0005:05:00.0 net0: 0x000052a0: 0x1464002b, 0x00000000, 0x00000000, 0x40000020
[ 3287.356766] tg3 0005:05:00.0 net0: 0x000052b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.356874] tg3 0005:05:00.0 net0: 0x000052c0: 0x00000000, 0x00000000, 0x0800180a, 0x00000000
[ 3287.356982] tg3 0005:05:00.0 net0: 0x00005300: 0x00009800, 0x80000000, 0x00000000, 0x00000000
[ 3287.357090] tg3 0005:05:00.0 net0: 0x00005310: 0x00000000, 0x00000000, 0x00000000, 0x080017e8
[ 3287.357199] tg3 0005:05:00.0 net0: 0x00005320: 0x1485000d, 0x00000000, 0x00000000, 0x40000020
[ 3287.357307] tg3 0005:05:00.0 net0: 0x00005330: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.357415] tg3 0005:05:00.0 net0: 0x00005340: 0x00000000, 0x00000000, 0x080029c0, 0x00000000
[ 3287.357524] tg3 0005:05:00.0 net0: 0x00005380: 0x00009800, 0x80004000, 0x00000000, 0x00000000
[ 3287.357633] tg3 0005:05:00.0 net0: 0x00005390: 0x00000000, 0x00000000, 0x00000000, 0x08001e10
[ 3287.357741] tg3 0005:05:00.0 net0: 0x000053a0: 0xafbf0014, 0x00000000, 0x00000000, 0x40000020
[ 3287.357850] tg3 0005:05:00.0 net0: 0x000053b0: 0x00000000, 0x0000001d, 0x00000000, 0x00000000
[ 3287.357959] tg3 0005:05:00.0 net0: 0x000053c0: 0x00000000, 0x00000000, 0x08001d66, 0x00000000
[ 3287.358068] tg3 0005:05:00.0 net0: 0x00005800: 0x03000000, 0x03000000, 0xfb000000, 0x00000000
[ 3287.358177] tg3 0005:05:00.0 net0: 0x00005810: 0x7c000000, 0x00000000, 0x14000000, 0x00000000
[ 3287.358285] tg3 0005:05:00.0 net0: 0x00005820: 0xd3000000, 0x00000000, 0x00000000, 0x00000000
[ 3287.358394] tg3 0005:05:00.0 net0: 0x00005860: 0x00000000, 0x00000000, 0x00000364, 0x00000364
[ 3287.358502] tg3 0005:05:00.0 net0: 0x00005880: 0x0000062b, 0x0000062b, 0x0000007c, 0x0000007c
[ 3287.358611] tg3 0005:05:00.0 net0: 0x00005890: 0x00000321, 0x00000321, 0x000000d4, 0x000000d4
[ 3287.358720] tg3 0005:05:00.0 net0: 0x00005900: 0x0000005c, 0x0000005c, 0x00000000, 0x00000000
[ 3287.358829] tg3 0005:05:00.0 net0: 0x00005980: 0x0000005c, 0x00000000, 0x00000000, 0x00000000
[ 3287.358937] tg3 0005:05:00.0 net0: 0x00005a00: 0x000f601f, 0x00000000, 0x00010000, 0x00000000
[ 3287.359046] tg3 0005:05:00.0 net0: 0x00006000: 0x00010082, 0x00000000, 0x00000000, 0x00000000
[ 3287.359154] tg3 0005:05:00.0 net0: 0x00006400: 0x00000000, 0x00000000, 0x00010991, 0xc0000000
[ 3287.359264] tg3 0005:05:00.0 net0: 0x00006410: 0x0a000064, 0x0a000064, 0x00000000, 0x00000000
[ 3287.359372] tg3 0005:05:00.0 net0: 0x00006430: 0x00000000, 0x14e41657, 0x04201014, 0x01020000
[ 3287.359480] tg3 0005:05:00.0 net0: 0x00006440: 0x0000304f, 0x000002e4, 0x00000000, 0x00000000
[ 3287.359589] tg3 0005:05:00.0 net0: 0x000064c0: 0x00000010, 0x00000004, 0x00000124, 0x00000000
[ 3287.359698] tg3 0005:05:00.0 net0: 0x000064d0: 0x00000000, 0x10008d81, 0x00000000, 0x00315e41
[ 3287.359806] tg3 0005:05:00.0 net0: 0x000064e0: 0x00000031, 0x0000001f, 0x00000000, 0x00000000
[ 3287.359914] tg3 0005:05:00.0 net0: 0x000064f0: 0x00000002, 0x00000031, 0x00000000, 0x00000000
[ 3287.360023] tg3 0005:05:00.0 net0: 0x00006500: 0x01e10003, 0x94028f64, 0x000098be, 0x00000003
[ 3287.360132] tg3 0005:05:00.0 net0: 0x00006510: 0x0007812c, 0x00058116, 0x0004610a, 0x00000000
[ 3287.360239] tg3 0005:05:00.0 net0: 0x00006530: 0x00000001, 0x00000000, 0x00000000, 0x00000000
[ 3287.360347] tg3 0005:05:00.0 net0: 0x00006550: 0x00000000, 0x02800000, 0x00000000, 0x00000000
[ 3287.360456] tg3 0005:05:00.0 net0: 0x000065f0: 0x00000000, 0x00000109, 0x00000000, 0x00000000
[ 3287.360565] tg3 0005:05:00.0 net0: 0x00006800: 0x041b0034, 0x20089082, 0x01060408, 0xc0ac6cfe
[ 3287.360674] tg3 0005:05:00.0 net0: 0x00006810: 0x01020000, 0xffffffff, 0x00000000, 0x00000000
[ 3287.360782] tg3 0005:05:00.0 net0: 0x00006830: 0xffffffff, 0xffffffff, 0x00000000, 0x00000000
[ 3287.360890] tg3 0005:05:00.0 net0: 0x00006840: 0x00000000, 0x00000001, 0x00000000, 0x00000000
[ 3287.360999] tg3 0005:05:00.0 net0: 0x00006890: 0x00000000, 0x88003800, 0x00000000, 0x04102040
[ 3287.361108] tg3 0005:05:00.0 net0: 0x000068a0: 0x00000020, 0x00000001, 0x03ff03ff, 0x00000000
[ 3287.361216] tg3 0005:05:00.0 net0: 0x000068b0: 0xe0011514, 0x00000000, 0x00000000, 0x00000000
[ 3287.361325] tg3 0005:05:00.0 net0: 0x000068e0: 0x00000000, 0x00000000, 0x00000000, 0x0000e204
[ 3287.361433] tg3 0005:05:00.0 net0: 0x000068f0: 0x00ff000e, 0x00ff0000, 0x00000000, 0x04444444
[ 3287.361542] tg3 0005:05:00.0 net0: 0x00006900: 0xb3c3f9a0, 0x14f28bed, 0x00000000, 0x00000000
[ 3287.361650] tg3 0005:05:00.0 net0: 0x00006920: 0x00000000, 0x00000000, 0x00000001, 0x00000000
[ 3287.361759] tg3 0005:05:00.0 net0: 0x00007000: 0x00000188, 0x00000000, 0x00000000, 0x0000022c
[ 3287.361879] tg3 0005:05:00.0 net0: 0x00007010: 0x04200420, 0x010080f3, 0x00d70081, 0x03008200
[ 3287.361988] tg3 0005:05:00.0 net0: 0x00007020: 0x00000000, 0x00000000, 0x00000406, 0x10004000
[ 3287.362097] tg3 0005:05:00.0 net0: 0x00007030: 0x00020000, 0x00000230, 0x001f0000, 0x00000000
[ 3287.362208] tg3 0005:05:00.0 net0: 0: Host status block [00000001:00000011:(0000:0241:0000):(0000:005c)]
[ 3287.362319] tg3 0005:05:00.0 net0: 0: NAPI info [00000011:00000011:(005c:005c:01ff):0000:(036e:0000:0000:0000)]
[ 3287.362444] tg3 0005:05:00.0 net0: 1: Host status block [00000001:00000004:(0000:0000:0000):(0634:0000)]
[ 3287.362555] tg3 0005:05:00.0 net0: 1: NAPI info [00000004:00000004:(0000:0000:01ff):0634:(0634:0634:0000:0000)]
[ 3287.362679] tg3 0005:05:00.0 net0: 2: Host status block [00000001:0000007c:(007c:0000:0000):(0000:0000)]
[ 3287.362790] tg3 0005:05:00.0 net0: 2: NAPI info [0000007c:0000007c:(0000:0000:01ff):007c:(007c:007c:0000:0000)]
[ 3287.362914] tg3 0005:05:00.0 net0: 3: Host status block [00000001:00000014:(0000:0000:0000):(0000:0000)]
[ 3287.363025] tg3 0005:05:00.0 net0: 3: NAPI info [00000014:00000014:(0000:0000:01ff):0321:(0321:0321:0000:0000)]
[ 3287.363148] tg3 0005:05:00.0 net0: 4: Host status block [00000001:000000d4:(0000:0000:00d5):(0000:0000)]
[ 3287.363260] tg3 0005:05:00.0 net0: 4: NAPI info [000000d4:000000d4:(0000:0000:01ff):00d5:(00d5:00d5:0000:0000)]
[ 3287.392917] tg3 0005:05:00.0 net0: Link is down
[ 3289.256766] tg3 0005:05:00.0 net0: Link is up at 100 Mbps, full duplex
[ 3289.256859] tg3 0005:05:00.0 net0: Flow control is on for TX and on for RX
[ 3289.256941] tg3 0005:05:00.0 net0: EEE is disabled
[ 3314.141595] Watchdog CPU:64 detected Hard LOCKUP other CPUS:0
[ 3314.141692] Watchdog CPU:0 Hard LOCKUP
[ 3314.141744] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 3314.142565] CPU: 0 PID: 9200 Comm: kworker/0:0 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 3314.142689] Workqueue: events wait_rcu_exp_gp
[ 3314.142753] task: c0000007f008b000 task.stack: c0000007f0160000
[ 3314.142829] NIP:  c0000000001b8028 LR: c000000000189f14 CTR: c0000000001b7f60
[ 3314.142919] REGS: c0000007f01638d0 TRAP: 0501   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 3314.143039] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 44002022  XER: 20000000
[ 3314.143156] CFAR: c0000000001b8030 SOFTE: 1 
GPR00: c000000000189420 c0000007f0163b50 c000000001403900 0000000000000060 
GPR04: c0000007ff526700 c00000000138ac80 0000000000000000 0000000000000000 
GPR08: 0000000000000001 0000000000000001 0000000000000001 c00000000138ac80 
GPR12: c000000000090180 c00000000fd60000 
[ 3314.143563] NIP [c0000000001b8028] smp_call_function_single+0xd8/0x1a0
[ 3314.143642] LR [c000000000189f14] sync_rcu_exp_select_cpus+0x3c4/0x550
[ 3314.143718] Call Trace:
[ 3314.143753] [c0000007f0163b50] [c0000000001b7f60] smp_call_function_single+0x10/0x1a0 (unreliable)
[ 3314.143866] [c0000007f0163bc0] [c000000000189f14] sync_rcu_exp_select_cpus+0x3c4/0x550
[ 3314.143962] [c0000007f0163c50] [c00000000018a7b8] wait_rcu_exp_gp+0x38/0x60
[ 3314.144044] [c0000007f0163c80] [c000000000122d60] process_one_work+0x1a0/0x490
[ 3314.144138] [c0000007f0163d20] [c0000000001230e8] worker_thread+0x98/0x520
[ 3314.144220] [c0000007f0163dc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3314.144301] [c0000007f0163e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3314.144394] Instruction dump:
[ 3314.144443] 7c895214 81240018 792a07e1 41820030 48000018 60000000 60000000 60000000 
[ 3314.144555] 60000000 60420000 7c210b78 7c421378 <81240018> 792807e1 4082fff0 7c2004ac 
[ 3379.041240] sd 0:2:0:0: [sda] tag#0 Resetting device
[ 3409.101139] ipr 0001:08:00.0: Timed out waiting for aborted commands
[ 3409.101252] sd 0:2:1:0: [sdb] tag#1 Resetting device
[ 3439.180914] ipr 0001:08:00.0: Timed out waiting for aborted commands
[ 3439.181028] ipr 0001:08:00.0: Adapter being reset as a result of error recovery.
[ 3456.100538] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3456.100665] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=10988 
[ 3456.100763] 	(detected by 32, t=24007 jiffies, g=17311, c=17310, q=15846)
[ 3456.100871] Sending NMI from CPU 32 to CPUs 24:
[ 3467.225107] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=96
[ 3467.225212] rcu_sched       I    0     9      2 0x00000800
[ 3467.225281] Call Trace:
[ 3467.225321] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3467.225422] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3467.225507] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3467.225590] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3467.225673] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3467.225771] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3467.225867] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3467.225951] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3636.149208] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3636.149318] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=19079 
[ 3636.149418] 	(detected by 64, t=42012 jiffies, g=17311, c=17310, q=24830)
[ 3636.149526] Sending NMI from CPU 64 to CPUs 24:
[ 3647.273942] rcu_sched kthread starved for 1109 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 3647.274056] rcu_sched       I    0     9      2 0x00000800
[ 3647.274126] Call Trace:
[ 3647.274168] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3647.274271] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3647.274356] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3647.274440] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3647.274524] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3647.274623] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3647.274720] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3647.274805] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3647.274941] Watchdog CPU:64 detected Hard LOCKUP other CPUS:128,152
[ 3816.197876] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3816.197983] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=27054 
[ 3816.198083] 	(detected by 88, t=60017 jiffies, g=17311, c=17310, q=33768)
[ 3816.198200] Sending NMI from CPU 88 to CPUs 24:
[ 3827.322627] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 3827.322732] rcu_sched       I    0     9      2 0x00000800
[ 3827.322801] Call Trace:
[ 3827.322843] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 3827.322945] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 3827.323032] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 3827.323116] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 3827.323199] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 3827.323298] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 3827.323396] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 3827.323480] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 3996.246543] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 3996.246639] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=34980 
[ 3996.246740] 	(detected by 88, t=78022 jiffies, g=17311, c=17310, q=42662)
[ 3996.246832] Sending NMI from CPU 88 to CPUs 24:
[ 4007.256506] Watchdog CPU:128 detected Hard LOCKUP other CPUS:88
[ 4007.371456] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=88
[ 4007.371560] rcu_sched       I    0     9      2 0x00000800
[ 4007.371629] Call Trace:
[ 4007.371667] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4007.371768] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4007.371852] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4007.371937] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4007.372020] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4007.372119] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4007.372216] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4007.372301] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4008.387138] Watchdog CPU:88 became unstuck
[ 4176.295213] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4176.295317] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=42870 
[ 4176.295417] 	(detected by 64, t=96027 jiffies, g=17311, c=17310, q=51572)
[ 4176.295531] Sending NMI from CPU 64 to CPUs 24:
[ 4187.419949] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 4187.420053] rcu_sched       I    0     9      2 0x00000800
[ 4187.420122] Call Trace:
[ 4187.420164] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4187.420267] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4187.420351] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4187.420435] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4187.420519] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4187.420619] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4187.420716] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4187.420800] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4356.343983] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4356.344090] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=50769 
[ 4356.344191] 	(detected by 40, t=114032 jiffies, g=17311, c=17310, q=60472)
[ 4356.344299] Sending NMI from CPU 40 to CPUs 24:
[ 4367.468407] rcu_sched kthread starved for 1109 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=88
[ 4367.468521] rcu_sched       I    0     9      2 0x00000800
[ 4367.468590] Call Trace:
[ 4367.468630] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4367.468733] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4367.468817] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4367.468900] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4367.468984] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4367.469083] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4367.469179] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4367.469263] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4536.392782] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4536.392881] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=58758 
[ 4536.392981] 	(detected by 48, t=132037 jiffies, g=17311, c=17310, q=69558)
[ 4536.393101] Sending NMI from CPU 48 to CPUs 24:
[ 4547.517491] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 4547.517620] rcu_sched       I    0     9      2 0x00000800
[ 4547.517689] Call Trace:
[ 4547.517731] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 4547.517833] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 4547.517919] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 4547.518003] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 4547.518086] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 4547.518185] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 4547.518281] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 4547.518365] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 4547.518490] Watchdog CPU:48 Hard LOCKUP
[ 4547.518491] Modules linked in: vhost_net vhost tap xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables ses enclosure scsi_transport_sas i2c_opal ipmi_powernv i2c_core ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc kvm_hv kvm_pr kvm scsi_dh_alua dm_service_time dm_multipath tg3 ptp pps_core
[ 4547.518583] CPU: 48 PID: 0 Comm: swapper/48 Tainted: G        W       4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le #1
[ 4547.518586] task: c000000ffb972e00 task.stack: c000000ffba20000
[ 4547.518588] NIP:  c0000000000165c8 LR: c0000000000165a4 CTR: c000000000182050
[ 4547.518590] REGS: c00000003fdbfd80 TRAP: 0900   Tainted: G        W        (4.14.0-1.rc4.dev.gitb27fc5c.el7.centos.ppc64le)
[ 4547.518591] MSR:  9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE>  CR: 28002044  XER: 20000000
[ 4547.518604] CFAR: c00000000009052c SOFTE: 0 
GPR00: c0000000000165a4 c000001fffe0bf20 c000000001403900 00000000000001a7 
GPR04: c000001fffe0be60 000000000000301a 0000000000000000 0000000000000180 
GPR08: 0000000000000000 b000000000009033 0000000000000000 0000000000000000 
GPR12: c0000000000903f0 c00000000fd7f800 c000000ffba23f90 0000000000000000 
GPR16: 0000000000200042 0000000100067b33 c000000ffba20000 0000000000000000 
GPR20: c000000000f74f00 c000000001433b00 c000000000f74f00 000000000000000a 
GPR24: c000000ffba20000 c000000ffba20000 c000000001432200 c000000ffba23b30 
GPR28: 0000000000000000 c000000000f74ee0 c000000ffba236e0 c000001fffe08000 
[ 4547.518654] NIP [c0000000000165c8] __do_irq+0x88/0x200
[ 4547.518657] LR [c0000000000165a4] __do_irq+0x64/0x200
[ 4547.518658] Call Trace:
[ 4547.518661] [c000001fffe0bf20] [c0000000000165a4] __do_irq+0x64/0x200 (unreliable)
[ 4547.518667] [c000001fffe0bf90] [c000000000029fa4] call_do_irq+0x14/0x24
[ 4547.518671] [c000000ffba23620] [c0000000000167dc] do_IRQ+0x9c/0x110
[ 4547.518676] [c000000ffba23670] [c000000000008c58] hardware_interrupt_common+0x158/0x160
[ 4547.518683] --- interrupt: 501 at arch_local_irq_restore+0x5c/0x90
    LR = arch_local_irq_restore+0x40/0x90
[ 4547.518688] [c000000ffba23960] [c000000000141904] vtime_account_irq_enter+0x64/0x80 (unreliable)
[ 4547.518694] [c000000ffba23980] [c000000000b0f690] __do_softirq+0xf0/0x3a4
[ 4547.518698] [c000000ffba23a70] [c0000000001064e8] irq_exit+0x118/0x130
[ 4547.518703] [c000000ffba23a90] [c00000000002469c] timer_interrupt+0xac/0xe0
[ 4547.518708] [c000000ffba23ac0] [c000000000009208] decrementer_common+0x158/0x160
[ 4547.518714] --- interrupt: 901 at replay_interrupt_return+0x0/0x4
    LR = arch_local_irq_restore+0x74/0x90
[ 4547.518717] [c000000ffba23db0] [c00000000143c0cc] cpu_idle_force_poll+0x0/0x4 (unreliable)
[ 4547.518726] [c000000ffba23dd0] [c0000000008e83f0] cpuidle_enter_state+0x110/0x3d0
[ 4547.518730] [c000000ffba23e30] [c00000000015f73c] call_cpuidle+0x4c/0x80
[ 4547.518735] [c000000ffba23e50] [c00000000015fbe0] do_idle+0x2b0/0x350
[ 4547.518739] [c000000ffba23ec0] [c00000000015fe8c] cpu_startup_entry+0x3c/0x50
[ 4547.518743] [c000000ffba23ef0] [c000000000048a74] start_secondary+0x4e4/0x530
[ 4547.518748] [c000000ffba23f90] [c00000000000b16c] start_secondary_prolog+0x10/0x14
[ 4547.518750] Instruction dump:
[ 4547.518754] 7d2903a6 7d2c4b78 4e800421 e8410018 894d028b 7948f7e3 554a003c 994d028b 
[ 4547.518765] 40820010 e92d0020 61298000 7d210164 <2fa30000> 41de012c 48161569 60000000 
[ 4547.522790] Watchdog CPU:48 became unstuck
[ 4716.441578] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 4716.441680] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=66701 
[ 4716.441781] 	(detected by 96, t=150042 jiffies, g=17311, c=17310, q=78447)
[ 4716.441894] Sending NMI from CPU 96 to CPUs 24:
[ 4727.566112] rcu_sched kthread starved for 1113 jiffies! g17311 c17310 f0x0 RCU_GP_WAIT4fc/0xa60
[ 5267.713414] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5267.713498] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 5268.788484] Watchdog CPU:104 became unstuck
[ 5436.636769] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 5436.636852] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=98515 
[ 5436.636953] 	(detected by 112, t=222062 jiffies, g=17311, c=17310, q=115020)
[ 5436.637067] Sending NMI from CPU 112 to CPUs 24:
[ 5447.761232] rcu_sched kthread starved for 1111 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=104
[ 5447.761360] rcu_sched       I    0     9      2 0x00000800
[ 5447.761429] Call Trace:
[ 5447.761469] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 5447.761571] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 5447.761656] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 5447.761740] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 5447.761824] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 5447.761923] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 5447.762019] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5447.762104] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 5616.685701] 	24-...: (1 GPs behind) idle=c92/140000000000000/0 softirq=3973/3974 fqs=106527 
[ 5616.685812] 	(detected by 8, t=240067 jiffies, g=17311, c=17310, q=123911)
[ 5616.685921] Sending NMI from CPU 8 to CPUs 24:
[ 5627.810360] rcu_sched kthread starved for 1110 jiffies! g17311 c17310 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=112
[ 5627.810475] rcu_sched       I    0     9      2 0x00000800
[ 5627.810544] Call Trace:
[ 5627.810584] [c0000007fad9b8c0] [c000000000063da8] ftrace_call+0x4/0xbc (unreliable)
[ 5627.810686] [c0000007fad9ba90] [c00000000001b018] __switch_to+0x2f8/0x440
[ 5627.810771] [c0000007fad9baf0] [c000000000b07ea8] __schedule+0x2a8/0x9e0
[ 5627.810854] [c0000007fad9bbc0] [c000000000b08628] schedule+0x48/0xc0
[ 5627.810937] [c0000007fad9bbf0] [c000000000b0d5b0] schedule_timeout+0x1f0/0x4d0
[ 5627.811036] [c0000007fad9bce0] [c00000000018d1ec] rcu_gp_kthread+0x4fc/0xa60
[ 5627.811132] [c0000007fad9bdc0] [c00000000012b578] kthread+0x168/0x1b0
[ 5627.811216] [c0000007fad9be30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c

Earlier, I have enabled function tracing as in below script:

mkdir -p /debug
mount -t debugfs nodev /debug 2>&1
echo '*' >/debug/tracing/set_ftrace_filter
echo function >/debug/tracing/current_tracer
echo 1 >/debug/tracing/tracing_on
read -p "press ENTER key to cancel .." var
echo 0 >/debug/tracing/tracing_on
cat /debug/tracing/trace > /tmp/tracing.out$$
echo "/tmp/tracing.out$$ is created .."
@cdeadmin
Copy link

cdeadmin commented Nov 2, 2017

------- Comment From [email protected] 2017-11-02 05:21:08 EDT-------
Discussed with Gowri, this issue is always recreatable with ftrace function on.. and after a while host is hung.. hard reset of host is required in order to recover from this issue..

@cdeadmin
Copy link

------- Comment From [email protected] 2017-11-24 02:10:54 EDT-------
(In reply to comment #11)
> (In reply to comment #10)
> > (In reply to comment #9)
> > > @satheesh,
> > >
> > > Any updates on this?
> >
> > Naveen,
> > do you think
> > https://patchwork.kernel.org/patch/10042881/ would help in this case.
> > do you know if your fix made it to 4.14-rc4?
>
> From the report above, this looks to be a ppc64le host. The patch above is
> for BE, so it is probably not applicable in this case. I am not able to
> conclude anything from the provided traces.
>
> Satheesh,
> Could you please test v4.14 to see if this problem can be reproduced there?
> If so, it would also be good to see if reverting commit
> 6b847d7 helps. That commit changed which
> functions get traced, so it might have an impact here.

I am able to hit with latest devel(4.14) 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le

Steps to reproduce:

  1. Run the below script in host, wait at ""press ENTER key to cancel" step,

cat ftrace.sh

mkdir -p /debug
mount -t debugfs nodev /debug 2>&1
echo '*' >/debug/tracing/set_ftrace_filter
echo function >/debug/tracing/current_tracer
echo 1 >/debug/tracing/tracing_on
read -p "press ENTER key to cancel .." var
echo 0 >/debug/tracing/tracing_on
cat /debug/tracing/trace > /tmp/tracing.out$$
echo "/tmp/tracing.out$$ is created .."

  1. Run a vm start stop in another terminal, able to hit hardlock immediately
    for i in {1..20};do virsh destroy vm1;virsh start vm1;virsh domstate vm1;sleep 2;done

Message from syslogd@localhost at Nov 24 02:03:07 ...
kernel:Watchdog CPU:56 detected Hard LOCKUP other CPUS:24
error: Failed to destroy domain vm1
error: Failed to terminate process 5481 with SIGKILL: Device or resource busy

error: Domain is already active

in shutdown

error: Failed to destroy domain vm1
error: Failed to terminate process 5481 with SIGKILL: Device or resource busy

Regards,
-Satheesh

@cdeadmin
Copy link

------- Comment From [email protected] 2017-11-24 05:33:38 EDT-------
> > Satheesh,
> > Could you please test v4.14 to see if this problem can be reproduced there?
> > If so, it would also be good to see if reverting commit
> > 6b847d7 helps. That commit changed which
> > functions get traced, so it might have an impact here.
>
> I am able to hit with latest devel(4.14)
> 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le

Thanks for confirming. It is not clear from your message if you tried by reverting the patch I mentioned above. There also seems to have been many watchdog fixes that went into v4.15, so it will be good to test with those for more reliable traces. I can take a look if you can spare the test machine for a day next week?

@cdeadmin
Copy link

------- Comment From [email protected] 2017-11-02 04:10:28 EDT-------
#22

------- Comment From [email protected] 2017-11-02 04:21:08 EDT-------

------- Comment From [email protected] 2017-11-02 05:54:46 EDT-------
It is unclear if the behavior is being reported on the guest or the host. We do notice devices resetting in the logs, so if these messages are from the guest, what is happening on the host? The lockups seem to coincide with the devices being locked up.

Can we please get sosreport from the host and the guest? There is not really enough data here to proceed on this.

------- Comment From [email protected] 2017-11-02 05:57:23 EDT-------
(In reply to comment #4)
> It is unclear if the behavior is being reported on the guest or the host. We
> do notice devices resetting in the logs, so if these messages are from the
> guest, what is happening on the host? The lockups seem to coincide with the
> devices being locked up.
> Can we please get sosreport from the host and the guest? There is not really
> enough data here to proceed on this.

From what I heard from Gowri and description of this issue.. hard lockup happens while booting guest with ftrace..
@gowri can you please help with confirming above and providing sosreports?

------- Comment From [email protected] 2017-11-02 06:00:02 EDT-------
(In reply to comment #5)
> @gowri can you please help with confirming above and providing sosreports?

Forgot to mention: @gowri sosreport running as-is on hostos has issues... instead run sosreport -n powerpc ...

------- Comment From [email protected] 2017-11-02 09:01:42 EDT-------
Reported trace was from host kernel, as we enable ftrace in host and then start guest.

Btw I am not quite sure how it would be possible to run sosreport as I could not execute any command once hard lockup detected and I find current ssh login as well as ipmi console login stops responding momentarily.

@srikant, hope your team would be able to assist providing any other needed info as server is not with me (I spotted it while debugging some other bug).

------- Comment From [email protected] 2017-11-02 09:09:30 EDT-------
(In reply to comment #7)
> Reported trace was from host kernel, as we enable ftrace in host and then
> start guest.
> Btw I am not quite sure how it would be possible to run sosreport as I could
> not execute any command once hard lockup detected and I find current ssh
> login as well as ipmi console login stops responding momentarily.
> @srikant, hope your team would be able to assist providing any other needed
> info as server is not with me (I spotted it while debugging some other bug).

Sure Gowri. Since we cannot run sosreport after hitting issue, we will try to reset host and capture logs.. @satheesh would help here in getting required logs..

------- Comment From [email protected] 2017-11-21 23:43:36 EDT-------
@satheesh,

Any updates on this?

------- Comment From [email protected] 2017-11-22 00:53:30 EDT-------
(In reply to comment #9)
> @satheesh,
>
> Any updates on this?

Naveen,
do you think
https://patchwork.kernel.org/patch/10042881/ would help in this case.
do you know if your fix made it to 4.14-rc4?

------- Comment From [email protected] 2017-11-23 11:38:01 EDT-------
(In reply to comment #10)
> (In reply to comment #9)
> > @satheesh,
> > Any updates on this?
>
> Naveen,
> do you think
> https://patchwork.kernel.org/patch/10042881/ would help in this case.
> do you know if your fix made it to 4.14-rc4?

From the report above, this looks to be a ppc64le host. The patch above is for BE, so it is probably not applicable in this case. I am not able to conclude anything from the provided traces.

Satheesh,
Could you please test v4.14 to see if this problem can be reproduced there? If so, it would also be good to see if reverting commit 6b847d7 helps. That commit changed which functions get traced, so it might have an impact here.

------- Comment From [email protected] 2017-11-24 01:10:54 EDT-------
> >
>
>

------- Comment From [email protected] 2017-11-24 01:14:09 EDT-------
Created attachment 122559
host cal trace

------- Comment From [email protected] 2017-11-24 04:33:38 EDT-------
>

------- Comment From [email protected] 2017-11-28 08:12:02 EDT-------
(In reply to comment #14)
> > > Satheesh,
> > > Could you please test v4.14 to see if this problem can be reproduced there?
> > > If so, it would also be good to see if reverting commit
> > > 6b847d7 helps. That commit changed which
> > > functions get traced, so it might have an impact here.
> >
> > I am able to hit with latest devel(4.14)
> > 4.14.0-3.dev.git68b4afb.el7.centos.ppc64le
>
> Thanks for confirming. It is not clear from your message if you tried by
> reverting the patch I mentioned above. There also seems to have been many
> watchdog fixes that went into v4.15, so it will be good to test with those
> for more reliable traces. I can take a look if you can spare the test
> machine for a day next week?

I had tried with latest hostos devel branch, yet to tryout with your suggestion of reverting that commit, planning to try out now, will update once have the results.

Regards,
-Satheesh.

------- Comment From [email protected] 2017-11-28 08:51:21 EDT-------
Hi Naveen,

I am able to hit hardlockup even with reverting commit 6b847d7,
test system: ltc-test-ci1.aus.stglabs.ibm.com
password: passw0rd

[ 226.151759] virbr0: topology change detected, propagating
[17582052814.647948] Delta way too big! 17582052588050456478 ts=17582052814647941894 write stamp = 226597485416
[ 244.398673] Watchdog CPU:24 detected Hard LOCKUP other CPUS:0
[ 286.645022] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 286.645131] 0-...: (1 GPs behind) idle=a82/140000000000000/0 softirq=3930/3930 fqs=2906
[ 286.645217] (detected by 32, t=6002 jiffies, g=2328, c=2327, q=6125)
[ 286.645322] Sending NMI from CPU 32 to CPUs 0:
[ 297.706160] rcu_sched kthread starved for 1104 jiffies! g2328 c2327 f0x0 RCU_GP_DOING_FQS(4) ->state=0x0 ->cpu=64
[ 297.706259] rcu_sched I 0 9 2 0x00000800
[ 297.706337] Call Trace:
[ 297.706383] [c0000007f864f8c0] [c000000000063b68] ftrace_call+0x4/0xbc (unreliable)
[ 297.706496] [c0000007f864fa90] [c00000000001b038] __switch_to+0x2f8/0x440
[ 297.706589] [c0000007f864faf0] [c000000000b09088] __schedule+0x2a8/0x9e0
[ 297.706679] [c0000007f864fbc0] [c000000000b09808] schedule+0x48/0xc0
[ 297.706770] [c0000007f864fbf0] [c000000000b0e790] schedule_timeout+0x1f0/0x4d0
[ 297.706875] [c0000007f864fce0] [c00000000018dc0c] rcu_gp_kthread+0x4fc/0xa60
[ 297.706979] [c0000007f864fdc0] [c00000000012beb8] kthread+0x168/0x1b0
[ 297.707071] [c0000007f864fe30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
[ 394.696906] Watchdog CPU:16 detected Hard LOCKUP other CPUS:40
[ 394.697040] Watchdog CPU:40 Hard LOCKUP
[ 394.697106] Modules linked in: vhost_net vhost tap act_police cls_u32 sch_ingress cls_fw sch_sfq sch_htb xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables i2c_opal ses i2c_core enclosure scsi_transport_sas ipmi_powernv ipmi_devintf ipmi_msghandler powernv_op_panel nfsd auth_rpcgss oid_registry nfs_acl lockd grace kvm_hv sunrpc kvm_pr kvm xfs libcrc32c tg3 ptp pps_core
[ 394.698496] CPU: 40 PID: 2789 Comm: python2 Not tainted 4.14.0+ #3
[ 394.698579] task: c0000007ca8b9080 task.stack: c00000079e24c000
[ 394.698661] NIP: c0000000001b91e8 LR: c0000000001b9188 CTR: c00000000008ff10
[ 394.698756] REGS: c00000079e24f780 TRAP: 0501 Not tainted (4.14.0+)
[ 394.698838] MSR: 900000010280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE,TM[E]> CR: 44244824 XER: 20000000
[ 394.699127] CFAR: c0000000001b91f0 SOFTE: 1
[ 394.699127] GPR00: c0000000001b9188 c00000079e24fa00 c000000001403900 0000000000000000
[ 394.699127] GPR04: 0000000000000400 0000000000000000 0000000000000000 c00000000fd60000
[ 394.699127] GPR08: 0000000000000001 0000000000000003 c0000007ff528ee0 0000000000000010
[ 394.699127] GPR12: c00000000008ff00 c00000000fd7a400
[ 394.699684] NIP [c0000000001b91e8] smp_call_function_many+0x388/0x420
[ 394.699769] LR [c0000000001b9188] smp_call_function_many+0x328/0x420
[ 394.699849] Call Trace:
[ 394.699894] [c00000079e24fa00] [c0000000001b9188] smp_call_function_many+0x328/0x420 (unreliable)
[ 394.700029] [c00000079e24fa70] [c000000000070ff8] pmdp_invalidate+0x98/0xe0
[ 394.700139] [c00000079e24faa0] [c00000000033b41c] change_huge_pmd+0x8c/0x390
[ 394.700325] [c00000079e24fb10] [c0000000002e6c3c] change_protection+0xb1c/0xfb0
[ 394.700512] [c00000079e24fca0] [c0000000003164f8] change_prot_numa+0x38/0xb0
[ 394.700698] [c00000079e24fcd0] [c000000000144510] task_numa_work+0x2f0/0x400
[ 394.700884] [c00000079e24fda0] [c000000000129100] task_work_run+0x140/0x1a0
[ 394.701045] [c00000079e24fe00] [c00000000001ca90] do_notify_resume+0xf0/0x100
[ 394.701233] [c00000079e24fe30] [c00000000000bec4] ret_from_except_lite+0x70/0x74
[ 394.701414] Instruction dump:
[ 394.701519] 812a0018 792807e1 41820034 4800001c 60000000 60000000 60000000 60000000
[ 394.701786] 60000000 60420000 7c210b78 7c421378 <812a0018> 792807e1 4082fff0 7c2004ac

Regards,
-Satheesh

------- Comment From [email protected] 2018-01-10 19:58:24 EDT-------
Naveen, Satheesh - this seems to have stalled. Is there a clear 'next action' here? This is, after all, listed as a Ship Issue.

------- Comment From [email protected] 2018-01-30 18:04:09 EDT-------
Downgrading to a normal, in the absence of any new information from Naveen or Satheesh.

It sounds like from comment 7 that we need a host running ftrace and then when we start a guest we will see the host develop problems. Correct?

Can I have access to a host that I can use to investigate this further? ltc-test-ci1.aus.stglabs.ibm.com no longer seems available.

------- Comment From [email protected] 2018-01-31 03:11:34 EDT-------
Rick,
Yes, sorry, I should have put out an update here. I discussed this with Satheesh in the context of https://bugzilla.linux.ibm.com/show_bug.cgi?id=164049 on Monday. I have started looking into this and should hopefully have some update this week.

------- Comment From [email protected] 2018-02-01 16:54:18 EDT-------
Added dependency to 164049. Depending on what Naveen and Satheesh discover, may even be a dup.

------- Comment From [email protected] 2018-02-22 17:23:49 EDT-------
Naveen, any update on this?

@cdeadmin
Copy link

------- Comment From [email protected] 2018-03-12 04:51:36 EDT-------
I missed putting an update here, but only did so on https://bugzilla.linux.ibm.com/show_bug.cgi?id=164049 . I am working on an upstream fix for this.

@cdeadmin
Copy link

------- Comment From [email protected] 2018-08-31 13:32:05 EDT-------
Just declaring this fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cdeadmin @gowrishankarm