[Bug 634605] [NEW] Kernel crash with invalid opcode: 0000 [1] SMP (crash utility compatible vmcore and vmlinux available)
chris
634605 at bugs.launchpad.net
Fri Sep 10 00:52:50 UTC 2010
Public bug reported:
I have a linux box using tun drivers with vpn kind of applications, it hosts http caching service for VPN clients. The throughput served by box is around 500Mbps.
Recently machine crashed when application handling the data traffic tried to do a write on the tun device(info from backtrace). About 500Mbps of http traffic was served by the machine at the time of crash. At first i started seeing a lot of printk message "time wait bucket table overflow" and then later
system crashed. As far as i understand these message shouldn't be directly related to crash, but mean that system is loaded with lot many TCP connections.
I have the core file with me and i am including a backtrace here. could someone help me with fixing this issue. Any pointers on debugging this issue could be helpful. I can provide any required information.
#Kernel version: 2.6.24-27 on ubuntu hardy.
# lsb_release -rd
Description: Ubuntu 8.04.3 LTS
Release: 8.04
# cat /proc/version_signature
Ubuntu 2.6.24-4.6-server
[365302.597824] printk: 10886 messages suppressed.
[365302.597834] TCP: time wait bucket table overflow
[365309.492501] printk: 10811 messages suppressed.
[365309.492508] TCP: time wait bucket table overflow
[371482.767760] invalid opcode: 0000 [1] SMP
[371482.767916] CPU 2
[371482.768021] Modules linked in: tipc mptctl af_packet xt_TPROXY xt_NOTRACK xt_MARK xt_tcpudp ipt_addrtype xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core iptable_mangle iptable_raw obp bonding iptable_filter ip_tables x_tables xfs sr_mod cdrom tcp_westwood tcp_veno tcp_vegas tcp_hybla ipv6 loop joydev e1000e psmouse shpchp evdev button tpm_infineon tpm tpm_bios i5000_edac edac_core pci_hotplug serio_raw pcspkr ext3 jbd mbcache pata_acpi sg usbhid hid usb_storage libusual ata_piix sd_mod ahci ehci_hcd ata_generic uhci_hcd libata usbcore mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal processor fan fuse
[371482.770941] Pid: 15939, comm: Dp[7] Not tainted 2.6.24-27-server-tproxy #1
[371482.771055] RIP: 0010:[<ffffffff805d1721>] [<ffffffff805d1721>] tcp_prot+0x1/0x160
[371482.771211] RSP: 0018:ffff81040e04bc00 EFLAGS: 00010282
[371482.771291] RAX: ffffffff805d1720 RBX: ffff81005b1b1000 RCX: ffff810409938208
[371482.771404] RDX: 0000000000000044 RSI: 000000003e741618 RDI: ffff81005b1b1000
[371482.771517] RBP: ffff81040e04bc28 R08: 0000000000005000 R09: 0000000000000044
[371482.771630] R10: 00000000a700040a R11: 00000000ad040013 R12: 0000000000000000
[371482.771743] R13: ffff81016c118000 R14: ffff8101f3909800 R15: ffff8102667e3200
[371482.771856] FS: 0000000045c29950(0063) GS:ffff810411401b80(0000) knlGS:0000000000000000
[371482.771972] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[371482.772055] CR2: 00007f11f0026f00 CR3: 0000000154117000 CR4: 00000000000006e0
[371482.772168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[371482.772281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[371482.772394] Process GtpDp[7] (pid: 15939, threadinfo ffff810150fbc000, task ffff8101785cf7d0)
[371482.772512] Stack: ffffffff803eec25 ffffffff88409102 ffff81005b1b1000 0000000000000000
[371482.772805] ffff81016c118000 ffff81040e04bc78 ffffffff884091b9 ffff810400000000
[371482.773060] ffffffff8043db8d 0000000000000001 0000000000000000 ffff81016c118070
[371482.773255] Call Trace:
[371482.773388] <IRQ> [<ffffffff803eec25>] sk_free+0x25/0x110
[371482.773514] [<ffffffff88409102>] :xt_socket:socket_mt+0xc2/0x1b0
[371482.773600] [<ffffffff884091b9>] :xt_socket:socket_mt+0x179/0x1b0
[371482.773687] [<ffffffff8043db8d>] tcp_v4_do_rcv+0x24d/0x6d0
[371482.773773] [<ffffffff8837fdeb>] :ip_tables:ipt_do_table+0x22b/0x510
[371482.773869] [<ffffffff883e5018>] :iptable_mangle:ipt_route_hook+0x18/0x20
[371482.773963] [<ffffffff80418b22>] nf_iterate+0x62/0xa0
[371482.774046] [<ffffffff8041f1c0>] ip_rcv_finish+0x0/0x3c0
[371482.774129] [<ffffffff80418bfb>] nf_hook_slow+0x9b/0xf0
[371482.774210] [<ffffffff8041f1c0>] ip_rcv_finish+0x0/0x3c0
[371482.774295] [<ffffffff8041fb2d>] ip_rcv+0x21d/0x300
[371482.774378] [<ffffffff803f8c8a>] netif_receive_skb+0x3aa/0x490
[371482.774464] [<ffffffff803fba6f>] process_backlog+0x7f/0xf0
[371482.774549] [<ffffffff803fb3ca>] net_rx_action+0x12a/0x230
[371482.774637] [<ffffffff80244db4>] __do_softirq+0x74/0xf0
[371482.774722] [<ffffffff8020d58c>] call_softirq+0x1c/0x30
[371482.774803] <EOI> [<ffffffff8020ed9d>] do_softirq+0x3d/0x90
[371482.774924] [<ffffffff803f9051>] netif_rx_ni+0x21/0x30
[371482.775014] [<ffffffff883ac90f>] :dp_write+0x14f/0x2b0
[371482.775103] [<ffffffff802b9abd>] vfs_write+0xed/0x190
[371482.775186] [<ffffffff802ba1a0>] sys_write+0x50/0x90
[371482.775269] [<ffffffff8020c3fe>] system_call+0x7e/0x83
[371482.775353]
[371482.775419]
[371482.775420] Code: ce 42 80 ff ff ff ff d0 f1 43 80 ff ff ff ff 00 e3 42 80 ff
[371482.776334] RIP [<ffffffff805d1721>] tcp_prot+0x1/0x160
[371482.776450] RSP <ffff81040e04bc00>
** Affects: linux (Ubuntu)
Importance: Undecided
Status: New
** Description changed:
- I have a linux box using tun drivers with vpn kind of applications. The throughput served by box is around 500Mbps.
- Recently machine crashed which application tried to do a write on the tun device. About 500Mbps of http traffic was served by the machine at the time of crash. At first i started seeing a lot of printk message "time wait bucket table overflow" and then later
+ I have a linux box using tun drivers with vpn kind of applications, it hosts http caching service for VPN clients. The throughput served by box is around 500Mbps.
+ Recently machine crashed when application handling the data traffic tried to do a write on the tun device(info from backtrace). About 500Mbps of http traffic was served by the machine at the time of crash. At first i started seeing a lot of printk message "time wait bucket table overflow" and then later
system crashed. As far as i understand these message shouldn't be directly related to crash, but mean that system is loaded with lot many TCP connections.
I have the core file with me and i am including a backtrace here. could someone help me with fixing this issue. Any pointers on debugging this issue could be helpful. I can provide any required information.
#Kernel version: 2.6.24-27 on ubuntu hardy.
# lsb_release -rd
Description: Ubuntu 8.04.3 LTS
Release: 8.04
# cat /proc/version_signature
Ubuntu 2.6.24-4.6-server
[365302.597824] printk: 10886 messages suppressed.
[365302.597834] TCP: time wait bucket table overflow
[365309.492501] printk: 10811 messages suppressed.
[365309.492508] TCP: time wait bucket table overflow
[371482.767760] invalid opcode: 0000 [1] SMP
[371482.767916] CPU 2
[371482.768021] Modules linked in: tipc mptctl af_packet xt_TPROXY xt_NOTRACK xt_MARK xt_tcpudp ipt_addrtype xt_socket nf_conntrack nf_defrag_ipv4 nf_tproxy_core iptable_mangle iptable_raw obp bonding iptable_filter ip_tables x_tables xfs sr_mod cdrom tcp_westwood tcp_veno tcp_vegas tcp_hybla ipv6 loop joydev e1000e psmouse shpchp evdev button tpm_infineon tpm tpm_bios i5000_edac edac_core pci_hotplug serio_raw pcspkr ext3 jbd mbcache pata_acpi sg usbhid hid usb_storage libusual ata_piix sd_mod ahci ehci_hcd ata_generic uhci_hcd libata usbcore mptsas mptscsih mptbase scsi_transport_sas scsi_mod thermal processor fan fuse
[371482.770941] Pid: 15939, comm: Dp[7] Not tainted 2.6.24-27-server-tproxy #1
[371482.771055] RIP: 0010:[<ffffffff805d1721>] [<ffffffff805d1721>] tcp_prot+0x1/0x160
[371482.771211] RSP: 0018:ffff81040e04bc00 EFLAGS: 00010282
[371482.771291] RAX: ffffffff805d1720 RBX: ffff81005b1b1000 RCX: ffff810409938208
[371482.771404] RDX: 0000000000000044 RSI: 000000003e741618 RDI: ffff81005b1b1000
[371482.771517] RBP: ffff81040e04bc28 R08: 0000000000005000 R09: 0000000000000044
[371482.771630] R10: 00000000a700040a R11: 00000000ad040013 R12: 0000000000000000
[371482.771743] R13: ffff81016c118000 R14: ffff8101f3909800 R15: ffff8102667e3200
[371482.771856] FS: 0000000045c29950(0063) GS:ffff810411401b80(0000) knlGS:0000000000000000
[371482.771972] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[371482.772055] CR2: 00007f11f0026f00 CR3: 0000000154117000 CR4: 00000000000006e0
[371482.772168] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[371482.772281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[371482.772394] Process GtpDp[7] (pid: 15939, threadinfo ffff810150fbc000, task ffff8101785cf7d0)
[371482.772512] Stack: ffffffff803eec25 ffffffff88409102 ffff81005b1b1000 0000000000000000
[371482.772805] ffff81016c118000 ffff81040e04bc78 ffffffff884091b9 ffff810400000000
[371482.773060] ffffffff8043db8d 0000000000000001 0000000000000000 ffff81016c118070
[371482.773255] Call Trace:
[371482.773388] <IRQ> [<ffffffff803eec25>] sk_free+0x25/0x110
[371482.773514] [<ffffffff88409102>] :xt_socket:socket_mt+0xc2/0x1b0
[371482.773600] [<ffffffff884091b9>] :xt_socket:socket_mt+0x179/0x1b0
[371482.773687] [<ffffffff8043db8d>] tcp_v4_do_rcv+0x24d/0x6d0
[371482.773773] [<ffffffff8837fdeb>] :ip_tables:ipt_do_table+0x22b/0x510
[371482.773869] [<ffffffff883e5018>] :iptable_mangle:ipt_route_hook+0x18/0x20
[371482.773963] [<ffffffff80418b22>] nf_iterate+0x62/0xa0
[371482.774046] [<ffffffff8041f1c0>] ip_rcv_finish+0x0/0x3c0
[371482.774129] [<ffffffff80418bfb>] nf_hook_slow+0x9b/0xf0
[371482.774210] [<ffffffff8041f1c0>] ip_rcv_finish+0x0/0x3c0
[371482.774295] [<ffffffff8041fb2d>] ip_rcv+0x21d/0x300
[371482.774378] [<ffffffff803f8c8a>] netif_receive_skb+0x3aa/0x490
[371482.774464] [<ffffffff803fba6f>] process_backlog+0x7f/0xf0
[371482.774549] [<ffffffff803fb3ca>] net_rx_action+0x12a/0x230
[371482.774637] [<ffffffff80244db4>] __do_softirq+0x74/0xf0
[371482.774722] [<ffffffff8020d58c>] call_softirq+0x1c/0x30
[371482.774803] <EOI> [<ffffffff8020ed9d>] do_softirq+0x3d/0x90
[371482.774924] [<ffffffff803f9051>] netif_rx_ni+0x21/0x30
[371482.775014] [<ffffffff883ac90f>] :dp_write+0x14f/0x2b0
[371482.775103] [<ffffffff802b9abd>] vfs_write+0xed/0x190
[371482.775186] [<ffffffff802ba1a0>] sys_write+0x50/0x90
[371482.775269] [<ffffffff8020c3fe>] system_call+0x7e/0x83
[371482.775353]
[371482.775419]
[371482.775420] Code: ce 42 80 ff ff ff ff d0 f1 43 80 ff ff ff ff 00 e3 42 80 ff
[371482.776334] RIP [<ffffffff805d1721>] tcp_prot+0x1/0x160
[371482.776450] RSP <ffff81040e04bc00>
--
Kernel crash with invalid opcode: 0000 [1] SMP (crash utility compatible vmcore and vmlinux available)
https://bugs.launchpad.net/bugs/634605
You received this bug notification because you are a member of Kernel
Bugs, which is subscribed to linux in ubuntu.
More information about the kernel-bugs
mailing list