[3.8.y.z extended stable] Patch "x86/ioapic/kcrash: Prevent crash_kexec() from deadlocking on" has been added to staging queue

[Kamal Mostafa]

This is a note to let you know that I have just added a patch titled

    x86/ioapic/kcrash: Prevent crash_kexec() from deadlocking on

to the linux-3.8.y-queue branch of the 3.8.y.z extended stable tree 
which can be found at:

 http://kernel.ubuntu.com/git?p=ubuntu/linux.git;a=shortlog;h=refs/heads/linux-3.8.y-queue

This patch is scheduled to be released in version 3.8.13.14.

If you, or anyone else, feels it should not be added to this tree, please 
reply to this email.

For more information about the 3.8.y.z tree, see
https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable

Thanks.
-Kamal

------

>From 04989392279e97ae515b4f8162788ac2d5d8c04e Mon Sep 17 00:00:00 2001
From: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez at hitachi.com>
Date: Tue, 20 Aug 2013 16:01:07 +0900
Subject: x86/ioapic/kcrash: Prevent crash_kexec() from deadlocking on
 ioapic_lock

commit 17405453f4ad0220721a29978692081be6392b8f upstream.

Prevent crash_kexec() from deadlocking on ioapic_lock. When
crash_kexec() is executed on a CPU, the CPU will take ioapic_lock
in disable_IO_APIC(). So if the cpu gets an NMI while locking
ioapic_lock, a deadlock will happen.

In this patch, ioapic_lock is zapped/initialized before disable_IO_APIC().

You can reproduce this deadlock the following way:

1. Add mdelay(1000) after raw_spin_lock_irqsave() in
   native_ioapic_set_affinity()@arch/x86/kernel/apic/io_apic.c

   Although the deadlock can occur without this modification, it will increase
   the potential of the deadlock problem.

2. Build and install the kernel

3. Set up the OS which will run panic() and kexec when NMI is injected
    # echo "kernel.unknown_nmi_panic=1" >> /etc/sysctl.conf
    # vim /etc/default/grub
      add "nmi_watchdog=0 crashkernel=256M" in GRUB_CMDLINE_LINUX line
    # grub2-mkconfig

4. Reboot the OS

5. Run following command for each vcpu on the guest
    # while true; do echo <CPU num> > /proc/irq/<IO-APIC-edge or IO-APIC-fasteoi>/smp_affinitity; done;
   By running this command, cpus will get ioapic_lock for setting affinity.

6. Inject NMI (push a dump button or execute 'virsh inject-nmi <domain>' if you
   use VM). After injecting NMI, panic() is called in an nmi-handler context.
   Then, kexec will normally run in panic(), but the operation will be stopped
   by deadlock on ioapic_lock in crash_kexec()->machine_crash_shutdown()->
   native_machine_crash_shutdown()->disable_IO_APIC()->clear_IO_APIC()->
   clear_IO_APIC_pin()->ioapic_read_entry().

Signed-off-by: Yoshihiro YUNOMAE <yoshihiro.yunomae.ez at hitachi.com>
Cc: Andi Kleen <ak at linux.intel.com>
Cc: Gleb Natapov <gleb at redhat.com>
Cc: Konrad Rzeszutek Wilk <konrad.wilk at oracle.com>
Cc: Joerg Roedel <joro at 8bytes.org>
Cc: Marcelo Tosatti <mtosatti at redhat.com>
Cc: Hidehiro Kawai <hidehiro.kawai.ez at hitachi.com>
Cc: Sebastian Andrzej Siewior <sebastian at breakpoint.cc>
Cc: Zhang Yanfei <zhangyanfei at cn.fujitsu.com>
Cc: Eric W. Biederman <ebiederm at xmission.com>
Cc: yrl.pp-manager.tt at hitachi.com
Cc: Masami Hiramatsu <masami.hiramatsu.pt at hitachi.com>
Cc: Seiji Aguchi <seiji.aguchi at hds.com>
Link: http://lkml.kernel.org/r/20130820070107.28245.83806.stgit@yunodevel
Signed-off-by: Ingo Molnar <mingo at kernel.org>
[ kamal: 3.8 stable prereq for
  522e664 x86/apic: Disable I/O APIC before shutdown of the local APIC ]
Signed-off-by: Kamal Mostafa <kamal at canonical.com>
---
 arch/x86/include/asm/apic.h    | 2 ++
 arch/x86/kernel/apic/io_apic.c | 5 +++++
 arch/x86/kernel/crash.c        | 4 +++-
 3 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
index 3388034..cb8bc81 100644
--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -688,4 +688,6 @@ extern int default_check_phys_apicid_present(int phys_apicid);

 #endif /* CONFIG_X86_LOCAL_APIC */

+extern void ioapic_zap_locks(void);
+
 #endif /* _ASM_X86_APIC_H */
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index b739d39..c605173 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -1513,6 +1513,11 @@ static void __init setup_timer_IRQ0_pin(unsigned int ioapic_idx,
 	ioapic_write_entry(ioapic_idx, pin, entry);
 }

+void ioapic_zap_locks(void)
+{
+	raw_spin_lock_init(&ioapic_lock);
+}
+
 __apicdebuginit(void) print_IO_APIC(int ioapic_idx)
 {
 	int i;
diff --git a/arch/x86/kernel/crash.c b/arch/x86/kernel/crash.c
index 74467fe..e0e0841 100644
--- a/arch/x86/kernel/crash.c
+++ b/arch/x86/kernel/crash.c
@@ -128,7 +128,9 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	cpu_emergency_svm_disable();

 	lapic_shutdown();
-#if defined(CONFIG_X86_IO_APIC)
+#ifdef CONFIG_X86_IO_APIC
+	/* Prevent crash_kexec() from deadlocking on ioapic_lock. */
+	ioapic_zap_locks();
 	disable_IO_APIC();
 #endif
 #ifdef CONFIG_HPET_TIMER
--
1.8.3.2