IRQ balancing problem on new Nehalem

[DAHLBOKUM Markus (IVECO MOTORENFORSCHUNG AG)]

Hello all,

I have got a new server for our calculations department and I have some troubles with the performance.
The server is:

CPU: XEON X5570 (1 Quad CPU)
24 GB Memory
2 10k Disks for the OS
10 15k Disks for scratch data
Adaptec RAID 51645

The server is used for numerical calculations, so there is always a high load on the CPUs and on the disks.
Simple benchmarks on the disks only or on the CPUs only give good performance values, but under full load the performance is not acceptable. The scratch disks have about 700 MB/s for reading and writing with bonnie++. This is ok. But within a calculation I only get 150 MB/s reading and 580 MB/s writing. Writing is ok because this is random access performance. But reading is a disaster. The calculation is reading 10h!!!
What I found out now is that the interrupts all go over the first CPU. So all I/O activities (which can be a lot during a calculation) are handled by one core.

What I did so far:
First I was running debian lenny (2.6.26) which was unstable -> several system freezes
Then I updated the system via backports to kernel 2.6.30 -> the system became stable but with the described issues
Now I'm running ubuntu 9.10 beta (2.6.31) -> still the same
Between the OS changes I tried Knoppix 6.0.1 -> the IRQs were balanced!!!! (But I don't want to run knoppix as the main OS on this machine :))

The ksoftirqd are running on the machine but with no effect (why???). I tried the irqbalance package which had the effect that the interrupts were balanced a little more, but the I/O was still on one core.

Is this problem known? What can I do to balance the IRQs?

Thank you for your help.
Markus

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20091007/9494dd4b/attachment.html>