Debugging extreme disk IO latency

[Rashkae]

Åsmund Grammeltvedt wrote:
> Hi.
> 
> I am running into some very nasty IO latency, using a Western Digital Caviar 
> Green 1.5TB (WD15EADS) with XFS. Adding one writer (cat /dev/zero | pv > 
> /tmp/foo) appears to cause a massive latency spike after ~10-20 minutes, where 
> almost no interaction, read or write, with the disk comes through. The spike 
> lasts for up to ~5 minutes, after which everything is back to normal.
> 
> Any ideas of what might be causing this or how to proceed with debugging this 
> issue would be much appreciated. Thanks.
> 

Can you replicate this if you boot from a live CD?  If yes, are you able
to run dmesg to see if there are any relevant messages during the spikes?

If you can reproduce the error with the BootCD, I would vet the drive
itself by testing on a different computer, if possible.