IO Wait very high and real IO speed is very low

Tang Jianwei myhnet at gmail.com
Mon Mar 3 07:43:05 UTC 2014 

Hi Marty,

     I am right now using 3.2.0-48

     Linux 3.2.0-48-generic #74-Ubuntu SMP Thu Jun 6 19:43:26 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux


Best Regards
Tomas

On 2014/3/3 14:55, Marty Sweet wrote:
> Hi Tomas,
>
> What kernel version are you currently running?
>
> $ uname -a
>
> Thanks,
> Marty
>
>
>
> -------- Original message --------
> From: Tang Jianwei
> Date:03/03/2014 03:36 (GMT+00:00)
> To: Marty Sweet ,ubuntu-users at lists.ubuntu.com
> Subject: Re: IO Wait very high and real IO speed is very low
>
> Hi Marty,
>
> Thank you very much, please see my comments:
>
> that machine is my backup server and the following is how my backup task
> working procedure:
> 1. make a local copy of the current backup with cp -al (hard link)
> 2. use rsync to sync remote files if there is change/new/delete data
>
> So, all the data should be write to local disk, and it won't send data 
> out.
>
> I understand that the write speed might be low for the reason that there
> is not too much data to write due to the hard link, but the problem is:
>
> when there is no backup tasks running(low load), the command "dd
> if=/dev/zero of=/backup/test10G bs=1M count=10240" write data in
> 500MBps, and it reduce to around 10-20MBps when the backup tasks
> running(high load).
>
> and the problem still exist after I run fsck as you suggested.
>
> Best Regards
> Tomas
>
> On 2014/3/2 20:32, Marty Sweet wrote:
> > Hi Tomas,
> >
> > So it appears you are reading from the disk but not putting it
> > anywhere, for example, backing it up via a network. A utility which
> > will probably help drill this down is $ iotop, here you will be able
> > to see what processes are using what disk %. You may also like to try
> > $ nmon, then press d for disk information.
> >
> > I would imagine it will be one of your rsync processes although those
> > should of been going onto the network.
> >
> > When you say 'the write speed can reach 500MB/s when there is no other
> > processes running', what size file are you testing this with?
> > Something reasonably small will be saved to cache (Write-Back) before
> > actually being committed to disk. As for the read speeds with those
> > Wait % something is definitely odd looking at your hardware.
> >
> > Just to be on the safe side, I would recommend running fsck on your
> > data volume, this will take the volume offline for some time.
> > umount /dev/sda2
> > fsck -f /dev/sda2
> > mount /dev/sda2
> >
> > Marty
> >
> > On 2 March 2014 11:10, Tang Jianwei <myhnet at gmail.com> wrote:
> >> Hi Marty,
> >>
> >>      Thank you very much for your time and help, here is the output 
> of $dstat
> >>      http://pastebin.com/kiHijEKY
> >>
> >> Best Regards
> >> Tomas
> >>
> >> On 2014/3/1 23:52, Marty Sweet wrote:
> >>> Hi Tomas,
> >>>
> >>> Re-reading your original messages: If the system is experiencing lots
> >>> of IO it will appear as wait %, this can also be seen by processes
> >>> excessively in D state.
> >>> It appears (looking at kern.log) that processes are getting blocked
> >>> from disk, which obviously isn't ideal.
> >>>
> >>> Does this happen at set intervals?
> >>>
> >>> I have greped out the D state processes here: 
> http://pastebin.com/b04bgHbL
> >>>
> >>> Could you run $ dstat and send a 10 lines when the issue occurs. This
> >>> will allow us to tell the traffic being written to disk / read from
> >>> network and the loads of the system.
> >>>
> >>> Marty
> >>>
> >>> On Sat, Mar 1, 2014 at 4:16 AM, Tang Jianwei <myhnet at gmail.com> wrote:
> >>>> Hi Marty,
> >>>>
> >>>> thank you very much for your reply and the following is the files 
> you are
> >>>> asking
> >>>>
> >>>> /var/log/kern.log (should be the same as what you were asking)
> >>>> http://pastebin.com/3fC1qxrq
> >>>>
> >>>> /var/log/dmesg
> >>>> http://pastebin.com/G059cxXN
> >>>>
> >>>> output of the commands
> >>>> http://pastebin.com/6faPXt43
> >>>>
> >>>> And thank you very much in advance for your great help.
> >>>>
> >>>>
> >>>> Best Regards
> >>>> Tomas
> >>>>
> >>>> On 2014/2/28 20:00, ubuntu-users-request at lists.ubuntu.com wrote:
> >>>>> Message: 3
> >>>>> Date: Fri, 28 Feb 2014 08:00:31 +0000
> >>>>> From: Marty Sweet <msweet.dev at gmail.com>
> >>>>>
> >>>>> To: "Ubuntu user technical support, not for general discussions"
> >>>>> <ubuntu-users at lists.ubuntu.com>
> >>>>> Subject: Re: IO Wait very high and real IO speed is very low
> >>>>> Message-ID:
> >>>>>
> >>>>> <CALoaWF5ACbWc=wwEiv4WorQZu0gpSEpk0UUSLMX6QhqnO1-m+w at mail.gmail.com>
> >>>>> Content-Type: text/plain; charset=ISO-8859-1
> >>>>>
> >>>>>
> >>>>> Hi Tomas,
> >>>>>
> >>>>> Could you provide the output of /var/log/kernel.log and 
> /var/log/dmesg
> >>>>> and $ df -h and $ fdisk -l ?
> >>>>> When the problem occurs, can you also run $ ps aux ?
> >>>>>
> >>>>> Once you have done this please post it on pastebin and send the 
> link,
> >>>>> as there will be a few hundred lines.
> >>>>>
> >>>>> Thanks,
> >>>>> Marty
> >>>>>
> >>>>> On Fri, Feb 28, 2014 at 6:00 AM, Tang Jianwei <myhnet at gmail.com> 
> wrote:
> >>>>>> Hi All,
> >>>>>>
> >>>>>>        recently I get a very weird problem from my Ubuntu 
> Server 1204
> >>>>>> LTS
> >>>>>> 64bit:
> >>>>>>        1. the write speed can reach 500MB/s when there is no other
> >>>>>> processes
> >>>>>> running
> >>>>>>        2. the write speed down to around 10MB/s when there are many
> >>>>>> other
> >>>>>> processes and meanwhile:
> >>>>>>                a. only very high usage from io wait
> >>>>>>                b. sys, user and all the other cpu usages are 
> very low
> >>>>>>                c. the system load average can reach to 15 and 
> even more
> >>>>>> higher
> >>>>>>
> >>>>>>        can any one give me some advice what is wrong with my 
> system, and
> >>>>>> what I
> >>>>>> should to adjust in hardware/software?
> >>>>>>
> >>>>>>        the platform is:
> >>>>>>        MB: Intel S3240GPLC
> >>>>>>        CPU: I3-540
> >>>>>>        RAM: Kingston 4G ECC
> >>>>>>        Storage: LSI MegaRaid 9270-8i, Raid 5 with 5 drives + 1 
> spare,
> >>>>>> ForcedWB
> >>>>>> with no BBU
> >>>>>>        Drive: WD 3T
> >>>>>>
> >>>>>>
> >>>>>>        thank you very much.
> >>>>>>
> >>>>>>
> >>>>>> Best Regards
> >>>>>> Tomas
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> ubuntu-users mailing list
> >>>>>> ubuntu-users at lists.ubuntu.com
> >>>>>> Modify settings or unsubscribe at:
> >>>>>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
> >>>>>
> >>>>>
> >>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/ubuntu-users/attachments/20140303/961a7032/attachment.html>

More information about the ubuntu-users mailing list