I only have this problem with VBR mp3s and weird formats (like wavpack or musepack). I guess it's mostly caused by either non-optimal seeking in the plugins or limited seeking ability in the file format (as in the mp3 case) -- [Dapper] Seek bar sometimes shows erratic behaviour https://launchpad.net/bugs/27939