prefetch still broken with readv and paramiko 1.6.1

[Robey Pointer]

On 31 Jul 2006, at 11:57, John Arbash Meinel wrote:

>>> I think the reason that paramiko-trunk is so slow is because we  
>>> have a
>>> fairly large queue at that point, and iterating through the queue  
>>> every
>>> time is slow. At least that is my guess. Something weird is going  
>>> on,
>>> considering it effects seek+read performance without prefetch or  
>>> readv
>>> enabled.
>>>
>>> I'm not positive, though.
>>>
>>> This are my results:
>>>
>>> paramiko-trunk readv    98.89
>>> paramiko-trunk seek+read    96.81
>>> request 32k readv    79.75
>>> request 32k seek+read    87.64
>>>
>>> Anyway, it might be possible to rewrite your trunk such that it  
>>> is still
>>> fast, and allows readv and prefetch to operate simultaneously.  
>>> I'm not
>>> 100% positive as to how to do that, though.
>>
>> This is bad.  Can you point me to the code you used to get those  
>> stats?
>> I'd like to dig in further and see if I can figure out what's  
>> going on
>> in paramiko-land.  The queue management when using readv without
>> prefetch ought to be basically the same as before...
>>
>> robey
>
> http://bzr.arbash-meinel.com/branches/real-world-benchmarks/jameinel/
>
> Is the branch where I was doing all of the testing. It really isn't a
> general purpose branch. It is just a bunch of scripts that I use to  
> run
> 'bzr something' multiple times against different branches, and collect
> the timing results.

Thanks, that was enough to get me started.  My macbook doesn't have  
"tc" but just trying it across loopback sftp was enough to show a  
significant drop.  Lsprof to the rescue!  Turned out to be bad logic  
in checking for prefetched buffers, so readv was reading a lot of the  
data twice.

I think this should be fixed in paramiko @382.  I'll probably cut a  
1.6.2 release soon.

robey