Merging changes for Bazaar beyond the default python

[Martin (gzlist)]

(I'm adding CC: jython-dev as Bazaar+Jython issues may also interest them)

On 02/11/2009, Ian Clatworthy <ian.clatworthy at canonical.com> wrote:
> Martin (gzlist) wrote:
>
>> <https://code.launchpad.net/~gzlist/bzr/noncpython>
>>
>> As may be guessed from the name, it also works on Jython.
>
> That's really exciting.
>
>> I may post
>> some follow up messages with additional details on some specific
>> issues.
>
> Please do.

Here's an overview of the (relatively few) changes needed to get
Bazaar into a semi-working state on Jython. Many of the issues were
the same as with IronPython, over platform detection, file lifetimes,
and absent binary modules. On the positive side, Jython is faster, and
supports a larger subset of the python standard library. However, it's
less compatible and a few key problems mean unlike IronPython, it's
not yet possible to use it to do useful work.

This revision on the branch has the majority of Jython specific
changes if anyone wants more detail on the workarounds mentioned
further down:

<http://bazaar.launchpad.net/~gzlist/bzr/noncpython/revision/4711>


The main deal-breaker at the moment is that the Jython zlib module
does not have a Z_SYNC_FLUSH flag. More generally, it does not support
any way of flushing the stream, except when closing it. This is
apparently a longstanding limitation of java.util.zip:

<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4206909>

There were discussions when bzrlib.chunk_writer was being written
about alternate ways of getting compressed chunks of the desired size,
including copying the Compress object:

<https://lists.ubuntu.com/archives/bazaar/2008q3/043779.html>

As far as I can tell, that won't work for Jython either, which doesn't
have Python 2.5's copy method.


Bazaar expects to be able to tell a lot from exceptions thrown by os
functions, even without reference to the original call. This doesn't
work with the Java IO layer, which provides less detail than errno
does for C:

    >>> from java.io import File
    >>> File("nonexistant").renameTo(File("whatever"))
    False

So, for os.rename with an absent source file Jython gives an errno of
0 rather than ENOENT. These issues don't seem robustly fixable.

Methods like file.fileno return a Java filestream object, which is
rather neat (and much better than making up an integer). However,
there is no os.fstat and I don't see a way of getting back to the name
of the file. Fortunately not hard to get round, because every use of
fstat in Bazaar also has either a file object or name nearby anyway.

Jython seems prone to falling over in odd ways whenever the filesystem
and unicode meet. To be fair, it does have
os.path.supports_unicode_filenames as False, but given that
java.io.File and java in general are happy with unicode, this is
surprising.

The terminal encoding is given wrong. Looks like another last
millennium Java bug:

<http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4170549>


And finally, the string-escape codec doesn't escape the escape
character, so can't round trip.


I wrote a few tests to keep track as I went along, the curious may
check them out using:

    bzr branch http://float.endofinternet.org/bazaar/simple/test_noncpython/

(add test_noncpython.py to the end of that URL to just see the script)


One thing I've not tracked down yet is when running selftest, Jython
gets slower and slower and eventually dies a messy OOM death. For
instance, these transform tests towards the end of the alphabet go
from taking half a second:

<http://float.endofinternet.org/temp/bazaar_j_noncpython_bt.test_transform#bt.test_transform.TestTransformRollback>

To nearly ten seconds when lots of other tests are run first:

<http://float.endofinternet.org/temp/bazaar_j_noncpython_bt.test_#bt.test_transform.TestTransformRollback>

Click on some of the other coloured boxes for more info. I have a
vague feeling this may be to do with http server thread leaking, but
that's just guessing.


I welcome comments and corrections on this post or any of the code.

Martin