Charm push crashes when uploading big charms

Mon Jun 20 18:29:52 UTC 2016

Thanks for the explanation, Jay.

I did some further testing. Charm upload fails for a 270MB Charm both from
my home, my work and our datacenter.

   - The datacenter is connected directly to Belnet (upload bandwith
   ~300Mbit/s).
   - My upload bandwidth at home is ~ 3 Mbps (speedtest.net) although
   during upload, system monitor shows ~400KiB/s.

This causes me to think there is more at play here then large file + slow
internet... Let me know if I can help to further debug this problem.

As an aside; I don't consider 270MB to be that large. Some examples:

   - Kubernetes is ~1G
   - Ubuntu docker base image is ~200MB

I think this is stuff we should be able to handle...

2016-06-20 18:41 GMT+02:00 Jay Wren <jay.wren at canonical.com>:

> Yes, files are broken up into many TCP packets, and they are all
> transmitted over a single TCP connection. TCP is a complex protocol which
> is well documented, so I'll not repeat that here. If you want lots of
> details, wikipedia is not bad:
> https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Protocol_operation
>
> In the abstract, when you connect to a server using TCP it is identified
> by a 4-tuple of source address, source port, target ip address, target
> port. These connections consume server resources and indeed a connection
> exhaustion is a popular denial of service attack.
>
> You are getting a tcp timeout due to of file size because the time it
> takes to send the entire content is longer than the TCP connection timeout.
>
> Yes, the resource upload command to charmstore will also be affected by
> this. Luckily, resources also have the ability to be uploaded specifically
> to a model, which might have greater network data rates from the resource
> uploader.
>
> --
> Jay
>
>
> On Mon, Jun 20, 2016 at 7:04 PM, Merlijn Sebrechts <
> merlijn.sebrechts at gmail.com> wrote:
>
>> Thanks for looking into this! I'll try the compression and see if that
>> works.
>>
>> Just curious; why does filesize affect tcp connection timeout? Aren't the
>> files broken up into a bunch of smaller tcp packets? Filesize should only
>> affect the number of tcp packets, not the size of the tcp packets? So
>> getting a tcp timeout because of filesize seems strange to me... Any idea
>> what exactly goes wrong here?
>>
>> Also, now that I think of it, the resource upload command might also be
>> affected by this, if it uses the same library and similar backend
>> infrastructure? I'll test this out.
>>
>>
>> Op maandag 20 juni 2016 heeft Jay Wren <jay.wren at canonical.com> het
>> volgende geschreven:
>> > Hello Merlijn,
>> > I can replicate the problem and I can work around it by using a faster
>> internet connection.
>> > At some point, tcp connections have to time out. I can only replicate
>> the issue when that timeout is reached. If you have the means to relocate
>> to a faster internet connection temporarily for pushing to charmstore,
>> please do. You might also try recompressing any items in the charm using a
>> higher compression level. xz -9 instead of gzip -3 or whatever things may
>> be using now.
>> > We are aware this is a poor longterm solution. We are investigating
>> better solutions for uploads. As you've mentioned, resources will also help
>> the situation.
>> > I am sorry that I do not have a better solution.
>> > --
>> > Jay
>> > On Fri, Jun 17, 2016 at 4:29 PM, Rick Harding <
>> rick.harding at canonical.com> wrote:
>> >>
>> >> Merlijn, thanks. I'm going to bet there's an issue with http request
>> sizes for the charmstore that the charm command talks do as we've got some
>> layers (Apache, Squid) in front of the actual application. The team is
>> looking into it. Thanks for giving us the heads up.
>> >> On Wed, Jun 15, 2016 at 10:28 AM Merlijn Sebrechts <
>> merlijn.sebrechts at gmail.com> wrote:
>> >>>
>> >>> Hi all
>> >>>
>> >>> I've hit a roadblock in setting up my CI pipeline. I have a charm
>> with a Java sdk blob of ~200MB. Pushing this charm to the store causes the
>> tool to crash. I put a bug report here:
>> https://bugs.launchpad.net/juju/+bug/1592822
>> >>> Juju resources would fix my problem, but then I'd need to move to
>> Juju 2.0 and I'm not ready to do that yet.
>> >>>
>> >>>
>> >>> Kind regards
>> >>> Merlijn Sebrechts
>> >>> --
>> >>> Juju mailing list
>> >>> Juju at lists.ubuntu.com
>> >>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>> >>
>> >> --
>> >> Juju mailing list
>> >> Juju at lists.ubuntu.com
>> >> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/juju
>> >>
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ubuntu.com/archives/juju/attachments/20160620/edfb60e1/attachment.html>