[ec2-beta] ec2 & upgrade testing

[simone brunozzi]

Thanks Eric,
that's quite an impressive list of tips, as always :)

Cheers,

Simone

<------------------------------------>
Simone Brunozzi
Amazon Web Services Evangelist - simoneb at amazon.com
Work: Mob +352 691-999-967 / Office +352 26-73-34-30
personal, EU: +39 340-57-68-488 - US: +1 206-499-3408
18 Rue Michel Welter - L-2790 Luxembourg
http://www.brunozzi.com
http://twitter.com/simon
http://www.linkedin.com/in/simonebrunozzi
<------------------------------------>


On Mon, Jan 19, 2009 at 4:04 AM, Eric Hammond <ehammond at thinksome.com>wrote:

> Michael:
>
> Here are the ideas I was thinking about that might help you get past the
> 10G limit on the root partition:
>
> (1) Run a base image.  On the running instance, copy some of the bigger
> directories to /mnt and then mount them back over top of the original
> location.  Install all the packages you need, then test the upgrade
> process.  E.g.,
>
>   for i in /usr /var/cache /var/lib; do
>     sudo rsync -PaR $i /mnt/
>     sudo mount --bind /mnt$i $i
>   done
>   apt-get install ...
>
> Pros/cons:
>
>   + Easy
>   - Process must be repeated every time you want to test upgrade
>   - Can't relocate directories which have open files (reboot?)
>
>
> (2) Same as (1) but instead of using /mnt (ephemeral storage), the
> directories could be copied to an EBS volume (persistent storage) and
> clean installs performed.  Take a snapshot of the EBS volume and bundle
> the rest of the root partition as a new AMI.  Figure out a way to start
> new instances with the AMI + new EBS volume created from the snapshot.
>
> Pros/cons:
>
>   + Faster to start up fresh systems (if it's even possible)
>   - More difficult to set up
>   - Must coordinate AMI + snapshot to be consistent
>   - Not sure if the AMI portion will contain tools necessary to mount
>     EBS volumes so this approach may not be feasible.
>
>
> (3) There has been some progress in the exploration of storing an entire
> root partition on an EBS volume of any size and using pivotroot at
> startup.  Here are some threads which talk about this:
>
> http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24091
> http://developer.amazonwebservices.com/connect/thread.jspa?threadID=24440
>
> Pros/cons:
>
>   + Everything on a single mount point
>   + EBS snapshot captures entire consistent install
>   + Faster to start up fresh systems
>   - Setup requires the use of advanced methods
>
>
> Let me know if you'd like assistance pursuing one of these or other
> alternatives.
>
> --
> Eric Hammond
> ehammond at thinksome.com
>
>
>
> Michael Vogt wrote:
> > On Fri, Jan 16, 2009 at 11:13:13PM -0800, Eric Hammond wrote:
> >> Michael:
> > Hi Eric,
> >
> >> There are a few ways to get more than 10 GB effectively or actually on a
> >> root file system on a running EC2 instance.
> >>
> >> Depending on your needs and the level of complexity you're willing to
> >> deal with, the data can even be persistent and/or can be used as an
> >> image to start multiple instances (though not exactly an AMI).
> >>
> >> Can you share more about what goals you are trying to accomplish and the
> >> processes you use?
> >
> > Currently I use the vmbuilder to create the AMI and have python code
> > that runs a instance based on this, installs additional packages into
> > it (ubuntu-desktop for example). Then it copies the current release
> > upgrader code into the instance and upgrades to the next version of
> > ubuntu to see if there are any issues with the package upgrades or the
> > upgrader code.
> >
> > What I would like do be able to do is have images/AMIs for the various
> > ubuntu flavours (ubuntu,kubuntu,..) available so that the upgrade test
> > does not have to construct them again. I would also like to run
> > upgrade tests with a lot of packages (e.g. most of main or even most of
> > universe). That can easily go much beyond the 10G limit. Having a
> > snapshoting ability would also be nice but its not that important
> > right now.
> >
> > I hope the above gives a bit of background, any hints/links to more
> > information how to get around the limit are much appreciated (I'm
> > still pretty new to ec2).
> >
> > Thanks,
> >  Michael
> >
> >
> >> Robbie Williamson wrote:
> >>> Great work, Michael!  I'm copying the ec2-beta list, as people on it
> will be
> >>> interested and can also help you, if needed. ;)
> >>>
> >>> -Robbie
> >>>
> >>> On 01/16/2009 08:56 AM, Michael Vogt wrote:
> >>>> Hi,
> >>>>
> >>>> I worked on the ec2 upgrade testing stuff today and yesterday and I
> >>>> have a working prototype now in the lp:~mvo/update-manager/ec2
> >>>> branch. It can test upgrades in a ec2 instance in a similar fashion as
> >>>> the current kvm backend. I created the base images with vmbuilder
> >>>> (from my lp:~mvo/vmbuilder/mvo branch that contains some fixes against
> >>>> trunk/).
> >>>>
> >>>> I did a successful automatic/unattended-upgrade for the server and the
> >>>> ubuntu-desktop profile. But my code is still prototypish and needs
> >>>> some cleanup love.
> >>>>
> >>>> One pretty anoying limiation currently seems to be that the size of a
> >>>> image (a AMI) seems to be limited to 10Gb [1]. That is not enough for
> a
> >>>> full blown upgrade test that includes most packages in main (it is
> >>>> sufficient for our various default installs).
> >>>>
> >>>> Its a good addition to the kvm based backend but its still useful to
> >>>> keep the kvm one around. Its nice to be able to boot a kvm upgraded
> >>>> image and login to test stuff manually (or verify upgrade bugs).
> >>>>
> >>>> There is still a lot of room for improvements in the ec2 backend. I
> >>>> just use a single base AMI right now for everything, this should be
> >>>> changed so that each base image becomes its own ami. I also don't make
> >>>> any use of the ec2-volumes yet (but I'm not quite sure yet if they are
> >>>> useful for my use-case). I'm still learning how everything fits
> >>>> together :)
> >>>>
> >>>> Cheers,
> >>>>  Michael
> >>>>
> >>>>
> >>>>
> >>>> [1]
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1145
> >>>> (question: "When I try to bundle my AMI, it fails. Why?")
> >>>
> >
>
> --
> Ec2-beta mailing list
> Ec2-beta at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ec2-beta
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/mailman/private/ec2/attachments/20090119/053e8250/attachment-0001.htm