[ec2-beta] data corruption

[Chuck Short]

Hi Ben,

Can you describe how you were setup so I can try reproducing the problem 
here?

Thanks
chuck

Ben Hendrickson wrote:
> On Tue, Apr 14, 2009 at 4:10 PM, Eric Hammond <ehammond at thinksome.com> wrote:
>   
>> Ben, it wasn't quite clear to me exactly how many instances you were
>> running for how long on each of the different images and how many
>> failures you experienced.  I'm trying to figure out if the failure rate
>> is statistically significant or if it might simply be an
>>     
>
> About 85% of the time I have 16-20 large instances running, and 15% of
> the time I have 0 machines running.  We switched to the beta images
> towards the end of January, and switched back to the Alestic images on
> March 29th.  So we were running the beta images for 8 weeks.  I don't
> actually have a record when I upgraded from Beta 1 to Beta 2, but I
> think I switched to the Beta 1 in early March, and afterward I
> continued to see corruptions at roughly the same pace.  I didn't keep
> track of exactly how many corruptions I saw, but roughly 12.  Most of
> the corruptions were on different machines (I started swapping
> machines out as soon as they had their first corruption).  Thinking it
> might be a data center issue, I swapped availability zones, but
> continued to see corruptions.
>
> Regarding time with the Alestic AMIs, I had been using the Alestic
> images for several months prior to switching to the beta, and since
> March 29th I've had 20 machines running continually with the Alestic
> images.  And during the time with the Alestic images, there have been
> zero corruptions.  Currently I'm using the 8.10 Alestic AMI
> (ami-bcfe19d5), but previously was using the Alestic 8.04 AMI without
> issue.
>
> On Tue, Apr 14, 2009 at 12:54 PM, Mark Shuttleworth <mark at ubuntu.com> wrote:
>   
>> Is ReiserFS integral to the solution, or a personal preference? It jumped
>> out at me as an area of risk, as it's not a filesystem we're particularly
>> focused on. Ext3, and the newer ext4 and ultimately btrfs would be the
>> "stable, next, future" default filesystems we'd recommend unless there was a
>> specific technical reason to do otherwise. If Reiser isn't integral I'd be
>> interested in your results with ext3, both performance and stability wise.
>>     
>
> I selected Reiser prior to the guilty verdict and before it was clear
> to me developers were backing away from it.  I wouldn't choose it now,
> but at the time, I went with it because it had a slight performance
> advantage when I benchmarked our code on it vs ext2.  I was thinking
> to change over to ext4 when that is available, as we do have a lot of
> large files which the extents might be good for.  Eric's comment he
> uses XFS suddenly makes that more attractive.
>
> For the record, I really appreciate your guy's work.  It is great
> having up-to-date Ubuntu AMIs.
>
> Ben
>
>