resource maps

Tue May 8 17:53:10 UTC 2012

Hey all

Following on from the UDS discussion on Monday morning [0], I footnote
an example of how I might expect a resource map to look for us-east-1 on
AWS [1].

It's very clearly AWS-focused, but I'm not sure that it's *necessarily*
AWS-*specific* -- that is, I don't think that there's anything in there
that couldn't translate to other cloud providers. If I'm wrong about
that in particular, please complain loudly [2].

Specific things to notice:

* The images' "hvm" field defaults to "false", and "ebs" defaults to
"true", but on reflection I think I'd prefer to eschew default values
entirely and require that all values be explicit.

* The instance-type fields which need to be cross-referenced with image
fields (arch, ebs-image, hvm-image) are all lists rather than naked
values. In the first two cases, this is clearly necessary because Amazon
itself already allows for variation in those fields per instance-type; 
and in the final case, it seems sensible because I don't believe there's 
anything fundamentally different about HVM to lead it to be classified
differently (that is, I can readily imagine a cloud provider that
allowed you to run a particular instance-type either with or without
HVM).

* The "cost" field happens to mean dollars-per-hour -- as published by
Amazon, excluding considerations of EBS storage cost, etc -- but doesn't
*necessarily* have that meaning: on a private cloud, for example, it may
be very difficult to come up with precise costs, but one will still want
some means of ranking instance types. (Note: I don't believe it makes a
great deal of sense for Juju to impose a computed "cost" ranking on
different instance types: while it's pretty easy to assume that larger
cpu and mem values cost more, it's not really sensible to globally
impose our own assumptions about the relative cost weighting of those
two constraints.)

* instance-store defaults to 0, but t1.micro is the only instance type 
that doesn't have instance storage, so that's not immediately obvious.

* As discussed, Juju would expect a "resource-uri" environment setting
to point to something like this; in EC2, and potentially other public
clouds, you'd leave it blank to get the standard data (that would be)
published by Canonical, but you'd provide your own data for private
clouds.

* I'm not sure that libcloud is going to be as helpful as we'd hoped. It
seems to be missing some important information, such as valid-arches-per-
instance-type and required-image-kinds-per-instance-type; and I'm a little 
confused about what exactly "disk" is meant to mean (t1.micro has no instance 
storage, but has a "disk" of 15; m1.small has a "disk" of 160, which apparently 
refers to its instance storage...). Can someone who knows libcloud better
than me confirm that it's not offering what I'm suggesting here? (Or perhaps, 
ofc, point out where I'm crackful ;).)

I hope this helps to crystallise the discussion a little bit...

Cheers
William

[0]
http://summit.ubuntu.com/uds-q/meeting/20509/servercloud-q-juju-resource-map/ 

[1]

-------------------------------------------------------------
images:
  ami-4bad7422: {arch: i386, ubuntu-series: oneiric}
  ami-4dad7424: {arch: amd64, ubuntu-series: oneiric}
  ami-4fad7426: {arch: amd64, hvm: true, ubuntu-series: oneiric}
  ami-61ea3408: {arch: i386, ebs: false, ubuntu-series: precise}
  ami-8baa73e2: {arch: amd64, ebs: false, ubuntu-series: oneiric}
  ami-b5ea34dc: {arch: amd64, ubuntu-series: precise}
  ami-b7ea34de: {arch: amd64, hvm: true, ubuntu-series: precise}
  ami-bdea34d4: {arch: i386, ubuntu-series: precise}
  ami-e1aa7388: {arch: i386, ebs: false, ubuntu-series: oneiric}
  ami-ebea3482: {arch: amd64, ebs: false, ubuntu-series: precise}
instance-types:
  c1.medium:
    arch: [i386, amd64]
    cost: 0.165
    cpu: 5
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 358400
    mem: 1740
  c1.xlarge:
    arch: [amd64]
    cost: 0.66
    cpu: 20
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 1730560
    mem: 7168
  cc1.4xlarge:
    arch: [amd64]
    cost: 1.3
    cpu: 33.5
    ebs-image: [true, false]
    hvm-image: [true]
    instance-store: 1730560
    mem: 23552
  cc2.8xlarge:
    arch: [amd64]
    cost: 2.4
    cpu: 88
    ebs-image: [true, false]
    hvm-image: [true]
    instance-store: 3450880
    mem: 61952
  cg1.4xlarge:
    arch: [amd64]
    cost: 2.1
    cpu: 33.5
    ebs-image: [true, false]
    hvm-image: [true]
    instance-store: 1730560
    mem: 22528
  m1.large:
    arch: [amd64]
    cost: 0.32
    cpu: 4
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 870400
    mem: 7680
  m1.medium:
    arch: [i386, amd64]
    cost: 0.16
    cpu: 2
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 419840
    mem: 3840
  m1.small:
    arch: [i386, amd64]
    cost: 0.08
    cpu: 1
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 163840
    mem: 1740
  m1.xlarge:
    arch: [amd64]
    cost: 0.64
    cpu: 8
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 1730560
    mem: 15360
  m2.2xlarge:
    arch: [amd64]
    cost: 0.9
    cpu: 13
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 870400
    mem: 35020
  m2.4xlarge:
    arch: [amd64]
    cost: 1.8
    cpu: 26
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 1730560
    mem: 70040
  m2.xlarge:
    arch: [amd64]
    cost: 0.45
    cpu: 6.5
    ebs-image: [true, false]
    hvm-image: [false]
    instance-store: 430080
    mem: 17510
  t1.micro:
    arch: [i386, amd64]
    cost: 0.02
    cpu: 0.1
    ebs-image: [true]
    hvm-image: [false]
    mem: 613
-------------------------------------------------------------

Note also that this was generated from (1) cloud-images.ubuntu.com and
(2) a handwritten list of instance-types and costs per region. There's
nothing stopping me from collecting the data from those sources at
runtime (and so there would be nothing stopping us from writing custom
code per provider to dynamically collate whatever's available from the
local APIs with whatever can't be got from them); but that feels to me
like a potential future enhancement rather than a strict necessity at
this stage.

[2] Ofc, please complain about anything else I may also be wrong about;
but I'm most concerned with exposing a sensible set of
image/instance-type properties that make some sort of sense across
providers. For example, I'm not bothered if some particular cloud *only*
handles EBS images -- we can accommodate that -- but I very much am
bothered if I'm exposing properties that are just plain crazy (in some
way, in some cloud).