API updates

[Ian Booth]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



On 29/05/13 21:24, John Arbash Meinel wrote:
> Ian, William, and I discussed the changes to how we implement API
> requests. And I think we sorted out a few points. Hopefully they agree
> that this is what we agreed on. :)
> 
> 1) The server side of the APIs should all support batch requests. (eg,
> we shouldn't have Unit.Get(one), but Units.Get([]many).)
>

+1


> 2) Client side, we'll still write a lot of stuff as
>    myunit = client.Unit.Get(one)
> 
> As most clients will be concerned with only 1 thing
> (Unit/Machine/Agent, etc).
>

Cautious +1


> 3) We aren't going to put versions into the API today. Instead, we
> expect that when we have to update (MVU) we'll insert them to the
> names. So something like Units.Get() will probably become Units.GetV2()
> 
> This will likely be versioned per-request, rather than versioning the
> API as a whole.
> 
> Lots of stuff not discussed here (how do we deprecate old things,
> eventually removing them, etc.)
>

Part of me still wishes that the API version be embedded in the request to keep
the method names "clean". I'd like to think we can still discuss this when the
time comes to implement it.


> 4) API requests will generally *not* return full objects as responses.
> (eg Machine.AddUnit()[0] won't return a full representation of the
> machine with the new unit added, and the dirty flag changed, and
> whatever other records that might have been changed by a 3rd party)
>

I disagree with this approach but sadly for me may have lost the argument here.
Given we are not implementing a fully stateless design, the potential for stale
data to cause issues is real, and complicates the programmimg model. We can
alleviate this somewhat by API requests returning the full objects as responses.
The added overhead is not high - a few bytes on the wire. We are making a round
trip anyway to invoke the API, and the API returns some data to the caller eg
error/success. So it is not too unreasonable to augment the returned data with
the new object state.


> 5) It is reasonable for the client side to update fields on the
> in-memory structure that it knows are modified as a side-effect of the
> API it called (eg Machine.AddUnit knows to set dirty=true locally,
> even though it isn't a Machine.SetDirty call). However, it is expected
> to *not* modify fields that aren't obviously related (after calling
> AddUnit, it shouldn't have the side effect of seeing a new
> Machine.Name value.)
> 

I have a real problem with this. If by "client side" you mean remote object in a
distributed system sense of the word, that means we are having to write the same
state manipulation logic in 2 places - the client proxy and the server in-memory
model. The client has no right to know what fields it needs to update as a
result of calling an API - that's the service's job. And it becomes moot if we
return the new object state when we make an API call. See point 4 above.

> 6) *Most* API calls are going to be field-level changes, with
> field-level transactional guarantees, rather than object-level
> guarantees.
> 

So it seems.

> eg, Machine.ChangeName(old, new) should probably take the old value
> and the new value, and do a DB transaction that asserts name is still
> old before setting it to new. However, Machine.AddUnit doesn't need to
> lock the whole Machine object in order to add a unit. Thus
> Machine.ChangeName from client A will conflict with a similar request
> from client B, but will *not* conflict with a call to Machine.AddUnit
> from client C.
> 

Yes, for field level changes, old/new make sense. For object level changes, an
object level version field makes more sense. So long as we have *some* form of
optimistic locking pattern in use, I think we will be ok.

> 7) There will be some logic server side to handle concurrency, and
> conflict resolution occurs in the API server as much as possible.
> 

+1

> 
> 8) It is expected that the primary mechanism for keeping clients up to
> date is actually Watchers.
> 
> eg: Machine.AddUnit won't notice that the Name changed. Instead, a
> client interested in Machine-0 needs to have a watcher on Machine-0,
> when something (such as name or units) changes on Machine-0 the
> watcher will get a notification "You need to refresh your information
> about Machine-0", and then the client can use Machine.Refresh() to
> actually get that information.
>

We need to be *very* careful about race condition and order of updates and other
such things with this approach. If a Machine.AddUnit() call results in fields of
that machine changing as a direct result of that API call, I really feel the
client memory model should immediately reflect those changes, which implies we
return the new state when making the call (see points 4,5). If we eschew this
approach, it will cause problems. Believe me. I've been there and it's not
pretty and really hard to debug.


> 9) Clients can be optimistic, but should be defensive. They can assume
> that their local information is correct (Machine.name is still the
> same), but should be prepared to get failures and retry with correct
> information.
> This is somewhat similar to ETAFFTP vs LYBL (easier to ask for
> forgiveness than permission, vs look-before-you-leap).
> 

+1

> 
> 10) If there are transactional requirements, these are taken care of
> in the state code (by doing assertions in the DB transaction
> generally). So if something needs to handle multiple-field
> correctness, then it should happen at this level.
> 

+1

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iJwEAQECAAYFAlGl9FcACgkQCJ79BCOJFcYt1gQAmrZhVePSnA8GF5pKomG5P3q6
OJ94f9L1Hsa+XAgE+Eq2G3gM3UOLfj5kw7tAfRSZm7LI0kyMCd1rkJCY+IHpeuuB
113bc1XjIA6fXrYsq5+k9KBANc3VtAP7duImKbtvAIWOXtqH3qV6m3Xw6dMvzcm3
ZHWZmUWvM5wLsOBMzBg=
=3rkZ
-----END PGP SIGNATURE-----