[Bug 1190986] Re: ERROR Nonce already used

[Greg Lutostanski]

** Description changed:

+ 
+ SRU for secondary fix (branch lp:~julian-edwards/maas/nonce-fix-bug-1190986)
+ [Impact]
+  * Minimize possibility of hitting a nonce collision by using a longer nonce (uuid4) in both python-maas-client and commissioning scripts.
+ 
+ [Testcase]
+  * Do some api calls and make sure that the piston_nonce postgres table is using the larger uuid nonce for the calls
+ 
+ [Regression Potential]
+  * Minimal; replace random string of length 8 with uuidgen.
+ 
+ 
  The error below happens sometimes. (~5% of all requests fail)
  
  System: Ubuntu 12.04.2
  MAAS-version: 1.2+bzr1373+dfsg-0ubuntu1~12.04.1
  Juju-version: 0.7+bzr628+bzr631~precise1
  
  $ juju status
  2013-06-14 13:59:36,213 INFO Connecting to environment...
  2013-06-14 13:59:36,295 ERROR Cannot connect to environment: Nonce already used: 55941407
  Traceback (most recent call last):
-   File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 43, in run
-     client = yield self._internal_connect(share)
-   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
-     result = result.throwExceptionIntoGenerator(g)
-   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
-     return g.throw(self.type, self.value, self.tb)
-   File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 59, in _internal_connect
-     candidates = yield self._provider.get_zookeeper_machines()
-   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
-     result = result.throwExceptionIntoGenerator(g)
-   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
-     return g.throw(self.type, self.value, self.tb)
-   File "/usr/lib/python2.7/dist-packages/juju/providers/common/findzookeepers.py", line 32, in find_zookeepers
-     machine = yield provider.get_machine(instance_id)
-   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
-     result = result.throwExceptionIntoGenerator(g)
-   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
-     return g.throw(self.type, self.value, self.tb)
-   File "/usr/lib/python2.7/dist-packages/juju/providers/maas/provider.py", line 125, in get_machines
-     instances = yield self.maas_client.get_nodes(instance_ids)
-   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
-     current.result = callback(current.result, *args, **kw)
-   File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 95, in _process_error
-     raise ProviderError(error.response)
+   File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 43, in run
+     client = yield self._internal_connect(share)
+   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
+     result = result.throwExceptionIntoGenerator(g)
+   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
+     return g.throw(self.type, self.value, self.tb)
+   File "/usr/lib/python2.7/dist-packages/juju/providers/common/connect.py", line 59, in _internal_connect
+     candidates = yield self._provider.get_zookeeper_machines()
+   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
+     result = result.throwExceptionIntoGenerator(g)
+   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
+     return g.throw(self.type, self.value, self.tb)
+   File "/usr/lib/python2.7/dist-packages/juju/providers/common/findzookeepers.py", line 32, in find_zookeepers
+     machine = yield provider.get_machine(instance_id)
+   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1037, in _inlineCallbacks
+     result = result.throwExceptionIntoGenerator(g)
+   File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 382, in throwExceptionIntoGenerator
+     return g.throw(self.type, self.value, self.tb)
+   File "/usr/lib/python2.7/dist-packages/juju/providers/maas/provider.py", line 125, in get_machines
+     instances = yield self.maas_client.get_nodes(instance_ids)
+   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
+     current.result = callback(current.result, *args, **kw)
+   File "/usr/lib/python2.7/dist-packages/juju/providers/maas/maas.py", line 95, in _process_error
+     raise ProviderError(error.response)
  ProviderError: Nonce already used: 55941407
  2013-06-14 13:59:36,297 ERROR Nonce already used: 55941407
  
  I think the problem is caused by the MAAS-database table "piston_nonce"
  In my system this table contains a hughe amount of tokens that are never deleted:
  maasdb=# SELECT count(id) FROM piston_nonce;
-   count  
+   count
  ---------
-  3619099
+  3619099
  (1 row)

** Summary changed:

- ERROR Nonce already used
+ [SRU] ERROR Nonce already used

-- 
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to juju in Ubuntu.
https://bugs.launchpad.net/bugs/1190986

Title:
  [SRU] ERROR Nonce already used

To manage notifications about this bug go to:
https://bugs.launchpad.net/juju-core/+bug/1190986/+subscriptions