[storm] No exception raised when hitting database locking error

[james at jamesh.id.au]

On Thu, Dec 16, 2010 at 3:25 PM, Steve Kieu <msh.computing at gmail.com> wrote:
>
> Hello all,
>
> It is kind of hard to explain - basically I wrote an application that
> trying to retrieve a set of objects from the db - then opening a config file
> and loop over the lines regex matching and if match, update a field from the
> object. At the end of function - call store.commit()
>
> Run it at the same time in multiple servers to connect to one central
> database (tested with postgre and mysql). The set of object mentioned above
> are a different set (as they run in different servers)
>
> If I use postgres and I often get error - the program quit with an error:
>
> could not serialize access due to concurrent update
>
> I tried to wrap store.commit()  in try and except but it never runs into
> except: it just passed the store.commit() as normal and exited.

It isn't unusual for serialisation errors to be reported prior to
commit: the database usually reports them as soon as it detects the
problem.  The traceback from the error you saw probably tells you
where it occurred.  Try wrapping your try/except block around the
entire transaction logic rather than just the commit() call.

If you're using PostgreSQL, the
psycopg2.extensions.TransactionRollbackError exception should cover
the cases you're interested in.


> The same problem with MySQL with innoDB (different error message though,
> something abt deadlock detected

I would guess MySQL is pretty much the same: report the error as soon
as the problem is detected rather than letting you continue until
commit.


> What surprises me is that why the db server  refuses to work. I am positive
> that they are completely different set and then it should be in different
> rows modifications.
>
> And then storm does not throw exception on this case so I can catch using
> except and try again or do something about it.
>
> The problem is fixed if: I call store.commit() after each object value
> update  (store.flush() is not enough) ; but it is painfully slow
>
>
>
> MyISAM does not have that problem as well

Well MyISAM doesn't support transactions, so you shouldn't expect it
to report transaction serialisation errors ...

James.