[Bug 1366997] [NEW] Duplicate entry error for primary key following cluster size change

[Launchpad Bug Tracker]

You have been subscribed to a public bug by Robie Basak (racb):


[Impact] 

 * Duplicate entry errors on the primary key for tables occurs, data
   can't be added to DB correctly and in time.

[Test Case]

 * Prepare a .sql file full of INSERT queries:

   INSERT INTO test.t VALUES(NULL,"miguel");

   that line million of times.

 * Start a 3 node clusters.

 * Load the data on node 1 (if you do several times in parallel you will
get the error sooner): cat data.sql | mysql

 * Restart node2 and wait until MySQL is running.

 * Restart node3 and wait until MySQL is running.

 * Repeat step 4 and 5 several times (sometimes lot of restarts are
needed, sometimes just a few) and the node1 load will fail with
duplicate key entry.

[Regression Potential]

 * The patch to fix this issue is backported from a nearest version 5.6.

[Other Info]


We (PagerDuty) have experienced on multiple occasions duplicate entry errors on the primary key for some tables after our cluster's size changes. In the cases we have experienced so far, this is when gracefully adding or removing a node to vertically scale the nodes in the cluster. We don't experience a total failure of all transactions. Instead, a small subset error out. In some cases, the problem corrects itself, but several times now we have had the errors persist for several hours. When the problem persists, we have found that restarting one of the cluster members will sometimes fix the issue.

Restarting all transactions by rebooting the application does not fix
the issue. Only a restart of a cluster member can eventually solve the
problem.

Here are the packages we are running

ii  percona-toolkit                   2.2.7                             Advanced MySQL and system command-line tools
ii  percona-xtrabackup                2.1.9-744-1.lucid                 Open source backup tool for InnoDB and XtraDB
ii  percona-xtradb-cluster-client-5.5 5.5.37-25.10-756.lucid            Percona XtraDB Cluster database client binaries
ii  percona-xtradb-cluster-common-5.5 5.5.37-25.10-756.lucid            Percona XtraDB Cluster database common files (e.g. /e
ii  percona-xtradb-cluster-galera-2.x 188.lucid                         Galera components of Percona XtraDB Cluster
ii  percona-xtradb-cluster-server-5.5 5.5.37-25.10-756.lucid            Percona XtraDB Cluster database server binaries

Here is an example error that we see.

Duplicate entry '623287' for key 'PRIMARY'

That error will be seen for multiple tables, and sometimes again for the
same table.

Please let me know what additional information can help. The MySQL error
log doesn't appear to have any interesting details in it (just node join
and leave events) but I would be glad to pass it along if you think it
would be helpful.

** Affects: codership-mysql
     Importance: Undecided
         Status: Confirmed

** Affects: codership-mysql/5.5
     Importance: Undecided
         Status: Confirmed

** Affects: codership-mysql/5.6
     Importance: Undecided
         Status: Confirmed

** Affects: percona-xtradb-cluster
     Importance: Undecided
         Status: Fix Released

** Affects: percona-xtradb-cluster/5.5
     Importance: Undecided
     Assignee: Alexey Kopytov (akopytov)
         Status: Fix Released

** Affects: percona-xtradb-cluster/5.6
     Importance: Undecided
         Status: Fix Released

** Affects: percona-xtradb-cluster-5.5 (Ubuntu)
     Importance: Undecided
         Status: Confirmed

** Affects: percona-xtradb-cluster-5.5 (Ubuntu Trusty)
     Importance: Undecided
         Status: New

** Affects: percona-cluster (Juju Charms Collection)
     Importance: Undecided
     Assignee: Edward Hope-Morley (hopem)
         Status: In Progress


** Tags: cts
-- 
Duplicate entry error for primary key following cluster size change
https://bugs.launchpad.net/bugs/1366997
You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to the bug report.