[Bug 1521958] Re: rabbit: starvation of connections for reply

[Corey Bryant]

** Also affects: python-oslo.messaging (Ubuntu)
   Importance: Undecided
       Status: New

** No longer affects: python-oslo.messaging (Ubuntu Trusty)

** No longer affects: python-oslo.messaging (Ubuntu Vivid)

** Also affects: oslo.messaging (Ubuntu Wily)
   Importance: Undecided
       Status: New

** Also affects: python-oslo.messaging (Ubuntu Wily)
   Importance: Undecided
       Status: New

** No longer affects: oslo.messaging (Ubuntu Wily)

** No longer affects: cloud-archive/juno

** No longer affects: cloud-archive

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/juno
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/juno
     Assignee: (unassigned) => Corey Bryant (corey.bryant)

** Changed in: python-oslo.messaging (Ubuntu Wily)
     Assignee: (unassigned) => Corey Bryant (corey.bryant)

** Changed in: oslo.messaging (Ubuntu Vivid)
     Assignee: (unassigned) => Corey Bryant (corey.bryant)

** Changed in: oslo.messaging (Ubuntu Trusty)
     Assignee: (unassigned) => Corey Bryant (corey.bryant)

** Changed in: oslo.messaging (Ubuntu)
       Status: New => Invalid

** Changed in: python-oslo.messaging (Ubuntu)
       Status: New => Invalid

** Changed in: cloud-archive
       Status: New => Invalid

** Changed in: python-oslo.messaging (Ubuntu Wily)
       Status: New => In Progress

** Changed in: oslo.messaging (Ubuntu Vivid)
       Status: Fix Released => In Progress

** Changed in: python-oslo.messaging (Ubuntu Wily)
   Importance: Undecided => Medium

** Changed in: cloud-archive/juno
   Importance: Undecided => Medium

** Changed in: oslo.messaging (Ubuntu)
   Importance: Medium => Undecided

** Changed in: oslo.messaging (Ubuntu Vivid)
       Status: In Progress => New

-- 
You received this bug notification because you are a member of Ubuntu
Sponsors Team, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1521958

Title:
  rabbit: starvation of connections for reply

Status in Ubuntu Cloud Archive:
  Invalid
Status in Ubuntu Cloud Archive juno series:
  New
Status in oslo.messaging:
  Fix Released
Status in oslo.messaging package in Ubuntu:
  Invalid
Status in python-oslo.messaging package in Ubuntu:
  Invalid
Status in oslo.messaging source package in Trusty:
  New
Status in oslo.messaging source package in Vivid:
  New
Status in python-oslo.messaging source package in Wily:
  In Progress

Bug description:
  Hi,

  When a client died/restart/stop but was waiting more replies that the rpc_connection_pool size,
  the server will hold all connections from the pool during the retry logic in case of the client come back with the same reply_queue_id (that occurs only is rabbit is restart, not the client).

  Cheers,

  
  ---------------------------

  [Impact]

  This patch addresses an issue when multiple clients lost
  networks/died/restart/stop, the server will hold all connections from
  the pool(rpc_connection_pool size) during the retry logic in case of
  the client come back with the same reply_queue_id (that occurs only is
  rabbit is restart, not the client), and which cause nova-conductor
  infinitely reconnects to rabbit if large nova-compute hosts are
  deployment until all of the connections of the old reply messages are
  expired, for the quite large scaled cloud, high availability is
  broken.

  [Test Case]

  Note steps are for trusty-icehouse, including latest oslo.messaging
  library (1.3.0-0ubuntu1.2 at the time of this writing).

  Deploy an OpenStack cloud w/ multiple rabbit nodes and multiple nova
  compute hosts then cut off the network between OpenStack services and
  RabbitMQ. Observe that the nova-conductor is infinitely reconnecting
  to rabbit nodes.

  [Regression Potential]

  None.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1521958/+subscriptions