[Bug 1907686] Re: ovn: instance unable to retrieve metadata

Thu May 6 05:21:25 UTC 2021

After applying the fix in bionic-proposed, 
only one compute node exhibited the logs mentioned in #5
2021-01-12 06:48:49.848 52569 ERROR neutron.agent.ovn.metadata.server [-] Unexpected error.: AttributeError: 'MetadataProxyHandler' object has no attribute 'sb_idl'

sb_idl is never initialised as the connection timed out during initialisation phase.

See error log:

2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn [-] OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Traceback (most recent call last):
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     self.ovsdb_connection.start()
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn Exception: Timeout
2021-04-15 22:30:08.542 69188 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager [-] Error during notification for neutron.agent.ovn.metadata.server.MetadataProxyHandler.post_fork_initialize-476074 process, after_init: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS database connection to OVN_Southbound failed with error: 'Timeout'. Verify that the OVS and OVN services are available and that the 'ovn_nb_connection' and 'ovn_sb_connection' configuration options are correct.
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Traceback (most recent call last):
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 67, in start_connection
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     self.ovsdb_connection.start()
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 79, in start
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     idlutils.wait_for_change(self.idl, self.timeout)
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager   File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 173, in wait_for_change
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager     raise Exception("Timeout")  # TODO(twilson) use TimeoutException?
2021-04-15 22:30:08.544 69188 ERROR neutron_lib.callbacks.manager Exception: Timeout

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to openvswitch in Ubuntu.
https://bugs.launchpad.net/bugs/1907686

Title:
  ovn: instance unable to retrieve metadata

Status in charm-ovn-chassis:
  Invalid
Status in Ubuntu Cloud Archive:
  Fix Released
Status in Ubuntu Cloud Archive ussuri series:
  Fix Committed
Status in Ubuntu Cloud Archive victoria series:
  New
Status in Ubuntu Cloud Archive wallaby series:
  Fix Released
Status in neutron:
  Invalid
Status in openvswitch package in Ubuntu:
  Fix Released
Status in openvswitch source package in Focal:
  Fix Committed
Status in openvswitch source package in Groovy:
  Fix Committed
Status in openvswitch source package in Hirsute:
  Fix Released

Bug description:
  [Impact]
  Cloud instances are unable to retrieve metadata on startup.

  [Test Case]
  Deploy OpenStack with OVN/OVS
  Restart OVN central controllers
  Create a new instance
  Instance will fail to retrieve metadata with the message from the original bug report displayed in the metadata agent log on the local hypervisor

  [Regression Potential]
  The fix for this issue is included in the upstream 2.13.3 release of OVS.
  The fix ensures that SSL related connection issues are correctly handling in python3-ovs avoiding an issue where the connection to the OVN SB IDL is reset and never recreated.
  The OVN drivers use python3-ovsdbapp which in turn bases off code provided by python3-ovs.

  [Original Bug Report]
  Ubuntu:focal
  OpenStack: ussuri
  Instance port: hardware offloaded

  instance created, attempts to access metadata - metadata agent can't
  resolve the port/network combination:

  2020-12-10 15:00:18.258 4732 INFO neutron.agent.ovn.metadata.agent [-] Port d65418a6-d0e9-47e6-84ba-3d02fe75131a in datapath 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 bound to our ch
  assis
  2020-12-10 15:00:31.672 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155
  2020-12-10 15:00:31.673 8062 INFO eventlet.wsgi.server [-] 10.5.1.155,<local> "GET /openstack HTTP/1.1" status: 404  len: 297 time: 0.0043790
  2020-12-10 15:00:34.639 8062 ERROR neutron.agent.ovn.metadata.server [-] No port found in network 37706e4d-ce2a-4d81-8c61-3fd12437a0a7 with IP address 10.5.1.155
  2020-12-10 15:00:34.639 8062 INFO eventlet.wsgi.server [-] 10.5.1.155,<local> "GET /openstack HTTP/1.1" status: 404  len: 297 time: 0.0040138

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686/+subscriptions