[Bug 2121812] Re: [SRU] cinder-netapp driver fails to start when a node is down

[Hemanth Nakkina]

** Tags removed: verification-needed-plucky
** Tags added: verification-done-plucky

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/2121812

Title:
  [SRU] cinder-netapp driver fails to start when a node is down

Status in Cinder:
  In Progress
Status in Ubuntu Cloud Archive:
  New
Status in Ubuntu Cloud Archive caracal series:
  New
Status in Ubuntu Cloud Archive epoxy series:
  New
Status in cinder package in Ubuntu:
  Fix Released
Status in cinder source package in Noble:
  New
Status in cinder source package in Plucky:
  Fix Committed
Status in cinder source package in Questing:
  Fix Released

Bug description:
  We're trying to use cinder-netapp, and the driver fails to start with the following error.
  https://pastebin.ubuntu.com/p/dwMdrVwdtf/

  After enabling debugging, I noticed that one node doesn't report its model.
  Checking on netapp side, the node is down due to a hardware issue, and another one is taking over the current node.

  If i add or ''  to get_cluster_nodes_info. it works fine.

  In this current setup, if any node goes down and the driver restarts,
  Cinder won't be able to recover.

  ***************************

  [SRU]

  [Impact]
  cinder-volume service fails to start when cinder-netapp driver is used with some netapp nodes in maintenance state.

  The service during driver initialization queries the netapp server for all the nodes information. Typically the node
  name, model and certain other attributes are expected. However in case if the node is in maintenance mode, the
  model information is missing.
  cinder-volume service does not handle properly in case the model value is None and so the service goes to failed state.

  The fix checks if the model value is None and assigns empty string as default value. In addition, a warning message is
  logged for missing model values.

  [Test Case]

  To test the bug, we need netapp storage nodes. Instead developed a
  small python script that responds to couple of netapp requests that
  are required to reproduce the bug.

  Here are the reproducer steps:

  1. Deploy regress-stack (https://github.com/canonical/regress-stack)
  2. Install the packages required for cinder service to get setup
     
     sudo snap install astral-uv --classic
     sudo apt-get update
     uvx pre-commit install
     sudo apt install dpkg-dev python3-dev python-apt-dev -y
     uv sync
     sudo apt install ceph mysql-server rabbitmq-server keystone cinder-api cinder-scheduler cinder-volume -y

  3. Run regress-stack setup step

     uv run regress-stack setup

  4. Simulate netapp server

     The simulated code responds to couple of initial netapp requests and you can
     see in L#33 one of the node has no model information.
     Python code: https://pastebin.ubuntu.com/p/7pBSXzFSGY/
     python <netapp.py>

  5. Update cinder.conf to add netapp configuration

     https://pastebin.ubuntu.com/p/xK52rT5f7V/ (modified cinder configs)
     systemctl restart cinder-volume.service

  6. Check for cinder-volume logs

     Non-Working case:
     Should fail with error `TypeError: argument of type 'NoneType' is not iterable`

     Working case:
     Should see a printout in logs: `Reported ONTAPI Version: 1.261`
     (At this point of time the code execution crossed the bug)

     Note: The service wont start and fail with following error as the
  simulation does not support all the netapp API calls.

     cinder.volume.drivers.netapp.dataontap.client.api.NaApiError:
  NetApp API failed. Reason - 400:BAD REQUEST

  
  [Regression Potential]
  In order to mitigate any regression potential, the fix has been tested with real hardware for jammy caracal. The unit test cases are also updated to verify the bug. Also the default logic is not changed when model is provided by netapp server.

  [Discussion]
  n/a

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/2121812/+subscriptions