[Bug 2051299] Re: Failed to wipe Micron 7400 MTFDKBA960TDZ during machine release

Wed Feb 7 14:00:42 UTC 2024

Thanks maasuser1 for the analysis.

The nmve-cli issue https://github.com/linux-nvme/nvme-cli/issues/1653
has been fixed in https://github.com/linux-nvme/nvme-
cli/commit/78b7ad235507ddd59c75c7fcc74fc6c927811f87 and released in
https://github.com/linux-nvme/nvme-cli/releases/tag/v2.2

jammy uses nvmi-cli in version 1.16, mantic has a later version (2.5).
I'm not sure if the bugfix can be backported to jammy, so nvme-cli
package was added as affected for further triage.

** Also affects: nvme-cli (Ubuntu)
   Importance: Undecided
       Status: New

** Bug watch added: github.com/linux-nvme/nvme-cli/issues #1653
   https://github.com/linux-nvme/nvme-cli/issues/1653

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to nvme-cli in Ubuntu.
https://bugs.launchpad.net/bugs/2051299

Title:
  Failed to wipe Micron 7400 MTFDKBA960TDZ during machine release

Status in MAAS:
  New
Status in nvme-cli package in Ubuntu:
  New

Bug description:
  - Both main and secondary controllers running MAAS 3.4.0-14321-g.1027c7664 installed by Snap.
  - Host: Supermicro SYS-6019P-WT with Micron 7400 MTFDKBA960TDZ
  - Both the first and second attempts of machine releasing failed.
  - All parts of commissioning and tests passed with no errors. Ubuntu was deployed to the host with no errors.
  - Newly shipped hardware so it can't be a hardware problem.

  ```
  2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: (Reading database ... #015(Reading database ... 5%#015(Reading database ... 10%#015(Reading database ... 15%#015(Reading database ... 20%#015(Reading database ... 25%#015(Reading database >2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: Preparing to unpack .../nvme-cli_1.16-3ubuntu0.1_amd64.deb ...
  2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: Unpacking nvme-cli (1.16-3ubuntu0.1) ...
  2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: Setting up nvme-cli (1.16-3ubuntu0.1) ...
  2024-01-25T16:56:27+00:00 dp2 systemd[1]: Reloading.
  2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service.
  2024-01-25T16:56:27+00:00 dp2 systemd[1]: Reloading.
  2024-01-25T16:56:27+00:00 dp2 cloud-init[2497]: Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service.
  2024-01-25T16:56:28+00:00 dp2 systemd[1]: Reloading.
  2024-01-25T16:56:28+00:00 dp2 cloud-init[2497]: nvmf-connect.target is a disabled or a static unit, not starting it.
  2024-01-25T16:56:28+00:00 dp2 systemd[1]: Condition check resulted in Auto-connect to subsystems on FC-NVME devices found during boot being skipped.
  2024-01-25T16:56:28+00:00 dp2 systemd[1]: Starting Connect NVMe-oF subsystems automatically during boot...
  2024-01-25T16:56:28+00:00 dp2 systemd[1]: nvmf-autoconnect.service: Deactivated successfully.
  2024-01-25T16:56:28+00:00 dp2 systemd[1]: Finished Connect NVMe-oF subsystems automatically during boot.
  2024-01-25T16:56:28+00:00 dp2 cloud-init[2497]: Processing triggers for man-db (2.10.2-1) ...
  2024-01-25T16:56:28+00:00 dp2 cloud-init[2497]: NEEDRESTART-VER: 3.5
  2024-01-25T16:56:29+00:00 dp2 cloud-init[2497]: NEEDRESTART-KCUR: 5.15.0-91-generic
  2024-01-25T16:56:29+00:00 dp2 cloud-init[2497]: NEEDRESTART-KSTA: 0
  2024-01-25T16:56:29+00:00 dp2 cloud-init[2497]: NEEDRESTART-UCSTA: 0
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]: Traceback (most recent call last):
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 542, in <module>
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     main()
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 522, in main
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     disk_info = get_disk_info()
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 165, in get_disk_info
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     return {kname: get_disk_security_info(kname) for kname in list_disks()}
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 165, in <dictcomp>
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     return {kname: get_disk_security_info(kname) for kname in list_disks()}
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 158, in get_disk_security_info
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     return get_nvme_security_info(disk)
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:   File "/tmp/user_data.sh.jNE4lC/bin/maas-wipe", line 64, in get_nvme_security_info
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]:     output = output.decode()
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 385: invalid start byte
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]: 2024-01-25 16:56:30,929 - cc_scripts_user.py[WARNING]: Failed to run module scripts_user (scripts in /var/lib/cloud/instance/scripts)
  2024-01-25T16:56:30+00:00 dp2 cloud-init[2497]: 2024-01-25 16:56:30,930 - util.py[WARNING]: Running module scripts_user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py>
  2024-01-25T16:56:31+00:00 dp2 cloud-init: #############################################################
  2024-01-25T16:56:31+00:00 dp2 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
  2024-01-25T16:56:31+00:00 dp2 cloud-init: 1024 SHA256:K01EHEVM4P7nsT++Sa/t2HdUJBmEtq0CzvTF80Nxl5U root at dp2 (DSA)
  2024-01-25T16:56:31+00:00 dp2 cloud-init: 256 SHA256:esLgsNXtlXz5DTOVoo87+TEwUoV5AOaNTKow7VU1is0 root at dp2 (ECDSA)
  2024-01-25T16:56:31+00:00 dp2 cloud-init: 256 SHA256:JyJHsc60D/M5slr2ERoYDtKaTA4y08hT4blFFR45Dn4 root at dp2 (ED25519)
  2024-01-25T16:56:31+00:00 dp2 cloud-init: 3072 SHA256:KdILKl8DlVf2DFhxbe6x4TY806Wv42peRfXrRWH/7gw root at dp2 (RSA)
  2024-01-25T16:56:31+00:00 dp2 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
  2024-01-25T16:56:31+00:00 dp2 cloud-init: #############################################################
  2024-01-25T16:56:31+00:00 dp2 cloud-init[2497]: Cloud-init v. 23.3.3-0ubuntu0~22.04.1 finished at Thu, 25 Jan 2024 16:56:31 +0000. Datasource DataSourceMAAS [http://192.168.0.2:5248/MAAS/metadata/].  Up 164.57 seconds
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Main process exited, code=exited, status=1/FAILURE
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Failed with result 'exit-code'.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Unit process 2496 (sh) remains running after unit stopped.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Unit process 2497 (tee) remains running after unit stopped.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Unit process 3187 (cloud-init) remains running after unit stopped.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: Failed to start Execute cloud user/final scripts.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: cloud-final.service: Consumed 13.354s CPU time.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: Reached target Cloud-init target.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: Startup finished in 22.611s (kernel) + 2min 22.111s (userspace) = 2min 44.723s.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: Removed slice Slice /system/modprobe.
  2024-01-25T16:56:31+00:00 dp2 systemd[1]: Stopped target Cloud-init target.
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/2051299/+subscriptions