[Bug 1692397] Please test proposed package
Corey Bryant
corey.bryant at canonical.com
Tue Jan 2 15:25:01 UTC 2018
Hello Zhenyu, or anyone else affected,
Accepted nova into mitaka-proposed. The package will build now and be
available in the Ubuntu Cloud Archive in a few hours, and then in the
-proposed repository.
Please help us by testing this new package. To enable the -proposed
repository:
sudo add-apt-repository cloud-archive:mitaka-proposed
sudo apt-get update
Your feedback will aid us getting this update out to other Ubuntu users.
If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-mitaka-needed to verification-mitaka-done. If it does
not fix the bug for you, please add a comment stating that, and change
the tag to verification-mitaka-failed. In either case, details of your
testing will help us make a better decision.
Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
advance!
** Changed in: cloud-archive/mitaka
Status: Triaged => Fix Committed
** Tags added: verification-mitaka-needed
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1692397
Title:
hypervisor statistics could be incorrect
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive mitaka series:
Fix Committed
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) newton series:
Fix Committed
Status in OpenStack Compute (nova) ocata series:
Fix Committed
Status in nova package in Ubuntu:
Fix Released
Status in nova source package in Xenial:
Fix Released
Bug description:
[Impact]
If you deploy a nova-compute service to a node, delete that service
(via the api), then deploy a new nova-compute service to that same
node i.e. same hostname, the database will now have two service
records one marked as deleted and the other not. So far so good until
you do an 'openstack hypervisor stats show' at which point the api
will aggregate the resource counts from both services. This has been
fixed and backported all the way down to Newton so the problem still
exists on Mitaka. I assume the reason why the patch was not backported
to Mitaka is that the code in
nova.db.sqlalchemy.apy.compute_node_statistics() changed quite a bit.
However it only requires a one line change in the old code (that does
the same thing as the new code) to fix this issue.
[Test Case]
* Deploy Mitaka with bundle http://pastebin.ubuntu.com/25968008/
* Do 'openstack hypervisor stats show' and verify that count is 3
* Do 'juju remove-unit nova-compute/2' to delete a compute service
but not its physical host
* Do 'openstack compute service delete <id>' to delete a compute
service we just removed (choosing correct id)
* Do 'openstack hypervisor stats show' and verify that count is 2
* Do juju add-unit nova-compute --to <machine id of deleted unit>
* Do 'openstack hypervisor stats show' and verify that count is 3
(not 4 as it would be before fix)
[Regression Potential]
None anticipated other than for clients that were interpreting invalid
counts as correct.
[Other Info]
===========================================================================
Hypervisor statistics could be incorrect:
When we killed a nova-compute service and deleted the service from nova DB, and then
start the nova-compute service again, the result of Hypervisor/statistics API (nova hypervisor-stats) will be
incorrect;
How to reproduce:
Step1. Check the correct statistics before we do anything:
root at SZX1000291919:/opt/stack/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 1 |
| current_workload | 0 |
| disk_available_least | 14 |
| free_disk_gb | 34 |
| free_ram_mb | 6936 |
| local_gb | 35 |
| local_gb_used | 1 |
| memory_mb | 7960 |
| memory_mb_used | 1024 |
| running_vms | 1 |
| vcpus | 8 |
| vcpus_used | 1 |
+----------------------+-------+
Step2. Kill the compute service:
root at SZX1000291919:/var/log/nova# ps -ef | grep nova-com
root 120419 120411 0 11:06 pts/27 00:00:00 sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root 120420 120419 0 11:06 pts/27 00:00:07 /usr/bin/python /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log
root at SZX1000291919:/var/log/nova# kill -9 120419
root at SZX1000291919:/var/log/nova# /usr/local/bin/stack: line 19: 120419 Killed sg libvirtd '/usr/local/bin/nova-compute --config-file /etc/nova/nova.conf --log-file /var/log/nova/nova-compute.log' > /dev/null 2>&1
root at SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:36.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:24:37.000000 | - |
| 8 | nova-compute | SZX1000291919 | nova | enabled | down | 2017-05-22T03:23:38.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step3. Delete the service from DB:
root at SZX1000291919:/var/log/nova# nova service-delete 8
root at SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:16.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:25:17.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step4. Start the compute service again:
root at SZX1000291919:/var/log/nova# nova service-list
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
| 4 | nova-conductor | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:55.000000 | - |
| 6 | nova-scheduler | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 7 | nova-consoleauth | SZX1000291919 | internal | enabled | up | 2017-05-22T03:48:56.000000 | - |
| 9 | nova-cert | SZX1000291919 | internal | enabled | down | 2017-05-17T02:50:13.000000 | - |
| 10 | nova-compute | SZX1000291919 | nova | enabled | up | 2017-05-22T03:48:57.000000 | - |
+----+------------------+---------------+----------+---------+-------+----------------------------+-----------------+
Step5. Check again the hyervisor statistics, the result is incorrect:
root at SZX1000291919:/var/log/nova# nova hypervisor-stats
+----------------------+-------+
| Property | Value |
+----------------------+-------+
| count | 2 |
| current_workload | 0 |
| disk_available_least | 28 |
| free_disk_gb | 68 |
| free_ram_mb | 13872 |
| local_gb | 70 |
| local_gb_used | 2 |
| memory_mb | 15920 |
| memory_mb_used | 2048 |
| running_vms | 2 |
| vcpus | 16 |
| vcpus_used | 2 |
+----------------------+-------+
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1692397/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list