[Bug 1724342] Re: test_drop_privileges AssertionError groups mismatch
OpenStack Infra
1724342 at bugs.launchpad.net
Sat Nov 4 00:25:27 UTC 2017
Reviewed: https://review.openstack.org/517718
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=5c564e53966b74d6d7589c3505dd33816b3c19f1
Submitter: Zuul
Branch: feature/deep
commit 77a8a4455d4a2bd0a9b9146a01acfe0d93fe7550
Author: Tim Burke <tim.burke at gmail.com>
Date: Tue Oct 3 00:11:07 2017 +0000
Let clients request heartbeats during SLO PUTs
An SLO PUT requires that we HEAD every referenced object; as a result, it
can be a very time-intensive operation. This makes it difficult as a
client to differentiate between a proxy-server that's still doing work and
one that's crashed but left the socket open.
Now, clients can opt-in to receiving heartbeats during long-running PUTs
by including the query parameter
heartbeat=on
With heartbeating turned on, the proxy will start its response immediately
with 202 Accepted then send a single whitespace character periodically
until the request completes. At that point, a final summary chunk will be
sent which includes a "Response Status" key indicating success or failure
and (if successful) an "Etag" key indicating the Etag of the resulting SLO.
This mechanism is very similar to the way bulk extractions and deletions
work, and even the way SLO behaves for ?multipart-manifest=delete requests.
Note that this is opt-in: this prevents us from sending the 202 response
to existing clients that may mis-interpret it as an immediate indication
of success.
Co-Authored-By: Alistair Coles <alistairncoles at gmail.com>
Related-Bug: 1718811
Change-Id: I65cee5f629c87364e188aa05a06d563c3849c8f3
commit 92705bb36b4a771a757977d11875e7b929c41741
Author: David Rabel <rabel at b1-systems.de>
Date: Thu Nov 2 12:38:41 2017 +0100
Fix indent in overview_policies.rst
Change-Id: I7f070956d8b996db798837392adfca4483067aea
commit feee3998408e5ed03563c317ad9506ead92083a6
Author: Clay Gerrard <clay.gerrard at gmail.com>
Date: Fri Sep 1 14:15:45 2017 -0700
Use check_drive consistently
We added check_drive to the account/container servers to unify how all
the storage wsgi servers treat device dirs/mounts. Thus pushes that
unification down into the consistency engine.
Drive-by:
* use FakeLogger less
* clean up some repeititon in probe utility for device re-"mounting"
Related-Change-Id: I3362a6ebff423016bb367b4b6b322bb41ae08764
Change-Id: I941ffbc568ebfa5964d49964dc20c382a5e2ec2a
commit 29e9ae1cc5b74dce205643c101601555c06b67dc
Author: Tim Burke <tim.burke at gmail.com>
Date: Thu Oct 19 18:53:04 2017 +0000
Make xml responses less insane
Looking at bulk extractions:
$ tar c *.py | curl http://saio:8090/v1/AUTH_test/c?extract-archive=tar \
-H accept:application/xml -T -
<?xml version="1.0" encoding="UTF-8"?>
<delete>
<number_files_created>2</number_files_created>
<response_body></response_body>
<response_status>201 Created</response_status>
<errors>
</errors>
</delete>
Or SLO upload failures:
$ curl http://saio:8090/v1/AUTH_test/c/slo?multipart-manifest=put -X PUT \
-d '[{"path": "not/found"}]' -H accept:application/xml
<delete>
<errors>
<object><name>not/found</name><status>404 Not Found</status></object></errors>
</delete>
Why <delete>? Makes no sense.
Drive-by: stop being so quadratic.
Change-Id: I46b233864ba2815ac632d856b9f3c40cc9d0001a
commit 449d83fb0c8668f444baf3c74cf15c26a70bb02e
Author: Matthew Oliver <matt at oliver.net.au>
Date: Tue Oct 31 15:55:13 2017 +1100
Doc uses alias instead of aliases
The overview_policies doc makes reference to an `alias` option when in
fact the option is `aliases`.
The sample storage policy snippet is correct, it's just incorrect when
listing the possible options.
This change changes the listed option to `aliases`.
Change-Id: Iddf0f19f4d50819ff6abd46e6a1156dc8e4a451d
commit 178d7f37cb80efa668bc81e79d1ee633204b8852
Author: HCLTech-SSW <hcl_ss_oss at hcl.com>
Date: Wed Oct 25 03:23:47 2017 -0700
Added the man page for container-reconciler.conf
Change-Id: Ic7fc6ddca2ab564b31156fa84b362bc9963825f1
Closes-Bug: #1607025
commit 83be309627e79c2f4dd66faf17b5459fdf5bfc25
Author: Kota Tsuyuzaki <tsuyuzaki.kota at lab.ntt.co.jp>
Date: Thu Oct 26 18:41:56 2017 +0900
Fix a small typo
That's redundant "to" in an inline comment.
Change-Id: Idab1e11fbbd80a97b0e4ba1c8ab046be99e47d2d
commit 8d882095375dc53acdafafac40aa2efc3bafbcd4
Author: Thiago da Silva <thiago at redhat.com>
Date: Thu Sep 15 16:45:06 2016 -0400
Return 404 on a GET if tombstone is newer
Currently the proxy keeps iterating through
the connections in hope of finding a success even
if it already has found a tombstone (404).
This change changes the code a little bit to compare
the timestamp of a 200 and a 404, if the tombstone is
newer, then it should be returned, instead of returning
a stale 200.
Closes-Bug: #1560574
Co-Authored-By: Tim Burke <tim.burke at gmail.com>
Change-Id: Ia81d6832709d18fe9a01ad247d75bf765e8a89f4
Signed-off-by: Thiago da Silva <thiago at redhat.com>
commit e199192caefef068b5bf57da8b878e0bc82e3453
Author: Romain LE DISEZ <romain.le-disez at corp.ovh.com>
Date: Wed Oct 26 10:53:46 2016 +0200
Replace replication_one_per_device by custom count
This commit replaces boolean replication_one_per_device by an integer
replication_concurrency_per_device. The new configuration parameter is
passed to utils.lock_path() which now accept as an argument a limit for
the number of locks that can be acquired for a specific path.
Instead of trying to lock path/.lock, utils.lock_path() now tries to lock
files path/.lock-X, where X is in the range (0, N), N being the limit for
the number of locks allowed for the path. The default value of limit is
set to 1.
Change-Id: I3c3193344c7a57a8a4fc7932d1b10e702efd3572
commit 9c97c80b2631bedebbd3196e87f0696a6c67b8be
Author: Samuel Merritt <sam at swiftstack.com>
Date: Fri Oct 20 15:31:07 2017 -0700
Clean up a couple hand-rolled mocks.
Change-Id: I6582985990e8b5e3a0c65bce5d3bb1e39d58dfb9
commit 9b2779ba131aa22893432b5c96fb19bf43656a4f
Author: Samuel Merritt <sam at swiftstack.com>
Date: Fri Oct 20 14:48:31 2017 -0700
Clean up memcache tests
This is mostly just using mock.patch instead of doing it by hand. One
test was doing some weird exception counting; now it just uses
assertRaises in a loop.
Change-Id: I54f903f170226720405df2c5f6845124909ab830
commit 0bca0d4d2bf95a37e4a403457ee5c7db4c0b6290
Author: Pete Zaitcev <zaitcev at kotori.zaitcev.us>
Date: Wed Oct 18 18:15:11 2017 -0500
Be more tolerant of exception messages from sqlite
Parsing the text of error messages was always perilous, perhaps.
Either way, this should fix the problem of test_get() failing.
Change-Id: I929c2f6e60624241075a292d020c5707ea0ff1c3
commit e001c02ff938d9aea560970fa70f3d8884a8ea33
Author: Tim Burke <tim.burke at gmail.com>
Date: Tue Oct 17 22:49:39 2017 +0000
Stop logging tracebacks on bad xLOs
The error messages alone should provide plenty enough information.
Plus, running functional tests really shouldn't cause tracebacks.
Also, tighten up tests for log messages.
Change-Id: I55136484d342af756fa153d971dcb9159a435f13
commit 356b110229ee5e5be93ce1ec60ed2de8f75090bf
Author: Samuel Merritt <sam at swiftstack.com>
Date: Wed Oct 18 15:09:12 2017 -0700
Remove swift-temp-url man page.
Commit 250da37a removed bin/swift-temp-url, but left the man page.
Change-Id: Ic518f23678e3c3134f02a46a51a2bcb90d92bdc2
commit 1aecd1dfc61988f12d4cdfda7110dd648133e4cf
Author: Alistair Coles <alistairncoles at gmail.com>
Date: Wed Oct 18 10:52:52 2017 +0100
tighten up drop_privileges unit tests
add more assertions about args that are passed to
os module functions
Related-Change: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
Change-Id: Iee483274aff37fc9930cd54008533de2917157f4
commit e6cf9b4758612f38f7317c52423f317b9a11bbc9
Author: Samuel Merritt <sam at swiftstack.com>
Date: Tue Oct 17 15:16:43 2017 -0700
Fix some spelling in a docstring
Change-Id: I6b32238da3381848ae56aed1c570d299be72473e
commit 646f7507a1f5cf479b8d2023645009de0a28cd79
Author: Tim Burke <tim.burke at gmail.com>
Date: Tue Oct 17 21:17:57 2017 +0000
Quiet test output when running test_utils.py in isolation
Change-Id: I4cf85d3cd5a20424e9bbbdf0213120b3c3d4b837
commit 0f91c862e19bb8d81c20d03e65e6e6657f7265dc
Author: Corey Bryant <corey.bryant at canonical.com>
Date: Tue Oct 17 15:04:45 2017 -0400
Drop group comparison from drop_privileges test
Drop the group comparison from drop_privileges test as it isn't
valid since os.setgroups() is mocked.
Change-Id: Ida15e72ae4ecdc2d6ce0d37bd99c2d86bd4e5ddc
Closes-Bug: #1724342
commit 5438fe7d9bcb7dc99118c12bdf1ccfd663390b08
Author: Samuel Merritt <sam at swiftstack.com>
Date: Fri Oct 13 17:03:18 2017 -0700
Clean up some leftover Python 2.6-isms.
Change-Id: I24f0aa3bf7314a669c4cf44eadc46e874d6803f1
commit 250da37a7b279b05a1ac8954b506add1661f194d
Author: Tim Burke <tim.burke at gmail.com>
Date: Fri Oct 13 23:27:11 2017 +0000
Remove swift-temp-url script
This has been deprecated since Swift 2.10.0 (Newton) including a
message that it would go away. Let's actually remove it.
Change-Id: I7d3659761c71119363ff2c0c750e37b4c6374a39
Related-Change: Ifa8bf636f20f82db4845b02d1b58699edaa39356
commit c118059719ded85381e9b134b091d04a7b5a3955
Author: Tim Burke <tim.burke at gmail.com>
Date: Mon Sep 11 22:08:12 2017 +0000
Respond 400 Bad Request when Accept headers fail to parse
Change-Id: I6eb4e4bca95e2ee4fecdb703394cb2419737922d
Closes-Bug: 1716509
commit 03e8ab7171580f3fb93131e9ccfb69fd35364672
Author: Samuel Merritt <sam at swiftstack.com>
Date: Thu Oct 12 17:15:54 2017 -0700
functests: don't crash if no second account
In test.functional.test_object.TestObject.setUp, we create a container
in account 2. However, if we've only got one account, we don't skip
this class, resulting in a TypeError down in requests somewhere and a
stack trace. Since we're using account 2 in setup, we should skip the
tests if account 2 is not configured.
Change-Id: I569d98baf071d2dce7cf34a9538070f00afda388
commit eaea0c49332273085b17852e22582940ab059006
Author: Samuel Merritt <sam at swiftstack.com>
Date: Thu Oct 12 16:58:41 2017 -0700
Skip cross-account-copy functest if only one account
This looks like a case of copy-paste-itis. The cross-account-copy
functest is skipped if we have no test accounts configured, but not if
we have only one.
Change-Id: Ifbefdd9aeb98e3d02c536e9d29759f86ec9af6a1
commit 24188beb81d39790034fa0902246163a7bf54c91
Author: Samuel Merritt <sam at swiftstack.com>
Date: Thu Oct 12 16:13:25 2017 -0700
Remove some leftover threadpool cruft.
Change-Id: I43a1a428bd96a2e18aac334c03743a9f94f7d3e1
commit 1d67485c0b935719e0c8999eb353dfd84713add6
Author: Samuel Merritt <sam at swiftstack.com>
Date: Fri Apr 15 12:43:44 2016 -0700
Move all monkey patching to one function
Change-Id: I2db2e53c50bcfa17f08a136581cfd7ac4958ada2
commit dc8da5bb194d0a544fb8065aa9a4c7484f605715
Author: Samuel Merritt <sam at swiftstack.com>
Date: Thu Oct 12 10:45:12 2017 -0700
Use "poll" or "selects" Eventlet hub for all Swift daemons.
Previously, Swift's WSGI servers, the object replicator, and the
object reconstructor were setting Eventlet's hub to either "poll" or
"selects", depending on availability. Other daemons were letting
Eventlet use its default hub, which is "epoll".
In any daemons that fork, we really don't want to use epoll. Epoll
instances end up shared between the parent and all children, and you
get some awful messes when file descriptors are shared.
Here's an example where two processes are trying to wait on the same
file descriptor using the same epoll instance, and everything goes
wrong:
[proc A] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = 0
[proc B] epoll_ctl(6, EPOLL_CTL_ADD, 3, ...) = -1 EEXIST (File exists)
[proc B] epoll_wait(6, ...) = 1
[proc B] epoll_ctl(6, EPOLL_CTL_DEL, 3, ...) = 0
[proc A] epoll_wait(6, ...)
This primarily affects the container updater and object updater since
they fork. I've decided to change the hub for all Swift daemons so
that we don't add multiprocessing support to some other daemon someday
and suffer through this same bug again.
This problem was made more apparent by commit 6d16079, which made our
logging mutex use file descriptors. However, it could have struck on
any shared file descriptor on which a read or write returned EAGAIN.
Change-Id: Ic2c1178ac918c88b0b901e581eb4fab3b2666cfe
Closes-Bug: 1722951
commit 37f23b072ee7a655ab9072f1d0bece0919dec61b
Author: Samuel Merritt <sam at swiftstack.com>
Date: Mon Oct 9 17:31:30 2017 -0700
Allow SLOs to have zero-byte last segments.
Since we used to allow zero-byte last segments but now we don't, it
can be difficult to deal with some old SLO manifests.
Imagine you're writing some code to sync objects from Swift cluster A
to Swift cluster B. You start off with just a GET from A piped into a
PUT to B, and that works great until you hit a SLO manifest and B
won't accept a 500GB object. So, you write some code to detect SLO
manifests, sync their segments, then take the JSON manifest
(?multipart-manifest=get) and sync *that* over.
Now, life is good... until one day you get an exception notification
that there's this manifest on cluster A that cluster B won't
accept. Turns out that, back when Swift would take zero-byte final
segments on SLOs (before commit 7f636a5), someone uploaded such a SLO
to cluster A. Now, however, zero-byte final segments are invalid, so
that SLO that exists over in cluster A can't just be copied to cluster
B.
A little coding later, your sync tool detects zero-byte final segments
and removes them when copying a manifest. But now your ETags don't
match between clusters, so you have to figure out some way to deal
with that, and so you put it in metadata, but then you realize that
your syncer might encounter a SLO which contains a sub-SLO which has a
zero-byte final segment, and it's right about then that you start
thinking about giving up on programming and getting a job as an
elevator mechanic.
This commit makes life easier for developers of such applications by
allowing SLOs to have zero-byte segments again.
Change-Id: Ia37880bbb435e269ec53b2963eb1b9121696d479
commit 348bd83b7eb638d3309ff6313d9a6501c540fa24
Author: Alistair Coles <alistairncoles at gmail.com>
Date: Wed Aug 30 15:29:14 2017 +0100
Respect co-builder partition moves when min_part_hours is zero
Repeated calls to each co-builder's _update_last_part_moves() are
unnecessary and have the unfortunate side effect of resetting the
_last_part_moved bitmap. When a component builder has zero
min_part_hours this results in it not preventing its co-builders from
moving parts that it has already moved.
This patch changes the CompositeRingBuilder to call each component
builder _update_last_part_moves() *once* before rebalancing any
component builder. CooperativeRingBuilder's no longer forward calls to
their _update_last_part_moves() method. Each component's
_last_part_moved bitmap is therefore preserved until for the duration
of the composite rebalance.
The initialisation of the RingBuilder _last_part_moves array is moved
to the RingBuilder __init__ method, so that calls to
_update_last_part_moves() are effective even when rebalance() has
never been called on that builder. Otherwise, during a composite
rebalance, a component that has not previously been rebalanced will
not have its _last_part_moves_epoch updated during rebalance and as a
result may report erroneous min_part_seconds_left after its first
rebalance.
Related-Change: I1b30cb3d776be441346a4131007d2487a5440a81
Closes-Bug: #1714274
Change-Id: Ib165cf974c865d47c2d9e8f7b3641971d2e9f404
commit a959d24bf5733650c6005412382fe24c65897c8b
Author: Alistair Coles <alistairncoles at gmail.com>
Date: Tue Aug 15 12:37:33 2017 +0100
Document keystone role element in container ACL
The use of a keystone role name in container ACLs is supported
and tested. This patch adds documentation.
[1] https://github.com/openstack/swift/blob/fb3d01a974fb7df8cfadc56ff15bdc04b3c90759/swift/common/middleware/keystoneauth.py#L491-L497
[2] test.unit.common.middleware.test_keystoneauth.TestAuthorize.test_authorize_succeeds_for_user_role_in_roles
Change-Id: I77df27393a10f1d8c5a43161fdd4eb08be632566
Closes-Bug: #1705300
** Tags added: in-feature-deep
--
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to swift in Ubuntu.
https://bugs.launchpad.net/bugs/1724342
Title:
test_drop_privileges AssertionError groups mismatch
Status in Ubuntu Cloud Archive:
Fix Released
Status in Ubuntu Cloud Archive pike series:
Fix Released
Status in OpenStack Object Storage (swift):
Fix Released
Status in swift package in Ubuntu:
Fix Released
Bug description:
This test is failing in Ubuntu builds on Artful. However I think it
generally just gets lucky when it is successful.
======================================================================
FAIL: test_drop_privileges (test.unit.common.test_utils.TestUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/test/unit/common/test_utils.py", line 2151, in test_drop_privileges
self.assertEqual(set(groups), set(os.getgroups()))
AssertionError: Items in the second set but not the first:
110
"""Fail immediately, with the given message."""
>> raise self.failureException('Items in the second set but not the first:\n110')
It seems that setgroups() is mocked out. Part of what drop_privileges() does is:
groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
os.setgroups(groups)
The failing part of test_drop_privileges() is:
groups = [g.gr_gid for g in grp.getgrall() if user in g.gr_mem]
groups.append(pwd.getpwnam(user).pw_gid)
self.assertEqual(set(groups), set(os.getgroups()))
The test is trying to assert that os.getgroups() was set in
drop_privileges. However os.getgroups() can't change since
os.setgroups() is mocked out, and os.getgroups() isn't mocked out.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1724342/+subscriptions
More information about the Ubuntu-openstack-bugs
mailing list