[Bug 1915819] Re: 'NoneType' object has no attribute 'encode' in requestReceived() when multipart body doesn't include content-disposition
Victor Tapia
1915819 at bugs.launchpad.net
Thu Feb 18 14:47:41 UTC 2021
** Description changed:
[impact]
python-twisted errors out with "'NoneType' object has no attribute
'encode' in requestReceived()" when it tries to parse a multipart mime
message and python3.7+ is used. This happens because before commit
cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
defined in "content-disposition" (or parts without headers for that
matter) but after 3.7+ the return of the function can now contain
NoneType keys, which fail to encode.
[scope]
- This bug affects all releases
+ Even though this bug affects all python3-twisted releases, I'll backport
+ the fix just to Focal, Groovy and Hirsute. Bionic and Xenial do not have
+ Python 3.7 as the default interpreter (required to trigger the issue),
+ and the delta in python3-twisted might be to big to backport as the
+ current packages do not contemplate _PY37PLUS at all.
Fixed upstream with commit 310496249, available since 21.2.0rc1
[test case]
1. Save the following code as webserver.py
from twisted.application.internet import TCPServer
from twisted.application.service import Application
from twisted.web.resource import Resource
from twisted.web.server import Site
-
class Foo(Resource):
- def render_POST(self, request):
- newdata = request.content.getvalue()
- print(newdata)
- return ''
-
+ def render_POST(self, request):
+ newdata = request.content.getvalue()
+ print(newdata)
+ return ''
root = Resource()
root.putChild("foo", Foo())
application = Application("cgi.parse_multipart test")
TCPServer(8080, Site(root)).setServiceParent(application)
2. Save the following code as client.py (python3-httplib2 is required)
#!/usr/bin/env python
import httplib2
-
def http_request(url, method, body=None, headers=None, insecure=False):
- """Issue an http request."""
- http = httplib2.Http(disable_ssl_certificate_validation=insecure)
- if isinstance(url, bytes):
- url = url.decode("ascii")
- return http.request(url, method, body=body, headers=headers)
-
+ """Issue an http request."""
+ http = httplib2.Http(disable_ssl_certificate_validation=insecure)
+ if isinstance(url, bytes):
+ url = url.decode("ascii")
+ return http.request(url, method, body=body, headers=headers)
url = "http://localhost:8080"
method = "POST"
headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
emptyh = '--8825899812428059282\n\n--8825899812428059282--'
print("== BODY: " + emptyh + "\n")
response, content = http_request(url, method, emptyh, headers)
3. Run the server with "twistd3 -y webserver.py"
4. Run the client
5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)
2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
- Traceback (most recent call last):
- File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
- return callWithContext({"system": lp}, func, *args, **kw)
- File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
- return context.call({ILogContext: newCtx}, func, *args, **kw)
- File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
- return self.currentContext().callWithContext(ctx, func, *args, **kw)
- File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
- return func(*args,**kw)
- --- <exception caught here> ---
- File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
- why = selectable.doRead()
- File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
- return self._dataReceived(data)
- File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
- rval = self.protocol.dataReceived(data)
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
- return self._channel.dataReceived(data)
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
- return basic.LineReceiver.dataReceived(self, data)
- File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
- why = self.rawDataReceived(data)
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
- self._transferDecoder.dataReceived(data)
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
- finishCallback(data[contentLength:])
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
- self.allContentReceived()
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
- req.requestReceived(command, path, version)
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
- self.args.update({
- File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
- x.encode('iso-8859-1'): \
- builtins.AttributeError: 'NoneType' object has no attribute 'encode'
+ Traceback (most recent call last):
+ File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
+ return callWithContext({"system": lp}, func, *args, **kw)
+ File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
+ return context.call({ILogContext: newCtx}, func, *args, **kw)
+ File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
+ return self.currentContext().callWithContext(ctx, func, *args, **kw)
+ File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
+ return func(*args,**kw)
+ --- <exception caught here> ---
+ File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
+ why = selectable.doRead()
+ File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
+ return self._dataReceived(data)
+ File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
+ rval = self.protocol.dataReceived(data)
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
+ return self._channel.dataReceived(data)
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
+ return basic.LineReceiver.dataReceived(self, data)
+ File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
+ why = self.rawDataReceived(data)
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
+ self._transferDecoder.dataReceived(data)
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
+ finishCallback(data[contentLength:])
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
+ self.allContentReceived()
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
+ req.requestReceived(command, path, version)
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
+ self.args.update({
+ File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
+ x.encode('iso-8859-1'): \
+ builtins.AttributeError: 'NoneType' object has no attribute 'encode'
[regression potential]
This affects the returned dictionaries with non-str keys, which were
discarded in python3.6 or earlier before they reached twisted, so
patching this will make its behavior consistent.
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to twisted in Ubuntu.
https://bugs.launchpad.net/bugs/1915819
Title:
'NoneType' object has no attribute 'encode' in requestReceived() when
multipart body doesn't include content-disposition
Status in twisted package in Ubuntu:
New
Status in twisted source package in Focal:
New
Status in twisted source package in Groovy:
New
Status in twisted source package in Hirsute:
New
Bug description:
[impact]
python-twisted errors out with "'NoneType' object has no attribute
'encode' in requestReceived()" when it tries to parse a multipart mime
message and python3.7+ is used. This happens because before commit
cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
defined in "content-disposition" (or parts without headers for that
matter) but after 3.7+ the return of the function can now contain
NoneType keys, which fail to encode.
[scope]
Even though this bug affects all python3-twisted releases, I'll
backport the fix just to Focal, Groovy and Hirsute. Bionic and Xenial
do not have Python 3.7 as the default interpreter (required to trigger
the issue), and the delta in python3-twisted might be to big to
backport as the current packages do not contemplate _PY37PLUS at all.
Fixed upstream with commit 310496249, available since 21.2.0rc1
[test case]
1. Save the following code as webserver.py
from twisted.application.internet import TCPServer
from twisted.application.service import Application
from twisted.web.resource import Resource
from twisted.web.server import Site
class Foo(Resource):
def render_POST(self, request):
newdata = request.content.getvalue()
print(newdata)
return ''
root = Resource()
root.putChild("foo", Foo())
application = Application("cgi.parse_multipart test")
TCPServer(8080, Site(root)).setServiceParent(application)
2. Save the following code as client.py (python3-httplib2 is required)
#!/usr/bin/env python
import httplib2
def http_request(url, method, body=None, headers=None, insecure=False):
"""Issue an http request."""
http = httplib2.Http(disable_ssl_certificate_validation=insecure)
if isinstance(url, bytes):
url = url.decode("ascii")
return http.request(url, method, body=body, headers=headers)
url = "http://localhost:8080"
method = "POST"
headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
emptyh = '--8825899812428059282\n\n--8825899812428059282--'
print("== BODY: " + emptyh + "\n")
response, content = http_request(url, method, emptyh, headers)
3. Run the server with "twistd3 -y webserver.py"
4. Run the client
5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)
2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
why = selectable.doRead()
File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
return self._dataReceived(data)
File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
return self._channel.dataReceived(data)
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
return basic.LineReceiver.dataReceived(self, data)
File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
why = self.rawDataReceived(data)
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
self._transferDecoder.dataReceived(data)
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
finishCallback(data[contentLength:])
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
self.allContentReceived()
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
self.args.update({
File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
x.encode('iso-8859-1'): \
builtins.AttributeError: 'NoneType' object has no attribute 'encode'
[regression potential]
This affects the returned dictionaries with non-str keys, which were
discarded in python3.6 or earlier before they reached twisted, so
patching this will make its behavior consistent.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/1915819/+subscriptions
More information about the foundations-bugs
mailing list