[Bug 1915819] Re: 'NoneType' object has no attribute 'encode' in requestReceived() when multipart body doesn't include content-disposition

Mon Feb 22 16:04:21 UTC 2021

Hello Victor, or anyone else affected,

Accepted twisted into groovy-proposed. The package will build now and be
available at
https://launchpad.net/ubuntu/+source/twisted/18.9.0-11ubuntu0.20.10.1 in
a few hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
to enable and use -proposed.  Your feedback will aid us getting this
update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, what testing has been
performed on the package and change the tag from verification-needed-
groovy to verification-done-groovy. If it does not fix the bug for you,
please add a comment stating that, and change the tag to verification-
failed-groovy. In either case, without details of your testing we will
not be able to proceed.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance for helping!

N.B. The updated package will be released to -updates after the bug(s)
fixed by this package have been verified and the package has been in
-proposed for a minimum of 7 days.

** Changed in: twisted (Ubuntu Groovy)
       Status: In Progress => Fix Committed

** Tags added: verification-needed verification-needed-groovy

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to twisted in Ubuntu.
https://bugs.launchpad.net/bugs/1915819

Title:
  'NoneType' object has no attribute 'encode' in requestReceived() when
  multipart body doesn't include content-disposition

Status in twisted package in Ubuntu:
  In Progress
Status in twisted source package in Focal:
  In Progress
Status in twisted source package in Groovy:
  Fix Committed
Status in twisted source package in Hirsute:
  In Progress

Bug description:
  [impact]

  python-twisted errors out with "'NoneType' object has no attribute
  'encode' in requestReceived()" when it tries to parse a multipart mime
  message and python3.7+ is used. This happens because before commit
  cc3fa20 in cpython, cgi.parse_multipart ignored parts without a name
  defined in "content-disposition" (or parts without headers for that
  matter) but after 3.7+ the return of the function can now contain
  NoneType keys, which fail to encode.

  [scope]

  Even though this bug affects all python3-twisted releases, I'll
  backport the fix just to Focal, Groovy and Hirsute. Bionic and Xenial
  do not have Python 3.7 as the default interpreter (required to trigger
  the issue), and the delta in python3-twisted might be to big to
  backport as the current packages do not contemplate _PY37PLUS at all.

  Fixed upstream with commit 310496249, available since 21.2.0rc1

  [test case]

  1. Save the following code as webserver.py

  from twisted.application.internet import TCPServer
  from twisted.application.service import Application
  from twisted.web.resource import Resource
  from twisted.web.server import Site

  class Foo(Resource):
      def render_POST(self, request):
          newdata = request.content.getvalue()
          print(newdata)
          return ''

  root = Resource()
  root.putChild("foo", Foo())
  application = Application("cgi.parse_multipart test")
  TCPServer(8080, Site(root)).setServiceParent(application)

  2. Save the following code as client.py (python3-httplib2 is required)

  #!/usr/bin/env python
  import httplib2

  def http_request(url, method, body=None, headers=None, insecure=False):
      """Issue an http request."""
      http = httplib2.Http(disable_ssl_certificate_validation=insecure)
      if isinstance(url, bytes):
          url = url.decode("ascii")
      return http.request(url, method, body=body, headers=headers)

  url = "http://localhost:8080"
  method = "POST"
  headers = {'Content-Type': 'multipart/form-data; boundary="8825899812428059282"'}
  emptyh = '--8825899812428059282\n\n--8825899812428059282--'

  print("== BODY: " + emptyh + "\n")
  response, content = http_request(url, method, emptyh, headers)

  3. Run the server with "twistd3 -y webserver.py"
  4. Run the client
  5. twistd will fail to encode the key and will drop this traceback in the log file (twistd.log)

  2021-02-16T13:41:39+0100 [_GenericHTTPChannelProtocol,7,127.0.0.1] Unhandled Error
          Traceback (most recent call last):
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 103, in callWithLogger
              return callWithContext({"system": lp}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 86, in callWithContext
              return context.call({ILogContext: newCtx}, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
              return self.currentContext().callWithContext(ctx, func, *args, **kw)
            File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
              return func(*args,**kw)
          --- <exception caught here> ---
            File "/usr/lib/python3/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
              why = selectable.doRead()
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 243, in doRead
              return self._dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/internet/tcp.py", line 249, in _dataReceived
              rval = self.protocol.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2952, in dataReceived
              return self._channel.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2245, in dataReceived
              return basic.LineReceiver.dataReceived(self, data)
            File "/usr/lib/python3/dist-packages/twisted/protocols/basic.py", line 579, in dataReceived
              why = self.rawDataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2252, in rawDataReceived
              self._transferDecoder.dataReceived(data)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 1699, in dataReceived
              finishCallback(data[contentLength:])
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2115, in _finishRequestBody
              self.allContentReceived()
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 2224, in allContentReceived
              req.requestReceived(command, path, version)
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 898, in requestReceived
              self.args.update({
            File "/usr/lib/python3/dist-packages/twisted/web/http.py", line 899, in <dictcomp>
              x.encode('iso-8859-1'): \
          builtins.AttributeError: 'NoneType' object has no attribute 'encode'

  [regression potential]

  This affects the returned dictionaries with non-str keys, which were
  discarded in python3.6 or earlier before they reached twisted, so
  patching this will make its behavior consistent.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/twisted/+bug/1915819/+subscriptions