[Bug 1838109] Re: civetweb does not allow tuning of maximum socket connections

[Edward Hope-Morley]

xenial-queens verified

test output:

root at juju-662fa0-xq-sru-test-7:~# sed -i -r 's/(rgw frontends = civetweb port=70).*/\1 max_connections=10000000000000000000000000000000/g' /etc/ceph/ceph.conf
root at juju-662fa0-xq-sru-test-7:~# systemctl restart ceph-radosgw at rgw.`hostname`
root at juju-662fa0-xq-sru-test-7:~# lsof -i :70
root at juju-662fa0-xq-sru-test-7:~# grep max_conn /var/log/ceph/ceph-client.rgw.juju-662fa0-xq-sru-test-7.log 
2019-10-30 11:11:22.085139 7f8c89268000  0 civetweb: 0x55e1294108e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:23.446174 7f961780d000  0 civetweb: 0x55d4401128e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:25.497004 7f3569786000  0 civetweb: 0x555c414398e0: max_connections value "10000000000000000000000000000000" is invalid
root at juju-662fa0-xq-sru-test-7:~# sed -i -r 's/(rgw frontends = civetweb port=70).*/\1 max_connections=1000/g' /etc/ceph/ceph.conf
root at juju-662fa0-xq-sru-test-7:~# systemctl restart ceph-radosgw at rgw.`hostname`
root at juju-662fa0-xq-sru-test-7:~# lsof -i :70
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
radosgw 7663 ceph   37u  IPv4  25171      0t0  TCP *:gopher (LISTEN)
root at juju-662fa0-xq-sru-test-7:~# grep max_conn /var/log/ceph/ceph-client.rgw.juju-662fa0-xq-sru-test-7.log 
2019-10-30 11:11:22.085139 7f8c89268000  0 civetweb: 0x55e1294108e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:23.446174 7f961780d000  0 civetweb: 0x55d4401128e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:25.497004 7f3569786000  0 civetweb: 0x555c414398e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:26.860549 7fbbe5e42000  0 civetweb: 0x55b8810748e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:11:28.070096 7f093f962000  0 civetweb: 0x55c44a9ac8e0: max_connections value "10000000000000000000000000000000" is invalid
2019-10-30 11:12:15.341958 7f14baf35000  1 mgrc service_daemon_register rgw.juju-662fa0-xq-sru-test-7 metadata {arch=x86_64,ceph_version=ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable),cpu=Intel Xeon E312xx (Sandy Bridge, IBRS update),distro=ubuntu,distro_description=Ubuntu 16.04.6 LTS,distro_version=16.04,frontend_config#0=civetweb port=70 max_connections=1000,frontend_type#0=civetweb,hostname=juju-662fa0-xq-sru-test-7,kernel_description=#195-Ubuntu SMP Tue Oct 1 09:35:25 UTC 2019,kernel_version=4.4.0-166-generic,mem_swap_kb=0,mem_total_kb=2047984,num_handles=1,os=Linux,pid=7663,zone_id=a3fe50ec-b375-4b10-ba6a-8e3bb8a15e89,zone_name=default,zonegroup_id=c35bd079-a42d-4540-acd1-bf77ba2ee5c8,zonegroup_name=default}


** Tags removed: verification-queens-needed
** Tags added: verification-queens-done

-- 
You received this bug notification because you are a member of Ubuntu
OpenStack, which is subscribed to Ubuntu Cloud Archive.
https://bugs.launchpad.net/bugs/1838109

Title:
  civetweb does not allow tuning of maximum socket connections

Status in Ceph:
  New
Status in OpenStack ceph-radosgw charm:
  Triaged
Status in Ubuntu Cloud Archive:
  Fix Committed
Status in Ubuntu Cloud Archive queens series:
  Fix Committed
Status in Ubuntu Cloud Archive rocky series:
  Fix Released
Status in Ubuntu Cloud Archive stein series:
  Fix Released
Status in Ubuntu Cloud Archive train series:
  Fix Committed
Status in ceph package in Ubuntu:
  Fix Released
Status in ceph source package in Bionic:
  Fix Released
Status in ceph source package in Disco:
  Fix Released
Status in ceph source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  RADOS gateway can run out of sockets prior to consuming the CPU and memory resources on the server on which it is running.

  [Test Case]
  Deploy RGW to a large server; scale test - RGW processes will only be able to service around 100 open connections.

  [Regression Potential]
  Medium; the fix introduces a new configuration option for civetweb (the web connector for RGW) to allow the max connections to be set via configuration, rather than being set during compilation; improvement has been accepted upstream in the civetweb project.

  [Original Bug Report]
  Civetweb does not offer an option for configuring the maximum number of sockets available. Some users run out of sockets and are left with no workaround.

  This patch adds a new user-configurable parameter, "max_connections".

  See:
  https://github.com/civetweb/civetweb/issues/775

To manage notifications about this bug go to:
https://bugs.launchpad.net/ceph/+bug/1838109/+subscriptions