[Bug 1835464] Re: nginx service fails after libssl update due to low entropy at boot
Dietmar May
1835464 at bugs.launchpad.net
Mon Jul 8 17:03:22 UTC 2019
@racb
I'm not sure that I would consider it normal or expected, though, for
system services to suddenly stop working due to regular updates, and for
a server to suddenly become unreachable and unresponsive just because it
was updated.
On the other hand, it's certainly not desirable for a system to silently
operate with poor entropy and poor encryption quality.
In my case, this is easily resolved due to the hardware RNG on the TI
AM335X chip.
However, AFAIK a Raspberry PI does not have a hardware RNG, nor do many
embedded processors / systems - meaning they would have low entropy at
boot, and rng-tools most likely won't help.
Without looking at any code, here are a few observations.
Does nginx really need to make this blocking call to openssl when the
service starts? or only when the first https request is made to the
service? That is, if no https request comes in for 2 min, or 10 min,
maybe there would be sufficient entropy by then due to system activity.
Does openssl really need to block on initialization until sufficient
entropy exists? Or could it defer that until some subsequent call that
does actually need adequate entropy? In other words, would moving this
blocking behavior to a different function satisfy the security need that
led to its implementation, without potentially blocking systemd services
at boot time?
Finally, I have a couple of the same devices that do not exhibit this
blocking behavior. I'm not sure exactly why, but the difference appears
somehow related to the way updates are applied. I've noticed a file
'/.rnd' (from memory) which is used and/or generated by openssl. Looks
like this file is used as an entropy seed. Once deleted (and the
hardware RNG is not used), the nginx systemd service will start blocking
and timing out. Attempts to create this file manually using openssl do
not allow the nginx service to start successfully at boot.
Maybe the simple fix is to find the right way to create and manage the
/.rnd file on devices with low entropy?
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to openssl in Ubuntu.
https://bugs.launchpad.net/bugs/1835464
Title:
nginx service fails after libssl update due to low entropy at boot
Status in nginx package in Ubuntu:
Opinion
Status in openssl package in Ubuntu:
New
Status in nginx source package in Bionic:
Opinion
Status in openssl source package in Bionic:
New
Bug description:
After updating libssl and related packages, nginx will no longer
autostart at system boot.
Immediately after boot, nginx.service is in a failed state.
# service nginx status
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: failed (Result: timeout) since Fri 2018-08-24 21:27:51 UTC; 32min ago
Docs: man:nginx(8)
systemd[1]: Starting A high performance web server and a reverse proxy server...
systemd[1]: nginx.service: Start-pre operation timed out. Terminating.
systemd[1]: nginx.service: Failed with result 'timeout'.
systemd[1]: Failed to start A high performance web server and a reverse proxy server.
The service can be manually started after boot.
# service nginx start
# service nginx status
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2018-08-24 22:02:06 UTC; 2s ago
Docs: man:nginx(8)
Process: 2704 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 2703 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 2705 (nginx)
CGroup: /system.slice/nginx.service
├─2705 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
└─2706 nginx: worker process
systemd[1]: Starting A high performance web server and a reverse proxy server...
systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
systemd[1]: Started A high performance web server and a reverse proxy server.
This happens on an ARMHF based microcontroller running ubuntu 18.04.2 raspi server distribution with a stock kernel.org 4.9-181 kernel.
Ubuntu repositories are not accessible from the device, so packages
are copied to the device, and apt install is used to upgrade them:
apt install --no-install-recommends $dir/updates/system/*.deb |
logger 2>&1
The following is a list of packages that, when upgraded, cause the nginx systemd service to fail to autostart at boot.
201,205c201,205
< ii libpython2.7:armhf 2.7.15-4ubuntu4~18.04 armhf Shared Python runtime library (version 2.7)
< ii libpython2.7-minimal:armhf 2.7.15-4ubuntu4~18.04 armhf Minimal subset of the Python language (version 2.7)
< ii libpython2.7-stdlib:armhf 2.7.15-4ubuntu4~18.04 armhf Interactive high-level object-oriented language (standard library, version 2.7)
< ii libpython3.6-minimal:armhf 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6)
< ii libpython3.6-stdlib:armhf 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (standard library, version 3.6)
---
> ii libpython2.7:armhf 2.7.15~rc1-1ubuntu0.1 armhf Shared Python runtime library (version 2.7)
> ii libpython2.7-minimal:armhf 2.7.15~rc1-1ubuntu0.1 armhf Minimal subset of the Python language (version 2.7)
> ii libpython2.7-stdlib:armhf 2.7.15~rc1-1ubuntu0.1 armhf Interactive high-level object-oriented language (standard library, version 2.7)
> ii libpython3.6-minimal:armhf 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6)
> ii libpython3.6-stdlib:armhf 3.6.7-1~18.04 armhf Interactive high-level object-oriented language (standard library, version 3.6)
225c225
< ii libssl1.1:armhf 1.1.1-1ubuntu2.1~18.04.2 armhf Secure Sockets Layer toolkit - shared libraries
---
> ii libssl1.1:armhf 1.1.0g-2ubuntu4.3 armhf Secure Sockets Layer toolkit - shared libraries
272c272
< ii openssl 1.1.1-1ubuntu2.1~18.04.2 armhf Secure Sockets Layer toolkit - cryptographic utility
---
> ii openssl 1.1.0g-2ubuntu4.3 armhf Secure Sockets Layer toolkit - cryptographic utility
282,283c282,283
< ii python3.6 3.6.8-1~18.04.1 armhf Interactive high-level object-oriented language (version 3.6)
< ii python3.6-minimal 3.6.8-1~18.04.1 armhf Minimal subset of the Python language (version 3.6)
---
> ii python3.6 3.6.7-1~18.04 armhf Interactive high-level object-oriented language (version 3.6)
> ii python3.6-minimal 3.6.7-1~18.04 armhf Minimal subset of the Python language (version 3.6)
nginx is used primarily as an https front-end for web services on the device.
libssl is the core dependency for all of the packages in the group
that, when upgraded, causes nginx to fail.
The nginx configuration includes the following SSL settings:
http {
##
# SSL Settings
##
ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
ssl_prefer_server_ciphers on;
}
server {
listen 443 ssl;
ssl_certificate /etc/certs/cert.crt;
ssl_certificate_key /etc/certs/cert.key;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!MD5;
}
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1835464/+subscriptions
More information about the foundations-bugs
mailing list