[Bug 503180] Re: [SRU] eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
Dustin Kirkland
dustin.kirkland at gmail.com
Thu Feb 11 20:56:55 GMT 2010
SRU instructions in the bug description above.
** Description changed:
Using 1.6.2~bzr1120-0ubuntu1, I can't get the credentials:
ubuntu at uec-cc:~$ sudo euca_conf --get-credentials mycreds.zip
ERROR: you need to be on the CLC host and the CLC needs to be running.
A wget on the register url also times out:
ubuntu at uec-cc:~$ wget -T 10 -t 1 -O - --no-check-certificate https://127.0.0.1:8443/register
--2010-01-04 20:39:34-- https://127.0.0.1:8443/register
Connecting to 127.0.0.1:8443... connected.
WARNING: cannot verify 127.0.0.1's certificate, issued by `/C=US/ST=CA/L=Santa Barbara/O=db/OU=Eucalyptus/CN=localhost':
- Self-signed certificate encountered.
+ Self-signed certificate encountered.
WARNING: certificate common name `localhost' doesn't match requested host name `127.0.0.1'.
HTTP request sent, awaiting response... Read error (Connection timed out) in headers.
Giving up.
The eucalyptus-cloud process is running a listening on port 8443.
I can see the following errors in /var/log/eucalyptus/cloud-error.log:
0:36:55 [log:653891498 at qtp-1693378617-9] ERROR /register
java.lang.RuntimeException: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
- at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46)
- at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98)
- at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91)
- at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117)
- at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199)
- at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210)
- at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
- at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
- at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
- at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
- at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
- at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
- at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
- at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
- at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
- at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
- at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
- at org.mortbay.jetty.Server.handle(Server.java:326)
- at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
- at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
- at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
- at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
- at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
- at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
- at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
+ at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:46)
+ at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:98)
+ at com.eucalyptus.util.EntityWrapper.<init>(EntityWrapper.java:91)
+ at edu.ucsb.eucalyptus.util.EucalyptusProperties.getSystemConfiguration(EucalyptusProperties.java:117)
+ at edu.ucsb.eucalyptus.admin.server.Registration.getRegistrationId(Registration.java:199)
+ at edu.ucsb.eucalyptus.admin.server.Registration.doGet(Registration.java:210)
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
+ at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
+ at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
+ at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
+ at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
+ at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
+ at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
+ at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
+ at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
+ at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
+ at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
+ at org.mortbay.jetty.Server.handle(Server.java:326)
+ at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
+ at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
+ at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
+ at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
+ at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
+ at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
+ at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
Caused by: javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Cannot open connection
- at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
- at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
- at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40)
- ... 24 more
+ at org.hibernate.ejb.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:614)
+ at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:41)
+ at com.eucalyptus.util.TxHandle.<init>(TxHandle.java:40)
+ ... 24 more
Caused by: org.hibernate.exception.JDBCConnectionException: Cannot open connection
- at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
- at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
- at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
- at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
- at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
- at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
- at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
- at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
- at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
- ... 25 more
+ at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:97)
+ at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
+ at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:52)
+ at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:449)
+ at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
+ at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:142)
+ at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:85)
+ at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1353)
+ at org.hibernate.ejb.TransactionImpl.begin(TransactionImpl.java:38)
+ ... 25 more
Caused by: java.sql.SQLException: Connection is broken: java.net.SocketException: Connection timed out
- at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
- at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source)
- at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
- at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
- at java.lang.reflect.Method.invoke(Method.java:616)
- at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162)
- at $Proxy27.getAutoCommit(Unknown Source)
- at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81)
- at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
- ... 30 more
+ at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
+ at org.hsqldb.jdbc.jdbcConnection.getAutoCommit(Unknown Source)
+ at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source)
+ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+ at java.lang.reflect.Method.invoke(Method.java:616)
+ at org.logicalcobwebs.proxool.WrappedConnection.invoke(WrappedConnection.java:162)
+ at $Proxy27.getAutoCommit(Unknown Source)
+ at org.hibernate.connection.ProxoolConnectionProvider.getConnection(ProxoolConnectionProvider.java:81)
+ at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
+ ... 30 more
+
+ == SRU ==
+
+ IMPACT: Users of 9.10 UEC will often experience long, non-deterministic
+ delays (10-20 minutes in many cases) when restarting eucalyptus services
+ (which includes reboots, package upgrades, service restarts). This is
+ highly inconvenient, yielding UEC unusable until database network
+ connections reset.
+
+ HOW FIXED: The fix consists of two trivial iptables calls being added to
+ the eucalyptus upstart script. Upstream had these calls in there init
+ scripts, but were inadvertently dropped when porting Eucalyptus to
+ upstart. These iptables commands will ensure that the iptables kernel
+ module (and most importantly, the ip connection tracker) is loaded and
+ active before Eucalyptus comes up. WIthout said ip connection tracker,
+ Eucalyptus will often establish a connection to the database, then
+ iptables is loaded and connections are mangled, breaking the connection
+ to the database. The user will see the problem in any one of a number
+ of disguising ways (front end not working, api tools not responding,
+ etc). All of these problems are due to an inaccessible database. After
+ a while (10-20 minutes), Eucalyptus will reset the database connection.
+ With this fix, the above problems should never happen. Eucalyptus
+ should be back up and running within 1-2 minutes of boot (if not
+ immediately).
+
+ MINIMAL PATCH:
+ diff -u eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
+ --- eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
+ +++ eucalyptus-1.6~bzr931/debian/eucalyptus-common.eucalyptus.upstart
+ @@ -11,6 +11,10 @@
+ # Check if installed
+ [ -f /usr/sbin/euca_conf ] || { stop; exit 0; }
+
+ + # Ensure that the iptables module gets loaded here
+ + iptables -t nat -L -n >/dev/null
+ + iptables -L -n > /dev/null
+ +
+ mkdir -p /var/run/eucalyptus/net
+ chown eucalyptus:eucalyptus /var/run/eucalyptus /var/run/eucalyptus/net
+
+
+ REPRODUCING THE BUG:
+ Reboot your UEC (or sudo restart eucalyptus). If restarting eucalyptus takes a *long* time, you are experiencing one symptom of this bug. Once upstart thinks that eucalyptus is up, try: $(sudo wget --no-check-certificate https://localhost:8443) If this takes a long time, or fails to work, you are experiencing a symptom of this bug. Note that the problem is inherently due to a race condition, and therefore may not be immediately reproducible. Try rebooting/restarting a few times, and you will likely hit it.
+
+ REGRESSION POTENTIAL:
+ I cannot see any possible regression potential. The iptables modules will be loaded eventually. This patch just ensures that they get loaded before Eucalyptus tries to start services.
--
[SRU] eucalyptus-cloud doesn't reply to requests (eucalyptus doesn't work after reboot or services restart issues due to upstart networking behavior)
https://bugs.launchpad.net/bugs/503180
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to eucalyptus in ubuntu.
More information about the Ubuntu-server-bugs
mailing list