[Bug 1103047] Re: mountall causes automatic mounting of gluster shares to fail
Bruno Léon
bruno.leon at nividic.org
Thu Mar 21 21:21:31 UTC 2013
Ok so the situation was as following.
In boot.log we have:
mounted event handled for /var/www
local 2/3 remote 1/1 virtual 9/10 swap 0/0
Mount failed. Please check the log file for more details.
mountall: mount /var/www [337] terminated with status 1
Filesystem could not be mounted: /var/www
The thing is the log file mentionned, which I suppose to be the gluster log file, does not exist.
Multiple attempts (and this one I do not explain) permitted me to have one time in boot.log a message stating that /va/log/glusterfs/var-www.log did not exist.
Actually the folder exist, and permissions are fine as well.
The only thing is that actually my /var/log is a dedicated partition like often on servers.
I commented this out in fstab for a try, and managed to have the /va/log/glusterfs/var-www.log writtend at boot time.
[2013-03-21 21:55:45.968441] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2013-03-21 21:55:45.970384] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2013-03-21 21:55:45.970471] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host storage.domain.com
[2013-03-21 21:55:45.970503] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2013-03-21 21:55:45.970523] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: 2 connect attempts left
The fstab line is:
storage.domain.com:web-g-www /var/www glusterfs defaults,_netdev,backupvolfile-server=172.18.2.130,fetch-attempts=1 0 0
I'm not sure where the "2 connect attempts left" is coming from as I
indicated fetch-attempts=1 is the fstab.
Still we can see that DNS resolution is failing, may be indicating that
the network is not really up when mountall is trying to mount glusterfs
filesytems.
Setting fetch-attempts=3 in the fstab managed to have all filesystem
properly mounted, including glusterfs.
So this tell us a few things:
- network does not seem to be up when mountall try to mount glustersfs filesystems (at least DNS is failing)
- there may be a missing dependency on having local fs up before trying to mount remote ones, because obviously the "log" file that could help in debugging this issue is not available for write when mountall (thus glusterfs) tries to write to it. (this is fixed by not separating /var/log but on a server may be dangerous then)
Hope this helps
--
Bruno
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mountall in Ubuntu.
https://bugs.launchpad.net/bugs/1103047
Title:
mountall causes automatic mounting of gluster shares to fail
Status in “mountall” package in Ubuntu:
Incomplete
Bug description:
Ubuntu Server 12.04.1 LTS (GNU/Linux 3.2.0-29-virtual x86_64)
glusterfs 3.3.1 (from semiosis PPA)
mountall 2.36.3
We are running an OpenStack cloud (essex). We use Gluster as a network filesystem for storage of data.
When we updated mountall on our client VMs we experienced a failure of the automatic mounting of the gluster-shares at boot-time. The gluster-shares can be mounted manually when the VMs are booted and online.
When we downgraded the mountall package from new version 2.36.3 to old
version 2.36, the gluster-shares mounted again during boot.
boot.log (added --verbose to /etc/init/mountall.conf)
=======
Connected to Plymouth
/ is local
/proc is virtual
/sys is virtual
/sys/fs/fuse/connections is virtual
/sys/kernel/debug is virtual
/sys/kernel/security is virtual
/dev is virtual
/dev/pts is virtual
/tmp is local
/run is virtual
/run/lock is virtual
/run/shm is virtual
/home is nowait
/scratch is nowait
/opt is nowait
/mnt is nowait
mounting event sent for /sys/fs/fuse/connections
mounting event sent for /sys/kernel/debug
mounting event sent for /sys/kernel/security
mounting event sent for /run/lock
mounting event sent for /run/shm
mounting event sent for /
mounted event handled for /sys
swap finished
local 0/2 remote 0/0 virtual 1/10 swap 0/0
mounted event handled for /dev/pts
local 0/2 remote 0/0 virtual 2/10 swap 0/0
mounting event handled for /sys/fs/fuse/connections
mounting /sys/fs/fuse/connections
mount /sys/fs/fuse/connections [282] exited normally
mounting event handled for /sys/kernel/debug
mounting /sys/kernel/debug
mount /sys/kernel/debug [296] exited normally
mounting event handled for /sys/kernel/security
mounting /sys/kernel/security
mount /sys/kernel/security [299] exited normally
mounting event handled for /run/lock
mounting /run/lock
mount /run/lock [301] exited normally
mounting event handled for /run/shm
mounting /run/shm
mount /run/shm [304] exited normally
mounted event handled for /proc
local 0/2 remote 0/0 virtual 3/10 swap 0/0
mounting event handled for /
remounting /
mount / [310] exited normally
mounted event handled for /dev
mount / [321] exited normally
mount /proc [322] exited normally
mount /sys [323] exited normally
mount /sys/fs/fuse/connections [324] exited normally
mount /sys/kernel/debug [325] exited normally
mount /sys/kernel/security [326] exited normally
mount /dev [327] exited normally
mount /dev/pts [328] exited normally
mount /run [329] exited normally
mount /run/lock [330] exited normally
mount /run/shm [331] exited normally
local 0/2 remote 0/0 virtual 4/10 swap 0/0
mounted event handled for /sys/fs/fuse/connections
local 0/2 remote 0/0 virtual 5/10 swap 0/0
mounted event handled for /sys/kernel/security
local 0/2 remote 0/0 virtual 6/10 swap 0/0
mounting event sent for /tmp
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event sent for /mnt
mounted event handled for /sys/kernel/debug
local 0/2 remote 0/0 virtual 7/10 swap 0/0
mounted event handled for /run/lock
local 0/2 remote 0/0 virtual 8/10 swap 0/0
mounted event handled for /run/shm
local 0/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /tmp
mounting event handled for /home
mounting /home
Mount failed. Please check the log file for more details.
mountall: mount /home [333] terminated with status 1
mountall: Filesystem could not be mounted: /home
mounting event handled for /scratch
mounting /scratch
Mount failed. Please check the log file for more details.
mountall: mount /scratch [389] terminated with status 1
mountall: Filesystem could not be mounted: /scratch
mounting event handled for /opt
mounting /opt
Mount failed. Please check the log file for more details.
mountall: mount /opt [424] terminated with status 1
mountall: Filesystem could not be mounted: /opt
mounting event handled for /mnt
mounting /mnt
mount /mnt [459] exited normally
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounted event handled for /tmp
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
mounted event handled for /mnt
local 1/2 remote 0/0 virtual 9/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
cloud-init start-local running: Tue, 22 Jan 2013 15:55:55 +0000. up 3.83 seconds
mounted event handled for /run
virtual finished
remote finished
local 1/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
no instance data found in start-local
mounting event sent for /mnt
mounting event handled for /mnt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Received SIGUSR1 (network device up)
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
ci-info: lo : 1 127.0.0.1 255.0.0.0 .
ci-info: eth0 : 1 192.168.0.12 255.255.255.0 fa:16:3e:0a:0b:4c
ci-info: route-0: 0.0.0.0 192.168.0.11 0.0.0.0 eth0 UG
ci-info: route-1: 192.168.0.0 0.0.0.0 255.255.255.0 eth0 U
cloud-init start running: Tue, 22 Jan 2013 15:55:57 +0000. up 5.78 seconds
found data source: DataSourceEc2
mounted event handled for /
local finished
All filesystems mounted
local 2/2 remote 0/0 virtual 10/10 swap 0/0
mounting event sent for /home
mounting event sent for /scratch
mounting event sent for /opt
mounting event handled for /home
mounting event handled for /scratch
mounting event handled for /opt
Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
* Starting AppArmor profiles [ OK ]
Starting memcached: * Stopping System V initialisation compatibility [ OK ]
* Starting System V runlevel compatibility [ OK ]
* Starting automatic crash report generation [ OK ]
* Starting deferred execution scheduler [ OK ]
* Starting regular background program processing daemon [ OK ]
* Starting ACPI daemon [ OK ]
* Starting save kernel messages [ OK ]
* Starting CPU interrupts balancing daemon [ OK ]
* Stopping save kernel messages [ OK ]
* Starting crash report submission daemon [ OK ]
* Stopping Handle applying cloud-config [ OK ]
memcached.
* Starting Name Service Cache Daemon nscd [ OK ]
* Starting Postfix Mail Transport Agent postfix [ OK ]
* Starting HTTP accelerator varnishd [ OK ]
* Starting NTP server ntpd [ OK ]
landscape-client is not configured, please run landscape-config.
Warning: DocumentRoot [/home/www] does not exist
* Starting web server apache2 [ OK ]
* Stopping System V runlevel compatibility [ OK ]
* Starting execute cloud user/final scripts [ OK ]
/etc/fstab
========
LABEL=cloudimg-rootfs / ext4 defaults 0 0
10.0.0.100:/home /home glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/scratch /scratch glusterfs defaults,nobootwait,comment=cloudconfig 0 0
10.0.0.100:/ubuntu_11_10_opt /opt glusterfs defaults,nobootwait,comment=cloudconfig 0 0
/dev/vdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2
gluster log of home volume
======================
[2013-01-22 13:17:58.788134] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2013-01-22 13:17:58.862333] E [common-utils.c:125:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2013-01-22 13:17:58.862395] E [name.c:243:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 10.0.0.100
[2013-01-22 13:17:58.862514] E [glusterfsd-mgmt.c:1787:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2013-01-22 13:17:58.862546] I [glusterfsd-mgmt.c:1790:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
[2013-01-22 13:17:58.863087] W [glusterfsd.c:831:cleanup_and_exit] (-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x1ff) [0x7f5d23e811af] (-->/usr/lib/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f5d237cd922] (-->/usr/sbin/glusterfs(+0xd486) [0x7f5d23e81486]))) 0-: received signum (1), shutting down
[2013-01-22 13:17:58.863134] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/home'.
Please let me know if you need any more info.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/1103047/+subscriptions
More information about the foundations-bugs
mailing list