Vagrant provisioning script runs before VM is fully up? - linux

Question

I have a Vagrant box that I am preparing, along with a shell script as the provisioning file. I have spent a couple of days getting it working, and it now seems stable and complete. The base box is Ubuntu 12.04 (32-bit), with Postgres, Redis and Memcached running on the VM. The provisioning script sets up Nginx configuration, creates a blank database, and does some basic housekeeping.
When I came to package the VM, and attempt to re-run it on a different machine at home, I kept coming across a problem on the first run (vagrant up) whereby none of the services were running - and so my attempt to run dropdb or createdb failed.
Digging in to why this occurred (and I'm an ex-Windows guy, so this took some doing) I found myself in the bowels of run levels and the /etc/rc[0-6,S].d files.
I have relevant S (start) files for the three services I'm interested in:
vagrant#precise32:~$ ls -l /etc/rc2.d
total 4
-rw-r--r-- 1 root root 677 Apr 14 2012 README
lrwxrwxrwx 1 root root 20 Dec 29 10:05 S19postgresql -> ../init.d/postgresql
lrwxrwxrwx 1 root root 19 Dec 29 10:05 S20memcached -> ../init.d/memcached
lrwxrwxrwx 1 root root 15 Dec 29 10:05 S20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 22 Dec 29 10:05 S20redis-server -> ../init.d/redis-server
...
and K files for run level 0 (shutdown), and so all seems in order:
vagrant#precise32:~$ ls -l /etc/rc0.d
total 4
lrwxrwxrwx 1 root root 19 Dec 29 10:05 K20memcached -> ../init.d/memcached
lrwxrwxrwx 1 root root 15 Dec 29 10:05 K20nginx -> ../init.d/nginx
lrwxrwxrwx 1 root root 22 Dec 29 10:05 K20redis-server -> ../init.d/redis-server
lrwxrwxrwx 1 root root 20 Dec 29 10:05 K21postgresql -> ../init.d/postgresql
....
This seemed to suggest that the underlying VM runlevel wasn't 2, and so in order to debug this issue, I created a new provisioning script to output a.) the runlevel at the time of provisioning, and b.) whether the expected processes were running (memcache, prostgres, redis):
ps aux | grep memcache
ps aux | grep postgres
ps aux | grep redis
# expected output is 'N 2'
runlevel
I ran vagrant destroy, then vagrant up on this, and the result is as follows:
[default] Running provisioner: Vagrant::Provisioners::Shell...
root 791 0.0 0.2 4624 840 ? S 10:33 0:00 grep memcache
root 793 0.0 0.2 4624 836 ? S 10:33 0:00 grep postgres
root 795 0.0 0.2 4624 840 ? S 10:33 0:00 grep redis
unknown
i.e. the services aren't running at the time the provisioning script is run, and more confusingly, the runlevel command isn't even recognised.
If I then repeatedly re-run the provisioning script on the running VM, using vagrant provision, I get the same results for the first few times I run it, and then eventually (after 2-3mins) I see what I had expected first time round:
[default] Running provisioner: Vagrant::Provisioners::Shell...
memcache 1103 0.2 0.2 46336 1072 ? Sl 10:56 0:00 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
root 1267 0.0 0.2 4624 840 ? S 10:56 0:00 grep memcache
postgres 1073 13.0 2.0 50440 7828 ? S 10:56 0:02 /usr/lib/postgresql/9.1/bin/postgres -D /var/lib/postgresql/9.1/main -c config_file=/etc/postgresql/9.1/main/postgresql.conf
postgres 1077 0.3 0.3 50440 1248 ? Ss 10:56 0:00 postgres: writer process
postgres 1078 0.3 0.3 50440 1244 ? Ss 10:56 0:00 postgres: wal writer process
postgres 1079 0.1 0.6 50860 2296 ? Ss 10:56 0:00 postgres: autovacuum launcher process
postgres 1080 0.0 0.3 20640 1284 ? Ss 10:56 0:00 postgres: stats collector process
root 1269 0.0 0.2 4624 836 ? S 10:56 0:00 grep postgres
redis 1123 0.6 0.2 3292 1036 ? Ss 10:56 0:00 /usr/bin/redis-server /etc/redis/redis.conf
root 1271 0.0 0.2 4624 840 ? S 10:56 0:00 grep redis
N 2
It looks like it's just taking a bit of time for everything to come up, which makes sense, hwoever presents a huge problem for me in that the provisioning script will always fail first time.
Is this a known situation, and if so, what is the solution? Ideally, the provisioning script would pause until the runlevel had changed to 2, i.e. the box was ready to accept the shell commands.
[UPDATE: HACK]
I have managed to work around this issue by hacking together the following script:
while [ "`runlevel`" = "unknown" ]; do
echo "runlevel is 'unknown' - waiting for 10s"
sleep 10
done
echo "runlevel is now valid ('`runlevel`'), kicking off provisioning..."
I have saved this as 'pre-provision.sh' and my Vagrantfile now looks like:
# Enable provisioning with a shell script.
config.vm.provision :shell, :path => "pre-provision.sh"
config.vm.provision :shell, :path => "provision.sh", :args => "myapp"
which gives the following ouput:
[default] Running provisioner: Vagrant::Provisioners::Shell...
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is 'unknown' - waiting for 10s
runlevel is now valid ('N 2'), kicking off provisioning...
[default] Running provisioner: Vagrant::Provisioners::Shell...
...
and then the original provision.sh runs, and everything is OK.
I have not marked this as the answer (although it is an answer) because I still want to know what I should have done - this cannot be the way it works, surely?

Answer 1

Turns out that the easiest way to do this is to look for the PID file for the relevant process (see this article for an explanation of the pid file - What is a .pid file and what does it contain?)
NGINX_PID=/var/run/nginx.pid
...
## set up nginx configuration using .config from shared directory
if [ ! -f $NGINX_PID ]; then
echo "---> Waiting for for Nginx process to spin up"
while [ ! -f $NGINX_PID ]; do
echo .
sleep 1
done
fi