How to start multiple processes inside a linux service construct? - linux

Question

I am attempting to start multiple memcached processes from the linux service framework using the following logic:
RETVAL=0
pcount="$CACHES"
if [ ! -z "$pcount" ]; then
while [ $pcount -gt 0 ];
do
(( pcount-- ))
(( port=PORT + pcount ))
daemon --pidfile ${pidfile}${pcount}.pid memcached -d -p $port -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}${pcount}.pid $OPTIONS
(( RETVAL=RETVAL + $? ))
done
else
daemon --pidfile ${pidfile}.pid memcached -d -p $PORT -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}.pid $OPTIONS
RETVAL=$?
fi
When run using the command service memcached start, it creates and updates pid files for each cycle in the loop, but only the last instance of the process remains running. That is, while each of the /var/run/memcached/memcached(1 through 5).pid are created and updated with a PID; those processes do not exist. /var/run/memcached/memcached0.pid is also created and updated and the PID points to a running process.
I turned on tracing and I can see that the loop is executed and the process invocation is made; however the process does not start (or likely, starts and immediately terminates so I dont see it as having started).
On the other hand, running this script directly as /etc/init.d/memcached start results in all the processes getting started correctly.
Can someone help me understand why the service framework is preventing the starting of the other instances except the last one?

Answer 1

As suggested by #nos, I added strace -f to trace the calls during the service memcached start operation. I compared the traced calls between the unsuccessful/terminated process and the successful process. The only lines of significant difference that I found were:
< bind(26, {sa_family=AF_INET, sin_port=htons(11216), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
< dup(2) = 27
< fcntl(27, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(27, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(27, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d780) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(27, 0, SEEK_CUR) = 0
< write(27, "bind(): Permission denied\n", 26) = 26
< close(27) = 0
< munmap(0x7f5dae958000, 4096) = 0
< close(26) = 0
< dup(2) = 26
< fcntl(26, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(26, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(26, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d730) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(26, 0, SEEK_CUR) = 0
< write(26, "failed to listen on TCP port 112"..., 54) = 54
< close(26) = 0
< munmap(0x7f5dae958000, 4096) = 0
< exit_group(71) = ?
---
> bind(26, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> listen(26, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 26, {EPOLLIN, {u32=26, u64=26}}) = 0
> socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 27
> fcntl(27, F_GETFL) = 0x2 (flags O_RDWR)
> fcntl(27, F_SETFL, O_RDWR|O_NONBLOCK) = 0
> setsockopt(27, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_LINGER, {onoff=0, linger=0}, 8) = 0
> setsockopt(27, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> bind(27, {sa_family=AF_INET6, sin6_port=htons(11211), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
> listen(27, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 27, {EPOLLIN, {u32=27, u64=27}}) = 0
> socket(PF_NETLINK, SOCK_RAW, 0) = 28
> bind(28, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
> getsockname(28, {sa_family=AF_NETLINK, pid=31943, groups=00000000}, [12]) = 0
> gettimeofday({1393735036, 191154}, NULL) = 0
> sendto(28, "\24\0\0\0\26\0\1\3|\265\22S\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0|\265\22S\307|\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"#\0\0\0\24\0\2\0|\265\22S\307|\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0|\265\22S\307|\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
> close(28) = 0
> socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 28
The top (<) one being from a terminated process and bottom one (>) from the last (successful) process. It is clear that the process is terminating due to lack of permission to bind to the port. On looking further, I realized that the SELinux was set to ENFORCE, which was preventing the memcached service from binding to port other than 11211 (the default port).
To the best of what I could figure, when I was running it without the service command, the behavior was simply that of a process (not a service) and hence the binding was not being enforced.
Turning off the ENFORCED mode of SELinux, got the service memcached start command working!