I am attempting to start multiple memcached processes from the linux service framework using the following logic:
RETVAL=0
pcount="$CACHES"
if [ ! -z "$pcount" ]; then
while [ $pcount -gt 0 ];
do
(( pcount-- ))
(( port=PORT + pcount ))
daemon --pidfile ${pidfile}${pcount}.pid memcached -d -p $port -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}${pcount}.pid $OPTIONS
(( RETVAL=RETVAL + $? ))
done
else
daemon --pidfile ${pidfile}.pid memcached -d -p $PORT -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}.pid $OPTIONS
RETVAL=$?
fi
When run using the command service memcached start, it creates and updates pid files for each cycle in the loop, but only the last instance of the process remains running. That is, while each of the /var/run/memcached/memcached(1 through 5).pid are created and updated with a PID; those processes do not exist. /var/run/memcached/memcached0.pid is also created and updated and the PID points to a running process.
I turned on tracing and I can see that the loop is executed and the process invocation is made; however the process does not start (or likely, starts and immediately terminates so I dont see it as having started).
On the other hand, running this script directly as /etc/init.d/memcached start results in all the processes getting started correctly.
Can someone help me understand why the service framework is preventing the starting of the other instances except the last one?
As suggested by #nos, I added strace -f to trace the calls during the service memcached start operation. I compared the traced calls between the unsuccessful/terminated process and the successful process. The only lines of significant difference that I found were:
< bind(26, {sa_family=AF_INET, sin_port=htons(11216), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
< dup(2) = 27
< fcntl(27, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(27, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(27, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d780) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(27, 0, SEEK_CUR) = 0
< write(27, "bind(): Permission denied\n", 26) = 26
< close(27) = 0
< munmap(0x7f5dae958000, 4096) = 0
< close(26) = 0
< dup(2) = 26
< fcntl(26, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(26, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(26, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d730) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(26, 0, SEEK_CUR) = 0
< write(26, "failed to listen on TCP port 112"..., 54) = 54
< close(26) = 0
< munmap(0x7f5dae958000, 4096) = 0
< exit_group(71) = ?
---
> bind(26, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> listen(26, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 26, {EPOLLIN, {u32=26, u64=26}}) = 0
> socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 27
> fcntl(27, F_GETFL) = 0x2 (flags O_RDWR)
> fcntl(27, F_SETFL, O_RDWR|O_NONBLOCK) = 0
> setsockopt(27, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_LINGER, {onoff=0, linger=0}, 8) = 0
> setsockopt(27, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> bind(27, {sa_family=AF_INET6, sin6_port=htons(11211), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
> listen(27, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 27, {EPOLLIN, {u32=27, u64=27}}) = 0
> socket(PF_NETLINK, SOCK_RAW, 0) = 28
> bind(28, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
> getsockname(28, {sa_family=AF_NETLINK, pid=31943, groups=00000000}, [12]) = 0
> gettimeofday({1393735036, 191154}, NULL) = 0
> sendto(28, "\24\0\0\0\26\0\1\3|\265\22S\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0|\265\22S\307|\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"#\0\0\0\24\0\2\0|\265\22S\307|\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0|\265\22S\307|\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
> close(28) = 0
> socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 28
The top (<) one being from a terminated process and bottom one (>) from the last (successful) process. It is clear that the process is terminating due to lack of permission to bind to the port. On looking further, I realized that the SELinux was set to ENFORCE, which was preventing the memcached service from binding to port other than 11211 (the default port).
To the best of what I could figure, when I was running it without the service command, the behavior was simply that of a process (not a service) and hence the binding was not being enforced.
Turning off the ENFORCED mode of SELinux, got the service memcached start command working!
Related
I am working on an embedded linux device, and the kernel has support for a USB serial console. You can log in fine, but when you type 'exit', the session is not exited, and you get stuck.
> exit
# The session hangs here forever
Hotplugging is enabled, so you can start a new session by removing and plugging the cable back in, but I don't want to do that every time I exit.
dmesg doesn't print anything useful after I type exit, so something is stuck.
I also tried attaching strace to getty, and it looks like it exited normally
) = 1 ([{fd=0, revents=POLLIN}])
read(0, "\n", 1) = 1
write(1, "\n", 1) = 1
open("/root/.ash_history", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3
_llseek(3, 0, [42052], SEEK_END) = 0
write(3, "exit\n", 5) = 5
close(3) = 0
ioctl(0, SNDCTL_TMR_START or TCSETS, {B115200 opost isig icanon echo ...}) = 0
rt_sigaction(SIGWINCH, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xb6dadd61}, NULL, 8) = 0
wait4(-1, 0xbee98a2c, WNOHANG|WSTOPPED, NULL) = -1 ECHILD (No child processes)
ioctl(10, TIOCSPGRP, [2578]) = 0
setpgid(0, 2578) = -1 EPERM (Operation not permitted)
close(10) = 0
exit_group(0) = ?
+++ exited with 0 +++
What is causing this session to hang forever?
I have a directory with a single file, one.txt. If I run ls | cat, it works fine. However, if I try to strace both sides of this pipeline, I do see the output of the command as well as strace, but the process doesn't finish.
strace ls 2> >(stdbuf -o 0 sed 's/^/command1:/') | strace cat 2> >(stdbuf -o 0 sed 's/^/command2:/')
The output I get is:
command2:execve("/usr/bin/cat", ["cat"], [/* 50 vars */]) = 0
command2:brk(0) = 0x1938000
command2:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f87e5a93000
command2:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command2:open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
command2:fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0
command2:mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f87def8a000
command2:close(3) = 0
command2:fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
command2:fstat(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = -1 ESPIPE (Illegal seek)
command2:read(0, "command1:execve(\"/usr/bin/ls\", ["..., 65536) = 4985
command1:execve("/usr/bin/ls", ["ls"], [/* 50 vars */]) = 0
command1:brk(0) = 0x1190000
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c3000
command1:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command1:close(3) = 0
command1:fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:write(1, "command1:close(3) "..., 115) = 115
command2:read(0, "command1:mmap(NULL, 4096, PROT_R"..., 65536) = 160
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c2000
one.txt
command1:write(1, "one.txt\n", 8) = 8
command2:write(1, "command1:mmap(NULL, 4096, PROT_R"..., 160) = 160
command2:read(0, "command1:close(1) "..., 65536) = 159
command1:close(1) = 0
command1:munmap(0x7fae869c2000, 4096) = 0
command1:close(2) = 0
command2:write(1, "command1:close(1) "..., 159) = 159
command2:read(0, "command1:exit_group(0) "..., 65536) = 53
command1:exit_group(0) = ?
command2:write(1, "command1:exit_group(0) "..., 53) = 53
command2:read(0, "command1:+++ exited with 0 +++\n", 65536) = 31
command1:+++ exited with 0 +++
command2:write(1, "command1:+++ exited with 0 +++\n", 31) = 31
and it hangs from then on. ps reveals that both commands in the pipeline (ls and cat here) are running.
I am on RHEL7 running Bash version 4.2.46.
I put a strace on your strace:
strace bash -c 'strace true 2> >(cat > /dev/null)'
It hangs on a wait4, indicating that it's stuck waiting on children. ps f confirms this:
24740 pts/19 Ss 0:00 /bin/bash
24752 pts/19 S+ 0:00 \_ strace true
24753 pts/19 S+ 0:00 \_ /bin/bash
24755 pts/19 S+ 0:00 \_ cat
Based on this, my working theory is that this effect is a deadlock because:
strace waits on all children, even the ones it didn't spawn directly
Bash spawns the process substitution as a child of the process. Since the process substitution is attached to stderr, it essentially waits for the parent to exit.
This suggests at least two workarounds, both of which appear to work:
strace -D ls 2> >(nl)
{ strace ls; true; } 2> >(nl)
-D, to quote the man page, "[runs the] tracer process as a detached grandchild, not as parent of the tracee". The second one forces bash to do another fork to run strace by adding another command to do after.
In both cases, the extra forks mean that the process substitution doesn't end up as strace's child, avoiding the issue.
While running time command, one of the programs gives following output:
real 1m33.523s
user 0m15.156s
sys 0m1.312s
Here the real and user+sys time have a lot of difference. This is most likely due to time spent on IO wait/calls. I want to measure total time spend by program in IO wait or IO calls. Is there any way to do that?
I tried using iotop. However, it doesnot report total time spent by the program performing IO.
Yes, strace - which can provide per-system-call statistics.
Example 1
I want to measure time spent on I/O while accessing stackoverflow.com:
$ time curl stackoverflow.com >/dev/null 2>&1
curl stackoverflow.com > /dev/null 2>&1 0.00s user 0.01s system 2% cpu 0.392 total
OK, 2% CPU and 0.01 s in system. Let's find out:
$ strace -c curl stackoverflow.com >/dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 240k 0 240k 0 0 127k 0 --:--:-- 0:00:01 --:--:-- 130k
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.12 0.005497 11 506 write
18.16 0.001845 43 43 fstat
11.95 0.001214 30 41 poll
5.75 0.000584 32 18 recvfrom
3.40 0.000345 3 101 mmap
2.51 0.000255 4 62 mprotect
1.98 0.000201 4 50 close
1.84 0.000187 31 6 getsockname
0.29 0.000029 1 42 1 open
Especially useful compare this results with results measured for runing curl without args.
Anyway. strace shows that curl mostly spends time in write, fstat and poll.
Another example
The first approach seems show incorrect results for sleep. If you are not satisfied with the first approach you can just print get times of each syscall (strace -T). Get this data and process them to find summary time of each syscall.
$ strace 2>&1 -T curl stackoverflow.com >/dev/null | head -n 20
execve("/usr/bin/curl", ["curl", "stackoverflow.com"], [/* 62 vars */]) = 0 <0.000219>
brk(0) = 0x186e000 <0.000175>
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc04c9e6000 <0.000166>
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000238>
open("/etc/ld.so.cache", O_RDONLY) = 3 <0.000144>
fstat(3, {st_mode=S_IFREG|0644, st_size=96498, ...}) = 0 <0.000175>
mmap(NULL, 96498, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc04c9ce000 <0.000164>
close(3) = 0 <0.000160>
open("/usr/lib64/libcurl.so.4", O_RDONLY) = 3 <0.000047>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\333\300\">\0\0\0"..., 832) = 832 <0.000160>
fstat(3, {st_mode=S_IFREG|0755, st_size=346008, ...}) = 0 <0.000216>
mmap(0x3e22c00000, 2438600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3e22c00000 <0.000189>
mprotect(0x3e22c51000, 2097152, PROT_NONE) = 0 <0.000032>
mmap(0x3e22e51000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x51000) = 0x3e22e51000 <0.000119>
close(3) = 0 <0.000110>
open("/lib64/libidn.so.11", O_RDONLY) = 3 <0.000257>
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0/#U1\0\0\0"..., 832) = 832 <0.000051>
fstat(3, {st_mode=S_IFREG|0755, st_size=209088, ...}) = 0 <0.000041>
mmap(0x3155400000, 2301736, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x3155400000 <0.000037>
mprotect(0x3155432000, 2093056, PROT_NONE) = 0 <0.000037>
I've been studying UNIX and system calls and I came across a low-level and tricky questions. The question asks what system calls are called for this command:
grep word1 word2 > file.txt
I did some research and I was unable to find a huge number of resources on the underlying UNIX calls. However, it seems to me that the answer would be open (to open and the file descriptor for the file file.txt), then dup2 (to change the STDOUT of grep to the file descriptor of open), then write to write the STDOUT of grep (which is now the file descriptor of file.txt), and finally close(), to close the file descriptor of file.txt... However, I have no idea if I am right or on the correct path, can anyone with experience in UNIX enlighten me on this topic?
You are on correct direction in your research. This command is very helpful to trace system calls in any program:
strace
On my PC it shows output (without stream redirection):
$ strace grep abc ss.txt
execve("/bin/grep", ["grep", "abc", "ss.txt"], [/* 237 vars */]) = 0
brk(0) = 0x13de000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1785694000
close(3) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("ss.txt", {st_mode=S_IFREG|0644, st_size=13, ...}) = 0
open("ss.txt", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffa0e4f370) = -1 ENOTTY (Inappropriate ioctl for device)
read(3, "abc\n123\n321\n\n", 32768) = 13
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f178568c000
write(1, "abc\n", 4abc
) = 4
read(3, "", 32768) = 0
close(3) = 0
close(1) = 0
munmap(0x7f178568c000, 4096) = 0
close(2) = 0
exit_group(0) = ?
i am trying to setup perforce server on cubieboard2 (raspberry-pi clone)
i have downloaded p4d and p4 for Linux ARM (2011.1)
i am running server like this:
p4d -d -r root_path -p 12345 -L log
statistics of running commands directly on server machine (server ip 192.168.1.1)
p4 -p 12345 users # it returns immediately
p4 -p 127.0.0.1:12345 users # it returns immediately
p4 -p localhost:12345 users # it returns immediately
p4 -p 192.168.1.1:12345 users # it returns immediately
p4 -p 12345 info # it returns immediately
p4 -p 127.0.0.1:12345 info # it returns immediately
p4 -p localhost:12345 info # it returns immediately
p4 -p 192.168.1.1:12345 info # returns after ~10 sec
i want to connect to this perforce server from my windows pc, since i am in LAN with the server, i am connecting with p4 -p 192.168.1.1:12345 users command returns immediately, info with 10sec lag (using p4 on my windows pc)
i didn't checked other commands, i am unable to connect with p4Admin or p4v, they hang/do nothing, thus my server is unusable
starting server with
p4d -d -r root_path -p 192.168.1.1:12345 -L log
makes it impossible to run commands with p4 -p 12345 or localhost:12345 or 127.0.0.1:12345
but its same story with 192.168.1.1:12345
Note. i am also running ftp on it, and it runs fine, when connecting through 192.168.1.1
Edit. added strace log for p4d
strace of 'p4 -p 192.168.1.1:12345 users' ran from win7 pc (ip ends with '13', returns immed.)
{sa_family=AF_INET, sin_port=htons(61483), sin_addr=inet_addr("192.168.1.13")}, [16]) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
wait4(-1, 0xbea31744, WNOHANG, NULL) = -1 ECHILD (No child processes)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb6e45068) = 2890
close(4) = 0
gettimeofday({1380359365, 320807}, NULL) = 0
accept(3, 0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) # 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 2890
wait4(-1, 0xbea3141c, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
sigreturn() = ? (mask now [HUP INT QUIT])
accept(3,
strace for p4 -p 192.168.1.1:12345 info
{sa_family=AF_INET, sin_port=htons(61486), sin_addr=inet_addr("192.168.1.13")}, [16]) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
wait4(-1, 0xbea31744, WNOHANG, NULL) = -1 ECHILD (No child processes)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb6e45068) = 2894
close(4) = 0
gettimeofday({1380359462, 822219}, NULL) = 0
accept(3,
now it lags, and after, following lines appear
0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) # 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 2894
wait4(-1, 0xbea3141c, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
sigreturn() = ? (mask now [HUP INT QUIT])
accept(3,
its not telling me much, but as you can see, the exact same line
accept(3, 0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
appears in both logs, and on the latter it stops for 10 sec after 'accept(3, '
As for swap, i had it enabled as swap file on external drive connected to the board, while i am booting from sd-card, either way, no change after disabling it.