Perforce strange 10s delay/lag on some commands [raspberry-pi / cubieboard] - linux

i am trying to setup perforce server on cubieboard2 (raspberry-pi clone)
i have downloaded p4d and p4 for Linux ARM (2011.1)
i am running server like this:
p4d -d -r root_path -p 12345 -L log
statistics of running commands directly on server machine (server ip 192.168.1.1)
p4 -p 12345 users # it returns immediately
p4 -p 127.0.0.1:12345 users # it returns immediately
p4 -p localhost:12345 users # it returns immediately
p4 -p 192.168.1.1:12345 users # it returns immediately
p4 -p 12345 info # it returns immediately
p4 -p 127.0.0.1:12345 info # it returns immediately
p4 -p localhost:12345 info # it returns immediately
p4 -p 192.168.1.1:12345 info # returns after ~10 sec
i want to connect to this perforce server from my windows pc, since i am in LAN with the server, i am connecting with p4 -p 192.168.1.1:12345 users command returns immediately, info with 10sec lag (using p4 on my windows pc)
i didn't checked other commands, i am unable to connect with p4Admin or p4v, they hang/do nothing, thus my server is unusable
starting server with
p4d -d -r root_path -p 192.168.1.1:12345 -L log
makes it impossible to run commands with p4 -p 12345 or localhost:12345 or 127.0.0.1:12345
but its same story with 192.168.1.1:12345
Note. i am also running ftp on it, and it runs fine, when connecting through 192.168.1.1
Edit. added strace log for p4d
strace of 'p4 -p 192.168.1.1:12345 users' ran from win7 pc (ip ends with '13', returns immed.)
{sa_family=AF_INET, sin_port=htons(61483), sin_addr=inet_addr("192.168.1.13")}, [16]) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
wait4(-1, 0xbea31744, WNOHANG, NULL) = -1 ECHILD (No child processes)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb6e45068) = 2890
close(4) = 0
gettimeofday({1380359365, 320807}, NULL) = 0
accept(3, 0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) # 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 2890
wait4(-1, 0xbea3141c, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
sigreturn() = ? (mask now [HUP INT QUIT])
accept(3,
strace for p4 -p 192.168.1.1:12345 info
{sa_family=AF_INET, sin_port=htons(61486), sin_addr=inet_addr("192.168.1.13")}, [16]) = 4
fcntl64(4, F_SETFD, FD_CLOEXEC) = 0
fcntl64(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
setsockopt(4, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
wait4(-1, 0xbea31744, WNOHANG, NULL) = -1 ECHILD (No child processes)
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb6e45068) = 2894
close(4) = 0
gettimeofday({1380359462, 822219}, NULL) = 0
accept(3,
now it lags, and after, following lines appear
0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
--- SIGCHLD (Child exited) # 0 (0) ---
wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 2894
wait4(-1, 0xbea3141c, WNOHANG, NULL) = -1 ECHILD (No child processes)
rt_sigaction(SIGCHLD, {0x16d160, [CHLD], SA_RESTART|0x4000000}, {0x16d160, [CHLD], SA_RESTART|0x4000000}, 8) = 0
sigreturn() = ? (mask now [HUP INT QUIT])
accept(3,
its not telling me much, but as you can see, the exact same line
accept(3, 0xbea31734, [16]) = ? ERESTARTSYS (To be restarted)
appears in both logs, and on the latter it stops for 10 sec after 'accept(3, '
As for swap, i had it enabled as swap file on external drive connected to the board, while i am booting from sd-card, either way, no change after disabling it.

Related

USB serial console is not exiting from Linux device

I am working on an embedded linux device, and the kernel has support for a USB serial console. You can log in fine, but when you type 'exit', the session is not exited, and you get stuck.
> exit
# The session hangs here forever
Hotplugging is enabled, so you can start a new session by removing and plugging the cable back in, but I don't want to do that every time I exit.
dmesg doesn't print anything useful after I type exit, so something is stuck.
I also tried attaching strace to getty, and it looks like it exited normally
) = 1 ([{fd=0, revents=POLLIN}])
read(0, "\n", 1) = 1
write(1, "\n", 1) = 1
open("/root/.ash_history", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3
_llseek(3, 0, [42052], SEEK_END) = 0
write(3, "exit\n", 5) = 5
close(3) = 0
ioctl(0, SNDCTL_TMR_START or TCSETS, {B115200 opost isig icanon echo ...}) = 0
rt_sigaction(SIGWINCH, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0xb6dadd61}, NULL, 8) = 0
wait4(-1, 0xbee98a2c, WNOHANG|WSTOPPED, NULL) = -1 ECHILD (No child processes)
ioctl(10, TIOCSPGRP, [2578]) = 0
setpgid(0, 2578) = -1 EPERM (Operation not permitted)
close(10) = 0
exit_group(0) = ?
+++ exited with 0 +++
What is causing this session to hang forever?

Why does this strace on a pipeline not finish

I have a directory with a single file, one.txt. If I run ls | cat, it works fine. However, if I try to strace both sides of this pipeline, I do see the output of the command as well as strace, but the process doesn't finish.
strace ls 2> >(stdbuf -o 0 sed 's/^/command1:/') | strace cat 2> >(stdbuf -o 0 sed 's/^/command2:/')
The output I get is:
command2:execve("/usr/bin/cat", ["cat"], [/* 50 vars */]) = 0
command2:brk(0) = 0x1938000
command2:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f87e5a93000
command2:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command2:open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
command2:fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0
command2:mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f87def8a000
command2:close(3) = 0
command2:fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
command2:fstat(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:fadvise64(0, 0, 0, POSIX_FADV_SEQUENTIAL) = -1 ESPIPE (Illegal seek)
command2:read(0, "command1:execve(\"/usr/bin/ls\", ["..., 65536) = 4985
command1:execve("/usr/bin/ls", ["ls"], [/* 50 vars */]) = 0
command1:brk(0) = 0x1190000
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c3000
command1:access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
<snip>
command1:close(3) = 0
command1:fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
command2:write(1, "command1:close(3) "..., 115) = 115
command2:read(0, "command1:mmap(NULL, 4096, PROT_R"..., 65536) = 160
command1:mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fae869c2000
one.txt
command1:write(1, "one.txt\n", 8) = 8
command2:write(1, "command1:mmap(NULL, 4096, PROT_R"..., 160) = 160
command2:read(0, "command1:close(1) "..., 65536) = 159
command1:close(1) = 0
command1:munmap(0x7fae869c2000, 4096) = 0
command1:close(2) = 0
command2:write(1, "command1:close(1) "..., 159) = 159
command2:read(0, "command1:exit_group(0) "..., 65536) = 53
command1:exit_group(0) = ?
command2:write(1, "command1:exit_group(0) "..., 53) = 53
command2:read(0, "command1:+++ exited with 0 +++\n", 65536) = 31
command1:+++ exited with 0 +++
command2:write(1, "command1:+++ exited with 0 +++\n", 31) = 31
and it hangs from then on. ps reveals that both commands in the pipeline (ls and cat here) are running.
I am on RHEL7 running Bash version 4.2.46.
I put a strace on your strace:
strace bash -c 'strace true 2> >(cat > /dev/null)'
It hangs on a wait4, indicating that it's stuck waiting on children. ps f confirms this:
24740 pts/19 Ss 0:00 /bin/bash
24752 pts/19 S+ 0:00 \_ strace true
24753 pts/19 S+ 0:00 \_ /bin/bash
24755 pts/19 S+ 0:00 \_ cat
Based on this, my working theory is that this effect is a deadlock because:
strace waits on all children, even the ones it didn't spawn directly
Bash spawns the process substitution as a child of the process. Since the process substitution is attached to stderr, it essentially waits for the parent to exit.
This suggests at least two workarounds, both of which appear to work:
strace -D ls 2> >(nl)
{ strace ls; true; } 2> >(nl)
-D, to quote the man page, "[runs the] tracer process as a detached grandchild, not as parent of the tracee". The second one forces bash to do another fork to run strace by adding another command to do after.
In both cases, the extra forks mean that the process substitution doesn't end up as strace's child, avoiding the issue.

why does shell save, dup, and restore redirected descriptor in the parent, rather than redirect in the child?

I wanted to understand how redirection is implemented in a (POSIX) shell, so I looked at dash (the simplest shell) code, and here is how it works.
A dash script like:
date > foobar.txt
date
is (as an SSCCE) handled like this:
int fd;
int saved;
fd = open64("foobar.txt", O_WRONLY|O_CREAT);
saved = fcntl(1, F_DUPFD, 10);
dup2(fd, 1);
if (!fork()) {
execl("/bin/date", "date", (char *)NULL);
}
dup2(saved, 1);
if (!fork()) {
execl("/bin/date", "date", (char *)NULL);
}
This is strange. Why save, dup and dup again to restore, descriptors in the parent, when it would be much simpler to just dup in the child, and not have to save and restore. This is simpler and I checked it works the same:
int fd;
if (!fork()) {
fd = open64("foobar.txt", O_WRONLY|O_CREAT);
dup2(fd, 1);
execl("/bin/date", "date", (char *)NULL);
}
if (!fork()) {
execl("/bin/date", "date", (char *)NULL);
}
I am sure there must be a good reason and I am not understanding something deeper. What is it?
No good reason as far as I can tell. Bash does things in the opposite order, and the externally observable behavior is the same.
I didn't bother reading the source code, it's easy enough to see what happens using strace. (The : is to prevent the shell from optimizing away the fork.)
$ strace -fetrace=dup2,file,process dash -c 'date > foobar.txt; :'
execve("/usr/bin/dash", ["dash", "-c", "date > foobar.txt; :"], [/* 71 vars */]) = 0
open("foobar.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
dup2(3, 1) = 1
stat("/usr/bin/date", {st_mode=S_IFREG|0755, st_size=105280, ...}) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6650bf2750) = 1948
strace: Process 1948 attached
[pid 1947] wait4(-1, <unfinished ...>
[pid 1948] execve("/usr/bin/date", ["date"], [/* 71 vars */]) = 0
[pid 1948] exit_group(0) = ?
[pid 1948] +++ exited with 0 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 1948
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1948, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
dup2(10, 1) = 1
exit_group(0) = ?
+++ exited with 0 +++
$ strace -fetrace=dup2,file,process bash -c 'date > foobar.txt; :'
execve("/usr/bin/bash", ["bash", "-c", "date > foobar.txt; :"], [/* 71 vars */]) = 0
stat("/usr/bin/date", {st_mode=S_IFREG|0755, st_size=105280, ...}) = 0
access("/usr/bin/date", R_OK) = 0
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fd5b78349d0) = 2026
strace: Process 2026 attached
[pid 2025] wait4(-1, <unfinished ...>
[pid 2026] open("foobar.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
[pid 2026] dup2(3, 1) = 1
[pid 2026] execve("/usr/bin/date", ["date"], [/* 71 vars */]) = 0
[pid 2026] exit_group(0) = ?
[pid 2026] +++ exited with 0 +++
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 2026
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2026, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
wait4(-1, 0x7ffeffe03c50, WNOHANG, NULL) = -1 ECHILD (No child processes)
exit_group(0) = ?
+++ exited with 0 +++

Implicit system calls in UNIX commands

I've been studying UNIX and system calls and I came across a low-level and tricky questions. The question asks what system calls are called for this command:
grep word1 word2 > file.txt
I did some research and I was unable to find a huge number of resources on the underlying UNIX calls. However, it seems to me that the answer would be open (to open and the file descriptor for the file file.txt), then dup2 (to change the STDOUT of grep to the file descriptor of open), then write to write the STDOUT of grep (which is now the file descriptor of file.txt), and finally close(), to close the file descriptor of file.txt... However, I have no idea if I am right or on the correct path, can anyone with experience in UNIX enlighten me on this topic?
You are on correct direction in your research. This command is very helpful to trace system calls in any program:
strace
On my PC it shows output (without stream redirection):
$ strace grep abc ss.txt
execve("/bin/grep", ["grep", "abc", "ss.txt"], [/* 237 vars */]) = 0
brk(0) = 0x13de000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1785694000
close(3) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
stat("ss.txt", {st_mode=S_IFREG|0644, st_size=13, ...}) = 0
open("ss.txt", O_RDONLY) = 3
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fffa0e4f370) = -1 ENOTTY (Inappropriate ioctl for device)
read(3, "abc\n123\n321\n\n", 32768) = 13
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f178568c000
write(1, "abc\n", 4abc
) = 4
read(3, "", 32768) = 0
close(3) = 0
close(1) = 0
munmap(0x7f178568c000, 4096) = 0
close(2) = 0
exit_group(0) = ?

How to start multiple processes inside a linux service construct?

I am attempting to start multiple memcached processes from the linux service framework using the following logic:
RETVAL=0
pcount="$CACHES"
if [ ! -z "$pcount" ]; then
while [ $pcount -gt 0 ];
do
(( pcount-- ))
(( port=PORT + pcount ))
daemon --pidfile ${pidfile}${pcount}.pid memcached -d -p $port -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}${pcount}.pid $OPTIONS
(( RETVAL=RETVAL + $? ))
done
else
daemon --pidfile ${pidfile}.pid memcached -d -p $PORT -u $USER -m $CACHESIZE -c $MAXCONN -P ${pidfile}.pid $OPTIONS
RETVAL=$?
fi
When run using the command service memcached start, it creates and updates pid files for each cycle in the loop, but only the last instance of the process remains running. That is, while each of the /var/run/memcached/memcached(1 through 5).pid are created and updated with a PID; those processes do not exist. /var/run/memcached/memcached0.pid is also created and updated and the PID points to a running process.
I turned on tracing and I can see that the loop is executed and the process invocation is made; however the process does not start (or likely, starts and immediately terminates so I dont see it as having started).
On the other hand, running this script directly as /etc/init.d/memcached start results in all the processes getting started correctly.
Can someone help me understand why the service framework is preventing the starting of the other instances except the last one?
As suggested by #nos, I added strace -f to trace the calls during the service memcached start operation. I compared the traced calls between the unsuccessful/terminated process and the successful process. The only lines of significant difference that I found were:
< bind(26, {sa_family=AF_INET, sin_port=htons(11216), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EACCES (Permission denied)
< dup(2) = 27
< fcntl(27, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(27, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(27, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d780) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(27, 0, SEEK_CUR) = 0
< write(27, "bind(): Permission denied\n", 26) = 26
< close(27) = 0
< munmap(0x7f5dae958000, 4096) = 0
< close(26) = 0
< dup(2) = 26
< fcntl(26, F_GETFL) = 0x8002 (flags O_RDWR|O_LARGEFILE)
< fstat(26, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
< ioctl(26, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff20d5d730) = -1 ENOTTY (Inappropriate ioctl for device)
< mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5dae958000
< lseek(26, 0, SEEK_CUR) = 0
< write(26, "failed to listen on TCP port 112"..., 54) = 54
< close(26) = 0
< munmap(0x7f5dae958000, 4096) = 0
< exit_group(71) = ?
---
> bind(26, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> listen(26, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 26, {EPOLLIN, {u32=26, u64=26}}) = 0
> socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 27
> fcntl(27, F_GETFL) = 0x2 (flags O_RDWR)
> fcntl(27, F_SETFL, O_RDWR|O_NONBLOCK) = 0
> setsockopt(27, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> setsockopt(27, SOL_SOCKET, SO_LINGER, {onoff=0, linger=0}, 8) = 0
> setsockopt(27, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> bind(27, {sa_family=AF_INET6, sin6_port=htons(11211), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
> listen(27, 1024) = 0
> epoll_ctl(3, EPOLL_CTL_ADD, 27, {EPOLLIN, {u32=27, u64=27}}) = 0
> socket(PF_NETLINK, SOCK_RAW, 0) = 28
> bind(28, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
> getsockname(28, {sa_family=AF_NETLINK, pid=31943, groups=00000000}, [12]) = 0
> gettimeofday({1393735036, 191154}, NULL) = 0
> sendto(28, "\24\0\0\0\26\0\1\3|\265\22S\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0|\265\22S\307|\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 108
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"#\0\0\0\24\0\2\0|\265\22S\307|\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
> recvmsg(28, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0|\265\22S\307|\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
> close(28) = 0
> socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 28
The top (<) one being from a terminated process and bottom one (>) from the last (successful) process. It is clear that the process is terminating due to lack of permission to bind to the port. On looking further, I realized that the SELinux was set to ENFORCE, which was preventing the memcached service from binding to port other than 11211 (the default port).
To the best of what I could figure, when I was running it without the service command, the behavior was simply that of a process (not a service) and hence the binding was not being enforced.
Turning off the ENFORCED mode of SELinux, got the service memcached start command working!

Resources