Error when spawning child process in node.js - node.js

I'm trying to get a little ffmpeg converter-service up and running, made pretty good progress so far. But when it comes to spawning the actual ffmpeg process for conversion, i'm hitting a brick wall.
// options.ffmpegopts is an array containing format-specific parameters
var args = [ '-y', '"' + options.targetfile + '"' ];
args = options.ffmpegopts.concat(args);
var ffmpegProc = spawn('ffmpeg ', args);
ffmpegProc.stderr.on('data', function(data) {
console.log('stderr: ' + data);
});
When executing this code, i get the following console output:
stderr: execvp(): No such file or directory
I already checked different node versions (0.4.0, 0.4.2 and 0.5.0-pre) without any effect.
Another really strange behavior is the fact that i have to call spawn including a space ('ffmpeg ' instead of just 'ffmpeg'). If i omit this space, i get a different error (stderr: "/path/to/my/movie.mpeg": no such file or directory). When calling ffmpeg directly from the shell, the command sent to child_process.spawn() executes without any problems.
Any hints on that one? I already checked other projects who achieve the same (like node-imagemagick or ffmpeg-node, but the enlightment didn't hit me...
Update: strace() output
When running my application using strace -fF -o strace.log node server.js, i can grep the following process spawning calls:
execve("/usr/local/sbin/ffmpeg", ["ffmpeg", "-i", "\"/data/media_dev/test/ORG_mymovi"..., "-sameq", "-ab", "128k", "-ar", "44100", "-b", "512k", "-r", "25", "-s", "320x240", "-f", "flv", ...], [/* 20 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/ffmpeg", ["ffmpeg", "-i", "\"/data/media_dev/test/ORG_mymovi"..., "-sameq", "-ab", "128k", "-ar", "44100", "-b", "512k", "-r", "25", "-s", "320x240", "-f", "flv", ...], [/* 20 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/sbin/ffmpeg", ["ffmpeg", "-i", "\"/data/media_dev/test/ORG_mymovi"..., "-sameq", "-ab", "128k", "-ar", "44100", "-b", "512k", "-r", "25", "-s", "320x240", "-f", "flv", ...], [/* 20 vars */]) = -1 ENOENT (No such file or directory)
execve("/usr/bin/ffmpeg", ["ffmpeg", "-i", "\"/data/media_dev/test/ORG_mymovi"..., "-sameq", "-ab", "128k", "-ar", "44100", "-b", "512k", "-r", "25", "-s", "320x240", "-f", "flv", ...], [/* 20 vars */]) = 0
After seeing that strangely escaped double quotes on the path, i tried to call ffmpeg without the quotes...worked like a charm. But the problem remains, i need to be able to work with spaces in my paths.
Any suggestions?
Update: Solution
Got it working with spaces, a simple inputfile.replace(' ', '\ ') was enough.

I'd wager money that the space at the end of "ffmpeg " is the cause of the current problem. A quick little C program will show that:
#include <unistd.h>
int main(int argc, char *argv[]) {
execvp(argv[1], &argv[1]);
perror(argv[0]);
}
gives the following output:
$ ./execvp "ffmpeg"
FFmpeg version 0.6-4:0.6-2ubuntu6, Copyright (c) 2000-2010 the FFmpeg developers
...
$ ./execvp "ffmpeg "
./execvp: No such file or directory
$
I suggest removing the space again, and re-run under strace(1) -fF. Look for the command that is actually executed, and look to see if the error message about /path/to/my/movie.mpeg is coming from ffmpeg or from node.js.

Related

Make a shell pipeline started from subprocess.Popen fail if the left-hand side of a pipe fails

Im running a bash command with subprocess.popen in python:
cmd = "bwa-mem2/bwa-mem2 mem -R \'#RG\\tID:2064-01\\tSM:2064-01\\tLB:2064-01\\tPL:ILLUMINA\\tPU:2064-01\' reference_genome/human_g1k_v37.fasta BHYHT7CCXY.RJ-1967-987-02.2_1.fastq BHYHT7CCXY.RJ-1967-987-02.2_2.fastq -t 14 | samtools view -bS -o dna_seq/aligned/2064-01/2064-01.6.bam -"
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
The problem is that I get returncode 0 even if the first command fails.
I have googled and found out about pipefail and it seems that this is what I should use.
However, I don't understand where to write it. I have tried:
"set -o pipefail && bwa-mem2/bwa-mem2 mem -R \'#RG\\tID:2064-01\\tSM:2064-01\\tLB:2064-01\\tPL:ILLUMINA\\tPU:2064-01\' reference_genome/human_g1k_v37.fasta BHYHT7CCXY.RJ-1967-987-02.2_1.fastq BHYHT7CCXY.RJ-1967-987-02.2_2.fastq -t 14 | samtools view -bS -o dna_seq/aligned/2064-01/2064-01.6.bam -"
which gives: /bin/sh: 1: set: Illegal option -o pipefail
any ideas how I should incorporate this?
Edit:
I'm not sure if it is correct to edit my answer when responding to an answer? there was not enough characters to respond in a comment:/
Anyway,
I tried your second approach without shell=True #Charles Duffy.
(cmd_1 and cmd_2 are equal to what you wrote in your solution)
This is the code I use:
try:
p1 = Popen(shlex.split(cmd_1), stdout=PIPE)
p2 = Popen(shlex.split(cmd_2), stdin=p1.stdout, stdout=PIPE, stderr=STDOUT, text=True)
p1.stdout.close()
output, error = p2.communicate()
p1.wait()
rc_1 = p1.poll()
rc_2 = p2.poll()
print("rc_1:", rc_1)
print("rc_2:", rc_2)
if rc_1 == 0 and rc_2 == 0:
self.log_to_file("DEBUG", "# Process ended with returncode = 0")
if text: self.log_to_file("INFO", f"{text} succesfully
else:
print("Raise exception")
raise Exception(f"stdout: {output} stderr: {error}")
except Exception as e:
print(f"Error: {e} in misc.run_command()")
self.log_to_file("ERROR", f"# Process ended with returncode != 0, {e}")
this is the result i get when deliberately causing an error by renaming one file:
[E::main_mem] failed to open file `/home/jonas/BASE/dna_seq/reads/2064-01/test_BHYHT7CCXY.RJ-1967-987-02.2_2.fastq.gz'.
free(): double free detected in tcache 2
rc_1: -6
rc_2: 0
Raise exception
Error: stdout: stderr: None in misc.run_command()
ERROR: # Process ended with returncode != 0, stdout: stderr: None
It seems to capture the faulty returncode.
But why is stdout empty and stderr= None?
How can I capture the output to have it logged to a logger both when the process is successful and when it fails?
First, With A Shell
Instead of letting shell=True specify sh by default, specify bash explicitly to ensure that pipefail is an available feature:
shell_script = r'''
set -o pipefail || exit
bwa-mem2/bwa-mem2 mem \
-R '#RG\tID:2064-01\tSM:2064-01\tLB:2064-01\tPL:ILLUMINA\tPU:2064-01' \
reference_genome/human_g1k_v37.fasta \
BHYHT7CCXY.RJ-1967-987-02.2_1.fastq \
BHYHT7CCXY.RJ-1967-987-02.2_2.fastq \
-t 14 \
| samtools view -bS \
-o dna_seq/aligned/2064-01/2064-01.6.bam -
'''
process = subprocess.Popen(["bash", "-c", shell_script],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
This works, but it's not the best available option.
Second, With No Shell At All
p1 = subprocess.Popen(
['bwa-mem2/bwa-mem2', 'mem',
'-R', r'#RG\tID:2064-01\tSM:2064-01\tLB:2064-01\tPL:ILLUMINA\tPU:2064-01',
'reference_genome/human_g1k_v37.fasta',
'BHYHT7CCXY.RJ-1967-987-02.2_1.fastq',
'BHYHT7CCXY.RJ-1967-987-02.2_2.fastq', '-t', '14'],
stdout=subprocess.PIPE)
p2 = subprocess.Popen(
['samtools', 'view', '-bS',
'-o', 'dna_seq/aligned/2064-01/2064-01.6.bam', '-'],
stdin=p1.stdout,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True)
p1.stdout.close()
output, _ = p2.communicate() # let p2 finish running
p1.wait() # ensure p1 has properly exited
print(f'bwa-mem2 exited with status {p1.returncode}')
print(f'samtools exited with status {p2.returncode}')
...which lets you check p1.returncode and p2.returncode separately.

Race condition when reading from /dev/pts/x

I read in this article that when a process reads from a tty device, say /dev/pts/1, the tty driver/line discipline (when in cooked mode) buffers the data, and only when enter is pressed, the data is passed to the process. I ran the following experiment:
I open a terminal, and note the tty it's using. Let's say it's /dev/pts/0.
Now I open another terminal, using /dev/pts/1 for the sake of the manner, and run a process, which only executes the following function:
// passing /dev/pts/0 to the function
int read_from_tty(char *tty)
{
int bytes_read = 0;
int fd = 0;
char buffer[100];
fd = open(tty, O_RDWR);
if(-1 == fd)
{
printf("Couldn't open pts\n");
return 1;
}
printf("File opened: %d\n", fd);
while(1)
{
bytes_read = read(fd, buffer, 100);
if(-1 == bytes_read)
{
perror("read");
}
if(-1 == write(1, buffer, bytes_read))
{
perror("write:");
}
memset(buffer, 0, 100);
}
}
Now I start typing characters in the /dev/pts/0 terminal, and I see that characters are appearing mostly in the /dev/pts/1 terminal, but one in something like 10 chars I see a char in the /dev/pts/0 terminal.
If what's written in the article is true, seemingly, the characters should have been buffered in the line discipline, and only when I press enter be passed to one of the reading processes (assuming the bash is just sitting blocked on read).
Could someone explain?
EDIT
Examining this a bit further. I add the following line to the code above:
...
bytes_read = read(fd, buffer, 100);
if(-1 == bytes_read)
{
perror("read");
}
printf("Bytes read: %d\n", bytes_read);
if(-1 == write(1, buffer, bytes_read))
{
...
I can see that when I read /dev/pts/0, it only reads 1 byte at a time. Whereas, if I run it with /dev/pts/1 (actually reading from stdin), it does read a whole line.
Could someone explain?
Actually, bash sets the terminal in non canonical mode when reading from the terminal and when it gets an end of line, it sets back the terminal in canonical mode to run the command line.
The same experience can be done with two terminals:
Terminal#1 (/dev/pts/6): launch strace /bin/bash
Terminal#2: launch strace cat /dev/pts/6
The bash shell on terminal#1 deactivates the canonical mode and calls pselect() to wait for the input:
$ strace /bin/bash
[...]
ioctl(0, TCGETS, {B38400 opost isig -icanon -echo ...}) = 0
[...]
pselect6(1, [0], NULL, NULL, NULL, {[], 8}
On terminal#2, the cat command merely calls a blocking read() to get chars from the terminal:
$ strace cat /dev/pts/6
[...]
openat(AT_FDCWD, "/dev/pts/6", O_RDONLY) = 3
[...]
read(3,
So, both bash and cat are reading concurrently on the terminal. When we type characters in terminal#1, pselect() returns to indicate that a character is available and then bash calls a blocking read() to get the char. But the concurrent read() from cat stills the characters between the calls to pselect() and read() by bash. Sometimes, bash is able to get a character before cat.
Here is an example where pselect() returns because a character is available (I typed "Y") and a following read() is called to get it on terminal#1:
pselect6(1, [0], NULL, NULL, NULL, {[], 8}) = 1 (in [0])
read(0,
But read() from cat on the other terminal succeeded to get the character before the read() of bash:
write(1, "Y", 1Y) = 1
read(3,
Sometimes, bash is able to get the typed character before cat. Generally, when it is blocked on its read() call (that is to say it missed the character detected by pselect() but it will be able to get one of the subsequent typed characters before one of the calls to read() by cat)...
Side note
When we launch stty -a under bash, the display shows that the terminal is in canonical mode:
$ stty -a
[...]
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt echoctl echoke -flusho -extproc
This is because of the fact that bash reactivates the canonical mode before launching the stty command.

How to quote part of a subprocess.run list? [duplicate]

This question already has answers here:
Python Subprocess: Unable to Escape Quotes
(2 answers)
Closed last year.
I need to quote part of the rsync line that subprocess.run uses that contains the ssh parameters, unfortunately nothing I have tried has worked so far.
Can someone please advise me on the correct way to quote the ssh parameters, so that it will run under rsync.
At first I had a list of lists that got passed to subprocess.run, that fails with:
Traceback (most recent call last):
File "./tmp.py", line 20, in <module>
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not list
Flatten it to an ordinary list:
Unexpected remote arg: example.com:/var/log/maillog
rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
Which makes sense, as part of the command line for rsync needs to be quoted.
So I try to quote it:
rsync: Failed to exec /usr/bin/ssh -F /home/rspencer/.ssh/config -o PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask -o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o Compression=yes: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(85) [Receiver=3.1.2]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in IPC code (code 14) at io.c(235) [Receiver=3.1.2]
Which is due, I expect, to it being a string instead of a list. Although I'm guessing and that does not make complete sense to me.
Summarized code of my last attempt:
#!/usr/bin/python3
import subprocess
ssh_args = [
"-F",
"/home/rspencer/.ssh/config",
"-o",
"PreferredAuthentications=publickey",
"-o",
"StrictHostKeyChecking=accept-new",
"-o",
"TCPKeepAlive=yes",
"-o",
"ServerAliveInterval=5",
"-o",
"ServerAliveCountMax=24",
"-o",
"ConnectTimeout=30",
"-o",
"ExitOnForwardFailure=yes",
"-o",
"ControlMaster=autoask",
"-o",
"ControlPath=/run/user/1000/foo-ssh-master-%C",
"-l",
"root",
"-p",
"234",
]
rsync_params = []
src = "example.com:/var/log/maillog"
dest = "."
# Build SSH command
ssh_cmd = ["/usr/bin/ssh"] + ssh_args
# Use basic compression
ssh_cmd.extend(["-o", "Compression=yes"])
ssh_cmd = " ".join(ssh_cmd)
ssh_cmd = f'"{ssh_cmd}"'
# Build rsync command
rsync_cmd = ["/usr/bin/rsync", "-vP", "-e", ssh_cmd] + rsync_params + [src, dest]
# Run rsync
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
if process.returncode != 0:
print(process.stderr.decode("UTF-8").strip())
What the correct command would look like on the command line:
/usr/bin/rsync -vP -e "/usr/bin/ssh -F /home/rspencer/.ssh/config -o \
PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o \
TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o \
ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask \
-o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o \
Compression=yes" example.com:/var/log/maillog .
Turns out the trick is to not try to quote it.
I removed the following line and it worked without further modification:
ssh_cmd = f'"{ssh_cmd}"'
I've read so much documentation and missed it until asking the question. Murphy.
Rereading the post "How not to quote argument in subprocess?" and finally understanding what Greg Hewgill was saying helped me. I blame lack of sleep.
"If you use quotes on the shell command line, then put the whole contents in one element of args (without the quotes). ..." - Greg Hewgill

Using a pipe character | with child_process spawn

I'm running nodejs on a raspberry pi and I want to run a child process to spawn a webcam stream.
Outside of node my command is:
raspivid -n -mm matrix -w 320 -h 240 -fps 18 -g 100 -t 0 -b 5000000 -o - | ffmpeg -y -f h264 -i - -c:v copy -map 0:0 -f flv -rtmp_buffer 100 -rtmp_live live "rtmp://example.com/big/test"
With child_process I have to break each argument up
var args = ["-n", "-mm", "matrix", "-w", "320", "-h", "240", "-fps", "18", "-g", "100", "-t", "0", "-b", "5000000", "-o", "-", "|", "ffmpeg", "-y", "-f", "h264", "-i", "-", "-c:v", "copy", "-map", "0:0", "-f", "flv", "-rtmp_buffer", "100", "-rtmp_live", "live", "rtmp://example.com/big/test"];
camera.proc = child.spawn('raspivid', args);
However it chokes on the | character:
error, exit code 64
Invalid command line option (|)
How do I use this pipe character as an argument?
This has been answered in another question: Using two commands (using pipe |) with spawn
In summary, with child.spawn everything in args should be an argument of your 'raspivid' command. In your case, the pipe and everything after it are actually arguments for sh.
A workaround is to call child.spawn('sh', args) where args is:
var args = ['-c', <the entire command you want to run as a string>];

How to test STDIN read error

I have a Linux program (currently in assembly) that has a check: if a read from STDIN failed, show an error message. The problem is that I do not know how to test this condition, how to execute the program so that it will fail reading from STDIN. IT must be run without STDIn or STDIN couldbe closed some how before the program starts?
Yes, you can close the file descriptor, that will trigger an error. Test using bash:
$ strace ./a.out 0<&-
execve("./a.out", ["./a.out"], [/* 32 vars */]) = 0
[ Process PID=4012 runs in 32 bit mode. ]
read(0, 0xffe13fec, 1) = -1 EBADF (Bad file descriptor)
You can also provoke other errors that are listed in the man page, such as:
$ strace ./a.out 0</tmp
execve("./a.out", ["./a.out"], [/* 32 vars */]) = 0
[ Process PID=4056 runs in 32 bit mode. ]
read(0, 0xffed5c0c, 1) = -1 EISDIR (Is a directory)

Resources