I am making a terminal program that is able to run any executable (please ignore safety concerns). I need to detect when the child process is waiting for the user input (from stdin). I start the child process using:
process = subprocess.Popen(command, close_fds=False, shell=True, **file_descriptors)
I can think of 2 ways of detecting if the child process is waiting for stdin:
Writing a character then backspace and checking if the child has processed those 2 bytes. But here it says that "CMD does support the backspace key". So I need to find a character that when printed to the screen will delete what ever is in the stdin buffer in the command prompt.
The second method is to use the pywin32 library and use the WaitForInputIdle function as described here. I looked at the source code for the subprocess library and found that it uses pywin32 and it keeps a reference to the process handle. So I tried this:
win32event.WaitForInputIdle(proc._handle, 100)
But I got this error:
(1471, 'WaitForInputIdle', 'Unable to finish the requested operation because the specified process is not a GUI process.')
Also in the windows api documentation here it says: "WaitForInputIdle waits only once for a process to become idle; subsequent WaitForInputIdle calls return immediately, whether the process is idle or busy.". I think that means that I can't use the function for its purpose more than once which wouldn't solve my problem
Edit:
This only needs to work on Windows but later I might try to make my program computable with Linux as well. Also I am using pipes for the stdin/stdout/stderr.
Why I need to know if the child is waiting for stdin:
Currently, when the user presses the enter key, I send all of the data, that they have written so far, to stdin and disable the user from changing it. The problem is when the child process is sleeping/calculating and the user writes some input and wants to change it before the process starts reading from stdin again.
Basically lets take this program:
sleep(10)
input("Enter value:")
and lets say that I enter in "abc\n". When using cmd it will allow me to press backspace and delete the input if the child is still sleeping. Currently my program will mark all of the text as read only when it detects the "\n" and send it to stdin.
class STDINHandle:
def __init__(self, read_handle, write_handle):
self.handled_write = False
self.working = Lock()
self.write_handle = write_handle
self.read_handle = read_handle
def check_child_reading(self):
with self.working:
# Reset the flag
self.handled_write = True
# Write a character that cmd will ignore
self.write_handle.write("\r")
thread = Thread(target=self.try_read)
thread.start()
sleep(0.1)
# We need to stop the other thread by giving it data to read
if self.handled_write:
# Writing only 1 "\r" fails for some reason.
# For good measure we write 10 "\r"s
self.write_handle.write("\r"*10)
return True
return False
def try_read(self):
data = self.read_handle.read(1)
self.handled_write = False
def write(self, text):
self.write_handle.write(text)
I did a bit of testing and I think cmd ignores "\r" characters. I couldn't find a case where cmd will interpret it as an actual character (like what happened when I did "\b"). Sending a "\r" character and testing if it stays in the pipe. If it does stay in the pipe that means that the child hasn't processed it. If we can't read it from the pipe that means that the child has processed it. But we have a problem - we need to stop the read if we can't read from stdin otherwise it will mess with the next write to stdin. To do that we write more "\r"s to the pipe.
Note: I might have to change the timing on the sleep(0.1) line.
I am not sure this is a good solution but you can give it a try if interested. I just assumed that we execute the child process for its output given 2 inputs data and TIMEOUT.
process = subprocess.Popen(command, close_fds=False, shell=True, **file_descriptors)
try:
output, _ = process.communicate(data, TIMEOUT)
except subprocess.TimeoutExpired:
print("Timeout expires while waiting for a child process.")
# Do whatever you want here
return None
cmd_output = output.decode()
You can find more examples for TimeoutExpired here.
Related
I was trying to build a YouTube streamer using Rust that uses mpv player. I've managed to extract URL of music video from the YouTube search page.
I have set up an input loop to take user's commands. Actions are taken according to user commands. When the user specifies play thisSong, the music video's URL is extracted and saved as a string. Now, I want to start a process by calling mpv player. The output of mpv player should be ignored, and the player should play music in the background, and the user should be back to prompt, from where he can supply commands again.
I tried to set it up, but the problem was that as soon as thempv child process starts, it starts to take commands supplied by the user to my main program. I want mpv to ignore those commands.
let mut youtube_mpv = match Command::new("mpv")
.arg(song_url)
.arg("--no-video")
.arg("--ytdl-format=worst")
.arg("--really-quiet")
.arg("&")
.stdout(Stdio::null())
.spawn()
{
Err(_why) => exit(1),
Ok(process) => process,
};
println!("Playing {} from YouTube", song_name);
Add .stdin(Stdio::null()).
By default, the subprocess will inherit all streams from the parent. If you don't want that, either pipe them (to interact with the subprocess via stdin/stdout) or null them (to redirect to / from /dev/null).
Incidentally I don't think this:
.arg("&")
makes any sense. It's going to pass an & argument to mpv, which mpv is going to assume is a file, look up, fail to find, and trigger an error. Assuming you eventually wait() on the mpv subprocess, it'll always report failure.
url request and sniff(count=x) don't work together. sniff(count) is waiting for x packets to sniff, and though I have to put the line before the url-request it blocks the program, the url-request never starts and it never sniffs any packet.
When I opened 2 Windows in ubuntu command line, it worked. In the first window I activated the interactive mode of python and activated the sniffer. After doing that, I started the web-crawler int the second window and the sniffer in the 1st window received the packets correctly and put it on the screen / into a pcap-file.
Now the easiest way would be to write 2 scripts and start them from 2 different Windows, but I want to do the complete work in one script: Webcrawling, sniffing the packets and putting them into a pcap-file
Here is the code that does not work:
class spider():
…
def parse():
a = sniff(filter="icmp and host 128.65.210.181", count=1)
req = urllib.request.urlopen(self.next_url.replace(" ",""))
a.nsummary()
charset = req.info().get_content_charset()
Now the first line blocks the program, waiting 4 the packet to come in, what cannot do so because only in the next line the request is done. Swapping the lines also doesn't work. I think that the only way to resolve the problem is to work with paralessisms, so I've also tried this:
class protocoller():
...
def run(self):
self.pkt = sniff(count=5) # and here it blocks
…
prot = protocoller()
Main.thr = threading.Thread(target=prot.run())
Main.thr.start()
I Always thought that the thread is running indipendently from the main program, but it blocks it as if it would be part of it. Any suggestions?
So what I would need is a solution in which the web-crawler and the IP/TCP protocoller based on scapy are running independently from each other.
Could the sr()-function of scapy be an alternative?
https://scapy.readthedocs.io/en/latest/usage.html
Is it possible to put the request manually in the packet and to put the received packet into the pcap-file?
Your example doesn't show what's going on in other threads so I assume you've got a second thread to do the request etc. If all that is in order the obvious error is here:
Main.thr = threading.Thread(target=prot.run())
This executes the function prot.run and passes the result to the target parameter of Thread. It should be:
Main.thr = threading.Thread(target=prot.run)
This passes the function itself into Thread
The other answer works great.
FYI, Scapy 2.4.3 also has a native way of doing this:
https://scapy.readthedocs.io/en/latest/usage.html#asynchronous-sniffing
Some developers and I are using KornShell (ksh) to run SAS programs in a Linux environment. The script invokes a SAS command line and I wish to collect the stdout from the SAS execution (a string defined and written by SAS) as well as the Linux return code (0/1).
My Code (collects stdout into envar, but return_code is always 0 because the envar assignment was successful):
envar=$(./sas XXXX/filename.sas -log $LOG_FILE)
return_code=$?
Is there a way to collect both the return code and the std out without having to submit this command twice?
SAS does not write anything to STDOUT when it is run as a non-interactive process. The log file contains the record of statements executed and step statistics; "printed" output (such as from proc print) is written to a "listing" file. By default, that file will be created using the name of your source file appended with ".lst" (in your case, filename.lst).
You are providing a file to accept the log output using the -log system option. The related option to define the listing file is the -print option. Of course, if the program does not create any listing output, such an option isn't needed.
And as you've discovered, the value returned by $? is the execution return code from SAS. Any non-zero value will indicate some sort of error occurred during program execution.
If you want to influence the return code, you can use the ABORT data step statement in your SAS program. That will immediately halt the SAS program as set the return code to something meaningful to you. For example, suppose you want to terminate further processing if a particular PROC SQL step fails:
data _null_;
rc = symgetn('SQLRC');
put rc=;
if rc > 0 then ABORT RETURN 10;
run;
This would set the return code to 10 and you could use your outer script to send an email to the appropriate person. Such a custom return code value must be greater than 6 and less than 976; other values are reserved for SAS. Here is the SAS doc link.
Is it safe to pass a key to the openssl command via the command line parameters in Linux? I know it nulls out the actual parameter, so it can't be viewed via /proc, but, even with that, is there some way to exploit that?
I have a python app that I want to use OpenSSL to do the encryption/description through stdin/stdout streaming in a subprocess, but I want to know my keys are safe.
Passing the credentials on the command line is not safe. It will result in your password being visible in the system's process listing - even if openssl erases it from the process listing as soon as it can, it'll be there for an instant.
openssl gives you a few ways to pass credentials in - the man page has a section called "PASS PHRASE ARGUMENTS", which documents all the ways you can pass credentials into openssl. I'll explain the relevant ones:
env:var
Lets you pass the credentials in an environment variable. This is better than using the process listing, because on Linux your process's environment isn't readable by other users by default - but this isn't necessarily true on other platforms.
The downside is that other processes running as the same user, or as root, will be able to easily view the password via /proc.
It's pretty easy to use with python's subprocess:
new_env=copy.deepcopy(os.environ)
new_env["MY_PASSWORD_VAR"] = "my key data"
p = subprocess.Popen(["openssl",..., "-passin", "env:MY_PASSWORD_VAR"], env=new_env)
fd:number
This lets you tell openssl to read the credentials from a file descriptor, which it will assume is already open for reading. By using this you can write the key data directly from your process to openssl, with something like this:
r, w = os.pipe()
p = subprocess.Popen(["openssl", ..., "-passin", "fd:%i" % r], preexec_fn=lambda:os.close(w))
os.write(w, "my key data\n")
os.close(w)
This will keep your password secure from other users on the same system, assuming that they are logged in with a different account.
With the code above, you may run into issues with the os.write call blocking. This can happen if openssl waits for something else to happen before reading the key in. This can be addressed with asynchronous i/o (e.g. a select loop) or an extra thread to do the write()&close().
One drawback of this is that it doesn't work if you pass closeFds=true to subprocess.Popen. Subprocess has no way to say "don't close one specific fd", so if you need to use closeFds=true, then I'd suggest using the file: syntax (below) with a named pipe.
file:pathname
Don't use this with an actual file to store passwords! That should be avoided for many reasons, e.g. your program may be killed before it can erase the file, and with most journalling file systems it's almost impossible to truly erase the data from a disk.
However, if used with a named pipe with restrictive permissions, this can be as good as using the fd option above. The code to do this will be similar to the previous snippet, except that you'll need to create a fifo instead of using os.pipe():
pathToFifo = my_function_that_securely_makes_a_fifo()
p = subprocess.Popen(["openssl", ..., "-passin", "file:%s" % pathToFifo])
fifo = open(pathToFifo, 'w')
print >> fifo, "my key data"
fifo.close()
The print here can have the same blocking i/o problems as the os.write call above, the resolutions are also the same.
No, it is not safe. No matter what openssl does with its command line after it has started running, there is still a window of time during which the information is visible in the process' command line: after the process has been launched and before it has had a chance to null it out.
Plus, there are many ways for an accident to happen: for example, the command line gets logged by sudo before it is executed, or it ends up in a shell history file.
Openssl supports plenty of methods of passing sensitive information so that you don't have to put it in the clear on the command line. From the manpage:
pass:password
the actual password is password. Since the password is visible to utilities (like 'ps' under Unix) this form should only be used where security is not important.
env:var
obtain the password from the environment variable var. Since the environment of other processes is visible on certain platforms (e.g. ps under certain Unix OSes) this option should be used with caution.
file:pathname
the first line of pathname is the password. If the same pathname argument is supplied to -passin and -passout arguments then the first line will be used for the input password and the next line for the output password. pathname need not refer to a regular file: it could for example refer to a device or named pipe.
fd:number
read the password from the file descriptor number. This can be used to send the data via a pipe for example.
stdin
read the password from standard input.
All but the first two options are good.
I'm writing a program that forks multiple child processes and I'd like for all of these child processes to be able to write lines to STDERR and STDOUT without the output being garbled. I'm not doing anything fancy, just emitting lines that end with a new line (that, at least in my understanding would be an atomic operation for Linux). From perlfaq it says:
Both the main process and the backgrounded one (the "child" process) share the same STDIN, STDOUT and STDERR filehandles. If both try to access them at once, strange things can happen. You may want to close or reopen these for the child. You can get around this with opening a pipe (see open) but on some systems this means that the child process cannot outlive the parent.
It says I should "close or reopen" these filehandles for the child. Closing is simple, but what does it mean by "reopen"? I've tried something like this from within my child processes and it doesn't work (the output still gets garbled):
open(SAVED_STDERR, '>&', \*STDERR) or die "Could not create copy of STDERR: $!";
close(STDERR);
# re-open STDERR
open(STDERR, '>&SAVED_STDERR') or die "Could not re-open STDERR: $!";
So, what am I doing wrong with this? What would the pipe example it alludes to look like? Is there a better way to coordinate output from multiple processes together to the console?
Writes to a filehandle are NOT atomic for STDOUT and STDIN. There are special cases for things like fifos but that's not your current situation.
When it says re-open STDOUT what that means is "create a new STDOUT instance" This new instance isn't the same as the one from the parent. It's how you can have multiple terminals open on your system and not have all the STDOUT go to the same place.
The pipe solution would connect the child to the parent via a pipe (like | in the shell) and you'd need to have the parent read out of the pipe and multiplex the output itself. The parent would be responsible for reading from the pipe and ensuring that it doesn't interleave output from the pipe and output destined to the parent's STDOUT at the same time. There's an example and writeup here of pipes.
A snippit:
use IO::Handle;
pipe(PARENTREAD, PARENTWRITE);
pipe(CHILDREAD, CHILDWRITE);
PARENTWRITE->autoflush(1);
CHILDWRITE->autoflush(1);
if ($child = fork) { # Parent code
chomp($result = <PARENTREAD>);
print "Got a value of $result from child\n";
waitpid($child,0);
} else {
print PARENTWRITE "FROM CHILD\n";
exit;
}
See how the child doesn't write to stdout but rather uses the pipe to send a message to the parent, who does the writing with its stdout. Be sure to take a look as I omitted things like closing unneeded file handles.
While this doesn't help your garbleness, it took me a long time to find a way to launch a child-process that can be written to by the parent process and have the stderr and stdout of the child process sent directly to the screen (this solves nasty blocking issues you may have when trying to read from two different FD's without using something fancy like select).
Once I figured it out, the solution was trivial
my $pid = open3(*CHLD_IN, ">&STDERR", ">&STDOUT", 'some child program');
# write to child
print CHLD_IN "some message";
close(CHLD_IN);
waitpid($pid, 0);
Everything from "some child program" will be emitted to stdout/stderr, and you can simply pump data by writing to CHLD_IN and trust that it'll block if the child's buffer fills. To callers of the parent program, it all just looks like stderr/stdout.