I am trying to build a python GUI app which needs to run bash scripts and capture their output. Most of the commands used in these scripts require further inputs at run time, as they deal with network and remote systems.
Python's subprocess module allows one to easily run external programs, while providing various options for convenience and customizability.
To run simple programs that do not require interaction, the functions call(), check_call(), and check_output() (omitting arguments) are very useful.
For more complex use cases, where interaction with the running program is required, Popen Objects can be used, where you can customize input/output pipes, as well as many other options - the aformentioned functions are wrappers around these objects. You can interact with a running process through the provided methods poll(), wait(), communicate(), etc.
As well, if communicate() doesn't work for your use case, you can get the file descriptors of the PIPEs via Popen.stdin, Popen.stdout, and Popen.stderr, and interact with them directly with select. I prefer Polling Objects :)
Related
In NodeJS, how does one make the c-level exec call, like other languages and runtimes allow (e.g. Python's os.exec() or even Bash's exec), please?
Unlike the child_process module (and all the wrappers I've found for it so far), this would be meant to completely replace the current NodeJS process by the execed process.
What I am aiming for is to create a NodeJS cli adaptor of sorts, that you provide with one set of configuration options and you can use it to dispatch different tools that use the same configuration, but each has a different expectations on the format (some expect ENV VARs, some command-line arguments, some need a file, ...).
The reason why I am looking for such an exec call is that, once the tool gets dispatched, I have no expectation on the NodeJS process sticking around -- there's no reason for it to continue existing. At the same time, I need to tool to become the foreground process (accept all signals, control characters, ...). It seems to me that it would be possible to forward all such events from the parent NodeJS to the child tool (if I used child_process), but it seems a bit too much to implement from scratch...
Cheers!
OK, so I think I have finally got it. I am posting this as an answer as it actually shows a real code example:
child_process.spawnSync(
"bash",
[
"-ic",
["psql"],
],
{
stdio: "inherit",
}
);
process.exit();
This will spin up bash (with control tty) and inside it psql. Pressing ctrl-c does not kill the node process; it is, as expected, sent to psql. I have no idea if it would be possible to rewrite this code in a way where the node process terminates before psql and bash, but for my use case, this does it. Maybe it would be even possible to get it done without bash, but I think that bash -i is actually the key here.
I see that Origen supports passing jobs to the program command in this video. What would be the preferred method to run the program command in a job loop (i.e. job == 'ws' then job == 'ft', etc.).
thx
The job is a runtime concept, not a compile/generate time concept, so it doesn't really make sense to run the program command (i.e. generate the program) against different settings of job.
Origen doesn't currently provide any mechanism to pass define-type arguments through to the program generator from the command line, though you could implement that in your app easily enough by overriding the program command - i.e. capture and store them somewhere in your app and then continue with the regular command.
The 'Origen-way' of doing things like this is to setup different target files with different variables set within them, then execute the program command for the different targets.
I'm mainly posting this as I'm not sure if the multiprocessing lib is available for python 3x and if that's not the case, I need something that will allow one python script to send data as cleanly and efficiently as possible to another. They are separate so I cannot call one of them using import.
To explain it more in detail, I have to bots running with the discord.py library and so I cannot run one under the other using a function or class, but I want to pass data between them that way they can communicate without having to write into a file or enter a submission in chat.
What you are looking for is called interprocess communication.
They are gathered together at https://docs.python.org/3/library/ipc.html - you can dig depper into module signal or mmap - which is using memory mapped files which you excluded by choice.
I only ever worked with named pipes - both programs use the same name and communicate over a named pipe:
FIFOs are pipes that can be accessed like regular files. FIFOs exist until they are deleted (for example with os.unlink()). Generally, FIFOs are used as rendezvous between “client” and “server” type processes: the server opens the FIFO for reading, and the client opens it for writing. Note that mkfifo() doesn’t open the FIFO — it just creates the rendezvous point.
Availability: Unix.
(cited from above link)
For windows use: win32pipe, win32file and win32api - see Windows named pipes in practice)
An unix example can be found in this answer to that question: Python read named PIPE
I'd like to create an auto-testing/grading script for students on a Linux system such that:
Any student user can initiate the script at any time.
A separate script (with root privileges) copies student code to a non-student-accessible file space, using non-student-accessible unit tests, etc.
The user receives limited feedback in the form of a text file generated by the grading script.
In short, I'm looking to create something similar to programming contest submission systems, but allowing richer feedback without revealing all teacher unit testing.
I would imagine that a spooling behavior between one initiating script and one root-permission cron script might be in order. Are there any models/examples of how one might best structure communication between a user-initiated script and a separate root-initiated script for such purposes?
There are many options.
The things I would mention at the first line:
Don't use su; use sudo; there are several reasons for it, and the main reason, that to use su you need the password of the user you want to be and with sudo — you don't;
Scripts can't be suid, you must use binaries or just a normal script that will be started using sudo (of course students must have sudoers entry that allows them to use the script);
Cron is not that fast, as you may theoretically need; cron runs tasks every minute; please consider inotify usage;
To communicate between components of your system you need something that will react in realtime; there are many opensource components/libraries/frameworks that could help you, but I would recommend you to take a look at ZeroMQ and Redis;
Results of the scripts' executions/tests can be written either to a filesystem (I think it would be better), or to a DBMS.
If you want to stick to shell scripting, the method I suggest for communicating between processes would be to have the root script continually check a named pipe for input (i.e. keep opening it after each eof) and send each input through whatever various tests must be done. Have part of the input be a 'return address' - where to send the result.
This should allow the tests to be performed in a privileged space without exposing any control over the privileged space to the students. The students don't need sudo, and you don't need to pull in libraries. Just have the students pipe their code into a non-privileged script that adds the return address and whatever other markup you may need, which then gives it to the named pipe.
What is the difference between a command, function and systemcall?
This is probably homework help, but anyhow:
Command - A program (or a shell built-in) you execute from (probably) your command shell.
Function - A logical subset of your program. Calling one is entirely within your process.
Systemcall - A function that is executed by your operating system; a primary way of using OS features like working with a file system or using the network.
A command can be a program, which in turn is comprised of functions, which themselves can execute system calls.
For example, the 'cp' command in Unix-like systems copies files. Its implementation includes functions which perform the copying. Those functions themselves execute system-calls like open() and read().
They are all just abstractions of a set of computer instructions which perform a given task.