What's the simplest way to send a simple string to a process (I know the pid of the process and I don't need to do any check of any sort) in python 3-4 ?
Should I just go for a communication via socket ?
If you want only two processes communicate, you can use a pipe (it can be named using mkfifo or "anonymous" using pipe & dup syscalls).
If you want one server and some clients, then you can use sockets. TCP socket is usable over network, but on unix/linux exists also so-called "unix socket", that look alike named pipe.
Next way to communicate between applicaitons are realtime signals and/or shared memory.
More information about unix socket: http://beej.us/guide/bgipc/output/html/multipage/unixsock.html
About pipes, google will tell you.
But, to correctly answer your question (which do not have only one "right answer"), I think that simplest is using named pipe, because it is used as writing to file on disk.
Related
I'm mainly posting this as I'm not sure if the multiprocessing lib is available for python 3x and if that's not the case, I need something that will allow one python script to send data as cleanly and efficiently as possible to another. They are separate so I cannot call one of them using import.
To explain it more in detail, I have to bots running with the discord.py library and so I cannot run one under the other using a function or class, but I want to pass data between them that way they can communicate without having to write into a file or enter a submission in chat.
What you are looking for is called interprocess communication.
They are gathered together at https://docs.python.org/3/library/ipc.html - you can dig depper into module signal or mmap - which is using memory mapped files which you excluded by choice.
I only ever worked with named pipes - both programs use the same name and communicate over a named pipe:
FIFOs are pipes that can be accessed like regular files. FIFOs exist until they are deleted (for example with os.unlink()). Generally, FIFOs are used as rendezvous between “client” and “server” type processes: the server opens the FIFO for reading, and the client opens it for writing. Note that mkfifo() doesn’t open the FIFO — it just creates the rendezvous point.
Availability: Unix.
(cited from above link)
For windows use: win32pipe, win32file and win32api - see Windows named pipes in practice)
An unix example can be found in this answer to that question: Python read named PIPE
Suppose I have application A that takes some time to load (opens a couple of libraries). A processes stdin into some stdout.
I want to serve A on a network over a socket (instead of stdin and stdout).
The simplest way of doing that efficiently that I can think of is by hacking at the code and adding a forking server loop, replacing stdin and stdout with socket input and output.
The performance improvement compared to having an independent server application that spawns A (fork+exec) on each connection comes at a cost however. The latter is much easier to write and I don't need to have access to the source code of A or know the language it's written in.
I want my cake and eat it too. Is there a mechanism that would extract that forking loop?
What I want is something like fast_spawnp("A", "/tmp/A.pid", stdin_fd, stdout_fd, stderr_fd) (start process A unless it's already running, clone A from outside and make sure the standard streams of the child point to the argument-supplied file descriptors).
Use named pipe some times very convenient, such as mkfifo file.fifo.
but the file.fifo is not persistent, if the computer restarted or the writer process crashed, I can get nothing from the pipe. so, are there any methods to let the piped data store in disk rather than memory?
thanks.
The simplest solution is to use plain files to store the data. For example, and use a pipe (or similar) to notify that there are new data, for example. You must take care of interprocess locking, of course.
Or you can use "message queues" (see mqueue.h). They are persistent in case of process crash, but not if the system is rebooted.
Or you can use a third-party library that implements "persistent message queues", such as MQTT or RabbitMQ.
Let's say I create a socketpair() and I pass the handle of one of the socket to a spawned process (popen), will the said process be able to communicate back with the parent?
The examples I saw are applied using fork() which is out of scope for my current project.
Updated: I tried a simple test:
Client: socketpair with sockets[0]
From Client use posix_spawn with sockets1 as command-line argument
Client: write to socket ... Client exits without any warning...
It would appear that there is a problem with this method.
UPDATED: I also found this note:
Pipes and socketpairs are limited to communication between processes with a common ancestor.
The man page for execve states:
File descriptors open in the calling process image remain open in the new
process image, except for those for which the close-on-exec flag is set
(see close(2) and fcntl(2)). Descriptors that remain open are unaffected
by execve().
Since functions like popen are based on execve, then the file descriptors that you got from your socketpair function should be good across both processes, and I don't see why you can't pass the descriptor in whatever manner pleases you. I'm assuming that in this case you mean to convert it to a string and set it over STDIN to the sub-process, which would convert it back to an int to use as a file descriptor.
It would certainly be worth writing some trial code for.
Yes, you can pass it to the child process. The trick is really that socketpair() gives you a pair of connected sockets - make sure that the child keeps one and the parent keeps the other (the parent should close the child's and vice versa).
Most cases use a pair of pipes instead though.
I would like to implement a socket-like object in user space. There's an important requirement that it should be pollable (i.e. it's state should be queryable via select or poll call).
Is there a platform neutral way of implementing such an object?
I'm aware that on Linux there's eventfd which kind of suits the needs except that there's no way to force it to signalize neither POLLIN nor POLLOUT.
You can use socketpair() to create a pair of connected AF_UNIX sockets. This is better than pipe() as it allows for bidirectional communication. If this isn't good enough for your needs, another option (which requires root for a daemon) would be to use the as-yet-not-in-mainline-Linux CUSE patches to create a device driver in userspace to do whatever you like. Or you can just hook into whatever event loop your user will be using...
The new linux eventfd can also emulate POLLIN/POLLOUT, although not both at once - set its value to 0xfffffffffffffffe for POLLIN but not POLLOUT, 0 for POLLOUT but not POLLIN, or anything else for both.
Other than these options, there's no platform-neutral way to do this, no. The usual pattern is to use a FIFO just to wake up the event loop, and have it poll using some other API once it's awake.
You want to build an user space object, that will be accessible through system call ?
ie open, read, write etc ... are redirected to your userspace object ?
You need either kernel support or libc support, otherwise I don't see how you can redirect your system call.
eventfd is not what you are asking for, it is implemented in kernel space. Did you describe your real problem ? Could fifo or unix domain socket fit your need ?
What about pseudo tty ? I don't know if you can block writing from the master side by faking the hardware flow control.
It's really not clear what you're trying to do; if you want a socket-like device, why not use sockets? You don't say ... And what's the deal with POLLIN and POLLOUT?
I kinda suspect you might be interested in using pseudo-terminal devices, see man 7 pty.
Use pipe(). It gives you two fd's, one to write, one to read.
Use the fd[1] to do your select/poll on.
Use the fd[0] to signal your select/poll for activity.