I am currently trying to make a server client architecture for any program using pipes as its stdin and stdout.
I do have the whole server-thing working, and it's communicating through named pipes(fifo).
What I am trying to do is to read the content of the stdout pipe into a vim buffer, constantly (asynchronously), and I am looking for the most elegant solution to this problem.
Say I have a program, python, communicating through 2(3 if you account stderr but it's not relevant to finding the solution) pipes, namely PIPE_IN as its stdin, and PIPE_OUT as its stdout.
I can write to the pipe easily from vim (with :w), how would I however read from PIPE_OUT into a vim or neovim buffer?
This is not supposed to be portable to non-UNIX systems, however I would like to keep compatibility with both vim and neovim as much as possible. What would be your recommendations to do so? Should I use the vimscript language for this? Is using vim's command-line mode better in that case? What other, probably more elegant solutions, am I missing?
Thanks in advance!
NB: The server project's code can be found at Soulthym/pyper on GitHub for testing purposes.
NeoVim and modern Vim support channels which can be used for asynchronous communication with external components.
In particular, NeoVim supports a sockconnect() which supports named pipes:
Connect a socket to an address. If {mode} is "pipe" then {address} should be the path of a named pipe. [...] Returns a channel ID.
When creating a channel, you can attach a callback to get notifications when there's data to read. You also have functions to poll a channel to see if there's data available.
(An aside, but consider using Unix-domain sockets rather than pipes, they're much more featureful. They're bidirectional, so you don't need multiple pairs of them. Also you can use them to implement servers that can listen and accept multiple connections at the same address.)
Related
ncurses' blocking mode appears to only allow for blocking on stdin - that is, I have no option of doing some select()-esque thing where getch() blocks until either a key is pressed or e.g. data arrives in a pipe. Am I missing something, or is there some other terminal control library that allows me to do this?
Edit: I'm targeting Linux platforms, although the more general the solution, the better.
There's an experimental configure-option, which is rarely used:
--enable-wgetch-events
Compile with experimental wgetch-events code. See ncurses/README.IZ
So I have finished reading an article here:
https://msdn.microsoft.com/en-us/library/ms809983.aspx
about why we have COM and how it lets us share code without worrying about name mangling of compilers or unicode/ascii issues or memory management in a language independent manner.
I have elsewhere read that COM isn't supposed by LINUX because COM basically uses the OS as the moderator for acquisition of these standardized objects. Shouldn't there be something similar in Linux? and if so, what is it?
On Linux you can run any program that accepts its input on standard input, and connect it, via a pipe, to any other program that generates its results on its standard output.
The simple, file and pipe-based input/output in POSIX predates MS-Windows by decades. And, as long as both sides of the pipe agree on the format of the data being interchanged, it doesn't matter which compiler was used to create each program (although, on Linux, there's pretty one de-factor compiler, so it's a moot point).
And by using a socket-pair, the pipe becomes bi-directional, so both processes can swap data with each other.
This is, generally, how processes interoperate on Linux:
1) A pipe, or a network socket, that connects the two processes together
2) An agreed, established standard for the format of the data exchanged between the two processes.
It is important to understand that there is no practical standard for all processes to use the same exact format for exchanging messages. The closest that would come to such a standard, I suppose, would be the remote procedure call, or RPC, standard that's used in some low-level protocols, like NFS, but, mostly, individual applications define and use a particular format that's tailored for them.
For example, the X Window System Protocol: http://www.x.org/releases/X11R7.7/doc/xproto/x11protocol.html -- this is a format definition of a protocol for communicating between an X server and an X client. Applications that are written to use this protocol (they'll typically use an intermediate library or a toolkit, actually) can establish a connection and use any X server that talks the same protocol, over a network connection or a local pipe.
I have some tasks requiring massive temporary named pipes to deal with.
Originally, I just simply think that generate random numbers, then append it as <number>.fifo be the name of named pipe.
However, I found this post: Create a temporary FIFO (named pipe) in Python?
It seems there is something I don't know that may cause some security issue there.
So my question here is that, what's the best way to generate a named pipe?
Notice that even though I am referencing a Python related post, I don't really mean to ask only in Python.
UPDATE:
Since I want to use a named pipe to connect unrelated processes, my plan is having process A call process B first via shell, and capture stdout to acquire the name of pipe, then both know what to open.
Here I am just worrying about whether leaking the name of pipe will become an issue. Before I never thought of it, until I read that Python post.
If you have to use named FIFOs and need to ensure that overlap/overwriting cannot occur, your best bet is probably to use some combination of mktemp and mkfifo.
Although mktemp itself cannot create FIFOs, it can be used to create unique temporary directories, which you can then put your FIFOs into.
The GNU mktemp documentation has an example of this.
Alternatively, you could create some name containing well random letters. You could read from /dev/random (or /dev/urandom, read random(4)) some random bytes to e.g. seed a PRNG (e.g. random(3) seeded by srandom), and/or mix the PID and time, etc.
And since named fifo(7) are files, you should use the permission system (and/or ACL) on them. In particular, you might create a command Linux user to run all your processes and restrict the FIFOs to be only owner-readable, etc.
Of course, and in all cases, you need to "store" or "transmit" securely these FIFO names.
If you start your programs in some bash script, you might consider making your fifo names using mktemp(1) as:
fifoname=$(mktemp -u -t yourprog_XXXXXX).fifo-$RANDOM-$$
mkfifo -m 0600 $fifoname
(perhaps in some loop). I guess it would be secure enough if the script is running in a dedicated user (and then pass the $fifoname in some pipe or file, not as a program argument)
The recent renameat2(2) syscall might be helpful (atomicity of RENAME_EXCHANGE).
BTW, you might want some SElinux. Remember that opened file descriptors -and that includes your fifos- are available as symlinks in proc(5) !
PS. it all depends upon how paranoid are you. A well sysadmined Linux system can be quite secure...
In Linux (2.6.35-22-generic), man pipe states that
pipe() creates a pipe, a unidirectional data channel that can be used for interprocess communication."
In FreeBSD (6.3-RELEASE-p5), man pipe states that
The pipe() system call creates a pipe, which is an object allowing bidirectional data flow, and allocates a pair of file descriptors."
One is unidirectional, the other is bidirectional. I hope this isn't a silly question, but which method is the standard way of doing this? Are they both POSIX compliant?
To make my intentions clear, I lost some points on an exam for believing pipe() was one way and am looking for some ammo to get any points back ;p
I started this as a comment on Greg's answer at first, but it occurs to me that it more closely answers your specific question:
pipe()s documentation in the POSIX standard explicitly states that the behavior in question is "unspecified" -- that is, pipe() is not required to be bidirectional, though it's not forbidden. Linux's is unidirectional, FreeBSD's is bidirectional. Both are compliant, one just implements additional behavior that is not required (but doesn't break apps built to work on compliant systems).
Data can be written to the file
descriptor fildes[1] and read from the
file descriptor fildes[0]. A read on
the file descriptor fildes[0] shall
access data written to the file
descriptor fildes[1] on a
first-in-first-out basis. It is
unspecified whether fildes[0] is also
open for writing and whether fildes[1]
is also open for reading.
I wouldn't count on getting the points back (though you should). Professors have a tendency to ignore the real world in favor of whatever they've decided is correct.
The FreeBSD man page for pipe is pretty clear on this point:
The bidirectional nature of this implementation of pipes is not portable to older systems, so it is recommended to use the convention for using the endpoints in the traditional manner when using a pipe in one direction.
I am developing under Linux with pretty tight constraints on disk usage. I'd like to be able to point logging to a fixed-size file. For example, if my application outputs all logs to stdout:
~/bin/myApp > /dev/debug1
and then, to see the last amount of output:
cat /dev/debug1
would write out however many bytes debug1 was setup to save (if at least that many had been written there).
This post suggests using expect or its library, but I was wondering if anyone has seen a "pseudo-tty" device driver-type implementation as I would prefer to not bind any more libraries to my executable.
I realize there are other mechanisms like logrotate, but I'd prefer to have a non-cron solution.
Pointers, suggestions, questions welcome!
Perhaps you could achieve what you want using mkfifo and something that reads the pipe with a suitable buffer. I haven't tried, but less --buffers=XXXXXX might work for this.