Bash read/write file descriptors -- seek to start of file - linux

Question

I tried to use the read/write file descriptor in bash so that I could delete the file that the file descriptor referred to afterward, as such:
F=$(mktemp)
exec 3<> "$F"
rm -f "$F"
echo "Hello world" >&3
cat <&3
but the cat command gives no output. I can achieve what I want if I use separate file descriptors for reading and writing:
F=$(mktemp)
exec 3> "$F"
exec 4< "$F"
rm -f "$F"
echo "Hello world" >&3
cat <&4
which prints Hello world.
I suspected that bash doesn't automatically seek to the start of the file descriptor when you switch from writing to reading it, and the following combination of bash and python code confirms this:
fdrw.sh
exec 3<> tmp
rm tmp
echo "Hello world" >&3
exec python fdrw.py
fdrw.py
import os
f = os.fdopen(3)
print f.tell()
print f.read()
which gives:
$ bash fdrw.sh
12
$ # This is the prompt reappearing
Is there a way to achieve what I want just using bash?

Answer 1

I found a way to do it in bash, but it's relying on an obscure feature of exec < /dev/stdin which actually can rewind the file descriptor of stdin according to http://linux-ip.net/misc/madlug/shell-tips/tip-1.txt:
F=$(mktemp)
exec 3<> "$F"
rm -f "$F"
echo "Hello world" >&3
{ exec < /dev/stdin; cat; } <&3
The write descriptor isn't affected by that so you can still append output to descriptor 3 before the cat.
Sadly I only got this working under Linux not under MacOS (BSD), even with the newest bash version. So it doesn't seem very portable.

Answer 2

If you ever do happen to want to seek on bash file descriptors, you can use a subprocess, since it inherits the file descriptors of the parent process. Here is an example C program to do this.
seekfd.c
#define _FILE_OFFSET_BITS 64
#include <string.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
int main(int argc, char* argv[])
{
/* Arguments: fd [offset [whence]]
* where
* fd: file descriptor to seek
* offset: number of bytes from position specified in whence
* whence: one of
* SEEK_SET (==0): from start of file
* SEEK_CUR (==1): from current position
* SEEK_END (==2): from end of file
*/
int fd;
long long scan_offset = 0;
off_t offset = 0;
int whence = SEEK_SET;
int errsv; int rv;
if (argc == 1) {
fprintf(stderr, "usage: seekfd fd [offset [whence]]\n");
exit(1);
}
if (argc >= 2) {
if (sscanf(argv[1], "%d", &fd) == EOF) {
errsv = errno;
fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
exit(1);
}
}
if (argc >= 3) {
rv = sscanf(argv[2], "%lld", &scan_offset);
if (rv == EOF) {
errsv = errno;
fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
exit(1);
}
offset = (off_t) scan_offset;
}
if (argc >= 4) {
if (sscanf(argv[3], "%d", &whence) == EOF) {
errsv = errno;
fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
exit(1);
}
}
if (lseek(fd, offset, whence) == (off_t) -1) {
errsv = errno;
fprintf(stderr, "%s: %s\n", argv[0], strerror(errsv));
exit(2);
}
return 0;
}

Answer 3

No. bash does not have any concept of "seeking" with its redirection. It reads/writes (mostly) from beginning to end in one long stream.

Answer 4

When you open a file descriptor in bash like that, it becomes accessible as a file in /dev/fd/.
On that you can do cat and it'll read from the start, or append (echo "something" >> /dev/fd/3), and it'll add it to the end.
At least on my system it behaves this way. (On the other hand, I can't seem to be able to get "cat <&3" to work, even if I don't do any writing to the descriptor).

Answer 5

Try changing the sequence of commands:
F=$(mktemp tmp.XXXXXX)
exec 3<> "$F"
echo "Hello world" > "$F"
rm -f "$F"
#echo "Hello world" >&3
cat <&3

Answer 6

#!/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<> $F
rm $F
echo "Hello world" >&3
cat /dev/fd/3
As suggested in other answer, cat will rewind the file descriptor for you before reading from it since it thinks it's just a regular file.

Answer 7

To 'rewind' the file descriptor, you can simply use /proc/self/fd/3
Test script :
#!/bin/bash
# Fill data
FILE=test
date +%FT%T >$FILE
# Open the file descriptor and delete the file
exec 5<>$FILE
rm -rf $FILE
# Check state of the file
# should return an error as the file has been deleted
file $FILE
# Check that you still can do multiple reads or additions
for i in {0..5}; do
echo ----- $i -----
echo . >>/proc/self/fd/5
cat /proc/self/fd/5
echo
sleep 1
done
Try to kill -9 the script while it is running, you will see that contrary to what happens with the trap method, the file is actually deleted.

Answer 8

Expansion on the answer by #sanmai...
And confirmation of what is going on...
#/bin/bash
F=$(mktemp tmp.XXXXXX)
exec 3<>$F # open the temporary file for read and write
rm $F # delete file, though it remains on file system
echo "Hello world!" >&3 # Add a line to a file
cat /dev/fd/3 # Read the whole file
echo "Bye" >>/dev/fd/3 # Append another line
cat /dev/fd/3 # Read the whole file
echo "Goodbye" >&3 # Overwrite second line
cat /dev/fd/3 # Read the whole file
cat <&3 # Try to Rewind (no output)
echo "Cruel World!" >&3 # Still adds a line on end
cat /dev/fd/3 # Read the whole file
shell_seek 3 6 0 # seek fd 3 to position 6
echo -n "Earth" >&3 # Overwrite 'World'
shell_seek 3 # rewind fd 3
cat <&3 # Read the whole file put 3 at end
Note that the echo Goodbye overwrites the second lineas the file descriptor &3 had not changed by the cat!
So I tried using cat <&3 which did not output anything, probably as the file descriptor was at the end of the file. To see it if it rewinds the descriptor it was given. It does not.
The last part is to use the 'C' program that was provided, compiled and named shell_seek and yes it seems it works as the first 'World' was replaced by 'Earth', the rewind (seek to start) worked allowing the last cat to again read the whole file. It would put the fd at the end of the file again!
Doing it using perl instead of C was not that hard either.
For example perl -e 'open(FD,">&3"); seek(FD,0,0);' will rewind file descriptor 3 back to the start of the file.
I have now made a perl version of shell_seek so I don't have to re-compile it all the time for different systems. Not only that but the script can also 'tell' you the current file descriptor offset, and also 'truncate' the file that file descriptor points too. Both operations are commonly used when using seek, so it seemed a good idea to include those functions. You can download the script from...
https://antofthy.gitlab.io/software/#shell_seek