While researching this question I came across the fact that in POSIX (and Linux) there simply is not a truncateat system call.
Certain system calls like for instance unlink have an equivalent alternative method with an added at suffix at the end of their names, i.e. unlinkat. The difference between those methods is that the variations with the at suffix accept an additional argument, a file descriptor pointing to a directory. Therefore, a relative path passed into unlinkat is not relative to the current working directory but instead relative to the provided file descriptor (an open directory). This is really useful under certain circumstances.
Looking at truncate, there only is ftruncate next to it. truncate works on paths - absolute or relative to the current working directory. ftruncate directly works on an open file handle - without any path being specified. There is no truncateat.
A lot of libraries (various "alternative" C-libraries) do what I did and mimic tuncateat by using an openat-ftruncate-close-sequence. This works, in most cases, except ...
I ran into the following issue. It took me months to figure out what was happening. Tested on Linux, different 3.X and 4.X kernels. Imagine two processes (not threads):
Process "A"
Process "B"
Now imagine the following sequence of events (pseudo code):
A: fd = open(path = 'filename', mode = write)
A: ftruncate(fd, 100)
A: write(fd, 'abc')
B: truncate('filename', 200)
A: write(fd, 'def')
A: close(fd)
The above works just fine. Just after process "A" has the file opened, set its size to 100 and written some stuff into it, process "B" re-sets its size to 200. Then process "A" continues. At the end, the file has a size of 200 and contains "abcdef" at its beginning followed by zero-bytes.
Now, let's try and mimic something like truncateat:
A: fd_a = open(path = 'filename', mode = write)
A: ftruncate(fd_a, 100)
A: write(fd_a, 'abc')
B: fd_b = openat(dirfd = X, path = 'filename', mode = write | truncate)
B: ftruncate(fd_b, 200)
B: close(fd_b)
A: write(fd_a, 'def')
A: close(fd_a)
My file has a length of 200, ok. It starts with three zero-bytes, not ok, then the "def", then then again zero-bytes. I have just lost the first write from process "A" while the "def" technically landed at the correct position (three bytes in, as if I had called seek(fd_a, 3) before writing it).
I can work with the first sequence of operations just fine. But in my use case, I can not rely on paths relative the current working directory as far as process "B" is concerned. I really want to work with paths relative to a file descriptor. How can achieve that - without running into the issue demonstrated in the second sequence of operations? Calling fsync from process "A" just after write(fd_a, 'abc') does not solve this.
The reason why your second case overwrites everything with zeroes is that mode=truncate (i.e. openat(.., O_TRUNC)) will first truncate the file to length 0.
If you instead ftruncate to 200 immediately without first truncating to 0, the existing data up until that point will remain untouched.
Related
Suppose I have a file named test.txt and it currently has the number 6 inside of it. I want to use a variable such as x=4 then write to the file and add the two numbers together and save the result in the file.
var1 = 4.0
f=open(test.txt)
balancedata = f.read()
newbalance = float(balancedata) + float(var1)
f.write(newbalance)
print(newbalance)
f.close()
It's probably simpler than you're trying to make it:
variable = 4.0
with open('test.txt') as input_handle:
balance = float(input_handle.read()) + variable
with open('test.txt', 'w') as output_handle:
print(balance, file=output_handle)
Make sure 'test.txt' exists before you run this code and has a number in it, e.g. 0.0 -- you can also modify the code to deal with creating the file in the first place if it's not already there.
Files only read and write strings (or bytes for files opened in binary mode). You need to convert your float to a string before you can write it to your file.
Probably str(newbalance) is what you want, though you could customize how it appears using format if you want. For instance, you could round the number to two decimal places using format(newbalance, '.2f').
Also note that you can't write to a file opened only for reading, so you probably need to either use mode 'r+' (which allows both reading and writing) combined with a f.seek(0) call (and maybe f.truncate() if the length of the new numeric string might be shorter than the old length), or close the file and reopen it in 'w' mode (which will truncate the file for you).
I don't find a way to check the free space available in a device using Haxe, Openfl, Lime or another library.
I would like to avoid download data that will exceed the size recommended for an app in each device.
What do you do to check that?
Try creating a file of that size! Then either delete it or reopen and write (not append) over its contents.
I don't know whether all platforms Haxe supports will work fine with this trick, but this algorithm is reported to work in many places and languages (I personally tested it in Ruby and saw the same suggestion for C++/.NET). To check whether X bytes of disk space are available:
open a new file for writing
seek X-1 bytes from the beginning
write a byte of data (whatever you want, 0, 42...)
close the file (probably unrelated to the task at hand, but don't forget to do that anyway)
If there's insufficient disk space, you'll likely get an exception at some point in this algorithm. You'll have to find out what errors to expect and process them properly.
Using ihx I've found this is working and requires nothing but Haxe Standard Library:
haxe interactive shell v0.3.4
type "help" for help
>> import sys.io.*;
>> var f = File.write('loca', true)
sys.io.FileOutput : { __f => #abstract }
>> f.seek(39999, FileSeek.SeekBegin)
Void : null
>> f.writeByte(0)
Void : null
>> f.close()
Void : null
After these manipulations, I had a file named loca of exactly 40000 bytes in my working directory.
By the way, be careful when doing things like these in ihx since it re-runs the entire session with the last entered line appended each time.
Ongoing experimentation:
However, when there's insufficient disk space, it may not fail with errors. In this case you'll have to check the real size with sys.FileSystem.stat(path).size. And don't forget to delete the file if there's not enough space.
The man page is here: http://man.cat-v.org/unix-6th/3/ttyn
This example:
if (ttyn(0) = 'x'){
...
}
The man page says "x is returned if the indicated file does not correspond to a
typewriter."
The indicated file would be argument 0, so the standardinput, right?
And what is a typewriter? My keyboard?
What are you checking with this line?
if (ttyn(0) = 'x')
At that point in time, a typewriter (or teletype, or tty) was an RS-232 terminal connected to the computer via a serial port. The device entries in /dev corresponding to these ports were named /dev/tty0, /dev/tty1, /dev/ttya, etc. Each of those files was a character special file, as opposed to an ordinary file.
When a terminal was detected by the system, typically by being turned on or connected through a modem, the init process opened the device on file descriptors 0, 1, and 2 in a new process, and those file descriptors persisted through the login process, a user's shell, and any processes forked from the shell.
As you said in your question, file descriptor 0 is also called standard input.
The ttyn function calls fstat on its argument, which returns some info about the file such as its inode number, permissions, etc. ttyn then reads through /dev, looking at each file that starts with "tty", to see which one has the same inode number as ttyn's argument. When it finds a match, it returns the 4th character of the filename, which would be '0', '1', 'a', etc. If no matches are found, it returns 'x'.
There were generally a console and a few 8-port serial interfaces on a PDP-11. so there was no ttyx. And you could name devices in /dev anything you wanted. So it was easy to avoid /dev/ttyx being an actual device.
Commands like goto could use ttyn(0) != 'x' to determine whether the user was actually typing the command on a terminal.
Here is the default config file, /etc/ttys, used by init in V6. The console was tty8.
In V7 Unix, the functionality of ttyn was replaced by ttyname, which could accommodate longer device names, and isatty, which returned true if the fle descriptor was a terminal device. The goto command was not present in V7.
I've never seen this library call before; I'm used to the more familiar ttyname. The webpage doesn't give a return value, but based on what the text says, it would give the last char value in the string returned by ttynam(3). So if stdin (fd0) was connected to "/dev/tty2", then the return value would be the char 2. And in C, you would be able to check it with:
if (ttyn(0) == '2') { ... }
Granted the documentation is not clear. And it is using bad terminology; instead of "typewriter", it should be using "teletype" or "terminal", which are the accepted terms. Remember that stdin can be different from stdout; it is perfectly possible to do run cat </dev/tty1 > /dev/tty2, assuming you have the permissions for it.
I have a perl script that traverses a set of directories and when it hits one of them it blows up with an Invalid Argument and I want to be able to programmatically skip it. I thought I could start by finding out the file type with the file command but it too blows up like this:
$ file /sys/devices/virtual/net/br-ex/speed
/sys/devices/virtual/net/br-ex/speed: ERROR: cannot read `/sys/devices/virtual/net/br-ex/speed' (Invalid argument)
If I print out the mode of the file with the perl or python stat function it tells me 33060 but I'm not sure what all the bits mean and I'm hoping a particular one would tell me not to try to look inside. Any suggestions?
To understand the stats number you got, you need to convert the number to octal (in python oct(...)).
Then you'll see that 33060 interprets to 100444. You're interested only in the last three digits (444). The first digit is file owner permissions, the second is group and the third is everyone else.
You can look at each of the numbers (in your case all are 4) as 3 binary bits in this order:
read-write-execute.
Since in your case owner, group & other has 4, it is translated (for all of them) to 100 (in binary) which means that only the read bit is on for all three - meaning that all three can only read the file.
As far as file permissions go, you should have been successful reading /sys/devices/virtual/net/br-ex/speed.
There are two reasons for the read to fail:
- Either speed is a directory, (directories require execute permissions to read inside).
- Or it's a special file - which can be tested using the -f flag in perl or bash, or using os.path.isfile(...) in python.
Anyhow, you can use the following links to filter files & directories according to their permissions in the 3 languages you mentioned:
ways to test permissions in perl.
ways to test permissions in python.
ways to test permissions in bash.
Not related to this particular case, but I hit the same error when I ran it on a malicious ELF (Linux executable) file. In that case it was because the program headers of the ELF was intentionally corrupted. Looking at the source code for file command, this is clear as it checks the ELF headers and bails out with the same error in case the headers are corrupted:
/*
* Loop through all the program headers.
*/
for ( ; num; num--) {
if (pread(fd, xph_addr, xph_sizeof, off) <
CAST(ssize_t, xph_sizeof)) {
file_badread(ms);
return -1;
}
TLDR; The file command checks not only the magic bytes, but it also performs other checks to validate a file type.
I'm trying to edit an existing binary file using NodeJS.
My code goes something like this:
file = fs.createWriteStream("/path/to/existing/binary/file", {flags: "a"});
file.pos = 256;
file.write(new Buffer([0, 1, 2, 3, 4, 5]));
In OS X, this works as expected (The bytes at 256..261 get replaced with 0..5).
In linux however, the 5 bytes get appended to the end of file. This is also mentioned in the NodeJS API Reference:
On Linux, positional writes don't work when the file is opened in append mode. The kernel ignores the position argument and always appends the data to the end of the file.
How do I get around this?
Open with a mode of r+ instead of a. r+ is the portable way to say that you want to read and/or write to arbitrary positions in the file, and that the file should already exist.