In Linux, for a single file what is the dirty page write back(to disk) order ? is it from beginning to end ? or out of order ?
Scenario 1 : without overwrite
A file(in disk) is created and large amount of data written (sequentially) quickly. Now I presume these would be in multiple page caches. When writing back the dirty pages is pages written back in order ?
e.g. Say server shutdown before file write completion.
Now after reboot can we have the disk file in below state
|--correct data --|---data unset/garbage--|--correct data--|
i.e. I understand last set of bytes in file can be incomplete, but can data in mid be incomplete
Scenario 2 : with overwrite (attempt to use file similar to a circular/ring buffer)
File created, data written, after reaching a max size, "fsync"
called (i.e. data+meta data synchronized).
Now, file pointer is
moved to the beginning of the file and data written sequentially.
(No fsync done)
Now due to a server shutdown can we have the disk file in below state after reboot
|--Newly written data--|--Old data--|--New data--|...
i.e. for new data, some pages were written to disk out of order
OR
can I assume it is always
|--Newly written data--|----Newly written data--|--Old data--|
i.e. old data and new data will not mix-up (if present old data would only be at the end of file)
Related
I am using fs.writeSync(fd, buffer[, options]) to append data to a file via multiple processes, the file handle is instantiated with correct flag for opening the file in append mode.
Question is, the above api returns number of bytes written to file which is good, what i need to understand is at what address(position) they were written to in the file, either start or end will suffice Eg: 10 bytes were written at start address of 123478899 or end address of (123478899+10)
Since it is been written by multiple processes and across different computers (file is actually stored on NAS) there are workarounds but all are complicated and needs synchronisation.
So is something like this is possible?
Suppose we have a file (less than 1MiB). This file can be modified (or deleted) by some program (git) from time to time. Also this file is being read by a Node application from time to time.
How to make sure that file contents that have been read by the Node application are in a consistent state?
Here's an example of what I'm afraid might happen.
Initial state of the file (A, B, C are just different parts of the file): ABC.
Node app reads file stats (mtimeMs).
Node app starts reading the file and reads A.
Other process starts writing to the file and writes to A making it A* and to B making it B* (file state: A*B*C).
Node app reads B* and C.
Node app reads file stats (mtimeMs).
Other process writes to C making it C* (file state: A*B*C*).
OS writes new file modification time.
As a result the Node application will get inconsistent file state AB*C while both mtimeMs read before and after operation are the same.
Note: temporarily blocking write operations to the file is not an option.
You could take a hash from the string you read and then after the file, compare those, if they are the same, you know the data is the same, if not you retry till it does.
I don't know if a hash from a string and the hash from the file would be the same tho... Then you could take a hash from the file before and after you read the data, if they are different, have your code retry until they aren't.
example: https://gist.github.com/GuillermoPena/9233069
You could start watching the file you are reading, taking appropriate action when it does.
https://nodejs.org/docs/latest/api/fs.html#fs_fs_watchfile_filename_options_listener
In linux when writing to a file, kernel maintains multiple in memory pages (4KB in size). Data is first written to the pages and background process bdflush sends these data to disk drive.
Is there a way to determine page boundaries when writing sequentially to a file ?
Can I assume it is always 1-4096:page 1 and 4097-8192:page 2 ?
or can it vary ?
say if I start writing from 10 (i.e. first 10 bytes already written to the file previously and I set file position to 10 before start writing) will the page boundary still be
1-4096 : page 1
OR
10-5096 : page 1 ?
Reason for asking,
I can use sync_file_range
http://man7.org/linux/man-pages/man2/sync_file_range.2.html
to flush data from kernel pages to disk drive in a orderly manner.
If I can determine page boundaries I can call sync_file_range only when a page boundary is reached, so that unnecessary sync_file_range calls are avoided.
Edit :
Only positive thing to suggest such boundary alignment I could find was in mmap page asking offset to be multiple of page size
,
offset must be a multiple of the page size as
returned by sysconf(_SC_PAGE_SIZE)
I want to read data from a tape and store that data on disk as a virtual tape. How to I maintain the original block structure of the tape? Some of the data I have requires the block structure stays the same. How to I establish what the block structure is on the source tape? I was thinking of writing the blocks to file with a header and footer structure and then using that to write back to tape/virtual tape maintaining the block structure. I can't work out how to establish the block structure of the data of the incoming data. I am doing this on Linux(Centos) in C. Language is not critical, will accept help in any language.
As far as I know, your analysis is correct. Tape will not maintain any structure of files which it has. You should use "file marks" to find out the location of a file in a tape.
Actually process of writing a set of files to a tape goes something like this : Write 1st file, Write 1st filemark, Write 2nd file, Write 2nd filemark & so on. While restoring, for example, you need 2nd file to be restore, just jump to the 1st filemark on the tape & start reading it using ReadFile, till you reach the next file mark.
Here are some APIs you can use to do these operations said above :
Writing a file to tape : BackupRead & WriteFile
Write a file mark : WriteTapeMark
Restoring a file from tape : ReadFile & BackupWrite
Jumping to a filemark : SetTapePosition
In case any doubts arise, please get back to me.
I want to make a multithread download using Idhttp (indy), so I have a principal thread that starts secondary threads, each secondary thread creates a file: "fileThreadNB" that is supposed to contain downloaded data, then this secondary thread downloads a part of the file on the server using idhttp.request.range and it writes downloaded data in fileThreadNB , then all these files (files created by secondary threads) are copied in one file to get the same file on the server, but the copy here takes a lot of time especially when the file on the server has a big size, so is there any other way that allows threads to write data in the same file, to be clearer; thread 0: downloads from position 0 to m, writes in fileX from position 0 to m .. thread n:downloads from position j to filesize-1, writes in fileX from position j to filesize-1.
Note: threads must write data in hard drive, so I can resume download later if something bad occurs.
I tried this code instead:
procedure TSecondaryThread.Execute;
begin
HTTP.Request.Range := Format('%d-%d',[BeginPos ,BeginPos +BlockSize -1]);
File.Position:=BeginPos;
HTTP.Get(url,File);
end;
BlockSize is the same for all threads, BeginPos changes from thread to other, the too variables are initialised in TSecondaryThread.Create.
NB:
when I try use one secondary thread, the file is well downloaded, but when I use more I get this error:External SIGSEGV, and the size of the downloaded file is bigger than the file on the server's size.
File is a global variable.
i guess that the problem is due to:File.Position:=BeginPos;but I don't know how to fix it, I would be grateful if someone helps me to solve this.
As you know the filesize you can create a empty fike with the allocation already configured to that size, then just take care to write for each thread to the right range. There should be no concurrence issues.