When running a parallel program, we might need to write result to files for post-processing. These files are often with specific format, for example: title (characters/strings) + binary array (real/float). This can be done with 2 steps: 1. use master processor to create a new file and write title. 2. use MPI IO to append the array data to this file. I tried to use MPI_MODE_APPEND as amod in MPI_FILE_OPEN to append data to the existing file, but this step fails. The FORTRAN code is (I tried MPI_MODE_APPEND only and MPI_MODE_APPEND+MPI_MODE_WRONLY)
call MPI_File_open(comm, filename, MPI_MODE_APPEND+MPI_MODE_WRONLY, &
MPI_INFO_NULL, fh, ierr)
Also, the new result overwrites the old data. How should I correct this statement? Many thanks.
Thanks #Vladimir for this reminder. I think I make it. The original code snippet was
integer (kind=MPI_OFFSET_KIND) :: disp = 0
integer :: n1, n2, n3
double precision, dimension(0:n1+1,0:n2+1,0:n3+1) :: iodata
integer, dimension(ndim) :: arraysize, arraystart
integer, dimension(ndim) :: arraygsize, arraysubsize
...
call MPI_Type_create_subarray(ndim, arraygsize, arraysubsize, arraystart, &
MPI_ORDER_FORTRAN, MPI_DOUBLE_PRECISION, &
filetype, ierr)
call MPI_Type_commit(filetype, ierr)
arraystart(:) = 1
call MPI_Type_create_subarray(ndim, arraysize, arraysubsize, arraystart, &
MPI_ORDER_FORTRAN, MPI_DOUBLE_PRECISION, &
mpi_subarray, ierr)
call MPI_Type_commit(mpi_subarray, ierr)
call MPI_File_open(cartcomm, filename, MPI_MODE_APPEND+MPI_MODE_WRONLY, &
MPI_INFO_NULL, fh, ierr)
call MPI_File_set_view(fh, disp, MPI_DOUBLE_PRECISION, filetype, 'native', &
MPI_INFO_NULL, ierr)
call MPI_File_write_all(fh, iodata, 1, mpi_subarray, status, ierr)
Indeed, I used the correct mode: MPI_MODE_APPEND+MPI_MODE_WRONLY. However, I did not set the offset/displacement and append the parallel data with 0 displacement, which means overwrite the file.
I should add another statement after MPI_FILE_OPEN which is
call MPI_FILE_GET_POSITION(fh, disp, ierr)
It gets the current position with argument disp. disp gives the correct position for appended data to the existing file.
Related
I'm implementing a file in /proc which I'd like to be a little more file-like than usual. In particular, I'd like to detect that someone is appending to the file and handle that correctly -- that is, I'd like to distinguish between someone doing
echo value > /proc/my_proc_file
and
echo value >> /proc/my_proc_file
Like all write functions, mine is handed an offset as its fourth argument:
ssize_t my_proc_write(struct file *file, const char __user *buf,
size_t count, loff_t *offs)
But *offs is always 0.
When I set up my proc file, I'm specifying seq_lseek as the lseek function:
struct file_operations my_proc_fops = {
.open = my_proc_open,
.read = seq_read,
.write = my_proc_write,
.llseek = seq_lseek,
};
Inspecting the source (in fs/seq_file.c), it looks like seq_lseek maintains file->f_pos appropriately, but when I look at file->f_pos in my write function, it's always 0, too. (This may not be surprising, since appending usually means opening with O_APPEND which doesn't result in any explicit calls to lseek.)
Anyway, is there a way to do this? Presumably these write functions wouldn't have been set up with offset pointer arguments if they weren't going to pass in useful, nonzero values from time to time...
first, from user perspective, file opened with O_APPEND will ALWAYS append data when you call write(), no matter where the f_pos is set by llseek(). but f_pos is still effective for read().
second, kernel framework dosn't know the file length unless it calls your llseek to find out, but that's not gonna happen because it will mess up f_pos,so kernel expect your driver, which is the only one who knows where is the true "end of the file", to act accordingly when (file->f_flags & O_APPEND) is true. basically, your driver needs to check that flag when write() is called and ignore the offs param and do the append.
I am new at MPI and parallel computing.I wrote simple MPI program in Fortran. But the problem is the MPI_FILE_READ_AT ignores and does not read the file.
My code is:
program O72
use mpi
!implicit none
INTEGER :: NINTS, DATATYPE, STATUS(MPI_STATUS_SIZE), mpierr, FH, i
integer :: taskid, no_tasks, FILESIZE, FILENO_2
INTEGER(KIND=MPI_OFFSET_KIND) :: OFFSET
INTEGER :: BUF(10)
character :: filename='ab.dat'
FILESIZE=10
BUF(:)=0
call MPI_INIT(mpierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD,no_tasks,mpierr)
call MPI_COMM_RANK(MPI_COMM_WORLD,taskid,mpierr)
NINTS = FILESIZE/no_tasks !*INTSIZE)
OFFSET= taskid*NINTS !*INTSIZE
call MPI_File_open(MPI_COMM_WORLD, "ab.dat", MPI_MODE_RDONLY, &
MPI_INFO_NULL, FH, mpierr)
call MPI_FILE_READ_AT(FH, OFFSET, BUF, NINTS, MPI_INT, STATUS, mpierr)
write(6,*) taskid, no_tasks
call MPI_BARRIER(MPI_COMM_WORLD, mpierr)
write(6,*) 'taskid=',taskid,'NINTS=', NINTS
write(6,*) 'taskid_',taskid,'OFFSET=',OFFSET
write(6,*) BUF(:)
!call MPI_FILE_WRITE(FILENO_2, OFFSET, BUF, NINTS, DATATYPE, STATUS, mpierr)
call MPI_FILE_CLOSE(FH, mpierr)
!call MPI_FILE_CLOSE(FILENO_2)
call MPI_FINALIZE(mpierr)
end
Then, the input file is an ASCII file with 2 columns in which 1st column is always 1 and 2nd column is always 2.
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
1 2
And the output comes like this with 2 processor used:
0 2
1 2
taskid= 1 NINTS= 5
taskid_ 1 OFFSET= 5
taskid= 0 NINTS= 5
taskid_ 0 OFFSET= 0
171051313 171051313 171051313 171051313 171051313 0 0 0 0 0
822751753 822751753 822751753 822751753 822751753 0 0 0 0 0
I don't understand where the problem is and why random values are assigned to BUF(:)
Could you please help me?
I guess the simple answer is that MPI-IO only performs binary IOs, while what you try to achieve here is formatted IOs.
So starting from here, you see that you have a few problems in your code:
You file is formatted. Since as I said in preamble, MPI-IO only performs binary IO, you'll have to, either switch to a binary input file, or to read the file as a set of strings which you'll have to make sense of internally. Let's assume for the rest of this post that your input file is now binary.
The offset to use in MPI_File_read_at() is expected in bytes. What you pass is a number of elements. You therefore need to multiply this offset by the size of what you want to read, aka the size of an integer.
Finally, you print the full of buf which is an array of 10 integer whereas you only read nints of them. You probably should only print buf(1:nints).
With that, your code should work.
I have text file with the name By_a_waterfal.txt and the following content:
By a waterfal I'm calling you.
We can share it all beneath a ceiling of blue.
We'll spend a heavenly day
Here where the whispering waters play.
I create memory mapping of this file by calls of function:
HANDLE hFileMapping = CreateFileMapping(handleOfFile, NULL, PAGE_READ, 0, 0, NULL);
Where handleOfFile is the handle of By_a_waterfal.txt file.
Then I do the following call:
LPVOID lpFileMap = MapViewOfFile(hFileMapping, FILE_MAP_READ, 0, 0, 0);
How can I get access to content of the memory mapped By_a_waterfal.txt file via the view created with MapViewOfFile call?
If you really have to use memory mapping instead if std::ifstream, then:
1) Check lpFileMap for NULL.
2) Get size of view as GetFileSize(handleOfFile) or VirtualQuery(lpFileMap).
3) If file has ANSI encoding (not Unicode) cast lpFileMap to char* and proceed (do not forget that there is no 0-terminator)
auto data = static_cast<const char*>(lpFileMap);
std::string strData(data, size);
std::cout << stdData;
lpFileMap is a pointer to the content. If you want the content as a const char* for instance, just cast it:
const char* content = static_cast<const char*>(lpFileMap);
We are currently using file I/O but need a better/faster way. Sample code would be appreciated.
By using files for transfer, you're already implementing a form of message passing, and so I think that would be the most natural fit for this sort of program. Now, you could write something yourself that uses shared memory when available and something like TCP/IP when not - or you could just use a library that already does that, like MPI, which is widely available, works, will take advantage of shared memory if you are running on the same machine, but would then also extend to letting you run them on different machines entirely without you changing your code.
So as a simple example of one program sending data to a second and then waiting for data back, we'd have two programs as follows; first.f90
program first
use protocol
use mpi
implicit none
real, dimension(n,m) :: inputdata
real, dimension(n,m) :: processeddata
integer :: rank, comsize, ierr, otherrank
integer :: rstatus(MPI_STATUS_SIZE)
call MPI_INIT(ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, comsize, ierr)
if (comsize /= 2) then
print *,'Error: this assumes n=2!'
call MPI_ABORT(1,MPI_COMM_WORLD,ierr)
endif
!! 2 PEs; the other is 1 if we're 0, or 0 if we're 1.
otherrank = comsize - (rank+1)
inputdata = 1.
inputdata = exp(sin(inputdata))
print *, rank, ': first: finished computing; now sending to second.'
call MPI_SEND(inputdata, n*m, MPI_REAL, otherrank, firsttag, &
MPI_COMM_WORLD, ierr)
print *, rank, ': first: Now waiting for return data...'
call MPI_RECV(processeddata, n*m, MPI_REAL, otherrank, backtag, &
MPI_COMM_WORLD, rstatus, ierr)
print *, rank, ': first: recieved data from partner.'
call MPI_FINALIZE(ierr)
end program first
and second.f90:
program second
use protocol
use mpi
implicit none
real, dimension(n,m) :: inputdata
real, dimension(n,m) :: processeddata
integer :: rank, comsize, ierr, otherrank
integer :: rstatus(MPI_STATUS_SIZE)
call MPI_INIT(ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, comsize, ierr)
if (comsize /= 2) then
print *,'Error: this assumes n=2!'
call MPI_ABORT(1,MPI_COMM_WORLD,ierr)
endif
!! 2 PEs; the other is 1 if we're 0, or 0 if we're 1.
otherrank = comsize - (rank+1)
print *, rank, ': second: Waiting for initial data...'
call MPI_RECV(inputdata, n*m, MPI_REAL, otherrank, firsttag, &
MPI_COMM_WORLD, rstatus, ierr)
print *, rank, ': second: adding 1 and sending back.'
processeddata = inputdata + 1
call MPI_SEND(processeddata, n*m, MPI_REAL, otherrank, backtag, &
MPI_COMM_WORLD, ierr)
print *, rank, ': second: completed'
call MPI_FINALIZE(ierr)
end program second
For clarity, stuff that the two programs must agree on could be ina module they both use, here protocol.f90:
module protocol
!! shared information like tag ids, etc goes here
integer, parameter :: firsttag = 1
integer, parameter :: backtag = 2
!! size of problem
integer, parameter :: n = 10, m = 20
end module protocol
(A makefile for building the executables follows:)
all: first second
FFLAGS=-g -Wall
F90=mpif90
%.mod: %.f90
$(F90) -c $(FFLAGS) $^
%.o: %.f90
$(F90) -c $(FFLAGS) $^
first: protocol.mod first.o
$(F90) -o $# first.o protocol.o
second: protocol.mod second.o
$(F90) -o $# second.o protocol.o
clean:
rm -rf *.o *.mod
and then you run the two programs as following:
$ mpiexec -n 1 ./first : -n 1 ./second
1 : second: Waiting for initial data...
0 : first: finished computing; now sending to second.
0 : first: Now waiting for return data...
1 : second: adding 1 and sending back.
1 : second: completed
0 : first: recieved data from partner.
$
We could certainly give you a more relevant example if you give us more information about the workflow between the two programs.
Are you using binary (unformatted) file I/O? Unless the data quantity is huge, that should be fast.
Otherwise you could use interprocess communication, but it would be more complicated. You might find code in C, which you could call from Fortran using the ISO C Binding.
How to read and write to binary files in D language? In C would be:
FILE *fp = fopen("/home/peu/Desktop/bla.bin", "wb");
char x[4] = "RIFF";
fwrite(x, sizeof(char), 4, fp);
I found rawWrite at D docs, but I don't know the usage, nor if does what I think. fread is from C:
T[] rawRead(T)(T[] buffer);
If the file is not opened, throws an exception. Otherwise, calls fread for the file handle and throws on error.
rawRead always read in binary mode on Windows.
rawRead and rawWrite should behave exactly like fread, fwrite, only they are templates to take care of argument sizes and lengths.
e.g.
auto stream = File("filename","r+");
auto outstring = "abcd";
stream.rawWrite(outstring);
stream.rewind();
auto inbytes = new char[4];
stream.rawRead(inbytes);
assert(inbytes[3] == outstring[3]);
rawRead is implemented in terms of fread as
T[] rawRead(T)(T[] buffer)
{
enforce(buffer.length, "rawRead must take a non-empty buffer");
immutable result =
.fread(buffer.ptr, T.sizeof, buffer.length, p.handle);
errnoEnforce(!error);
return result ? buffer[0 .. result] : null;
}
If you just want to read in a big buffer of values (say, ints), you can simply do:
int[] ints = cast(int[]) std.file.read("ints.bin", numInts * int.sizeof);
and
std.file.write("ints.bin", ints);
Of course, if you have more structured data then Scott Wales' answer is more appropriate.