Related
I am writing the following simple routine:
program scratch
character*4 :: word
word = 'hell'
print *, concat(word)
end program scratch
function concat(x)
character*(*) x
concat = x // 'plus stuff'
end function concat
The program should be taking the string 'hell' and concatenating to it the string 'plus stuff'. I would like the function to be able to take in any length string (I am planning to use the word 'heaven' as well) and concatenate to it the string 'plus stuff'.
Currently, when I run this on Visual Studio 2012 I get the following error:
Error 1 error #6303: The assignment operation or the binary
expression operation is invalid for the data types of the two
operands. D:\aboufira\Desktop\TEMP\Visual
Studio\test\logicalfunction\scratch.f90 9
This error is for the following line:
concat = x // 'plus stuff'
It is not apparent to me why the two operands are not compatible. I have set them both to be strings. Why will they not concatenate?
High Performance Mark's comment tells you about why the compiler complains: implicit typing.
The result of the function concat is implicitly typed because you haven't declared its type otherwise. Although x // 'plus stuff' is the correct way to concatenate character variables, you're attempting to assign that new character object to a (implictly) real function result.
Which leads to the question: "just how do I declare the function result to be a character?". Answer: much as you would any other character variable:
character(len=length) concat
[note that I use character(len=...) rather than character*.... I'll come on to exactly why later, but I'll also point out that the form character*4 is obsolete according to current Fortran, and may eventually be deleted entirely.]
The tricky part is: what is the length it should be declared as?
When declaring the length of a character function result which we don't know ahead of time there are two1 approaches:
an automatic character object;
a deferred length character object.
In the case of this function, we know that the length of the result is 10 longer than the input. We can declare
character(len=LEN(x)+10) concat
To do this we cannot use the form character*(LEN(x)+10).
In a more general case, deferred length:
character(len=:), allocatable :: concat ! Deferred length, will be defined on allocation
where later
concat = x//'plus stuff' ! Using automatic allocation on intrinsic assignment
Using these forms adds the requirement that the function concat has an explicit interface in the main program. You'll find much about that in other questions and resources. Providing an explicit interface will also remove the problem that, in the main program, concat also implicitly has a real result.
To stress:
program
implicit none
character(len=[something]) concat
print *, concat('hell')
end program
will not work for concat having result of the "length unknown at compile time" forms. Ideally the function will be an internal one, or one accessed from a module.
1 There is a third: assumed length function result. Anyone who wants to know about this could read this separate question. Everyone else should pretend this doesn't exist. Just like the writers of the Fortran standard.
I receive the runtime error "bad integer for item 11 in list input" referencing the following line of code.
read(2,*)a,b,c,d,e,f,g,h,theta1,phi1,k,l,m,n,o,
$ p,theta2,phi2,s,theta3,phi3,
$ r1,x1,y1,r2,x2,y2,r3,x3,y3,z1,z2,z3
The line its reading from is
1 255.11211 0.2876 165.11404 90 4 8 0.19173 90 165.11404 0.09587 90 345.11404 4 4 0.0764 89.99915 -64.51149 0.11131 90.0015 24.23892 470.10565 -454.32263 120.7902 264.91144 114.00389 -239.12589 322.2894 293.87778 132.3114 0.01236 0.00697 0.00006 0.42619 -0.19278
so a mix of integers and reals, but I thought this was okay since I used the * format descriptor rather than a specified format.
I'm using the gcc compiler.
Item 11 in your list is 0.09587, a real. You are using list directed input, which for a real is interpreted a numeric value using an F edit descriptor (Fortran 2008 Cl. 10.10.3 paragraph 4). An input list item corresponding to an F edit descriptor must be real or complex (Fortran 2008 Cl. 10.7.2.3.1 paragraph 1).
To fix this you need to define k as a real variable. You note in a comment that you are using implicit typing (where variables starting with i-n are integer types and the rest are real by default). To explicitly declare k as a real, use:
real :: k
Best practices are to disable implicit typing (implicit none) and explicitly type all of your variables.
Preface: I needed to figure out the structure of a binary grid_data_file. From the Fortran routines I figured that the first record consists of 57 bytes and has information in the following order.
No. of the file :: integer*4
File name :: char*16
file status :: char*3 (i.e. new, old, tmp)
.... so forth (rest is clear from write statement in the program)
Now for the testing I wrote a simple program as follows: (I haven't included all the parameters)
Program testIO
implicit none
integer :: x, nclat, nclon
character :: y, z
real :: lat_gap, lon_gap, north_lat, west_lat
integer :: gridtype
open(11, file='filename', access='direct', form='unformatted', recl='200')
read(11, rec=1) x,y,z,lat_gap,lon_gap, north_lat,west_lat, nclat, nclon, gridtyp
write(*,*) x,y,z,lat_gap,lon_gap, north_lat,west_lat, nclat, nclon, gridtyp
close(11)
END
To my surprise, when I change the declaration part to
integer*4 :: x, nclat, nclon
character*16 :: y
character*3 :: z
real*4 :: lat_gap, lon_gap, north_lat, west_lat
integer*2 :: gridtype
It gives me SOME correct information, albeit not all! I can't understand this. It would help me to improve my Fortran knowledge if someone explains this phenomenon.
Moreover, I can't use ACCESS=stream due to machine being old and not supported, so I conclude that above is the only possibility to figure out the file structure.
From your replies and what others have commented, I think your problem might be a misunderstanding of what a Fortran "record" is:
You say that you have a binary file where each entry (you said record, but more on that later) is 57 bytes.
The problem is that a "record" in Fortran I/O is not what you would expect it is coming from a C (or anywhere else, really) background. See the following document from Intel, which gives a good explanation of the different access modes:
https://software.intel.com/sites/products/documentation/hpc/composerxe/en-us/2011Update/fortran/lin/bldaps_for/common/bldaps_rectypes.htm
In short, it has extra data (a header) describing the data in each entry.
Moreover, I can't use ACCESS=stream due to machine being old and not supported, so I conclude that above is the only possibility to figure out the file structure. Any guidance would be a big help!
If you can't use stream, AFAIK there is really no simple and painless way to read binary files with no record information.
A possible solution which requires a C compiler is to do IO in a C function that you call from Fortran, "minimal" example:
main.f90:
program main
integer, parameter :: dp = selected_real_kind(15)
character(len=*), parameter :: filename = 'test.bin'
real(dp) :: val
call read_bin(filename, val)
print*, 'Read: ', val
end program
read.c:
#include <string.h>
#include <stdio.h>
void read_bin_(const char *fname, double *ret, unsigned int len)
{
char buf[256];
printf("len = %d\n", len);
strncpy(buf, fname, len);
buf[len] = '\0'; // fortran strings are not 0-terminated
FILE* fh = fopen(buf, "rb");
fread(ret, sizeof(double), 1, fh);
fclose(fh);
}
Note that there is an extra parameter needed in the end and some string manipulation because of the way Fortran handles strings, which differs from C.
write.c:
#include <stdio.h>
int main() {
double d = 1.234;
FILE* fh = fopen("test.bin", "wb");
fwrite(&d, sizeof(double), 1, fh);
fclose(fh);
}
Compilation instructions:
gcc -o write write.c
gcc -c -g read.c
gfortran -g -o readbin main.f90 read.o
Create binary file with ./write, then see how the Fortran code can read it back with ./readbin.
This can be extended for different data types to basically emulate access=stream. In the end, if you can recompile the original Fortran code to output the data file differently, this will be the easiest solution, as this one is pretty much a crude hack.
Lastly, a tip for getting into unknown data formats: The tool od is your friend, check its manpage. It can directly convert binary represantations into a variety of different native datatypes. Try with the above example (the z adds the character representation in the right-hand column, not very useful here, in general yes):
od -t fDz test.bin
I wanted to know what the best way to write a large fortran array ( 5000 x 5000 real single precision numbers) to a file. I am trying to save the results of a numerical calculation for later use so they do not need to be repeated. From calculation 5000 x 5000 x 4bytes per number number is 100 Mb, is it possible to save this in a form that is only 100Mb? Is there a way to save fortran arrays as a binary file and read it back in for later use?
I've noticed that saving numbers to a text file produces a file much larger than the size of the data type being saved. Is this because the numbers are being saved as characters?
The only way I am familiar with to write to file is
open (unit=41, file='outfile.txt')
do i=1,len
do j=1,len
write(41,*) Array(i,j)
end do
end do
Although I'd imagine there is a better way to do it. If anyone could point me to some resources or examples to approve my ability to write and read larger files efficiently (in terms of memory) that would be great.
Thanks!
Write data files in binary, unless you're going to actually be reading the output - and you're not going to be reading a 2.5 million-element array.
The reasons for using binary are threefold, in decreasing importance:
Accuracy
Performance
Data size
Accuracy concerns may be the most obvious. When you are converting a (binary) floating point number to a string representation of the decimal number, you are inevitably going to truncate at some point. That's ok if you are sure that when you read the text value back into a floating point value, you are certainly going to get the same value; but that is actually a subtle question and requires choosing your format carefully. Using default formatting, various compilers perform this task with varying degrees of quality. This blog post, written from the point of view of a games programmer, does a good job of covering the issues.
Let's consider a little program which, for a variety of formats, writes a single-precision real number out to a string, and then reads it back in again, keeping track of the maximum error it encounters. We'll just go from 0 to 1, in units of machine epsilon. The code follows:
program testaccuracy
character(len=128) :: teststring
integer, parameter :: nformats=4
character(len=20), parameter :: formats(nformats) = &
[ '( E11.4)', '( E13.6)', '( E15.8)', '(E17.10)' ]
real, dimension(nformats) :: errors
real :: output, back
real, parameter :: delta=epsilon(output)
integer :: i
errors = 0
output = 0
do while (output < 1)
do i=1,nformats
write(teststring,FMT=formats(i)) output
read(teststring,*) back
if (abs(back-output) > errors(i)) errors(i) = abs(back-output)
enddo
output = output + delta
end do
print *, 'Maximum errors: '
print *, formats
print *, errors
print *, 'Trying with default format: '
errors = 0
output = 0
do while (output < 1)
write(teststring,*) output
read(teststring,*) back
if (abs(back-output) > errors(1)) errors(1) = abs(back-output)
output = output + delta
end do
print *, 'Error = ', errors(1)
end program testaccuracy
and when we run it, we get:
$ ./accuracy
Maximum errors:
( E11.4) ( E13.6) ( E15.8) (E17.10)
5.00082970E-05 5.06639481E-07 7.45058060E-09 0.0000000
Trying with default format:
Error = 7.45058060E-09
Note that even using a format with 8 digits after the decimal place - which we might think would be plenty, given that single precision reals are only accurate to 6-7 decimal places - we don't get exact copies back, off by approximately 1e-8. And this compiler's default format does not give us accurate round-trip floating point values; some error is introduced! If you're a video-game programmer, that level of accuracy may well be enough. If you're doing time-dependant simulations of turbulent fluids, however, that might absolutely not be ok, particularly if there's some bias to where the error is introduced, or if the error occurs in what is supposed to be a conserved quantity.
Note that if you try running this code, you'll notice that it takes a surprisingly long time to finish. That's because, maybe surprisingly, performance is another real issue with text output of floating point numbers. Consider the following simple program, which just writes out your example of a 5000 × 5000 real array as text and as unformatted binary:
program testarray
implicit none
integer, parameter :: asize=5000
real, dimension(asize,asize) :: array
integer :: i, j
integer :: time, u
forall (i=1:asize, j=1:asize) array(i,j)=i*asize+j
call tick(time)
open(newunit=u,file='test.txt')
do i=1,asize
write(u,*) (array(i,j), j=1,asize)
enddo
close(u)
print *, 'ASCII: time = ', tock(time)
call tick(time)
open(newunit=u,file='test.dat',form='unformatted')
write(u) array
close(u)
print *, 'Binary: time = ', tock(time)
contains
subroutine tick(t)
integer, intent(OUT) :: t
call system_clock(t)
end subroutine tick
! returns time in seconds from now to time described by t
real function tock(t)
integer, intent(in) :: t
integer :: now, clock_rate
call system_clock(now,clock_rate)
tock = real(now - t)/real(clock_rate)
end function tock
end program testarray
Here are the timing outputs, for writing to disk or to ramdisk:
Disk:
ASCII: time = 41.193001
Binary: time = 0.11700000
Ramdisk
ASCII: time = 40.789001
Binary: time = 5.70000000E-02
Note that when writing to disk, the binary output is 352 times as fast as ASCII, and to ramdisk it's closer to 700 times. There are two reasons for this - one is that you can write out data all at once, rather than having to loop; the other is that generating the string decimal representation of a floating point number is a surprisingly subtle operation which requires a significant amount of computing for each value.
Finally, is data size; the text file in the above example comes out (on my system) to about 4 times the size of the binary file.
Now, there are real problems with binary output. In particular, raw Fortran (or, for that matter, C) binary output is very brittle. If you change platforms, or your data size changes, your output may no longer be any good. Adding new variables to the output will break the file format unless you always add new data at the end of the file, and you have no way of knowing ahead of time what variables are in a binary blob you get from your collaborator (who might be you, three months ago). Most of the downsides of binary output are avoided by using libraries like NetCDF, which write self-describing binary files that are much more "future proof" than raw binary. Better still, since it's a standard, many tools read NetCDF files.
There are many NetCDF tutorials on the internet; ours is here. A simple example using NetCDF gives similar times to raw binary:
$ ./array
ASCII: time = 40.676998
Binary: time = 4.30000015E-02
NetCDF: time = 0.16000000
but gives you a nice self-describing file:
$ ncdump -h test.nc
netcdf test {
dimensions:
X = 5000 ;
Y = 5000 ;
variables:
float Array(Y, X) ;
Array:units = "ergs" ;
}
and file sizes about the same as raw binary:
$ du -sh test.*
96M test.dat
96M test.nc
382M test.txt
the code follows:
program testarray
implicit none
integer, parameter :: asize=5000
real, dimension(asize,asize) :: array
integer :: i, j
integer :: time, u
forall (i=1:asize, j=1:asize) array(i,j)=i*asize+j
call tick(time)
open(newunit=u,file='test.txt')
do i=1,asize
write(u,*) (array(i,j), j=1,asize)
enddo
close(u)
print *, 'ASCII: time = ', tock(time)
call tick(time)
open(newunit=u,file='test.dat',form='unformatted')
write(u) array
close(u)
print *, 'Binary: time = ', tock(time)
call tick(time)
call writenetcdffile(array)
print *, 'NetCDF: time = ', tock(time)
contains
subroutine tick(t)
integer, intent(OUT) :: t
call system_clock(t)
end subroutine tick
! returns time in seconds from now to time described by t
real function tock(t)
integer, intent(in) :: t
integer :: now, clock_rate
call system_clock(now,clock_rate)
tock = real(now - t)/real(clock_rate)
end function tock
subroutine writenetcdffile(array)
use netcdf
implicit none
real, intent(IN), dimension(:,:) :: array
integer :: file_id, xdim_id, ydim_id
integer :: array_id
integer, dimension(2) :: arrdims
character(len=*), parameter :: arrunit = 'ergs'
integer :: i, j
integer :: ierr
i = size(array,1)
j = size(array,2)
! create the file
ierr = nf90_create(path='test.nc', cmode=NF90_CLOBBER, ncid=file_id)
! define the dimensions
ierr = nf90_def_dim(file_id, 'X', i, xdim_id)
ierr = nf90_def_dim(file_id, 'Y', j, ydim_id)
! now that the dimensions are defined, we can define variables on them,...
arrdims = (/ xdim_id, ydim_id /)
ierr = nf90_def_var(file_id, 'Array', NF90_REAL, arrdims, array_id)
! ...and assign units to them as an attribute
ierr = nf90_put_att(file_id, array_id, "units", arrunit)
! done defining
ierr = nf90_enddef(file_id)
! Write out the values
ierr = nf90_put_var(file_id, array_id, array)
! close; done
ierr = nf90_close(file_id)
return
end subroutine writenetcdffile
end program testarray
OPEN the file for reading and writing as "unformatted", and READ and WRITE the data without supplying a format, as demonstrated in the program below.
program xunformatted
integer, parameter :: n = 5000, inu = 20, outu = 21
real :: x(n,n)
integer :: i
character (len=*), parameter :: out_file = "temp_num"
call random_seed()
call random_number(x)
open (unit=outu,form="unformatted",file=out_file,action="write")
do i=1,n
write (outu) x(i,:) ! write one row at a time
end do
print*,"sum(x) =",sum(x)
close (outu)
open (unit=inu,form="unformatted",file=out_file,action="read")
x = 0.0
do i=1,n
read (inu) x(i,:) ! read one row at a time
end do
print*,"sum(x) =",sum(x)
end program xunformatted
I just started my path on Erlang and I'm facing a problem I can't sort out a solution about:
I wrote a metod to take a domain espressed as a binary string, i.e. <<"www.404pagenotfound.com">> and convert it in the domain format as required for DNS protocol, so: <<3,"www",15,"pagenotfound",3,"com">>.
In the following the code (I rewroted it many times in different ways):
domainbyte(Bin) ->
if byte_size(Bin) > 0 ->
Res = binary:split(Bin, <<".">>),
[Chunk|[RestList]] = Res,
ChunkSize = byte_size(Chunk),
if length(RestList) > 0 ->
Rest = domainbyte(RestList), %% <- Got "bad argument" here!
<<ChunkSize/binary,Chunk,Rest>>;
true ->
<<ChunkSize/binary,Chunk>>
end
end.
Thx in advance for any clues.
PS.
Thx to comments I've found the error in the code abobe:
if length(RestList) > 0 -> %% here RestList is binary data so length throw "bad argument" error.
I've rewroted the method this way, but still with no luck:
**NOTE: I was able to fix the code in the following, problem is that if you have a binary chunk and you want to use it in another binary string, you must specify /binary on it: something not obvious to me.
I.e.: consider this small code snip:
**
TT = <<"com">>,
SS = <<3, TT, 0>> %% <- you get error: bad argument
**
must be fixed this way:
**
TT = <<"com">>,
SS = <<3, TT/binary, 0>>
domainbyte(Bin) ->
if byte_size(Bin) > 0 ->
Res = binary:split(Bin, <<".">>),
if length(Res) > 1 ->
[Chunk|[RestList]] = Res,
ChunkSize = byte_size(Chunk),
Rest = domainbyte(RestList),
<<ChunkSize,Chunk,Rest>>;
true ->
[Chunk] = Res,
ChunkSize = byte_size(Chunk),
<<ChunkSize,Chunk>>
end
end.
MdP
I think the easiest solution is to define the function using a binary comprehension:
domainbyte(Bin) ->
Chunks = binary:split(Bin, <<".">>, [global]), %A list of all the chunks
<< <<byte_size(C),C/binary>> || C <- Chunks >>. %Build output binary
It might be slightly faster to build the output binary as a list of segments in a separate function then put them together in an iolist_to_binary/1. Note that if a '.' occurs outermost in the binary then this code will take that as an empty segment of length 0. If these should be discarded then you nee to add the option trim to binary:split/3. Note also that the size will occupy only one byte.
#Alnitak has the separate function but builds the binary one segment at time so it is not more efficient than the binary comprehension which does the same thing.
N.B. that if you have a binary segment Chunk/binary when constructing a binary this means that Chunk IS a binary, not that it should become one. Binaries are flat structures, think byte arrays, so everything becomes a binary. Or rather the binary.
EDIT: Though I see I missed the 0 which should be at the end. That is left as an exercise to the reader.
EDIT: Being in teaching mode, apart from the constructing binaries, a key to writing good Erlang code is understanding pattern matching. You use a bit but could do it more:
domainbyte(Bin) ->
case binary:split(Bin, <<".">>) of
[Chunk,Rest] -> %There was a '.'
RestBin = domainbyte(Rest),
Size = byte_size(Chunk),
<<Size,Chunk/binary,RestBin/binary>>;
[Chunk] -> %This was the last chunk
Size = byte_size(Chunk),
<<Size,Chunk/binary,0>> %Add terminating 0
end.
This is basically doing the same as your code but we are using pattern matching to select a clause, not only to pull apart a known structure. Pattern matching is the basic method for control, not just in case as here but also in functions and receive. This results in if being used quite sparingly.
Enough from me for now.
I'm a bit rusty on Erlang, but I think your problem is that RestList is an array of chunks from the output of binary:split.
So when you chuck it recursively back into domainbyte it's in the wrong format.
Also - don't forget the terminating NUL byte to represent the root label!
FWIW, here's my working version:
label([]) ->
<< 0 >>;
label([H|T]) ->
D = label(H),
P = label(T),
<< D/binary, P/binary>>;
label(A) ->
L = byte_size(A),
<< <<L>>/binary, A/binary>>.
domainbyte(A) ->
Res = binary:split(A, <<".">>, [global, trim]),
label(Res).
It correctly adds a trailing NUL byte, and trims any extra trailing dots.
The concat of binaries should be written like this:
<< <<ChunkSize>>/binary, Chunk/binary, Rest/binary>>
% and
<< <<ChunkSize>>/binary, Chunk/binary>>