Bin data read differently in AIX vs Linux - linux

Question

I have a .bin file that contains slopes and intercepts. I'm using Fortran to read the values and I'm getting different values on machines running AIX and Linux. I believe the Linux data to be accurate. Does this have something to do with stack size or endians?
For example, AIX max value is: 0.3401589687E+39 while Linux max value is: 6.031288
program read_bin_files
REAL :: slope(2500,1250)
INTEGER :: recl=2500*1250*4
OPEN(UNIT=8, FILE='MODIS_AVHRR_years_slope.bin', ACTION='READ', ACCESS='direct', FORM='unformatted', RECL=recl, IOSTAT=iostat)
READ(unit=8, REC = 1, IOSTAT = iostat) slope
print *, "Max slope value is:", maxval(slope)
CLOSE(8)
end

Answer 1

AIX runs (these days) on POWER CPUs, which are usually big-endian, whereas Linux is usually run on x86es, which are little-endian. So you are correct to suspect that endianness may be a problem. You report that the result of running this program
program read_bin_files
INTEGER*4 :: slope(2500,1250)
INTEGER :: recl=2500*1250*4
OPEN(UNIT=8, FILE='MODIS_AVHRR_years_slope.bin', ACTION='READ', &
ACCESS='direct', FORM='unformatted', RECL=recl)
READ(unit=8, REC = 1) slope
DO i = 1, 10
WRITE(*, '(Z8.8)') slope(1, i)
END DO
CLOSE(8)
end
is the following. ("AIX" and "Linux" are in quotes in the column headers because it's the CPU that matters here, not the operating system.)
"Linux" | "AIX"
------------+------------
3E C2 61 8F | 8F 61 C2 3E
3E F5 64 52 | 52 64 F5 3E
BC F3 E0 7E | 7E E0 F3 BC
BF B9 71 0D | 0D 71 B9 BF
3E F5 B9 73 | 73 B9 F5 3E
3F 29 3C 2F | 2F 3C 29 3F
3E DC C2 09 | 09 C2 DC 3E
3F 66 86 89 | 89 86 66 3F
3E 5B 91 A9 | A9 91 5B 3E
3F 67 73 25 | 25 73 67 3F
In each row, the right-hand half is the mirror image of the left-hand half. That demonstrates that the issue is endianness. What we still don't know is which byte order is correct. The answer to that question will almost certainly be "the byte order used by the CPU that ran the program that generated the file."
If you are using GNU Fortran, the CONVERT specifier to OPEN should solve the problem, provided you can figure out which way around the data is supposed to be interpreted. However, I think that's an extension. In the general case, I don't know enough FORTRAN to tell you what to do.
If you have control over the process generating these data files, you can avoid the entire problem in the future by switching both sides to a self-describing data format, such as HDF.

Answer 2

Your AIX machine is likely big-endian RISC and your Linux is likely PC or other Intel platform. Just convert the endianness.
I use these procedures for 4 byte and 8 byte variables (use iso_fortran_env in the module):
elemental function SwapB32(x) result(res)
real(real32) :: res
real(real32),intent(in) :: x
character(4) :: bytes
integer(int32) :: t
real(real32) :: rbytes, rt
equivalence (rbytes, bytes)
equivalence (t, rt)
rbytes = x
t = ichar(bytes(4:4),int32)
t = ior( ishftc(ichar(bytes(3:3),int32),8), t )
t = ior( ishftc(ichar(bytes(2:2),int32),16), t )
t = ior( ishftc(ichar(bytes(1:1),int32),24), t )
res = rt
end function
elemental function SwapB64(x) result(res)
real(real64) :: res
real(real64),intent(in) :: x
character(8) :: bytes
integer(int64) :: t
real(real64) :: rbytes, rt
equivalence (rbytes, bytes)
equivalence (t, rt)
rbytes = x
t = ichar(bytes(8:8),int64)
t = ior( ishftc(ichar(bytes(7:7),int64),8), t )
t = ior( ishftc(ichar(bytes(6:6),int64),16), t )
t = ior( ishftc(ichar(bytes(5:5),int64),24), t )
t = ior( ishftc(ichar(bytes(4:4),int64),32), t )
t = ior( ishftc(ichar(bytes(3:3),int64),40), t )
t = ior( ishftc(ichar(bytes(2:2),int64),48), t )
t = ior( ishftc(ichar(bytes(1:1),int64),56), t )
res = rt
end function
usage:
SLOPE = SwapB32(SLOPE)
There are other ways. Some compilers support non-standard OPEN(...,CONVERT='big_endian',... and some have command line options like -fconvert=big-endian.

Answer 3

That elemental function SwapB64 is elegant, and good to have for these issues.
Or
Try these using big_endian, little endian etc.
(Personally I would do both)
!OPEN(UNIT=8, FILE='MODIS_AVHRR_years_slope.bin', ACTION='READ', ACCESS='direct', FORM='unformatted', RECL=recl, IOSTAT=iostat)
OPEN(UNIT=8, FILE='MODIS_AVHRR_years_slope.bin', CONVERT='big_endian', ACTION='READ', ACCESS='direct', FORM='unformatted', RECL=recl, IOSTAT=iostat)