Problem
How can modern Fortran navigate a file with double forward slashes between entries, as in the *.obj format? The goal is to extract the vertex index (first entry) and ignore the vertex normal index (second entry).
Example
For example, for this snippet,
f 297//763 298//763 296//763
f 296//764 298//764 295//764
f 384//765 385//765 382//765
f 384//766 382//766 383//766
the goal is to create an array like this:
face ( 1 ) = [297, 298, 296]
face ( 2 ) = [296, 298, 295]
face ( 3 ) = [384, 385, 382]
face ( 4 ) = [384, 382, 383]
Extra points for an answer which would adopt to a richer format like
f a//b//c d//e//f g//h//i j//k//l
Other posts
The answer for [How to get Wavefront .obj file with 3 faces (traingle) points to a deleted blog. This post [How to read numeric data from a string in FORTRAN was not relevant.
References
Three references on the *.obj format:
Object Files (.obj), B1. Object Files (.obj), Wavefront .obj file
As suggested in the comments, we can replace / by a space using sed etc. We can also scan each character one by one in Fortran (see below). We then read in all integers into vals and select the desired part as vals( 1:6:2 ). A similar approach can be used for f a//b//c d//e//f ... etc by changing 1:6:2 to 1:12:3 etc.
[test.f90]
program main
implicit none
character(100) buf
integer vals(10), ios, i
open(10, file="test.dat", status="old")
do
read(10, "(a)", iostat=ios) buf
if (ios /= 0) exit
do i = 1, len(buf)
if (buf(i:i) == "/") buf(i:i) = " " !! replace "/" by " "
enddo
read(buf(2:), *) vals( 1:6 ) !! read all items
print *, vals( 1:6:2 ) !! select items 1, 3, 5
enddo
close(10)
end
[test.dat]
f 297//763 298//763 296//763
f 296//764 298//764 295//764
f 384//765 385//765 382//765
f 384//766 382//766 383//766
$ gfortran test.f90 && ./a.out
297 298 296
296 298 295
384 385 382
384 382 383
Just for fun, here is a similar code in Python, which is shorter thanks to replace() and split(). If we have some similar routines, I guess the above code may also become shorter.
dat = []
for line in open("test.dat").readlines():
dat.append( line[1:] .replace("/", " ") .split() [::2] )
import numpy as np
face = np.array(dat, dtype=int)
print(face)
$ python test.py
[[297 298 296]
[296 298 295]
[384 385 382]
[384 382 383]]
Related
I want to pretty print an AST using Haskell and (currently) wl-pprint-annotated (willing to switch to a different library).
How can I make the renderer prefer breaking the softlines of the outer group over the softlines of the inner group?
Minimal Example
Take for example the tuple ((1234, 5678), (abcd, efgh)).
The output I want:
// line width: 10
(
(
1234,
5678
),
(
abcd,
efgh
)
)
// line width: 16
(
(1234, 5678),
(abcd, efgh)
)
// line width: 32
((1234, 5678), (abcd, efgh))
The output I get:
// line width: 10
((1234,
5678),
(abcd,
efgh))
// line width: 16
((1234, 5678), (
abcd, efgh))
// line width: 32
((1234, 5678), (abcd, efgh))
Code:
module Main where
import qualified Prelude
import Prelude hiding((<>))
import Text.PrettyPrint.Annotated.WL
main :: IO ()
main = do
putStrLn $ pp 10
putStrLn $ pp 16
putStrLn $ pp 32
pp w = "// line width: " ++ show w ++ "\n" ++
display (renderPretty 1.0 w doc) ++ "\n"
doc = pair (pair (text "1234") (text "5678"))
(pair (text "abcd") (text "efgh"))
pair x y = group (nest 2 (lparen <//> x <> comma </> y) <//> rparen)
pair x y = group (nest 2 (lparen <##> x <> comma <#> y) <##> rparen)
As ekim found out, I've mixed up </> with <#>
I found the documentation to be confusing, so let me clear it up a little.
First of all the operators </> and <#> are just sugar for line and softline.
See definitions:
x </> y = x <> softline <> y
x <#> y = x <> line <> y
My problem was that I was using softline when what I wanted was line.
Commonalities between line and softline
Both are printed as space when the whole line fits the page. Both are replaced with a line break when the line does not fit the page.
Difference between line and softline
When a group foes not fit the page, all lines of the whole group are replaced with line breaks. That's the behavior I've wanted.
When the line does not fit the page, only the last softline still fitting the page is replaced.
Not the whole group.
It's like the word wrapping in our text editors: Just breaking after the last word that fits to the page.
For example
doc = paragraph p1
paragraph = foldr (</>) mempty . map text . words
p1 = "I want to pretty print an AST using Haskell and (currently) wl-pprint-annotated (willing to switch to a different library)."
is printed as
I want to pretty print an AST using Haskell and
(currently) wl-pprint-annotated (willing to
switch to a different library).
I have specific dataformat, say 'n' (arbitrary) row and '4' columns. If 'n' is '10', the example data would go like this.
1.01e+00 -2.01e-02 -3.01e-01 4.01e+02
1.02e+00 -2.02e-02 -3.02e-01 4.02e+02
1.03e+00 -2.03e-02 -3.03e-01 4.03e+02
1.04e+00 -2.04e-02 -3.04e-01 4.04e+02
1.05e+00 -2.05e-02 -3.05e-01 4.05e+02
1.06e+00 -2.06e-02 -3.06e-01 4.06e+02
1.07e+00 -2.07e-02 -3.07e-01 4.07e+02
1.08e+00 -2.08e-02 -3.08e-01 4.07e+02
1.09e+00 -2.09e-02 -3.09e-01 4.09e+02
1.10e+00 -2.10e-02 -3.10e-01 4.10e+02
Constraints in building this input would be
data should have '4' columns.
data separated by white spaces.
I want to implement a feature to check whether the input file has '4' columns in every row, and built my own based on the 'M.S.B's answer in the post Reading data file in Fortran with known number of lines but unknown number of entries in each line.
program readtest
use :: iso_fortran_env
implicit none
character(len=512) :: buffer
integer :: i, i_line, n, io, pos, pos_tmp, n_space
integer,parameter :: max_len = 512
character(len=max_len) :: filename
filename = 'data_wrong.dat'
open(42, file=trim(filename), status='old', action='read')
print *, '+++++++++++++++++++++++++++++++++++'
print *, '+ Count lines +'
print *, '+++++++++++++++++++++++++++++++++++'
n = 0
i_line = 0
do
pos = 1
pos_tmp = 1
i_line = i_line+1
read(42, '(a)', iostat=io) buffer
(*1)! Count blank spaces.
n_space = 0
do
pos = index(buffer(pos+1:), " ") + pos
if (pos /= 0) then
if (pos > pos_tmp+1) then
n_space = n_space+1
pos_tmp = pos
else
pos_tmp = pos
end if
endif
if (pos == max_len) then
exit
end if
end do
pos_tmp = pos
if (io /= 0) then
exit
end if
print *, '> line : ', i_line, ' n_space : ', n_space
n = n+1
end do
print *, ' >> number of line = ', n
end program
If I run the above program with a input file with some wrong rows like follows,
1.01e+00 -2.01e-02 -3.01e-01 4.01e+02
1.02e+00 -2.02e-02 -3.02e-01 4.02e+02
1.03e+00 -2.03e-02 -3.03e-01 4.03e+02
1.04e+00 -2.04e-02 -3.04e-01 4.04e+02
1.05e+00 -2.05e-02 -3.05e-01 4.05e+02
1.06e+00 -2.06e-02 -3.06e-01 4.06e+02
1.07e+00 -2.07e-02 -3.07e-01 4.07e+02
1.0 2.0 3.0
1.08e+00 -2.08e-02 -3.08e-01 4.07e+02 1.00
1.09e+00 -2.09e-02 -3.09e-01 4.09e+02
1.10e+00 -2.10e-02 -3.10e-01 4.10e+02
The output is like this,
+++++++++++++++++++++++++++++++++++
+ Count lines +
+++++++++++++++++++++++++++++++++++
> line : 1 n_space : 4
> line : 2 n_space : 4
> line : 3 n_space : 4
> line : 4 n_space : 4
> line : 5 n_space : 4
> line : 6 n_space : 4
> line : 7 n_space : 4
> line : 8 n_space : 3 (*2)
> line : 9 n_space : 5 (*3)
> line : 10 n_space : 4
> line : 11 n_space : 4
>> number of line = 11
And you can see that the wrong rows are properly detected as I intended (see (*2) and (*3)), and I can write 'if' statements to make some error messages.
But I think my code is 'extremely' ugly since I had to do something like (*1) in the code to count consecutive white spaces as one space. I think there would be much more elegant way to ensure the rows contain only '4' column each, say,
read(*,'4(X, A)') line
(which didn't work)
And also my program would fail if the length of 'buffer' exceeds 'max_len' which is set to '512' in this case. Indeed '512' should be enough for most practical purposes, I also want my checking subroutine to be robust in this way.
So, I want to improve my subroutine in at least these aspects
Want it to be more elegant (not as (*1))
Be more general (especially in regards to 'max_len')
Does anyone has some experience in building this kind of input-checking subroutine ??
Any comments would be highly appreciated.
Thank you for reading the question.
Without knowledge of the exact data format, I think it would be rather difficult to achieve what you want (or at least, I wouldn't know how to do it).
In the most general case, I think your space counting idea is the most robust and correct.
It can be adapted to avoid the maximum string length problem you describe.
In the following code, I go through the data as an unformatted, stream access file.
Basically you read every character and take note of new_lines and spaces.
As you did, you use spaces to count to columns (skipping double spaces) and new_line characters to count the rows.
However, here we are not reading the entire line as a string and going through it to find spaces; we read char by char, avoiding the fixed string length problem and we also end up with a single loop. Hope it helps.
EDIT: now handles white spaces at beginning at end of line and empty lines
program readtest
use :: iso_fortran_env
implicit none
character :: old_char, new_char
integer :: line, io, cols
logical :: beg_line
integer,parameter :: max_len = 512
character(len=max_len) :: filename
filename = 'data_wrong.txt'
! Output format to be used later
100 format (a, 3x, i0, a, 3x , i0)
open(42, file=trim(filename), status='old', action='read', &
form="unformatted", access="stream")
! set utils
old_char = " "
line = 0
beg_line = .true.
cols = 0
! Start scannig char by char
do
read(42, iostat = io) new_char
! Exit if EOF
if (io < 0) then
exit
end if
! Deal with empty lines
if (beg_line .and. new_char==new_line(new_char)) then
line = line + 1
write(*, 100, advance="no") "Line number:", line, &
"; Columns: Number", cols
write(*,'(6x, a5)') "EMPTYLINE"
! Deal with beginning of line for white spaces
elseif (beg_line) then
beg_line = .false.
! this indicates new columns
elseif (new_char==" " .and. old_char/=" ") then
cols = cols + 1
! End of line: time to print
elseif (new_char==new_line(new_char)) then
if (old_char/=" ") then
cols = cols+1
endif
line = line + 1
! Printing out results
write(*, 100, advance="no") "Line number:", line, &
"; Columns: Number", cols
if (cols == 4) then
write(*,'(6x, a5)') "OK"
else
write(*,'(6x, a5)') "ERROR"
end if
! Restart with a new line (reset counters)
cols = 0
beg_line = .true.
end if
old_char = new_char
end do
end program
This is the output of this program:
Line number: 1; Columns number: 4 OK
Line number: 2; Columns number: 4 OK
Line number: 3; Columns number: 4 OK
Line number: 4; Columns number: 4 OK
Line number: 5; Columns number: 4 OK
Line number: 6; Columns number: 4 OK
Line number: 7; Columns number: 4 OK
Line number: 8; Columns number: 3 ERROR
Line number: 9; Columns number: 5 ERROR
Line number: 10; Columns number: 4 OK
Line number: 11; Columns number: 4 OK
If you knew your data format, you could read your lines in a vector of dimension 4 and use iostat variable to print out an error on each line where iostat is an integer greater than 0.
Instead of counting whitespace you can use manipulation of substrings to get what you want. A simple example follows:
program foo
implicit none
character(len=512) str ! Assume str is sufficiently long buffer
integer fd, cnt, m, n
open(newunit=fd, file='test.dat', status='old')
do
cnt = 0
read(fd,'(A)',end=10) str
str = adjustl(str) ! Eliminate possible leading whitespace
do
n = index(str, ' ') ! Find first space
if (n /= 0) then
write(*, '(A)', advance='no') str(1:n)
str = adjustl(str(n+1:))
end if
if (len_trim(str) == 0) exit ! Trailing whitespace
cnt = cnt + 1
end do
if (cnt /= 3) then
write(*,'(A)') ' Error'
else
write(*,*)
end if
end do
10 close(fd)
end program foo
this should read any line of reasonable length (up to the line limit your compiler defaults to, which is generally 2GB now-adays). You could change it to stream I/O to have no limit but most Fortran compilers have trouble reading stream I/O from stdin, which this example reads from. So if the line looks anything like a list of numbers it should read them, tell you how many it read, and let you know if it had an error reading any value as a number (character strings, strings bigger than the size of a REAL value, ....). All the parts here are explained on the Fortran Wiki, but to keep it short this is a stripped down version that just puts the pieces together. The oddest behavior it would have is that if you entered something like this with a slash in it
10 20,,30,40e4 50 / this is a list of numbers
it would treat everything after the slash as a comment and not generate a non-zero status return while returning five values. For a more detailed explanation of the code I think the annotated pieces on the Wiki explain how it works. In the search, look for "getvals" and "readline".
So with this program you can read a line and if the return status is zero and the number of values read is four you should be good except for a few dusty corners where the lines would definitely not look like a list of numbers.
module M_getvals
private
public getvals, readline
implicit none
contains
subroutine getvals(line,values,icount,ierr)
character(len=*),intent(in) :: line
real :: values(:)
integer,intent(out) :: icount, ierr
character(len=:),allocatable :: buffer
character(len=len(line)) :: words(size(values))
integer :: ios, i
ierr=0
words=' '
buffer=trim(line)//"/"
read(buffer,*,iostat=ios) words
icount=0
do i=1,size(values)
if(words(i).eq.'') cycle
read(words(i),*,iostat=ios)values(icount+1)
if(ios.eq.0)then
icount=icount+1
else
ierr=ios
write(*,*)'*getvals* WARNING:['//trim(words(i))//'] is not a number'
endif
enddo
end subroutine getvals
subroutine readline(line,ier)
character(len=:),allocatable,intent(out) :: line
integer,intent(out) :: ier
integer,parameter :: buflen=1024
character(len=buflen) :: buffer
integer :: last, isize
line=''
ier=0
INFINITE: do
read(*,iostat=ier,fmt='(a)',advance='no',size=isize) buffer
if(isize.gt.0)line=line//buffer(:isize)
if(is_iostat_eor(ier))then
last=len(line)
if(last.ne.0)then
if(line(last:last).eq.'\\')then
line=line(:last-1)
cycle INFINITE
endif
endif
ier=0
exit INFINITE
elseif(ier.ne.0)then
exit INFINITE
endif
enddo INFINITE
line=trim(line)
end subroutine readline
end module M_getvals
program tryit
use M_getvals, only: getvals, readline
implicit none
character(len=:),allocatable :: line
real,allocatable :: values(:)
integer :: icount, ier, ierr
INFINITE: do
call readline(line,ier)
if(allocated(values))deallocate(values)
allocate(values(len(line)/2+1))
if(ier.ne.0)exit INFINITE
call getvals(line,values,icount,ierr)
write(*,'(*(g0,1x))')'VALUES=',values(:icount),'NUMBER OF VALUES=',icount,'STATUS=',ierr
enddo INFINITE
end program tryit
Honesty, it should work reasonably with just about any line you throw at it.
PS:
If you are always reading four values, using list-directed I/O and checking the iostat= value on READ and checking if you hit EOR would be very simple (just a few lines) but since you said you wanted to read lines of arbitrary length I am assuming four values on a line was just an example and you wanted something very generic.
I saved this data (20 vectors v)into csv file like this
v=:<"1 (? 20 2 $ 20)
makecsv v
v writecsv jpath'~temp/position.csv'
]vcsv =: freads jpath '~temp/position.csv'
fixcsv vcsv
, and I could import the csv file by
readcsv jpath '~temp/position.csv'
However, it doesn't give same result if I name it as
w=: readcsv jpath '~temp/position.csv'
diff=: ([{]) ,. ]
0 diff v
0 diff w
Actually, 0 diff w gives a length error
Is there any other approach should I use to have same results from both v(original) and w(imported csv data)?
Thank you!
I'm a J beginner so you may get a better answer later, but poking at it I think I have found something.
First, the tables/csv addon docs state that readcsv "Reads csv file into a boxed array," emphasis mine, while writecsv "Writes an array to a csv file." In other words, readcsv and writecsv are not symmetric operations. And the shapes of the values seem to confirm that:
$ w
1 20
$ v
20
This is also why the diff works for v but not w. If you simply unbox the result, it seems to work better:
0 diff 0 { w
┌───┬─────┐
│1 3│1 3 │
├───┼─────┤
...
├───┼─────┤
│1 3│5 8 │
└───┴─────┘
However, the shapes are still not exactly the same:
$ > v
20 2
$ > 0 { w
20 5
I think this is because readcsv doesn't know that your values are numeric; you probably need to throw a ". in there somewhere to decode them.
When you write the CSV file, you just have a bunch of ASCII characters. In this case, you've got numbers, spaces, and commas.
When you read the CSV, J has no guarantees about the format or contents. fixcsv gets your commas and line-breaks translated into a grid of cells, but J boxes it all to be safe, because it's a bunch of variable-length ASCII strings.
If you want to get back to v, you have two things you need to do. The first is to get the dimensions right. CSV files, pretty much by definition, are two-dimensional. If you change your example to write a two-dimensional array to the CSV, you'll find that you have the same shape after fixcsv readcsv.
u =: 4 5 $ v
u writecsv jpath'~temp/position.csv'
104
] t =: fixcsv freads jpath '~temp/position.csv'
┌────┬─────┬────┬────┬─────┐
│9 11│1 4 │8 3 │3 12│5 4 │
├────┼─────┼────┼────┼─────┤
│7 11│10 11│9 10│0 8 │6 16 │
├────┼─────┼────┼────┼─────┤
│13 8│17 12│13 2│5 19│17 14│
├────┼─────┼────┼────┼─────┤
│2 15│19 10│3 1 │12 7│14 13│
└────┴─────┴────┴────┴─────┘
$ v
20
$ u
4 5
$ t
4 5
If you're definitely dealing with a one-dimensional list (albeit of boxed number pairs), then you can Ravel (,) what you read to get it down to one dimension.
$ w
1 20
$ , w
20
Once you have them in the same shape, you need to convert the ASCII text into number arrays. Do that with Numbers (".).
10 * > {. v
90 110
10 * > {. , w
|domain error
| 10 *>{.,w
'a' , > {. , w
a9 11
10 * _ ". > {. , w
90 110
I have a huge csv file (as in: more than a few gigs) and would like to read it in Matlab and process each file. Reading the file in its entirety is impossible so I use this code to read in each line:
fileName = 'input.txt';
inputfile = fopen(fileName);
while 1
tline = fgetl(inputfile);
if ~ischar(tline)
break
end
end
fclose(inputfile);
This yiels a cell array of size(1,1) with the line as string. What I would like is to convert this cell to a normal array with just the numbers.
For example:
input.csv:
0.0,0.0,3.201,0.192
2.0,3.56,0.0,1.192
0.223,0.13,3.201,4.018
End result in Matlab for the first line:
A = [0.0,0.0,3.201,0.192]
I tried converting tline with double(tline) but this yields completely different results. Also tried using a regex but got stuck there. I got to the point where I split up all values into a different cell in one array. But converting to double with str2double yields only NaNs...
Any tips? Preferably without any loops since it already takes a while to read the entire file.
You are looking for str2num
>> A = '0.0,0.0,3.201,0.192';
>> str2num(A)
ans =
0 0 3.2010 0.1920
>> A = '0.0 0.0 3.201 0.192';
>> str2num(A)
ans =
0 0 3.2010 0.1920
>> A = '0.0 0.0 , 3.201 , 0.192';
>> str2num(A)
ans =
0 0 3.2010 0.1920
e.g., it's quite agnostic to input format.
However, I will not advise this for your use case. For your problem, I'd do
C = dlmread('input.txt',',', [1 1 1 inf]) % for first line
C = dlmread('input.txt',',') % for entire file
or
[a,b,c,d] = textread('input.txt','%f,%f,%f,%f',1) % for first line
[a,b,c,d] = textread('input.txt','%f,%f,%f,%f') % for entire file
if you want all columns in separate variables:
a = 0
b = 0
c = 3.201
d = 0.192
or
fid = fopen('input.txt','r');
C = textscan(fid, '%f %f %f %f', 1); % for first line only
C = textscan(fid, '%f %f %f %f', N); % for first N lines
C = textscan(fid, '%f %f %f %f', 1, 'headerlines', N-1); % for Nth line only
fclose(fid);
all of which are much more easily expandable (things like this, whatever they are, tend to grow bigger over time :). Especially dlmread is much less prone to errors than writing your own clauses is, for empty lines, missing values and other great nuisances very common in most data sets.
Try
data = dlmread('input.txt',',')
It will do exactly what you want to do.
If you still want to convert string to a vector:
line_data = sscanf(line,'%g,',inf)
This code will read the entire coma-separated string and convert each number.
Say you have 20 files and you don't won't to look at each one but instead have a script determine the format of the file.
ie bash findFileFormat direcName
Then loops through each file in a directory and print out the filename plus whether it has a delimiter (in which case is it a comma, pipe or otherwise) or fixed with for field separator and then what is the record separator. ie CR, LF, Ctrl+Z character.etc
I was thinking because some files may have a lot of pipes and commas in the data, that it could use a count of each character per line to determine what the delimiter is --> if this process does not produce consistent numbers of the character per line it is safe to assume that the file uses a fixed width field separator.
Is there a command or script that can be used to determine these 2 bits of info for each file?
Here's a small python script that will do as a starting point for what you need:
import sys
separators = [',', '|']
file_name = sys.argv[1]
def sep_cnt(line):
return {sep:line.count(sep) for sep in separators}
with open(file_name, 'r') as inf:
lines = inf.readlines()
cnts = [sep_cnt(line) for line in lines]
print(cnts)
def cnts_red(a, b):
c = {}
for k, v in a.iteritems():
if v > 0 and v == b[k]:
c[k] = v
return c
final = reduce(cnts_red, cnts[1:], cnts[0])
if len(final) == 0:
ftype = 'fixed'
else:
ftype = 'sep by ' + str(final.iteritems().next()[0])
print(ftype)
Name the above heur_sep.py and run this somewhere safe (e.g. /tmp):
# Prepare
rm *.txt
# Commas
cat >f1.txt <<e
a,a,a,a
b,b,b,b
c,c,c,c
e
# Pipes
cat >f2.txt <<e
a|a|a|a
b|b|b|b
c|c|c|c
e
# Fixed width
cat >f3.txt <<e
1 2 3
1 2 3
1 2 3
e
# Fixed width with commas
cat >f4.txt <<e
1, 2 3
1 2, 3
1 2, 3,
e
for i in *.txt; do
echo --- $i
python heur_sep.py $i
done
You would have to do some more work to make this resistant to different kinds of errors, but should be a good starting point. Hope this helps.