script not reading last line of a file - linux

Question

i have a file created in windows using notepad:
26453215432460
23543265235421
38654365876325
12354152435243
I have a script which will read every line, and create a command like below in other file for every line and will not consider blank lines:
CRE:EQU,264532154324600,432460,1;
Now if I save my input file after hitting enter after the last line of number 12354152435243, then the output file consists the command above corresponding to all numbers(including the last 12354152435243:
CRE:EQU,264532154324600,432460,1;
CRE:EQU,235432652354210,235421,1;
CRE:EQU,386543658763250,876325,1;
CRE:EQU,123541524352430,435243,1;
but if I save the file, without hitting enter after the last number is keyed in i.e after this 12354152435243, then after the script executes, I don't see the output file have the command for the last number:
CRE:EQU,264532154324600,432460,1;
CRE:EQU,235432652354210,235421,1;
CRE:EQU,386543658763250,876325,1;
Can somebody explain the error in the code:
while read LINE
do
[ -z "$LINE" ] && continue
IMEI=`echo $LINE | sed 's/ //g' | sed -e 's/[^ -~]//g'`
END_SERIAL=`echo $IMEI | cut -c9- | sed 's/ //g' | sed -e 's/[^ -~]//g'`
echo "CRE:EQU,${IMEI}0,${END_SERIAL},${list},,${TODAY};" >> /apps/ins/list.out
done < "${FILE_NAME}"
kindly help

Answer 1

Use
grep . "${FILE_NAME}" | while read LINE
or
while read LINE
do
....
done < <(grep . "${FILE_NAME}")
The grep is less sensible to line-ending, and you will get empty-line skip for a free... :)
Honestly, never tried windows, all above is OK for unix...
EDIT Explanation:
make the next file:
echo -n -e 'line\n\nanother\nno line ending here>' >file.txt
the file contains 4 lines (although the last "line" is not a "correct" one)
line
another
no line ending here>
Usual shell routines, as read or wc looking for line ending. Therefore,
$ wc -l file.txt
3 file.txt
When you grepping for '' (empty string) the grep returns every line where found the string, so
$ grep '' file.txt
prints
line
another
no line ending here>
When grep prints out the found lines - ensures than one `\n' exists at the end, so
$ grep '' file.txt | wc -l
returns
4
therefore, for these situations, is better to use grep with -c (count) and not wc.
$ grep -c '' file.txt
4
Now, the . dot. The dot mean any character. So, when you grepping for a ., you get all lines what contain at least one character. And therefore, it will skip all lines what doesn't contain any character = skips empty lines. So,
$ grep . file.txt
line
another
no line ending here>
again, with added line ending to the last line (and skipped the empty line). Remember, the (space) is character too, so when the line contains only one space it is NOT EMPTY. Counting non-empty lines
$ grep . file.txt | wc -l
3
or faster
$ grep -c . file.txt
3

Answer 2

If you do a help read it says for -d delim continue until the first character of DELIM is read, rather than newline.
So read will continue until it hits a \n or if you specify -d delim.
So you probably need to change the delim or you can try read -e

Answer 3

read will read untill a new line has been found and when it finds a new line, it will return you the line. But if the file ends without a new line, read treats this as an error. So, even if read has set the returning variable with the line read till now, the return code of read is set to indicate an error. Now the while read ... this loop body will only executes if the command executes with a success which is not a case here. Thus you miss the last line.
For overcoming this , you can change the condition to also check the returning variable is empty or not. Hence the condition succeeds even if read fails, as the variable is already set till the end of the file.
This is not related to line ending in different OS, i mean it's somehow related, but the exact root cause is always the read fails to find a new line at the end of the line/file, and the last line is missing the loop body.
Below is an example
[[bash_prompt$]]$ echo -ne 'hello\nthere' > log
[[bash_prompt$]]$ while read line; do echo $line; done < log
hello
[[bash_prompt$]]$ while read line || [ -n "$line" ]; do echo $line; done < log
hello
there
[[bash_prompt$]]$

Answer 4

read nees the end of line to read the input. Try
echo -n $'a\nb' | while read x ; do echo $x ; done
It only prints a.

Answer 5

To prevent script not reading last line of a file:
cat "somefile" | { cat ; echo ; } | while read line; do echo $line; done
Source : My open source project https://sourceforge.net/projects/command-output-to-html-table/