How to flush output of dlmwrite in Octave - io

dlmwrite command writes to a file, but without last few hundreds bytes. How can i flush these bytes. Seems i can't use fflush as i don't have file id. My Octave version is 4.2.1
EDIT: I conducted small experiment:
>> A=[1 1];
>> dlmwrite('A',A); % after this line you will have an empty file
>> exit % after this line the file will contain "1,1"

Well, you've found a bug in GNU Octave in dlmwrite if the filename is only one char long. Have a look at dlmwrite.m around line 197
...
if (! isscalar (file))
fclose (fid);
endif
...
This code snippet tries to check if file is a filename or a fid (see alternative calling methods on dlmwrite). Obviously this check fails if the filename is only one byte long and is thus a scalar.
Please confirm that all works as expected if you use filenames with more than one char.
EDIT:
Bugreport on Savannah
Bugfix on the stable branch

Related

How do I implement "file -s <file>" on Linux in pure Go?

Intent:
Does Go have the functionality (package or otherwise) to perform a special file stat on Linux akin to the command file -s <path>
Example:
[root#localhost ~]# file /proc/uptime
/proc/uptime: empty
[root#localhost ~]# file -s /proc/uptime
/proc/uptime: ASCII text
Use Case:
I have a fileglob of files in /proc/* that I need to very quickly detect if they are truly empty instead of appearing to be empty.
Using The os Package:
Code:
result,_ := os.Stat("/proc/uptime")
fmt.Println("Name:",result.Name()," Size:",result.Size()," Mode:",int(result.Mode()))
fmt.Printf("%q",result)
Result:
Name: uptime Size: 0 Mode: 292
&{"uptime" '\x00' 'Ĥ' {%!q(int64=63606896088) %!q(int32=413685520) %!q(*time.Location=&{ [] [] 0 0 <nil>})} {'\x03' %!q(uint64=4026532071) '\x01' '脤' '\x00' '\x00' '\x00' '\x00' '\x00' 'Ѐ' '\x00' {%!q(int64=1471299288) %!q(int64=413685520)} {%!q(int64=1471299288) %!q(int64=413685520)} {%!q(int64=1471299288) %!q(int64=413685520)} ['\x00' '\x00' '\x00']}}
Obvious Workaround:
There is the obvious workaround of the following. But it's a little over the top to need to call in a bash shell in order to get file stats.
output,_ := exec.Command("bash","-c","file -s","/proc/uptime").Output()
//parse output etc...
EDIT/MY PRACTICAL USE CASE:
Quickly determining which files are zero size without needing to read each one of them first.
file -s /cgroup/memory/lsf/<cluster>/*/tasks | <clean up commands> | uniq -c
6 /cgroup/memory/lsf/<cluster>/<jobid>/tasks: ASCII text
805 /cgroup/memory/lsf/<cluster>/<jobid>/tasks: empty
So in this case, I know that only those 6 jobs are running and the rest (805) have terminated. Reading the file works like this:
# cat /cgroup/memory/lsf/<cluster>/<jobid>/tasks
#
or
# cat /cgroup/memory/lsf/<cluster>/<jobid>/tasks
12352
53455
...
I'm afraid you might be confusing matters here: file is special in precisely a way it "knows" a set of heuristics to carry out its tasks.
To my knowledge, Go does not have anything like this in its standard library, and I've not came across a 3rd-party package implementing a file-like functionality (though I invite you to search by relevant keywords on http://godoc.org)
On the other hand, Go provides full access to the syscall interface of the underlying OS so when it comes to querying the OS in a way file does it, there's nothing you could not do in plain Go.
So I suggest you to just fetch the source code of file, learn what it does in its mode turned on by the "-s" command-line option and implement that in your Go code.
We'll try to have you with specific problems doing that — should you have any.
Update
Looks like I've managed to grasp the OP is struggling with: a simple check:
$ stat -c %s /proc/$$/status && wc -c < $_
0
849
That is, the stat call on a file under /proc shows it has no contents but actually reading from that file returns that contents.
OK, so the solution is simple: instead of doing a call to os.Stat() while traversing the subtree of the filesystem one should instead merely attempt to read a single byte from the file, like in:
var buf [1]byte
f, err := os.Open(fname)
if err != nil {
// do something, or maybe ignore.
// A not existing file is OK to ignore
// (the POSIX error code will be ENOENT)
// because after the `path/filepath.Walk()` fetched an entry for
// this file from its directory, the file might well have gone.
}
_, err = f.Read(buf[:])
if err != nil {
if err == io.EOF {
// OK, we failed to read 1 byte, so the file is empty.
}
// Otherwise, deal with the error
}
f.Close()
You might try to be more clever and first obtain the stat information
(using a call to os.Stat()) to see if the file is a regular file—to
not attempt reading from sockets etc.
I have a fileglob of files in /proc/* that I need to very quickly
detect if they are truly empty instead of appearing to be empty.
They are truly empty in some sense (eg. they occupy no space on file system). If you want to check whether any data can be read from them, try reading from them - that's what file -s does:
-s, --special-files
Normally, file only attempts to read and
determine the type of argument files which stat(2) reports are
ordinary files. This prevents problems, because reading special files
may have peculiar consequences. Specifying the -s option causes file
to also read argument files which are block or character special
files. This is useful for determining the filesystem types of the
data in raw disk partitions, which are block special files. This
option also causes file to disregard the file size as reported by
stat(2) since on some systems it reports a zero size for raw disk
partitions.

Concatenating string read from file with string literals creates jumbled output

My problem is that the result is jumbled. Consider this script:
#!/bin/bash
INPUT="filelist.txt"
i=0;
while read label
do
i=$[$i+1]
echo "HELLO${label}WORLD"
done <<< $'1\n2\n3\n4'
i=0;
while read label
do
i=$[$i+1]
echo "HELLO${label}WORLD"
done < "$INPUT"
filelist.txt
5
8
15
67
...
The first loop, with the immediate input (through something I believe is called a herestring (the <<< operator) gives the expected output
HELLO1WORLD
HELLO2WORLD
HELLO3WORLD
HELLO4WORLD
The second loop, which reads from the file, gives the following jumbled output:
WORLD5
WORLD8
WORLD15
WORLD67
I've tried echo $label: This works as expected in both cases, but the concatenation fails in the second case as described. Further, the exact same code works on my Win 7, git-bash environment. This issue is on OSX 10.7 Lion.
How to concatenate strings in bash |
Bash variables concatenation |
concat string in a shell script
Well, just as I was about to hit post, the solution hit me. Sharing here so someone else can find it - it took me 3 hours to debug this (despite being on SO for almost all that time) so I see value in addressing this specific (common) use case.
The problem is that filelist.txt was created in Windows. This means it has CRLF line endings, while OSX (like other Unix-like environments) expects LF only line endings. (See more here: Difference between CR LF, LF and CR line break types?)
I used the answer here to convert the file before consumption. Using sed I managed to replace only the final line's carriage return, so I stuck to known guns and went for the perl approach. Final script is below:
#!/bin/bash
INPUTFILE="filelist.txt"
INPUT=$(perl -pe 's/\r\n|\n|\r/\n/g' "$INPUTFILE")
i=0;
while read label
do
i=$[$i+1]
echo "HELLO${label}WORLD"
done <<< $'INPUT'
Question has been asked in a different form at Bash: Concatenating strings fails when read from certain files

Linux: For "tail -f filename", does the follow read lines only after a \n?

Initial File:
line 1
line 2
line 3
File after append 1:
line 1
line 2
line 3
lin
File after append 2:
line 1
line 2
line 3
line 4
If I do a "tail -f filename", and then the file gets updated as above,
Does tail follow on with updates of half a line, or does it only return atomic lines?
i.e.
Am I guaranteed to see/not-see "lin" after append 1?
If there is no \n after "line 4", will I see it?
In particular, I'm interested in whether I can read atomic lines. I'm running a log monitoring application, and I only want to parse complete lines. The log files might be updated to half a line, and if tail doesn't guarantee atomic lines, I will have to guard against that (i.e. read char by char and then identify a line when I see the \n).
On my mac (9.8.0 Darwin Kernel Version 9.8.0) the built-in tail shows characters appended to the file. Here is a small test:
import sys, time
def write(fname):
fd = open(fname, 'wb')
try:
while True:
for i in range(0, 5):
fd.write('a')
fd.flush()
print >>sys.stderr, "Wrote a char"
time.sleep(2)
fd.write('\n')
fd.flush()
print >>sys.stderr, "Wrote newline"
time.sleep(2)
finally:
fd.close()
if __name__ == '__main__':
write(sys.argv[1])
Run: python test.py test.txt and then tail -f test.txt. It shows appearing a-s.
Well according to the man page:
-f, --follow[={name|descriptor}]
output appended data as the file grows; -f, --follow, and --follow=descriptor are equivalent
so that implies you'll see each character as it's appended, not just on new line. However, I imagine tail employs some kind of flushing strategy, so what you see may depend on how fast the file is being updated. I can only suggest you experiment a bit.
The thing is that there is buffering.
Tail will probably do buffering, but the source may well be linebuffering. If the source is under your control, try to disable output buffering and see how that goes

Resolving patch conflicts manually [duplicate]

I'm having trouble applying a patch to my source tree, and it's not the usual -p stripping problem. patch is able to find the file to patch.
Specifically, my question is how to read / interpret the .rej files patch creates when it fails on a few hunks. Most discussions of patch/diff I have seen don't include this.
A simple example:
$ echo -e "line 1\nline 2\nline 3" > a
$ sed -e 's/2/b/' <a >b
$ sed -e 's/2/c/' <a >c
$ diff a b > ab.diff
$ patch c < ab.diff
$ cat c.rej
***************
*** 2
- line 2
--- 2 -----
+ line b
As you can see: The old file contains line 2 and the new file should contain line b. However, it actually contains line c (that's not visible in the reject file).
In fact, the easiest way to resolve such problems is to take the diff fragment from the .diff/.patch file, insert it at the appropriate place in the file to be patched and then compare the code by hand to figure out, what lines actually cause the conflict.
Or - alternatively: Get the original file (unmodified), patch it and run a three way merge on the file.
Wiggle is a great tool for applying .rej files when patch does not succeed.
I'm not an expert on dealing with patch files, but I'd like to add some clarity on how to read them based on my understanding of the information they contain.
Your .rej files will tell you:
the difference between the original and the .rej file;
where the problem code starts in the original file, how many lines it goes on
for in that file;
and where the code starts in the new file, and how many lines it goes on for in that file.
So given this message, noted in the beginning of my .rej file:
diff a/www/js/app.js b/www/js/app.js (rejected hunks)
## -4,12 +4,24 ##
I see that for my problem file (www/js/app), the difference between the original (noted as a/www/js/app.js on the first line) and the .rej file (noted as b/www/js/) starts on line 4 of the original and goes on for 12 lines (the part before the comma in ## -4,12, +4,24 ## on line two), and starts on line 4 of the new version of the file and goes on for 24 lines (the part after the comma in ## -4,12, +4,24 ##.
For further information, see the excellent overview of patch files (containing the information I note above, as well as details on lines added and/or between file versions) at http://blog.humphd.org/vocamus-906/.
Any corrections or clarifications welcome of course.

How to tell binary from text files in linux

The linux file command does a very good job in recognising file types and gives very fine-grained results. The diff tool is able to tell binary files from text files, producing a different output.
Is there a way to tell binary files form text files? All I want is a yes/no answer whether a given file is binary. Because it's difficult to define binary, let's say I want to know if diff will attempt a text-based comparison.
To clarify the question: I do not care if it's ASCII text or XML as long as it's text. Also, I do not want to differentiate between MP3 and JPEG files, as they're all binary.
file is still the command you want. Any file that is text (according to its heuristics) will include the word "text" in the output of file; anything that is binary will not include the word "text".
If you don't agree with the heuristics that file uses to determine text vs. not-text, then the question needs to be better specified, since text vs. non-text is an inherently vague question. For example, file does not identify a PGP public key block in ASCII as "text", but you might (since it is composed only of printable characters, even though it is not human-readable).
The diff manual specifies that
diff determines whether a file is text
or binary by checking the first few
bytes in the file; the exact number of
bytes is system dependent, but it is
typically several thousand. If every
byte in that part of the file is
non-null, diff considers the file to
be text; otherwise it considers the
file to be binary.
A quick-and-dirty way is to look for a NUL character (a zero byte) in the first K or two of the file. As long as you're not worried about UTF-16 or UTF-32, no text file should ever contain a NUL.
Update: According to the diff manual, this is exactly what diff does.
This approach defers to the grep command in determining whether a file is binary or text:
is_text_file() { grep -qIF '' "$1"; }
grep options used:
-q Quiet; Exit immediately with zero status if any match is found
-I Process a binary file as if it did not contain matching data
-F Interpret PATTERNS as fixed strings, not regular expressions.
grep pattern used:
'' Empty string. All files (except an empty file)
will match this pattern.
Notes
An empty file is not considered a text file according to this test. (The GNU file command agrees with this assessment.)
A file with one printable character, say a, is considered a text file according to this test. (Makes sense to me.) (The file command disagrees with this assessment. (Tested with GNU file))
This approach requires only one child process to test whether a file is text or binary.
Test
# cd into a temp directory
cd "$(mktemp -d)"
# Create 3 corner-case test files
touch empty_file # An empty file
echo -n a >one_byte_a # A file containing just `a`
echo a >one_line_a # A file containing just `a` and a newline
# Another test case: a 96KiB text file that ends with a NUL
head -c 98303 /usr/share/dict/words > file_with_a_null_96KiB
dd if=/dev/zero bs=1 count=1 >> file_with_a_null_96KiB
# Last test case: a 96KiB text file plus a NUL added at the end
head -c 98304 /usr/share/dict/words > file_with_a_null_96KiB_plus1
dd if=/dev/zero bs=1 count=1 >> file_with_a_null_96KiB_plus1
# Defer to grep to determine if a file is a text file
is_text_file() { grep -qI '^' "$1"; }
# Test harness
do_test() {
printf '%22s ... ' "$1"
if is_text_file "$1"; then
echo "is a text file"
else
echo "is a binary file"
fi
}
# Test each of our test cases
do_test empty_file
do_test one_byte_a
do_test one_line_a
do_test file_with_a_null_96KiB
do_test file_with_a_null_96KiB_plus1
Output
empty_file ... is a binary file
one_byte_a ... is a text file
one_line_a ... is a text file
file_with_a_null_96KiB ... is a binary file
file_with_a_null_96KiB_plus1 ... is a text file
On my machine, it seems grep checks the first 96 KiB of a file for a NUL. (Tested with GNU grep). The exact crossover point depends on your machine's page size.
Relevant source code: https://git.savannah.gnu.org/cgit/grep.git/tree/src/grep.c?h=v3.6#n1550
You could try to give a
strings yourfile
command and compare the size of the results with the file size ... i'm not totally sure, but if they are the same the file is really a text file.
These days the term "text file" is ambiguous, because a text file can be encoded in ASCII, ISO-8859-*, UTF-8, UTF-16, UTF-32 and so on.
See here for how Subversion does it.
A fast way to do this in ubuntu is use nautilus in the "list" view. The type column will show you if its text or binary
Commands like less, grep detect it quite easily(and fast). You can have a look at their source.

Resources