I have a file called this.txt that has this content:
a
b
c
d
Which I generate using: ls /home > this.txt
Then I create a file called that.txt that has this content:
a
c
d
f
Which I generate using: ssh -p 1111 root#176.178.1.8 'ls /home' > that.txt
When I compare both using diff this.txt that.txt I get normal results.
Then I get the file that2.txt using an expect script to avoid typing the password for the ssh connection, with this content
a
c
d
f
Using cat I compare (visually) both files and are the same, but when I use diff this.txt that.txt I get results with no sense (it says that nothing from this.txt is in that2.txt).
Also if I use diff that.txt that2.txt I get the no sense result.
Maybe is because I'm using two different interpreters (because I use expect and bash) and the files are coded different? Any ideas?
PD: hopefully I explained myself. I'm not an English speaker and this is my first question.
I’d assume you have files with either blanks at the ends of lines or different end-of-line markers, possibly both. Please compare the outputs of od -c that.txt and od -c that2.txt. Also, it may be worth checking the file sizes.
Oh, and I should add that you do not need to put your password into an expect script. ssh can work with public key pairs, a much safer alternative, and not really hard to set up. Check man ssh-keygen for a start.
Related
I'm trying to display the output of an AWS lambda that is being captured in a temporary text file, and I want to remove that file as I display its contents. Right now I'm doing:
... && cat output.json && rm output.json
Is there a clever way to combine those last two commands into one command? My goal is to make the full combined command string as short as possible.
For cases where
it is possible to control the name of the temporary text file.
If file is not used by other code
Possible to pass "/dev/stdout" as the.name of the output
Regarding portability: see stack exchange how portable ... /dev/stdout
POSIX 7 says they are extensions.
Base Definitions,
Section 2.1.1 Requirements:
The system may provide non-standard extensions. These are features not required by POSIX.1-2008 and may include, but are not limited to:
[...]
• Additional character special files with special properties (for example, /dev/stdin, /dev/stdout, and /dev/stderr)
Using the mandatory supported /dev/tty will force output into “current” terminal, making it impossible to pipe the output of the whole command into different program (or log file), or to use the program when there is no connected terminals (cron job, or other automation tools)
No, you cannot easily remove the lines of a file while displaying them. It would be highly inefficient as it would require removing characters from the beginning of a file each time you read a line. Current filesystems are pretty good at truncating lines at the end of a file, but not at the beginning.
A simple but extremely slow method would look like this:
while [ -s output.json ]
do
head -1 output.json
sed -i 1d output.json
done
While this algorithm is plain and simple, you should know that each time you remove the first line with sed -i 1d it will copy the whole content of the file but the first line into a temporary file, resulting in approximately 0.5*n² lines written in total (where n is the number of lines in your file).
In theory you could avoid this by do something like that:
while [ -s output.json ]
do
line=$(head -1 output.json)
printf -- '%s\n' "$line"
fallocate -c -o 0 -l $((${#len}+1)) output.json
done
But this does not account for variable newline characters (namely DOS-formatted newlines) and fallocate does not always work on xfs, among other issues.
Since you are trying to consume a file alongside its creation without leaving a trace of its existence on disk, you are essentially asking for a pipe functionality. In my opinion you should look into how your output.json file is produced and hopefully you can pipe it to a script of your own.
I want to store output of ls command in my bash script in a variable and use each file name in a loop, but for example one file in the directory has name "Hello world", when I do variable=$(ls) "Hello" and "world" end up as two separate entries, and when I try to do
for i in $variable
do
mv $i ~
done
it shows error that files "Hello" and "world" doesn't exist.
Is there any way I can access all files in current directory and run some command even if the files have space(s) in their names.
If you must, dirfiles=(/path/of/interest/*).
And accept the admonition against parsing the output of ls!
I understand you are new to this and I'd like to help. But it isn't easy for me (us?) to provide you with an answer that would be of much help to you by the way you've stated your question.
Based on what I hear so far, you don't seem to have a basic understanding on how parameter expansions work in the shell. The following two links will be useful to you:
Matching Pathnames, Parameters
Now, if your task at hand is to operate on files meeting certain criteria then find(1) will likely to do the job.
Say it with me: don't parse the output of ls! For more information, see this post on Unix.SE.
A better way of doing this is:
for i in *
do
mv -- "$i" ~
done
or simply
mv -- * ~
How do I get the first n lines of the output of a makefile (specifically, my complier is g++). Either a script in linux or in the makefile would work (if you could provide both, that'll be even better).
I have tried
make | head -n 5
but it's not working.
Currently, the process I go through is tedious; I'm piping the output to a text file before using head on it (then having to delete the file).
Given that the messages from the compiler appear on standard error rather than standard output, you need to redirect both:
make 2>&1 | head -n 20
(I think 5 lines will be too small to be useful.)
I'd like to merge two files by doing the following:
Output the diff of the two files into a temp file and
Manually select the lines I want to copy/save.
The problem here is that diff -u only gives me a file lines of context, while I want to output the entire file in a unified format.
Is there any way diff can do this?
One option that might fit the bill for you,
sdiff : side-by-side diff of files.
sdiff -o merged.file left.file right.file
Once there, it will prompt you with what lines you want to keep from which file. Hit ? and then enter for a little help. Also man sdiff with the detailed goods.
(In my distro, these come packaged in the "diffutils" package [fedora,centos])
If you need to automate the process, you might want to try the util merge, which will mark conflicts in the files. However, that might put you back at square one.
"I want to output the entire file in a unified format. Is there any way diff can do this?"
Yes.
diff -U 9999999 file1.txt file2.txt > diff.txt
This should work, provided your files are less than 10 million lines long.
You can merge/combine the two files with diff using --
diff --line-format %L file1 file2
The easy answer is to use the -D flag to merge the files and surround the differences with C style #ifdef statements.
From the documentation:
-D NAME --ifdef=NAME
Output merged file to show `#ifdef NAME' diffs.
You can use it as follows:
$ diff -D NEWSTUFF file1 file2 > merged_file
I usually then just open the merged file in an editor and resolve the merge conflicts by hand.
You also can use options to output an ed script, etc.
If you are an emacs user, you can do this directly in emacs using the "emerge" tool:
https://www.gnu.org/software/emacs/manual/html_node/emacs/Emerge.html
Issuing M-x emerge-files will open an interactive prompt with a view of files A, B, and the merged file to allow choosing text that differs between files A & B, inserting part of A into B, and more.
I like to compare two text files and save the difference under linux.
I know there are tools like kdiff, diff vimdiff etc. but my expectation are as follows.
Output should be in a separate file
The difference should be quoted with colours, ex: delete line in red and added line in green something like that
It should ignore space differences
It should be an opensource tool
use tkdiff4 -w file-name1 file-name2
It fulfills all your requirements. Specific color might be an issue.
try colordiff and man diff for options for ignoring whitespace etc
Like,
#!/bin/bash
wdiff -w "\e[31m" -x "\e[0m" -y "\e[32m" -z "\e[0m" "$#";
replace \e by, well, the ASCII character with value 0x1A. Put the two commands into some file, and run it using redirection.
Save the changes to a file:
diff -Nur originalfile newfile > patchfile
Use the difference file to change the origin file:
patch originfile patchfile
I think this is the easiest way to save the changes and reload the changes.
By the way, you can use this command the create an update-package.