I've copied a file from HDFS into my local file system (all on RH linux). However, after the copy, if I cat the file, I see the following:
[me#ac12 ~]$ cat file_copy
0|name|string
1|phone|string
2|age|string[me#ac12 ~]$
What I expected was this:
[me#ac12 ~]$ cat file_copy
0|name|string
1|phone|string
2|age|string
[me#ac12 ~]$
You can see that a newline seems to be missing in the first cat, and the shell prompt is on the same line as the last line. Why would this be and how can I diagnose the issue?
EDIT: I can't edit the output file (well, maybe I could but I really don't want to - I'd rather fix the problem at the source). I want to know why there is no newline character..
You can use this sed to add a newline after last line:
sed -i.bak $'$s/$/\\n/' file_copy
EDIT:
Or else use (thanks to #JonathanLeffler):
echo '' >> file_copy
Related
I have a file contained \n hidden behind each line:
input:
s3741206\n
s2561284\n
s4411364\n
s2516482\n
s2071534\n
s2074633\n
s7856856\n
s11957134\n
s682333\n
s9378200\n
s1862626\n
I want to remove \n behind
desired output:
s3741206
s2561284
s4411364
s2516482
s2071534
s2074633
s7856856
s11957134
s682333
s9378200
s1862626
however, I try this:
tr -d '\n' < file1 > file2
but it goes like below without space and new line
s3741206s2561284s4411364s2516482s2071534s2074633s7856856s11957134s682333s9378200s1862626
I also try sed $'s/\n//g' -i file1 and it doesn't work in mac os.
Thank you.
This is a possible solution using sed:
sed 's/\\n/ /g'
with awk
awk '{sub(/\\n/,"")} 1' < file1 > file2
What you are describing so far in your question+comments doesn't make sense. How can you have a multi-line file with a hidden newline character at the end of each line? What you show as your input file:
s3741206\n
s2561284\n
s4411364\n
etc.
where each "\n" above according to your comment is a single newline character "\n" is impossible. If those "\n"s were newline characters then your file would simply look like:
s3741206
s2561284
s4411364
etc.
There's really only 2 possibilities I can think of:
You are wrongly interpreting what you are seeing in your input file
and/or using the wrong terminology and you actually DO have \r\n
at the end of every line. Run cat -v file to see the \rs as
^Ms and run dos2unix or similar (e.g. sed 's/\r$//' file) to
remove the \rs - you do not want to remove the \ns or you will
no longer have a POSIX text file and so POSIX tools will exhibit
undefined behavior when run on it. If that doesn't work for you then
copy/paste the output of cat -v file into your question so we can
see for sure what is in your file.
Or:
It's also entirely possible that your file is a perfectly fine POSIX
text file as-is and you are incorrectly assuming you will have a
problem for some reason so also include in your question a
description of the actual problem you are having, include an example
of the command you are executing on that input file and the output
you are getting and the output you expected to get.
You could use bash-native string substitution
$ cat /tmp/newline
s3741206\n
s2561284\n
s4411364\n
s2516482\n
s2071534\n
s2074633\n
s7856856\n
s11957134\n
s682333\n
s9378200\n
s1862626\n
$ for LINE in $(cat /tmp/newline); do echo "${LINE%\\n}"; done
s3741206
s2561284
s4411364
s2516482
s2071534
s2074633
s7856856
s11957134
s682333
s9378200
s1862626
I have a file that contains a list of file names to be opened later.
After I load lines (file names) to variables, for a reason unknown to me I cannot open it as a file later.
Here is a simplified example of what i'm trying to do:
Main file's contents:
first_file.txt
second_file.txt
Bash commands:
read line < $main_file # file with the list, received as an argument
echo $line # to check that correct filename has been read
cat $line # attempt to dump "first_file.txt" contents <- FAILS
cat first_file.txt # read "first_file.txt" contents manually
Execution esult:
first_file.txt
: No such file or directory
*** this is 1st file's contents ***
*** ....
So, cat first_file.txt works, $line contains "first_file.txt", but cat $line fails...
I obviously misunderstand something here, suggestions are welcomed!
Edit:
As requested, here is cat -v $main_file's output:
first_file.txt^M
second_file.txt^M
third_file.txt^M
^M
^M
The ^M characters are carriage returns (a.k.a. \r) and are often part of a Windows line ending. They don't show up when you echo them, but they are messing up your ability to open a file with the text having it at the end.
The best solution is to remove them from your "main file." You could use the dos2unix tool if you have it, or you could use GNU sed like sed -i -e 's/\s+$//g' $main_file to edit it in place and remove the extra white space (which includes ^M) from the end of each line.
I searched the Internet, but maybe I used the wrong keyword, but I couldn't find the syntax to my very simple problem below:
How do I redirect a file as command line arguments to the Linux command "touch"? I want to create a file with "touch abc.txt", but the filename should come from the filename "file.txt" which contains "abc.txt", not manually typed-in.
[root#machine ~]# touch < file.txt
touch: missing file operand
Try `touch --help' for more information.
[root#machine ~]# cat file.txt
abc.txt
Try
$ touch $(< file.txt)
to expand the content of file.txt and give it as argument to touch
Alternatively, if you have multiple filenames stored in a file, you could use xargs, e.g.,
xargs touch <file.txt
(It would work for just one, but is more flexible than a simple "echo").
I am using this command in a shell script
lnum=5
str="Hello foo"
filename="/path/fiename"
sed -i "$lnum i $str" $filename
Getting the following error
sed: -e expression #1, char 3: : doesn't want any addresses
I had used this command before for other script it worked fine, the only change i made this time is file-name has a path to the file, but I tried it with just giving file-name and not the path by getting into the path and executing the script but still it doesn't work
I am unable to resolve it can anybody help
If you are using OSX , BSD (and AIX) versions of sed, the backup extension for the -i in place editing flag is not optional.
GNU sed differs on this I believe, so the script may work on Linux.
This is a bit of a pain for portability - but it gets worse with "in-place" editing when BSD derived sed is used. This version of sed is arguably more "standard" in some ways (as in: "lowest common denominator across POSIX systems") but this behaviour seems like a bug:
sed: 1: "5 i hello foo": command i expects \ followed by text
Here is how I made your script work on several BSD flavors:
lnum="5"
str="Hello foo"
filename="sed-mess.txt"
sed -i "" "$lnum i\^M
$str" $filename
I had to enter a literal line end character with Ctrl-v [Return] to get the i command to work since \ is required and has to have nothing following it. Not sure how GNU sed would handle this.
Can you use perl ? ;-)
The old farts use ed for this type of problem:
cat /tmp/sample
#!/bin/ksh
lnum=5
str="Hello bar"
filename="/tmp/foo"
ed - $filename <<EOF
$lnum
i
$str
.
w
q
EOF
cat /tmp/foo
line
line
line
line
line
line
line
line
line
wc /tmp/foo
9 9 45 /tmp/foo
/tmp/sample
line
wc /tmp/foo
10 11 55 /tmp/foo
cat /tmp/foo
line
line
line
line
Hello bar
line
line
line
line
line
i is "insert" before while a is "append" after. I don't know what the sed i command does exactly but I would expect it would be the same as i
Hope this helps
I want to edit a file from the command line, because opening it in vim or other editors takes forever (a large file). I want to add a string ('chr') to the beginning of every line that is not commented out with a #. The command I am using is this:
cat '/home/me/37.vcf' | sed s/^/chr/>'sp.vcf'
But it adds a chr to the beginning of EVERY line and a > to the END of every line. I don't want either of those things to occur.
Can anyone offer any suggestions to improve my results?
To apply the substitution to only the lines that don't start with a #:
sed '/^[^#]/s/^/chr/' file > output
Note: the command cat is for concatenating files, it is useless here.
You can syntax error in your sed command. Use this syntactically correct sed command:
sed -E 's/^([^#]|$)/chr/' /home/me/37.vcf > sp.vcf
OR on Linux:
sed -r 's/^([^#]|$)/chr/' /home/me/37.vcf > sp.vcf
This might work for you (GNU sed):
sed '/^\s*#/!s/^/chr/' file > new_file