I have a text file, each line is one or more file paths separated with space, all the file has suffix dl, e.g.
/some/path/file.dl
/some/other/path/file2.dl /some/other/path2/file3.dl
/some/other/path3/file4.dl /some/other/path4/file5.dl ...
...
Now I need to transform the above file to another text file. Only the first file of every line should be changed to /out/P{fileName}.h:, {fileName} is the original file name without directory and suffix. e.g.
/out/Pfile.h:
/out/Pfile2.h: /some/other/path2/file3.dl
/out/Pfile4.h: /some/other/path2/file5.dl ...
...
So how can I write the linux shell script?
Try this command:
$ sed -r 's#^\S*/(\S*)\.dl#/out/P\1.h:#' input
/out/Pfile.h:
/out/Pfile2.h: /some/other/path2/file3.dl
/out/Pfile4.h: /some/other/path4/file5.dl
Related
I would like to run a command on a list of paired files with the following format
SAMPLE_1.1.fq.gz SAMPLE_1.2.fq.gz
SAMPLE_2.1.fq.gz SAMPLE_2.2.fq.gz
etc...
etc. That are in a directory called ../cleaned-trimmed
I have the list of samples in a txt file (samples_final.txt - one sample per line) in a directory called info.
SAMPLE_1
SAMPLE_2
SAMPLE_3
I would like to run the following command on all the samples:
gsnap <args> --output-file=./alignments.gsnap/SAMPLE_1.mapped.sam --failed-input=./alignments.gsnap/SAMPLE_1.unmapped.fa ../cleaned-trimmed/SAMPLE_1.1.fq.gz ../cleaned-trimmed/SAMPLE_1.2.fq.gz
Where the args are the database used, command flags, etc.
I modified the script from a previous answer in stackoverflow to build a loop as follows:
for file in $(<../info/samples_final.txt)
do
gsnap <args> --output-file=./alignments.gsnap/${file}.mapped.sam --failed-input=./alignments.gsnap/${file}.unmapped.fa ../cleaned-trimmed/${file}.1.fq.gz ../cleaned-trimmed/${file}.2.fq.gz
done
but it does not pass the variables on correctly.
How do I pass on the values from samples_final.txt to the command?
At the moment the script garbles the file names when I run the loop. So, for example, if I run a test on the file "for_test2.txt":
SAMPLE_1
SAMPLE_2
Using the echo command:
for file in $(<../info/for_test2.txt)
do
echo ../cleaned-trimmed/${file}.1.fq.gz
done
I get the following output:
.1.fq.gzed-trimmed/SAMPLE_1
.1.fq.gzed-trimmed/SAMPLE_2
.1.fq.gzed-trimmed/
So it seems to have replaced ../clean with the .1.fq.gz
I genuinely do not understand the logic of this.
You have dos line endings in your file. The "carrier return" character makes the cursor jump to the beginning of the current line, that's why the .1.fq.gz part in your last code snipped is printed on the beginning of the line. You can first convert your file to normal line endings:
dos2unix ../info/for_test2.txt
Then read the file line by line and execute your command. Remember to quote your variables:
while IFS= read -r file; do
# protect against empty lines in input file
if [ -z "$file" ]; then continue; fi
gsnap <args> --output-file=./alignments.gsnap/"$file".mapped.sam --failed-input=./alignments.gsnap/"$file".unmapped.fa ../cleaned-trimmed/"$file".1.fq.gz ../cleaned-trimmed/"$file".2.fq.gz
done <../info/for_test2.txt
or like a pro with xargs:
<../info/for_test2.txt xargs -I{} gsnap <args> --output-file=./alignments.gsnap/{}.mapped.sam --failed-input=./alignments.gsnap/{}.unmapped.fa ../cleaned-trimmed/{}.1.fq.gz ../cleaned-trimmed/{}.2.fq.gz
I am trying replace a line with another line and add new line below that using shell script. My file content looks like below. I want to replace export PATH with JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm-1.8.0.2.10-1jpp.1.el7.x86_64
export PATH JAVA_HOME
before:
export PATH
It should be after:
JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm-1.8.0.2.10-1jpp.1.el7.x86_64
export PATH JAVA_HOME
That means, it has to replace replace export PATH with these two lines.
Please help me how can I do with sed or shell script
Thanks,
Kumar.
sed -e 's#^export PATH$#JAVA_HOME=/usr/lib/jvm/java-1.8.0-ibm-1.8.0.2.10-1jpp.1.el7.x86_64\n\nexport PATH JAVA_HOME#' <yourfile.txt
This is just a single substitute command in sed.
Some notes:
The # is used after s instead of the usual / to avoid having to quote all slashes in the path. Otherwise you have to quote each / in the path this way: /. ow you only have to quote #.
The escape sequence \n insert a newline.
Add the -i parameter to sed if you want to actually update the file. The code above only prints the new file to stdout.
The regex is anchored (^...$) so that it matches the whole line and not just a part of it.
This question already has answers here:
sed edit file in place
(15 answers)
Closed 7 years ago.
I want to remove the headers of a file and replace its content without headers in the same file.
Example: file_student
name age
XYS 24
RTF 56
The output should be:
XYS 24
RTF 56
The scenario is that I do not want to create any new file for this change. Can sed do this?
I tried:
sed 1d /tmp/file_student.txt |
hadoop fs -copyfromLocal /tmp/file_student.txt /tmp/file_student_no_header.txt
But that does not work. Any help is appreciated!
an extract from sed's man page
-i[SUFFIX]'
--in-place[=SUFFIX]'
This option specifies that files are to be edited in-place. GNU
`sed' does this by creating a temporary file and sending output to
this file rather than to the standard output.(1).
This option implies `-s'.
When the end of the file is reached, the temporary file is renamed
to the output file's original name. The extension, if supplied,
is used to modify the name of the old file before renaming the
temporary file, thereby making a backup copy(2)).
This rule is followed: if the extension doesn't contain a `*',
then it is appended to the end of the current filename as a
suffix; if the extension does contain one or more `*' characters,
then _each_ asterisk is replaced with the current filename. This
allows you to add a prefix to the backup file, instead of (or in
addition to) a suffix, or even to place backup copies of the
original files into another directory (provided the directory
already exists).
If no extension is supplied, the original file is overwritten
without making a backup.
so you need to change your sed command to sth like
sed -i 1d file | whatever
hope this helps.
If you don't want to use sed, try tail - e.g. you have a file called xxx:
tail -n +2 xxx > xxx.tmp && mv xxx.tmp xxx
What I'm trying to accomplish is having a central configuration file, in bash, that defines some variables that are re-used in different bash files. The example below attempts to generate a file name with the current date included in the file name as well as a variable defined in another shell script. However whenever I try to concatenate this external variable it doesn't work. I can concatenate the variable in any other situation.
Example Code:
../config/vars.sh
#!/bin/bash
mysqlUser="backupuser"
mysqlPwd="fakePwd"
mysqlSocket="/var/run/mysqld/mysqld.sock"
mysqlPort="3306"
serverName="s01.catchyservername.com"
./dbBackup.sh
#!/bin/bash
source ../config/vars.sh
tempName=$(date +"%Y%m%d.sql.gz")
fileName="mysqld_${mysqlPort}_${tempName}"
echo "mysqld_${mysqlPort}"
echo ${tempName}
echo ${fileName}
output of dbBackup.sh
mysqld_3306
20140926.sql.gz
_20140926.sql.gz
As you can see when echoing "mysqld_${mysqlPort}" I get the expected output, but when echoing ${fileName} the entire first half of the string is ignored. What am I misunderstanding?
Your vars.sh file was probably created with a DOS/windows text editor:
$ ./dbBackup.sh
mysqld_3306
20140926.sql.gz
_20140926.sql.gz
$ dos2unix vars.sh
dos2unix: converting file vars.sh to Unix format ...
$
$ ./dbBackup.sh
mysqld_3306
20140926.sql.gz
mysqld_3306_20140926.sql.gz
$
As you can see above, I use the dos2unix utility to convert the line separators to Unix style.
I am using Shell scripting. I tried the below option.
I want to append the part of a file to another part of a file.
I tried this command to append the content of a file.
Command:
/bin/cat ../../../test_op.txt/sql/part_code.txt >> ../../../PartitioningUtility/log/test_op.txt
To append the text of part_code.txt to test_op.txt.
No errors but the text is not appending.
Any Solution to this problem.
Any other option to append the data of a one file to another
If you want to append all content from '/bin/cat ../../../test_op.txt/sql/part_code.txt' to '../../../PartitioningUtility/log/test_op.txt' you could add 'cat' in the beginning:
cat ../../../test_op.txt/sql/part_code.txt >> ../../../PartitioningUtility/log/test_op.txt
That is assuming that file paths are correct.