Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I have a csv file which has two columns: column A for image links and column B for names. First and second columns are comma separated. I need to download all files in column A and assign them names in column B. I have tried the syntax below:
#!/bin/bash
while IFS=',' read -r url filename
do
wget -O $filename $url
done < urls.csv
However, I got these errors:
line 2: $'\r': command not found
line 6: syntax error near unexpected token `done'
line 6: `done < urls.csv'
I am beginner with bash, any help with this?
There are several alternatives, for example using awk, to process a file with field separator characters like a CSV.
However, I will try to KISS this specific case:
Iterate through your CSV file (while)
Get each line fields (IFS is used to set ',' as field separator)
Use them with wget -O option to specify a filename
e.g. something like this:
#!/bin/bash
while IFS=',' read -r url filename
do
wget -O $filename $url
done < yourfile.csv
edit. Just copy pasted your snippet (which lacks proper identation inside the while loop..), and works properly
Perhaps you could share how are you executing that snippet ?
I'm saving it in 'test.sh' and launching it like this, having "urls.csv" file in same folder:
./test.sh
Related
This question already has answers here:
Rename multiple files based on pattern in Unix
(24 answers)
Closed 2 years ago.
I have a lot of file with the same format (???_ideal.sdf) name and I need to rename all (???.sdf) removing form the name "ideal". For example:
Files:
002_ideal.sdf
ERT_ideal.sdf
234_ideal.sdf
sCX_idel.sdf
New Files:
002.sdf
ERT.sdf
234.sdf
SCX.sdf
I thought to using a loop but I don’t know how to indicate that in the new file name should be removed "individual".
For example:
for file in ???_individual.sdf; mv $file what?
With your shown samples, could you please try following. This will only print the rename command on terminal(for safer side, check and make sure if commands are fine and looking ok to you first before actually renaming files), to perform actual rename remove echo from following.
for file in *_ideal.sdf
do
echo mv "$file" "${file/_ideal/}"
done
Based on this answer you can also write it as one line with:
rename 's/^(.*)_ideal.png/$1.png/s' **/**
First parameter replaces the the _ideal.png, second one means all files.
Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 2 years ago.
Improve this question
I am trying to translate markdown file into confluence markup as a complete beginner.
I need to make [Title](https:// site.com) into [Title|https:// site.com]. If it was just one link, i could add it to a var and printf it, but I am having trouble figuring out how to do it if I have 10 links for example.
Previously I used CONTENT=$(echo "${CONTENT//# /h1. }") to replace strings but since now every string is different, I am stuck at how to solve this. I found the solution written in javascript: http://chunpu.github.io/markdown2confluence/browser but fail to understand how to do it in bash.
For this test file
$ cat file
[Title](https://site1.com)
[Title](https://site2.com)
[Title](https://site3.com)
[Title](https://site4.com)
[Title](https://site5.com)
[Title](https://site6.com)
[Title](https://site7.com)
[Title](https://site8.com)
[Title](https://site9.com)
[Title](https://site10.com)
Sed variant:
$ sed 's/\](/|/;s/)/\]/' file
[Title|https://site1.com]
[Title|https://site2.com]
[Title|https://site3.com]
[Title|https://site4.com]
[Title|https://site5.com]
[Title|https://site6.com]
[Title|https://site7.com]
[Title|https://site8.com]
[Title|https://site9.com]
[Title|https://site10.com]
Bash variant:
while read -r line; do
line=${line//](/|}
line=${line//)/]}
echo $line
done < file
[Title|https://site1.com]
[Title|https://site2.com]
[Title|https://site3.com]
[Title|https://site4.com]
[Title|https://site5.com]
[Title|https://site6.com]
[Title|https://site7.com]
[Title|https://site8.com]
[Title|https://site9.com]
[Title|https://site10.com]
Awk variant:
$ awk '{ sub(/\]\(/, "|"); sub(/\)/, "]"); print }' file
[Title|https://site1.com]
[Title|https://site2.com]
[Title|https://site3.com]
[Title|https://site4.com]
[Title|https://site5.com]
[Title|https://site6.com]
[Title|https://site7.com]
[Title|https://site8.com]
[Title|https://site9.com]
[Title|https://site10.com]
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 5 years ago.
Improve this question
I have a text file of 75000 items, 2 lines for each item. line 1 has an identifier, line 2 a text string.
I need to remove 130 items, random identifiers that I have in a list or can put in a file.
I can carry out the removal for one item, but not for more than one.
I tried piping the identifiers and get an empty output file.
I tried repeated commands of sed -e 'expression' inputfile > outfile. This works, but requires a new output file that then becomes the inputfile for the next iteration and so on. this might be the last resort.
I tried sed -i in iteration; this crashes and the error is that there is no file by the name of the inputfile. Which is clearly not the case, as I can see it, ls it and grep the number of identifiers in it. Only sed can't seem to read it.
I even found a python/biopython script online for this exact problem, it is very simple and does not give error messages, but it also removes only the first item.
I think it has something to do with file properties/temporary files that don't really exist (?).
I am using Ubuntu 12.04 'Precise'
How can I get around this issue?
quick and dirty (no check if modification file is created, ...)
sed
Assuming there is no special meta character in your pattern list
sed 's#.*#/&/{N;d;}#' YourListToExclude > /tmp/exclude.sed
sed -f /tmp/exclude.sed YourDataFile > /tmp/YourDataFile.tmp
mv /tmp/YourDataFile.tmp YourDataFile
rm /tmp/exclude.sed
awk
awk 'FNR==NR{ex=(ex==""?"":ex"|")$0;next}$0!~ex{print;getline;print;next}{getline}' YourListToExclude YourDataFile > /tmp/YourDataFile.tmp
mv /tmp/YourDataFile.tmp YourDataFile
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I'm working on redHat linux.
I've a file which looks like :
$vi filename
Jan,1,00:00:01,someone checked your file
Jan,3,09:38:02,applebee
Jan,16,10:20:03, ****************
Jan,18,03:04:03, ***************
I want the output to look like:
2015/01/01,00:00:01,someone checked your file
2015/01/03,3,09:38:02,applebee
2015/01/16,16,10:20:03, ****************
2015/01/18,03:04:03, ***************
Please help me to do this. Thanks
If you have GNU date, try:
$ awk -F, '{cmd="date -d \""$1" "$2"\" +%Y/%m/%d"; cmd|getline d; print d","$3","$4; close(cmd)}' file
2015/01/01,00:00:01,someone checked your file
2015/01/03,09:38:02,applebee
2015/01/16,10:20:03, ****************
2015/01/18,03:04:03, ***************
This approach cannot be used with the BSD (OSX) version of date because it does not support any comparable -d option.
How it works
awk implicitly loops over lines of input, breaking each line into fields.
-F,
This tells awk to use a comma as the field separator
cmd="date -d \""$1" "$2"\" +%Y/%m/%d"
This creates a string variable, cmd, and contains a date command. I am assuming that you have GNU date.
cmd|getline d
This runs the command and captures the output in variable d.
print d","$3","$4
This prints the output that you asked for.
close(cmd)
This closes the command.
This question already has answers here:
How to concatenate string variables in Bash
(30 answers)
Closed 9 years ago.
I am trying to download files from a database using wget and url. E.g.
wget "http://www.rcsb.org/pdb/files/1BXS.pdb"
So format of the url is as such: http://www.rcsb.org/pdb/files/($idnumber).pdb"
But I have many files to download; so I wrote a bash script that reads id_numbers from a text file, forms url string and downloads by wget.
!/bin/bash
while read line
do
url="http://www.rcsb.org/pdb/files/$line.pdb"
echo -e $url
wget $url
done < id_numbers.txt
However, url string is formed as
.pdb://www.rcsb.org/pdb/files/4H80
So, .pdb is repleced with http. I cannot figure out why. Does anyone have an idea?
How can I format it so url is
"http://www.rcsb.org/pdb/files/($idnumber).pdb"
?
Thanks a lot.
Note. This question was marked as duplicate of 'How to concatenate strings in bash?' but I was actually asking for something else. I read that question before asking this one and it turns out my problem was with preparing the txt file in Windows not really string concetanation. I edited question title. I hope it is more clear now.
It sounds like your id_numbers.txt file has DOS/Windows-style line endings (carriage return followed by linefeed characters) instead of plain unix line endings (just linefeed). The result is that read thinks the line ends with a carriage return, $line actually has a carriage return at the end, and that gets embedded in the url, causing various confusion.
There are several ways to solve this. You could have bash trim the carriage return from the variable when you use it:
url="http://www.rcsb.org/pdb/files/${line%$'\r'}.pdb"
Or you could have read trim it by telling it that carriage return counts as whitespace (read will trim leading and trailing whitespace from what it reads):
while IFS=$'\r' read line
Or you could use a command like dos2unix (or whatever the equivalent is on your OS) to convert the id_numbers.txt file.
The -e echo option is used to output the desired content without inserting a new line, you do not need it here.
Also I suspect your file containing the ids to be malformed, on which OS did you create it?
Anyway, you can simplify your script this way:
!/bin/bash
while read line
do
wget "http://www.rcsb.org/pdb/files/$line.pdb"
done < id_numbers.txt
I was able to successfully test it with an id_numbers.txt file generated like so:
for i in $(0 9) ; do echo "$i" >> id_numbers.txt ; done
Try this:
url="http://www.rcsb.org/pdb/files/"$line
$url=$url".pdb"
For more info, check How to concatenate string variables in Bash?