Small Note: I removed the http:// from infront each link, because stackoverflow isn't allowing me to post it in original way.
I wrote a script which access to a webpage, to catch a URL and download it. One of the urls makes curl stop working and the whole URLS in the list to the same.
The script works as following:-
PAGE=$(curl -sL pageurl)
FILE_URL=$(echo $PAGE | sed -e 's/^.*<a href=\"\(.*\)\">\(.*\) alt="File" \/><\/a>.*$/\1/')
The FILE_URL VALUE is
URL/files/PartOne - Booke (Coll).pdf
webprod25.megashares.com/index.php?d01=3109985&lccdl=9e8e091ef33dd103&d01go=1&fln=/adobe reader exe.rar
AND SO One for others
When curl tried to catch this url it shows the following error using the debug mode of bash
++ curl -sOL 'webprod37.megashares.com/index.php?d01=3109985&lccdl=9e8e091ef33dd103&d01go=1&fln=/adobe' reader exe.rar fileshare273.depositfiles.com/auth-13023763920cd7ec18a0fdbfa8b62d35-188.165.197.50-43792102-7713641/FS273-7/PageMaker.rar -sOLJg fileshare601.depositfiles.com/auth-1302376689013d421df6c01e7f64c8d2-188.165.197.50-43801594-82379659/FS601-2/Adobe_Flash_Player_v10.3.180.65.2.rar -sOLJg 'webprod37.megashares.com/index.php?d01=de48789&lccdl=9e8e091ef33dd103&d01go=1&fln=/KAZAMIZA.COM.Adobe.Flash' Player-10.3.180.65.Beta-2.JUDGMENT DAY.rar bellatrix.oron.com/spzsttzwytpflwd76j3ne2moukomuhcdxg6llddfztqa2ztd7cplwwp457h3mxuacq3pbxzs/An-Beat - Mentally Insine '(Original' 'Mix).mp3'
curl: option -: is unknown
curl: try 'curl --help' or 'curl --manual' for more information
The quote marks the curl put it itself, I tried to do some workarounds like escaping url but it not works.
The basic problem seems to be that you are using $() expansion for something that looks to me like a multi line value. You should try iterating over each line.
The other problem looks like one of improper quoting of URLs containing spaces. There's a lone dash (-) in "An-Beat - Mentally Insine"
Oh, one more problem: The sed part to catch the href="..." contents only works if there's exactly one href on the line. If there are two or more, your \(.*\) will match everything else up to the last href. You should use something like href="\([^"]*\)", matching "any number of non-doublequotes followed by a doublequote".
Quote your variables as in:
pageurl='the url'
PAGE=$(curl -sL "$pageurl")
FILE_URL=$(echo "$PAGE" | sed -e 's/^.*<a href=\"\(.*\)\">\(.*\) alt="File" \/><\/a>.*$/\1/')
Otherwise, shell expansion will occur. The error "option -: is unknown" comes from the final part:
An-Beat - Mentally Insine
Because you didn't apply quotes to it, it got parsed as arguments, which you can clearly see in the syntax-highlighted code.
Related
I am working through the Node.js The Right Way book by Jim Wilson. I am currently trying to use a PUSH request to create a new bundle with the specified name.
* curl -X POST http://:/api/bundle?name=
However, when I use the command:
$ curl -s -X POST localhost:60702/api/bundle?name=light%20reading | jq '.'
rather than getting the JSON indicating that a Bundle has been created, I get: zsh: no matches found: localhost:60702/api/bundle?name=light%20reading
The command should be using a POST request to create a new All of my code is bit for bit identical to the code listed in the book. Any ideas?
Can you try
curl -s -X POST 'localhost:3000/api/bundle?name=light%20reading'
i.e wrap the url within '
This seems to be an issue with zsh solved here.
There are several ways to solve this:
You can escape the question mark ? in the url by quoting the url as explained by #huzaifa-saifuddin to avoid zsh treating it as a wildcard character.
As explained here, you can create an alias for curl: alias curl='noglob curl'
As explained here, you can disable to nomatch handling by adding the following to your ~/.zshrc: unsetopt nomatch
I want to get some data from a HTTP server. What it sends me depends on what I put in a POST request.
What I put in the INPUT_TEXT field is a sequence of words. When I run the following command, I get good looking output.
$ curl http://localhost:59125/process -d INPUT_TEXT="here are some words"
I want a bash script to take some string as a command line argument, and pass it appropriately to curl. The first thing I tried was to put the following in a script:
sentence=$1
command="curl http://localhost:59125/process -d INPUT_TEXT=\"${sentence}\""
$command
I then run the script like so:
$ ./script "here are some words"
But then I get a curl Couldn't resolve host error for each of "are", "some", and "words". It would seem that "here" got correctly treated as the INPUT_TEXT, but the rest of the words were then considered to be hosts, and not part of the option.
So I tried:
command=("curl" "http://localhost:59125/process" "-d" "INPUT_TEXT='$sentence'")
${command[#]}
I got the same output as the first script. I finally got what I wanted with:
result=$(curl http://localhost:59125/process -d INPUT_TEXT="${sentence}")
echo $result
I'm still unsure as to what the distinction is. In the first two cases, when I echoed out the contents of command, I get exactly what I input from the interactive Bash prompt, which had worked fine. What caused the difference?
The following will work:
command=("curl" "http://localhost:59125/process"
"-d" "INPUT_TEXT=$sentence")
"${command[#]}"
That has two changes from yours:
I removed the incorrect quotes around $sentence since you don't want to send quotes to the server (as far as I can see).
I put double-quotes around the use of "${command[#]}". Without the double quotes, the array's elements are concatenated with spaces between them and then the result is word-split. With double quotes, the individual array elements are used as individual words.
The second point is well-explained in the bash FAQ and a bunch of SO answers dealing with quotes.
The important thing to understand is that quotes only quote when a command is parsed. A quote which is a character in a variable is just a character; it is not reinterpreted when the value of the variable expanded. Whitespace in the variable is used for word-splitting if the variable expansion is unquoted; the fact that the whitespace was quoted in the the command which defined the variable is completely irrelevant. In this sense, bash is just the same as any other programming language.
Im trying to create a bash script to make github pull requests here its my code:
Gist Code
I get this error:
curl: (6) Could not resolve host: on
curl: (3) [globbing] unmatched close brace/bracket in column 63
{
"message": "Problems parsing JSON",
"documentation_url": "https://developer.github.com/v3"
}
Please help
More/proper quoting:
body=$(printf '{"title":"%s","body":"%s","head":"clamour:%s","base":"%s"}' "$TITLE" "$DESCRIPTION" "$TARGET" "$SOURCE")
curl -H "$auth" -d "$body" "https://api.github.com/repos/clamour/$PROJECT/pulls"
All user-supplied variables must be quoted (unless you know exactly why you want to leave then unquoted).
Enclosing a variable name in ${braces} is not the same as "$quoting".
I find using printf tends to be more clear than mixing double and single quotes with variable interpolation.
Get out of the habit of using ALLCAPS variable names: one day you'll use PATH=... and then wonder why your script is broken
you are using a lot of variables
try to do it like this
BODY="{\"title\":\"$TITLE\",\"body\":\"$DESCRIPTION\",\"head\":\"clamour\":\"$TARGET\",\"base\":\"$SOURCE\"}"
"https://api.github.com/repos/clamour/$PROJECT/pulls"
read also this Difference between single and double quotes in Bash
So, im making a small script to do an entire task for me. The task is to get the output of the dmidecode -Fn into a text file and then take a part of the dmidecode output, in my example, the Address (0xE0000) as the file name of the txt.
My script goes as follows and does work, i have tested it. The only little issue that i have, is that the file name of the txt appears as "? 0xE0000.txt"
My question is, why am i getting a question mark followed by a space in the name?
#!/bin/bash
directory=$(pwd)
name=$(dmidecode|grep -i Address|sed 's/Address://')
inxi -Fn > $directory/"$name".txt
The quotes in the "$name".txt is to avoid an "ambiguous redirect" error i got when running the script.
Update #Just Somebody
root#server:/home/user/Desktop# dmidecode | sed -n 's/Address://p'
0xE0000
root#server:/home/user/Desktop#
Solution
The use of |sed -n 's/^.*Address:.*0x/0x/p' got rid of the "? " in 0xE0000.txt
A big thanks to everyone!
You've got a nonprinting char in there. Try:
dmidecode |grep -i Address|sed 's/Address://'| od -c
to see exactly what you're getting.
UPDATE: comments indicate there's a tab char in there that needs to be cleaned out.
UPDATE 2: the leading tab is before the word Address. Try:
name=$(dmidecode |grep -i Address|sed 's/^.*Address:.*0x/0x/')
or as #just_somebody points out:
name=$(dmidecode|sed -n 's/^.*Address:.*0x/0x/p')
UPDATE 3
This changes the substitution regex to replace
^ (start of line) followed by .* (any characters (including tab!)) followed by Address: followed by .* (any characters (including space!)) followed by 0x (which are always at the beginning of the address since it's in hex)
with
0x (because you want that as part of the result)
If you want to learn more, read about sed regular expressions and substitutions.
EDIT (Side Question)
Can someone please explain what this line does?
eval website=\${$#}
The script reads a lot of paremeters, it's called somewhat like this
./script.sh -t 30 -n 100 -a test http://www.google.com
I have trouble reading the url ( http://www.google.com )
I am opening firefox using urls passed to a bash script. How do I encode them? Some of these urls are causing issue.
Some code
eval website=\${$#} // takes as argument
firefox -width 1280 -height 8000 ${website} &
Problematic URL
http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/1$41912
In firefox, it opens as
http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/141912
$ sign gets removed
The easiest way is probably to escape the characters that cause some problems.
Unless your url contain some unusual characters as ', or \, you should be fine just by putting your url between tow ':
$ firefox 'YOUR_URL'
This will prevent YOUR_URL content to be evaluated.
Edit, to reflect updated answer:
You can see using echo command how bash expands your parameters.
In your example, bash thinks $ is used to identify a variable (a variable named 4), thus it substitutes $4 with the value of variable 4, which is not defined (thus just removes $4):
$ echo http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/1$41912
http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/11912
$ echo 'http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/1$41912'
http://www.airportbusiness.com//print/Airport-Business-Magazine/Expo-Returns-to-Vegas/1$41912
Never use eval, and always quote your variables. The first argument ist stored in the parameter 1:
firefox "$1" &
This line:
eval website=\${$#}
sets the variable to the last positional parameter, regardless of how many there are.
Change it to:
website=${#: -1}
which is a Bashism, by the way.
Here are a few other Bashisms that accomplish the same thing:
echo "${!#}"
echo "${#:$#}"
echo "${BASH_ARGV[0]}"