How to extract text from web and add into another file in shell script - linux

I've one shell script where i mentioned as
#!/bin/sh
.........
curl -X POST -d text={varX} $WEB_LINK
.............
So I want to extract text of WEB_LINK and add into another file like xyz.txt. Could you please help me to do it.


Try this:
#!/bin/sh
text=$(curl -s $WEB_LINK)
echo $text > xyz.txt


Try following:
#!/bin/sh
cmd="curl -X POST -d text={varX} $WEB_LINK > xyz.txt"
bash -c "$cmd"

Related

How to pass a parameter to a downloaded script

I'm stuck passing a parameter(URL to download) to a script.
My goal is to create a script for deployment that downloads and installs an app.
The script I run:
curl url_GitHub | bash -s url_download_app
The script on GitHub:
#! /bin/sh
url="$2"
filename=$(basename "$url")
workpath=$(dirname $(readlink -f $0))
curl $url -o $workpath/$filename -s
sudo dpkg --install $workpath/$filename
As I understood it doesn't pass the URL to download the app to the URL="$2" variable.
If I run the GitHub script locally, and pass the URL to download the app, it executes successfully.
Smth like:
bash install.sh -s url_download_app
Please help=)

-s appears to be an option intended for the downloaded script. However, it is also an option accepted by bash, so what I think you want is
curl url_GitHub | bash -s -- -s url_download_app

As the script on GitHub use $2, we should pass it as second argument :
curl url_GitHub | bash -s _ url_download_app
_ url_download_app will be passed to the script on GitHub.

What about the following (using process substitution):
bash <(curl -Ss url_GitHub) url_download_app
I did a proof of concept with the following script:
$ cat /tmp/test.sh
#!/bin/bash
echo "I got '$1'"
exit 0
and when you run it you get:
$ bash <(cat /tmp/test.sh) "test input argument"
I got 'test input argument'

How do i make my bash script on download automatically turn into a terminal command? [duplicate]

Say I have a file at the URL http://mywebsite.example/myscript.txt that contains a script:
#!/bin/bash
echo "Hello, world!"
read -p "What is your name? " name
echo "Hello, ${name}!"
And I'd like to run this script without first saving it to a file. How do I do this?
Now, I've seen the syntax:
bash < <(curl -s http://mywebsite.example/myscript.txt)
But this doesn't seem to work like it would if I saved to a file and then executed. For example readline doesn't work, and the output is just:
$ bash < <(curl -s http://mywebsite.example/myscript.txt)
Hello, world!
Similarly, I've tried:
curl -s http://mywebsite.example/myscript.txt | bash -s --
With the same results.
Originally I had a solution like:
timestamp=`date +%Y%m%d%H%M%S`
curl -s http://mywebsite.example/myscript.txt -o /tmp/.myscript.${timestamp}.tmp
bash /tmp/.myscript.${timestamp}.tmp
rm -f /tmp/.myscript.${timestamp}.tmp
But this seems sloppy, and I'd like a more elegant solution.
I'm aware of the security issues regarding running a shell script from a URL, but let's ignore all of that for right now.

source <(curl -s http://mywebsite.example/myscript.txt)
ought to do it. Alternately, leave off the initial redirection on yours, which is redirecting standard input; bash takes a filename to execute just fine without redirection, and <(command) syntax provides a path.
bash <(curl -s http://mywebsite.example/myscript.txt)
It may be clearer if you look at the output of echo <(cat /dev/null)

This is the way to execute remote script with passing to it some arguments (arg1 arg2):
curl -s http://server/path/script.sh | bash /dev/stdin arg1 arg2

For bash, Bourne shell and fish:
curl -s http://server/path/script.sh | bash -s arg1 arg2
Flag "-s" makes shell read from stdin.

Use:
curl -s -L URL_TO_SCRIPT_HERE | bash
For example:
curl -s -L http://bitly/10hA8iC | bash

Using wget, which is usually part of default system installation:
bash <(wget -qO- http://mywebsite.example/myscript.txt)

You can also do this:
wget -O - https://raw.github.com/luismartingil/commands/master/101_remote2local_wireshark.sh | bash

The best way to do it is
curl http://domain/path/to/script.sh | bash -s arg1 arg2
which is a slight change of answer by #user77115

You can use curl and send it to bash like this:
bash <(curl -s http://mywebsite.example/myscript.txt)

I often using the following is enough
curl -s http://mywebsite.example/myscript.txt | sh
But in a old system( kernel2.4 ), it encounter problems, and do the following can solve it, I tried many others, only the following works
curl -s http://mywebsite.example/myscript.txt -o a.sh && sh a.sh && rm -f a.sh
Examples
$ curl -s someurl | sh
Starting to insert crontab
sh: _name}.sh: command not found
sh: line 208: syntax error near unexpected token `then'
sh: line 208: ` -eq 0 ]]; then'
$
The problem may cause by network slow, or bash version too old that can't handle network slow gracefully
However, the following solves the problem
$ curl -s someurl -o a.sh && sh a.sh && rm -f a.sh
Starting to insert crontab
Insert crontab entry is ok.
Insert crontab is done.
okay
$

Also:
curl -sL https://.... | sudo bash -

Just combining amra and user77115's answers:
wget -qO- https://raw.githubusercontent.com/lingtalfi/TheScientist/master/_bb_autoload/bbstart.sh | bash -s -- -v -v
It executes the bbstart.sh distant script passing it the -v -v options.

Is some unattended scripts I use the following command:
sh -c "$(curl -fsSL <URL>)"
I recommend to avoid executing scripts directly from URLs. You should be sure the URL is safe and check the content of the script before executing, you can use a SHA256 checksum to validate the file before executing.

instead of executing the script directly, first download it and then execute
SOURCE='https://gist.githubusercontent.com/cci-emciftci/123123/raw/123123/sample.sh'
curl $SOURCE -o ./my_sample.sh
chmod +x my_sample.sh
./my_sample.sh

This way is good and conventional:
17:04:59#itqx|~
qx>source <(curl -Ls http://192.168.80.154/cent74/just4Test) Lord Jesus Loves YOU
Remote script test...
Param size: 4
---------
17:19:31#node7|/var/www/html/cent74
arch>cat just4Test
echo Remote script test...
echo Param size: $#

If you want the script run using the current shell, regardless of what it is, use:
${SHELL:-sh} -c "$(wget -qO - http://mywebsite.example/myscript.txt)"
if you have wget, or:
${SHELL:-sh} -c "$(curl -Ls http://mywebsite.example/myscript.txt)"
if you have curl.
This command will still work if the script is interactive, i.e., it asks the user for input.
Note: OpenWRT has a wget clone but not curl, by default.

bash | curl http://your.url.here/script.txt
actual example:
juan#juan-MS-7808:~$ bash | curl https://raw.githubusercontent.com/JPHACKER2k18/markwe/master/testapp.sh
Oh, wow im alive
juan#juan-MS-7808:~$

Curl command doesn't work in bash script

I am trying to upload a JSON file into my noSQL database using a bash script, but it doesn't work and I don't understand why.
This is the script :
test='{"evaluation": "none"}'
test="'$test'"
command="curl -XPUT localhost:9200/test/evaluation/$i -d $test"
echo "$command"
$command
This is the error :
curl -XPUT localhost:9200/test/evaluation/0 -d '{"evaluation": "none"}'
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}curl: (3) [globbing] unmatched close brace/bracket in column 7
When I do the command given in my command line it works fine though.
What is the error here ? Thank you

Don't store a command in a variable; if you absolutely must have something usable with logging, put the arguments in an array.
test='{"evaluation": "none"}'
args=( -XPUT localhost9200/test/evaluation/"$i" -d "$test" )
echo "curl ${args[*]}"
curl "${args[#]}"

Linux bash script: share variable among terminal windows

If I do this:
#!/bin/bash
gnome-terminal --window-with-profile=KGDB -x bash -c 'VAR1=$(tty);
echo $VAR1; bash'
echo $VAR1
How can I get the last line from this script to work? I.e., be able to access the value of $VAR1 (stored on the new terminal window) from the original one? Currently, while the first echo is working, the last one only outputs an empty line.

The short version is that you can't share the variable. There's no shared channel for that.
You can write it to a file/pipe/etc. and then read from it though.
Something like the following should do what you want:
#!/bin/bash
if _file=$(mktemp -q); then
gnome-terminal --window-with-profile=KGDB -x bash -c 'VAR1=$(tty); echo "$VAR1"; declare -p VAR1 > '\'"$_file"\''; bash'
cat "$_file"
. "$_file"
echo "$VAR1"
fi

grep in bash script not working as expected

If I run
grep -i "echo" *
I get the results I want, but if I try the following simple bash script
#search.sh
grep -i "$1" *
echo "####--DONE--####"
and I run it with sh -x search.sh "echo" I get the following error output:
' grep -i echo '*
: No such file or directory
' echo '####--DONE--####
####--DONE--####
How come? I'm on CentOS

Add the sha-bang line at the top of your script
#!/bin/bash
and after making it executable, run the script using
./search.sh "echo"

The "sh -x" should print the files that '*' matches. It looks like it's not matching any files. Are you maybe running it in a directory with no readable files?

Resources