What I want to do is something like below. I have a URL, say http://www.google.com/one/two/three
I need to extract the main domain name "www.google.com", to feed it to nslookup (As nslookup/dig does not seem to work with full URL) and then replace the URL with resolved IP address.e.g.
$ echo "http://www.google.com/one/two/three" | sed "s/<pattern>//g"
$ www.google.com
The problem is that "http://" may not always be there. And then
$ echo "http://www.google.com/one/two/three" | sed "s/<pattern>//g"
$ http://11.22.33.44/one/two/three
Can anyone provide any related link or related examples ?
Try this sed command:
echo "http://www.google.com/one/two/three" | sed -r 's#(https?://)?([^/]+).*#\2#'
OUTPUT:
www.google.com
And when you have fetched IP address:
$> IP="11.22.33.44"
$> echo "https://www.google.com/one/two/three" | sed -r "s#(https?://)?([^/]+)(.*)#\1$IP\3#"
https://11.22.33.44/one/two/three
This will work with http://, https:// or without any https?:// before the URL.
Since URL's can be messy ("http://user:pass#www.example.com:8080/one/two/three") in the general case I'd recommend using a language with a URI parsing library. For example
url="http://stackoverflow.com/questions/18087200/replace-part-of-url-with-ip-adress"
newurl=$(perl -MURI -le '
$uri = URI->new(shift);
$cmd = "nslookup " . $uri->host;
$out = qx($cmd);
if ($out =~ /Name:.*?\KAddress:\s+([\d.]+)/s) {
$ip = $1;
$uri->host($ip);
print $uri
} else {warn "cannot find IP address in nslookup output"}
' "$url")
echo "$newurl"
outputs
http://198.252.206.16/questions/18087200/replace-part-of-url-with-ip-adress
Related
Trying to execute remotely a bunch of commands in a perl script
This looks like that :
$CMD1 = "/usr/sbin/mminfo -av -q \"savetime>'-1 day 18:00:00',savetime<'17:59:59'\" -r \"ssid,totalsize,nfiles,pool\"|grep \"xxxxx\"|/usr/bin/awk '!seen[\$1]++'";
print Dumper $CMD1;
$CMD = "/usr/bin/ssh xxxx\#$SRV \'$CMD1\' 2>&1";
print Dumper $CMD;
But I still have problem with the $1 in the awk command, It seems to be cancelled when running.
What I can see :
$VAR1 = '/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxxx"|/usr/bin/awk \'!seen[$1]++\'';
$VAR1 = '/usr/bin/ssh xxxxx#\'xxxxxx\' \'/usr/sbin/mminfo -av -q "savetime>\'-1 day 18:00:00\',savetime<\'17:59:59\'" -r "ssid,totalsize,nfiles,pool"|grep "xxxxx"|/usr/bin/awk \'!seen[$1]++\'\' 2>&1';
So the '$1' of the awk command is passed correctly to the remote but when running :
#RESU = `$CMD`;
print Dumper #RESU;
I can see that my $1 is missing (or interpretated by the remote shell as a null value) :
$VAR1 = 'awk: ligne de commande:1: !seen[]++
';
$VAR2 = 'awk: ligne de commande:1: ^ syntax error
';
$VAR3 = 'awk: ligne de commande:1: error: expression indice non valide
';
I've tried many things like quoting or double-quoting the string, creating the string with perl 'qq' function, putting value of $CMD1 directly in $CMD and escaping quotes but no way.
And of course, my awk is piped to another awk (not provided here).
I don't want a solution which runs awk localy since I've millions lines returned from the 'mminfo' command.
Any clue (or a better way to do that !) ?
You might want to break it into smaller pieces for readability, and use the multi-arg invocation of system to avoid perl having to spawn a shell. The q() function goes a long way toward avoiding quoting hell.
$mminfo = q{/usr/sbin/mminfo -av -q "savetime>'-1 day 18:00:00',savetime<'17:59:59'" -r "ssid,totalsize,nfiles,pool"};
$awk = q{/usr/bin/awk '/xxxxx/ && !seen[$1]++');
print Dumper [$mminfo, $awk];
#cmd = ( "/usr/bin/ssh", "xxxx\#$SRV", "$mminfo | $awk" );
print Dumper \#cmd;
system #cmd;
Even if you can not use modules in your final environment, you may be able to use them in your local machine. In that case you can use them to quote the command programmatically and then just copy and paste the quoted string into the script you are developing. For instance:
use strict;
use warnings;
use Net::OpenSSH;
my $quoted_cmd1 = Net::OpenSSH->shell_quote('/usr/sbin/mminfo', '-av',
-q => q(savetime>'-1 day 18:00:00',savetime<'17:59:59'),
-r => 'ssid,totalsize,nfiles,pool',
\\'|',
'grep', 'xxxxx',
\\'|',
'/usr/bin/awk', '!seen[$1]++');
my $SRV = "foo";
my $quoted_cmd = Net::OpenSSH->shell_quote('/usr/bin/ssh', "xxxx\#$SRV",
$quoted_cmd1,
\\'2>&1');
print "$quoted_cmd\n";
Which outputs...
/usr/bin/ssh xxxx#foo '/usr/sbin/mminfo -av -q '\''savetime>'\''\'"''"'-1 day 18:00:00'\''\'"''"',savetime<'\''\'\''17:59:59\'\'' -r ssid,totalsize,nfiles,pool | grep xxxxx | /usr/bin/awk '\''!seen[$1]++'\' 2>&1
I am working on a bash script and I got a list of IP's that I wanted to add one by one in a CURL command.
For example given list on a file named list.txt
8.8.8.8
10.10.10.10
136.34.24.22
192.168.10.32
I wanted to add each value on curl command
curl -k -u $user:$password "https://logservice/jobs" --data-urlencode 'search=search index=test $ARRAYVALUE | head 1' > output.txt
Where $ARRAYVALUE is the IP address to be used on the command.
I will appreciate any hint.
Thanks
If I understood correctly, you want to:
map each line of a "list.txt" to an item of an array
loop over the newly created array inserting items one by one into your command invocation
Consider this, heavily commented, snippet. Look especially at mapfile and how variable is used in curl invocation, surrounded by double quotes.
#!/bin/bash
# declare a (non-associative) array
# each item is indexed numerically, starting from 0
declare -a ips
#put proper values here
user="userName"
password="password"
# put file into array, one line per array item
mapfile -t ips < list.txt
# counter used to access items with given index in an array
ii=0
# ${#ips[#]} returns array length
# -lt makes "less than" check
# while loops as long as condition is true
while [ ${ii} -lt ${#ips[#]} ] ; do
# ${ips[$ii]} accesses array item with the given (${ii}) index
# be sure to use __double__ quotes around variable, otherwise it will not be expanded (value will not be inserted) but treated as a string
curl -k -u $user:$password "https://logservice/jobs" --data-urlencode "search=search index=test ${ips[$ii]} | head -1" > output.txt
# increase counter to avoid infinite loop
# and access the next item in an array
((ii++))
done
You may read about mapfile in GNU Bash reference: Built-ins.
You may read about creating and accessing arrays in GNU Bash reference: Arrays
Check this great post about quotes in bash.
I hope you found this answer helpful.
I believe you need something like this :
#!/bin/bash
function FN()
{
filename=$1
declare -a IPs_ARRAY
i=0
user=$2
password=$3
while read ip
do
IPs_ARRAY[$i]=$ip
echo ${IPs_ARRAY[$i]}
# Uncomment for your actions ::
#curl -k -u $user:$password "https://logservice/jobs" --data-urlencode 'search=search index=test ${IPs_ARRAY[$i]} | head 1' > output.txt
(( i++ ))
done < $filename
}
#############
### MAIN ###
###########
read -p "Enter username: " username
read -p "Enter password: " password
# Call your function
filename="list.txt"
FN $filename $username $password
(* In the following post, all IP's, Ports and Passwords have been changed. Sorry about the formatting of this post, the editor doesn't seem to like new lines.)
Question: How do I store integers as signed 32bit little endian?
Background: Im attempting to use RCon to connect to a minecraft server in bash. So far the server shows the connection is being received but I can't get the packet formatted correctly. I can connect to the server using mcrcon and see the packets in wireshark but when I attempt using my bash script, the packet length, requestid and type values look wrong.
The following is some of my sources, trouble shooting data and my code which may help in answering the question.
Source: https://wiki.vg/RCON
Implementation of: https://developer.valvesoftware.com/wiki/Source_RCON_Protocol
Server console:
[22:24:09 WARN]: Can't keep up! Is the server overloaded? Running 3190ms or 63 ticks behind
[22:24:23 INFO]: Rcon connection from: /164.256.8.10
[22:24:34 WARN]: Can't keep up! Is the server overloaded? Running 9961ms or 199 ticks behind
[22:24:55 WARN]: Can't keep up! Is the server overloaded? Running 2006ms or 40 ticks behind
[22:25:12 INFO]: Rcon connection from: /164.256.8.10
.
Wireshark: (mcrcon data)
Code:
#!/bin/bash
# Length int Length of remainder of packet
# Request ID int Client-generated ID
# Type int 3 for login, 2 to run a command, 0 for a multi-packet response
# Payload byte[] ASCII text
# 2-byte pad byte, byte Two null bytes
# Connection details
RCON_HEADER=$(echo -e "\xff\xff\xff\xff")
HOST="192.168.0.173"
PORT=12345
LENGTH=0 # Length of packet
REQUESTID=$RANDOM
PASSWORD="$1"
RES=0
COM=2
AUTH=3
NULL="\0"
COMMAND=${#:2}
echo "command: $COMMAND"
## Packet Format as per docs
#Packet Size in Bytes
#Request ID any int
#Type as above
#Body null terminated ascii string
#Empty string null terminated
build_packet()
{
local TYPE="$1";
$([ "$TYPE" == "$AUTH" ]) && local BODY="$PASSWORD" || local BODY=$COMMAND;
local DATA="$REQUESTID$TYPE$BODY";
local LENGTH=${#DATA};
local PACKET="$LENGTH$DATA";
echo $PACKET;
}
send()
{
#local PACKET="$1"
echo "sending: $PACKET"
printf "$PACKET%s\0%s\0" >&5 &
}
read ()
{
LENGTH="$1"
RETURN=`dd bs=$1 count=1 <&5 2> /dev/null`
}
echo "trying to open socket"
# try to connect
if ! exec 5<> /dev/tcp/$HOST/$PORT; then
echo "`basename $0`: unable to connect to $HOST:$PORT"
exit 1
fi
echo "socket is open"
PACKET=$(build_packet $AUTH $PASSWORD);
echo "Command: $COMMAND"
echo "Packet: $PACKET"
send $PACKET
read 7
echo "RETURN: $RETURN"
PACKET=$(build_packet $COM $COMMAND);
echo "Command: $COMMAND"
echo "Packet: $PACKET"
send $PACKET
read 7
echo "RETURN: $RETURN"
.
Referenced Code: https://blog.chris007.de/using-bash-for-network-socket-operation/
The solution was to hex encode and decode as follows:
build_packet()
{
local TYPE="$1"
local BODY=""
$([ "$TYPE" == "$AUTH" ]) && BODY="$PASSWORD" || BODY="$COMMAND"
# Add null chars "\0" and hex encode to preserve them "xxd -p".
# David C Rankin also mentioned, you could possibly printf them in using flags.
local DATA=`echo -ne "$ID\0\0\0\0$TYPE\0\0\0$BODY\0\0" | xxd -p | tr -d '\n'`
local LEN="${#DATA}"
LEN=$((LEN / 2))
LEN="$(asc $LEN)"
LEN=`echo -ne "$LEN\0\0\0" | xxd -p | tr -d '\n'`
local PACKET="$LEN$DATA"
echo $PACKET
}
Notes: # (asc) is a custom function that converts LEN to ascii char. I have no idea why the packet length has to be an ascii char but the type, which is also an integer does not. Perhaps someone else can answer that.
At transmission time the hex is decoded "echo -ne "$PACKET" | xxd -r -p >&5 &".
Pure bash
There is a bash way, for creating a binary packet by some build_pkt function.
build_pkt function
But as you could not store null character (\0) is bash strings, you have to use printf to develop your string:
declare -i REQUESTID=1
build_pkt() {
local pkt='' len
case $1 in
AUTH ) local TYPE=3 BODY="$PASSWORD" ;;
CMD ) local TYPE=2 BODY="${#:2}" ;;
* ) return 1;;
esac
addtopkt() {
local i v c
printf -v v "%08x" $1
for i in {3..0..-1};do
printf -v c %03o 0x${v:2*i:2}
pkt+=\\$c
done
}
len=$(( ${#BODY} >0 ? ${#BODY}+10 : 10 ))
addtopkt $len
REQUESTID+=1
addtopkt $REQUESTID
addtopkt $TYPE
pkt+="$BODY\0\0"
printf "$pkt"
}
TCP connection
As my version of bash don't recognize /dev/tcp/, I use netcat (1 fork to background task) for this:
TMPFIFO=/tmp/.mcrcon-$$
mkfifo $TMPFIFO
exec 8> >(exec stdbuf -i 0 -o 0 nc $HOST $PORT >$TMPFIFO 2>&1)
exec 9<$TMPFIFO
rm $TMPFIFO
Then a quick'n dirty script to dump answer:
readrawanswer() {
local foo LANG=C
while IFS= read -u 9 -t .01 -r foo ;do
printf "%q\n" "$foo"
done
}
Let's go:
readrawanswer
build_pkt AUTH >&8
readrawanswer
I'm ready now to send help request to my spigot (minecraft) servr:
build_pkt CMD "help" >&8
readrawanswer
On my host, this will print:
$'\272\001\'\302\247e--------- \302\247fHelp: Index (1/10) \302\247e--------------------'
$'\302\2477Use /help [n] to get page n of help.'
$'\302\2476Aliases: \302\247fLists command aliases'
$'\302\2476Bukkit: \302\247fAll commands for Bukkit'
$'\302\2476Minecraft: \302\247fAll commands for Minecraft'
$'\302\2476/advancement: \302\247fA Mojang provided command.'
$'\302\2476/ban: \302\247fA Mojang provided command.'
$'\302\2476/ban-ip: \302\247fA Mojang provided command.'
$'\302\2476/banlist: \302\247fA Mojang provided command.'
$'\302\2476/bossbar: \302\247fA Mojang provided command.'
Then now, for 2nd page, I could:
build_pkt >&8 CMD "help 2" && readrawanswer
$'\307\001)\302\247e--------- \302\247fHelp: Index (2/10) \302\247e--------------------'
$'\302\2476/clear: \302\247fA Mojang provided command.'
$'\302\2476/clone: \302\247fA Mojang provided command.'
$'\302\2476/data: \302\247fA Mojang provided command.'
$'\302\2476/datapack: \302\247fA Mojang provided command.'
$'\302\2476/debug: \302\247fA Mojang provided command.'
$'\302\2476/defaultgamemode: \302\247fA Mojang provided command.'
$'\302\2476/deop: \302\247fA Mojang provided command.'
$'\302\2476/difficulty: \302\247fA Mojang provided command.'
$'\302\2476/effect: \302\247fA Mojang provided command.'
And finally, the whole help list, with pages 1 to 10:
for i in {1..10};do
build_pkt >&8 CMD "help $i" && readrawanswer
done |
sed 's/^.*[0-9]\/\([a-z:-]\+\): .*$/\1/p;d' |
xargs |
fold -s
On my host, this will render:
advancement ban ban-ip banlist bossbar clear clone data datapack debug
defaultgamemode deop difficulty effect enchant execute experience fill
forceload function gamemode gamerule give help kick kill list locate loot me
minecraft:help minecraft:reload msg op pardon pardon-ip particle playsound
plugins publish recipe reload replaceitem restart save-all save-off save-on say
schedule scoreboard seed setblock setidletimeout setworldspawn spawnpoint
spigot spreadplayers stop stopsound summon tag team teammsg teleport tell
tellraw time timings title tm tp tps trigger version w weather whitelist
worldborder xp
bash completion:
Once mcrcmd ready:
mcrcmd() {
build_pkt CMD "$#" 1>&8 && readrawanswer
}
Create a $mcrcmds array, holding all available commands:
mcrcmds=($(
for i in {1..10};do
build_pkt >&8 CMD "help $i" && readrawanswer
done |
sed 's/^.*[0-9]\/\([a-z:-]\+\): .*$/\1/p;d'
))
Then:
complete -W "${mcrcmds[*]}" mcrcmd
Ready.
Then now mcrcmd is a function with bash completion ;-)
We are logged into a bash session, connected to minecraft server, ready to interact with them, by sending commands and retrieving answers.
Thanks to the link:
Source RCON Protocol
Further, go to Classic server protocol for translating $'\302\247'[0-f] to terminal colors...
Let close nc:
exec 8>&- ; exec 9<&-
But, please consider doing this in perl, python if not c
I need that scipt enter in my router on page and find the code and compared with earlier this IP information recorded, and if not changed, then stop the script.
I run this script - $ perl ~/test.pl
no error, but file my_ip.txt is not created.
In originals script must check my ip through host example.dyndns.org, but my ip is gray.
So I need to be determined through a router
#!/usr/bin/perl
use LWP::UserAgent;
my $routeraddress = `addr admin:Tavolzhansky#192.168.1.1/RST_conn_status.htm`;
if ($routeraddress =~ /var info_get_wanip="((\d+\.){3}(\d+))"/) {
my $ip = "$1.$2.$3.$4";
#Добавлено:
open (FILE,"my_ip.txt");
my #lines = <FILE>;
$old_ip = $lines[0]; #Считываем IP из файла
$old_ip =~ s/^\s+|\s+$//g; #trim
close(FILE);
if ($old_ip eq $ip) {
die "IP not changed"; # Выходим из скрипта, если IP не изменился
}
open (FILE,">my_ip.txt");
print FILE $ip; # Записываем в файл новый IP
close(FILE);
...... (this code is OK)
}
Do not bring the end of the code because the problem by connecting to the router
$ perl -cw -e 'print `addr admin:foo#192.168.1.1/bar`'
Possible unintended interpolation of #192 in string at -e line 1.
-e syntax OK
It doesn't work because #192 is interpolated as an array, and you are running the command addr admin:Tavolzhansky.168.1.1/RST_conn_status.htm instead of the command you meant to.
This would have been very easy to spot and fix if you would use warnings, if you would step through the code with the debugger, or as Andy suggests, if you would examine return values.
To get #192 to not be interpolated, escape the #:
my $routeraddress = `addr admin:Tavolzhansky\#192.168.1.1/RST_conn_status.htm`;
It's not working because of this:
my $routeraddress = `addr admin:Tavolzhansky#192.168.1.1/RST_conn_status.htm`;
if ($routeraddress =~ /var info_get_wanip="((\d+\.){3}(\d+))"/) {
...
}
Try adding a couple debug prints to confirm like:
my $routeraddress = `addr admin:Tavolzhansky#192.168.1.1/RST_conn_status.htm`;
print "Router IP = $routeraddress/n";
if ($routeraddress =~ /var info_get_wanip="((\d+\.){3}(\d+))"/) {
...
} else {
print "Router address didn't match.\n";
}
You should use the strict and warning pragmas, but as you said this worked when you were getting your IP from a different source, the rest of the code will work.
I'm guessing you're reading an HTML page and expecting to see this string somewhere:
var info_get_wanip="xxx.xxx.xxx.xxx";
However, you need to confirm if $routeraddress is actually getting that, it looks like
my $routeraddress = `addr admin:Tavolzhansky#192.168.1.1/RST_conn_status.htm`;
Is just essentially making a get request for the page, but isn't returning "var info_get_wanip="xxx.xxx.xxx.xxx";". If you sort the assignment of the variable $routeraddress, then it'll work.
I can't say how, as I don't know what it's returning now.
Hello: I have a lot of files called test-MR3000-1.txt to test-MR4000-1.nt, where the number in the name changes by 100 (i.e. I have 11 files),
$ ls test-MR*
test-MR3000-1.nt test-MR3300-1.nt test-MR3600-1.nt test-MR3900-1.nt
test-MR3100-1.nt test-MR3400-1.nt test-MR3700-1.nt test-MR4000-1.nt
test-MR3200-1.nt test-MR3500-1.nt test-MR3800-1.nt
and also a file called resonancia.kumac which in a couple on lines contains the string XXXX.
$ head resonancia.kumac
close 0
hist/delete 0
vect/delete *
h/file 1 test-MRXXXX-1.nt
sigma MR=XXXX
I want to execute a bash file which substitutes the strig XXXX in a file by a set of numbers obtained from the command ls *MR* | cut -b 8-11.
I found a post in which there are some suggestions. I try my own code
for i in `ls *MR* | cut -b 8-11`; do
sed -e "s/XXXX/$i/" resonancia.kumac >> proof.kumac
done
however, in the substitution the numbers are surrounded by sigle qoutes (e.g. '3000').
Q: What should I do to avoid the single quote in the set of numbers? Thank you.
This is a reproducer for the environment described:
for ((i=3000; i<=4000; i+=100)); do
touch test-MR${i}-1.nt
done
cat >resonancia.kumac <<'EOF'
close 0
hist/delete 0
vect/delete *
h/file 1 test-MRXXXX-1.nt
sigma MR=XXXX
EOF
This is a script which will run inside that environment:
content="$(<resonancia.kumac)"
for f in *MR*; do
substring=${f:7:3}
echo "${content//XXXX/$substring}"
done >proof.kumac
...and the output looks like so:
close 0
hist/delete 0
vect/delete *
h/file 1 test-MR300-1.nt
sigma MR=300
There are no quotes anywhere in this output; the problem described is not reproduced.
or if it could be perl:
#!/usr/bin/perl
#ls = glob('*MR*');
open (FILE, 'resonancia.kumac') || die("not good\n");
#cont = <FILE>;
$f = shift(#ls);
$f =~ /test-MR([0-9]*)-1\.nt/;
$nr = $1;
#out = ();
foreach $l (#cont){
if($l =~ s/XXXX/$nr/){
$f = shift(#ls);
$f =~ /test-MR([0-9]*)-1\.nt/;
$nr = $1;
}
push #out, $l;
}
close FILE;
open FILE, '>resonancia.kumac' || die("not good\n");
print FILE #out;
That would replace the first XXXX with the first filename, what seemed to be the question before change.