I read some data out of a file, used grep for the only two columns needed, and redirected the output into a variable.
My script looks like this:
#!/bin/bash
cat hosts.cfg | grep 'address\|host_name' | sed -e 's/\<address\>//g' | while read line; do
echo $line | sed 's/host_name//g' | sed -r 's/\s+//g' ;
done
The output looks something like this now:
Host1
xx.xx.xx.xx
Host2
xx.xx.xx.xx
The problem is that hosts and ips must be saved into an array, not a file!
Output must look like this:
Host1(tab)xx.xx.xx.xx
Host2(tab)xx.xx.xx.xx
You are looking for process substitution. $(command), or old-style in `s.
(sorry, the description of how it should work is not clear enough for me to show modified version of your code)
You can use awk:
echo $output | awk 'NR%2{printf $0"\t";next;}1'
To save any command output, wrap it in backticks, or the newer (but less backward compatible) $(command) style substitution. E.g.:
result=`echo $output | awk 'NR%2{printf $0"\t";next;}1'`
using sed 'N;s/\n/\t/g'
change
Host1
xx.xx.xx.xx
Host2
xx.xx.xx.xx
to
Host1(tab)xx.xx.xx.xx
Host2(tab)xx.xx.xx.xx
You can use "set" to get faster output
exg:
I have file best2,
# cat best2
Host1 xx.xx.xx.xx Host2 xx.xx.xx.xx
make a script called: tabcheck.sh
# cat tabcheck.sh
#!/bin/bash
out=$(cat best2)
set $out
echo -e "$1\t$2\n$3\t$4"
# ./tabcheck.sh
Host1 xx.xx.xx.xx
Host2 xx.xx.xx.xx
If you use shift command(Usually only nine command line arguments can be accessed using positional parameters. The shift command gives access to command line arguments greater than nine by shifting each of the arguments.) as well.
Thanks.
Related
I'm developping a script to search for patterns within scripts executed from CRON on a bunch of remote servers through SSH.
Script on client machine -- SSH --> Remote Servers CRON/Scripts
For now I can't get the correct output.
Script on client machine
#!/bin/bash
server_list=( '172.x.x.x' '172.x.x.y' '172.x.x.z' )
for s in ${server_list[#]}; do
ssh -i /home/user/.ssh/my_key.rsa user#${s} crontab -l | grep -v '^#\|^[[:space:]]*$' | cut -d ' ' -f 6- | awk '{print $1}' | grep -v '^$\|^echo\|^find\|^PATH\|^/usr/bin\|^/bin/' | xargs -0 grep -in 'server.tld\|10.x.x.x'
done
This only gives me the paths of scripts from crontab, not the matched lines and line number plus the first line is prefixed with "grep:" keyword (example below):
grep: /opt/directory/script1.sh
/opt/directory/script2.sh
/opt/directory/script3.sh
/opt/directory/script4.sh
How to get proper output, meaning the script path plus line number plus line of matching pattern?
Remote CRON examples
OO 6 * * * /opt/directory/script1.sh foo
30 6 * * * /opt/directory/script2.sh bar
Remote script content examples
1 ) This will match grep pattern
#!/bin/bash
ping -c 4 server.tld && echo "server.tld ($1)"
2 ) This won't match grep pattern
#!/bin/bash
ping -c 4 8.x.x.x && echo "8.x.x.x ($1)"
Without example input, it's really hard to see what your script is attempting to do. But the cron parsing could almost certainly be simplified tremendously by refactoring all of it into a single Awk script. Here is a quick stab, with obviously no way to test.
#!/bin/sh
# No longer using an array for no good reason, so /bin/sh will work
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
# no -0; use grep -E and properly quote literal dot
xargs grep -Ein 'server\.tld|10.x.x.x'
done
Your command would not output null-delimited data to xargs so probably the immediate problem was that xargs -0 would receive all the file names as a single file name which obviously does not exist, and you forgot to include the ": file not found" from the end of the error message.
The use of grep -E is a minor hack to enable a more modern regex syntax which is more similar to that in Awk, where you don't have to backslash the "or" pipe etc.
This script, like your original, runs grep on the local system where you run the SSH script. If you want to run the commands on the remote server, you will need to refactor to put the entire pipeline in single quotes or a here document:
for s in 172.x.x.x 172.x.x.y 172.x.x.z; do
ssh -i /home/user/.ssh/my_key.rsa "user#${s}" <<\________HERE
crontab -l |
awk '! /^#|^[[:space:]]*$/ && $6 !~ /^$|^(echo|find|PATH|\/usr\/bin|\/bin\/)/ { print $6 }' |
xargs grep -Ein 'server\.tld|10.x.x.x'
________HERE
done
The refactored script contains enough complexities in the quoting that you probably don't want to pass it as an argument to ssh, which requires you to figure out how to quote strings both locally and remotely. It's easier then to pass it as standard input, which obviously just gets transmitted verbatim.
If you get "Pseudo-terminal will not be allocated because stdin is not a terminal.", try using ssh -t. Sometimes you need to add multiple -t options to completely get rid of this message.
I need to generate a script from within a script but am having problems because some of the commands going into the new script are being interpreted rather than written to the new file. For example i want to create a file called start.sh in it I want to set a variable to the current IP address:
echo "localip=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')" > /start.sh
what gets written to the file is:
localip=192.168.1.78
But what i wanted was the following text in the new file:
localip=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')"
so that the IP is determined when the generated script is run.
What am i doing wrong ?
You're making this unnecessary hard. Use a heredoc with a quoted sigil to pass literal contents through without any kind of expansion:
cat >/start.sh <<'EOF'
localip=$(ip addr | grep 'state UP' -A2 | tail -n1 | awk '{print $2}' | cut -f1 -d'/')
EOF
Using <<'EOF' or <<\EOF, as opposed to just <<EOF, is essential; the latter will perform expansion just as your original code does.
If anything you're writing to start.sh needs to be based on current variables, by the way, be sure to use printf %q to safely escape their contents. For instance, to set your current $1, $2, etc. to be active during start.sh execution:
# open start.sh for output on FD 3
exec 3>/start.sh
# build a shell-escaped version of your argument list
printf -v argv_str '%q ' "$#"
# add to the file we previously opened a command to set the current arguments to that list
printf 'set -- %s\n' "$argv_str" >&3
# pass another variable through safely, just to be sure we demonstrate how:
printf 'foo=%q\n' "$foo" >&3
# ...go ahead and add your other contents...
cat >&3 <<'EOF'
# ...put constant parts of start.sh here, which can use $1, $2, etc.
EOF
# close the file
exec 3>&-
This is far more efficient than using >>/start.sh on every line that needs to append: Using exec 3>file and then >&3 only opens the file once, rather than opening it once per command that generates output.
I was just wondering how I would go about writing a shell script to separate proxy IPs from their ports.
The proxies are stored in this format
ip:port
ip:port
ip:port
How can I use a shell script to separate the IP on the left side of the colon from the Port on the right side, and put the IP and Port lists in separate .txt files with the same order? Is this even possible?
If the proxies are listed that way in a file, say proxy.txt, then all you need is cut:
cut -f1 -d: proxy.txt > proxy_ip.txt
cut -f2 -d: proxy.txt > proxy_port.txt
Try something like this:
#!/bin/bash
ips="1.2.3.4:123 2.3.4.5:356 4.5.6.7:576"
# or get IPs from stdin
# split them
ips_array=($ips)
for w in ${ips_array[#]}
do
echo $w | sed -e 's/:.*$//g' >> ips.txt
echo $w | sed -e 's/^.*://g' >> ports.txt
done
Key is using the ($ips) to split the list up.
EDIT:
I just realized that you didn't format your question correctly so it's not a single line with IP:PORTs separated by spaces, but one on a line by itself. You just need this then:
#!/bin/bash
while read w
do
echo $w | sed -e 's/:.*$//g' >> ips.txt
echo $w | sed -e 's/^.*://g' >> ports.txt
done
And you read from stdin.
I have a file named email.txt like these one :
Subject:My test
From:my email <myemail#gmail.com>
this is third test
I want to take out only the email address in this file by using bash script.So i put this script in my bash script named myscript:
#!/bin/bash
file=$(myscript)
var1=$(awk 'NR==2' $file)
var2=$("$var1" | (grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b'))
echo $var2
But I failed to run this script.When I run this command manually in bash i can obtain the email address:
echo $var1 | grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b'
I need to put the email address to store in a variable so i can use it in other function.Can someone show me how to solve this problem?
Thanks.
I think this is an overly complicated way to go about things, but if you just want to get your script to work, try this:
#!/bin/bash
file="email.txt"
var1=$(awk 'NR==2' $file)
var2=$(echo "$var1" | grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b')
echo $var2
I'm not sure what file=$(myscript) was supposed to do, but on the next line you want a file name as argument to awk, so you should just assign email.txt as a string value to file, not execute a command called myscript. $var1 isn't a command (it's just a line from your text file), so you have to echo it to give grep anything useful to work with. The additional parentheses around grep are redundant.
What is happening is this:
var2=$("$var1" | (grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b'))
^^^^^^^ Execute the program named (what is in variable var1).
You need to do something like this:
var2=$(echo "$var1" | grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b')
or even
var2=$(awk 'NR==2' $file | grep -Eio '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b')
There are very helpful flags for bash: -xv
The line with
var2=$("$var1" | (grep...
should be
var2=$(echo "$var1" | (grep...
Also my version of grep doesn't have -o flag.
And, as far as grep patterns are "greedy" even as the following code runs, it's output is not exactly what you want.
#!/bin/bash -xv
file=test.txt
var1=$(awk 'NR==2' $file)
var2=$(echo "$var1" | (grep -Ei '\b[A-Z0-9._%+-]+#[A-Z0-9.-]+.[A-Z]{2,4}\b'))
echo $var2
Use Bash parameter expansion,
var2="${var1#*:}"
There's a cruder way:
cat $file | grep # | tr '<>' '\012\012' | grep #
That is, extract the line(s) with # signs, turn the angle brackets into newlines, then grep again for anything left with an # sign.
Refine as needed...
I am trying to write a script that uses a user-created MAC address when using the wlan0 interface.
To test, I made a file called testFile.txt, which is a copy of /etc/network/interfaces. If there are interfaces after wlan0, I cannot use echo "$var" >> testFile.txt because that simply adds the text to the end.
I am able to find the end of the wlan0 interface text, but I am not sure how to insert there. Below is what I currently have:
#!/bin/bash
echo "Enter MAC Address"
read var
log=$(cat testFile.txt | grep -o "wlan0.*" | grep -o dhcp)
echo $log
echo $log prints dhcp.
I tried adding | echo "hwaddress ether $var" >> testFile.txt to $log but that still appends to the end of the file.
How do I insert directly after $log?
The Problem
Useless use of cat. Don't do that.
You're not using the right tool for the job. Use GNU sed to make life easier on yourself.
The Solution
# Insert hwaddress line into /etc/network/interfaces.
read -p "Enter MAC Address: "
sudo sed -i.bak "/iface wlan0/a\ hwaddress $REPLY" /etc/network/interfaces
# Replace existing hwaddress line in /etc/network/interfaces.
read -p "Enter MAC Address: "
sudo sed -ri.bak "s/(hwaddress).*/\1 $REPLY/"
You may want to use sed to insert the MAC address. If you want to insert it at the end of the line with wlan0, that would be
sed "s/\(wlan0.*\)/\1 $var/" testFile.txt > testFile.txt.tmp
If that works, then
mv testFile.txt.tmp testFile.txt
you need a different tool, SED for example
$ seq 5 | sed '/3/ s/.*/something else/'
1
2
something else
4
5
for example you can Substitute the line with '3' by 'something else' - and there is no limits