I have below script which will append the file awslogs.conf on search of folder names under directory but if I re-execute this script, this appends again. How can I avoid doing this?
for NAME in $(ls -1p /home/ec2-user/src/ |grep -v "^_" | grep -v "/$" |cut -d. -f1);
do
sudo tee -a /etc/awslogs/awslogs.conf << END
[$NAME]
datetime_format = %b %d %H:%M:%S
file = /var/log/airflow/$NAME/*/*/*.log
buffer_duration = 5000
log_stream_name = $NAME
initial_position = start_of_file
log_group_name = CLOUDWATCH_LOGS
END
done
sudo service awslogsd start
sudo systemctl enable awslogsd
The tee command by default replaces the content of the file. -a overrides that to append instead. If you want to replace the content simply remove -a from the command.
If you want to append the content only if the file does not already contain it, that's an entirely different proposition. You could for example pull out the contents as a multi-line string and check for the entire thing using GNU grep:
contents="[$NAME]
datetime_format …"
if ! grep --fixed-strings --null --quiet "$contents" /etc/awslogs/awslogs.conf
then
tee -a …
Related
I have a folder structure like the following:
2020-123-1
2020-123-2
2020-123-3
2020-124-1
2020-124-2
...
I need to create folders from the first 2 numbers and omit whatever follows the second dash (-). Then I need to put the prior folders under the newly created ones with the correct name.
2020-123
->2020-123-1
->2020-123-2
->2020-123-3
2020-124
->2020-124-1
->2020-124-2
I tried to write a script in bash like this:
ls -d */ > folder.txt
cut -f1,2 -d"-" folder.txt |cut -f1 -d"/" |sort|uniq > mainfolder.txt
while read line; do mkdir $line ; done < mainfolder.txt
while read line; do mv $(cut -f1,2 -d"-" $line) $line/ ; done < folder.txt
I couldn't make the last line work, I know it has issues.
Actually, you don't have to parse the directory names and build the hierarchy. You can make use of the -p option of mkdir, thus, an awk one-liner will do the job:
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' dir.txt
The output with your example:
mkdir -p 2020-123; mv 2020-123-1 2020-123
mkdir -p 2020-123; mv 2020-123-2 2020-123
mkdir -p 2020-123; mv 2020-123-3 2020-123
mkdir -p 2020-124; mv 2020-124-1 2020-124
mkdir -p 2020-124; mv 2020-124-2 2020-124
Note
This one-liner just print the commands without executing them, you just pipe the output to |sh if everything looks fine. Examine the output commands, change the printf format/values for adjustment.
I didn't quote the filenames, since your example doesn't contain any special chars. Do it if it is in the case.
So the final script is as follows:
ls -d */ | cut -f1 -d"/" > folder.txt
awk -F'-' '{top=$1 FS $2;printf "mkdir -p %s; mv %s %s\n",top, $0, top}' folder.txt |sh
In pure bash:
#!/bin/bash
for src in *-*-*; do
destdir=${src%-*}
[[ -d $destdir ]] || mkdir "$destdir" || exit
# This just prints out the command that will be called.
# Remove the "echo" in actual script after making sure it will run as intented
echo mv "$src" "$destdir"
done
In the script above it is assumed that each file name to be moved contains exactly two dashes. If it can contain two or more dashes then the destdir=${src%-*} line should be replaced with these two lines:
suffix=${src#*-*-}
destdir=${src%"-$suffix"}
For detailed information read the "shell parameter expansion" section in bash reference.
Additionally, a good read article is: Why you shouldn't parse the output of ls
I'm trying to find all the init scripts created for websphere.
I know all the scripts end up with -init, so the first part of the code is:
find /etc/rc.d/init.d -name "*-init"
Also, I need all the script that run on an specific path, so the second part would be
| grep -i "/opt/ibm"
Finally, I need help with the last part. I have found the scripts I need to run them with the stop argument.
find /etc/rc.d/init.d -name "*-init" | grep -i "/opt/ibm" | <<run script found with stop argument>>
How can I run the command found with find?
Use a loop so that we are a little more careful while executing them:
#!/bin/bash
shopt -s globstar
for file in /etc/rc.d/init.d/**/*-init; do # grab all -init scripts
script=$(readlink -f "$file") # grab the actual file in case of a symlink
[[ -f $script ]] || continue # skip if not a regular file
[[ $file = */opt/ibm/* ]] || continue # not "/opt/ibm/", skip
printf '%s\n' "Executing script '$script'"
"$script" stop; exit_code=$?
printf '%s\n' "Script '$script' finished with exit_code $exit_code"
done
If you omit the 'find' and use grep directly you could do something like this:
grep -i "/opt/ibm" /etc/rc.d/init.d/* | sed 's/:.*/ stop/g' | sort -u | bash
it uses grep directly, which adds the filename to the output: filename:matched line
since you only need the filename and not the match, use sed to replace the ':' and the rest of the line with ' stop' (see the space before stop)
use sort -u (make sure, to execute each script only once)
Pipe the result into a shell
I have use command to get response time :
curl -s -w "%{time_total}\n" -o /dev/null https://www.ziffi.com/suggestadoc/js/ds.ziffi.https.v308.js
and I also need download time of this below mentioned js file link so used wget command to download this file but i get multiple parameter out put. I just need download time from it
$ wget --output-document=/dev/null https://www.ziffi.com/suggestadoc/js/ds.ziffi.https.v307.js
please suggest
I think what you are looking for is this:
wget --output-document=/dev/null https://www.ziffi.com/suggestadoc/js/ds.ziffi.https.v307.js 2>&1 >/dev/null | grep = | awk '{print $5}' | sed 's/^.*\=//'
Explanation:
2>&1 >/dev/null | --> Makes sure stderr gets piped instead of stdout
grep = --> select the line that contains the '=' symbol
sed 's/^.*\=//' --> deletes everything from linestart to the = symbol
In order to identify if my installation has errors that I should notice, I am using grep command on the file and write the file using tee because I need to elevate permissions.
sudo grep -inw ${LOGFOLDER}/$1.log -e "failed" | sudo tee -a ${LOGFOLDER}/$1.errors.log
sudo grep -inw ${LOGFOLDER}/$1.log -e "error" | sudo tee -a ${LOGFOLDER}/$1.errors.log
The thing is that the file is created even if the grep didn't find anything.
Is there any way I can create the file only if the grep found a match ?
Thanks
You may replace tee with awk, it won't create file if there is nothing to write to it:
... | sudo awk "{print; print \$0 >> \"errors.log\";}"
But such feature of awk is rarely used. I'd rather remove empty error file if nothing is found:
test -s error.log || rm -f error.log
And, by the way, you may grep for multiple words simultaneously:
grep -E 'failed|error' ...
I believe this is a simple syntax issue on my part but I have been unable to find another example similar to what i'm trying to do. I have a variable taking in a specific disk location and I need to use that location in an hdparm /grep command to pull out the max LBA
targetDrive=$1 #/dev/sdb
maxLBA=$(hdparm -I /dev/sdb |grep LBA48 |grep -P -o '(?<=:\s)[^\s]*') #this works perfect
maxLBA=$(hdparm -I $1 |grep LBA48 |grep -P -o '(?<=:\s)[^\s]*') #this fails
I have also tried
maxLBA=$(hdparm -I 1 |grep LBA48 |grep -P -o '(?<=:\s)[^\s]*')
maxLBA=$(hdparm -I "$1" |grep LBA48 |grep -P -o '(?<=:\s)[^\s]*')
Thanks for the help
So I think here is the solution to your problem. I did basically the same as you but changed the way I pipe the results into one another.
grep with regular expression to find the line containing LBA48
cut to retrieve the second field when the resulting string is divided by the column ":"
then trim all the leasding spaces from the result
Here is my resulting bash script.
#!/bin/bash
target_drive=$1
max_lba=$(sudo hdparm -I "$target_drive" | grep -P -o ".+LBA48.+:.+(\d+)" | cut -d: -f2 | tr -d ' ')
echo "Drive: $target_drive MAX LBA48: $max_lba"