Extracting zip file and then cd into it with different filename - linux

I am creating a bash script to extract a tar file and cd'ing into it and then it runs another script. So far this has been working pretty well with my code below, however, i ran into a case where if the extracted folder is different than the .tar file name then it would cause an issue. So my question is, how should I handle unique cases where the file name is different than then .tar filename.
e.g,) my_file.tar ---> after extraction ----> my_different_file_name
#!/bin/bash
fname=$1
echo the file you are about to extract is $fname
if [ -f $fname ]; then #if the file exists
tar -xvzf $fname #tar it
cd ${fname%.*} #the `%.*` will extract filename from filename.tgz and cd into it
echo ${fname%.*}
echo $PWD
loadIt #another script to load
fi

You could do a:
topDir=$(tar -xvzf $fname | sed "s|/.*$||" | uniq)
[ $(wc -w <<< $topDir) == 1 ] || exit 1
echo topDir=$topDir
Explanation: the first command untars vebosely (outputs all files it's untarring), and then gets all the leading directory names, and pipes them into uniq. (so basically it returns a list of all the top level directories in the tar file). The next line checks that there's exactly one entry in topDir, otherwise it exits.
At this point $topdir will be the directory you want to cd into.

Maybe you could do something like that:
cd $(tar -tf $fname | head -1)

If you don't mind moving the directory around after you extract it you can do something like this
# Create a temporary directory
$ tmpd=$(mktemp -d)
# Change to the temporary directory
$ pushd "$tmpd"
# Extract the tarball
$ tar -xf "$fname"
# Glob the directory name
$ d=(*)
# Error if we have more (or less) than one directory
$ [ "${#d}" = 0 ] || exit 1
# Explicitly use just the first directory (optional since `$d` does the same thing)
$ d=${d[0]}
# Move the extracted directory to the previous directory
$ mv "$d" "$OLDPWD"
# Change back to the starting directory
$ popd
# Remove the (now empty) temporary directory
$ rmdir "$tmpd"
# Change into the extracted directory
$ cd "$d"
# Run 'loadIt'
$ loadIt

Related

Shell script to copy one file at a time in a cron job

I have some csv files in location A like this
abc1.csv,
abc2.csv,
abc3.csv
I have a cron job which runs every 30 mins and in each execution I want to copy only 1 file(which shouldn't be repeated) and placed it in location B
I had though of 2 ways of doing this
1)I will pick the first file in the list of files and copy it to location B and will delete it once copied.Problem with this is I am not sure when the file will get copied completely and if i delete before its completed copied it can be an issue
2)I will have a temp folder.So i will copy the file from location A to location B and also keep it in temp location.In next iteration, when I pick the file from list of files I will compare its existence in the temp file location.If it exists I will move to next file .I think this will be more time consuming etc.
Please suggest if there is any other better way
you can use this bash script for your use case:
source="/path/to/.csv/directory"
dest="/path/to/destination/directory"
cd $source
for file in *.csv
do
if [ ! -f $dest/"$file" ]
then
cp -v $file $dest
break
fi
done
You can ensure you move the already copied file with:
cp abc1.csv destination/ && mv abc1.csv.done
(here you can make your logic to find only *.csv files, and not take into account *.done files.. that have been already processed by your script... or use any suffix you want..
if the cp does not succeed, nothing after that will get executed, so the file will not be moved.
You can also replace mv with rm to delete it:
cp abc1.csv destination/ && rm -f abc1.csv
Further more, you can add to the above commands error messages in case you want to be informed if the cp failed:
cp abc1.csv destination/ && mv abc1.csv.done || echo "copy of file abc1.csv failed"
And get informed via CRON/email output
Finally I took some idea from both the opted solution.Here is the final script
source="/path/to/.csv/directory"
dest="/path/to/destination/directory"
cd $source
for file in *.csv
do
if [ ! -f $dest/"$file" ]
then
cp -v $file $dest || echo "copy of file $file failed"
rm -f $file
break
fi
done

Linux mv command and then shutdown

I have this small bash script (simplified) which runs on Ubuntu 16.04:
tar zxvf fileNameHere.tgz <-- Untar tgz file in $SRC_DIR
files=$(ls $SRC_DIR)
echo "Extracting $files" >> $APP_LOG_DIR/update.log
mv $SRC_DIR/* $OUTPUT_DIR
shutdown -r now
I've noticed that, after rebooting, only sometimes files are not moved to target and I was wondering if that shutdown command could be the a problem. Is it necessary to call 'sync' before shutting down?
Fixed script with comments:
#!/usr/bin/env bash
# Test if SRC_DIR and OUTPUT_DIR are actual directories
if [ -d "$SRC_DIR" ] && [ -d "$OUTPUT_DIR" ]; then
# Populates the arguments array with the content
# of SRC_DIR rather than parsing the output of ls
set -- "$SRC_DIR/"*
# Prints joined file entries of the arguments array
# while stripping their leading directory path
printf 'Extracting %s\n' "${*#*/}" >> "$APP_LOG_DIR/update.log"
# Moves all the arguments array's entries (the actual
# content of the SRC_DIR) into OUTPUT_DIR
mv -- "$#" "$OUTPUT_DIR/"
shutdown -r now
fi

how to filter out / ignore specific lines when comparing text files with diff

To further clarify what I am trying to do, I wrote the script below. I am attempting to audit some files between my QA and PRD environments and would like the final Diff output to Ignore hard coded values such as sql connections. I have about 6 different values to filer. I have tried several ways thus far I am not able to get any of them to work as needed. I am open to doing this another way if anyone has any ideas. I am pretty new to script development so Im open to any ideas or information. Thanks :)
#!/bin/bash
#*********************************************************************
#
# Name: compareMD5.sh
# Date: 02/12/2018
# Script Location:
# Author: Maggie o
#
# Description: This script will pull absolute paths from a text file
# and compare the files via ssh between QA & PRD on md5sum
# output match or no match
# Then the file the non matching files will be imported to a
# tmp directory via scp
# Files will be compared locally and exclude whitespace,
# spaces, comments, and hard coded values
# NOTE: Script may take a several minutes to run
#
# Usage: Auditing QA to PRD Pass 3
# nohup ./compareMD52.sh > /output/compareMD52.out 2> /error/compareMD52.err
# checking run ps -ef | grep compareMD52*
#**********************************************************************
rm /output/no_matchMD5.txt
rm /output/filesDiffer.txt
echo "Filename | Path" > /output/matchingMD5.txt
#Remove everything below tmp directory recursivly as it was created by previous script run
rm -rf /tmp/*
for i in $(cat /input/comp_list.txt) #list of files with absolute paths output by compare script
do
export filename=$(basename "$i") #Grab just the filename
export path=$(dirname "$i") #Just the Directory
qa_md5sum=$(md5sum "$i") #Get the md5sum
qa_md5="${qa_md5sum%% *}" #remove the appended path
export tmpdir=(/tmp"$path")
# if the stat is not null then run the if, if file is exisiting
if ssh oracle#Someconnection stat $path'$filename' \> /dev/null 2\>\&1
then
prd_md5sum=$(ssh oracle#Somelocation "cd $path; find -name '$filename' -
exec md5sum {} \;")
prd_md5="${prd_md5sum%% *}" #remove the appended path
if [[ $qa_md5 == $prd_md5 ]] #Match hash as integer
then
echo $filename $path " QA Matches PRD">> /output/matchingMD5.txt
else
echo $i
echo $tmpdir
echo "Copying "$i" to "$tmpdir >> /output/no_matchMD5.txt
#Copy the file from PRD to a tmp Dir in QA, keep dir structure to avoid issues of same filename exisiting in diffrent directorys
mkdir -p $tmpdir # -p creates only if not exisiting, does not produce errors if exisiting
scp oracle#Somelocation:$i $tmpdir # get the file from Prd, Insert into tmp Directory
fi
fi
done
for x in $(cat /output/no_matchMD5.txt) #do a local comapare using diff
do
comp_filename=$(basename "$x")
#Ignore Comments, no white space, no blank lines, and only report if different but not How different
qa=(/tmp"$x")
#IN TEST
if diff -bBq -I '^#' $x $qa >/dev/null
# Fails to catch files if the Comment then the start of a line
then
echo $comp_filename " differs more then just white space, or
comment"
echo $x >> /output/filesDiffer.txt
fi
done
You can pipe the output into grep -v
Like this:
diff -bBq TEST.sh TEST2.sh | grep -v "^#"
I was able to get this figured out using this method
if diff -bBqZ -I '^#' <(grep -vE '(thing1|thing2|thing3)' $x) <(grep -vE '(thing1|thing2|thing3)' $prdfile)

Create .txt of all files in each subdirectory

I need to create a text file in each subdirectory of all files in the list.
For example, subdirectory1 would contain a list of all of its files as a .txt and subdirectory2 would also contain a list of all of subdirectory2 files as a .txt.
I have tried
#!/bin/bash
for X in "$directory" *
do
if [ -d "$X" ];
then
cd "$X"
files="$(ls)"
echo "$files" >> filesNames.txt
fi
done
However this did not generate anything. I absolutely need it as a shell script because it will be part of a pipeline script, but I cannot seem to get it to work.
Here is the adjusted script giving me the no such file or directory comment. I know that the folder exists and have used it in commands that are run before this command.
#!/bin/bash
#Retrieve the base directory path
baseDir=$(dirname "$ini")
#Retrieve the reference genome path
ref=$(dirname "$genome")
#Create required directory structure
tested="$baseDir/tested"
MarkDups1="$baseDir/MarkDups1"
#don't create if already exists
[[ -d "tested" ]] || mkdir "$tested"
[[ -d "MarkDups1" ]] || mkdir "$MarkDups1"
#create a text file with all sorted and indexed bam files paths
#!/bin/bash
for x in $MarkDups1/*/;
do
(cd "$x"; ls > filesNames.txt)
done
The sequence to iterate over should be "$directory"/*/.
for x in "$directory"/*/; do
(cd "$x"
files=(*)
printf '%s\n' "${files[#]}" > filesNames.txt
)
done

Bash: move file/directory and create a link of it

I am trying to make a bash script that moves a file or directory from source directory to destination directory and puts a symlink to it into source directory.
So, <source_path> can be a file or directory, <destination_dir_path> is the directory where I want the original moved to.
Sample usage:
$ mvln /source_dir/file.txt /destination_dir/
OR
$ mvln /source_dir/dir_I_want_to_move/ /destination_dir/
This is what I have managed to put together, but it does not work properly.
It works only if source is a directory, otherwise mv returns an error:
mv: unable to rename `/source_dir/some_file.txt': Not a directory
And the directory is not moved into destination_directory but only its contents are moved.
#!/bin/bash
SCRIPT_NAME='mvln'
USAGE_STRING='usage: '$SCRIPT_NAME' <source_path> <destination_dir_path>'
# Show usage and exit with status
show_usage_and_exit () {
echo $USAGE_STRING
exit 1
}
# ERROR file does not exist
no_file () {
echo $SCRIPT_NAME': '$1': No such file or directory'
exit 2
}
# Check syntax
if [ $# -ne 2 ]; then
show_usage_and_exit
fi
# Check file existence
if [ ! -e "$1" ]; then
no_file $1
fi
# Get paths
source_path=$1
destination_path=$2
# Check that destination ends with a slash
[[ $destination_path != */ ]] && destination_path="$destination_path"/
# Move source
mv "$source_path" "$destination_path"
# Get original path
original_path=$destination_path$(basename $source_path)
# Create symlink in source dir
ln -s "$original_path" "${source_path%/}"
Can some one please help?
The problem is that $destination_path refers to a directory that doesn't exist. Something like this:
mv /path/to/file.txt /path/to/non/existent/directory/
returns an error, and
mv /path/to/directory/ /path/to/non/existent/directory/
will rename /path/to/directory/ to /path/to/non/existent/directory/ (provided that /path/to/non/existent/ is an existent directory, just without a subfolder named directory).
If you are expecting that $destination_path doesn't already exist, then you can add a mkdir command:
mkdir "$destination_path"
mv "$source_path" "$destination_path"
if you're expecting that it might not exist, then you can add it conditionally:
[[ -d "$destination_path" ]] || mkdir "$destination_path"
mv "$source_path" "$destination_path"
and if you're expecting that it does exist, then you have some debugging to do!
(By the way, depending on your exact situation, you might find mkdir -p to be helpful. It recursively creates a directory and all necessary parent directories, and it doesn't mind if the directory already exists.)

Resources