shell script to move data from other server's (or nodes) sub-dirs to current server(node) matching sub-dir - linux

Question

I've .parquet files for multiple dates (from 20190927 to 20200131) inside /data/pg/export/schema.table_YYYYMMDD<random alphanumric string> directory structure in seven different nodes. When process ran, it created sub-directory in schema.table_YYYYMMDD<random alphanumric string> format (such as schema.table_20190927) inside /data/pg/export path for each date. However, it did append some random letter on sub-dir on other hosts. So for instance, I've folder, files in following format:
on node#1 (10.245.122.100)
/data/pg/export/schema.table_20190927 contains:
----1.parquet
----2.parquet
----3.parquet
on node#2 (10.245.122.101)
/data/pg/export/schema.table_20190927S8rW4dQ2 contains:
----4.parquet
----5.parquet
----6.parquet
on node#3 (10.245.122.102)
/data/pg/export/schema.table_20190927P5SJ9aX4 contains:
----7.parquet
----8.parquet
----9.parquet
and so on for other nodes.
How I can bring files from /data/pg/export/schema.table_20190927S8rW4dQ2 on node#2 (10.245.122.101) and /data/pg/export/schema.table_20190927P5SJ9aX4 on node#3 (10.245.122.102) (and similar for other hosts) to /data/pg/export/schema.table_20190927 on node#1 (10.245.122.100) so
final output look like:
***on node#1 (10.245.122.100)***
/data/pg/export/schema.table_20190927 will have:
----1.parquet
----2.parquet
----3.parquet
----4.parquet
----5.parquet
----6.parquet
----7.parquet
----8.parquet
----9.parquet

Answer 1

Welcome to SO. Since it is your first question (well the first I see), and I liked the challenge, here is a script that will do that. For your next question, you must provide your own code with a specific problem you are having, and not expect a complete script as an answer. See my comment for stuff to read on using SO.
The bash knowledge required to make this work is:
while loop
date calculation
variable value incrementation (so basic math)
I made some assumptions:
you have a single user on all nodes which can be used to do scp from node1
that user is hopefully setup to use ssh keys to login, otherwise you will type your password a lot of times!
you have connected at least 1 time to each node, and they are listed in your known_hosts file
on each node, there is 1 and only one directory with a specific date in the name.
all files are copied in each directory. You can modify the scp command to get only the .parquet files if you want.
Basic ideas in the code
loop on each node, so from 2 to 7
loop on dates, so from 20190927 to 20200131
copy files for each node, each date within the loops
this was tested on Linux Mint (== Ubuntu) so the date command is the gnu version, which allows for date calculation the way I did it.
Before use, modify the value of the user variable with your user name.
DISCLAIMER: I did not have multiple systems to test the scp command, so this command was added by memory.
The code:
#!/bin/bash
#
# This script runs on node1
# The node1 IP is 10.245.122.100
#
# This script assumes that you want to copy all files under
# /data/pg/export/schema.table_YYYMMDD<random>
#
###############################################################################
# node1 variables
targetdirprefix="/data/pg/export/schema.table_"
user="YOURUSER"
# Other nodes variables
total_number_nodes=7 # includes node1
ip_prefix=10.245.122
ip_lastdigit_start=99 # node1 == 100, so start at 99
# loop on nodes ---------------------------------------------------------------
# start at node 2, node1 is the target node
nodecount=2
# Stop at maxnode+1, here he last node will be 7
(( whileexit = total_number_nodes + 1 ))
while [[ "$nodecount" -lt "$whileexit" ]]
do
# build the current node IP
(( currentnode_lastdigit = ip_lastdigit_start + nodecount ))
currentnode_ip="${ip_prefix}.${currentnode_lastdigit}"
# DEBUG
echo "nodecount=$nodecount, ip=$currentnode_ip"
# loop on dates ---------------------------------------
firstdate="20190927"
lastdate="20200131"
loopdate="$firstdate"
while [[ "$loopdate" -le "$lastdate" ]]
do
# DEBUG
echo "loopdate=$loopdate"
# go into the target directory (create it if required)
targetdir="${targetdirprefix}${loopdate}"
if [[ -d "$targetdir" ]]
then
cd "$targetdir"
else
mkdir -p "$targetdir"
if [[ "$?" -ne 0 ]]
then
echo "ERROR: could not create directory $targetdir, exiting."
exit 1
else
cd "$targetdir"
fi
fi
# copy the date's file into the target dir (i.e. localy, since we did a cd before)
# the source directory is the same as the targetdir, with extra chars at the end
# this script assumes there is only 1 directory with that particular date!
scp ${user}#${currentnode_ip}:${targetdir}* .
if [[ "$?" -ne 0 ]]
then
echo "WARNING: copy failed from node $nodecount, date $loopdate."
echo " The script will continue for other dates and nodes..."
fi
loopdate=$(date --date "$loopdate +1 days" +%Y%m%d)
done
(( nodecount += 1 ))
done