Currently, I have a server that needs to batch process a bunch of files. All of the files are on Server A, which is running Ubuntu, but I need them to be processed on a macos server. Right now, I have a script that will transfer all the files from Server A to Server B, process all the files, then transfer all the files back to Server A.
The bash file looks like this (simplified):
script -q -c "scp -r files_to_process b:process_these 2>&1"
ssh b "process_all.sh"
script -q -c "scp -r processed_files a:final_dir 2>&1"
My question is this: is there any easy way to implement a simple queue between these servers?
Once a file has been transferred to b, I am wasting time by not just processing the files immediately.
Related
I am connected to a server through putty, and I want to download (to my PC) certain files on a regular basis using a shell script. Specifically, these are the files...
ls -t ~/backup | head -n2
What is the best strategy for this? I was trying with command line FTP but I am prompted to login to something. I'm already logged into the server that has the files I need to download, so I am missing something.
The SSH protocol can be a good way, with scp command. You can take a look at this thread
To automate the process and script a solution, you will need use password-less ssh and ssh keys.
The first step will be to get the list of files to copy and so:
fils=$(ssh username#host ls -t ~/backup | head -n2)
Then once we have the files read into a variable fils, we can loop on the entries and run a secure copy command:
while read fyle
do
scp username#host:~/"$fyle" "$fyle"
done <<< "$fils"
I am trying to transfer a large dataset (768 Gigs) from one remote machine to another using bash on ubuntu 16.04. The problem I appear to be having is that I use rsync and the machine will transfer for a few hours and then quit when the connection inevitably gets interrupted. So suppose Im on machine A and the remote servers are machines B and C (all machines using ubuntu 16.04). I ssh to machine B and use this command:
nohup rsync -P -r -e ssh /path/to/files/on/machine_B user#machine_C:directory &
note that I have the authorized key setup so no password is required between machines B and C
A few hours later I get the following in the nohup file:
sending incremental filelist
file_1.bam
90,310,583,648 100% 36.44MB/s 0:39:23 (xfr#4, to-chk=5/10)
file_2.bam
79,976,321,885 100% 93.25MB/s 0:13:37 (xfr#3, to-chk=6/10)
file_3.bam
88,958,959,616 88% 12.50MB/s 0:15:28 rsync error: unexplained error (code 129) at rsync.c(632) [sender=3.1.1]
rsync: [sender] write error: Broken pipe (32)
I used nohup because I though it would keep running even if there was a hangup. I have not tried sh -c and I have not tried running the command from machine A because at this point whatever I try would be guesswork, ideas would be appreciated.
for those that are interested I also tried running the following script with the nohup command on machine B.
script:
chomp( my #files = `ls /path/to/files/on/machineB/*` );
foreach ( #files ) { system("scp $_ user#machineC:destination/"); }
I still got truncated files.
at the moment the following command appears to be working:
nohup rsync -P --append -r -e ssh /path/to/files/on/machine_B user#machine_C:directory &
you just have to check the nohup file for a broken pipe error and re-enter the command if necessary.
I had the same problem and solved it in multiple steps:
First I made sure that I ran all commands on tmux terminals. This adds a layer of safety on top of nohup, as it keeps connections alive: https://en.wikipedia.org/wiki/Tmux
I combined the rsync command with the while command to enforce that the copy is attempted an infinite number of times even if the pipe breaks:
while ! rsync <your_source> <your_destination>; do echo "Rsync failed. Retrying ..."; done
This approach is brute force and it will work if for each attempt, rsync manages to copy at least a few files. Eventually, even with wasteful repeats and multiple failures, all the files will be copied and my command above will exit gratefully.
I have a server where files are transferred thru FTP to a location. All files are there since transfer beginning (January 2015).
I want to make a new server and transfer the files from first server's location.
Basically, I need a cron job to run scp and transfer only new files since last run.
Connection between servers with ssh is working and I can transfer files without restiction between servers.
How can I achieve this in Ubuntu?
The possible duplicate with the other question doesn't stand because, on my destination server I will have just one file where I should keep the date of last cron run and the files which will be copied from first server will be parsed and deleted afterwards.
rsync will simply make sure that all files exists in both servers, correct?
I manage to set-up the cron job on remote computer using the following:
I created first a timestamp file which will keep the last timestamp when cron job run:
touch timestamp
Then I copy all files with ssh and scp commands:
ssh username#remote find <files_path> -type f -newer timestamp | xargs -i scp -i username#remote:'{}' <local_path>
Then I touch timestamp file with new modified time:
touch -m timestamp
The only problem with this script is that, if a file is copied to remote host during ssh run before touching timestamp second time, this file is ignored on later runs.
Later edit:
To be sure that there is no gap between timestamp file and actual run because of ssh command duration, the script was changed to:
touch timestamp_new
ssh username#remote find <files_path> -type f -newer timestamp | xargs -i scp -i username#remote:'{}' <local_path>
rm -rf timestamp
mv timestamp_new timestamp
there are 4 files which will be generated each day in a Linux server. the files has to be sent daily via connect direct to another server which is in unix.
eg..ABC_1JUNE.txt, BCD_1JUNE.txt, CDE_1JUNE.txt, DEF_1JUNE.txt
how to do this in shell script...
To schedule daily jobs on a UNIX-like system you can usually do that with cron. Create a script for that job in the /etc/cron.daily directory on the Linux server and have the cron daemon run it automatically. The script should simply contain the commands to be run. In this case it could look something like this:
#!/usr/bin/env bash
source=<local-dir>
destination=<remote-server>:<remote-dir>
suffix=1JUNE.txt
for file in {ABC,BCD,CDE,DEF}_${suffix}; do
scp "$source/$file" "$destination"
done
This assumes there is a SSH daemon running on the remote server that you can connect to with scp. Replace the values for source and desination to match your real server name and file structures. The source here could also be a remote server.
mount the target server shared folder in the source server folder (or vice versa) and copy the files there using cp command.
For filesystem mount :
http://linux.about.com/od/commands/l/blcmdl8_mount.htm
Remote mounting
I have a bash script on a remote host that produces a large amount of data on fd=3 as well as some possibly interesting data on stdout and stderr. I want to:
Log stdout and stderr to a file on my local machine.
Write the data on fd=3 to stdout on my local machine.
Here's how it could be done if my big script were local:
exec 3> >(cat)
./big_script.sh -o /dev/fd/3 2>&1 >big_script.log
exec 3>&-
However, I want to run big_script.sh on a remote machine and have all three pipes (fd=1, fd=2, and fd=3) come out of the ssh program as separate. What is the best way to that?
nc (netcat) and tunnels ? you can make kinda log radio on your net this way!
SSH opens just a single tty, so you just get a single stream that has all the data. You cannot tell apart what went to what the other side seen as stdout and stderr.
You could log to files on the remote host, and then ssh remote tail -f each of the log files from your local machine.