I need to send multiple files to multiple location, but can't find the proper way.
e.g. I need to send file1 to location1 and file2 to location2. This is what I am doing:
scp file1 file2 root#192.168.1.114:/location1 /location2
But this is not working. Any suggestion?
It's not possible to send to multiple remote locations with a single scp command. It can accommodate multiple source files but only one destination. Using just the scp command itself, it must be run twice.
scp file1 file2 user#192.168.1.114:/location1
scp file1 file2 user#192.168.1.114:/location2
It is possible to turn this into a "one-liner" by running a for loop. In bash for example:
for REMOTE in "user#192.168.1.114:/location1" "user#192.168.1.114:/location2"; do scp file1 file2 $REMOTE; done
scp still runs multiple times but the loop takes care of the iteration. That said, I find it easier to run the command once, hit the Up Arrow (which brings the original command back up in most environments) and just change the remote location and resend.
you can put your scp command in .sh file and execute scp commands with the user name and password in one line
vim ~/fileForSCP.sh
#!/bin/sh
sshpass -p {password} scp file1 root#192.168.1.114:/location1
sshpass -p {password} scp file2 root#192.168.1.114:/location2
sshpass -p {password} scp file3 root#192.168.1.114:/location3
...
sshpass -p {password} scp file{n} root#192.168.1.114:/location{n}
and then:
chmod 777 ~/fileForSCP.sh
~/fileForSCP.sh
You cannot do it with single scp command. Just use scp twice:
scp file1 root#192.168.1.114:/location1
scp file2 root#192.168.1.114:/location2
you can build the desired remote tree structure using sym links
for example
ln -s file1 target/location1/file1
... etc
after this you can use a dereferencing copy to push these files
rsync -rL target -e ssh user#host:/tmp
Hi there is a Hack to go about this query since SCP targets only single Destination, but warn you this might be a bit Naive but gets your work done.
use the && operator between two SCP calls
I use it like this:
scp fileTosend userName1#IPAddress_1:path/to/folder && scp fileToSend userName2#IPAddress_2:path/to/folder
This will simultaneously send the data to both the destinations.
Related
I am trying to execute the scp command in such a way that it can copy .csv files from source to sink, except a few specific CSV file.
For example in the source folder I am having four files:
file1.csv, file2.csv, file3.csv, file4.csv
Out of those four files, I want to copy all files, except file4.csv, to the sink location.
When I was using the below scp command:
scp /tmp/source/*.csv /tmp/sink/
It would copy all the four CSV files to the sink location.
How can I achieve the same by using the scp command or through writing a shell script?
You can use rsync with the --exclude switch, e.g.
rsync /tmp/source/*.csv /tmp/sink/ --exclude file4.csv
Bash has an extended globbing feature which allows for this. On many installations, you have to separately enable this feature with
shopt -e extglob
With that in place, you can
scp tmp/source/(!fnord*).csv /tmp/sink/
to copy all *.csv files except fnord.csv.
This is a shell feature; the shell will expand the glob to a list of matching files - scp will have no idea how that argument list was generated.
As mentioned in your comment, rsync is not an option for you. The solution presented by tripleee works only if the source is on the client side. Here I present a solution using ssh and tar. tar does have the --exclude flag, which allows us to exclude patterns:
from server to client:
$ ssh user#server 'tar -cf - --exclude "file4.csv" /path/to/dir/*csv' \
| tar -xf - --transform='s#.*/##' -C /path/to/destination
This essentially creates a tar-ball which is send over /dev/stdout which we pipe into a tar extract. To mimick scp we need to remove the full path using --transform (See U&L). Optionally you can add the destination directory.
from client to server:
We do essentially the same, but reverse the roles:
$ tar -cf - --exclude "file4.csv" /path/to/dir/*csv \
| ssh user#server 'tar -xf - --transform="s#.*/##" -C /path/to/destination'
You could use a bash array to collect your larger set, then remove the items you don't want. For example:
files=( /tmp/src/*.csv )
for i in "${!files[#]}"; do
[[ ${files[$i]} = *file4.csv ]] && unset files[$i]
done
scp "${files[#]}" host:/tmp/sink/
Note that our for loop steps through array indices rather than values, so that we'll have the right input for the unset command if we need it.
I am trying to use scp to copy large log files from a remote server. However I want only the lines in remote log file that has a string "Fail".
This is how I am doing it currently
scp user#ip:remote_folder/logfile* /localfolder
This copies all the files starting with logfile in remote server to my local folder. The files are pretty large and I need to copy only the lines in those log file, containing the string "Fail" from remote server. Can any body tell me how to do this? Can I use cat or grep command?
Use grep on the remote machine and filter the output into file name and content:
#!/usr/bin/env bash
BASEDIR=~/temp/log
IFS=$'\n'
for match in `ssh user#ip grep -r Fail "remote_folder/logfile*"`
do
IFS=: read file line <<< $match
mkdir -p `dirname $BASEDIR/$file`
echo $line >> $BASEDIR/$file
done
You might want to look at an explanation to IFS in combination with read.
ssh user#ip grep Fail remote_folder/logfile*
The goal is I want to monitor one directory from different server. For example the remote server is user#host.
I have list.txt that contents list of file that will be moved. And list.txt located in a remote server.
Currently I have this code.
ssh user#host cat /full-path/list.txt |
{
while read line;
do mv user#host:/full-path/$line user#host:/full-path/done/;
done;
}
When I run the code above, error exists. There's no such file or directory.
But when I log in to user#host and cat one file randomly from list.txt, the file exists.
The while loop runs on the local server. You need to put the script in quotes so it's an argument to the ssh command.
... Or a here document, like this:
ssh user#host <<':'
while read line; do
mv /full-path/"$line" /full-path/done/
done </full-path/list.txt
:
... or more succinctly
ssh user#host 'cd /full-path && xargs -a list.txt mv -t done'
Notice also the absence of a useless cat and the local file name resolution (mv would have no idea about the SSH remote path syntax you were trying to use).
I have a shell script that I am using to compare directory contents. The script has to ssh to different servers to get a directory listing. When I run the script below, I am getting the contents of the server that I am logged into's /tmp directory listing and not that of the servers I am trying to ssh to. Could you please tell me what I am doing wrong?
The config file used in the script is as follows (called config.txt):
server1,server2,/tmp
The script is as follows
#!/bin/sh
CONFIGFILE="config.txt"
IFS=","
while read a b c
do
SERVER1=$a
SERVER2=$b
COMPDIR=$c
`ssh user#$SERVER1 'ls -l $COMPDIR'`| sed -n '1!p' >> server1.txt
`ssh user#$SERVER2 'ls -l $COMPDIR'`| sed -n '1!p' >> server2.txt
done < $CONFIGFILE
When I look at the outputs of server1.txt and server2.txt, they are both exactly the same - having the contents of /tmp of the server the script is running on (not server1 or 2). Doing the ssh +dir listing on command line works just fine. I am also getting the error "Pseudo-terminal will not be allocated because stdin is not a terminal". Adding the -t -t to the ssh command isnt helping either
Thank you
I have the back ticks in order to execute the command.
Backticks are not needed to execute a command - they are used to expand the standard output of the command into the command line. Certainly you don't want the output of your ssh commands to be interpreted as commands. Thus, it should work fine without the backticks:
ssh user#$SERVER1 "ls -l $COMPDIR" | sed -n '1!p' >>server1.txt
ssh user#$SERVER2 "ls -l $COMPDIR" | sed -n '1!p' >>server2.txt
(provided that double quotes to allow expansion of $COMPDIR are used).
first you need to generate keys to login to remote without keys
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host
then try to ssh without pass
ssh remote-host
then try to invoke in your script but first make sanity check
var1=$(ssh remote-host) die "Cannot connect to remote host" unless $var1;
We have a repository of about 3000 MP3 files. Many of these files have our old domain name within their name.
For example: somesong_oldDomainName.mp3
I need to SSH into the site, find all instances of those files and rename them with the new domain name.
Example: somesong_NEWDomainName.mp3
I know the basic SSH commands but not something advanced like this.
Pretty sure it'll be a combination of multiple commands.
Assuming you get an interactive shell when you ssh into your linux server, this might be a possible way:
ssh user#machine-name-or-ip
then you will get some sort of terminal like
user#machine-name:~$
where you enter the commands to execute on that remote machine.
As mentioned in the comments, the answer here might just fit very well:
Bash: Rename small part of multiple files in middle of name
user#machine-name:~$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/_oldDomainName/_NEWDomainName/g')"; done
This assumes, your current directory is the one with all the MP3 files in it.
If you dont want interactivly operate on your files, e.g. because they change very often and you want a script to perform this action, SSH can also execute a command and/or shell script remotely.
To pass the command directly with the SSH call:
SSH error when executing a remote command: "stdin: is not a tty"
To pipe a local shell script into the SSH connection: How to use SSH to run a shell script on a remote machine?
Run a remote shell script via SSH: how to run a script file remotely using ssh
Edit:
Assume you are connected via SSH to your remote machine and have somewhat similar versions of bash and sed, it should work like this:
$ ls
bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv $i $(echo $i | sed 's/chosefil/tamasha/g'); done
$ ls
bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3
Versions involved:
bash: 4.2.25
sed: 4.2.1
mv: 8.13
Edit 2:
Updated the command to work with blanks in filenames
$ ls
asd chosefil.mp3 bar_chosefil.mp3 boo_chosefil.mp3 foo_chosefil.mp3
$ for i in *.mp3; do mv "$i" "$(echo "$i" | sed 's/chosefil/tamasha/g')"; done
$ ls
asd tamasha.mp3 bar_tamasha.mp3 boo_tamasha.mp3 foo_tamasha.mp3