On a computer with IP address like 10.11.12.123, I have a folder document. I want to copy that folder to my local folder /home/my-pc/doc/ using the shell.
I tried like this:
scp -r smb:10.11.12.123/other-pc/document /home/my-pc/doc/
but it's not working.
So you can use below command to copy your files.
scp -r <source> <destination>
(-r: Recursively copy entire directories)
eg:
scp -r user#10.11.12.123:/other-pc/document /home/my-pc/doc
To identify the location you can use the pwd command, eg:
kasun#kasunr:~/Downloads$ pwd
/home/kasun/Downloads
If you want to copy from B to A if you are logged into B: then
scp /source username#a:/destination
If you want to copy from B to A if you are logged into A: then
scp username#b:/source /destination
In addition to the comment, when you look at your host-to-host copy options on Linux today, rsync is by far, the most robust solution around. It is brought to you by the SAMBA team[1] and continues to enjoy active development. Most distributions include the rsync package by default. (if not, you should find an easily installable package for your distro or you can download it from rsync.samba.org ).
The basic use of rsync for host-to-host directory copy is:
$ rsync -uav srchost:/path/to/dir /path/to/dest
-uav simply recursively copies -ua only new or changed files preserving file & directory times and permissions while providing -v verbose output. You will be prompted for the username/password on 10.11.12.123 unless your have setup ssh-keys to allow public/private key authentication (see: ssh-keygen for key generation)
If you notice, the syntax is basically the same as that for scp with a slight difference in the options: (e.g. scp -rv srchost:/path/to/dir /path/to/dest). rsync will use ssh for secure transport by default, so you will want to insure sshd is running on your srchost (10.11.12.123 in your case). If you have name resolution working (or a simple entry in /etc/hosts for 10.11.12.123) you can use the hostname for the remote host instead of the remote IP. Regardless, you can always transfer the files you are interested in with:
$ rsync -uav 10.11.12.123:/other-pc/document /home/my-pc/doc/
Note: do NOT include a trailing / after document if you want to copy the directory itself. If you do include a trailing / after document (i.e. 10.11.12.123:/other-pc/document/) you are telling rsync to copy the contents, (i.e. the files and directories under) document to 10.11.12.123:/other-pc/ without also copying the document directory.
The reason rsync is far superior to other copy apps is it provides options to truly synchronize filesystems and directory trees both locally and between your local machine and remote host. Meaning, in your case, if you have used rsync to transfer files to /home/my-pc/doc/ and then make changes to the files or delete files on 10.11.12.123, you can simply call rsync again and have the changes/deletions reflected in /home/my-pc/doc/. (look at the several flavors of the --delete option for details in rsync --help or in man 1 rsync)
For these, and many more reasons, it is well worth the time to familiarize yourself with rsync. It is an invaluable tool in any Linux user's hip pocket. Hopefully this will solve your problem and get you started.
Footnotes
[1] the same folks that "Opened Windows to a Wider World" allowing seemless connection between windows/Linux hosts via the native windows server message block (smb) protocol. samba.org
If the two directories (document and /home/my-pc/doc/) you mentioned are on the same 10.11.12.123 machine.
then:
cp -ai document /home/my-pc/doc/
else:
scp -r document/ root#10.11.12.123:/home/my-pc/doc/
Related
I have directory(/usr/share/hub-bucket/GameImages/) which has sub directories which contains files. And I want to transfer the sub-directories to machine to location /usr/share/hub-bucket/GameImages/. Both are remote machine, I can access remote destination using SSH private key and passphrase. And in future I will need to sync both the remote source and remote destination folder/files. How can this be implemented ?? I have used SCP for file transfer but haven't used for folder/sub-folder.
You can use the flag -r to copy files recursively with scp.
scp -r /usr/share/hub-bucket/GameImages/ user#remotehost:/usr/share/hub-bucket/GameImages/
A better and often faster option is to use rsync, which is usually more efficient since it only transfers files that differ between the two hosts.
if you use scp you can use -r option, like this
scp -r /usr/share/hub-bucket/GameImages/ user#remote-host:/usr/share/hub-bucket/GameImages/
you can also use rsync command
rsync -avz /usr/share/hub-bucket/GameImages/ user#remote-host:/usr/share/hub-bucket/GameImages/
Can any one help me writing a shell script to Download files from Linux/UNIX system?
Regards
On UNIX systems, such as Linux and OSX, you have access to a utility called rsync. It is installed by default and is the tool to use to download files from another UNIX system.
It is a drop-in replacement for the cp (copy) command, but it is much more powerful.
To copy a directory from a remote system to yours, using SSH, you would do this:
rsync username#hostname:path/to/dir .
(notice the dot at the end, this means 'place everything here please', you can also give the name of the local dir where the files should be placed.)
To download only some specific files, use this:
rsync 'username#hostname:path/to/dir/*.txt' .
(notice the quotes: if you omit them, your shell will try to expand the *.txt part locally, will fail and give you an error.)
Useful flags:
--progress: show a progress bar
--append: if a file has only partially downloaded, resume it where it left off
I find the rsync utility so useful, I've created an alias for it in my shell and use it as a 'super-copy':
alias cpa 'rsync -vae ssh --progress --append'
With that alias, copying files between machines is just as easy as copying files locally:
cpa user#host:file .
Making it even better
Since rsync is using SSH, it helps to setup a private/public key pair, so you don't have to type in your password every time:
How do I setup Public-Key Authentication?
Futhermore, you can write down your username in your .ssh/config file and give the remote host a short name: read about it here.
For example, I have something like this:
Host panda
Hostname panda.server.long.hostname.com
User rodin
With this setup, my command to download files from the panda server is just:
cpa panda:path/to/my/files .
And there was much rejoicing.
Is there any option to tell scp command - not copy file from current machine in case file exists on remote machine
For example
On my machine I have the file -
/etc/secret-pw.txt
On Remote machine I have also the file -
/etc/secret-pw.txt
So
scp /etc/secret-pw.txt $remote_machine:/etc
Will destroy the secret-pw.txt, and scp will not ask questions about: overwrite the target file
Is there any option to avoid copy if file exist on target machine by scp?
Update: I can't install rsync or any other program.
You should be using rsync instead of scp. It will give you what you need.
If you can't install rsync (as you mentioned in the comments) you need to run a script beforehand to check if file exists and run it with ssh.
SCP does not offer any option, unfortunately.
But you can resort standard tools, like this:
ssh $remote_machine -- cp --no-clobber /dev/stdin /etc/secret-pw.txt < /etc/secret-pw.txt
Note that with this trick you gain all the functionalities of cp.
I am trying to rsync directory A of server1 with directory B of server2.
Sitting in the directory A of server1, I ran the following commands.
rsync -av * server2::sharename/B
but the interesting thing is, it synchronizes all files and directories except .htaccess or any hidden file in the directory A. Any hidden files within subdirectories get synchronized.
I also tried the following command:
rsync -av --include=".htaccess" * server2::sharename/B
but the results are the same.
Any ideas why hidden files of A directory are not getting synchronized and how to fix it. I am running as root user.
thanks
This is due to the fact that * is by default expanded to all files in the current working directory except the files whose name starts with a dot. Thus, rsync never receives these files as arguments.
You can pass . denoting current working directory to rsync:
rsync -av . server2::sharename/B
This way rsync will look for files to transfer in the current working directory as opposed to looking for them in what * expands to.
Alternatively, you can use the following command to make * expand to all files including those which start with a dot:
shopt -s dotglob
See also shopt manpage.
For anyone who's just trying to sync directories between servers (including all hidden files) -- e.g., syncing somedirA on source-server to somedirB on a destination server -- try this:
rsync -avz -e ssh --progress user#source-server:/somedirA/ somedirB/
Note the slashes at the end of both paths. Any other syntax may lead to unexpected results!
Also, for me its easiest to perform rsync commands from the destination server, because it's easier to make sure I've got proper write access (i.e., I might need to add sudo to the command above).
Probably goes without saying, but obviously your remote user also needs read access to somedirA on your source server. :)
I had the same issue.
For me when I did the following command the hidden files did not get rsync'ed
rsync -av /home/user1 server02:/home/user1
But when I added the slashes at the end of the paths, the hidden files were rsync'ed.
rsync -av /home/user1/ server02:/home/user1/
Note the slashes at the end of the paths, as Brian Lacy said the slashes are the key. I don't have the reputation to comment on his post or I would have done that.
I think the problem is due to shell wildcard expansion. Use . instead of star.
Consider the following example directory content
$ ls -a .
. .. .htaccess a.html z.js
The shell's wildcard expansion translates the argument list that the rsync program gets from
-av * server2::sharename/B
into
-av a.html z.js server2::sharename/B
before the command starts getting executed.
The * tell to rsynch to not synch hidden files. You should not omit it.
On a related note, in case any are coming in from google etc trying to find while rsync is not copying hidden subfolders, I found one additional reason why this can happen and figured I'd pay it forward for the next guy running into the same thing: if you are using the -C option (obviously the --exclude would do it too but I figure that one's a bit easier to spot).
In my case, I had a script that was copying several folders across computers, including a directory with several git projects and I noticed that the I couldn't run any of the normal git commands in the copied repos (yes, normally one should use git clone but this was part of a larger backup that included other things). After looking at the script, I found that it was calling rsync with 7 or 8 options.
After googling didn't turn up any obvious answers, I started going through the switches one by one. After dropping the -C option, it worked correctly. In the case of the script, the -C flag appears to have been added as a mistake, likely because sftp was originally used and -C is a compression-related option under that tool.
per man rsync, the option is described as
--cvs-exclude, -C auto-ignore files in the same way CVS does
Since CVS is an older version control system, and given the man page description, it makes perfect sense that it would behave this way.
I have online account with some Host which give me FTP account with username and password .
i have another with copany which gave me FTP and rsync .
Now i want to transfer all my files from old FTP to NEW FTP with rync.
Now is it possible to do it via rsync only because i don't want to first copy on computer and then upload again
Lets call the machine with only FTP src.
Lets call the machine with FTP and SSH dst.
ssh dst
cd destination-direction
wget --mirror --ftp-user=username --ftp-password=password\
--no-host-directories ftp://src/pathname/
Note that running wget with --ftp-password on the command line will give away the password to anyone else on the system. (As well as transferring it over the wire in the clear, but you knew that.)
If you don't have access to wget, then they might have ncftp or lftp or ftp installed. I just happen to know wget the best. :)
Edit To use ftp, you'll need to do something more like:
ftp src
user username
pass password
bin
cd /pathname
ls
At this point, note all the directories on the remote system. Create each one with !mkdir. Then change into the directory both locally and remotely:
lcd <dirname>
cd <dirname>
ls
Repeat for all the directories. Use mget * to get all the files.
If this looks awful, it is because it is. FTP wasn't designed for this, and if your new host doesn't have better tools (be sure to look for ncftp and lftp and maybe something like ftpmirror), then either compile better tools yourself or get good at writing scripts around the bad tools. :)
Or if you could get a shell on src, that'd help immensely too. FTP is just not intended for transferring thousands of files.
Anyway, this avoids bouncing through your local system, which ought to help throughput significantly.
There's always the trusty FUSE filesystems, CurlFtpFS and SSHFS. Mount each server with the appropriate filesystem and copy across using standard utilities. Probably not the fastest way to do it, but quite possibly the least labor-intensive.
I was looking for a simple solution to sync a remote folder to a local folder via FTP while only replacing new files. I got stuck with a little wget script based on sarnold's answer that I thought might be helpful to others, too, so here it is:
#!/bin/bash
HOST="1.2.3.4"
USER="username"
PASS="password"
LDIR="/path/to/local/dir" # can be empty
RDIR="/path/to/remote/dir" # can be empty
cd $LDIR && \ # only start if the cd was successful
wget \
--continue \ # resume on files that have already been partially transmitted
--mirror \ # --recursive --level=inf --timestamping --no-remove-listing
--no-host-directories \ # don't create 'ftp://src/' folder structure for synced files
--ftp-user=$USER \
--ftp-password=$PASS \
ftp://$HOST/$RDIR