I'm using lftp to mirror files from external server but now what I need is to after sucessful download rename source directory (on remote server). Basicaly what I need is to open connection on remote server list directories, download all dirs that name starts from "todo" i.e. todo.20121019 after sucess I must rename downloaded directory to "done.20121019". There might be more than one dir on the server.
Remote FTP server works only with active connection.
#!/bin/bash
directories=`lftp -f lftp_script_file.lf |grep done|awk '{print $NF}'`
for i in $directories
do
echo $i //here I get list of directories that should be downloaded and renamed
done
lftp_script_file.lf just list directires:
set ftp:passive-mode false;
open ftp://user:pass$#10.10.10.123
ls my_sub_dir/
Is there a way to:
open connection to ftp server
find directories that I want to download
add those dirs to queue and download
rename directories on remote server
in batch file?
What I was trying to achive was to list dirs find interesing ones, download and rename but I cant find a way to post list of dirs to lftp via bash script and "set ftp:passive-mode false".
To be able to substitute variables into lftp commands use something like this:
lftp -e "cmd1;cmd2"
Related
I am building FTP bash script to generate a .csv file and transfer from a Linux machine to another server, but i have problems because it tigers an error and the file is not transferred on the 2nd server. What could be the problem?
This is the error:
TEST: A file or directory in the path name does not exist.
Filename invalid
And it doesn't matter if i put the / before the TEST, it will trigger the same issue.
This is my script
HOST='ipadress'
USER='user'
PASSWD=''
TARGET='TEST'
#Paramenters
set -x
DATE=`date +%Y%m%d%H%M`
SQL=/home/sql_statement.sql
QUERYCMD=/home/report.sh
CSV=/home/csv/test_$DATE.csv
#Interogate the sql and put in the folder
$QUERYCMD ${SQL} ${CSV}
#Send the .csv file in the target folder
cd /home/csv
ftp -n $HOST <<EOF
quote USER $USER
quote PASS $PASSWD
lcd $TARGET
put $CSV $TARGET
quit
EOF
exit 0
does the symbol TARGET refer to a dir on the remote host?
ftp command lcd would change-dir on the local (client) side, while cd would change-dir on the remote (server) side; also for the remote side there's usually a designated ftp root dir, adjust any path in relation to that starting point; to confirm dir contents, you might add ftp commands ls and !ls on separate lines right after the PASS line
Current code
#!/bin/bash
SFTP_SERVER="sftp.url.com:/csv/test/10"
SFTP_USER="user"
SFTP_PWD="pwd"
## not sure if this line is needed given I specify the local directory
# in the next block of code.
cd /mnt/c/Users/user/Documents/new_directory
lftp sftp://$SFTP_USER:$SFTP_PWD#$SFTP_SERVER
lftp -e mget *.csv mirror sftp.user.com:/csv/test/10 /mnt/c/Users/user/Documents/new_directory
Objective
Download all csv files and mirror my local directory folder with the remote server, so when the code is run again it won't download a second file.
Error received
open: *.csv: Name or service not known
Comments
From what I understood of the lftp man page I should be able to get all wildcard files by using mget instead of the standard get, provided I use -e to use external commands. I've run mget manually and can download the files without issue but it doesn't seem to support the *.csv in the script.
Appreciate any feedback you can provide as to why my code won't download the files and what I might have misunderstood from the man pages.
It should be like:
lftp sftp://$SFTP_USER:$SFTP_PWD#$SFTP_SERVER -e "mget *.csv; bye"
I'm trying to download multiple files trough SFTP on a linux server using
sftp -o IdentityFile=key <user>#<server><<END
get -r folder
exit
END
which will download all contents on a folder. It appears that find and grep are invalid commands, so are for loops.
I need to download files having a name containing a string e.g.
test_0.txt
test_1.txt
but no file.txt
Do you really need the -r switch? Are there really any subdirectories in the folder? You do not mention that.
If there are no subdirectories, you can use a simple get with a file mask:
cd folder
get *test*
Are you required to use sftp? A tool like rsync that operates over ssh has flexible include/exclude options. For example:
rsync -a <user>#<server>:folder/ folder/ \
--include='test_*.txt' --exclude='*.txt'
This requires rsync to be installed on the remote system, but that's very common these days. If rsync isn't available, you could do something similar using tar:
ssh <user>#<server> tar -cf- folder/ | tar -xvf- --wildcards '*/test_*.txt'
This tars up all the files remotely, but then only extracts files matching your target pattern on the receiving side.
scp user#server:/home/loghost??/logfiles.log .
i'm using above scp command in my unix script to download all the logs from loghost folder.
there are mutliple loghost are avaible in my server(i.e. loghost01,loghost02,loghost03)
The log name is same in all the loghost folder. So while scping, the logs are getting override. Is there a way to change the logname while copying?
for server in loghost01 loghost02 loghost03; do
mkdir -p $server;
scp user#$server:/home/$server/logfiles.log $server/;
done
I think something like that might help.
It takes a list of your servers, scps files over to a folder named loghost##/logfiles.log.
If you have a list of servers in a text file, replace the top line with:
for server in `cat file_containing_servers`; do
Put logs from different servers into different directories:
for server in loghost{01,02,03}
do
mkdir -p $server
scp user#$server:/home/$server/logfiles.log ./$server/
done
Put logs from different servers into the same directory with different names:
for server in loghost{01,02,03}
do
scp user#$server:/home/$server/logfiles.log ./$server.log
done
Sorry if it's too much simple question. But I am Java developer, no idea of shell scripting.
I googled, but couldn't find exactly what I am looking for.
My requirement
Connect to remote server using Sftp [authentication based on pub/pri
keys]. A variable to point to private key file
Transfer files with
specific extension [.log] to local server folder. Variable to set
remote server path and local folder
Rename the transferred file in
remote server
Log all the transferred files in a .txt file
Can any one give me shell script for this?
This is so far I framed from suggestions.
Still some questions left on my side ;)
export PRIVKEY=${private_key_path}
export RMTHOST=user#remotehost
export RMTDIR=/logs/*.log
export LOCDIR=/downloaded/logs/
export LOG=sucess.txt
scp -i $PRIVKEY $RMTHOST:$RMTDIR $LOCDIR
for i in 'ls -1 $LOCDIR/*.log'
do
echo $i >> $LOG
done
ssh $RMTHOST -c "for i in `ls -1 $RMTDIR; do mv /logs/$i /logs/$i.transferred; done"
What about this approach?
Connect to remote server using Sftp [authentication based on pub/pri keys]. A variable to point to private key file
Transfer files with specific extension [.log] to local server folder. Variable to set remote server path and local folder
scp your_user#server:/dir/of/file/*.log /your/local/dir
Log all the transferred files in a .txt file
for file in /your/local/dir/*.log
do
echo "$file" >> $your_txt
done
Rename the transferred file in remote server
ssh your_user#server -c "for file in /dir/of/file/*.log; do mv /dir/of/file/"$file" /dir/of/file/new_name_based_on"$file"; done"
Use scp(secure copy) command to transfer file. You may also want to add the -C switch, which compresses the file. That can speed things up a bit. i.e. copy file1 from server1 to server2,
On server1:
#!/bin/sh
scp -C /home/user/file1 root#server2.com:/home/user
Edit:
#!/bin/sh
scp -i {path/to/pub/pri/key/file} /home/user/file1 root#server2.com:/home/user