I'm running the command -ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429 /SRR5907429 .sra ~/sra_download with Linux
and I get this error -
"user#host:" in all sources must match
What does this mean?How to solve it?
First,"-private"should be removed.Secondly,need to correct the space error in the sentence,example "SRR5907429 ".'ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m anonftp#ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429/SRR5907429.sra ~/sra_download'is the correct answer we need.enter image description here
Your problem:
the ascp syntax is:
Usage: ascp [OPTION] SRC... DEST
SRC to DEST, or multiple SRC to DEST dir
SRC, DEST format: [[user#]host:]PATH
Display full usage: -h,--help
You get this by simply executing ascp, get more with "ascp -h" and have a manual for it as well, or https://download.asperasoft.com/download/docs/entsrv/3.9.1/es_admin_linux/webhelp/index.html#dita/ascp_2.html
it is pretty much like "scp", but works also in "pull" mode.
so, you have:
options then one or multiple sources, then a single destination (always the last argument).
if the destination is: user#server:folder, then you do a push
if source is user#server:folder, then you do a pull
globally, you can only do a push or a pull at the same time. but there can be multiple sources, and always a single destination (on command line).
in you case you have:
options: -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m
sources: anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429 /SRR5907429 .sra
destination:~/sra_download
the first source is: anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429
the other sources are: /SRR5907429 .sra
so you specify one remote source, two local sources, and one local destination.
This is the error you get.
My advice:
do not use the legacy syntax, as you did, but instead, use the advanced syntax:
ascp [options] --mode=<send|recv> --user=<user> --host=<server> sources... destination
There are plenty of options, for instance, if all your source files are in the same folder, you can use: --source-prefix=
you can also use file list file (i.e. a file that contains the list of files you want to transfer, in case it is long and generated by a script) or even file par list file.
Note also, that there is an interesting front end for aspera command line transfers:
https://www.rubydoc.info/gems/asperalm
Related
This the log file that I want to monitor:
/test/James-2018-11-16_15215125111115-16.15.41.111-appserver0.log
I want Nagios to read it this log file so I can monitor a specific string.
The issue is with 15215125111115 this is the random id that gets generated
Here is my script where the Nagios is checking for the Logfile path:
Veriables:
HOSTNAMEIP=$(/bin/hostname -i)
DATE=$(date +%F)
..
CHECK=$(/usr/lib64/nagios/plugins/check_logfiles/check_logfiles
--tag='failorder' --logfile=/test/james-${date +"%F"}_-${HOSTNAMEIP}-appserver0.log
....
I am getting the following output in nagios:
could not find logfile /test/James-2018-11-16_-16.15.41.111-appserver0.log
15215125111115 This number is always generated randomly but I don't know how to get nagios to identify it. Is there a way to add a variable for this or something? I tried adding an asterisk "*" but that didn't work.
Any ideas would be much appreciated.
--tag failorder --type rotating::uniform --logfile /test/dummy \
--rotation "james-${date +"%F"}_\d+-${HOSTNAMEIP}-appserver0.log"
If you add a "-v" you can see what happens inside. Type rotating::uniform tells check_logfiles that the rotation scheme makes no difference between current log and rotated archives regarding the filename. (You frequently find something like xyz..log). What check_logfile does is to look into the directory where the logfiles are supposed to be. From /test/dummy it only uses the directory part. Then it takes all the files inside /test and compares the filenames with the --rotation argument. Those files which match are sorted by modification time. So check_logfiles knows which of the files in question was updated recently and the newest is considered to be the current logfile. And inside this file check_logfiles searches the criticalpattern.
Gerhard
I'm looking for a way to flatten directories from a /year/month/day/directory format to just /directory via rsync
The source directories, containing data files, are formatted like this:
/year/month/day/round-number/files
Source:
/2018/06/01/round-1111/(files)
/2018/06/01/round-1112/(etc)
/2018/06/01/round-1113
/2018/06/02/round-1114
/2018/06/02/round-1115
/2018/06/02/round-1116
/2018/06/03/round-1117
/2018/06/03/round-1118
/2018/06/03/round-1119
I need them to come out like this at the Destination:
/round-1111/(files)
/round-1112/(etc)
/round-1113
/round-1114
/round-1115
/round-1116
/round-1117
/round-1118
/round-1119
The command I'm using right now is basically "rsync -a source destination"
I'd like to keep the processing load of this command low, as it needs to run frequently while also not disturbing the source server too much.
According to the man page of wget, --acccept-regex is the argument to use when I need to selectively transfer files whose names matching a certain regular expression. However, I am not sure how to use --accept-regex.
Assuming I want to obtain files diffs-000107.tar.gz, diffs-000114.tar.gz, diffs-000121.tar.gz, diffs-000128.tar.gz in IMDB data directory ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/. "diffs\-0001[0-9]{2}\.tar\.gz" seems to be an ok regex to describe the file names.
However, when executing the following wget command
wget -r --accept-regex='diffs\-0001[0-9]{2}\.tar\.gz' ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/
wget indiscriminately acquires all files in the ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/ directory.
I wonder if anyone could tell what I have possibly done wrong?
Be careful --accept-regex is for the complete URL. But our target is some specific files. So we will use -A.
For example,
wget -r -np -nH -A "IMG[012][0-9].jpg" http://x.com/y/z/
will download all the files from IMG00.jpg to IMG29.jpg from the URL.
Note that a matching pattern contains shell-like wildcards, e.g. ‘books’ or ‘zelazny196[0-9]*’.
reference:
wget manual: https://www.gnu.org/software/wget/manual/wget.html
regex: https://regexone.com/
I'm reading in wget man page:
--accept-regex urlregex
--reject-regex urlregex
Specify a regular expression to accept or reject the complete URL.
and noticing that it mentions the complete URL (e.g. something like ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/diffs-000121.tar.gz)
So I suggest (without having tried it) to use
--accept-regex='.*diffs\-0001[0-9][0-9]\.tar\.gz'
(and perhaps give the appropriate --regex-type too)
BTW, for such tasks, I would also consider using some scripting language à la Python (or use libcurl or curl)
I am new to Bowtie. I am trying to use Bowtie for end to end local alignment. I've got this error message:
Could not locate a Bowtie index corresponding to basename "/bowtie2-index/hg19"
In my installation and in the bowtie2-index/hg19 folder there are six bt2 files. I am using the following command:
/opt/bowtie2/bowtie2-align-s --wrapper basic-0 -p 64 -x /mnt/miczfs/tide/bowtie2-index/hg19 -S /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.sam -1 /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.f1.fastq -2 /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.f2.fastq
This is a perennial question, I guess the documentation isn't explicit enough here. By using -x /mnt/miczfs/tide/bowtie2-index/hg19, you're telling bowtie2 that you have files like /mnt/miczfs/tide/bowtie2-index/hg19.1.bt2 that it should use. You don't specify a folder, you specify a "basename". You probably meant -x /mnt/miczfs/tide/bowtie2-index/hg19/hg19 or something like that.
I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure.
Determine the directory structure first,
then build and use a download description file:
aria2c -i uri.txt
where uri.txt might contain
http://serverA/file1.iso http://mirror-serverB/file1.iso
# parameters must begin with a space, otherwise it's treatened as url!
dir=/downloads/a
# not mandatory
out=file1.iso
http://serverA/file2.iso http://mirror-serverB/file2.iso
dir=/downloads/b
out=file2.iso
Keep in mind that aria2 is a download util - not an sync util, like rsync or lftp.
Referencing an rsync answer: https://stackoverflow.com/a/4147263/1163786
and an lftp answer: https://superuser.com/a/305236.