How to solve 'ascp: "user#host:" in all sources must match' when download SRA data with linux? - linux

I'm running the command -ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429 /SRR5907429 .sra ~/sra_download with Linux
and I get this error -
"user#host:" in all sources must match
What does this mean?How to solve it?

First,"-private"should be removed.Secondly,need to correct the space error in the sentence,example "SRR5907429 ".'ascp -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m anonftp#ftp.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429/SRR5907429.sra ~/sra_download'is the correct answer we need.enter image description here

Your problem:
the ascp syntax is:
Usage: ascp [OPTION] SRC... DEST
SRC to DEST, or multiple SRC to DEST dir
SRC, DEST format: [[user#]host:]PATH
Display full usage: -h,--help
You get this by simply executing ascp, get more with "ascp -h" and have a manual for it as well, or https://download.asperasoft.com/download/docs/entsrv/3.9.1/es_admin_linux/webhelp/index.html#dita/ascp_2.html
it is pretty much like "scp", but works also in "pull" mode.
so, you have:
options then one or multiple sources, then a single destination (always the last argument).
if the destination is: user#server:folder, then you do a push
if source is user#server:folder, then you do a pull
globally, you can only do a push or a pull at the same time. but there can be multiple sources, and always a single destination (on command line).
in you case you have:
options: -v -i ~/.aspera/connect/etc/asperaweb_id_dsa.openssh -k 1 -T -l200m
sources: anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429 /SRR5907429 .sra
destination:~/sra_download
the first source is: anonftp#ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/SRR/SRR590/SRR5907429
the other sources are: /SRR5907429 .sra
so you specify one remote source, two local sources, and one local destination.
This is the error you get.
My advice:
do not use the legacy syntax, as you did, but instead, use the advanced syntax:
ascp [options] --mode=<send|recv> --user=<user> --host=<server> sources... destination
There are plenty of options, for instance, if all your source files are in the same folder, you can use: --source-prefix=
you can also use file list file (i.e. a file that contains the list of files you want to transfer, in case it is long and generated by a script) or even file par list file.
Note also, that there is an interesting front end for aspera command line transfers:
https://www.rubydoc.info/gems/asperalm

Related

How do I use Nagios to monitor a log file that generates a random ID

This the log file that I want to monitor:
/test/James-2018-11-16_15215125111115-16.15.41.111-appserver0.log
I want Nagios to read it this log file so I can monitor a specific string.
The issue is with 15215125111115 this is the random id that gets generated
Here is my script where the Nagios is checking for the Logfile path:
Veriables:
HOSTNAMEIP=$(/bin/hostname -i)
DATE=$(date +%F)
..
CHECK=$(/usr/lib64/nagios/plugins/check_logfiles/check_logfiles
--tag='failorder' --logfile=/test/james-${date +"%F"}_-${HOSTNAMEIP}-appserver0.log
....
I am getting the following output in nagios:
could not find logfile /test/James-2018-11-16_-16.15.41.111-appserver0.log
15215125111115 This number is always generated randomly but I don't know how to get nagios to identify it. Is there a way to add a variable for this or something? I tried adding an asterisk "*" but that didn't work.
Any ideas would be much appreciated.
--tag failorder --type rotating::uniform --logfile /test/dummy \
--rotation "james-${date +"%F"}_\d+-${HOSTNAMEIP}-appserver0.log"
If you add a "-v" you can see what happens inside. Type rotating::uniform tells check_logfiles that the rotation scheme makes no difference between current log and rotated archives regarding the filename. (You frequently find something like xyz..log). What check_logfile does is to look into the directory where the logfiles are supposed to be. From /test/dummy it only uses the directory part. Then it takes all the files inside /test and compares the filenames with the --rotation argument. Those files which match are sorted by modification time. So check_logfiles knows which of the files in question was updated recently and the newest is considered to be the current logfile. And inside this file check_logfiles searches the criticalpattern.
Gerhard

Rsync directories into a flatter structure

I'm looking for a way to flatten directories from a /year/month/day/directory format to just /directory via rsync
The source directories, containing data files, are formatted like this:
/year/month/day/round-number/files
Source:
/2018/06/01/round-1111/(files)
/2018/06/01/round-1112/(etc)
/2018/06/01/round-1113
/2018/06/02/round-1114
/2018/06/02/round-1115
/2018/06/02/round-1116
/2018/06/03/round-1117
/2018/06/03/round-1118
/2018/06/03/round-1119
I need them to come out like this at the Destination:
/round-1111/(files)
/round-1112/(etc)
/round-1113
/round-1114
/round-1115
/round-1116
/round-1117
/round-1118
/round-1119
The command I'm using right now is basically "rsync -a source destination"
I'd like to keep the processing load of this command low, as it needs to run frequently while also not disturbing the source server too much.

wget: obtaining files matching regex

According to the man page of wget, --acccept-regex is the argument to use when I need to selectively transfer files whose names matching a certain regular expression. However, I am not sure how to use --accept-regex.
Assuming I want to obtain files diffs-000107.tar.gz, diffs-000114.tar.gz, diffs-000121.tar.gz, diffs-000128.tar.gz in IMDB data directory ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/. "diffs\-0001[0-9]{2}\.tar\.gz" seems to be an ok regex to describe the file names.
However, when executing the following wget command
wget -r --accept-regex='diffs\-0001[0-9]{2}\.tar\.gz' ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/
wget indiscriminately acquires all files in the ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/ directory.
I wonder if anyone could tell what I have possibly done wrong?
Be careful --accept-regex is for the complete URL. But our target is some specific files. So we will use -A.
For example,
wget -r -np -nH -A "IMG[012][0-9].jpg" http://x.com/y/z/
will download all the files from IMG00.jpg to IMG29.jpg from the URL.
Note that a matching pattern contains shell-like wildcards, e.g. ‘books’ or ‘zelazny196[0-9]*’.
reference:
wget manual: https://www.gnu.org/software/wget/manual/wget.html
regex: https://regexone.com/
I'm reading in wget man page:
--accept-regex urlregex
--reject-regex urlregex
Specify a regular expression to accept or reject the complete URL.
and noticing that it mentions the complete URL (e.g. something like ftp://ftp.fu-berlin.de/pub/misc/movies/database/diffs/diffs-000121.tar.gz)
So I suggest (without having tried it) to use
--accept-regex='.*diffs\-0001[0-9][0-9]\.tar\.gz'
(and perhaps give the appropriate --regex-type too)
BTW, for such tasks, I would also consider using some scripting language à la Python (or use libcurl or curl)

Could not locate a Bowtie index corresponding to basename

I am new to Bowtie. I am trying to use Bowtie for end to end local alignment. I've got this error message:
Could not locate a Bowtie index corresponding to basename "/bowtie2-index/hg19"
In my installation and in the bowtie2-index/hg19 folder there are six bt2 files. I am using the following command:
/opt/bowtie2/bowtie2-align-s --wrapper basic-0 -p 64 -x /mnt/miczfs/tide/bowtie2-index/hg19 -S /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.sam -1 /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.f1.fastq -2 /mnt/miczfs/tide/Data/chr2chr3/chr2chr3.f2.fastq
This is a perennial question, I guess the documentation isn't explicit enough here. By using -x /mnt/miczfs/tide/bowtie2-index/hg19, you're telling bowtie2 that you have files like /mnt/miczfs/tide/bowtie2-index/hg19.1.bt2 that it should use. You don't specify a folder, you specify a "basename". You probably meant -x /mnt/miczfs/tide/bowtie2-index/hg19/hg19 or something like that.

How to keep directory structure with aria2?

I need to download files simultaneously- wget doesn't support that so I want to try aria2. But I don't see an option in aria2 to keep directory structure.
Determine the directory structure first,
then build and use a download description file:
aria2c -i uri.txt
where uri.txt might contain
http://serverA/file1.iso http://mirror-serverB/file1.iso
# parameters must begin with a space, otherwise it's treatened as url!
dir=/downloads/a
# not mandatory
out=file1.iso
http://serverA/file2.iso http://mirror-serverB/file2.iso
dir=/downloads/b
out=file2.iso
Keep in mind that aria2 is a download util - not an sync util, like rsync or lftp.
Referencing an rsync answer: https://stackoverflow.com/a/4147263/1163786
and an lftp answer: https://superuser.com/a/305236.

Resources