File Not Found When Running Docker Script But Is Found In Another Docker Script - linux

I am trying to run a script inside a docker that was published by google.
The command I use mounts some datafiles onto the docker in a file called '/input' (inside the docker).
When I run the script, it says that it does not find the input file.
However, I do use the -v flag, and I ran a script that makes sure the input file is there (inside the docker).
So in summary - when I run
find /input -name "*.fasta"
It outputs:
/input/ucsc.hg19.chr20.unittest.fasta
As needed, but when I run the script, it says
./dv-quick-start: 19: ./dv-quick-start: --ref=/input/ucsc.hg19.chr20.unittest.fasta: not found
Full Script:
#!/bin/sh
BIN_VERSION="1.0.0"
INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"
OUTPUT_DIR="${PWD}/quickstart-output"
sudo docker run \
-v "${INPUT_DIR}":"/input" \
-v "${OUTPUT_DIR}":"/output" \
google/deepvariant:"${BIN_VERSION}" \
find /input -name "*.fasta"
sudo docker run \
-v "${INPUT_DIR}":"/input" \
-v "${OUTPUT_DIR}":"/output" \
google/deepvariant:"${BIN_VERSION}" \
/opt/deepvariant/bin/run_deepvariant \
--model_type=WGS \ **Replace this string with exactly one of the following [WGS,WES,PACBIO,HYBRID_PACBIO_ILLUMINA]**
--ref=/input/ucsc.hg19.chr20.unittest.fasta \
--reads=/input/NA12878_S1.chr20.10_10p1mb.bam \
--regions "chr20:10,000,000-10,010,000" \
--output_vcf=/output/output.vcf.gz \
--output_gvcf=/output/output.g.vcf.gz \
--intermediate_results_dir /output/intermediate_results_dir \ **This flag is optional. Set to keep the intermediate results.
Full output:
/input/ucsc.hg19.chr20.unittest.fasta
--ref is required.
Pass --helpshort or --helpfull to see help on flags.
./dv-quick-start: 19: ./dv-quick-start: --ref=/input/ucsc.hg19.chr20.unittest.fasta: not found
I feel there is some misunderstanding on my behalf, and I would appreciate any help.
Should more information be needed to answer the question, let me know.

You have some extraneous text in your shell script that's causing a problem. Delete the "replace this string" and "this flag is optional" text and all of the whitespace before them, making the \ the very last character on those lines.
In a shell script you can break commands across multiple lines using a \. But, the \ must be the absolute very last character in the line; if it's not, it escapes the character that comes after it.
# one line: ls -al $HOME
ls -al \
$HOME
# two lines: ls -al " " more text here; $HOME
ls -al \ more text here
$HOME
In your example you've left some explanatory text in
sudo docker run \
...\
--model_type=WGS \ **Replace this string with exactly one of the following [WGS,WES,PACBIO,HYBRID_PACBIO_ILLUMINA]**
# This is seen as a separate command
--ref=/input/ucsc.hg19.chr20.unittest.fasta \
...
Since the "Replace this string..." text makes the \ not be the absolute last character in the line, it causes the shell to break the command. You then get two commands, a docker run command without the --ref option and what looks like trying to run --ref=... as a separate command; that corresponds to the two errors you get.

Related

Getting this error: curl: (3) Host name ' --url' contains bad letter

I am trying to run this line on a linux machine:
curl --request GET \ --url 'https://www.tenable.com/downloads/api/v2/pages/nessus/files/Nessus-10.4.2-ubuntu1404_amd64.deb' \ --output 'nessus.deb'
but I am getting this error:
curl: (3) Host name ' --url' contains bad letter
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: " to save to a file.
I ended up using: wget https://www.tenable.com/downloads/api/v2/pages/nessus/files/Nessus-10.4.2-ubuntu1404_amd64.deb
but it seems I should not have done it this way because now the subsequent line I want to run isn't working: dpkg -i nessus.deb
The output for the curl was 'nessus.deb'. So to install you had to use 'dpkg -i nessus.deb'.
However, because you used wget, it did not create the output name, but just saved 'INSERTFILENAMEHERE'. So to install it, you would use:
'dpkg -i *INSERTFILENAMEHERE*'
For example, you would have put in
'wget https://www.tenable.com/downloads/api/v2/pages/nessus/files/Nessus-10.4.2-ubuntu1404_amd64.deb'
Then it would say something like
''Nessus.deb' saved'
So you would then put in.
'dpkg -i Nessus.deb'
And it would install the package. In short, the reason why the line was not working was because the file was there, but the name did not match.

Need to run multiple feature file in behave using python

My command in python is
feature_file_folder -f allure_behave.formatter:AllureFormatter -o target/allure-results --no-capture --no-capture-stderr
This runs all the feature files but if i want to run only 2 feature files out of 10 i am having problems i can run 1 or all feature file.
You can mention feature file names while executing.
behave -f allure_behave.formatter:AllureFormatter \
-o target/allure-results \
--no-capture \
--no-capture-stderr \
example1.feature example2.feature
This is what tags are used for. Lets say, hypothetically, the two you want to run are related to logging into a system:
login-from-main-page.feature:
#login-tests
Feature: Test logging in from the main page
Scenario: ...
login-from-mobile.feature
#login-tests
Feature: Test logging in from iOS App
Scenario: ...
Then you would run behave, specifying those tags:
feature_file_folder -f allure_behave.formatter:AllureFormatter \
-o target/allure-results \
--no-capture \
--no-capture-stderr \
--tags login-tests

Execution CMake from within a Bash Script

I've built a script to automate a CMake build of OpenCV4. The relevant part of the script is written as:
install.sh
#!/bin/bash
#...
cd /home/pi/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=ON \
-D BUILD_TESTS=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D BUILD_EXAMPLES=OFF ..
This part of the code is first executed from /home/pi/ directory. If I execute these lines within the cli they work and the cmake file is built without error. If I run this same code form a bash script it results in the cmake command resulting in -- Configuring incomplete, errors occurred!.
I believe this is similar to these two SO threads (here and here) in as much as they both describe situations where calling a secondary script from the first script creates a problem (or at least that's what I think they are saying). If that is the case, how can you start a script from /parent/, change to /child/ within the script, execute secondary script (CMake) as though executed from the /child/ directory?
If I've missed my actual problem - highlighting taht would be even more helpful.
Update with Full Logs
Output log for CMakeOutput.log and CMakeError.log as unsuccessfully run from the bash script.
When executed from the cli the successful logs are success_CMakeOutput.log and success_CMakeError.log
Update on StdOut
I looked through the files above and they look the same... Here is the failed screen output (noting the bottom lines) and the successful screen output.
You are running your script as the root user with the /root home directory, while the opencv_contrib directory is in /home/pi directory. The /home/pi is most probably the home directory of the user pi.
Update the:
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
With proper path to opencv_contrib. Either provide opencv_contrib in the home directory of the root user, if you aim to run the script as root, or provide full, non-dependent on HOME, path to opencv_contrib directory.
-D OPENCV_EXTRA_MODULES_PATH=/home/pi/opencv_contrib/modules \

sidekiq.yml file is not being considered

I have installed gitlab community edition on my raspberry pi 3. Everything is working fine. But when the application is up there are 25 sidekiq threads. It's eating up my memory and I don't want so many threads.
I tried controlling by adding the file /opt/gitlab/embedded/service/gitlab-rails/config/sidekiq.yml.
# Sample configuration file for Sidekiq.
# Options here can still be overridden by cmd line args.
# Place this file at config/sidekiq.yml and Sidekiq will
# pick it up automatically.
---
:verbose: false
:concurrency: 5
# Set timeout to 8 on Heroku, longer if you manage your own systems.
:timeout: 30
# Sidekiq will run this file through ERB when reading it so you can
# even put in dynamic logic, like a host-specific queue.
# http://www.mikeperham.com/2013/11/13/advanced-sidekiq-host-specific-queues/
:queues:
- critical
- default
- <%= `hostname`.strip %>
- low
# you can override concurrency based on environment
production:
:concurrency: 5
staging:
:concurrency: 5
I have restarted the application many times and even ran "reconfigure". It's not helping. It's not considering the sidekiq.yml file at all.
Can anybody please let me know where I am going wrong?
i found your question by searching for a solution for the same problem. All i found doesn't work. So i tried bye myself and found the right place for reducing sidekiq from 25 to 5. I use the gitlab omnibus version. I think the path is idetical to yours:
/opt/gitlab/sv/sidekiq/run
In this file you find the following code:
#!/bin/sh
cd /var/opt/gitlab/gitlab-rails/working
exec 2>&1
exec chpst -e /opt/gitlab/etc/gitlab-rails/env -P \
-U git -u git \
/opt/gitlab/embedded/bin/bundle exec sidekiq \
-C /opt/gitlab/embedded/service/gitlab-rails/config/sidekiq_queues.yml \
-e production \
-r /opt/gitlab/embedded/service/gitlab-rails \
-t 4 \
-c 25
Change the last line to "-c 5". The result should look like this:
#!/bin/sh
cd /var/opt/gitlab/gitlab-rails/working
exec 2>&1
exec chpst -e /opt/gitlab/etc/gitlab-rails/env -P \
-U git -u git \
/opt/gitlab/embedded/bin/bundle exec sidekiq \
-C /opt/gitlab/embedded/service/gitlab-rails/config/sidekiq_queues.yml \
-e production \
-r /opt/gitlab/embedded/service/gitlab-rails \
-t 4 \
-c 5
Last but no least yout have to resart gitlab service
sudo gitlab-ctl restart
No idea, what happening on the gitlab update. I think i have to change this value again. It would be nice, if the gitlab developers add this option to gitlab.rb in /etc/gitlab directory.

Bash completion for path in argument (with equals sign present)

I used to be able to type the following:
$> ./foo --arg=<TAB>
Where foo is any program I wrote, and it would give me a list of files in the current directory, just like tab-completion normally does. I didn't have to make any changes to /etc/bash_completion.
Recently, however, this has gone away for some unknown reason. Does anyone know how to re-enable this feature?
FWIW, this still does the correct thing (notice the lack of an equals sign):
$> ./foo --arg <TAB>
I removed all bash completion scripts and started adding them one by one to if any of them cause the problem.
In my case it turned out to be the npm completion script was the cause of this problem.
Not sure (yet) what the problem is, but this is the completion script which caused equal sign values not working as before:
###-begin-npm-completion-###
#
# npm command completion script
#
# Installation: npm completion >> ~/.bashrc (or ~/.zshrc)
# Or, maybe: npm completion > /usr/local/etc/bash_completion.d/npm
#
COMP_WORDBREAKS=${COMP_WORDBREAKS/=/}
COMP_WORDBREAKS=${COMP_WORDBREAKS/#/}
export COMP_WORDBREAKS
if type complete &>/dev/null; then
_npm_completion () {
local si="$IFS"
IFS=$'\n' COMPREPLY=($(COMP_CWORD="$COMP_CWORD" \
COMP_LINE="$COMP_LINE" \
COMP_POINT="$COMP_POINT" \
npm completion -- "${COMP_WORDS[#]}" \
2>/dev/null)) || return $?
IFS="$si"
}
complete -F _npm_completion npm
elif type compdef &>/dev/null; then
_npm_completion() {
si=$IFS
compadd -- $(COMP_CWORD=$((CURRENT-1)) \
COMP_LINE=$BUFFER \
COMP_POINT=0 \
npm completion -- "${words[#]}" \
2>/dev/null)
IFS=$si
}
compdef _npm_completion npm
elif type compctl &>/dev/null; then
_npm_completion () {
local cword line point words si
read -Ac words
read -cn cword
let cword-=1
read -l line
read -ln point
si="$IFS"
IFS=$'\n' reply=($(COMP_CWORD="$cword" \
COMP_LINE="$line" \
COMP_POINT="$point" \
npm completion -- "${words[#]}" \
2>/dev/null)) || return $?
IFS="$si"
}
compctl -K _npm_completion npm
fi
###-end-npm-completion-###
Not sure what environment you're in, but on a recent CentOS
complete -D -o default
enables filename completion after a token w/o whitespace as the default. To toggle it in the other direction:
complete -D -o nospace
However, it looks like older versions of the builtin don't have the -D option.
I resolved the same trouble with Ubuntu 12.04 by using https://github.com/ai/rake-completion.
You need to
download the file wget -O ~/scripts/rake https://raw.githubusercontent.com/ai/rake-completion/master/rake
add to your .bashrc: . ~/scripts/rake
or You can use one of other ways on that page.

Resources