using wget in a cron job to download files onto my server - cron

I use a service to generate sitemaps and i'm trying to automate the retrieval process.
I have been using wget to fetch the data and add it to my server
here is my wget statement:
wget --no-check-certificate --quiet \
--output-document sitemap.xml \
--method POST \
--timeout=0 \
--header 'Content-Type: application/x-www-form-urlencoded' \
--body-data 'method=download_sitemap&api_key=[SECRET_KEY]&site_id=[SECRET_ID]&sitemap_id=sitemap.xml' \
'https://pro-sitemaps.com/api/'
^this code works great for me no issues.
I cronjob -e and added the code to my cron folder using nano which looks like this
25 0 * * * "/etc/letsencrypt"/acme.sh --cron --home "/etc/letsencrypt" > /dev/null
07 18 * * * wget --no-check-certificate --quiet \ --output-document "/FILE/PATH/sitemap.xml" \ --method POST \ --timeout=0 \ --header 'Content-Type: application/x-www-form-urlencoded' \ --body-data 'method=download_sitemap&api_key=[SECRET_KEY]&site_id=[SECRET]&sitemap_id=sitemap.xml' \ 'https://pro-sitemaps.com/api/'
My problem is that my code is not running in the cron folder. I have set up the time, so my server time matches my local time. I have tried running the wget statement all on one line and removing any extra spacing in the code block. I tried shorthanding the commands (-T instead of --timeout) & I have tried adding a space at the end of each cron job. I am a bit stumped. Its probably something really simple that I missed in the documentation. Does anybody have any suggestions or notice anything off with what i'm doing in my cron folder?
I have observed these two q's which is where I have gotten the ideas so far troubleshooting: How to get CRON to call in the correct PATHs
& this q: CronJob not running
Again, I have no issues when I run the wget statement in my terminal. It pulls everything just as expected. My issues is just when I put the wget command in my cron folder the command wont run.

\ denotes line continuation, you should not use it if you have all in one line command, for example
wget --continue \
https://www.example.com
after conversion to one line is
wget --continue https://www.example.com
Regarding cron if you have working command you might create file with it and use it through bash e.g. you might create fetcher.sh with content as follows
wget -P /path/to/catalog https://www.example.com
inside /path/to/catalog and then add
58 23 * * * bash /path/to/catalog/fetcher.sh
where /path/to/catalog is path to existing catalog, then it would download example domain into /path/to/catalog each 2 minutes to midnight.

Related

File Not Found When Running Docker Script But Is Found In Another Docker Script

I am trying to run a script inside a docker that was published by google.
The command I use mounts some datafiles onto the docker in a file called '/input' (inside the docker).
When I run the script, it says that it does not find the input file.
However, I do use the -v flag, and I ran a script that makes sure the input file is there (inside the docker).
So in summary - when I run
find /input -name "*.fasta"
It outputs:
/input/ucsc.hg19.chr20.unittest.fasta
As needed, but when I run the script, it says
./dv-quick-start: 19: ./dv-quick-start: --ref=/input/ucsc.hg19.chr20.unittest.fasta: not found
Full Script:
#!/bin/sh
BIN_VERSION="1.0.0"
INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"
OUTPUT_DIR="${PWD}/quickstart-output"
sudo docker run \
-v "${INPUT_DIR}":"/input" \
-v "${OUTPUT_DIR}":"/output" \
google/deepvariant:"${BIN_VERSION}" \
find /input -name "*.fasta"
sudo docker run \
-v "${INPUT_DIR}":"/input" \
-v "${OUTPUT_DIR}":"/output" \
google/deepvariant:"${BIN_VERSION}" \
/opt/deepvariant/bin/run_deepvariant \
--model_type=WGS \ **Replace this string with exactly one of the following [WGS,WES,PACBIO,HYBRID_PACBIO_ILLUMINA]**
--ref=/input/ucsc.hg19.chr20.unittest.fasta \
--reads=/input/NA12878_S1.chr20.10_10p1mb.bam \
--regions "chr20:10,000,000-10,010,000" \
--output_vcf=/output/output.vcf.gz \
--output_gvcf=/output/output.g.vcf.gz \
--intermediate_results_dir /output/intermediate_results_dir \ **This flag is optional. Set to keep the intermediate results.
Full output:
/input/ucsc.hg19.chr20.unittest.fasta
--ref is required.
Pass --helpshort or --helpfull to see help on flags.
./dv-quick-start: 19: ./dv-quick-start: --ref=/input/ucsc.hg19.chr20.unittest.fasta: not found
I feel there is some misunderstanding on my behalf, and I would appreciate any help.
Should more information be needed to answer the question, let me know.
You have some extraneous text in your shell script that's causing a problem. Delete the "replace this string" and "this flag is optional" text and all of the whitespace before them, making the \ the very last character on those lines.
In a shell script you can break commands across multiple lines using a \. But, the \ must be the absolute very last character in the line; if it's not, it escapes the character that comes after it.
# one line: ls -al $HOME
ls -al \
$HOME
# two lines: ls -al " " more text here; $HOME
ls -al \ more text here
$HOME
In your example you've left some explanatory text in
sudo docker run \
...\
--model_type=WGS \ **Replace this string with exactly one of the following [WGS,WES,PACBIO,HYBRID_PACBIO_ILLUMINA]**
# This is seen as a separate command
--ref=/input/ucsc.hg19.chr20.unittest.fasta \
...
Since the "Replace this string..." text makes the \ not be the absolute last character in the line, it causes the shell to break the command. You then get two commands, a docker run command without the --ref option and what looks like trying to run --ref=... as a separate command; that corresponds to the two errors you get.

Need to run multiple feature file in behave using python

My command in python is
feature_file_folder -f allure_behave.formatter:AllureFormatter -o target/allure-results --no-capture --no-capture-stderr
This runs all the feature files but if i want to run only 2 feature files out of 10 i am having problems i can run 1 or all feature file.
You can mention feature file names while executing.
behave -f allure_behave.formatter:AllureFormatter \
-o target/allure-results \
--no-capture \
--no-capture-stderr \
example1.feature example2.feature
This is what tags are used for. Lets say, hypothetically, the two you want to run are related to logging into a system:
login-from-main-page.feature:
#login-tests
Feature: Test logging in from the main page
Scenario: ...
login-from-mobile.feature
#login-tests
Feature: Test logging in from iOS App
Scenario: ...
Then you would run behave, specifying those tags:
feature_file_folder -f allure_behave.formatter:AllureFormatter \
-o target/allure-results \
--no-capture \
--no-capture-stderr \
--tags login-tests

Execution CMake from within a Bash Script

I've built a script to automate a CMake build of OpenCV4. The relevant part of the script is written as:
install.sh
#!/bin/bash
#...
cd /home/pi/opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
-D ENABLE_NEON=ON \
-D ENABLE_VFPV3=ON \
-D BUILD_TESTS=OFF \
-D OPENCV_ENABLE_NONFREE=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D BUILD_EXAMPLES=OFF ..
This part of the code is first executed from /home/pi/ directory. If I execute these lines within the cli they work and the cmake file is built without error. If I run this same code form a bash script it results in the cmake command resulting in -- Configuring incomplete, errors occurred!.
I believe this is similar to these two SO threads (here and here) in as much as they both describe situations where calling a secondary script from the first script creates a problem (or at least that's what I think they are saying). If that is the case, how can you start a script from /parent/, change to /child/ within the script, execute secondary script (CMake) as though executed from the /child/ directory?
If I've missed my actual problem - highlighting taht would be even more helpful.
Update with Full Logs
Output log for CMakeOutput.log and CMakeError.log as unsuccessfully run from the bash script.
When executed from the cli the successful logs are success_CMakeOutput.log and success_CMakeError.log
Update on StdOut
I looked through the files above and they look the same... Here is the failed screen output (noting the bottom lines) and the successful screen output.
You are running your script as the root user with the /root home directory, while the opencv_contrib directory is in /home/pi directory. The /home/pi is most probably the home directory of the user pi.
Update the:
-D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \
With proper path to opencv_contrib. Either provide opencv_contrib in the home directory of the root user, if you aim to run the script as root, or provide full, non-dependent on HOME, path to opencv_contrib directory.
-D OPENCV_EXTRA_MODULES_PATH=/home/pi/opencv_contrib/modules \

virt-install script in crontab how to control tty

I have a script that creates a virtual machine using virt-install. This script uses a kickstart file for unattended installation. It works perfectly fine when triggered through shell but its throws the following error when triggered through crontab:
error: Cannot run interactive console without a controlling TTY
The VM creation process continues at the backend but in my script it doesn't wait for the virt-install to complete and moves on to the next commands. I wanted my script to wait for the virt-install command to complete its job and then move to the next command. Is there any way i can either get a controll on TTY or make my script to wait for virt-install to complete?
Edit
Here is the virt-install command that my script executes (in case it helps you figuring out the issue):
virt-install --connect=qemu:///system \
--network=bridge:$BRIDGE \
$nic2 \
--initrd-inject=$tmp_ks_file \
--controller type=scsi,model=virtio-scsi \
--extra-args="ks=file:/$(basename $tmp_ks_file) console=tty0 console=ttyS0,115200" \
--name=$img_name \
--disk $libvirt_dir/$img_name.img,size=$disk \
--ram $mem \
--vcpus=2 \
--check-cpu \
--accelerate \
--hvm \
--location=$tree \
--nographics
Thanks in advance,
Kashif
I finally able to cater this issue through two steps:
First of all remove the 'console' related configurations from the virt-install command. See extra-args in above command.
Put some logic to wait for virt-install to complete. I did add shutdown in the post install section of kickstart file so that VM shutoff after it is done installing all the packages. Then in my script i actually 'waited' for VM to go to shutdown state before moving to the next command.
This way I am able to run my script in crontab. It also worked with jenkins too.
Hope this helps someone facing the same issue.

How to call a URL periodically form a linux box (cron job)?

I have a shell access on a Linux box where my Website resides.
I want to call a URL each hour on that Website (it's not a specific PHP file, I have codeigniter has a framework and some Apache redirects, so I really need to call a URL).
I guess I can use wget and crontab? How?
Add this to /etc/crontab
0 * * * * wget -O - -q -t 1 http://www.example.com/cron.php
Alternatively, create a shell script in the /etc/cron.hourly
File: wget
#!/bin/sh
wget -O - -q -t 1 http://www.example.com/cron.php
make sure you chmod +x wget (or whatever filename you picked)

Resources