Install docker and docker-compose on Azure using cloud-Init - azure

I wanted to bootstrap my Azure VM with Docker and Docker-compose using cloud-init. So far I tried something like below.
#cloud-config
package_update: true
package_upgrade: true
groups:
- docker: [default]
runcmd:
- [ sh, -c, "curl -sSL https://get.docker.com/ | sh" ]
- [ sh, -c, "sudo curl -L "https://github.com/docker/compose/releases/download/$(git ls-remote https://github.com/docker/compose | grep refs/tags | grep -oP "[0-9]+\.[0-9][0-9]+\.[0-9]+$" | tail -n 1)/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose" ]
- [ sh, -c, "sudo chmod +x /usr/local/bin/docker-compose" ]
But it doesn't install the docker/ docker-compose and get below error
cloud-init[958]: 2019-02-28 00:51:02,447 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 32: "while scanning a plain scalar
[ 333.241244] cloud-init[958]: in "<unicode string>", line 11, column 32:
[ 333.245521] cloud-init[958]: - [ sh, -c, "sudo curl -L "https://github.com/docker/compos ...
2019/02/28 00:51:04.015216 INFO Daemon Wire protocol version:2012-11-30
[ 333.245619] cloud-init[958]: ^
[ 333.259509] cloud-init[958]: found unexpected ':'
[ 333.259584] 2019/02/28 00:51:04.019282 INFO Daemon Server preferred version:2015-04-05
cloud-init[958]: in "<unicode string>", line 11, column 37:
[ 333.474295] cloud-init[958]: - [ sh, -c, "sudo curl -L "https://github.com/docker/compose/rel ...
[ 333.514672] cloud-init[958]: ^
[ 333.554215] cloud-init[958]: Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details."
[ 333.593764] cloud-init[958]: 2019/02/28 00:51:04.361760 INFO Daemon Waiting for ssh host key be generated at /etc/ssh/ssh_host_rsa_key.pub [1800 attempts remaining, sleeping 1s]
2019-02-28 00:51:02,470 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 32: "while scanning a plain scalar
[ 333.705350] cloud-init[958]: in "<unicode string>", line 11, column 32:
[ 333.731074] cloud-init[958]: - [ sh, -c, "sudo curl -L "https://github.com/docker/compos ...
[ 333.769118] cloud-init[958]: ^
[ 333.789992] cloud-init[958]: found unexpected ':'
[ 333.808855] cloud-init[958]: in "<unicode string>", line 11, column 37:
[ 333.831502] cloud-init[958]: - [ sh, -c, "sudo curl -L "https://github.com/docker/compose/rel ...
[ 333.863028] cloud-init[958]: ^
[ 333.885648] cloud-init[958]: Please check http://pyyaml.org/wiki/YAMLColonInFlowContext for details."
[ 333.915174] cloud-init[958]: 2019-02-28 00:51:02,470 - util.py[WARNING]: Failed at merging in cloud config part from part-001
Apart from it I doubt how can I add Azure auto created default user (which was provided from UI) to docker group.

Just tested by myself on Azure using Ubuntu 18.04-LTS:
#cloud-config
package_upgrade: true
packages:
- docker.io
- docker-compose
# create the docker group
groups:
- docker
# assign a VM's default user, which is mydefaultuser, to the docker group
users:
- default
- name: mydefaultuser
groups: docker
runcmd can be left empty for this setup.

you have " inside of other " which doesnt work without escaping. as for the user. I think default user created has id of 1000, you can just use that (probably test this hypothesis before).

After spending some time I solved it myself.
#cloud-config
package_update: true
# Setup swap memory
disk_setup:
ephemeral0:
table_type: mbr
layout: [66, [33, 82]]
overwrite: True
fs_setup:
- device: ephemeral0.1
filesystem: ext4
- device: ephemeral0.2
filesystem: swap
mounts:
- ["ephemeral0.1", "/mnt"]
- ["ephemeral0.2", "none", "swap", "sw", "0", "0"]
# Enable Docker's swap limit support
bootcmd:
- [ sh, -c, 'sudo echo GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1" >> /etc/default/grub' ]
- [ sh, -c, 'sudo update-grub' ]
- [ cloud-init-per, once, mymkfs, mkfs, /dev/vdb ]
# Install docker and docker-compose
runcmd:
- [ sh, -c, 'curl -sSL https://get.docker.com/ | sh' ]
- [ sh, -c, 'sudo curl -L https://github.com/docker/compose/releases/download/$(curl -s https://api.github.com/repos/docker/compose/releases/latest | grep "tag_name" | cut -d \" -f4)/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose' ]
- [ sh, -c, 'sudo chmod +x /usr/local/bin/docker-compose' ]
# Add default auto created user to docker group
system_info:
default_user:
groups: [docker]

Related

Cronjob doesn't execute "bundle exec rake -f ..."

I am trying to automate redmine's issue creation via email within a docker container using cronjob.
I use a slightly modified version of the offical redmine docker image with cron installed and some additional environment variables.
Manually, when I run the following line in the terminal, everything works fine.
# cd /usr/src/redmine && bundle exec rake -f /usr/src/redmine/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=$IMAP_HOST port=$IMAP_PORT ssl=1 username=$SMTP_USER password=$SMTP_PASSWORD -r "openssl" -E "OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE"
(eval):1: warning: already initialized constant OpenSSL::SSL::VERIFY_PEER
W, [2020-12-11T14:18:43.730617 #86] WARN -- : Creating scope :system. Overwriting existing method Enumeration.system.
W, [2020-12-11T14:18:45.264865 #86] WARN -- : Creating scope :sorted. Overwriting existing method Group.sorted.
W, [2020-12-11T14:18:45.273745 #86] WARN -- : Creating scope :sorted. Overwriting existing method User.sorted.
I, [2020-12-11T14:18:46.534306 #86] INFO -- : Performing ActionMailer::DeliveryJob (Job ID: eda80c3a-dc7d-4dcc-9fb9-ad25414cbad2) from Inline(mailers) with arguments: "Mailer", "issue_edit", "deliver_now", #<GlobalID:0x000055f759eb7818 #uri=#<URI::GID gid://redmine-app/User/1>>, #<GlobalID:0x000055f759eb6940 #uri=#<URI::GID gid://redmine-app/Journal/1>>
I, [2020-12-11T14:18:46.560561 #86] INFO -- : Rendering mailer/issue_edit.text.erb within layouts/mailer
I, [2020-12-11T14:18:46.597403 #86] INFO -- : Rendered mailer/_issue.text.erb (34.9ms)
I, [2020-12-11T14:18:46.597516 #86] INFO -- : Rendered mailer/issue_edit.text.erb within layouts/mailer (36.9ms)
I, [2020-12-11T14:18:46.600656 #86] INFO -- : Rendering mailer/issue_edit.html.erb within layouts/mailer
I, [2020-12-11T14:18:46.613771 #86] INFO -- : Rendered mailer/_issue.html.erb (0.9ms)
I, [2020-12-11T14:18:46.613876 #86] INFO -- : Rendered mailer/issue_edit.html.erb within layouts/mailer (13.1ms)
I, [2020-12-11T14:18:46.620620 #86] INFO -- : Performed ActionMailer::DeliveryJob (Job ID: eda80c3a-dc7d-4dcc-9fb9-ad25414cbad2) from Inline(mailers) in 86.17ms
I, [2020-12-11T14:18:46.621236 #86] INFO -- : Enqueued ActionMailer::DeliveryJob (Job ID: eda80c3a-dc7d-4dcc-9fb9-ad25414cbad2) to Inline(mailers) with arguments: "Mailer", "issue_edit", "deliver_now", #<GlobalID:0x000055f75a681d00 #uri=#<URI::GID gid://redmine-app/User/1>>, #<GlobalID:0x000055f75a681530 #uri=#<URI::GID gid://redmine-app/Journal/1>>
I, [2020-12-11T14:18:46.622072 #86] INFO -- : MailHandler: issue #1 updated by Redmine Admin
Now, I want to create a cronjob that executes this line, say every minute.
I added the cronjob below, but the desired outcome remains absent.
# crontab -l
* * * * * cd /usr/src/redmine && bundle exec rake -f /usr/src/redmine/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=$IMAP_HOST port=$IMAP_PORT ssl=1 username=$SMTP_USER password=$SMTP_PASSWORD -r "openssl" -E "OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE"
From the /var/log/cron.log file I know, that the cronjob is executed.
Dec 11 14:43:01 26c0ec148cea CRON[155]: (root) CMD (cd /usr/src/redmine && bundle exec rake -f /usr/src/redmine/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=$IMAP_HOST port=$IMAP_PORT ssl=1 username=$SMTP_USER password=$SMTP_PASSWORD -r "openssl" -E "OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE")
If I append &> /home/redmine/cron.log to the job, an empty file gets created.
I assume cron somehow breaks the command.
I already tried to use the full paths to bundle (/usr/local/bin/bundle) and rake (/usr/local/bundle/bin/rake) without success.
Can you help me to fix this cronjob or tell me how to debug this further?
Setup:
Dockerfile:
FROM redmine:latest
ENV SMTP_AUTHENTICATION="login" \
SMTP_DOMAIN="gmail.com" \
SMTP_HOST="smtp.gmail.com" \
SMTP_PASSWORD="<password>" \
SMTP_PORT="587" \
SMTP_TLS="true" \
SMTP_USER="<email>" \
IMAP_HOST="imap.gmail.com" \
IMAP_PORT="993"
RUN apt-get update > /dev/null && apt-get install -y cron vim rsyslog > /dev/null
RUN service rsyslog start
RUN crontab -l | { cat; echo '* * * * * cd /usr/src/redmine && bundle exec rake -f /usr/src/redmine/Rakefile redmine:email:receive_imap RAILS_ENV="production" host=$IMAP_HOST port=$IMAP_PORT ssl=1 username=$SMTP_USER password=$SMTP_PASSWORD -r "openssl" -E "OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE"'; } | crontab -
docker-compose.yml:
version: '3.1'
services:
redmine:
#image: redmine
build: '.'
restart: always
ports:
- 80:3000
volumes:
- 'redmine_data:/redmine'
environment:
REDMINE_DB_MYSQL: db
REDMINE_DB_PASSWORD: <password>
REDMINE_SECRET_KEY_BASE: <secretkey>
db:
image: mysql:5.7
restart: always
volumes:
- 'db_data:/db'
environment:
MYSQL_ROOT_PASSWORD: <password>
MYSQL_DATABASE: redmine
volumes:
db_data:
driver: local
redmine_data:
driver: local

Snakemake gives InputFunctionException when using --profile slurm

I'm creating a pipeline using snakemake to call methylation in nanopore sequencing data. I've run snakenake using the --dryrun option and the dag is constructed successfully. But when I add the option --profile slurm, I get the following error:
(nanopolish) [danielle.perley#talonhead2 nanopolish-CpG-calling]$ snakemake -np --use-conda --profile slurm test_data/20-001-002/20-001-002_fastq_pass.gz
Building DAG of jobs...
Job counts:
count jobs
1 combine_tech_reps
1
InputFunctionException in line 32 of /home/danielle.perley/nanopolish-CpG-calling/Snakefile:
Error:
SyntaxError: invalid syntax (<string>, line 1)
Wildcards:
sample=20-001-002
Traceback:
File "/home/danielle.perley/miniconda3/envs/nanopolish/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 115, in run_jobs
File "/home/danielle.perley/miniconda3/envs/nanopolish/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 120, in run
File "/home/danielle.perley/miniconda3/envs/nanopolish/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 131, in _run
File "/home/danielle.perley/miniconda3/envs/nanopolish/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 151, in printjob
File "/home/danielle.perley/miniconda3/envs/nanopolish/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 137, in printjob
Line 33 is rule combine_tech_reps in my snakefile. (I'm only showing the first part of my snakefile here)
from snakemake.utils import validate
import pandas as pd
import os.path
import glob
configfile: "config.yaml"
samples_df = pd.read_table(config["samples"],sep = '\t')
samples_df = samples_df.set_index("Sample")
samples = list(samples_df.index.unique())
wildcard_constraints:
sample = "|".join(samples)
def get_fast5(wildcards):
f5 = glob.glob(os.path.join(config["raw_data"],wildcards.sample,"2*","fast5_pass"))
return(f5)
localrules: all,build_index
rule all:
input:
expand("results/Methylation/{sample}_frequency.tsv",sample=samples),
expand("results/alignments/{sample}_flagstat.txt",sample=samples),
expand("resources/QC/{sample}_pycoQC.json",sample=samples),
expand("results/QC/{sample}_pycoQC.html",sample=samples),
"report/multiQC.html"
rule combine_tech_reps:
input:
fqs = lambda wildcards: glob.glob(os.path.join(config["raw_data"],"{sample}","2*","{sample}_fastq_pass.gz").format(sample=wildcards.sample))
output:
fq = os.path.join(config["raw_data"],"{sample}","{sample}_fastq_pass.gz")
shell: """
zcat {input} > {output}
"""
I have a slurm profile file in the directory:
~/.config/snakemake/slurm/config.yaml
jobs: 10
cluster: "sbatch -p talon -t {resources.time} --mem={resources.mem} -c {resources.cpus} -o logs_slurm/{rule}_{wildcards} -e logs_slurm/{rule}_{wildcards}"
default-resources: [cpus=1, mem=2000, time=10:00]
use-conda: true
I'd really like to use this pipeline on our HPC, but I'm not sure what's causing this error.
I was able to solve my problem with the help of this post:
InputFunctionException: unexpected EOF while parsing
By adding the verbose flag:
snakemake -np --verbose --use-conda --profile slurm test_data/20-001-002/20-001-002_fastq_pass.gz
I could see that snakemake was having issues with the default resources:
10:00
^
Changing the default resources line of my config.yaml file:
default-resources: [cpus=1, mem=2000, time=600]
removed the error.
I am not sure if default-resources is a valid key in the config.
What happens if you try this as config.yaml:
jobs: 10
cluster: "sbatch -p talon -t {resources.time} --mem={resources.mem} -c {resources.cpus} -o logs_slurm/{rule}_{wildcards} -e logs_slurm/{rule}_{wildcards}"
use-conda: true
__default__:
time: 10
cpus: 1
mem: 2GB

How to quote part of a subprocess.run list? [duplicate]

This question already has answers here:
Python Subprocess: Unable to Escape Quotes
(2 answers)
Closed last year.
I need to quote part of the rsync line that subprocess.run uses that contains the ssh parameters, unfortunately nothing I have tried has worked so far.
Can someone please advise me on the correct way to quote the ssh parameters, so that it will run under rsync.
At first I had a list of lists that got passed to subprocess.run, that fails with:
Traceback (most recent call last):
File "./tmp.py", line 20, in <module>
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
File "/usr/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not list
Flatten it to an ordinary list:
Unexpected remote arg: example.com:/var/log/maillog
rsync error: syntax or usage error (code 1) at main.c(1361) [sender=3.1.2]
Which makes sense, as part of the command line for rsync needs to be quoted.
So I try to quote it:
rsync: Failed to exec /usr/bin/ssh -F /home/rspencer/.ssh/config -o PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask -o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o Compression=yes: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(85) [Receiver=3.1.2]
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in IPC code (code 14) at io.c(235) [Receiver=3.1.2]
Which is due, I expect, to it being a string instead of a list. Although I'm guessing and that does not make complete sense to me.
Summarized code of my last attempt:
#!/usr/bin/python3
import subprocess
ssh_args = [
"-F",
"/home/rspencer/.ssh/config",
"-o",
"PreferredAuthentications=publickey",
"-o",
"StrictHostKeyChecking=accept-new",
"-o",
"TCPKeepAlive=yes",
"-o",
"ServerAliveInterval=5",
"-o",
"ServerAliveCountMax=24",
"-o",
"ConnectTimeout=30",
"-o",
"ExitOnForwardFailure=yes",
"-o",
"ControlMaster=autoask",
"-o",
"ControlPath=/run/user/1000/foo-ssh-master-%C",
"-l",
"root",
"-p",
"234",
]
rsync_params = []
src = "example.com:/var/log/maillog"
dest = "."
# Build SSH command
ssh_cmd = ["/usr/bin/ssh"] + ssh_args
# Use basic compression
ssh_cmd.extend(["-o", "Compression=yes"])
ssh_cmd = " ".join(ssh_cmd)
ssh_cmd = f'"{ssh_cmd}"'
# Build rsync command
rsync_cmd = ["/usr/bin/rsync", "-vP", "-e", ssh_cmd] + rsync_params + [src, dest]
# Run rsync
process = subprocess.run(rsync_cmd, stderr=subprocess.PIPE)
if process.returncode != 0:
print(process.stderr.decode("UTF-8").strip())
What the correct command would look like on the command line:
/usr/bin/rsync -vP -e "/usr/bin/ssh -F /home/rspencer/.ssh/config -o \
PreferredAuthentications=publickey -o StrictHostKeyChecking=accept-new -o \
TCPKeepAlive=yes -o ServerAliveInterval=5 -o ServerAliveCountMax=24 -o \
ConnectTimeout=30 -o ExitOnForwardFailure=yes -o ControlMaster=autoask \
-o ControlPath=/run/user/1000/foo-ssh-master-%C -l root -p 234 -o \
Compression=yes" example.com:/var/log/maillog .
Turns out the trick is to not try to quote it.
I removed the following line and it worked without further modification:
ssh_cmd = f'"{ssh_cmd}"'
I've read so much documentation and missed it until asking the question. Murphy.
Rereading the post "How not to quote argument in subprocess?" and finally understanding what Greg Hewgill was saying helped me. I blame lack of sleep.
"If you use quotes on the shell command line, then put the whole contents in one element of args (without the quotes). ..." - Greg Hewgill

Unable to execute script using user_data

I'm trying to execute a script once while ec2 boots up so have following in instance.tf.
resource "aws_instance" "test" {
ami = "i-33434"
user_data = "${data.template_file.user-data.rendered}"
}
data "template_file" "user-data" {
template = "${file("templates/init.tpl")}"}
And have init.tpl file created under templates folder with below content:
#!/bin/bash
sudo mkdir /ecs
mkfs -t ext4 /dev/xvdt10
mkfs -t ext4 /dev/xvdt11
mkdir /ecs/folder1
mkdir /ecs/folder2
mount /dev/xvdt10 /ecs/folder1
mount /dev/xvdt11 /ecs/folder2
echo /dev/xvdt10 /ecs/folder1 ext4 defaults,nofail 0 2 >> /etc/fstab
echo /dev/xvdt11 /ecs/folder2 ext4 defaults,nofail 0 2 >> /etc/fstab

Integrity Measurement Architecture(IMA) & Linux Extended Verification Module (EVM)

I am trying to activate IMA appraisal & EVM modules.
After compiling linux kernel 3.10.2 on my bt5R3 and setting kernel boot option in a first time like this:
GRUB_CMDLINE_LINUX="rootflags=i_version ima_tcb ima_appraise=fix ima_appraise_tcb evm=fix"
and after running this command to generate xattr security.ima and security.evm
find / \( -fstype rootfs -o -fstype ext4 \) -type f -uid 0 -exec head -c 1 '{}' \;
like this:
GRUB_CMDLINE_LINUX="rootflags=i_version ima_tcb ima_appraise=enforce ima_appraise_tcb evm=enforce"
I try to create digital signature of xattr like it's recommended on this tutorial
Tutorial to IMA & EVM
Every steps have been followed, creating RSA keys, loading them early at boot in initramfs with keyctl.
Session Keyring
-3 --alswrv 0 65534 keyring: _uid_ses.0
977514165 --alswrv 0 65534 \_ keyring: _uid.0
572301790 --alswrv 0 0 \_ user: kmk-user
126316032 --alswrv 0 0 \_ encrypted: evm-key
570886575 --alswrv 0 0 \_ keyring: _ima
304346597 --alswrv 0 0 \_ keyring: _evm
However as soon as I reboot my OS when I try to read a signed and hashed file I get the error "Permission Denied"
Running dmesg tells me :
[ 5461.175996] type=1800 audit(1375262160.913:57): pid=1756 uid=0 auid=4294967295 ses=4294967295 op="appraise_data" cause="**invalid-HMAC**" comm="sh" name="/root/Desktop/new.sh" dev="sda1" ino=546526 res=0
Have you any idea why i get invalid HMAC ?
They keys are loaded like the tutorial says...
#!/bin/sh -e
PREREQ=""
# Output pre-requisites
prereqs()
{
echo "$PREREQ"
}
case "$1" in
prereqs)
prereqs
exit 0
;;
esac
grep -q "ima=off" /proc/cmdline && exit 1
mount -n -t securityfs securityfs /sys/kernel/security
IMA_POLICY=/sys/kernel/security/ima/policy
LSM_POLICY=/etc/ima_policy
grep -v "^#" $LSM_POLICY >$IMA_POLICY
# import EVM HMAC key
keyctl show |grep -q kmk || keyctl add user kmk "testing123" #u
keyctl add encrypted evm-key "load `cat /etc/keys/evm-key`" #u
#keyctl revoke kmk
# import Module public key
mod_id=`keyctl newring _module #u`
evmctl import /etc/keys/pubkey_evm.pem $mod_id
# import IMA public key
ima_id=`keyctl newring _ima #u`
evmctl import /etc/keys/pubkey_evm.pem $ima_id
# import EVM public key
evm_id=`keyctl newring _evm #u`
evmctl import /etc/keys/pubkey_evm.pem $evm_id
# enable EVM
echo "1" > /sys/kernel/security/evm
# enable module checking
#echo "1" > /sys/kernel/security/module_check
Thanks for your help
Solved, new kernel use HMAC v2 and you have to activate asymmetric key when you compile kernel.
cat .config should have this entries:
CONFIG_EVM_HMAC_VERSION=2
CONFIG_ASYMMETRIC_KEY_TYPE=y
Then when you hash or sign a file use
evmctl -u - -x --imasig/--imahash $file
As well you should have create the asymetric keys and load them in _evm and _ima keyring with keyctl with initramfs.

Resources