cloud-init is powerful to inject user-data in to VM instance, and its existing module provides lots of possibility.
While to make it more easy to use, I want to define my own tag like coreos below, see detail in running coreos in openstack
#cloud-config
coreos:
etcd:
# generate a new token for each unique cluster from https://discovery.etcd.io/new
discovery: https://discovery.etcd.io/<token>
# multi-region and multi-cloud deployments need to use $public_ipv4
addr: $private_ipv4:4001
peer-addr: $private_ipv4:7001
units:
- name: etcd.service
command: start
- name: fleet.service
command: start
So I could have something like below using my own defined tag/config myapp
#cloud-config
myapp：
admin: admin
database: 192.168.2.3
I am new to cloud-init, is it called module ? it is empty in document http://cloudinit.readthedocs.org/en/latest/topics/modules.html
Can you provide some information to describe how I can write my own module ?
You need to write a "cc" module in a suitable directory, and modify a few configurations. It is not terribly easy, but certainly doable (we use it a lot).
Find the directory for cloudconfig modules. On Amazon Linux, this is /usr/lib/python2.6/site-packages/cloudinit/config/, but the directory location differs in different cloud init versions and distributions. The easiest way to find this is to find a file named cc_mounts.py.
Add a new file there, in your case cc_myapp.py. Copy some existing script as a base to know what to write there. The important function is def handle(name,cfg,cloud,log,args): which is basically the entrypoint for your script.
Implement your logic. The cfg parameter has a python object which is the parsed YAML config file. So for your case you would do something like:
myapp = cfg.get('myapp')
admin = myapp.get('admin')
database = myapp.get('database')
Ensure your script gets called by cloud-init. If your distribution uses the standard cloud-init setup, just adding the file might work. Otherwise you might need to add it to /etc/cloud/cloud.cfg.d/defaults.cfg or directly to /etc/cloud/cloud.cfg. There are keys called cloud_init_modules, cloud_config_modules, etc. which correspond to different parts of the init process where you can get your script run. If this does not work straight out of the box, you'll probably have to do a bit of investigation to find out how the modules are called on your system. For example, Amazon Linux used to have a hardcoded list of modules inside the init.d script, ignoring any lists specified in configuration files.
Also note that by default your script will run only once per instance, meaning that rerunning cloud-init will not run your script again. You need to either mark the script as being per boot by setting frequency to always in the configuration file listing your module, or remove the marker file saying that the script has run, which lives somewhere under /var/lib/cloud like in /var/lib/cloud/instances/i-86ecc763/sem/config_mounts.
paste my note for you:
config: after installed cloud-init in VM,if u want to have root permission to access with passwd, do simple config below
modify /etc/cloud/cloud.cfg like below
users：
- defaults
disable_root：0
ssh_pwauth： 1
Note: ssh_pwauth: "it will modify PasswordAuthentication in sshd_config automatically, 1 means yes
Usage:
the behavior of cloud-init can configured using user data. user data can be filled by user during the start of instance (user data is limited to 16K).
Mainly there are several ways to do (tested):
user-data script
$ cat myscript.sh
#!/bin/sh
echo "Hello World. The time is now $(date -R)!" | tee /root/output.txt
when starting instance, add parameter --user-data myscript.sh, and the instance will run the script once during startup and only once.
cloud config syntax:
It is YAML-based, see http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/files/head:/doc/examples/
run script
#cloud-config
runcmd:
- [ ls, -l, / ]
- [ sh, -xc, "echo $(date) ': hello world!'" ]
- [ sh, -c, echo "=========hello world'=========" ]
- ls -l /root
- [ wget, "http://slashdot.org", -O, /tmp/index.html ]
change hostname, password
#cloud-config
chpasswd:
list: |
root:123456
expire: False
ssh_pwauth: True
hostname: test
include format
run url script, it will download URL script and execute them sequence, this can help to manage the scripts centrally.
#include
http://hostname/script1
http://hostname/scrpt2
Related
I have a sshd_config configuration file which contains commented as well as uncommented lines. I want to comment all the uncommented lines in that file using puppet. Is there any optimal/simple way to do this? Or is there a way to run bash command (maybe sed to replace) via puppet? I am not sure that using bash command is a right approach.
It would be really helpful is someone guides me with this. Thanks in advance!
Is there any optimal/simple way to do this?
There is no built-in resource type or well-known module that specifically ensures that non-blank lines of a file start with a # character.
Or is there a way to run bash command (maybe sed to replace) via puppet?
Yes, the Exec resource type. That's your best bet short of writing a custom resource type.
I am not sure that using bash command is a right approach.
In a general sense, it's not. Appropriate, specific resource types are better than Exec. But when you don't have a suitable one and can't be bothered to make one, Exec is available.
It might look like this:
# The file to work with, so that we don't have to repeat ourselves
$target_file = '/etc/ssh/sshd_config'
exec { "Comment uncommented ${target_file} lines":
# Specifying the command in array form avoids complicated quoting or any
# risk of Puppet word-splitting the command incorrectly
command => ['sed', '-i', '-e', '/^[[:space:]]*[^#]/ s/^/# /', $target_file],
# If we didn't specify a search path then we would need to use fully-qualified
# command names in 'command' above and 'onlyif' below
path => ['/bin', '/usr/bin', '/sbin', '/usr/sbin'],
# The file needs to be modified only if it contains any non-blank, uncommented
# lines. Testing that via an 'onlyif' ensures that Puppet will not
# run 'sed' or (more importantly) report the file changed when it does
# not initially contain any lines that need to be commented
onlyif => [['grep', '-q', '^[[:space:]]*[^#]', $target_file]],
# This is the default provider for any target node where the rest of this
# resource would work anyway. Specifying it explicitly will lead to a more
# informative diagnostic if there is an attempt to apply this resource to
# a system to which it is unsuited.
provider => 'posix',
}
That does not rely on bash or any other shell to run the commands, but it does rely on sed and grep being available in one of the specified directories. In fact, it relies specifically on GNU sed or one that supports an -i option with the same semantics. Notably, that does not include BSD-style sed, such as you will find on macOS.
I'm putting together a snakemake slurm workflow and am having trouble with my working directory becoming cluttered with slurm output files. I would like my workflow to, at a minimum, direct these files to a 'slurm' directory inside my working directory. I currently have my workflow set up as follows:
config.yaml:
reads:
1:
2:
samples:
15FL1-2: /datasets/work/AF_CROWN_RUST_WORK/2020-02-28_GWAS/data/15FL1-2
15Fl1-4: /datasets/work/AF_CROWN_RUST_WORK/2020-02-28_GWAS/data/15Fl1-4
cluster.yaml:
localrules: all
__default__:
time: 0:5:0
mem: 1G
output: _{rule}_{wildcards.sample}_%A.slurm
fastqc_raw:
job_name: sm_fastqc_raw
time: 0:10:0
mem: 1G
output: slurm/_{rule}_{wildcards.sample}_{wildcards.read}_%A.slurm
Snakefile:
configfile: "config.yaml"
workdir: config["work"]
rule all:
input:
expand("analysis/fastqc_raw/{sample}_R{read}_fastqc.html", sample=config["samples"],read=config["reads"])
rule clean:
shell:
"rm -rf analysis logs"
rule fastqc_raw:
input:
'data/{sample}_R{read}.fastq.gz'
output:
'analysis/fastqc_raw/{sample}_R{read}_fastqc.html'
log:
err = 'logs/fastqc_raw/{sample}_R{read}.out',
out = 'logs/fastqc_raw/{sample}_R{read}.err'
shell:
"""
fastqc {input} --noextract --outdir 'analysis/fastqc_raw' 2> {log.err} > {log.out}
"""
I then call with:
snakemake --jobs 4 --cluster-config cluster.yaml --cluster "sbatch --mem={cluster.mem} --time={cluster.time} --job-name={cluster.job_name} --output={cluster.output}"
This does not work, as the slurm directory does not already exist. I don't want to manually make this before running my snakemake command, that will not work for scalability. Things I've tried, after reading every related question, are:
1) simply trying to capture all the output via the log within the rule, and setting cluster.output='/dev/null'. Doesn't work, the info in the slurm output isn't captured as it's not output of the rule exactly, its info on the job
2) forcing the directory to be created by adding a dummy log:
log:
err = 'logs/fastqc_raw/{sample}_R{read}.out',
out = 'logs/fastqc_raw/{sample}_R{read}.err'
jobOut = 'slurm/out.err'
I think this doesn't work because sbatch tries to find the slurm folder before implementing the rule
3) allowing the files to be made in the working directory, and adding bash code to the end of the rule to move the files into a slurm directory. I believe this doesn't work because it tries to move the files before the job has finished writing to the slurm output.
Any further ideas or tricks?
You should be able to suppress these outputs by calling sbatch with --output=/dev/null --error=/dev/null. Something like this:
snakemake ... --cluster "sbatch --output=/dev/null --error=/dev/null ..."
If you want the files to go to a directory of your choosing you can of course change the call to reflect that:
snakemake ... --cluster "sbatch --output=/home/Ensa/slurmout/%j.out --error=/home/Ensa/slurmout/%j.out ..."
So this is how I solved the issue (there's probably a better way, and if so, I hope someone will correct me). Personally I will go to great lengths to avoid hard-coding anything. I use a snakemake profile and an sbatch script.
First, I make a snakemake profile that contains a line like this:
cluster: "sbatch --output=slurm_out/slurm-%j.out --mem={resources.mem_mb} -c {resources.cpus} -J {rule}_{wildcards} --mail-type=FAIL --mail-user=me#me.edu"
You can see the --output parameter to redirect the slurm output files to a subdirectory called slurm_out in the current working directory. But AFAIK, slurm can't create that directory if it doesn't exist. So...
Next I make a small sbatch script whose only job is to make the subdirectory, then call the sbatch script to submit the workflow. This "wrapper" looks like:
#!/bin/bash
mkdir -p ./slurm_out
sbatch snake_submit.sbatch
And finally, the snake_submit.sbatch looks like:
#!/bin/bash
ml snakemake
snakemake --profile <myprofile>
In this case both the wrapper and the sbatch script that it calls will have their slurm out files in the current working directory. I prefer it that way because it's easier for me to locate them. But I think you could easily re-direct by adding another #SBATCH --output parameter to the snake_submit.sbatch script (but not the wrapper, then it's turtles all the way down, you know?).
I hope that makes sense.
As several other users who have posted to StackOverflow, I ran into problems with file provisioners, and the Terraform documentation says we should not rely on them.
What's the best way to work around file provisioners - specifically for local config files and scripts?
One solution, which works very well and does not require a direct connection to the instance, is to use the userdata as a hook to "install" the files from the base64 version of the file(s).
We can actually embed the files as base64 strings in the userdata initialization scripts. This works for both Windows and Linux instances in AWS, and is compatible also with having a userdata script run on startup.
Solution Description:
During terraform plan, encode whatever local files you need as base64 strings using terraform functions base64encode(file("path/to/file")).
(Optional) Save a marker file (_INIT_STARTED_) at the start of userdata execution; this file will have the creation timestamp of when the userdata execution started.
Before running the actual userdata script, write the base64 strings to text files. (The actual command varies between windows and linux, see examples below.)
Run the userdata script itself (userdata_win.bat or userdata_lin.sh)
(Optional) Finally, save a second marker file (_INIT_COMPLETE_) which will have the creation timestamp of when the userdata script completed. (The absence of this file is also helpful to detect script failures and/or still-running scripts after logging into the instance.)
For AWS Linux instances:
data "template_file" "userdata_lin" {
template = <<EOF
#!/bin/bash
mkdir -p /home/ubuntu/setup-scripts
cd /home/ubuntu/setup-scripts
touch _INIT_STARTED_
echo ${base64encode(file("${path.module}/userdata_lin.sh"))} | base64 --decode > userdata.sh
echo ${base64encode(file("${path.module}/config.json"))} | base64 --decode > config.json
${file("${path.module}/userdata_lin.sh")}
sudo chmod 777 *
touch _INIT_COMPLETE_
EOF
}
# ...
resource "aws_instance" "my_linux_instance" {
# ...
user_data = data.template_file.userdata_lin.rendered
}
For AWS Windows instances:
data "template_file" "userdata_win" {
template = <<EOF
<script>
mkdir C:\Users\Administrator\setup-scripts
cd C:\Users\Administrator\setup-scripts
echo "" > _INIT_STARTED_
echo ${base64encode(file("${path.module}/userdata_win.bat"))} > tmp1.b64 && certutil -decode tmp1.b64 userdata.bat
echo ${base64encode(file("${path.module}/config.json"))} > tmp2.b64 && certutil -decode tmp2.b64 config.json
${file("${path.module}/userdata_win.bat")}
echo "" > _INIT_COMPLETE_
</script>
<persist>false</persist>
EOF
}
# ...
resource "aws_instance" "my_windows_instance" {
# ...
user_data = data.template_file.userdata_win.rendered
}
I want to learn to create a wrapper around a program in linux. How does one do this? A tutorial reference web-page/link or example will do. To clarify what I want to learn, I will explain with an example.
I use vim for editing text files. And use rcs as my simple revision control system. rcs allows you to check-in and checkout-files. I would like to create a warpper program named vir which when I type in the shell as:
$ vir temp.txt
will load the file temp.txt into rcs with ci -u temp.txt and then allows me to edit the file using vim.
When I get out and go back in, It will need to check out the file first, using ci -u temp.txt and allow me to edit the file as one normally does with vim, and then when I save and exit, it should check-in the file using co -u temp.txt and as part of that I should be able to add a version control comment.
Basically, all I want to be doing on the command line is:
$ vir temp.txt
as one would with vim. And the wrapper should take care of the version control for me.
Take a look at rcsvers.vim, a vim plugin for automatically saving versions in RCS; you could modify that. There are also other RCS plugins for vim at vim.org
I have a wrapper to enhance the ping command (using zsh) it could, maybe help you:
# ping command wrapper - Last Change: out 27 2019 18:47
# source: https://www.cyberciti.biz/tips/unix-linux-bash-shell-script-wrapper-examples.html
ping(){
# Name: ping() wrapper
# Arg: (url|domain|ip)
# Purpose: Send ping request to domain by removing urls, protocol, username:pass using system /usr/bin/ping
local array=( $# ) # get all args in an array
local host=${array[-1]} # get the last arg
local args=${array[1,-2]} # get all args before last arg in $#
#local _ping="/usr/bin/ping"
local _ping="/bin/ping"
local c=$(_getdomainnameonly "$host")
[ "$host" != "$c" ] && echo "Sending ICMP ECHO_REQUEST to \"$c\"..."
# pass args and host
# $_ping $args $c
# default args for ping
$_ping -n -c 2 -i 1 -W1 $c
}
_getdomainnameonly(){
# Name: _getdomainnameonly
# Arg: Url/domain/ip
# Returns: Only domain name
# Purpose: Get domain name and remove protocol part, username:password and other parts from url
# get url
local h="$1"
# upper to lowercase
local f="${h:l}"
# remove protocol part of hostname
f="${f#http://}"
f="${f#https://}"
f="${f#ftp://}"
f="${f#scp://}"
f="${f#scp://}"
f="${f#sftp://}"
# Remove username and/or username:password part of hostname
f="${f#*:*#}"
f="${f#*#}"
# remove all /foo/xyz.html*
f=${f%%/*}
# show domain name only
echo "$f"
}
What it hides the local ping using a function called "ping", so if your script has precedence on your path it will find at first the function ping. Then inside the script I define an internal variable called ping that points out to the real ping command:
local _ping="/bin/ping"
You can also notice that the args are stored in one array.
Goal: Run a custom bash script as a service in linux and allow me to pass commands to it like you can to most other services via terminal/bash.
I already have a script set up and tested that can do what I need it to, except I can't figure out how to be able to pass commands to it like you would other services.
Example: nano opens up the nano editor to read specified file.
I want to perform 'fan on', which will call the new service 'fan' and execute the 'on' command.
Bonus: Being able to save the variables into a config file to be modified later on. But for now, I have the variables set at top of script so its not totally necessary.
Current Unit File:
[Unit]
Description=Fan control Service
[Service]
Type=simple
Restart=always
RestartSec=30
ExecStart=/home/pi/Documents/FanControl.sh
User=pi
[Install]
WantedBy=multi-user.target
Script:
#!/bin/bash
#########################################################
# User Settings
#GPIO Pin Number to use to control fan transistor.
fanpin=3
#Celsius temp to turn fan on/off
offtemp=55
ontemp=60
#Turn on the looping script automatically or not
autostart=TRUE
#Determine how often to scan temp and turn fan on/off if in auto
sleepinterval=10
#########################################################
#Misc Variables used in script - Leave these alone - Base Settings
MaxTemp=0
FanState=OFF
mode=MANUAL
auto=FALSE
#########################################################
# Functions Described Below
#########################################################
before-start() {
# Check if gpio is already exported
if [ ! -d /sys/class/gpio/gpio$fanpin ]
then
#Export the Pin
echo $fanpin > /sys/class/gpio/export
sleep 1 ;# Short delay while GPIO permissions are set up
echo Fan Pin Exported Successfully.
# Setup the pin as an output
sudo echo "out" > /sys/class/gpio/gpio$fanpin/direction
fi
}
#Function to turn fan on
on() {
# Sets FanPin to high
echo "1" > /sys/class/gpio/gpio$fanpin/value
FanState=ON
mode=Manual
auto=FALSE
echo Fan Turned on -- Mode set to Manual.
echo
}
#Function to turn fan off
off() {
# Sets FanPin to low
echo "0" > /sys/class/gpio/gpio$fanpin/value
FanState=OFF
mode=Manual
auto=FALSE
echo Fan Turned off -- Mode set to Manual.
echo
}
#Function to set the variables to values
#Haven't actually tested this function yet
Set() {
if $2 = "ontemp"
then
ontemp=$2
else
$2=$3
fi
}
#########################################################
# Begin Service Execution Code
#########################################################
#Don't know whow to write this section to keep it running as a service
#But it works well for testing purposes
before-start
on
sleep 3
off
echo
read -p "Select an action": Q
$Q
echo
action="$1"
serviceName="Fan-Control Service"
echo Exiting Fan Service
I can use systemctl daemon-reload and it can load the service.
I can use 'systemctl start fan' and the service starts successfully without error. it will also run the fan for a few seconds, so i know its starting fine.
When attempting to use 'fan on' as a bash command, i get "command not found"
- how can i get this as a working command?
- what changes do I need in the script to keep it alive to be able to pass such commands to it later on?
It is atypical for services to respond to interactive commands other than service-management commands drawn from a pre-defined set. If that's the concept you are trying to prove then I would suggest choosing a different direction.
But if you must do it, then you have viable alternatives:
for an on / off toggle, consider setting up a signal handler in the service script, say for SIGUSR1. You can send signals directly via the kill command, but I would recommend instead providing a separate script for the purpose.
for more complex instructions or more than a very small number of distinct instructions, you need some kind of communication channel. The script could potentially use a FIFO or a socket for this purpose, and in that case I would again recommend a separate script for sending commands. But this will require your service to be substantially more complex.
In any case, do note that although there's a long tradition of service-control scripts, service implementations themselves are rarely scripts. They are normally compiled programs, typically written in C, as this provides more direct access to the system and finer control, and is also considered by many to be more secure (when done competently).