I'm trying to setup a docker container that starts cloud spanner and that initializes it.
Using this official docker image provided by google: gcr.io/cloud-spanner-emulator/emulator
I'm looking to automatically initialize spanner on start.
I tried various things with the docker file, to summarize:
FROM gcr.io/cloud-spanner-emulator/emulator
RUN some gcloud command after to initialize local spanner db
But absence of informations on how this image works makes it hard to find if it's even possible to initialize it everytime the container starts.
I repeat myself, I need this to be run automatically when the container mounts. It's going to build pipelines.
Is there a way to do that with this provided docker image? Or should I create my own dockerfile that installs the emulator via gcloud cli?
Here is a docker file example that allows to start docker emulator and to initialize with some custom gcloud commands
FROM google/cloud-sdk:slim
RUN apt-get install -y google-cloud-sdk google-cloud-sdk-spanner-emulator
COPY ./start_spanner.bash start_spanner.bash
COPY ./schema.ddl schema.ddl
CMD ./start_spanner.bash
Here is a sample content of file start_spanner.bash.
See https://docs.docker.com/config/containers/multi-service_container/ for details about set -m.
See https://cloud.google.com/spanner/docs/emulator for info about how to use gcloud and emulator together
#!/bin/bash
set -m
gcloud beta emulators spanner start --host-port=0.0.0.0:9010 &
# configure gcloud cli to connect to emulator
gcloud config set auth/disable_credentials true
gcloud config set project someproject
gcloud config set api_endpoint_overrides/spanner http://localhost:9020/
# create spanner instance
gcloud spanner instances create someinstance \
--config=emulator-config \
--description="Test Instance" \
--nodes=1
# create spanner database with the given schema
gcloud spanner databases create somedb \
--instance=someinstance \
--ddl-file=schema.ddl
fg %1
And the file schema.ddl is just your spanner ddl to run to create tables in the emulator as per spanner's documentation.
If I understand correctly, you wish to start up the Spanner emulator within a dockerfile.
You can create your own dockerfile to build the emulator.
Please utilize these docker commands:
docker pull gcr.io/cloud-spanner-emulator/emulator
docker run -p 9010:9010 -p 9020:9020 gcr.io/cloud-spanner-emulator/emulator
To startup the spanner emulator.
EDIT:
With further clarification, it seems that what is being attempted is currently not an available feature of the official Docker image "gcr.io/cloud-spanner-emulator/emulator".
No documentation identifies how to utilize the official Spanner Emulator Docker image to run initialization scripts, neither the GCP Documentation nor the Spanner emulator README. Such, as running gcloud commands on startup to create a spanner instance, upload a DDL file and add a database using said file.
The workaround identified by OP in the comments, seems to be the best solution as of now.
Related
I am new to running containers in Azure and I am puzzled with the use case below,
If you feel I am wrongly using containers, feel free to give me an alternative approach.
Core problem: I am not able/ don't know how to create a container instance in the "stopped" state either via command line or ARM template deployment
Long read use case:
I created a docker image that runs a python job.
The job needs to run daily and is triggered via a data factory. The data factory will figure out the environment, set up the docker commands, update the container image and then execute the container via a batch account. The job itself does an api call and writes some data to sql. This part works fine, the container status goes to running and stops afterwards (I put auto-restart off)
In Azure DevOps, I have a pipeline that builds the image of the job and stores it in an azure repo. this works fine.
As I need a container instance as a resource in my resource group, I decided to put them in my infra ARM template. The problem: When deploying the container using DevOps / Arm template deployment,
It deploys and runs the job instance and this is not great, I would like to have the container created in a "stopped" state. The reason for this is that the job otherwise writes data to our database, and that is unwanted.
I am wondering what would be the best approach / what the best guidelines are, I thought about these two scenarios, but for both, my intuition says no.
scenario 1: have an all-time running container (let it execute bin/bash) and deliver the command using "az container exec".
Why I don't want to do this: I currently have an image per environment that has the exact job for that environment and I don't see the usefulness of having 3 standby containers running all the time to be triggered once a day
scenario 2: Instead of handling the container instance creation via DevOps, ignore the problem and create it using data-factory and the batch account. This implies that, when the job is triggered, It will create (and therefore run the container). Subsequently, I could delete it after usage.
Why I don't want to do this: I see a container instance as part of the infrastructure (as it is something you deploy inside a resource group, correct me if my point of view is wrong) So in that sense, managing resources via a scheduled data factory job doesn't look good and is a kind of hack to overcome the problem that you cannot deploy a container instance in a stopped state.
# set base image (host OS)
FROM python:3.7-buster
# Argument for environment selection
ARG environment
ENV environment ${environment}
# set the working directory in the container
WORKDIR /
# copy the dependencies file to the working directory
COPY requirements.txt .
# install FreeTDS and dependencies
RUN apt-get update && apt-get -y install apt-transport-https curl
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN exit
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get -y install msodbcsql17
RUN apt-get install unixodbc-dev
RUN pip install pyodbc
RUN pip install -r requirements.txt
# copy the content of the local src directory to the working directory
COPY /src .
# command to run on container start
CMD python "./my_data_job.py" "./my_config.ini" ${environment}
For Azure Container Instance, the container group will always be in the running state until you stop it. But containers inside it can be in the terminated state. In fact, if your image is a one-time image, then the container will be in the terminated state when the job is finished. You can use the CLI command az container exec to run the job again as you need.
So it's impossible to create an ACI in the stopped state. Maybe you can use the AKS, create different deployments for different environments, and when you need a container to run the job, then scale up to one replica. When you don't need the container, you can scale down to zero.
I'm using docker/elk image to display my data in kibana dashboard (Version 6.6.0) and It works pretty good. I started the service like using below command.
Docker Image git repo:
https://github.com/caas/docker-elk
Command:
sudo docker-compose up --detach
Expecting that it will run background, and did as expected. After two days the server up and running the and third day the kibana alone getting stopped. and Used below command to make it up and running.
sudo docker run -d <Docer_image_name>
It's up and running when I use docker ps command. But when I tried to hit the kibana server in chrome browser it says not reachable.
So I just used to below command to restart the service.
sudo docker-compose down
After that I can see kibana server in chrome browser which is up and running but I do see all my data is lost.
I used below URL in jenkins to collect the data.
`http://hostname:9200/ecdpipe_builds/extern`al
Any idea how can I resolve this issue?
I did not see the persistent storage configuration the image you mentioned in their GitHub docker-compose file.
This is common to lost data in case of docker container if you did not provide persistent storage configuration. so docker-compose down may cause to lost you data if there is no persistent configuration docker-compose file.
Persisting log data
In order to keep log data across container restarts, this image mounts
/var/lib/elasticsearch — which is the directory that Elasticsearch
stores its data in — as a volume.
You may however want to use a dedicated data volume to persist this
log data, for instance to facilitate back-up and restore operations.
One way to do this is to mount a Docker named volume using docker's -v
option, as in:
$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 \
-v elk-data:/var/lib/elasticsearch --name elk sebp/elk
This command mounts the named volume elk-data to
/var/lib/elasticsearch (and automatically creates the volume if it
doesn't exist; you could also pre-create it manually using docker
volume create elk-data).
So you can set these paths in your docker-compose file accordingly. Here is the link that you can check elk-docker-persisting-log-data
Use docker volume or file location as persistant space
Steps followed during rolling updates:
Create an image for the v2 version of the application with some changes
Re-Build a Docker Image with Maven. pom.xml. Run command in SSH or Cloud Shell:
docker build -t gcr.io/satworks-1/springio/gs-spring-boot-docker:v2 .
Push the new updated docker image to the Google Container Registry. Run command in SSH or Cloud Shell
gcloud docker -- push gcr.io/satworks-1/springio/gs-spring-boot-docker:v2
Apply a rolling update to the existing deployment with an image update. Run command in SSH or Cloud Shell
kubectl set image deployment/spring-boot-kube-deployment-port80 spring-boot-kube-deployment-port80=gcr.io/satworks-1/springio/gs-spring-boot-docker:v2
Revalidate the application again through curl or browser
curl 35.227.108.89
and observe the changes take effect.
When do we come across the "CrashLoopBackOff" error and how can we resolve this issue? Does it happen at application level or at kubernetes pods level?
I've been looking at various methods to run commands upon creation of EC2 instances using Elastic Beanstalk on AWS. I've been given different methods to do this through AWS Tech Support, including life cycle hooks, custom AMI's, and .ebextensions. I've been having issues getting the first 2 methods (life cycle hooks and custom AMIs) to work with EB.
I'm currently using .ebextensions to run commands upon deploy, but not sure if there's a way to run a set of commands upon creation only instead of every time I deploy code. For instance, I have a file .ebextensions/03-commands.config that contains the following code:
container_commands:
01_npm_install:
command: "npm install -g -f npm#latest"
However, I only want this code to run upon instance creation, not every time I deploy, as it currently does. Does anyone know a way to accomplish this?
Thanks in advance!
I would recommend creating an idempotent script in your application that leaves a marker file on the instance in some location say /var/myapp/marker using something like mkdir -p /var/myapp-state/; touch /var/myapp-state/marker on successful execution. That way in your script you can check that if the marker file exists you can make your script a no-op.
Then you can call your script from container commands but it will be a no-op everytime because on first successful execution it will create the marker file and subsequent executions will be no-ops.
Create a custom AMI. This way you can setup your instances whoever you want and they will launch faster
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.customenv.html
As I see from you question, you're using: container_commands, that is means you are using Elastic Beanstalk with Docker. Right? In this case I would recommended to read: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html.
The idea is following, that you can create own Dockerfile, where you can specify all commands that you need to build a docker container, for example to install all dependencies.
I would recommended to use .ebextensions for the Elastic Beanstalk customization and configuration, for example to specify ELB or RDS configuration. In the Dockerfile make sense to specify all commands, that you need to build a container for your application, that includes setup of the web server, dependencies etc.
With this approach, Elastic Beanstalk will build a container, that each time when you do deploy, it execute a docker instance with deployed source code.
There is a simple option leader_only: true you need to use as per current AWS Elasticbeanstalk configurations, You simply need to add this under
container_commands:
01_npm_install:
command: "npm install -g -f npm#latest"
leader_only: true
This is the link as per AWS
AWS Elasticbeanstalk container command option
I have a newbie regarding docker. I would like to know if it is possible to export a docker image created for AWS to Bluemix or Azure. My docker image contains a websocket server under NodeJS and a MongoDB database.
Thank you for your help
Access your aws cloud and use:
docker save -o image.tar image:1.0 #exporte docker image
After concluded that, access your new cloud and use:
docker load -i image.tar #load your image to the new cloud
Having the dockerfile you used to create your AWS container, you can simply use it to build the container on Bluemix using cf ic client or the docker native one
Following the reference doc for Bluemix docker cli
https://www.ng.bluemix.net/docs/containers/container_cli_reference_ov.html
https://www.ng.bluemix.net/docs/containers/container_cli_ov.html