GKE private image registry is missing images. No changes to the environment have been done, this process was working fine until about 2 weeks ago. Here's the process
(This environment was handed to me and it is my first time into the CI/CD process and I am a newbie on the GKE environment as well.)
I have a GitLab pipeline that builds and deploys my app to a GKE dev environment when triggered. There are no errors reported in this process and it completes using gitlab.com in 4-5 minutes. )
The issue that manifested is that many of the images in a google private registry are no longer there, the current version is gone. The pod is trying to pull that image and it is failing with the ImagePullBackoff error, which makes sense due to the missing images. (That is most of them have disappeared, over 40 past versions are not longer in the registry, some older images are still there. )
First, I cannot tell how the images, from the CI/CD process, get placed into the private registry. There is only a reference to pull the registry.gitlab.com and no corresponding push to eu.gcr.io references at all (in the ci/cd files) which is the location of the gke image registry.
There are 3 files related to the ci/cd process:
gitlab-ci.yaml
kube-init.sh
migration.sh
All the secrets are in place and none have been changes. It seems there is a piece missing which moves/saves the files to the private google image registry...where would that be defined?
I can post the files in this process but since there are no errors there, I am not sure that would help. (Let me know if they are needed.)
Thanks in advance...I can't wait to get a DevOps engineer:)
-glen
As a summary of the conclusion reached in the comments:
The images are hosted on gitlab and aren't pushed to the GKE registry. as can be seen here.
The issue OP had was related to the token created for the pipeline from Google Cloud Platform to Gitlab, as it was linked to the previous account which is no longer associated. A new token was issued and the images can be pulled from Gitlab.
Related
Tried to update our infra to use our pipeline images from self-hosted GitLab using Container Registry (prev. using DO's Container Registry, all working).
The pipeline image is in a separate repo on the same GitLab instance and pushed to the Container Registry.
The application's pipeline sometimes fails with a permission error that it cannot pull the image. Which is weird as if I restart the pipeline it works and can pull the image. I'm not using any env like DOCKER_AUTH_CONFIG as from the docs it should be able to access images from the private project's container registry. Also, it always works on master branch.
If the image is pulled, it usually works around 1-2 hours then it starts to fail.
We are using auto-scaled workers if that's important.
Issue was a syntax error in config.toml. Gitlab runner did not throw any error just randomly failing to load it.
I am setting up two Django Web App on the Azure using their App Services - one for staging and one for production.
I have dockerized my Django App and stored the image in the Azure Container Registries.
Now, I have noticed that when I create the webapp, it asks for the specific tag from the registry repo that I want to use, which I can't seem to change after the webapp is created.
My plan is to tag the images with their versions (e.g. :090920201) and promote this version (tag) first on staging, test it there and if it works as expected, use the same version (tag) image and promote it on prod.
For now, I am tagging the images as :staging for staging environment and :prod for the production environment. When I am happy with a specific version of my local, I push the image with their respective tags.
Now the problem is since I cannot change the tag of the registry repo after the app is created, I have to push the same image twice, one with :staging tag and if this image is working as expected, push the same image again with the prod tag.
This could work for the time being, until the dev and staging are in sync.
So what's the problem?
Is there a way to change the tag of the image after the web app was created in Azure?
Is there a way to use a consistent tag (let's say :latest) but only deploy to staging first, test there and then promote it to production?
If we completely ignore what I am doing, how else do you suggest I utilize the same image and manage the promotion of the image first on the staging, and then to the production environment?
Apologies for the world tour for a straight forward question.
Not sure if you looked at registry best practices, but you can't retag an image after it's been deployed.
You can consider tagging your images by including the slots in the namespace name e.g. <registry-name>.azurecr.io/<app-svc-slot>/<image-name>:<version>. If you're going to use the latest moniker, I would tag latest docker build twice, the actual release version and latest. You can then push both images to your registry, or just the one tagged latest, like what's explained in https://learn.microsoft.com/en-us/azure/container-registry/container-registry-image-tag-version.
I’m trying to set up GitLab CI/CD for an old client-side project that makes use of Grunt (https://github.com/yeoman/generator-angular).
Up to now the deployment worked like this:
run ’$ grunt build’ locally which built the project and created files in a ‘dist’ folder in the root of the project
commit changes
changes pulled onto production server
After creating the .gitlab-ci.yml and making a commit, the GitLab CI/CD job passes but the files in the ‘dist’ folder in the repository are not updated. If I define an artifact, I will get the changed files in the download. However I would prefer the files in ‘dist’ folder in the to be updated so we can carry on with the same workflow which suits us. Is this achievable?
I don't think commiting into your repo inside a pipeline is a good idea. Version control wouldn't be as clear, some people have automatic pipeline trigger when their repo is pushed, that'd trigger a loop of pipelines.
Instead, you might reorganize your environment to use Docker, there are numerous reasons for using Docker in a professional and development environments. To name just a few: that'd enable you to save the freshly built project into a registry and reuse it whenever needed right with the version you require and with the desired /dist inside. So that you can easily run it in multiple places, scale it, manage it etc.
If you changed to Docker you wouldn't actually have to do a thing in order to have the dist persistent, just push the image to the registry after the build is done.
But to actually answer your question:
There is a feature request hanging for a very long time for the same problem you asked about: here. Currently there is no safe and professional way to do it as GitLab members state. Although you can push back changes as one of the GitLab members suggested (Kamil Trzciński):
git push http://gitlab.com/group/project.git HEAD:my-branch
Just put it in your script section inside gitlab-ci file.
There are more hack'y methods presented there, but be sure to acknowledge risks that come with them (pipelines are more error prone and if configured in a wrong way, they might for example publish some confidential information and trigger an infinite pipelines loop to name a few).
I hope you found this useful.
Not easy to make good title for this question so if someone have better idea please edit.
That's what I have:
VPS (KVM)
Docker
Nginx-proxy so all docker containers supposed to be exposed are automatically exposed to appropriate domain.
Some apps like Wordpress are just using container with connected volumes which are accesible by FTP so this is not an issue to manage them/update stuff etc.
I have SailsJS app (NodeJS) which I have to dockerize. It will be kept updated quite often.
I will have some apps written in C#(ASP.NET) / Java (Spring) with similar scenario as in point 5.
Both 5 and 6 source code is stored on BitBucket but can be changed if it would be better to have self hosted git server to solve issues.
What I am looking for is to have automated process which will build the docker image when I do commit and make sure that docker will pull the new image and restart container with new content. I do not want to use DockerHub as there is only 1 private repository so it will not work for long term.
I thought I can do it with Jenkins somehow but have no idea how...
You can setup private GitLab server.
It provides THREE necessary things - Git repository (managed as admin by your own), completely private Docker registry (so you can privately store your own docker images) , and own CI - complete and sufficient to do what you request, integrated seamlessly and working with former two.
You would setup GitLab runner so when you do commit image being rebuilt and pushed to component-specific registry, and there are hooks and environments which allow you to set up back connection.
I've been successfully using Git deploy (via Kudu) to a couple of Azure websites (e.g., beta/prod) for several months, and it's worked quite well. Starting today, I noticed that when I push to the appropriate respective git branch, my Azure websites will supposedly deploy - i.e., the deploy kicks off, everything builds, all my tests run, and the Azure management portal swears up and down that it's deployed my website - but ... nothing happens. My websites don't change. (Beta and prod pull from different branches of the same git repo, but no matter which I push to, none of the changes included in the latest push show up on either website.)
There are no errors or any other indication of a problem in the logs. The Azure portal detects the git pushes, runs the deployments, and swears that they've happened successfully. But the changes - some very simple ones, i.e., text on a certain page - simply aren't there.
This is the sort of thing that I'd normally contact Azure support for, but my subscription doesn't include tech support :-(. The Azure site recommends asking here on SO, and hence my post.
Any suggestions for further troubleshooting this?
Well, I don't know what was triggering the problem, but resetting the website - by adding a bogus key/value pair to the configuration, and saving it - triggers the website(s) to pick up the changes. Apparently the underlying issue is that the Kudu deploy doesn't seem to be triggering the website to reset itself. I'll add more details in the future if I run into the problem again.
[Edit 2013-10-15 - Today, deploys seem to be working normally again. My guess is that it was some sort of transient Azure bug that's now fixed.]