Yesterday, I have moved my GitLab installation to another machine.
It was installed with docker-compose, and I followed the official GitLab guide to back up and restore GitLab including the 'secrets' files.
Everything works so far, except the CI/CD variables in the admin area.
I get the error 'There was an error fetching the variables.' when I navigate to this site.
Can you give me a hint in which log I can found more information about this error?
Finally I could solve the problem.
With the Doctor Rake tasks I could determine where the problem was.
Afterwards I followed the steps to reset the runner registration tokens.
Finally I deleted al the instance variables in the dbconsole, by deleting them out of the database.
Check first if this is similar to gitlab-org/gitlab issue 218913 which includes two possible root causes:
Either you have an adblocker on, which could affect that functionality
Or:
go to the project settings general -> Visibility, project features, permissions
In Pipelines (Build, test, and deploy your changes) select Only Project Members
I had the same issus after restore a backup :
My solution was to delete variables from the database :
sudo gitlab-rails dbconsole --database main
gitlabhq_production=>delete from ci_instance_variables;
gitlabhq_production=>delete from ci_variables;
then it works
Related
I started running GitLab CE inside of an x86 Debian VM locally about two years ago, and last year I decided to migrate the GitLab CE instance to a dedicated Intel NUC server. Everything appeared to go well with no issues, and my GitLab CE instance is up-to-date as of today (running 13.4.2).
I discovered recently though, that some repos that were moved give a "NO REPOSITORY!" error when visiting their project pages, and if they had any issue boards, merge requests, etc, that these were also gone. But you wouldn't suspect it since the broken repos appear in the repo lists along with working repos that I use all the time.
If I had to reason about these broken repos, it would be that they had their last activity over a year ago, with either no pushes ever made to them other than an initial push, or if changes were made, issues created, or merge requests created, it was literally over a year ago.
Some of these broken repos are rather large with a lot of history, whereas others are super tiny (literally just tracking changes to a shell script), so I don't think repo size itself has anything to do with it.
If I run the GitLab diagnostic check sudo gitlab-rake gitlab:check, everything looks good except for "hashed storage":
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
But then running sudo gitlab-rake gitlab:storage:migrate_to_hashed doesn't appear to complete (with something like six failed jobs in the dashboard), and running the "gitlab:check" again still indicates this "hashed storage" problem. I've also tried running sudo gitlab-rake gitlab:git:fsck and sudo gitlab-rake cache:clear but these commands don't seem to make a difference.
Luckily I have the latest versions of all the missing repos on my machine, and in fact, I still have the original VM running GitLab CE 12.8.5 (with slightly out of date copies of the repos.)
So my questions are:
Is it possible to "repair" the broken repos on my current instance? I suspect I could just "re-push" my local copies of these repos back up to my server, but I really don't want to lose any metadata like issues / merge requests and such.
Is there any way to resolve the "not all projects are in hashed storage" issue? (Again the migrate_to_hashed task fails to complete.)
Would I be able to do something like "backup", "inspect / tweak backup", "restore backup" kind of thing to fix the broken repos, or at least the metadata?
Thanks in advance.
Okay, so I think I figured out what happened.
I found this thread on the GitLab User Forums.
Apparently the scenario here is:
Have a GitLab instance that has repos not in "hashed storage"
Backup your repo
Restore your repo (either to the same server or migrating to another server)
Either automatically or manually, attempt to update your repos to "hashed storage"
You'll now find that any repo with a "ci runner" (continuous integration runner) will now be listed as "NO REPOSITORY!" and be completely unavailable, since the "hashed storage" migration process will fail
The fix is to:
Reset runner registration tokens as listed in this article in the GitLab documentation
Re-run the sudo gitlab-rake gitlab:storage:migrate_to_hashed process
Once the background jobs are completed, run sudo gitlab-rake gitlab:check to ensure the output contains the message:
All projects are in hashed storage? ... yes
If successful, the projects that stated "NO REPOSITORY!" should now be fully restored.
A key to know if you need to run this process is if you:
Log in to your GitLab CE instance as an admin
Go to the Admin Area
Look under Monitoring->Background Jobs->Dead
and see a job with the name
hashed_storage:hashed_storage_project_migrate
with the error
OpenSSL::Cipher::CipherError:
I set up a workspace and I am following the Enforce Policy with Sentinel hands on guide.
I see the following message in the run tab:
As soon as I try to press the queue plan button I receive this error:
My configured variables are:
Is there something else I need to configure to be able to queue a plan?
Executing from the cli I was able to trigger a run (in TF Cloud) that only included the plan step. The run execution can be viewed if I access the specific run url directly.
Any help, suggestions are more than welcome!
I guess you've already resolved this issue, but I post my resolution anyway.
When this issue occurred, I had wrong workspace's name settings.
There were two workspaces with similar names, and I don't know why this happened. Then I deleted one of them.
On the other one occurred this issue.
In the end, when I deleted the workspace and recreated it, I didn't have the issue occurring anymore.
In my case I didn't have duplicate / similar names workspaces (only one workspace). I found that after running terraform apply locally once first, the UI controls started to work as expected.
I'm running into a strange problem whenever I start a particular build, and I can't get my head around it.
I just imported an existing VSTS-repository into my new GIT-Repository on Azure DevOps. My next step is to create a Build-pipeline which should lead to an artifact which I can deploy. For the company I work for I've done this many times, but I've never seen this error before.
The buildpipeline is setup, and as soon as I start a build it immediately fails with the following error;
Hopefully somebody can help out in resolving this.
UPDATE - Added settings for retrieving sources
After posting the second screenshot and going through everything again properly, I saw that I didn't point the Build Pipeline to the proper GIT-Repository in Azure Devops. After updating this, the issue was resolved.
I have an Azure Web application that I checked into TFS yesterday with no issues. Upon checking in, the resource manager will inject our nuget packages and deploy if it builds successfully.
I made a few changes (added a class) and checked in today. I received this error on the build:
Here's the quote to help the future search bots:
Exit code 1 returned from process: file name 'tf', arguments 'vc unshelve Gated_xxxxxx;****** /loginType:OAuth /login:.,******** /noprompt'.
I looked into the log response, to see if I get more detail, but it says the exact same thing. I have not changed my password or username.
How can I debug this to figure out the issue?
UPDATE
To save others from the headache. The issue was that we had CI builds per project. A file from another project had snuck in as well. So I was checking in for 2 different projects on 2 different solutions (Which both go to the same TFS server). So make sure you only check in for that one project!
To save others from the headache. The issue was that we had CI builds per project. A file from another project had snuck in as well. So I was checking in for 2 different projects on 2 different solutions (Which both go to the same TFS server). So make sure you only check in for that one project!
You can receive that error as well if you try to check in a file that is in a project that is not mapped in your build definition.
Let's say that you have a file named FileA.cs that is in a project named ProjectA.csproj. If you do changes in FileA.cs and this file is included in your changes, you need to map ProjectA.csproj in the Get source step of the build definition.
I have a custom deployment script (*.sh script) defined for my azure deployment.
Just today, I have found that I am unable to publish. I updated my bitbucket repository and after a while I get an error similar to the following:
Command 'starter.cmd deploy_pvl_cont ...' was aborted due to no output nor CPU activity for 180 seconds. You can increase the SCM_COMMAND_IDLE_TIMEOUT app setting (or WEBJOBS_IDLE_TIMEOUT if this is a WebJob) if needed.\r\nstarter.cmd deploy_pvl_content.sh
I have tried a number of things to try to diagnose the problem.
Increase SCM_COMMAND_IDLE_TIMEOUT to 300
Run the script locally (Works)
Set up a new fresh deployment slot and try publishing same commit (Same error)
Tried publishing the previously successful commit (Same error)
Look for useful error messages in a diagnostic log dump (Coldn't find anything more useful)
Tried running the deployment script from the Kudu Console (No output returned, like it didn't actually run)
Tried reverting git to a previous version as suggested by #david-ebbo
Tried simplifying my script to a single echo command with the same results
Not sure what I can do to debug this further. Ideally I would like to get the output of the shell script on the azure host but don't know how to get it. Any ideas?
Updated answer
This is a regression caused by the move to git 2.8.x in Azure. The issue is tracked by https://github.com/projectkudu/kudu/issues/2041.
Here is a very simple workaround (and you don't need to bring in the old git tools): instead of setting your COMMAND to deploy_pvl_content.sh, set it to bash deploy_pvl_content.sh
We'll address the issue, but this workaround will get you going.
Original answer (only leaving for context)
You could be running into some flavor of this issue, which is caused by the upgrade to git 2.8.1 that we just did.
While we're trying to get to the bottom of it, please try this workaround to see if that helps:
Go to Kudu Console
Create a d:\home\bin folder
Copy the old Windows git 1.8.x folder in there. You can get the content from here. If you drag and drop the zip into Kudu console, there is a special unzip drop area that will expand it.
Try your deployment again