We've been using Gitlab on my current job for a while now, and have encountered some instability which express itself in various ways.
The most recent one: projects that should be deleted are flagged as such but actual deletion never occurs.
Some research has allowed me to see the probable cause of the problem, but not how to resolve it: the ProjectDestroyWorker hasn't run for over 10 days.
Could someone point me to some documentation on the mechanism(s) that trigger the workers, and how to monitor them?
Version: GitLab Community Edition 8.5.0 a513e09
You have a few issues with this kind of problem: issue 15334, issue 20984.
Checking the backtrace from sidekiq.log can help
Merge Requests 5695 and 4341 (for GitLab 8.11) should fix some of those issues, like:
There is a race condition in DestroyGroupService now that projects are deleted asynchronously:
User attempts to delete group
DestroyGroupService iterates through all projects and schedules a Sidekiq job to delete each Project
DestroyGroupService destroys the Group, leaving all its projects without a namespace
Projects::DestroyService runs later but the can? (current_user, :remove_project) is false because the user no longer has permission to destroy projects with no namespace.
This leaves the project in pending_delete state with no namespace/group.
I came across this issue during the migration to hashed storage in v13.5.7
Drop to the ruby console & destroy projects that are pending
deletion
In my particular case this allowed gitlab-rake gitlab:storage:legacy_projects to correctly show 0 so further upgrades were possible.
Related
I have set up atlantis and configured multiple projects.
I am not using workspaces (therefore, for each project only the default workspace should be applicable).
However, when creating a GitHub Pull Request that includes changes to multiple projects, I get the following error(s)
dir: terragrunt/path1/to/something workspace: default
The default workspace is currently locked by another command that is running for this pull request.
Wait until the previous command is complete and try again.
dir: terragrunt/path1/to/anotherthing workspace: default
dir: terragrunt/path2/to/anotherthing workspace: default
The default workspace is currently locked by another command that is running for this pull request.
Wait until the previous command is complete and try again.
This is despite the fact that docs state:
Only the directory in the repo and Terraform workspace are locked, not the whole repo.
Any idea why is this happening?
I saw something similar after setting
parallel_plan: true
parallel_apply: true
in my atlantis.yaml.
Removing these fixed the issue for me, and I assume setting them to false would achieve the same thing.
I am not 100% clear WHY this is happening yet, but it appears to have something to do with the way Atlantis locks and terraform workspaces interact, as the default workspace is called default and it appears that Atlantis locks, might be related to the workspace name.
It looks like you have already reported the issue here and that the maintainers responded that there is a merged but currently (2022-04-29) unreleased fix.
Hope this helps others who happen to stumble upon this...
I started running GitLab CE inside of an x86 Debian VM locally about two years ago, and last year I decided to migrate the GitLab CE instance to a dedicated Intel NUC server. Everything appeared to go well with no issues, and my GitLab CE instance is up-to-date as of today (running 13.4.2).
I discovered recently though, that some repos that were moved give a "NO REPOSITORY!" error when visiting their project pages, and if they had any issue boards, merge requests, etc, that these were also gone. But you wouldn't suspect it since the broken repos appear in the repo lists along with working repos that I use all the time.
If I had to reason about these broken repos, it would be that they had their last activity over a year ago, with either no pushes ever made to them other than an initial push, or if changes were made, issues created, or merge requests created, it was literally over a year ago.
Some of these broken repos are rather large with a lot of history, whereas others are super tiny (literally just tracking changes to a shell script), so I don't think repo size itself has anything to do with it.
If I run the GitLab diagnostic check sudo gitlab-rake gitlab:check, everything looks good except for "hashed storage":
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
But then running sudo gitlab-rake gitlab:storage:migrate_to_hashed doesn't appear to complete (with something like six failed jobs in the dashboard), and running the "gitlab:check" again still indicates this "hashed storage" problem. I've also tried running sudo gitlab-rake gitlab:git:fsck and sudo gitlab-rake cache:clear but these commands don't seem to make a difference.
Luckily I have the latest versions of all the missing repos on my machine, and in fact, I still have the original VM running GitLab CE 12.8.5 (with slightly out of date copies of the repos.)
So my questions are:
Is it possible to "repair" the broken repos on my current instance? I suspect I could just "re-push" my local copies of these repos back up to my server, but I really don't want to lose any metadata like issues / merge requests and such.
Is there any way to resolve the "not all projects are in hashed storage" issue? (Again the migrate_to_hashed task fails to complete.)
Would I be able to do something like "backup", "inspect / tweak backup", "restore backup" kind of thing to fix the broken repos, or at least the metadata?
Thanks in advance.
Okay, so I think I figured out what happened.
I found this thread on the GitLab User Forums.
Apparently the scenario here is:
Have a GitLab instance that has repos not in "hashed storage"
Backup your repo
Restore your repo (either to the same server or migrating to another server)
Either automatically or manually, attempt to update your repos to "hashed storage"
You'll now find that any repo with a "ci runner" (continuous integration runner) will now be listed as "NO REPOSITORY!" and be completely unavailable, since the "hashed storage" migration process will fail
The fix is to:
Reset runner registration tokens as listed in this article in the GitLab documentation
Re-run the sudo gitlab-rake gitlab:storage:migrate_to_hashed process
Once the background jobs are completed, run sudo gitlab-rake gitlab:check to ensure the output contains the message:
All projects are in hashed storage? ... yes
If successful, the projects that stated "NO REPOSITORY!" should now be fully restored.
A key to know if you need to run this process is if you:
Log in to your GitLab CE instance as an admin
Go to the Admin Area
Look under Monitoring->Background Jobs->Dead
and see a job with the name
hashed_storage:hashed_storage_project_migrate
with the error
OpenSSL::Cipher::CipherError:
I want to create rollback task for Release Management in visual basic online.
i have some steps but it will take more time.
steps are:
Back Up: perform a backup of the original files to use for rollback later.
Deploy:Copy latest files from artifact to the target folder.
3.Configure — Make configuration changes to the setup.
4.Rollback — Rollback the files from backup in case Deploy failed. Delete backup before exiting.
we can see it will take long time while backup a database . So how we can optimize ?
is there any other method so we can implement and take less time when we will do rollback task ?
There isn’t the built-in feature/task to rollback changes in release, you need to do it by your own script.
There are various options that can be considered as a rollback
strategy.
Option#1 : Undo the change by redeploying the previous release
The first option is to simply redeploy the previously successful
release. This might work well for standalone applications.
Whenever an application depends on some external services or has a
database involved, this approach does not work well. The dependent
services might have upgraded and no longer compatible with the
previous release. The database might have changed the schema making
the previous release no longer healthy.
Option#2 : Fix the issue, do another release
The second option is to simply do nothing. Something went wrong,
troubleshoot it and fix it. Once we fix the issue, we can do another
release.
This however means the environment would remain unhealthy for some
time, as long as it takes for the fix to be ready and deployed.
Option#3 : Understand what failed in the deployment and make a temporary change for the time being
Both the above options are both valid approaches but with some
limitations.
That brings us to a third option. While the fix is getting ready
(option 2), make a minimum change to the environment to get it
temporarily healthy.
More information, please refer to: Implement Rollback with Release Management for TFS 2015 (Apply to VSTS)
You can't really optimise the backup/restore process.
If you use a Virtual Machene to deploy to then you should snapshot it prior to the upgrade, then a roll back is simply restoring the snapshot.
This will be much quicker, seconds, rather than minutes or hours...
Short version: In gitlab how do I view all issues resolved in a commit (or merge request) given that I close issues using git comments (e.g. 'fix #10')?
Long version: In case there is an alternative solution what I'm ultimately trying to accomplish is this. Let's say developer fixes 10 issues, commits, creates merge request and deploys to the staging server. Now QA team needs to know the list of issues that were released so that they can test them. I know this can be done by manually labeling each issue(or assigning a milestone). But since gitlab already knows what issues were resolved in a commit it should be easy to view all issues associated with a commit/merge request. Is it possible to view them?
We currently have an issue with our TFS server locking up whenever any of our developers attempt to check in a pending delete on a folder, however, this is a relatively new (within the last couple of months) issue, and we have been struggling to determine the root cause.
Essentially, the process that is occurring is something like this:
Create folder in TFS.
Branch from Dev trunk, creating the branch in the new folder.
Make any code changes in branch.
Merge changes from branch into Dev trunk.
Delete branch.
Realize we no longer need the folder.
Delete the folder.
Attempt to check-in folder deletion.
TFS server locks up, not allowing any operations to be performed until the server is restarted.
So, I am looking for two answers:
Why is TFS locking up when attempting to check in a folder deletion?
What can we do to resolve the issue?
A little bit more info:
This problem occurs for every user, including out TFS administrator(s).
It may occur even when there was never anything placed in the folder, but we haven't attempted that, so that we can avoid having to restart the server.
This has not always been an issue on the server, as it use to be relatively common for us to delete folders.
This issue occurs regardless of how the delete is being performed (VS2012, VS2015, command line, etc.).
We primarily work with a single trunk, but have potentially hundreds of branches being created over the course of a month.
The solution contains roughly 20-25,000 files, currently.