I started running GitLab CE inside of an x86 Debian VM locally about two years ago, and last year I decided to migrate the GitLab CE instance to a dedicated Intel NUC server. Everything appeared to go well with no issues, and my GitLab CE instance is up-to-date as of today (running 13.4.2).
I discovered recently though, that some repos that were moved give a "NO REPOSITORY!" error when visiting their project pages, and if they had any issue boards, merge requests, etc, that these were also gone. But you wouldn't suspect it since the broken repos appear in the repo lists along with working repos that I use all the time.
If I had to reason about these broken repos, it would be that they had their last activity over a year ago, with either no pushes ever made to them other than an initial push, or if changes were made, issues created, or merge requests created, it was literally over a year ago.
Some of these broken repos are rather large with a lot of history, whereas others are super tiny (literally just tracking changes to a shell script), so I don't think repo size itself has anything to do with it.
If I run the GitLab diagnostic check sudo gitlab-rake gitlab:check, everything looks good except for "hashed storage":
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
But then running sudo gitlab-rake gitlab:storage:migrate_to_hashed doesn't appear to complete (with something like six failed jobs in the dashboard), and running the "gitlab:check" again still indicates this "hashed storage" problem. I've also tried running sudo gitlab-rake gitlab:git:fsck and sudo gitlab-rake cache:clear but these commands don't seem to make a difference.
Luckily I have the latest versions of all the missing repos on my machine, and in fact, I still have the original VM running GitLab CE 12.8.5 (with slightly out of date copies of the repos.)
So my questions are:
Is it possible to "repair" the broken repos on my current instance? I suspect I could just "re-push" my local copies of these repos back up to my server, but I really don't want to lose any metadata like issues / merge requests and such.
Is there any way to resolve the "not all projects are in hashed storage" issue? (Again the migrate_to_hashed task fails to complete.)
Would I be able to do something like "backup", "inspect / tweak backup", "restore backup" kind of thing to fix the broken repos, or at least the metadata?
Thanks in advance.
Okay, so I think I figured out what happened.
I found this thread on the GitLab User Forums.
Apparently the scenario here is:
Have a GitLab instance that has repos not in "hashed storage"
Backup your repo
Restore your repo (either to the same server or migrating to another server)
Either automatically or manually, attempt to update your repos to "hashed storage"
You'll now find that any repo with a "ci runner" (continuous integration runner) will now be listed as "NO REPOSITORY!" and be completely unavailable, since the "hashed storage" migration process will fail
The fix is to:
Reset runner registration tokens as listed in this article in the GitLab documentation
Re-run the sudo gitlab-rake gitlab:storage:migrate_to_hashed process
Once the background jobs are completed, run sudo gitlab-rake gitlab:check to ensure the output contains the message:
All projects are in hashed storage? ... yes
If successful, the projects that stated "NO REPOSITORY!" should now be fully restored.
A key to know if you need to run this process is if you:
Log in to your GitLab CE instance as an admin
Go to the Admin Area
Look under Monitoring->Background Jobs->Dead
and see a job with the name
hashed_storage:hashed_storage_project_migrate
with the error
OpenSSL::Cipher::CipherError:
Related
I have a Cloudflare page that uses python-sphinxto build docs. For some of my commits, it downloads a different version of sphinx than others and fails to build docs correctly.
What I tried:
Adding a dummy commit on top of a failing build seems to fix an issue and force Cloudflare builder to download the correct sphinx version
re-running deployments doesn't fix the issue
creating a new branch with the same head(failing commit) and running another deployment doesn't fix the issue
changing between preview/production deployments has no impact on this issue
Here is a dummy commit I added to make the docs build correctly
Commit one result vs. Commit two result
The diff in deployment logs old commits, the left is Commit 1(not working), and the right, Commit two, correctly builds all three tasks and the releases.
https://www.diffchecker.com/ZpV8vE9D
I have tried making different branches and re-run deployments to check whether the sphinx version will change, but it seems like it's bounded the the "old commit". This is also an issue for other Cloudflare Pages, and using preview/production deployments has no impact on this problem.
The issue in this case was actually not with sphinx version but with the fact that I was using:
git fetch --all
Which does not guarantee pulling the tags with it.
The --all pull from all remotes instead of "everything" as I thought.
Using git fetch --tags instead fixed the issue
Indeed it seems to be a bug, in this case I recommend you to consult directly with CloudFlare support, sometimes they are errors that remain internally in your account and you unbug it, CloudFlare Pages for the moment continues to improve its system, there are details to be corrected.
Yesterday, I have moved my GitLab installation to another machine.
It was installed with docker-compose, and I followed the official GitLab guide to back up and restore GitLab including the 'secrets' files.
Everything works so far, except the CI/CD variables in the admin area.
I get the error 'There was an error fetching the variables.' when I navigate to this site.
Can you give me a hint in which log I can found more information about this error?
Finally I could solve the problem.
With the Doctor Rake tasks I could determine where the problem was.
Afterwards I followed the steps to reset the runner registration tokens.
Finally I deleted al the instance variables in the dbconsole, by deleting them out of the database.
Check first if this is similar to gitlab-org/gitlab issue 218913 which includes two possible root causes:
Either you have an adblocker on, which could affect that functionality
Or:
go to the project settings general -> Visibility, project features, permissions
In Pipelines (Build, test, and deploy your changes) select Only Project Members
I had the same issus after restore a backup :
My solution was to delete variables from the database :
sudo gitlab-rails dbconsole --database main
gitlabhq_production=>delete from ci_instance_variables;
gitlabhq_production=>delete from ci_variables;
then it works
TLDR; My version of the SVN-repo differs from my teamates. Even though the repo-UUID is identical and we are using the same branch (exactly the same repo-links).
My SVN was working fine before i went home for vacation.
I came back, updated my repo, commit some changes. Everything seemed to work fine. But it turns out that my team cannot see my commits and when updating - my commit seems to be the latest one. Looking through their clients, the revision-numbers i have checked in collide with other commits.
Im using Ubuntu 18.04.1 LTS -
svn, version 1.9.7 (r1800392)
I tried removing the repository and checking it out (removing the .svn dir as well) - and when i do, my latest commit is HEAD. (Instead of the real HEAD which has a revision number way higher).
When i browse the repo from the web-browser, my commit is HEAD.
When they browse the very same link - their commit is HEAD.
Restarting computer does nothing. Reinstalling subversion (removing ~/.subversion) did nothing.
We tried checking out the repo using different user - still my commit is HEAD.
If i use my account on different machine, it works fine.
The repo-UUID is the same for me and my colleagues.
Im thinking this might be some kind of cache issue, but what cache is both used for svn and browser?
Additional thoughts:
I am also using a VPN to access the corporate network. Could there be a cache there? But the SVN-traffic is using TLS, what could possibly cache TLS-data?
#Simion pointed out that it might be a good idea to make sure the hostname of the Repo resolves in the same IP.
Turns out that was the problem. The CM-Center had moved the repo to another server (and for unknown reasons kept a copy of the repo on the old server - which caused all this confusion) and the Infra department had changed the IP of the repo hostname.
Flusing my local DNS-cache fixed the problem!
Seemingly both GitLab and gitlab-mirrors are setup and working correctly for me. However not long after I create a new branch under username Admin using the default root account for example (not to be confused with the gitmirror user created not long after), I'll read in the commit history that the admin user deleted the branch entirely. Typically an hour later.
Prior to the new branch being deleted, everything seems 100%, and it is really cool to see my branch alongside the history of previous commits to the repo I mirrored, using git-mirror.
To be more clear, I am using gitmirror to pull a GitHub-hosted repo to GitLab. This repo already has two branches, one is called master (which is HEAD) and I can see those fine. The next thing I do is create a new branch for my own development as the Admin user, and while it seems to work just fine. Within an hour the log will read that the Admin user deleted the newly created branch (which happens every time automatically, and I certainly didn't do this manually myself).
I've checked cron, and it seems ok. During setup, as the gitmirror user I used 'crontab e' to install the following text (tweaked for my own domain of course):
Host gitlab.example.com
User git
Since the machine is a virtual machine, it is easy to save snapshots which I've done. So it is possible to recover a snapshot with all the changes described, and I can watch the Admin user automatically delete my new branch not long after restoration (because the snapshot was saved hours ago already); which is handy for debugging purposes.
Should I not be creating repos using the default GitLab root account? Is there a log somewhere that shows more than the recent GIT history on the GitLab projects page? What else can I try?
I've created two different servers from scratch using the latest software versions now, and I get identical results on each.
I have an Azure Website configured to deploy from a Bitbucket repository. This works fine.
Since the application is still in active development, I update the nuget packages it uses quite frequently. This causes the packages folder to keep growing indefinitely, unless I go and manually delete the packages.
Now, in my local machine this is not a big issue. Space is cheap. But in Azure, this makes us go over the quota really fast, as old packages accumulate.
How can I customize the Azure deploy process so that it deletes all the packages after a successful deployment?
(I am open to other solutions as well)
You can utilize the custom deployment script feature where you add a step that cleans up the packages directory.
You can read about it here:
http://blog.amitapple.com/post/38418009331/azurewebsitecustomdeploymentpart2/
Another option is to add a post deployment action, by adding a script file (.cmd/.bat) that has the clenup logic to the following directory in your site: d:\home\site\deployments\tools\PostDeploymentActions\, this script will run after the deployment completes successfully.
Read more about it here:
https://github.com/projectkudu/kudu/wiki/Post-Deployment-Action-Hooks