Gitlab-ce upgrade issue while upgrading from 14.0.12 to 14.3.6 below error is coming and container is not coming up - gitlab-ce

This is Nithish Kumar G working as a Devops Engineer in Metricstream company,we are using a gitlab community edition(13.12.15 version) , We are facing issue while upgrading gitlab-ce edition , we are following the upgrade path mentioned in the document as 13.12.15 -> 14.0.12 -> 14.3.6 , we successfully upgraded to 14.0.12 and container was up and running and background migrations completed successfully after sometime and then we tried to upgrade to 14.3.6 the container was not coming up and continously restarting in a loop, when we go through the logs found some below error
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20221207-31-1820bvw" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210616134905 AddTimestampToSchemaMigration: migrating ====================
-- add_column(:schema_migrations, :finished_at, :timestamptz)
STDERR:
---- End output of "bash" "/tmp/chef-script20221207-31-1820bvw" ----
Ran "bash" "/tmp/chef-script20221207-31-1820bvw" returned 1
I tried upgrading gitlab-ce from 13.12.15 to 14.0.12 and it is successfull and container is coming up but when i tried to upgrade from 14.0.12 to 14.3.6 after completing background migrations in 14.0.12 it is not coming up and container restarting in loop.

Related

GitLab reports an error : PG::DuplicateSchema: ERROR: schema "gitlab_partitions_dynamic" already exists

Installed by GitLab Docker Compose and upgraded according to the official documentation
docker compose pull
docker compose up -d
The gitlab container cannot be started, the detailed error log is as follows:
...
StandardError: An error has occurred, this and all later migrations canceled:
PG::DuplicateSchema: ERROR: schema "gitlab_partitions_dynamic" already exists
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181228175414_init_schema.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:265:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:264:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::DuplicateSchema: ERROR: schema "gitlab_partitions_dynamic" already exists
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181228175414_init_schema.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:265:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:264:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::DuplicateSchema: ERROR: schema "gitlab_partitions_dynamic" already exists
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20181228175414_init_schema.rb:7:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:125:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:77:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:265:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:264:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:63:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20221119-30-1pungg2" ----
Ran "bash" "/tmp/chef-script20221119-30-1pungg2" returned 1
I tried to stop the gitlab container and restart it, still not working, how can I fix this, thanks!

gitlab.com/gitlab-org/labkit#v1.14.0: Get "https://proxy.golang.org/gitlab.com/gitlab-org/labkit/#v/v1.14.0.zip": dial tcp: i/o timeout

Due to some reason, I have to install GitLab from source code.
When I install 'gitlab-shell' follow the guide:
# Run the installation task for gitlab-shell:
sudo -u git -H bundle exec rake gitlab:shell:install RAILS_ENV=production
I got results:
client/gitlabnet.go:14:2: github.com/golang-jwt/jwt/v4#v4.4.1: Get "https://proxy.golang.org/github.com/golang-jwt/jwt/v4/#v/v4.4.1.zip": dial tcp 142.251.43.17:443: i/o timeout
internal/gitaly/gitaly.go:8:2: github.com/grpc-ecosystem/go-grpc-middleware#v1.3.0: Get "https://proxy.golang.org/github.com/grpc-ecosystem/go-grpc-middleware/#v/v1.3.0.zip": dial tcp 142.251.43.17:443: i/o timeout
internal/gitaly/gitaly.go:9:2: github.com/grpc-ecosystem/go-grpc-prometheus#v1.2.0: Get "https://proxy.golang.org/github.com/grpc-ecosystem/go-grpc-prometheus/#v/v1.2.0.zip": dial tcp 142.251.43.17:443: i/o timeout
internal/metrics/metrics.go:6:2: github.com/prometheus/client_golang#v1.12.1: Get "https://proxy.golang.org/github.com/prometheus/client_golang/#v/v1.12.1.zip": dial tcp 142.251.43.17:443: i/o timeout
……
make: *** [Makefile:57: bin/gitlab-shell] Error 1
rake aborted!
command failed: make build
/home/git/gitlab/lib/tasks/gitlab/shell.rake:37:in `block (5 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/shell.rake:35:in `each'
/home/git/gitlab/lib/tasks/gitlab/shell.rake:35:in `block (4 levels) in <top (required)>'
/home/git/gitlab/lib/tasks/gitlab/shell.rake:20:in `chdir'
/home/git/gitlab/lib/tasks/gitlab/shell.rake:20:in `block (3 levels) in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:58:in `load'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:58:in `kernel_load'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli/exec.rb:23:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli.rb:484:in `exec'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli.rb:31:in `dispatch'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/cli.rb:25:in `start'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/exe/bundle:48:in `block in <top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
/home/git/gitlab/vendor/bundle/ruby/2.7.0/gems/bundler-2.3.6/exe/bundle:36:in `<top (required)>'
/usr/local/bin/bundle:25:in `load'
/usr/local/bin/bundle:25:in `<main>'
/usr/local/bin/ruby_executable_hooks:22:in `eval'
/usr/local/bin/ruby_executable_hooks:22:in `<main>'
Tasks: TOP => gitlab:shell:install
(See full trace by running task with --trace)
I have modified my GO111MODULE=on and GOPROXY=https://goproxy.cn,direct, but it still shows ... Get "https://proxy.golang.org/... ... i/o timeout, so what should I do or where should I modify the source code? Or are there actually other issues here that need to be addressed?

How to fix Gitlab CE 14.1.6 upgrade failed

I wanted to update my GitLab CE installed on a EC2 instance following the recommendation. I went from 13.0.x to 13.9.2 -> 13.12.12 -> 14.0.11 -> 14.1.6 as the documentation recommend.
Every time I check if the server was responding. I just had to restart the process when I jump to 14.X.
But when I went to 14.1.6 I got an error on the update :
[...]
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action restart (up to date)
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrating ==================
-- add_column(:plan_limits, :pages_file_entries, :integer, {:default=>200000, :null=>false})
-> 0.0021s
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrated (0.0022s) =========
== 20210622045705 FinalizeEventsBigintConversion: migrating ===================
STDERR:
---- End output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
Ran "bash" "/tmp/chef-script20211028-68712-eysjor" returned 1
Running handlers complete
Chef Infra Client failed. 27 resources updated in 51 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
So as the output told me I tried :
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
and got
rake aborted!
JSON::ParserError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Oj::ParseError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)
Any idea ?
After upgrading from 13 to 14.0 gitlab will commence several database migrations in the background (see yourgitlabserver.example.com/admin/background_migrations).
You have to wait for these to complete before advancing any further.
If you already did, there are several ways to fix this (see documentation).
I had the same error in this situation. To find out what's going on I've added following code into the
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17
puts "JSON: '#{args[:job_arguments]}'"
to have the code looks like this:
puts "JSON: '#{args[:job_arguments]}'"
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.finalize(
args[:job_class_name],
args[:table_name],
args[:column_name],
Gitlab::Json.parse(args[:job_arguments])
)
And I noticed that command args[:job_arguments] returns invalid JSON string.
So finally for me (zsh) worked to execute the following command:
sudo gitlab-rake 'gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,[["id"]\, ["id_convert_to_bigint"]]]'
The difference is in single quotes.

Gitlab-ce backup restore failing for certain repos

I'm trying to set up a perpetually 1 day old clone of a gitlab server. The gitlab version on both machines 12.4.1-ce.0. There's a daily crontab for the following:
gitlab-rake GZIP_RSYNCABLE=yes gitlab:backup:create
Then it transfers the backup to my clone machine and it runs
gitlab-ctl stop unicorn && gitlab-ctl stop sidekiq && gitlab-rake gitlab:backup:restore BACKUP=most_recent force=yes --trace && gitlab-ctl restart
All of the infrastructure there works and I'm happy with it, but then during the restore, many many repos are failing to restore, throwing one of two messages for one of two types of restores:
repos larger than ~800MB
Error: 4:Deadline Exceeded
all wikis
Error 13:CreateRepositoryFromBundle: cmd wait failed: exit status 128
Searching around online said that this is probably a gitaly timeout issue, so I changed the default timeout from 55s to 475s, the fast from 10s to 30s, and the medium from 30s to 115s, but that made no difference.
At the end of the restore, I get this message:
2019-11-20 12:46:14 -0500 -- Restoring lfs objects ...
2019-11-20 12:46:14 -0500 -- done
** Invoke gitlab:backup:registry:restore (first_time)
** Invoke gitlab_environment
** Execute gitlab:backup:registry:restore
2019-11-20 12:46:14 -0500 -- Restoring container registry images ...
rake aborted!
Backup::Error: Backup failed
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:81:in `run_pipeline!'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/files.rb:44:in `restore'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:234:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:71:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.6.0/monitor.rb:230:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:registry:restore
I'm not even sure if that's because of the same problem, or if it's entirely unrelated. Anyone know what's wrong, or where I can start looking next?

Updating failure - Gitlab CE 7.0.0 - Ubuntu 12.04 - Package installation

Upgrading Gitlab CE 7.0.0 to latest version using the package at https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/precise/gitlab-ce_8.12.4-ce.0_amd64.deb failed.
I have followed this guide.
Please see below the log entries below after runnnig sudo gitlab-rake gitlab:check SANITIZE=true.
Checking GitLab Shell ...
GitLab Shell version >= 3.6.1 ? ... OK (3.6.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column projects.pending_delete does not exist
LINE 1: SELECT COUNT(*) FROM "projects" WHERE "projects"."pending_de...
^
: SELECT COUNT(*) FROM "projects" WHERE "projects"."pending_delete" = 'f'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:270:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:227:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:133:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:48:in `count'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:13:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:469:in `check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:348:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
PG::UndefinedColumn: ERROR: column projects.pending_delete does not exist
LINE 1: SELECT COUNT(*) FROM "projects" WHERE "projects"."pending_de...
^
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `async_exec'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `block in exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:592:in `exec_no_cache'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql_adapter.rb:584:in `execute_and_clear'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:160:in `exec_query'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:356:in `select'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/database_statements.rb:32:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/query_cache.rb:70:in `select_all'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:270:in `execute_simple_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:227:in `perform_calculation'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:133:in `calculate'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/relation/calculations.rb:48:in `count'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/querying.rb:13:in `count'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:469:in `check_repos_hooks_directory_is_link'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/check.rake:348:in `block (3 levels) in <top (required)>'
Here is the environment info run with sudo gitlab-rake gitlab:env:info
System: Ubuntu 12.04
Current User: git
Using RVM: no
Ruby Version: 2.3.1p112
Gem Version: 2.6.6
Bundler Version:1.13.1
Rake Version: 10.5.0
Sidekiq Version:4.1.4
rake aborted!
ActiveRecord::UnknownAttributeError: unknown attribute 'pending_delete' for Project.
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:59:in `rescue in _assign_attribute'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:54:in `_assign_attribute'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:41:in `block in assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:169:in `initialize_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:251:in `block in build_record'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:465:in `block in initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:283:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:142:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:463:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/reflection.rb:141:in `build_association'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:250:in `build_record'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_association.rb:146:in `build'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_proxy.rb:259:in `build'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:33:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
NoMethodError: undefined method `pending_delete=' for #<Project:0x00000007337b90>
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activemodel-4.2.7.1/lib/active_model/attribute_methods.rb:433:in `method_missing'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:54:in `public_send'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:54:in `_assign_attribute'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:41:in `block in assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `each'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/attribute_assignment.rb:35:in `assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:28:in `perform_attribute_assignment'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/attr_encrypted-3.0.3/lib/attr_encrypted/adapters/active_record.rb:36:in `assign_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:169:in `initialize_attributes'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:251:in `block in build_record'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:465:in `block in initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/core.rb:283:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/default_value_for-3.0.2/lib/default_value_for.rb:142:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/state_machines-activerecord-0.4.0/lib/state_machines/integrations/active_record.rb:463:in `initialize'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/inheritance.rb:61:in `new'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/reflection.rb:141:in `build_association'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/association.rb:250:in `build_record'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_association.rb:146:in `build'
/opt/gitlab/embedded/service/gem/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/associations/collection_proxy.rb:259:in `build'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/info.rake:33:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:env:info
Following steps helped me complete this upgrade:
Upgrade to 7.10 using the package at https://packages.gitlab.com/gitlab/gitlab-ce/packages/el/6/gitlab-ce-7.10.0~omnibus-1.x86_64.rpm
Then update to latest version: curl -sS https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash followed by sudo apt-get install gitlab-ce

Resources