How to fix Gitlab CE 14.1.6 upgrade failed - gitlab

Question

I wanted to update my GitLab CE installed on a EC2 instance following the recommendation. I went from 13.0.x to 13.9.2 -> 13.12.12 -> 14.0.11 -> 14.1.6 as the documentation recommend.
Every time I check if the server was responding. I just had to restart the process when I jump to 14.X.
But when I went to 14.1.6 I got an error on the update :
[...]
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action restart (up to date)
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrating ==================
-- add_column(:plan_limits, :pages_file_entries, :integer, {:default=>200000, :null=>false})
-> 0.0021s
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrated (0.0022s) =========
== 20210622045705 FinalizeEventsBigintConversion: migrating ===================
STDERR:
---- End output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
Ran "bash" "/tmp/chef-script20211028-68712-eysjor" returned 1
Running handlers complete
Chef Infra Client failed. 27 resources updated in 51 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
So as the output told me I tried :
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
and got
rake aborted!
JSON::ParserError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Oj::ParseError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)
Any idea ?

Answer 1

After upgrading from 13 to 14.0 gitlab will commence several database migrations in the background (see yourgitlabserver.example.com/admin/background_migrations).
You have to wait for these to complete before advancing any further.
If you already did, there are several ways to fix this (see documentation).

Answer 2

I had the same error in this situation. To find out what's going on I've added following code into the
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17
puts "JSON: '#{args[:job_arguments]}'"
to have the code looks like this:
puts "JSON: '#{args[:job_arguments]}'"
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.finalize(
args[:job_class_name],
args[:table_name],
args[:column_name],
Gitlab::Json.parse(args[:job_arguments])
)
And I noticed that command args[:job_arguments] returns invalid JSON string.
So finally for me (zsh) worked to execute the following command:
sudo gitlab-rake 'gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,[["id"]\, ["id_convert_to_bigint"]]]'
The difference is in single quotes.