GitLab upgrade from 9.5.10 to 10.8.7 unsuccessful - gitlab

Question

After installing the package for 10.8.7, which is the next supported upgrade path per https://docs.gitlab.com/ee/update/#upgrade-paths, gitlab-ctl reconfigure fails with:
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================
NoMethodError
-------------
undefined method `[]' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:81:in `write_to_gitlab_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:186:in `generate_secrets'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:259:in `generate_config'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/secrets_helper.rb:
74: },
75: 'registry' => {
76: 'http_secret' => Gitlab['registry']['http_secret'],
77: 'internal_certificate' => Gitlab['registry']['internal_certificate'],
78: 'internal_key' => Gitlab['registry']['internal_key']
79: },
80: 'letsencrypt' => {
81>> 'auto_enabled' => Gitlab['letsencrypt']['auto_enabled']
82: },
83: 'mattermost' => {
84: 'email_invite_salt' => Gitlab['mattermost']['email_invite_salt'],
85: 'file_public_link_salt' => Gitlab['mattermost']['file_public_link_salt'],
86: 'sql_at_rest_encrypt_key' => Gitlab['mattermost']['sql_at_rest_encrypt_key']
87: },
88: 'postgresql' => {
89: 'internal_certificate' => Gitlab['postgresql']['internal_certificate'],
90: 'internal_key' => Gitlab['postgresql']['internal_key']
Seeing the error, I attempted to add the following to /etc/gitlab/gitlab-secrets.json:
"letsencrypt": {
"auto_enabled": false
},
Now I see a warning along with the error:
Ignoring section letsencrypt in /etc/gitlab/gitlab-secrets.json, does not exist in gitlab.rb
So I added the following to /etc/gitlab/gitlab.rb:
letsencrypt['auto_enabled'] = false
But now I get a different error when running gitlab-ctl reconfigure:
*ERROR*: Encountered unsupported config key 'letsencrypt' in /etc/gitlab/gitlab.rb.
================================================================================
ERROR: Encountered unsupported config key 'letsencrypt' in /etc/gitlab/gitlab.rb.
================================================================================
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb
================================================================================
NoMethodError
-------------
undefined method `[]=' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `block in from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:28:in `auto_vivify'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:100:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:125:in `block in from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:28:in `auto_vivify'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/gitlab.rb:125:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:23:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in `from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:20:in `from_file'
Relevant File Content:
----------------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/helpers/settings_helper.rb:
93: def ee_attribute(name, **config)
94: config = { ee: true }.merge(config)
95: attribute(name, **config)
96: end
97:
98: def from_file(_file_path)
99: # Allow auto mash creation during from_file call
100>> Gitlab::ConfigMash.auto_vivify { super }
101: end
102:
103: def method_missing(method_name, *arguments) # rubocop:disable Style/MethodMissing
104: # Give better message for NilClass errors
105: # If there are no arguments passed, this is a 'GET' call, and if
106: # there is no matching key in the configuration, then it has not been set (not even to nil)
107: # and we will output a nicer error above the exception
108: if arguments.length.zero? && !configuration.key?(method_name)
109: breaktxt = '=' * 80
Additionally, I get the following errors if I try to run gitlab-rake cache:clear:
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/elastic.rake:61: warning: already initialized constant INDEXABLE_CLASSES
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/gitlab/elastic.rake:61: warning: previous definition of INDEXABLE_CLASSES was here
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab.rb:31: warning: already initialized constant Gitlab::VERSION
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/2_app.rb:6: warning: previous definition of VERSION was here
WARNING: Unable to check/update clone_url_prefix for Geo: undefined method `connected?' for Gitlab::Geo:Module
Did you mean? concern
rake aborted!
NameError: uninitialized constant API::Entities::ProjectPushRule
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:28:in `block (2 levels) in <class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:27:in `block in <class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:10:in `<class:ProjectPushRule>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/project_push_rule.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:23:in `block (2 levels) in <class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:22:in `block in <class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:13:in `<class:ProjectGitHook>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:3:in `<module:V3>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/v3/project_git_hook.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:66:in `block in <class:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:25:in `<class:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:2:in `<module:API>'
/opt/gitlab/embedded/service/gitlab-rails/lib/api/api.rb:1:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb:12:in `<top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/config/environment.rb:11:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => cache:clear => cache:clear:redis => environment
(See full trace by running task with --trace)

Answer 1

In my case, I was able to successfully upgrade to 10.8.7 by selecting an arbitrary intermediate version (in my case, 10.5.8). Not sure why directly upgrading as per the upgrade path did not work. I will continue my journey to GitLab 14 now!