This is Nithish Kumar G working as a Devops Engineer in Metricstream company,we are using a gitlab community edition(13.12.15 version) , We are facing issue while upgrading gitlab-ce edition , we are following the upgrade path mentioned in the document as 13.12.15 -> 14.0.12 -> 14.3.6 , we successfully upgraded to 14.0.12 and container was up and running and background migrations completed successfully after sometime and then we tried to upgrade to 14.3.6 the container was not coming up and continously restarting in a loop, when we go through the logs found some below error
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20221207-31-1820bvw" ----
STDOUT: rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
ActiveRecord::StatementInvalid: PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Caused by:
PG::InsufficientPrivilege: ERROR: must be owner of table schema_migrations
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20210616134905_add_timestamp_to_schema_migration.rb:6:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:260:in `block in transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database.rb:259:in `transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210616134905 AddTimestampToSchemaMigration: migrating ====================
-- add_column(:schema_migrations, :finished_at, :timestamptz)
STDERR:
---- End output of "bash" "/tmp/chef-script20221207-31-1820bvw" ----
Ran "bash" "/tmp/chef-script20221207-31-1820bvw" returned 1
I tried upgrading gitlab-ce from 13.12.15 to 14.0.12 and it is successfull and container is coming up but when i tried to upgrade from 14.0.12 to 14.3.6 after completing background migrations in 14.0.12 it is not coming up and container restarting in loop.
I wanted to update my GitLab CE installed on a EC2 instance following the recommendation. I went from 13.0.x to 13.9.2 -> 13.12.12 -> 14.0.11 -> 14.1.6 as the documentation recommend.
Every time I check if the server was responding. I just had to restart the process when I jump to 14.X.
But when I went to 14.1.6 I got an error on the update :
[...]
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action restart (up to date)
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrating ==================
-- add_column(:plan_limits, :pages_file_entries, :integer, {:default=>200000, :null=>false})
-> 0.0021s
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrated (0.0022s) =========
== 20210622045705 FinalizeEventsBigintConversion: migrating ===================
STDERR:
---- End output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
Ran "bash" "/tmp/chef-script20211028-68712-eysjor" returned 1
Running handlers complete
Chef Infra Client failed. 27 resources updated in 51 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
So as the output told me I tried :
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
and got
rake aborted!
JSON::ParserError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Oj::ParseError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)
Any idea ?
After upgrading from 13 to 14.0 gitlab will commence several database migrations in the background (see yourgitlabserver.example.com/admin/background_migrations).
You have to wait for these to complete before advancing any further.
If you already did, there are several ways to fix this (see documentation).
I had the same error in this situation. To find out what's going on I've added following code into the
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17
puts "JSON: '#{args[:job_arguments]}'"
to have the code looks like this:
puts "JSON: '#{args[:job_arguments]}'"
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.finalize(
args[:job_class_name],
args[:table_name],
args[:column_name],
Gitlab::Json.parse(args[:job_arguments])
)
And I noticed that command args[:job_arguments] returns invalid JSON string.
So finally for me (zsh) worked to execute the following command:
sudo gitlab-rake 'gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,[["id"]\, ["id_convert_to_bigint"]]]'
The difference is in single quotes.
I'm using
Ubuntu 18.04 server
Trying to install Gitlab standalone using this blog and official one too
I am getting the below error:-
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20180927-7185-157g8fm" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:51:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20180927-7185-157g8fm" ----
Ran "bash" "/tmp/chef-script20180927-7185-157g8fm" returned 1
Running handlers complete
Chef Client failed. 4 resources updated in 01 minutes 21 seconds
I thought I was messing up with the URL.
$grep "external" /etc/gitlab/gitlab.rb | head -n 3
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'http://192.168.4.215'
I tried varius options for external_url like
IP address
domain name
Let me know what is going wrong.
I had the same problem and found a workaround here:
gitlab.com/gitlab-org/omnibus-gitlab/issues/3744
Doing the following steps solved the problem:
sudo systemctl disable gitlab-runsvdir
sudo systemctl enable gitlab-runsvdir
sudo gitlab-ctl restart
I'm trying to get a node.js app to start using foreman. I run the forman start command in the projects root directory and I keep getting the following error:
/Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/dotenv-0.8.0/lib/dotenv/environment.rb:34:in `block in load': Line "== project-ps Config Vars" doesn't match format (Dotenv::FormatError)
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/dotenv-0.8.0/lib/dotenv/environment.rb:27:in `each'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/dotenv-0.8.0/lib/dotenv/environment.rb:27:in `load'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/dotenv-0.8.0/lib/dotenv/environment.rb:23:in `initialize'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/foreman-0.63.0/lib/foreman/engine.rb:172:in `new'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/foreman-0.63.0/lib/foreman/engine.rb:172:in `load_env'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/foreman-0.63.0/lib/foreman/cli.rb:136:in `load_environment!'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/foreman-0.63.0/lib/foreman/cli.rb:38:in `start'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/command.rb:27:in `run'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/invocation.rb:120:in `invoke_command'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor.rb:363:in `dispatch'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/thor-0.18.1/lib/thor/base.rb:439:in `start'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/gems/foreman-0.63.0/bin/foreman:7:in `<top (required)>'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/bin/foreman:23:in `load'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/bin/foreman:23:in `<main>'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `eval'
from /Users/Hebime/.rvm/gems/ruby-1.9.3-p448/bin/ruby_noexec_wrapper:14:in `<main>'
The project works perfectly fine on my co-workers machines using the same command.
I'm using OSX 10.8.4.
Any ideas?
Thanks!
I ran into this error too. In my case I had accidentally included an "activate" command inside my .env file:
. venv/bin/activate
export DEBUG=True
...
When I ran foreman, it parsed the .env file and it stumbled on that first line. By commenting out that activate command, the problem went away.
Showing Error while running script by using Firefox
With user of Roo & test/unit
I call information from excel sheet & user in executing but showing error
Showing Error as below
C:/Ruby193/lib/ruby/gems/1.9.1/gems/ffi-1.0.9-x86-mingw32/lib/ffi/platform.rb:2
: Use RbConfig instead of obsolete and deprecated Config.
Started ChromeDriver
port=52228
version=20.0.1133.0
log=D:\chromedriver.log
C:/Ruby193/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock': An existing c
nnection was forcibly closed by the remote host. (Errno::ECONNRESET)
from C:/Ruby193/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
from C:/Ruby193/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
from C:/Ruby193/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_
equest'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:1293:in `request'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:1064:in `head'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.21.2/lib/
elenium/webdriver/chrome/service.rb:62:in `block in stop'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:745:in `start'
from C:/Ruby193/lib/ruby/1.9.1/net/http.rb:557:in `start'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.21.2/lib/
elenium/webdriver/chrome/service.rb:58:in `stop'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.21.2/lib/
elenium/webdriver/chrome/service.rb:52:in `block in start'
from C:/Ruby193/lib/ruby/gems/1.9.1/gems/selenium-webdriver-2.21.2/lib/
elenium/webdriver/common/platform.rb:124:in `block in exit_hook'
Please check whether you have latest ffi gem installed.
gem install ffi
Then give it a shot.