My MySQL server just stopped working after ESXi server restart...
It does not start automatically, and I can't start it from my Unix shell.
~# service mysql restart
restart: Unknown instance:
~# sudo service mysql start
start: Job failed to start
~# sudo -u mysql mysqld
~#
~# netstat -tap | grep mysql
~#
~# service mysql restart
restart: Unknown instance:
Files at /var/log/ : mysql.err and mysql.log are empty.
File here /var/log/mysql/error.log has some info:
Version: '5.1.62-0ubuntu0.11.10.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)
130410 9:50:54 [ERROR] /usr/sbin/mysqld: Table './radius/radcheck' is marked as crashed and should be repaired
130410 9:50:54 [Warning] Checking table: './radius/radcheck'
130410 9:50:54 [ERROR] /usr/sbin/mysqld: Table './radius/radpostauth' is marked as crashed and should be repaired
130410 9:50:54 [Warning] Checking table: './radius/radpostauth'
130410 9:50:54 [ERROR] /usr/sbin/mysqld: Table './radius/resv' is marked as crashed and should be repaired
130410 9:50:54 [Warning] Checking table: './radius/resv'
130819 22:31:21 [Note] Plugin 'FEDERATED' is disabled.
130819 22:31:21 InnoDB: Initializing buffer pool, size = 8.0M
130819 22:31:21 InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
130819 22:31:21 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
Could anyone tell me, what could be wrong here?
Ugh... that's ugly.
Here's a fix. When uninstalling, copy the list of packages to be removed, then use that list to reinstall them afterwards.
http://www.blog.webcare.pk/2012/05/ubuntu-mysql-cannot-start-after-upgrade.html
Related
I am trying to install mysql non-interactively (without being asked any configuration questions) with the Dockerfile, and I found the solution to how to do that in : how to use here-strings in the Dockerfile?
but now I am facing another problem when I am trying to start mysql using service mysql start inside the container It won't work and I wonder why? because it was working perfectly when I was doing the installation manually inside the container of debian:buster image.
As you can see in the following it won't start :
root#839a7dc4869a:/# service --status-all
[ ? ] hwclock.sh
[ - ] mysql
root#839a7dc4869a:/# service mysql start
....................................................................................
....................................................................................
[info] MySQL Community Server 5.7.28 did not start. Please check logs for more details..
and this is my Dockerfile:
From debian:buster
RUN apt update
RUN apt install -y gnupg wget lsb-release
RUN wget https://dev.mysql.com/get/mysql-apt-config_0.8.13-1_all.deb
RUN printf "1\n1\n4\n" | dpkg -i mysql-apt-config_0.8.13-1_all.deb
RUN apt update
RUN printf '%s\n' 'mysql-community-server mysql-community-server/root-pass password your_password' | debconf-set-selections && \
printf '%s\n' 'mysql-community-server mysql-community-server/re-root-pass password your_password' | debconf-set-selections && \
apt-get -y install mysql-server
CMD bash
this is the logfile using less /var/log/mysql/error.log
2020-01-07T21:47:54.400816Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-07T21:47:55.708390Z 0 [Warning] InnoDB: New log files created, LSN=45790
2020-01-07T21:47:55.886509Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2020-01-07T21:47:56.038387Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 5d1f693e-3197-11ea-95e2-0242ac110002.
2020-01-07T21:47:56.040635Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-01-07T21:47:56.456659Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-07T21:47:56.740853Z 1 [Warning] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2020-01-07T21:47:59.268086Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-07T21:47:59.269832Z 0 [Note] mysqld (mysqld 5.7.28) starting as process 334 ...
2020-01-07T21:47:59.274112Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-01-07T21:47:59.274187Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-07T21:47:59.274191Z 0 [Note] InnoDB: Uses event mutexes
2020-01-07T21:47:59.274194Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-01-07T21:47:59.274196Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-07T21:47:59.274199Z 0 [Note] InnoDB: Using Linux native AIO
2020-01-07T21:47:59.274482Z 0 [Note] InnoDB: Number of pools: 1
2020-01-07T21:47:59.274639Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-01-07T21:47:59.276370Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-01-07T21:47:59.281119Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-07T21:47:59.282302Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-07T21:47:59.294898Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-01-07T21:47:59.301611Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-07T21:47:59.301665Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-07T21:47:59.344888Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-07T21:47:59.345338Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-01-07T21:47:59.345345Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-01-07T21:47:59.346096Z 0 [Note] InnoDB: Waiting for purge to start
2020-01-07T21:47:59.396626Z 0 [Note] InnoDB: 5.7.28 started; log sequence number 2628227
2020-01-07T21:47:59.396804Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-07T21:47:59.396852Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-01-07T21:47:59.397717Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200107 21:47:59
2020-01-07T21:47:59.400400Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-01-07T21:47:59.400413Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-01-07T21:47:59.400770Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-07T21:47:59.400790Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-01-07T21:47:59.400992Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2020-01-07T21:47:59.401006Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2020-01-07T21:47:59.401021Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-01-07T21:47:59.403395Z 0 [Warning] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-07T21:47:59.408666Z 0 [Note] Event Scheduler: Loaded 0 events
2020-01-07T21:47:59.408808Z 0 [Note] Execution of init_file '/var/lib/mysql-files/7pxJ7t37bI' started.
2020-01-07T21:47:59.409234Z 0 [Note] Execution of init_file '/var/lib/mysql-files/7pxJ7t37bI' ended.
2020-01-07T21:47:59.409284Z 0 [Note] mysqld: ready for connections.
Version: '5.7.28' socket: '/tmp/tmp.q7db4MMdAE/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2020-01-07T21:47:59.409594Z 0 [Note] Giving 0 client threads a chance to die gracefully
2020-01-07T21:47:59.409606Z 0 [Note] Shutting down slave threads
2020-01-07T21:47:59.409609Z 0 [Note] Forcefully disconnecting 0 remaining clients
2020-01-07T21:47:59.409612Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-01-07T21:47:59.409663Z 0 [Note] Binlog end
2020-01-07T21:47:59.409938Z 0 [Note] Shutting down plugin 'ngram'
2020-01-07T21:47:59.409944Z 0 [Note] Shutting down plugin 'partition'
2020-01-07T21:47:59.409946Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2020-01-07T21:47:59.409948Z 0 [Note] Shutting down plugin 'ARCHIVE'
2020-01-07T21:47:59.409950Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2020-01-07T21:47:59.409962Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2020-01-07T21:47:59.409964Z 0 [Note] Shutting down plugin 'MyISAM'
2020-01-07T21:47:59.409969Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2020-01-07T21:47:59.409971Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2020-01-07T21:47:59.409973Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2020-01-07T21:47:59.409975Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2020-01-07T21:47:59.409976Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2020-01-07T21:47:59.409978Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2020-01-07T21:47:59.409980Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2020-01-07T21:47:59.409981Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2020-01-07T21:47:59.409983Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2020-01-07T21:47:59.409985Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2020-01-07T21:47:59.409986Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2020-01-07T21:47:59.409988Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2020-01-07T21:47:59.409990Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2020-01-07T21:47:59.409991Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2020-01-07T21:47:59.409993Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2020-01-07T21:47:59.409995Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2020-01-07T21:47:59.409996Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2020-01-07T21:47:59.409998Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2020-01-07T21:47:59.409999Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2020-01-07T21:47:59.410001Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2020-01-07T21:47:59.410003Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2020-01-07T21:47:59.410004Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2020-01-07T21:47:59.410006Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2020-01-07T21:47:59.410008Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2020-01-07T21:47:59.410009Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2020-01-07T21:47:59.410011Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2020-01-07T21:47:59.410012Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2020-01-07T21:47:59.410014Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2020-01-07T21:47:59.410016Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2020-01-07T21:47:59.410017Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2020-01-07T21:47:59.410019Z 0 [Note] Shutting down plugin 'InnoDB'
2020-01-07T21:47:59.410083Z 0 [Note] InnoDB: FTS optimize thread exiting.
2020-01-07T21:47:59.410230Z 0 [Note] InnoDB: Starting shutdown...
2020-01-07T21:47:59.511556Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-01-07T21:47:59.511701Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200107 21:47:59
2020-01-07T21:48:00.958542Z 0 [Note] InnoDB: Shutdown completed; log sequence number 2628246
2020-01-07T21:48:00.960660Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-01-07T21:48:00.960692Z 0 [Note] Shutting down plugin 'MEMORY'
2020-01-07T21:48:00.960697Z 0 [Note] Shutting down plugin 'CSV'
2020-01-07T21:48:00.960701Z 0 [Note] Shutting down plugin 'sha256_password'
2020-01-07T21:48:00.960703Z 0 [Note] Shutting down plugin 'mysql_native_password'
2020-01-07T21:48:00.960789Z 0 [Note] Shutting down plugin 'binlog'
2020-01-07T21:48:00.962661Z 0 [Note] mysqld: Shutdown complete
and this is the logfile after i run service mysql start :
2020-01-07T21:47:54.400816Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-07T21:47:55.708390Z 0 [Warning] InnoDB: New log files created, LSN=45790
2020-01-07T21:47:55.886509Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2020-01-07T21:47:56.038387Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 5d1f693e-3197-11ea-95e2-0242ac110002.
2020-01-07T21:47:56.040635Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2020-01-07T21:47:56.456659Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-07T21:47:56.740853Z 1 [Warning] root#localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2020-01-07T21:47:59.268086Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-07T21:47:59.269832Z 0 [Note] mysqld (mysqld 5.7.28) starting as process 334 ...
2020-01-07T21:47:59.274112Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-01-07T21:47:59.274187Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-07T21:47:59.274191Z 0 [Note] InnoDB: Uses event mutexes
2020-01-07T21:47:59.274194Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-01-07T21:47:59.274196Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-07T21:47:59.274199Z 0 [Note] InnoDB: Using Linux native AIO
2020-01-07T21:47:59.274482Z 0 [Note] InnoDB: Number of pools: 1
2020-01-07T21:47:59.274639Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-01-07T21:47:59.276370Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-01-07T21:47:59.281119Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-07T21:47:59.282302Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-07T21:47:59.294898Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-01-07T21:47:59.301611Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-07T21:47:59.301665Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-07T21:47:59.344888Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-07T21:47:59.345338Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-01-07T21:47:59.345345Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-01-07T21:47:59.346096Z 0 [Note] InnoDB: Waiting for purge to start
2020-01-07T21:47:59.396626Z 0 [Note] InnoDB: 5.7.28 started; log sequence number 2628227
2020-01-07T21:47:59.396804Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-07T21:47:59.396852Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-01-07T21:47:59.397717Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200107 21:47:59
2020-01-07T21:47:59.400400Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-01-07T21:47:59.400413Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-01-07T21:47:59.400770Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-07T21:47:59.400790Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-01-07T21:47:59.400992Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2020-01-07T21:47:59.401006Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2020-01-07T21:47:59.401021Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-01-07T21:47:59.403395Z 0 [Warning] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2020-01-07T21:47:59.408666Z 0 [Note] Event Scheduler: Loaded 0 events
2020-01-07T21:47:59.408808Z 0 [Note] Execution of init_file '/var/lib/mysql-files/7pxJ7t37bI' started.
2020-01-07T21:47:59.409234Z 0 [Note] Execution of init_file '/var/lib/mysql-files/7pxJ7t37bI' ended.
2020-01-07T21:47:59.409284Z 0 [Note] mysqld: ready for connections.
Version: '5.7.28' socket: '/tmp/tmp.q7db4MMdAE/mysqld.sock' port: 3306 MySQL Community Server (GPL)
2020-01-07T21:47:59.409594Z 0 [Note] Giving 0 client threads a chance to die gracefully
2020-01-07T21:47:59.409606Z 0 [Note] Shutting down slave threads
2020-01-07T21:47:59.409609Z 0 [Note] Forcefully disconnecting 0 remaining clients
2020-01-07T21:47:59.409612Z 0 [Note] Event Scheduler: Purging the queue. 0 events
2020-01-07T21:47:59.409663Z 0 [Note] Binlog end
2020-01-07T21:47:59.409938Z 0 [Note] Shutting down plugin 'ngram'
2020-01-07T21:47:59.409944Z 0 [Note] Shutting down plugin 'partition'
2020-01-07T21:47:59.409946Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2020-01-07T21:47:59.409948Z 0 [Note] Shutting down plugin 'ARCHIVE'
2020-01-07T21:47:59.409950Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2020-01-07T21:47:59.409962Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2020-01-07T21:47:59.409964Z 0 [Note] Shutting down plugin 'MyISAM'
2020-01-07T21:47:59.409969Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2020-01-07T21:47:59.409971Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2020-01-07T21:47:59.409973Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2020-01-07T21:47:59.409975Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2020-01-07T21:47:59.409976Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2020-01-07T21:47:59.409978Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2020-01-07T21:47:59.409980Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2020-01-07T21:47:59.409981Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2020-01-07T21:47:59.409983Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2020-01-07T21:47:59.409985Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2020-01-07T21:47:59.409986Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2020-01-07T21:47:59.409988Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2020-01-07T21:47:59.409990Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2020-01-07T21:47:59.409991Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2020-01-07T21:47:59.409993Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2020-01-07T21:47:59.409995Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2020-01-07T21:47:59.409996Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2020-01-07T21:47:59.409998Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2020-01-07T21:47:59.409999Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2020-01-07T21:47:59.410001Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2020-01-07T21:47:59.410003Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2020-01-07T21:47:59.410004Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2020-01-07T21:47:59.410006Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2020-01-07T21:47:59.410008Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2020-01-07T21:47:59.410009Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2020-01-07T21:47:59.410011Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2020-01-07T21:47:59.410012Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2020-01-07T21:47:59.410014Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2020-01-07T21:47:59.410016Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2020-01-07T21:47:59.410017Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2020-01-07T21:47:59.410019Z 0 [Note] Shutting down plugin 'InnoDB'
2020-01-07T21:47:59.410083Z 0 [Note] InnoDB: FTS optimize thread exiting.
2020-01-07T21:47:59.410230Z 0 [Note] InnoDB: Starting shutdown...
2020-01-07T21:47:59.511556Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-01-07T21:47:59.511701Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200107 21:47:59
2020-01-07T21:48:00.958542Z 0 [Note] InnoDB: Shutdown completed; log sequence number 2628246
2020-01-07T21:48:00.960660Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-01-07T21:48:00.960692Z 0 [Note] Shutting down plugin 'MEMORY'
2020-01-07T21:48:00.960697Z 0 [Note] Shutting down plugin 'CSV'
2020-01-07T21:48:00.960701Z 0 [Note] Shutting down plugin 'sha256_password'
2020-01-07T21:48:00.960703Z 0 [Note] Shutting down plugin 'mysql_native_password'
2020-01-07T21:48:00.960789Z 0 [Note] Shutting down plugin 'binlog'
2020-01-07T21:48:00.962661Z 0 [Note] mysqld: Shutdown complete
2020-01-07T21:59:26.471325Z mysqld_safe Logging to '/var/log/mysql/error.log'.
2020-01-07T21:59:26.486959Z mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2020-01-07T21:59:26.643199Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2020-01-07T21:59:26.644096Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.28) starting as process 408 ...
2020-01-07T21:59:26.646176Z 0 [Note] InnoDB: PUNCH HOLE support available
2020-01-07T21:59:26.646190Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2020-01-07T21:59:26.646193Z 0 [Note] InnoDB: Uses event mutexes
2020-01-07T21:59:26.646196Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2020-01-07T21:59:26.646198Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2020-01-07T21:59:26.646201Z 0 [Note] InnoDB: Using Linux native AIO
2020-01-07T21:59:26.646331Z 0 [Note] InnoDB: Number of pools: 1
2020-01-07T21:59:26.646391Z 0 [Note] InnoDB: Using CPU crc32 instructions
2020-01-07T21:59:26.647165Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2020-01-07T21:59:26.651798Z 0 [Note] InnoDB: Completed initialization of buffer pool
2020-01-07T21:59:26.653130Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-01-07T21:59:27.636212Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2020-01-07T21:59:27.642182Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2020-01-07T21:59:27.642232Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2020-01-07T21:59:27.689063Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-01-07T21:59:27.689519Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2020-01-07T21:59:27.689527Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2020-01-07T21:59:27.689913Z 0 [Note] InnoDB: Waiting for purge to start
2020-01-07T21:59:27.740941Z 0 [Note] InnoDB: 5.7.28 started; log sequence number 2628246
2020-01-07T21:59:27.741098Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2020-01-07T21:59:27.741221Z 0 [Note] Plugin 'FEDERATED' is disabled.
2020-01-07T21:59:27.810722Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2020-01-07T21:59:27.810737Z 0 [Note] Skipping generation of SSL certificates as certificate files are present in data directory.
2020-01-07T21:59:27.811201Z 0 [Warning] CA certificate ca.pem is self signed.
2020-01-07T21:59:27.811224Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
2020-01-07T21:59:27.811663Z 0 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
2020-01-07T21:59:27.811680Z 0 [Note] - '127.0.0.1' resolves to '127.0.0.1';
2020-01-07T21:59:27.811738Z 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-01-07T21:59:27.813580Z 0 [Note] InnoDB: Buffer pool(s) load completed at 200107 21:59:27
2020-01-07T21:59:27.816611Z 0 [ERROR] Fatal error: Can't open and lock privilege tables: Table storage engine for 'user' doesn't have this option
2020-01-07T21:59:27.816628Z 0 [ERROR] Fatal error: Failed to initialize ACL/grant/time zones structures or failed to remove temporary table files.
2020-01-07T21:59:27.816674Z 0 [ERROR] Aborting
2020-01-07T21:59:27.816694Z 0 [Note] Binlog end
2020-01-07T21:59:27.816756Z 0 [Note] Shutting down plugin 'ngram'
2020-01-07T21:59:27.816763Z 0 [Note] Shutting down plugin 'partition'
2020-01-07T21:59:27.816765Z 0 [Note] Shutting down plugin 'BLACKHOLE'
2020-01-07T21:59:27.816767Z 0 [Note] Shutting down plugin 'ARCHIVE'
2020-01-07T21:59:27.816769Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
2020-01-07T21:59:27.816803Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
2020-01-07T21:59:27.816806Z 0 [Note] Shutting down plugin 'MyISAM'
2020-01-07T21:59:27.816811Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
2020-01-07T21:59:27.816814Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
2020-01-07T21:59:27.816816Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
2020-01-07T21:59:27.816817Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
2020-01-07T21:59:27.816819Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
2020-01-07T21:59:27.816821Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
2020-01-07T21:59:27.816822Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
2020-01-07T21:59:27.816824Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
2020-01-07T21:59:27.816826Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
2020-01-07T21:59:27.816828Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
2020-01-07T21:59:27.816829Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
2020-01-07T21:59:27.816831Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
2020-01-07T21:59:27.816833Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
2020-01-07T21:59:27.816834Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
2020-01-07T21:59:27.816836Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
2020-01-07T21:59:27.816838Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
2020-01-07T21:59:27.816839Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
2020-01-07T21:59:27.816841Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
2020-01-07T21:59:27.816843Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
2020-01-07T21:59:27.816844Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
2020-01-07T21:59:27.816846Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
2020-01-07T21:59:27.816848Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
2020-01-07T21:59:27.816849Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
2020-01-07T21:59:27.816851Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
2020-01-07T21:59:27.816852Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
2020-01-07T21:59:27.816854Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
2020-01-07T21:59:27.816856Z 0 [Note] Shutting down plugin 'INNODB_CMP'
2020-01-07T21:59:27.816857Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
2020-01-07T21:59:27.816859Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
2020-01-07T21:59:27.816861Z 0 [Note] Shutting down plugin 'INNODB_TRX'
2020-01-07T21:59:27.816863Z 0 [Note] Shutting down plugin 'InnoDB'
2020-01-07T21:59:27.816936Z 0 [Note] InnoDB: FTS optimize thread exiting.
2020-01-07T21:59:27.817047Z 0 [Note] InnoDB: Starting shutdown...
2020-01-07T21:59:27.922074Z 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2020-01-07T21:59:27.922231Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 200107 21:59:27
2020-01-07T21:59:29.468679Z 0 [Note] InnoDB: Shutdown completed; log sequence number 2628265
2020-01-07T21:59:29.470792Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2020-01-07T21:59:29.470822Z 0 [Note] Shutting down plugin 'MEMORY'
2020-01-07T21:59:29.470827Z 0 [Note] Shutting down plugin 'CSV'
2020-01-07T21:59:29.470830Z 0 [Note] Shutting down plugin 'sha256_password'
2020-01-07T21:59:29.470832Z 0 [Note] Shutting down plugin 'mysql_native_password'
2020-01-07T21:59:29.470919Z 0 [Note] Shutting down plugin 'binlog'
2020-01-07T21:59:29.471136Z 0 [Note] /usr/sbin/mysqld: Shutdown complete
2020-01-07T21:59:29.582468Z mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
So any Idea why I am facing this problem or any hint that can lend me to the right way to do it.
When you create the docker image, it generates an initial database.
The problem is that the files in this location are then mounted on an overlay file system when you try running.
These files can't be used by mysql when it starts up because they're not modifiable in-place.
When you chown, it creates 'fresh' copies of the files, which allows mysql to start up as these files are now modifiable in-place.
If you want a container that can just launch a fresh mysql, the best workaround for this is to delete everything below /var/lib/mysql as part of the image creation step. It will recreate the database on first start, allowing it to be used as needed.
After I did a little research I found the solution, I just need to run this following command :
chown -R mysql:mysql /var/lib/mysql /var/run/mysqld // change the ownership of all files and subdirectories under the /var/lib/mysql and /var/run/mysqld
then I can start mysql with no problems
service mysql start
and now it works perfectly
..
[info] MySQL Community Server 5.7.28 is started.
or I can do it this way :
find /var/lib/mysql -type f -exec touch {} \; //find every file in that directory and execute touch to touch all the files and change the files access.
or you can do it as #Petesh said:
RUN printf '%s\n' 'mysql-community-server mysql-community-server/root-pass password your_password' | debconf-set-selections && \
printf '%s\n' 'mysql-community-server mysql-community-server/re-root-pass password your_password' | debconf-set-selections && \
apt -y install mysql-server && \
rm -rf /var/lib/mysql/* // remove all the files in this directory and when you will run `service mysql start` it'll recreate the database and mysql will start.
I really don't understand why I need to change the user and group of the files in that directories, maybe it's due the build way of the Dockerfile because it uses intermediate containers during the build.
because when I build the image debian:buster without Dockerfile and run the container then install the mysql-server manually and start it, it started perfectly.
correct me if I am wrong.
for more information mysql __ Chown Command __ github
I have deployed this PostgreSQL image to the IBM Cloud, Google Cloud Platform and Microsoft Azure using Kubernetes. https://github.com/paunin/PostDock
It was successfully deployed on all 3 platforms with identical configurations and an identical process. The IBM cloud fails with the error "psql: FATAL: password authentication failed for user "replica_user""
You can find below the logs from all 3 cloud platforms. Has anyone experienced this?
IBM Cloud Log
>>> Setting up STOP handlers...
>>> STARTING SSH (if required)...
>>> SSH is not enabled!
>>> STARTING POSTGRES...
>>> TUNING UP POSTGRES...
>>> Cleaning data folder which might have some garbage...
psql: FATAL: password authentication failed for user "replica_user"
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node2-service" (172.30.65.206) and accepting
TCP/IP connections on port 5432?
>>> Auto-detected master name: ''
>>> Setting up repmgr...
>>> Setting up repmgr config file '/etc/repmgr.conf'...
>>> Setting up upstream node...
cat: /var/lib/postgresql/data/standby.lock: No such file or directory
>>> Previously Locked standby upstream node LOCKED_STANDBY=''
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
psql: FATAL: password authentication failed for user "replica_user"
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node1-service:5432 (will try 30 times more)
....
The last couple of lines are then repeated many times.
This is the log file from deploying the same application, using identical processes on the Google Cloud. It works just fine on the Google Cloud Platform.
Google Cloud Log
>>> Setting up STOP handlers...
>>> STARTING SSH (if required)...
>>> SSH is not enabled!
>>> STARTING POSTGRES...
>>> TUNING UP POSTGRES...
>>> Cleaning data folder which might have some garbage...
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node1-service" (10.52.0.11) and accepting
TCP/IP connections on port 5432?
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node2-service" (10.52.0.12) and accepting
TCP/IP connections on port 5432?
>>> Auto-detected master name: ''
>>> Setting up repmgr...
>>> Setting up repmgr config file '/etc/repmgr.conf'...
>>> Setting up upstream node...
cat: /var/lib/postgresql/data/standby.lock: No such file or directory
>>> Previously Locked standby upstream node LOCKED_STANDBY=''
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node1-service" (10.52.0.11) and accepting
TCP/IP connections on port 5432?
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node1-service:5432 (will try 30 times more)
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node1-service:5432 (will try 29 times more)
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node1-service" (10.52.0.11) and accepting
TCP/IP connections on port 5432?
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node1-service" (10.52.0.11) and accepting
TCP/IP connections on port 5432?
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node1-service:5432 (will try 28 times more)
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
>>> REPLICATION_UPSTREAM_NODE_ID=1
>>> Sending in background postgres start...
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
>>> Starting standby node...
>>> Instance hasn't been set up yet.
>>> Clonning primary node...
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
NOTICE: destination directory '/var/lib/postgresql/data' provided
INFO: connecting to upstream node
INFO: Successfully connected to upstream node. Current installation size is 34 MB
INFO: checking and correcting permissions on existing directory /var/lib/postgresql/data ...
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
NOTICE: starting backup (using pg_basebackup)...
INFO: executing: '/usr/lib/postgresql/9.5/bin/pg_basebackup -l "repmgr base backup" -D /var/lib/postgresql/data -h cyclos-postgres-node1-service -p 5432 -U replica_user -c fast -X stream '
NOTICE: standby clone (using pg_basebackup) complete
NOTICE: you can now start your PostgreSQL server
HINT: for example : pg_ctl -D /var/lib/postgresql/data start
HINT: After starting the server, you need to register this standby with "repmgr standby register"
[REPMGR EVENT] Node id: 2; Event type: standby_clone; Success [1|0]: 1; Time: 2018-02-02 13:24:32.87843+00; Details: Cloned from host 'cyclos-postgres-node1-service', port 5432; backup method: pg_basebackup; --force: Y
>>> Configuring /var/lib/postgresql/data/postgresql.conf
>>>>>> Will add configs to exists file
>>> Starting postgres...
>>> Waiting for local postgres server start...
>>> Wait db replica_db on cyclos-postgres-node2-service:5432(user: replica_user,password: *******), will try 60 times with delay 10 seconds (TIMEOUT=600)
LOG: incomplete startup packet
LOG: incomplete startup packet
LOG: database system was interrupted; last known up at 2018-02-02 13:24:31 UTC
FATAL: the database system is starting up
psql: FATAL: the database system is starting up
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node2-service:5432 (will try 60 times more)
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/20000F8
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
>>>>>> Db replica_db exists on cyclos-postgres-node2-service:5432!
>>> Waiting for replication on this node is over(if any in progress): CLEAN_UP_ON_FAIL=, INTERVAL=30
>>> Replication is done
>>> Unregister the node if it was done before
DELETE 0
>>> Registering node with role standby
INFO: connecting to standby database
INFO: connecting to master database
INFO: retrieving node list for cluster 'postgres_cluster'
INFO: registering the standby
[REPMGR EVENT] Node id: 2; Event type: standby_register; Success [1|0]: 1; Time: 2018-02-02 13:24:51.891592+00; Details:
INFO: standby registration complete
NOTICE: standby node correctly registered for cluster postgres_cluster with id 2 (conninfo: user=replica_user password=replica_pass host=cyclos-postgres-node2-service dbname=replica_db port=5432 connect_timeout=2)
Locking standby (NEW_UPSTREAM_NODE_ID=1)...
>>> Starting repmgr daemon...
[2018-02-02 13:24:53] [NOTICE] looking for configuration file in current directory
[2018-02-02 13:24:53] [NOTICE] looking for configuration file in /etc
[2018-02-02 13:24:53] [NOTICE] configuration file found at: /etc/repmgr.conf
[2018-02-02 13:24:53] [INFO] connecting to database 'user=replica_user password=replica_pass host=cyclos-postgres-node2-service dbname=replica_db port=5432 connect_timeout=2'
[2018-02-02 13:24:53] [INFO] connected to database, checking its state
[2018-02-02 13:24:53] [INFO] connecting to master node of cluster 'postgres_cluster'
[2018-02-02 13:24:53] [INFO] retrieving node list for cluster 'postgres_cluster'
[2018-02-02 13:24:53] [INFO] checking role of cluster node '1'
[2018-02-02 13:24:53] [INFO] checking cluster configuration with schema 'repmgr_postgres_cluster'
[2018-02-02 13:24:53] [INFO] checking node 2 in cluster 'postgres_cluster'
[2018-02-02 13:24:53] [INFO] reloading configuration file
[2018-02-02 13:24:53] [INFO] configuration has not changed
[2018-02-02 13:24:53] [INFO] starting continuous standby node monitoring
ERROR: cannot execute DELETE in a read-only transaction
STATEMENT: DELETE FROM repmgr_postgres_cluster.repl_nodes WHERE conninfo LIKE '%host=cyclos-postgres-node3-service%'
And on the Azure Cloud, it works just fine as well.
Azure Cloud Log
>>> Setting up STOP handlers...
>>> STARTING SSH (if required)...
>>> SSH is not enabled!
>>> STARTING POSTGRES...
>>> TUNING UP POSTGRES...
>>> Cleaning data folder which might have some garbage...
psql: could not connect to server: Connection refused
Is the server running on host "cyclos-postgres-node2-service" (10.244.0.9) and accepting
TCP/IP connections on port 5432?
>>> Auto-detected master name: 'cyclos-postgres-node1-service'
>>> Setting up repmgr...
>>> Setting up repmgr config file '/etc/repmgr.conf'...
>>> Setting up upstream node...
cat: /var/lib/postgresql/data/standby.lock: No such file or directory
>>> Previously Locked standby upstream node LOCKED_STANDBY=''
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
>>> REPLICATION_UPSTREAM_NODE_ID=1
>>> Sending in background postgres start...
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
>>> Starting standby node...
>>> Instance hasn't been set up yet.
>>> Clonning primary node...
>>> Waiting for upstream postgres server...
>>> Wait db replica_db on cyclos-postgres-node1-service:5432(user: replica_user,password: *******), will try 30 times with delay 10 seconds (TIMEOUT=300)
NOTICE: destination directory '/var/lib/postgresql/data' provided
INFO: connecting to upstream node
>>>>>> Db replica_db exists on cyclos-postgres-node1-service:5432!
INFO: Successfully connected to upstream node. Current installation size is 34 MB
INFO: checking and correcting permissions on existing directory /var/lib/postgresql/data ...
NOTICE: starting backup (using pg_basebackup)...
INFO: executing: '/usr/lib/postgresql/9.5/bin/pg_basebackup -l "repmgr base backup" -D /var/lib/postgresql/data -h cyclos-postgres-node1-service -p 5432 -U replica_user -c fast -X stream '
NOTICE: standby clone (using pg_basebackup) complete
NOTICE: you can now start your PostgreSQL server
HINT: for example : pg_ctl -D /var/lib/postgresql/data start
HINT: After starting the server, you need to register this standby with "repmgr standby register"
[REPMGR EVENT] Node id: 2; Event type: standby_clone; Success [1|0]: 1; Time: 2018-02-02 06:50:47.340146+00; Details: Cloned from host 'cyclos-postgres-node1-service', port 5432; backup method: pg_basebackup; --force: Y
>>> Configuring /var/lib/postgresql/data/postgresql.conf
>>>>>> Will add configs to exists file
>>> Starting postgres...
>>> Waiting for local postgres server start...
>>> Wait db replica_db on cyclos-postgres-node2-service:5432(user: replica_user,password: *******), will try 60 times with delay 10 seconds (TIMEOUT=600)
LOG: incomplete startup packet
LOG: database system was interrupted; last known up at 2018-02-02 06:50:46 UTC
LOG: incomplete startup packet
FATAL: the database system is starting up
psql: FATAL: the database system is starting up
>>>>>> Db replica_db is still not accessable on cyclos-postgres-node2-service:5432 (will try 60 times more)
LOG: entering standby mode
LOG: redo starts at 0/2000028
LOG: consistent recovery state reached at 0/2000130
LOG: database system is ready to accept read only connections
LOG: started streaming WAL from primary at 0/3000000 on timeline 1
>>>>>> Db replica_db exists on cyclos-postgres-node2-service:5432!
>>> Waiting for replication on this node is over(if any in progress): CLEAN_UP_ON_FAIL=, INTERVAL=30
>>> Replication is done
>>> Unregister the node if it was done before
DELETE 0
>>> Registering node with role standby
INFO: connecting to standby database
INFO: connecting to master database
INFO: retrieving node list for cluster 'postgres_cluster'
INFO: registering the standby
[REPMGR EVENT] Node id: 2; Event type: standby_register; Success [1|0]: 1; Time: 2018-02-02 06:51:05.083455+00; Details:
INFO: standby registration complete
NOTICE: standby node correctly registered for cluster postgres_cluster with id 2 (conninfo: user=replica_user password=replica_pass host=cyclos-postgres-node2-service dbname=replica_db port=5432 connect_timeout=2)
Locking standby (NEW_UPSTREAM_NODE_ID=1)...
>>> Starting repmgr daemon...
[2018-02-02 06:51:05] [NOTICE] looking for configuration file in current directory
[2018-02-02 06:51:05] [NOTICE] looking for configuration file in /etc
[2018-02-02 06:51:05] [NOTICE] configuration file found at: /etc/repmgr.conf
[2018-02-02 06:51:05] [INFO] connecting to database 'user=replica_user password=replica_pass host=cyclos-postgres-node2-service dbname=replica_db port=5432 connect_timeout=2'
[2018-02-02 06:51:06] [INFO] connected to database, checking its state
[2018-02-02 06:51:06] [INFO] connecting to master node of cluster 'postgres_cluster'
[2018-02-02 06:51:06] [INFO] retrieving node list for cluster 'postgres_cluster'
[2018-02-02 06:51:06] [INFO] checking role of cluster node '1'
[2018-02-02 06:51:06] [INFO] checking cluster configuration with schema 'repmgr_postgres_cluster'
[2018-02-02 06:51:06] [INFO] checking node 2 in cluster 'postgres_cluster'
[2018-02-02 06:51:06] [INFO] reloading configuration file
[2018-02-02 06:51:06] [INFO] configuration has not changed
[2018-02-02 06:51:06] [INFO] starting continuous standby node monitoring
ERROR: cannot execute DELETE in a read-only transaction
STATEMENT: DELETE FROM repmgr_postgres_cluster.repl_nodes WHERE conninfo LIKE '%host=cyclos-postgres-node3-service%'
I was able to run this on a paid cluster in IBM Cloud and it appears to be working. I did NOT use the persistent volumes and I was on a paid cluster. Please note that persistent volumes are not available on free clusters, so if you are testing on a free cluster you will get issues if you use persistent volumes.
My cluster has 3 workers of size u2c.2x4 (the smallest available) and is on the default version of Kubernetes for IBM Cloud (1.8.6), if that helps you debug at all. Please try again or if your setup is different than mine, let me know and I can try with a matching setup.
$ kubectl logs --namespace=mysystem mysystem-db-node1-0
>>> Setting up STOP handlers...
>>> STARTING SSH (if required)...
>>> SSH is not enabled!
>>> STARTING POSTGRES...
>>> TUNING UP POSTGRES...
>>> Cleaning data folder which might have some garbage...
psql: could not translate host name "mysystem-db-node1-service" to address: Name or service not known
psql: could not translate host name "mysystem-db-node2-service" to address: Name or service not known
>>> Auto-detected master name: ''
>>> Setting up repmgr...
>>> Setting up repmgr config file '/etc/repmgr.conf'...
>>> Setting up upstream node...
>>> Sending in background postgres start...
>>> Waiting for local postgres server start...
>>> Wait db replica_db on mysystem-db-node1-service:5432(user: replica_user,password: *******), will try 60 times with delay 10 seconds (TIMEOUT=600)
psql: could not translate host name "mysystem-db-node3-service" to address: Name or service not known
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
psql: could not connect to server: Connection refused
Is the server running on host "mysystem-db-node1-service" (172.30.207.54) and accepting
TCP/IP connections on port 5432?
selecting default shared_buffers ... >>>>>> Db replica_db is still not accessable on mysystem-db-node1-service:5432 (will try 60 times more)
128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
WARNING: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....LOG: could not bind IPv6 socket: Cannot assign requested address
HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
LOG: database system was shut down at 2018-02-14 15:40:14 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
done
server started
CREATE DATABASE
CREATE ROLE
/docker-entrypoint.sh: running /docker-entrypoint-initdb.d/entrypoint.sh
>>> Configuring /var/lib/postgresql/data/postgresql.conf
>>>>>> Config file was replaced with standard one!
>>>>>> Adding config 'wal_keep_segments'='250'
>>>>>> Adding config 'shared_buffers'='300MB'
>>>>>> Adding config 'archive_command'=''/bin/true''
>>> Creating replication user 'replica_user'
CREATE ROLE
>>> Creating replication db 'replica_db'
LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: autovacuum launcher shutting down
waiting for server to shut down....LOG: shutting down
LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
LOG: database system was shut down at 2018-02-14 15:40:16 UTC
LOG: MultiXact member wraparound protections are now enabled
LOG: database system is ready to accept connections
LOG: autovacuum launcher started
LOG: incomplete startup packet
LOG: incomplete startup packet
>>>>>> Db replica_db exists on mysystem-db-node1-service:5432!
>>> Registering node with role master
INFO: connecting to master database
INFO: master register: creating database objects inside the 'repmgr_mysystem_cluster' schema
INFO: retrieving node list for cluster 'mysystem_cluster'
[REPMGR EVENT] Node id: 1; Event type: master_register; Success [1|0]: 1; Time: 2018-02-14 15:40:27.337393+00; Details:
[REPMGR EVENT] will execute script '/usr/local/bin/cluster/repmgr/events/execs/master_register.sh' for the event
[REPMGR EVENT::master_register] Node id: 1; Event type: master_register; Success [1|0]: 1; Time: 2018-02-14 15:40:27.337393+00; Details:
[REPMGR EVENT::master_register] Locking master...
[REPMGR EVENT::master_register] Unlocking standby...
NOTICE: master node correctly registered for cluster 'mysystem_cluster' with id 1 (conninfo: user=replica_user password=replica_pass host=mysystem-db-node1-service dbname=replica_db port=5432 connect_timeout=2)
>>> Starting repmgr daemon...
[2018-02-14 15:40:27] [NOTICE] looking for configuration file in current directory
[2018-02-14 15:40:27] [NOTICE] looking for configuration file in /etc
[2018-02-14 15:40:27] [NOTICE] configuration file found at: /etc/repmgr.conf
[2018-02-14 15:40:27] [INFO] connecting to database 'user=replica_user password=replica_pass host=mysystem-db-node1-service dbname=replica_db port=5432 connect_timeout=2'
[2018-02-14 15:40:27] [INFO] connected to database, checking its state
[2018-02-14 15:40:27] [INFO] checking cluster configuration with schema 'repmgr_mysystem_cluster'
[2018-02-14 15:40:27] [INFO] checking node 1 in cluster 'mysystem_cluster'
[2018-02-14 15:40:27] [INFO] reloading configuration file
[2018-02-14 15:40:27] [INFO] configuration has not changed
[2018-02-14 15:40:27] [INFO] starting continuous master connection check
We have implemented MongoDB master-slave replication and database got deleted.
We restored deleted files from Mongo data directory using extundelete command.
Undelete Blog https://github.com/RIKSOF/development/wiki/Restoring-deleted-files-on-Linux
We wanted to restore at least one collection for which we got the WT (wired timer) file. We are trying to restore that file using the following commands and we are getting the following errors
Recovering a WiredTiger collection from a corrupt MongoDB installation
wget http://source.wiredtiger.com/releases/wiredtiger-2.7.0.tar.bz2
tar xvf wiredtiger-2.7.0.tar.bz2
cd wiredtiger-2.7.0
sudo apt-get install libsnappy-dev build-essential
./configure --enable-snappy
make
./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2657--1723320556100349955.wt
Error -
./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-246--7553069514495955510.wt
[1488888117:36780][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 12288: block header checksum of 1955562709 doesn't match expected checksum of 3146787951
[1488888117:36809][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value
[1488888117:36817][13536:0x7f7b633cd740], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic
lt-wt: WT_PANIC: WiredTiger library panic
This could be related to corrupted .wt files (e.g. WiredTiger.wt/WiredTiger.turtle) as per SERVER-31076 bug report.
Try to run repair on all databases by the following command:
mongod --repair --dbpath /path/to/data/db
Also make sure all data/db files have the right read and write permission.
In my case, I had recovered mac from Time Machine. I had mongodb installed via brew install. When I started mongo, the service did not start cleanly.
vi /usr/local/var/log/mongodb/mongo.log
enter code here
2020-01-30T17:06:10.655-0500 E STORAGE [initandlisten] WiredTiger error (0) [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12 Raw: [1580421970:655376][62181:0x112233dc0], file:WiredTiger.wt, connection: __wt_block_read_off, 283: WiredTiger.wt: read checksum error for 4096B block at offset 12288: block header checksum of 0x325e318 doesn't match expected checksum of 0x65ef0e12
What I saw was that because of a corrupt mongodb collection.
So, I ran the cleanup command for the brew install, which has the db in a different location.
mongod --repair --dbpath /usr/local/var/mongodb
2020-01-30T17:06:11.741-0500 I STORAGE [initandlisten] Verify failed on uri table:collection-12--8361386101775862971. Running a salvage operation.
2020-01-30T17:06:11.745-0500 I STORAGE [initandlisten] Repairing collection wlchat.jobs
2020-01-30T17:06:11.765-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 100
2020-01-30T17:06:11.769-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 200
2020-01-30T17:06:11.774-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 300
2020-01-30T17:06:11.783-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 400
2020-01-30T17:06:11.789-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 500
2020-01-30T17:06:11.793-0500 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.verify 600
2020-01-30T17:06:11.795-0500 I STORAGE [initandlisten] Verify succeeded on uri table:collection-10--8361386101775862971. Not salvaging.
2020-01-30T17:06:11.795-0500 I STORAGE [initandlisten] Repairing collection wlchat.SovrenJobs
2020-01-30T17:06:11.801-0500 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0-8794251073515214768. Not salvaging.
2020-01-30T17:06:11.814-0500 I INDEX [initandlisten] index build: starting on wlchat.users properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "wlchat.users" } using method: Foreground
and restart mongodb-community
brew services restart mongodb-community.
==> Successfully started `mongodb-community` (label: homebrew.mxcl.mongodb-community)
Things are back to normal.
I got this error and trying a lot of way to resolve but not reached to successfully.
I am currently using 5.0.1 mongodb standalone version,
removed currently installed mongodb ( 5.0.1 )
installed latest version ( 6.0 )
and reapaired all corrupted *.wt file and verified.
mongod --repair --dbpath /path/to/data/db
after upgrading a node in cassandra, these error in log occured:
I want to investigate but dont have a direction, any clue will help
thanks
2016/09/19 06:24:49 [INFO] core: post-unseal setup starting
2016/09/19 06:24:49 [INFO] core: mounted backend of type generic at secret/
2016/09/19 06:24:49 [INFO] core: mounted backend of type cubbyhole at cubbyhole/
2016/09/19 06:24:49 [INFO] core: mounted backend of type system at sys/
2016/09/19 06:24:49 [INFO] core: mounted backend of type cassandra at cassandra/
2016/09/19 06:24:49 [INFO] rollback: starting rollback manager
2016/09/19 06:24:50 [INFO] expire: restored 2 leases
2016/09/19 06:24:50 [INFO] core: post-unseal setup complete
2016/09/19 06:24:55 gocql: unable to dial control conn node-0.cassandra-app.mesos:9042: dial tcp 10.0.2.42:9042: getsockopt: connection refused
2016/09/19 06:25:12 error: failed to connect to 10.0.2.42:9042 due to error: gocql: no response to connection startup within timeout
Cassandra cluster is not cross-version compatible, you can not upgrade a node only, you have to upgrade the cluster. This is a common mistake people tend to do, please see this video here it mentiones this problem, also it is very very useful with lots of good info.
I have the latest Amazon Linux AMI running on a medium instance with 4gb of Ram.
LAMP server is setup with a single Wordpress instance. I am having memory issues that I can't seem to pinpoint my issue. I am new to linux, and am unsure of how to pinpoint our issue.
Apache will report:
Sat Apr 18 04:27:36 2015 error (12)Cannot allocate memory: fork: Unable to fork new process
MYSQL will report:
150418 4:28:24 InnoDB: The InnoDB memory heap is disabled
150418 4:28:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
Version: '5.5.42' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
150418 04:28:24 mysqld_safe Number of processes running now: 0
150418 04:28:24 mysqld_safe mysqld restarted
150418 4:28:24 Note Plugin 'FEDERATED' is disabled.
150418 4:28:24 InnoDB: The InnoDB memory heap is disabled
150418 4:28:24 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150418 4:28:24 InnoDB: Compressed tables use zlib 1.2.7
150418 4:28:24 InnoDB: Using Linux native AIO
150418 4:28:24 InnoDB: Initializing buffer pool, size = 64.0M
InnoDB: mmap(68681728 bytes) failed; errno 12
150418 4:28:24 InnoDB: Completed initialization of buffer pool
150418 4:28:24 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150418 4:28:24 ERROR Plugin 'InnoDB' init function returned error.
150418 4:28:24 ERROR Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150418 4:28:24 ERROR Unknown/unsupported storage engine: InnoDB
150418 4:28:24 ERROR Aborting
150418 4:28:24 Note /usr/libexec/mysqld: Shutdown complete
150418 04:28:24 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
MYSQL Conf:
mysqld
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
Settings user and group are ignored when systemd is used.
If you need to run mysqld under a different user or group,
customize your systemd unit file for mysqld according to the
instructions in http://fedoraproject.org/wiki/Systemd
query_cache_size=8M
tmp_table_size=16M
max_heap_table_size=16M
thread_cache_size=4
innodb_buffer_pool_size=64M
mysqld_safe
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
HTTPD Conf: (Prefork)
prefork MPM
StartServers: number of server processes to start
MinSpareServers: minimum number of server processes which are kept spare
MaxSpareServers: maximum number of server processes which are kept spare
ServerLimit: maximum value for MaxClients for the lifetime of the server
MaxClients: maximum number of server processes allowed to start
MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 15
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 3000
</IfModule>