E 2017-08-07 21:50:36 0x7fc8cc108700 InnoDB: Assertion failure in thread 140500393821952 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 21:50:36 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=5
E max_threads=4000
E thread_count=3
E connection_count=3
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x55607658e000
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7fc8cc103e40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E New value of fp=0 failed sanity check, terminating stack trace!
E mysqld(handle_fatal_signal+0x390) [0x556064b51050]
E libc.so.6(gsignal+0x3b) [0x7fc95aa6e2fb]
E libc.so.6(abort+0x14f) [0x7fc95aa7000f]
E mysqld(+0x702791) [0x556064b14791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x5560652dcb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x5560654233e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x55606542354c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x55606542370d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x55606532f4a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x55606532fae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x55606535e569]
E mysqld(_Z18row_purge_poss_secP12purge_node_tP12dict_index_tPK8dtuple_t+0x229) [0x556065390909]
E mysqld(+0xf7ebdf) [0x556065390bdf]
E mysqld(_Z14row_purge_stepP9que_thr_t+0x6ba) [0x5560653926ba]
E mysqld(_Z15que_run_threadsP9que_thr_t+0x998) [0x5560653e0a58]
E mysqld(_Z9trx_purgemmb+0x10f8) [0x55606533e4f8]
E mysqld(srv_purge_coordinator_thread+0x1c9) [0x55606535aa99]
E libpthread.so.0(+0x6890) [0x7fc95b303890]
E libc.so.6(clone+0x6d) [0x7fc95aa572ed]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
I [mysqld]
I allow_client_hostname_forwarding=ON
I binlog-format=ROW
I block_user_access=33280 # FILE_ACL | SUPER_ACL
I character_set_server=utf8
I datadir=/mysql/datadir
I general_log_file=/mysql/logs/general.log
I hostname=localhost
I innodb_buffer_pool_dump_at_shutdown=ON
I innodb_buffer_pool_load_at_startup=ON
I innodb_file_format=Barracuda
I innodb_file_per_table=ON
I innodb_flush_method=O_DIRECT
I innodb_flush_neighbors=2
I innodb_flushing_avg_loops=5
I innodb_io_capacity=5000
I innodb_log_file_size=512M
I innodb_lru_scan_depth=2048
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I force_ssl_for_tcp=X509
I server-id=4078416117
I innodb_buffer_pool_size=1379874816
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-07T21:50:39.963095Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-07T21:50:39.963132Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-07T21:50:39.963154Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-07T21:50:39.970731Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-07T21:50:39.970752Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-07T21:50:39.971078Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-07T21:50:39.971102Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-07T21:50:39.971107Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-07T21:50:39.971109Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-07T21:50:39.971111Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-07T21:50:39.971113Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-07T21:50:39.971115Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-07T21:50:39.976515Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-07T21:50:39.976900Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-07T21:50:39.978638Z 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
E 2017-08-07T21:50:40.111401Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-07T21:50:40.162065Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-07T21:50:40.186160Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-07T21:50:40.192989Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6675084209
E 2017-08-07T21:50:40.193686Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6675088158
E 2017-08-07T21:50:40.200272Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6675088158
E 2017-08-07T21:50:40.201148Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-07T21:50:40.201572Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-07T21:50:40.451240Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-07T21:50:40.981117Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-07T21:50:40.981752Z 0 [Note] InnoDB: Last MySQL binlog file position 0 19400, file name mysql-bin.001294
E 2017-08-07T21:50:41.091670Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-07T21:50:41.092443Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-07T21:50:41.092934Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-07T21:50:41.274501Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-07T21:50:41.276009Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-07T21:50:41.276468Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-07T21:50:41.277251Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-07T21:50:41.327848Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6675088158
E 2017-08-07T21:50:41.329350Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-07T21:50:41.331629Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-07T21:50:41.332219Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-07T21:50:41.383054Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-07T21:50:41.384038Z 0 [Note] Starting crash recovery...
E 2017-08-07T21:50:41.384478Z 0 [Note] Crash recovery finished.
E 2017-08-07T21:50:41.384971Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.001503' size: 18873
E 2017-08-07T21:50:41.386207Z 0 [Note] Starting ack receiver thread
E 2017-08-07T21:50:41.397658Z 0 [Note] Found gtids in binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-4014510'
E 2017-08-07T21:50:41.398735Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3149075'
E 2017-08-07T21:50:41.404845Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-07T21:50:41.406127Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-07T21:50:41.406641Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-07T21:50:41.407577Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-07T21:50:41.408048Z 0 [Note] IPv6 is available.
E 2017-08-07T21:50:41.408512Z 0 [Note] - '::' resolves to '::';
E 2017-08-07T21:50:41.408975Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-07T21:50:41.412444Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.412947Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.413372Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-07T21:50:41.413868Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.414334Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.414965Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.415389Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.415873Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.430960Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.436537Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-07T21:50:41.437253Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-07T21:50:41.455411Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170807 21:50:41
E 2017-08-07T21:50:45.264838Z 3 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.001504, pos: 561), semi-sync up to file , position 0.
E 2017-08-07T21:50:45.264877Z 3 [Note] Semi-sync replication switched OFF.
E 2017-08-07T21:51:21.932513Z 8 [Note] Aborted connection 8 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:21.933091Z 2 [Note] Aborted connection 2 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.336547Z 22 [Note] Aborted connection 22 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.337185Z 21 [Note] Aborted connection 21 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.550910Z 0 [Note] Giving 1 client threads a chance to die gracefully
E 2017-08-07T21:51:22.550933Z 0 [Note] Shutting down slave threads
E 2017-08-07T21:51:24.551046Z 0 [Note] Forcefully disconnecting 1 remaining clients
E 2017-08-07T21:51:24.551081Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 3 user: 'root'
E 2017-08-07T21:51:24.551098Z 0 [Note] Event Scheduler: Purging the queue. 0 events
E 2017-08-07T21:51:24.552286Z 0 [Note] Binlog end
E 2017-08-07T21:51:24.555274Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
E 2017-08-07T21:51:24.555577Z 0 [Note] Shutting down plugin 'partition'
E 2017-08-07T21:51:24.555836Z 0 [Note] Shutting down plugin 'ngram'
E 2017-08-07T21:51:24.556081Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
E 2017-08-07T21:51:24.556089Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
E 2017-08-07T21:51:24.556091Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
E 2017-08-07T21:51:24.556093Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
E 2017-08-07T21:51:24.556094Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
E 2017-08-07T21:51:24.556095Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
E 2017-08-07T21:51:24.556097Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
E 2017-08-07T21:51:24.556099Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
E 2017-08-07T21:51:24.556100Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
E 2017-08-07T21:51:24.556102Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
E 2017-08-07T21:51:24.556103Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
E 2017-08-07T21:51:24.556105Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
E 2017-08-07T21:51:24.556106Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
E 2017-08-07T21:51:24.556107Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
E 2017-08-07T21:51:24.556112Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
E 2017-08-07T21:51:24.556113Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
E 2017-08-07T21:51:24.556114Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
E 2017-08-07T21:51:24.556116Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
E 2017-08-07T21:51:24.556117Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
E 2017-08-07T21:51:24.556118Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
E 2017-08-07T21:51:24.556120Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
E 2017-08-07T21:51:24.556121Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
E 2017-08-07T21:51:24.556123Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
E 2017-08-07T21:51:24.556124Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
E 2017-08-07T21:51:24.556126Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
E 2017-08-07T21:51:24.556127Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
E 2017-08-07T21:51:24.556131Z 0 [Note] Shutting down plugin 'INNODB_CMP'
E 2017-08-07T21:51:24.556133Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
E 2017-08-07T21:51:24.556134Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
E 2017-08-07T21:51:24.556136Z 0 [Note] Shutting down plugin 'INNODB_TRX'
E 2017-08-07T21:51:24.556137Z 0 [Note] Shutting down plugin 'InnoDB'
E 2017-08-07T21:51:24.556557Z 0 [Note] InnoDB: FTS optimize thread exiting.
E 2017-08-07T21:51:24.556652Z 0 [Note] InnoDB: Starting shutdown...
E 2017-08-07T21:51:24.656885Z 0 [Note] InnoDB: Dumping buffer pool(s) to /mysql/datadir/ib_buffer_pool
E 2017-08-07T21:51:24.657132Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 170807 21:51:24
E 2017-08-07T21:51:41.305784Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709546192 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
E 2017-08-07 21:51:41 0x7f887ea72700 InnoDB: Assertion failure in thread 140224217163520 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 21:51:41 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=5
E max_threads=4000
E thread_count=0
E connection_count=0
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x564fa6786000
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7f887ea6de40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E Stack trace seems successful - bottom reached
E mysqld(handle_fatal_signal+0x390) [0x564f93f21050]
E libc.so.6(gsignal+0x3b) [0x7f890d3d82fb]
E libc.so.6(abort+0x14f) [0x7f890d3da00f]
E mysqld(+0x702791) [0x564f93ee4791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x564f946acb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x564f947f33e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x564f947f354c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x564f947f370d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x564f946ff4a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x564f946ffae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x564f9472e569]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I enforce_storage_engine=InnoDB
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I server-id=4292715972
I innodb_buffer_pool_size=4217853542
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-08T14:41:10.903955Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-08T14:41:10.903993Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-08T14:41:10.904012Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-08T14:41:10.908514Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-08T14:41:10.908546Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-08T14:41:10.909011Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-08T14:41:10.909047Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-08T14:41:10.909058Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-08T14:41:10.909066Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-08T14:41:10.909072Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-08T14:41:10.909093Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-08T14:41:10.909100Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-08T14:41:10.912950Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-08T14:41:10.913111Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-08T14:41:10.914564Z 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
E 2017-08-08T14:41:11.136802Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-08T14:41:11.196973Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-08T14:41:11.223178Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-08T14:41:11.234344Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6672642734
E 2017-08-08T14:41:11.234388Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672646997
E 2017-08-08T14:41:11.247010Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672646997
E 2017-08-08T14:41:11.247148Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-08T14:41:11.247156Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-08T14:41:11.535829Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-08T14:41:12.129444Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-08T14:41:12.129482Z 0 [Note] InnoDB: Last MySQL binlog file position 0 70560779, file name mysql-bin.000132
E 2017-08-08T14:41:12.240364Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-08T14:41:12.240384Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-08T14:41:12.240417Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-08T14:41:12.345314Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-08T14:41:12.346271Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-08T14:41:12.346282Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-08T14:41:12.346522Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-08T14:41:12.397948Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6672646997
E 2017-08-08T14:41:12.398526Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-08T14:41:12.401353Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-08T14:41:12.401372Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-08T14:41:12.483339Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-08T14:41:12.483373Z 0 [Note] Starting ack receiver thread
E 2017-08-08T14:41:12.483474Z 0 [Note] Starting crash recovery...
E 2017-08-08T14:41:12.483492Z 0 [Note] Crash recovery finished.
E 2017-08-08T14:41:12.483550Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.000158' size: 18586
E 2017-08-08T14:41:12.493085Z 0 [Note] Found gtids in binlog: '052ac796-7c44-11e7-93db-42010a84007d:1-1320,
E 542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3684139'
E 2017-08-08T14:41:12.493100Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-2850351'
E 2017-08-08T14:41:12.500533Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170808 14:41:12
E 2017-08-08T14:41:12.500710Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-08T14:41:12.501884Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-08T14:41:12.501927Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-08T14:41:12.502966Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-08T14:41:12.503008Z 0 [Note] IPv6 is available.
E 2017-08-08T14:41:12.503014Z 0 [Note] - '::' resolves to '::';
E 2017-08-08T14:41:12.503020Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-08T14:41:12.507540Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507570Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507576Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-08T14:41:12.507597Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507619Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508335Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508352Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508354Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.524474Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.540810Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-08T14:41:12.541274Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-08T14:41:15.714525Z 2 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.000159, pos: 601), semi-sync up to file , position 0.
E 2017-08-08T14:41:15.714567Z 2 [Note] Semi-sync replication switched OFF.
E 2017-08-08T14:42:12.360008Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709541672 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
E 2017-08-08 14:42:12 0x7ff6dbeee700 InnoDB: Assertion failure in thread 140698228549376 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 14:42:12 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=4
E max_threads=4000
E thread_count=3
E connection_count=3
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x55d5efe40800
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7ff6dbee9e40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E New value of fp=0 failed sanity check, terminating stack trace!
E mysqld(handle_fatal_signal+0x390) [0x55d5d3c94050]
E libc.so.6(gsignal+0x3b) [0x7ff7ed9542fb]
E libc.so.6(abort+0x14f) [0x7ff7ed95600f]
E mysqld(+0x702791) [0x55d5d3c57791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x55d5d441fb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x55d5d45663e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x55d5d456654c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x55d5d456670d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x55d5d44724a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x55d5d4472ae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x55d5d44a1569]
E mysqld(_Z18row_purge_poss_secP12purge_node_tP12dict_index_tPK8dtuple_t+0x229) [0x55d5d44d3909]
E mysqld(+0xf7ebdf) [0x55d5d44d3bdf]
E mysqld(_Z14row_purge_stepP9que_thr_t+0x6ba) [0x55d5d44d56ba]
E mysqld(_Z15que_run_threadsP9que_thr_t+0x998) [0x55d5d4523a58]
E mysqld(_Z9trx_purgemmb+0x10f8) [0x55d5d44814f8]
E mysqld(srv_purge_coordinator_thread+0x1c9) [0x55d5d449da99]
E libpthread.so.0(+0x6890) [0x7ff7ee1e9890]
E libc.so.6(clone+0x6d) [0x7ff7ed93d2ed]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
I [mysqld]
I allow_client_hostname_forwarding=ON
I binlog-format=ROW
I block_user_access=33280 # FILE_ACL | SUPER_ACL
I character_set_server=utf8
I datadir=/mysql/datadir
I general_log_file=/mysql/logs/general.log
I hostname=localhost
I innodb_buffer_pool_dump_at_shutdown=ON
I innodb_buffer_pool_load_at_startup=ON
I innodb_file_format=Barracuda
I innodb_file_per_table=ON
I innodb_flush_method=O_DIRECT
I innodb_flush_neighbors=2
I innodb_flushing_avg_loops=5
I innodb_io_capacity=5000
I innodb_log_file_size=512M
I innodb_lru_scan_depth=2048
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I enforce_storage_engine=InnoDB
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I server-id=4292715972
I innodb_buffer_pool_size=4217853542
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-08T14:42:15.633907Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-08T14:42:15.633944Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-08T14:42:15.633962Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-08T14:42:15.637549Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-08T14:42:15.637572Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-08T14:42:15.637868Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-08T14:42:15.637892Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-08T14:42:15.637897Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-08T14:42:15.637899Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-08T14:42:15.637901Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-08T14:42:15.637903Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-08T14:42:15.637906Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-08T14:42:15.641182Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-08T14:42:15.641289Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-08T14:42:15.642723Z 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
E 2017-08-08T14:42:15.854815Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-08T14:42:15.912061Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-08T14:42:15.936072Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-08T14:42:15.946772Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6672642734
E 2017-08-08T14:42:15.946814Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672647605
E 2017-08-08T14:42:15.958760Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672647605
E 2017-08-08T14:42:15.958901Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-08T14:42:15.958908Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-08T14:42:16.188562Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-08T14:42:16.778561Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-08T14:42:16.778601Z 0 [Note] InnoDB: Last MySQL binlog file position 0 70560779, file name mysql-bin.000132
E 2017-08-08T14:42:16.888214Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-08T14:42:16.888239Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-08T14:42:16.888379Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-08T14:42:16.989889Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-08T14:42:16.990765Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-08T14:42:16.990777Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-08T14:42:16.991037Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-08T14:42:17.042278Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6672647605
E 2017-08-08T14:42:17.043013Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-08T14:42:17.045458Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-08T14:42:17.045472Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-08T14:42:17.134897Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170808 14:42:17
E 2017-08-08T14:42:17.135878Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-08T14:42:17.136199Z 0 [Note] Starting crash recovery...
E 2017-08-08T14:42:17.136251Z 0 [Note] Crash recovery finished.
E 2017-08-08T14:42:17.136341Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.000159' size: 18913
E 2017-08-08T14:42:17.136954Z 0 [Note] Starting ack receiver thread
E 2017-08-08T14:42:17.143780Z 0 [Note] Found gtids in binlog: '052ac796-7c44-11e7-93db-42010a84007d:1-1377,
E 542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3684139'
E 2017-08-08T14:42:17.143803Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-2850351'
E 2017-08-08T14:42:17.148899Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-08T14:42:17.149474Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-08T14:42:17.149527Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-08T14:42:17.149985Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-08T14:42:17.150022Z 0 [Note] IPv6 is available.
E 2017-08-08T14:42:17.150026Z 0 [Note] - '::' resolves to '::';
E 2017-08-08T14:42:17.150030Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-08T14:42:17.152789Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152824Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152837Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-08T14:42:17.152886Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152911Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152984Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152994Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152996Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.166564Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.180096Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-08T14:42:17.180265Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-08T14:42:20.980545Z 3 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.000160, pos: 601), semi-sync up to file , position 0.
E 2017-08-08T14:42:20.980588Z 3 [Note] Semi-sync replication switched OFF.
Hi!
-- On Monday (170807) I noticed that the connection from our services to our Cloud SQL instance (2nd generation) started to get shakey. Today (Tuesday 170808) it totally broke down on us.
If I check under the "Operations" tab, I can see that an backup failed with "An unknown error occurred":
and now the entire instance is locked (pretty much everything greyed out) and it says it is "Under maintenance":
Checking the error logs for the instance, I found this:
ERROR] InnoDB: Cannot allocate 18446744073709546192 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
(For the complete log, please see the attached file mysql_err_log_1.txt)
This made me find this link: https://bugs.mysql.com/bug.php?id=70185 where the user seems to have nearly exactly the same problem as we're experiencing. He also suggests two solutions:
1) add 'innodb_force_recovery=4' to [mysqld] in my.cnf or.... 2) set 'innodb_change_buffering=none' in my.cnf
But I am however not able to test them since gcloud won't allow me to neither access the cnf file, nor set the flags using gcloud:
gcloud sql instances patch [INSTANCE-NAME] --database-flags innodb_force_recovery=4
The following message will be used for the patch API method.
{"project": [PROJECT-NAME], "name": [INSTANCE-NAME], "settings": {"databaseFlags": [{"name": "innodb_force_recovery", "value": "4"}]}}
WARNING: This patch modifies a value that requires your instance to be
restarted. Submitting this patch will immediately restart your
instance if it's running.
Do you want to continue (Y/n)? Y
ERROR: (gcloud.sql.instances.patch) HTTPError 404: Flag requested cannot be set.
I've also tried to create two new instances: One exactly the same as the original one, and one with double the resources (vCPU, RAM and SSD) just in case we'd hit the resource ceiling, and then restore these new instances to a previous version of the now failed instance using the backups. I've tried two different backup dates: The first one one day before this problem was noticed (170806) and one a couple of days before (170804), but both of the new instances start immediately to experience the same issues (Please see the logs from one of the new instances in attached file mysql_err_log_2.txt)
I've also tried to do an export using one of the new instances, via the Cloud Web UI, but that results in this error: mysqldump: Couldn't execute 'show table status like 'email\_meta'': Lost connection to MySQL server during query (2013).
All our services that tires to access any of the new instances get one of the following errors:
(2013, 'Lost connection to MySQL server during query')
(2006, 'MySQL server has gone away')
I'm also not able to do an export using a mysql client application. I've tried with Sequel Pro (for mac) without luck, I just get an error saying the connection failed.
What does this log mean? What can I do to check my tables in case any of them are corrupted etc?
Any kind of help and suggestion is highly appreciated!
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/c80901c6-2d2f-42c4-b948-c80087fcb986%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
No comments:
Post a Comment