E 2017-08-07 21:50:36 0x7fc8cc108700 InnoDB: Assertion failure in thread 140500393821952 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 21:50:36 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=5
E max_threads=4000
E thread_count=3
E connection_count=3
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x55607658e000
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7fc8cc103e40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E New value of fp=0 failed sanity check, terminating stack trace!
E mysqld(handle_fatal_signal+0x390) [0x556064b51050]
E libc.so.6(gsignal+0x3b) [0x7fc95aa6e2fb]
E libc.so.6(abort+0x14f) [0x7fc95aa7000f]
E mysqld(+0x702791) [0x556064b14791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x5560652dcb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x5560654233e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x55606542354c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x55606542370d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x55606532f4a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x55606532fae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x55606535e569]
E mysqld(_Z18row_purge_poss_secP12purge_node_tP12dict_index_tPK8dtuple_t+0x229) [0x556065390909]
E mysqld(+0xf7ebdf) [0x556065390bdf]
E mysqld(_Z14row_purge_stepP9que_thr_t+0x6ba) [0x5560653926ba]
E mysqld(_Z15que_run_threadsP9que_thr_t+0x998) [0x5560653e0a58]
E mysqld(_Z9trx_purgemmb+0x10f8) [0x55606533e4f8]
E mysqld(srv_purge_coordinator_thread+0x1c9) [0x55606535aa99]
E libpthread.so.0(+0x6890) [0x7fc95b303890]
E libc.so.6(clone+0x6d) [0x7fc95aa572ed]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
I [mysqld]
I allow_client_hostname_forwarding=ON
I binlog-format=ROW
I block_user_access=33280 # FILE_ACL | SUPER_ACL
I character_set_server=utf8
I datadir=/mysql/datadir
I general_log_file=/mysql/logs/general.log
I hostname=localhost
I innodb_buffer_pool_dump_at_shutdown=ON
I innodb_buffer_pool_load_at_startup=ON
I innodb_file_format=Barracuda
I innodb_file_per_table=ON
I innodb_flush_method=O_DIRECT
I innodb_flush_neighbors=2
I innodb_flushing_avg_loops=5
I innodb_io_capacity=5000
I innodb_log_file_size=512M
I innodb_lru_scan_depth=2048
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I force_ssl_for_tcp=X509
I server-id=4078416117
I innodb_buffer_pool_size=1379874816
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-07T21:50:39.963095Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-07T21:50:39.963132Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-07T21:50:39.963154Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-07T21:50:39.970731Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-07T21:50:39.970752Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-07T21:50:39.971078Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-07T21:50:39.971102Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-07T21:50:39.971107Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-07T21:50:39.971109Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-07T21:50:39.971111Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-07T21:50:39.971113Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-07T21:50:39.971115Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-07T21:50:39.976515Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-07T21:50:39.976900Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-07T21:50:39.978638Z 0 [Note] InnoDB: Initializing buffer pool, total size = 2G, instances = 8, chunk size = 128M
E 2017-08-07T21:50:40.111401Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-07T21:50:40.162065Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-07T21:50:40.186160Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-07T21:50:40.192989Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6675084209
E 2017-08-07T21:50:40.193686Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6675088158
E 2017-08-07T21:50:40.200272Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6675088158
E 2017-08-07T21:50:40.201148Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-07T21:50:40.201572Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-07T21:50:40.451240Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-07T21:50:40.981117Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-07T21:50:40.981752Z 0 [Note] InnoDB: Last MySQL binlog file position 0 19400, file name mysql-bin.001294
E 2017-08-07T21:50:41.091670Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-07T21:50:41.092443Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-07T21:50:41.092934Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-07T21:50:41.274501Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-07T21:50:41.276009Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-07T21:50:41.276468Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-07T21:50:41.277251Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-07T21:50:41.327848Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6675088158
E 2017-08-07T21:50:41.329350Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-07T21:50:41.331629Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-07T21:50:41.332219Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-07T21:50:41.383054Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-07T21:50:41.384038Z 0 [Note] Starting crash recovery...
E 2017-08-07T21:50:41.384478Z 0 [Note] Crash recovery finished.
E 2017-08-07T21:50:41.384971Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.001503' size: 18873
E 2017-08-07T21:50:41.386207Z 0 [Note] Starting ack receiver thread
E 2017-08-07T21:50:41.397658Z 0 [Note] Found gtids in binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-4014510'
E 2017-08-07T21:50:41.398735Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3149075'
E 2017-08-07T21:50:41.404845Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-07T21:50:41.406127Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-07T21:50:41.406641Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-07T21:50:41.407577Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-07T21:50:41.408048Z 0 [Note] IPv6 is available.
E 2017-08-07T21:50:41.408512Z 0 [Note] - '::' resolves to '::';
E 2017-08-07T21:50:41.408975Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-07T21:50:41.412444Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.412947Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.413372Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-07T21:50:41.413868Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.414334Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.414965Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.415389Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.415873Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.430960Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-07T21:50:41.436537Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-07T21:50:41.437253Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-07T21:50:41.455411Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170807 21:50:41
E 2017-08-07T21:50:45.264838Z 3 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.001504, pos: 561), semi-sync up to file , position 0.
E 2017-08-07T21:50:45.264877Z 3 [Note] Semi-sync replication switched OFF.
E 2017-08-07T21:51:21.932513Z 8 [Note] Aborted connection 8 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:21.933091Z 2 [Note] Aborted connection 2 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.336547Z 22 [Note] Aborted connection 22 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.337185Z 21 [Note] Aborted connection 21 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2017-08-07T21:51:22.550910Z 0 [Note] Giving 1 client threads a chance to die gracefully
E 2017-08-07T21:51:22.550933Z 0 [Note] Shutting down slave threads
E 2017-08-07T21:51:24.551046Z 0 [Note] Forcefully disconnecting 1 remaining clients
E 2017-08-07T21:51:24.551081Z 0 [Warning] /usr/sbin/mysqld: Forcing close of thread 3 user: 'root'
E 2017-08-07T21:51:24.551098Z 0 [Note] Event Scheduler: Purging the queue. 0 events
E 2017-08-07T21:51:24.552286Z 0 [Note] Binlog end
E 2017-08-07T21:51:24.555274Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
E 2017-08-07T21:51:24.555577Z 0 [Note] Shutting down plugin 'partition'
E 2017-08-07T21:51:24.555836Z 0 [Note] Shutting down plugin 'ngram'
E 2017-08-07T21:51:24.556081Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
E 2017-08-07T21:51:24.556089Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
E 2017-08-07T21:51:24.556091Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
E 2017-08-07T21:51:24.556093Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
E 2017-08-07T21:51:24.556094Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
E 2017-08-07T21:51:24.556095Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
E 2017-08-07T21:51:24.556097Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
E 2017-08-07T21:51:24.556099Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
E 2017-08-07T21:51:24.556100Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
E 2017-08-07T21:51:24.556102Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
E 2017-08-07T21:51:24.556103Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
E 2017-08-07T21:51:24.556105Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
E 2017-08-07T21:51:24.556106Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
E 2017-08-07T21:51:24.556107Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
E 2017-08-07T21:51:24.556112Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
E 2017-08-07T21:51:24.556113Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
E 2017-08-07T21:51:24.556114Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
E 2017-08-07T21:51:24.556116Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
E 2017-08-07T21:51:24.556117Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
E 2017-08-07T21:51:24.556118Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
E 2017-08-07T21:51:24.556120Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
E 2017-08-07T21:51:24.556121Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
E 2017-08-07T21:51:24.556123Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
E 2017-08-07T21:51:24.556124Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
E 2017-08-07T21:51:24.556126Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
E 2017-08-07T21:51:24.556127Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
E 2017-08-07T21:51:24.556131Z 0 [Note] Shutting down plugin 'INNODB_CMP'
E 2017-08-07T21:51:24.556133Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
E 2017-08-07T21:51:24.556134Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
E 2017-08-07T21:51:24.556136Z 0 [Note] Shutting down plugin 'INNODB_TRX'
E 2017-08-07T21:51:24.556137Z 0 [Note] Shutting down plugin 'InnoDB'
E 2017-08-07T21:51:24.556557Z 0 [Note] InnoDB: FTS optimize thread exiting.
E 2017-08-07T21:51:24.556652Z 0 [Note] InnoDB: Starting shutdown...
E 2017-08-07T21:51:24.656885Z 0 [Note] InnoDB: Dumping buffer pool(s) to /mysql/datadir/ib_buffer_pool
E 2017-08-07T21:51:24.657132Z 0 [Note] InnoDB: Buffer pool(s) dump completed at 170807 21:51:24
E 2017-08-07T21:51:41.305784Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709546192 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
E 2017-08-07 21:51:41 0x7f887ea72700 InnoDB: Assertion failure in thread 140224217163520 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 21:51:41 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=5
E max_threads=4000
E thread_count=0
E connection_count=0
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x564fa6786000
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7f887ea6de40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E Stack trace seems successful - bottom reached
E mysqld(handle_fatal_signal+0x390) [0x564f93f21050]
E libc.so.6(gsignal+0x3b) [0x7f890d3d82fb]
E libc.so.6(abort+0x14f) [0x7f890d3da00f]
E mysqld(+0x702791) [0x564f93ee4791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x564f946acb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x564f947f33e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x564f947f354c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x564f947f370d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x564f946ff4a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x564f946ffae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x564f9472e569]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I enforce_storage_engine=InnoDB
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I server-id=4292715972
I innodb_buffer_pool_size=4217853542
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-08T14:41:10.903955Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-08T14:41:10.903993Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-08T14:41:10.904012Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-08T14:41:10.908514Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-08T14:41:10.908546Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-08T14:41:10.909011Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-08T14:41:10.909047Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-08T14:41:10.909058Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-08T14:41:10.909066Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-08T14:41:10.909072Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-08T14:41:10.909093Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-08T14:41:10.909100Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-08T14:41:10.912950Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-08T14:41:10.913111Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-08T14:41:10.914564Z 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
E 2017-08-08T14:41:11.136802Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-08T14:41:11.196973Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-08T14:41:11.223178Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-08T14:41:11.234344Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6672642734
E 2017-08-08T14:41:11.234388Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672646997
E 2017-08-08T14:41:11.247010Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672646997
E 2017-08-08T14:41:11.247148Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-08T14:41:11.247156Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-08T14:41:11.535829Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-08T14:41:12.129444Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-08T14:41:12.129482Z 0 [Note] InnoDB: Last MySQL binlog file position 0 70560779, file name mysql-bin.000132
E 2017-08-08T14:41:12.240364Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-08T14:41:12.240384Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-08T14:41:12.240417Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-08T14:41:12.345314Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-08T14:41:12.346271Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-08T14:41:12.346282Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-08T14:41:12.346522Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-08T14:41:12.397948Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6672646997
E 2017-08-08T14:41:12.398526Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-08T14:41:12.401353Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-08T14:41:12.401372Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-08T14:41:12.483339Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-08T14:41:12.483373Z 0 [Note] Starting ack receiver thread
E 2017-08-08T14:41:12.483474Z 0 [Note] Starting crash recovery...
E 2017-08-08T14:41:12.483492Z 0 [Note] Crash recovery finished.
E 2017-08-08T14:41:12.483550Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.000158' size: 18586
E 2017-08-08T14:41:12.493085Z 0 [Note] Found gtids in binlog: '052ac796-7c44-11e7-93db-42010a84007d:1-1320,
E 542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3684139'
E 2017-08-08T14:41:12.493100Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-2850351'
E 2017-08-08T14:41:12.500533Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170808 14:41:12
E 2017-08-08T14:41:12.500710Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-08T14:41:12.501884Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-08T14:41:12.501927Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-08T14:41:12.502966Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-08T14:41:12.503008Z 0 [Note] IPv6 is available.
E 2017-08-08T14:41:12.503014Z 0 [Note] - '::' resolves to '::';
E 2017-08-08T14:41:12.503020Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-08T14:41:12.507540Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507570Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507576Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-08T14:41:12.507597Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.507619Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508335Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508352Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.508354Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.524474Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:41:12.540810Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-08T14:41:12.541274Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-08T14:41:15.714525Z 2 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.000159, pos: 601), semi-sync up to file , position 0.
E 2017-08-08T14:41:15.714567Z 2 [Note] Semi-sync replication switched OFF.
E 2017-08-08T14:42:12.360008Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709541672 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
E 2017-08-08 14:42:12 0x7ff6dbeee700 InnoDB: Assertion failure in thread 140698228549376 in file ut0ut.cc line 931
E InnoDB: Failing assertion: !m_fatal
E InnoDB: We intentionally generate a memory trap.
E InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
E InnoDB: If you get repeated assertion failures or crashes, even
E InnoDB: immediately after the mysqld startup, there may be
E InnoDB: corruption in the InnoDB tablespace. Please refer to
E InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
E InnoDB: about forcing recovery.
E 14:42:12 UTC - mysqld got signal 6 ;
E This could be because you hit a bug. It is also possible that this binary
E or one of the libraries it was linked against is corrupt, improperly built,
E or misconfigured. This error can also be caused by malfunctioning hardware.
E Attempting to collect some information that could help diagnose the problem.
E As this is a crash and something is definitely wrong, the information
E collection process might fail.
E key_buffer_size=8388608
E read_buffer_size=131072
E max_used_connections=4
E max_threads=4000
E thread_count=3
E connection_count=3
E It is possible that mysqld could use up to
E key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1597723 K bytes of memory
E Hope that's ok; if not, decrease some variables in the equation.
E Thread pointer: 0x55d5efe40800
E Attempting backtrace. You can use the following information to find out
E where mysqld died. If you see no messages after this, something went
E terribly wrong...
E Cannot determine thread, fp=7ff6dbee9e40, backtrace may not be correct.
E Stack range sanity check OK, backtrace follows:
E New value of fp=0 failed sanity check, terminating stack trace!
E mysqld(handle_fatal_signal+0x390) [0x55d5d3c94050]
E libc.so.6(gsignal+0x3b) [0x7ff7ed9542fb]
E libc.so.6(abort+0x14f) [0x7ff7ed95600f]
E mysqld(+0x702791) [0x55d5d3c57791]
E mysqld(_ZN2ib14fatal_or_errorD2Ev+0x248) [0x55d5d441fb78]
E mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x5f7) [0x55d5d45663e7]
E mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x4c) [0x55d5d456654c]
E mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x4d) [0x55d5d456670d]
E mysqld(_Z25trx_undo_get_undo_rec_lowmP16mem_block_info_tb+0x1a2) [0x55d5d44724a2]
E mysqld(_Z27trx_undo_prev_version_buildPKhP5mtr_tS0_P12dict_index_tPmP16mem_block_info_tPPhS7_PPK8dtuple_tm+0x2a2) [0x55d5d4472ae2]
E mysqld(_Z28row_vers_old_has_index_entrymPKhP5mtr_tP12dict_index_tPK8dtuple_tmm+0x1289) [0x55d5d44a1569]
E mysqld(_Z18row_purge_poss_secP12purge_node_tP12dict_index_tPK8dtuple_t+0x229) [0x55d5d44d3909]
E mysqld(+0xf7ebdf) [0x55d5d44d3bdf]
E mysqld(_Z14row_purge_stepP9que_thr_t+0x6ba) [0x55d5d44d56ba]
E mysqld(_Z15que_run_threadsP9que_thr_t+0x998) [0x55d5d4523a58]
E mysqld(_Z9trx_purgemmb+0x10f8) [0x55d5d44814f8]
E mysqld(srv_purge_coordinator_thread+0x1c9) [0x55d5d449da99]
E libpthread.so.0(+0x6890) [0x7ff7ee1e9890]
E libc.so.6(clone+0x6d) [0x7ff7ed93d2ed]
E Please read http://dev.mysql.com/doc/refman/5.1/en/resolve-stack-dump.html
E and follow instructions on how to resolve the stack trace.
E Resolved stack trace is much more helpful in diagnosing the
E problem, so please do resolve it
E Trying to get some variables.
E Some pointers may be invalid and cause the dump to abort.
E Query (0): Connection ID (thread ID): 0
E Status: NOT_KILLED
E The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
E information that should help you find out what is causing the crash.
E + ROOT=/mysql
E + rm -f /mysql/mysql.sock.lock
E + cat /mysql/my.cnf
I [mysqld]
I allow_client_hostname_forwarding=ON
I binlog-format=ROW
I block_user_access=33280 # FILE_ACL | SUPER_ACL
I character_set_server=utf8
I datadir=/mysql/datadir
I general_log_file=/mysql/logs/general.log
I hostname=localhost
I innodb_buffer_pool_dump_at_shutdown=ON
I innodb_buffer_pool_load_at_startup=ON
I innodb_file_format=Barracuda
I innodb_file_per_table=ON
I innodb_flush_method=O_DIRECT
I innodb_flush_neighbors=2
I innodb_flushing_avg_loops=5
I innodb_io_capacity=5000
I innodb_log_file_size=512M
I innodb_lru_scan_depth=2048
I innodb_monitor_enable=all
I innodb_page_cleaners=8
I master_info_repository=TABLE
I max_allowed_packet=32M
I max_binlog_size=104857600 # 100 MB
I max_connections=4000
I performance_schema=OFF
I pid_file=/mysql/mysql.pid
I relay_log=/mysql/relaylog/relay-log
I relay_log_info_repository=TABLE
I relay_log_recovery=ON
I reserved_super_connections=20
I rpl_semi_sync_master_enabled=1
I rpl_semi_sync_master_timeout=3000 # 3 second
I rpl_semi_sync_master_async_notify_socket=/semisync/semisync.sock
I rpl_semi_sync_master_do_async_notify=ON
I rpl_semi_sync_master_wait_point=AFTER_SYNC
I skip_name_resolve=ON
I slave_net_timeout=30
I slow_query_log_file=/mysql/logs/slow.log
I socket=/mysql/mysql.sock
I ssl_ca=/mysql/datadir/client_ca_cert.pem
I ssl_cert=/mysql/datadir/server_cert.pem
I ssl_cipher=ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
I ssl_key=/mysql/datadir/server_pkey.pem
I sync_binlog=1
I system_user_table=true
I tmpdir=/mysql/tmp
I require_super_for_system_table_writes=event,func,plugin,proc,heartbeat
I enforce_storage_engine=InnoDB
I log_bin=/mysql/binlog/mysql-bin
I gtid_mode=ON
I log_slave_updates=ON
I enforce_gtid_consistency=ON
I server-id=4292715972
I innodb_buffer_pool_size=4217853542
E + exec /usr/sbin/mysqld --defaults-file=/mysql/my.cnf
E 2017-08-08T14:42:15.633907Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
E 2017-08-08T14:42:15.633944Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path.
E 2017-08-08T14:42:15.633962Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.14-google-log) starting as process 1 ...
E 2017-08-08T14:42:15.637549Z 0 [Note] Plugin 'AUDIT_LOG' is disabled.
E 2017-08-08T14:42:15.637572Z 0 [Note] Plugin 'FEDERATED' is disabled.
E 2017-08-08T14:42:15.637868Z 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
E 2017-08-08T14:42:15.637892Z 0 [Note] InnoDB: PUNCH HOLE support not available
E 2017-08-08T14:42:15.637897Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
E 2017-08-08T14:42:15.637899Z 0 [Note] InnoDB: Uses event mutexes
E 2017-08-08T14:42:15.637901Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
E 2017-08-08T14:42:15.637903Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
E 2017-08-08T14:42:15.637906Z 0 [Note] InnoDB: Using Linux native AIO
E 2017-08-08T14:42:15.641182Z 0 [Note] InnoDB: Number of pools: 1
E 2017-08-08T14:42:15.641289Z 0 [Note] InnoDB: Using CPU crc32 instructions
E 2017-08-08T14:42:15.642723Z 0 [Note] InnoDB: Initializing buffer pool, total size = 4G, instances = 8, chunk size = 128M
E 2017-08-08T14:42:15.854815Z 0 [Note] InnoDB: Completed initialization of buffer pool
E 2017-08-08T14:42:15.912061Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
E 2017-08-08T14:42:15.936072Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
E 2017-08-08T14:42:15.946772Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 6672642734
E 2017-08-08T14:42:15.946814Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672647605
E 2017-08-08T14:42:15.958760Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 6672647605
E 2017-08-08T14:42:15.958901Z 0 [Note] InnoDB: Database was not shutdown normally!
E 2017-08-08T14:42:15.958908Z 0 [Note] InnoDB: Starting crash recovery.
E 2017-08-08T14:42:16.188562Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
E InnoDB: Progress in percent: 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
E 2017-08-08T14:42:16.778561Z 0 [Note] InnoDB: Apply batch completed
E 2017-08-08T14:42:16.778601Z 0 [Note] InnoDB: Last MySQL binlog file position 0 70560779, file name mysql-bin.000132
E 2017-08-08T14:42:16.888214Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
E 2017-08-08T14:42:16.888239Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
E 2017-08-08T14:42:16.888379Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
E 2017-08-08T14:42:16.989889Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
E 2017-08-08T14:42:16.990765Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
E 2017-08-08T14:42:16.990777Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
E 2017-08-08T14:42:16.991037Z 0 [Note] InnoDB: Waiting for purge to start
E 2017-08-08T14:42:17.042278Z 0 [Note] InnoDB: 5.7.14 started; log sequence number 6672647605
E 2017-08-08T14:42:17.043013Z 0 [Note] InnoDB: Loading buffer pool(s) from /mysql/datadir/ib_buffer_pool
E 2017-08-08T14:42:17.045458Z 0 [Note] Semi-sync replication initialized for transactions.
E 2017-08-08T14:42:17.045472Z 0 [Note] Semi-sync replication enabled on the master.
E 2017-08-08T14:42:17.134897Z 0 [Note] InnoDB: Buffer pool(s) load completed at 170808 14:42:17
E 2017-08-08T14:42:17.135878Z 0 [Note] Recovering after a crash using /mysql/binlog/mysql-bin
E 2017-08-08T14:42:17.136199Z 0 [Note] Starting crash recovery...
E 2017-08-08T14:42:17.136251Z 0 [Note] Crash recovery finished.
E 2017-08-08T14:42:17.136341Z 0 [Note] Performed crash recovery to binlog file '/mysql/binlog/mysql-bin.000159' size: 18913
E 2017-08-08T14:42:17.136954Z 0 [Note] Starting ack receiver thread
E 2017-08-08T14:42:17.143780Z 0 [Note] Found gtids in binlog: '052ac796-7c44-11e7-93db-42010a84007d:1-1377,
E 542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-3684139'
E 2017-08-08T14:42:17.143803Z 0 [Note] Found gtids purged from binlog: '542217ec-9b5b-11e6-9e03-42010a840ffd:1-21109977,
E b0979fb3-571b-11e7-b051-42010a84001c:1-2850351'
E 2017-08-08T14:42:17.148899Z 0 [Note] Skipping generation of SSL certificates as options related to SSL are specified.
E 2017-08-08T14:42:17.149474Z 0 [Warning] CA certificate /mysql/datadir/client_ca_cert.pem is self signed.
E 2017-08-08T14:42:17.149527Z 0 [Note] Skipping generation of RSA key pair as key files are present in data directory.
E 2017-08-08T14:42:17.149985Z 0 [Note] Server hostname (bind-address): '*'; port: 3306
E 2017-08-08T14:42:17.150022Z 0 [Note] IPv6 is available.
E 2017-08-08T14:42:17.150026Z 0 [Note] - '::' resolves to '::';
E 2017-08-08T14:42:17.150030Z 0 [Note] Server socket created on IP: '::'.
E 2017-08-08T14:42:17.152789Z 0 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152824Z 0 [Warning] 'user' entry 'cloudsqlimport@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152837Z 0 [Note] Loaded users from system_user table: root@localhost,cloudsqlreplica@%,cloudsqlimport@localhost.
E 2017-08-08T14:42:17.152886Z 0 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152911Z 0 [Warning] 'user' entry 'gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152984Z 0 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152994Z 0 [Warning] 'db' entry 'insider\_trading gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.152996Z 0 [Warning] 'db' entry 'textual-db gordon@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.166564Z 0 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.
E 2017-08-08T14:42:17.180096Z 0 [Note] Event Scheduler: Loaded 0 events
E 2017-08-08T14:42:17.180265Z 0 [Note] /usr/sbin/mysqld: ready for connections.
E Version: '5.7.14-google-log' socket: '/mysql/mysql.sock' port: 3306 (Google)
E 2017-08-08T14:42:20.980545Z 3 [Warning] Timeout waiting for reply of binlog (file: mysql-bin.000160, pos: 601), semi-sync up to file , position 0.
E 2017-08-08T14:42:20.980588Z 3 [Note] Semi-sync replication switched OFF.
Hi!
-- On Monday (170807) I noticed that we started to get unstable connections to our Cloud SQL instance (2nd generation). Under the "Operations" tab I noticed that a backup had failed.
The database seemed to work on and off, until today Tuesday (170808) when it totally broke down on us. Now it has a yellow warning sign beside it, saying "Under maintenance", and I cannot do anything with it.
Checking the logs, and I found this:
[ERROR] InnoDB: Cannot allocate 18446744073709546192 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
(See the attached mysql_err_log_1.txt for complete log)
and I found this MySQL bug: https://bugs.mysql.com/bug.php?id=70185 which I find very similar to the error message that I got. That person is suggesting two things:
1) add 'innodb_force_recovery=4' to [mysqld] in my.cnf or.... 2) set 'innodb_change_buffering=none' in my.cnf
But these are flags that gcloud won't allow me to change:
gcloud sql instances patch [INSTANCE-NAME] --database-flags innodb_force_recovery=4
The following message will be used for the patch API method.
{"project": [PROJECT-NAME], "name": [INSTANCE-NAME], "settings": {"databaseFlags": [{"name": "innodb_force_recovery", "value": "4"}]}}
WARNING: This patch modifies a value that requires your instance to be
restarted. Submitting this patch will immediately restart your
instance if it's running.
Do you want to continue (Y/n)? Y
ERROR: (gcloud.sql.instances.patch) HTTPError 404: Flag requested cannot be set.
And non of those flags can be found here: https://cloud.google.com/sql/docs/mysql/flags
I have also tried to spin up multiple new SQL instances, one with the same spec as the original, and one with everything doubled (vCPU, RAM, SSD) in case there was any resource limits that hit the ceiling, and tried to restore to a previous point in time using our backups, but without any luck. I've tried to restore to the backup one day before this happened (One backup at 6 Aug 2017, 03:23:17, and one at 4 Aug 2017, 00:55:01)
All my services that tries to access the database get one of either (2013, 'Lost connection to MySQL server during query') or (2006, 'MySQL server has gone away').
I've also tried to do an export using the Google cloud web UI, but that export fails with the following error message:
mysqldump: Couldn't execute 'show table status like 'email\_meta'': Lost connection to MySQL server during query (2013)
mysqldump: Couldn't execute 'show table status like 'email\_meta'': Lost connection to MySQL server during query (2013)
Trying to do an export using Sequel Pro (a mysql client for mac) yields the same problem: The connection fails.
This means I cannot even export my data so that I can manually re-create all my tables and insert data.
I'm running out of ideas of how to find what is wrong, does anyone have any idea and can help me, it will be highly appreciated!
I'll also attach the logs for one of the newly created instances which I used to restore the 170804 backup. There it can be seen that the mysql server restarts multiple times but fails to keep alive. That file is called mysql_err_log_2.txt
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/88aaf57f-f17c-4461-a655-f1e0669b6826%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
No comments:
Post a Comment