[google-cloud-sql-discuss] Connection issues using cloudsqlproxy and Cloud SQL 2 gen HA

Hi there!

We recently ran into networking issues between App servers on Compute Engine and SQL MYSql server (2 gen. HA).

This issue has affected us now two times and during the incident we are unable to connect to the SQL MYSql server via cloudsqlproxy or directly via IP.

During the incident we also noticed several updates being applied to the SQL MYSql 2 Gen server, even if out of the scheduled update time, (evidence further below).

Please note that we did not face any kind of networking issues since the deployment of the infrastructure, which happened ~14 moths ago.

Evidence of connection issues:

• 2019/10/17 07:18:32 couldn't connect to "#######-website:us-east1:#####mysql01": dial tcp ########:3307: getsockopt: connection timed out

Example of logs collected on the Cloud SQL console:

• Aborted connection ###### to db: '######' user: '#######' host: 'cloudsqlproxy~########' (Got an error reading communication packets)
• Slave I/O for channel: error reconnecting to master 'cloudsqlreplica@#########:3306' - retry-time: 60  retries: 1, Error_code: 2003

Several updates being applied during incident:

Updates being applied outside planned window:

Solution:

The only way to reestablish the connection between App servers and SQL MYSql server was manually triggering the failover.

Once the process completes, the connection is reestablished and works as expected.

Did anybody out there face the same issue?

Thank you very much!

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/50bfe831-c6bf-47ef-9673-3ada33a26ab2%40googlegroups.com.