Wednesday, January 24, 2018

[google-cloud-sql-discuss] Re: Three nights in a row, synchronised outage of all connections from GAE to Cloud SQL

I've opened a private issue to facilitate investigation into this further and posed some initial questions.  This is currently scoped to the gmail address used to post here, but I can add other users upon request.

On Tuesday, January 23, 2018 at 6:11:48 PM UTC-8, Robin Donaldson wrote:
(Second time attempting to post this)

We have a simple App Engine (PHP) with PDO connection to Cloud SQL (MySQL) architecture.  We have programmed for stale connections;  at the start of every PHP request, we get a connection to the database and check if it's active, and if not we time out within 4 seconds and get a new connection.  This code is good and works well, we see in the logs that this code works as expected.  

We have a Staging project (Cloud SQL 2nd gen) and Production project (Cloud SQL 1st gen).

For the past three nights, we've had SYNCHRONIZED outages across Staging and Production.  All connections to Cloud SQL go stale and do not time out according to our 4s rule.  The page requests all time out.  

Tonight, during the latest outage, I restarted the database twice which restarts the connection pool.  This did NOT help.  

I conclude that this is a Google Cloud issue and not my issue because:
- It has happened each night around the same time
- It is synchronized across PROJECTS and PRODUCTS (Cloud SQL 1st/2nd gen)
- Restarting the DB did not fix it.

I now demand answers.  I have customers who are complaining about these outages and we as growing business have a reputation to keep.

Please assist.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/6770c243-67ca-4af2-a338-eaa8a5ec3086%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

No comments:

Post a Comment