[google-cloud-sql-discuss] Re: [cloud-sql-proxy] couldn't connect to database error, stream error: stream ID 1; PROTOCOL_ERROR

For those of us who use the cloudsql proxy docker image, the Google docs indicate to pin to a specic version. Currently 1.09 is the recommended version here (https://cloud.google.com/sql/docs/mysql/connect-docker).

When I initially setup the cloudsql proxy docker image for production that document said to pull 1.05 - and it bit us when Google decided deprecate support for older versions of their official cloudsql proxy docker images?

Is using the "latest" image bad practice here? Is Google just going to keep updating that doc and deprecating old images without any warning?

On Wednesday, May 31, 2017 at 11:13:14 AM UTC-4, Skyler Slade wrote:

This worked: we updated the version of Cloud Proxy we were using from v1.05 to v1.09 and the problem appears to have gone away. 

That worked for us as well.

It was super fun to have a ~15 minute production outage caused by that this past Monday :)

On Tuesday, May 30, 2017 at 1:25:49 PM UTC-4, Siyuan Zhang wrote:

Hi,

We are using cloud-sql-proxy together with kubernetes to deploy our app, it works fine since Nov. 2016. For the first time, today, 2 of our 5 pods reported bad connection when trying to connect to google-cloud-sql with cloud-sql-proxy. here are parts the logs:

2017/05/30 16:42:44 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:44 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:44 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:44 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:3306

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 277.184µs ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 1.194005ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 1.644907ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 382.94776ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 383.939371ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 384.954228ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 385.11944ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 388.399746ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 389.191251ms ago

2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/sql/v1beta4/projects/teak-clarity-132123/instances/diningzoodatabase/createEphemeral?alt=json: stream error: stream ID 1; PROTOCOL_ERROR

2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase"

2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1:diningzoodatabase): it was only called 497.448221ms ago

We resolved this by deleting the pod, and after 30mins we restarted the first pod with this issue, another pod reported the same errors. and the end, we have restarted all pods. it seems gone now, but dont know if it will occur again.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f8f3ad41-6a53-479f-8f9d-e2421a9f3cbc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.