Note: this was not a deprecation. There is a bug in the version of 1.05 which was triggered by a rollout done by a completely different team at Google (inside Google there is a lot of shared infrustructure so it's likely that something is being updated all the time, so this rollout was certainly not anything my team was directly responsible for). We found the rollout, proved it was causing this problem, and asked that team for a prompt rollback. Version 1.05 was never broken globally (getting a PROTOCOL_ERROR would have happened depending on where your service was running geographically), and should be working perfectly fine as of yesterday afternoon (pacific time) after a quick rollback of the other team's software. If you are still being affected, please reopen this github issue.
On Wednesday, May 31, 2017 at 8:20:26 AM UTC-7, Thad Megow wrote:
-- We will update the documentation and I will personally work to find a better system to always keep the versions up to date in the documentation; as the one who normally does releases, I apologize for not syncing with our documentation writer to update the pages. In any case, the github page for releases is the canonical list of versions; please subscribe to that repository for version update announcements.
As for the 'latest' version of docker: I have heard from numerous people in the Docker community that the 'latest' tag is consistently more trouble than it's worth due to a similar (but somewhat opposite issue): if there is a bug in a 'new' version of the Cloud SQL Proxy, we certainly don't want to immediately break our users. Instead, I encourage users to periodically update (all of) their dependencies as is needed and is convenient for your own software's rollout schedule. I periodically update the version that the 'latest' tag points to since I expect there are those who chose to ignore this advice, but I definitely want to wait some time to ensure that any bugs in the new version are dealt with before making it any sort of default.
On Wednesday, May 31, 2017 at 8:20:26 AM UTC-7, Thad Megow wrote:
For those of us who use the cloudsql proxy docker image, the Google docs indicate to pin to a specic version. Currently 1.09 is the recommended version here (https://cloud.google.com/sql/docs/mysql/connect-docker ).When I initially setup the cloudsql proxy docker image for production that document said to pull 1.05 - and it bit us when Google decided deprecate support for older versions of their official cloudsql proxy docker images?Is using the "latest" image bad practice here? Is Google just going to keep updating that doc and deprecating old images without any warning?
On Wednesday, May 31, 2017 at 11:13:14 AM UTC-4, Skyler Slade wrote:This worked: we updated the version of Cloud Proxy we were using from v1.05 to v1.09 and the problem appears to have gone away.That worked for us as well.It was super fun to have a ~15 minute production outage caused by that this past Monday :)
On Tuesday, May 30, 2017 at 1:25:49 PM UTC-4, Siyuan Zhang wrote:Hi,We are using cloud-sql-proxy together with kubernetes to deploy our app, it works fine since Nov. 2016. For the first time, today, 2 of our 5 pods reported bad connection when trying to connect to google-cloud-sql with cloud-sql-proxy. here are parts the logs:central1:diningzoodatabase" 2017/05/30 16:42:44 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:44 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:44 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:45 Client closed local connection on 127.0.0.1:33062017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 277.184µs ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 1.194005ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 1.644907ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 382.94776ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 383.939371ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 384.954228ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 385.11944ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 388.399746ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 389.191251ms ago 2017/05/30 16:42:46 couldn't connect to "teak-clarity-132123:us-central1:diningzoodatabase": Post https://www.googleapis.com/ sql/v1beta4/projects/teak- : stream error: stream ID 1; PROTOCOL_ERRORclarity-132123/instances/ diningzoodatabase/ createEphemeral?alt=json 2017/05/30 16:42:46 New connection for "teak-clarity-132123:us-central1:diningzoodatabase" 2017/05/30 16:42:46 Thottling refreshCfg(teak-clarity-132123:us-central1: diningzoodatabase): it was only called 497.448221ms ago We resolved this by deleting the pod, and after 30mins we restarted the first pod with this issue, another pod reported the same errors. and the end, we have restarted all pods. it seems gone now, but dont know if it will occur again.
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/2fb8545b-da6f-425e-9dd3-1b841481ac87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
No comments:
Post a Comment