Monday, December 31, 2018

[google-cloud-sql-discuss] Re: How to restore multiple PostgreSQL databases on a single instance to the same point in time?

The Postgres file system level backup technical details can be verified here. However, please note that the internal technical details, for instance, how the Cloud SQL Postgres managed database service backups are performed, are only available to the Cloud SQL engineering team. That way, they can guarantee that the service will remain reliable and secure.

On Friday, December 14, 2018 at 5:44:12 PM UTC-5, Alon Herbst wrote:
Thank you for the answer Aref.
Can you give more technical details on how the backup and restore are done? Which PostgreSQL files? And how is the snapshot taken without service interruption?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f6f1cc7f-afec-4eaa-a008-7dc3545022bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Friday, December 28, 2018

Re: [google-cloud-sql-discuss] Re: CloudSQL proxy & corp firewall question

That doesn't help. I need to know what ip's the sql proxy server uses to whitelist. My security team will not open the port to all ip's

On Fri, Dec 7, 2018, 10:42 AM 'Katayoon (Cloud Platform Support)' via Google Cloud SQL discuss <google-cloud-sql-discuss@googlegroups.com wrote:
Hi Scott,

When you connect using TCP, you only need to ensure outgoing TCP (3306) on remote port 3307 is allowed, however you can specify another port for it to use as well.


--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/9b1bc87c-1b99-4e68-80dd-52cea363c110%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/CAOTThP%3DShxJVSbvQ%3DZYpw1ZcTQAFKz2wamHbbFQm5dMQrAZx_g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Wednesday, December 19, 2018

[google-cloud-sql-discuss] "SQLSTATE[HY000] [2006] MySQL server has gone away" error

Hi,

My SQL Cloud Instance had a problem for ~20 hours (13:27 GMT yesterday to 09:47 today), it randomly dropped connections with the following error, without any reason :
SQLSTATE[HY000] [2006] MySQL server has gone away

The graphs were also dead for this period :

Screenshot_20181219_111405.png



I've found these errors in the logs, occurring only at the beginning and the end of this abnormal situation :

E 2018-12-18T13:27:08.099086Z 583928 [Note] Aborted connection 583928 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-18T13:27:09.334641Z 583904 [Note] Aborted connection 583904 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-18T13:27:09.335742Z 24 [Note] Aborted connection 24 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-18T13:27:11.526683Z 9 [Note] Aborted connection 9 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-19T09:47:58.863151Z 594976 [Note] Aborted connection 594976 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-19T09:47:58.864093Z 594975 [Note] Aborted connection 594975 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-19T09:47:58.914624Z 594974 [Note] Aborted connection 594974 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)
E 2018-12-19T09:48:06.935322Z 594973 [Note] Aborted connection 594973 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)

My project only uses the cloudsql/gce-proxy docker image to connect to the sql server.

Any idea what could have caused this situation ?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/54e7390b-d864-425f-b04a-2d404b77406e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tuesday, December 18, 2018

[google-cloud-sql-discuss] (gcloud.sql.instances.patch) HTTPError 409: The instance or operation is not in an appropriate state to handle the request.

(gcloud.sql.instances.patch) HTTPError 409: The instance or operation is not in an appropriate state to handle the request.

We have been down for about an hour. All UI options are grayed out. Can we get a restart?

Project is:
 jcps-backpack

Cloud sql name:
jcps-backpack


Thanks

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/89272117-bd20-4ad9-8212-85838e46c3c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Friday, December 14, 2018

[google-cloud-sql-discuss] Re: How to restore multiple PostgreSQL databases on a single instance to the same point in time?

Thank you for the answer Aref.
Can you give more technical details on how the backup and restore are done? Which PostgreSQL files? And how is the snapshot taken without service interruption?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/de380d20-ebaf-4d77-af60-f244922497b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: How to Monitor DML GoogleCloud postgres

Hi Jonathan,


At the moment this feature is only available for Cloud SQL MySQL instances and not available for Cloud PostgreSQL. There is however a feature request for this on issue tracker that you can follow for updates.


Hope this helps.



On Friday, December 14, 2018 at 12:11:39 PM UTC-5, Jonathan Delgado wrote:
Good morning, someone can help me with the following:
How to monitor DML queries in GoogleCloud postgres?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/4a75e584-c626-4f9c-b856-e4c9cdb51ba1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] How to Monitor DML GoogleCloud postgres

Good morning, someone can help me with the following:
How to monitor DML queries in GoogleCloud postgres?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/4878c94f-b51e-4976-a929-72332b468feb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wednesday, December 12, 2018

[google-cloud-sql-discuss] Re: How to restore multiple PostgreSQL databases on a single instance to the same point in time?

To avoid operation disruption, the backup and recovery is done using file-system backup.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/7163505c-30b7-499c-aa9e-9f32014b23f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Friday, December 7, 2018

[google-cloud-sql-discuss] Re: How to restore multiple PostgreSQL databases on a single instance to the same point in time?

Can you explain which kind of backup is used? Is it pg_dump like / file system backup / something else?
This will help to understand how the promise to restore all databases in the instance to the exact same point in time works.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/465c3193-2ad5-423b-8be9-5133d3eff485%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: How to restore multiple PostgreSQL databases on a single instance to the same point in time?

If you restore a Cloud SQL instance backup, it will restore all the databases in that instance to the point when backup was taken. One thing to note here is that if you restore the backup in the same instance it will overwrite the current data with the one in backup [1].


You can see the General tips about performing a restore at [2].


[1]https://cloud.google.com/sql/docs/postgres/backup-recovery/restoring#restorebackups-same-instance

[2] https://cloud.google.com/sql/docs/postgres/backup-recovery/restore#tips-restore



On Friday, December 7, 2018 at 10:25:09 AM UTC-5, Alon Herbst wrote:
I have 2 separate PostgreSQL databases in Google Cloud SQL. The databases should be separate due to security concerns, but they can reside in the same instance. When restoring from a backup, it is important for me to have the databases restored to the same point in time, due to data relations.
I tried to understand how the backup works. I tried to research the backup documentation and the restore one, but found no clue.
Can I be sure that restoring an instance would bring all its databases to the same point in time?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/a4f1796e-09d2-4723-8aa4-142f7efb858a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: CloudSQL proxy & corp firewall question

Hi Scott,

When you connect using TCP, you only need to ensure outgoing TCP (3306) on remote port 3307 is allowed, however you can specify another port for it to use as well.


--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/9b1bc87c-1b99-4e68-80dd-52cea363c110%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] How to restore multiple PostgreSQL databases on a single instance to the same point in time?

I have 2 separate PostgreSQL databases in Google Cloud SQL. The databases should be separate due to security concerns, but they can reside in the same instance. When restoring from a backup, it is important for me to have the databases restored to the same point in time, due to data relations.
I tried to understand how the backup works. I tried to research the backup documentation and the restore one, but found no clue.
Can I be sure that restoring an instance would bring all its databases to the same point in time?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/6f4a805c-a034-429a-803a-6b786079b3a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thursday, December 6, 2018

[google-cloud-sql-discuss] Re: Cloud SQL Proxy host

Apparently you have created a duplicate thread on the same question. In order to avoid duplicating efforts I recommend to work with the duplicate one.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/c1cb1771-3702-4c87-9f3c-5f1e6903fe66%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Very high CPU Usage for few hrs every day

Hi,

Diagnosing issues with your Cloud SQL instance, you may inspect the Cloud SQL instance operation logs or the instance error logs. There is a likelihood that you are running a workload with intensive queries at the mentioned interval. 

If you couldn't find the root cause in the logs, I recommend to raise a PRIVATE report in the Issue Tracker and provide us with your project number, instance id and the timestamp you get these spikes along with your zone to investigate the issue efficiently.


--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/dac86c72-b53d-41de-9637-21c7794f2107%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Backup restore to different instance, in different project

Found a way, you can do:
gcloud sql export sql db-production gs://someproject-development-bucket/db-production_dump.sql --project someproject-production && \
gcloud sql import sql db-development gs://someproject-development-bucket/db-production_dump.sql --project someproject-development

As long as permissions are set, you can import/export to/from any project to/from any project.

---

Our setup:
  
project-development:
- Cloud SQL: db-development
- Cloud Storage: bucket-development
- Cloud Build: clone-prod-to-dev

project-production:
- Cloud SQL: db-production

// YAML from Cloud Build trigger:
steps:
  
  # export SQL databases
  - name: 'gcr.io/cloud-builders/gcloud'
    args: [
      'sql', 'export', 'sql',
      'db-production',
      'gs://someproject-development-bucket/db-production_dump-$BUILD_ID.sql',
      '--project', 'someproject-production'
    ]
    waitFor: ['-']
    id: 'sql-export'
    
  # import SQL databases
  - name: 'gcr.io/cloud-builders/gcloud'
    args: [
      'sql', 'import', 'sql',
      'db-development',
      'gs://someproject-development-bucket/db-production_dump-$BUILD_ID.sql',
      '--project', 'someproject-development'
    ]
    waitFor: ['sql-export']


On Tuesday, January 24, 2017 at 6:25:52 PM UTC+1, Dan Gardner wrote:
I'm failing to restore a backup from a source instance to a target instance when each instance is in a different project.  Is this simply not supported ?

I get ambiguous permissions error from the target instance restore cmd**.   Also if I look in the console no option is offered to target instances in diff projects,  resonable I suppose for thr console.  

I guess unless you can pass a project in the restoreBackupContext json, as below, this isn't supported yet ?
  
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
     
--header 'Content-Type: application/json' \
     
--data '{ "restoreBackupContext": {"backupRunId": "[BACKUP_ID]", "instanceId": "[SOURCE_INSTANCE_NAME]", "kind": "sql#restoreBackupContext"}}' \
     
-X POST \
     https
://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances/[TARGET_INSTANCE_NAME]/restoreBackup

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/e42b0387-9395-417a-a98b-44c849061ae2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wednesday, December 5, 2018

[google-cloud-sql-discuss] Cloud SQL Proxy host

I am behind a corp firewall. My security team is willing to open the port but they want to allow it to only certain hosts. Does anyone know if there is a range or lists of hosts that needs to be allowed?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/47bf97ad-82e4-4b45-99a6-d83dcb4b3c21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] CloudSQL proxy & corp firewall question

Our corp firewall is blocking outbound connections to port 3307. The security team is willing to work with me on opening it up, but they want to open it up to only specific IP's or ranges. 

Does anyone know of the hosts that would need to be allowed? im assuming its by zone. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/1501b197-45cd-4f60-a523-c29c14517f27%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tuesday, December 4, 2018

[google-cloud-sql-discuss] Very high CPU Usage for few hrs every day

Hi

Our instance configuration is this

6 core with 18 GB RAM

We dont get much traffic to our site, but for last 1 week we have been noticing that the mysql server CPU usage is every high, something like 500%. After 2-3 hrs it goes back to normal. This usually happens between around 11 AM  to 2 PM. I tried process list but there is nothing abnormal. Not sure where to look for as i am mostly a dev and dont have much db experience 

Any pointers ??? 

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/16296360-a322-4bc0-9db1-93b88cdf540e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Saturday, December 1, 2018

[google-cloud-sql-discuss] Flag "lower_case_table_names" dont change after server restart

The specific flag "lower_case_table_names" don't change after server restart. Its disabled by Google SQL or need additional config ?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/5a3885ca-636c-466c-a81e-00d6af04ef3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Friday, November 30, 2018

[google-cloud-sql-discuss] Re: MySQL 8.0 support

Hi,

You may follow this feature request which already exists in the Issue Tracker via this link. I transferred your interest on having MySQL 8.0 support to Google Cloud SQL product team as well. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/d446d469-e366-4ee6-bc4e-acc7993c1eba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Logical Decoding

Hello Sri Krishna, 

You are right, there is work in progress on exactly the same issue; you may follow title "Change the wal_level to logical for logical decoding" in Issue Tracker

It may be worthwhile mentioning here that, not surprisingly, this feature is present for instances deployed in Compute Engine; details on the "How to Set Up PostgreSQL for High Availability and Replication with Hot Standby" page

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/ccc76562-3ac8-426d-9040-87888807de42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: MySQL 8.0 support

aws supports it now: https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-rds-now-supports-mysql-8/

any word yet, google?

On Wednesday, November 7, 2018 at 5:11:41 PM UTC-5, Ryan Toth wrote:
Any word on this yet performance difference is huge!!! thinking about mariadb if this isn't rolling out sometime soon.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/0d4e7a65-1158-4a3c-a8e2-5625b429a2ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thursday, November 29, 2018

[google-cloud-sql-discuss] Logical Decoding

Hi,

We want to use logical decoding to read and push any changes to our database into data processing pipelines. I've tried doing it as documented in the official Postgres docs for 9.6. But cloud sql is disallowing me to change the wal_level to logical. I've looked for this in other providers and Amazon RDS already provides this and has a blog posted about it here.

Is there a way do this already? If yes then can you point me to the documentation?
If there isn't is it part of the roadmap?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/0aefe94b-c437-434f-b82a-1d7d4729af04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Monday, November 26, 2018

[google-cloud-sql-discuss] Re: Cloud SQL PostgreSQL Point in time recovery

Hi Jonathan,

I should note that Point-in-time recovery for Cloud PostgreSQL isn't available at this time, however there is a feature request on it and the Cloud SQL product team are evaluating this feature request. You may star it for further updates.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/abe7e18e-bc8d-47ae-af91-1f7a195c938e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Cloud SQL PostgreSQL Point in time recovery

Does anyone have any solutions to implementing a point in time recovery on a PostgreSQL hosted on GCP?

A couple ways that I have found that could be a potential solution is if Cloud SQL PostgreSQL could create an external read replica so that I would be able to have the replica update the write ahead log file every time it changes, or somehow get access to the CloudSQL write ahead logs. To my knowledge, I don't think these features have been implemented yet.

Any suggestions or leads will be helpful. Thank you.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/a551ea73-f598-4a6b-95bd-3d0d02b8802c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Saturday, November 24, 2018

[google-cloud-sql-discuss] Re: Google Cloud SQL (Postgres) 30% slower than local PC

Thanks Dan. We got it to a point now where the performance is fair enough. It's a tidy bit slower then our previous setup but it can handle load better, so we are fine with it for now. Will go through the articles soon to see if we can improve it a bit more. 

Op maandag 19 november 2018 22:09:26 UTC+1 schreef Dan (Cloud Platform Support):

Hello,


It seems that you didn't create this report two times, there are no worries.


Unfortunately, there is no way to compare the Cloud SQL Postgre performance with another Cloud SQL service from my side.


It seems that you're following the right steps to improve the Database performance, such as check the location, increase the tier when the instance looks throttled and upgrade the disk since larger disks have more IOPS available. I would add on your to-do list to implement cache, case your queries will not be constrained to I/O performance. Ideally, the entire data set can fit in 70% of RAM. Additionally, I recommend that you investigate non-Cloud SQL issues, like optimized queries by adding appropriate indexing, reduce the scanned data and avoid extra round trips. Uou can find some information regarding this in the following website[1].


You can troubleshoot performance issues by following the instructions in the following documentation[2].


I hope this documentation could help you to determine the root cause of this performance issue,


[1] https://use-the-index-luke.com/

[2] https://cloud.google.com/sql/docs/postgres/diagnose-issues



On Thursday, November 15, 2018 at 11:07:03 AM UTC-5, Marcel wrote:
Hi,

I hope I don't post this duplicate because I already posted this, but I cannot find it anywhere so maybe something went wrong.

We are migrating our APP + DB to Google Cloud. However, we can't get the database to a level of performance as needed. We created a heavy select query with lots of joins and on my local PC it's way faster than it's on Google Cloud SQL. We are moving from a managed Amazon solution, and the current setup on Amazon is also faster. On both my PC and Amazon it's somewhere nere 30% faster on any query. 

How come Google Cloud SQL Postgres is so slow? I tried a lot of things to increase the performance:
  • Upgraded the disk to improve read/write speed as suggested by Google
  • Added more CPUs and RAM (makes no difference)
  • The database instance runs in Frankfurt which is close to my home so I don't think there is any noticable latency on this end
What else can I try to improve the performance? I find it very strange that my PC (which is not optimized to run Postgres) is 30% faster than a optimized instance on Google Cloud. 

How can I debug this and how can I improve this? 

Thanks! 

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/78df2a8f-acf3-4c9b-8955-5640af3cee18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thursday, November 22, 2018

[google-cloud-sql-discuss] Re: Migrating from Cloud SQL to Datastore

Hello Sajid, 

How did you import datastore from google.cloud? The entirety of your code is needed for examination. This is not the right place for such extensive coding issue. As pointed out in a previous answer, if your issue is not a platform issue but rather a problem with how you've configured your code, you should ask for help in specialized forums such as stackoverflow. 

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/22bc2da7-73c5-4e99-98b1-26771d681961%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: What is the timeline for when Cloud SQL upgrades to Postgres 11

Hello Ray, 

This is a known feature request, and in progress with Engineering, under an issue in the Public Issue Tracker. You may follow developments up to final implementation in that PIT issue

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/1f957c52-0e6b-4f5f-a539-f79bd534da50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Wednesday, November 21, 2018

[google-cloud-sql-discuss] What is the timeline for when Cloud SQL upgrades to Postgres 11

Where can I find information about when Cloud SQL will be supported newer versions of postgres (like Postgres 11)?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/c4c93677-eeeb-41e4-a9e5-41dc845a4f29%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: LDAP authentication for PostgreSQL

Hello,

It is currently not possible to use LDAP authentication in PostgreSQL. However, I have found an open feature request for this on the issue tracker website [1]. I suggest you star it to get email updates. In addition, users have no access to pg_hba.conf in Cloud SQL.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/b15d4cb6-65c1-4394-b488-fc6f8ba24349%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Cannot connect to Cloud SQL with Java SocketFactory Library

Yeah, it was a firewall issue indeed.
Linked SO question: https://stackoverflow.com/questions/53306610/cannot-connect-to-cloud-sql-with-java-socketfactory-library

On Tuesday, November 20, 2018 at 8:09:44 PM UTC-2, Yasser Karout (Cloud Platform Support) wrote:
Hello,

Are you connecting to Cloud SQL through a proxy? If you have a corporate firewall setup, you might need to add an exception to allow connections to the instance. It is worth looking into this if the connection is being blocked. 

I found a similar issue on the issue tracker website that suggest this is a proxy issue [1]. There is also a similar issue posted on Stackoverflow but for MySQL [2]. 

It might be better to post this issue on Stackoverflow as it is more suited for development questions.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/654a684f-edb4-4e65-9eca-236188f85881%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Migrating from Cloud SQL to Datastore

Hi,
Thanks for looking into this.
Actually I solved this and it seems that the documentation is not clear about the importance of custom namespaces.

All I had to do was change:
$datastore = new DatastoreClient(['projectId' => $projectId]);
to 
$datastore = new DatastoreClient(['projectId' => $projectId, 'namespaceId' => 'ssd']);

I was not using 'default' namespace. That was causing all the problem.
Hope someone will add this to the documentation.


On Wednesday, November 21, 2018 at 8:26:48 AM UTC-8, Mohammad I (Cloud Platform Support) wrote:

Hello Sajid,


I have reproduced your code as below with some modification after executing the commands and PHP code outlined here.


$datasetId = $projectId;

$datastore = new DatastoreClient(['projectId' => $projectId]);

function llist($datastore){

$query = $datastore->query()

       ->kind('Task');

   $results = $datastore->runQuery($query);

$entries = [];

$count = 0;

foreach ($results as $entity) {

$count++;

}


echo $count; // this shows me '1' results when I have used Kind('Task') and shows me '0' for Kind('Task1') .

}

llist($datastore);


The kind, 'Task' was defined in the PHP code of the above mentioned link as a result when the modified version of your code was executed it has shown the result 1. When the kind, 'Task' was modified with 'Task1' it has shown the result 0.


From the above observation, it is appearing the kind, 'keypad_research' that you used may not exist or your typed it wrong. In addition the variable $cursor is not defined in your code which I removed during reproduction as it was showing the following error:


PHP Notice:  Undefined variable: cursor in /home/.../php/index.php on line 17


As it was mentioned earlier, If your issue is not a platform issue but rather a problem with how you've configured your code, although you're not sure what it may be specifically, you should post to StackOverflow and community of developers will assist you.



On Tuesday, November 20, 2018 at 12:50:02 AM UTC-6, sajid saiyed wrote:
Thanks Omair for your guidance.

So I tried to first see if my PHP can talk to Datastore and retrieve data.
I created two entities nuder the kind "keypad_research".

This is how my PHP looks like:

<?php
require __DIR__ . '/../../vendor/autoload.php';
use Google\Cloud\Datastore\DatastoreClient;
use Google\Cloud\Datastore\Entity;
$projectId = "__my projectID__";
$datasetId = $projectId;
$datastore = new DatastoreClient(['projectId' => $projectId]);
function llist($datastore){
$query = $datastore->query()
        ->kind('keypad_research')
->start($cursor);

    $results = $datastore->runQuery($query);
$entries = [];
$count = 0;
foreach ($results as $entity) {
$count++;
}

echo $count; // this shows me '0' results even when I have 2 entities.
}
llist($datastore);
?>

Do I need to do any addition configuration or edit my PHP to be able to read from datastore?

I am thinking of writing a PHP script that will first read from SQL database and insert entities in datastore.
But my very basic attempt at reading data from datastore using PHP is failing.

Any advice is appreciated.


On Monday, November 19, 2018 at 9:28:34 AM UTC-8, Omair (Cloud Platform Support) wrote:

Hi Sajid,


Unfortunately there is no automatic way to migrate a Cloud SQL database to Datastore.  The general process for migrating is as follows:


  1. Map your relational model to datastore kinds and entities

  2. [Optional] Download your relational database as csv

  3. Write a script using one of our Cloud Datastore Client Libraries that would transfer your data from Cloud SQL or the csv in step 2 to Datastore

  4. If you have a large database, it would speed up the migration process by hosting your script on App Engine.



If you require assistance in writing your script for migrating relational database to Datastore it is recommended to post your complete question to Stack Overflow using one of our supported tags, as your question will receive a lot more attention there.


I have found a useful post on Stack Overflow for you that I think might help you.



On Sunday, November 18, 2018 at 4:22:23 PM UTC-5, sajid saiyed wrote:
Hi,
I am currently using Cloud SQL but its an over kill for the small application that I moved from another host to Google Cloud.
I wanted to know how can I move my data from SQL to Datastore?

Is there an automatic way to do that?
It would be impossible to manually re-create the existing database in Datastore (100's of rows of sql data).

Any tips are appreciated.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/0e445e0f-33bd-458b-8e8d-4d3859e25de0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: The default value about “idle_in_transaction_session_timeout” of PostgreSQL in CloudSQL?

You can set idle_in_transaction_session_timeout on the session level in PostgreSQL 9.6 using SET [1]:

SET SESSION idle_in_transaction_session_timeout = '5min';

If you need this to be set at instance level, I recommend opening a Feature Request on the issue tracker website [2] so that the Cloud SQL Team can look into it as this flag is not implemented for Postgres in Cloud SQL [3]

[1] https://www.postgresql.org/docs/current/sql-set.html
[2] https://issuetracker.google.com/
[3] https://cloud.google.com/sql/docs/postgres/flags#list-flags-postgres

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/dae505d3-390c-4f49-b529-a3c89fb2699a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-cloud-sql-discuss] Re: Migrating from Cloud SQL to Datastore

Hello Sajid,


I have reproduced your code as below with some modification after executing the commands and PHP code outlined here.


$datasetId = $projectId;

$datastore = new DatastoreClient(['projectId' => $projectId]);

function llist($datastore){

$query = $datastore->query()

       ->kind('Task');

   $results = $datastore->runQuery($query);

$entries = [];

$count = 0;

foreach ($results as $entity) {

$count++;

}


echo $count; // this shows me '1' results when I have used Kind('Task') and shows me '0' for Kind('Task1') .

}

llist($datastore);


The kind, 'Task' was defined in the PHP code of the above mentioned link as a result when the modified version of your code was executed it has shown the result 1. When the kind, 'Task' was modified with 'Task1' it has shown the result 0.


From the above observation, it is appearing the kind, 'keypad_research' that you used may not exist or your typed it wrong. In addition the variable $cursor is not defined in your code which I removed during reproduction as it was showing the following error:


PHP Notice:  Undefined variable: cursor in /home/.../php/index.php on line 17


As it was mentioned earlier, If your issue is not a platform issue but rather a problem with how you've configured your code, although you're not sure what it may be specifically, you should post to StackOverflow and community of developers will assist you.



On Tuesday, November 20, 2018 at 12:50:02 AM UTC-6, sajid saiyed wrote:
Thanks Omair for your guidance.

So I tried to first see if my PHP can talk to Datastore and retrieve data.
I created two entities nuder the kind "keypad_research".

This is how my PHP looks like:

<?php
require __DIR__ . '/../../vendor/autoload.php';
use Google\Cloud\Datastore\DatastoreClient;
use Google\Cloud\Datastore\Entity;
$projectId = "__my projectID__";
$datasetId = $projectId;
$datastore = new DatastoreClient(['projectId' => $projectId]);
function llist($datastore){
$query = $datastore->query()
        ->kind('keypad_research')
->start($cursor);

    $results = $datastore->runQuery($query);
$entries = [];
$count = 0;
foreach ($results as $entity) {
$count++;
}

echo $count; // this shows me '0' results even when I have 2 entities.
}
llist($datastore);
?>

Do I need to do any addition configuration or edit my PHP to be able to read from datastore?

I am thinking of writing a PHP script that will first read from SQL database and insert entities in datastore.
But my very basic attempt at reading data from datastore using PHP is failing.

Any advice is appreciated.


On Monday, November 19, 2018 at 9:28:34 AM UTC-8, Omair (Cloud Platform Support) wrote:

Hi Sajid,


Unfortunately there is no automatic way to migrate a Cloud SQL database to Datastore.  The general process for migrating is as follows:


  1. Map your relational model to datastore kinds and entities

  2. [Optional] Download your relational database as csv

  3. Write a script using one of our Cloud Datastore Client Libraries that would transfer your data from Cloud SQL or the csv in step 2 to Datastore

  4. If you have a large database, it would speed up the migration process by hosting your script on App Engine.



If you require assistance in writing your script for migrating relational database to Datastore it is recommended to post your complete question to Stack Overflow using one of our supported tags, as your question will receive a lot more attention there.


I have found a useful post on Stack Overflow for you that I think might help you.



On Sunday, November 18, 2018 at 4:22:23 PM UTC-5, sajid saiyed wrote:
Hi,
I am currently using Cloud SQL but its an over kill for the small application that I moved from another host to Google Cloud.
I wanted to know how can I move my data from SQL to Datastore?

Is there an automatic way to do that?
It would be impossible to manually re-create the existing database in Datastore (100's of rows of sql data).

Any tips are appreciated.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/448dd779-1bae-4943-bf21-98f278074ca6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.