[google-cloud-sql-discuss] Re: Use `gcloud sql export csv ... ` for multiple calls at the same time

There is already a feature request [1] for a Cloud SQL export with multi-threaded process, however do note that feature requests do not have ETA's nor guarantee to be implemented. I would invite you to log a comment on the public issue tracker to indicate that you are also requesting for the feature. 

Additionally, kindly take a look at export is taking too long [2, 3] section as it might be applicable on your use case. 

[1] https://issuetracker.google.com/141981674

[2] https://cloud.google.com/sql/docs/mysql/import-export#troubleshooting-export
[3] https://cloud.google.com/sql/docs/mysql/import-export#serverless

On Monday, March 28, 2022 at 12:56:55 PM UTC-4 har...@moonfire.com wrote:

Thanks for the suggestion @babdelrahman.

It seems as if the orchestration software i'm using is programmatically executing each `gcloud sql export csv ... ` command on separate threads. This is causing massive problems with which operation to `wait` at any given moment. Especially since am not sure if I can easily re/name each operation.

Are you aware of any other tools or methods that are better suited for downloading/exporting these tables in a multithreaded setting? Perhaps something similar to the multithreaded upload command `gsutil cp -m ... ` ?

Best wishes

Harpal

On Friday, 25 March 2022 at 19:32:30 UTC babdelrahman wrote:

You have received a HTTP:409 error because most operations in Cloud SQL are synchronous and you can only run one operation at a time. You can perform only one import or export operation at a time for each instance [1].

Exporting using REST or CLI creates a lock on the sql instance while the exporting process is in progress.

To fix this, you need to list all the operations running on the instance, and then use 'gcloud sql operations wait' on all the other operations [2].

[1]https://cloud.google.com/sql/docs/mysql/import-export#reduce_long-running_import_and_export_processes

[2] https://cloud.google.com/sdk/gcloud/reference/sql/operations/wait

On Thursday, March 24, 2022 at 9:36:21 AM UTC-4 har...@moonfire.com wrote:

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/5a1faa44-0526-4abc-a62f-8da0a9590083n%40googlegroups.com.

[google-cloud-sql-discuss] Re: Use `gcloud sql export csv ... ` for multiple calls at the same time

Thanks for the suggestion @babdelrahman.

It seems as if the orchestration software i'm using is programmatically executing each `gcloud sql export csv ... ` command on separate threads. This is causing massive problems with which operation to `wait` at any given moment. Especially since am not sure if I can easily re/name each operation.

Are you aware of any other tools or methods that are better suited for downloading/exporting these tables in a multithreaded setting? Perhaps something similar to the multithreaded upload command `gsutil cp -m ... ` ?

Best wishes

Harpal

On Friday, 25 March 2022 at 19:32:30 UTC babdelrahman wrote:

You have received a HTTP:409 error because most operations in Cloud SQL are synchronous and you can only run one operation at a time. You can perform only one import or export operation at a time for each instance [1].

Exporting using REST or CLI creates a lock on the sql instance while the exporting process is in progress.

To fix this, you need to list all the operations running on the instance, and then use 'gcloud sql operations wait' on all the other operations [2].

[1]https://cloud.google.com/sql/docs/mysql/import-export#reduce_long-running_import_and_export_processes

[2] https://cloud.google.com/sdk/gcloud/reference/sql/operations/wait

On Thursday, March 24, 2022 at 9:36:21 AM UTC-4 har...@moonfire.com wrote:

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/e29aed16-0ac5-474d-a7fd-078b080275efn%40googlegroups.com.

[google-cloud-sql-discuss] Re: Use `gcloud sql export csv ... ` for multiple calls at the same time

You have received a HTTP:409 error because most operations in Cloud SQL are synchronous and you can only run one operation at a time. You can perform only one import or export operation at a time for each instance [1].

Exporting using REST or CLI creates a lock on the sql instance while the exporting process is in progress.

To fix this, you need to list all the operations running on the instance, and then use 'gcloud sql operations wait' on all the other operations [2].

[1]https://cloud.google.com/sql/docs/mysql/import-export#reduce_long-running_import_and_export_processes

[2] https://cloud.google.com/sdk/gcloud/reference/sql/operations/wait

On Thursday, March 24, 2022 at 9:36:21 AM UTC-4 har...@moonfire.com wrote:

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/50f0ff21-e1bd-4401-b018-7246da853494n%40googlegroups.com.

[google-cloud-sql-discuss] Use `gcloud sql export csv ... ` for multiple calls at the same time

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/42bebd42-9372-4725-ab2a-acc2cc39e0e9n%40googlegroups.com.

[google-cloud-sql-discuss] Re: Connection to On Premise SQL Server

This documentation from Google Cloud could help you to achieve what you want to do:

"This tutorial shows you how to create and deploy a pipeline that continuously replicates changed data from a Microsoft SQL Server database to a BigQuery table."

This other page shows another alternative to connect SQL Server with BigQuery.

But consider that the tutorial from Google Cloud relies on the CDC (Change Data Capture) feature, which

"is supported in the Developer and Enterprise Editions of SQL Server up to and including SQL Server 2016 RTM."

As stated on Microsoft's site; MS SQL Server 2014, which is the one that you're using, has reached its End of Life, and is beyond support.

On Monday, March 21, 2022 at 9:22:49 AM UTC-6 da...@nutracapusa.com wrote:

 We have a SQL Server 2014 Edition on premise database for an application internal to the company  and would like to access its data from Bigquery, so we can run BI tasks using it.

 I've been struggling with this problem for some time and would like some input from other people - I come from an Analytics background and never had to deal with this sort of thing prior to leading this project. Also, I'd like to point out that we would prefer not to use any pre-built data connectors third-party services, if possible.

 Given the context, I basically have two questions:

•  I know that Bigquery can access Cloud SQL data using federated queries, so creating a SQL Server instance in Cloud SQL and somehow connecting it to our on premise db is an option - although I might be completely out of range here, since I'm not sure if that's the best option. If this is a good option, how exactly should I build this connection (which kind of services should I use and documentation should I read)? Any network configurations necessary  in the on premise db server  can be carried out by our internal specialists.
• If the previous item seems absurd to you or if you simply see an easier/cheaper option, how would that go?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/2f32857e-31d8-4902-be88-0da47976d6c2n%40googlegroups.com.

[google-cloud-sql-discuss] Connection to On Premise SQL Server

 We have a SQL Server 2014 Edition on premise database for an application internal to the company  and would like to access its data from Bigquery, so we can run BI tasks using it.

 I've been struggling with this problem for some time and would like some input from other people - I come from an Analytics background and never had to deal with this sort of thing prior to leading this project. Also, I'd like to point out that we would prefer not to use any pre-built data connectors third-party services, if possible.

 Given the context, I basically have two questions:

•  I know that Bigquery can access Cloud SQL data using federated queries, so creating a SQL Server instance in Cloud SQL and somehow connecting it to our on premise db is an option - although I might be completely out of range here, since I'm not sure if that's the best option. If this is a good option, how exactly should I build this connection (which kind of services should I use and documentation should I read)? Any network configurations necessary  in the on premise db server  can be carried out by our internal specialists.
• If the previous item seems absurd to you or if you simply see an easier/cheaper option, how would that go?

Thanks in advance

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/25e49b3b-d4e8-4cf2-97ed-936a4ffecf03n%40googlegroups.com.

[google-cloud-sql-discuss] Re: where is the cloudsql.googleapis.com/mysql-general.log

Please be sure your database has the correct flags configuration. In this page you will find how to see if the flags were applied correctly by checking the Current values of the database flags and Determine what database flags have been set for an instance. Once confirming the flags are correct, you might want to check the Storing, viewing, and managing logs.

I also want to mention that the flag `log_out` should be `log_output`. Please check all the configuration is done properly and correct the typo and provide more information according to this documentation, please.

On Thursday, March 17, 2022 at 10:14:29 AM UTC-6 zhaox...@gmail.com wrote:

Hi,

We have configured the cloud SQL instance tag 

log_out FILE

slow_query_log on

general_log on

It is promised in this [document](https://cloud.google.com/sql/docs/mysql/logging) that there should be a file "cloudsql.googleapis.com/mysql-general.log" that could be found in the log explorer that should record all queries, but we can't find it!

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/36b4843d-557e-4e23-aff0-69f0a403502dn%40googlegroups.com.

[google-cloud-sql-discuss] where is the cloudsql.googleapis.com/mysql-general.log

Hi,

We have configured the cloud SQL instance tag 

log_out FILE

slow_query_log on

general_log on

It is promised in this [document](https://cloud.google.com/sql/docs/mysql/logging) that there should be a file "cloudsql.googleapis.com/mysql-general.log" that could be found in the log explorer that should record all queries, but we can't find it!

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/7d11a210-31ec-466d-a1b9-b954e286f5f0n%40googlegroups.com.

Re: [google-cloud-sql-discuss] Integration CloudPostgreSQL to PowerBi.

Thanks, @bala @peter @scott I will try to implement my setup with the above suggestions.
Thanks for the inputs :-) will ping here if required more clarification

On Tue, Mar 15, 2022 at 8:20 PM Scott Ertel <jsecpa@gmail.com> wrote:

Hello, 

An option you could look at is creating a windows VM in google cloud and install the enterprise gateway connector on it. Then you would need to install the Postgres connector on that machine. This will allow the powerBI service to reach the database. 

I'm assuming connectivity is already in place and  users can already connect to the database in your internal network. 

Sent from my iPhone

On Mar 15, 2022, at 2:26 AM, Sujit Ramugade <sujit.ramugade@searce.com> wrote:

Hi Team,

We have a requirement to connect Cloud PostgreSQL w/ PowerBI for reporting needs. Could you please suggest what is the best possible solution here?

Since there is no direct/easy connectivity - team is taking the reporting data out of cloud (postgre) sql --> moving it to SQL express --> connecting with PowerBI. Is there any better solution?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f25bee9e-bb48-46cf-b166-d499985b065bn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/D5F4D41E-B3EA-4EB7-A968-DB2E6A4E570C%40gmail.com.

--

Sujit Ramugade CloudOps - DBA M:+91-7387107873 www.searce.com solving for better

 

The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. Searce is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/CABwOSmzho%3DdeEYdRA44D4vrG%3DyhX6bMS4y2rOXvzvdU0MK4UPg%40mail.gmail.com.

Re: [google-cloud-sql-discuss] Integration CloudPostgreSQL to PowerBi.

Hello, 

An option you could look at is creating a windows VM in google cloud and install the enterprise gateway connector on it. Then you would need to install the Postgres connector on that machine. This will allow the powerBI service to reach the database. 

I'm assuming connectivity is already in place and  users can already connect to the database in your internal network. 

Sent from my iPhone

On Mar 15, 2022, at 2:26 AM, Sujit Ramugade <sujit.ramugade@searce.com> wrote:

Hi Team,

We have a requirement to connect Cloud PostgreSQL w/ PowerBI for reporting needs. Could you please suggest what is the best possible solution here?

Since there is no direct/easy connectivity - team is taking the reporting data out of cloud (postgre) sql --> moving it to SQL express --> connecting with PowerBI. Is there any better solution?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f25bee9e-bb48-46cf-b166-d499985b065bn%40googlegroups.com.

Re: [google-cloud-sql-discuss] Integration CloudPostgreSQL to PowerBi.

Hi Sujit,

Since you're a bit sparse on your environment in which you use PowerBI, it's hard to answer, but I would suggest looking into running Cloud SQL Auth Proxy locally (on server or network) to your PowerBI environment. 

https://cloud.google.com/sql/docs/postgres/sql-proxy

Thx

Peter

On Tue, Mar 15, 2022 at 1:26 AM Sujit Ramugade <sujit.ramugade@searce.com> wrote:

Hi Team,

We have a requirement to connect Cloud PostgreSQL w/ PowerBI for reporting needs. Could you please suggest what is the best possible solution here?

Since there is no direct/easy connectivity - team is taking the reporting data out of cloud (postgre) sql --> moving it to SQL express --> connecting with PowerBI. Is there any better solution?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f25bee9e-bb48-46cf-b166-d499985b065bn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/CAC%2B%3DFfbKkdfULg3%3DZW8J741sLhC5d7NWaB94dUvyTJopwyBVSw%40mail.gmail.com.

[google-cloud-sql-discuss] Integration CloudPostgreSQL to PowerBi.

Hi Team,

We have a requirement to connect Cloud PostgreSQL w/ PowerBI for reporting needs. Could you please suggest what is the best possible solution here?

Since there is no direct/easy connectivity - team is taking the reporting data out of cloud (postgre) sql --> moving it to SQL express --> connecting with PowerBI. Is there any better solution?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/f25bee9e-bb48-46cf-b166-d499985b065bn%40googlegroups.com.

Re: [google-cloud-sql-discuss] Re: Size of Cloudsql Instance reduced from 70 GB to 51GB

Hi David.

Thanks for your reply. Yes I am talking about used storage. You may be right. Google update activity may purge binlogs. How can we verify it? Can we access the instance?

On Sat, Mar 12, 2022 at 1:15 AM 'David (Cloud Platform Support)' via Google Cloud SQL discuss <google-cloud-sql-discuss@googlegroups.com> wrote:

Hello,

I'm not sure whether you are referring to used storage or total available storage. However, if it's used storage, this may be a case where the bin logs were purged. However, to know for a fact, this issue requires someone from the support team to inspect your instance. I would recommend you to contact GCP support as they can surely provide you with information about this storage reduction you are seeing on your instance.

On Friday, March 11, 2022 at 8:02:32 AM UTC-5 masif...@gmail.com wrote:

Hi Geeks,

 We are using cloudsql instance which size gradually increased to 70 GB. Yesterday morning it was 70 GB. But today I notice that it is reduced to 51GB. Please guide me what may be the cause?

Yesterday there was some update activity happen on our DB not by us but by Google. 

What this updation activity can do. May be some other reasons. Please guide.

Regards 

M.Asif

--
You received this message because you are subscribed to a topic in the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-cloud-sql-discuss/Rlm3Z2xDGBg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/387978c3-2f08-4196-9c84-738fd66b29abn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/CADovsB%3DpaQb9EcLGmTyZxKeBGgnbvFmqxrsWRDxGWZOCSEK4Dw%40mail.gmail.com.

[google-cloud-sql-discuss] Re: Size of Cloudsql Instance reduced from 70 GB to 51GB

Hello,

I'm not sure whether you are referring to used storage or total available storage. However, if it's used storage, this may be a case where the bin logs were purged. However, to know for a fact, this issue requires someone from the support team to inspect your instance. I would recommend you to contact GCP support as they can surely provide you with information about this storage reduction you are seeing on your instance.

On Friday, March 11, 2022 at 8:02:32 AM UTC-5 masif...@gmail.com wrote:

Hi Geeks,

 We are using cloudsql instance which size gradually increased to 70 GB. Yesterday morning it was 70 GB. But today I notice that it is reduced to 51GB. Please guide me what may be the cause?

Yesterday there was some update activity happen on our DB not by us but by Google. 

What this updation activity can do. May be some other reasons. Please guide.

Regards 

M.Asif

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/387978c3-2f08-4196-9c84-738fd66b29abn%40googlegroups.com.

[google-cloud-sql-discuss] Size of Cloudsql Instance reduced from 70 GB to 51GB

Hi Geeks,

 We are using cloudsql instance which size gradually increased to 70 GB. Yesterday morning it was 70 GB. But today I notice that it is reduced to 51GB. Please guide me what may be the cause?

Yesterday there was some update activity happen on our DB not by us but by Google. 

What this updation activity can do. May be some other reasons. Please guide.

Regards 

M.Asif

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/93088467-3555-451c-bd6e-ddce1ed9fb9fn%40googlegroups.com.

[google-cloud-sql-discuss] Re: Data engineer

You may check the Exam Guide and the FAQs with preparation advice. There are various books available, for instance this O'Reilly one. Towards the end of your studies, you may check various books with collections of exam questions, to get a feeling of how the exam would look like. 

On Sunday, 06 March 2022 at 13:13:21 UTC-5 abd.al...@gmail.com wrote:

How can I train google cloud data engineer in the easiest way

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/1357e6f9-aba6-441e-a3d8-e677cbc29ec3n%40googlegroups.com.

[google-cloud-sql-discuss] Data engineer

How can I train google cloud data engineer in the easiest way

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/75f7b942-ebbe-426f-8f9e-90e1ef52a59en%40googlegroups.com.

[google-cloud-sql-discuss] Re: Charged when instance is stopped - test sql instance

A stopped instance does not incur charges, but all the resources that are attached to the instance will still be charged. 

For example, you are charged for persistent disks and external IP addresses according to the price sheet, even if an instance is stopped. 

To stop being charged for the attached resources, you can reconfigure a stopped instance to not use those resources, and then delete the resources.

Here is some documentation that could help you:

• Stop and start a VM: https://cloud.google.com/compute/docs/instances/stop-start-instance

• gcloud alpha compute instances suspend: https://cloud.google.com/sdk/gcloud/reference/alpha/compute/instances/suspend

• Suspending and resuming an instance: https://cloud.google.com/compute/docs/instances/suspend-resume-instance

• Save money by stopping and starting Compute Engine instances on schedule: https://cloud.google.com/blog/products/storage-data-transfer/save-money-by-stopping-and-starting-compute-engine-instances-on-schedule

On Wednesday, March 2, 2022 at 10:38:24 AM UTC-6 atree...@gmail.com wrote:

Hi, I have a PostgreSQL 13 database instance, its a 'test' instance with the lowest spec I could find (end of msg for spec).  The pricing calculator estimated about £6.50 per month I thought.

I created the new instance on the 2/2/22, created maybe three tables and added maybe 4 or five rows of data in each table (so almost nothing in there).  I then stopped the instance on the 3/2/22.  I used it for less than 48 hours.

Looking at the charts section in the console and apart from those two days, no activity has occurred; active connections, storage use, total memory use, CPU utilisation, etc all show usage for the two days I had the database running but blank since the 3/2/22 when I stopped the instance.

I've read a number of places that say that I shouldn't be charged when my database instance is stopped.  However, I've been charged what seems to be the full £6.50 (its actually £6.49) and for March I'm forecast the same.  Why am I still being charged?  If I don't want to be charged do I need to delete the instance every time I'm not using it?

Configuration:

vCPUs: 1, Memory: 614.4 MB, HDD storage: 10 GB

Database version is PostgreSQL 13.4
Auto storage increase is disabled
Automated backups are disabled
Point-in-time recovery is disabled
Located in europe-west1-d
Not highly available (zonal)
No database flags set
No labels set

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/fbd46522-de11-4de8-9926-32cfdd892b91n%40googlegroups.com.

[google-cloud-sql-discuss] Charged when instance is stopped - test sql instance

Hi, I have a PostgreSQL 13 database instance, its a 'test' instance with the lowest spec I could find (end of msg for spec).  The pricing calculator estimated about £6.50 per month I thought.

I created the new instance on the 2/2/22, created maybe three tables and added maybe 4 or five rows of data in each table (so almost nothing in there).  I then stopped the instance on the 3/2/22.  I used it for less than 48 hours.

Looking at the charts section in the console and apart from those two days, no activity has occurred; active connections, storage use, total memory use, CPU utilisation, etc all show usage for the two days I had the database running but blank since the 3/2/22 when I stopped the instance.

I've read a number of places that say that I shouldn't be charged when my database instance is stopped.  However, I've been charged what seems to be the full £6.50 (its actually £6.49) and for March I'm forecast the same.  Why am I still being charged?  If I don't want to be charged do I need to delete the instance every time I'm not using it?

Configuration:

vCPUs: 1, Memory: 614.4 MB, HDD storage: 10 GB

Database version is PostgreSQL 13.4
Auto storage increase is disabled
Automated backups are disabled
Point-in-time recovery is disabled
Located in europe-west1-d
Not highly available (zonal)
No database flags set
No labels set

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/5b4402c7-ed74-4935-99a8-bddad6a0862bn%40googlegroups.com.

[google-cloud-sql-discuss] Re: Full table scan perform by spanner database

Hi,

I found the following Official Documentation where it is explained that the scenario where a query needs to look up all the rows in a table is one of the worst cases, so in my understanding it is a practice to try to avoid and not recommended. 

Regarding your concern about how to avoid it at a query level I found this Best Practices documentation where it is explained that one way to avoid this full table scan from happening is by defining a secondary index.

I hope it helps.

On Monday, February 28, 2022 at 9:52:27 AM UTC+1 senth...@gmail.com wrote:

We are using a query  which has multiple left outer joins which perform full table scan while running in google cloud spanner database. In which scenarios usually it performs full table scan and how to avoid it in query level?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/3d005d57-c46a-4e95-aea0-35e74c47d865n%40googlegroups.com.