[google-cloud-sql-discuss] Re: Use `gcloud sql export csv ... ` for multiple calls at the same time

You have received a HTTP:409 error because most operations in Cloud SQL are synchronous and you can only run one operation at a time. You can perform only one import or export operation at a time for each instance [1].

Exporting using REST or CLI creates a lock on the sql instance while the exporting process is in progress.

To fix this, you need to list all the operations running on the instance, and then use 'gcloud sql operations wait' on all the other operations [2].

[1]https://cloud.google.com/sql/docs/mysql/import-export#reduce_long-running_import_and_export_processes

[2] https://cloud.google.com/sdk/gcloud/reference/sql/operations/wait

On Thursday, March 24, 2022 at 9:36:21 AM UTC-4 har...@moonfire.com wrote:

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.

Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/50f0ff21-e1bd-4401-b018-7246da853494n%40googlegroups.com.