Thursday, March 24, 2022

[google-cloud-sql-discuss] Use `gcloud sql export csv ... ` for multiple calls at the same time

So I've built my ETL pipeline to move 5 tables from my cloud-sql server to Google Cloud Storage as .csv files at the same time.

However my pipeline fails because the `gcloud sql export csv ... ` command cannot be run while another instance of it is running. Is there a way I can execute these to either run simultaneously?

If not then will I have the same problem with the REST API implementation?

I'd rather not spend time setting up Auth2.0 for the REST API if it's going to have the same problem. See code and error message below.

Thanks in advance!

Harpal

________________

$ gcloud sql export csv my-instance-id gs://bucket-name/sector_cls/automatic/test.csv --database=my-db --query="SELECT * FROM public.my_table"
>>>

ERROR: (gcloud.sql.export.csv) HTTPError 409: Operation failed because another operation was already in progress. Try your request after the current operation is complete.



Moonfire Ventures LLP (FRN: 932750) is authorised and regulated by the Financial Conduct Authority.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/42bebd42-9372-4725-ab2a-acc2cc39e0e9n%40googlegroups.com.

No comments:

Post a Comment