Friday, December 1, 2017

[google-cloud-sql-discuss] Re: “cat urls.txt | gsutil -m cp -I gs://target-bucket-name/” consistently hangs after transferring ~10,000 files


Hello Alex,


What is the size of the file, combined?


Are you having the same issue when trying to upload from the Developers Console website interface ?


As the issue occurs especially on large upload jobs, After commenting out "parallel_composite_upload_threshold = 0", could you verify of using Parallel Composite Uploads improves the throughput by using the following command:


$ gsutil -o GSUtil:parallel_composite_upload_threshold=150M cp I  gs://target-bucket-name.


If you still see issue, Please run a perfdiag to the bucket in question to get some diagnostics on what is going on and send me in a private message.


As an alternative to gsutil, you can also absolutely try Cloud Storage Transfer Service.


--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/03058386-c1d9-410a-b9b9-e0f660fd9c6f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

No comments:

Post a Comment