Re: [google-cloud-sql-discuss] import with csv fails

Do you mind sending your project and instance names to cloud-sql@google.com so that we can take a deeper look?

On Fri, Mar 11, 2016 at 12:35 PM Augustin Wenger <wengerboy@gmail.com> wrote:

HI,

I'm using a Second Generation instance.

Mo ang Go  are french for Megabyte and Gigabyte, sorry I'm using them without thinking.

Since last time, I managed to improve performance of loading by increasing temporarily the performance of the instance, which helped a bit

I believe my approach (using huge csv files and uploading them through Cloud Storage one at a time) is actually not optimal :

  - errors are not properly tracked

  - I actually might have to transform some columns in a few tables, so I would need to specify a more precise command

For this time, I will probably split my csv file of chunks of a few dozens Mb and upload them one by one via a custom LOAD DATA LOCAL INFILE which works better and can be automated more easily on my side.

However, I still have the problem of my instance being 'stucked in importation' : since a few hours I cannot connect to my instance and the graph of the storage used is frighteningly discontinuous, as seen in the attached file.

 (I tried to import a 500 Mb file, which was not the biggest I ever managed to import)

 

I looked it up a bit, but I did not see if I can do anything to restart it manually. How long should I wait before I panick ? It has been 4 hours.

On Friday, March 11, 2016 at 5:48:07 AM UTC+1, Vadim Berezniker wrote:

Are you importing into a First Generation or Second Generation instance? 

Sorry for the stupid question, but what unit is "Mo" and "Go"? I am not familiar with that terminology.

One thing you may want to do to improve the speed of large imports is to disable binary logs on the instance prior to the import.

I apologize about the vague error messages, we aim to provide good actionable error messages when things go wrong, but that didn't happen in this case.

On Thu, Mar 10, 2016 at 4:52 PM Augustin Wenger <weng...@gmail.com> wrote:

I'm trying to load several csv files in the range of 100Mo-8Go on Cloud SQL.

I expected the best way to to it would be to put the csv files in CloudStorage, then through an importation in the UI of Cloud SQL

When I tried to do this, after a very long time of "Importing data...", a notification of error appears, with no precision of what caused it, just a red message "Unknown Error"

I tried to split the file in chunks and it is currently loading without error using the command described in the documentation 

LOAD DATA LOCAL INFILE 'myfile-chunk-143.csv' 

 REPLACE INTO TABLE mytable CHARACTER SET 'utf8' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' ESCAPED BY '"'

So my guess it's either  a faulty line in my file  (but how can I detect it if I don't know the error?)

Is there any way for me to access a more precise description of the error that occured ?

Am I using the best way to put this amount of data ? 

also, my menu is now stuck in mode "Importing data..." while the job has failed an hour ago. Should I just wait or is it possible it got stuck ?

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/aa404aaa-af8c-4d34-963f-2a638230ccbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/08d77184-d445-474a-a363-41d5b50140e5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Google Cloud SQL discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-sql-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-sql-discuss/CABDU3p1o5-Hi%3DQYhzgB937-nkOYcbdCwk9xP2YeNA5Cf89uAtQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.