Tuesday, November 15, 2011

Import taking too long?

Folks,

I've tried twice now to run an import for a mysqldump via Cloud
Storage that contains roughly 1 million records (~250MB file).
The .sql file contains the usual - a series of CREATE TABLE and INSERT
() statements. There are 24 tables involved, each with < 5 indexes
and a couple of many-to-many maps in the Django style. I'd love to do
them all at once instead of running 24 separate imports - the DB size
isn't very large at all in the terms that I usually work.

I've attempted to run the import twice, and each time it got cut off
at the 24 hour mark with a generic error (I'm guessing Google cuts off
imports at 24 hours, yes?). At that time, it had completed roughly
100,000 records at a rate of a few per second (I was able to query the
tables with count(*)'s as they imported to check the rate), decreasing
in rate as the tables grew. Any ideas why this was so slow? Each
record is relatively small (think 10 VARCHAR(100) columns and a couple
of dates). I'm close to just giving up on it as impossible to import
at my scale, unless I get clued in on why it's taking so long.

Thanks,

Jesse

No comments:

Post a Comment