The answer was indeed this, the problem was reading the database table
results = db.query( 'select * from mytable', :stream => true, :cache_rows => false )
The writing of the csv was done asynchronously
質問
I export the following from a mysql database using a little ruby script and the mysql2 gem
require 'csv'
require 'mysql2'
begin
db = Mysql2::Client.new( :host => ".", :username => "xyz", :password => 'xyz', :database => 'xyz')
results = db.query 'select * from mytable'
CSV.open("C:/tmp/mytable.csv", "w") do |csv|
csv << results.fields
results.each do |row|
csv << row.values
end
end
end
This works fine for small tables but I think I should be doing things differently for big tables. I could be wrong here but it seems to read the entire table out to memory and then write it all in one go. If i've got a huge table with tens of millions of rows I'd like to stream to the file constantly.
How would I go about doing that?
I tried running this against a table with ~10 million rows and got the following
[FATAL] failed to allocate memory
I've found this, will report back if it works
result = client.query("SELECT * FROM really_big_Table", :stream => true)
解決
The answer was indeed this, the problem was reading the database table
results = db.query( 'select * from mytable', :stream => true, :cache_rows => false )
The writing of the csv was done asynchronously
他のヒント
This isn't perhaps the Ruby solution you are looking for, but it is one solution:
MySQL supports direct output to a CSV file. In your case if all you need is a CSV dump, then the most efficient way is probably to run a query like this:
SELECT * FROM mytable
INTO OUTFILE 'C:\\tmp\\mytable.csv'
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
This works if the MySQL server is on the same computer as the output file, since it will be the server that writes the file, not the client.
http://www.tech-recipes.com/rx/1475/save-mysql-query-results-into-a-text-or-csv-file/