Infobright goes open source!

Long time without write in this blog...have been too too busy in my new job.

Anyway since yesterday I'm glad to see that infobright, a data warehousing storage engine for mysql has been released as open source.

You can see the news here .

To the mysqlers that are using this good database for data warehousing this is a pretty good new one. As Robin states in the article when you reach the 1-2 TB in your mysql db you must begin to do lot of tricks to make things works.

Please check the Infobright website! http://www.infobright.org/

Learning to dump: The hard a way

Remember the problem with ARCHIVE engine in the upgrade from 5.0 to 5.1?

Well...I had to mount a parallel MySQL 5 server and restore those tables from the backup. Now the trick is to do a mysqldump...the thing is...what would be the best way to dump the tables (as the avg number of rows is around 20millon)?

The first approach I tried: a bash script that goes table to table doing:
-mysqldump thedb thetable |mysql -htheotherserver

Problems? The dump is going very slow....moreover I learn something the hard way: Mysqldump by default load all the table in memory before dump it....the result? somewhen in the morning the server ran out of memory and crashed......

This week that comes I will try other approaches to the problem....

Migrating the platform to mysql 5.1

Last week I began the actualization of the server platform (CentOS versión, MySQL version, GFS version) for the dwh I manage.

Everything went quite right except the MySQL upgrade (from 5.0 to 5.1). As you may know MyISAM tables are composed by three files: a frm file that contains information of the table structure, a myd file that contains the data and a myi file that contains the index data. The MyISAM structure didn't change from 5.0 to 5.1 so I just did a copy of the files....but that was not totally true for ARCHIVE engine...

The ARCHIVE engine is a compressed data table which allow less disk usage (you should know disk usage is important for DWH :)) ...the scans are fater because rows are compressed and the server can read more rows from disk (obviously with the penalty of processor time for decompression). I use ARCHIVE for historic and less frequently queried tables...

So what was the problem? MySQL 5.1 crashes on some (not all) ARCHIVE tables....I am working around this problem.....