A good new one

I think this post will be quite large...sorry if You get bored with it......
The good one: I will change my job. I resigned yesterday to my current job as an information system analyst in a telco operator. I will move to a big cable and tv company as Traffic and Dwh Director.

First, I have to say I'm quite happy about the change but I have a mix of feelings as is natural. I was engineer there for three years and a half. In that time I made a lot of good friends and workmates. Also I think this three years was a period where I learn a LOT of useful things. In this job I had the opportunity to play with linux, MySQL, tomcat, mondrian, pentaho, kettle, clusters, SANs etc. and while I am not an expert in any of those things I have a good knowledge in each of them.

Sadly things have been going to the wrong side in the area ( caution note: these are personal opinions :) ). Since Millicom bought the company and the managers went off the area has become less and less relevant for the organization. The hope that our developed software and knowledge can be extended to other operations worldwide (a thing that can be possible because Millicom have more than twenty operations around the world) is almost lost. I have to say Millicom doesn't like home development teams so our existance is practically hidden from the headquarters.

Moreover the new management style is very precipitated (the high level management not my direct boss...my direct boss is a very good leader and friend). Today we have a new project, tomorrow is canceled, today we exist, tomorrow not. Things were needed in 4 hours no matters the quality cost and the entropy added to the team.

You would think that my ex job was a bad job. That is not true. As I said i faced a lot of interesting problems, and learn lot of things. Also there are good organizational practices there.

Anyway I'm pretty happy because:

• I will manage the dwh cycle
  
• I will have the opportunity to lead a technical team
• Is a demostration that the good job is rewarded sooner or later
• I will have to show that peter principle is wrong

And many other reasons! so expect technical and managerial posts from now! My desire is not to abandon the technical skills only to use them to lead software projects to the expected results...so my leadership will be from a technical perspective.

And to my good friends in ##pentaho irc channel....I will ask a lot there!...be prepared! :)

Back from holidays

Hey!
Today is my first job day after the holidays and I have to say: I'm out of form.

Anyway...holidays were great, as I told in the last post I went to Valledupar and to Santa Marta. There I take a very good rest. The two weeks I stayed in Bogotá I did this:

Bought a new pc: Intel Quad Core 64 bits, 4GB Ram, 320GB, Nvidia graphics card, etc...This pc will be my home development server.

Did the eye surgery: I no longer use lens!

Played a LOT of squash: Two times a day!

I'm very happy with the holidays I took..I feel they changed my daily routine and now I'm ready to work.

Btw...Colombia is a great place to take holidays!

Holidays!

Is official tomorrow my holidays vacations begin!...I will take 14 work days which are like 3 weeks in calendar days...

The last year and the current has been very heavy in work load terms...The managers of the area and the vice-presidency where I work for were changed. The new managers came with new ideas, strategies, etc. Furthermore we had to implement several SOX controls in the area.

I really feet tired in this year and I think I really need a time to rest...What will I do in the holidays? I'll go a week to Valledupar and the other weeks I will spend some time mounting my home wlan, buying a new desktop computer and practicing squash.

Squash Tournament

I just entered to the squash tournament of the company I work for. Its a very big tournament with 6 extreme players (3 from TI, 3 from Network) :-). We will use English scoring and double yellow dot balls. I

hope to post some interesting reports about the games I will play, hope my not so disciplined drilling practices will serve in the games.

Regarding squash this is another cool website for reading tips and watching videos: squash club

Backup, backup, backup

Perhaps every system administrator/DBA talks about this issue...and this recommendation is based on the common sense: Make valid backups of your data.

We made backups at all level: operating system, user file system directories and the database. Because we use MyISAM and ARCHIVE engines we simply flush the table we want to backup and then copy the files to the tape.

The backup must be an automated labor that must run at different level (full, incremental and differential), and there are many good open source tools for this: we use Bacula with great results.

Why I am talking about this? this last week I must drop a temporary table with so bad luck that I dropped the wrong table (an was an aggregated historic table). Those are the moments when you say...thank goodness I have a backup!

Learning to dump: The hard a way

Remember the problem with ARCHIVE engine in the upgrade from 5.0 to 5.1?

Well...I had to mount a parallel MySQL 5 server and restore those tables from the backup. Now the trick is to do a mysqldump...the thing is...what would be the best way to dump the tables (as the avg number of rows is around 20millon)?

The first approach I tried: a bash script that goes table to table doing:
-mysqldump thedb thetable |mysql -htheotherserver

Problems? The dump is going very slow....moreover I learn something the hard way: Mysqldump by default load all the table in memory before dump it....the result? somewhen in the morning the server ran out of memory and crashed......

This week that comes I will try other approaches to the problem....

Migrating the platform to mysql 5.1

Last week I began the actualization of the server platform (CentOS versión, MySQL version, GFS version) for the dwh I manage.

Everything went quite right except the MySQL upgrade (from 5.0 to 5.1). As you may know MyISAM tables are composed by three files: a frm file that contains information of the table structure, a myd file that contains the data and a myi file that contains the index data. The MyISAM structure didn't change from 5.0 to 5.1 so I just did a copy of the files....but that was not totally true for ARCHIVE engine...

The ARCHIVE engine is a compressed data table which allow less disk usage (you should know disk usage is important for DWH :)) ...the scans are fater because rows are compressed and the server can read more rows from disk (obviously with the penalty of processor time for decompression). I use ARCHIVE for historic and less frequently queried tables...

So what was the problem? MySQL 5.1 crashes on some (not all) ARCHIVE tables....I am working around this problem.....