Infobright goes open source!

Long time without write in this blog...have been too too busy in my new job.

Anyway since yesterday I'm glad to see that infobright, a data warehousing storage engine for mysql has been released as open source.

You can see the news here .

To the mysqlers that are using this good database for data warehousing this is a pretty good new one. As Robin states in the article when you reach the 1-2 TB in your mysql db you must begin to do lot of tricks to make things works.

Please check the Infobright website! http://www.infobright.org/

Office, visio, project

Seems like when you are in a non technical position you must use these Microsoft tools very often....which is sad because I don't want lose my Linux desktop as the main os I use in the work...

So in these last days I basically tested: vmware and wine.

• Vmware

The thing with vmware...it works fine but you must startup windows every time inside vm...and...the applications aren't integrated into the desktop.

• Wine

The good thing with wine is that windows apps are integrated in the window manager....the bad...make programs works is a pain....Now it seems that word is running but not sure what surprise it will give when I use it with more detail....

Thanks to my friend Tom for the support...

A good new one

I think this post will be quite large...sorry if You get bored with it......
The good one: I will change my job. I resigned yesterday to my current job as an information system analyst in a telco operator. I will move to a big cable and tv company as Traffic and Dwh Director.

First, I have to say I'm quite happy about the change but I have a mix of feelings as is natural. I was engineer there for three years and a half. In that time I made a lot of good friends and workmates. Also I think this three years was a period where I learn a LOT of useful things. In this job I had the opportunity to play with linux, MySQL, tomcat, mondrian, pentaho, kettle, clusters, SANs etc. and while I am not an expert in any of those things I have a good knowledge in each of them.

Sadly things have been going to the wrong side in the area ( caution note: these are personal opinions :) ). Since Millicom bought the company and the managers went off the area has become less and less relevant for the organization. The hope that our developed software and knowledge can be extended to other operations worldwide (a thing that can be possible because Millicom have more than twenty operations around the world) is almost lost. I have to say Millicom doesn't like home development teams so our existance is practically hidden from the headquarters.

Moreover the new management style is very precipitated (the high level management not my direct boss...my direct boss is a very good leader and friend). Today we have a new project, tomorrow is canceled, today we exist, tomorrow not. Things were needed in 4 hours no matters the quality cost and the entropy added to the team.

You would think that my ex job was a bad job. That is not true. As I said i faced a lot of interesting problems, and learn lot of things. Also there are good organizational practices there.

Anyway I'm pretty happy because:

• I will manage the dwh cycle
  
• I will have the opportunity to lead a technical team
• Is a demostration that the good job is rewarded sooner or later
• I will have to show that peter principle is wrong

And many other reasons! so expect technical and managerial posts from now! My desire is not to abandon the technical skills only to use them to lead software projects to the expected results...so my leadership will be from a technical perspective.

And to my good friends in ##pentaho irc channel....I will ask a lot there!...be prepared! :)

Back from holidays

Hey!
Today is my first job day after the holidays and I have to say: I'm out of form.

Anyway...holidays were great, as I told in the last post I went to Valledupar and to Santa Marta. There I take a very good rest. The two weeks I stayed in Bogotá I did this:

Bought a new pc: Intel Quad Core 64 bits, 4GB Ram, 320GB, Nvidia graphics card, etc...This pc will be my home development server.

Did the eye surgery: I no longer use lens!

Played a LOT of squash: Two times a day!

I'm very happy with the holidays I took..I feel they changed my daily routine and now I'm ready to work.

Btw...Colombia is a great place to take holidays!

Holidays!

Is official tomorrow my holidays vacations begin!...I will take 14 work days which are like 3 weeks in calendar days...

The last year and the current has been very heavy in work load terms...The managers of the area and the vice-presidency where I work for were changed. The new managers came with new ideas, strategies, etc. Furthermore we had to implement several SOX controls in the area.

I really feet tired in this year and I think I really need a time to rest...What will I do in the holidays? I'll go a week to Valledupar and the other weeks I will spend some time mounting my home wlan, buying a new desktop computer and practicing squash.

Squash Tournament

I just entered to the squash tournament of the company I work for. Its a very big tournament with 6 extreme players (3 from TI, 3 from Network) :-). We will use English scoring and double yellow dot balls. I

hope to post some interesting reports about the games I will play, hope my not so disciplined drilling practices will serve in the games.

Regarding squash this is another cool website for reading tips and watching videos: squash club

Backup, backup, backup

Perhaps every system administrator/DBA talks about this issue...and this recommendation is based on the common sense: Make valid backups of your data.

We made backups at all level: operating system, user file system directories and the database. Because we use MyISAM and ARCHIVE engines we simply flush the table we want to backup and then copy the files to the tape.

The backup must be an automated labor that must run at different level (full, incremental and differential), and there are many good open source tools for this: we use Bacula with great results.

Why I am talking about this? this last week I must drop a temporary table with so bad luck that I dropped the wrong table (an was an aggregated historic table). Those are the moments when you say...thank goodness I have a backup!

Learning to dump: The hard a way

Remember the problem with ARCHIVE engine in the upgrade from 5.0 to 5.1?

Well...I had to mount a parallel MySQL 5 server and restore those tables from the backup. Now the trick is to do a mysqldump...the thing is...what would be the best way to dump the tables (as the avg number of rows is around 20millon)?

The first approach I tried: a bash script that goes table to table doing:
-mysqldump thedb thetable |mysql -htheotherserver

Problems? The dump is going very slow....moreover I learn something the hard way: Mysqldump by default load all the table in memory before dump it....the result? somewhen in the morning the server ran out of memory and crashed......

This week that comes I will try other approaches to the problem....

Migrating the platform to mysql 5.1

Last week I began the actualization of the server platform (CentOS versión, MySQL version, GFS version) for the dwh I manage.

Everything went quite right except the MySQL upgrade (from 5.0 to 5.1). As you may know MyISAM tables are composed by three files: a frm file that contains information of the table structure, a myd file that contains the data and a myi file that contains the index data. The MyISAM structure didn't change from 5.0 to 5.1 so I just did a copy of the files....but that was not totally true for ARCHIVE engine...

The ARCHIVE engine is a compressed data table which allow less disk usage (you should know disk usage is important for DWH :)) ...the scans are fater because rows are compressed and the server can read more rows from disk (obviously with the penalty of processor time for decompression). I use ARCHIVE for historic and less frequently queried tables...

So what was the problem? MySQL 5.1 crashes on some (not all) ARCHIVE tables....I am working around this problem.....

Mondrian at last

Finally I did something in production using Mondrian. For those who don't know what it is, Mondrian is an open source rolap server written in Java. What is the purpose of OLAP? analyze information or measures across different dimensions. So, if you want to use Mondrian, you should follow the dimensional modeling principles in your db design and then build a XML document that maps dimensions and fact tables into the relational entity model. Here you can see some screenshots of Jpivot+Mondrian (Jpivot is a web UI for mondrian)

I did a couple of cubes in order to calculate the Payback of the BTS (base stations) of the mobile network. The payback is the capex of the bts divided by the cash flow that the station generates. I have two cubes one for traffic and revenue and one for the capex costs, also I have three dimensions: geographic, date and type of coverage. With those two cubes I made a virtual cube for the payback calculated measure.

On the top of the cubes you write the MDX (kinda SQL for Multidimensional analysis) for each report. MDX imho is an awesome expression language that lets you write many differents analysis,

The user's response to this app was great althought Jpivot is a little bit oldie...Anyway if you plan to build a data warehouse using open source software and you'd like to have this nice feature read more deeply about Mondrian.

The guide to squash

Since I moved out to my new apartment I began to learn and practice squash. I discover squash is a very entertain and fully workout game. While my objective is not to be a pro I want to have a good level. What am I doing to improve my game? At the beginning I took some squash lessons. After that I watched the videos and tips from a web site I recommend: The Guide-To-Squash

Here you can find beginner and intermediate drills, moves, hits, tactics, etc. Also you can find game videos from top squash players. Is a pity the creators didn't complete the advanced section.

I recommend to practice the hits and drills two or three times in the week.

Hope you enjoy the link.

The first post

Ok, this is my first post...this blog is intended to write about reporting and data warehousing with open source (primarily Pentaho and Mysql), but I also will write about stocks analysis and maybe squash...although my native language is spanish I decided to write this blog in english to practice....if you want me to write a post regarding the topics above in spanish just contact me.