Andres Chaves blog

Software Engineering for Software as a Service - Statement

2012-04-11T09:02:00.000-07:00

If you want to have an idea of how is the statement here is a picture of it:

Software Engineering for Software as a Service

2012-04-11T08:56:00.000-07:00

Hi,
I had the experience to take the Software Engineering for Software as a Service course offered by professor Armando Fox and David Patterson from University of Berkeley through the coursera startup.

The course used the same material, quizzes and videos from the official Berkeley class. And at then end deliver an statement of accomplishment. I'll try to upload it later.

In general terms I think the experience was great and I am very grateful with Professors Fox and Patterson. There will be a new offering of the course if you are interested. The url of the course is http://saas-class.org

The topics that the course covers are Architectural patterns, software design, code coverage, unit and integration tests, agile development and Ruby. If you like those topics you'll have a great time.

I have to say I liked the Ruby language and the rails framework. In a non expert and scientific perception I think it is more suitable for most of the projects than Java in terms of speed and velocity of development and value added to enterprise software.

Tutorial CDE

2011-07-21T09:08:00.000-07:00

Inteligencia de Negocio y Pentaho: Cómo hacer cuadros de mando: V: Un tutorial recomendado para manejar las excelentes herramientas que Pedro Alves y Webdetails han desarrollado, en especial CDE.

Your faithful employee

2011-07-18T07:42:00.000-07:00

Hello,

In the area I work for, we have one duty among several others more: Maximize the availability of the network management systems. The NMSs must run almost every time because the NOCs (network operation centers) monitor them 7x24.

But thats not an easy task, sometimes the planning area delivers the NMS implementation with many flaws, other times the machine you get is not what you expected or the application servers freezes continously and research the root cause can take weeks.

As we're not big fan of attending service disruption calls at 3am, we deployed a nice and useful service in all our Linux machines: Monit. This nice program monitor the service existence,availability and performance and take automated actions when the rules/thresholds are exceeded.

For example, we had a Tomcat container that was getting frozen several times at the week...the rule for Monit was something like this:

check process tomcat5 with pidfile /var/run/tomcat5.pid
group tomcat5
start program = "/etc/init.d/tomcat5 start" with timeout 120 seconds
stop  program = "/etc/init.d/tomcat5 stop" with timeout 120 seconds
if failed host 127.0.0.1 port 8080 
protocol HTTP request /archivos/gestion.jpg 
TIMEOUT 3 SECONDS then restart
if cpu usage > 95% for 10 cycles then restart
if 5 restarts within 5 cycles then timeout

So, this loyalty automated employee check the port 8080, check the HTTP request of gestion.jpg to be less than 3 seconds and check the cpu usage of the process to be under 95%. If Monit sees any of these rules broken then it begins to restart the service. As the good employee he is, Monit sends email notification of every step he takes.

Hope this application be useful for you,

Quick update: Even thought Monit is doing a great job is important to find the root cause. Regarding the tomcat issue i found this useful site to tune the JVM parameters: http://wiki.alfresco.com/wiki/JVM_Tuning

pentaho CDE CDF CDA CD* tutorials

2011-03-30T07:27:00.000-07:00

Quick post to help people trying to make stuff on those great tools developed by webdetails.pt . They are great but have a lack of docs, so this two links would help:

http://www.vinzi.nl/media/CDE-introduction_V0_8.pdf
http://www.tikalk.com/incubator/blog/creating-bugzilla-dashboard-%E2%80%93-hands-cde-tutorial-%E2%80%93-fuse-day-3-session-summary

I got those links in IRC ##pentaho channel on freenode btw

PD: If you want to install CDE in a context path different to /pentaho/, keep in mind that uri is hardcoded in all js and templates files of the pentaho-cde-dd directory

Messing with Roo

2011-03-10T17:27:00.000-08:00

In the last Google I/O was anounced that GWT 2 will have support in Spring Roo. I never heard of Roo before and I wanted to take look (http://www.springsource.org/roo).

It seems that Roo improves the development by maintaining all the interfaces, stubs and glue code that is boring when you use GWT or Entities.

I just ran two of the demos...a classic Spring Controller Roo Project with handling the CRUD of a Concat entity (http://s3.springsource.com/MRKT/roo/2010-01-Five_Minutes_Roo.mov):

And a GWT integration handling the CRUD of an equipment (http://www.thescreencast.com/2010/05/how-to-gwt-roo.html)

So far it was pretty impressive to make an entity based application in 5 minutes without all the glue code, but as this is a new concept I wonder if anyone has done any real large sized project supported in Roo....

Render a Prpt report correctly in a Pentaho's BI Platform xaction

2011-02-17T05:42:00.000-08:00

Hi, a quick hint about writing a Xaction who must render a Prpr Report. If you try to use the xaction writing plugin for Eclipse, the xaction probably wont work. Here is the correct xaction, and you should overwrite the xaction with this in Eclipse (changing the prpt name of course):

<?xml version="1.0" encoding="UTF-8"?>
<action-sequence>
<title>Test report in PDF format</title>
<version>1</version>
<logging-level>TRACE</logging-level>
<documentation>
    <author>Emmanuel Guiton</author>
    <description>Generates a PDF test report.</description>
    <help/>
    <result-type/>
    <icon/>
</documentation>
<inputs/>
<outputs>
    <test_report type="content">
      <destinations>
        <response>content</response>
      </destinations>
    </test_report>
</outputs>
<resources>
    <report-definition>
      <solution-file>
        <location>test-report.prpt</location>
        <mime-type>application/zip</mime-type>
      </solution-file>
    </report-definition>
</resources>

<actions>
    <action-definition>
      <component-name>SimpleReportingComponent</component-name>
      <action-type>Test report</action-type>
      <action-resources>
        <report-definition type="resource"/>
      </action-resources>
      <action-outputs>
        <outputstream type="content" mapping="test_report"/>
      </action-outputs>
      <component-definition>
        <outputType><![CDATA[application/pdf]]></outputType>
        <useContentRepository><![CDATA[true]]></useContentRepository>
        <paginate><![CDATA[false]]></paginate>
      </component-definition>
    </action-definition>

</actions>
</action-sequence>

Hope this help.

Latest CDF - script.js minified

2011-01-27T13:15:00.000-08:00

Hi,
Just a quick post about the latest CDE-CDF etc that the guys of webdetails have been developing. As In the previous version, distributed in Pentaho platform, I had to tweak a little the javascript of the Map component, I wanted to do the same in this version.

You'll notice a new js file called script.js. That's a minified js generated when the server is started. If you need to twek the js and test you'll have to add this to the dashboard's url: &debug=true&bypassCache=true

Hope this helps anyone else

Aberrant Behavior Detection

2010-12-03T14:35:00.000-08:00

As I wrote before, most of the time of this year I worked in Network Management Systems. Managing a huge, constantly growing, network is a heavy task. One part of this task is measurement collection, and for this task we use Cacti, which a powerful visualization and rrd graphics template tool.

Cacti as others open source measurement collections systems rely the data storing and charting in the RRDTool software.

In our Cacti instances we are collecting measurements from 15000 different router/cmts interfaces, which includes common measures like bandwidth, cpu, memory and others not so common like SNR, Cable Modems in the upstream, modulation profile.

As the NOC is unable to watch and control this amount of charts, there is a nice functionality, not widely used, in the rrds: Aberrant Behavior Detection (ABD). ABD adds a set of measurements and algorithms that adapt to the seasonal trend of the chart and detects when a current value is too deviated from the expected value. You can read about the mathematical details here and here.

You can see some examples here:

The classic bandwidth measurement

Cable Modems in the upstream channel

You can write a short Perl script to check if there is an aberrant behavior in the rrd and do something about about it (for example, send a trap or a syslog with the alert). This way you add a great feature to the network management tasks.

Back to Pentaho again

2010-11-26T11:55:00.000-08:00

So, this year I have been spending most of my time digging in Network Management Systems both open and closed source. Think on Net-SNMP, SNMPTT, NAGIOS, CENTREON and you will get an idea. I promise i'll post more about this because is such an interesting topic.

Lately I have been with free time so I went back to Pentaho and discovered huge advances in the community ecosystem. Specially Pedro with CDF and Tom with Pat (erm now called Haiku).

As my boss wanted to show the quality status of our customers in a map, what I did was to use CDF with openlayers and open street maps. With some adjustments to the CDF's js libraries (the marker layer) I changed the presentation to render a circle with a color scale and whose size depends on the number of clients in the xy position. Here is the result:

Isn't open source software nice? :-)

User Defined Java Class (UDJC) step

2010-01-28T08:04:00.000-08:00

Hey!, I saw the flash recording demo of the new plugin developed by Daniel Einspanjer, User Defined Java Class UDJC and the performance improvement of the Kettle PDI Transformations are awesome. Now Matt Casters is including it in the source tree of PDI 4.

Click here to see Matt's post.
Click here to see the awesome demo.

Coming from vacations

2010-01-12T06:58:00.000-08:00

I just came back from a 3 weeks vacations which I enjoyed so much because I havent taken a rest since a year and a half. As is usual I went to Valledupar, nice Caribbean city of Colombia, and stayed three days at Parque Tayrona.

I totally recommend go to Parque Tayrona, very nice beaches and weather and relativelly cheap acommodations.

We slept in hammocks in a site called Arrefices. We went to many beaches to rest and also a indigenous ruins called pueblito indigena which is around 2 hours walking from the beach.

I rested a lot and got back very motivated, I feel I really needed the vacations to acomplish the year's objectives.

Pentaho Reporting 3.5 for Java Developers - Review

2009-12-21T15:09:00.000-08:00

So, I've finally finished the book Pentaho Reporting 3.5 for Java Developers. Thanks to Richard Dias from Packet Publishing for the opportunitty and apologies for writing this so late.

So, as I told in the last pentaho reporting book post this book is really good if you need a faster learning curve or if you need advanced reporting capabilities like subreporting, crosstabs, complex functions, styling, etc; but is totally needed if you plan to extend the capabilities of the reporting engine such as Swing or JS events, weird data sources, new report elements or direct interaction trough the reporting API.

So the first chapter An Introduction to Pentaho Reporting , gives you a nice introduction to the reporting software as well the architecture of the product.

Chapter two, Pentaho Reporting Client and Enterprise Examples ,show you how to build basic reports and also how to deploy them into a Swing application or Java web container.

Chapter three, Pentaho Reporting Examples in the Real World, chapter six, Including Charts and Graphics in Reports, and chapter seven, Parameterization, Functions, Formulas, and Internationalization in Reports are the parts I recommend if you are interested in building complex and nice reports (you know, with images, sub reports, well suited to be exported to Excel, with groups, formulas, charts, etc). If you follow the samples in each chapter you'll become almost an advanced report builder.

The fourth chapter, Design and Layout in Pentaho's Report Designer explains the detail of every option in the Report Designer: menu, toolbars, bands, report elements, shortcuts, etc. While this chapter can be a little bit boring I recommend to give it an overview reading cause it has some tips inline while is explaining every detail of the items.

Fifth chapter, Working with Data Sources gives you detailed information of the existing data sources who can feed the report and also how to build custom data sources. If all your reports will be from a database you can skip the chapter.

Chapter eight, Adding Sub-Reports and Cross Tabs to Reports gives you a little more insight in subreporting and also how to construct reports from MDX queries (OLAP). If you plan to make reports who need a pivot table layout you need to read the chapter.

Chapter nine, Building Interactive Reports explains how to use events in your reports both in JS (click events, hyperlinks, DOM, Jquery) or in Swing. Nice chapter if you need to deploy reports with high interaction or very dynamic from the UI perspective.

Chapter ten, API-based Report Generation, and eleven, Extending Pentaho Reporting, gives you the developer perspective. Things like how to generate a report on the fly, custom expression/functions, new report elements (and how to show them in the designer) and bean shell expressions are covered in those two chapters. This chapters are recommended for developers who plan to extend the nice features of the designer or need to do something very weird.

Finally chapter eleven, Additional Pentaho Reporting Topics show other issues like how to embed reports in the Pentaho BI Engine, publish to mobile browsers and how to ask for help in the Pentaho Communitty.

Well I hope this review to be useful for you, next post will be the same review but in Spanish

A status update

2009-12-04T05:56:00.001-08:00

So, I totally forgot my blog until the people from Packt Publishing asked me for the reviewing of Pentaho Reporting book. Now I want to speak about what I have been doing this year.

While I continued with the KPI reporting and information systems the objective of the area I'm managing became broader: Now I'm in charge of the network management systems.

We have several of these systems most of them are propietary and focused in one technology or subset of the HFC and Core networks. The really interesting system is a generic one that We built using open source tools: CACTI for high scale network SNMP measurement collection, NAGIOS/CENTREON/NAGVIS for active ICMP/service checking, and a combination of NET-SNMP, SNMPTT and SYSLOG-NG.

While it is difficult to integrate and manage, its a very good solution to scale and manage the events and traps of all the network with a low cost (the same engineer you would dedicate to a propietary solution plus the hardware).

Also I dedicated a lot of effort customizing TRAC for managing all the requirements to the area. I've found in TRAC a useful tool but with a lack of good reporting modules.

So thats the summary of the year...while I still follow pentaho and databases with passion also I have learnt about SNMP, monitoring, traps.

Pentaho Reporting 3.5 for Java Developers

2009-11-05T10:39:00.001-08:00

So, it seems I can't get used to write in my blog often. Lately I've been working with monitoring tools (cacti, nagios, snmptt, etc), so I have left BI and pentaho a little bit aside.

A couple of weeks ago I had the oportunity to review the Pentaho Reporting for Java Developers book, so It reminds me about the cool open source BI platform again.

So far, I've read half of the book and all the insides of this nice reporting tool are very well explained. I haven't used pentaho reporting before but I used Crystal Reports a couple of years ago so I knew the basics of the reporting tool.

When you use reporting tools, the learning curve is slow, so if you can get the insides and details to accomplish not only the basic stuff but things like subreporting, complex styles, etc it would be much better. And I think that's the objective of the book...explain you how to do useful things with reporting.

I'll review the complete book later when I finish reading it.

Infobright goes open source!

2008-09-16T17:27:00.000-07:00

Long time without write in this blog...have been too too busy in my new job.

Anyway since yesterday I'm glad to see that infobright, a data warehousing storage engine for mysql has been released as open source.

You can see the news here .

To the mysqlers that are using this good database for data warehousing this is a pretty good new one. As Robin states in the article when you reach the 1-2 TB in your mysql db you must begin to do lot of tricks to make things works.

Please check the Infobright website! http://www.infobright.org/

Office, visio, project

2008-05-15T07:39:00.000-07:00

Seems like when you are in a non technical position you must use these Microsoft tools very often....which is sad because I don't want lose my Linux desktop as the main os I use in the work...

So in these last days I basically tested: vmware and wine.

Vmware

The thing with vmware...it works fine but you must startup windows every time inside vm...and...the applications aren't integrated into the desktop.

Wine

The good thing with wine is that windows apps are integrated in the window manager....the bad...make programs works is a pain....Now it seems that word is running but not sure what surprise it will give when I use it with more detail....

Thanks to my friend Tom for the support...

A good new one

2008-04-18T12:34:00.000-07:00

I think this post will be quite large...sorry if You get bored with it......
The good one: I will change my job. I resigned yesterday to my current job as an information system analyst in a telco operator. I will move to a big cable and tv company as Traffic and Dwh Director.

First, I have to say I'm quite happy about the change but I have a mix of feelings as is natural. I was engineer there for three years and a half. In that time I made a lot of good friends and workmates. Also I think this three years was a period where I learn a LOT of useful things. In this job I had the opportunity to play with linux, MySQL, tomcat, mondrian, pentaho, kettle, clusters, SANs etc. and while I am not an expert in any of those things I have a good knowledge in each of them.

Sadly things have been going to the wrong side in the area ( caution note: these are personal opinions :) ). Since Millicom bought the company and the managers went off the area has become less and less relevant for the organization. The hope that our developed software and knowledge can be extended to other operations worldwide (a thing that can be possible because Millicom have more than twenty operations around the world) is almost lost. I have to say Millicom doesn't like home development teams so our existance is practically hidden from the headquarters.

Moreover the new management style is very precipitated (the high level management not my direct boss...my direct boss is a very good leader and friend). Today we have a new project, tomorrow is canceled, today we exist, tomorrow not. Things were needed in 4 hours no matters the quality cost and the entropy added to the team.

You would think that my ex job was a bad job. That is not true. As I said i faced a lot of interesting problems, and learn lot of things. Also there are good organizational practices there.

Anyway I'm pretty happy because:

I will manage the dwh cycle
I will have the opportunity to lead a technical team
Is a demostration that the good job is rewarded sooner or later
I will have to show that peter principle is wrong

And many other reasons! so expect technical and managerial posts from now! My desire is not to abandon the technical skills only to use them to lead software projects to the expected results...so my leadership will be from a technical perspective.

And to my good friends in ##pentaho irc channel....I will ask a lot there!...be prepared! :)

Back from holidays

2008-04-14T08:32:00.000-07:00

Hey!
Today is my first job day after the holidays and I have to say: I'm out of form.

Anyway...holidays were great, as I told in the last post I went to Valledupar and to Santa Marta. There I take a very good rest. The two weeks I stayed in Bogotá I did this:

Bought a new pc: Intel Quad Core 64 bits, 4GB Ram, 320GB, Nvidia graphics card, etc...This pc will be my home development server.

Did the eye surgery: I no longer use lens!

Played a LOT of squash: Two times a day!

I'm very happy with the holidays I took..I feel they changed my daily routine and now I'm ready to work.

Btw...Colombia is a great place to take holidays!

Holidays!

2008-03-19T09:42:00.000-07:00

Is official tomorrow my holidays vacations begin!...I will take 14 work days which are like 3 weeks in calendar days...

The last year and the current has been very heavy in work load terms...The managers of the area and the vice-presidency where I work for were changed. The new managers came with new ideas, strategies, etc. Furthermore we had to implement several SOX controls in the area.

I really feet tired in this year and I think I really need a time to rest...What will I do in the holidays? I'll go a week to Valledupar and the other weeks I will spend some time mounting my home wlan, buying a new desktop computer and practicing squash.

Squash Tournament

2008-03-14T13:14:00.000-07:00

I just entered to the squash tournament of the company I work for. Its a very big tournament with 6 extreme players (3 from TI, 3 from Network) :-). We will use English scoring and double yellow dot balls. I

hope to post some interesting reports about the games I will play, hope my not so disciplined drilling practices will serve in the games.

Regarding squash this is another cool website for reading tips and watching videos: squash club

Backup, backup, backup

2008-02-29T18:14:00.000-08:00

Perhaps every system administrator/DBA talks about this issue...and this recommendation is based on the common sense: Make valid backups of your data.

We made backups at all level: operating system, user file system directories and the database. Because we use MyISAM and ARCHIVE engines we simply flush the table we want to backup and then copy the files to the tape.

The backup must be an automated labor that must run at different level (full, incremental and differential), and there are many good open source tools for this: we use Bacula with great results.

Why I am talking about this? this last week I must drop a temporary table with so bad luck that I dropped the wrong table (an was an aggregated historic table). Those are the moments when you say...thank goodness I have a backup!

Learning to dump: The hard a way

2008-02-23T16:11:00.000-08:00

Remember the problem with ARCHIVE engine in the upgrade from 5.0 to 5.1?

Well...I had to mount a parallel MySQL 5 server and restore those tables from the backup. Now the trick is to do a mysqldump...the thing is...what would be the best way to dump the tables (as the avg number of rows is around 20millon)?

The first approach I tried: a bash script that goes table to table doing:
-mysqldump thedb thetable |mysql -htheotherserver

Problems? The dump is going very slow....moreover I learn something the hard way: Mysqldump by default load all the table in memory before dump it....the result? somewhen in the morning the server ran out of memory and crashed......

This week that comes I will try other approaches to the problem....

Migrating the platform to mysql 5.1

2008-02-20T13:04:00.000-08:00

Last week I began the actualization of the server platform (CentOS versión, MySQL version, GFS version) for the dwh I manage.

Everything went quite right except the MySQL upgrade (from 5.0 to 5.1). As you may know MyISAM tables are composed by three files: a frm file that contains information of the table structure, a myd file that contains the data and a myi file that contains the index data. The MyISAM structure didn't change from 5.0 to 5.1 so I just did a copy of the files....but that was not totally true for ARCHIVE engine...

The ARCHIVE engine is a compressed data table which allow less disk usage (you should know disk usage is important for DWH :)) ...the scans are fater because rows are compressed and the server can read more rows from disk (obviously with the penalty of processor time for decompression). I use ARCHIVE for historic and less frequently queried tables...

So what was the problem? MySQL 5.1 crashes on some (not all) ARCHIVE tables....I am working around this problem.....

Mondrian at last

2008-02-14T15:07:00.000-08:00

Finally I did something in production using Mondrian. For those who don't know what it is, Mondrian is an open source rolap server written in Java. What is the purpose of OLAP? analyze information or measures across different dimensions. So, if you want to use Mondrian, you should follow the dimensional modeling principles in your db design and then build a XML document that maps dimensions and fact tables into the relational entity model. Here you can see some screenshots of Jpivot+Mondrian (Jpivot is a web UI for mondrian)

I did a couple of cubes in order to calculate the Payback of the BTS (base stations) of the mobile network. The payback is the capex of the bts divided by the cash flow that the station generates. I have two cubes one for traffic and revenue and one for the capex costs, also I have three dimensions: geographic, date and type of coverage. With those two cubes I made a virtual cube for the payback calculated measure.

On the top of the cubes you write the MDX (kinda SQL for Multidimensional analysis) for each report. MDX imho is an awesome expression language that lets you write many differents analysis,

The user's response to this app was great althought Jpivot is a little bit oldie...Anyway if you plan to build a data warehouse using open source software and you'd like to have this nice feature read more deeply about Mondrian.