Aberrant Behavior Detection

As I wrote before, most of the time of this year I worked in Network Management Systems. Managing a huge, constantly growing, network is a heavy task. One part of this task is measurement collection, and for this task we use Cacti, which a powerful visualization and rrd graphics template tool.

Cacti as others open source measurement collections systems rely the data storing and charting in the RRDTool software.

In our Cacti instances we are collecting measurements from 15000 different router/cmts interfaces, which includes common measures like bandwidth, cpu, memory and others not so common like SNR, Cable Modems in the upstream, modulation profile.

As the NOC is unable to watch and control this amount of charts, there is a nice functionality, not widely used, in the rrds: Aberrant Behavior Detection (ABD). ABD adds a set of measurements and algorithms that adapt to the seasonal trend of the chart and detects when a current value is too deviated from the expected value. You can read about the mathematical details here and here.

You can see some examples here:

The classic bandwidth measurement

Cable Modems in the upstream channel

You can write a short Perl script to check if there is an aberrant behavior in the rrd and do something about about it (for example, send a trap or a syslog with the alert). This way you add a great feature to the network management tasks.

Back to Pentaho again

So, this year I have been spending most of my time digging in Network Management Systems both open and closed source. Think on Net-SNMP, SNMPTT, NAGIOS, CENTREON and you will get an idea. I promise i'll post more about this because is such an interesting topic.

Lately I have been with free time so I went back to Pentaho and discovered huge advances in the community ecosystem. Specially Pedro with CDF and Tom with Pat (erm now called Haiku).

As my boss wanted to show the quality status of our customers in a map, what I did was to use CDF with openlayers and open street maps. With some adjustments to the CDF's js libraries (the marker layer) I changed the presentation to render a circle with a color scale and whose size depends on the number of clients in the xy position. Here is the result:

Isn't open source software nice? :-)

User Defined Java Class (UDJC) step

Hey!, I saw the flash recording demo of the new plugin developed by Daniel Einspanjer, User Defined Java Class UDJC and the performance improvement of the Kettle PDI Transformations are awesome. Now Matt Casters is including it in the source tree of PDI 4.

Click here to see Matt's post.
Click here to see the awesome demo.

Coming from vacations

I just came back from a 3 weeks vacations which I enjoyed so much because I havent taken a rest since a year and a half. As is usual I went to Valledupar, nice Caribbean city of Colombia, and stayed three days at Parque Tayrona.

I totally recommend go to Parque Tayrona, very nice beaches and weather and relativelly cheap acommodations.

We slept in hammocks in a site called Arrefices. We went to many beaches to rest and also a indigenous ruins called pueblito indigena which is around 2 hours walking from the beach.

I rested a lot and got back very motivated, I feel I really needed the vacations to acomplish the year's objectives.