Latest CDF - script.js minified

Hi,
Just a quick post about the latest CDE-CDF etc that the guys of webdetails have been developing. As In the previous version, distributed in Pentaho platform, I had to tweak a little the javascript of the Map component, I wanted to do the same in this version.

You'll notice a new js file called script.js. That's a minified js generated when the server is started. If you need to twek the js and test you'll have to add this to the dashboard's url: &debug=true&bypassCache=true

Hope this helps anyone else

Aberrant Behavior Detection

As I wrote before, most of the time of this year I worked in Network Management Systems. Managing a huge, constantly growing, network is a heavy task. One part of this task is measurement collection, and for this task we use Cacti, which a powerful visualization and rrd graphics template tool.

Cacti as others open source measurement collections systems rely the data storing and charting in the RRDTool software.

In our Cacti instances we are collecting measurements from 15000 different router/cmts interfaces, which includes common measures like bandwidth, cpu, memory and others not so common like SNR, Cable Modems in the upstream, modulation profile.

As the NOC is unable to watch and control this amount of charts, there is a nice functionality, not widely used, in the rrds: Aberrant Behavior Detection (ABD). ABD adds a set of measurements and algorithms that adapt to the seasonal trend of the chart and detects when a current value is too deviated from the expected value. You can read about the mathematical details here and here.

You can see some examples here:

The classic bandwidth measurement

Cable Modems in the upstream channel

You can write a short Perl script to check if there is an aberrant behavior in the rrd and do something about about it (for example, send a trap or a syslog with the alert). This way you add a great feature to the network management tasks.

Back to Pentaho again

So, this year I have been spending most of my time digging in Network Management Systems both open and closed source. Think on Net-SNMP, SNMPTT, NAGIOS, CENTREON and you will get an idea. I promise i'll post more about this because is such an interesting topic.

Lately I have been with free time so I went back to Pentaho and discovered huge advances in the community ecosystem. Specially Pedro with CDF and Tom with Pat (erm now called Haiku).

As my boss wanted to show the quality status of our customers in a map, what I did was to use CDF with openlayers and open street maps. With some adjustments to the CDF's js libraries (the marker layer) I changed the presentation to render a circle with a color scale and whose size depends on the number of clients in the xy position. Here is the result:

Isn't open source software nice? :-)

User Defined Java Class (UDJC) step

Hey!, I saw the flash recording demo of the new plugin developed by Daniel Einspanjer, User Defined Java Class UDJC and the performance improvement of the Kettle PDI Transformations are awesome. Now Matt Casters is including it in the source tree of PDI 4.

Click here to see Matt's post.
Click here to see the awesome demo.

Coming from vacations

I just came back from a 3 weeks vacations which I enjoyed so much because I havent taken a rest since a year and a half. As is usual I went to Valledupar, nice Caribbean city of Colombia, and stayed three days at Parque Tayrona.

I totally recommend go to Parque Tayrona, very nice beaches and weather and relativelly cheap acommodations.

We slept in hammocks in a site called Arrefices. We went to many beaches to rest and also a indigenous ruins called pueblito indigena which is around 2 hours walking from the beach.

I rested a lot and got back very motivated, I feel I really needed the vacations to acomplish the year's objectives.

Pentaho Reporting 3.5 for Java Developers - Review

So, I've finally finished the book Pentaho Reporting 3.5 for Java Developers. Thanks to Richard Dias from Packet Publishing for the opportunitty and apologies for writing this so late.

So, as I told in the last pentaho reporting book post this book is really good if you need a faster learning curve or if you need advanced reporting capabilities like subreporting, crosstabs, complex functions, styling, etc; but is totally needed if you plan to extend the capabilities of the reporting engine such as Swing or JS events, weird data sources, new report elements or direct interaction trough the reporting API.

So the first chapter An Introduction to Pentaho Reporting , gives you a nice introduction to the reporting software as well the architecture of the product.

Chapter two, Pentaho Reporting Client and Enterprise Examples ,show you how to build basic reports and also how to deploy them into a Swing application or Java web container.

Chapter three, Pentaho Reporting Examples in the Real World, chapter six, Including Charts and Graphics in Reports, and chapter seven, Parameterization, Functions, Formulas, and Internationalization in Reports are the parts I recommend if you are interested in building complex and nice reports (you know, with images, sub reports, well suited to be exported to Excel, with groups, formulas, charts, etc). If you follow the samples in each chapter you'll become almost an advanced report builder.

The fourth chapter, Design and Layout in Pentaho's Report Designer explains the detail of every option in the Report Designer: menu, toolbars, bands, report elements, shortcuts, etc. While this chapter can be a little bit boring I recommend to give it an overview reading cause it has some tips inline while is explaining every detail of the items.

Fifth chapter, Working with Data Sources gives you detailed information of the existing data sources who can feed the report and also how to build custom data sources. If all your reports will be from a database you can skip the chapter.

Chapter eight, Adding Sub-Reports and Cross Tabs to Reports gives you a little  more insight in subreporting and also how to construct reports from MDX queries (OLAP). If you plan to make reports who need a pivot table layout you need to read the chapter.

Chapter nine, Building Interactive Reports explains how to use events in your reports both in JS (click events, hyperlinks, DOM, Jquery) or in Swing. Nice chapter if you need to deploy reports with high interaction or very dynamic from the UI perspective.

Chapter ten, API-based Report Generation, and eleven, Extending Pentaho Reporting, gives you the developer perspective. Things like how to generate a report on the fly, custom expression/functions, new report elements (and how to show them in the designer) and bean shell expressions are covered in those two chapters. This chapters are recommended for developers who plan to extend the nice features of the designer or need to do something very weird.

Finally chapter eleven, Additional Pentaho Reporting Topics show other issues like how to embed reports in the Pentaho BI Engine, publish to mobile browsers and how to ask for help in the Pentaho Communitty.

Well I hope this review to be useful for you, next post will be the same review but in Spanish

A status update

So, I totally forgot my blog until the people from Packt Publishing asked me for the reviewing of Pentaho Reporting book. Now I want to speak about what I have been doing this year.

While I continued with the KPI reporting and information systems the objective of the area I'm managing became broader: Now I'm in charge of the network management systems.

We have several of these systems most of them are propietary and focused in one technology or subset of the HFC and Core networks. The really interesting system is a generic one that We built using open source tools: CACTI for high scale network SNMP measurement collection, NAGIOS/CENTREON/NAGVIS for active ICMP/service checking, and a combination of NET-SNMP, SNMPTT and SYSLOG-NG.

While it is difficult to integrate and manage, its a very good solution to scale and manage the events and traps of all the network with a low cost (the same engineer you would dedicate to a propietary solution plus the hardware).

Also I dedicated a lot of effort customizing TRAC for managing all the requirements to the area. I've found in TRAC a useful tool but with a lack of good reporting modules.

So thats the summary of the year...while I still follow pentaho and databases with passion also I have learnt about SNMP, monitoring, traps.