Pentaho Business Analytics Cookbook

Hi,
I had the opportunity to review the book Pentaho Business Analytics Cookbook published by Packtpub. I have been delayed with this review because most of the time was consumed by the master I have been studying.

But now that I read it, these are my comments:

Pentaho Business Analytics Cookbook is a book that gives you a good overview of the Pentaho Platform, for both the enterprise and community version. For each major part of the platform, except Pentaho Data Integration and Weka, it give you a set of recipes to accomplish the most common tasks. These are the topics covered in each chapter:

• Chapter 1: Getting Familiar with Pentaho User Console
• Chapter 2: Configuring Your BA Server Instance
• Chapter 3: Defining BA Server Data Sources
• Chapter 4: Defining Business Models with the Pentaho Metadata Editor
• Chapter 5: Creating Reports Using Pentaho Interactive Reporting
• Chapter 6: Creating Analysis Reports
• Chapter 7: Creating Reports Using Pentaho Report Designer
• Chapter 8: Creating Dashboards
• Chapter 9: Scheduling Content
• Chapter 10: Working with Pentaho Mobile BI
• Chapter 11: Customizing Pentaho BA to Meet Your Business Needs

For example, in chapter 2: Configuring your BA Server Instance, the author gives you an overview of the BA Platform and recipes for managing Pentaho solution files, users and roles. These are things that every Pentaho operator should accomplish day by day.

All the recipes given are well detailed and written with proper images and steps. This is a great book if you plan to be a Pentaho devop.

From my perspective, the book allowed me to:

1. Understand what's new in Pentaho 5, how is the UI organized, how the Pentaho solution is managed on it.
2. Grasp a very good understanding of how to use Pentaho Metadata. Pentaho Metadata is a powerfull business abstraction of the physical db model, sadly I never saw a detailed guide about how to use it. The recipes given in the book are quite accurate explaining how to use it.
3. Know what is the Pentaho Interactive Reporting. As this feature is only available in the Enterprise version I have never used it.  It also give you a good understanding of how WAQR can be used.
4. See how is the Ipad mobile version of Pentaho.

It also cover some few recipes about using Saiku in Analytical Reports and CDE in Dashboard edition.

I hope this review would be useful for Pentaho newbies,

All the best,

Andres

Quick update

Hey, a quick update,

I decided to resign my job in Colombia and go to study overseas. I am currently studying a Master of Information Technology - Distributed Systems in The University of Melbourne, Australia.

I just been in Melbourne 3 weeks and everything so far is very interesting and challenging.

Regards,


Andres

Pentaho for Big Data Analytics

Despite I'm currently not an active community user of Pentaho, mainly because right now I'm focused on network and it managament, I still follow the evolution of this great platform. In the last years there have been great new improvements:

 - The acquisition of Webdetails with all the useful Ctools
- The integration of connectors to several NoSQL technologies allowing the use of big data in all the components of the platform (BI Server, Kettle, Mondrian, etc).

 Recently I got the opportunity to review the Pentaho for Big Data Analytics book published by Packt. My expectations on the book were quite high. I hoped the book would help me to stay updated with the latest improvements of Pentaho and clarify a lot of the marketing buzzword around Big Data.

What I found was the following:

 - The book intention was to give a broad overview of Pentaho components and spend a lot of chapters setting up Pentaho platform: One would expect that if someone buy this book is because he already have a background of Pentaho and want to detail the relationship of it with Big Data.
 - The Bigdata theory and examples were oriented to using Apache Hive: It is explained how to handle files in HDFS and how to handle Big Data analysis through Apache Hive (which at the end is a SQL layer over Hadoop). While the examples and theory are a good introduction to the topic, there a lot of issues not handled: How to do a map/reduce job directly to Hadoop? What about other NoSQL technologies like MongoDB, Reddis, etc?
 - There are very good examples about how to use Ctools: Handling the CDE, CDF and CDA tools is not easy at the beginning, so the "Visualization of Big Data" chapter is very helpful for this.

I think the book is worth of read if you are a new user of Pentaho and Hadoop, want a introduction about how to install, run them, etc, and need the first steps to handle data through Hadoop.

Software Engineering for Software as a Service - Statement

If you want to have an idea of how is the statement here is a picture of it:

Software Engineering for Software as a Service

Hi,
I had the experience to take the Software Engineering for Software as a Service course offered by professor Armando Fox and David Patterson from University of Berkeley through the coursera startup.

The course used the same material, quizzes and videos from the official Berkeley class. And at then end deliver an statement of accomplishment. I'll try to upload it later.

In general terms I think the experience was great and I am very grateful with Professors Fox and Patterson. There will be a new offering of the course if you are interested. The url of the course is http://saas-class.org

The topics that the course covers are Architectural patterns, software design, code coverage, unit and integration tests, agile development and Ruby. If you like those topics you'll have a great time.

I have to say I liked the Ruby language and the rails framework. In a non expert and scientific perception I think it is more suitable for most of the projects than Java in terms of speed and velocity of development and value added to enterprise software.

Tutorial CDE

Inteligencia de Negocio y Pentaho: Cómo hacer cuadros de mando: V: Un tutorial recomendado para manejar las excelentes herramientas que Pedro Alves y Webdetails han desarrollado, en especial CDE.

Your faithful employee

Hello,

In the area I work for, we have one duty among several others more: Maximize the availability of the network management systems. The NMSs must run almost every time because the NOCs (network operation centers) monitor them 7x24.

But thats not an easy task, sometimes the planning area delivers the NMS implementation with many flaws, other times the machine you get is not what you expected or the application servers freezes continously and research the root cause can take weeks.

As we're not big fan of attending service disruption calls at 3am, we deployed a nice and useful service in all our Linux machines: Monit. This nice program monitor the service existence,availability and performance and take automated actions when the rules/thresholds are exceeded.

For example, we had a Tomcat container that was getting frozen several times at the week...the rule for Monit was something like this:

check process tomcat5 with pidfile /var/run/tomcat5.pid
group tomcat5
start program = "/etc/init.d/tomcat5 start" with timeout 120 seconds
stop  program = "/etc/init.d/tomcat5 stop" with timeout 120 seconds
if failed host 127.0.0.1 port 8080 
protocol HTTP request /archivos/gestion.jpg 
TIMEOUT 3 SECONDS then restart
if cpu usage > 95% for 10 cycles then restart
if 5 restarts within 5 cycles then timeout

So, this loyalty automated employee check the port 8080, check the HTTP request of gestion.jpg to be less than 3 seconds and check the cpu usage of the process to be under 95%. If Monit sees any of these rules broken then it begins to restart the service. As the good employee he is, Monit sends email notification of every step he takes.

Hope this application be useful for you,


Quick update: Even thought Monit is doing a great job is important to find the root cause. Regarding the tomcat issue i found this useful site to tune the JVM parameters: http://wiki.alfresco.com/wiki/JVM_Tuning

pentaho CDE CDF CDA CD* tutorials

Quick post to help people trying to make stuff on those great tools developed by webdetails.pt . They are great but have a lack of docs, so this two links would help:

http://www.vinzi.nl/media/CDE-introduction_V0_8.pdf
http://www.tikalk.com/incubator/blog/creating-bugzilla-dashboard-%E2%80%93-hands-cde-tutorial-%E2%80%93-fuse-day-3-session-summary

I got those links in IRC ##pentaho channel on freenode btw

PD: If you want to install CDE in a context path different to /pentaho/, keep in mind that uri is hardcoded in all js and templates files of the pentaho-cde-dd directory

Messing with Roo

In the last Google I/O was anounced that GWT 2 will have support in Spring Roo. I never heard of Roo before and I wanted to take look (http://www.springsource.org/roo).

It seems that Roo improves the development by maintaining all the interfaces, stubs and glue code that is boring when you use GWT or Entities.

I just ran two of the demos...a classic Spring Controller Roo Project with handling the CRUD of a Concat entity (http://s3.springsource.com/MRKT/roo/2010-01-Five_Minutes_Roo.mov):

And a GWT integration handling the CRUD of an equipment (http://www.thescreencast.com/2010/05/how-to-gwt-roo.html)

So far  it was pretty impressive to make an entity based application in 5 minutes without all the glue code, but as this is a new concept I wonder if anyone has done any real large sized project supported in Roo....

Render a Prpt report correctly in a Pentaho's BI Platform xaction

Hi, a quick hint about writing a Xaction who must render a Prpr Report. If you try to use the xaction writing plugin for Eclipse, the xaction probably wont work. Here is the correct xaction, and you should overwrite the xaction with this in Eclipse (changing the prpt name of course):

<?xml version="1.0" encoding="UTF-8"?>
<action-sequence>
  <title>Test report in PDF format</title>
  <version>1</version>
  <logging-level>TRACE</logging-level>
  <documentation>
    <author>Emmanuel Guiton</author> 
    <description>Generates a PDF test report.</description> 
    <help/> 
    <result-type/> 
    <icon/>
  </documentation>
  <inputs/>
  <outputs>
    <test_report type="content">
      <destinations>
        <response>content</response>
      </destinations>
    </test_report>
  </outputs>
  <resources>
    <report-definition>
      <solution-file>
        <location>test-report.prpt</location> 
        <mime-type>application/zip</mime-type>
      </solution-file>
    </report-definition>
  </resources>
 
  <actions>
    <action-definition>
      <component-name>SimpleReportingComponent</component-name>
      <action-type>Test report</action-type>
      <action-resources>
        <report-definition type="resource"/>
      </action-resources>
      <action-outputs>
        <outputstream type="content" mapping="test_report"/>
      </action-outputs>
      <component-definition>
        <outputType><![CDATA[application/pdf]]></outputType> 
        <useContentRepository><![CDATA[true]]></useContentRepository> 
        <paginate><![CDATA[false]]></paginate>
      </component-definition>
    </action-definition>

  </actions>
</action-sequence>

Hope this help.

Latest CDF - script.js minified

Hi,
Just a quick post about the latest CDE-CDF etc that the guys of webdetails have been developing. As In the previous version, distributed in Pentaho platform, I had to tweak a little the javascript of the Map component, I wanted to do the same in this version.

You'll notice a new js file called script.js. That's a minified js generated when the server is started. If you need to twek the js and test you'll have to add this to the dashboard's url: &debug=true&bypassCache=true

Hope this helps anyone else

Aberrant Behavior Detection

As I wrote before, most of the time of this year I worked in Network Management Systems. Managing a huge, constantly growing, network is a heavy task. One part of this task is measurement collection, and for this task we use Cacti, which a powerful visualization and rrd graphics template tool.

Cacti as others open source measurement collections systems rely the data storing and charting in the RRDTool software.

In our Cacti instances we are collecting measurements from 15000 different router/cmts interfaces, which includes common measures like bandwidth, cpu, memory and others not so common like SNR, Cable Modems in the upstream, modulation profile.

As the NOC is unable to watch and control this amount of charts, there is a nice functionality, not widely used, in the rrds: Aberrant Behavior Detection (ABD). ABD adds a set of measurements and algorithms that adapt to the seasonal trend of the chart and detects when a current value is too deviated from the expected value. You can read about the mathematical details here and here.

You can see some examples here:

The classic bandwidth measurement

Cable Modems in the upstream channel

You can write a short Perl script to check if there is an aberrant behavior in the rrd and do something about about it (for example, send a trap or a syslog with the alert). This way you add a great feature to the network management tasks.

Back to Pentaho again

So, this year I have been spending most of my time digging in Network Management Systems both open and closed source. Think on Net-SNMP, SNMPTT, NAGIOS, CENTREON and you will get an idea. I promise i'll post more about this because is such an interesting topic.

Lately I have been with free time so I went back to Pentaho and discovered huge advances in the community ecosystem. Specially Pedro with CDF and Tom with Pat (erm now called Haiku).

As my boss wanted to show the quality status of our customers in a map, what I did was to use CDF with openlayers and open street maps. With some adjustments to the CDF's js libraries (the marker layer) I changed the presentation to render a circle with a color scale and whose size depends on the number of clients in the xy position. Here is the result:

Isn't open source software nice? :-)

User Defined Java Class (UDJC) step

Hey!, I saw the flash recording demo of the new plugin developed by Daniel Einspanjer, User Defined Java Class UDJC and the performance improvement of the Kettle PDI Transformations are awesome. Now Matt Casters is including it in the source tree of PDI 4.

Click here to see Matt's post.
Click here to see the awesome demo.

Coming from vacations

I just came back from a 3 weeks vacations which I enjoyed so much because I havent taken a rest since a year and a half. As is usual I went to Valledupar, nice Caribbean city of Colombia, and stayed three days at Parque Tayrona.

I totally recommend go to Parque Tayrona, very nice beaches and weather and relativelly cheap acommodations.

We slept in hammocks in a site called Arrefices. We went to many beaches to rest and also a indigenous ruins called pueblito indigena which is around 2 hours walking from the beach.

I rested a lot and got back very motivated, I feel I really needed the vacations to acomplish the year's objectives.

Pentaho Reporting 3.5 for Java Developers - Review

So, I've finally finished the book Pentaho Reporting 3.5 for Java Developers. Thanks to Richard Dias from Packet Publishing for the opportunitty and apologies for writing this so late.

So, as I told in the last pentaho reporting book post this book is really good if you need a faster learning curve or if you need advanced reporting capabilities like subreporting, crosstabs, complex functions, styling, etc; but is totally needed if you plan to extend the capabilities of the reporting engine such as Swing or JS events, weird data sources, new report elements or direct interaction trough the reporting API.

So the first chapter An Introduction to Pentaho Reporting , gives you a nice introduction to the reporting software as well the architecture of the product.

Chapter two, Pentaho Reporting Client and Enterprise Examples ,show you how to build basic reports and also how to deploy them into a Swing application or Java web container.

Chapter three, Pentaho Reporting Examples in the Real World, chapter six, Including Charts and Graphics in Reports, and chapter seven, Parameterization, Functions, Formulas, and Internationalization in Reports are the parts I recommend if you are interested in building complex and nice reports (you know, with images, sub reports, well suited to be exported to Excel, with groups, formulas, charts, etc). If you follow the samples in each chapter you'll become almost an advanced report builder.

The fourth chapter, Design and Layout in Pentaho's Report Designer explains the detail of every option in the Report Designer: menu, toolbars, bands, report elements, shortcuts, etc. While this chapter can be a little bit boring I recommend to give it an overview reading cause it has some tips inline while is explaining every detail of the items.

Fifth chapter, Working with Data Sources gives you detailed information of the existing data sources who can feed the report and also how to build custom data sources. If all your reports will be from a database you can skip the chapter.

Chapter eight, Adding Sub-Reports and Cross Tabs to Reports gives you a little  more insight in subreporting and also how to construct reports from MDX queries (OLAP). If you plan to make reports who need a pivot table layout you need to read the chapter.

Chapter nine, Building Interactive Reports explains how to use events in your reports both in JS (click events, hyperlinks, DOM, Jquery) or in Swing. Nice chapter if you need to deploy reports with high interaction or very dynamic from the UI perspective.

Chapter ten, API-based Report Generation, and eleven, Extending Pentaho Reporting, gives you the developer perspective. Things like how to generate a report on the fly, custom expression/functions, new report elements (and how to show them in the designer) and bean shell expressions are covered in those two chapters. This chapters are recommended for developers who plan to extend the nice features of the designer or need to do something very weird.

Finally chapter eleven, Additional Pentaho Reporting Topics show other issues like how to embed reports in the Pentaho BI Engine, publish to mobile browsers and how to ask for help in the Pentaho Communitty.

Well I hope this review to be useful for you, next post will be the same review but in Spanish

A status update

So, I totally forgot my blog until the people from Packt Publishing asked me for the reviewing of Pentaho Reporting book. Now I want to speak about what I have been doing this year.

While I continued with the KPI reporting and information systems the objective of the area I'm managing became broader: Now I'm in charge of the network management systems.

We have several of these systems most of them are propietary and focused in one technology or subset of the HFC and Core networks. The really interesting system is a generic one that We built using open source tools: CACTI for high scale network SNMP measurement collection, NAGIOS/CENTREON/NAGVIS for active ICMP/service checking, and a combination of NET-SNMP, SNMPTT and SYSLOG-NG.

While it is difficult to integrate and manage, its a very good solution to scale and manage the events and traps of all the network with a low cost (the same engineer you would dedicate to a propietary solution plus the hardware).

Also I dedicated a lot of effort customizing TRAC for managing all the requirements to the area. I've found in TRAC a useful tool but with a lack of good reporting modules.

So thats the summary of the year...while I still follow pentaho and databases with passion also I have learnt about SNMP, monitoring, traps.

Pentaho Reporting 3.5 for Java Developers

So, it seems I can't get used to write in my blog often. Lately I've been working with monitoring tools (cacti, nagios, snmptt, etc), so I have left BI and pentaho a little bit aside.

A couple of weeks ago I had the oportunity to review the Pentaho Reporting for Java Developers book, so It reminds me about the cool open source BI platform again.

So far, I've read half of the book and all the insides of this nice reporting tool are very well explained. I haven't used pentaho reporting before but I used Crystal Reports a couple of years ago so I knew the basics of the reporting tool.

When you use reporting tools, the learning curve is slow, so if you can get the insides and details to accomplish not only the basic stuff but things like subreporting, complex styles, etc it would be much better. And I think that's the objective of the book...explain you how to do useful things with reporting.

I'll review the complete book later when I finish reading it.

Infobright goes open source!

Long time without write in this blog...have been too too busy in my new job.

Anyway since yesterday I'm glad to see that infobright, a data warehousing storage engine for mysql has been released as open source.

You can see the news here .

To the mysqlers that are using this good database for data warehousing this is a pretty good new one. As Robin states in the article when you reach the 1-2 TB in your mysql db you must begin to do lot of tricks to make things works.

Please check the Infobright website! http://www.infobright.org/

Office, visio, project

Seems like when you are in a non technical position you must use these Microsoft tools very often....which is sad because I don't want lose my Linux desktop as the main os I use in the work...

So in these last days I basically tested: vmware and wine.

• Vmware

The thing with vmware...it works fine but you must startup windows every time inside vm...and...the applications aren't integrated into the desktop.

• Wine

The good thing with wine is that windows apps are integrated in the window manager....the bad...make programs works is a pain....Now it seems that word is running but not sure what surprise it will give when I use it with more detail....

Thanks to my friend Tom for the support...