Agile Data Warehousing and Business Intelligence

When you hear the word ‘agile’, what do you think of?  Your users being agile in the sense of being able to manipulate data quickly in response to business conditions?  Or the approach to business intelligence being one that responds to business conditions and delivers accurate results quickly?  It’s really both.  There is the end product of an agile business intelligence environment which comes as the result of an agile business intelligence approach.

Business value, like a new car, declines quickly as even a short amount of time passes.  I suggest that long development cycles has a lot to do with the loss of business value of data.  The longer it takes to deliver the data – in say, a data warehouse – the longer the business waits to exploit that data and improve business results.  It is imperative that business intelligence organizations learn to deliver more quickly than they have been.

Let’s also face the fact that we in information management are sitting on the organization’s jewels and it’s a gold rush right now.  We have a lot of work to do, no room for unnecessary activities and (we should have) motivation to deliver quickly and efficiently to keep the backlog from growing and causing more desperate and risky moves.  We have a lot of vessels to deliver data into as well.  HDFS, data streams, master data hubs, data warehouses, data marts, columnar databases, cubes, in-memory databases and all of this potentially in the cloud now make for interesting, and very busy, days ahead.  It’s time to reflect on how we deliver.

In the projects which I am involved in (advisor, architect, program leader), all are interested in this notion of “agile” delivery, which is something I’ve been promoting for many years.  Traditional waterfall approaches are simply not fast enough.  The waterfall cycle needs to be broken.  It is costly, inefficient, and it’s risky to take months for a delivery.  Yet, we must maintain a balance between rigor and creativity.  That’s where leadership and judgment come into play and no methodology will ever replace that!  Any methodology that claims you just put the people into place and it works without major decision making along the way is folly.  As I discuss agile approaches and SCRUM in particular, please note I am not discounting the need for leadership and judgment in the process.

The now somewhat-famous Agile Manifesto has become the foundation of several methodologies, including SCRUM.  It simply reads:

·         Individuals and interactions over processes and tools

·         Working software over comprehensive documentation

·         Customer collaboration over contract negotiation

·         Responding to change over following a plan

The principles behind the Agile Manifesto include early and continuous delivery of valuable software, deliver working software frequently, business people and developers must work together, build projects around motivated individuals, simplicity is essential, self-organizing teams and regular reflection and tuning.  SCRUM is based on these principles and is the most widely adopted of the agile methodologies and, I suggest, the most applicable to business intelligence.

Your judgment and leadership (there it is!) will have to come into play here to determine to what degree you take it.  Many organizations are forcing the issue by adopting SCRUM company-wide, which makes it easy as to which profile you take on:

·         Company-wide adoption of SCRUM with obvious effects on the business intelligence program

·         Business intelligence-wide (your domain) adoption of SCRUM

·         Business intelligence-wide adoption of some level of agile/SCRUM principles

I suggest the last bullet is the minimum requirement so I will delve into some applicable aspects of SCRUM to business intelligence:

·         A general emphasis on people over processes

·         An emphasis on a working project over all else 

·         Minimizing long cycles of acute time planning and opting instead for educated speculation

·         Collecting project requirements into a project backlog, to be implemented in “sprints”

·         Executing full lifecycle (“potentially shippable”) sprints every 3-6 weeks based on value of business requirements

·         Choosing another batch of requirements to work on in the next sprint

To accomplish this, SCRUM suggests requirements (“user stories”) be tiered from short- to longer-term deliverables of sprints, releases and products.  Stories may be themed and longer stories are called epics.  Other product artifacts include Impediments (know them to avoid them), a Product Roadmap, a Release Vision, Team Capacity, Sprint Backlogs, Sprint Burndowns (resource utilization) and Sprint Retrospectives. 

Some of the hallmarks of SCRUM, applicable to business intelligence, are:

·         Hands-off of the requirements once the sprint begins; whatever the accepted criteria of a sprint - even if found halfway through to be wrong or not priority – will be delivered; this forces some serious discipline into the process of building the Sprint Backlog

·         Multi-capable teams (Scrum Teams) of 5-7 (lead by a Scrum Master) per sprint; many business intelligence professionals can handle data integration and data delivery so their focus may change from sprint to sprint according to the needs of that sprint

·         An eschewing of multitasking in favor of focus; studies of multitasking show a serious drop-off in doing value-added work once the number of tasks exceeds 2; in SCRUM individuals are focused for the duration of the sprint on “1” task (1 is in quotes because we know how nuanced business intelligence tasks can be)

·         Since planning is essential, Sprint Retrospectives are done to assess what worked, what didn’t and what to take-away into future sprints

Please note I’m only referring to project work in terms of using SCRUM.  Production work – the arguably more important half of the team’s workload – is still more queue-based work. 

Some struggle with breaking down requirements into such short cycles of work effort.  This struggle can be quite productive and result in some eye-opening removal of unnecessary process (i.e., no data modeling of tables that will not be populated in this sprint.)  In terms of delivery, keep in mind some sprints will deliver nothing more than the ability to do one query, albeit a complex one.

To achieve agility for the users, you need not only an agile process, but also tools that are aligned with this approach.  It would be a shame to deliver quickly while users still experience poor performance.  The increase in interest in memory-utilizing tools is partly a reflection of this need to “finish off” an agile process with a corresponding well-performing environment.