Characterize the Environment
The following list outlines the items and issues to cover in attempting to characterize the environment within which your challenge is encapsulated.
Characterize the Environment
Organizational Scope
Project Sponsorship
Business Context
Business Objectives
Current Data Landscape
Solution Objectives
Technical Environment
Resources
Funding
The topics outlined above may not all be necessary for some smaller organizations and in any case, it is not recommended to take a heavy approach to most of these items. All that is required is that they be given some thought and be documented if there is any useful information to be captured.
Organizational Scope
It may be that you are launching a BI initiative that is intended to eventually serve the entire organization. Titles like 'Enterprise Data Warehouse' (EDW) and 'Corporate Information Factory' (CIF) have frequently been given to such initiatives.
On the other hand, it may be that only a subset of your organization is the intended user and customer of the product. A division of your company, or possibly even a single department.
An all-too-common practice has been the creation of Data Marts an a virtually ad-hoc basis, resulting in an unknown and uncontrolled proliferation of such entities. It was recently published that Hewlett Packard launched an offensive against such a proliferation when it was discovered that there were more than 750 Data Marts in existence across the organization. This is when Business Intelligence could start to lose its true intended purpose.
Capturing the details of the organizational scope of your BI Initiative will serve to define the limits of the project.
Project Sponsorship
Any project needs to have its sponsors well defined. It is a good strategy to encourage a vibrant relationship between the sponsors and the delivery team. Record who they are and work on that relationship. If you are the sponsor then be prepared to maintain close contact and involvement with the person or team implementing your BI solution.
Business Context
Within the organization that is sponsoring or requesting a BI facility, there may any number of business processes, databases, application systems and sub-organizations. Take time to make a record of those for which the solution is to relevant, either as data sources or expected areas of enhancement.
Business Objectives
Why do we need a BI solution? Someone in the organization came to the conclusion that this would be beneficial. Someone is aware that the solution will meet one or more needs. What are those needs? What does that person or group expect to achieve? In what way will their aspirations be met? They must have a reason, or many reasons, perhaps for going ahead. Those reasons will indicate and help crystallize the objectives of the BI Initiative.
Current Data Landscape
Now that we have organizational scope, project sponsors, a business context and objectives, we should try to define the data sources that need to be leveraged in order to provide the candidate information from which some intelligent business decisions can be derived.
At this level it is sufficient to speak of this in terms of databases or applications or functional areas covered by an application.
Additional information about the data sources should be collected. How easy will it be to learn about the data source? Is it well documented? Are there still people available who developed, maintained or supported the database that can contribute inside knowledge? Is the source 'out-of-bounds'? Can we only obtain data from it under some constraint, such as time, frequency, security policy, etc.?
Solution Objectives
The information gathered so far will help direct your efforts and maintain control over scope, cost and delivery aspects of the project. They may not, however, define everything we might be able to learn about the nature of the target solution. It is possible that corporate or IT policy may also impact the size and scope of the project.
It is possible that technical limitations exist or resource availability limitations exist. There may even be limits on funding that prevent all the business objectives from being pursued all at once.
Sometimes, it makes sense to extend the scope a little because the effort of visiting one or more data sources just to obtain the requested data me be wasting an opportunity to pull more useful (but not yet requested) data, This is a very important consideration because returning to the same source later is far more expensive than picking up data now that may even be in adjacent columns to that which we have come looking for. This is one area where an experienced Dimensional Modeler is able to point to 'easy gains' that could soon satisfy a growing appetite for analytic data.
Technical Environment
It will be much easier to estimate the size of the project if we know something about the challenges it may present. One such challenge is dealing with the issues surrounding the way the data is currently hosted. Is it all in the same kind of database? Is it all in the same location? Are there common standards in place governing naming conventions, data type usage, data structure designs, etc?
Data is not always sitting in a database, nor is it always easily accessible. Sometimes it is necessary to access data that has already been extracted from its original source and stored somewhere else. It may be necessary to reuse the data feed of such a transfer of data from the destination. This can introduce serious issues of data reliability and stability. if the second destination has itself transformed the data or if the data has been summarized to a higher level of granularity or even stored as a periodic snapshot instead of at an atomic level, there will be issues.
Resources
Besides the need for resources on the BI project, there is often the need to call on the help of others outside the project. The lack of such resources, or limitations on their availability can cause problems for the BI Initiative.
Funding
Large companies tend to be able to fund large projects. Conversely, small companies cannot. However, outside of this fact, there is a need to consider whether a large project is actually the most effective approach to BI. Many initiatives have come to grief because the challenges involved, compounded by the inherent difficulty of coordinating large scale projects, leading to massive over-runs and wasted time.
One of the key areas for devastating misjudgments is that in which organizations inexperienced in DW and BI take on projects that are too large and too complex for their relatively inexperienced team.
Regardless of whether funding is easily obtained or not, it is strongly recommended that some kind of 'pilot project' be initiated first, to gain experience and confidence in dealing with the new technologies involved. Then after an initial smaller success, the team (or individual, as is often the case) can move on to take more territory.
Key to success is knowing how much to attempt rather than how much to fund.
Establish a Roadmap
The mode under which the BI Initiative will be conducted is the most important consideration at the start of the project.
Historically, most software application development projects followed a phased plan that was essentially serial in nature. Often referred to as 'waterfall model' processes, these were so named because the product of the first phase 'poured' into the second phase, the second into the third and so on, regardless of the name or purpose of each phase.
Eventually, it became obvious that this approach could be responsible for many of the ills that plagued such projects. The fact that any given phase is dependent entirely on the quality, appropriateness and timeliness of the products of its predecessor meant that any single phase had the potential of setting the limit on the overall project success, of being the weakest link.
Furthermore, the very act of feeding one phase with the documented results of another could introduce defects due to the misunderstandings, ambiguities, lack of clarity, etc., resulting from this form of communication. Surprisingly, the creators of one phase's deliverable were often not available for consultation with the recipients of their efforts in the next phase. Many weaker processes did not mandate a review (or quality gate) at the end of each phase.
Yet another issue with Waterfall models is that the project is not finished until all phases are complete. As this could extend into years, it would be necessary to 'freeze' requirements until the project was complete. Business changes would have to go unattended to because interrupting this process was too disruptive to be tolerated. Also, any error in delivering against the original requirements may not come to light until the very end of the string of phases, when it is then the most expensive to repair.
Evolving the Solution
Avoiding all the above issues is not a trivial task. Splitting the project into 'waves' is sometimes attempted to reduce the work going through the sequence of phases and therefore reducing the time frame of each wave is one, often used approach for very large projects. However, this does not remove the problem-causing imperfections of waterfall models, only 'divides' in order to 'conquer' them.
The approach that often produces the most successful result is that of 'Iterative' development, or prototyping. The main aim in this method is to work as quickly as possible toward a reduced end-product but one which can still be used, or at least, demonstrated.
Its advantages are: a quicker end point at which the product can be evaluated; an opportunity to learn from mistakes, misunderstandings or unseen challenges and the availability of a version of the solution that the sponsors can try out to see if their ideas were sound or in need of refinement. Indeed, the all-round learning process afforded by the first iteration is one of the strongest arguments for the approach. This takes us away from the need to 'get everything absolutely correct, at the first and only attempt.
The virtual impossibility of such an outcome with the waterfall model is certainly what leads to the creation of a 'Phase II' on most projects. Phase II may include new features but it usually includes a lot more in the way of 'rework' of the initial release.
Establishing a road-map for the project involves the decision to use the appropriate style of process and determining what to deliver by way of a prototype.
Dividing the project into two or more iterations offers many advantages if that option is available.
..to be continued in Part Three of 'How do I Implement Business Intelligence?'.
Noted software engineering expert, Martin Fowler, turns his attention to enterprise application development. He helps professionals understand the complex--yet critical--aspects of architecture. Enables the reader to make proper choices when faced with a difficult design decision.