17 Aug 2022 BI Project Management Part 2 – The Agile Approach to Business Intelligence
In the first part of this mini blog series we described the different approaches or methodologies used when implementing a BI project. In this part we are going to focus on the different areas of a BI project implementation using the Agile approach. For our software development projects, and as referenced in this article, we employed the Scrum methodology because it’s a complete methodology and it suited our purpose perfectly.
How to Apply the Agile Methodology to BI Projects
Project Set-Up
As we commented in the first part, the main reason for using an Agile approach is to get our BI systems up and running as quickly as possible. To do that, we will divide up all of our work (product backlog defined) into iterations. We will separate the scope of the project into several short sprints/iterations, and at the end of this process we will deliver the increased value of the product and do a demo for the stakeholders and end-users, for their acceptance and subsequent production.
So, how do we build our product using Agile – and in particular, Scrum? Let’s look at the roadmap we’re going to follow:
Sprint 0:
When the BI strategy definition is complete and we have the initial product backlog defined (an artifact that can be modified over time, as agreed on by all parties). In this iteration we must create all the technical items required to be able to start building the system in the following sprints. This will include:
- Provisioning the work environments: Development, Staging, Testing, Pre-production and Production
- Software installation and configuration
- Software to version control system
- Software for DevOps
- Project management software configuration
Sprint 1… N:
BI projects are mainly composed of these basic elements:
- Architecture
- Data sources and model
- Data integration
- Data visualisation
All of them are interconnected, generating dependencies. Of course, we can’t develop a report or dashboard if we haven’t analysed and implemented the previous components in the chain (architecture, data model and the data integration pipeline).
We will organise our work using the Agile Scrum methodology, noting the following:
- In Scrum we divide the project scope into several User Stories (as many as we need) that we will work across our iterations
- Every user story defines one functional requirement
- We will work a group of user stories into every iteration, aiming to implement them completely
- To implement a functional requirement, we have to develop a part of the architecture, data model, integration, and visualisation
User Stories
A well-defined user story is the work unit needed to start building and evolving our BI system. An example could be:
UH_1
As Sales Analyst
Needs the capacity to visualise the evolution of sales by month and year
To identify and analyse the months with less sales
With this user story we can:
When we work the other user stories we will increase all these components, adding new data sources, pipelines, evolving the data model, adding measures and aggregations, and developing all the reports and dashboards for the end-user.
Now let’s delve into the different elements that make up a user story:
Data Model
The evolution of the data model, the core of our system, is known as an evolutive dimensional model, and as shown in the following example, it will increase with the implementation of user stories:
Data Sources and ETL
When the dimensional model is better defined, we can develop the ETL pipelines. These processes are in charge of ETL, moving the data from the identified source systems to the staging area and later to the dimensional model with the right shape. The Data Governance process is important too, covering the management of all related metadata.
Data Visualisation
Once all the data is loaded into the dimensional model, the next step is to build the BI solution for the end-users (dashboards, reporting, web interfaces, apps, mobile apps, alerts, etc.). They will use these functionalities to obtain information about the business situation, analyse it and take the right decisions.
Testing
Every implemented user story must be unit tested to guarantee that the purpose is achieved. We should do performance, regressive and data volume testing too, to collectively validate the implementation of all the user stories developed during the iteration.
DevOps
And finally we need to manage the delivery of the following DevOps tasks into every sprint:
Project Planning
Following the iterative work structure explained above, we can organise our sprint plan projects with different approaches. Agile means that we should follow the option that best suits our reality. Consider the following:
Iterative planning, developing stories that contain all the component parts (architecture, data model and sources, ETL, and reporting) and fit them into the planned iteration slot time:
If the user stories are very complex, we can divide them into sub-tasks based on the type of element to implement and develop during continuous iterations:
Mixing a linear and Scrum methodology, depending on the situation and the requirements to implement. Sometimes, especially at the beginning of a project, we can opt to apply a linear and iterative development separately (in parallel) for different work components (for instance, architecture as linear, and modelling, ETL, and reporting as Scrum):
Scrum Best Practices
Here are examples of the top best practices to follow when implementing the Agile Scrum framework in our projects:
Just-in-Time Modelling (JIT) | Don’t go into all the details at the beginning of the project, but during the project. The requirements will change in the future, and at the beginning of the project we have a great amount of knowledge to include in the model, so we don’t want the modelling phase to be a stopper. | |
|---|---|---|
Top-to-bottom | In Scrum every iteration should deliver a vertical slide (tested and working) top-to-bottom functionality, offering added value to the end-user. With this approach, we chop up large blocks of functionality and then work on them in each iteration, avoiding the Waterfall method. | |
Focus on usage, not data | Fully understand the business needs. The system should be defined on use cases and scenarios, not driven by data models. Data sources are essential, of course, but the features and functionalities required by users should be defined first. | |
Work based on requirements not on technical questions | The project must be planned around defined priority requirements, not technical issues. | |
Prioritise according to value | When the functionality is defined by the end-users (product backlog), Scrum requires us to divide it into smaller deliverables developed in every iteration. The end-users will need to prioritise these features carefully. | |
Promote stakeholder collaboration | Involve the stakeholders in the project as much as possible to get better requirement definitions and to avoid future reworkings. If the stakeholders feel like they part of the project, the acceptance of the result is almost guaranteed. | |
Involve end-users early and often | They should be involved in the project from the beginning and should participate on a daily or weekly basis. This helps to ensure the correct gathering of requirements. | |
Early architecture vision | You don’t need to define all the details at the beginning, but it’s important to have a simple deployment plan and a great idea to drive the strategy. | |
Agile architecture and technology | The architecture and technology implemented should be agile. | |
Test automation | Use automated testing tools to be able to deliver functionality with quality. Manual testing will consume too much time and can produce unexpected results. | |
Promote team collaboration | Collaboration between the development team and different roles (data modellers, DBAs, BI developers) increases the likelihood of a successful project. | |
Manage the risk | You need an efficient project control mechanism to minimise risks. | |
Defined iterations | The iteration periods should be fixed and short, thus promoting focus and motivation in the team. | |
Adopt organisational standards | Define common guidelines and best practices for the development team – naming convention, modelling style, report design, etc. |
Agile Aligned With the ClearPeaks Project Methodology
As we explained previously, one of the core values is to value working software over comprehensive documentation, but this doesn’t mean that we shouldn’t support the project with a strongly defined process, steps, and records.
Our very own ClearPeaks Delivery Methodology provides a framework for delivering data analytics applications efficiently, ensuring that project objectives are valid and the deliverables are aligned with the defined requirements, always following the best practices and standards. The methodology is based on industry standards like the Project Management Institute (PMI), Agile frameworks and the Kimball methodology, as well as the experience gained during hundreds of successful implementations.
Our Delivery Methodology defines a project lifecycle composed of 4 workstreams:
- Project Management
- Infrastructure
- Data
- Front-end Applications
A set of processes and tools are defined for each workstream, as we can see below:
Conclusion
Depending on the different use cases that we need to handle, we can adopt one of the following approaches:
- Waterfall: Suitable for stable projects where the requirements and technologies to be used are clear from the beginning and are unlikely to change during the implementation phase. The big catch here is that the product is not validated until the end of the project, creating a potential risk of rejection by users, leading to an increase of related costs with the possibility of change management.
- Incremental: As in the Waterfall approach, you need to have clear requirements and a project strategy defined at the beginning, but in this case, the incremental approach reduces the time to market due to the planned smaller releases, leading to better user adoption. However, the duration and cost of the entire project will be greater due to the complexity of managing different workstream teams. Scope changes are also expensive, although less so than in the Waterfall approach.
- Agile (Scrum): Optimal for projects with a dynamic change of requirements and when the technical solution is innovative and not clearly defined. We will deliver the product continually across short time periods, gathering user feedback, making it easy and inexpensive to apply changes. Agile requires the end-users to be involved in the project lifecycle.
Whilst our ClearPeaks Delivery Methodology encourages the Agile approach, it is flexible and can adapt all the processes to any of the approaches described in part one of this series, depending on the nature of the project, specific requirements, and customer needs.
Why not contact our team of experts today to see how our tried and trusted BI solutions can help you grow your business?
