Business intelligence and data warehouse methodologies

Methodologies provide a best practice framework for delivering successful business intelligence and data warehouse projects.

Depending on your requirements, we will draw on one or more of the following established methodologies.

Data Vault 

The way we manage data must be flexible and able to adapt to today’s fast-changing and dynamic businesses and agile working practices. But traditional data modelling methodologies (such as Kimball) aren’t designed to respond to fast changes. Implementing Data Vault modelling can help address this challenge. 

The Data Vault methodology approach

Data Vault architecture is a hybrid approach that combines the best of 3rd Normal Form (3NF) and dimension modeling. This data modeling technique enables historical storage of data, integration of data from different operational systems as well as traceability and auditability of the data being loaded into the database.

The Data Vault approach is based on the concept of Hubs, Links and Satellites:

• Hubs represent business keys that are uniquely identified, with a very low tendency to change.
• Links represent relationships (transactions, hierarchies or associations) between hubs. A single link is created for each type of relationship, e.g., a single link record indicates that there is a relationship between customers and products called Purchases.
• Satellites represent attributes of hubs or links, consisting of data that tends to change over time. A single satellite record is created for each instance of a relationship, e.g., a single satellite record is created for each instance of a customer purchasing a product, defining attributes of that purchase such as date, quantity and price.

Data Vault architecture: benefits and advantages

Data Vault architecture is designed to be flexible and adaptable to meet the changing needs of an enterprise. With traditional data models, it could take BI teams months to add new relationships into the data warehouse, because those relationships are built into the schema of the warehouse. So any change requires extensive governance and testing. The Data Vault approach makes enterprise data warehouses more agile, because the relationships are not part of the schema – they are just data rows in the Links and Satellite tables. This enables rapid implementation of evolving data relationships.

The advantages Data Vault brings are:

• Simplified and high-performance data ingestion
• Comprehensive audit trail of data
• Easy incorporation of changed data sources without disruption to existing schema
• Increased usability by business users as a Data Vault is modeled after the business domain
• High performance
• Support of near-real-time loads as well as batch loads
• Terabytes to petabytes of information (Big data)
• Supports isolated, flexible and incremental development (organic growth).
• Dynamic model can be incrementally built, easily extended
• No rework is required when adding additional information to the core data warehouse model
• Supports business rule changes with ease

Data Vault also lends itself to pattern-based, metadata-driven development. In many cases this can lead to reduced time spent developing and maintaining the data warehouse and associated extract, transform and load processes.

Our Data Vault experience 

At Theta we have a team of Data Vault certified experts located in our Auckland, Wellington and Christchurch offices.  Our consultants have experience in all aspects of Data Vault architecture:

• Enterprise data modelling
• Data warehouse design
• Development and maintenance of data warehouses
• Accelerated pattern-based development approaches

Data Vault courses and training

We partner with Hans Hultgren (Genesee Academy), one of the leading proponents of Data Vault worldwide. Hans provides training and best practice advice on Data Vault techniques.

If you are interested in Data Vault courses or training please get in touch.

Kimball data warehouse methodology

The Kimball Data Warehouse Methodology was developed by Ralph Kimball, who is widely regarded as the father of the data warehouse. Guidelines that every Kimball data warehouse should follow include:

• The primary objectives of a data warehouse should be performance and ease of use.
• Dimensional models can only be developed once information requirements have been understood and agreed.
• While the data warehouse will constantly evolve, each iteration should be considered a project life cycle consisting of predictable activities with a finite start and end.

Theta ETL infrastructure methodology

Theta’s ETL infrastructure methodology draws on experience implementing many data warehouse solutions using both WhereScape RED & Microsoft SQL Server Integration Services (SSIS).  The ETL infrastructure is a set of processes, procedures and code that can be reused in a customer environment. 

This infrastructure deals with:

• consistent method and content for entering operational metadata for loading data into the data warehouse environment;
• consistent methods for ETL error handling;
• consistent methods for ETL notifications;

The advantages of implementing the Theta ETL infrastructure methodology include;

• faster delivery (no re-inventing the wheel);
• consistency and simplicity for maintenance;
• minimise bugs and rework;
• leveraging of best practice and experience.

Theta’s Pragmatic Agile methodology

Theta uses an iterative approach to deploy data warehouse and BI solutions, where we prioritise components or features of the solution for deployment based on a combination of:

• high business impact (value);
• low cost (including risk);
• explicitly stated priorities of the business;
• dependencies between components.

The objective is to get as much value (usually in the form of information assets) into the hands of users, where it can provide the most value as quickly as possible. 

This goes hand in hand with our iterative approach to BI development. Key elements include:

• Getting information to the users as quickly as possible, so they can see data and request relevant changes while development is underway. 
• Developing and testing reports and dashboards in close collaboration with users. 

The advantages of deploying solutions in a pragmatic, agile and iterative way include:

• early and visible return on investment;
• quick delivery to stakeholders and users;
• user requirements can be reviewed and revisited regularly;
• better control of the project from a delivery perspective;
• better financial control of the project.

See also: Data quality