Closing the loop with business intelligence

From my perspective any means of representing data qualifies, from the static TPS reports featured in this clip from Office Space through to customer segmentation, rank, reach and any number of alphabet soup concepts.

With that rather broad definition established, what I want to explore and share is the concept of an end-to-end business intelligence solution. This is the first in a series of blog posts where I'll examine what an end-to-end business intelligence solution looks like; how we can enrich human intelligence and supplement it with machine intelligence, and lastly, a few tools to help you in the exploration of this rather exciting space.

I’ll start with an overview and will gradually incorporate a few more technical concepts, so if you’re keen to delve deeper – stick around.

The current state of things

In traditional business intelligence solutions you will likely see something similar to this:

This represents a reasonably solid foundation. Data will flow from a number of source systems through into an operational data store (ODS). The ODS and other, potentially external sources will be collated, cleaned and sanitised in the data warehouse and made available to some form of analytical database. From there data is delivered to the presentation layer, where reports and other end-user analytical tools place the data in the hands of the users.

Let’s repeat that diagram and see where humans interact with the solution. For now, let’s ignore all the development, analysis and building that go into a solution like this and focus purely on people interacting with the solution on a daily basis for operational and work related activities.

‍

Traditionally humans are the generators of data. They buy things online or in a shop. Other humans capture that data, generate customer lists and a smorgasbord of different, data generating activities. Machines then collate that data and transform it into knowledge, using rules defined by humans. And in the end humans consume that knowledge and use it to start the whole cycle again with a slightly better KPI.

One big problem is that the flow of information is mandatory only in one direction. You will very rarely see a coded, defined mechanism built to ensure the outcomes delivered by the business intelligence solution are used back at the start of the process.

This, to me, represents a major oversight in the realm of business intelligence. Lots of money is spent building solutions that will place knowledge in the hands of the users; but the tools to action and use that knowledge are rarely part of the solution.

Closing the loop

Let’s enrich that diagram and change the flow of knowledge slightly.

To avoid cluttering this diagram with too many arrows I’ve highlighted only a few potential scenarios and I’m going to avoid a technical deep-dive at this stage. You can see there is a data flow back from the presentation layer to the analytical layer. And the analytical layer and the data warehouse itself communicate back through a process and to the source systems.

Example 1: Users closing the loop

Say a data analyst has identified a customer whose support agreements we are at risk of breaching. Normal business process would be to extract the supporting information and send those details via email to the relationship manager. The relationship manager will create an activity in the CRM solution to review the findings and contact the customer if appropriate.

How can the business intelligence solution improve this process?

By directly pushing the activity into the CRM solution with the relevant detail already included. The objective is to close the loop from the analyst perspective and feed directly back into the source system where another end-user can act upon it immediately. This avoids losing a task through email and helps keep the required action visible.

As you can well imagine the number of tasks triggered through actual end-user use of the knowledge distilled from a business intelligence solution is massive. Being able to feed that directly into an operational system and close that loop is invaluable.

Example 2: Systems closing the loop

The second use-case to highlight is that of data quality. For this example, let us assume that the end-result of a customer data load should be one or more means of contacting the customer - a phone number, an email address or a postal address. A new employee has written a simple SQL statement to remove an email address for one customer; but in their haste to get out the door they somehow managed to forget the where clause and updated all customer records to remove the email address. The employee checks in the CRM system and sees the customer email address has been removed, so they log off and head out the door. Oops.

How can the business intelligence solution improve this process?

In that evening’s data load, the data warehouse runs its data validation rules and discovers that almost 30% of the customer records are suddenly missing at least one means of contacting the customer. It immediately flags an action for the owner of the customer data to investigate this and correct the issue.

In this example the solution has effectively become a co-owner in the data it holds and has the knowledge and ability to communicate and contact the relevant people when its data falls outside of expected boundaries. It has, in effect, become a part of the data ecosystem for the organisation, playing a far greater role than just that of a data repository.

Data-knowledge-action

I hope this has highlighted some of the benefits of closing the loop so data flows to knowledge and knowledge flows through to action. That, to me, is a true end-to-end solution. In my next post I’ll look at how this can be achieved and how we could supplement this with machine intelligence. And if you’re interested in reading more about machine intelligence in the meantime, check out Jim Taylor’s series of posts on Microsoft Azure Machine Learning.