Big data

At the simplest level, big data is data that is too big to be processed on a conventional relational database system.

The Vs of big data, originally coined by Gartner, usefully enhance this definition:

• Volume - Big data is big! 
• Variety - There are many forms, structured and unstructured, including those traditionally outside the scope of existing analytics platforms (eg photo, video, audio) 
• Velocity - The frequency of incoming data (especially machine-generated data) is coming thick and fast, and demands faster platforms for management and analysis 
• Veracity - Challenges are exacerbated around quality and accuracy 

Examples of big data sources and types of data

• Social media data
• Wearables and internet of things (IoT) data
• Environmental data
• Customer behaviour data
• Sensor data

Big data technologies

We work with our customers to select the best technologies from leading vendors across business intelligence, data warehouse, big data and analytics.   

The data lake and big data

A data lake is a storage repository that holds a vast amount of raw data in its native format, including structured, semi-structured and unstructured data. The data structure and requirements are not defined until the data is needed. Data is stored once, in one place but can then be used and accessed in many different ways – for example queried, searched, visualised. It complements, rather than completely replacing, the enterprise data warehouse.

Azure Data Catalog

Azure Data Catalog is an Azure cloud service that allows for enterprise-wide metadata discovery and data source access across an organisation's data stores. It’s a fully managed service and lets any user—from analyst to data scientist to developer—register, enrich, discover, understand, and consume data sources. 

Supported objects that can be catalogued include:

• SQL Server databases, Analysis Services and Reporting Services
• Azure Data Lake Store File
• Azure Storage
• HDSF and Hadoop
• Oracle
• SQL Hana
• OData, and more.

Metadata captured in Azure Data Catalog includes the data source, asset definitions (tables, views, reports) and data profiles (types, row counts, max/min). Users can enrich the metadata by adding documentation, notes, tags or common terms to build up a business glossary.

Users can work with data in the tool of your choice, eg Power BI, Reporting Services or Excel. Your data stays where you want it, and the Data Catalog user portal helps you discover it and work with it where you want. It’s also possible to integrate into existing tools and processes with open REST APIs.

Implementing Azure Data Catalog will enable users to get more value out of the BI environment and encourage self-service reporting and analysis. 

Find out more

Big data and analytics can now be implemented cost-effectively and deliver high return on investment. Contact Theta’s data specialists to learn more about extracting maximum value and business impact from your data assets, and making the most of big data and data-driven insights.