To extract business value from data, enterprises need to have the right data architecture, with the right leadership and business culture being critical.
When it comes to business information, CIOs (CIOs) and data directors (CDOs) are tasked with bringing order to the chaos.
As companies collect more and more data, they face both commercial pressures to do more with the information they have, and growing regulatory requirements for data management, especially if it involves customers.
The situation is further complicated by the range of tools available to store and manipulate data, from data lakes and data centers to object storage, machine learning and artificial intelligence.
According to a study by Seagate, up to 68 percent of business data goes unused. As a result, companies are missing out on the benefits that data should provide. At the same time, they face regulatory and compliance risks if they don’t know what data they have and where they store it.
To solve this problem and make data work for the business, companies need to look at their data architecture. At its simplest level, data architecture is about knowing where an organization’s data resides and mapping how data flows through it. However, given the myriad of data sources and ways to manipulate and use data, there is no single blueprint for this. Each organization needs to create a data architecture that meets its own needs.
But part of the problem for CIOs and CDOs is that technology leads to increased complexity in both data management and use. As the consulting firm McKinsey noted in 2020, technical additions — from data lakes to client analytics and streaming platforms — have made data architecture extremely complex. This makes it difficult for companies to manage existing data and deliver new capabilities.
The shift from traditional relational database systems to much more flexible data structures – and the collection and processing of unstructured data – gives organizations the ability to do much more with data than ever before.
The challenge for CIOs and CDOs is to connect these capabilities to business needs. Creating a data architecture should be more than just maintaining IT or ensuring compliance.
What is data architecture
A data architecture is often described as a data management scheme. Of course, an effective data architecture must map the flow of information within an organization.
This, in turn, relies on a good understanding of the data being collected and stored, the systems in which it is stored, and the regulatory, compliance and security regimes that apply to that data.
Companies also need to understand which data is critical to operations and which data provides the most value. As organizations store and process more and more information, it becomes increasingly important. Sometimes it’s more art than science.
Data architecture must be tied to an organization’s data strategy and data lifecycle, but it also depends on sound data management.
Often organizations divide their data architecture into two parts: data provisioning and data consumption or use.
On the provisioning side, CIOs and CDOs need to look at data sources, including transactions, business applications, customer actions and even sensors. On the data consumption side, it can be about reporting, business intelligence, advanced analytics, and even IO and AI. Some companies may also seek to further use data by selling it or using it to create new products.
Why and how to implement data architecture
The reason for creating or updating a data architecture can be either a change in technology or a change in business.
Changing the core component of an organization’s IT or analytics systems provides another way to look at data flows. And moving to the cloud offers a way to update data flows without having to lift and shift systems. At the same time, changes can be made for each application or each project.
The shift from data warehouses to data lakes also facilitates this, as data no longer has to be tied to specific applications.