Data is a ubiquitous asset that all organizations own at various levels of extent. In 2006, British Mathematician Clive Humby coined the phrase "Data is the New Oil", alluding to the importance and potential, Data can have when treated, refined, and used correctly. Industry experts picked up this phrase and played out for several years to the point that it became cliché. While the original intent was to advocate for effective processing and use of data, we live in times where it is no longer necessary to insist on treating data as a valuable resource. Nowadays, it's an infinite commodity from which a well-established discipline is required to extract value. However, with the exponential growth in the volume of data being created, it is crucial to master the foundational disciplines.
Traditionally, Data points used to be collected far less frequently and only during key points of transactions. Relational databases were often enough to store them, and they were inherently well-structured owing to their simplicity. With the advent of large volumes of structured and unstructured data generated by online entities such as online users, bots, logs, wearable devices, IoT devices, and AI entities, there is an enormous need for data management and governance principles to handle them. In addition to these sources, introducing AI-generated synthetic data will add a substantial volume of data streams that need to be differentiated.
In this article, I will be sharing my thoughts on the three fundamental pillars of the Big Data discipline that bear the load and enable extracting valuable insights from sources of Big Data.
The Data engineering discipline is focused on collecting and ingesting data in a scalable way that benefits organizational standards for Data quality. Infrastructures need to be in place to effectively process and store data that comes in structured and unstructured formats and a blend of both. Data architecture systems consisting of databases, data marts, and warehouses built by data engineers play an important role in preparing the datasets and democratizing them for use by data scientists. Another important role of this discipline is to maintain consistency and quality of data across all delivery systems based on the overall governing principles of the Data organization.
To illustrate the role of Data Engineering, let's consider a hypothetical business that sells wearable step counters. Their data sources may range from structured data, such as sales transactional data collected at the point of checkout, to unstructured chat response data collected each time the website chatbot gets engaged. The Data engineering discipline is responsible for storing and retrieving both types of data in a manner that's interpretable for all data users.
The Data Science discipline is focused on extracting meaningful insights from large volumes of data in the interest of the organization's goals and objectives. Data Science practitioners are skilled in putting data to use to identify patterns based on existing data and predict future outcomes by modeling past events. The Data science discipline is broad and constantly evolving. New aspects of this discipline, such as Machine learning and Deep Learning, involve building predictive models at scale to continuously learn from feedback and eventually operate with little to no human interference.
In our example Wearable step counter business, the Data Science discipline ensures that sales and product performance are on track based on analyzing data collected (Descriptive Analytics), predicting seasonality and likelihood of purchases and customer churn using forecasting (Predictive analytics), process device usage activities at scale to push nudges and notifications on the wearable devices using recommendation engines (Machine Learning).
The Data Governance discipline is focused on creating and maintaining policies and guidelines that ensure that all data sources across the organization are consistently high quality, accessible, and secure. Data governance plays a key role in making data trustworthy, which eventually enables Data scientists to provide objective and unbiased insights. As the complexity, volume, and variety of data sources increase, it is important to have a centralized body of experts and systems that ensure that decisions are made from reliable data sources.
In our example, the wearable step counter business, the data governance guidelines ensure that the sources used by various data science teams are consistent and that the data sources exist as a single source of truth. The organization could have aggregated data such as Total Devices sold or sensitive data such as their username to access their profile. Data governance policies will enforce various levels of security needed to store each of these data points.
As we prepare to enter a big data-dominant world where trillions are data points are generated in real time by both humans and AI equally, solid foundational disciplines are essential for organizations to adapt and thrive. Mastering the three essential pillars of Big Data management I shared in this article will help organizations navigate, innovate, and succeed in the constantly evolving Big Data space.
Nithhyaa Ramamoorthy is a data subject matter expert with over a decade's worth of industry experience in product analytics and big data, specifically at the intersection of healthcare and consumer behavior. She regularly contributes long-form thought leadership and career advice content to various Data Science publications. She is passionate about leveraging her analytics skills to drive business decisions that create inclusive and equitable digital products rooted in empathy. Opinions are her own.
Disclaimer: The author is completely responsible for the content of this article. The opinions expressed are their own and do not represent IEEE's position nor that of the Computer Society nor its Leadership.
