Data Complexity: What Is It?

Why is data complexity more common today 

As companies across industry segments – from software, to manufacturing, energy, you name it – digitize and innovate their business models with new product or service offerings, the complexity of their data increases exponentially.

91% of businesses are engaged in digital initiatives (Gartner) 

Digital transformation has been a hot topic for a very long time. If you attended conferences of any kind in the past decade, chances are a couple of sessions have covered it from an industry perspective. And today, we’ve reached that threshold where 91% of businesses are already knee-deep into adding at least a layer of digital. Whether the transformation is a success or not is a whole other point. But today, the world is digital. 

Volume of data/information created, captured, copied, and consumed worldwide will reach 147 zettabytes by 2024 (Statista) 

A consequence of digital transformation is the exponential amount of data being created every year. From social media to SaaS applications, both the B2C and B2B world contribute to generating an increase in digital information … that needs to be captured, processed and analyzed to run a data-driven company. 

67% of enterprises are relying on data integration to support analytics and BI platforms today (Forbes) 

Because data comes from a larger array of sources and types, there is a greater need for data integration tools, like ETL (extract, load, transform), to combine that data into a single, unified view. But not all tools are created equal, and not all of them can handle the volume and variety (ie, data complexity) arising from all the different sources and types.

Common data challenges 

Complexity can occur for any type of data, but for this blog post we will focus specifically for usage data, the record of how and when customers interact with a product or service. 

Companies leveraging usage data typically encounter six main challenges: 

Integration – With the increasing number of data sources, the cost and speed of data integration becomes a challenge for many organizations. 

Real-time processing  – Real-time processing of usage data is critical for organizations that need to make quick decisions based on the data. 

Data volume – Data volumes are exploding, and traditional data processing solutions cannot handle the sheer volume of data generated by various sources. 

Data variety – The intricacy and diversity of the data being processed, including the structure, format, and relationships between data sets. 

Data quality  – Data cleansing, validation, and enrichment capabilities, ensuring that the data is accurate, consistent, and complete. 

Monetization – Usage data is a valuable asset for organizations, enabling them to invoice accurately, create new revenue streams and improve customer experiences.  

Now, going back to data complexity; it usually arises from the combined effects of volume and variety of data, posing challenges for managing and analyzing large volumes of diverse data types and formats. 

Companies that grow in subscriber base and product portfolio encounter this complexity of data, and risk facing inefficiencies, increased propensity for errors and revenue leakage, and potential for damaged customer experience if they don’t cater for it as they scale for growth.  

Since data volume and variety are the root cause of data complexity, it’s also worth mentioning how it differs from big data (another term you might remember from attending conferences…). Big data itself is a combination of volume, variety but also a couple of other “V’s” depending on which definition you refer to, but would generally include velocity, veracity, and value.  

Let’s dig deeper into what data volume and variety are. 

What do we mean by volume of data? 

Data volume is the amount of data being generated, processed, and stored by an organization. 

What it means to growing businesses: 

• Data storage capacity under stress 
• Unacceptable time required to process and support revenues or GTM (1hr –1 day) 
• Cost of processing becomes excessive (additional FTEs or H/W) 
• Data quality deterioration due to analysis process under stress 
• Time and effort required to process data detracts from the core business functions 

What creates volume? 

Dramatic subscriber growth, such as the increasing number of subscribers and their associated records, compounded by organizations relying on data, use of data for analytical purposes and cloud storage. This is prevalent in any type of company transitioning or growing their SaaS business model with recurring revenue.

New data sources: collecting data from multiple sources, such as different sensors or devices, each source will contribute its own data points. The more sources you add, the more data you will have overall.  This is a growing trend in the industrial sector, where newly developed products are “connected” from inception and older products are retrofitted with sensors.

Mergers & acquisitions: bringing together datasets from multiple sources, gaining access to new sources of data, and partnering with other companies to collect data. Large enterprises who have grown through M&A’s might find themselves with a multitude of legacy ERP, CRM, or billing systems, for example.

Additional products or services: more transactions, more customers, more channels, and more features. This increase in data can pose a challenge to manage. 

What do we mean by variety? 

Data variety is the intricacy, diversity and incremental challenge posed by the data being processed, including the structure, format, and relationships between data sets. 

What it means to evolving businesses: 

• Data sources – where the digital information is located. There are two types of data sources: machine data source and file data source. 
• Real-time events – an increasing amount of data needs to be processed and acted upon in real-time (e.g. password verification for users logging into a web product) 
• Data formats – to feed your billing systems or BI tools, the data gathered in various formats will need to be transformed in a unified view. 
• Customized contracts – how unique or customized to each customer are the products or services rendered. 
• Entitlement – understanding which features of a product customers are allowed or enabled to use in order to support add-ons, upsells, or reduce churn. 

What creates the data variety challenge? 

Modified business model: does the company only have one way to sell its products (for example, perpetual licenses) or are the rolling out additional business models such as subscription, consumption-based, etc? 

Additional product: Is the company structured around a single service? Or do they have multiple products? 

Additional company: Is the company a simple and unified entity? Or has it been growing through mergers and acquisitions or rapid organic growth? Duplicate revenue systems could be a challenge. 

Additional partner: Is the company’s service entirely their own? Or is it assembled (or augmented) from an ecosystem of partners? 

In all logic, the greater the combination of the above, the more complex will data be. If a software company, for example, has a hybrid model of selling perpetual licences as well as subscriptions, for multiple products, via different entities, and through a partner ecosystem as well as a direct motion, they will encounter exponentially more complexity than a company that only ticks one of the four variety boxes.

Conclusion  

As companies digitize their business models and start to develop new services or products, data – and usage data, in particular – begins to play an increasingly important role for success. But with growth comes a larger set of data volume and variety; organizations need to have that in mind and a plan to be able to deal with it as their businesses scale up, otherwise they will eventualy face unmanageable data complexity.

Thomas Igou

Head of Content Marketing @ DigitalRoute

Thomas heads the content marketing team at DigitalRoute. He’s been working with content his whole career, in various formats (hosting podcasts, organizing conferences, writing blogs and reports) and on various topics (usage data, digital sales room, servitization, IT security, i-gaming).

He has two passions in life: football and music. When he’s not playing, watching, or coaching a football game, you’ll likely find him strumming his guitar to a classic 70’s rock song (very likely a Led Zeppelin one).