The Cloud Data Warehouse Paradox - Reducing Costs While Encouraging Data Sharing

The Cloud Data Warehouse Paradox - Reducing Costs While Encouraging Data Sharing

by Sachin Wadhwa, CEO and Co-founder of Q Spark Group

You’ve implemented a cloud data warehouse. Good. Employees are adding data and sharing it. Great. Your cloud DW vendor sends you a monthly bill and it’s higher than you budgeted. Uh oh. Welcome to the cloud data warehouse paradox. Balancing the dual priorities of encouraging data sharing and containing cloud data warehouse costs isn’t easy. Over-emphasizing one leads to sub-optimizing the other. Or does it? 

Many organizations have implemented the practice of “Cloud FinOps” to manage and reduce their cloud costs. This includes the costs of cloud data warehouses. Reporting to finance and sitting within operations, these teams are tasked with keeping cloud spending costs under control. Initially, many have good results. Identifying the low-hanging fruit can result in 30% cost reduction, according to Forrester Research. Post migration cloud data warehouse costs are easy to identify and reduce. Temp files sitting around. Copies of instances. Wrong-sized dev environments. Some shutdown actions are easy by identifying old or ownerless cloud instances. Those are zero-conflict situations. But recent instances with valid owners are not so easy to shut down. And right-size actions are always met with some resistance. Why? Conflict. Each of those scenarios is based on you asking people to change and give-up something. The next level of cloud data warehouse efficiency is more difficult to realize. 

When it comes to FinOps, data-centric clouds are more challenging than other types of cloud instances. Cloud data warehouses are the prime culprits. Why? People hoard data. They may not use it, but they think it’s valuable. So they hold onto it for as long as they can. A copy of data might have a few manual modifications made for one analytic use case. That’s all the justification a data hoarder needs to claim they still need that file. Who’s to say what’s valid and what’s redundant? 

That’s where data-centric FinOps comes in. Managing cloud data warehouse costs comes down to striking a balance between the needs of business users and data owners vs. financial objectives. There are three essential characteristics of a cloud data warehouse FinOps team that produce ongoing efficiencies. 

1 – Holistic Understanding of Data, Analytics and Business Functions

The cloud dw FinOps team needs to understand data. Specifically, where it comes from and what its intended uses are. You have to be a data-person to manage data requests. Expertise in cloud data warehouses, usage of data warehouses and marts, temporary files, ad-hoc sandboxes, data lineage and processing, and temporal right-sizing for various environments (production, dev, QA). Temporal right-sizing requires expertise in data patterns. Data surges. Inquiry spikes. Is that a run-away query or a valid one-off analysis? When should it be performed? The data expertise required for cost reduction is multi-faceted.

The team also needs expertise in analytics to determine how much data is required to build and operate advanced analytic and AI models, and for how long that data is needed. And of course, that same team needs expertise in business functions. What are data and analytics used for? What’s driving the request and how long will it be needed? Are business users reading reports and benefiting from the analysis? Ultimately this team needs to make a business judgment between the potential benefit of the data usage and its cost. 

2 – Neutrality

The data team needs to be neutral for many reasons. The first is quite obvious – to make good decisions from a balanced perspective. The second is less obvious, but more important. Buy-in. If the decision-making is perceived by data owners and users to be tough-but-fair, then it is much more likely that they will abide by the decision. You can’t have finance alone dictating costs and constantly asking for reductions without data and business context. You can’t have data-leaders prioritizing requests as they too-often will favor data usage vs. cost reduction. You need neutrality as well as the strong perception of neutrality in order to have an effective cloud data warehouse FinOps program.

3 – Drive Adoption via Change Management

Cloud Data Warehouse FinOps is a change management program! Your cloud DW FinOps team needs to not only make decisions, but help users adopt them. This isn’t just a decision-making body. It’s a change management team. That is where a lot of programs stumble. They think it is as easy as setting rules, publishing them, making decisions, and policing them. It isn’t. Data owners and users will figure out the rules, and if they don’t agree, they will figure out ways around them. There are parallels to other data-centric practices. Take Data Governance, for example. Once upon a time, data governance was seen as a compliance exercise. Rules for use would be written down, published to a shared directory, and everyone would read them and abide by them. Did they? Of course not. Data governance didn’t evolve and become more effective until the people leading it evolved from “people who say no” to “people who understand what the business is trying to do and try to find a way for data to make it possible.” The evolution of cloud dw FinOps will follow the same trend. Why not skip the learning curve and just start out with the principle that “Cloud Data Warehouse FinOps is first and foremost a CHANGE MANAGEMENT EXERCISE.”

The Cloud Data Warehouse Paradox doesn’t need to be a blocker for you. It isn’t really a paradox. It’s only a problem if you take a simplistic approach to cloud data warehouse FinOps. But if you take a more evolved approach, one that creates a team with holistic expertise, neutrality, and a change management mindset, you will solve the paradox. 

Comments

One Response

Leave a Reply

Your email address will not be published. Required fields are marked *

Related blogs

Governing Data and AI
It's Time to Start Governing Data and AI Together
Governing data and AI together is better than governing data and using it for AI. It's required for compliance....
Read More
What is Data Modernization?
What is Data Modernization?
Data Modernization is a continual business strategy that drives utilization of data across the enterprise...
Read More
MDM Integration
The Current State of MDM - Part 4 - How Integrated, and Integral, is MDM?
Modern MDM solutions have impressive Data Quality functionality. But do they have impressive data quality?...
Read More
MDM Data Quality
The Current State of MDM - Part 3 - MDM & Data Quality Have Had a Long Relationship. Is it Time to Renew Their Vows?
Modern MDM solutions have impressive Data Quality functionality. But do they have impressive data quality?...
Read More
Being Together While Staying Apart
The Current State of MDM - Part 2 - The Curious Relationship Between MDM and Data Governance
Twenty plus years after the emergence of MDM and data governance, their relationship remains unfulfilled....
Read More
Current State of MDM
The Current State of MDM - Part 1 - Operational MDM Projects in No-man's Land
Operational MDM projects that are caught in no-man's land. In this sort of project, MDM ingests data...
Read More
7 steps for marketing analytics blog
7 Steps for Successful B2B Marketing Analytics
In the ever-evolving B2B marketing analytics landscape, mastering the art of marketing has become the...
Read More
B2B Marketing Attribution
Turbocharge B2B Marketing Attribution with AI
I started my morning by downloading a research report from a software company’s website. Now, I’m their...
Read More
B2B Marketing Multi-touch Attribution
Two B2B Marketing Analytics Use Cases Every CMO is Talking About
2024 will be an eventful year for B2B marketing analytics. We’ve been working with and talking to a number...
Read More
An analyst uses a computer and dashboard for data business analysis and Data Management System with KPI and metrics connected to the database for technology finance, operations, sales, marketing
Top 5 B2B Marketing Analytics Opportunities in 2024
B2B marketing evolves quickly. In 2024, marketing analytics will drive strategies. Learn the top 5 opportunities...
Read More

Top categories

Popular blogs

Governing Data and AI
It's Time to Start Governing Data and AI Together
Read More
What is Data Modernization?
What is Data Modernization?
Read More
MDM Integration
The Current State of MDM - Part 4 - How Integrated, and Integral, is MDM?
Read More
MDM Data Quality
The Current State of MDM - Part 3 - MDM & Data Quality Have Had a Long Relationship. Is it Time to Renew Their Vows?
Read More
Being Together While Staying Apart
The Current State of MDM - Part 2 - The Curious Relationship Between MDM and Data Governance
Read More
Current State of MDM
The Current State of MDM - Part 1 - Operational MDM Projects in No-man's Land
Read More
7 steps for marketing analytics blog
7 Steps for Successful B2B Marketing Analytics
Read More
B2B Marketing Attribution
Turbocharge B2B Marketing Attribution with AI
Read More
B2B Marketing Multi-touch Attribution
Two B2B Marketing Analytics Use Cases Every CMO is Talking About
Read More
An analyst uses a computer and dashboard for data business analysis and Data Management System with KPI and metrics connected to the database for technology finance, operations, sales, marketing
Top 5 B2B Marketing Analytics Opportunities in 2024
Read More

Sign up for SparkPlug
Q Spark Group's Monthly Newsletter

QSG’s monthly newsletter is filled with insights, best practices, and success stories from our customers’ experiences in utilizing modern technology to improve their business.