How Vendr built a high-leverage data team that saves 8 hours every week
“Metaplane is the most exciting tool in our data stack. It’s the command-and-control center where we have a comprehensive view of the health of our data.”
“You have carte blanche. Paint the vision for where you think we need to be in the next few years.”
That was Erik Edelmann’s mission when he joined Vendr in February 2021. At the time, the company had just 30 employees and no data team to speak of. Their resourceful CTO had dabbled in data, adopting a couple of fundamental tools, but knew they could be doing more.
As Senior Analytics Engineering Manager, Erik was responsible for the entire spectrum of data—from ingesting raw data to crafting models to creating dashboards for reporting. Fortunately, he was empowered not only to re-architect the data stack from scratch, but to develop a forward-thinking vision for how data should be done
“Leadership gave me the space and resources to plan ahead and build a data stack that scales,” Erik said.
But with great power comes great responsibility. To deliver on his promise, Erik needed a technology partner that could empower him to transform his vision into a reality.
Monitoring what matters
Erik’s first goal was to expand his awareness of data quality issues.
He knew schema changes were inevitable, for example. Like any startup, every team at Vendr contributes to data creation and the company’s data models change rapidly as a result. So, he needed to know who changed what as fast as possible.
“This is probably the best team of engineers and product managers I’ve ever worked with when it comes to giving us a heads up about changes,” Erik said. “But they can’t possibly tell us about every change that has an impact on the work we do.”
That’s where Metaplane comes in. The tool’s schema change alerts instantly notify Erik of changes that require his attention, on the channels he prefers, empowering him to follow up on changes within minutes, before downstream issues occur.
Automated anomaly detection also notifies him about any unexpected changes found in the data, which happens once every couple of weeks on average.
“We love getting those notifications because they’re genuinely helpful,” Erik said. “Every time it happens, it feels like we’ve landed on a trampoline from a long fall.”
One alert, for example, revealed that an upstream team member had accidentally updated a contract end date to the Bronze Age. Vendr has built a business on saving customers a significant amount of money on their SaaS contracts. Had the team done so recently, the customer success manager on the account wouldn’t have known. And if a renewal was on the horizon, that would have a huge impact on the company’s bottom line.
“Catching the issue could be the difference between keeping and losing a customer,” Erik said. “We’re a B2B company that cares deeply about each customer. One customer is a big deal for us.”
The confidence Erik has gained, knowing that he’ll be the first to learn about outliers, enables his team to ship work with confidence and speed.
“Only Metaplane empowers us to detect anomalies in our data. There’s nothing else that can so quickly and efficiently enable us to know when numbers are fishy.”
Leading with context
As the company scaled, Erik needed a nuanced understanding of Vendr’s evolving data stack to better serve the organization.
In the past, the process of gaining visibility into the data stack was clunky and disjointed. To understand lineage, Erik had to intentionally break something in dbt to see how it affected the corresponding Looker dashboards. And he had to write separate queries to figure out usage patterns.
Doing things manually took too much time and made it difficult to be comprehensive, which introduced doubt about whether their data was reliable.
After implementing Metaplane, Erik was able to pair the tool’s warehouse-to-BI lineage and usage analytics features together to figure out which dashboards were affected by changes and whether they were still relevant to the company’s business users.
“The combination is a dream,” Erik said. “One or the other is great, but together they’re greater than the sum of their parts”
Over the last fifteen months, Vendr has grown to 300 employees. During that time, Erik has added five data engineers to his dream team.
To maximize the leverage of his team, Erik followed a handful of principles:
- Don’t reinvent the wheel. If a solution to a problem already exists, adopt that solution.
“There’s a proliferation of tools that do a lot of what would previously be fairly cumbersome tasks,” Erik said. “You don’t have to do it by yourself.”
- Make our jobs easier. By investing in things like data testing, governance, and overall data hygiene, the Vendr team is able to prevent the accumulation of technical debt—and ultimately save around eight hours per week.
- Save time for innovation. Erik protects his team’s time, using up just 80% of their bandwidth for day-to-day data work. The other 20% is reserved for special projects that reduce technical debt, like their recent overhaul of dbt.
“We want to have breathing room because we’re humans, but also because that time is necessary to think about how we can be and do better,” Erik said.
- Invest in onboarding. To maximize his team’s capacity, Erik has developed an efficient yet effective onboarding process, which includes time spent interfacing with Metaplane.
“When a new person joins the team, one of the first things I point them to is Metaplane because it empowers them to see how pieces of our stack relate to one another,” Erik said.