How Appcues reduced data quality issues by 77% using Metaplane, Snowflake, and dbt

“The important thing is that when things break, I know immediately—and I can usually fix them before any of my stakeholders find out.”

Andrew Mackenzie
Business Intelligence Architect
77%
Reduction in data quality issues
10X
Reduction in WTF messages from stakeholders
Appcues is the most flexible way to build personalized product experiences. From user onboarding through adoption and retention.
http://appcues.com
Industry
SaaS
Size
90 employees
STACK
Snowflake
dbt
Looker
Metaplane

Not enough resources, difficulty storing large amounts of data, and not enough trust

As a growing company, Appcues depends on data for product, operations, sales, marketing, and customer success. But the data team is small– Andy Mackenzie, Business Intelligence Architect at Appcues, is the sole individual working exclusively on data.

❝Virtually everyone in the company uses the data I produce on a daily basis,” said Andy. “They count on me to keep that data clean and error-free.

To serve his team, Andy has to be as efficient as possible. “The faster I can rapidly build, test, and publish changes to our data schema, the more effective I can be,” he said.

Andy’s job centers around managing data and then piping it into Snowflake and other third-party tools. He also manages that data with dbt and ETL to prep it for Looker, the team’s visualization tool.

Faced with not enough resources, difficulty storing large amounts of data, and a general loss of trust in data in the organization at large, Appcues turned to Snowflake, dbt, and Metaplane and saw substantial improvements in data management.

Specifically, these three tools together allowed Andy and the Appcues team to scale performance, iterate quickly, and monitor data quality.

Snowflake + dbt + Metaplane = The perfect combination

Andy and the Appcues team adopted Snowflake, dbt, and Metaplane to help solve their challenges. But why did Andy select each tool? And, how does using these tools together lead to success?

Snowflake offered integrations and the ability to scale performance quickly with simple SQL commands

Andy reviewed a number of tools before settling on Snowflake, ultimately finding that its ability to scale and wide set of integrations set it apart from the pack.

❝Every tool seems to integrate with Snowflake, so the speed with which you can get data into Snowflake is fast

Snowflake is incredibly easy to set up and affordable, even for a team of one. With its deep ecosystem of integrations, Snowflake allowed Andy to adopt other best-in-class tools that helped increase his productivity and leverage.

❝Snowflake has a ton of native integrations with our other data pipeline tools like Segment which can send events to Snowflake

Not only that, but Snowflake can easily scale when Andy needs to run compute-intensive queries, like running analysis over product event data. This can be done with simple SQL commands.

“At any given moment, we could be dealing with 10s of billions of rows. If I need to work with a really large data set and it's worth spending the compute to get that query back in a reasonable time, I can run a simple SQL command to scale my warehouse, and I immediately get that compute power and run my query on those 10 billion rows,” said Andy. “This type of scaling is incredibly easy with Snowflake but it isn’t with other tools.”

dbt was an inexpensive, flexible ETL tool that saved Appcues substantial time and money

Alongside Snowflake, Andy needed an ETL tool that could combine data from multiple data sources. He knew of a number of proprietary tools, but they were too expensive. He also considered building his own tool, but it would’ve taken a substantial amount of time, ultimately wasting resources.

“dbt was open source and it was free for me to get started, so it was a no-brainer to try it out,” he said. Being able to quickly see how it fit within his workflow made it easy for him to adopt the tool. Andy was able to deploy dbt to production in just under 24 hours. Features like Jinja templating allowed him to write hundreds of lines of SQL that would have been thousands of lines of unmaintainable SQL queries if he wrote them without dbt. 

“There’s no way I could do what I do on my own if I didn’t have a tool like dbt,” he said. “Without dbt, you’d need a data engineer to pair with me all the time. Plus, I actually have free time to do analysis, which I definitely wouldn’t if I wasn’t using dbt.”

Andy estimates that using dbt has saved the business tens of thousands of dollars due to not needing to increase engineering resources or spend hours maintaining models.

Metaplane, a data observability solution that offered visibility and insight into data

To complement Snowflake and dbt, Andy also needed a data observability solution. This was particularly important, as he wanted to be the first one in the organization to know about data issues.

Before Metaplane, data quality issues were flagged by Andy’s stakeholders — sometimes several times in one week.

“There were weeks in which I spent much of my time dealing with data quality issues that my colleagues identified,” Andy said. “I wouldn’t even know about an issue until someone pinged me – it reflected poorly on me.”

Without a data observability platform, Andy was flying blind. He would try to get it right the first time when he was pushing out changes, but if something broke, he’d need to go and fix it. This process wasted time and burned him out.

Appcues leverages Metaplane's integrations with Snowflake, dbt, and Looker to alert on data incidents, provide root cause analysis, and show downstream impact to BI dashboards.

Then Andy found Metaplane. He liked that it was free to try and that he could automatically add tests across his warehouse. Ultimately, Andy configured Metaplane to monitor data quality across hundreds of tables in Snowflake and dbt, a task that would have been impossible for Andy to do on his own. 

Plus, he could receive alerts in Slack and adjust the machine learning models with the click of a button. This allowed him to quickly learn how his data behaves and be the first to know about data quality issues, not the last.

Easily store and model trusted data all in one place with fast deployment

By using Snowflake, dbt, and Metaplane, Andy is better positioned to help his team. He’s able to easily store and model trusted data all in one place and make changes quickly and efficiently. 

“Snowflake and dbt are huge factors in enabling me to get our data to our tools and ultimately at the fingertips of our sales team,” said Andy. “Using these with Metaplane ensures that not only is the data where it needs to be, but that it’s the right data that can help my team make decisions.”

Results

  • Appcues now has a scalable data stack built on Snowflake, dbt, and Metaplane that can service the entire organization and scale with a growing team
  • Snowflake offered integrations and the ability to scale performance quickly with simple SQL commands, making it easy for one individual to manage
  • With dbt, Appcues could easily scale up writing data models without needing to build their own framework or hire additional data engineers
  • After adopting Metaplane, Andy now experiences 77% fewer data quality issues.  The entire team is more productive because they spend on average 20 to 40 hours fewer per month investigating issues with their data.

More customer stories

Rinsed washes away data quality issues with Metaplane
Census Activates Data Quality Improvements with Metaplane
Veronica Beard sets up a data stack (almost) as good looking as their clothes
Parachute Home Makes Their Data as High Quality as their Bedding with Metaplane
Dribbble designs a data quality solution with Metaplane
Metaplane keeps data right for Upright
Metaplane propels LogRocket's data quality forward
How Clear Street uses Metaplane to Prevent $100M+ Worth of Data Quality Issues
How Sigma uses Metaplane to track impacts to Sigma Workbooks
Vivian Health Improves Data Quality with dbt and Metaplane to Connect Healthcare Professionals with their Dream Job
Gorgias knows their data is always accurate and gorgeous with Metaplane
How Clearbit cleared up data quality to increase customer retention and improve detection time by 3x+
How Teachable Used Metaplane and Sigma to Achieve Their Data Quality and Visibility Goals
How SpotOn reduced time to actionable data by 6x and increased data engineering contribution by 8.5x using Snowflake, dbt Cloud, and Metaplane
​​How Car and Classic’s adoption of Metaplane, Snowflake, and dbt led to a 10x improvement in report load time and ensured trust in the data
How Imperfect Foods uses Metaplane, Snowflake, and dbt to break down data silos
How Metaplane, Snowflake, and dbt help Vendr run a lean and adaptable data team
How Mux increased test coverage from 10% to 95% with just a few clicks
How Reforge used Snowflake, dbt, and Metaplane to 3x their team and save 18 hours a week
Ensure trust in data

Start monitoring your data in minutes.

Connect your warehouse and start generating a baseline in less than 10 minutes. Start for free, no credit-card required.