Metaplane’s New Monitoring Platform: Product Deep-Dive

If you’ve followed Metaplane for a while, you’ll know that the first half of 2023 was a big one for us.

We shipped tags and monitors as code. We built our first public API. We further invested in our data CI/CD functionality with data test previews. We redesigned nearly our entire app, from lineage to incidents to monitor management. And we completed several new integrations: Databricks, Fivetran, Microsoft Teams,  Power BI, and dbt Core. All in the first few months of the year (we’re pretty amazed, too).

And then…radio silence. Not because our software engineers were taking a well-deserved break (although they’ve certainly earned one). But because we were working on something big. Something like fundamentally re-architecting how Metaplane monitors your data.

As we wrote in our recent blog post, Metaplane’s monitoring platform is now more powerful, more flexible, and easier to use than ever. This platform comes with a slew of new updates, including:

• An entirely redesigned monitor page
• A more powerful and intuitive way to build group by monitors (formerly called partition monitors)
• Greater visibility into underlying data for Snowflake customers
• The ability to accelerate our development on monitors going forward

Monitors, but better

The next time you go to a monitor page in Metaplane, you’ll notice that it’s had a bit of a glow up. We’ve made several changes to filter out the noise, streamline your workflows, and add useful context to help you get the most out of your monitors.

The first thing you’ll notice is that the graph at the top of the page makes the historical state of this monitor abundantly clear. We’ve done away with the traditional y-axis and instead show the range of the data to better contextualize the most recent data points. And it’s now clearer than ever to see the exact values Metaplane’s ML model predicted, when your monitor ran, and when you or someone on your team marked a data point as normal.

Now you can see at a glance whether an observed data point was in the expected range, and if it wasn’t, how much higher or lower than expected it was. For monitors with user-defined manual thresholds, you’ll now be able to see that same information relative to the threshold, adding clarity to the model’s behavior.

You’ll also notice a new “Overview” section at the top right, which shows high-level information about the monitor, including the current status of the monitor, when it was last run, and when it’s scheduled to run next.

We’ve also streamlined the configuration options. There’s now an explicit option to switch the between default, ML model-based monitoring and manual, threshold-based monitoring for special cases. There’s also a new option which allows you to select a specific range of dates to incorporate into the prediction model—useful for major seasonal spikes or significant changes to your data. These configuration options, plus all the ones you’re used to, are just a click away in a side menu.

Powerful and intuitive group by monitoring

As part of this update, we’ve overhauled how we do group by monitoring (formerly called partition monitoring), a feature which lets you monitor data by distinct segments within a table, such as source or customer. Metaplane has two “modes” for building group by monitors: a  simple mode, which lets you segment by a single (or multiple) columns, and a custom mode, which lets you write complex SQL to determine what Metaplane should track. The simple mode is much the same as the previous version, but if you try building a custom partition monitor, you’ll find that the interface is significantly more powerful and flexible.

Let’s say you want to monitor your conversions grouped by channel and country. Starting today, you can navigate Metaplane’s familiar Custom SQL Monitor UI. Then you simply write your SQL, label the column you’d like us to monitor the value of, and test your query.

Once you’ve created a group monitor, you’ll notice that there’s now a new interface for viewing it. Just navigate to the table where you’ve applied the monitor and select the monitor in the “Monitors” table. From the new group by monitor page, you can see the status of all the segments at a glance, as well as update the configuration for all segments in the monitor and dig into the data for an individual segment.

We’re delighted to add that this update has let us add a much-requested feature: the ability to add multiple different segment monitors to a single table. 🎉

More visibility into your metadata

Most companies choose not to monitor everything in their database—for good reason. Between the noise and the cost, often it’s simply not worth it. But that doesn’t mean that the data in those unmonitored tables isn’t important, or that you won’t sometimes want to dig into the metadata.

Now, for Metaplane users with Snowflake or BigQuery warehouses, we’ll be capturing snapshots of the freshness and row count of every table or view in your database. You won’t get an alert about any drops or spikes in this data, but if you notice something’s amiss and you want to dig in, that historical metadata will be available to you.

And, if at some point down the road, you decide you want to add a monitor to the data, you won’t need to wait for the standard 3-5 day training period. Since Metaplane already has the historical data, we can create the model and have you monitoring that table’s row count and freshness in moments.

Best of all, because we’re able to get this metadata through Snowflake’s information schema, this extra context only costs pennies.

Accelerated development for monitoring capabilities

While there’s a lot to love about this update inside of the Metaplane app, the biggest update is under the hood. We’ve completely rewritten how we do monitoring to be more performant and more flexible. This means that we’ll be able to accelerate our development speed when it comes to new monitor types and capabilities going forward, getting useful updates into your hands faster.

We hope these updates were well worth the wait, and we’re so grateful for the input of our customers as we architected a platform built to solve their thorniest data observability challenges. For Metaplane users, you can see those polished new screens and try out the new features today.

If you’re not yet a Metaplane user, you can sign up for a 14-day free trial today, and use many of these new features forever as a part of Metaplane’s free tier.