In this blog, we’ll discuss all things data quality and data observability. We’ll present interviews with leading data scientists, and we’ll discuss our own working building data observability solutions for data engineers and data scientists working in a broad range of industries.
If you’re responsible for data pipelines or any other aspect of your company’s data, you’re busy. You might want to try out new data services, but you probably don’t want to get bogged down in lengthy vendor negotiations or complicated configurations for trials....
One of the fundamental challenges of data quality is freshness: Did the data pipelines run and push new data into your data warehouse tables as expected? With all the tools available for transforming and loading data, it can be difficult to ensure that all data has...
A question we often get asked at Data Culpa is, “If you guys are going to monitor our data, isn’t that going to take a lot of custom engineering to set up?” The answer is, “No.” Data Culpa Validator can be set up in less than an hour. No lengthy professional services...
Every data team needs to keep an eye on its data. At the same time, no data team wants to be deluged with alerts. Ideally, alerts should direct your attention to the most important changes taking place in your data. If they call attention to every change, you’ll...
Observability is the analysis of a system based on its outputs. By analyzing the outputs of a system at various points, it should be possible to infer the internal state of the system and to diagnose problems the system is experiencing. This sounds useful for data...
What’s your data doing? Can you tell? Data teams tell us they need better visibility into data pipelines, integrations, repositories, and data lakes. You can hand-code a bunch of unit tests to check for known boundary cases. But coverage will always be limited. And...
A company’s data is its most valuable asset — that's a commonplace observation in board rooms and IT labs today. Every organization recognizes the importance of its data. Somewhere in the mission statement for most digital transformation projects, there’s probably a...
As a leader in healthcare innovation at Philips Healthcare, Shawn Stapleton manages a portfolio of technology solutions that have immediate impact for hospitals around the world. Shawn works with a network of clinical partners to identify key needs and to develop...
As a leader in healthcare innovation at Philips Healthcare, Shawn Stapleton manages a portfolio of technology solutions that have immediate impact for hospitals around the world. Shawn works with a network of clinical partners to identify key needs and to develop...
Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability...
Gordon Wong has been working with data since the early 90’s. His accomplishments include building Fitbit’s data analytics platform from the ground up and serving as HubSpot’s VP of Business Intelligence. He has also worked or consulted for leading brands such as...
Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability...
Gordon Wong has been working with data since the early 90’s. His accomplishments include building Fitbit’s data analytics platform from the ground up and serving as HubSpot’s VP of Business Intelligence. He has also worked or consulted for leading brands such as...
Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability...
Data engineering teams need to be able to find and fix data quality problems quickly. In part 2 of our interview with Michal Klos, Michal discusses the challenges of discovering and troubleshooting data quality problems in pipelines. Michal is Sr. Director of...
Data quality is an ongoing challenge for data scientists and data engineers. In this interview, Michal Klos share the insights he's gleaned from a long career building data pipelines and ensuring they deliver high-quality results. Michal is Sr. Director of Engineering...
At Data Culpa, we're building solutions to help data engineers and data scientists catch data quality problems before they jeopardize data pipeline results. Nearly every business today is becoming more data-centric. Data not only drives operations and decision making;...
Here’s what we’re writing on Medium:
The Data Quality Gap (June 11, 2020)
4 Lessons for Data Scientists from the UK’s A-Levels Algorithm Debacle (August 18, 2020)
Feel free to reach out to us!
Subscribe to the Data Culpa Newsletter