John Bennett, Author at Data Culpa

Where Most Data Observability Solutions Fall Short

Data Culpa Validator showing cohesion bullseye

Observability is the analysis of a system based on its outputs. By analyzing the outputs of a system at various points, it should be possible to infer the internal state of the system and to diagnose problems the system is experiencing.

This sounds useful for data science and engineering, and it is. Data systems working with large volumes of complex data are obviously difficult to monitor and troubleshoot. A growing number of new software products promise to deliver “data observability” for pipelines and other data systems, so that data engineers and data scientists can discover and troubleshoot problems more quickly. Understandably, there’s a lot of interest in these products.

But there are two shortcomings with this model of data observability: lack of support for external data, and lack of support for historical comparisons.

External Data Sources

The first problem with data observability tools is that today’s data pipelines increasingly rely on external data sources. If external data sources change in unexpected ways, they can corrupt a data pipeline’s results. Measuring outputs of the pipeline at various points will indicate there’s a problem, but it won’t not pinpoint the source.

This isn’t a trivial point. Monitoring external data is different from monitoring internal data. Chances are, someone in your organization is in full control of the schemas for your internal data. A business partner or other third party may have provided a description of their data schema, but chances are you’re not in as close contact with that organization’s data team as you are your own. And chances are, you don’t have the time or budget to write countless unit tests to check for changes in data values and data types in every field of every schema from every external data source.

So you need an automated way of detecting changes in external data — a way that analyzes that data’s schema and values over time and automatically builds a model for that data you can rely on.

This type of modeling lies outside the scope of traditional observability, which is focused solely on outputs from internal processes designed and controlled by the organization itself.

Comparisons Over Time

The second shortcoming is that there are lots of reasons why data outputs might change, especially in the world of Big Data. Sales data, for example, might change because of seasonality, sales promotions, breaking news stories, compliance deadlines, or even weather events that disrupt supply chains or trigger changes in consumer behavior.

To analyze these changes, you need to understand context. Context requires understanding changes over time. Is a surge in data volumes the last week of March a cause for alarm? Or is it tied to economic activity at the end of the quarter? To answer that question, it’s useful to compare that week’s data to data at the end of March a year ago. Or data at the end of December. Or even data at the end of February.

In other words, observability needs to take into account a system’s history. It needs to support historical comparisons, so that, for example, a data team can easily compare one day’s data to another, or one hour’s data to another.

This support for historical comparison needs to be built into the observability solution from the ground up. You can’t just tack something like this on and hope to have any effective way of comparing one massive data set to another. You’ll end up with hugely inefficient tools and processes. You need to think about this problem from the beginning.

Data Culpa Validator Gets Observability Right

That’s what we’ve done at Data Culpa. We’ve designed our data observability platform, Data Culpa Validator, to take into account the concept of time. We make it easy for data teams to compare one batch of data to another, one data stream to that stream from an earlier time period, and one data set to a data set selected by the data team as the “gold standard” for the application.

You need to understand what’s happening with your data. That means understanding that data wherever it comes from — an external data feed or internal data source. It also means understanding how much it varies from what you’re expecting, given the full context of your data application or use case.

We make those comparisons quickly, easy, and powerful with Validator.

Try Validator today, and see for yourself.

Request Early Access

Introducing Data Culpa Validator

Data Culpa Validator User Interface What’s your data doing? Can you tell?

Data teams tell us they need better visibility into data pipelines, integrations, repositories, and data lakes.

You can hand-code a bunch of unit tests to check for known boundary cases. But coverage will always be limited.

And unit tests don’t give you visibility across time.

How does today’s data ingest compare to yesterday’s or last week’s? Are schemas changing? Data values drifting? Is this week’s spike an anomaly or just a typical end-of-quarter surge?

Is your external data feed about to corrupt your results?

Observe Data Changes over Time

At Data Culpa, we’ve built a SaaS service that gives you a birds-eye view of your data.

Using our service, you can:

See what’s happening with your data at a glance.
Compare the data you’re getting to the data you’re expecting. Compare activity across time.
Receive alerts, and dig into what’s happening with your data before it jeopardizes business results.

We call our service Validator, because it helps you validate the data your company depends on.

Our early access program is open. Interested? Sign up today.

Why You Need a Data Quality Strategy for External Data

city scape with bright lights showing data connections

A company’s data is its most valuable asset — that’s a commonplace observation in board rooms and IT labs today. Every organization recognizes the importance of its data. Somewhere in the mission statement for most digital transformation projects, there’s probably a remark about becoming “a more data-driven organization” or “more fully leveraging the data that we have.”

Data is obviously part of what makes a company unique and therefore valuable. No two companies have the same data. (If they did, the less capable of those two companies wouldn’t be around very long.)

But even if your organization has a wealth of proprietary data, your organization’s success probably depends on enhancing that data with external data and broadening the scope of your data applications.

Data scientists recognize this. In a recent Deloitte survey, 92% of data analytics professionals said their companies should increase their use of external data.

A 2018 MIT Sloan Management Review report on data and data analytics found that:

The most analytically mature organizations use more data sources, including data from customers, vendors, regulators, and competitors. “Analytical innovators,” or companies that incorporate analytics into most aspects of decision-making, are four times more likely than less mature organizations to use all four data sources, and are more likely to use a variety of data types, including mobile, social, and public data.

There’s a wealth of external data out there to tap into, everything from transaction data from partners to financial market data to weather data to data feeds from IoT devices — the list goes on.

Ingesting that data and combining it in innovative ways can yield powerful analytical insights. It can also lead to the development of valuable new products and services.

Data Quality Challenges with External Data

But external data brings challenges, too, particularly in the area of data quality.

Your company already has data quality controls in place for internal data, especially structured data such as records in customer databases and financial applications. Your company controls the data objects in those repositories and applications. You can define the data schemas and put data management controls in place to ensure that a customer record in your CRM, for example, matches the format of a customer record in your billing application. If something goes wrong with internal data, you can assign a data steward to investigate the difference and correct it.

You don’t have that kind of visibility and control with external data. Schemas can change unexpectedly. Columns might be added, removed, or reshuffled. Column names might change without notice. And data values might drift slowly or lurch suddenly, breaking analytical results further down a pipeline.

You don’t have that kind of visibility and control with external data. Schemas can change unexpectedly. Columns might be added, removed, or reshuffled.

Ideally, the company or organization providing the external data should warn you about these changes, but the data teams we’ve talked to tell us they hardly ever do.

So data engineers and data scientists discover these changes the hard way — after the changes have already broken a data pipeline’s functioning or skewed its results. In some cases, undetected changes in external data could corrupt business transactions or render a business intelligence report meaningless.

To catch these problems, a data team can sit down and write unit tests, defining expectations for all these columns in all their external data feeds. But that’s a mind-boggling amount of work. There are simply too many fields with too many expectations in too many data feeds to manually write unit tests for all of them. And companies are adding new data feeds and data use cases all the time.

Unit tests or expectations have a role to play. But they’ll never provide the comprehensive coverage that data teams need for discovering problems in external data feeds before those problems jeopardize business results.

Automating Data Quality Analysis with Data Culpa

That’s why at Data Culpa, we’re building a new approach to data quality intelligence, automatically detecting changes in data patterns and alerting data engineers and data scientists in time for them to inspect the changes and, if necessary, take corrective action.

Interested in trying the Data Culpa solution yourself? Contact us at freetrial@dataculpa.com.

Data Quality in Healthcare: An Interview with Shawn Stapleton, PhD, Part 2

As a leader in healthcare innovation at Philips Healthcare, Shawn Stapleton manages a portfolio of technology solutions that have immediate impact for hospitals around the world. Shawn works with a network of clinical partners to identify key needs and to develop novel solutions that improve the quality and efficiency of clinical care delivery. Shawn sits on several steering committees within Philips to develop and implement innovation strategies across multiple businesses, as well as with multiple granting agencies to advise on national research initiatives in healthcare. As Director of Innovation, Shawn strives put research into practice, bringing ideas to market, and working with worldwide leaders to deploy the next generation of healthcare innovations.

Shawn has a PhD in Medical Biophysics from the University of Toronto and a B.S. with honors in Physics and Computer Science from the University of Victoria.

Shawn is an investor in Data Culpa, Inc. and a member of the company’s Board of Advisors.

Read Part 1 of our interview here.

Data Quality and the Lack of Standardized Data Tools in Healthcare

Data Culpa: At Data Culpa, our big theme is data quality. I’m wondering if a data team working with an ad hoc assortment of tools as you’ve described creates any data quality challenges. Does a lack of standardization pose any risks for the consistency or quality of data?

Shawn Stapleton: There are many layers of complexity in this question. I’ll take you through how data is generated and flows from source to data scientist in healthcare to try and shed light on data quality challenges in healthcare.

Healthcare data is largely produced to drive the business of healthcare delivery. We tend to think that healthcare data primarily consists of clinical information: lab results, genomic information, radiology images and reports, pathology reports, and so on.

Healthcare data is largely produced to drive the business of healthcare delivery.

However, there’s an entire operational machinery to drive a patient examination and the generation of a clinical result. Ordering, exam scheduling, staff scheduling, examination workflow, and billing are all examples of operational healthcare data.

Today this data is stored in a collection of hospital information systems. The main system we tend to discuss is the electronic medical record (EMR) system. In reality, information is also stored in lab information systems, radiology information systems, billing systems, scheduling systems, and more. Some of these systems are department-dependent or are duplicated across departments. Rarely are these systems static. Systems are being upgraded, added to, and migrated over time. The integration, support, and maintenance of these systems results in data quality issues.

Nowadays healthcare data is largely digital. But this wasn’t always the case. Back in the day, hospitals , clinics, and private practices managed their business using paper records. The switch to digital is actually quite recent, starting in the 80’s and only really become widely adopted after passing of the HITECH Act, which was part of the American Recovery and Reinvestment Act (ARRA) in 2009 which introduced the concept of meaningful use. The switch to digital meant that a paper cased records need to be digitized to maintain continuity of patient care. This been an ongoing and challenging process, fraught with data quality issues.

Nowadays healthcare data is largely digital.

So what we have in healthcare today is a collection of operational and clinical data stored across multiple systems, with varying data quality. This has profound implications for data science and machine learning. Getting access to data means working closely with the hospital IT to query multiple systems. When you run in data linking and/or data quality issues, or missing data you need to go back to hospital IT to understand the data quality issues and/or get new data. Hospital IT is not in the business of machine learning; they’re in the business of supporting hospital operations. So, you can imagine this process is time consuming. Sometimes you settle for the data you get, and do your best to remove data quality issues with little understanding if the data you removed is indeed that was important and representative.

So what typically happens for data science projects in healthcare is the collecting of heterogeneous data with limited metadata providing context about how the data was collected and how the data is changing over time. Data was likely integrated from multiple healthcare systems, each with varying data quality challenges. The evolution of healthcare systems and devices result in data drift and increased potential for new data quality issues to arise. Without context, data scientists rely on subject matter experts who has been around long enough with knowledge of the data and context in which it was generated.

What typically happens for data science projects in healthcare is the collecting of heterogeneous data with limited metadata providing context about how the data was collected and how the data is changing over time.

Data Culpa: Do you see a movement underway to standardize tools and processes to make these changes more likely? For example, let’s say a hospital group decides they want to improve radiology outcomes for their patients, and they think that applying AI might be able to do that. By now, they probably recognize that they need to address data quality problems like data drift, and they need to standardize tools. Do you see any sort of top-down initiative looking at addressing this as an issue of enterprise architecture?

Shawn Stapleton: Absolutely. I mean, it’s top-down and bottom-up, and sideways and it’s coming from every side. There’s a lot of work being done here, but I don’t believe there’s any solid production tools to accomplish this. The real challenge in my mind is understanding data quality across an intertwined collected of systems that evolve over time. There’s substantial value in provided insights into data quality issues that are connected across data sources used in healthcare and processing pipelines used for machine learning.

TO BE CONTINUED.

End of Part 2. Go back and read Part 1.

Data Science in Healthcare: An Interview with Shawn Stapleton, PhD, Part 1

Shawn has a PhD in Medical Biophysics from the University of Toronto and a B.S. with honors in Physics and Computer Science from the University of Victoria.

Shawn is an investor in Data Culpa, Inc. and a member of the company’s Board of Advisors.

Building Actionable Analytics for Healthcare Data

Data Culpa: Can you tell us about your current position and how you got started in data science?

Shawn Stapleton: I’m the Director of Innovation at Philips, developing innovative solutions for radiology departments. My team’s focus is largely on building actionable analytics and AI on top of hospital big data. I manage an innovation portfolio to drive continuous improvement in the quality and efficiency of delivery of radiology services. This includes everything from managing a patient’s order, scheduling and executing an examination, helping radiologists’ interpretation of the images, and ensuring that appropriate follow-up clinical care based on radiologists’ recommendations are being communicated and managed appropriately.

My team is really on the forefront of this area. We work really closely with hospitals around the country – in fact, around the world. Our mission is to develop and prove out novel solutions that have broad impact using technologies like artificial intelligence, simulations, statistics, business intelligence – just a whole gamut of tools. Some solutions are solely data aggregation and retrospective business intelligence, and some are focused on predictive algorithms. Through collaborations with hospital IT and healthcare workers, we validate our solutions through proof of concept studies, and then scale our solution to multiple hospitals as a product.

Through collaborations with hospital IT and healthcare workers, we validate our solutions through proof of concept studies, and then scale our solution to multiple hospitals as a product.

Using Machine Learning to Model Drug Interactions

How did I get here? It was a long journey. I started off studying computer science and physics in my undergraduate years. It was clear that software development would be useful in my “toolbox” of skills, and I primarily focused on the application of physics in healthcare.

For my PhD, I specifically focused on understanding where cancer drugs go in the body and what that means for cancer treatment response. I essentially integrated disparate data sources together to figure out where a cancer drug goes in the body after injection into a patient. I worked with a multi-disciplinary group of scientists to establish if it can be reliably tracked with imaging, building mathematical models to understand how a patient’s biology affects where the drug goes, and use machine learning to predict how much drug would be delivered to the cancer, with the ultimate goal of providing more context on response to that drug.

This was the origins of my data science career, although I was saddled with the need to collect and manage my own data. I didn’t have access to a database of already collected data. I continued this work as a post-doc at Harvard Medical School, at the Center for Systems Biology located in Massachusetts General Hospital. Essentially continuing to increase the types of data I was collecting to add more specificity to the modeling of drug transport in the body and predicting drug delivery to cancer.

At the end of my post-doc, I had an urge to move away from the research bench. Collecting my own data was getting tedious! I wanted to work with real healthcare patient data. So I joined Optum, which is a division of UnitedHealth Group. At Optum, I focused on both educating and leading a team of data scientists to develop machine-learning-based tools that could improve healthcare. We focused on operational delivery of healthcare and on building clinical decision support tools.

The Challenge of Hiring and Educating Data Scientists

As I mentioned, a large portion of my role was to educate junior data scientists. At that time, it wasn’t easy to hire data scientists. Moreover, no one was really sure what a data scientist role really meant, so how do you hire into it?

To me a data scientist was fundamentally a scientist, which resonated with the leadership at Optum. I tried to instill in my team the ability to develop a solution to a complex problem using solid scientific processes. Machine learning, statistics, etc. were just methodologies and tools used to gain insight into the data. I was then recruited by Philips Research as a senior data scientist, focusing on AI applied to radiology.

To me a data scientist was fundamentally a scientist . . . . I tried to instill in my team the ability to develop a solution to a complex problem using solid scientific processes.

The hot topic was, and still is, building AI algorithms to support a radiologist in their task to identify and classify abnormalities in medical images. I progressed through Philips being promoted to a principle scientist, and then to my current role as the director of innovation.

The Evolution of Data Science Toolkits

Data Culpa: Is there a standard sort of data science tool kit that you’re working with?

Shawn Stapleton: The evolution of data science toolkits is absolutely fascinating. Tools have matured remarkably from when I started. I started in MATLAB, which had limited machine learning tools libraries. You could build a neural network in MATLAB, but it wasn’t turnkey. I had no idea how to make it work and I also didn’t have enough data to do anything remarkable. TensorFlow didn’t exist, and scikit learn certainly wasn’t popular amongst my healthcare scientist colleagues.

Today, there are numerous open-source and proprietary/commercial tools for machine learning. So what tools are my team and I working with? Typically the “tool du jour” and/or the tools they have the most experience with. It’s less coordinated than one might expect. From an innovation standpoint, it’s great not to be limited by conforming to a specific toolset. I like the mindset of “use what you like.”

However, from a product development standpoint, this mindset is a nightmare. During my academic career using whatever tool I wanted never posed any problems, since sharing code or data wasn’t commonplace, and the goal wasn’t to build a product. However, in a company it can be hard to take innovations built using multiple different tools and transfer them into a unified product, particularly in a the highly regulated healthcare environment.

From an innovation standpoint, it’s great not to be limited by conforming to a specific toolset. I like the mindset of “use what you like.” However, from a product development standpoint, this mindset is a nightmare.

We use tools like Docker to help manage this challenge, but there’s still complexity in the flow of data across microservices. We use tools like MLFLow to help track machine learning experiments. However, the management of data needed to build algorithms isn’t straightforward; it’s always a struggle.

A Lack of Tools for AI and Machine Learning

There is a lack of tools to manage the data pipeline for machine learning in healthcare. Healthcare data is stored by the hospital across multiple disparate systems; and it’s not generally available to researchers or companies for building algorithms. Moreover, the ability to gather a high-quality healthcare data set isn’t straightforward. I need to collaborate with hospitals’ healthcare workers and IT to access, collate, and create a data set that’s sufficient to build AI algorithms.

The tools to deploy AI algorithms add an additional challenge. There is not a straightforward method to deploy AI in healthcare, but companies like Philips are making huge strides to tackle this problem. Once an algorithm is deployed, there is still a major challenge keeping the algorithm “fresh.”

Healthcare data is constantly evolving, and algorithms need to be updated to reflect these changes. Until recently, there haven’t been any tools that could manage and monitor the flow of data into AI algorithms to ensure they are behaving as expected. In my opinion this is a very cool topic. It’s why I’m so excited to be working with Data Culpa now.

End of Part 1.

Accounting for Bias in Data Analytics: An Interview with Lauren S. Moores, PhD, Part 3

Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability and build life sciences companies. She is also Chair of the Data Advisory Board for USA for UNHCR, a non-profit organization supporting the UN Refugee Agency, and a Teaching Fellow at the Harvard Business Analytics Program. We spoke to Lauren about building data science systems for companies of all sizes, ensuring those platforms maintain high data quality, and accounting for bias in data analytics. (Read Part 1 of our interview here and Part 2 of our interview here.)

Continue reading “Accounting for Bias in Data Analytics: An Interview with Lauren S. Moores, PhD, Part 3”

Data Coverage and Other Pillars of Data Science: An Interview with Gordon Wong, Part 2

Gordon Wong has been working with data since the early 90’s. His accomplishments include building Fitbit’s data analytics platform from the ground up and serving as HubSpot’s VP of Business Intelligence. He has also worked or consulted for leading brands such as ezCater, edX, and ZipCar. In all his engagements, Gordon focuses on repeatable processes, continuous insights, and team building. In Part 2 of our interview, we spoke to Gordon about data coverage and other pillars of data science, and the importance of testing for data quality. You can read Part 1 of our interview here.

Continue reading “Data Coverage and Other Pillars of Data Science: An Interview with Gordon Wong, Part 2”

Different Roles in Data Science: An Interview with Lauren S. Moores, PhD, Part 2

Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability and build life sciences companies. She is also Chair of the Data Advisory Board for USA for UNHCR, a non-profit organization supporting the UN Refugee Agency, and a Teaching Fellow at the Harvard Business Analytics Program. We spoke to Lauren about building data science systems for companies of all sizes, ensuring those platforms maintain high data quality, and different roles in data science for engineers and subject matter experts. (Read Part 1 of our interview here.)

Continue reading “Different Roles in Data Science: An Interview with Lauren S. Moores, PhD, Part 2”

Continuous Analysis, Continuous Insights, and Data Quality: An Interview with Gordon Wong, Part 1

Gordon Wong has been working with data since the early 90’s. His accomplishments include building Fitbit’s data analytics platform from the ground up and serving as HubSpot’s VP of Business Intelligence. He has also worked or consulted for leading brands such as ezCater, edX, and ZipCar. In all his engagements, Gordon focuses on repeatable processes, continuous insights, and team building. We spoke to him about data science, the surprising staying power of SQL, and data quality.

Continue reading “Continuous Analysis, Continuous Insights, and Data Quality: An Interview with Gordon Wong, Part 1”

Building Data Science Systems: An Interview with Lauren S. Moores, PhD, Part 1

Lauren S. Moores, a.k.a. “the Data Queen,” is Head of Data Sciences Innovation for FL65 and Invaio Sciences, two companies launched by Flagship Pioneering, a venture capital and private equity firm dedicated to creating breakthroughs in human health and sustainability and build life sciences companies. She is also Chair of the Data Advisory Board for USA for UNHCR, a non-profit organization supporting the UN Refugee Agency, and a Teaching Fellow at the Harvard Business Analytics Program. We spoke to Lauren about building data science systems for companies of all sizes and ensuring those platforms maintain high data quality.

Continue reading “Building Data Science Systems: An Interview with Lauren S. Moores, PhD, Part 1”