“An Overview of Methods for Handling Missing Values”

Abstract

The problem of missing values has existed since the earliest attempts to use data as a source of knowledge, as it is an inherent part of the process of collecting, recording, and preparing the data itself. It is all the more unavoidable given that vast amounts of data are currently being collected from various sources: “One of the ironies of Big Data is that missing data plays an increasingly important role.” There is a vast body of literature on this topic, and a recent survey even identified more than 150 different implementations.

In this presentation, I will share my experience on this topic. I will begin by discussing the inferential framework and then show how missing values create additional challenges for supervised learning, since traditional machine learning algorithms cannot handle incomplete data. Finally, I will illustrate the impact of methods developed in the field of causal inference for estimating treatment effects from clinical data.