Chapter 13: Missing Data¶
In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import statsmodels.formula.api as smf
Read in a missing version of the Chicago Insurance data:
In [2]:
chmiss = pd.read_csv("data/chmiss.csv", index_col=0)
chmiss.head()
Out[2]:
Describe does give observed count but can only tell the number of missing by subtraction
In [3]:
chmiss.describe().round(2)
Out[3]:
Number of missing values per variable
In [4]:
chmiss.isna().sum(axis=0)
Out[4]:
Number of missing values per case
In [5]:
chmiss.isna().sum(axis=1)
Out[5]:
Make a plot of the missing data:
In [6]:
plt.imshow(~chmiss.isna(), aspect='auto')
plt.xlabel("variables")
plt.ylabel("cases")
plt.gray()
plt.show()
Full data model
In [7]:
chredlin = pd.read_csv("data/chredlin.csv", index_col=0)
lmod = smf.ols(formula='involact ~ race + fire + theft + age + np.log(income)', data=chredlin).fit()
lmod.summary()
Out[7]:
Missing data model
In [8]:
lmodm = smf.ols(formula='involact ~ race + fire + theft + age + np.log(income)', data=chmiss).fit()
lmodm.summary()
Out[8]:
Mean values for the variables
In [9]:
cmeans = chmiss.mean(axis=0)
cmeans
Out[9]:
Fill in missing values with means
In [10]:
mchm = chmiss.copy()
mchm.race.fillna(cmeans['race'],inplace=True)
mchm.fire.fillna(cmeans['fire'],inplace=True)
mchm.theft.fillna(cmeans['theft'],inplace=True)
mchm.age.fillna(cmeans['age'],inplace=True)
mchm.income.fillna(cmeans['income'],inplace=True)
imod = smf.ols(formula='involact ~ race + fire + theft + age + np.log(income)', data=mchm).fit()
imod.summary()
Out[10]:
In [11]:
lmodr = smf.ols(formula='race ~ fire + theft + age + np.log(income)', data=chmiss).fit()
mv = chmiss.race.isna()
lmodr.predict(chmiss)[mv]
Out[11]:
In [12]:
chmiss.race.fillna(lmodr.predict(chmiss))[mv]
Out[12]:
Use logit transformation to restrict to [0,1]
In [13]:
def logit(x): return(np.log(x/(1-x)))
In [14]:
def ilogit(x): return(np.exp(x)/(1+np.exp(x)))
In [15]:
ilogit(0)
Out[15]:
In [16]:
lmodr = smf.ols(formula='logit(race/100) ~ fire + theft + age + np.log(income)', data=chmiss).fit()
(ilogit(lmodr.predict(chmiss))*100)[mv]
Out[16]:
Multiple imputation¶
statsmodels has Multiple Imputation with Chained Equations (MICE). Not the same as the R package demonstrated in R.
In [17]:
import statsmodels.imputation.mice as smi
imp = smi.MICEData(chmiss)
fm = 'involact ~ race + fire + theft + age + np.log(income)'
mmod = smi.MICE(fm, sm.OLS, imp)
results = mmod.fit(10, 10)
print(results.summary())
As might be expected, the inference is not as sharp as seen in the full data version with somewhat larger p-values.
In [18]:
%load_ext version_information
%version_information pandas, numpy, matplotlib, seaborn, scipy, patsy, statsmodels
Out[18]: