<jupyter_start><jupyter_text>Setup and load data.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format = 'svg' 
# Set some Pandas options
pd.set_option('html', False)
pd.set_option('max_columns', 30)
pd.set_option('max_rows', 10)<jupyter_output><empty_output><jupyter_text>Using 2007 Q1 for now<jupyter_code>data = pd.read_hdf('/var/datasets/dshs/CD2007Q1/reduced_PUDF_base1q2007.h5','data')<jupyter_output><empty_output><jupyter_text>It consists of ~740K claims.<jupyter_code>nclaims = data.Discharge.count()
nclaims<jupyter_output><empty_output><jupyter_text>It can be used to create about 75M diagnosis triplets.<jupyter_code>diagnosisView = data[['Princ_Diag_Code'] + ['Oth_Diag_Code_'+str(i) for i in range(1,25)]]
diagnosis = diagnosisView.copy()
diagnosis.count()
cnt = diagnosis.count().values
diff = cnt[:-1]-cnt[1:]
diff = np.append(diff,cnt[-1])
trigrams=[diff[i-1]*i*(i-1)*(i-2)/6 for i in range(3, 26)]
trigrams
sum(trigrams)
def calcNumberOfTrigrams(nvars):
    trigrams=[diff[i-1]*i*(i-1)*(i-2)/6 for i in range(3, nvars + 1)]
    return sum(trigrams)
def calcNumberOfTrigramsJoin(nvars, claimsperpatient=10):
    trigrams=[diff[i-1]*(i*claimsperpatient)*((i*claimsperpatient)-1)*((i*claimsperpatient)-2)/6 for i in range(3, nvars + 1)]
    return sum(trigrams)
nvars=range(3,12)
ntri = [calcNumberOfTrigrams(x) for x in nvars]
ntrij = [calcNumberOfTrigramsJoin(x) for x in nvars]
plt.bar(nvars, ntri)
plt.title('PUF 2007 Q1')
plt.xlabel('Number of diagnosis features used')
plt.ylabel('Number of extracted triplets')<jupyter_output><empty_output><jupyter_text>The above is sligtly incorrect as there are repetitions in the list of codes for every claim. Most importantly:  Other stats to consider: - Major hospitals- Popular diagnoses- Basic demographics- Rural vs urban- Got data for 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006 and 2007: cross reference with major events, seasonal occurences.<jupyter_code>dif = diff.tolist()
dif.insert(0,0)
diff = np.array(dif)
diff
sum(diff)
nclaims
cdf = np.cumsum(diff)/float(sum(diff))
pdf = diff/float(sum(diff))
plt.bar(range(0, len(pdf)), pdf)
plt.title('Number of *new* triplets contributed by including features')
plt.xlabel('Diagnosis feature #')
plt.ylabel('Fraction of triplets contributed')<jupyter_output><empty_output><jupyter_text>Including DIAG9 is (for some reason) especially beneficial  What if claims were linked? This section tries to predict how many triplets could be extracted from the data, if we had a fixed number of linked claims for each patient.The PUF from DSHS does **not** contain linkable claims. This is just an estimate of what linkable claims **would** be able to give us.<jupyter_code>pdf2 = np.convolve(pdf, pdf)
plt.bar(range(0, len(pdf2)), pdf2)
pdf4 = np.convolve(pdf2, pdf2)
plt.bar(range(0, len(pdf4)), pdf4)
pdf8 = np.convolve(pdf4, pdf4)
pdf16 = np.convolve(pdf8, pdf8)
plt.bar(range(0, len(pdf16)), pdf16)
def calcNumberOfTrigramsPMF(nvars, pmf, claimsperpatient, nclaims=nclaims):
    pmfcut = pmf.copy()
    pmfcut[nvars+1:]=0
    pmfcut[nvars] = 1 - sum(pmfcut[:nvars])
    pconv = pmfcut.copy()
    for k in range(1,claimsperpatient):
        pconv = np.convolve(pconv.copy(),pmfcut)
    trigrams=[(nclaims/float(claimsperpatient))*pconv[i]*i*(i-1)*(i-2)/6 for i in range(3, len(pconv))]
    return sum(trigrams)
ntri = calcNumberOfTrigramsPMF(5, pdf, 1)
ntri
nvars = range(3,15)<jupyter_output><empty_output><jupyter_text>Set the average number of linked claims per patient<jupyter_code>numberOfClaimsPerPatient = 1
ntrip = [12*calcNumberOfTrigramsPMF(x, pdf, numberOfClaimsPerPatient) for x in nvars]
plt.bar(nvars, ntrip)
plt.title('2007-2010 PUF, assuming ' + str(numberOfClaimsPerPatient) + ' claims per patient.')
plt.xlabel('Number of diagnosis features')
plt.ylabel('Number of extracted triplets')
sum((nclaims*pdf-diff)/nclaims)
diff<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup