Like a needle in a haystack

Predictive models are only effective if their predictions are useful, or in this case, correct most of the time. The past week was dedicated to evaluating the predictive capacity of our discriminant function developed last week. To do this, we ran our test set of data through the function to obtain a predicted MDI classification for each case in the test set. At first glance, our model appeared to work well, having predicted about the expected number of patients with higher MDIs versus lower MDIs. The next step was to compare the predicted MDI scores to the actual values in the test set. It was here that we ran into a problem: the MDI scores of the patients in the test set were missing. Either the patients had never returned for their follow up, or their appointment for evaluation had not yet taken place. To get around this we did the next best thing; we took a look at their charts to see if there was any mention that the patient was developmentally delayed, and classified them accordingly. Using this method of classification, our model only made a correct prediction about half of the time. This meant it was time to take a step back and re-evaluate which variables to use when we create our discriminant function. To do this, we've decided to cast a large net and create a cross-correlation matrix of all the variables we have access to in the training set. The idea is that we will then pick out the variables that have a high correlation with MDI score and then use those in our discriminant analysis. More number crunching to come next week!

Getting down to what matters

This fifth week of Immersion Term saw the lift-off of my summer project with Dr. William Frayer, which involves some statistical analysis of data on patients from the neonatal ICU. It was a very productive week; on Monday we only had a rough idea of what needed to be done, but by Friday we had obtained a copy of SAS (very expensive statistical analysis software made by SAS Inc.), I had learned the required syntax, and we had started generating some meaningful statistics. In doing so I had expanded my own knowledge of statistical techniques. Now for some fun details...

The first goal of the project is to determine which patient factors have a strong correlation with low mental developmental index (MDI, measured at approximately 1 year of age) in low birth weight babies. The second goal is to use these factors to create a discriminant function that will allow physicians to make predictions about the MDI of their patients. A discriminant function is a linear function that sums the products of each independent variable with its correlation coefficient (a value determined though canonical correlation analysis). The discriminant function can be created from any data set containing a categorical dependent variable (the variable you want to make a prediction about) and several independent variables (the predictors). This set is called the training set. To validate the function, a second data set with the same variables (the test set) is needed. The test set is then run through the discriminant function, the predictions made by the function are compared to the actual dependent variables in the test set, and the frequency of Type I and Type II errors (false positives and false negatives respectively) can be determined. Having low error frequencies means that the discriminant function is a good predictor of the dependent variable.

This type of analysis may sound complicated, but luckily it only takes about 4 lines of code in SAS to complete. I can't imagine how long it would take by hand! Next week will most likely involve fine-tuning our analysis and maybe even some validation if we're lucky. Stay tuned...