Use of Structured and Unstructured Data to Identify Contraceptive Use in Women Veterans

by Julie A. Womack, CNM, APRN, PhD; Matthew Scotch, MPH, PhD; Sylvia N. Leung, MS; Melissa Skanderson, MSW; Harini Bathulapalli, MPH; Sally G. Haskell, MD; and Cynthia A. Brandt, MD, MPH

Abstract

Contraceptive use among women Veterans may not be adequately captured using administrative and pharmacy codes. Clinical progress notes may provide a useful alternative. The objectives of this study were to validate the use of administrative and pharmacy codes to identify contraceptive use in Veterans Health Administration data, and to determine the feasibility and validity of identifying contraceptive use in clinical progress notes. The study included women Veterans who participated in the Women Veterans Cohort Study, enrolled in the Veterans Affairs Connecticut Health Care System, completed a baseline survey, and had clinical progress notes from one year prior to survey completion. Contraceptive ICD-9-CM codes, V-codes, CPT codes, and pharmacy codes were identified. Progress notes were annotated to identify contraceptive use. Self-reported contraceptive use was identified from a baseline survey of health habits and healthcare practices and utilization. Sensitivity, specificity, and positive predictive value were calculated comparing administrative and pharmacy contraceptive codes and progress note–based contraceptive information to self-report survey data. Results showed that administrative and pharmacy codes were specific but not sensitive for identifying contraceptive use. For example, oral contraceptive pill codes were highly specific (1.00) but not sensitive (0.41). Data from clinical progress notes demonstrated greater sensitivity and comparable specificity. For example, for oral contraceptive pills, progress notes were both specific (0.85) and sensitive (0.73). Results suggest that the best approach for identifying contraceptive use, through either administrative codes or progress notes, depends on the research question.

Keywords: contraception, electronic health records, Veterans, clinical coding

Introduction

Healthcare delivery in the United States is fragmented. Although this fragmentation is widespread, one example exists within the Veterans Health Administration (VHA), where some women Veterans enrolled for care within the VHA receive reproductive healthcare, including contraception, outside of the VHA system. Contraception is a key component of reproductive healthcare as it has important implications for women’s health and the health of their children. All providers need to be aware of the contraceptive methods used by their patients. However, if patients do not receive contraception within the VHA system, administrative and pharmacy codes may not be helpful for identifying the methods used by patients. We sought to explore the sensitivity and specificity of administrative codes for contraceptive use in a cohort of women Veterans and to explore the feasibility of using text notes (clinical progress notes) to identify contraceptive use in this sample.

Background

In the United States, an individual’s healthcare is typically provided by multiple providers, often from multiple healthcare systems.¹ This fragmentation of care delivery can be detrimental to patient health as it may result in the absence of important information, such as medication use or adverse drug reactions, from one or many of the patient’s medical records. Care within the Veterans Health Administration (VHA) is an example of fragmented care: patients registered for care within the VHA often receive care from providers outside of that system either by choice or when the care required is not provided by the VHA facility.² Women’s healthcare is an example of this fragmentation and is of particular importance to the VHA.

Women are the fastest-growing segment of eligible healthcare users in the VHA.³ There are currently more than 170,000 women Veterans of Operation Enduring Freedom (OEF; October 7, 2001–present), Operation Iraqi Freedom (OIF; March 19, 2003–September 1, 2010), and Operation New Dawn (OND; September 1, 2010–present).⁴ Of these women, more than 50 percent have enrolled in the VHA for healthcare, and 43.8 percent of the women who have enrolled have presented for at least two healthcare visits.^5-7 While the mean age of all women Veterans in 2007 was 47 years, approximately 90 percent of the returning OEF, OIF, or OND women Veterans who enroll in the VHA are between the ages of 20 and 40 years.⁷ With the growing women Veteran patient population in the VHA, contraception and reproductive health are top priorities for the VHA Women’s Health Research Agenda.⁸

To respond to this increase in women Veterans registering for care in the VHA system, the VHA now provides gender-specific healthcare services for women at all Veterans Affairs Medical Centers and Community-Based Outpatient Clinics.⁹ Nonetheless, some women choose to receive reproductive healthcare from providers in the community. Therefore, contraceptive procedures performed on women (e.g., implant insertion, intrauterine device insertion) or contraceptives prescribed to women (e.g., oral contraceptive pills, the contraceptive ring, or the contraceptive patch) by community providers would likely not be recorded in the VHA medical record as administrative or pharmacy data (in the form of International Classification of Diseases, Ninth Revision, Clinical Modification [ICD-9-CM] codes, V codes [codes used to describe encounters with circumstances other than disease or injury], Current Procedural Terminology [CPT] codes, and pharmacy codes). Therefore, these codes may not adequately capture the contraceptive care provided to women. However, accurate documentation of contraceptive use is important for women’s health. Contraceptives can interact with commonly prescribed medications such as antiepileptic drugs¹⁰ and antibiotics.¹¹ In addition, documentation of effective contraception is crucial when prescribing teratogenic medications to women.^12,13

If administrative data do not provide the necessary information, providers and researchers need to look elsewhere for information on contraceptive use. Clinical notes comprise approximately 60 percent of all clinical information captured by healthcare providers.¹⁴ Researchers and clinicians have captured information from discharge summaries, emergency room reports, radiology reports, and clinical notes for a wide variety of purposes including the development of clinical decision support tools, the conduct of quality improvement projects, and the identification of clinically important syndromes.^15-18 As any thorough health history for women Veterans should include information on contraceptive use, regardless of clinical setting, the use of clinical progress notes is an attractive alternative source of information about contraceptive use by women who receive care in the VHA and merits further exploration.

The purpose of our research was first to validate administrative and pharmacy codes for contraception within the VA Connecticut Healthcare System. We compared administrative codes obtained from the electronic health record (EHR) to results obtained from a survey that included questions about contraceptive use that was completed by women OEF/OIF/OND Veterans. We calculated sensitivity, specificity, and positive predictive value of these administrative codes. We then explored the feasibility of identifying contraceptive use through progress notes, comparing results obtained through chart review with results from the same survey, identifying sensitivity, specificity, and positive predictive value of chart-identified contraceptive use.

Methods

Sample

This study is part of the Women Veterans Cohort Study (WVCS). The WVCS is a two-phase longitudinal study examining healthcare utilization, health outcomes, and costs of care among male and female OEF/OIF/OND Veterans in VHA care. The sampling frame for the overall study is the OEF/OIF/OND roster, provided to the VA by the Department of Defense Manpower Data Center’s Contingency Tracking System. The OEF/OIF/OND roster is a database of Veterans who have separated from OEF/OIF/OND military service and enrolled in VA healthcare between October 1, 2001, and April 30, 2008 (N = 406,802). Roster information includes Veterans’ sex, race, date of birth, deployment dates, armed forces branch (Army, Navy, Air Force, Marines, or Coast Guard), and component (National Guard, reserve, or active duty).

Phase I of the WVCS is an EHR-based analysis of gender differences in healthcare utilization, healthcare costs, and health outcomes of the 406,802 male and female Veterans included in the OEF/OIF/OND roster. Data on Veterans identified through the OEF/OIF/OND roster were linked to administrative and clinical data contained within the VHA National Patient Care Database (NPCD) and Decision Support System (DSS). These databases provide healthcare utilization information and cost data, pharmacy and laboratory data, and coded diagnostic and procedure data associated with inpatient and outpatient encounters.

Phase II of the WVCS adds survey data to the EHR data from WVCS Phase I, and is a prospective survey of male and female OEF/OIF/OND Veterans at two large VA facilities, one in the Northeast and one in the Midwest. Letters were sent to all male and female Veterans on the OEF/OIF/OND roster who lived within 100 miles of each facility (n = 2,000). Patients expressing interest in the study either met with or called the research coordinator. Prospective participants were read a description of the study, had questions answered about enrollment and possible adverse consequences of participation, and were screened for eligibility. Eligibility criteria included the ability to speak and read English, and participation in OEF, OIF, or OND. Those who agreed to participate were given an appointment at which they signed the informed consent form and were asked to complete the baseline survey. Participants were also asked to complete follow-up surveys one and two years after the baseline survey. The WVCS was approved by the VA Connecticut Healthcare System and by Yale University.

The OEF/OIF/OND Veteran roster included 650 women registered to receive care at the Northeast VHA facility. Of these women, 208 (32 percent) agreed to participate in Phase II (the prospective survey) of the study. In addition, because we wished to explore documentation of contraceptive use in clinical progress notes, we limited our sample to women who had progress notes in their EHR during the year prior to baseline survey completion, for a total of 59 (of 208) women (28 percent) and 1,739 progress notes. We further limited the sample to those who completed all of the contraceptive questions on the survey: 57 women (27 percent) and 1,686 clinical progress notes.

Survey

The baseline survey for the WVCS¹⁹ is a web-based, self-reported structured survey that included items on combat exposure, trauma, pain, social support, health status, and reproductive health. It also captured information on participants’ health habits, healthcare practices, and utilization of healthcare services. The contraceptive question of interest was “During the past 12 months, have you taken prescription medicine for birth control?” If the response was positive, participants were asked to list the brand. Of note, the formulation of this question did not allow for the capture of information on bilateral tubal ligations or on hysterectomies. In addition, participants were asked, “Thinking back about the last time you had sex, did you or your partner use a condom?”

Administrative Codes

We identified ICD-9-CM codes, CPT codes, and VHA pharmacy codes associated with contraceptive use for the women in our sample for the year prior to baseline survey completion (see Table 1).

Table 1
Administrative Codes for Contraceptive Use

Progress Notes

Progress notes dating back one year from survey completion were imported into Protégé 2.0 (Stanford Center for Biomedical Informatics Research), a free, open-source ontology editor and knowledge-base framework. We used the Knowtator plug-in (BioNLP, Center for Computational Pharmacology, University of Colorado Denver Health Sciences Center), a general-purpose text annotation tool that is integrated with the Protégé knowledge representation system, to annotate the progress notes for contraceptive use. We identified specific contraceptives used, when they were initiated, and the duration of use. One author (J.W.), a nurse-midwife and family nurse practitioner, did the initial annotation. The results were then confirmed by the senior author (C.B.), also a clinician. After completing the annotation, we converted the Knowtator files into XML files and imported them into an SAS (SAS Institute Inc.) data set. Because the data included multiple files per person, we transposed the data so that there would be only a single file per person.

The three files—survey, administrative codes, and progress notes—were then merged into one SAS data set and analyzed together.

Statistical Methods

The survey results were used as the reference standard for the evaluation. Both administrative codes and progress note annotations were compared to the reference standard to determine sensitivity and specificity. These measures rely on comparing the results obtained from the test cases (administrative codes and annotations) to those obtained in the survey (the reference standard). Sensitivity (recall) is the proportion of documents that are relevant to the query and that are successfully retrieved compared to all reports determined to be relevant by the gold standard. Specificity identifies the proportion of documents correctly identified as negative, or not relevant to the query. Positive predictive value is the proportion of documents retrieved that are relevant to the user’s needs compared to all documents, relevant or irrelevant, that were retrieved by the method being evaluated.²⁰

In the current analysis, “true positives” were annotations or administrative codes that identified individuals who used a specific type of contraceptive and that were in agreement with survey results. “True negatives” were annotations or administrative codes that identified individuals who did not use a specific type of contraception and that again agreed with survey results. “False positives” occurred when annotations or administrative codes identified contraceptive use that was not confirmed by survey results. Finally, “false negatives” occurred when annotations or administrative codes did not identify contraceptive use that was identified by the survey. All analyses were conducted using SAS software, version 9.2 (SAS Institute Inc.).

Results

The mean age of the participants was 31 ± 9 years. Seventy-five percent were white, 98 percent had at least 12 years of education, and 21 percent were married. More than 80 percent had enlisted rank, and 67 percent of the women had served in the Army (see Table 2).

Table 2
Sample Description for Women Enrolled in the Women Veterans Cohort Study (WVCS) Study with Progress Notes during the Year Prior to Survey Completion, from the VA Connecticut Healthcare System, West Haven Campus

Survey

According to the survey, emergency contraception, combined oral contraceptive pills, NuvaRing or the contraceptive patch, intrauterine device (IUD), depot medroxyprogesterone acetate (DMPA), and male condoms were all reported to have been used by at least one woman (see Table 3). No one reported using progestin-only pills, female condoms, diaphragm, or e. Because of how the contraceptive question was phrased, the survey did not identify women who had had a hysterectomy or bilateral tubal ligation. Twenty-two women did not report contraceptive use over the past year. The remaining 35 women accounted for 57 instances of contraceptive use. Two women reported using more than one type of prescription contraceptive in the past year, while seven women reported using both condoms and a prescription contraceptive. The most commonly used contraceptive was combined oral pills (22 instances; 51 percent). Twelve women reported using condoms the last time they had intercourse (26 percent), and six reported using either NuvaRing or the contraceptive patch (13 percent). One woman reported using emergency contraception, one reported using an IUD, and one reported using DMPA during the year prior to survey completion.

Table 3
Instances of Contraceptive Use by Survey, Annotation, and Administrative Data (N = 57)

Administrative Codes

Administrative codes identified a total of 15 instances of contraceptive use: 9 instances of oral contraceptive pill use, 3 instances of NuvaRing or patch use, 2 instances of IUD use, and 1 instance of DMPA use.

Progress Notes

The annotated progress notes captured bilateral tubal ligation (one instance) and hysterectomies (four instances), as well as the contraceptive types measured in the survey (see Table 3). Twenty-seven women had no documentation of contraceptive use. Excluding the 5 women with permanent contraception, 30 women accounted for 43 separate instances of contraceptive use. Nine women had documentation indicating that they had used more than one type of contraceptive over the past year. As in the survey, combined oral contraceptive pills were the most commonly used form of contraception (21 instances). Eight women used NuvaRing or the contraceptive patch, one woman reported having used emergency contraception, eight women had male condom use documented, three women had IUD use documented, and two women had DMPA use documented.

Comparison of Different Approaches

Comparing administrative codes with survey data, the codes missed many of the contraceptive instances identified in the survey (see Table 4). For emergency contraception and male condom use, sensitivity and positive predictive value either were 0 or could not be calculated. Specificity was excellent for both (1.00). Oral contraceptive use and NuvaRing/patch use both demonstrated excellent specificity (1.00 for both) and positive predictive value (1.00 for both), but sensitivity (0.41 and 0.50, respectively) was not as good. IUD and DMPA use demonstrated excellent sensitivity (1.00 for both) and specificity (0.98 and 1.00, respectively). Positive predictive value (1.00) was equally outstanding for DMPA use, but was less strong for IUD use (0.50).

Table 4
Survey Results (Reference Standard) versus Annotations and Administrative Data

Comparing progress notes annotated for contraceptive use with survey results, we found excellent positive predictive value, sensitivity, and specificity for emergency contraceptive use (1.00 for all; see Table 4). Positive predictive value of male condom use in annotated progress notes was moderate (0.63), and sensitivity was poor (0.42), but specificity was excellent (0.93). Oral contraceptive pills demonstrated acceptable positive predictive value (0.76), sensitivity (0.73), and specificity (0.85). Positive predictive value and specificity were excellent for NuvaRing/contraceptive patch use (1.00 for both) with acceptable sensitivity (0.63. Sensitivity (1.00) and specificity (0.96) were excellent for IUD use, but positive predictive value (0.33) was less reassuring. Results were similar for DMPA use: sensitivity (1.00) and specificity (0.98) were excellent, but positive predictive value (0.50) was not as good.

As noted earlier, the survey did not capture information on hysterectomy and tubal ligation; thus comparisons with the reference standard were not possible. We did find one instance of tubal ligation in the annotated progress notes and four instances of hysterectomy. Administrative/pharmacy codes identified none of these.

Discussion

The data suggest that administrative and pharmacy codes are specific but not sensitive for identifying contraceptive use in this sample of women Veterans. In addition, we have demonstrated the feasibility of using progress notes to identify contraceptive use among women Veterans who receive their care at the VA Connecticut Healthcare System. Contraceptive use is well documented, demonstrating good sensitivity and specificity when compared with the reference standard of patient self-reporting, and thus the documentation potentially provides an important source of information that will help minimize the negative effects of care fragmentation.

Positive predictive value was good for the administrative codes but poor for the annotated progress notes. Because positive predictive value is dependent on prevalence, and the prevalence of many of these contraceptives was low in our sample, exploration of positive predictive value of annotated progress notes in a larger sample will be important.

In comparison to the reference standard, annotated progress notes were more sensitive than administrative/pharmacy codes. However, while administrative/pharmacy codes are easily and quickly available to researchers or clinicians, the annotation process that was required to identify contraceptive use in progress notes was labor intensive, time consuming, and thus costly.

Considerations about the appropriate use of the different sources of contraceptive data are important. Researchers, clinicians, and health information management professionals must thus think carefully about how they want to use data before identifying the appropriate source of the data: administrative codes or text notes. For example, if clinicians want to develop a clinical decision support tool that will identify contraceptive use in women who are being prescribed teratogenic medications, the highly specific administrative codes will be adequate. The high false negative rate, which is common with highly specific tests, is not a problem: the women who are falsely identified as not using an effective form of contraception can simply inform their clinical provider of their contraceptive method. Few women who are not using contraception will be misidentified as using it, which is the key. In contrast, if researchers want to gather information about the number of women in the VHA who use contraception, and characteristics of these women and their healthcare use, highly specific tests would not be as helpful because they would eliminate a number of women who were using contraception. For this type of research, contraceptive information obtained from progress notes might provide more meaningful information.

This study highlights the value of data extracted from progress notes. To maximize the information found in these notes, providers need to be encouraged to document thoroughly all healthcare, including contraceptive use, received by their patients from healthcare systems external to the VHA.

This study has a number of strengths, including the variety of data types that were available. Self-reported survey results provided a reference standard by which we could evaluate how well annotated progress notes identified contraceptive use compared with administrative/pharmacy data. EHR data provided access to clinical progress notes. Structured administrative and pharmacy data were also easily accessible.

Our sample size was small but was appropriate for a feasibility study. A next step will be to develop a more time-efficient approach to identifying contraceptive use in progress notes. Annotated notes can be utilized by biomedical informatics applications such as ontologies for automated extraction of clinical data from EHRs.²¹ We can then use this approach to identify contraceptive use in a national sample of women Veterans so as to explore questions relating to contraceptive use in this population. This work can also be extended to other healthcare systems with EHRs.

Conclusions

Administrative and pharmacy codes were specific but not sensitive for identifying contraceptive use among women Veterans, while information obtained from clinical progress notes was more sensitive with comparable specificity. However, obtaining information on contraceptive use from clinical progress notes is far more time consuming and thus expensive than using administrative and pharmacy codes. This study highlights the importance of considering the application and intended use of data before identifying structured or unstructured data as the appropriate source.

Funding

This work was supported by Veterans Affairs Health Services Research & Development (HSR&D) grants HIR 09-007 and DHI 07-065-1, and is a translational use case project within the VA-funded Consortium for Healthcare Informatics Research (CHIR); by the National Institute of Nursing Research of the National Institutes of Health (NIH) grant number 1K01NR013437-01; and by the National Center for Research Resources, the National Center for Advancing Translational Sciences, and the Center for Advancing Translational Science, components of the National Institutes of Health, and NIH Roadmap for Medical Research Clinical and Translational Science Award (CTSA) grant number UL1 RR024138. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the NIH or the Department of Veterans Affairs.

Author’s note: We followed the Veterans Administration convention of capitalizing “Veteran” as a mark of respect.

Julie A. Womack, CNM, APRN, PhD, is an assistant professor at the Yale School of Nursing in New Haven, CT.

Matthew Scotch, MPH, PhD, is an assistant professor in Arizona State University’s Department of Biomedical Informatics in Scottsdale, AZ.

Sylvia N. Leung, MS, is a graduate student at the Harvard Kennedy School of Government in Cambridge, MA.

Melissa Skanderson, MSW, is a programmer at the VA Connecticut Healthcare System, West Haven Campus, in West Haven, CT.

Harini Bathulapalli, MPH, is a biostatistician at the VA Connecticut Healthcare System, West Haven Campus, in West Haven, CT.

Sally G. Haskell, MD, is an associate professor at the VA Connecticut Healthcare System, West Haven Campus, in West Haven, CT.

Cynthia A. Brandt, MD, MPH, is a professor at the VA Connecticut Healthcare System, West Haven Campus, in West Haven, CT.

Notes

¹ Walker, J., E. Pan, D. Johnston, J. Adler-Milstein, D. W. Bates, and B. Middleton. “The Value of Health Care Information Exchange and Interoperability.” Health Affairs web suppl. (2005): W5-10–W5-18. doi: 10.1377/hlthaff.w5.10.

² US Department of Veterans Affairs. United States Department of Veterans Affairs: Non-VA Care. November 15, 2012. Available at http://www.nonvacare.va.gov/providers.asp (accessed November 27, 2012).

³ Yano, E. M., L. A. Bastian, S. M. Frayne, A. L. Howell, L. R. Lipson, G. McGlynn, P. P. Schnurr, M. R. Seaver, A. M. Spungen, and S. D. Fihn. “Toward a VA Women’s Health Research Agenda: Setting Evidence-based Priorities to Improve the Health and Health Care of Women Veterans.” Journal of General Internal Medicine 21, suppl. 3 (2006): S93–S101.

⁴ Epidemiology Program, Post-Deployment Health Group, Office of Public Health,  Veterans Health Administration, Department of Veterans Affairs. (2012). VA Health Care Utilization (Inpatient and Outpatient) Among 171,434 Female and 1,306,936 Male Operation Enduring Freedom (OEF), Operation Iraqi Freedom (OIF), and Operation New Dawn (OND)* Veterans through 2nd Qtr FY 2012, version 1a.

⁵ Hayes, P., and M. Krauthamer. “Changing the Face of Health Care for Women Veterans.” HealthJobsNationwide.com. October 21, 2009. Available at http://www.healthjobsnationwide.com/news.php?articleID=63.

⁶ Hayes, P. “Women Veterans Health Care.” Presented at the Women Veterans Resource Fair, Denver, 2012.

⁷ US Department of Veterans Affairs, Office of Policy and Planning. Women Veterans: Past, Present and Future. September 2007. Available at http://www1.va.gov/womenvet/docs/WomenVet_History.pdf.

⁸ Yano, E. M., L. A. Bastian, S. M. Frayne, A. L. Howell, L. R. Lipson, G. McGlynn, P. P. Schnurr, M. R. Seaver, A. M. Spungen, and S. D. Fihn. “Toward a VA Women’s Health Research Agenda: Setting Evidence-based Priorities to Improve the Health and Health Care of Women Veterans.”

⁹ Department of Veterans Affairs, Veterans Health Administration. (2010) VHA Handbook 1330.01: Healthcare Services for Women Veterans. VHA Publications.

¹⁰ Sabers, A. “Pharmacokinetic Interactions between Contraceptives and Antiepileptic Drugs.” Seizure 17, no. 2 (2008): 141–44. doi: 10.1016/j.seizure.2007.11.012.

¹¹ Dickinson, B. D., R. D. Altman, N. H. Nielsen, M. L. Sterling, and American Medical Association Council on Scientific Affairs. “Drug Interactions between Oral Contraceptives and Antibiotics.” Obstetrics and Gynecology 98, no. 5, pt. 1 (2001): 853–60.

¹² Schwarz, E. B., J. Maselli, M. Norton, and R. Gonzales. “Prescription of Teratogenic Medications in United States Ambulatory Practices.” American Journal of Medicine 118, no. 11 (2005): 1240–49. doi: 10.1016/j.amjmed.2005.02.029.

¹³ Schwarz, E. B., S. M. Parisi, S. L. Williams, G. J. Shevchik, and R. Hess. “Promoting Safe Prescribing in Primary Care with a Contraceptive Vital Sign: A Cluster-Randomized Controlled Trial.” Annals of Family Medicine 10, no. 6 (2012): 516–22. doi: 10.1370/afm.1404.

¹⁴ Health Story Project. “HIMSS10:Chair of HL7, Bob Dolin, MD, Recommends Incremental Interoperability Strategy to Reach Meaningful Use.” March 10, 2010. Available at http://www.healthstory.com/news/releases/bobdolin.htm.

¹⁵ Demner-Fushman, D., W. W. Chapman, and C. J. McDonald. “What Can Natural Language Processing Do for Clinical Decision Support?” Journal of Biomedical Informatics 42, no. 5 (2009): 760–72. doi: 10.1016/j.jbi.2009.08.007.

¹⁶ Dara, J., J. N. Dowling, D. Travers, G. F. Cooper, and W. W. Chapman. “Evaluation of Preprocessing Techniques for Chief Complaint Classification.” Journal of Biomedical Informatics 41, no. 4 (2008): 613–23. doi: 10.1016/j.jbi.2007.11.004.

¹⁷ Wilson, R. A., W. W. Chapman, S. J. Defries, M. J. Becich, and B. E. Chapman. “Automated Ancillary Cancer History Classification for Mesothelioma Patients from Free-Text Clinical Reports.” Journal of Pathology Informatics 1 (2010): 24. doi: 10.4103/2153-3539.71065.

¹⁸ Womack, J. A., M. Scotch, C. Gibert, W. Chapman, M. Yin, A. C. Justice, and C. Brandt. “A Comparison of Two Approaches to Text Processing: Facilitating Chart Reviews of Radiology Reports in Electronic Medical Records.” Perspectives in Health Information Management (Fall 2010): 1-11. At (at http://perspectives.ahima.org/a-comparison-of-two-approaches-to-text-processing-facilitating-chart-reviews-of-radiology-reports-in-electronic-medical-records/#.UZ_DabV9DjA)

¹⁹ Haskell, S. G., K. Mattocks, J. L. Goulet, E. E. Krebs, M. Skanderson, D. Leslie, A. C. Justice, E. M. Yano, and C. Brandt. “The Burden of Illness in the First Year Home: Do Male and Female VA Users Differ in Health Conditions and Healthcare Utilization.” Women’s Health Issues 21, no. 1 (2011): 92–97. doi: 10.1016/j.whi.2010.08.001.

²⁰ Taylor, J. R. An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements. Sausalito, CA: University Science Books, 1999.

²¹ Scotch, M., C. A. Brandt, S. Leung, and J. A. Womack. “Natural Language Processing for Understanding Contraceptive Use at the VA.” Annals of Information Science. 2012. In press.