Evaluation of an Interactive Surveillance System for Monitoring Acute Bacterial Infections in Nigeria

by Ashish Joshi, MD, MPH, PhD; Chioma Amadi, MPH; Kate Trout, MPH; and Stephen Obaro, MBBS, PhD

Abstract

Objective: The objective of the study is to evaluate the usability of a stand-alone, Internet-enabled interactive surveillance system designed to monitor the burden of invasive bacterial infections among children in Nigeria.

Method: A convenience sample of 10 participants were enrolled in a training session on using the system at a hospital in Nigeria. The participants performed a series of tasks assessing their ability to use the system. System usability was assessed using a System Usability Scale (SUS) questionnaire.

Results: The majority of participants found the system easy to use (90 percent; n = 9) and reported confidence in using the system. The average SUS score was 77.8. A total of 30 percent (n = 3) of the study participants had exceptional usability scores, 20 percent (n = 2) showed acceptable scores, and 10 percent (n = 1) had a good score.

Conclusion: Further evaluation of the system will help gauge additional challenges during its long-term utilization. If successful, the system could also be deployed in other resource poor-environments.

Keywords: data capture, data collection, developing countries, research, health records storage

Introduction

The major challenge hindering public health research in Nigeria is the difficulty of obtaining accurate data.¹ Lack of accurate data limits the proper assessment of the magnitude and distribution of disease conditions across various geographic regions.² Timely and accurate data are essential to develop evidence-based informed decisions.³ The lack of reliable data has been attributed to factors including limited financial resources, poor human resource training, and the use of paper-based methods of data collection.⁴ The ability to obtain reliable data also largely depends on the mechanism adopted to collect data.⁵

Electronic methods of data collection are frequently utilized in developed countries and have been shown to provide more benefits than paper-based data collection and alleviate most issues associated with paper-based methods. Traditional paper-based methods of data collection have been previously associated with several issues, including error frequency, duplication, and storage costs.⁶ Some of the advantages of electronic data collection over paper methods include timeliness, avoidance of record duplication, cost-effectiveness, facilitation of data transmission and synchronization, higher data quality, data completeness, and higher acceptance.⁷ Prior studies which examined the use of electronic systems in data collection in comparison to paper-based methods have reported higher patient acceptance and preference among patients in using electronic systems ^8,9. In developing countries, paper-based data collection methods are currently being replaced by electronic means of health research in order to facilitate more effective data collection and storage.¹⁰Despite these efforts, several impediments in the structure and functioning of the systems in these countries hamper the sustained progress of such activities.¹¹ These impediments include limited infrastructure and finances, and poor adoption by physicians, which has been attributed to a lack of familiarity with the use of such systems.¹²

The lack of surveillance systems in Nigeria is a major factor that currently hinders the assessment of the disease burden or other health conditions.¹³ Health surveillance systems essentially serve as tools for monitoring the incidence of health conditions across geographic boundaries.¹⁴ By doing so, they prompt the discovery of various patterns and relationships that were previously unknown.¹⁵ The data obtained from such systems could serve several purposes, including estimation of the incidence of a disease, visualization of its distribution and spread, hypothesis generation regarding causal factors, implementation of adequate control strategies to foster early detection, and short- and long-term outcome evaluation of the impact of preventive strategies.¹⁶

An interactive surveillance system was designed and developed for deployment across study sites in several regions of Nigeria to monitor the incidence of acute bacterial infections among children less than five years of age. Children under five years of age in Nigeria have one of the highest burdens of acute respiratory infections, with an estimated carriage of 67.4 percent, compared to 26 percent in adults.¹⁷About one-fifth of deaths due to acute respiratory infections are found in this segment of the population.¹⁸ Children residing in southwestern states of Nigeria have an estimated occurrence of at least three episodes of such infections annually, with pneumonia being the most prevalent infection.¹⁹Data regarding the distribution and determinants of acute respiratory infections among Nigerian children are limited, though these diseases have been reported to be more prevalent during the rainy season.²⁰

Invasive bacterial infections that are associated with clinical syndromes such as pneumonia, sepsis, and meningitis are the leading causes of mortality and morbidity in several developing countries, with exceptional vulnerability in children aged less than five years.^{21, 22} The risk factors associated with these diseases differ depending on populations and geographic regions and have not been well characterized.²³ The lack of sufficient data on the distribution of these diseases may have led to their underestimation.²⁴ A reliable estimation of the incidence, prevalence, and risk factors associated with this group of diseases is necessary in order to guide effective public health research and implementation of appropriate preventive and therapeutic interventions. A computer-based surveillance system was designed to gather information regarding the risk factors and other variables associated with the incidence of acute respiratory infections across different regions of the country.

Prior to deployment of the surveillance system, usability testing was conducted to evaluate the acceptance of the system among potential users so that any necessary adjustments could be incorporated in a timely manner. Usability is an integral determinant of user satisfaction and utilization of health technologies.²⁵ Interactivity, which is often hindered by poor usability testing, has been identified as a major issue in the deployment of interventions designed for various patient and clinician groups.²⁶ This issue is attributed to the fact that the design framework of most electronic systems is essentially “technology driven” rather than “user driven.”²⁷ The incompatibilities that result due to poor consideration of the user’s perspectives largely account for the reduced efficiency and utilization of such systems.²⁸ The design of user-centered systems ultimately depends on the level of integration of user characteristics alongside system requirements.²⁹ The lack of prioritization of user characteristics will lead to challenges in user satisfaction.³⁰ The objective of this pilot study is to evaluate the usability of a stand-alone, Internet-enabled interactive surveillance system designed to gather data from various sources for use in monitoring the burden of acute bacterial infections among children in Nigeria.

Conceptual Framework

The study was designed using the combined principles of a human-centered approach and behavioral and information processing. The human-centered approach gives specific considerations to users’ knowledge, expertise, and utilization of the interaction techniques to represent tasks performed by the users.^31-35 Processing of different types of information will be affected by the type of visual display that is used to present the information. When the information presented does not match the ultimate needs of the user performing the task, the result is decreased accuracy and increased time required to perform the task.³⁶ These benefits translate into system- and task-related performance factors. The existing consumer health information platform was modified to meet the needs of the users to gather data related to invasive bacterial infections among children living in diverse settings in Nigeria. The information processing theory facilitates presenting information in small chunks with each screen presenting limited information. The human-centered approach takes into consideration the users, their backgrounds, and the tasks the users need to perform.

Design and Development of an Interactive Surveillance System

A stand-alone, Internet-enabled interactive surveillance system was designed and developed to monitor the burden of invasive bacterial infections among children living in diverse settings of Nigeria. The system will be deployed in three sites, including the University of Abuja teaching hospital in central Nigeria, the Aminu Kano University teaching hospital in northern Nigeria, and University College Hospital in Ibadan (Figure 1).

Figure 1: Map of Nigeria Showing the Location of the Study Sites

Software Description

The technology involved in the development of the overall application included PHP version 5.1.2 for user and data management, log-in, and user roles. A MySQL database was utilized to store the data gathered, and jQuery was utilized for generating tabs, toggles, sliders, and all animations in the system. Figure 2 and Figure 3 describe the flow of data for both the Internet-enabled and the stand-alone applications of the system.

Figure 2: Data Flow for the Internet-enabled Interactive Surveillance System


Figure 3: Data Flow for the Stand-alone Desktop-enabled Interactive Surveillance System

Users are required to register themselves (Figure 4) and be approved by the administrator before they can gain access to the system components. Once users are registered, they can use either the Internet-enabled or the stand-alone application to collect data electronically across multiple sites in Nigeria. The users can register new patients into the system or track, and follow patients already registered into the system by clicking on the ‘search button.’ (Figure 5, Figure 6, and Figure 7). Data on various variables are gathered based on the study protocol. A data synchronization capability allows data gathered in the stand-alone system to be remotely uploaded to the centralized database so that data from various sites across Nigeria can be updated and so that the users can download the updated data from the centralized database onto their computers (Figure 3).

Figure 4: User Registration


Figure 5: Patient Registration


Figure 6: Data Entry Forms (Highlighted)


Figure 7: Viewing Existing Patient Records

Components of the Interactive Surveillance System

The components of the system include the following:

• User registration: Initially, users are required to register through the Internet portal and await administrator approval before they are able to use the system (Figure 4). The user registration gathers the username, password, full name, and e-mail address of the user. After registration, the user is able to log into the stand-alone system to collect data in environments with no Internet access.
• Patient registration: The user proceeds to register the individual patients. The system utilizes five major patient details to generate a unique identifier for each patient record entered: the patient’s name, date of birth, e-mail address, home address, and phone number (Figure 5).
• Data entry: The user enters the patient data corresponding to each of the 36 surveys for acute bacterial infections contained in the system (Figure 8). The user is required to complete each survey with the information obtained while interviewing the patient. The user can also choose to complete the surveys at a later time, and the patient details can be located by searching for existing patient records (Figure 7).

Figure 8: Data Entry

• Data validation: Clinical rules are integrated into the system, and if data outside the set threshold are entered into the system, the system generates a flag to reflect that the data value entered is a possible outlier.

• Data completion: The user interface displays color codes reflecting which forms have been completed and which forms need to be completed (Figure 6). It also displays a task progression pane reflecting the extent of completion of the form to help both the user and the patient approximate the time that will be needed to complete the forms (Figure 8). Each screen contains limited information to prevent information overload and to ensure that the user interface is easy to use and can help the user complete tasks efficiently and in a timely manner.

• Data synchronization: Data synchronization is a bidirectional process that allows data gathered through the stand-alone system to be transmitted to the central server and allows the central server to relay updates back to the stand-alone computers. This process facilitates the systematic synchronization of data across multiple locations that might not always be networked (Figure 9).
• Tracking patient progress: This component allows users to monitor the progress of the data gathered on every individual so that data collection is not missed at any of the important time intervals.

Figure 9: Data Synchronization Process

Methods

This pilot exploratory study was conducted during July 2013 at University College Hospital in Ibadan, a city in southwestern Nigeria. This hospital was one of the sites that would be utilizing the proposed interactive surveillance system to monitor the incidence of acute bacterial infections among children aged less than five years. Ten study participants from diverse backgrounds were enrolled into this pilot study to assess the usability of the system. These participants were the end users who would be responsible for data collection using the surveillance system. A training module was developed for the study participants to guide them on how to navigate through the system and use its various functions. A training session was conducted by a team of researchers consisting of an epidemiologist, a public health informatics specialist, and an infectious disease pediatrician from the University of Nebraska Medical Center. The participants were provided with information on how to carry out several tasks including (1) registering new users, (2) logging into the system, (3) registering patients, (4) viewing records, (5) navigating through the system, (6) performing data entry, and (7) synchronizing the entered records. After the one-hour training session, the users were asked to complete these seven tasks. Details of the tasks are provided in Table 1.

The System Usability Scale (SUS) was administered to the study participants after the training session to assess their level of comfort or difficulty in performing the stated tasks. The SUS was developed by Brooke in 1986 as a tool for usability testing and has been used extensively in evaluating user acceptance of systems including websites, telephones, and electronic applications.³⁷ Prior studies have shown that the SUS yields the most reliable results across sample sizes, and provides a good and valid method of assessing interface usability.³⁸ The optimum number of participants for in-depth interviews typically ranges from 6 to 10 people with similar backgrounds who participate in the interview for one to two hours; however, interview times may vary.³⁹ Prior literature has suggested that samples of 6 to 10 participants are large enough to gain a variety of perspectives, yet small enough not to become disorderly or fragmented.⁴⁰ Vast amounts of usability problems and issues can be identified with only a small number of test subjects, as few as 8 to 10 participants.⁴¹ Usability scores were recorded for the 10 study participants.

Variables Description

Socio demographics: Information regarding the age (years), gender, and level of education of the participants was recorded.

Familiarity with the use of computer systems: Information regarding the ease of use of computers, use of the Internet, and prior use of electronic data capturing systems was recorded.

System Usability Scale (SUS): The SUS assesses the participants’ ease of use and satisfaction with the interactive surveillance system. The SUS is a 10-item questionnaire with five response options that range from strong agreement to strong disagreement. Possible scores are 0, 1, 2, 3, or 4 for each question. The questions elucidate the complexity, functioning, ease of use, self-efficacy, and difficulty level associated with operating the system from the user’s perspective. To minimize bias based on agreement or disagreement, odd-numbered items of the SUS questionnaire are given more points for strong agreement, and even-numbered items are given more points for strong disagreement. The total score is calculated by adding up the converted responses for each user and multiplying that total by 2.5. This calculation makes the range of possible values 0 to 100 instead of 0 to 40. Prior research has shown that average SUS scores in the 90s are exceptional, scores in the 80s are good, scores in the 70s are acceptable, and scores below 70 indicate some usability concerns.⁴²

Statistical Analysis

Descriptive analysis was performed using univariate statistics with means and standard deviations reported for the continuous variable, age. The frequency distribution for the categorical variables, including gender, level of education, familiarity with the use of computer systems, and percentage distributions of the various SUS score categories, were also computed. All the analysis was performed using SAS version 9.1.

Results

Sociodemographic Characteristics

The average age of the study participants was 29.6 years (SD = 2.5) and ranged from 22 to 44 years. More than half of the study participants were male (70 percent; n = 7); 80 percent (n = 8) of the participants were college graduates and 20 percent (n = 2) had a postgraduate study

Familiarity with the Use of Computer Systems

Nine participants (90 percent) reported that they were very familiar with the use of computer systems.

System Usability Scale (SUS) Results

The average SUS score for all the participants was 77.8 with a standard deviation of 15.7. The SUS scores ranged from 60 to 100. Two participants had a score of 100, and these scores were followed by scores of 95 (n = 1), 85 (n = 1), 70 (n = 2), 67.5 (n = 1), 65 (n = 2), and 60 (n = 1). SUS scores were classified into four major categories: exceptional (scores in the 90s or above), good (scores in the 80s), acceptable (scores in the 70s), and minor usability issues (scores below 70) (Figure 10). On the basis of this classification, 30 percent (n = 3) of the study participants had exceptional usability scores and 20 percent (n = 2) showed acceptable scores, while 10 percent (n = 1) had a good score (Figure 10).⁴³

Figure 10: Percent Distribution of Individuals in the Four System Usability Scale (SUS) Score Categories

Additional analysis was performed to compare the differences in sociodemographic characteristics and familiarity with the use of computers among individuals who found the interactive surveillance system acceptable and those who did not. The results showed that the participants who were above 30 years of age, had a postgraduate study, and were somewhat familiar with prior use of computers had SUS scores that were in the acceptable range (greater than or equal to 70) (Table 3). Individuals younger than 30 years of age were more likely to report minor usability issues (SUS score less than 70) (Table 3).

Discussion

Results of this pilot study reflect high acceptability of the use of electronic data capture systems in Nigeria. Nine participants (90 percent) reported prior familiarity with the use of computers. The majority of the participants found the system easy to use (90 percent, n = 9), and all the participants reported confidence in using the system (100 percent, n = 10). Individuals had limited difficulties in using the system. Only one individual found the system cumbersome to use.

This surveillance system will address several limitations of the traditional paper-based methods of data collection, including record duplication, storage costs, and the difficulty of record retrieval.⁴⁴ The synchronization function of the interactive surveillance system limits data duplication. The surveillance system will also facilitate quick and efficient patient record retrieval as compared to time-consuming paper-based methods. Studies that analyzed the cost-effectiveness of electronic data collection methods over paper-based methods showed a 55 percent cost savings for electronic systems.⁴⁵ The surveillance system is interactive and easy to use, which may facilitate its adoption among users with limited computer experience. The surveillance system functions as a stand-alone system to enable real-time electronic data collection and stores the data in an encrypted format.

This study has several limitations. First, the study was cross-sectional in design and would need more long-term evaluation to identify the difficulties that the users might encounter during the use of the system. The system would need further evaluation with users having limited or no prior use of computers. To the best of our knowledge, this system is the first electronic disease surveillance system deployed for use in Nigeria on a research platform to monitor the burden of disease. The results of this usability testing showed great prospects for future deployment of the surveillance system across other locations. The high acceptability of the system in the study implies that usability errors, which often hinder adoption of such technology, are negligible in this case.

Electronic data collection systems require constant electricity supply, which is a limiting factor in Nigeria due to poor power generation.⁴⁶ However, some other factors that need to be taken into consideration during the deployment of electronic surveillance systems in resource-poor environments include cultural perceptions regarding the use of the system, given the ethnic and cultural diversity of the country. The study participants also expressed concerns regarding the timeliness in completing the patient records given the large volume of patients being seen at the same time.

Conclusion

The current usability study describes the design and development of a stand-alone and Internet-enabled interactive surveillance system that aimed to monitor the burden of acute bacterial infections among children aged less than five years in Nigeria. The study emphasizes the significance of usability evaluation of the health information systems among the actual users so that any challenges identified can be appropriately addressed before the final deployment of the system. Identifying such challenges is key to successful adoption of health technologies, especially in developing countries where users might have limited computer familiarity. Use of electronic data collection can help to improve quality of data collection, which will facilitate informed, evidence-based, and data-driven decision making.

The interactive surveillance system designed for use in data collection will potentially facilitate data completeness, improve data quality, foster timeliness in data entry, and support data validation and rapid synchronization. The successful implementation and utilization of the interactive surveillance system will aid in providing relevant statistics regarding the burden and determinants of acute bacterial infections across various geographical locations in Nigeria. It will also foster the identification of regions where disease is endemic and further guide the development of appropriate intervention strategies and their long-term outcome evaluation. The high usability acceptance of the surveillance system observed in this study will also facilitate its rapid adoption for use in the collection of data on other related health conditions across other Nigerian states and resource-poor countries. A long-term evaluation of the system is ongoing to gauge any challenges that users might encounter during its long-term utilization.

Ashish Joshi, MD, MPH, PhD, is an assistant professor at the College of Public Health, University of Nebraska Medical Center, in Omaha, NE.

Chioma Amadi, MPH, is a graduate research assistant at the College of Public Health, University of Nebraska Medical Center, in Omaha, NE.

Kate Trout, MPH, is a research coordinator at the College of Public Health, University of Nebraska Medical Center, in Omaha, NE.

Stephen Obaro, MBBS, PhD, is a professor in the Division of Infectious Diseases at the University of Nebraska Medical Center in Omaha, NE.

Acknowledgement

This pilot evaluation was conducted through support from Grant number OPP1034619 for the Community Acquired Pneumonia and Invasive Bacterial Disease (CAPIBD) in Nigerian children, awarded to Stephen Obaro.

Notes

1.   Oladapo, O. T., O. O. Adetoro, O. Fakeye, B. A. Ekele, A. O. Fawole, A. Abasiattai, and O. A. Dada. “National Data System on Near Miss and Maternal Death: Shifting from Maternal Risk to Public Health Impact in Nigeria.” Reproductive Health 6, no. 8 (2009): 1–11.

2.   Tindana, P. O., J. A. Singh, C. S. Tracy, R. E. Upshur, A. S. Daar, P. A. Singer, J. Frohlich, and J. V. Lavery. “Grand Challenges in Global Health: Community Engagement in Research in Developing Countries.” PLoS Medicine 4, no. 9 (2007): e273.

3.   Oladapo, O. T., O. O. Adetoro, O. Fakeye, B. A. Ekele, A. O. Fawole, A. Abasiattai, and O. A. Dada. “National Data System on Near Miss and Maternal Death: Shifting from Maternal Risk to Public Health Impact in Nigeria.”

4.   Walsham, G., and S. Sahay. “Research on Information Systems in Developing Countries: Current Landscape and Future Prospects.” Information Technology for Development 12, no. 1 (2006): 7–24.

5.   Tindana, P. O., J. A. Singh, C. S. Tracy, R. E. Upshur, A. S. Daar, P. A. Singer, J. Frohlich, and J. V. Lavery. “Grand Challenges in Global Health: Community Engagement in Research in Developing Countries.”

6.   Pavlović, I., T. Kern, and D. Miklavčič. “Comparison of Paper-based and Electronic Data Collection Process in Clinical Trials: Costs Simulation Study.” Contemporary Clinical Trials 30, no. 4 (2009): 300–316.

7.   Ibid

8.   Salaffi, F., Gasparini, S., Ciapetti, A., Gutierrez, M., & Grassi, W. (2013). Usability of an   Innovative and Interactive Electronic System for Collection of Patient-reported Data in Axial Spondyloarthritis: Comparison with the Traditional Paper-administered Format. Rheumatology, 52(11), 2062-2070.

9.   Wood, C., von Baeyer, C. L., Falinower, S., Moyse, D., Annequin, D., & Legout, V.    (2011). Electronic and Paper Versions of a Face’s Pain Intensity Scale: Concordance and Preference in Hospitalized Children.BMC Pediatrics, 11.

10. Benson, M. B. “Adoption of Hospital Information Systems in Nigeria.” Journal of Global Health Care Systems 1, no. 3 (2011).

11. Kushniruk, A. W., V. L. Patel, and J. J. Cimino. “Usability Testing in Medical Informatics: Cognitive Approaches to Evaluation of Information Systems and User Interfaces.” Proceedings of the AMIA Annual Fall Symposium (1997): 218.

12. Bangor, A., P. Kortum, and J. A. Miller. “The System Usability Scale (SUS): An Empirical Evaluation.” International Journal of Human-Computer Interaction 24, no. 6 (2008): 574–94.

13. Adetifa, I. M., M. Antonio, C. A. Okoromah, C. Ebruke, V. Inem, D. Nsekpong, A. Bojang, and R. A. Adegbola. “Pre-vaccination Nasopharyngeal Pneumococcal Carriage in a Nigerian Population: Epidemiology and Population Biology.” PloS One 7, no. 1 (2012): e30548.

14  Teutsch, S. M., and R. E. Churchill (Editors). Principles and Practice of Public Health Surveillance. Oxford: Oxford University Press, 2000.

15. Ibid.

16. Ibid.

17. Adetifa, I. M., M. Antonio, C. A. Okoromah, C. Ebruke, V. Inem, D. Nsekpong, A. Bojang, and R. A. Adegbola. “Pre-vaccination Nasopharyngeal Pneumococcal Carriage in a Nigerian Population: Epidemiology and Population Biology.”

18. Akanbi, M. O., C. O. Ukoli, G. E. Erhabor, F. O. Akanbi, and S. B. Gordon. “The Burden of Respiratory Disease in Nigeria.” African Journal of Respiratory Medicine 4 (2009): 10–17.

19. Johnson, A. B., K. Osinusi, W. I. Aderele, and F. A. B. Adeyemi-Doro. “Bacterial Aetiology of Acute Lower Respiratory Infections in Pre-school Nigerian Children and Comparative Predictive Features of Bacteraemic and Non-bacteraemic Illnesses.” Journal of Tropical Pediatrics 39, no. 2 (1993): 97–106.

20. Onipede, A. O., A. A. Onayade, J. B. Elusiyan, P. O. Obiajunwa, E. O. Ogundare, O. O. Olaniran, L. A. Adeyemi, and O. O. Oyelami. “Invasive Bacteria Isolates from Children with Severe Infections in a Nigerian Hospital.” Journal of Infection in Developing Countries 3, no. 6 (2009): 429–36.

21. Hotez, P. J., and A. Kamath. “Neglected Tropical Diseases in Sub-Saharan Africa: Review of Their Prevalence, Distribution, and Disease Burden.” PLoS Neglected Tropical Diseases 3, no. 8 (2009): e412.

22. Peltola, H. “Burden of Meningitis and Other Severe Bacterial Infections of Children in Africa: Implications for Prevention.” Clinical Infectious Diseases 32, no. 1 (2001): 64–75.

23. Hotez, P. J., and A. Kamath. “Neglected Tropical Diseases in Sub-Saharan Africa: Review of Their Prevalence, Distribution, and Disease Burden.”

24. Ibid.

25. Zayas-Cabán, T., and B. E. Dixon. “Considerations for the Design of Safe and Effective Consumer Health IT Applications in the Home.” Quality and Safety in Health Care 19, suppl. 3 (2010): i61–i67.

26. Ibid.

27. Ibid.

28. Ibid.

29. Ibid.

30. Ibid.

31. Koua, E. L., and M. J. Kraak. “A Usability Framework for the Design and Evaluation of an Exploratory Geovisualization Environment.” Proceedings of the Eighth International Conference on Information Visualization. London: IEEE, 2004.

32. Hollan, J., E. Hutchins, and D. Kirsh. “Distributed Cognition: Toward a New Foundation for Human-Computer Interaction Research.” ACM Transactions on Computer-Human Interaction (TOCHI) 7, no. 2 (2000): 174–96.

33. Zhang, J., and K. Butler. “UFuRT: A Work-centered Framework and Process for Design and Evaluation of Information Systems.” In Proceedings of the 12th International Conference, HCI International 2007. Springer, 2007.

34. Dennis, A. R., and T. A. Carte. “Using Geographical Information Systems for Decision Making: Extending Cognitive Fit Theory to Map-based Presentations.” Information Systems Research 9, no. 2 (1998): 194–203.

35. Andrienko, G., N. Andrienko, P. Jankowski, D. Keim, M. J. Kraak, A. MacEachren, and S. Wrobel. “Geovisual Analytics for Spatial Decision Support: Setting the Research Agenda.” International Journal of Geographical Information Science 21, no. 8 (2007): 839–57.

36. Dennis, A. R., and T. A. Carte. “Using Geographical Information Systems for Decision Making: Extending Cognitive Fit Theory to Map-based Presentations.”

37. Bangor, A., P. Kortum, and J. A. Miller. “The System Usability Scale (SUS): An Empirical Evaluation.”

38. Ibid.

39. Benson, M. B. “Adoption of Hospital Information Systems in Nigeria.”

40. Ibid.

41. Idowu, P., D. Cornford, and L. Bastin. “Health Informatics Deployment in Nigeria.” Journal of Health Informatics in Developing Countries 2, no. 1 (2008).

42. Bangor, A., P. Kortum, and J. A. Miller. “The System Usability Scale (SUS): An Empirical Evaluation.”

43. Ibid.

44. Yu, P., M. de Courten, E. Pan, G. Galea, and J. Pryor. “The Development and Evaluation of a PDA-based Method for Public Health Surveillance Data Collection in Developing Countries.” International Journal of Medical Informatics 78, no. 8 (2009): 532–42.

45. Benson, M. B. “Adoption of Hospital Information Systems in Nigeria.”

46. Kushniruk, A. W., V. L. Patel, and J. J. Cimino. “Usability Testing in Medical Informatics: Cognitive Approaches to Evaluation of Information Systems and User Interfaces.”