Insights from a Human Factors Usability Study 

By Marla M. Husch, RPh & Gary Fennessy, MBA

This article provides a current, practical methodology for choosing a smart infusion system for your organization.

Click here to download Additional Tables available only online from this article. 

Click here to Download PDF  

Technology has created fundamental changes in the way care is provided to patients. In the past, technology was commonly employed to resolve tactical problems that occurred within specific domains. Today, technology decisions must be more strategic and multidisciplinary in order to ensure a successful implementation and on-going commitment from end-users. This article provides a current, practical methodology for choosing a smart infusion system for your organization.

First Things First
Appoint a project manager and form a multidisciplinary task force. Implementing a smart infusion system is not a small task and, therefore, requires a leader with strong and proven project-management skills. Many times, the project manager will be a pharmacist, since pharmacy typically owns the drug library; however, this role can be taken on by any number of disciplines. At Northwestern Memorial Hospital (NMH), we utilized the talent of a nurse materials utilization manager, who was able to oversee the project from the contracting phase all the way through implementation.

Effective implementation of a smart infusion system requires significant participation from multiple disciplines. Thus, the first mission of the project manager is to establish a multidisciplinary task force. The importance of the multidisciplinary nature of this group cannot be overstated. Dedicated representation at all levels of the system (management and frontline staff) and from nursing, information systems, pharmacy, physicians, biomedical engineering, and materials management is essential. Because previous generations of IV pumps did not employ drug libraries or a connection to a wireless server, this multidisciplinary team concept is new to IV pump implementation and maintenance, but is nonetheless imperative to success.

System Selection
Once the key individuals have been identified, the process of system selection begins. This can be a frustrating process, since information is scarce regarding the breadth of functionality that the newest smart infusion systems can offer, making it difficult to determine what this technology has really been designed to do. In a nutshell, current smart infusion systems are designed to alert the programming clinician — via dose error reduction software (DERS) — when a dose, rate, or duration-of-infusion limit has been exceeded. The DERS is customized by each organization after the purchase has been made. Typically, both upper and lower limits are set, and are established as either hard limits — that give the clinician no other option except to reprogram or opt out of the DERS — or soft limits — that give the clinician an option to override the alert and continue programming or reprogram within limits. Additionally, some vendors now offer integrated PCA monitoring via end tidal CO2 monitoring (EtCO2) and pulse oximetry (SpO2).

Critical future developments will be the full integration of smart infusion systems with computerized physician order entry (CPOE), pharmacy information systems, and bar coded medication administration systems. With this integration, the power of information technology will be able to address many of the IV medication errors that are generated during medication ordering, dispensing, documentation, and monitoring. All major vendors are presently working in this direction.

Perhaps most importantly, smart infusion systems are currently capable of capturing and storing continuous quality improvement (CQI) data, which indicates when and where clinicians are infusing IV medications at a dose or rate that exceeds established limits. While the amount of specific information that the CQI data can offer is somewhat limited, it is one step in the right direction toward understanding frontline practice. CQI data is also a key to understanding the extent to which your drug library database is effective and usable.

Once the task force is clear on what current smart infusion technology is designed to accomplish, it is time to get up to speed on the smart infusion systems available for purchase and the differences between them. A good, non-biased source of information and comparisons is the ECRI Institute (www.ecri.org), particularly the October 2007 issue of ECRI’s journal Health Devices, which gives a comprehensive overview of the smart infusion systems currently on the market.

Once you know what options are available, consulting other institutions that work with the devices and vendors under consideration will add to your smart infusion system knowledge base. It is important to get a feel for the services and cooperation that a vendor provides, both before and after implementation, because this purchase will lead to a long-term relationship with your chosen vendor. Since smart infusion systems now include software that will be upgraded over time, it is likely that interactions with the vendor will occur more frequently than they did with previous generations of IV infusion pumps. 

At this point, you should contact the vendors your multidisciplinary group feels might meet your needs. Request a full in-service of their current technology, as well as an unambiguous presentation regarding their system’s future capabilities. Be sure to include all members of your multidisciplinary task force in these presentations. ECRI provides a comprehensive checklist in the October 2007 issue of Health Devices, and it should be expected that the vendor can speak to all of the items on this checklist, which include: “Can dose error reduction system data be transferred via a wireless network?” and “Must the clinician confirm whether the patient is the same as before shutoff?” The outcome of this phase in the selection process will be a list of vendors that can meet your needs from a time, cost, and future vision/partnership perspective.   

After your team has concluded which systems/vendors can potentially meet your organizational needs, it is time to further differentiate the systems from the end-user’s usability perspective. The increased complexity of IV infusion systems makes usability an important selection criterion. Therefore, the functionality of each infusion system from your own users’ perspective is a critical component of the evaluation. A human factors usability study will give you both objective and subjective data from your end-users that will assist you immensely in the selection of the appropriate smart infusion system for your organization.

What is a Human Factors Usability Study?
A human factors usability study identifies device design problems that can lead to medical errors and user dissatisfaction. Heuristic evaluation is one method of human factors analysis that is high yield, easy to learn, and low cost. This technique requires evaluators (end-users) to independently apply a set of usability heuristics (design “rules of thumb”) to a system, identify violations of the heuristics, and assess the severity of each violation.¹
Nielsen et al developed the technique of heuristic evaluation and described major heuristics that should be followed by good user interface design2, and Schneiderman also describes the eight rules of thumb that all good user interface design should follow³. Zhang J et al combined the work of these researchers to develop 14 heuristics, listed in Table 1, that should be followed when designing medical equipment.¹

Northwestern Memorial Hospital’s Usability Study
NMH utilized these 14 heuristics to develop a questionnaire (See Table 2) designed to solicit information from end-users regarding the safe use of each of the systems we selected for evaluation. We then compared the results side-by-side to determine which system would best suit our organizational needs from a safety and usability perspective. Since concern about being away from their clinical area is often a deterrent for nurses to participate in these types of evaluations, we were careful to make the questionnaire as short and concise as possible. We focused only on the heuristics found to be the most important in a published study on the impact device components have on usability⁵. Other organizations may decide to include all (or a different set) of the Nielsen-Schneiderman heuristics in their evaluation.

At NMH, two smart infusion systems were selected for the human factors usability study. The vendors were invited to bring their systems in for a full, eight-hour day. Ideally, the evaluation would be performed by the same individual with the systems side-by-side. However, the logistics involved proved to be overwhelming, and therefore, each system was evaluated on different days and, in some cases, by different nurses. Despite this potential shortcoming, we feel we received valid feedback from our end-users.

Nurses were invited from all clinical areas of the hospital to take part in this study. We provided two mock orders for the nurses to program into the smart infusion systems. The nurses did not receive any training on the equipment prior to attempting to program the system. We felt that the ideal IV infusion device should be intuitive; an experienced registered nurse should not need to be taught extensively to safely program and utilize it. A trained evaluator from our own organization observed each nurse as they programmed the mock order and documented any difficulties they experienced. The nurses were also asked to “think aloud” as they were programming, specifically regarding any confusion or frustration they were having while programming or setting up the system, so the evaluator could record these comments as well. The nurses then completed the questionnaire outlined in Table 2.

We only utilized registered nurses as end-users in the study, because knowledge in the domain in which the medical device is used typically increases the number of problems identified and, thus, makes the study most efficient and effective.^2,5 Additionally, research shows that as few as three to five individuals can be representative of an end-user population when it comes to detecting design flaws in this type of equipment.¹ However, 60 nurses from a variety of clinical areas participated in our evaluation.

It would be regrettable to omit from this article a prolific but unexpected aftereffect of performing this usability study within our own organization: The end-users’ plenteous, anecdotal feedback indicates they valued their involvement in the purchase decision, which ultimately resulted in their sustained acceptance of the technology.

Results of the Usability Study
To our surprise, it was clear from our study that one system was substantially superior from a human factors/end-user usability perspective. This information weighed significantly into our purchase decision. To view the detailed study results, download Table 3 (Results of the Usablility Study) and Table 4 (User Comments) at www.pppmag.com.

Leasing vs. Purchasing
For those who recognize that their current IV pumps are near obsolescence and need replacement, but who are not satisfied with the newer smart infusion system technology or are unsure that it will decrease error rates, a lease – as opposed to a purchase – may be a good option until the smart infusion system market matures. A cost analysis should be performed to compare a lease to a purchase. If the decision is made to purchase a smart infusion system prior to full market maturity, ask the vendor for information regarding the feasibility and cost of software upgrades as they become available.

Conclusion
Smart infusion systems with dose error reduction software have quickly become the preferred way to administer IV medications to patients. Utilizing technology to reduce medication errors is a giant gain for the patient safety movement.  The implementation of the correct system can prove fruitful, but take care to employ strong project-management leadership, sanctify the multidisciplinary nature of the decision-makers, thoroughly research the literature, and perform a usability study with your own end-users.
Keep in mind that, from the perspective of harm to patients and type and cause of errors, not all hospitals are created equal. Thus, when it comes to technology, there is no “one-size-fits-all” solution. Finally, it is critical to recognize that smart infusion systems are only part of a longer patient safety journey, which will require continuous multidisciplinary attention and enhancement in years to come.

The recipient of a BS in business education from Western Illinois University and an MBA from DePaul University, Gary Fennessy is the vice president of operations at Northwestern Memorial Hospital in Chicago and has administrative oversight for all laboratories, pharmacy, supply chain activities, environmental services, patient support services, and food services.

Marla M. Husch, RPh, is the program manager for medication systems at Northwestern Memorial Hospital. She earned her bachelor’s degree at the Purdue University School of Pharmaceutical Sciences in West Lafayette, Indiana, and is currently working toward a master’s in medical informatics at Northwestern University. Formerly a clinical pharmacist in oncology at Northwestern Memorial, Husch also serves as a member of the hospital’s multidisciplinary patient safety research team. 

A full list of references is available on the article PDF