Using Longitudinal Data to Understand Performance
Life science, clinical and IVD manufacturing laboratories have benefited from the recent technological advances in liquid handling instrumentation, from the handheld pipette to the automated liquid handler. As automated liquid handling is a relatively new approach, standardized performance assessment and calibration guidelines do not currently exist. Yet the consequences of poor performance and liquid handling errors can be severe, ranging from inefficiency to failed experiments to the release of inaccurate results.
A critical component in a quality control program for automated liquid handlers is knowledge regarding the performance of each instrument. These data allow laboratories to take corrective action before the instrument is out of specification, ensuring maximum efficiency, and accurate and precise results.
Liquid handling accuracy and precision are critical for molecular diagnostic companies like CareDx that focus on gene expression testing. The company’s AlloMap® Molecular Expression Testing is an in-vitro diagnostic multivariate index assay (IVDMIA) test service, performed in a single laboratory for assessing the gene expression profile of RNA isolated from peripheral blood mononuclear cells (PBMC). AlloMap testing is intended to aid in the identification of heart transplant recipients with stable allograft function who have a low probability of moderate/severe acute cellular rejection (ACR) at the time of testing in conjunction with standard clinical assessment.
Automated liquid handlers are critical to the successful manufacture of AlloMap qRT-PCR assay plates as well as the preparation of patient samples for proper testing. Steps including normalizing RNA mass for cDNA synthesis and setting up qRT-PCR reactions all involve automated liquid handling. Additionally, because these assays are largely volume-dependent, inaccurate volume transfers can affect results. Due to the small liquid volumes typically handled at CareDx (five microliters on average), inaccuracy of even a fraction of one microliter can affect results and impede efficiency. Consequences of liquid handling errors include inaccurate test results and products that fail the quality control process, leading to wasted time and materials, inefficiency, and unnecessary costs. For example, the universal master mix alone can cost $20,000 for a set of 300 qRT-PCR plates for testing.
As part of our master calibration and validation plan and to avoid the costs and consequences of liquid handling error, CareDx maintains a rigorous liquid handling performance monitoring process (outlined in Figure 1) that produces a wealth of instrument accuracy and precision data. Collectively, the data provide significant insight into the need for a routine performance assessment of critical automated liquid handling transfers, especially on a tip-by tip basis. This article provides a snapshot of the longitudinal data (data showing performance over time) collected between January 2006 and March 2008 and offers guidance on optimization practices as well as frequency between calibration intervals for automated liquid handlers.
Figure 1 – CareDx’s Liquid Handling Performance Monitoring Process
Study Parameters
The data included in this liquid handling longitudinal study were compiled using both gravimetric and photometric calibration methods. Gravimetry was employed as a standard method of performance assessment prior to July 2007, where an analytical balance was used to weigh liquid volumes. After July 2007, CareDx assessed liquid handling instruments with the Artel MVS® Multichannel Verification System, which is based on dual-dye ratiometric photometry and provides tip-by-tip accuracy and precision data for transfers into microtiter plates.
The study included performance data collected from ten different liquid handling instruments (LHIs): four Biomek® 2000 instruments, one Biomek® 3000 instrument (grouped with the Biomek 2000 instruments for data analysis purposes) and five Biomek® FXp Span-8 systems (with 1000-microliter syringes). See Table 1 for a list of the liquid handlers included in the study.
Table 1: Liquid Handlers Included in the Longitudinal Study
| LHI Type | LHI ID Number | Tools/Techniques Tested |
| Biomek 2000 | 2K-1 | P20 Tool P200 Tool MP20 Tool MP20 Repeat Tool MP200 Tool P1000 Tool |
| 2K-2 | ||
| 2K-3 | ||
| 2K-4 | ||
| Biomek 3000 | 3K-1 | |
| Biomek FXp | FXp-1 | Span-8 2-10 Technique Span-8 10-100 Technique Span-8 Repeat Technique Span-8 50-1000 Technique |
| FXp-2 | ||
| FXp-3 | ||
| FXp-4 | ||
| FXp-5 |
Data from the Biomek 2000 and 3000 liquid handlers were further analyzed by tool, which is a term for different interchangeable pipetting heads that are used depending on desired pipetting function and/or volume. Biomek FXp instruments were further analyzed by pipetting technique. Each technique can have different defined pipetting parameters for the transfer of liquid. Examples of such parameters include but are not limited to aspirate height, dispense height, speed and trailing air gap. The data are grouped together by liquid handling platform (i.e., Biomek 2000/3000 and Biomek FXp) as well as by tool/technique type. Each tool has a discrete set of test volumes. Table 1 also lists the liquid handling tools and techniques included in the study.
The Biomek 2000s and Biomek 3000 are volumetrically verified on a monthly basis at CareDx and the Biomek FXp instruments are verified twice per year. Deviations from the documented frequency of maintenance can occur when an instrument is not in-service for an extended period of time or when monitoring of high usage instruments is used to generate data for longitudinal analysis (i.e., the Biomek FXp instruments were checked quarterly during this study). The performance of each instrument is qualified according to internal accuracy and precision specifications. These specifications were defined according to specific acceptance criteria required for applications at CareDx.
Tips for Optimization
After analyzing the longitudinal data, several truisms were discovered with regard to optimal use of automated liquid handlers and performance monitoring practices. From these observations, several guidelines for optimal liquid handling practices were determined.
1. Determine if error is systematic or random
The first step in optimizing liquid handling operations is to understand the type of error that exists: random or systematic. Systematic (consistent) variation indicates that the liquid handler requires maintenance due to instrument issues, such as faulty or damaged o-rings, tubing problems, or motors and pumps that have lost their home positions. Conversely, random failure is non-systematic and generally not instrument-specific, and requires a different course for corrective action. To identify, account for, and mitigate random error, regular sampling, performance assessment, and calibration are required.
In analyzing the longitudinal performance data, systematic changes were not common. Rather, the majority of variation in liquid handler performance was random. The data indicate that serious maintenance was not often required, and that regular calibration was necessary to identify and correct random variation.
2. Consider the calibration process when developing liquid handling methods
When developing methods for liquid handling operations, ensure that performance assessment and calibration processes can be implemented to verify the instrument’s performance when the assay-defined techniques and methods are employed. It is critical that pipetting parameters for a given method or assay are predefined and not deviated from. When pipetting parameters are continuously modified and customized for a particular assay, it becomes increasingly difficult to have a true determination of liquid handler performance due to routine method manipulations.
3. Mimic assay parameters during calibration
Parameters tested in the liquid handler verification process should mimic the parameters used in the normal liquid handling operation. Therefore, volume verification processes should use the same combination of tip type, liquid type and liquid reservoir as employed during the actual assay. Factors such as liquid handling mode (forward vs. reverse), dispense (wet or dry), microtiter plate geometry, aspirate and dispense speed, and air gap usage can influence volume transfer accuracy and precision. When feasible, the pipetting technique should be mimicked in the calibration process to provide a clear indication of actual instrument performance under assay conditions.
4. Choose an appropriate calibration frequency
To minimize laboratory error, automated liquid handler performance should be verified at intervals shorter than the average time that lapses between pipetting steps that drift out of specifications. Historical pipetting data can indicate instrument propensity for error over time and reveal the appropriate frequency for calibration. The calibration frequency must be balanced against the cost of calibrating as well as the time required for the technical expert to perform the volumetric verification. Calibrating too frequently can waste time, effort and money while calibrating too infrequently can allow instrumentation to operate outside of tolerances for a specific protocol and affect data quality.
To initiate a calibration plan, a conservative approach would be to verify calibration twice or even three times as frequently as specified by the manufacturer. As data is acquired on the performance of an instrument, calibration frequency can then be adjusted appropriately. Longitudinal pipetting data continually helps us to refine our calibration schedule and specifications for the liquid handling instruments.
5. Verify performance of individual channels
Often, the average dispense for all channels (tips) in an automated liquid handler will be within performance specifications while one or more channels may not meet performance criteria. Figure 2 shows varying performance information when the data are analyzed on a tip-by-tip basis:
Figure 2: Performance verification on a channel-by-channel basis for a Biomek FXp instrument
In Figure 2, channel 2 consistently failed to meet acceptance criteria. After several runs (only the first and second runs are graphed above), it was evident that this channel was not functioning properly. In this case, for a Biomek FXp, the syringe pump that controls channel 2 was replaced. Following repair and calibration, the final run data (in green) shows that all channels were able to pass internal performance specifications.
Another example highlighting the importance of individual channel performance information can be seen in Figure 3, which shows the performance of the mean over three replicates (as shown by the pink line) for the FXp-1 instrument at one calibration interval (January 2007) using the Span-8 2-10 technique. It is often the case that performance data are analyzed per system and per calibration interval. This instrument’s average volume transferred for all channels falls just within specifications, and does not indicate an issue with the liquid handler.
Figure 3: Individual channel performance compared to the average of all channels for a Biomek FXp instrument
When evaluating tip-by-tip performance for the average data shown in Figure 3, however, it is clear that multiple channels of this liquid handler (channels 5, 7, and 8) fall outside of the defined specifications. This critical information is not discovered when only an average of the eight channels is analyzed.
The channel-to-channel analysis shown in Figure 3 was possible by using a calibration method for capturing tip-by-tip data for every volume transfer. Alternatively, calibration methods that only provide aggregate information on the accuracy and precision of all channels in a liquid handler, such as with gravimetric methods, fail to offer the same critical information needed to ensure the proper performance of all liquid handling steps. As a result of only using aggregate information, subsequent assays performed with poor performing tips might be unknowingly compromised. The data indicate that the overall, average dispense should not be relied upon to reflect the accuracy of all individual channels. Each tip’s performance is essential to the success of individual assays at CareDx. For data quality, each channel and the average of the channels of an automated liquid handler should meet performance specifications.
Because the Artel MVS is employed for standardized performance monitoring, information on a per-channel basis is provided in one rapid experiment. Using gravimetry to obtain tip-by-tip data would require a skilled technician and an arduous, time-consuming process. Tip-by-tip information can be captured with the MVS within minutes for each liquid handler, whereas a comparable gravimetric method employed for each liquid handler would require multiple hours, or even days.
6. Obtain accuracy and precision information
The precision and accuracy of automated liquid handlers must be assessed in order to have a complete sense of the performance of an automated liquid handler. Data in the longitudinal study indicate several instances where an automated liquid handling tool or technique is precise and inaccurate or is accurate and imprecise (see Figure 4, which shows average volume values over replicate trials).
The accuracy of a group of repeated (replicate) measurements can be determined by first calculating the mean of the group, and then comparing that average value with the target value. Accuracy for a group of volume measurements refers to the deviation of the group’s mean value from the target volume.
Precision indicates how close a group of measurements are to one another. The closer the data replicates, the more predictable future results will be. For this reason, good precision has predictive value and gives confidence in future results. A precise or closely clustered data set has a smaller coefficient of variation (CV) and is generally more reliable than one that is widely scattered.
Most automated liquid handler manufacturers only specify precision performance. However, consider an instrument that performs precisely but not accurately, as indicated in the right side of Figure 4. According to precision-only specifications, the instrument would appear to be performing well even though almost all of the volume dispenses fell outside of our designated tolerance range for accuracy.
In addition, knowing accuracy alone is of limited use. Data on the left side of Figure 4 show that the eight replicate measurements averaged 10 microliters. However, it is impossible to predict how likely it is that the next dispense will be within the specified limits. The automated liquid handler might deliver 9.9, 10.0 and 10.1 microliters, while another might deliver 8.0, 10.0 and 12.0. The averages of both sets of data are 10 microliters and both are perfectly accurate, but for volume-dependent protocols, the first data set is clearly preferred.
Figure 4: Target Volume of 10 microliters with a +/- 6% Accuracy Tolerance
7. Use longitudinal data to troubleshoot and adjust performance specifications
Based on longitudinal data, it might be possible for laboratories to tighten certain pipetting specifications to improve the performance of their automated liquid handlers. However, it is important to balance the tightening of specifications with pushing the liquid handler past the point of its ability.
For example, in certain high-volume transfers, little variation has been observed over time, allowing specifications to be tightened to decrease the potential variability of assay performance. In Figure 5, target volumes of 700 and 900 µL +/- 7% could be tightened to +/- 3% leading to better reliability of the method.
Figure 5: Certain high volume transfers show little variation over time
In another example, we employed a serial dilution protocol (Biomek 2000 with the P1000 tool) using a concentrated template. By evaluating the longitudinal data, it was determined that the accuracy specification of each volume transfer could easily be tightened from 6% to 3% without having the instrument operate out of tolerance. The tightened specifications resulted in improved efficiency and better performance of the method using the P1000 protocol, serving to reduce waste in terms of dead volume without extra time or labor hours required.
If operations are not negatively affected, it is often beneficial for laboratories to loosen certain pipetting specifications. This loosening reduces the burden on technical experts performing the volumetric verification in assessing the performance of automated liquid handlers. Certain tools/techniques have traditionally been very problematic due to their high variance and frequency of drifting outside of the specifications. After conducting studies to determine confidence intervals around expected values, or “guard band studies,” it is then possible to adopt looser specifications for particularly troublesome pipetting steps.
Historical performance data also provide input to laboratory managers to assist in troubleshooting assays that produce incorrect results. Without historical data, it is difficult and time consuming to rule out the instrument as the root cause of the questionable data, requiring a lengthy calibration process (4-6 hours). Alternatively, when historical data are available, the instrument has already been characterized and only a quick spot check needs to be conducted, requiring less than 30 minutes of time, to eliminate the instrument as the error source.
For example, in Figure 6, the March 2008 data points for two different techniques of the FXp-1 automated liquid handler are all within the acceptable range but are on the low end of specifications for the 5µL and 10µL target volume transfers. In the case where a liquid handing method utilizes pipetting steps with these two techniques, all calibrated on the low end of the acceptable accuracy range, the cumulative effect could be detrimental to the final results. In this example, the method was not designed with the specifications of the volume transfers in mind and therefore longitudinal data helped to identify root cause of the errors.
Figure 6: Longitudinal data for 5mL & 10mL Biomek FXp techniques
Conclusion
There are a number of steps that can be implemented to optimize performance monitoring of automated liquid handlers. Critical to the process is the consistent collection of tip-by-tip performance data on automated liquid handlers. This information can shape quality control practices and improve confidence in data produced using automated liquid handling instruments. If implemented properly, these suggestions can improve the robustness of laboratory operations and, in the clinical setting, reduce risk to patient health.
Sidebar 2: Quick Tips for Optimizing Automated Liquid Handlers
- Determine if error is systematic or random
- Consider the calibration process when developing liquid handling methods
- Mimic assay parameters during calibration
- Choose an appropriate calibration frequency
- Verify performance of individual channels
- Obtain accuracy and precision information on a per tip basis
- Use longitudinal data to troubleshoot and adjust performance specifications
About the Authors
Wade Yandell, Automation Specialist, CareDx
Wade has been with CareDx for five years, where he has worked within the Automation group in the evaluation, maintenance, and development of programs for a variety of automated liquid handling instruments. He was also the point person in the implementation and integration of the Artel MVS into regular maintenance procedures. He earned a B.S. in Biology from the University of Oregon.
David Wexler, PhD, Associate Director, Automation, CareDx
Dr. Wexler received a PhD in Physical Chemistry from the University of California, Los Angeles in 1994. Subsequently, he worked as a NIH postdoctoral fellow at the University of California, Berkeley where he studied the photophysics of rhodopsin. During his time in Berkeley, he designed, built, and patented a high speed rotary confocal fluorescence scanner for detection of capillary array electrophoresis chips. Dr. Wexler was recruited in 1998 by GeneTrace Systems, Inc. as an automation scientist developing methods for high-throughput purification of RNA and DNA for further analysis by mass spectrometry. He then joined AGY Therapeutics, Inc. in 2000 where he established a high-throughput micro array platform for the identification and validation of novel CNS therapeutic targets. As AGY evolved into a drug discovery company, Dr. Wexler led the effort to create a state of the art high throughput screening facility for small molecules. During his last 2 years as the Head of Automation, Dr. Wexler was overseeing the development and screening ofin vitro ADME-Tox assays. Dr. Wexler currently leads the Automation group at CareDx. His team is responsible for researching, implementing and maintaining the instrumentation for developing and running the gene expression testing of transplantation and autoimmune patient samples.
About CareDx
Based in Brisbane, California, CareDx is a molecular diagnostics company focused on the discovery, development and commercialization of non-invasive gene expression testing in the areas of transplant medicine and autoimmunity. The company has developed a proprietary method of utilizing gene expression in blood that provides a new tool for physicians to manage the care of heart transplant patients. The molecular expression technology developed and implemented by CareDx in heart transplant patient management is currently being explored to assist with other diseases that involve the immune system. Learn more at http://caredxinc.com/.
