CHAPTER 1
The Potential Use of Quality Control Data to Validate Pesticide Residue Method Performance
William Honvitz
CENTER FOR FOOD SAFETY AND APPLIED NUTRITION HFS-500, US FOOD AND DRUG ADMINISTRATION, WASHINGTON DC 20204, USA
Full scale interlaboratory (collaborative) studies are becoming too expensive and time-consuming to support their use as the only way to validate methods of analysis. Furthermore, reliable estimates of method performance parameters, such as accuracy, precision, and limits of applicability, cannot be achieved by individual collaborative studies at the concentration levels of 0.01-1 mg. kg-1 the region of interest for residue analysis, when the expected random error among laboratories is of the order of 20-30% of the mean. Performance data from proficiency studies of tens of thousands of control determinations accumulated over the past decade from individual and multiple laboratories are being examined to determine their potential as a substitute for interlaboratory performance data. An analysis of variance indicates that as much as 80% of the total variability of pesticide residue analysis is "random error." If this is the case, proficiency data may be substituted for method-performance data, when recovery is acceptable, because the individual factors of analyte, method, matrix, laboratory, and time contribute little to overall variability.
1 INTRODUCTION
A full-scale method performance study, utilizing the IUPAC-AOAC harmonized protocol,' requires a minimum of 8 laboratories to analyze at least 5 materials related to the analyte-matrix-concentration combinations of interest by the proposed method. Although such a study is probably the most pertinent and reliable way of demonstrating the performance of a method with a specific test sample, several alternatives are also available. The Youden pair technique (split level design) conducted at several relevant concentration levels is the model favored by the extensive water analysis program of the U.S. Environmental Protection Agency (EPA). In Europe, the "uncertainty" budget approach has been advocated, particularly in conjunction with laboratory accreditation, although this design has been modified during the past few years to become more similar to the method-performance model.
All of these models are expensive and time-consuming. What few calculations that have been conducted independently suggest that they do not provide the same performance parameters with the same method from the same data and information. Proficiency testing, also advocated by the FAO/WHO Codex Alimentarius Program as a way of demonstrating equivalency in laboratory results, has been suggested as still another potential substitute for the method-performance trials. The purpose of this paper is to explore this possibility.
2 AVAILABLE PROFICIENCY STUDIES
Data from several large scale proficiency programs have been made available to us in order to examine their potential use as a substitute for method performance in the determination of pesticide residues in food. As of late 1999, data from three programs have been examined. These include the following:
2.1 Total Diet Program (FDA-TD)
The Total Diet Program of the U.S. Food and Drug Administration (FDA), Kansas City Laboratory, has been in continuous operation for almost 40 years. The recoveries of approximately 7000 control determinations of analytes added to individual foods in the pesticide residue portion of the 4 annual market baskets analyzed during 1991-1997 were available as databases in EXCEL4. This is a within-laboratory program.
2.2 Food Analysis Performance Assessment Schemes (FAPAS)
This is an international fee-based program conducted by the Ministry of Agriculture, Fisheries, and Food of the United Kingdom. Each test material is thoroughly homogenized to a smooth paste and aliquots of stock standard solutions of pesticides of known purity are added with additional mixing. The fortified test material is measured into screw top glass jars, tested for homogeneity, and stored at -20° until shipped. More than 100 laboratories participated in many of the tests. Participants were required to submit their results within 8 weeks for their values to be included in a report that was distributed. The pesticide residue reports (Nos. 1901-1905), uncorrected for recovery, and percent recovery of about 1100 concurrent controls that were submitted, were the data used in the present examination.
2.3 State of California Quality Assurance Program (CA)
The State of California has been monitoring the quality of their Residue Enforcement Program since 1988, utilizing their headquarters and field laboratories and an occasional contract laboratory. The laboratories are supplied centrally with a control spiking solution containing 3-8 pesticides quarterly. The pesticides are selected from those encountered routinely in the state. The laboratories selected a test sample daily to be fortified with the control solution. Both test samples, spiked and unspiked, were conducted through the entire analytical procedure of extraction, isolation, and measurement. The commodity selected for fortification could be new, one that had shown problems, or was the subject of a special application investigation. The results are submitted to the Quality Assurance Unit of the State Laboratory for collation and about 15000 data points are reviewed here.
3 PROCEDURE
The data was usually available as percent recovery for each control sample (fortified analyte/food) from relatively long time periods as in the case of the FDA-TD and the CA programs and from many laboratories in the case of the FAPAS program. These records were calculated to an overall pooled recovery and relative standard deviation for each analyte/food combination and, when available, for substantially different characteristics (laboratory, method, type of food). To keep the examination within reasonable limits, only those combinations with at least 8 values were reviewed. The HORRAT values were also calculated for each group from the following formulae:
Relative standard deviation = RSD = s x 100/[bar.x], (1)
where s is the standard deviation within-laboratory from FDA-TD, and among-laboratories for the FAPAS and CA programs.
Predicted RSD = PRSD = 2 C(-0.1505), (2)
where C is the concentration of the pesticide added, as a decimal fraction (1 mg.kg-1 = 10-6). Equation (2) is the so-called "Horwitz-curve,n6 and
HORRAT = RSD/PRSD. (3)
A HORRAT value of about 0.5-0.7 is expected from within-laboratory studies (FDA-TD) and about 1 from among-laboratories studies (FAPAS and CA ), although values in the interval from about 0.5 to 2 times the expected value may be acceptable.
3.1 Outlier Removal
A concurrent investigation suggests that all values outside of a recovery of 100 [+ or -] 50% be removed as beyond acceptable limits for the concentration levels examined in these studies (about 0.01-1 x 10-6). Such limits usually result in the removal of an acceptably small fraction (i.e., <5%) of results. These limits also happen to approximate the 3-sigma quality control limits that would be calculated from the Horwitz formula (1) for a concentration between 0.5 and 1.0 x 10-6.
4 RESULTS
Considering all of the studies as a group, the overall recovery of pesticide residues in general is about 90%. The among-laboratory precision is about 15%. These values have remained approximately constant since the introduction of multiresidue methods. The FDA-TD program utilizes a within-laboratory model whose variability is expected to be one half to two thirds that of the among-laboratories models used in the other two programs. Each of the programs exhibit some special features discussed below.
4.1 FDA-TD
To keep this database within reasonable bounds, Table 1 summarizes the salient information only from those analytes with approximately 100 or more records and related entries. For this database, outliers, defined as recoveries >150% or <50%, are removed prior to calculating the statistics. In most cases few values had to be removed. If a substantial number of values had to be removed, the method(s) is considered inappropriate for the analyte.
Table 1 gives a summary of the analytical characteristics of the major analytes included in the FDA-TD market baskets over a recent 5-year period where a substantial number of values were available for examination. A more detailed examination of all of the data will be presented separately. In general, for most of the pesticides, the differences in extraction and measurement methods that are made to accommodate high or low fat, moisture, and sugar contents of various foods and the use of different types of columns, conditions and detectors make little difference in the overall recoveries. The extensive details that would be required to list each different method or condition used would require too much space and are not pertinent to the present summary. Those pesticides that deviate substantially in recovery and precision (ETU, herbicides) are known to produce problems in routine analytical work or an inapplicable method may have been used (e.g., pentachlorophenol).
The acceptable HORRAT values in this data set average about 0.5, which is somewhat better than would be expected for multiple-analyst, within-laboratory data. The typical overall analytical variability is so large (ca 10%) at these low levels (20-500 µg.kg-1) that statistical tests of significance are generally meaningless. In most cases there were insufficient numbers of values to provide reliable statistics for individual analyte/food combinations, aside from the general categories of high and low fat and moisture with some analytes.
4.2 FAPAS
This is a fee-based program operated to meet the ISO requirements for accreditation. Although a list of about 20 potential analytes is usually supplied to participants, no more than about 6 pesticides are present in any single test sample. About half of the participants voluntarily supplied control data in addition to the required program data. This resulted in the availability of analyte/commodity data from test material analyzed both as a known and as an unknown. This fortuity permitted applying a correlation coefficient (r) calculation to the two types of data with the unexpected result of an average r ≈ 0, i.e., the analysis of a test sample as a known does not correlate with its measurement as an unknown. Another interesting finding was that the analyte thiabendazole analyzed by HPLC by 15 laboratories showed definite evidence of censoring the data when analyzed as a known (matrix recovery) compared to the similar analyses analyzed as an unknown (spike recovery), as shown in Figure 1. The values exhibit a much tighter cluster when reported as a known, as compared to the wider cluster when reported as an unknown. Although rare, such reports occasionally do appear in the literature. Until the questions raised by these findings are resolved, control values reported on known additions must be viewed with skepticism.
4.3 CA Program
Almost 15000 values from about 65 analytes in numerous foods, analyzed by typically 5 laboratories, are available from this pioneer quality control program that has been operating for over a decade. A preliminary analysis of the available data is given in Table 1. The original data was first checked for the effect of outlier removal, using limits of 100 [+ or -] 40% (maximum removal), 100 [+ or -] 50%, and 100 [+ or -] 60% (minimum removal). Although, on an overall basis, outlier removal has a negligible impact on the statistical parameters calculated from 15000 records, they may affect calculations of the analyte/food/laboratory combinations.
Some preliminary conclusions are: (1) Most of the outliers (values outside 100 [+ or -] 50%) are from the smaller analyte/matrix combinations (<8 values). (2) Limits of 100 [+ or -] 50% strike a reasonable balance between excessive and restrained outlier removal. (3) A small percent of values are reported as "0." On review, these may be found to reflect clerical errors (failure to record a value, recording an incorrect value, or not adhering to a scheduled protocol) rather than an analytical failure. The implication of their presence is considerably greater if the data is used to support method performance rather than routine laboratory quality control.
Table 2 provides the initial analysis of variance of all the data from the CA program, with values outside of 100 [+ or -] 40%, 100 [+ or -] 50%, and 100 [+ or -] 60% recovery removed as "outliers." With no values removed, 82% of the variance is "random." Only 18% of the variance is attributable to specific factors, primarily analyte and method. Food and laboratory make a negligible contribution. If "outliers" are removed, about 75% of the variance is "random," but which outlier-removal procedure is used is immaterial. Only 25% of the variance is attributable to specific factors, with food and laboratory again making a negligible contribution.
5 DISCUSSION
The analysis of variance shows that the major factor in the variability of pesticide analysis is "random error" and therefore is irreducible under the conditions of these studies. This conclusion is reinforced by the constancy of the precision and recovery of pesticide residue analysis over the last quarter century. The improvements that have been made in the direction of better columns and instrumentation require relaxing control of operating conditions to permit optimizing resolution, sensitivity, and peak sharpness and shape, and minimizing baseline interference. Such general directions provided under the title "system suitability tests" at best can only maintain current among-laboratory precision as characterized by equation (2). System suitability requirements have attained a high degree of refinement in the pharmaceutical analysis and are recognized in the official compendia.
The necessity for the use of broad control limits (i.e, [+ or -] 50%) for individual analyses or for numerous replicates to reduce variability, decreases the value of quality control specifications to monitor analyst or method performance. The tendency of analysts to censor their values, as illustrated in Figure 1 by the tighter clusters of results when controls are analyzed as knowns (right side) as compared with results from similar analyses when conducted as unknowns (left side), must be overcome. The use of automation, at least in the chromatographic, measurement, and calculation steps, may surmount the natural tendency of analysts to provide expected rather than actual values.
Imposition of general specifications is likely to be self-defeating because they would have to be sufficiently broad to overcome the local systematic errors of individual laboratories. But, necessarily, imposition of local specifications would not apply to other laboratories.
