CHAPTER 1
Proficiency Testing in the Context of Valid Analytical Measurement
1.1 What is Proficiency Testing?
A proficiency testing (PT) scheme comprises the regular distribution of test materials to participating laboratories for independent testing. The results are returned to the organiser of the scheme who makes an analysis of the results and reports them to all of the participants. The primary function of the scheme is. to assist the participants to detect shortcomings in their execution of the test procedures and apply suitable remedial measures to make up any deficiency. This is a particularly important function in analytical measurement, which is fraught with practical difficulties and prone to unsuspected errors. As well as this primary 'self-help' ethos, there is also an element of accreditation in many proficiency testing schemes. Accreditation agencies will normally expect candidate laboratories to participate satisfactorily in a proficiency test in areas of analytical measurement where one is available.
Proficiency tests in analytical chemistry usually comply with a particular type, where the distributed materials are portions taken from an effectively homogeneous bulk material that resembles the normal test materials as closely as possible. The materials are distributed for analysis with choice of methods open, although participants should use their usual procedures. The estimated true value of the measurand is not revealed to the participants until after the collection of all of the results. The proficiency test therefore provides for the participant an independent check on the accuracy of the analytical result, albeit only at the time of the test. As the scheme is a regular event, it allows a laboratory to compare its results (i) with an external standard of quality, (ii) with the results of its peers and (iii) with its own past performance. Participation in a proficiency testing scheme therefore results in a much higher proportion of the participant laboratories reaching a satisfactory standard than would otherwise be the case. However, no consistent long-term improvements in performance will result unless the participation is in the context of an integrated quality assurance system within each laboratory.
Care must be taken not to confuse proficiency testing with other types of interlaboratory study, that is those designed for validating analytical methods or certifying reference materials. The operational protocols for these three types of study are quite distinct (Section 4.3).
1.2 Importance of Valid Analytical Measurements
Accurate and dependable analytical measurements are essential requirements for sound decision-making on many issues of vital interest to society. Environmental scientists and regulatory authorities need reliable data on the nature and extent of environmental contamination in order to identify cost-effective pollution-abatement technologies. The health and safety of the working population is safe-guarded by a number of regulations that prescribe maximum permissible levels of hazardous substances in the workplace environment. The effective enforcement of these regulations depends critically on the availability of sound analytical data. The welfare of the population at large is dependent on water and food supplies of high quality and this quality can be checked only by the careful application of a variety of different analytical techniques, each of which is often of considerable technical complexity, and therefore prone to error.
When valid measurements are not realised, data of poor quality are reported by a laboratory. In such circumstances the ensuing problems and costs for the end-user of the analytical data can be substantial and are associated with such consequences as:
• the costs involved in repeat measurements to correct poor data;
• the faulty decision-making that ensues when invalid results are acted upon;
• damage to reputation and credibility that results when an end-user is associated with poor data;
• possible loss of business where the end-user's customer is compromised by poor data;
• any legal and financial liability incurred from the use of poor data.
Hence it is important that those who commission laboratories to undertake analytical measurements on their behalf appreciate the critical need to select only competent laboratories.
There are a number of key concepts and practices that laboratories can apply to help them to improve, maintain and demonstrate the validity of their data, namely the use of properly validated methods, the use of internal quality control procedures, the use of certified reference materials, third party accreditation and participation in proficiency testing schemes.
1.3 The VAM Proficiency Testing Project
The material in this book has been enhanced by unique information gathered by the authors during a project on proficiency testing completed in 1994 as part of the UK Government's Department of Trade and Industry's Valid Analytical Measurement (VAM) Initiative. This wide ranging project studied a number of important aspects of proficiency testing (PT). An efficacy study was undertaken to evaluate the effectiveness of proficiency testing as a means of improving the quality of analytical data and the essential features of effective schemes were identified. An investigation was carried out to establish the degree to which good performance in one particular test in a PT scheme correlates with good performance in a second, different, test within the same scheme. Such information contributes to the design of cost-effective schemes, by enabling optimum information on analytical proficiency to be obtained on the basis of a laboratory's performance in a limited number of critically selected tests. Other factors relating to the costs, benefits and resource requirements associated with proficiency testing were also reviewed.
The structure and operation of several UK proficiency testing schemes were surveyed, from the perspectives of scheme organisers, participant laboratories and end-users of analytical data (i.e. customers of analytical laboratories). Organisers of 18 schemes in the fields of chemistry and microbiology took part in the survey, which was conducted by means of face-to-face interviews and questionnaires; the participant survey secured questionnaire returns from over 200 participants in 13 different schemes. The proficiency testing schemes involved in these surveys are listed in Appendix 2. Telephone interviews were used to carry out a survey of 61 individuals classed as end-users of analytical data. The objective of the survey work was to identify current best practice and to pinpoint areas where improvements in the operation and use of proficiency testing procedures are required.
It was recognised at the outset that proficiency testing is only one of several quality. assurance measures available to analytical laboratories. Consequently, the way proficiency testing interacts with and complements other quality assurance measures was investigated, so that the particular and unique role of PT could be identified and developed. A review of the statistical procedures required to handle the data generated in PT schemes was also undertaken, with the objective of identifying appropriate procedures for evaluating the performance of scheme participants.
In order to test or validate specific aspects of proficiency testing procedures recommended in this book, certain practical studies were carried out. Some of these involved collaboration with two proficiency testing schemes set up by the Laboratory of the Government Chemist as part of the VAM proficiency testing project. These schemes were the CONTEST scheme dealing with the analysis of contaminated land and the BAPS scheme dealing with the analysis of beer. The validation studies included a comparison of different procedures for establishing the assigned values of test materials; an assessment of the uncertainty associated with assigned values; an appraisal of the value of ranking laboratories according to their performance score, and the use of formal measures of efficacy to assess scheme effectiveness.
1.4 The Origin and Development of Proficiency Testing
Interlaboratory cooperation in pursuit of analytical quality began in earnest towards the end of the last century. Collaborative studies to validate analytical methods were being organised in the USA by the Association of Official Agricultural (now Analytical) Chemists in the 1880s and data from such studies have been reviewed by Horwitz. Similar studies were underway in the UK by the early 1930s under the direction of the Society for Analytical Chemistry and since 1980 this work has been continued by the Analytical Methods Committee of The Royal Society of Chemistry. In the early 1900s the first chemical reference materials (cast irons) certified for composition were produced in the USA by the National Bureau of Standards (now the National Institute for Science and Technology). In contrast to the above, interlaboratory programmes for the formal assessment of the proficiency of analytical laboratories are a rather more recent innovation.
In the field of clinical chemistry, where the analysis of human tissue has a direct impact on patient care, the first formalised survey of the proficiency of clinical laboratories took place in the USA in 1947. The survey revealed large variations in results from different laboratories and, as a consequence, by the early 1950s the College of American Pathologists (CAP) had instituted regular proficiency testing in several laboratory disciplines in the health care area. A subsequent development in the USA has been the legislative requirement (e.g. Clinical Laboratory Improvement Amendments, 1988) that clinical laboratories participate satisfactorily in an appropriately designed proficiency scheme.
According to Bullock, the first proficiency survey of UK clinical laboratories was reported in 1953 and revealed a wide spread of results for the determination of common constituents of blood. There followed other ad hoc surveys in the 1950s and 1960s, all of which confirmed the need for proficiency assessments on a regular basis. In 1969, the National Quality Control Scheme, in which test specimens prepared from human serum were distributed to the 200 participating laboratories every 14 days, was initiated by the Wolfson Research Laboratories, Birmingham, with funding from the UK's then Department of Health and Social Security. Now known as the UK National External Quality Assessment Scheme (UK NEQAS) for general clinical chemistry, this scheme currently has about 600 participating laboratories. Concurrently, several other UK NEQAS activities have been developed, covering such areas as haematology, microbiology, immunology and drug assays. Within the last few years the funding of the UK NEQASs has shifted significantly to a self-financing basis, with operational costs now being covered by fees paid by participants. In addition to schemes such as the above which originated in the public sector, schemes have also been developed as commercial activities by private sector organisations. For example, in 1971 Wellcome Diagnostics (now Murex Diagnostics) initiated a clinical chemistry scheme, which was followed in 1978 by a scheme covering immunoassays.
The occupational hazard arising from exposure to asbestos dust has long been recognised and in order to assess the quality of asbestos monitoring a proficiency testing scheme was established in the USA in 1972, by the National Institute for Occupational Safety and Health. A similar scheme was established in the UK in 1979 by the Institute of Occupational Medicine (IOM), following a recommendation of the UK's Advisory Committee on Asbestos.
Initially the activity was confined to a small group of specialist laboratories, but in 1984 the scheme, under the name of RICE (Regular Interlaboratory Counting Exchanges), was expanded to incorporate routine testing laboratories and it now has about 300 participants. A stimulus to participation in the RICE scheme is the legislative requirement of the Control of Asbestos at Work Regulations (1987) that laboratories undertaking asbestos fibre counting have the necessary facilities for producing reliable results.
The UK Control of Substances Hazardous to Health Regulations (1988) limit exposure to toxic chemicals in the work environment, and the proficiency testing scheme known as WASP (Workplace Analysis Scheme for Proficiency) deals with measurements of this type. WASP was set up in 1988 by the Health and Safety Executive and currently has about 200 member laboratories.
International legislation has also contributed to the development of proficiency testing in recent years. For example, the UK Ministry of Agriculture, Fisheries and Food initiated the Food Analysis Performance Assessment Scheme (FAPAS) in 1990, in part to meet UK obligations arising from various EU Directives covering the control of foodstuffs. A further impetus to the development of FAPAS was the evidence from ad hoc interlaboratory trials of poor quality analytical data for certain types of food analysis. The AQUACHECK proficiency testing scheme for water analysis was established by the Water Research Centre in 1985 and the proficiency assessments reflect, in part, the requirements of relevant EU legislation on water quality.
Laboratory accreditation bodies have also encouraged the growth of proficiency testing schemes. In 1981, the National Association of Testing Authorities (NATA) in Australia set up a Proficiency Testing Advisory Committee and since then NATA have completed over 100 proficiency tests, some of which were one-off exercises, while others are on-going. In the UK, the UK Accreditation Service (UKAS) strongly endorses the role of proficiency testing in quality assurance and in some areas, for example asbestos testing, maintenance of a laboratory's accredited status is dependent on satisfactory performance in an appropriate PT scheme, such as RICE.
In addition to schemes such as those above that are run primarily on a national basis (although a small proportion of their participants may be from other countries), some schemes have been developed from their inception as international activities. Thus the scheme referred to as the Asbestos Fibre Regular Informal Counting Arrangement (AFRICA) and established in the mid-1980s by the IOM operates on a broadly similar basis to the RICE scheme, but has participants from over 30 different countries. The Wageningen Agricultural University in the Netherlands currently operates a number of international proficiency testing schemes concerned with the analysis of environmental samples such as soils, sediments and plant tissue. Participants number around 250 and are drawn from about 70 countries. With the increasing importance of the internationalisation of trade, environmental protection, law enforcement, consumer safety, etc. the need to demonstrate comparability of data between laboratories on an international basis will continue to grow and the need for fully international proficiency testing schemes can be expected to grow also. In this respect it is of interest to note that the EU, through its Standards, Measurements and Testing Programme, is currently (1995) proposing the establishment of a 'pan-European' proficiency testing network, aimed at supporting particular EU directives and regulations.
A further current development, stemming from the significant increase in proficiency testing activities in recent years, has been an international initiative to harmonise the operational procedures adopted by the organisers of proficiency testing schemes. Agreement on the protocols to be followed is considered essential if all parties are to have the necessary confidence in the assessments of laboratory proficiency ultimately produced. In a further recent activity (June 1995) an expert group was set up by the International Organisation for Standardisation (ISO) to undertake an extensive revision of the ISO Guide on proficiency testing first issued in 1984.
It is difficult to estimate the total number of proficiency testing schemes currently operating, but a UK listing published in 1994 (see Appendix I) recorded 26 different schemes, covering a wide range of measurement disciplines in the chemical and microbiological areas. For example, in addition to the schemes already referred to, there are schemes dealing with the analysis of such materials as fertilizers, animal feedstuffs, alcoholic beverages, contaminated soils, petroleum products, cereals and cane sugar. There are also schemes covering forensic investigations, the microbiological examination of foods and waters and veterinary investigations. It is possible that there are yet other schemes currently operating and the need to minimise duplication of effort in the establishment of new schemes means that there is a requirement for a comprehensive register of all existing proficiency testing activities. This particular issue is currently being addressed on a Europe-wide basis in the VAM 1994/1997 project on proficiency testing being carried out at the Laboratory of the Government Chemist.
