Introduction
Clinical laboratories report patient blood or other body fluids test results that are widely used in clinical and public health settings. The outcome on the patients depends on the accuracy of the testing and reporting. It is always a double target to improve the quality of test reports accompanied by reduction of cost in healthcare system of both public and private sectors. This pressurizes to implement total quality management in the clinical laboratory which includes quality planning, quality laboratory process, quality control, quality assessment and quality improvement.
Quality refers to satisfaction of the needs and expectations of the users or customers. Fundamental requirements for all objective quality control systems are clearly defined quality goals. Laboratories must define their service goals and establish clinical analytical requirements for testing processes. Without such quality goals, there is no objective way to determine whether acceptable quality is being achieved.
There is a link between sigma level of an analyte and the QC rules that are applicable in terms of number of QC runs per day, size of patient samples between the runs and the applicability / exemption of outliers as per Westgard rules.1 Therefore, regular monitoring of sigma levels of individual analytes have an impact on improving efficiency and cost of laboratory operations.
Sigma score also gives an objective measurement of quality of the method, reagent kits and analyzer used. This not only helps better patient care but also help improve clinicians’ confidence on the reports.
Aims and Objectives
The study was undertaken to assess and compare performance of routine assays in terms of sigma score year-on-year in the laboratory and overall performance with published studies. This helps to improve the quality and reliability of assays based on their baseline sigma score, as applicable, with minimum 3 sigma for baseline and 6 sigma score as target score for each assay. The second aim of this study was to assess sigma metrics for tests used for analytes which are considered as routine diagnostic tests (not screening tests) and having a high severity risk index such as Creatinine, Glucose, Amylase, Calcium and Uric Acid.2 Effect of revision of CLIA’s allowable total error {TE(a)} of some analytes on sigma level was also explored retrospectively.
Materials and Methods
Sigma metrics for performance of routine analytes was estimated by Sigma score estimation on 16 biochemical analytes which were routinely performed in VITROS 5600 integrated system (Ortho Clinical Diagnostics, USA), based on IQC data.
Internal quality control analysis was performed twice a day for Cholesterol, Triglyceride, HDL, Creatinine, Uric Acid, SGPT, Glucose analytes using two levels of controls and once a day of both levels for Albumin, Total Protein, Chloride, Potassium, Sodium, Phosphorous, Urea, Calcium, AST. The quality control (QC) outliers were identified based on Westgard multi-QC rules adopted by the laboratory, including 1-2s, 1-3s, 2-2s and R-4s. Root cause analysis of QC outliers were performed, corrective and preventive actions were carried out effectively and documented. The long-term performance of the assay in both level controls were evaluated by calculating the cumulative co-efficient of variation (CV%) estimated on monthly basis. The clinical laboratory participated in the Randox External Quality control program (RIQAS) on monthly basis and the obtained results were analyzed in comparison with the peer group mean values. This helps to calculate the bias in performance of each assay in our laboratory in comparison with the peer group performance.
Sigma metric calculation of routine chemistries in analytical phase
By considering the performance of each of the 16 routine assays in both internal quality control program and external quality assurance program, Sigma (σ) metrics was calculated for each analyte, monthly with the following formula.
Formula used for estimating Sigma score (monthly):
Where TEa% and CV% indicates total allowable error percentage, as per CLIA and coefficient of variation percentage, respectively. Coefficient of variation (CV%) was derived from calculated laboratory mean and standard deviation (SD) of internal QC data for both level controls using the formula,
CV% for each analyte was extracted month wise from VITROS 5600 integrated system, as the system has internal software for quality control data analysis. Percentage of Bias was calculated for each analyte month wise, by doing comparison between the value obtained in the laboratory for each RIQAS sample and the peer group mean value obtained for each analyte, from monthly RIQAS report by using the following formula,
Peer group mean is the mean of values obtained for each analyte by the laboratories who are using the same method in instruments from the same vendor (Ortho Clinical Diagnostics). Total allowable error (TEa) of each analyte was selected from acceptance limits of proficiency testing guidelines of Clinical Laboratory Improvement Amendment (CLIA) 2012 and 2019.
The quality performance of analytes (individual lab test) were classified based on the achieved average of monthly sigma scores across Quality Control levels, for each year viz ≥6 sigma level as very good performance, ≤5.9 to 3.0 as good performance, <3 sigma: moderate performance for the current study as well as in analysis of the cited publications.
Improving performance of analytes
Quality improvement was compared year to year based on average Sigma score achieved for each analyte using internal quality control (IQC) levels 2 and 3 which cover the reference range and medical decision limits (level 1 is not applicable for VITROS). QGI as an index of corrective action required in terms of imprecision or bias was also calculated. The QGI ratio was calculated using the following formula, QGI = Bias/1.5 × CV%. The criteria used for interpreting QGI when test parameters fall short of Six Sigma quality is as follows::
< 0.8 indicates imprecision, 0.8 to 1.2 indicates both bias and imprecision and > 1.2 indicates bias. Corrective actions for quality improvement were based on QGI and was done on a monthly basis, but only for analytes which were performing at lower than six sigma.
Results
Figure 1 shows Graphical representation of the sigma levels of all analytes.
Table 1 and Table 2 show CV% & Bias% along with Sigma Score all sixteen analytes from April 2017 to March 2021.
Table 3 shows the comparison of performance in terms of improvement or decline (blocked cells) for IQC levels 2 and 3 for analytes with sigma scores more than six initially.
Table 4 shows the comparison of performance in terms of improvement or decline (blocked cells) for IQC levels 2 and 3 for analytes initially scoring less than six sigma.
Table 5 shows performance comparison of all analytes in cited studies and current study in slabs namely > 6 sigma, sigma between 3 and 6 and sigma score below 3.
Table 6 shows the QGI scores for analytes for which sigma score <6 and where laboratory had taken action as per specification from Randox EQAS for the periods of the study.
Table 7 shows the comparison of performance in terms of sigma scores for analytes for which the TE(a) has been revised by CLIA with effect from 2019.
Figure 1
Sigma scores as bar chart over four years of 2017, 2018, 2019 and 2020 along with table showing sigma level scores
Table 1, Table 2 below shows average CV%, average Bias and Sigma scores for years 2017-18, 2018-19, 2019-20 and 2020-21.
Table 1
Table 2
Table 3
Analytes with sigma scores at or above 6
Table 4
Analytes with sigma scores less than 6
Table 5
Comparing sigma scores with published literature
|
Average Sigma score |
Study reference #3 using TEa from CLIA guidelines = 16 analytes (Total #, % analytes out of total) |
Study reference # 4, using TEa from CLIA guidelines = 13 analytes |
Study reference # 5, using TEa from CLIA guidelines = 10 analytes |
Study reference # 6, using TEa from CLIA guidelines = 14 analytes |
Study Reference # 7 using TEa from CLIA Guideline = 19 analytes |
Study Reference # 8 = 20 analytes |
Sigma scores on VITROS 5600 analyser using Tea from CLIA guidelines in 2020 – 2021 = 16 analytes |
|
|
Average Sigma score of > 6 sigma |
ALP, Mg, TRIG, HDL (Total 4, 25%) |
Uric Acid, Total Bilirubin, AST, ALT, ALP (Total 5, 38%) |
TRIG, HDL and creatinine (Total 3, 30%). |
ALP, ALT, AST, TBIL, HDL, Triglycerides and Uric Acid, LDL (Total 8, 57%) |
TG, CK, GGT, TBIL, UA, ALP (Total 6, 32%) |
ALKP, AMYLASE, AST, GGT, MG, TRIG (Total 6, 30%) |
ALB, GLU, TP, UA, TRIG, K, PHOS, Creatinine, Calcium, AST, ALT and HDL (Total 12, 75%) |
|
|
Average Sigma score between 3 and 6 (>3<6) |
Creatinine, AST, ALT, Total protein, Calcium, Phosphorus and Sodium (Total 7) |
Glucose, Urea, Creatinine, Triglycerides (Total 4) |
Cholesterol and direct bilirubin (Total 2) |
Albumin, Cholesterol Creatinine, Glucose, Total Protein (Total 5) |
AST, CHOL, TP, CREAT, ALB, GLU, K+ and NA+, ALT (Total 9) |
ALT, TBIL, Calcium, HDL Cholesterol, Urea, Uric acid, Albumin, Cholesterol (Total 8) |
CHOL, CL, and Urea. (Total 3) |
|
|
|
||||||||
|
Average Sigma score less than 3 |
Urea, Total bilirubin, Albumin, Cholesterol and Potassium (Total 5) |
Total Protein, Albumin, Total Cholesterol, Chloride (Total 4) |
Glucose, chloride, albumin, total bilirubin and protein (Total 5) |
Urea (Total 1) |
CA, UREA, PHOS, CL- (Total 4) |
Creatinine, Glucose, Phosphorus, Potassium, Na+ and Total protein (Total 6) |
Na+ (Total 1) |
Table 6
QGI scores considered for action only for analytes scoring on average less than six sigma
|
Test |
2017 |
2018 |
2019 |
2020 |
Impression |
|
QGI |
QGI |
QGI |
QGI |
||
|
CHOL |
2.66 |
2.33 |
1.91 |
2.54 |
Bias |
|
CL- |
0.57 |
0.42 |
0.46 |
0.62 |
Imprecision |
|
NA+ |
0.28 |
0.39 |
0.39 |
0.61 |
Imprecision |
|
UREA |
1.75 |
2.56 |
1.99 |
3.06 |
Bias |
Table 7
Sigma score comparison based on revised guidelines for TEa with effect from 2019 derived from tables 3 and 4
Discussion
Performance of routine analytes based on allowable total error-TE(a)
Table 3 shows that nine analytes for IQC Level 2 & 11 analytes for IQC Level 3 for the year 2017-18, 10 analytes for IQC Level 2 & 11 analytes for IQC Level 3 in 2018-19, 11 analytes for IQC Level 2 &12 analytes for IQC Level 3 in the year 2019-20 and 10 analytes for IQC Level 2 and 13 analytes for IQC Level 3 in 2020-21 are above 6 Sigma score which denotes that the performance was very good. Overall, across all four years of this analysis, 09 routine analytes showed performance at sigma score six or more, in at least two levels of internal quality control, as shown in Table 3. Since sigma scores are dependent upon the allowable total error allotted by the guidelines, the performance has been analyzed separately for those having TE(a) more than 10% and lower than 10%.
Table 4 shows that six out of 16 analytes (37.5%) scored less than six sigma. Out of these, there were two analytes whose Sigma score performance was between 4 to 6 for all the four years on both IQC levels between 2017 and 2021 (March) namely, cholesterol and urea. It is brought to notice that only for Sodium in Level 3 in 2017 level 2 & 3 2018, 2019, 2020 and 2021 (March), Sigma score was below 3.
Analytes with TE(a) beyond 10%
Some analytes have high allowable total error (TE(a)) viz >10 (Table 1, Table 2), for example, Uric acid, Triglycerides, Potassium, Creatinine, AST, ALT and HDL. Consequently, when the performance on bias and imprecision (CV%) are monitored well and controlled, these analytes can have high sigma scores as seen in this study. These analytes can be regarded as having potential for higher than six sigma scores. In the current study, all these analytes except ALT had higher than six sigma average score each year, while ALT was at threshold level sigma of 5.9 and 5.5 in 2017, 2018 and were improved beyond six sigma in consequent years (newer version ALTV was used in year 2019-20 onwards). When performance of this group of analytes are reviewed in published studies, it can be seen that, while other analytes scored more than six sigma, Creatinine in five out of six studies performed at lower than six sigma levels, in contrast to the current study where Creatinine was consistently scoring above six sigma across four years, as can be derived from Table 5 and Table 1, Table 2. On the other side, Uric acid was one analyte showed consistency in scoring across sites at more than six sigma in four out of 5 studies including this study, the outlier being in study by Kavita Aggarwal et al.8 Out of 7 study sites including this study referred here as in Table 5, six sites included ALT for sigma analysis; it is noteworthy to mention that this study compares well with two sites viz Nanda et al4 and V Thomas et al6 for performance of ALT at more than six sigma level while others saw this analyte performing between 3 and 6 sigma.
Analytes with TE(a) <10%
Some analytes have low TE(a) < 10 (Figure 1), for example, Albumin, Glucose, Total Protein, Cholesterol, chloride, Phosphorus, Sodium and Urea. Hence, even if the bias and precision are maintained well for these analytes, it can be expected typically to have sigma scores lower than six sigma, due to the low target TE(a). Yet, in this study, Glucose, Total Protein and phosphorus in this group, performed on average above six sigma level each year whereas Albumin, Cholesterol, chloride and urea had average sigma score between 4 and 6 excepting Albumin which was performing at threshold level of 5.7 to 5.9 sigma in 2017 and 2018 but improved beyond six sigma in consequent years.
Performance of analytes based on severity index for harm
Some analytes are considered as diagnostic tests (not screening tests) and having a high severity index for harm if TE(a) is exceeded, namely Creatinine, Glucose, Amylase, Calcium, Uric Acid and Sodium.2 Therefore, it is important to have minimum sigma score of 3 in these tests. In this study, the average sigma level for these analytes, except sodium, were more than six sigma in all four years (2017-18 to 2020-21) which shows excellent quality performance of these critical analytes on VITROS 5600 analyzer used in this study. Amylase was not included in this study.
Sodium is a classic example of an analyte with a challenge of having high severity index for harm.2 It has a narrow range of biological variation and small changes are clinically significant and hence has been allotted a low target TE(a) in CLIA guidelines. Westgard S et al9 mention that the tightening of goals is definitely reducing the Sigma-metric of the assays. Sodium and Urea, in particular, were among the tightest goals in the current CLIA goal system, and they haven't gotten any better. It is, therefore, difficult to sustain sigma score near an acceptable level of 3 sigma score, for these two analytes. However, in this study, average sigma score for both IQC for Sodium was closer to acceptable minimum of three sigma performing between 2.2 to 3.1 sigma level (average / year for both IQC levels) with an average of 2.64 across all four years of the study while urea performed better with sigma scores between 4 and 6 across four years Table 1, Table 2. Comparing the performance of sodium in a study by Bhakti Gami et al10 wherein they obtained sigma values of 1.71, 1.20 and 2.64 for three different analysers for the same set of samples used, the performance of sodium in this study was comparable with one of the analyzers in the study. In the context of improving performance using QGI score, Sodium QGI was less <0.8 across four years, indicating opportunity for improvement being imprecision (Table 6). However, efforts were made to improve imprecision but couldn’t be improved from imprecision level of 0.8% CV which in itself is low imprecision for any analyte. Hence, unless the guidelines revise the TE(a) to higher level, sodium is likely to remain at low levels of sigma, across methods / analyzers.
Comparing sigma scores with published literature for all analytes overall: The total number of analytes performing at more than six sigma level were 12 out of 16 (75%) in this study in 2020 -2021 (Table 5), which is seen to be better than studies by Zhou et al7 who found that 6 out 19 analytes (32%), Sahar Iqbal and Tazeen Mustansar5 who found 3 out of 10 analytes (30%), Kavita Agarwal et al8 who found 6 out of 20 analytes (30%), Vijatha Thomas et al6 who found 8 out of 14 analytes (57%) and Nanda et al4 5 out of 13 analytes (38%) performing at ≥ 6 sigma level. Overall, it can be deduced from Table 5 that while 25 to 57% of routine analytes performed > six sigma in published studies cited, the current study in year 2020-21, had higher percentage of analytes (75%) ≥ six sigma. Such difference in sigma levels of performance of individual analytes amongst clinical labs, as reflected in this study, may be due to use of different analyzers/ kits used, status of process controls in place and for some analytes, target for TEa which is different from published CLIA guidelines was used.7, 3
Analytes in this study performing below six sigma viz between sigma levels of 3 and 6 in 2020-2021 (Table 4) is compared with performance in other studies as shown in Table 5. These were three analytes namely Urea at sigma 4.36, Chloride at sigma 5.26 and Cholesterol at sigma 5.19. It is noticed that these analytes have performed similarly at sigma scores below 6 in other studies as well. The reason for lower than 6 sigma performance for them could be the target allowable error is 10% or below for these analytes as mentioned in Table 1, Table 2.
Regarding use of QGI in comparing the cause of poor performance of analytes (below six sigma), Vinodh Kumar and Thuthi Mohan3 observed that for all analytes <6 sigma level, the quality goal index (QGI) was <0.8 indicating the area requiring improvement to be imprecision except cholesterol whose QGI >1.2 which required improvement in accuracy. In current study, only analytes with less than six sigma were acted upon for improvement based on QGI. For example, for analytes urea, chloride and cholesterol which scored <6 sigma, Urea and Cholesterol had QGI > 1.2 showing the cause to be inaccuracy (bias) and for chloride QGI <0.8 showing the cause of imprecision (Table 6). While monitoring and using QGI constantly to improve performance in terms of sigma score does help, for some analytes the improvement reaches a plateau before reaching the desired six sigma. For example, efforts were taken to remedy bias for urea and cholesterol and but the imprecision for chloride being already at very low level of 0.84 CV%, may be difficult to improve drastically further.
In summary, as in Table 5, 12 out of 16 i.e. a high percentage of routine analytes (75%) performed comparatively better at an average > 6 sigma on V5600 analyzer in 2020-21, in this study.
Applying new TE(a) targets as per recently revised CLIA guidelines
Recently in 2019, CLIA guidelines revised TE(a) for the analytes listed in Table 7. Hence, the new applicable TE(a) was applied across 2017 to 2021 for comparison of performance for these analytes as shown in this table. It can be seen that with the new TE(a), the analytes whose average performance across four years shifted lower than six sigma were Glucose, Total protein, potassium, creatinine, ALT, Albumin & HDL. Yet Glucose, Total Protein and HDL were at the threshold value of 5.7 to 6.0 sigma with the new TE(a). As cited by Westgard S et al,9 following analytes are known to shift to lower sigma levels on application of the revised CLIA TE(a) goals, namely Glucose, Creatinine, HDL, ALT, Mg, Alkaline Phosphatase, Triglycerides and potassium. Hence, the shift noticed in the analytes on applying the revised CLIA TE(a) goals is expected. However, efforts are ongoing to improve their performance as per the revised CLIA guidelines.
Path forward
As part of efforts for continuous improvement of quality assurance in the laboratory, it is intended by the authors to consider risk analysis for laboratory processes using sigma metrics in next phase, whereby its utility in identifying risk mitigating mechanism will contribute to overall process improvement.
Conclusion
Sigma score for each assay, in addition to QGI score, helps to focus on improving assay performance in terms bias or imprecision, on a monthly basis, thereby increasing accuracy of all results reported. All analytes with an allowable total error of > 10% as per CLIA performed at more than six sigma across four years of the study. While 25% to 57% of routine analytes performed ≥ six sigma across published studies cited, the current study in year 2020-21, in comparison, had higher number (12 out of 16) and percentage of analytes (75%) ≥ six sigma on the VITROS 5600 analyzer. Tests used for diagnostics which have high severity risk index such as Sodium, Creatinine, Glucose, Calcium and Uric Acid, all except Sodium had average sigma scores of more than six in VITROS 5600 analyzer used in this laboratory. On comparing performance of these important analytes across studies cited here, it was noticed that creatinine was a differentiator in this study which consistently performed at >6 sigma while in five of six studies cited the sigma score is less than six. In the context of the revised CLIA guidelines in 2019 wherein TE(a) for ten routine analytes have been reduced, it was noticed that with the lowering of TE(a) for many of these analytes, except uric acid, Triglycerides and AST, the performance shifted from > six sigma to a sigma lower than six. Yet, Glucose, Total protein and HDL among these, were at the threshold values of 5.7 to 6 sigma with the downward revised CLIA TE(a).
