Application of Balanced Scorecard in the Evaluation of a Complex Health System Intervention: 12 Months Post Intervention Findings from the BHOMA Intervention: A Cluster Randomised Trial in Zambia

Introduction

In many low income countries, the delivery of quality health services is hampered by health system-wide barriers which are often interlinked, however empirical evidence on how to assess the level and scope of these barriers is scarce. A balanced scorecard is a tool that allows for wider analysis of domains that are deemed important in achieving the overall vision of the health system. We present the quantitative results of the 12 months follow-up study applying the balanced scorecard approach in the BHOMA intervention with the aim of demonstrating the utility of the balanced scorecard in evaluating multiple building blocks in a trial setting.

Methods

The BHOMA is a cluster randomised trial that aims to strengthen the health system in three rural districts in Zambia. The intervention aims to improve clinical care quality by implementing practical tools that establish clear clinical care standards through intensive clinic implementations. This paper reports the findings of the follow-up health facility survey that was conducted after 12 months of intervention implementation. Comparisons were made between those facilities in the intervention and control sites. STATA version 12 was used for analysis.

Results

The study found significant mean differences between intervention(I) and control (C) sites in the following domains: Training domain (Mean I:C; 87.5.vs 61.1, mean difference 23.3, p = 0.031), adult clinical observation domain (mean I:C; 73.3 vs.58.0, mean difference 10.9, p = 0.02 ) and health information domain (mean I:C; 63.6 vs.56.1, mean difference 6.8, p = 0.01. There was no gender differences in adult service satisfaction. Governance and motivation scores did not differ between control and intervention sites.

Conclusion

This study demonstrates the utility of the balanced scorecard in assessing multiple elements of the health system. Using system wide approaches and triangulating data collection methods seems to be key to successful evaluation of such complex health intervention.

Trial number

ClinicalTrials.gov {"type":"clinical-trial","attrs":{"text":"NCT01942278","term_id":"NCT01942278"}}NCT01942278     

Measuring Health System Strengthening: Application of the Balanced Scorecard Approach to Rank the Baseline Performance of Three Rural Districts in Zambia

Introduction

There is growing interest in health system performance and recently WHO launched a report on health systems strengthening emphasising the need for close monitoring using system-wide approaches. One recent method is the balanced scorecard system. There is limited application of this method in middle- and low-income countries. This paper applies the concept of balanced scorecard to describe the baseline status of three intervention districts in Zambia.

Methodology

The Better Health Outcome through Mentoring and Assessment (BHOMA) project is a randomised step-wedged community intervention that aims to strengthen the health system in three districts in the Republic of Zambia. To assess the baseline status of the participating districts we used a modified balanced scorecard approach following the domains highlighted in the MOH 2011 Strategic Plan.

Results

Differences in performance were noted by district and residence. Finance and service delivery domains performed poorly in all study districts. The proportion of the health workers receiving training in the past 12 months was lowest in Kafue (58%) and highest in Luangwa district (77%). Under service capacity, basic equipment and laboratory capacity scores showed major variation, with Kafue and Luangwa having lower scores when compared to Chongwe. The finance domain showed that Kafue and Chongwe had lower scores (44% and 47% respectively). Regression model showed that children''s clinical observation scores were negatively correlated with drug availability (coeff −0.40, p = 0.02). Adult clinical observation scores were positively association with adult service satisfaction score (coeff 0.82, p = 0.04) and service readiness (coeff 0.54, p = 0.03).

Conclusion

The study applied the balanced scorecard to describe the baseline status of 42 health facilities in three districts of Zambia. Differences in performance were noted by district and residence in most domains with finance and service delivery performing poorly in all study districts. This tool could be valuable in monitoring and evaluation of health systems.     

The Sensitivity and Specificity of Using a Computer Aided Diagnosis Program for Automatically Scoring Chest X-Rays of Presumptive TB Patients Compared with Xpert MTB/RIF in Lusaka Zambia

Objective

To determine the sensitivity and specificity of a Computer Aided Diagnosis (CAD) program for scoring chest x-rays (CXRs) of presumptive tuberculosis (TB) patients compared to Xpert MTB/RIF (Xpert).

Method

Consecutive presumptive TB patients with a cough of any duration were offered digital CXR, and opt out HIV testing. CXRs were electronically scored as normal (CAD score ≤60) or abnormal (CAD score>60) using a CAD program. All patients regardless of CAD score were requested to submit a spot sputum sample for testing with Xpert and a spot and morning sample for testing with LED Fluorescence Microscopy-(FM).

Results

Of 350 patients with evaluable data, 291 (83.1%) had an abnormal CXR score by CAD. The sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of CXR compared to Xpert were 100% (95%CI 96.2–100), 23.2% (95%CI 18.2–28.9), 33.0% (95%CI 27.6–38.7) and 100% (95% 93.9–100), respectively. The area under the receiver operator curve (AUC) for CAD was 0.71 (95%CI 0.66–0.77). CXR abnormality correlated with smear grade (r = 0.30, p<0.0001) and with Xpert C_T(r = 0.37, p<0.0001).

Conclusions

To our knowledge this is the first time that a CAD program for TB has been successfully tested in a real world setting. The study shows that the CAD program had high sensitivity but low specificity and PPV. The use of CAD with digital CXR has the potential to increase the use and availability of chest radiography in screening for TB where trained human resources are scarce.     

Implementation Research to Inform the Use of Xpert MTB/RIF in Primary Health Care Facilities in High TB and HIV Settings in Resource Constrained Settings

BackgroundThe current cost of Xpert MTB RIF (Xpert) consumables is such that algorithms are needed to select which patients to prioritise for testing with Xpert.ObjectiveTo evaluate two algorithms for prioritisation of Xpert in primary health care settings in a high TB and HIV burden setting.MethodConsecutive, presumptive TB patients with a cough of any duration were offered either Xpert or Fluorescence microscopy (FM) test depending on their CXR score or HIV status. In one facility, sputa from patients with an abnormal CXR were tested with Xpert and those with a normal CXR were tested with FM (“CXR algorithm”). CXR was scored automatically using a Computer Aided Diagnosis (CAD) program. In the other facility, patients who were HIV positive were tested using Xpert and those who were HIV negative were tested with FM (“HIV algorithm”).ResultsOf 9482 individuals pre-screened with CXR, Xpert detected TB in 2090/6568 (31.8%) with an abnormal CXR, and FM was AFB positive in 8/2455 (0.3%) with a normal CXR. Of 4444 pre-screened with HIV, Xpert detected TB in 508/2265 (22.4%) HIV positive and FM was AFB positive in 212/1920 (11.0%) in HIV negative individuals. The notification rate of new bacteriologically confirmed TB increased; from 366 to 620/ 100,000/yr and from 145 to 261/100,000/yr at the CXR and HIV algorithm sites respectively. The median time to starting TB treatment at the CXR site compared to the HIV algorithm site was; 1(IQR 1-3 days) and 3 (2-5 days) (p<0.0001) respectively.ConclusionUse of Xpert in a resource-limited setting at primary care level in conjunction with pre-screening tests reduced the number of Xpert tests performed. The routine use of Xpert resulted in additional cases of confirmed TB patients starting treatment. However, there was no increase in absolute numbers of patients starting TB treatment. Same day diagnosis and treatment commencement was achieved for both bacteriologically confirmed and empirically diagnosed patients where Xpert was used in conjunction with CXR.