Benchmarking Outcomes That Matter Most to Patients: The Globe Programme - European Medical Journal

Benchmarking Outcomes That Matter Most to Patients: The Globe Programme

6 Mins
Download PDF
Andre G. Dias,1 *Charlotte J. Roberts,2 Jacob Lippa,1 Jason Arora,2 Mats Lundström,3 Ola Rolfson,4 Sarah T. Tonn1

The authors have declared no conflicts of interest.


The GLOBE programme is funded by ICHOM, sponsoring organisations with contributors from ICHOM’s data partner, ICON. All seven authors contributed to the concept, content, and editing of this article.

EMJ. ;2[2]:42-49. DOI/10.33590/emj/10310677.
Outcomes, benchmarking, standardisation, patient reported outcome measure (PROM), patientcentred, value-based healthcare

Each article is made available under the terms of the Creative Commons Attribution-Non Commercial 4.0 License.


Correction Notice: This article was first published online 7th June 2017. Since then a correction has been made. In Figure 1, the sentence ‘Brigham and Women’s Hospital, Massachusetts, USA’ has been removed from the top left text box, as the institution became uninvolved in the programme.

Significant variation in health outcomes exists around the world. The International Consortium for Health Outcomes Measurement (ICHOM) has developed Standard Sets of outcomes for important medical conditions and populations to enable outcome measurement and comparision in order to understand variation and stimulate improvement. ICHOM has recently launched a prospective, non-interventional, observational pilot benchmarking programme. This article reviews the pilot methods, timelines, expected outputs, lessons learnt to date, and the next steps. We believe this programme is truly innovative as it will be the first global initiative in Standard Sets benchmarking, provider engagement, and risk-adjustment on the outcomes of care of most importance to patients. It has the potential to bring significant changes to compare the quality of healthcare and health systems around the world and ultimately to improve patient care.


In 2012 the International Consortium for Health Outcomes Measurement (ICHOM) was founded as a non-profit organisation to develop Standard Sets of outcome measures for different medical conditions and populations, as well as drive their adoption by healthcare institutions. It is our belief that the systematic measurement of Standard Sets of outcomes by institutions around the world will enable, for the first time, global outcome comparisons. We think this will catalyse a new wave of innovative learning for healthcare professionals, as institutions will be able to see where the greatest outcomes are being achieved and then learn from the processes that they have resulted from, as well as really inform patient choice. ICHOM has now created Standard Sets of outcomes that matter most to patients for 21 medical conditions, including prostate cancer, cataracts (CAT), hip and knee osteoarthritis (HKO), and stroke amongst many others.

The concept of value-based healthcare1,2 has been defined by Prof Michael E. Porter, Bishop William Lawrence University Professor, Harvard Business School, Boston, USA, and Prof Elizabeth Teisberg, Full Professor, Dell Medical School, University of Texas, Austin, USA, as the patient outcomes achieved per dollar expended.3,4 Outcomes are the results people care about most when seeking treatment, including functional improvement and the ability to live ‘normal’, productive lives. Current metrics for care across many conditions tend to capture processes and costs and do not measure whether they achieve the outcomes which matter most to patients. The process of developing a Standard Set has been previously described.5-7 In developing globally-agreed-upon Standard Sets of outcomes for different medical conditions and patient populations, we aim to enable providers to measure the most important outcomes for their patients and then compare themselves, in a consistent manner, with other countries around the world.8

The application of performance benchmarking, transparency in reporting, improved performance, and collaborative learning, gained from the processes that the best performers share, ultimately informs patient choice towards value. The ICHOM Global Outcomes Benchmarking (GLOBE) programme was launched in May 2016 and aims to globally compare standardised outcomes between international partners to enable the identification of treatment paradigms that are more effective, leading to improvements in healthcare outcomes. The launch of two prospective, non-interventional, observational pilot studies using the ICHOM HKO,9,10 and CAT11,12 Standard Sets which include patient-reported outcome measurement (PROM) instruments, assess the feasibility of collecting outcome data from multiple international institutions, and provide risk-adjusted benchmarks. The GLOBE pilot programme will bring together highly motivated participants including institutions, hospitals, and registries that are keen to compare outcomes and learn from each other.


The idea of performance benchmarking as a mechanism to drive improvement is by no means a new one. A myriad of examples in the literature indicate that, when thoughtfully organised and well managed, the systematic collection of standardised data sets accelerates improvement upon measurement of process, structure, and outcome indicators which inherently reduces total cost (see Table 1).

Table 1: Five notable initiatives to systematically collect standardised datasets.
PB: performance benchmarking; PR: public reporting; BP/G: identification and dissemination of Best Practices or Clinical Guidelines.
Adapted from Larsson et al.18

At its most basic level, benchmarking sets the bar with regards to what is achievable, providing motivation for clinicians, care teams, and institutions to adopt best practices and/or evidence-based guidelines, or to innovate on their own. When combined with efforts to identify the underlying factors that lead to high performance, and co-ordinated quality improvement activities focussed on disseminating these lessons, the benefits of benchmarking can be amplified.

ICHOM’s GLOBE programme builds on available examples in two important ways. Firstly, it will be the primary truly global initiative, unlike numerous existing national or regional benchmarking programmes. By including institutions from around the world, we expect to see wider variation with regard to both processes and outcomes, thereby expanding the possible research questions that can be investigated and increasing the likelihood of discovering new best practices in pockets of innovation and excellence. Secondly, this will be the first initiative to emphasise PROM such as mobility, pain, and quality of life. This is critically important if we are to guide improvement efforts in a way that is compatible with patients’ preferences and values.

Global Outcomes Benchmarking Pilot Programme Objectives

The GLOBE pilot programme is a proof-of-concept initiative that aims to test the feasibility of setting up an international benchmarking programme based on the ICHOM Standard Sets. The pilot has five primary objectives:

a. Identify and overcome legal and technical hurdles to aggregating data from an international set of providers

b. Assess the appropriateness of the ICHOM CAT, and HKO sets for benchmarking and define the required changes

c. Design risk adjustment methodologies to adequately adjust participant outcomes between sites

d. Deliver risk-adjusted outcomes, reporting to participating institutions/hospitals so that they can understand their relative performance

e. Determine how pilot sites can leverage outcomes data to learn from one another

Pilot Overview

International Consortium for Health Outcomes Measurement community

ICHOM supports institutions, hospitals, and organisations in implementing Standard Sets of outcomes and develops case studies of examples of outcome measurement around the world. These have been described in further detail in case studies,19-21 and an example of implementation has been described in Box 1. The importance of supporting implementation of the Standard Sets is two-fold: scaling the effective use of standard PROM instruments for outcomes; and for decision-making in performance benchmarking, transparency, and narrowing variation toward improved outcomes. The programme design and success of the GLOBE pilot programme is largely based on the ICHOM community of implementers of ICHOM Standard Sets.

Box 1: Implementation Example: ABUHB and the implementation of ICHOM Standard Sets.
ABUHB: Aneurin Bevan University Health Board; VBHC: value-based healthcare; ICHOM: International Consortium for Health Outcomes Measurement.


The two ICHOM Standard sets, HKO9 and CAT,10 were chosen for initial examination in the GLOBE pilot. According to a 2010 assessment, 51% of world blindness is caused by CAT,22 and HKO has been ranked as the 11th highest contributor to global disability in the 2010 Global Burden of Disease.23 It is therefore globally appropriate to start with these two conditions. Using the lessons derived from this pilot, ICHOM intends to launch benchmarking programmes across additional Standard Sets going forward.


The GLOBE pilot participants were recruited from ICHOM’s international group of strategic partners, members of Standard Set implemention communities, the working groups of the ICHOM HKO, and CAT Standard Sets, and from the ICHOM global community. Participants determined which hospitals within their institutions would be participating. All self-identified as being interested in the ICHOM GLOBE pilot given their engagement with the ICHOM community. Participating institutions were required to sign a memorandum of understanding (MOU) governing data privacy and security, namely requiring the de-identification of data gathered by participating institutions, prior to transmission of data. As the need and requirements for ethics boards and patient consent varies by country, the responsibility of acquiring regulatory approvals and/or ethics committee review for participation rested with the institution. Each institution sought approval for i) the use of data for benchmarking analysis and ii) the transfer of de-identified patient data outside the operating jurisdiction. The project is best classified as a service evalution or clinical audit that does not require informed consent as no patient care is impacted. The GLOBE programme provides a resource guide, study plan, data dictionary, and MOU to aid approval processes.

As of December 2016, the institutions identified in Figure 1 had agreed to contribute data to the pilot. Due to the popularity of the programme, recruitment is ongoing for both pilots at the time of this publication.

Figure 1: Hip and knee osteoartritis and cataract GLOBE pilot participants as of December 2016.
GLOBE: The ICHOM Global Outcomes Benchmarking (GLOBE) programme; ICHOM: International Consortium for Health Outcomes Measurement.

Outcomes collected

The endpoints collected are based on the ICHOM HKO and CAT Standard Sets that have been previously detailed and published.10,12 Standard Sets include baseline conditions and risk factors to enable meaningful case-mix adjustment globally, ensuring that comparisons of outcomes account for differences in patient populations across not just providers but also countries and regions. High-level treatment variables are included to allow stratification of outcomes by major treatment types. Additional measurement time points have been added, where appropriate, to decrease the timeline needed to observe outcome differences between sites. Additionally, a pilot objective is to test the Standard Sets for the purposes of detailed benchmarking and identify required modifications to ensure the ICHOM HKO, and CAT sets can be used for benchmarking going forward.


Both the CAT and HKO programmes are prospective, non-interventional observational studies that are expected to run for 15 and 18 months respectively. Each pilot is divided into three main phases: Programme Design, Data Collection, and Analysis and Reporting, each with a specific goal (Figure 2).

Figure 2: Overview of pilot phases for the HKO programme.
HKO: hip and knee osteoarthritis.

Programme design

The aim of the Programme Design phase was to recruit potential pilot participants and provide a framework to work together to prepare sites to sign up for the pilot. There are four major engagement steps that were explained to onboard pilot participants:

a. Onboarding session: For each interested institution, a call was set up that included representation from the hospital leadership, clinical and programme management teams to discuss the pilot methodology and objectives. The aim of this session was to assess alignment with pilot objectives and ensure adequate resources were available at the hospital site to execute the pilot objectives.

b. Gap analysis assessment: All interested sites completed a gap analysis which assessed their sites’ overall alignment with the ICHOM Standard Sets. Sites were vetted to ensure that they collected the correct outcomes, which were based on the ICHOM defined outcome definitions, at acceptable time points. Sites that diverged from the ICHOM Standard Set recommendations or collected an incomplete set were coached on what gaps needed addressing to participate in the pilot.

c. Legal session: An exploratory discussion was held with interested sites to understand the legal and regulatory requirements for their participation in the pilot programme. These discussions were used to inform the legal contract ICHOM developed for this pilot discussed in Box 2.

Box 2: Developing a legal framework for data transfer.
CAT: cataract; HKO: hip and knee osteoarthritis; ICHOM: The International Consortium for Health Outcomes Measurement.

d. Technical: A technical discussion was held with a member of the Technical/Informatics department to provide an overview of the data transfer process and assess if sites were technically able to extract and transfer the complete ICHOM Standard Set data.

Data collection

During the data collection phase participants submit data through a secure ftp protocol to ICHOM’s data partner’s (ICON) server on a monthly basis and receive validation or data quality reports post-data submission. This process is followed so that institutions and their hospital sites can receive timely feedback on their collection and make any required adjustments as early as possible in the pilot. An example of the data collection process for the CAT pilot is shown in Figure 3.

Figure 3: Overview of the CAT pilot data collection phase.
CAT: cataract.

Analysis and reporting

The final phase of the pilot will focus on i) testing and implementing risk adjustment techniques to allow for adjusted comparisons between institutions and their hospital sites, ii) assessing potential visualisation techniques to allow for comparisons between providers, and iii) developing reporting options to showcase relative performance and facilitate collaborative sharing of best practices.


Currently the HKO and CAT pilots are in the data collection phase with both expected to deliver results by the end of 2017, after which each pilot is expected to transition into a fully defined benchmarking programme. To date, the pilots have overcome a number of potential hurdles including, but not limited to, defining a legal framework (Box 2) to allow for data sharing, developing a data transfer process that accommodates institutions with different collection systems and technical capabilities, and identifying new variable definitions/requirements to help capture process differences between sites.


The HKO and CAT pilots will provide ICHOM with important lessons to allow it to develop a benchmarking programme that draws outcomes data from an international set of contributors and both analyses and reports back to providers on their relative performance. Lessons learned from evaluating risk adjustment within these two pilot conditions will also inform future Standard Sets and improve upon an ongoing benchmarking programme.

The risk-adjusted benchmarking performance data will provide an immense opportunity to improve patient care by identifying differences in hospital performance that can be used to facilitate learning programmes between institutions. Over time, the programme will likely incorporate additional data sources and expand its scope of analyses. Potential options include but are not limited to the identification of determinants of hospital performance, an understanding of individualised treatment response at the patient level, or an assessment of the most efficacious set of medical interventions for a given patient subpopulation. Outcomes data sharing programmes like this will provide the largest repository of structured data that has been collected to date, linking both underlying patient characteristics and treatment options to patient outcomes. In this exciting future, we expect the ICHOM GLOBE programme to serve a pivotal role in facilitating, sharing, and utilisation of data at an international level.

Porter ME. A strategy for health care reform--toward a value-based system. N Engl J Med. 2009;361(2):109-12. Porter ME. What is value in health care? N Engl J Med. 2010;363(26):2477-81. Porter ME, Teisberg EO. Redefining Health Care: Creating Value-Based Competition on Results. N Engl J Med. 2007;356:316-7. Porter ME, Teisberg EO. Redefining Competition in Health Care. Harv Bus Rev. 2004;82:64-76. Martin NE et al. Defining a standard set of patient-centered outcomes for men with localized prostate cancer. Eur Urol. 2015;67(3):460-7. McNamara RL et al. Standardized Outcome Measurement for Patients With Coronary Artery Disease: Consensus From the International Consortium for Health Outcomes Measurement (ICHOM). J Am Heart Assoc. 2015;4(5):e001767. Allori AC et al. A Standard Set of Outcome Measures for the Comprehensive Appraisal of Cleft Care. Cleft Palate Craniofac J. 2016. [Epub ahead of print]. Porter ME et al. Standardizing Patient Outcomes Measurement. N Engl J Med. 2016;374(6):504-6. ICHOM. Hip & Knee Osteoarthritis Standard Set Reference Guide and Flyer. 2015. Available at: Last accessed: 31 October 2016. Rolfson O et al. Defining an International Standard Set of Outcome Measures for Patients With Hip or Knee Osteoarthritis: Consensus of the International Consortium for Health Outcomes Measurement Hip and Knee Osteoarthritis Working Group. Arthritis Care Res (Hoboken). 2016;68(11):1631-9. ICHOM. Cataracts Standard Set Reference Guide and Flyer. 2015. Available at: Last accessed: 31 October 2016. Mahmud I et al. A Proposed Minimum Standard Set of Outcome Measures for Cataract Surgery. JAMA Ophthalmol. 2015;133(11):1247-52. Cystic Fibrosis Foundation. Cystic Fibrosis Patient Registry: Annual data reports. 1995–2009. Available at: Last accessed: 2 November 2016. Van Leersum NJ et al. The Dutch surgical colorectal audit. Eur J Surg Oncol. 2013;39(10):1063-70. Lundström M et al. Evidence-based guidelines for cataract surgery. Guidelines based on data in the EUREQUO database. J Cataract and Refract Surg. 2012;38(6):1086-93. Jernberg T et al. Association between adoption of evidence-based treatment and survival for patients with ST-elevation myocardial infarction. JAMA. 2011;305(16):1677-84. Kärrholm J et al. Improved Analyses in the Swedish Hip Arthroplasty Register. Annual Meeting of the American Academy of Orthopaedic Surgeons. 2007. Larsson et al. Use of 13 disease registries in 5 countries demonstrates the potential to use outcome data to improve health care’s value. Health Aff (Millwood). 2012;31(1):220-7. Arora J et al. How Payers and Clinicians Partnered to Build National Outcomes Registries in the Netherlands: The Dutch Institute of Clinical Auditing (DICA) London, UK: International Consortium for Health Outcomes Measurement (ICHOM). 2015. Available at: Last accessed: 4 November 2016. Arora J, Aisenbrey J. Aligning payers and providers around value: Blue Cross Blue Shield of Michigan’s Collaborative Quality Initiatives. Cambridge, MA: International Consortium for Health Outcomes Measurement (ICHOM). 2016. Available at: Last accessed: 4 November 2016. Arora J, Lippa J, Stoefs, J. Stanford Health Care: A Seamless Outcomes Data Capture Solution. Cambridge, MA: International Consortium for Health Outcomes Measurement (ICHOM). 2014. Available at: Last accessed: 4 November 2016. World Health Organisation. Visual impairment and blindness 2010. 2012. Available at: Last accessed: 2 November 2016. Cross M et al. The global burden of hip and knee osteoarthritis: estimates from the global burden of disease 2010 study. Ann Rheum Dis. 2014;73(7):1323-30.

Please rate the quality of this content

As you found this content interesting...

Follow us on social media!

We are sorry that this content was not interesting for you!

Let us improve this content!

Tell us how we can improve this content?