Data for improvement

When testing your change ideas with PDSA cycles, you will need to start collecting data to be able to determine if the changes made have resulted in an improvement. This data should be collected in 'real time' rather than retrospectively.

What is a Family of Measures?

One measure alone is insufficient to determine if improvement has occurred. You are advised to include one or two measures from each of the following three categories.

It is important to define the numerator and denominator and provide an operational definition for each measure to ensure data consistency.

QIDS icon
Measurement examples

Note: Join the Med Rec QI COP to access QIDS

Use the measurement strategy template to record all details of your measures if you don't have access to QIDS.

Outcome measures are closely aligned with your aim statement or the overall outcome you are trying to achieve. They relate to how the overall process or system is performing. In other words, the end result.

  • Example: Completion rate of admission medication reconciliation within two business days of admission.
    • Numerator: Number of patients with admission reconciliation completed within two business days
    • Denominator: Total number of patients admitted to the ward for more than two business days
    • Operational definition:
      • Admission Reconciliation – Medication Reconciliation completed with two business days of admission (includes Best Possible Medication History)
      • Completion of Admission Reconciliation – the green tick that appears on the Cerner eMeds Admission Medication Reconciliation module
    • Data source: obtain from eMeds report

Process measures are the parts or steps in the process performing as planned. They are logically linked to achieve the intended outcome or aim.

  • Example: Percentage of medical officers trained on completing Best Possible Medication History (BPMH) (from education attendance sheet).

Balancing measures look at the system from different directions or dimensions. They determine whether changes designed to improve one part of the system are causing new problems in another part of the system.

  • Example: Rate of patients who did not receive discharge counselling from pharmacist.

What to consider before collecting data?

Think critically about the data you collect, how much, where to record and who can assist.

Before commencing PDSA cycles, the project team should:

  • Review any baseline or historical data on performance of the process to be improved
  • Consider the need to collect baseline data for the measures in order to determine the impact of the project
  • Agree upon what should be measured - outline how, who, when and where the project data will be collected from for each measure
  • Determine the most efficient way to access and collect the data
  • Consider how useful the data will be and how you will present it (don't collect unnecessary data that won’t be used)
  • Make sure to speak with staff and patients to hear about their experiences whilst you are testing
  • Decide where to record data and how it will be accessed by the team (e.g. spreadsheet, QIDS - preferred)
  • Consider assigning responsibility to individual team members for data collection for each measure
  • You will still need to continue collecting data after the project to check that the improvements are sustained.

Rather than collecting a big sample size, you want to make sure the data is project specific and can be collected continuously. Consult your QI advisor if you are unsure.

  • For example, if you need to manually collect data by going through progress notes, you may choose to audit a random selection of five patients per week. In contrast, you can collect longer periods (e.g. 6-12 months) of consecutive data if you are generating data from an eMeds report.

Contact data experts or colleagues who can advise you on the type of data to collect and how much to collect (they may already have the data you need):

  • Health information data team
  • ICT / eMeds team
  • Local clinical governance / patient safety team
  • Medical records
  • Pharmacy department
  • Executive Sponsor.

How do you make sense of and present your data?

Once data has been collected and entered in a spreadsheet or QIDS, the project team needs to interpret the data in a meaningful way to determine if an improvement has occurred. The QIDS – improvement projects platform also has the functionality for you to easily build different charts suitable for your project.

Run charts are line graphs showing data over time. Run charts are an effective tool to tell the project story and communicate the project’s achievements with stakeholders. Run charts illustrate what progress has occurred, what impact the changes are having and ultimately, if improvement is happening. Including annotations in your run chart will help to show when change ideas have been tested and may be associated with an improvement. There are specific rules to interpreting run charts which can be found via the CEC QI Academy website. Your local QI advisor may be able to assist with the display and analysis of data.

The example below is a run chart on admission medication reconciliation completion rate by week. The stretch goal is 100% and the project team has tracked a median Med Rec rate of 90%.

Med Rec Run Chart example

QIDS icon
Run Chart examples

Alternatively use the Med Rec Run Chart template.

There are a number of different charts (e.g. Histogram, SPC Chart etc.) which can be used to present your data. Visit the CEC QI Academy webpages for more information.

Determining if improvement has really happened and if it is lasting requires observing patterns over time.

Probability-based rules are helpful to detect non-random evidence of change:

For example, if you are using a run chart, an improvement is considered reliable when six consecutive data points are above 95%. That is, compliance with the new process implemented occurs 95% of the time.

For more information on types of data, minimum data point and probability-based rules visit the CEC QI Academy website. You can also contact your local QI advisor for assistance.

How do you implement the change?

Once the project team is confident that the change idea (or bundle of change ideas) is resulting in a reliable improvement, the project team can focus on implementing the change ideas across the project scope.

Use the flowchart for implementing formal medication reconciliation processes.