Investigating Performance
March 2017
Introduction
Rotamap provides twice-yearly benchmarks for our client hospital organisations. We review these at our autumn and spring events where we also discuss interesting questions and themes that have emerged from operational data.
In this article:
Investigating performance
On 24 March 2017 at the Royal Society of Arts in central London we held our Spring event.
Using data from different departments and specialties, we looked into a number of stories covering the relationship between correlation and causation, and the importance of corroborative evidence.
Let us first consider the Extras boxplot produced in advance of the event. This boxplot shows us, for anaesthetics departments using CLWRota, the percentage of total sessions delivered as an extra or locum over the course of 2016.
The total height of the boxplot represents the range of data, from the lowest occurring percentage to the highest. It is worth noting that dots above and below the range are outliers that are not incorporated into the range to prevent the boxplot being skewed heavily. The thicker band in the boxplot represents the standard deviation, or in other words, where the bulk of the variance from the average lies. The white line is the average.
Fig 1. Extras/All boxplot with two departments highlighted.
We have two departments highlighted in red and blue. Both departments have a high level of variance which is indicated by the heights of their boxplots. The story behind the high variance of both these departments are different - something that can't be determined from looking at their boxplots alone.
These boxplots are a way of showing a department's percentage extra usage. As such, we can learn more about this by instead looking directly at the data that generated the plot.
We can start with the department in red. The graph below shows the percentage of sessions that were delivered by extras and locums over 2016.
Fig 2. The percentage of sessions delivered through extra and locum sessions for the department in red.
For the first six months of 2016, extra and locum sessions make up 5.6% of the department's delivery. In the latter six months, the average has increased to 17.6%. If we attempt to translate this into a monetary figure, that is an additional spend of over £120,000[1].
Now let us look into the department in blue.
Fig 3. The percentage of sessions delivered through extra and locum sessions for the department in blue.
A discernible trend is harder to spot here, however large spikes can be seen over half-terms, junior doctor strikes and summer holidays. Seasonal trends may appear to be the source for the peaks and troughs of the extra and locum usage at this department.
It could be notionally said that both departments are facing staffing issues.
Is this true? And if so, how can we find out?
For the red department, the cause for this increase is due to the department running into staffing troubles with two of their consultants[2]. This was a problem that was able to be remedied on a personal basis and their extra and locum usage dropped again recently. This is information that is impossible to glean from the data alone, and can only be known from the feet-on-the-ground perspective of those running the department themselves.
It is less clear for the blue department and to learn more we need to look at some further data.
As an initial theory into the spikes in extra and locum usage was due to seasonal trends, an interesting next step could be to compare the graph to points in time where the department's leave pools have become overdrawn.
Weeks where the consultant (red) and trainee (orange) leave pools were overdrawn have been highlighted on the graph below.
Fig 4. Highlighted weeks where consultant or trainee leave pools were overdrawn.
Interestingly there is little correlation between the spikes and where too many people have been away.
Perhaps this means that the department's stresses do not come from being overstretched when short staffed.
Next step could be to compare the percentage extra and locum usage to the percentage of additional (i.e. sessions with no original template) sessions.
Fig 5. The percentage of sessions delivered through extra and locum sessions for the department in blue alongside the percentage of additional sessions.
There is a trend to be found here - spikes in extra usage generally follow spikes in the number of sessions outside of their plan.
Though, to consider the initial theme of correlation not always implying causation, we should look at another way of observing this before making a decision on how the department could remedy their situation.
We could compare this to the departments demand versus actual delivery.
Fig 6. The departments demand vs actual delivery.
It is clear that the department is consistently delivering more than it has planned, and that this disparity is being covered by extra and locum sessions.
This, in conjunction with the correlation between extra and locum usage and additional sessions, may well provide us enough evidence to come to the conclusion that the department is simply not able to plan for the number of sessions that must be delivered.
With this data it can be found that the department is on average delivering 7 additional sessions per week. Considered financially, these additional sessions would be costing over £170,000[3]. Hiring a full time equivalent member of staff could generate a saving of around £40,000[4].
With two similar boxplots telling two different stories, we must rely on further introspection and investigation in order to truly learn how a department is operating.
In the situations we have been exploring, looking at only one data set can only ever tell us what is happening, not why it is happening.
Benchmarks
The benchmarking data for this season are below, including the first Medirota Demand/Actual boxplot.
CLWRota
Medirota
Questions
If you have any questions about the above ideas or would like to know more about how to get reports from your system please contact the Rotamap support team at support@rotamap.net or 020 7631 1555.
Footnotes
- [1] Latter 6 months contains 258 additional locum sessions, and treating a
locum session as costing £500 gives £129000 additional spend.
- [2] This was discovered simply though discussion with the rota coordinators of
the department.
- [3] 352 additional sessions at £500 gives £176,000.
- [4] FTE staff member would cover around 280 of those additional sessions for a
salary of around £100,000. This leaves 72 remaining extra sessions costing
£36,000. This leaves £40,000 saved.