Lessons from an IT disaster

By Tony Collins

Only rarely is an independent report on an IT-related disaster published.  So North Bristol NHS Trust deserves credit for publishing a report  by Pricewaterhousecoopers into the problematic go-live of Cerner Millennium in December 2011.  PwC calls the Cerner system a “business-critical patient record system”.

The implementation, says PwC,  resulted in significant continuing  operational difficulty. PwC was asked to review the implementation, identify what went wrong and make recommendations.

What is clear from PWC’s report is that North Bristol NHS Trust repeated the known mistakes of other trusts that had gone live with Cerner Millennium:

–          A lack of independent challenge

–          Not enough testing of the system and new business processes

–          Inadequate contingency arrangements

–          Not enough time for data migration

–          Training systems not the same as those to be used

–          Preparations treated as an IT project, not a change programme.

–          Differences between legacy and Cerner systems not fully understood before go live

–          Staff did not always understand new or changed business processes

In 2007 the National Audit Office reported in detail on the lessons from the go-live of Cerner Millennium at Nuffield Orthopaedic Centre, Oxford in December 2005.

One of those lessons was that the Trust did not learn lessons from earlier NPfIT Cerner Millennium go-lives. This happened again at North Bristol, suggests the PwC report:

“There were not dissimilar Cerner implementations within the Greenfield [other ex-Fujitsu and now BT-managed Cerner Millennium implementations under the NPfIT] systems running a few months before NBT’s [North Bristol Trust] implementation. Similar difficulties were experienced there, but they were more successfully addressed.”

Below are extracts from PwC’s report “Independent review of Cerner Millennium implementation North Bristol NHS Trust”.

“The success of an implementation of this scale, complexity and timing depends on substantial, robust and enduring programme management focusing on:

–          The IT implementation. Incorporating configuration of Cerner Millennium, infrastructure, security, interfaces and testing;

–          The migration of data from the two legacy PAS systems into Cerner Millennium;

–          Change management to engage and train stakeholders, embed change in the organisation and ensure that processes and procedures are aligned to the new system;

–          Continuous communication with users about changes to business processes as a result of the implementation; and

–          Quality control criteria and the association governance to ensure that go-live went ahead in a safe and sustainable manner.

–          The Trust needed stringent programme management with programme and project managers of the highest quality, to ensure that effective governance and project planning procedures were followed.

–          The go-live decision and assurances needed to pass strict criteria with sufficient evidence to provide assurance to the board that all necessary activities were completed prior to go-live.

The implementation in both the wards and the Emergency Department (ED) went well. Staff in ED were well engaged in the project and as a result were fully aware of the changes to their business processes at go live. There were some minor system issues initially but these were resolved quickly and ED was fully operational with Cerner Millennium soon after go live. One of the underlying factors in the success of the deployment to ED was that there was no data migration required as the historical data remains in the old system.

The launch in the wards went as expected; the functionality was tested well and the data was loaded manually, although there now appear to be issues with staff engaging and using the system as intended.

The majority of problems encountered at go live related to the theatre and outpatient clinic builds.

Outpatients had the most disruption immediately after go live. The Trust’s back office team had not finished building the outpatient clinics in Cerner Millennium, so the new and old systems did not mirror each other and data could not successfully migrate. Changes continued to be made to clinics in the old PAS systems, and these were not all reflected in Cerner Millennium.

Ad hoc clinics were used in the old PAS system to allow overbooking to maximise activity. These were not separated from real clinics at go live and migrated to Cerner Millennium as real clinics. The ad hoc clinics in PAS had deliberately abnormal timings so they could be excluded from time-based reports, for example 12:30am and 5:30am. The system generated letters for these ad hoc out- of-hours clinics, and many were sent to patients.

In the old system, clinics for a number of consultants could be pooled to facilitate patients seeing the next available consultant.  All clinics in Cerner Millennium are specific to a consultant and this caused significant confusion to administration staff using the new system.

PAS [the legacy patient administration system] treats “weeks” differently to Cerner Millennium. On migration, weeks were misaligned and the dates for clinics and theatres was incorrect. This created huge confusion as patient notes did not agree with Cerner Millennium , despite exhaustive work before go live to ensure that all patient notes were ready for the clinics that should have been on the system.  This also affected information in letters, with patients advised to attend their appointment on the wrong date.

There was a further issue in theatres relating to theatre procedure codes. The Trust did not map the old procedure codes to the new to ensure that all the required procedures would be available in Cerner Millennium for the data to migrate successfully. The Trust identified this issue soon after go live and has run a parallel manual process to ensure patients received the correct procedures.

The training provided to staff by the Trust did not equip them to be able to use Cerner Millennium at go live. The training environment did not mirror the system the Trust implemented as certain elements of the system were not complete when the training domain was created. Theatre staff and outpatient appointments could not train on a system with theatre schedules and outpatient clinics built in.

The Trust is now beginning to move out of the crisis and return to normal operations.

Lack of effective quality controls

There was insufficient rigour over the controls criteria and sign off of the gateway reviews.

There was inadequate operational control over the go live process, such as clinic freeze and updates pre-, during, and post go-live. Evidence from the interviews suggests that:

• There was little challenge to confirm that the gateway criteria had in fact been met.
• There was no evidence presented to the Cerner Programme Board or the Trust Board to demonstrate that the gateway criteria had been met.
• There was not enough focus on or monitoring of risks and issues and their impact on go live.
• The cleansing of old and out-of-date data from the legacy PAS systems was inadequate; as a result, erroneous data became live data in the Cerner system.
• Data Migration issues were not all resolved and their impact on go live was not considered.
• The outpatient and theatre builds were neither complete nor accurate, and there were no controls which could have detected this before go live.
• There were inadequate controls over clinic freeze and clinic changes prior to go live.

Lack of effective programme planning

Programme plans were not rigorously updated as the programme progressed and planning around training, testing and data migration and build was not robust. The Trust failed to recognise this programme as a change programme and did not effectively manage the engagement and feedback from their stakeholders. Evidence from the interviews suggests that:

• The Trust did not factor contingency into its programme plan to account for changes to the go live date.
• The Cerner Programme Management Office was not effective because of inadequate resource and programme tools.
• The Trust had a lack of sufficiently skilled resources for a project on this scale.
• The Trust’s operational staff were not fully engaged in the Cerner project.
• The Cerner project was treated as an IT project and not a business change programme.
• The training was inadequate and did not provide users with the skills they needed to be able to use the system at go live.
• The testing focused on the functionality of the system and not end-user testing of the outpatient and theatre builds.
• There was no end-user testing of the final outpatient clinic and theatre builds prior to go live.
• There was lack of understanding of roles within the wider programme team.
• External parties offered NBT help and advice. They felt that the advice was not taken and the help was refused.

Lack of effective programme governance

Programme governance processes were not reviewed and updated regularly to ensure that they were adequate and there was inappropriate accountability for key decision making. During the implementation, the Trust established new overarching change management arrangements for the Building our Future programme. Evidence from the interviews suggests that:

• The Cerner Project team failed to comply with the Trust’s Building our Future governance processes
• The information presented to the Cerner Programme Board and the Trust board by the Cerner Project team was inadequate for them to make informed decisions;
• The Cerner Programme Board was not effective; and
• Significant issues relating to the theatre and outpatient clinic build were not escalated to the Cerner Programme Board or the Trust board.

PwC’s Conclusions

For a programme of this scale and complexity, the management arrangements were not sufficiently extensive or robust. There were many issues with the software and data migration, the training of users and operational go live planning. The Trust Board and the Cerner Programme Board did not plan to have, and did not receive, independent assurance that the state of the programme supported a decision to go-live.

Complex IT implementations are never without risks and issues that need to be managed, even at the point of go live. The scale of the issues in this implementation was not properly understood by those with responsibility, and as a result they were not in a position to make sound decisions.

Many of the problems are associated with poor data and process migration. Staff found that a significant proportion of migrated data was incorrect in the new system, and this had rapid and substantial operational impact which has taken a considerable time to rectify with manual processes. Staff needed to be more directly involved in migration and process testing.

The implementation was manifestly a complex change programme. But IT took the lead, and there was no intelligent customer with sufficient distance from IT to ensure products and progress were properly challenged.

There were not dissimilar Cerner implementations within the Greenfield running a few months before NBT implementation. Similar difficulties were experienced there, but they were more successfully addressed.”

PwC recommends that:

–  the Trust “stop and take stock”. It says  “The Trust needs to take stock of its position and develop a coherent and detailed plan for the remainder of the recovery stage. The Trust then needs to ensure that effective cross programme planning and governance arrangements are enforced for all current projects, especially those under the Building Our Future programme.”

PwC also recommends that the Trust carry out a:

–  Governance review

– Capability/capacity review

– Cross programme plan review

– Operational assessment

– Review of process and controls

– Review of information requirement

– Technical resilience/infrastructure review

– Review of access controls

Comment:

To me the PwC report throws up at least six points:

1) Are NPfIT go-lives more political than pragmatic?

In the 1990s Barclays Bank went live with new systems for all its branches. During the night (I was invited to watch the go-live at head office) the most striking element was a check list that asked questions on progress so far. The answers determined whether the go-live would happen. The check-list was completed repeatedly – seemingly endlessly – during the night.

Many  different types of mishaps could have stopped the go-live.  None did.  Go-lives of Cerner Millennium are different. They seem unstoppable, whatever the circumstances, whatever the problems.  There was nothing political about the Barclays go-live. But NPfIT go-lives are intensely political.

Would North Bristol’s board have accepted with equanimity a last-minute cancellation, especially after go-lives had been postponed at least twice before?

2)  Are NHS boards too focused on “good news” to oversee an NPfIT go live?

North Bristol NHS Trust deserves praise for publishing the PwC report.  But it’s not the whole story.  The report says little about any potentially serious impact on patients. Also it mentions (almost in passing) that the Trust board discussed in November 2011 the readiness of Cerner Millennium to go live. That discussion was probably positive because Millennium went live a month later. But there is no mention of that discussion in the Trust’s board papers for November 2011.

Why did the Trust discuss its readiness to go live in secret? And why did it keep secret its November 2011 report on its readiness to go live?

If North Bristol, like so many NHS trusts, is congenitally beset with a good news culture at board level, can the full truth ever be properly discussed?

3) Isn’t it time Cerner lessons were learnt?

After seven years of Cerner implementations in the NHS, several of them notorious failures, isn’t it time Trusts learnt the lessons?

4)  What’s the current position?

PwC’s report is succinct and professional. It’s also diplomatically-worded. There is little in the report that points to how the Trust is coping with the operational difficulties. Indeed it suggests the Trust is returning to normal. “The Trust is now beginning to move out of the crisis and return to normal operations,” says the PwC report. But that is, in essence, what the Trust has been saying publicly since January 2012.  PwC says nothing about whether the safety of patients has been jeopardized by the go-live.

5) Where were the Trust’s Audit Committee – and internal auditors?

Every NHS Trust has an audit committee and internal auditors to warn about things that are going wrong, or may go wrong. It appears that they were out to lunch when it came to North Bristol’s Cerner Millennium project and its consequences.  The Audit Committee seems hardly to have mentioned the project. Should North Bristol’s board hold the Audit Committee and internal auditors to account?

6) Is the Trust board to blame?

Perhaps rightly PwC does not seek to apportion blame. But did the Trust board ask the right questions often enough?  The tacit criticism in the PwC report is of the IT department and layers of management below board level. But is that criticism misdirected? If the board’s culture of encouraging good news – of “bring me solutions not problems” –  has not changed, perhaps little or nothing will have been learned from North Bristol’s IT-related disaster.

PWC report Independent review of Cerner Millennium implementation North Bristol NHS Trust.

Lessons from Nuffield Orthopaedic’s Cerner Millennium implementation in 2005.

North Bristol apologises over Cerner go-live.

New hospital system caused chaos.

MP asks why two Cerner systems cost vastly different prices.