Category Archives: LPfIT

When “life and death” NHS IT goes down

By Tony Collins

Almost unnoticed outside the NHS an email was circulated by health officials last weekend about a national “severity 1” incident involving the Electronic Prescription Service, running on BT’s data Spine .

“The EPS [electronic prescriptions service] database is currently experiencing severe degradation of performance. … BT engineers [are] currently investigating with the database application support team,” said the email.

A severity 1 or 2 incident, which involves a temporary loss of, or disruption to, the Spine or other national NHS system,  is not unusual, according to a succession of emails forwarded to Campaign4Change.

The Department of Health defines a severity 1 incident as a  failure that has the potential to:

— have a significant adverse impact on the provision of the service to a large number of users; or

— have a significant adverse impact on the delivery of patient care to a large number of patients; or

— cause significant financial loss and/or disruption to NHS Connecting for Health [now the Health and Social Care Information Centre], or the NHS; or

— result in any material loss or corruption of health data, or in the provision of incorrect data to an end user.

The Health and Social Care Information Centre, which manages BT’s Spine and other former NPfIT contracts, reports that the spine availability is 99.9% or 100%. But the HSCIC’s emails tell a story of service outage or disruption that is almost routine.

If the spine and other national services  are really available 99.9% of the time, is that good enough for the NHS, especially when ministers and officials are increasingly expecting clinicians and nurses to depend on electronic patient records and electronic prescriptions?  In short, are national NHS IT systems up to the job?

NHS staff access the spine tens of millions of times every month, often to trace patients before accessing their electronic records.  The spine is pivotal to the use of patient records held on Rio and Cerner Millennium systems in London. It is critical to the operation of Choose and Book, the Summary Care Record, Electronic Prescription Service pharmacy systems, GP2GP, iPM/Lorenzo, and the Personal Demographics Service.

According to a Department of Health letter sent to the Public Accounts Committee, payments to BT for the Spine totalled £1.08bn by March 2013.

BT says on its website that its 10-year NHS Spine contract involves developing systems and software to support more than 899,000 registered NHS users. The HSCIC says the Spine is used and supported 24 hours a day, 365 days a day.

“There is a huge amount of industrial-strength robustness, availability, disaster recovery, that you cannot get someplace else,” said a BT executive when he appeared before MPs in May 2011.

Life and death  

Sir David Nicholson spoke of the importance of the spine and other national NHS systems at a hearing of the Public Accounts Committee in 2011. He said they were

“providing services that literally mean life and death to patients today … So the Spine, and all those things, provides really, really important services for our patients…”

When Croydon Health Services NHS Trust went live with a Cerner Millennium patient records system at the end of September a “significant network downtime” – of BT’s N3 network – had an effect on patients.

A trust board paper, dated 25 November 2013 says:

“CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT …”

Below are some of the emails passed to Campaign4Change in the past four months. Written by the Health and Social Care Information, the emails alert NHS users to outages or disruption to GP or national NHS IT systems.

Some HSCIC messages of disruption to service

October 2013

Severity 2
HSCIC
National
CQRS has not received a number of participation status messages.
Also affecting: GPES
USER IMPACT:
CQRS Users are not able to manually submit specific information, this will impact the users’ business process for entry of achievement data.
ACTION BEING TAKEN:
Following a configuration change by the GPES Business Unit a specific code has now been added to the GPET-Q Database. We are currently awaiting confirmation that the addition of the relevant code has been successful. Discussions are taking place regarding the re-submission of status messages. HSCIC conference calls are on-going.

[A severity 2 service failure is a failure [that] has the potential to:

– have a significant adverse impact on the provision of the service to a small or moderate number of service users; or

– have a moderate adverse impact on the delivery of patient care to a significant number of service users; or

– have a significant adverse impact on the delivery of patient care to a small or moderate number of patients; or

– have a moderate adverse impact on the delivery of patient care to a high number of patients; or

– cause a financial loss and/or disruption … which is more than trivial but less severe than the significant financial loss described in the definition of a Severity 1 service failure.]

**

Severity 2
HSCIC
BT Spine
National
Intermittent performance issues on TSPINE.
T-Spine
RESOLUTION:
BT Spine have confirmed that the incident has been resolved and users are able to perform routine business processes without delays.

November 2013

Severity 1
BT Spine
HSCIC
National
Users are unable to log into PDS.
USER IMPACT:
All sites are currently unable to access PDS, this is causing a delay to normal services.
ACTION BEING TAKEN:
BT Spine are working to restore service.

**

Severity 2
BT Spine
HSCIC
National EPS users.
Slow performance on reliable and unreliable messages for EPS.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
BT investigating.

**

Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
ACTION BEING TAKEN:
BT moved the database to an alternate node following application server restarts. This temporarily restored normal message response times however performance has started to degrade again. BT Investigation continues.

**
Severity 1
Atos
HSCIC
National
Multiple users were unable to log in to the Choose & Book application.
ATOS made some network configuration changes overnight 19th/20th November which restored service. After a period of monitoring throughout the day yesterday the service has remained stable and at expected levels. Further activities and investigation will be carried out by several resolver teams which will be scheduled through change management.

**
Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
No further issues of slow response times with EPS messaging have occurred today. BT Spine to continue root cause investigation.

**

Severity 2
Cegedim RX
HSCIC
National
Cegedim RX – Users are experiencing slow performance in EPS 1 and EPS 2.
USER IMPACT:
Users are experiencing slow performance and delays to routine business processes when using EPS 1 and EPS 2.
ACTION BEING TAKEN:
Following a restart of application services, traffic has improved for all new EPS messages. However there is a backlog of EPS messages which may cause delays to routine business processes. Cegedim RX to continue to investigate.

**

December 2012

Severity 1
BT Spine
HSCIC
National
Performance issues have been detected with the transaction messaging system (TMS).
Also affecting: Choose and Book, GP2GP
USER IMPACT:
This may cause delays to routine business processes. This may have an effect on all Spine related systems. This includes PDS, Choose and Book, PSIS, SCR, ACF Services.
ACTION BEING TAKEN:
This has been resolved but BT are currently monitoring performance. Further investigation is required by BT into the root cause.

**

Severity 2
GDIT – CQRS
HSCIC
National
DTS has not processed a CQRS payment file.
CQRS
Also affecting: GPES
USER IMPACT:
This is causing delays to routine business processes.
ACTION BEING TAKEN:
GDIT are currently developing a fix which will be rolled out tomorrow evening, pending successful testing.

January 2014

Severity 1
BT Spine
HSCIC
National
TMS reliable messaging unavailable.
USER IMPACT:
TMS reliable messaging unavailable and users having to implement manual workarounds.
ACTION BEING TAKEN:
Issues experienced due to a planned change overrunning, BT Spine continue to implement the transition activity in order to restore service.

**

Severity 2
BT Spine
HSCIC
National
Users have experienced intermittent issues with the creation and cancellation of smartcards in CMS [Card Management Service for managing smartcards].
CMS
USER IMPACT:
This is intermittently causing delays to routine business processes as some users have been unable to create, cancel, cut or print cards in CMS.
ACTION BEING TAKEN
Users may experience issues with the creation and cancellation of cards in CMS. BT have identified a fix for the issue which is currently undergoing testing prior to deployment into the live environment.

**

Severity 2
BT Spine
HSCIC
National
The maternity browser was unavailable within NN4B.
RESOLUTION:
BT identified a problematic server which was recycled to restore system functionality.

**

Spine scheduled outage for essential maintenance activity.

During critical work to migrate to a new storage solution on Spine an issue was experienced on the Transaction Messaging Service (TMS) in September of this year. The issue resulted in BT failing over the TMS database from its usual site on Live B to Live A to restore service. The failover was completed well within the Service Level Agreement and no detrimental long term impacts to the service were incurred.

On the 15th January 2014 between approximately 22:00-23:30, HSCIC, in conjunction with BT, are planning to relocate the TMS database back to Live B, this is for several critical reasons:

  1. The issues experienced, which prompted the failover, are fully resolved and will not be experienced again as the storage migration work is now complete.
  2. The Spine service is designed to operate with all databases running on Live B so this work supports the optimum configuration for the service.
  3. Most critically the transition for all data on Spine to Spine2 has been designed to operate from a standby site with no live databases on it. Therefore to support the Spine2 transition this work is absolutely essential.

In order to facilitate a safe relocation of the database a 1.5 hour outage is required to TMS. The impact of this to Spine is significant and results in effectively an outage for Spine and its interfaces to connecting systems for that period. The time and date is aimed at the lowest times of utilisation for Spine, to minimise impact to end users, as well as not impacting critical batch processing and Choose & Book slot polls.

 

Date & Time

Change Start Change Finish Services Affected Outage Duration
15/01/2013 22:00 15/01/2013 23:30 Transaction Messaging Service (TMS) 1.5 hours
Service  Impact Description
Choose and Book The Choose and Book service will be available but functionality will be limited until the TMS database has switched over.Users of the web application will experience limited retrievals during the outage window.The system will not be able to create shared-secret for patients who have not been referred via Choose and Book before.Service Providers will be unable to:

  • Perform clinic re-structures and re-arrange appointments for patients for directly bookable services
  • Send DNA messages to Choose and Book.

For directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Choose & Book systems will need to queue the messages and resend to Spine once the TMS service is enabled.

Due to the timing of the outage slot polls will not be affected.

Summary Care Record application (SCRa) The SCRa application will be available but functionality will be limited until the TMS database has switched over. Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates)
DSA The DSA application will be available but functionality will be limited until the TMS database has switched over.Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates).
Electronic Prescription Service (EPS)Pharmacy Systems Reliable messaging will be unavailable for the duration of the switchover work as the TMS service will be suspended dual site. All messages received from EPS systems will be rejected and not go into retry.EPS systems will need to queue the messages and resend to Spine once the TMS service is enabled.
EPS Batch The PPA response for any “claim” messages will not be sent to PPA/PPD. However, EPS will send those response(s) again when the retry jobs are re-activated after the switchover exercise is over. Response for any “claim” messages will not be received until after the switchover. Retry jobs will resend the responses once the TMS service is enabled.
Existing Service Providers (ESPs) There will be varying impacts depending on the product, release version and Spine compliant modules of the solution.ESP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP2GP GP2GP will be unavailable until the TMS database has switched over.GP2GP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP Extraction Service (GPES) GPES functionality will be unavailable until the TMS database has switched over.Messages will be queued on Spine and processed once the TMS service is restored.
GP Systems Functionality for Choose & Book, EPS and GP2GP, SCR will be limited until the TMS database has switched over.For Choose & Book directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Systems will need to queue the messages and resend to Spine once the TMS service is enabled.

iPM/Lorenzo The real-time connection to Spine will be unavailable during the TMS outage. However both systems can be disconnected from Spine and operate without synchronised PDS data.iPM/Lorenzo will need to queue the messages and resend to Spine once the TMS service is enabled.
Millennium An outage of PDS reliable messaging will impact Millennium users.Users will be unable to:

  • trace patients
  • register new patients on PDS
  • book or reschedule appointments

Millennium will need to queue the messages and resend to Spine once the TMS service is enabled.

NN4B Trusts will need to be aware that during the outage NHS numbers cannot be generated, new-births cannot be registered and blood-spot labels cannot be generated and should plan accordingly.All birth notifications will be queued and processed once the TMS service is enabled.
Personal Demographics Service (PDS) Simple traces can be completed on PDS data.PDS reliable messaging will be unavailable until the TMS database has switched over.
RiO Users will be unable to:

  • trace patients
  • register new patients
  • book or reschedule appointments

The RiO system will need to queue the messages and resend to Spine once the TMS service is enabled.

TMS Event Service (TES) The majority of TES functionality will be unavailable during the outage.Trusts will need to be aware EPS, Death notifications, and Patient Care Provision Notifications (change of pharmacy) will be queued and sent to the receiving systems once the TMS service is restored.Any impacted notifications will be queued and sent to the receiving systems once TMS is restored.
TMS Batch (DBS, CHRIS, ONS) DBS will be unavailable until the TMS database has switched over (DBS processing will be suspended for the duration of the exercise).As the TMS switchover will be scheduled to start at 22:00, CHRIS batch should complete before the outage starts (CHRIS batch runs at 20:00 nightly).ONS processing will start at 18:00 nightly. If it doesn’t complete before 22:00, the messages will be queued and processed once the TMS service is restored.

**

Severity 2
BT Spine
HSCIC
National
Users are unable to grant worklist items within UIM.
USER IMPACT:
This is causing delays to routine business processes as users are unable to complete their worklist items within the UIM application.
ACTION BEING TAKEN:
BT investigating.

**

Severity 1
BT Spine
HSCIC
National
The EPS database is currently experiencing severe degradation of performance.
USER IMPACT:
Delays to routine business processes.
ACTION BEING TAKEN:
BT engineers currently investigating with the database application support team.

Comment

David Nicholson is right. The NHS has become dependent on systems such as the Spine. But can doctors ever trust any aspect of the safety of patients to systems that are not available 24×7 as they need to be in a national health service?

It appears that BT and other suppliers have not been in breach of service level agreements, and the HSCIC has a good relationship with the companies.  But does the HSCIC have too great an interest in not finding fault with its suppliers or the contracts, for finding fault  could draw attention to any defects in a service for which the HSCIC is responsible?

Have national NHS IT suppliers a strong enough commercial or reputational interest  in avoiding  a disruption or loss of service, so long as they keep within their service level agreements? 

If nobody sees anything wrong with the reliability of existing national NHS IT services improvements are unlikely. Diane Vaughan’s book on the culture and organisation of NASA shows that experts in a big organisation can do everything right according to the rules  and procedures – and still have a disastrous outcome.

Top 5 posts on this site in last 12 months

Below are the top 5 most viewed posts of 2013.  Of other posts the most viewed includes “What exactly is HMRC paying Capgemini billions for?” and “Somerset County Council settles IBM dispute – who wins?“.

1) Big IT suppliers and their Whitehall “hostages

Mark Thompson is a senior lecturer in information systems at Cambridge Judge Business School, ICT futures advisor to the Cabinet Office and strategy director at consultancy Methods.

Last month he said in a Guardian comment that central government departments are “increasingly being held hostage by a handful of huge, often overseas, suppliers of customised all-or-nothing IT systems”.

Some senior officials are happy to be held captive.

“Unfortunately, hostage and hostage taker have become closely aligned in Stockholm-syndrome fashion.

“Many people in the public sector now design, procure, manage and evaluate these IT systems and ignore the exploitative nature of the relationship,” said Thompson.

The Stockholm syndrome is a psychological phenomenon in which hostages bond with their captors, sometimes to the point of defending them.

This month the Foreign and Commonwealth Office issued  a pre-tender notice for Oracle ERP systems. Worth between £250m and £750m, the framework will be open to all central government departments, arms length bodies and agencies and will replace the current “Prism” contract with Capgemini.

It’s an old-style centralised framework that, says Chris Chant, former Executive Director at the Cabinet Office who was its head of G-Cloud, will have Oracle popping champagne corks.

2) Natwest/RBS – what went wrong?

Outsourcing to India and losing IBM mainframe skills in the process? The failure of CA-7 batch scheduling software which had a knock-on effect on multiple feeder systems?

As RBS continues to try and clear the backlog from last week’s crash during a software upgrade, many in the IT industry are asking how it could have happened.

3) Another Universal Credit leader stands down

Universal Credit’s Programme Director, Hilary Reynolds, has stood down after only four months in post. The Department for Work and Pensions says she has been replaced by the interim head of Universal Credit David Pitchford.

Last month the DWP said Pitchford was temporarily leading Universal Credit following the death of Philip Langsdale at Christmas. In November 2012 the DWP confirmed that the then Programme Director for UC, Malcolm Whitehouse, was stepping down – to be replaced by Hilary Reynolds. Steve Dover,  the DWP’s Corporate Director, Universal Credit Programme Business, has also been replaced.

4) The “best implementation of Cerner Millennium yet”?

Edward Donald, the chief executive of Reading-based Royal Berkshire NHS Foundation Trust, is reported in the trust’s latest published board papers as saying that a Cerner go-live has been relatively successful.

“The Chief Executive emphasised that, despite these challenges, the ‘go-live’ at the Trust had been more successful than in other Cerner Millennium sites.”

A similar, stronger message appeared was in a separate board paper which was released under FOI.  Royal Berkshire’s EPR [electronic patient record] Executive Governance Committee minutes said:

“… the Committee noted that the Trust’s launch had been considered to be the best implementation of Cerner Millennium yet and that despite staff misgivings, the project was progressing well. This positive message should also be disseminated…”

Royal Berkshire went live in June 2012 with an implementation of Cerner outside the NPfIT.  In mid-2009, the trust signed with University of Pittsburgh Medical Centre to deliver Millennium.

Not everything has gone well – which raises questions, if this was the best Cerner implementation yet,  of what others were like.

5) Universal Credit – the ace up Duncan Smith’s sleeve?

Some people, including those in the know, suspect  Universal Credit will be a failed IT-based project, among them Francis Maude. As Cabinet Office minister Maude is ultimately responsible for the Major Projects Authority which has the job, among other things, of averting major project failures.

But Iain Duncan Smith, the DWP secretary of state, has an ace up his sleeve: the initial go-live of Universal Credit is so limited in scope that claims could be managed by hand, at least in part.

The DWP’s FAQs suggest that Universal Credit will handle, in its first phase due to start in October 2013, only new claims  – and only those from the unemployed.  Under such a light load the system is unlikely to fail, as any particularly complicated claims could managed clerically.

 

MP calls for candour after Cerner NPfIT go-live at Croydon

By Tony Collins

Richard Bacon, a long-standing member of the House of Commons’ Public Accounts Committee, has called on Croydon Health Services NHS Trust to be more open about problems it faces after deploying a Cerner Millennium patient records system at the end of September.

The installation was carried out by BT under the London Programme for IT – a branch of the NPfIT.  The Health and Social Care Information Centre, which has taken on BT and CSC contracts under the NPfIT, was the trust’s partner for the Cerner deployment.

Bacon has closely followed the NPfIT and written a chapter on it in his book, “Conundrum: Why every government gets things wrong and what we can do about it” which he co-wrote with Christopher Hope, the Telegraph’s senior political correspondent.

According to fragments of information in Croydon Health Services’ latest board papers, dated 25 November 2013, the trust has faced a series of problems after the NPfIT Cerner go-live.

They included:

–  N3 Network downtime and waiting time breaches.

 Excessive waits for patients in A&E

 Going over budget.

– Significant loss of income.

 A bid to recover Cerner costs.

– A need for HSCIC support for delays. 

-A need for extra investment in Cerner to “stabilise the operational position”

The trust has not published any specific report on the implementation’s problemsNow Bacon says it is “unacceptable for any trust not to disclose the problems it faces – and possibly patients face – after a major IT implementation such as Cerner”.

He adds:

“If these implementations go wrong they can affect the safety of patients.  We know this from some NPfIT deployments at other  trusts. For Croydon to say that board members have been kept informed of the potential risks of the Cerner implementation through the “Corporate Risk and Board Assurance Framework”  is not reassuring.

“This is putting a matter of importance in the small print. Indeed, for officials to brief board members on the potential risks, rather than actual events, is also of concern.

“Patients need to know that Croydon takes a duty of candour seriously. If the Trust cannot be open about its IT-related problems, how can we be sure it will be open about anything else to do with patient safety?”

Patient records go-live “success” – or a new NPfIT failure 

Patient records go-live “success” – or a new NPfIT failure?

By Tony Collins

John Goulston says the go-live of a new patient records system at his trust is a “success”.

He should know. He’s Chief Executive of Croydon Health Services NHS Trust. He’s also chair of the trust’s Informatics Programme Board which has taken charge of bringing Cerner Millennium to Croydon’s community health services and the local University Hospital, formerly the Mayday.

He was formerly Programme Director of the London Programme for IT at NHS London – a branch of the NPfIT.

In a report two weeks ago Goulston said the trust deployed the “largest number of clinical applications in a single implementation in the NHS”. Croydon went live with Cerner Millennium on 30 September and 1 October 2013.

Said Goulston in his report:

“Administrative functions do not engage clinicians; providing them with a suite of clinical functionality has been justified as each weekday approx. 1,000 staff are logged on and using the system. CHS [Croydon Health Services] has in Phase 1 deployed, in addition to patient administration, the largest number of clinical applications in a single implementation in the NHS England.”

BT helped install Millennium at Croydon under the National Programme for IT.  The trust’s spokesman says the Department of Health provided central funding, and the trust paid for implementation “overheads”.  The Health and Social Care Information Centre was the trust’s partner for the go-live.

The Centre is the successor for Connecting for Health. It has taken on CfH’s officials who continue to help run the NPfIT contracts with BT and  CSC.

Goulston said that Cerner and BT have paid tribute to the trust which installed Millennium in A&E, outpatients, secretarial support and cancer services, and elsewhere.

“Our partners Cerner, BT and Ideal have commented that the Trust has undertaken one of the most efficient roll-outs of the system they have worked on, with more users adopting the system more quickly and efficiently than other trusts … the success we have achieved to date is the result of the efforts of every single system user and all staff members,” said Goulston.

Best Cerner implementation yet?

Optimistic remarks about their launch of Cerner Millennium were also made in 2012 by executives at the Royal Berkshire NHS Foundation Trust.  Their optimism proved ill-judged.

Of the Millennium go-live at Royal Berkshire, trust executives said that it “had been considered to be the best implementation of Cerner Millennium yet and that despite staff misgivings, the project was progressing well”.   This positive message should be disseminated, they said.

Months later they told the Reading Chronicle of patient safety issues and a financial crisis arising from the Millennium implementation.

A Royal Berkshire governors Rebecca Corre was quoted as saying: “There is a patient safety issue when staff write down observations and then there is an hour before they can get it onto the computer. If it is an experienced nurse, they may pick up a problem, but others may not.”

Ed Donald, Chief Executive of Royal Berkshire was quoted as saying:

“Unfortunately, implementing the EPR [electronic patient record] system has at times been a difficult process and we acknowledge that we did not fully appreciate the challenges and resources required in a number of areas.”

Are executives and managers at Croydon Health Services NHS Trust  now similarly afflicted with an unjustified optimism about the success of their Cerner go-live?  

Past consequences of NPfIT go-lives hidden?

The Department of Health has claimed benefits for the NPfIT of £3.7bn to March 2012 but there have been trust-wide failures: thousands of patients have had their appointments, care or treatment delayed by difficulties arising from past implementations of patient record systems under the NPfIT.  For thousands of patients waiting time standards have been exceeded or “breached” because of disruption arising from troubled go-lives.

In nearly every case trusts made it difficult for the facts to come out publicly. Vague or unexplained fragments of information about the consequences of the NPfIT implementation appeared  in different board papers over several months. The facts only emerged after a journalistic investigation that required scrutiny of many board papers and follow-up questions to the trust’s press office.

So Campaign4Change investigated Croydon Health’s implementation of Cerner Millennium to see if the Francis report’s call for a “duty of candour” over mistakes and problems in the NHS have made any difference to the traditional fragmentation of facts after NPfIT go-lives of patient record systems.

The Francis report called for “openness, transparency and candour“.  Trusts were told not to hide sub-standard practices under the carpet. The health secretary Jeremy Hunt said it can be “disastrous” when bad news does not emerge quickly and the public are kept in the dark about poor care.

To my questions about the Cerner Millennium implementation Croydon trust’s spokesman always responded promptly and tried to be helpful. But it appears that trust executives have given him limited information about consequences of the go-live, and have preferred to indulge the “good news” NHS culture that Jeremy Hunt warned about.

On being asked what problems the trust has faced since the go-live the spokesman gave various answers that made no mention of the problems.

“All of our staff received training on the system, and we are continuing to offer our teams support as it is embedded.”

What of the problems arising from the implementation, and has the board been fully informed?

“Millennium has featured regularly on the Corporate Risk Register presented to each Part 1 Board meeting.   In addition, implementation has received detailed confidential consideration at Part 2 of Board meetings, (which is why you won’t find it in our public board papers).”

Given Francis’s call for duty of candour,  should the trust be more open about its problems?

“The initial roll out for CRS Millennium was introduced over three days at the Trust, with a phased approach.  We did this to ensure the system was working in each department, before introducing it in another area.

“We are monitoring waiting time performance and records management so we can identify any issues if they emerge. The system is still being introduced in some services and when this is completed we will be able to assess the overall programme,” said the spokesman.

Does Croydon’s unwillingness to give in its statements to me any details of problems indicate that the culture of a lack of transparency in the NHS will be hard to change, no matter how many times Jeremy Hunt talks about the need for candour when things go wrong?

The spokesman:

“I’d like to be clear about the Trust’s approach:

  • The Trust board has been cited on the roll out of CRS Millennium and any potential risks throughout the process.  As I previously noted, the board received an update in September.  The board meeting, which will take place on Monday of next week, will receive a further update from the Chief Executive.  The papers from this meeting will be published on our website and the meeting takes place in public;
  • A meeting chaired by the Chief Operating Officer has reviewed any operational matters arising on a daily basis.  This is an internal meeting for clinicians and managers which has informed the implementation process;
  • Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“These actions would suggest that the Trust has been transparent in its approach.  You are welcome to review the board papers when they are published.”

Serious problems now emerge

Croydon did indeed publish its board papers on 25 November 2013 – which is to its credit because not all NHS trusts publish timely board papers.

But it’s mostly in the small print of various board papers that details emerge of Millennium-related problems. The shortcomings are mentioned as individual items rather than in a single, detailed Cerner Millennium deployment report.  This leaves one to question whether trust directors have an overview of the seriousness of the difficulties arising from its implementation of a new patient records system.

These are some excerpts from deep inside Croydon’s latest board papers:

Breaches in waiting time standards

– “CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT providing CHS with the required scale of N3 access (>600 concurrent users and >1,600 users on any day – which is the largest network usage of any trust in England).”

– • “Hospital Based Pathways: The deployment of CRS Millennium was a particular challenge in the month across the multiple service areas within the Directorate of A&E, Surgery and Maternity.

• “Cancer & Core Functions: With the implementation of CRS Millennium, the open pathways part of RTT [referral to treatment – patient waiting times) may fail the standard – validation will be completed after the narrative for this report… “

Excessive waits in A&E

– “The main drivers adversely affecting the performance in the month [October 2013) for A&E were the deployment of CRS Millennium and the commencement of winter pressures due to the seasonality change.  A&E  4-Hour Total Time in Department Target: 95.00%. Actual: 91.57%.”

Over budget

“The Trust position as at October is an adverse variance of £4.1m. This is a significant deterioration on the Month 6 position. The movement is mainly due to a significant reduction in income mainly as a result of operating issues caused by the Cerner deployment (£0.9m)…  Actual £14.8 (£14.8)m; Budget £10.7m; Variance £4.1m.”

“Cerner Millennium: Plan YTD [year-to-date] £245,000; Actual YTD  £621,000;

Significant loss in income

“… A new patient administration system was deployed in the Trust on the 30th September and 1st October (Cerner Millennium). The deployment has resulted in significant loss in income in September and October £ 1.1m. Trust performance on Activity Planning Assumptions and Key Performance Indicators is substantially worse than plan …”

Extra costs

“Medical £412k and admin £148k agency levels continue to be high due to cover for vacancies, annual leave, sickness and release of staff for Cerner training. The Trust has also incurred additional costs associated with the Cerner deployment (£600k) including overtime payments to administration staff and training costs.”

Bid to recover Cerner costs?

“… The Trust is currently forecasting a deficit position of £17.8m, which is £3.3m off the plan submitted to the NHS Trust Development Authority. This is a £3m movement from the month 6 forecast and is as a result of operational issues caused by the Cerner deployment. The current projected impact is an additional costs £1.7m and a loss in activity £1.1m . An application is to be made to recover the additional cost/losses relating to the Cerner deployment [of £2.9m] …”

HSCIC support for delays

“Cerner Millennium – Revised implementation date to Sept 2013 (achieved) ,with resultant additional costs including additional PC requirements of £146k, specialist support services £300k, procurement costs £91k, data cleansing costs £200k.

“Health& Social Care Information Centre (HSCIC) has confirmed support for the delayed implementation will be provided, accounting treatment of support to be confirmed with Department of Health.”

More money to stabilise operational position?

“As a result of operational issues caused by the Cerner deployment , Income is significantly reduced in October. The forecast assumes that the Trust will resume normal operating levels from November and that an element of the income lost will be will be recovered in the latter part of the year. A business case is being submitted to the Trust Board for additional investment in Cerner to stabilise the operational position.

“If there are further operational issues due to the Cerner deployment then this will significantly impact on the year end forecast…”

Over-optimism?

Principal risk -reporting output from Cerner is not accurate or timely. Officer in charge: CEO. Before go-live risk scores: June 2013 – 16; July – 16; Aug  – 10; Sept – 10. After go-live risk score (for Oct): 20 [high risk of likelihood and consequences]

Principal risk – operational readiness following the implementation of Cerner. Officer in charge: COO.  Before go-live risk score 15. Post go-live: 20. Risk rating before go-live – Green. After go-live – Red.

Red risks

Corporate Risk Assurance Framework

Nine risks are reported as Red [two of which relate directly to Millennium]:

“… Reporting output from Cerner is not accurate or timely. Data migration was successful. However reliance on external provider as internal knowledge has not yet been fully gained. A data quality dashboard with exception reporting is in place.

“… Operational readiness following the implementation of Cerner CRS Millennium impact conveyed to Trust Development Authority e.g. ED [Emergency Department] reporting and cost overruns

Risk scores

– Failure of CRS millennium to deliver anticipated benefits – 12. Officer in charge: CEO

– Reporting output from Cerner is not accurate or timely – 20. Officer in charge: CEO

– Operational readiness following the implementation of Cerner – 20. Officer in charge: COO

Croydon’s trust’s response to problems

Said John Goulston, Croydon’s CEO, in his latest [November 2013] report to the board of directors:

“The issues being encountered now with CRS Millennium are not due to any lack of integration testing with legacy applications or testing of workflow. They can be attributed to changing from a 25 year old Patient Administration System (Patient Centre) which did not require working in real time, was simple and intuitive to use, easily configurable and flexible to our needs.

“CRS Millennium’s patient administration functions are almost the complete opposite and the language used is new for our staff i.e. conversations, encounters etc. For our staff it has been a big ask for them to step into and up to such a complex application.”

He added: “The benefits of the new system are that each patient will have a single accurate electronic record that can be viewed and kept up-to-date by hospital and community clinical staff. This will eventually mean less time searching for patient notes, missing documentation and duplicating patient information…

“As with any massive change, there are still some challenges to tackle in making the system work effectively for every single user, in a diverse and complex organisation.

“However the success we have achieved to date is the result of the efforts of every single system user and all staff members. I would like to thank all our staff for their hard work in getting the Trust to this important stage.”

The trust spokesman gave me this statement on the problems:

“The Trust board has been given regular reports on the roll out of CRS Millennium and any potential risks throughout the process, not least through its regular reviews of the Corporate Risk and Board Assurance Frameworks.  As I previously noted, the board received a specific update in September.

“As you already know, November’s board meeting received a further update from the Chief Executive.  The papers from this meeting were published and the meeting takes place in public;  Those attending are invited to put forward questions.

“A meeting chaired by the Chief Operating Officer continues to review operational matters.  This is an internal meeting for clinicians and managers which has informed the implementation process;

“Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“As you highlight from the board report, Cerner & BT noted that ‘the Trust has undertaken one of the most efficient roll-outs of the system they have worked on’   The papers also note some operational challenges as the system was rolled out.  These have been addressed as part of the daily meetings I reference above – these are mainly concerned with users familiarising themselves with the system and have been addressed through the support and training staff received.

“In terms of the costs, the introduction of CRS Millennium has been supported by central funding from the Department of Health with the Trust paying the implementation overheads.   These costs are a matter of public record and the Trust publishes annual Accounts as part of its Annual Report.”

Comment

When you go into hospital it’s reassuring to know the directors will be well informed and open about problems that could affect you.

The approach of Croydon Health Services NHS Trust to openness about its problems is not reassuring. It is no better or worse than other trusts that have implemented Cerner’s Millennium. In fact the timely publication of its board papers means it is more open than some.

But it should not require a time-consuming journalistic investigation to establish the consequences for patients of an NPfIT go-live. It has required just such an investigation after the go-live of Millennium at Croydon.

Board directors will not have the time to dig for, and piece together, information about internal problems that could delay patient appointments, treatment and care. They need the unpalatable facts in one place. Croydon Health Services has failed to make it easy for patients or board directors to see what has gone wrong.

NPfIT deployments at other trusts have led, cumulatively, to thousands of patients having appointments that were disrupted, or who had to wait longer to be seen than necessary, or whose records were not available, or who were seen with another patient’s records.

In shying away from telling the whole truth trusts take their cue from the top: the Department of Health has always made it hard to establish facts about anything to do with the NPfIT.  Said the Public Accounts Committee in its report The National Programme for IT in the NHS: an update on the delivery of detailed care records systems in July 2011:

 “It is unacceptable that the Department [of Health] has neglected its duty to provide timely and reliable information to make possible Parliament’s scrutiny of this project.

“Basic information provided by the Department to the National Audit Office was late, inconsistent and contradictory.”

Unanswered questions

Croydon has questions to answer, such as how many breaches of waiting time standards it has had, and may still be having, due to problems arising from the go-live. Other unanswered questions:

– What does a “a large number of breaches” in the Emergency Department mean? Have each the patients affected been told?

– Why are the risks related to the implementation much higher after go-live than before, given that the trust has had years to prepare for the go-live, and the many lessons it could have learned from other trusts?

– Exactly what problems are still affecting patients?

In a post-Francis NHS, Jeremy Hunt has demanded openness about mistakes and problems. There is an agreed need for change – but how can Hunt change an NHS culture – indeed a public sector culture – in which senior executives, in troubled IT implementations, will always emphasise the good news over the bad, perhaps hoping the bad will always remain hidden?

Croydon trust plans “high-risk” Cerner go-live in secret

By Tony Collins

NHS trusts have gone live with Cerner Millennium with mixed success. Eight trust implementations went seriously awry. A list is at the end of this post.

The flawed go-lives have meant that hospital managers have lost track, cumulatively, of thousands of patients and found that treatments,  including those for cancer, were delayed or care pathways interrupted. At times medical notes have not been available, or clinicians have been given the wrong notes.

In July 2008 E-Health Insider reported that the deployment of a national programme care records system at Milton Keynes Hospital NHS Foundation Trust “developed into an untenable situation which resulted in near melt down of the organisation”.

One of the lessons from the problematic go-lives is that trusts, when they have reported on the difficulties later, have said they underestimated the risks.

In particular they regarded the patient records system as mainly administrative with no risks to the health and care of patients.  Yet it is possible for incidents arising from flawed patient record implementations to be judged “clinical”.

Is Croydon Health Services NHS Trust, which includes the former Mayday hospital, about to repeat this same mistake – of regarding the risks as non-clinical?

The Trust is due to go live this weekend with the start of one of the largest patient record go-lives in the UK.  The deployment is being run under the NPfIT, which, in the capital, is now called the London Programme for IT. The plans are to install Millennium at Croydon University Hospital (formerly Mayday) and at some community sites.

The Trust says its preparations are designed to ensure a safe and effective deployment that will replace systems that are more than 20 years old and lack “many of the functions one expects in the modern healthcare digital age”.

But the signs are, from the trust’s board papers, that the risks of a flawed implementation are being seen largely as financial and administrative. For patients it appears that the worst that is envisaged is a “poor experience”.  

Have the risks to patients been properly flagged to the board’s directors?

I looked through Croydon’s most recently published board papers to see how well the trust’s directors have been kept informed of the risks of the go-live and could see almost no mention of Cerner – except a risk that income to the Trust could be affected if it is unable to produce timely reports. There is no specific mention of risks to patients.

Given the history of failed implementations of Cerner Millennium under the NPfIT, I asked  Croydon Health NHS Trust what directors have been told about the risks.

This was the trust’s reply:

“CRS Millennium has featured regularly on the Corporate Risk Register presented to each Part 1 Board meeting.

“In addition, implementation has received detailed confidential consideration at Part 2 of Board meetings, (which is why you won’t find it in our public board papers).”

I then put it to the trust that I was not sure why, in a new era of openness and transparency in the NHS,  that board discussions should be in private on a matter that could affect large numbers of patients to judge from past implementations at other trusts. The spokesperson – the interim stakeholder relations manager – made no further comment.

Richard Granger,  when head of the NPfIT, was quoted as saying he was ashamed of some of the Cerner deployments . Granger quit the programme in January 2008 – and since then several more Cerner deployments at trusts have gone badly wrong, at North Bristol NHS Trust, for example.

After the NPfIT go-live of Cerner at the Nuffield Orthopaedic Centre, the National Audit Office, at the request of a member of the Public Accounts Committee Conservative MP Richard Bacon, investigated and produced a public report on the lessons learned.

The Nuffield go-live was in December 2005. Since then managers at all trusts that have gone live with Cerner have claimed they have learned those lessons.

Comment:

It’s true that Croydon Trust’s risk register has shown “red” for the Cerner implementation. It is indeed deemed “high risk”. But how many of its board directors will understand the nature and substance of the risks to patients from a graphic?

It’s easy for the trust to say that board directors have been informed through confidential discussions but how would anyone outside the trust be able to judge whether that’s true?

Trust boards can never be held fully accountable after a patient record go-live goes wrong. They hold their own investigations or sometimes commission inquiries by consultants known to them, and the final reports usually indicate that no patients have been harmed.  Even after a trust has lost track of thousands of patient appointments for more than a year, it has later reported that in the end all was well.

As trust boards are not voted in and out by the public they have a special duty to ensure they are fully informed on things that can go wrong. Is Croydon Health’s board fully informed on the risks of the Cerner go-live? I doubt it.  

Some flawed NHS patient record go-lives:

Barts and The London

Royal Free Hampstead

Weston Area Health Trust

Milton Keynes Hospital NHS Trust

Worthing and Southlands

Barnet and Chase Farm Hospitals NHS Trust

Nuffield Orthopaedic

North Bristol.

St George’s Healthcare NHS Trust

Lorenzo:

University Hospitals of Morecambe Bay NHS Foundation Trust

Birmingham Women’s Foundation Trust

NHS Bury