Category Archives: BT

50,000 on waiting list and cancer test delays after NPfIT go-live

By Tony Collins

Croydon hospitals have built up a waiting list of 50,000 patients since a Cerner electronic patient record system go-live last October, according the trust’s latest board papers.

And, since the go-live, more than 2,200 patients have waited at least 6 weeks for diagnostic tests, of which 160 have been identified as “urgent suspected cancer and urgent patients”.  This backlog may take until the end of August to clear, say the board papers of the Croydon Health Services NHS Trust which includes Croydon’s Mayday Hospital, now the University Hospital.

The trust has declared a “serious incident” as a result of the diagnostics backlog. An SI can be reported when there is possibility of unexpected or avoidable death or severe harm to one or more patients.

“No harm”

The trust concedes that its waiting times pose a “potential clinical risk” but the board papers say several times that there is no evidence any patient has come to harm.  This assurance has been questioned by some trust board members. The trust continues to investigate.

Croydon is the latest in a long line of trusts to have had serious disruption after a Cerner go-live under the NPfIT, with BT as the installation partner.

The trust has kept the implications for patients confidential. This may contravene the NHS’s “duty of candour” – to report publicly on things that go wrong. The duty has come about in the wake of the suffering of hundreds of people in the care of Mid Staffordshire NHS Trust.

Croydon Health Services NHS Trust has decided not to publish its “Cerner Deep Dive” or Cerner “Lessons Learnt” reports, and discussions on the reports have been in Part 2 confidential sections of board meetings.

The trust defended its “Part 2” approach in its statement (below).

Meanwhile the Health and Social Care Information Centre, which runs the NPfIT local service provider contracts, including BT’s agreement to supply Cerner to hospitals in London,  has commissioned Cerner to capture the benefits nationally of Cerner installations.

Q&A

My questions and points to the trust, and its responses are below.

From me to the trust:

Croydon had good reasons to go live with Cerner, and DH funding was a further incentive but the trust does not appear to have been in a position to go live – at any stage – with a Big Bang Cerner implementation. The 7 aborted official go-live dates might have been a sign of why.  It would have been a brave decision to cancel the implementation, especially as:

–  the trust had spent 2 years preparing for it

– DH, BT and Cerner had put a lot of work into it

– there was DH pressure to go live especially after all the missed go-live dates.

The latest board papers say 6 or more times in different places that there has been no harm to patients as a result of the delays and waits.  Some members have raised questions on this and there is the matter of whether the trust is commissioning its own assessments (marking its own work).

On this:

– 50,000 on waiting list

– Cerner deep dive not published

– Lessons Learnt not published (concealment of failures, against the spirit of duty of candour called for by Robert Francis QC and Jeremy Hunt?)

– Diagnostics – an SI reported. The trust has considered the contributing issues which related to Cerner implementation but has not published details of the discussion. Again a concealment of failures?

– An accumulation of over 2,200 patients that were waiting over 6 weeks for diagnostics. Out of that number 160 patients were identified as urgent suspected cancer (USC) and urgent patients.  Can the trust – and patients – be sure there has been no harm?

– “… external assurance through an external clinician will provide the assurance that no patients have suffered harm as a result of the length of the waiting times”. Bringing in an external clinician to provide an assurance no patients have been harmed seems to pre-judge the outcome.  The trust appears to be marking its own work, especially as the backlog of patients awaiting diagnostics may not be cleared until the end of August.

– Managing public and GP perceptions? “Members agreed that GP interactions should be held off until the investigations had produced definite findings. However the Communications Department are on standby to publish information to GPs if required, and the Trust is ready to react to other enquiries. The Trust will in any event publish the incident report after the investigation has been completed.”

– “… the implementation of Cerner in October 2013 had an impact on activity levels and the delivery of RTT standards”. Again no report on this published.

– “An independent assessor would re-check all patients to assure that no harm has resulted. The Committee noted the progress report and requested that this is referred to a Part 2 meeting of the Trust Board …” Concealment of failures again?

– In the past the DH has been prepared to treat patients as guinea pigs in Cerner Big Bang implementations. The philosophy appears to be that the implementations will inevitably be disruptive but it’s for the good of patients in the longer term. That this approach may be unfair on patients in the short term, however, seems not to trouble the NHS hierarchy.

It’s clear clinicians and IT staff are doing their best and working hard for the benefit of patients but the implementation was beyond their control. Meanwhile complaints are increasing, Croydon Health Services was one of the lowest rated trusts for overall patient experience and a sizeable minority of local residents don’t choose the local hospitals for care or treatment. That said some patients rate their care very highly on NHS Choices (although some don’t). The University hospital is rated 2.5 stars out of 5.

One of the most surprising statements in the board papers is this: “… despite the weaknesses in the programme, the overall success of the deployment had been recognised at a national level”. A success? Can the trust in essence say what it likes? Nobody knows for sure what the facts are, given that the trust decides on what to publish and not to publish.

The trust’s response to the above points and questions:

“Due to a temporary failure of our administrative systems, the Trust found in February 2014 that a number of patients who needed to be seen by the imaging service were in breach of the six week waiting standard.

“We have taken immediate action to correct this and are undertaking a thorough review to confirm that no patients were harmed as a result.  The Trust is now working hard to treat patients currently on our waiting lists.  This is referenced in our publicly available Board papers.

“CRS Millennium has delivered a number of improvements that support improving patient experience at the Trust, including more efficient management of medicines, more detailed patient information being conveyed between shifts and departments and better management of beds within the organisation.”

Lessons?

Below are some of the lessons from Croydon’s Cerner go-live. Although the trust hasn’t published its “Lessons Learnt” report, some of lessons are mentioned in its latest board papers:

  • Insufficient engagement from operational and clinical colleagues
  • Time pressures were felt when a full dress rehearsal stretched the capabilities of the information team.
  • Insufficient time and resources were allocated to completion of the outline business and full business cases, as well as to due diligence on the options and costs.  [Business cases for Cerner are still unpublished.]
  • Trust directors agreed that a business case for a project of the size and complexity of the CRS Millennium should have taken longer than 6 weeks to prepare.
  • A failure of senior managers to take stock of the project at its key stages.
  • Too strong a focus on technical aspects
  • Clinicians not always fully appreciating the impact of the changes the system would deliver
  • The hiring of an external change manager to lead the deployment who proved to be “less than wholly successful because of the resulting deficiency in previous experience or knowledge of the culture of the organisation”.
  • The individual left the organisation part way through deployment which led to further challenges.
  • The right people with the right skills mix were not in place at the outset to achieve the transformational change necessary to successfully deploy a new system such as CRS

Comment 

NHS trusts have good reason to modernise their IT using the widely-installed  Cerner electronic patient record system, especially  if it’s a go-live under the remnants of the NPfIT, in which case hospitals receive DH funding and gain from having BT as their installation partner.

But why does a disruption that borders on chaos so often follow NPfIT Cerner implementations? Perhaps it’s partly because the benefits of Cerner, and the extra work required by nurses and doctors and clerical staff to harvest the benefits, is underestimated.

It is in any case difficult to convey to busy NHS staff that the new technology will, in the short-term, require an increase in their workload. Staff and clinicians will need to capture more data than they did on the old system, and with precision. The new technology will change how they work, so doctors may resent it initially, especially as there may be shortcomings in the way it has been implemented which will take time to identify and solve.

The problem with NPfIT go-lives is that they take place in an accountability void. Nobody is held responsible when things go badly wrong, and it’s easy for trusts to play down what has gone wrong. They have no fear of authoritative contradiction because they keep their implementation assessments confidential.

What a difference it would make if trusts had an unequivocal duty of candour over electronic health record – EHR – deployments. They would not be able to go live until they were ready.

The disruption that has followed NPfIT Cerner go-lives has been serious. Appointments and tests for suspected cancer have been lost in the administrative confusion that follows go-live. There have been backlogs of appointments for tens of thousands of patients. Operating theatres have gone under-used because of mis-scheduled appointments.

Now and again a patient may die unnecessarily but the problems have been regarded by the NHS centrally as collateral damage, the price society pays for the technological modernisation of the NHS.

Richard Granger, when head of the NPfIT, said he was ashamed of some Cerner installations. He described some of them as “appalling” but since he made his comments in 2007, some of the Cerner installations have been more disruptive than those he was referring to.

Provided each time there is no incontrovertible evidence of harm to patients as a result of a go-live, officials give the go ahead for more NPfIT Cerner installations.

Guinea pigs?

Disruption after go-live is too often treated as an administrative problem. Croydon’s statement refers to a “temporary problem with our administrative systems”. But new patient record systems can harm patients, as the inquest on 3-year-old Samuel Starr heard.

It’s time officials stopped regarding patients as guinea pigs in IT go-lives. It compounds the lack of accountability when trusts such as Croydon keep the reports from the go-live secret.

Trusts need better technological support but not at the cost of treating any harm to patients as collateral damage.

A tragic outcome for Cerner implementation at Bath?

Openness and honesty is a rarity after health IT problems

Mishandled electronic health record transition

A botched Cerner EHR implementation?

Trinity Medical Center reaches Cerner settlement

CEO and CIO resign after troubled EHR go-live

By Tony Collins

At the foot of the Blue Ridge Mountains, Georgia, in America’s deep south, about 70 miles from Atlanta, is Athens .

It was named at the turn of the 19th century to associate its university with Aristotle and Plato’s academy in Greece. It is home to the Athens Regional Medical Centre, one of the USA’s top hospitals.

There on 4 May 2014 the Centre went live with what it described as the most meaningful and largest scale information technology system in its 95-year history – a Cerner EHR implementation.

Now the Centre’s CEO James Thaw and CIO Gretchen Tegethoff have resigned. The Centre’s implementation of the electronic health record system seems to have been no more or less successful than at UK hospitals.

The main difference is that more than a dozen doctors complained in a letter to Thaw and Tegethoff.  A doctor leaked their letter to the local paper.

“Medication errors”

The letter said the timescales to install the Cerner EHR system were too “aggressive” and there was a “lack of readiness” among the intended users. They called the system cumbersome.

The letter referred to “medication errors … orders being lost or overlooked … (emergency department) and patients leaving after long waits”. An inpatient wasn’t seen by a physician for five days.

“The Cerner implementation has driven some physicians to drop their active staff privileges at ARMC [Athens Regional Medical Centre],” said the letter. “This has placed an additional burden on the hospitalists, who are already overwhelmed. Other physicians are directing their patients to St. Mary’s (an entirely separate local hospital) for outpatient studies, (emergency room) care, admissions and surgical procedures. … Efforts to rebuild the relationships with patients and physicians (needs) to begin immediately.”

The boldness of the letter has won praise in parts of the wider American health IT community.

It was signed by the centre’s most senior medical representatives: Carolann Eisenhart, president of the medical staff; Joseph T. Johnson, vice president of the medical staff; David M. Sailers, surgery department chair; and, Robert D. Sinyard, medicine department chair.

A doctor who provided the letter to the Athens Banner-Herald refused a request to openly discuss the issues with the computer system and asked to remain anonymous at the urging of his colleagues.

Swift action

One report said that at a meeting of medical staff 200 doctors were “solid in their vote of no confidence in the present hospital administration.”

Last week Thaw wrote in an email to staff: “From the moment our physician leadership expressed concern about the Cerner I.T. conversion process on May 15, we took swift action and significant progress has been made toward resolving the issues raised … Providing outstanding patient care is first and foremost in our minds at Athens Regional, and we have dedicated staff throughout the hospital to make sure the system is functioning as smoothly as possible through this transition.”

UK implications?

The problems at the Athens centre raise questions about whether problematic Cerner installations in the NHS should have consequences for CEOs.  Health IT specialists say that, done well, EHR implementations can improve the chances of a successful recovery. Done badly an EHR implementation can harm patients and contribute to death.

The most recent installations of Cerner in the NHS, at Imperial College Healthcare NHS Trust and Croydon Health Services NHS Trust, follow the pattern of other Cerner EHR go-lives in the NHS where there have been hints of problems but the trusts are refusing to publish a picture of how patients are being affected.

What has gone wrong at Athens Regional?

IT staff, replying to the Banner-Herald’s article, have given informed views on what has gone wrong. It appears that the Athens Regional laid off about a third of the IT staff in February 2014, about three months before go-live.

Past project disasters have shown that organisations often need more, not fewer, IT staff, advisers and helpers, at the time of a major go-live.

A further problem is that there appears to have been little understanding or support among doctors for the changes they would need to make in their business practices to accommodate the new system.  Had the organisation done enough to persuade doctors and nurses of the benefits to them of changing their ways of working?

If clinicians do not support the need for change, they may focus unduly on what is wrong with the new system. An organisation that is inherently secretive and resentful of constructive criticism will further alienate doctors and nurses.

Doctors who fully support an EHR implementation may find ways around problems, without complaining.

One comment on the Banner-Herald website says:

“While I have moved on from Athens Regional, I still have many friends and colleagues that are trying to work through this mess. Here is some information that has been reported to me…

“Medications, labs and diagnostic exams are not getting done in a timely manner or even missed all together. Some of this could be training issues and some system.

“Already over worked clinical staff are having to work many extra hours to get all the information in the system. This obviously takes away from patient care.

“Senior leadership tried to implement the system in half the amount of time that is usually required to do such things, with half the staff needed to do it. Why?

“Despite an environment of fear and intimidation the clinical staff involved with the project warned senior administration that the system was not ready to implement and posed a safety risk.

“I have ex-colleagues that know staff and directors that are involved with the project. They have made a valiant effort to make things right. Apparently an 80 to even a 100 hour work week has been the norm of late.

“Some questions that I have: where does the community hospital board stand with all this? Were they asking the questions that need to be asked? Why would the software company agree to do such a tight timeline? Shouldn’t they have to answer some questions as well?”

“Hopefully, this newspaper will continue to investigate what has happened here and not cave to an institution that spends a lot of money on frequent giant full page ads.

“Please remember there are still good people (staff, managers and administrators) that work at ARMC and I am sure they care about the community they serve and will make sure they provide great patient care.”

“The last three weeks have been very challenging for our physicians, nurses, and staff,” said Athens Regional Foundation Vice President Tammy Gilland. “Parts of the system are working well while others are not. The medical staff leadership has been active in relaying their concerns to the administration and the administration has taken these concerns very seriously. Maintaining the highest quality of patient care has always been the guiding principle of Athens Regional Health System.”

Keeping quiet

NHS trusts go quiet about the effect on patients of EHR implementations despite calls by Robert Francis QC and health secretary Jeremy Hunt for openness when things go wrong.

Imperial College Healthcare NHS Trust, which comprises St Mary’s Paddington, Hammersmith Hospital, Charing Cross Hospital, Queen Charlotte’s and Chelsea Hospital, and Western Eye hospital in Marylebone Road, went live with Cerner– but its managers and CEO are refusing to say what effect the system is having on patients.

An FOI request by eHealth Insider elicited the fact that Imperial College Healthcare had 55 different consultants working on the Cerner Millennium project and 45 Trust staff. The internal budget for electronic patient record deployment was £14m.

Croydon Health Services NHS Trust, which comprises Croydon University Hospital (formerly Mayday) and the Purley War Memorial Hospital, went live with Cerner last year, also under BT’s direction.

The trust has been a little more forthcoming than Imperial about the administrative disruption, unforeseen extra  costs and effects on patients, but Croydon’s officials, like Imperial College Healthcare’s spokespeople,  refuse to give any specific answers to Campaign4Change’s questions on the Cerner implementation.

Comment

It was probably unfair of doctors at Athens Regional to judge the Cerner system so soon after go-live but their fierce reaction is a reminder that doctors exist to help patients, not spend time getting to grips with common-good IT systems.

Would an NHS CEO resign after a rebellion by UK doctors over a problematic EHR implementation? It’s highly unlikely – especially if trusts can stop news leaking out of the effects on patients. In the NHS that’s easy to do.

Athens Regional CEO resigns

A tragic outcome for Cerner Millennium implementation?

Athens Regional is addressing computer problems encountered by doctors

Athens Regional is addressing computer problems after patients put at risk

CEO forced out?

 

Should Liverpool Council smile now it’s ending BT joint venture?

By Tony Collins

Liverpool Direct Ltd describes itself as the largest public/private partnership of its kind in the UK. BT and Liverpool City Council formed the joint venture in 2001. At one point it employed more than 1,300 people.

Last year the joint venture had a visit from  Prince Edward who met its apprentices and trainees.

Now Liverpool City Council is taking full ownership of the joint venture. BT is handing back its 60% share in Liverpool Direct to the council. But the way the dissolution is being handled is like a theatre compere smiling exaggeratedly at the audience while he pushes off stage a performer who has overstayed his welcome.

Indeed the council’s report on why BT is being pushed out has an oversized grin on every page. Too much self-conscious praise for BT, perhaps. Which may show how political outsourcing deals have become.

This is the first sentence of the council’s report on why the joint venture with BT is ending:

“BT and Liverpool City Council have enjoyed a long and successful partnership through the joint venture company Liverpool Direct Limited.”

And then:

“The ethos of the Partnership was to place the ‘customer at the heart’ of the organisation through the development of innovative new ways of working building on BT’s global brand and reputation.”

There’s much more praise for BT. From the council’s report:

Groundbreaking achievements have included:

  • Establishment of the first ever 24x7x365 local government contact centre including a call centre which is top quartile
  • The only ‘Benefits Plus’ service in the UK.
  • A comprehensive and integrated network of One Stop Shops serving 350,000 visitors each year.
  • First class ICT infrastructure.
  • Creation of 300 new jobs supporting 3nd party business won by LDL.

But there’s a give-away line in one of the sets of bullet points on some of the benefits of the partnership. In 2011 came a refresh of the 10 year-old deal. The benefits of the refresh:

  • Further price reduction of £22.5m.
  • Increased share of third party business. Potential investment of £17m.
  • Continued sponsorship of ( e.g. BT Convention Centre 2012-2017)
  • The ‘write off’ by both parties of potential legal claims against Liverpool City Council estimated by BT of approximately £56m.
  • Increased ownership level from 20% to 40% in favour of the council.

Spot the anomaly – a write-of legal claims against each other of £56m? So the partnership wasn’t quite so wonderful. But that was 2011. Why is the council now pushing out BT from the Liverpool Direct joint venture – what the council calls officially “The Way Forward”?

Amid all the praise for BT it is not easy to see at first glance why Liverpool Direct is being taken into the council’s full ownership. It turns out that austerity is the reason. The council needs to make more cuts than BT is willing to make, and it recognises that BT needs to make a profit. Which raises the question of whether the council was willing to pay BT a decent profit during bountiful times until cuts began to bite.

From Liverpool Council’s report:

“In the early Autumn of 2013, both parties were in active discussions in an effort to resolve the serious financial savings Liverpool City Council needed to make between 2014 and 2017.

“As a result of these discussions and negotiations, BT agreed a further price reduction of £5m contribution to the budget process for 2014/15 together with a further £5m for the following financial year.

“Whilst the Council really appreciated BT’s continued commitment to the city, the current budget deficit would require a far more substantive financial contribution from the Contract both for 2014/15 and for future years.

“Unfortunately BT feels unable to commit to any further price reduction within the Contract as they need to sustain their own financial position. Moreover, the City Council is now well placed, as a result of the long collaboration with BT and the learning gained from the Partnership, to continue to drive forward business transformation and run the services with consequent cost savings to the city.”

The result is that negotiations will continue with a view to transferring BT’s 60% share in Liverpool Direct to the council by 31 March 2014; and the good news doesn’t stop there.

“The City Council and BT both believe that the transfer will enable additional savings to flow to the council including all income from third party contracts.

“BT remains committed to serving residents and businesses in Liverpool and its long and successful relationship with Liverpool City Council will carry on with BT continuing to provide a range of services to the council. During negotiations in late 2013 BT announced it plans to recruit a further 240 staff in Liverpool to support the growth of high-speed broadband services and has recently committed to being a major sponsor of the 2014 International Festival of Business.”

A dark side?

Behind the smiles Liverpool City Council has, it seems, an unusually secretive side.   Richard Kemp CBE has been a member of Liverpool City Council for 30 years having held major portfolios in both control and opposition. He is leader of the Liberal Democrats on the Council. He was Vice Chair of the Local Government Association of England & Wales for more than 6 years.

He says on his blog that has “taken the very unusual step of asking for two independent enquiries into activities of Liverpool City Council”. He adds: “The cases are related and refer to the tangled web of relationships which surround the Liverpool/Liverpool Direct Ltd/Lancashire/One Call Ltd/BT activities.

“In the first instance I have asked that the Lancashire Police extend their Lancashire investigation into Liverpool. In the second I have asked the Information Commissioner to look at the appalling record of the council in responding to freedom of information requests about any matter relating to Liverpool Direct Ltd.”

He says the council has an excellent record of responding to FOI requests – except when it comes to LDL. “When I raised this with the Mayor at the Mayoral Select Committee I didn’t get any answers …”

He also says:

“I find it amazing that I have been told that no contract exists between Liverpool, Lancashire and BT only to find that there is a legal agreement! As a layman I am unclear as to what the difference is between these two positions.

“We now need external scrutiny and investigation to examine these tangled relationships and work out not only who agreed what and when but also whether Liverpool and Lancashire tax payers are getting value for money for this deal.

“In a system where there is no internal scrutiny, Liverpool Labour members have to ask permission to raise issues in the scrutiny process, I feel that I have no alternative but to ask for help in looking at these affairs outside the council.

“In my career I have not only been a councillor for a long time but also asked to work in other councils which were in severe difficulties with their governance structures. Liverpool feels as bad as any council that I have worked in. There is no clear definition of Member and Officer roles.

“ No effective challenges exist within the system and a centralised almost Stalinist decision-making process pertains … I hope that these external investigations will take place and then that they force change in this secretive and opaque council.”

Infighting

A local paper, the Liverpool Echo, has also been investigating the council and its deal with BT.

It says the deal “has been dogged by accusations of infighting between BT and the Council, after top QCs were brought in to settle disputes over how much work would be awarded to LDL under the terms of its contract”.

An internal council report obtained by the Echo before agreement for the contract refresh in 2011 “showed the it took place amid the threat of costly legal claims by BT if city bosses pulled the plug and did not stay in partnership with them until 2017”.

Private/public deals too secretive – MPs today

A report by the Public Accounts Committee published today Private Contractors and Public Spending says private and public partnerships are too secretive – and they lie largely outside the FOI Act. Indeed a BBC File on 4 investigation into the growing influence of accountancy companies such as Capita in public life reached similar conclusions. File on 4 suggested that even if the contract between Service Birmingham, Capita and Birmingham City Council were published in full it could prove impenetrably complicated.

Margaret Hodge MP, chairman of the Public Accounts Commitee, said today:

 “There is a lack of transparency and openness around Government’s contracts with private providers, with ‘commercial confidentiality’ frequently invoked as an excuse to withhold information.

“It is vital that Parliament and the public are able to follow the taxpayers’ pound to ensure value for money. So, today we are calling for three basic transparency measures:

– the extension of Freedom of Information to public contracts with private providers;

– access rights for the National Audit Office; and

– a requirement for contractors to open their books up to scrutiny by officials.

Comment:

It’s remarkable how council outsourcing deals are becoming more cabalistic despite many initiatives toward more open government.

It’s a pity that things have reached a point where Richard Kemp, a Liverpool councillor of 30 years, ends up reporting his authority to the police and the Information Commissioner.

Meanwhile Liverpool City Council, which is one of the most self-image-protecting authorities in the UK, ends a long-standing joint venture with BT by giving the supplier nothing but praise – in public.

Democracy is a form of government in which all eligible citizens participate equally, either directly or indirectly through elected representatives. Clearly that’s not happening properly in Liverpool – or  some other parts of local government.

Reasons I have asked police and Information Commissioner to come in 

BT ad Capita  –  outsourcing joint ventures under pressure in Liverpool and Birmingham 

Private contractors and public spending – Public Accounts Committee report published today

Coroner criticises hospital’s new IT after boy’s death

By Tony Collins

Avon coroner Maria Voisin said yesterday that a new booking system at the Royal United Bath was responsible for three year-old Samuel Starr not being seen and given timely treatment.

It’s rare for a coroner to criticise a hospital’s new IT in direct terms. It is also rare for evidence to emerge of a link between the introduction of new hospital IT and harm to a patient.

Since patient administration systems began to be installed as part of the National Programme for IT in 2005, the disruption arising from go-lives has led to thousands of patient appointments being delayed at a number of trusts. Usually  trusts contend that no patients have suffered serious harm as a result.

Yesterday at the coroners court near Bristol, the coroner Voisin said: “Due to the failure of the hospital outpatients booking system, there was a five-month delay in Samuel being seen and receiving treatment. Samuel’s heart was disadvantaged and he died following urgent surgery.”

Catherine Holley, Samuel’s mother, said her first warning of Cerner implementation problems came in June 2012, about three months before her son’s death, when the RUH cardiologist’s post clinic letter said:

“I apologised to the parents that I should have seen him in January of this year, but I am afraid a few patients have fallen foul of the Millennium changeover and I suspect this is the case here”.

The RUH installed the Cerner Millennium system at the end of July 2011. After the go-live the trust’s board reports, and a post-implementation review of the system by Pwc in February 2012, raised no serious concerns.

One of Pwc’s main criticisms of the system was the lack of clinical engagement – a common problem with NPfIT deployments.

Some excerpts from Pwc’s report:

“Given the issues and delays there is a still a good deal of uncertainty about whether the planned benefits can be realised. It is therefore important that there is a concerted effort to focus on driving out these benefits, early in 2012.”

“The go live itself did have some issues, but most of those interviewed felt that, relative to other Cerner Millennium implementations, it had gone well. Since the go-live the Trust has been working hard to address the issues, and to help users use the system more effectively…”

“There are a number of issues which arose during the audit which, although not directly covered within the audit objectives, are important in ensuring that the trust realises all the potential benefits from the Cerner Millennium implementation. These are as follows:

“There is a need for greater clinical engagement to ensure that clinicians are using the system in the most effective way – the Trust operates a Clinical Engagement Group which meets on a monthly basis.

“However feedback from a number of interviewees is that this meeting isn’t as productive as it might be and that the Trust needs to think of other ways to improve clinical engagement.”

Pwc said that the trust needed a clinical lead for Cerner Millennium who would focus on key clinical priorities and “sell” the benefits of Millennium to other clinicians. The trust also needed a CIO who would be responsible for IM&T, the Business Intelligence Unit, the Care Records Service team, Records Management, and managing the end of a [local service provider] contract with BT in 2015.

Pwc said the risks included:

  • The end of the contract with BT in 2015
  • Recent issues with BT delivery, with several significant outages of service
  • The issues with clinical engagement.

None of the trust reports – or Pwc’s review –  mentioned any adverse impact of the new system’s implementation on patients.

In 2011, several months after the go-live, the RUH’s “Insight” magazine quoted the Chief Executive James Scott as suggesting the go-live had been a success.

“Thanks to everyone for their hard work and patience during the switchover period,” Scott is quoted as saying. “It has been a major project for the RUH (and our partners BT and Cerner are describing it as the smoothest deployment yet), but we now have the foundation in place to meet the future needs of the Trust and the NHS.”

Care plan?

Catherine Holley expected Samuel to go onto a care plan after he had successful heart surgery at the age of nine months, in 2010. But it proved difficult to arrange an appointment for a scan on the new Cerner Millennium system installed at Royal United Hospital, Bath.

When the scan was eventually carried out Samuel’s heart had deteriorated and he died in September 2012 after an operation that was more complicated than expected.

This week’s three-day inquest at Flax Bourton Coroners Court heard that although Samuel’s medical records had been created on Millennium, no appointments were transferred across.

Speaking after the inquest, Samuel’s parents said they believe the system at Bath failed their son and that mistakes were made.

“It was devastating to hear evidence that an improperly implemented computer appointments system and a series of human errors resulted in the death of our son,”said Paul Starr’s Samuel’s father.

“We accept that mistakes happen but we believe that leaving a child unmonitored for as long as Samuel was, with so many opportunities to attend to him, goes beyond a simple error.

“Our three year old son had a complex cardiac condition and had been scheduled for regular cardiac reviews. In fact he was not properly assessed or nearly 21 months because of further delays.”

A Freedom of Information request revealed that there were 63 missed paediatric cardiac appointments as a result of problems arising from the Cerner implementation – some of which took nearly two years to discover.

Comment

It’s unlikely to have been the Cerner system itself at fault but an  implementation that involved several organisations including the Southern Programme for IT, a branch of the dismantled National Programme for IT.  As such nobody is likely to held responsible for what the coroner called a “failure” of the IT system.

Then again blaming an individual would be pointless.  Learning lessons – the right ones, and not just technical ones – is the objective. It may make a world of difference if those in charge of major go-lives in hospitals acknowledged that delayed appointments, or difficulties arranging them, and a disrupted administration, can harm patients – or worse.

IT in the NHS is seen as an integral part of hospital care when the technology, clinicians, nurses, administrators and other users are working in alignment. But when the IT is not working well, it is seen as something techie which has little or nothing to do with patient care and treatment.

Samuel Starr’s inquest draws attention to the fact that the difference between a good and bad IT implementation in a large hospital can be the difference between life and death.

But hospital deployments of patient administration systems have been going wrong with remarkable regularity for many years.  Isn’t it time to learn the right lessons?   

A tragic outcome for Cerner Millennium implementation at Bath?

By Tony Collins

Three year-old Samuel Starr died in the arms of his parents as his they read him his favourite stories at the local hospital. 

At an inquest this week his parents, and specialists, raised questions about whether long delays in arranging appointments on a new Cerner Millennium system at Bath’s Royal United Hospital, which replaced an old “TDS” patient administration system, was a factor in his death.

Ben Peregrine, the speciality manager for paediatrics at the RUH in Bath,  told the inquest:

“Samuel’s appointment request must have fallen through the cracks between the old and new system.”

After successful heart surgery at 9 months, Samuel should have had regular scans to see if his condition had worsened. But he didn’t have any scans for 20 months, in part because of difficulties in organising the appropriate appointments on Bath’s new Millennium systems.

Though there is no certainty, Samuel may be alive today if he’d had the scans.

In a review of Samuel’s death, which took place in November 2012, the details of which have only just been made public, Bristol Children’s Hospital concluded that appointment delays might have played a part.

It said: “Death was felt to be possibly modifiable if [there had been] earlier surgery before cardiac function deteriorated.”

Samuel had his first surgical procedure, open heart surgery at Bristol Royal Hospital for Children, on 3 March 2010. He was discharged six days later, and referred to the Paediatric Cardiac Clinic at the Royal United Hospital in Bath for check-ups.

This week’s inquest heard that the first check-up took place in Bristol in October 2010, when an echocardiogram, also known as an ‘echo’, was carried out. Samuel’s parents, Paul Starr and Catherine Holley, expected a follow-up appointment in January 2012 but by March they’d not received one.

Their community nurse rang the hospital five times in as many months for a follow-up appointment but could not arrange one. When another echo was eventually taken in June 2012 – 20 months after the first – it was found that Samuel needed urgent surgery which proved more complicated than expected. He died on 6 September 2012.

Paul Starr told the BBC that during the long delays in obtaining an appointment for a further scan Samuel’s heart function went from good to bad. He said: “It is not like he had bad care in that time. He had no care at all.”

Ben Peregrine, the speciality manager for paediatrics at the hospital, told the inquest:

“The new system is now up and running as best as it can be, but as long as there is still humans entering the information there will always be room for error.”

The BBC reported that the delay in Samuel’s treatment “came after a new computerised appointment booking system was introduced at the RUH in 2011. It was only after an appointment had been set that doctors discovered the three-year-old, from Frome in Somerset, needed open heart surgery.”

BBC West’s Inside Out obtained a hospital document “Issues for discussion including any action or learning to be taken as a result of the child’s death. Issues that require broader multi-agency discussion” that has as its first bullet point:

“Failure of the RUH Millennium computer software to organise appointments at the designated time leading to a delay of three months before Samuel was seen by (redacted) in Bath.

“Parents have since told me that Samuel had not had an ECHO for 20 months prior to June 2012. At his previous cardiac appointment (April 2011) [redacted] failed to carry out an ECHO because he was not expecting to see Samuel despite Samuel’s parents being sent an appointment for this day.”

It appears that events at Bath after the Cerner go-live have, in the main, followed a pattern at a dozen or so other trusts that have installed the Millennium system.

The pattern was outlined in a Campaign4change post in December 2012:

– go-live

– chaos

– a trust admission that potential problems, costs and risks were underestimated

– a public apology to patients

– a trust promise that the problems have been fixed

– trust board papers that show the problems haven’t been fixed or new ones have arisen

– ongoing difficulties producing statutory and regulatory reports

– provision in trust accounts for unforeseen costs

– continuing questions about the impact of the new system on patients

– a drying-up of information from the trust on the full consequences of the EPR implementation, other than public announcements on its successful aspects.

Catherine Holley, Samuel’s mother, believes the Millennium implementation at the Royal United Hospital at Bath might have followed the above cycle.

Bath went live with Cerner Millennium at the end of July 2011. An upbeat trust statement at the time to E-Health Insider said:

“We can confirm that the new Cerner Millennium IT system successfully went live on Friday 29 July – as planned – at Royal United Hospital Bath NHS Trust.

“BT and Cerner worked closely with the trust and the Southern Programme for IT on the implementation over the past year – a complex and major change management programme.”

As part of its investigation into Samuel’s death, the BBC asked the RUH how many appointments were overdue to delayed because of the new computer system. Said the BBC’s Inside Out West programme:

“They told us there were 63 overdue appointments some with delays of up to 2 years before they were discovered.”

Separately an FOI request to the trust on the Millennium installation brought the response that there have been 65 cardiac outpatients’ appointments “that have been identified as being were missed due to problems with the delayed and that occurred around implementation of Cerner Millennium… All of these appointments have been followed up and actioned as required.”

The RUH is not discussing Samuel Starr’s death. A spokesperson said the inquest is expected to give Samuel’s family and everyone involved in his care a clearer indication of the circumstances surrounding his death. “We have offered our sincere condolences to the family of Samuel Starr following his sad death.”

Contradictory

RUH Board reports on Millennium’s deployment have had a general “good news” tone. But some of the reports have mentioned potentially serious problems. This was in an RUH board report in 2011 on Millennium:

“… there were significant issues with clinic templates and data that had not been migrated. This affected encounters with long term follow up appointments. As a result this meant that there was unplanned downtime across Outpatients and backlogs developed in addition to those produced as a result of planned downtime”.

Comment:

What’s striking about the reports to the Bath board of directors on the Cerner Millennium implementation is their similarity, in tone and substance, to the “good news” reports of deployments of Millennium at other trusts.

The go lives are nearly always depicted as successes for clinicians that have had minor irritations for administrators.

Now we know from the RUH Bath’s implementation of Millennium that when appointments are delayed as a result of inadequate preparations for, and structural settlement of, a new patient administration system, it can be a matter of life and death.

Indeed the BBC, in its investigation into Samuel Starr’s short life, raises the question of whether delayed appointments have been a factor in other deaths.

But do trusts genuinely care about the bigger picture, or do they regard each case of harm or death as an individual, unique event, to be reviewed after the problems come to light?

At the RUH Bath, IT appears to be treated as a separate department, too little interweaved with care and treatment. Managers talked enthusiastically of smartcard use, the work of the service desk, the need for more printers, resolving BT outages, the benefits of the service security model, champion users and floorwalkers, completing the Readiness Workbook, and keeping the Deployment Hazard Document up to date – while the parents of Samuel Starr could not get an appointment on the new system for their son to have a vital heart scan.

In 2011 a senior executive at Bath told his trust staff: “Our partners BT and Cerner are describing it [the go-live of Cerner Millennium] as the smoothest deployment yet” and “we now have the foundation in place to meet the future needs of the Trust and the NHS”.

Will things improve?

The comment in my post of December 2012, which was about Royal Berkshire’s implementation of Cerner Millennium, seems apt so some it may be worth repeating (below).

“Some Cerner implementations go well and bring important benefits to hospitals and their patients. Some implementations go badly. One question the NHS doesn’t ask, but perhaps should, is: what level of problems is acceptable with a new electronic patient record system?

“It appears from some EPR implementations in the NHS that there is no such thing as a low point. No level of disruption or damage to healthcare is deemed unacceptable.

“Berkshire’s chief executive Edward Donald speaks the truth when he says that the trust’s implementation of Cerner was more successful than at other NHS sites. This is despite patients at his trust attending for clinics that did not exist, receiving multiple requests to attend clinics and not receiving follow-up appointments…

“The worrying thing for those who use the NHS is that, as far as new IT is concerned, it is like flying in a plane that has not been certified as safe – indeed a plane for which there has been no statutory requirement for safety tests. And if the plane crashes it’ll be easy for its operators and supplier to deny any responsibility. They can argue that their safety and risk ratings were at “green” or “amber-green”.

“The lack of interest in the NHS over the adverse effect on patients of patient record implementations means that trusts can continue to go ahead with high-risk electronic patient record system go-lives without independent challenge.”

This very thing seems to have happened at the RUH Bath – with possibly tragic consequences.

Thank you to openness campaigner Dave Orr for drawing my attention to the BBC’s investigation into Samuel Starr’s death.

RUH booking system might have contributed to boy’s death – BBC

Boy died after scan delay – BBC

Best Cerner Millennium implementation yet?

BT earns £1.3bn extra from “dismantled” NHS IT scheme

By Tony Collins

The Department of Health paid BT £1.3bn more from the “dismantled” NHS IT contracts than the company first expected.

In 2004 BT expected £2.1bn from its contracts under the NHS IT scheme, the National Programme for IT. In fact BT’s payments totalled  £3.4bn to March 2013, according to information contained in a DH letter to the Public Accounts Committee.  The DH’s letter has gone unpublicised until now.

The size of the payments to BT, in the light of financial pressures elsewhere in the NHS, indicate that Connecting for Health, and its successor the Health and Social Care Information Centre,  regard BT’s data spine, the N3 broadband network,  and Cerner and Rio patient administration systems as indispensable.

The Public Accounts Committee has described the NHS IT scheme, the NPfIT,  as a “failed” programme.

Though important parts of BT’s work on the scheme have been successful, a national care records service in which an individual’s electronic patient record can be accessed across  the NHS, hasn’t materialised.

A cut-down version, the Summary Care Record, exists but the NHS and MPs regarded the creation of a detailed national electronic patient record as the main reason for the National Programme for IT.

Despite the extra money  is delivering far fewer Cerner Millennium systems to London’s acute trusts than originally intended, and none of the GP systems.

Payments to BT

After BT won three NPfIT contracts in 2003, the company said in its annual report of 2004 that the deals would be worth a total of £2.1 billion. The NHS deals were among “some of the largest BT has ever won”, said BT’s  2004 annual report. 

Now the DH’s letter to the Public Accounts Committee shows the amounts paid under NPfIT contracts to March 2013: 

  • N3 broadband network  – £937.7 m [BT]. Original contract value £533m.
  • Spine (including Secondary Uses Service)   £1.083.8m [BT]. Original contract value £620m.
  • Core contracts for local clinical systems in London (London Programme for IT, formerly part of the NPfIT ) –  £865.9m [BT]. Original contract value £996m. BT is delivering to far fewer trusts than it originally envisaged.
  •  Core contracts for the south of England – £586.3m. [BT]. No payments were due to BT for the south of England in the original contracts. BT replaced Fujitsu as the local service provider in the south. The DH spent a total of 737.3m on NPfIT contracts in the south of England to March 2013 but of this £151m had been paid to Fujitsu. The Fujitsu NPfIT local service provider contract is the subject of a protracted legal dispute between the company and the DH.

MP Richard Bacon, a member of the Public Accounts Committee, has criticised the size of some of the payments to BT.

Further payments are due to BT under the NPfIT contracts and it may also receive new payments for work under the Care.data project.

Comment

BT’s stunning financial success from the NHS IT scheme shows the value, from a supplier’s perspective, of getting a foot in the door. For some time it has been a monopoly supplier to the NHS. Its grip on the NHS, the HSCIC and the Department of Health, could be diminished if the HSCIC split up its work and awarded a set of new contracts. That is unlikely to happen. Indeed the signs are that some Whitehall officials would like to tie in the NHS to BT for the foreseeable future.

NHS database: is it a top IT priority?

By Tony Collins

It’s called the NHS database but the new “giant” medical records system is to be run by the Health and Social Care Information Centre, largely for the benefit of researchers.

Though it may help patients in the longer term, say by helping to identify what treatments work and don’t, it is arguably not the NHS’s most immediate IT priority.

I said on BBC R4’s Today programme this morning that a top NHS IT priority is providing secure links to health records so that patients with acute and chronic illnesses can be treated in one part of the NHS one week and another part of the health service the following week – perhaps in a different county – and have their updated records accessible wherever they go.

At present patients with multiple problems can end up being treated in different NHS or non-NHS centres without each organisation knowing what the other is doing.  This is dangerous for patients and gives the impression the NHS is technologically backwards.

Links can be made to existing medical records – there are millions of electronic records already in the NHS – without creating a big central database. The records can reside where they are at the moment, inside and outside the NHS, and be linked to securely by clinicians and nurses, subject to the patient’s specific consent.

Indeed patients should be able to look at their record online and correct any mistakes.

Research database

My comment on BBC R4 Today that a research database is a good idea has brought a mixed response – understandably, because are risks. We need some facts from the Health and Social Care Information Centre on who is going to run the database, and how information will be made genuinely anonymous.

The HSCIC concedes in its information material that some patient information on the database will be potentially identifiable, but it implies this is acceptable if the organisations using the data can be trusted.

Why must information be potentially identifiable? And to what extent can the HSCIC be trusted to run the database? It is, after all, managing contracts under the National Programme for IT, a scheme which Jeremy Hunt called a “huge disaster”.

How much extra will be paid to BT which runs the SUS database under the “dismantled” NPfIT? It is likely that BT’s Spine and SUS-related work will link into the new “NHS database”. Have any new contracts gone to open competitive tender?

Hospital group wins $106m settlement in Cerner dispute

By Tony Collins

 A US health organisation Trinity Medical Centre has won a $106m settlement in  a legal dispute with Cerner, which is one the main suppliers of patient record systems to NHS trusts in London and the south.

Under the NPfIT BT has installed Cerner at trusts that include the Royal Free, London, Barnet and Chase Farm Hospitals NHS Trust, Weston Area Health NHS Trust, Barts Health NHS Trust, North Bristol NHS Trust and more recently at Croydon Health Services NHS Trust.

The Wall St Journal says a clinical patient accounting program Trinity bought from Cerner in 2008 was defective and didn’t deliver the promised benefits, which Cerner disputed. Trinity sought about $240m in damages; Cerner estimated $4m.

The companies agreed to submit the dispute to arbitration which began in October 2013.

Cerner said it “strongly disagrees” with the award and believes the claim was based on unique circumstances. It called the award the only material judgment against Cerner in its 34-year history.

US lawyer Michael Dagley says his firm won a $106m settlement for North Dakota-based Trinity Medical Centre in an arbitration case against Cerner.

The firm says that Trinity alleged in 2012 that patient accounting software and other services purchased from Cerner were defective, producing thousands of billing errors.

“We think it’s tremendously significant because it represents the first major victory that we’re aware of by a health care provider against a software vendor,” Dagley said in a statement.

“Providers are under pressure to automate and vendors are under pressure to offer integrated products. Providers want one vendor for all their IT needs, so the vendors have this incentive to deliver software to the market as quickly as possible, and that can lead to products being introduced that are immature and defective, which in health care, can cause tremendous damage.”

Last year Cerner said it believed the chance of a material loss related to the matter was remote and it had 147 hospitals and 735 clinics using the patient accounting program.

Despite the settlement Cerner’s share price has held up well.

Trinity Medical Centre is a non-profit organisation with about 2,700 employees.

Dare anyone criticise this IT project – with the CEO as leader?

By Tony Collins

Croydon Health Services NHS Trust has had mixed success with its go-live of the Cerner Millennium system.

It is said to be a technical success but last week board members of the Croydon Clinical Commissioning Group expressed concerns about ongoing problems with the system.

Fouzia Harrington, director of quality and governance told the Croydon Advertiser: “The implementation [of Cerner] itself went well in technical terms, but there have been some implications about how it has been used by staff.

“It’s had far more impact in terms of the time it takes to book people in, for example. There have also been implications in terms of lost information about patients.

“There has been a lack of information about hospital activity, which has an impact on finances and, potentially,the quality of services patients are receiving…”

David Hughes, a lay member of the board, was not satisfied with that reassurance.

“You say that no harm has occurred,” he said,  “but while we’ve had no direct incident so far, patient care has definitely suffered.

“You talk about increased waiting times and there’s a risk that harm may occur because of the difficulty in getting in touch with clinicians who actually know what is going on with the patient.

“I’m very concerned from a quality point of view that our main provider has a serious problem with its information systems.”

Hughes called for action. Although the trust may not be aware of an incident yet it may “come out through further investigation that there has been”.

Some waiting times have increased,  the CCG cannot be certain of exact levels of activity at the hospital, and missing information has made it difficult to commission some services.

The concerns were raised at a board meeting on Tuesday.

Dr Tony Brzezicki, chairman of the CCG, said new system would eventually lead to improvements.  “Hospital patients had five sets of notes before. That in itself posed a risk that Cerner will mitigate,” he said.

“However, there have been administrative delays which mean longer waiting times for patients.There are also issues for the service to primary care which is a significant risk. Some of the problems have been resolved though I am concerned at the time scale because they are certainly impacting on my practice.”

Success

John Goulston is the Croydon Health Services NHS Trust CEO. One of his previous jobs was as Programme Director of the London Programme for IT at NHS London. The LPfIT was formerly part of the National Programme for IT. 

As well as CEO, he chairs the trust’s Informatics Programme Board which has taken charge of bringing Cerner Millennium to Croydon’s community health services and the local University Hospital, formerly the Mayday.

Goulston reported to his board that the Cerner go-live – on 30 September and 1 October last year – was a success.

“Our partners Cerner, BT and Ideal have commented that the Trust has undertaken one of the most efficient roll-outs of the system they have worked on, with more users adopting the system more quickly and efficiently than other trusts … the success we have achieved to date is the result of the efforts of every single system user and all staff members,” said Goulston.

Goulston has said the trust deployed the “largest number of clinical applications in a single implementation in the NHS”. 

The Department of Health provided central funding, and the trust paid for implementation “overheads”.  The Health and Social Care Information Centre was the trust’s partner for the go-live.

The Croydon Advertiser asked Croydon Health Servicesa series of questions about Cerner, including its cost to the NHS, but was sent a short statement.

A spokesman told the Advertiser the system would improve patient administration and means that nurses have access to “quality, detailed information” when delivering care.

He added: “During the initial switch over of systems in September while staff were getting used to the system, some patients did need to wait slightly longer to check in for their clinic appointments.

“The trust has maintained and surpassed our 18 week referral to treatment targets from the initial roll out.”

Croydon’s response

Campaign4Change put some questions to the Croydon trust. These are the questions and its responses: .

Is the trust being completely open – taking seriously the duty of candour –  about problems arising from the Cerner Millennium go-live?

“The Trust takes its duty of candour on all issues very seriously; we believe that transparency is essential in running a modern NHS organisation. We are held to account by our board at public meetings, where the public are able to attend and question our senior management team, by our local health overview and scrutiny committee and our commissioners.

“Recent press coverage on CRS Millennium appeared in the local press when the system was discussed in a public meeting of our commissioners.”

As the CEO is leading the Cerner Millennium project, does this make it difficult for trust staff and trust directors to say anything even mildly critical about the implementation?

“Staff opinions on the implementation of CRS Millennium, both positive and negative, are welcomed by the Trust. Staff have given their frank opinions of the system directly to the Chief Executive both in our monthly all staff meetings and at the open staff engagement surgeries held by our Chief Executive and Chairman. All staff opinions are taken seriously and are acted upon appropriately.”

Given the CEO’s enthusiasm for the implementation is there a special onus on the press office to defend the implementation and play down problems? [I note that the Croydon Advertiser implied its questions had not been answered, and that the Trust gave a short statement instead.]

“The communications team respond to and facilitate a large number of external requests, including from the media, in a transparent, timely and appropriate manner. This same approach is followed on questions about CRS Millennium.

“CRS Millennium will bring about many improvements to patient care and Trust efficiency and we are enthusiastic about communicating these; it is unfortunate that recent press coverage did not consider these positive benefits in any depth.”

A comment on the Croydon Advertiser’s website says:

“When I checked in to out-patients I supplied all my personal details; however the post code I gave was declared invalid by the new system. That filled me with confidence. I also gave my contact as a mobile; however they tried to ring me using an old landline number.”

Comment

It’s generally accepted that having a high-level sponsor for an IT project is essential but when the lead is the CEO, does that make it difficult for people to challenge and constructively criticise?

A “good news” culture tends to prevail – as happened on Universal Credit, on the BBC’s Digital Media Initiative, and within the Department of Health on the NPfIT. Nobody dared to speak the whole truth to power. The truth tends to surface only when a new administration takes over or, in the case of Universal Credit, the minister obtained his own independent reports on project progress.

Campaign4Change put it to the Croydon trust that board directors see reports on the Cerner implementation only every two months and much can happen in the intervening period. This it did not deny.

Even if the trust’s directors met daily would they dare to challenge the CEO? And will the full facts  ever emerge? Things could be much better than CCG directors believe  – or much worse.

After nearly every major NPfIT implementation of the Cerner Millennium system in London and beyond (such as North Bristol) the facts were scarce, and reassurances that no patients had come to harm were plentiful. 

Here we go again?

**

Should lessons have been learned from these Cerner go-lives?

Barts and The London

Royal Free Hampstead

Weston Area Health Trust

Milton Keynes Hospital NHS Trust

Worthing and Southlands

Barnet and Chase Farm Hospitals NHS Trust

Nuffield Orthopaedic

North Bristol.

St George’s Healthcare NHS Trust

University Hospitals of Morecambe Bay NHS Foundation Trust

Birmingham Women’s Foundation Trust

NHS Bury

GPs asked to contact hundreds of patients who may have missed treatment after hospital’s cancer referrals blunder  – Pulse

London LMCs alert over Imperial cancer waits mix-up – Pulse.

GPs kept in the dark over hospital cancer blunder – Pulse

 IT system has increased waiting times and led to lost patient data.

Patient records go-live success – or NPfIT failure

When “life and death” NHS IT goes down

By Tony Collins

Almost unnoticed outside the NHS an email was circulated by health officials last weekend about a national “severity 1” incident involving the Electronic Prescription Service, running on BT’s data Spine .

“The EPS [electronic prescriptions service] database is currently experiencing severe degradation of performance. … BT engineers [are] currently investigating with the database application support team,” said the email.

A severity 1 or 2 incident, which involves a temporary loss of, or disruption to, the Spine or other national NHS system,  is not unusual, according to a succession of emails forwarded to Campaign4Change.

The Department of Health defines a severity 1 incident as a  failure that has the potential to:

— have a significant adverse impact on the provision of the service to a large number of users; or

— have a significant adverse impact on the delivery of patient care to a large number of patients; or

— cause significant financial loss and/or disruption to NHS Connecting for Health [now the Health and Social Care Information Centre], or the NHS; or

— result in any material loss or corruption of health data, or in the provision of incorrect data to an end user.

The Health and Social Care Information Centre, which manages BT’s Spine and other former NPfIT contracts, reports that the spine availability is 99.9% or 100%. But the HSCIC’s emails tell a story of service outage or disruption that is almost routine.

If the spine and other national services  are really available 99.9% of the time, is that good enough for the NHS, especially when ministers and officials are increasingly expecting clinicians and nurses to depend on electronic patient records and electronic prescriptions?  In short, are national NHS IT systems up to the job?

NHS staff access the spine tens of millions of times every month, often to trace patients before accessing their electronic records.  The spine is pivotal to the use of patient records held on Rio and Cerner Millennium systems in London. It is critical to the operation of Choose and Book, the Summary Care Record, Electronic Prescription Service pharmacy systems, GP2GP, iPM/Lorenzo, and the Personal Demographics Service.

According to a Department of Health letter sent to the Public Accounts Committee, payments to BT for the Spine totalled £1.08bn by March 2013.

BT says on its website that its 10-year NHS Spine contract involves developing systems and software to support more than 899,000 registered NHS users. The HSCIC says the Spine is used and supported 24 hours a day, 365 days a day.

“There is a huge amount of industrial-strength robustness, availability, disaster recovery, that you cannot get someplace else,” said a BT executive when he appeared before MPs in May 2011.

Life and death  

Sir David Nicholson spoke of the importance of the spine and other national NHS systems at a hearing of the Public Accounts Committee in 2011. He said they were

“providing services that literally mean life and death to patients today … So the Spine, and all those things, provides really, really important services for our patients…”

When Croydon Health Services NHS Trust went live with a Cerner Millennium patient records system at the end of September a “significant network downtime” – of BT’s N3 network – had an effect on patients.

A trust board paper, dated 25 November 2013 says:

“CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT …”

Below are some of the emails passed to Campaign4Change in the past four months. Written by the Health and Social Care Information, the emails alert NHS users to outages or disruption to GP or national NHS IT systems.

Some HSCIC messages of disruption to service

October 2013

Severity 2
HSCIC
National
CQRS has not received a number of participation status messages.
Also affecting: GPES
USER IMPACT:
CQRS Users are not able to manually submit specific information, this will impact the users’ business process for entry of achievement data.
ACTION BEING TAKEN:
Following a configuration change by the GPES Business Unit a specific code has now been added to the GPET-Q Database. We are currently awaiting confirmation that the addition of the relevant code has been successful. Discussions are taking place regarding the re-submission of status messages. HSCIC conference calls are on-going.

[A severity 2 service failure is a failure [that] has the potential to:

– have a significant adverse impact on the provision of the service to a small or moderate number of service users; or

– have a moderate adverse impact on the delivery of patient care to a significant number of service users; or

– have a significant adverse impact on the delivery of patient care to a small or moderate number of patients; or

– have a moderate adverse impact on the delivery of patient care to a high number of patients; or

– cause a financial loss and/or disruption … which is more than trivial but less severe than the significant financial loss described in the definition of a Severity 1 service failure.]

**

Severity 2
HSCIC
BT Spine
National
Intermittent performance issues on TSPINE.
T-Spine
RESOLUTION:
BT Spine have confirmed that the incident has been resolved and users are able to perform routine business processes without delays.

November 2013

Severity 1
BT Spine
HSCIC
National
Users are unable to log into PDS.
USER IMPACT:
All sites are currently unable to access PDS, this is causing a delay to normal services.
ACTION BEING TAKEN:
BT Spine are working to restore service.

**

Severity 2
BT Spine
HSCIC
National EPS users.
Slow performance on reliable and unreliable messages for EPS.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
BT investigating.

**

Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
ACTION BEING TAKEN:
BT moved the database to an alternate node following application server restarts. This temporarily restored normal message response times however performance has started to degrade again. BT Investigation continues.

**
Severity 1
Atos
HSCIC
National
Multiple users were unable to log in to the Choose & Book application.
ATOS made some network configuration changes overnight 19th/20th November which restored service. After a period of monitoring throughout the day yesterday the service has remained stable and at expected levels. Further activities and investigation will be carried out by several resolver teams which will be scheduled through change management.

**
Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
No further issues of slow response times with EPS messaging have occurred today. BT Spine to continue root cause investigation.

**

Severity 2
Cegedim RX
HSCIC
National
Cegedim RX – Users are experiencing slow performance in EPS 1 and EPS 2.
USER IMPACT:
Users are experiencing slow performance and delays to routine business processes when using EPS 1 and EPS 2.
ACTION BEING TAKEN:
Following a restart of application services, traffic has improved for all new EPS messages. However there is a backlog of EPS messages which may cause delays to routine business processes. Cegedim RX to continue to investigate.

**

December 2012

Severity 1
BT Spine
HSCIC
National
Performance issues have been detected with the transaction messaging system (TMS).
Also affecting: Choose and Book, GP2GP
USER IMPACT:
This may cause delays to routine business processes. This may have an effect on all Spine related systems. This includes PDS, Choose and Book, PSIS, SCR, ACF Services.
ACTION BEING TAKEN:
This has been resolved but BT are currently monitoring performance. Further investigation is required by BT into the root cause.

**

Severity 2
GDIT – CQRS
HSCIC
National
DTS has not processed a CQRS payment file.
CQRS
Also affecting: GPES
USER IMPACT:
This is causing delays to routine business processes.
ACTION BEING TAKEN:
GDIT are currently developing a fix which will be rolled out tomorrow evening, pending successful testing.

January 2014

Severity 1
BT Spine
HSCIC
National
TMS reliable messaging unavailable.
USER IMPACT:
TMS reliable messaging unavailable and users having to implement manual workarounds.
ACTION BEING TAKEN:
Issues experienced due to a planned change overrunning, BT Spine continue to implement the transition activity in order to restore service.

**

Severity 2
BT Spine
HSCIC
National
Users have experienced intermittent issues with the creation and cancellation of smartcards in CMS [Card Management Service for managing smartcards].
CMS
USER IMPACT:
This is intermittently causing delays to routine business processes as some users have been unable to create, cancel, cut or print cards in CMS.
ACTION BEING TAKEN
Users may experience issues with the creation and cancellation of cards in CMS. BT have identified a fix for the issue which is currently undergoing testing prior to deployment into the live environment.

**

Severity 2
BT Spine
HSCIC
National
The maternity browser was unavailable within NN4B.
RESOLUTION:
BT identified a problematic server which was recycled to restore system functionality.

**

Spine scheduled outage for essential maintenance activity.

During critical work to migrate to a new storage solution on Spine an issue was experienced on the Transaction Messaging Service (TMS) in September of this year. The issue resulted in BT failing over the TMS database from its usual site on Live B to Live A to restore service. The failover was completed well within the Service Level Agreement and no detrimental long term impacts to the service were incurred.

On the 15th January 2014 between approximately 22:00-23:30, HSCIC, in conjunction with BT, are planning to relocate the TMS database back to Live B, this is for several critical reasons:

  1. The issues experienced, which prompted the failover, are fully resolved and will not be experienced again as the storage migration work is now complete.
  2. The Spine service is designed to operate with all databases running on Live B so this work supports the optimum configuration for the service.
  3. Most critically the transition for all data on Spine to Spine2 has been designed to operate from a standby site with no live databases on it. Therefore to support the Spine2 transition this work is absolutely essential.

In order to facilitate a safe relocation of the database a 1.5 hour outage is required to TMS. The impact of this to Spine is significant and results in effectively an outage for Spine and its interfaces to connecting systems for that period. The time and date is aimed at the lowest times of utilisation for Spine, to minimise impact to end users, as well as not impacting critical batch processing and Choose & Book slot polls.

 

Date & Time

Change Start Change Finish Services Affected Outage Duration
15/01/2013 22:00 15/01/2013 23:30 Transaction Messaging Service (TMS) 1.5 hours
Service  Impact Description
Choose and Book The Choose and Book service will be available but functionality will be limited until the TMS database has switched over.Users of the web application will experience limited retrievals during the outage window.The system will not be able to create shared-secret for patients who have not been referred via Choose and Book before.Service Providers will be unable to:

  • Perform clinic re-structures and re-arrange appointments for patients for directly bookable services
  • Send DNA messages to Choose and Book.

For directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Choose & Book systems will need to queue the messages and resend to Spine once the TMS service is enabled.

Due to the timing of the outage slot polls will not be affected.

Summary Care Record application (SCRa) The SCRa application will be available but functionality will be limited until the TMS database has switched over. Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates)
DSA The DSA application will be available but functionality will be limited until the TMS database has switched over.Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates).
Electronic Prescription Service (EPS)Pharmacy Systems Reliable messaging will be unavailable for the duration of the switchover work as the TMS service will be suspended dual site. All messages received from EPS systems will be rejected and not go into retry.EPS systems will need to queue the messages and resend to Spine once the TMS service is enabled.
EPS Batch The PPA response for any “claim” messages will not be sent to PPA/PPD. However, EPS will send those response(s) again when the retry jobs are re-activated after the switchover exercise is over. Response for any “claim” messages will not be received until after the switchover. Retry jobs will resend the responses once the TMS service is enabled.
Existing Service Providers (ESPs) There will be varying impacts depending on the product, release version and Spine compliant modules of the solution.ESP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP2GP GP2GP will be unavailable until the TMS database has switched over.GP2GP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP Extraction Service (GPES) GPES functionality will be unavailable until the TMS database has switched over.Messages will be queued on Spine and processed once the TMS service is restored.
GP Systems Functionality for Choose & Book, EPS and GP2GP, SCR will be limited until the TMS database has switched over.For Choose & Book directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Systems will need to queue the messages and resend to Spine once the TMS service is enabled.

iPM/Lorenzo The real-time connection to Spine will be unavailable during the TMS outage. However both systems can be disconnected from Spine and operate without synchronised PDS data.iPM/Lorenzo will need to queue the messages and resend to Spine once the TMS service is enabled.
Millennium An outage of PDS reliable messaging will impact Millennium users.Users will be unable to:

  • trace patients
  • register new patients on PDS
  • book or reschedule appointments

Millennium will need to queue the messages and resend to Spine once the TMS service is enabled.

NN4B Trusts will need to be aware that during the outage NHS numbers cannot be generated, new-births cannot be registered and blood-spot labels cannot be generated and should plan accordingly.All birth notifications will be queued and processed once the TMS service is enabled.
Personal Demographics Service (PDS) Simple traces can be completed on PDS data.PDS reliable messaging will be unavailable until the TMS database has switched over.
RiO Users will be unable to:

  • trace patients
  • register new patients
  • book or reschedule appointments

The RiO system will need to queue the messages and resend to Spine once the TMS service is enabled.

TMS Event Service (TES) The majority of TES functionality will be unavailable during the outage.Trusts will need to be aware EPS, Death notifications, and Patient Care Provision Notifications (change of pharmacy) will be queued and sent to the receiving systems once the TMS service is restored.Any impacted notifications will be queued and sent to the receiving systems once TMS is restored.
TMS Batch (DBS, CHRIS, ONS) DBS will be unavailable until the TMS database has switched over (DBS processing will be suspended for the duration of the exercise).As the TMS switchover will be scheduled to start at 22:00, CHRIS batch should complete before the outage starts (CHRIS batch runs at 20:00 nightly).ONS processing will start at 18:00 nightly. If it doesn’t complete before 22:00, the messages will be queued and processed once the TMS service is restored.

**

Severity 2
BT Spine
HSCIC
National
Users are unable to grant worklist items within UIM.
USER IMPACT:
This is causing delays to routine business processes as users are unable to complete their worklist items within the UIM application.
ACTION BEING TAKEN:
BT investigating.

**

Severity 1
BT Spine
HSCIC
National
The EPS database is currently experiencing severe degradation of performance.
USER IMPACT:
Delays to routine business processes.
ACTION BEING TAKEN:
BT engineers currently investigating with the database application support team.

Comment

David Nicholson is right. The NHS has become dependent on systems such as the Spine. But can doctors ever trust any aspect of the safety of patients to systems that are not available 24×7 as they need to be in a national health service?

It appears that BT and other suppliers have not been in breach of service level agreements, and the HSCIC has a good relationship with the companies.  But does the HSCIC have too great an interest in not finding fault with its suppliers or the contracts, for finding fault  could draw attention to any defects in a service for which the HSCIC is responsible?

Have national NHS IT suppliers a strong enough commercial or reputational interest  in avoiding  a disruption or loss of service, so long as they keep within their service level agreements? 

If nobody sees anything wrong with the reliability of existing national NHS IT services improvements are unlikely. Diane Vaughan’s book on the culture and organisation of NASA shows that experts in a big organisation can do everything right according to the rules  and procedures – and still have a disastrous outcome.