Category Archives: e-Referral Service

After a major IT failure, how did Barts NHS trust manage its image?

Posted on 05/05/2017 | 2 comments

By Tony Collins

It sounded serious. Under the headline

“Cancer patients in limbo as five hospitals suffer ‘major’ IT crash”

the Daily Telegraph said,

“Hundreds of cancer patients have been denied treatment at one of England’s biggest hospital trusts due to a major IT failure that ground basic services to a halt.

“Doctors at five large London hospitals have reported 11 days of “chaos” after the systems used to prescribe chemotherapy doses and share x-ray and MRI images broke down on April 20.

“Barts Health NHS Trust said at least 136 operations had been cancelled due to the crash, as well as “hundreds” of cancer treatment sessions.

“The computer failure also means frantic staff have been unable to process blood tests for all but the most critical cases…

“A doctor at the Royal London Hospital told the Daily Telegraph: ‘We have been forced to leave sick patients on the ward while we go down 16 floors to catch a glance at an x-ray image, then come back and make treatment decisions based on a hazy recollection of it…

“An email sent by managers to staff last week said the crisis had forced cancer teams to rebuild patient records ‘from scratch’.

A medic at Whipps Cross hospital was quoted as saying that a lot of people were stuck in hospital needlessly which increased the likelihood of infection.

The trust runs Mile End Hospital, Newham University Hospital, The Royal London Hospital, St Bartholomew’s Hospital and Whipps Cross University Hospital as well as other NHS sites.

The Barts trust website says it delivers “high quality compassionate care to the 2.5 million people of east London and beyond”.

It has a turnover of £1.25 bn and a workforce of 15,000, making it the largest NHS trust in the country.

According to Health Service Journal, an internal email from Barts’ chief clinical information officer Tim Peachy said the IT failure was primarily a result of an “unexpected failure of a small number of physical disks on which data is stored”.

At one point the trust was manually processing blood test results and X-rays, and arranging for porters at its hospitals to hand deliver paperwork to clinicians.

Barts’ reputation 

In the light of the failure and disclosures in Health Service Journal, Barts confirmed the IT problems in statements to the media. It also contacted patients who were affected by the problems. A Barts statement this week said,

“A major computer equipment failure on Thursday 20 April resulted in a number of IT applications being unavailable to staff.

“‘Unfortunately, it has been necessary to cancel 136 operations, representing about 2.5% of our usual weekly in-patient activity. Several hundred chemotherapy appointments have been cancelled, however we have now recovered the chemotherapy prescribing database.

“Clinical teams have completed a patient-by-patient review to ensure that the appropriate course of action is taken for each of them, endeavouring to keep the disruption to an absolute minimum.

“We apologise to those affected and will reschedule their appointment for as soon as we are able.

“A number of applications have been affected to varying degrees. We have made significant progress in many areas including pathology (blood testing), with image viewing now also restored across the Trust. There are still some other areas where it will take time before we are on track again.”

It added,

“We continue to work urgently to maintain the operational resilience of our services, using tried and tested contingency plans to keep our patients safe.”

Despite the seriousness of the problems, the effect on patients and the uncertainties that media coverage might have created in the minds of those intending to go to Barts’ hospitals, the trust made no mention of the difficulties on its website – where it has a “latest news” section –  or on Twitter.

Barts uses Twitter for good news announcements, comments and congratulations, sometimes with dozens of daily tweets.

But why no mention of the IT problems?

On this point, a Barts spokeswoman said,

“We do not rely on social media to update patients. As a proportionately small number of people will be impacted on by the IT situation we are communicating directly to those affected including at outpatients clinics and via phone, letter as well as through communications with our healthcare partners including GPs.”

Comment

In its media statements Barts has been more open than some NHS organisations.

The usual NHS cycle after a major IT-related failure is a statement saying teething problems have been resolved, or are being resolved, followed by a succession of similar statements over the next few days, weeks or months when it becomes clear the problems haven’t been resolved.

This is what happened with e-Referral Service and Capita’s problems handling GP support services.

That hasn’t happened at Barts. But despite its openness with the media, it’s odd  the trust has published many congratulatory tweets in the past two weeks without a mention of any IT-related problems. They are not even alluded to.

It’s also odd that on its website the Barts “Latest News” section has no mention of the difficulties. But the website does have various good news announcements, including a reference to a positive Care Quality Commission report in April 2007.

Trusts do not have to account to patients, Parliament or anybody for IT-related problems. They are under no obligation to apologise to patients whose stays in hospital are unnecessarily prolonged, or whose appointments, operations and blood tests are cancelled or delayed because of IT-related difficulties.

Back-up systems? 

They also have no obligation to give the public any reason for the failure or explain why there was no back-up system that ensured patients were unaffected.

But amid so many positive announcements, statements and comments to the public on its website and on Twitter, should Barts have left out the other side of the story?

The NHS is an organisation that’s attuned to promulgating good news. It’s rare for a trust board paper and or a trust website to have anything but a good news feel to it.

But telling the public one side of the story does not encourage the public to believe officialdom when it says: “Trust us. We know what we’re talking about.”

Thank you to Zara Pradyer for letting me know about the Daily Telegraph article.

 

2 Comments

Posted in Campaign4Change, e-Referral Service, IT-related failures, NHS

is London Ambulance Service’s back-up system “public endurance”?

Posted on 05/01/2017 | 3 comments

By Tony Collins

In November 2016 London Ambulance Service had its busiest week for seriously ill and injured incidents in the history of the Service.

“The Service is …expecting demand to increase even further throughout December,” said London Ambulance Service at the time.

A few weeks later, on one of the busiest nights of the year, the systems went down, from 12.30am to 5.15am on 1 January 2017. The result was that 999 calls were logged  by pen and paper.

When systems are working normally  an incoming 999 call displays the address registered to that number – if the address is registered.  The London Ambulance operator confirms the location, assesses the severity and an ambulance can be despatched within seconds, with the address on its screen and a satnav pointing the way, according to a comment on The Register.

Pen and paper takes longer because the address and other details need to be given over a radio, which can take minutes.

But pen and paper is the London Ambulance Service’s back-up for IT failures.  Whether it can cope with unprecedented demand – or with a major incident in London – is in doubt.

A former London Ambulance Service paramedic told the BBC there had been waits of an hour for ambulances on 1 January 2017. He said call handlers had been “amazingly helpful”, but it was “easy to become overwhelmed especially in the midst of high call volumes”.

London Ambulance Service declined to answer any questions on its latest system failure.

Malcolm Alexander of the Patients’ Forum for the London Ambulance Service said: “We want to know why it is that this system that cost so much money and is supposed to be so effective is not fail-safe.”

He added: “If this system fails at a time when there is huge pressure in the system, for example if there was a major disaster or a terrorist attack, we are going to be in trouble. We really need to make sure it doesn’t collapse again.”

1992

A report into the collapse of London Ambulance Service systems found that they had had failed for many reasons. The Service had taken a “high-risk” IT approach and did not test systems thoroughly before putting them into service.

(Some may question how much has been learned since then.)

2006

In 2006 the London Ambulance Service systems crashed nine times in a fortnight. Each time staff reverted to pen and paper.

2008

In 2008, when systems failed,  repairs took 12 hours. Again the Service reverted to pen and paper.

2011

In June 2011 an IT upgrade caused the system to go down for about three and half hours. Pen and paper was again the back-up “system”. At the time the London Ambulance Service was upgrading the Commandpoint system, supplied by Northrop Grunman, which the Service deployed in 2010 and still uses.

2013

In 2013 on Christmas Day and Boxing Day the systems went down for separate reasons for several hours each day, with staff reverting to pen and paper.

2015

The Chief Inspector of Hospitals, Mike Richards, recommended that the London Ambulance Service be placed into special measures.

He said at the time,

“The Trust has been performing poorly on response times since March 2014. This is a very serious problem, which the trust clearly isn’t able to address alone, and which needs action to put right.”

Comment

It’s becoming the norm for parts of the public sector to regard the public as captive customers when it comes to going live with new IT or upgraded software.

Rather than test new systems, procedures and upgrades thoroughly before introducing them, some parts of the public sectors are going live with a “let’s see what happens and fix things then” approach.

This has become the semi-official approach to the introduction of Universal Credit – with long delays in payments for some claimants.

Within the NHS, at some hospitals introducing new patient record systems, there has been an internal acceptance that patients may suffer from delays,  perhaps with tragic consequences, at least for three year-old Samuel Starr.

The NHS e-referral service was launched with nine pages of known problems.  And when NHS England launched a streamlined GP support service with Capita, officials knew of the possible problems. But it launched anyway.

After the London Ambulance Service’s IT failure on New Year’s Day, it’s clear that many emergency workers did their best to give a normal 999 service. St John’s Ambulance helped.

But to what extent does senior management at the London Ambulance Service have a “stuff happens” mindset when IT goes seriously wrong?

There’s no individual accountability and no commercial imperative to learn lessons from any of the failures.

And there’s no fervent business or political will to ensure the same or similar mistakes don’t recur.

Every time systems fail, the London Ambulance Service promises an investigation. But where are the results published so that lessons can be learned?

Pen and paper is tried and tested. But demands on the London Ambulance Service are much greater than in the past.

With an unprecedented demand for its services how is it London Ambulance Service’s senior management can comfortably rely on pen and paper as its back-up system?

It can – if nobody in power requires an earnest answer to the question.

Another wider question is whether it’s acceptable to use the public as guinea pigs for new or upgraded IT, with potentially serious or even tragic consequences.

London Ambulance Service suffers New Year’s crash – Computer Weekly

London Ambulance Service hit by new year fault – BBC online

 

 

3 Comments

Posted in Campaign4Change, e-Referral Service, Government IT, IT-related failures, NHS England, Orcale, public sector, public services, Universal Credit

Tagged

Yet another NHS IT mess?

Posted on 09/07/2015 | Leave a comment

By Tony Collins

Last week the National Audit Office reported on the failure of the GP Extraction Service. Health officials  had signed off and paid for a contract even though the system was unfit for use.

The officials worked for organisations that have become part of the Health and Social Care Information Centre.

An unapologetic HSCIC issued a statement on its website in response to the Audit Office report. It said, in essence, that the problems with the GP Extraction Service were not the fault of the HSCIC but rather its predecessor organisations (ignoring the fact that many of the officials and contractors from those defunct organisations moved to the HSCIC).

Now it transpires that the HSCIC may have a new IT-related mess on its hands, this time one that is entirely of its own making – the e-Referral Service.

Last month the HSCIC went live with its e-Referral service without testing the system properly. It says it tested for thousands of hours but still the system went live with 9 pages of known problems.

Problems are continuing. Each time in their routine bulletins officials suggest that an upgrade will solve e-Referral’s problems. But each remedial upgrade is followed by another that does not appear to solve the problems.

The system went live on 15 June, replacing Choose and Book which was part of an earlier NHS IT disaster the £10bn National Programme for IT.

Problems more than teething?

Nobody expects a major new IT system to work perfectly first time but regular outages of the NHS e-Referral Service may suggest that it has more than teething problems.

It’s a common factor in IT-based project failures that those responsible have commissioned tests for many hours but with inadequately designed tests that did not always reflect real-world use of the system. They might also have underestimated loads on the available hardware and networks.

This means that after the system goes live it is brought down for regular hardware and software fixes that don’t solve the problems.  End-users lose faith in the system – as many GPs did with the Choose and Book system – and a misplaced optimism takes the place of realism in the thinking of managers who don’t want to admit the system may need a fundamental redesign.

On the day the e-Referral Service launched, a Monday, doctors had difficulties logging in. Software “fixes” that day made little difference. By the next day HSCIC’s optimism has set in. Its website said:

“The NHS e-Referral Service has been used by patients and professionals today to complete bookings and referrals comparable with the number on a typical Tuesday but we were continuing to see on-going performance and stability issues after yesterday’s fixes.

“We suspend access to the system at lunchtime today to implement another fix and this improved performance and stability in the afternoon.”

The “fix” also made little apparent difference. The next day, Wednesday 17 June, the entire system was “unavailable until further notice” said the HSCIC’s website.

By early evening all was apparently well. An HSCIC bulletin said:

“The NHS e-Referrals Service is now available again. We apologise for the disruption caused to users and thank everyone for their patience.”

In fact, by the next day, Thursday 18 June, all was not well. Said another bulletin:

“Yesterday’s outage enabled us to implement a number of improvements and hopefully this is reflected in your user experience today.

“This morning users reported that there were ongoing performance issues so work has now taken place to implement changes to the configuration to the NHS e-Referral Service hardware and we are currently monitoring closely to see if this resolved the issue.”

About 2 weeks later, on 30 June, HSCIC’s officials said there were ongoing problems, because of system performance in provider organisations that were processing referrals.

Was this HSCIC’s way of, again, blaming other organisations – as they did after the NAO report’s on the failure of the GP Extraction Service project? Said a statement on the HSCIC’s website on 30 June 2015:

“Since transition to the NHS e-Referral Service on Monday 15th June, we have unfortunately experienced a number of problems… Although most of the initial problems were related to poor performance of the system, some residual functional and performance issues persist and continue to affect some of our colleagues in their day-to-day working.

“Most of these on-going problems relate to the performance of the system in provider organisations that are processing referrals, though this does of course have a knock-on effect for referrers.

“Please be assured that the team are working to identify root causes and fixes for these issues.”

By last week – 2 July 2015 – HSCIC warned that it will require a “period of planned downtime on the NHS e-Referral Service tonight which is currently scheduled for between 21:00 and 23:00 for some essential maintenance to fix a high priority functional Incident.”

The fix worked – or did it? HSCIC told Government Computing: “An update was applied to the system overnight from Thursday (July 2) into Friday (July 3) which was successful.”

But …

Monday 6 July 2015 4.15pm. HSCIC e-Referral Service bulletin:

“We would like to apologise for the interruption to service between 13:15 and 13:54 today.  This was not a planned outage and we are investigating the root cause.  If any remedial activity is required we will give notice to all users. Once again please accept our sincere apologies for any inconvenience this caused.”

Why was testing inadequate?

Did senior managers go live without testing how the system would work in the real world, or did they select as test end-users only IT enthusiasts?

Perhaps managers avoided challenging the test system too much in case it gave poor results that could force a redesign.

We probably won’t know what has gone wrong unless the National Audit Office investigates. Even then it could be a year or more before a report is published. A further complicating factor is that the HSCIC itself may not know yet what has gone wrong and may be receiving conflicting reports on the cause or causes of the problems.

An IT failure? – change the organisation’s name

What’s certain is that the NHS has a history of national IT project failures which cause organisational embarrassment that’s soon assuaged by changing the name of the organisation, though the officials and contractors just switch from one to the next.

NHS Connecting for Health, which was largely responsible for the NPfIT disaster, was blended into the Department of Health’s informatics function which was then blended into the HSCIC.

Similarly the NHS Information Centre which was largely responsible for the GP Extraction Service disaster was closed in 2013 and its staff and contractors blended into the HSCIC.

Now, with the e-Referral Service, the HSCIC at least has a potential IT project mess that can be legitimately regarded as its own.

When will a centrally-run national NHS IT-based turn out to be a success? … care.data?

New SRO

Meanwhile NHS England is looking for a senior responsible owner for e-Referral Service on a salary of up to £98,453.

Usually in central government, SROs do the job as an adjunct to their normal work. It’s unusual for the NHS to employ a full-time project SRO which the NAO will probably welcome as a positive step.

But the job description is vague. NHS England says that the SRO for NHS e-Referrals programme will help with a switch from paper to digital for 100% of referrals in England by March 2018.

“The SRO … will have responsibility for the strategic and operational development of the digital journey, fulfilment of the patient and clinical process and the performance of the service. Plans to achieve the strategy will be underpinned by the delivery of short to medium term objectives, currently commissioned from HSCIC and other third party suppliers.”

Key aspects of this role will be to:-

– Ensure the strategy is formulated, understood by all stakeholders and is delivered utilising all available resources efficiently and effectively.

– Ensure the development and management of plans.

– Ensure appropriate system and processes are in place to enable the uptake and on-going use of digital referrals by GP’s, hospitals, patients and commissioners.

– Proactively manage the key risks and issues associated with ensuring appropriate actions are taken to mitigate or respond.

– Monitor and establish accountability on the overall progress of the strategy to ensure completion within agreed timescales.

– Manage the budgetary implications of activity.

– Avoid the destabilisation of business as usual.

– Manage and actively promote the relationships with key stakeholders.

The job will be fixed-term until 31/03/2017 and interviews will be held in London on the 20th July 2015.

The big challenge will be to avoid the destabilisation of business as usual – a challenge beyond the ability of one person?

Government Computing. 

Another fine NHS IT mess

Why was e-Referral Service launched with 9 pages of known problems?

National e-Referral Service unavailable until further notice

 

Leave a comment

Posted in Agile, Campaign4Change, consultants, e-health, e-Referral Service, Government IT, HSCIC, IT-related failures, National Audit Office, NHS, npfit, operational issues

Tagged , , ,

New national e-Referral Service “unavailable until further notice”

Posted on 17/06/2015 | Leave a comment

By Tony Collins

The NHS e-Referral Service which launched nationally on Monday was “unavailable until further notice”, the Health and Social Care Information Centre said at 9.30am today.

“Due to issues experienced overnight the NHS e-Referral Service is unavailable until further notice while essential maintenance is performed. If you have local business continuity processes available, we recommend that you consider invoking them,” says the HSCIC on its website.

“We are working hard to resolve these issues as quickly as possible and to keep disruption to a minimum… We apologise for the disruption caused to some users and thank everyone for their patience.”

Late yesterday afternoon the Health and Social Care Information Centre warned GPs and other users of its e-Referral Service that technical problems were continuing.

The difficulties have aggravated cynicism in the GP community about the ability of centrally-based officials to implement national IT systems.

Is it too soon to question whether e-Referrals is the first IT disaster of the new government? There is also the question of whether GPs have been used as guinea pigs to test for problems with the new system.

Until the service went down GPs were in any case unable to log in or were experiencing long delays in arranging referrals. Some reverted to sending letters by post – or always did use the post and avoided the NPfIT Choose and Book system which e-Referral is replacing.

Fewer than 60% of GPs used Choose and Book to hospital appointments for patients.

On its website at 17.30 yesterday the HSCIC said:

“PLEASE PASS THIS ON TO COLLEAGUES WHO USE THE NHS e-REFERRAL SERVICE

“The NHS e-Referral Service has been used by patients and professionals today to complete bookings and referrals comparable with the number on a typical Tuesday but we were continuing to see on-going performance and stability issues after yesterday’s fixes.

“We suspend access to the system at lunchtime today to implement another fix and this improved performance and stability in the afternoon.

“We are continuing to monitor the service and will implement further fixes if required. If users notice any further issues they should log them with their local service desk in the usual way…

“We apologise for the disruption caused to some users and thank everyone for their patience.

Update 14.00 17 June

The Health and Social Care Information Centre said the e-Referral Service was still down.

“HSCIC are completing the final stage of testing a number of fixes to the NHS e-Referrals Service. It is hoped that the service will be available again later today. A further update will be issued at 15:00 today.”

Update 18.00 17 June

Said the HSCIC:

“The NHS e-Referrals Service is now available again. We apologise for the disruption caused to users and thank everyone for their patience.

Update 15.00 18 June – ongoing problems

“Yesterday’s outage enabled us to  implement a number of improvements and hopefully this is reflected in your user experience today,” said HSCIC’s website.

“This morning users reported that there were ongoing performance issues so work has now taken place to implement changes to the configuration to the NHS e-Referral Service hardware and we are currently monitoring closely to see if this resolved the issue.”

Leave a comment

Posted in Agile, Campaign4Change, e-health, e-Referral Service, Government IT, IT-related failures, Major Projects Authority, NHS, npfit, open source

Tagged , ,

Why was NHS e-Referral service launched with 9 pages of known problems?

Posted on 17/06/2015 | 2 comments

By Tony Collins

Were GPs guinea pigs for live testing of the new national NHS e-Referral Service?

Between 2004 and 2010 the Department of Health marked as confidential its lists of problems with national NPfIT systems, in particular Choose and Book.

So the Health and Social Care Information Centre deserves praise for publishing a list of problems when it launched the national “e-Referrals” system on Monday. But that list was 9 pages long.

The launch brought unsurprised groans from GPs who are used to new national systems going live with dozens of known problems.

The e-Referral Service, built on agile “techniques” and based on open source technology, went live early on Monday to replace “Choose and Book” for referring GP patients to hospitals and to other parts of the NHS.

Some GPs found they could not log on.

“As expected – cannot refer anything electronically this morning. Surprise surprise,” said one GP in a comment to “Pulse” on its article headlined “Patient referrals being delayed as GPs unable to access e-Referrals system on launch day.”

A GP practice manager said: “Cannot access in south London. HSCIC debacle…GPs pick up the pieces. Changing something that wasn’t broken.”

Another GP said: “I was proud never to have used Choose and Book once. Looks like this is even better!”

Other GPs said they avoided using technology to refer patients.

“Why delay referral? Just send a letter. (Some of us never stopped).”

Another commented: “I still send paper referrals – no messing, you know it has gone, no time wasted.”

Dr Faisal Bhutta, a GP partner in Manchester, said his practice regularly used Choose and Book but on Monday morning he couldn’t log in. “You can’t make a referral,” he said.

The Health and Social Care Information Centre has apologised for the disruption. A statement on its website says:

“There are a number of known issues, which are currently being resolved. It is not anticipated that any of these issues will pose a clinical safety risk, cause any detriment to patient care or prevent users from carrying out essential tasks. We have published the list of known issues on our website along with details of how to provide feedback .”

But why did the Centre launch the e-Referral Service with 9 pages of known problems? Was it using GPs as guinea pigs to test the new system?

Comment

The Health and Social Care Information Centre is far more open, less defensive and a better communicator than the Department of Health ever was when its officials were implementing the NPfIT.

But is the HSCIC’s openness a good thing if it’s accompanied by a brazen and arrogant acceptance that IT can be introduced into the NHS without a care whether it works properly or not?

In parts of the NHS, IT works extraordinarily well. Those who design, test, implement and support such systems care deeply about patients. In many hospitals the IT reduces risks and helps to improve the chances of successful outcomes.

But in other parts of the NHS are some technology enthusiasts – at the most senior board level – who seem to believe that all major IT implementations will be flawed and will be improved by user feedback.

The result is that IT that’s inadequately designed, tested and implemented is foisted on doctors and nurses who are expected to get used to “teething” troubles.

This is dangerous thinking and it’s becoming more and more prevalent.

Many poorly-considered implementations of the Cerner Millennium electronic patient record system have gone live in hospitals across England with known problems.

In some cases, poor implementations – rather than any faults with the system itself – have affected the care of patients and might have contributed to unnecessary deaths when records needed urgently were not available, or hospitals lost track of urgent appointments.

A CQC report in March 2015 said IT was a possible factor in the death of a patient because NHS staff were unable to access electronically-held information.

In another incident a coroner criticised a patient administration system for being a factor in the death of three year-old Samuel Starr whose appointment for a vital scan got lost in the system.

Within NHS officialdom is a growing cultural acceptance that somehow a poor IT implementation is different to a faulty x-ray machine that delivers too high a dose of radiation.

NHS officials will always brush off IT problems as teething and irrelevant to the care and safety of patients. Just apologise and say no patient has come to any harm.

So little do IT-related problems matter in the NHS that unaccountable officials at the HSCIC have this week felt sufficiently detached from personal accountability to launch a national system knowing there are dozens of problems with the use of it.

Their attitude seems to be: “We can’t know everything wrong with the system until it’s live. So let’s launch the system and fix the problems as GPs give us their feedback.”

This is a little like the NHS having a template letter of regret to send to relatives and families of patients who die unexpectedly in the care of the NHS. Officials simply fill in the appropriate name and address. The NHS can then fix the problems as and when patients die.

It’s surely time that bad practice in NHS IT was eradicated.  Board members need to question more. When necessary directors must challenge the blind positivism of the chief executive.

Some managers can learn much about the culture of care at the hospitals that implement IT successfully.

Patients, nurses and doctors do not exist to tell hospital managers and IT suppliers when electronic records are wrong, incomplete, not available or are somebody else’s record with a similar name.

And GPs do not exist to be guinea pigs for testing and providing feedback on new national systems such as the e-Referral Service.

e-Referral Service “unavailable until further notice”

Hundreds of patients lost in NPfIT systems

Hospital has long-term NPfIT problems

An NPfIT success at Croydon? – Really?

Physicians’ views on electronic patient records

Patient record systems raise some concerns, says report

Electronic health records and safety concerns

2 Comments

Posted in Agile, Campaign4Change, change management, Choose and Book, Department of Health, e-health, e-Referral Service, Government IT, health, health records, HSCIC, IT successes, IT-related failures, LPfIT, NHS, npfit, open source, open standards, operational issues, procurement, project management

Tagged , ,