Category Archives: npfit

When “life and death” NHS IT goes down

By Tony Collins

Almost unnoticed outside the NHS an email was circulated by health officials last weekend about a national “severity 1” incident involving the Electronic Prescription Service, running on BT’s data Spine .

“The EPS [electronic prescriptions service] database is currently experiencing severe degradation of performance. … BT engineers [are] currently investigating with the database application support team,” said the email.

A severity 1 or 2 incident, which involves a temporary loss of, or disruption to, the Spine or other national NHS system,  is not unusual, according to a succession of emails forwarded to Campaign4Change.

The Department of Health defines a severity 1 incident as a  failure that has the potential to:

— have a significant adverse impact on the provision of the service to a large number of users; or

— have a significant adverse impact on the delivery of patient care to a large number of patients; or

— cause significant financial loss and/or disruption to NHS Connecting for Health [now the Health and Social Care Information Centre], or the NHS; or

— result in any material loss or corruption of health data, or in the provision of incorrect data to an end user.

The Health and Social Care Information Centre, which manages BT’s Spine and other former NPfIT contracts, reports that the spine availability is 99.9% or 100%. But the HSCIC’s emails tell a story of service outage or disruption that is almost routine.

If the spine and other national services  are really available 99.9% of the time, is that good enough for the NHS, especially when ministers and officials are increasingly expecting clinicians and nurses to depend on electronic patient records and electronic prescriptions?  In short, are national NHS IT systems up to the job?

NHS staff access the spine tens of millions of times every month, often to trace patients before accessing their electronic records.  The spine is pivotal to the use of patient records held on Rio and Cerner Millennium systems in London. It is critical to the operation of Choose and Book, the Summary Care Record, Electronic Prescription Service pharmacy systems, GP2GP, iPM/Lorenzo, and the Personal Demographics Service.

According to a Department of Health letter sent to the Public Accounts Committee, payments to BT for the Spine totalled £1.08bn by March 2013.

BT says on its website that its 10-year NHS Spine contract involves developing systems and software to support more than 899,000 registered NHS users. The HSCIC says the Spine is used and supported 24 hours a day, 365 days a day.

“There is a huge amount of industrial-strength robustness, availability, disaster recovery, that you cannot get someplace else,” said a BT executive when he appeared before MPs in May 2011.

Life and death  

Sir David Nicholson spoke of the importance of the spine and other national NHS systems at a hearing of the Public Accounts Committee in 2011. He said they were

“providing services that literally mean life and death to patients today … So the Spine, and all those things, provides really, really important services for our patients…”

When Croydon Health Services NHS Trust went live with a Cerner Millennium patient records system at the end of September a “significant network downtime” – of BT’s N3 network – had an effect on patients.

A trust board paper, dated 25 November 2013 says:

“CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT …”

Below are some of the emails passed to Campaign4Change in the past four months. Written by the Health and Social Care Information, the emails alert NHS users to outages or disruption to GP or national NHS IT systems.

Some HSCIC messages of disruption to service

October 2013

Severity 2
HSCIC
National
CQRS has not received a number of participation status messages.
Also affecting: GPES
USER IMPACT:
CQRS Users are not able to manually submit specific information, this will impact the users’ business process for entry of achievement data.
ACTION BEING TAKEN:
Following a configuration change by the GPES Business Unit a specific code has now been added to the GPET-Q Database. We are currently awaiting confirmation that the addition of the relevant code has been successful. Discussions are taking place regarding the re-submission of status messages. HSCIC conference calls are on-going.

[A severity 2 service failure is a failure [that] has the potential to:

– have a significant adverse impact on the provision of the service to a small or moderate number of service users; or

– have a moderate adverse impact on the delivery of patient care to a significant number of service users; or

– have a significant adverse impact on the delivery of patient care to a small or moderate number of patients; or

– have a moderate adverse impact on the delivery of patient care to a high number of patients; or

– cause a financial loss and/or disruption … which is more than trivial but less severe than the significant financial loss described in the definition of a Severity 1 service failure.]

**

Severity 2
HSCIC
BT Spine
National
Intermittent performance issues on TSPINE.
T-Spine
RESOLUTION:
BT Spine have confirmed that the incident has been resolved and users are able to perform routine business processes without delays.

November 2013

Severity 1
BT Spine
HSCIC
National
Users are unable to log into PDS.
USER IMPACT:
All sites are currently unable to access PDS, this is causing a delay to normal services.
ACTION BEING TAKEN:
BT Spine are working to restore service.

**

Severity 2
BT Spine
HSCIC
National EPS users.
Slow performance on reliable and unreliable messages for EPS.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
BT investigating.

**

Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
ACTION BEING TAKEN:
BT moved the database to an alternate node following application server restarts. This temporarily restored normal message response times however performance has started to degrade again. BT Investigation continues.

**
Severity 1
Atos
HSCIC
National
Multiple users were unable to log in to the Choose & Book application.
ATOS made some network configuration changes overnight 19th/20th November which restored service. After a period of monitoring throughout the day yesterday the service has remained stable and at expected levels. Further activities and investigation will be carried out by several resolver teams which will be scheduled through change management.

**
Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
No further issues of slow response times with EPS messaging have occurred today. BT Spine to continue root cause investigation.

**

Severity 2
Cegedim RX
HSCIC
National
Cegedim RX – Users are experiencing slow performance in EPS 1 and EPS 2.
USER IMPACT:
Users are experiencing slow performance and delays to routine business processes when using EPS 1 and EPS 2.
ACTION BEING TAKEN:
Following a restart of application services, traffic has improved for all new EPS messages. However there is a backlog of EPS messages which may cause delays to routine business processes. Cegedim RX to continue to investigate.

**

December 2012

Severity 1
BT Spine
HSCIC
National
Performance issues have been detected with the transaction messaging system (TMS).
Also affecting: Choose and Book, GP2GP
USER IMPACT:
This may cause delays to routine business processes. This may have an effect on all Spine related systems. This includes PDS, Choose and Book, PSIS, SCR, ACF Services.
ACTION BEING TAKEN:
This has been resolved but BT are currently monitoring performance. Further investigation is required by BT into the root cause.

**

Severity 2
GDIT – CQRS
HSCIC
National
DTS has not processed a CQRS payment file.
CQRS
Also affecting: GPES
USER IMPACT:
This is causing delays to routine business processes.
ACTION BEING TAKEN:
GDIT are currently developing a fix which will be rolled out tomorrow evening, pending successful testing.

January 2014

Severity 1
BT Spine
HSCIC
National
TMS reliable messaging unavailable.
USER IMPACT:
TMS reliable messaging unavailable and users having to implement manual workarounds.
ACTION BEING TAKEN:
Issues experienced due to a planned change overrunning, BT Spine continue to implement the transition activity in order to restore service.

**

Severity 2
BT Spine
HSCIC
National
Users have experienced intermittent issues with the creation and cancellation of smartcards in CMS [Card Management Service for managing smartcards].
CMS
USER IMPACT:
This is intermittently causing delays to routine business processes as some users have been unable to create, cancel, cut or print cards in CMS.
ACTION BEING TAKEN
Users may experience issues with the creation and cancellation of cards in CMS. BT have identified a fix for the issue which is currently undergoing testing prior to deployment into the live environment.

**

Severity 2
BT Spine
HSCIC
National
The maternity browser was unavailable within NN4B.
RESOLUTION:
BT identified a problematic server which was recycled to restore system functionality.

**

Spine scheduled outage for essential maintenance activity.

During critical work to migrate to a new storage solution on Spine an issue was experienced on the Transaction Messaging Service (TMS) in September of this year. The issue resulted in BT failing over the TMS database from its usual site on Live B to Live A to restore service. The failover was completed well within the Service Level Agreement and no detrimental long term impacts to the service were incurred.

On the 15th January 2014 between approximately 22:00-23:30, HSCIC, in conjunction with BT, are planning to relocate the TMS database back to Live B, this is for several critical reasons:

  1. The issues experienced, which prompted the failover, are fully resolved and will not be experienced again as the storage migration work is now complete.
  2. The Spine service is designed to operate with all databases running on Live B so this work supports the optimum configuration for the service.
  3. Most critically the transition for all data on Spine to Spine2 has been designed to operate from a standby site with no live databases on it. Therefore to support the Spine2 transition this work is absolutely essential.

In order to facilitate a safe relocation of the database a 1.5 hour outage is required to TMS. The impact of this to Spine is significant and results in effectively an outage for Spine and its interfaces to connecting systems for that period. The time and date is aimed at the lowest times of utilisation for Spine, to minimise impact to end users, as well as not impacting critical batch processing and Choose & Book slot polls.

 

Date & Time

Change Start Change Finish Services Affected Outage Duration
15/01/2013 22:00 15/01/2013 23:30 Transaction Messaging Service (TMS) 1.5 hours
Service  Impact Description
Choose and Book The Choose and Book service will be available but functionality will be limited until the TMS database has switched over.Users of the web application will experience limited retrievals during the outage window.The system will not be able to create shared-secret for patients who have not been referred via Choose and Book before.Service Providers will be unable to:

  • Perform clinic re-structures and re-arrange appointments for patients for directly bookable services
  • Send DNA messages to Choose and Book.

For directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Choose & Book systems will need to queue the messages and resend to Spine once the TMS service is enabled.

Due to the timing of the outage slot polls will not be affected.

Summary Care Record application (SCRa) The SCRa application will be available but functionality will be limited until the TMS database has switched over. Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates)
DSA The DSA application will be available but functionality will be limited until the TMS database has switched over.Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates).
Electronic Prescription Service (EPS)Pharmacy Systems Reliable messaging will be unavailable for the duration of the switchover work as the TMS service will be suspended dual site. All messages received from EPS systems will be rejected and not go into retry.EPS systems will need to queue the messages and resend to Spine once the TMS service is enabled.
EPS Batch The PPA response for any “claim” messages will not be sent to PPA/PPD. However, EPS will send those response(s) again when the retry jobs are re-activated after the switchover exercise is over. Response for any “claim” messages will not be received until after the switchover. Retry jobs will resend the responses once the TMS service is enabled.
Existing Service Providers (ESPs) There will be varying impacts depending on the product, release version and Spine compliant modules of the solution.ESP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP2GP GP2GP will be unavailable until the TMS database has switched over.GP2GP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP Extraction Service (GPES) GPES functionality will be unavailable until the TMS database has switched over.Messages will be queued on Spine and processed once the TMS service is restored.
GP Systems Functionality for Choose & Book, EPS and GP2GP, SCR will be limited until the TMS database has switched over.For Choose & Book directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Systems will need to queue the messages and resend to Spine once the TMS service is enabled.

iPM/Lorenzo The real-time connection to Spine will be unavailable during the TMS outage. However both systems can be disconnected from Spine and operate without synchronised PDS data.iPM/Lorenzo will need to queue the messages and resend to Spine once the TMS service is enabled.
Millennium An outage of PDS reliable messaging will impact Millennium users.Users will be unable to:

  • trace patients
  • register new patients on PDS
  • book or reschedule appointments

Millennium will need to queue the messages and resend to Spine once the TMS service is enabled.

NN4B Trusts will need to be aware that during the outage NHS numbers cannot be generated, new-births cannot be registered and blood-spot labels cannot be generated and should plan accordingly.All birth notifications will be queued and processed once the TMS service is enabled.
Personal Demographics Service (PDS) Simple traces can be completed on PDS data.PDS reliable messaging will be unavailable until the TMS database has switched over.
RiO Users will be unable to:

  • trace patients
  • register new patients
  • book or reschedule appointments

The RiO system will need to queue the messages and resend to Spine once the TMS service is enabled.

TMS Event Service (TES) The majority of TES functionality will be unavailable during the outage.Trusts will need to be aware EPS, Death notifications, and Patient Care Provision Notifications (change of pharmacy) will be queued and sent to the receiving systems once the TMS service is restored.Any impacted notifications will be queued and sent to the receiving systems once TMS is restored.
TMS Batch (DBS, CHRIS, ONS) DBS will be unavailable until the TMS database has switched over (DBS processing will be suspended for the duration of the exercise).As the TMS switchover will be scheduled to start at 22:00, CHRIS batch should complete before the outage starts (CHRIS batch runs at 20:00 nightly).ONS processing will start at 18:00 nightly. If it doesn’t complete before 22:00, the messages will be queued and processed once the TMS service is restored.

**

Severity 2
BT Spine
HSCIC
National
Users are unable to grant worklist items within UIM.
USER IMPACT:
This is causing delays to routine business processes as users are unable to complete their worklist items within the UIM application.
ACTION BEING TAKEN:
BT investigating.

**

Severity 1
BT Spine
HSCIC
National
The EPS database is currently experiencing severe degradation of performance.
USER IMPACT:
Delays to routine business processes.
ACTION BEING TAKEN:
BT engineers currently investigating with the database application support team.

Comment

David Nicholson is right. The NHS has become dependent on systems such as the Spine. But can doctors ever trust any aspect of the safety of patients to systems that are not available 24×7 as they need to be in a national health service?

It appears that BT and other suppliers have not been in breach of service level agreements, and the HSCIC has a good relationship with the companies.  But does the HSCIC have too great an interest in not finding fault with its suppliers or the contracts, for finding fault  could draw attention to any defects in a service for which the HSCIC is responsible?

Have national NHS IT suppliers a strong enough commercial or reputational interest  in avoiding  a disruption or loss of service, so long as they keep within their service level agreements? 

If nobody sees anything wrong with the reliability of existing national NHS IT services improvements are unlikely. Diane Vaughan’s book on the culture and organisation of NASA shows that experts in a big organisation can do everything right according to the rules  and procedures – and still have a disastrous outcome.

BBC World at One’s focus on Government IT

By Tony Collins

The lead item on BBC R4’s World at One on Friday was about Government IT contracts.

On the programme were the government’s Chief Procurement Officer Bill Crothers, Cabinet Office minister Francis Maude, the chairman of the Public Accounts Committee Margaret Hodge, the UK IT Association, and me.

Some of the points made:

–  Bill Crothers gave an example of what he called “abuse” by some big IT suppliers. He said a young man who works for him lost his power cable. The supplier quoted £65 for a replacement. The price should have been £5 or £6.  When Crothers queried it, the supplier justified its price on grounds of security. Crothers could not believe that a power lead had security implications so he questioned the price again and received several pages of explanation from the supplier, which he did not read. Eventually the supplier “was good enough to reduce the price to £37”.

– HMRC was charged £30,000 for changing some text on its website.

– Francis Maude said a DWP team and a further 12 people from the Cabinet Office’s Government Digital Service had built – in only three months – a prototype of a digital solution to support the introduction of Universal Credit. The system cost just over £1m, he said. [Separately big IT suppliers at DWP have been paid £303m up to March 2013 for Universal Credit work.] Maude declined to predict the outcome of the “twin-track” work on the UC project.

– Some big legacy systems may soon need replacing – those that pay about £60bn a year in state pensions and collect nearly £100bn a year in VAT. “Those are going to be big projects,” said Margaret Hodge. “I don’t think we have seen the end of big projects, or the end of disasters.”

World at One in detail

Presenter Shaun Ley and BBC political correspondent Ross Hawkins focused on government IT because of an announcement by the Cabinet Office that it is drawing the line on “bloated and wasteful IT contracts”. The Cabinet Office was pitching its announcement as marking a “massive change,” said Hawkins.

Ley said Francis Maude announced the safeguards  in an attempt to ensure that IT contracts don’t become multi-billion pound failures. He said that the abandoned NPfIT had cost close to £10bn.

Hawkins quoted the UK IT Association as saying that  government did not know how to do deals with smaller suppliers. On the government’s relationship with big suppliers UKITA said the government was like a “battered wife or husband who doesn’t seem to know how to leave.”

Appalling

Hawkins said Crothers has the air of a man going to war. Crothers’ conclusion on the way things are at the moment:

“This is about the oligopoly, the cluster of big suppliers that have had it took good for too long. It’s reflective of monopolistic or oligopolistic behaviour.  It is not acting as if they are in hungry and in a competitive market.  That’s appalling.”

Universal Credit

Hawkins asked Francis Maude how confident he was that what was being put in place on Universal Credit would work.

“I hope it will work,” said Maude. “The digital solution was created by a team within DWP with a dozen or so GDS [Government Digital Service] staff assisting.

“They created a working prototype for a digital solution within 3 months at a cost of only a bit over £1m. That certainly can be basis of a successful long-term solution.”

Hawkins [to Maude] “I asked you whether you were confident the approach with DWP would work and you said you hoped it would. That suggests to me that maybe you are not (confident).”

Maude: “N0-one knows with these things. Anyone who says you are certain everything is going to succeed … the way we do things now is build something quickly, test it, prove it, test it with users, and so you can’t have certainty about any of these outcomes.”

Outsourcing failures

Hawkins said “We have had story after embarrassing story about outsourcing failures [such as the] government being charged for tagging dead people … now ministers  have an interest in coming out on the front foot and just for once being on the attack and having a whack at the IT companies.

“You don’t need to be a political genius to work out why they would like to do that rather than be endlessly explaining themselves after embarrassing stories in the papers.”

Ley (to me): “Is this the best way to deal with the problems government has experienced? The journalist Tony Collins has written widely  about project failures in IT in both the public and private sectors.”

I replied that big companies have sometimes charged a lot to make small software changes.  The Cabinet Office’s “red lines” were a good idea though they were a formalising of restrictions that had been in place some time.

The Cabinet Office doesn’t have the power to make changes happen because departments are accountable to Parliament for their spend and so don’t want much interference from the Cabinet Office. But the Cabinet Office is right to try and reduce the amounts spent on big projects.

Ley: “What will be the effect of breaking up contracts?”

I said I hoped the Cabinet Office’s restrictions would bring about a change in culture in departments against the assumption that big is beautiful. Big projects should be split into components which would give SMEs a greater involvement and could reduce the risks of projects failing.

More project disasters?

Hodge gave her reaction to the Cabinet Office’s restrictions in the context of the Universal Credit project.

“Francis Maude and Cabinet Office have been trying really hard to get some sense into the way that project has developed. But sadly the news we have had lately suggests to me that they have failed. It is about £400m so far on IT.

“What went wrong there was that the department [DWP] thought it [UC] was a big IT project instead of thinking:  we are going to be changing our business; we are going to get 6 benefits rolled into one. They [the DWP] have not written off that money [£303m] which is what my committee thinks they should have done, because they want to save face. Down the line I think we’ll see some disasters there.

“There are a lot of projects around  government, what are called legacy projects, where old systems need to be replaced . They are big projects – pensions in DWP where £60bn is given out a year;  VAT receipts  in HMRC where nearly £100bn is collected. Those are going to be big projects. I don’t think we have seen the end of big projects, or the end of disasters.”

Ley: “What about breaking them up into smaller projects? Won’t that reduce potential risks?”

Hodge: “The important thing is what Tony Collins was saying to you. What we find is that the skills don’t exist within departments, either to commission the IT properly or to manage the suppliers once they have the IT in place.

“We are about to examine the army recruitment contract – I think that is what we’ll find.  The MoD hasn’t got the skills to manage it.

Ley: “Do you welcome the ending of automatic contract extensions?”

“I warmly welcome that. This is a small step in the right direction. Having an expert as we have in Bill Crothers in the Cabinet Office is really important. What we haven’t got are skills in the departments. It is not like a business. If it was, Bill Crothers would probably run IT across the whole of government. Our departments run in silos. They haven’t got the skills. They have this demand for big, big programmes in the future and I don’t think we have seen, sadly, the end of IT disasters.”

Update

Thank you to Dave Orr for drawing my attention to an excellent piece on the World at One item by procurement expert Peter Smith who concludes:

“… There is a big issue – large suppliers have not covered themselves in glory, but small suppliers just can’t develop huge systems for DWP or MOD.

“The large suppliers must have a role, but we have to manage these contracts better. And the answer can’t just be a small hit squad in Cabinet Office. This needs real capability development across government, which we haven’t really seen as yet in a coordinated fashion.”

BBC World at One – Government IT contracts

Bill Crothers on BBC Radio 4 – suppliers get another good kicking

Are Govt IT-based project disasters over? Ask the Army

By Tony Collins

When senior civil servants know an IT-based project is in trouble and they’re unsure how bad things are, they sometimes offer their minister an all-encompassing euphemism to publicly describe the status of the scheme – teething.

Which may be why the defence secretary Philip Hammond told the House of Commons in November 2013 that the IT project to support army recruiting was having “teething” problems.

Now Hammond knows more, he says the problems are “big”. He no longer uses the “t” word. Speaking about the £440m 10-year Recruitment Partnering Project in the House of Commons this week Hammond said:

“Yes, there are big problems with the IT and I have told the House on repeated occasions that we have IT challenges…”

Only a few days ago Cabinet Office minister Francis Maude suggested that Government IT was no longer a byword for disaster, though he accepted there were still challenges.

In a speech on how he expected the UK to become the G8’s most digital government by next year (whatever that means) Maude said: “… it’s great news that DVLA is about to launch online driving records which can be used by anyone with a driving licence as well as by the insurance industry.

“Back in 2010 our digital offering was limited at best and government IT was a by-word for disaster … There are still challenges but with the help of the Government Digital Service I am determined that the UK will be the G8’s most digital government by next year.”

A few days later The Times reported on a leaked Gartner report on the army Recruitment Partnering Project. The report expressed concerns about the entire plan, including a poor project management team and delays that were allowed to spiral out of control.

It claimed that the Army’s recruitment division had failed to challenge MoD policy in 2011 that had apparently favoured the less suitable of the two competing bidders chasing the contract.

Hammond is said to be mulling over a £50m payout for Capita to build a new infrastructure for the recruiting system instead of trying to integrate it with systems supplied by the “Atlas” consortium under the Defence Information Infrastructure project. Hammond told the House of Commons this week:

“… there have been initial difficulties with that recruiting process as we transition to the new recruiting arrangements with Capita.

“In particular, we have encountered difficulties with the IT systems supporting the application and enlistment process. The decision to use the legacy Atlas IT platform was deemed at the time to be the quickest and most cost-effective way of delivering the new recruitment programme.

“An option to revert to a Capita hosting solution was included in the contracts as a back-up solution.

“I was made aware in the summer of last year that the Army was encountering problems with the integration of the Capita system into the Atlas platform. Since then we have put in place a number of workarounds and mitigation measures for the old IT platform to simplify the application process, and we have reintroduced military personnel to provide manual intervention to support the process.

“Having visited the Army’s recruitment centre in Upavon [Wiltshire] on 30 October, it became clear to me that, despite the Army putting in place measures to mitigate those problems in the near term, further long-term action was needed to fix the situation.

“It was agreed in principle at that point that the Atlas system was not capable of timely delivery of the Capita-run programme and that we would need to take up the option of reverting to Capita building a new IT platform specifically to run its system, which will be ready early next year.

“… we have already taken action to bring in a range of new initiatives that will make it progressively easier and quicker for applicants … the introduction this month of a new front-end web application for Army recruitment; a simplified online application form; more streamlined medical clearance processes …

“With an improved Army recruitment website, streamlined medicals and an increase in the number of recruiting staff, recruits should see a much-improved experience by the end of this month.

“.. we are looking at further ways of improving the management of the recruiting process in the intervening period before the introduction of the advanced IT system now being developed in partnership with Capita, which is expected to be deployed in February 2015…”

Vernon Croaker, Labour’s defence spokesman, said the recruitment project was an IT fiasco. He wondered why Hammond had initially described the problems as teething.

“Today we have learned [from newspapers] that the problems are even worse than anyone thought and still have not been fixed.

“Will the Defence Secretary tell the House which Minister signed off the deal and who has been responsible for monitoring it?

“… Will the Secretary of State also confirm that £15.5m has been spent building the existing flawed computer system behind the project? Finally, is it correct that this continuing disaster is costing taxpayers £1 million every month?…”

Croaker quoted a minister Andrew Robathan as telling MPs on 10 April 2013 that the “Recruiting Partnering Project with Capita…will lead to a significant increase in recruiting performance”.

Croaker said: “Is there any Member of this House, any member of our armed forces or, indeed, any member of the British public who still believes that?”

In March 2012 Capita announced that the Recruitment Partnering Project was valued at about £44m a year for 10 years and was expected to deliver benefits in excess of £300m to the armed forces. It would “release military recruiters back to the front line” said Capita.

Comment. Francis Maude is probably right: there don’t seem to be as many big IT-based project failures as in previous decades. But then the truth isn’t known because progress reports on big IT-related schemes are not published.

Indeed little would be known about the Capita Recruitment Partnering Project is not for the leaked report to The Times. Without the leak, public information on the state of the project would be confined to Hammond’s “teething problems” comment to MPs last November.

Internal and external reports on the state of the Universal Credit IT project continue to be kept secret.  It’s not even clear whether ministers are properly briefed on their big IT projects. Hammond almost certainly wasn’t last year. IDS was left to commission his own “red team” review of Universal Credit IT.

Perhaps the “good news” reporting culture in Whitehall explains why the NHS IT scheme, the NPfIT, continued to die painfully slowly for 7 years before senior officials and ministers started to question whether all was well.

Hammond is still getting wrong information. He described “Atlas” systems in the House of Commons as the “legacy IT platform”.

The Atlas contract for the Defence Information Infrastructure was awarded in 2005 for 10 years. It doesn’t even expire until next year. It may be convenient for officials to suggest that the reason Capita has been unable to link new recruitment systems into the DII network is because DII is old – legacy IT.  But the multi-billion pound Atlas DII project cannot be accurately described as “legacy” yet.

If ministers don’t get the truth about their big IT projects until serous problems are so obvious they can no longer be denied, how can Parliament and taxpayers expect to get the truth?

Lessons from NASA?

NASA put in place processes, procedures and rules to ensure engineers were open and deliberately adversarial in challenging assumptions. Even so it has had difficulties getting engineers to express  their views freely.

Diane Vaughan in her excellent book “The Challenger Launch Decision” referred to large organisations that proceeded as if nothing was wrong “in the face of evidence that something was wrong”.  She said NASA made a series of seemingly harmless decisions that “incrementally moved the space agency towards a catastrophic outcome”.

After the loss of Challenger NASA made many changes. But an investigation into the subsequent tragedy of the Columbia space shuttle indicated that little had actually changed – even though few of the top people who had been exposed to the lessons of Challenger were still in position.

If NASA couldn’t change when lives depended on it, is it likely the UK civil service will ever change?  A political heavyweight,  Francis Maude has tried and failed to get departments to be more open about progress or otherwise on their big IT-based projects.  Permanent secretaries now allow the out-of-date “traffic light” status of some projects to be published in the annual report of the Major Projects Authority. That is not openness.

The failure so far of the Recruitment Partnering Project, the routine suppression of information on technology-based scheme such as this, and the circumscribed “good news” briefings to ministers, suggest that government IT-based project failures are here to stay, despite the best intentions of the Cabinet Office, GDS and the Major Projects Authority.

Thank you to campaigner Dave Orr for his email on the recruitment project

Top 5 posts on this site in last 12 months

Below are the top 5 most viewed posts of 2013.  Of other posts the most viewed includes “What exactly is HMRC paying Capgemini billions for?” and “Somerset County Council settles IBM dispute – who wins?“.

1) Big IT suppliers and their Whitehall “hostages

Mark Thompson is a senior lecturer in information systems at Cambridge Judge Business School, ICT futures advisor to the Cabinet Office and strategy director at consultancy Methods.

Last month he said in a Guardian comment that central government departments are “increasingly being held hostage by a handful of huge, often overseas, suppliers of customised all-or-nothing IT systems”.

Some senior officials are happy to be held captive.

“Unfortunately, hostage and hostage taker have become closely aligned in Stockholm-syndrome fashion.

“Many people in the public sector now design, procure, manage and evaluate these IT systems and ignore the exploitative nature of the relationship,” said Thompson.

The Stockholm syndrome is a psychological phenomenon in which hostages bond with their captors, sometimes to the point of defending them.

This month the Foreign and Commonwealth Office issued  a pre-tender notice for Oracle ERP systems. Worth between £250m and £750m, the framework will be open to all central government departments, arms length bodies and agencies and will replace the current “Prism” contract with Capgemini.

It’s an old-style centralised framework that, says Chris Chant, former Executive Director at the Cabinet Office who was its head of G-Cloud, will have Oracle popping champagne corks.

2) Natwest/RBS – what went wrong?

Outsourcing to India and losing IBM mainframe skills in the process? The failure of CA-7 batch scheduling software which had a knock-on effect on multiple feeder systems?

As RBS continues to try and clear the backlog from last week’s crash during a software upgrade, many in the IT industry are asking how it could have happened.

3) Another Universal Credit leader stands down

Universal Credit’s Programme Director, Hilary Reynolds, has stood down after only four months in post. The Department for Work and Pensions says she has been replaced by the interim head of Universal Credit David Pitchford.

Last month the DWP said Pitchford was temporarily leading Universal Credit following the death of Philip Langsdale at Christmas. In November 2012 the DWP confirmed that the then Programme Director for UC, Malcolm Whitehouse, was stepping down – to be replaced by Hilary Reynolds. Steve Dover,  the DWP’s Corporate Director, Universal Credit Programme Business, has also been replaced.

4) The “best implementation of Cerner Millennium yet”?

Edward Donald, the chief executive of Reading-based Royal Berkshire NHS Foundation Trust, is reported in the trust’s latest published board papers as saying that a Cerner go-live has been relatively successful.

“The Chief Executive emphasised that, despite these challenges, the ‘go-live’ at the Trust had been more successful than in other Cerner Millennium sites.”

A similar, stronger message appeared was in a separate board paper which was released under FOI.  Royal Berkshire’s EPR [electronic patient record] Executive Governance Committee minutes said:

“… the Committee noted that the Trust’s launch had been considered to be the best implementation of Cerner Millennium yet and that despite staff misgivings, the project was progressing well. This positive message should also be disseminated…”

Royal Berkshire went live in June 2012 with an implementation of Cerner outside the NPfIT.  In mid-2009, the trust signed with University of Pittsburgh Medical Centre to deliver Millennium.

Not everything has gone well – which raises questions, if this was the best Cerner implementation yet,  of what others were like.

5) Universal Credit – the ace up Duncan Smith’s sleeve?

Some people, including those in the know, suspect  Universal Credit will be a failed IT-based project, among them Francis Maude. As Cabinet Office minister Maude is ultimately responsible for the Major Projects Authority which has the job, among other things, of averting major project failures.

But Iain Duncan Smith, the DWP secretary of state, has an ace up his sleeve: the initial go-live of Universal Credit is so limited in scope that claims could be managed by hand, at least in part.

The DWP’s FAQs suggest that Universal Credit will handle, in its first phase due to start in October 2013, only new claims  – and only those from the unemployed.  Under such a light load the system is unlikely to fail, as any particularly complicated claims could managed clerically.

 

MP calls for candour after Cerner NPfIT go-live at Croydon

By Tony Collins

Richard Bacon, a long-standing member of the House of Commons’ Public Accounts Committee, has called on Croydon Health Services NHS Trust to be more open about problems it faces after deploying a Cerner Millennium patient records system at the end of September.

The installation was carried out by BT under the London Programme for IT – a branch of the NPfIT.  The Health and Social Care Information Centre, which has taken on BT and CSC contracts under the NPfIT, was the trust’s partner for the Cerner deployment.

Bacon has closely followed the NPfIT and written a chapter on it in his book, “Conundrum: Why every government gets things wrong and what we can do about it” which he co-wrote with Christopher Hope, the Telegraph’s senior political correspondent.

According to fragments of information in Croydon Health Services’ latest board papers, dated 25 November 2013, the trust has faced a series of problems after the NPfIT Cerner go-live.

They included:

–  N3 Network downtime and waiting time breaches.

 Excessive waits for patients in A&E

 Going over budget.

– Significant loss of income.

 A bid to recover Cerner costs.

– A need for HSCIC support for delays. 

-A need for extra investment in Cerner to “stabilise the operational position”

The trust has not published any specific report on the implementation’s problemsNow Bacon says it is “unacceptable for any trust not to disclose the problems it faces – and possibly patients face – after a major IT implementation such as Cerner”.

He adds:

“If these implementations go wrong they can affect the safety of patients.  We know this from some NPfIT deployments at other  trusts. For Croydon to say that board members have been kept informed of the potential risks of the Cerner implementation through the “Corporate Risk and Board Assurance Framework”  is not reassuring.

“This is putting a matter of importance in the small print. Indeed, for officials to brief board members on the potential risks, rather than actual events, is also of concern.

“Patients need to know that Croydon takes a duty of candour seriously. If the Trust cannot be open about its IT-related problems, how can we be sure it will be open about anything else to do with patient safety?”

Patient records go-live “success” – or a new NPfIT failure 

Patient records go-live “success” – or a new NPfIT failure?

By Tony Collins

John Goulston says the go-live of a new patient records system at his trust is a “success”.

He should know. He’s Chief Executive of Croydon Health Services NHS Trust. He’s also chair of the trust’s Informatics Programme Board which has taken charge of bringing Cerner Millennium to Croydon’s community health services and the local University Hospital, formerly the Mayday.

He was formerly Programme Director of the London Programme for IT at NHS London – a branch of the NPfIT.

In a report two weeks ago Goulston said the trust deployed the “largest number of clinical applications in a single implementation in the NHS”. Croydon went live with Cerner Millennium on 30 September and 1 October 2013.

Said Goulston in his report:

“Administrative functions do not engage clinicians; providing them with a suite of clinical functionality has been justified as each weekday approx. 1,000 staff are logged on and using the system. CHS [Croydon Health Services] has in Phase 1 deployed, in addition to patient administration, the largest number of clinical applications in a single implementation in the NHS England.”

BT helped install Millennium at Croydon under the National Programme for IT.  The trust’s spokesman says the Department of Health provided central funding, and the trust paid for implementation “overheads”.  The Health and Social Care Information Centre was the trust’s partner for the go-live.

The Centre is the successor for Connecting for Health. It has taken on CfH’s officials who continue to help run the NPfIT contracts with BT and  CSC.

Goulston said that Cerner and BT have paid tribute to the trust which installed Millennium in A&E, outpatients, secretarial support and cancer services, and elsewhere.

“Our partners Cerner, BT and Ideal have commented that the Trust has undertaken one of the most efficient roll-outs of the system they have worked on, with more users adopting the system more quickly and efficiently than other trusts … the success we have achieved to date is the result of the efforts of every single system user and all staff members,” said Goulston.

Best Cerner implementation yet?

Optimistic remarks about their launch of Cerner Millennium were also made in 2012 by executives at the Royal Berkshire NHS Foundation Trust.  Their optimism proved ill-judged.

Of the Millennium go-live at Royal Berkshire, trust executives said that it “had been considered to be the best implementation of Cerner Millennium yet and that despite staff misgivings, the project was progressing well”.   This positive message should be disseminated, they said.

Months later they told the Reading Chronicle of patient safety issues and a financial crisis arising from the Millennium implementation.

A Royal Berkshire governors Rebecca Corre was quoted as saying: “There is a patient safety issue when staff write down observations and then there is an hour before they can get it onto the computer. If it is an experienced nurse, they may pick up a problem, but others may not.”

Ed Donald, Chief Executive of Royal Berkshire was quoted as saying:

“Unfortunately, implementing the EPR [electronic patient record] system has at times been a difficult process and we acknowledge that we did not fully appreciate the challenges and resources required in a number of areas.”

Are executives and managers at Croydon Health Services NHS Trust  now similarly afflicted with an unjustified optimism about the success of their Cerner go-live?  

Past consequences of NPfIT go-lives hidden?

The Department of Health has claimed benefits for the NPfIT of £3.7bn to March 2012 but there have been trust-wide failures: thousands of patients have had their appointments, care or treatment delayed by difficulties arising from past implementations of patient record systems under the NPfIT.  For thousands of patients waiting time standards have been exceeded or “breached” because of disruption arising from troubled go-lives.

In nearly every case trusts made it difficult for the facts to come out publicly. Vague or unexplained fragments of information about the consequences of the NPfIT implementation appeared  in different board papers over several months. The facts only emerged after a journalistic investigation that required scrutiny of many board papers and follow-up questions to the trust’s press office.

So Campaign4Change investigated Croydon Health’s implementation of Cerner Millennium to see if the Francis report’s call for a “duty of candour” over mistakes and problems in the NHS have made any difference to the traditional fragmentation of facts after NPfIT go-lives of patient record systems.

The Francis report called for “openness, transparency and candour“.  Trusts were told not to hide sub-standard practices under the carpet. The health secretary Jeremy Hunt said it can be “disastrous” when bad news does not emerge quickly and the public are kept in the dark about poor care.

To my questions about the Cerner Millennium implementation Croydon trust’s spokesman always responded promptly and tried to be helpful. But it appears that trust executives have given him limited information about consequences of the go-live, and have preferred to indulge the “good news” NHS culture that Jeremy Hunt warned about.

On being asked what problems the trust has faced since the go-live the spokesman gave various answers that made no mention of the problems.

“All of our staff received training on the system, and we are continuing to offer our teams support as it is embedded.”

What of the problems arising from the implementation, and has the board been fully informed?

“Millennium has featured regularly on the Corporate Risk Register presented to each Part 1 Board meeting.   In addition, implementation has received detailed confidential consideration at Part 2 of Board meetings, (which is why you won’t find it in our public board papers).”

Given Francis’s call for duty of candour,  should the trust be more open about its problems?

“The initial roll out for CRS Millennium was introduced over three days at the Trust, with a phased approach.  We did this to ensure the system was working in each department, before introducing it in another area.

“We are monitoring waiting time performance and records management so we can identify any issues if they emerge. The system is still being introduced in some services and when this is completed we will be able to assess the overall programme,” said the spokesman.

Does Croydon’s unwillingness to give in its statements to me any details of problems indicate that the culture of a lack of transparency in the NHS will be hard to change, no matter how many times Jeremy Hunt talks about the need for candour when things go wrong?

The spokesman:

“I’d like to be clear about the Trust’s approach:

  • The Trust board has been cited on the roll out of CRS Millennium and any potential risks throughout the process.  As I previously noted, the board received an update in September.  The board meeting, which will take place on Monday of next week, will receive a further update from the Chief Executive.  The papers from this meeting will be published on our website and the meeting takes place in public;
  • A meeting chaired by the Chief Operating Officer has reviewed any operational matters arising on a daily basis.  This is an internal meeting for clinicians and managers which has informed the implementation process;
  • Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“These actions would suggest that the Trust has been transparent in its approach.  You are welcome to review the board papers when they are published.”

Serious problems now emerge

Croydon did indeed publish its board papers on 25 November 2013 – which is to its credit because not all NHS trusts publish timely board papers.

But it’s mostly in the small print of various board papers that details emerge of Millennium-related problems. The shortcomings are mentioned as individual items rather than in a single, detailed Cerner Millennium deployment report.  This leaves one to question whether trust directors have an overview of the seriousness of the difficulties arising from its implementation of a new patient records system.

These are some excerpts from deep inside Croydon’s latest board papers:

Breaches in waiting time standards

– “CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT providing CHS with the required scale of N3 access (>600 concurrent users and >1,600 users on any day – which is the largest network usage of any trust in England).”

– • “Hospital Based Pathways: The deployment of CRS Millennium was a particular challenge in the month across the multiple service areas within the Directorate of A&E, Surgery and Maternity.

• “Cancer & Core Functions: With the implementation of CRS Millennium, the open pathways part of RTT [referral to treatment – patient waiting times) may fail the standard – validation will be completed after the narrative for this report… “

Excessive waits in A&E

– “The main drivers adversely affecting the performance in the month [October 2013) for A&E were the deployment of CRS Millennium and the commencement of winter pressures due to the seasonality change.  A&E  4-Hour Total Time in Department Target: 95.00%. Actual: 91.57%.”

Over budget

“The Trust position as at October is an adverse variance of £4.1m. This is a significant deterioration on the Month 6 position. The movement is mainly due to a significant reduction in income mainly as a result of operating issues caused by the Cerner deployment (£0.9m)…  Actual £14.8 (£14.8)m; Budget £10.7m; Variance £4.1m.”

“Cerner Millennium: Plan YTD [year-to-date] £245,000; Actual YTD  £621,000;

Significant loss in income

“… A new patient administration system was deployed in the Trust on the 30th September and 1st October (Cerner Millennium). The deployment has resulted in significant loss in income in September and October £ 1.1m. Trust performance on Activity Planning Assumptions and Key Performance Indicators is substantially worse than plan …”

Extra costs

“Medical £412k and admin £148k agency levels continue to be high due to cover for vacancies, annual leave, sickness and release of staff for Cerner training. The Trust has also incurred additional costs associated with the Cerner deployment (£600k) including overtime payments to administration staff and training costs.”

Bid to recover Cerner costs?

“… The Trust is currently forecasting a deficit position of £17.8m, which is £3.3m off the plan submitted to the NHS Trust Development Authority. This is a £3m movement from the month 6 forecast and is as a result of operational issues caused by the Cerner deployment. The current projected impact is an additional costs £1.7m and a loss in activity £1.1m . An application is to be made to recover the additional cost/losses relating to the Cerner deployment [of £2.9m] …”

HSCIC support for delays

“Cerner Millennium – Revised implementation date to Sept 2013 (achieved) ,with resultant additional costs including additional PC requirements of £146k, specialist support services £300k, procurement costs £91k, data cleansing costs £200k.

“Health& Social Care Information Centre (HSCIC) has confirmed support for the delayed implementation will be provided, accounting treatment of support to be confirmed with Department of Health.”

More money to stabilise operational position?

“As a result of operational issues caused by the Cerner deployment , Income is significantly reduced in October. The forecast assumes that the Trust will resume normal operating levels from November and that an element of the income lost will be will be recovered in the latter part of the year. A business case is being submitted to the Trust Board for additional investment in Cerner to stabilise the operational position.

“If there are further operational issues due to the Cerner deployment then this will significantly impact on the year end forecast…”

Over-optimism?

Principal risk -reporting output from Cerner is not accurate or timely. Officer in charge: CEO. Before go-live risk scores: June 2013 – 16; July – 16; Aug  – 10; Sept – 10. After go-live risk score (for Oct): 20 [high risk of likelihood and consequences]

Principal risk – operational readiness following the implementation of Cerner. Officer in charge: COO.  Before go-live risk score 15. Post go-live: 20. Risk rating before go-live – Green. After go-live – Red.

Red risks

Corporate Risk Assurance Framework

Nine risks are reported as Red [two of which relate directly to Millennium]:

“… Reporting output from Cerner is not accurate or timely. Data migration was successful. However reliance on external provider as internal knowledge has not yet been fully gained. A data quality dashboard with exception reporting is in place.

“… Operational readiness following the implementation of Cerner CRS Millennium impact conveyed to Trust Development Authority e.g. ED [Emergency Department] reporting and cost overruns

Risk scores

– Failure of CRS millennium to deliver anticipated benefits – 12. Officer in charge: CEO

– Reporting output from Cerner is not accurate or timely – 20. Officer in charge: CEO

– Operational readiness following the implementation of Cerner – 20. Officer in charge: COO

Croydon’s trust’s response to problems

Said John Goulston, Croydon’s CEO, in his latest [November 2013] report to the board of directors:

“The issues being encountered now with CRS Millennium are not due to any lack of integration testing with legacy applications or testing of workflow. They can be attributed to changing from a 25 year old Patient Administration System (Patient Centre) which did not require working in real time, was simple and intuitive to use, easily configurable and flexible to our needs.

“CRS Millennium’s patient administration functions are almost the complete opposite and the language used is new for our staff i.e. conversations, encounters etc. For our staff it has been a big ask for them to step into and up to such a complex application.”

He added: “The benefits of the new system are that each patient will have a single accurate electronic record that can be viewed and kept up-to-date by hospital and community clinical staff. This will eventually mean less time searching for patient notes, missing documentation and duplicating patient information…

“As with any massive change, there are still some challenges to tackle in making the system work effectively for every single user, in a diverse and complex organisation.

“However the success we have achieved to date is the result of the efforts of every single system user and all staff members. I would like to thank all our staff for their hard work in getting the Trust to this important stage.”

The trust spokesman gave me this statement on the problems:

“The Trust board has been given regular reports on the roll out of CRS Millennium and any potential risks throughout the process, not least through its regular reviews of the Corporate Risk and Board Assurance Frameworks.  As I previously noted, the board received a specific update in September.

“As you already know, November’s board meeting received a further update from the Chief Executive.  The papers from this meeting were published and the meeting takes place in public;  Those attending are invited to put forward questions.

“A meeting chaired by the Chief Operating Officer continues to review operational matters.  This is an internal meeting for clinicians and managers which has informed the implementation process;

“Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“As you highlight from the board report, Cerner & BT noted that ‘the Trust has undertaken one of the most efficient roll-outs of the system they have worked on’   The papers also note some operational challenges as the system was rolled out.  These have been addressed as part of the daily meetings I reference above – these are mainly concerned with users familiarising themselves with the system and have been addressed through the support and training staff received.

“In terms of the costs, the introduction of CRS Millennium has been supported by central funding from the Department of Health with the Trust paying the implementation overheads.   These costs are a matter of public record and the Trust publishes annual Accounts as part of its Annual Report.”

Comment

When you go into hospital it’s reassuring to know the directors will be well informed and open about problems that could affect you.

The approach of Croydon Health Services NHS Trust to openness about its problems is not reassuring. It is no better or worse than other trusts that have implemented Cerner’s Millennium. In fact the timely publication of its board papers means it is more open than some.

But it should not require a time-consuming journalistic investigation to establish the consequences for patients of an NPfIT go-live. It has required just such an investigation after the go-live of Millennium at Croydon.

Board directors will not have the time to dig for, and piece together, information about internal problems that could delay patient appointments, treatment and care. They need the unpalatable facts in one place. Croydon Health Services has failed to make it easy for patients or board directors to see what has gone wrong.

NPfIT deployments at other trusts have led, cumulatively, to thousands of patients having appointments that were disrupted, or who had to wait longer to be seen than necessary, or whose records were not available, or who were seen with another patient’s records.

In shying away from telling the whole truth trusts take their cue from the top: the Department of Health has always made it hard to establish facts about anything to do with the NPfIT.  Said the Public Accounts Committee in its report The National Programme for IT in the NHS: an update on the delivery of detailed care records systems in July 2011:

 “It is unacceptable that the Department [of Health] has neglected its duty to provide timely and reliable information to make possible Parliament’s scrutiny of this project.

“Basic information provided by the Department to the National Audit Office was late, inconsistent and contradictory.”

Unanswered questions

Croydon has questions to answer, such as how many breaches of waiting time standards it has had, and may still be having, due to problems arising from the go-live. Other unanswered questions:

– What does a “a large number of breaches” in the Emergency Department mean? Have each the patients affected been told?

– Why are the risks related to the implementation much higher after go-live than before, given that the trust has had years to prepare for the go-live, and the many lessons it could have learned from other trusts?

– Exactly what problems are still affecting patients?

In a post-Francis NHS, Jeremy Hunt has demanded openness about mistakes and problems. There is an agreed need for change – but how can Hunt change an NHS culture – indeed a public sector culture – in which senior executives, in troubled IT implementations, will always emphasise the good news over the bad, perhaps hoping the bad will always remain hidden?

More IT-based megaprojects derail amid claims all is well

By Tony Collins

If one thing unites all failing IT-based megaprojects in the public sector it is the defensive shield of denial that suppliers and their clients hold up when confronted by bad news.

It has happened in the US and UK this week. On the Universal Credit  project, the minister in charge of the scheme, Lord Freud, accepted none of the criticisms in a National Audit Office report “Universal Credit: early progress”.   In a debate in the House of Lords Lord Freud quoted from two tiny parts of the NAO report that could be interpreted as positive comments.

“Spending so far is a small proportion of the total budget … and it is still entirely feasible that [universal credit] goes on to achieve considerable benefits for society,” said Lord Freud, quoting the NAO report.

But he mentioned none of the criticisms in the 55-page NAO report which concluded:

“At this early stage of the Universal Credit programme the Department has not achieved value for money. The Department has delayed rolling out Universal Credit to claimants, has had weak control of the programme, and has been unable to assess the value of the systems it spent over £300 million to develop.

“These problems represent a significant setback to Universal Credit and raise wider concerns about the Department’s ability to deal with weak programme management, over-optimistic timescales, and a lack of openness about progress.”

And a shield of denial went up in the US this week where newspapers on the east and west coast published stories on failing public sector IT-based megaprojects.  The LA [Los Angeles] Times said:

As many as 300,000 jobless affected by state software snags

“California lawmakers want to know why Deloitte’s unemployment benefits system arrived with major bugs and at almost double the cost estimate. The firm says the system is working.”

The LA Times continued:

“Problems are growing worse for the state’s Employment Development Department after a new computer system backfired, leaving some Californians without much-needed benefit cheques for weeks.”

The Department said the problems affected 80,000 claims but the LA Times obtained internal emails that showed the software glitches stopped payment to as many as 300,000 claimants.

Now lawmakers are setting up a hearing to determine what went wrong with a system that cost taxpayers $110m, almost double the original estimate.

Some blame the Department’s slow response to the problems. Others point the finger at a Deloitte Consulting.

The LA Times says that Deloitte has a “history of delivering projects over budget and with problematic results”. Deloitte also has been blamed, in part, for similar troubles with upgrades to unemployment software in Massachusetts, Pennsylvania and Florida, says the paper.

“We keep hiring the same company, and they keep having the same issues,” said Senator Anthony Cannella.  “At some point, it’s on us for hiring the same company. It’s faulty logic, and we’ve got to get better.”

In 2003 California planned to spend $58m upgrading its 30-year-old unemployment benefits system. By the time the state awarded Deloitte the contract in 2010  the cost estimate had grown by more than $30m.

The Department handed out $6.6bn to about 1 million unemployed Californians in 2012. The software was expected to ease the agency’s ability to verify who was eligible to receive benefits.

Problems began when the Department transferred old unemployment data to the new system. The software flagged claims for review — requiring state workers to manually process them.

The LA Times says that officials thought initially the workload would be manageable, but internal emails showed the agency was quickly overwhelmed. Phone lines were jammed. For weeks, the Department’s employees have been working overtime to clear the backlog.

A poor contract?

In a contract amendment signed two months ago California agreed to pay Deloitte $3.5m for five months of maintenance and operations costs. Those costs should have been anticipated in the contract said Michael Krigsman, a software consultant who is an expert on why big IT-based contracts go awry. He told the LA Times:

“It’s a striking oversight that maintenance was not anticipated at the beginning of the contract when the state was at a much stronger negotiation position.”

By the time the middle of a project is reached, the state has no choice but to stick with Deloitte to work out bugs that arise when the system goes live, he said.

System works

Loree Levy, a spokeswoman for the Department, said the system is working, processing 80% of claims on time. As for the troubles, she said, “There is a period of transition or adjustment with any large infrastructure upgrade like this one.”

Deloitte spokeswoman Courtney Flaherty said the new California system is working and that problems are not the result of a “breakdown or flaw in the software Deloitte developed”.

System not working?

While there seems to be no project disaster in the eyes of the Department and Deloitte Consulting, some of the unemployed see things differently. One wrote:

“I am a contract worker who had to fight for my unemployment benefits. I won my case and yet they still cannot pay me… It’s been more than 3 weeks since I won my appeal and as of this moment, I am owed 13 weeks of back payments. To add insult to injury, they cannot send me current weeks to certify and they refuse to even try to help me to get back into the online system.

“I blame Deloitte, but it is California that carries the heaviest burden of fault… We’re nearing November and they still haven’t fixed an issue that began over Labor Day? Nonsense!

“This is untenable for everyone affected …We are owed reparations as well as our money at this point. It’s a funny word, affected. That means families and individuals are going hungry but can’t get food stamps or welfare. It means evictions and repossessed cars. It means destroyed credit, late fees, years of turmoil and shame for people already dealing with unemployment. Shame on you California.”

Another wrote:

“ … Not communicating is NOT an answer. Unemployed individuals caught up in the nightmare were told to be patient.  Rents and other expenses were still accumulating.  But [when you] add on additional fees: late fees, restoral fees, interest fees, etc…….you get the picture.

“Dear Governor Brown,

“Please reimburse me for all additional fees I’ve had to absorb to survive this fiasco.  You are going to make me payback any overpayments, but ignore the cost to the unemployed taxpayer.  This is  appears to unfair.  Perhaps Deloitte should pay us back from their contracted funds before they receive their final payment.  I am saving all of my receipts to deduct from my 2013 tax return.

“BTW Gov Brown – I am still waiting on additional payments as of today and DMV registration for my vehicle was due on 10/20/13.  Are you going to waive the penalty for late payment? Am I the only one with this question?”

Scrutiny

California’s state Assembly has set a date of 6 November 2013 for a hearing into the Department’s system upgrade.

“We’re going to look at EDD, the contractors and others to see how the system broke down so we can avoid this in the future,” said Henry Perea, chair of the Assembly’s Insurance Committee, which has oversight over the jobless benefits program.

On its website Deloitte says:

“Deloitte continues to help EDD [Employment Development Department] transform the level of service it provides to unemployed workers and improve the quality of information collected by EDD. The next time unemployment spikes, California should be ready to meet the increased demand for services.”

Massachutsetts IT disaster?

On the opposite coast the Boston Globe reported on an entirely separate debacle (which also involved Deloitte):

          None admit fault on troubled jobless benefits system

“… even with the possibility that unemployed workers could face months more of difficulties and delays in getting benefits, officials from the Labor Department and contractor, Deloitte Consulting of New York, testified before the Senate Committee on Post Audit that the rollout of the computer system was largely a success.

“‘I am happy with the launch,’ said Joanne F. Goldstein, secretary of Labour and Workforce Development, noting that she would have liked some aspects to have gone better.

“Mark Price, a Deloitte principal in charge of the firm’s Massachusetts business, acknowledged that software has faced challenges during the rollout, but insisted, ‘We have a successful working system today. ‘’’

NPfIT shield

A shield of denial was up for years at the Department of Health whose CIOs and other spokespeople repeatedly claimed that the NPfIT was a success.

Comment

If you didn’t know that Universal Credit IT wasn’t working, or that thousands of people on the east and west coasts of the US hadn’t been paid unemployment benefits because of IT-related problems, and you had to rely on only the public comments of the IT suppliers and government spokespeople, you would have every reason to believe that Universal Credit and the jobless systems in Massachusetts and California were working well.

Why is it that after every failed IT-based megaproject those in charge can simply blow the truth gently away like soap bubbles?

When confronted by bad news, suppliers and their customers tend to join hands behind their defensive shields. On the other side are politicians, members of the public affected by the megaprojects and the press who have all, according to suppliers and officials, got it wrong.

Is this why lessons from public sector IT-based project disasters are not always learned? Because, in the eyes of suppliers and their clients, the disasters don’t really exist?

None admit fault on troubled jobless benefit system

State fired Deloitte

Complaints continue despite claims system is under control

As many as 300,000 affected by California’s software problems

California’s predictable fiasco?

Are Whitehall IT business cases largely fictional?

By Tony Collins

Today’s report on the e-Borders programme by John Vine, the Chief Inspector of Borders and Immigration, is a reminder that central government business cases for major IT-based projects can be largely fictional.

Says the Vine report:

“The failure to identify these risks in the 2007 business plan meant that the original data collection targets, set out in the e-Borders delivery plan, were unrealistic and were always likely to be missed.”

It adds:

“The e-Borders programme business case indicated that e-Borders would allow foreign national passengers to be counted in and counted out of the UK, providing more reliable data for the purposes of migration and population statistics, and in planning the provision of public services. However, we found that the data set collected by e-Borders was not extensive enough for these purposes.”

And:

“Management information shows that between January and September 2012, 2,200 arrests took place as a direct result of the identification of wanted persons. This was less than the original estimate provided in the 2007 business case, which had anticipated 8,200 arrests per year based on the Semaphore pilot.”

One Whitehall insider said that experts are employed to write business cases to a template.  But do any of the promises in the business cases have to be fulfilled? It seems not.  Do business cases have to be realistic? The history of IT-based projects and programmes in central government shows that they don’t have to be.  

Business cases make promises on targets, any savings and costs.  When the targets in the business prove unachievable a new business case is written, and when the revised targets also prove unachievable another is written and so forth.

By the time assumptions in the business case have been properly tested the writers of the business cases are likely to have moved to other departments. Nobody is ever held responsible for writing a business case that proves to have been fictional. And why should they be? The writers of the business cases are in no way responsible for delivering the results.

The National Programme for IT in the NHS – NPfIT –had so many revised business cases nobody counted them.  Perhaps officials at the Department of Health knew they were largely fictional or, to put it more politely, aspirational. But the Treasury requires tick-box business cases to be written to justify money allocated to a project. Is there any point in a business case that’s not realistic? Perhaps. It allows money to be spent on a project that, based on realistic assumptions, would probably not be approved.

Below are the results of the e-Borders business case of 2007. Most of the promises haven’t been fulfilled.

The e-Borders system was based on Project Semaphore which was delivered by IBM in 2004 and it’s clear from the Vine report that the system  has been a success. Project Semaphore is still used because its replacement, which was commissioned in 2007, has been a standard government IT-based disaster with suppliers claiming that government kept changing its mind and the requirements, and the government saying milestones were not met.  In July 2010 the e-borders contract with “Trusted Borders” was terminated.

Vine’s report today,  Exporting the border’? An inspection of e-Borders October 2012 – March 2013, has a table (figure 18) that shows how much the Border Force has been able to meet the promises in the 2007 business case for the e-borders programme:  

1. Improved security by supporting the security and intelligence agencies to track and analyse the activities of terrorists and other national security targets across the border. Delivered? Partially.

2. Increased ability to identify and arrest those of interest to the police. Delivered? Yes.

3. Improved effectiveness and efficiency of border control activity by providing a risk assessment of passengers, facilitating expedited processing of passengers at the border and providing a platform for automated clearance services. Delivered? No.

4. Benefits will accrue from process cost savings as a result of the phasing
out of landing cards and the ability to access electronic movement
records when determining applications for extensions of stay. Delivered? No.

5. Enable the identification of those involved in excise duty avoidance and
impact on the market penetration of smuggled goods. Delivered? Partially.

6. Enable HMRC and DWP to establish the length of time spent in the
UK by an individual permitting easy identification of benefit claimants
living outside the UK and those falsely claiming non domicile status for
income tax purposes. Delivered? No.

7. Benefits to ports and carriers such as:
• reductions in removal and detention costs of those refused entry
(subject to implementation of an authority to carry scheme);
• more effective use of detention space at ports, provided free of
rent to control agencies; and
• remove requirement to procure and administer landing cards.

Delivered? No.

8. The ability to count all foreign national passengers into and out of the
UK enabling the provision of accurate statistical data to support the
provision of services. Delivered? No.

**

The Home Office is now writing a further business case for a new e-Borders programme, and will appoint a new IT supplier. Are its business case  authors expecting their work to be published under fiction or non-fiction? History, it seems, will provide the answer.

[The Home Office said its e-Borders technology was the most advanced in Europe – which says much for the 2004 IBM Semaphore system.]

John Vine’s report.

JohnVine “surprised” by findings

Croydon trust plans “high-risk” Cerner go-live in secret

By Tony Collins

NHS trusts have gone live with Cerner Millennium with mixed success. Eight trust implementations went seriously awry. A list is at the end of this post.

The flawed go-lives have meant that hospital managers have lost track, cumulatively, of thousands of patients and found that treatments,  including those for cancer, were delayed or care pathways interrupted. At times medical notes have not been available, or clinicians have been given the wrong notes.

In July 2008 E-Health Insider reported that the deployment of a national programme care records system at Milton Keynes Hospital NHS Foundation Trust “developed into an untenable situation which resulted in near melt down of the organisation”.

One of the lessons from the problematic go-lives is that trusts, when they have reported on the difficulties later, have said they underestimated the risks.

In particular they regarded the patient records system as mainly administrative with no risks to the health and care of patients.  Yet it is possible for incidents arising from flawed patient record implementations to be judged “clinical”.

Is Croydon Health Services NHS Trust, which includes the former Mayday hospital, about to repeat this same mistake – of regarding the risks as non-clinical?

The Trust is due to go live this weekend with the start of one of the largest patient record go-lives in the UK.  The deployment is being run under the NPfIT, which, in the capital, is now called the London Programme for IT. The plans are to install Millennium at Croydon University Hospital (formerly Mayday) and at some community sites.

The Trust says its preparations are designed to ensure a safe and effective deployment that will replace systems that are more than 20 years old and lack “many of the functions one expects in the modern healthcare digital age”.

But the signs are, from the trust’s board papers, that the risks of a flawed implementation are being seen largely as financial and administrative. For patients it appears that the worst that is envisaged is a “poor experience”.  

Have the risks to patients been properly flagged to the board’s directors?

I looked through Croydon’s most recently published board papers to see how well the trust’s directors have been kept informed of the risks of the go-live and could see almost no mention of Cerner – except a risk that income to the Trust could be affected if it is unable to produce timely reports. There is no specific mention of risks to patients.

Given the history of failed implementations of Cerner Millennium under the NPfIT, I asked  Croydon Health NHS Trust what directors have been told about the risks.

This was the trust’s reply:

“CRS Millennium has featured regularly on the Corporate Risk Register presented to each Part 1 Board meeting.

“In addition, implementation has received detailed confidential consideration at Part 2 of Board meetings, (which is why you won’t find it in our public board papers).”

I then put it to the trust that I was not sure why, in a new era of openness and transparency in the NHS,  that board discussions should be in private on a matter that could affect large numbers of patients to judge from past implementations at other trusts. The spokesperson – the interim stakeholder relations manager – made no further comment.

Richard Granger,  when head of the NPfIT, was quoted as saying he was ashamed of some of the Cerner deployments . Granger quit the programme in January 2008 – and since then several more Cerner deployments at trusts have gone badly wrong, at North Bristol NHS Trust, for example.

After the NPfIT go-live of Cerner at the Nuffield Orthopaedic Centre, the National Audit Office, at the request of a member of the Public Accounts Committee Conservative MP Richard Bacon, investigated and produced a public report on the lessons learned.

The Nuffield go-live was in December 2005. Since then managers at all trusts that have gone live with Cerner have claimed they have learned those lessons.

Comment:

It’s true that Croydon Trust’s risk register has shown “red” for the Cerner implementation. It is indeed deemed “high risk”. But how many of its board directors will understand the nature and substance of the risks to patients from a graphic?

It’s easy for the trust to say that board directors have been informed through confidential discussions but how would anyone outside the trust be able to judge whether that’s true?

Trust boards can never be held fully accountable after a patient record go-live goes wrong. They hold their own investigations or sometimes commission inquiries by consultants known to them, and the final reports usually indicate that no patients have been harmed.  Even after a trust has lost track of thousands of patient appointments for more than a year, it has later reported that in the end all was well.

As trust boards are not voted in and out by the public they have a special duty to ensure they are fully informed on things that can go wrong. Is Croydon Health’s board fully informed on the risks of the Cerner go-live? I doubt it.  

Some flawed NHS patient record go-lives:

Barts and The London

Royal Free Hampstead

Weston Area Health Trust

Milton Keynes Hospital NHS Trust

Worthing and Southlands

Barnet and Chase Farm Hospitals NHS Trust

Nuffield Orthopaedic

North Bristol.

St George’s Healthcare NHS Trust

Lorenzo:

University Hospitals of Morecambe Bay NHS Foundation Trust

Birmingham Women’s Foundation Trust

NHS Bury

NPfIT central costs rise by tens of millions – even after “dismantling”

By Tony Collins

On 22 September 2011 the Department of Health announced the dismantling of the NPfIT. As the press release was being issued some officials at the department were aware that they were continuing to spend tens of millions on central administrative costs of the programme.

Today’s report of the Public Accounts Committee has a figure for the central costs of the NPfIT until the end of March 2012 of about £890m. Before the DH announced the dismantling of the programme, in March 2011, the DH put the central costs at £817m.

So there has been a rise in central admin costs of about £70m since the NPfIT was supposedly dismantled.

The administrative costs are separate from spending on the contracts with BT or CSC. The admin costs don’t include the delivery of a single laptop to the NHS under the NPfIT. They are simply the central costs of administering the programme – including day rates for consultants – such as day rates of £1,700 to help senior officials prepare for appearances before MPs on the Public Accounts Committee.

The central costs have never been explained, not even by the National Audit Office which has published several reports on the NPfIT.  It is known that some central costs are explained by items of questionable benefit such as the commissioning of DVD films that marketed the NPfIT.

Some of the cost categories have emerged as a result of an FOI request (below).  Officials made regular visits to various parts of the globe to promote the success of the NPfIT. It’s thought that the DH has spent more than £100m on consultants for the programme.

Millions of pounds have been spent with public relations companies. The DH spent about £30,000 on press cuttings in two years alone.  Released central costs for just two years of the NPfIT between 2005 and 2007 include:

  • £1.23m with Expotel Hotel Reservations
  • £1.87 Harry Weeks Business Travel
  • BT conferencing – £1.15m
  • Intercall video conferencing – £274,973
  • MWB (Serviced Offices) – £15.8m
  • Regus – offices and meeting rooms – £3.17m
  • Spring International Express (courier and other services) – £192, 662
  • Cision UK (press cuttings) – £30,000
  • Fishburn Hedges (includes public relations) – £559,310
  • Good Relations (public relations] – £1.55m
  • Porter Novelli (public relations and information) – £943,000
  • ASE Consulting – £31.7m
  • Capgemini – £15m
  • Deloitte MCS – £42.8m
  • Atos Consulting – £32.3m
  • Gartner – £3.8m
  • QI Consulting – £14.5m
  • Tribal Consulting – £6.9m

Comment

Central administrative costs of nearly £900m on a single IT programme are breathtaking. That makes the National Programme for IT in the NHS one of the world’s largest public sector IT projects – before a penny has been spent on deliveries of hardware or software to the NHS.

It’s almost as surprising that not even the National Audit Office has been able to obtain a breakdown. Has central spending been properly controlled? Perhaps not, given that the DH, even this year, spent up to £1,700 a day on consultants to brief a senior official for a hearing of the Public Accounts Committee in June 2013.

Maybe the taxpayer should be grateful that the consultants were hired for only 52 days between February and June 2013 to prepare for the Committee’s hearing, and that the DH managed to renegotiate the day rate down from £1,714 to £1,000 a day between April and June.

Maybe the taxpayer should be grateful that the total cost of the consultancy for preparing for the PAC hearing was only £73,563.

But the £73,563 was spent after the DH estimated its central administrative costs on the NPfIT at nearly £900m – which are costs up to 31 March 2012.

It’s also remarkable that some at the DH still consider the NPfIT a success. This was the NAO’s conclusions on the NPfIT in its May 2011 report on the NPfIT Care Records Service:

“Central to achieving the Programme’s aim of improving services and the quality of patient care, was the successful delivery of an electronic patient record for each NHS patient. Although some care records systems are in place, progress against plans has fallen far below expectations and the Department has not delivered care records systems across the NHS, or with anywhere near the completeness of functionality that will enable it to achieve the original aspirations of the Programme.

“The Department has also significantly reduced the scope of the Programme without a proportionate reduction in costs, and is in negotiations to reduce it further still. So we are seeing a steady reduction in value delivered not matched by a reduction in costs.

“On this basis we conclude that the £2.7 billion spent on care records systems so far does not represent value for money, and we do not find grounds for confidence that the remaining planned spend of £4.3 billion will be different.”

But this was the Department of Health’s view on NPfIT Care Records Service value for money:

“The Department considers, however, that the money spent to date has not been  wasted and will potentially deliver value for money… The Department believes that the flexibility provided by the future delivery model for the programme will deliver functionality that best fits the needs of the clinical and managerial community. The future architecture of the programme allows many sources of information to be connected together as opposed to assuming that all relevant information will be stored in a single system. This approach has been proven in other sectors and is fully consistent with the Government’s recently published ICT strategy.”

This contradiction between the DH’s view of the NPfIT, and the NAO’s, indicates, perhaps, that the DH continues to live in a world not entirely attached to reality.

From April 2013, the DH’s central team and some local programme teams responsible for the NPfIT moved to the Health and Social Care Information Centre which has taken over the local service provider contracts with BT and CSC. Will it be able to control central spending on the very-much-alive NPfIT?

Update:

The central costs could rise much further – possibly by more than £100m – if the eventual settlement of the legal case between the DH and Fujitsu works out badly for the taxpayer. Legal costs on the case so far are about £31m.