Category Archives: IT-related failures

More public sector IT-related failures for which nobody will be accountable – a solution?

The Times front page – 23 January 2019

By Tony Collins

Criminal trials were delayed, jurors unable to enrol and witness statements inaccessible.

Quoting a tweet by the authoritative @BarristerSecret, the BBC said the “entire digital infrastructure” of courts was “broken for days”.

@BarristerSecret added,

“No accountability, no lessons learned.”

In the Spectator, Matthew Scott, a criminal barrister at Pump Court Chambers, said,

“Nobody seems to know exactly what has gone wrong or, if they do, they do not like to say.”

His Spectator blog was headlined,

The Spectator – 24 Jan 2019

 

 

“The most irritating fault has been for a few days the near total seizure (or ‘major service degradation’ to use the official non-explanation) of the secure email system (‘CJSM’) which for several years now has been the only authorised means of written communication between the Crown Prosecution Service and defence lawyers, probation, prisons, police and others.”

The Law Society Gazette said,

Law Society Gazette – 22 Jan 2019

 

 

 

The Law Society Gazette gave examples of how the problems had caused disruption and angst in the criminal justice system. It said,

“Major disruption that affected multiple Ministry of Justice IT systems last week continues to cause chaos.

“Lawyers on the front line have told the Gazette that trials have been delayed, jurors have been unable to enrol and practitioners have been prevented from confirming attendance that will enable them to get paid.

“Last week the ministry’s digital and technology team said most systems were improving. However, the Gazette has spoken to practitioners whose experiences suggest otherwise.”

A criminal barrister who spent the day in Leicester Crown Court said  none of the court’s computer systems was operational, jurors could not be enrolled, and no advocates could sign into the Ministry of Justice’s XHIBIT system, an online service that logs lawyers’ attendance so they can get paid.

A lawyer at Lincoln Crown Court said the XHIBIT system was down again. The Crown Court Digital Case System, on which all cases are accessed, was also down.

A criminal defence solicitor arrived at Highbury Magistrates’ Court in London at 9.15am, where there were several clients in the cells. But jailers did not know which courts the cases would be heard in and  because there was no wi-fi in the building magistrates had no access to any papers on their ipads before the hearings.

“The Gazette was told that several people attended Scarborough Magistrates’ Court last week to make statutory declarations in respect of driving matters. ‘Most of these people had come suited and booted, with all the anxiety that marks ordinary members of the public out as different from the frequent flyers who regularly come before the courts.

“These poor souls were left hanging around all morning, until 1pm, when they were advised that the systems were still not back up. Two of them agreed to come back on an adjourned date, 14 days later, but one of them explained that he couldn’t take further time off work. He was asked to come back in the afternoon, in the vain hope that the case management system might be back online.”

Former government chief technology officer Andy Beale quoted The Times in a tweet,

 

 

 

In another tweet, Beale said,

 

 

 

The Guardian reported yesterday (28 January 2019) that the Ministry of Justice knew its court computer systems were “obsolete” and “out of support” long before the network went into meltdown, internal documents have revealed.

The MoJ document, entitled Digital & Technology, said, “Historical under-investment in ageing IT systems has built our technical debt to unacceptable levels and we are carrying significant risk that will result in a large-scale data breach if the vulnerabilities are exploited.”

It added, “We have a Technology 2022 strategy, but it is not funded to help us address the long-term issues with current systems and allow us to make best use of new technologies to improve service delivery.”

It referred to a database used by 16 employment tribunal administrative offices in which the “scale of outage” accounted for 33% of incidents over the previous six months. Users were unable to access systems for a “significant number of hours”.

The report cited problems such as “risk of database corrupted leading to data loss; unable to restore service in a timely manner”, and added: “Judges say they will put tribunal activity on hold because of the poor running of the application.”

Government response

In the Commons, the government’s justice minister Lucy Frazer, responding to an urgent Labour request for a statement on the IT problems, was relaxed in her comments. She said the disruption was “intermittent” and the problems were merely “frustrating”. She added,

“The issue that has arisen relates mainly to email systems. There has been minimal disruption, I am told, to the courts system as a whole.”

She said there had been an “infrastructure failure in our supplier’s data centre”.

“The Prison Service has not been affected and—to correct inaccurate reporting—criminals have not gone free as a result of the problem. We have been working closely with our suppliers, Atos and Microsoft, to get our systems working again, and yesterday we had restored services to 180 court sites, including the largest ones.

“Today (23 January 2019), 90% of staff have working computer systems. Work continues to restore services and we expect the remainder of the court sites to be fully operational by the time they open tomorrow morning. We are very disappointed that our suppliers have not yet been able to resolve the network problems in full.

“This afternoon, the permanent secretary, Sir Richard Heaton, will meet the chief executive of Atos and write personally to all members of the judiciary. I am very grateful to all our staff who have been working tirelessly and around the clock, alongside our suppliers, to resolve the issues.”

Labour’s Yasmin Qureshi asked if Microsoft and Atos have paid any penalties to which Frazer gave a vague, non-committal reply,

“… the permanent secretary is meeting the supplier’s chief executive this afternoon and of course we will look carefully at the contracts, which include penalty clauses.”

Frazer later said the problem related to a “server” which raised questions about how the failure of a single server, or servers, could cause widespread chaos in the courts.

Labour’s Steve McCabe said the server problem was not a  single or unusual event.

“… her Department has been receiving reports of failures in the criminal justice secure email service for at least six months now”.

Police systems

The BBC reported last week that problems with a police IT system were causing some criminals to escape justice.

Nine forces in England and Wales use Athena from Northgate Public Services. They are Bedfordshire, Cambridgeshire, Essex, Hertfordshire, Kent, Norfolk, Suffolk, Warwickshire and West Mercia. The system is designed to help speed up the detection of crimes.

But officers told the BBC’s Victoria Derbyshire programme that it crashes regularly and is overly complicated, meaning some cases are not built in time or dropped.

Developers Northgate Public Services apologised for problems “in small areas”, which it said it was fixing.

A joint response from nine police forces said Athena – which has cost £35m over the past 10 years – had been “resilient and stable, although no system is perfect”.

The system was introduced following a government directive for forces to share intelligence after the Soham murders of Holly Wells and Jessica Chapman, in 2002.

Officers said the intelligence-sharing function works well but problems arise when they use the system to build cases for the Crown Prosecution Service.

The delays it causes means officers can struggle to get the information together in time to charge suspects or the cases are not up to a high-enough standard and are dropped.

Serving officers at Bedfordshire, Hertfordshire and Essex told the programme the process could now take up to twice as long.

The BBC did not name any officers who revealed details of the problems because they could face disciplinary action for speaking out. Their comments included:

  • “The first two weeks it (the system) was brought in were the worst two weeks of my entire career. It’s overly bureaucratic. It doesn’t understand the police investigative process at all. From day one, it malfunctioned. Four years on, it is still malfunctioning”
  • “It often requires information that is totally irrelevant and if you miss just one data entry point (like whether a solicitor is male or female), I have to reject the whole case and send it back to the officer”
  • “Even for a simple shoplift, I probably have to press about 50 buttons, with a 30-second minimum loading time between each task”
  • “There have been incidents where charges have been dropped because of the inadequacies of the system. There have been cases of assaults, albeit fairly minor assaults, but these are still people who should be facing criminal charges”
  • “It slows the whole criminal justice system down. At the moment, it is not fit for purpose. This is the most challenging time I have come across. We’re at breaking point already. This has pushed some officers over the edge”
  • “When you’ve got detainees in a custody block who’ve got various illnesses and ailments, medical conditions that are all recorded on there and they need medication at certain times – it became very dangerous because we were unable to access the records”

The nine forces – which also include those in Cambridgeshire, Kent, Norfolk, Suffolk, Warwickshire and West Mercia – said in a joint statement that they had been working with the supplier to identify and correct issues as they arose.

“Over the 12 months up to November 2018, there have only been 72 hours of total downtime and there are detailed plans in place of how to manage business when this occurs.”

Northgate Public Services, which created Athena, said 40,000 officers accessed the system and benefited from improved criminal intelligence.

It said it was working to make improvements to the “complex system”.

“We recognise there are a small number of areas of the solution where improvements can be made and we apologise for any difficulties this has caused.

“We are working hard with the customer and other parties to make these improvements as a priority.”

Comment:

As @BarristerSecret said,

“No accountability, no lessons learned.”

In central and local government, accountability means suppliers sometimes have to pay small penalties. Outsourcing supplier Capita last year paid Barnet Council about £4.2m in compensation for poor performance.

It was a fraction of the hundreds of millions Capita has received from Barnet Council.

Sometimes the opposite happens and it is the supplier that wins money from the government after a failure.

The Home Office sacked Raytheon over problems on an e-borders IT systems and ended up paying Raytheon £224m in compensation.

The Department of Heath ended up paying Fujitsu hundreds of millions of pounds after the supplier’s contract to deliver systems under the National Programme for IT [NPfIT] was ended.

A major failure in one area of the public sector will not  stop or deter officials from awarding the same supplier a major contract in the same or another part of the public sector.

Were a major failure or legal dispute to preclude a supplier from bidding for further UK public sector work, most if not all major suppliers would today have little UK government business.

A solution?

There is an effective way to encourage IT suppliers and the public sector to avoid public service failures. But the senior civil service isn’t interested.

That solution would be to publish – after every major public services failure – a full, independent third-party report into what went wrong and why.

Some senior officials seem unruffled by public criticism or even contempt after a services failure. But particularly in some of the major departments, there is a high-level fear of the full truth emerging after an administrative disaster.  Departments would do almost anything to avoid IT-related failures if reports on the causes were routinely published.

But unless there is a Parliamentary or public clamour for such internal analyses to be published, they will remain hidden or uncommissioned.

When the National Audit Office publishes a report on a departmental failure, the report has usually been agreed and signed off by the department; and it is usually a one-off report.

When public services descend into chaos, as happened in the court service last week, immense pressure falls on the IT teams to restore normal services urgently. But without the routine publication of reports on major IT-related public service failures, where is the motivation for senior officials to avoid chaos in the first place?

House of Commons debate on the courts’ IT failures

Thank you to Celina Bledowska for her tweet alerting me to the criminal justice IT problems.

Advertisements

Uupublished plan to throw another £13bn at the NHS’s IT problems?

By Tony Collins

The Health Service Journal yesterday revealed details of NHS IT investment plans that have been costed at about £12.9bn over the next five years.

The HSJ’s award-winning technology correspondent Ben Heather  says the sums currently involved – which could reduce as proposals are “reined in” – are on a par with the notorious National Programme for IT in the NHS.

He says that officials working on the plan have produced an estimate of between £10.9bn and £12.9bn for the cost of supporting proposals across 15 long-term plan “workstreams” ranging from creating personalised care to improving cancer survival.

The figures form part of the work of the digital and technology workstream for the long term plan, which is being developed by NHS England and NHS Improvement.

“The sum would be on par with the National Programme for IT, the most expensive push to improve IT systems in NHS history and an infamously costly and troubled project. It is likely to reduce substantially, however, as ambitions for the plan are negotiated and reined in over coming weeks.”

The plan is due to be published in late November or early December. The health secretary is known to be a keen advocate of new IT-related investments.

It is likely that a sizeable portion of the new £20bn planned for the NHS – which will be financed partly by tax increases that are due to be announced in the budget later this month – will go on NHS technology.

But the Health Service Journal suggests the investments will be controlled centrally, which may be a bad sign given that one of the major flaws in the failed £13bn NPfIT was that money was controlled centrally rather than by local groups of doctors and nurses.

Comment

On the face of it the current investment proposals bear no resemblance to the NHS IT programme NPfIT which was “dismantled” in 2011.

The NPfIT comprised a handful of specific major projects that were to be implemented nationally under the umbrella of “ruthless standardisation”.

The current proposals look very different. The investments fall into vague categories such as digitalising secondary care, improvements to IT infrastructure, data gathering and analytics.

The proposals have all the appearance of a different way the NHS has found to waste vast sums of public money.

It has never been acknowledged by the Treasury, NHS England or the Department of Health that the NPfIT wasted billions on spending that was invisible to the public, such as numerous consultants, years of globe-trotting by officials, first-class hotels across the world, sponsored conferences and unreported funds for marketing items that included DVDs and board games designed especially to promote the IT programme.

For officials, there’s nothing more exciting than going to work on a £13bn technology programme where money flows more freely than water. It’s no wonder officialdom is lobbying for the money.

No doubt it will be easy for officials to obtain the new billions. At any time in the recent history of the NHS it would have been easy on paper to justify £13bn for new NHS technology. Much of the £13bn could be justified simply enough by submitting plans to HM Treasury to modernise what already exists.

It was easy to justify the NPfIT. Tony Blair approved it at a Downing Street meeting that lasted 40 minutes. Computer Weekly obtained minutes of the Downing Street meeting after various FOI appeals.

But the NHS needs £13bn to be spent wisely on technology. The last thing the NHS needs is for Whitehall officials to be involved. History shows that Whitehall has the reverse Midas touch when it comes to major NHS IT investments. It is local groups of doctors and nurses who know how to spend the money wisely.

If either NHS England or the Department of Health and Social Care is involved in the new proposals for NHS IT investments – and they both are – it’s almost certain the new plans will end up as costly failures.

How would the public feel if they realised that a sizeable portion of their increased taxes for the NHS is almost certainly destined for the dustbin marked “mismanaged Whitehall IT schemes”.

Revealed: Officials’ £13bn funding ask to modernise NHS IT

Another NPfIT scandal in the making?

£20bn for the NHS? – not spent like this please

Johnathan Lewis, CEO Capita (right) and Simon Stevens, Chief Executive, NHS England (left) at Monday’s Public Accounts Committee.

By Tony Collins

Capita apologies for working “blind” on NHS outsourcing contract – but no humility from NHS England

Capita’s CEO Johnathan Lewis was contrite and authoritative when he appeared before public accounts MPs in the House of Commons on Monday.

He apologised unreservedly for what the committee chairwoman Meg Hillier called “a shambles”, which was Capita’s £330 seven to ten-year contract to run a range of services for GPs, dentists and ophthalmologists, as well as handle invitations and test results for cervical screening.

Capita’s Primary Care Support Services contract began in 2015 and complaints about the service from medical practitioners began to flow months later.

Capita made mistakes, said Lewis who was supported by his colleague Stephen Sharp, who reports directly to Lewis on public sector contracts. One mistake was that Capita tried to save money too soon by folding the work of 47 local NHS offices with 1650 staff into three offices without fully understanding that each office had a different way of working and a different way of delivering NHS services.

[A similar mistake helped to floor the £10bn National Programme for IT in the NHS (NPfIT), where suppliers and Whitehall officials tried unsuccessfully to use computers to standardise working practices and services in hundreds of hospitals before they fully understood the widely-different approaches of each hospital.]

Lewis told the Public Accounts Committee on Monday,

“This was an extremely complex outsourcing of services that I think both parties would recognise were not fully understood when the work was outsourced – the volumes, the scope, the fact that the service was being delivered in different ways across the different regions that became NHS England. At the same time I recognise the pressure NHS England were under to reduce costs and hence the pressure on them to outsource.”

His colleague Stephen Sharp added,

“I think mistakes were made. During the bid stage, NHS England did say there were some inconsistencies and differences within the various operations. But once Capita got into all the offices and looked at it, the inconsistencies and differences were not inconsequential. It was more or less 45 different services being run from 45 different offices, so the closure programme, which we adhered to and carried on with, we maybe should have stopped. We just made the problem worse as we went along.”

Why didn’t you stop the office closures, asked Conservative MP Anne Marie Morris who added that “even the NHS said, ‘We think you need to stop’.”

Sharp replied,

“We were actually working blind for a period of time. It was only once the service had been running under our control for a few months that complaints started to come in and we started to see visibility that there were bigger issues than we thought there were.”

With hindsight he said he would not have closed offices “until we had got the procedures operating on a national basis”. He conceded that if NHS England and Capita had deferred closing offices, the first two years of savings of about £60m would not have been achieved.

Capita’s losses of £140m

Lewis said that Capita had invested £125m in the contract but, given the loss of profit margin, the losses would be closer to £140m. “We will not make money over the life of this contract,” said Lewis.

An MP asked: why not walk away?

Lewis replied, “Because we made a commitment to deliver this service and reputations depend on that commitment. We see the public sector as a segment of our market that helps us achieve a diversified revenue base. It is a segment where we have services and solutions, where we can create value for the taxpayer and that is why it is an attractive segment.”

Capita is now meeting 41 of the 45 KPIs and, though the company is making good progress against the remaining four KPIs, it doesn’t change the fact that “our initial execution on this contract was not good and for that we apologise unreservedly,” said Lewis.

There were failings on the part of NHS England too. Health officials were so anxious to achieve the savings from closing offices and replacing old IT that couldn’t be relied on that they failed to test new national, standardised working practices and services before they asked a supplier to implement this strategy.

The result was that officials at NHS England had no clear idea of how much work they were outsourcing. They left due diligence to Capita; and Capita admitted at the hearing it did not do enough due diligence at the bid stage. If it had understood how much work was involved it would have bid a higher price or not bid at all.

NHS England also failed to involve most of the potential end-users – GPs, dentists and ophthalmologists in the design and planning of new services that would directly affect them such as pensions and payments.

Lewis said.

“There are other stakeholders that have historically not been brought into this process to the extent that they should have been, such as the BMA [British Medical Association] in how we might implement the digitisation of pension payments and the management of its pensions, or the Confederation of Dental Employers with regard to ophthalmic payments.

“We want to bring them into the process in ways that they have not been historically because we think that that will ultimately lead to a more successful roll out of the technology… They rightly have influence over the process. If we are going to roll out a process for digitising the 20,000 paper documents that cover the process by which you get refunded for an ophthalmic prescription today, surely those people need to be involved in the final roll-out and configuration of that solution.”

Absence of humility?

When MPs questioned the top official at NHS England, Simon Stevens, there was little sign of humility, contrition or regret. He left an impression that the same problems could end up being repeated by a different supplier under a different contract. One Conservative MP Bim Afolami found himself “sticking up for Capita”.

Afolami said,

“Do you feel, Mr Stevens, that criticism of this contract is in any way unfair on Capita? The more I hear, the more I feel that Capita has taken the sharp end of this and NHS England, despite slight reputational difficulty, has saved £60 million. To what extent do you feel that you should take more of the blame here and Capita should take less of it?”

Stevens emphasised the £60m savings but made no mention any of the contract’s specific problems such as the thousands of patient records that went missing, dozens of women left off cancer-screening lists, the qualified GPs who were unable to work for months while the system delayed verifying their entitlement to go onto a “National Performers List”, the GPs who ran short of basic supplies or the GPs and ophthalmologists who suffered financial detriment because of delayed payments.

Said Stevens,

“First, let me say that this has clearly been a rocky road, and the National Audit Office accurately described the bumps along the way, which are regrettable. That should not obscure the fact that, notwithstanding the economic pain that Capita has experienced, the contract has saved taxpayers £60 million in lower administrative costs in the National Health Service over the first two years of its life … that £60 million of savings is not to be sniffed at; it is the equivalent of 30,000 operations.”

Comment:

Campaign4Change has repeatedly criticised Capita’s performance on Barnet’s outsourcing contract, in part because Capita and the council have been markedly defensive – thin-skinned.

It was refreshing, therefore, to hear Capita’s newish CEO Jonathan Lewis being openly contrite over highly-visible failings in the NHS contract. He gave the impression to public accounts MPs of being a CEO who is determined to put right the failings for the sake of Capita’s reputation. The cost of correcting the problems seemed a secondary consideration.

With Lewis at the helm, Capita’s share price has continued to rise in recent weeks.

Less impressive at Monday’s hearing was Simon Stevens, NHS England’s chief executive, who seemed to imply that NHS England had done nothing wrong.  It was a reaction we’ve come to expect from top civil servants after an IT-related programme disaster. It’s never the fault of officialdom.

The reality is that NHS England was almost as culpable as Capita. NHS England rushed the whole outsourcing exercise – which doomed it from the start. It didn’t listen to critics who warned that primary care support services were too locally diverse and inherently problematic to standardise as part of a national  outsourcing deal.

Instead of first piloting and agreeing with GPs, dentists and ophthalmologists fundamental changes in working practices that would be needed across the country, NHS England went ahead with signing a co-called transformation deal with Capita.

NHS England paid only lip service to engagement with the new system’s end-users in the medical professions. By its own admission Capita, because of its own internal shortcomings, went into the contract blind.

What’s worrying is the way civil servants blithely repeat mistakes of the past and later say they did everything right.

The National Programme for IT in the NHS – NPfIT – failed in part because it was rushed, the implications of “ruthless standardisation” were not fully understood at the outset and there was a lack of proper engagement with potential end-users in hospitals and GP practices. All these same mistakes were made by Capita and NHS England on the Primary Care Support Services contract.

When ordinary human beings become senior civil servants there seems to be a requirement that they lose at a cellular level the facility to express humility and contrition. That loss is replaced by an overly prominent complacency. Whatever goes wrong is not their fault.

Stevens said in essence that NHS England did everything right. Through its unpublished project reviews, the Major Projects Authority – now the Infrastructure and Projects Authority –  endorsed NHS England’ s plans. All the so-called experts gave the outsourcing deal what Stevens called a “thumbs-up”.

It would have been surprising if Stevens had said the public sector was in any way to blame.

At least Capita has learned the lessons. It has a financial interest in doing so.

Ministers can learn from Capita’s candid chief executive

NHS England’s management of Primary Care Support Services contract with Capita – National Audit Office report

Monday’s televised Public Accounts Committee hearing with Capita’s Jonathan Lewis and Simon Stevens of NHS England

Ministers told of major problem on Capita NHS contract more than a year later

By Tony Collins

Today’s Financial Times and other newspapers cover a National Audit Office report into GP clinical notes and correspondence, some of it urgent, that was not directed to the patient’s GP.

The correspondence was archived by Capita under its contract to provide GP support services. But patient notes were still “live”. They included patient invitation letters, treatment/diagnosis notes, test results and documents/referrals marked ‘urgent’.

What isn’t well reported is that ministers were left in the dark about the problems for more than a year. The National Audit Office does not blame anyone – its remit does not include questioning policy decisions – but its report is impressive in setting out of the facts.

Before NHS England outsourced GP support services to Capita in 2015, GPs practices sent correspondence for patients that were not registered at their practice to local primary care services centres, which would attempt to redirect the mail.

By the time Capita took over GP support services on 1 September 2015, GPs were supposed to “return to sender” any correspondence that was sent to them incorrectly – and not send it to primary care services centres that were now run, in part, by Capita.

But some GPs continued to send incorrectly-addressed correspondence to the primary care services centres. Capita’s contract did not require it to redirect clinical correspondence.

An unknown number of GP practices continued to send mail to the centres, expecting the centre’s staff to redirect it. A further complication was that Capita had “transformation” plans to cut costs by closing the primary care services support centres.

Capita made an inventory of all records at each site and shared this with NHS England. The inventories made reference to ‘clinical notes’ but at this point no one identified these notes as live clinical correspondence. Capita stored the correspondence in its archive.

In line with its contract, Capita did not forward the mail. It was not until May 2016 – eight months after Capita took over the primary care services centres – that Capita told a member of NHS England’s primary care support team that there was a problem with an unquantified accumulation of clinical notes.

It was a further five months before Capita formally reported the incident to NHS England. At that time Capita estimated that there was an accumulation of hundreds of thousands of clinical notes. When the National Audit Office questioned Capita on the matter, it replied that, with hindsight, it believes it could have reported the backlog sooner.

In November 2016, Capita and NHS England carried out initial checks on the reported backlog of 580,000 clinical notes. It wasn’t until December 2016 that ministers were informed of problems – more than a year after Capita took over the contract.

Even in December 2016 ministers were not fully informed. Information about a backlog of live clinical notes was within in a number of items in the quarterly ministerial reports. NHS England did not report the matter to the Department of Health until April 2017 – about two years after the problems began.

Even then, officials told ministers that clinical notes had been sampled and were considered “low clinical and patient risk”. But a later study by NHS England’s National Incident Team identified a backlog of 1,811 high priority patient notes such as documents deemed to be related to screening or urgent test results.

The National Audit Office says, “NHS England expects to know by March 2018 whether there has been any harm to patients as a result of the delay in redirecting correspondence. NHS England will investigate further where GPs have identified that there could be potential harm to patients. The review will be led by NHS England’s national clinical directors, with consultant level input where required.”

Last month Richard Vautrey, chairman of British Medical Association’s General Practitioners Committee, wrote to the NHS Chief Executive Simon Stevens criticising a lack of substantial improvement on Capita’s contract to run primary care service centres.

In December, the GP Committee surveyed practices and individual GPs on the Capita contract. The results showed a little improvement across all service lines, when compared to its previous survey in October 2016, but a “significant deterioration” in some services. Vautrey’s letter said,

“While any new organisation takes time to take over services effectively, the situation has gone from bad to worse since Capita took over the PCSE [Primary Care Support England] service almost two and a half years ago …

“This situation is completely unacceptable. As a result of the lack of improvement in the service delivery of PCSE we are now left with no option but to support practices and individual doctors in taking legal routes to seek resolution. While this is taking place, we believe it is imperative that NHS England conducts a transparent and comprehensive review of all policy, procedures and processes used by PCSE across each service line.”

Comment:

It’ll be clear to some who read the NAO report that the problems with urgent patient notes going astray or being put mistakenly into storage, stems from NHS England’s decision to outsource a complex range of GP support services without fully considering – or caring about – what could go wrong.

It’s not yet known if patients have come to harm. It’s clear, though, that patients have been caught in the middle of a major administrative blunder that has complex causes and for which nobody in particular can be held responsible.

That ministers learned of a major failure on a public sector outsourcing deal over a year after live patient notes began to be archived is not surprising.

About four million civil and public servants have strict rules governing confidentiality. There are no requirements for civil and public service openness except when it comes to the Freedom of Information Act which many officials can – and do – easily circumvent.

Even today, the fourth year of Capita’s contract to run GP support services, the implications for patients of what has gone wrong are not yet fully known or understood.

It’s a familiar story: a public sector blunder for which nobody will take responsibility, for which nobody in particular seems to care about, and for which the preoccupation of officialdom will be to continue playing down the implications or not say anything at all.

Why would they be open when there is no effective requirement for it? It’s a truism that serious problems cannot be fixed until they are admitted. In the public sector, serious problems on large IT-related contracts are not usually fixed until the seriousness of the problems can no longer be denied.

For hundreds of years UK governments have struggled to reconcile a theoretical desire for openness with an instinctive and institutional need to hide mistakes. Nothing is likely to change now.

National Audit Office report – Investigation into clinical correspondence handling in the NHS.

Companies nervous over HMRC customs IT deadline?

By Tony Collins

This Computer Weekly article in 1994 was about the much-delayed customs system CHIEF. Will its CDS replacement that’s being built for the post-Brexit customs regime also be delayed by years?

The Financial Times  reported this week that UK companies are nervous over a deadline next year for the introduction of a new customs system three months before Brexit.

HMRC’s existing customs system CHIEF (Customs Handling of Import Export Freight) copes well with about 100 million transactions a year. It’s expected a £157m replacement system using software from IBM and European Dynamics will have to handle about 255 million transactions and with many more complexities and interdependencies than the existing system.

If the new system fails post-Brexit and CHIEF cannot be adapted to cope, it could be disastrous for companies that import and export freight. A post-Brexit failure could also have a serious impact on the UK economy and the collection of billions of pounds in VAT, according to the National Audit Office.

The FT quoted me on Monday as calling for an independent review of the new customs system by an outside body.

I told the FT of my concern that officials will, at times, tell ministers what they want to hear. Only a fully independent review of the new customs system (as opposed to a comfortable internal review conducted by the Infrastructure and Projects Authority) would stand a chance of revealing whether the new customs system was likely to work on time and whether smaller and medium-sized companies handling freight had been adequately consulted and would be able to integrate the new system into their own technology.

The National Audit Office reported last year that HMRC has a well-established forum for engaging with some stakeholders but has

“significant gaps in its knowledge of important groups. In particular it needs to know more about the number and needs of the smaller and less established traders who might be affected by the customs changes for the first time”.

The National Audit Office said that the new system will need to cope with 180,000 new traders who will use the system for the first time after Brexit, in addition to the 141,000 traders who currently make customs declarations for trade outside the EU.

The introduction in 1994 of CHIEF was labelled a disaster at the time by some traders,  in part because it was designed and developed without their close involvement. CHIEF  was eventually accepted and is now much liked – though it’s 24 years old.

Involve end-users – or risk failure

Lack of involvement of prospective end-users is a common factor in government IT disasters. It happened on the Universal Credit IT programme, which turned out to be a failure in its early years, and on the £10bn National Programme for IT which was dismantled in 2010. Billions of pounds were wasted.

The FT quoted me as saying that the chances of the new customs system CDS [Customs Declaration Service) doing all the things that traders need it to do from day one are almost nil.

The FT quotes one trader as saying,

“HMRC is introducing a massive new programme at what is already a critical time. It would be a complex undertaking at the best of times but proceeding with it at this very moment feels like a high stakes gamble.”

HMRC has been preparing to replace CHIEF with CDS since 2013. Its civil servants say that the use of the SAFe agile methodology when combined with the skills and capabilities of its staff mean that programme risks and issues will be effectively managed.

But, like other government departments, HMRC does not publish its reports on the state of major IT-related projects and programmes. One risk, then,  is that ministers may not know the full truth until a disaster is imminent.

In the meantime ministerial confidence is likely to remain high.

Learning from past mistakes?

HMRC has a mixed record on learning from past failures of big government IT-based projects.  Taking some of the lessons from “Crash”, these are the best  things about the new customs project:

  • It’s designed to be simple to use – a rarity for a government IT system. Last year HMRC reduced the number of system features it plans to implement from 968 to 519. It considered that there were many duplicated and redundant features listed in its programme backlog.
  • The SAFe agile methodology HMRC is using is supposed to help organisations implement large-scale, business-critical systems in the shortest possible time.
  • HMRC is directly managing the technical development and is carrying out this work using its own resources, independent contractors and the resources of its government technology company, RCDTS. Last year it had about 200 people working on the IT programme.

These are the potentially bad things:

  • It’s not HMRC’s fault but it doesn’t know how much work is going to be involved because talks over the post-Brexit customs regime are ongoing.
  • It’s accepted in IT project management that a big bang go-live is not a good idea. The new Customs Declaration Service is due to go live in January 2019, three months before Britain is due to leave the EU. CHIEF system was commissioned from BT in 1989 and its scheduled go-live was delayed by two years. Could CDS be delayed by two years as well? In pre-live trials CHIEF rejected hundreds of test customs declarations for no obvious reason.
  • The new service will use, at its core,  commercially available software (from IBM) to manage customs declarations and software (from European Dynamics) to calculate tariffs. The use of software packages is a good idea – but not if they need large-scale modification.  Tampering with proven packages is a much riskier strategy than developing software from scratch.  The new system will need to integrate with other HMRC systems and a range of third-party systems. It will need to provide information to 85 systems across 26 other government bodies.
  • If a software package works well in another country it almost certainly won’t work when deployed by the UK government. Core software in the new system uses a customs declaration management component that works well in the Netherlands but is not integrated with other systems, as it would be required to do in HMRC, and handles only 14 million declarations each year.
  • The IBM component has been tested in laboratory conditions to cope with 180 million declarations, but the UK may need to process 255 million declarations each year.
  • Testing software in laboratory conditions will give you little idea of whether it will work in the field. This was one of the costly lessons from the NHS IT programme NPfIT.
  • The National Audit Office said in a report last year that HMRC’s contingency plans were under-developed and that there were “significant gaps in staff resources”.

Comment

HMRC has an impressive new CIO Jackie Wright but whether she will have the freedom to work within Whitehall’s restrictive practices is uncertain. It seems that the more talented the CIO the more they’re made to feel like outsiders by senior civil servants who haven’t worked in the private sector.  It’s a pity that some of the best CIOs don’t usually last long in Whitehall.

Meanwhile HMRC’s top civil servants and IT specialists seem to be confident that CDS, the new customs system, will work on time.  Their confidence is not reassuring.  Ministers and civil servants publicly and repeatedly expressed confidence that Universal Credit would be fully rolled by the end of 2017. Now it’s running five years late.  The NHS IT programme NPfIT was to have been rolled out by 2015.  By 2010 it was dismantled as hopeless.

With some important exceptions, Whitehall’s track record on IT-related projects is poor – and that’s when what is needed is known. Brexit is still being negotiated. How can anyone build a new bridge when you’re not sure how long it’ll need to be and what the many and varied external stresses will be?

If the new or existing systems cannot cope with customs declarations after Brexit it may not be the fault of HMRC. But that’ll be little comfort for the hundreds of thousands of traders whose businesses rely, in part, on a speedy and efficient customs service.

FT article – UK companies nervous over deadline for new Customs system

Capita under fire again over GP support contract – but NHS England praises “improvements”

By Tony Collins

Hundreds of trainee GPs have not received their salaries from Capita, which is under contract to pay them, reports The Guardian.

Some of the trainees have applied for emergency funds from The Cameron Fund, a charity for the prevention of hardship among GPs and their dependents.

Capita administers training grants for GPs under its wide-ranging £1bn contract with NHS England to provide primary care services.

In November 2016 the then Health minister Nicola Blackwood described failings on Capita’s GP support contract as “entirely unacceptable”. 

She said Capita had inadequately prepared for delivering a “complex transition”.

In response,  Capita said it adding the full-time equivalent of 500 extra staff on the contract.

But in February 2017, after continuing complaints,  the Health Secretary Jeremy Hunt said he would be prepared to end Capita’s contract if necessary.

Since then, though, NHS England has praised “improvements” in the contract, according to Pulse.

Yesterday The Guardian reported extracts from a letter the British Medical Association sent to NHS England on 30 October 2017.

It said some GP practices were “having to pay trainees out of already overstretched practice budgets, or trainees are going months without being paid if the practice cannot cover the shortfall”.

Capita confirmed it had outstanding payments to some trainee GPs but was unable to say how many it is responsible for paying, or how many it has not paid.

It said that it had not received all the information it needed to pay salaries from the relevant employers. A Capita spokesperson told The Guardian that the problems were an inevitable part of “a major transformation project to modernise a localised and unstandardised service”.

It added: “We have made significant investment to deliver improvements and these have been recognised by NHS England and demonstrated through improved service performance and improved customer satisfaction.”

The Cameron Fund’s treasurer Dr David Wrigley described the outsourcing of GP support services as a “botched privatisation”.

“NHS England has commissioned out what was a very efficient service run within the NHS, and now Capita runs this contract in what I’d call another botched privatisation.”

One trainee GP went unpaid two consecutive months.  At the end of October she posted on a private message board for GPs: “Anyone know of how I access hardship funds (quickly) to feed children/pay nursery/mortgage (quickly)?”

Her surgery gave her a loan last month to tide her over but did not have enough surplus funds to do the same thing again.

She said that in the last 24 hours partners have stepped forward and have all taken a pay cut to provide a loan “to get me through the month as they were worried about my family”.

An NHS England spokesperson said it was “holding Capita’s feet to the fire on needed improvements”.

It added: “In the meantime, the lead employer for Health Education England or the GP practice are responsible for paying their GP trainee salaries and are subsequently reimbursed for this. Backlogs are being prioritised by Capita.”

The BMA’s letter to the NHS chief executive Simon Stevens criticises Capita.

“We are disappointed at the lack of progress that has been made … These issues have been ongoing since NHS England commissioned Capita … and it is unacceptable that more progress has not been made to getting these resolved …

Wrigley wants the House of Commons’ public accounts committee to investigate the contract.

“NHS England have known about this for a while and the BMA has been putting constant pressure on, and it’s all promises that it’ll get better but it doesn’t.”

New systems for cervical screening and GP payments and pensions that are also contracted out to Capita are due to go live next July. The BMA has told NHS England that it has “no confidence” in Capita’s ability to deliver the services.

Comment

It’s possible to have some sympathy for Capita which has the daunting task of trying to standardize a wide range of systems for supporting disparate GP support services.

But, as Campiagn4Change has reported many times on Barnet Council’s Capita outsourcing contract, it can be difficult if not impossible to make huge savings in the cost of running services (£40m in the case of the GP support contract), deliver an IT-based transformation based on new investment and provide a healthy profit for the supplier’s shareholders while at the same time making internal efficiency savings.

Capita’s share price is relatively low and under continuing pressure but is holding up reasonably well given the company’s varied problems.

Still, we wonder whether the company can afford to put large sums into sorting out problems on the GP support contract, at Barnet Council and on other well-publicised contracts?

The MoD has ended a Capita contract early, the company faces litigation from the Co-op and its staff are staging nine days of strikes over pensions.

Who’s to blame?

If anyone is to blame in this NHS saga it is NHS England for not fully understanding the scale and complexity of the challenges when it outsourced to Capita.

The first rule of outsourcing is: Don’t outsource a problem.

Doctors warned NHS England against signing the contract. Under financial pressure to do so – it needed the promised savings  – NHS England’s public servants signed the deal.

Those public servants will not be held accountable for their decision. In which case, what’s to stop public and civil servants making the wrong decisions time and again?

Two further questions:

Is NHS England too close to Capita to see the faults?

Do public servants have a vested interest in not criticising their outsourcing suppliers, in case opprobrium falls on both parties? 

Thank you to Zara Pradyer for drawing my attention to the Guardian article.

Hundreds of trainee GPs facing hardship as outsourcing firm Capita fails to pay – The Guardian.

 

A proposed Bill and charter that could change the face of Whitehall IT and save billions

By Tony Collins

A government-commissioned review yesterday backed a Bill that could, if enacted and applied to Whitehall generally, prevent billions of pounds being lost on wasteful projects.

The Public Authority Accountability Bill – known informally as the Hillsborough Law – would establish an offence of intentionally or recklessly misleading the public, media or court proceedings.

It would also impose a legal requirement on public authorities to act with candour, transparency and frankness when things go wrong.

Although the Bill was a reaction, in part, to the cover up by public authorities of their failings in the light of Hillsborough, it could, if enacted, deter public authorities from covering up failings generally – including on major IT programmes.

For decades public authorities have had the freedom – unrestricted by any legislation – to cover up failures and issue misleading statements to the public, Parliament and the media.

In the IT sphere, early problems with the Universal Credit IT programme were kept secret and misleadingly positive statements issued. The National Audit Office later criticised a “good news” culture on the Universal Credit programme.

And still the DWP is fighting to block the disclosure of five project assessment reviews that were carried out on the Universal Credit IT programme between 2012 and 2015.

It could be argued that billions of pounds lost on the NPfIT – the National Programme for IT in the NHS – would have been avoided if the Department of Health had been open and candid at the start of the programme about the programme’s impractically ambitious aims, timescales and budgets.

The Department for Business, Energy and Industrial Strategy is currently keeping secret its progress reports on the £111bn smart meters rollout – which independent experts have said is a failing programme.  The department routinely issues positive statements to the media on the robust state of the programme.

The Public Authority Accountability Bill was drafted by lawyers who had been involved with representing bereaved Hillsborough families. It is aimed mainly at government inquiries, court proceedings and investigations into lapses of public services.

But it would also enshrine into law a duty on public authorities, public servants, officials and others to act within their powers with “transparency, candour and frankness”.

Lawyers who drafted the Bill refer on their website to “institutional defensiveness and a culture of denial” when things go wrong. They say,

“In 2017 we expect public authorities and individuals acting as public servants to be truthful and act with candour. Unfortunately, repeated examples have shown us that this is not generally the case.

“Instead of acting in the public interest by telling the truth, public authorities have tended to according to narrow organisational and individual motives by trying to cover up faults and deny responsibility …”

Backing for the Bill came yesterday from a 117-page report on the Hillsborough disaster by Bishop James Jones. The government commissioned him to produce a report on the experiences of the Hillsborough families so that their “perspective is not lost”.

Jones’ impressive report refers to institutions that “closed ranks, refused to disclose information, used public money to defend its interests and acted in a way that was both intimidating and oppressive”

His report refers to public bodies in general when it points to a “cultural condition” and “mindset” that features an “instinctive prioritisation of the reputation of an organisation over the citizen’s right to expect people to be held to account for their actions”. This, says the report, “represents a barrier to real accountability”.

It adds,

“As a cultural condition, this mindset is not automatically changed, still less dislodged, by changes in policies or processes. What is needed is a change in attitude, culture, heart and mind.”

The report urges leaders of “all public bodies” to make a commitment to cultural change by publicly signing a new charter.

The charter commits public bodies to:

  •  Place the public interest above its own reputation.
  • Approach forms of scrutiny with candour, in an open, honest and transparent way, making full disclosure of relevant documents, material and facts.
  • Learn from the findings of external scrutiny and from past mistakes.
  • Avoid seeking to defend the indefensible or to dismiss or disparage those who may have suffered where the organisation has fallen short.
  • When falling short, apologise straightforwardly and genuinely.
  • Not knowingly mislead the public or the media.

The report says that institutional defensiveness and a culture of denial are “endemic amongst public institutions as has been demonstrated not only by the Hillsborough cover up but countless other examples.”

Stuart Hamilton, son of Roy Hamilton who died at Hillsborough, is quoted in the report as saying,

“Police, officials and civil servants should have a duty of revealing the full facts and not merely selecting some truths to reveal but not others. Not lying or not misleading is simply not good enough. Without this, future disasters cannot be averted and appropriate policies and procedures cannot be developed to protect society.

“Such selective revealing of information also results in the delay of justice to the point where it cannot be served”.

He added,

“I believe that without a change not only in the law but also in the mindset of the public authorities (which a law can encourage) then very little exists to stop the post-event actions happening again.”

IT-enabled projects

Whitehall departments and the Infrastructure and Projects Authority publish their own narratives on the progress on major IT-enabled projects and programmes such as Universal Credit and smart meters.

But their source reports aren’t published.

Early disclosure of failings could have prevented hundreds of millions of pounds being lost on FireControl project, BBC’s Digital Media Initiative, the Home Office Raytheon e-borders and C-Nomis national offender management information projects and the Rural Payments Agency’s CAP delivery programme (which, alone, contributed to EU penalties of about £600m).

Comment:

Yesterday’s beautifully-crafted report into the Hillsborough disaster – entitled “The patronising disposition of unaccountable power” – is published on the Gov.uk website.

It has nothing to do with IT-enabled projects and programmes. But, in an unintentional way, it sums up a public sector culture that has afflicted nearly every Whitehall IT-based project failure in the last 25 years.

A culture of denial is not merely prevalent today; it is pervasive. All Whitehall departments keep quiet about reports on their failings. It is “normal” for departments to issue misleadingly positive statements to the media about progress on their programmes.

The statements are not lies. They deploy facts selectively, in a way that covers up failings. That’s the Whitehall culture. That’s what departments are expected to do.

According to Bishop Jones’ Hillsborough report, one senior policeman told bereaved families that he was not obliged to reveal the contents of his reports. He could bury them in his garden if he wished.

It’s the same with government departments. There is no legal duty to keep programme reports, still less any requirement to publish them.

If Bishop Jones’ charter is signed by leaders of public authorities including government departments, and Andy Burnham’s Bill becomes law,  the requirement for candour and transparency could mean that IT programme progress reports are made available routinely.

If this happened – a big if – senior public officials would have to think twice before risking billions of pounds on a scheme that held out the prospect of being fun to work on but which they knew had little chance of success within the proposed timescales, scope and budget.

It’s largely because of in-built secrecy that the impossibly impractical NPfIT was allowed to get underway. Billions of pounds was wasted.

Some may say that the last thing ministers and their permanent secretaries will want is the public, media and MPs being able to scrutinise what is really happening on, say, a new customs IT project to handle imports and exports after Brexit.

But the anger over the poor behaviour of public authorities after Hillsborough means that the Bill has an outside chance of eventually becoming law. Meanwhile public sector leaders could seriously consider signing Jones’ charter.

John Stuart Mill wrote in 1859 (On Liberty and The Subjection of Women) that the “only stimulus which can keep the ability of the [public] body itself up to a high standard is liability to the watchful criticism of equal ability outside the body”.

 

Why are councils hiding exit costs of outsourcing deals – embarrassment perhaps?

Tony Collins

Excerpt from Taunton Deane council’s confidential “pink pages”.
The last sentence contains a warning that IBM-owned SWO – Southwest One – may try to “maximise revenues” on exiting its joint venture with the council.

Somerset County Council has refused a Freedom of Information request for the costs of exiting its joint venture with IBM.

But a secret report written last year by officers at Taunton Deane Borough Council – which was a party to the IBM-owned joint venture company Southwest One  – warned that the supplier could attempt to “maximise revenues on exit”.

It said,

“… from experience anything slightly ambiguous within the contract is likely to be challenged by SWO [Southwest One] in order to push it into the chargeable category as they attempt to maximise revenues on exit”.

A separate section of the confidential report said,

“disaggregating from the SWO [Southwest One] contract will be complex and expensive …”

Taunton Deane Borough Council did not tell councillors what the exit turned out to be. The figures are also being kept secret by Somerset County Council which signed the “transformative” SWO joint venture deal with IBM in 2007.

Both councils have now brought back services in-house.

Secrecy over the exit costs is in contrast to Somerset’s willingness to talk in public about the potential savings when local television news covered the setting up of Southwest One in 2007.

The silence will fuel some local suspicions that exit costs have proved considerable and will have contributed to the justifications for Somerset’s large council tax rise this year.

£69m losses?

David Orr, a former Somerset County Council IT employee, has followed closely the costs of the joint venture, and particularly its SAP-based “transformation.

It was his FOI request for details of the exit costs that the council refused.

Orr says that Somerset has lost money as a result of the Southwest One deal. Instead of saving £180m, the joint venture has cost the council £69m, he says.

FOI

Under the Freedom of Information Act, Orr asked Somerset for the “total contract termination costs” including legal, consultancy, negotiation, asset valuations, audit and extra staffing.

He also asked whether IBM was paid compensation for early termination of the Southwest One contract. In replying, the council said,

“The Authority exited from a significant contract with Southwest One early, and the services delivered through this contract were brought back in-house in November 2016.

“The Authority expects the costs to fall significantly now it has regained control of those services.

“Somerset County Council made payment under the ‘Termination for Convenience’ provisions of the original contract. We do hold further information but will not be releasing it at this point as we believe to do so would damage the commercial interests of the County Council, in that it would prejudice the our negotiating position in future contract termination agreements in that it would give contractors details on what terms the Council was willing to settle …”

Orr will appeal. He says the Information Commissioner has already established a principle with Suffolk Coastal District Council that the termination costs of a contract with a third party should be disclosed. The commissioner told Suffolk Coastal council that, in opting out of FOI,

“there is no exemption for embarrassment”

Hidden costs

Taunton’s pink pages paper said that the Southwest One contract’s Exit Management Plan provided for a smooth transfer of services and data, and for access to staff to assess skills and do due diligence.

In practice, though, there were many exit-related complications and costs – potential and actual. The paper warned that Taunton would need to find the money for:

  • Exit programme and project management costs
  • Early termination fees
  • Contingency
  • ICT infrastructure disaggregation
  • Service transition and accommodation costs
  • Disaggregating SAP from Southwest One. Also the council would need to exit its SAP-based shared services with Somerset County Council because the estimated costs were lower when run on a non shared services basis. SAP covered finance, procurement, HR, payroll, website and customer relationship management.
  • Costs involved in a “soft” or “hard” (adversarial) exit.
  • Estimating council exit costs when IBM was keeping secret its own Southwest One running costs.
  • Staff transfer issues.

Comment

So much for open government. It tends to apply when disclosures will not embarrass local government officials.

In 2007 Somerset County Council enjoyed local TV, radio and newspaper coverage of the new joint venture with IBM. Officials spoke proudly on camera of the benefits for local taxpayers, particularly the huge savings.

Now, ten years later, the losses are stacking up. Former Somerset IT employee and FOI campaigner Dave Orr puts the losses at £69m. And local officials are keeping secret the further exit costs.

Suffolk Coastal District Council lost an FOI case to withhold details of how much it paid in compensation to a third party contractor to terminate a contract. But at least it had published its other exit costs.

Somerset is more secretive. It is withholding details of the sums it paid to IBM in compensation for ending the joint venture early; it also refuses to publish its other exit costs.

Trust?

Can anything said by councils such as Somerset or Barnet in support of major outsourcing/joint venture deals be trusted if the claimed savings figures are not audited and the other side of the story – the hidden costs – are, well, hidden?

In local elections, residents choose councillors but they have no say over the appointment of the permanent officials. It’s the officials who decide when to refuse FOI requests; and they usually decide whether the council will tell only one side of the story when public statements are made on outsourcing/joint ventures.

Across the UK, local councils employed 3,400 press and communications staff –  about double the total number in central government – in part to promote the authorities’ services and activities.

What’s the point if they publicise only one side of the story – the benefits and not the costs?

Somerset’s decision to refuse Orr’s reasonable FOI request makes, in its own small way, a mockery of open government.

It also gives just cause for Somerset residents to be sceptical about any council statement on the benefits of its services and activities.

Will MPs call BA to account over IT power problems?

By Tony Collins

Experts are questioning BA’s explanation of the power problems that disrupted the travel plans and arrangements for 75,000 people at the weekend.

BA says it is “reviewing” what went wrong at the weekend but is under no regulatory duty to publish the findings.

There is little pressure from shareholders to hold BA to account. The share price of BA’s parent International Airlines Group is higher today than a month ago.

Sceptical

Yesterday the BBC’s business editor Simon Jack accused IAG of dodging tough questions it will “surely have to answer” and the FT quoted IT and electricity experts who are sceptical of the airline’s explanations.

But MPs on the Transport select committee – a new one will be formed after the general election next week – could decide, if pressed by their constituents, to have an inquiry into BA’s power problems.

If so, they could question BA’s chief Alex Cruz or Willie Walsh, the chief executive of IAG.

In 1997 the committee held an inquiry into the escalating costs and problems on IT contracts at the Swanwick air traffic control centre in Hampshire. MPs decided to publish the contents of an independent report into the problems by technology consultancy Arthur D Little.

Any 2017 inquiry by the committee could hold BA to account in a way that would not otherwise be possible. Lessons from the failures may be useful to the public and private sectors.

UPS failure

Meanwhile what went wrong and why seems confused.

The Telegraph says the BA review is focusing on the uninterruptible power supply (UPS) to Boadicea House, one of two data centres close to Heathrow airport.

The UPS in question delivers power through the mains, diesel and batteries.

On Saturday morning, shortly after 8.30am, power to Boadicea House through its UPS was shut down. The reasons are unclear.

If power had returned to the servers in Boadicea House slowly this would have allowed the airline’s other Heathrow data centre, at Comet House, to take up some of the slack, said the Telegraph.

But, on Saturday morning, just minutes after the UPS went down, power was resumed in what one Telegraph source described as “uncontrolled fashion.”

This caused “catastrophic physical damage” to BA’s servers, which contain everything from customer and crew information to operational details and flight paths.

The Telegraph said that if power had been restored more gradually, BA would have been able to cope with the outage, and return services far more quickly than was the case.

The FT said yesterday that the UPS malfunctioned, cutting off the power supply. But it said that “some people working in the field have questioned” the explanation. They said it is very rare for UPS systems to fail. Even if they do, it should not affect the continued supply of mains electricity to the data centres they serve.

Not a technology problem?

BA has said there was an “immediate loss of power” from the UPS. When power returned, a surge physically damaged its IT servers. It had to replace the damaged equipment.

Willie Walsh said the meltdown was not a technology problem. The FT quoted him as saying, “You give me any IT system in the world and I’ll show you how good it is when it doesn’t have any electrical power going to it.”

Walsh insisted there was “no data loss, no data corruption”. He said the IT systems “functioned how they are supposed to function.”

But the FT quoted Jonathan Glover, co-founder of PSI, a company that helps businesses protect their equipment against sudden, unexpected power surges, who said the failure of a UPS “was relatively unlikely as they are robust and well-proven pieces of equipment”.

He added that, even if the UPS system did fail, it should not make a difference to the power supply to the airline’s IT system. The answers given don’t make a lot of sense, he said.

Alan Woodward, visiting professor at the department of computer science at the University of Surrey, agreed. He told the FT,

“It is like on your laptop and if you just pull the plug out of the back, it shouldn’t affect your laptop. It keeps running until the battery runs down. Even if you unplug the battery [of a laptop], it doesn’t like it from a data perspective, but plug it back in again, you don’t suddenly get a big power surge.”

Woodward said one possible explanation was that a voltage regulator contained within the UPS might have malfunctioned but when they fail the power usually stops, he added.

Another expert on UPS technology said that even if the system had failed, it would simply have been bypassed and normal electricity supply should have continued.

Why would the failure of the UPS affect BA’s back-up data centre?  The answer is unknown. BA would not comment on whether their two Heathrow-based data centres relied on the same UPS.

Ryanair on Tuesday pointed out that it had IT systems in three locations around Europe and if one went down, there were backups at each of its data centres. Ryanair’s data centres are not close to each other.

Two electricity companies whose low-voltage networks cover Heathrow airport and the surrounding area have denied there were any issues on their networks on Saturday morning.

Transient voltage surge arresters can shield against power surges from the local electricity network and malfunctions in a company’s own equipment but it is unclear whether BA had these fitted and if it did whether they worked.

The FT quoted an expert as saying that BA either had inadequate defences or  didn’t have the right level of industrial-level surge protection. BA has not commented on what protection measures it had.

Will BA publish its review?

BA may be reluctant to reveal the results of its review for various reasons. Parts of its IT appear in the UK could be run by non-BA staff. The failures could raise questions about the corporate oversight of any non BA specialists, possibly at board level.

It is also possible that an internal review could highlight fundamental managerial weaknesses – such as unclear or confused IT responsibilities in the UK or at IAG – after the outsourcing of IT skills to India last year.

Damian Brewer, an analyst at RBC Capital Markets, told the Telegraph that if BA’s early diagnosis of the cause of the crisis is correct, bosses’ failure to prepare for such an incident in the light of other carriers’ problems “suggests fundamental management and planning weakness”.

“It seems highly questionable why similar incidents with major US carriers in the last year have failed to see IAG move to ensure its airlines had plans in place to mitigate this risk, already seen elsewhere, and also to have contingency plans in place,” he said.

“At present, it appears that BA management have seemingly not taken account of IT risk precedent already seen and already known at other carriers.”

In what BA has said publicly about the IT problems, much of it has focused on what didn’t happen (a cyber attack) and on the people who were not responsible (Tata in India or energy companies). It told the BBC  the problems were “definitely not a consequence of underinvestment or cost-cutting.”

“All the parties involved around this particular event have not been involved with any type of outsourcing in any foreign country,” said Cruz.  “They have all been local issues around a local data centre who [sic] has been managed and fixed by local resources.”

Comment

Without an inquiry by the newly-formed Transport Committee, BA will find it easy to keep the lid on the results of its inquiry into the failures.  This would be a pity given the lessons that could be learned.

It’s ironic that the aviation industry has an exemplary reputation for reporting even minor problems that relate to safety. There is a duty to report even a ruffled carpet in an aircraft aisle that could trip up passengers or crew.

But there is no duty to account for an IT failure that disrupted the lives of 75,000 people across the world because it was not a safety issue. Provided the company pays satisfactory compensation, the fiasco will probably be out of the public eye in a few months.

But MPs, on behalf of their constituents,  could hold BA to account.

Anyone who wants to ask MPs to hold an inquiry into the BA failures could write to:

Transport Committee
House of Commons
London
SW1A 0AA

Telephone: 020 7219 3266
transcom@parliament.uk
Twitter: @CommonsTrans

The Committee’s clerk is Gordon Clarke: clarkeg@parliament.uk

Thank you to Dave Orr for his regular updates on the BA problems

BA’s IT: Will Transport Committee MPs ask the tough questions? – Government Computing

Full details of meltdown revealed (says Daily Telegraph)

BA board to demand IT chaos inquiry – Simon Jack, BBC

Aftermath of the cyber attack – will ministers learn the wrong lessons?

By Tony Collins

At least 16 NHS trusts out of 47 that were hit by the ransomware attack continue to face problems, according to BBC research.

And, as some patients continued to have their cancer treatments postponed, Tory, Labour and Lib-dem politicians told of their plans to spend more money on NHS IT.

But will any new money promised by government focus on basic weaknesses – such as the lack of interoperability and the structural complexities that made the health service vulnerable to cyber attack?

Last year when the health secretary Jeremy Hunt announced £4bn for NHS IT, his focus was on new technologies such as smartphone apps to order repeat prescriptions rather than any urgent need to upgrade MRI, CT and other medical devices that rely on Windows XP.

Similarly the government-commissioned Wachter review “Making IT Work: Harnessing the Power of HealthInformation Technology to Improve Care in England made no mention of Windows XP or any operating system – perhaps because ministers were much more likely to welcome a review of NHS IT that focused on innovation and new technologies.

Cancer treatments postponed

The Government’s position is that the NHS was not specifically targeted in the cyber attack and that the Tories are putting £2bn into cyber security over the next year.

Theresa May said yesterday,

“It was clear warnings were given to hospital trusts but this is not something that was focused on attacking the NHS. 150 countries are affected. Europol says there are 200,000 victims across the world. Cyber security is an issue we need to address.

“That’s why the government, when we came into government in 2010, put money into cyber security. It’s why we are putting £2bn into cyber security over the coming year.”

Similarly Jeremy Hunt, health secretary, told the BBC that the attack affected international sites that have “some of the most modern IT systems”.

But the BBC’s World at One gave an example of how the NHS’s IT problems were affecting the lives of patients.

It cited the case of Claire Hobday whose radiography appointment for breast cancer at Lincoln County Hospital was cancelled on Friday (12 May 2017) and she still doesn’t know when she’ll receive treatment. Hobday said,

“I turned up by hospital transport for my second radiotherapy session, and I, along with many other patients – at least 20 other people were waiting – and they said the computers weren’t working.

“I do have to say the staff were very good and very quickly let us all know that they were having trouble with the computers. They didn’t want to misinform us, so they were going to come and talk to us all individually and hoped they would be able to rectify it.

“Within half an hour or so they came out and said, ‘We’re really sorry but it’s not going to get sorted. We’ll send you all home and give you a call on Sunday’ which didn’t happen.

“But they did ring me this morning (15 May 2017) to say it’s not happening today and if transport turns up please don’t get in it, and it’s very unlikely it will happen tomorrow.

“It is just a bit upsetting that other authorities have managed to sort it but Lincolnshire don’t seem to have been able to do that.”

United Lincolnshire Hospitals Trust told World at One it will be back in touch with patients once the IT system is restored.

Roy Grimshaw was in the middle of an MRI scan – after dye was injected into his blood stream –  when the scan was stopped and he was asked to go back into the waiting room in his gown, with tubes attached to him, while staff investigated a computer problem. After half an hour he was told the NHS couldn’t continue the scan.

Budgets “not an issue”?

GP practices continue to be affected. Keiran Sharrock, GP and medical director of Lincolnshire local medical committee, said yesterday (15 Mat 2017) that systems were switched off in “many” practices.

“We still have no access to medical records of our patients. We are asking patients to only contact the surgery if they have an urgent or emergency problem that needs dealing with today. We have had to cancel routine follow-up appointments for chronic illnesses or long-term conditions.”

Martha Kearney – BBC World at One presenter –  asked Sharrock about NHS Digital’s claim that trusts were sent details of a security patch that would have protected against the latest ransomware attack.

“I don’t think in general practice we received that information or warning. It would have been useful to have had it,” replied Sharrock.

Kearney – What about claims that budget is an aspect of this?

Sharrock: “Within general practice that doesn’t seem to be the reason this happened. Most general practices have people who can work on their IT and if we’d been given the patch and told it needed to be installed, most practices would have done that straight away.”

GCHQ

World at One also spoke to Ciaran Martin, Director General for Government and Industry Cyber Security.  He is a member of the GCHQ board and its senior information risk owner.  He used to be Constitution Director at the Cabinet Office and was lead negotiator for the Prime Minister in the run-up to the Edinburgh Agreement in 2012 on a referendum on independence for Scotland.

Kearney: Did your organisation issue any warnings to the health service?

Martin: “We issue warnings and advice on how to upgrade defences constantly. It’s generally public on our website and it’s made very widely available for all organisations. We are a national organisation protecting all critical sectors and indeed individuals and smaller organisations as well.”

Huge sums spent on paying ransoms?

Kearney asked Martin, “How much money are you able to estimate is being spent on ransoms as a result of these cyber attacks?” She added,

“I did hear one astonishing claim that in the first quarter of 2016 more money was spent in the USA on responding to ransomware than [was involved] in armed robberies for the whole of that year?”

Martin: “First let me make clear that we don’t condone the payment of ransoms and we strongly advise bodies not to pay and indeed in this case the Department of Health and the NHS have been very clear that affected bodies are not to pay ransoms. Across the globe there is, sadly, a market in ransomware. It is often the private sector in shapes and sizes that is targeted.”

Martha Kearney said the UK may be a target because it has a reputation for being willing to pay ransoms.

Martin, “We are no more or less a target for ransomware than anywhere else. It’s a global business; and it is a business. It is all about return on investment for the attacker.

“What’s important about that is that it’s all about upgrading defences because you can make the return on investment lower by making it harder to get in.”

If an attacker gets in the aim must be to make it harder to get anything useful, in which case the “margin on investment goes down”. He added,

“That’s absolutely vital to addressing this problem.”

Are governments at fault?

Martin,

“Vulnerabilities will always exist in software. Regardless of who finds the underlying software defect, it’s incumbent on the entire cyber security ecosystem – individual users, enterprises, governments or whoever – to work together to mitigate the harm.”

He added that there are “all sorts of vulnerabilities out there” including with open source software.

Windows XP

Computer Weekly reports – convincingly – that the government did not cancel an IT support contract for XP.

Officials decided to end a volume pricing deal with Microsoft which left NHS organisations to continue with XP support if they chose to do so. This was clearly communicated to affected departments.

Government technology specialists, reports Computer Weekly, did not want a volume pricing deal with Microsoft to be  “comfort blanket” for organisations that – for their own local reasons – were avoiding an upgrade from XP.

Computer Weekly also reported that civil servants at the Government Digital Service expressed concerns about the lack of technical standards in the NHS to the then health minister George Freeman.

Freeman was a Department of Health minister until July 2016. In their meeting with Freeman, GDS officials  emphasised the need for a central body to set technical standards across the NHS, with the authority to ensure trusts and other organisations followed best practice, and with the transparency to highlight those who chose not to.

A source told Computer Weekly that Jeremy Hunt was also briefed on the security risks that a lack of IT standards would create in a heavily-federated NHS but it was not considered a priority at that top political level.

“Hunt never grasped the problem,” said the source.

There are doubts, though, that Hunt could have forced trusts to implement national IT security standards even if he’d wanted to. NHS trusts are largely autonomous and GDS has no authority to mandate technical standards. It can only advise.

How our trust avoided being hit

A comment by an NHS IT lead on Digital Health’s website gives an insight into how his trust avoided being hit by the latest cyber attack.  He said his trust had a “focus on perimeter security” and then worked back to the desktop.

“This is then followed up by lots of IG security pop ups and finally upgrading (painfully) windows XP to windows 7…” He added,

“NHS Digital have to take a lead on this and enforce standards for us locally to be able to use.”

He also suggests that NHS Digital sign a Microsoft Enrollment for Windows Azure [EWA] agreement as it is costly arranging such a deal locally.

 “NHS Digital must for me, step in and provide another MS EWA as I am sure the disruption and political fall-out will cost more. Introduce an NHS MS EWA, introduce standards for software suppliers to comply with latest OS and then use CQC to rate organisations that do not upgrade.”

Another comment on the Digital Health website says that even those organisations that could afford the deployment costs of moving from XP to Windows 7 were left with the “professional” version, which “Microsoft has mercilessly withdrawn core management features from (e.g. group policy features)”.

The comment said,

“There are a lot of mercenary enterprises taking advantage of the NHS’s inability to mandate and coordinate the required policies on suppliers which would at least give the under-funded and under-appreciated IT functions the ability to provide the service they so desperately want to.”

A third comment said that security and configuration management in the NHS is “pretty poor”. He added, “I don’t know why some hospitals continue to invest in home-brew email systems when there is a national solution ready and paid for.

“In this recent attack most the organisations hit seem to use local email systems.”

He also criticised NHS organisations that:

  • Do not properly segment their networks
  • Allow workstations to openly and freely connect to each other in a trusted zone.
  • Do not have a proper patch / update management regime
  • Do not firewall legacy systems
  • Don’t have basic ACLs [access control lists)

Three lessons?

  • Give GDS the ability to mandate no matter how many Sir Humphreys would be upset at every challenge to their authority. Government would work better if consensus and complacency at the top of the civil service were regarded as vices, while constructive, effective and forceful criticism was regarded as a virtue.
  • Give the NHS money to spend on the basic essentials rather than nice-to-haves such as a paperless NHS, trust-wide wi-fi, smartphone apps, telehealth and new websites. The essentials include interoperability – so that, at the least, all trusts can send test results and other medical information electronically to GPs –  and the upgrading of medical devices that rely on old operating systems.
  •  Plan for making the NHS less dependent on monolithic Microsoft support charges.

On the first day of the attacks, Microsoft released an updated patch for older Windows systems “given the potential impact to customers and their businesses”.

Patches are available for: Windows Server 2003 SP2 x64Windows Server 2003 SP2 x86, Windows XP SP2 x64Windows XP SP3 x86Windows XP Embedded SP3 x86Windows 8 x86, and Windows 8 x64.

Reuters reported last night that the share prices of cyber security companies “surged as investors bet on governments and corporations spending to upgrade their defences”.

Network company Cisco Systems also closed up (2.3%), perhaps because of a belief that it would benefit from more network spending driven by security needs.

Security company Avast said the countries worst affected by WannaCry – also known as Wannacypt – were Russia, Taiwan, Ukraine and India.

Comment

In a small room on the periphery of an IT conference on board a cruise ship , nearly all of the senior security people talked openly about how their board directors had paid ransoms to release their systems after denial of service attacks.

Some of the companies – most of them household names – had paid ransoms more than once.

Until then, I’d thought that some software suppliers tended to exaggerate IT security threats to help market their solutions and services.

But I was surprised at the high percentage of large companies in that small room that had paid ransoms. I no longer doubted that the threats – and the damage – were real and pervasive.

The discussions were not “off-the-record” but I didn’t report their comments at the time because that would doubtless have had job, and possibly even career ramifications, if I had quoted the security specialists by name.

Clearly ransomware is, as the GCHQ expert Kieran Martin put it, a global business but, as ransoms are paid secretly – there’s not a whisper in corporate annual accounts – the threat has not been taken seriously enough in some parts of the NHS.

The government’s main defence is that the NHS was not targeted specifically and that many private organisations were also affected.

But the NHS has responsibility for lives.

There may be a silver lining if a new government focuses NHS IT priorities on the basics – particularly the structural defects that make the health service an easy target for attackers.

What the NHS doesn’t need is a new set of politicians and senior civil servants who can’t help massaging their egos and trying to immortalise their legacy by announcing a patchwork of technological marvels that are fun to work on, and spend money on, but which gloss over the fact that much of the NHS is, with some notable exceptions, technologically backward.

Microsoft stockpiled patches – The Register

UK government, NHS and Windows XP support – what really happened – Computer Weekly

NHS letter on patches to counter cyber attack

Multiple sites hit by ransomware attack – Digital Health (31 comments)

Lessons from the WannaCrypt – Wannacry – cyber attack according to Microsoft