Aftermath of the cyber attack – will ministers learn the wrong lessons?

By Tony Collins

At least 16 NHS trusts out of 47 that were hit by the ransomware attack continue to face problems, according to BBC research.

And, as some patients continued to have their cancer treatments postponed, Tory, Labour and Lib-dem politicians told of their plans to spend more money on NHS IT.

But will any new money promised by government focus on basic weaknesses – such as the lack of interoperability and the structural complexities that made the health service vulnerable to cyber attack?

Last year when the health secretary Jeremy Hunt announced £4bn for NHS IT, his focus was on new technologies such as smartphone apps to order repeat prescriptions rather than any urgent need to upgrade MRI, CT and other medical devices that rely on Windows XP.

Similarly the government-commissioned Wachter review “Making IT Work: Harnessing the Power of HealthInformation Technology to Improve Care in England made no mention of Windows XP or any operating system – perhaps because ministers were much more likely to welcome a review of NHS IT that focused on innovation and new technologies.

Cancer treatments postponed

The Government’s position is that the NHS was not specifically targeted in the cyber attack and that the Tories are putting £2bn into cyber security over the next year.

Theresa May said yesterday,

“It was clear warnings were given to hospital trusts but this is not something that was focused on attacking the NHS. 150 countries are affected. Europol says there are 200,000 victims across the world. Cyber security is an issue we need to address.

“That’s why the government, when we came into government in 2010, put money into cyber security. It’s why we are putting £2bn into cyber security over the coming year.”

Similarly Jeremy Hunt, health secretary, told the BBC that the attack affected international sites that have “some of the most modern IT systems”.

But the BBC’s World at One gave an example of how the NHS’s IT problems were affecting the lives of patients.

It cited the case of Claire Hobday whose radiography appointment for breast cancer at Lincoln County Hospital was cancelled on Friday (12 May 2017) and she still doesn’t know when she’ll receive treatment. Hobday said,

“I turned up by hospital transport for my second radiotherapy session, and I, along with many other patients – at least 20 other people were waiting – and they said the computers weren’t working.

“I do have to say the staff were very good and very quickly let us all know that they were having trouble with the computers. They didn’t want to misinform us, so they were going to come and talk to us all individually and hoped they would be able to rectify it.

“Within half an hour or so they came out and said, ‘We’re really sorry but it’s not going to get sorted. We’ll send you all home and give you a call on Sunday’ which didn’t happen.

“But they did ring me this morning (15 May 2017) to say it’s not happening today and if transport turns up please don’t get in it, and it’s very unlikely it will happen tomorrow.

“It is just a bit upsetting that other authorities have managed to sort it but Lincolnshire don’t seem to have been able to do that.”

United Lincolnshire Hospitals Trust told World at One it will be back in touch with patients once the IT system is restored.

Roy Grimshaw was in the middle of an MRI scan – after dye was injected into his blood stream –  when the scan was stopped and he was asked to go back into the waiting room in his gown, with tubes attached to him, while staff investigated a computer problem. After half an hour he was told the NHS couldn’t continue the scan.

Budgets “not an issue”?

GP practices continue to be affected. Keiran Sharrock, GP and medical director of Lincolnshire local medical committee, said yesterday (15 Mat 2017) that systems were switched off in “many” practices.

“We still have no access to medical records of our patients. We are asking patients to only contact the surgery if they have an urgent or emergency problem that needs dealing with today. We have had to cancel routine follow-up appointments for chronic illnesses or long-term conditions.”

Martha Kearney – BBC World at One presenter –  asked Sharrock about NHS Digital’s claim that trusts were sent details of a security patch that would have protected against the latest ransomware attack.

“I don’t think in general practice we received that information or warning. It would have been useful to have had it,” replied Sharrock.

Kearney – What about claims that budget is an aspect of this?

Sharrock: “Within general practice that doesn’t seem to be the reason this happened. Most general practices have people who can work on their IT and if we’d been given the patch and told it needed to be installed, most practices would have done that straight away.”

GCHQ

World at One also spoke to Ciaran Martin, Director General for Government and Industry Cyber Security.  He is a member of the GCHQ board and its senior information risk owner.  He used to be Constitution Director at the Cabinet Office and was lead negotiator for the Prime Minister in the run-up to the Edinburgh Agreement in 2012 on a referendum on independence for Scotland.

Kearney: Did your organisation issue any warnings to the health service?

Martin: “We issue warnings and advice on how to upgrade defences constantly. It’s generally public on our website and it’s made very widely available for all organisations. We are a national organisation protecting all critical sectors and indeed individuals and smaller organisations as well.”

Huge sums spent on paying ransoms?

Kearney asked Martin, “How much money are you able to estimate is being spent on ransoms as a result of these cyber attacks?” She added,

“I did hear one astonishing claim that in the first quarter of 2016 more money was spent in the USA on responding to ransomware than [was involved] in armed robberies for the whole of that year?”

Martin: “First let me make clear that we don’t condone the payment of ransoms and we strongly advise bodies not to pay and indeed in this case the Department of Health and the NHS have been very clear that affected bodies are not to pay ransoms. Across the globe there is, sadly, a market in ransomware. It is often the private sector in shapes and sizes that is targeted.”

Martha Kearney said the UK may be a target because it has a reputation for being willing to pay ransoms.

Martin, “We are no more or less a target for ransomware than anywhere else. It’s a global business; and it is a business. It is all about return on investment for the attacker.

“What’s important about that is that it’s all about upgrading defences because you can make the return on investment lower by making it harder to get in.”

If an attacker gets in the aim must be to make it harder to get anything useful, in which case the “margin on investment goes down”. He added,

“That’s absolutely vital to addressing this problem.”

Are governments at fault?

Martin,

“Vulnerabilities will always exist in software. Regardless of who finds the underlying software defect, it’s incumbent on the entire cyber security ecosystem – individual users, enterprises, governments or whoever – to work together to mitigate the harm.”

He added that there are “all sorts of vulnerabilities out there” including with open source software.

Windows XP

Computer Weekly reports – convincingly – that the government did not cancel an IT support contract for XP.

Officials decided to end a volume pricing deal with Microsoft which left NHS organisations to continue with XP support if they chose to do so. This was clearly communicated to affected departments.

Government technology specialists, reports Computer Weekly, did not want a volume pricing deal with Microsoft to be  “comfort blanket” for organisations that – for their own local reasons – were avoiding an upgrade from XP.

Computer Weekly also reported that civil servants at the Government Digital Service expressed concerns about the lack of technical standards in the NHS to the then health minister George Freeman.

Freeman was a Department of Health minister until July 2016. In their meeting with Freeman, GDS officials  emphasised the need for a central body to set technical standards across the NHS, with the authority to ensure trusts and other organisations followed best practice, and with the transparency to highlight those who chose not to.

A source told Computer Weekly that Jeremy Hunt was also briefed on the security risks that a lack of IT standards would create in a heavily-federated NHS but it was not considered a priority at that top political level.

“Hunt never grasped the problem,” said the source.

There are doubts, though, that Hunt could have forced trusts to implement national IT security standards even if he’d wanted to. NHS trusts are largely autonomous and GDS has no authority to mandate technical standards. It can only advise.

How our trust avoided being hit

A comment by an NHS IT lead on Digital Health’s website gives an insight into how his trust avoided being hit by the latest cyber attack.  He said his trust had a “focus on perimeter security” and then worked back to the desktop.

“This is then followed up by lots of IG security pop ups and finally upgrading (painfully) windows XP to windows 7…” He added,

“NHS Digital have to take a lead on this and enforce standards for us locally to be able to use.”

He also suggests that NHS Digital sign a Microsoft Enrollment for Windows Azure [EWA] agreement as it is costly arranging such a deal locally.

 “NHS Digital must for me, step in and provide another MS EWA as I am sure the disruption and political fall-out will cost more. Introduce an NHS MS EWA, introduce standards for software suppliers to comply with latest OS and then use CQC to rate organisations that do not upgrade.”

Another comment on the Digital Health website says that even those organisations that could afford the deployment costs of moving from XP to Windows 7 were left with the “professional” version, which “Microsoft has mercilessly withdrawn core management features from (e.g. group policy features)”.

The comment said,

“There are a lot of mercenary enterprises taking advantage of the NHS’s inability to mandate and coordinate the required policies on suppliers which would at least give the under-funded and under-appreciated IT functions the ability to provide the service they so desperately want to.”

A third comment said that security and configuration management in the NHS is “pretty poor”. He added, “I don’t know why some hospitals continue to invest in home-brew email systems when there is a national solution ready and paid for.

“In this recent attack most the organisations hit seem to use local email systems.”

He also criticised NHS organisations that:

  • Do not properly segment their networks
  • Allow workstations to openly and freely connect to each other in a trusted zone.
  • Do not have a proper patch / update management regime
  • Do not firewall legacy systems
  • Don’t have basic ACLs [access control lists)

Three lessons?

  • Give GDS the ability to mandate no matter how many Sir Humphreys would be upset at every challenge to their authority. Government would work better if consensus and complacency at the top of the civil service were regarded as vices, while constructive, effective and forceful criticism was regarded as a virtue.
  • Give the NHS money to spend on the basic essentials rather than nice-to-haves such as a paperless NHS, trust-wide wi-fi, smartphone apps, telehealth and new websites. The essentials include interoperability – so that, at the least, all trusts can send test results and other medical information electronically to GPs –  and the upgrading of medical devices that rely on old operating systems.
  •  Plan for making the NHS less dependent on monolithic Microsoft support charges.

On the first day of the attacks, Microsoft released an updated patch for older Windows systems “given the potential impact to customers and their businesses”.

Patches are available for: Windows Server 2003 SP2 x64Windows Server 2003 SP2 x86, Windows XP SP2 x64Windows XP SP3 x86Windows XP Embedded SP3 x86Windows 8 x86, and Windows 8 x64.

Reuters reported last night that the share prices of cyber security companies “surged as investors bet on governments and corporations spending to upgrade their defences”.

Network company Cisco Systems also closed up (2.3%), perhaps because of a belief that it would benefit from more network spending driven by security needs.

Security company Avast said the countries worst affected by WannaCry – also known as Wannacypt – were Russia, Taiwan, Ukraine and India.

Comment

In a small room on the periphery of an IT conference on board a cruise ship , nearly all of the senior security people talked openly about how their board directors had paid ransoms to release their systems after denial of service attacks.

Some of the companies – most of them household names – had paid ransoms more than once.

Until then, I’d thought that some software suppliers tended to exaggerate IT security threats to help market their solutions and services.

But I was surprised at the high percentage of large companies in that small room that had paid ransoms. I no longer doubted that the threats – and the damage – were real and pervasive.

The discussions were not “off-the-record” but I didn’t report their comments at the time because that would doubtless have had job, and possibly even career ramifications, if I had quoted the security specialists by name.

Clearly ransomware is, as the GCHQ expert Kieran Martin put it, a global business but, as ransoms are paid secretly – there’s not a whisper in corporate annual accounts – the threat has not been taken seriously enough in some parts of the NHS.

The government’s main defence is that the NHS was not targeted specifically and that many private organisations were also affected.

But the NHS has responsibility for lives.

There may be a silver lining if a new government focuses NHS IT priorities on the basics – particularly the structural defects that make the health service an easy target for attackers.

What the NHS doesn’t need is a new set of politicians and senior civil servants who can’t help massaging their egos and trying to immortalise their legacy by announcing a patchwork of technological marvels that are fun to work on, and spend money on, but which gloss over the fact that much of the NHS is, with some notable exceptions, technologically backward.

Microsoft stockpiled patches – The Register

UK government, NHS and Windows XP support – what really happened – Computer Weekly

NHS letter on patches to counter cyber attack

Multiple sites hit by ransomware attack – Digital Health (31 comments)

Lessons from the WannaCrypt – Wannacry – cyber attack according to Microsoft

 

MPs suggest Cabinet Office is losing its grip on departments – but does it care?

By Tony Collins

The Register has an excellent piece by Kat Hall on how the Cabinet Office is losing its grip on Government departments.

Citing the annual report of the all-party Public Accounts Committee, Hall says there are issues where “departments repeatedly don’t do what they have been told or asked to do by the centre”.

An analysis by The Register found that

“government departments are winning significantly more exemptions to splash the cash on expensive IT projects since the departure of former Cabinet Office minister Francis “Mad Frankie” Maude last year”.

Chair of the Public Accounts Committee Meg Hillier said: “After my second year as Chair I am increasingly concerned about the long-term accountability of senior civil servants.

“The game of musical chairs starts as one Permanent Secretary moves on and they all change jobs in the system. And few are in post long enough to have a vested interest in the long-term aims of their department or a project.

“And there is the age-old tension between a department and central Whitehall through the Cabinet Office.”

Universal Credit and HMRC’s plans to overhaul its Aspire IT contract – the biggest in Europe – were outlined as being two areas of concern. As was the Home Office’s Emergency Services Network.

“The Home Office seemed to downplay the risks to the contract and its being caught unawares by the contractor does not reassure us that the Department is on top of the contract or this project. This could cost the taxpayer dear,” it said.

Comment:

It’s hard to argue with a comment on Hall’s piece by @JagPatel3 who suggests that some in Whitehall are as preoccupied with spin as with the efficient delivery of public services.

“… Government is preoccupied with presentation, manipulation of words and the dark art of spinning – instead of working on its programme of reform to deliver public services efficiently, to satisfy the wants, needs and expectations of the electorate.

“The political imperative of needing to put a positive slant on everything the Government does or will do, irrespective of whether it is true or not, is the reason why spin has become the centrepiece of this Government’s communications strategy.

“And because Government has got a monopoly on inside information (enabling it to maintain extremely tight control), it uses spin to divert attention away from the key issues that really matter to citizens …

“the eagerness with which senior Civil Servants have complied with their political masters’ desire to see policy announcements framed around presentation and spin, at the expense of substance, would explain why their skills set has been narrowed down to this single, dark art.”

The commentator also says that the “intense focus of attention on presentation alone has resulted in a massive gap opening up between the leadership and lower ranks of the Civil Service, who have to deal with the reality of delivering public services on the ground, on a day-to-day basis, which has in itself, led to alienation and disaffection”.

A good summary. Many ordinary civil servants are doing the hard work of delivering public services while a few of their masters are preoccupied with keeping what they do secret and justifying or defending all else that is published in National Audit Office reports, other third-party reports or leaked emails.

It’s hardly surprising the Cabinet Office is losing control of departments. Since Maude’s departure it doesn’t want control. It has become clear that it wants, in a hassle-free way,  to continue with Sir Humphrey’s non-integrated approach to government.

The Cabinet Office is just another Whitehall department. Why would it want to be an “enforcer?”

After a major IT failure, how did Barts NHS trust manage its image?

By Tony Collins

It sounded serious. Under the headline

“Cancer patients in limbo as five hospitals suffer ‘major’ IT crash”

the Daily Telegraph said,

“Hundreds of cancer patients have been denied treatment at one of England’s biggest hospital trusts due to a major IT failure that ground basic services to a halt.

“Doctors at five large London hospitals have reported 11 days of “chaos” after the systems used to prescribe chemotherapy doses and share x-ray and MRI images broke down on April 20.

“Barts Health NHS Trust said at least 136 operations had been cancelled due to the crash, as well as “hundreds” of cancer treatment sessions.

“The computer failure also means frantic staff have been unable to process blood tests for all but the most critical cases…

“A doctor at the Royal London Hospital told the Daily Telegraph: ‘We have been forced to leave sick patients on the ward while we go down 16 floors to catch a glance at an x-ray image, then come back and make treatment decisions based on a hazy recollection of it…

“An email sent by managers to staff last week said the crisis had forced cancer teams to rebuild patient records ‘from scratch’.

A medic at Whipps Cross hospital was quoted as saying that a lot of people were stuck in hospital needlessly which increased the likelihood of infection.

The trust runs Mile End Hospital, Newham University Hospital, The Royal London Hospital, St Bartholomew’s Hospital and Whipps Cross University Hospital as well as other NHS sites.

The Barts trust website says it delivers “high quality compassionate care to the 2.5 million people of east London and beyond”.

It has a turnover of £1.25 bn and a workforce of 15,000, making it the largest NHS trust in the country.

According to Health Service Journal, an internal email from Barts’ chief clinical information officer Tim Peachy said the IT failure was primarily a result of an “unexpected failure of a small number of physical disks on which data is stored”.

At one point the trust was manually processing blood test results and X-rays, and arranging for porters at its hospitals to hand deliver paperwork to clinicians.

Barts’ reputation 

In the light of the failure and disclosures in Health Service Journal, Barts confirmed the IT problems in statements to the media. It also contacted patients who were affected by the problems. A Barts statement this week said,

“A major computer equipment failure on Thursday 20 April resulted in a number of IT applications being unavailable to staff.

“‘Unfortunately, it has been necessary to cancel 136 operations, representing about 2.5% of our usual weekly in-patient activity. Several hundred chemotherapy appointments have been cancelled, however we have now recovered the chemotherapy prescribing database.

“Clinical teams have completed a patient-by-patient review to ensure that the appropriate course of action is taken for each of them, endeavouring to keep the disruption to an absolute minimum.

“We apologise to those affected and will reschedule their appointment for as soon as we are able.

“A number of applications have been affected to varying degrees. We have made significant progress in many areas including pathology (blood testing), with image viewing now also restored across the Trust. There are still some other areas where it will take time before we are on track again.”

It added,

“We continue to work urgently to maintain the operational resilience of our services, using tried and tested contingency plans to keep our patients safe.”

Despite the seriousness of the problems, the effect on patients and the uncertainties that media coverage might have created in the minds of those intending to go to Barts’ hospitals, the trust made no mention of the difficulties on its website – where it has a “latest news” section –  or on Twitter.

Barts uses Twitter for good news announcements, comments and congratulations, sometimes with dozens of daily tweets.

But why no mention of the IT problems?

On this point, a Barts spokeswoman said,

“We do not rely on social media to update patients. As a proportionately small number of people will be impacted on by the IT situation we are communicating directly to those affected including at outpatients clinics and via phone, letter as well as through communications with our healthcare partners including GPs.”

Comment

In its media statements Barts has been more open than some NHS organisations.

The usual NHS cycle after a major IT-related failure is a statement saying teething problems have been resolved, or are being resolved, followed by a succession of similar statements over the next few days, weeks or months when it becomes clear the problems haven’t been resolved.

This is what happened with e-Referral Service and Capita’s problems handling GP support services.

That hasn’t happened at Barts. But despite its openness with the media, it’s odd  the trust has published many congratulatory tweets in the past two weeks without a mention of any IT-related problems. They are not even alluded to.

It’s also odd that on its website the Barts “Latest News” section has no mention of the difficulties. But the website does have various good news announcements, including a reference to a positive Care Quality Commission report in April 2007.

Trusts do not have to account to patients, Parliament or anybody for IT-related problems. They are under no obligation to apologise to patients whose stays in hospital are unnecessarily prolonged, or whose appointments, operations and blood tests are cancelled or delayed because of IT-related difficulties.

Back-up systems? 

They also have no obligation to give the public any reason for the failure or explain why there was no back-up system that ensured patients were unaffected.

But amid so many positive announcements, statements and comments to the public on its website and on Twitter, should Barts have left out the other side of the story?

The NHS is an organisation that’s attuned to promulgating good news. It’s rare for a trust board paper and or a trust website to have anything but a good news feel to it.

But telling the public one side of the story does not encourage the public to believe officialdom when it says: “Trust us. We know what we’re talking about.”

Thank you to Zara Pradyer for letting me know about the Daily Telegraph article.

 

Some officials “smuggle their often half-baked proposals past ministers” says Cabinet Office adviser who quits

By Tony Collins

Jerry Fishenden has resigned from the Cabinet Office‘s Privacy and Consumer Advisory Group after nearly six years. First he was its chairman and more recently co-chairman.

The Privacy and Consumer Advisory Group comprises privacy and security experts who give the government independent analysis and guidance on personal data and privacy initiatives by departments, agencies and other public sector bodies. This includes GOV.UK Verify.

The group’s advice has had the citizens’ interests in mind. But the group might have been seen by some Whitehall officials as having an open and frank “outsiders” culture.

Francis Maude, then Cabinet Office minister, helped to set up the group but he left in 2015 and none of his replacements has had a comparable willingness to challenge the civil service culture.

Maude welcomed the help of outsiders in trying to change the civil service.  He tried to bring down the costs of Government IT and sought to stop unnecessary or failing projects and programmes. He also wanted to end the “oligopoly” of a handful of large IT suppliers. But Maude’s initiatives have had little continuing support among some Whitehall officials.

Fishenden said in a blog post this week that Maude had wanted the Privacy and Consumer Advisory Group to be a “critical friend” – a canary that could detect and help fix policy and technology issues before they were too far down the policy / Bill process.

“The idea was to try to avoid a repeat of previous fiascos, such as the Identity Card Act, where Whitehall generalists found themselves notably out of their depth on complex technical issues and left Ministers to pick up the pieces.”

He added that “since Francis Maude’s departure, there has been only one meeting” with subsequent Cabinet Office ministers.

“Without such backing, those officials who find the group’s expert reviews and analyses “challenging” have found it easier to ignore, attempting instead to smuggle their often half-baked proposals past Ministers without the benefit of the group’s independent assistance…

“Let’s just hope that after the election the value of the group will be rediscovered and government will breathe life back into the canary. Doing so would help realise Francis Maude’s original purpose – and bring significant benefits to us all, whether inside or outside of government.”

Comment

One of the Privacy and Consumer Group’s strengths has been its independent view of Government IT-related initiatives  – which is probably the main reason it has been marginalised.

Fishenden’s departure is further confirmation that since Maude’s departure, the Cabinet Office – apart from the Government Digital Service – has settled back into the decades-old Whitehall culture of tinkering with the system while opposing radical change.

While Whitehall’s culture remains unreformable, central government will continue to lose the best IT people from the private sector. Some of these include the former Government Digital Service executive director Mike Bracken, Stephen Foreshew-Cain, who took over from Bracken, Janet Hughes, programme director of Verify,  Andy Beale, GDS’s chief technology officer, Paul Maltby, GDS’s director of data and former Whitehall chief information officers Joe Harley, Steve Lamey, Andy Nelson and Mark Dearnley.

The unfortunate thing is that a few powerful career civil servants, including some permanent secretaries, will be delighted to lose such outsiders.

Jerry Fishenden is simply the latest casualty of a civil service tradition that puts the needs of the department before those of the citizen.

It’s a culture that hasn’t changed for decades.

The canary that ceased to be – Jerry Fishenden’s blog on his departure

Privacy and Consumer Advisory Group

Does Universal Credit make a mockery of Whitehall business cases?

By Tony Collins

Does Universal Credit make a mockery of this Treasury guidance on business cases?

It’s supposed to be mandatory for Whitehall departments to produce business cases. They show that big projects are “unequivocally” affordable and will work as planned.

But Computer Weekly said yesterday that the Department for Work and Pensions has not yet submitted a full business case for Universal Credit although the programme has been running for six years.

The result is that the Universal Credit IT programme may be the first big government computer project to have reached the original completion date before a full business case has been finalised.

Its absence suggests that the Department for Work and Pensions has not yet been able to produce a convincing case to the Treasury that the IT programme will either work or be affordable when it is due to roll out to millions of claimants.

The absence also raises a question of why the Department for Work and Pensions was able to award contracts and proceed with implementation without having to be accountable to Parliament for milestones, objectives, projected costs and benefits – all things that would have been recorded in the full business case.

If the DWP can proceed for years with project implementation without a full business case, does this mean that other Whitehall department need have no final structured plan to justify spending of billions on projects?

Will Universal Credit work?

By early March 2017, fewer than 500,000 people were on Universal Credit. On completion, the system will be expected to cope with seven million claimants.

Although the rollout of the so-called “digital” system – which can handle all types of claim online – is going well (subject to long delays in payments in some areas and extreme hardship for some), there are uncertainties about whether it will cope with millions of claimants.

Universal Credit campaigner John Slater has been unable to obtain any confirmation from the DWP on whether it is planning to complete the rollout by 2022 – five years later than originally scheduled.

Business cases present arguments that justify the spending of public money. They also provide a “clear audit trail for purposes of public accountability,” says Cabinet Office guidance on business cases.

But hundreds of millions has already been spent on Universal Credit IT, according to the National Audit Office.

Business cases are mandatory … sort of

The Treasury says that production of business cases is a

“mandatory part of planning a public sector spending proposal …”

Yesterday, however, Computer Weekly reported that,

“Amazingly, given the programme has been going since 2011, the full business case for Universal Credit has still not been submitted or signed off by the Treasury – that’s due to take place in September this year.”

The Treasury says that preparation of the Full Business Case is “completed following procurement of the scheme – but prior to contract signature – in most public sector organisations.”

But by March 2013, the Department for Work and Pensions had already spent about £303m on Universal Credit IT, mostly with Accenture (£125m), IBM (£75m), HP (49m) and BT (£16m), according to the National Audit Office.

Why a business case is important

The Treasury sums up the importance of business cases in its guidance to departments,

“… it is vital that capital spending decisions are taken on the basis of highly competent professionally developed spending proposals.

The business case provides a

“structured process for appraising, developing and planning to deliver best public value.”

The full business case, in particular, sets out the

  • contractual arrangements
  • funding and affordability
  • detailed management arrangements
  • plans for successful delivery and post evaluation.

In the absence of a full business case the DWP was able to start the Universal Credit IT programme with little structured control on costs. The National Audit Office found in 2013 that there was

  • Poorly managed and documented financial governance
  • Limited evidence that supplier invoices were properly checked before payments were made.
  • Inadequate challenge of purchase decisions
  • Insufficient information on value for money of contracts before ministers approved them
  • Insufficient challenge of suppliers’ cost changes
  • Over-reliance on performance information from suppliers that the Department for Work and Pensions didn’t validate.
  • No enforcement by the DWP of key parts of the supplier contracts

Comment

Officials at the Department for Work and Pensions have gone to the bank for money for a new business venture – the building of Universal Credit IT – and said in effect,

“We’ll let you have an outline business case that may change a few times and in a few years, perhaps on completion of the programme or thereabouts, we’ll provide a full business case. But we’d like the money now please.”

In response the bank – HM Treasury – has replied in effect,

“You’re supposed to supply a full business plan before we decide on whether to give you the money but we know how important Universal Credit is.

“We’ll tell you what: we’ll let you have a few tens of millions here and there and see how you get on.

“For the time being, without a full business case, you’re restricted to an IT spend of around £300m.

“In terms of the eligibility criteria for the money, you can let us know what this should be when you’re a few years down the road.

“We accept that you’ll be in a much better position to know why you should be given the money once you’ve spent it.”

Does “mandatory” mean anything when there is no sanction against non-compliance?

And when the DWP is able to embark on a multi-billion pound programme without submitting a full business case until after the original completion date (2017), what’s the point of a business case?

The fact that the DWP is six years into implementation of Universal Credit without a full business case suggests that departments make up the rules as they go along.

What if the Treasury rejects the Universal Credit business case when it’s eventually submitted?

Will the DWP wait another few years to submit a case, when an entirely new set of officials will be in place? By then, perhaps, the Universal Credit rollout will have finished (or been aborted) and nobody at that stage could be effectively held to account if the scheme didn’t work or money had been wasted.

If Whitehall routinely waits until an IT-based programme is finished before presenting a full business case for Treasury approval, there’s nothing the Treasury can do if it wants and needs the programme.

Sir Humphrey is all-powerful.  Why should officials worry about presenting full business cases on programmes they know there’s a political imperative to deliver?

Can DWP meet its revised 2022 target for completion of Universal Credit? – Computer Weekly

Treasury guidance on business cases

 

 

Will MPs’ report on Capita’s BBC contract make any difference?

By Tony Collins

At one level, Capita’s contract to handle most of the BBC’s TV licensing work is, in general, a success, at least according to statements made to the media.

Were it not for the National Audit Office and the Public Accounts Committee, a fuller story would not have emerged.

Today in The Guardian, a BBC spokesperson speaks of the Capita TV licensing contract in glowing terms. Through the contract, the BBC has reduced collection costs by 25% and increased revenue for programmes and services.

A Capita spokesperson spoke in similar terms. Capita has helped the BBC to collect more TV licence fee revenue every year since 2010-2011.

The only blip in the contract had seemed to be the heavy-handed tactics of some Capita staff. The Daily Mail reported in February 2017 that vulnerable people were hounded as some Capita staff tried to catch 28 TV licence evaders a week for bonuses of £15,000 a year.

This blip aside, has anything else gone wrong? There’s no hint of any technological problems on Capita’s website – or the BBC’s.

The BBC reported in 2011 that Capita will transform the TV licensing service, “using advances in technology and analytics to increase revenue and reduce costs”.

Capita’s website has a case study on its work for the BBC that refers to cost savings of £220m over the life of the contract, organisation-wide efficiencies and “protected brand image” among other benefits.

In December 2016, Capita described the “partnership” with the BBC  as a “success”.

The bigger picture

Capita processes TV licence payments, collects arrears and enforces licence fee collection. Its current contract with the BBC began in July 2012 and, after a recent renegotiation, ends in 2022 with the option to extend by up to a further five years.The BBC paid Capita £59 million in 2015–16.

The BBC has had a long-standing ambition to improve its main TV licensing databases so that they are structured by individual customers rather than households.

This was one of the hopes for the contract with Capita but it hasn’t happened. Capita had partly subcontracted work on the BBC’s legacy databases to CSC Computer Sciences.

Manual workarounds

The BBC, in its contract with Capita, aimed to upgrade ICT as part of a wider transition programme. The BBC paid Capita £22.9m for parts of the programme that were delivered, including restructuring contact centres, updating the TV Licensing website and upgrading handheld units for field staff.

The Public Accounts Committee says in today’s report,

“However, improvements with a contract value of £27.9m, primarily related to replacing legacy ICT systems, were not delivered by Capita and its subcontractor (CSC), and were not paid for by the BBC.

“As a result of the transition programme being only partly completed and subsequently stopped, the BBC and Capita currently have to do resource-intensive manual workarounds between inefficient ICT systems.

“Capita informed us that it was bearing the additional costs associated with undelivered elements of the transition programme. However, the BBC has had to allocate £9m to Capita to support the ongoing use of legacy systems, costs which the BBC told us were compensated for elsewhere in the renegotiated contract.

“It is unclear to us why ICT database improvements have proved so difficult over the last 15 years, particularly when competitors and other organisations can make similar changes.

“The BBC acknowledges that its current database is not fit for purpose for the future but does not yet have a clear plan to replace it.”

Comment

All outsourcing contracts have their strengths and failures – including early promises that don’t come to anything.

But it’s unlikely councils and other public sector organisations that are seriously considering outsourcing will take into account the past failures and broken promises of their potential suppliers.

If officials and councillors want to outsource IT and other services they probably will, whatever the record of their favoured potential suppliers.

They will see reports of the National Audit Office and Public Accounts Committee as biased towards negative disclosures.

Indeed the BBC and Capita, in their responses to today’s TV licensing report of the Public Accounts Committee, have drawn attention to the positive aspects of the report and not mentioned the technological failures.

Where does this leave councils and other organisations that are considering IT-related outsourcing and are seeking reference sites as part of the bid process?

Will those reference sites give only the positive aspects and not mention, or successfully deprecate, any media, PAC or NAO reports on contract failures?

Negative findings by the National Audit Office and Public Accounts Committee are usually important. Were it not for their scrutiny would not know how public money is being spent and misspent.

But their reports will have little or no effect as warnings to organisations that want to outsource.

Public Accounts Committee – BBC Licence Fee – 26 April 2017

 

Whitehall to auto-extend outsourcing deals using Brexit as excuse?

By Tony Collins

Type of government procurement spend 2014-2015. ICT is the top item.
Source: National Audit Office

Under a headline “UK outsourcing deals extended because of Brexit workload”, the Financial Times has reported that “hundreds of government contracts with the private sector that were due to expire are to be automatically extended because civil servants are too busy with Brexit to focus on new and better-value tenders”.

The FT says the decision to roll over the contracts could prove expensive for taxpayers because it limits competition and undermines government efforts to improve procurement.

A “procurement adviser to the government” whom the FT doesn’t name, said more than 250 contracts were either close to expiring or had already expired in 2016-17. The adviser told the FT,

“Brexit has pushed them down the list of priorities so there are lots of extensions and re-extensions of existing deals.”

The adviser added that this was the only way civil servants could prioritise the huge increase in Brexit-related work since the referendum.

Extensions

The FT provides no evidence of automatic contract extensions or the claim that deals will be extended because of the civil service’s Brexit workload.

There is evidence, however, that Whitehall officials tend to extend contracts beyond their original expiry date.

In a report published this year on the Cabinet Office’s Crown Commercial Service, the National Audit Office identified 22 framework contracts that were due to expire in 2016-17. Half of them (eleven) were extended beyond their original expiry date.

[The Crown Commercial Service was set up in 2014 to improve state procurement.]

The NAO also found that Whitehall departments – and the Crown Commercial Service – have been awarding contracts using expired framework deals, even though this contravenes public contracting regulations.

In 2015-16, 21 of the 39 frameworks that were due to expire were extended without competition or market testing, according to the NAO.

One example of an extended contract is a deal between Capita and the Department for Work and Pensions which started in 2010. Capita provides eligibility assessments for the personal independent payment allowance, which supports for people with long-term ill health or disability.

The five-year deal was extended by two years until July 2019.

Capita has also won a three-year extension to a contract with the Pensions Regulator and the BBC has extended a deal with Capita that was signed originally in 2002 to June 2022 – a total of at least 20 years.

Open competition?

The NAO has found that extending ICT contracts may not always be good for taxpayers. In the later years of their government contracts, suppliers tend to make higher margins (though not always).

There are also suggestions that civil servants will sometimes sign contract extensions when the performance of the supplier does not meet expected standards.

On ICT, the Cabinet Office asks central departments to complete a return every six months for each business process outsourcing and facilities management contract above £20m with strategic suppliers.

The survey asks whether the contract is being delivered on time, to scope, to budget, to the appropriate standards, and whether there have been any disputes.

In one study of government contracts with ICT suppliers, the NAO found that, of 259 returns from departments, 42 highlighted problems that included,

  • failure to achieve milestones
  • dissatisfaction with quality of outputs
  • errors and other issues with delivery
  • poor customer engagement and end user dissatisfaction and
  • failure to meet key performance indicators.

Comment

For taxpayers there is some good news.

A break-up of “Aspire”, the biggest IT outsourcing long-term deal of all, between HMRC and Capgemini (and to a lesser extent Fujitsu) – worth about £9bn – is going ahead this June. An HMRC spokesman says,

“HMRC is on track to complete the phased exit from Aspire, as planned, by June 2017.”

And according to Government Computing, Defra’s IT outsourcing contracts with IBM and Capgemini under a £1.6bn contract called “Unity” are due to expire in 2018 and there are no signs the deals will be extended.

But the Department for Work and Pensions’ huge IT outsourcing contracts with the same major suppliers are renewed routinely and not always with open competition. The DWP says on its website,

“DWP contracts are awarded by competition between potential suppliers, unless there are compelling reasons why competition cannot be used.”

The DWP doesn’t define “compelling”. Nor is it clear whether its auditors look at whether the DWP has put up a compelling case for not putting a large IT contract out to open competition.

In 2014 the Public Accounts Committee, after investigating major suppliers to government, concluded,

“Government is clearly failing to manage performance across the board, and to achieve the best for citizens out of the contracts into which they have entered.

“Government needs a far more professional and skilled approach to managing contracts and contractors, and contractors need to demonstrate the high standards of ethics expected in the conduct of public business, and be more transparent about their performance and costs”.

Breaking up is hard to do

The break up of the huge Aspire IT outsourcing contract at HMRC is an exception, not the rule. The NAO has found that civil servants regard their major incumbent suppliers as safe and less risky than hiring a smaller company (that’s not steeped in Whitehall’s culture).

The NAO has also found that in some cases officials don’t know whether their suppliers are performing well or not. On many ICT contracts there is “open book” accounting, but not all departments have the staff or expertise to check regularly on whether their suppliers’ profits are excessive.

If Whitehall, with exceptions, is continuing to roll over contracts whether it’s legal to do so or not, what incentive exists to stick to the rules?

Brexit?

The FT story suggests Brexit is the reason hundreds of contracts are to be extended automatically. There’s probably truth in the automatic extension of some contracts – but it’s unlikely to be because of Brexit.

It’s unlikely that the civil servants involved in Brexit will be the same ones who are handling ICT contract extensions. That said, Brexit will inevitably put a higher workload on lawyers working for government.

If contracts are being extended automatically, it’s probably because that’s the way it has always been, at least within living memory.

While Sir Humphrey and his senior officials remain only nominally accountable to Parliament for how they spend taxpayers’ money, the easiest option of renewing or extending existing contracts will usually be seen as the best option.

It can be justified with “compelling” arguments such as a need to make an urgent decision in difficult circumstances, or the absence of alternative suppliers who have the necessary skills or the financial strength to accept the risks of failure.

Will anything change?

Until departments have to publish contemporaneously their intentions to award contracts without open competition or there is effective accountability within the civil service for major decisions, little is likely to change.

It hasn’t happened yet and there’s no reason to believe it will.  Many politicians including prime ministers have tried to reform the civil service and they haven’t ruffled a single carpet in the corridors of Whitehall.

As Antony Jay, co-writer of Yes Minister,  said in January 2013,

“The central anomaly is that civil servants have years of experience, jobs for life, and a budget of hundreds of billions of pounds, while ministers have, usually, little or no experience of the job and could be kicked out tomorrow.

” After researching and writing 44 episodes and a play, I find government much easier to understand by looking at ministers as public relations consultants to the real government – which is, of course, the Civil Service.”

In short, Brexit is likely to be officialdom’s up-to-date excuse for carrying on much as before.

Thank you to @TimMorton2 for alerting me to the FT article.

Another Whitehall failure: no officials responsible, fluid facts and doubtful ethics. Plus ça change?

By Tony Collins

It’s rare for truth to emerge from the ashes of a failed contract.

The disastrous contract between Siemens and the BBC (the so-called Digital Media Initiative) was a rarity. Various reports provided confidence that the relevant facts had emerged.

It’s more usual for MPs to report that they haven’t got to bottom of what happened after a Whitehall contract failure.

Indeed today’s report by the Public Accounts Committee says of its inquiry into PA Consulting’s contract with the UK Trade and Investment:

“We cannot remember a previous inquiry in which so many witnesses corrected their evidence after a public session.”

UK Trade and Investment, now the Department for International Trade,  helps UK businesses to export more goods and services and encourages overseas organisations to invest in the UK.

It is funded by the Department for Business, Innovation & Skills and the Foreign & Commonwealth Office.

In May 2014, UK Trade and Investment’s  officials entered into a three-year contract with PA Consulting for the supply of consultants in a contract that involved ICT support.

On small example of unclear facts: in its bid, PA stated that cost categories including ICT were “already included in the costings and will not be charged for separately”. This implied that ICT would be included in the consultants’ day rates.  Today’s Public Accounts Committee report says,

“However, there were separate charges in the pricing schedule for HR, ICT,  legal and professional, quality, and knowledge management.”

After the contract had started, officials became concerned about:

  • the way PA had priced the contract
  • PA’s transparency in its communications with Whitehall.

The contractual relationship eventually broke down and officials terminated the contract in January 2016.

The two sides agreed a settlement in which the taxpayer would pay the balance of PA’s outstanding invoices less a £3m reduction. Officials paid £18.8m for the first 11 months of the contract.

Labour MP Meg Hillier, chairman of the Public Accounts Committee, said today (5 April 2017),

“Even now, ten months after the parties reached a settlement and four months after we took oral evidence, our Committee cannot say with confidence that it has got to the bottom of what happened.”

Poor record-keeping

Even the National Audit Office was unable to obtain a full picture. The NAO said in its 2016 report on PA’s contract,

“Understanding exactly what happened in letting and negotiating this contract is difficult due to the lack of proper documentation, the disagreement between parties and, now, the absence of a number of people who were involved on either side.”

MPs on the Public Accounts Committee say that Whitehall officials:

  • did not keep proper records
  • negotiated significant changes to the contract with PA when they should have gone back to the market
  • pushed for a signing of the contract before they had finished negotiations.

For its part, PA “fell well short of the appropriate duty of care that we expect contractors to demonstrate when in receipt of taxpayers’ money”.

According to the Public Accounts Committee, PA

  • took advantage of the department’s poor decision making
  • sold Whitehall a service it is not clear it needed
  • failed to give the fair breakdown of its costs and profit that officials had asked for
  • used the negotiations to pass on costs to Whitehall that it had said in its bid that it would bear
  • increased its profit from the contract while telling officials that its profit had not increased.

PA Consulting obfuscation?

The Committee says in its report,

“PA has not convinced us that it takes full responsibility for its actions. Its many explanations of its charges both at the time and since have been loosely worded, inconsistent and seemingly designed to obfuscate.

“It is unclear to us how such behaviour would be possible in a well managed professional practice.”

The Committee adds,

“Government’s lack of commercial expertise to get the best deals on behalf of the taxpayer has been a regular cause for concern for this Committee.”

In 2015 RSM UK Consulting produced a draft report on the contract. It included a finding that PA had “consistently made incorrect and misleading representations relating to £3.9m of the overheads charged”.

PA disputed RSM’s findings, stating that it had invoiced according to the agreed charging mechanism.

Overly long contract?

The contract was 596 pages – “difficult to read, understand and use, for a relatively simple service”. The Committee adds,

“The contract incorporates the ITT and bid, both of which are focused on the outcomes and how they will be achieved, and not on the way the contract would actually be run and charged for.

“Furthermore, the bid and contract are not clear on important aspects of the pricing and are often self-contradictory.”

[One would have thought that after decades of practice, Whitehall departments would understand how to commission a clear and unambiguous contract.]

Comment:

There is not even any evidence that some key decisions by Whitehall officials were approved by any formal decision-making body.

This and other findings by the National Audit Office and the Public Accounts Committee are astonishing, not because of their momentousness but because of they are almost routine.

There were similar findings after National Audit Office investigations into early spending on the Universal Credit IT programme.

It’s beginning to look as if Whitehall officials can sometimes hand over money to the private sector without any firm controls at all, which could encourage corruption and, at the least, incompetence and waste.

What’s the civil service’s solution?

To make sure that no officials are held responsible. The civil service is all about collective responsibility. In other words no responsibility.

Is the civil service’s message to the private sector now clear: “Get whatever Whitehall business you can because though the terms of the contract may be tough, what we pay you afterwards may be, for us, a matter of indifference.”

PA Consulting’s contract with UK Trade and Investment

 

Large suppliers still dominate government IT

By Tony Collins

In 2012, the then Cabinet Office minister Francis Maude, lamented the high costs of government IT and spoke of an “oligopoly” of large suppliers. He suggested things would change.

“… contracts were consistently awarded to a limited number of very large suppliers on long-term exclusive contracts.

“As a result there was inadequate competition and an abdication of control. The concept of having one supplier, aggregated supply, increased project risk and removed competitive tension.

“The Government repeatedly found itself paying large amounts for systems that were delivered late, over budget and which often did not fully meet the original policy requirement.  If indeed, they were delivered at all. There are plenty of well-documented disasters – such as DH’s now terminated National programme for IT.

“Ultimately, the last Government lost control of IT – it outsourced not only delivery, but its entire strategy and ability to shape the future of our public services.

“At the same time smaller, more innovative and efficient suppliers were finding themselves locked out of the supply of services to Government because of what was described by Parliament as a powerful “oligopoly” of large suppliers.

“Procurements took so long only the big companies could absorb the cost – which they naturally passed on to us.

“All in all, we had an approach that was bad for users, bad for the taxpayer and bad for growth.”

Public sector IT spending was up to £20bn a year, he said, adding that “public sector productivity was actually declining”.  He outlined how things were changing.

What has happened since?

A report published today by the National Audit “Digital Transformation in Government” raises a question of how much has changed.

Efforts to boost the SME share of government IT business “have had some impact”, says the National Audit Office, but it adds that “most government procurement with digital and technology suppliers continues to be with large organisations”.

“In 2015-16, 94% of such spending was with large enterprises, a fall of less than one percentage point since 2012-13.”

Today’s NAO report is mainly about the Cabinet Office’s Government Digital Service – GDS. It points out GDS’s strengths and weaknesses but in general does not give any advice on the sensitive point of whether it should have more or less influence on government IT.

On digital transformation, it says that the work of the NAO shows that attempts to transform government have had mixed success.

“Many public services appear increasingly unsustainable. Those responsible for major programmes have continued to exhibit over-optimism and make slow progress towards their objectives.”

It adds,

“Digital transformation has a mixed track record across government. It has not yet provided a level of change that will allow government to further reduce costs while still meeting people’s needs.

“GDS has also struggled to demonstrate the value of its own flagship initiatives such as Verify, or to set out clear priorities between departmental and cross-government objectives.

“GDS’s renewed approach aims to address many of these concerns as it expands and develops into a more established part of government. But there continues to be a risk that GDS is trying to cover too broad a remit with unclear accountabilities.

“To achieve value for money and support transformation across government, GDS needs to be clear about its role and strike a balance between robust assurance and a more consultative approach.”

Comment

The National Audit Office report is strong on facts and quality of research but avoids the big question of how GDS can bring about change when the top brass in departments prefer autonomy to what they see as GDS’s interference.

GDS’s existence goes to the heart of how the civil service runs. It is one part of the civil service trying to bring about change in other parts of the civil service.

And the evidence so far is that the civil service doesn’t like change.

The NAO report disappoints because it doesn’t address how government IT is to change if departments are to continue to run empires unchallenged by GDS or the heads of the civil service. Sir Humphrey is still king.

GDS scrutinises departmental IT spending – spending applications are reviewed by a team of eight people within GDS’s Standards Assurance team – but, much to Sir Humphrey’s delight, GDS’s influence seems to be waning.

When Jack Straw was Justice secretary, he told MPs in 2007 that when he abandoned projects there was a fuss at first and soon nobody noticed the project did not exist.

“There is always the option to abandon things. I did that in the Foreign Office with much complaint that the world might end.

“What happened was that we saved a lot of money and no one ever noticed the fact that that scheme did not exist…it is very frustrating that so many people, including the private sector, are taken in by snake oil salesmen from IT contractor who are not necessarily very competent and make a lot of money out of these things. I am pretty intolerant of this.”

How much has changed? Outsiders including Jack Straw and Francis Maude, together with insiders such as Chris Chant have pointed to the need for major changes in the way departments manage huge IT budgets and there have been some improvements: HMRC’s is breaking up its monolithic “Aspire” contract, citizens may notice that it is possible now to renew passports and driving licences online and GDS has had an impact in making departments think hard about whether they really need to spend the amounts they do on major IT contracts.

But major change in the costs of government IT seems not just a long way off but unattainable while the dominance of Sir Humphrey remains unchallenged.

Digital Transformation in Government – NAO report

Shedding new light on the Post Office Horizon controversy?

By Tony Collins

Dozens of families gathered in the ballroom of a Hilton hotel to hear independent investigators announce the most likely cause of an air crash that killed 132 air passengers.

Some wondered whether official investigations into air crashes always ended up protecting powerful corporate interests. For several years the manufacturer Boeing had denied that a technical malfunction was the cause of the crash. It blamed the pilots.

This was the longest inquiry in the history of the National Transportation Safety Board, an investigative organisation funded by the US government. Congress has mandated the Board’s independence and objectivity.

At first, each Boeing 737 incident was treated as a single unique event.  In the absence of any clear evidence of a technical malfunction, suspicion fell on the pilots.

The 737 is, after all, the best-selling commercial jet airliner in history. It has an extraordinary safety record.

Then evidence began to mount that various 737 incidents might have been linked.

After thousands of tests over several years, air crash investigators made a discovery – that a particular technical malfunction could, after all, have caused the incidents.

It was an intermittent malfunction – and one that occurred in a rare set of circumstances. It left no trace. It might have caused a succession of seemingly-unique major incidents.

Now the final verdict on the likely cause of USAir Flight 427’s destruction was imminent. As families sat in silence at the Hilton Hotel, Springfield, Virginia, five board members of the National Transportation Safety Board voted – in public – on whether they accepted the findings of their staff investigators who’d pointed to the likely cause being a technical malfunction, not the pilots.

The vote was unanimous; and some relatives wept.  The probable cause was not the pilots. It was “most likely” to have been a technical malfunction.

Boeing accepted the final report into the crash of Flight 427. “We respect the Board’s opinion,” said Boeing after the vote. It made rudder-related design changes that eventually cost more than $100m.

Human or machine?

What do various incidents involving Boeing 737s have to do with a campaign for justice for 198 former sub-postmasters and their families?

At issue in both cases is whether human or machine was to blame for a plethora of incidents.

Former sub-postmasters, who used to run local post offices across the UK, say that technical malfunction, or a combination of human error and unusual, unexpected equipment behaviour, was the cause of their distress, misfortune, jailing or bankruptcy.

The Post Office blamed them for losses shown on its “Horizon” system and required that they pay the shortfall in question. This led to financial ruin for some of them. The Post Office insisted its equipment was not at fault. It pointed to the lack of evidence of any technical malfunction.

But investigations into rare crashes of 737s show that it’s possible for a major corporation to be mistaken when it clears its own equipment and blames the equipment’s human operators.

The 737 investigations found that “no evidence of a technical malfunction” did not necessarily mean “no technical malfunction”.

The UK government reached a similar conclusion at the end of a campaign by families to set aside an RAF finding of gross negligence against two pilots, Flight Lieutenants Jonathan Tapper and Rick Cook, who died when a Chinook helicopter, ZD576, crashed on the Mull of Kintyre in June 1994.

For 16 years the RAF and Ministry of Defence insisted that there was no evidence of a relevant technical malfunction on the last flight of Chinook ZD576. They blamed the pilots for the crash. But leaked MoD technical papers established that the Chinook’s engine computer systems could fail in unpredictable ways – sometimes intermittently – and leave no evidence.

In the end – after a 17-year campaign for justice by the pilots’ families – the UK government set aside the RAF’s finding against Tapper and Cook, mainly because of doubts over whether the pilots or technical malfunction, or a combination of both, caused the crash.

Arguably, the Chinook and 737 controversies established the principle that, despite the absence of firm evidence of a technical malfunction, a major incident could still be caused by one, or a series of them.

This may be an important consideration in Post Office cases because, in some criminal trials of sub-postmasters, the absence of evidence of a technical malfunction that caused the losses shown on Horizon has counted against the defendants.

It counted against former sub-postmaster Lee Castleton who disputed in a civil action the Post Office’s claim that he owed amounts totalling £27,000. These sums were shown on Horizon as losses.

The judge in the case said, “It is inescapable that the Horizon system was working properly in all material respects.” Castleton lost the case and was left with costs of £321,000. The following year he filed for bankruptcy.

In a separate case, a criminal hearing where former sub-postmaster Seema Misra was the defendant, a jury agreed with the Post Office’s case that the Horizon system was tried and tested, had been in use at thousands of Post Offices for several years, and was fundamentally reliable and robust.

Misra was jailed for the theft of £75,000 in a case based on the Post Office’s computer evidence. She said she hadn’t taken a penny.

When sub-postmasters could not prove the existence of a fault on Horizon that explained the losses, the conclusion was that they were personally responsible for the shortfall.

About 30 of the 198 individual complaints against Horizon are from former sub-postmasters who received criminal convictions over the losses.

Boeing and the Post Office

With its turnover of about $94bn [£76bn), Boeing is nearly ten times the size of the Post Office. The Post Office has a turnover of less than £1bn. Boeing has vast facilities and specialist teams to investigate crashes full-time. Still, its judgments on the probable cause or causes of major incidents are not infallible.

A number of 737 incidents have shown that, even with relevant incident data available, it may take years of assiduous and expensive independent investigations to get to the likely truth.

In the case of the 737 incidents, the suspect component at the centre of investigations, a power control unit, was based on an old design (certified in the 1960s) – and straightforward in its operation relative to the Horizon system.

In comparison, the Horizon system has hundreds of thousands of lines of code and is complex, taking into account its many upgrades over more than a decade and its interactions with different hardware, networks, interfaces and a central data centre. Adding to this complexity are user uncertainties over procedures for dealing with problems.

But one of the most striking single aspects of any comparison between 737 crashes and the Horizon controversy is that it took professional full-time independent investigators in the US several years and thousands of tests on one suspect component only, before they were able to establish not that the component in question had been the cause of two fatal crashes and a succession of other incidents but that it had been the “probable” cause.

More than $1m was spent investigating the power control unit and still there was no firm evidence that the suspect component was the cause.

The Post Office has, arguably, required a higher standard of proof from local sub-postmasters.

By insisting that there was no evidence of a malfunction that resulted in losses, the Post Office put the onus on sub-postmasters to prove otherwise. Establishing that Horizon was the “probable” or “likely” cause – the standard of proof required in commercial aircraft accidents – was not good enough in cases of sub-postmaster complaints.

In response to the complaints of former sub-postmasters, the Post Office has made a number of similar statements:

“There is no evidence that faults with the computer system caused money to go missing at these Post Office branches. There is evidence that user actions, including dishonest conduct, were responsible for missing money.”

Another Post Office statement said,

“To date, and after two and half years of investigation and independent review, the facts are that Post Office has found no evidence, nor has any been advanced by either an Applicant [former sub-postmaster] or Second Sight [the independent investigators of sub-postmaster complaints], which suggests that Horizon does not accurately record and store branch transaction data or that it is not working as it should.”

Boeing made similar points in its submission to the National Transportation Safety Board on the crash of Flight 427. Boeing pointed to a lack of evidence of technical malfunction while pointing to evidence of the actions of human operators (pilots).

Boeing said,

“There is no evidence to support a conclusion that an uncommanded full rudder deflection occurred (the rudder moving in the opposite direction to that commanded by the pilots).

“While there is not conclusive evidence of a crew-commanded, sustained left-rudder input, such a possibility is plausible and must be seriously considered, especially given the lack of evidence of an airplane-induced rudder deflection.”

Indeed Boeing’s conclusion in its submission to investigators of 737 incidents was similar to the Post Office’s position that there was “no systemic problem” with Horizon.

Boeing said,

“There is no data to indicate that the Eastwind Flight 517 event, the United Flight 585 accident, and USAir Flight 427 accident were caused by a common airplane malfunction.” [Boeing had argued that each incident was different – a similar argument to the Post Office which said each complaint by sub-postmasters  about the Horizon system was “demonstrably different and influenced by its own particular facts”.]

In a separate submission to the National Transportation Safety Board, the manufacturer of the 737’s suspect power control unit, Parker Hannifin, made a point similar to Boeing’s.

“In sum, after years of one of the most critical examinations in aviation history, there is no evidence that the main rudder PCU [power control unit] from Flight 427 malfunctioned or was other than fully operational.”

Last word

But the National Transportation Safety Board, as a statutory authority, had the last word.

Its conclusion did not coincide with the view of Boeing or Parker Hannifin.

It said the most likely cause of the crash of Flight 427 was that the rudder moved in the opposite direction to that commanded by the flight crew. The final investigation report said,

“Probable Cause

“The National Transportation Safety Board determines that the probable cause of the USAir flight 427 accident was a loss of control of the airplane resulting from the movement of the rudder surface to its blowdown limit [full aerodynamic limit].

“The rudder surface most likely deflected in a direction opposite to that commanded by the pilots as a result of a jam of the main rudder power control unit servo valve secondary slide to the servo valve housing offset from its neutral position and over-travel of the primary slide.”

Could both sides be right?

On the face of it, the Post Office and former sub-postmasters have contradictory arguments, just as Boeing’s assertions and the investigators’ finding of likely technical malfunction may seem contradictory.

It’s possible, though, that these arguments are not as contradictory as they seem.

It is conceivable the Post Office was correct when it said there was no conclusive evidence of a technical malfunction; and it’s equally conceivable the former sub-postmasters were correct when they said a technical malfunction was partly or entirely to blame for the losses.

Possible similarities and differences

Campaign4Change has looked closely at some of the similarities and differences between 737 rudder incidents and the Post Office cases.

The Post Office and Boeing investigated each incident as a separate matter. Both organisations found no systemic problems. But, unlike Boeing, the Post Office always had the upper hand in its investigations: it was able to require that sub-postmasters pay, in many cases, tens of thousands of pounds that were shown as losses on Horizon.

There’s a risk of trivialising the consequences of 737 crashes when making comparisons with the Horizon controversy. It can be argued, though, that both involved major incidents that ruined lives; and both cases raise the question of whether any large corporation, once it has taken a position that its equipment was not to blame for a single major incident – let alone a number of incidents – will ever change its mind unless forced to.

One particular difference between the UK and US investigations into major incidents is that the US regulatory system allows Boeing to make a submission to the investigations board – which it did, contesting the board’s draft finding that blamed technical malfunction for 737 incidents and crashes – but Boeing had to abide by the independent board’s final decision.

The Post Office did not have to abide by the findings of its independent investigators Second Sight and was able to end Second Sight’s contract. The Post Office said it had given Second Sight “notice regarding its contract“.

Another difference: in the US, the regulatory system allowed the National Transportation Safety Board to require information from the various equipment manufacturers; and the Board’s investigators could obtain information independently of the manufacturers, usually with their cooperation but not necessarily.

In comparison, the Post Office determined what information it passed to Second Sight and the families. On this point Second Sight had its concerns.

In one of its reports for the Post Office, Second Sight said,

“We have experienced significant difficulty in obtaining access to a number of documents we believe are necessary for the purposes of our investigation, notwithstanding Post Office’s commitment to make requested documents available to us.”

The Post Office says it made available to Second Sight thousands of documents but not those that were the subject of legal privilege .

There’s a further difference between the US and UK investigations. In the US, the National Transportation Safety Board did its own investigations or supervised those carried out by equipment manufacturers. It even had the power to exclude equipment owners from participating in the inquiry.

In 2010 American Airlines was excluded from participating in an investigation into an incident involving one of its 757 aircraft because its technicians downloaded and accessed information from the plane’s black box [digital flight data recorder] before it was examined by independent investigators.

US regulations require that the National Transportation Safety Board is the first to see, download or access information from the black boxes.

A Board press release criticised American Airlines. It said,

“Although a thorough examination by our investigators determined that no information from the DFDR [digital flight data recorder] was missing or altered in any way, the breach of protocol by American Airlines personnel violates the Safety Board’s standards of conduct for any organization granted party status in an NTSB investigation.

“Because maintaining and enforcing strict investigative protocols and procedures is vital to the integrity of our investigative processes, we have revoked the party status of American Airlines and excused them from further participation in this incident investigation.”

When the Post Office investigated Horizon systems in the light of losses shown on the systems, it had the authority to retain full control of system information throughout.

As well as being the owner of the system, the Post Office was responsible for commissioning the investigations into the actions of the sub-postmasters. It was also the prosecuting authority and supplier of the material facts involved.

Other possible considerations

  1. In the US, there was no procedure for pilots to follow if they had a rudder hardover (where the rudder moves to its fullest extent and jams against a mechanical stop). The principle was that pilots were not trained to cope with problems that theoretically couldn’t occur. Were sub-postmasters faced with malfunctions that were considered impossible and so hadn’t been trained to cope with them?
  2. Human operators may make the ultimate mistake but they might have been reacting to malfunctions, problems with design, inaccurate information or confusing interfaces. [The Post Office had 1.5 million Horizon helpline calls in a three-year period which is a possible sign that many local post office staff did not fully understand the system or how it worked.]
  3. The US pilots’ trade union ALPA [Airline Pilots Association] was formed partly because of a perception that the government’s automatic response to major incidents was to blame pilots.
  4. After major incidents, the Post Office and Boeing have pointed to the extraordinary record of reliability of their equipment, the implication being that a systemic problem is highly unlikely. The 737 had (and still has) an extraordinary safety record: 264 million flight hours and an uncommonly low crash rate. Airlines have ordered at least 11,550 of them, more than any other commercial aircraft in history. It’s in use in 111 countries. Its reliability record is the best in the world. On average more than 2,000 737s are in the air at any one time. It has carried 17 billion passengers – about twice the world’s total population. It has flown about 120 billion miles, the equivalent of 640 round trips from the earth to the sun. The Post Office says of Horizon: “Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations.   There have been 500,000 users of the system since it was introduced.”
  5. The design of the 737 rudder system had been considered fail-safe. It was thought it would work properly even when problems occurred. The system had built-in “redundancy”. Every lever inside the lower power control unit had a second lever that moved in concert, in case one should break. There were two hydraulic systems in case one should fail. There was a standby actuator in case the main power control unit stopped working. Even so, after thousands of tests, investigators found it could fail in very rare circumstances.
  6. The Post Office has listed the many procedures and processes in place for subpostmasters to handle problems or technical failures. The Post Office said, “Horizon is capable of handling power and telecommunications problems. In Post Office branches, postmasters are responsible for power supplies and the cabled telecommunications lines. Interruptions in power supplies and telecommunication lines are a risk faced by all IT systems. There are, however, recovery systems built into Horizon to prevent losses occurring where there is a power or telecommunication failure. There is no evidence to suggest that either of these events would cause losses in branches where the recovery process has been correctly followed by branch staff. There is however evidence of branch staff failing to follow the recovery process properly. This would cause discrepancies in a branch accounts and could be a cause of losses. It is however the result of human error by Applicants [former sub-postmasters] or their staff, and not a failing of the Post Office or Horizon.”
  7. US air crash investigators were able to glean much from listening to voices in the cockpit shortly before incidents occurred. No such luxury existed in the investigation of Post Office Horizon losses. The Post Office cannot have known what was in the minds of the sub-postmasters at the time: whether they had criminal intent or were utterly baffled by what was appearing on their screens.
  8. The National Transportation Safety Board after its initial investigation into the fatal crash of United Airlines 585 at Colorado Springs in 1991, reached a conclusion that the probable cause was “undetermined reasons”. Would the Post Office consider such a possibility in the case of Horizon losses?
  9. After the unexplained crash of Flight 585, the National Transportation Safety Board kept tabs on 737 rudder problems even without evidence they were the likely cause of any serious incidents. Does this mark a different investigative approach to the Post Office which appears to have had a mindset that its equipment could not be to blame for losses?
  10. The fact that five leading members of the National Transportation Safety Board voted publicly on the probable cause, or causes, of a major incident limited the potential for an institutional mindset to develop. The Board often modified or rejected the findings of its investigators.
  11. Tests could not be carried out on 737 equipment until all parties agreed on how each piece would be tested. Agreement involved the Federal Aviation Authority as regulator, Boeing, the pilots’ union ALPA and the machinists’ union. In contrast the Post Office was in complete control of its investigations into Horizon losses.
  12. The existence of the National Transportation Safety Board is a check against parties protecting their own corporate interests, namely the reputation of their equipment, after a major incident. What similar check exists to prevent the Post Office from seeking to protect its corporate interests – namely the reputation of its equipment – after a number of major incidents?
  13. Would the conclusions of the investigations into the 737 incidents have been different if Boeing had been the authority in charge of the final report?

A useful book on the crash of Flight 427 is by Bill Adair, which is an inside account of the 737 rudder incidents. He had access to all the main parties involved.

Also useful is the final report of the National Transportation Safety Board into the crash of Flight 427. It contains Boeing’s submission.

In January 2017, the High Court granted Justice for Subpostmasters Alliance, which represents the accused former sub-postmasters, a Group Litigation Order against the Post Office.  There are 198 sub-postmasters on the High Court claim form and several hundred more are likely to join as claimants.

If the case goes to appeal, it could continue for years.

Or the Post Office could choose to settle rather than spend public money fighting a case which could be seen as a self-vindicating exercise – one that prolongs the misery for the subpostmasters and their families.

Campaign4change emailed the Post Office a list of detailed questions, based on this article. A Post Office spokeswoman replied that, “given that there is currently litigation it’s not appropriate for Post Office to comment”.

Last year, after a BBC Panorama documentary on the complaints of sub-postmasters and the Horizon system, the Post Office issued the following statement:

BBC Panorama – Our response

The Post Office wholly rejects extremely serious allegations repeated in BBC’s Panorama programme of 17 August 2015. The allegations are based on partial, selective and misleading information.

  • The Post Office does not prosecute people for making innocent mistakes and never has   
  • There is no evidence that faults with the computer system caused money to go missing at these Post Office branches 
  • There is evidence that user actions, including dishonest conduct, were responsible for missing money

We are sorry if a small number of people feel they have not been treated fairly in the past but we have gone to enormous lengths to re-investigate their cases, doing everything and more than we committed to do.

All of the allegations presented in the programme have been exhaustively investigated and tested by the Post Office and various specialists over the past three years or more.   The unsubstantiated claims and theories that continue to be levelled against the Post Office are at odds with the facts and are constructed from highly partial, selective and inaccurate information.

This is about individual cases and the Post Office will not discuss those in public for very good reason.  The Criminal Cases Review Commission (CCRC) is reviewing a small number of cases involving criminal convictions. It will be provided with all available information including confidential legal material not available to others and we believe the CCRC should be allowed to complete its reviews without external comment.  We also gave a commitment of confidentiality to people who put forward cases to us for re-investigation.

The Horizon computer system is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations.

Background facts

Prosecutions

The Post Office has always taken its duty to act fairly, proportionately and with the public interest in mind extremely seriously.  The Prosecutions it brings are scrutinised by defence lawyers before they advise their clients and are, ultimately, ruled upon by the courts.

If money is missing from a Post Office branch and the fact that cash is missing has been dishonestly disguised by falsifying figures in the branch accounts, the Post Office is entitled to take action and does so based on the facts and circumstances of that specific case. Though rare, where there is evidence of criminal conduct, a decision may be made to prosecute.

Prosecutions are brought to determine whether there was criminal conduct in a branch, not for the Post Office’s financial considerations.

Post Office prosecutors are all experienced criminal lawyers, many of whom have significant experience in prosecuting for both Post Office and the Crown Prosecution Service.   In the rare instances that prosecutions are undertaken, the Post Office follows the Code for Crown Prosecutors (the same code as the Crown Prosecution Service).  The Code requires a prosecution to have sufficient evidence and be in the public interest, both of which are kept under review right up to and including any trial.   It means there must be sufficient evidence for each charge – if a theft charge is brought, there must be sufficient evidence for a realistic prospect of a conviction for theft.

A charge upon which there is no evidence will inevitably fail. It is the duty of the defence lawyers to identify to the court where there is insufficient evidence to sustain a charge.  If the court agrees then the Judge must dismiss that charge.

The Post Office takes extremely seriously any allegation that there may have been a miscarriage of justice. We have seen no evidence to support this allegation.   The Post Office has a continuing duty after a prosecution has concluded to disclose any information that subsequently comes to light which might undermine its prosecution  or support the case of the defendant and continues to act in compliance with that duty.

The Horizon Computer System

Horizon is robust and effective in dealing with the six million transactions put through the system every day by our postmasters and employees at 11,500 Post Office branches. It is independently audited and meets or exceeds industry accreditations.   There have been 500,000 users of the system since it was introduced.

Nevertheless, rigorous re-investigations were undertaken into claims made by 136 mainly former postmasters that the system caused losses in their branches.

There is overwhelming evidence that the losses complained of were caused by user actions, including in some cases deliberate dishonest conduct. The investigations have not identified any transaction caused by a technical fault in Horizon which resulted in a postmaster wrongly being held responsible for a loss of money.

There is also no evidence of transactions recorded by branches being altered through ‘remote access’ to the system.  Transactions as they are recorded by branches cannot be edited and the Panorama programme did not show anything that contradicts this.

Resolution of cases

The Post Office was approached in 2012 by a small number of largely former Postmasters and MPs with the concern that faults in the Horizon computer system had caused losses at their Post Office branches.

In response the Post Office set up an independent inquiry and, when that found nothing wrong with the system, established a scheme to enable people to put forward individual complaints, providing financial support to those making claims so that they could obtain independent professional advice.

There were 150 cases put forward, 43 of which involved criminal convictions.

A number of the cases are now resolved, through mediation or otherwise, and the remainder of cases where the courts have not previously ruled have been put forward for mediation.

Mediation is overseen by the Centre for Effective Dispute Resolution (CEDR), an established leading and entirely independent organisation.   Those who have been offered mediation can still exercise their available rights if mediation is not successful – mediation itself doesn’t stop that.

Mediation cannot overturn a previous court ruling – only the courts can do so.

Campaign4Change’s questions to the Post Office

Based on this article, Campaign4Change put some questions to the Post Office:

  1. If an organisation the size of Boeing can be mistaken when it clears its own equipment and blames the human operators (pilots), it is possible that the Post Office was mistaken when it cleared its own equipment and blamed the sub-postmasters? [Boeing, which is much bigger than the Post Office, has vast test facilities and matching resources for investigations.]
  2. One outcome of the US investigations was that “no firm evidence of a technical malfunction” did not necessarily mean there was no technical malfunction. The 737 rudder system malfunction was found eventually to have been intermittent. It left no trace. [We know from the crash of a Chinook helicopter on the Mull of Kintyre in June 1994 that it’s possible for computer systems to fail to work properly – sometimes with an intermittent fault – and leave no trace.) Does the Post Office accept that mechanical or digital equipment can suffer from an intermittent fault that leaves no trace?
  3. Any comment please on the point that “no evidence of a technical malfunction” does not necessarily mean “no technical malfunction”?
  4. Any comment please on the point that large corporations, once they have cleared their equipment from blame after a single major incident – or further similar incidents – are unlikely ever to change their minds unless forced to?
  5. One of the most striking single aspects of any comparison between 737 crashes and the Horizon controversy is that it took professional full-time independent investigators in the US several years, millions of dollars and thousands of tests on one suspect component only, before they were able to establish not that the component in question had been the cause of two fatal crashes and a succession of other incidents but that it was the “probable” cause. There was no evidence that the suspect component was the cause. Has the Post Office required a higher standard of proof from sub-postmasters by requiring “evidence” to suggest that a Horizon malfunction or malfunctions caused the incidents in question?
  6. Boeing had to abide by the findings of the National Transportation Safety Board even though the Board did not agree with Boeing’s conclusions. The Post Office did not have to abide by the findings of its independent investigators Second Sight and was able to end Second Sight’s contract. Any comment please?
  7. In the US, the regulatory system allowed the National Transportation to require information from the various equipment manufacturers; and it could obtain information independently of the manufacturers, usually with their cooperation but not necessarily.   In comparison, the Post Office determined what information it passed to Second Sight and the families. On this point Second Sight had its concerns. In one of its reports for the Post Office, Second Sight said, “We have experienced significant difficulty in obtaining access to a number of documents we believe are necessary for the purposes of our investigation, notwithstanding Post Office’s commitment to make requested documents available to us.” Any comment please?
  8. The National Transportation Safety Board had the power (which it exercised) to exclude organisations that owned the equipment in question from participating in the inquiry. When the Post Office investigated Horizon systems in the light of losses shown on the systems, the Post Office, although owner and operator of the equipment in question, had the authority to retain full control of system information throughout.  Any comment please?
  9. The design of the 737 rudder system had been considered fail-safe and was certified on this basis. It had built-in “redundancy”. Even so, after thousands of tests, investigators found it could fail in very rare circumstances. The Post Office has explained at some length its Horizon failure back-up processes and procedures. Nevertheless could these prove fallible in very rare circumstances, in ways not yet fully understood?
  10. Boeing said it was open to any theory even if it meant Boeing was at fault. Is this the Post Office’s position?
  11. After the crash of United Airlines Flight 585 at Colorado Springs in 1991, the National Transportation Safety Board kept tabs on 737 rudder problems even without evidence they were the likely cause of any serious incidents.  Does this mark a different investigative approach to the Post Office which appears to have had a mindset that its equipment could not be to blame for losses?
  12. The NTSB after its initial investigation into the fatal crash of United Airlines 585 reached a conclusion that the probable cause was “undetermined reasons”. Would the Post Office consider such a possibility in the case of Horizon losses?
  13. Tests could not be carried out on 737 equipment until all parties agreed on how each piece would be tested. Agreement involved the Federal Aviation Authority as regulator, Boeing, the pilots’ union ALPA and the machinists’ union. In contrast the Post Office was in complete control of its investigations into Horizon losses.  Any comment please?
  14. The existence of the National Transportation Safety Board is a check against parties protecting their own corporate interests, namely the reputation of their equipment, after a major incident. What similar check exists to prevent the Post Office from seeking to protect its corporate interests – namely the reputation of its equipment – after a number of major incidents?

The Post Office’s reply (as mentioned earlier) was that “given that there is currently litigation it’s not appropriate for Post Office to comment”.

Postmasters tell their story – Computer Weekly investigation in 2009

Sub-postmasters and Horizon – timeline of events, 2009 to 2016 – Computer Weekly