Most of the underlying causes of the Post Office IT scandal are unfixable

“Success is walking from failure to failure with no loss of enthusiasm” – Winston Churchill

By Tony Collins

Why the Post Office IT scandal happened and some of its largely-fixable contributory causes were covered in a Campaign4Change post yesterday. One conclusion was that the Post Office IT scandal was waiting to happen.

Today’s piece looks at some of the underlying causes of the scandal that are mostly unfixable. One problem is that parts of the government and Whitehall machine are deeply dysfunctional. It’s not the fault of any individuals. Author Diane Vaughan describes banal corporate cultures that lead to disaster as the “normalisation of deviance”.

Summary of mostly-unfixable underlying causes of the Post Office IT scandal

– It is perfectly normal in Whitehall to buy, accept and roll out defective systems. It has happened routinely since early government computerisations in the 50s, 60s, and 70s. This blog post lists some of relevant projects. In the context of government’s poor record on IT, Horizon was simply one of many flawed computer systems that were bought with public money, formally accepted by officials who would not expect to be held responsible for their decisions and rolled out to thousands of unsuspecting end-users.

– It would be a major break with convention for any publicly-funded institution, including the Post Office, to reveal publicly the faults in its core IT systems unless the faults were already public knowledge. Weaknesses in Horizon were known at the time of the system’s national roll-out. Had software specialists been free to call out those weaknesses publicly at the roll-out stage, it’s unlikely the Post Office would have been able to confiscate businesses and homes, remove livelihoods and secure hundreds of convictions based on Horizon’s robustness.

Cultish ideology

– In the absence of effective independent scrutiny or challenge, and within a Whitehall bubble that is itself not subject to effective scrutiny, a mini-authoritarian ideology took hold at the Post Office over Horizon. Little exists in Whitehall’s armoury to stop a cultish ideology over an IT system again taking hold, unacknowledged and under cover of confidentiality, almost anywhere in central government or the wider public sector.

Faulty presumption

– If other public institutions take innocent people to court based on data from a flawed IT system, judges are likely to take the prosecution’s computer evidence at face value, as they did with many victims of the Post Office IT scandal. An obsolescent Law Commission “Presumption” means that judges can regard complex IT systems much as a mechanical clock that is assumed to be working if the hands can be seen moving. But Horizon had thousands of faults, some of which were blame for intermittent balance discrepancies in branches. For the Law Commission to normalise treating such a complex system as a mechanical clock underlines how deviant and deeply dysfunctional the Whitehall machine can be. The backward-facing Ministry of Justice has refused to modernise the “presumption”. Barrister Paul Marshall has written an article headlined, “Government failings on computer evidence risk more miscarriages of justice.”

No consequences

– Whitehall has a convention of no individual consequences for failure. Sometimes there is accountability but “responsibility” is normally collective not individual. In more than 40 years of government-funded IT-based project failures, nobody was held responsible except at the BBC where the chief technology lost his job in 2013 over the failed £100m Digital Media Initiative. Post Office higher-ups had no fear of action being taken against them for collectively bad decisions. It appears that a demonstrable belief in Horizon was seen as an act of loyalty.

– Every major government IT project has a “senior responsible owner” who is ultimately accountable for a programme or project meeting its objectives, delivering the projected outcomes and realising the required benefits. In practice the SROs, as they are called, are not always told the whole truth about the projects for which they are responsible. Sometimes a project or programme has several SROs.

– The lack of effective checks and balances across central government is currently illustrated by the national roll-out today of defective IT to the courts. The “Common Platform” is a £250m IT project run by HM Courts and Tribunals Service which is part of the Ministry of Justice. It has taken a BBC “File on 4” documentary, which was broadcast last month, to show that the system has led to wrongful arrests. Indeed, File on 4 compares Common Platform to the Post Office IT scandal. The Common Platform is following the usual lifecycle of a failing major IT project.

The whole truth?

– A stage in this lifecycle is that senior officials make public statements in direct opposition to those made by the system’s end-users who, in the case of Common Platform, are judges, barristers and solicitors. It is commonplace in failing IT projects for protected interests in government to make public statements that contradict the criticisms of those who are simply using the system to do their work. This happened at the Post Office for 19 years. Sir Wyn Williams, the much-respected chairman of the Horizon IT inquiry, will have no power to change the Whitehall culture that encourages protected interests to present IT project failure as success.

– Senior officials have a difficult job to do when it comes to public communications on big projects that go wrong. One of their top priorities is to protect ministers and the government from embarrassment. Thus they will not want to say anything that could make the civil service look dysfunctional. To this end, they will keep quiet about facts that could be used by opposition MPs to make mischief. This helps to explain why successive ministers have, based on drafts written by senior officials, routinely given misleadingly positive statements on the status of IT systems such as Horizon.

– Such unjustified official optimism has dogged nearly every government-funded IT project disaster listed here.

– On the Common Platform project a government spokesperson told Mail Online that the system has “already successfully managed over 158,000 criminal cases and there is no evidence that Common Platform is compromising justice or putting parties at risk”. But the BBC said, “File on 4 has spoken with whistle-blowers from within the court service who say the system is unsafe, unfinished and beset with bugs, errors and glitches”.

– Similarly, the GOV.UK website has an unremittingly positive “overview” of the UK’s most costly non-military IT project, the Home Office’s Emergency Services Network. The overview gives no hint that the project is billions of pounds over budget, more than five years behind schedule and may never work. The £11.4bn project seems to be following the path of the £10bn National Programme for IT in the NHS, the government’s most costly IT project failure to date. The NPfIT, as it was known, was “dismantled” in 2011 after eight years of ministerial assurances about its success. Official statements on Horizon’s robustness followed a familiar general pattern of untruthfulness that concealed IT faults. This pattern is probably too ingrained within central government to be fixable.

Honesty and integrity unscrutinised

– It is enshrined into law that the civil servants who brief ministers must be honest, act with integrity, be neutral and tell the truth. But there are no punishments for breaches. Governments indeed expect that ministerial statements, at least on IT projects, will not cause any political embarrassment. Thus official briefings and draft ministerial statements are tailored accordingly: a selected set of facts usually gives the impression that all is well.

Lessons must be learnt – but how?

– “Lessons must be learned” is almost as familiar as the phrase “Post Office IT Scandal”. But the Whitehall machine is not set up to learn lessons from big IT-based project calamities. There is no institutional memory, civil servants come and go and each big IT project is regarded internally as unique.

– The private sector also has notorious IT disasters but, because the job of one or more senior executives may be at risk and the bottom line is of deep concern, the same mistakes tend not to be repeated.

– The public sector has its major and less visible IT successes. Perhaps the most unexpected success was the performance of Universal Credit during the pandemic.

Arguably the success of Universal Credit in recent years is because the systems had such a deserved bad press in their early years that it smoothed the path, politically, for serious faults to be admitted and therefore addressed.

Normalising deviance

– The deviant Whitehall conventions that are among the contributory underlying causes of the Post Office IT scandal don’t need to be written down or spoken about because they are an accepted part of Whitehall’s culture. Senior officials know how to act on the conventions without being told. But that self-protective, secretive, fault-denying culture is one reason few major figures from the world of private sector IT have lasted long in the Whitehall system.

– No incoming government has a vested interest in correcting deviant conventions. Prime ministers and their senior officials benefit from an absence of effective, independent and contemporaneous checks and balances and the non-reporting of IT project weaknesses. The non-reporting especially. After decades of campaigns for open government on IT projects still today there is no public reporting of faults in government systems other than in spasmodic, though usually thorough, reports of the National Audit Office. The Post Office is legally exempt from the National Audit Office’s scrutiny as are many public bodies.

Whitehall’s versions of the truth

– Ministers and officials have responded to criticisms about the non-reporting of publicly-funded failing IT-based projects by publishing a yearly (sanitised) report under the banner “Transparency Data” in which departments give their own summary of the status of their major programmes and projects. The government’s Infrastructure and Projects Authority publishes the report. But officials can in practice say what they want in the summaries. A project that is many years late and over budget can be recorded as on time and to budget if the project was secretly “reset”. The summaries give a red/amber/green status on major projects but the colour-coding is a subjective assessment that is not open to scrutiny. Convention dictates that departments mustn’t publish the internal reports on which their annual summaries are based. Thus nobody knows whether the annual summaries of projects tell the whole truth or not.

– The often-excellent recommendations in reports of the National Audit Office and Parliamentary committees that look at major IT and other projects are unenforceable.

– Given that public institutions operate in a world where they cannot go bust and there is no fear of individual consequences when things go wrong, the checks and balances need to be demonstrably rigorous and independent rather than deferential, in-house and largely confidential as now.

As the main fear of senior officials and political leaders is embarrassment, contemporaneous publication of internal reports on IT project faults could make the difference between success and failure on major projects such as Horizon. But some mandarins may choke on the words “full disclosure”.

– In the Post Office IT scandal, the main defence of those who allowed it happen is expected to be that they didn’t know the truth. Nobody seems to have known the whole truth. The Post Office didn’t even tell the whole truth to a High Court judge in the Bates v Post Office litigation. The systemic problem is that the whole truth doesn’t seem to have mattered to anyone in authority. It didn’t matter to governments if business officials were misled by the Post Office over Horizon’s robustness. No business department officials were going to jeopardise their job by accepting the Post Office’s version of events rather than sub-postmasters’. Business department officials would occasionally question the Post Office on its finances but rarely on its IT systems. On IT, the Post Office had operational autonomy. It was entirely within Whitehall conventions that officials usually took the Post Office’s assurances about Horizon at face value.

Remarkable lack of suspicion

– Some former postal ministers have suggested they were given misleading briefings about Horizon which is likely to be true. Officials had themselves been misled by the Post Office. It’s also possible that the whole truth became lost in the Post Office’s secretive silo culture. But the general lack of curiosity among those in authority seems remarkable. Early in the new millennium, Alan Bates, later joined by other former sub-postmasters and sub-postmistresses, wrote to constituency MPs about losing their livelihoods after unexplained shortfalls shown on Horizon. Those MPs then wrote to ministers. James (now Lord) Arbuthnot was a particularly active and campaigning MP as was Kevan Jones. Computer Weekly had published its first in-depth piece on the scandal, written by Rebecca Thomson, in 2009. It was also generally known that no IT system is infallible especially a complex one – and Horizon was said to have been the biggest system of its kind in Europe. An obvious question for incoming postal ministers or business officials to keep asking was: “Could the sub-postmasters be right about Horizon causing shortfalls given that no system is perfect?” Suspicion ought to have been the reaction to assurances that the system could not cause unexplained shortfalls. But neither officials nor ministers had any need for suspicion. A lack of suspicion was entirely within Whitehall’s conventions. Nobody was going to be fired for a lack of suspicion – or complacency. Indeed postal ministers have been briefed in recent years that if there were any genuine miscarriages of justice to do with Horizon there were procedures within the criminal justice system to correct them. Whitehall conventions almost guarantee official complacency when things go wrong.

– Nothing can be done to change the Whitehall convention in which senior officials will tell their ministers only what the officials want the minister to know.

– A technology journalist was some years ago asked to have an unofficial meeting in the House of Commons with a health minister who was sceptical about a briefing she’d received from her officials. It transpired that the minister was right to be concerned: the briefing paper she’d been given was one-sided and reflected the department’s self-interests. But it’s conceivable that officials who wrote the paper were acting in what they considered the national interest.

Jonathan Swift

– One business minister, exceptionally, seems to have challenged the Post Office’s complacency over Horizon. In 2015 Baroness Neville-Rolfe asked the Post Office’s incoming chairman to look at possible miscarriages of justice and whether the Post Office had behaved fairly and responsibly. The Post Office agreed but ordered an investigation from a safe pair of hands, a former top government lawyer memorably named Jonathan Swift. An important ten-part analysis of Swift’s final report has been written by Richard Moorhead, a professor of law and professional ethics at Exeter Law School. One of his conclusions is that the Swift review was a lost opportunity to “surface and deal with life-shattering miscarriages of justice years sooner rather than later”.

– Strangely the Swift report was not shown to the Post Office board. Indeed, it remained undisclosed in the Bates v Post Office litigation. Its public disclosure could have prevented the litigation which cost a total of nearly £100m.

Secrecy – a British disease?

– Nobody can do anything about Whitehall’s preoccupation with secrecy. It is said to the British disease. Francis (now Lord) Maude, Cabinet minister in the Coalition government, tried to bring into effect what he called the “most transparent government ever” but his reforms were widely derided among the higher-ups in the civil service and most of the changes he initiated have since been reversed.

– It’s difficult to see a time when any part of central government will embrace for its own good the idea of genuine openness and transparency. Indeed the concealing of faults on a major publicly-funded IT system such as Horizon is as easy today as it was between 2000 and 2019.

– The Freedom of Information Act has made government a little more open than it used to be. But most of what is released under the Act by the Post Office and the Department for BEIS has entire paragraphs and pages redacted (blacked out). The Act was supposed to give the public a right to know. Rather it seems to be interpreted by many public institutions as giving officials the right not to disclose the whole truth.

Government and big, complex technology-based schemes don’t always mix. Genuine successes seem to defy the natural order. The National Audit Office was right to point out last year that,

“Despite 25 years of government strategies and countless attempts to deliver digital business change successfully, the findings of this report show a consistent pattern of underperformance.

“This underperformance can often be the result of programmes not being sufficiently thought through before key decisions on technology solutions are made. This means that there is a gap between what government intends to achieve and what it delivers to citizens and service users, which wastes taxpayers’ money and delays improvements in public services.”

Comment:

Much credit is due Sir Wyn Williams, chairman of the Horizon inquiry, who has understood completely the frustrations and the harm to decent honest people who had the misfortune to be using an accounts system when it was not working as expected.

There is no doubt Sir Wyn will do what he can to mitigate the wrongs within the inquiry’s terms of reference. But an inquiry into the Post Office and Fujitsu’s Horizon system cannot fix the underlying wrongs that cost hundreds of self-made people their livelihoods, their savings, their homes, their savings and sometimes their lives.

Indeed, looking back on the scandal from its origins, it is remarkable how well the government machine has managed to brush off 19 years of official untruths and the biggest miscarriage of justice in British legal history.

It is as if a Whitehall conveyor belt is feeding all the untruths and injustices into a large box marked “Post Office Horizon system – untruths and injustices for archiving”.

Whitehall officials and successive government ministers have escaped scrutiny or criticism. The only people who have suffered are the hundreds who were punished by a system that will continue as it is. The “us and them” system will continue to punish the unprotected, which is why sub-postmasters and sub-postmistresses are still fighting for full and fair compensation.

The current government machine is ripe for misuse, witting or unwitting, by any number of the hundreds of other publicly-funded institutions that exist in the UK. Objective truth will continue to be nowhere in sight when big government IT-based projects go wrong. This is currently the case on the Ministry of Justice’s £250m Common Platform and the Home Office’s £11.4bn Emergency Services Network.

It is a deeply dysfunctional Whitehall system that political and civil service leaders will continue to describe as “world-beating” or the Rolls-Royce of government administrations, the envy of democratic countries across the world. All these claims are no more credible than the official claims – which lasted 19 years – that Horizon was robust.

Failures in the government administrative machine ought in no way to reflect badly on the many excellent and fully committed civil servants who often have to work against the system to succeed in what they do.

Full and fair compensation

How, in the Post Office scandal, will it be possible to single out one or two people as culpable?

The Horizon ideology gripped generations of Post Office boards, officials and prosecutors. State-supported aggression and mendacity in the name of the national interest was widespread within a formerly much-respected public institution. Within government, nobody persisted with asking the right questions about Horizon or the persecution of sub-postmasters because such challenges to the system were unwelcome. The main perpetrator of the Post Office IT scandal, the Mr Big, is not any one individual because the miscarriages of justice have continued so far for 22 years. About 80 convictions have been overturned but most victims are still waiting for full and fair compensation.

Journalist Nick Wallis who wrote the book The Great Post Office Scandal is unaware of anyone who has received full and fair compensation. It is remarkable that still after 22 years nobody in government is demanding that, finally, right is done before any more victims die.

The Mr Big in the Post Office IT scandal is the unreformable Whitehall system that not only let the large-scale misery happen but remained the Post Office’s silent investor in the extraordinarily unequal fight against sub-postmasters in the High Court. It wasn’t just unequal in money terms. One one side were sub-postmasters who’d had their livelihoods, homes and money taken from them by an aggressive state institution where responsibility was collective. The sub-postmasters were fighting for a return of their money and a semblance of their former lives. On the other side was the Post Office whose officials had lost nothing. The Whitehall system of operational autonomy allowed the Post Office to pour tens of millions of pounds into a legal fight to prove a point: that they were right about Horizon all along.

Justice in the case came only because of the single-minded tenacity of Alan Bates, his good fortune in having a gifted legal team and the final judgments of a genuinely independent and proficient High Court judge, Mr Justice Fraser, whom the Post Office tried to remove.

The government machine has reacted to the litigation it lost by managing to escape paying full and fair compensation in most cases although it is two years since the litigation ended. The Horizon inquiry may name and shame which is an important part of its task. But apart from tweaks the government machine will go on much as before.

Nobody doubts that Sir Wyn cares – really cares – but he has no power to change the Whitehall conventions that let the Post Office scandal happen. Untruths will continue, unsuccessful IT-based projects will continue to be passed off as successes, a pre-occupation with positive statements that conceal the truth will continue, individual consequences for bad decisions will continue to be non-existent, and the system as a whole will continue to escape scrutiny as the focus, whenever anything goes seriously wrong, will be on the specific institution(s) involved.

Given that most of the underlying causes of the Post Office IT scandal cannot be tackled, it is to Sir Wyn’s credit that he is concentrating the inquiry’s attention on full and fair compensation and on naming and shaming. These are challenges enough.

Perhaps nothing less than a Royal Commission into the Civil Service could bring any hope of a wider change in Whitehall’s deviant, reform-resisting conventions that let the Post Office IT scandal happen – and are now helping it to ulcerate.

Origins of a disaster by researcher Eleanor Shaikh

The Great Post Office Scandal by journalist Nick Wallis

Post Office attacked sub-postmasters who questioned Horizon, say victims – Karl Flinders, Computer Weekly

Prof Richard Moorhead’s 10-part analysis of the “Swift” review of Horizon miscarriages of justice

Investigating the Post Office scandal – podcast by journalist Nick Wallis and Rebecca Thomson who wrote the first articles on the scandal for Computer Weekly in 2009

The harm judges do – Lee Castleton’s story by barrister Paul Marshall

Scandal at the Post Office – The intersection of law, ethics and politics – Paul Marshall

Horizon issues log – campaigner Tim McCormack’s blog