We know how the Post Office IT scandal happened – but why?

The scandal’s fixable and unfixable underlying causes

By Tony Collins

Retired judge Sir Wyn Williams deserves much credit for the sensitive and thorough way he is chairing a public inquiry into the Post Office IT scandal. He said on You Tube last week that the evidence he has read and heard on the human impact has “made a deep impression upon me”. He added,

“On any view, the scale of the human suffering has been and in many cases continues to be very significant.”

He said he had an overriding duty to lay bare who knew what, when they knew it and what they did with the knowledge they acquired. “I am determined to expose the truth about these matters,” he said.

He may go much further. It’s conceivable he will urge Parliament to act on continuing delays in paying victims full and fair compensation.

But why the scandal happened and its underlying causes are questions not often asked. These articles look at possible answers to those questions. One conclusion is that the Post Office IT scandal was waiting to happen.

Part A is a summary of what could be said to be the scandal’s more obvious contributory causes, most of which are fixable. Part B is a summary of the underlying causes, few of which are fixable.

First, some of the scandal’s more obvious causes:

Part A: summary of the scandal’s more obvious contributory causes

– The Post Office is 100% owned by government and has many more branches than Tesco has stores. In 2000, the Post Office gave the go-ahead for a nationwide roll-out of a flawed outsourced accounts system, Horizon.

– The Post Office board knew that Horizon’s “accounting integrity” was uncertain at the time. But still Horizon was rolled out to 18,000 branches, according to Post Office board minutes in January 2000. Arguably, the single most important feature of any accounts system is its accounting integrity.

– Any large or small company that mistakenly buys an accounting system with uncertain accounting integrity is likely to reject it. The Post Office didn’t.

– Rejecting Horizon, even if the Post Office had wanted to, was fraught with difficulty. The contractor ICL (owned by Fujitsu) had strong political support.

– On the previous version of Horizon, a telegram from the British ambassador in Tokyo to No 10, the Cabinet Office and HM Treasury referred to a dispute over the project’s uncertain future. The telegram said,

“PROJECT HORIZON: ICL/FUJITSU VIEWS …FUJITSU COULD NOT UNDERSTAND HOW HMG [Her Majesty’s Government] COULD CONTEMPLATE QUOTE DESTROYING ICL UNQUOTE. IF THE PROJECT FAILED, THE FLOTATION OF ICL WOULD BE UNDERMINED AND FUJITSU WOULD RECONSIDER ALL ITS OPTIONS ON ICL. THIS MIGHT … INCLUDE SALE … A FAILURE OF PROJECT HORIZON WOULD UNDERMINE FLOTATION AND WOULD RESULT IN 700- 900 JOB LOSSES.”

The then prime minister Tony Blair preferred a reconciliation with Fujitsu. A settlement came, in part, in the form of the purchase of a truncated version of Horizon [called Horizon mark 2 for the purpose of this article], which was the system at the heart of the Post Office IT scandal.

Convention on not suing major IT suppliers to government

– The supply of big systems to central government has pros and cons. Among the cons are that the supplier may have to pay for large IT teams assigned to a particular project such as Horizon while government officials take weeks and sometimes months to take key decisions. One of the pros is that, once you are the chosen supplier, the chances of being de-selected are slim indeed, no matter what problems arise. The Post Office’s accepted a flawed Horizon mark 2 against this backdrop: there was little or no chance in practice of Fujitsu’s being asked to quit.

– Major international IT companies are willing to sue government if the circumstances warrant it but that willingness is not mutual. Senior politicians and civil service leaders do not want to confront one of their major technology suppliers in an open courtroom. One reason is that officials in the witness box may be challenged in public on their truthfulness and whether parts of Whitehall are dysfunctional. This reluctance of governments to sue their most familiar IT suppliers can put international technology companies in an extraordinarily powerful position in any serious dispute.

Lack of courage

– One of a series of Downing Street papers unearthed by researcher Eleanor Shaikh said that a lesson from the failure of Horizon [mark one] was that government lacked “courage” when it came to terminating a failing IT project. The paper said,

“Perhaps the most important lesson is a more general one: namely that when a project is clearly failing government needs to be bolder about cutting its losses, and tougher in its negotiating stance. There was a clear case for termination 12 months ago, although the Treasury and DTI [now the Department for Business, Energy and Industrial Strategy which funds the Post Office] favoured continuation. In effect, inertia led to continuation, since no-one at the centre had a sufficiently clear remit or reason to terminate. Throughout this period ICL assumed that the government would not have the courage to terminate. Only at the last minute, when they believed that cancellation was a real possibility, did they make real concessions. As a result the Post Office slipped a year in its automation strategy, and large sums of money were wasted both by government and by ICL.”

– Several decades ago, when government took the risk of putting officials in the witness box in a public context, two cases, Matrix Churchill and Peter Wright, proved particularly embarrassing for Whitehall and the government: Whitehall’s sometimes-distant relationship with the truth was exposed in both cases. A senior official in the Matrix Churchill case said, under questioning, that “truth is a difficult concept”. Journalist and author Richard Norton-Taylor wrote a book about the case. A different senior official in the Peter Wright case referred to a letter that had been “economical with the truth”. By not suing international IT suppliers in an open courtroom, government and Whitehall avoid the risk of this sort of embarrassing testimony.

Lack of open competition

– For the specific Horizon system the Post Office accepted for roll-out to branch post offices, there was no open competition. Competition was for the previous mark 1 version of Horizon. Fujitsu was then, as now, in an unassailably powerful position as a major IT supplier to the UK government.

– Big companies in the private sector, if contracting out core IT, usually maintain a level of control of the outsourced system and spend much time and care to ensure they fully understand the technology’s weaknesses throughout its life. Evidence has come to light that the Post Office’s understanding of Horizon’s faults was limited; it did not always want to know the truth and for years did not know of, or at least denied, the existence of Fujitsu’s Horizon Known Error Log which contained records of thousands of faults. A further complication in understanding Horizon was that, when unexplained shortfalls appeared on branch systems, the Post Office was, in part, beholden to Fujitsu to determine whether its IT was at fault and in what way.

Lack of competence?

– A paper by Geoff Mulgan, then head of No. 10’s Strategy Unit at No 10 summarised the lessons learnt from the Horizon mark 1 disaster. Mulgan addressed his paper to Tony Blair. It was copied to leading officials and ministers: Jeremy Heywood, David Miliband, Geoffrey Norris, Lord Falconer, Sir Richard Wilson and Jonathan Powell. This and other No. 10 documents are included in Eleanor Shaikh’s remarkable 579-page document on the origins of the Horizon scandal. Mulgan’s paper to Blair said, “Throughout this process, the relative lack of competence of the Post Office and their failure to develop a proper business strategy has been a key failing.”

– The lessons-learnt paper for Blair on Horizon mark 1 also said that “by most criteria Horizon has been a fairly disastrous project”. It was “misconceived from the start, has faced continual delays and problems, has over the last year taken up huge amounts of ministerial and official time and has delivered in the end a far from optimal solution”. It was an inauspicious start for Horizon mark 2.

Scandal could have been avoided

– Almost as soon as the truncated mark 2 version of Horizon began rolling out across the UK, sub-postmasters and sub-postmistresses began noticing unexplained shortfalls in their branch balances. Had the Post Office, knowing from the start that Horizon’s accounting integrity was uncertain, tenaciously pursued an explanation for every shortfall with Fujitsu and written off losses where necessary, as supermarkets write off shrinkages, a scandal could have been avoided.

– Corporate Post Office staff knew that Horizon had faults but it was difficult for them to establish the facts after sub-postmaster complaints, for at least two reasons. First the system was not owned or controlled by the Post Office and access to the main audit database could be costly, beyond a pre-defined “free” level of queries. Second, staff regularly raised concerns about Horizon but their queries, it seems, usually went unanswered or unresolved: there appeared to be little appetite among the higher-ups at the Post Office for thorough investigations of Horizon.

Us and them

– Among the Post Office’s higher-ups there appeared to be a belief that Horizon’s faults could not become public knowledge in case it undermined ongoing legal cases or the institution’s reputation. There was also a concern that any general knowledge of bugs could be exploited by unscrupulous postal workers for their own financial gain.

– Before, during and after the roll-out of Horizon there appeared to be an “us and them” corporate scepticism of the honesty of sub-postmasters and their staff. One of those who worked on the original Horizon project said he suspected that many sub-postmasters would exploit system weaknesses if given the chance. In nearly every dispute with sub-postmasters over Horizon shortfalls, the Post Office appears to have been pre-disposed to assume the sub-postmaster was dishonest or incompetent.

– Large sums of money were handled by branch postal workers. Horizon, with the help of manual “transaction corrections”, had to be trusted as an electronic police force watching out for theft. But the commercial imperative to trust Horizon ought to have been tempered by a full knowledge of its weaknesses. Without this knowledge, the commercial need to believe in Horizon turned quickly into a corporate ideology – the system was to be trusted whatever the circumstances. This was an irrational approach to take with any large-scale IT system but the buzz-phrase in Whitehall for the Post Office was “operational independence“. It’s still a Whitehall buzz-phrase today. It means that when things go wrong at the Post Office, its operations can be said to be independent of the government. Thus the Post Office was left unchallenged over Horizon matters. It had the operational freedom to believe in the rationality of the irrational.

– When shortfalls began to appear on Horizon, there was no need to investigate them thoroughly because the Post Office had an assured way of covering losses that appeared on the system: the institution had one-sided contracts between the Post Office and branch sub-postmasters and sub-postmistresses. The Post Office interpreted these contracts as requiring that sub-postmasters pay losses shown on the system, however big. The Post Office did not need to prove any actual money had gone missing. Sometimes it claimed phantom losses of more than £100,000.

– The Post Office went much further than claiming Horizon’s losses from sub-postmasters. It had its own 700-strong security community including a prosecuting authority that had no reliance on the Crown Prosecution Service or the police. Thus it began prosecuting postal workers for fraud, false accounting and theft on the basis of Horizon data. In the criminal and civil courts it achieved victory after victory. Each successful case added to the mythical status of Horizon as invincible. Successful cases also enabled the Post Office to confiscate homes, cars and other goods.

– Lee Castleton was a sub-postmaster who ended up having to pay the Post Office around £350,000. He was forced into bankruptcy. His case sent a clear message to other sub-postmasters who were experiencing unexplained shortfalls: if you try and challenge Horizon’s reliability you may go bankrupt too. [Today, 16 years later, Castleton still hasn’t recouped his losses despite various government and Post Office compensation schemes.] Castleton’s case was another symbol of the Post Office’s extraordinary operational freedom. If the Post Office today wanted to again spend £320,000 chasing a debt of £27,0000 in a different case it could. Barrister Paul Marshall has covered the Castleton case in his paper “The harm that judges do“.

– Judges were content to trust Horizon without proof of the system’s integrity or reliability. They sent sent dozens of people to prison on the basis of data from a faulty system. There was no requirement for juries to see facts such as Horizon’s error logs. In this respect, the UK joins authoritarian regimes across the world where people can be imprisoned without fair trials.

– No action is taken against prosecutors who don’t disclose relevant evidence. The UK courts, as currently set up, seem unwilling to punish the prosecution’s concealment of IT facts.

– When presented with Horizon cases, the courts had before them one poorly-funded man or woman in the dock. On the prosecution side were the statements of an international computer company and the Post Office, a well-funded public institution with a reputation for being a vital asset to local communities. For the person in the dock, establishing any credibility against such formidable legal opponents was difficult or impossible.

– Judges and juries, when presented with data from a system that was used successfully by thousands of postal workers every day, could not see how Horizon would be at fault in the one particular case before them.

– The accused stood little or no chance of proving their innocence by establishing that Horizon’s faults could show themselves in out-of-the-ordinary circumstances. The Post Office did not have always have unlimited “free” access to Horizon’s audit data and the courts did not require its disclosure.

– No serving judge has called for a reform of the criminal justice system in the light of wrongful convictions of more than 700 sub-postmasters and sub-postmistresses. This implies that judges and the courts would be content to convict another 700 innocent people if a different public institution were gradually, over 13 years, to prosecute hundreds of unfortunates by mistake on the basis of data from faulty computer system. In recent years, faulty data used by two other public institutions, the Ministry of Justice [“Common Platform”] and the Department of Work and Pensions [Universal Credit] have already had harmful consequences for individuals.

– The Post Office seems not to have noticed that a mass of prosecutions (more than 700) followed Horizon’s introduction. Before Horizon prosecutions were rare. Such was the Post Office lack of corporate awareness of its Horizon prosecutions that it kept assuring ministers and constituency MPs that the number of complaints about Horizon was “tiny”. In some letters to constituency MPs, the Post Office referred to Horizon complaints as numbering two, three or four – in total.

-The Post Office had no need to care whether the number of prosecutions was tiny, dozens or hundreds. As at the business department, an absence of the whole truth goes unpunished.

– The Post Office is without effective independent scrutiny. Had it not been for the public inquiry into the Horizon scandal, which was prompted by a High Court action against the Post Office led by former sub-postmaster Alan Bates, at a cost to his side of £46m, the Post Office would have remained protected from effective independent scrutiny. After the public inquiry, the Post Office is likely to revert to being a protected organisation.

– The business department BEIS may say that its oversight of the Post Office is subject to independent and effective scrutiny, at least though the Government Internal Audit Agency.

But the audit agency is, arguably, neither independent nor effective. On independence, it is one group of government officials marking the homework of another group of government officials. On effectiveness, its reporting is wordy, deferential in tone, generous in the use of business jargon and vague in a way that seems to say, “would you mind acting on our suggestions, perhaps?” Indeed, the agency suggested minor improvements to BEIS’s scrutiny of the Post Office in October 2019 which was three years after Alan Bates and his team had sued the Post Office. The agency’s audit reports on the Post Office are not published routinely. Nor do we know when or whether its audit suggestions are acted on. [What little we do know about the audit agency in relation to Horizon was unearthed by Eleanor Shaikh under the FOI Act.]

Shocked

– The business department BEIS could argue that the Post Office has independent scrutiny through Parliamentary committees and MPs’ questions. But Parliamentary committees have no power to enforce their recommendations. Nor can they force officials to be truthful. Former head of the Public Accounts Committee, MP Margaret Hodge, wrote a book “Called to Account” which suggested officials (out of loyalty to their department and ministers) rarely told the whole truth. At one hearing, Hodge shocked civil service leaders by calling for a bible and requiring a government lawyer to swear an oath to tell the truth. Later, the then head of the civil service made a formal complaint about Hodge.

– On whether MPs’ written and verbal Parliamentary questions provide effective and independent scrutiny, ministers, based on their civil service briefings, rarely give statements on IT projects that tell the whole truth. It’s not that ministers and or their officials are being mendacious: they are not telling the whole truth in the national interest, to insulate the government from possible embarrassment.

– The public spending watchdog the National Audit Office [NAO] is not permitted by law to audit the Post Office’s accounts, its computer systems or whether its decisions represent value for money for taxpayers. To a limited degree, the NAO was allowed to audit Horizon mark 1 but not subsequent versions. Mark 1 was eligible for investigation only because of the involvement in the scheme of a government department, now the Department for Work and Pensions. When the department ceased to be involved in Horizon, which was more than 20 years ago, the NAO ceased its investigations of the Post Office.

– The Post Office resisted independent scrutiny on the basis that it could restrict its commercial freedom. It opposed a political move to make the Post Office subject to oversight by the National Audit Office and, in the past, has challenged the role of the government’s one shareholder on the Post Office board.

A scandal waiting to happen

– The lack of effective independent scrutiny gives the Post Office an extraordinary operational autonomy. It has no shareholders except the government, it cannot go bust because it is publicly-owned, funding from taxpayers for costly mistakes is potentially unlimited and it has a remit to be commercially opportunistic but with no personal consequences for high-risk or for bad decisions. It could be argued that the Post Office IT scandal was waiting to happen.

– Government statements about the Post Office usually begin with words about the importance of branches to local communities and the country. It is therefore conceivable that Post Office’s senior managers and directors saw Horizon’s reliability as tied in with their organisation’s commercial interests and, therefore, the national interest, the two being the same. But almost anything can be justified in the name of the national interest, such as the shredding of important documents and denying that Fujitsu staff could change the balances of branch accounts without the knowledge of sub-postmasters.

– Some describe the Post Office as evil because it destroyed many lives. In some cases the Post Office’s pursuit of sub-postmasters and sub-postmistresses over supposed Horizon losses is said to have been a contributory factor in premature deaths and suicides. The events of the scandal indeed have no equal in scale and mendacity with any that have darkened the UK’s corporate record. But an allied tragedy may be that those at the Post Office, whose aggressive actions inflicted harm on thousands of independent business people who ran branches, might have been serving the Post Office as best they could. Doing the right thing didn’t come into it, except that right was whatever was judged to be in the Post Office’s best commercial interests. One illustration of this remarkable loyalty to the Post Office was a lack of truthfulness in some of the evidence given to the High Court in the Bates v Post Office civil litigation. The judge Mr Justice Fraser said several of the Post Office’s senior witnesses gave what he called a “party line” which was not always truthful. But how did a publicly-funded institution end up with the freedom to be untruthful and act as aggressively it did? One answer is that it had no effective independent scrutiny. Indeed, the organisation that supposed to be scrutinising the Post Office – the Department for Business, Energy and Industrial Strategy – was not itself subject to effective independent scrutiny (Part B).

– Mr Justice Fraser’s High Court judgments included a finding that the Post Office seemed to operate in a parallel world. He compared some of the Post Office’s logic to a Lewis Carroll nonsense rhyme. But these findings caused no alarm in government or the business department BEIS which funds the Post Office.

– By not imposing any effective independent scrutiny, successive governments seem to have left the Post Office with one foot in reality and the other in a parallel world where reality was sometimes interpreted in an abnormal way.

– Trigger events for going into a parallel world seemed to be the perception of external threats such as Mr Justice Fraser’s unwelcome findings, thorough analyses of Horizon-related problems by forensic accountants Second Sight and unflattering coverage of Horizon by media organisations such as BBC’s Panorama. In response to these perceived “threat” events, the Post Office sacked Second Sight, tried to remove the judge and made representations to the BBC and other media organisations about their actual or intended coverage of Horizon-related matters.

– But by far the biggest challenge to the Post Office’s Horizon ideology came from sub-postmasters and sub-postmistresses. We know how the Post Office reacted.

– One major contributory cause of the Post Office IT scandal, therefore, is that nobody in government or the business department BEIS had any particular wish to ensure the Post Office always had both feet in the real world.

– Another major contributory cause is that criminal court judges can also operate in a parallel world when it comes to complex IT systems. They will assume a massively-complex computer system works properly merely on the basis of the owners’ say-so. The owners need to produce no evidence of what they say and can usually hide the system’s faults with impunity.

Most of the above contributory causes of the scandal can be fixed. Most of the underlying causes listed in Part B (which is scheduled for tomorrow) cannot.

Origins of a disaster by researcher Eleanor Shaikh

The Great Post Office Scandal by journalist Nick Wallis

Post Office attacked sub-postmasters who questioned Horizon, say victims – Karl Flinders, Computer Weekly

Prof Richard Moorhead’s 10-part analysis of the “Swift” review of Horizon miscarriages of justice

Investigating the Post Office scandal – podcast by journalist Nick Wallis and Rebecca Thomson who wrote the first articles on the scandal for Computer Weekly in 2009

The harm judges do – Lee Castleton’s story by barrister Paul Marshall

Scandal at the Post Office – The intersection of law, ethics and politics – Paul Marshall

Horizon issues log – campaigner Tim McCormack’s blog