Category Archives: Police IT

A classic “waterfall” IT project disaster – yet officials went by the book

By Tony Collins

Some of those who read “Crash – 10 easy ways to avoid a computer disaster” may remember a warning that buying an IT system on the basis that it works well in another country and can therefore be adapted to the UK’s needs, is flirting with disaster.

First published in 1999, Crash said,

“There are graveyards of computer projects that began life as a simple adaptation of a package used elsewhere in the world.”

One example at that time was the failure of the London Stock Exchange’s Taurus project.

Now a report published today by Audit Scotland on the “i6” project goes into forensic – but lucid – detail on what went wrong and the conflicting views of police and the supplier Accenture.

Says the report,

“The belief that most of the i6 system could be based on an existing IT system proved incorrect.”

It became clear well into project that

“a virtually fully bespoke system was required”.

The plan was for i6 to replace 130 paper-based processes and IT systems but on 1 July 2016, after many well-publicised difficulties and delays, the Scottish Police Authority and Accenture agreed to terminate the i6 contract.

Police in Scotland had chosen Accenture’s bid in 2013 largely because it had successfully implemented a system for Spain’s Guardia Civil police service.

To its credit Accenture refunded all the money the police in Scotland had paid for the i6 system, £11.06m, plus a further £13.56m – but Audit Scotland says the failure of the project …

“means that some of the benefits that should have arisen from implementing it, have been, at best, delayed. There was a need to modernise police ICT systems six years ago when the procurement of i6 began. That need has not been met. Police officers and staff continue to struggle with out-of-date, inefficient and poorly integrated systems.

“This also hinders how Police Scotland interacts and shares information and intelligence with the other parts of the justice system. There is an urgent need to determine what the next steps should be…”

The lessons are clear from the report:

  • Don’t buy an overseas system without realising that it’ll need to be built almost from scratch for the UK. The ideal is for the business processes to be greatly simplified and adapted to fit a tried and tested system, not the other way around. Audit Scotland says the police programme team and Accenture believed that the majority of the i6 system could be based on an existing IT system that Accenture had developed for Spain,  with the remainder being bespoke development work.  But there was an “over-reliance” on Accenture’s work for Guardia Civil”.
  • The “waterfall” systems development contributed to the fact that Police Scotland “only discovered the true extent of problems with the system when it was delivered for testing”.  Waterfall meant that Accenture produced the software in distinct phases, in a sequence resembling a waterfall. Once a phase was complete, the process moved to the next phase – and no turning back. “It meant that all of the design, coding and construction of i6 would be completed before Accenture released it to Police Scotland for testing. Police Scotland would pay for each phase when it was completed.” [Agile, on the other hand, is a “test and see” approach and is far more flexible. It can adapted according to what the end-user needs and wants, and changes in those needs and wants.]
  • Don’t trust the demonstration of a waterfall system. The demo may look great but rolling it out successfully across various regions may be a different story. Accenture had demonstrated i6 but much later, after a period of testing, the i6 programme team reported to the programme board in August 2015 that there were: critical errors in the technical coding, flaws that Accenture was unable to resolve as quickly as expected, serious concerns about the criminal justice module, which did not comply with the Integrated Scottish Criminal Justice Information System data standards, errors in the search and audit modules and “problems around the limited functionality in the administration module”.
  • External assurance reports may tell you that you have complied with good practice and they may give you detailed praise for your attention to detail but they probably haven’t looked at the big question: will the systems ever work? Audit Scotland said external assurance reports such as the Scottish Government’s “Gateway reviews” suggested improvements but “raised no major concerns”.  Throughout the course of the i6 programme, most of the external reviews suggested that delivery confidence was either amber or green.
  • If the plan is for a waterfall development, doing everything by the book before a contract is awarded will not guarantee success, or even make it more likely, if you haven’t asked the big question: Is this ever likely to work given the complexities we don’t yet understand? For officials in Scotland, everything went smoothly before the award of contract: there were even 18 months of pre-contract discussions. But within weeks of the contract’s start, Police Scotland and Accenture disagreed about whether the proposed system would deliver the requirements set out in the contract. Soon there was a “breakdown in relationships and a loss of trust between Police Scotland and Accenture that never fully recovered,” said Audit Scotland.
  • The supplier may be just as optimistic as you. “As the design and development of i6 progressed, it became apparent that Accenture would need to develop significantly more than had been originally anticipated. Despite delays and serious problems throughout the lifetime of the programme, Accenture provided regular assurance, in the face of strong challenge, about their confidence in delivering the i6 system. This assurance proved misplaced.”
  • When planning a waterfall system that has complexities and inter-dependencies that are not fully understood at the outset, expect ever-lengthening delays and projected costs to soar. At one point Police Scotland estimated that the level of effort Accenture would require to complete i6 was around eight times greater than the resources Accenture had estimated when signing the original contract. “The i6 programme team believed that the functionality of Accenture’s solution did not meet the requirements it had agreed in the contract. Accenture maintained that Police Scotland had not specified a detailed description of business requirements. This issue had not emerged during months of pre-award dialogue. Accenture also believed that it had set out clearly what its solution would do and maintained that Police Scotland, as part of procurement process, had accepted its qualified solution. A dispute followed about the interpretation of the contract requirements. Police Scotland argued that, after months of competitive dialogue, the requirements of the i6 system were well-defined, and that in line with the contract, these took precedence. Accenture argued its solution had precedence and that Police Scotland was trying to extend the scope of the programme. Accenture stated that, to meet Police Scotland’s interpretation of requirements, it would require more time and money.”
  • As soon as things start going badly awry, stop and have a re-think. Cancel all existing work if necessary rather than plough on simply because failure isn’t an option. Above all, take politics out of the equation. The Scottish Police Authority was anxious about i6 being seen to be a success after the failure of a previous police ICT project in 2012 – the Common Performance Management Platform. At the same time the i6 programme was “extremely important to Accenture at a global level. “This may have led to misplaced optimism about the prospects of success and unwillingness to consider terminating the programme,” says Audit Scotland.
  • When things start to go wrong, the truth is unlikely to emerge publicly. Even those accountable for the project may be kept in the dark. “Police Scotland were cautious of commercial sensitivities when providing assurances on i6 publicly. The Scottish Parliament’s Justice Sub-Committee on Policing held a number of evidence sessions with the Scottish Police Authority and Police Scotland to explore progress with the i6 programme. In March 2014, the Sub-Committee expressed frustration at the lack of information about the problems with the i6 programme that had been ongoing since August 2013. Police Scotland did not disclose the severity of the issues facing the programme, nor was it overly critical of Accenture. This may have reflected a desire to maintain relationships with Accenture to keep the programme on track or to maintain the commercial confidentiality of the contract.”

Accenture’s response

Accenture said,

“As the report acknowledges, the scope and the complexity of the solution for i6 increased significantly during the project.  This was driven by the client.  There were challenges and issues on both sides, but we worked closely with Police Scotland to review the programme and recommend revised plans to successfully deliver i6.

Despite our best efforts, it was not possible to agree the necessary changes and we mutually agreed to end the project.”

In May 2017 Audit Scotland is due to publish a report that summarises the lessons from a number of public sector ICT projects it has investigated.

Some of what i6 was intended to cover …

Comment

Tis a pity officials in Scotland hadn’t read Crash before they embarked on the i6 project – or if they had, taken more notice of the dangers of assuming a system that works overseas can be tweaked to work in the UK.

We commend Audit Scotland for its expert investigation and a fine report.

Clearly the failure of i6 is not entirely Accenture’s fault.  The project was commissioned on the basis of assumption and when things went wrong politics intervened to prevent a complete stop and a fundamental re-think.

Fatally, perhaps, there appears to have been no discussion about simplifying police administration to make the IT more straightforward. If police administration is so enshrined in law that it cannot be simplified, officials would have to accept before awarding the contract that they were buying an entirely new system.

The UK armed services simplified volumes of rules and practices before it introduced pay and personnel administration systems. It was hard, inglorious work. But simplifying ways of working first can make the difference between IT success and failure.

i6 – a review. Audit Scotland’s report. 

Waterfall approach damns £46m Scottish police system – Government Computing

Another public sector IT disaster – but useful if the lessons are learned.

Advertisements

Another public sector IT project disaster – but a useful failure if lessons are disseminated

By Tony Collins

Comment and analysis

Government Computing reported on 1 July 2016 that the Scottish Police Authority has agreed with Accenture to end their “i6” programme.

It’s a classic public sector IT project disaster. It failed for the usual reasons (see below). What marks it out is the unusual post-failure approach: a limited openness.

Police in Scotland and the Scottish Government plan a review of what went wrong, which is likely to be published.

Usually senior civil and public servants in Whitehall, local government in England and Wales and the NHS rush to shut the blinds when an IT-enabled change project goes awry, which is what has happened recently after failures of the GP Support Services contract with Capita.  [GP magazine Pulse reports that NHS England is to withhold report on primary care support problems until 2017.]

The police in Scotland and the Scottish Parliament are being open but not completely. Their settlement with Accenture remains confidential, but the Scottish Police Authority has published the full business case for i6 and – under FOI – early “Gateway” reviews and “Healthcheck” reports, though with quite a few redactions.

Despite FOI, it’s almost unknown for Whitehall, the NHS or local government in England and Wales, to publish Gateway reviews of big IT projects.

All this means there may be a genuine attempt in Scotland to learn lessons from the failure of the i6 project, and perhaps even let the public sector as a whole benefit from them (if it’s interested),

Due originally to go live last December, and then in the autumn this year, i6 hit problems within months of the start of the contract with Accenture. The contract was signed in June 2013, work started in July and the two sides were reported as being in mediation by August 2013.

Exemplar?

But the programme had followed well-established preparatory routines. One internal report described the procurement approach as an exemplar for the rest of the public sector. Yet it still ended in failure.

In fact i6 followed the classic script of a traditional public sector IT-based project disaster:

  1. An over-ambitious plan for widespread “integration” – which is one of the most dangerous words in the history of public sector IT-enabled change projects. It seemed a great idea at the time: to save vast sums by bringing together in a single system similar things done in different ways by formerly separate organisations.
  2. A variety of early independent reports that highlighted risks and strengths of the programme but didn’t ask the biggest question of all: could a single national system ever work satisfactorily given the amount of organisational change required – changes that would impose on the system design constant modification as end-users discovered new things they wanted and didn’t want that were in the original design – and changes that would require a large team on the police side to have the time to understand the detail and convey it accurately to Accenture.
  3. An assumption that the supplier would be able to deliver an acceptable system within tight deadlines in a fast-changing environment.
  4. Milestones that were missed amid official denials that the project was in disarray.
  5. An agreement to end the contract that was on the basis of a secret settlement, which brought little or no accountability for the failure. Nobody knows how much has been spent on the project in staff and managerial time, hiring of various consultancies, the commissioning of various reports, and money paid over to the supplier.

What are the lessons?

 

The 10-year programme, which was said to cost between £40m and £60m, was ambitious. It was supposed to replace 135 IT-and paper-based systems across Scotland with a single national integrated system that would be rolled out to all Police Scotland divisions.

A “Gateway review” of the project in March 2013 said the project involved the “largest organisagtional change in the history of Scottish policing”.

The released documents have much praise for the police’s preparatory work on the contract with Accenture. Private consultants were involved as the technical design authority. Deloitte was hired for additional support. There were regular “healthcheck” and Gateway reviews.

Too ambitious?

Bringing together dozens of systems and paper-based processes into a new standardised system that’s supposed to work across a variety of business units, requires – before a single new server is installed – agreement over non-IT changes that are difficult in practice to achieve. It’s mainly a business-change project rather than an IT one.

The business case promised “Full interoperability, of processes and technology, at local and national level.” Was that ever really possible?

The disastrous Raytheon/Home Office e-borders project was a similar classic public sector project failure based on “integration”.  Although it was a much bigger project and far more complex than i6, it followed similar principles: a new national system that would replace a  patchwork of different systems and business processes.

Raytheon could not force change on end-users who did not want change in the way Raytheon envisaged. The Home Office wasted hundreds of millions on the project, according to the National Audit Office which said,

“During the period of the e-borders programme the Department made unrealistic assumptions about programme delivery without recognising the importance of managing a diverse range of stakeholders.

“Delivering the e-borders vision requires that more than 600 air, ferry and rail carriers supply data on people they are bringing in and out of the country, while around 30 government agencies supply data on persons of interest.

“During the e-borders period, the contract made Raytheon responsible for connecting e-borders to these stakeholders’ systems, under the Department’s strategic direction. But carriers and agencies expressed general concerns about the costs and other implications of revising their systems to connect to e-borders, including the interfaces they were expected to use.

“The contract strongly incentivised Raytheon to deliver the roll-out to the agreed schedules but provided less incentive for Raytheon to offer a wider choice of interfaces…Lack of clarity on what was legal under European law further exacerbated the difficult relationships with carriers. These difficulties affected progress in rolling out e-borders from the outset…

“Following the cancellation of the e-borders contract in 2010, the Department [Home Office] took more direct ownership of external relationships instead of working through Raytheon. Transport carriers told us there is now a better understanding of needs and requirements between themselves and the Department.”

The NHS National Programme for IT [NPfIT] was another similar failure, in part because of overly ambitious plans for “integration” – on a scale that could never be imposed on a diverse range of largely autonomous NHS organisations. Some hospitals and GPs did not want a national system that did less than their existing systems. Why would they want to replace their own proven IT with cruder standardised systems for the sake of the common good?

More recently the GP support services contract with Capita has run into serious problems largely because of an overly ambitious objective of replacing fragmented ways of working with a national “common good” system.

A Capita spokesperson said of the new system: ‘NHS England asked Capita to transform what was a locally agreed, fragmented primary care support service, to a national standardised system.”

It’s naïve for politicians and senior public servants to view integration as a public benefit without questioning its necessity in the light of the huge risks.

[Mao Tsedong saw the Great Leap Forward as a public benefit. It was a costly catastrophe, in human and financial terms. ]

Disputes over whether proposals would meet actual needs?

It appears that i6 officials found Accenture’s solutions unconvincing; but it’s likely Accenture found that requirements were growing and shifting, leading to disagreements over varying interpretations of different parts of the contract. Accenture could not compel cooperation by various forces even it wanted to.

It may work elsewhere – but that doesn’t mean it’ll work for you.

This is one of the oldest lessons from countless disaster in the history of the IT industry. It was listed as a key factor in some of the world’s biggest IT disasters in “Crash”.

The business case for i6 says:

“The [Accenture] solution is based on a system delivered to 80,000 officers in the Guardia Civil, Spain’s national police force.

“The procured solution includes software components, software licences, specialist hardware, integration tools and services, business change activities, implementation services, reporting capabilities, data management activities, ongoing support, optional managed service arrangements, additional integration services and other relevant services necessary for the successful implementation of the solution.”

Is it wise to promise huge savings many times greater than projected costs?

Clearly i6 is a political scheme. It’s easy in the public sector to declare at the outset any amount of anticipated savings when it’s clear to everyone that the actual audited savings – or losses – will probably never be announced.

Initial costs were put at £12m, but later revisions put the cost nearer to £46m. More recently costs of £60m have been reported. In 2013, cashable savings to be made by developing i6 were said to be over £61m, with the total cashable and non-cashable savings estimated to be £218m over ten years.

That said, the police appear to have paid over relatively small sums to Accenture, not tens of millions of pounds.

Lessons from past failures have been learned – really?

The Scottish Police Authority gave an unequivocal assurance to its members in June 2013 that i6 will “not suffer the same fate as other high profile large scale IT projects”. This is what the Authority said to its members,

“Delivery Assurance – SPA [Scottish Police Authority] members have sought and been provided with significant assurance that the i6 programme will deliver the intended outcomes and not suffer the same fate as other high profile large scale IT projects.

“The robustness and diligent detail that has gone into the full business case itself provides much of that assurance. Further delivery confidence around i6 comes from a number of sources including:
1. Rigorous Programme Governance.
2. Widespread User Engagement and Robust Requirements Gathering.
3. The creation of a ‘live’ multi-sector i6 Learning Network.
4. The formation of strategic partnership groups.
5. Alignment to the wider Scottish Government Digital Strategy.
6. Active learning from the Audit Scotland Review of Public Sector IT Projects and the Common Performance Management Project (‘Platform’).
7. Significant time and investment in the use of Competitive Dialogue.
8. The formation of a strong and consistent programme team with integrated professional advice & support.
9. Exposure to the full independent OGC Gateway Review Process.
10. An independent Scottish Government Technical Assurance Review.

A growing list of changes.

In February 2016 Accenture said, “This is a very complex project. The complexity of the solution, which has been driven by the client, has increased significantly over the last two years.”

This suggests the scope and specification grew as the many different stakeholders gradually formed a view of what they wanted.

Criticism of the supplier, as if it were the only party responsible or delivering the system.

Police Scotland told members of the Scottish Parliament in February 2016 that Accenture has let the police down.

One question auditors may ask is whether it would have been better for local policing divisions to keep control of their own IT.

Internal reviews too soft, too reassuring?

A technical assurance review in June 2013 gave the i6 project an “amber/green” status.

A secret settlement leaves taxpayers having no clue of how much money has gone down the drain.

The Scottish Police Authority says the settlement is confidential. “The terms of the agreement are commercially confidential. However we can confirm that the settlement results in no financial detriment to the police budget.”

The current police budget may not be affected but how much has already been paid and how much of this is wasted? If no figures are ever given, how can there be proper accountability that could deter a new set of officials making similar mistakes in a future project?

Doomsday Register?

If the public sector kept a published “Doomsday” register of failed projects and programmes and the mistakes made in them, as identified by auditors, the same mistakes would be less likely to be repeated.

Perhaps i6 could be the first entry into a new Doomsday register.

The future’s looking bright (?).

When a project is cancelled, it’s almost inevitable that the consequences will be declared to be minimal; and we’re all left wondering why the project was needed in the first place if the future is so rosy.

Half the story

As things stand,  when a council, police, NHS, or Whitehall project fails and millions of pounds, sometimes tens of millions, even billions, are lost, there’s no incentive for anyone but taxpayers to care – and even then they don’t know half the story.

In the case of i6, once the settlement with Accenture is finalised – with hardly anyone knowing the details – officialdom is free to embark on a similar project in a few years time, with different people involved, and describing it in a different way.

Who cares when the public sector has another IT disaster that follows an age-old script?

**

Project summary

The i6 project was introduced to merge more than 130 different computer and paper systems left in place after eight regional forces were merged to form Police Scotland.

Police Scotland told MSPs in February that they were looking at contingency options because they could not solve scores of faults that had emerged during testing.

Officers involved in the tests said at one point they had found 12 critical errors that made it unusable, and a total of 76 defects that required further work.

Accenture said in February that i6 passed its internal testing but flaws emerged when Police Scotland tested the programme.

**

The Guardian reports on another IT-enabled project problems in Scotland.

“Scottish ministers have already been forced to seek an extension from the European commission after its new £178m farming payments system had to be dramatically scaled back and failed to meet an EU deadline.

“There have been significant delays and cost rises too in a new call-handling and IT system for NHS Scotland’s telephone advice service, NHS 24, which has not yet become operational. Its budget has risen by 55% to nearly £118m, and it is four years late.”

Scottish Police Authority and Accenture terminate i6 contract – Government Computing