Category Archives: private sector lessons for the public sector

Lessons from “stupid” NHS IT scheme – Logica boss

Some wise words from Andy Green, CE of Logica, on lessons from the NPfIT and other failures

By Tony Collins

Andy Green, CE, Logica

Andy Green, chief executive of Logica, speaking to the BBC’s Evan Davis about the NHS National Programme for IT, NPfIT, said:

“It is a stupid thing for the supply chain to have answered, and it’s a stupid thing for the customer to have asked for.”

Green was speaking on Radio 4’s The Bottom Line about corporate “cock-ups and conspiracies”. Other guests were Phil Smith, chief executive of Cisco UK and Ireland, and entrepreneur Luke Johnson.

Green, who joined Logica as CEO in January 2008, said he was in one of the bidders for the NPfIT when he was at BT.

The plan, he said, had been to put the same system into every hospital but later foundation hospitals were able to opt out of the NPfIT.

“Half way through [the NHS IT programme] foundation hospitals were invented,  and suddenly foundation hospitals did not have to go with what the NHS said at all”.

He added: “There were fundamental errors in the whole procurement process, and then real difficulty in delivering what had been promised.”

Evan Davis said the NHS IT scheme had cost billions, achieved little and had been running for years. He asked Green: “What’s the story?”

Green said some things went well including the supply of a network that connects pharmacies and doctors. But …

“What  had been promised by the supply chain was fantastic software that had not been designed yet that was going to completely revolutionise hospitals and delivering that proved to be horrendous… in the end it is foolish to set out on a programme that is going to take seven years with a fixed procurement up front, which says we all know everything about it …”

Lessons

Green spoke of the need for the supplier to understand exactly what the customer wants and whether it is deliverable before the parties agree to draw up a project specification.

“I think the world is beginning to learn about incrementalism. Let’s do something that we can all see and understand.

“Some of our clients we now work with in common teams – we call it co-management – and only when we have worked out exactly what is going to work in the client, and we can deliver, do we specify it as a project.

“Those things tend to go a lot better. We have got used to the fact that we don’t know everything.”

Luke Johnson

Luke Johnson, who is a former chairman of Channel 4, criticised IT suppliers for not getting it right often enough.  “I have bought quite a lot of projects and been involved as a customer many times… As a customer it is a very scary thing because clearly you are not an expert. Your providers are experts and yet they do not seem to be able to get it right often enough it seems to me, given how much they charge.”

Green said there is a high failure rate in the IT industry. “The client sets out one view at the beginning and then they have to change. The sensible defence to this is the partitioning into smaller items and relationships.

“We bluntly always think of our clients over the long run. You need to know people so that you can sit down and have a decent conversation. Too often when these things start to go wrong everybody runs for the contract. Experienced buyers and sellers do not do that: they run for each other and they talk it through, and they work it out, and they put it back on track.

“It’s value that matters. It’s doing something that really changes Patisserie Valerie’s business. [Luke Johnson is chairman of Patisserie Valerie.] What can you do that would transform that. If you can get that done, then if it over-runs by 20% it probably does not matter.”

Luke Johnson: “It depends how much money you’ve got.”

Lowest-price bids

Phil Smith, Cisco

Phil Smith of Cisco said government often has the biggest problems because “they squeeze so much in procurement there is little good value and goodwill left”.He said that on good projects problems are tackled by cooperation but “if every piece of value has been squeezed out before you procure it, your only option is to get something back from it”.

Beware procurement experts

Johnson said if procurement experts take control, and their mantra is to save money, it can often lead to trouble. “I fear that in many aspects of business, it gets down exclusively to price rather than value.

“Quality is out the window. They [procurement experts] can show a saving so they have justified their bonus but the supplier may be rubbish.”

Green said government is in a difficult position when a project starts to go wrong. “You are stuck in a procurement and the poor individual responsible is almost certainly facing a union or a consumer group or a doctor who doesn’t want the thing to happen anyway.”

Evan Davis made the valid point that the costs of projects in the public sector have to be underestimated to get approved. Realistic estimates would be rejected as too costly.

“… The person who is championing this project has to demonstrate to superiors that it is not too expensive. It is only by taking the cheapest bid and starting the thing off that you can sell the project higher up and of course down the line it costs a heck of a lot more.”

Luke Johnson: “We all know in many sectors there are providers that will take things at cost or even less with a view that they will somehow bulk it out and make a margin on the way. They know the client will need variations.

Innovation means taking risks

Luke Johnson: “If you want an innovative society, if you want one that is willing to take risks, to generate new technologies, new jobs, new businesses, then it involves failures and cock-ups.

“I think the British have got vastly better in recent years in accepting that as part of the journey and that is incredibly healthy.”

BBC R4’s The Bottom Line – Cock-ups and conspiracies.

New child support system has 90,000 requirements – in phase one

                               A new old-style government IT disaster?

By Tony Collins

While officials in the Cabinet Office offcials try to simplify and cut costs of Government IT, a part of the Department for Work and Pensions has commissioned a system with 90,000 requirements in phase one.

The projected costs of the child maintenance system have risen by 85% and the delivery date has slipped by more than two years.

Even with 90,000 requirements, phase one, which is due to go live in October, excludes 70 requirements that are “deemed critical” says a report published today by the National Audit Office.

The NAO report indicates that the Child Maintenance and Enforcement Commission has commissioned an old-style large IT system using traditional developing techniques and relying on large companies.

G-Cloud and SMEs have not featured in the Commission’s IT strategy – and it abandoned agile techniques last year on its child maintenance project.

The Commission put the cost of its new child maintenance system at £149m in January 2011. Ten months later it put the cost at £275m, an 85% increase. The Commission was unable to give the NAO a full explanation for the difference.

Lessons from past failures not learned?

Today’s NAO report says there is a risk the Commission will repeat mistakes by the Child Support Agency whose IT system and business processes were criticised in several Parliamentary reports. The Commission takes in the work of the Child Support Agency – and indeed runs its own systems and the Child Support Agency’s in parallel.

Officials at the Commission told the NAO they have a good track record of holding back IT releases until they are satisfied they will work.  “Nevertheless, we found that the Commission is at risk of repeating many of the mistakes of 2003,” said the NAO. Those mistakes include over-optimism and a lack of internal expertise to handle suppliers.

Mixing “agile” and “waterfall” doesn’t work

Initially civil servants at the Commission tried to “mix and match” agile and traditional developing techniques – which Agile advocates say should not be attempted.

In 2011 the Commission gave up on agile and “reverted to a more traditional approach to system development” says the NAO report.

The mix and match approach meant there were two distinct routes for specifying requirements and “resulted in duplicated, conflicting and ambiguous specifications”.  The Commission did not have previous experience of using the agile approach.

The Commission’s child maintenance system was due to go live in April 2010 but the delivery date has slipped three times. Phase one is now due to go live in October 2012 and phase two in July next year but the NAO report raises questions about whether the go-lives will happen successfully. The Commission has not planned in its financial estimates for the failure of the system.

The NAO finds that the Commission has struggled to make its requirements for the new system clear. The Commission’s main developer Tata Consulting Services has had protracted discussions over the meaning and implementation of requirements.

The NAO also hints that IT costs may be out of control. It says the Commission may not secure value for money without properly considering alternative options for restructuring and “adequately controlling its IT development …”

These are some of the NAO’s findings:

IT costs could increase further

“The new system is based on ‘commercial off-the-shelf’ products. However, a recent audit by Oracle identified that the performance, maintainability and adaptability of the new system would be key risks. This could increase the cost of supporting the system. The scheme does not yet include plans for the integration with HM Revenue & Customs’ Real Time Information system due to be implemented in 2013, or introducing Universal Credit because of the differing timescales,” says the NAO which adds:

“Achieving the Commission’s plans without further cost increases or delays appears unlikely. The Commission reported to the audit committee in October 2011 on the high risk that the change programme may not deliver phase two functionality within agreed timescales … The Commission did not develop a benefits realisation plan until November 2011.”

103,000 of Commission’s 1.1m cases are handled manually

“Ongoing technical problems have resulted in a large number of cases being removed from the IT system and managed manually. These are known as clerical cases … The Commission has had to operate the ‘old’ and ‘current’ schemes in parallel.  Due to flaws in the IT systems for each scheme, some 100,000 cases have had to be processe:d separately by clerical staff at a cost of £48 million,” says the NAO. It takes 900 contractors to manage the clerical cases.

Comment

Despite numerous NAO reports on failures of Government IT-based projects over the past 30 years the disasters are still happening, with the same mistakes repeated: over-optimism in every aspect of the project including timetables and financial estimates; excessive complexity and over-specification, no sign of cost-consciousness and, worst of all, an apparent indifference to being held accountable for a major failure.

A glance at the monthly outgoings of the Commission (well done to the coalition for requiring departments and agencies to publish contracts over £25,000) show sizeable and regular payments to familiar names among the large suppliers: HP Enterprise Services (formerly EDS), Capgemini, Tata Consultancy Services, BT Global Services and Capita. There is hardly an SME in sight and no sign of imaginative thinking.

Meanwhile some senior officials at the Commission put in monthly expenses for thousands of pounds in travel, accomodation and subsistence for “Commission meetings”. One wonders: to what useful effect?

Officials at the Cabinet Office are trying to change the culture of departments and agencies. They are encouraging departmental heads to do things differently. They advocate the use of  SMEs to show how new ways of working can trounce traditional approaches to projects.

But the Cabinet Office has little influence on the Department of Work and Pensions. Indeed the DWP has lost its impressive chief innovator James Gardner.

We praise the NAO for noting that the Commission risks repeating the IT-related and project management mistakes of the Child Support Agency. But we note with concern that the NAO still puts up with Whitehall’s non-publication of  Gateway reviews, which are independent reports on the progress or otherwise of big and risky IT-based projects.

Would the Commission have been so apparently careless of the risks if it had known that regular Gateway reports on its shortcomings would be published?

How many more government IT-based projects are late, over budget and at risk of failing, their weaknesses hidden by an unwritten agreement between the coalition and civil servants to keep Gateway reviews secret?

NAO report – Child Maintenance and Enforcement Commission: cost reduction

Government repeating child support mistakes – ComputerworldUK

Mutuals: a novel means of driving down demand for public services?

By David Bicknell

A recent piece in the Guardian local government network has come up with the intriguing idea that mutuals can help drive down demand for public services.

The article, by Ross Griffiths, a partner at law firm Cobbetts,  suggests that if  as a service user, you are dealing with a provider that is your mutual, you are more likely to think twice about the demands you are making on it, and the effect that might have on the service and other users. It argues that this is the ‘Holy Grail’ of the mutual project – allowing providers to deliver services more cheaply not by making cuts, but by reducing demand.

The piece asks whether in today’s local government, where efficiency must be a big part of any changes to services, this is something that mutual structures can deliver. Or are they, as the article asks, ‘little more than a frivolity that should be saved for less straitened times?’

Links

http://www.guardian.co.uk/local-government-network/2011/dec/08/new-mutuals-pick-winners

http://www.number10.gov.uk/take-part/public-services/start-a-public-service-mutual/

DWP defends £316m HP contract

By Tony Collins

The Department for Work and Pensions could lead the public sector in technical innovations. It has had some success in cutting its IT-related costs. It has also had some success so far with Universal Credit, which is based on agile principles.

It has further launched an imaginative welfare-to-work scheme , the so-called Work Programme, which seeks to get benefit claimants into jobs they keep.

Despite media criticism of the way the scheme has been set up – especially in the FT – a report by the NAO this week made it clear that the DWP has, for the most part, taken on risks that officials understand.

Some central government departments have updated business cases as they went through a major business-change programme and not submitted the final case until years into the scheme, as in parts of the NPfIT.

But the DWP has implemented the Work Programme unusually quickly, in a little more than a year, by taking sensible risks.  The NAO report on the scheme said the business case and essential justification for the Work Programme were drawn up after key decisions had already been made. But the NAO also picked out some innovations:

– some of the Work Programme is being done manually rather than rush the IT

– suppliers get paid by results, when they secure jobs that would not have occurred without their intervention. And suppliers get more money if the former claimant stays in the job.

– the scheme is cost-justified in part on the wider non-DWP societal benefits of getting the long-term unemployed into jobs such as reduced crime and improved health.

So the DWP is not frightened of innovation. But while Universal Credit and welfare-to-work scheme are centre stage, the DWP is, behind the safety curtain, awarding big old-style contracts to the same suppliers that have monopolised government IT for decades.

Rather than lead by example and change internal ways of working – and thus take Bunyan’s steep and cragged paths – the DWP is taking the easy road.

It is making sure that HP, AccentureIBM and CapGemini are safe in its hands. Indeed the DWP this week announced a £316m desktop deal with HP.  EDS, which HP acquired in 2008, has been a main DWP supplier for decades.

DWP responds to questions on £316m HP deal 

I put it to the DWP that the £316m HP deal was olde worlde, a big contract from a former era. These were its responses. Thank you to DWP press officer Sandra Roach who obtained the following responses from officials. A DWP spokesperson said:

“This new contract will deliver considerable financial savings and a range of modern technologies to support DWP’s strategic objectives and major initiatives such as Universal Credit.

“The DWP has nearly 100,000 staff, processing benefits and pensions, delivering services to 22 million people.

“DWP is on schedule to make savings of over £100m in this financial year for it’s Baseline IT operational costs, including the main IT contracts with BT and HPES [Hewlett Packard Enterprise Services].

“All contracts have benchmarking clauses to ensure best value for money in the marketplace.

“The five year contract was awarded through the Government Procurement framework and has been scrutinised to ensure value for money.”

My questions and the DWP’s answers:

Why has the DWP awarded HP a £316m contract when the coalition has a presumption against awarding contracts larger than £100m?

DWP spokesperson: “The Government IT Strategy says (page 10) ‘Where possible the Government will move away from large and expensive ICT projects, with a presumption that no project will be greater than £100m. Moving to smaller and more manageable projects will improve project delivery timelines and reduce the risk of project failure’.

“HM Treasury, Cabinet Office and DWP’s commercial and finance teams have scrutinised the DWP Desktop Service contract to ensure that it represents the most economically advantageous proposition.”

What is the role, if any for SMEs ?

DWP: “There are a number of SMEs whose products or services will form part of or contribute to the DWP Desktop Service being delivered by HP, for example ActivIdentity, Anixter, AppSense, Azlan, Click Stream, Cortado, Juniper Networks, Quest Software, Repliweb Inc, Scientific Computers Limited (SCL), Westcon etc.”

Why is there no mention of G-Cloud?

DWP: “Both the new contract and the new technical solution are constructed in such a way as to support full or partial moves to cloud services at DWP’s discretion.”

Comment:

For the bulk of its IT the DWP is trapped by a legacy of complexity. It is arguably too welcoming of the safety and emollients offered by its big suppliers.

The department is not frightened by risk – hence the innovative Work Programme which the NAO is to be commended on for monitoring at an early stage of the scheme. So if the DWP is willing to take on sensible risks, why does it continue to bathe its major IT suppliers in soothingly-large payments, a tradition that dates back decades? What about G-Cloud?

DWP reappoints HP on £316m desktop deal

DWP signs fifth large deal with HP

“DWP awards Accenture seven year application services deal”

“DWP awards IT deals to IBM and Capgemini”

NPfIT Cerner go-live at Bristol has “more problems than anticipated”

By Tony Collins

The BBC reports that there are “more problems than anticipated” with a patient-booking system at two Bristol hospitals run by North Bristol NHS Trust.

The trust describes the problems as “teething”.  Consultants say the problems are “potentially dangerous”.

Last month North Bristol went live with the Cerner Millennium system under an NPfIT contract with BT. The Trust says problems are due to software being used incorrectly. They have led to some patients missing their operations and the wrong patients being booked for operations, says the BBC.

Emails from executives at Frenchay and Southmead hospitals, seen by the BBC, said staff should be “vigilant” to check lists were “completely accurate”.

BBC Points West’s health correspondent Matthew Hill said emails sent by consultants to hospital bosses claimed operation lists printed by the system were “complete fiction” and “potentially dangerous”.

One consultant told the BBC he had been put down to operate on patients from a completely different speciality.

The trust said there had been “teething problems” and that there had been “more problems than anticipated”.

In an email to staff the trust said the change of system had been “a very big change” so there was “no surprise” there had been difficulties.

A trust spokesman said there were a series of problems around outpatients and the associated clinics and some of the data moved from old systems had not migrated as planned.

“We need to ensure that we rebuild and recreate the clinics to match what people expect them to be on the ground,” he said.

“In theatres we have had some issues but have absolutely ensured from the outset that clinical safety has been at the top and have ensured any risks and issues have been mitigated.”

Conservative MP Richard Bacon, a member of the Public Accounts Committee, has established through a Parliamentary question that the cost of the North Bristol Cerner implementation is much higher than for a non-NPfIT installation in the same city.

Health Minister Simon Burns told Bacon that the costs of a Cerner Millennium deployment at the North Bristol NHS Trust were £15.2m for deployment and an annual service charge of £2m.

This brought the total cost of the Cerner system over seven years to about £29m, which was more than three times the £8.2m price of a similar deployment outside of the NPfIT at University Hospitals Bristol Foundation Trust.

Comment

Several Cerner implementations under the NPfIT have gone awry but the problems have eventually been resolved. The question is whether patient care and treatment is affected in the meantime. The lack of openness over problems with patient care in the NHS mean that the answer will probably never be known, which underlines the need for better regulation of hospital IT implementations.

Does hospital IT need airline-style safety certification?

HM Courts Service hides “Libra” IT’s new shortcomings

By Tony Collins

A report published today by the National Audit Office highlights how limitations in Libra, a case management IT system in use across magistrates’ courts, has contributed towards  HM Courts Service’s inability to provide basic financial information to support the accounts.

HM Courts Service claimed a success for the troubled Libra system in 2008 – but the failure of the system was more enduring and deep-rooted than thought. The problems were kept hidden until today’s NAO report because the present and past governments have kept “Gateway” progress reports on IT-based projects confidential.

In an unusual step, the head of the NAO, Amyas Morse, has “disclaimed” his audit opinion on the accounts of the HM Courts Service, largely because of a lack of financial information.

Disclaiming an audit opinion is more serious than qualifying the accounts of a government department or agency. Qualifying the accounts means that Morse has reservations on whether figures presented to the NAO are accurate. Disclaiming an audit opinion means that Morse lacks the basic information on which to give any opinion on the accounts.

MP Richard Bacon, a member of the Public Accounts Committee, says that disclaiming an audit opinion is the “auditor’s nuclear button”.

The NAO report today puts the focus on inadequacies in the “Libra” system which is supplied by Fujitsu and STL, with integration work by Accenture.

Fujitsu originally estimated the cost of Libra, a case management system for magistrates’ courts, at £146m. By March the estimated costs were £447m and were expected to rise further. The Libra project took 16 years to complete.

Problems and cost increases on the Libra system were well known in 2003 when the NAO criticised the management of the project. After that all went quiet until in 2008 when HM Courts Service declared Libra a success.

Now the NAO’s Morse says:

“Because of limitations in the underlying systems, HM Courts Service has not been able to provide me with proper accounting records relating to the collection of fines, confiscation orders and penalties. I have therefore disclaimed my audit opinion on its Trust Statement accounts.”

In a statement the NAO criticises the Libra system directly:

“Today’s report highlights how limitations in Libra, the case management IT system in use across magistrates’ courts, and similar systems have contributed towards  HM Courts Service’s inability to provide information at an individual transactions level to support the accounts.”

The NAO says that the Ministry of Justice plans to investigate further the functionality of Libra to determine whether it is possible to provide evidence to support accruals-based financial reporting.

Says the NAO:

“In particular, the Ministry and HMCTS [HM Courts and Tribunals Service] believe that it may be possible to obtain evidence over fines and confiscation orders if a suitable report is run shortly after the month end.

“ However, the Ministry and HMCTS have informed me that they may not be able to address these fundamental issues until Libra is significantly enhanced or replaced with a new case management and accounting system. The timing of this enhancement or replacement is currently uncertain. However, the Ministry have committed to ensuring that any replacement for Libra includes accounting functionality to enable financial reporting.”

MP Richard Bacon, who has followed the Libra project for many years, says:

“This is a disgraceful position for the Courts Service to have reached.  It is true that the Libra computer system is both expensive and useless but we have known this for many years (Cost of Courts’ IT system triples) and public bodies still have a duty to keep proper records.

“We are now looking at a possible £1.4 billion loss in uncollected fines and penalties partly because of the longstanding shambles that passes for record-keeping in the courts service.

“For centuries, people have kept accurate records and accounts using pen and paper. This could still be done now if needed and if there were sufficient will to do it.”

Margaret Hodge MP, Chair of the Committee of Public Accounts, said:

“It is really worrying that HM Courts and Tribunals Service can’t produce basic financial records.  HM Courts Service is responsible for collecting fines and penalties, but we can’t tell if this money is accounted for properly.

“The Comptroller and Auditor General has taken the rare step of disclaiming his audit opinion – the Committee will be looking for HM Courts and Tribunals Service to improve.”

Comment:

It is astonishing that HM Courts Service has been able to continue in operation without MPs having idea until today that the costly Libra computer system was unable to provide basic financial information.

Parliament was kept in the dark about Libra’s new shortcomings because “Gateway” review reports in IT-based projects and programmes are kept confidential. It is a pity for taxpayers and accountability on major projects that ministers are surrendering to the wishes of civil servants who want Gateway reports kept confidential.

NAO report on Courts Service.

Why hospital IT needs airline safety culture

By Tony Collins

Earlier this month the pilots of a Boeing 787 “Dreamliner” carrying 249 passengers aborted a landing at Okayama airport in Japan when the wheels failed to deploy automatically. The pilots circled and deployed the landing gear manually.

A year ago pilots of an Airbus A380, the world’s largest passenger plane, made an emergency landing at Singapore on landing gear that they deployed using gravity, the so-called “gravity drop emergency extension system”.

In both emergencies the contingencies worked.  The planes landed safely and nobody was hurt.

Five years earlier, during tests, part of the landing gear on a pre-operational A380 failed initially to drop down using gravity.

The Teflon solution

The problem was solved, thanks in part to the use of Teflon paint (see below). Eventually the A380 was certified to carry 853 passengers.

Those who fly sometimes owe their lives to the proven and certified backup arrangements on civil aircraft. Compare this safety culture to the improvised and sometimes improvident way some health IT systems are tested and deployed.

Routinely hospital boards order the installation of information systems without proven backup arrangements and certification. Absent in health IT are the mandatory standards that underpin air safety.

When an airliner crashes investigators launch a formal inquiry and publish their findings. Improvements usually follow, if they haven’t already, which is one reason flying is so safe today.

Shutters come down when health IT fails 

When health IT implementations go wrong the effect on patients is unknown. Barts and The London NHS Trust, the Royal Free Hampstead, the Nuffield Orthopaedic Centre, Barnet and Chase Farm Hospitals NHS Trust and other trusts had failed go-lives of NPfIT patient administration systems. They have not published reports on the consequences of the incidents, and have no statutory duty to do so.

Instead of improvements triggered by a public report there may, in health IT, be an instinctive and systemic cover-up, which is within the law. Why would hospitals own up to the seriousness of any incidents brought about by IT-related confusion or chaos? And under the advice of their lawyers suppliers are unlikely to own up to weaknesses in their systems after pervasive problems.

Supplier “hold harmless” clauses

Indeed a “hold harmless” clause is said to be common in contracts between electronic health record companies and healthcare provider organisations. This clause helps to shift liability to the users of EHRs in that users are liable for adverse patient consequences that result from the use of clinical software, even if the software contains errors.

That said the supplier’s software will have been configured locally; and it’s those modifications that might have caused or contributed to incidents.

Done well, health IT implementations can improve the care and safety of patients. But after the go-live of a patient administration system Barts and The London NHS Trust lost track of thousands of patient appointments and had no idea how many were in breach of the 18-week limit for treatment after being referred by a GP. There were also delays in appointments for cancer checks.

At the Royal Free Hampstead staff found they had to cope with system crashes, delays in booking patient appointments, data missing in records and extra costs.

And an independent study of the Summary Care Records scheme by Trisha Greehalgh and her team found that electronic records can omit allergies and potentially dangerous reactions to certain combinations of drugs. Her report also found that the SCR database:

–  Omitted some medications

–  Listed ‘current’ medication the patient was not taking

–  Included indicated allergies or adverse reactions which the patient probably did not have

Electronic health records can also record wrong dosages of drugs, or the wrong drugs, or fail to provide an alert when clinical staff have come to wrongly rely on such an alert.

A study in 2005 found that Computerized Physician Order Entry systems, which were widely viewed as a way of reducing prescribing errors, could lead to double the correct doses being prescribed.

One problem of health IT in hospitals is that computer systems are introduced alongside paper where neither one nor the other is a single source of truth. This could cause mistakes analogous to the ones made in early air crashes which were caused not by technology alone but pilots not fully understanding how the systems worked and not recognising the signs and effects of systems failing to work as intended.

In air crashes the lessons are learned the hard way. In health IT the lessons from failed implementations will be learned by committed professionals. But what when a hospital boss is overly ambitious, is bowled over by unproven technology and is cajoled into a premature go-live?

In 2011, indeed in the past few months, headlines in the trade press have continued to flow when a hospital’s patient information system goes live, or when a trust reaches a critical mass of Summary Care Record uploads of patient records (although some of the SCR records may or not be accurate, and may or may not be correctly updated).

What we won’t see are headlines on any serious or even tragic consequences of the implementations. A BBC File on 4 documentary this month explained how hospital mistakes are unlikely to be exposed by coroners or inquests.

So hospital board chief executives can order new and large-scale IT systems without the fear of any tragic failure of those implementations being exposed, investigated and publicly reported on. The risk lies with the patient. Certification and regulation of health IT systems would reduce that risk.

Should health IT systems be tested as well as the A380’s landing gear? – those tests in detail

Qantas Flight 32 was carrying 466 passengers when an engine exploded. The Airbus A380 made an emergency landing at Singapore Changi Airport on 4 November 2010. The failure was the first of its kind for the four-engined A380.

Shrapnel from the exploding engine punctured part of the wing and damaged some of the systems, equipment and controls. Pilots deployed the landing gear manually, using gravity  – and it worked well. Despite many technical problems the plane landed safely.

Five years earlier, tests of a manual deployment of the A380’s landing gear failed initially. It happened in a test hangar, more than a year before the A380 received regulatory approval to carry 853 passengers.

The story of the landing gear tests is told by Channel 4 as part of a well-made documentary, “Giant of the Skies” on the construction and assembly of the A380.  Against a backdrop of delays and a budget overspend, Airbus engineers must show that if power is lost the wheels will come down on their own, using gravity.

The film shows an Airbus A380 suspended in a hangar while the undercarriage test gets underway. The undercarriage doors open under their own weight and a few seconds later the locks that hold up the outer wheels release. Two massive outer sets of four wheels each fall down through a 90-degree arc. Something goes wrong.  At about 45 degrees, one of the Michelin tyres catches on an undercarriage door which looks as if it has not opened as fully as it would have if powered electrically. Only after 16 seconds does the jammed wheel set slip free. Engineers watching the test look mortified.

Simon Sanders, head of landing gear design at Airbus, tells Channel 4: “We need to understand and find a solution.”

An engineer smeared some grease (Aeroshell 70022 from Shell, Houston) on a guide ramp where the A380’s wheels are supposed to push the door open in an emergency loss of power. This worked and the test was successful: the left-side outer landing gear doors opened under their own weight; a few seconds later the wheels fell down, also under their own weight, and this time the tyre that had jammed earlier hit the grease on the door and slid down without any delay. But a permanent solution was needed.

A month later Airbus repeated the undercarriage “gravity” test. Sanders told Channel 4: “We have applied a layer of Teflon paint which is similar to the Teflon coasting you have on non-stick flying pans. This will reduce the friction when we do the free-fall [of the undercarriage]. We are now going to perform the test to demonstrate that with this low-friction Teflon coating we have solved the problem.”

This time the A380’s chief test pilot Gérard Desbois was watching. If the wheels got struck again Desbois could refuse to accept the aircraft for its first test flight, which would mean a further delay.

The loss-of-power test began. The outer landing gear doors opened … the wheels fell down under their own weight … and jammed again. This time they freed themselves quicker than before. After some hesitation Desbois accepted the aircraft on the basis that if power were lost and the left outer landing wheels took a little longer to come down than the right outer set this would not be a problem.  The gravity free-fall backup system was further refined before the A380 went into service.

The importance of the tests was shown in 2010 when an exploding Rolls-Royce Trent 900 engine on an Airbus A380,  Qantas Flight 32 from Singapore to Sydney, caused damage to various aircraft systems including the landing gear computer which stopped working.  The pilots had to deploy the landing gear manually. The official incident report shows that all of the A380’s 22 wheels deployed fully under the gravity extension backup arrangements.

If a hospital board had been overseeing the A380’s tests back in 2005, would directors have taken the view that the test was very nearly successful, so the undercarriage could be left to prove itself in service?

For the test engineers at Airbus, safety was not a matter of choice but of regulation and certification. It is a pity that the deployment of health IT, which can affect the safety of patients, is not a matter of regulation or certification.

Links:

Oxford University Hospitals NHS Trust postpones major IT go-live.

Giant of the Skies – Channel 4 documentary on manufacture and testing of the Airbus A380

Officials pay supplier invoices – then raise purchase orders

This morning the National Audit Office has published a report that says the Equality and Human Rights Commission, in up to 35% of cases, raises its purchase order after it gets the invoice from suppliers.

It’s unlikely that any private sector company could survive if it didn’t know what it owed, didn’t know what it had bought, and had to wait for an invoice from the supplier to raise the purchase order.

Amyas Morse, the head of the NAO, says in his report today:

“While I welcome the considerable improvements that the Commission has made in its controls over procurement, there are still areas where it needs to make improvements. In particular, up to 35% of the Commission’s purchase orders are still not raised until after the Commission has received an invoice for goods and services.

“This means that Commission staff are committing funds without going through proper processes and are avoiding some of the checking processes. Consequently the Commission does not have an accurate understanding of its committed expenditure at any one point in time.

“The Chief Executive has made it clear that he takes noncompliance with these processes seriously such that in cases of repeated non-compliance delegations will be withdrawn.”

A common practice? 

Is this absence of proper accounting worryingly common in central government and its agencies, particularly on IT contracts?

Auditors told us that in the case of NPfIT contracts they found some invoices that were paid when they came in, awaiting reconciliation with any past paperwork.

This, perhaps, ties in with the experiences of Conservative MP Richard Bacon, a member of Public Accounts Committee who, when asking civil servants for a breakdown of IT spending has, in the past, been referred to the department’s IT supplier.

On the C-Nomis IT project for prisons, the National Offender Management Service paid £161m without keeping any record of what the payments were for.

The Cabinet Office wants to cut the £17bn or so spent every year on public sector IT. But before departments, agencies and other organisations cut their costs they’ll need to know what those costs are. Maybe they should ask their major IT suppliers? We wonder if the domination of GovIT by a small number of suppliers has got to the stage where it’s the suppliers managing the civil service IT budgets. If that’s the case it is not the fault of suppliers.

UK GovIT often a barrier not enabler says Cabinet Office official

By Tony Collins

In an interview for UKauthority.com Chris Chant, Executive Director at the Cabinet Office and head of the G-Cloud programme,  debunks the claims of some that GovIT doing a great job and should remain largely untouched.

Chant says: “IT is supposed to be an enabler. Quite often in my experience in government IT it is actually a barrier to getting things done. That’s no way to use IT. It is supposed to support what we do.”

His criticism puts into context claims by some in the civil service that GovIT is an unpublicised success because of the ease and success of online re-taxing of vehicles, the payment of benefits to millions of people and the collection of taxes.

Chant has made clear his concern that some departments are locked into major IT suppliers through costly, inflexible long-term contracts that, in some cases, are being signed anew.

“In the main we are not delivering good quality IT to government and public sector workers. We are not delivering good IT solutions to the citizen …”

He calls for internal change and describes SMEs as “front and centre to what we need”.

“It is with SMEs that agility and innovation lie, and it is that market we are really encouraging… Good IT is not developed by spending a long time trying to work out a definitive answer, and then taking ages over delivering it only to discover it is not what we needed in the first place. It is about iteration. I have said all along that we do not have all the answers. We will develop as we go and take SMEs with us.”

Asked whether the public sector is ready for the cloud Chant replies: “No we are not. We are quite a way from that… We are very well positioned to operate in a world where our IT is delivered by multinationals but now it is a different world.”

He says that the cloud has security limitations. “It is difficult to see the cloud in the short term handling some of the higher security aspects of what we do but for a lot of what government does it’s about commodity products and we need to get people in who know how to handle that.”

The focus he says must always be on the citizen – assumptions should not start from a departmental or systems standpoint. “We will need to change the way we do things; we will need some new people and I suspect a lot of retraining. I think we will need a lot fewer people working on the client side of government IT…

“We are in really tough times and the idea that we can operate with [current] cost levels is wrong…”

Government clouds take shape – UKauthority.com.

The unavoidable truths about GovIT – Chris Chant.

Vested interests will try to stop GovIT changing.

What exactly is HM Revenue and Customs paying Capgemini billions for?

DWP signs new large contracts with HP, Accenture, IBM and Capgemini.

What exactly is HMRC paying Capgemini billions for?

By Tony Collins

When the National Audit Office published a largely-positive report on HMRC and its online filing systems last month, the department received some justifiably good media coverage.

What was little noticed was that auditors were unable to get a breakdown of what HMRC is paying its “Aspire” systems suppliers Capgemini and Fujitsu for online filing.

Collect your car after a service and your bill has a breakdown of the parts used, their cost, and the cost of labour. But when HMRC pays around £8bn to Capgemini for its Aspire IT service, a clear breakdown of costs is not provided.

Says the NAO report:  HM Revenue & Customs – The expansion of online filing of tax returns:

“HMRC has a high-level view of the overall costs of ICT provision through the ASPIRE contract. It has been taking steps to improve that information and achieve cost savings. It does not yet have a detailed breakdown of the costs of online filing services, so it cannot benchmark those costs to assess their value for money.

“HMRC is currently negotiating with the ASPIRE contractors to obtain a clearer breakdown of the costs of ICT services provided.”

In case you think the NAO has made a mistake, and that HMRC must surely have a breakdown of the costs of Capgemini’s services, the NAO makes it completely clear that the Department has no such breakdown.

“The ASPIRE contract includes a rolling programme of benchmarking the prices HMRC pays for the various contracted services, including those relevant to online filing … Since 2010, HMRC has introduced new processes to improve information on the cost and use of ICT and benchmarking of key ICT service lines. These processes cannot yet provide information in sufficient detail to benchmark and challenge the cost of individual online filing services…”

Unfortunately for taxpayers it is not unusual for a department to pay its main IT supplier without having a full breakdown of the bills.

Several years ago the Conservative MP Richard Bacon asked criminal justice officials for a breakdown of costs on the “Libra” contract for magistrates’ courts IT. The Department didn’t know. So it referred Bacon to Fujitsu, Libra’s main supplier.

Fujitsu eventually provided a breakdown so vague – with high-level categories such as “network services” – that Bacon had little choice but to ask the same questions repeatedly to find out how public funds were being spent with Fujitsu.

In the end Bacon failed – and he had little support from departmental officials.

Now, about 10 years on, Capgemini is keeping HMRC in a similar level of ignorance.

Can any department be trusted with the public funds to pay its IT suppliers billions of pounds without a clear and unambiguous breakdown of what it is paying for?

A supplier’s reluctance to supply a breakdown of costs is understandable.  A clear breakdown could clear a path through the fog of supplier pricing, so it could make price comparisons easier.

It is up to HMRC to insist on a breakdown.  Its IT services have been outsourced since 1994. Shouldn’t it know exactly what it is paying billions for by now?

Chris Chant, an Executive Director in the Cabinet Office, has deplored the high costs of locked-in long-term contracts with out-of-season monolithic suppliers.  Does the Aspire contract alone make a good case for the reform of central government?

The unavoidable truths about GovIT – Chris Chant.