Category Archives: IT-related failures

Why are councils hiding exit costs of outsourcing deals – embarrassment perhaps?

Tony Collins

Excerpt from Taunton Deane council’s confidential “pink pages”.
The last sentence contains a warning that IBM-owned SWO – Southwest One – may try to “maximise revenues” on exiting its joint venture with the council.

Somerset County Council has refused a Freedom of Information request for the costs of exiting its joint venture with IBM.

But a secret report written last year by officers at Taunton Deane Borough Council – which was a party to the IBM-owned joint venture company Southwest One  – warned that the supplier could attempt to “maximise revenues on exit”.

It said,

“… from experience anything slightly ambiguous within the contract is likely to be challenged by SWO [Southwest One] in order to push it into the chargeable category as they attempt to maximise revenues on exit”.

A separate section of the confidential report said,

“disaggregating from the SWO [Southwest One] contract will be complex and expensive …”

Taunton Deane Borough Council did not tell councillors what the exit turned out to be. The figures are also being kept secret by Somerset County Council which signed the “transformative” SWO joint venture deal with IBM in 2007.

Both councils have now brought back services in-house.

Secrecy over the exit costs is in contrast to Somerset’s willingness to talk in public about the potential savings when local television news covered the setting up of Southwest One in 2007.

The silence will fuel some local suspicions that exit costs have proved considerable and will have contributed to the justifications for Somerset’s large council tax rise this year.

£69m losses?

David Orr, a former Somerset County Council IT employee, has followed closely the costs of the joint venture, and particularly its SAP-based “transformation.

It was his FOI request for details of the exit costs that the council refused.

Orr says that Somerset has lost money as a result of the Southwest One deal. Instead of saving £180m, the joint venture has cost the council £69m, he says.

FOI

Under the Freedom of Information Act, Orr asked Somerset for the “total contract termination costs” including legal, consultancy, negotiation, asset valuations, audit and extra staffing.

He also asked whether IBM was paid compensation for early termination of the Southwest One contract. In replying, the council said,

“The Authority exited from a significant contract with Southwest One early, and the services delivered through this contract were brought back in-house in November 2016.

“The Authority expects the costs to fall significantly now it has regained control of those services.

“Somerset County Council made payment under the ‘Termination for Convenience’ provisions of the original contract. We do hold further information but will not be releasing it at this point as we believe to do so would damage the commercial interests of the County Council, in that it would prejudice the our negotiating position in future contract termination agreements in that it would give contractors details on what terms the Council was willing to settle …”

Orr will appeal. He says the Information Commissioner has already established a principle with Suffolk Coastal District Council that the termination costs of a contract with a third party should be disclosed. The commissioner told Suffolk Coastal council that, in opting out of FOI,

“there is no exemption for embarrassment”

Hidden costs

Taunton’s pink pages paper said that the Southwest One contract’s Exit Management Plan provided for a smooth transfer of services and data, and for access to staff to assess skills and do due diligence.

In practice, though, there were many exit-related complications and costs – potential and actual. The paper warned that Taunton would need to find the money for:

  • Exit programme and project management costs
  • Early termination fees
  • Contingency
  • ICT infrastructure disaggregation
  • Service transition and accommodation costs
  • Disaggregating SAP from Southwest One. Also the council would need to exit its SAP-based shared services with Somerset County Council because the estimated costs were lower when run on a non shared services basis. SAP covered finance, procurement, HR, payroll, website and customer relationship management.
  • Costs involved in a “soft” or “hard” (adversarial) exit.
  • Estimating council exit costs when IBM was keeping secret its own Southwest One running costs.
  • Staff transfer issues.

Comment

So much for open government. It tends to apply when disclosures will not embarrass local government officials.

In 2007 Somerset County Council enjoyed local TV, radio and newspaper coverage of the new joint venture with IBM. Officials spoke proudly on camera of the benefits for local taxpayers, particularly the huge savings.

Now, ten years later, the losses are stacking up. Former Somerset IT employee and FOI campaigner Dave Orr puts the losses at £69m. And local officials are keeping secret the further exit costs.

Suffolk Coastal District Council lost an FOI case to withhold details of how much it paid in compensation to a third party contractor to terminate a contract. But at least it had published its other exit costs.

Somerset is more secretive. It is withholding details of the sums it paid to IBM in compensation for ending the joint venture early; it also refuses to publish its other exit costs.

Trust?

Can anything said by councils such as Somerset or Barnet in support of major outsourcing/joint venture deals be trusted if the claimed savings figures are not audited and the other side of the story – the hidden costs – are, well, hidden?

In local elections, residents choose councillors but they have no say over the appointment of the permanent officials. It’s the officials who decide when to refuse FOI requests; and they usually decide whether the council will tell only one side of the story when public statements are made on outsourcing/joint ventures.

Across the UK, local councils employed 3,400 press and communications staff –  about double the total number in central government – in part to promote the authorities’ services and activities.

What’s the point if they publicise only one side of the story – the benefits and not the costs?

Somerset’s decision to refuse Orr’s reasonable FOI request makes, in its own small way, a mockery of open government.

It also gives just cause for Somerset residents to be sceptical about any council statement on the benefits of its services and activities.

Advertisements

Will MPs call BA to account over IT power problems?

By Tony Collins

Experts are questioning BA’s explanation of the power problems that disrupted the travel plans and arrangements for 75,000 people at the weekend.

BA says it is “reviewing” what went wrong at the weekend but is under no regulatory duty to publish the findings.

There is little pressure from shareholders to hold BA to account. The share price of BA’s parent International Airlines Group is higher today than a month ago.

Sceptical

Yesterday the BBC’s business editor Simon Jack accused IAG of dodging tough questions it will “surely have to answer” and the FT quoted IT and electricity experts who are sceptical of the airline’s explanations.

But MPs on the Transport select committee – a new one will be formed after the general election next week – could decide, if pressed by their constituents, to have an inquiry into BA’s power problems.

If so, they could question BA’s chief Alex Cruz or Willie Walsh, the chief executive of IAG.

In 1997 the committee held an inquiry into the escalating costs and problems on IT contracts at the Swanwick air traffic control centre in Hampshire. MPs decided to publish the contents of an independent report into the problems by technology consultancy Arthur D Little.

Any 2017 inquiry by the committee could hold BA to account in a way that would not otherwise be possible. Lessons from the failures may be useful to the public and private sectors.

UPS failure

Meanwhile what went wrong and why seems confused.

The Telegraph says the BA review is focusing on the uninterruptible power supply (UPS) to Boadicea House, one of two data centres close to Heathrow airport.

The UPS in question delivers power through the mains, diesel and batteries.

On Saturday morning, shortly after 8.30am, power to Boadicea House through its UPS was shut down. The reasons are unclear.

If power had returned to the servers in Boadicea House slowly this would have allowed the airline’s other Heathrow data centre, at Comet House, to take up some of the slack, said the Telegraph.

But, on Saturday morning, just minutes after the UPS went down, power was resumed in what one Telegraph source described as “uncontrolled fashion.”

This caused “catastrophic physical damage” to BA’s servers, which contain everything from customer and crew information to operational details and flight paths.

The Telegraph said that if power had been restored more gradually, BA would have been able to cope with the outage, and return services far more quickly than was the case.

The FT said yesterday that the UPS malfunctioned, cutting off the power supply. But it said that “some people working in the field have questioned” the explanation. They said it is very rare for UPS systems to fail. Even if they do, it should not affect the continued supply of mains electricity to the data centres they serve.

Not a technology problem?

BA has said there was an “immediate loss of power” from the UPS. When power returned, a surge physically damaged its IT servers. It had to replace the damaged equipment.

Willie Walsh said the meltdown was not a technology problem. The FT quoted him as saying, “You give me any IT system in the world and I’ll show you how good it is when it doesn’t have any electrical power going to it.”

Walsh insisted there was “no data loss, no data corruption”. He said the IT systems “functioned how they are supposed to function.”

But the FT quoted Jonathan Glover, co-founder of PSI, a company that helps businesses protect their equipment against sudden, unexpected power surges, who said the failure of a UPS “was relatively unlikely as they are robust and well-proven pieces of equipment”.

He added that, even if the UPS system did fail, it should not make a difference to the power supply to the airline’s IT system. The answers given don’t make a lot of sense, he said.

Alan Woodward, visiting professor at the department of computer science at the University of Surrey, agreed. He told the FT,

“It is like on your laptop and if you just pull the plug out of the back, it shouldn’t affect your laptop. It keeps running until the battery runs down. Even if you unplug the battery [of a laptop], it doesn’t like it from a data perspective, but plug it back in again, you don’t suddenly get a big power surge.”

Woodward said one possible explanation was that a voltage regulator contained within the UPS might have malfunctioned but when they fail the power usually stops, he added.

Another expert on UPS technology said that even if the system had failed, it would simply have been bypassed and normal electricity supply should have continued.

Why would the failure of the UPS affect BA’s back-up data centre?  The answer is unknown. BA would not comment on whether their two Heathrow-based data centres relied on the same UPS.

Ryanair on Tuesday pointed out that it had IT systems in three locations around Europe and if one went down, there were backups at each of its data centres. Ryanair’s data centres are not close to each other.

Two electricity companies whose low-voltage networks cover Heathrow airport and the surrounding area have denied there were any issues on their networks on Saturday morning.

Transient voltage surge arresters can shield against power surges from the local electricity network and malfunctions in a company’s own equipment but it is unclear whether BA had these fitted and if it did whether they worked.

The FT quoted an expert as saying that BA either had inadequate defences or  didn’t have the right level of industrial-level surge protection. BA has not commented on what protection measures it had.

Will BA publish its review?

BA may be reluctant to reveal the results of its review for various reasons. Parts of its IT appear in the UK could be run by non-BA staff. The failures could raise questions about the corporate oversight of any non BA specialists, possibly at board level.

It is also possible that an internal review could highlight fundamental managerial weaknesses – such as unclear or confused IT responsibilities in the UK or at IAG – after the outsourcing of IT skills to India last year.

Damian Brewer, an analyst at RBC Capital Markets, told the Telegraph that if BA’s early diagnosis of the cause of the crisis is correct, bosses’ failure to prepare for such an incident in the light of other carriers’ problems “suggests fundamental management and planning weakness”.

“It seems highly questionable why similar incidents with major US carriers in the last year have failed to see IAG move to ensure its airlines had plans in place to mitigate this risk, already seen elsewhere, and also to have contingency plans in place,” he said.

“At present, it appears that BA management have seemingly not taken account of IT risk precedent already seen and already known at other carriers.”

In what BA has said publicly about the IT problems, much of it has focused on what didn’t happen (a cyber attack) and on the people who were not responsible (Tata in India or energy companies). It told the BBC  the problems were “definitely not a consequence of underinvestment or cost-cutting.”

“All the parties involved around this particular event have not been involved with any type of outsourcing in any foreign country,” said Cruz.  “They have all been local issues around a local data centre who [sic] has been managed and fixed by local resources.”

Comment

Without an inquiry by the newly-formed Transport Committee, BA will find it easy to keep the lid on the results of its inquiry into the failures.  This would be a pity given the lessons that could be learned.

It’s ironic that the aviation industry has an exemplary reputation for reporting even minor problems that relate to safety. There is a duty to report even a ruffled carpet in an aircraft aisle that could trip up passengers or crew.

But there is no duty to account for an IT failure that disrupted the lives of 75,000 people across the world because it was not a safety issue. Provided the company pays satisfactory compensation, the fiasco will probably be out of the public eye in a few months.

But MPs, on behalf of their constituents,  could hold BA to account.

Anyone who wants to ask MPs to hold an inquiry into the BA failures could write to:

Transport Committee
House of Commons
London
SW1A 0AA

Telephone: 020 7219 3266
transcom@parliament.uk
Twitter: @CommonsTrans

The Committee’s clerk is Gordon Clarke: clarkeg@parliament.uk

Thank you to Dave Orr for his regular updates on the BA problems

BA’s IT: Will Transport Committee MPs ask the tough questions? – Government Computing

Full details of meltdown revealed (says Daily Telegraph)

BA board to demand IT chaos inquiry – Simon Jack, BBC

Aftermath of the cyber attack – will ministers learn the wrong lessons?

By Tony Collins

At least 16 NHS trusts out of 47 that were hit by the ransomware attack continue to face problems, according to BBC research.

And, as some patients continued to have their cancer treatments postponed, Tory, Labour and Lib-dem politicians told of their plans to spend more money on NHS IT.

But will any new money promised by government focus on basic weaknesses – such as the lack of interoperability and the structural complexities that made the health service vulnerable to cyber attack?

Last year when the health secretary Jeremy Hunt announced £4bn for NHS IT, his focus was on new technologies such as smartphone apps to order repeat prescriptions rather than any urgent need to upgrade MRI, CT and other medical devices that rely on Windows XP.

Similarly the government-commissioned Wachter review “Making IT Work: Harnessing the Power of HealthInformation Technology to Improve Care in England made no mention of Windows XP or any operating system – perhaps because ministers were much more likely to welcome a review of NHS IT that focused on innovation and new technologies.

Cancer treatments postponed

The Government’s position is that the NHS was not specifically targeted in the cyber attack and that the Tories are putting £2bn into cyber security over the next year.

Theresa May said yesterday,

“It was clear warnings were given to hospital trusts but this is not something that was focused on attacking the NHS. 150 countries are affected. Europol says there are 200,000 victims across the world. Cyber security is an issue we need to address.

“That’s why the government, when we came into government in 2010, put money into cyber security. It’s why we are putting £2bn into cyber security over the coming year.”

Similarly Jeremy Hunt, health secretary, told the BBC that the attack affected international sites that have “some of the most modern IT systems”.

But the BBC’s World at One gave an example of how the NHS’s IT problems were affecting the lives of patients.

It cited the case of Claire Hobday whose radiography appointment for breast cancer at Lincoln County Hospital was cancelled on Friday (12 May 2017) and she still doesn’t know when she’ll receive treatment. Hobday said,

“I turned up by hospital transport for my second radiotherapy session, and I, along with many other patients – at least 20 other people were waiting – and they said the computers weren’t working.

“I do have to say the staff were very good and very quickly let us all know that they were having trouble with the computers. They didn’t want to misinform us, so they were going to come and talk to us all individually and hoped they would be able to rectify it.

“Within half an hour or so they came out and said, ‘We’re really sorry but it’s not going to get sorted. We’ll send you all home and give you a call on Sunday’ which didn’t happen.

“But they did ring me this morning (15 May 2017) to say it’s not happening today and if transport turns up please don’t get in it, and it’s very unlikely it will happen tomorrow.

“It is just a bit upsetting that other authorities have managed to sort it but Lincolnshire don’t seem to have been able to do that.”

United Lincolnshire Hospitals Trust told World at One it will be back in touch with patients once the IT system is restored.

Roy Grimshaw was in the middle of an MRI scan – after dye was injected into his blood stream –  when the scan was stopped and he was asked to go back into the waiting room in his gown, with tubes attached to him, while staff investigated a computer problem. After half an hour he was told the NHS couldn’t continue the scan.

Budgets “not an issue”?

GP practices continue to be affected. Keiran Sharrock, GP and medical director of Lincolnshire local medical committee, said yesterday (15 Mat 2017) that systems were switched off in “many” practices.

“We still have no access to medical records of our patients. We are asking patients to only contact the surgery if they have an urgent or emergency problem that needs dealing with today. We have had to cancel routine follow-up appointments for chronic illnesses or long-term conditions.”

Martha Kearney – BBC World at One presenter –  asked Sharrock about NHS Digital’s claim that trusts were sent details of a security patch that would have protected against the latest ransomware attack.

“I don’t think in general practice we received that information or warning. It would have been useful to have had it,” replied Sharrock.

Kearney – What about claims that budget is an aspect of this?

Sharrock: “Within general practice that doesn’t seem to be the reason this happened. Most general practices have people who can work on their IT and if we’d been given the patch and told it needed to be installed, most practices would have done that straight away.”

GCHQ

World at One also spoke to Ciaran Martin, Director General for Government and Industry Cyber Security.  He is a member of the GCHQ board and its senior information risk owner.  He used to be Constitution Director at the Cabinet Office and was lead negotiator for the Prime Minister in the run-up to the Edinburgh Agreement in 2012 on a referendum on independence for Scotland.

Kearney: Did your organisation issue any warnings to the health service?

Martin: “We issue warnings and advice on how to upgrade defences constantly. It’s generally public on our website and it’s made very widely available for all organisations. We are a national organisation protecting all critical sectors and indeed individuals and smaller organisations as well.”

Huge sums spent on paying ransoms?

Kearney asked Martin, “How much money are you able to estimate is being spent on ransoms as a result of these cyber attacks?” She added,

“I did hear one astonishing claim that in the first quarter of 2016 more money was spent in the USA on responding to ransomware than [was involved] in armed robberies for the whole of that year?”

Martin: “First let me make clear that we don’t condone the payment of ransoms and we strongly advise bodies not to pay and indeed in this case the Department of Health and the NHS have been very clear that affected bodies are not to pay ransoms. Across the globe there is, sadly, a market in ransomware. It is often the private sector in shapes and sizes that is targeted.”

Martha Kearney said the UK may be a target because it has a reputation for being willing to pay ransoms.

Martin, “We are no more or less a target for ransomware than anywhere else. It’s a global business; and it is a business. It is all about return on investment for the attacker.

“What’s important about that is that it’s all about upgrading defences because you can make the return on investment lower by making it harder to get in.”

If an attacker gets in the aim must be to make it harder to get anything useful, in which case the “margin on investment goes down”. He added,

“That’s absolutely vital to addressing this problem.”

Are governments at fault?

Martin,

“Vulnerabilities will always exist in software. Regardless of who finds the underlying software defect, it’s incumbent on the entire cyber security ecosystem – individual users, enterprises, governments or whoever – to work together to mitigate the harm.”

He added that there are “all sorts of vulnerabilities out there” including with open source software.

Windows XP

Computer Weekly reports – convincingly – that the government did not cancel an IT support contract for XP.

Officials decided to end a volume pricing deal with Microsoft which left NHS organisations to continue with XP support if they chose to do so. This was clearly communicated to affected departments.

Government technology specialists, reports Computer Weekly, did not want a volume pricing deal with Microsoft to be  “comfort blanket” for organisations that – for their own local reasons – were avoiding an upgrade from XP.

Computer Weekly also reported that civil servants at the Government Digital Service expressed concerns about the lack of technical standards in the NHS to the then health minister George Freeman.

Freeman was a Department of Health minister until July 2016. In their meeting with Freeman, GDS officials  emphasised the need for a central body to set technical standards across the NHS, with the authority to ensure trusts and other organisations followed best practice, and with the transparency to highlight those who chose not to.

A source told Computer Weekly that Jeremy Hunt was also briefed on the security risks that a lack of IT standards would create in a heavily-federated NHS but it was not considered a priority at that top political level.

“Hunt never grasped the problem,” said the source.

There are doubts, though, that Hunt could have forced trusts to implement national IT security standards even if he’d wanted to. NHS trusts are largely autonomous and GDS has no authority to mandate technical standards. It can only advise.

How our trust avoided being hit

A comment by an NHS IT lead on Digital Health’s website gives an insight into how his trust avoided being hit by the latest cyber attack.  He said his trust had a “focus on perimeter security” and then worked back to the desktop.

“This is then followed up by lots of IG security pop ups and finally upgrading (painfully) windows XP to windows 7…” He added,

“NHS Digital have to take a lead on this and enforce standards for us locally to be able to use.”

He also suggests that NHS Digital sign a Microsoft Enrollment for Windows Azure [EWA] agreement as it is costly arranging such a deal locally.

 “NHS Digital must for me, step in and provide another MS EWA as I am sure the disruption and political fall-out will cost more. Introduce an NHS MS EWA, introduce standards for software suppliers to comply with latest OS and then use CQC to rate organisations that do not upgrade.”

Another comment on the Digital Health website says that even those organisations that could afford the deployment costs of moving from XP to Windows 7 were left with the “professional” version, which “Microsoft has mercilessly withdrawn core management features from (e.g. group policy features)”.

The comment said,

“There are a lot of mercenary enterprises taking advantage of the NHS’s inability to mandate and coordinate the required policies on suppliers which would at least give the under-funded and under-appreciated IT functions the ability to provide the service they so desperately want to.”

A third comment said that security and configuration management in the NHS is “pretty poor”. He added, “I don’t know why some hospitals continue to invest in home-brew email systems when there is a national solution ready and paid for.

“In this recent attack most the organisations hit seem to use local email systems.”

He also criticised NHS organisations that:

  • Do not properly segment their networks
  • Allow workstations to openly and freely connect to each other in a trusted zone.
  • Do not have a proper patch / update management regime
  • Do not firewall legacy systems
  • Don’t have basic ACLs [access control lists)

Three lessons?

  • Give GDS the ability to mandate no matter how many Sir Humphreys would be upset at every challenge to their authority. Government would work better if consensus and complacency at the top of the civil service were regarded as vices, while constructive, effective and forceful criticism was regarded as a virtue.
  • Give the NHS money to spend on the basic essentials rather than nice-to-haves such as a paperless NHS, trust-wide wi-fi, smartphone apps, telehealth and new websites. The essentials include interoperability – so that, at the least, all trusts can send test results and other medical information electronically to GPs –  and the upgrading of medical devices that rely on old operating systems.
  •  Plan for making the NHS less dependent on monolithic Microsoft support charges.

On the first day of the attacks, Microsoft released an updated patch for older Windows systems “given the potential impact to customers and their businesses”.

Patches are available for: Windows Server 2003 SP2 x64Windows Server 2003 SP2 x86, Windows XP SP2 x64Windows XP SP3 x86Windows XP Embedded SP3 x86Windows 8 x86, and Windows 8 x64.

Reuters reported last night that the share prices of cyber security companies “surged as investors bet on governments and corporations spending to upgrade their defences”.

Network company Cisco Systems also closed up (2.3%), perhaps because of a belief that it would benefit from more network spending driven by security needs.

Security company Avast said the countries worst affected by WannaCry – also known as Wannacypt – were Russia, Taiwan, Ukraine and India.

Comment

In a small room on the periphery of an IT conference on board a cruise ship , nearly all of the senior security people talked openly about how their board directors had paid ransoms to release their systems after denial of service attacks.

Some of the companies – most of them household names – had paid ransoms more than once.

Until then, I’d thought that some software suppliers tended to exaggerate IT security threats to help market their solutions and services.

But I was surprised at the high percentage of large companies in that small room that had paid ransoms. I no longer doubted that the threats – and the damage – were real and pervasive.

The discussions were not “off-the-record” but I didn’t report their comments at the time because that would doubtless have had job, and possibly even career ramifications, if I had quoted the security specialists by name.

Clearly ransomware is, as the GCHQ expert Kieran Martin put it, a global business but, as ransoms are paid secretly – there’s not a whisper in corporate annual accounts – the threat has not been taken seriously enough in some parts of the NHS.

The government’s main defence is that the NHS was not targeted specifically and that many private organisations were also affected.

But the NHS has responsibility for lives.

There may be a silver lining if a new government focuses NHS IT priorities on the basics – particularly the structural defects that make the health service an easy target for attackers.

What the NHS doesn’t need is a new set of politicians and senior civil servants who can’t help massaging their egos and trying to immortalise their legacy by announcing a patchwork of technological marvels that are fun to work on, and spend money on, but which gloss over the fact that much of the NHS is, with some notable exceptions, technologically backward.

Microsoft stockpiled patches – The Register

UK government, NHS and Windows XP support – what really happened – Computer Weekly

NHS letter on patches to counter cyber attack

Multiple sites hit by ransomware attack – Digital Health (31 comments)

Lessons from the WannaCrypt – Wannacry – cyber attack according to Microsoft

 

After a major IT failure, how did Barts NHS trust manage its image?

By Tony Collins

It sounded serious. Under the headline

“Cancer patients in limbo as five hospitals suffer ‘major’ IT crash”

the Daily Telegraph said,

“Hundreds of cancer patients have been denied treatment at one of England’s biggest hospital trusts due to a major IT failure that ground basic services to a halt.

“Doctors at five large London hospitals have reported 11 days of “chaos” after the systems used to prescribe chemotherapy doses and share x-ray and MRI images broke down on April 20.

“Barts Health NHS Trust said at least 136 operations had been cancelled due to the crash, as well as “hundreds” of cancer treatment sessions.

“The computer failure also means frantic staff have been unable to process blood tests for all but the most critical cases…

“A doctor at the Royal London Hospital told the Daily Telegraph: ‘We have been forced to leave sick patients on the ward while we go down 16 floors to catch a glance at an x-ray image, then come back and make treatment decisions based on a hazy recollection of it…

“An email sent by managers to staff last week said the crisis had forced cancer teams to rebuild patient records ‘from scratch’.

A medic at Whipps Cross hospital was quoted as saying that a lot of people were stuck in hospital needlessly which increased the likelihood of infection.

The trust runs Mile End Hospital, Newham University Hospital, The Royal London Hospital, St Bartholomew’s Hospital and Whipps Cross University Hospital as well as other NHS sites.

The Barts trust website says it delivers “high quality compassionate care to the 2.5 million people of east London and beyond”.

It has a turnover of £1.25 bn and a workforce of 15,000, making it the largest NHS trust in the country.

According to Health Service Journal, an internal email from Barts’ chief clinical information officer Tim Peachy said the IT failure was primarily a result of an “unexpected failure of a small number of physical disks on which data is stored”.

At one point the trust was manually processing blood test results and X-rays, and arranging for porters at its hospitals to hand deliver paperwork to clinicians.

Barts’ reputation 

In the light of the failure and disclosures in Health Service Journal, Barts confirmed the IT problems in statements to the media. It also contacted patients who were affected by the problems. A Barts statement this week said,

“A major computer equipment failure on Thursday 20 April resulted in a number of IT applications being unavailable to staff.

“‘Unfortunately, it has been necessary to cancel 136 operations, representing about 2.5% of our usual weekly in-patient activity. Several hundred chemotherapy appointments have been cancelled, however we have now recovered the chemotherapy prescribing database.

“Clinical teams have completed a patient-by-patient review to ensure that the appropriate course of action is taken for each of them, endeavouring to keep the disruption to an absolute minimum.

“We apologise to those affected and will reschedule their appointment for as soon as we are able.

“A number of applications have been affected to varying degrees. We have made significant progress in many areas including pathology (blood testing), with image viewing now also restored across the Trust. There are still some other areas where it will take time before we are on track again.”

It added,

“We continue to work urgently to maintain the operational resilience of our services, using tried and tested contingency plans to keep our patients safe.”

Despite the seriousness of the problems, the effect on patients and the uncertainties that media coverage might have created in the minds of those intending to go to Barts’ hospitals, the trust made no mention of the difficulties on its website – where it has a “latest news” section –  or on Twitter.

Barts uses Twitter for good news announcements, comments and congratulations, sometimes with dozens of daily tweets.

But why no mention of the IT problems?

On this point, a Barts spokeswoman said,

“We do not rely on social media to update patients. As a proportionately small number of people will be impacted on by the IT situation we are communicating directly to those affected including at outpatients clinics and via phone, letter as well as through communications with our healthcare partners including GPs.”

Comment

In its media statements Barts has been more open than some NHS organisations.

The usual NHS cycle after a major IT-related failure is a statement saying teething problems have been resolved, or are being resolved, followed by a succession of similar statements over the next few days, weeks or months when it becomes clear the problems haven’t been resolved.

This is what happened with e-Referral Service and Capita’s problems handling GP support services.

That hasn’t happened at Barts. But despite its openness with the media, it’s odd  the trust has published many congratulatory tweets in the past two weeks without a mention of any IT-related problems. They are not even alluded to.

It’s also odd that on its website the Barts “Latest News” section has no mention of the difficulties. But the website does have various good news announcements, including a reference to a positive Care Quality Commission report in April 2007.

Trusts do not have to account to patients, Parliament or anybody for IT-related problems. They are under no obligation to apologise to patients whose stays in hospital are unnecessarily prolonged, or whose appointments, operations and blood tests are cancelled or delayed because of IT-related difficulties.

Back-up systems? 

They also have no obligation to give the public any reason for the failure or explain why there was no back-up system that ensured patients were unaffected.

But amid so many positive announcements, statements and comments to the public on its website and on Twitter, should Barts have left out the other side of the story?

The NHS is an organisation that’s attuned to promulgating good news. It’s rare for a trust board paper and or a trust website to have anything but a good news feel to it.

But telling the public one side of the story does not encourage the public to believe officialdom when it says: “Trust us. We know what we’re talking about.”

Thank you to Zara Pradyer for letting me know about the Daily Telegraph article.

 

Will MPs’ report on Capita’s BBC contract make any difference?

By Tony Collins

At one level, Capita’s contract to handle most of the BBC’s TV licensing work is, in general, a success, at least according to statements made to the media.

Were it not for the National Audit Office and the Public Accounts Committee, a fuller story would not have emerged.

Today in The Guardian, a BBC spokesperson speaks of the Capita TV licensing contract in glowing terms. Through the contract, the BBC has reduced collection costs by 25% and increased revenue for programmes and services.

A Capita spokesperson spoke in similar terms. Capita has helped the BBC to collect more TV licence fee revenue every year since 2010-2011.

The only blip in the contract had seemed to be the heavy-handed tactics of some Capita staff. The Daily Mail reported in February 2017 that vulnerable people were hounded as some Capita staff tried to catch 28 TV licence evaders a week for bonuses of £15,000 a year.

This blip aside, has anything else gone wrong? There’s no hint of any technological problems on Capita’s website – or the BBC’s.

The BBC reported in 2011 that Capita will transform the TV licensing service, “using advances in technology and analytics to increase revenue and reduce costs”.

Capita’s website has a case study on its work for the BBC that refers to cost savings of £220m over the life of the contract, organisation-wide efficiencies and “protected brand image” among other benefits.

In December 2016, Capita described the “partnership” with the BBC  as a “success”.

The bigger picture

Capita processes TV licence payments, collects arrears and enforces licence fee collection. Its current contract with the BBC began in July 2012 and, after a recent renegotiation, ends in 2022 with the option to extend by up to a further five years.The BBC paid Capita £59 million in 2015–16.

The BBC has had a long-standing ambition to improve its main TV licensing databases so that they are structured by individual customers rather than households.

This was one of the hopes for the contract with Capita but it hasn’t happened. Capita had partly subcontracted work on the BBC’s legacy databases to CSC Computer Sciences.

Manual workarounds

The BBC, in its contract with Capita, aimed to upgrade ICT as part of a wider transition programme. The BBC paid Capita £22.9m for parts of the programme that were delivered, including restructuring contact centres, updating the TV Licensing website and upgrading handheld units for field staff.

The Public Accounts Committee says in today’s report,

“However, improvements with a contract value of £27.9m, primarily related to replacing legacy ICT systems, were not delivered by Capita and its subcontractor (CSC), and were not paid for by the BBC.

“As a result of the transition programme being only partly completed and subsequently stopped, the BBC and Capita currently have to do resource-intensive manual workarounds between inefficient ICT systems.

“Capita informed us that it was bearing the additional costs associated with undelivered elements of the transition programme. However, the BBC has had to allocate £9m to Capita to support the ongoing use of legacy systems, costs which the BBC told us were compensated for elsewhere in the renegotiated contract.

“It is unclear to us why ICT database improvements have proved so difficult over the last 15 years, particularly when competitors and other organisations can make similar changes.

“The BBC acknowledges that its current database is not fit for purpose for the future but does not yet have a clear plan to replace it.”

Comment

All outsourcing contracts have their strengths and failures – including early promises that don’t come to anything.

But it’s unlikely councils and other public sector organisations that are seriously considering outsourcing will take into account the past failures and broken promises of their potential suppliers.

If officials and councillors want to outsource IT and other services they probably will, whatever the record of their favoured potential suppliers.

They will see reports of the National Audit Office and Public Accounts Committee as biased towards negative disclosures.

Indeed the BBC and Capita, in their responses to today’s TV licensing report of the Public Accounts Committee, have drawn attention to the positive aspects of the report and not mentioned the technological failures.

Where does this leave councils and other organisations that are considering IT-related outsourcing and are seeking reference sites as part of the bid process?

Will those reference sites give only the positive aspects and not mention, or successfully deprecate, any media, PAC or NAO reports on contract failures?

Negative findings by the National Audit Office and Public Accounts Committee are usually important. Were it not for their scrutiny would not know how public money is being spent and misspent.

But their reports will have little or no effect as warnings to organisations that want to outsource.

Public Accounts Committee – BBC Licence Fee – 26 April 2017

 

A classic “waterfall” IT project disaster – yet officials went by the book

By Tony Collins

Some of those who read “Crash – 10 easy ways to avoid a computer disaster” may remember a warning that buying an IT system on the basis that it works well in another country and can therefore be adapted to the UK’s needs, is flirting with disaster.

First published in 1999, Crash said,

“There are graveyards of computer projects that began life as a simple adaptation of a package used elsewhere in the world.”

One example at that time was the failure of the London Stock Exchange’s Taurus project.

Now a report published today by Audit Scotland on the “i6” project goes into forensic – but lucid – detail on what went wrong and the conflicting views of police and the supplier Accenture.

Says the report,

“The belief that most of the i6 system could be based on an existing IT system proved incorrect.”

It became clear well into project that

“a virtually fully bespoke system was required”.

The plan was for i6 to replace 130 paper-based processes and IT systems but on 1 July 2016, after many well-publicised difficulties and delays, the Scottish Police Authority and Accenture agreed to terminate the i6 contract.

Police in Scotland had chosen Accenture’s bid in 2013 largely because it had successfully implemented a system for Spain’s Guardia Civil police service.

To its credit Accenture refunded all the money the police in Scotland had paid for the i6 system, £11.06m, plus a further £13.56m – but Audit Scotland says the failure of the project …

“means that some of the benefits that should have arisen from implementing it, have been, at best, delayed. There was a need to modernise police ICT systems six years ago when the procurement of i6 began. That need has not been met. Police officers and staff continue to struggle with out-of-date, inefficient and poorly integrated systems.

“This also hinders how Police Scotland interacts and shares information and intelligence with the other parts of the justice system. There is an urgent need to determine what the next steps should be…”

The lessons are clear from the report:

  • Don’t buy an overseas system without realising that it’ll need to be built almost from scratch for the UK. The ideal is for the business processes to be greatly simplified and adapted to fit a tried and tested system, not the other way around. Audit Scotland says the police programme team and Accenture believed that the majority of the i6 system could be based on an existing IT system that Accenture had developed for Spain,  with the remainder being bespoke development work.  But there was an “over-reliance” on Accenture’s work for Guardia Civil”.
  • The “waterfall” systems development contributed to the fact that Police Scotland “only discovered the true extent of problems with the system when it was delivered for testing”.  Waterfall meant that Accenture produced the software in distinct phases, in a sequence resembling a waterfall. Once a phase was complete, the process moved to the next phase – and no turning back. “It meant that all of the design, coding and construction of i6 would be completed before Accenture released it to Police Scotland for testing. Police Scotland would pay for each phase when it was completed.” [Agile, on the other hand, is a “test and see” approach and is far more flexible. It can adapted according to what the end-user needs and wants, and changes in those needs and wants.]
  • Don’t trust the demonstration of a waterfall system. The demo may look great but rolling it out successfully across various regions may be a different story. Accenture had demonstrated i6 but much later, after a period of testing, the i6 programme team reported to the programme board in August 2015 that there were: critical errors in the technical coding, flaws that Accenture was unable to resolve as quickly as expected, serious concerns about the criminal justice module, which did not comply with the Integrated Scottish Criminal Justice Information System data standards, errors in the search and audit modules and “problems around the limited functionality in the administration module”.
  • External assurance reports may tell you that you have complied with good practice and they may give you detailed praise for your attention to detail but they probably haven’t looked at the big question: will the systems ever work? Audit Scotland said external assurance reports such as the Scottish Government’s “Gateway reviews” suggested improvements but “raised no major concerns”.  Throughout the course of the i6 programme, most of the external reviews suggested that delivery confidence was either amber or green.
  • If the plan is for a waterfall development, doing everything by the book before a contract is awarded will not guarantee success, or even make it more likely, if you haven’t asked the big question: Is this ever likely to work given the complexities we don’t yet understand? For officials in Scotland, everything went smoothly before the award of contract: there were even 18 months of pre-contract discussions. But within weeks of the contract’s start, Police Scotland and Accenture disagreed about whether the proposed system would deliver the requirements set out in the contract. Soon there was a “breakdown in relationships and a loss of trust between Police Scotland and Accenture that never fully recovered,” said Audit Scotland.
  • The supplier may be just as optimistic as you. “As the design and development of i6 progressed, it became apparent that Accenture would need to develop significantly more than had been originally anticipated. Despite delays and serious problems throughout the lifetime of the programme, Accenture provided regular assurance, in the face of strong challenge, about their confidence in delivering the i6 system. This assurance proved misplaced.”
  • When planning a waterfall system that has complexities and inter-dependencies that are not fully understood at the outset, expect ever-lengthening delays and projected costs to soar. At one point Police Scotland estimated that the level of effort Accenture would require to complete i6 was around eight times greater than the resources Accenture had estimated when signing the original contract. “The i6 programme team believed that the functionality of Accenture’s solution did not meet the requirements it had agreed in the contract. Accenture maintained that Police Scotland had not specified a detailed description of business requirements. This issue had not emerged during months of pre-award dialogue. Accenture also believed that it had set out clearly what its solution would do and maintained that Police Scotland, as part of procurement process, had accepted its qualified solution. A dispute followed about the interpretation of the contract requirements. Police Scotland argued that, after months of competitive dialogue, the requirements of the i6 system were well-defined, and that in line with the contract, these took precedence. Accenture argued its solution had precedence and that Police Scotland was trying to extend the scope of the programme. Accenture stated that, to meet Police Scotland’s interpretation of requirements, it would require more time and money.”
  • As soon as things start going badly awry, stop and have a re-think. Cancel all existing work if necessary rather than plough on simply because failure isn’t an option. Above all, take politics out of the equation. The Scottish Police Authority was anxious about i6 being seen to be a success after the failure of a previous police ICT project in 2012 – the Common Performance Management Platform. At the same time the i6 programme was “extremely important to Accenture at a global level. “This may have led to misplaced optimism about the prospects of success and unwillingness to consider terminating the programme,” says Audit Scotland.
  • When things start to go wrong, the truth is unlikely to emerge publicly. Even those accountable for the project may be kept in the dark. “Police Scotland were cautious of commercial sensitivities when providing assurances on i6 publicly. The Scottish Parliament’s Justice Sub-Committee on Policing held a number of evidence sessions with the Scottish Police Authority and Police Scotland to explore progress with the i6 programme. In March 2014, the Sub-Committee expressed frustration at the lack of information about the problems with the i6 programme that had been ongoing since August 2013. Police Scotland did not disclose the severity of the issues facing the programme, nor was it overly critical of Accenture. This may have reflected a desire to maintain relationships with Accenture to keep the programme on track or to maintain the commercial confidentiality of the contract.”

Accenture’s response

Accenture said,

“As the report acknowledges, the scope and the complexity of the solution for i6 increased significantly during the project.  This was driven by the client.  There were challenges and issues on both sides, but we worked closely with Police Scotland to review the programme and recommend revised plans to successfully deliver i6.

Despite our best efforts, it was not possible to agree the necessary changes and we mutually agreed to end the project.”

In May 2017 Audit Scotland is due to publish a report that summarises the lessons from a number of public sector ICT projects it has investigated.

Some of what i6 was intended to cover …

Comment

Tis a pity officials in Scotland hadn’t read Crash before they embarked on the i6 project – or if they had, taken more notice of the dangers of assuming a system that works overseas can be tweaked to work in the UK.

We commend Audit Scotland for its expert investigation and a fine report.

Clearly the failure of i6 is not entirely Accenture’s fault.  The project was commissioned on the basis of assumption and when things went wrong politics intervened to prevent a complete stop and a fundamental re-think.

Fatally, perhaps, there appears to have been no discussion about simplifying police administration to make the IT more straightforward. If police administration is so enshrined in law that it cannot be simplified, officials would have to accept before awarding the contract that they were buying an entirely new system.

The UK armed services simplified volumes of rules and practices before it introduced pay and personnel administration systems. It was hard, inglorious work. But simplifying ways of working first can make the difference between IT success and failure.

i6 – a review. Audit Scotland’s report. 

Waterfall approach damns £46m Scottish police system – Government Computing

Another public sector IT disaster – but useful if the lessons are learned.

What Google looks for when hiring staff … traits Whitehall’s culture abhors?

By Tony Collins

The contrast between what Google looks for when hiring staff and what Whitehall looks for when making some of its top appointments, could give clues as to why many government IT-based projects and programmes fail.

First, the strengths Google looks for.  These were set out yesterday on BBC R4 by Laszlo Bock,  human resources chief at Google for 10 years.

Google was named “Best Company to Work For” more than 30 times around the world and received over 100 awards as a top employer during Bock’s time.

In 2010, he was named “Human Resources Executive of the Year”. Under him, Google changed its clunky, arduous recruitment processes that relied on gimmicks like maths puzzles to those that helped the company grow to about 60,000 employees in less than two decades.

In 2015 he  published his first book, The New York Times bestseller Work Rules!, a practical guide to help people find meaning in work and improve the way they live and lead. He resigned from Google in 2016.

On the BBC  “Analysis” programme on Monday evening – which looked at intelligence and talent and what they mean, if anything, in job interviews –  Bock said the least important attribute Google screens for is whether someone knows about the job they are taking on. Crunching the data on successful hiring led Google instead to look for these characteristics:

  • Humility
  • Conscientiousness
  • A sense of responsibility not to quit until the job is done well
  • Comfort with ambiguity
  • A sense of fun
  • Courage

Why courage?

Bock said,

“It’s about the importance of people being able to raise their voices in organisations. One of the things that happens is, when organisations get large, people stop raising their voices and really bad things happen as a result. That’s where you get whistleblowing, insider trading, all kinds of things.

“Human beings are evolved, biologically, as social, hierarchy-seeking animals. We tend to conform. So courage is important because the really innovative, creative stuff comes from ‘I got this crazy idea’ and the bad problems get flagged by people who are willing to raise their hand and say ‘I don’t think this is a good thing to do’.

“Without that you can’t do great things.”

Comment

It’s too easy to generalise about the hiring and appointment of senior civil servants. But it’s possible to understand a little about the hiring culture within Whitehall’s biggest department, the Department for Work and Pensions.

An insight into DWP culture and thinking can be gleaned from the many Lever arch folders of documents filed by the DWP as part of an FOI case in which it spent several years fighting to stop the release of documents about the Universal Credit IT programme.

The documents include DWP witness statements on the “harm” that would be caused if the IT documents in question were published.

The judge in the case, Chris Ryan, challenged most of the DWP’s arguments.

In one of his rulings, Judge Ryan described the DWP’s claims as:

  • alarming and surprising
  • overstated
  • unconvincing
  • close to fanciful

He said that public confidence in the Universal Credit IT programme had been maintained for some time “on a false basis”; and he raised the possibility that an “unhealthily collegiate relationship had developed” between the DWP and private sector IT suppliers. [Campaign4Change will publish a separate blog post on this ruling in the next few days.]

As well as the insight into DWP culture that one can gain from the FOI case, it’s possible to gauge culture and thinking within Whitehall departments from the talented, free-thinking IT individualists who have joined the top layer of the civil service, quit and returned to the private sector.

It would be invidious to pick out some names as there are so many.

What all this suggests is that Whitehall’s culture appreciates conformity and consensus and shuns boat-rocking.

When top IT professionals who joined HMRC and the DWP spoke publicly at conferences about institutional problems that needed to be tackled, mandarins reacted quickly – and such disclosures were never repeated.

And after a leak to the Guardian about the results of a DWP staff survey of morale on the Universal Credit IT programme, the department launched a formal leak inquiry headed by a senior member of the security services.

At the same time, Universal Credit IT programme documents were no longer emailed but transferred around in taxis.

This bout of nervous introspection (the judge described the DWP’s arguments in the FOI case as “defensive”) when taken together with what else we know, indicate that Whitehall’s culture is insular, distrustful and inimical to open challenge and problem-solving (though there are some within the senior Whitehall ranks who successfully defy that culture).

When Bock talks of conformity being a danger within large organisations he would not have had the DWP in mind – but he aptly describes its culture.

When he speaks about the “importance of people being able to raise their voices in organisations” he was probably unaware of the extent to which Whitehall culture abhors raised voices.

As Bock says, when people don’t raise their voices “really bad things happen as a result”. Perhaps the lack of internal challenge was one reason the NHS IT programme – NPfIT – lost billions of pounds, and the DWP’s Universal Credit programme went badly awry for several years.

When Bock says the “really innovative, creative stuff comes from ‘I got this crazy idea’, he could have been describing the culture of the Government Digital Service. But that refreshing GDS culture is being slowly choked by the conservatism of traditional Whitehall departments.

As Bock says, “the bad problems get flagged by people who are willing to raise their hand and say ‘I don’t think this is a good thing to do’.”  But bad problems are things senior civil servants avoid talking about, even internally. A Disneyland”good news” culture pervades central departments.

A National Audit Office report on the Universal Credit programme referred to a “fortress mentality” within the DWP.

Maybe the consensus-seeking John Manzoni, head of the civil service, and his colleague Sir Jeremy Heywood, Cabinet Secretary, could seek to employ Bock as an adviser on appointments and recruitment.

Bock’s brief? To turn around the senior civil service’s culture of conformity, groupthink, denial, selective use of “good news” facts and a lack of open challenge.

Recognising the destructiveness within a big organisation of having the wrong culture – as Bock does – could be the start of a genuine Whitehall transformation.

BBC R4 “Analysis” on talent, intelligence and recruitment

Laszlo Bock steps down

Southwest One – a positive postscript

By Tony Collins

somerset county council2IBM-led Southwest One has had a mostly bad press since it was set up in 2007. But the story has a positive postscript.

Officials at Somerset County Council now understand what has long been obvious to ICT professionals: that the bulk of an organisation’s savings come from changing the way people work – and less from the ICT itself.

Now that Somerset County Council has the job of running its own IT again – its IT-based relationship with Southwest One ended prematurely in December 2016 – the council’s officials have realised that technology is not an end in itself but an “enabler” of headcount reductions and improvements in productivity.

A 2017 paper by the county council’s “Programme Management Office”  says the council has begun a “technology and people programme” to “contribute to savings via headcount reduction by improving organisational productivity and process efficiency using technology as the key enabler”.

Outsourcing IT a “bad mistake” 

It was in 2007 that Somerset County Council and IBM launched a joint venture, Southwest One. The new company took over the IT staff and some services from the council.

In the nine years since then the council has concluded that outsourcing ICT – thereby separating it from the council’s general operations – was not a good idea.

The same message – that IT is too integral and important to an organisation  to be outsourced – has also reached Whitehall’s biggest department, the Department for Work and Pensions.

Yesterday (8 February 2017) Lord Freud,  who was the Conservative minister in charge of Universal Credit at the Department for Work and Pensions, told MPs that outsourcing IT across government had proved to be a “bad idea”.  He said,

“What I didn’t know, and I don’t think anyone knew, was how bad a mistake it had been for all of government to have sent out their IT…

“You went to these big firms to build your IT. I think that was a most fundamental mistake, right across government  and probably across government in the western world …

” We talk about IT as something separate but it isn’t. It is part of your operating system. It’s a tool within a much better system. If you get rid of it, and lose control of it, you don’t know how to build these systems.

” So we had an IT department but it was actually an IT commissioning department. It didn’t know how to do the IT.

“What we actually discovered through the (Universal Credit) process was that you had to bring the IT back on board. The department has been rebuilding itself in order to do that. That is a massive job.”

Task facing Somerset officials

Somerset County Council says in its paper that the council now suffers from what it describes as:

  • Duplicated effort
  • Inefficient business processes
  • A reliance on traditional ways of working (paper-based and meeting-focused).
  • Technology that is not sufficient to meet business needs
  • Inadequate data extraction that does not support evidence based decision making.
  • “Significant under-investment in IT”.

To help tackle these problems the council says it needs a shift in culture. This would enable the workforce to change the way it works.  

From January 2017 to 2021, the council plans “organisation and people-led transformational change focused on opportunities arising from targeted systems review outcomes”.

The council’s officers hope this will lead to

  • Less unproductive time in travelling and  attending some statutory duties such as court proceedings.
  • Fewer meetings.
  • Reduced management time because of fewer people to manage e.g. supervision, appraisal, performance and sickness.
  • Reduced infrastructure spend because fewer people will mean cuts in building and office costs, and IT equipment. Also less training would be required.
  • Reduction in business support process and roles.
  • Reduction in hard copy file storage and retention.

 The council has discovered that it could, for instance, with changes in working practices supported by the right technology,  conduct the same number of social services assessments with fewer front- line social workers or increase the level of assessments with the same number of staff.

Southwest One continues to provide outsourced services to Avon and Somerset Police. The contract expires next year.

Comment

Somerset County Council is taking a bold, almost private sector approach to IT.

Its paper on “technology and people” says in essence that the council cannot  save much money by IT change alone.

Genuine savings are to be found in changing ways of working and thus reducing headcount. This will require very close working – and agreement – between IT and the business end-users within the council.

It is an innovative approach for a council.

The downside is that there are major financial risks, such as a big upfront spend with Microsoft that may or may not more than pay for itself.

Does outsourcing IT ever make sense?

Somerset County Council is not an international organisation like BP where outsourcing and standardising IT across many countries can make sense.

The wider implication of Somerset’s experience – and the experience of the Department for Work and Pensions – is that outsourcing IT in the public sector is rarely a good idea.

Thank you to Dave Orr, who worked for Somerset County Council as an IT analyst and who has, since the Southwest One contract was signed in 2007, campaigned for more openness over the implications of the deal.

He has been more effective than any Somerset councillor in holding to account the county council, Taunton Deane Borough Council and Avon and Somerset Police, over the Southwest One deal.  He alerted Campaign4Change to Somerset’s “Technology and People Programme” Somerset paper.

One of Orr’s recent discoveries is that the council’s IT assets at the start of the Southwest One contract were worth about £8m and at hand-back in December 2016 were worth just £0.32m, despite various technology refreshes.

Somerset County Council’s “Technology and People Programme” paper

Whitehall’s outsourcing IT a “bad mistake” – and other Universal Credit lessons, by a former DWP minister

is London Ambulance Service’s back-up system “public endurance”?

By Tony Collins

In November 2016 London Ambulance Service had its busiest week for seriously ill and injured incidents in the history of the Service.

“The Service is …expecting demand to increase even further throughout December,” said London Ambulance Service at the time.

A few weeks later, on one of the busiest nights of the year, the systems went down, from 12.30am to 5.15am on 1 January 2017. The result was that 999 calls were logged  by pen and paper.

When systems are working normally  an incoming 999 call displays the address registered to that number – if the address is registered.  The London Ambulance operator confirms the location, assesses the severity and an ambulance can be despatched within seconds, with the address on its screen and a satnav pointing the way, according to a comment on The Register.

Pen and paper takes longer because the address and other details need to be given over a radio, which can take minutes.

But pen and paper is the London Ambulance Service’s back-up for IT failures.  Whether it can cope with unprecedented demand – or with a major incident in London – is in doubt.

A former London Ambulance Service paramedic told the BBC there had been waits of an hour for ambulances on 1 January 2017. He said call handlers had been “amazingly helpful”, but it was “easy to become overwhelmed especially in the midst of high call volumes”.

London Ambulance Service declined to answer any questions on its latest system failure.

Malcolm Alexander of the Patients’ Forum for the London Ambulance Service said: “We want to know why it is that this system that cost so much money and is supposed to be so effective is not fail-safe.”

He added: “If this system fails at a time when there is huge pressure in the system, for example if there was a major disaster or a terrorist attack, we are going to be in trouble. We really need to make sure it doesn’t collapse again.”

1992

A report into the collapse of London Ambulance Service systems found that they had had failed for many reasons. The Service had taken a “high-risk” IT approach and did not test systems thoroughly before putting them into service.

(Some may question how much has been learned since then.)

2006

In 2006 the London Ambulance Service systems crashed nine times in a fortnight. Each time staff reverted to pen and paper.

2008

In 2008, when systems failed,  repairs took 12 hours. Again the Service reverted to pen and paper.

2011

In June 2011 an IT upgrade caused the system to go down for about three and half hours. Pen and paper was again the back-up “system”. At the time the London Ambulance Service was upgrading the Commandpoint system, supplied by Northrop Grunman, which the Service deployed in 2010 and still uses.

2013

In 2013 on Christmas Day and Boxing Day the systems went down for separate reasons for several hours each day, with staff reverting to pen and paper.

2015

The Chief Inspector of Hospitals, Mike Richards, recommended that the London Ambulance Service be placed into special measures.

He said at the time,

“The Trust has been performing poorly on response times since March 2014. This is a very serious problem, which the trust clearly isn’t able to address alone, and which needs action to put right.”

Comment

It’s becoming the norm for parts of the public sector to regard the public as captive customers when it comes to going live with new IT or upgraded software.

Rather than test new systems, procedures and upgrades thoroughly before introducing them, some parts of the public sectors are going live with a “let’s see what happens and fix things then” approach.

This has become the semi-official approach to the introduction of Universal Credit – with long delays in payments for some claimants.

Within the NHS, at some hospitals introducing new patient record systems, there has been an internal acceptance that patients may suffer from delays,  perhaps with tragic consequences, at least for three year-old Samuel Starr.

The NHS e-referral service was launched with nine pages of known problems.  And when NHS England launched a streamlined GP support service with Capita, officials knew of the possible problems. But it launched anyway.

After the London Ambulance Service’s IT failure on New Year’s Day, it’s clear that many emergency workers did their best to give a normal 999 service. St John’s Ambulance helped.

But to what extent does senior management at the London Ambulance Service have a “stuff happens” mindset when IT goes seriously wrong?

There’s no individual accountability and no commercial imperative to learn lessons from any of the failures.

And there’s no fervent business or political will to ensure the same or similar mistakes don’t recur.

Every time systems fail, the London Ambulance Service promises an investigation. But where are the results published so that lessons can be learned?

Pen and paper is tried and tested. But demands on the London Ambulance Service are much greater than in the past.

With an unprecedented demand for its services how is it London Ambulance Service’s senior management can comfortably rely on pen and paper as its back-up system?

It can – if nobody in power requires an earnest answer to the question.

Another wider question is whether it’s acceptable to use the public as guinea pigs for new or upgraded IT, with potentially serious or even tragic consequences.

London Ambulance Service suffers New Year’s crash – Computer Weekly

London Ambulance Service hit by new year fault – BBC online

 

 

Days from taking back outsourced IT, Somerset Council is unsure what it’ll find

By Tony Collins

Facing the TV cameras, officials at Somerset County Council spoke with confidence about the new joint venture company they had set up with the “world-class” IT supplier IBM.

“The contract has to succeed; we will make it succeed, ” a senior official said at the time. Greater choice for residents, more control, sustained improvement of services, improved efficiency, tens of millions in savings and enhanced job prospects for staff.

These were some of the promises in 2007.

Since then, Somerset County Council has been through a costly legal dispute with IBM; projected savings have become losses, and Somerset is days away from taking back the service early.

Now the council faces new IT-related risks to its reputation and finances, warns a team of auditors.

In several audit reports on the exit arrangements, auditors warn of a series of uncertainties about:

  •  what exactly IT assets the council will own as of 1 December 2016, when the joint venture hands back IT and staff.
  • how much software may not be licensed, therefore being used illicitly.
  • how much software is being paid for without being needed or used, wasting council tax money.
  • whether thousands of pieces of hardware have been disposed of securely over the years of the contract, or whether confidential data could later turn up in the public domain.
  • the accuracy of some supplied information. “… the same networking hardware items have the same value associated with them even though one is twelve years old and the other only four” said auditors.

Comment

That Somerset County Council laments setting up the Southwest One joint venture with IBM is not new. What continues to surprise is the extent of the difficulties of ending the joint venture cleanly – despite months, indeed more than a year – of preparatory work.

The realty is that uncertainties and risks abound.

When IT journalists ask leading councillors and officers at the start of outsourcing/joint venture deals whether all the most potentially serious risks have been given proper consideration, the spokespeople inevitably sound supremely confident.

If things go wrong, they are sure the council will be able to take back the service under secure arrangements that have been properly planned and written into the contract.

Yet today some of the most potentially serious risks to Somerset’s finances and reputation come from continuing threats such as the possibility confidential data being found on old hardware not securely disposed of.

Or the council may be paying for unneeded software licences.

In short Somerset County Council is taking back the IT service on 1 December 2016 without being certain what it will find.

In future, therefore, when councillors and officials across the country talk with supreme confidence at the start of an outsourcing deal or joint venture about large savings, sustained efficiencies, and a step-change improvement in services that comes with the benefits of collaborating with a world-class private-sector partner, local residents will have every right to be deeply sceptical.

For the reality is more likely to be that the council and its world-class supplier are about to embark on a journey into the unknown.

Thank you to campaigner Dave Orr for alerting me to the council audit reports that made this post possible.

TV broadcast in 2007 days after the council and IBM signed the Southwest One joint venture deal.

**

Excerpts from reports due to be considered by Somerset County Council’s Audit Committee next week (29 November 2016):

“… laptops, servers, storage devices, networking equipment, etc.) have been disposed of without the correct documentation historically, throughout the term with SWO [Southwest One]. There is a high likelihood that without the documentation to show that SWO were meant to have previously disposed of any specific data baring assets in a compliant manner then subsequent fines and loss of reputation will need to be dealt with by the Council.

“This is being addressed as part of the exit works but initial investigations show an expected lack of documentation.

**

“The quality of asset management and therefore exposure to risk (over and above this inherited risk) is expected to improve significantly once asset management returns to SCC [Somerset County Council).”

**

“Asset locations have been updated and improved though there are still issues regarding all asset details not being recorded accurately in the Asset Register. There is a risk that if wrong details are recorded against an asset then incorrect decisions could be made regarding these assets which may in turn cause the Council financial loss and/or loss of reputation.”

**

“… the same networking hardware items have the same value associated with them even though one is twelve years old and the other only four.”

**

“Software assets are now included in the monthly asset register report though the information collected and lack of correlation to meaningful license information means the original risk is not fully mitigated.

“This continued lack of software asset usage information against licensing proof of entitlement as well as the obvious risk of illegally using non licensed software there is also a risk that the Council is wasting public funds and Council officer’s time to manage unnecessary software. This means the Council will not be able to show “Best Value” in these purchases which could lead to fines being imposed by Central Government and loss of reputation by the inefficiencies being reported in the media.”

**

“I cannot though see evidence of the warranty & support arrangements being recorded or accurate recording of end of life assets. Due to a lack of or incorrect detail on the asset information there is the risk of incorrect decisions being made regarding an asset’s usage which could then lead to loss of money or reputation for the Council.”