Category Archives: supplier relationships

Somerset Council publishes “lessons learnt” from IBM contract

By Tony Collins

It’s rare for any council to publish the lessons learned from its outsourcing/joint venture contract, but Somerset County Council has set an example.

The council has produced its report to “inform future commissioning”. The council held a workshop to help identify the right lessons.

Written by Kevin Nacey, the council’s Director of Finance and Performance, and published by the Audit Committee, the document is diplomatically worded because the South West One joint venture contract with IBM continues; it was renegotiated in 2013 when the council took back some services and about 100 staff that had been seconded to South West One.

IBM is the majority shareholder in the joint venture company, and is its main funder. The minority shareholders comprise the county council, Taunton Deane Borough Council and Avon and Somerset Police. The joint venture company still provides ICT, finance and human resources/payroll.

Dave Orr, a former council IT employee and campaigner for openness, who spotted Nacey’s report, says it misses some important lessons, which are in Orr’s From Hubris to High Court (almost) – the story of Southwest One.

Lessons

From Somerset County Council’s Audit Committee report:

Contract too long and complicated

“One of the most significant lessons learnt is not to make contracts overly complicated. Both the provider and the Council would agree that the contract is incredibly complicated. A contract with over 3,000 pages was drawn up back in 2007 which was considered necessary at the time given the range of services and the partnership and contractual arrangements created.”

Expectations not met

“The partnership between the provider and the three clients has at times been adversarial and at times worked well. What has become clear over time is that any such partnership depends upon having similar incentives and an understanding of each partner’s requirements.

“Requirements change and the nature of local government changed considerably as a result of the national austerity programme. The well-documented financial difficulties faced by the provider early into the contract life also affected its ability to meet client expectations. The net effect is that at times the provider and partner aims in service delivery do not always match and discord and dissatisfaction can occur.”

Client team to monitor supplier “too small”

“The Client function monitoring a major contract needs to be adequately resourced. At the outset the size of the client unit was deemed commensurate with the tasks ahead, such as monitoring a range of performance measures and reporting on such to various management and Member forums.

“Liaison between partners, approving service development plans and approval of payments under the contract were other significant roles performed by the client unit. However, as performance issues became evident and legal and other contractual disputes escalated, the team had to cope with increasing workloads and increasing pressure from service managers and Council Members to address these issues.

“This is a difficult balancing act. You do not want to assemble a large client function that in part duplicates the management of the services being provided nor overstaff to the extent that there is insufficient work if contract performance is such that no issues are created.

“With hindsight, the initial team was too small to manage the contract when SAP and other performance issues were not resolved quickly enough. Sizing the function is tricky but we do now have an extremely knowledgeable and experienced client team.”

Some contract clauses “too onerous”

“Performance indicators need to be meaningful rather than simply what can be measured. Agreement between the provider and the SCC client of all the appropriate performance measures was a long and difficult exercise at the beginning of the contract. Early on in the first year of the contract, there were a large number of meetings held to agree how to record performance and what steps would be necessary should performance slip below targets.

“Internal audit advice was taken (and has been at least twice since under further reviews) on the quality and value of the performance indicator regime. It is regrettable and again with hindsight a learning point that too much attention was paid to these contractual mechanisms rather than ensuring the relationship between provider and SCC was positive. Perhaps the regime was too onerous for both sides to administer.”

Too ambitious

“Contract periods need to be different for different services as the pace of change is different. The range of services provided under the initial few years of the contract were quite extensive. On another related point the provider also had to manage different services for different clients.

“This level of complexity was perhaps too ambitious for all parties. Although there are many successful parts to the contract, it is inevitable that most will remember those that did not work so well.

“The contract period of 10 years is a long time for 9 different services to change at the same pace. Of course, service development plans were agreed for each service to attempt to keep pace with service needs as they changed. The secondment model introduced as part of the contract arrangements had been used elsewhere in the country before this contract used it.

“Nevertheless, it was the first time that 3 separate organisations had seconded staff into one provider. In many ways the model worked as staff felt both loyalty to their “home” employer, keeping the public service ethos we all felt to be important, and to Southwest One as they merged staff into a centre of excellence model.”

Hampered by terms of staff contracts

“The disadvantage was that Southwest One was hampered by the terms and conditions staff kept as they tried to find savings for their business model and to provide savings to the Council in recent years given the changing financial conditions we now operate under.

“Another aspect of this contract in terms of complexity is the nature of the partnering arrangement. It is not easy for all partners to have exactly the same view or stance on an issue. Southwest One had to manage competing priorities from its clients and the partners also had varying opinions on the level of performance provided.”

In summary

“This was a very ambitious venture. The service provided in some cases got off to an unfortunate start with the issues generated by SAP problems and relationships were strained and attracted much inside and outside attention.

“All parties have been working very hard to keep good relationships and to fix service issues as they arise. The sheer size and complexity of this contract has proven difficult to manage and future commissioning decisions will bear this in mind.

“Over the years officers running services that receive support from Southwest One have been surveyed regularly on how they feel the contract has been progressing. Despite all of the issues and lessons learnt outlined in this report, it is worth pointing out that many of the customer satisfaction and performance levels under the contract have been met by Southwest One.”

Comment

Well done to Somerset County Council’s Audit Committee and the Council’s auditors Grant Thornton for asking for the report. The South West One contract has been a costly embarrassment for IBM and the council. It has also had mixed results for Taunton Deane and the local police though officials in these two organisations seem locked into a “good news” culture and cannot admit it.

Perhaps the best thing to emerge from the IBM-led joint venture is this “lessons learnt” report. Without it, what would be the point of the millions lost and the damage to council services?

From contracts that don’t work out as expected, councils and central government departments rarely produce a “lessons” report , because nobody requires them to. Why should they bother, especially when it may be hard to get an internal consensus on what the lessons are, and especially when a report may mean admitting that mistakes were made when the contract was drawn up and signed.

Public sector organisations will sometimes do anything to avoid admitting they’ve made mistakes. Somerset County Council and its Audit Committee have shown they are different. Maybe the rest of the public sector will start to follow their example.

Dave Orr’s well-informed analysis of the lessons from Southwest One.

Officials black out IT security report after it’s published in full

By Tony Collins

In one of the most bizarre regressions since the FOI Act came into force in 2005, officials at Somerset County Council have redacted an audit report on SAP security weaknesses after the report was published in full.

The result is that anyone can see links to both reports. This is the report with parts of it redacted – blacked out. These are links to the full versions, which were published before the redactions – here and here.

The report was written by auditors Grant Thornton for Somerset County Council and highlights weaknesses in a database that is shared by the council, Taunton Deane Borough Council and Avon and Somerset Police.  The database is part of a SAP system run by Southwest One on behalf of the three authorities.

Southwest One is an IBM-led enterprise that provides IT and other services to the three authorities under a controversial outsourcing contract. Dave Orr has written comprehensively about the deal.

Somerset published the Grant Thornton report in full. The media including Campaign4Change published some details of the IT security weaknesses mentioned in the Grant Thornton report. It appears that Avon and Somerset Police asked officials at Somerset to black out details of some of the weaknesses.

Somerset-based FOI campaigner Dave Orr says the blacking out is to save the blushes of the police.

Says Orr: “Much of the redaction in the Somerset County Council IT Controls report by Grant Thornton, especially generic and available password advice in Section 3, is not based in a genuine security threat, but looks to be rooted in a Police culture that seeks to avoid criticism and/or embarrassment.”

Somerset MP Ian Liddell-Grainger says:

“SAP was built on the cheap by IBM to serve three different customers – the County Council, Taunton Deane district council and the Police. It would have made sense to bung in a few partitions to stop council eyes taking a peek at police matters, or vice versa. But that would have cost money – perish the thought.”   

 Police SAP systems’s “significant” security weaknesses. 

When “life and death” NHS IT goes down

By Tony Collins

Almost unnoticed outside the NHS an email was circulated by health officials last weekend about a national “severity 1” incident involving the Electronic Prescription Service, running on BT’s data Spine .

“The EPS [electronic prescriptions service] database is currently experiencing severe degradation of performance. … BT engineers [are] currently investigating with the database application support team,” said the email.

A severity 1 or 2 incident, which involves a temporary loss of, or disruption to, the Spine or other national NHS system,  is not unusual, according to a succession of emails forwarded to Campaign4Change.

The Department of Health defines a severity 1 incident as a  failure that has the potential to:

— have a significant adverse impact on the provision of the service to a large number of users; or

— have a significant adverse impact on the delivery of patient care to a large number of patients; or

— cause significant financial loss and/or disruption to NHS Connecting for Health [now the Health and Social Care Information Centre], or the NHS; or

— result in any material loss or corruption of health data, or in the provision of incorrect data to an end user.

The Health and Social Care Information Centre, which manages BT’s Spine and other former NPfIT contracts, reports that the spine availability is 99.9% or 100%. But the HSCIC’s emails tell a story of service outage or disruption that is almost routine.

If the spine and other national services  are really available 99.9% of the time, is that good enough for the NHS, especially when ministers and officials are increasingly expecting clinicians and nurses to depend on electronic patient records and electronic prescriptions?  In short, are national NHS IT systems up to the job?

NHS staff access the spine tens of millions of times every month, often to trace patients before accessing their electronic records.  The spine is pivotal to the use of patient records held on Rio and Cerner Millennium systems in London. It is critical to the operation of Choose and Book, the Summary Care Record, Electronic Prescription Service pharmacy systems, GP2GP, iPM/Lorenzo, and the Personal Demographics Service.

According to a Department of Health letter sent to the Public Accounts Committee, payments to BT for the Spine totalled £1.08bn by March 2013.

BT says on its website that its 10-year NHS Spine contract involves developing systems and software to support more than 899,000 registered NHS users. The HSCIC says the Spine is used and supported 24 hours a day, 365 days a day.

“There is a huge amount of industrial-strength robustness, availability, disaster recovery, that you cannot get someplace else,” said a BT executive when he appeared before MPs in May 2011.

Life and death  

Sir David Nicholson spoke of the importance of the spine and other national NHS systems at a hearing of the Public Accounts Committee in 2011. He said they were

“providing services that literally mean life and death to patients today … So the Spine, and all those things, provides really, really important services for our patients…”

When Croydon Health Services NHS Trust went live with a Cerner Millennium patient records system at the end of September a “significant network downtime” – of BT’s N3 network – had an effect on patients.

A trust board paper, dated 25 November 2013 says:

“CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT …”

Below are some of the emails passed to Campaign4Change in the past four months. Written by the Health and Social Care Information, the emails alert NHS users to outages or disruption to GP or national NHS IT systems.

Some HSCIC messages of disruption to service

October 2013

Severity 2
HSCIC
National
CQRS has not received a number of participation status messages.
Also affecting: GPES
USER IMPACT:
CQRS Users are not able to manually submit specific information, this will impact the users’ business process for entry of achievement data.
ACTION BEING TAKEN:
Following a configuration change by the GPES Business Unit a specific code has now been added to the GPET-Q Database. We are currently awaiting confirmation that the addition of the relevant code has been successful. Discussions are taking place regarding the re-submission of status messages. HSCIC conference calls are on-going.

[A severity 2 service failure is a failure [that] has the potential to:

– have a significant adverse impact on the provision of the service to a small or moderate number of service users; or

– have a moderate adverse impact on the delivery of patient care to a significant number of service users; or

– have a significant adverse impact on the delivery of patient care to a small or moderate number of patients; or

– have a moderate adverse impact on the delivery of patient care to a high number of patients; or

– cause a financial loss and/or disruption … which is more than trivial but less severe than the significant financial loss described in the definition of a Severity 1 service failure.]

**

Severity 2
HSCIC
BT Spine
National
Intermittent performance issues on TSPINE.
T-Spine
RESOLUTION:
BT Spine have confirmed that the incident has been resolved and users are able to perform routine business processes without delays.

November 2013

Severity 1
BT Spine
HSCIC
National
Users are unable to log into PDS.
USER IMPACT:
All sites are currently unable to access PDS, this is causing a delay to normal services.
ACTION BEING TAKEN:
BT Spine are working to restore service.

**

Severity 2
BT Spine
HSCIC
National EPS users.
Slow performance on reliable and unreliable messages for EPS.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
BT investigating.

**

Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
USER IMPACT:
This is causing delays to routine business processes as some users may be experiencing slow performance with the EPS service.
ACTION BEING TAKEN:
BT moved the database to an alternate node following application server restarts. This temporarily restored normal message response times however performance has started to degrade again. BT Investigation continues.

**
Severity 1
Atos
HSCIC
National
Multiple users were unable to log in to the Choose & Book application.
ATOS made some network configuration changes overnight 19th/20th November which restored service. After a period of monitoring throughout the day yesterday the service has remained stable and at expected levels. Further activities and investigation will be carried out by several resolver teams which will be scheduled through change management.

**
Severity 2
BT Spine
HSCIC
National
Slow performance on EPS Messaging.
No further issues of slow response times with EPS messaging have occurred today. BT Spine to continue root cause investigation.

**

Severity 2
Cegedim RX
HSCIC
National
Cegedim RX – Users are experiencing slow performance in EPS 1 and EPS 2.
USER IMPACT:
Users are experiencing slow performance and delays to routine business processes when using EPS 1 and EPS 2.
ACTION BEING TAKEN:
Following a restart of application services, traffic has improved for all new EPS messages. However there is a backlog of EPS messages which may cause delays to routine business processes. Cegedim RX to continue to investigate.

**

December 2012

Severity 1
BT Spine
HSCIC
National
Performance issues have been detected with the transaction messaging system (TMS).
Also affecting: Choose and Book, GP2GP
USER IMPACT:
This may cause delays to routine business processes. This may have an effect on all Spine related systems. This includes PDS, Choose and Book, PSIS, SCR, ACF Services.
ACTION BEING TAKEN:
This has been resolved but BT are currently monitoring performance. Further investigation is required by BT into the root cause.

**

Severity 2
GDIT – CQRS
HSCIC
National
DTS has not processed a CQRS payment file.
CQRS
Also affecting: GPES
USER IMPACT:
This is causing delays to routine business processes.
ACTION BEING TAKEN:
GDIT are currently developing a fix which will be rolled out tomorrow evening, pending successful testing.

January 2014

Severity 1
BT Spine
HSCIC
National
TMS reliable messaging unavailable.
USER IMPACT:
TMS reliable messaging unavailable and users having to implement manual workarounds.
ACTION BEING TAKEN:
Issues experienced due to a planned change overrunning, BT Spine continue to implement the transition activity in order to restore service.

**

Severity 2
BT Spine
HSCIC
National
Users have experienced intermittent issues with the creation and cancellation of smartcards in CMS [Card Management Service for managing smartcards].
CMS
USER IMPACT:
This is intermittently causing delays to routine business processes as some users have been unable to create, cancel, cut or print cards in CMS.
ACTION BEING TAKEN
Users may experience issues with the creation and cancellation of cards in CMS. BT have identified a fix for the issue which is currently undergoing testing prior to deployment into the live environment.

**

Severity 2
BT Spine
HSCIC
National
The maternity browser was unavailable within NN4B.
RESOLUTION:
BT identified a problematic server which was recycled to restore system functionality.

**

Spine scheduled outage for essential maintenance activity.

During critical work to migrate to a new storage solution on Spine an issue was experienced on the Transaction Messaging Service (TMS) in September of this year. The issue resulted in BT failing over the TMS database from its usual site on Live B to Live A to restore service. The failover was completed well within the Service Level Agreement and no detrimental long term impacts to the service were incurred.

On the 15th January 2014 between approximately 22:00-23:30, HSCIC, in conjunction with BT, are planning to relocate the TMS database back to Live B, this is for several critical reasons:

  1. The issues experienced, which prompted the failover, are fully resolved and will not be experienced again as the storage migration work is now complete.
  2. The Spine service is designed to operate with all databases running on Live B so this work supports the optimum configuration for the service.
  3. Most critically the transition for all data on Spine to Spine2 has been designed to operate from a standby site with no live databases on it. Therefore to support the Spine2 transition this work is absolutely essential.

In order to facilitate a safe relocation of the database a 1.5 hour outage is required to TMS. The impact of this to Spine is significant and results in effectively an outage for Spine and its interfaces to connecting systems for that period. The time and date is aimed at the lowest times of utilisation for Spine, to minimise impact to end users, as well as not impacting critical batch processing and Choose & Book slot polls.

 

Date & Time

Change Start Change Finish Services Affected Outage Duration
15/01/2013 22:00 15/01/2013 23:30 Transaction Messaging Service (TMS) 1.5 hours
Service  Impact Description
Choose and Book The Choose and Book service will be available but functionality will be limited until the TMS database has switched over.Users of the web application will experience limited retrievals during the outage window.The system will not be able to create shared-secret for patients who have not been referred via Choose and Book before.Service Providers will be unable to:

  • Perform clinic re-structures and re-arrange appointments for patients for directly bookable services
  • Send DNA messages to Choose and Book.

For directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Choose & Book systems will need to queue the messages and resend to Spine once the TMS service is enabled.

Due to the timing of the outage slot polls will not be affected.

Summary Care Record application (SCRa) The SCRa application will be available but functionality will be limited until the TMS database has switched over. Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates)
DSA The DSA application will be available but functionality will be limited until the TMS database has switched over.Simple traces can be completed on PDS data but users will be unable to perform any PSIS updates (e.g. GP summary updates).
Electronic Prescription Service (EPS)Pharmacy Systems Reliable messaging will be unavailable for the duration of the switchover work as the TMS service will be suspended dual site. All messages received from EPS systems will be rejected and not go into retry.EPS systems will need to queue the messages and resend to Spine once the TMS service is enabled.
EPS Batch The PPA response for any “claim” messages will not be sent to PPA/PPD. However, EPS will send those response(s) again when the retry jobs are re-activated after the switchover exercise is over. Response for any “claim” messages will not be received until after the switchover. Retry jobs will resend the responses once the TMS service is enabled.
Existing Service Providers (ESPs) There will be varying impacts depending on the product, release version and Spine compliant modules of the solution.ESP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP2GP GP2GP will be unavailable until the TMS database has switched over.GP2GP systems will need to queue the messages and resend to Spine once the TMS service is enabled.
GP Extraction Service (GPES) GPES functionality will be unavailable until the TMS database has switched over.Messages will be queued on Spine and processed once the TMS service is restored.
GP Systems Functionality for Choose & Book, EPS and GP2GP, SCR will be limited until the TMS database has switched over.For Choose & Book directly bookable services the following functionality will be unavailable:

  • Booking appointments
  • Rearranging appointments
  • Creating new patient accounts

Systems will need to queue the messages and resend to Spine once the TMS service is enabled.

iPM/Lorenzo The real-time connection to Spine will be unavailable during the TMS outage. However both systems can be disconnected from Spine and operate without synchronised PDS data.iPM/Lorenzo will need to queue the messages and resend to Spine once the TMS service is enabled.
Millennium An outage of PDS reliable messaging will impact Millennium users.Users will be unable to:

  • trace patients
  • register new patients on PDS
  • book or reschedule appointments

Millennium will need to queue the messages and resend to Spine once the TMS service is enabled.

NN4B Trusts will need to be aware that during the outage NHS numbers cannot be generated, new-births cannot be registered and blood-spot labels cannot be generated and should plan accordingly.All birth notifications will be queued and processed once the TMS service is enabled.
Personal Demographics Service (PDS) Simple traces can be completed on PDS data.PDS reliable messaging will be unavailable until the TMS database has switched over.
RiO Users will be unable to:

  • trace patients
  • register new patients
  • book or reschedule appointments

The RiO system will need to queue the messages and resend to Spine once the TMS service is enabled.

TMS Event Service (TES) The majority of TES functionality will be unavailable during the outage.Trusts will need to be aware EPS, Death notifications, and Patient Care Provision Notifications (change of pharmacy) will be queued and sent to the receiving systems once the TMS service is restored.Any impacted notifications will be queued and sent to the receiving systems once TMS is restored.
TMS Batch (DBS, CHRIS, ONS) DBS will be unavailable until the TMS database has switched over (DBS processing will be suspended for the duration of the exercise).As the TMS switchover will be scheduled to start at 22:00, CHRIS batch should complete before the outage starts (CHRIS batch runs at 20:00 nightly).ONS processing will start at 18:00 nightly. If it doesn’t complete before 22:00, the messages will be queued and processed once the TMS service is restored.

**

Severity 2
BT Spine
HSCIC
National
Users are unable to grant worklist items within UIM.
USER IMPACT:
This is causing delays to routine business processes as users are unable to complete their worklist items within the UIM application.
ACTION BEING TAKEN:
BT investigating.

**

Severity 1
BT Spine
HSCIC
National
The EPS database is currently experiencing severe degradation of performance.
USER IMPACT:
Delays to routine business processes.
ACTION BEING TAKEN:
BT engineers currently investigating with the database application support team.

Comment

David Nicholson is right. The NHS has become dependent on systems such as the Spine. But can doctors ever trust any aspect of the safety of patients to systems that are not available 24×7 as they need to be in a national health service?

It appears that BT and other suppliers have not been in breach of service level agreements, and the HSCIC has a good relationship with the companies.  But does the HSCIC have too great an interest in not finding fault with its suppliers or the contracts, for finding fault  could draw attention to any defects in a service for which the HSCIC is responsible?

Have national NHS IT suppliers a strong enough commercial or reputational interest  in avoiding  a disruption or loss of service, so long as they keep within their service level agreements? 

If nobody sees anything wrong with the reliability of existing national NHS IT services improvements are unlikely. Diane Vaughan’s book on the culture and organisation of NASA shows that experts in a big organisation can do everything right according to the rules  and procedures – and still have a disastrous outcome.

BBC World at One’s focus on Government IT

By Tony Collins

The lead item on BBC R4’s World at One on Friday was about Government IT contracts.

On the programme were the government’s Chief Procurement Officer Bill Crothers, Cabinet Office minister Francis Maude, the chairman of the Public Accounts Committee Margaret Hodge, the UK IT Association, and me.

Some of the points made:

–  Bill Crothers gave an example of what he called “abuse” by some big IT suppliers. He said a young man who works for him lost his power cable. The supplier quoted £65 for a replacement. The price should have been £5 or £6.  When Crothers queried it, the supplier justified its price on grounds of security. Crothers could not believe that a power lead had security implications so he questioned the price again and received several pages of explanation from the supplier, which he did not read. Eventually the supplier “was good enough to reduce the price to £37”.

– HMRC was charged £30,000 for changing some text on its website.

– Francis Maude said a DWP team and a further 12 people from the Cabinet Office’s Government Digital Service had built – in only three months – a prototype of a digital solution to support the introduction of Universal Credit. The system cost just over £1m, he said. [Separately big IT suppliers at DWP have been paid £303m up to March 2013 for Universal Credit work.] Maude declined to predict the outcome of the “twin-track” work on the UC project.

– Some big legacy systems may soon need replacing – those that pay about £60bn a year in state pensions and collect nearly £100bn a year in VAT. “Those are going to be big projects,” said Margaret Hodge. “I don’t think we have seen the end of big projects, or the end of disasters.”

World at One in detail

Presenter Shaun Ley and BBC political correspondent Ross Hawkins focused on government IT because of an announcement by the Cabinet Office that it is drawing the line on “bloated and wasteful IT contracts”. The Cabinet Office was pitching its announcement as marking a “massive change,” said Hawkins.

Ley said Francis Maude announced the safeguards  in an attempt to ensure that IT contracts don’t become multi-billion pound failures. He said that the abandoned NPfIT had cost close to £10bn.

Hawkins quoted the UK IT Association as saying that  government did not know how to do deals with smaller suppliers. On the government’s relationship with big suppliers UKITA said the government was like a “battered wife or husband who doesn’t seem to know how to leave.”

Appalling

Hawkins said Crothers has the air of a man going to war. Crothers’ conclusion on the way things are at the moment:

“This is about the oligopoly, the cluster of big suppliers that have had it took good for too long. It’s reflective of monopolistic or oligopolistic behaviour.  It is not acting as if they are in hungry and in a competitive market.  That’s appalling.”

Universal Credit

Hawkins asked Francis Maude how confident he was that what was being put in place on Universal Credit would work.

“I hope it will work,” said Maude. “The digital solution was created by a team within DWP with a dozen or so GDS [Government Digital Service] staff assisting.

“They created a working prototype for a digital solution within 3 months at a cost of only a bit over £1m. That certainly can be basis of a successful long-term solution.”

Hawkins [to Maude] “I asked you whether you were confident the approach with DWP would work and you said you hoped it would. That suggests to me that maybe you are not (confident).”

Maude: “N0-one knows with these things. Anyone who says you are certain everything is going to succeed … the way we do things now is build something quickly, test it, prove it, test it with users, and so you can’t have certainty about any of these outcomes.”

Outsourcing failures

Hawkins said “We have had story after embarrassing story about outsourcing failures [such as the] government being charged for tagging dead people … now ministers  have an interest in coming out on the front foot and just for once being on the attack and having a whack at the IT companies.

“You don’t need to be a political genius to work out why they would like to do that rather than be endlessly explaining themselves after embarrassing stories in the papers.”

Ley (to me): “Is this the best way to deal with the problems government has experienced? The journalist Tony Collins has written widely  about project failures in IT in both the public and private sectors.”

I replied that big companies have sometimes charged a lot to make small software changes.  The Cabinet Office’s “red lines” were a good idea though they were a formalising of restrictions that had been in place some time.

The Cabinet Office doesn’t have the power to make changes happen because departments are accountable to Parliament for their spend and so don’t want much interference from the Cabinet Office. But the Cabinet Office is right to try and reduce the amounts spent on big projects.

Ley: “What will be the effect of breaking up contracts?”

I said I hoped the Cabinet Office’s restrictions would bring about a change in culture in departments against the assumption that big is beautiful. Big projects should be split into components which would give SMEs a greater involvement and could reduce the risks of projects failing.

More project disasters?

Hodge gave her reaction to the Cabinet Office’s restrictions in the context of the Universal Credit project.

“Francis Maude and Cabinet Office have been trying really hard to get some sense into the way that project has developed. But sadly the news we have had lately suggests to me that they have failed. It is about £400m so far on IT.

“What went wrong there was that the department [DWP] thought it [UC] was a big IT project instead of thinking:  we are going to be changing our business; we are going to get 6 benefits rolled into one. They [the DWP] have not written off that money [£303m] which is what my committee thinks they should have done, because they want to save face. Down the line I think we’ll see some disasters there.

“There are a lot of projects around  government, what are called legacy projects, where old systems need to be replaced . They are big projects – pensions in DWP where £60bn is given out a year;  VAT receipts  in HMRC where nearly £100bn is collected. Those are going to be big projects. I don’t think we have seen the end of big projects, or the end of disasters.”

Ley: “What about breaking them up into smaller projects? Won’t that reduce potential risks?”

Hodge: “The important thing is what Tony Collins was saying to you. What we find is that the skills don’t exist within departments, either to commission the IT properly or to manage the suppliers once they have the IT in place.

“We are about to examine the army recruitment contract – I think that is what we’ll find.  The MoD hasn’t got the skills to manage it.

Ley: “Do you welcome the ending of automatic contract extensions?”

“I warmly welcome that. This is a small step in the right direction. Having an expert as we have in Bill Crothers in the Cabinet Office is really important. What we haven’t got are skills in the departments. It is not like a business. If it was, Bill Crothers would probably run IT across the whole of government. Our departments run in silos. They haven’t got the skills. They have this demand for big, big programmes in the future and I don’t think we have seen, sadly, the end of IT disasters.”

Update

Thank you to Dave Orr for drawing my attention to an excellent piece on the World at One item by procurement expert Peter Smith who concludes:

“… There is a big issue – large suppliers have not covered themselves in glory, but small suppliers just can’t develop huge systems for DWP or MOD.

“The large suppliers must have a role, but we have to manage these contracts better. And the answer can’t just be a small hit squad in Cabinet Office. This needs real capability development across government, which we haven’t really seen as yet in a coordinated fashion.”

BBC World at One – Government IT contracts

Bill Crothers on BBC Radio 4 – suppliers get another good kicking

Big 4 Universal Credit IT suppliers punished?

By Tony Collins

The  latest draft business case for Universal Credit suggests existing IT suppliers will have little to do with the “end-state digital solution” that is  due eventually to support the roll-out of UC.

The Department for Work and Pensions will use a mixture of its own and external people for the end-state digital solution.

Computer Weekly quotes part of the draft business case as saying:

“To extend the current IT solution we will be using a standard waterfall delivery approach largely using existing suppliers and commercial frameworks, in order to de-risk delivery and ensure UC continues to have a safe and secure introduction.

“The end-state digital solution will be delivered using an agile, and therefore iterative, approach as advocated by the Cabinet Office with significantly less reliance on the large IT suppliers delivering the current UC IT service.”

Politicalscrapbook.net picks up Computer Weekly’s report and says that Iain Duncan Smith “punishes Universal Credit IT suppliers“. 

Costs

Computer Weekly quotes the draft business case as putting the cost of the end-state solution at £106m – comprising external IT costs of £69m and in-house “Design and Build” team costs of £37m.

The total cost of UC IT is now put at £535m – down substantially on the £673m estimate in the DWP’s December 2012 UC business case.

UC project at “red” 

Yesterday the Guardian reported that Francis Maude and his team at the government digital service have objected to the twin-track approach to UC but were outflanked by “a majority” of other government ministers and project advisers, leaked minutes say.

The twin-track approach to UC IT means that the DWP and its main suppliers – HP, Accenture, IBM and BT – continue to develop existing systems (a blend of legacy and new technology) while a separate team develops a new “end-state” system for use by the end of 2017. It’s unclear how the two systems will differ. 

Computer Weekly quotes the latest draft business case as saying it is “unclear what the digital service will deliver and to what timescales”.

Due to the multitude of problems facing universal credit, the project has been coded “red” overall, according to the Guardian.

Comment

Computer Weekly has done well to gain sight of the latest draft business case for UC.

Whoever wrote the draft appears to accept the Cabinet Office’s case for departments to “move away from large ICT projects” and thus “reduce waste, provide a more flexible approach to complex business requirements that are likely to change over time and reduce the risk of project failures”. (National Audit Office, Universal Credit: early progress). 

But is the DWP simply telling the Cabinet Office what it wants to hear?  All the signs are that the big money at the DWP will continue to go to its main IT suppliers. 

The £106m agile “end-state digital solution” is a bonus system which may or may not materialise.  It is in essence a big, agile research project and the DWP is having trouble finding IT professionals to work on it.

If ever it’s a success it could start to replace existing UC IT in 2017 or beyond. But that may never happen. The DWP has already spent more than £300m on existing UC technology and is set to spend a lot more: around £90m. The DWP is unlikely to scrap it.

So HP, IBM, Accenture and BT are all but guaranteed a large income stream from the non end-state UC technology.

Even without the UC project the big 4 are guaranteed a large income from the DWP’s other work which includes:

– Personal Independence Implementation – 2.8bn 2011–2016
– Fraud and error programme – £770m  2012–2015
– Child maintenance group change 1.2bn 2009–2014
– Pensions reform Enabling Retirement Savings programme 1.04bn 2007–2018
– State Pension reform – single tier £114m 2012–2017
– Specialist Disability Employment programme – £203m 2012–2014

The big 4 will also continue to receive a large chunk of the DWP’s IT budget for maintaining and upgrading the existing software, hardware and networks.

Business cases are written by experts in the writing of Whitehall business cases.  Their main purpose is to provide a case for the Treasury to release funds for a project. They give current thinking on costs and benefits. The documents are revised when these change significantly.

So the statement in the UC draft business case that the new end-state digital solution will rely “significantly less” on existing UC IT suppliers means little: it is subject to change.

And the words “significantly less” are  unexplained. They may have no scientific basis. 

Worrying

The big 4 suppliers continue to be all-important to the DWP – and are so enmeshed that they decide at times how much they should be paid, suggests the NAO.

From its latest report on the UC project, the NAO comments on the DWP’s lack of control of suppliers :

– “In February 2013, the Major Projects Authority reported there was no evidence of the Department actively managing its supplier contracts and recommended that the Department needed to urgently get a grip of its supplier management.”

– “[The DWP has] limited IT capability and ‘intelligent client’ function leading to a risk of supplier self-review.”

– “[The DWP has] inadequate controls over what would be supplied, when and at what cost because deliverables were not always defined before contracts were signed.”

– “[The DWP has an] over-reliance on performance information that was provided by suppliers without Department validation.”

– ” … the Department did not enforce all the key terms and conditions of its standard contract management framework, inhibiting its ability to hold suppliers to account.”

So it would be naively optimistic to suppose that if the big 4 were to be frozen out of the end-state solution for UC that it would make much difference to their income from the DWP.      

UC in chaos or not?

A generous interpretation of all the available evidence on the UC project so far is that the DWP is working through, and understanding, the difficulties on an immensely complicated IT-enabled project.

And supporters of the twin-track approach could argue that two completely independent sets of teams are working in parallel and in discreet competition to produce the most successful system. One team comprises the big 4 using waterfall and the other a largely in-house team using agile.  Eventually one system will prevail, even if it’s 2020 or beyond that it handles securely online all types of claims. On completion the system will simplify benefit claims and cut the costs of administration.

A less generous interpretation of the available facts is that the UC IT project  is in chaos and that vast sums continue to be poured into a poorly formed strategy that nobody in government will concede is failing;  all parties are preoccupied with resolving problems as they arise and expecting irrationally that things will come good in the end.  Nobody should expect the full truth to emerge from those who have a deep interest in the project’s success including IDS and his permanent secretary Robert Devereux.

Howard Shiplee, head of the UC project, may still be getting his head around how chaotic things are. The highly capable David Pitchford, who headed UC  for a few months before he quit the civil service last year, came close to saying the project was in chaos. His Major Projects Authority said in February 2013 that the DWP needed to “rethink the delivery approach”, said the NAO.

Indeed the UC project shows many of the usual signs of a government IT-based project failure:

– major changes in the basic assumptions between the business case of December 2012 and the latest draft business case
– excessive secrecy (keeping secret a succession of internal and external reports on the project).
– defensiveness (continued DWP claims that problems are historic)
– a high turnover of leaders
– a culture of good news that “limited open discussion and stifled challenge”, said the NAO
– a lack of control of suppliers (NAO)
– repeated delays
– suppliers that get paid regardless of whether their systems are contributing to a  successful project.

To me things look chaotic but I hope I’m wrong. I’d like UC IT to work. IDS and Shiplee will probably know the whole truth – and they are still in post, to date.  If Shiplee leaves the project before the general election that could be an indication of how bad things really are.   

Is £40m write-off on a big software project normal?

By Tony Collins

On BBC R4’s “Week in Westminster” on Saturday morning (14/12/13)  guest presenter Isabel Hardman of The Spectator spoke to Conservative MP Richard Bacon and me about big government projects that go wrong.

Hardman mentioned that Bacon has co-written a book  on government failures Conundrum: Why every government gets things wrong and what we can do about it.

Referring to write-offs so far of about £40m on Universal Credit, Hardman asked me whether it was normal for such a write-off on a big project.

I said it wasn’t. The work and pensions secretary Iain Duncan Smith has said it was. When questioned by MPs of the work and pensions committee on 9 December, IDS implied that it was not unusual to write-off a third on large software-based projects. He suggested that research by Forrester supported this view.

Software coding for Universal Credit has cost about £120m so far (excluding hardware, infrastructure, consultancy or other IT-related costs). So IDS suggested that a write off of £40m was only about a third of the software coding costs.

But I haven’t seen any evidence that suggests write-offs of a third of the software costs on a big project are typical.   

I replied to Hardman that although there has been much trial and error on Universal Credit IT, £40m is a lot to write off.

[Trial and error included an attempt, from 2011 onwards, to adopt an agile approach but the National Audit Office said the DWP “experienced problems incorporating the agile approach into existing contracts, governance and assurance structures”. The NAO added that the Cabinet Office “did not consider that the Department (DWP) had at any point prior to the reset [Feb-May 2013]  appropriately adopted an agile approach to managing the Universal Credit programme”. The DWP has now introduced what it calls Agile 2.0, a hybrid approach incorporating elements of  agile with waterfall, though agile purists say it is impossible to combine the two.]

I told Hardman that the write-offs were largely because the DWP was unclear at the outset what the software was supposed to do.”With big IT projects it’s a bit like designing a bridge and you know where one side begins but you’re not sure where the other side ends. They have been learning as they go along and that’s probably why there have been large write-offs,” I said.

Hardman asked Richard Bacon whether it was normal to set out on these big projects without knowing where the bridge was going. Bacon agreed, citing the NPfIT which had led to large write-offs on failed work for England-wide electronic patient records. He said it was not at all abnormal for ministers to set off on big projects without knowing where they were going.  

The good news?

I told Hardman that IDS was at least well informed. He now has the NAO scrutinising the project as well as his own external consultants and the independently-minded Howard Shiplee as head of the project.

But I didn’t think UC would be complete until 2020 at the earliest given that the last big computerisation of benefit systems, Operational Strategy, took about 10 years to complete. Hardman said: “That would be a humiliation for IDS surely?”

I replied that IDS may not even be in politics in 2017. I also said that UC will probably not bring the financial benefits predicted, to judge from the last big computerisation of benefits.  But UC has wide support. Perhaps, I said, it has to work … eventually. 

BBC R4 Week in Westminster – 14/12/13

Patient records go-live “success” – or a new NPfIT failure?

By Tony Collins

John Goulston says the go-live of a new patient records system at his trust is a “success”.

He should know. He’s Chief Executive of Croydon Health Services NHS Trust. He’s also chair of the trust’s Informatics Programme Board which has taken charge of bringing Cerner Millennium to Croydon’s community health services and the local University Hospital, formerly the Mayday.

He was formerly Programme Director of the London Programme for IT at NHS London – a branch of the NPfIT.

In a report two weeks ago Goulston said the trust deployed the “largest number of clinical applications in a single implementation in the NHS”. Croydon went live with Cerner Millennium on 30 September and 1 October 2013.

Said Goulston in his report:

“Administrative functions do not engage clinicians; providing them with a suite of clinical functionality has been justified as each weekday approx. 1,000 staff are logged on and using the system. CHS [Croydon Health Services] has in Phase 1 deployed, in addition to patient administration, the largest number of clinical applications in a single implementation in the NHS England.”

BT helped install Millennium at Croydon under the National Programme for IT.  The trust’s spokesman says the Department of Health provided central funding, and the trust paid for implementation “overheads”.  The Health and Social Care Information Centre was the trust’s partner for the go-live.

The Centre is the successor for Connecting for Health. It has taken on CfH’s officials who continue to help run the NPfIT contracts with BT and  CSC.

Goulston said that Cerner and BT have paid tribute to the trust which installed Millennium in A&E, outpatients, secretarial support and cancer services, and elsewhere.

“Our partners Cerner, BT and Ideal have commented that the Trust has undertaken one of the most efficient roll-outs of the system they have worked on, with more users adopting the system more quickly and efficiently than other trusts … the success we have achieved to date is the result of the efforts of every single system user and all staff members,” said Goulston.

Best Cerner implementation yet?

Optimistic remarks about their launch of Cerner Millennium were also made in 2012 by executives at the Royal Berkshire NHS Foundation Trust.  Their optimism proved ill-judged.

Of the Millennium go-live at Royal Berkshire, trust executives said that it “had been considered to be the best implementation of Cerner Millennium yet and that despite staff misgivings, the project was progressing well”.   This positive message should be disseminated, they said.

Months later they told the Reading Chronicle of patient safety issues and a financial crisis arising from the Millennium implementation.

A Royal Berkshire governors Rebecca Corre was quoted as saying: “There is a patient safety issue when staff write down observations and then there is an hour before they can get it onto the computer. If it is an experienced nurse, they may pick up a problem, but others may not.”

Ed Donald, Chief Executive of Royal Berkshire was quoted as saying:

“Unfortunately, implementing the EPR [electronic patient record] system has at times been a difficult process and we acknowledge that we did not fully appreciate the challenges and resources required in a number of areas.”

Are executives and managers at Croydon Health Services NHS Trust  now similarly afflicted with an unjustified optimism about the success of their Cerner go-live?  

Past consequences of NPfIT go-lives hidden?

The Department of Health has claimed benefits for the NPfIT of £3.7bn to March 2012 but there have been trust-wide failures: thousands of patients have had their appointments, care or treatment delayed by difficulties arising from past implementations of patient record systems under the NPfIT.  For thousands of patients waiting time standards have been exceeded or “breached” because of disruption arising from troubled go-lives.

In nearly every case trusts made it difficult for the facts to come out publicly. Vague or unexplained fragments of information about the consequences of the NPfIT implementation appeared  in different board papers over several months. The facts only emerged after a journalistic investigation that required scrutiny of many board papers and follow-up questions to the trust’s press office.

So Campaign4Change investigated Croydon Health’s implementation of Cerner Millennium to see if the Francis report’s call for a “duty of candour” over mistakes and problems in the NHS have made any difference to the traditional fragmentation of facts after NPfIT go-lives of patient record systems.

The Francis report called for “openness, transparency and candour“.  Trusts were told not to hide sub-standard practices under the carpet. The health secretary Jeremy Hunt said it can be “disastrous” when bad news does not emerge quickly and the public are kept in the dark about poor care.

To my questions about the Cerner Millennium implementation Croydon trust’s spokesman always responded promptly and tried to be helpful. But it appears that trust executives have given him limited information about consequences of the go-live, and have preferred to indulge the “good news” NHS culture that Jeremy Hunt warned about.

On being asked what problems the trust has faced since the go-live the spokesman gave various answers that made no mention of the problems.

“All of our staff received training on the system, and we are continuing to offer our teams support as it is embedded.”

What of the problems arising from the implementation, and has the board been fully informed?

“Millennium has featured regularly on the Corporate Risk Register presented to each Part 1 Board meeting.   In addition, implementation has received detailed confidential consideration at Part 2 of Board meetings, (which is why you won’t find it in our public board papers).”

Given Francis’s call for duty of candour,  should the trust be more open about its problems?

“The initial roll out for CRS Millennium was introduced over three days at the Trust, with a phased approach.  We did this to ensure the system was working in each department, before introducing it in another area.

“We are monitoring waiting time performance and records management so we can identify any issues if they emerge. The system is still being introduced in some services and when this is completed we will be able to assess the overall programme,” said the spokesman.

Does Croydon’s unwillingness to give in its statements to me any details of problems indicate that the culture of a lack of transparency in the NHS will be hard to change, no matter how many times Jeremy Hunt talks about the need for candour when things go wrong?

The spokesman:

“I’d like to be clear about the Trust’s approach:

  • The Trust board has been cited on the roll out of CRS Millennium and any potential risks throughout the process.  As I previously noted, the board received an update in September.  The board meeting, which will take place on Monday of next week, will receive a further update from the Chief Executive.  The papers from this meeting will be published on our website and the meeting takes place in public;
  • A meeting chaired by the Chief Operating Officer has reviewed any operational matters arising on a daily basis.  This is an internal meeting for clinicians and managers which has informed the implementation process;
  • Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“These actions would suggest that the Trust has been transparent in its approach.  You are welcome to review the board papers when they are published.”

Serious problems now emerge

Croydon did indeed publish its board papers on 25 November 2013 – which is to its credit because not all NHS trusts publish timely board papers.

But it’s mostly in the small print of various board papers that details emerge of Millennium-related problems. The shortcomings are mentioned as individual items rather than in a single, detailed Cerner Millennium deployment report.  This leaves one to question whether trust directors have an overview of the seriousness of the difficulties arising from its implementation of a new patient records system.

These are some excerpts from deep inside Croydon’s latest board papers:

Breaches in waiting time standards

– “CRS Millennium (Cerner) Deployment -Network downtime – Week 1.  In particular, the significant network downtime in week 1 (BT N3 problem) led to no electronic access to Pathology and Radiology which resulted in longer waits for patients in the Emergency Department (ED) leading to a large number of breaches. This was a BT N3 problem which has been rectified with BT providing CHS with the required scale of N3 access (>600 concurrent users and >1,600 users on any day – which is the largest network usage of any trust in England).”

– • “Hospital Based Pathways: The deployment of CRS Millennium was a particular challenge in the month across the multiple service areas within the Directorate of A&E, Surgery and Maternity.

• “Cancer & Core Functions: With the implementation of CRS Millennium, the open pathways part of RTT [referral to treatment – patient waiting times) may fail the standard – validation will be completed after the narrative for this report… “

Excessive waits in A&E

– “The main drivers adversely affecting the performance in the month [October 2013) for A&E were the deployment of CRS Millennium and the commencement of winter pressures due to the seasonality change.  A&E  4-Hour Total Time in Department Target: 95.00%. Actual: 91.57%.”

Over budget

“The Trust position as at October is an adverse variance of £4.1m. This is a significant deterioration on the Month 6 position. The movement is mainly due to a significant reduction in income mainly as a result of operating issues caused by the Cerner deployment (£0.9m)…  Actual £14.8 (£14.8)m; Budget £10.7m; Variance £4.1m.”

“Cerner Millennium: Plan YTD [year-to-date] £245,000; Actual YTD  £621,000;

Significant loss in income

“… A new patient administration system was deployed in the Trust on the 30th September and 1st October (Cerner Millennium). The deployment has resulted in significant loss in income in September and October £ 1.1m. Trust performance on Activity Planning Assumptions and Key Performance Indicators is substantially worse than plan …”

Extra costs

“Medical £412k and admin £148k agency levels continue to be high due to cover for vacancies, annual leave, sickness and release of staff for Cerner training. The Trust has also incurred additional costs associated with the Cerner deployment (£600k) including overtime payments to administration staff and training costs.”

Bid to recover Cerner costs?

“… The Trust is currently forecasting a deficit position of £17.8m, which is £3.3m off the plan submitted to the NHS Trust Development Authority. This is a £3m movement from the month 6 forecast and is as a result of operational issues caused by the Cerner deployment. The current projected impact is an additional costs £1.7m and a loss in activity £1.1m . An application is to be made to recover the additional cost/losses relating to the Cerner deployment [of £2.9m] …”

HSCIC support for delays

“Cerner Millennium – Revised implementation date to Sept 2013 (achieved) ,with resultant additional costs including additional PC requirements of £146k, specialist support services £300k, procurement costs £91k, data cleansing costs £200k.

“Health& Social Care Information Centre (HSCIC) has confirmed support for the delayed implementation will be provided, accounting treatment of support to be confirmed with Department of Health.”

More money to stabilise operational position?

“As a result of operational issues caused by the Cerner deployment , Income is significantly reduced in October. The forecast assumes that the Trust will resume normal operating levels from November and that an element of the income lost will be will be recovered in the latter part of the year. A business case is being submitted to the Trust Board for additional investment in Cerner to stabilise the operational position.

“If there are further operational issues due to the Cerner deployment then this will significantly impact on the year end forecast…”

Over-optimism?

Principal risk -reporting output from Cerner is not accurate or timely. Officer in charge: CEO. Before go-live risk scores: June 2013 – 16; July – 16; Aug  – 10; Sept – 10. After go-live risk score (for Oct): 20 [high risk of likelihood and consequences]

Principal risk – operational readiness following the implementation of Cerner. Officer in charge: COO.  Before go-live risk score 15. Post go-live: 20. Risk rating before go-live – Green. After go-live – Red.

Red risks

Corporate Risk Assurance Framework

Nine risks are reported as Red [two of which relate directly to Millennium]:

“… Reporting output from Cerner is not accurate or timely. Data migration was successful. However reliance on external provider as internal knowledge has not yet been fully gained. A data quality dashboard with exception reporting is in place.

“… Operational readiness following the implementation of Cerner CRS Millennium impact conveyed to Trust Development Authority e.g. ED [Emergency Department] reporting and cost overruns

Risk scores

– Failure of CRS millennium to deliver anticipated benefits – 12. Officer in charge: CEO

– Reporting output from Cerner is not accurate or timely – 20. Officer in charge: CEO

– Operational readiness following the implementation of Cerner – 20. Officer in charge: COO

Croydon’s trust’s response to problems

Said John Goulston, Croydon’s CEO, in his latest [November 2013] report to the board of directors:

“The issues being encountered now with CRS Millennium are not due to any lack of integration testing with legacy applications or testing of workflow. They can be attributed to changing from a 25 year old Patient Administration System (Patient Centre) which did not require working in real time, was simple and intuitive to use, easily configurable and flexible to our needs.

“CRS Millennium’s patient administration functions are almost the complete opposite and the language used is new for our staff i.e. conversations, encounters etc. For our staff it has been a big ask for them to step into and up to such a complex application.”

He added: “The benefits of the new system are that each patient will have a single accurate electronic record that can be viewed and kept up-to-date by hospital and community clinical staff. This will eventually mean less time searching for patient notes, missing documentation and duplicating patient information…

“As with any massive change, there are still some challenges to tackle in making the system work effectively for every single user, in a diverse and complex organisation.

“However the success we have achieved to date is the result of the efforts of every single system user and all staff members. I would like to thank all our staff for their hard work in getting the Trust to this important stage.”

The trust spokesman gave me this statement on the problems:

“The Trust board has been given regular reports on the roll out of CRS Millennium and any potential risks throughout the process, not least through its regular reviews of the Corporate Risk and Board Assurance Frameworks.  As I previously noted, the board received a specific update in September.

“As you already know, November’s board meeting received a further update from the Chief Executive.  The papers from this meeting were published and the meeting takes place in public;  Those attending are invited to put forward questions.

“A meeting chaired by the Chief Operating Officer continues to review operational matters.  This is an internal meeting for clinicians and managers which has informed the implementation process;

“Patients and visitors to the hospital have been kept fully appraised of the introduction of the system and were made aware that they may experience some delays to the check-in process while staff became familiar with the new computer system;

“As you highlight from the board report, Cerner & BT noted that ‘the Trust has undertaken one of the most efficient roll-outs of the system they have worked on’   The papers also note some operational challenges as the system was rolled out.  These have been addressed as part of the daily meetings I reference above – these are mainly concerned with users familiarising themselves with the system and have been addressed through the support and training staff received.

“In terms of the costs, the introduction of CRS Millennium has been supported by central funding from the Department of Health with the Trust paying the implementation overheads.   These costs are a matter of public record and the Trust publishes annual Accounts as part of its Annual Report.”

Comment

When you go into hospital it’s reassuring to know the directors will be well informed and open about problems that could affect you.

The approach of Croydon Health Services NHS Trust to openness about its problems is not reassuring. It is no better or worse than other trusts that have implemented Cerner’s Millennium. In fact the timely publication of its board papers means it is more open than some.

But it should not require a time-consuming journalistic investigation to establish the consequences for patients of an NPfIT go-live. It has required just such an investigation after the go-live of Millennium at Croydon.

Board directors will not have the time to dig for, and piece together, information about internal problems that could delay patient appointments, treatment and care. They need the unpalatable facts in one place. Croydon Health Services has failed to make it easy for patients or board directors to see what has gone wrong.

NPfIT deployments at other trusts have led, cumulatively, to thousands of patients having appointments that were disrupted, or who had to wait longer to be seen than necessary, or whose records were not available, or who were seen with another patient’s records.

In shying away from telling the whole truth trusts take their cue from the top: the Department of Health has always made it hard to establish facts about anything to do with the NPfIT.  Said the Public Accounts Committee in its report The National Programme for IT in the NHS: an update on the delivery of detailed care records systems in July 2011:

 “It is unacceptable that the Department [of Health] has neglected its duty to provide timely and reliable information to make possible Parliament’s scrutiny of this project.

“Basic information provided by the Department to the National Audit Office was late, inconsistent and contradictory.”

Unanswered questions

Croydon has questions to answer, such as how many breaches of waiting time standards it has had, and may still be having, due to problems arising from the go-live. Other unanswered questions:

– What does a “a large number of breaches” in the Emergency Department mean? Have each the patients affected been told?

– Why are the risks related to the implementation much higher after go-live than before, given that the trust has had years to prepare for the go-live, and the many lessons it could have learned from other trusts?

– Exactly what problems are still affecting patients?

In a post-Francis NHS, Jeremy Hunt has demanded openness about mistakes and problems. There is an agreed need for change – but how can Hunt change an NHS culture – indeed a public sector culture – in which senior executives, in troubled IT implementations, will always emphasise the good news over the bad, perhaps hoping the bad will always remain hidden?

More IT-based megaprojects derail amid claims all is well

By Tony Collins

If one thing unites all failing IT-based megaprojects in the public sector it is the defensive shield of denial that suppliers and their clients hold up when confronted by bad news.

It has happened in the US and UK this week. On the Universal Credit  project, the minister in charge of the scheme, Lord Freud, accepted none of the criticisms in a National Audit Office report “Universal Credit: early progress”.   In a debate in the House of Lords Lord Freud quoted from two tiny parts of the NAO report that could be interpreted as positive comments.

“Spending so far is a small proportion of the total budget … and it is still entirely feasible that [universal credit] goes on to achieve considerable benefits for society,” said Lord Freud, quoting the NAO report.

But he mentioned none of the criticisms in the 55-page NAO report which concluded:

“At this early stage of the Universal Credit programme the Department has not achieved value for money. The Department has delayed rolling out Universal Credit to claimants, has had weak control of the programme, and has been unable to assess the value of the systems it spent over £300 million to develop.

“These problems represent a significant setback to Universal Credit and raise wider concerns about the Department’s ability to deal with weak programme management, over-optimistic timescales, and a lack of openness about progress.”

And a shield of denial went up in the US this week where newspapers on the east and west coast published stories on failing public sector IT-based megaprojects.  The LA [Los Angeles] Times said:

As many as 300,000 jobless affected by state software snags

“California lawmakers want to know why Deloitte’s unemployment benefits system arrived with major bugs and at almost double the cost estimate. The firm says the system is working.”

The LA Times continued:

“Problems are growing worse for the state’s Employment Development Department after a new computer system backfired, leaving some Californians without much-needed benefit cheques for weeks.”

The Department said the problems affected 80,000 claims but the LA Times obtained internal emails that showed the software glitches stopped payment to as many as 300,000 claimants.

Now lawmakers are setting up a hearing to determine what went wrong with a system that cost taxpayers $110m, almost double the original estimate.

Some blame the Department’s slow response to the problems. Others point the finger at a Deloitte Consulting.

The LA Times says that Deloitte has a “history of delivering projects over budget and with problematic results”. Deloitte also has been blamed, in part, for similar troubles with upgrades to unemployment software in Massachusetts, Pennsylvania and Florida, says the paper.

“We keep hiring the same company, and they keep having the same issues,” said Senator Anthony Cannella.  “At some point, it’s on us for hiring the same company. It’s faulty logic, and we’ve got to get better.”

In 2003 California planned to spend $58m upgrading its 30-year-old unemployment benefits system. By the time the state awarded Deloitte the contract in 2010  the cost estimate had grown by more than $30m.

The Department handed out $6.6bn to about 1 million unemployed Californians in 2012. The software was expected to ease the agency’s ability to verify who was eligible to receive benefits.

Problems began when the Department transferred old unemployment data to the new system. The software flagged claims for review — requiring state workers to manually process them.

The LA Times says that officials thought initially the workload would be manageable, but internal emails showed the agency was quickly overwhelmed. Phone lines were jammed. For weeks, the Department’s employees have been working overtime to clear the backlog.

A poor contract?

In a contract amendment signed two months ago California agreed to pay Deloitte $3.5m for five months of maintenance and operations costs. Those costs should have been anticipated in the contract said Michael Krigsman, a software consultant who is an expert on why big IT-based contracts go awry. He told the LA Times:

“It’s a striking oversight that maintenance was not anticipated at the beginning of the contract when the state was at a much stronger negotiation position.”

By the time the middle of a project is reached, the state has no choice but to stick with Deloitte to work out bugs that arise when the system goes live, he said.

System works

Loree Levy, a spokeswoman for the Department, said the system is working, processing 80% of claims on time. As for the troubles, she said, “There is a period of transition or adjustment with any large infrastructure upgrade like this one.”

Deloitte spokeswoman Courtney Flaherty said the new California system is working and that problems are not the result of a “breakdown or flaw in the software Deloitte developed”.

System not working?

While there seems to be no project disaster in the eyes of the Department and Deloitte Consulting, some of the unemployed see things differently. One wrote:

“I am a contract worker who had to fight for my unemployment benefits. I won my case and yet they still cannot pay me… It’s been more than 3 weeks since I won my appeal and as of this moment, I am owed 13 weeks of back payments. To add insult to injury, they cannot send me current weeks to certify and they refuse to even try to help me to get back into the online system.

“I blame Deloitte, but it is California that carries the heaviest burden of fault… We’re nearing November and they still haven’t fixed an issue that began over Labor Day? Nonsense!

“This is untenable for everyone affected …We are owed reparations as well as our money at this point. It’s a funny word, affected. That means families and individuals are going hungry but can’t get food stamps or welfare. It means evictions and repossessed cars. It means destroyed credit, late fees, years of turmoil and shame for people already dealing with unemployment. Shame on you California.”

Another wrote:

“ … Not communicating is NOT an answer. Unemployed individuals caught up in the nightmare were told to be patient.  Rents and other expenses were still accumulating.  But [when you] add on additional fees: late fees, restoral fees, interest fees, etc…….you get the picture.

“Dear Governor Brown,

“Please reimburse me for all additional fees I’ve had to absorb to survive this fiasco.  You are going to make me payback any overpayments, but ignore the cost to the unemployed taxpayer.  This is  appears to unfair.  Perhaps Deloitte should pay us back from their contracted funds before they receive their final payment.  I am saving all of my receipts to deduct from my 2013 tax return.

“BTW Gov Brown – I am still waiting on additional payments as of today and DMV registration for my vehicle was due on 10/20/13.  Are you going to waive the penalty for late payment? Am I the only one with this question?”

Scrutiny

California’s state Assembly has set a date of 6 November 2013 for a hearing into the Department’s system upgrade.

“We’re going to look at EDD, the contractors and others to see how the system broke down so we can avoid this in the future,” said Henry Perea, chair of the Assembly’s Insurance Committee, which has oversight over the jobless benefits program.

On its website Deloitte says:

“Deloitte continues to help EDD [Employment Development Department] transform the level of service it provides to unemployed workers and improve the quality of information collected by EDD. The next time unemployment spikes, California should be ready to meet the increased demand for services.”

Massachutsetts IT disaster?

On the opposite coast the Boston Globe reported on an entirely separate debacle (which also involved Deloitte):

          None admit fault on troubled jobless benefits system

“… even with the possibility that unemployed workers could face months more of difficulties and delays in getting benefits, officials from the Labor Department and contractor, Deloitte Consulting of New York, testified before the Senate Committee on Post Audit that the rollout of the computer system was largely a success.

“‘I am happy with the launch,’ said Joanne F. Goldstein, secretary of Labour and Workforce Development, noting that she would have liked some aspects to have gone better.

“Mark Price, a Deloitte principal in charge of the firm’s Massachusetts business, acknowledged that software has faced challenges during the rollout, but insisted, ‘We have a successful working system today. ‘’’

NPfIT shield

A shield of denial was up for years at the Department of Health whose CIOs and other spokespeople repeatedly claimed that the NPfIT was a success.

Comment

If you didn’t know that Universal Credit IT wasn’t working, or that thousands of people on the east and west coasts of the US hadn’t been paid unemployment benefits because of IT-related problems, and you had to rely on only the public comments of the IT suppliers and government spokespeople, you would have every reason to believe that Universal Credit and the jobless systems in Massachusetts and California were working well.

Why is it that after every failed IT-based megaproject those in charge can simply blow the truth gently away like soap bubbles?

When confronted by bad news, suppliers and their customers tend to join hands behind their defensive shields. On the other side are politicians, members of the public affected by the megaprojects and the press who have all, according to suppliers and officials, got it wrong.

Is this why lessons from public sector IT-based project disasters are not always learned? Because, in the eyes of suppliers and their clients, the disasters don’t really exist?

None admit fault on troubled jobless benefit system

State fired Deloitte

Complaints continue despite claims system is under control

As many as 300,000 affected by California’s software problems

California’s predictable fiasco?

Who polices police IT reports?

By Tony Collins

The police, and civil and public servants in central government, the NHS and local authorities criticise journalists for biased reporting – taking selected facts out of context.

They’re sometimes right.  Journalists working for national newspapers can draft an article that is diligently balanced only to find, by the time it’s published, it leaves out facts which would have complicated, blunted, or contradicted the main points.

It’s one thing for this to happen in the world of journalism. You don’t expect public bodies to report on their own affairs with a partiality that rivals out-of-context reporting by some newspapers.

But it appears to be happening so regularly that one-sided self-reporting on organisational performance may be becoming the norm in the public sector.

In the NHS subjective, positive reporting in board papers – where managers tell directors what they think they want to hear – could help to explain why Cerner patient record implementations have, for years, gone badly wrong for the same reasons.

In recent months reports without balance have been published on the performance of Avon and Somerset Police’s IT outsourcing contract with IBM. 

Somerset County Council, Taunton Deane Borough Council and Avon and Somerset Police  are minority shareholders in a private company, Southwest One,  which is owned by IBM.

Confusingly, Taunton Deane Borough Council issued positive reports about its successful partnership with Southwest One – and then it decided to take some services back in-house.

Now it has emerged – only as a result of FOI requests by Somerset resident and campaigner Dave Orr – that two independent organisations, the National Audit Office, and HM Inspectorate of Constabulary, have commented positively on Avon and Somerset Constabulary’s partnership with Southwest One, based entirely on the unaudited opinions of the police force itself.

SAP

From his FOI requests Orr learned that the Avon and Somerset’s outsourcing deal with Southwest One has not gone entirely as expected. The National Audit Office’s FOI team has released notes of a joint visit by the NAO and HM Inspectorate of Constabulary to Avon and Somerset police in December 2012.  The visit was to find out about how well Southwest One was delivering services to the police force.  

The NAO’s notes are positive in parts. They say that performance has improved considerably since the implementation of the contract.

“Implementation of SAP improving the accounts close-down process, initial issues being resolved and a good quality of service being provided regularly.”

But there is another side to the story that is not reflected in the published accounts of Avon and Somerset’s relationship with Southwest One. The NAO’s [unpublished] field notes say:

“Fewer than expected benefits have been realised from IT due to the considerably different security requirements of the Police compared to the Councils.

“It also took a long time for SAP to be implemented. There has yet to be a duty management system implemented by SWOne which is part of the contract… SAP would have benefited from some pre-launch testing or piloting.”

A letter to Orr from the Home Office appears to confirm that Avon and Somerset Police’s participation in Southwest One is an unequivocal success.

“The private sector can help to deliver police support services better and at lower cost. Every pound saved means more money for the front line, putting officers on the streets…

“In its report “Policing in Austerity: rising to the challenge [2013] Her Majesty’s Inspectorate of Constabulary identified the Southwest One partnership as being a key element in achieving savings for Avon and Somerset Constabulary while ensuring better procurement, streamlining business support processes, and ensuring better use of police officer time.

“The report also noted that the Southwest One collaboration was the first of its kind for policing in England and Wales and that to date, no other force has delivered this level of partnership with local authorities.”

A little of the other side of the story comes in the last sentence of the Home Office letter to Orr which says: “We understand that Avon and Somerset Constabulary continues to work closely with IBM to resolve any technical difficulties and improve the services provided by Southwest One.”

Indeed a table on page 155 of HMIC ‘s 2013 report Policing in austerity: rising to the challenge indicates that Avon and Somerset Constabulary has one of the worst records of any police force when it comes to savings delivered between 2010/11 and 2012/13. [Table: Key indicators of the challenge – quartile analysis.]

Southwest One began a 10-year contract providing services to Avon and Somerset Police in 2008. The services included enquiry offices, district HR, estates, financial services, site administration, facilities, corporate human resources, information services, purchasing and supply, and reprographics. The contract involves 554 seconded staff.

Comment

Police forces, councils, the NHS and central government departments need  a few Richard Feymans to report on their organisation’s performance. Feynman was a gifted scientist, MIT graduate and noble prize winner who was chosen as a commissioner to report on the cause , or causes, of the Challenger Space Shuttle “O” rings accident on 28 January 1986.

He reported with such independence of mind and diligence that his hard-hitting findings were not considered acceptable to be included in the main report of the Presidential Commission of inquiry into the accident.  Feynman had to be content with having his findings published as an appendix to the Commission’s report – and an edited appendix at that.  

He suggested in his book “What do you care what other people think?” that his appendix was the only genuinely balanced part of the official inquiry report. 

“For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled,” said Feynman.

One of his questions was whether “organisation weaknesses that contributed to the [Shuttle] accident [was] confined to the solid rocket booster sector, or were they a more general characteristic of NASA.”

One of Feynman’s conclusions:

“It would appear that, for whatever purpose – be it for internal or external consumption – the management of NASA exaggerates the reliability of its product to the point of fantasy.”

If such exaggeration happens at NASA it can happen in UK police force IT reports, and in board papers on the performance of councils and NHS trusts.

When journalists get it wrong it’s usually to their eternal regret. In the public sector positive unbalanced reporting is so “normal” that hardly anyone involved realises it’s a deviant practice. The US author Diane Vaughan coined a phrase for such corporate behaviour.  She called it the normalisation of deviance.  

It’s surely time for public bodies to move away from the norm and start reporting on their performance, and the performance of their outsourcing other private sector contracts, with balance, objectivity and independence of mind.   

If managers knew that reports on the progress of their contracts would be audited for impartiality and competence over organisational self-interest, perhaps they would have a greater incentive to avoid badly thought through outsourcing deals and IT implementations.

Is this why some council and NHS scandals stay hidden for years?

NAO report “Private sector partnering in the police service”

Dave Orr’s HMIC FOI requests and answers

NAO’s FOI responses on Avon and Somerset Police

Capita has duty to promote success of Barnet contract

By Tony Collins

Capita has a contractual duty to promote the success of the “One Barnet” outsourcing deal with Barnet Council – apparently without taking into account facts that may count against success.

Within the 2,400 pages that make up contracts between Capita and Barnet Council, Unison has discovered clauses that appear to put the onus on the service provider to talk up the success of Barnet’s outsourcing deal.

These are excerpts from the contracts:

“The Service Provider shall use its relationships to create advocates of the success of the One Barnet programme by informing the Department of Communities and Local Government and the Local Government Association of key milestones and achievements within the programme thereby supporting increased political awareness of the Authority and the Service Provider shall utilise its corporate and personal networks to support the communication of the success of the Partnership via appropriate case studies.”

The contract points out that the service provider has “frequent meetings across central government at official level and occasional meetings at ministerial level”. It also sits on the Public Services Strategy Board, the Whitehall & Industry Group, Reform, Policy Exchange and Localis.

“The Service Provider shall use its relationships to create opportunities for the successes of the Partnership to be promoted enhancing the profile of the Authority at strategic level across the public sector,” says one of the contractual clauses.

Thank you to Dave Orr, a campaigner for openness over local government outsourcing deals, for drawing my attention to the Barnet Council clauses.

Comment

It now seems to be official – that outsourcing deals in local government have to be perceived as successful. Perhaps these sorts of clauses in local government outsourcing contracts help to explain why the public don’t learn of failing “partnerships” and joint ventures until what has gone wrong can be hidden no longer.

This is not open government. This is a contractual expectation that the supplier’s representatives should smile, and smile broadly, whenever the subject of an outsourcing deal with Barnet is discussed, or there is an opportunity to discuss it.

Which rather undermines the credibility of the Public Services Strategy Board, the Whitehall & Industry Group, Reform, Policy Exchange and Localis if supplier’s representatives are there to pass on PR messages about their outsourcing deals, whatever the truth.

“Smile and others will smile back. Smile to show how transparent, how candid you are. Smile if you have nothing to say. Most of all, do not hide the fact you have nothing to say nor your total indifference to others. Let this emptiness, this profound indifference, shine out spontaneously in your smile.” Jean Baudrillard.