Category Archives: supplier relationships

Will coalition sign a new NPfIT deal with CSC?

Posted on 14/08/2012 | 1 comment

By Tony Collins

CSC has told investors that its discussions with the UK government on an interim agreement for deploying Lorenzo to the NHS are “continuing positively”.

CSC says that an agreement could commit a certain number of NHS trusts to take Lorenzo. Some of those trusts would be named in the interim agreement and the remainder within six months. CSC refers to them as “committed named trusts”.

[Such a legal commitment for named NHS trusts to take Lorenzo may run counter to the post-NPfIT coalition philosophy of giving trusts the freedom to buy what they want, when they want, and from whom they want. The named trusts might have indicated on a  DH questionnaire a wish to take Lorenzo but an agreement between the government and CSC would commit the trusts irrevocably, or the DH could have to pay CSC compensation for non-deployment.]

CSC says the deployed product would be categorized as “base product” or “additional product” for pricing purposes. The DH would commit money to the base product. Other funds would be available centrally available for “additional products,  supplemental trust activity and local configuration”.

The DH would give CSC a structured set of payments following certain product deliveries, as well as additional payments to cover various deployments for the named trusts and payments for work already performed.

If the government does not sign a new deal, and allows CSC’s existing contracts to run down until they expire formally in 2015, this could keep further NPfIT-related costs to the taxpayer to a minimum.  But it risks legal action from CSC, which says the NHS contract is enforceable and that the NHS has no existing right to terminate the contract, unless for convenience (which is unlikely).

If the government had terminated CSC’s contracts for its convenience (as opposed to alleged breach of contract) it would have had to pay CSC a termination fee capped at £329m as of 29 June 2012. CSC would also have been entitled to compensation for the profit it would have earned for the 12 months after the contract was terminated.

If the contract is not terminated, a new deal not signed, and no legal action is taken by either side, the amounts the UK government would have to pay CSC are likely to be minimised.  It is in CSC’s interests to maintain and enhance Lorenzo for those NHS sites that have deployed it.

So will the government sign a new deal with CSC at least to reduce the risks of CSC legal action? Or could the government hold out not signing any agreement until expiry of the contracts in 2015 on the basis that CSC has not delivered all it promised?

If a new deal is signed – and CSC indicates that an agreement is likely – the government may face accusations that it has broken its undertaking to dismantle the NPfIT.

David Camerson intervened personally to have the Cabinet Office’s Major Projects Authority look closely at NPfIT commitments.  His “efficiency” minister Francis Maude is likely to resist the signing of any new agreement

But will CSC accept the government’s refusal to sign a new deal, when such a deal could enable CSC to recover at least some of the $1.485bn (£0.95bn) it recorded as an NPfIT contract charge in the third quarter of 2012?

1 Comment

Posted in CSC, e-health, Government IT, health, health records, NHS, npfit, supplier relationships

Tagged , , , , , ,

Some lessons from a major outage

Posted on 31/07/2012 | 2 comments

By Tony Collins

One of the main reasons for remote hosting is that you don’t have to worry about security and up-time is guaranteed. Support is 24x7x365. State-of-the-art data centres offer predictable, affordable, monthly charges.

In the UK more hospitals are opting for remote hosting of business-critical systems. Royal Berkshire NHS Foundation Trust and Homerton University Hospital NHS Foundation Trust are among those taking remote hosting from Cerner, their main systems supplier.

More trusts are expected to do the same, for good reasons: remote hosting from Cerner will give Royal Berkshire a single point of contact to deal with on technical problems without the risks and delay of ascertaining whether the cause is hardware, third party software or application related.

But what when the network goes down – across the country and possibly internationally?

With remote hosting of business-critical systems becoming more widespread it’s worth looking at some of the implications of a major outage.

A failure at CSC’s Maidstone data centre in 2006 was compounded by problems with its recovery data centre in Royal Tunbridge Wells. Knock-on effects extended to information services in the North and West Midlands. The outage affected 80 trusts that were moving to CSC’s implementation of Lorenzo under the NPfIT.

An investigation later found that CSC had been over-optimistic when it informed NHS Connecting for Health that the situation was under control. Chris Johnson, a professor of computing science at Glasgow University, has written an excellent case study on what happened and how the failure knocked out primary and secondary levels of protection. What occured was a sequence of events nobody had predicted.

Cerner outage

Last week Cerner had a major outage across the US. Its international customers might also have been affected.

InformationWeek Healthcare reported that Cerner’s remote hosting service went down for about six hours on Monday, 23 July. It hit “hospital and physician practice clients all over the country”. Information Week said the unusual outage “reportedly took down the vendor’s entire network” and raised “new questions about the reliability of cloud-based hosting services”.

A Cerner spokesperson Kelli Christman told Information Week,

“Cerner’s remote-hosted clients experienced unscheduled downtime this week. Our clients all have downtime procedures in place to ensure patient safety. The issue has been resolved and clients are back up and running. A human error caused the outage. As a result, we are reviewing our training protocol and documented work instructions for any improvements that can be made.”

Christman did not respond to a question about how many Cerner clients were affected. HIStalk, a popular health IT blog, reported that hospital staff resorted to paper but it is unclear whether they would have had access to the most recent information on patients.

One Tweet by @UhVeeNesh said “Thank you Cerner for being down all day. Just how I like to start my week…with the computer system crashing for all of NorCal.”

Another by @wsnewcomb said “We have not charted any pts [patients] today. Not acceptable from a health care leader.”

Cerner Corp tweeted “Our apologies for the inconvenience today. The downtime should be resolved at this point.”

One HIStalk reader praised Cerner communications. Another didn’t:

“Communication was an issue during the downtime as Cerner’s support sites was down as well. Cerner unable to give an ETA on when systems would be back up. Some sites were given word of possible times, but other sites were left in the dark with no direction. Some sites only knew they were back up when staff started logging back into systems.

“Issue appears to have something to do with DNS entries being deleted across RHO network and possible Active Directory corruption. Outage was across all North America clients as well as some international clients.”

Colleen Becket, chairman and co-CEO of Vurbia Technologies, a cloud computing consultancy, told InformationWeek Healthcare that NCH Healthcare System, which includes two Tampa hospitals, had no access to its Cerner system for six hours. The outage affected the facilities and NCH’s ambulatory-care sites.

Lessons?

A HIStalk reader said Cerner has two electronic back-up options for remote hosted clients. Read-only access would have required the user to be able to log into Cerner’s systems, which wouldn’t have been possible with the DNS servers out of action last week.

Another downtime service downloads patient information to local computers, perhaps at least one on each floor, at regularly scheduled intervals, updated say every five minutes. “That way, even if all connection with [Cerner’s data centre] is lost, staff have information (including meds, labs and more) locally on each floor which is accurate up to the time of the last update”.

Finally, says the HIStalk commentator, “since this outage was due to a DNS problem, anyone logged into the system at the time it went down was able to stay logged in. This allowed many floors to continue to access the production system even while most of the terminals couldn’t connect.”

But could the NHS afford a remote hosted service, and a host of on-site back-up systems?

Common factors in health IT implementation failures

In its discussion on the Cerner outage, HIStalk put its finger on the common causes of hospital IT implementation failures. It says the main problems are usually:

– a lack of customer technical and implementation resources;
– poorly developed, self-deceiving project budgets that don’t support enough headcount, training, and hardware to get the job done right;
– letting IT run the project without getting users properly involved
– unreasonable and inflexible timelines as everybody wants to see something light quickly up after spending millions; and
– expecting that just implementing new software means clearing away all the bad decisions (and indecisions) of the past and forcing a fresh corporate agenda on users and physicians, with the suppplier being the convenient whipping boy for any complaints about ambitious and sometimes oppressive changes that the culture just can’t support.

Cerner hosting outage raises concerns

HIStalk on Cerner outage

Case study on CSC data centre crash in 2006

2 Comments

Posted in Cerner, Disaster recovery, e-health, health, health records, IT-related failures, NHS, npfit, public sector, public services, Remote Hosting, software quality, supplier relationships

Trust buys extras-laden system for 20% of BT’s price

Posted on 30/07/2012 | Leave a comment

By Tony Collins

An NHS Trust has paid £1.8m for the “Rio” patient record system and a host of extras – which raises fresh questions about why the Department of Health paid BT £9m for each Rio deployment under the NPfIT.

BT’s deal with the Department of Health under the NPfIT costs taxpayers £224.3m for 25 deployments of Rio – about £9m for each one. The deal included support for five years, until October 2015.

But a deal struck directly between Cambridgeshire and Peterborough NHS Foundation Trust and Rio’s supplier CSE-Healthcare Systems includes support for a year longer – six years in total. At £1.806m the cost of Cambridgeshire and Peterborough’s deal is about 20% BT’s price.

Cambridgeshire and Peterborough has secured more extras. CSE-Healthcare Systems will supply Cambridgeshire and Peterborough with:

  – Medical software development services

– Software consultancy services

 – Computer support services

– System implementation planning services

– Software implementation services

– Project management consultancy services

– Business and management consultancy services

– Medical software package

– Database and operating software package

– Information systems and servers

– Computer and related services

– Software support services

It’s not clear though whether the deal includes disaster recovery – which the Department of Health argues is one reason for the price paid to BT for Rio.

Comment:

Disaster recovery does not explain how or why the Department of Health agreed to pay BT five times more for Rio than paid by Cambridgeshire and Peterborough.

What is not in doubt is that if BT, like Accenture and Fujitsu before it, had left the NPfIT, the programme would have been indelibly stamped a disaster – at a time when the Department of Health and ministers were proclaiming it an unacknowledged success (in the run up to a general election).

Was BT paid a hugely costly premium to stay in the NPfIT?  If so, should tax money have been used to secure the affections of a supplier whose profits were otherwise being squeezed on the contract?

Last year the Department of Health gave the Public Accounts Committee an explanation of its Rio payments to BT, some of which is below. The explanation is long, vague and defensive enough to sound like the excuses of a young boy who, when questioned by his teacher, gives various accounts of why he took a classmate’s sweets.

What the DH told MPs

This is what the DH told the Public Accounts Committee last year, after one of its MPs Richard Bacon queried the payments to BT for Rio (and Cerner).

“Assessing the value for money of the RiO prices was part of the CCN3 [change control note 3] negotiation process … The CCN3 negotiation process was lengthy and involved many iterations challenging the component parts of the BT cost model. This included ensuring that rates offered were competitive and that the effort ascribed to various activities was justifiable.

Taken together, competitive rates and reasonable effort comprise value for money. BT was required to provide numerous iterations of financial models. These models were reviewed in detail by the Authority, resulting in multi-million pound savings.

The Authority negotiated reduced day rates on all of the BT labour within the contract. In addition, BT’s profit margin on the contract was also significantly reduced. Other commitments were also obtained by the Authority, in particular around sub-contractor pricing; for example, Cerner has confirmed that the pricing provided to the Authority (via BT) is the best it provides to any of its customers.

The Authority then requested further financial assurances and agreed with BT that a requirement of signing CCN3 would be that a verification exercise would be conducted by third party, independent financial experts (KPMG)…

In October 2010 KPMG were requested by the Authority to verify the costs presented by BT, including those for RiO, in the CCN3 Financial Model.

The approach adopted by KPMG was as follows:

—  Their work focussed on the Cost Data sheet within the CCN3 Financial Model and was conducted on a sample basis, designed to provide a high coverage of costs with a reasonable sample size.

—  The cost elements for potential duplicate entries were reviewed.

—  The cost rates associated with BT labour were validated to cost rate cards and payroll records.

—  The hours presented in the Model associated with BT labour were reviewed for reasonableness.

—  Sub-contractor and other supplier costs were validated to the agreements entered into by BT with their suppliers.

—  Cost elements and supporting documentation requested from BT were sampled to substantiate the costs provided…

On 3 November 2010 KPMG concluded that “BT has provided underpinning evidence to support the agreed delivery costs” and that “no proposed adjustments are required for Agreed Delivery Costs”.

None of the trusts consulted had purchased the same RiO product offering and all trusts varied significantly from the offering provided to NPFIT trusts, making a direct price comparison difficult. However, trusts within the programme typically had significant advantages to those outside the programme, namely:

—  The ability to influence the functionality of the product.

—  Centrally provided and hosted hardware.

—  Centrally-provided disaster recovery with 100% capacity and availability.

—  No additional development costs for subsequent releases.

—  Spine connectivity.

BT estimate that the monthly charge for hardware, disaster recovery, service management and Spine connectivity to be in the region of £42,500 per month or just over £2 million of value over a 48 month contract term.

Furthermore the NPFIT investment in the development of the RiO has significantly enhanced the functionality of the product to the benefit of all trusts. Examples of functionality in the latest deployed version (v5) and soon to be deployed (R1, 2011) of RiO include:

—  Standard assessment forms.

—  Care-plans and reports.

—  Spine connectivity, enabling integration with central demographics services, and functionality to support smart cards and role based access controls.

—  Waiting lists.

—  Results reporting.

—  Prevention, screening and surveillance.

—  SNOMED.

—  Inpatient prescribing.

—  Functionality to support multi-disciplinary care planning.

DH statement to MPs on Rio (and Cerner) prices paid to BT

MP seeks inquiry into £546m NHS deal with BT.

Breakdown of £546m NPfIT payment to BT

Leave a comment

Posted in BT Health, Government IT, health records, NHS, npfit, public sector, public services, supplier relationships

Tagged

How to identify a high-risk supplier – Cabinet Office works out details

Posted on 25/07/2012 | 5 comments

By Tony Collins

Francis Maude, the Cabinet Office minister, has agreed mechanisms for officials to identify high-risk suppliers where “material and substantial underperformance is evident”.

On his blog Spend Matters, Peter Smith has published parts of a letter from Maude.

Where under-performing suppliers are identified “departments will be asked to engage with the Cabinet Office at each stage of any procurement process involving the affected supplier to ensure that performance concerns are taken fully into account before proceeding”.

The implication is that the Cabinet Office will draw up a blacklist of bad suppliers which departments will take account of when buying. Smith says that two suppliers are already on the blacklist.

Comment: 

For more than 20 years the trade press has identified the same suppliers in a succession of failed or failing IT-based projects but poor performance has never been taken seriously into account.

This is usually because the suppliers argue that the media and/or Parliament has got it all wrong.  Departments, it appears, will always prefer a potential supplier’s version to whatever is said in the media or in Parliament.

The Office of Government Commerce, now part of the Cabinet Office, kept intelligence information on suppliers but it seems to have made no difference in procurements.

It is unlikely the Cabinet Office’s blacklist will rule out any suppliers from a shortlist. As Smith says, suppliers will claim that any problem was all the fault of ministers or civil servants who kept changing their minds, were not around to make key decisions, or didn’t understand the nature of the work.

But still the blacklist is a worthwhile innovation. At least one big IT supplier has made a habit of threatening to withdraw from existing assignments when officials have refused to revise terms, prices or length of contract. The blacklist will strengthen the negotiating hand of officials.

The challenge for Maude will be persuading departments to take the blacklist idea seriously.

Peter Smith, Spend Matters.

5 Comments

Posted in Campaign4Change, EU procurement, Government IT, innovation, IT-related failures, legal disputes, managing outsourced services, operational issues, private sector lessons for the public sector, supplier relationships

Tagged , , ,

IBM won bid without lowest-price – council gives detail under FOI

Posted on 10/07/2012 | Leave a comment

By Tony Collins

Excessive secrecy has characterised a deal between IBM and Somerset County Council which was signed in 2007.

Indeed I once went to the council’s offices in Taunton, on behalf of Computer Weekly, for a pre-arranged meeting to ask questions about the IBM contract. A council lawyer refused to answer most of my questions because I did not live locally.

Now (five years later) Somerset’s Corporate Information Governance Officer Peter Grogan at County Hall, Taunton, has shown that the council can be surprisingly open.

He has overturned a refusal of the council to give the bid prices. Suppliers sometimes complain that the public sector awards contracts to the lowest-price bidder. But …

Supplier / Bid Total cost over 10 years
BT Standard bid £220.552M
BT Variant Bid £248.055M
Capita Standard Bid £256.671M
Capita Variant Bid £267.687M
IBM Standard Bid £253.820M
IBM Variant Bid £253.820M

The FOI request was made by former council employee Dave Orr who has, more than anyone, sought to hold Somerset and IBM to account for what has turned out to be a questionable deal.

Under the FOI Act, Orr asked Somerset County Council for the bid totals. It refused saying the suppliers had given the information  in confidence. Orr appealed. In granting the appeal Grogan said:

“I would also consider that the passage of time has a significant impact here as the figures included under the exemption are now some 5 years old and their commercial sensitivity is somewhat eroded.

“Whilst, at the time those companies tendering for the contract would justifiably expect the information to be confidential and that they could rely upon confidentiality clauses, I am not able to support the non-disclosure due the fact that the FOI Act creates a significant argument for disclosure that outweighs the confidentiality agreement once the tender exercise is complete and a reasonable amount of time has passed.

“I therefore do not consider this exemption [section 41] to be engaged. Please find the information you requested below…”

[In my FOI experience – making requests to central government departments – the internal review process has always proved pointless. So all credit to Peter Grogan for not taking the easy route, in this case at least.]

MP Ian Liddell-Grainger ‘s website on the “Southwest One” IBM deal.

IBM struggles with SAP two years on – a shared services warning.

Council accepts IBM deal as failing.

Was Audit Commission Somerset and IBM’s unofficial PR agents?

Leave a comment

Posted in Campaign4Change, change, change management, excessive secrecy, Freedom of Information, governance, IBM, IT-related failures, managing change, managing outsourced services, SAP, shared services, software quality, supplier relationships

Tagged ,

HMRC “still plagued by IT problems”

Posted on 28/06/2012 | Leave a comment

By Tony Collins

HMRC has one of the biggest IT outsourcing contracts in central government, a deal worth about £8bn with Capgemini, which began in 2004. Before that, between 1994 and 2004, the main IT supplier was EDS, now HP. But HMRC has had pervasive IT-related challenges for more than a decade.

Today Margaret Hodge, Chair of the Committee of Public Accounts, commented   on a report by the National Audit Office on HM Revenue & Customs’ 2011-12 accounts.

“Sadly it is no surprise that the NAO has found substantial problems with the HMRC’s accounts. This year has seen a litany of tax errors and scandals come to light with mistakes made at the most senior level from the Permanent Secretary for Tax downwards.

“The sheer scale of waste and mismanagement at HMRC never ceases to shock me. Without even mentioning the tax gap, in 2011-12 the Department wrote off a staggering £5.2 billion of tax owed, overpaid nearly £2.5 billion in tax credits due to fraud and error and underpaid around £290 million.

“In some areas the Department is moving in the right direction and has made progress to implement improvement plans. But the Department is still plagued by IT problems; limiting, for example, its ability to link together the debts owed by tax payers across different tax streams.

“With its long history of large scale IT failures, the Department needs to get a grip before it introduces its new real time PAYE information systems and begins the high-risk move from tax credits to the Universal Credit.”

Timetable for HMRC’s work on Universal Credit is “challenging” says NAO.

NAO – HMRC’s 2011/12 accounts

Time for truth on Universal Credit.

Leave a comment

Posted in Government IT, HMRC, IT projects, outsourcing, supplier relationships

Tagged ,

Poor IT suppliers to face ban from contracts?

Posted on 28/06/2012 | Leave a comment

By Tony Collins

The Cabinet Office minister Francis Maude is due to meet representatives of suppliers today, including  Accenture BT,Capgemini, Capita, HP, IBM, Interserve, Logica, Serco, and Steria.

They will be warned that suppliers with poor performance may find it more difficult to secure new work with the Government. The Cabinet Office says that formal information on a supplier’s performance will be available and will be taken into consideration at the start of and during the procurement process (pre-contract).

Maude will tell them that the Government is strengthening its supplier management by monitoring suppliers’ performance for the Crown as a whole.

“I want Whitehall procurement to become as sharp as the best businesses”, says Maude. “Today I will tell companies that we won’t tolerate poor performance and that to work with us you will have to offer the best value for money.”

The suppliers at today’s meeting represent around £15bn worth of central government contract spend.

The representatives will also be:

– asked their reactions on the government’s approach to business over the past two years

– briefed on the expanded Cabinet Office team of negotiators (Crown Representatives) from the private and public sectors. Maude says these negotiators aim to maximise the Government’s bulk buying power to obtain strategic discounts for taxpayers and end the days of lengthy and inflexible contracts.

Spending controls made permanent

Maude is announcing today that cross-Whitehall spending controls will be a permanent way of life. The Government introduced in 2010 temporary controls on spending in areas such as ICT  and consultancy. It claims £3.75bn of cash savings in 2010/11, and efficiency savings for 2011/12, which it says are being audited.

The Cabinet Office says: “By creating an overall picture of where the money is going, the controls allow government to act strategically in a way it never could before. For example, strict controls on ICT expenditure do not just reduce costs but also reveal the software, hardware and services that departments are buying and whether there is a competitive mix of suppliers and software standards across government.”

Maude said: “Our cross-Whitehall controls on spending have made billions of cash savings for the taxpayer – something that has never been done before. That’s why I’m pleased to confirm that our controls will be a permanent feature, helping to change fundamentally the way government operates.”

Leave a comment

Posted in cost reduction, efficiency savings, Government IT, IT-related failures, operational issues, outsourcing, oversight, Politics, private sector lessons for the public sector, procurement, project management, public sector, software quality, supplier relationships

Tagged , ,

Why is MoD spending more on IT when its data is poor?

Posted on 28/06/2012 | Leave a comment

By Tony Collins

The Ministry of Defence and the three services have spent many hundreds of millions of pounds on logistics IT systems over the past 20 years, and new IT projects are planned.

But the National Audit Office, in a report published today – Managing the defence investory –  found that logistics data is so unreliable and limited that it has hampered its investigations into stock levels.

“During the course of our study,” says the NAO, “the Department provided data for our analyses from a number of its inventory systems. However, problems in obtaining reliable information have limited the scope of our analysis…”

The NAO does not ask the question of why the MoD is spending money on more IT while data is unreliable and there are gaps in the information collected.

But the NAO does question whether new IT will solve the MoD’s information problems.

“The Department has acknowledged the information and information systems gaps and committed significant funds to system improvements. However these will not address the risk of failure across all of the inventory systems nor resolve the information shortfall.”

MPs on the Public Accounts Committee, who will question defence staff on the NAO report, may wish to ask why the MoD’s is so apparently anxious to hand money to IT suppliers when data is poor and new technology will not plug information gaps.

Comment:

MPs on the Public Accounts Committee found in 2003 (Progress in reducing stocks) that the MoD was buying and storing stock it did not need. Indeed after two major fires at the MoD’s warehouses at Donnington in 1983 and 1988 more than half of the destroyed stock did not need replacing. Not much has changed judging by the NAO’s latest report.

It’s clear that the MoD lacks good management information. Says the NAO in today’s report:

“The summary management and financial information on inventory that is provided to senior staff within Defence Equipment and Support is not sufficient for them to challenge and hold to account the project teams…”

But will throwing money at IT suppliers make much difference? The MoD plans the:

–  Future Logistics Information Services project, which is intended to bring together and replace a number of legacy inventory management systems; and

–  Management of the Joint Deployed Inventory system which will provide the armed services with a common system for the inventory they hold and manage.

But is the  MoD using IT spending as proof of its conviction to improve the quality of data and the management of its inventory?

Managing the defence inventory

Leave a comment

Posted in cost reduction, Defence, Government IT, IT projects, IT-related failures, oversight, private sector lessons for the public sector, project management, public sector, public services, software quality, supplier relationships

Tagged ,

RBS/Natwest: Some lessons from the IT crash – Bank of England Governor.

Posted on 26/06/2012 | 1 comment

By Tony Collins

Sir Mervyn King, Governor Bank of England, promised today that there would be a “very detailed inquiry” once problems at RBS/Natwest are back to normal.

Such a report would be unusual because the cause or causes of IT-related crashes in the public and private sectors are usually kept secret unless in rare cases a legal action comes to court.

Mervyn King told the Treasury Committee today:

“Once the current difficulties are over then we will need the FSA to go in and carry out a very detailed investigation to find out first of all what went wrong but even more importantly why it took so long to recover.

“Computer systems will always go wrong from time to time. The important things are your back-up systems and the time it takes to implement recovery. As of now we have kept in very close touch. My office was in touch with senior RBS management right through the weekend. Our banking director was in touch with RBA and FSA on this right since this problem began … It is still going to take time to catch up, to get back to normal.

“The important thing now is that we provide whatever support is needed to let them put it right. Once it is back to normal then we must carry out a very detailed inquiry.

“To my mind, one of the big lessons from this is that it shows everyone is how important the basic functions of banking actually are: what can go wrong when the system of payments from person to another is interrupted. Fortunately it has been one bank, albeit a very big bank, and customers of that bank have been affected, and of course customers of other banks have been affected, and payments have not gone through.

“I hope this is a reminder, a demonstration, to everyone, for example, of what might have happened if we had not rescued RBS in the autumn of 2008. The whole payment system would have collapsed. [It is] why it is so important to ensure you have a banking system where the people running it are completely focused on this essential service function of banking to provide … customers with a functioning payment system.

Learning from supermarkets.

“I have been driven by the belief that the nature of banking and providing these kinds of services is very different from investment banking operations. Those are important but they are very different. When you go out and see how supermarkets operate, the senior management is utterly focused on ensuring that the IT systems, the ordering systems, the delivery system, works hour by hour. That is very important to ensure that that is true of the banking system as well…”

Comment:

History  is, to some extent, the story of the unforeseen, in which case a published report on the cause of the problems at RBS/Natwest could be helpful to other banks and major organisations whose ageing systems are vulnerable to an unforeseen failure of huge proportions.

A published report on the crisis may show systemic management failures. The mere fear of such a report would be an added deterrent – additional to potential losses and payments of compensation – to any bank that does not give the attention it should to operational systems, even when those systems support a retail banking operation that may represent a small part, perhaps only 2% of a bank’s balance sheet.

It is ironic that RBS is publicly owned. Will the IT disaster now be added to the list of other public sector failures? Did RBS, now in the public sector, drop its IT-related standards and caution in part because the commercial imperative was absent?

Natwest/RBS – what went wrong?

1 Comment

Posted in Government IT, IBM, IT projects, IT-related failures, managing outsourced services, operational issues, software quality, supplier relationships

Natwest/RBS – what went wrong?

Posted on 25/06/2012 | 4 comments

By Tony Collins

Outsourcing to India and losing IBM mainframe skills in the process? The failure of CA-7 batch scheduling software which had a knock-on effect on multiple feeder systems?

As RBS continues to try and clear the backlog from last week’s crash during a software upgrade, many in the IT industry are asking how it could have happened.

Stephen Hester, RBS’s boss, told the BBC today:

“In simple terms there was a software change which didn’t go right. Although that was put right quickly there then was a big backlog of things that had to be reprocessed in sequence. That got on top of our technical teams … it is like the landing path at Heathrow. Once you get out of sequence it takes a time to get back into sequence even if the original fault is put right.

“Our people are working incredibly hard … I am pleased to report that as of today RBS and Natwest systems are operating normally.

“We need to make sure they stay normal for the next few days. There is still some significant catch-up today, much less tomorrow and so on as we go through the week.”

The immediate technical cause of the problems might not have been too difficult for those inside the bank to establish – but finding out how and why it happened, why processes were not in place to stop a backlog of work building up, and why testing of the upgrade did not pre-empt the failure may take weeks and possibly months to establish.

Attributing blame could take many years. After BSkyB appointed EDS to supply a CRM system in 2000, and the project failed, it was ten years later before a court reached a judgment on blame. The cause of the failed project was never definitively established.

Official cause of system crash

The official cause of RBS/Natwest’s problems was given at the weekend by Susan Allen, Director of Customer Services, RBS Group which includes Natwest and Ulster Bank. She told Paul Lewis of BBC’s Moneybox programme:

“Earlier this week we had a problem in our overnight backup. So a piece of software failed that started all the updates that happened to our systems overnight.

“What that has meant practically is that information on customers’ accounts has not been updated… It is horrendous.

“The underlying problem has been fixed, so the computer software that failed has been replaced. That is in and working. The challenge we now have is bringing all the systems back up and working through all the data that should have been gone through over the last three nights …

“We have 12 million customers in Natwest and RBS and just over 100,000 in Ulster Bank. So it is affecting a serious number of people. It is having a terrible impact.

“We are encouraging all of our customers to call us, come and see us in our branches … we have branches open late .. and have doubled the number of people on the phone. Call centres are open 24 hours a day.”

Call centres use 0845 numbers which are chargeable for some. Lewis asked, Why are you making people pay to fix a problem that’s your fault?

“Customers should not be having to pay for those calls,”replied Allen. “If that is a problem for people we will take a look at that.”

Lewis: Will you re-imburse people for their calls?

“Absolutely. We recognise there will be lots of different expenses as a result of this. We apologise and want to make sure they are not out of pocket. If people have got claims they should put them through to us…we will need the information to deal with the claims.”

Lewis: Will you refund charges by credit card companies for late payments?

“We will. We will… we will make sure nobody is out of pocket… in one instance we got cash in a cab to a customer’s home… clearly we trust our customers so if we can see that somebody has a certain amount coming in every week we will give them money against that. So we ask people to come in and bring identification with them such as their bank card, we will do what we can to help.

“We will look after our customers. We realise this has had a huge impact on people. We are not underestimating it … clearly there are things that have gone wrong and we cannot put everything right.”

Lewis: How much damage has this done to the reputation of the bank?

“Time will tell. For us it is pretty devastating. We pride ourselves on being a bank that really cares about our customers and wants to deliver great service. We absolutely mean it.”

Lewis: Should you get a bonus?

“We only get performance bonuses when we perform and this has not been a good performance.”

Comment:

Her explanation of the cause of the IT crash is unclear but otherwise Susan Allen’s answers to Paul Lewis’s questions were exemplary. Her openness and unaffected humility is surely the best way to handle a PR crisis. Small comfort for the millions affected though.

Technical cause of the crash?

Some of those commenting to The Register appear to have a good knowledge of RBS systems. There are suggestions RBS has lost some important IBM mainframe software skills in outsourcing.

One or two have suggested that the crash was caused by a failure of the bank’s CA-7 batch scheduling software. In February RBS had an “urgent requirement” in Hyderabad, India, for people with four to seven years experience of CA7.

One comment on The Register said that RBS runs updates on customer accounts overnight on an IBM mainframe, via a number of feeder systems that include BACS. “The actual definitive customer account updates were carried out by a number of programs written in assembly language dating back to about 1969-70, and updated since then. These were also choc-full of obscure business rules … and I do not believe anyone there really knew how it all worked anymore, even back in 2001…

“Of course the moral is complex mainframe systems require staff with the skills, and in this case, the specific system knowledge to keep things smooth. The fewer of these you have, the more difficult it is to recover from problems like this.”

Robert Peston, the BBC’s Business Editor, asks whether outsourcing was to blame.

“In my conversations with RBS bankers, there is an implication that outsourcing contributed to the problems – though they won’t say whether this is an issue of basic competence or of the complexities of co-ordinating a rescue when a variety of parties are involved.”

An RBS spokesperson told The Register that the software error occurred on a UK-based piece of software.

Some lessons from the crisis – Bank of England Governor.

4 Comments

Posted in Campaign4Change, India, IT and the Business, IT projects, IT-related failures, M&A, managing outsourced services, operational issues, outsourcing, supplier relationships

Tagged