Bank Watch ...

[woodburner]

Hell’s bells!! how much more would it have cost if it was phased? they must be a couple of legs short by now, and working quickly through the first arm.

[Potemkin Villager]

Apparently all in one go is cheaper. Now they've drafted in IBM to help things can only go from very bad to even worse.

Install Bigger Memory!

[johnhemming2]

Apparently all in one go is cheaper.

It is a question of risk management. Software is complex and it is difficult to find all the problems with an implementation before it is fully running.

[woodburner]

Using the term “management� in this case is a tad euphemistic.

[PS_RalphW]

There is a report on the guardian site that the software update was a disaster waiting to happen. Tsb has been using a clone of Lloyd s software since it was split off a few years ago, at a licence cost of £100m per year. This is a monstrosity of 6 or more merged systems from all the banks and building societies absorbed by Lloyds over the decades. The upgrade was a to move to a much simpler system based on a Spanish bank. The budget was £450m and that was deeply inadequate. The project was already 6 months late and major code changes were made days before going live. Result? I doubt that many people will get their money back, it will be lost in the ether. Tsb's only hope is to renegotiate the Lloyds lease on the old software, or beg the government to bail them out

[fuzzy]

Thank god I was just a worker bee. I used to work in a datacentre that housed many companies financial systems, from water cooled Joe90 mainframes, down to 'that black box over there'.
2 things I remember were a financial transfer link to outside with no modern encryption [X25] and an upgrade disaster. Our company had a guaranteed up time contract with someone, and an upgrade went pear shaped that I think was our problem [I didn't ask too much] I was told after 24 hours offline we were penalised £30000 per hour and this was 20 years ago!

[adam2]

TSB fiasco still ongoing.
A local pub has had to pay thousands of pounds of cash takings into the account of a customer, in order that said customer may pay an urgent bill on behalf of the pub.

Staff at another local business have not been paid on Friday because their employer banks with TSB. The employer states that ample money was in the company account, yet bank transfers for wages did not go ahead.

[johnhemming2]

Really successful companies that use tech are run by people who understand tech and can do it themselves.

eg Musk, Bezos

Those companies that don't have directors that really understand tech end up in situations like TSB.

It isn't actually how much money they spend, but how effectively they spend that money.

[woodburner]

That is an unsupportable postulation. The function of a boss of a company is to select the people able to do the job required, not to do the job themselves. Thare are also companies run by people who do understand the technical side, but as managers they are failures, consequently so are their companies.

[vtsnowedin]

That is an unsupportable postulation. The function of a boss of a company is to select the people able to do the job required, not to do the job themselves. Thare are also companies run by people who do understand the technical side, but as managers they are failures, consequently so are their companies.

If you don't understand all that a job requires, ie 'are able to do it yourself', how would you ever be able to select people capable of doing the job for you or the company?

[raspberry-blower]

More on the on going fiasco at TSB:

At the end of last week, TSB’s CEO Paul Preston announced he was drafting in a crack unit of IT technicians from IBM to “put things right� once and for all, no doubt at substantial extra cost to the bank. It was a bif snub for TSB’s parent company, Banco Sabadell, which oversaw the platform switch and claims (or at least claimed) to be a wizard in the art of migrating data. Indeed, TSB’s data migration was intended as a showcase to the world of Sabadell’s savoir faire in the IT department.

Initially, the IBM team had until the end of Saturday to fix the IT problems Sabadell helped to create, yet on Monday morning (April 30), the online system appeared to be no less broken than it was on Thursday when Preston succinctly told ITV News that the bank was “on its knees.� On Sunday night a spokesperson for the bank warned customers that Internet banking is still operating at around 50% capacity. “For every 10 customers who try to access our internet banking, five will be able to access this service.� And five won’t

The alleged crack team from IBM failing to come up with the goods. No surprise there, then

The time period to develop the new system and migrate TSB over to it was just 18 months,� one insider has complained to The Guardian. “I thought this was ridiculous. TSB people were saying that Sabadell had done this many times in Spain. But tiny Spanish local banks are not sprawling legacy systems� like Lloyds Banking Group’s.

To compound matters, the Sabadell development team did not have full access to or control of the system they were trying to migrate customer data and systems from, since Lloyds Banking Group was still the supplier. “This turned what was a super-hard systems job [into] a clusterfuck in the making,� the insider said.

If, as the insider alleges, the disastrous roll-out of Proteo4UK at TSB was foreseeable a long time ago, at least to many of the tech workers involved in the project, and yet the management went ahead anyway, adopting a hope-and-pray attitude, TSB, and by extension Sabadell, are likely to pay an even higher price for this epic IT disaster — especially if, as feared, some records are indeed lost or corrupted.

The wing-and-a-prayer attitude that is still very prevalent in Financial IT Management I see.

Don Quijones: UK Bank's IT disaster enters second week: Risk of contagion grows

[clv101]

That is an unsupportable postulation. The function of a boss of a company is to select the people able to do the job required, not to do the job themselves. Thare are also companies run by people who do understand the technical side, but as managers they are failures, consequently so are their companies.

If you don't understand all that a job requires, ie 'are able to do it yourself', how would you ever be able to select people capable of doing the job for you or the company?

That's bonkers - have you ever worked in a large company? Ever been involved in recruitment? More often than not, managers hire people to do roles they don't fully understand / are capable of themselves. In my career, more often than not I've had skills my management lacked - they couldn't do my job.

[adam2]

This is getting serious, with bonuses said to be at risk !

"In the absolute worst case scenario, TSB’s senior management may even lose their bonuses." By Don Quijones.

I can imagine a rush to push through bonus payments for PREVIOUS years, nothing to do with the present disaster which will be considered in the fullness of time. In time, large bonuses could be paid for this year due to the "outstanding way in which senior figures handled a crisis" But not perhaps just yet.

I do wonder if wonder the final great crash of a major bank, followed by contagion to other banks, and then TEOTWAWKI, might be precipitated by an IT failure rather than by underlying insolvency.

[raspberry-blower]

More from Naked Capitalism on the continuing fiasco at TSB:

On the IT testing:

And on Tuesday, before the Parliamentary hearings, a Wired story volunteered the notion that TSB had tested its systems over the migration weekend. Um, by the time you’ve migrated your production system, you are past the point of testing. Vlade poured cole water on this idea:


TSB testing on the weekend. Experts, huh?

You DON’T do testing of migration on the production system at the weekend of the migration. That would be even more idiotic than not doing it at all TBH, as it has all the potential to screw up the production, reduces time available for migration and any inevitable cock-ups (and hence also ability to roll back), and a lot of other problems I could think of.

The testing of migration is done on parallel test systems (that are, as far as practicable) perfect copies of production (some brave souls could use the DR site, which of course is ok until a problem strikes and you need that..) – so called dry-runs. You may do those on the weekends (to simulate the whole migration, so for example be able to detach/re-attach integrated systems etc., plus it takes a lot more of effort than usual working day), but not on the weekend of migration.

It’s expensive to have a full parallel system, which is why partial migrations are preferable. But to migrate a full system like TSBs w/o at least two full-population dry-runs (first one tends to find quite a few problems, second hopefully fixes them and establishes better runbook) is negligency par excellence.

Of course, then you also need to have a way of testing the dry-run – otherwise I can dry-run trivially (>/dev/null will always work… ) and claim sucess. In my experience, you might have done a dry-run on the weekend, and then spend the rest of the week testing and evaluating it.

This is getting worse every time there is a new revelation. There is a clear breakdown in command and now it is the blind leading the blind.

So what have TSB done?
If in doubt, hire a consultancy firm:

TSB has hired a law firm, Slaughter & May, to do an investigation. Help me. This is garbage in, garbage out, by design. As we’ve said before, when banks are in hot water and want to or are forced to come clean by regulators, they usually hire consulting firms. When they want to deliver carefully crafted cover-ups, they hire law firms because the work will be attorney-client privileged.

Pester said TSB would make everyone whole, or more specifically, “No customers end up out of pocket.� Good luck with that. How do you compensate someone for not being able to buy a house thanks to a records stuff up? MP Nicky Morgan also mentioned a customer dying…and didn’t quite make clear how big a role TSB-induced stress played.

If TSB does not have its problems almost entirely fixed in two weeks, the bank is toast. It will only be a matter of time before the damage reaches a tipping point. The only upside is if TSB turns out to have suffered as much self-inflicted damage as we think it has, this may finally lead regulators to get way more serious about the sorry state of bank systems.

Naked Capitalism: #TSBfail: MPs fail to land a punch as CEO Pester discredits himself on his own

[RenewableCandy]

That is an unsupportable postulation. The function of a boss of a company is to select the people able to do the job required, not to do the job themselves. Thare are also companies run by people who do understand the technical side, but as managers they are failures, consequently so are their companies.

If you don't understand all that a job requires, ie 'are able to do it yourself', how would you ever be able to select people capable of doing the job for you or the company?

Jolly good question! But trust - as in this case being able to trust someone's CV and references, is crucial to Business and everybody tends to forget that!