Toyota Servers Ran Out Of Storage, Crashed Production At 14 Plants In Japan

Toyota has revealed a server running out of disk space after botched maintenance was the cause of an outage that forced it to shut down 14 manufacturing plants across Japan last week.

“The system malfunction was caused by the unavailability of some multiple servers (sic) that process parts orders,” states a company announcement posted on Wednesday.

Those servers went down after regular maintenance work was performed on August 27. But someone botched that job.

"During the maintenance procedure, data that had accumulated in the database was deleted and organized, and an error occurred due to insufficient disk space, causing the system to stop," Toyota’s announcement explains.

"Since these servers were running on the same system, a similar failure occurred in the backup function, and a switchover could not be made. We think that means Toyota's production and backup servers are identical, so failover failed because the backup rig was also out of space."

Whatever the exact circumstances, the mess led to the plug being pulled on vehicle production.

• Microsoft admits slim staff and broken automation contributed to Azure outage
• UK flights disrupted by 'technical issue' with air traffic computer system
• Bank of Ireland outage sees customers queue for 'free' cash – or maybe any cash
• Voyager 2 found! Deep Space Network hears it chattering in space

The fix?

"The system was restored after the data was transferred to a server with a larger capacity on August 29," Toyota's statement reveals. But by then around 36 hours of production time had been lost.

The Register feels on solid ground suggesting that Toyota – a master of just-in-time manufacturing – has a recovery time objective rather shorter than 36 hours.

The Japanese auto giant's techies have determined the root cause of the issue, and developed countermeasures to prevent its recurrence. The Corolla-maker has pledged to "review our maintenance work procedures and strengthen our efforts to prevent recurrence in order to deliver as many cars as possible to our customers as quickly as possible.”

A small piece of good news to emerge from the mess is that Toyota's original assertion it was not the victim of a cyber attack has proved correct. ®