Rackspace Runs Short Of Cloud Files Storage In LON Region

Updated Rackspace is running out of, er, space. At least as far as a portion of its Cloud Files customers served by the LON (London) region are concerned.

According to the company, problems began on November 17 at 1545 UTC, as some customers experienced 503 errors when attempting to access their files.

On November 24 – yes, a week later – the company said its engineers increased storage capacity in the region. However, problems have continued this week. A Register reader, working for a SaaS biz that uses Rackspace's Cloud Files product, told us: "Since Monday we've been seeing a big jump in errors as they tried to upload files, and have had to delay a release because we can't publish assets to their CDN.

"To be clear, some operations succeed, but we're seeing between 1 and 6 percent fail in any given hour, and it's not getting any better."

By November 28, Rackspace told customers - in a communication seen by The Reg - that its engineers continued working and introducing additional storage "at a deliberate pace" to avoid worsening the situation. It said "customers should be able to access already uploaded content but may experience issues processing new uploads or delete actions."

As of November 29, more storage was added – and will continue to be added – to, in the words of Rackspace, "restore the environment's health."

The company said: "Engineers continue to recommend that customers retry upload and delete actions as needed as this is the only workaround currently available."

• How much to clean up a ransomware infection? For Rackspace, about $11M
• Facing a 30% price rise to park servers in a colo? Blame AI
• Rackspace datacenter infrastructure took 12-hour nap in London, Sydney, Hong Kong
• Rackspace racks up job cuts amid market downturn and talk of offshoring

The embarrassing event is yet another reminder of the fallibility of cloud storage, although unlike the problems reported with Google Drive earlier this week, Rackspace has at least admitted that, yes, it isn't your imagination. Something has gone wrong.

However, the question remains – how did this happen? Storage needs are relatively easy to predict when paying customers are involved. Cloud Files itself is powered by OpenStack. Rackspace says of the product: "Whether your storage needs are modest - or monumental - you enjoy built-in redundancy."

Unless, it seems, the disk fills up.

The Register contacted Rackspace for comment on the incident and will update this piece should the company respond. ®

Updated at 16.32 UTC on December 1, 2023, to add:

A Rackspace spokesperson told us that no data has been lost in the incident and the issue was related to LON Cloud Files customers uploading new objects.

"Actions regarding existing objects are not affected by this issue."

As for how the company failed to plan for storage running out, the spokesperson added: "Reasonably foreseeable storage needs were forecasted and accounted for. High demand for Rackspace's Object Storage services significantly exceeded those forecasts."