Client description:

Needing to deliver important health services, Berkshire-based charity Ostomy Lifestyle’s advice to stoma surgery patients offered on its 0800 telephone lines must be dispensed with 100% efficiency.

The medical conditions that lead to stoma surgery affect people of any age, gender, profession or race. The charity was established in 2007, in response to the lack of accessible information about ostomy issues.

The charity seeks to empower people, enabling them to make informed choices and to communicate frankly about all aspects of the surgery. “We are working to raise awareness of ostomy topics among the medical profession and the general public,” explains the charity’s Neil Basil. “Having a stoma should not be a barrier to leading a full life.”

He and his 16 colleagues offer impartial advice, and aim to inspire callers that barriers can be overcome.

The charity produces a regular newsletter, information sheets, and a website.

Advice given on the telephone complements existing healthcare services, to reduce the challenges that patients have to confront head-on.

“Our support services must be accessible to people of all backgrounds – so using our telephone system is perfect for this,” says Neil Basil. Someone can call needing basic stoma care advice and have their problems solved quickly, without needing to visit a hospital or local GP.”

Analysis and history of Gradwell’s VoIP service:

“We were still establishing Ostomy Lifestyle in 2007, and moving into new premises - so were looking for a solution that would adapt and grow with us,” explains Neil Basil. Gradwell’s advice and ability to provide what the charity required was well-documented and explained.

“Most of our staff work at our office in Reading – but some of our volunteers and Trustees benefit from the flexibility that working from home offers. We are now more likely to conduct meetings over the telephone, taking advantage of teleconferencing facilities.”

Helpline staff spend their time responding to queries from callers, as well as researching bowel and bladder topics and conducting training courses for nurses, support groups, and medical industry companies.

“VoIP worked very well from the start for us, once basic glitches were resolved,” comments Neil Basil.

Calls to Gradwell’s support team have been entirely satisfactory, he added.

“VoIP has given us the flexibility we need; it allows us capacity and functionality that we would otherwise not achieve.

“VoIP is a very technical service - so it needs a fair bit of involvement from our own staff; but we are learning all the time and Gradwell’s support staff resolve any problems quickly.”

Please see www.ostomylifestyle.org

Dear Customers,

Starting from Monday 19th January, core Gradwell services (email, web hosting and VoIP) suffered from repeated outages.  Each outage lasted several hours at a time, and all customers were affected.

The team have now completed all planned work to improve the stability of our services, and are now monitoring the situation to determine whether any further emergency work is required.  I apologise for the disruption caused by these outages, and want to reassure our customers that this continues to be my top priority.

In this blog post, my Technical Manager, Stuart and I want to explain to our customers what went wrong, and the changes we have made to put it right.

Ultimately, the root cause is technical equipment failure, but our design decisions have exacerbated the situation. We have received, and are grateful for, the large amount of customer feedback and have identified opportunities to improve our communication, operational process and the management of both our operations and customer support teams.

Some of these improvements have been implemented immediately, others will take a little longer. However, Gradwell is a well supported business, by customers, our staff (and even our bankers!) and there is no question that we cannot progress forward and work with customers to rectify circumstances as required and reconfirm our position as a leading provider of Internet Services to UK Small Business.

Stuart explains below in more detail what went wrong and how we’ve taken action to prevent it re-occurrence. We caused severe disruption to all, both customers and staff and we sincerely regret our failure to deliver excellent service in the last few weeks.

Yours sincerely

Peter Gradwell

Technical Overview

Gradwell relies heavily on server virtualisation, using the industry-leading VMWare ESX platform.  All of our services run on the ESX platform, and rely on virtualised storage running on dedicated networked storage servers.

On Monday 19th January, we began to experience multiple issues with two of our networked storage servers - a combination of faulty hardware and trying to work the networked storage servers too hard.  These simultaneous issues caused the networked storage servers to hang, which in turn caused all of the ESX servers to hang.  During each individual outage, we recovered services by resetting the networked storage servers and then resetting all of the ESX servers.  Regrettably, the process of rebooting all of these servers in the correct sequence took several hours to complete from start to finish, causing our services to be unavailable for extended periods of time.

Gradwell has addressed these issues by:

1. The faulty hardware has been either replaced or removed from service.
2. We have purchased and installed five additional HP storage servers partly to replace the failed hardware, partly to reduce the demands on the remaining storage servers, and partly to ensure better redundancy against future failures.

We apologise for the length of time it has taken to complete the work to address these issues.  This was due to the lead times for purchasing, testing and installing the new storage servers.

We apologise for the inconvenience caused by these outages, and want to reassure our customers that we will be continuing to improve to our infrastructure to ensure we do not have future outages like this.

If you’d like the full technical details of the faults and how they have been addressed, please read on for more details.

Read the rest of this entry »