24/7 Standby Service – What Does It Mean?
A 24/7 standby service is a critical offering that ensures a rapid response to critical issues at any time, regardless of the time of day or night. The primary goal of this service is to provide the client with peace of mind that, should a critical system failure occur, there is someone ready to respond immediately. The standby service operates by having the on-call developer always carry a dedicated mobile phone that the client can call if they identify a Priority 1 (P1) incident. This way, we guarantee that problem analysis will begin as quickly as possible, and ideally, a solution will be implemented relatively soon, significantly reducing the risk of prolonged system downtime.
Interview with Stefan Bolesnikov
We spoke with Stefan Bolesnikov, Delivery Manager, about this interesting topic and the unique experience of providing a 24/7 standby service for one of our key customers.
Why was this service important for the client?
At the time the standby service was introduced, Incision was in a phase where they needed reliable support and minimal downtime to guarantee high uptime to their clients. For some of their clients, this was a key requirement. That’s why they needed the security of a 24/7 standby service. As their main development partner, we provided this service because it is a standard part of the suite of services that Levi9 offers.
What were the biggest challenges you anticipated at the beginning, and how did the situation unfold?
The biggest challenge was the fear of a call at three in the morning, which nobody wants, and the question of whether the on-call developer would be able to resolve the problem. At the start, developers organized internal Knowledge Sharing sessions for each other to prepare for providing this service, and that helped significantly.
Great reassurance also came from certain senior colleagues who volunteered to be available at any time if the on-call developer couldn’t resolve the issue alone and had exhausted all options they could think of. As a last resort option, even the Delivery Director offered to be called, since he had previously been a software architect on the project. And I, as a Delivery Manager, expressed my willingness to be called at absolutely any time if I could help in any way, even just as moral support or a “rubber duck”.
Thanks to all these support mechanisms, the burden became lighter. We felt most secure during working hours or even at team buildings, because we knew we would all together shift our focus to solving the problem. In the end, after 15 months, the phone never rang, not even once, with a Priority 1 issue.
In your opinion, what contributed most to there being no P1 incidents for 15 months?
What contributed most was that we developed the entire system for the client and have complete technical ownership, so we know it well. Although not everyone worked on all parts of the system, most concepts are familiar to everyone.
Everyone took the consequences of a potential P1 call seriously and professionally approached practices that reduce risk. One of the key practices was banning production deployments on Fridays, and on other days only until 2 PM, so we would have enough time to resolve any issues that might have come up by the end of the workday. We also carefully planned the timing for releasing riskier changes. And of course, the focus on delivering quality that all developers have, combined with these preventive measures, led to the phone never ringing once. And yes, we checked every week whether the phone was working properly.
Project tradition
The nicest tradition that emerged from this service was taking photos during each phone handover. The previous on-call person takes a picture with the next one, shaking hands and symbolically handing over the phone. This way, we created an unbroken chain of on-call duty pictures. With on-call shifts lasting one week, about 65 photos have been collected in total. Even when someone is on call for two weeks in a row, they traditionally take a photo handing the phone to themselves.
In this article:
