Agile, Scrum, and Kanban were fancy in the 2000s. Nowadays, almost everyone in software development either tried (and dropped) them or is a staunch devotee. But today’s story has some peculiarities that make our experience with Agile particularly interesting in the 2020s. Not only did we implement it in telecom – one of the most conservative information technology industries – it also all started in the late 2000s, exactly when Agile delivery in telecom was new and fancy.
So now, a decade later, we’re ready to share our implementation lessons. More than that, we’ll share our real trench (and console) experience: from MR24 to MR104. Whether you’re in telecom, fintech, or data processing, you might find a few of these moments relevant to your own new Agile journey.
Managing Those Last-Minute Change Requests (and Improving the Efficiency of the Release Efforts)
By 2008, PortaOne had over 300 regular clients and was using the classic software release scheme: one big release every 6–9 months, followed by minor updates. We also had our ongoing “every day implementing what seems impossible; miracles require three-day notice” corporate mentality. That combo meant we sometimes had dozens of feature requests filed weeks (or even days!) before the planned release date. That’s when last-minute change requests (LMCRs) started to overwhelm our business. Limiting those requests was not an option: it would hurt our sales strategy and lead to unhappy customers.
Release discipline was another issue. Tackling many features at the same time made it difficult for our release managers to set accurate time frames. The outcome? Either heavy overtime and the need for testing the “improvement to the improvement, which is actually an earlier improvement requested by another client” (leading to dev team burnout), or deadline shifts with our clients. Obviously, both were painful. We realized we had to change our underlying approach if we wanted to keep delivering software enhancements to our customers in a sustainable way.
The Agile Book and AgileZen
“I started scouting for literature that could explain practical aspects of Agile methodology, besides merely quoting the Agile Manifesto,” recalls Andriy Zhylenko. One such book was the second edition of Agile Project Management: Creating Innovative Products by Jim Robert Highsmith, the executive consultant at ThoughtWorks. Andriy bought two copies: one for his Seattle home; another to send to the PortaOne development team in Chernihiv.
Andriy also discovered AgileZen — a company that created online Kanban boards. (This was several years before Trello launched.) AgileZen ended up becoming a part of Broadcom after a series of mergers and acquisitions throughout the 2010s and was ultimately lost among all that corporate conundrum. Andriy wrote the Perl API for AgileZen, enabling integration with the programming language we used for our own software and RequestTracker support ticketing system.
Then PortaOne started experimenting with scrum boards as part of our Agile delivery in telecom cycle. Our biggest challenge was the “yes, but” stigma: while most agreed Agile was great for small web projects, many thought it was not applicable to a “serious” telecom business. And that “many” included our own developers and telecom executives.
“To the Islands!” or Why Shifting Perspective Is Important for Agile Delivery in Telecom
Sometimes you have to reboot the system in order to apply changes to its configuration. That’s true even in our digital age of seamless handover and devices that can exist for months and years without any reboot. In 2011, we felt our team needed a reboot. In theory, every engineer knew how Agile worked and why it was a framework we should go with. But we needed to change our daily habits and routines — that’s the difference between real change and “innovation theater”.
Corporate outings was a worldwide trend right at that time. Many of our developers either hadn’t been abroad or had gone only on family vacations to conventional “all-inclusive” destinations. In Ukraine, like with most of the European working class, those were in Egypt and Turkey (this was before tourist operators turned Thailand into “a Turkey that has 🐒 monkeys, 🐍 cobras, and a longer flight”). This “home sitting” lifestyle was in contrast to the frequent training junkets our sales teams and support engineers were going on – often in places you couldn’t buy a tour to.
So, to take our revenge on sales, all the team leads and PMs went on a joint Agile vacation to Thailand. Andriy distributed multiple copies of Highsmith’s book, somebody took out playing cards, and the airline offered free refills on the red wine. By the end of the 11-hour plane flight, even those who weren’t that in to reading the book knew its contents. “We looked funny on a beach,” recalls Mike Kiddik, our Product Development Officer. From a distance it appeared like a library discussion club.
By the end of the joint vacation, everybody loved Scrum, Kanban, and the Agile Manifesto. Especially the younger software engineers. For some, it was their first trip abroad.
And Our First Sprint for Agile Delivery in Telecom Was… Whoops, There Was None
Winston Churchill (or was it the unnamed Free Printable Country Hill Cottage series author?) once said that “happiness is a way of life, not a destination.” And, as it happens, our transformation into an Agile team did not go according to schedule. Instead, it was a number of slow, steady independent processes across various teams and individuals that steadfastly produced tangible outcomes. It also created a work ethic that survived to this decade. People did come and go, but many stayed and grew with us.
But yes – we invited people to “take new approaches” and made it evident that adopting aspects of Scrum into daily coding routines is good. We also showed that it positively impacts the yearly bonus and promotion chances. So, everybody started innovating, except for the laziest who came in at the very end, adopted the best practices from the experimentation wave, improved them slightly, and started treating them as their own. That’s the Three Virtues in action.
MR24: The Agile (2011)
Nevertheless, while the Agile transformation won its way through our team peacefully and quietly, we still had to “tell” our clients. That’s why management decided to name MR24 “PortaOne’s first Agile release”. It was important to communicate our new release philosophy to our existing customer base and to upcoming prospects.
From that point forward, our salespeople and customer success managers had seven weeks to discuss any code changes with the PMs on a client’s side. And if a feature required more than one Agile cycle, we were able to make a clear forecast: “This feature could require another 7, 14, 21… weeks”.
Not everyone was happy with this New World Order (read: those “loud-yelling last minute scope-changers”). But it only took a few MRs for clients to discover the benefits and peace of mind of this new system. In the end, our telecom-blended Agile landed well with customers, too.
The Life of MR: 7 Builds, 7 Weeks, and 2 Years
Rephrasing George Orwell: some MRs are “more equal” than others. We have two MR types that differ by release strategy and backwards compatibility. Understanding this basic difference will greatly help you grasp the PortaOne product release process.
A Leap Forward (LP) Release
In terms of naming and numbering: LP is any MR whose number cannot be cleanly divided by 5. (For example, MR81, 82, 83, or 84.) Usually, LPs are usually supported until the next Long Term Support (LTS) release comes out, which is roughly one year. In other words: MR81, 82, 83, and 84 have our support guarantee until MR85 comes out. Don’t confuse LP MRs with “builds” of LTS, which contain bug fixes and critical patches. LPs have their builds as well. However, the number is usually limited to two.
A Long Term Support (LTS) Release
In terms of our naming conventions, LTS is any MR whose number can be divided by 5: MR80, MR85, MR90 and so on. This began with MR25 (i.e., the next MR after the first “Agile Release”). Primarily, LTS releases have an extended guaranteed support term of two years since the release date of build 0. (See our explainer on builds below.)
The key differentiator of LTS is backwards compatibility, which covers data models. You can jump forward as many LTS as you want. But if you want a safe rollback, you can only update from MR80, MR81, etc., to MR85.
Why is safe rollback important? Here’s a brief client story. We have a client in the US who was migrating from MR60 to MR75. It turned out their existing self-care portal did not support the new API of PortaSwitch. So, we rolled the affected clients back to MR70, while the rest of the customer base was already on MR75.
Take this example: MR88 or MR87 are backwards compatible with MR85. That means all the API methods, data, metadata structures, and support of third-party software will stay the same. We might even ensure this between several LTS releases, as was the case for most of the features between releases MR75 to MR90. However, we don’t guarantee it.
A Build
We use the term “build” in its conventional meaning. LTS and LP releases both contain new features, the difference between them being the “essentiality” of these features. Builds usually do not contain any new functionality. They only deliver security patches, bug fixes, and improvements of existing features.
Here’s a Zendesk example from MR86: while MR86 build 0 became available on July 6, 2020, build 1 of the same MR (solving the issue of “No informative message while trying to re-assign Zendesk organization to another customer”) arrived two months later, on September 11 (sad date, indeed).
Each LTS usually consists of seven builds (0 to 6), spanning for nearly a year after we publish build 0. Each LP usually consists of two builds, and we usually publish build 1 three to five months after build 0.
Agile Delivery Decade — the Retro
So how do we keep track of all the lessons we learned? We use an Agile practice that we discovered via the Retromat knowledge repository. Retromat is one of the many resources specializing in helping Agile teams discover various techniques for conducting retrospective meetings (or simply “retros”) — an essential element of the Agile delivery cycle. The practice is called “Back to the Future”. To undergo it, you ask three questions: 1) What do you definitely keep? 2) What do you skip or do differently this time? 3) What were your biggest learnings? We compile the answers from our dev team members via Slack voting.
Keep: The Release Discipline — (Arguably) the Most Valuable Lesson for Agile Delivery in Telecom
“For 10 years since MR24,” says Andriy Zhylenko, “we haven’t had a single delayed release.”
Okay, he does add one thing: “To be totally frank, we shifted several times to January 4th from New Year’s eve.” (If you didn’t guess, this is the time when everyone in Ukraine drinks sparkling wine, watches YouTube greetings from some political leader, and eats Stolichny salad, which post-Soviet Ukrainians prefer calling 🇫🇷 “Olivier” in a French manner, as if the naming makes it really French.) “All other times it works strictly to the clock like the 🇩🇪Berlin U-Bahn,” Andriy closes the thought.
So since Agile came fully into force, it’s always on time. This enables transparent client communication: say a feature does not get to MR89, you wait seven weeks and it’s in MR90. Most of the competition does “a release per year” — still “Agile”, maybe, given traditional telco dev cycle duration. But it’s “not Agile enough” to enable fast adaptation to the ever-changing business landscape.
Keep: “One Feature at a Time” and the Features Curfew
The features curfew stems out of the release discipline, and that’s actually what enables it. It might be a non-canonical approach to Agile, but it’s the last principle of the Agile Manifesto. That is: “at regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” So, we reflected and adjusted our behavior to have green🍀 (weeks 1—2), yellow🌕 (weeks 3—4) and red🌹 (weeks 5—6) zones.
As is obvious from that color scheme: the green zone is when any scope changes are ok. The yellow zone is when we can change features that were selected during “green” weeks. And red is when we finalize testing and documentation for the upcoming MR. No scope changes are allowed. Once again: this is a telecom specific rework, and it might limit the 100% Agile flexibility. But it works, and it makes life easier for both our dev team and the clients.
“One feature at a time” stems from the concepts of “continuous delivery” and “continuous integration”. We try to preserve optimal feature v. component balance by creating cross-functional dev teams and taking both feature and component improvements into the MR roadmap. Luckily, agile delivery in telecom allows sufficient choice of both elements of our software architecture.
Keep: “Take It (Maturely) Easy” Attitude
Agile delivery in telecom stuck well with PortaOne without the loud “declarations” and “transformation committees”. Why? Because we did not impose it on people. Things that were destined to work worked; other things — we moved past them. Please avoid misinterpreting this as an invitation to be loose or irresponsible. This is really about avoiding any tension that can be avoided.
Skip: Vague Feature Specifications
If we could send an SMS from our 2021 selves to 2011 it would be this: “Begin your Agile journey by creating a workflow for Agile specifications”. We weren’t that smart in 2011. We were hoping for “Scrum magic” to help us design around the customer’s needs. The painful truth is that the customer does not (and should not) know the code-making process or the architecture underlying it. That’s the job of the product delivery team.
The big difference between the big “mass market” coding (Google, Apple, Microsoft or Facebook) or web-based startups (Notion, Loom, Trello, Asana) and coding in telecom is that, besides the front-end frameworks, you have to work with huge amounts of back-end architectures. These can be a decade old, but you still have to implement them to support all the technologies your customers will need. Imagine making a VoIP call only to hear, “Sorry, this call cannot be completed because the technology of the destination device is outdated”. When launching for telecom, plan accordingly.
Skip: The Great MR Gap
In the mass market, coding customers go after the latest versions and brag about being beta-testers. Things work vice-versa in B2B, especially in VoIP telecom. Here, dev teams are loyal adepts of the Three Virtues. Why change something that just works? Especially when it requires additional engineering efforts and OpEX.
At first, we were proud to have an “eternal” software and discover clients who are still on MR10 in 2014, “and it works”. Then we started to realize why garment manufacturers make jeans that don’t last. How can you sell your “most recent innovation” to a client who is more than satisfied with MR10 a decade after the release? And there’s another “MR Gap” situation: clients arrive to our customer conference, listen to presentations containing new features and feel disappointed because they won’t be able to roll the MR that contains the various exciting new features for another X years. (Say because they have a lack of personnel to perform the data migration or lack the spare server capacity.)
Luckily, not every client’s mind works this way. But even those interested in our product innovation had concerns regarding the migration to new releases and the risks it posed. It took us several years to come up with a solution and then implement it. (Stay tuned for our next story to hear how we cured the Great MR Gap with Dual Version.)
Learn: Programmers Should Talk to Customers to Enable Agile Delivery in Telecom
We started implementing design sprints and empathy interviews back in 2019. Since then, these design thinking tools, as well as value proposition canvas etc., became our regular routine. It was hard to make our developers sit down and do nothing or make them “draw the nonsense” on a piece of paper. However, it slowly took off and now various design thinking ideation sessions are a part of our development process.
Andriy Zhylenko recalls: “Back in the Telenor times, I had a colleague who had a previous job in IKEA as a sysadmin.” It was back in those distant times when IKEA actually hired locals to manage its IT systems. That dude told “crazy and scary” stories about how management “made” him work periodically in the actual store, selling furniture, sofas, and lamps. “It all looked strange back then,” he says.
Two decades later, this practice doesn’t seem strange at all. How can people write good code without knowing and seeing whom they are actually coding for?
