A good product team often resembles a living organism. And its ability to self-reflect, its “corporate metacognition,” if you will, is what helps it survive and adapt to ever-changing market realities. But the big problem with metacognition is finding time to actually perform it. PortaOne, for example, is an engineer-centered and engineer-run business. And that’s why various feature requests, code reviews, and other daily operational trenchworks regularly flood over us. Add Kodawari perfectionism to the mix (you’ll soon get what that has to do with our business), and you can empathize with our “no time to stand and stare” tune.
Still, we were able to block a few time slots on our management’s calendar and actually get them to “tell their story.” So let’s start at the beginning – that’s 1997 (yes, it’s the same core team) – and end with our most recent trials, errors, and successes. The organization is now over 300 employees, some of whom are younger than PortaOne itself. And, to be honest, the closer this story got to “You Are Here,” the more intense our “founding fathers” became in telling it.
The Story Split
So we’re splitting up the autobiography of PortaOne: the “safe” remote past, and the vibrant, sometimes turbulent, “not so remote” past. This part, Part 1, deals with the early 00s and the one question we always hear from new customers and employees. That is: “What the hell is MR?”
We’ll answer this question, and we’ll also show you a few other early events, like how the Kodawari philosophy impacted our software license type, how PortaSwitch started off with FreeBSD and then switched to Linux, and what Oracle (the corporation, not any kind of prophetic deity) has to do with all this.
MR Stands for “Maintenance Release”
Software versions are among the many relics of the 1990s. The “Major.Minor.Patch” semantic versioning appeared by then and is still in use today. In the 2000s, Steve Jobs created the fashion for fancy code names with his Felidae-alpestrine theme for macOS (Cheetah, Snow Leopard, Yosemite). Then, Google followed suit with its confectionery Android theme (Cupcake, Donut, Jelly Bean).
The big difference in how versions are handled now compared to the 1990s is the business model. Back then, everyone followed the Microsoft method, in which each new version (e.g., Windows 95, Windows 98, Windows Me) was purchased separately.
MR and the Perpetual Unlimited License
Being adepts of Kodawari, the PortaOne “founding fathers” wanted to emphasize right from the company’s inception that our software is “the long game.” That, at that time, meant a perpetual unlimited license. (BTW, apologies to our feminist readers for the patriarchal language — let’s remember we are speaking about Eastern Europe in the late 1990s.) Since then, two significant developments have occurred: (1) the lifetime license stopped being the long game with the advent of SaaS, and (2) several women joined the ranks of our top executives (currently two, to be precise).
So, we decided to abandon the classic “version” scheme. Now, we liken purchasing a software license from PortaOne to buying a sapling when you want a tree: pay once, and then (with some support and nourishment) it grows big and tall. All those new branches and all that summer fruit is yours to keep. And the MRs? Those are merely the notches that mark its growth.
Kodawari?
So, here is Kodawari in one paragraph, for those who don’t have time to follow links. Kodawari is a mental framework that comes from Japan. Very briefly, it’s being a perfectionist in what you love to do: be it a gardener growing melons, a chef making sushi, a katana blacksmith forging blades, or a telecom entrepreneur selling a softswitch with convergent billing.
What Does Kodawari Have to Do with Our MRs?
One of our first clients back in the 2000s was Ed Masaitis and his ApolloPhone. At the time, their core team was located in St. Petersburg. ApolloPhone changed their business model several times throughout their two-decade-long journey, and now they serve customers on all continents. They started with calling cards, then a Vonage-like “replace your phone with VoIP” solution. Now they have a cloud UCaaS solution. An endless evolution indeed… but their PortaOne license still works. Plus, we never charged them another penny for expanding it to cover their new business models.
Perpetual unlimited is better understood now, in the era of post-SaaS and customer lifetime value. But in our story, we’re still in the early 2000s, remember? So how were we to explain to our customers and prospects that this is radically different from Windows 95, 98, or whatever switch solution they had at that time?
Forget the Upgrade Fees!
Our answer: “maintenance release” is our long-term commitment to making MR25, MR75, or MR95 (coming this autumn) available to any customer who has already paid for the license (or signed up for PortaSwitch SaaS) as a part of their regular support contract. This includes all the new functionality and every component added to PortaSwitch. No “module fee.” No “upgrade fee.” Ever.
This also made the procurement process infinitely easier on the customer’s side. Anyone who ever dealt with a “typical” sales order for enterprise software with myriad options will understand. A mere human can’t grasp how each option will affect price, how they interrelate, or which ones you really need.
Core Product (i.e., PortaSwitch + PortaBilling) License Is Perpetual; Service Agreements and iPaaS Add-ons Are Not
“So,” you ask as an attentive and shrewd reader. “Does your business model always depend on new sales?” The answer is no. We would not have survived and thrived for over two decades on a business model like that. Instead, people buy our software, and then (likely after a few trials, errors, and successes) hire us to manage and support it.
There are thousands of things that can go wrong in the telecom business. Who could support your business’s core systems (switch, billing) better than the people who wrote them? Over twenty-plus years, we have handpicked and educated some of the best support engineers in the industry. Some then move into writing our code. Some prefer to continue “making all the systems work.” And that is our primary revenue driver, at least beyond software sales.
Hire Vendors to Do Integrations or Start Treating Integrations as Your Own Product! Please, Please, Please!
Part 2 of this “perpetual v. not perpetual” story is coming soon in a future post. So without spoiling too much about iPaaS, Workflows, and how we started slowly switching our key modules to cloud and microservices (using Kubernetes as the workhorse), here are the basics you need to know.
Somewhere around 2009, we realized that SaaS disruption is happening in telecom. And it culminated in 2016 with the Twilio IPO. We started receiving hundreds (literally) of integration requests from customers all around the globe. Why? Because all those great and wonderful telecom startups each launched their own little API for something.
Seeing the Whole Picture
At first, we tried to copy the best solutions, but we are neither Cisco nor Google, and we couldn’t invest millions into buying out the brightest challengers. So we decided to explain to our customers that they had a choice. They could (1) pay us extra to hire additional outside developers, (2) wait until we reach their issue in the backlog, or (3) hire and manage their own “integration team.”
That last approach was one of the most painful experiences in our entire corporate history. When your client hires a freelancer for $10/hour to perform a softswitch and billing integration, that freelancer is fated to take shortcuts and ignore the holistic picture. It has nothing to do with professionalism or commitment, it’s just reality. A limited task scope will always limit the perspective.
Staging Systems and Custom Patches
We decided we should help. At around MR12, we started to provide staging systems (a fancy name for “developer’s sandbox”). Then, with MR19, we added the tool to maintain custom patches (third-party changes to the source code – sometimes a band-aid and sometimes a significant transformation). These patches ensure that modified client code will “survive” the next update. The system automatically verifies patches for compatibility with the new release, then “applies” them to the latest version of the code. Therefore, in normal circumstances (i.e., no conflicts discovered), this happens with zero involvement of the admins or developers.
This way, your proprietary improvements to our code don’t get lost when the next MR comes out. Even if the developers who initially did the change are no longer available or don’t remember what exactly they did! (Though clients using custom patches do have to monitor our documentation and update their code.) New Jersey-based Telinta was particularly pleased about this innovation back in 2010.
What Does a Custom-Build Hypercar Have to Do with MRs and a Softswitch?
Still, it left two problems unsolved: (1) in a typical enterprise application everything is interconnected, and one wrong line of code can damage it all, and (2) writing code for just one system is costly (look up the MSRP for a custom-build Ferrari). And one more: deploying the code developed by a third party on your system is risky.
iPaaS — the Kodawari Marketplace for the Code (Finally)
So, we thought. What if we could create a system where our customers could gain access to new integrations? What if they could do it quickly and without risk? And what if those willing to do software development were rewarded for their efforts and could recoup their development costs? Enter: the iPaaS Marketplace.
This is how we keep the license to our core product perpetual while ensuring the quality of all integrations. Using Kubernetes containers, we are managing to separate our code and any third-party add-ons. The process is ongoing, and it takes time and resources. But the (successful) end of this transition is in sight. By launching iPaaS Marketplace, we have enabled our customers to monetize their improvements to PortaSwitch via revenue sharing.
MR1 — MR19: FreeBSD (2001–2008)
At first, we thought this topic deserved a separate story from the birth of PortaOne and the first MR. Then it turned out we have enough material for a series. (Not for Netflix, so far 😉, but you never know.) So here, now, we’re only going to explain our initial version of our versioning (pun intended). The story of our “founding fathers,” how they met, and what the city of Prague has to do with it will come soon. As will stories about how we implemented Agile, created Dual Version, envisioned iPaaS, and did an awful lot of other great things.
FreeBSD
In 2002 we completely rewrote our code, keeping only FreeBSD as the OS and using the other three parts of LAMP (Linux, Apache, MySQL, Perl), the “gold standard” of Internet-related software development in the 2000s. Why FAMP instead of LAMP? At that time, FreeBSD (a variant of Unix) was particularly popular in telecom. This was no surprise, as this operating system did partially derive from AT&T Research Unix.
After establishing our R&D office back in our hometown of Kyiv, we hired many people from the Ukrainian FreeBSD community. (Fun fact: in summer 2020, a client from Bosnia and Herzegovina approached us, saying, “Your system works well; we just wanted to ask for a minor improvement.” It turned out that they were still running FreeBSD-based MR19 from 2007 on our perpetual unlimited license.)
PortaBilling + PortaSIP = PortaSwitch
In the early 2000s (MR7), we added a new product: VoIP softswitch. (That’s a fancy name for something that intelligently connects calls and doesn’t require a connection to physical copper wires.) We called it PortaSIP. Obviously, we did our best to sell both products (the billing and the switch) together. It made sense, since the real-time connection between them allowed the operator to control all aspects of calling and avoid situations where end customers spend more than they have. But having to use “plus” each time we were mentioning PortaBilling and PortaSIP was tiring. It also decreased our sales efficiency. Hence, PortaSwitch appeared.
Here’s a short story that highlights the relevance of billing and switch integration. In the 2000s, we received a call from a customer in Montreal saying, “Your billing lost our money!” After an investigation, we discovered that their VoIP gateway didn’t send all the data to our billing for a while because of a bug in that third-party gateway. Days passed where multiple people were making calls that were supposed to be charged $0.05–$0.50/minute. Grrr. We made sure this wouldn’t happen with telcos who use PortaSwitch.
Procinctus
Being an integrated switch-plus-billing solution gave us unique convergence opportunities and allowed us to compete with giants like Cisco and Alcatel Lucent. We fell in love with the fancy Apple naming (think back to felidae-alpestrine), and decided to invent our own system. MR11 came with a cluster functionality, and so we called it PortaSwitch Procinctus (Latin: “ready for the battle”) — a product bundle that provided high-availability and scalability for every component, making good-old Latin and its gladiator flare “our thing.”
MR19 — MR21: Oracularius (2007); Switching to Linux
By 2006 PortaOne had over 100 telecom operators (in 40+ countries) using our system to run various VoIP concerns. Some were traditional (like calling cards) and some were more like creative “hacks” (like Lithuanian-issued SIM cards for use while a person is using their phone in a foreign country). It was evident that we succeeded in building a sustainable mature business, and it was time to grow fast.
Oracularius
Well, our Procinctus had an Achilles heel: a single “master” instance of MySQL database (this was before Oracle acquired MySQL in 2009). Slowly moving from smaller telecom operators (the Wild West pioneers of communications) into the segment of large, established operators, we started getting more and more inquiries to have complete high-availability use of PortaSwitch and PortaBilling with the Oracle RAC (Real-time Application Cluster) database. In 2008 we decided we were finally ready for Oracle, and so we released PortaBilling Oracularius. (We had moved into a mood for marrying Greek divine ancestry with our Latin gladiators and legionnaires.)
A variety of large operators worldwide (e.g., Vodafone, Telekom Malaysia) have deployed PortaBilling Oracularius since then. But one of the brightest moments came when Bezeq deployed it. Bezeq is a national operator of Israel, a country historically strong in technology and a homeland for many established telco billing players🥴, such as Amdocs or Mind CTI.
The Unbreakable
Oracle had its own Oracle Enterprise Linux distribution by that time. Okay, to be precise: Red Hat had it, and Oracle bought it and used their distro. The Oracle database would not run on FreeBSD, and even connecting to it from FreeBSD was painful. It became evident that a task with the complexity of a multimillion-dollar corporation would eventually overwhelm us. So, it was time to say goodbye to old good FreeBSD and switch to Linux. We are still on it today, more than a decade since.
In 2010 Oracle improved what they got from Red Hat with Unbreakable Enterprise Kernel, dubbed “Oracle Unbreakable Linux.” Their purpose was to “address the needs of customers running demanding software such as Oracle Database on large-scale systems.”
This totally resonated with us, and we started using this distro for PortaSwitch.
The Big Reinstall
The migration from FreeBSD to Linux was not an easy task for our existing clients. In the 2000s, changing the operating system on a server running your core business systems (that is, softswitch and billing) was like changing the tires on a car that was still moving. Pre-Kubernetes, it: (1) took hours before the server was completely functional, (2) was risky — one mistake “bricked” the server, and (3) typically had to be done “on-site.” That meant our engineers had to travel all across the globe (not that they were against that) or our customers had to hire local experts.
The outcome: most software vendors solved this conundrum by holding on to their existing OS as long as they could. When contemplating the switch from FreeBSD to Linux, we knew: this was gonna hurt. But we also knew that if we didn’t just do it, it would eventually hurt just as much. And that hurt would arrive again and again, with every new version of OS.
So How Do We Change Those Tires?
So, PortaOne decided this:
- Have a “safe” way to update the OS (and roll back if required). Use a spare partition for the new version, so it only takes a reboot to “turn it on.”
- Package OS + software like MySQL and our code into an “image.” Then use the same “verified and tested” MR image for all customers.
- Use a centralized configuration server to automatically populate configuration settings into OS and all applications, thus reducing hours of manual config file editing (and the risk of “oops!”) to almost zero. We’d already developed our own configurator by that time, but that’s a separate story. Stay tuned.
So our promise to our existing clients was this: “Go through the pain once, gain peaceful administration for decades.”
We provided automated install disks for new Linux versions, but our customers still had to manually insert and launch the install. And they had to shift the workload between servers, pretty much like in that tire-change video above. But it ultimately worked: since then, we’ve gone through many changes in the Linux version, and nobody’s really noticed. Contrast that with all the scary stories from the market.
Do you know who else introduced a similar update cycle around that time? Apple and Google. Seeing that, we realized that PortaOne was ready for another giant leap forward. And this time, it would be our communication with our customers, not our code. Watch this space for that story.
