As you move your infrastructure to the cloud, it’s important to consider how to not become beholden to a single cloud vendor. In other words, to attain cloud neutrality. I’m afraid that I don’t have all the answers on how to do this, but I do have some thoughts. In any case, I recommend that you think about it now, before you throw a lot of money into one vendor’s vision of the “cloud.”
The Sad History of Neutrality (besides Switzerland)
Will you think less of me if I admit that I started in IT in the early ’90s? Back then, one concern of IT managers was not being too tied in to a single technology vendor. Once you adopted a platform, you were making a long term decision. If you were IBM AS400 today, you were likely going to be IBM AS400 for a long time.
Managers were rightly worried about being locked into platforms that might get super expensive, wouldn’t keep up with the times or, even worse, get totally dropped by their vendors.
However, a couple of saviors came along to save the day! Unix and SQL. Both were platform-neutral technologies. They were built on global standards so, for example, if your SQL code ran on one database it would run on all the others.
Except, it didn’t.
It turned out that neutrality was good for customers but bad for vendors. While tech vendor PR campaigns jumped on the neutrality bandwagon, their products did not. Of course not; neutrality largely turned them into commodities. If SQL was standard, you could buy your database, for instance, from Oracle, IBM, Informix (remember them?), Ingres (remember them?), Microsoft (remember them? Oh wait, they’re still here)… Assuming that performance was similar, you, the customer, would gain bargaining power. Oracle get too expensive? Move to Informix, or IBM, or any of the numerous other vendors.
So, vendors fought neutrality quite effectively. They did it by taking the standards and “extending” them with their own, proprietary implementations (anyone who remembers doing outer joins with + signs in Informix knows what I mean). Perhaps the most obvious extensions were procedural languages; Oracle’s PL/SQL is very different than Microsoft’s T-SQL.
The same happened on the UNIX front. Each vendor delivered their own “flavor,” be it AIX, HP-UX, Solaris, etc.
Why You Need Neutrality
In many cases vendor extensions allow you to do things you can’t do easily with the open, or standard, versions of the technology. On the other hand, however, they bind you to one vendor. Once you write in PL/SQL, for example, it’s really hard to move away from Oracle. And, anyone who\’s received a renewal bill from Oracle knows how painful that linkage can be.
Neutrality and the Cloud
Today, companies are moving their infrastructures to the cloud and that presents them with an opportunity to break free of proprietary technologies. However, many (most?) companies are falling back into the trap. Yes, I\’m no longer using SSIS for ETL but am I just shifting that work to Amazon Glue, or Microsoft ADF? If so, I’m back to using proprietary, vendor-specific tools.
Steps You Can Take
So, how to break the cycle? As I noted at the top of this post, I, regrettably, don’t have all the answers. But, I do have a few suggestions:
Use DB-Independent Middleware
There are technologies available to help make your applications database independent. For example, you may know that we\’re making an investment in developing a unique data matching technology called Golden Record (GR). GR is built on relational technologies. However, rather than writing it for SQL Server, MySQL, or Redshift, we’re employing a Python framework called SQLAlchemy. SQLAlchemy allows us to use largely DB-independent SQL. We can work with virtually any SQL database and SQLAlchemy will translate our calls into the version of SQL we’re running on. Is it perfect? Absolutely not. But, I’m guessing that as our requirements for improved performance increase, the effort we’ll need to port from one database to another will be reduced by 60 – 80% versus if we had written directly to a vendor’s SQL implementation.
Use Open Source ETL Technologies
The market now provides a number of open source technologies for moving data. Check out things like Apache Kafka and Airflow and Stitch to name a few. Will these be free? In the end, likely not. But, they will help keep you vendor neutral.
Stick to Ansi Standard SQL
When writing SQL, avoid vendor-specific, procedural extensions. There actually is an ISO standard for procedural database calls, SQL/PSM although, to be honest, I have never encountered it in 30 years of practice. I have found, however, that smart developers can frequently find ways to solve problems using traditional SQL when they originally thought they needed procedural commands. Even better, they frequently found that the traditional SQL actually provided better performance than did the extensions.