Menu
Menu
Since its announcement as a first-party service on Microsoft Azure at the end of 2017, Databricks has seen a remarkable growth in usage. However, the service and its success were around long before Microsoft came into play. Going by the fact that you are reading this blog at this very moment, I’ll assume that you have used Databricks before, or at least have heard that Databricks is in the data business. And cousin, business is a-boomin’! One aspect that may have left users of the service frazzled, is the steady stream of updates it receives. This begs the questions, why are there updates so often? And what is the impact on maintenance of your dearest Databricks-projects?
Databricks was first released in limited availability in November 2014. With minor updates happening more or less monthly and major releases every 6 to 12 months, we’re now on version 8.2. So lets first take a look at what these minor and major updates generally change.
Databricks is a web-based platform that is built on top of the Apache Spark framework. In fact, the company was founded by the original creators of Apache Spark. This means that not only do the updates incorporate changes to the Databricks platform itself, it also upgrades the Spark engine it is built on. This is reflected in the type of releases. A minor release contains only updates to Databricks, while major releases incorporate updates to Spark.
So why are there so many updates? While Databricks isn’t open source, many of the systems used are – most notably the Spark engine. Given the engine’s popularity, this makes for a very dynamic development process, resulting in a lot of updates. And as mentioned above, changes to Spark end up triggering a major release for Databricks too.
Furthermore, you can’t teach an old dog new tricks. While the Databricks platform isn’t open source, the developers have kept their habit of being deeply ingrained in their online communities and incorporating their feedback. Coupled with the fact that Databricks is used in several rapidly developing fields such as data science and machine learning, this makes for an explosive mix that triggers fast minor and major releases.
Let’s quickly check in on what the exact Databricks Runtime Versions lifecycle is, since it impacts the maintenance of your Databricks-projects.
Releases start in beta, which has three modes:
After beta there is a full support release:
The next phase is End of Support (EOS):
Finally, there is End of Life (EOL):
Two years might not be what a lot of us envision when we hear the phrase ‘Long Term Support’. Not only are the timeframes shorter compared to conventional platforms such as SQL Server that keep churning away even after support ends, your Databricks project on an abandoned version can fully stop working at any time. All the more reason to keep close tabs.
So finally, we arrive at the big question: how does all of this impact us when it comes to maintaining our flourishing Databricks projects? This question is largely decided by the role the Databricks platforms plays for you. If you are a data scientist, chances are you are very actively working and reworking your notebooks on a day-to-day basis. If you are a data engineer on the other hand like most of us at Kohera are, you might be looking more into building a reliable ELT platform.
In the first case, updates can quickly be incorporated into your work and it’s worth it to keep close tabs on the newest release. For the latter, incremental updates are welcome, but realistically it’s more beneficial to limit the amount of time spent on maintenance.
So as a data engineer, it’s best to check in and update your clusters at least every 3 to 6 months. If you are particularly enthusiastic, you can always be up to date of course. However, setting a hard limit of 6 months with a bit of leeway beforehand, keeps you safe from your environments suddenly grinding to a halt. Additionally, we would suggest sticking mostly with full support releases. If you do opt for beta releases as a data engineer, it is advisable to give these releases a few months at least to mature before jumping in.
“Alright, cool, good to know, but what does the actual maintenance entail then?”, I hear you say. Well, so far Databricks has shown the tendency to mostly add new features or improve existing ones. This is great news of course, as it allows us to upgrade our platforms, while still being relatively sure that our notebooks will keep working. Still, when you do push an upgrade, it remains prudent to check out the release notes and think about what changes might impact your notebooks. The easiest way though to check, is to upgrade your dev platform first and run some sample notebooks with different scenarios. The proof is in the pudding, as they say. I haven’t really encountered breaking code on minor releases so far – knock on wood – but I did on major releases. Especially when there was a Spark upgrade incorporated. Keep an eye out for those.
Important to note: pay extra attention if you use third-party libraries. These can add a lot of functionality to your clusters, but the downside is they add a potential weakness in your notebooks when Spark receives an upgrade. The library might not work with the newer version of Spark.
This could always be resolved, of course, if there is still support and development for the library. However, this results in time pressure and makes you reliant on the third party providing updates. I’ve had good experiences with this in the past – shoutout to Cobrix for very quick responses to fellow Koherians in such cases – but it’s best to always remain vigilant regarding this.
Finally, you may be thinking “Alright, yeah, most interesting blog I’ve ever read in my entire life. If only they’d also tell me where to look for release notes, more interesting blogs regarding big changes or maybe something in the genre of a lively vlog on the topic.”. I get your need for a guide towards the Databricks resources treasure! So I have provided links to other Kohera blogs regarding Databricks, as well as documentation links below. Also, I would be remiss to not mention Simon Whiteley’s videos.
