“Connecting to an (Azure) database from within Databricks using revoscalepy by deploying Microsoft’s Machine Learning Server”. I am sure that I am not the first guy typing this line into a search engine, and I’m very sure that I won’t be the last neither. The real power of Python resides in the flexibility of the language and the power of its data frames and the agility to play with data in a very easy way.
I made a very useful ELT program in python and wanted it do run inside a Databricks cluster. Databricks on Azure fully supports python 3 so I thought I was up for a walk in the park. Trying to import the database connection classes already gave a small hint of the troubles ahead.
The easy part was to create the workspace, once you have created the Databricks environment, it’s easy to create a new environment by clicking the Launch Workspace button:
Because I am creating my workspace for a rather simple purpose, I create a very simple cluster with limited resources.
For the purpose if this demo, a F4s will largely be sufficient, as it has 4 cores and 8GB of ram. Autoscaling will not be needed, and the cluster can easily destroy itself after 50 minutes of inactivity. (This is a demo anyway)
Spinning up a simple cluster like this also has the advantage that it goes fast, and I don’t need to drink 5 cups of coffee before I get my resources (I only needed 1 <Evil Grin>)
Once the cluster is available, we can now create a new notebook. To create the notebook, click on the Azure Databricks button
Here you can select, New Notebook
Make sure you connect to your newly created cluster, and the language is on Python
This opens your workbook, and now we are in our Python workspace.
It is here that I started to get some worries almost everything that I could find about working with Azure Database involved JDBC, and somehow I couldn’t get them working as I wanted.
Importing the base libraries, work like a charm, loading as we expect, and working just great
Once I started to load, my favorite SQL libraries, like RevoScalePy for example, it’s not loaded.
My first (and obvious) guess was to look for an instruction how to load revoscalepy on Databricks.
For reasons unbeknownst to me, this instruction doesn’t exist, neither on the Databricks pages, nor on on the Ravoscalepy pages.
Ok bummer…
But this never stopped me before, so why would it stop me now…
First thing that we can now (ab)use is knowing what platform Databricks is built on. This is well documented on docs.databricks.com
Because we created a Databricks 5.0 environment, we now know that we are essentially running with the following important actors:
But most important of all this piece of information:
Ok great, we are running on Ubuntu 16.04, time to dust off those Linux Shell skills (always knew they would come in handy sometime) and there is an installation manual for Ubuntu and Revoscalepy Yay!
The easiest way to test your shell scripts before turning them into a initialization script, is to run them from inside your notebook.
Adding the %sh command, puts your notebook in shell mode, it’s now a graphical linux shell.
Ok now let’s get cooking, Ubuntu’s default package manager is apt, so let’s start
Let us first update the package manager and it’s repos.
Woot, this works!
Now let us try Microsoft’s installation manual, I just made some minor adjustments so that it could run in this shell environment
Installing the package:
And then… Bummer
Running the following query shows us that the file has been created though
Changing Microsoft script to:
Fixes that issue, allowing you to rerun apt-get update
Be aware that this is a rather large install (almost 10 GB) and will take several minutes to finish, but after approximately 10 minutes we got the following message:
%sh
|
Done!
Cookie | Duration | Description |
---|---|---|
ARRAffinity | session | ARRAffinity cookie is set by Azure app service, and allows the service to choose the right instance established by a user to deliver subsequent requests made by that user. |
ARRAffinitySameSite | session | This cookie is set by Windows Azure cloud, and is used for load balancing to make sure the visitor page requests are routed to the same server in any browsing session. |
cookielawinfo-checkbox-advertisement | 1 year | Set by the GDPR Cookie Consent plugin, this cookie records the user consent for the cookies in the "Advertisement" category. |
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
CookieLawInfoConsent | 1 year | CookieYes sets this cookie to record the default button state of the corresponding category and the status of CCPA. It works only in coordination with the primary cookie. |
elementor | never | The website's WordPress theme uses this cookie. It allows the website owner to implement or change the website's content in real-time. |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
Cookie | Duration | Description |
---|---|---|
__cf_bm | 30 minutes | Cloudflare set the cookie to support Cloudflare Bot Management. |
pll_language | 1 year | Polylang sets this cookie to remember the language the user selects when returning to the website and get the language information when unavailable in another way. |
Cookie | Duration | Description |
---|---|---|
_ga | 1 year 1 month 4 days | Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. |
_ga_* | 1 year 1 month 4 days | Google Analytics sets this cookie to store and count page views. |
_gat_gtag_UA_* | 1 minute | Google Analytics sets this cookie to store a unique user ID. |
_gid | 1 day | Google Analytics sets this cookie to store information on how visitors use a website while also creating an analytics report of the website's performance. Some of the collected data includes the number of visitors, their source, and the pages they visit anonymously. |
ai_session | 30 minutes | This is a unique anonymous session identifier cookie set by Microsoft Application Insights software to gather statistical usage and telemetry data for apps built on the Azure cloud platform. |
CONSENT | 2 years | YouTube sets this cookie via embedded YouTube videos and registers anonymous statistical data. |
vuid | 1 year 1 month 4 days | Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. |
Cookie | Duration | Description |
---|---|---|
ai_user | 1 year | Microsoft Azure sets this cookie as a unique user identifier cookie, enabling counting of the number of users accessing the application over time. |
VISITOR_INFO1_LIVE | 5 months 27 days | YouTube sets this cookie to measure bandwidth, determining whether the user gets the new or old player interface. |
YSC | session | Youtube sets this cookie to track the views of embedded videos on Youtube pages. |
yt-remote-connected-devices | never | YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. |
yt-remote-device-id | never | YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. |
yt.innertube::nextId | never | YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. |
yt.innertube::requests | never | YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. |
Cookie | Duration | Description |
---|---|---|
WFESessionId | session | No description available. |