Managing files from other devices in a Fabric Lakehouse using the Python Azure SDK

In this blogpost, you’ll see how to manage files in OneLake programmatically using the Python Azure SDK. Very little coding experience is required and yet you can create very interesting automations this way.

I started testing this approach when I was creating a proof-of-concept. I was working on a virtual machine where database tables were exported to Parquet files on a daily base. These Parquet files needed to be moved to OneLake. However, at the time of writing the on-premise gateway for pipelines in Fabric was not yet fully supported.

Instead of using a pipeline, I started thinking about uploading the files directly to OneLake from the machine and then proceed in Fabric for further processing.

The process for doing this is described below.

Authentication and authorization configuration in Azure and Fabric

Step 1: Create an app registration (service principal) in Azure and create a client secret.
Write down the tenant_id, client_id and client_secret. A detailed guide on how to create the service principal can be found here: https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-register-app

Step 2: Add the service principal as contributor to your Fabric workspace.

Manage OneLake from other devices using Python

Establish the connection to OneLake

Step 1: First create a JSON file with the Service Principal credentials. I put this file in the config folder of my project

Step 2: Next, import the dependencies. Make sure they’re installed in the Python environment you’ll be using. From the Azure SDK we import the package to manage the datalake (OneLake) and we also import the package for authenticating with the Service Principal using a client secret. Lastly, the JSON package is imported to read our config file created in step 1.

Step 3: Make a Credential object for the Service Principal.

Step 4: Put the name of the workspace and the lakehouse in variables so it can be easily reused.

Step 5: Create a DataLakeServiceClient object. This is an object at the OneLake level. Next use this object to create a FileSystemClient object. The FileSystemClient is on the workspace level. Once we have this, the preparation is done. Now we can start doing stuff. 😊

Playtime!

Below are some examples of what we can do with our FileSystemClient object.

Example 1: List all the folders starting from a specific path in OneLake

Example 2: Create a new (sub)folder on OneLake

Example 3: Upload a file to OneLake

Example 4: Download a file from OneLake

Example 5: Append to a CSV file on OneLake

Example 6: Delete a file from OneLake

Example 7: Delete a directory from OneLake

The data is in OneLake. What’s next?

Let’s go back to the setup of my proof-of-concept. On my virtual machine I uploaded the exported tables to a folder on OneLake named ‘Exports’. Each table had its own subfolder. It looked like this:

To expose the data in Lakehouse Tables, I made a very simple pipeline. It starts by creating a list of all the exported tables by looking at the exports folder. Next, a foreach loop is initiated with a copy activity to copy the data into a Lakehouse table. In my setup, the tables are overwritten in every load.

The structure of the pipeline is shown below:

Conclusion

As you can see, it’s very easy to manage files on OneLake with Python, especially for people who have used the Python Azure SDK for data lakes before. OneLake can be approached in an unthinkable number of ways. So there is a lot of room for creativity when developing your architectures. I would definitely recommend playing around with it. Have fun!