Caching in Databricks? Yes, you can!
Sometimes, Databricks can be a bit sluggish. Especially when working with many small parquet files on Azure Data Lake. This sluggishness is often due to the security and read/write access requests that the Databricks cluster needs to maintain. This slows down the processing significantly, as the Databricks cluster reserves memory space for every I/O thread […]
