In October 2017 at the dataMinds conference (former SQL User Days) I gave a presentation about Data Lake Analytics and one of my slides was the following:
This picture was taken and I found that it was quite a nice topic to write something about. So here we are!
In Azure there are 3 different big data stores that you can use and in this blog post I want to explain the differences between them.
Azure Cosmos DB (former Document DB) is a multi-model database or NoSQL database. What does that mean and what are some features of Azure Cosmos DB?
Azure SQL Data Warehouse is the easiest to explain if you are already using SQL Server. It’s the same as an ordinary SQL Server database but developed for Petabyte scaled databases. By making use of Massively Parallel Processing it can quickly run complex queries across petabytes of data.
It stores relational tables with columnar storage. That way it can significantly reduce the data storage cost and improves query performance. Compared with traditional database systems, analysis queries finish in seconds instead of minutes, or hours instead of days.
Some features:
More info can be found at: https://docs.microsoft.com/en-us/azure/sql-data-warehouse/
What is a data lake? One of my colleagues explained it quite nicely. If you take a bottle of water, it’s all cleansed and ready to drink. That water can be poured back into a lake or a river, and nothing will happen. Now if we do that the other way around and we fill our bottle with water from the lake, the bottle might be contaminated with trash or diseases.
So in a data lake we can store any kind of data, unprocessed (river water) and processed (bottled water). And the nicest thing is that it does not have to be structured. Relational data in any sort is not necessary. You can add images, music, documents, databases, text files, … No size limit to files or total storage (That is not possible with Azure Blob Storage). That are the storage capabilities of Data Lake. But an extra resource that is available is Data Lake Analytics that has a query language called “U-SQL”. This so called U-SQL gives us the ability to query text-files but also documents (docx, pdf, …) and images.
How nice would it be if you have a collection of photos and you would want to see only the photos that have a horse on it. Or only with people who are smiling. Or what about executing Python or R code on data that is just there. It’s all possible.
Some features:
When do we use what? If you need a multi-model database go to Cosmos DB. If you need a relational database make use of SQL Data Warehouse, but if your data exists of all kind of different types of files then make use of Data Lake to clean and transform it, so you can store it in one of the others if necessary.
© 2023 Kohera
Crafted by
© 2022 Kohera
Crafted by
Cookie | Duration | Description |
---|---|---|
ARRAffinity | session | ARRAffinity cookie is set by Azure app service, and allows the service to choose the right instance established by a user to deliver subsequent requests made by that user. |
ARRAffinitySameSite | session | This cookie is set by Windows Azure cloud, and is used for load balancing to make sure the visitor page requests are routed to the same server in any browsing session. |
cookielawinfo-checkbox-advertisement | 1 year | Set by the GDPR Cookie Consent plugin, this cookie records the user consent for the cookies in the "Advertisement" category. |
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
CookieLawInfoConsent | 1 year | CookieYes sets this cookie to record the default button state of the corresponding category and the status of CCPA. It works only in coordination with the primary cookie. |
elementor | never | The website's WordPress theme uses this cookie. It allows the website owner to implement or change the website's content in real-time. |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |
Cookie | Duration | Description |
---|---|---|
__cf_bm | 30 minutes | Cloudflare set the cookie to support Cloudflare Bot Management. |
pll_language | 1 year | Polylang sets this cookie to remember the language the user selects when returning to the website and get the language information when unavailable in another way. |
Cookie | Duration | Description |
---|---|---|
_ga | 1 year 1 month 4 days | Google Analytics sets this cookie to calculate visitor, session and campaign data and track site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognise unique visitors. |
_ga_* | 1 year 1 month 4 days | Google Analytics sets this cookie to store and count page views. |
_gat_gtag_UA_* | 1 minute | Google Analytics sets this cookie to store a unique user ID. |
_gid | 1 day | Google Analytics sets this cookie to store information on how visitors use a website while also creating an analytics report of the website's performance. Some of the collected data includes the number of visitors, their source, and the pages they visit anonymously. |
ai_session | 30 minutes | This is a unique anonymous session identifier cookie set by Microsoft Application Insights software to gather statistical usage and telemetry data for apps built on the Azure cloud platform. |
CONSENT | 2 years | YouTube sets this cookie via embedded YouTube videos and registers anonymous statistical data. |
vuid | 1 year 1 month 4 days | Vimeo installs this cookie to collect tracking information by setting a unique ID to embed videos on the website. |
Cookie | Duration | Description |
---|---|---|
ai_user | 1 year | Microsoft Azure sets this cookie as a unique user identifier cookie, enabling counting of the number of users accessing the application over time. |
VISITOR_INFO1_LIVE | 5 months 27 days | YouTube sets this cookie to measure bandwidth, determining whether the user gets the new or old player interface. |
YSC | session | Youtube sets this cookie to track the views of embedded videos on Youtube pages. |
yt-remote-connected-devices | never | YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. |
yt-remote-device-id | never | YouTube sets this cookie to store the user's video preferences using embedded YouTube videos. |
yt.innertube::nextId | never | YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. |
yt.innertube::requests | never | YouTube sets this cookie to register a unique ID to store data on what videos from YouTube the user has seen. |
Cookie | Duration | Description |
---|---|---|
WFESessionId | session | No description available. |