Azure's Tools for Scalable Analytics

What is Azure Synapse Analytics?

Microsoft describe Azure Synapse as "a limitless analytics service that brings together enterprise data warehousing and Big Data analytics", but what does that mean? It essentially means that Synapse is a container for a whole suite of tools that have been brought together from across the Microsoft Azure data stack, starting with what was once called Azure SQL Data Warehouse, then bringing in technologies like Polybase and Apache Spark for unstructured data and big data analytics, Azure Data Explorer for log and time series analytics, and Data Factory for orchestration of complex ELT\ETL processes.

It means you have a one-stop-shop to build out an enterprise analytics platform that can consume data of all shapes and sizes, and serve a whole range of user personas and skill sets, all within a single collaborative workspace. Some of the key benefits include:

• Limitless scale - separating compute and storage enables more efficient and cost effective scaling whilst providing insights across data warehouses, data lakes, operational databases, and big data systems.

• Reduced development time - converging several different experiences in one environment makes it easy to share data without having to move it first and keeps development all within the one unified console.

• Unified workspace - Data engineers, data scientists, database administrators, and business analysts all work within the same analytics service, making it much easier to collaborate.

• Converge data - Simplify access to data for analysts and data scientists by consolidating in in Synapse. Components like Azure Synapse Link provide near real-time access to operational data in SQL or Dataverse by automatically synchronising it into Synapse.

• Secure data - A unified approach to access control makes it simple to control secure access to resources, and advanced security and privacy features such as automated threat detection and always-on encryption, keep your data safe and protected.

What are the key components of Synapse?

• Workspace - the first Synapse resource to create, the workspace is a securable container that represents the boundary of a collection of Synapse resources

• Studio - the unified administration console that is used to manage and interact with all resources in a Synapse workspace

• ADLS Gen 2 Storage - Every workspace must have an associated storage account and file system for temporary data storage. This can be created as part of the workspace creation process

• Serverless SQL Pool - A distributed query processing engine that can be used to query data from the data lake using T-SQL

  • Every workspace has a serverless pool by default

• Dedicated SQL Pool - Evolved from Azure SQL Data Warehouse, a massively parallel processing (MPP) engine to support analytical workloads across very large volumes of data, with its own dedicated storage and compute resource

  • No dedicated pools are created by default in a Synapse workspace, but any number can be added

• Apache Spark Pool - Provision compute resource to process Spark analytics workloads

  • Apache spark pools are not created by default in a Synapse workspace but can be added as required

• Synapse Pipelines - An implementation of Azure Data Factory within Synapse that provides orchestration of data movement and transformation to maintain data lakes, lakehouses, and warehouses through variety of methods

How much does it cost?

It's complicated! Each of the different workloads in Synapse has their own distinct consumption requirements and associated billing model, which makes predicting overall spend quite challenging and can be something which goes up and down over time depending on usage patterns.

You can explore the modular pricing of various Synapse components on Microsoft's website but we have included the simplest version of that here, which is using Pre-Purchase plans, whereby you commit to a total volume of consumption, measured in Synapse Commit Units (SCUs), which represent a shared pool of resource to be used by all workloads. This gives you a single bill for your whole Synapse workspace, and also has the benefit of a 28% saving, but you must carefully plan your anticipated workload to ensure you are provisioning enough resource for your solution.

How can PTR help you?

As a Microsoft Solutions Partner for Data & AI in Azure, we specialise in helping our customers navigate the complex array of Microsoft Azure resources, including Azure Synapse, to design robust data and analytics platforms.

• Provide strategic advice and develop sustainable technical strategies

• Understand the difference and choose between the different components available

• Understand the different billing models and tiers to make sensible selections and estimate consumption and costs

• Implement and configure Azure Synapse resources

• Implement secure networking and integrate with existing infrastructure within Azure, across other cloud providers, and on-prem

• Understand and implement secure user and access control practices using Microsoft Entra ID