Many organisations have built up their business processes and data within the Dynamics environment. One of the key features that Dynamics offers its users is the ability to export data from Dynamics 365 into an Azure SQL Database using the Dynamics Data Export Service (DES). However, in November 2022 Dynamics Data Export Services will be deprecated, leaving users searching for a suitable replacement - Azure Synapse Link for Dataverse.
This blog post highlights the design options available from a Azure Synapse network connectivity perspective. Initial planning for this aspect is essential as some changes cannot be made after deployment.
Listen to Crimson's Head of Data, Mark Cotton, and Solutions Architecture Practice Lead, Matthew Lancashire discuss this topic:
Dynamics Data Export Services – is it actually deprecation?
Deprecation means the use of something is being discouraged, however, Microsoft have announced it will reach end of life in November 2022. Therefore, it is advised that users should assume it will be shut off in November 2022.
The replacement: Azure Synapse Link for Dataverse
Although deploying an Azure Synapse instance is a few clicks in the Azure Portal, the component parts deployed include other Azure services such as Workspace, SQL and Storage Account - understanding how these services interact makes the planning stage easier.
From a deployment connectivity perspective one of the main choices is whether to use a Managed virtual network or not , this option can only be set at deployment and cannot be changed afterwards.
It's not a 'real' virtual network - it's a Managed one!
The current option to join Azure Synapse to a Virtual Network is to use a Managed Virtual Network. These offer less configuration options than a ‘normal’ Virtual Network and it is important to plan these into design stage. If Azure Synapse is to be deployed into an existing Azure architecture, it is also important to consider the differences listed below:
- Managed Virtual network cannot be peered with other Virtual Networks so routing considerations are required
- It is not currently supported to add User Defined Routes or Network Security Groups to Managed Virtual networks, somewhat restricting classic routing and security options
- There is more of a reliance on Private Endpoints to surface and route network traffic
- There is no option to set the Managed Virtual Network IP ranges
Synapse Network Components
Because Synapse can contain a diverse set of Azure services, they don't always ‘sit’ within the Managed Virtual Network. With the Managed Virtual Network, those multi–tenant services use Private Endpoints to communicate and offer network level isolation. A Private Endpoint is a way to provide secure networking to an Azure PaaS service that does not fully support deploying to a Virtual Network, although implementing Private Endpoints can add to the complexity of design. Also for consideration is the location of the services being connected to - there may be liaising with different teams or 3rd parties which may impact timelines in a project plan.
Creating a workspace that has a Managed workspace Virtual Network associated with it ensures that your workspace is network isolated from other workspaces. Azure Synapse provides various analytic capabilities within workspaces: Data integration, serverless Apache Spark pool, dedicated SQL pool, and serverless SQL pool.
If your workspace has a Managed workspace Virtual Network, then Data integration and Spark resources are deployed in it. A Managed workspace Virtual Network also provides user-level isolation for Spark activities because each Spark cluster is in its own subnet.
Dedicated SQL pool and serverless SQL pool are multi-tenant capabilities and therefore reside outside of the Managed workspace Virtual Network. Intra-workspace communication to dedicated SQL pool and serverless SQL pool use Azure private links. These private links are automatically created for you when you create a workspace with a Managed workspace Virtual Network associated to it.
What's the cost difference?
For many Microsoft services that use Virtual Network integration, this can mean moving to the highest or premium tier. There is currently no such cost model for Synapse, and the only place where the additional cost of a Managed Virtual Network is shown is in the Data Pipelines runtime costs which are considerably more when using Managed Virtual Network integration.
So, what's the conclusion?
If you have an Azure Synapse deployment that uses Dynamics Data Export Service (DES), you will need to replace it, and Azure Synapse Link for Dataverse is the ideal replacement. Azure Synapse Link for Dataverse offers better integration with other Azure services than DES did. Therefore, if you are using Dynamics Data Export Service (DES), we recommend that you migrate to Azure Synapse Link for Dataverse.
