Posts

Showing posts from June, 2024

Your Azure data factory sink might not work as expected, have you not considered these settings already

Image
    ADF is a very convinient way of moving data from external system to Azure SQL storages (Azure SQL, Azure MySQL, Azure cosmos DB, Cassandra: not to mention blob, table and file systems), without writing any code. In fact dataflows are so convinent ways of handling data, that it definitely does have an upperhand as compared to Azure Synapses, where you need to write a lot of staff in Python on JuPyter Notebook to instigate the cleaning and transformation processes. However your ADF dataflows could fail very weirdly, if you have not considered these options. These happen typically when any of your Source/sink involves an Azure SQL connection: A. Missing property error: Your piepline gave a nasty error as:  What?!!! Wait -- I have validated the pipeline and published everything without any errors. Even then, why on earth, did it happen? Solution : While creating the sink/source Linked Service, you need to select 'Legacy' instead of 'Recommended' in  Version  par...

Create and deploy a web app on Azure, using Azure bicep step by step

Image
¡Hola! I am sure you must have gone through my earlier post that housed an introduction to Azure Bicep and how we can easily manage mass deployment of resources, dynamically using the same: https://subsd365.blogspot.com/2024/05/manage-your-azure-resource-deployments.html This blog itself gave you an insight of deploying Azure resources and resource groups using Azure biceps. However in my next blog, I will take a deeper look at Azure bicep, giving you a feel of creating and managing and deploying a bit more complex resources which have dependencies and a lot of overheads to consider. One of the shortfalls of the ARM template was: your ARM template is always needed to be tagged to a subscription or a resource group. Consequently it's very difficult to create a resource group and then add resources to that resource group. What you used to do as a workaround was: to create two independent templates: one for resource group and then another for resources, and then execute them sequent...