PinnedSlowly Changing Dimensions (SCD) using Databricks (python +SQL)Slowly Changing Dimensions (SCD) — These are dimensions records which slowly changes with time or specific periods. In Data Warehouse…Aug 10, 20201Aug 10, 20201
Parallel processing of REST API in python/databricksProcessing 1000s of API pages in parallel can be challenging, especially in sync mode as each time the for-loop waits for the event to get…May 30May 30
Parallel processing/Multiprocessing using python and databricksThis is a simple program to find the power of a number. There are 10000 numbers in a list and the processing is done first synchronously…May 24May 24
Flattening deep nested JSON using python and DatabricksThe article tells how to flatten a deeply nested json file using python and databricks. Reaching every node in a heavily nested JSON can be…Mar 201Mar 201
Neo4J Kubernetes Cluster deploymentCreate a kubernetes cluster with azure cli(Recommended). Here you can choose the number of agent or the master nodes. You can refer this…Oct 29, 2020Oct 29, 2020
Deploying a ML service to the Azure Container Service (AKS)In this setup we need to create two resource group using Machine Learning workbench CLI and Azure portal itself. At the end of deployment…Oct 7, 2020Oct 7, 2020
Real time dashboards with SQL server and logic appsVisualization of real time data over dashboards can be complicated because refresh rate of incoming data at a particular time frame is…Oct 6, 2020Oct 6, 2020
Pipeline monitoring & sending email based notification using ADF and Logic appsLogic apps are powerful services that are meant to provide useful APIs which are used for handling many use cases.Jul 13, 2020Jul 13, 2020
Proactive monitoring of cloud services & pipelines using Azure MonitorWhen Data integration activities fails, constant manual monitoring becomes very time consuming, boring and tedious. To improve the overall…Jun 9, 2020Jun 9, 2020
Databricks Structured streaming using EventHub, Kafka & PowerBIThe use case for this scenario is to capture real time streaming data from sensors and IoT devices and then process them for data analysis…May 3, 2020May 3, 2020
