PinnedSanajit GhoshSlowly Changing Dimensions (SCD) using Databricks (python +SQL)Slowly Changing Dimensions (SCD) — These are dimensions records which slowly changes with time or specific periods. In Data Warehouse…2 min read·Aug 10, 2020--1--1
Sanajit GhoshFlattening deep nested JSON using python and DatabricksThe article tells how to flatten a deeply nested json file using python and databricks. Reaching every node in a heavily nested JSON can be…3 min read·Mar 20, 2024--1--1
Sanajit GhoshNeo4J Kubernetes Cluster deploymentCreate a kubernetes cluster with azure cli(Recommended). Here you can choose the number of agent or the master nodes. You can refer this…3 min read·Oct 29, 2020----
Sanajit GhoshDeploying a ML service to the Azure Container Service (AKS)In this setup we need to create two resource group using Machine Learning workbench CLI and Azure portal itself. At the end of deployment…6 min read·Oct 7, 2020----
Sanajit GhoshReal time dashboards with SQL server and logic appsVisualization of real time data over dashboards can be complicated because refresh rate of incoming data at a particular time frame is…3 min read·Oct 6, 2020----
Sanajit GhoshPipeline monitoring & sending email based notification using ADF and Logic appsLogic apps are powerful services that are meant to provide useful APIs which are used for handling many use cases.3 min read·Jul 13, 2020----
Sanajit GhoshProactive monitoring of cloud services & pipelines using Azure MonitorWhen Data integration activities fails, constant manual monitoring becomes very time consuming, boring and tedious. To improve the overall…4 min read·Jun 9, 2020----
Sanajit GhoshDatabricks Structured streaming using EventHub, Kafka & PowerBIThe use case for this scenario is to capture real time streaming data from sensors and IoT devices and then process them for data analysis…4 min read·May 3, 2020----
Sanajit GhoshData Visualization using Databricks and PowerBIIn my previous post, I have explained the workflow of processing real-time data from sensors using Azure databricks and Event Hubs.3 min read·May 3, 2020----
Sanajit GhoshManage CI/CD pipelines using Azure DevOps, Azure Data Factory & Azure DatabricksIn Part 1 of this post, I will show you the process of automating a data factory pipeline using Azure DevOps.5 min read·Mar 8, 2020--3--3