PinnedSanajit GhoshSlowly Changing Dimensions (SCD) using Databricks (python +SQL)Slowly Changing Dimensions (SCD) — These are dimensions records which slowly changes with time or specific periods. In Data Warehouse…Aug 10, 20201Aug 10, 20201
Sanajit GhoshParallel processing of REST API in python/databricksProcessing 1000s of API pages in parallel can be challenging, especially in sync mode as each time the for-loop waits for the event to get…May 30May 30
Sanajit GhoshParallel processing/Multiprocessing using python and databricksThis is a simple program to find the power of a number. There are 10000 numbers in a list and the processing is done first synchronously…May 24May 24
Sanajit GhoshFlattening deep nested JSON using python and DatabricksThe article tells how to flatten a deeply nested json file using python and databricks. Reaching every node in a heavily nested JSON can be…Mar 201Mar 201
Sanajit GhoshNeo4J Kubernetes Cluster deploymentCreate a kubernetes cluster with azure cli(Recommended). Here you can choose the number of agent or the master nodes. You can refer this…Oct 29, 2020Oct 29, 2020
Sanajit GhoshDeploying a ML service to the Azure Container Service (AKS)In this setup we need to create two resource group using Machine Learning workbench CLI and Azure portal itself. At the end of deployment…Oct 7, 2020Oct 7, 2020
Sanajit GhoshReal time dashboards with SQL server and logic appsVisualization of real time data over dashboards can be complicated because refresh rate of incoming data at a particular time frame is…Oct 6, 2020Oct 6, 2020
Sanajit GhoshPipeline monitoring & sending email based notification using ADF and Logic appsLogic apps are powerful services that are meant to provide useful APIs which are used for handling many use cases.Jul 13, 2020Jul 13, 2020
Sanajit GhoshProactive monitoring of cloud services & pipelines using Azure MonitorWhen Data integration activities fails, constant manual monitoring becomes very time consuming, boring and tedious. To improve the overall…Jun 9, 2020Jun 9, 2020
Sanajit GhoshDatabricks Structured streaming using EventHub, Kafka & PowerBIThe use case for this scenario is to capture real time streaming data from sensors and IoT devices and then process them for data analysis…May 3, 2020May 3, 2020
