iLearn

Hey guys, wanted to talk about delta lake in data bricks and the ADLS ecosystem. I have found an issue while changing the schema of delta tables.  Problem Statement - Let's say you have a table Table1 with columns column1 and column2 as a delta table and the requirement is to get rid of column2 from Table1, something like below Using T SQL we could easily achieve this by using ALTER Table drop column ...command However, in the spark SQL and Databricks realm, I couldn't find any such feature. There are options to re-order, add columns but not drop as per  https://docs.databricks.com/delta/delta-batch.html Solution - Seems like the only possible solution is to drop the folder/delta table in the ADLS You would have the table/folder recreated after running the above spark SQL query (screenshot) Does anyone have a proper solution to this? This looks like a legit limitation as changing schemas is very common in real-time projects

  Prerequisites  Azure SQL Server with a source and target database instance GitHub repo Visual code for SQL Server Database Objects Familiarity about dacpac Azure DevOps Account Things I didn't check  Since I was an admin on azure, DevOps, I am not sure on the permissions required. Ideally contrubutor access should be enough to implement this solution.  Problem Statement and Solution We are using dacpac for deploying our SQL artifacts from source to target server/database. The SQL server code, as well as the dacpac, is checked inside the GitHub repo. We need to develop an azure DevOps pipeline that uses the dacpac and does the deployment.  I have found if you change the dacpac location to a custom folder and sync it to the Github repo, we could achieve what we had set out to.  Logical Steps 1. Create a SSDT project  2. Link SSDT to github repo with a local repo 3. Create a folder for dacpac in local repo 4. Add settings in visual studio to point dacpa...