iLearn

  So got this error today and at first glance, I thought I inadvertently cancelled the job. Tried running it twice and the job got cancelled again.  Full marks to the DevOps developers for diagnosing and proposing the solution.  Changing the agent job machine to windows-latest worked for me

  Super quick solution if you ever encountered/stuck with this scenario in Azure Data Factory  Let's say there is an if condition activity and you need to evaluate the output value from a previous activity (could be metadata or could be inside for each loop ).  The values to be evaluated against are more than two in number.  The programming equivalent of it would           when variable1 not in (Value 1, Value 2, Value 3, Value 4) If you have some experience working in the azure data factory, you would like to use the following expression  @and(not(contains(item().name, 'Value1')), not(contains(item().name, 'Value2')), not(contains(item().name, 'Value3')), not(contains(item().name, 'Value4')), not(contains(item().name, 'Value5')), not(contains(item().name, 'Value6')) ) However, this will not work out and you will get an error message.  Instead, try using a variable. Assign the value to the variable before evaluating and then use this

  Man! The past couple of weeks has been really tough. Hardcore development on Azure Data Factory, and Azure Databricks as we are up against a tight deadline (again  :-) ). Loads of different scenarios and loads of new learnings. Sharing one below, keep reading. We are receiving a source file (let's call it Test.csv) which has a blank row before the header row  1 2 "colname1", "colname2" 3 "value1","value2" we are using  spark.read.format to load this into a data frame.  Looking at the file contents, one would assume that you need to somehow skip the first blank row.  So I began researching it. Found that spark.read.format does nt provide any such property.  After spending couple of hours with no major break through, I thought of testing the code as it is  val rawdataframe=  spark.read.format("csv").option("header","true").option("inferSchema","true").option("delimiter", s",&qu

  Hey guys, time to share something new that I have learned in the past few days.  I am going to share my thought process on how I viewed the problem statement and arrived at the solution.  So, let's get rolling, as part of the requirements, we were given an API endpoint and were told that we need to ingest data and show it on the power bi report.  I began brainstorming on how do we pull data from API either to ADLS storage in the form of CSV or txt files or to Azure SQL DB in the form of a managed table.  Disclaimer -  I am new to extracting information from API endpoints using Azure Data Factory, so had to read quite a few blogs. I discovered that we could either use Rest API connectors or HTTP connectors to talk to the API endpoint and pull out the information needed.  So I tried using both of the above connectors and found that it is becoming challenging to extract the information in the CSV file. Although I did get some success in fetching only the first row from the API endpo

Continuing our series on SQL Query Questions Asked During Interview. This is part 2.  You can find part 1  here You might not encounter the following scenario in real-time projects, however, it would help you exercise your SQL skills and might help if you are preparing for an Interview.  Please refer to the picture below. On the left side, you have a table with a single column called teamname which consists of various acronyms of countries situated in Asia that play cricket. And on the right side, is the expected output. Essentially, we need to come up with the fixtures of teams playing each other and no two teams playing twice .  Common give it a try For answer and explanation, refer to the video below