A developer on the team wrote an ETL that runs once a day as a Spark job. Every day it reads a CSV file that shows the total value of each customer's transactions of that day and writes them as a parquet file partitioned by date and customer id. Below you can see an example of the CSV file. Note that each customer has one entry representing the total transaction value it did on that day. However, sometimes the CSV file contains a correction for a sum reported in the past. For example - this file represents the transactions on 1/10. You can see that customer 1002 has 2 entries. One for 1/10 and one for 30/9. This means that the total sum of transactions the customer did on 1/10 is 70, but the total sum of transactions it did on 30/9 was 40 and this sum should replace the value already reported on 30/9. current date file: 2020-10-01 date,customer,price 2020-10-01,1000,40 2020-10-01,1001,10 2020-09-30,1002,40 2020-10-01,1002,70 2020-10-01,1003,10 2020-09-29,1004,10 2020-10-01,1004,10 This function represents the ETL. It runs once a day with a string representing the current day. It reads the CSV file, does some transformations, and writes it. Please help us find the bug in the code above, and return the right results
Data Engineer Interview Questions
Data Engineer Interview Questions
Data engineers are IT professionals who are needed in almost every industry. Data engineers monitor data trends to determine best next steps for companies. A critical part of a data engineer job is to process raw data into usable data by creating data pipelines and building data systems.
Top Data Engineer Interview Questions & How To Answer
Question #1: Can you describe in detail your level of expertise with programming languages?
Question #2: Explain data engineering in your own words.
Question #3: Can you describe your experience working with Apache Hadoop and cloud data management environments?
20,205 data engineer interview questions shared by candidates
Our developer had to join the results with a dimensional table of categories. The join works, but its a bit slow, see if you can understand why and whether it can run faster
Questions about Python and SQL, and were both reasonable to do. They ask to get an interval range in the SQL part and the Python to find the missing number in an unsorted array
How to apply union between 2 tables if table A has 5 columns and table B has 8 columns
About project Previous experience Some random unrelated stuff about which I doubt even he had any idea
Describe your last project with the last Employer
1. Get the top 10 IP addresses from 1TB data files having semi-structured data using map reduce. 2. The intersection of two lists, with the output being in a sorted order. 3. Third highest salary in each department (SQL)
2. Find the sum of contiguous subsequence with maximum sum. (The list contains both positive and negative integers)
Medium level question from leetcode.
Situational Question- If a doctor at the NHS asks- "will Hepatitis B increase or decrease in my area in the next 10 years?" What answer will you give him?
Viewing 1451 - 1460 interview questions