TransUnion Interview Question

Remove duplicates from spark dataframe without using in built function. Handle null values.