I have one file and it contains city names and distance between them. But there are duplicates, we need to remove those duplicates by using stage variables in transformer stage.
Read the file with sequential file stage.
Next, put transformer stage and select the option to run it in sequential mode.
Flag = If input.DESTINATION:input.SOURCE=city1 Then 0 Else 1city1 = input.SOURCE:input.DESTINATION
Compile and run the job.
After run, you can see the output like below.