Thank you for the detailed explanation Ariel ! We used Redis in the process of de-duplication by storing the file name in Redis in order to avoid copies of the same S3 file into Redshift.