I have a sync that pulls in entire files from an SFTP source. Sometimes the files are CSVs and sometimes they’re zipped CSVs, and when they’re zip files the Pipeline Builder transform that I’m using to parse the files (extract rows from a CSV file) breaks. Is there any way to handle this in Pipeline Builder or Data Source? I’d like to avoid using a python transform if possible.
I can answer on the Pipeline Builder side – currently we don’t support unzipping CSV’s but we’re actively tracking this feature request and can update this when we start working on it
That’s great to hear, is there a projected rollout date for this feature? And will it support some kind of logic that’s able to deal with the uncertainty around whether or not the file is zipped (as far as I’m aware, I can’t use a Case statement to transform a file with no schema – is that correct?)
I misread this as the feature being actively developed, my bad! An update when work starts would be great