Dynamically Generating Transform

Ask the Community
,
1

Hi,

I hope you are well. I am trying to dynamically make multiple output datasets based on a list of files paths I have. I managed to do this when I have the list of file paths written out manually.

However i want to do this completely dynamic, by fetching the list of files. However can I do this? Because I’m not allowed to use the transform decorator. Is there anyway I can fetch information from a dataset, file, media set without using that decorator?

Also I understand that in a production workflow this isn’t common as we normally define our datasets and schemas, however I was doing this for a specific use case and just wanted to know if it’s possible.

Thanks

Sam

2

I recently ran into a similar issue and have not solved it. Two resources that might be helpful though are:

Here’s a full workflow that might help you pull the input file RIDs: Platform governance application using the Platform SDK.

If that doesn’t work, there’s also an API for getting children of a folder: List Children Of Folder • API Reference • Palantir

3

Thanks for this, I’m very interested in the first link. That will be useful.

However I am writing all my csv files into one dataset, thus the links don’t help me obtain the actual data in the dataset. Which is what I need. I’m hoping there an api I can use to obtain this.

Kind regards

Sam

4

Could you use pipeline builder to extract the file information?

5

Yes for sure, but that’s not the bit I’m stuck on. I need to get this data inputted into my python file without using the transform decorator so I got a list of files to dynamically make the transform for each individual dataset.

I would never need to do this for most use cases if ever, however I’m interested in if it’s possible.