Create filter on a dataset based on another dataset and use this values as output names

Ask the Community
1

Hi Community,

I have the following:

def compute():
    transforms = []
    entitys_list = var.entity_list

    for entity in entitys_list:
        def create_transform_function(entity):
            @transform(
                outputTbl=Output(path + '{entity}'.format(entity=entity_name)),
                sourceTbl=Input("ri.foundry.main.dataset.xxxxxxxxxxxxx"),
                )
            def write_entitys(sourceTbl, outputTbl):
		filter the input_data based on entity
                data transformation based on entity
            return write_entitys
        transforms.append(create_transform_function(entity))

    return transforms

TRANSFORMS = compute()

this is working with a fixed entity list, but I would like for the entitys_list be based on a value from another dataset.
I understand that due to Continuous integration (CI) checks we cannot have two transforms in this case. But I wonder if there is another way I am not thinking that I might be able to do this even outside repository.

3

As far as i know it is not possible to have dynamic names for outputs in foundry, due to the creation of datasets only during CI. If you don’t need first class datasets, you could have a process which creates files inside of a dataset, and have control over those names, using the input_ds.filesystem() methods.
Python • Basic transforms • Read and write unstructured files • Palantir