Is there a way I can programatically store all the information related to any metadata, changes or events related to specific lineages in a table/object that I can later query or analyze.
Consider this table/object to have all the info like file location, file inputs, file outputs, file rid, filename, file type (py/sql/dbo or csv) etc.
Hey @upangotra there isn’t currently any transform or function in Pipeline builder that supports this.
I’ll let the folks from code repositories comment for their end
Hey @upangotra you probably will need to query directly the relevant Foundry API endpoints, you can do it via code repository or directly from data connectors.
Palantir have also prepared two libraries that will give you easy programmatic access via Python to some of the endpoints (e.g. list all files related to a dataset)
foundry-platform-python
https://github.com/palantir/palantir-python-sdk/ (deprecated)
The responses you can save as your metadata dataset and schedule it.