I was wondering what is the best way to export data from Foundry to a local Kafka setup on my computer, for development.
Is it possible to do this using Data Connection and creating a Kafka source? Or is there another way to do this for local setups?
It is possible to export data from Foundry to a local Kafka setup using the Data Connection feature by creating a Kafka source. The process involves setting up a streaming export to Kafka, which allows you to stream records to your local Kafka setup as long as the export job is running in Foundry. You need to enable exports for your Kafka connector and then create a new export to your local Kafka setup.
https://www.palantir.com/docs/foundry/available-connectors/kafka#export-data-to-kafka
https://www.palantir.com/docs/foundry/data-connection/export-overview#streaming-exports
I created a new data source connection to Kafka, but I am not sure if I should select direct connection or agent. If I choose agent, do I need to install the agent on my local computer?
That’s right - I’d recommend using direct connection which would be quicker to setup.