Enable API endpoint to receive streaming events

Hello, Im currently facing a challenge trying to extract the data from a source that acts the following manner:

  • In the source, you have to declare an endpoint

  • In Palantir, you have to expose an endpoint

  • The source, when some requirements are met, will POST a request to Palantir Foundry containing a JSON file in the body calling the endpoint.

Ideally, we would like to land the information into a streaming dataset so we can process the information in “streaming” mode, but meantime, batch would be a good approach, how I can proceed with such implementation?

I cant see a data connector that allows me to do that. Ive read that maybe this is achievable through compute modules…

Regards

If you’re able to authenticate with Foundry (potentially meaning writing some custom code in your source), you can utilize our streaming endpoints here (non-beta versions available here).

Otherwise, more permissive (receiving webhooks, using alternative forms of authentication, less structured, etc.) creation of ingest endpoints is currently in the works, so keep an eye out for announcements in the coming months.

Alternatively if the streaming endpoint approach doesn’t work for you, you may choose to, for example, write an AWS lambda function that receives the requests and forwards them to the streaming endpoint, or some other middleware approach where you land the data in some database and do a standard data ingest (depending on your latency requirements, this may be insufficient).