Receive Large Pushes from External Source

mmagid · June 24, 2026, 6:29pm

I am trying to receive a push from an external source (Oracle Fusion I believe). Initially we tried this via streams, but the payload did not conform to standard stream schemas. We tried listeners next, but some of the json payloads exceeded the 1 MB payload size limit (maximum payload is ~5 MB). I am experimenting with custom endpoints next and compute modules, but I couldn’t get either working. Does anyone have any other thoughts of what to try? Or an idea on how to get compute modules working?

nicornk · June 25, 2026, 8:58pm

Many systems do support s3 - did you explore the s3 compatible API to push files into a foundry dataset?

mmagid · July 28, 2026, 7:46pm

@nicornk solution worked for me, but I wanted to write out a step-by-step guide for others interested in my approach for exposing an ingestion endpoint that an external client can call with a JSON payload of any size.

1. Overview

The goal: an external client sends a (possibly large, deeply-nested, schema-drifting) JSON document, and Foundry ingests it and updates automatically. Foundry streaming records are capped at ~1 MB each, so documents can exceed that limit. The robust pattern is therefore two channels:

Upload the full document as a FILE via Foundry’s S3-compatible API (no ~1 MB cap; supports multipart for very large files).
Publish a small POINTER record (order id + file path + metadata, well under 1 MB) to a Foundry STREAM, to trigger low-latency downstream processing.
A streaming/batch pipeline reads the pointer, loads the referenced file, parses it, and writes a clean table that backs an Ontology object type.

Flow: External client → (S3 PutObject) → Payloads dataset + (publish pointer) → Pointer stream → Pipeline Builder parse → Clean dataset → Ontology object.
If you do not need real-time triggering, you can skip the stream entirely and just upload files (Step 6a) — a scheduled batch pipeline over the payloads dataset is simpler. The pointer stream is what makes it event-driven.

2. Naming used in this guide

Placeholder	Meaning
`<FOUNDRY_URL>`	Your stack host, e.g. https://acme.palantirfoundry.com
Project “PO Ingestion”	The Project (Compass folder) that holds all resources. RID looks like ri.compass.main.folder..
`po_payloads_raw`	Dataset that stores the full JSON documents as files (the S3 “bucket”).
`po_pointers`	Stream with a single string column ‘payload’ holding the small pointer JSON.
App “PO External Uploader”	Third-party OAuth2 application whose service user the client authenticates as.

3. Prerequisites

Edit access to a Project where these resources will live.
Permission to register a third-party application (Developer Console or Control Panel).
For STATIC S3 credentials: the ‘User experience administrator’ role on the Organization (required to mint S3 keys). If you do not have this, use TEMPORARY credentials (Step 6a, Option B) instead.

4. Create the Foundry resources

4a. Payloads dataset (the S3 bucket)

In your Project, select + New → Dataset and name it po_payloads_raw. It can start empty; files land in it via the S3 API. The dataset’s RID (ri.foundry.main.dataset.<uuid>) is the S3 bucket name. See Datasets.

4b. Pointer stream

Select + New → Stream. Define a single-column schema payload: String and Normal throughput (one partition is fine). Full walkthrough: Push data into a stream. A single string column deliberately avoids schema validation, so the pointer JSON shape can evolve freely.

5. Register the app and set permissions

5a. Register a third-party OAuth2 application

Create a Confidential client with the Client Credentials grant enabled. Record the Client ID and secret. Enable the streams-write operation on the app. Guides: Third-party applications, Writing OAuth2 clients, Register a third-party application.

5b. Grant the service user access (the single key step)

Each app has a service user (find it on the app’s Manage page). Add that service user as an Editor on the Project (or share the specific stream and dataset with it as Editor). This one grant covers BOTH channels:

S3 read/write requires s3-proxy:datasets-read / s3-proxy:datasets-write, granted by default to Viewer / Editor.
Publishing to the stream requires Editor on the stream.

Reference: the Authentication section of the S3 API guide and Part 3 (“Share the stream”) of Push data into a stream.

6. Give the client its credentials

6a. S3 credentials for the file upload

Option A — Static credentials (long-lived, recommended for a service). Requires the ‘User experience administrator’ org role. Generates an access key/secret tied to the app’s service user and restricted to specific Projects:

curl -X POST \
  -H "Authorization: Bearer <TOKEN>" \
  -H "Content-type: application/json" \
  --data '{"clientId":"<CLIENT_ID>","projectRestrictions":["<PROJECT_RID>"]}' \
  https://<FOUNDRY_URL>/io/s3/api/v2/credentials

# Response contains AccessKeyId + SecretAccessKey. List: GET .../credentials  Revoke: DELETE .../credentials/<ACCESS_KEY_ID>

Option B — Temporary credentials (<= 1 hour, no admin role). Exchange any valid Foundry token (e.g. a client-credentials token) via the standard STS AssumeRoleWithWebIdentity call. Returns AccessKeyId + SecretAccessKey + SessionToken:

curl -X POST \
  "https://<FOUNDRY_URL>/io/s3?Action=AssumeRoleWithWebIdentity&WebIdentityToken=<TOKEN>"

# RoleArn / RoleSessionName are ignored by Foundry but AWS SDK STS clients require them to be set.

S3 client settings (all clients): Endpoint https://<FOUNDRY_URL>/io/s3, Region foundry, Path-style access true, Bucket = dataset RID. Details & AWS CLI / Spark / Cyberduck examples: S3-compatible API for Foundry datasets.

6b. A token for publishing the pointer

The client mints an OAuth2 token from https://<FOUNDRY_URL>/multipass/api/oauth2/token (grant_type=client_credentials) with the streams scopes. The exact push URL + ready-made code is generated for you on the stream’s Connect via API → Push page — use that to get the stack-correct endpoint. See Push data into a stream.

7. What the client runs

A single script does both channels: upload the full payload to S3, then publish the pointer to the stream.

import json, boto3, requests

FOUNDRY = "https://<FOUNDRY_URL>"
PAYLOADS_DATASET = "ri.foundry.main.dataset.<payloads-uuid>"
POINTER_STREAM   = "ri.foundry.main.dataset.<stream-uuid>"
CLIENT_ID, CLIENT_SECRET = "<id>", "<secret>"
S3_KEY_ID, S3_SECRET = "<access-key-id>", "<secret-access-key>"  # from Step 6a

def ingest(order_number, payload_bytes):
    key = f"incoming/{order_number}.json"

    # 1) Upload the full document as a file (no ~1 MB limit; multipart is automatic)
    s3 = boto3.client('s3', endpoint_url=f"{FOUNDRY}/io/s3", region_name='foundry',
                      aws_access_key_id=S3_KEY_ID, aws_secret_access_key=S3_SECRET,
                      config=boto3.session.Config(s3={'addressing_style': 'path'}))
    s3.put_object(Bucket=PAYLOADS_DATASET, Key=key, Body=payload_bytes,
                  ContentType='application/json')

    # 2) Publish a small pointer record to the stream
    token = requests.post(f"{FOUNDRY}/multipass/api/oauth2/token", data={
        'grant_type': 'client_credentials', 'client_id': CLIENT_ID,
        'client_secret': CLIENT_SECRET, 'scope': 'api:usage:streams-write api:streams-write'},
        headers={'Content-Type': 'application/x-www-form-urlencoded'}).json()['access_token']
    pointer = {'orderNumber': order_number, 'datasetRid': PAYLOADS_DATASET,
               'filePath': key, 'sizeBytes': len(payload_bytes)}
    r = requests.post(
        f"{FOUNDRY}/api/v2/highScale/streams/datasets/{POINTER_STREAM}/streams/master/publishRecords",
        data=json.dumps({'records': [{'payload': json.dumps(pointer)}]}),
        headers={'Authorization': f'Bearer {token}', 'Content-Type': 'application/json'})
    r.raise_for_status()   # 204 on success

Note on the publish URL: use the exact endpoint shown on the stream’s Connect via API → Push page for your stack. If a publish returns a server error, confirm the token includes both ‘api:usage:streams-write’ and ‘api:streams-write’ (the second lets the service resolve the stream), and that the service user has Editor on the stream.

8. Downstream: parse and update the Ontology

Create a Pipeline Builder pipeline that consumes the po_pointers stream, reads each referenced file from po_payloads_raw, parses the JSON (Parse JSON expression) into typed columns, and writes a clean output dataset. Route unparseable rows to a separate rejects output. Then back an Ontology object type (e.g. ‘Purchase Order’) with the clean dataset. Helpful guides: Create a streaming pipeline with Pipeline Builder and Integrate your stream with the Ontology.
Simpler batch alternative: skip the stream, and run a scheduled batch pipeline directly over po_payloads_raw that parses any new files. See Datasets and streaming vs. batch.

9. What to hand the client

Item	Used for
S3 endpoint https://<FOUNDRY_URL>/io/s3, region ‘foundry’, path-style	Connecting the S3 client
S3 access key ID + secret (+ session token if temporary)	Uploading the full payload
OAuth client ID + secret	Minting a token to publish the pointer
Payloads dataset RID + pointer stream RID + publish URL	Targets for the two calls

10. Limits & sizing

Streaming records are capped at ~1 MB each (both string and binary). Above ~1 MiB you get 413 RecordTooLarge. Keep the pointer well under this.
The S3 / file path has no ~1 MB cap (multi-GB, multipart supported) — this is why the full document goes to S3, not the stream.
S3 API supports path-style only; the bucket is the dataset RID and dataset RIDs contain ‘.’, which some Hadoop clients dislike (workaround: replace ‘.’ with ‘-’).

11. Security best practices

Restrict static S3 credentials to only the Project(s) that hold the payloads dataset (projectRestrictions).
Give the service user the least role needed — Editor on this Project only, not broad access.
Never commit client secrets or S3 keys; inject them via environment variables / a secrets manager. Rotate on exposure.
Prefer temporary S3 credentials for interactive/user workflows; use static credentials for a dedicated service identity.

12. Validation status (what was tested)

Step	Status
Create stream + publish string/binary record	Verified end to end (204).
Streaming ~1 MB record cap	Verified: 413 RecordTooLarge at 1 MiB.
Upload large file (4.2 MB, 10 MB) to a dataset	Verified end to end (200, committed).
S3 temporary credentials (STS AssumeRoleWithWebIdentity)	Verified: returns access key/secret/session token (200).
S3 PutObject with those credentials	Confirmed via public docs (standard boto3). Not run from our sandbox because its internal proxy host rejects S3 Host validation; this does not affect an external caller on <FOUNDRY_URL>.
S3 static credential generation	Requires ‘User experience administrator’ org role (per docs); confirmed our service user lacked it (403 needs s3-proxy:manage-credentials).