My ontology indexing costs are high. What can I do to decrease them?

Ask the Community
1

Hi all, after seeing quite a few Funnel syncs, wanted to share some strategies for decreasing Funnel costs. Curious for what has worked best for you!

2

Before going any further, it can be helpful to understand how Funnel indexing works. Here are some docs on the process. In short, a Funnel batch pipeline has four stages. Steps 2–4 scale with changed rows. Step 1 is the one that scales with how the data arrives.

  1. Changelog – Funnel computes the data difference for each datasource and writes it to an internal (user-inaccessible) changelog dataset as APPEND transactions.
  2. Merge changes – changelog rows plus any recent user edits from Actions are joined on the object type’s primary key.
  3. Indexing – one job per object database converts merged rows into index files.
  4. Hydration – the object database (OSv2) downloads index files onto its search node disks; this is the last step before the data is queryable.

Now, on some more general Funnel cost reduction strategies, I’ll start with easiest and most common (from my experience), then get to a few more in depth options:

  1. Sync the data less frequently. This one may seem obvious, but it’s important to analyze if you really need the data on the cadence it’s currently syncing at. What would the business impact of decreasing the frequency be? Most of the time, a 3am schedule is unnecessary and wasteful.
  2. Reduce the number of rows that have any changes. The changelog stages “incrementalizes” everything for all stages downstream, meaning if 1% of rows change then all downstream stages (merge changes, indexing, hydrating) are super quick. If a single column changes on every single row, then all downstream stages have to reprocess 100% of the data, costing significantly less - which then also has a chain reaction effect of causing much more frequent background replacement pipeline refreshes to rebalance the shard indexes. Some common pitfalls: a current timestamp / date column that changes on every row on every run, or an array of strings that is non deterministically sorted (this is a massive one, you need to sort every “collect distinct” array to avoid this for example).
  3. Shrink the object itself. Indexing compute is driven by number of records, number of properties per object, and size of each property — a large text property costs much more to analyze and index than a numeric one (Compute usage: Ontology indexing). Dropping unused properties and moving big blobs of text out of frequently-reindexed object types is often a faster win than restructuring for incrementally.
  4. Consider switching automates from live monitoring. Automates which are set to “live monitoring” have a separate indexing job, and may drive additional cost. If your object is used anywhere in a non-deleted automate (even if the automate is paused), then it will be indexed an additional time at extra cost. If you can switch those automates to running on a schedule then you can eliminate up to half of your Funnel cost for that object. Note the automate performance best practices as well.
  5. Consider moving away from streaming object types. Funnel streaming pipelines are always-on, cannot be cancelled by users. If you do not need data at a very low latency, this might be unnecessarily costly.
  6. Use incremental/cdc/iceberg. If you are constantly snapshotting your data, it’s worthwhile to consider if an incremental approach to changes in data would be possible. There are several different potential flavors to this, and each of them could probably use their own post, but I’ll go through a couple basics below. CDC/Iceberg in particular both contain their own nuances and complex set of tradeoffs.
    1. Snapshot - What happens: each build replaces the entire dataset view with a new set of files. Funnel then has to read the full new view and diff it against prior state to produce the changelog, so the changelog stage’s cost scales with total dataset size on every single run, even if nothing changed. Only the diff flows through merge/index/hydrate. If >80% of rows change in a given transaction, you will get a full re-index.
    2. Incremental - The datasource receives APPEND transactions containing only new or changed rows. Funnel’s changelog stage therefore only needs to read the newly added files rather than the entire dataset—this is where the actual cost savings occur. OSv2 resolves object state using a “most recent transaction wins” strategy: if a primary key appears in multiple transactions, the row from the latest transaction is written to the Ontology. In theory, the only performance difference between snapshot and incremental processing is the cost of the changelog stage. Once that stage has “incrementalized” the input, every downstream stage processes only the changes. For example, if just 1% of rows change, all downstream stages should complete very quickly.
  7. Use multi-dataset backed objects (MDOs). Objects can be backed by multiple datasources. This can be really helpful if you have some properties which update very frequently (ie inventory quantities updating hourly) and others updating very infrequently (ie plant location). However, MDOs cannot be used with object security policies, so you must use restricted views for any granular policies, which can fragment your security controls. It also can mean splitting up your pipeline, which can be a time consuming exercise.
3

So are you saying that incremental datasets can back ontology (without a view dedupiing them first)? It resolved primary key collision based solely on the latest transaction winning?

4

Correct - You can look at https://www.palantir.com/docs/foundry/object-indexing/funnel-batch-pipelines#incremental-and-full-reindexing for more details.

The main thing to make sure, is to not have any duplicated primary key in a given transaction (otherwise, there is no way to know which record takes precedence over each other)

5

This is helpful, went to create yet another post on it then found yours :wink:

Trying to help a customer get a handle on high ontology indexing spend and wanted to quantify some of the levers. Results / conclusions from a series of tests I ran below.

Based on these tests + prior experience, this is my understanding of the key variables that impact spend in order. Are these #s indicative of what you would expect?

# Cost driver Our understanding and observations
1 Backing-dataset update frequency More frequent updates drive more indexing runs.. e.g., a 3-minute schedule has 5Ă— as many runs as a 15-minute schedule. Would spend scale roughly with that when each run processes a small delta?
2 Stream vs dataset backing Streaming used ~72% less indexing compute than the comparable batch cohort. However benefits are situational given the shared CDC connector/transform spend, always-on costs and ofc the complexity that comes w/ operationalizing streaming.
3 Number of backing datasets Three backing datasets used ~60% more indexing compute than joining those inputs upstream into one incrementally updated backing dataset. Including the join/transform cost, the three-source path still used ~37% more total compute. Your example suggests separating frequently/infrequently updated properties can reverse this—is their update cadence the deciding factor?
4 Snapshot vs incremental updates The snapshot-backed version used ~17% more indexing compute than the incremental one. Based on your explanation, we’d expect the extra work primarily in the changelog stage.
5 Number of records Very minimal impact at the tested scale. 1K, 10K and 50K baseline-population arms were within ~1–2% of each other. Does this hold at larger scales if the number of records changing each day is the same, and when would segmenting operational/analytical data help?

Other variables I could foresee impacting compute but haven’t quantified. What impact should we expect from these?

# Variable What should we expect?
6 Duplicate-key history The client already deduplicates within each incremental transaction to pick the right row per key. If those rows are appended, that sounds like the intended pattern. Does retaining older versions across transactions add meaningful cost during normal indexing or replacement pipelines?
7 Data volume and change patterns Your timestamp/array examples explain how unchanged business data can still trigger indexing. Beyond avoiding that churn, how much do additional properties, large text/arrays, deletes and user edits affect compute?
8 Dataset layout Do compaction, partitioning, sorting or projections help Funnel? Can many small files substantially increase cost?
9 Replacement pipelines Your shard-rebalancing explanation helps explain replacements without repo/Ontology changes. What other performance heuristics trigger them, and can we inspect the reason or reduce their frequency?
10 Direct Ontology writes Should we expect lower total compute than dataset or stream backing, or mainly lower latency? There are a lot of described downsides, is this intended for external to palantir writes or similar or what pattern is this looking to unlock?
11 Explicit dataset schema Does explicitly declaring and enforcing column types reduce indexing work versus an inferred schema with the same resulting types?
12 Other “black magic” / data guarantees Beyond the live-monitoring Automate overhead you mentioned, are there guarantees or settings that let indexing do less work? What breaks those assumptions or forces a more expensive path?