# My ontology indexing costs are high. What can I do to decrease them? **URL:** \ **Category:** Ask the Community\ **Tags:** ontology-management\ **Created:** [September 4, 2026, 2:10pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178 "2026-09-04T14:10:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ckucera](https://avatars.discourse-cdn.com/v4/letter/c/f0a364/32.png) [@ckucera](https://community.palantir.com/u/ckucera)\ **Post date:** [September 4, 2026, 2:10pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/1 "2026-09-04T14:10:05Z")
Hi all, after seeing quite a few Funnel syncs, wanted to share some strategies for decreasing Funnel costs. Curious for what has worked best for you! ---
**Author:** ![ckucera](https://avatars.discourse-cdn.com/v4/letter/c/f0a364/32.png) [@ckucera](https://community.palantir.com/u/ckucera)\ **Post date:** [September 4, 2026, 2:10pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/2 "2026-09-04T14:10:22Z")
Before going any further, it can be helpful to understand how Funnel indexing works. [Here](https://www.palantir.com/docs/foundry/object-indexing/funnel-batch-pipelines) are some docs on the process. In short, a Funnel batch pipeline has four stages. Steps 2–4 scale with _changed rows_. Step 1 is the one that scales with _how the data arrives_. 1. Changelog – Funnel computes the data difference for each datasource and writes it to an internal (user-inaccessible) changelog dataset as `APPEND` transactions. 2. Merge changes – changelog rows plus any recent user edits from Actions are joined on the object type’s primary key. 3. Indexing – one job per object database converts merged rows into index files. 4. Hydration – the object database (OSv2) downloads index files onto its search node disks; this is the last step before the data is queryable. Now, on some more general Funnel cost reduction strategies, I’ll start with easiest and most common (from my experience), then get to a few more in depth options: 1. **Sync the data less frequently.** This one may seem obvious, but it’s important to analyze if you really need the data on the cadence it’s currently syncing at. What would the business impact of decreasing the frequency be? Most of the time, a 3am schedule is unnecessary and wasteful. 2. **Reduce the number of rows that have any changes.** The changelog stages “incrementalizes” everything for all stages downstream, meaning if 1% of rows change then all downstream stages (merge changes, indexing, hydrating) are super quick. If a single column changes on every single row, then all downstream stages have to reprocess 100% of the data, costing significantly less - which then also has a chain reaction effect of causing much more frequent background replacement pipeline refreshes to rebalance the shard indexes. Some common pitfalls: a current timestamp / date column that changes on every row on every run, or **an array of strings that is non deterministically sorted** (this is a massive one, you need to sort every “collect distinct” array to avoid this for example). 3. **Shrink the object itself**. Indexing compute is driven by number of records, number of properties per object, and size of each property — a large text property costs much more to analyze and index than a numeric one ([Compute usage: Ontology indexing](https://rm-demo.palantirfoundry.com/docs/foundry/ontologies/compute-usage#drivers-of-usage-for-ontology-indexing)). Dropping unused properties and moving big blobs of text out of frequently-reindexed object types is often a faster win than restructuring for incrementally. 4. **Consider switching automates from live monitoring.** Automates which are set to “live monitoring” have a separate indexing job, and may drive additional cost. If your object is used anywhere in a non-deleted automate ( **even if the automate is paused** ), then it will be indexed an additional time at extra cost. If you can switch those automates to running on a schedule then you can eliminate up to half of your Funnel cost for that object. Note the [automate performance best practices](https://www.palantir.com/docs/foundry/automate/performance-best-practices#condition-configuration-lever-1) as well. 5. **Consider moving away from streaming object types.** Funnel streaming pipelines are [always-on,](https://www.palantir.com/docs/foundry/object-indexing/funnel-streaming-pipelines) cannot be cancelled by users. If you do not need data at a very low latency, this might be unnecessarily costly. 6. **Use incremental/cdc/iceberg.** If you are constantly snapshotting your data, it’s worthwhile to consider if an incremental approach to changes in data would be possible. There are several different potential flavors to this, and each of them could probably use their own post, but I’ll go through a couple basics below. CDC/Iceberg in particular both contain their own nuances and complex set of tradeoffs. 1. **Snapshot** - What happens: each build replaces the entire dataset view with a new set of files. Funnel then has to read the full new view and diff it against prior state to produce the changelog, so the changelog stage’s cost scales with total dataset size on every single run, even if nothing changed. Only the diff flows through merge/index/hydrate. If \>80% of rows change in a given transaction, you will get a full re-index. 2. **Incremental** - The datasource receives APPEND transactions containing only new or changed rows. Funnel’s changelog stage therefore only needs to read the newly added files rather than the entire dataset—this is where the actual cost savings occur. OSv2 resolves object state using a “most recent transaction wins” strategy: if a primary key appears in multiple transactions, the row from the latest transaction is written to the Ontology. In theory, the only performance difference between snapshot and incremental processing is the cost of the changelog stage. Once that stage has “incrementalized” the input, every downstream stage processes only the changes. For example, if just 1% of rows change, all downstream stages should complete very quickly. 7. **Use [multi-dataset backed objects](https://www.palantir.com/docs/foundry/object-permissioning/multi-datasource-objects) (MDOs).** Objects can be backed by multiple datasources. This can be really helpful if you have some properties which update very frequently (ie inventory quantities updating hourly) and others updating very infrequently (ie plant location). However, MDOs cannot be used with [object security policies](https://www.palantir.com/docs/foundry/object-permissioning/object-security-policies), so you must use restricted views for any granular policies, which can fragment your security controls. It also can mean splitting up your pipeline, which can be a time consuming exercise. ---
**Author:** ![tmundt](https://avatars.discourse-cdn.com/v4/letter/t/b5a626/32.png) [@tmundt](https://community.palantir.com/u/tmundt)\ **Post date:** [September 7, 2026, 8:17am UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/3 "2026-09-07T08:17:26Z")
So are you saying that incremental datasets can back ontology (without a view dedupiing them first)? It resolved primary key collision based solely on the latest transaction winning? ---
**Author:** ![VincentF](https://avatars.discourse-cdn.com/v4/letter/v/8491ac/32.png) [@VincentF](https://community.palantir.com/u/VincentF)\ **Post date:** [September 7, 2026, 8:18am UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/4 "2026-09-07T08:18:48Z")
Correct - You can look at [https://www.palantir.com/docs/foundry/object-indexing/funnel-batch-pipelines#incremental-and-full-reindexing](https://www.palantir.com/docs/foundry/object-indexing/funnel-batch-pipelines#incremental-and-full-reindexing) for more details. The main thing to make sure, is to not have any duplicated primary key **in a given transaction** (otherwise, there is no way to know which record takes precedence over each other) ---
**Author:** ![23dimethylpant](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/23dimethylpant/32/2277_2.png) [@23dimethylpant](https://community.palantir.com/u/23dimethylpant)\ **Post date:** [September 21, 2026, 4:17pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/5 "2026-09-21T16:17:56Z")
This is helpful, went to create yet another post on it then found yours 😉 Trying to help a customer get a handle on high ontology indexing spend and wanted to **quantify** some of the levers. Results / conclusions from a series of tests I ran below. Based on these tests + prior experience, this is my understanding of the key variables that impact spend in order. Are these #s indicative of what you would expect? | # | Cost driver | Our understanding and observations | | --- | --- | --- | | 1 | **Backing-dataset update frequency** | More frequent updates drive more indexing runs.. e.g., a 3-minute schedule has 5× as many runs as a 15-minute schedule. Would spend scale roughly with that when each run processes a small delta? | | 2 | **Stream vs dataset backing** | Streaming used ~72% less indexing compute than the comparable batch cohort. However benefits are situational given the shared CDC connector/transform spend, always-on costs and ofc the complexity that comes w/ operationalizing streaming. | | 3 | **Number of backing datasets** | Three backing datasets used ~60% more indexing compute than joining those inputs upstream into one incrementally updated backing dataset. Including the join/transform cost, the three-source path still used ~37% more total compute. Your example suggests separating frequently/infrequently updated properties can reverse this—is their update cadence the deciding factor? | | 4 | **Snapshot vs incremental updates** | The snapshot-backed version used ~17% more indexing compute than the incremental one. Based on your explanation, we’d expect the extra work primarily in the changelog stage. | | 5 | **Number of records** | Very minimal impact at the tested scale. 1K, 10K and 50K baseline-population arms were within ~1–2% of each other. Does this hold at larger scales if the number of records changing each day is the same, and when would segmenting operational/analytical data help? | Other variables I could foresee impacting compute but haven’t quantified. What impact should we expect from these? | # | Variable | What should we expect? | | --- | --- | --- | | 6 | **Duplicate-key history** | The client already deduplicates within each incremental transaction to pick the right row per key. If those rows are appended, that sounds like the intended pattern. Does retaining older versions across transactions add meaningful cost during normal indexing or replacement pipelines? | | 7 | **Data volume and change patterns** | Your timestamp/array examples explain how unchanged business data can still trigger indexing. Beyond avoiding that churn, how much do additional properties, large text/arrays, deletes and user edits affect compute? | | 8 | **Dataset layout** | Do compaction, partitioning, sorting or projections help Funnel? Can many small files substantially increase cost? | | 9 | **Replacement pipelines** | Your shard-rebalancing explanation helps explain replacements without repo/Ontology changes. What other performance heuristics trigger them, and can we inspect the reason or reduce their frequency? | | 10 | **[Direct Ontology writes](https://www.palantir.com/docs/foundry/object-indexing/direct-datasources)** | Should we expect lower total compute than dataset or stream backing, or mainly lower latency? There are a lot of described downsides, is this intended for external to palantir writes or similar or what pattern is this looking to unlock? | | 11 | **Explicit dataset schema** | Does explicitly declaring and enforcing column types reduce indexing work versus an inferred schema with the same resulting types? | | 12 | **Other “black magic” / data guarantees** | Beyond the live-monitoring Automate overhead you mentioned, are there guarantees or settings that let indexing do less work? What breaks those assumptions or forces a more expensive path? | ---
**Author:** ![Vinh](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/vinh/32/8365_2.png) [@Vinh](https://community.palantir.com/u/Vinh)\ **Post date:** [September 30, 2026, 3:06pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/6 "2026-09-30T15:06:24Z")
A few thoughts after digging into the current Funnel / OSv2 docs — especially for the 6–12 items: **6. Duplicate-key history** My understanding is that keeping historical versions across transactions is compatible with the intended incremental pattern. OSv2 uses “most recent transaction wins” across transactions, with the main constraint being no duplicate primary keys within a single transaction. So I would not expect historical rows alone to force re-indexing on every incremental update; the bigger concern is when something causes Funnel to scan/recompute a much larger history, such as a full reindex or replacement. **7. Data volume / change patterns** This seems like one of the clearest levers. Palantir explicitly calls out number of objects, number of properties per object, and property size as indexing-cost drivers. So beyond avoiding false changes, I’d expect large text/arrays and simply carrying unnecessary properties to matter materially. Deletes and user edits should also be treated as changes entering the Funnel pipeline rather than “free” operations. A useful optimization principle seems to be: `avoid unnecessary changes > reduce properties > reduce property size` The deterministic-array example is especially important here. **8. Dataset layout** I’d separate this from the actual object-indexing cost. Projections/compaction are clearly useful for dataset/query performance and for controlling small-file proliferation, but I don’t see them documented as a direct Funnel indexing-cost reduction mechanism. My guess would be that many small files primarily hurt the input/changelog side through read/metadata overhead, while projection benefits are more obvious for downstream consumers. Would be interesting to know whether Funnel’s changelog stage sees a measurable compute penalty from high file counts. **9. Replacement pipelines** The docs explicitly say Funnel can provision replacement pipelines for performance reasons based on various heuristics, but don’t enumerate those heuristics publicly. The one concrete threshold I found is the ~80% changed-row threshold, where a full re-index can become cheaper than computing and applying the changelog incrementally. It would be very useful if Builds/Ontology Manager exposed the reason for a replacement, e.g. shard imbalance, index fragmentation, changed-row density, schema evolution, etc. That would make this much more actionable from a cost-optimization perspective. **10. Direct Ontology writes** I would be cautious about treating direct writes as a cheaper indexing path. The documented tradeoff is primarily low latency vs throughput, and they seem intended for workloads where batch/stream-backed indexing latency is unacceptable, especially frequent small updates and workflows needing low-latency edits. So I’d expect the main benefit to be latency/architecture rather than a guaranteed lower total compute bill. It would be interesting to see an apples-to-apples cost model for direct writes vs a small incremental datasource. **11. Explicit schema** I’d expect explicit typing to help mainly with upstream pipeline performance, schema stability, and avoiding repeated/dynamic inference. The docs explicitly note that dynamic schema inference has a performance cost. I’m less convinced that explicitly declaring the schema would reduce OSv2 indexing compute if the final materialized schema and values are identical. That feels worth benchmarking separately. **12. “Black magic” / guarantees** The strongest optimizations I can see are really about making incrementalization deterministic: - stable primary keys - at most one row per PK per transaction - deterministic ordering of collections/arrays - deterministic event ordering for streams - avoiding volatile fields that cause logically unchanged objects to look changed - keeping object/property shape as small as practical - avoiding workloads that cross the full-reindex threshold ---
**Author:** ![coffee-operator](https://yyz1.discourse-cdn.com/palantir/user_avatar/community.palantir.com/coffee-operator/32/5223_2.png) [@coffee-operator](https://community.palantir.com/u/coffee-operator)\ **Post date:** [October 1, 2026, 11:16pm UTC](https://community.palantir.com/t/my-ontology-indexing-costs-are-high-what-can-i-do-to-decrease-them/7178/7 "2026-10-01T23:16:53Z")
We’ve found that SNAPSHOT refresh (100% of rows changed) can be cheaper for Ontology indexing on OsV2 relative to INCREMENTAL builds, i.e. for relatively small objects backed by datasets. You have to prove out the right approach.