Object type not syncing Timestamp property from backing dataset after pipeline backfill (dataset shows values, object type does not)

Ask the Community
, ,
1

I’m working on an ontology-backed application with an external frontend and a Node.js backend proxy that calls Foundry actions. The create object rule action works reliably for creating objects and updating related objects. The challenge is setting a dynamic timestamp on new objects (since create object rules can’t set “current time”).

What I tried before:
I initially attempted a function-backed action (Python) to set timestamps directly, but the function parser consistently failed with AttributeError: create_object even with minimal code. A TypeScript function repository also had module resolution issues. Since the create object rule action works perfectly, I pivoted to a pipeline-based backfill approach.

Current approach (pipeline backfill):

  1. Object Dataset: Created qc_review_all_objects — a materialized dataset containing all objects (both from initial CSV and those created via actions). This dataset updates automatically as new objects are created.

  2. Pipeline in Pipeline Builder:

    • Input: qc_review_all_objects

    • Transform: Used COALESCE(review_date, CURRENT_TIMESTAMP) to fill null timestamps

    • Output: New dataset qc_review_fixedv2

    • Write mode: Snapshot replace (with review_id as primary key)

    • Build succeeded: ri.foundry.main.build.0f840769-0332-36b9-a387-ddd672863291

  3. Object Type Update:

    • Changed backing datasource from qc_review_fixed to qc_review_fixedv2

    • Verified property mapping: review_date → review_date column (type Timestamp)

    • Migrated edits to new datasource

    • Unpublished and republished the object type

    • Unregistered and reregistered the backing datasource to force reindex

The Issue:

  • Dataset qc_review_fixedv2 preview shows timestamps for all rows, including new objects :white_check_mark:

  • Property mapping is correct :white_check_mark:

  • Build succeeded, reindex completed :white_check_mark:

  • However, in Object Explorer (and Workshop), the review_date property for new objects still shows “No value”

  • Existing objects (from initial CSV) display their timestamps correctly

Troubleshooting already done:

  • Waited >1 hour after reindex

  • Cleared browser cache and refreshed

  • Confirmed new objects exist (object count increased)

  • Verified in Object Explorer (not just Workshop) to rule out Workshop caching

  • Checked that __is_deleted and __patch_offset columns are present (2 of 8 columns unmapped, but review_date is mapped)

  • Tried unpublishing/republishing the object type

  • Tried unregistering/reregistering the backing datasource

Question:
Why is the object storage not syncing the review_date values from the backing dataset? Is there an additional step to force a full refresh of the object type’s data? Has anyone encountered a similar issue where dataset changes don’t propagate to the object type despite correct mapping and successful builds?

Any guidance would be appreciated!

2

Hello - thank you for the question. This sounds like a conflict resolution issue rather than a reindexing issue. Your pipeline updated the backing dataset, but Object Explorer/Workshop show the Ontology object, which combines:

  • values from the backing datasource, and

  • any persisted user edits

The behavior depends on the object type’s conflict resolution strategy, which you can check in the Datasources tab in Ontology Manager. If the object type is using Apply user edits, then existing edits to the property can continue to override values coming from the backing dataset. Switching to Apply most recent value might be more appropriate for this use case, as the newer datasource values can overwrite older edits when the datasource timestamp is newer than the edit timestamp; however, this would affect all dataset-backed properties.

Here is the documentation on how user edits are applied to objects: https://www.palantir.com/docs/foundry/object-edits/how-edits-applied/

1 Like
3

Thank you very much for pointing me toward the conflict resolution strategy. That was indeed the core issue.

After reading your reply and the documentation, I did the following:

  1. Switched the object type’s conflict resolution to “Apply most recent value” and configured the timestamp property correctly.

  2. Updated my pipeline to set review_date = CURRENT_TIMESTAMP for all rows (so the datasource timestamp would be newer than any user edit).

  3. Also modified my backend to pass a specific reviewDate when creating each review.

With these changes, the object type now shows timestamps for new reviews.

The only remaining nuance is that for audit purposes, I need each review to have its actual creation timestamp, not the time of the last pipeline run. To achieve that, I ended up:

  • Letting the backend set the timestamp at creation time.

  • Changing the pipeline to COALESCE(review_date, CURRENT_TIMESTAMP) so it only fills nulls and preserves the backend’s value.

This way, every review gets its own accurate creation time, and the conflict resolution still works because the datasource timestamp (which is only filled when null) doesn’t override the backend timestamp.

Thank you again for the clear guidance – it saved me a lot of time. I’ll mark this as resolved.