Hello,
I am trying to understand why a property value is not propagating from my backing dataset to my object type for newly created objects.
I have created a minimal test object with the following backing dataset and properties:
Next, I have an action to create new objects of this type — which sets Key and Value 1, but not Value 2:
I executed the action to create a new object:
Next, I added this row to the backing dataset, with the goal being to populate Value 2 in my edits-created object:
At this point, what I expect to happen and what actually happens diverges:
- Expectation: my object
72ac8812should have Value 1 of new object from actions (because Conflict resolution strategy is set to “Apply user edits”) and Value 2 of bar (because this property was never edited)
- Reality: the object still has Value 2 of
null after the dataset synced
My expectation is that the conflict resolution happens by property, so “Value 2” which has never been edited should populate from the dataset.
Am I missing something, or is this actually expected behaviour? How can I accomplish what I am trying to do, i.e. being able to create objects first and potentially later having them appear in the backing datasets?
Hi Haavard,
When an object is created via an action, the edit layer records the entire object — not just the properties explicitly set in the action. The properties that weren’t set (like Value 2) are implicitly stored as null in the edit layer (which used to be a key-value pair store in Cassandra back in the days but unsure of the Foundry backend with OSS v2).
With the “Apply user edits” conflict resolution strategy, the edit takes precedence for all properties, including the ones that were implicitly set to null. So even though the backing dataset provides Value 2 = "bar", the edit’s null wins because the system sees it as “this property was part of the user edit.” As a consequence, the edit win strategy will never push backing dataset attributes for a primary-key that gets instantiated at Ontology level before appearing in the backing dataset.
How To Achieve The Desired Behavior
There are a few options:
-
Use “Apply most recent value” instead of “Apply user edits.” Add a timestamp property to the object type for conflict resolution. Have the action set it to now() at creation time, and have the pipeline set it to a later timestamp on each run. The pipeline data will then win for the full row, filling in Value 2.
-
Set all properties in the action — if the action is supposed to create the object but you know Value 2 will come from the pipeline, pre-populate it in the action if possible.
-
Don’t create the object via an action — instead, write a row to the backing dataset (e.g., via a function-backed action that writes to the dataset directly), so the object is created in the pipeline layer rather than the edit layer. Then subsequent pipeline runs can update it without conflict.
Edge Case: Handling Deletions From the Backing Dataset
There is one edge case you need to be aware of: if records ingested from your pipeline are deleted at the source and you need the ontology to reflect that. When using edits-enabled object types, the primary key and its associated property values persist in the ontology even if the corresponding row disappears from the backing dataset. The object will remain discoverable in the ontology indefinitely.
To handle this, the main go to is : incremental pipeline with tombstoning — use an incremental transform that reads its own previous output to detect which primary keys have disappeared from the source. For each deleted record, write a tombstone row with a status like "archived" or "deleted" and a current timestamp. Then manage discoverability at the ontology level (e.g., filter out archived objects in object sets and Workshop modules). Alternatively, or in addition, you can have an event-based workflow with Automate, for instance, that regularly comes and cleans up your ontology.
Thanks
Nico from Sibyl
