The difference between when the set of Merge, ChangeLog, and Sync operations is executed only once and when it is executed twice during ontology synchronization seems unclear. Could you explain the reason for this difference?
There are many reasons why a job tracker view of Funnel pipeline jobs and backing dataset updates can be more complex than a simple one-to-one correspondence between dataset update → Changelog → Merge Changes → OSv2 sync jobs. These are not generally causes for concern. If you have reason to believe that in your environment, you are seeing something out of the ordinary that is having a negative impact in some way, please contact your Palantir support team. For reference, here are just a few of the factors that can cause what you’re seeing.
① Changelog, merge, and sync jobs are pipelined together sequentially, but independently. The triggering conditions for each of these operations is as follows - Each phase is triggered upon (1) the previous phase completing and (2) no existing operation in a previous phase being in flight. Because of (2), we can end up with changelog running twice but sync running only once, for example.
② Per https://www.palantir.com/docs/foundry/object-indexing/funnel-batch-pipelines/#live-pipelines, live pipelines run every six hours regardless of any backing dataset update if user edits on objects are detected.
③ We are tracking a minor bug that can cause extra Funnel batch pipeline triggers on a single dataset update in some cases. It appears that when this happens, the extra triggered jobs exit without doing any substantial computational work, so this is an issue with very minor impact, but noting it here because if this bug if fixed in the future, you may see this phenomenon happen less.
2 Likes
Thank you for your response!