[Incremental] Read output takes a long time (download) and incremental mode is not 100% reliable

Ask the Community
1

Hi,

I am in a situation I need to read the previous state of a dataset.

However I encounter 2 limitations:

  1. Downloading the output dataset sometimes take more than 30s for a light one (50MB), and 2min for a heavy one (several GBs) causing CPU to consume time for “nothing”
  2. Even if my inputs are all as snapshot inputs, sometimes it runs as snapshot

The second issue is a true problem and can cause data loss.
The first one is “only” a waste of resources.

Do you have any insights or recommendations?