i need to access dataset from downstream as an input, if it data exist. Foundry prevent me to do so because its cyclic (i understand the cyclic problem), but i want to overcome the cyclic, and be able to read previous version of already processed data. Is there a way to work around it?
Thanks,
Hello,
I think there maybe two options.
The first is cleaner, and would require using an @incremental transform. The you could perhaps use the previous read mode, see inputs and outputs modes here. That would allow you to access previous versions of the data without having to cyclicly try and load some of the output again from downstream.
A hackier second option would be to re-use the method described in this post where you can use the ctx._foundry.input() method to read a dataset, that isn’t listed in the decorator, or the JobSpec, and so doesn’t trigger the cyclic warning. But this comes with various downsides, which are listed in that post, but in this case you have to be careful that you don’t end up with schedules that build when the last input builds, otherwise you end up in an endless build cycle.