Hello,
I am processing large data and it takes a while to get the output and write to Pandas or Spark DF and to finally see the result. Is there a way to stream the output as they process? As far as I understand, write_dataframe or can be called only once.
Interactions with the filesystem() will immediately show up in the Files section of the open transaction of the output dataset on the history tab.
You could write some intermediary results to the filesystem(). files or folder which start with _ (underscore) are considered hidden by many Foundry Filesystem components.
The other option would be to write your progress to stdout (lightweight) or use the logging library (spark based) and use the Build Ui to tail.