Can I have multiple APPENDs in a single incremental spark transaction?

I have a spark job that runs for a long time. And I get issues like POD not responding after the build is running for a long time (1day) and the build retries. So, I loose 1 day worth of computation.

The output is an incremental dataset and I’m performing APPEND transaction. Is it possible to periodically write the result to the dataset during the job?

I tried using “.write_dataframe” method to flush the data periodically but foundry didn’t allow it.

Hi,

You cannot periodically commit partial results within a single Foundry Spark transform job. The output transaction is committed when the job completes successfully. APPEND changes how the committed data is added to the dataset and does not enable partial commits during execution.

write_dataframe() is not a flush or checkpoint operation, and it can only be called once per output in a job.

To preserve progress, I would suggest splitting the computation into smaller jobs. If your input contains multiple append transactions, Foundry supports incremental input transaction limits, such as Input(..., transaction_limit=1). Each successful job processes and commits a batch, so a later failure does not discard previously completed batches. You can configure the schedule to re-trigger after a successful build until the output catches up.

See Limit batch size of incremental inputs.

Hi Maxh, Thank you, it’s helpful. But i have one doubt.

Transaction limit cannot limit the number of records that the transform view right?

Say if I have 100k records added in each transaction. In this case, even if I limit the number of transactions lookup to one, I would end up with with 100k rows.

Is there a way to restrict the number of incremental rows that the transaction can look up?

Hi,

To your first question, transaction_limit=1 limits the input to one whole transaction, so if that transaction contains 100,000 rows, all 100,000 rows are available to the transform. To my knowledge, there isn’t an equivalent built-in row limit that automatically resumes from the remaining rows in the next build. See incremental transaction limits.

FYI, I would refrain from simply applying .limit(10000) to the incremental input as after a successful build, Foundry records the input transaction as processed. The remaining 90,000 rows would not automatically appear in the next build’s incremental input.

Instead, to process smaller batches safely:

  • Create smaller upstream transactions, if you control ingestion. For example, append 10,000 rows per transaction, then use transaction_limit=1 downstream.
  • Implement explicit row-level progress tracking. For example, keep unprocessed rows in a separate backlog output dataset, read that backlog in the next build, process a bounded batch, and persist the remainder alongside the results. Foundry supports reading previous output state. You would also need to trigger further builds until that backlog is empty.

Hope that helps!