jkane
June 4, 2024, 2:15pm
1
I can see in the job spec the field maxAllowedDuration
.
This appears to be able to be set with python transforms and magritte extract.
For python transfomrs the allowed_run_duration
argument can be given in the transforms decorator [source ]
For magritte extracts the UI can be used to set the maximum duration:
Doing one of these two things sets a value in the jobspec for maxAllowedDuration
for any datasets produced.
The question is:
For job types that are not these two, is there a way to set the maxAllowedDuration
field? For example, in a dataset generated by java transforms?
Editing the jobspec directly gives a permissions error.
2 Likes
I confirm that editing job specs directly is not a supported user workflow.
For Java, the feature is not generally available yet so there is no documentation, but it will work in a similar way by annotating the transforms method.
jkane
August 27, 2024, 9:56am
3
You can now set the build duration limit in a java transform.
See the documentation for that here: https://www.palantir.com/docs/foundry/transforms-java/advanced-configuration/#maximum-build-duration
package myproject.datasets;
import com.palantir.transforms.lang.java.api.*;
import java.time.Duration;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
public final class FilterTransform {
@MaxAllowedDuration(value = "PT2H")
@Compute
public void myComputeFunction(
@Input("/examples/students_hair_eye_color") FoundryInput myInput,
@Output("/examples/students_hair_eye_color_filtered") FoundryOutput myOutput) {
Dataset<Row> inputDf = myInput.asDataFrame().read();
myOutput.getDataFrameWriter(inputDf.filter("eye = 'Brown'")).write();
}
}
1 Like