we would like to add cost / compute seconds per job to our Foundry Metadata Ontology therefore we would need an API to retrieve that information for a batch of jobRIds.
As a workaround we are already parsing the spark Plan and for static execution we have a way to roughly calculate the compute seconds (even though we are not sure it’s 100% correct).
Given that the variety of jobs are increasing in Foundry (dynamic spark jobs, workspaces, builder, builder lightweight…) we can’t keep up with re-engineering the compute seconds logic and would like to request an API where we can pass in a batch of jobRIDs and get the compute seconds plus compute seconds converted into currency after any applied discounts as response.
Thanks for reaching out! A few additional questions which will help with scoping this feature request:
How much does retention period matter here? Are you typically interested in looking at job level data over the past week? The past month? The past year?
How do you anticipate this data will fit into user workflows? Are users interested in immediate access to this data for improving/optimizing transforms? Tracking changes in job usage over time? etc. Any information you can share here about how you anticipate this data will be used will be helpful!
Permissions wise, who do you expect to be able to access this data? Users who already have access to RMA? Or users who are working on the transforms themselves?
good to see you again after all those years and thanks for the great questions.
How much does retention period matter here? Are you typically interested in looking at job level data over the past week? The past month? The past year?
In our Foundry Metadata Ontology we are currently storing ~21M Jobs going back roughly 1.5 years. I’d like to add a column with the true compute seconds and cost associated with each job. Retention would be handled by us for our copy of the data from the API. For the actual API I would be finde with retention of 6 month or so. Naturally the existing jobs API would be a good fit to return this data. I am okay with a lag of computation - data only being available after a few hours (but directly available would be super cool )
How do you anticipate this data will fit into user workflows? Are users interested in immediate access to this data for improving/optimizing transforms? Tracking changes in job usage over time? etc. Any information you can share here about how you anticipate this data will be used will be helpful!
Taking a step back: We currently have two parallel user workflows that are both not ideal: a) processes & automation around our own Metadata Ontology: Pros: We can tie it easily to our own governance objects such as Use Cases or Data Treasury Items and associated properties or linked properties Cons: We are more or less rebuilding RMA one step at a time b) access to RMA: This is unfortunately not very well integrated into Compass and does not provide very granular permissions. We have given access to “Everyone” but there is no entry point from a Compass Project or from a Build. In addition RMA provides only resource level granularity and there is no drill down into individual Jobs which makes optimizing / comparing Job Compute Seconds almost impossible (unless you only send one job a day )
If there was a greenfield I would like to have RMA deeper integrated into Compass Folders (for example as layover) and RMA with Job level granularity.
Permissions wise, who do you expect to be able to access this data? Users who already have access to RMA? Or users who are working on the transforms themselves?
This would be for the developers of transforms, pipeline builder, … all builders on the stack. More granular permission for RMA data would be appreciated. Maybe connecting visibility to membership of -editor group on a project? Maybe -viewer - not sure. I think currently it’s all or nothing in RMA which is not really helpful.
It’s good to see you too! Happy to be able to action feature requests this time around instead of passing on signal .
Thank you for the additional context, and we’ll be sure to keep you in mind to help us with beta testing when we’ve got a V0 of this feature to roll out. We’re working through an RFC for a prototype, so stay turned.
Independent of job level data, the Resource Management team is currently working on a feature to make project level usage visible to project owners via an overlay in compass. We’ll make sure to communicate when that’s available as well!
and thanks for the great questions.
)
)