Batched Execution in Python

Ask the Community
,
1

I can’t find any clear references in the documentation, but does Python support batch execution for an ontology action?

I ran into an issue where the user needs to upload a csv file to an object. Using the standard create action throws an error if there are more than 20 rows using a function baked action. Our FDE recommended using batched execution with typescript v1. Our team uses Python usually so if like to keep our code as much Python as I can.

2

What kind of error are you getting exactly? Too many edits at once? Or timeout issue?

3

Hi there,

A few approaches you could take on this:

1 - Write back action
Python functions support ontology edit actions which return a list of ontology edits, so instead of trying to edit the ontology on a per row basis, the python function would handle editing (creating) multiple objects inside a single function call

2 - Pipelining
Is there a reason this data cannot be pipelined into the ontology? Tabular data in a CSV file can be easily pipelined into the ontology.

Is this creation of objects from a CSV file something you expect to happen multiple times or is this a one time operation? Is the object you are trying to update already backed by a pipelined dataset or is a purely an ontology edits based object type?

This will give us a better idea as to what the best solution is

thanks

4

Thanks for the reply and clarifying questions!

Also this is a user facing application (workshop app) where 30 or so analysts will upload their reports on a regular basis and have to make edits to the rows in those reports when needed. It’s a purely ontology edits based object. This app is the source for this data.

  1. We did start off with a python ontology edit function. The issue was when trying to apply more than 20 rows in the workshop app, the action failed saying there is a 20 row limit. This is what led us to using the batched-execution typescript solution (which works pretty well; just becomes a liability because none of us know typescript)
  2. Pipeline i don’ t think makes sense because I still have to have the ingestion point at the workshop app/user-facing level. Plus there would be a good bit of overhead i’d think (a few mins for the pipeline to run and populate the ontology so the users can see their uploaded data. Then we have to do error handling and these users are business users so they won’t be able to troubleshoot a pipeline failure)
5

Super interesting, thanks for the clarification

Given that this object is purely edits based, you are right, pipelining is not the way to go.

I would be super interested to see what your current implementation of the python function looks like. This should inform why you are running into the rate limits of 20 object changes per action.

From my understanding, if you used the platform SDK to pull the CSV file from a media set (after being uploaded in the front end) into your python function, you should be able to (for each row) create a new ontology object (limit is way higher than 20), add it to the list of edits, and return all the new objects at the end of the function.

I think the next steps is to figure out why your current python function implementation is limiting you. Could you share code snippets / implementation logic?

6

Yea, let me add a bit more context too. The users are adding data via the Inline Action Form or Table widget.

image
image972×199 13.6 KB

This adds the entries to a forecast object in the ontology. The import data button let’s us submit a CSV file but if there are more than 20 rows, using the ontology action create forecast throws a 20 row limit error.

Here’s the general code.

@function(edits=[Forecast])
def create_forecast(
    p01: date,                  # period anchor
    p02: str,                   # id component 1
    p03: date,                  # creation date
    p04: str,                   # creator
    p05: str,                   # category A
    p06: bool,                  # flag
    p07: str,                   # id component 2
    p08: str,                   # label A
    p09: str,                   # category B
    p10: Double,                # total amount
    p11: Double | None = None,  # component 1
    p12: Double | None = None,  # component 2
    p13: Double | None = None,  # component 3
    p14: str | None = None,     # free text
    p15: Double | None = None,  # component 4
    p16: str | None = None,     # optional grouping
    p17: Double | None = None,  # component 5
    p18: str | None = None,     # optional tag
) -> list[OntologyEdit]:
    ontology_edits = FoundryClient().ontology.edits()

    # Data Validation
    v13 = p13 or 0
    v17 = p17 or 0
    v11 = p11 or 0
    v12 = p12 or 0
    v15 = p15 or 0

    obj_id = f"{p01} | {p02} | {p07}"
    v_cost = v11 + v12 + v17 + v15
    v_expected = v13 + v_cost

    if v_expected != 0 and p10 != v_expected:
        raise UserFacingError(
            f"Invalid input: aggregate amount check failed. Expected {v_expected}, got {p10}."
        )

    if p01.day != 1:
        raise UserFacingError(
            f"Invalid input: period anchor must be first day of month. Received {p01}."
        )
    # Create Forecast
    ontology_edits = FoundryClient().ontology.edits()
    forecast = ontology_edits.objects.Forecast.create(obj_id)
    forecast.p02 = p02
    ...

    return ontology_edits.get_edits()
7

Ahhhhh right that would be the problem. There IS a limit on 20 actions when editing via an inline form.

Is there a reason why you want to do data upload via an inline form specifically?

If it was completely up to me here is how I would orchestrate this upload process:

  1. Create an ontology object called “CSV upload” or something similar
  2. Use the Media Uploader widget that will take the CSV and upload it to a Media Set
  3. Upon the successful upload, run an action to create a new “CSV upload” object and set the media reference property to the “Uploaded File” in the Media Uploader widget configuration (see notional configuration in image below)
    image
    image624×933 64.5 KB
  4. Trigger an automation that runs on all newly created “CSV upload” to parse the CSV file in the media reference property and create new ontology objects

Please let me know if that would work in your use case / if you need any further clarification

1 Like
8

that’s a pretty slick way to do it. I’m gonna work on it and let you know how it goes in a day or two. Thanks for the help!

1 Like