Automate settings and development best practices to optimize resources usage and cost?

Resource management in Foundry mainly relates to 3 factors:

  • compute (CPU time or related)
  • storage (dataset/storage of files)
  • indexed storage (e.g. Ontology)

The exact implementation and configuration of a use-case affects how those metrics are consumed. In other words: a given business need can translate into different implementations, with different resource consumption profiles.
See https://www.palantir.com/docs/foundry/resource-management/usage-types/

Use-cases are often implemented using a variety of tools available in Foundry, for example Automate or Pipeline Builder.

What are the settings and best practices to have performance and resource efficient Automation setup ?

5 Likes

Automate can be a source of high resource consumption if the setup/settings of a given Automate don’t follow best practices.

High level

  • :green_circle: Do things in bulk
  • :orange_circle: If the goal is to manually change a property (e.g. change a property of a manually defined object set) you don’t need automate in the first place.
  • :orange_circle: Be aware of highly frequently updating objects that will generate a lot of automate executions
  • :orange_circle: Be aware of the performance/cost of a single action or function execution. Automating such action or function (in case of Function’s backed Actions) will see its effect(s) multiplied by the scale.

Automate

Automate has 3 levels to be aware of, for cost, performance and parallelism management:

  • Level 1 - What triggers an Automation
    e.g. a time base evaluation, on each object update, or a combination of all
  • Level 2 - How a trigger firing will generate effect
    e.g. for each object trigger an action vs trigger an action for all objects in one go
  • Level 3 - What the effect will do
    e.g. an Action backed by a function that takes only one object as input vs an object set, and that iterate over them vs doing bulk computation on them

If level 1 is not “bulk” then there is no way for level 2 and 3 to be “bulk”.
e.g. trigger on each object update (level 1) then you will have only one object as input, so you can trigger only one action per object (level 2), with only one object as input to process (level 3)

As a high level guidance, rule of thumb: Always do things as bulk as possible.
Note: the exact guidance depends on the context (frequency of updates of objects, business and use-case needs, …)

There are cases where this is not necessary to use Automate at all.
Namely: if you intent to manually change a property of an object to a hardcoded or computed value, you don’t need Automate. You can simply create an Action and call it from a UI (Workshop, Object Explorer, …) on the object set you want.

Level 1 - What triggers an Automation

For highly frequently updating objects: use a time based evaluation AND an object update evaluation. The number of evaluation of the Automation will get capped by the number of evaluations of the time trigger.

e.g. You have an Object instance updating 100 times per day. You have 1000 objects in your Object Type. On “on object update” this translate to 100 to 100k automations triggered per day (100x1000 at worst, if no object update is happening at the same time)
Using a time trigger of 5 minutes, the number of execution will be capped to 1440/5 = 288 per day, a potential ~x340 reduction compared to the worst case.

5 Automate executions without time trigger

3 Automate executions with time trigger

Note: Other configuration (Queue, Automate dependencies …) can also affect this level.

Level 2 - How a trigger firing will generate effect

This is directly configurable and illustrated in the UI. A “Single execution” mode is the default, and is usually the best to pick if you have no requirements that contradict its use.
e.g. you need the objects to be processed by the effects in specific batches because your action/function/etc. has this convention as an input. For example if an action sends a notification per a number of “alerts” raised on one object instance, you then might want to group the alerts by “source_object_instance” and send emails this way. In practice, there are often other ways to define this and this is not necessary.

Level 3 - What the effect will do

The effect (an Action, potentially backed by a function) needs to handle objects in bulk. So the Action needs to take numerous objects instances as input, in one go.

The best option is to use “Object set” as input parameter of your action. This allows to pass very large set of objects, in a very efficient way.
See https://www.palantir.com/docs/foundry/action-types/use-actions

e.g. you need to pass 10M objects as an input to your action, you pass it via an Object Set.

Object set configured as input in Action

Object set can be passed to the Action in Automate

A least good but still possible option is to pass an “array of objects instances”. You need to select “Object reference” and tick the “allow multiple values” so that the input parameter is effectively an array of objects.
With this type of input, the action can perform changes by itself, in bulk, on all objects passed via parameter without the need for a function.

The least preferable option is to pass single object references. Again, in some case this might be needed or relevant (e.g. you only have one object instance in your object type) but generally speaking, working with any of the above (object set in particular) is preferable.

Implicit - Effects triggering new Automations

Automate triggering an effect might edit objects, which might serves as a condition for other Automate to start.
AIP Autopilot should be helpful to understand how Automate chains together. Workflow Lineage (available today) is also another way.

1 Like

Function aspects

The Action can be backed by a function. The same reasoning applies: the function should take batch input (ObjectSets, Object arrays …) whenever possible.

Python syntax

from functions.api import function, Date, Integer, String, Float
from ontology_sdk.ontology.objects import ExampleDataAircraft
from ontology_sdk.ontology.object_sets import ExampleDataAircraftObjectSet

# 🔴 Object: An instance of an ontology object type.
@function()
def get_aircraft_name(aircraft: ExampleDataAircraft) -> str:
    return aircraft.display_name
    
# 🟠 Object Array: An array of objects.
@function()
def get_aircrafts_names(aircraft_array: list[ExampleDataAircraft]) -> list[str]:
    return [aircraft.display_name for aircraft in aircraft_array]
   
# 🟢 Object Set: A queryable set of objects e.g. ObjectSet<Employee>.
@function()
def count_aircrafts(aircraft_set: ExampleDataAircraftObjectSet) -> Float:
    return aircraft_set.count().compute()

Typescript v2 syntax

/// in "getAircraftName.ts"
// 🔴 Object: An instance of an ontology object type.
import { Client, Osdk } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk"; // Note that your OSDK may have a different name.

export default function getAircraftName(aircraft: Osdk.Instance<ExampleDataAircraft>): string {
    return aircraft.displayName!;
}

/// in "getAircraftsNames.ts"
// 🟠 Object Array: An array of objects.
import { Client, Osdk } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk"; // Note that your OSDK may have a different name.

// Default export is required for TSv2 functions
export default function getAircraftsNames(aircraftArray: Osdk.Instance<ExampleDataAircraft>[]): string[] {
    return aircraftArray.map(e => e.displayName!);
}


/// in "countAircrafts.ts"
// 🟢 Object Set: A queryable set of objects e.g. ObjectSet<Employee>.
import { ObjectSet} from "@osdk/client";
import { Integer } from "@osdk/functions";
import { ExampleDataAircraft } from "@ontology/sdk"; 

export default async function countAircrafts(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
    const nbAircrafts = await aircraftSet.aggregate({
        $select: {
            $count: "unordered" 
        } 
    });
    return nbAircrafts.$count;
}

Typescript v1 syntax

import { Function, Integer} from "@foundry/functions-api";
import { ObjectSet, ExampleDataAircraft } from "@foundry/ontology-api";

export class MyFunctions {
    // 🔴 Object: An instance of an ontology object type.
    @Function()
    public getAircraftName(aircraft: ExampleDataAircraft): string {
        return aircraft.displayName!;
    }
    
    // 🟠 Object Array: An array of objects.
    @Function()
    public getAircraftsNames(aircraftArray: ExampleDataAircraft[]): string[] {
        return aircraftArray.map(e => e.displayName!);
    }

    // 🟢 Object Set: A queryable set of objects e.g. ObjectSet<Employee>.
    @Function()
    public async countAircrafts(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
        const nbAircrafts = await aircraftSet.count()
        return nbAircrafts!;
    }
}

Example: Running those 3 implementations above shows different query patterns to the ontology

The behavior of the function itself is important. A call towards the ontology will take some time, and so has a performance/cost impact. Same rule of thumb as above: Always do things as bulk as possible.
You should prefer loading a whole object set in memory in one call instead of loading objects one by one.

The cost of a function has multiple components:

  • the overhead : calling a function = 4 compute seconds, regardless of what it is doing
  • the compute time: how much time the function needs to execute (the vCPU-“used time”)
  • the calls its performing to other parts of the platform, which might incur a cost by itself (ontology queries, models inference, LLM calls,…)

See https://www.palantir.com/docs/foundry/ontologies/query-compute-usage

Ontology Queries in Functions

Understand when the Ontology is called is important. Each query (fetching objects, aggregations, …) actually executes a call to the Ontology to perform this operation.

One thing to consider: Objects passed as parameter will trigger queries. Even if a single object is passed as an input, this will trigger a call to the Ontology to load this object in memory, so that you can - for example - edit it in the function.
Hence it is preferable to use Objects Sets to pass references, to avoid this upfront “loading cost” (e.g. if you don’t actually need the values of the object, then no loading will happen).

Outside of this one case, the calls to the ontology are explicit with a syntax like Objects.search() .myObjectType().... for example (exact syntax depends on language).

Below are examples of how to load objects in memory and how to edit them, when a single object, an array of object or an object set is passed as parameter.

Python syntax

from functions.api import function, Date, Integer, String, Float, OntologyEdit
from ontology_sdk.ontology.objects import ExampleDataAircraft
from ontology_sdk.ontology.object_sets import ExampleDataAircraftObjectSet
from ontology_sdk import FoundryClient

# 🔴 Object: An instance of an ontology object type. Single call to the Ontology upfront.
@function(edits=[ExampleDataAircraft])
def edit_aircraft_name(aircraft: ExampleDataAircraft) -> list[OntologyEdit]:
    ontology_edits = FoundryClient().ontology.edits()

    editable_aircraft = ontology_edits.objects.ExampleDataAircraft.edit(aircraft)
    editable_aircraft.display_name = "new display name"

    return ontology_edits.get_edits()


# 🟠 Object Array: An array of objects. Single call to the Ontology upfront.
@function(edits=[ExampleDataAircraft])
def edit_aircrafts_names(aircraft_array: list[ExampleDataAircraft]) -> list[OntologyEdit]:
    ontology_edits = FoundryClient().ontology.edits()

    for aircraft in aircraft_array:
        editable_aircraft = ontology_edits.objects.ExampleDataAircraft.edit(aircraft)
        editable_aircraft.display_name = "new display name"

    return ontology_edits.get_edits()


# 🟢 Object Set: A queryable set of objects e.g. ObjectSet<Employee>.
# No call to the Ontology upfront. Implicit calls handled by the "iterate" operator
@function(edits=[ExampleDataAircraft])
def edit_all_aircrafts_paging(aircraft_set: ExampleDataAircraftObjectSet) -> list[OntologyEdit]:
    ontology_edits = FoundryClient().ontology.edits()

    aircraft_array = aircraft_set.iterate()
    for aircraft in aircraft_array:
        editable_aircraft = ontology_edits.objects.ExampleDataAircraft.edit(aircraft)
        editable_aircraft.display_name = "new display name"

    return ontology_edits.get_edits()

# 🟢 Object Set: A queryable set of objects e.g. ObjectSet<Employee>. 
# No call to the Ontology upfront. Only explicit (e.g. one per page loaded)
@function(edits=[ExampleDataAircraft])
def edit_all_aircrafts_iterate(aircraft_set: ExampleDataAircraftObjectSet) -> list[OntologyEdit]:
    ontology_edits = FoundryClient().ontology.edits()

    next_page_token: str = None;

    while True:
        page_iterator = aircraft_set.page(1000, next_page_token)

        # Parsing out the fetched page
        aircraft_array = page_iterator.data
        next_page_token = page_iterator.next_page_token

        for aircraft in aircraft_array:
            editable_aircraft = ontology_edits.objects.ExampleDataAircraft.edit(aircraft)
            editable_aircraft.display_name = "new display name"

        if next_page_token is None:
            break

    return ontology_edits.get_edits()

Typescript v2 syntax

/// in "editAircraftName.ts"
// 🔴 Object: An instance of an ontology object type. Single call to the Ontology upfront.

import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Edits, createEditBatch } from "@osdk/functions";

type OntologyEdit = Edits.Object<ExampleDataAircraft>;

async function editAircraftName(client: Client, aircraft: Osdk.Instance<ExampleDataAircraft>): Promise<OntologyEdit[]> {
    const batch = createEditBatch<OntologyEdit>(client);
    batch.update(aircraft, { displayName: "new display name" });
    return batch.getEdits();
}

export default editAircraftName;

/// in "editAircraftsNames.ts"
// 🟠 Object Array: An array of objects. Single call to the Ontology upfront.
import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Edits, createEditBatch } from "@osdk/functions";

type OntologyEdit = Edits.Object<ExampleDataAircraft>;

async function editAircraftsNames(client: Client, aircraftArray: Osdk.Instance<ExampleDataAircraft>[]): Promise<OntologyEdit[]> {
    const batch = createEditBatch<OntologyEdit>(client);
    aircraftArray.map(aircraft => batch.update(aircraft, { displayName: "new display name" }));
    return batch.getEdits();
}

export default editAircraftsNames;

/// in "editAllAircrafts_iterate.ts"
// 🟢 No call to the Ontology upfront. Implicit calls handled by the "iterate" operator

import { Client, ObjectSet } from "@osdk/client";
import { Integer, Edits, createEditBatch } from "@osdk/functions";
import { ExampleDataAircraft } from "@ontology/sdk"; 

type OntologyEdit = Edits.Object<ExampleDataAircraft>;

export default async function editAllAircrafts_iterate(client: Client, aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<OntologyEdit[]> {
    const batch = createEditBatch<OntologyEdit>(client);

    for await(const aircraft of aircraftSet.asyncIter()) {
        batch.update(aircraft, { displayName: "new display name" });
    }

    return batch.getEdits();
}

/// in "editAllAircrafts_page.ts"
// 🟢 No call to the Ontology upfront. Only explicit (e.g. one per page loaded)

import { Client, ObjectSet } from "@osdk/client";
import { Integer, Edits, createEditBatch } from "@osdk/functions";
import { ExampleDataAircraft } from "@ontology/sdk"; 

type OntologyEdit = Edits.Object<ExampleDataAircraft>;

export default async function editAllAircrafts(client: Client, aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<OntologyEdit[]> {
    const batch = createEditBatch<OntologyEdit>(client);

    var newToken: string | undefined = undefined;
    var nextPageToken: string | undefined = undefined;

    do {
        const { data, nextPageToken: newToken } = await aircraftSet.fetchPage({
            $pageSize: 1000, // Adjust page size as needed
            $nextPageToken: nextPageToken,
        });

        data.forEach(aircraft => {
            batch.update(aircraft, { displayName: "new display name" });
        });

        nextPageToken = newToken;
    } while (nextPageToken);

    return batch.getEdits();
}

Typescript v1 syntax

import { Function, OntologyEditFunction, Edits, Integer} from "@foundry/functions-api";
import { ObjectSet, ExampleDataAircraft } from "@foundry/ontology-api";

export class MyFunctions {
    // Object: An instance of an ontology object type. Single call to the Ontology upfront.
    @Edits(ExampleDataAircraft)
    @OntologyEditFunction()
    public editAircraftName(aircraft: ExampleDataAircraft): void {
        aircraft.displayName = "new display name";
    }
    
    // Object Array: An array of objects. Single call to the Ontology upfront.
    @Edits(ExampleDataAircraft)
    @OntologyEditFunction()
    public editAircraftsNames(aircraftArray: ExampleDataAircraft[]): void {
        aircraftArray.map(currentAircraft => currentAircraft.displayName = "new display name");
    }

    // Object Set: A queryable set of objects, e.g. ObjectSet<Employee>.
    // No call to the Ontology upfront. Only explicit (e.g. when calling `.all()`)
    @Edits(ExampleDataAircraft)
    @OntologyEditFunction()
    public doNothingWithObjectSetAircrafts(aircraftSet: ObjectSet<ExampleDataAircraft>): void {
        // Loading with ".all()" = Ontology call
        // aircraftSet.all().map(currentAircraft => currentAircraft.displayName = "new display name");
        // No loading with ".all()" = no Ontology call
    }
}

Any implementation that doesn’t load objects in bulk is likely an anti-pattern.
For example: :red_circle: A “for loop” loading one object per iteration vs :green_circle: loading all objects upfront

https://www.palantir.com/docs/foundry/functions/api-object-sets/#object-search

Python syntax

from functions.api import function
from ontology_sdk import FoundryClient
from ontology_sdk.ontology.objects import ExampleDataAircraft

# 🔴 Objects are loaded one by one, and the aggregation is performed in the function
@function()
def for_loop_worst(pks_list: list[str]) -> int:
    client = FoundryClient()
    seats_count = 0

    for current_pk in pks_list:
        aircrafts = client.ontology.objects.ExampleDataAircraft.where(
            ExampleDataAircraft.object_type.tail_number == current_pk
        ).page()
        # Get the one aircraft
        aircraft : ExampleDataAircraft = aircrafts.data[0]
        seats_count += aircraft.number_of_seats or 0

    return seats_count

# 🟠 The Ontology backend returns all objects in one go, and the aggregation is performed in the function
@function()
def for_loop_better_iterate(pks_list: list[str]) -> int:
    client = FoundryClient()
    seats_count = 0

    # Get all objects
    aircrafts = client.ontology.objects.ExampleDataAircraft.where(
        ExampleDataAircraft.object_type.tail_number.in_(pks_list)
    )

    # Iterate on each object
    for aircraft in aircrafts:
        seats_count += aircraft.number_of_seats or 0
    return seats_count


@function
def for_loop_better_page(pks_list: list[str]) -> int:
    client = FoundryClient()
    seats_count = 0

    # Fetch all objects
    aircrafts_page = client.ontology.objects.ExampleDataAircraft.where(
        ExampleDataAircraft.object_type.tail_number.in_(pks_list)
    ).page(page_size=1000)

    # Iterate on each
    for aircraft in aircrafts_page.data:
        seats_count += aircraft.number_of_seats or 0

    return seats_count


# 🟢 The Ontology backend perform the aggregation and only returns the result
@function()
def for_loop_best(pks_list: list[str]) -> int:
    client = FoundryClient()

    # Perform the aggregation
    result = client.ontology.objects.ExampleDataAircraft.where(
        ExampleDataAircraft.object_type.tail_number.in_(pks_list)
    ).sum(ExampleDataAircraft.object_type.number_of_seats).compute()

    # result.data[0].metrics[0].value contains the sum
    return int(result or 0)

Typescript v2 syntax

/// in "forLoopWorst.ts"

import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Integer } from "@osdk/functions";

export default async function forLoopWorst(client: Client, pks_list: string[]): Promise<Integer> {
    // 🔴 Objects are loaded one by one, and the aggregation is performed in the function
    var seatsCount = 0;

    // Iterate over each primary key
    for (var currentPk of pks_list) { 
        // Fetch the object
        const fetchedPage = await client(ExampleDataAircraft).where({
            tailNumber: { $eq: currentPk }
        }).fetchPage();
        const relevantObject = fetchedPage.data[0];

        // Do something with the object - e.g. access a property
        seatsCount += relevantObject!.numberOfSeats ?? 0;
    }

    return seatsCount
}

/// in "forLoopBetter_iterate.ts"

import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Integer } from "@osdk/functions";

export default async function forLoopBetter_iterate(client: Client, pks_list: string[]): Promise<Integer> {
    // 🟠 The Ontology backend returns all objects in one go, and the aggregation is performed in the function
    var seatsCount = 0;

    // Fetch all the objects
    const allObjects = client(ExampleDataAircraft).where({
        tailNumber: { $in: pks_list }
    });

    // Iterate over each object
    for await(const currentObject of allObjects.asyncIter()) { 
        // Do something with the object - e.g. access a property
        seatsCount += currentObject!.numberOfSeats ?? 0;
    }

    return seatsCount
}

/// in "forLoopBetter_page.ts"

import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Integer } from "@osdk/functions";

export default async function forLoopBetter_page(client: Client, pks_list: string[]): Promise<Integer> {
    // 🟠 The Ontology backend returns all objects in one go, and the aggregation is performed in the function
    var seatsCount = 0;

    // Fetch all the objects
    const allObjects = await client(ExampleDataAircraft).where({
        tailNumber: { $in: pks_list }
    }).fetchPage({
        $pageSize: 1000
    });

    // Iterate over each object
    for (const currentObject of allObjects.data) { 
        // Do something with the object - e.g. access a property
        seatsCount += currentObject!.numberOfSeats ?? 0;
    }

    return seatsCount
}

/// in "forLoopBest.ts"

import { Osdk, Client } from "@osdk/client";
import { ExampleDataAircraft } from "@ontology/sdk";
import { Integer } from "@osdk/functions";

export default async function forLoopBest(client: Client, pks_list: string[]): Promise<Integer> {
    // 🟢 The Ontology backend perform the aggregation and only returns the result

    // Perform the aggregation
    const seatsCount = await client(ExampleDataAircraft).where({
        tailNumber: { $in: pks_list }
    }).aggregate({
        $select: { "numberOfSeats:sum" : "unordered" }
    });

    return seatsCount.numberOfSeats.sum!
}

Typescript v1 syntax

import { Objects } from "@foundry/ontology-api";

export class MyFunctions {
    @Function()
    public forLoopInefficient(pks_list: string[]): Integer {
        // 🔴 Objects are loaded one by one, and the aggregation is performed in the function
        var seatsCount = 0;

        // Iterate over each primary key
        for (var currentPk of pks_list) { 
            // Fetch the object
            // Note, if you have the primary key, you can use: Objects.search().exampleDataAircraft([currentPk]).all()[0];
            const relevant_object: ExampleDataAircraft = Objects.search().exampleDataAircraft().filter(o => o.tailNumber.exactMatch(currentPk)).all()[0];
            // Do something with the object - e.g. access a property
            seatsCount += relevant_object.numberOfSeats ?? 0;
        }

        return seatsCount
    }

    @Function()
    public forLoopBetter(pks_list: string[]): Integer {
        // 🟠 The Ontology backend returns all objects in one go, and the aggregation is performed in the function
        // Bulk loading of objects
        // Note, if you have the primary key, you can use: var allObjects: ExampleDataAircraft[] = Objects.search().exampleDataAircraft(pks_list).all();
        var allObjects: ExampleDataAircraft[] = Objects.search().exampleDataAircraft().filter(o => o.tailNumber.exactMatch(...pks_list)).all();

        var seatsCount = 0;

        // Iterate over each object
        for (var currentObject of allObjects) { 
            // Do something with the object - e.g. access a property
            seatsCount += currentObject.numberOfSeats ?? 0;
        }

        return seatsCount
    }

    @Function()
    public async forLoopBest(pks_list: string[]): Promise<Integer> {
        // 🟢 The Ontology backend perform the aggregation and only returns the result
        // Note, if you have the primary key, you can use: Objects.search().exampleDataAircraft(pks_list).sum(o => o.numberOfSeats)
        const seatsCount = await Objects.search().exampleDataAircraft().filter(o => o.tailNumber.exactMatch(...pks_list)).sum(o => o.numberOfSeats)

        return seatsCount!
    }
}

A note on async operations: even if the usage of asynchronous operations can speed up the execution of a function, it might not reduce its resource usage (and so its cost). For example:

  • A “for loop” of object loading is bad for performance, and non-optimal for resource consumption
  • An asynchronous Promise.all() on all iterations of the for loop will be better for speed, but still non-optimal for resource consumption.
  • A bulk loading upfront of the for loop will be good for performance and for resource consumption.

Python syntax

from functions.api import function, Date, Integer, String, Float
from ontology_sdk import FoundryClient
from ontology_sdk.ontology.objects import ExampleDataAircraft
from ontology_sdk.ontology.object_sets import ExampleDataAircraftObjectSet

# No other working example for now

# 🟢 The Ontology backend moves from one Object set to another, perform the aggregation and only returns the result
@function
def bulk_processing(aircraft_set: ExampleDataAircraftObjectSet) -> Float:
    all_maintenance_events = aircraft_set.search_around_example_data_aircraft_maintenance_event()
    maintenance_events = all_maintenance_events.count().compute()
    return maintenance_events

Typescript v2 syntax

/// in "forLoopInefficientNotAsync.ts"
// 🔴 For loop, where a search around is performed on each object, and then contributed to the aggregation - done in the function

import { Osdk, Client, ObjectSet } from "@osdk/client";
import { ExampleDataAircraft, ExampleDataAircraftMaintenanceEvent} from "@ontology/sdk";
import { Integer } from "@osdk/functions";

// Utility function: Fetch the number of maintenance events for a given Aircraft.
export async function getNumberOfMaintenanceEvents(client: Client, aircraft: Osdk.Instance<ExampleDataAircraft>): Promise<Integer> {
    const maintenanceEvents = aircraft.$link.exampleDataAircraftMaintenanceEvent
    const nbMaintenanceEvents = await maintenanceEvents.aggregate({
        $select: {
            $count: "unordered" 
        } 
    });
    return nbMaintenanceEvents.$count ?? 0;
}

export default async function forLoopInefficientNotAsync(client: Client, aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
    // Get all objects - Could also be done using fetchPage
    const allObjects : Osdk.Instance<ExampleDataAircraft>[] = [];
    for await(const obj of aircraftSet.asyncIter()) {
        allObjects.push(obj);
    }

    let maintenanceEvents = 0;

    // Iterate over each object
    for (const currentObject of allObjects) { 
        maintenanceEvents += await getNumberOfMaintenanceEvents(client, currentObject);
    }

    return maintenanceEvents;
}


/// in "forLoopAsync.ts"
// 🟠 Each object's search around and calculation is launched in parallel. Final result is aggregated after everything executes.

import { Osdk, Client, ObjectSet } from "@osdk/client";
import { ExampleDataAircraft, ExampleDataAircraftMaintenanceEvent} from "@ontology/sdk";
import { Integer } from "@osdk/functions";


// Utility function: Fetch the number of maintenance events for a given Aircraft.
export async function getNumberOfMaintenanceEvents(client: Client, aircraft: Osdk.Instance<ExampleDataAircraft>): Promise<Integer> {
    const aircraftAsObjectSet = client(ExampleDataAircraft).where({
        tailNumber: { $eq: aircraft.tailNumber }
        });

    const maintenanceEvents = aircraftAsObjectSet.pivotTo("exampleDataAircraftMaintenanceEvent")
    const nbMaintenanceEvents = await maintenanceEvents.aggregate({
        $select: {
            $count: "unordered" 
        } 
    });
    return nbMaintenanceEvents.$count ?? 0;
}

export default async function forLoopAsync(client: Client, aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
    // Get all objects - Could also be done using asyncIter
    // Get all objects - Could also be done using fetchPage
    const allObjects : Osdk.Instance<ExampleDataAircraft>[] = [];
    for await(const obj of aircraftSet.asyncIter()) {
        allObjects.push(obj);
    }    
    const futures: Promise<number>[] = [];
    let maintenanceEvents = 0;

    // Iterate over each object and trigger the computation
    for (const currentObject of allObjects) { 
        futures.push(getNumberOfMaintenanceEvents(client, currentObject));
    }

    // Wait for the computation to finish
    const results = await Promise.all(futures);

    // Iterate over the results to consolidate into final result
    for (const result of results) { 
        maintenanceEvents += result;
    }

    return maintenanceEvents;
}

/// in "bulkProcessing.ts"
// 🟢 The Ontology backend moves from one Object set to another, perform the aggregation and only returns the result

import { Osdk, Client, ObjectSet } from "@osdk/client";
import { ExampleDataAircraft, ExampleDataAircraftMaintenanceEvent} from "@ontology/sdk";
import { Integer } from "@osdk/functions";

export default async function bulkProcessing(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
    const allMaintenanceEvents = aircraftSet.pivotTo("exampleDataAircraftMaintenanceEvent");

    const nbMaintenanceEvents = await allMaintenanceEvents.aggregate({
        $select: {
            $count: "unordered" 
        } 
    });
    return nbMaintenanceEvents.$count ?? 0;
}

Typescript v1 syntax

import { Function, OntologyEditFunction, Edits, Integer} from "@foundry/functions-api";
import { ObjectSet, ExampleDataAircraft, ExampleDataAircraftMaintenanceEvent } from "@foundry/ontology-api";
import { Objects } from "@foundry/ontology-api";

export class MyFunctions {

    // Utility function. Fetch the number of maintenance event for a given Aircraft.
    public async getNumberOfMaintenanceEvents(aircraft: ExampleDataAircraft): Promise<Integer> {
        // See https://www.palantir.com/docs/foundry/functions/api-objects-links#link-types 
        // on the syntax when moving from one object to a set of linked objects
        const aircraftObjectSet = Objects.search().exampleDataAircraft([aircraft])
        const maintenanceEvents = aircraftObjectSet.searchAroundExampleDataAircraftMaintenanceEvent();

        return await maintenanceEvents.count() ?? 0;
    }
    
    // 🔴 For loop, where a search around is performed on each object, and then contributed to the aggregation - done in the function
    @Function()
    public async forLoopInefficientNotAsync(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
        const allObjects = aircraftSet.all();
        var maintenanceEvents = 0;

        // Iterate over each object
        for (var currentObject of allObjects) { 
            maintenanceEvents += await this.getNumberOfMaintenanceEvents(currentObject);
        }

        return maintenanceEvents
    }

    // 🟠 Each object's search around and calculation is launched in parallel. Final result is aggregated after everything executes.
    @Function()
    public async forLoopAsync(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
        const allObjects = aircraftSet.all();
        var futures = []
        var maintenanceEvents = 0;

        // Iterate over each object and trigger the computation
        for (var currentObject of allObjects) { 
            futures.push(this.getNumberOfMaintenanceEvents(currentObject))
        }

         // Wait for the computation to finish
        const results = await Promise.all(futures);

        // Iterate over the results to consolidate into final result
        for (var result of results) { 
            maintenanceEvents += result;
        }

        return maintenanceEvents
    }

    // 🟢 The Ontology backend moves from one Object set to another, perform the aggregation and only returns the result
    @Function()
    public async bulkProcessing(aircraftSet: ObjectSet<ExampleDataAircraft>): Promise<Integer> {
        const allMaintenanceEvents = aircraftSet.searchAroundExampleDataAircraftMaintenanceEvent();
        var maintenanceEvents = await allMaintenanceEvents.count();
        return maintenanceEvents ?? 0;
    }
}

8 Likes