Hi everyone,
We are currently building a user-facing application backed by AIP Logic. Because this is an interactive API, latency is critical, and we have strict SLOs regarding response times.
We are observing significant, non-deterministic latency spikes during the execution of AIP Logic functions. Through tracing, we’ve narrowed the bottleneck down to the “Security Context” evaluation phase.
Initially, we thought this was due to LLM-driven SQL generation. However, to isolate the root cause, we created a minimal reproduction setup:
-
A single AIP Logic block.
-
No SQL lookups or Ontology edits.
-
No tools attached.
-
Minimal/Empty context.
Even in this bare-bones setup, the time taken for the “Security Context” evaluation to clear before the LLM starts generating varies significantly between runs - between 300ms to almost 3000ms. Sometimes it is instant; other times it adds a substantial delay that violates our latency requirements.
Apparently Security Context is enforced on every call (including LLM nodes) and cannot be bypassed or selectively disabled, regardless of whether we use OAuth or other authentication methods.
Has anyone else dealt with this specific overhead for real-time/user-facing use cases? Since we cannot disable it, I am looking for architectural workarounds or optimization strategies.
Specifically:
-
Architectural alternatives: For low-latency requirements, have you had better luck moving the LLM orchestration out of AIP Logic (e.g., using pure Functions on Foundry with manual calls to the language model) to have finer control over the execution environment?
-
Determinism: Are there any settings regarding isolation levels or resource profiles that make this evaluation time more consistent?
Any insights regarding AIP Logic latency would be greatly appreciated.
Thanks!