Use LLM build job cannot read MediaSet when mediaReference comes from an upstream code repo transform

Ask the Community
,
1

Hi all,

I am using the Use LLM node in Pipeline Builder with a vision prompt. My setup is as follows:

  • A code repository transform joins a MediaSet with product data and outputs an enriched dataset. The mediaReference column is present in this output dataset.
  • In Pipeline Builder, I feed that enriched dataset directly into the Use LLM node. The MediaSet itself is not a node in the Pipeline Builder graph — only the enriched dataset is.

Trial runs work without issue. Full builds fail with:

PERMISSION_DENIED (MediaSet:CannotReadMediaSetBranch)
mediaSetRid = ri.mio.main.media-set.<redacted>

The build job does not have read access to the MediaSet because it is not declared as a connected input node in the pipeline graph. The build token is scoped only to resources explicitly present in the graph, so even though the mediaReference values are sitting in the enriched dataset, the build job cannot resolve them against the MediaSet at runtime.

I know that adding the MediaSet as a connected input node in the pipeline graph (with a join back to the enriched dataset) resolves the permission issue, and I have confirmed this works. However, this is a problematic workaround for two reasons:

  1. The mediaReference values are already in the enriched dataset. Re-introducing the MediaSet into the pipeline graph just to satisfy the build job’s permission scoping is unnecessary overhead.
  2. More importantly, this creates a hidden dependency problem for consumers of the enriched dataset. Any downstream user who builds a pipeline on top of that dataset and uses the Use LLM node will hit the same PERMISSION_DENIED failure. They have no visibility into the upstream MediaSet, no reason to know the workaround exists, and will burn build credits on repeated failures before understanding why. This has already happened in practice.

This is not just a usability issue — it has a real cost impact for teams consuming enriched datasets that contain mediaReference columns.

My questions:

  1. Is there a supported way to grant a Pipeline Builder build job read access to a MediaSet without adding it as a connected input node in the graph?
  2. If not, is the lightweight join pattern (adding the MediaSet as a connected node with a minimal join that adds no new columns) the only supported approach right now?
  3. Is there a platform-level mechanism planned to allow MediaSet read access to be declared or inherited without requiring an explicit graph node, so that downstream consumers are not silently blocked and do not incur unnecessary build costs?

Thank you.

2

Hi @akash.menon,

Great questions, and I completely understand the implications. Currently, yes, the only solution is to add the media set as an input and join it to the pipeline, as you described. And you are correct that the join does not need to add any columns and, in fact, it does not need to join on any rows either – you are free to filter out all rows of the media set upstream of the join, which may offer a performance benefit if you’re working with a large number of media items.

You are correctly identifying the underlying challenge, which is that without a media set explicitly declared, there is no way to perform input resolution on the media references. This is an essential step that ensures the user has permission to read those media items, which is why the error being thrown is PERMISSION_DENIED. Consider a scenario where media references from multiple media sets are contained in the same column and the user has access to only a subset of media sets, or the media reference column contains media references that point to a media set view prior to when the user was granted permission.

That being said, support for performing the input resolution implicitly is currently in development. Until then, please continue to join the media set.