Making incremental pipeline from mediaset

Hi,

I’m making an app which allows the user to upload an image of their meal using the media uploader widget. I send these images to a media set in pipeline builder and then process the data using transformations and a LLM to extract the food items from the image. I want to make it so that every time a new image is uploaded to the media set, it gets sent through the pipeline and a new object is created for that image.

The way that I tried to solve this was first by converting the mediaset into a dataset in a separate pipeline, then running the transformations / LLM on that outputted dataset in another pipeline. However, I’m getting “Failed to load preview” when I do this. Any solutions?

Hey @jaysen_quan is it only preview that’s failing or is it the build too? Are you able to share screenshots of what you’re seeing? And to confirm what exactly are you trying to preview (the llm output? If so can I see the configuration of the LLM node)

Separately, I’d recommend putting your pipeline on a schedule to automatically build your datasets whenever a new media is uploaded

Hey @helenq, both the preview and the build are failing. I’ve attached screenshots of the workflow and the error messages that I’m getting.




.

On the first pipeline, I was able to set up a schedule so that every time an image is added to the media set, the “meals data” dataset is rebuilt. But, since I’m unable to build the other pipeline, this cuts off my continuous building process.

Got it, can you check/try these two things:

  1. Make sure you have permissions to see the backing media sets
  2. Add the original media set to the pipeline eg. instead of using the dataset that references the media set, try going from the media set (or just try importing it into the pipeline builder file)

(cc: @Isy from the unstructed/media team)

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.