I have 503 rows of text data (~250 words each) and want to embed the column with Text Embedding 3 Large. The vector embeddings are visible in the output preview but the transform cuts 483 of the rows (leaving 20) and restricts the preview window from 500 to 10. Additionally, the transform causes my pipeline to fail deployment.
How can I stop the transform from removing data and blocking deployment?
Hi Zack. Pipeline Builder team here.
I would double check that you don’t have any sampling strategies on your input datasets. Generally for previews, we only show a truncated (or filtered) version so it’s not unexpected that you only see a subset of rows in the preview.
Aside from preview filtering, I’m curious why this transform causes failed deployment. Is the problem that the “deploy” button is disabled, or that the builds of your outputs fail? For the former, I would dive into any error messages you see in the deploy errors tab. For the latter, I would take a closer look at any failed builds and any associated logs you might see.
