Speeding up mediaset to object

[jackmiller2003]

Speeding up pipeline builder.

Hey everyone! I’ve got a pipeline which runs to turn a mediaset into an object everytime the mediaset changes… this allows me to upload from my OSDK frontend into a mediaset and then provide some metadata around that upload.

However, the latency on my pipeline build is like a minute! This is too long for the user to get feedback from the upload. Is there anyway to speed this up? I’ve uploaded an image of my pipeline.

[bkaplan]

I think the best way to improve latency would be to have the upload run an action to write directly to the ontology.

[jackmiller2003]

Awesome thanks! I did this in the end

[jackmiller2003]

Hi all,

Just for reference, here’s how I did the mediaset stuff in the end!

Basically I maintain two things on Palantir’s system:

1. Mediaset in a given project.
2. A dataset in the same project pointing to an object which essentially “tracks” the mediaset

When uploading a document I use the following three step process:

1. Upload the file and get a mediaItemRid
2. Get a media reference to the mediaItemRid
3. Create an entry in the object tracking the mediaset

Some important things to watchout for:

1. Don’t use a regular OSDK app for the token to use the mediaset APIs, rather create a non-ODSK app, giving it the appropriate permissions, namely: “api:mediasets-write” and “api:mediasets-read”.
2. The media reference is for later use. It’s important to set the string formatting of this correctly, and use the manipulation code I have in step 3 as what is returned from the endpoint does not line up with the way media references are interpreted on other parts of the platform!

How would you do this with restricted views? (I haven’t yet done this, but will soon! Also Boaz Francis is the real genius here for figuring it out)

1. Start by putting the mediaset in a different project where users only have view access and can only upload (they can’t read from the mediaset).
2. Then create an entry in the object tracking the mediaset (which is in a different project) with the media reference and mediaset rid.
3. Update the object with the media reference by using an automate on platform which calls a typescript function.

Here’s the code:

  // Step 1: Upload file and get mediaItemRid
  const uploadFile = async (file: File): Promise<string> => {
      setUploadProgress("Uploading file...");
      const mediaItemRid = await uploadDocument(mediaSetRid, file, file.name);
      return mediaItemRid;
  };

  // Step 2: Get media reference
  const getMediaReference = async (mediaItemRid: string): Promise<string> => {
      setUploadProgress("Getting media reference...");
      const response = await fetch(
          `${config.palantirFoundryUrl}/api/v2/mediasets/${mediaSetRid}/items/${mediaItemRid}/reference?preview=true`,
          {
              headers: {
                  'Authorization': `Bearer ${auxiliaryAuth.getTokenOrUndefined()}`
              }
          }
      );

      if (!response.ok) {
          throw new Error('Failed to get media reference');
      }

      const data = await response.json();

      const reference = {
          type: "mediaSetViewItem",
          mediaSetViewItem: {
              mediaSetRid: data.reference.mediaSetRid,
              mediaSetViewRid: data.reference.mediaSetViewRid,
              mediaItemRid: data.reference.mediaItemRid
          }
      };

      data.reference = reference;

      console.log(data);

      return data;
  };

  // Step 3: Initialize metadata
  const initializeMetadata = async (mediaItemRid: string, mediaReference: string) => {
      setUploadProgress("Initializing document metadata...");
      const result = await client(createDocumentMediaset).applyAction(
          {
              media_item_rid: mediaItemRid,
              client_id: selectedClientId,
              document_name: documentName,
              mediaset_rid: mediaSetRid,
              path: selectedFile?.name || "",
              tags: selectedTags,
              timestamp: new Date().toISOString(),
              mediaset_reference: JSON.stringify(mediaReference)
          },
          {
              $returnEdits: true,
          }
      );

      if (result.type !== "edits") {
          throw new Error("Failed to initialize metadata");
      }

      return result;
  };

With the upload utility:

export async function uploadDocument(
    mediaSetRid: string,
    file: File,
    filename: string
): Promise<string> {
    // Encode the filename for the URL
    const encodedFilename = encodeURIComponent(filename);

    // Get the file contents as a blob/binary data
    const fileContents = await file.arrayBuffer();

    // Construct the upload URL using the Palantir URL from config
    const uploadUrl = `${config.palantirFoundryUrl}/api/v2/mediasets/${mediaSetRid}/items?mediaItemPath=${encodedFilename}&preview=true`;

    try {
        const response = await fetch(uploadUrl, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/octet-stream',
                'Authorization': `Bearer ${auxiliaryAuth.getTokenOrUndefined()}`
            },
            body: fileContents
        });

        if (!response.ok) {
            throw new Error(`Upload failed: ${response.statusText}`);
        }

        const result = await response.json();
        return result.mediaItemRid;
    } catch (error) {
        console.error('Error uploading document:', error);
        throw error;
    }
}