Difficulties with Push Based Streaming Data Sources

Overview

Hi all!

I am attempting to test out Data Connection - Push Based Ingestion to a Stream. The ultimate goal will be to have a python application that steadily pushing information to a Foundry stream from a OPC-UA server (which is why many of the Foundry resources are named things like OPC-UA Streaming in the various screenshots). As a first testing step I am attempting to get the simple Python-requests based write to work.

I am largely trying to follow this guide:

The script executing is resulting in:

403 Forbidden {"errorCode":"PERMISSION_DENIED","errorName":"Security:PermissionDenied","errorInstanceId":"xxxx","parameters":{"rid":"xxxxx","operation":"streaming:write"}}

I have turned on various permissions as much as the documentation and my exploration can tell is necessary.

Based on the details of my configuration below, a guide on what I may be doing wrong would be appreciated. Is there a way to determine what set of security codes are being violated? Am I missing a configuration?

Details:

When I attempt to write to the stream using a personal token and the following script (generated from Foundry’s streaming write console page) it succeeds:

import requests
import json


FOUNDRY_TOKEN = "TOKEN"


# We define a row that matches the schema we described earlier
sample_data = {"timestamp":1722015471624,"value":"value"}
post_uri = "CORRECT URL"


# We use requests to create a post request with an array of streaming rows, in this case we have one row to push
response = requests.post(
   post_uri,
   data=json.dumps([{"value": sample_data}]),
   headers={
       "Authorization": "Bearer " + FOUNDRY_TOKEN,
       "Content-Type": "application/json",
   }
)


print(response.status_code, response.reason, response.text)

response:

200 OK {"topic":"ri.foundry-streaming.main.topic.7e244b10-ace2-4c56-a4ca-f431550015de","offsetAndPartitionIds":[{"offset":3,"partitionId":0}]}

When I attempt to use the following OAuth based script it fails:

import requests
import json


def push_row():
  # We make a call to the OAuth2 endpoint with our id and secret to get an access token
  token_response = requests.post("CORRECT URL",
      data={
          "grant_type": "client_credentials",
          "client_id": "CLIENT ID",
          "client_secret": "SECRET",
	   "scope": "compass:edit"
     },
     headers={
         "Content-Type": "application/x-www-form-urlencoded",
     }
  )
  access_token = token_response.json()["access_token"]
  print('ACCESSS TOKENN:')
  print(access_token)
 
  # We define a row that matches the schema we described earlier
  sample_data = {"timestamp":1722007394978,"value":"value"}
  postUri = "CORRECT URL"
 
  # We use requests to create a post request with an array of streaming rows, in this case we have one row to push
  response = requests.post(
      postUri,
      data=json.dumps([{"value": sample_data}]),
      headers={
          "Authorization": "Bearer " + access_token,
          "Content-Type": "application/json",
      },
  )
  return response


# Call the function
response = push_row()
print(response.status_code, response.reason, response.text)

response

ACCESSS TOKENN:
<access-token printed out>
403 Forbidden {"errorCode":"PERMISSION_DENIED","errorName":"Security:PermissionDenied","errorInstanceId":"xxxxx","parameters":{"rid":"ri.foundry-streaming.main.view.xxxxxx","operation":"streaming:write"}}

Conjecture:

  • Both scripts are executed from the same local laptop - so it is not a ingress issue or both would not work
  • There must some level of within Foundry permissions that my personal token (as an Owner of much of the Foundry instance) has that the Applicaiton OAuth user does not have

The OAuth user, however, appears to have edit access to the appropriate resources. I even elevated the permissions as high as I could for the OAuth User:

Additionally the application is enabled:

Questions

I am not sure what could be causing this permissions issue:

Is it a problem with the scope of the application/OAuth token request?
A missing configuration?
There is some additional permissions that one needs to grant to enable streaming:write or to the "rid":"ri.foundry-streaming.main.view...?

Any guidance would be appreciated!

You’ll likely have to add the streaming:write scope to the requested scopes in the OAuth flow. The set of operations your third party application can perform is determined by the intersection of the resources/roles it has access to and which scopes you have requested in the OAuth flow.

Thanks @jett

That makes sense! Unfortunately, I cannot seem to understand how to add that to the scope of the application.

When I attempt to add things to the application scope, it navigates me to the SDK portion of the console page. On this page I can only find how to add Ontology API Objects, Actions and functions.

Searching for portions of the term streaming:write in various sections does not return any success?

How can an app be given streaming:write permissions? I had not yet ontologized any of the resources I had spun up. Is that necessary? Am I in the wrong place in the console?

(i would add a screenshot but it won’t let me)

You don’t have to do anything on the foundry side.
You either have to pass the additional scope in the client credentials request to the oauth2 endpoint or you just omit and remove scopes from the arguments at all and your issued token gets all scopes of the service user.

Unfortunately, this is not so well documented and Palantir is also hesitant to publish a full list of scopes.

1 Like

Thanks for the advice @nicornk!

I am attempting to add the scope to the Access token request:

   token_response = requests.post("https://<>/multipass/api/oauth2/token",
       data={
           "grant_type": "client_credentials",
            "client_id": "<>",
           "client_secret": "<>",
           "scope": "compass:edit streaming:write"
      },
      headers={
          "Content-Type": "application/x-www-form-urlencoded",
      }
   )
   access_token = token_response.json()["access_token"]
   print('TOKEN RESPONSE CONTENTS:')
   print(token_response.json())

The token response, however, indicates that no scope was registered for the access token. The response object has an empty string in the scope section:

{'access_token': '<>', 'scope': '', 'expires_in': 3600, 'token_type': 'bearer'}

Additionally, the attempt to write data continues to be denied.


I attempted a few different combinations of string in the scope parameter:
compass:edit - the web console, auto code generated scope string
streaming:write - what I want to do!
compass:edit streaming:write - documentation suggests that multiple scopes can be submitted as space delimited strings. A list of relevant scopes still

Any ideas of how to continue debugging this?? I’ve tested:

  • adding access to different sets of ontology resources - in case some are required to also be granted streaming:write
  • enabling scoped sessions for the relevant organization (no scoped sessions are also allowed)

Have you tried just leaving out the scope entirely, as also mentioned as an option by @nicornk above? scope is now optional for client_credentials (we’ll be updating the documentation to make that clear in a day or so).

(It’s interesting that you’re getting an empty string for scope in the response, but the documentation doesn’t indicate that scope is supposed to be in the response body at all, so maybe that partially explains it)

We‘ve had long discussion threads with support / PD on this and back than they couldn’t or wouldn’t want to fix the scope value in the response body.

Anyway, leaving out scope in the request has been working as implemented here in Foundry DevTools.

Does that work? Fwiw, are you sure streaming:write is the correct scope? If I remember correctly it should be something with stream-proxy.

Hi!

Yes, my apologies for forgetting to mention it but I tried leaving the scope empty several times - to the same PERMISSION DENIED and operation... streaming:write results.

I have to double check… did you leave the scope empty or omit it in the request? Did you use the example code I linked above?

thanks all for your inputs. Some updates:

leaving out scope

I have tried leaving scope out of the data dictionary. I have also tried submitting an empty string and a None.

All result in the same PERMISSION_DENIED and operation: streaming:write error.

dev tools

I did review the foundry_dev_tools package. I attempted to use it. I submitted mutiple inputs to the scopes parameter:

  • scopes = None
  • scopes = "offline_access compass:view compass:edit compass:discover api:write-data api:read-data streaming:write"

All succeed in getting a token, but a data write results in the same PERMISSION_DENIED and operation: streaming:write error.

getting scopes back in the oauth2 response:

One thing I did note in foundry_dev_tools package was the function _get_palantir_oauth_token uses a set of DEFAULT_SCOPES. I explored submitting these to the console generated code.

When doing so I started to get more than a empty string in the Oauth2 API Response! Only, however, for a couple of the scopes: 'scope': 'api:read-data api:write-data'

code

import requests
import json
import os

DEFAULT_TPA_SCOPES = [
    "offline_access",
    "compass:view",
    "compass:edit",
    "compass:discover",
    "api:write-data",
    "api:read-data",
    "streaming:write",
    "streaming:write-data",
    "stream-proxy:write-data",
    "stream-proxy:write",
]
SCOPES_STRING = ' '.join(DEFAULT_TPA_SCOPES)

print(f'SCOPES STRING ATTEMPTED: {SCOPES_STRING}')

def push_row():
   # We make a call to the OAuth2 endpoint with our id and secret to get an access token
   token_response = requests.post(f"{os.environ['FOUNDRY_URL']}/multipass/api/oauth2/token",
       data={
            "grant_type": "client_credentials",
            "client_id": os.environ['APP_CLIENT_ID'],
            "client_secret": os.environ['APP_CLIENT_SECRET'],
            "scope": SCOPES_STRING
      },
      headers={
          "Content-Type": "application/x-www-form-urlencoded",
      }
   )
   access_token = token_response.json()["access_token"]
   print('TOKEN RESPONSE CONTENTS:')
   print(token_response.json())
   
   # We define a row that matches the schema we described earlier
   sample_data = {"timestamp":1722007394978,"value":"value"}
   postUri = os.environ['POST_URI']
   
   # We use requests to create a post request with an array of streaming rows, in this case we have one row to push
   response = requests.post(
       postUri,
       data=json.dumps([{"value": sample_data}]),
       headers={
           "Authorization": "Bearer " + access_token,
           "Content-Type": "application/json",
       },
   )
   return response

# Call the function
response = push_row()
print(response.status_code, response.reason, response.text)

response

COPES STRING ATTEMPTED: offline_access compass:view compass:edit compass:discover api:write-data api:read-data streaming:write streaming:write-data stream-proxy:write-data stream-proxy:write
TOKEN RESPONSE CONTENTS:
{'access_token': '<>', 'scope': 'api:read-data api:write-data', 'expires_in': 3600, 'token_type': 'bearer'}
403 Forbidden {"errorCode":"PERMISSION_DENIED","errorName":"Security:PermissionDenied","errorInstanceId":"b12750c3-d30d-4d5e-ad36-6be9ff1bb463","parameters":{"rid":"<>","operation":"streaming:write"}}

conjecture/questions

The compass:edit scope is included in the console generated code. It and the other compass scopes are in the foundry dev tools package. Are these scopes no longer active/respected? Why are they not showing up when the api scopes are?

Given that the response is now returning some scopes… what is the correct scope that includes permissions to execute the streaming:write operation? I attempted many combinations of the words suggested… and still receiving the same error.

We have a custom role that controls who can create streams in which projects. It’s used in addition to a customized editor role (therefore the following permissions may not be everything you need).

I notice that our custom streaming role, which has been working just fine for us so far but hasn’t been exahaustively tested, has the following permissions:

streaming:create
streaming:delete
streaming:write
streaming:read
streaming:read-resource
build2:cancel-build-with-output
streaming:manage-resource

Thank you all for your inputs. I have gotten the streaming push to work and I’ve learned a bunch through the process!

The fix: making sure the service user associated with the access is related to an Oauth Client and not an Application.

Reproducing the issue

As part of the process of creating an application you can choose if it has access to the ontology. On my first sets of applications I allowed them access to the ontology. This is labeled as a choice one cannot go back on later.

Apps with access to the ontology (even if don’t select any object types or actions) end up in the Applications portion of the Developer Console. Apps that are created without access to the ontology end up in the OAuth Clients section of the Developer Console.

When I attempted to control for all of other conditions - Oauth access tokens associated with Applications Service Users get the PERMISSIONS DENIED error (even with the same project, file and roles). Oauth access tokens with associated with OAuth Client Service Users were able to streaming:write.

As far as I can tell everything else about this applications and their service users is the same: it is just the access to the ontology that differs:

1 Like