How upload a .csv file from a local computer into a Foundry dataset?

Ask the Community
1

Hi there,

My final goal is to upload a file into a foundry dataset.

For that, I get the transaction id thanks to this documentation :
https://www.palantir.com/docs/foundry/api/datasets-v2-resources/transactions/create-transaction/.


import requests

HOSTNAME = <your_hostname>
TOKEN = <your_token>
headers = {
    "content-type": "application/json",
    "authorization": "Bearer {}".format(TOKEN)
}
request_body={"transactionType":"SNAPSHOT"}

response = requests.post('https://{}/api/v1/datasets/<your_rid>/transactions?branchId=master'.format(HOSTNAME), headers=headers, json=request_body)
print(response)

Then, I’m using the code based on this page https://www.palantir.com/docs/foundry/api/datasets-v2-resources/files/upload-file

The python code which allows to push the file stored in a local computer is:


import requests

HOSTNAME = <your_hostname>
TOKEN = <your_token>
headers = {
    "content-type": "application/octet-stream",
    "authorization": "Bearer {}".format(TOKEN)
}

response = requests.post('https://{}/api/v2/datasets/<your_rid>/files/2020/09/30/trades.csv/upload?branchName=master&transactionType=APPEND&transactionRid=<your_rid>'.format(HOSTNAME), headers=headers, data=open('/path/to/file', 'rb'))
print(response)

In the response variable I’m able to identify dataset rid and transaction id but I don’t know what “/2020/09/30/trades.csv” means in the path. I guess that’s why the code doesn’t work.

Any help would be greatly appreciated.

2

Hi @Arnaud_B , please check out this guide. It’s using Boto3 to upload directly into S3. This might help you. The code is available in GitHub if you want to do a quick C&P.

Hope this helps

3

Thanks for your help it works very well.

Is there a way to solve the issue I had on my code ?
The reason is in a perfect world, I would not use “foundry_dev_tool” package.

4

Hey Arnaud,

Could you elaborate on what you mean by “I guess that’s why the code doesn’t work.”? From your code, it looks like you might not be committing the transaction that you opened (https://www.palantir.com/docs/foundry/api/datasets-v2-resources/transactions/commit-transaction) which is what ultimately persist the file to the dataset.

The “/2020/09/30/trades.csv” part is the path of the file in the dataset. A dataset can contain multiple files and the path is the identifier of a file (seen under files on the dataset).

Finally, the transactionRid in upload-files can be omitted which automatically opens a transactions, uploads the file to the transaction, and then commits the transaction for you.