I have a table that I am syncing into Foundry that contains URLs of websites I need to web scrape content from to use downstream. However, since Foundry has strict egress limitations, the content can’t be scraped from within Foundry. I am debating between 2 approaches -
Create a connection to an external server/serverless function that can scrape and return content from a URL
Pre-scrape the content before it is ingested into Foundry - In the database, instead of storing just the URL to scrape the content from, run a background task (from the application server) to scrape the actual content and store it in the database. Then ingest and process the content in Foundry.
Wondering if anyone has had a similar use case before and how they approached it.
There’s an important point to make here around security. The reason the egress is limited, is that this is one of the guardrails protecting your data and users. Without this, you will be vulnerable to data leaving the platform (e.g. by a GET-request).
I would go with an external service that you set up, where you ensure that the incoming data has been parsed, cleaned and analysed, so you can lower your risk of incoming data containing things like prompt injections, etc.
This is an important component in scraping systems, and I always see it as a good lesson for checking your priors and understanding the potential attack surface that this can open up for. Building this service also gives you the opportunity to build in things like caching, etc. which is kinder on network traffic.