Aggregating news from external sources

Ask the Community
,
1

Does anyone have any use-cases that pull news stories from external sources and maybe summarises them using AIP?

2

Yes - I’m currently working on a marketplace deployable product which will solve this problem. That having been said, it probably won’t be ready in the timeline you presumably need, so here’s the gist of what we’re putting together:

  1. Get yourself a subscription to Moody’s NewsEdge or similar
  2. Set up a REST API source in data connection, storing your Moody’s NewsEdge account and password as secrets on it.
  3. Set up an incremental external transform which takes as input the source
  4. In your external transform, hit the get token endpoint for Moody’s NewsEdge, using the secrets stored on the source as inputs to this query.
  5. Using the token you receive back, hit the GET or POST endpoints on NewsEdge for predefined queries related to your organisation. I set up an object and a small workshop to be able to manage the queries that are being asked of Moody’s. I recommend asking for 1 or 2 days worth of articles at a time - preferably 1 day or less.
  6. Put this external transform on a schedule that is as frequent as the time period you specified in step 5.
  7. Use pipeline builder to build an incremental pipeline off this, which parses the JSON text as a struct, then explodes/flattens it until you have usable columns with the article contents, etc.
  8. Feed this exploded version of the JSON into another pipeline builder pipeline which uses LLM blocks to summarise incoming articles. Ensure that caching is enabled on the LLM blocks in case duplicates occur - this is quite likely. I recommend extracting the following - “relevance to my organisation”, “sentiment towards my organistion”, “summary in ” etc.
  9. Create appropriate objects off the outputs of this.
  10. Optionally, create AIP logics, notepad tempates and automations to generate a weekly summary from these summaries, selecting the most critical stories, and email to individuals who need this information. You can create a subscription object to allow users to choose to subscribe and unsubscribe themselves - they can even subscribe only to certain topics - based on tags returned by Moody’s.
3

Thanks for the detailed instructions!

What’s the timeline on your Marketplace product?