I have a Time series object type containing approximately 500,000 objects. I’m using FoundryTS for transforming these time series data. My goal is to process the entire dataset by fetching every distinct time series node within my object type.
Currently, I’m encountering a limitation with fts.search.series(), which only retrieves up to 10,000 nodes. For my current approach, I’ve tried creating a property filled with the same value using fts.search.series() to fetch every object, but this method doesn’t work beyond 10,000 nodes. I am also noticing duplicates of some of the nodes.
Could someone advise on an alternative approach to fetch all 500,000 time series nodes without relying on fts.search.series()? I’m looking for a method that allows me to efficiently process the entire dataset.
Your insights or examples on how to achieve this would be greatly appreciated!
I have a Time series object type containing approximately 500,000 objects. I’m using FoundryTS for transforming these time series data. My goal is to process the entire dataset by fetching every distinct time series node within my object type.
Currently, I’m encountering a limitation with fts.search.series(), which only retrieves up to 10,000 nodes. For my current approach, I’ve tried creating a property filled with the same value using fts.search.series() to fetch every object, but this method doesn’t work beyond 10,000 nodes.
Could someone advise on an alternative approach to fetch all 500,000 time series nodes without relying on fts.search.series()? I’m looking for a method that allows me to efficiently process the entire dataset, possibly using batch processing or iterative fetching techniques.
Your insights or examples on how to achieve this would be greatly appreciated!
Hi @etuhabonye, thanks for reaching out with your question! To better assist you, could you share more details about the analysis you’re aiming to perform, particularly the final form you expect your data to take? Understanding the desired shape of your result will allow us to guide you more effectively.
Regarding your query with the 500,000 series from your time series properties, it seems you’re applying transformations to each series independently. If this results in 500,000 separate rows in your final dataframe, you should consider batching your queries from the beginning, even before you make the search query. Handling queries on such a large number of series will overwhelm the FoundryTS computation backend, even if you get past search.
The FoundryTS backend is more efficient at processing queries with large volumes of data points within fewer series rather than handling a large number of series each with a few points. I would recommend processing your dataset in smaller chunks, such as batches of 100, and then combining the results at the end.
Could you confirm if this approach aligns with your current process, and let us know if implementing batching makes a difference in your analysis? I’m happy to help with any follow-up questions you might have.