Hi all,
I would like to perform a fleet analysis based on timeseries data. For that purpose I will use foundryTS library to perform queries on the timeseries.
I understand that the analysis can be quite heavy in terms of computation time if I want to analyse 100 aircraft that performed 500.000 flights over the last 5 years. That’s the reason why I’m looking for some best practices with respect to timeseries analysis with foundryTS.
Here is my use case: for each flight, I need to extract some features during the take off phase. I need 3 parameters to identify the events during the flight (altitude and engine temperatures), then I want to average 60 parameters over a window of 4 seconds around the identified events.
I tried several strategies on a sample of aircraft before launching the analysis on the complete fleet.
- First strategy: for each msn and each flight (interval is one flight), I first perform an interpolation at 1Hz to align all the data on the same timestamp, then I apply a udf that identify the event and aggregate the data for the 60 parameters
- Second strategy: for each msn and each flight (for loop), I perform several foundryTS queries to identify my event (time_series_search) then another query to compute statistics
- Third strategy: for each msn and each flight (interval is one flight), I build a first query to identify my events (with interpolation and a udf but only for the three parameters required), then I build a second query with all the parameters to compute the statistics on the window of 4 seconds (without interpolation)
My conclusion so far are the following: - First strategy works well but might be a little expensive due to many interpolation (on the 60 parameters I’m interested in) and usage of udf not as optimized as native foundryTS functions (like statistics)
- Second strategy is very slow because there are too many foundryTS API calls
- Third strategy is not as good as expected. The first build to detect the events is very fast (a couple of minutes per msn) but the second is desperately long.
Before I launch the analysis on the whole fleet, I would like to have an expert opinion about the best way to launch such an analysis with foundryTS, how to optimize the code so that it minimizes compute time and ressources.
Thanks and regards
Thomas