We can monitor derived streams with some of the monitoring view options for checkpoints, but raw streams from a data ingest do not have checkpoints. How can I ensure that records are flowing through my raw stream?
Hi! Monitoring for raw stream ingests is in active development. We will reach back out once this is generally available.
Out of curiosity – when would you want an alert to fire? For example, “alert when 0 records have been ingested in the last 30 minutes”.
Hi @tpark there would be a few scenarios I would want to alert on
- The one you mentioned about number of records
- Looking at the timestamps in incoming records to see if they are more than X minutes in the past
- Whether the stream is actually running or not right now. The schedule monitoring is not good enough for this because streaming schedules always end in failure.
@dherls Thanks for the context.
Looking at the timestamps in incoming records to see if they are more than X minutes in the past
Would you alert on now - time record is created in source system or now - time record enters Foundry?
Whether the stream is actually running or not right now.
Gotcha, we’re thinking of this as a “Liveness” monitor. We will decomp this and and look into adding it to our roadmap.
For raw streams, definitely now - source system. For later derived streams, now - time record enters foundry would be a good indicator of total e2e processing time
1 Like