Understanding compaction configuration

Redis modules RedisTimeSeries
1

Hi there,

I have a use case where I would like to save page views with a configurable retention period (let’s say 1 year). When queries are made, my purpose is to have low latency to get:

  • page views per day for a given time period (e.g. week, month, year, or custom interval).
  • page views per hour for the last 2 days.

After reading the Redis time series documentation, aggregation, compaction, and downsampling features look perfect for my use case. However, several questions that seem not addressed in the docs came to my mind:

  1. When TS.CREATERULE is used, can we use the same key as sourceKey and destKey? or is it not recommended at all?
  2. If a compaction rule is created, and only the destKey is used for queries, meaning the sourceKey (that is a timeseries) is only there to act as a buffer for compaction rules, can I use a sourceKey with a very short retention period (e.g. 1000ms) or this may prevent the compaction rule to handle all incoming adds if the module do some processing in the background? (I have no knowledge about redistimeseries internal).
  3. It seems possible to configure compaction when the redistimeseries module is loaded. What’s the difference with creating a compaction rule? Is it the same as creating compaction rules for each timeseries, with the same key as source and dest? or does compaction rules defined at module configuration create new keys (timeseries)? In that case, what is the name?
  4. Why is there no mean to define a compaction rule when a new timeseries is created with TS.CREATE?
1 Like
2

Hi @lpellegr,

  1. I think its not possible, you can’t create a rule that “reads” and writes to the same key.
  2. Yes! thats totally possible, just note that if you are doing out-of-order updates on your source key it will reject any samples out of the retention time.
    You can even create multiple rules with different time buckets.
  3. Its possible to configure automatic rules at boot time: https://oss.redislabs.com/redistimeseries/configuration/#compaction_policy-policy
    The way it works is that if you don’t call ts.create, but just use ts.add it will create the source key + all the downsampled keys and the rules.
    The created keys will be called SOURCE_KEY_AGGREGATION_TIMEBUCKET.
    You can also configure the default retention period that way.
  4. Since rule creation could fail, it would make ts.create more cumbersome so we opted to create a different command for that .
1 Like
3

Thanks for your clear explanations.

If I understand properly, using an intermediate timeseries (i.e. having 2 timeseries) is required for compaction. It is not possible to have compaction applied directly to a single timeseries.

4

@lpellegr You are correct, a rule needs a source and a destination key. Having said that you can also have multiple rules from the same source key.

5

I want to configure compaction for a TS:

  • All data points (one every 10 minutes) kept for 4 weeks
  • one data point per hour (sum 6 dp from source) for 6 months
  • one data point per 4 hours (sum 4 dp from previous) for 2 years

Following this discussion, I used the following setup:

ONE_HOUR=3600000
FOUR_WEEKS=2419200000
SIX_MONTH=14515200000
TWO_YEARS=58060800000

TS.CREATE mytskey         RETENTION $FOUR_WEEKS
TS.CREATE mytskey_SUM_M6  RETENTION $SIX_MONTH
TS.CREATE mytskey_SUM_Y2  RETENTION $TWO_YEARS
TS.CREATERULE mytskey mytskey_SUM_M6 AGGREGATION sum $ONE_HOUR
TS.CREATERULE mytskey mytskey_SUM_Y2 AGGREGATION sum $((4*${ONE_HOUR}))

And plan to add entries only on the first one.

Is it the right way to proceed ?