How to fine tune CHUNK_SIZE value?

Redis modules RedisTimeSeries
1

Hi there,

When using TS.CREATE, we can specify a value for the CHUNK_SIZE option. Based on the documentation, this value defines the amount of memory, in bytes, allocated for data. However, to what data are you referring?

How to set an appropriate value based on the values associated with samples and the number of samples we expect to add in a timeseries? what happens if the value is too small or too big?

2

The data in each key is stores in chunks, each chunks contains some of the data. you can use the default and you don’t have to define the chunk size on every key. The current default is 4K bytes. which is basically a single memory page. there are tradeoffs for having smaller or larger sizes of chunks and it really depends on your usage.

3

Thanks for helping. Since timestamps and values from samples are stored in chunks, I guess the CHUNK_SIZE value really depends on how things work internally.

Do you have an example where a higher value than the default is a benefit and another where a lower value is a benefit? I don’t know if that’s the best scenario to consider, but for instance when the injection and query rate is constant and high, and another where the injection rate and query rate have gaps and fluctuates?

4

the chunk size will impact at :

  1. insert time - if you allocate more chunks per samples you occur more penalty due to that.
  2. query time - if you have a very large key, if you have a query for a small subset of data that contains in a very large chunk we will iterate over the chunk to find your data.
  3. retention - if your retention time means that you keep only 10 samples, you probably don’t need 4K ram for that, you can live with a much smaller size.

If you are not sure about your use-case i would suggest to stick with the default, if you don’t have a lot of samples per-key having a smaller chunk would benefit you more (space wise, you can go down to 256).

5

That’s helping a lot. Thanks again!

6

Decemialise the frequency create even size files at an even static, 20, I hope this helps.

7

Partially related, you can find some details on chunk size vs compression vs backfilling performance here: https://oss.redislabs.com/redistimeseries/out-of-order_performance_considerations/ This is however on the write path.