Using HSET increases memory consumption

monte
Thu, 28 Sep 2023 19:24:22 +0000
Redis commands & data structures

I have a groupid and a clientid and the value is val. All are ints, the length of groupid and clientid can go up to 7 digits from a minimum of 5 digits, while the val can only be three digits.

I tried storing this data by following two methods:

1. method 1: I am trying to store this data using HSET, by using this logic, hash I create by concatenating groupid and clientid and then doing floor division by 400, the key I am creating by concatenating groupid and clientid and then converting to int. The hash-max-ziplist-entries is 512 (hence the floor division by 400) and hash-max-ziplist-value is 64.

2. method 2: I create a 64 bit int where groupid is stored in 32 bit and clientid is also stored in 32 bit but left shifted. val is same, three digit int.

When I compare these two methods, method 2 takes less memory than method 1. I thought this should not be the case, as using hset is a way to optimise space (as indicated in this article). So I am presuming, I am doing something wrong in method 1.

For small dataset (roughly 100k pairs) I did see, method 1 taking less memory but when I attempt to store 75 million pairs, method 2 turns out to be better.

1 post - 1 participant