We have .net applications hosted in kubernetes that connect to the AWS Elasticache Redis cluster (cluster mode enabled) which has 3 shards with 2 nodes each - instances -cache.r7g.large.
These are some of the connection timeout errors:
StackExchange.Redis.RedisTimeoutException: Timeout performing PEXPIREAT (5000ms), next: PEXPIREAT qa:b86f017510dc4231897568f5a42bb3d7, inst: 0, qu: 0, qs: 1, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 4, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 4932, async-ops: 11160, serverEndpoint: serverIP:6379, conn-sec: 48208.56, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: some-api-54b23447f6-p4985(SE.Redis-v2.7.10.12442), PerfCounterHelperkeyHashSlot: 12840, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=19,Free=32748,Min=2,Max=32767), POOL: (Threads=19,QueuedItems=7,CompletedItems=4438108,Timers=240), v: 2.7.10.12442 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5820ms elapsed, timeout is 5000ms), command=HEXISTS, next: HEXISTS qa:api-client:9ee:ticket, inst: 0, qu: 0, qs: 5, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 21, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 55175, async-ops: 138784, serverEndpoint: serverIP:6379, conn-sec: 442957.14, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: some-api-78j58b9799-77ds2(SE.Redis-v2.7.10.12442), PerfCounterHelperkeyHashSlot: 3054, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=23,Free=32744,Min=3,Max=32767), POOL: (Threads=31,QueuedItems=36,CompletedItems=7640769,Timers=36), v: 2.7.10.12442 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts
Timeout awaiting response (outbound=4KiB, inbound=0KiB, 5728ms elapsed, timeout is 5000ms), command=HSET, next: HSET qa:a11fb30932a04e81aa4b658fbd853410, inst: 0, qu: 0, qs: 8, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 486, in-pipe: 0, out-pipe: 0, last-in: 1, cur-in: 0, sync-ops: 930, async-ops: 1438, serverEndpoint: serverIP:6379, conn-sec: 5066.95, aoc: 1, mc: 1/1/0, mgr: 10 of 10 available, clientName: some-api-57dcr4558-vd7t9(SE.Redis-v2.7.10.12442), PerfCounterHelperkeyHashSlot: 15135, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=26,Free=32741,Min=3,Max=32767), POOL: (Threads=30,QueuedItems=39,CompletedItems=386484,Timers=37), v: 2.7.10.12442 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts
We are currently investigating this issue.
What could be the possible reasons for such frequent timeouts to occur?