I have a very strange crash. I have a redis cluster (two nodes master-slave) with sentinels. Every 5 minutes I have file synchronization process to the disk.
I am starting receive an error from redis in java code and in a half of an hour all data was erased.
All requests from java are logged and there no any DELETE or FLUSH requests. in 14:54 I am starting receive empty data from redis and aboult 15:28 the data was erased (in data metrics).
No any logs in redis from 14:54 to 15:28.
I see the in 15:28 connection with master and replica was lost.
And before the crash, there was a strange memory desynchronization between the master and the slave, a very interesting question is why?
I don’t understand how it can be, maybe I am losing something, need help.
redis 1 log
950:M 18 Aug 2023 14:53:02.052 * 10 changes in 300 seconds. Saving...
950:M 18 Aug 2023 14:53:02.291 * Background saving started by pid 11769
11769:C 18 Aug 2023 14:53:02.298 * DB saved on disk
11769:C 18 Aug 2023 14:53:02.532 * RDB: 2 MB of memory used by copy-on-write
950:M 18 Aug 2023 14:53:02.794 * Background saving terminated with success
950:M 18 Aug 2023 15:28:58.624 # Connection with replica 10.11.78.5:6379 lost.
950:signal-handler (1692361845) Received SIGTERM scheduling shutdown...
950:M 18 Aug 2023 15:30:45.946 # User requested shutdown...
950:M 18 Aug 2023 15:30:45.946 * Saving the final RDB snapshot before exiting.
950:M 18 Aug 2023 15:30:45.949 * DB saved on disk
950:M 18 Aug 2023 15:30:45.949 * Removing the pid file.
950:M 18 Aug 2023 15:30:45.949 # Redis is now ready to exit, bye bye...
16576:C 18 Aug 2023 15:31:03.823 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16576:C 18 Aug 2023 15:31:03.823 # Redis version=6.2.9, bits=64, commit=00000000, modified=0, pid=16576, just started
16576:C 18 Aug 2023 15:31:03.823 # Configuration loaded
16576:M 18 Aug 2023 15:31:03.824 * monotonic clock: POSIX clock_gettime
16576:M 18 Aug 2023 15:31:03.825 * Running mode=standalone, port=6379.
16576:M 18 Aug 2023 15:31:03.825 # Server initialized
16576:M 18 Aug 2023 15:31:03.825 * Loading RDB produced by version 6.2.9
16576:M 18 Aug 2023 15:31:03.825 * RDB age 18 seconds
16576:M 18 Aug 2023 15:31:03.825 * RDB memory usage when created 7748.97 Mb
16576:M 18 Aug 2023 15:31:03.825 # Done loading RDB, keys loaded: 0, keys expired: 0.
16576:M 18 Aug 2023 15:31:03.825 * DB loaded from disk: 0.000 seconds
16576:M 18 Aug 2023 15:31:03.825 * Ready to accept connections
16576:S 18 Aug 2023 15:31:14.454 * Before turning into a replica, using my own master parameters to synthesize a cached master: I may be able to synchronize with the new master with just a partial transfer.
16576:S 18 Aug 2023 15:31:14.455 * Connecting to MASTER 10.11.78.5:6379
16576:S 18 Aug 2023 15:31:14.455 * MASTER <-> REPLICA sync started
16576:S 18 Aug 2023 15:31:14.455 * REPLICAOF 10.11.78.5:6379 enabled (user request from 'id=3 addr=10.11.78.5:59715 laddr=10.11.80.101:6379 fd=8 name=sentinel-e8d3f92d-cmd age=10 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=199 qbuf-free=40755 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
16576:S 18 Aug 2023 15:31:14.460 # CONFIG REWRITE executed with success.
16576:S 18 Aug 2023 15:31:14.460 * Non blocking connect for SYNC fired the event.
16576:S 18 Aug 2023 15:31:14.461 * Master replied to PING, replication can continue...
16576:S 18 Aug 2023 15:31:14.462 * Trying a partial resynchronization (request 925b873f88472f093bacba80764ee9113c45f743:1).
16576:S 18 Aug 2023 15:31:19.340 * Full resync from master: 8e60450bb6636938508cb0ebc0faebaff3e8ee78:2902471125
16576:S 18 Aug 2023 15:31:19.340 * Discarding previously cached master state.
16576:S 18 Aug 2023 15:31:19.349 * MASTER <-> REPLICA sync: receiving streamed RDB from master with EOF to disk
16576:S 18 Aug 2023 15:31:19.356 * MASTER <-> REPLICA sync: Flushing old data
16576:S 18 Aug 2023 15:31:19.356 * MASTER <-> REPLICA sync: Loading DB in memory
16576:S 18 Aug 2023 15:31:19.364 * Loading RDB produced by version 6.2.9
16576:S 18 Aug 2023 15:31:19.364 * RDB age 0 seconds
16576:S 18 Aug 2023 15:31:19.364 * RDB memory usage when created 11.62 Mb
16576:S 18 Aug 2023 15:31:19.369 # Done loading RDB, keys loaded: 510, keys expired: 0.
16576:S 18 Aug 2023 15:31:19.369 * MASTER <-> REPLICA sync: Finished with success
16576:S 18 Aug 2023 15:36:04.083 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 15:36:04.084 * Background saving started by pid 17126
17126:C 18 Aug 2023 15:36:04.132 * DB saved on disk
17126:C 18 Aug 2023 15:36:04.132 * RDB: 0 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:36:04.184 * Background saving terminated with success
16576:S 18 Aug 2023 15:39:42.138 * 10000 changes in 60 seconds. Saving...
16576:S 18 Aug 2023 15:39:42.140 * Background saving started by pid 17527
17527:C 18 Aug 2023 15:39:42.378 * DB saved on disk
17527:C 18 Aug 2023 15:39:42.380 * RDB: 1 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:39:42.442 * Background saving terminated with success
16576:S 18 Aug 2023 15:44:43.062 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 15:44:43.065 * Background saving started by pid 18569
18569:C 18 Aug 2023 15:44:43.460 * DB saved on disk
18569:C 18 Aug 2023 15:44:43.463 * RDB: 1 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:44:43.467 * Background saving terminated with success
16576:S 18 Aug 2023 15:49:44.056 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 15:49:44.060 * Background saving started by pid 19039
19039:C 18 Aug 2023 15:49:44.596 * DB saved on disk
19039:C 18 Aug 2023 15:49:44.600 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:49:44.665 * Background saving terminated with success
16576:S 18 Aug 2023 15:54:45.039 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 15:54:45.043 * Background saving started by pid 19705
19705:C 18 Aug 2023 15:54:45.655 * DB saved on disk
19705:C 18 Aug 2023 15:54:45.659 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:54:45.747 * Background saving terminated with success
16576:S 18 Aug 2023 15:59:46.098 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 15:59:46.103 * Background saving started by pid 20478
20478:C 18 Aug 2023 15:59:46.802 * DB saved on disk
20478:C 18 Aug 2023 15:59:46.807 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 15:59:46.908 * Background saving terminated with success
16576:S 18 Aug 2023 16:04:47.063 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 16:04:47.069 * Background saving started by pid 21002
21002:C 18 Aug 2023 16:04:47.850 * DB saved on disk
21002:C 18 Aug 2023 16:04:47.856 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 16:04:47.872 * Background saving terminated with success
16576:S 18 Aug 2023 16:09:48.085 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 16:09:48.091 * Background saving started by pid 21554
21554:C 18 Aug 2023 16:09:48.958 * DB saved on disk
21554:C 18 Aug 2023 16:09:48.965 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 16:09:48.996 * Background saving terminated with success
16576:S 18 Aug 2023 16:14:49.000 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 16:14:49.008 * Background saving started by pid 22208
22208:C 18 Aug 2023 16:14:49.976 * DB saved on disk
22208:C 18 Aug 2023 16:14:49.983 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 16:14:50.014 * Background saving terminated with success
16576:S 18 Aug 2023 16:19:51.090 * 10 changes in 300 seconds. Saving...
16576:S 18 Aug 2023 16:19:51.097 * Background saving started by pid 22663
22663:C 18 Aug 2023 16:19:52.128 * DB saved on disk
22663:C 18 Aug 2023 16:19:52.135 * RDB: 2 MB of memory used by copy-on-write
16576:S 18 Aug 2023 16:19:52.204 * Background saving terminated with success
sentinel 1 log
931:X 18 Aug 2023 15:28:58.404 # Executing user requested FAILOVER of 'bills'
931:X 18 Aug 2023 15:28:58.404 # +new-epoch 2
931:X 18 Aug 2023 15:28:58.404 # +try-failover master bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:58.493 # +vote-for-leader 4e236586f916fb5b96fe83434b701993c17c06ce 2
931:X 18 Aug 2023 15:28:58.493 # +elected-leader master bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:58.493 # +failover-state-select-slave master bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:58.552 # +selected-slave slave 10.11.78.5:6379 10.11.78.5 6379 @ bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:58.552 * +failover-state-send-slaveof-noone slave 10.11.78.5:6379 10.11.78.5 6379 @ bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:58.623 * +failover-state-wait-promotion slave 10.11.78.5:6379 10.11.78.5 6379 @ bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:59.504 # +promoted-slave slave 10.11.78.5:6379 10.11.78.5 6379 @ bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:59.504 # +failover-state-reconf-slaves master bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:59.564 # +failover-end master bills 10.11.80.101 6379
931:X 18 Aug 2023 15:28:59.564 # +switch-master bills 10.11.80.101 6379 10.11.78.5 6379
931:X 18 Aug 2023 15:28:59.564 * +slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:09.709 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:19.754 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:29.825 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:39.839 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:49.868 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:29:59.915 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:30:09.957 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:30:19.989 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:30:30.028 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:X 18 Aug 2023 15:30:40.085 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
931:signal-handler (1692361845) Received SIGTERM scheduling shutdown...
931:X 18 Aug 2023 15:30:45.950 # User requested shutdown...
931:X 18 Aug 2023 15:30:45.950 * Removing the pid file.
931:X 18 Aug 2023 15:30:45.950 # Sentinel is now ready to exit, bye bye...
16569:X 18 Aug 2023 15:31:03.607 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
16569:X 18 Aug 2023 15:31:03.607 # Redis version=6.2.9, bits=64, commit=00000000, modified=0, pid=16569, just started
16569:X 18 Aug 2023 15:31:03.607 # Configuration loaded
16569:X 18 Aug 2023 15:31:03.608 * monotonic clock: POSIX clock_gettime
16569:X 18 Aug 2023 15:31:03.608 * Running mode=sentinel, port=26379.
16569:X 18 Aug 2023 15:31:03.608 # Sentinel ID is 4e236586f916fb5b96fe83434b701993c17c06ce
16569:X 18 Aug 2023 15:31:03.608 # +monitor master bills 10.11.78.5 6379 quorum 2
16569:X 18 Aug 2023 15:31:13.674 # +sdown sentinel 84aa500347818b93129695809e01c7c45f62cfc4 10.63.1.12 26753 @ bills 10.11.78.5 6379
redis 2 log
26943:S 18 Aug 2023 14:50:07.261 * Background saving terminated with success
26943:S 18 Aug 2023 14:51:39.602 * 10000 changes in 60 seconds. Saving...
26943:S 18 Aug 2023 14:51:39.609 * Background saving started by pid 3628
3628:C 18 Aug 2023 14:51:39.644 * DB saved on disk
3628:C 18 Aug 2023 14:51:39.649 * RDB: 7 MB of memory used by copy-on-write
26943:S 18 Aug 2023 14:51:39.709 * Background saving terminated with success
26943:S 18 Aug 2023 14:56:40.008 * 10 changes in 300 seconds. Saving...
26943:S 18 Aug 2023 14:56:40.013 * Background saving started by pid 4206
4206:C 18 Aug 2023 14:56:40.016 * DB saved on disk
4206:C 18 Aug 2023 14:56:40.019 * RDB: 0 MB of memory used by copy-on-write
26943:S 18 Aug 2023 14:56:40.114 * Background saving terminated with success
26943:M 18 Aug 2023 15:28:58.623 # Connection with master lost.
26943:M 18 Aug 2023 15:28:58.623 * Caching the disconnected master state.
26943:M 18 Aug 2023 15:28:58.623 * Discarding previously cached master state.
26943:M 18 Aug 2023 15:28:58.623 # Setting secondary replication ID to b311e4fbc56acdd45eea0cbbd90d74ae5a4dd263, valid up to offset: 2899375868. New replication ID is 8e60450bb6636938508cb0ebc0faebaff3e8ee78
26943:M 18 Aug 2023 15:28:58.623 * MASTER MODE enabled (user request from 'id=6 addr=10.11.80.101:59607 laddr=10.11.78.5:6379 fd=12 name=sentinel-4e236586-cmd age=488987 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=188 qbuf-free=40766 argv-mem=4 obl=45 oll=0 omem=0 tot-mem=61468 events=r cmd=exec user=default redir=-1')
26943:M 18 Aug 2023 15:28:58.627 # CONFIG REWRITE executed with success.
26943:M 18 Aug 2023 15:30:19.319 * 1 changes in 900 seconds. Saving...
26943:M 18 Aug 2023 15:30:19.324 * Background saving started by pid 7947
7947:C 18 Aug 2023 15:30:19.330 * DB saved on disk
7947:C 18 Aug 2023 15:30:19.333 * RDB: 0 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:30:19.424 * Background saving terminated with success
26943:M 18 Aug 2023 15:31:14.462 * Replica 10.11.80.101:6379 asks for synchronization
26943:M 18 Aug 2023 15:31:14.462 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for '925b873f88472f093bacba80764ee9113c45f743', my replication IDs are '8e60450bb6636938508cb0ebc0faebaff3e8ee78' and 'b311e4fbc56acdd45eea0cbbd90d74ae5a4dd263')
26943:M 18 Aug 2023 15:31:14.462 * Delay next BGSAVE for diskless SYNC
26943:M 18 Aug 2023 15:31:19.340 * Starting BGSAVE for SYNC with target: replicas sockets
26943:M 18 Aug 2023 15:31:19.344 * Background RDB transfer started by pid 8064
8064:C 18 Aug 2023 15:31:19.360 * RDB: 0 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:31:19.360 # Diskless rdb transfer, done reading from pipe, 1 replicas still up.
26943:M 18 Aug 2023 15:31:19.369 * Background RDB transfer terminated with success
26943:M 18 Aug 2023 15:31:19.369 * Streamed RDB transfer with replica 10.11.80.101:6379 succeeded (socket). Waiting for REPLCONF ACK from slave to enable streaming
26943:M 18 Aug 2023 15:31:19.369 * Synchronization with replica 10.11.80.101:6379 succeeded
26943:M 18 Aug 2023 15:35:20.040 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 15:35:20.045 * Background saving started by pid 8485
8485:C 18 Aug 2023 15:35:20.064 * DB saved on disk
8485:C 18 Aug 2023 15:35:20.068 * RDB: 0 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:35:20.146 * Background saving terminated with success
26943:M 18 Aug 2023 15:39:05.633 * 10000 changes in 60 seconds. Saving...
26943:M 18 Aug 2023 15:39:05.640 * Background saving started by pid 8894
8894:C 18 Aug 2023 15:39:05.823 * DB saved on disk
8894:C 18 Aug 2023 15:39:05.828 * RDB: 2 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:39:05.841 * Background saving terminated with success
26943:M 18 Aug 2023 15:44:06.025 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 15:44:06.032 * Background saving started by pid 9336
9336:C 18 Aug 2023 15:44:06.345 * DB saved on disk
9336:C 18 Aug 2023 15:44:06.352 * RDB: 2 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:44:06.435 * Background saving terminated with success
26943:M 18 Aug 2023 15:49:07.014 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 15:49:07.023 * Background saving started by pid 10112
10112:C 18 Aug 2023 15:49:07.460 * DB saved on disk
10112:C 18 Aug 2023 15:49:07.467 * RDB: 3 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:49:07.527 * Background saving terminated with success
26943:M 18 Aug 2023 15:54:08.068 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 15:54:08.077 * Background saving started by pid 10555
10555:C 18 Aug 2023 15:54:08.612 * DB saved on disk
10555:C 18 Aug 2023 15:54:08.619 * RDB: 3 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:54:08.680 * Background saving terminated with success
26943:M 18 Aug 2023 15:59:09.010 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 15:59:09.019 * Background saving started by pid 11149
11149:C 18 Aug 2023 15:59:09.591 * DB saved on disk
11149:C 18 Aug 2023 15:59:09.599 * RDB: 4 MB of memory used by copy-on-write
26943:M 18 Aug 2023 15:59:09.624 * Background saving terminated with success
26943:M 18 Aug 2023 16:04:10.034 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 16:04:10.045 * Background saving started by pid 11942
11942:C 18 Aug 2023 16:04:10.703 * DB saved on disk
11942:C 18 Aug 2023 16:04:10.712 * RDB: 4 MB of memory used by copy-on-write
26943:M 18 Aug 2023 16:04:10.748 * Background saving terminated with success
26943:M 18 Aug 2023 16:09:11.038 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 16:09:11.049 * Background saving started by pid 12472
12472:C 18 Aug 2023 16:09:11.761 * DB saved on disk
12472:C 18 Aug 2023 16:09:11.770 * RDB: 3 MB of memory used by copy-on-write
26943:M 18 Aug 2023 16:09:11.853 * Background saving terminated with success
26943:M 18 Aug 2023 16:14:12.066 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 16:14:12.077 * Background saving started by pid 13047
13047:C 18 Aug 2023 16:14:12.843 * DB saved on disk
13047:C 18 Aug 2023 16:14:12.853 * RDB: 4 MB of memory used by copy-on-write
26943:M 18 Aug 2023 16:14:12.882 * Background saving terminated with success
26943:M 18 Aug 2023 16:19:13.085 * 10 changes in 300 seconds. Saving...
26943:M 18 Aug 2023 16:19:13.096 * Background saving started by pid 13673
13673:C 18 Aug 2023 16:19:13.928 * DB saved on disk
13673:C 18 Aug 2023 16:19:13.938 * RDB: 7 MB of memory used by copy-on-write
26943:M 18 Aug 2023 16:19:14.002 * Background saving terminated with success
sentinel 2 log
26937:X 18 Aug 2023 15:28:59.310 # +new-epoch 2
26937:X 18 Aug 2023 15:28:59.570 # +config-update-from sentinel 4e236586f916fb5b96fe83434b701993c17c06ce 10.11.80.101 26379 @ bills 10.11.80.101 6379
26937:X 18 Aug 2023 15:28:59.570 # +switch-master bills 10.11.80.101 6379 10.11.78.5 6379
26937:X 18 Aug 2023 15:28:59.570 * +slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:09.604 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:19.619 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:29.665 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:39.707 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:49.766 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:29:59.820 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:09.892 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:19.930 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:29.996 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:40.036 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:56.008 # +sdown sentinel 4e236586f916fb5b96fe83434b701993c17c06ce 10.11.80.101 26379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:30:56.751 # +sdown slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:31:03.775 # -sdown sentinel 4e236586f916fb5b96fe83434b701993c17c06ce 10.11.80.101 26379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:31:04.409 * +reboot slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:31:04.463 # -sdown slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
26937:X 18 Aug 2023 15:31:14.454 * +convert-to-slave slave 10.11.80.101:6379 10.11.80.101 6379 @ bills 10.11.78.5 6379
And desynchronization memory between master and slave. Master is green, slave - orange.
