Redis 隔几分钟“抽风”一次：一个 200 万字段的 Hash，拖着全公司业务一起卡

$ redis-cli -h 10.0.0.21 -p 6379 --latency-history -i 5
min: 0, max: 1, avg: 0.19 (1314 samples) -- 5.01 seconds range
min: 0, max: 1, avg: 0.18 (1299 samples) -- 5.00 seconds range
min: 0, max: 2314, avg: 27.65 (312 samples) -- 5.01 seconds range
min: 0, max: 1, avg: 0.20 (1301 samples) -- 5.00 seconds range

$ redis-cli -h 10.0.0.21 config get slowlog-log-slower-than
1) "slowlog-log-slower-than"
2) "10000"
$ redis-cli -h 10.0.0.21 config get slowlog-max-len
1) "slowlog-max-len"
2) "128"

$ redis-cli -h 10.0.0.21 slowlog get 5
1) 1) (integer) 4823
   2) (integer) 1730272931
   3) (integer) 1874553
   4) 1) "HGETALL"
      2) "cart:promo:hot"
   5) "10.0.2.15:52344"
   6) "order-service"
2) 1) (integer) 4822
   2) (integer) 1730272611
   3) (integer) 2130971
   4) 1) "HGETALL"
      2) "cart:promo:hot"
   5) "10.0.2.16:41188"
   6) "order-service"
3) 1) (integer) 4821
   2) (integer) 1730272301
   3) (integer) 312044
   4) 1) "SMEMBERS"
      2) "user:tags:all"
   5) "10.0.3.7:38820"
   6) "profile-service"
...

$ redis-cli -h 10.0.0.21 info commandstats | egrep 'hgetall|smembers|keys|get:|hget:'
cmdstat_get:calls=918273645,usec=1102394817,usec_per_call=1.20,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=88123411,usec=112009231,usec_per_call=1.27,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=48211,usec=39823310977,usec_per_call=826043.51,rejected_calls=0,failed_calls=0
cmdstat_smembers:calls=9021,usec=1204993211,usec_per_call=133572.09,rejected_calls=0,failed_calls=0

$ redis-cli -h 10.0.0.22 --bigkeys -i 0.1

# Scanning the entire keyspace to find biggest keys as well as
# average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
# per 100 SCAN commands (not usually needed).

[00.00%] Biggest string found so far '"session:8a3f..."' with 512 bytes
[03.11%] Biggest hash   found so far '"cart:promo:hot"' with 412007 fields
[17.52%] Biggest set    found so far '"user:tags:all"' with 903112 members
[41.90%] Biggest hash   found so far '"cart:promo:hot"' with 2018733 fields
...
-------- summary -------

Sampled 4823017 keys in the keyspace!
Total key length in bytes is 121803321 (avg len 25.25)

Biggest   hash found '"cart:promo:hot"' has 2018733 fields
Biggest    set found '"user:tags:all"' has 903112 members
Biggest string found '"cfg:rules:json"' has 3145728 bytes

0 lists with 0 items (00.00% of keys, avg size 0.00)
2811003 hashs with 6871390 fields (58.29% of keys, avg size 2.44)
1930811 strings with 812340111 bytes (40.03% of keys, avg size 420.71)
...

$ redis-cli -h 10.0.0.22 --memkeys -i 0.1
...
Biggest   hash found '"cart:promo:hot"' has 187436412 bytes
Biggest    set found '"user:tags:all"' has 74510223 bytes
Biggest string found '"cfg:rules:json"' has 3145784 bytes

$ redis-cli -h 10.0.0.22 memory usage cart:promo:hot samples 0
(integer) 187436472
$ redis-cli -h 10.0.0.22 hlen cart:promo:hot
(integer) 2018733
$ redis-cli -h 10.0.0.22 type cart:promo:hot
hash
$ redis-cli -h 10.0.0.22 object encoding cart:promo:hot
"hashtable"

$ redis-cli -h 10.0.0.21 config set latency-monitor-threshold 100
OK
# 等下一次卡顿后：
$ redis-cli -h 10.0.0.21 latency latest
1) 1) "command"
   2) (integer) 1730273251
   3) (integer) 1874
   4) (integer) 2130
$ redis-cli -h 10.0.0.21 latency doctor
Dave, I have observed latency spikes in this Redis instance. You don't mind talking about it, do you Dave?

1. command: 5 latency spikes (average 1533ms, mean deviation 412ms, period 302.20 sec). Worst all time event 2130ms.

I have a few advices for you:

- Check your Slow Log to understand what are the commands you are running which are too slow to execute. Please check http://redis.io/commands/slowlog for more information.
- Deleting, expiring or evicting (because of maxmemory policy) large objects is a blocking operation. If you have very large objects that are often deleted, expired, or evicted, try to fragment those objects into multiple smaller objects.

// 每次下单前，把活动参与记录整个拉回来，在内存里判断用户是否参与过
Map<String, String> all = stringRedisTemplate.opsForHash().entries("cart:promo:hot");
boolean joined = all.containsKey(String.valueOf(userId));

# 先看看有多大，再决定怎么删
$ redis-cli -h 10.0.0.21 hlen cart:promo:hot
(integer) 2018733

# 方案 A：整个 key 都不要了，异步删除（Redis 4.0+）
$ redis-cli -h 10.0.0.21 unlink cart:promo:hot
(integer) 1

# 方案 B：key 还要留，但想分批瘦身（避免一次性大删除）——用 HSCAN 分批删 field
$ redis-cli -h 10.0.0.21 hscan cart:promo:hot 0 count 500
1) "1048576"
2) 1) "u_100001"
   2) "{\"joined\":true}"
   ...

public void shrinkBigHash(String key) throws InterruptedException {
    ScanOptions opts = ScanOptions.scanOptions().count(500).build();
    try (Cursor<Map.Entry<Object, Object>> cursor = redis.opsForHash().scan(key, opts)) {
        List<Object> batch = new ArrayList<>(500);
        while (cursor.hasNext()) {
            batch.add(cursor.next().getKey());
            if (batch.size() >= 500) {
                redis.opsForHash().delete(key, batch.toArray());
                batch.clear();
                Thread.sleep(10);          // 让出主线程，别人还要用
            }
        }
        if (!batch.isEmpty()) redis.opsForHash().delete(key, batch.toArray());
    }
}

# 用 SCAN 替代 KEYS 的命令行写法
$ redis-cli -h 10.0.0.22 --scan --pattern 'cart:promo:*' -i 0.01 | head
# 用 HEXISTS 判断用户是否参与（O(1)）
$ redis-cli -h 10.0.0.21 hexists cart:promo:hot u_100001
(integer) 1

boolean joined = Boolean.TRUE.equals(stringRedisTemplate.opsForHash().hasKey("cart:promo:hot", String.valueOf(userId)));

// 分桶键：cart:promo:{bucket}，bucket = hash(userId) % 1024
private String bucketKey(long userId) {
    int bucket = (int) (Math.abs(Long.hashCode(userId)) % 1024);
    return "cart:promo:" + bucket;
}
redis.opsForHash().put(bucketKey(userId), String.valueOf(userId), json);
redis.opsForHash().hasKey(bucketKey(userId), String.valueOf(userId));

# 每周在从库上跑一次，输出存档；-i 0.1 降低压力
$ redis-cli -h 10.0.0.22 --bigkeys -i 0.1 > /data/redis-audit/bigkeys-$(date +%F).txt

# 同时打开延迟监控和慢日志，并调小慢日志阈值（单位微秒，这里 5ms）
$ redis-cli -h 10.0.0.21 config set slowlog-log-slower-than 5000
$ redis-cli -h 10.0.0.21 config set slowlog-max-len 1000
$ redis-cli -h 10.0.0.21 config set latency-monitor-threshold 50

# 让 DEL/过期/淘汰默认走后台线程回收（Redis 4.0+ 的 lazy free）
$ redis-cli -h 10.0.0.21 config set lazyfree-lazy-user-del yes
$ redis-cli -h 10.0.0.21 config set lazyfree-lazy-expire yes
$ redis-cli -h 10.0.0.21 config set lazyfree-lazy-eviction yes

$ redis-cli -h 10.0.0.21 --latency-history -i 15
min: 0, max: 1, avg: 0.17 (1350 samples) -- 15.01 seconds range
min: 0, max: 1, avg: 0.18 (1348 samples) -- 15.00 seconds range
min: 0, max: 2, avg: 0.18 (1349 samples) -- 15.00 seconds range

$ redis-cli -h 10.0.0.21 slowlog get 3
(empty array)