deduplication.keepLastIfActive performs an O(active) LRANGE + Lua scan on every deduplicated add

Type: Performance
Component: Deduplication / Lua scripts
Affected versions: v5.72.0 (2026-04-01) through current master (6.1.2), all backends that use the Redis scripts
File: src/commands/includes/storeDeduplicatedNextJob.lua

Summary

When deduplication.keepLastIfActive is enabled, every deduplicated add() triggers a full read of the queue's active list into the Lua VM, followed by a linear scan in Lua to test membership of a single job ID.

Because the whole point of the option is to collapse a high rate of repeated adds, the hit rate on this path is high by construction. The cost therefore scales as O(active_depth × deduplicated_add_rate), executed serially on the Redis/Valkey main thread inside an atomic script.

The codebase already has a version-gated LPOS-based membership check used for exactly this purpose elsewhere (getStateV2-8.lua, RedisQueueBackend#isJobInState). Applying the same pattern here appears to be a one-line change.

Details

src/commands/includes/storeDeduplicatedNextJob.lua, lines 12–15 at master:

if deduplicationOpts['keepLastIfActive'] and currentDeduplicatedJobId then
    local activeKey = prefix .. "active"
    local activeItems = rcall('LRANGE', activeKey, 0, -1)
    if checkItemInList(activeItems, currentDeduplicatedJobId) then

with src/commands/includes/checkItemInList.lua:

local function checkItemInList(list, item)
  for _, v in pairs(list) do
    if v == item then
      return 1
    end
  end
  return nil
end

storeDeduplicatedNextJob is reached from both deduplicateJob (replace path) and deduplicateJobWithoutReplace, i.e. from addStandardJob, addDelayedJob, addPrioritizedJob, and addParentJob.

To be clear about scope: the keepLastIfActive guard is the first condition in the function, so users who do not enable the option pay nothing. This is strictly an opt-in cost — but it is a cost that grows precisely where the feature is most useful.

Why this is more expensive than the comparison count suggests

Each invocation forces Redis to:

1. Materialize the entire active list as a multibulk reply,
2. Convert it into a Lua table in the shared interpreter,
3. Walk it with pairs() (non-array iteration over what is an array),
4. Discard the table, leaving GC pressure in the Lua VM.

Only step 3 is the nominal O(N) scan; steps 1, 2 and 4 dominate the constant factor.

Rough magnitude

A moderately sized deployment — say 20 workers at concurrency 50, so ~1,000 jobs active — with a 1,000/sec deduplicated add rate performs on the order of 10⁶ element allocations and comparisons per second on the single-threaded event loop, entirely inside atomic scripts that block all other clients for their duration. Larger active depths scale this linearly.

Proposed fix

Replace the list materialization with a server-side membership test.

Preferred: LPOS

 if deduplicationOpts['keepLastIfActive'] and currentDeduplicatedJobId then
     local activeKey = prefix .. "active"
-    local activeItems = rcall('LRANGE', activeKey, 0, -1)
-    if checkItemInList(activeItems, currentDeduplicatedJobId) then
+    if rcall('LPOS', activeKey, currentDeduplicatedJobId) then

This keeps identical semantics (membership of active) while eliminating the reply serialization, the Lua table allocation, and the interpreted scan. The remaining traversal happens inside Redis's own C loop.

This is the same substitution already made in getStateV2-8.lua relative to the legacy getState-8.lua:

-- getState-8.lua (legacy)
local active_items = rcall("LRANGE", KEYS[4], 0, -1)

-- getStateV2-8.lua
if rcall("LPOS", KEYS[4], ARGV[1]) then return "active" end

and in RedisQueueBackend#isJobInState, which calls client.lpos() directly. lpos() is implemented across all four client adapters (ioredis, node-redis, bun-redis-client, valkey-glide-client).

Handling the version gate

LPOS requires Redis ≥ 6.0.6, and RedisConnection.minimumVersion is still 5.0.0, so a fallback is needed. getState and isJobInState gate at the TypeScript call site via isRedisVersionLowerThan(..., '6.0.6', ...), which isn't directly available here because storeDeduplicatedNextJob is an include shared by several add scripts rather than a script selected per call.

Two options, in rough order of preference:

(a) Capability flag through the packed options. The client already knows queue.redisVersion. Passing a boolean alongside the existing de options and branching in Lua keeps checkItemInList as the ≤6.0.5 path with no script duplication.

(b) redis.pcall probe, self-contained.

local pos = redis.pcall('LPOS', activeKey, currentDeduplicatedJobId)
if type(pos) == 'table' and pos.err then
    -- Redis < 6.0.6: fall back to the existing scan
    pos = checkItemInList(rcall('LRANGE', activeKey, 0, -1), currentDeduplicatedJobId)
end
if pos then

No plumbing required, at the cost of one failed dispatch per call on legacy servers only.

Alternative considered and rejected: lock-key existence

EXISTS <prefix><jobId>:lock would be true O(1) and version-independent, since prepareJobForProcessing creates that key on transition to active. We do not propose it, because a stalled job remains in the active list after its lock has expired but before moveStalledJobsToWait reclaims it. That window would change the option's observable semantics. LPOS preserves them exactly.

Minor, independent of the above

If checkItemInList is retained as a fallback, pairs() over an array-shaped table can become a numeric for loop — free, and marginally faster.

Suggested verification

A benchmark that holds the deduplicated add rate constant while sweeping active depth (e.g. 10 / 100 / 1,000 / 10,000) should show the current implementation scaling linearly in p99 add() latency and in INSTANTANEOUS_OPS-adjacent blocking, and the LPOS version scaling far more gently. Measuring latency on an unrelated command against the same instance during the sweep is the clearest demonstration of the head-of-line blocking, since it isolates main-thread occupancy from client-side effects.

Background / prior art in this repo

keepLastIfActive shipped in v5.72.0 via #3902. Subsequent changes to this file have been correctness-focused rather than performance-focused:

I could not find any existing issue or PR raising the performance characteristics of this path. #3912 ("consider other non finished states in keepLastIfActive") was closed unmerged; note that widening the check to additional states would compound this cost unless the underlying membership test is addressed first.

Questions

1. Is there a reason the LPOS pattern used in getStateV2 was not applied here, e.g. a semantic difference I've missed?
2. If a capability flag is the preferred gating approach, is there an existing convention for threading Redis-version capabilities into shared includes that we should follow?
3. Is a fix along these lines something you'd accept as a community PR, or would you prefer to handle the version-gating plumbing in-house?