fix(bigtable): sessionTableHandle self-heals across cache eviction - #20296
Conversation
#6) TableShim caches a *sessionTableHandle at Open time and holds it for the process lifetime. When the sessionTableCache TTL sweeper evicts that handle, subsequent ReadRow/Apply through the stale pointer hit a Close'd pool and return wrapped btransport.ErrPoolClosed. Because that sentinel is codes.Unknown (not codes.Unimplemented), InterceptUnimplemented does not fall back to classic. The user's *Table stays poisoned until process restart. Reproduced end-to-end against sandbox in session_table_evict_repro_test.go (pre-fix FAIL / post-fix PASS). Fix (Design C): keep TableShim caching the resolved handle at Open time so the RPC hot path pays only an atomic Load. Move all recovery logic INTO the handle: - sessionTableHandle gains an evicted atomic.Bool + captured openFn. - Close() sets evicted.Store(true) BEFORE api.Close so concurrent readers see the flag and route through the self-heal path. - ReadRow / MutateRow atomic-Load evicted; on true, call reopenAfterEviction (identity-checked removeEntry + getOrOpen) to get a live successor and dispatch on it. Happy path is unchanged: atomic Load + touch + api.ReadRow. Trade-offs vs alternative designs considered on this branch: - Per-RPC provider closure (03a8dcde89, reverted): ~30ns/RPC cache- map lookup on every call. This design: ~2ns/RPC atomic Load. Cost moves back to Open time (~30ns once per table opened). - Cache-side ErrPoolClosed catch: leaked transport-layer error sentinel into the cache/handle layer. Nothing changes in TableShim, open.go, or the test files that access shim.session. The fix is entirely inside session_table_cache.go — correct layer for eviction semantics. Change surface: - bigtable/session_table_cache.go +openFn, +evicted atomic.Bool, +reopenAfterEviction, wrap ReadRow/MutateRow, update Close ordering. - bigtable/open_test.go +TestSessionTableHandle_EvictedSelfHeals (pins openFn re-invocation + cache re-install + handle- identity stability). +TestSessionTableHandle_EvictedReopenFailsGracefully (pins terminal cache-close branch doesn't loop). - bigtable/docs/specs/SESSION_POOL_SPEC.md #3 TableShim bullet extended: handle self- heal semantics + link to session_table_cache.go invariants + test name.
Addresses the round-2 reviews on 82cd0a3 across three reviewers. Iterative dispatch() helper (thermo + igor blockers). ReadRow / MutateRow are shape-twins; both used to duplicate the self-heal conditional (drifting at comment level already, would drift at logic level as future methods land). Extract dispatch() on *sessionTableHandle that returns the live handle via a for-loop (not recursion — igor's catch: a hair-trigger TTL test could otherwise recurse if the fresh successor gets evicted mid-flight). ReadRow / MutateRow collapse to 3-line pass-throughs: dispatch → touch → api.op. Any future proxied method drops in as a 3-liner with zero branch structure to keep in sync. Bonus: touch fires on the live handle, not the doomed original. Rename reopenAfterEviction → resolveSuccessor (igor). "Reopen" reads as "revive h in place"; we're actually asking the cache for whoever's live under h.key now — could be a peer's winner, could be a fresh one we just minted. New name is honest about what happens. Kill dead fresh==h guard (sushanb + thermo). After removeEntry(h), getOrOpen can't return h (hit branch requires entries[key]==h, slow path mints a fresh *sessionTableHandle). The !ok half of the type assertion stays as a defensive check but the fresh==h half is provably unreachable. Close-ordering doc reworded (all three reviewers). "Nanoseconds" window claim was wrong — a reader preempted between Load(false) and api.dispatch can span arbitrary wall-clock under scheduler pressure. Rewrite to the honest three-step invariant: (1) evicted.Store(true) routes future readers through self-heal, (2) removeEntry drops from the map, (3) api.Close tears down. (1)→(3) is the load-bearing ordering; (2) tightens the window. Also documents which readers the flag actually helps (post-Load-true only, not in-flight pre-Load). WHY on openFn field (sushanb + igor). Documents the set-once *Client.sessionImpl invariant (if that ever becomes reconfigurable, self-heal will stale) AND the "invoked only via getOrOpen which nil-guards on close" invariant (direct openFn calls would lose the guard). Fix EvictedReopenFailsGracefully (all three). Was Logf-not-assert; docstring claimed to test a deadApi that wasn't wired. Rewritten: wrap ReadRow in context.WithTimeout(1s) for hang detection; assert cache.entries stays empty after the call (the closed-cache guard inside getOrOpen must have refused to install the freshly-opened api, releasing it — otherwise we'd leak). Three new tests (sushanb): - TestSessionTableHandle_SweeperEvictionSelfHeals — the actual production trigger. Prior tests used explicit handle.Close; this drives sweepOnce after TTL and asserts the held pointer still works. - TestSessionTableHandle_ConcurrentEvictAndReadRace — -race stress: evictor spins Close+reopen while reader spins ReadRow through the original held handle. - TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled — N=32 concurrent post-eviction RPCs. Pins the real invariant getOrOpen guarantees (exactly one installed successor + every loser api closed), not the aspirational single-flight-openFn one my first draft assumed. Making openFn genuinely single-flight would be a real perf win but is out of scope for this fix. Split test file (thermo blocker). open_test.go crossed 1006 lines after the new tests landed. All cache-focused tests + helpers + fakes (fakeClock, newTestSessionTableCache, openNoop, openClosing, closeCountingTable, all TestSessionTableCache_* and TestSessionTableHandle_*) move to a new session_table_cache_test.go (597 lines). open_test.go drops to 629. Both under threshold. SPEC trim (sushanb + igor). Trimmed the TableShim bullet from a paragraph to an invariant + Java-parity note ("Go-specific hazard — Java's session-table cache has no TTL sweeper, so Design C has no direct Java analog"). Non-blockers deferred: - Single-flight openFn in getOrOpen (would save N-1 pool creations in an eviction storm; separate perf follow-up). - Narrowing getOrOpen's return to *sessionTableHandle (thermo nit; touches the interface at the public boundary — separate PR).
There was a problem hiding this comment.
Code Review
This pull request implements a self-healing mechanism for sessionTableHandle in the Bigtable session pool, allowing evicted handles to transparently resolve and route subsequent RPCs (such as ReadRow and MutateRow) through a freshly-minted successor handle. Unit tests for the cache and handle have been moved to a dedicated test file, with new tests added to verify self-healing, concurrent races, and eviction storms. The review feedback recommends optimizing the self-healing path to avoid acquiring the global cache mutex repeatedly after eviction by caching the resolved successor handle directly inside the evicted handle using an atomic.Pointer and applying path compression in dispatch().
| evicted atomic.Bool | ||
| closeOnce sync.Once | ||
| closeErr error | ||
| } |
There was a problem hiding this comment.
To avoid a performance bottleneck where every single RPC after the first eviction has to acquire the global cache mutex twice (via removeEntry and getOrOpen), we can cache the resolved successor directly in the evicted handle using an atomic.Pointer[sessionTableHandle].
| evicted atomic.Bool | |
| closeOnce sync.Once | |
| closeErr error | |
| } | |
| evicted atomic.Bool | |
| sucessor atomic.Pointer[sessionTableHandle] | |
| closeOnce sync.Once | |
| closeErr error | |
| } |
There was a problem hiding this comment.
Good catch — adopted in 2552f50. Under TableShim's process-lifetime pointer-caching pattern, every RPC on a stale handle would pay 2× cache.mu forever without this. Field is now atomic.Pointer[sessionTableHandle] (fixed typo sucessor → successor) with a WHY comment on the field explaining why the Store races are safe by construction (getOrOpen's at-most-one-per-key invariant makes concurrent Stores idempotent; a divergent-next race just walks one extra hop on the next dispatch — no correctness bug, just lagging compression).
| func (h *sessionTableHandle) dispatch() *sessionTableHandle { | ||
| cur := h | ||
| for cur.evicted.Load() { | ||
| next := cur.resolveSuccessor() | ||
| if next == nil { | ||
| return cur // reopen refused; let cur.api surface the terminal error | ||
| } | ||
| cur = next | ||
| } | ||
| return cur | ||
| } |
There was a problem hiding this comment.
Update dispatch() to use the cached successor pointer. If a successor is already resolved, we can load it atomically without acquiring any locks. We also apply path compression when returning the active successor so that intermediate evicted handles can be garbage collected and subsequent lookups are O(1) atomic loads.
func (h *sessionTableHandle) dispatch() *sessionTableHandle {
cur := h
for cur.evicted.Load() {
if next := cur.sucessor.Load(); next != nil {
cur = next
continue
}
next := cur.resolveSuccessor()
if next == nil {
return cur // reopen refused; let cur.api surface the terminal error
}
cur.sucessor.Store(next)
cur = next
}
if cur != h {
h.sucessor.Store(cur) // path compression to bypass intermediate evicted handles
}
return cur
}There was a problem hiding this comment.
Adopted in 2552f50 with two small modifications:
- Single
Store(next)site inside the loop rather than two — the per-iteration Store handles the fresh-resolve case; the terminalh.successor.Store(cur)handles the "walked a chain" case. Same shape, one Store to reason about. - Preserved the existing
dispatch()doc block (eviction branch + loop rationale + cache-closed fallthrough) and added the path-compression paragraph on top of it.
Final shape:
func (h *sessionTableHandle) dispatch() *sessionTableHandle {
cur := h
for cur.evicted.Load() {
next := cur.successor.Load()
if next == nil {
next = cur.resolveSuccessor()
if next == nil {
return cur // reopen refused; let cur.api surface the terminal error
}
cur.successor.Store(next)
}
cur = next
}
if cur != h {
h.successor.Store(cur)
}
return cur
}Coverage: TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled already exercises N=32 concurrent post-eviction dispatch calls under -race, which is where the successor-cache races would surface. No new test needed.
Follow-up to igor round-2 on PR googleapis#20296. - Typo: openFn doc "staless" → "goes stale". - openFn field: add invariant note that resolveSuccessor threads the captured closure through unchanged, so all handles that ever share the key share the same openFn (guards future contributors from silently diverging successors by threading a per-call openFn override into getOrOpen). - SPEC bullet: clarify TableShim's `session` field is typed session.TableAPI (concrete value happens to be *sessionTableHandle); drop Java-parity phrasing per author preference. - Trim double blank line above the "─── sessionTableCache tests" divider. No behavior change.
Adopts gemini-code-assist's PR googleapis#20296 review suggestion (with modifications approved by igor round-2): Add `successor atomic.Pointer[sessionTableHandle]` to sessionTableHandle and use it in dispatch() to cache the resolved live successor. Without this, EVERY RPC on a stale handle pays 2× cache.mu (removeEntry + getOrOpen) forever, because TableShim's cached pointer never gets re-fetched. With it, only the FIRST post-eviction RPC pays that cost; subsequent ones are one extra atomic.Load. dispatch() also applies path compression on return so a chain of evicted handles collapses on the next observation — union-find with path compression, standard technique. Intermediate handles become unreachable from h and are eligible for GC. Store races are documented on the field and safe by construction: - Two concurrent dispatches on the same evicted handle race into getOrOpen, which is at-most-one-per-key (loser closes its local api and returns the winner's handle), so both racers see the same `next` — the Store is idempotent. - Divergent-`next` race (successor itself evicted between one dispatch's resolveSuccessor and another's Store) writes a stale pointer; the next dispatch's outer for loop walks one extra hop through it and updates. No correctness bug, just a lagging compression. Preserves the existing dispatch() doc block (eviction branch + loop rationale + cache-closed fallthrough) and adds the path-compression paragraph. Modifications from gemini's original suggestion: - Typo: sucessor → successor. - Single Store site inside the loop (gemini had two writes: per-iteration and at the return). One is enough — the per-iteration Store handles the fresh-resolve case; the terminal h.successor.Store(cur) covers the "we walked a chain" case. - Preserved the fallthrough-to-cur-on-nil semantic + comment. No new tests — TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled already exercises N=32 concurrent post-eviction dispatch calls under -race, which is where the successor-cache races would show up.
🤖 I have created a release *beep* *boop* --- ## [1.52.0](bigtable/v1.51.0...bigtable/v1.52.0) (2026-08-03) ### Features * **bigtable:** Add AFE picker (Simple / LeastInFlight / LeastLatency) ([#20204](#20204)) ([bcbf714](bcbf714)) * **bigtable:** Add ClientConfig.DisableSession to opt out of session backend ([#20297](#20297)) ([7ee5e44](7ee5e44)) * **bigtable:** Add getClientConfigDirectAccessChecker for session pools ([#20209](#20209)) ([3b8d30a](3b8d30a)) * **bigtable:** Add NoOpChannelPrimer for session channel pools ([#20208](#20208)) ([d055a8a](d055a8a)) * **bigtable:** Add per-AFE sessionList for the two-tier session pool ([#20224](#20224)) ([dbf0c3f](dbf0c3f)) * **bigtable:** Add protoRowToRow conversion helper for TableShim ([#20257](#20257)) ([1297143](1297143)) * **bigtable:** Add Session debug surface (observability fields + methods) ([#20211](#20211)) ([d8d3e16](d8d3e16)) * **bigtable:** Add Session lifecycle (Start, Close, ForceClose, readLoop, heartBeatLoop) ([#20215](#20215)) ([b9e53c6](b9e53c6)) * **bigtable:** Add Session struct + state machine ([#20117](#20117)) ([09acbb3](09acbb3)) * **bigtable:** Add session.Config.EnableDebug to gate sessionz debug state ([#20247](#20247)) ([ce74c31](ce74c31)) * **bigtable:** Add SessionClient + SessionTable + lazyPool ([#20228](#20228)) ([ab2c96c](ab2c96c)) * **bigtable:** Add SessionPoolImpl (two-tier pool + scaling + debug) ([#20225](#20225)) ([683eda8](683eda8)) * **bigtable:** Rename session pool display to <resource-id>-<PERM> ([#20248](#20248)) ([35e146e](35e146e)) * **bigtable:** Route Client.Open()-returned *Table through the Diverter ([#20273](#20273)) ([2b81c7d](2b81c7d)) * **bigtable:** State-based classification for abnormal session close ([#20243](#20243)) ([f2905b7](f2905b7)) * **bigtable:** TableShim fallback to classic on session UNIMPLEMENTED ([#20269](#20269)) ([36540af](36540af)) * **bigtable:** TTL-on-idle cache for per-resource session.TableAPI ([#20263](#20263)) ([00b2a49](00b2a49)) * **bigtable:** Wire Diverter on Client and route Open* via TableShim ([#20256](#20256)) ([b32fbd7](b32fbd7)) ### Bug Fixes * **bigtable:** AFE picker latency signal — subtract poolWait and compute TransportLatency = wire − backend at source ([#20281](#20281)) ([bb8c4d5](bb8c4d5)) * **bigtable:** Guard NewStream OnFinish against grpc-go double-fire ([#20295](#20295)) ([b51da29](b51da29)) * **bigtable:** Real per-resource pool teardown on sessionTable.Close + cache close-race gate ([#20264](#20264)) ([599aea9](599aea9)) * **bigtable:** Session.durations / session.uptime — set explicit histogram bucket boundaries ([#20276](#20276)) ([97eee22](97eee22)) * **bigtable:** SessionTableHandle self-heals across cache eviction ([#20296](#20296)) ([0dd98cd](0dd98cd)) * **bigtable:** Translate ctx errors to gRPC status on session vRPC ([#20299](#20299)) ([0f3b2a5](0f3b2a5)) * **bigtable:** Treat PingAndWarm NotFound as a successful prime ([#20219](#20219)) ([a1557ad](a1557ad)) ### Performance Improvements * **bigtable:** Delete periodic Tick loop; sizing is event-driven ([#20285](#20285)) ([2c096bd](2c096bd)) * **bigtable:** Drop pick_lost_race debug tag from CheckoutSession hot path ([#20280](#20280)) ([bd0e400](bd0e400)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Summary
`TableShim` caches a `*sessionTableHandle` at Open time and holds it for the process lifetime. When `sessionTableCache`'s TTL sweeper evicts that handle (default 1h idle), subsequent `ReadRow` / `Apply` through the stale pointer hit a `Close`d pool and return wrapped `btransport.ErrPoolClosed`. That sentinel is `codes.Unknown` (not `codes.Unimplemented`), so `InterceptUnimplemented` does NOT fall back to classic. The user's `*Table` stays poisoned until process restart.
Fix (Design C)
Extract an iterative `dispatch()` helper on `*sessionTableHandle`:
Design tradeoffs considered
Test plan
Bug reference: internal audit finding #6 ("TTL evict → ErrPoolClosed").