feat: logs search v2 by carderne · Pull Request #4615 · triggerdotdev/trigger.dev

carderne · 2026-08-14T08:43:50Z

Summary

Improves global Logs search with a smaller, bounded search representation that is indexed asynchronously. Source event inserts are no longer coupled to search indexing, and new logs can become searchable within seconds while a slower pass preserves complete coverage.

Existing reads remain on v1 by default.

Design

A best-effort preview pass processes complete five-second windows after a short safety delay. An authoritative pass processes one-minute windows after a longer delay and records each completed window as an append-only PostgreSQL checkpoint.

PostgreSQL keeps one low-churn control row for initialization and pause state. Redis coordinates projection work and stores the disposable preview watermark, so normal projection does not repeatedly overwrite a PostgreSQL watermark row.

Both passes use the existing LOGS_CLICKHOUSE_URL connection. They share a strengthened, bounded projection fingerprint so retries and preview/finalized overlap do not produce duplicate results, while distinct logs that share a span and timestamp remain separate.

The destination is partitioned by insertion date to keep each projection insert localized, while retention remains based on the event timestamp. Reads stay behind LOGS_SEARCH_TABLE_VERSION until v2 has sufficient forward coverage.

changeset-bot · 2026-08-14T08:43:56Z

⚠️ No Changeset found

Latest commit: 4ebd9ac

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-08-14T08:44:24Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 010f945c-8ee2-41e7-a121-f32d69678646

📥 Commits

Reviewing files that changed from the base of the PR and between d789149 and cc3213c.

📒 Files selected for processing (1)

internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql

📜 Recent review details

⏰ Context from checks skipped due to timeout. (23)

GitHub Check: report
GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 12)
GitHub Check: obsmap / 🧪 Unit Tests: Observability Map
GitHub Check: fk-cascade-guard / fk-cascade-guard
GitHub Check: typecheck / typecheck
GitHub Check: internal / 🧪 Unit Tests: Internal
GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
GitHub Check: runops-guard / runops-guard
GitHub Check: code-quality / code-quality
GitHub Check: audit
GitHub Check: audit
GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (1)

internal-packages/clickhouse/schema/[0-9][0-9][0-9]_*.sql

📄 CodeRabbit inference engine (internal-packages/clickhouse/CLAUDE.md)

internal-packages/clickhouse/schema/[0-9][0-9][0-9]_*.sql: Migration file numbering: name files as 0(N+1)_descriptive_name.sql where N is the largest existing migration number in schema/; rebase and renumber if main adds migrations before opening a PR
DDL in migrations must be idempotent: use ALTER TABLE ... ADD COLUMN IF NOT EXISTS, CREATE TABLE IF NOT EXISTS, DROP TABLE IF EXISTS, ADD INDEX IF NOT EXISTS, DROP INDEX IF EXISTS, and CREATE MATERIALIZED VIEW IF NOT EXISTS forms to allow out-of-order and retry-safe application

Files:

internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql

🔇 Additional comments (1)

internal-packages/clickhouse/schema/039_create_task_events_search_v2.sql (1)

3-3: LGTM!

Walkthrough

Added a ClickHouse V2 search table and projection pipeline with persistent state, leases, backfills, telemetry, scheduled processing, and admin controls. Updated log search normalization, minimum-length validation, bounded pagination, cursor handling, and period expansion. Centralized logs access checks across navigation and routes. Added ClickHouse command support, runtime configuration, database state storage, and integration tests.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 7.89% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description explains the design but omits the required issue reference, checklist, testing, changelog, and screenshots sections.	Add the template sections, link the closing issue, complete the checklist, document testing steps, provide a changelog entry, and include screenshots or state that none apply.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: adding logs search version 2.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch feat/log-search-v2

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-08-14T08:45:48Z

Observability map

As of cc3213c.

20/100 over 426 measured of 442 entry points (base 20, no change)

What this PR changed

route	base	head	now failing
/admin/api/v1/logs-search-projector	new	50	request-context

FIX FIRST

/api/v1/projects/:projectRef/envvars (sensitive) - auth-boundary, request-context
/auth/sso (sensitive) - auth-boundary, request-context
/_app/orgs/:organizationSlug/settings/team (sensitive) - error-classification, auth-scope, request-context

AUDIT 3 of 50 sensitive mutations record an actor. 47 without one.
CONTEXT 22 of 426 entry points name a tenant on a failure path. 325 appear only here, 39 of them sensitive, in the JSON rather than the fix list.

What the score is made of

CHECKS
  error-classification  179 applicable, 102 pass,   0 sole, global without it 12
  auth-boundary          62 applicable,  57 pass,   0 sole, global without it 17
  auth-scope             19 applicable,  17 pass,   0 sole, global without it 20
  request-context       426 applicable,  22 pass, 225 sole, global without it 64
  audit-trail            50 applicable,   3 pass,   0 sole, not in the score

The score and findings here are report-only and never gate the merge. Separately, a required test suite keeps this tool's symbol and route lists in sync with the code they name, and can fail a pull request that renames or removes a symbol they reference, or that adds the first route with a segment they anticipate. Each failure names the list to edit. The rules and their reasons: internal-packages/observability-map/README.md.

devin-ai-integration

Devin Review found 3 new potential issues.

devin-ai-integration

Devin Review found 4 new potential issues.

Project closed source windows with durable watermarks and leases. Keep v2 reads and backfill disabled by default until sufficient history exists.

Fetch bounded extra rows and remove duplicate projection identities in the application. Keep exact keyset pagination while background merges collapse physical copies.

Create the scheduled-projector schema directly in migration 038 and remove the intermediate migration.

Renumber search_v2 table migration to 039 and drop the inserted_at index DDL, which now ships as standalone migration 038.

pkg-pr-new · 2026-08-17T14:41:04Z

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@70280f5

trigger.dev

npm i https://pkg.pr.new/trigger.dev@70280f5

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@70280f5

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@70280f5

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@70280f5

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@70280f5

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@70280f5

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@70280f5

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@70280f5

commit: 70280f5

This comment was marked as resolved.

Sign in to view

carderne marked this pull request as ready for review August 14, 2026 09:27

This comment was marked as resolved.

Sign in to view

devin-ai-integration Bot reviewed Aug 14, 2026

View reviewed changes

Comment thread internal-packages/clickhouse/src/taskEventsSearchProjector.ts

Comment thread apps/webapp/app/presenters/v3/LogsListPresenter.server.ts

Comment thread internal-packages/clickhouse/schema/040_create_task_events_search_v2.sql

This comment was marked as resolved.

Sign in to view

carderne marked this pull request as draft August 14, 2026 13:23

devin-ai-integration Bot reviewed Aug 14, 2026

View reviewed changes

carderne added 10 commits August 17, 2026 14:08

feat(webapp,clickhouse): add bounded global log search

40282f0

feat(webapp,clickhouse): decouple log search indexing from event inserts

dcb0a03

Project closed source windows with durable watermarks and leases. Keep v2 reads and backfill disabled by default until sufficient history exists.

perf(webapp,clickhouse): preserve early exit when hiding log retries

fd578d4

Fetch bounded extra rows and remove duplicate projection identities in the application. Keep exact keyset pagination while background merges collapse physical copies.

refactor(clickhouse): consolidate log search v2 schema migration

e1922d0

Create the scheduled-projector schema directly in migration 038 and remove the intermediate migration.

fix coderabbit comments

22a6899

fix test

2b0d2f1

respond to devin comments

ae45385

fix(webapp): make log search normalization locale independent

45d729a

refactor(clickhouse): split inserted_at index into its own migration

233beba

Renumber search_v2 table migration to 039 and drop the inserted_at index DDL, which now ships as standalone migration 038.

feat(webapp,clickhouse,database): add fast and finalized log projection

70280f5

carderne force-pushed the feat/log-search-v2 branch from cc3213c to 70280f5 Compare August 17, 2026 14:11

refactor(webapp): reuse logs ClickHouse connection for projection

4ebd9ac

Uh oh!

Uh oh!

Conversation

carderne commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Design

Uh oh!

changeset-bot Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ No Changeset found

Uh oh!

coderabbitai Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reviews paused

Walkthrough

❌ Failed checks (2 warnings)

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Observability map

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pkg-pr-new Bot commented Aug 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

carderne commented Aug 14, 2026 •

edited

Loading

changeset-bot Bot commented Aug 14, 2026 •

edited

Loading

coderabbitai Bot commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading