Skip to content

feat(files): upload and safely extract ZIP archives - #6782

Merged
waleedlatif1 merged 11 commits into
stagingfrom
feat/uploading-and-unzipping-zips
Aug 18, 2026
Merged

feat(files): upload and safely extract ZIP archives#6782
waleedlatif1 merged 11 commits into
stagingfrom
feat/uploading-and-unzipping-zips

Conversation

@j15z

@j15z j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator

Summary

The Files page previously rejected ZIP uploads and provided no way to unpack an archive. This PR adds an end-to-end ZIP extraction flow and fixes safety, cleanup, notification, and upload reliability issues uncovered while building it.

Features

  1. Upload ZIP files

    • Accept .zip files through the file picker and drag-and-drop.
    • Display ZIP files with the correct file-type label.
  2. Unzip files from the Files page

    • Open a confirmation modal when a ZIP file is clicked.
    • Extract through a new authenticated workspace-file endpoint: POST /api/workspaces/[id]/files/[fileId]/extract
    • Require a signed-in user with workspace write access.
    • Show success or error feedback and refresh the file browser afterward.
  3. Create a predictable extracted folder

    • Extract archive.zip into a sibling folder named archive.
    • Add a numeric suffix when the destination folder already exists.
    • Preserve nested folders from the archive.
    • Add numeric suffixes when extracted filenames conflict.
    • Skip common macOS and Windows metadata files.
    • Report the destination folder and extracted/skipped file counts.

Fixes and hardening

  1. Prevent duplicate extraction

    • Reuse the existing IdempotencyService and idempotency_key table as a six-minute distributed lease, preventing two servers from extracting the same workspace ZIP concurrently.
  2. Validate the complete output before writing

    • Count all files, nested folders, and the destination folder before extraction.
    • Reject output trees above the 5,000-item workspace limit.
    • Validate the full destination path before creating folders.
    • Reuse existing protections against malformed ZIPs, ZIP bombs, oversized files, symlinks, and path traversal.
  3. Make failed extraction rollback safe

    • Permanently remove files created by a failed extraction instead of only hiding them.
    • Remove a file only when its identity still matches the version created by that extraction.
    • Correct storage accounting in the same transaction as metadata removal.
    • Remove created folders deepest-first and only while they remain empty.
    • Preserve pre-existing folders and content added by collaborators.
  4. Make storage cleanup durable

    • Record physical storage deletion in the outbox transaction.
    • Attempt cleanup immediately and retry it through the background outbox processor if storage is unavailable.
    • Treat already-missing objects as successfully cleaned up.
  5. Batch workspace notifications

    • Suppress per-file notifications during extraction.
    • Send one file-tree notification after success or rollback.
  6. Improve upload and drag-and-drop reliability

    • Use multipart uploads for local files larger than 8 MB to stay within proxy limits.
    • Sort provider-returned multipart parts before completing an upload.
    • Clear the upload overlay when a file is dropped onto a folder.
    • Keep the currently opened folder selected after a spring-open drop.

Test coverage

Added or updated Vitest coverage for:

  • Extraction authentication, authorization, response data, and safe error mapping.
  • Destination naming, name conflicts, non-ZIP rejection, and concurrent extraction.
  • Output item limits and complete path validation.
  • File and folder rollback, storage accounting, and collaborator-content preservation.
  • Durable storage cleanup and retry behavior.
  • Batched notifications and client cache invalidation.
  • Large local multipart uploads and out-of-order provider parts.

@vercel

vercel Bot commented Aug 17, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
docs Skipped Skipped Aug 17, 2026 11:56pm

Request Review

@cursor

cursor Bot commented Aug 17, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Large changes to archive extraction, storage accounting, durable deletion, and workspace file mutations; failures mid-extraction could affect partial trees and billing unless rollback and outbox cleanup behave correctly.

Overview
Adds end-to-end ZIP support on the Files page: .zip uploads, clicking a ZIP opens an Unzip confirmation, and a new POST /api/workspaces/[id]/files/[fileId]/extract route (session + write) runs extractWorkspaceFile to unpack beside the archive into a sibling folder (numeric suffix on conflicts), with UI refresh and toasts.

Extraction pipeline is hardened in decompressArchiveBufferToWorkspaceFiles: pre-checks total materialized items (default 5k bulk limit), full destination path validation, optional prepareRootFolder, AbortSignal between entries, batched realtime via notifyWorkspaceChange: false, and rollback through purgeCreatedWorkspaceFile plus archiveWorkspaceFileFolderIfEmpty instead of soft-delete/recursive folder wipe. Durable storage cleanup is enqueued in the metadata transaction and handled by a new outbox handler wired into the outbox processor.

Concurrency and limits: distributed lease via IdempotencyService blocks duplicate unzips; 180s wall-clock budget maps timeouts to caller-safe errors. Upload sessions use multipart for local files above 8 MB and sort multipart parts before completion. Shared statusForArchiveError maps archive failures to 400/413 across tool routes and internal error policies.

Reviewed by Cursor Bugbot for commit 52b5c2f. Configure here.

@greptile-apps

greptile-apps Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds authenticated ZIP extraction, rollback-safe storage cleanup, multipart upload improvements, and Files-page extraction UI. One destination-path validation gap remains:

  • Intermediate folder names may receive conflict suffixes after preflight without validating the final full path.
  • Rollback metadata, accounting, and physical storage cleanup are coordinated through a durable outbox.
  • ZIP uploads, extraction feedback, cache invalidation, and drag-and-drop behavior are integrated into the Files page.

Confidence Score: 4/5

The PR should not merge until final deduplicated intermediate-folder paths are validated before insertion.

The reply states that destination paths are now fully validated before mutation, but intermediate folder conflicts still append suffixes after preflight without rebuilding the full path, allowing an over-limit folder path to be persisted.

Files Needing Attention: apps/sim/lib/uploads/archive.ts and apps/sim/lib/uploads/contexts/workspace/workspace-file-folder-manager.ts

Important Files Changed

Filename Overview
apps/sim/lib/uploads/archive.ts Implements bounded two-pass ZIP validation and extraction, but intermediate conflict suffixes can make final paths exceed limits after preflight.
apps/sim/lib/uploads/contexts/workspace/workspace-file-folder-manager.ts Adds resolved-name validation for prepared roots, while intermediate folder deduplication still inserts names without validating the resulting full path.
apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Coordinates rollback metadata deletion, storage accounting, and durable physical cleanup without an accepted blocking issue.
apps/sim/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox.ts Implements idempotent storage cleanup with immediate processing and generic outbox retry support.
apps/sim/lib/workspace-files/application/extract-workspace-file.ts Adds the authorized extraction use case, destination-root locking, lease handling, and batched notifications.
apps/sim/app/workspace/[workspaceId]/files/files.tsx Adds ZIP upload recognition, extraction confirmation, feedback, and drag-overlay fixes without an accepted blocking issue.
apps/sim/lib/uploads/upload-session/service.ts Improves local multipart selection and provider-part ordering without an accepted blocking issue.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[ZIP archive] --> B[Validate original entry paths]
  B --> C[Resolve destination root]
  C --> D[Materialize intermediate folders]
  D --> E{Folder-name conflict?}
  E -- No --> F[Create folder and upload files]
  E -- Yes --> G[Append numeric suffix]
  G --> H[Revalidate final full path]
  H --> F
  F --> I[Commit extraction]
  F -- Failure --> J[Rollback metadata and accounting]
  J --> K[Enqueue durable storage cleanup]
Loading

Comments Outside Diff (1)

  1. apps/sim/lib/uploads/archive.ts, line 471-478 (link)

    P1 Deduplicated paths bypass preflight

    When an archive's intermediate folder conflicts with an existing sibling and its original path is near the 4,096-byte limit, folder creation appends a numeric suffix after preflight without validating the resulting full path. This can persist an over-limit folder path, causing subsequent path reconstruction and folder operations to throw FolderHierarchyError.

    Knowledge Base Used: Upload sessions

Reviews (6): Last reviewed commit: "fix(files): only remap the deadline abor..." | Re-trigger Greptile

Comment thread apps/sim/lib/uploads/archive.ts Outdated
Comment thread apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Outdated
Comment thread apps/sim/lib/uploads/archive.ts
Comment thread apps/sim/lib/uploads/archive.ts
@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@cursor review

Comment thread apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Outdated

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit c779a15. Configure here.

@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f139e28. Configure here.

Comment thread apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts
@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile

@j15z

j15z commented Aug 17, 2026

Copy link
Copy Markdown
Collaborator Author

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 0d255a2. Configure here.

@j15z j15z changed the title feat(files): support zip extraction feat(files): upload and safely extract ZIP archives Aug 17, 2026
Comment thread apps/sim/lib/workspace-files/application/extract-workspace-file.ts
Comment thread apps/sim/app/api/workspaces/[id]/files/[fileId]/extract/route.ts
@j15z
j15z requested review from waleedlatif1 and a lite review from Copilot August 17, 2026 23:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds an end-to-end ZIP extraction workflow for workspace files, including a new authenticated API endpoint, a Files-page UX flow to confirm extraction, and extraction hardening (path/item-limit validation, rollback semantics, and batched notifications). It also improves upload reliability for larger local files by preferring multipart uploads and ensures multipart completion is stable even when providers return parts out of order.

Changes:

  • Add POST /api/workspaces/[id]/files/[fileId]/extract backed by a new authorized workspace-files use case and operation.
  • Harden archive extraction: validate destination paths and projected item counts before writing, use a distributed lease to prevent concurrent extracts, and improve rollback + durable storage cleanup via outbox.
  • Improve upload-session behavior for local storage (proxy-safe multipart threshold) and sort provider-returned multipart parts before completion; update UI to accept/label ZIPs and prompt unzip confirmation.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
scripts/check-api-validation-contracts.ts Updates the API-contract audit baseline to reflect the newly added route.
apps/sim/lib/workspace-files/application/operations.ts Registers a new files.extract_archive workspace operation with session-only write access.
apps/sim/lib/workspace-files/application/extract-workspace-file.ts Implements the authorized extraction use case with idempotency-lease protection and batched notifications.
apps/sim/lib/workspace-files/application/extract-workspace-file.test.ts Adds unit coverage for extraction behavior, authorization, leasing, and error cleanup.
apps/sim/lib/workspace-files/application/create-workspace-file.ts Adds notifyWorkspaceChange plumbing to suppress per-file notifications during bulk operations.
apps/sim/lib/workspace-files/api/internal-error-policies.ts Adds an internal error policy that maps ArchiveError to caller-safe status codes.
apps/sim/lib/workspace-files/api/internal-error-policies.test.ts Tests archive-error → HTTP status mapping behavior.
apps/sim/lib/uploads/upload-session/service.ts Adjusts transfer method selection for local uploads and returns sorted multipart parts for completion.
apps/sim/lib/uploads/upload-session/service.test.ts Tests the local multipart threshold and out-of-order provider parts handling.
apps/sim/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox.ts Introduces a durable outbox event/handler for deferred workspace object deletion.
apps/sim/lib/uploads/contexts/workspace/workspace-file-storage-cleanup-outbox.test.ts Tests payload validation, ENOENT handling, and retry-on-failure behavior for cleanup outbox.
apps/sim/lib/uploads/contexts/workspace/workspace-file-storage-accounting.test.ts Extends storage/accounting tests for purge rollback, durable cleanup, and notification suppression.
apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts Adds durable purge semantics for archive-created files and integrates cleanup outbox + notification suppression.
apps/sim/lib/uploads/contexts/workspace/workspace-file-folder-manager.ts Adds exact-name/dedup hooks for folder creation and an “archive if empty” rollback helper.
apps/sim/lib/uploads/contexts/workspace/workspace-file-folder-manager.test.ts Adds coverage for folder dedup behavior and empty-folder archival semantics.
apps/sim/lib/uploads/archive.ts Enhances archive extraction with preflight path/item validation, rollback via purge + empty-folder archival, and batched notifications.
apps/sim/lib/uploads/archive.test.ts Updates/extends extraction tests for notifications, rollback semantics, and item/path pre-validation.
apps/sim/lib/api/contracts/workspace-files.ts Adds a shared route contract + response schema for the new extract endpoint.
apps/sim/hooks/queries/workspace-file-folders.ts Adds a React Query mutation hook for extraction and invalidates file browser caches.
apps/sim/hooks/queries/workspace-file-folders.test.ts Tests cache invalidation and error handling for the new extraction mutation hook.
apps/sim/app/workspace/[workspaceId]/files/files.tsx Updates Files UI to accept/label ZIPs and adds an unzip confirmation modal flow.
apps/sim/app/api/workspaces/[id]/files/[fileId]/extract/route.ts Adds the new internal JSON route adapter for archive extraction.
apps/sim/app/api/workspaces/[id]/files/[fileId]/extract/route.test.ts Adds route-level tests for auth, input mapping, success response, and archive error mapping.
apps/sim/app/api/webhooks/outbox/process/route.ts Registers the new workspace-file storage cleanup outbox handler with the outbox processor.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/sim/app/workspace/[workspaceId]/files/files.tsx
…r's option surface

`maxMaterializedItems` was opt-in, so only the new unzip route bounded its output
tree — the copilot `materialize_file` and `POST /api/tools/file/manage` extract
paths had no cap on folder creation at all. An archive within MAX_ARCHIVE_ENTRIES
can still imply far more folders than files, so the cap now defaults to
MAX_WORKSPACE_FILE_BULK_AFFECTED_ITEMS and applies to all three callers.

`materializedRootFolderCount` was a hand-maintained number that had to agree with
what an opaque callback would create, and the callee could not check it; drift
surfaced only as an over-limit archive slipping past the cap. It is now derived
from whether `prepareRootFolder` ran, so the contract is just "the callback
creates exactly one folder".

Also single-sources the ArchiveError -> HTTP status map (it was copied into both
the internal error policy and the tools route), drops IdempotencyService config
that only `executeWithIdempotency` reads (the extraction lease uses
atomicallyClaim/release, so no result is ever stored), hoists the duplicated
predicates in purgeCreatedWorkspaceFile and archiveWorkspaceFileFolderIfEmpty so
a lock and its write cannot diverge, and names UPLOAD_SESSION_LOCAL_PUT_MAX_BYTES
rather than overloading the multipart part size as the local single-PUT ceiling.

Adds coverage for the two guards nothing exercised: the re-validation of the
segments `prepareRootFolder` actually returned, and the default cap applying with
no caller opt-in.

UI: the drop overlay used --surface-4 unconditionally, which renders grey over the
light-mode canvas; matches the canonical overlay's --white/dark:--surface-4 and
swaps arbitrary px type sizes for named tokens.
…lease

Cursor Bugbot flagged two related holes, both rooted in the write loop being
unbounded:

1. `maxDuration` is a Next.js route-segment config that serverless platforms
   enforce and self-hosted deployments do not. A slow extraction (up to 1000
   sequential uploads) could therefore outrun the six-minute lease, and
   `IdempotencyService` reclaims an expired in-progress claim — so a second unzip
   of the same archive could start beside the first.
2. Nothing rolls back a process killed mid-pass-2, so a timeout stranded the
   destination folder and every file written so far.

`decompressArchiveBufferToWorkspaceFiles` now takes an `AbortSignal` and checks it
between entries in both passes, and the extraction use case supplies a 180s
deadline. The abort unwinds through the existing all-or-nothing rollback, so the
work stops on our terms with the tree cleaned up, well inside both the route's
300s budget and the 360s lease. That closes (1) outright — the holder can no
longer outlive its lease on any platform — and converts (2) from a stranded
partial tree into a clean rollback for the slow case that actually triggers it. A
SIGKILL still cannot be caught; that needs a durable job and is out of scope here.

The overrun surfaces as a caller-fixable 413 naming the archive rather than an
opaque 500 from the raw DOMException.
@waleedlatif1

Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator

@cursor review

Comment thread apps/sim/lib/workspace-files/application/extract-workspace-file.ts
Comment thread apps/sim/lib/workspace-files/application/extract-workspace-file.ts
…ng rollback

Two follow-ups on the budget deadline, both reported by Cursor Bugbot:

`deadline.aborted` stays true for the rest of the request once the timer fires, so
it cannot decide whether *this* error was the abort. An `ArchiveError` or storage
failure thrown mid-entry after the timer fired was being relabelled as a timeout
and returned as a 413, hiding the real cause. The catch now matches the thrown
value against `deadline.reason` — `throwIfAborted()` throws exactly that object,
so the check is identity-exact and cannot capture an unrelated failure.

The message also claimed a rollback that has not necessarily happened: the budget
covers the archive download too, so it can fire before the first write, when there
is nothing to roll back. It now says the unzip was cancelled and claims nothing
about what was written. Including the download in the budget is deliberate — the
lease it has to fit inside starts earlier still — so the TSDoc says that rather
than "the extraction itself".
@waleedlatif1

Copy link
Copy Markdown
Collaborator

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator

@cursor review

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 52b5c2f. Configure here.

@waleedlatif1
waleedlatif1 merged commit 0c34e69 into staging Aug 18, 2026
30 checks passed
@waleedlatif1
waleedlatif1 deleted the feat/uploading-and-unzipping-zips branch August 18, 2026 00:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants