Release the GIL in one-shot AEAD encrypt/decrypt by reaperhulk · Pull Request #15361 · pyca/cryptography

reaperhulk · 2026-07-30T13:46:49Z

Follow-up to #15359, carrying the AEAD portion that was split out during review: EvpCipherAead encrypt/decrypt (the AESGCM, ChaCha20Poly1305, AESGCMSIV, AESOCB3, AESSIV, and AESCCM one-shot APIs) now run the process_aad + process_data + tag region via run_with_gil_detached when plaintext + AAD is at least the 2KiB threshold.

Because the detached region must not touch Python objects, the AAD buffers are extracted and length-checked up front (extract_aad), keeping the CffiBufs alive on the calling frame while only borrowed slices cross into the detached closure. The BoringSSL/AWS-LC EvpAead path is unchanged.

Benchmarks (Linux x86-64, 4 cores, CPython 3.11, release build):

4 threads each AESGCM-encrypting 16MiB:
    9.3 ms -> 5.1 ms wall (1.8x; memory-bandwidth bound)
AESGCM encrypt 64B: 617 -> 630 ns (noise)

🤖 Generated with Claude Code

https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

Generated by Claude Code

Follow-up to the hash/HMAC/cipher GIL-release work: EvpCipherAead encrypt/decrypt (the AESGCM, ChaCha20Poly1305, AESGCMSIV, AESOCB3, AESSIV, and AESCCM one-shot APIs) now run the process_aad + process_data + tag region via run_with_gil_detached when plaintext + AAD is at least the 2KiB threshold. Because the detached region must not touch Python objects, the AAD buffers are now extracted and length-checked up front (extract_aad), keeping the CffiBufs alive on the calling frame while only borrowed slices cross into the detached closure. The BoringSSL/AWS-LC EvpAead path is unchanged. Benchmark (Linux x86-64, 4 cores, CPython 3.11, release build): 4 threads each AESGCM-encrypting 16MiB: 9.3 ms -> 5.1 ms wall (1.8x; memory-bandwidth bound) AESGCM encrypt 64B: 617 -> 630 ns (noise) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

Fold the AAD slice harvesting and total-length computation into extract_aad itself: the returned ExtractedAad carries the raw slices (with the full GIL lifetime) alongside the CffiBufs that keep the buffers alive, so call sites just pass extracted.slices() through and the single-AAD path allocates nothing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

alex · 2026-07-30T14:00:59Z

+    /// Like `as_bytes`, but the returned slice carries the full `'p`
+    /// lifetime instead of borrowing from `self`. The caller is
+    /// responsible for keeping this `CffiBuf` (which keeps the underlying
+    /// buffer alive and pinned) around for as long as the slice is used.
+    pub(crate) fn as_bytes_full(&self) -> &'a [u8] {
+        self.buf
+    }


Is there a reason not to extend the lifetime on teh regular as_bytes...

The &self-tied lifetime on as_bytes is doing borrow-check work: it makes something.extract::<CffiBuf>()?.as_bytes() a compile error when the slice outlives the temporary. With the 'p lifetime that pattern compiles and dangles — the CffiBuf drop releases the buffer view, so a bytearray can be resized/freed out from under the slice. Keeping the wide-lifetime version as a separate, documented method (only needed by callers that explicitly manage the owner's lifetime, like extract_aad) preserves that protection everywhere else. Happy to widen as_bytes itself instead if you'd rather have one method.

Generated by Claude Code

Tha makes it sound like this API is completely unsound and shouldn't be offered at all....

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

Split the AAD extraction result into an AadOwner (the Python buffer keepalives, which stay on the calling frame) and an ExtractedAad (raw slices + total length, no Python references) so process_aad takes the ExtractedAad directly and the whole thing can cross into the detached closure, which is now written inline at each call site. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

Per review: store Py rather than Bound inside CffiBuf (PyBuffer is already Send + Sync), which makes CffiBuf itself usable from a detached region as long as it's kept alive. ExtractedAad is then just the CffiBufs, process_aad takes it directly, and the separate owner/slices split and the wide-lifetime as_bytes_full accessor are gone. into_pyobj now takes py to rebind. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4

alex · 2026-07-30T14:20:28Z

+    }
+}
+
+fn extract_aad(aad: Option<Aad<'_>>) -> CryptographyResult<ExtractedAad<'_>> {


oh my god, I'm losing my mind: I told you to pre-return the length in a tuple. this is not that fucking complicated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C5bsKENedjVD9gcLApf1x4