Module Spoc_core.Transfer

Packages a backend buffer of a scalar vector as a DEVICE_BUFFER. A buffer that is_zero_copy (Native, Interpreter: the device uses the host array itself) needs no transfer; any other one, OpenCL host-pointer buffers included, goes through its backend's transfers, which synchronise (map/unmap) instead of copying when the buffer aliases the host pointer.

Allocate a device buffer of its own for a scalar vector.

The backend's alloc_zero_copy: a buffer that is the host array itself (Native, Interpreter). None on every backend with memory of its own.

The backend's alloc_host_ptr: a buffer created over ba's memory. Only Host_ptr_buffer calls this; None when dev's framework is not registered.

Allocate a device buffer for a custom vector

Record buf as vec's buffer on dev (no buffer there yet): the vector's finaliser frees it, and Gpu_memory counts its bytes.

Ensure vector has a device buffer, allocating if needed. On Native and Interpreter the buffer is the host array itself (alloc_zero_copy); every other device, OpenCL CPU devices included, gets memory of its own. Sharing the host array on an OpenCL CPU device is the explicit Host_ptr_buffer opt-in, which installs its buffer before this function would allocate.

Everything that reads device memory back into host storage, and NOTHING else. The module exists for its signature: copy_device_to_host is defined inside and deliberately absent from it, so the only code that can call the packed-buffer read directly is Read_back.read_back_to_host below.

Transfer vector data to a device

Transfer vector data from device to CPU.

• parameter force

  If true, always transfer even if location is Both (useful after kernel writes)

Ensure vector is fully synchronized

A value rooted only for its liveness, whatever its type; never inspected. One 2-word block per root: an [@@unboxed] existential is rejected while float arrays are flat.

Implicit paths that run on the default stream (host read-back, auto-sync, default-stream launches) cannot rely on the legacy-stream hidden synchronisation for work on a non-blocking stream. settle_vector vec waits, on the host, for every non-blocking stream still referencing vec. Streams from create_stream without ~non_blocking are not touched. Explicit-stream operations never call this: ordering between explicit streams is the caller's, through events.

Refuse to free device memory that pending stream-ordered work may still touch (FR-021): freeing it would put the DMA engine or a kernel on freed memory.

true iff vec has no host storage (create_device_only on a non-CPU device).

Release the device memory vec holds on dev, draining it to host storage first.

FOUR steps, and not one of them may sit inside get_buffer's Some arm. Under the SoA ABI the leaves are the only device memory the vector has and the packed buffer is never allocated, so on a transparent vector get_buffer returns None ALWAYS — an early return there does not skip a corner case, it skips the whole function.

The previous version of this comment already said "under the SoA ABI they are the only device memory this vector has" while the bookkeeping below it still lived in the Some buf arm and assumed a packed buffer existed. That mismatch was the bug, not a wording slip: the location reset is the only code here that assigns CPU, so a transparent SoA vector came out of this function with its leaves freed and location still naming dev. Two measured consequences, both on a vector whose data was intact in host storage:

• to_cpu ~force:true raised Failure "to_cpu: no device buffer to transfer from" — Both dev plus force means "read the device back", and there is nothing there to read;
• to_device on the same device took the "skip (Both)" short-circuit and allocated nothing, reinstating the exact short-circuit the Stale_CPU work earlier in this branch exists to eliminate.

free_all_buffers was never exposed to this because it assigns CPU unconditionally at the end. This function now does the same, scoped to dev.

Free all device buffers for a vector

Synchronize all pending operations on a device and release the ledger entries of its streams; the same as Device.synchronize.

Raise Feature_not_supported for backend: the capability refusal of this section, as opposed to Invalid_argument for caller errors.

Stream handle - packages backend stream with its operations.

Create a new stream on a device. ~non_blocking:true (default false) creates a stream that does not synchronise implicitly with the default stream; order work across streams with record_event and stream_wait_event.

Get default stream for a device. All handles for one device share one ledger state. Work given ~stream:(default_stream dev) is explicit-stream work: it is not settled against non-blocking streams (only the no-~stream paths are); order it with events.

The device a stream belongs to and its backend-tagged handle — what Execute needs to launch on it.

Root keep (and record vec_ids as in use) until the work just enqueued on s has completed — what Execute does for the vectors of a launch it enqueued on s (the backend's own kernel-argument retention roots the argument cells, not the vectors whose buffers the kernel touches).

Recover the backend's own stream from a core stream; refused when it belongs to another backend or another device than dev.

An event on one device. ev_recorded remembers which stream and ledger position the latest record_event covered, so synchronising or querying the event can release exactly the keep-alive entries it proves complete. Recording again replaces it (the latest record is what waits, queries and releases refer to, as in CUDA/HIP).

Create an event on dev. The device is selected first: HIP and CUDA create events on the CURRENT device, which a GC finaliser freeing a buffer may have changed.

Record ev on s: it completes once all work enqueued on s so far has completed. Refused across devices.

Make every operation enqueued on s after this call wait for the work ev recorded. Refused across devices. On Native/Interpreter streams are synchronous, so the wait is already satisfied.

true once the recorded work has completed (or the event was never recorded). Never blocks. A true answer releases what the event covers.

Block until the recorded work has completed, then release what it covers.

Destroy the event. Keep-alive entries it would have released stay until their stream is synchronised.

Allocate a vector of length elements that lives only on dev.

On a GPU device no host storage is allocated at all: host element access (Vector.get/set/fill/iter/to_bigarray/...), whole-vector read-back (to_cpu, auto-sync) and host-side copies all raise Invalid_argument naming the vector as device-only. Data moves in and out only through the ranged copies below (copy_to_device, copy_bytes_to_device, copy_ptr_to_device, copy_to_host, copy_bytes_to_host) or kernels. Freeing its buffer (free_buffer, free_all_buffers) discards the contents, and the vector is unusable afterwards.

On Metal the buffer is a shared-storage MTLBuffer (Apple silicon unified memory): still no host mirror, and the same refusals apply.

On a CPU device (Native, Interpreter) device memory IS host memory — kernels there read the vector's host storage — so the vector gets ordinary host storage, aliased by its device buffer, and host element access works. Portable code must not rely on that.

Refused (Invalid_argument): length outside 1 .. Int32.max_int (kernels index with int32; split larger caches over several vectors) and custom (record) kinds. Refused (Feature_not_supported): backends without ranged copies (OpenCL, WebGPU), where nothing but a kernel could ever fill the vector.

vec's buffer on d is its host storage (Memory.Host_storage), so host and device copies cannot diverge.

The device a ranged copy on vec targets, and its buffer there, after checking the preconditions. A ranged copy needs the device copy to be CURRENT: a partial write into a stale device copy would leave neither copy authoritative, and a read from one would return stale data.

Current means: vec is device-only (device-current by definition), or its location is GPU d, Both d or Stale_CPU d, or — when its buffer on d is the host storage itself (Memory.Host_storage: Native, Interpreter) — any location naming d.

Data address of ba. The address does not root ba: the caller keeps ba reachable until every use of the address has completed (for an enqueued copy, until the stream ledger releases it).

Copy all of src into dst starting at element dst_offset.

Without ~stream the copy has completed when the call returns. With it, the copy is enqueued on the stream and may still be running; SPOC keeps src and dst alive until the stream (or an event recorded after the copy) is synchronised. Writing into src before then races with the DMA: ordering that is the caller's job.

dst must be device-current (see range_target); a vector in Both becomes Stale_CPU, so auto-sync later reads the device copy. Raises Invalid_argument when the range does not fit, and Feature_not_supported on backends without ranged copies. Slice the source with Bigarray.Array1.sub.

Pageable sources: HIP and CUDA stage a stream copy from pageable memory through a driver bounce buffer. It is correct, but the host call may block and the copy may not overlap compute; source from alloc_pinned or register_host memory when overlap matters. Not refused, because the result is right and the rate can be equal (28.1 vs 28.6 GB/s measured on an RX 7900 XTX, CHANGES.md 2026-10).

Byte-level copy_to_device: copy every byte of src (any element kind) into dst at byte offset dst_byte_offset. For blocks that mix element types (e.g. int32 quant nibbles followed by f32 scales) read into a char buffer. Same stream, keep-alive and location contract.

Raw-address copy_bytes_to_device. src_ptr is NOT rooted: the caller keeps byte_size bytes at src_ptr alive and unmodified until the copy has completed — the call's return without ~stream, the stream or a later event's completion with it.

Copy Bigarray.Array1.dim dst elements of src, starting at element src_offset, into dst — the explicit read-back for device-only vectors. src must be device-current. With ~stream the copy may still be running on return and SPOC roots dst until it has completed; read dst only after synchronising (see copy_bytes_to_host for the destination's state meanwhile). src's location is unchanged.

Byte-level copy_to_host: copy Bigarray.Array1.size_in_bytes dst bytes of src (any element kind), starting at byte src_byte_offset, into dst (any element kind) — e.g. a range of an int32 vector into a char slot. Slice the destination with Bigarray.Array1.sub. The mirror of copy_bytes_to_device; same location rules as copy_to_host: src must be device-current and its location is unchanged. Raises Invalid_argument when the byte range does not fit and in the other cases copy_to_host refuses (host-authoritative or freed source, destroyed stream, stream of another device), and Feature_not_supported on backends without ranged copies.

Without ~stream the copy has completed when the call returns. With it, the copy is enqueued and may still be running: until the stream has been synchronised, an event recorded after the copy has been synchronised or queried complete, or flush, dst's contents are undefined — reading it may see old, new or mixed bytes, and writing it races with the DMA, which may overwrite it. SPOC keeps dst and src alive meanwhile and records dst's bytes as in flight, so free_pinned and unregister_host refuse a buffer they overlap. HIP and CUDA may stage a stream copy into pageable memory through a driver bounce buffer and block the host call; copy into alloc_pinned or register_host memory when overlap matters.

true iff ba is exactly a live alloc_pinned allocation or a live register_host registration.

Allocate n elements of page-locked host memory for DMA from/to dev, as a Bigarray (read into it with pread/O_DIRECT, then copy it with ~stream at full link rate). HIP: hipHostMalloc (portable); CUDA: cuMemHostAlloc (portable); Metal: page-aligned host memory (unified memory has nothing to page-lock); Vulkan: driver-allocated mapped memory, owned by the device and unmapped when it is destroyed (free the buffer first; later access is undefined behaviour; allocation can fail with an out-of-memory error, not a Feature_not_supported refusal); Native/Interpreter: ordinary host memory. Refused with Feature_not_supported elsewhere.

The memory is released ONLY by free_pinned: there is no finaliser (sub-arrays could outlive the parent). Using the Bigarray, or any sub-array of it, after free_pinned is undefined behaviour, as after munmap. Bigarray.float16 is refused: ctypes cannot build that kind over foreign memory; use a char or int16_unsigned buffer and copy_bytes_to_device.

Release a buffer returned by alloc_pinned. Refused (Invalid_argument) for anything else — a sub-array, a registered buffer, a buffer already freed — and while a stream-ordered copy may still read or write it.

Page-lock an existing Bigarray for DMA from/to dev. SPOC keeps ba alive until unregister_host.

Measured pitfalls on HIP (RX 7900 XTX, ROCm 7.2, Linux 7.1.2, 2026-10-10, under memory pressure from other workloads; specs/transfer-pitfalls.md):

• Copies right after registering can be slow. bench_hip_register_host, 8 GiB registrations, synchronous 5.5 MiB copies: 7 of 50 first passes ran at 0.95-6.17 GB/s (about 26 GB/s otherwise; one repeat pass at 1.66), whatever the preparation: memory written before registering, never touched, written after registering, or faulted in with madvise(MADV_POPULATE_WRITE) first. spoc_llm saw 0.45 GB/s for memory registered before it was touched, and attributed it to that; repeated runs here do not support a dependence on touching, only an intermittent, state-dependent effect. hipHostRegister faults every page in itself (VmRSS grew by the full 8 GiB for untouched memory).
• A copy whose host range spans two registrations fails with hipErrorInvalidValue, synchronous or on a stream. Registering a large buffer in regions means slicing every copy inside one region.
• spoc_llm observed copies from ONE 8 GiB registration fall to 2.3 GB/s after CPU writes into it, against 26.3 GB/s with 64 MiB registrations. That did not reproduce later the same day (26.3 GB/s after an O_DIRECT fill of a single 8 GiB registration): treat region size as something to measure on the target machine, not a rule.
• ROCm does not count registered pages as pinned (VmPin: 0 kB in /proc/self/status with 8 GiB registered), so the kernel may still migrate or compact them. Whether that is what makes copies slow under memory pressure is NOT established. alloc_pinned memory is allocated by the driver instead.

Undo register_host; refused while stream-ordered copies may use it.

Largest source stage_to_device accepts, in bytes (64 KiB).

Write all of src (at most max_staged_bytes) into dst at element dst_offset, WITHOUT waiting for the GPU: the cheap way to update a small vector that kernels read, e.g. per-layer routing indices.

Vector.set on such a vector is expensive. After a launch the vector is Stale_CPU, so the first set downloads it (waiting for every queued kernel), and the next launch uploads it again with a synchronous copy that waits for the queue once more. Measured (bench_hip_small_writes, RX 7900 XTX, 8 int32 + 8 float32 rewritten, then 300 us of host work, then 4 launches, three runs, 2026-10-10, raw output in docs/measurements/transfer-pitfalls-2026-10-10/bench_hip_small_writes.txt): 705.2-715.1 us per iteration with Vector.set (341.3-346.4 us of it in the writes), 330.1-338.9 us with stage_to_device (9.1-11.5 us in the writes), 322.0-322.5 us with no rewrite at all.

src is copied into SPOC-owned pinned staging memory before the call returns, so the caller may overwrite it immediately. The copy to dst is enqueued on stream (default: dev's default stream), after the work already there and before what follows; streams from create_stream are blocking by default, so they order with it too (a ~non_blocking:true stream does not: order it with events; without ~stream the vector is settled first). SPOC keeps dst alive until the copy has completed, and refuses to free it until the stream or device is synchronised (or the ring has reused the range).

Staging is 1 MiB of pinned memory per device, allocated on first use and kept for the process lifetime; when every byte of it is still in flight, the call waits for the oldest staged copy. Same preconditions as copy_to_device: dst device-current, Both becomes Stale_CPU; out of range, or a source above max_staged_bytes, raises Invalid_argument before any device call.

Known issue: on a Ryzen 7950X iGPU (HIP, ROCm 7.2) about 1 run in 7 of 300 staged writes interleaved with launches lost one launch (its kernel never wrote). A caller-managed pinned ring using only copy_to_device and events shows it too, so the cause lies below this function. It depends on GC timing: running pending finalisers first (Gc.full_major ()) removed it in every run (0 of 20 against 4 of 20), so finaliser work during the loop is the lead; the mechanism is not established. The RX 7900 XTX never showed it (specs/transfer-pitfalls.md C-9, issue #427, docs/measurements/transfer-pitfalls-2026-10-10/stress_hip_staged_launches.txt, sarek-hip/test/stress_hip_staged_launches.ml).

HIP, CUDA: as above. Native, Interpreter, Metal, Vulkan (no asynchronous copies): a synchronous copy_to_device, complete on return (on Metal it first waits for kernels that use dst; on Vulkan for every pending launch of the device). OpenCL, WebGPU: Feature_not_supported.