Two weeks ago we posted "PDL in Rust - A Native
Reimplementation of the Perl Data Language". At the time the
score was 45 tests, all green. That was enough to say "it compiles,
it runs, here is the arithmetic surface." It was not enough to say
"you can use this."

We are now on the second number. The current PDL implementation in pperl covers roughly 3,000 assertions end-to-end: about 1,400 on the Perl-facing connector side and about 1,600 on the engine side. As of this writing roughly 98% of the connector assertions match upstream PDL 2.103 exactly, and most of the remaining couple of dozen we already know why they fail. By the time you read this the numbers will have drifted a little in our favour - give or take - but the shape is the point, not the decimal.

Everything described below is shipping, today, at https://perl.petamem.com.

What Part One Was

Part One was a flag-planting exercise. Someone on Reddit asked whether pperl would support PDL; we said yes; we then had to actually do it. Forty-five tests covering construction, arithmetic, reductions, a handful of transcendentals, and operator overloading. The point was to prove the mechanism - that a pure-Rust engine behind a pperl native module could stand up to Perl-side code that expects `use PDL; my $x = pdl([1,2,3]);` to just work. It did. But forty-five tests is not a usable PDL.

This post is about what happened when we turned the heat up.

What's in the Box

The Perl-facing module hierarchy now mirrors upstream PDL's own, file-for-file where it matters:

PDL              - top-level boot, exports pdl/zeroes/ones/...
PDL::Core        - constructors, accessors, DESTROY, magic
PDL::Ops         - arithmetic, comparison, bitwise, unary
PDL::Ufunc       - reductions, sorting, statistics
PDL::Math        - trig, hyperbolic, Bessel, erf, special functions
PDL::Primitive   - matmult, which, where, clip, append, convolve
PDL::Slices      - slice, xchg, mv, reshape, clump, dummy, range
PDL::Basic       - sequence, xvals, yvals, zvals, rvals, linvals
PDL::MatrixOps   - det, inv, LU, eigens, trace, norm
PDL::Bad         - bad-value flags, setbadat, copybad, locf
PDL::FFT         - Cooley-Tukey radix-2 FFT/IFFT
PDL::NiceSlice   - source filter: $x(:,0) syntax
PDL::Lite        - lightweight import variant
PDL::LiteF       - lightweight import with fewer functions

Fifteen data types. Broadcasting. Bad values. Dataflow. Affine slicing. Storable round-trip. FFT. 2D image convolutions. Matrix decompositions. Source-filter support for PDL::NiceSlice's $x(:,0) syntax. Operator overloading for ~40 operators. It is PDL, not a demo subset. Whether it walks quite like the grown-up yet is for the user to judge.

The Connector, in One Example

Part One described the split at a high level - a standalone Rust crate for the engine, a pperl native module for the bridge. This time let us walk one operation end to end, because the connector is where the Perl audience will want the detail.

Consider $A x $B, the matrix-multiply overload:

use PDL;
my $A = pdl([[1,2],[3,4]]);
my $B = pdl([[5,6],[7,8]]);
my $C = $A x $B;

Here is what the x overload looks like on our side. The handler is a plain Rust function registered into the PDL stash at boot. No use overload block, no BEGIN-time aliasing, no XSUB wrapper file:

// At boot, overload entries go straight into the PDL stash.
let overloads: &[(&[u8], unsafe extern "C" fn(*mut CV))] = &[
    (b"(+\0",    xs_plus_overload),
    (b"(*\0",    xs_mult_overload),
    (b"(x\0",    xs_matmult_overload),   // matrix multiply
    // ~40 more operators …
];
for &(name, func) in overloads {
    newXS(/* PDL::(X */, Some(func), file);
}

The handler itself extracts the PDLs from their SVs, validates dimensions, calls into the engine, wraps the result:

unsafe extern "C" fn xs_matmult_overload(_cv: *mut CV) {
    let mark  = xs_pop_mark();
    let items = xs_items(mark);
    let (a_sv, b_sv, _swap) = binop_args(mark, items);
    xs_sp_set(mark);
    let mut a_temp = false;
    let mut b_temp = false;
    let a = sv_to_pdl_or_scalar(a_sv, &mut a_temp);
    let b = sv_to_pdl_or_scalar(b_sv, &mut b_temp);
    let a_dims = core_vtable::pdl_dims(a);
    let b_dims = core_vtable::pdl_dims(b);
    if a_dims[0] != b_dims[1] {
        Perl_croak(b"PDL: matmult: dimension mismatch\0".as_ptr() as *const c_char);
    }
    let c = core_vtable::pdl_alloc_output();
    // --- THE ENTIRE DISPATCH ---
    // One call into Rust-PDL. No XS wrapping, no interpreter
    // round-trip, no hidden allocations.
    let err = rust_pdl::primitive::pp_matmult
                 ::pdl_run_matmult(a, b, c);
    if a_temp { core_vtable::pdl_destroy(a); }
    if b_temp { core_vtable::pdl_destroy(b); }
    xs_push(sv_2mortal(pdl_to_sv(c)));
}

Three things about this example are worth drawing out for a Perl audience.

The overload dispatch is a function-pointer table. $A x $B resolves as: stash lookup → function-pointer call → Rust. Upstream PDL's equivalent path is: use overload table lookup → dispatch to a Perl sub → XSUB boundary → PP-generated C → inner kernel. pperl removes the two middle layers entirely, because the handler is the kernel-caller, compiled into the interpreter binary.

sv_to_pdl is a magic walk, not a method call. PDL SVs in pperl store the engine's *mut c_pdl as PERL_MAGIC_ext on the inner SV - the same mechanism Perl itself uses for tied variables, dualvars, and a dozen other features. Extracting the pointer is a flag check plus a linked-list walk. No method dispatch, no can(), no string comparisons:

let flags = SvFLAGS(inner);
if flags & (SVs_GMG | SVs_SMG | SVs_RMG) != 0 {
    let mut mg = (*sv_any).xmg_u.xmg_magic;
    while !mg.is_null() {
        if (*mg).mg_virtual == &PDL_MAGIC_VTBL {
            return (*mg).mg_ptr as *mut c_pdl;  // ← the c_pdl pointer, raw
        }
        mg = (*mg).mg_moremagic;
    }
}

So $A->at(0) in pperl is: one stash method lookup, one magic walk, one pointer-arithmetic offset inside a Rust function. We did not invent a new mechanism for binding Rust objects to Perl variables - we reused Perl's own magic infrastructure, because that is what it is for.

Type promotion: upstream's rule, reimplemented from scratch. Matrix multiply with mixed types - long x float - must promote to double for numerical stability. Upstream's rule is "an NV scalar forces minimum promotion to double; an NV PDL keeps its wider type if already ≥ double." We ported the rule verbatim; the implementation is thirty lines of Rust match arms. No C preprocessor switch tables. No .xs file. No PDL::PP code-generation templates. When we discovered that pperl was keeping float instead of promoting to double for float + 0.2 - the fix was two match arms, one rebuild, twenty-five tests recovered. In the upstream world the equivalent change would touch PDL::PP's templates and trigger a rebuild of forty op files.

pd2multi - We build a Compiler

Upstream PDL has PDL::PP: a macro/template processor that weaves .pd definitions with C fragments and hands the output to cc. It has served PDL for thirty years. It works. But the output is always C, and it is a text-substitution engine, not a compiler.

We could not extend it. We needed something that could emit Rust today, and potentially other targets later - a C backend to validate against upstream's own output, a GPU backend for OpenCL, a Perl backend to let pperl's JIT see through the operation into the surrounding loop. So we built a proper parser → AST → emitter pipeline. pd2multi ingests the same input language, builds an explicit intermediate representation, and delegates output to pluggable backend emitters.

This is internal infrastructure. Nothing user-facing ships from it yet; it is the scaffolding under the engine. It is roughly two orders of magnitude more work than the template-substitution approach it replaces. The payoff is optionality: when we want to add a GPU backend, it is one more emitter walking the AST, not a new compiler.

Correctness

The connector test suite runs every assertion against both upstream perl5+PDL and pperl+Rust-PDL, compares outputs, and categorizes each result. A representative snapshot from the harness at the time of writing:

Tests: ~1380 matched, ~25 mismatched, ~110 extra / ~1400 total

Matched - the large majority, give or take: pperl produces the same output as upstream.

Mismatched - on the order of two dozen: pperl differs from upstream. Most of these are real work in progress and we know where the gaps are. The biggest cluster is in forward-flow dataflow - when a parent PDL is mutated, downstream slice children should see the update eagerly. Our implementation is currently lazy, which catches most cases via dirty-flag refresh but misses the paths that read child data without going through the normal accessor. A handful of tests in 220-broadcasting fail on that. The rest are distributed drift: error message text, one edge case in unbroadcast, STL round-trip issues. None are conceptual; all are bounded.

Extra - around a hundred: tests that pperl passes and upstream does not. Some of these are test-surface coverage - we wrote more thorough tests for overload dispatch, import semantics, and native-function dispatch than upstream did. Others are legitimate behavioural differences where we produce a correct result and upstream does not; the precision and diagnostics subsections below cover the interesting cases.

There is a separate, smaller number worth singling out.

Where pperl and Upstream Disagree, and pperl is Right

A handful of tests produce different results between the two implementations where pperl's result is more correct. Six in the first sweep; we are no longer surprised when we find another. Three patterns are worth naming.

Precision. pctover computes the k-th percentile along a dimension:

my $x = pdl(0,0,6,3,5,0,7,14,94,5,5,8,7,7,1,6,7,13,10,2,101,19,7,7,5);
my $r = $x->pctover(0.9);
printf "%.20f\n", $r->at();

Upstream says 17.00000000000000710543. pperl says 17.00000000000000000000. Both stringify to 17, so to the eye they are identical. But $r->at() == 17 is false on upstream (the 7e-15 drift catches the numeric comparison) and true on pperl. The test was written as a plain == 17 assertion, assuming the kernel would return an exact integer.

Why does pperl get it right? Same numerical recipe, different float accounting. Upstream's PP-generated C coerces intermediates into PDL_Double early and rounds once per sample. The Rust kernel, passing through LLVM's optimizer, keeps a wider accumulator and rounds once at the final divide. It is not cleverness on our side - it is what the modern toolchain does with the same numerical recipe when nothing in the code forces early narrowing. The same pattern appears in xlogvals and a couple of other reductions.

Perl context. lgamma returns two values: the log-gamma value, and the sign. Upstream's PP-generated XS returns both as a two-element list regardless of context. In scalar context, Perl takes the last element - so my $y = lgamma($x) assigns the sign (always 1), not the actual gamma. That is not what a Perl programmer expects. pperl respects GIMME_V: scalar context returns the value, list context returns the pair.

This is not a numerical-methods win. It is a Perl-semantics win - the kind that only a Perl-aware reimplementation can notice. We didn't set out to behave differently from upstream; we set out to behave the way Perl specifies.

Diagnostics. pdl("nonsense blurb") should reject its input. Upstream does reject it - but the error message reads found 'e' as part of a larger word in nonsense blurb, because the parser is reporting that it saw e (Euler's constant) embedded in a longer token, and that is what it complains about. pperl reports found disallowed character(s) 'nonsense,blurb', which is what an actual user wanted to know. A smaller case of the same pattern: $x->slice('0:-14') on a 10-element PDL silently returns a garbage view upstream; pperl croaks with an out-of-bounds error. Good diagnostics are a correctness property.

And in the interest of honest accounting: we have also found places where our implementation was wrong and upstream was right, and we conformed. The float + 0.2 promotion rule mentioned earlier was one of those: we were returning float, upstream correctly promoted to double, we fixed ourselves to match. The flow of corrections is bidirectional and that is the point - two independent implementations of the same spec are a stronger guarantee of correctness than either one alone.

Performance

The startup story from Part One holds and has tightened up. Here is the harness output from the current connector test suite, picking a representative sample:

PDL/010-constructor.t     ... ok  (17/17)   perl5: 84ms  / pperl: 9ms
PDL/110-core.t            ... ok  (121/121) perl5: 88ms  / pperl: 13ms
PDL/120-ops.t             ... ok  (75/75)   perl5: 94ms  / pperl: 12ms
PDL/150-primitive.t       ... ok  (60/60)   perl5: 86ms  / pperl: 11ms
PDL/190-matrixops.t       ... ok  (35/35)   perl5: 90ms  / pperl: 11ms
PDL/260-pdl-from-string.t ... ok  (60/60)   perl5: 86ms  / pperl: 13ms
PDL/300-niceslice.t       ... ok  (30/30)   perl5: 137ms / pperl: 75ms
PDL/330-flexraw-fastraw.t ... ok  (30/30)   perl5: 96ms  / pperl: 20ms

Across the full PDL suite, pperl's per-test wall time sits at 9–13ms for the typical path and climbs to 20–75ms for tests that do real I/O or source filtering. Upstream's corresponding numbers are 82–137ms. The typical startup advantage is 8–10×; the fastest cases reach 13×. This is not a compute-throughput benchmark - it is the cost of use PDL plus a few operations. And it holds because the entire PDL stack is compiled into the pperl binary. There is no module loading, no PP compilation, no dynamic linking.

Compute-throughput numbers are what most readers will actually care about, so here is a table from the current benchmark suite - release build, microseconds per call, averaged over a warm loop. Worth stating up front: these are dry numbers. No JIT loop-fusion, no Rayon parallelism, no GPU offload - none of the mechanisms that define pperl's architectural advantage are engaged here. What is being measured is the raw kernel-versus-kernel cost and the per-call dispatch overhead. The ceiling is considerably higher than what these numbers show; the floor is what they are.

op                    size    upstream µs    pperl µs    ratio
---------------   --------    -----------    --------    -----
pdl_from_str            10           66.6         3.4    20x
pdl_from_str           100          583.4        18.3    32x
pdl_from_str          1000         6667.7       183.4    36x

stringify               10            8.6         1.5     5.6x

stringify              100           60.0        13.1     4.6x

stringify             1000          620.6       140.8     4.3x

add_scalar             100            3.4         1.2     2.8x

matmult_sq              10            5.4         2.3     2.3x

slice                  100            2.0         1.3     1.5x

slice                10000            2.1         1.3     1.7x

slice              1000000            2.2         1.4     1.5x

add_vec            1000000         1860.4      1939.1     0.96x

mul_vec            1000000         1879.3      1790.6     1.05x

sum                1000000         1468.3      1344.1     1.09x

exp                1000000         8174.9      8110.6     1.01x

matmult_sq             200         6329.4      7070.6     0.90x

sumover            1000000         4723.4     13214.5     0.36x

Three things stand out.

Parsing and formatting win cleanly. pdl_from_str - the pdl("1 2 3; 4 5 6") literal parser - is 20-36× faster because ours is a native Rust tokenizer where upstream's is a pure-Perl regex chain. Stringify is 4-6× faster for the same structural reason: a native formatter rather than Perl-side string assembly. These gains are not tuning artefacts; they will persist.

Small-op dispatch wins. On small arrays (100-element add_scalar, 10×10 matmult), pperl is 2-3× faster because the XS overhead is gone. This is the function-pointer overload dispatch from the connector walkthrough, now with numbers: Perl → stash lookup → Rust kernel, no intervening XSUB layer. On large arrays the kernel dominates and the dispatch win disappears into the noise - which is the correct outcome, not a disappointment.

Slice is zero-copy and measurably so. A slice operation on a 1M-element PDL takes about 1.4 µs on pperl, independent of input size - a constant-time affine view construction, as it should be. Upstream is within the same order of magnitude (about 2 µs); both implementations do the right thing here.

Outliers do happen. sumover at 1M elements is currently slower than upstream; it was detected, it is understood, and it is being worked on. That is the expected shape of a young engine against a thirty-year-old one.

These numbers are just a baseline. The "dry" performance without JIT, Autoparallelization or GPU. Autovec by the compiler happens for both. The numbers for JIT+Rayon and/or GPU will land in a dedicated post once the relevant parts of the JIT are ready.

How pperl PDL Compares to Upstream

On Compatibility

Our commitment is to upstream PDL's specified behaviour. When the specification and the current implementation disagree, we side with the specification. lgamma in scalar context is the cleanest illustration: Perl's GIMME_V contract is unambiguous; upstream's PP-generated XS happens not to honour it; ours does. We did not set out to deviate - we set out to mirror Perl.

Everywhere else, upstream is the reference. Thirty years of PDL carry numerical decisions and edge-case wisdom that the documentation does not always explain. Whenever we disagreed with upstream during implementation, our default assumption was that we were missing context. The float + 0.2 promotion case earlier in this post is exactly that: we thought we were right. However, upstream was right - we conformed.

What's Next

Three threads, in rough priority order.

Rayon-based parallelism. First prototypes are running: PDL operations inside pperl's JIT-compiled while-loops distribute across cores via work-stealing. Early results look promising. Mandelbrot-scale workloads already parallelize cleanly; the interesting work is in making parallelism show up for realistic PDL pipelines without per-call overhead eating the win.

OpenCL / GPU acceleration. This is a priority goal, and it is the reason pd2multi exists as a proper compiler rather than a template system. A GPU backend is one more emitter walking the AST, not a separate codegen project. The engineering cost is real but bounded.

JIT fusion across PDL boundaries. Cranelift compilation of surrounding Perl loops that call PDL operations, so the JIT can see through the op and fuse it with the loop body. This is the structural payoff of having a pure-Rust engine - the JIT can look inside - and it is the thing that will make the case for reimplementation in performance terms, not just compatibility terms. Serious work on this is pending the Rayon and GPU threads stabilizing.

No dates. We will post numbers when there are numbers to post.

In the meantime: use PDL; in pperl, today, at https://perl.petamem.com. Roughly three thousand assertions of work, about ninety-eight per cent of them matching the upstream beacon, a double-digit startup win, and a handful of cases where independent implementation caught things that a single implementation had missed.

- Richard C. Jelinek, PetaMem s.r.o.

A while back, I received a pull request suggesting that I update the performance comparison with Encode.pm in my module, Unicode::UTF8. When I originally wrote Unicode::UTF8, Encode.pm used its own UTF-8 validation implementation. Since then, Karl Williamson has done extensive work improving Perl, and Encode.pm now relies on those validation routines based on Björn Höhrmann’s UTF-8 DFA decoder. It’s an elegant piece of code, widely adopted across many projects.

That said, I wasn’t particularly motivated to revisit the comparison, so I decided instead to look into faster scalar C implementations. While it has been suggested that Unicode::UTF8 should gain a SIMD implementation, and that may well be worthwhile, I wanted to first explore improvements to the scalar path, which would still be required as a fallback.

After some searching, I came across a tweet by Russ Cox showing performance numbers for a shift-based DFA implementation in Go, along with a link to a gist by Per Vognsen describing the technique.

It turned out that this shift-based DFA approach was quite popular a few years ago. Several implementations appeared in different programming languages and even in some RDBMs. However, I couldn’t find any reusable code, so I decided to implement it as a header-only C library, updating a UTF-8 validator I originally wrote in 2017.

I won’t go into the details of shift-based DFA encoding here. For a thorough explanation, I recommend Per Vognsen’s article, and I cover my own implementation in more detail here.

The main difference from a traditional DFA, such as Björn Höhrmann’s implementation, lies in how table lookups are handled. A conventional byte-class + transition-table DFA requires two lookups per byte: one to map the byte to a character class, and another to determine the next state. The shift-based DFA combines both steps into a single lookup, at the cost of a larger table. Both DFAs have a serial chain dependency where each byte’s state transition depends on the previous one, which limits instruction-level parallelism.

Two functions are provided for UTF-8 validation:

utf8_valid has no data-dependent branches. Each DFA step is a table lookup combined with a bitwise shift, with no conditional branches on the input byte value. Execution time scales with byte count, not byte values.

utf8_valid_ascii adds an ASCII fast path. There are different approaches to this: Perl, for instance, scans a leading ASCII prefix and falls back to DFA validation upon encountering a non-ASCII byte, never re-entering the fast path. I am not particularly fond of that approach, since even predominantly ASCII content today tends to contain non-ASCII bytes. Instead, I opted for a 16-byte chunk fast path that skips the DFA for chunks consisting entirely of ASCII bytes, and resumes DFA validation in a non-clean state when a non-ASCII byte is encountered.

The implementation showed promising results, but can we go faster? Yes! By splitting the UTF-8 stream in two, we can break the serial chain dependency and run two independent DFA chains in a single interleaved loop. Since the chains share no data dependencies, the CPU’s out-of-order engine can overlap their shift operations on cores with multiple shift-capable execution ports. I did try three interleaved streams but saw no improvement on the hardware available to me, the added overhead of splitting the stream further likely offsets any potential gain.

Whether utf8_valid_ascii is profitable depends heavily on the content. The following benchmark output illustrates this with ASCII-heavy content.

The benchmark includes two reference implementations: hoehrmann and utf8_valid_old, the previous scalar implementation in the C library.

en.txt: 80 KB; 82K code points; 1.00 units/point
  U+0000..U+007F          82K  99.9%
  U+0080..U+07FF           18   0.0%
  U+0800..U+FFFF           49   0.1%
  hoehrmann                    572 MB/s
  utf8_valid_old              3677 MB/s  (6.42x)
  utf8_valid                  6466 MB/s  (11.30x)
  utf8_valid_ascii           41001 MB/s  (71.63x)

sv.txt: 94 KB; 93K code points; 1.04 units/point
  U+0000..U+007F          90K  96.4%
  U+0080..U+07FF           3K   3.5%
  U+0800..U+FFFF          171   0.2%
  hoehrmann                    572 MB/s
  utf8_valid_old              1536 MB/s  (2.68x)
  utf8_valid                  6600 MB/s  (11.53x)
  utf8_valid_ascii            8581 MB/s  (15.00x)

units/point is a rough content-mix indicator: 1.00 is near-pure ASCII, ~1.7-1.9 is common for 2-byte-heavy text, and ~2.7-3.0 for CJK-heavy text. The code point distribution breaks down the input by Unicode range. Results are sorted slowest to fastest; the multiplier shows throughput relative to the slowest implementation.

Output from multibyte-heavy content:

ja.txt: 176 KB; 65K code points; 2.79 units/point
  U+0000..U+007F           7K  10.7%
  U+0080..U+07FF           30   0.0%
  U+0800..U+FFFF          58K  89.3%
  hoehrmann                    572 MB/s
  utf8_valid_old              2046 MB/s  (3.58x)
  utf8_valid_ascii            4200 MB/s  (7.34x)
  utf8_valid                  6492 MB/s  (11.34x)

ar.txt: 25 KB; 14K code points; 1.81 units/point
  U+0000..U+007F           3K  18.9%
  U+0080..U+07FF          12K  81.1%
  hoehrmann                    572 MB/s
  utf8_valid_old               984 MB/s  (1.72x)
  utf8_valid_ascii            4163 MB/s  (7.28x)
  utf8_valid                  6486 MB/s  (11.34x)

utf8_valid reaches approximately 0.71 cycles per byte on wide-issue cores, that’s pretty good for a C scalar implementation. See the Performance section in the GitHub repository for a more in-depth analysis and comparison across different architectures and compiler optimizations.

I couldn’t stop at just a validator. With spare bits left in the DFA table, I went ahead and implemented a complete UTF-8 library covering validation, decoding, navigation, and transcoding. The code is available on GitHub.

Unicode::UTF8 internally uses utf8_valid_ascii in its encode and decode routines.

Next steps: I have no further ideas for how to improve the scalar DFA implementation, and I am satisfied with the performance. A SIMD implementation may be worth exploring in the future, but the more immediate goal would be to get this incorporated into Perl core.

Enjoy! PS Now I can disregard the PR and tune up the numbers in the comparison ;o)

I found this sufficient of an obstacle that I wanted to post about it for future posterity.

I was able to cpanm Google::ProtocolBuffers::Dynamic after installing these packages on Debian Trixie.

• build-essential (unsurprisingly)
• cmake
• libprotobuf-dev
• libprotoc-dev

The last library eluded me and caused the most frustration. Anyway on to other things.

All three of us attended this long meeting consisting entirely of dealing with release blockers.

• We found no blockers among the issues and patches new since last week.

• We weighed up #23676 again and decided that it merits blocker status even though the breakage was the result of a C compiler change, not of a change to perl (in this dev cycle or otherwise).

• We then reviewed all of the blockers we were already tracking and made decisions on how to proceed with all of them. Of particular note,

• We agreed that #23131 simply cannot be pursued in this form – not just in this cycle but at all. We may eventually be able to virtualize the stash entirely, allowing much deeper optimization by using a more rational internal data structure while maintaing the Perl-land illusion that nothing has changed (effectively turning the stash hash tree into an API in the same way that a tied hash is an API that looks like a data structure). But until such time, Perl-visible changes to the stash data structure are simply too disruptive.

• #24340 is part of a series that constitutes a promising-looking fix which we want to pursue, but it didn’t get fully stabilized soon enough in this cycle, and because it’s in a dark and tricky corner of the codebase, we don’t want to take the risk of shipping with undiscovered new breakage there rather than a longstanding bug. Reapplying this patch series early in the next cycle should give it ample time to bake and settle down, and we hope it will eventually ship successfully.

• We spent some time debating #24341, tracing commit and issue tracker history to try to assess the correctness of the changes. We lean toward the view that this is breakage that core should fix, but discussion in the comments is ongoing.

[P5P posting of this summary]

So you've got a bunch of Perl worker processes and they need to share state. A work queue, a counter, a lookup table - the usual. What do you reach for?

Perl has solid options here, and they've been around for a while. File::Map gives you clean zero-copy access to mmap'd files - substr, index, even regexes run directly on mapped memory. LMDB_File wraps the Lightning Memory-Mapped Database - mature, ACID-compliant, lock-free concurrent readers via MVCC, crash-safe persistence. Hash::SharedMem offers a purpose-built concurrent hash with lock-free reads and atomic copy-on-write updates. Cache::FastMmap is the workhorse for shared caching - mmap-backed pages with per-page fcntl locking, LRU eviction, and optional compression.

These are all good, proven tools. But they have something in common: they're about storage. You put data in, you get data out. They don't give you a queue that consumers can block on. They don't give you a pub/sub channel, a ring buffer, a semaphore, a priority heap, or a lock-free MPMC algorithm. They don't do atomic counters or futex-based blocking with timeouts.

That's the gap the Data::*::Shared family fills - fourteen Perl modules that give you proper, typed, concurrent data structures backed by mmap. Not better storage - concurrent data structures that happen to live in shared memory. Queues, hash maps, pub/sub, stacks, ring buffers, heaps, graphs, sync primitives - the works. All written in XS/C, all designed to work across fork()'d processes with zero serialization overhead.

Let me walk you through what's in the box.

The Approach

Every module in the family uses the same core recipe:

• mmap(MAP_SHARED) for the actual shared memory - no serialization, no copies, just raw memory visible to all processes
• Linux futex for blocking/waiting - when a queue is empty and you want to wait for data, you sleep in the kernel, not in a spin loop
• CAS (compare-and-swap) for lock-free operations where possible - no mutex, no contention, just atomic CPU instructions
• PID-based crash recovery - if a process dies holding a lock, other processes detect the stale PID and recover automatically

Requires Linux (futex, memfd), 64-bit Perl 5.22+. A deliberate tradeoff - portable it isn't, but fast it is.

Three ways to create the backing memory:

# File-backed - persistent, survives restarts
my $q = Data::Queue::Shared::Int->new('/tmp/myq.shm', 1024);

# Anonymous - fork-inherited, no filesystem footprint

my $q = Data::Queue::Shared::Int->new(undef, 1024);

# memfd - passable via Unix socket fd, no filesystem visibility

my $q = Data::Queue::Shared::Int->new_memfd("my_queue", 1024);

The Modules

Here's the full roster, grouped by use case.

Message Passing

Data::Queue::Shared - Your bread-and-butter MPMC (multi-producer, multi-consumer) bounded queue. Integer variants use the Vyukov lock-free algorithm; string variant uses a mutex with a circular arena. Blocking and non-blocking modes, batch operations, the whole deal.

use Data::Queue::Shared;

my $q = Data::Queue::Shared::Int->new(undef, 4096);

# In producer

$q->push(42);

$q->push_multi(1, 2, 3, 4, 5);

# In consumer

my $val = $q->pop_wait(1.5);    # block up to 1.5s

my @batch = $q->pop_multi(100);

Single-process throughput: ~5M ops/s for integers. That's roughly 3x MCE::Queue and 6x POSIX message queues.

Data::PubSub::Shared - Broadcast pub/sub over a ring buffer. Publishers write, subscribers each track their own cursor. If a subscriber falls behind, it auto-recovers to the oldest available message. No back-pressure on writers.

my $ps = Data::PubSub::Shared::Int->new(undef, 8192);
$ps->publish(42);

my $sub = $ps->subscribe;

my $val = $sub->poll_wait(1.0);

Batch publishing hits ~170M msgs/s for integers. Yes, really. It's just writing to mapped memory.

Data::ReqRep::Shared - Request-response pattern with per-request reply routing. Client acquires a response slot, sends a request carrying the slot ID, server replies to that specific slot. Supports both sync and async client styles.

# Server
my ($request, $id) = $rr->recv_wait(1.0);
$rr->reply($id, "processed: $request");

# Client (async)

my $id = $rr->send("do something");

my $response = $rr->get_wait($id, 2.0);

Around 200K req/s cross-process - competitive with Unix domain sockets but with true MPMC support.

Key-Value

Data::HashMap::Shared - This is the big one. Concurrent hash map with elastic capacity, optional LRU eviction (clock algorithm with lock-free reads), optional per-key TTL, atomic counters, sharding, cursors. Eleven type variants from II (int-int) to SS (string-string).

use Data::HashMap::Shared::SS;

my $map = Data::HashMap::Shared::SS->new('/tmp/cache.shm', 100_000);

$map->put("user:123", "alice");

my $name = $map->get("user:123");

# LRU cache with max 10K entries

my $cache = Data::HashMap::Shared::SS->new('/tmp/lru.shm', 100_000, 10_000);

# TTL - entries expire after 60 seconds

my $ttl = Data::HashMap::Shared::II->new('/tmp/ttl.shm', 100_000, 0, 60);

# Atomic counter (lock-free fast path under read lock)

$map->incr("hits:page_a");

Cross-process string reads: 3.25M/s. Integer lookups hit ~10M/s. And you get built-in LRU and TTL without an external cache layer.

Sequential & Positional

Data::Stack::Shared - Lock-free LIFO stack. Push, pop, peek. ~6.4M ops/s.

Data::Deque::Shared - Double-ended queue. Push/pop from both ends. Lock-free CAS. ~6.3M ops/s.

Data::RingBuffer::Shared - Fixed-size circular buffer that overwrites on wrap. No consumer tracking - you just read by position. Great for metrics windows and rolling logs. ~11.7M writes/s.

Data::Log::Shared - Append-only log. Unlike Queue (consumed on read) or RingBuffer (overwritten), Log retains everything until explicitly truncated. CAS-based append, cursor-based reads. ~8.9M appends/s.

Resource Management

Data::Pool::Shared - Object pool with allocate/free. CAS-based bitmap allocation, typed slots (I64, I32, F64, Str), scope guards for automatic cleanup, raw C pointers for FFI integration. PID-tracked slots are auto-recovered when a process dies.

my $pool = Data::Pool::Shared::I64->new(undef, 256);
my $idx = $pool->alloc;
$pool->set($idx, 42);
# ...
$pool->free($idx);

# Or with auto-cleanup

{

    my $guard = $pool->alloc_guard;

    $pool->set($$guard, 99);

}  # auto-freed here

Data::BitSet::Shared - Fixed-size bitset with per-bit atomic CAS operations. Good for flags, membership tracking, allocation bitmaps. ~10.5M ops/s.

Data::Buffer::Shared - Type-specialized arrays (I8 through F64, plus Str) with atomic per-element access. Seqlock for bulk reads, RW lock for bulk writes. Think shared sensor arrays or metric buffers.

Graphs & Priority

Data::Graph::Shared - Directed weighted graph with mutex-protected mutations. Node bitmap pool, adjacency lists, per-node data. ~3.9M node adds/s, ~13.3M lookups/s.

Data::Heap::Shared - Binary min-heap for priority queues. Mutex-protected, futex blocking when empty. ~5.3M pushes/s.

Synchronization Primitives

Data::Sync::Shared - Five cross-process sync primitives in one module: Semaphore, Barrier, RWLock, Condvar, and Once. All futex-based, all with PID-based stale lock recovery, all with scope guards.

use Data::Sync::Shared;

my $sem = Data::Sync::Shared::Semaphore->new(undef, 4);  # 4 permits

{

    my $guard = $sem->acquire_guard;

    # at most 4 processes here concurrently

}

my $barrier = Data::Sync::Shared::Barrier->new(undef, $num_workers);

$barrier->wait;  # blocks until all workers arrive

my $once = Data::Sync::Shared::Once->new(undef);

if ($once->enter) {

    init_expensive_thing();

    $once->done;

}

At a Glance

Type Specialization

Most modules come in typed variants - Int16, Int32, Int64, Str, and so on. This isn't just for type safety. An Int16 queue uses half the memory of an Int64 queue, which means double the cache density on the same hardware. When you're doing millions of operations per second, cache lines matter.

Event Loop Integration

Every module supports eventfd() for integration with event loops like EV, Mojo, or AnyEvent:

my $fd = $q->eventfd;
# register $fd with your event loop
# on readable: $q->eventfd_consume; then poll/pop

Signaling is explicit ($q->notify) so you can batch writes before waking consumers.

Playing Nice with Others: PDL, FFI::Platypus, OpenGL::Modern

One thing I want to highlight is that these aren't isolated islands. Because everything lives in mmap'd memory with known layouts, you get natural interop with other systems that work with raw pointers and packed data.

PDL is the obvious one. If you're doing numerical work in Perl - signal processing, image manipulation, statistics - PDL is your workhorse. The Buffer module's as_scalar returns a zero-copy scalar reference directly over the mmap'd region. Feed that to PDL and you've got an ndarray backed by shared memory:

use Data::Buffer::Shared::F64;
use PDL;

my $buf = Data::Buffer::Shared::F64->new('/tmp/signal.shm', 10000);

# one process fills the buffer with sensor data...

# another process reads it as a PDL:

my $pdl = PDL->new_from_specification(double, 10000);

${$pdl->get_dataref} = ${$buf->as_scalar};

$pdl->upd_data;

printf "mean=%.4f stddev=%.4f\n", $pdl->stats;

For typed arrays you can also use get_raw/set_raw for bulk transfers - a single memcpy under the hood, seqlock-guarded for consistency. That means you can build a multiprocess image pipeline where one process captures frames into a shared U8 buffer, another runs PDL convolutions on it, and a third renders the result - all communicating through shared memory with eventfd notifications, no serialization anywhere.

FFI::Platypus works just as naturally. Pool and Buffer both expose ptr() / data_ptr() - raw C pointers as unsigned integers, ready to hand to any C function through FFI. Need to call libc qsort directly on your shared data? Go ahead:

use Data::Pool::Shared;
use FFI::Platypus;

my $pool = Data::Pool::Shared::I64->new(undef, 1000);

# ... alloc and fill slots ...

my $ffi = FFI::Platypus->new(api => 2);

$ffi->lib(undef);  # libc

$ffi->attach([qsort => 'c_qsort'] =>

    ['opaque', 'size_t', 'size_t', '(opaque,opaque)->int'] => 'void');

c_qsort($pool->data_ptr, 1000, 8, $comparator);

# slots are now sorted in-place, visible to all processes

Pool slots are contiguous in memory (data_ptr + idx * elem_size), so any C library that expects a flat array works out of the box.

OpenGL::Modern is where it gets fun. Buffer::F32 is essentially a shared vertex buffer. One process computes positions, another renders them - connected by a shared mmap region and eventfd:

# Compute process:
my $verts = Data::Buffer::Shared::F32->new('/tmp/verts.shm', 30000);
$verts->set_slice(0, @new_positions);
$verts->notify;

# Render process:

my $ref = $verts->as_scalar;

# on eventfd readable:

glBufferSubData_p(GL_ARRAY_BUFFER, 0, $$ref);  # zero-copy upload

Pool goes further - it's a natural fit for particle systems. Particles are dynamically spawned (alloc) and despawned (free), each with a fixed-size state struct. A spawner process allocates particles, a physics process updates them, and the renderer uploads the live slots to a VBO via ptr(). The raw pointer goes straight to glBufferSubData_c - no packing, no intermediate copies.

The common thread here is that the data is already in the format the consuming library expects. F32 buffers are packed floats. I64 pools are packed int64s. There's no Perl-side serialization layer to bypass because there was never one to begin with.

Optional Keyword API

If you install XS::Parse::Keyword, several modules expose lexical keywords that bypass Perl method dispatch entirely:

use Data::Queue::Shared;

q_int_push $q, 42;

my $v = q_int_pop $q;

Zero dispatch overhead. The XS function gets called directly. It's optional - the method API works fine - but it's there when you need every last microsecond.

The Big Picture

Here's how the pieces fit together in a typical system:

• Data::Queue::Shared distributes work from producers to a pool of workers
• Data::HashMap::Shared acts as a shared cache or config store that all workers read from
• Data::PubSub::Shared broadcasts events or status updates to whoever's listening
• Data::Sync::Shared coordinates startup (Barrier), limits concurrency (Semaphore), and protects shared initialization (Once)
• Data::Pool::Shared manages reusable resource slots
• Data::RingBuffer::Shared or Data::Log::Shared holds recent metrics or audit trails

All of this running across fork()'d processes, communicating through shared memory at millions of operations per second, no serialization overhead.

Getting Started

Values are typed C scalars or fixed-length strings - no automatic serialization of arbitrary Perl structures. That's by design: raw mmap'd memory is what makes everything fast and FFI-friendly, but it means you won't be sharing hashrefs or blessed objects directly.

All modules follow the same pattern:

use Data::Queue::Shared;

# Pick your backing: file, anonymous, or memfd

my $q = Data::Queue::Shared::Int->new(undef, 4096);

if (fork() == 0) {

    $q->push($$);  # child pushes its PID

    exit;

}

my $child_pid = $q->pop_wait(5.0);

say "Child reported in: $child_pid";

The modules are on GitHub under the vividsnow account. Each one has its own repo, test suite, and benchmarks you can run yourself.

If you've ever wished Perl had something like Go's channels and sync primitives but for fork()'d processes - well, now it does. Fourteen of them, actually.

Happy sharing