Add comments from Unicode to script_run()
embed.fnc: Use correct name for formal parameter The prototype here did not match the actual function
grok_bin_oct_dec: Speed up overflow detection We can compute outside the loop the exact value at which the next iteration wil overflow, saving some operations
grok_bin_oct_hex: Add/revise some comments grok_bin_oct_hex: Move comments And reflow. This is in preparation for the next commit.
grok_bin_oct_hex: Add two UNLIKELY()s Underscores in numbers are much less common than digits, and its unlikely that this iteration of the loop through all the digits will still have a running total of 0.
When writing perl functions that operate on arrays I use the tail of @_ as the "array argument" and when returning I return them by value. This keep the api simple
When I write map like functions:
sub mymap (&@) { ... }
I receive the list @ as last argument. Assuming I know these functions will operate over big arrays (#>10000) is it worth to write it to receive a reference instead of the array?
The same question applies to returning arrays, if I return a big array, is it worth to return in a reference?
By "worth" here I means, are there any gains in performance and how much?
Or is perl smart enought to optimize these cases?
My current understanding is that when I do
foo(@bar)
Perl create an alias for each value in @bar in foo's stack, if I replace @bar with \ @bar a single alias is created. This should not matter for functions that receive like 10 arguments, but for functions like map, grep etc, that operate on arrays, they may easily receive >10000 arguments
Another question, are map and grep optimized to operate on big arrays?
As suggested by Ikegami, here is a benchmark for reference
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Data::Dumper qw(Dumper);
use List::Util;
use Benchmark qw(:all);
sub array_as_value {
my @list = (0 .. shift);
for (@_) {
}
return @list;
}
sub array_as_ref {
my @list = (0 .. shift);
for (shift->@*) {
}
return \@list;
}
my %test;
for (my $i = 100; $i < 1_000_000; $i *= 10) {
my @big_array = (0 .. $i);
say "Testing with $i values";
cmpthese(-5, {
as_value => sub { my @result = array_as_value($i, @big_array); scalar @result; },
as_ref => sub { my $result = array_as_ref($i, \@big_array); scalar $result->@*; }
});
say "";
}
Here are the results
Testing with 100 values
Rate as_value as_ref
as_value 217634/s -- -40%
as_ref 365740/s 68% --
Testing with 1000 values
Rate as_value as_ref
as_value 23641/s -- -43%
as_ref 41365/s 75% --
Testing with 10000 values
Rate as_value as_ref
as_value 2053/s -- -46%
as_ref 3813/s 86% --
Testing with 100000 values
Rate as_value as_ref
as_value 200/s -- -50%
as_ref 402/s 101% --
So from what I understand, passing references is considerably faster for big arrays
examples/json_schema_validate.pl
use JSON::Schema::Validate;
use JSON ();
use open qw( :std :encoding(UTF-8) );
my $schema = {
'$schema' => 'https://json-schema.org/draft/2020-12/schema',
'$id' => 'https://example.org/s/root.json',
type => 'object',
required => [ 'name' ],
properties => {
name => { type => 'string', minLength => 5 },
next => { '$dynamicRef' => '#Node' },
},
'$dynamicAnchor' => 'Node',
additionalProperties => JSON::false,
};
my $js = JSON::Schema::Validate->new( $schema )
->compile
->content_checks
->ignore_unknown_required_vocab
->prune_unknown
->register_builtin_formats
->trace
->trace_limit(200) # 0 means unlimited
->unique_keys; # enable uniqueKeys
#my $data = {
# name => 'head',
# next => {
# name => 'tail'
# }
#};
#my $data = {
# name => 23,
# next => {
# name => 'tail'
# }
#};
#my $data = {
# name => 'head',
#};
my $data = {
name => 'head big',
};
my $ok = $js->validate($data)
or die( $js->error );
print "ok\n";
This is a series of post of my experiences learning Perl web development with Vuejs. These are all the posts:
For anyone interested in helping me shakedown PAGI (https://github.com/jjn1056/pagi/) docs in preparation for publishing to CPAN, I'd love feedback on the tutorial:
https://github.com/jjn1056/pagi/blob/main/lib/PAGI/Tutorial.pod
Or any part of the docs really. But the tutorial is aimed to get people up to speed so any feedback on that is really helpful
[link] [comments]
I have a regular expression in an extension of java by florian ingerl
to parse a latex command,
\\\\DocumentMetadata(?<docMetadata>\\{(?:[^{}]|(?'docMetadata'))*\\})
but the important thing is that it allows nested braces.
The concrete string to be matched is
\DocumentMetadata{pdfversion=1.7,pdfstandard={a-3b,UA-1}}
but earlier or later time comes where deeper nesting is required.
To that end, I use the extension com.florianingerl.util.regex.MatchResult of the builtin java regular expressions.
Now I want to use latexmk for latex which is in Perl and need to adapt .latemkrc which is just Perl code.
So i need the same regular expression or at least something similar i can automatically transform.
Up to now each expression worked in both worlds. But this one does not match in Perl.
Maybe there is some extension which does.
I found in Perl recursion but not with named groups.
This text was translated using software. However, I wrote almost all of it myself. So please bear with me if the language sounds a bit…
Releasing update to LinkedList::Single based on Object::Pad. Faster, cleaner, much more complete module.
Catch: It breaks the old interface & requires v5.40. Handling the Perl version is easy -- same way as FindBin::Libs w/ a ./version subdir, nobody gets onevthat uses a later Perl version. The interface changes can be handled with:
a. Release it as LinkedList::Single v2.0.0. b. Release is as LinkedList::Single2.
Catch with option b is nobody will know it's there or bother to use it. Then. again, maybe nobody uses it or cares... the old version can be installed with an ENV setting if people don't want O::P.
Q: Does anybody out there actually use the existing LinkedList::Single?
Thanks
[link] [comments]
This week's challenges were early and on the easy side, which was appreciated as the holidays loom. The musical interlude this week ignores the challenge tasks in favor of seasonal light jazz; I have Spotify playing "Winter's Songs: A Windham Hill Christmas."
Task 1: Max Words
The Task
You are given an array of sentences. Write a script to return the maximum number of words that appear in a single sentence.
-
Example 1:
-
Input:
@sentences = ("Hello world", "This is a test", "Perl is great") - Output: 4
-
Input:
The Solution
An easy one -- an early Xmas gift? Split each sentence into words and count them, then take the maximum. Completely ignoring all the complexities of punctuation and internationalization, it's a one-liner:
sub maxWords(@sentence)
{
use List::Util qw/max/;
return max map { scalar(split(" ", $_)) } @sentence;
}
The only thing mildly tricky here is transforming the list of words into a count. Inside the body of the map block, we're in array context, so split will return a list. Coercing it with scalar gives the count.
Task 2: Validate Coupon
The Task
You are given three arrays, @codes, @names and @status. Write a script to validate codes in the given array. A code is valid when the following conditions are true:
-
codes[i]is non-empty and consists only of alphanumeric characters (a-z, A-Z, 0-9) and underscores (_). -
names[i]is one of the following four categories: "electronics", "grocery", "pharmacy", "restaurant". -
status[i]is true.
Return an array of booleans indicating validity: output[i] is true if and only if codes[i], names[i] and status[i] are all valid.
-
Example 1:
-
Input:
@codes = ("A123", "B_456", "C789", "D@1", "E123")@names = ("electronics", "restaurant", "electronics", "pharmacy", "grocery")@status = ("true", "false", "true", "true", "true")
-
Output:
(true, false, true, false, true)
-
Input:
Thoughts
The volume of text for this one was daunting at first glance, but it turns out to be mostly example data and actually quite easy.
The validation is straight-forward. @codes is a regular expression match; @names is a keyword lookup from a dictionary table (hash); and @status is just a string compare.
We'll be operating on parallel arrays. I see two ways of doing it. First, we could use a variable to loop over the indexes. That's going to yield code with a lot of array subscripts. Or second, we could use each_array from List::More::Utils to do the indexing for us. I'm curious which is more efficient.
To the code-sleigh, Rudolph!
Loop with indexed
sub valid($code, $name, $status)
{
state %ValidName = ( electronics => true, grocery => true,
pharmacy => true, restaurant => true );
my @valid = (true) x $code->@*;
for my ($i, $c) ( indexed $code->@* )
{
$valid[$i] &&= ( $code->[$i] !~ m/[^a-zA-Z0-9_]/ );
$valid[$i] &&= ( exists $ValidName{$name->[$i]} );
$valid[$i] &&= $status->[$i] eq "true";
}
return \@valid;
}
Notes:
- Passing multiple arrays as subroutine parameters implies using references. We could use traditional prototype attributes to make it possible to pass arrays, but let's not.
-
state %ValidName-- These are valid keywords for the@namesinput array. Usingstatemakes this a static table, initialized only the first time the subroutine is entered. -
@valid-- Our output array, same size as the input array. Assume they're all valid and turn elements false if any check fails. -
indexed-- My favorite recent Perl improvement. Returns both the index and the value from an array during aforloop. -
valid->[$i] &&= ...-- Any failing test will turn the value to false. -
$code->[$i] !~ m/[^a-zA-Z0-9_]/-- The code has to consist entirely of valid characters, or conversely, it must not match (!~) anything that contains a character other than those. Later I went back to look up exactly what\wand\Wmean in Perl regular expressions and realized that I could have used that instead of the character classes. -
exists $ValidName{...}-- Useexistsexplicitly to return a boolean value. A simple lookup that fails would returnundef, and will make the result of the expressionundefinstead offalse. Althoughundefis treated as false in many contexts, it failed the unit test. -
return \@valid-- Return a reference. This is convenient for theis()function from theTest2::V0unit testing framework, and returning a reference also improves performance by avoiding an array copy.
Loop with array iterator
sub validEach($code, $name, $status)
{
state %ValidName = ( electronics => true, grocery => true,
pharmacy => true, restaurant => true );
my @valid;
use List::MoreUtils qw/each_arrayref/;
my $ea = each_arrayref($code, $name, $status);
while ( my ($c, $n, $s) = $ea->() )
{
push @valid, ( $c !~ m/[^a-zA-Z0-9_]/ )
&& ( exists $ValidName{$n} )
&& ( $s eq "true" );
}
return \@valid;
}
Notes:
- This is basically the same function, except that the explicit indexing has been replaced by an iterator.
-
each_arrayref-- There's aneach_arrayfunction, but since we have array references, this one is convenient. -
my $ea = ...-- Creates an iterator over the three arrays. -
ea->()-- Call the iterator.eais the equivalent of a function pointer, and this is function dereference syntax. -
while ( my ($c, $n, $s) = ea->() )-- Each call to the iterator returns a tuple of the next element from each of the$code,$name, and$statusarrays. -
push @valid, ...-- The same three checks as before, but building the list instead of modifying it, because we don't have an index in hand.
Performance check
I think the iterator version is a little cleaner, because it eliminates the clutter of repeated array subscripts. But I was curious how it compared in efficiency, so I set up a quick benchmark using the examples.
It turns out that the explicit indexing is about twice as efficient as the each_arrayref iterator. Still gonna use it, though.
$ perl ch-2.pl -b 100000
Rate iterator forindex
iterator 72464/s -- -47%
forindex 136986/s 89% --
To do the benchmarking, I set up the examples as an array of test cases, and used Benchmark::cmpthese to do a lot of iterations.
sub runBenchmark($repeat)
{
use Benchmark qw/cmpthese/;
cmpthese($repeat, {
forindex => sub {
for ( @case )
{
valid($_->{codes}, $_->{names}, $_->{status});
}
},
iterator => sub {
for ( @case )
{
validEach($_->{codes}, $_->{names}, $_->{status});
}
},
});
}
And the input data looks like this:
my @case = (
{ id => 'Example 1',
codes => [ 'A123', 'B_456', 'C789', 'D@1', 'E123' ],
names => [ 'electronics', 'restaurant', 'electronics', 'pharmacy', 'grocery' ],
status => [ 'true', 'false', 'true', 'true', 'true' ],
expect => [ true, false, true, false, true ],
},
{ id => 'Example 2',
codes => [ 'Z_9', 'AB_12', 'G01', 'X99', 'test' ],
names => [ 'pharmacy', 'electronics', 'grocery', 'electronics', 'unknown' ],
status => [ 'true', 'true', 'false', 'true', 'true' ],
expect => [ true, true, false, true, false ],
},
{ id => 'Example 3',
codes => [ '_123', '123', '', 'Coupon_A', 'Alpha' ],
names => [ 'restaurant', 'electronics', 'electronics', 'pharmacy', 'grocery' ],
status => [ 'true', 'true', 'false', 'true', 'true' ],
expect => [ true, true, false, true, true ],
},
{ id => 'Example 4',
codes => [ 'ITEM_1', 'ITEM_2', 'ITEM_3', 'ITEM_4' ],
names => [ 'electronics', 'electronics', 'grocery', 'grocery' ],
status => [ 'true', 'true', 'true', 'true' ],
expect => [ true, true, true, true ],
},
{ id => 'Example 5',
codes => [ 'CAFE_X', 'ELEC_100', 'FOOD_1', 'DRUG_A', 'ELEC_99' ],
names => [ 'restaurant', 'electronics', 'grocery', 'pharmacy', 'electronics' ],
status => [ 'true', 'true', 'true', 'true', 'false' ],
expect => [ true, true, true, true, false ],
},
);
There are two issues with event loop coding, related to the need to maintain an asynchronous, non-blocking style.
- It's harder to write and maintain than linear, blocking code.
- Despite all the asynchronous behaviour, it's still single threaded.
You can break out of the async/non-blocking mode by forking, of course, but it's not a lightweight operation and creates the risk of orphaned processes even if most of the IPC work is hidden by a good library.
Wouldn't it be nice if you could simply write subs in the plain old linear, blocking style and then call them asynchronously, letting them run in parallel to your main thread until they're ready, no forking required? After all, you're probably already using some kind of async result mechanism like callbacks, or promises, or AnyEvent condition variables, or Future objects to manage existing async behaviour. Wouldn't it be nice if you could just call a sub and deal with it using one of those mechanisms instead of the usual synchronous behaviour?
Enter Thread::Subs.
Thread::Subs grants you this very powerful ability to execute subs in parallel using Perl's not-much-loved "iThreads" mechanism, but with almost all of the sharp edges that make parallel programming hard carefully hidden behind a simple API. So long as your subs pass basic data types in and out (within the limits of threads::shared), you'll be able to execute the sub in parallel, the only visible difference being that it returns a "result" object like an AnyEvent condition variable, which is easily converted to a real AnyEvent condition variable, a Future, a Mojo::Promise, or a callback.
This functionality facilitates several frequently-encountered use cases.
- Generic worker pools to execute CPU-intensive or otherwise slow operations
- Resource-limited pools, such as DB workers, where concurrency must be capped
- Operations like log-writing which require serial execution but can operate concurrently with other activities
Worker threads are spawned early in the process lifecycle and persist after that point, meaning that you don't have anywhere near as much per-call overhead as forking or spawning additional threads. It also means you don't have to manage workers once they start up: it's set and forget.
Thread::Subs uses Perl's attribute mechanism to mark specific subs as "Thread" and can then replace the sub with a shim that calls the sub asynchronously in a worker thread. As such, a generic blocking function like the following ...
sub munge {
my ($foo, $bar) = @_;
# perform heavyweight munge operation
return $munged;
}
my $munged = munge('fred', 123);
... can be converted into an asynchronous, parallel operation like so.
use threads;
use Thread::Subs;
sub munge :Thread {
my ($foo, $bar) = @_;
# perform heavyweight munge operation
return $munged;
}
Thread::Subs::startup(); # using default pool sizes
my $result = munge('fred', 123);
# do something else while munging
my $munged = $result->recv;
This is the kind of low-boilerplate code that's hard to pull off in anything but Perl. Java has had something very similar in the form of ExecutorService for a long time, but you have to construct the object yourself and feed it specific functions. Furthermore, Thread::Subs has three attributes, "qlim", "clim", and "pool", which allow fine-grain control over the parallelism not available in most other languages except on a roll-your-own basis. In particular, requests for such sub calls go into a FIFO queue prior to execution by workers, and you can limit that queue independently of the concurrency.
pool- worker pool name - subs can be tied to named pools or even given a private pool of workers; you can set the number of workers per poolqlim- queue limit - the maximum number of requests for a sub which can be in the queue before requests start to block; default no limitclim- concurrency limit - the maximum number of workers which can execute this sub simultaneously; default all workers in the pool
These parameters can be specified as part of the Thread attribute, as in sub foo :Thread(clim=1), or they can be set via a function, Thread::Subs::define(). The specific combination of a concurrency limit of one and a private pool of one worker, Thread(clim=1, pool=SUB), effectively offers the sub its own private Perl interpreter. It has free reign to maintain any kind of state it needs in this environment.
Thread::Subs is designed to make basic parallelism trivial to implement with no further dependencies (nothing outside core Perl as of v5.22), but it's also designed with event loops in mind. It has native support for AnyEvent, Mojolicious, and anything which works with Future objects. For example, if foo() is a sub with the Thread attribute, then foo()->future returns a Future object (so long as you use Future; first).
Give it a try and put some of those underutilised CPU cores to work.
Originally published at Perl Weekly 752
Hi there,
Marlin? Yet another object-oriented programming framework?
There are plenty of choices available already, but this one is worth trying, to be honest. A quick introductory post on the subject by the creator, Toby Inkster is worth reading.
Funky? PSPWA Framework?
It's what happens when a PWA and an SPA have a baby, and that baby is raised by a camel who really cares about user experience. The pattern of combining PWA capabilities with SPA architecture has been around for years. Please find out more about it on the official page.
We also had another development release of Perl v5.43.6 a couple of days ago. The main change is that using goto to jump into the body of a loop or other block construct from outside is no longer permitted.
The TIOBE Index for December 2025 shows positive signs for Perl. However, you shouldn't take it too seriously. That said, I really enjoy reading it.
This is the last edition of the year 2025 for me - the 196th issue overall - so Merry Christmas and Happy New Year 2026 to all the readers. Please stay safe and healthy.
Enjoy rest of the newsletter.
--
Your editor: Mohammad Sajid Anwar.
Announcements
Introducing Marlin
Yet another OOP framework, where most of your constructors and accessors will be implemented in XS.
Funky - The PSPWA Framework
A comprehensive guide to building modern, secure, real-time web applications with Perl and vanilla JavaScript.
Foswiki 2.1.10 is released
We are delighted to announce the new release, which includes 57 significant bug fixes compared to the previous 2.1.8 version. This update addresses a range of important issues and enhances the overall stability and performance.
The corner of Gabor
A couple of entries sneaked in by Gabor.
Perl Maven online: Live Open Source contribution
During the holidays, on December 26, there is still an online Perl event to encourage more people to contribute to open source Perl projects. You are invited!
Articles
Masters of Destiny
This post is primarily a philosophical essay. Saif uses a fictional dialogue to express skepticism about over-reliance on AI and "dispassionate algorithms," particularly in fields like medicine that traditionally value human judgment and compassion.
Mid-life upgrade to the MailBox suite completed
This is a high-quality project maintenance post from Mark. It provides valuable insight into the pragmatic considerations of modernising a massive, real-world Perl codebase—decisions about language versions, exception frameworks, and managing breaking changes.
TIOBE Index for December 2025
This is fun index and not to be taken seriously. Perl has +1.33% increase and currently ranked #9.
Perl Advent Calendar
Using Mojolicious::Plugin::Mount to help test your applications
This is an insightful, practical tutorial that presents a clever and elegant pattern for integration testing in Mojolicious. It demonstrates a sophisticated understanding of the Mojolicious ecosystem and provides a viable alternative to the more common approach of mocking HTTP user agents.
Auto-instrument your code with OpenTelemetry
The post introduces a new experimental approach for automatically generating OpenTelemetry traces for Perl code. The traditional options are either 1) use a pre-built instrumentation library, 2) write a custom instrumentation library (hard), or 3) manually instrument your code (tedious). This module offers a fourth, automated path: dynamically instrumenting Perl modules at runtime without modifying their source code.
The Elves Learn to be Lazy
This narrative post is a superb piece of technical communication. It successfully argues that adopting a modern OO framework like Moose isn't about "cool new features" for their own sake, but about practical engineering benefits: safer refactoring, better testing, and more maintainable code.
Safer last-minute hotfixes before Christmas
App::Transpierce is a Perl-based command-line utility for managing configuration changes on production systems. Its core philosophy is to provide a structured, safe workflow for emergency "hotfixes" where direct editing on a live server is unavoidable.
Advent of the Underbar
This is a well-conceived and thoughtfully launched community project. The post successfully makes the case for the podcast's existence by grounding it in the Phileppe's genuine passion and a clear, unmet need for recorded oral history within the Perl ecosystem.
How SUSE is Using Perl
The primary coverage of the talk concerns SUSE's openQA and the Open Build Service. These both lean heavily on Perl and are tools that you may find useful in your own work.
The Gift of Readability
The article is a well-crafted narrative that argues readable code is an act of empathy and professionalism, not just a stylistic preference. It stands out by framing technical advice within a memorable story (the elves at the Present Delivery Network), making the concepts more engaging than a dry style guide.
The Weekly Challenge
The Weekly Challenge by Mohammad Sajid Anwar will help you step out of your comfort-zone. You can even win prize money of $50 by participating in the weekly challenge. We pick one champion at the end of the month from among all of the contributors during the month, thanks to the sponsor Lance Wicks.
The Weekly Challenge - 353
Welcome to a new week with a couple of fun tasks "Max Words" and "Validate Coupon". If you are new to the weekly challenge then why not join us and have fun every week. For more information, please read the FAQ.
RECAP - The Weekly Challenge - 352
Enjoy a quick recap of last week's contributions by Team PWC dealing with the "Match String" and "Binary Prefix" tasks in Perl and Raku. You will find plenty of solutions to keep you busy.
TWC352
Overall solid solutions with correct logic. Task2 is particularly well-implemented. The code demonstrates good Perl idioms and problem-solving skills.
Binary Match
Arne Sommer presents Raku solutions to the weekly challenge #352, demonstrating both practical implementations and exploratory algorithmic thinking. It is worth reading for Raku learners and those interested in different approaches to these challenges.
Five is the one-liest number
Solves the problem correctly, efficiently, and readably. Suitable for Perl beginners to intermediate programmers.
Perl Weekly Challenge: Week 352
Excellent solutions with clear explanations! Your approach shows good understanding of both Raku and Perl idioms.
Triangular Squares
This is an excellent and sophisticated technical write-up. It goes beyond simply providing answers by delving into algorithmic logic, implementing solutions in multiple languages (Perl and J), and including high-quality visual aids for understanding complex array operations.
and here comes Christmas
Impressive polyglot implementation - 5 languages plus SQL variants. Consistent algorithm approach across languages. Good use of language-specific features. Clear code structure and organization
Perl Weekly Challenge 352
This is competition-level algorithmic thinking combined with production-quality code. Truly impressive work!
Doodling with matches and prefixes
This is a solid and practical solution guide for Perl Weekly Challenge #352. Its primary strength lies in the Packy's exploration of the same solution logic across four different programming languages (Raku, Perl, Python, Elixir), providing excellent comparative value for polyglot programmers.
Bits of strings and strings of bits
The solutions are not just correct, but thoughtfully optimized and well-engineered. Clear documentation of design decisions.
The Weekly Challenge #352
This post presents a functional but simplistic approach to solving the two challenges.
Prefix the Matches in Strings of Binary
This is excellent work from a clearly experienced developer. The solutions are clean, maintainable code. Roger demonstrates strong algorithmic thinking and practical implementation skills.
Strings and Binaries
This is a solid, practical implementation of coding challenges that correctly solves the problems. Simon demonstrates good programming fundamentals and language knowledge.
Not So Loopy Digits
This solution demonstrates expert-level understanding of formal language theory and automata. This is PhD-level computer science applied to a coding challenge.
Rakudo
2025.50 Exemplar Poll
Weekly collections
NICEPERL's lists
Great CPAN modules released last week;
MetaCPAN weekly report.
Events
Perl Maven online: Live Open Source contribution
December 26, 2025
Boston.pm - online
January 13, 2025
German Perl/Raku Workshop 2026 in Berlin
March 16-18, 2025
You joined the Perl Weekly to get weekly e-mails about the Perl programming language and related topics.
Want to see more? See the archives of all the issues.
Not yet subscribed to the newsletter? Join us free of charge!
(C) Copyright Gabor Szabo
The articles are copyright the respective authors.
-
App::Netdisco - An open source web-based network management tool.
- Version: 2.097000 on 2025-12-16, with 810 votes
- Previous CPAN version: 2.096001 was 2 days before
- Author: OLIVER
-
CPANSA::DB - the CPAN Security Advisory data as a Perl data structure, mostly for CPAN::Audit
- Version: 20251221.001 on 2025-12-21, with 25 votes
- Previous CPAN version: 20251214.001 was 7 days before
- Author: BRIANDFOY
-
Dist::Zilla::Plugin::Test::Compile - Common tests to check syntax of your modules, using only core modules
- Version: 2.059 on 2025-12-16, with 13 votes
- Previous CPAN version: 2.058 was 7 years, 11 months, 27 days before
- Author: ETHER
-
Image::ExifTool - Read and write meta information
- Version: 13.44 on 2025-12-15, with 44 votes
- Previous CPAN version: 13.36 was 3 months, 6 days before
- Author: EXIFTOOL
-
JSON::Schema::Modern - Validate data against a schema using a JSON Schema
- Version: 0.630 on 2025-12-14, with 16 votes
- Previous CPAN version: 0.629 was 2 days before
- Author: ETHER
-
List::Gen - provides functions for generating lists
- Version: 0.979 on 2025-12-21, with 24 votes
- Previous CPAN version: 0.978
- Author: SOMMREY
-
Minilla - CPAN module authoring tool
- Version: v3.1.29 on 2025-12-17, with 98 votes
- Previous CPAN version: v3.1.28 was 3 months, 2 days before
- Author: SYOHEX
-
Module::CoreList - what modules shipped with versions of perl
- Version: 5.20251220 on 2025-12-20, with 44 votes
- Previous CPAN version: 5.20251120 was 1 month before
- Author: BINGOS
-
Mouse - Moose minus the antlers
- Version: v2.6.1 on 2025-12-20, with 63 votes
- Previous CPAN version: v2.6.0 was 1 month, 20 days before
- Author: SKAJI
-
PGXN::API - Maintain and serve a REST API to search PGXN mirrors
- Version: v0.21.0 on 2025-12-15, with 18 votes
- Previous CPAN version: v0.20.2 was 1 year, 9 months before
- Author: DWHEELER
-
Sidef - The Sidef Programming Language
- Version: 25.12 on 2025-12-21, with 121 votes
- Previous CPAN version: 24.11 was 1 year, 22 days before
- Author: TRIZEN
-
Text::Markup - Parse text markup into HTML
- Version: 0.41 on 2025-12-18, with 12 votes
- Previous CPAN version: 0.40 was 3 days before
- Author: DWHEELER
-
Unicode::UTF8 - Encoding and decoding of UTF-8 encoding form
- Version: 0.63 on 2025-12-20, with 20 votes
- Previous CPAN version: 0.62 was 8 years, 8 months, 9 days before
- Author: CHANSEN
-
Zonemaster::Backend - A system for running Zonemaster tests asynchronously through an RPC-API
- Version: 12.0.0 on 2025-12-19, with 16 votes
- Previous CPAN version: 11.5.0 was 5 months, 22 days before
- Author: ZNMSTR
-
Zonemaster::Engine::Exception::NormalExit - run Zonemaster tests from the command line
- Version: 8.000001 on 2025-12-19, with 23 votes
- Previous CPAN version: 8.000000 was 5 months, 22 days before
- Author: ZNMSTR
-
Zonemaster::Engine - A tool to check the quality of a DNS zone
- Version: 8.001000 on 2025-12-19, with 35 votes
- Previous CPAN version: 8.000000 was 5 months, 22 days before
- Author: ZNMSTR
This is the weekly favourites list of CPAN distributions. Votes count: 43
Week's winner: MCP (+3)
Build date: 2025/12/21 13:03:54 GMT
Clicked for first time:
- App::BlurFill - Blurred background fill image processor
- Complete::Getopt::Long - Complete command-line argument using Getopt::Long specification
- Data::Turtle - Turtle Movement and State Operations
- Marlin - ð pretty fast class builder with most Moo/Moose features ð
- Mojo::Collection::XS - Fast XS subclass of Mojo::Collection with XS-based while
- SimpleFlow - easy, simple workflow manager (and logger); for keeping track of and debugging large and complex shell command workflows
Increasing its reputation:
- Affix (+1=4)
- App::shcompgen (+1=3)
- Complete::Bash (+1=5)
- Complete::Util (+1=2)
- Const::Fast (+1=37)
- DateTime::Format::Strptime (+1=26)
- File::HomeDir (+1=35)
- File::XDG (+1=10)
- Getopt::Long::Complete (+1=15)
- Getopt::Long::More (+1=2)
- IPC::Run3 (+1=25)
- JQ::Lite (+1=7)
- JSON::Schema::Modern (+1=9)
- JSON::XS (+1=121)
- MCP (+3=7)
- Melian (+2=3)
- MooX::Singleton (+1=6)
- OpenGL (+1=14)
- OpenGL::Modern (+1=3)
- PAGI (+2=2)
- Path::Iterator::Rule (+1=26)
- Perl::Types (+1=2)
- Prima (+1=46)
- SDL3 (+2=2)
- sealed (+1=2)
- Storage::Abstract (+2=2)
- Sub::Throttler (+1=2)
- Test2::Plugin::SubtestFilter (+1=3)
- Text::Markup (+1=12)
- Thread::Subs (+2=2)
Weekly Challenge 352
Each week Mohammad S. Anwar sends out The Weekly Challenge, a chance for all of us to come up with solutions to two weekly tasks. My solutions are written in Python first, and then converted to Perl. It's a great way for us all to practice some coding.
Task 1: Match String
Task
You are given an array of strings.
Write a script to return all strings that are a substring of another word in the given array in the order they occur.
My solution
The Python solution is relatively straight forward. I have a loop that has i as the index and word as the word I are matching. I then check if any words in the words list are a substring (including a complete match) that are not at the same position. If there are, I append it to the matches list. I also do a check to ensure a word does not appear twice.
def match_string(words: list) -> list:
matches = []
for i, word in enumerate(words):
if word in matches:
continue
if any(word in other_word for j, other_word in enumerate(words) if i != j):
matches.append(word)
return matches
The Perl solution follows the same logic, but does it slightly differently. Perl has a (documented) quirk that calling each on a array (or hash) does not reset the counter when called a second time. I even made a blog post about the dangers of each in Perl. For the Perl solution, I use a foreach loop to find a match.
sub main (@words) {
my @other_words = @words;
my @matches = ();
while ( my ( $i, $word ) = each(@words) ) {
if ( any { $_ eq $word } @matches ) {
next;
}
my $j = 0;
foreach my $other_word (@other_words) {
if ( $i != $j and index( $other_word, $word ) != -1 ) {
push @matches, $word;
last;
}
$j++;
}
}
say '(' . join( ', ', map { "\"$_\"" } @matches ) . ')';
Examples
$ ./ch-1.py cat cats dog dogcat dogcat rat ratcatdogcat
['cat', 'dog', 'dogcat', 'rat']
$ ./ch-1.py hello hell world wor ellow elloworld
['hell', 'world', 'wor', 'ellow']
$ ./ch-1.py a aa aaa aaaa
['a', 'aa', 'aaa']
$ ./ch-1.py flower flow flight fl fli ig ght
['flow', 'fl', 'fli', 'ig', 'ght']
$ ./ch-1.py car carpet carpenter pet enter pen pent
['car', 'pet', 'enter', 'pen', 'pent']
Task 2: Binary Prefix
Task
You are given an array, @nums, where each element is either 0 or 1.
Define xi as the number formed by taking the first i+1 bits of @nums (from $nums[0] to $nums[i]) and interpreting them as a binary number, with $nums[0] being the most significant bit.
Write a script to return an array @answer where $answer[i] is true if xi is divisible by 5, otherwise false.
My solution
This is the more straight forward of the two tasks. For this task, I have a variable called current_binary that starts with an empty string. For each iteration of nums, I append the value to the current_binary string. I then use int (Python) or oct (Perl) to convert the value to an integer. I then see if it is divisible by five, and append the boolean (string in Perl) to the results list.
def binary_prefix(nums: list) -> list:
if any(type(n) != int or n not in (0, 1) for n in nums):
raise ValueError("Input list must contain only 0s and 1s")
current_binary = ''
results = []
for n in nums:
current_binary += str(n)
results.append(int(current_binary, 2) % 5 == 0)
return results
Examples
$ ./ch-2.py 0 1 1 0 0 1 0 1 1 1
[True, False, False, False, False, True, True, False, False, False]
$ ./ch-2.py 1 0 1 0 1 0
[False, False, True, True, False, False]
$ ./ch-2.py 0 0 1 0 1
[True, True, False, False, True]
$ ./ch-2.py 1 1 1 1 1
[False, False, False, True, False]
$ ./ch-2.py 1 0 1 1 0 1 0 0 1 1
[False, False, True, False, False, True, True, True, False, False]
Task 1: Match String
The Task
You are given an array of strings. Write a script to return all strings that are a substring of another word in the given array in the order they occur.
-
Example 1:
-
Input:
@words = ("cat", "cats", "dog", "dogcat", "dogcat", "rat", "ratcatdogcat") -
Output:
("cat", "dog", "dogcat", "rat")
-
Input:
The Deep Thoughts
Example 1 implies a couple of constraints. A complete match counts as a substring. Words may be repeated in the input. Repeated words are not duplicated in the output. The output order must match the input order. Noted.
For each word, we'll need to take it out of the list, check for substrings, and then put it back for the next round. A relatively simple way of doing that would be to shift it off the front of the list, do what needs to be done, and then return it by pushing it onto the back.
I don't see much of a way of getting around the idea that every word must be compared to every other word, so we're looking at an O(n2) performance algorithm.
One possibility for reducing the operation count might be to compare a pair of words both ways. If I have $w1 and $w2 in hand, I can check both directions for substring-i-ness. That way, instead of having to loop over the entire NxN matrix, we would only have to loop over the upper or lower triangle of the matrix. It's fewer looping operations, but still the same number of string comparisons. But it would mean more complexity in keeping track of what's been checked, so, meh.
The Code
sub matchString(@words)
{
use List::Util qw/any/;
my %seen;
my @match;
for ( 1 .. @words )
{
my $w = shift @words;
push @match, $w if ( ! $seen{$w}++ && any { index($_, $w) >= 0 } @words );
push @words, $w;
}
return \@match;
}
Notes:
- Use a hash (
%seen) to eliminate duplicates, and coincidentally optimize a little by not searching multiple times. - The main loop is once per word, so simply count. Normally, I would prefer something obviously tied to the array (like
for (@words)), but since I'm changing the array within the loop, I don't want to have to think too hard about whether the iterator remains valid after removing and adding an element. -
indexis a cheaper operation than a regular expression match. Benchmarking bears this out -- it's about three times more efficient in my environment and test data. -
List::Util::anystops as soon as something works, so it's cheaper thangrep, which would evaluate the entire list every time.
Task 2: Binary Prefix
The Task
You are given an array, @nums, where each element is either 0 or 1.
Define xi as the number formed by taking the first i+1 bits of @nums (from $nums[0] to $nums[i]) and interpreting them as a binary number, with $nums[0] being the most significant bit.
For example: If @nums = (1, 0, 1), then:
x0 = 1 (binary 1)
x1 = 2 (binary 10)
x2 = 5 (binary 101)
For each i, check whether xi is divisible by 5.
Write a script to return an array @answer where $answer[i] is true if xi is divisible by 5, otherwise false.
-
Example 1:
- Input:
@nums = (0,1,1,0,0,1,0,1,1,1) - Output: (true, false, false, false, false, true, true, false, false, false)
- Input:
Binary numbers formed (decimal values):
0: 0 (false)
01: 1 (false)
011: 3 (false)
0110: 6 (false)
01100: 12 (false)
011001: 25 ( true)
0110010: 50 ( true)
01100101: 101 (false)
011001011: 203 (false)
0110010111: 407 (false)
The Deep Thoughts
Weird flex, but OK.
This puts me in mind of bit-twiddling with C code, but bit-twiddling in Perl is equally possible. The other alternative is probably to build strings and do binary number conversions by prefixing with '0b'.
I kind of want to do a list-based solution, using map to convert each 0 or 1 to a true or false, but I think that would be obfuscating a pretty simple loop.
The Code
sub bpre(@nums)
{
my @isFive;
my $b = 0;
while ( defined(my $bit = shift @nums) )
{
$b = ($b << 1) | $bit;
push @isFive, ( $b % 5 == 0 );
}
return \@isFive;
}
Notes:
- Destroy the input by shifting a 1 or 0 off the left until the list is gone.
- Do bit operations to form a new number.
- Build an answer array consisting of boolean values.
Musical Interlude
The solution to task 1 reminds me of Katy Perry's Hot N Cold -- "You're yes, then you're no. You're in, then you're out."
And for Task 2, Mambo Number 5 is oddly appropriate. "Take one step left and one step right, One to the front and one to the side. Clap your hands once and clap your hands twice, And if it looks like this, then you're doin' it right."
Foswiki 2.1.10 can now be downloaded - landing right before Christmas, a full year since the last version dropped. Please be advised that this release includes several security fixes that require your attention. We would like to express our gratitude to Evgeny Kopytin of Positive Technologies for conducting a thorough audit of Foswiki and providing a comprehensive vulnerability report. Despite adhering closely to our security procedures, we were unable to obtain a response from the CVE Assignment Team regarding the allocation of official CVE-IDs. It is for this reason that the new security alerts covered by the 2.1.10er release had to be documented with a "CVE-2025-Unassigned" tag, since no better option was available.
See the release notes for additional information.
Does the Perl world need another object-oriented programming framework?
To be honest, probably not.
But here’s why you might want to give Marlin a try anyway.
-
Most of your constructors and accessors will be implemented in XS and be really, really fast.
-
If you accept a few basic principles like “attributes should usually be read-only”, it can be really, really concise to declare a class and its attributes.
An example
use v5.20.0; use experimental qw(signatures); # Import useful constants, types, etc. use Marlin::Util -all, -lexical; use Types::Common -all, -lexical; package Person { use Marlin 'given_name!' => NonEmptyStr, 'family_name!' => NonEmptyStr, 'name_style' => { enum => [qw/western eastern/], default => 'western' }, 'full_name' => { is => lazy, builder => true }, 'birth_date?'; sub _build_full_name ( $self ) { return sprintf( '%s %s', uc($self->family_name), $self->given_name ) if $self->name_style eq 'eastern'; return sprintf( '%s %s', $self->given_name, $self->family_name ); } } package Payable { use Marlin::Role -requires => [ 'bank_details' ]; sub make_payment ( $self ) { ...; } } package Employee { use Marlin -extends => [ 'Person' ], -with => [ 'Payable' ], 'bank_details!' => HashRef, 'employee_id!' => Int, 'manager?' => { isa => 'Employee' }; } my $manager = Employee->new( given_name => 'Simon', family_name => 'Lee', name_style => 'eastern', employee_id => 1, bank_details => {}, ); my $staff = Employee->new( given_name => 'Lea', family_name => 'Simons', employee_id => 2, bank_details => {}, manager => $manager, ); printf( "%s's manager is %s.\n", $staff->full_name, $staff->manager->full_name, ) if $staff->has_manager;
Some things you might notice:
-
It supports most of the features of Moose… or most of the ones you actually use anyway.
-
Declaring an attribute is often as simple as listing it’s name on the
use Marlinline. -
It can be followed by some options, but if you’re happy with Marlin’s defaults (read-only attributes), it doesn’t need to be.
-
You can use the
!to quickly mark an attribute as required instead of the longer{ required => true }. -
You can use
?to request a predicate method instead of the longer{ predicate => true }.
Benchmarks
My initial benchmarking shows that Marlin is fast.
Constructors
Rate Tiny Plain Moo Moose Marlin Core Tiny 1317/s -- -2% -48% -53% -54% -72% Plain 1340/s 2% -- -47% -53% -53% -72% Moo 2527/s 92% 89% -- -11% -12% -47% Moose 2828/s 115% 111% 12% -- -2% -40% Marlin 2873/s 118% 114% 14% 2% -- -39% Core 4727/s 259% 253% 87% 67% 65% --
Only the new Perl core class keyword generates a constructor faster than Marlin’s. And it is significantly faster; there’s no denying that. However, object construction is only part of what you are likely to need.
Accessors
Rate Tiny Moose Plain Core Moo Marlin Tiny 17345/s -- -1% -3% -7% -36% -45% Moose 17602/s 1% -- -2% -6% -35% -44% Plain 17893/s 3% 2% -- -4% -34% -44% Core 18732/s 8% 6% 5% -- -31% -41% Moo 27226/s 57% 55% 52% 45% -- -14% Marlin 31688/s 83% 80% 77% 69% 16% --
By accessors, I’m talking about not just standard getter and setters, but also predicate methods and clearers. Marlin and Moo both use Class::XSAccessor when possible, giving them a significant lead over the others. Marlin uses some sneaky tricks to squeeze out a little bit of extra performance by creating aliases for parent class methods directly in the child class symbol tables, allowing Perl to bypass a lot of the normal method resolution stuff.
I really expected class to do a lot better than it does. Its readers and writers are basically implemented in pure Perl currently, though I guess there’s scope to improve them in future releases.
Native Traits / Handles Via / Delegations
Rate Tiny Core Plain Moose Moo Marlin Tiny 675/s -- -56% -57% -59% -61% -61% Core 1518/s 125% -- -4% -8% -13% -13% Plain 1581/s 134% 4% -- -4% -9% -10% Moose 1642/s 143% 8% 4% -- -5% -6% Moo 1736/s 157% 14% 10% 6% -- -1% Marlin 1752/s 160% 15% 11% 7% 1% --
If you don’t know what I mean by native traits, it’s the ability to create small methods like this:
sub add_language ( $self, $lang ) { push $self->languages->@*, $lang; }
As part of the attribute definition:
use Marlin languages => { is => 'ro', isa => ArrayRef[Str], default => [], handles_via => 'Array', handles => { add_language => 'push', count_languages => 'count' }, };
There’s not an awful lot of difference between the performance of most of these, but Marlin slightly wins. Marlin and Moose are also the only frameworks that include this out of the box without needing extension modules.
By the way, that default => [] was not a typo. You can set an empty arrayref or empty hashref as a default, and Marlin will assume you meant something like default => sub { [] }, but it cleverly skips over needing to actually call the coderef (slow), instead creating a reference to a new empty array in XS (fast)!
Combined
Rate Tiny Plain Core Moose Moo Marlin Tiny 545/s -- -48% -56% -58% -60% -64% Plain 1051/s 93% -- -16% -19% -22% -31% Core 1249/s 129% 19% -- -4% -8% -18% Moose 1304/s 139% 24% 4% -- -4% -14% Moo 1355/s 148% 29% 8% 4% -- -11% Marlin 1519/s 179% 45% 22% 17% 12% --
A realistic bit of code that constructs some objects and calls a bunch of accessors and delegations on them. Marlin performs very well.
Lexical accessors and private attributes
Marlin has first class support for lexical methods!
use v5.42.0; package Widget { use Marlin name => { isa => Str }, internal_id => { reader => 'my internal_id', storage => 'PRIVATE' }; ... printf "%d: %s\n", $w->&internal_id, $w->name, } # dies because internal_id is lexically scoped Widget->new->&internal_id;
Support for the ->& operator was added in Perl 5.42. On older Perls (from Perl 5.12 onwards), lexical methods are still supported but you need to use function call syntax (internal_id($w)).
The storage => "PRIVATE" hint tells Marlin to use inside-out storage for that attribute, meaning that trying to access the internal_id by poking into the object’s internals ($obj->{internal_id}) won’t work.
This gives you true private attributes.
On Perl 5.18 and above, you can of course declare lexical methods using the normal my sub foo syntax, so you have private attributes as well as private methods.
Constant attributes
package Person { use Marlin name => { isa => Str, required => true }, species_name => { isa => Str, constant => "Homo sapiens" }; }
Constant attributes are declared like regular attributes, but are always very read-only and illegal to pass to the constructor.
Like other attributes, they support delegations, provided the delegated method isn’t one which could change the value.
Perl version support
Although some of the lexical features need newer versions of Perl, Marlin runs on Perl versions as old as 5.8.8.
Future directions
Some ideas I’ve had:
- If Moose is loaded, create meta object protocol stuff for Marlin classes and roles, like Moo does.
Does the Perl world need another object-oriented programming framework?
To be honest, probably not.
But here’s why you might want to give Marlin a try anyway.
Most of your constructors and accessors will be implemented in XS and be really, really fast.
If you accept a few basic principles like “attributes should usually be read-only”, it can be really, really concise to declare a class and its attributes.
An example
use v5.20.0;
use experimental qw(signatures);
# Import useful constants, types, etc.
use Marlin::Util -all, -lexical;
use Types::Common -all, -lexical;
package Person {
use Marlin
'given_name!' => NonEmptyStr,
'family_name!' => NonEmptyStr,
'name_style' => { enum => [qw/western eastern/], default => 'western' },
'full_name' => { is => lazy, builder => true },
'birth_date?';
sub _build_full_name ( $self ) {
return sprintf( '%s %s', uc($self->family_name), $self->given_name )
if $self->name_style eq 'eastern';
return sprintf( '%s %s', $self->given_name, $self->family_name );
}
}
package Payable {
use Marlin::Role
-requires => [ 'bank\_details' ];
sub make_payment ( $self ) {
...;
}
}
package Employee {
use Marlin
-extends => [ 'Person' ],
-with => [ 'Payable' ],
'bank_details!' => HashRef,
'employee_id!' => Int,
'manager?' => { isa => 'Employee' };
}
my $manager = Employee->new(
given_name => 'Simon',
family_name => 'Lee',
name_style => 'eastern',
employee_id => 1,
bank_details => {},
);
my $staff = Employee->new(
given_name => 'Lea',
family_name => 'Simons',
employee_id => 2,
bank_details => {},
manager => $manager,
);
printf(
"%s's manager is %s.\n",
$staff->full_name,
$staff->manager->full_name,
) if $staff->has_manager;
Some things you might notice:
It supports most of the features of Moose… or most of the ones you actually use anyway.
Declaring an attribute is often as simple as listing it’s name on the
use Marlinline.It can be followed by some options, but if you’re happy with Marlin’s defaults (read-only attributes), it doesn’t need to be.
You can use the
!to quickly mark an attribute as required instead of the longer{ required => true }.You can use
?to request a predicate method instead of the longer{ predicate => true }.
Benchmarks
My initial benchmarking shows that Marlin is fast.
Constructors
Rate Tiny Plain Moo Moose Marlin Core
Tiny 1317/s -- -2% -48% -53% -54% -72%
Plain 1340/s 2% -- -47% -53% -53% -72%
Moo 2527/s 92% 89% -- -11% -12% -47%
Moose 2828/s 115% 111% 12% -- -2% -40%
Marlin 2873/s 118% 114% 14% 2% -- -39%
Core 4727/s 259% 253% 87% 67% 65% --
Only the new Perl core class keyword generates a constructor faster than Marlin’s. And it is significantly faster; there’s no denying that. However, object construction is only part of what you are likely to need.
Accessors
Rate Tiny Moose Plain Core Moo Marlin
Tiny 17345/s -- -1% -3% -7% -36% -45%
Moose 17602/s 1% -- -2% -6% -35% -44%
Plain 17893/s 3% 2% -- -4% -34% -44%
Core 18732/s 8% 6% 5% -- -31% -41%
Moo 27226/s 57% 55% 52% 45% -- -14%
Marlin 31688/s 83% 80% 77% 69% 16% --
By accessors, I’m talking about not just standard getter and setters, but also predicate methods and clearers. Marlin and Moo both use Class::XSAccessor when possible, giving them a significant lead over the others. Marlin uses some sneaky tricks to squeeze out a little bit of extra performance by creating aliases for parent class methods directly in the child class symbol tables, allowing Perl to bypass a lot of the normal method resolution stuff.
I really expected class to do a lot better than it does. Its readers and writers are basically implemented in pure Perl currently, though I guess there’s scope to improve them in future releases.
Native Traits / Handles Via / Delegations
Rate Tiny Core Plain Moose Moo Marlin
Tiny 675/s -- -56% -57% -59% -61% -61%
Core 1518/s 125% -- -4% -8% -13% -13%
Plain 1581/s 134% 4% -- -4% -9% -10%
Moose 1642/s 143% 8% 4% -- -5% -6%
Moo 1736/s 157% 14% 10% 6% -- -1%
Marlin 1752/s 160% 15% 11% 7% 1% --
If you don’t know what I mean by native traits, it’s the ability to create small methods like this:
sub add_language ( $self, $lang ) {
push $self->languages->@*, $lang;
}
As part of the attribute definition:
use Marlin
languages => {
is => 'ro',
isa => ArrayRef[Str],
default => [],
handles_via => 'Array',
handles => { add_language => 'push', count_languages => 'count' },
};
There’s not an awful lot of difference between the performance of most of these, but Marlin slightly wins. Marlin and Moose are also the only frameworks that include this out of the box without needing extension modules.
By the way, that default => [] was not a typo. You can set an empty arrayref or empty hashref as a default, and Marlin will assume you meant something like default => sub { [] }, but it cleverly skips over needing to actually call the coderef (slow), instead creating a reference to a new empty array in XS (fast)!
Combined
Rate Tiny Plain Core Moose Moo Marlin
Tiny 545/s -- -48% -56% -58% -60% -64%
Plain 1051/s 93% -- -16% -19% -22% -31%
Core 1249/s 129% 19% -- -4% -8% -18%
Moose 1304/s 139% 24% 4% -- -4% -14%
Moo 1355/s 148% 29% 8% 4% -- -11%
Marlin 1519/s 179% 45% 22% 17% 12% --
A realistic bit of code that constructs some objects and calls a bunch of accessors and delegations on them. Marlin performs very well.
Lexical accessors and private attributes
Marlin has first class support for lexical methods!
use v5.42.0;
package Widget {
use Marlin
name => { isa => Str },
internal_id => { reader => 'my internal_id', storage => 'PRIVATE' };
...
printf "%d: %s\n", $w->&internal_id, $w->name,
}
# dies because internal_id is lexically scoped
Widget->new->&internal_id;
Support for the ->& operator was added in Perl 5.42. On older Perls (from Perl 5.12 onwards), lexical methods are still supported but you need to use function call syntax (internal_id($w)).
The storage => "PRIVATE" hint tells Marlin to use inside-out storage for that attribute, meaning that trying to access the internal_id by poking into the object’s internals ($obj->{internal_id}) won’t work.
This gives you true private attributes.
On Perl 5.18 and above, you can of course declare lexical methods using the normal my sub foo syntax, so you have private attributes as well as private methods.
Constant attributes
package Person {
use Marlin
name => { isa => Str, required => true },
species_name => { isa => Str, constant => "Homo sapiens" };
}
Constant attributes are declared like regular attributes, but are always very read-only and illegal to pass to the constructor.
Like other attributes, they support delegations, provided the delegated method isn’t one which could change the value.
Perl version support
Although some of the lexical features need newer versions of Perl, Marlin runs on Perl versions as old as 5.8.8.
Future directions
Some ideas I’ve had:
- If Moose is loaded, create meta object protocol stuff for Marlin classes and roles, like Moo does.
When trying to match the "randomart" part of ssh-keygen's output using Expect, expect() finds an empty string after the "randomart", but not the expected eof, but I don't understand why.
I'm using this code fragment:
# other values are:
# DB<2> x CMD_SSH_KEYGEN, @params
#0 'ssh-keygen'
#1 '-a'
#2 123
#3 '-b'
#4 4096
#5 '-f'
#6 'keys/host-aa71f91580'
#7 '-t'
#8 'rsa'
#9 '-C'
#10 'testXXX'
#11 '-N'
#12 'Rssg2(pm)j'
if (defined(my $exp = Expect->spawn(CMD_SSH_KEYGEN, @params))) {
my $timeout = 30;
my $CRLF = '\r?\n';
$exp->exp_internal(1); # for debugging
$exp->log_stdout(0);
my ($match_pos, $err, $match, $before, $after) = $exp->expect(
$timeout,
# ... matches for other parts
[qr/^The key's randomart image is:$CRLF/,
sub ($$) {
my ($exp, $tag) = @_;
my $r;
while (($r = $exp->expect(
$timeout,
'-re', qr/^[|+](.+?)[|+]$CRLF/,
'eof')) && $r == 1) {
if (my $m = $exp->match()) {
$m =~ s/$CRLF\$//;
print "match($tag): ", $m, "\n";
}
}
return exp_continue_timeout;
},
'randomart'],
[qr/^.*?$CRLF/,
sub ($$) {
my ($exp, $tag) = @_;
$exp = $exp->match(); # ugly, sorry!
$exp =~ s/$CRLF\$//;
print "match($tag): ", $exp, "\n";
return exp_continue_timeout;
},
'default'],
# ... more matches
);
When debugging I see:
#...
spawn id(7): list of patterns:
#1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'
#2: -ex `eof'
spawn id(7): Does `+----[SHA256]-----+\r\n'
match:
pattern #1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'? YES!!
Before match string: `'
Match string: `+----[SHA256]-----+\r\n'
After match string: `'
Matchlist: (`----[SHA256]-----')
\atch(randomart): "+----[SHA256]-----+\
"
#...
spawn id(7): list of patterns:
#1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'
#2: -ex `eof'
spawn id(7): Does `'
match:
pattern #1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'? No.
pattern #2: -ex `eof'? No.
Continuing expect...
^Z
I pressed ^Z when expect was waiting for timeout. So the obvious problem is the empty string when I expected "eof"; what's wrong?
(The other thing is that "match(randomart):..." got mangled to "\atch(randomart):..." for some reason.)
When I redirect the command output to a file, then +----[SHA256]-----+ is the last line.
So maybe when testing use this output:
Generating public/private rsa key pair.
Your identification has been saved in keys/host-a03a9693b2
Your public key has been saved in keys/host-a03a9693b2.pub
The key fingerprint is:
SHA256:d85NzqB1gvlyxUjP27q4KlzqMRklpb2zGbqwLlbezr4 testXXX
The key's randomart image is:
+---[RSA 4096]----+
| . |
| + |
| o o . |
| o = = |
| S B * B |
| . =.% X o |
| o.o=o= = = .|
| o .+=+ o . . |
| . oo+Eo..o.o. |
+----[SHA256]-----+
Experiment
When I added the empty string to the list of patterns, then the wait for timeout does not happen, but still I don't understand why eof does not match instead:
...
spawn id(7): list of patterns:
#1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'
#2: -ex `'
#3: -ex `eof'
spawn id(7): Does `'
match:
pattern #1: -re `(?^:^[|+](.+?)[|+]\\r?\\n)'? No.
pattern #2: -ex `'? YES!!
Before match string: `'
Match string: `'
After match string: `'
Matchlist: ()
Continuing expect...
...
Software versions used are: perl 5.26.1, Expect.pm 1.35
I get a "Prototype mismatch" warning in code that either imports 'blessed' from Scalar::Util or defines it depending on the version of Scalar::Util. Is there a way to suppress the warning, or do I need to turn off signatures and add a prototype to my own blessed() sub as shown below? Am I correct that the latter is only a good practice if Scalar::Util::blessed will have a prototype forever? (I don't understand why it has a prototype now.)
use strict; use warnings;
use feature qw( unicode_strings signatures );
no warnings 'experimental::signatures';
use utf8;
use version;
require Scalar::Util;
if ( version->parse( Scalar::Util->VERSION ) >= version->parse( '1.53' ) ) {
STDERR->print( "Using Scalar::Util::blessed()\n" );
Scalar::Util->import( 'blessed' );
}
else {
STDERR->print( "Using our own blessed()\n" );
no feature 'signatures';
sub blessed($) {
# Workaround for Scalar-List-Utils bug #124515 fixed in 1.53 (Perl v5.31.6)
my $class = Scalar::Util::blessed($_[0]);
utf8::decode($class) if defined($class) && ! utf8::is_utf8($class);
return $class;
}
}
sub new ( $class, %arg ) {
return bless( { CODE => '', %arg }, $class );
}
In my previous post, in February, I announced the overhaul of the MailBox software. The MailBox suite of distributions implement automatic email handling processes. I started development back in 1999, so it had aged a bit. And I can now proudly tell you that the work has been completed!
As you may have experienced yourself: software ages. It's not directly that it does not work anymore, however your own opinion about programming, the features of the language and libraries you use, and the source specifications keep on changing. Basic maintenance picks some of the low-hanging fruits as refreshment, but you usually stay away from major rewrites. Well, the marvelous NLnet Foundation helped me to realize just that!
Some of the changes:
- Supported Perl went from 5.8.5 to 5.16 (2015), to get some nice syntax features like
`//` and `s///r` - OODoc improvements make the documentation even better manageable
- Major updates on the documentation text
- New HTML version of the generated docs, purely using client-side rendering
- Code simplifications and less line folding (80 -> 132 chars)
- Some code-style preference changes
- replaced the error handling into real exceptions
The latter change is backwards compatibility breaking: Major version 3 will be maintenance releases for the old error handling. Major version 4 releases will be the main branch for maintenance and development with the new system.
Gladly, the software had over 15.000 regression tests. Also, cpantesters is so incredibly useful, with its active disciples Slaven and Andreas! Thanks so much, guys! However, I do expect some fall-out, especially because exceptions are produced at error conditions, and errors rarely occur. Hence, the code path of errors is rarely used to tend to contain the most issues.
As module author, I am always glad to receive bug-reports; it's much better to have bugs (even tiny typos) fixed, than have them linger around. Help authors by reporting issues!
For exception infra, I picked (my own) Log::Report, which is extremely powerful. I should post a few examples on this medium soon. It integrates nicely with many other exception frameworks, but also opens the path to translation.
Many, many thanks to NLnet and cpantesters.
A working link for Tom Christiansen's slides on "Unicode, The Good, the Bad, and the (mostly) Ugly" is at https://web.archive.org/web/20121224081332/http://98.245.80.27/tcpc/OSCON2011/gbu.html. (We are writing a book on debugging at home, and I needed a usable link to Tom's talk.)
A simple proxy in Perl runs as a CGI on the webserver of my ISP. It's purpose is to forward https GET and POST to a http webserver running on my PC. With that https works without the need of any TLS certificate and domain on my PC.
It generally works, but the original headers can't be forwarded, because I have not found a method accessing them. I can only get them via %ENV, but then "Content-Type" becomes "CONTENT_TYPE", and that of course isn't recognized by the PC webserver.
Is there a way to access the original headers? I don't want to retranslate them to the original ones. There should be an easier way, but I found nothing on the web.
We’ve just published a new Perl School book: Design Patterns in Modern Perl by Mohammad Sajid Anwar.
It’s been a while since we last released a new title, and in the meantime, the world of eBooks has moved on – Amazon don’t use .mobi any more, tools have changed, and my old “it mostly works if you squint” build pipeline was starting to creak.
On top of that, we had a hard deadline: we wanted the book ready in time for the London Perl Workshop. As the date loomed, last-minute fixes and manual tweaks became more and more terrifying. We really needed a reliable, reproducible way to go from manuscript to “good quality PDF + EPUB” every time.
So over the last couple of weeks, I’ve been rebuilding the Perl School book pipeline from the ground up. This post is the story of that process, the tools I ended up using, and how you can steal it for your own books.
The old world, and why it wasn’t good enough
The original Perl School pipeline dates back to a very different era:
-
Amazon wanted
.mobifiles. -
EPUB support was patchy.
-
I was happy to glue things together with shell scripts and hope for the best.
It worked… until it didn’t. Each book had slightly different scripts, slightly different assumptions, and a slightly different set of last-minute manual tweaks. It certainly wasn’t something I’d hand to a new author and say, “trust this”.
Coming back to it for Design Patterns in Modern Perl made that painfully obvious. The book itself is modern and well-structured; the pipeline that produced it shouldn’t feel like a relic.
Choosing tools: Pandoc and wkhtmltopdf (and no LaTeX, thanks)
The new pipeline is built around two main tools:
-
Pandoc – the Swiss Army knife of document conversion. It can take Markdown/Markua plus metadata and produce HTML, EPUB, and much, much more.
-
wkhtmltopdf– which turns HTML into a print-ready PDF using a headless browser engine.
Why not LaTeX? Because I’m allergic. LaTeX is enormously powerful, but every time I’ve tried to use it seriously, I end up debugging page breaks in a language I don’t enjoy. HTML + CSS I can live with; browsers I can reason about. So the PDF route is:
- Markdown → HTML (via Pandoc) → PDF (via
wkhtmltopdf)
And the EPUB route is:
- Markdown → EPUB (via Pandoc) → validated with
epubcheck
The front matter (cover page, title page, copyright, etc.) is generated with Template Toolkit from a simple book-metadata.yml file, and then stitched together with the chapters to produce a nice, consistent book.
That got us a long way… but then a reader found a bug.
The iBooks bug report
Shortly after publication, I got an email from a reader who’d bought the Leanpub EPUB and was reading it in Apple Books (iBooks). Instead of happily flipping through Design Patterns in Modern Perl, they were greeted with a big pink error box.
Apple’s error message boiled down to:
There’s something wrong with the XHTML in this EPUB.
That was slightly worrying. But, hey, every day is a learning opportunity. And, after a bit of digging, this is what I found out.
EPUB 3 files are essentially a ZIP containing:
-
XHTML content files
-
a bit of XML metadata
-
CSS, images, and so on
Apple Books is quite strict about the “X” in XHTML: it expects well-formed XML, not just “kind of valid HTML”. So when working with EPUB, you need to forget all of that nice HTML5 flexibility that you’ve got used to over the last decade or so.
The first job was to see if we could reproduce the error and work out where it was coming from.
Discovering epubcheck
Enter epubcheck.
epubcheck is the reference validator for EPUB files. Point it at an .epub and it will unpack it, parse all the XML/XHTML, check the metadata and manifest, and tell you exactly what’s wrong.
Running it on the book immediately produced this:
Fatal Error while parsing file: The element type
brmust be terminated by the matching end-tag</br>.
That’s the XML parser’s way of saying:
-
In HTML,
<br>is fine. -
In XHTML (which is XML), you must use
<br />(self-closing) or<br></br>.
And there were a number of these scattered across a few chapters.
In other words: perfectly reasonable raw HTML in the manuscript had been passed straight through by Pandoc into the EPUB, but that HTML was not strictly valid XHTML, so Apple Books rejected it. I should note at this point that the documentation for Pandoc’s EPUB creation explicitly says that it won’t touch HTML fragments it finds in a Markdown file when converting it to EPUB. It’s down to the author to ensure they’re using valid XHTML
A quick (but not scalable) fix
Under time pressure, the quickest way to confirm the diagnosis was:
-
Unzip the generated EPUB.
-
Open the offending XHTML file.
-
Manually turn
<br>into<br />in a couple of places. -
Re-zip the EPUB.
-
Run
epubcheckagain. -
Try it in Apple Books.
That worked. The errors vanished, epubcheck was happy, and the reader confirmed that the fixed file opened fine in iBooks.
But clearly:
Open the EPUB in a text editor and fix the XHTML by hand
is not a sustainable publishing strategy.
So the next step was to move from “hacky manual fix” to “the pipeline prevents this from happening again”.
HTML vs XHTML, and why linters matter
The underlying issue is straightforward once you remember it:
-
HTML is very forgiving. Browsers will happily fix up all kinds of broken markup.
-
XHTML is XML, so it’s not forgiving:
-
empty elements must be self-closed (
<br />,<img />,<hr />, etc.), -
tags must be properly nested and balanced,
-
attributes must be quoted.
-
EPUB 3 content files are XHTML. If you feed them sloppy HTML, some readers (like Apple Books) will just refuse to load the chapter.
So I added a manuscript HTML linter to the toolchain, before we ever get to Pandoc or epubcheck.
Roughly, the linter:
-
Reads the manuscript (ignoring fenced code blocks so it doesn’t complain about
<in Perl examples). -
Extracts any raw HTML chunks.
-
Wraps those chunks in a temporary root element.
-
Uses
XML::LibXMLto check they’re well-formed XML. -
Reports any errors with file and line number.
It’s not trying to be a full HTML validator; it’s just checking: “If this HTML ends up in an EPUB, will the XML parser choke?”
That would have caught the <br> problem before the book ever left my machine.
Hardening the pipeline: epubcheck in the loop
The linter catches the obvious issues in the manuscript; epubcheck is still the final authority on the finished EPUB.
So the pipeline now looks like this:
-
Lint the manuscript HTML
Catch broken raw HTML/XHTML before conversion. -
Build PDF + EPUB via
make_book-
Generate front matter from metadata (cover, title pages, copyright).
-
Turn Markdown + front matter into HTML.
-
Use
wkhtmltopdffor a print-ready PDF. -
Use Pandoc for the EPUB.
-
-
Run
epubcheckon the EPUB
Ensure the final file is standards-compliant. -
Only then do we upload it to Leanpub and Amazon, making it available to eager readers.
The nice side-effect of this is that any future changes (new CSS, new template, different metadata) still go through the same gauntlet. If something breaks, the pipeline shouts at me long before a reader has to.
Docker and GitHub Actions: making it reproducible
Having a nice Perl script and a list of tools installed on my laptop is fine for a solo project; it’s not great if:
-
other authors might want to build their own drafts, or
-
I want the build to happen automatically in CI.
So the next step was to package everything into a Docker image and wire it into GitHub Actions.
The Docker image is based on a slim Ubuntu and includes:
-
Perl +
cpanm+ all CPAN modules from the repo’scpanfile -
pandoc -
wkhtmltopdf -
Java +
epubcheck -
The Perl School utility scripts themselves (
make_book,check_ms_html, etc.)
The workflow in a book repo is simple:
-
Mount the book’s Git repo into
/work. -
Run
check_ms_htmlto lint the manuscript. -
Run
make_bookto buildbuilt/*.pdfandbuilt/*.epub. -
Run
epubcheckon the EPUB. -
Upload the
built/artefacts.
GitHub Actions then uses that same image as a container for the job, so every push or pull request can build the book in a clean, consistent environment, without needing each author to install Pandoc, wkhtmltopdf, Java, and a large chunk of CPAN locally.
Why I’m making this public
At this point, the pipeline feels:
-
modern (Pandoc, HTML/CSS layout, EPUB 3),
-
robust (lint +
epubcheck), -
reproducible (Docker + Actions),
-
and not tied to Perl in any deep way.
Yes, Design Patterns in Modern Perl is a Perl book, and the utilities live under the “Perl School” banner, but nothing is stopping you from using the same setup for your own book on whatever topic you care about.
So I’ve made the utilities available in a public repository (the perlschool-util repo on GitHub). There you’ll find:
-
the build scripts,
-
the Dockerfile and helper script,
-
example GitHub Actions configuration,
-
and notes on how to structure a book repo.
If you’ve ever thought:
I’d like to write a small technical book, but I don’t want to fight with LaTeX or invent a build system from scratch…
then you’re very much the person I had in mind.
eBook publishing really is pretty easy once you’ve got a solid pipeline. If these tools help you get your ideas out into the world, that’s a win.
And, of course, if you’d like to write a book for Perl School, I’m still very interested in talking to potential authors – especially if you’re doing interesting modern Perl in the real world.
The post Behind the scenes at Perl School Publishing first appeared on Perl Hacks.
-
App::Greple - extensible grep with lexical expression and region handling
- Version: 10.00 on 2025-12-11, with 56 votes
- Previous CPAN version: 9.23 was 7 months, 4 days before
- Author: UTASHIRO
-
App::Netdisco - An open source web-based network management tool.
- Version: 2.096001 on 2025-12-13, with 804 votes
- Previous CPAN version: 2.096000 was 5 days before
- Author: OLIVER
-
Beam::Wire - Lightweight Dependency Injection Container
- Version: 1.027 on 2025-12-06, with 18 votes
- Previous CPAN version: 1.026 was 1 year, 2 months, 23 days before
- Author: PREACTION
-
Bitcoin::Crypto - Bitcoin cryptography in Perl
- Version: 4.003 on 2025-12-11, with 14 votes
- Previous CPAN version: 4.002 was 27 days before
- Author: BRTASTIC
-
CPANSA::DB - the CPAN Security Advisory data as a Perl data structure, mostly for CPAN::Audit
- Version: 20251207.001 on 2025-12-07, with 25 votes
- Previous CPAN version: 20251130.001 was 6 days before
- Author: BRIANDFOY
-
DateTime::Format::Strptime - Parse and format strp and strf time patterns
- Version: 1.80 on 2025-12-06, with 25 votes
- Previous CPAN version: 1.79 was 4 years, 7 months, 3 days before
- Author: DROLSKY
-
DateTime::TimeZone - Time zone object base class and factory
- Version: 2.66 on 2025-12-11, with 22 votes
- Previous CPAN version: 2.65 was 8 months, 15 days before
- Author: DROLSKY
-
DBIx::Class::DeploymentHandler - Extensible DBIx::Class deployment
- Version: 0.002235 on 2025-12-12, with 21 votes
- Previous CPAN version: 0.002234 was 1 year, 4 months, 26 days before
- Author: WESM
-
Exporter::Tiny - an exporter with the features of Sub::Exporter but only core dependencies
- Version: 1.006003 on 2025-12-07, with 24 votes
- Previous CPAN version: 1.006002 was 2 years, 8 months, 6 days before
- Author: TOBYINK
-
JSON::Schema::Modern - Validate data against a schema using a JSON Schema
- Version: 0.629 on 2025-12-12, with 16 votes
- Previous CPAN version: 0.628 was 5 days before
- Author: ETHER
-
Mail::Box - complete E-mail handling suite
- Version: 4.01 on 2025-12-13, with 16 votes
- Previous CPAN version: 4.00 was 1 day before
- Author: MARKOV
-
Module::Release - Automate software releases
- Version: 2.137 on 2025-12-12, with 12 votes
- Previous CPAN version: 2.136 was 11 months, 8 days before
- Author: BRIANDFOY
-
Number::Phone - base class for Number::Phone::* modules
- Version: 4.0009 on 2025-12-10, with 24 votes
- Previous CPAN version: 4.0008 was 2 months, 27 days before
- Author: DCANTRELL
-
Object::Pad - a simple syntax for lexical field-based objects
- Version: 0.823 on 2025-12-08, with 46 votes
- Previous CPAN version: 0.822 was 7 days before
- Author: PEVANS
-
Release::Checklist - A QA checklist for CPAN releases
- Version: 0.18 on 2025-12-09, with 16 votes
- Previous CPAN version: 0.17 was 2 years, 7 months, 9 days before
- Author: HMBRAND
-
Spreadsheet::Read - Meta-Wrapper for reading spreadsheet data
- Version: 0.94 on 2025-12-09, with 31 votes
- Previous CPAN version: 0.93 was 8 months, 22 days before
- Author: HMBRAND
-
SPVM - The SPVM Language
- Version: 0.990109 on 2025-12-08, with 36 votes
- Previous CPAN version: 0.990108 was 4 days before
- Author: KIMOTO
-
Test::Simple - Basic utilities for writing tests.
- Version: 1.302219 on 2025-12-09, with 199 votes
- Previous CPAN version: 1.302218 was before
- Author: EXODIST
-
WebService::Fastly - an interface to most facets of the [Fastly API](https://www.fastly.com/documentation/reference/api/).
- Version: 13.01 on 2025-12-09, with 18 votes
- Previous CPAN version: 13.00 was 1 month, 8 days before
- Author: FASTLY
Tony write:
``` In addition to the typical stream of small changes to review, Dave's second AST rebuild of ExtUtils::ParseXS arrived (#23883), and I spent several hours reviewing it.
In response to #23918 I worked on adding numeric comparison APIs, which are complicated by overloading, NaNs, SVs dual IV/NV implmentation, and of course by overloading. This includes some fixes for the existing sv_numeq() API. You can see the current state of this work in #23966.
[Hours] [Activity] 2025/11/03 Monday 0.37 #23886 review and approve 0.22 #23873 review other comments and follow-up 0.47 #23887 review, research and approve 1.72 #23890 review, testing 0.23 #23890 comment 0.08 #23891 review and approve 0.18 #23895 review and approve
0.67 #23896 review and comment
3.94
2025/11/04 Tuesday 0.57 coverity scan results, testing, comment on #23871 1.15 #23885 review and comment 1.03 #23871 testing per wolfsage’s example, work on a regression test and fix, testing, push to PR 23897 1.67 #21877 debugging, fix my understanding on PerlIO and the
code, testing
4.42
2025/11/05 Wednesday 0.70 #23897 fix non-taint perl, testing and update PR 0.58 #23896 recheck 1.50 #23885 comment 0.57 #21877 look into remaining test failure, find the cause
and workaround it
3.35
2025/11/06 Thursday 0.08 #23902 review and approve 0.08 #23898 review and approve 0.55 #23899 review and approve 0.97 #23901 review and approve 0.95 #23883 review
1.40 #23883 review up to Node::include
4.03
2025/11/10 Monday 1.60 #23795 review updates, comment 0.35 #23907 review, research and approve 1.07 #23908 review, research, comment (fixed while I worked)
0.63 #23883 continue review, comment
3.65
2025/11/11 Tuesday 0.57 #23908 review updates and approve 0.40 #23911 review, review history of associated ticket and approve 0.85 #23883 more review
1.37 #23883 more review
3.19
2025/11/12 Wednesday 0.73 #23913 review, research and approve 0.77 #23914 review, check for SvIsUV() usage on CPAN 0.83 #23910 testing, get some strange results 0.82 #23910 debugging, can’t reproduce in new builds
0.67 #23883 more review
3.82
2025/11/13 Thursday 0.73 #23918 review discussion and research 0.75 #23917 review and approve 0.23 #23919 review and approve 1.03 #23883 more review
1.27 #23883 more review
4.01
2025/11/17 Monday 1.13 testing, comments on new XS API list thread 0.97 #23923 review and approve 1.25 #23914 testing, comment, review 0.43 #23914 more review and approve
0.93 #23888 review, comments, some side discussion of 23921
4.71
2025/11/18 Tuesday 0.50 #23888 review updates, testing,approve 0.27 #23943 review and approve 0.52 #23883 more review
1.27 #23883 more review
2.56
2025/11/19 Wednesday 0.78 #23922 review and approve 1.08 #23918 work on new compare APIs 0.53 #23918 debugging 1.22 #23918 testing, cleanup
0.82 #23918 re-work documentation
4.43
2025/11/20 Thursday 2.50 #23918 work on sv_numcmp(), research, test code, testing, debugging 1.07 #23918 work out an issue, more testing, document sv_numcmp
variants
3.57
2025/11/24 Monday 0.08 #23819 review and approve 2.77 #23918 NULL tests and fix, test for NV/IV mishandling and fix 0.82 #23918 open #23956, start on le lt ge gt implementation
1.20 #23918 finish implementation, test code, testing
4.87
2025/11/25 Tuesday 0.67 #23885 review, comment 1.13 #23885 more review
1.03 #23918 some polish
2.83
2025/11/26 Wednesday 0.07 #23960 review and approve 2.07 #23885 review, research and comments 0.48 #23918 more polish, testing
1.60 #23918 finish polish, push for CI
4.22
2025/11/27 Thursday 0.58 #23918 check CI, add perldelta and push 0.58 check CI results and make PR 23966
0.48 comment on dist discussion on list
1.64
2025/11/28 Friday
0.18 #23918 fix a minor issue
0.18
Which I calculate is 59.42 hours.
Approximately 32 tickets were reviewed or worked on. ```
Paul writes:
A mix of things this month, though I didn't get much done in the final week because of preparations for my talk at LPW2025. A useful event though because a few ideas came out of discussions that I shall be looking at for core perl soon.
- 4 = Mentoring preparation for BooK + Eric on PPC 0014
- 4.5 = attributes-v2 branch
- https://github.com/Perl/perl5/pull/23923
- 3 = Experiments with refalias in signatures in XS::Parse::Sublike
- 4 = Support for signature named parameters in
meta - 3 = Experiments with lexical class constructor functions in Object::Pad.
- While this is a CPAN module and not directly core perl, it serves as the experimental base for what gets implemented in future versions of perl, so it is still of interest to core development.
- 1 = Other github code reviews
Total: 19.5 hours
My aim for December is to continue the attributes-v2 branch, and get
into a good position to implement perhaps the :abstract and
:lexical_new attributes on classes.
Dave writes:
Last month was relatively quiet.
I worked on a couple of bugs and did some final updates to my branch which rewrites perlxs.pod - which I intend to merge in the next few days.
Summary:
- 10:33 GH #16197 re eval stack unwinding
- 4:47 GH #18669 dereferencing result of ternary operator skips autovivification
- 2:06 make perl -Dx display lexical variable names
- 10:58 modernise perlxs.pod
Total:
- 28:24 TOTAL (HH::MM)
-
App::cpm - a fast CPAN module installer
- Version: 0.998002 on 2025-12-04, with 177 votes
- Previous CPAN version: 0.998001 was 21 days before
- Author: SKAJI
-
App::HTTPThis - Export the current directory over HTTP
- Version: 0.010 on 2025-12-04, with 24 votes
- Previous CPAN version: 0.009 was 2 years, 5 months, 12 days before
- Author: DAVECROSS
-
App::Netdisco - An open source web-based network management tool.
- Version: 2.095006 on 2025-11-30, with 800 votes
- Previous CPAN version: 2.095005
- Author: OLIVER
-
CPANSA::DB - the CPAN Security Advisory data as a Perl data structure, mostly for CPAN::Audit
- Version: 20251130.001 on 2025-11-30, with 25 votes
- Previous CPAN version: 20251123.001 was 7 days before
- Author: BRIANDFOY
-
JSON::Schema::Modern - Validate data against a schema using a JSON Schema
- Version: 0.627 on 2025-12-04, with 15 votes
- Previous CPAN version: 0.626 was 2 days before
- Author: ETHER
-
MetaCPAN::Client - A comprehensive, DWIM-featured client to the MetaCPAN API
- Version: 2.034000 on 2025-12-03, with 27 votes
- Previous CPAN version: 2.033000 was 1 year, 8 days before
- Author: MICKEY
-
Minion::Backend::mysql - MySQL backend
- Version: 1.007 on 2025-12-01, with 13 votes
- Previous CPAN version: 1.006 was 1 year, 6 months, 9 days before
- Author: PREACTION
-
Object::Pad - a simple syntax for lexical field-based objects
- Version: 0.822 on 2025-11-30, with 46 votes
- Previous CPAN version: 0.821 was 4 months, 18 days before
- Author: PEVANS
-
Sisimai - Mail Analyzing Interface for bounce mails.
- Version: v5.5.0 on 2025-12-05, with 81 votes
- Previous CPAN version: v5.4.1 was 3 months, 5 days before
- Author: AKXLIX
-
SPVM - The SPVM Language
- Version: 0.990108 on 2025-12-03, with 36 votes
- Previous CPAN version: 0.990107 was 15 days before
- Author: KIMOTO
-
Sys::Virt - libvirt Perl API
- Version: v11.10.0 on 2025-12-01, with 17 votes
- Previous CPAN version: v11.8.0 was 24 days before
- Author: DANBERR
-
Time::Moment - Represents a date and time of day with an offset from UTC
- Version: 0.46 on 2025-12-04, with 76 votes
- Previous CPAN version: 0.44 was 7 years, 6 months, 25 days before
- Author: CHANSEN
This is the weekly favourites list of CPAN distributions. Votes count: 72
Week's winner: JSON::Schema::Validate (+3)
Build date: 2025/12/06 16:48:33 GMT
Clicked for first time:
- Chess::Plisco - Representation of a chess position with move generator, legality checker etc.
- Dev::Util - Utilities useful in the development of perl programs
- Disk::SmartTools - Provide tools to work with disks via S.M.A.R.T.
- Dump::Krumo - Fancy, colorful, human readable dumps of your data
- Melian - Perl client to the Melian cache
Increasing its reputation:
- Algorithm::Diff (+1=24)
- AnyEvent (+1=167)
- App::perlimports (+1=22)
- Beekeeper (+1=3)
- BioPerl (+1=36)
- CGI::Compile (+1=2)
- CGI::Emulate::PSGI (+1=3)
- Crypt::SecretBuffer (+1=3)
- Data::Processor (+1=2)
- DBD::Oracle (+1=32)
- Devel::Examine::Subs (+1=4)
- Devel::NYTProf (+1=196)
- Dpkg (+1=4)
- Email::Filter (+1=2)
- Email::Simple (+1=23)
- GD (+1=32)
- Geo::Parser::Text (+1=3)
- Hash::Merge::Simple (+1=18)
- IO::K8s (+1=5)
- IO::Uring (+1=2)
- JSON::Schema::Modern (+2=8)
- JSON::Schema::Validate (+3=4)
- libapreq2 (+1=5)
- List::Gen (+1=24)
- Log::Any (+1=68)
- LWP::Protocol::https (+1=22)
- Math::BigInt (+1=14)
- MCP (+1=4)
- meta (+1=15)
- MIME::Base64 (+1=25)
- Module::Generic (+1=4)
- Mojolicious (+1=509)
- Mojolicious::Plugin::OpenAPI::Modern (+1=6)
- Moose (+1=334)
- MooX::TypeTiny (+1=11)
- Nice::Try (+1=11)
- OpenAPI::Modern (+2=4)
- OpenTelemetry (+1=5)
- Params::Validate::Strict (+1=2)
- perl (+1=440)
- Perl::Critic (+1=135)
- Perl::Critic::Community (+1=8)
- Perl::Tidy (+1=147)
- Safe (+1=14)
- Scalar::List::Utils (+1=184)
- Scope::Guard (+1=21)
- Sereal::Encoder (+1=25)
- signatures (+1=8)
- Software::License (+1=17)
- Sub::Quote (+1=11)
- Syntax::Keyword::Try (+1=47)
- Template::Nest (+1=2)
- Test2::Harness (+1=20)
- Test::Harness (+1=65)
- Test::Simple (+1=199)
- Text::CSV (+1=82)
- Text::Diff (+1=17)
- Text::PO (+2=2)
- Time::HiRes (+1=65)
- Time::Piece (+1=66)
- Types::JSONSchema (+2=2)
It’s December, and I should’ve posted this in early November, but I didn’t. I did other stuff. Now, though, I’m on a long plane flight, so I guess it’s time to write a bit of bloggery. (Did I really never write up 2024? Well, maybe later.)
Here’s what we watched for Horror Movie Month this year!
October 1: Heart Eyes (2025)
This is “what if a rom-com was also a slasher?” It was a bit uneven, and didn’t quite nail either part – understandable, but still, you hope for excellence, right? It was novel and fun enough that I’m glad to have watched it. It’s from Josh Ruben, who did some other stuff I also liked.
October 2: The Borderlands (2012)
Hey, this was surprisingly decent. Vatican-sponsored ghost hunters are investigating a haunting. Is it bunk or not? There was nothing particularly special about this movie, except that it was well done. Good cast, good pacing. It’s nice to find a new competent, enjoyable, unexpected movie like this!
October 3: Departing Seniors (2023)
There’s a murderer at a high school, and somebody can see the murders coming in visions. It wasn’t good. It wasn’t terrible, but it was retreading old material, and there were lots of little problems. Why was the school always so empty? If it’s nearly graduation day, why are people wearing big coats?
Watch It’s a Wonderful Knife instead, maybe.
October 5: V/H/S/Halloween
I think this was my favorite V/H/S movie so far. They’re all sort of uneven messes, but this one had the most fun I can remember. I especially enjoyed the weird framing story about the soft drink focus group. Also, Fun Size was fun and weird. More like this one, please, V/H/S people!
October 5: Bring Her Back (2025)
I saw this as “best new horror of 2025” a bunch of places. It was a very well made movie, well written, well-acted. It was a good film. It was also just so grim, for much of it. It was a movie full of desperation. Also, it had “people in authority do bad things to kids”, which I don’t like watching. Still, I’m glad I saw it, and yeah, it was good, but I think we usually are looking for something more fun in Horror Movie Month.
October 6: Presence (2024)
Remember in The Menu, how the chef is making this extremely high-technique food that is hard to criticize for any lack of technical merit, but which sparks no joy? That’s this movie. It’s a well-made haunted house movie, but I didn’t care about what happened to anybody in it.
October 7: Ick (2024)
This movie reminded me of Detention (2011) and that’s a good thing. Brandon Routh plays a guy who was a high school football star and is now sort of washed up. There’s a weird black fungus that grows all over everything, and has been there for decades, and nobody cares. It’s just there. Then, one day, things change. It was weird and fun and unexpected. It wasn’t as good as Detention, but I liked it a lot.
October 8: Somnium (2024)
Big change in gears here from the previous movie. This was a slow-paced, moody movie about a would-be actor in LA who gets a job at some kind of weird dream therapy place. It looks right out of an early 80s Cronenberg movie. She walks around the place at three in the morning, surrounded by sleeping people in wood-paneled rooms. I liked it! I also liked that it was very happy to have a simple moral.
October 9: The Collector (2009)
This was pitched as “Would you like to see a movie that’s kind of like Saw but has been mostly forgotten?” It wasn’t as smart as the original Saw trilogy, and went on way too long. It’s sort of “Home Alone, with adults, and lots of blood.”
October 10: Found Footage: The Making of the Patterson Project (2025)
This little indie movie was a mess, but I had fun. There’s a guy who wants to make an indie horror movie, and is not a skilled filmmaker. His crew have a lot of heart, but are sort of hopeless. They send a guy in a bigfoot costume out into the woods in hunting season. Also, the set might be haunted? I enjoyed it.
October 11: I Know What You Did Last Summer (2025)
This could’ve been a lot worse. If you have any fond memories of the original, go for it. Otherwise, eh.
October 11: [REC]⁴ Apocalypse
Gloria and I had seen the previous REC movies, but I don’t think we’d seen this one. It was good! With Satanic zombies back on the mainland, a bunch of people are aboard a ship trying to study the problem. So: people in an enclosed space, at sea, with a zombie out break. You can imagine the rest, mostly. It brought back the newscaster from the first two movies, which was fun. (The original REC is definitely worth watching!)
October 13: How to Kill Monsters (2024)
A police station is taken over by monsters and demons and the riff raff inside have to save themselves. There’s a fun framing device where the movie starts at the end of a horrible bloodbath and then you’re sort of figuring out how it all fits together. Like a bunch of other movies on this list, it was enjoyable, but not great.
October 14: Psycho Therapy: The Shallow Tale of a Writer Who Decided to Write About a Serial Killer (2024)
Every year we end up with one or two movies that we thought were going to be sort of horror-y, or maybe just creepy thrillers, but don’t work out that way. This was one, this year, and no regrets! Steve Buscemi plays a retired serial killer who approaches a struggling writer to pitch a collaboration on memoirs. But after a little bit of farce, he’s roped into pretending to be a marriage councelor for the writer and his wife. Really weird, and who doesn’t want to see more Steve Buscemi?
October 17: Vicious (2025)
Polygon suggested this would be a great pick if you liked Weapons. It wasn’t. I guess I was supposed to feel sympathy and horror for the protagonist, but I didn’t care. It was not compelling or scary.
October 18: Dark Match (2024)
Another “flawed but worth it” entry! A group of near-nobodies in the local professional wrestling circuit book a high-paying gig at a private party in the woods. Could it be that the private party is actually a horrible, horrible place to end up? Yes, it could.
Special note: I kept thinking, “Who is this actor?” Turns out he played Bill in GTA V, and I recognized his face from his likeness in a video game. Woah!
October 19: Together
This was another “best of the year” candidate, said the Internet. It wasn’t my favorite, but it was good. It had some serious ideas, and was nicely creepy, but also had a great sense of humor. It’s hard for me to explain much without saying too much. There’s a couple with a long-term relationship. They move to the sticks for a fresh start. Things get weird. Great pacing in this one, I thought, too.
October 20: Here for Blood (2022)
This was definitely one of the best movies we watched! It reminded us of McG’s The Babysitter (2017), and in a good way. There’s a busy young woman who is a bunch of college classes and maybe a couple jobs, one of which is babysitting. She needs to cram, so her boyfriend agrees to take one of her babysitting jobs. Meanwhile, somebody has planned to stage a home invasion while she’s babysitting. They expect the petit woman, not her massive mixed martial artist boyfriend. It’s a trip!
October 21: Minutes Past Midnight (2016)
I don’t remember it well at this point. It was an anthology. I remember that some of it was terrible. On the other hand, there was one lovely and funny bit with Arthur Darvill as a serial killer who falls in love.
October 22: Willy’s Wonderland (2021)
Not good, per se, but I think I’ll still recommend it. It’s a rip off of Five Nights at Freddy’s, where Nic Cage is conscripted to spend a night cleaning an abandoned Chuck E. Cheese style place. The animatronics are out to get him. He fights back. Also, there are dying teenagers.
This movie is all about Nicholas Cage, who has zero lines and punctuates his night of cleaning and demonocide by drinking soda and playing pinball. Unhinged.
October 23: Cherry Falls (2000)
This got on my radar as “the most underappreciated slasher of the post-Scream boom”. Yeah, I could buy that. It wasn’t great but it was definitely fun and quirky. Also, my household likes Brittany Murphy. The premise: a serial killer is at work in the town of Cherry Falls. Their schtick? They only kill virgins. (The sherrif comes to this conclusion incredibly quickly with a shockingly small amount of evidence. But he is right.)
The town’s teenagers are urged to lose their virginity as quickly as possible.
If this movie had been made 15 years earlier, it would’ve featured a staggering amount of nudity. In 2000, though, it would have none. I wonder just when that changed!
October 24: Scared Shitless (2024)
A plumber enlists the help of his germaphobic son to go clean and fix some toilets. Little did they know that these toilets… are haunted! It was short and stupid and fun, and I am glad we watched it.
October 25: Death of a Unicorn
I don’t know why this wasn’t good. It felt like they overworked the dough. Everything was there for greatness, but it didn’t pan out. Dude hits a unicorn with his car. The unicorn’s parents come back for revenge. At one point, Richard E. Grant eats a piece of unicorn meat. Anthony Carrigan is in it and, as always, is great.
October 26: Grafted (2024)
Kind of a mess. A young woman with a skin disorder is trying to find a new skin graft technique so she can fit in. But she accidentally kills a few people and it gets worse. It wasn’t bad, but it wasn’t great. Some good Kiwi accents in it, though!
October 28: Boys from County Hell (2020)
This one had been sitting in our Shudder queue for ages! It was worth the wait. In a village in Ireland, there’s a cairn that attracts a very modest number of tourists. The claim is that it’s the grave of an Irish vampire whose story inspired Bram Stoker to write Dracula. The level to which anybody believes this is unclear. Meanwhile, the cairn has to be knocked down to make way for a new motorway. Could this free the evil lurking beneath? Yes.
This was another case of “an unexceptional premise made really well”. The pacing, the casting, the sense of humor, all worked well to make a really enjoyable little movie.
October 29: Good Boy (2025)
Was this whole movie a metaphor? I’m not sure.
There’s a guy who is either sick or suicidal or haunted, and he skips town to go stay at his dead uncle’s house, which even the casual observer can tell is a really bad idea. His sister tells him so, but he’s a jerk and won’t listen. He does take his dog, though. His dog is also worried about him, and the movie is entirely from the perspective of the dog – it’s not all in the dog’s point of view, but we’re following the dog, so we have to work things out from what the dog can see.
The dog was great. The movie was only okay.
Ghosted
Oops, this is a later addition! Sometimes we fill in some days with TV. In the past, this was often when we’d watch American Horror Story. This year, we watched about half of Ghosted, a 2017 show starring Adam Scott and Craig Robinson as an odd couple of kinda-losers who get recruited into a secret government agency that pursues X-files. How had I never heard of this before? And, in fact, nearly nobody I’ve talked to has.
Gloria and I liked it, but it was pretty uneven. The structure of the show takes a hard left turn partway through, and it’s worse for sure.
That’s it!
We watched some other things not reflected here. Most notably, we started watching “It: Welcome to Derry”, which was fine.
I think it was a decent year, and I’ll have to see if I can remember enough about 2024’s movies to write those up too…
A language awakens the moment its community shares what it has lived and built.
If you were building web applications during the first dot-com boom, chances are you wrote Perl. And if you’re now a CTO, tech lead, or senior architect, you may instinctively steer teams away from it—even if you can’t quite explain why.
This reflexive aversion isn’t just a preference. It’s what I call Dotcom Survivor Syndrome: a long-standing bias formed by the messy, experimental, high-pressure environment of the early web, where Perl was both a lifeline and a liability.
Perl wasn’t the problem. The conditions under which we used it were. And unfortunately, those conditions, combined with a separate, prolonged misstep over versioning, continue to distort Perl’s reputation to this day.
The Glory Days: Perl at the Heart of the Early Web
In the mid- to late-1990s, Perl was the web’s duct tape.
-
It powered CGI scripts on Apache servers.
-
It automated deployments before DevOps had a name.
-
It parsed logs, scraped data, processed form input, and glued together whatever needed glueing.
Perl 5, released in 1994, introduced real structure: references, modules, and the birth of CPAN, which became one of the most effective software ecosystems in the world.
Perl wasn’t just part of the early web—it was instrumental in creating it.
The Dotcom Boom: Shipping Fast and Breaking Everything
To understand the long shadow Perl casts, you have to understand the speed and pressure of the dot-com boom.
We weren’t just building websites.
We were inventing how to build websites.
Best practices? Mostly unwritten.
Frameworks? Few existed.
Code reviews? Uncommon.
Continuous integration? Still a dream.
The pace was frantic. You built something overnight, demoed it in the morning, and deployed it that afternoon. And Perl let you do that.
But that same flexibility—its greatest strength—became its greatest weakness in that environment. With deadlines looming and scalability an afterthought, we ended up with:
-
Thousands of lines of unstructured CGI scripts
-
Minimal documentation
-
Global variables everywhere
-
Inline HTML mixed with business logic
-
Security holes you could drive a truck through
When the crash came, these codebases didn’t age gracefully. The people who inherited them, often the same people who now run engineering orgs, remember Perl not as a powerful tool, but as the source of late-night chaos and technical debt.
Dotcom Survivor Syndrome: Bias with a Backstory
Many senior engineers today carry these memories with them. They associate Perl with:
-
Fragile legacy systems
-
Inconsistent, “write-only” code
-
The bad old days of early web development
And that’s understandable. But it also creates a bias—often unconscious—that prevents Perl from getting a fair hearing in modern development discussions.
Version Number Paralysis: The Perl 6 Effect
If Dotcom Boom Survivor Syndrome created the emotional case against Perl, then Perl 6 created the optical one.
In 2000, Perl 6 was announced as a ground-up redesign of the language. It promised modern syntax, new paradigms, and a bright future. But it didn’t ship—not for a very long time.
In the meantime:
-
Perl 5 continued to evolve quietly, but with the implied expectation that it would eventually be replaced.
-
Years turned into decades, and confusion set in. Was Perl 5 deprecated? Was Perl 6 compatible? What was the future of Perl?
To outsiders—and even many Perl users—it looked like the language was stalled. Perl 5 releases were labelled 5.8, 5.10, 5.12… but never 6. Perl 6 finally emerged in 2015, but as an entirely different language, not a successor.
Eventually, the community admitted what everyone already knew: Perl 6 wasn’t Perl. In 2019, it was renamed Raku.
But the damage was done. For nearly two decades, the version number “6” hung over Perl 5 like a storm cloud – a constant reminder that its future was uncertain, even when that wasn’t true.
This is what I call Version Number Paralysis:
-
A stalled major version that made the language look obsolete.
-
A missed opportunity to signal continued relevance and evolution.
-
A marketing failure that deepened the sense that Perl was a thing of the past.
Even today, many developers believe Perl is “stuck at version 5,” unaware that modern Perl is actively maintained, well-supported, and quite capable.
While Dotcom Survivor Syndrome left many people with an aversion to Perl, Version Number Paralysis gave them an excuse not to look closely at Perl to see if it had changed.
What They Missed While Looking Away
While the world was confused or looking elsewhere, Perl 5 gained:
-
Modern object systems (Moo, Moose)
-
A mature testing culture (Test::More, Test2)
-
Widespread use of best practices (Perl::Critic, perltidy, etc.)
-
Core team stability and annual releases
-
Huge CPAN growth and refinements
But those who weren’t paying attention, especially those still carrying dotcom-era baggage, never saw it. They still think Perl looks like it did in 2002.
Can We Move On?
Dotcom Survivor Syndrome is real. So is Version Number Paralysis. Together, they’ve unfairly buried a language that remains fast, expressive, and battle-tested.
We can’t change the past. But we can:
-
Acknowledge the emotional and historical baggage
-
Celebrate the role Perl played in inventing the modern web
-
Educate developers about what Perl really is today
-
Push back against the assumption that old == obsolete
Conclusion
Perl’s early success was its own undoing. It became the default tool for the first web boom, and in doing so, it took the brunt of that era’s chaos. Then, just as it began to mature, its versioning story confused the industry into thinking it had stalled.
But the truth is that modern Perl is thriving quietly in the margins – maintained by a loyal community, used in production, and capable of great things.
The only thing holding it back is a generation of developers still haunted by memories of CGI scripts, and a version number that suggested a future that never came.
Maybe it’s time we looked again.
The post Dotcom Survivor Syndrome – How Perl’s Early Success Created the Seeds of Its Downfall first appeared on Perl Hacks.
I was writing a data intensive code in Perl relying heavily on PDL for some statistical calculations (estimation of percentile points in some very BIG vectors, e.g. 100k to 1B elements), when
I noticed that PDL was taking a very (and unusually long!) time to produce results compared to my experience in Python.
This happened irrespective of whether one used the pct or oddpct functions in PDL::Ufunc.
The performance degradation had a very interesting quantitative aspect: if one asked PDL to return a single percentile it did so very fast;
but if one were to ask for more than one percentiles, the time-to-solution increased linearly with the number of percentiles specified.
Looking at the source code of the pct function, it seems that it is implemented by calling the function pctover, which according to the PDL documentation “Broadcasts over its inputs.”
But what is exactly broadcasting? According to PDL::Broadcasting : “[broadcasting] can produce very compact and very fast PDL code by avoiding multiple nested for loops that C and BASIC users may be familiar with. The trouble is that it can take some getting used to, and new users may not appreciate the benefits of broadcasting.” Reading the relevant PDL examples and revisiting the NumPy documentation (which also uses this technique), broadcasting : treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python..
It seems that when one does something like:
use PDL::Lite;
my $very_big_ndarray = ... ; # code that constructs a HUGE PDL ndarrat
my $pct = sequence(100)/100; # all percentiles from 0 to 100%
my $pct_values = pct( $very_big_ndarray, $pct);
the broadcasting effectively executes sequentially the code for calculating a single percentile and concatenates the results.
The problem with broadcasting for this operation is that the percentile calculation includes a VERY expensive operation, namely the
sorting of the $very_big_darray before the (trivial) calculation of the percentile from the sorted values as detailed in
Wikipedia. So when the percentile operation is broadcast by PDL, the sorting is repeated for each
percentile value in $pct, leading to catastrophic loss of performance!
How can we fix this? It turns out to be reasonably trivial : we need to reimplement the percentile function so that it does not broadcast.
One of the simplest quantile functions to implement, is the one based on the empirical cumulative distribution function (this corresponds to the Type 3 quantile in the
classification by Hyndman and Fan).
This one can be trivially implemented in Perl using PDL as:
sub quantile_type_3 {
my ( $data, $pct ) = @_;
my $sorted_data = $data->qsort;
my $nelem = $data->nelem;
my $cum_ranks = floor( $pct * $nelem );
$sorted_data->index($cum_ranks);
}
(The other quantiles can be implemented equally trivially using affine operations as explained in R’s documentation of the quantile function).
To see how well this works, I wrote a Perl benchmark script that benchmarks
the builtin function pct, the quantile_type_3 function on synthetic data and then calls the companion
R script to profile the 9 quantile functions and the 3 sort
functions in R for the same dataset.
I obtained the following performance figures in my old Xeon: the “de-broadcasted” version of the quantile function achieves the same performance as the R implementations, whereas the
PDL broadcasting version is 100 times slower.
| Test | Iterations | Elements | Quantiles | Elapsed Time (s) |
|---|---|---|---|---|
| pct | 10 | 1000000 | 100 | 132.430000 |
| quantile_type_3 | 10 | 1000000 | 100 | 1.320000 |
| pct_R_1 | 10 | 1000000 | 100 | 1.290000 |
| pct_R_2 | 10 | 1000000 | 100 | 1.281000 |
| pct_R_3 | 10 | 1000000 | 100 | 1.274000 |
| pct_R_4 | 10 | 1000000 | 100 | 1.283000 |
| pct_R_5 | 10 | 1000000 | 100 | 1.290000 |
| pct_R_6 | 10 | 1000000 | 100 | 1.286000 |
| pct_R_7 | 10 | 1000000 | 100 | 1.233000 |
| pct_R_8 | 10 | 1000000 | 100 | 1.309000 |
| pct_R_9 | 10 | 1000000 | 100 | 1.291000 |
| sort_quick | 10 | 1000000 | 100 | 1.220000 |
| sort_shell | 10 | 1000000 | 100 | 1.758000 |
| sort_radix | 10 | 1000000 | 100 | 0.924000 |
As can be seen from the table, the sorting operations account mostly for the bulk of the execution time of the quantile functions.
Two major takehome points:
1) don’t be afraid to look under the hood/inside the blackbox when performance is surprisingly disappointing!
2) be careful of broadcasting operations in PDL, NumPy, or Matlab.