Beautiful Perl series

This post is part of the beautiful Perl features series.
See the introduction post for general explanations about the series.

The last post was about BLOCKs and lexical scoping, a feature present in almost all programming languages. By contrast, today's topic is about another feature quite unique to Perl : dynamic scoping, introduced through the 'local' keyword. As we shall see, lexical scoping and dynamic scoping address different needs, and Perl allows us to choose the scoping strategy most appropriate to each situation.

Using local : an appetizer

Let us start with a very simple example:

say "current time here is ", scalar localtime;

foreach my $zone ("Asia/Tokyo", "Africa/Nairobi", "Europe/Athens") {
  local $ENV{TZ} = $zone;
  say "in $zone, time is ", scalar localtime;
}

say "but here current time is still ", scalar localtime;

Perl's localtime builtin function returns the time "in the current timezone". On Unix-like systems, the notion of "current timezone" can be controlled through the TZ environment variable, so here the program temporarily changes that variable to get the local time in various cities around the world. At the end of the program, we get back to the usual local time. Here is the output:

current time here is Sun Feb  8 22:21:41 2026
in Asia/Tokyo, time is Mon Feb  9 06:21:41 2026
in Africa/Nairobi, time is Mon Feb  9 00:21:41 2026
in Europe/Athens, time is Sun Feb  8 23:21:41 2026
but here current time is still Sun Feb  8 22:21:41 2026

The gist of that example is that:

1. we want to call some code not written by us (here: the localtime builtin function);
2. the API of that code does not offer any parameter or option to tune its behaviour according to our needs; instead, it relies on some information in the global state of the running program (here: the local timezone);
3. apart from getting our specific result, we don't want the global state to be durably altered, because this could lead to unwanted side-effects.

Use cases similar to this one are very common, not only in Perl but in every programming language. There is always some kind of implicit global state that controls how the program behaves internally and how it interacts with its operating system: for example environment variables, command-line arguments, open sockets, signal handlers, etc. Each language has its strategies for dealing with the global state, but often the solutions are multi-faceted and domain-specific. Perl shines because its local mechanism covers a very wide range of situations in a consistent way and is extremely simple to activate.

Before going into the details, let us address the bad reputation of dynamic scoping. This mechanism is often described as something that should be absolutely avoided because it implements a kind of action at distance, yielding unpredictable program behaviour. Yes, it is totally true, and therefore languages that only have dynamic scoping are not suited for programming-in-the-large. Yet in specific contexts, dynamic scoping is acceptable or even appropriate. The most notable examples are Unix shells and also Microsoft Powershell for Windows, that still today all rely on dynamic scoping, probably because it's easier to implement.

Perl1 also used dynamic scoping in its initial design, presumably for the same reason of easiness of implementation, and also because it was heavily influenced by shell programming and was addressing the same needs. When later versions of Perl evolved into a general-purpose language, the additional mechanism of lexical scoping was introduced because it was indispensable for larger architectures. Nevertheless, dynamic scoping was not removed, partly for historical reasons, but also because action at distance is precisely what you want in some specific situations. The next chapters will show you why.

The mechanics of local

In the last post we saw that a my declaration declares a new variable, possibly shadowing a previous variable of the same name. By contrast, a local declaration is only possible on a global variable that already exists; the effect of the declaration is to temporarily push aside the current value of that global variable, leaving room for a new value. After that operation, any code anywhere in the program that accesses the global variable will see the new value ... until the effect of local is reverted, namely when exiting the scope in which the local declaration started.

Here is a very simple example, again based on the %ENV global hash. This hash is automatically populated with a copy of the shell environment when perl starts up. We'll suppose that the initial value of $ENV{USERNAME}, as transmitted from the shell, is Alex:

sub greet_user ($salute) {
  say "$salute, $ENV{USERNAME}";
}

greet_user("Hello");                 # Hello, Alex

{ local $ENV{USERNAME} = 'Brigitte';
  greet_user("Hi");                  # Hi, Brigitte

  { local $ENV{USERNAME} = 'Charles';
    greet_user("Good morning");      # Good morning, Charles
  }

  greet_user("Still there");         # Still there, Brigitte
}

greet_user("Good bye");              # Good bye, Alex

The same thing would work with a global package variable:

our $username = 'Alex';             # "our" declares a global package variable

sub greet_user ($salute) {
  say "$salute, $username";
}

greet_user("Hello");
{ local $username = 'Brigitte';
  greet_user("Hi");
  ... etc.

but if the variable is not pre-declared, we get an error:

Global symbol "$username" requires explicit package name (did you forget to declare "my $username"?)

or if the variable is pre-declared as lexical through my $username instead of our $username, we get another error:

Can't localize lexical variable $username

So the local mechanism only applies to global variables. There are three categories of such variables:

1. standard variables of the Perl interpreter;
2. package members (of modules imported from CPAN, or of your own modules). This includes the whole symbol tables of those packages, i.e. not only the global variables, but also the subroutines or methods;
3. what the Perl documentation calls elements of composite types, i.e. individual members of arrays or hashes. Such elements can be localized even if the array or hash is itself a lexical variable, because while the entry point to the array or hash may be on the execution stack, its member elements are always stored in global heap memory. This last category of global variables is perhaps more difficult to understand, but we will give examples later to make it clear.

As we can see, the single and simple mechanism of dynamic scoping covers a vast range of applications! Let us explore the various use cases.

Localizing standard Perl variables

The Perl interpreter has a number of builtin special variables, listed in perlvar. Some of them control the internal behaviour of the interpreter; some other are interfaces to the operating system (environment variables, signal handlers, etc.). These variables have reasonable default values that satisfy most common needs, but they can be changed whenever needed. If the change is for the whole program, a regular assignment instruction is good enough; but if it is for a temporary change in a specific context, localis the perfect tool for the job.

Internal variables of the interpreter

The examples below are typical of idiomatic Perl programming: altering global variables so that builtin functions behave differently from the default.

# input record separator ($/)
my @lines         = <STDIN>;                     # regular mode, separating by newlines
my @paragraphs    = do {local $/ = ""; <STDIN>}; # paragraph mode, separating by 2 or more newlines
my $whole_content = do {local $/; <STDIN>};      # slurp mode, no separation

# output field and output record separators
sub write_csv_file ($rows, $filename) {
  open my $fh, ">:unix", $filename or die "cannot write into $filename: $!";
  local $, = ",";               # output field separator -- inserted between columns
  local $\ = "\r\n";            # output row separator   -- inserted between rows
  print $fh @$_ foreach @$rows; # assuming that each member of @$rows is an arrayref
}

# list separator in interpolated strings
my @perl_files   = <*.pl>; # globbing perl files in the current directory
my @python_files = <*.py>; # idem for python files
{ local $" = ", ";         # lists will be comma-separated when interpolated in a string
  say "I found these perl files: @perl_files and these python files: @python_files";
}

Interface to the operating system

We have already seen two examples involving the %ENV hash of environment variables inherited from the operating system. Likewise, it is possible to tweak the @ARGV array before parsing the command-line arguments. Another interesting variable is the %SIG hash of signal handlers, as documented in perlipc:

{ local $SIG{HUP} = "IGNORE"; # don't want to be disturbed for a while
  do_some_tricky_computation();
}

Predefined handles like STDIN, STDOUT and STDERR can also be localized:

say "printing to regular STDOUT";
{ local *STDOUT;
  open STDOUT, ">", "captured_stdout.txt" or die $!;
  do_some_verbose_computation();
}
say "back to regular STDOUT";

Localizing package members

Package global variables

Global variables are declared with the our keyword. The difference with lexical variables (declared with my) is that such global variables are accessible, not only from within the package, but also from the outside, if prefixed by the package name. So if package Foo::Bar declares our ($x, @a, %h), these variables are accessible from anywhere in the Perl program under $Foo::Bar::x, @Foo::Bar::a or %Foo::Bar::h.

Many CPAN modules use such global variables to expose their public API. For example, the venerable Data::Dumper chooses among various styles for dumping a data structure, depending on the $Data::Dumper::Indent variable. The default (style 2) is optimized for readability, but sometimes the compact style 0 is more appropriate:

print Dumper($data_tree);
print do {local $Data::Dumper::Indent=0; Dumper($other_tree)};

Carp or URI are other examples of well-known modules where global variables are used as a configuration API.

Modules with this kind of architecture are often pretty old; more recent modules tend to prefer an object-oriented style, where the configuration options are given as options to the new() method instead of using global variables. Of course the object-oriented architecture offers better encapsulation, since a large program can work with several instances of the same module, each instance having its own configuration options, without interference between them. This is not to say, however, that object-oriented configuration is always the best solution: when it comes to tracing, debugging or profiling needs, it is often very convenient to be able to tune a global knob and have its effect applied to the whole program: in those situations, what you want is just the opposite of strict encapsulation! Therefore some modules, although written in a modern style, still made the wise choice of leaving some options expressed as global variables; changing these options has a global effect, but thanks to local this effect can be limited to a specific scope. Examples can be found in Type::Tiny or in List::Util.

Subroutines (aka monkey-patching)

Every package has a symbol table that contains not only its global variables, but also the subroutines (or methods) declared in that package. Since the symbol table is writeable, it is possible to overwrite any subroutine, thereby changing the behaviour of the package - an operation called monkey-patching. Of course monkey-patching could easily create chaos, so it should be used with care - but in some circumstances it is extremely powerful and practical. In particular, testing frameworks often use monkey-patching for mocking interactions with the outside world, so that the internal behaviour of a sofware component can be tested in isolation.

The following example is not very realistic, but it's the best I could come up with to convey the idea in just a few lines of code. Consider a big application equipped with a logger object. Here we will use a logger from Log::Dispatch:

use Log::Dispatch;
my $log = Log::Dispatch->new(
  outputs   => [['Screen', min_level => 'info', newline => 1]],
);

The logger has methods debug(), info(), error(), etc. for accepting messages at different levels. Here it is configured to only log messages starting from level info; so when the client code calls info(), the message is printed, while calls to debug() are ignored. As a result, when the following routine is called, we normally only see the messages "start working ..." and "end working ...":

sub work ($phase) {
  $log->info("start working on $phase");
  $log->debug("weird condition while doing $phase"); # normally not seen - level below 'info'
  $log->info("end working on $phase\n");
}

Now suppose that we don't want to change the log level for the whole application, but nevertheless we need to see the debug messages at a given point of execution. One (dirty!) way of achieving this is to temporarily treat calls to debug() as if they were calls to info(). So a scenario like

work("initial setup");
{ local *Log::Dispatch::debug = *Log::Dispatch::info; # temporarily alias the 'debug' method to 'info'
  work("core stuff")}
work("cleanup");

logs the following sequence:

start working on initial setup
end working on initial setup

start working on core stuff
weird condition while doing core stuff
end working on core stuff

start working on cleanup
end working on cleanup

Monkey-patching techniques are not specific to Perl; they are used in all dynamic languages (Python, Javascript, etc.), not for regular programming needs, but rather for testing or profiling tasks. However, since other dynamic languages do not have the local mechanism, temporary changes to the symbol table must be programmed by hand, by storing the initial code reference in a temporary variable, and restoring it when exiting from the monkey-patched scope. This is a bit more work and is more error-prone. Often there are library modules for making the job easier, though: see for example https://docs.pytest.org/en/7.1.x/how-to/monkeypatch.html in Python.

Monkey-patching in a statically-typed language like Java is more acrobatic, as shown in Nicolas Fränkel's blog.

Localizing elements of arrays or hashes

The value of an array at a specific index, or the value of a hash for a specific key, can be localized too. We have already seen some examples with the builtin hashes %ENV or %SIG, but it works as well on user data, even when the data structure is several levels deep. A typical use case for this is when a Web application loads a JSON, YAML or XML config file at startup. The config data becomes a nested tree in memory; if for any reason that config data must be changed at some point, local can override a specific data leaf, or any intermediate subtree, like this:

local $app->config->{logger_file} = '/opt/log/special.log';
local $app->config->{session}     = {storage => '/opt/data/app_session',
                                     expires => 42900,
                                    };

Another example can be seen in my Data::Domain module. The job of that module is to walk through a datatree and check if it meets the conditions expected by a "domain". The inspect() method that does the checking sometimes needs to know at which node it is currently located; so a $context tree is worked upon during the traversal and passed to every method call. With the help of local, temporary changes to the shape of $context are very easy to implement:

  for (my $i = 0; $i < $n_items; $i++) {
    local $context->{path} = [@{$context->{path}}, $i];
    ...

An alternative could have been to just push $i on top of the @{$context->{path}} array, and then pop it back at the end of the block. But since this code may call subclasses written by other people, with little guarantee on how they would behave with respect to $context->{path}, it is safer to localize it and be sure that $context->{path} is in a clean state when starting the next iteration of the loop. Interested readers can skim through the source code to see the full story. A similar technique can also be observed in Catalyst::Dispatcher.

A final example is the well-known DBI module, whose rich API exploits several Perl mechanisms simultaneously. DBI is principally object-oriented, except that the objects are called "handles"; but in addition to methods, handles also have "attributes" accessible as hash members. The DBI documentation explicitly recommends to use local for temporary modifications to the values of attributes, for example for the RaiseError attribute. This is interesting because it shows a dichotomy between API styles: if DBI had a purely object-oriented style, with usual getter and setter methods, it would be impossible to use the benefits of local - temporary changes to an attribute followed by a revert to the previous value would have to be programmed by hand.

How other languages handle temporary changes to global state

As argued earlier, the need for temporary changes to global state occurs in every programming language, in particular for testing, tracing or profiling tasks. When dynamic scoping is not present, the most common solution is to write specific code for storing the old state in a temporary variable, implement the change for the duration of a given computation, and then restore the old state. A common best practice for such situations is to use a try ... finally ... construct, where restoration to the old state is implemented in the finally clause: this guarantees that even if exceptions occur, the code exits with a clean state. Most languages do possess such constructs - this is definitely the case for Java, JavaScript and Python.

Python context managers

Python has a mechanism more specifically targeted at temporary changes of context : this is called context manager, triggered through a with statement. A context manager implements special methods __enter__() and __exit__() that can be programmed to operate changes into the global context. This technique offers more precision than Perl's local construct, since the enter and exit methods are free to implement any kind of computation; however it is less general, because each context manager is specialized for a specific task. Python's contextlib library provides a collection of context managers for common needs.

Wrapping up

Perl's local mechanism is often misunderstood. It is frowned upon because it breaks encapsulation - and this criticism is perfectly legitimate as far as regular programming tasks are concerned; but on the other hand, it is a powerful and clean mechanism for temporary changes to the global execution state. It solves real problems with surprising grace, and it’s one of the features that makes Perl uniquely expressive among modern languages. So do not follow the common advice to avoid local at all costs, but learn to identify the situations where local will be a helpful tool to you!

Beyond the mere mechanism, the question is more at the level of philosophy of programming: to which extent should we enforce strict encapsulation of components? In an ideal world, each component has a well-defined interface, and interactions are only allowed to go through the official interfaces. But in a big assembly of components, we may encounter situations that were not foreseen by the designers of the individual components, and require inserting some additional screws, or drilling some additional holes, so that the global assembly works satisfactorily. This is where Perl's local is a beautiful device.

About the cover picture

The cover picture¹ represents a violin with a modified global state: the two intermediate strings are crossed! This is the most spectacular example of scordatura in Heirich Biber's Rosary sonatas (sometimes also called "Mystery sonatas"). Each sonata in the collection requires to tune the violin in a specific way, different from the standard tuning in fifths, resulting in very different atmospheres. It requires some intellectual gymnastics from the player, because the notes written in the score no longer represent the actual sounds heard, but merely refer to the locations of fingers on a normal violin; in other words, this operation is like monkey-patching a violin!

1. CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=110521795 ↩

This week, I challenged myself to write small but meaningful Perl programs for Weekly Challenge — focusing on clean logic, edge cases, and test-driven validation.

Sometimes, the simplest problems teach the strongest lessons.

Challenge 1: Text Justifier (Centering Text with Custom Padding)

Problem Statement
Write a script to return the string that centers the text within that width using asterisks * as padding.

If the string length is greater than or equal to the width, return the string as-is.

Example
Input:

("Hi", 5)

Output:

*Hi**

Implementation Highlights

• Calculated total padding required.
• Split padding evenly left and right.
• Handled odd padding differences correctly.
• Covered edge cases like:
  • Empty string
  • Width equal to string length
  • Width smaller than string length

Key Logic

my $pad_total = $width - $len;
my $left_pad = int($pad_total / 2);
my $right_pad = $pad_total - $left_pad;

This ensures perfect centering even when padding is odd.

What I Learned

• Importance of integer division in layout formatting
• Handling edge cases makes logic robust
• Writing test cases increases confidence immediately

Challenge 2: Word Sorter (Alphabetical Word Sorting)

Problem Statement
Write a script to order words in the given sentence alphabetically but keeps the words themselves unchanged.

Example
Input:

"I have 2 apples and 3 bananas!"

Output:

2 3 and apples bananas! have I

Implementation Highlights

• Split words using whitespace regex
• Used Perl’s built-in sort
• Rejoined words with single spaces
• Managed multiple spaces correctly

Core Logic

my @words = split /\s+/, $str;
return join ' ', sort @words;

Simple. Elegant. Effective.

What I Learned

• Perl’s sort is powerful and concise
• Regex-based splitting handles messy input cleanly
• Even small scripts benefit from structured testing

PAGI 0.001017 is on CPAN. The headline feature is HTTP/2 support in PAGI::Server, built on the nghttp2 C library and validated against h2spec's conformance suite. HTTP/2 is marked experimental in this release -- the protocol works, the compliance numbers are solid, and we want production feedback before dropping that label.

This post focuses on why h2c (cleartext HTTP/2) between your reverse proxy and backend matters, and how PAGI::Server implements it.

The Quick Version

# Install the HTTP/2 dependency
cpanm Net::HTTP2::nghttp2

# Run with HTTP/2 over TLS
pagi-server --http2 --ssl-cert cert.pem --ssl-key key.pem app.pl

# Run with cleartext HTTP/2 (h2c) -- behind a proxy
pagi-server --http2 app.pl

Your app code doesn't change. HTTP/2 is a transport concern handled entirely by the server. The same async handlers that serve HTTP/1.1 serve HTTP/2 without modification.

"I Have nginx in Front, Why Do I Care?"

If nginx terminates TLS and speaks HTTP/2 to clients, why does the backend need HTTP/2 too?

The answer is h2c -- cleartext HTTP/2 between your proxy and your application server. No TLS overhead, but all of HTTP/2's protocol benefits on the internal hop: stream multiplexing over a single TCP connection, HPACK header compression (especially effective for repetitive internal headers like auth tokens and tracing IDs), and per-stream flow control so a slow response on one stream doesn't block others.

The practical wins: fewer TCP connections between proxy and backend (one multiplexed h2c connection replaces a pool of HTTP/1.1 connections), less file descriptor and kernel memory pressure, and no TIME_WAIT churn from connection recycling.

Where h2c Matters

gRPC requires HTTP/2 -- it doesn't work over HTTP/1.1 at all. If you're building gRPC services, h2c is mandatory.

API gateway fan-out is where multiplexing shines. When your gateway fans out to 10 backend services per request, h2c means 1-2 connections per backend instead of a pool of 50-100.

Service mesh environments (Envoy/Istio sidecars) default to HTTP/2 between services. A backend that speaks h2c natively means one less protocol translation.

A Note on Proxies

Not all proxies handle h2c equally:

• Envoy has the best h2c upstream support with full multiplexing
• Caddy makes it trivial: reverse_proxy h2c://localhost:8080
• nginx supports h2c via grpc_pass for gRPC workloads, but its generic proxy_pass doesn't support proxy_http_version 2.0

For full multiplexing to backends, Envoy or Caddy are better choices than nginx today.

HTTP/2 Over TLS -- No Proxy Required

h2c isn't the only mode. PAGI::Server also does full HTTP/2 over TLS with ALPN negotiation:

pagi-server --http2 --ssl-cert cert.pem --ssl-key key.pem app.pl

This is useful when you don't want the overhead or complexity of a reverse proxy -- internal tools, admin dashboards, development servers, or any app where the traffic doesn't justify a separate proxy layer. Browsers get HTTP/2 directly, with TLS, no nginx required.

What PAGI::Server Does

Dual-Mode Protocol Detection

With TLS, PAGI::Server uses ALPN negotiation during the handshake -- advertising h2 and http/1.1, letting the client choose. The protocol is decided before the first byte of application data.

Without TLS (h2c mode), PAGI::Server inspects the first 24 bytes of each connection for the HTTP/2 client connection preface. If it matches, the connection upgrades to HTTP/2. If not, it falls through to HTTP/1.1. Both protocols coexist on the same port, same worker -- no configuration needed beyond --http2.

Either way, HTTP/1.1 clients are still served normally. The server handles both protocols on the same port.

WebSocket over HTTP/2 (RFC 8441)

Most HTTP/2 implementations skip this. PAGI::Server supports the Extended CONNECT protocol from RFC 8441, which tunnels WebSocket connections over HTTP/2 streams. Multiple WebSocket connections multiplex over a single TCP connection instead of requiring one TCP connection each.

Compliance

Built on nghttp2 (the same C library behind curl, Firefox, and Apache's mod_http2). PAGI::Server passes 137 of 146 h2spec conformance tests (93.8%). The 9 remaining failures are in nghttp2 itself and shared with every server that uses it. Load tested with h2load at 60,000 requests across 50 concurrent connections with no data loss or protocol violations.

Full test-by-test results are published: HTTP/2 Compliance Results.

Multi-Worker and Tunable

HTTP/2 works in multi-worker prefork mode. Each worker independently handles HTTP/2 sessions:

pagi-server --http2 --workers 4 app.pl

Protocol settings are exposed for environments that need fine-tuning:

my $server = PAGI::Server->new(
    app   => $app,
    http2 => 1,
    h2_max_concurrent_streams => 50,   # default: 100
    h2_initial_window_size => 131072,  # default: 65535
    h2_max_frame_size => 32768,        # default: 16384
    h2_max_header_list_size => 32768,  # default: 65536
);

Most deployments won't need to touch these. The defaults follow the RFC recommendations.

Context in the Perl Ecosystem

Perl has had HTTP/2 libraries on CPAN (Protocol::HTTP2, Net::HTTP2), but application servers haven't integrated them with validated compliance testing. PAGI::Server is the first to publish h2spec results and ship h2c with automatic protocol detection alongside HTTP/1.1. If you're currently running Starman, Twiggy, or Hypnotoad, none of them offer HTTP/2.

What Else Is in 0.001017

The rest of the release is operational improvements:

• Worker heartbeat monitoring -- parent process detects workers with blocked event loops and replaces them via SIGKILL + respawn. Default 50s timeout. Only triggers on true event loop starvation; async handlers using await are unaffected.
• Custom access log format -- format strings with atoms like %a (address), %s (status), %D (duration).
• TLS performance fix -- shared SSL context via SSL_reuse_ctx eliminates per-connection CA bundle parsing. 26x throughput improvement at 8+ concurrent TLS connections.
• SSE wire format fix -- now handles CRLF, LF, and bare CR line endings per the SSE specification.
• Multi-worker fixes -- shutdown escalation, parameter pass-through, and various stability improvements.

Getting Started

# Install PAGI
cpanm PAGI

# Install HTTP/2 support (optional)
cpanm Net::HTTP2::nghttp2

# Run your app with HTTP/2
pagi-server --http2 app.pl

Links

• PAGI 0.001017 on CPAN
• HTTP/2 Compliance Results
• PAGI on GitHub
• h2spec -- HTTP/2 conformance testing tool

It's been a while since I commented on a Weekly Challenge solution, but here we are at week 360. Such a useful number. So divisible, so circular. It deserves twenty minutes.

Task 2: Word Sorter

The task

You are given a sentence. Write a script to order words in the given
sentence alphabetically but keep the words themselves unchanged.

# Example 1 Input: $str = "The quick brown fox"
#           Output: "brown fox quick The"
# Example 2 Input: $str = "Hello    World!   How   are you?"
#           Output: "are Hello How World! you?"
# Example 3 Input: $str = "Hello"
#           Output: "Hello"
# Example 4 Input: $str = "Hello, World! How are you?"
#           Output: "are Hello, How World! you?"
# Example 5 Input: $str = "I have 2 apples and 3 bananas!"
#           Output: "2 3 and apples bananas! have I"

The thoughts

This should be quite simple: split the words, sort them, put them back together. The sort should be case-insensitive.

join " ", sort { lc($a) cmp lc($b) } split(" ", $str);

Creeping doubt #1

Is converting to lowercase with lc the right way to do case-insenstive compares? Not really. Perl has the fc -- fold-case -- function to take care of subtleties in Unicode. We won't see those in simple ASCII text, but for the full rabbit hole, start with the documentation of fc.

Creeping doubt #2

Doing the case conversion inside the sort means that we will invoke that every time there's a string comparison, which will be quite redundant. We could (probably?) speed it up by pre-calculating the conversions once.

The solution

sub sorter($str)
{
    return join " ",
        map { $_->[0] }
        sort { $a->[1] cmp $b->[1] }
        map { [$_, fc($_)] }
        split(" ", $str);
}

This solution uses the idiom of Schwartzian transformation. Every word gets turned into a pair of [original_word, case_folded_word]. That array of pairs gets sorted, and then we select the original words out of the sorted pairs. This is best read bottom-up.

• split(" ", $str) -- turns the string into an array of words, where words are loosely defined by being white-space separated.
• map { [$_, fc($_)] } -- every word turns into a pair: the original, and its case-folded variation. The result is a list of array references.
• sort { $a->[1] cmp $b->[1] } -- sort by the case-folded versions. The result is still a list of array references.
• map { $_->[0] } -- select the original word from each pair
• join " " -- Return a single string, where the words are separated by one space.

Does it blend?

A quick benchmark shows that it is indeed faster to pre-calculate the case folding. This example used a text string of about 60 words.

           Rate oneline  pre_lc
oneline 32258/s      --    -45%
pre_lc  58140/s     80%      --

My intuition says that when the strings are much shorter, the overhead of the transform might not offset the gains in the sort, but as is so often true, my intuition is crap. This is the result for a string of five words:

oneline  470588/s      --    -59%
pre_lc  1142857/s    143%      --

Beautiful Perl series

This post is part of the beautiful Perl features series.
See the introduction post for general explanations about the series.

BLOCK : sequence of statements

Today's topic is the construct named BLOCK in the perl documentation: a sequence of statements enclosed in curly brackets {}. Both the concept and the syntax are common to many programming languages - they can also be found in languages of C heritage like Java, JavaScript and C++, among others; but Perl is different on various aspects - read on to dig into the details.

BLOCKs in Perl may be used:

1. as part of a compound statement, after an initial control flow construct like if, while, foreach, etc.;
2. as the body of a sub declaration (subroutine or function);
3. at any location where a single statement is expected. Obviously it would also be possible to just insert a plain sequence of statements, but enclosing them in a BLOCK has the advantage of creating a new delimited lexical scope so that the effect of inner declarations is guaranteed to end when control flow exits the BLOCK. Details about lexical scopes are discussed below;
4. as part of a do expression, so that the whole BLOCK becomes a value that can be inserted within a more complex expression. This may be convenient for clarity of thought in some algorithms, and also for avoiding a subroutine call when efficiency is at stake.

All modern programmming languages have constructs equivalent to usages 1 and 2, because these are crucial for structuring algorithms and for handling complexity in programming. Usage 3 is less common, and usage 4 is quite particular to Perl. The next chapters will cover these various aspects in more depth.

Lexical scope

In all usage situations listed above, a Perl BLOCK always opens a new lexical scope, i.e. a portion of code that delimits the effect of inner declarations. Things that can be temporarily declared inside a lexical scope are:

• lexical variables, temporarily binding a variable name to a memory location on the stack - declared through keywords my or state;
• lexical pragmata, temporarily importing semantics into the current BLOCK - introduced through keywords use or no;
• (beginning with Perl 5.18) lexical subroutines, only accessible within the scope - also declared through keywords my or state.

When a BLOCK is used as part of a compound statement (if, foreach, etc.), the initial clause before the BLOCK is already part of the lexical scope, so that variables declared in that clause can then be used within the BLOCK:

foreach my $member (@list) {
  work_with($member); # here $member can be used
}
say $member;          # ERROR : $member is no longer in scope

This is also true when a compound statement has several clauses and therefore several BLOCKs, like if (...) {...} elsif {...} else {...}. Further examples, together with detailed explanations, can be found in perlsub.

Declarations of variables, pragmata or subroutines have the following facts in common:

• they take effect starting from the statement after the declaration. Technically declarations can occur anywhere in the BLOCK; the most common usage is to put them at the beginning, but there is no obligation to do so;
• they may temporarily shadow the effect of declarations in higher scopes;
• their effect ends when control flow exits the BLOCK, whatever the exit cause may be (normal end of the block, explicit exit through instructions like return, next or goto, or exit because an exception was encountered).

Declarations in lexical scopes have effects both at compile-time (the Perl interpreter temporarily alters its parsing rules) and at runtime (the interpreter temporarily allocates or releases resources). For example in the following snippet:

{
  my $db_handle   = DBI->connect(@db_connection_args);
  my @large_array = $db_handle->selectall_array($some_sql, @some_bind_values);

  open my $file_handle, ">", $output_file or die "could not open $output_file: $!";
  print $file_handle formatted_ouput($_) foreach @large_array;
} 

the interpreter knows at compile-time that variables $db_handle, @large_array and $file_handle are allowed within the BLOCK, but not outside of it, so it can check statically for typos or other misuses of variables names; then at runtime the interpreter will dynamically allocate and release memory and external handles when control flow crosses the BLOCK. This is quite similar to what happens in a statically typed language like Java. By contrast, Python, often grouped in the same family as Perl because it is also a dynamically-typed language, does not have the same treatment of lexical scopes.

Lexical scopes in Python are not like Perl

In Python, there is no generally available construct as versatile as a Perl BLOCK. Subsequences of statements are expressed through indentation, but this is only allowed as part of a function definition or as part of a compound statement. A compound statement must always start with a keyword (if, for, while, etc.) that opens the header clause and is followed by a suite:

if is_success():
    summary = summarize_results()
    report_to_user(summary)
    cleanup_resources()

A 'suite' in Python is not to be confused with a 'block'. The official documentation is very careful about the distinction, but many informal texts in Python literature make the confusion; yet the difference is quite important because:

• a Python block, "a piece of Python program text that is executed as a unit", only occurs within a module, a function body or a class definition (https://docs.python.org/3/reference/executionmodel.html#structure-of-a-program);
• a Python suite, "a group of statements controlled by a clause", occurs whenever a clause in a compound statement expects to be followed by some instructions.

Blocks and suites look similar, because they are both expressed as indented sequences of statements; but the difference is that a 'block' opens a new lexical scope, while a 'suite' does not. In particular, this means that variables declared in a compound statement are still available after the statement has ended, which is quite surprising for programmers coming from a C-like culture (including Perl and Java). So for example

for i in [1, 2, 3]:
    pass
print(i)

is valid Python code and prints 3. This example is taken from https://eli.thegreenplace.net/2015/the-scope-of-index-variables-in-pythons-for-loops/ which does a very good job at explaining why Python in this respect works differently from many other languages.

Lexical scope vs dynamic scope

In addition to traditional lexical scoping, Perl also has another construct named dynamic scoping, introduced through the keyword local. Dynamic scoping is a vestige from Perl1, but still very useful in some specific use cases; it will be discussed in a future post in this series. For the moment let us just say that in all common situations, lexical scoping is the most appropriate mechanism for working with variables guaranteed not to interfere with the global state of the program.

Lexical variables

Lexical variables in Perl are introduced with the my keyword. Several variables, possibly of different types, can be declared and initialized in one single statement:

my @table               = ([qw/x y z/], [1, 2, 3], [9, 8, 7]);
my ($nb_rows, $nb_cols) = (scalar @table, scalar $table[0]->@*); 
my (@db_connection_args, %select_args);

Variable initialization and list destructuring

Lexical variables can only be used starting from the statement after the declaration. Therefore it is illegal in Perl to write something like:

my ($x, $y, $z) = (123, $x + 1, $x + 2);

because at the point where expression $x + 1 is encountered, variable $x is not yet in scope. By contrast, Java would accept int x = 123, y = x + 1, z = x + 2, or JavaScript would accept let x = 123, y = x + 1, z = x + 2, because in those languages variable initializations occur in sequence, while Perl starts by evaluating the complete list on the right-hand side of the assignment, and then distributes the values into variables on the left-hand side.

Python is like Perl: it does not accept x, y = 123, x + 1 because x is not defined in the right-hand side. Both Perl and Python had from the start the notion of "destructuring" a list into several individual variables. Other languages adopted similar features much later:

• JavaScript has an advanced mechanism of destructuring since ES6 (2015), that can be applied not only to lists, but also to objects (records). For destructuring a list the syntax variables must be put in angle brackets on the left-hand side of the assignment, in order to avoid ambiguity with sequences of ordinary assignments:

let x = 123, y = x + 1, z = x + 2;  // sequence of assignments
let [a, b, c] = [123, 124, 125];    // list destructuring

• The Amber project for Java recently introduced several mechanisms for pattern matching, which is quite close to the idea of destructuring. However for the moment it can only be used for destructuring records, but not yet for lists.

Coming back to Perl, list destructuring has always been part of common idioms, notably for:

• extracting items from command-line arguments

my ($user, $password, @others) = @ARGV;

• extracting items from the argument list to a subroutine

my ($height, $width, $depth) = @_;

• switching variables

($x, $y) = ($y, $x);

Shadowing variables from higher scopes

Like in other languages, a lexical variable in Perl can shadow another variable of the same name at a higher lexical scope. However, the shadowing effect only starts at the statement after the declaration of the variable. As a result, the shadowed value can still be used in the initializing expression:

my $x = 987;
{ my ($x, $y) = (123, $x + 1);
  say "inner scope, x is $x and y is $y"; # "inner scope, x is 123 and y is 988"
}
say "outer scope, x is $x";               # "outer scope, x is 987"

Now let us see how other dynamically typed languages handle variable shadowing.

Shadowing variables in Python

Python has no explicit variable declarations; instead, any assignment instruction implicitly declares the target of the assignment to be a lexical variable in the current lexical scope.

def foo():
    x = 123 # declares lexical variable x
    y = 456 # declares lexical variable y
    x = 789 # assigns a new value to existing variable x

Since the intention of declaring is not explicitly stated by the programmer, the interpreter is of little help for detecting errors that would be identified as typos in other languages. In the example above, one could suspect that the intent was to declare a z variable instead of assigning a new value to x.

If an assignment occurs in the middle of a lexical scope, the corresponding variable is nevertheless treated as being lexical from the very beginning of the scope. As a consequence, newcomers to Python can easily be surprised by an UnboundLocalError, which can be shortly demonstrated by this example from the official documentation:

x = 10
def foo():
    print(x)
    x += 1

foo()

Here the assignment x += 1 implicitly declares x to be a lexical variable for the whole body of the foo() function, even if the assignment comes at the end. In this situation the print() statement raises an exception because at this point lexical variable x is not bound to a value. By contrast, if the assignment is commented out

x = 10
def foo():
    print(x)
    # x += 1

foo()

the program happily prints 10, because here x is no longer interpreted as a lexical variable, but as the global x.

Python statements global and nonlocal can instruct the parser that some specific variables should not be declared in the current lexical scope, but should instead be taken from the global module scope, or, in case of nested functions or classes, from the next higher scope. So on this respect, Python programming is just the opposite to Perl or Java : instead of explicitly declaring lexical variables, one must explicitly declare the variables that are not lexical. Furthermore, since such declarations apply to the whole current lexical scope, independently of the place where they are inserted, it is an exclusive choice : any use of a named variable must be either from the current lexical scope or from a higher scope or from the global scope. Therefore it is not possible, like in Perl, to use the value of global x in the initialization expression for local lexical x.

Shadowing variables in JavaScript

The historical construct for declaring lexical variables in JavaScript was through the var keyword, which is still present in the language. The behaviour of var is quite similar to Python lexical variables: variables appear to exist even before they are declared (which is called hoisting in JavaScript); they are scoped by functions or modules, not by blocks, so they still hold values afer exiting from the block; and the interpreter does not complain if a variable is declared twice. For all these reasons, var is now deprecated, replaced since ES6 (2015) by keywords const (for variables that do not change after initialization) or let (for mutable variables).

These new constructs indeed introduced more security in usage of lexical variables in JavaScript: such variables can no longer be used after exiting from the block, and redeclarations raise syntax errors. Yet one ambiguity remains: the shadowing effect of a variable declared with let does not start at the location of the declaration, but at the beginning of the enclosing block. This is no longer called "hoisting", but still it means that from the beginning of the block that variable name shadows any variable with the same name in higher scopes. This is called temporal dead zone in JavaScript literature.

Shadowing is prohibited in Java

Java has no ambiguity with shadowing ... because it has a more radical approach: it raises an exception when a variable is declared in an inner block with a name already in use at a higher scope! The following snippet

public class ScopeDemo {
  public static void main(String[] args) {
    int x = 987;
    {
      int x = 123, y = x + 1, z = x + 2;
      System.out.println("here x is " + x + " and y is " + y);
    }
    System.out.println("here x is " + x);
  }
}

yields:

ScopeDemo.java:6: error: variable x is already defined in method main(String[])
      int x = 123, y = x + 1, z = x + 2;
          ^
1 error
error: compilation failed

Lexical pragmata

In Perl, lexical scopes are not only used to control the lifetime of lexical variables: they are also used for lexical pragmata that temporarily alter the behaviour of the interpreter, either by adding some semantics (through the keyword use) or by removing some semantics (through the keyword no). Here is an example of one very common idiom:

use strict;
use warnings;
foreach my $user (get_users_from_database()) {
  no warnings 'uninitialized';
  my $body = "Dear $user->{firstname} $user->{lastname}, bla bla bla";
  ...
}

At the beginning of the program, the warnings pragma is activated, because this is general good practice, so that the interpreter can detect suspect situations and warn about them. But when working with a $user record from the database, some fields might be undef, which is OK, there is no reason to issue a warning - so within that BLOCK the interpreter is instructed to treat undefined data as empty strings, without complaining.

In a similar vein, it is sometimes necessary to alleviate the controls performed by use strict, in particular on the subject of symbolic references. This control forbids programmatic insertion of new subroutines into the symbol table of a module, a safe measure of prevention; yet this feature is powerful and very useful in some specific situations - so when needed, one can temporarily disable the control:

foreach my $method_name (@list_of_names) {
  no strict 'refs';
  *{$method_name} = generate_closure_for($method_name);
}

This technique is used quite extensively for example in the Object-Relational Mapping module DBIx::DataModel for generating methods that implement navigation from one table to another related table. The source code can demonstrate usage patterns.

Some pragmata can also reinforce controls instead of alleviating them. One very good example is the autovivification module, which changes the default behaviour of Perl on implicit creation of intermediate references:

my $tree;                  # at this point, $tree is undef
$tree->{foo}{bar}[1] = 99; # no error; now $tree is {foo => {bar => [undef, 99]}}

Autovivification can be very handy, but it can be dangerous too. If we want to be on the safe side, we can write

{ no autovivification qw/fetch store/;
  my $tree;                  # at this point, $tree is undef
  $tree->{foo}{bar}[1] = 99; # ERROR: Can't vivify reference
}

Like for lexical variables, lexical pragmata can be nested, the innermost use or no declaration temporarily shadowing previous declarations on the same pragma.

Other examples of lexical pragmata include bigint, which transparently transforms all arithmetic operations to work with instances of Math::BigInt; or the incredible Regexp::Grammars module that adds grammatical parsing features to Perl regexes. The perlpragma documentation explains how module authors can implement new lexical pragmata.

'do': transform a BLOCK into an expression

In all examples seen so far, BLOCKs were treated as statements; but thanks to the do BLOCK construct, it is also possible to insert a BLOCK anywhere in an expression. The value from the last instruction in the BLOCK is then processed by operators in the expression. This is very convenient for performing a small computation in-place, either for clarity or for efficiency reasons.

The first example is a cheap version of XML entity encoding, slight adaptation from my module Excel::ValueWriter::XLSX:

my %ENTITY_TABLE = ( '<' => '&lt;', '>' => '&gt;', '&' => '&amp;' );
my $entity_regex = do {my $chars = join "", keys %ENTITY_TABLE; qr/[$chars]/};
...
$text =~ s/($entity_regex)/$ENTITY_TABLE{$1}/g; # encode entity characters in $text

The second example is from the cousin module Excel::ValueReader::XLSX. Here we are parsing the content of table in an Excel sheet, and the data is returned either in the form of a list of arrayrefs (plain values), or in the form of a list of hashrefs (column name => value), depending on an option given by the caller:

my $row = $args{want_records} ? do {my %r; @r{@{$args{columns}}} = @$vals; \%r}
                              : $vals;

If the caller wants records, the do block performs a hash slice assignment into a lexical hash variable to create a new record on the fly.

Wrapping up

Thanks to BLOCKs, lexical scoping can be introduced very flexibly almost anywhere in Perl code. The semantics of lexical variable and lexical pragmata cleanly defines that the lexical effect starts at the next statement after the declaration, and that it ends at exit from the block, without any of the surprises that we have seen in some other languages. The shadowing effect of lexical variables in inner scopes is easily understandable and consistent across all higher scopes, including the next englobing lexical scopes and the global module scope.

What a beautiful language design !

The next post will be about dynamic scoping through the local keyword - another, complementary way for temporarily changing the behaviour of the interpreter.

About the cover image

The picture is an excerpt from the initial movement of Verdi's Requiem, at a place where Verdi shadows several characteristics of the movement : for a short while, the orchestra stays still, leaving the choir a cappella, with a different speed, different tonality and different dynamics; then after this parenthesis, all parameters come back to their initial state, come prima as stated in the score.