Your services have dashboards, tracing, and alerting. Your CLI tools print to STDOUT and exit. When something breaks, debugging starts at the API gateway -- everything upstream is a black box. This makes no sense.

If your CLI talks to an API, it's part of the request path. Instrument it like any other participant.

This post describes how we instrumented an internal Perl CLI -- the same mycli tool from our earlier post on fatpacking -- with syslog logging, StatsD metrics, and correlation IDs. The post is strongly biased towards tooling internal to an organisation, which has the luxury of being opinionated: you control the deployment targets, you know where syslog goes, and you can lean on solved infrastructure rather than building your own. The principles generalise to any language and any CLI that talks to an API.

Why observability matters in CLI tools

Web services get dashboards as a matter of course[1]. Error rates, latency percentiles, request counts -- these are table stakes for any production service. CLI tools rarely get the same treatment, even when they're used just as heavily.

Once your CLI emits metrics, you can build per-tool dashboards that show error rates broken down by command, by user cohort, by API version, by CLI version, by deployment target. This is the same dimensional analysis you'd do for a web service, applied to a tool that runs on someone's laptop.

This integrates naturally with operational practices you're probably already using:

• Continuous deployment. When you ship a new CLI version, the dashboard shows whether error rates changed. If command.device_list.errors spikes after a release, you know immediately -- not when someone files a ticket three days later.
• Rollback decisions. If error rates climb after a release, the dashboard tells you in minutes -- roll back now, debug later. Without metrics, you're guessing whether the new version is the cause or a coincidence.
• Canary deployments. Roll the new version to 10% of jumpboxes. Compare http.timing and http.errors between the canary and the stable cohort. The same deployment strategy that works for services works for CLI tools, but only if you have the metrics to compare.
• Feature flags. If a new feature is gated behind a flag, metrics tell you whether the flagged code path is slower, more error-prone, or unused. Without instrumentation, feature flag decisions are based on "nobody complained".
• Incident management. During a site event, the CLI dashboard shows whether the tool is contributing to or affected by the problem. A spike in http.status.503 from the CLI tells the incident commander that the API is rejecting requests before users report it. Conversely, if the CLI error rate is flat during an incident, you can rule it out as a contributing factor.
• Adoption and deprecation. Metrics answer "is anyone still using the v1 endpoint?" and "has the team migrated to the new auth flow?" without surveys or guesswork.

The point is not that CLI tools are special -- it's that they're not. They're participants in the same distributed system as your services, and they deserve the same observability treatment. The investment is small: a correlation ID, a handful of counters, and a logging lifecycle. The return is that your CLI becomes a first-class citizen in your operational tooling rather than a blind spot.

[1] Yours does, right?

The three layers

We instrument at three levels, each serving a different audience and persistence model:

 Layer           Audience              Persistence
 -----           --------              -----------
 Verbose mode    Developer at terminal Ephemeral (STDERR)
 Syslog          Ops / incident review Durable (centralised logs)
 StatsD          Dashboards / alerting Aggregated (time-series)

A developer debugging their own command uses --verbose. An on-call engineer investigating a reported issue searches syslog by invocation ID. A platform team monitors command usage and error rates on dashboards. Same underlying data, different consumers, different retention.

Each layer is controlled independently and opt-in:

# Syslog only
MYCLI_LOG=1 mycli device list

# Verbose only (no syslog, no metrics)
mycli device list --verbose

# Everything
MYCLI_LOG=1 mycli device list --verbose

StatsD metrics emit whenever a statsd_host is configured -- no-ops otherwise. Syslog requires MYCLI_LOG=1 -- deliberately opt-in, since CLI tools run on personal machines and writing to syslog on every invocation without consent would be surprising.

The verbose layer itself has depth. --verbose shows the shape of the HTTP conversation -- method, URL, status, timing -- but deliberately omits headers and bodies to keep the output scannable. When that isn't enough, plugging in LWP::ConsoleLogger::Everywhere via perl -M gives a full HTTP trace without the CLI needing to build one. More on this in the debugging spectrum section below.

Invocation ID: the correlation key

Every mycli invocation generates a random 8-character hex ID at startup:

my @chars = ('0' .. '9', 'a' .. 'f');
my $id = join '', map { $chars[ int(rand @chars) ] } 1 .. 8;

This ID appears in three places:

1. Every syslog message -- prefixed as [f7a3b1c2]
2. Every HTTP request -- sent as the X-Invocation-Id header
3. Verbose STDERR output -- printed at startup

The server-side API logs this header alongside its own request ID. To trace a failing command end-to-end:

# Find the CLI side
grep 'f7a3b1c2' /var/log/mycli.log

# Find the server side
grep 'f7a3b1c2' /var/log/api.log

One string, full picture. No timestamps to correlate, no guessing which request came from which terminal.

User-Agent

In addition to the invocation ID, set the User-Agent header to mycli/<version>. This is trivial and gives the server side a way to filter by CLI version without any custom header support -- useful for canary deployment analysis and for spotting users running outdated versions.

Two-way correlation

The API returns its own request ID in a response header (X-Request-Id). The CLI logs this too:

[f7a3b1c2] http: 200 OK (142ms, application/json, 8431 bytes) req=a1b2c3d4

This gives you a join key in both directions: from the CLI's invocation ID you can find the server's request ID, and vice versa. When a user reports "mycli gave me an error", the request ID in the error message leads straight to the server-side trace.

What the server needs to do

The correlation only works if the server participates. The requirements are minimal:

1. Log the X-Invocation-Id header from incoming requests. Most API frameworks can do this with a single middleware or access log configuration change.
2. Return a request ID in every response (e.g., X-Request-Id). Many frameworks generate this by default.
3. Propagate both IDs into the server's own tracing and logging. If the API uses structured logging or distributed tracing, attach the invocation ID as a field or span attribute so it appears in the same search results.

If the server doesn't log the invocation ID, the CLI-side correlation still works (you can grep your CLI logs by invocation ID), but you lose the end-to-end join. If the server doesn't return a request ID, the CLI can still log its own invocation ID, but the user can't hand a request ID to the API team and say "look this up".

The ideal state is both: the CLI sends its ID, the server sends its ID, and both sides log both. This is a two-line change on the server and it makes every future debugging session faster.

Structured syslog

Every invocation logs a structured lifecycle to syslog:

Startup

[f7a3b1c2] startup: cli: mycli device list --status Active
[f7a3b1c2] startup: perl: 5.36.0 on linux
[f7a3b1c2] startup: env: API_KEY=ab12****, SERVER_URL=https://api.internal
[f7a3b1c2] config: key source: file (~/.config/mycli/api-key)
[f7a3b1c2] config: format: table, fields: all, tty

The API key is masked -- first four characters, then ****. Enough to identify which key is in use without leaking it to logs.

HTTP requests

[f7a3b1c2] http: GET https://api.internal/v1/devices
[f7a3b1c2] http: 200 OK (142ms, application/json, 8431 bytes) req=a1b2c3d4

Every request/response pair is logged with method, URL, status, elapsed time, content type, response size, and the server's request ID.

Shutdown

[f7a3b1c2] device_list: done (387ms, 24 results, 2 requests, cache 3/1)

One line summarising the entire command: wall-clock time, result count, number of HTTP requests made, and resolve cache statistics (3 items cached across 1 resource type).

Always format, conditionally emit

A subtle design choice: the logger always formats every message, even when logging is disabled. Only the syslog() call is conditional:

sub _emit {
    my ($self, $priority, $context, $detail) = @_;
    my $msg = sprintf '[%s] %s: %s', $self->{_id}, $context, $detail;
    syslog($priority, '%s', $msg) if $self->{_enabled};
    return $msg;
}

This means formatting bugs surface during normal development, not only when someone enables logging in production. The cost is negligible -- sprintf is fast.

A note on philosophy: when syslog is enabled, all levels are transmitted -- info, debug, error. There is no runtime knob to suppress debug messages. The belief behind this is that logging should always be on in production, not enabled after a problem is suspected. The time you most need debug-level detail is exactly the time you can't reproduce the issue. You can never have too much log detail, with the obvious exception of user or employee personal data, which should never be logged at any level.

What not to log

The API key masking (ab12****) is one example of a broader principle: log enough to identify, not enough to exploit.

• Credentials and secrets -- mask API keys, tokens, and passwords. Show enough characters to distinguish between keys (we show four), then mask the rest. Apply the same caution to environment variables and URL query parameters that may carry tokens.
• Request and response bodies -- don't log them. They may contain customer data, PII, or sensitive business logic. Log metadata (status, timing, size) but never content. Body inspection is what LWP::ConsoleLogger is for -- interactive, ephemeral, on-demand.

StatsD metrics

Every command emits a standard set of metrics to StatsD:

Per-command metrics

The command name is derived from the class hierarchy: MyCLI::App::Command::device::list becomes device_list.

Per-HTTP metrics

Operational metrics

What this tells you

The metrics answer questions that logs can't:

• What commands are people actually using? -- sort command.*.calls by count. If nobody uses crossconnect list, don't spend time improving it.
• Is the API getting slower? -- http.timing percentiles over time. The CLI is seeing the same latency as your users, including TLS negotiation and DNS.
• Are auth errors increasing? -- http.status.401 spike means keys are being rotated or revoked.
• How are people authenticating? -- auth.key_source.env vs auth.key_source.file tells you whether your team has adopted the recommended credential flow.
• What output formats matter? -- if 90% of usage is output.format.json, your table renderer is mostly aesthetic.

Metric naming conventions

Prefix every metric with the tool name (mycli.*) to avoid collisions in a shared StatsD instance. Use a consistent dot-separated hierarchy (mycli.command.<cmd>.calls) rather than flat names -- this makes metrics discoverable by browsing the tree. Watch cardinality: derive command names from a fixed set (like the class hierarchy) rather than user input, and keep dynamic segments like http.status.<code> to naturally bounded sets.

Verbose mode and the debugging spectrum

The three layers above cover durable observability -- data that outlives the terminal session. But the most common debugging scenario is someone at a keyboard wondering why their command isn't working. For this, the CLI has three levels of HTTP visibility:

Level 1: Silent (default)

No HTTP output. The user sees formatted results only. Syslog and metrics still capture everything in the background.

Level 2: --verbose

--> GET https://api.internal/v1/devices?status=Active
<-- 200 OK (142ms, application/json, 8431 bytes)

Printed to STDERR so it doesn't interfere with STDOUT piping. Shows method, URL, status, timing, and size. This is enough for "is my request hitting the right endpoint?" and "why is this slow?".

The design choice here is restraint. Verbose mode shows the shape of the conversation -- what was asked, what came back, how long it took. It deliberately omits headers and bodies. This keeps the output scannable when a command makes multiple requests.

Level 3: LWP::ConsoleLogger::Everywhere

When --verbose isn't enough -- when you need to see request headers, response headers, and full bodies -- plug in LWP::ConsoleLogger::Everywhere:

# From source
perl -MLWP::ConsoleLogger::Everywhere -Ilib bin/mycli device get 42

# Fatpacked binary (with API key redaction)
LWPCL_REDACT_HEADERS=Authorization \
  PERL5OPT="-MLWP::ConsoleLogger::Everywhere" \
  ./mycli-packed device get 42

This is a full HTTP trace: every header, every byte of the request and response body, formatted and syntax-highlighted. It's invaluable for debugging serialisation issues, unexpected headers, or auth failures.

The reason we don't build this into --verbose is that it's a different tool for a different job. Verbose mode is for operators; full HTTP tracing is for developers debugging the CLI itself. The -M flag means the capability is always available without cluttering the option namespace or adding a dependency that most users will never need.

Error reporting and surfacing correlation IDs

When the API returns an error, the CLI needs to show the user enough information to report the problem without overwhelming them with internals. Our error output includes the server's request ID:

Error: 403 Forbidden
  The API key does not have permission to access this resource.
  Request ID: a1b2c3d4

The request ID is the bridge between the user and the operations team. "It gave me a 403, request ID a1b2c3d4" is a complete bug report. The on-call engineer greps the server logs for a1b2c3d4, finds the full request context (authenticated user, requested resource, policy that denied access), and resolves the issue -- without asking the user to reproduce it, enable verbose mode, or paste terminal output.

The invocation ID doesn't appear in normal error output -- it's an internal correlation key for log analysis, not a user-facing artifact. If syslog is enabled, the invocation ID is already in the logs alongside the request ID, providing the join in both directions.

The execution wrapper

All of this comes together in the base command's execute() method, which wraps every leaf command:

sub execute {
    my ($self, $opt, $args) = @_;
    my $cmd     = $self->_metric_name;
    my $start   = Time::HiRes::time();

    $self->logger->info($cmd, 'start');
    $self->metrics->increment("command.$cmd.calls");

    eval { $self->_execute($opt, $args) };

    my $elapsed_ms   = int((Time::HiRes::time() - $start) * $MS_PER_SEC);
    my $requests     = $self->client->request_count;
    my $result_count = $self->{_result_count};

    $self->metrics->timing("command.$cmd.timing", $elapsed_ms);
    $self->metrics->gauge("command.$cmd.results", $result_count)
        if defined $result_count;

    if (my $err = $@) {
        $self->metrics->increment("command.$cmd.errors");
        $self->logger->error($cmd, $err);
        die $err;
    }

    $self->logger->info($cmd, sprintf 'done (%dms, %s results, %d requests)',
        $elapsed_ms, $result_count // 'n/a', $requests);
}

Leaf commands implement _execute() and don't think about observability at all. They call $self->client->get(...), render results, and return. The wrapper handles timing, logging, metrics, and error reporting. This is the single place where the observability contract is enforced -- no leaf command can accidentally skip it.

Design principles

A few principles that guided these choices:

1. Zero cost when off. Logging and metrics are lazy-initialised. If you never enable syslog or configure StatsD, the modules aren't even loaded.

2. Instrument the framework, not the features. Leaf commands don't contain observability code. The base command wrapper and HTTP client handle everything. New commands get full instrumentation for free.

3. Correlate by default. The invocation ID requires no opt-in. Every request carries it. The server just has to log it.

4. Separate concerns by audience. Verbose mode is for the person at the terminal. Syslog is for the person investigating after the fact. Metrics are for the person watching trends. Don't conflate them.

5. Don't build what you can plug in. Full HTTP tracing via LWP::ConsoleLogger is better than anything we'd build ourselves. Keep verbose mode lean and let the specialist tool handle the rest.

Testing observability

Instrumentation code is easy to write and easy to break silently. If nobody notices that the invocation ID stopped appearing in syslog, it might be months before an incident reveals the gap. A few testing strategies:

• Unit test the logger's formatting. The _emit method returns the formatted message even when syslog is disabled. Assert that the invocation ID, context, and detail appear in the expected format.
• Unit test metric emissions. Mock the StatsD client and assert that command.<cmd>.calls is incremented, command.<cmd>.timing receives a value, and command.<cmd>.errors fires on exception. These are contract tests -- they verify that the execution wrapper keeps its promises.
• Assert the invocation ID propagates. Mock the HTTP client and verify that outgoing requests carry the X-Invocation-Id header with the same value the logger is using.
• Integration test the full lifecycle. Run a command against a mocked API, capture STDERR with --verbose, and assert the --> / <-- lines appear with the expected method, URL, and status.

The "always format, conditionally emit" pattern helps here: the logger exercises all formatting code paths in every test run, even when syslog isn't available in the test environment.

Tracing an incident: a walkthrough

Here's how the instrumentation plays out during a real debugging scenario. This walkthrough exercises every layer described above: error output with a request ID, the metrics dashboard, syslog correlation, and two-way ID join.

A user reports: "mycli device list is failing intermittently." They include the error message:

Error: 503 Service Unavailable
  The API is temporarily unable to handle the request.
  Request ID: e4f5a6b7

Step 1: Find the server side. The on-call engineer greps the API logs for e4f5a6b7 and finds the request hit a backend that was in the middle of a deployment. The 503 was a transient error from a rolling restart.

Step 2: Assess the blast radius. But is it just this one user? The engineer checks the CLI dashboard: mycli.http.status.503 shows a spike over the last 20 minutes, coinciding with the deployment window. It's not one user -- it's everyone hitting that backend.

Step 3: Find the CLI side. The server log for e4f5a6b7 also contains the X-Invocation-Id: c8d9e0f1. Grepping the centralised CLI logs for c8d9e0f1 shows the full client-side context: which command was run, which user ran it, what arguments were passed, and that the request took 12 seconds before returning 503 (suggesting the backend was hanging, not failing fast).

Step 4: Verify the fix. After the deployment completes, the 503 counter drops to zero. The engineer confirms on the dashboard that error rates are back to baseline across all commands.

Total debugging time: minutes. Without instrumentation, this would have been a ticket saying "it's broken sometimes" followed by back-and-forth to reproduce, enable verbose mode, and collect output.

Summary

 +-------------------+    X-Invocation-Id    +-------------------+
 | mycli             |-----------------------| API               |
 |                   |    X-Request-Id       |                   |
 | - syslog [id]     |<----------------------| - access log [id] |
 | - StatsD metrics  |                       | - request trace   |
 | - verbose STDERR  |                       |                   |
 +-------------------+                       +-------------------+
         |                                           |
         v                                           v
 +-------------------+                       +-------------------+
 | Centralised logs  |<--- grep by ID ------>| Centralised logs  |
 | Metrics dashboard |                       | APM / tracing     |
 +-------------------+                       +-------------------+

Key takeaways:

1. Your CLI is part of the distributed system. If it talks to an API, it's a participant in the request path -- treat it like a service, not a script.
2. A correlation ID is the single most valuable thing you can add. One random string, sent as an HTTP header, ties client logs to server logs. Everything else builds on this.
3. Separate layers by audience. Verbose mode for the developer at the terminal, structured logs for the on-call engineer after the fact, metrics for dashboards and alerting. Same data, different consumers, different lifetimes.
4. Instrument the framework, not the features. A single execution wrapper gives every command logging, metrics, and error reporting for free. Leaf commands shouldn't contain observability code.
5. The server needs to participate. Log the client's invocation ID, return your own request ID. Without this, correlation is one-sided.
6. Log everything except secrets and personal data. Mask credentials, never log request bodies, and keep logging always on -- the time you need debug detail is the time you can't reproduce the issue.
7. Start simple, keep the door open. Wrap your logging backend so the rest of the codebase never touches it directly. Start with whatever works for your deployment targets today -- Sys::Syslog, Fluent::Logger, a file. When your infrastructure is ready for OpenTelemetry or wide events (see Appendix A), the swap is localised.

The investment is small: a correlation ID, a handful of counters, and a logging lifecycle. The return is that your CLI becomes a first-class citizen in your operational tooling rather than a blind spot.

References

• LWP::ConsoleLogger::Everywhere -- drop-in full HTTP tracing for any LWP-based client
• NO_COLOR -- convention for suppressing colour output, relevant to verbose/debug output formatting
• OpenTelemetry -- industry-standard observability framework; Perl SDK on CPAN
• StatsD -- the metrics aggregation protocol used for CLI instrumentation
• Shipping a Perl CLI as a single file with App::FatPacker -- companion post on building and distributing mycli

Getting started

If you want to add observability to an existing CLI tool, here's a practical order of operations. Each step is independently useful -- you don't need to do all five before any of them pay off.

1. Generate a random invocation ID at startup. Eight hex characters is enough. Send it as an X-Invocation-Id header on every HTTP request. This single change makes every future debugging session easier.
2. Set User-Agent to <tool>/<version>. Trivial, and it lets the server side filter by CLI version without any custom header support.
3. Log three lifecycle events. Startup (command line, environment, config source), each HTTP request/response (method, URL, status, timing), and shutdown (duration, result count). Even logging to STDERR behind a --debug flag is better than nothing.
4. Emit one counter per command invocation. If you have StatsD or a metrics collector, mycli.command.<cmd>.calls is the single most useful metric -- it tells you what people are actually using. If you don't have a metrics pipeline, a cheap alternative is to emit key=value pairs in your log lines (e.g. command=device_list duration_ms=387 status=ok) -- most log aggregation tools, including Grafana itself, can extract fields from these lines and build charts and dashboards without a separate metrics stack.
5. Wrap your command entry point. Move timing, logging, and metric emission into a single wrapper around leaf command execution. New commands get instrumentation for free, and no leaf command can accidentally skip it.

Appendix A: Wide events

Our implementation uses separate syslog lines for each lifecycle phase (startup, HTTP, shutdown) and separate StatsD counters for aggregation. This works, but it means correlating data across multiple log lines at query time -- you need the invocation ID to join them together.

An increasingly popular alternative is the wide event (or what Stripe called a canonical log line in 2019): a single, information-dense structured record emitted once per unit of work, containing every attribute you collected along the way. Instead of five syslog lines and ten StatsD counters, you emit one event with fields like command=device_list duration_ms=387 results=24 http_requests=2 http_status=200 auth_source=file output_format=table cache_hits=3.

The advantages are significant:

• Faster queries -- all the data is colocated in one record. No joins, no correlation by ID.
• Ad hoc analysis -- during an incident you can group by any combination of fields without having pre-defined a metric for it.
• Simpler pipeline -- one event replaces multiple log lines and multiple metric emissions. Less code, fewer failure modes.

We didn't take this approach because our logging infrastructure is syslog-based and doesn't support high-cardinality structured queries. If you have access to a columnar store (Honeycomb, ClickHouse, a data warehouse), wide events are the stronger choice. The execution wrapper already collects all the data in one place -- the change would be emitting it as a single structured record instead of spreading it across syslog and StatsD.

For more on wide events, see A Practitioner's Guide to Wide Events and All You Need Is Wide Events, Not Metrics.

Appendix B: Why Sys::Syslog and not a logging framework?

Perl has several mature logging frameworks -- Log::Any, Log::Dispatch, Log::Log4perl -- any of which would be a fine choice here. We went with Sys::Syslog directly. This is an opinionated trade-off worth explaining.

What Sys::Syslog gives you

Syslog is a solved problem on servers and jumpboxes. The local syslog daemon (rsyslog, syslog-ng, journald) handles buffering, rotation, compression, and forwarding to a central log aggregator. The CLI doesn't need to know where the logs go, how to authenticate to a remote endpoint, or what to do when the network is down. It calls syslog(), the daemon takes it from there. This is a clean separation of concerns: the application produces structured messages, the infrastructure handles transmission.

There are no extra dependencies beyond core Perl. No configuration files, no adapter registration, no output plugin selection. The logger module is ~50 lines. For a fatpacked binary where every dependency has a cost, this matters.

What a framework would give you

A framework like Log::Any or Log::Dispatch provides output abstraction: you write $log->info(...) and configure the destination at deployment time -- syslog, a file, STDERR, a network endpoint, or multiple at once. The application code doesn't change when the destination does. This is a genuine advantage when the tool runs in environments with different logging infrastructure, or when libraries you depend on already use Log::Any.

Where the trade-off bites

The opinionated choice of Sys::Syslog works well when every target machine runs a syslog daemon. It falls apart on developer laptops and desktops.

macOS ships with a syslog-compatible interface via Apple System Log, but the log viewer has moved to Console.app and the unified logging system. Messages from syslog() end up in a different place than most macOS users expect, and the retention policy may discard them quickly. On Windows, there is no syslog daemon at all.

You have two choices here:

Accept the gap. Detect the platform at startup and disable syslog on macOS and Windows. The CLI still has --verbose for interactive debugging, and StatsD metrics still flow if a collector is configured. You lose durable logging on developer machines, but you avoid adding complexity to the CLI itself. This is the approach we took -- the primary deployment targets are Linux servers and jumpboxes where syslog is reliable.

Solve logging everywhere. Use a framework like Log::Dispatch with pluggable outputs: syslog on Linux, a file on macOS, a network endpoint everywhere. This means the CLI now owns the full logging pipeline: transport selection, buffering when the destination is unavailable, possibly TLS for log data in transit, possibly client-side authentication to a log aggregator. Each of these is individually tractable, but collectively they add configuration surface, failure modes, and dependencies that the syslog approach avoids entirely.

There is a middle ground: In an organization with tight control of staff laptops and desktops (as is increasingly common), solving the logging problems in the CLI or having a local logging daemon is very feasible.

Another opinionated choice: Fluent::Logger

If your infrastructure runs Fluentd or Fluent Bit, Fluent::Logger is worth considering as an equally opinionated alternative to Sys::Syslog. It sends structured events directly to a Fluent collector over a local socket or TCP, which then handles routing, buffering, and delivery to whatever backend you use (Elasticsearch, S3, a data warehouse). Like Sys::Syslog, it delegates transport to purpose-built infrastructure. Unlike syslog, the events are natively structured -- key-value pairs rather than format strings -- which makes the path to wide events shorter.

The advantage of making an opinionated backend choice -- whether that's Sys::Syslog, Fluent::Logger, or something else entirely -- is that it removes abstraction layers that aren't adding value. If you know where your logs go, a framework like Log::Any is indirection without a benefit. You pay for adapter registration, output plugin configuration, and an extra dependency, but you only ever use one backend. An abstraction earns its keep when requirements are genuinely uncertain; when they're known, it's just ceremony.

The elephant in the room: OpenTelemetry

Of course, the industry is converging on OpenTelemetry as the standard answer to all of the above. Perl has solid support via the OpenTelemetry distribution on CPAN. If your organisation already runs an OTel collector, plumbing it into your CLI from the start is the right long-term bet.

Keeping the door open

The important thing is that the rest of the codebase never touches Sys::Syslog directly. Every module calls $self->logger->info(...), ->error(...), or ->debug(...). The actual syslog calls are isolated to two private methods in the logger class: _emit (which formats and transmits) and _open_syslog (which calls openlog). Swapping Sys::Syslog for Log::Dispatch, Fluent::Logger, or an OpenTelemetry log bridge would mean changing those two methods and nothing else.

This is the pragmatic middle path: start with the simplest backend that works for your deployment targets, but wrap it so the choice is easy to revisit. For a server-side CLI deployed to a controlled fleet, Sys::Syslog is a sensible default -- zero-config, zero-dependency, and delegates the hard problems to purpose-built infrastructure. If the tool later needs to run on developer laptops as a primary deployment target, the logging framework swap is a localised change rather than a rewrite.

Discussion

Have you plumbed observability into a CLI tool? I'd love to hear what worked and what didn't -- whether you went with OpenTelemetry traces, wide events from day one, or bolted logging on after the fact. What was the moment that made you invest in CLI instrumentation? Was it an incident that was hard to trace, a question about adoption you couldn't answer, or just good hygiene? And if you haven't done it yet -- what's holding you back?

A Side note: I'm currently on look out for a new contract or permanent opportunity. If you know of any relevant openings, I'd appreciate hearing from you. I am a proficient front-end developer also - email@lnation.org

If you've spent any time debugging Perl, you've used Data::Dumper. It's one of those modules that ships with every Perl installation, gets loaded into every debugging session, and does its job without complaint. But it also hasn't changed much in a long time. The output is monochrome, the internals are pure Perl, and code references remain opaque sub { "DUMMY" } blobs unless you enable Deparse and even then you do not get the response one would expect.

Loo is a new take on the same problem: dump Perl data structures to readable output, but do it in C, with colour, and with a built-in code deparser that walks the op tree directly.

Why Another Dumper?

Three reasons drove the creation of Loo:

Speed. Loo is implemented entirely in XS. The dump logic, string escaping, colour code generation, and op tree walking all happen in C. For large or deeply nested structures, this matters.

Colour out of the box. When your terminal supports it, Loo's output is coloured by default. Strings, numbers, hash keys, braces, blessed class names, regex patterns, and code all get distinct colours that can be customised. There's no separate module to install, no formatter to configure. It auto-detects terminal capability, respects $ENV{NO_COLOR}, and falls back to plain text when appropriate.

Code deparsing without B::Deparse. When you pass a code reference to Loo with deparsing enabled, it walks Perl's internal op tree in C and reconstructs the source. This is not a wrapper around B::Deparse — it's a standalone implementation that lives in the same XS compilation unit as the introspecter itself.

Getting Started

Loo provides a functional interface that mirrors Data::Dumper closely enough that switching is straightforward:

use Loo qw(Dump cDump ncDump dDump);

# Colour auto-detected based on terminal
print Dump({ name => 'Perl', version => 5.40 });

# Force colour on (useful when piping to a pager that supports ANSI)
print cDump([1, 2, 3]);

# Force colour off (useful for logging or file output)
print ncDump(\%ENV);

# Dump with code deparsing enabled
print dDump(sub { my ($x) = @_; return $x * 2 });

The OO interface supports method chaining and gives you access to the full set of configuration options:

my $loo = Loo->new([{ key => 'value' }], ['data']);
$loo->Indent(1)->Sortkeys(1)->Theme('monokai');
print $loo->Dump;

Data::Dumper Compatibility

Loo implements the same accessor interface as Data::Dumper: Indent, Terse, Varname, Useqq, Quotekeys, Sortkeys, Maxdepth, Maxrecurse, Purity, Deepcopy, Pair, Freezer, Toaster, Bless, Deparse, Sparseseen, and Pad. If you have existing code that configures a Data::Dumper object, the same method calls work on a Loo object and is faster.

Beyond that, Loo adds a few options that Data::Dumper doesn't have:

• Indentwidth($n) — control the number of characters per indent level (default 2)
• Usetabs($bool) — indent with tabs instead of spaces
• Trailingcomma($bool) — add trailing commas after the last element in arrays and hashes

Colour Customisation

Loo ships with four built-in themes: default, light (optimised for light terminal backgrounds), monokai, and none.

For fine-grained control, the Colour method accepts a hash specifying foreground and background colours for 17 distinct syntax elements:

$loo->Colour({
    string_fg  => 'green',
    key_fg     => 'magenta',
    number_fg  => 'cyan',
    brace_fg   => 'bright_black',
    undef_fg   => 'red',
});

The colorable elements cover every visual component of the output: string, number, key, brace, bracket, paren, arrow, comma, undef, blessed, regex, code, variable, quote, keyword, operator, and comment.

Colour codes are pre-computed once at configuration time, so there's no per-character overhead during the dump.

The Deparser

The most unusual feature of Loo is its built-in code deparser. When you enable deparsing, Loo walks the Perl op tree directly in C — the same internal structure that the Perl interpreter executes — and reconstructs Perl source code from it.

my $loo = Loo->new([\&Some::function]);
$loo->Deparse(1);
print $loo->Dump;

This means code references in your data structures are no longer black boxes. You see the actual code that will be run from the code. NOTE: It may not be identical to the code written as some postfix operations are lost as I am deparsing the compiled op tree itself.

Getting this to work across Perl versions was one of the harder parts of the project. The op tree structure has changed across Perl releases — OP_PADSV_STORE appeared in 5.38, OP_EMPTYAVHV landed in 5.36 as an enum rather than a #define, and PADNAME typedefs shifted between 5.20 and 5.22. Loo handles all of this with version-conditional compilation, supporting Perl 5.10 through to the latest releases.

Reusable C Headers

Loo's XS code is organised into modular C headers:

• loo.h — core definitions, themes, colour element names
• loo_colour.h — ANSI colour code generation
• loo_escape.h — string escaping
• loo_dump.h — recursive data structure dumping
• loo_deparse.h — op tree walking and code reconstruction

These headers are installed alongside the Perl module, and Loo->include_dir returns their path. This means other XS modules can reuse Loo's colour or escaping logic without duplicating the C code.

Auto-Detection Done Right

Loo follows the no-color.org convention and layers several checks to decide whether to emit ANSI codes:

1. If $Loo::USE_COLOUR is set, that takes precedence
2. If $ENV{NO_COLOR} is set, colour is disabled
3. If $ENV{TERM} is "dumb", colour is disabled
4. If STDOUT is not a terminal, colour is disabled
5. Otherwise, colour is enabled

This means Dump() does the right thing whether you're debugging interactively, piping to a file, or running in CI. And cDump() / ncDump() are there when you need to override.

Installation

Loo is available on CPAN:

cpanm Loo

It requires Perl 5.008003 or later and a C compiler (which you already have if you've ever installed an XS module).

Closing Thoughts

Data::Dumper is a workhorse that has served the Perl community well for decades. Loo isn't trying to replace it everywhere — but if you spend a lot of time reading dump output, colour and deparsing make a real difference. And if you're dumping large structures in production logging or tooling, the XS implementation gives you that output faster.

Give it a look. Your eyes may thank you.

Modern software distribution has converged on a simple idea: ship a self-contained artifact. Whether that means a statically linked binary, a container image, or a snap/flatpak, the benefits are the same -- dependency management is solved at build time, platform differences are absorbed, and upgrades and rollbacks reduce to swapping a single file.

Perl's App::FatPacker applies the same principle to Perl scripts. It bundles every pure-Perl dependency into a single executable file. No cpanm, no local::lib, no Makefile on the target -- just copy the file and run it. The technique is well-established -- cpm (the CPAN installer we use in the build) is itself distributed as a fatpacked binary.

The distribution pipeline looks like this:

 Code repo --> CI --> fatpack --> deploy --> laptops / jumpboxes / servers
                       |
                 single file,
                 no dependencies

This post walks through how we fatpacked an internal CLI we'll call mycli, a ~90-module Perl app, into a single file. The approach generalises to any App::Cmd-based tool.

A good practice for internal tools is to provide all three interfaces: a web frontend, an API, and a CLI. The web frontend is the easiest to discover; the API enables automation and integration; the CLI is the fastest path for engineers who live in a terminal. FatPacker makes the CLI trivially deployable.

mycli is a thin client -- it talks to an internal REST API over HTTPS and renders the response locally. There is no local state beyond a config file and environment variables. You could build an equivalent tool against a binary RPC protocol such as gRPC or Thrift -- the fatpacking approach is the same.

 +--------------------+           +-------------------+
 | Workstation        |   HTTPS   | Server            |
 |                    |           |                   |
 |  $ mycli resource  |---------->|  REST API ---+    |
 |    list ...        |<----------|  (JSON)  DB  |    |
 +--------------------+           +-------------------+

Despite being a thin client, mycli is not trivial. It includes:

• Pluggable output renderers (table, JSON, YAML, CSV, plain text)
• Colour output with NO_COLOR support
• Automatic pager integration (less -RFX) and pipe/TTY detection
• Activity spinner
• Multi-format ID resolution (numeric, UUID prefix, name lookup)
• Command aliases (ls/list, get/show)
• Config file discovery chain (env var, XDG path, dotfile)
• Timezone-aware timestamp rendering
• Structured syslog logging with per-invocation correlation IDs
• StatsD metrics instrumentation
• HTTP debugging hooks

All of this fatpacks cleanly because each feature is backed by pure-Perl modules.

This makes it an ideal fatpack candidate: the only XS dependency is Net::SSLeay for TLS, which is typically already present on the target system. Everything else is pure Perl.

Why FatPacker over PAR::Packer?

The other well-known option for single-file Perl distribution is PAR::Packer. PAR bundles everything -- including XS modules and even the perl interpreter itself -- into a self-extracting archive. At runtime it unpacks to a temp directory and executes from there.

FatPacker takes a different approach: modules are inlined as strings inside the script and served via a custom @INC hook. There is no extraction step, no temp directory, and no architecture coupling. The trade-off is that FatPacker only handles pure Perl -- XS modules must already be on the target.

For a thin REST client where the only XS dependency is Net::SSLeay, FatPacker wins on simplicity: the output is a plain Perl script, it starts instantly, and it runs on any architecture with a compatible perl. PAR is the better choice when you need to bundle XS-heavy dependencies or ship a binary to machines without Perl at all.

What fatpacking does

FatPacker prepends a BEGIN block to your script containing every dependency as a string literal, keyed by module path. A custom @INC hook serves these strings to require instead of reading from disk. The original script is appended unchanged.

$ wc -l bin/mycli mycli-packed
      13 bin/mycli
   48721 mycli-packed

That ~49k line file runs identically to the original, on any machine with Perl 5.24+.

The problem with naive fatpacking

The standard FatPacker workflow is:

fatpack trace bin/mycli
fatpack packlists-for $(cat fatpacker.trace) > packlists
fatpack tree $(cat packlists)
fatpack file bin/mycli > mycli-packed

This breaks for non-trivial apps because fatpack trace uses compile-time analysis (B::minus_c). It misses anything loaded at runtime via require:

• App::Cmd discovers commands via Module::Pluggable at runtime
• Text::ANSITable loads border styles and colour themes dynamically
• LWP::UserAgent loads protocol handlers on first request
• YAML::Any probes for available backends at runtime

If the trace misses a module, the packed binary dies with Can't locate Foo/Bar.pm in @INC at the worst possible moment.

The solution: a custom trace helper

Instead of relying on fatpack trace, we wrote a helper script that requires every module the app could ever load, then dumps %INC at exit. This captures the complete runtime dependency tree.

#!/usr/bin/env perl
# bin/trace-helper -- not shipped, build-time only
use strict;
use warnings;
use lib 'lib';

# Modules loaded lazily that fatpack misses
require Data::Unixish::Apply;
require Digest::SHA;
require HTTP::Request;
require LWP::UserAgent;
require String::RewritePrefix;

# Exercise objects to trigger deep runtime loads
{
    require Text::ANSITable;
    my $t = Text::ANSITable->new(use_color => 1, use_utf8 => 1);
    $t->border_style('UTF8::SingleLineBold');
    $t->color_theme('Text::ANSITable::Standard::NoGradation');
    $t->columns(['a']);
    $t->add_row(['1']);
    $t->draw;  # forces all rendering deps to load
}

# Every App::Cmd leaf command
require MyCLI::App;
require MyCLI::App::Command::device::list;
require MyCLI::App::Command::device::get;
# ... all 80+ command modules ...

END {
    open my $fh, '>', 'fatpacker.trace' or die $!;
    for my $inc (sort keys %INC) {
        next unless defined $INC{$inc};
        next if $inc =~ m{\AMyCLI/};  # our own modules come from lib/
        print $fh "$inc\n";
    }
}

Key points:

• Don't call ->run -- App::Cmd subdispatch will die on duplicate command names across namespaces. Just require every leaf.
• Exercise both code paths -- Text::ANSITable loads different modules for colour vs plain, UTF-8 vs ASCII. Instantiate both.
• Exclude your own namespace -- FatPacker embeds modules from fatlib/; your lib/ modules are embedded separately. Including them in the trace causes duplicates.

Forcing pure-Perl backends

FatPacker can only bundle pure Perl. Many popular modules ship dual XS/pure-Perl backends and prefer XS at runtime. If XS is available during the trace, the pure-Perl fallback won't appear in %INC and won't get bundled.

Force pure-Perl mode during the build:

# In the fatpack build script
export B_HOOKS_ENDOFSCOPE_IMPLEMENTATION=PP
export LIST_MOREUTILS_PP=1
export MOO_XS_DISABLE=1
export PACKAGE_STASH_IMPLEMENTATION=PP
export PARAMS_VALIDATE_IMPLEMENTATION=PP
export PERL_JSON_BACKEND=JSON::PP
export PUREPERL_ONLY=1

PUREPERL_ONLY=1 is a convention respected by many dual XS/PP distributions at install time, preventing XS compilation entirely. The per-module variables above cover modules that don't check PUREPERL_ONLY.

Combine this with --pp at install time to avoid pulling in XS at all:

cpm install -L local --target-perl 5.24.0 --pp

Pinning the target Perl version

The --target-perl flag to cpm is critical and easy to overlook. Without it, cpm resolves dependency versions against your build machine's Perl. If you're building on 5.38 but deploying to a jumpbox running 5.24, you'll silently install module versions that use postfix dereferencing, subroutine signatures, or other features that don't exist on the target.
The packed binary will fail at runtime with a syntax error -- far from the build where you could catch it.

This tells cpm's resolver to only consider module versions whose metadata declares compatibility with 5.24.0. Combined with perl -c as a post-install sanity check, this catches version mismatches before the slow trace step.

The complete build script

Here is the full pipeline, wrapped in a shell script. It supports incremental builds (reuses local/ and trace cache) and --clean for full rebuilds.

#!/bin/sh
set -e

CLEAN=0
[ "$1" = "--clean" ] && CLEAN=1

# 0. Prerequisites
for cmd in cpm fatpack perl; do
    command -v "$cmd" >/dev/null 2>&1 || {
        echo "Error: '$cmd' is not installed." >&2; exit 1
    }
done

export PERL_USE_UNSAFE_INC=1  # Perl 5.26+ removed . from @INC

# 1. Install deps (pure-perl only)
if [ "$CLEAN" = 1 ] || [ ! -d local/ ]; then
    rm -rf local/
    cpm install -L local --target-perl 5.24.0 --pp
fi

# 2. Set up paths
export PERL5LIB=$PWD/lib:$PWD/local/lib/perl5
export PATH=$PWD/local/bin:$PATH

# 3. Force pure-perl backends
export B_HOOKS_ENDOFSCOPE_IMPLEMENTATION=PP
export LIST_MOREUTILS_PP=1
export MOO_XS_DISABLE=1
export PACKAGE_STASH_IMPLEMENTATION=PP
export PARAMS_VALIDATE_IMPLEMENTATION=PP
export PERL_JSON_BACKEND=JSON::PP
export PUREPERL_ONLY=1

# 4. Verify compilation
perl -c bin/mycli || exit 1

# 5. Trace
if [ "$CLEAN" = 1 ] || [ ! -f fatpacker.trace ]; then
    perl -Ilib bin/trace-helper
    echo "Trace: $(wc -l < fatpacker.trace) modules"
fi

# 6. Pack
fatpack packlists-for $(cat fatpacker.trace) > packlists
fatpack tree $(cat packlists)

# Strip arch-specific dirs and non-essential files
rm -rf fatlib/$(perl -MConfig -e 'print $Config{archname}')
find fatlib -name '*.pod' -delete
find fatlib -name '*.pl'  -delete

# Bundle
fatpack file bin/mycli > mycli-packed
chmod +x mycli-packed
echo "Built mycli-packed ($(wc -c < mycli-packed) bytes)"

Step by step: what happens

1. Prerequisites -- verify cpm, fatpack, and perl are available
2. Install -- cpm installs all dependencies into local/ as pure Perl, targeting 5.24.0
3. Paths and env -- set PERL5LIB, PATH, and pure-Perl overrides
4. Compile check -- perl -c bin/mycli catches syntax errors before the slow trace step
5. Trace -- the helper script loads everything and writes the module list to fatpacker.trace
6. Packlists and tree -- fatpack packlists-for maps module names to installed packlist files; fatpack tree copies the .pm files into fatlib/
7. Clean up -- remove .pod, .pl, and arch-specific directories to reduce size
8. Bundle -- fatpack file inlines everything from fatlib/ into the script

Makefile integration

For teams that prefer make, add targets that delegate to the shell script:

# In Makefile.PL, inside MY::postamble
.PHONY: pack clean_fatpack

pack:
    ./fatpack

clean :: clean_fatpack

clean_fatpack:
    rm -rf fatlib fatpacker.trace packlists mycli-packed local/

Then building is just:

perl Makefile.PL
make pack

Adding a new dependency

When someone adds use Some::New::Module to the codebase, the fatpacked binary will break with Can't locate Some/New/Module.pm in @INC unless the build picks it up. The workflow is:

1. Add the module to cpanfile
2. If the module is loaded at runtime (via require or a plugin mechanism), add a require Some::New::Module line to the trace helper
3. Rebuild with --clean

./fatpack --clean

The --clean flag is important. Without it, the build reuses the cached local/ directory and fatpacker.trace from the previous run. The new module won't appear in either, and the packed binary will silently ship without it.

A good safeguard is to run perl -c mycli-packed after every build -- this catches missing modules at build time rather than in production.

What about perlstrip?

Perl::Strip can reduce the packed file by ~30% by removing comments, POD, and whitespace from bundled modules. We deliberately left it off. For an internal tool, the size saving (~1.7 MB) is not worth the trade-off: stripped files are harder to debug with stack traces, and perlstrip has a known issue corrupting files that contain use utf8.

Gotchas and tips

XS modules cannot be fatpacked

Modules with C extensions (.so/.xs) cannot be inlined. They must already exist on the target system. If your app has many XS dependencies, consider PAR::Packer instead (see above).

PERL_USE_UNSAFE_INC

Perl 5.26 removed . from @INC. Some older CPAN modules assume it's there during install or test. Set PERL_USE_UNSAFE_INC=1 during the build to avoid spurious failures. This only affects the build environment, not the packed binary.

Pinto / private CPAN

If your organisation runs a private CPAN mirror (Pinto, OrePAN2, etc.), point cpm at it with --resolver:

cpm install -L local --resolver 02packages,$PINTO_REPO --pp

Docker builds

FatPacker and Docker are complementary. Use Docker for the build environment (consistent Perl version, cpm, fatpack installed), and ship either the container image or just the packed file:

COPY mycli-packed /usr/local/bin/mycli
RUN chmod +x /usr/local/bin/mycli

Summary

The core recipe is three pieces:

1. A trace helper that loads every module your app could use at runtime, capturing the full dependency tree via %INC
2. Pure-Perl enforcement via environment variables and cpm --pp
3. The standard fatpack pipeline: packlists, tree, clean up, bundle

The result is a single file you can scp to any box with Perl 5.24+ and run immediately. No CPAN, no Makefile, no containers required.

References

• App::FatPacker on CPAN
• FatPacking Perl applications -- talk by Andrew Rodland covering the core technique, pure-Perl enforcement, and cpm
• arodland/swr fatpack script -- a clean, minimal reference implementation of the full pipeline
• App::cpm -- fast CPAN installer (itself shipped as a fatpacked binary); --target-perl and --pp flags are essential for fatpack builds

Originally published at Perl Weekly 766

Hi there,

This week's Perl landscape firmly establishes that while the history of Perl is rich and exciting, it is also a place for experimentation and innovation in the future. There have been handful of releases of Perl v5.43.9 which came up with plenty of changes and major one for me was the enhanced /xx pattern modifier. In between there was another very important patch was released, Perl v5.42.2-RC1, and Perl v5.40.4 addressing the vulnerability in Compress::Raw::Zlib. Don't dare call Perl is dead.

Ever worked with XS modules? Well we have three related XS modules that made it looks so simple and easy. The benefit of XS helps creating efficient and high speed unique identifier creation using Horus, Apophis, and Sekhmet. Bonus, you get to see how they can be used together. Robert seems to be on the roll with his another gem, Eshu, a code formatter written entirely in C and exposed to Perl through XS.

Not everything have to be, XS. Dave showed how you can work with TOON (Token-Oriented Object Notation), textual format for representing structured data, same data model as JSON. Using his new creation TOON, one can easily work with TOON data model. If you are XS fan, feel free to create XS version of TOON.

Do you use Java? If yes then you now have the choice of using Perl power inside Java. The project, PerlOnJava, gives us handy tool to get the job done: jperl, jcpan, and jprove.

The week was fun, too much to handle in such a short time but I am not complaining. I am finding it hard to keep up, how about you?

Enjoy rest of the newsletter.

--
Your editor: Mohammad Sajid Anwar.

Announcements

TPRF Board Announces the 2025 Annual Report

The Board is pleased to share the 2025 Annual Report from the The Perl and Raku Foundation.

Articles

Beautiful Perl feature: "heredocs", multi-line strings embedded in source code

This article on the Beautiful Perl Feature - Heredocs and MultiLine Strings provides a nice introductory overview of how to use Perl's heredoc syntax to create readable, maintainable multiline text. It provides practical examples combined with a simple explanation which allows experienced programmers and novices alike to have a fresh look at an item that has been around for many years.

Perl, the Strange Language That Built the Early Web

The unusual language that made the early web; a glimpse at the history of Perl less than truly alien to the average user; The original dynamic/interactive media for the internet; with its contribution to automation processes (primarily text); through CGI scripting - both technically and culturally; In terms of practicality or versatility; play a significant role in creating and supporting how the first wave of web interactivity was created and how it became an integral part of the early days of the world wide web.

Horus, Apophis, and Sekhmet: An C/XS Identifier Stack for Perl

This post focuses on three related XS modules for efficient and high speed unique identifier creation (UUID, ULID and deterministic) and content-addressable storage in Perl. It provides a comprehensive overview of how to use these tools in conjunction with each other to create an efficient and scalable unique ID workflow. It also demonstrates how they can be used together.

Eshu: Indentation Fixer for Eight Languages, Written in C

This article discusses a portable C-based program that formats code and will uniformly line up the indentation across eight different programming languages. It describes examples to show you how Eshu can help you make the indentation to your programming code consistent with very little effort and no extra heavy duty tools required. For Developers who choose to use other than traditional language-specific formattors, this document presents an overview of how Eshu creates a lightweight formatting solution that developers may find useful.

Writing a TOON Module for Perl

The article presents TOON (Token-Oriented Object Notation) which aims to be simple for both people and LMs to construct and understand while using as few punctuation marks as possible and maintaining an easily accessible structure of data. It also discusses the reasons why TOON will be beneficial and provides a Perl implementation module for TOON with a familiar interface to those that have used JSON.pm.

CPAN

Graphics::Toolkit::Color 2.0 feature overview

The Graphics::Toolkit::Color 2.0 feature overview post provides an impressive look at all of the most significant improvements that have been made in developing GTC 2.0. The description outlines how GTC has grown beyond only doing basic coloring routines to include now a much richer, more complex, multi-space colored library complete with the ability to create beautiful gradients, accurately measure colors for perceptual purposes, and a variety of tools for use by both designers and developers. Overall, this is a succinct overview that does an excellent job of showcasing the reasons why GTC 2.0 is a unique addition to CPAN.

PerlOnJava Gets a CPAN Client

This is a great update regarding the addition of native CPAN support to Perl-on-JVM tooling. The example uses the ability to use an already developed CPAN client for installing modules and accessing the overall CPAN ecosystem in a more natural way than would be done with the non-JVM versions of the clients. It gives many real-world examples and is an excellent source of information for those who want to connect Perl and Java.

Lingua::* - From 17 to 61 Languages: Resurrecting and Modernizing PetaMem's Number Conversion Suite

The blog entry, "Lingua Revival", is an interesting way to reintroduce Lingua by combining elements of memories with new features that apply to modern day Perl. The story is easy to follow and focuses on being usable in today's world, which will be beneficial to both long-time users and new users of the project.

The Weekly Challenge

The Weekly Challenge by Mohammad Sajid Anwar will help you step out of your comfort-zone. You can even win prize money of $50 by participating in the weekly challenge. We pick one champion at the end of the month from among all of the contributors during the month, thanks to the sponsor Lance Wicks.

The Weekly Challenge - 367

Welcome to a new week with a couple of fun tasks "Max Odd Binary" and "Conflict Events". If you are new to the weekly challenge then why not join us and have fun every week. For more information, please read the FAQ.

RECAP - The Weekly Challenge - 366

Enjoy a quick recap of last week's contributions by Team PWC dealing with the "Count Prefixes" and "Valid Times" tasks in Perl and Raku. You will find plenty of solutions to keep you busy.

Count the Times

Raku Musings has a clearly written "Count the Times" post that gives a well-organised overview. It shows how idiomatic features work together effectively in Raku, resulting in a clear and elegant solution. There is an excellent balance between compact code and an informative explanation. The post demonstrates the use of expressive constructs that lend themselves to solving this type of problem using Raku.

Could We Start Again, Please

Bob Lied writes an engaging post about a problem in a clear manner, interspersing logic with humor; making it a pleasure to read! The author reviews alternative methods and their advantages/disadvantages and demonstrates a practical approach as well as demonstrating good Perl coding skill.

Valid Times

Bob Lied's "Valid Times" post systematically breaks down the issue into distinct steps while also providing significant attention to detail in regards to possible edge cases and practical limitations of validation of times. It presents a succinct but complete Perl code and corresponds with sound logic behind the choice of this Perl implementation, which allows readers to follow along easily and use as a reference when addressing the same type of parsing problems.

Perl Weekly Challenge: Week 366

The Jaldhar's blog has written an extensive, interesting post detailing how to perform Week 366 tasks. It does so by separating them into two sections: Problem 1 and Problem 2. This helps readers easily understand the problems themselves, as well as providing a clear path to solution using Perl. The blog also clearly states the logic behind each step, allowing readers to learn from the blogs experience while still being able to easily move on to solving this week's challenges independently.

Pre-Timed Counters

The blog post written for the week of 366 by Jörg, presents an elegant solution that exemplifies the use of clear and concise Perl programming techniques with a command of idiomatic constructs. The reader will appreciate Sommrey's clean, logical approach in solving the problem space and his appreciation for the use of expressive and efficient code, which reflects both familiarity and appreciation for the inherent beauty of programming.

what time is it?

Luca Ferrari's post is a further example of his continuing theme of approaching Perl Weekly Challenge in an analytical as well as exploratory way; frequently developing solutions in several languages and platforms to help him better understand the challenges. Luca's posts provide very useful instruction/examples; help you learn through experimentation/experience; and help you to truly think about and re-examine/consider the real-world nature of the solution.

Perl Weekly Challenge 366

The write-up gives a very reasoned overview of the problem with easy-to-follow methods of solving it using logical thinking. There is a good amount of coding as well as thorough explanations that create a valid and helpful source for those searching for an understanding of this issue and its methods of resolution using Perl.

Counting Times Without Questions

The article from Matthias Muth, entitled Matthias Muth's Week 366, is a clear and precise description of his thought processes relating to solutions presented in a concise manner while also being well-organised so as to make it easy to understand the underlying concept. It has an elegant and idiomatic style similar to that of Matthias' other contributions to the Perl Weekly Challenge, and it has a very clean decomposition of the problem that allows more experienced readers to develop an appreciation for it.

The Times They Are A-Countin'

The entertaining narrative of Packy Anderson's post combines humor and solid technical approaches to a problem to keep readers interested and provide them with an enjoyable and intuitive experience involved in the solution. His creative approach to framing the challenge and providing clear examples of how to solve it makes for a positive experience for all.

Prefixes and times?

Peter Campbell Smith's Week 366 Write-up provides an unambiguous, pragmatic solution style representing a strong real-world Perl mindset. The emphasis is placed on solving the problem in an accurate and efficient manner through simple implementation methods. The provided solution is straightforward and effective; he understands the relevant tasks thoroughly and prefers to solve issues clearly and without complexity (typical of all Weekly Challenges).

The Weekly Challenge - 366: Count Prefixes

The writing style used by Reinier Maliepaard in his submission demonstrates a logical and coherent framework and logical correctness; Making it easy for the reader to follow. Reinier’s structure of writing reflects discipline and analytic thought, along with succinctness, resulting in a Combination of Clear and Robust Perl Source Code, which matches the strategy of problem-solving as well.

The Weekly Challenge - 366: Valid Token Counter

The Week 366 second post by Reinier Maliepaard provides another example of his methodical and rational approach to problem-solving with a detailed logical breakdown along with concise Perl code to solve the problem. The article focuses on providing a clear, correct, and easily read explanation of how to work through validation problems, offering users of all skill levels an accessible, educational account.

The Weekly Challenge #366

Robbie Hatley's Week 366 Answers includes usable Perl solutions, as well as easy to follow logical documentation for each step of reasoning. What is accomplished is a practical, understandable solution. While the solutions provide a clear method to convey both the "how-to" and "why" of developing the final product, they also teach the reader to think through each implementation logically.

The Time of the Count is Over

Week 366 of Roger's post is an impressive example of multi-language exploration with Ruby, Lua, PostScript and Raku; it illustrates how to solve problems in Perl and develop cross-language thinking by presenting examples of various programming paradigms solving the same problem as well as having clear and entertaining explanatory text.

Happy 7th birthday TWC!

Simon Green's 7th Anniversary Post for The Weekly Challenge is an amazing, heartfelt reflection of how far we've come as a community over the past seven years, combining his personal experience with his deep appreciation for all the amazing contributors & readers to our community. It's an uplifting and well-written post that captures the essence of what The Weekly Challenge is about and how it's positively affected our lives.

Rakudo

2026.12 Ich bin ein Berliner

Weekly collections

NICEPERL's lists

Great CPAN modules released last week;
MetaCPAN weekly report.

Events

Perl Maven online: Testing in Perl - part 2

April 2, 2026

Perl Maven online: Testing in Perl - part 3

April 9, 2026

Perl Toolchain Summit 2026

April 23-26, 2026

The Perl and Raku Conference 2026

June 26-29, 2026, Greenville, SC, USA

You joined the Perl Weekly to get weekly e-mails about the Perl programming language and related topics.

Want to see more? See the archives of all the issues.

Not yet subscribed to the newsletter? Join us free of charge!

(C) Copyright Gabor Szabo
The articles are copyright the respective authors.

Introduction

Since I last wrote a XS tutorial my knowledge within C has increased this has come from improvements in LLM software that has assisted in improving my knowledge where previously I would be stuck. This knowledge and software has since enabled me to craft more elegant and efficient XS implementations.

Today I will share with you my technique for writing reusable C/XS code.

One of the most powerful patterns in XS development is writing your core logic in pure C header files. This gives you:

• Zero-cost reuse - no runtime linking, no shared libraries, just a #include line.
• No Perl dependency in the C layer - your headers work in any C project
• Compile-time inlining - the compiler sees everything, optimises aggressively
• Simple distribution - headers are installed alongside the Perl module via PM

This tutorial walks through the complete pattern step by step, using a minimal working example you can build and run yourself.

The Example

We will create two distributions:

1. Abacus - a provider distribution that ships a reusable pure-C abacus_math.h header containing simple arithmetic functions
2. Tally - a consumer distribution that #includes the Abacus header to build its own XS module, without duplicating any C code

As always lets start by creating the distributions that we will need for this tutorial. Open your terminal and run module-starter. If you are using a modern version of Module::Starter then the command has changed slightly since my last posts.

  module-starter --module=Abacus --author="LNATION <email@lnation.org>"
  module-starter --module=Tally --author="LNATION <email@lnation.org>"

Part 1: The Provider Distribution (Abacus)

Write the pure-C header

This is the reusable part. It has zero Perl dependencies - just standard C.

Now enter the Abacus and create the include directory:

  cd Abucus
  mkdir include

Then create a new file abacus_math.h:

  touch abacus_math.h
  vim abacus_math.h

Paste the following code into the file:

#ifndef ABACUS_MATH_H
#define ABACUS_MATH_H

/*
 * abacus_math.h - Pure C arithmetic library (no Perl dependencies)
 *
 * This header is the reusable entry point for any C or XS project
 * that needs basic arithmetic operations. It has ZERO Perl/XS
 * dependencies.
 *
 * Usage from another XS module:
 *
 *     #include "abacus_math.h"
 *
 * Build: add -I/path/to/Abacus/include to your compiler flags.
 */

#include <stdint.h>

/* ── Error handling hook ─────────────────────────────────────────
 *
 * Consumers can #define ABACUS_FATAL(msg) before including this
 * header to route errors through their own mechanism.
 *
 * In an XS module you would typically do:
 *
 *     #define ABACUS_FATAL(msg) croak("%s", (msg))
 *     #include "abacus_math.h"
 *
 * In plain C the default behaviour is fprintf + abort.
 */
#ifndef ABACUS_FATAL
#  include <stdio.h>
#  include <stdlib.h>
#  define ABACUS_FATAL(msg) do { \
       fprintf(stderr, "abacus fatal: %s\n", (msg)); \
       abort(); \
   } while (0)
#endif

/* ── Arithmetic operations ───────────────────────────────────── */

static inline int32_t
abacus_add(int32_t a, int32_t b) {
    return a + b;
}

static inline int32_t
abacus_subtract(int32_t a, int32_t b) {
    return a - b;
}

static inline int32_t
abacus_multiply(int32_t a, int32_t b) {
    return a * b;
}

static inline int32_t
abacus_divide(int32_t a, int32_t b) {
    if (b == 0) {
        ABACUS_FATAL("division by zero");
    }
    return a / b;
}

static inline int32_t
abacus_factorial(int32_t n) {
    int32_t result = 1;
    int32_t i;
    if (n < 0) {
        ABACUS_FATAL("factorial of negative number");
    }
    for (i = 2; i <= n; i++) {
        result *= i;
    }
    return result;
}

#endif /* ABACUS_MATH_H */

The code above demonstrates three critical design patterns for reusable C headers:

static inline functions eliminate linker complications by giving each translation unit its own copy of the function. The compiler can then inline these small arithmetic operations directly into the call site, producing zero-overhead abstractions. This is key to the "zero-cost reuse" principle—there is no shared library dependency, no function call overhead, just pure generated code.

The ABACUS_FATAL macro hook provides a customization point for error handling. By default, it calls fprintf() and abort() in standalone C programs; but consumers can #define ABACUS_FATAL(msg) croak("%s", (msg)) before including the header to integrate seamlessly with Perl's exception system. This single mechanism allows the same C header to work across Perl XS, plain C, and other environments without code duplication.

The use of only stdint.h integers and no Perl types ensures the header remains truly portable. There are no SV* pointers, no pTHX context variables, no XSUB.h includes—just standard C99 types. This purity is what allows the header to be #included into any C or XS project without creating hidden Perl dependencies at the C layer.

Write the Perl-facing XS header

Next we will add another header which will hold the Perl/XS specific logic. Inside the include directory create a new file called abacus.h. The rational behind this thin wrapper is to pull in Perl's headers and sets up the ABACUS_FATAL macro to use croak(). To reiterate only a XS distribution should include this header, whereas abacus_math.h is generic and could be used by other languages which bind C.

    touch abacus.h
    vim abacus.h

Paste the following code into the file

#ifndef ABACUS_H
#define ABACUS_H

/*
 * abacus.h - Perl XS wrapper header for the Abacus library
 *
 * This header sets up Perl-specific error handling and includes
 * the pure C core library.
 *
 * For reuse from OTHER XS modules without Perl overhead, include
 * abacus_math.h directly instead (see that header for usage).
 */

#define PERL_NO_GET_CONTEXT
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"

/* Route fatal errors through Perl's core croak() */
#define ABACUS_FATAL(msg) croak("%s", (msg))

/* Pull in the pure-C library */
#include "abacus_math.h"

#endif /* ABACUS_H */

Now any .xs file can #include "abacus.h" and get the full Perl/XS environment plus all the pure-C functions, with errors properly integrated.

Write the XS file

Next we will create the XS file, return to the root directory and then enter the lib directoy where you should see the Abacus.pm file already. Create a new XS file called Abacus.xs. This will be the glue that exposes the C functions to Perl.

  cd ../lib
  touch Abacus.xs
  vim Abacus.xs

#include "abacus.h"

MODULE = Abacus  PACKAGE = Abacus

PROTOTYPES: DISABLE

int
add(a, b)
    int a
    int b
  CODE:
    RETVAL = abacus_add(a, b);
  OUTPUT:
    RETVAL

int
subtract(a, b)
    int a
    int b
  CODE:
    RETVAL = abacus_subtract(a, b);
  OUTPUT:
    RETVAL

int
multiply(a, b)
    int a
    int b
  CODE:
    RETVAL = abacus_multiply(a, b);
  OUTPUT:
    RETVAL

int
divide(a, b)
    int a
    int b
  CODE:
    RETVAL = abacus_divide(a, b);
  OUTPUT:
    RETVAL

int
factorial(n)
    int n
  CODE:
    RETVAL = abacus_factorial(n);
  OUTPUT:
    RETVAL

As you can see we include abacus.h which pulls in all the C that we need to create our XS module. We then define add, subtract, multiply, divide and factorial as XSUBs. As you should know by now XSUBs can be called directly from your perl code.

Write the Perl module with include_dir()

Next open the pm file and update to add an exporter for the XSUBS we have just created.

package Abacus;

use 5.008003;
use strict;
use warnings;

our $VERSION = '0.01';

use Exporter 'import';
our @EXPORT_OK = qw(add subtract multiply divide factorial);

require XSLoader;
XSLoader::load('Abacus', $VERSION);

Now the critical piece that makes header sharing work. A include_dir() method which returns the path to the installed headers so that consumer distributions can find them at build time.

sub include_dir {
    my $dir = $INC{'Abacus.pm'};
    $dir =~ s{Abacus\.pm$}{Abacus/include};
    return $dir;
}

1;

How include_dir() works:

1. When Perl loads Abacus.pm, it records the full path in %INC (e.g. /usr/lib/perl5/site_perl/Abacus.pm)
2. include_dir() replaces Abacus.pm with Abacus/include
3. That directory exists because Makefile.PL installs the headers there (see next step)

Write the Makefile.PL that installs headers

The PM hash is what makes headers available to other distributions after install. It maps source files to their installation destinations.

Abacus/Makefile.PL

use 5.008003;
use strict;
use warnings;
use ExtUtils::MakeMaker;

WriteMakefile(
    NAME             => 'Abacus',
    AUTHOR           => 'Your Name <you@example.com>',
    VERSION_FROM     => 'lib/Abacus.pm',
    ABSTRACT_FROM    => 'lib/Abacus.pm',
    LICENSE          => 'artistic_2',
    MIN_PERL_VERSION => '5.008003',
    CONFIGURE_REQUIRES => {
        'ExtUtils::MakeMaker' => '0',
    },
    TEST_REQUIRES => {
        'Test::More' => '0',
    },
    PREREQ_PM => {},
    XSMULTI => 1,
    # XS configuration
    INC    => '-I. -Iinclude',
    OBJECT => '$(O_FILES)',

    # *** THIS IS THE KEY PART ***
    # Install headers alongside the module so dependent
    # distributions can find them via Abacus->include_dir()
    PM => {
        'lib/Abacus.pm'           => '$(INST_LIB)/Abacus.pm',
        'include/abacus.h'        => '$(INST_LIB)/Abacus/include/abacus.h',
        'include/abacus_math.h'   => '$(INST_LIB)/Abacus/include/abacus_math.h',
    },

    dist  => { COMPRESS => 'gzip -9f', SUFFIX => 'gz' },
    clean => { FILES => 'Abacus-*' },
);

The PM hash does two things:

1. Installs Abacus.pm as normal
2. Copies the header files into Abacus/include/ alongside the module

After make install, the filesystem looks like:

site_perl/
  Abacus.pm
  Abacus/
    include/
      abacus.h
      abacus_math.h

Write a test

Abacus/t/01-basic.t

use strict;
use warnings;
use Test::More;
use Abacus qw(add subtract multiply divide factorial);

is(add(2, 3),       5,   'add');
is(subtract(10, 4), 6,   'subtract');
is(multiply(3, 7),  21,  'multiply');
is(divide(20, 4),   5,   'divide');
is(factorial(5),     120, 'factorial');

eval { divide(1, 0) };
like($@, qr/division by zero/, 'divide by zero croaks');

eval { factorial(-1) };
like($@, qr/negative/, 'negative factorial croaks');

done_testing;

Build and install Abacus

cd Abacus
perl Makefile.PL
make
make test
make install          # installs headers into site_perl

Part 2: The Consumer Distribution (Tally)

Tally is a separate distribution that reuses Abacus's C arithmetic without duplicating any code. It adds its own "running total" functionality on top.

Write the Makefile.PL that finds Abacus headers

This is where the consumer locates the provider's headers. The two-step resolution strategy supports both installed (CPAN) and development (sibling
directory) scenarios.

Tally/Makefile.PL

use 5.008003;
use strict;
use warnings;
use ExtUtils::MakeMaker;

# Resolve Abacus include directory:
#   1. Try installed Abacus module (CPAN / system)
#   2. Fall back to sibling directory (development)
my $abacus_inc;
eval {
    no warnings 'redefine';
    local *XSLoader::load = sub {};  # skip XS bootstrap
    require Abacus;
    my $dir = Abacus->include_dir();
    $abacus_inc = $dir if $dir && -d $dir;
};
if (!$abacus_inc && -d '../Abacus/include') {
    $abacus_inc = '../Abacus/include';
}
die "Cannot find Abacus include directory.\n"
  . "Install Abacus or place it as a sibling directory.\n"
    unless $abacus_inc;

WriteMakefile(
    NAME             => 'Tally',
    AUTHOR           => 'Your Name <you@example.com>',
    VERSION_FROM     => 'lib/Tally.pm',
    ABSTRACT_FROM    => 'lib/Tally.pm',
    LICENSE          => 'artistic_2',
    MIN_PERL_VERSION => '5.008003',
    CONFIGURE_REQUIRES => {
        'ExtUtils::MakeMaker' => '0',
        'Abacus'              => '0.01',
    },
    TEST_REQUIRES => {
        'Test::More' => '0',
    },
    PREREQ_PM => {
        'Abacus' => '0.01',
    },

    # Point the compiler at Abacus's installed headers
    INC    => "-I$abacus_inc",
    OBJECT => '$(O_FILES)',

    dist  => { COMPRESS => 'gzip -9f', SUFFIX => 'gz' },
    clean => { FILES => 'Tally-*' },
);

Let's walk through the header resolution:

1. Try the installed path first - require Abacus loads the module, then Abacus->include_dir() returns the path where the headers were installed. We stub out XSLoader::load because we only need the pure-Perl include_dir() method, not the XS functions.
2. Fall back to sibling directory - during development, Abacus and Tally often live side by side. ../Abacus/include handles this case.
3. Die with a clear message if neither path works.

The resolved path is passed to INC, which adds it to the C compiler's include search path (-I/path/to/Abacus/include).

Abacus is listed in both CONFIGURE_REQUIRES and PREREQ_PM:

• CONFIGURE_REQUIRES ensures Abacus is installed before Makefile.PL runs (needed because we require Abacus at configure time)
• PREREQ_PM ensures it is available at runtime too

Write the XS file

This is where the reuse happens. Tally includes abacus_math.h directly -
no Perl coupling, just pure C function calls.

Tally/Tally.xs

#define PERL_NO_GET_CONTEXT
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

/* Hook Abacus errors into Perl's croak() */
#define ABACUS_FATAL(msg) croak("%s", (msg))

/* Include the pure-C header from Abacus - no Perl deps in the header */
#include "abacus_math.h"

/* ── Tally's own C logic, built on top of Abacus ─────────────── */

typedef struct {
    int32_t total;
} tally_state_t;

static inline void
tally_init(tally_state_t *state) {
    state->total = 0;
}

static inline int32_t
tally_add(tally_state_t *state, int32_t value) {
    state->total = abacus_add(state->total, value);
    return state->total;
}

static inline int32_t
tally_subtract(tally_state_t *state, int32_t value) {
    state->total = abacus_subtract(state->total, value);
    return state->total;
}

static inline int32_t
tally_multiply_total(tally_state_t *state, int32_t value) {
    state->total = abacus_multiply(state->total, value);
    return state->total;
}

static inline int32_t
tally_get(tally_state_t *state) {
    return state->total;
}

static inline void
tally_reset(tally_state_t *state) {
    state->total = 0;
}

/* ── XS bindings ─────────────────────────────────────────────── */

MODULE = Tally  PACKAGE = Tally

PROTOTYPES: DISABLE

SV *
new(class)
    const char *class
  CODE:
    tally_state_t *state;
    Newxz(state, 1, tally_state_t);
    tally_init(state);
    RETVAL = newSV(0);
    sv_setref_pv(RETVAL, class, (void *)state);
  OUTPUT:
    RETVAL

int
add(self, value)
    SV *self
    int value
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    RETVAL = tally_add(state, value);
  OUTPUT:
    RETVAL

int
subtract(self, value)
    SV *self
    int value
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    RETVAL = tally_subtract(state, value);
  OUTPUT:
    RETVAL

int
multiply_total(self, value)
    SV *self
    int value
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    RETVAL = tally_multiply_total(state, value);
  OUTPUT:
    RETVAL

int
total(self)
    SV *self
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    RETVAL = tally_get(state);
  OUTPUT:
    RETVAL

void
reset(self)
    SV *self
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    tally_reset(state);

void
DESTROY(self)
    SV *self
  CODE:
    tally_state_t *state = INT2PTR(tally_state_t *, SvIV(SvRV(self)));
    Safefree(state);

Notice that Tally includes abacus_math.h (the pure C header), not abacus.h (the Perl-facing wrapper). This is intentional - Tally has its own Perl/XS setup and only needs the C functions.

Write the Perl module

Tally/lib/Tally.pm

package Tally;

use 5.008003;
use strict;
use warnings;

our $VERSION = '0.01';

require XSLoader;
XSLoader::load('Tally', $VERSION);

1;

__END__

=head1 NAME

Tally - Running total calculator using Abacus C headers

=head1 SYNOPSIS

    use Tally;

    my $t = Tally->new;
    $t->add(10);        # total is now 10
    $t->add(5);         # total is now 15
    $t->subtract(3);    # total is now 12
    $t->multiply_total(2);  # total is now 24
    say $t->total;      # 24
    $t->reset;          # back to 0

=cut

Write a test

Tally/t/01-basic.t

use strict;
use warnings;
use Test::More;

use_ok('Tally');

my $t = Tally->new;
isa_ok($t, 'Tally');

is($t->total, 0, 'starts at zero');

is($t->add(10), 10, 'add 10');
is($t->add(5),  15, 'add 5');
is($t->subtract(3), 12, 'subtract 3');
is($t->multiply_total(2), 24, 'multiply by 2');
is($t->total, 24, 'total is 24');

$t->reset;
is($t->total, 0, 'reset to zero');

done_testing;

Build Tally (development mode)

cd Tally
perl Makefile.PL     # finds ../Abacus/include automatically
make
make test

I hope you found this tutorial useful! If you have questions about XS, C header reuse, or building modular Perl/C libraries, please leave a message.