It was Christmas Eve, and Santa was facing a nightmare. The sleigh’s new GPS system, upgraded for the first time in centuries from following the Star of Bethlehem, was malfunctioning. As Santa checked the screen, the map was a chaotic mess — locations were wrong, some coordinates did not make sense, and the sleigh was heading straight into the mountains instead of over the City of New Orleans!

Santa's CURRENT position read-out indicated he was over the Appalachians at the following coordinates:

  Sleigh Latitude  Sleigh Longitude  Sleigh Altitude (ft)
       38.0000° N        -81.500° W              300

When he should be nearly exactly at,

  Sleigh Latitude  Sleigh Longitude  Sleigh Altitude (ft)
       29.9500° N        -90.070° W              300

"We can't afford this!" Santa shouted. "We’ve got billions of presents to deliver, and the system's down. And I want some gumbo!"

"Santa, we can fix it," Jingles, the lead elf, said with a frantic smile. "We just need more computing power. If we use OpenMP, we can parallelize the calculations and solve this quickly."

Santa stared. “What are you talking about, Jingles? I'm a giant elf, not a Scottish engineer on a television show for nerds!”

“We’ll solve the GPS problem by recalculating the distances to each delivery location. The sleigh’s GPS relies on triangulating data from multiple satellites. Right now, it's too slow to process the data for each location one by one. We can use OpenMP to divide the problem into smaller parts, calculate distances in parallel, and get this fixed fast!”

The data format of the positional satellites consisted of their positions in latitude, longitude, with an altitude; so the computations are necessarily in 3D space, and could contain any number of lines:

The distance formula was computationally expensive, involving square roots and trigonometry. But by using OpenMP, the task could be split into multiple OS threads, each calculating the distance to one satellite at the same time. The results would then be aggregated, and the sleigh's GPS would know exactly where to direct its heading.

The code would parallelize the work by dividing the list of satellites among a number of pthreads using OpenMP's #pragma omp for construct! OpenMP::Simple handles the required #include <omp.h> and makes it easy to query the environment for information, such as OMP_NUM_THREADS.

Jingles quickly typed up a solution in Perl, his favorite programming language. The solution was simple but critically offloaded the triangulation computations to Inline::C code containing OpenMP directives, using the OpenMP module on CPAN.

use strict;
use warnings;

use OpenMP;

use Inline (
    C    => 'DATA',
    with => qw/OpenMP::Simple/,
);

my $omp = OpenMP->new;

$omp->env->omp_num_threads($ENV{OMP_NUM_THREADS} // 8); # accept number of threads via commandline

# Sleigh's CURRENT position (latitude, longitude, altitude)
my $sleigh_lat = 38.0000;
my $sleigh_lon = -81.500;
my $sleigh_alt = 300.0; # in meters

# Satellite positions in tab-delimited format
my @satellites = ();
open my $FH, "<", "./satellite-data.txt" || die $!;

foreach my $line (<$FH>) {
  chomp $line;
  push @satellites, [split(/[\s\t]+/, $line)];
}

# Function to calculate distance from sleigh to each satellite using OpenMP::Simple,
# a subclass of Inline::C!
my $distances = calculate_distances($sleigh_lat, $sleigh_lon, $sleigh_alt, \@satellites);

# Print the calculated distances
foreach my $distance (@$distances) {
    print "Distance: $distance meters\n";
}

Santa watched as Jingles split up the computational load. Each satellite’s data was handled in parallel by different threads, and within seconds, the recalculations were done. The GPS latency issues were fixed. Jingles already had some thoughts of improvements he could make using OpenMP's ability to target GPUs directly, but was quite happy with the current resolution.

Just as the final calculations finished, a gentle voice spoke, “When Faith sanctifies morally neutral technology, Good things are possible at Internet scale.”

Santa turned to see the Holy Family - Baby Jesus in the arms of His Virgin Mother accompanied by His foster father, Joseph; by the sleigh, smiling serenely. “Thank you,” Santa whispered. “Merry Christmas.”

With the sleigh back on track, Santa soared into the night sky.

When things settled down, Santa was able to look at Jingles' full code, and it was something to behold!

use strict;
use warnings;

use OpenMP;

use Inline (
    C    => 'DATA',
    with => qw/OpenMP::Simple/,
);

my $omp = OpenMP->new;

$omp->env->omp_num_threads($ARGV[0] // 8); # accept number of threads via commandline

# Sleigh's CURRENT position (latitude, longitude, altitude)
my $sleigh_lat = 38.0000;
my $sleigh_lon = -81.500;
my $sleigh_alt = 300.0; # in meters

# Satellite positions in tab-delimited format
my @satellites = ();
open my $FH, "<", "./satellite-data.txt" || die $!;

foreach my $line (<$FH>) {
  chomp $line;
  push @satellites, [split(/[\s\t]+/, $line)];
}

# Function to calculate distance from sleigh to each satellite using Inline::C
my $distances = calculate_distances($sleigh_lat, $sleigh_lon, $sleigh_alt, \@satellites);

# Print the calculated distances
foreach my $distance (@$distances) {
    print "Distance: $distance meters\n";
}

__DATA__
__C__

#include <math.h>

// Function to convert geographic coordinates to Cartesian (x, y, z)
void geo_to_cartesian(double lat, double lon, double alt, double *x, double *y, double *z) {
  double R = 6371000;  // Earth's radius in meters
  double lat_rad = lat * M_PI / 180.0;
  double lon_rad = lon * M_PI / 180.0;

  *x = (R + alt) * cos(lat_rad) * cos(lon_rad);
  *y = (R + alt) * cos(lat_rad) * sin(lon_rad);
  *z = (R + alt) * sin(lat_rad);
}

// Function to calculate Euclidean distance between two points (x1, y1, z1) and (x2, y2, z2)
double calculate_distance(double x1, double y1, double z1, double x2, double y2, double z2) {
  return sqrt(pow(x2 - x1, 2) + pow(y2 - y1, 2) + pow(z2 - z1, 2));
}

/* C function parallelized with OpenMP */

// Main function to calculate the distances from the sleigh to each satellite
AV* calculate_distances(double sleigh_lat, double sleigh_lon, double sleigh_alt, AV *satellites) {
    int satellite_count = av_len(satellites) + 1;  // The number of satellites
    AV* distances = newAV();  // Create a new Perl array (AV) to hold the distances

    double sleigh_x, sleigh_y, sleigh_z;

    // Convert sleigh's geographic coordinates to Cartesian coordinates
    geo_to_cartesian(sleigh_lat, sleigh_lon, sleigh_alt, &sleigh_x, &sleigh_y, &sleigh_z);

    // Fetch satellite data into local arrays
    double *sat_latitudes = malloc(satellite_count * sizeof(double));
    double *sat_longitudes = malloc(satellite_count * sizeof(double));
    double *sat_altitudes = malloc(satellite_count * sizeof(double));

    // Populate satellite data into local arrays (from the Perl array)
    for (int i = 0; i < satellite_count; i++) {
      AV *satellite = (AV*) SvRV(*av_fetch(satellites, i, 0));  // Fetch the satellite at index i
      sat_latitudes[i] = ((double) SvNV(*av_fetch(satellite, 0, 0)));  // Latitude
      sat_longitudes[i] = ((double) SvNV(*av_fetch(satellite, 1, 0)));  // Longitude
      sat_altitudes[i] = ((double) SvNV(*av_fetch(satellite, 2, 0)));  // Altitude
    }

    // Declare a temporary array to hold distances for each thread
    double *_distances = malloc(satellite_count * sizeof(double));

    PerlOMP_GETENV_BASIC // read common environmental variables, provided by OpenMP::Simple

    // Start parallel region
    #pragma omp parallel shared(_distances)
    {
      // Parallel for loop to compute distances in parallel
      #pragma omp for
      for (int i = 0; i < satellite_count; i++) {
          _distances[i] = 0.0;  // Initialize

          double sat_x, sat_y, sat_z;

          // Convert satellite's geographic coordinates to Cartesian coordinates
          geo_to_cartesian(sat_latitudes[i], sat_longitudes[i], sat_altitudes[i], &sat_x, &sat_y, &sat_z);

          // Calculate the distance from the sleigh to this satellite
          _distances[i] = calculate_distance(sleigh_x, sleigh_y, sleigh_z, sat_x, sat_y, sat_z);
      }
    }

    // Combine the results from all threads into the main distances array inside the parallel region
    for (int i = 0; i < satellite_count; i++) {
      av_push(distances, newSVnv(_distances[i]));
    }

    // Free the private distance arrays and satellite data arrays
    free(_distances);
    free(sat_latitudes);
    free(sat_longitudes);
    free(sat_altitudes);

    // Return the AV containing the distances to Perl
    return distances;
}

__END__

See More:

https://metacpan.org/pod/OpenMP

https://metacpan.org/pod/OpenMP::Simple

https://metacpan.org/pod/OpenMP::Environment

Everyone is welcome to join the Perl+OpenMP Project at https://github.com/Perl-OpenMP/!

Christmas eve was quickly approaching, and Santa was trying to decide whether to purchase masks for all of the reindeer again this year (he was aware that there were documented cases of SARS-CoV-2 infecting White-Tail Deer (O.virginianaus), and he knew that Reindeer (R.tarandus) were closely related), when the Chief Elf interrupted his train of thought to inform him that two of the elves in workshop were in the infirmary with respiratory distress. Testing had revealed that they were both infected with Respiratory Syncitial Virus (RSV), specifically with the RSVA subtype.

Santa asked "Do we know where they were exposed, or if one of them transmitted it to the other? We may need to require everyone at the North Pole to mask again until after Christmas eve . . . "

"The diagnostic lab says the two viruses are similar but not identical. Also they are not confident of the lineage calls and the clade assignments because we are using a new set of PCR primers designed in Edinburgh, which are based on a newer RSVA reference sequence (RefSeq). The problem is that we are using the RSVA phylogenetic tree available at usher.bio and that tree was built using the NCBI RefSeq as the root. You can see where the problem arises . . . "

Santa thought for a moment and said "Oh, blast! We aligned our fastq reads to the other RefSeq, and at the primer trimming step we are using the genomic coordinates from that different virus strain. So if we first align against the NCBI RefSeq genome (to get more accurate lineage calls and clade assignments) then the primer trimming step will not work as intended. Huh! Hmmm, do you think we could align those two different RSVA RefSeqs to each other and then modify the coordinates in the primer bed file so it uses the NCBI RefSeq's genomic coordinates instead?!?"

The Chief Elf sketched this out on the iceboard and said, "I think this could work. I will ask our bioinformatic software developers to tackle the problem."

And so gentle reader that is how we arrived at a simple, but tedious task of converting the genomic coordinates in this bed file:

  RS20000581      44      66      RSVA_1_LEFT     1       +
  RS20000581      434     464     RSVA_1_RIGHT    1       -
  RS20000581      359     385     RSVA_2_LEFT     2       +
  RS20000581      749     773     RSVA_2_RIGHT    2       -
  RS20000581      669     699     RSVA_3_LEFT     1       +
  RS20000581      1057    1083    RSVA_3_RIGHT    1       -
  RS20000581      990     1016    RSVA_4_LEFT     2       +
  RS20000581      1366    1389    RSVA_4_RIGHT    2       -

So that it would contain the correct coordinates from a different RSVA subtype RefSeq. It quickly became apparent that this problem could be solved much more quickly using the Awesome Power of Perl, unfortunately there was no out-of-the-box solution available in the existing base of Perl code. But how to proceed?

Using the Awesome Power of Perl in Bioinformatics

Each of the two genome files are simple text files consisting of an alphabet of five letters (A, C, G, and T (occasionally an N)) (technically RSV, similar to SARS-CoV-2 is an RNA virus, but life is simpler if we use 'T' instead of 'U'), all we need is a so-called "Global Alignment" of the two full-length sequences (they are each just over 15,000 bases in length). What we really want to know is where all of the longest stretches of identical sequences are located in the viral chromosomes. Fortunately an algorithm to calculate this was published by Needleman and Wunsch back in 1970; it is considered a classic example of dynamic programming (which builds up the answer without consuming vast amounts of memory). Furthermore, the National Center of Biotechnology Information (NCBI) maintains a server we can use to upload any two closely related nucleotide sequences.

The output from the N-W Global Alignment looks like this:

  Query  2177   GTGTGATTAACTACAGTGTATTAGATTTGACAGCAGAAGAACTAGAGGCTATCAAACATC  2236
                |||||||||||||||||||| |||| ||||||||||||||||||||||||||||||||||
  Sbjct  2221   GTGTGATTAACTACAGTGTACTAGACTTGACAGCAGAAGAACTAGAGGCTATCAAACATC  2280

  Query  2237   AGCTTAATCCAAAAGATAATGATGTAGAGCTTTGAGTTAATAAAAAGGTGGGGCAAATAA  2296
                ||||||||||||||||||||||||||||||||||||||||||||||  ||||||||||||
  Sbjct  2281   AGCTTAATCCAAAAGATAATGATGTAGAGCTTTGAGTTAATAAAAAA-TGGGGCAAATAA  2339

Where the the top sequence in each pair is a stretch of 60 bases from the Edinburgh RefSeq (Labeled 'Query') and the bottom sequence in each pair is the matching stretch from the NCBI RefSeq (Labeled 'Sbjct'). The unix pipe symbols denote a perfect match between the sequences at that base.

Notice that the numbering systems are slightly offset, AND that there is a gap introduced with a hyphen in-between the A at 2327, and the T at 2328 in the Sbjct, directly across from the G at position 2384 in the Query. That can be interpreted as a single nucleotide insertion in the Query genome. So creating the mapping table between the two coordinate systems is not trivial. The sequences only share 94 percent identity over about 15,200 nucleotides, which means they diverge 6 percent of the time.

To simplify the construction of our lookup table we can use grep to filter the starting alignment into two different subsets,

grep ^Query nw_alignment.txt > just_rows_from_edinburgh_refseq.txt
grep ^Sbjct nw_alignment.txt > just_rows_from_ncbi_refseq.txt

N.B. you will have to go in and manually remove the first extraneous Query line from the first file. These become the first two input files to our script, the third file is the primer bed file we are starting with.

Our Script that Creates a Lookup Table, and then Swaps Coordinates in a bed File

#!/usr/bin/env perl
use strict;
use warnings;

my $query_input = $ARGV[0];
my $subject_input = $ARGV[1];
my $input_bed = $ARGV[2];

open my ($FH1), '<', $query_input or die "Could not open $query_input for reading\n";
open my ($FH2), '<', $subject_input or die "Could not open $subject_input for reading\n";

my @global_array = ();
$global_array[0] = { query => undef,
       sbjct => undef,
     };

# This global counter serves as the index for the first data structure
# that captures the information at each place in the aligned sequences
my $counter = 0;

# Parse the rows of the first file
while ( <$FH1> ) {
    chomp;
    # Each of these filtered rows has the same structure, in addition to
    # A, C, G, and T, the N-W output can also contain hyphens (dashes), so
    # our regex has to include that in the string we are capturing
    my ($seq) = $_ =~ m/Query\s+\d+\s+([\w\-]+)\s+\d+/;
    # Parse the 60 nucleotide string into individual characters
    my @qchars = split(//, $seq);
    foreach my $q ( @qchars ) {
        $counter++;
        # Load up the array of hashrefs with the symbols
	$global_array[$counter]{query} = $q;
    }
}
close $FH1;


$counter = 0;
while ( <$FH2> ) {
    chomp;
    my ($seq) = $_ =~ m/Sbjct\s+\d+\s+([\w\-]+)\s+\d+/;
    my @schars = split(//, $seq);
    foreach my $s ( @schars ) {
        $counter++;
 $global_array[$counter]{sbjct} = $s;
    }
}
close $FH2;

# Now we process the information in each element of the
# @global_array, and store that in this global hash
# the keys of this hash are genomic coordinates for the
# Edinburgh RSV-A RefSeq and the values are the genomic
# coordinates of the NCBI RSV-A RefSeq
my %ncbi_coordinates_of = ();

my $qcounter = 0;
my $scounter = 0;

# There is nothing useful in the [0] element
foreach my $i ( 1..$#global_array ) {
    my $key = undef;
    my $value = undef;
    # If the string from the N-W output contains a '-' then we
    # need to handle those differently.  We only increment the
    # $qcounter when the value in 'query' matches a letter
    if ( $global_array[$i]{query} =~ m/\w/ ) {
        $qcounter++;
        $key = $qcounter;
        if ( $global_array[$i]{sbjct} =~ m/\w/ ) {
            $scounter++;
     $value = $scounter;
 }
        $ncbi_coordinates_of{$key} = $value;
    }
    else {
 $scounter++;
 next;
    }
}

# After all of that manipulation and data wrangling, the problem now becomes
# very simple.  We read in the starting primer bed file and iterate over each
# TSV record.  We use the information in columns 2 and 3 to query the
# %ncbi_coordinates_of hash, and insert the coordinates we want from the hash
# values.  The modified record is immediately printed to STDOUT

open my ($FH3), '<', $input_bed or die "Could not open $input_bed for reading\n";

while ( <$FH3> ) {
    my @fields = split(/\t/, $_);
    # Change the sequence name in column 1
    $fields[0] = 'NC_038235.1';
    # It is formally possible that either the 'start' or the 'end' of the
    # Query sequence is NOT in the %ncbi_coordinates_of hash, so check for this
    unless ( exists $ncbi_coordinates_of{$fields[1]} and exists $ncbi_coordinates_of{$fields[2]} ) {
        $fields[1] = $fields[2] = 'undef';
        print join("\t", @fields);
        next;
    }
    # Sometimes the 'start' and 'end' coordinates from the Query sequence ARE keys in the
    # %ncbi_coordinates_of hash, but the corresponding values from the Sbjct sequence are undef
    # so continue processing gracefully when this occurs
    $fields[1] = $ncbi_coordinates_of{$fields[1]} //= 'undef';
    $fields[2] = $ncbi_coordinates_of{$fields[2]} //= 'undef';
    print join("\t", @fields);
}
close $FH3;

exit;

Conclusion

When you run the script, save the output to file, and then use grep undef to see if there were any coordinates which could NOT be successfully mapped. If there are any then you can use grep -v undef to save a copy without those lines.

What did we learn? After running this script to create the new primer bed file for the NCBI RSV-A RefSeq, and then rerunning their viral sequencing pipeline to align the amplified reads to the NCBI RefSeq, the bioinformaticians working with the infirmary could see that the two isolates from the workshop elves were more closely related than they had previously thought, but since the viruses still had many mutations separating them, it seemed like these were two separate exposures (or introductions) of RSV-A to the North Pole compound.

You can see that for yourself in this image:

N.B.: In accordance with the laws of Canada, Denmark, and Russia, the clinical metadata and any other personal health information (PHI) for these samples has been de-identified.

These are two versions of the RSV-A Global phylogenentic tree, showing a subtree of the samples that the most highly related to the two sequences from Santa's workshop. The trees are shown rotated 90 degrees, so the "root" of the tree (which is not shown), is on the LEFT. You can see the "branches" as horizontal lines; and at the tips of each branch is a "leaf" (or a node). The leaves represent the sequences from different individual viruses in the database. Each tip is labeled with the name of the sample, and the two samples from the North Pole are colored black. The Y-axis has no units, and is arbitrary. The data depicting similarity is contained in the pattern of the branches (which are technically internal nodes), and in the LENGTH of the branches. The X-axis conveys information about the number of mutations in one sample, which is the number of nucleotide sequence changes that it takes to "get back" to the root of the tree.

Panel A. shows where the UShER program placed the two samples from the North Pole, using the Edinburgh RefSeq genome, whereas Panel B. shows where they were placed after the same samples were aligned to the NCBI RefSeq genome, and the new Primer bed file created with our script was used to remove (or trim) the primers from the end of the reads. You can see that the placement of the Elf_0001 sample is almost the same in the two trees, whereas the placement of the Elf_0002 sample has moved. It is now forking off at a different branch of the tree and is closer (and therefore more similar, or more related) to Elf_0001.

In principal, the flow and the steps used here could work with any paired RefSeqs for a pathogenic virus where you have created DNA sequencing amplicons with PCR primers from one coordinate system, but you want to be able to use your regular pipelines to align the fastq reads to a different RefSeq.

If you have some paired-end fastq files from RSV (or another RNA virus) you can get an idea of the processing steps to use to generate consensus.fasta files here.

During the COVID19 Global Pandemic these consensus.fasta files were essential for constructing the phylogenetic trees used by Epidemiologists to identify outbreaks of new strains.

THM: Best to wear masks at the North Pole until after DEC-24.

I had never met Randal Schwartz before, but when I reached out to him about contributing to year 25 of the Perl Advent Calendar, he immediately agreed. This year I wanted to try some new things and Randal suggested giving a Perl-specific talk which he had given once before, but which was lacking in recording quality. This gave Randal a chance to re-record his talk and it gave the Perl communities a chance to watch in real time. We've never before had a video recording as a Perl Advent article, but I believe quite strongly that this is in the spirit of the Advent Calendar project, which is about giving something back. Also, when I mentioned it to Mark Fowler back in October, he didn't object, so I'll take that as a seal of approval.

-- Olaf Alders

Randal's advance summary of this talk:

    Having been there, at the beginning with Perl, I will recount the early
    days through the modern era (or as much as I can cover in the time
    provided). I’ll deliver first-hand experience of the creation of the Camel
    Book, the Llama book, and the way I invaded comp.unix.questions with Perl 2
    answers so often that people would post “no Perl please”. Oh, and my
    version of the story of the Schwartzian Transform.

If you enjoyed the video below, you may also enjoy the next in the series, which will be presented by Dave Cross: Still Munging Data with Perl. If you are at all interested, please register now, even if you cannot attend, as that will allow us to share the recording URL with you after the fact.

Now, please enjoy Randal Schwartz's "Half My Life with Perl":

Another busy Christmas period loomed, and the North Pole was tirelessly working to get the final preparations ready. Inside a brightly lit meeting room Santa cleared his throat.

"This is all very interesting, very... *yawn* ...stimulating", he said, picking at some cookie crumbs that fell from his beard to the table. "But how is that going to help us? How does ... absorbability help us deliver gifts?"

"Observability", corrected Ada Slashdóttir, the team's senior elf.

"Yeah, that's what I said", said Santa. "What does it do for us?"

As their platform had started to grow, the elves had struggled to keep track on where the bottlenecks were. And now that they had migrated parts of it to microservices and their platform was becoming more distributed, even the tools they were familiar were starting to become unwieldy.

"We need to have a way to see from the outside how our system behaves internally, Santa", said Ada. "That's observability. So it doesn't directly help us send gifts, but it helps us keep things running smoothly, and that helps us send gifts".

"Isn't that what logs do?", asked Duende Juniorsson, the team's junior elf, who was trying to catch up. "We already have logs in our services. Can't we use those?"

Santa liked logs. Yule logs in particular.

"We can, for sure", said Gnomo Knullpointer, who had recently joined Santa's workshop as a coding elf. "Logs are useful because they have been around forever, so they are well supported. But they have their limitations, specially now that a single request can touch several separate microservices before we are done with it. We'd have to correlate logs across services, which is not easy."

"Correlating logs is difficult, but there are other kinds of telemetry we can use", said Ada. "In particular, we can uses traces to track a request across services. This is called 'distributed tracing'. Let me show you how it works."

Integrating with Mojolicious

Ada then opened the code for the Naught-or-Not service, a Mojolicious application that checked whether a specific child has been naughty or nice:

# Naught-or-Not: the 'naughty or nice' service
use Mojolicious::Lite -signatures;
use Mojo::SQLite;

helper sql => sub { state $sql = Mojo::SQLite->new };

app->sql->migrations->name('santa')->from_string(<<'EOF')->migrate;
-- 1 up
create table naughty (id integer unique, naughty bool);
-- 1 down
drop table naughty;
EOF

get '/is-naughty/:id' => sub ($c) {
    my $id = $c->param('id');
    my $db = app->sql->db;

    # If we have a value for this ID, return it

    my $row = $db
      ->select( naughty => ['naughty'] => { id => $id } )
      ->hashes
      ->first;

    return $c->render( json => { naughty => $row->{naughty} } ) if $row;

    # Otherwise, generate one and store it before returning

    my $naughty = !!( int rand 2 ); # Randomised naughtiness!

    $db->insert( naughty => { id => $id, naughty => $naughty } );

    $c->render( json => { naughty => $naughty } );
};

"The first thing we'll need to do", said Ada, "is get this code to start generating telemetry data. Traces in particular."

"Wait a second", said Gnomo. "If we start touching the code to generate this telemetry, we run the risk of introducing bugs. Not to mention that we have a lot of code, we'll never be able to do it all by hand..."

Santa was starting to sweat.

"We won't have to. We can use OpenTelemetry to do what is called 'zero code instrumentation', which will get us most of the way there. The first thing we'll need to do is load up the OpenTelemetry::SDK in our application. This will read the configuration from the environment and set things up internally so any telemetry we generate gets exported correctly. And since we are using Mojolicious we can use Mojolicious::Plugin::OpenTelemetry to actually generate telemetry data from our routes, all without actually touching any of the controller code."

use Mojolicious::Lite -signatures;
use OpenTelemetry::SDK;
use Mojo::SQLite;

plugin 'OpenTelemetry';

...

"And that's it?", asked Duende a little disappointed. He was already looking forward to touching a lot of code.

"Pretty much. The rest is just configuration. You can already see this in motion by running the code and telling it to export the traces to the console", said Ada, feeling pedagogic.

$ OTEL_TRACES_EXPORTER=console ./server get '/is-naughty/123'
[2024-12-16 23:24:29.37268] [821150] [trace] [nLc70JabTxup] GET "/is-naughty/123"
[2024-12-16 23:24:29.37288] [821150] [trace] [nLc70JabTxup] Routing to a callback
{
    'attributes' => {
        'client.address' => '127.0.0.1',
        'client.port' => '47534',
        'http.request.method' => 'GET',
        'http.response.status_code' => 200,
        'http.route' => '/is-naughty/:id',
        'network.protocol.version' => '1.1',
        'server.address' => '127.0.0.1',
        'server.port' => '36703',
        'url.path' => '/is-naughty/123',
        'user_agent.original' => 'Mojolicious (Perl)'
    },
    'dropped_attributes' => 0,
    'dropped_events' => 0,
    'dropped_links' => 0,
    'end_timestamp' => '1734391469.37503',
    'events' => [],
    'instrumentation_scope' => {
        'name' => 'server',
        'version' => ''
    },
    'kind' => 2,
    'links' => [],
    'name' => 'GET /is-naughty/:id',
    'parent_span_id' => '0000000000000000',
    'resource' => {
        'process.command' => './server',
        'process.command_args' => [
            'get',
            '/is-naughty/123'
        ],
        'process.executable.name' => 'perl',
        'process.executable.path' => '/home/user/.perl/perls/perl-5.40.0/bin/perl',
        'process.pid' => 821150,
        'process.runtime.name' => 'perl',
        'process.runtime.version' => 'v5.40.0',
        'telemetry.sdk.language' => 'perl',
        'telemetry.sdk.name' => 'opentelemetry',
        'telemetry.sdk.version' => '0.024'
    },
    'span_id' => '3823100b9ca26a91',
    'start_timestamp' => '1734391469.3732',
    'status' => {
        'code' => 0,
        'description' => ''
    },
    'trace_flags' => 1,
    'trace_id' => '795c45630d3615ba1ebf524661bf01ac',
    'trace_state' => ''
}

A wall of text (pretty printed here for clarity) appeared in front of the elves. Santa seized the opportunity to look like he knew what was going on by looking at the text in silence while thinking about what to have for lunch.

After a couple of seconds Gnomo and Duende had pieced it together. It was all there! Data about the request itself, how long it took, where it came from, and even details about what code had generated it... and all of that without having to touch any of the core logic!

"This is very cool", said Gnomo, who had been reading through OpenTelemetry::Guides::Quickstart and was already making sense of things. "And it seems we can get even more data if we load some of the instrumentation libraries that are already out there. This route uses a database, so we could use the OpenTelemetry::Instrumentation::DBI to get DBI to also generate telemetry".

use Mojolicious::Lite -signatures;
use OpenTelemetry::SDK;
use OpenTelemetry::Instrumentation 'DBI';
use Mojo::SQLite;

plugin 'OpenTelemetry';

...

Now running the test command printed even more output, this time including traces for all the DB operations, including the ones that executed when the service initially came up and ran the database migrations, etc.

"What about our other services? Not all of them are using Mojolicious", said Duende, thinking that maybe then he'd have the chance to touch some code.

Integrating with Dancer2

Ada loaded up the code for the Gift Allocation service, which was implemented in Dancer2. This service made a call to Naught-or-Not to check whether this child had been naughty, and allocated either a lump of coal or one of the gifts from the child's wishlist.

package Gift::Assignment;

use Dancer2;
use HTTP::Tiny;

set serializer => 'JSON';

my $ua = HTTP::Tiny->new;

post '/gift/assign' => sub {
    my $id = body_parameters->get('id');
    my $res = $ua->get("$ENV{NAUGHT_OR_NOT_HOST}/is-naughty/$id");

    unless ( $res->{success} ) {
        status $res->{status};
        return { error => $res->{reason} };
    }

    my $body = decode_json $res->{content};
    return { gift => 'coal' } if $body->{naughty};

    my @wants = body_parameters->get_all('wants');
    { gift => $wants[ int rand @wants ] };
};

"Like with the Mojolicious service, integrating this with OpenTelemetry requires no code changes", said Ada. Duende's heart sank. Was he going to get to touch any code? "All we need to do is load the Dancer2::Plugin::OpenTelemetry plugin".

"And like with the DBI instrumentation, since this route is using an HTTP client, we can load OpenTelemetry::Instrumentation::HTTP::Tiny to get telemetry for the requests it makes!", chimed in Gnomo, who was neck-deep in the documentation for instrumentation libraries and getting very excited.

The import list now looked like this:

package Gift::Assignment;

use Dancer2;
use Dancer2::Plugin::OpenTelemetry;
use OpenTelemetry::Instrumentation 'HTTP::Tiny';

set serializer => 'JSON';

...

"Hold on a second... what happened to the HTTP::Tiny import?", asked Duende, who was trying to follow along to find ways to contribute.

"Oh, well that's done automatically when we load the instrumentation library, so we don't need to write that twice", said Ada".

"And what about the OpenTelemetry::SDK import? Don't we need it here as well?", continued Duende.

"Ah, well spotted, Duende!", replied Ada. "We do need it, but we only need to load it once. Since this service will have more controllers once it's finished, loading it in each one would be a hassle, so it's better to load it once at the entry point. In this case, since we are using Plack to mount it, we can load that in the top-level PSGI file".

#!/usr/bin/env perl

use strict;
use warnings;
use lib 'lib';

use Gift::Assignment;
use OpenTelemetry::SDK;
use Plack::Builder;

builder {
    Gift::Assignment->psgi_app;
}

Once again they tested it. They brought up the server configured to send traces to the terminal, and made a request:

$ curl -X POST \
    -H 'Content-Type: application/json' \
    -d '{"id":123, "wants":["teddy_bear", "crochet"]}' \
    http://localhost:5000/gift/assign

As the traces appeared in the logs now for both services, Gnomo spotted something. "Hey look! Ignore everything in the traces except for those span and trace IDs". Pretty printed for your convenience, this is what he was looking at:

    naught-or-not  | {
      'name' => 'SELECT "naughty" FROM "naughty" WHERE "id" = ?',
      'parent_span_id' => '88b066b24f372f39',
      'span_id' => '932cf74c171b705e',
      'trace_id' => '16073bf43d281a0693ec7d5a9f4860a4',
    }
    naught-or-not  | {
      'name' => 'GET /is-naughty/:id',
      'parent_span_id' => '7bd198e5a6ddd5fd',
      'span_id' => '88b066b24f372f39',
      'trace_id' => '16073bf43d281a0693ec7d5a9f4860a4',
    }
    gifts          | {
      'name' => 'GET',
      'parent_span_id' => 'd9718758cefd39d4',
      'span_id' => '7bd198e5a6ddd5fd',
      'trace_id' => '16073bf43d281a0693ec7d5a9f4860a4',
    }
    gifts          | {
      'name' => 'POST /gift/assign',
      'parent_span_id' => '0000000000000000',
      'span_id' => 'd9718758cefd39d4',
      'trace_id' => '16073bf43d281a0693ec7d5a9f4860a4',
    }

Santa wondered if this is what it felt like to see the woman in the red dress.

"What are we looking at?", said Duende.

"They are all linked!", replied Gnomo excitedly. "They have the same trace_id, and the parent_span_id in each one is the same as the span_id of the one that came before it, which appears lower in the logs".

"Well, except for that one at the bottom. That one just has a bunch of zeroes as the parent_span_id", commented Duende.

"Yes, that is called the root span", said Ada. "The fact that it has a null parent_span_id tells us that this is the span that initiated the trace".

"We could write something that would aggregate these data and present them in a neater way", said Gnomo. "Maybe something using a kind of flamegraph like the ones from Devel::NYTProf, which are great".

"We don't have to!", said Ada. "The OpenTelemetry project has already done that for us. We can use the OpenTelemetry Collector to aggregate all sorts of telemetry data generated via OpenTelemetry and then configure it to send that data to whatever external service or platform we want".

"Oh, so it's vendor agnostic too?", said Gnomo. "That's very good".

Santa's eye twitched at the mention of "agnostic". This was a touchy subject in the North Pole. He looked at Helga Hakrniüs, the Public Relations elf in the corner of the room who quietly shook their head. Santa let his breath out and wiped his brow.

"Yes, that's one of OpenTelemetry's key selling points", said Ada. "Vendor agnostic on the consumer side, and platform agnostic on the producer side, so you can use it in services written not only in all sorts of frameworks like we just saw, but also in all sorts of languages. They even have a demo application where you can see this in motion".

"Is this true for Perl as well?", said Gnomo. "We can obviously use it in Mojolicious and Dancer2, but what about other frameworks, like Catalyst? I heard that's what the Milk-and-Cookies team was using for their service. Or plain CGI? We still have a bunch of those in the older parts of the code".

"Yes!", said Ada. "The Perl implementation is relatively new, so there's still work to do, but we do have Plack::Middleware::OpenTelemetry that we can use for any server that uses Plack. The integration might not be as close, but it does work. And it's only a matter of time before more instrumentation libraries get released".

"So how do we use this Collector?", asked Duende.

"I'm glad you asked", replied Ada.

From the terminal to the browser

"The OpenTelemetry distribution on CPAN has an example collector stack that we can use to see how things go together", continued Ada. "There is also a similar example managed by the developers of the collector itself, and a more realistic example in the OpenTelemetry demo I mentioned before. We can use the one from CPAN to illustrate things for now".

git clone https://github.com/jjatria/perl-opentelemetry
cd perl-opentelemetry/examples/collector
docker compose up --build

"And then we tell the services to talk to it?", asked Gnomo.

"Yes", said Ada. "We need to set a couple of environment variables for each deployment so that they can talk to the collector. It's probably easier to show this with a docker compose file".

services:
  naught-or-not:
    build:
      context: mojo
    container_name: naught-or-not
    volumes:
    - ./mojo:/app
    command:
    - hypnotoad
    - ./server
    - --foreground
    environment:
      IO_ASYNC_LOOP: Mojo
      OTEL_BSP_MAX_EXPORT_BATCH_SIZE: 1
      OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost
      OTEL_SERVICE_NAME: naught-or-not
      OTEL_TRACES_EXPORTER: otlp
    network_mode: host

  gifts:
    build:
      context: dancer
    container_name: gifts
    volumes:
    - ./dancer:/app
    command:
    - plackup
    - --server
    - Net::Async::HTTP::Server
    - ./server.psgi
    environment:
      NAUGHT_OR_NOT_HOST: http://localhost:8080
      OTEL_BSP_MAX_EXPORT_BATCH_SIZE: 1
      OTEL_EXPORTER_OTLP_ENDPOINT: http://localhost
      OTEL_SERVICE_NAME: gifts
      OTEL_TRACES_EXPORTER: otlp
    network_mode: host

"In both cases we need to set OTEL_SERVICE_NAME to something meaningful so we can identify where the traces come from. We also need to set OTEL_TRACES_EXPORTER to otlp, which is the protocol used by the collector, and OTEL_EXPORTER_OTLP_ENDPOINT to the host that we are sending the telemetry to. In this case, both of these are set to their default values, but it's good for now to be explicit".

"What about that OTEL_BSP_MAX_EXPORT_BATCH_SIZE?", asked Duende. He had already learned the hard way that a batch of one was a little pointless.

"Oh, that's only for testing. In a realistic scenario we can increase that to limit the number of requests that we make to the collector", said Ada.

"So the communication with the collector happens off-band?", asked Gnomo.

"Yes. When exporting to the console we got each span as it was produced, but in a production environment you'd expect many times more spans to be produced, and it would be very expensive to export each one as it came. It's more efficient to batch them and send them all together when enough of them are ready. The Perl OpenTelemetry implementation uses a batch span processor written on top of IO::Async to do this".

Santa wondered if his elves lived in the same planet as he did. He was glad there were people in the room who knew about this so he didn't have to.

"Is that why we need the IO_ASYNC_LOOP variable in the Mojolicious environment?", asked Gnomo.

"Yes", said Ada. "We need that because Mojolicious has its own event loop, so we need to tell IO::Async to use it".

"I see. And that's why in the Dancer2 environment we use Net::Async::HTTP::Server, because Dancer2 doesn't have an event loop, and this gives it one", said Gnomo.

"Exactly", said Ada. "And now, if we run this so it talks to the collector stack, and make some queries, we can see the traces in our browser. That stack connects the collector to a Jaeger instance, which we should be able to see at http://localhost:16686".

"Incidentally", said Ada, "I've put all the files for what we've been talking about in a repository you can check out if you want to run things yourself".

Santa was happy to finally be able to see some pretty pictures. And as he looked at his elves, all excitedly looking around and exploring the data, he felt a warm sense of pride and joy to see them learning and growing and getting things done.

With this out of the way, he could finally get back to what mattered most: having those milk and cookies.

Lesser-known businesses

Everybody knows that the North Pole™ produces and delivers Christmas gifts, but it’s not the sole business they’re into. In fact, they’ve recently engaged in production of Christmas ornaments and Christmas cards, too.

As is common, starting a new business brings new problems. The elves needed a way to design decorated Christmas trees to evaluate their ornament proposals and their combinations. They started by planting small trees and decorating them by hand, but quickly found out this approach didn’t scale as more and more (and bigger and bigger) trees were needed.

At a C-level meeting, Santa listened to laments of the managing elf responsible for the ornaments and narrowed his eyes at another young elf at the other corner of the table.

“You had something for arranging trees, right?” asked Santa.

“Yes, but…” spluttered the elf.

“Let’s meet after lunch and see how we can share the knowledge,” commanded Santa.

There’s trees and there’s trees

In the afternoon (by the way, it was already dark, since it was the North Pole and summer was over) Santa met with the COO and CCO (where the second letter stands for Ornaments or Cards, respectively).The CBO (Chief Baubles Officer) was missing as his department was merged with the Ornaments in the last workforce shaping to trim the fat.

“Can you show us what your department uses to visualise trees?” asked Santa, turning to the CCO.

“Our developers found this open source tool called TrEd, which stands for ‘Tree Editor’,” replied the elf. “And they’re still discovering new features it has. You can do much more than view the trees: you can easily change their structure, add attributes to nodes and edges, add secondary relations that turn the trees into full graphs, and there are also tools for searching large treebanks.”

“I hate the jargon,” muttered Santa and turned to the COO, “but I guess you’re following.”

“Actually, not really,” replied the COO, “we need to arrange the ornaments, but we don’t want to change the structure of the trees. How is such a thing needed to produce a Christmas card, anyway?”

“We hear similar questions often,” said the CCO keeping a stiff upper lip. “At the beginning, we only produced English Christmas cards, so we didn’t need anything like that. But several years ago we started printing the cards in other languages, too, and we needed a way to translate all the greetings and wishes. We started with elvish translators, but we found ourselves in your boots, so to say: the approach didn’t scale.

“We needed an automated process. We reached for statistical machine translation, but for that, we needed large aligned corpora in both the source and target languages.”

“Corpora?” asked Santa raising an eyebrow.

“Large collections of texts. And we quickly found out aligning the individual words wasn’t enough, as the grammar in various languages can change the words in different roles. The sentence structure stays usually much more similar across languages than individual words and their order. That’s why we started annotating the trees.”

“Decorating,” said Santa and nodded to the COO hopefully.

“No, annotating,” explained the CCO. “I’m talking about trees in the graph-theory sense. We arrange the words in a sentence to a tree and annotate the relations between them with their syntactic roles: this word is a subject of this verb, that word is an adverbial of that word,” and he started to gesticulate wildly.

“Wait, wait,” the COO interrupted him, “can you show us what you’re talking about? I still have no idea.”

“Ho ho ho,” said Santa, “a picture is worth a thousand words!”

Diving deeper

The CCO opened his ChristmasPad and typed something into a terminal. “See? This is Ukrainian, by the way.”

“That’s impressive,” admitted Santa, “but I fear there’s some kind of confusion.”

“Let’s have a look at a simpler example in English,” replied the CCO and quickly typed on the keyboard. “The annotated sentence is Is that Microwave that you gave Dan really expensive?”

“You can see the pronoun that references the word microwave, and the word microwave is an object of the verb give in a semantic sense which we can also capture.”

“No, no,” tried Santa to stop him, “linguistics is not our concern.”

“That”s great!” rejoiced the CCO. “I’ve always wondered whether TrEd can be used outside of linguistics. There already is one such use: The tree editor serves as a client to a search engine. You assemble a tree and the engine searches your tree data to find where the tree would fit. The trick is you can specify different relations than the ordinary parent–child one.”

“Ho ho ho,” nodded Santa, “Christmas is a family time!”.

“I mean this,” explained the CCO and again showed them his screen.

“Normally, the parent would be at the top, but here, we’re using the reversed relation, so the query will search for all nominal subjects whose parent is not a verb.”

“How can something that’s not a verb have a subject?” wondered Santa.

“Let me show you the English example with the microwave again. The Universal Dependencies style uses adjectives in copula constructions as parents of the subject and the auxiliary verb. The word expensive is not a verb, but the microwave is its nominal subject.”

“I fear this whole thing is of no use for us,” sighed the COO. “What programming language is the tool written in?”

“Perl,” replied the CCO. “It uses Tk::Canvas to edit the trees, which makes it rather easy to extend if you need more features.”

“At least something our team would understand. And the search engine is also written in Perl?” asked the COO again.

“There are in fact two implementations,” replied the CCO. “One uses SQL on Postgres to store and query the data, but it’s only suitable for data that don’t change, as updating the database is quite slow. The second implementation uses Perl and is great for querying frequently changing data. If the data are large, you need some kind of parallelism to compensate its less favourable speed, we run it over slurm. But you can also write your queries directly in Perl. This will show you exactly the same trees as the query I showed you before.” And he again used the terminal.

btred -N -T -e '
    FPosition()
        if $this->{deprel} eq "nsubj"
        && $this->parent->{upostag} ne "VERB"
    ' data/*.conllu | tred -l-

“Also, if you need to process the data without all the power TrEd offers, you can just use Treex::PML, the library that TrEd is based on. It implements the Prague Markup Language used as TrEd’s native data format. The previous five-liner turns almost into a screenful,” and he opened Elven Mate at Creating Scripts (EMaCS) and started to type, interrupted several times by squinting into the documentation.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

my $ud_path;
BEGIN { $ud_path = $ENV{UD_DIR} }

use lib "$ud_path/libs";

use Treex::PML qw{ ImportBackends AddResourcePath };

my @backends = ImportBackends('UD');
AddResourcePath("$ud_path/resources");

my $schema = 'Treex::PML::Factory'->createPMLSchema({
    use_resources => 1,
    filename      => "ud_schema.xml"});

for my $file (@ARGV) {
    my $doc = 'Treex::PML::Factory'->createDocumentFromFile(
        $file, {backends => \@backends});

    my $tree_no = 1;
    for my $tree ($doc->trees) {
        my $node_no = 1;
        for my $node ($tree->descendants) {
            say "$file##$tree_no.$node_no"
                if $node->{deprel} eq "nsubj"
                && $node->parent->{upostag} ne "VERB";
            ++$node_no;
        }
        ++$tree_no;
    }
}

No happy ending?

“It’s interesting, but I don’t see how our department could benefit from it,” shrugged the COO.

Santa seemed lost in thought. “Maybe your department can’t,” murmured he, “but we have many other departments that need solutions…”

He dismissed the meeting by pointing at the door and strode towards his office. Can you think of a way how you could benefit from the tool?