Michael Limberger

Need me? Email mike@limberger.ca

Patterns that have been in a fight

These are battle-tested one-liners for common tasks.

Logs

Show me.

perl -ne 'print "$1\n" if /(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/' app.log
perl -ne '$c++ if /error/i; END { print "$c\n" }' app.log
perl -ne 'print "$1\n" if /(\d+\.\d+\.\d+\.\d+)/' access.log | sort -u

Timestamps. Count lines containing "error". Unique IP addresses.

CSV

perl -F',' -ane 'print "$F[1]\n"' data.csv
perl -F',' -ane 'print "$F[1]\n" if $. > 1' data.csv
perl -F',' -ane 'print "$F[2],$F[0],$F[1]\n"' data.csv

Second column. Skip the header ($. is the line number). Reorder columns 3, 1, 2.

JSON, crudely

perl -ne 'print "$1\n" if /"name":\s*"([^"]+)"/' data.json

For serious JSON work, use jq. For a quick extract, this works.

Line endings and blank lines

perl -i -pe 's/\r\n/\n/' *.txt
perl -i -pe 's/\n/\r\n/' *.txt
perl -ne 'print unless /^$/' file.txt
perl -00 -pe '' file.txt

Windows to Unix. Unix to Windows. Remove blank lines. Collapse multiple blank lines to one. The -00 flag enables paragraph mode.

Base64, again

This comes up constantly with AI APIs.

cat image.b64 | perl -pe 's/\s//g'
perl -pe 's/(.{76})/$1\n/g' raw.b64

Strip whitespace. Add newlines every 76 characters, the standard wrap.

Vision responses

Given:

NUDE_CHEST: NO
NUDE_LOWER: NO
WEAPON: YES
CONFIDENCE: 0.87

Show me.

perl -ne 'print "$1\n" if /WEAPON:\s*(\w+)/' response.txt
perl -ne 'print "$1\n" if /(\w+):\s*YES/' response.txt
perl -ne 'print "$1\n" if /CONFIDENCE:\s*(\d+\.\d+)/' response.txt

Fix a typo across files

perl -i -pe 's/teh/the/g' *.txt
perl -i -pe 's/version: 1\.0/version: 1.1/' config.yml
perl -i -pe 's/^(.*DEBUG.*)$/# $1/' code.py

Reformat

perl -pe 's/\s+/,/g' data.txt
perl -pe 's/\t/    /g' file.txt
perl -pe 's/  +/ /g' file.txt

Spaces to commas. Tabs to four spaces. Collapse duplicate spaces.

URLs and email

perl -ne 'print "$1\n" while /(https?:\/\/[^\s"<>]+)/g' page.html
perl -ne 'print "$1\n" while /([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})/g' file.txt

The while loop finds all matches, not just the first.

Pipes from other commands

curl -s https://example.com | perl -ne 'print "$1\n" if /<title>([^<]+)</'
find . -name "*.log" | perl -ne 'chomp; print "Found: $_\n"'
ack -o "\d+" | perl -ne '$sum += $_; END { print "Total: $sum\n" }'

The whole file as one string

perl -0777 -pe 's/old/new/g' file.txt
perl -0777 -pe 's|/\*.*?\*/||gs' code.c

-0777 slurps the whole file. Useful for patterns spanning lines. The second one strips C-style comments.

BEGIN and END

perl -ne 'BEGIN { $sum = 0 } $sum += $_; END { print $sum }' nums.txt
perl -pe 'BEGIN { print "=== OUTPUT ===\n" }' file.txt

The -l flag

perl -lne 'print uc' file.txt

Automatically chomps input and adds newlines to output. Cleaner than handling newlines by hand.

See what it is doing

perl -ne 'print "LINE: $_"; print "MATCH: $1\n" if /(\d+)/' file.txt
perl -MData::Dumper -ne '@F = split /,/; print Dumper(\@F)' data.csv

When the one-liner gets too long

Make it a script:

Show me.

#!/usr/bin/env perl
use strict;
use warnings;

while (<>) {
    s/old/new/g;
    print;
}

Save as script.pl, chmod +x, and run on any file. <> is the diamond operator: read from files named on the command line, or from stdin.

Speed, if the file is huge

Avoid capturing groups if you do not need them. Use index() for literal string matching. Pre-compile regex with qr// for repeated matches.

perl -ne 'BEGIN { $re = qr/pattern/ } print if /$re/' huge.txt