NerdVana

freq with me

Written 8 years ago by Eric Schwimmer

I often need to count the frequency of events occurring in log files and what not, and I was never able to find a super clean way to do it in real time, so this perl script was born:

#!/usr/bin/perl

use IO::Select;
use Fcntl;

my (%counts, $start, @lines, $flags, $buffer);

my $select = IO::Select->new();
$select->add(\*STDIN);

my $flags;
fcntl(STDIN, F_GETFL, $flags);
$flags |= O_NONBLOCK;
fcntl(STDIN, F_SETFL, $flags);
my $start = time;

while(1)
{
    print "\033[2J";
    print "\033[0;0H";
    sleep(1);
    @lines = ();
    if ($select->can_read(0))
    {
        @lines = <STDIN>;
        $lines[0] = $buffer . $lines[0];
        $buffer = ($lines[$#lines] =~ /\n$/) ? "" : pop @lines;

        for my $line (@lines)
        {
            chomp $line;
            $counts{$line}->[0]++;
        }

        my $elapsed = time - $start;
        for my $line (sort keys %counts)
        {
            my $rates = $counts{$line};
            my $sec_rate = $rates->[0];
            $rates->[3] += $sec_rate;
            my $total = $rates->[3];

            push @{$rates->[1]}, $sec_rate;
            shift @{$rates->[1]} if scalar @{$rates->[1]} > 60;
            push @{$rates->[2]}, $sec_rate;
            shift @{$rates->[2]} if scalar @{$rates->[2]} > 300;
            my $min1_rate = avg($rates->[1]);
            my $min5_rate = avg($rates->[2]);

            printf "%s -- total: %s, 1 sec: %s, 1 min: %.2f/s, 5 min: %.2f/s\n",
                $line, $total, $sec_rate, $min1_rate, $min5_rate;

            $rates->[0] = 0;
        }
    }
}

sub avg 
{
    my ($list) = @_;
    my $total = 0;
    $total += $_ for (@$list);
    return $total / scalar @$list;
}

Assuming you have jq installed and your logs are in JSON format, you can do something nifty like this:

tail -F /data/nginx/access_log | jq .status | freq

And it will spit out a real time display that looks like this (updated in real time):

"200" -- total: 45226, 1 sec: 676, 1 min: 674.75/s, 5 min: 675.01/s
"204" -- total: 133, 1 sec: 2, 1 min: 2.00/s, 5 min: 1.99/s        
"304" -- total: 70, 1 sec: 1, 1 min: 1.03/s, 5 min: 1.04/s         
"403" -- total: 3, 1 sec: 0, 1 min: 0.08/s, 5 min: 0.08/s          
"404" -- total: 253, 1 sec: 3, 1 min: 4.02/s, 5 min: 3.89/s        
"500" -- total: 52, 1 sec: 0, 1 min: 0.80/s, 5 min: 0.78/s         
"502" -- total: 1, 1 sec: 0, 1 min: 0.05/s, 5 min: 0.05/s