cat access.log | awk ‘{print $1 ” ” $4}’<\/p>\n
Below are some simple awk\/sed\/etc command line scripts to parse apache logs and get quick statistics<\/p>\n
Unique visitors per day<\/p>\n
Where access.log is your combined log file with typical format as below:<\/p>\n
access.log
\n69.175.xxx.yyy – – [13\/Jul\/2013:06:28:31 -0500] “GET \/some\/web\/folder\/somewebpage2 HTTP\/1.0” 404 4212 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/some_web_page1” “Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/535.1 (KHTML, like Gecko) Chrome\/13.0.782.112 Safari\/535.1”
\n69.175.xxx.yyy – – [13\/Jul\/2013:06:28:35 -0500] “GET \/some\/web\/folder\/?do=register HTTP\/1.0” 302 599 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/535.1 (KHTML, like Gecko) Chrome\/13.0.782.112 Safari\/535.1”
\n69.175.xxx.yyy – – [13\/Jul\/2013:06:28:36 -0500] “GET \/some\/web\/folder\/somewebpage2 HTTP\/1.0” 404 4212 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/535.1 (KHTML, like Gecko) Chrome\/13.0.782.112 Safari\/535.1”
\n69.175.xxx.yyy – – [13\/Jul\/2013:06:28:41 -0500] “POST \/some\/web\/folder\/somewebpage2 HTTP\/1.0” 200 2439 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64) AppleWebKit\/535.1 (KHTML, like Gecko) Chrome\/13.0.782.112 Safari\/535.1”<\/p>\n
94.123.xxx.yyy – – [13\/Jul\/2013:06:32:49 -0500] “GET \/some\/web\/folder\/some_web_page1 HTTP\/1.1” 200 5121 “http:\/\/somesubdomain.example.org\/” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0”
\n94.123.xxx.yyy – – [13\/Jul\/2013:06:32:49 -0500] “GET \/some\/web\/folder\/?do=login HTTP\/1.1” 302 599 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/some_web_page1” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0”
\n94.123.xxx.yyy – – [13\/Jul\/2013:06:32:50 -0500] “GET \/some\/web\/folder\/somewebpage2 HTTP\/1.1” 404 4214 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/some_web_page1” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0”
\n94.123.xxx.yyy – – [13\/Jul\/2013:06:32:50 -0500] “GET \/some\/web\/folder\/?do=register HTTP\/1.1” 302 599 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0”
\n94.123.xxx.yyy – – [13\/Jul\/2013:06:32:50 -0500] “GET \/some\/web\/folder\/somewebpage2 HTTP\/1.1” 404 4214 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0”
\n94.123.xxx.yyy – – [13\/Jul\/2013:06:32:51 -0500] “POST \/some\/web\/folder\/somewebpage2 HTTP\/1.1” 200 2530 “http:\/\/somesubdomain.example.org\/some\/web\/folder\/somewebpage2” “Mozilla\/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko\/20100101 Firefox\/5.0″
\nBelow is the command line script. This gets the unique hits per day. There is a grep at the very end to do a final filter for the Month and Year you may be looking for.<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’ | sed ‘s\/\\[\/\/’ | cut -d”:” -f1 | awk ‘{print $2 ” ” $1}’ | \\
\nsort | uniq | awk ‘{print $1}’ | uniq -c | grep “Feb\/2013″
\nThe output is as below<\/p>\n
52 01\/Feb\/2013
\n 63 02\/Feb\/2013
\n 47 03\/Feb\/2013
\n 62 04\/Feb\/2013
\n 59 05\/Feb\/2013
\n 63 06\/Feb\/2013
\n …
\netc.
\nExplanation of the command<\/p>\n
This may be useful if you want to tweak it.<\/p>\n
Break down 1<\/p>\n
Get IP and date (unformatted at this stage)<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’
\nOutput of the above<\/p>\n
69.175.xxx.yyy [13\/Jul\/2013:06:28:31
\n69.175.xxx.yyy [13\/Jul\/2013:06:28:35
\n69.175.xxx.yyy [13\/Jul\/2013:06:28:36
\n69.175.xxx.yyy [13\/Jul\/2013:06:28:41
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:49
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:49
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:50
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:50
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:50
\n94.123.xxx.yyy [13\/Jul\/2013:06:32:51
\nBreak down 2<\/p>\n
Remove the [ bracket with sed. Remove the time portion of the output with cut.<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’ | sed ‘s\/\\[\/\/’ | cut -d”:” -f1
\nOutput of the above<\/p>\n
69.175.xxx.yyy 13\/Jul\/2013
\n69.175.xxx.yyy 13\/Jul\/2013
\n69.175.xxx.yyy 13\/Jul\/2013
\n69.175.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\n94.123.xxx.yyy 13\/Jul\/2013
\nBreak down 3<\/p>\n
Swap IP and Date such that Date is 1st and IP is 2nd<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’ | sed ‘s\/\\[\/\/’ | cut -d”:” -f1 | awk ‘{print $2 ” ” $1}’ |
\nOutput of the above<\/p>\n
13\/Jul\/2013 69.175.xxx.yyy
\n13\/Jul\/2013 69.175.xxx.yyy
\n13\/Jul\/2013 69.175.xxx.yyy
\n13\/Jul\/2013 69.175.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\nBreak down 4<\/p>\n
Remove duplicate IPs – to get unique IP hits per day<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’ | sed ‘s\/\\[\/\/’ | cut -d”:” -f1 | awk ‘{print $2 ” ” $1}’ | \\
\nsort | uniq
\nOutput of the above<\/p>\n
13\/Jul\/2013 69.175.xxx.yyy
\n13\/Jul\/2013 94.123.xxx.yyy
\nBreak down 5<\/p>\n
Now the IPs are no longer interesting as we only need their count. So remove IP by printing only date. Then do a uniq -c on the output to get the counts for the dates<\/p>\n
cat access.log | awk ‘{print $1 ” ” $4}’ | sed ‘s\/\\[\/\/’ | cut -d”:” -f1 | awk ‘{print $2 ” ” $1}’ | \\
\nsort | uniq | awk ‘{print $1}’ | uniq -c
\nOutput of the above. So there are only two unique hits in our example for the one date.<\/p>\n
http:\/\/tech.snathan.org\/tech\/apache\/log_parsing<\/p>\n
2 13\/Jul\/2013<\/p>\n","protected":false},"excerpt":{"rendered":"
cat access.log | awk ‘{print $1 ” ” $4}’ Below are some simple awk\/sed\/etc command line scripts to parse apache logs and get quick statistics Unique visitors per day Where access.log is your combined log file with typical format as below: access.log 69.175.xxx.yyy – –…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_angie_page":false,"page_builder":"","footnotes":""},"categories":[5],"tags":[],"class_list":["post-2801","post","type-post","status-publish","format-standard","hentry","category-apache2"],"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts\/2801","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/comments?post=2801"}],"version-history":[{"count":0,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/posts\/2801\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/media?parent=2801"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/categories?post=2801"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.deuzebranaweb.com.br\/index.php\/wp-json\/wp\/v2\/tags?post=2801"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}