Script to fetch top 25 errors from Apache and Php Logs

Script to Fetch Error Logs

Script Overview

In this article, I will provide you bash script which helps you to find top 25 Errors from Apache and Php logs.

Every Time when we face the issue on websites or any application. We need to go on the server and check for error logs. These error logs tell us about what is going wrong with the website. When I faced any issues I am doing the same like searching error logs etc. This is a very time-consuming process to search every log file.
So I have created this script which helps us in a scenario where we face the issue on our website or application.

Script which helps to sync production apache and PHP logs at one place and then process below conditions.

  • Top 25 403 & 404 Error from Apache Access logs with No of counts & URL
  • Top 25 Errors from Apache error logs with No of counts & complete error information
  • Top 25 Errors from PHP error logs with No of counts & complete error information

With help of this script, we are going to fetch or read one-day old error logs files and find out top 25 Errors.

vim /opt/scripts/FetchApacheError.sh

#!/bin/bash

YESTERDAY_DATE=`date --date="Yesterday" +"%Y%m%d"`
APACHE_ACCESS_LOG_FILE="access.log-${YESTERDAY_DATE}.gz"
APACHE_ERROR_LOG_FILE="error.log-${YESTERDAY_DATE}.gz"
PHP_ERROR_LOG_FILE="php-errors.log-${YESTERDAY_DATE}.gz"
MYSQL_SLOW_LOG_FILE=""

RSYNC=`which rsync`

#Add Webserver information

SSH_USERNAME='production-server-username'
SSH_HOSTNAME='production-server-hostname'
#Add Your WebServer IP address from where you want to fetch logs
WEB_SERVERS='webserver-ip webserver-ip2 webserver-ip3'

#Add Status code which wants to monitor or fetch info.
HTTP_STATUS_CODE='403 404'
#Provide path of log file on server
LOG_DIR='/var/log/sites/prod/logs'
#Create directory and touch file to save logs in tmp directory.
TMP_DIR='/tmp/prod_logs'
TMP_LOG_FILE='/tmp/prod_logs_mail.log'

echo "" > $TMP_LOG_FILE

#For loop to which used to fetch errors from all the servers one by one
for WEB_SERVER in ${WEB_SERVERS}; do
[[ ! -d ${TMP_DIR}/${WEB_SERVER} ]] && mkdir -p ${TMP_DIR}/${WEB_SERVER}
$RSYNC -avzP ${SSH_USERNAME}@${WEB_SERVER}${SSH_HOSTNAME}:${LOG_DIR}/${WEB_SERVER}/{$APACHE_ACCESS_LOG_FILE,$APACHE_ERROR_LOG_FILE,$PHP_ERROR_LOG_FILE} ${TMP_DIR}/${WEB_SERVER}
done

#For loop to search status code in each error log
for STATUS_CODE in $HTTP_STATUS_CODE; do
echo -e "Top 25 $STATUS_CODE errors:n" >> $TMP_LOG_FILE
zgrep "" $STATUS_CODE " ${TMP_DIR}/*/${APACHE_ACCESS_LOG_FILE} | awk -F ']' '{print $2}' | awk '{print $2}' | sort | uniq -c | sort -nr | head -n25 >> $TMP_LOG_FILE
echo -e "nn" >> $TMP_LOG_FILE
done

#Below code will fetch Top 25 Apache Errors, PHP Errors.
echo -e "Top 25 Apache errors:n" >> $TMP_LOG_FILE
zcat ${TMP_DIR}/*/${APACHE_ERROR_LOG_FILE} | awk -F ']' '{print $4}' | sort | uniq -c | sort -nr | head -n25 >> $TMP_LOG_FILE
echo -e "nn" >> $TMP_LOG_FILE

echo -e "Top 25 PHP errors:n" >> $TMP_LOG_FILE
zcat ${TMP_DIR}/*/${PHP_ERROR_LOG_FILE} | awk -F ']' '{print $2}' | awk -F 'request_id' '{print $1}' | sort | uniq -c | sort -nr | head -n25 >> $TMP_LOG_FILE
echo -e "nn" >> $TMP_LOG_FILE

Save & close the file.

Note : Remove the comments from script followed with # before setting up or running the script.

Execute Permission:

Give execute permission which is important to run the .sh file.

chmod +x /opt/scripts/FetchApacheError.sh

Also, you can set cron to run the script on a daily or weekly basis which will provide you data or logs.

That’s it . You can modify above script according to your requirement. If you still face any issues to setup this script then comments down below in comment box. Stay connected & subscribe with techthings.org for more such articles.

Leave a Reply

Your email address will not be published. Required fields are marked *