Analyze Hosting Access Logs to Find Bots and Security Issues

Your hosting server's access log is one of the most valuable diagnostic tools available to a website owner. Every HTTP request — whether from a real visitor, a search engine crawler, a security scanner, or a brute force attacker — leaves a trace in this file. If your website has slowed down inexplicably, is receiving suspicious traffic, or you suspect a security incident, analyzing the access log is always the right first step. This guide walks you through reading Apache access logs on DirectAdmin hosting, extracting meaningful insights, and identifying bots and security threats using standard Linux command-line tools.

What Is an Access Log and Where to Find It in DirectAdmin

Apache Web Server, the engine behind most shared hosting environments, writes a record of every incoming HTTP request to the access log automatically. This includes page views, image loads, API calls, and crawl requests from bots of all kinds — legitimate and otherwise.

In DirectAdmin hosting, you can access log files in several ways:

• Via DirectAdmin Panel: Navigate to Advanced Features → Error/Access Logs, select your domain, then view or download the log file directly from the browser.
• Via File Manager: Browse to the ~/logs/ or ~/access-logs/ directory. You will typically find a file named domainname.com-access_log.
• Via SSH: If your hosting plan supports SSH access, connect and read the file directly at /home/username/logs/domainname.com-access_log.

Typical log file paths in a DirectAdmin environment:

/home/username/logs/domainname.com-access_log
/home/username/logs/domainname.com-error_log

Understanding the Access Log Format — Field by Field

Apache's default Combined Log Format structures each log entry as follows:

IP - - [DATE TIME +ZONE] "METHOD URL PROTOCOL" STATUS BYTES "REFERER" "USER-AGENT"

Here is a real-world example with two entries — one from a legitimate visitor and one from a scanning bot:

203.150.45.12 - - [09/Jun/2026:10:23:41 +0700] "GET /about.html HTTP/1.1" 200 8452 "https://google.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
185.220.101.5 - - [09/Jun/2026:10:24:02 +0700] "GET /wp-login.php HTTP/1.1" 404 1234 "-" "python-requests/2.28.0"

Breaking down each field:

Essential grep and AWK Commands for Log Analysis

With the log format understood, you can use standard Linux tools — grep, awk, sort, and uniq — to filter, count, and surface patterns. These tools come pre-installed on every Linux server and require no additional software.

View the 20 most recent log entries

tail -n 20 ~/logs/domainname.com-access_log

Count requests per IP address (Top 20)

awk '{print $1}' ~/logs/domainname.com-access_log | sort | uniq -c | sort -rn | head -20

Show all requests from a specific IP

grep "^185.220.101.5" ~/logs/domainname.com-access_log

Find all 404 errors and group by URL

grep '" 404 ' ~/logs/domainname.com-access_log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Identify non-browser User-Agents

grep -v "Mozilla\|Googlebot\|Bingbot\|Twitterbot\|facebookexternalhit" ~/logs/domainname.com-access_log | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20

Identifying Bad Bots in Access Logs

Not every automated request is harmful, but certain bot types consume server resources without contributing value, or actively probe for vulnerabilities. Understanding how to distinguish bad bots from legitimate crawlers is a fundamental security skill for any website operator.

Characteristics that flag a bad bot in access logs:

• Script-like User-Agents: Strings such as python-requests, Go-http-client, curl/, libwww-perl, scrapy indicate automated tools with no disguise.
• Empty or minimal User-Agent: A User-Agent of "-", a single word like "test", or just "Mozilla" without version information is suspicious.
• Rapid sequential scanning of non-existent paths: Requests to /.env, /config.php, /backup.zip, /admin, /.git/config in quick succession indicate a vulnerability scanner.
• High 404 rate from a single IP: A legitimate user who mistyped a URL will generate one or two 404s. A scanner generating hundreds of 404s from one IP in a short window is a clear signal.

Find the IPs generating the most 404 errors (scanner signature):

grep '" 404 ' ~/logs/domainname.com-access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

List suspicious User-Agents by request volume:

awk -F'"' '{print $6}' ~/logs/domainname.com-access_log | sort | uniq -c | sort -rn | grep -i "python\|curl\|go-http\|libwww\|scan\|bot\|crawler" | head -20

Detecting Brute Force Attacks on WordPress Sites

WordPress login endpoints are among the most attacked paths on the internet. If you run a WordPress site, monitoring /wp-login.php and /xmlrpc.php in your access log should be a regular habit. A brute force attack shows up as a single IP — or a coordinated set of IPs from the same subnet — sending repeated POST requests to these paths in a short period.

# Count POST requests to wp-login.php by IP
grep '"POST /wp-login.php' ~/logs/domainname.com-access_log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

# Count POST requests to xmlrpc.php by IP
grep '"POST /xmlrpc.php' ~/logs/domainname.com-access_log | awk '{print $1}' | sort | uniq -c | sort -rn

If any IP has more than 20 POST attempts in a 10-minute window, block it immediately via .htaccess:

# Block a single IP in .htaccess
Order allow,deny
Allow from all
Deny from 185.220.101.5

# Block an entire /24 subnet
Deny from 185.220.101.0/24

For persistent or distributed brute force attacks, consider adding a CAPTCHA to the WordPress login page, disabling xmlrpc.php entirely if you do not use it, or moving the login URL using a security plugin.

Detecting Other Security Issues in Access Logs

Access logs expose much more than brute force activity. The following patterns in log entries are indicators of other common web application attacks:

Path Traversal Attempts — trying to access files outside the web root

grep "\.\./\.\." ~/logs/domainname.com-access_log | head -20

SQL Injection Probing — keywords appearing in request URLs

grep -i "union\|select\|from\|where\|drop\|insert\|update\|delete\|cast(" ~/logs/domainname.com-access_log | head -20

Sensitive File Enumeration — scanning for exposed configuration files

grep -E "\.env|config\.php|wp-config\.php|\.git|backup\.zip|database\.sql|phpinfo\.php" ~/logs/domainname.com-access_log

Webshell Upload Probing — POST requests targeting shell execution

grep -i "\.php.*upload\|upload.*\.php\|POST.*shell\|cmd=\|exec=\|system=" ~/logs/domainname.com-access_log

When you find any of these patterns, record the offending IPs and note the timestamp range. This information is essential if you need to file an abuse report or work with your hosting provider to investigate further.

Traffic Pattern Analysis Using Access Logs

Beyond security, access logs are an excellent source of traffic intelligence. You can identify your most visited pages, peak traffic hours, large response payloads, and unusual data egress — all without installing any additional analytics software.

Top 20 most requested pages (excluding static assets)

awk '{print $7}' ~/logs/domainname.com-access_log | grep -v "\.jpg\|\.png\|\.css\|\.js\|\.ico\|\.woff" | sort | uniq -c | sort -rn | head -20

Request volume by hour of day

awk '{print $4}' ~/logs/domainname.com-access_log | cut -d: -f2 | sort | uniq -c

HTTP status code distribution

awk '{print $9}' ~/logs/domainname.com-access_log | sort | uniq -c | sort -rn

Detect abnormally large responses (potential data exfiltration)

awk '{if($10 > 1000000) print $1, $7, $10}' ~/logs/domainname.com-access_log | sort -k3 -rn | head -10

This last command surfaces requests that returned responses larger than 1 MB, listed with the originating IP and the requested URL. If a URL that should return a small response (such as a login page) is sending megabytes of data back to an unusual IP, it may indicate unauthorized data extraction.

Automating Log Monitoring with a Daily Shell Script

Running individual commands manually is useful for one-off investigations, but setting up an automated daily report lets you catch problems quickly without needing to remember all the commands. The following script summarizes the key security signals from the current day's log:

#!/bin/bash
LOGFILE="$HOME/logs/domainname.com-access_log"
TODAY=$(date "+%d/%b/%Y")
echo "=== Daily Log Report: $TODAY ==="
echo ""
echo "--- Top 10 IP Addresses ---"
grep "$TODAY" "$LOGFILE" | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
echo ""
echo "--- HTTP Status Code Breakdown ---"
grep "$TODAY" "$LOGFILE" | awk '{print $9}' | sort | uniq -c | sort -rn
echo ""
echo "--- Top 10 URLs Returning 404 ---"
grep "$TODAY" "$LOGFILE" | grep '" 404 ' | awk '{print $7}' | sort | uniq -c | sort -rn | head -10
echo ""
echo "--- Non-Browser User-Agents ---"
grep "$TODAY" "$LOGFILE" | grep -v "Mozilla" | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -10

Save this file at ~/scripts/log-report.sh and schedule it as a daily cron job:

0 0 * * * /bin/bash ~/scripts/log-report.sh >> ~/logs/daily-report.txt 2>&1

Review ~/logs/daily-report.txt each morning to stay on top of unusual activity before it escalates.

Frequently Asked Questions

Where are access logs located in DirectAdmin hosting?

In DirectAdmin, go to Advanced Features then Apache Logs, or browse directly through File Manager to the ~/logs/ or ~/access-logs/ directory. You will typically find files named domainname.com-access_log there. If your hosting plan supports SSH, you can also access logs at /home/username/logs/ directly from the command line.

How do I tell a good bot from a bad bot in access logs?

Good bots such as Googlebot and Bingbot declare their User-Agent clearly, respect your robots.txt rules, and originate from IP addresses that can be verified via reverse DNS lookup. Bad bots typically use fake or empty User-Agents, send requests to non-existent paths in rapid succession, generate unusually high 404 error rates, or originate from datacenter IP ranges with no legitimate business identity.

Why is the same IP address sending thousands of requests per day?

There are three common causes: 1) A vulnerability scanner probing various paths and parameters looking for weaknesses. 2) A brute force attack targeting login endpoints such as /wp-login.php or /xmlrpc.php, trying many username and password combinations. 3) A botnet DDoS where some nodes send repeated requests from the same IP. Mitigation involves blocking the offending IP via .htaccess, using a web application firewall, or installing fail2ban if root access is available.

How long should I retain access logs for security purposes?

The general best practice is to retain access logs for at least 90 days for security investigation purposes, and up to 1 year for compliance in regulated industries. If your hosting plan rotates logs aggressively, set up a cron job to compress and archive logs to a remote location daily. Having sufficient historical data is critical for forensic analysis when a security incident is discovered weeks after it occurred.