What to Do When Your Server Is Compromised: Investigate, Clean Up, and Respond

When your server starts behaving strangely, don't panic — and don't rush to delete things. The correct order for incident response is: confirm the signs → isolate and contain → preserve evidence → clean up or rebuild → harden afterward. The steps below use Ubuntu/Debian as the reference.

1. Recognizing the Signs

Common red flags include:

• CPU or memory pegged at full load with no matching legitimate workload (a classic sign of a cryptominer).
• A spike in outbound traffic, or connections to unfamiliar IPs and ports.
• Processes, cron jobs, or user accounts you don't recognize.
• Successful logins from strange IPs or at odd hours in your login history.

2. Investigating: Find the Problem

Check processes and resource use

top                            # watch top CPU/memory consumers live
ps aux --sort=-%cpu | head     # list processes sorted by CPU

For anything suspicious, note the PID and find its binary: ls -l /proc/<PID>/exe.

Check network connections

ss -tunp            # list all TCP/UDP connections and their processes

Pay attention to ESTAB (established) outbound connections and anything listening on unusual ports.

Check login history

last -a             # recent logins/logouts with source IPs
lastb               # failed login attempts (signs of brute forcing)

Then scan the auth log for successful logins from unfamiliar IPs:

grep -i "accepted" /var/log/auth.log

Look for persistence (backdoors)

Attackers usually leave a way back in. Check these one by one:

crontab -l                        # current user's cron jobs
ls -l /etc/cron.* /etc/crontab    # system-wide cron jobs
cat ~/.ssh/authorized_keys        # any unfamiliar public keys added?

Also inspect /etc/passwd for unknown users and systemctl list-units --type=service for suspicious services.

3. Responding: Isolate and Preserve Evidence

Once a compromise is confirmed, contain it first:

• Immediately change every relevant password (system accounts, databases, control panels).
• Cut off malicious outbound traffic at the firewall — for example, temporarily block a suspicious destination with ufw deny out to <IP>. Tighten your security group so only your own IP can SSH in if needed.

The key principle: preserve evidence and back up important data before you clean anything. This protects both the investigation and your data:

# Archive key logs and configs as evidence
tar czf /root/forensic-$(date +%F).tar.gz \
  /var/log /etc/crontab /etc/passwd ~/.ssh

Copy your website files, a database dump, and logs to a separate, clean machine — and don't overwrite your existing backups.

4. Clean Up or Rebuild?

If it's a single, well-understood vulnerability with a clear blast radius, a targeted cleanup may do: kill the malicious processes, delete backdoor files and cron jobs, remove unfamiliar keys and users, then patch.

But whenever you can't be certain the system is free of a hidden backdoor (a rootkit), the safest move is to rebuild the server from scratch and restore data from a clean, pre-compromise backup. Don't keep a questionable system just to save time — a second breach costs far more than a reinstall.

5. Hardening Afterward

Before going back online, harden the box:

• Disable password login in favor of SSH keys; change the default SSH port.
• Install fail2ban to block brute-force attempts, and configure your firewall (ufw) to allow only the ports you need.
• Keep the system patched: apt update && apt upgrade.
• Set up regular offline backups and basic monitoring/alerts so you catch the next anomaly sooner.

Summary

The golden order for incident response is: confirm, isolate, preserve evidence, then clean up or rebuild. Use top/ss/last to pin down the anomaly, and check cron jobs and authorizedkeys for backdoors. Always back up important data for forensics before you touch anything, and if you can't be sure the system is clean, just rebuild and restore from a clean backup — then finish with key-based login, a tight firewall, and regular backups.