Linux reference
Task-first recipes for the things you actually do at 2am.
Find what is using a port
Problem · Something owns :8080 and your service can't bind.
ss -tulpn | grep :8080 sudo lsof -i :8080
ss/lsof map the listening socket to a PID and command — that's who to stop.
`netstat` is deprecated; prefer `ss`. Run with sudo to see other users' processes.
Check why a service failed
Problem · A unit is 'failed' and you need the real reason.
systemctl status <svc> journalctl -u <svc> -n 50 --no-pager journalctl -u <svc> -b
status = current state; journal = the story. Look for the last FATAL/exit-code line.
After editing a unit file you must `systemctl daemon-reload` or changes are ignored.
Find large files safely
Problem · Disk is filling and you need the offenders fast.
df -h du -xsh /* 2>/dev/null | sort -h | tail du -xsh /var/* | sort -h | tail
df finds the full filesystem; du drills into it. `-x` stays on one filesystem.
If df and du disagree, suspect a deleted-but-open file: `lsof | grep deleted`.
Inspect logs with journalctl
Problem · You need recent, relevant log lines without drowning.
journalctl -u nginx -f journalctl --since '10 min ago' journalctl -p err -b
-u scopes to a unit, -f follows, -p err shows errors this boot.
Journald may be volatile (lost on reboot) unless persistent storage is enabled.
Check memory pressure
Problem · Is the box actually low on memory, or is that just cache?
free -h cat /proc/meminfo | grep -i available dmesg -T | grep -i oom
Look at 'available', not 'free' — Linux uses spare RAM as cache. OOM kills show in dmesg.
High 'used' with healthy 'available' is normal. Don't panic at buff/cache.
Understand load average
Problem · Load is high but you're not sure what it means.
uptime top # check %wa (IO wait) iostat -x 1 3 ps -eo state,pid,cmd | grep '^D'
Load counts runnable AND uninterruptible (D-state) tasks. High load + idle CPU = I/O wait.
Adding CPUs won't help an I/O-bound box. Check the disk first.
Trace DNS resolution
Problem · A name won't resolve and you need to know which step fails.
dig +trace example.com dig @<resolver> example.com getent hosts example.com cat /etc/resolv.conf
Compare what your resolver returns vs the authoritative answer; getent honours nsswitch (incl. /etc/hosts).
/etc/hosts overrides DNS. A wrong answer on one box is usually local config.
Check disk and inode usage
Problem · Writes fail with 'No space left' but df shows free space.
df -h df -i
You can exhaust inodes (df -i) while bytes are free — common with millions of tiny files.
'No space left on device' can mean inodes, not bytes. Always check `df -i` too.