Linux Commands for Debugging a Remote Box

Your app works on your laptop. It returns 502s on the server. Your options are to SSH in and find out why, or to guess. This section gives you a small set of commands and a fixed order to run them in. The same set covers most production incidents you will meet as an AI engineer, from a FastAPI service that will not start, to a full disk, to an inference worker the kernel killed for using too much memory.

1. Getting onto the box

Save the details in ~/.ssh/config so you can type ssh app instead:

Useful companions:

• scp app:/var/log/app/error.log . copies a file down to your machine. rsync -avz ./dist/ app:/srv/app/ syncs a directory up.
• tmux new -s debug starts a session that survives a dropped connection. Reattach with tmux attach -t debug. Run long jobs inside tmux.
• If SSH fails with Permission denied (publickey), check the user name first, then the key path, then the key's permissions: chmod 600 ~/.ssh/app-key.pem.

2. The triage order

When something is broken, work through these questions in order. Most incidents are answered by step 3.

1. Is the process running?
2. Is it listening on the expected port and interface?
3. What do the logs say?
4. Is the box out of a resource such as disk, memory or CPU?
5. Can it reach its dependencies, such as Postgres or an external API?

3. Processes: is it running?

htop is friendlier if it is installed. In top, a high load average relative to the CPU count (nproc) means work is queueing. A high wa value means the CPU is waiting on disk.