98:I[12737,["/p/_next/static/chunks/3fntmmi971322.js","/p/_next/static/chunks/1hqu0koe_wtia.js","/p/_next/static/chunks/13bbhchlfynjz.js","/p/_next/static/chunks/2se6u63yuc59i.js"],"LabScreen"] 9f:I[27201,["/p/_next/static/chunks/3fntmmi971322.js"],"IconMark"] 99:T43b,Last night a teammate ran `git reset --hard origin/release/2.3` to clean up before a deploy. `release/2.3` lost two commits: the OPS-311 DB-retry hotfix and the worker memory bump. The redeploy then crash-looped. I copied the box's `git reflog` into `data/reflog_release.txt` and the `docker compose up` output into `data/compose_up.log`. `python main.py` should print the incident summary we paste into the channel. It doesn't, because `find_lost_commit` (recovery.py) and `summarize_container_errors` (triage.py) are still stubs, so the report says nothing was lost and nothing failed. The parsers already work. Read `reflog.py`, `composelog.py` and `models.py` before you write any splitting of your own; the rules you need are mostly there already. I wrote the failing tests in `tests/test_oncall.py` first, the way we drive Claude Code and Cursor here. Get them green with small diffs, and leave the read-only files alone. The docstrings are the spec. Two things to keep in mind: the reflog is printed newest first, and `exit 137` means something very different from `exit 1`.9a:T50f,from dataclasses import dataclass ERROR_LEVELS = {"ERROR", "CRITICAL", "FATAL"} @dataclass(frozen=True) class ReflogEntry: """One line of `git reflog`: : : .""" sha: str selector: str # "HEAD@{3}" action: str # "commit", "commit (amend)", "checkout", "reset", "merge hotfix/x", ... subject: str # the commit message for commits, otherwise what the action did @property def makes_commit(self) -> bool: """True if this entry recorded a new commit, so subject is that commit's message.""" verb = self.action.split(" ", 1)[0] return verb in {"commit", "cherry-pick", "revert"} or self.action.endswith("(pick)") @dataclass(frozen=True) class LogEvent: """One line of `docker compose up` output: a service log line or a container exit.""" container: str message: str = "" level: str = "" # "INFO", "ERROR", ... or "" if the line has no level exit_code: int | None = None @property def is_exit(self) -> bool: return self.exit_code is not None @property def is_error(self) -> bool: return self.level in ERROR_LEVELS @dataclass(frozen=True) class ContainerFailure: container: str exit_code: int count: int last_error: str 9b:T4fa,from exit_codes import explain_exit_code from recovery import find_lost_commit from triage import summarize_container_errors def read_lines(path: str) -> list[str]: with open(path, encoding="utf-8") as f: return f.read().splitlines() def build_report(reflog_lines: list[str], log_lines: list[str], wanted: list[str]) -> str: """The incident summary on-call pastes into the channel.""" out = ["== Lost commits =="] for message in wanted: sha = find_lost_commit(reflog_lines, message) if sha is None: out.append(f"{message!r}: not in the reflog; ask the author to push it again") else: out.append(f"{message!r}: {sha} -> git branch rescue/{sha} {sha}") out += ["", "== Failed containers =="] failures = summarize_container_errors(log_lines) if not failures: out.append("none") for f in failures: out.append(f"{f.container}: exit {f.exit_code} x{f.count} ({explain_exit_code(f.exit_code)})") out.append(f" last error: {f.last_error or '(nothing logged; check docker inspect)'}") return "\n".join(out) def run(reflog_path: str, log_path: str, wanted: list[str]) -> None: print(build_report(read_lines(reflog_path), read_lines(log_path), wanted)) 9c:T577,import re from models import LogEvent EXIT_LINE = re.compile(r"^(?P[\w.-]+) exited with code (?P

\d+)\s*$")
SERVICE_LINE = re.compile(r"^(?P[\w.-]+)\s+\|(?P.*)$")
LEVEL = re.compile(r"^\[?(?P[A-Za-z]+)\b")
LEVELS = {"DEBUG", "INFO", "WARN", "WARNING", "ERROR", "CRITICAL", "FATAL"}


def parse_log_line(line: str) -> LogEvent | None:
    """
    Parse one line of `docker compose up` output:
        worker-1   | ERROR redis not reachable  -> LogEvent("worker-1", "ERROR redis not reachable", "ERROR")
        worker-1 exited with code 137           -> LogEvent("worker-1", exit_code=137)
    message is the text after the "|", stripped. level is its first word, upper-cased,
    if that word is a log level ("error:" and "[ERROR]" count), otherwise "".
    Compose's own chatter ("Attaching to ...", "[+] Running 4/4") and blank lines give None.
    """
    line = line.rstrip()
    exit_match = EXIT_LINE.match(line)
    if exit_match:
        return LogEvent(exit_match["container"], exit_code=int(exit_match["code"]))
    service_match = SERVICE_LINE.match(line)
    if not service_match:
        return None
    message = service_match["message"].strip()
    level_match = LEVEL.match(message)
    level = level_match["level"].upper() if level_match else ""
    return LogEvent(service_match["container"], message, level if level in LEVELS else "")
9d:T672,[+] Running 4/4
 ✔ Container app-db-1       Started
 ✔ Container app-migrate-1  Started
 ✔ Container app-api-1      Started
 ✔ Container app-worker-1   Started
Attaching to api-1, db-1, migrate-1, worker-1
db-1       | PostgreSQL init process complete; ready for start up.
migrate-1  | INFO  [alembic.runtime.migration] Running upgrade 1f2e -> 3a4b, add orders index
migrate-1 exited with code 0
api-1      | INFO:     Started server process [1]
api-1      | INFO:     Waiting for application startup.
worker-1   | INFO worker booting, concurrency=4
api-1      | ERROR:    could not connect to postgres at db:5432: connection refused
api-1      | ERROR:    Application startup failed. Exiting.
api-1 exited with code 1
db-1       | database system is ready to accept connections
worker-1   | ERROR redis not reachable at cache:6379, retrying in 2s
worker-1   | INFO redis connected
worker-1   | INFO loading embeddings cache (230 MB)
api-1      | INFO:     Started server process [1]
api-1      | INFO:     Application startup complete.
api-1      | INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
worker-1 exited with code 137
worker-1   | INFO worker booting, concurrency=4
worker-1   | INFO loading embeddings cache (230 MB)
worker-1   | WARNING memory at 251 MiB of 256 MiB limit
worker-1 exited with code 137
api-1      | INFO:     172.18.0.1:50132 - "GET /health HTTP/1.1" 200 OK
worker-1   | INFO worker booting, concurrency=4
worker-1   | INFO loading embeddings cache (230 MB)
worker-1 exited with code 137
Gracefully stopping... (press Ctrl+C again to force)
api-1 exited with code 0
db-1 exited with code 0
9e:T8bb,from recovery import find_lost_commit
from triage import summarize_container_errors


def summary(log_lines):
    return [(f.container, f.exit_code, f.count, f.last_error)
            for f in summarize_container_errors(log_lines)]


def test_finds_commit_by_message():
    reflog = [
        "8a1f2b3 HEAD@{0}: checkout: moving from hotfix/OPS-311-db-retry to release/2.3",
        "b7c41e9 HEAD@{1}: commit: OPS-311: retry DB connect on startup",
        "8a1f2b3 HEAD@{2}: checkout: moving from release/2.3 to hotfix/OPS-311-db-retry",
    ]
    assert find_lost_commit(reflog, "ops-311") == "b7c41e9"


def test_newest_matching_commit_wins():
    reflog = [
        "d41f0a2 HEAD@{0}: commit (amend): OPS-311: retry DB connect on startup, with backoff",
        "b7c41e9 HEAD@{1}: commit: OPS-311: retry DB connect on startup",
        "8a1f2b3 HEAD@{2}: commit: OPS-298: add /health endpoint",
    ]
    assert find_lost_commit(reflog, "retry DB connect") == "d41f0a2"


def test_groups_nonzero_exits():
    log = [
        "migrate-1 exited with code 0",
        "api-1      | ERROR:    could not connect to postgres",
        "api-1 exited with code 1",
        "api-1      | ERROR:    could not connect to postgres",
        "api-1 exited with code 1",
    ]
    assert summary(log) == [("api-1", 1, 2, "ERROR:    could not connect to postgres")]


def test_last_error_is_per_run():
    log = [
        "worker-1   | ERROR redis not reachable",
        "api-1      | ERROR:    Application startup failed.",
        "worker-1 exited with code 1",
        "worker-1   | INFO worker booting",
        "worker-1 exited with code 137",
        "api-1 exited with code 3",
    ]
    assert summary(log) == [
        ("api-1", 3, 1, "ERROR:    Application startup failed."),
        ("worker-1", 1, 1, "ERROR redis not reachable"),
        ("worker-1", 137, 1, ""),
    ]


def test_sorted_by_count_then_name():
    log = [
        "web-1 exited with code 1",
        "api-1 exited with code 2",
        "web-1 exited with code 1",
        "api-1 exited with code 1",
    ]
    assert [(f.container, f.exit_code, f.count) for f in summarize_container_errors(log)] == [
        ("web-1", 1, 2), ("api-1", 1, 1), ("api-1", 2, 1),
    ]
90:["$","$L98",null,{"lab":{"id":27,"slug":"ops-318-reflog-deploy-triage","title":"Recover the lost hotfix and triage the crash-looping deploy","afterSection":"1.5","manifest":{"slug":"ops-318-reflog-deploy-triage","files":[{"path":"composelog.py","language":"python","read_only":true},{"path":"data/compose_up.log","language":"log","read_only":true},{"path":"data/reflog_release.txt","language":"txt","read_only":true},{"path":"exit_codes.py","language":"python","read_only":true},{"path":"main.py","language":"python","read_only":true},{"path":"models.py","language":"python","read_only":true},{"path":"oncall.py","language":"python","read_only":true},{"path":"recovery.py","language":"python","read_only":false},{"path":"reflog.py","language":"python","read_only":true},{"path":"tests/test_oncall.py","language":"python","read_only":true},{"path":"triage.py","language":"python","read_only":false}],"graph":{"edges":[{"to":"models.py","from":"composelog.py","names":["LogEvent"]},{"to":"oncall.py","from":"main.py","names":["run"]},{"to":"exit_codes.py","from":"oncall.py","names":["explain_exit_code"]},{"to":"recovery.py","from":"oncall.py","names":["find_lost_commit"]},{"to":"triage.py","from":"oncall.py","names":["summarize_container_errors"]},{"to":"reflog.py","from":"recovery.py","names":["parse_reflog_line"]},{"to":"models.py","from":"reflog.py","names":["ReflogEntry"]},{"to":"recovery.py","from":"tests/test_oncall.py","names":["find_lost_commit"]},{"to":"triage.py","from":"tests/test_oncall.py","names":["summarize_container_errors"]},{"to":"composelog.py","from":"triage.py","names":["parse_log_line"]},{"to":"models.py","from":"triage.py","names":["ContainerFailure"]}],"nodes":[{"file":"main.py","kind":"entry","layer":0,"defines":[],"you_write":null},{"file":"tests/test_oncall.py","kind":"test","layer":0,"defines":["summary","test_finds_commit_by_message","test_newest_matching_commit_wins","test_groups_nonzero_exits","test_last_error_is_per_run","test_sorted_by_count_then_name"],"you_write":null},{"file":"oncall.py","kind":"read_only","layer":1,"defines":["read_lines","build_report","run"],"you_write":null},{"file":"exit_codes.py","kind":"read_only","layer":2,"defines":["explain_exit_code"],"you_write":null},{"file":"recovery.py","kind":"editable","layer":2,"defines":["find_lost_commit"],"you_write":"find_lost_commit"},{"file":"triage.py","kind":"editable","layer":2,"defines":["summarize_container_errors"],"you_write":"summarize_container_errors"},{"file":"composelog.py","kind":"read_only","layer":3,"defines":["parse_log_line"],"you_write":null},{"file":"reflog.py","kind":"read_only","layer":3,"defines":["parse_reflog_line"],"you_write":null},{"file":"models.py","kind":"read_only","layer":4,"defines":["ReflogEntry","LogEvent","ContainerFailure"],"you_write":null}]},"tests":{"hidden_count":7,"visible_names":["test_finds_commit_by_message","test_newest_matching_commit_wins","test_groups_nonzero_exits","test_last_error_is_per_run","test_sorted_by_count_then_name"],"cases_per_check":2,"check_timeout_seconds":15,"timeout_seconds_per_test":5},"title":"Recover the lost hotfix and triage the crash-looping deploy","course":"software-engineer-to-ai-engineer","ticket":{"id":"OPS-318","body":"$99","type":"feature","title":"On-call report: find the commits the reset wiped and which containers crashed","priority":"P1","reporter":"platform (on-call)","acceptance":[{"text":"Finds the commit by ticket id or message words, ignoring case, and skips checkout/merge/reset lines that only mention it","tests":["test_finds_commit_by_message"]},{"text":"When a commit was amended, returns the newest version","tests":["test_newest_matching_commit_wins"]},{"text":"One entry per (container, exit code) with a count; clean exits (code 0) are left out","tests":["test_groups_nonzero_exits"]},{"text":"last_error comes from the run that ended in that exit, not from an earlier crash or another container","tests":["test_last_error_is_per_run"]},{"text":"Sorted by count (highest first), then container, then exit code","tests":["test_sorted_by_count_then_name"]}]},"scoring":{"max":100,"hint_cost":5,"time_bonus":{"points":10,"full_under_minutes":20,"zero_after_minutes":60},"hidden_tests":50,"visible_tests":40},"editable":[{"file":"recovery.py","function":"find_lost_commit"},{"file":"triage.py","function":"summarize_container_errors"}],"language":"python","entrypoint":"main.py","hint_count":4,"open_files":["recovery.py","triage.py","oncall.py"],"after_section":"1.5","python_version":"3.12","editable_regions":[{"file":"recovery.py","end_line":19,"function":"find_lost_commit","start_line":18},{"file":"triage.py","end_line":21,"function":"summarize_container_errors","start_line":20}]},"starter":{"main.py":"from oncall import run\n\nif __name__ == \"__main__\":\n    run(\"data/reflog_release.txt\", \"data/compose_up.log\", [\"OPS-311\", \"worker memory\"])\n","models.py":"$9a","oncall.py":"$9b","reflog.py":"from models import ReflogEntry\n\nHEX = set(\"0123456789abcdef\")\n\n\ndef parse_reflog_line(line: str) -> ReflogEntry | None:\n    \"\"\"\n    Parse one line of `git reflog` output:\n        d41f0a2 HEAD@{4}: commit (amend): OPS-311: retry DB connect on startup\n    Subjects can contain \": \" themselves, so only the first two separators split.\n    Returns None for blank lines and anything git didn't print as an entry.\n    \"\"\"\n    sha, _, rest = line.strip().partition(\" \")\n    if not 4 <= len(sha) <= 40 or not set(sha) <= HEX:\n        return None\n    parts = rest.split(\": \", 2)\n    if len(parts) != 3 or \"@{\" not in parts[0]:\n        return None\n    selector, action, subject = parts\n    return ReflogEntry(sha, selector, action.strip(), subject.strip())\n","triage.py":"from composelog import parse_log_line\nfrom models import ContainerFailure\n\n\ndef summarize_container_errors(log_lines: list[str]) -> list[ContainerFailure]:\n    \"\"\"\n    Summarise the failed container exits in `docker compose up` output.\n\n    - Skip lines parse_log_line returns None for.\n    - Exit code 0 is a clean stop, not a failure: leave it out.\n    - One ContainerFailure per (container, exit_code); count is how many\n      times that container exited with that code.\n    - last_error is the message of the last error line (event.is_error)\n      the container logged in the run that ended with its most recent\n      exit with that code. A run starts after the container's previous\n      exit (any code, 0 included), so errors from an earlier run don't\n      carry over. Use \"\" if that run logged no error.\n    - Sort by count, highest first, then container name, then exit code.\n    \"\"\"\n    # TODO: implement\n    return []\n","recovery.py":"from reflog import parse_reflog_line\n\n\ndef find_lost_commit(reflog_lines: list[str], message: str) -> str | None:\n    \"\"\"\n    Find the commit to recover after a bad reset, given `git reflog`\n    output (newest entry first, exactly as git prints it).\n\n    Match entries whose subject contains `message`, ignoring case and\n    the message's surrounding whitespace. Only entries that recorded a\n    new commit count (entry.makes_commit): a checkout, merge or reset\n    line that mentions the same words is not the commit.\n    If several commits match (e.g. a commit and its amend), return the\n    newest one. Skip lines parse_reflog_line can't read.\n    Return the short SHA, or None if nothing matches.\n    Raise ValueError if message is blank.\n    \"\"\"\n    # TODO: implement\n    return None\n","composelog.py":"$9c","exit_codes.py":"\"\"\"What a container's exit code usually means. Above 128 it is 128 + the signal that killed it.\"\"\"\n\nMEANINGS = {\n    0: \"clean exit\",\n    1: \"application error; read the last error line\",\n    2: \"bad arguments or shell misuse\",\n    125: \"docker could not start the container\",\n    126: \"command found but not executable\",\n    127: \"command not found; check CMD / ENTRYPOINT\",\n    137: \"SIGKILL (128+9): usually the OOM killer; check OOMKilled in docker inspect\",\n    139: \"SIGSEGV (128+11): segmentation fault\",\n    143: \"SIGTERM (128+15): asked to stop, e.g. docker stop or a redeploy\",\n}\n\n\ndef explain_exit_code(code: int) -> str:\n    if code in MEANINGS:\n        return MEANINGS[code]\n    if code > 128:\n        return f\"killed by signal {code - 128}\"\n    return \"application-defined error\"\n","data/compose_up.log":"$9d","tests/test_oncall.py":"$9e","data/reflog_release.txt":"8a1f2b3 HEAD@{0}: reset: moving to origin/release/2.3\ne93b7c5 HEAD@{1}: commit: chore: raise worker memory limit to 512m\nd41f0a2 HEAD@{2}: merge hotfix/OPS-311-db-retry: Fast-forward\n8a1f2b3 HEAD@{3}: checkout: moving from hotfix/OPS-311-db-retry to release/2.3\nd41f0a2 HEAD@{4}: commit (amend): OPS-311: retry DB connect on startup, with backoff\nb7c41e9 HEAD@{5}: commit: OPS-311: retry DB connect on startup\n8a1f2b3 HEAD@{6}: checkout: moving from release/2.3 to hotfix/OPS-311-db-retry\n8a1f2b3 HEAD@{7}: commit: OPS-298: add /health endpoint for the load balancer\n3c4d5e6 HEAD@{8}: checkout: moving from main to release/2.3\n"},"course":{"slug":"software-engineer-to-ai-engineer","title":"AI Engineer","prefix":"Software Engineer →"},"chapter":{"num":"01","title":"The AI-Native Engineer's Toolkit: Git, Linux, Docker and AI Coding Tools"}}}]
93:null
97:[["$","title","0",{"children":"Lab OPS-318 · Recover the lost hotfix and triage the crash-looping deploy · TheNeuralNetwork"}],["$","meta","1",{"name":"description","content":"Job-ready tech courses crafted from professional JDs, years of experience and wisdom from people at the cutting edge of technology."}],["$","link","2",{"rel":"icon","href":"/p/icon.svg?icon.3dus6kvmx1art.svg","sizes":"any","type":"image/svg+xml"}],["$","$L9f","3",{}]]