Day 10: Files, paths, and encodings

count = 0
with open("notes.txt", encoding="utf-8") as f:
for line in f:
if line.strip():
count += 1
print(count, f.closed)
Output: 3 True. Trace: the loop receives 'alpha\n', 'beta\n', 'gamma\n' (or 'gamma' when the file has no final newline); each is non-empty after strip(), so count reaches 3, and a trailing newline on the last line does not create a fourth line. print runs after the with block has exited, so f is already closed and f.closed is True.

Observed output of the exercise script (the same on every machine, because the fixture is private):

[project] exit=0
cwd is project: True
exists: True
lines: 3
[outside] exit=1
cwd is project: False
exists: False
FileNotFoundError: [Errno 2] No such file or directory: 'data/notes.txt'

Diagnosis: the file exists (exists: True when the cwd is the project), but the second run has a different cwd (cwd is project: False), so the relative path resolves to outside/data/notes.txt, which does not exist. The cause is the working directory, not an absent file. Minimal fix: anchor the path to the script with BASE = Path(__file__).resolve().parent and open BASE / "data" / "notes.txt"; the counting line stays unchanged. The complete reference below writes the anchored reader into a fresh fixture, runs it from both folders, then deletes the data file to show the other cause: when the anchored path itself does not exist, no code change helps, and restoring the fixture does.

import subprocess
import sys
import tempfile
from pathlib import Path
FIXED = (
    "from pathlib import Path\n"
    "BASE = Path(__file__).resolve().parent\n"
    "with open(BASE / 'data' / 'notes.txt', encoding='utf-8') as f:\n"
    "    print('lines:', sum(1 for line in f if line.strip()))\n"
)

def run(script, cwd):
    done = subprocess.run([sys.executable, str(script)],
                          cwd=cwd, capture_output=True, text=True)
    if done.returncode == 0:
        return f"exit=0 {done.stdout.strip()}"
    last = done.stderr.strip().splitlines()[-1]
    return f"exit={done.returncode} {last.split(':')[0]}"

with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp).resolve()
    project = root / "project"
    outside = root / "outside"
    notes = project / "data" / "notes.txt"
    notes.parent.mkdir(parents=True)
    outside.mkdir()
    notes.write_text("alpha\nbeta\ngamma\n", encoding="utf-8")
    reader = project / "reader.py"
    reader.write_text(FIXED, encoding="utf-8")
    print("anchored, from outside:", run(reader, outside))
    print("anchored, from project:", run(reader, project))
    notes.unlink()
    print("fixture deleted, anchored path exists:", notes.exists())
    print("anchored, from project:", run(reader, project))
    notes.write_text("alpha\nbeta\ngamma\n", encoding="utf-8")
    print("fixture restored:", run(reader, outside))

Expected output:

anchored, from outside: exit=0 lines: 3
anchored, from project: exit=0 lines: 3
fixture deleted, anchored path exists: False
anchored, from project: exit=1 FileNotFoundError
fixture restored: exit=0 lines: 3

Prediction and observed output:

{'ana': 7, 'cy': 5} [2]

Line 2 (bo,x9) raises ValueError inside the try, so its number goes into skipped and continue skips the assignment; lines 1 and 3 reach scores[name] = score. The try wraps only int(raw), so a failing open() or a bug in the split still surfaces as a traceback instead of being silently counted as a bad row. enumerate(f, 1) numbers physical lines, so a blank line still advances the count. A decoding problem is a different failure class; this program contrasts it with ValueError:

data = "café".encode("latin-1")
print(data)
try:
    data.decode("utf-8")
except UnicodeDecodeError as exc:
    print(type(exc).__name__ + ":", exc)
print(data.decode("latin-1") == "café")
print(b"plain ascii".decode("utf-8") == b"plain ascii".decode("latin-1"))
both = b"caf\xc3\xa9"
as_utf8 = both.decode("utf-8")
as_latin1 = both.decode("latin-1")
print(as_utf8 == "café", as_latin1 == "café", len(as_utf8), len(as_latin1))

Expected output:

b'caf\xe9'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data
True
True
True True 4 5

b'caf\xe9' is café in Latin-1 and is not valid UTF-8, so decoding it as UTF-8 raises UnicodeDecodeError; the fix is to declare the real encoding (here encoding="latin-1"). Pure ASCII decodes the same under both encodings, and b'caf\xc3\xa9' decodes successfully under both but to different text (café, 4 characters, versus café, 5 characters), so a decode that succeeds is not proof that the declared encoding is right. ValueError, by contrast, means the text decoded fine but one row's value is invalid.

Reference implementation with a self-contained fixture harness (compare it with your own version only after your unaided attempt is recorded):

import tempfile
from pathlib import Path

def summarize_scores(path):
    result = {"scores": {}, "skipped": [], "missing": False}
    try:
        f = open(path, encoding="utf-8")
    except FileNotFoundError:
        result["missing"] = True
        return result
    with f:
        for lineno, line in enumerate(f, 1):
            line = line.strip()
            if not line:
                continue
            name, sep, raw = line.rpartition(",")
            try:
                score = int(raw)
            except ValueError:
                score = None
            if not sep or not name or score is None:
                result["skipped"].append(lineno)
                continue
            best = result["scores"].get(name)
            if best is None or score > best:
                result["scores"][name] = score
    return result

with tempfile.TemporaryDirectory() as tmp:
    base = Path(tmp)
    (base / "s1.txt").write_text("ana,7\nbo,9\nana,8\n", encoding="utf-8")
    (base / "s2.txt").write_text("ana,7\nbo,x9\ncy,5\n", encoding="utf-8")
    print(summarize_scores(base / "s1.txt"))
    print(summarize_scores(base / "s2.txt"))
    print(summarize_scores(base / "missing.txt"))

Expected output:

{'scores': {'ana': 8, 'bo': 9}, 'skipped': [], 'missing': False}
{'scores': {'ana': 7, 'cy': 5}, 'skipped': [2], 'missing': False}
{'scores': {}, 'skipped': [], 'missing': True}

Why it is shaped this way: open() is the only call inside the FileNotFoundError handler, so only a missing file sets missing to True and other errors still surface. The with f: block closes the file on every path. int(raw) is the only call inside the ValueError handler; a row is skipped when it has no comma, an empty name, or a non-integer score, and its 1-based line number is recorded. The best score per name is kept by comparing with the stored value, so ana ends at 8.