Latest Results
fix: decode non-UTF-8 pages correctly (CAS-228) and make ys.crawl(limit=N) a real page cap (CAS-232) (#120)
fix(fetcher,crawl): decode non-UTF-8 pages correctly and make ys.crawl(limit=N) a real page cap
CAS-228 — non-UTF-8 pages became mojibake because only the HTTP charset header was
honored. Charset resolution now walks BOM -> HTTP header -> <meta charset> -> UTF-8 ->
byte-level detection (charset-normalizer), decoding each candidate strictly so a lying
header falls through instead of producing replacement characters. A lossy UTF-8 decode
is the last resort only. The same header-only bug in verifier.quick_test and
extractor.quick_extract is fixed through the shared helper.
CAS-232 — ys.crawl(limit=N) only set CrawlTarget.max_budget_pages, a field nothing
reads, so the crawl fetched the same page count for every limit. limit now lowers
CrawlBudget.max_pages, the field the frontier enforces, and lifts a preset's
max_pages_per_host to match so crawl.conservative's cap of 30 cannot silently answer
limit=40 with 30 pages. Politeness levers are untouched.
Fixtures and verification:
- tests/fixtures/encoding: 11 committed byte fixtures (shift_jis, windows-1251, koi8-r,
gb18030, euc-kr, latin-1, BOM, UTF-8) across every declaration pattern, plus a
regenerator script and a manifest.
- tests/fixtures/local_site.py: a real localhost HTTP server so the tests exercise the
transport rather than a mocked response object.
- tests/integration (marker: local_http, wired into CI): the corpus served over HTTP,
and a calendar/facet link-trap site where the limit is asserted against the server's
own request log.
- tests/smoke (YOSOI_LIVE_SMOKE=1): live non-UTF-8 pages (kakaku.com meta-only shift_jis,
lib.ru header-only koi8-r/windows-1251) and a live capped crawl of books.toscrape.com. fix(fetcher,crawl): decode non-UTF-8 pages correctly and make ys.crawl(limit=N) a real page cap
CAS-228 — non-UTF-8 pages became mojibake because only the HTTP charset header was
honored. Charset resolution now walks BOM -> HTTP header -> <meta charset> -> UTF-8 ->
byte-level detection (charset-normalizer), decoding each candidate strictly so a lying
header falls through instead of producing replacement characters. A lossy UTF-8 decode
is the last resort only. The same header-only bug in verifier.quick_test and
extractor.quick_extract is fixed through the shared helper.
CAS-232 — ys.crawl(limit=N) only set CrawlTarget.max_budget_pages, a field nothing
reads, so the crawl fetched the same page count for every limit. limit now lowers
CrawlBudget.max_pages, the field the frontier enforces, and lifts a preset's
max_pages_per_host to match so crawl.conservative's cap of 30 cannot silently answer
limit=40 with 30 pages. Politeness levers are untouched.
Fixtures and verification:
- tests/fixtures/encoding: 11 committed byte fixtures (shift_jis, windows-1251, koi8-r,
gb18030, euc-kr, latin-1, BOM, UTF-8) across every declaration pattern, plus a
regenerator script and a manifest.
- tests/fixtures/local_site.py: a real localhost HTTP server so the tests exercise the
transport rather than a mocked response object.
- tests/integration (marker: local_http, wired into CI): the corpus served over HTTP,
and a calendar/facet link-trap site where the limit is asserted against the server's
own request log.
- tests/smoke (YOSOI_LIVE_SMOKE=1): live non-UTF-8 pages (kakaku.com meta-only shift_jis,
lib.ru header-only koi8-r/windows-1251) and a live capped crawl of books.toscrape.com.andberg9/cas-228-cas-232-encoding-and-crawl-limit fix(fetcher,crawl): decode non-UTF-8 pages correctly and make ys.crawl(limit=N) a real page cap
CAS-228 — non-UTF-8 pages became mojibake because only the HTTP charset header was
honored. Charset resolution now walks BOM -> HTTP header -> <meta charset> -> UTF-8 ->
byte-level detection (charset-normalizer), decoding each candidate strictly so a lying
header falls through instead of producing replacement characters. A lossy UTF-8 decode
is the last resort only. The same header-only bug in verifier.quick_test and
extractor.quick_extract is fixed through the shared helper.
CAS-232 — ys.crawl(limit=N) only set CrawlTarget.max_budget_pages, a field nothing
reads, so the crawl fetched the same page count for every limit. limit now lowers
CrawlBudget.max_pages, the field the frontier enforces, and lifts a preset's
max_pages_per_host to match so crawl.conservative's cap of 30 cannot silently answer
limit=40 with 30 pages. Politeness levers are untouched.
Fixtures and verification:
- tests/fixtures/encoding: 11 committed byte fixtures (shift_jis, windows-1251, koi8-r,
gb18030, euc-kr, latin-1, BOM, UTF-8) across every declaration pattern, plus a
regenerator script and a manifest.
- tests/fixtures/local_site.py: a real localhost HTTP server so the tests exercise the
transport rather than a mocked response object.
- tests/integration (marker: local_http, wired into CI): the corpus served over HTTP,
and a calendar/facet link-trap site where the limit is asserted against the server's
own request log.
- tests/smoke (YOSOI_LIVE_SMOKE=1): live non-UTF-8 pages (kakaku.com meta-only shift_jis,
lib.ru header-only koi8-r/windows-1251) and a live capped crawl of books.toscrape.com.andberg9/cas-228-cas-232-encoding-and-crawl-limit fix(fetcher,crawl): decode non-UTF-8 pages correctly and make ys.crawl(limit=N) a real page cap
CAS-228 — non-UTF-8 pages became mojibake because only the HTTP charset header was
honored. Charset resolution now walks BOM -> HTTP header -> <meta charset> -> UTF-8 ->
byte-level detection (charset-normalizer), decoding each candidate strictly so a lying
header falls through instead of producing replacement characters. A lossy UTF-8 decode
is the last resort only. The same header-only bug in verifier.quick_test and
extractor.quick_extract is fixed through the shared helper.
CAS-232 — ys.crawl(limit=N) only set CrawlTarget.max_budget_pages, a field nothing
reads, so the crawl fetched the same page count for every limit. limit now lowers
CrawlBudget.max_pages, the field the frontier enforces, and lifts a preset's
max_pages_per_host to match so crawl.conservative's cap of 30 cannot silently answer
limit=40 with 30 pages. Politeness levers are untouched.
Fixtures and verification:
- tests/fixtures/encoding: 11 committed byte fixtures (shift_jis, windows-1251, koi8-r,
gb18030, euc-kr, latin-1, BOM, UTF-8) across every declaration pattern, plus a
regenerator script and a manifest.
- tests/fixtures/local_site.py: a real localhost HTTP server so the tests exercise the
transport rather than a mocked response object.
- tests/integration (marker: local_http, wired into CI): the corpus served over HTTP,
and a calendar/facet link-trap site where the limit is asserted against the server's
own request log.
- tests/smoke (YOSOI_LIVE_SMOKE=1): live non-UTF-8 pages (kakaku.com meta-only shift_jis,
lib.ru header-only koi8-r/windows-1251) and a live capped crawl of books.toscrape.com.andberg9/cas-228-cas-232-encoding-and-crawl-limit fix(qa): restore the CAS-270 QA index implementation dropped from #117 (#118)
* feat(providers): make OpenCode default subscription discovery, gate Claude SDK opt-in (CAS-242)
* scaffold the observations qa base for today
* test(observations): scaffold the boss-fight evaluation harness
Adversarial evaluation workloads for the modality-specific observation
pruners, organized by failure shape rather than site popularity. Adds the
tests/boss_fights/ tree (one directory per modality plus controls,
cross_modal, episodes, generators and scoreboard), a session-scoped
boss_fights_root fixture, and boss_fights.md stating the principles and the
gate/selfhost/dogfood/live corpus lanes.
Registers a `boss_fight` marker, auto-applies it (plus `eval`) to anything
under tests/boss_fights/, and adds a `boss-fights` poe task. Live sites never
gate CI: public pages are frozen into pinned artifacts first.
* feat(observations): implement the static HTML pruning slice (CAS-262)
Port the spike's source_outline reducer into HtmlPruner as addressable
fragments, and complete ROADMAP delivery step 1: static HTML artifact ->
PrunedView -> flat index -> bounded inspection.
* test(observations): add 10,000-row repeat-collapse workload
* docs(observations): record the 10k-row gate; chore: lower coverage floor to 70
* feat(observations): anchored refs, stable ref ids, budgeted rendering, and multi-hop navigation (CAS-262)
- anchor addresses to durable ancestors; derive snapshot-independent ref_id
(anchored + stable + positional-free, else None)
- implement index/render.py: token-budgeted overview, headings before regions,
omission always stated
- add ObservationInspector.rebind: carry an exemplar-learned route onto a branch
the index never described
- gates: reference_stability (6 mutations), wikipedia_web_scraping (frozen pinned
revision), multi_hop (nested repeats, session budget)
- bump both HTML pruner versions to 2: emitted addresses changed
* feat(observations): add structured DOM pruning beta
* feat(observations): capture live TodoMVC DOM episode
* fix(observations): bind DOM artifacts and coverage
* test(observations): cover duplicate DOM member keys
* test(observations): restore rendered-DOM artifact schema coverage
These 6 tests were added alongside `models/dom.py` but were dropped when the
commit that introduced them was squashed away; the models survived, the tests
did not. They pass unchanged against the consolidated schema.
* spike(qa-index): restore the QA index spike (screencast, interrupt takeover, VoidCrawl blocker)
6.9k lines under spikes/qa_index/ that were stranded on a side head when the
observations work was squashed forward. Self-contained spike directory: it adds
no imports to the yosoi package and is outside pytest's testpaths.
* spike(gemma): add Gemma 4 multimodal inference harness (CAS-268)
Standalone gemma-spike harness: docker-compose deployment, inference client,
benchmark/smoke scripts, a static-HTML boss-fight runner, and captured QAT results.
Also gates Gemma discovery config behind an explicit opt-in.
* fix(outputs): make xlsx writer type-safe without changing behavior
pyrefly rejected `wb.active` (Worksheet | None) being subscripted/appended to,
and `record.get(col)` being passed a header cell value that is not `str`.
Guard the active worksheet explicitly and skip lookups for non-string header
cells. The header list stays positional -- index i is column i+1 -- so a gap in
the header row still maps to a blank data cell; a test pins that alignment.
* fix(gemma-spike): ask the endpoint which model it serves
The three scripts each hard-coded a default MODEL_ID. When the deployment moved
to the QAT checkpoint only static_html_boss_fight.py was updated, so smoke_test
and bench_inference asked for google/gemma-4-12b-it against a server offering
google/gemma-4-12b-it-qat-w4a16-ct and got a bare HTTP 404.
The worker serves exactly one model and /v1/models reports it, so resolve it at
runtime unless --model or MODEL_ID pins one. Verified live against Echo: vision
smoke, static-HTML structured output, and the microbench all run unpinned.
* fix(observations): write a real newline after the DOM capture manifest
The literal was '\\n', so each capture appended a backslash and an n to
capture_manifest.json instead of terminating the line.
* feat(observations): name which members a collapsed DOM region stands for
A DOM region reported "×3 li; states=visible×3" and no member text, so the
index said how many members a run had but nothing about which — a list of
groceries and a list of build failures read identically until you spent an
expand. The HTML pruner already samples distinguishing member text; this brings
the DOM pruner to parity via dom_subtree_text, which crosses shadow boundaries
because a member's visible content is what distinguishes it.
Bounded the same way: three samples plus an explicit withheld count. Measured on
a live TodoMVC capture the index grew 3,031 -> 3,131 bytes (3%) and the region
became: ×3 li "Ship beta", "Read design", "Buy milk".
Pruner version 1 -> 2: emitted summaries changed.
* test(observations): add opt-in live dogfood for the HTML and DOM pruners
The boss fights gate CI against frozen bytes, which is the only way to assert a
digest or an exact member count -- and it means nothing checks the pruners
against a page as served today. These do, and never gate: they assert
invariants that survive any capture (order-of-magnitude reduction, repeated runs
cost regions, members stay reachable, no index address is a dead pointer)
instead of counts that legitimately drift.
HTML runs over plain HTTP; DOM drives a real VoidCrawl browser through the same
TodoMVC states as the frozen episode. Neither writes into tests/, so the frozen
artifacts stay byte-exact. Both skip unless YOSOI_LIVE_SMOKE=1; verified passing
live, 8/8.
* fix(observations): name the DOM depth limit instead of leaking a JSON error
A rendered-DOM payload nesting deeper than 99 elements failed inside pydantic's
recursive-descent JSON validator, surfacing as 'Invalid JSON: recursion limit
exceeded'. That reads as a corrupt artifact and sends the reader looking for
malformed bytes that are not there — the real cause is a page that nests too far.
The ceiling was measured by bisection, not chosen, and is now stated as
MAX_PARSED_DEPTH, documented among the schema's deliberate limits, and pinned by
a test at the limit and one past it. Ordinary validation failures still report
as themselves.
Deeply wrapped applications need a subtree capture until the ceiling is lifted.
* fix(observations): disclose where the DOM walk stopped descending
_walk returned silently at MAX_DEPTH, so the deepest indexed entry reported its
child count and read exactly like a fully indexed node. A 41-element chain
emitted 26 entries and content planted at depth 30 was unreachable, with nothing
in the index indicating a subtree had been skipped -- the omission the reader
cannot see that render.py warns about.
The HTML pruner already says 'below index depth -- inspect to descend' at its
own ceiling; this brings the DOM pruner to parity. Reachable in practice, not
theoretical: measured live, GitHub nests 38 deep and Reddit 23 against a walk
ceiling of 24.
* chore(repo): fix .venv ignore, protect frozen artifacts, drop scratch file
- .gitignore matched only `.venv/` (a directory), so a `.venv` symlink got
committed by the gemma spike; match the bare name instead.
- end-of-file-fixer/trailing-whitespace were rewriting frozen boss-fight
artifacts, breaking the `artifact_sha256` digest assertions that pin them;
exclude the artifact directories.
- temp.py was a scratch contract doodle containing a `>>>` prompt line, so it
is not valid Python and failed check-ast and ruff.
* fix(observations): repair DOM rebind, unify region coverage, bound expand summaries
* feat(observations): stop the DOM index spending its budget on non-evidence
* feat(observations): collapse DOM records by shape, not by exact sameness
* feat(observations): page the index instead of truncating it at 1,000
* feat(observations): describe an oversized page at coarser depth instead of only paging it
* feat(observations): fold pass-through wrapper chains into one entry
* feat(observations): compare two indexes over identity, not position
* feat(observations): accessibility-tree modality on the shared kernel (CAS-263)
* chore(observations): integrate the AX and network modalities
* fix(observations): harden four pruners and indexed evidence routing
* feat(qa): expose read-only indexed evidence surface
* fix(qa): enforce evidence budgets and clarify public surfaces
* test(qa): prove all-four composition and bounded expansion progress
* fix(qa): make capabilities and MCP schemas truthful
* fix(docs): resolve source links from jj workspaces
* feat(actions): add retained transition ledger and QA seam (CAS-270)
* fix(build): sort console-script entries so pyproject-fmt passes Latest Branches
0%
andberg9/cas-228-cas-232-encoding-and-crawl-limit 0%
andberg9/release-0-0-3a27 0%
andberg9/restore-cas-270-qa-index © 2026 CodSpeed Technology