0% 5h quota used · resets in 2h live · 08/09/2026 12:12 CEST

de/wasserversorger-enrich

Enrich the next batch of German water suppliers (Wasserversorger) — contact details plus the published water-analysis URL — and append results to the campaign JSONL.

de:wasserversorger-enrich 245 lines

Wasserversorger enrichment

You are in the country workspace. Campaign files: campaigns/wasserversorger/. Plan and measured pilot results: _tasks/netzbetreiber-contact/german-water/PLAN.md.

This is the water counterpart of netzbetreiber-enrich. Same guards, same append-only discipline, two differences that matter: the data is mostly published (so outreach is a fallback, not the goal), and company identity is ours, not the register's.

Scope — this skill is Step 2, not Step 0

It enriches suppliers that are already identified. Finding out who exists is the separate discovery sweep: coverage.mjs ranks the unclaimed municipalities by population into residual-top.csv, a sweep resolves them into roster.part-*.jsonl + roster-links.part-*.jsonl, and only then do those companies become enrichment input. Do not start here on an empty roster.

Batch selection

  1. Read the roster (roster.part-*.jsonl, and seed.csv once it exists) plus campaigns/wasserversorger/enrichment.jsonl. A row is pending when its company_id has no JSONL record.
  2. Take the next 45 pending rows, unless requester instructions say otherwise (a different count, or specific company_ids to re-research). Water enrichment measured a median of 5 fetches per company against 1-2 for netzbetreiber, so 30 is the working batch — raise it only on measured evidence from the ledger, never on a guess.
  3. Order the batch by priority, not by file order: a. pending_domain rows first. A company with no official_domain cannot be deduped and cannot be imported at all — domain is the dedup key. 107 of the roster are in this state as of 2026-08-27 and discovery keeps producing more, because it finds delegated suppliers by name off a Verbandsgemeinde page without resolving their own site. Clearing these unblocks the load and surfaces domain collisions while they are still few. b. Then by population served (sum the bevoelkerung of the municipalities linked to that company_id in roster-links.jsonl — grep it, it is 3 MB+, never read it whole). The consumer is energiemarie's city→supplier finder, queried in proportion to population. c. Then the rest.
  4. Append-only. A re-research appends a new record; exports take the latest per company_id.

Identity — read before recording anything

company_id (wv-de-NNN) is our internal primary key. official_domain is the dedup/merge key (decided by Stefanie, 2026-08-12). Do not key records on handelsregister_nr, water_delegate_id, or any cities-api id. Mapping our companies onto cities-api rows is a separate late-binding pass run at import time.

Why not Handelsregister (measured 2026-08-12, pilot 3): only 16% of suppliers have a number at all — Zweckverbände are Körperschaften des öffentlichen Rechts and are not in the register.

That 16% is a Zweckverband rate, not a national one. The population-weighted roster sweep (2026-08-12, 19 big-city suppliers) found 68% with a number printed in the Impressum: the head of the country is GmbHs and AGs, which are registered. Handelsregister coverage therefore correlates with size, which is exactly what makes it useless as a key — it is present where dedup is easy and absent where dedup is hard. Domain does not have that skew.

Legal forms that justify a null handelsregister_nr are KdöR and AöR — Berliner Wasserbetriebe is an Anstalt des öffentlichen Rechts, not a Körperschaft. Record whichever the Impressum states. Worse, where a number exists it may belong to an operating subsidiary rather than the public-law body holding the supply relationship (Aggerverband has no entry; AV Aggerwasser GmbH is HRB 39328). Matching on it attaches the wrong company, undetectably.

Domain has full coverage instead (website_url was 100% in pilot 3) and the field already exists in Stefanie's analysis contract as official_domain.

Normalise before comparing: lowercase, strip scheme and www., reduce to the registrable domain (eTLD+1). https://WWW.Beispiel.de/wasser/ → beispiel.de.

Two failure modes to guard, not ignore:

  1. One domain, several suppliers. Small Eigenbetriebe live on the municipality's domain, and German Samtgemeinden / Verwaltungsgemeinschaften often share one website across several municipalities — so two genuinely distinct suppliers can present the same domain. Parent / operating-subsidiary pairs do the same. Never auto-merge on a domain collision. Flag any official_domain claimed by more than one company_id into a review queue and record both rows; a wrong merge here is the same undetectable error we rejected Handelsregister for.
  2. One supplier, several domains. Rebrands, .de/.com pairs, and old domains that redirect. Record the domain you verified via the Impressum and put observed alternates in notes, so a later rebrand appends an alias rather than creating a duplicate company.

Priority input: pending_domain rows

Discovery records a delegated supplier it could name but could not locate — official_domain: null with domain_status: "pending_domain" — because the VG page that names the supplier usually does not link its website. Those rows block the load: domain is the dedup key, so a row without one cannot be deduped or imported.

Take them before the normal batch. The company name is already verbatim in the row's notes along with the page that named it; the job is to find the official site, verify the Impressum names that entity, and set official_domain + domain_status: "resolved". If it genuinely has no website — small Zweckverbände sometimes publish only through their member municipalities — say so in reason and leave the status as is rather than attaching a member's domain, which would key the supplier to the wrong organisation.

Research each supplier

Spawn at most 2 parallel subagents (each is a separate multi-GB claude process; the container has a 6 GB memory cap — 4+ concurrent agents OOM the run). Work supplier by supplier.

Part 1 — contact and detail fields

Target fields: name, short_name, address, phone, address_opening_hours, phone_opening_hours, email, contact_form_url, website_url, emergency_phone_number, supply_area_municipalities.

  1. Official website. Verify the Impressum names this supplier. For a Zweckverband the Impressum usually also states the legal form (Körperschaft des öffentlichen Rechts) — record that verbatim when present, it is what justifies a null handelsregister_nr.
  2. Contact path. Kontakt / Kontaktformular. Record contact_form_url only if an actual form is present, not merely an address block.
  3. Emergency number. Look for Notdienst, Störung, Entstörung, Havarie, Bereitschaft — often a separate page, not the Impressum.
  4. Supply area. The member/Versorgungsgebiet list. This feeds the pivot; see Part 3.

supply_area_plz is not collected. Nobody publishes it (0/25 in pilot 3) and it is derivable: take the member municipalities and read plz from _tasks/netzbetreiber-contact/german-water/gemeinden_deutschland.csv. Compute it downstream; never scrape it, never email anyone for it.

Part 2 — the analysis locator

Port Stefanie's validated prompt contract verbatim — do not redesign it. Field list and her worked example (Stadtwerke Kempen) are in _tasks/netzbetreiber-contact/german-water/basecamp-todo-10101567321-find-prompt-analysis-url.md: official_domain, analysis_url, index_url, url_type, tier, year, multi_zone, zone_urls, zone_url_pattern, access_method, evidence_value, url_looks_dated, third_party_fallback, confidence, notes.

Published analyses are obligatory under TrinkwV §45/§46, so expect to find one — pilot 3 got 21/25, which is 21 of the 21 entities that actually supply drinking water; the 4 misses were all roster defects (flood-protection Verbände). multi_zone ran at 15/21 = 71% of real suppliers (60% of all 25 rows), so treat multiple supply zones as the normal case.

Useful side effect: "no analysis page anywhere + no Trinkwasser content" is a cheap automated filter for a non-supplier that has crept onto the roster.

Part 3 — municipality links

One record per published municipality↔company link in links.jsonl, carrying the verbatim published name. Normalisation and AGS resolution happen only in map_ags.mjs — never in the crawl. The crawl records what the supplier published; every normalisation decision stays in one reviewable file so the match rate is reproducible from raw evidence.

Classify each member with member_kind: gemeinde, ortsteil, landkreis, utility, unknown. Landkreise and utilities are legitimate members but have no AGS — they must not count against the match rate.

Set quartier when the supplier names a sub-area (Ortsteil / Versorgungszone) rather than the whole municipality, and type to aep for drinking water.

Hard rules

Country rules in ../../CLAUDE.md apply in full (never invent contact data; research only, never contact anyone; record a source per field group). Plus three the pilots proved:

  1. Every field carries its own source_url and confidence. Not one per record.
  2. A negative must be earned. Before recording not_published, follow the site's own labelled links. Pilot 3's single false negative was an emergency_phone_number marked not_published where the homepage linked "Notdienst" three times and the page was never opened. Use status: found / not_published (verified absent) / not_found (searched, inconclusive) / not_applicable (cannot exist for this legal form).
  3. Label inference as inference. A handelsregister_nr of not_applicable justified by an Impressum statement is a scraped fact; the same conclusion drawn from the legal form in the supplier's name is an inference. Say which in reason. 14 of 21 in pilot 3 were the latter — legally sound, but not evidence.

Memory safety

When searching saved HTML use fixed-string searches (grep -F -c 'g-recaptcha') or read the file directly — never wide-context regexes like .{0,150}pattern.{0,150} or unbounded .*pattern.* with -o. On minified single-line HTML these allocate gigabytes and have OOM-killed the host (2026-07-22).

Fetch politely: no parallel hammering of a single host. Known host traps, all measured:

  • bodensee-wasserversorgung.de serves an incomplete TLS chain — curl and WebFetch both fail cert verification; it needs a CA workaround or it reads as unreachable rather than absent.
  • 78.46.40.200 has hard-banned this IP after ~7 requests, and the ban is durable (still refusing an hour later, not a 429). Two campaign companies share that one host — reckenberg-gruppe.de and fernwasser-franken.de — so one ban took out two suppliers.
  • l.de (Leipziger Gruppe) returns 403 to both curl and WebFetch. Bot protection, not absence. Record it as unresolved; do not conclude the data is unpublished.
  • Impressum paths are not guessable. /impressum 404s on Stadtwerke Düsseldorf (/rechtliches/impressum/), Stadtwerke Karlsruhe (/de/impressum/) and N-ERGIE (a long portal URL). Fetch the homepage and read the footer link rather than guessing paths.
  • Normalise Unicode before matching. One site's heading Öffungszeiten (their typo) did not match a grep -F for the same string because the Ö was not the precomposed codepoint.
  • PDFs may be pure scans with no text layer — pdftotext returned 6 bytes for a supplier whose 26 zone reports are all Konica-Minolta scans. Render and read, or OCR; a pdftotext-only pipeline scores that entire supplier as "publishes nothing".

Record

Append one JSON object per supplier (one line) to enrichment.jsonl. Each field is an object:

{"company_id": "wv-de-001", "name": "<from seed>",
 "fields": {
   "email": {"value": "info@…", "status": "found",
             "source_url": "https://…/impressum/", "confidence": "high", "reason": null},
   "handelsregister_nr": {"value": null, "status": "not_applicable", "source_url": "https://…",
             "confidence": "high", "reason": "Impressum states Körperschaft des öffentlichen Rechts"}
 },
 "analysis": { …Stefanie's 15 keys… },
 "fetches": 5, "searches": 1, "checked_at": "<today YYYY-MM-DD>", "notes": "… or null"}

Resumability

A fresh session must be able to continue mid-campaign without re-fetching completed work:

  1. Pending detection is company_id absent from enrichment.jsonl — nothing else.
  2. Append after each supplier, never batch at the end. An interrupted run keeps its completed work.
  3. Write raw crawl shards per subagent (enrichment.part-*.jsonl) and keep them as the evidence trail; map_ags.mjs merges.
  4. Do all work synchronously inside the run. Never spawn background agents and end your turn "waiting" — the run terminates the moment you produce your final message, killing anything still running (learned on netzbetreiber, 2026-08-04).

Finish

  1. Run node campaigns/wasserversorger/map_roster.mjs (merges the crawl shards into roster-links.jsonl and resolves each raw_name to an AGS), then node …/coverage.mjs (population coverage + the residual queue). Do not run map_ags.mjs / export_csv.mjs — those are pilot-3-era scripts that read the dead companies.part-a/b.jsonl → companies.jsonl/links.jsonl/companies.csv pair (25 rows, frozen since 2026-08-12), not the live enrichment.jsonl/roster-links.jsonl data. Running them is harmless (they just regenerate the same stale 25-row files) but wastes a step — multiple runs since 2026-09-03 have done this out of habit before catching it. There is currently no CSV export from enrichment.jsonl for the team's Sheet; that script does not exist yet. Persisted campaign scripts are .mjs so any session can run them. python3 is available at /usr/bin/python3 and is fine for throwaway analysis, but do not add Python to the campaign's committed toolchain. Rerun build_ags_map.mjs whenever normName changes — ags-city-map.csv caches name_norm, and a stale cache silently depresses the exact-match rate.
  2. Journal entry (see country CLAUDE.md).
  3. Final message: batch size, per-field recovery counts, how many are complete vs needing outreach, municipalities newly claimed and population coverage (the metric that matters — not row count), plus anything odd worth a human's attention.