Israel SchoolsTechnical Reference

This page covers the technical detail behind Israel Schools as implemented in the current codebase: the data sources and the join key that links them, the ETL pipeline, the layered data model, the mvp-v2 scoring model, peer grouping, the multilingual system, the API surface, the open-versus-geo-restricted source architecture, and deployment. The build runs with no external runtime dependencies. For the product thinking and design rationale, start with the narrative.

System Overview

The platform has three parts: an ETL pipeline that pulls public data into a local database, a stdlib-only HTTP server exposing a JSON API over that database, and a vanilla JavaScript map front end. There is no application framework, no ORM, and no Node toolchain. The database is a single SQLite file built by the pipeline; the server opens it read-only for everything except the right-of-reply endpoint.

Tech Stack

The dependency-light choice is deliberate. The product is a data problem, not a framework problem, and a stdlib-only stack keeps the running cost low and the operational surface small for a one-person public-good project.

Data Sources & Join Key

Every source is joined on the semel_mosad (institution symbol) for schools and on authority_code for municipalities. The institution symbol is stable across datasets, which is what makes the join possible at all.

Every observation row carries its source_dataset, source_resource_id and a JSON copy of the raw_record, so any value on a profile can be traced back to the dataset, year and original source row it came from. That traceability is both a data-quality tool and the substantive basis for the truth-in-publication defence described in the narrative.

ETL Pipeline

A single orchestrator runs every loader in dependency order and is safe to rerun. Each loader is small, isolated, and lands its source into the shared model. A failure in one loader is logged and skipped (unless run in strict mode) so a single flaky source does not abort the whole build.

The loaders carry the accumulated “tribal knowledge” of these sources: Hebrew quote-unescaping, an authority-name alias map for normalising inconsistent spellings, un-truncation of a fixed-width institution-type column, separation of Meitzav climate from achievement, Tnufa year parsing, and settlement-to-authority socioeconomic propagation. Each ETL run is recorded in an audit table with row counts in, row counts written, status and any error, which feeds the public “Sources” view and a maintenance dashboard.

Data Model

The schema is layered into reference dimensions, institutional identity, raw observations, and a regenerable derived layer.

Reference and identity

Observations and derived

The single generic school_indicators table is the load-bearing design choice: because every assessment fact is stored as a group, key, value and provenance rather than as a typed column, current Bagrut, current climate data, or any future indicator can be ingested as new rows without restructuring the schema or the scoring contract.

Scoring Model (mvp-v2)

The current model, mvp-v2, computes a per-dimension picture from the most recent available evidence. Academic prefers Tnufa 2022/23 and falls back to Meitzav percentile rankings within a peer group; wellbeing prefers the 2023/24 student-and-teacher survey, the freshest signal, then Tnufa, then Meitzav climate. The principle is that the freshest evidence wins per dimension. The overall result is a weighted average of the components actually present, renormalised by what is available.

The weights deliberately avoid test-score dominance. Bagrut is computed separately and shown on its own card with a staleness warning rather than blended into the headline band, because it is only relevant to high schools and the data is from 2013 to 2016. The overall is the weighted average of the components actually present, renormalised by what is available. In the current build roughly 3,600 schools are fully scored and the rest are returned as “Insufficient data”.

Model honesty: The model is a useful discovery score, not a fair contemporaneous ranking. The next pass enforces stage-aware grouping, age-weights stale indicators, and surfaces per-dimension provenance everywhere. The constraints are documented in the methodology rather than hidden behind a number.

Evidence Bands & Confidence

Numeric scores are never shown to users as precise values. Instead they map to five honest evidence bands, with anything below a confidence threshold withheld entirely.

A school with data_confidence='low' or fewer than two available components is published no overall band. Confidence and recency are linked: a recent survey adds confidence and lifts a school’s most-recent-data year to 2024, where it was previously capped at 2022. The API maps these display bands back to underlying model values for filtering, so a parent filtering for “Strong” gets the schools whose model output supports it, and the unrated remain explicitly unrated rather than silently ranked last.

Peer Groups & Baselines

Comparison only means something within a like-for-like group. Peer groups are built by a dedicated step (build_peer_groups.py) and keyed by education stage, sector, supervision and socioeconomic band. Education stage replaces raw institution type deliberately, so an elementary school is never ranked against a high school; null dimensions are stored as an unknown sentinel so grouping stays well-defined. The current build forms around 140 groups, with roughly 3,500 rated schools falling in a group of five or more.

The /api/peers/{semel} endpoint returns the school’s persistent peer group, its co-members for the map and list, and a benchmark per achievement indicator. The benchmark places the focal school’s latest value against three references at once: the peer-group average, its local-authority average, and the national average for the same grade and subject. The authority and national figures come from ready-made baselines (assessment_baselines, loaded by load_baselines.py) rather than being modelled on the fly, so a profile shows the school in peer, authority and national context with no extra computation in the request path.

Direction over time

Alongside the cross-sectional comparison, each profile carries a trajectory: improving, stable or declining. It is taken from the Tnufa peer-normalised trend category where current data exists, and otherwise computed from the historical Meitzav achievement panel (first year versus last year per subject). A school’s own direction is frequently more meaningful to a family than its rank against others.

Multilingual System

Source data is almost entirely in Hebrew, so presentation is separated from source values throughout. Two tables drive localisation, loaded once at server start and cached.

Translation lookups fall back gracefully (requested language, then English, then Hebrew, then the raw source value) so a missing string degrades to something readable rather than breaking. Proper nouns such as school and authority names are intentionally never translated. The four supported languages are en, he, ar and ru, with he and ar presented right-to-left. The server resolves the language per request from a query parameter and returns labels alongside raw values in every payload.

API Surface

A small, read-mostly JSON API. Every read endpoint accepts a lang parameter and returns translated labels next to raw values.

The schools query builds its WHERE clause dynamically from validated parameters, with grade-stage filters mapping to SQL ranges, radius search pre-filtered by a bounding box and refined with a haversine distance computed in a registered SQLite function. A Latin-transliteration normaliser lets searches like “LeZion” match the official “LEZIYYON” spelling.

Access & Catchment

Whether a child can actually attend a school is the most-requested and most-fragmented piece of information, so eligibility is treated as first-class, in two layers. A national rule layer derives each school’s admission basis from its type, supervision and stage, which in Israel is a structural feature of the system rather than a local quirk: state elementary and kindergarten by home-address zone, religious-stream by affiliation, state-secondary mixed, special-education by placement committee, gifted selective, boarding non-regional. Each rule row carries an honest amber “general rule, confirm your exact zone locally” flag and never fabricates a street boundary. A thin curated layer overrides the rule with municipality specifics where they have been hand-sourced into a dated seed file; Modi’in-Maccabim-Re’ut is the first such authority. Schools that are administrative entities rather than places a child attends are left in an honest unknown empty state. Adding another city is a data-curation task: drop a seed file and the loader picks it up, with no code change.

On top of eligibility, build_catchment.py estimates a catchment reach for each precisely-geolocated school, because real street-level zones are not published as open data anywhere and shift yearly. Reach is modelled as a labelled scalar (tight, typical or wide, as a national tercile within the school’s stage) from the median distance to the nearest same-stage, same-stream peers, with a confidence flag for isolated schools. When a home address is set, the API computes a per-address in-zone likelihood (likely, boundary or unlikely) from the exact home-to-school distance against that reach, but only for distance-gated schools; selective, stream and special-education schools get none, because distance does not gate entry there. It is surfaced as green, amber and red chips on cards and the map, always labelled as an estimate from the school pattern with a “confirm with the municipality before any property decision” warning, and is overridden by curated data where held.

Right of Reply & Privacy

Every profile exposes a structured “the data on this page is wrong” route into POST /api/corrections, which writes a correction_requests row pointing at the disputed datum. The endpoint validates input, caps the body size, confirms the school exists, and enforces a per-IP soft rate limit using a salted SHA-256 hash of the submitter IP, never the raw address. Contact details are stored separately and never returned by any read endpoint.

Privacy is enforced defensively. A minimum-sample-size threshold suppresses any survey indicator answered by too few students, so no value can identify an individual child. Published RAMA data is already pre-suppressed; the threshold is enforced again on any value that does carry a sample size. There is no third-party analytics, no fingerprinting, and no user-generated content surface. The server opens the database read-only for every endpoint except the corrections POST.

Geo-Restricted Sources

The architecture draws a hard line between sources reachable from anywhere and sources that need, or might need, an Israeli egress. The line is drawn conservatively: where a source’s access was only ever confirmed from an Israeli connection, it is treated as geo-restricted rather than assumed open.

Because both paths land into the same tables, the public-facing site serves every user identically regardless of which sources are reachable from the user’s own IP. The geo-restriction is an operational concern at refresh time, never a constraint on who the product can serve, which is what lets it work for diaspora and pre-aliyah families outside Israel.

Frontend

The front end is a single vanilla JavaScript application with no build step. A Leaflet map renders every school as a clustered marker coloured by evidence band, with schools placed only at a locality centroid drawn hollow and dashed and labelled approximate so they are never passed off as exact; OpenStreetMap supplies the tiles directly to the browser. The first-visit picker is a short stepped wizard, child (stage, stream, special-education needs) then location (address or dropped pin), after which results are ranked by distance and each card carries compact transit, eligibility and catchment-zone chips. Estimated catchment reach is drawn as a map overlay rather than a card pill. The profile, the up-to-five compare bar, the visit checklist and the sources view are all driven by the JSON API. The interface is built to the WCAG 2.0 AA standard required of Israeli public-facing sites: keyboard navigation through filters and results, ARIA labelling on markers and band badges, and a verified contrast ramp, with a published accessibility statement alongside the methodology and legal pages.

Running & Deployment

Building the database is a single command (python etl/run_all.py) that downloads from the public APIs and writes data/schools.db; starting the server is another (python app/server.py). There are no external dependencies, no framework and no npm, and a small pytest suite covers the server and ETL helpers. The intended deployment is a small VPS outside Israel for around a hundred dollars a month, with any geo-restricted refresh running from a separate Israeli egress on a schedule. Hosting and the corporate vehicle sit outside Israel deliberately, as part of the legal posture described in the narrative.

Ongoing maintenance is explicit rather than discovered after the first breakage: government-portal HTML drift, CKAN schema changes, geo-block changes, Hebrew encoding drift, translation upkeep, school-identity churn, per-municipality quirks, correction handling, freshness-band recomputes and tile-cost monitoring each have a defined cadence and trigger, surfaced through a single health dashboard and a monitored complaints mailbox.