38 Commits

Author SHA1 Message Date
lilleman 976d2e5ac1 Rank understanding the code second, and say why it is not an afterthought
Tests / vet + fmt + tests (pull_request) Successful in 1m25s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 7s
Tests / Gitea release from CHANGELOG.md (push) Successful in 4s
2026-09-20 23:19:53 +02:00
lilleman 343c9fa98a Make being the best fake-data tool, and known for it, the goal the rest serve
Tests / vet + fmt + tests (pull_request) Successful in 1m23s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 23:01:30 +02:00
lilleman f206a18027 Gate everything on the comprehension score, and replay the round until it passes
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 22:53:14 +02:00
lilleman 3dc1cf82f1 Make the cost of reading the code a goal, and answer to a reader panel each PR
Tests / vet + fmt + tests (pull_request) Successful in 1m26s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 22:45:22 +02:00
lilleman 0d627d2b11 Plan the fence layer's readability first, as eight chunks in order
Tests / vet + fmt + tests (pull_request) Successful in 5s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 5s
Tests / Gitea release from CHANGELOG.md (push) Successful in 5s
2026-09-20 22:11:27 +02:00
lilleman ce8ea550a3 Correct what the readers proved false, and lift the file map out of last place
Tests / vet + fmt + tests (pull_request) Successful in 1m26s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 7s
Tests / Gitea release from CHANGELOG.md (push) Successful in 5s
2026-09-20 21:47:17 +02:00
lilleman 1035f8bb2a Name the register read before claiming none publishes a spelling
Tests / vet + fmt + tests (pull_request) Successful in 6s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 5s
Tests / Gitea release from CHANGELOG.md (push) Successful in 4s
2026-09-20 21:41:00 +02:00
lilleman ce59653ccd Say what selects, name the decision the keyword instances, and cite POSIX's macros
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 7s
Tests / Gitea release from CHANGELOG.md (push) Successful in 4s
2026-09-20 21:29:55 +02:00
lilleman ea9c1ad436 Name only the checker the keyword answers to, and file the Error near miss
Tests / vet + fmt + tests (pull_request) Successful in 1m26s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 21:26:54 +02:00
lilleman e09e92ff03 Re-pin the shipped shape for misc.loglevel's keyword
Tests / vet + fmt + tests (pull_request) Successful in 1m23s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 21:21:10 +02:00
lilleman 81cf04ae8b Render the syslog keyword POSIX publishes, keeping RFC 5424's severity beside it 2026-09-20 21:20:44 +02:00
lilleman 5715e516aa Expect the syslog keyword misc.loglevel renders, beside the RFC severity 2026-09-20 21:19:11 +02:00
lilleman 621e6f4548 Drop the loglevel decision's false goal-10 claim
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 20:52:07 +02:00
lilleman 839b7bd164 Cut the importer's rationale, pin the name selection the docs show, and spell the PRI alike
Tests / vet + fmt + tests (pull_request) Successful in 1m30s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 20:47:19 +02:00
lilleman d081ec85d0 Re-pin the shipped shape for misc.loglevel
Tests / vet + fmt + tests (pull_request) Successful in 1m28s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 20:39:52 +02:00
lilleman 474f73444b Read misc.loglevel from RFC 5424's severity table 2026-09-20 20:39:21 +02:00
lilleman edce645f34 Expect RFC 5424's severities as a table, selectable by code or name 2026-09-20 20:37:24 +02:00
lilleman b581daf0ab Record why misc.tld keys carry the dot, and file the miss it costs
Tests / vet + fmt + tests (pull_request) Successful in 1m28s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 6s
Tests / Gitea release from CHANGELOG.md (push) Successful in 4s
2026-09-20 18:20:16 +02:00
lilleman 815a24c186 Prove every tld row's type and displayed spelling, which 100 draws of 1438 cannot
Tests / vet + fmt + tests (pull_request) Successful in 1m22s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 18:18:45 +02:00
lilleman 878cab95e7 Re-pin the shipped shape for misc.tld
Tests / vet + fmt + tests (pull_request) Successful in 1m25s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 18:12:44 +02:00
lilleman e7a81459cd Read misc.tld from the IANA Root Zone Database, and file the .eh no root zone holds 2026-09-20 18:12:16 +02:00
lilleman 15af49e618 Expect the IANA root zone as a table, selectable by either label 2026-09-20 18:06:01 +02:00
lilleman b3be1ee137 Record why a register with no frequency draws evenly, and the rule three misses shared
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
Tests / vet + fmt + tests (push) Successful in 6s
Tests / Gitea release from CHANGELOG.md (push) Successful in 4s
2026-09-20 00:48:50 +02:00
lilleman 5083b79730 Credit RFC 5789 for PATCH, which RFC 9110 does not define
Tests / vet + fmt + tests (pull_request) Successful in 1m21s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:45:23 +02:00
lilleman b3ead0ca5e Check every method's booleans on every seed, where the old guard checked none 2026-09-20 00:45:23 +02:00
lilleman 78d7e513f9 Give the datatype item the reason that holds, and rewrap a line
Tests / vet + fmt + tests (pull_request) Successful in 1m26s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:38:20 +02:00
lilleman 4472c8d92b Stop naming what the register's other entries are, and say where the booleans stay text
Tests / vet + fmt + tests (pull_request) Successful in 1m26s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:36:38 +02:00
lilleman caec22001d Ship the methods a request carries, as booleans a consumer can read
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:32:42 +02:00
lilleman e7a906debf Expect the nine core methods, and a Go bool reading safe and idempotent 2026-09-20 00:32:42 +02:00
lilleman df8d15aeb6 Say what the loader does with a repeated name, and name the constant this one mirrors
Tests / vet + fmt + tests (pull_request) Successful in 1m25s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:25:14 +02:00
lilleman 93d216616b Say what the loader refuses and what this script refuses on its own, and mirror the selector fence
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:22:29 +02:00
lilleman c93281b8cc Pin a port whose winner would move if the registry were re-sorted 2026-09-20 00:22:29 +02:00
lilleman f081fa1433 Re-pin the shipped shape for port.service and protocol's name option
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:16:59 +02:00
lilleman baed259802 Drop a filter that matched nothing, keep the 79 ports it hid, and exit on a value the registers should not hold 2026-09-20 00:16:59 +02:00
lilleman ec6f0e2edb Pin the port tie-break on a port where it discriminates, and protocol selection by name 2026-09-20 00:16:56 +02:00
lilleman d1fbc96a34 Re-pin the shipped shape for the three IANA network tables
Tests / vet + fmt + tests (pull_request) Successful in 1m24s
Tests / Gitea release from CHANGELOG.md (pull_request) Has been skipped
2026-09-20 00:02:37 +02:00
lilleman 2016a7408c Read misc.httpmethod, misc.protocol and misc.port from their IANA registries 2026-09-20 00:01:57 +02:00
lilleman 9feadf4d15 Expect IANA's method, protocol and port registries as tables 2026-09-20 00:01:52 +02:00
27 changed files with 8230 additions and 141 deletions
+9
View File
@@ -11,8 +11,17 @@
- Gitea queues Actions runs that `/repos/{owner}/{repo}/actions/tasks` does not - Gitea queues Actions runs that `/repos/{owner}/{repo}/actions/tasks` does not
list, so an empty task list says nothing about whether CI ran; read list, so an empty task list says nothing about whether CI ran; read
`/repos/{owner}/{repo}/commits/<sha>/status`. `/repos/{owner}/{repo}/commits/<sha>/status`.
- README goal 2 is gated at 7.0 on the `comprehension-panel` skill, and below it
nothing else merges — no feature, no category, no data — bar a security fix and a
dependency bump. `todo.md` carries the round: the nine-seat findings run at depth 1,
a PR per item it names, then the run again, until the score passes. At or above 7.0
every pull request scores with the four-seat run and answers it in one run. No merge
lowers the last score.
- Hard tabs. No comment by default; delete a restatement, a rationale, history, or a file preamble. - Hard tabs. No comment by default; delete a restatement, a rationale, history, or a file preamble.
- One spelling per result: reject the other at `New`, and let the error name the spelling to use. - One spelling per result: reject the other at `New`, and let the error name the spelling to use.
- A standing choice a reader would relitigate goes under Decisions in the README, not in a comment. - A standing choice a reader would relitigate goes under Decisions in the README, not in a comment.
- Prose naming a source states what that source states: read the register's own field
before paraphrasing it, and name the register you read before claiming none
publishes a spelling.
- A README example is a `json` block that loads and renders as a category; `readme_test.go` runs every one. - A README example is a `json` block that loads and renders as a category; `readme_test.go` runs every one.
- Cyclomatic complexity is gated at 14: the table-shaped dispatches (`linkParent`, `compileTemplate`, `renderEdges`) sit at it and stay whole; a function that would pass it is decomposed. - Cyclomatic complexity is gated at 14: the table-shaped dispatches (`linkParent`, `compileTemplate`, `renderEdges`) sit at it and stay whole; a function that would pass it is decomposed.
+13
View File
@@ -43,6 +43,19 @@ replacement, and each removed path, column or flag.
in. `misc.useragent` is the top-user-agents desktop and mobile lists with `browser`, in. `misc.useragent` is the top-user-agents desktop and mobile lists with `browser`,
`device` and `os` columns. `misc.car` keeps its makes and models in a `make` and a `device` and `os` columns. `misc.car` keeps its makes and models in a `make` and a
`model` column. `model` column.
- `misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries as tables.
`misc.httpmethod` is the eight methods RFC 9110 defines and PATCH, carrying
`safe` and `idempotent` as `true` or `false`; `misc.protocol` the keyword, its
`number` and its `name`, selectable by either; `misc.port` renders a TCP port number,
with the IANA `service` name beside it, keeping the first service the registry
describes for a port so a number selects one row.
- `misc.tld` is the IANA root zone as a table: every delegated TLD keyed with its
leading dot, `.se` and `.xn--p1ai`, the register's `type` beside it, and a `unicode`
column holding the form the register displays, so `misc.tld[.рф]` selects the row
`.xn--p1ai` renders. A delegation the register marks "Not assigned" does not ship.
- `misc.loglevel` is the eight syslog severities: the `keyword` a configuration writes,
`err` and `info`, keyed by the numerical `code` a PRI encodes and selectable by
either, with the `severity` RFC 5424 spells beside it.
- `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`,
`locality`, `postal-code` and `street`, weighted by population and address counts `locality`, `postal-code` and `street`, weighted by population and address counts
and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an
+6 -1
View File
@@ -14,13 +14,18 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
| `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` | | `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` |
| `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` | | `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` |
| `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` | | `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 10) | — | — | — | | `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 12) | — | — | — |
| `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` | | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` |
| `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` |
| `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` |
| `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` | | `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` |
| `misc/loglevel.tsv` | [RFC 5424](https://www.rfc-editor.org/rfc/rfc5424.txt) Table 2, the syslog message severities; keywords from the [POSIX](https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html) `<syslog.h>` severity macros | facts | none required | `data-import/loglevel.py` |
| `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` | | `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` |
| `misc/port.tsv` | [IANA Service Name and Transport Protocol Port Number Registry](https://www.iana.org/assignments/service-names-port-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/port.py` |
| `misc/protocol.tsv` | [IANA Protocol Numbers](https://www.iana.org/assignments/protocol-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/protocol.py` |
| `misc/territory.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/territory.py` | | `misc/territory.tsv` | [datasets/country-codes](https://github.com/datasets/country-codes) | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/territory.py` |
| `misc/timezone.tsv` | [IANA tzdb](https://www.iana.org/time-zones) 2026d `zone.tab` and the standard offset of each zone; weights from [GeoNames](https://www.geonames.org/) `cities15000` populations | [public domain](https://data.iana.org/time-zones/tzdb/LICENSE); [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | "Populations from GeoNames, www.geonames.org" | `data-import/timezone.py` | | `misc/timezone.tsv` | [IANA tzdb](https://www.iana.org/time-zones) 2026d `zone.tab` and the standard offset of each zone; weights from [GeoNames](https://www.geonames.org/) `cities15000` populations | [public domain](https://data.iana.org/time-zones/tzdb/LICENSE); [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | "Populations from GeoNames, www.geonames.org" | `data-import/timezone.py` |
| `misc/tld.tsv` | [IANA Root Zone Database](https://www.iana.org/domains/root/db), checked against the root zone's [TLD list](https://data.iana.org/TLD/tlds-alpha-by-domain.txt) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/tld.py` |
| `misc/useragent.tsv` | [top-user-agents](https://github.com/microlinkhq/top-user-agents) desktop and mobile lists | [MIT](https://github.com/microlinkhq/top-user-agents/blob/master/LICENSE.md) | "Copyright © 2020 Kiko Beats" | `data-import/useragent.py` | | `misc/useragent.tsv` | [top-user-agents](https://github.com/microlinkhq/top-user-agents) desktop and mobile lists | [MIT](https://github.com/microlinkhq/top-user-agents/blob/master/LICENSE.md) | "Copyright © 2020 Kiko Beats" | `data-import/useragent.py` |
Every other category is hand-written under [`data/`](data), MIT like the code. Every other category is hand-written under [`data/`](data), MIT like the code.
+125 -57
View File
@@ -162,14 +162,40 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`
`username`, `version` and `word`, formatted per locale; `sv_SE` adds `username`, `version` and `word`, formatted per locale; `sv_SE` adds
`personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc` `personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc`
carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217), carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217),
`datetime` (RFC 3339), `emoji`, `httpstatus`, `language` (ISO 639-1, with its `datetime` (RFC 3339), `emoji`, `httpmethod`, `httpstatus`, `language` (ISO 639-1,
639-2/T code), `mac`, `mimetype`, `objectid`, `territory` (ISO 3166-1), `timezone` with its 639-2/T code), `loglevel` (syslog severity), `mac`, `mimetype`, `objectid`,
(IANA), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, `port`, `protocol`, `territory` (ISO 3166-1), `timezone` (IANA), `tld` (IANA root
zone), `useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`,
`misc.territory.alpha2`, `misc.httpstatus.code` — which `--list` shows. `car`, `misc.territory.alpha2`, `misc.httpstatus.code` — which `--list` shows. `car`,
`currency`, `httpstatus`, `language`, `mimetype`, `territory`, `timezone` and `currency`, `httpmethod`, `httpstatus`, `language`, `loglevel`, `mimetype`, `port`,
`useragent` are [tables](#table), so `misc.territory[SE].capital` and `protocol`, `territory`, `timezone`, `tld` and `useragent` are [tables](#table), so
`misc.currency[Euro].symbol` select a row; `car` and `useragent` carry no key or `misc.territory[SE].capital` and `misc.currency[Euro].symbol` select a row; `car` and
name, so they are drawn from rather than selected in. `useragent` carry no key or name, so they are drawn from rather than selected in.
`misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries.
`misc.httpmethod` is the eight methods RFC 9110 defines and PATCH; the register's
other entries, WebDAV and DeltaV among them, do not ship. Its `safe` and
`idempotent` are `true` or `false`, so a Go `bool` reads them, though `--format json`
and `sql` write them as text, a table column carrying no datatype. `misc.protocol`
carries a `number` and a `name`, the keyword itself where the register spells none out,
and selects by either spelling.
`misc.port` is the TCP assignments, rendering the number a port field holds with the
IANA `service` beside it, and selecting by number alone. A draw spans the whole
register, so pin `misc.port[443]` where a fixture needs a port a reader recognises;
it stops at 49150, IANA assigning nothing above, so an ephemeral source port is
`{int(49152,65535)}`.
`misc.tld` is the root zone's delegated TLDs, keyed with the leading dot
`misc.territory.tld` already carries, `.se`. Its `type` is the register's —
`country-code`, `generic`, `generic-restricted`, `infrastructure` or `sponsored` — and
its `unicode` the form the register displays, `.рф` for `.xn--p1ai` and the key itself
for an ASCII one, which selects the row too, so `misc.tld[.рф]` renders `.xn--p1ai`. A
draw spans the whole zone, `.arpa` and `.zuerich` alike, so pin `misc.tld[.com]` where
a fixture needs one a reader recognises.
`misc.loglevel` is the eight syslog severities, rendering the keyword a configuration
writes, `err` and `info`, keyed by the numerical code a PRI encodes and selectable by
either, so `misc.loglevel[3]` and `misc.loglevel[err]` are one row.
`misc.loglevel.severity` is the name RFC 5424 spells, `Error` beside `err`; it
selects nothing, the code and the keyword do.
[`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence.
`misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most `misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most
@@ -244,7 +270,7 @@ f, err := fejkdata.New(fejkdata.WithSeed(42))
if err != nil { if err != nil {
log.Fatal(err) log.Fatal(err)
} }
v, err := f.Fake("sv_SE.address") // "Kungsvägen 68\n379 17 Stockholm" v, err := f.Fake("sv_SE.address") // "Järvedsvägen 43\n891 77 Järved"
paths := f.List() // every path Fake accepts, sorted paths := f.List() // every path Fake accepts, sorted
v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call v, err = f.FakeTemplate("name: {/sv_SE.person.last}") // compile + render in one call
t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once t, err := f.NewTemplate(`{"format":"name: {x}","x":["bosse","lina"]}`) // compile once
@@ -736,7 +762,7 @@ a minor only adds, and a major is the only release that changes what exists.
| Surface | Major | Minor | | Surface | Major | Minor |
|---------|-------|-------| |---------|-------|-------|
| Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row | | Shipped data | remove or rename a path; change a category's format; remove a value, or change a weight or a repeat; add a reference from one shipped category into another; change a table's key, name, weight or parent column, or remove a row | a path outside a record's columns, a locale, a value in a list, a row |
| Records | remove, rename, retype or add a column; let a column be null | a record, as a new category | | Records (a table is one) | remove, rename, retype or add a column; let a column be null | a record, as a new category |
| Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin | | Data format | a fence: a spelling `New` rejects that it accepted; a template option, since it reserves a field name | a builtin |
| CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format | | CLI | remove or rename a flag, or change its default; change what an exit code means; change the framing a `--format` writes (header, quoting, statement shape), the `--list` layout, or what an error names | a flag, a format |
| Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option | | Library | change or remove an exported name; raise the lowest supported Go | an exported name, a `With…` option |
@@ -775,45 +801,57 @@ the Development section below, and who ships a register the four above then draw
## Goals ## Goals
1. **Valid by construction** — every value passes the check its real consumer 1. **The best fake-data tool available, and used because it is** — a developer
choosing a seed-data library or CLI, in any language, should find this the one
worth switching to, and every goal below it is what that costs. Merit is only
half the goal: once the grammar, the data and the CLI are complete and stable,
it ships where those developers install from and is announced where they read.
2. **Understanding the code is part of the product** — it is a compiler with static
fences behind a small API, it will grow, and the people who extend it will not be
the people who wrote it. So reading cost is judged alongside behaviour: reaching
the unit behind a symptom without asking a person, changing one piece without
holding the rest, names that tell the truth, and a file that stands with no second
document open beside it. A simulated reader panel scores those four, the score is a
ratchet no release lowers, and [`AGENTS.md`](AGENTS.md) says what each pull request
owes it.
3. **Valid by construction** — every value passes the check its real consumer
applies; facts that belong together come from one draw, within a value and applies; facts that belong together come from one draw, within a value and
across categories. across categories.
2. **Text means what it says** — a format renders as written; only `{…}` varies, 4. **Text means what it says** — a format renders as written; only `{…}` varies,
random characters included (`{digits(3)}`). One spelling per result; the wrong random characters included (`{digits(3)}`). One spelling per result; the wrong
one is a load error naming the right one. one is a load error naming the right one.
3. **Every mistake is a load error** — `New` rejects the data and `NewTemplate` 5. **Every mistake is a load error** — `New` rejects the data and `NewTemplate`
the inline template; on a loaded generator `Fake` fails only for an unknown the inline template; on a loaded generator `Fake` fails only for an unknown
path, `FakeStruct` only for a non-struct argument or a type its tags do not path, `FakeStruct` only for a non-struct argument or a type its tags do not
describe, with the same error every call, and `Template.Fake` cannot fail at all. describe, with the same error every call, and `Template.Fake` cannot fail at all.
4. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`: 6. **Zero to a value in one command** — `go install`, then `fejkdata sv_SE.person`:
no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position; no checkout, no flag. Flags are GNU-form (`--seed 42`, `-n 3`) in any position;
the first custom template needs no escape and no option. the first custom template needs no escape and no option.
5. **Data lives in JSON** — a builtin only for what data can't express. 7. **Data lives in JSON** — a builtin only for what data can't express.
6. **Reproducible** — seed in, same stream out; no builtin reads a clock. 8. **Reproducible** — seed in, same stream out; no builtin reads a clock.
7. **Zero dependencies** — standard library only. 9. **Zero dependencies** — standard library only.
8. **Docs index the grammar** — every syntax feature is a heading; every example 10. **Docs index the grammar** — every syntax feature is a heading; every example
runs under test and shows its output; a rule is stated once. runs under test and shows its output; a rule is stated once.
9. **Fast enough to be free** — a value renders in about a microsecond and `New` 11. **Fast enough to be free** — a value renders in about a microsecond and `New`
parses and validates the whole set once upfront, so generating fixtures stays parses and validates the whole set once upfront, so generating fixtures stays
noise against a test's own runtime. noise against a test's own runtime.
10. **Data is sourced, or on its way there** — a shipped fact, a name, place, 12. **Data is sourced, or on its way there** — a shipped fact, a name, place,
code, id or classification, is read from a register or open dataset by a code, id or classification, is read from a register or open dataset by a
[`data-import/`](data-import) script wherever one exists to read; where none [`data-import/`](data-import) script wherever one exists to read; where none
does yet a small hand-written set ships and [`todo.md`](todo.md) carries the does yet a small hand-written set ships and [`todo.md`](todo.md) carries the
step that replaces it. Only non-factual copy stays authored. A sourced table step that replaces it. Only non-factual copy stays authored. A sourced table
holds the rows its source holds: none is added by hand, and one is dropped holds the rows its source holds: none is added by hand, and one is dropped
only by a rule the script states. only by a rule the script states.
11. **Breadth follows what most systems store** — a category is added in proportion 13. **Breadth follows what most systems store** — a category is added in proportion
to how many real schemas hold it: names, addresses, phones, ids, money and to how many real schemas hold it: names, addresses, phones, ids, money and
timestamps before anything domain-specific, and a catalogue serving one niche timestamps before anything domain-specific, and a catalogue serving one niche
waits behind everything serving many. waits behind everything serving many.
12. **Realism is the default, inertness is selectable** — where a value could reach 14. **Realism is the default, inertness is selectable** — where a value could reach
something real, a domain anyone may register or an account a bank could issue, something real, a domain anyone may register or an account a bank could issue,
the realistic breadth ships *and* so does the subset that provably reaches the realistic breadth ships *and* so does the subset that provably reaches
nothing, each on its own path. A fixture that looks nothing like production nothing, each on its own path. A fixture that looks nothing like production
tests nothing; the caller who needs a value that can touch nothing asks for it tests nothing; the caller who needs a value that can touch nothing asks for it
by name. by name.
## Decisions ## Decisions
- **Options and fields share one namespace.** `format`, `weight`, `repeat`, - **Options and fields share one namespace.** `format`, `weight`, `repeat`,
@@ -1184,6 +1222,16 @@ the Development section below, and who ships a register the four above then draw
worth more than the last rows of a table. Where no such link can hold the fact worth more than the last rows of a table. Where no such link can hold the fact
stays a column. Layer your own `misc.territory` over the shipped one and you stays a column. Layer your own `misc.territory` over the shipped one and you
must layer `misc.timezone` too, or the link fails at load naming the row. must layer `misc.timezone` too, or the link fails at load naming the row.
- **A table whose register publishes no frequency draws evenly.** `misc.httpmethod`,
`misc.port`, `misc.httpstatus`, `misc.mimetype` and `misc.tld` weigh every row alike,
so GET is a ninth of the methods drawn. Goal 10 keeps an authored fact out of a
sourced table, and no register publishes how often a method, a port or a TLD is used,
so a weight here would be invented. Where one exists it is read, as `misc.timezone`
reads GeoNames populations and `sv_SE.first-name` SCB bearers.
- **`misc.port` selects by number, and carries no `name`.** 29 of its services sit on
more than one port, `http-alt` on three, so `misc.port[http-alt]` could name no one
row. The number is what a port field holds anyway, and `misc.port[443].service`
reads the other way.
- **`misc.car` is one flat table, not a make linked to its models.** A row is a make - **`misc.car` is one flat table, not a make linked to its models.** A row is a make
and a model drawn together, so no render pairs a Volvo with a RAV4. Two linked and a model drawn together, so no render pairs a Volvo with a RAV4. Two linked
tables would reach the same pairs and add a selector nothing asks for; a make alone tables would reach the same pairs and add a selector nothing asks for; a make alone
@@ -1199,6 +1247,21 @@ the Development section below, and who ships a register the four above then draw
every value names a row instead. A territory the register records no state for every value names a row instead. A territory the register records no state for
stands alone, which is `EH` alone, and naming one for it would be a claim stands alone, which is `EH` alone, and naming one for it would be a claim
fejkdata has no business making. fejkdata has no business making.
- **`misc.loglevel` is a table keyed by the code, rendering POSIX's keyword.** A flat
list of names carries neither the code a PRI encodes nor a selector reaching it, and
goal 3 draws the two as one fact. The canonical spelling losing to the one its domain
writes, above, settles the rest: a configuration writes `info`, so RFC 5424's
`Informational` stays the `severity` column, and the code and the keyword hold the
two selector slots.
- **`misc.tld` keys carry the leading dot, where other tables key on a bare code.**
The register spells a TLD `.se` and `misc.territory.tld` already ships it so, which a
bare key would make two spellings of one fact; `{/misc.tld}` also composes onto a
host with no separator. `misc.tld[se]` misses for it, which `todo.md` carries.
- **`misc.tld` is a table of its own, and `misc.territory.tld` stays a column.** A
`parent` demands a child for every parent row, so linking them would drop every root
zone row naming no territory, which is most of them, and goal 12 holds a sourced
table whole. The loader refuses the link outright anyway: `tld` is a column of
`misc.territory`, and a table may not be named like a column of its ancestor.
- **`misc.territory` carries a currency code, it does not link to `misc.currency`.** - **`misc.territory` carries a currency code, it does not link to `misc.currency`.**
A `parent` demands a child for every parent row, and ISO 4217 registers codes no A `parent` demands a child for every parent row, and ISO 4217 registers codes no
country's row can name: the funds codes (Mvdol, WIR Euro, US Dollar (Next day)), country's row can name: the funds codes (Mvdol, WIR Euro, US Dollar (Next day)),
@@ -1229,6 +1292,38 @@ the Development section below, and who ships a register the four above then draw
through a chain of five tables is every subsequence of it, and the direct chain is through a chain of five tables is every subsequence of it, and the direct chain is
the one a reader can predict from the tables' parents. the one a reader can predict from the tables' parents.
## Layout
```
fejkdata.go Generator, New, options, the embedded data set, List
node.go the node model and JSON -> node compilation
table.go tables: the rows TSV, its options and links, row selection and draws
path.go the dotted-path walk with its selectors, and proving a path resolves
render.go Fake and the recursive renderer (choices, format strings, expansions)
record.go records: Record, the JSON/CSV/SQL serializers, and their entry points
struct.go structs: FakeStruct, fake tags, and a field's Go type as its column's datatype
inline.go inline templates: Template, NewTemplate, FakeTemplate, IsTemplate, and their compile and link
template.go the {token} grammar: scanning, tokens, operands, validation, compiling a format
hold.go the hold: one draw per expansion for paths and operands, and its fences
draw.go one reference draw per render and group: draw sets, the group option, and its fence
family.go a family of linked tables: the rows a render pins, and the fence over paths into one family
reference.go reference sigils, and binding references across the tree
graph.go the render graph: edges, cycles, the repeat bound, tree walks
builtins.go the {name()} function registry and its implementations
layout.go date and time layouts: the instants one is proved against, and the two samples
checksum.go the check characters a derivation appends, and the IBAN they sit inside
transform.go the builtins that rewrite an operand's value, and the ASCII folding
calc.go the {calc()} arithmetic evaluator: parser, eval, validation
datatype.go column datatypes: DataType, where datatype and null may sit, a column's datatype
value.go the value proof: what a typed column or calc operand holds, checked at load
data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge
cmd/fejkdata/ the fejkdata CLI
data/ shipped data (JSON, and a TSV per table), embedded at build: locale folders, geo, misc
data-import/ the scripts that rebuild each sourced table (see DATA-LICENSES.md)
release-tooling/ the release CI publishes from the changelog heading
testdata/ the pinned shipped shape (see Versioning)
```
## Development ## Development
Everything runs in Docker — **no local tooling beyond Docker is needed**. Everything runs in Docker — **no local tooling beyond Docker is needed**.
@@ -1296,13 +1391,18 @@ a copy of the surname file:
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/currency.py
docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py docker compose run --rm --user "$(id -u):$(id -g)" -e TRAFIKVERKET_API_KEY data-import data-import/geo-se.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-us.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpmethod.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/loglevel.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/port.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/protocol.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/territory.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/timezone.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/tld.py
docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/useragent.py
``` ```
@@ -1311,38 +1411,6 @@ and merge: once `main` passes the gate, CI tags that commit `vX.Y.Z` and publish
the Gitea release with the section as its body. A top heading of `[Unreleased]` the Gitea release with the section as its body. A top heading of `[Unreleased]`
publishes nothing. publishes nothing.
## Layout
```
fejkdata.go Generator, New, options, the embedded data set, List
node.go the node model and JSON -> node compilation
table.go tables: the rows TSV, its options and links, row selection and draws
path.go the dotted-path walk with its selectors, and proving a path resolves
render.go Fake and the recursive renderer (choices, format strings, expansions)
record.go records: Record, the JSON/CSV/SQL serializers, and their entry points
struct.go structs: FakeStruct, fake tags, and a field's Go type as its column's datatype
inline.go inline templates: Template, NewTemplate, FakeTemplate, IsTemplate, and their compile and link
template.go the {token} grammar: scanning, tokens, operands, validation, compiling a format
hold.go the hold: one draw per expansion for paths and operands, and its fences
draw.go one reference draw per render and group: draw sets, the group option, and its fence
family.go a family of linked tables: the rows a render pins, and the fence over paths into one family
reference.go reference sigils, and binding references across the tree
graph.go the render graph: edges, cycles, the repeat bound, tree walks
builtins.go the {name()} function registry and its implementations
layout.go date and time layouts: the instants one is proved against, and the two samples
checksum.go the check characters a derivation appends, and the IBAN they sit inside
transform.go the builtins that rewrite an operand's value, and the ASCII folding
calc.go the {calc()} arithmetic evaluator: parser, eval, validation
datatype.go column datatypes: DataType, where datatype and null may sit, a column's datatype
value.go the value proof: what a typed column or calc operand holds, checked at load
data.go data loading: fs.FS folders/files -> namespace tree, multi-source merge
cmd/fejkdata/ the fejkdata CLI
data/ shipped data (JSON, and a TSV per table), embedded at build: locale folders, geo, misc
data-import/ the scripts that rebuild each sourced table (see DATA-LICENSES.md)
release-tooling/ the release CI publishes from the changelog heading
testdata/ the pinned shipped shape (see Versioning)
```
## License ## License
MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes). MIT — see [LICENSE](LICENSE). Forked from [github.com/Timewave-AB/fakes](https://github.com/Timewave-AB/fakes).
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry.
data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
A method ships when the register cites RFC 9110 section 9.3 or RFC 5789 for it, which
is the eight HTTP defines and PATCH; the register's other entries do not. The
register's `yes` and `no` ship as `true` and `false`, which a consumer's boolean reads.
"""
import argparse
import csv
import io
import re
import sys
from pathlib import Path
import source
import tsv
SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["idempotent", "method", "safe"]
CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]")
BOOLEAN = {"yes": "true", "no": "false"}
SELECTOR = '[]{}"|' # mirrors inSelector in table.go
EXPECTED = 9
def rows(text):
for r in csv.DictReader(io.StringIO(text)):
method = (r["Method Name"] or "").strip()
if not CORE.search(r["Reference"] or ""):
continue
if set(method) & set(SELECTOR):
sys.exit(f"{method}: a method holding one of {SELECTOR} is one a selector cannot spell")
answers = {}
for column in ("Safe", "Idempotent"):
answer = (r[column] or "").strip()
if answer not in BOOLEAN:
sys.exit(f"{method}: {column} {answer!r} is neither yes nor no")
answers[column] = BOOLEAN[answer]
yield {"idempotent": answers["Idempotent"], "method": method, "safe": answers["Safe"]}
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--source", default=SOURCE)
p.add_argument("--out", default=str(OUT))
a = p.parse_args()
table = sorted(rows(source.fetch(a.source, a.cache, "http-methods.csv").decode("utf-8")), key=lambda r: r["method"])
if len(table) != EXPECTED:
sys.exit(f"{len(table)} methods cite RFC 9110 section 9.3 or RFC 5789, not {EXPECTED}; the register's Reference column has moved")
tsv.write(a.out, COLUMNS, table)
if __name__ == "__main__":
main()
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Rebuild data/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE]
Every severity in Table 2 ships, paired to POSIX's keywords by the order each source
lists them, which holds only because a keyword prefixes the severity RFC 5424 numbers
alike.
"""
import argparse
import html
import re
import sys
from pathlib import Path
import source
import tsv
SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt"
POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["code", "keyword", "severity"]
TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S)
ROW = re.compile(r"^ +(\d) +([A-Z][a-z]+): ", re.M)
SECTION = re.compile(r"severity level portion of the.*?macros:(.*?)The following shall be declared as functions", re.S)
MACRO = re.compile(r"\bLOG_([A-Z]+)\b")
EXPECTED = 8
def severities(text):
table = TABLE.search(text)
if not table:
sys.exit("the RFC holds no text between the facility and the severity caption; its tables have moved")
return ROW.findall(table.group(1))
def keywords(page):
section = SECTION.search(html.unescape(re.sub(r"<[^>]+>", "", page)))
if not section:
sys.exit("POSIX spells no severity list between its own caption and the function declarations; the page has moved")
return [m.lower() for m in MACRO.findall(section.group(1))]
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--source", default=SOURCE)
p.add_argument("--posix", default=POSIX)
p.add_argument("--out", default=str(OUT))
a = p.parse_args()
table = sorted(severities(source.fetch(a.source, a.cache, "rfc5424.txt").decode("utf-8")))
codes = [code for code, _ in table]
if codes != [str(i) for i in range(EXPECTED)]:
sys.exit(f"the severity table numbers {codes}, not 0 through {EXPECTED - 1}")
words = keywords(source.fetch(a.posix, a.cache, "posix-syslog.h.html").decode("utf-8"))
if len(words) != EXPECTED:
sys.exit(f"POSIX lists {len(words)} severity macros, not {EXPECTED}: {words}")
rows = []
for (code, severity), keyword in zip(table, words):
if not severity.lower().startswith(keyword):
sys.exit(f"severity {code} is {severity} in the RFC and {keyword} in POSIX, which is no prefix of it")
rows.append({"code": code, "keyword": keyword, "severity": severity})
tsv.write(a.out, COLUMNS, rows)
if __name__ == "__main__":
main()
+55
View File
@@ -0,0 +1,55 @@
#!/usr/bin/env python3
"""Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
A row needs a service name and a numeric TCP port. A port the registry lists more than
once keeps the first service it describes, or the first of them where it describes none,
so a number selects one row.
"""
import argparse
import csv
import io
import sys
from pathlib import Path
import source
import tsv
SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["number", "service"]
FLOOR = 4000
def rows(text):
best, described = {}, set()
for r in csv.DictReader(io.StringIO(text)):
service, number = (r["Service Name"] or "").strip(), (r["Port Number"] or "").strip()
transport, description = (r["Transport Protocol"] or "").strip(), (r["Description"] or "").strip()
if transport != "tcp" or not service or not number.isdigit():
continue
if not 1 <= int(number) <= 65535:
sys.exit(f"{service}: port {number} is outside 1-65535")
if number not in best or (description and number not in described):
best[number] = {"number": number, "service": service}
if description:
described.add(number)
return sorted(best.values(), key=lambda r: int(r["number"]))
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--source", default=SOURCE)
p.add_argument("--out", default=str(OUT))
a = p.parse_args()
table = rows(source.fetch(a.source, a.cache, "service-names-port-numbers.csv").decode("utf-8"))
if len(table) < FLOOR:
sys.exit(f"only {len(table)} ports named a TCP service; the registry's columns have moved")
tsv.write(a.out, COLUMNS, table)
if __name__ == "__main__":
main()
+67
View File
@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry.
data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
A row needs a number and a keyword, which drops the seven numbers assigned to a class of
protocols rather than to one; a keyword the registry reserves or marks deprecated goes
too. Where the registry names no protocol beside the keyword, the keyword is the name.
"""
import argparse
import csv
import io
import sys
from pathlib import Path
import source
import tsv
SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["keyword", "name", "number"]
SKIP = ("Reserved", "deprecated")
SELECTOR = '[]{}"|' # mirrors inSelector in table.go
def rows(text):
for r in csv.DictReader(io.StringIO(text)):
number, keyword = (r["Decimal"] or "").strip(), (r["Keyword"] or "").strip()
if not number.isdigit() or not keyword or any(s in keyword for s in SKIP):
continue
if not 0 <= int(number) <= 255:
sys.exit(f"{keyword}: protocol number {number} is outside 0-255")
yield {"keyword": keyword, "name": " ".join((r["Protocol"] or "").split()) or keyword, "number": number}
def refuse_an_unselectable_row(table):
"""What the loader refuses at New, and a name that names two rows rather than one."""
keywords, names = {}, {}
for r in table:
for column in ("keyword", "name"):
if set(r[column]) & set(SELECTOR):
sys.exit(f"{r['keyword']}: {column} {r[column]!r} holds one of {SELECTOR}, which a selector cannot spell")
if r["keyword"] in keywords:
sys.exit(f"{r['keyword']}: repeats the keyword of protocol {keywords[r['keyword']]}")
keywords[r["keyword"]] = r["number"]
for r in table:
if r["name"] != r["keyword"] and r["name"] in keywords:
sys.exit(f"{r['keyword']}: name {r['name']!r} is another protocol's keyword")
if r["name"] in names:
sys.exit(f"{r['keyword']}: name {r['name']!r} repeats {names[r['name']]}'s, so it selects neither")
names[r["name"]] = r["keyword"]
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--source", default=SOURCE)
p.add_argument("--out", default=str(OUT))
a = p.parse_args()
table = sorted(rows(source.fetch(a.source, a.cache, "protocol-numbers-1.csv").decode("utf-8")), key=lambda r: int(r["number"]))
refuse_an_unselectable_row(table)
tsv.write(a.out, COLUMNS, table)
if __name__ == "__main__":
main()
+68
View File
@@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""Rebuild data/misc/tld.tsv from the IANA Root Zone Database.
data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE]
A TLD ships when the database names a manager for it; one it marks "Not assigned" is a
delegation the root zone no longer holds, and the zone file is fetched to prove the two
agree. The key is the A-label the root zone holds; `unicode` is the form the database
displays, less the bidi marks it wraps a right-to-left label in, and proved to punycode
back to the key.
"""
import argparse
import html
import re
import sys
from pathlib import Path
import source
import tsv
SOURCE = "https://www.iana.org/domains/root/db"
ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv"
CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["tld", "type", "unicode"]
TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored")
BIDI = {0x200E: None, 0x200F: None}
ROW = re.compile(r'<a href="/domains/root/db/([^"]+)\.html">([^<]+)</a>.*?</td>\s*<td>([^<]*)</td>\s*<td>(.*?)</td>', re.S)
def rows(page):
for a_label, shown, kind, manager in ROW.findall(page):
if "Not assigned" in manager:
continue
kind = kind.strip()
if kind not in TYPES:
sys.exit(f"{a_label}: the database types it {kind!r}, which is none of {TYPES}")
yield {"tld": "." + a_label, "type": kind, "unicode": html.unescape(shown).strip().translate(BIDI)}
def prove_each_unicode_form_is_its_key(table):
"""The displayed form is the key's own spelling, which punycode encodes back to."""
for r in table:
label = r["unicode"][1:]
encoded = label if label.isascii() else "xn--" + label.encode("punycode").decode("ascii")
if "." + encoded != r["tld"]:
sys.exit(f"{r['tld']}: the database displays {r['unicode']!r}, which is no spelling of it")
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
p.add_argument("--cache", default=str(CACHE))
p.add_argument("--source", default=SOURCE)
p.add_argument("--zone", default=ZONE)
p.add_argument("--out", default=str(OUT))
a = p.parse_args()
table = sorted(rows(source.fetch(a.source, a.cache, "root-db.html").decode("utf-8")), key=lambda r: r["tld"])
prove_each_unicode_form_is_its_key(table)
listing = source.fetch(a.zone, a.cache, "tlds-alpha-by-domain.txt").decode("utf-8")
delegated = {"." + line.strip().lower() for line in listing.splitlines() if line.strip() and not line.startswith("#")}
shipped = {r["tld"] for r in table}
if len(table) != len(delegated) or shipped != delegated:
sys.exit(f"the database assigns {len(table)} rows where the root zone holds {len(delegated)} TLDs: {sorted(shipped ^ delegated)[:10]}")
tsv.write(a.out, COLUMNS, table)
if __name__ == "__main__":
main()
+1
View File
@@ -0,0 +1 @@
{ "format": "{method}", "rows": "httpmethod.tsv", "key": "method" }
+10
View File
@@ -0,0 +1,10 @@
idempotent method safe
false CONNECT false
true DELETE false
true GET true
true HEAD true
true OPTIONS true
false PATCH false
false POST false
true PUT false
true TRACE true
1 idempotent method safe
2 false CONNECT false
3 true DELETE false
4 true GET true
5 true HEAD true
6 true OPTIONS true
7 false PATCH false
8 false POST false
9 true PUT false
10 true TRACE true
+1
View File
@@ -0,0 +1 @@
{ "format": "{keyword}", "rows": "loglevel.tsv", "key": "code", "name": "keyword" }
+9
View File
@@ -0,0 +1,9 @@
code keyword severity
0 emerg Emergency
1 alert Alert
2 crit Critical
3 err Error
4 warning Warning
5 notice Notice
6 info Informational
7 debug Debug
1 code keyword severity
2 0 emerg Emergency
3 1 alert Alert
4 2 crit Critical
5 3 err Error
6 4 warning Warning
7 5 notice Notice
8 6 info Informational
9 7 debug Debug
+1
View File
@@ -0,0 +1 @@
{ "format": "{number}", "rows": "port.tsv", "key": "number" }
+5890
View File
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
{ "format": "{keyword}", "rows": "protocol.tsv", "key": "keyword", "name": "name" }
+140
View File
@@ -0,0 +1,140 @@
keyword name number
HOPOPT IPv6 Hop-by-Hop Option 0
ICMP Internet Control Message 1
IGMP Internet Group Management 2
GGP Gateway-to-Gateway 3
IPv4 IPv4 encapsulation 4
ST Stream 5
TCP Transmission Control 6
CBT CBT 7
EGP Exterior Gateway Protocol 8
IGP any private interior gateway (used by Cisco for their IGRP) 9
BBN-RCC-MON BBN RCC Monitoring 10
NVP-II Network Voice Protocol 11
PUP PUP 12
EMCON EMCON 14
XNET Cross Net Debugger 15
CHAOS Chaos 16
UDP User Datagram 17
MUX Multiplexing 18
DCN-MEAS DCN Measurement Subsystems 19
HMP Host Monitoring 20
PRM Packet Radio Measurement 21
XNS-IDP XEROX NS IDP 22
TRUNK-1 Trunk-1 23
TRUNK-2 Trunk-2 24
LEAF-1 Leaf-1 25
LEAF-2 Leaf-2 26
RDP Reliable Data Protocol 27
IRTP Internet Reliable Transaction 28
ISO-TP4 ISO Transport Protocol Class 4 29
NETBLT Bulk Data Transfer Protocol 30
MFE-NSP MFE Network Services Protocol 31
MERIT-INP MERIT Internodal Protocol 32
DCCP Datagram Congestion Control Protocol 33
3PC Third Party Connect Protocol 34
IDPR Inter-Domain Policy Routing Protocol 35
XTP XTP 36
DDP Datagram Delivery Protocol 37
IDPR-CMTP IDPR Control Message Transport Proto 38
TP++ TP++ Transport Protocol 39
IL IL Transport Protocol 40
IPv6 IPv6 encapsulation 41
SDRP Source Demand Routing Protocol 42
IPv6-Route Routing Header for IPv6 43
IPv6-Frag Fragment Header for IPv6 44
IDRP Inter-Domain Routing Protocol 45
RSVP Reservation Protocol 46
GRE Generic Routing Encapsulation 47
DSR Dynamic Source Routing Protocol 48
BNA BNA 49
ESP Encap Security Payload 50
AH Authentication Header 51
I-NLSP Integrated Net Layer Security TUBA 52
NARP NBMA Address Resolution Protocol 54
Min-IPv4 Minimal IPv4 Encapsulation 55
TLSP Transport Layer Security Protocol using Kryptonet key management 56
SKIP SKIP 57
IPv6-ICMP ICMP for IPv6 58
IPv6-NoNxt No Next Header for IPv6 59
IPv6-Opts Destination Options for IPv6 60
CFTP CFTP 62
SAT-EXPAK SATNET and Backroom EXPAK 64
KRYPTOLAN Kryptolan 65
RVD MIT Remote Virtual Disk Protocol 66
IPPC Internet Pluribus Packet Core 67
SAT-MON SATNET Monitoring 69
VISA VISA Protocol 70
IPCV Internet Packet Core Utility 71
CPNX Computer Protocol Network Executive 72
CPHB Computer Protocol Heart Beat 73
WSN Wang Span Network 74
PVP Packet Video Protocol 75
BR-SAT-MON Backroom SATNET Monitoring 76
SUN-ND SUN ND PROTOCOL-Temporary 77
WB-MON WIDEBAND Monitoring 78
WB-EXPAK WIDEBAND EXPAK 79
ISO-IP ISO Internet Protocol 80
VMTP VMTP 81
SECURE-VMTP SECURE-VMTP 82
VINES VINES 83
IPTM Internet Protocol Traffic Manager 84
NSFNET-IGP NSFNET-IGP 85
DGP Dissimilar Gateway Protocol 86
TCF TCF 87
EIGRP EIGRP 88
OSPFIGP OSPFIGP 89
Sprite-RPC Sprite RPC Protocol 90
LARP Locus Address Resolution Protocol 91
MTP Multicast Transport Protocol 92
AX.25 AX.25 Frames 93
IPIP IP-within-IP Encapsulation Protocol 94
SCC-SP Semaphore Communications Sec. Pro. 96
ETHERIP Ethernet-within-IP Encapsulation 97
ENCAP Encapsulation Header 98
GMTP GMTP 100
IFMP Ipsilon Flow Management Protocol 101
PNNI PNNI over IP 102
PIM Protocol Independent Multicast 103
ARIS ARIS 104
SCPS SCPS 105
QNX QNX 106
A/N Active Networks 107
IPComp IP Payload Compression Protocol 108
SNP Sitara Networks Protocol 109
Compaq-Peer Compaq Peer Protocol 110
IPX-in-IP IPX in IP 111
VRRP Virtual Router Redundancy Protocol 112
PGM PGM Reliable Transport Protocol 113
L2TP Layer Two Tunneling Protocol 115
DDX D-II Data Exchange (DDX) 116
IATP Interactive Agent Transfer Protocol 117
STP Schedule Transfer Protocol 118
SRP SpectraLink Radio Protocol 119
UTI UTI 120
SMP Simple Message Protocol 121
PTP Performance Transparency Protocol 123
ISIS over IPv4 ISIS over IPv4 124
FIRE FIRE 125
CRTP Combat Radio Transport Protocol 126
CRUDP Combat Radio User Datagram 127
SSCOPMCE SSCOPMCE 128
IPLT IPLT 129
SPS Secure Packet Shield 130
PIPE Private IP Encapsulation within IP 131
SCTP Stream Control Transmission Protocol 132
FC Fibre Channel 133
RSVP-E2E-IGNORE RSVP-E2E-IGNORE 134
Mobility Header Mobility Header 135
UDPLite UDPLite 136
MPLS-in-IP MPLS-in-IP 137
manet MANET Protocols 138
HIP Host Identity Protocol 139
Shim6 Shim6 Protocol 140
WESP Wrapped Encapsulating Security Payload 141
ROHC Robust Header Compression 142
Ethernet Ethernet 143
AGGFRAG AGGFRAG encapsulation payload for ESP 144
NSH Network Service Header 145
Homa Homa 146
BIT-EMU Bit-stream Emulation 147
1 keyword name number
2 HOPOPT IPv6 Hop-by-Hop Option 0
3 ICMP Internet Control Message 1
4 IGMP Internet Group Management 2
5 GGP Gateway-to-Gateway 3
6 IPv4 IPv4 encapsulation 4
7 ST Stream 5
8 TCP Transmission Control 6
9 CBT CBT 7
10 EGP Exterior Gateway Protocol 8
11 IGP any private interior gateway (used by Cisco for their IGRP) 9
12 BBN-RCC-MON BBN RCC Monitoring 10
13 NVP-II Network Voice Protocol 11
14 PUP PUP 12
15 EMCON EMCON 14
16 XNET Cross Net Debugger 15
17 CHAOS Chaos 16
18 UDP User Datagram 17
19 MUX Multiplexing 18
20 DCN-MEAS DCN Measurement Subsystems 19
21 HMP Host Monitoring 20
22 PRM Packet Radio Measurement 21
23 XNS-IDP XEROX NS IDP 22
24 TRUNK-1 Trunk-1 23
25 TRUNK-2 Trunk-2 24
26 LEAF-1 Leaf-1 25
27 LEAF-2 Leaf-2 26
28 RDP Reliable Data Protocol 27
29 IRTP Internet Reliable Transaction 28
30 ISO-TP4 ISO Transport Protocol Class 4 29
31 NETBLT Bulk Data Transfer Protocol 30
32 MFE-NSP MFE Network Services Protocol 31
33 MERIT-INP MERIT Internodal Protocol 32
34 DCCP Datagram Congestion Control Protocol 33
35 3PC Third Party Connect Protocol 34
36 IDPR Inter-Domain Policy Routing Protocol 35
37 XTP XTP 36
38 DDP Datagram Delivery Protocol 37
39 IDPR-CMTP IDPR Control Message Transport Proto 38
40 TP++ TP++ Transport Protocol 39
41 IL IL Transport Protocol 40
42 IPv6 IPv6 encapsulation 41
43 SDRP Source Demand Routing Protocol 42
44 IPv6-Route Routing Header for IPv6 43
45 IPv6-Frag Fragment Header for IPv6 44
46 IDRP Inter-Domain Routing Protocol 45
47 RSVP Reservation Protocol 46
48 GRE Generic Routing Encapsulation 47
49 DSR Dynamic Source Routing Protocol 48
50 BNA BNA 49
51 ESP Encap Security Payload 50
52 AH Authentication Header 51
53 I-NLSP Integrated Net Layer Security TUBA 52
54 NARP NBMA Address Resolution Protocol 54
55 Min-IPv4 Minimal IPv4 Encapsulation 55
56 TLSP Transport Layer Security Protocol using Kryptonet key management 56
57 SKIP SKIP 57
58 IPv6-ICMP ICMP for IPv6 58
59 IPv6-NoNxt No Next Header for IPv6 59
60 IPv6-Opts Destination Options for IPv6 60
61 CFTP CFTP 62
62 SAT-EXPAK SATNET and Backroom EXPAK 64
63 KRYPTOLAN Kryptolan 65
64 RVD MIT Remote Virtual Disk Protocol 66
65 IPPC Internet Pluribus Packet Core 67
66 SAT-MON SATNET Monitoring 69
67 VISA VISA Protocol 70
68 IPCV Internet Packet Core Utility 71
69 CPNX Computer Protocol Network Executive 72
70 CPHB Computer Protocol Heart Beat 73
71 WSN Wang Span Network 74
72 PVP Packet Video Protocol 75
73 BR-SAT-MON Backroom SATNET Monitoring 76
74 SUN-ND SUN ND PROTOCOL-Temporary 77
75 WB-MON WIDEBAND Monitoring 78
76 WB-EXPAK WIDEBAND EXPAK 79
77 ISO-IP ISO Internet Protocol 80
78 VMTP VMTP 81
79 SECURE-VMTP SECURE-VMTP 82
80 VINES VINES 83
81 IPTM Internet Protocol Traffic Manager 84
82 NSFNET-IGP NSFNET-IGP 85
83 DGP Dissimilar Gateway Protocol 86
84 TCF TCF 87
85 EIGRP EIGRP 88
86 OSPFIGP OSPFIGP 89
87 Sprite-RPC Sprite RPC Protocol 90
88 LARP Locus Address Resolution Protocol 91
89 MTP Multicast Transport Protocol 92
90 AX.25 AX.25 Frames 93
91 IPIP IP-within-IP Encapsulation Protocol 94
92 SCC-SP Semaphore Communications Sec. Pro. 96
93 ETHERIP Ethernet-within-IP Encapsulation 97
94 ENCAP Encapsulation Header 98
95 GMTP GMTP 100
96 IFMP Ipsilon Flow Management Protocol 101
97 PNNI PNNI over IP 102
98 PIM Protocol Independent Multicast 103
99 ARIS ARIS 104
100 SCPS SCPS 105
101 QNX QNX 106
102 A/N Active Networks 107
103 IPComp IP Payload Compression Protocol 108
104 SNP Sitara Networks Protocol 109
105 Compaq-Peer Compaq Peer Protocol 110
106 IPX-in-IP IPX in IP 111
107 VRRP Virtual Router Redundancy Protocol 112
108 PGM PGM Reliable Transport Protocol 113
109 L2TP Layer Two Tunneling Protocol 115
110 DDX D-II Data Exchange (DDX) 116
111 IATP Interactive Agent Transfer Protocol 117
112 STP Schedule Transfer Protocol 118
113 SRP SpectraLink Radio Protocol 119
114 UTI UTI 120
115 SMP Simple Message Protocol 121
116 PTP Performance Transparency Protocol 123
117 ISIS over IPv4 ISIS over IPv4 124
118 FIRE FIRE 125
119 CRTP Combat Radio Transport Protocol 126
120 CRUDP Combat Radio User Datagram 127
121 SSCOPMCE SSCOPMCE 128
122 IPLT IPLT 129
123 SPS Secure Packet Shield 130
124 PIPE Private IP Encapsulation within IP 131
125 SCTP Stream Control Transmission Protocol 132
126 FC Fibre Channel 133
127 RSVP-E2E-IGNORE RSVP-E2E-IGNORE 134
128 Mobility Header Mobility Header 135
129 UDPLite UDPLite 136
130 MPLS-in-IP MPLS-in-IP 137
131 manet MANET Protocols 138
132 HIP Host Identity Protocol 139
133 Shim6 Shim6 Protocol 140
134 WESP Wrapped Encapsulating Security Payload 141
135 ROHC Robust Header Compression 142
136 Ethernet Ethernet 143
137 AGGFRAG AGGFRAG encapsulation payload for ESP 144
138 NSH Network Service Header 145
139 Homa Homa 146
140 BIT-EMU Bit-stream Emulation 147
+1
View File
@@ -0,0 +1 @@
{ "format": "{tld}", "rows": "tld.tsv", "key": "tld", "name": "unicode" }
+1439
View File
File diff suppressed because it is too large Load Diff
+38 -24
View File
@@ -95,30 +95,44 @@ func TestShippedMiscCategories(t *testing.T) {
func TestShippedMiscReferenceData(t *testing.T) { func TestShippedMiscReferenceData(t *testing.T) {
f := newGenerator(t, "data/misc", WithSeed(1)) f := newGenerator(t, "data/misc", WithSeed(1))
re := map[string]*regexp.Regexp{ re := map[string]*regexp.Regexp{
"currency": regexp.MustCompile(`^[A-Z]{3}$`), "currency": regexp.MustCompile(`^[A-Z]{3}$`),
"currency.name": regexp.MustCompile(`\p{L}`), "currency.name": regexp.MustCompile(`\p{L}`),
"currency.symbol": regexp.MustCompile(`^\S+$`), "currency.symbol": regexp.MustCompile(`^\S+$`),
"territory": regexp.MustCompile(`\p{L}`), "territory": regexp.MustCompile(`\p{L}`),
"territory.alpha2": regexp.MustCompile(`^[A-Z]{2}$`), "territory.alpha2": regexp.MustCompile(`^[A-Z]{2}$`),
"territory.alpha3": regexp.MustCompile(`^[A-Z]{3}$`), "territory.alpha3": regexp.MustCompile(`^[A-Z]{3}$`),
"territory.country": regexp.MustCompile(`^[A-Z]{2}$`), "territory.country": regexp.MustCompile(`^[A-Z]{2}$`),
"language": regexp.MustCompile(`\p{L}`), "language": regexp.MustCompile(`\p{L}`),
"language.code": regexp.MustCompile(`^[a-z]{2}$`), "language.code": regexp.MustCompile(`^[a-z]{2}$`),
"language.code3": regexp.MustCompile(`^[a-z]{3}$`), "language.code3": regexp.MustCompile(`^[a-z]{3}$`),
"timezone": regexp.MustCompile(`^[A-Za-z]+(/[A-Za-z0-9_+-]+)+$`), "timezone": regexp.MustCompile(`^[A-Za-z]+(/[A-Za-z0-9_+-]+)+$`),
"timezone.offset": regexp.MustCompile(`^[+-](0\d|1[0-4]):[0-5]\d$`), "timezone.offset": regexp.MustCompile(`^[+-](0\d|1[0-4]):[0-5]\d$`),
"timezone.territory": regexp.MustCompile(`^[A-Z]{2}$`), "timezone.territory": regexp.MustCompile(`^[A-Z]{2}$`),
"timezone.weight": regexp.MustCompile(`^[1-9]\d*$`), "timezone.weight": regexp.MustCompile(`^[1-9]\d*$`),
"mimetype": regexp.MustCompile(`^[a-z]+/[a-z0-9.+-]+$`), "mimetype": regexp.MustCompile(`^[a-z]+/[a-z0-9.+-]+$`),
"mimetype.ext": regexp.MustCompile(`^\.[a-z0-9_-]+$`), "mimetype.ext": regexp.MustCompile(`^\.[a-z0-9_-]+$`),
"httpstatus": regexp.MustCompile(`^[1-5]\d{2} \S.*$`), "httpmethod": regexp.MustCompile(`^(CONNECT|DELETE|GET|HEAD|OPTIONS|PATCH|POST|PUT|TRACE)$`),
"httpstatus.code": regexp.MustCompile(`^[1-5]\d{2}$`), "httpmethod.safe": regexp.MustCompile(`^(true|false)$`),
"useragent": regexp.MustCompile(`^Mozilla/5\.0 .+`), "httpmethod.idempotent": regexp.MustCompile(`^(true|false)$`),
"useragent.browser": regexp.MustCompile(`^(Chrome|Edge|Firefox|Opera|Safari|Samsung Internet)$`), "port": regexp.MustCompile(`^[1-9]\d{0,4}$`),
"useragent.device": regexp.MustCompile(`^(desktop|mobile)$`), "port.service": regexp.MustCompile(`^\S+$`),
"useragent.os": regexp.MustCompile(`^(Android|ChromeOS|Linux|Windows|iOS|macOS)$`), "protocol": regexp.MustCompile(`^\S+( \S+)*$`),
"car": regexp.MustCompile(`^\S.* \S`), "protocol.name": regexp.MustCompile(`\S`),
"car.make": regexp.MustCompile(`^\S`), "protocol.number": regexp.MustCompile(`^\d{1,3}$`),
"loglevel": regexp.MustCompile(`^(alert|crit|debug|emerg|err|info|notice|warning)$`),
"loglevel.code": regexp.MustCompile(`^[0-7]$`),
"loglevel.severity": regexp.MustCompile(`^(Alert|Critical|Debug|Emergency|Error|Informational|Notice|Warning)$`),
"tld": regexp.MustCompile(`^\.[a-z0-9]([a-z0-9-]*[a-z0-9])?$`),
"tld.type": regexp.MustCompile(`^(country-code|generic|generic-restricted|infrastructure|sponsored)$`),
"tld.unicode": regexp.MustCompile(`^\.\S+$`),
"httpstatus": regexp.MustCompile(`^[1-5]\d{2} \S.*$`),
"httpstatus.code": regexp.MustCompile(`^[1-5]\d{2}$`),
"useragent": regexp.MustCompile(`^Mozilla/5\.0 .+`),
"useragent.browser": regexp.MustCompile(`^(Chrome|Edge|Firefox|Opera|Safari|Samsung Internet)$`),
"useragent.device": regexp.MustCompile(`^(desktop|mobile)$`),
"useragent.os": regexp.MustCompile(`^(Android|ChromeOS|Linux|Windows|iOS|macOS)$`),
"car": regexp.MustCompile(`^\S.* \S`),
"car.make": regexp.MustCompile(`^\S`),
} }
for i := 0; i < 100; i++ { for i := 0; i < 100; i++ {
for p, rx := range re { for p, rx := range re {
+6 -6
View File
@@ -5,8 +5,8 @@
// then generate values by path: // then generate values by path:
// //
// f, _ := fejkdata.New(fejkdata.WithSeed(42)) // f, _ := fejkdata.New(fejkdata.WithSeed(42))
// f.Fake("sv_SE.address") // "Storgatan 12\n234 56 Göteborg" // f.Fake("sv_SE.address") // "Järvedsvägen 43\n891 77 Järved"
// f.Fake("sv_SE.address.locality") // "Göteborg" // f.Fake("sv_SE.address.locality") // "Sundbyberg": a second call draws afresh
// //
// Several sources merge in order, the last winning a name clash, so custom data // Several sources merge in order, the last winning a name clash, so custom data
// layers over the built-ins. The JSON template format is documented in the README. // layers over the built-ins. The JSON template format is documented in the README.
@@ -29,8 +29,8 @@ import (
//go:embed data //go:embed data
var shippedFS embed.FS var shippedFS embed.FS
// MaxRepeat caps a repeat, and the renders nested repeats multiply to along any // MaxRepeat caps a repeat, and caps the renders nested repeats multiply to along
// path; the CLI's --repeat shares it. // any one path; the CLI's --repeat shares it.
const MaxRepeat = 1 << 20 const MaxRepeat = 1 << 20
var _ [^uint(0)>>63 - 1]struct{} // 64-bit only, per the README's Decisions var _ [^uint(0)>>63 - 1]struct{} // 64-bit only, per the README's Decisions
@@ -40,8 +40,8 @@ var ErrNoData = errors.New("no data: WithoutShippedData needs at least one WithD
// Generator generates fake data from a loaded namespace tree. Create one with [New]. // Generator generates fake data from a loaded namespace tree. Create one with [New].
// It is safe for concurrent use; a seeded sequence is reproducible only when drawn // It is safe for concurrent use; a seeded sequence is reproducible only when drawn
// from one goroutine. The compiled tree is immutable after [New], and Fake, // from one goroutine. The compiled tree is immutable after [New], so List reads it
// NewTemplate and List read it concurrently without a lock. // without a lock; mu guards the rest, which every other entry point takes.
type Generator struct { type Generator struct {
mu sync.Mutex mu sync.Mutex
rand *session rand *session
+3 -3
View File
@@ -9,9 +9,9 @@ import (
"strings" "strings"
) )
// node is a compiled template element: a choice or a template. Compiling JSON // node is a compiled element of the namespace tree: a folder, choice, null,
// into these once (see compile) means rendering never re-inspects the raw JSON or // template, table, column or row. Compiling JSON into these once (see compile)
// re-sums weights. // means rendering never re-inspects the raw JSON or re-sums weights.
type node interface{ isNode() } type node interface{ isNode() }
// folder is a namespace of named children, built from a directory of JSON files // folder is a namespace of named children, built from a directory of JSON files
+2 -2
View File
@@ -5,8 +5,8 @@ import (
"strings" "strings"
) )
// rng is the randomness the renderer draws from. Passing it in keeps the render // rng is the randomness a builtin sample draws from; *rand.Rand satisfies it. The
// functions a pure core over an explicit effect; *rand.Rand satisfies it. // render path takes the concrete *session instead, for the reason below.
type rng interface { type rng interface {
IntN(n int) int IntN(n int) int
Float64() float64 Float64() float64
+1 -1
View File
@@ -23,7 +23,7 @@ type table struct {
whole *row // the pinned row rendered by the format whole *row // the pinned row rendered by the format
cells []string // rows × columns, flat cells []string // rows × columns, flat
tokens map[int]*template tokens map[int]*template
key int // column index, or -1 key int // key, name, weight and parent are column indexes, or -1
name int name int
weight int weight int
parent int parent int
+112 -39
View File
@@ -9,6 +9,7 @@ import (
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
"unicode"
) )
// writeFiles writes a data directory from a map of relative file name, extension // writeFiles writes a data directory from a map of relative file name, extension
@@ -663,45 +664,77 @@ func TestTableOverridesByLayering(t *testing.T) {
func TestShippedTables(t *testing.T) { func TestShippedTables(t *testing.T) {
f := newGenerator(t, "data/misc", WithSeed(1)) f := newGenerator(t, "data/misc", WithSeed(1))
for path, want := range map[string]string{ for path, want := range map[string]string{
"territory[SE]": "Sweden", "territory[SE]": "Sweden",
"territory[Sweden].alpha3": "SWE", "territory[Sweden].alpha3": "SWE",
"territory[SE].numeric": "752", "territory[SE].numeric": "752",
"territory[SE].tld": ".se", "territory[SE].tld": ".se",
"territory[SE].calling-code": "46", "territory[SE].calling-code": "46",
"territory[SE].capital": "Stockholm", "territory[SE].capital": "Stockholm",
"territory[SE].currency": "SEK", "territory[SE].currency": "SEK",
"territory[SE].flag": "🇸🇪", "territory[SE].flag": "🇸🇪",
"territory[SE].country": "SE", "territory[SE].country": "SE",
"territory[GL]": "Greenland", "territory[GL]": "Greenland",
"territory[GL].country": "DK", "territory[GL].country": "DK",
"territory[Åland Islands].country": "FI", "territory[Åland Islands].country": "FI",
"currency[SEK].name": "Swedish Krona", "currency[SEK].name": "Swedish Krona",
"currency[SEK].symbol": "kr", "currency[SEK].symbol": "kr",
"currency[SEK].numeric": "752", "currency[SEK].numeric": "752",
"currency[SEK].decimals": "2", "currency[SEK].decimals": "2",
"currency[Euro].code": "EUR", "currency[Euro].code": "EUR",
"language[sv]": "Swedish", "language[sv]": "Swedish",
"language[sv].code3": "swe", "language[sv].code3": "swe",
"language[Swedish].code": "sv", "language[Swedish].code": "sv",
"language[nl]": "Dutch", "language[nl]": "Dutch",
"httpstatus[200]": "200 OK", "httpstatus[200]": "200 OK",
"httpstatus[404]": "404 Not Found", "httpstatus[404]": "404 Not Found",
"httpstatus[404].reason": "Not Found", "httpstatus[404].reason": "Not Found",
"httpstatus[451].reason": "Unavailable For Legal Reasons", "httpstatus[451].reason": "Unavailable For Legal Reasons",
"httpstatus[500].reason": "Internal Server Error", "httpstatus[500].reason": "Internal Server Error",
"mimetype[application/json].ext": ".json", "mimetype[application/json].ext": ".json",
"mimetype[text/markdown].ext": ".md", "mimetype[text/markdown].ext": ".md",
"mimetype[.jpg]": "image/jpeg", "mimetype[.jpg]": "image/jpeg",
"mimetype[.mov]": "video/quicktime", "mimetype[.mov]": "video/quicktime",
"mimetype[.mp3]": "audio/mpeg", "mimetype[.mp3]": "audio/mpeg",
"mimetype[.mp4]": "video/mp4", "mimetype[.mp4]": "video/mp4",
"mimetype[.ogg]": "audio/ogg", "mimetype[.ogg]": "audio/ogg",
"timezone[Europe/Stockholm]": "Europe/Stockholm", "timezone[Europe/Stockholm]": "Europe/Stockholm",
"timezone[Europe/Stockholm].offset": "+01:00", "timezone[Europe/Stockholm].offset": "+01:00",
"timezone[Asia/Kathmandu].offset": "+05:45", "timezone[Asia/Kathmandu].offset": "+05:45",
"timezone[America/New_York].offset": "-05:00", "timezone[America/New_York].offset": "-05:00",
"territory[SE].timezone": "Europe/Stockholm", "territory[SE].timezone": "Europe/Stockholm",
"territory[NP].timezone.offset": "+05:45", "httpmethod[GET].safe": "true",
"httpmethod[GET].idempotent": "true",
"httpmethod[POST].safe": "false",
"httpmethod[POST].idempotent": "false",
"httpmethod[PUT].idempotent": "true",
"protocol[TCP].number": "6",
"protocol[UDP].number": "17",
"protocol[ICMP].name": "Internet Control Message",
"port[443]": "443",
"port[443].service": "https",
"port[22].service": "ssh",
"port[3306].service": "mysql",
"port[465].service": "urd",
"port[2049].service": "nfs",
"protocol[Transmission Control].number": "6",
"tld[.se]": ".se",
"tld[.se].type": "country-code",
"tld[.com].type": "generic",
"tld[.museum].type": "sponsored",
"tld[.arpa].type": "infrastructure",
"tld[.рф]": ".xn--p1ai",
"tld[.xn--p1ai].unicode": ".рф",
"territory[NP].timezone.offset": "+05:45",
"loglevel[0]": "emerg",
"loglevel[1]": "alert",
"loglevel[2]": "crit",
"loglevel[3]": "err",
"loglevel[4]": "warning",
"loglevel[5]": "notice",
"loglevel[6]": "info",
"loglevel[7]": "debug",
"loglevel[err].code": "3",
"loglevel[6].severity": "Informational",
} { } {
if got := fake(t, f, path); got != want { if got := fake(t, f, path); got != want {
t.Errorf("Fake(%q) = %q, want %q", path, got, want) t.Errorf("Fake(%q) = %q, want %q", path, got, want)
@@ -739,6 +772,22 @@ func TestShippedTables(t *testing.T) {
t.Fatalf("%q: a territory and a timezone in one render disagree on the territory", got) t.Fatalf("%q: a territory and a timezone in one render disagree on the territory", got)
} }
} }
var row struct {
Idempotent bool `fake:"misc.httpmethod.idempotent"`
Method string `fake:"misc.httpmethod.method"`
Safe bool `fake:"misc.httpmethod.safe"`
}
if err := newGenerator(t, "data", WithSeed(1)).FakeStruct(&row); err != nil {
t.Fatalf("FakeStruct into a bool = %v, want the register's yes and no shipped as true and false", err)
}
want, named := map[string][2]bool{
"CONNECT": {false, false}, "DELETE": {false, true}, "GET": {true, true},
"HEAD": {true, true}, "OPTIONS": {true, true}, "PATCH": {false, false},
"POST": {false, false}, "PUT": {false, true}, "TRACE": {true, true},
}[row.Method]
if !named || row.Safe != want[0] || row.Idempotent != want[1] {
t.Fatalf("%s drew safe=%v idempotent=%v", row.Method, row.Safe, row.Idempotent)
}
zones := map[string]int{} zones := map[string]int{}
for i := 0; i < 2000; i++ { for i := 0; i < 2000; i++ {
zones[fake(t, f, "territory[US].timezone")]++ zones[fake(t, f, "territory[US].timezone")]++
@@ -799,6 +848,30 @@ func TestEveryTimezoneRowIsWellFormed(t *testing.T) {
} }
} }
// TestEveryTldRowSpellsItsKey scans the file rather than drawing: 100 draws leave
// most of 1438 rows unrendered, and the register wraps a right-to-left label in bidi
// marks that belong to its display, not to the label.
func TestEveryTldRowSpellsItsKey(t *testing.T) {
head, rows := shippedRows(t, "tld.tsv", "tld", "type", "unicode")
types := map[string]bool{"country-code": true, "generic": true, "generic-restricted": true, "infrastructure": true, "sponsored": true}
for i, r := range rows {
key, kind, shown := r[head["tld"]], r[head["type"]], r[head["unicode"]]
if !types[kind] {
t.Errorf("tld.tsv line %d: %s is typed %q, which the register assigns no TLD", i+2, key, kind)
}
for _, c := range shown {
if unicode.Is(unicode.Cf, c) {
t.Errorf("tld.tsv line %d: %s displays %q, carrying the format character %U", i+2, key, shown, c)
}
}
if idn := strings.HasPrefix(key, ".xn--"); idn && shown == key {
t.Errorf("tld.tsv line %d: the IDN %s displays its A-label, not a Unicode form", i+2, key)
} else if !idn && shown != key {
t.Errorf("tld.tsv line %d: the ASCII TLD %s displays %q", i+2, key, shown)
}
}
}
// shippedRows reads a shipped misc TSV as the loader does, proving it names the // shippedRows reads a shipped misc TSV as the loader does, proving it names the
// columns asked for and that every row fills them. // columns asked for and that every row fills them.
func shippedRows(t *testing.T, file string, want ...string) (map[string]int, [][]string) { func shippedRows(t *testing.T, file string, want ...string) (map[string]int, [][]string) {
+19
View File
@@ -229,6 +229,10 @@ misc.currency.numeric string
misc.currency.symbol string misc.currency.symbol string
misc.datetime format "{date(2000-01-01,2029-12-31,'2006-01-02T15:04:05Z')}" misc.datetime format "{date(2000-01-01,2029-12-31,'2006-01-02T15:04:05Z')}"
misc.emoji misc.emoji
misc.httpmethod format "{method}" key method
misc.httpmethod.idempotent string
misc.httpmethod.method string
misc.httpmethod.safe string
misc.httpstatus format "{code} {reason}" key code name reason misc.httpstatus format "{code} {reason}" key code name reason
misc.httpstatus.code string misc.httpstatus.code string
misc.httpstatus.reason string misc.httpstatus.reason string
@@ -236,11 +240,22 @@ misc.language format "{name}" key code name name
misc.language.code string misc.language.code string
misc.language.code3 string misc.language.code3 string
misc.language.name string misc.language.name string
misc.loglevel format "{keyword}" key code name keyword
misc.loglevel.code string
misc.loglevel.keyword string
misc.loglevel.severity string
misc.mac format "{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}" misc.mac format "{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}:{hex(2)}"
misc.mimetype format "{type}" key type name ext misc.mimetype format "{type}" key type name ext
misc.mimetype.ext string misc.mimetype.ext string
misc.mimetype.type string misc.mimetype.type string
misc.objectid format "{hex(24)}" misc.objectid format "{hex(24)}"
misc.port format "{number}" key number
misc.port.number string
misc.port.service string
misc.protocol format "{keyword}" key keyword name name
misc.protocol.keyword string
misc.protocol.name string
misc.protocol.number string
misc.territory format "{name}" key alpha2 name name misc.territory format "{name}" key alpha2 name name
misc.territory.alpha2 string misc.territory.alpha2 string
misc.territory.alpha3 string misc.territory.alpha3 string
@@ -263,6 +278,10 @@ misc.timezone.offset string
misc.timezone.territory string misc.timezone.territory string
misc.timezone.weight string misc.timezone.weight string
misc.timezone.zone string misc.timezone.zone string
misc.tld format "{tld}" key tld name unicode
misc.tld.tld string
misc.tld.type string
misc.tld.unicode string
misc.useragent format "{ua}" misc.useragent format "{ua}"
misc.useragent.browser string misc.useragent.browser string
misc.useragent.device string misc.useragent.device string
+86 -8
View File
@@ -4,12 +4,55 @@ Ordered as the releases that ship it.
## v0.1.0 ## v0.1.0
### Comprehension, until the panel scores 7.0
Nothing else ships while the score is under 7.0 — no feature, no category, no data —
bar a security fix and a dependency bump, per README goal 2. This section is one
round, replayed: the `comprehension-panel` skill's nine-seat findings run at depth 1,
a PR per item it names, then the run again, until the score passes. The run is the
nine-seat one: the round is meant to buy architectural change, and the four-seat
scoring run names too little to steer one. A fresh run replaces the items below.
Scores, newest last: 5.9 on 2026-09-20 (Navigation 7.0, Locality 5.2, Shape 5.9,
Self-sufficiency 5.7), eight of nine seats capped by Locality or Self-sufficiency, all
nine naming `hold.go`, `draw.go` and `family.go` first for reading cost: 1,138 lines in
which each file carries both a load-time fence and render-time machinery, so no file
name says whether a unit runs at `New` or at `Fake`.
This round, in order:
- Define the fence vocabulary where a reader of the code meets it: fence, hold, pin,
draw, draw group, expansion against render, family, whole. It exists only across five
README sections today, and every reader paid for it before the three files read at all.
- Say on the `template` struct which pass fills which field, and that `compileFormat`
runs a second time once `linkTemplateRefs` binds the references: nine of its fields
are written by four other files, and the order lives only in `loadData`'s body.
- State on `rowSet` that it simulates `(*draws).pin` and `rowOf`, and prove the two
agree with a test: the load-time walk reimplements the render-time pinning, nothing
catches them drifting, and the failure is data that loads and then renders a family
that disagrees.
- Put `draws` and its methods in one file: it is declared in `hold.go:262` and all
eight of its methods sit in `family.go`, so a jump to the type lands where none of
the behaviour is.
- Split `hold.go`, `draw.go` and `family.go` so each file holds one phase, the
load-time fences apart from the render-time machinery, changing no logic.
- Move `nodeScope`, `checkScope` and `checkRenders` out of `graph.go` and refresh the
README Layout lines for `graph.go` and `data.go`: the order every load fence runs in
is the load path's spine, and no file name carries it.
- Name the `draw` stem apart: the free `pinned` in `hold.go` is not `(*draws).pinned`
in `family.go`, `compileFormat` is a method on both `*template` and `*table`, and
`draws`, `drawSet`, `drawScope`, `drawWalk`, `drawAt`, `drawKey`, `drawVisit` and
`drawRoute` no longer tell each other apart.
- Spell one fresh draw set one way: `newDrawSet` exists, and `render.go` and `draw.go`
hand-roll `drawSet{unnamed: draws{s: s}}` at three sites; name the lazy-map variant
if the difference is deliberate.
### Data ### Data
Shape: T = table, t = template, c = choice. Read every factual list from a register Shape: T = table, t = template, c = choice. Read every factual list from a register
by a `data-import/` script, as README goal 10 asks. by a `data-import/` script, as README goal 12 asks.
- Add the remaining `misc` tables and templates from the table below. - Add the remaining `misc` tables and templates, one row of the table below per chunk.
- Read `misc.emoji` from the Unicode `emoji-test.txt` register, and `misc.car` from an - Read `misc.emoji` from the Unicode `emoji-test.txt` register, and `misc.car` from an
international make and model source; vPIC and Mobility Sweden are national, so they international make and model source; vPIC and Mobility Sweden are national, so they
build `en_US.car` and `sv_SE.car`. build `en_US.car` and `sv_SE.car`.
@@ -20,13 +63,25 @@ by a `data-import/` script, as README goal 10 asks.
`municipality` as columns on `geo.SE.address` too. `municipality` as columns on `geo.SE.address` too.
- Fill the 398 Swedish localities weighted 200 from SCB småorter. - Fill the 398 Swedish localities weighted 200 from SCB småorter.
- Give `url` and `email` a path that draws only domains nobody can register, keeping - Give `url` and `email` a path that draws only domains nobody can register, keeping
the wide set as the default, per goal 12: 17 of the 40 distinct ones shipped today the wide set as the default, per goal 14: 17 of the 40 distinct ones shipped today
sit on `.se`, `.nu`, `.io` and `.co`, which anyone may register, and only RFC 2606's sit on `.se`, `.nu`, `.io` and `.co`, which anyone may register, and only RFC 2606's
`example.com`, `.net`, `.org`, `.test`, `.example`, `.invalid` and `.localhost` `example.com`, `.net`, `.org`, `.test`, `.example`, `.invalid` and `.localhost`
provably reach nothing. provably reach nothing.
- Audit the rest of the shipped set against goal 12 and give each a never-reaching - Audit the rest of the shipped set against goal 14 and give each a never-reaching
path where it lacks one: `phone` draws live PTS and NANP ranges, `bankgiro`, path where it lacks one: `phone` draws live PTS and NANP ranges, `bankgiro`,
`plusgiro` and `routing` draw live prefixes. `plusgiro` and `routing` draw live prefixes.
- Stop `misc.territory[EH].tld` rendering `.eh`, the one shipped TLD `misc.tld` does
not hold: ISO 3166 reserves it for Western Sahara and the root zone has never been
delegated it, so no resolver answers for it and goal 3 promises otherwise.
- Draw `en_US.phone`'s `exch` as a NANP central office code: `{int(100,999)}` renders
a leading 1 in about an eighth of draws, which libphonenumber rejects, and
`TestShippedUSPhone` proves the shape rather than the rule. Assert the rule with it.
- Give a `sv_SE.personnummer` over 100 the `+` separator Skatteverket spells, or stop
drawing birthdates that reach 100: the format hard-codes `-`, and the 1930 floor
makes the oldest draws invalid from 2031.
- Settle `en_US.ip` and `sv_SE.ip`, today byte-identical: either one `misc.ip` as the
decision "`misc` is what every locale shares" asks, or a decision saying why an
address that carries no locale stays per-locale.
- Accept a middle name, and draw a shipped `personnummer` inside a *selected* sex; - Accept a middle name, and draw a shipped `personnummer` inside a *selected* sex;
both want a draw group sharing its family's pins. Stop the conflict error naming a both want a draw group sharing its family's pins. Stop the conflict error naming a
rewrite that returns a different value where the read it conflicts with sits inside rewrite that returns a different value where the read it conflicts with sits inside
@@ -36,7 +91,7 @@ by a `data-import/` script, as README goal 10 asks.
beside the family: require the path step to reach a sibling category, and seed every beside the family: require the path step to reach a sibling category, and seed every
draw group of the render from the outer selector's pins. draw group of the render from the outer selector's pins.
- Read `email.local`'s and `username`'s handles from the shipped name tables, which - Read `email.local`'s and `username`'s handles from the shipped name tables, which
goal 10 asks for and the hand-written list they use today does not meet. Keep the goal 12 asks for and the hand-written list they use today does not meet. Keep the
handle shape: `username` shortens a surname to `ahl` or `sjo`, and the US one draws handle shape: `username` shortens a surname to `ahl` or `sjo`, and the US one draws
unisex given names, neither of which falls out of the tables unaided. unisex given names, neither of which falls out of the tables unaided.
- Decide whether `email.local` and `username` share one list: a reference between - Decide whether `email.local` and `username` share one list: a reference between
@@ -46,7 +101,6 @@ by a `data-import/` script, as README goal 10 asks.
| Category | Shape | Source | Licence | | Category | Shape | Source | Licence |
|---|---|---|---| |---|---|---|---|
| `httpmethod`, `port`, `tld`, `protocol`, `loglevel` | T/c | IANA | CC0 |
| `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts | | `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts |
| `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts | | `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts |
| `bic`, `isin`, `cusip` | t | structure rules | facts | | `bic`, `isin`, `cusip` | t | structure rules | facts |
@@ -102,8 +156,29 @@ by a `data-import/` script, as README goal 10 asks.
- Add `{base64url(n)}` for JWT shapes. - Add `{base64url(n)}` for JWT shapes.
- Add a `unix` layout to `date()` once something needs it. - Add a `unix` layout to `date()` once something needs it.
### Library and CLI
- Name a spelling that works when a row selector misses: `misc.protocol[tcp]`,
`misc.httpmethod[get]` and `misc.territory[se]` all answer "no row … has key or name"
and stop there, where a case-insensitive match could name the row that exists, and a
table with no `name` column could say it selects by key alone. `misc.tld[se]` misses
on the leading dot its keys carry rather than on case, and `misc.loglevel[Error]`
misses although a column carries `Error`, so the near miss is worth naming whatever
shape it takes.
- Let a table column carry a `datatype`, so `--format json` writes
`"safe": true` and `--format sql` a boolean rather than the text `'true'`. Today only
a JSON field takes one, so `misc.httpmethod`'s booleans are typed in Go and text
everywhere else. Typing a shipped column changes what `json` and `sql` write, so the
capability is a minor and applying it to `misc.httpmethod` is a major.
### Open questions to settle ### Open questions to settle
- Decide where the module is found from, before goal 1's announcement: the import
path is `gitea.larvit.se/larvit/fejkdata`, a developer looking for a fake-data
library searches GitHub and pkg.go.dev, and a module path is a major to change.
- Decide whether `data/misc`'s 33 flat files gain a level before v1.0.0: a folder is a
path segment, so `misc/net/tld` is a rename a consumer pays for, and goal 13 promises
the directory keeps growing.
- Decide whether `misc.browser` becomes a parent of `misc.useragent`, so - Decide whether `misc.browser` becomes a parent of `misc.useragent`, so
`misc.browser[Chrome].useragent` resolves. Adding a `parent` after v0.1.0 breaks a `misc.browser[Chrome].useragent` resolves. Adding a `parent` after v0.1.0 breaks a
consumer, so it rides a 0.(x+1).0. consumer, so it rides a 0.(x+1).0.
@@ -158,8 +233,11 @@ because pairing a street with its exact postnummer rewrites shipped rows.
## v1.0.0 ## v1.0.0
- Cut it once the shipped data is in its record shape and one full minor has shipped - Cut it once the shipped data is in its record shape and one full minor has shipped
with no breaking change, per the README's Versioning table. It plans no work of its with no breaking change, per the README's Versioning table: v0.1.0 settles the
own: v0.1.0 settles the record shape and v0.3.0 is the quiet minor. record shape and v0.3.0 is the quiet minor.
- Announce it where a developer choosing a fake-data tool already reads, which is
goal 1's second half and waits for the tag: complete, stable, installable without
Go. Write the README's first screen for someone deciding in a minute.
## Not release-bound ## Not release-bound