Answer architecture round 5: the shipped packages are enus, svse, geose, geous and misc, each holding its files at its root under its manifest's namespace, and the errors use the wording a bench of the Audience chose

This commit is contained in:
2026-10-11 12:15:41 +02:00
committed by Lillem4n
parent 29a76d5d9a
commit 7b70530a8a
152 changed files with 217 additions and 132 deletions
+1
View File
@@ -47,6 +47,7 @@ In [docs/decisions.md](docs/decisions.md):
- `FakeTemplate` and `NewTemplate` both stay - `FakeTemplate` and `NewTemplate` both stay
- The shipped data is embedded, not discovered - The shipped data is embedded, not discovered
- The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0 - The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
- A shipped package is named by its namespace in lower case without separators, and its manifest mounts it there with `namespace`
- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads - A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads
- Every `{…}` draws afresh, and only a name keeps a pick - Every `{…}` draws afresh, and only a name keeps a pick
- Reference sigils follow the filesystem - Reference sigils follow the filesystem
+6 -5
View File
@@ -126,13 +126,14 @@ replacement, and each removed path, column or flag.
`sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record `sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
that gives it one. that gives it one.
- The shipped data comes in Go packages, one per module, each exporting `FS`: - The shipped data comes in Go packages, one per module, each exporting `FS`:
`github.com/larvit/fejkdata/data/en_US`, `…/sv_SE`, `…/misc`, `…/geo/SE` and `github.com/larvit/fejkdata/data/enus`, `…/svse`, `…/misc`, `…/geose` and
`…/geo/US`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one, `…/geous`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
is given. A manifest's `reads` lists the modules its source reads by default, and a is given. A manifest's `namespace` mounts its source under a dot path, and its `reads` lists the
reference to a path nothing provides fails to load, naming every module the loaded modules its source reads by default; a reference to a path nothing provides fails to
manifests read, or saying there are none. load, asking whether a module is missing and naming every module the loaded manifests
read, or saying there are none.
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each - A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
category on the first call reaching it, as each shipped module does. Data failing to load, category on the first call reaching it, as each shipped module does. Data failing to load,
in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1. in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1.
+9 -9
View File
@@ -6,15 +6,15 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
| Table | Source | Licence | Attribution | Rebuild | | Table | Source | Licence | Attribution | Rebuild |
|-------|--------|---------|-------------|---------| |-------|--------|---------|-------------|---------|
| `geo/SE/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` | | `geose/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` |
| `geo/SE/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` | | `geose/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` |
| `geo/SE/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` | | `geose/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` |
| `geo/US/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` | | `geous/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` |
| `geo/US/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` | | `geous/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` |
| `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` | | `svse/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` |
| `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` | | `enus/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` |
| `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` | | `enus/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — | | `svse/sex.tsv`, `svse/birth-number.tsv`, `svse/title.tsv`, `enus/sex.tsv`, `enus/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — |
| `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` | | `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` |
| `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` | | `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` |
| `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` |
+24 -20
View File
@@ -160,15 +160,14 @@ it to a name and read the name
## Data ## Data
The shipped data comes in modules, each a Go package under [`data/`](data): one per The shipped data comes in modules, each a Go package under [`data/`](data) named by its
locale (`en_US`, `sv_SE`), one per country's tree of places (`geo/SE`, `geo/US`), and namespace in lower case without separators: one per locale (`enus` for `en_US`, `svse`
the locale-neutral `misc`. Each embeds its files, so nothing is read from disk. The CLI for `sv_SE`), one per country's tree of places (`geose` for `geo.SE`, `geous` for
carries every module, and a library loads those it passes to `WithDataFS` `geo.US`), and the locale-neutral `misc`. Each embeds its files, so nothing is read from
([Library](#library)). A disk. The CLI carries every module, and a library loads those it passes to `WithDataFS`
directory is a namespace: each JSON file is a category named after the file, each ([Library](#library)). A directory is a namespace: each JSON file is a category named
subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is `sv_SE.person` and after the file, each subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is
replaces the shipped one. A module's tree spells its namespace the same way, so `sv_SE.person` and replaces the shipped one.
`sv_SE`'s person is `data/sv_SE/sv_SE/person.json`.
Sources merge in order; matching folders combine, any other clash is won by the Sources merge in order; matching folders combine, any other clash is won by the
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `, last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `,
which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed
@@ -185,19 +184,24 @@ every category of its source, so a category file it leaves out never loads.
The index maps each category's dot path to an entry: `parent`, the table it links to, The index maps each category's dot path to an entry: `parent`, the table it links to,
which needs its own entry in the same folder; and `paths`, every path `List` advertises which needs its own entry in the same folder; and `paths`, every path `List` advertises
below the category, `""` for the category itself. Each shipped module's manifest, such as below the category, `""` for the category itself. Each shipped module's manifest, such as
[`data/sv_SE/.fejkdata.json`](data/sv_SE/.fejkdata.json), is one, so delete it from a [`data/svse/.fejkdata.json`](data/svse/.fejkdata.json), is one, so delete it from a
copy you edit, or keep its index current. A table whose parent column differs from its copy you edit, or keep its index current. A table whose parent column differs from its
entry's `parent` fails at first reach, and a table linking under another source's table entry's `parent` fails at first reach, and a table linking under another source's table
needs a source without an index. `//go:embed` of a directory leaves the manifest out, and needs a source without an index. `//go:embed` of a directory leaves the manifest out, and
the source then loads whole in `New`; name it in the pattern too. the source then loads whole in `New`; name it in the pattern too.
A manifest's `namespace` is the dot path its source's root sits under: a folder holding
`person.json` with `"namespace": "sv_SE"` provides `sv_SE.person`, its subfolders adding
segments below that. Each shipped module carries one, so `data/svse/person.json` is
`sv_SE.person`.
A manifest's `reads` lists, by any name its author chooses, every module the source's A manifest's `reads` lists, by any name its author chooses, every module the source's
references read unless another source provides the same paths, directly or through references read unless another source provides the same paths, directly or through
another module; a manifest may carry it without an index. Nothing loads these modules: another module. Nothing loads these modules: pass them yourself. A shipped module names a
pass them yourself. A shipped module names a Go package: `sv_SE`'s manifest reads Go package: `svse`'s manifest reads `github.com/larvit/fejkdata/data/geose`. Data whose
`github.com/larvit/fejkdata/data/geo/SE`. Data whose reference names a path nothing reference names a path nothing provides fails to load, asking whether a module is
provides fails to load, naming every module the loaded manifests read, or saying they missing and naming every module the loaded manifests read, or saying they name none.
name none. `namespace` and `reads` need no index.
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`, Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`, `ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
@@ -366,14 +370,14 @@ only the ones you read, pass each package's `FS` with the modules its manifest r
```go ```go
import ( import (
"github.com/larvit/fejkdata" "github.com/larvit/fejkdata"
"github.com/larvit/fejkdata/data/geo/SE" "github.com/larvit/fejkdata/data/geose"
"github.com/larvit/fejkdata/data/sv_SE" "github.com/larvit/fejkdata/data/svse"
) )
f, err := fejkdata.New(fejkdata.WithDataFS(sv_SE.FS, SE.FS)) f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS))
``` ```
`en_US` reads `geo/US`; `misc` reads none. `New` fails where no `WithDataFS` or `enus` reads `geous`; `misc` reads none. `New` fails where no `WithDataFS` or
`WithDataPath` names the data to load. `WithDataPath` names the data to load.
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid `New` refuses data that breaks the grammar, could mean two things or cannot render a valid
@@ -793,7 +797,7 @@ A reference renders a node from elsewhere in the data — the path `Fake` takes,
across every loaded source — so one category borrows another. The sigil says across every loaded source — so one category borrows another. The sigil says
where the path starts, as in a filesystem: `{/en_US.person}` from the data root, where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
`{.username}` from the folder this file sits in, `{..username}` from the folder `{.username}` from the folder this file sits in, `{..username}` from the folder
above. `data/sv_SE/sv_SE/email.json` can therefore read its own locale's `username` above. `sv_SE.email` can therefore read its own locale's `username`
without naming `sv_SE`: without naming `sv_SE`:
```json ```json
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode). """Rebuild data/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode).
data-import/currency.py [--source URL_OR_FILE] [--symbols URL_OR_FILE ...] [--cache DIR] [--out FILE] data-import/currency.py [--source URL_OR_FILE] [--symbols URL_OR_FILE ...] [--cache DIR] [--out FILE]
@@ -19,7 +19,7 @@ SYMBOLS = [
"https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/en.xml", "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/en.xml",
"https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/root.xml", "https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/root.xml",
] ]
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "currency.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "currency.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["code", "decimals", "name", "numeric", "symbol"] COLUMNS = ["code", "decimals", "name", "numeric", "symbol"]
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/geo/SE/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0). """Rebuild data/geose/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR] TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR]
""" """
+1 -1
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/geo/US/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain). """Rebuild data/geous/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR] data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR]
""" """
+2 -2
View File
@@ -5,14 +5,14 @@ import math
import sys import sys
from pathlib import Path from pathlib import Path
DATA = Path(__file__).resolve().parent.parent / "data" / "geo" DATA = Path(__file__).resolve().parent.parent / "data"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
def parser(doc, country, **own): def parser(doc, country, **own):
"""A parser for the options every country's import takes, plus the script's own: each keyword names one option, with """A parser for the options every country's import takes, plus the script's own: each keyword names one option, with
"_" standing for "-".""" "_" standing for "-"."""
options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / country / "geo" / country)}, "streets_per_locality": {"type": int, "default": 10}, **own} options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / ("geo" + country.lower()))}, "streets_per_locality": {"type": int, "default": 10}, **own}
p = argparse.ArgumentParser(description=doc.splitlines()[0]) p = argparse.ArgumentParser(description=doc.splitlines()[0])
for name in sorted(options): for name in sorted(options):
p.add_argument("--" + name.replace("_", "-"), **options[name]) p.add_argument("--" + name.replace("_", "-"), **options[name])
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/httpmethod.tsv from the IANA HTTP Method Registry. """Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry.
data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -18,7 +18,7 @@ import source
import tsv import tsv
SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv" SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpmethod.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["idempotent", "method", "safe"] COLUMNS = ["idempotent", "method", "safe"]
CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]") CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]")
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/httpstatus.tsv from the IANA HTTP Status Code Registry. """Rebuild data/misc/httpstatus.tsv from the IANA HTTP Status Code Registry.
data-import/httpstatus.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/httpstatus.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -14,7 +14,7 @@ import source
import tsv import tsv
SOURCE = "https://www.iana.org/assignments/http-status-codes/http-status-codes-1.csv" SOURCE = "https://www.iana.org/assignments/http-status-codes/http-status-codes-1.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpstatus.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpstatus.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["code", "reason"] COLUMNS = ["code", "reason"]
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register. """Rebuild data/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register.
data-import/language.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/language.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -15,7 +15,7 @@ import source
import tsv import tsv
SOURCE = "https://raw.githubusercontent.com/datasets/language-codes/main/data/language-codes-full.csv" SOURCE = "https://raw.githubusercontent.com/datasets/language-codes/main/data/language-codes-full.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "language.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "language.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["code", "code3", "name"] COLUMNS = ["code", "code3", "name"]
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>. """Rebuild data/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE] data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -18,7 +18,7 @@ import tsv
SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt" SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt"
POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html" POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "loglevel.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["code", "keyword", "severity"] COLUMNS = ["code", "keyword", "severity"]
TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S) TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S)
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions. """Rebuild data/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions.
data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -13,7 +13,7 @@ import source
import tsv import tsv
SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json" SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "mimetype.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "mimetype.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["ext", "type"] COLUMNS = ["ext", "type"]
FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"} FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"}
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/sv_SE/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB"). """Rebuild data/svse/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR] data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR]
@@ -14,7 +14,7 @@ import tsv
import xlsx import xlsx
SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx" SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx"
OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE" / "sv_SE" OUT = Path(__file__).resolve().parent.parent / "data" / "svse"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"} SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"}
SURNAMES = "Efternamn" SURNAMES = "Efternamn"
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/en_US/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain). """Rebuild data/enus/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR] data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR]
@@ -20,7 +20,7 @@ import tsv
NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt" NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt"
SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip" SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
OUT = Path(__file__).resolve().parent.parent / "data" / "en_US" / "en_US" OUT = Path(__file__).resolve().parent.parent / "data" / "enus"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
MC = re.compile(r"^Mc([a-z])") MC = re.compile(r"^Mc([a-z])")
PLACEHOLDERS = {"Baby", "Female", "Infant", "Male", "Notnamed", "Unknown", "Unnamed"} PLACEHOLDERS = {"Baby", "Female", "Infant", "Male", "Notnamed", "Unknown", "Unnamed"}
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry. """Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -17,7 +17,7 @@ import source
import tsv import tsv
SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv" SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "port.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["number", "service"] COLUMNS = ["number", "service"]
FLOOR = 4000 FLOOR = 4000
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/protocol.tsv from the IANA Protocol Numbers registry. """Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry.
data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -17,7 +17,7 @@ import source
import tsv import tsv
SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv" SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "protocol.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["keyword", "name", "number"] COLUMNS = ["keyword", "name", "number"]
SKIP = ("Reserved", "deprecated") SKIP = ("Reserved", "deprecated")
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/territory.tsv from datasets/country-codes (PDDL). """Rebuild data/misc/territory.tsv from datasets/country-codes (PDDL).
data-import/territory.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] data-import/territory.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
""" """
@@ -14,7 +14,7 @@ import source
import tsv import tsv
SOURCE = "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" SOURCE = "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "tld"] COLUMNS = ["alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "tld"]
SOVEREIGN = re.compile(r"^(?:Part of|Territor(?:y|ies) of|Crown dependency of|Commonwealth of|Associated with) ([A-Z]{2})$") SOVEREIGN = re.compile(r"^(?:Part of|Territor(?:y|ies) of|Crown dependency of|Commonwealth of|Associated with) ([A-Z]{2})$")
+3 -3
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0). """Rebuild data/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0).
data-import/timezone.py [--source URL_OR_FILE] [--cities URL_OR_FILE] [--cache DIR] [--out FILE] [--territories FILE] data-import/timezone.py [--source URL_OR_FILE] [--cities URL_OR_FILE] [--cache DIR] [--out FILE] [--territories FILE]
@@ -22,8 +22,8 @@ import tsv
SOURCE = "https://data.iana.org/time-zones/tzdata-latest.tar.gz" SOURCE = "https://data.iana.org/time-zones/tzdata-latest.tar.gz"
CITIES = "https://download.geonames.org/export/dump/cities15000.zip" CITIES = "https://download.geonames.org/export/dump/cities15000.zip"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "timezone.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "timezone.tsv"
TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv" TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["offset", "territory", "weight", "zone"] COLUMNS = ["offset", "territory", "weight", "zone"]
REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera", "europe", "northamerica", "southamerica"] REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera", "europe", "northamerica", "southamerica"]
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/tld.tsv from the IANA Root Zone Database. """Rebuild data/misc/tld.tsv from the IANA Root Zone Database.
data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE] data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE]
@@ -20,7 +20,7 @@ import tsv
SOURCE = "https://www.iana.org/domains/root/db" SOURCE = "https://www.iana.org/domains/root/db"
ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt" ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "tld.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["tld", "type", "unicode"] COLUMNS = ["tld", "type", "unicode"]
TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored") TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored")
+2 -2
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Rebuild data/misc/misc/useragent.tsv from microlinkhq/top-user-agents (MIT). """Rebuild data/misc/useragent.tsv from microlinkhq/top-user-agents (MIT).
data-import/useragent.py [--source URL] [--cache DIR] [--out FILE] data-import/useragent.py [--source URL] [--cache DIR] [--out FILE]
@@ -16,7 +16,7 @@ import source
import tsv import tsv
SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/src" SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/src"
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "useragent.tsv" OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
CACHE = Path(__file__).resolve().parent / "cache" CACHE = Path(__file__).resolve().parent / "cache"
COLUMNS = ["browser", "device", "os", "ua"] COLUMNS = ["browser", "device", "os", "ua"]
# First match wins: a Chromium fork this list does not name ships as Chrome. # First match wins: a Chromium fork this list does not name ships as Chrome.
+1
View File
@@ -34,6 +34,7 @@ func loadSources(sources []datafiles.Source) (folder, []string, error) {
} }
func addSource(root *folder, src datafiles.Source, m datafiles.Manifest) error { func addSource(root *folder, src datafiles.Source, m datafiles.Manifest) error {
src = src.Mounted(m.Namespace)
if m.Index != nil { if m.Index != nil {
placeIndex(root, src, m.Index) placeIndex(root, src, m.Index)
return nil return nil
-9
View File
@@ -1,9 +0,0 @@
// Package en_US is fejkdata's en_US locale module.
package en_US
import "embed"
// FS holds the en_US categories and the module's manifest; pass it to fejkdata.WithDataFS.
//
//go:embed en_US .fejkdata.json
var FS embed.FS
@@ -23,5 +23,6 @@
"en_US.version": {"paths":["","n","pre","suffix"]}, "en_US.version": {"paths":["","n","pre","suffix"]},
"en_US.word": {"paths":[""]} "en_US.word": {"paths":[""]}
}, },
"reads": ["github.com/larvit/fejkdata/data/geo/US"] "namespace": "en_US",
"reads": ["github.com/larvit/fejkdata/data/geous"]
} }
+9
View File
@@ -0,0 +1,9 @@
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places.
package enus
import "embed"
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
-9
View File
@@ -1,9 +0,0 @@
// Package SE is fejkdata's Swedish geography module.
package SE
import "embed"
// FS holds the geo.SE categories and the module's manifest; pass it to fejkdata.WithDataFS.
//
//go:embed geo .fejkdata.json
var FS embed.FS
-9
View File
@@ -1,9 +0,0 @@
// Package US is fejkdata's US geography module.
package US
import "embed"
// FS holds the geo.US categories and the module's manifest; pass it to fejkdata.WithDataFS.
//
//go:embed geo .fejkdata.json
var FS embed.FS
@@ -6,5 +6,6 @@
"geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]}, "geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]},
"geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]}, "geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
"geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]} "geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]}
} },
"namespace": "geo.SE"
} }
+9
View File
@@ -0,0 +1,9 @@
// Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*.
package geose
import "embed"
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
@@ -6,5 +6,6 @@
"geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]}, "geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]},
"geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]}, "geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
"geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]} "geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]}
} },
"namespace": "geo.US"
} }
+9
View File
@@ -0,0 +1,9 @@
// Package geous is fejkdata's US places module, whose categories are geo.US.*.
package geous
import "embed"
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
//
//go:embed *.json *.tsv .fejkdata.json
var FS embed.FS
+2 -1
View File
@@ -20,5 +20,6 @@
"misc.tld": {"paths":["","tld","type","unicode"]}, "misc.tld": {"paths":["","tld","type","unicode"]},
"misc.useragent": {"paths":["","browser","device","os","ua"]}, "misc.useragent": {"paths":["","browser","device","os","ua"]},
"misc.uuid": {"paths":["","variant"]} "misc.uuid": {"paths":["","variant"]}
} },
"namespace": "misc"
} }
+3 -3
View File
@@ -1,9 +1,9 @@
// Package misc is fejkdata's misc module. // Package misc is fejkdata's locale-neutral module, whose categories are misc.*.
package misc package misc
import "embed" import "embed"
// FS holds the misc categories and the module's manifest; pass it to fejkdata.WithDataFS. // FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
// //
//go:embed misc .fejkdata.json //go:embed *.json *.tsv .fejkdata.json
var FS embed.FS var FS embed.FS

Some files were not shown because too many files have changed in this diff Show More