Answer architecture round 5: the shipped packages are enus, svse, geose, geous and misc, each holding its files at its root under its manifest's namespace, and the errors use the wording a bench of the Audience chose
This commit is contained in:
@@ -47,6 +47,7 @@ In [docs/decisions.md](docs/decisions.md):
|
||||
- `FakeTemplate` and `NewTemplate` both stay
|
||||
- The shipped data is embedded, not discovered
|
||||
- The shipped data is Go packages a library imports by choice, and the CLI carries every one, from 0.1.0
|
||||
- A shipped package is named by its namespace in lower case without separators, and its manifest mounts it there with `namespace`
|
||||
- A source whose manifest carries an index loads each category on first reach; any other loads in `New`, with what it reads
|
||||
- Every `{…}` draws afresh, and only a name keeps a pick
|
||||
- Reference sigils follow the filesystem
|
||||
|
||||
+6
-5
@@ -126,13 +126,14 @@ replacement, and each removed path, column or flag.
|
||||
`sv_SE.sex[f].first-name[Kim]`, and a category with no columns names the record
|
||||
that gives it one.
|
||||
- The shipped data comes in Go packages, one per module, each exporting `FS`:
|
||||
`github.com/larvit/fejkdata/data/en_US`, `…/sv_SE`, `…/misc`, `…/geo/SE` and
|
||||
`…/geo/US`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
|
||||
`github.com/larvit/fejkdata/data/enus`, `…/svse`, `…/misc`, `…/geose` and
|
||||
`…/geous`. `data.Modules()`, from `github.com/larvit/fejkdata/data`, lists every one,
|
||||
and the CLI carries them all. `New` loads the modules `WithDataFS(modules...)` names and
|
||||
the folders `WithDataPath` names, and fails, naming `WithDataFS`, where no such option
|
||||
is given. A manifest's `reads` lists the modules its source reads by default, and a
|
||||
reference to a path nothing provides fails to load, naming every module the loaded
|
||||
manifests read, or saying there are none.
|
||||
is given. A manifest's `namespace` mounts its source under a dot path, and its `reads` lists the
|
||||
modules its source reads by default; a reference to a path nothing provides fails to
|
||||
load, asking whether a module is missing and naming every module the loaded manifests
|
||||
read, or saying there are none.
|
||||
- A data source whose root holds a `.fejkdata.json` manifest with an `index` loads each
|
||||
category on the first call reaching it, as each shipped module does. Data failing to load,
|
||||
in `New` or on that first call, gives an error matching `ErrLoad`, and the CLI exits 1.
|
||||
|
||||
+9
-9
@@ -6,15 +6,15 @@ Every shipped dataset, its source, its licence and the attribution it asks for.
|
||||
|
||||
| Table | Source | Licence | Attribution | Rebuild |
|
||||
|-------|--------|---------|-------------|---------|
|
||||
| `geo/SE/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` |
|
||||
| `geo/SE/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` |
|
||||
| `geo/SE/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` |
|
||||
| `geo/US/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` |
|
||||
| `geo/US/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` |
|
||||
| `sv_SE/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` |
|
||||
| `en_US/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` |
|
||||
| `en_US/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
|
||||
| `sv_SE/sex.tsv`, `sv_SE/birth-number.tsv`, `sv_SE/title.tsv`, `en_US/sex.tsv`, `en_US/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — |
|
||||
| `geose/locality.tsv`, `postal-code.tsv` | [GeoNames](https://www.geonames.org/) postal codes for SE; populations from SCB tätorter 2023 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); CC0 1.0 | "Postal codes from GeoNames, www.geonames.org" | `data-import/geo-se.py` |
|
||||
| `geose/region.tsv`, `municipality.tsv` | [SCB](https://www.scb.se/) län and kommun codes 2026 and population 2024 | [CC0 1.0](https://creativecommons.org/publicdomain/zero/1.0/) | none required | `data-import/geo-se.py` |
|
||||
| `geose/street.tsv` | [Trafikverket NVDB](https://www.trafikverket.se/) Gatunamn, through the open API | CC0 1.0 | none required | `data-import/geo-se.py` |
|
||||
| `geous/region.tsv`, `municipality.tsv`, `locality.tsv` | [Census Bureau](https://www.census.gov/) Gazetteer 2026 and population estimates 2025 | [public domain](https://www.usa.gov/government-works) | none required | `data-import/geo-us.py` |
|
||||
| `geous/postal-code.tsv`, `street.tsv` | Census Bureau ZCTA to place relationships 2020 and TIGER/Line 2025 address ranges and feature names | public domain | none required | `data-import/geo-us.py` |
|
||||
| `svse/first-name.tsv`, `last-name.tsv` | [SCB](https://www.scb.se/) names with at least two bearers, 31 December 2022 | CC0 1.0 | "Källa: SCB" | `data-import/names-se.py` |
|
||||
| `enus/first-name.tsv` | [SSA](https://www.ssa.gov/oact/babynames/) baby names, births 1930 to 2020, through [hackerb9/ssa-baby-names](https://github.com/hackerb9/ssa-baby-names) | public domain | none required | `data-import/names-us.py` |
|
||||
| `enus/last-name.tsv` | Census Bureau surnames occurring 100 or more times, 2010 | public domain | none required | `data-import/names-us.py` |
|
||||
| `svse/sex.tsv`, `svse/birth-number.tsv`, `svse/title.tsv`, `enus/sex.tsv`, `enus/title.tsv`, `misc/car.tsv` | curated; Skatteverket's test birth numbers are facts, and `car.tsv` waits for an international source (goal 14) | — | — | — |
|
||||
| `misc/currency.tsv` | [datasets/currency-codes](https://github.com/datasets/currency-codes); symbols from [Unicode CLDR](https://github.com/unicode-org/cldr) `en.xml` and `root.xml` | PDDL 1.0; [Unicode License v3](https://www.unicode.org/license.txt) | CLDR: "Copyright © 1991-2025 Unicode, Inc. Unicode and the Unicode Logo are registered trademarks of Unicode, Inc. in the United States and other countries." | `data-import/currency.py` |
|
||||
| `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` |
|
||||
| `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` |
|
||||
|
||||
@@ -160,15 +160,14 @@ it to a name and read the name
|
||||
|
||||
## Data
|
||||
|
||||
The shipped data comes in modules, each a Go package under [`data/`](data): one per
|
||||
locale (`en_US`, `sv_SE`), one per country's tree of places (`geo/SE`, `geo/US`), and
|
||||
the locale-neutral `misc`. Each embeds its files, so nothing is read from disk. The CLI
|
||||
carries every module, and a library loads those it passes to `WithDataFS`
|
||||
([Library](#library)). A
|
||||
directory is a namespace: each JSON file is a category named after the file, each
|
||||
subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is `sv_SE.person` and
|
||||
replaces the shipped one. A module's tree spells its namespace the same way, so
|
||||
`sv_SE`'s person is `data/sv_SE/sv_SE/person.json`.
|
||||
The shipped data comes in modules, each a Go package under [`data/`](data) named by its
|
||||
namespace in lower case without separators: one per locale (`enus` for `en_US`, `svse`
|
||||
for `sv_SE`), one per country's tree of places (`geose` for `geo.SE`, `geous` for
|
||||
`geo.US`), and the locale-neutral `misc`. Each embeds its files, so nothing is read from
|
||||
disk. The CLI carries every module, and a library loads those it passes to `WithDataFS`
|
||||
([Library](#library)). A directory is a namespace: each JSON file is a category named
|
||||
after the file, each subdirectory a dot-path segment, so `mydata/sv_SE/person.json` is
|
||||
`sv_SE.person` and replaces the shipped one.
|
||||
Sources merge in order; matching folders combine, any other clash is won by the
|
||||
last loaded. Names may not use `.`, `|`, `(`, `{`, `}`, `[`, `]`, `"`, `/` or ` as `,
|
||||
which binds a [name](#names), nor be `-`, which a struct tag reserves. A dot-prefixed
|
||||
@@ -185,19 +184,24 @@ every category of its source, so a category file it leaves out never loads.
|
||||
The index maps each category's dot path to an entry: `parent`, the table it links to,
|
||||
which needs its own entry in the same folder; and `paths`, every path `List` advertises
|
||||
below the category, `""` for the category itself. Each shipped module's manifest, such as
|
||||
[`data/sv_SE/.fejkdata.json`](data/sv_SE/.fejkdata.json), is one, so delete it from a
|
||||
[`data/svse/.fejkdata.json`](data/svse/.fejkdata.json), is one, so delete it from a
|
||||
copy you edit, or keep its index current. A table whose parent column differs from its
|
||||
entry's `parent` fails at first reach, and a table linking under another source's table
|
||||
needs a source without an index. `//go:embed` of a directory leaves the manifest out, and
|
||||
the source then loads whole in `New`; name it in the pattern too.
|
||||
|
||||
A manifest's `namespace` is the dot path its source's root sits under: a folder holding
|
||||
`person.json` with `"namespace": "sv_SE"` provides `sv_SE.person`, its subfolders adding
|
||||
segments below that. Each shipped module carries one, so `data/svse/person.json` is
|
||||
`sv_SE.person`.
|
||||
|
||||
A manifest's `reads` lists, by any name its author chooses, every module the source's
|
||||
references read unless another source provides the same paths, directly or through
|
||||
another module; a manifest may carry it without an index. Nothing loads these modules:
|
||||
pass them yourself. A shipped module names a Go package: `sv_SE`'s manifest reads
|
||||
`github.com/larvit/fejkdata/data/geo/SE`. Data whose reference names a path nothing
|
||||
provides fails to load, naming every module the loaded manifests read, or saying they
|
||||
name none.
|
||||
another module. Nothing loads these modules: pass them yourself. A shipped module names a
|
||||
Go package: `svse`'s manifest reads `github.com/larvit/fejkdata/data/geose`. Data whose
|
||||
reference names a path nothing provides fails to load, asking whether a module is
|
||||
missing and naming every module the loaded manifests read, or saying they name none.
|
||||
`namespace` and `reads` need no index.
|
||||
|
||||
Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name`,
|
||||
`ip`, `last-name`, `person`, `phone`, `price`, `sentence`, `sex`, `time`, `url`,
|
||||
@@ -366,14 +370,14 @@ only the ones you read, pass each package's `FS` with the modules its manifest r
|
||||
```go
|
||||
import (
|
||||
"github.com/larvit/fejkdata"
|
||||
"github.com/larvit/fejkdata/data/geo/SE"
|
||||
"github.com/larvit/fejkdata/data/sv_SE"
|
||||
"github.com/larvit/fejkdata/data/geose"
|
||||
"github.com/larvit/fejkdata/data/svse"
|
||||
)
|
||||
|
||||
f, err := fejkdata.New(fejkdata.WithDataFS(sv_SE.FS, SE.FS))
|
||||
f, err := fejkdata.New(fejkdata.WithDataFS(svse.FS, geose.FS))
|
||||
```
|
||||
|
||||
`en_US` reads `geo/US`; `misc` reads none. `New` fails where no `WithDataFS` or
|
||||
`enus` reads `geous`; `misc` reads none. `New` fails where no `WithDataFS` or
|
||||
`WithDataPath` names the data to load.
|
||||
|
||||
`New` refuses data that breaks the grammar, could mean two things or cannot render a valid
|
||||
@@ -793,7 +797,7 @@ A reference renders a node from elsewhere in the data — the path `Fake` takes,
|
||||
across every loaded source — so one category borrows another. The sigil says
|
||||
where the path starts, as in a filesystem: `{/en_US.person}` from the data root,
|
||||
`{.username}` from the folder this file sits in, `{..username}` from the folder
|
||||
above. `data/sv_SE/sv_SE/email.json` can therefore read its own locale's `username`
|
||||
above. `sv_SE.email` can therefore read its own locale's `username`
|
||||
without naming `sv_SE`:
|
||||
|
||||
```json
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode).
|
||||
"""Rebuild data/misc/currency.tsv from datasets/currency-codes (PDDL) and CLDR's symbols (Unicode).
|
||||
|
||||
data-import/currency.py [--source URL_OR_FILE] [--symbols URL_OR_FILE ...] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -19,7 +19,7 @@ SYMBOLS = [
|
||||
"https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/en.xml",
|
||||
"https://raw.githubusercontent.com/unicode-org/cldr/main/common/main/root.xml",
|
||||
]
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "currency.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "currency.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["code", "decimals", "name", "numeric", "symbol"]
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/geo/SE/geo/SE/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
|
||||
"""Rebuild data/geose/*.tsv from SCB (CC0), GeoNames (CC BY 4.0) and Trafikverket NVDB (CC0).
|
||||
|
||||
TRAFIKVERKET_API_KEY=… data-import/geo-se.py [--key-file FILE] [--cache DIR] [--streets-per-locality N] [--out DIR]
|
||||
"""
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/geo/US/geo/US/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
|
||||
"""Rebuild data/geous/*.tsv from the Census Bureau's Gazetteer, population estimates, ZCTA relationships and TIGER/Line files (public domain).
|
||||
|
||||
data-import/geo-us.py [--cache DIR] [--min-population N] [--streets-per-locality N] [--out DIR]
|
||||
"""
|
||||
|
||||
+2
-2
@@ -5,14 +5,14 @@ import math
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
DATA = Path(__file__).resolve().parent.parent / "data" / "geo"
|
||||
DATA = Path(__file__).resolve().parent.parent / "data"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
|
||||
|
||||
def parser(doc, country, **own):
|
||||
"""A parser for the options every country's import takes, plus the script's own: each keyword names one option, with
|
||||
"_" standing for "-"."""
|
||||
options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / country / "geo" / country)}, "streets_per_locality": {"type": int, "default": 10}, **own}
|
||||
options = {"cache": {"default": str(CACHE)}, "out": {"default": str(DATA / ("geo" + country.lower()))}, "streets_per_locality": {"type": int, "default": 10}, **own}
|
||||
p = argparse.ArgumentParser(description=doc.splitlines()[0])
|
||||
for name in sorted(options):
|
||||
p.add_argument("--" + name.replace("_", "-"), **options[name])
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/httpmethod.tsv from the IANA HTTP Method Registry.
|
||||
"""Rebuild data/misc/httpmethod.tsv from the IANA HTTP Method Registry.
|
||||
|
||||
data-import/httpmethod.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -18,7 +18,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://www.iana.org/assignments/http-methods/methods.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpmethod.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpmethod.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["idempotent", "method", "safe"]
|
||||
CORE = re.compile(r"\[RFC9110, Section 9\.3\.|\[RFC5789, Section 2\]")
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/httpstatus.tsv from the IANA HTTP Status Code Registry.
|
||||
"""Rebuild data/misc/httpstatus.tsv from the IANA HTTP Status Code Registry.
|
||||
|
||||
data-import/httpstatus.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -14,7 +14,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://www.iana.org/assignments/http-status-codes/http-status-codes-1.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "httpstatus.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "httpstatus.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["code", "reason"]
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register.
|
||||
"""Rebuild data/misc/language.tsv from datasets/language-codes (PDDL), the LoC ISO 639-2 register.
|
||||
|
||||
data-import/language.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -15,7 +15,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://raw.githubusercontent.com/datasets/language-codes/main/data/language-codes-full.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "language.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "language.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["code", "code3", "name"]
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
|
||||
"""Rebuild data/misc/loglevel.tsv from RFC 5424 and POSIX <syslog.h>.
|
||||
|
||||
data-import/loglevel.py [--source URL_OR_FILE] [--posix URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -18,7 +18,7 @@ import tsv
|
||||
|
||||
SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt"
|
||||
POSIX = "https://pubs.opengroup.org/onlinepubs/9799919799/basedefs/syslog.h.html"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "loglevel.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["code", "keyword", "severity"]
|
||||
TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions.
|
||||
"""Rebuild data/misc/mimetype.tsv from mime-db (MIT), the IANA media type registry with extensions.
|
||||
|
||||
data-import/mimetype.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -13,7 +13,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://raw.githubusercontent.com/jshttp/mime-db/master/db.json"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "mimetype.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "mimetype.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["ext", "type"]
|
||||
FIXUPS = {"application/mp4": "mp4s", "audio/mpeg": "mp3", "audio/ogg": "ogg", "video/quicktime": "mov"}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/sv_SE/sv_SE/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
|
||||
"""Rebuild data/svse/first-name.tsv and last-name.tsv from SCB's whole-population name counts (CC0, "Källa: SCB").
|
||||
|
||||
data-import/names-se.py [--source URL_OR_FILE] [--first N] [--last N] [--cache DIR] [--out DIR]
|
||||
|
||||
@@ -14,7 +14,7 @@ import tsv
|
||||
import xlsx
|
||||
|
||||
SOURCE = "https://www.scb.se/contentassets/9fe7dbb460994c72b835163dbc491ef9/namn-med-minst-tva-barare-31-december-2022.xlsx"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "sv_SE" / "sv_SE"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "svse"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
SHEETS = {"f": "Tilltalsnamn kvinnor", "m": "Tilltalsnamn män"}
|
||||
SURNAMES = "Efternamn"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/en_US/en_US/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
|
||||
"""Rebuild data/enus/first-name.tsv and last-name.tsv from SSA baby names and the Census 2010 surnames (public domain).
|
||||
|
||||
data-import/names-us.py [--names URL_OR_FILE] [--surnames URL_OR_FILE] [--from-year YEAR] [--first N] [--last N] [--cache DIR] [--out DIR]
|
||||
|
||||
@@ -20,7 +20,7 @@ import tsv
|
||||
|
||||
NAMES = "https://raw.githubusercontent.com/hackerb9/ssa-baby-names/master/alldata.txt"
|
||||
SURNAMES = "https://www2.census.gov/topics/genealogy/2010surnames/names.zip"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "en_US" / "en_US"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "enus"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
MC = re.compile(r"^Mc([a-z])")
|
||||
PLACEHOLDERS = {"Baby", "Female", "Infant", "Male", "Notnamed", "Unknown", "Unnamed"}
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
|
||||
"""Rebuild data/misc/port.tsv from the IANA Service Name and Transport Protocol Port Number Registry.
|
||||
|
||||
data-import/port.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -17,7 +17,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "port.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "port.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["number", "service"]
|
||||
FLOOR = 4000
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/protocol.tsv from the IANA Protocol Numbers registry.
|
||||
"""Rebuild data/misc/protocol.tsv from the IANA Protocol Numbers registry.
|
||||
|
||||
data-import/protocol.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -17,7 +17,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://www.iana.org/assignments/protocol-numbers/protocol-numbers-1.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "protocol.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "protocol.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["keyword", "name", "number"]
|
||||
SKIP = ("Reserved", "deprecated")
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/territory.tsv from datasets/country-codes (PDDL).
|
||||
"""Rebuild data/misc/territory.tsv from datasets/country-codes (PDDL).
|
||||
|
||||
data-import/territory.py [--source URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
"""
|
||||
@@ -14,7 +14,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["alpha2", "alpha3", "calling-code", "capital", "country", "currency", "flag", "languages", "name", "numeric", "tld"]
|
||||
SOVEREIGN = re.compile(r"^(?:Part of|Territor(?:y|ies) of|Crown dependency of|Commonwealth of|Associated with) ([A-Z]{2})$")
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0).
|
||||
"""Rebuild data/misc/timezone.tsv from the IANA tzdb tarball (public domain) and GeoNames (CC BY 4.0).
|
||||
|
||||
data-import/timezone.py [--source URL_OR_FILE] [--cities URL_OR_FILE] [--cache DIR] [--out FILE] [--territories FILE]
|
||||
|
||||
@@ -22,8 +22,8 @@ import tsv
|
||||
|
||||
SOURCE = "https://data.iana.org/time-zones/tzdata-latest.tar.gz"
|
||||
CITIES = "https://download.geonames.org/export/dump/cities15000.zip"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "timezone.tsv"
|
||||
TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "territory.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "timezone.tsv"
|
||||
TERRITORIES = Path(__file__).resolve().parent.parent / "data" / "misc" / "territory.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["offset", "territory", "weight", "zone"]
|
||||
REGIONS = ["africa", "antarctica", "asia", "australasia", "backward", "etcetera", "europe", "northamerica", "southamerica"]
|
||||
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/tld.tsv from the IANA Root Zone Database.
|
||||
"""Rebuild data/misc/tld.tsv from the IANA Root Zone Database.
|
||||
|
||||
data-import/tld.py [--source URL_OR_FILE] [--zone URL_OR_FILE] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -20,7 +20,7 @@ import tsv
|
||||
|
||||
SOURCE = "https://www.iana.org/domains/root/db"
|
||||
ZONE = "https://data.iana.org/TLD/tlds-alpha-by-domain.txt"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "tld.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "tld.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["tld", "type", "unicode"]
|
||||
TYPES = ("country-code", "generic", "generic-restricted", "infrastructure", "sponsored")
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Rebuild data/misc/misc/useragent.tsv from microlinkhq/top-user-agents (MIT).
|
||||
"""Rebuild data/misc/useragent.tsv from microlinkhq/top-user-agents (MIT).
|
||||
|
||||
data-import/useragent.py [--source URL] [--cache DIR] [--out FILE]
|
||||
|
||||
@@ -16,7 +16,7 @@ import source
|
||||
import tsv
|
||||
|
||||
SOURCE = "https://raw.githubusercontent.com/microlinkhq/top-user-agents/master/src"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "misc" / "useragent.tsv"
|
||||
OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "useragent.tsv"
|
||||
CACHE = Path(__file__).resolve().parent / "cache"
|
||||
COLUMNS = ["browser", "device", "os", "ua"]
|
||||
# First match wins: a Chromium fork this list does not name ships as Chrome.
|
||||
|
||||
@@ -34,6 +34,7 @@ func loadSources(sources []datafiles.Source) (folder, []string, error) {
|
||||
}
|
||||
|
||||
func addSource(root *folder, src datafiles.Source, m datafiles.Manifest) error {
|
||||
src = src.Mounted(m.Namespace)
|
||||
if m.Index != nil {
|
||||
placeIndex(root, src, m.Index)
|
||||
return nil
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
// Package en_US is fejkdata's en_US locale module.
|
||||
package en_US
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the en_US categories and the module's manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed en_US .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -23,5 +23,6 @@
|
||||
"en_US.version": {"paths":["","n","pre","suffix"]},
|
||||
"en_US.word": {"paths":[""]}
|
||||
},
|
||||
"reads": ["github.com/larvit/fejkdata/data/geo/US"]
|
||||
"namespace": "en_US",
|
||||
"reads": ["github.com/larvit/fejkdata/data/geous"]
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
// Package enus is fejkdata's en_US locale module, whose categories are en_US.*. Its address reads geous's places.
|
||||
package enus
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -1,9 +0,0 @@
|
||||
// Package SE is fejkdata's Swedish geography module.
|
||||
package SE
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the geo.SE categories and the module's manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed geo .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -1,9 +0,0 @@
|
||||
// Package US is fejkdata's US geography module.
|
||||
package US
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the geo.US categories and the module's manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed geo .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -6,5 +6,6 @@
|
||||
"geo.SE.postal-code": {"parent":"locality","paths":["","code","locality"]},
|
||||
"geo.SE.region": {"paths":["","code","municipality","municipality.code","municipality.locality","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.locality","municipality.locality.street.name","municipality.locality.street.segments","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
|
||||
"geo.SE.street": {"parent":"locality","paths":["","locality","name","segments"]}
|
||||
}
|
||||
},
|
||||
"namespace": "geo.SE"
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
// Package geose is fejkdata's Swedish places module, whose categories are geo.SE.*.
|
||||
package geose
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -6,5 +6,6 @@
|
||||
"geo.US.postal-code": {"parent":"locality","paths":["","addresses","code","locality"]},
|
||||
"geo.US.region": {"paths":["","abbr","code","municipality","municipality.code","municipality.locality","municipality.locality.code","municipality.locality.lat","municipality.locality.lon","municipality.locality.municipality","municipality.locality.name","municipality.locality.population","municipality.locality.postal-code","municipality.locality.postal-code.addresses","municipality.locality.postal-code.code","municipality.locality.postal-code.locality","municipality.locality.street","municipality.locality.street.addresses","municipality.locality.street.locality","municipality.locality.street.name","municipality.name","municipality.population","municipality.region","name","population","timezone"]},
|
||||
"geo.US.street": {"parent":"locality","paths":["","addresses","locality","name"]}
|
||||
}
|
||||
},
|
||||
"namespace": "geo.US"
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
// Package geous is fejkdata's US places module, whose categories are geo.US.*.
|
||||
package geous
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
@@ -20,5 +20,6 @@
|
||||
"misc.tld": {"paths":["","tld","type","unicode"]},
|
||||
"misc.useragent": {"paths":["","browser","device","os","ua"]},
|
||||
"misc.uuid": {"paths":["","variant"]}
|
||||
}
|
||||
},
|
||||
"namespace": "misc"
|
||||
}
|
||||
|
||||
+3
-3
@@ -1,9 +1,9 @@
|
||||
// Package misc is fejkdata's misc module.
|
||||
// Package misc is fejkdata's locale-neutral module, whose categories are misc.*.
|
||||
package misc
|
||||
|
||||
import "embed"
|
||||
|
||||
// FS holds the misc categories and the module's manifest; pass it to fejkdata.WithDataFS.
|
||||
// FS holds the module's categories and its manifest; pass it to fejkdata.WithDataFS.
|
||||
//
|
||||
//go:embed misc .fejkdata.json
|
||||
//go:embed *.json *.tsv .fejkdata.json
|
||||
var FS embed.FS
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user