diff --git a/CHANGELOG.md b/CHANGELOG.md index 005dec6..04238bc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -53,6 +53,9 @@ replacement, and each removed path, column or flag. leading dot, `.se` and `.xn--p1ai`, the register's `type` beside it, and a `unicode` column holding the form the register displays, so `misc.tld[.рф]` selects the row `.xn--p1ai` renders. A delegation the register marks "Not assigned" does not ship. +- `misc.loglevel` is RFC 5424's severity table: the eight levels a syslog PRI encodes, + keyed by the numerical `code` and rendering the `severity` the RFC spells, so + `misc.loglevel[3]` and `misc.loglevel[Error]` select one row. - `geo.SE` and `geo.US`: five linked tables per country, `region`, `municipality`, `locality`, `postal-code` and `street`, weighted by population and address counts and built from SCB, GeoNames, Trafikverket NVDB and the US Census Bureau, and an diff --git a/DATA-LICENSES.md b/DATA-LICENSES.md index 033cbb1..26f4bfa 100644 --- a/DATA-LICENSES.md +++ b/DATA-LICENSES.md @@ -19,6 +19,7 @@ Every shipped dataset, its source, its licence and the attribution it asks for. | `misc/httpmethod.tsv` | [IANA HTTP Method Registry](https://www.iana.org/assignments/http-methods/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpmethod.py` | | `misc/httpstatus.tsv` | [IANA HTTP Status Code Registry](https://www.iana.org/assignments/http-status-codes/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/httpstatus.py` | | `misc/language.tsv` | [datasets/language-codes](https://github.com/datasets/language-codes), the [Library of Congress](https://www.loc.gov/standards/iso639-2/) ISO 639-2 register | [PDDL 1.0](https://opendatacommons.org/licenses/pddl/1-0/) | none required | `data-import/language.py` | +| `misc/loglevel.tsv` | [RFC 5424](https://www.rfc-editor.org/rfc/rfc5424.txt) Table 2, the syslog message severities | facts | none required | `data-import/loglevel.py` | | `misc/mimetype.tsv` | [mime-db](https://github.com/jshttp/mime-db), the IANA media type registry with filename extensions | [MIT](https://github.com/jshttp/mime-db/blob/master/LICENSE) | "Copyright (c) 2014 Jonathan Ong, Copyright (c) 2015-2022 Douglas Christopher Wilson" | `data-import/mimetype.py` | | `misc/port.tsv` | [IANA Service Name and Transport Protocol Port Number Registry](https://www.iana.org/assignments/service-names-port-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/port.py` | | `misc/protocol.tsv` | [IANA Protocol Numbers](https://www.iana.org/assignments/protocol-numbers/) | [public domain](https://www.iana.org/help/licensing-terms) | none required | `data-import/protocol.py` | diff --git a/README.md b/README.md index b0756fc..8305d43 100644 --- a/README.md +++ b/README.md @@ -163,14 +163,14 @@ Each locale carries `address`, `color`, `company`, `date`, `email`, `first-name` `personnummer` and `samordningsnummer`, `en_US` adds `ssn` and `itin`. `misc` carries `car`, `coordinate`, `creditcard` (Luhn-valid), `currency` (ISO 4217), `datetime` (RFC 3339), `emoji`, `httpmethod`, `httpstatus`, `language` (ISO 639-1, -with its 639-2/T code), `mac`, `mimetype`, `objectid`, `port`, `protocol`, -`territory` (ISO 3166-1), `timezone` (IANA), `tld` (IANA root zone), `useragent` and -`uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, `misc.territory.alpha2`, -`misc.httpstatus.code` — which `--list` shows. `car`, `currency`, `httpmethod`, -`httpstatus`, `language`, `mimetype`, `port`, `protocol`, `territory`, `timezone`, -`tld` and `useragent` are [tables](#table), so `misc.territory[SE].capital` and -`misc.currency[Euro].symbol` select a row; `car` and `useragent` carry no key or name, -so they are drawn from rather than selected in. +with its 639-2/T code), `loglevel` (RFC 5424), `mac`, `mimetype`, `objectid`, `port`, +`protocol`, `territory` (ISO 3166-1), `timezone` (IANA), `tld` (IANA root zone), +`useragent` and `uuid` (v4). Many carry sub-fields — `misc.currency.symbol`, +`misc.territory.alpha2`, `misc.httpstatus.code` — which `--list` shows. `car`, +`currency`, `httpmethod`, `httpstatus`, `language`, `loglevel`, `mimetype`, `port`, +`protocol`, `territory`, `timezone`, `tld` and `useragent` are [tables](#table), so +`misc.territory[SE].capital` and `misc.currency[Euro].symbol` select a row; `car` and +`useragent` carry no key or name, so they are drawn from rather than selected in. `misc.httpmethod`, `misc.protocol` and `misc.port` are IANA's registries. `misc.httpmethod` is the eight methods RFC 9110 defines and PATCH; the register's @@ -191,6 +191,9 @@ its `unicode` the form the register displays, `.рф` for `.xn--p1ai` and the ke for an ASCII one, which selects the row too, so `misc.tld[.рф]` renders `.xn--p1ai`. A draw spans the whole zone, `.arpa` and `.zuerich` alike, so pin `misc.tld[.com]` where a fixture needs one a reader recognises. +`misc.loglevel` is RFC 5424's eight severities, keyed by the numerical code a syslog +PRI carries and named by the severity, so `misc.loglevel[3]` and `misc.loglevel[Error]` +are one row and `misc.loglevel.code` the number beside the name. [`DATA-LICENSES.md`](DATA-LICENSES.md) names each table's source and licence. `misc.timezone` is every zone tzdb gives a shipped territory, from one apiece for most @@ -1230,6 +1233,11 @@ the Development section below, and who ships a register the four above then draw every value names a row instead. A territory the register records no state for stands alone, which is `EH` alone, and naming one for it would be a claim fejkdata has no business making. +- **`misc.loglevel` is a table keyed by the code, not a list of names.** A syslog + message carries the severity as the number inside its PRI, so the code and the name + are one fact and goal 1 draws them together. The `emerg`, `err` and `info` an + rsyslog config takes are BSD's spellings, which RFC 5424 writes nowhere, so shipping + them would be the hand-written mapping goal 10 keeps out of a sourced table. - **`misc.tld` keys carry the leading dot, where other tables key on a bare code.** The register spells a TLD `.se` and `misc.territory.tld` already ships it so, which a bare key would make two spellings of one fact; `{/misc.tld}` also composes onto a @@ -1339,6 +1347,7 @@ docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/geo-u docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpmethod.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/httpstatus.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/language.py +docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/loglevel.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/mimetype.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-se.py docker compose run --rm --user "$(id -u):$(id -g)" data-import data-import/names-us.py diff --git a/data-import/loglevel.py b/data-import/loglevel.py new file mode 100644 index 0000000..439e067 --- /dev/null +++ b/data-import/loglevel.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Rebuild data/misc/loglevel.tsv from RFC 5424's severity table. + + data-import/loglevel.py [--source URL_OR_FILE] [--cache DIR] [--out FILE] + +Table 2 is the whole set, read between the facility table's caption and its own. The +RFC describes both tables for informational purposes, so a severity is what an +originator writes rather than something a collector rejects. +""" +import argparse +import re +import sys +from pathlib import Path + +import source +import tsv + +SOURCE = "https://www.rfc-editor.org/rfc/rfc5424.txt" +OUT = Path(__file__).resolve().parent.parent / "data" / "misc" / "loglevel.tsv" +CACHE = Path(__file__).resolve().parent / "cache" +COLUMNS = ["code", "severity"] +TABLE = re.compile(r"Table 1\.\s+Syslog Message Facilities(.*?)Table 2\.\s+Syslog Message Severities", re.S) +ROW = re.compile(r"^ +(\d) +([A-Z][a-z]+): ", re.M) +EXPECTED = 8 + + +def rows(text): + table = TABLE.search(text) + if not table: + sys.exit("the RFC holds no text between the facility and the severity caption; its tables have moved") + for code, severity in ROW.findall(table.group(1)): + yield {"code": code, "severity": severity} + + +def main(): + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("--cache", default=str(CACHE)) + p.add_argument("--source", default=SOURCE) + p.add_argument("--out", default=str(OUT)) + a = p.parse_args() + table = sorted(rows(source.fetch(a.source, a.cache, "rfc5424.txt").decode("utf-8")), key=lambda r: r["code"]) + codes = [r["code"] for r in table] + if codes != [str(i) for i in range(EXPECTED)]: + sys.exit(f"the severity table numbers {codes}, not 0 through {EXPECTED - 1}") + tsv.write(a.out, COLUMNS, table) + + +if __name__ == "__main__": + main() diff --git a/data/misc/loglevel.json b/data/misc/loglevel.json new file mode 100644 index 0000000..9e9d3f1 --- /dev/null +++ b/data/misc/loglevel.json @@ -0,0 +1 @@ +{ "format": "{severity}", "rows": "loglevel.tsv", "key": "code", "name": "severity" } diff --git a/data/misc/loglevel.tsv b/data/misc/loglevel.tsv new file mode 100644 index 0000000..8ba335c --- /dev/null +++ b/data/misc/loglevel.tsv @@ -0,0 +1,9 @@ +code severity +0 Emergency +1 Alert +2 Critical +3 Error +4 Warning +5 Notice +6 Informational +7 Debug diff --git a/todo.md b/todo.md index 514a1f8..6db496a 100644 --- a/todo.md +++ b/todo.md @@ -49,7 +49,6 @@ by a `data-import/` script, as README goal 10 asks. | Category | Shape | Source | Licence | |---|---|---|---| -| `loglevel` | c | RFC 5424 severities | facts | | `ip` v4 documentation ranges, private, CIDR; `ipv6` `2001:db8::/32`; `mac` locally administered | t | RFC 5737, 1918, 9637 | facts | | `creditcard` per network: IIN, length, CVV, expiry | T+t | network rules, `{luhn()}` | facts | | `bic`, `isin`, `cusip` | t | structure rules | facts |