mirror of
https://github.com/larvit/adf-codec.git
synced 2026-11-20 07:05:44 +01:00
State the fetch's cost as measured
This commit is contained in:
+1
-1
@@ -1203,7 +1203,7 @@ A site joins only with a published licence. All six are hosted by the Linux Foun
|
|||||||
replaces it in `corpus-gathering/sites.json`.
|
replaces it in `corpus-gathering/sites.json`.
|
||||||
|
|
||||||
The tool stops at 20 result pages of 100 per source, one request a second as the sites' robots.txt
|
The tool stops at 20 result pages of 100 per source, one request a second as the sites' robots.txt
|
||||||
asks, and 150 documents of at most 64 KiB. Those numbers keep a run under two hours and the corpus
|
asks, and 150 documents of at most 64 KiB. Those numbers keep a fetch to minutes and the corpus
|
||||||
near 1 MB. Selection takes new node and mark types first, then the most new features per byte.
|
near 1 MB. Selection takes new node and mark types first, then the most new features per byte.
|
||||||
|
|
||||||
22,769 fetched documents hold no `bodiedSyncBlock`, `decisionItem`, `decisionList`,
|
22,769 fetched documents hold no `bodiedSyncBlock`, `decisionItem`, `decisionList`,
|
||||||
|
|||||||
Reference in New Issue
Block a user