company_dns

Data package work item: the US SIC file and its division narrative

A brief for an engineer or coding agent who has not seen this project before. You do not need to know anything about earlier versions of the service. Everything you need is in this file and the repository. Facts below were checked on 2026-10-08, after two deliveries of a corrected US data file (the second, us_flat_embedded.feather dated 2026-10-08 06:06, is the current one).

1. Context in five lines

# Item Verdict What you do
1a The old US SIC file had no column for the division narrative text Fixed in the data (new delivery, 2026-10-08) and in the server (see Results) Done
1b The first new file’s narratives for sections A, B, C and G ended with a stray space Fixed in the data (second delivery, 2026-10-08 06:06; validator passes); server trim and divisions.csv fixed Done
2 EDGAR catalog does not contain one company a reference system finds Not a defect: the company is outside the catalog’s stated scope Documented in the plan and the API docs (see Results)
3 Japan SIC descriptions are upper case and there are more of them than a reference system returns Not a defect: the data matches the official source Completeness check done: no gaps (see Results)

2. Vocabulary

3. Item 1: the US SIC division narrative

3.1 Background and what the delivery contains

The API must return a long explanatory paragraph (the narrative) for each US SIC section on GET /V3.0/na/sic/division/{letter}, in the field data.division.{letter}.full_description. The text lives in the repository’s source data, source_data/sic_data/divisions.csv (header Division,Description,Full Description, 10 rows, A to J).

The old us_flat_embedded.feather (1,005 rows, one per 4-digit class) had no column for it, so the server could not answer from the data. As a stopgap, commit 2025b52 added a hand-generated side file that the server reads instead (section 3.4).

The current us_flat_embedded.feather (delivered 2026-10-08 06:06, 1,509,186 bytes) adds one column, section_full_desc (placed right after section_desc; read it by name, not by position), repeated on every row of the section. Verified:

section,full_description_chars
A,2649
B,1901
C,5549
D,4981
E,1970
F,3161
G,3311
H,699
I,588
J,510

The validator passes on this file (PASS, no failures).

3.2 Defect 1b (fixed in the second delivery): stray trailing space in four narratives

History, so the safeguard below makes sense. The first corrected file (dated 2026-10-08 05:42) failed the validator on 179 rows (python3 docs/plans/research/check_ic_feather.py <path>/us_flat_embedded.feather, check “full-width char, NBSP, tab/newline, stray or double spaces in English text”): every row of sections A (58), B (31), C (26) and G (64) had a narrative ending in one space. The space came from the source text (divisions.csv). The second delivery strips it, and the validator now passes.

Wrong (first delivery) and right (current) for the end of each narrative:

section,length_first_delivery,length_current,last_14_chars_first_delivery,last_14_chars_current
A,2650,2649,"ps 01 and 02. |","s 01 and 02.|"
B,1902,1901,"stablishment. |","tablishment.|"
C,5550,5549," restaurants. |","restaurants.|"
G,3312,3311,"ry Group 596. |","y Group 596.|"

(The | marks the end of the string and is not part of the data.)

Tasks for 1b (small, because the data is fixed)

  1. Keep the validator rule as it is; do not relax it.
  2. Keep the server tolerant: when it reads section_full_desc (3.3), apply trim() to the value, so the API output stays clean even if a later delivery regresses. Add a Rust unit test with a value "text. " that expects "text.".
  3. source_data/sic_data/divisions.csv still has the trailing space on the four rows (A, B, C, G). Remove it there too, so the repository’s source matches the delivered data; confirm the other six rows have none.

3.3 Tasks for 1a: use the column in the server

  1. Read the column. In v4/crates/sic/src/systems.rs, change division_narrative (and the System::Us / Level::Section branch that calls it) to take the narrative from the loaded sic_data rows (section_full_desc) instead of the side file. The query that builds section rows is in v4/crates/sic/src/lookup.rs and systems.rs (find_in_system); the table is registered in SicCatalog::open (v4/crates/sic/src/lib.rs).
  2. Fail soft on old files. If a data file has no section_full_desc column, log one warning at startup and return an empty full_description for that file. The server must not refuse to start. Add a test using us_flat.feather (which lacks the column).
  3. Remove the stopgap (3.4): delete v4/crates/sic/data/us_division_narratives.json and the include_str!/OnceLock code, and update the unit test in systems.rs that checks division E’s text.
  4. Add a data-readiness test in api_tests/test_data_readiness.py: GET /V3.0/na/sic/division/{A..J} returns a non-empty full_description with no leading or trailing whitespace, and with these exact lengths after trimming: A 2649, B 1901, C 5549, D 4981, E 1970, F 3161, G 3311, H 699, I 588, J 510.
  5. Validator check for presence. In docs/plans/research/check_ic_feather.py, for ic_module == "us", fail when section_full_desc is missing or empty for any section A to J. (The existing hygiene check already covers whitespace once the column exists.)
  6. Look for the same omission elsewhere. For NACE, ISIC and Japan, list any column or table in source_data/ that the feather files lack and report it in “Results” (do not fix it in this change).
  7. Update docs: the “known intentional differences” paragraph in docs/plans/v4-release-to-staging.md and the non-US paragraph in v4/README.md both mention the narrative; make them say it comes from the data file.

3.4 The stopgap that exists today (you will remove it)

It is not acceptable as the end state: nothing regenerates the JSON, so it can drift from the CSV and from future data deliveries. (It also carries the same four trailing spaces, since it was copied from the CSV.)

3.5 Definition of done

4. Item 2 (not a defect): the EDGAR catalog does not contain “PINEAPPLE, INC.”

Observation. GET /V3.0/na/companies/edgar/ciks/Apple returns 7 companies. The reference system returns 8: the extra one is PINEAPPLE, INC. (CIK 1654672).

Finding (checked 2026-10-08). The catalog file, edgar_10x_catalog.feather (72,813 rows; columns cik, company_name, form_type, year, month, day, accession, url), is built by ingest-edgar (v4/crates/edgar/src/periods.rs) from the SEC’s quarterly filing indexes for 10-K and 10-Q forms in the last eight completed quarters (2024 to 2026 in the current file). It contains no row for CIK 1654672. The SEC’s own record for that company (https://data.sec.gov/submissions/CIK0001654672.json) shows its last 10-Q on 2024-01-12, an NT 10-K on 2024-04-01 and a 15-12G (deregistration) on 2024-05-07, so it has no 10-K or 10-Q in the window. The catalog is working as designed: it lists companies that have recently filed periodic reports. The reference system searched a larger list.

Tasks.

  1. Record this finding in docs/plans/v4-release-to-staging.md as an intentional difference: “the EDGAR catalog covers 10-K and 10-Q filers in a rolling eight-quarter window; companies that stopped filing before the window (for example PINEAPPLE, INC., CIK 1654672) are not returned.”
  2. Add that sentence to the description of the EDGAR ciks operation in the API docs (v4/crates/server/src/main.rs, the utoipa::path for edgar_ciks), so users know the scope.
  3. Do not widen the window or change ingest-edgar unless the owner asks. If the owner does want lapsed filers, the change is to the form-type filter and the window in periods.rs; report the row-count and ingest-time cost first.
  4. In api_tests/test_edgar_wikipedia_parity.py, the ciks test compares only the structure, so no test change is needed.

5. Item 3 (not a defect): Japan SIC descriptions

Observation. GET /V3.0/japan/sic/major_group/09 returns MANUFACTURE OF FOOD; the reference system returns Manufacture of food. GET /V3.0/japan/sic/description/food returns 18 entries here and 13 there.

Finding (checked 2026-10-08). The official source, https://www.soumu.go.jp/english/dgpp_ss/seido/sangyo/san13-3a.htm (Japan Standard Industrial Classification, Rev. 13, October 2013), prints major-group and group names in upper case (09 MANUFACTURE OF FOOD, 091 LIVESTOCK PRODUCTS) and class names in sentence case (0911 Frozen meat and subprimal products). It contains the entries that only this server returns (for example Sozai, Cured food). The reference system was built from a partial extract of that classification. The data here is right. Do not change the case and do not change the stored data.

Task (one check, report only). Compare the descriptions the server holds for Japan (japan_rev13_flat_embedded.feather, 1,460 class rows) with the entries on the source page. List, in this file, any entry on the page that the file lacks, and any file entry whose text differs from the page other than by case. An empty list is a valid result. The author fetched the page once; fetch it with a normal browser User-Agent, no more than once, and cite it rather than copying it into the repository.

6. How to run things

7. Constraints

8. Results (2026-10-08, by the V4 maintainer)

All V4-side work in this brief is done. Nothing here is left for another agent, except what is listed under “Open”.

Item 1a and 1b: US division narrative

Item 2: EDGAR catalog scope

Item 3: Japan completeness (page fetched once, 2026-10-08)

Open

This site is open source. Improve this page.