READER TRIAL: COUNTS BY ARM (n = 24 reads of 12 notes, 4 reviewers; 24 of 24 reads in the key scored)
EXAMPLE DATA: made-up scores that show the analysis runs; these are not trial results (n = 24 reads of 12 notes, 4 reviewers)
Scope: inputs A, B; arms plain, pack, hybrid; models EXAMPLE-model; hosts EXAMPLE-host (n = 24 reads of 12 notes, 4 reviewers)
Reference movements: A 12, B 12 (n = 24 items, frozen before the runs)

1. Omissions: reference movements a note did not cover (n = 24 reads of 12 notes, 4 reviewers)
  plain: 16 of 96 item-reads omitted; 8 of 8 reads omitted at least one (n = 8 reads of 4 notes, 4 reviewers)
  pack: 17 of 96 item-reads omitted; 6 of 8 reads omitted at least one (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 13 of 96 item-reads omitted; 8 of 8 reads omitted at least one (n = 8 reads of 4 notes, 4 reviewers)

2. Statements the data does not support (n = 24 reads of 12 notes, 4 reviewers)
  plain: 6 statements; 5 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)
  pack: 9 statements; 8 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 6 statements; 4 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)

3. Wrong figures (n = 24 reads of 12 notes, 4 reviewers)
  plain: 3 wrong figures; 2 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)
  pack: 9 wrong figures; 4 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 7 wrong figures; 5 of 8 reads had at least one (n = 8 reads of 4 notes, 4 reviewers)

4. Reads with an omission, an unsupported statement or a wrong figure (n = 24 reads of 12 notes, 4 reviewers)
  plain: 8 of 8 reads (n = 8 reads of 4 notes, 4 reviewers)
  pack: 8 of 8 reads (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 8 of 8 reads (n = 8 reads of 4 notes, 4 reviewers)

5. Review time (n = 24 reads of 12 notes, 4 reviewers)
  plain: median 18.5 minutes, range 10 to 23, total 137 (n = 8 reads of 4 notes, 4 reviewers)
  pack: median 23 minutes, range 11 to 27, total 163 (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: median 17.5 minutes, range 11 to 23, total 136 (n = 8 reads of 4 notes, 4 reviewers)

6. Would the reviewer send it? (n = 24 reads of 12 notes, 4 reviewers)
  plain: as is 0 of 8, with edits 4 of 8, not at all 4 of 8 (n = 8 reads of 4 notes, 4 reviewers)
  pack: as is 0 of 8, with edits 2 of 8, not at all 6 of 8 (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: as is 0 of 8, with edits 5 of 8, not at all 3 of 8 (n = 8 reads of 4 notes, 4 reviewers)

7. The pack's own checks against what reviewers recorded (n = 24 reads of 12 notes, 4 reviewers)
  plain: no pack checks ran on these notes (n = 8 reads of 4 notes, 4 reviewers)
  pack: 6 of 8 reads were of notes that passed every check; in 6 of 6 of those a reviewer recorded an omission, an unsupported statement or a wrong figure (n = 8 reads of 4 notes, 4 reviewers)
  pack: 2 of 8 reads were of notes that failed a check (binding 2); in 0 of 2 of those a reviewer recorded a wrong figure (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 6 of 8 reads were of notes that passed every check; in 6 of 6 of those a reviewer recorded an omission, an unsupported statement or a wrong figure (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: 2 of 8 reads were of notes that failed a check (contract 2); in 0 of 2 of those a reviewer recorded a wrong figure (n = 8 reads of 4 notes, 4 reviewers)

8. By input (n = 24 reads of 12 notes, 4 reviewers)
  A plain: omitted 5 of 48 item-reads; unsupported 4; wrong figures 3; median 17 minutes, range 11 to 22, total 67; as is 0, with edits 2, not at all 2 (n = 4 reads of 2 notes, 4 reviewers)
  A pack: omitted 13 of 48 item-reads; unsupported 5; wrong figures 4; median 24.5 minutes, range 17 to 25, total 91; as is 0, with edits 0, not at all 4 (n = 4 reads of 2 notes, 4 reviewers)
  A hybrid: omitted 7 of 48 item-reads; unsupported 1; wrong figures 6; median 16.5 minutes, range 14 to 18, total 65; as is 0, with edits 3, not at all 1 (n = 4 reads of 2 notes, 4 reviewers)
  B plain: omitted 11 of 48 item-reads; unsupported 2; wrong figures 0; median 18.5 minutes, range 10 to 23, total 70; as is 0, with edits 2, not at all 2 (n = 4 reads of 2 notes, 4 reviewers)
  B pack: omitted 4 of 48 item-reads; unsupported 4; wrong figures 5; median 17 minutes, range 11 to 27, total 72; as is 0, with edits 2, not at all 2 (n = 4 reads of 2 notes, 4 reviewers)
  B hybrid: omitted 6 of 48 item-reads; unsupported 5; wrong figures 1; median 18.5 minutes, range 11 to 23, total 71; as is 0, with edits 2, not at all 2 (n = 4 reads of 2 notes, 4 reviewers)

9. Omissions by reference item, largest movement first (n = 24 reads of 12 notes, 4 reviewers)
  A-01 Revenue, year to date (215,850.80): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-04 Revenue in July 2023 (91,478.80): omitted in plain 0 of 4, pack 0 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-05 Revenue in September 2023 (83,446.85): omitted in plain 0 of 4, pack 2 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-03 Revenue in January 2023 (63,364.08): omitted in plain 4 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-07 Salaries and wages (58,448.96): omitted in plain 0 of 4, pack 4 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-06 Gross margin (43,415.98): omitted in plain 1 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-08 Technology (31,876.06): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-12 Marketing, timing (25,542.48): omitted in plain 0 of 4, pack 4 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-09 Other operating (18,801.44): omitted in plain 0 of 4, pack 1 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-02 Operating income, year to date (16,690.26): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-10 Professional fees (16,015.68): omitted in plain 0 of 4, pack 0 of 4, hybrid 1 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  A-11 Freight out (14,758.52): omitted in plain 0 of 4, pack 2 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-01 Revenue, year to date (143,550.52): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-02 Operating income, year to date (96,386.58): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-04 Revenue in March 2022 (85,058.16): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-05 Revenue in July 2022 (71,148.60): omitted in plain 0 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-06 Gross margin (40,461.72): omitted in plain 0 of 4, pack 0 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-12 Rent and occupancy, timing (30,921.63): omitted in plain 1 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-07 Salaries and wages (27,930.36): omitted in plain 2 of 4, pack 2 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-08 Freight out (16,147.17): omitted in plain 2 of 4, pack 2 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-09 Technology (12,471.38): omitted in plain 2 of 4, pack 0 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-03 August 2022 on its own (12,326.11): omitted in plain 2 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-10 Other operating (11,616.44): omitted in plain 0 of 4, pack 0 of 4, hybrid 2 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)
  B-11 Professional fees (9,664.48): omitted in plain 2 of 4, pack 0 of 4, hybrid 0 of 4 reads (n = 12 reads of 6 notes, 4 reviewers)

10. Paired: the same reviewer reading two arms on the same input (n = 24 reads of 12 notes, 4 reviewers)
  pack against plain, problems (omissions + unsupported + wrong figures): fewer in 2 of 8, the same in 1 of 8, more in 5 of 8 pairs (n = 8 pairs, 4 reviewers)
  pack against plain, minutes: less in 2 of 8, the same in 0 of 8, more in 6 of 8 pairs (n = 8 pairs, 4 reviewers)
  pack against plain, decision: nearer to send as is in 1 of 8, the same in 4 of 8, further in 3 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against plain, problems (omissions + unsupported + wrong figures): fewer in 3 of 8, the same in 2 of 8, more in 3 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against plain, minutes: less in 4 of 8, the same in 0 of 8, more in 4 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against plain, decision: nearer to send as is in 2 of 8, the same in 5 of 8, further in 1 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against pack, problems (omissions + unsupported + wrong figures): fewer in 6 of 8, the same in 0 of 8, more in 2 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against pack, minutes: less in 5 of 8, the same in 0 of 8, more in 3 of 8 pairs (n = 8 pairs, 4 reviewers)
  hybrid against pack, decision: nearer to send as is in 4 of 8, the same in 3 of 8, further in 1 of 8 pairs (n = 8 pairs, 4 reviewers)

11. By reviewer (n = 24 reads of 12 notes, 4 reviewers)
  R1 plain: omitted 5 of 24 item-reads; unsupported 1; wrong figures 1; minutes 40; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R1 pack: omitted 5 of 24 item-reads; unsupported 3; wrong figures 5; minutes 52; as is 0, with edits 0, not at all 2 (n = 2 reads)
  R1 hybrid: omitted 2 of 24 item-reads; unsupported 0; wrong figures 2; minutes 26; as is 0, with edits 2, not at all 0 (n = 2 reads)
  R2 plain: omitted 4 of 24 item-reads; unsupported 2; wrong figures 2; minutes 37; as is 0, with edits 0, not at all 2 (n = 2 reads)
  R2 pack: omitted 5 of 24 item-reads; unsupported 2; wrong figures 2; minutes 39; as is 0, with edits 0, not at all 2 (n = 2 reads)
  R2 hybrid: omitted 4 of 24 item-reads; unsupported 2; wrong figures 1; minutes 38; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R3 plain: omitted 4 of 24 item-reads; unsupported 1; wrong figures 0; minutes 28; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R3 pack: omitted 3 of 24 item-reads; unsupported 2; wrong figures 0; minutes 36; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R3 hybrid: omitted 3 of 24 item-reads; unsupported 2; wrong figures 1; minutes 37; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R4 plain: omitted 3 of 24 item-reads; unsupported 2; wrong figures 0; minutes 32; as is 0, with edits 2, not at all 0 (n = 2 reads)
  R4 pack: omitted 4 of 24 item-reads; unsupported 2; wrong figures 2; minutes 36; as is 0, with edits 1, not at all 1 (n = 2 reads)
  R4 hybrid: omitted 4 of 24 item-reads; unsupported 2; wrong figures 3; minutes 35; as is 0, with edits 1, not at all 1 (n = 2 reads)

12. Reading order: the reviewer's first, second, third read of the same input (n = 24 reads of 12 notes, 4 reviewers)
  read 1 of its input, plain: median 20 minutes, range 17 to 22, total 59 (n = 3 reads of 3 notes, 3 reviewers)
  read 1 of its input, pack: median 23 minutes, range 22 to 24, total 46 (n = 2 reads of 2 notes, 2 reviewers)
  read 1 of its input, hybrid: median 17 minutes, range 11 to 18, total 46 (n = 3 reads of 2 notes, 3 reviewers)
  read 2 of its input, plain: median 17 minutes, range 11 to 23, total 34 (n = 2 reads of 2 notes, 2 reviewers)
  read 2 of its input, pack: median 17 minutes, range 11 to 27, total 55 (n = 3 reads of 3 notes, 3 reviewers)
  read 2 of its input, hybrid: median 18 minutes, range 15 to 23, total 56 (n = 3 reads of 3 notes, 3 reviewers)
  read 3 of its input, plain: median 14 minutes, range 10 to 20, total 44 (n = 3 reads of 2 notes, 3 reviewers)
  read 3 of its input, pack: median 25 minutes, range 12 to 25, total 62 (n = 3 reads of 2 notes, 3 reviewers)
  read 3 of its input, hybrid: median 17 minutes, range 14 to 20, total 34 (n = 2 reads of 2 notes, 2 reviewers)

13. Blinding: the reviewers' guess of the arm (n = 24 reads of 12 notes, 4 reviewers)
  plain: guessed right 1 of 8, guessed another arm 6 of 8, could not tell 1 of 8 (n = 8 reads of 4 notes, 4 reviewers)
  pack: guessed right 1 of 8, guessed another arm 5 of 8, could not tell 2 of 8 (n = 8 reads of 4 notes, 4 reviewers)
  hybrid: guessed right 1 of 8, guessed another arm 6 of 8, could not tell 1 of 8 (n = 8 reads of 4 notes, 4 reviewers)

14. Agreement: notes read by two or more reviewers (n = 12 notes, 24 reads)
  the same omitted items: 8 of 12 notes (n = 12 notes)
  the same count of unsupported statements: 8 of 12 notes (n = 12 notes)
  the same count of wrong figures: 7 of 12 notes (n = 12 notes)
  the same decision: 11 of 12 notes (n = 12 notes)
  Disagreements, for the trial lead to look at (n = 12 notes):
    7N7N (A, pack): differ on omitted items, count of unsupported statements: R1 omitted A-05 A-07 A-12, unsupported 2, wrong 2, not_at_all; R4 omitted A-05 A-07 A-09 A-12, unsupported 1, wrong 2, not_at_all (n = 2 reads)
    7YYH (B, hybrid): differ on count of unsupported statements, count of wrong figures: R1 omitted B-06, unsupported 0, wrong 0, with_edits; R4 omitted B-06, unsupported 1, wrong 1, with_edits (n = 2 reads)
    ANAT (B, plain): differ on omitted items: R1 omitted B-08 B-11 B-12, unsupported 0, wrong 0, with_edits; R4 omitted B-08 B-11, unsupported 0, wrong 0, with_edits (n = 2 reads)
    KUNF (A, plain): differ on count of unsupported statements: R3 omitted A-03, unsupported 0, wrong 0, with_edits; R4 omitted A-03, unsupported 2, wrong 0, with_edits (n = 2 reads)
    TR43 (B, pack): differ on count of wrong figures: R1 omitted B-07 B-08, unsupported 1, wrong 3, not_at_all; R2 omitted B-07 B-08, unsupported 1, wrong 2, not_at_all (n = 2 reads)
    V4E7 (A, hybrid): differ on count of wrong figures: R1 omitted A-04, unsupported 0, wrong 2, with_edits; R3 omitted A-04, unsupported 0, wrong 1, with_edits (n = 2 reads)
    VAPC (A, plain): differ on omitted items, count of wrong figures: R1 omitted A-03 A-06, unsupported 1, wrong 1, not_at_all; R2 omitted A-03, unsupported 1, wrong 2, not_at_all (n = 2 reads)
    XMXM (A, hybrid): differ on omitted items, count of unsupported statements, count of wrong figures, decision: R2 omitted A-11 A-12, unsupported 0, wrong 1, with_edits; R4 omitted A-10 A-11 A-12, unsupported 1, wrong 2, not_at_all (n = 2 reads)

These counts describe these 12 notes, written on EXAMPLE-host by EXAMPLE-model for 2 inputs, and these 4 reviewers; they are not rates for the arms in general (n = 24 reads of 12 notes, 4 reviewers).

CAPSTONE: CREDIT BY ARTIFACT, scored with data/capstone/rubric.md and answers/capstone/assessor-key.md (n = 54 scores of 3 readers by 2 scorers)
EXAMPLE DATA: made-up scores that show the analysis runs; these are not results (n = 54 scores of 3 readers by 2 scorers)
  process screen: full 3 of 6, partial 3 of 6, none 0 of 6 (n = 6 scores, 3 readers)
  work map: full 1 of 6, partial 0 of 6, none 5 of 6 (n = 6 scores, 3 readers)
  sort sheet: full 1 of 6, partial 0 of 6, none 5 of 6 (n = 6 scores, 3 readers)
  skill file: full 0 of 6, partial 4 of 6, none 2 of 6 (n = 6 scores, 3 readers)
  script: full 2 of 6, partial 4 of 6, none 0 of 6 (n = 6 scores, 3 readers)
  output contract: full 4 of 6, partial 2 of 6, none 0 of 6 (n = 6 scores, 3 readers)
  test case and checks: full 1 of 6, partial 5 of 6, none 0 of 6 (n = 6 scores, 3 readers)
  test run: full 2 of 6, partial 1 of 6, none 3 of 6 (n = 6 scores, 3 readers)
  release decision: full 4 of 6, partial 2 of 6, none 0 of 6 (n = 6 scores, 3 readers)
  trap T01: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T02: caught in 4 of 6 scores where it was assessed (n = 6 scores)
  trap T03: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T04: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T05: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T06: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T07: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T08: caught in 4 of 6 scores where it was assessed (n = 6 scores)
  trap T09: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T10: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T11: caught in 6 of 6 scores where it was assessed (n = 6 scores)
  trap T12: caught in 4 of 6 scores where it was assessed (n = 6 scores)
  trap T13: caught in 4 of 6 scores where it was assessed (n = 6 scores)
  hours per reader: median 13.5, range 9 to 17 (n = 3 readers)
  scorers agree on the credit in 22 of 27 reader-artifact pairs scored twice (n = 27 pairs)
    C1, test run: S1 partial, S2 none (n = 2 scores)
    C2, test case and checks: S1 partial, S2 full (n = 2 scores)
    C3, process screen: S1 full, S2 partial (n = 2 scores)
    C3, sort sheet: S1 none, S2 full (n = 2 scores)
    C3, work map: S1 none, S2 full (n = 2 scores)
