v1.4 PCS Trainability Audit Chapter

Companion to: v1.0 (base validation whitepaper) + v1.2 SIOP/AERA psychometric framework + v1.3 cross-cultural chapter. Purpose: Document Talentopian's systematic trainability audit and how its findings inform the PCS scoring framework. v1.4 addresses the canonical SIOP/AERA critique of game-based assessment: "games are trainable, so they measure practice not innate talent." Audience: SIOP / AERA readers, counselors evaluating PCS validity concerns, and game-based-assessment researchers comparing methodologies. Status: Self-published chapter documenting the trainability-audit framework and preliminary pilot findings. This is a first-pass evidence-generation document; formal validation is planned through academic partnership. Author: Talentopian Research (Joyeux Réalité étendue Inc.). Version: 1.4.0 trainability chapter

1. Why this chapter exists — the trainability critique

The dominant critique of game-based assessment (GBA) in the SIOP / AERA literature: "games are trainable, so they measure practice effects rather than innate talent or stable competencies." Several commercial GBA platforms have faced this criticism, and the critique has empirical support — cognitive tasks like reaction-time games, working-memory tasks, and pattern recognition show substantial practice gains within a handful of attempts.

If a GBA platform does not address trainability, its scores conflate three sources of variance:

1. Innate / stable competency (what the platform claims to measure)
2. Practice / training (what improves with repetition)
3. Motivation / engagement (what varies with state, not trait)

Talentopian's response: systematically audit each game in the library for trainability, classify each game by trainability level, and down-weight trainable games in the PCS aggregation so that scores reflect (1) more than (2).

v1.4 documents this framework and the way audit classifications feed the scoring model.

2. Methodology — the trainability-audit framework

The audit classifies each game on three dimensions:

Classification decision tree:

• REJECT: HIGH trainability + non-motor → score is dominated by practice; exclude from PCS aggregation OR use as engagement-only signal
• CAUTION: MEDIUM trainability OR motor-dim with practice effects → include with reduced weight + first-attempt-only
• KEEP: LOW trainability + valid first-attempt → include at full weight

3. Preliminary pilot findings

An initial pilot audit has classified a sub-sample of the game library into the three tiers. Preliminary internal-structure analysis is underway; findings will inform framework consolidation and formal validation through academic partnership. The consistent signal from this early pass is that a substantial share of games warrant some degree of down-weighting — meaning a naive (un-audited) PCS aggregator would over-weight trainable signal. The audit's mitigation is designed to reduce this artifact.

Honest scope of the pilot audit:

• The pilot covers a sub-sample of Talentopian's game library; later audit batches will extend coverage.
• The HIGH/MEDIUM/LOW trainability ratings are expert-rater classifications based on game mechanics and literature review; empirical retest correlations (per v1.2 §2) would strengthen these classifications in a future revision.

4. How the audit informs PCS scoring

The audit classifications become scoring levers in the aggregation model. Conceptually, each game carries:

• a trainability tier (REJECT / CAUTION / KEEP),
• a corresponding aggregation weight (heavily down-weighted for REJECT, moderately down-weighted for CAUTION, full weight for KEEP),
• a first-attempt-only flag applied where practice effects are indicated, and
• a motor-dimension flag separating motor from cognitive measurement.

Consumer integration (planned, not yet wired):

• The scoring service joins each game attempt to its trainability classification.
• It applies the corresponding weight to the game's contribution to the per-parameter score.
• It uses the first-attempt-only flag to filter multi-attempt sessions when the audit flags practice effects.

The end-to-end path = audit classification → scoring-model wiring → aggregator consumption. v1.4 documents the framework and the first stages; consumer integration remains planned.

5. What this means for PCS validity claims (the SIOP/AERA-relevant framing)

Trainability-adjusted PCS scores answer the SIOP/AERA critique more rigorously than un-audited GBA scores:

Honest caveat: the audit improves discriminant validity (separating practice from competency) but does not by itself validate the underlying competency claims. The v1.2 psychometric framework's 4-coefficient validation (test-retest / concurrent / convergent / criterion) remains the deeper validation path; the audit is a NECESSARY-not-sufficient precondition for those coefficients to be interpretable as competency rather than practice.

6. Open work + v2 trainability research directions

Extended audit:

• Extend coverage from the pilot sub-sample to the full game library.
• Re-audit with empirical retest data once enough users have accumulated multi-session histories (currently sparse).
• Add inter-rater agreement on the classifications (Cohen's κ across two or more raters).

Consumer integration (planned):

• The scoring service joins the trainability classification and applies the audit-derived weights and first-attempt flags.
• Historical scores are re-aggregated under audit-weighted rules.

Empirical validation of the classifications:

• Retest reliability per tier: KEEP games should show higher test-retest than CAUTION/REJECT.
• Practice-curve fitting per game: empirical practice effects vs the audit's a-priori ratings.
• Per-tier external-criterion correlation (when criterion data lands).

7. v1.4 honest scope

What v1.4 IS:

• A documentation of Talentopian's systematic response to the SIOP/AERA GBA trainability critique
• A report of the pilot audit framework and preliminary tier findings
• A framework specification of the audit methodology, the scoring-model wiring, and the consumer-integration path
• A bridge between the audit work and the deeper validation path (v1.2 + v1.4)

What v1.4 IS NOT:

• NOT a claim that consumer integration is complete — the scoring service's join to the trainability classification is not yet wired (planned)
• NOT a claim that the pilot covers the full library — the pilot is a sub-sample; later batches extend coverage
• NOT an empirical validation of the classifications — classifications are expert-rater + literature-review based; empirical practice-curve fitting and Cohen's κ inter-rater agreement are open work
• NOT a substitute for v1.2 4-coefficient validation — the audit is a precondition, not a replacement; both are needed
• NOT a peer-reviewed analysis — self-published research draft; a future peer-reviewed, IRB-gated study is planned through academic partnership

8. Cross-references

• v1.0 — base whitepaper; v1.4 extends the Coherence Triangulation and Validation Studies sections.
• v1.2 SIOP/AERA psychometric — the v1.2 4-coefficient framework plus v1.4 trainability form a combined validity story.
• v1.3 Cross-cultural — v1.3 plus v1.4 combine international and trainability-adjusted validity.
• the Methodology & Citations brief — trainability literature citations should be added at its next revision (e.g., Salgado et al. on trainability of cognitive ability tests; Sackett et al. on practice effects in selection assessments).

This chapter is designed to be relevant to Korean career-counseling practice as well; we hope to engage the Korean career-counseling community as the framework matures.

— Talentopian Research (v1.4 PCS Trainability Audit Chapter — SIOP/AERA-audience response to the GBA trainability critique)

1,254 words.  ·  All research  ·  Talentopian home