What is learner analytics?
Learner analytics is the use of completion and assessment data to establish whether training achieved anything. It covers what the available data supports concluding, which is a narrower set of claims than most dashboards make. Timing and progress data describe behaviour inside a module and not competence.
Also called: Learning analytics
Why most learning dashboards report effort rather than effect
The standard learning dashboard reports hours consumed and modules completed, neither of which is evidence that anybody can do anything differently. Those measures are presented because they are easy to collect, and they survive because no one asks the harder question of what the data can actually support. Time spent on a page, for instance, cannot distinguish careful reading from an unattended browser. An analytics practice worth having states the limits of each measure alongside the number, so a board is not invited to conclude more than the evidence allows.
- Hours consumed measures delivery effort rather than any change in capability.
- Time on page cannot separate careful reading from an unattended screen.
- Assessment and retention data support claims about competence, where progress data does not.
- Stating the limit of a measure protects the credibility of the measures that hold.
How learner analytics works
- The question the analysis must answer is stated before any measure is selected.
- Measures are classified by whether they evidence behaviour or competence.
- Assessment results are reported per competency rather than as a single score.
- Cohort comparisons are checked for differences in population before they are interpreted.
- Each reported figure carries the claim it supports and the claim it does not.
How learner analytics is evaluated
Analytics is itself evaluated on whether its conclusions would survive a challenge. Examine the following for each reported measure:
- the specific claim each measure is used to support
- proportion of reported figures with a stated denominator
- agreement between assessment results and supervisory judgement
- coverage of the cohort behind each aggregate figure
- stability of a measure across consecutive cohorts
Limitations, risks and common failure modes
- Time on page is reported as engagement and then as effectiveness.
- Completion percentages are quoted without the cohort they are calculated against.
- Dashboards aggregate across roles whose obligations are entirely different.
- Correlations between training and performance are presented as causal at board level.