Interpretable Video Action Quality Assessment
Combined I3D features, Transformer cross-attention, and pose-derived supervision to reduce relative L2 error by 6.46% on FineDiving, with frame-aligned diagnostic evidence.
From Video Scores to Phase-Level Evidence
A single diving score gives little guidance about what went wrong. For IDSS, a CMU project on interpretable diving assessment, I combined I3D video features with procedure-aware Transformer cross-attention to align action phases between a query video and an exemplar. Pose-derived auxiliary supervision lets the model jointly learn score regression, temporal segmentation, and interpretable motion-quality prediction.
Using Detectron2 and HRNet, I built a pipeline for five phase-specific quality targets: tuck tightness, body straightness, entry angle, splash size, and distance from the platform. Gaussian mixtures supported platform-distance diagnostics. These assessments were linked to frame-aligned video evidence and presented through a Flask interface, connecting the predicted score to specific aspects of the dive.
Better Scoring with Faster Convergence
On FineDiving, a dataset of 3,000 videos, the model matched the baseline’s 100-epoch regression error in 30 epochs. The final comparison also showed improvements in score correlation, temporal alignment, and relative L2 error:
| Metric | Baseline | IDSS |
|---|---|---|
| Spearman Rank Correlation | 0.9272 | 0.9302 |
| tIoU@0.5 | 93.73% | 95.59% |
| tIoU@0.75 | 54.07% | 57.14% |
| Relative L2 Distance (R-ℓ2 ×100) | 0.3313 | 0.3099 |
Relative L2 error fell by 6.46%. Alongside the scoring gains, the pose-derived targets supplied interpretable, frame-aligned evidence for the final report.