In order to evaluate our benchmark, we extend four methods from image domain. The first two are EWC and MAS. These are regularized methods that penalize changes to the most relevant parameters for the previous tasks. The other two, iCaRL and BIC, are memory-based methods that select and store samples of the current task into a memory buffer for future replay. Consistent with image benchmarks, the memory-based approaches significantly outperform the regularized methods.
| Model | Num. Task | Kinetics | ActivityNet-Trim | UCF101 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mem. Video Instances | Mem. Frame Capacity | Acc | BWF | Mem. Video Instances | Mem. Frame Capacity | Acc | BWF | Mem. Video Instances | Mem. Frame Capacity | Acc | BWF | ||
| EWC | 10 | None | None | 5.81% | 16.05% | None | None | 4.02% | 5.32% | None | None | 9.51% | 98.94% |
| MAS | 10 | None | None | 7.81% | 10.12% | None | None | 8.11% | 0.18% | None | None | 10.89% | 11.11% |
| EWC | 20 | None | None | 2.95% | 32.70% | None | None | 1.28% | 3.77% | None | None | 4.71% | 92.12% |
| MAS | 20 | None | None | 4.25% | 5.54% | None | None | 4.61% | 0.1% | None | None | 5.90% | 5.31% |
| Naive | 10 | 8000 | 2 × 106 | 30.14% | 41.30% | 4000 | 15.5 × 106 | 47.20% | 20.64% | 2020 | 3.69 × 105 | 91.42% | 7.43% |
| iCaRL | 10 | 8000 | 2 × 106 | 32.04% | 38.74% | 4000 | 15.5 × 106 | 48.53% | 19.72% | 2020 | 3.69 × 105 | 80.97% | 18.11% |
| BiC | 10 | 8000 | 2 × 106 | 27.90% | 51.96% | 4000 | 15.5 × 106 | 51.96% | 24.27% | 2020 | 3.69 × 105 | 78.16% | 18.49% |
| Naive | 20 | 8000 | 2 × 106 | 23.47% | 48.05% | 4000 | 15.5 × 106 | 40.78% | 23.18% | 2020 | 3.69 × 105 | 87.40% | 10.96% |
| iCaRL | 20 | 8000 | 2 × 106 | 26.73% | 42.25% | 4000 | 15.5 × 106 | 43.33% | 21.57% | 2020 | 3.69 × 105 | 76.59% | 21.83% |
| BiC | 20 | 8000 | 2 × 106 | 23.06% | 58.97% | 4000 | 15.5 × 106 | 46.53% | 15.95% | 2020 | 3.69 × 105 | 70.69% | 24.90% |