Challenge of Remembering from fewer frames

The video instances contain a temporal dimension size that could show large variability, and some action classes do not require considering all frames to understand them. To favor fair comparisons between methods and datasets and reduce the huge memory consumption, we define the working memory size of the rehearsal methods in terms of stored frames. This creates a new unique scenario of Class Incremental Learning in video data, in which rehearsal methods must decide first what subset of frames should be selected and then decide what video to store according to selected frames per video. We encourage future works on Class Incremental Learning for Video Understanding to focus on this and beat our Temporal Consistency Regulation strategy.