Efficient Temporal Sentence Grounding in Videos with Multi-Teacher Knowledge Distillation (2024)

2.1 Proposal-based Methods

The majority of proposal-based approaches rely on some carefully thought-out dense sample strategies, which gather a set of video segments as candidate proposals and rank them in accordance with the scores obtained for the similarity between the proposals and the query to choose the most compatible pairs.Zhang etal. [25] convert visual features into a 2D temporal map and encode the query in sentence-level representation, which is the first solution to model proposals with a 2D temporal map (2D-TAN).BAN-APR [3] utilize a boundary-aware feature enhancement module to enhance the proposal feature with its boundary information by imposing a new temporal difference loss. Currently, most proposal-based methods are time-consuming due to the large number of proposal-query interactions.

2.2 Proposal-free Methods

Actually, the caliber of the sampled proposals has a significant impact on the impressive performance obtained by proposal-based methods. To avoid incurring the additional computational costs associated with the production of proposal features, proposal-free approaches directly regress or forecast the beginning and end times of the target moment.VSLNet Zhang etal. [22] exploits context-query attention modified from QANet Yu etal. [19] to perform fine-grained multimodal interaction. Then a conditioned span predictor computes the probabilities of the start/end boundaries of the target moment.SeqPAN [23] design a self-guided parallel attention module to effectively capture self-modal contexts and cross-modal attentive information between video and text inspired by sequence labeling tasks in natural language processing.Yang and Wu [17] propose Entity-Aware and Motion-Aware Transformers (EAMAT) that progressively localize actions in videos by first coarsely locating clips with entity queries and then finely predicting exact boundaries in a shrunken temporal region with motion queries.Nevertheless, with the improvement of performance, huge and complex architectures inevitably result in higher computational cost during inference phase.

2.3 Fast Video Temporal Grounding

Recently, fast video temporal grounding has been proposed for more practical applications.According to [5], the standard TSGV pipeline can be divided into three components. The visual encoder and the text encoder are proved to have little influence in model testing due to the features pre-extracted and stored at the beginning of the test, and cross-modal interaction is the key to reducing the test time. Thus, a fine-grained semantic distillation framework is utilized to leverage semantic information for improving performance. Besides, Wu etal. [16] utilize commonsense knowledge to obtain bridged visual and text representations, promoting each other in common space learning.However, based on our previous analysis, the inferred time proposed by [5] is only a part of the entire prediction processing. The processing from inputting video features to predicting timestamps is still time-consuming.

3.1 Problem Formulation

Given an untrimmed video $V=[f_{t}]_{t=1}^{T}$ and the language query $Q=[q_{j}]_{j=1}^{m}$, where $T$ and $m$ are the numbers of frames and words, respectively. The start and end times of the ground truth moment are indicated by $\tau^{s}$ and $\tau^{e}$, $1\leq\tau_{s}<\tau_{e}\leq T$.Mathematically, TSGV is to retrieve the target moment starting from $\tau_{s}$ and ending at $\tau_{e}$ by giving a video $V$ and query $Q$, i.e., $\mathcal{F}_{TSGV}:(V,Q)\mapsto(\tau_{s},\tau_{e})$.

3.2 General Scheme

3.2.4 Knowledge Aggregation Unit

Our goal is to combine all the unified predictions from $b$ branches to establish a strong teacher distribution.Drawing inspiration from [9], we developed the Knowledge Aggregation Unit (KAU). The KAU integrates parallel transformations with varying receptive fields, harnessing both local and global contexts to derive a more accurate target probability. The KAU’s design is illustrated in Figure 3.

Considering saving more original information, we first take the video features $V^{\prime}$ in the eq. (5) as input and then add convolution layers. The convolution operation is conducted with a small kernel size of 3 initially and then consistently increased to 5 and 7.Further, we incorporate the average pooling of query features $Q^{\prime}$ in the eq. (5) for richer representations. Then we concatenate all the splits and obtain the intermediate vector $v$, denoted as:

$\displaystyle v=[q_{avg},g([v_{conv3},v_{conv5},v_{conv7}])]$

(9)

where $q_{avg}$ denotes the result of $Q^{\prime}$ after average pooling, g(·) denotes the global pooling function, $v_{conv3}$, $v_{conv5}$, and $v_{conv7}$ denote the results after the convolution layers with kernel size 3, 5, and 7, respectively.

Passing through a fully connected layer $FC$, a channel-wise softmax operator is applied to obtain the soft attention $a$.

$\displaystyle a=Softmax(\mathtt{FC}(v))\in\mathbb{R}^{2b\times n}$

(10)

where the $b$ denote the number of teacher branch, $2b$ is because there are two probability distributions (i.e., start and end).

Finally, we fuse prediction results from multiple branches via an element-wise summation to obtain the weighted ensemble probability.

$\displaystyle\widetilde{P}=\sum_{i=1}^{b}{a^{i}\otimes\hat{P}^{i}}\in\mathbb{R$

(11)

where $\widetilde{P}$ denotes the ensemble probability, $P^{i}\in\mathbb{R}^{2\times n}$ means the start and end distribution from $i$-th teacher branch, and $\otimes$ refers to the channel-wise multiplication. Our experiments (see Section 4.5.1) prove that the weights generated by KAU can achieve better distillation performance.

3.3 Training and Inference

4.1 Datasets

To evaluate the performance of TSGV, we conduct experiments on three challenging datasets, all the queries in these datasets are in English. Details of these datasets are shown as follows:

Charades-STA[6] is composed of daily indoor activities videos, which is based on Charades dataset[14]. This dataset contains 6672 videos, 16,128 annotations, and 11,767 moments. The average length of each video is 30 seconds. $12,408$ and $3,720$ moment annotations are labeled for training and testing, respectively;

ActivityNet Caption[1] is originally constructed for dense video captioning, which contains about $20$k YouTube videos with an average length of 120 seconds. As a dual task of dense video captioning, TSGV utilizes the sentence description as a query and outputs the temporal boundary of each sentence description.

TACoS[12] is collected from MPII Cooking dataset[12], which has 127 videos with an average length of $286.59$ seconds.

4.2 Evaluation Metrics

Following existing video grounding works, we evaluate the performance on two main metrics:

mIoU:“mIoU" is the average predicted Intersection over Union in all testing samples. The mIoU metric is particularly challenging for short video moments;

Recall:We adopt “$\textrm{R@}n,\textrm{IoU}=\mu$” as the evaluation metrics, following[6]. The “$\textrm{R@}n,\textrm{IoU}=\mu$” represents the percentage of language queries having at least one result whose IoU between top-$n$ predictions with ground truth is larger than $\mu$. In our experiments, we reported the results of $n=1$ and $\mu\in\{0.3,0.5,0.7\}$.

The Metric of Efficiency:Time, FLOPs, and Params are used to measure the efficiency of the model.Specifically, the time refers to the entire inferring time from the input of the video and query pair to the output of the prediction. FLOPs refers to floating point operations, which is used to measure the complexity of the model. Params refers to the model parameter size except the word embedding.

4.3 Implementation Details

For language query $Q$, we use the $300$-D GloVe[11] vectors to initialize each lowercase word, which are fixed during training. Following the previous methods, 3D convolutional features (I3D) are extracted to encode videos.We set the dimension of all the hidden layers as $128$, the kernel size of the convolutional layer as $7$, and the head size of multi-head attention as $8$ in our model.For all datasets, models are trained for $100$ epochs. The batch size is set to $16$. The dropout rate is set as 0.2.Besides, an early stopping strategy is adopted to prevent overfitting.The whole framework is trained by Adam optimizer with an initial learning rate of 0.0001. The loss weight $\alpha$ is set as 0.1 in all the datasets. The temperate was set to 1, 3, 3 on Charades-STA, ActivityNet, and TACoS.The pre-trained teacher models are selected in SeqPAN, BAN-APR, EAMAT, and CCA.More ablation studies can be found in Section4.5. All experiments are conducted on an NVIDIA RTX A5000 GPU with 24GB memory. All experiments were performed three times, and reporting the average of performance.

4.4 Comparison with State-of-the-art Methods

We strive to gather the most current approaches, and compare our proposed model with the following state-of-the-art baselines on three benchmark datasets:

Overall Efficiency-Accuracy Analysis

Considering that fast TSGV task pays the same attention to efficiency as accuracy, we evaluate FLOPs, Params, and Times for each model. For a fair comparison, the batch size is set to 1 for all methods during inference. Besides, we also calculate the sum of the accuracy in terms of “R1@0.3” and “R1@0.5”, named sumACC to evaluate the whole performance of each model.

As Table 2 shows, our method surpasses all other methods and achieves the highest speed, minimal FLOPs and Params on all three datasets.We note that EMTM is at least 2000 times fewer in FLOPs than state-of-the-art proposal-based models (SCDM and 2D-TAN). According to sumACC, EMTM outperforms these two models by gains of at most 26.75% on Charades-STA and 14.30% on ActivtyNet.Despite the parameter size of VSLNet is at the same level as our method, we outperform it significantly in terms of accuracy, achieveing 15.30% absolute improvement by “sumACC” on Charades-STA.When it comes to CCA, which is proposed for fast TSGV, EMTM outperforms 16950x fewer in FLOPs and 121x fewer in model parameter size on Charades-STA.The above comparison illustrates that our method has significant efficiency and accuracy advantages.

Accuracy Analysis

As shown in Table 3, we can observe that our method performs better than extensive methods in most metrics on three benchmark datasets. Compared with FVMR and CCA, our model performers better in all metrics. Especially, EMTM achieves an absolute improvement of 4.58% on Charades- STA and 5.34% on TACoS on the metric "R@1, IoU=0.7", which is a more crucial criterion with higher quality.

The performance of our model on ActivityNet is slightly lower than CCA. The supposed reason may be that ActivityNet is more challenging since it covers a wide range of videos, not limited to daily indoor activities videos in Charades-STA and cooking videos in TACoS. In such cases, label distillation may not effectively capture the key features of the dataset, resulting in limited performance gains. Besides, the effectiveness of label distillation often depends on the performance of the teacher model. The backbone we use is SeqPAN, which has relatively poor performance on ActivityNet, limiting our upper bound even with label distillation. However, our framework can be adapted to any VMR model. If we replace it with the more powerful upcoming backbone models, the performance will surpass the current version.

4.5 Ablation Studies

In this part, we perform ablation studies to analyze the effectiveness of the EMTM. All experiments are performed three times with different random seeds.

4.5.2 Effect of Number of Teacher Models

We investigate the influence of different numbers of teacher models on Charades-STA. As shown in Figure 4, the performance presents a rising tendency with the increase of teachers.According to the results, we realize that our improvements are not only from soft targets with one single teacher, but also from the learning of structural knowledge and intermediate-level knowledge with fused multi-teacher teaching. Multiple teachers make knowledge distillation more flexible, while ensemble helps improve the training of students and transfer related information of examples to them.

4.6 Qualitative Analysis

Two samples of prediction on Charades-STA are depicted in Figure 6.The first sample indicates our approach can refine the predictions when the basic model already obtained satisfactory results.The second sample shows the basic model tends to predict the boundary position, possibly due to its limited understanding of the video.As a result, the model relies on biased positional information to make moment predictions.However, utilizing a shared encoder and label distillation approach can provide additional information that enables the model to more precisely predict the moment boundary.