Realtime Video QA

Setup

Setup TMLGA

1. Download requirements

cd TMLGA
sh ./downlad.sh

2. Download pretrained weights here and move them to TMLGA/checkpoints/charades_sta
3. To test that everything works, run

python main.py --config-file=experiments/charades-sta.yaml

Setup VideoQA

1. Download the VGG16 checkpoint and C3D checkpoint and put them in VideoQA/util
2. Download the word embeddings trained over 6B tokens (glove.6B.zip) from GloVe, unzip them, and put the 300d file in directory VideoQA/util
3. pip install -r PSAC/requirements.txt

Pre-process Dataset

1. Download the MSVD-QA dataset and place it in PSAC/MSVD-QA
2. Download the youtube videos from the MSVD dataset (YouTubeClips.tar in the downloads section) and unzip them into PSAC/MSVD-QA/video
3. Convert video names from garbled YouTubeClips video names into their corresponding video IDs, using this file as the mapping between YouTubeClips.tar name and the real ID
4. Delete problematic videos with IDs:

• 451
• 745
• 1106
• 1120
• 1258
• 1357
• 1475
• 1595 (TODO: LATER -- Figure out how to deal with these)

5. Pre-process the videos by running:

python preprocess_msvdqa.py PSAC/MSVD-QA