Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📝 A List of Welcoming Features #1446

Open
kennymckormick opened this issue Feb 15, 2022 · 12 comments
Open

📝 A List of Welcoming Features #1446

kennymckormick opened this issue Feb 15, 2022 · 12 comments
Labels
help wanted Extra attention is needed

Comments

@kennymckormick
Copy link
Member

kennymckormick commented Feb 15, 2022

Call for Contributors: https://openmmlab.medium.com/be-an-openmmlab-contributor-c9087428062a

Here we maintain a list of welcoming features proposed by the community. Our developer may consider integrating them into MMAction2. Besides, we highly encourage users to implement them and raise a PR.

  1. A Custom Training Pipeline for Spatio-Temporal Action Detection. How can I train a spatio-temporal action detection network only for action recognition? #1443 , how to make my custom ava dataset  #1318 , about AVA #1292 , Spatio-temporal region proposals #604
  2. Support AVA-Kinetics. How to train or Fine-tuning a specific class on the AVA-Kinetics #853
  3. Support HACS. support hacs dataset? #1331
  4. Support Charades. (The feature is partially implemented in [Feature] Support Charades Dataset #788 , thanks @congee524 . Anyone who would like to work on this feature can refer to the PR [Feature] Support Charades Dataset #788. )
  5. Lack of SOTA Action Detection model. Lack of SOTA temporal action detection models #1124
  6. TensorRT Support. export #1424
  7. Support SGN. Implementation of Semantics Guided Network(SGN) #1463 (It is easy to implement, I guess it may take at most 1 or 2 days)
@kennymckormick kennymckormick pinned this issue Feb 15, 2022
@kennymckormick
Copy link
Member Author

@gengenkai , you can check if there are some features you are interested in.

@kennymckormick kennymckormick changed the title [TODO] A List of Welcoming Features 📝 A List of Welcoming Features Feb 15, 2022
@kennymckormick kennymckormick added the help wanted Extra attention is needed label Feb 17, 2022
@hukkai
Copy link
Collaborator

hukkai commented Feb 28, 2022

I did run action detection models on AVA-Kinetics. The changes on codes are not very complicated, but the data processing is rather painful. Many videos in the Kinetics 700 dataset are no longer available. Many videos can be corrupt during download, i.e., shorter than 10 seconds. This is not a serious problem for classification tasks but for detection, the missing part can have annotations...

The CVDF repo (https://github.com/cvdfoundation/kinetics-dataset) provides Kinetics dataset (400, 600, 700-2020) but not Kinetics700. Also there are many videos in the repo which is corrupted.

If there could be a good solution to prepare the dataset, that is super helpful and I can submit a PR for AVA-kinetics training.

@kennymckormick
Copy link
Member Author

Hi, @hukkai , if you are eager to implement this feature in MMAction2, you can provide me a list of missing videos to be. I will see if these videos exist in my database and provide them to you (after ECCV ddl).

@hukkai
Copy link
Collaborator

hukkai commented Mar 9, 2022

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

@kennymckormick
Copy link
Member Author

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

I guess we can not directly share the download link, host a dataset of dozens of GB is expensive. A practical approach would be: users who require the missing data send me the missing list, and I will share the missing videos to them if I have.

@irvingzhang0512
Copy link
Contributor

irvingzhang0512 commented Mar 10, 2022

@hukkai You can take a look at kinetics-dataset, maybe CVDF has already provide a link to download ava-kinetics videos and annotations

@hukkai
Copy link
Collaborator

hukkai commented Mar 10, 2022

@irvingzhang0512 My first attempt is the CVDF repo. One issue is that they have K400, K600 and K700-2020 but not K700. Some videos are different for K700-2020 and K700 which is required by AVA-Kinetics. This is not a big issue. The problem is that many videos in that repo are corrupted. I also put an issue (cvdfoundation/kinetics-dataset#4 (comment)) but it seems that they only handle the val/test set.

@kennymckormick
Copy link
Member Author

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.

For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

Do you still need any help in finding missing videos for implementing AVA-Kinetics in MMAction2 ?

@hukkai
Copy link
Collaborator

hukkai commented Mar 21, 2022

Hi @kennymckormick I can implement the AVA-Kinetics dataset and training. Training on AVA-Kinetics indeed helps the performance on AVA a lot.
For the AVA-Kinetics, I plan to use YouTube-dl to download from YouTube. Some videos are missing and I can provide the list. Then how do you want to combine these missing videos with these which are still on YouTube? Will mmaction2 provide a link to download them?

Do you still need any help in finding missing videos for implementing AVA-Kinetics in MMAction2 ?

No, I will implement in a way that remove missing or incomplete videos from the training list. I will submit a pull request recently.

@rlleshi
Copy link
Contributor

rlleshi commented May 26, 2022

I had written a multimodal_demo script some time ago that late fuses the scores of rgb-, skeleton- & audio-based models.
Does it make sense to make a PR out of it?

@borijang
Copy link

Is development of SOTA temporal action detection models (such as ASM-Loc) still an active task?

@coldmanck
Copy link

Hi @hukkai @kennymckormick , is there any update on supporting AVA-Kinetics? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

6 participants