Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VC Noro model #247

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open

Add VC Noro model #247

wants to merge 12 commits into from

Conversation

kenxxxxx
Copy link

✨ Description

In this PR, we release an unofficial PyTorch implementation of Noro, a Noise-Robust One-shot Voice Conversion (VC) system. This model is designed to convert the timbre of speech from a source speaker to a target speaker using only a single reference speech sample while preserving the semantic content of the original speech. Noro introduces innovative components tailored for VC using noisy reference speeches, including a dual-branch reference encoding module and a noise-agnostic contrastive speaker loss.

The main purpose of this PR is to provide a noise-robust VC solution that performs effectively even with noisy reference speeches, making it suitable for real-world applications. Additionally, we explore the hidden speaker representation capabilities of the VC system by repurposing its reference encoder as a speaker encoder, demonstrating competitive performance with advanced self-supervised learning models.

To test this PR, follow the instructions in the updated README.md to set up the environment, train the model, and evaluate its performance under different acoustic environments.

🚧 Related Issues

None

👨‍💻 Changes Proposed

  • Implemented the Noro model with a dual-branch reference encoding module.
  • Added the training and evaluation scripts for the Noro model.
  • Added detailed documentation and examples for training and testing the model.

🧑‍🤝‍🧑 Who Can Review?

@RMSnow @HarryHe11 @Adorable-Qin

✅ Checklist

  • Code has been reviewed
  • Code complies with the project's code standards and best practices
  • Code has passed all tests
  • Code does not affect the normal use of existing features
  • Code has been commented properly
  • Documentation has been updated (if applicable)
  • Demo/checkpoint has been attached (if applicable)

Copy link
Collaborator

@RMSnow RMSnow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your efforts! Great job! This is our first time to introduce VC. So let us set high criteria for future developers!

README.md Show resolved Hide resolved
config/vc.json Outdated Show resolved Hide resolved
egs/vc/README.md Outdated Show resolved Hide resolved
egs/vc/exp_config_4gpu_clean.json Outdated Show resolved Hide resolved
models/base/vc_dataset.py Outdated Show resolved Hide resolved
models/vc/ns2_uniamphion.py Outdated Show resolved Hide resolved
models/vc/vc_loss.py Outdated Show resolved Hide resolved
models/vc/vc_trainer.py Outdated Show resolved Hide resolved
models/vc/vc_utils.py Outdated Show resolved Hide resolved
models/vc/vc_utils.py Outdated Show resolved Hide resolved
@HarryHe11
Copy link
Collaborator

@RMSnow Thank you, Xueyao, for your detailed comments! @kenxxxxx Yuchen, please familiarize yourself with Git-based development and directly update your code on your fork so we can track your revision progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants