-
Notifications
You must be signed in to change notification settings - Fork 19
-
Notifications
You must be signed in to change notification settings - Fork 19
QuantizeDequantizeG's backward() function doesn't be executed!!!! #3
Comments
Sorry, I couldn't understand. Please post the exact error that you are getting and describe the situation in further detail. |
Hi wuzhiyang2016, Reading your comment carefully, I understood better. What you are saying is that the backward method of QuantizeDequantizeG is not called during back-propagation (loss.backward()). This is a good question and it shows that you have tried to analyze and understand what is happening. Let me answer in detail: This is the crux of Straight Through Estimation (STE) - is that backward does not involve any quantization - it goes backward straight through as though no quantization happened in the forward. We support three kinds of Quant Estimation methods, that you can see as defined in qaunt_base_module.py: STRAIGHT_THROUGH_ESTIMATION is default. You can see this being set in quant_train_module.py If you change the above to QUANTIZED_THROUGH_ESTIMATION, you can see that the backward of QuantizeDequantizeG will be called. I would like to highlight a couple of limitations: I hope this helps. Best regards, |
very thanks, i just see your reply, let me take some time to understand your great reply~ |
hello, i have check the class QuantizeDequantizeG it's backward code, dx is the same with the paper's formula (8), but the scale derivative is not the same with the paper's formula (6), could you give me some advice? the paper which i mean is "TRAINED QUANTIZATION THRESHOLDS FOR ACCURATE AND EFFICIENT FIXED-POINT INFERENCE OF DEEP NEURAL NETWORKS |
Hi, The backward of QuantizeDequantizeG has numerical gradient: https://en.wikipedia.org/wiki/Numerical_differentiation Also our recommended quantized_estimation_type is STE, in which case this gradient is not used at all. I hope it is clear. |
i get, it's really clear ! very thanks!!! |
Keeping this open as an FAQ item, so that others can also benefit. |
hello, when i train quantized model by pytorch-jacinto-ai-devkit, a class called QuantizeDequantizeG, when doing loss.backward(), it's backword() function is not used,,, the code is in xnn/layers/funtion.py
Look forward to your favourable reply sincerely ~
The text was updated successfully, but these errors were encountered: