Running on yolov4 wrapper gives error

[asluborski98]

Hello - I am getting error when trying to run with my custom yolov4 model

I am running with code from this discussion: https://github.com/obss/sahi/issues/139

I am running this part in my Colab

detection_model = Yolov4DetectionModel(
    model_path='/content/overextrusion_model.weights',
    config_path= '/content/overextrusion_model.cfg',
    names_path='/content/overextrusion.names',
    device="cpu", # or 'cuda:0'
)

Here is the error:

i was here
 i was here 2
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-2-df39683a195a>](https://localhost:8080/#) in <module>()
      3     config_path= '/content/overextrusion_model.cfg',
      4     names_path='/content/overextrusion.names',
----> 5     device="cpu", # or 'cuda:0'
      6 )

2 frames
[/usr/local/lib/python3.7/dist-packages/sahi/model.py](https://localhost:8080/#) in category_names(self)
    592     @property
    593     def category_names(self):
--> 594         return list(self.model.config.names.values())
    595 
    596     def _create_object_prediction_list_from_original_predictions(

AttributeError: 'NoneType' object has no attribute 'config'

Here is my model.py updated with YoloV4 wrapper

# OBSS SAHI Tool
# Code written by Fatih C Akyon, 2020.

import numpy as np

from sahi.prediction import ObjectPrediction
from sahi.utils.torch import cuda_is_available, empty_cuda_cache
from typing import List, Dict, Optional, Union


class DetectionModel:
    def __init__(
        self,
        model_path: str,
        config_path: Optional[str] = None,
        names_path: Optional[str] =None,
        device: Optional[str] = None,
        mask_threshold: float = 0.5,
        prediction_score_threshold: float = 0.3,
        category_mapping: Optional[Dict] = None,
        category_remapping: Optional[Dict] = None,
        load_at_init: bool = True,
    ):
        """
        Init object detection/instance segmentation model.

        Args:
            model_path: str
                Path for the instance segmentation model weight
            config_path: str
                Path for the mmdetection instance segmentation model config file
            device: str
                Torch device, "cpu" or "cuda"
            mask_threshold: float
                Value to threshold mask pixels, should be between 0 and 1
            prediction_score_threshold: float
                All predictions with score < prediction_score_threshold will be discarded
            category_mapping: dict: str to str
                Mapping from category id (str) to category name (str) e.g. {"1": "pedestrian"}
            category_remapping: dict: str to int
                Remap category ids based on category names, after performing inference e.g. {"car": 3}
            load_at_init: bool
                If True, automatically loads the model at initalization
        """
        self.model_path = model_path
        self.config_path = config_path
        self.names_path = names_path
        self.model = None
        self.device = device
        self.mask_threshold = mask_threshold
        self.prediction_score_threshold = prediction_score_threshold
        self.category_mapping = category_mapping
        self.category_remapping = category_remapping
        self._original_predictions = None
        self._object_prediction_list = None

        # automatically set device if its None
        if not (self.device):
            self.device = "cuda:0" if cuda_is_available() else "cpu"

        # automatically load model if load_at_init is True
        if load_at_init:
            self.load_model()

    def load_model(self):
        """
        This function should be implemented in a way that detection model
        should be initialized and set to self.model.
        (self.model_path, self.config_path, and self.device should be utilized)
        """
        NotImplementedError()

    def unload_model(self):
        """
        Unloads the model from CPU/GPU.
        """
        self.model = None
        empty_cuda_cache()

    def perform_inference(self, image: np.ndarray):
        """
        This function should be implemented in a way that prediction should be
        performed using self.model and the prediction result should be set to self._original_predictions.

        Args:
            image: np.ndarray
                A numpy array that contains the image to be predicted.
        """
        NotImplementedError()

    def _create_object_prediction_list_from_original_predictions(
        self,
        shift_amount: Optional[List[int]] = [0, 0],
        full_shape: Optional[List[int]] = None,
    ):
        """
        This function should be implemented in a way that self._original_predictions should
        be converted to a list of prediction.ObjectPrediction and set to
        self._object_prediction_list. self.mask_threshold can also be utilized.

        Args:
            shift_amount: list
                To shift the box and mask predictions from sliced image to full sized image, should be in the form of [shift_x, shift_y]
            full_shape: list
                Size of the full image after shifting, should be in the form of [height, width]
        """
        NotImplementedError()

    def _apply_category_remapping(self):
        """
        Applies category remapping based on mapping given in self.category_remapping
        """
        # confirm self.category_remapping is not None
        assert self.category_remapping is not None, "self.category_remapping cannot be None"
        # remap categories
        for object_prediction in self._object_prediction_list:
            old_category_id_str = str(object_prediction.category.id)
            new_category_id_int = self.category_remapping[old_category_id_str]
            object_prediction.category.id = new_category_id_int

    def convert_original_predictions(
        self,
        shift_amount: Optional[List[int]] = [0, 0],
        full_shape: Optional[List[int]] = None,
    ):
        """
        Converts original predictions of the detection model to a list of
        prediction.ObjectPrediction object. Should be called after perform_inference().

        Args:
            shift_amount: list
                To shift the box and mask predictions from sliced image to full sized image, should be in the form of [shift_x, shift_y]
            full_shape: list
                Size of the full image after shifting, should be in the form of [height, width]
        """
        self._create_object_prediction_list_from_original_predictions(
            shift_amount=shift_amount,
            full_shape=full_shape,
        )
        if self.category_remapping:
            self._apply_category_remapping()

    @property
    def object_prediction_list(self):
        return self._object_prediction_list

    @property
    def original_predictions(self):
        return self._original_predictions

    def _create_predictions_from_object_prediction_list(
        object_prediction_list: List[ObjectPrediction],
    ):
        """
        This function should be implemented in a way that it converts a list of
        prediction.ObjectPrediction instance to detection model's original prediction format.
        Then returns the converted predictions.
        Can be considered as inverse of _create_object_prediction_list_from_predictions().
        Args:
            object_prediction_list: a list of prediction.ObjectPrediction
        Returns:
            original_predictions: a list of converted predictions in models original output format
        """
        NotImplementedError()


class MmdetDetectionModel(DetectionModel):
    def load_model(self):
        """
        Detection model is initialized and set to self.model.
        """
        try:
            import mmdet
        except ImportError:
            raise ImportError(
                'Please run "pip install -U mmcv mmdet" ' "to install MMDetection first for MMDetection inference."
            )

        from mmdet.apis import init_detector

        # set model
        model = init_detector(
            config=self.config_path,
            checkpoint=self.model_path,
            device=self.device,
        )
        self.model = model

        # set category_mapping
        if not self.category_mapping:
            category_mapping = {str(ind): category_name for ind, category_name in enumerate(self.category_names)}
            self.category_mapping = category_mapping

    def perform_inference(self, image: np.ndarray):
        """
        Prediction is performed using self.model and the prediction result is set to self._original_predictions.

        Args:
            image: np.ndarray
                A numpy array that contains the image to be predicted.
        """
        try:
            import mmdet
        except ImportError:
            raise ImportError(
                'Please run "pip install -U mmcv mmdet" ' "to install MMDetection first for MMDetection inference."
            )

        # Confirm model is loaded
        assert self.model is not None, "Model is not loaded, load it by calling .load_model()"

        # Supports only batch of 1
        from mmdet.apis import inference_detector

        prediction_result = inference_detector(self.model, image)

        self._original_predictions = prediction_result

    @property
    def num_categories(self):
        """
        Returns number of categories
        """
        if isinstance(self.model.CLASSES, str):
            num_categories = 1
        else:
            num_categories = len(self.model.CLASSES)
        return num_categories

    @property
    def has_mask(self):
        """
        Returns if model output contains segmentation mask
        """
        has_mask = self.model.with_mask
        return has_mask

    @property
    def category_names(self):
        if type(self.model.CLASSES) == str:
            # https://github.com/open-mmlab/mmdetection/pull/4973
            return (self.model.CLASSES,)
        else:
            return self.model.CLASSES

    def _create_object_prediction_list_from_original_predictions(
        self,
        shift_amount: Optional[List[int]] = [0, 0],
        full_shape: Optional[List[int]] = None,
    ):
        """
        self._original_predictions is converted to a list of prediction.ObjectPrediction and set to
        self._object_prediction_list.

        Args:
            shift_amount: list
                To shift the box and mask predictions from sliced image to full sized image, should be in the form of [shift_x, shift_y]
            full_shape: list
                Size of the full image after shifting, should be in the form of [height, width]
        """
        original_predictions = self._original_predictions
        category_mapping = self.category_mapping

        # parse boxes and masks from predictions
        num_categories = self.num_categories
        if self.has_mask:
            boxes = original_predictions[0]
            masks = original_predictions[1]
        else:
            boxes = original_predictions

        object_prediction_list = []

        # process predictions
        for category_id in range(num_categories):
            category_boxes = boxes[category_id]
            if self.has_mask:
                category_masks = masks[category_id]
            num_category_predictions = len(category_boxes)

            for category_predictions_ind in range(num_category_predictions):
                bbox = category_boxes[category_predictions_ind][:4]
                score = category_boxes[category_predictions_ind][4]
                if self.has_mask:
                    bool_mask = category_masks[category_predictions_ind]
                else:
                    bool_mask = None
                category_name = category_mapping[str(category_id)]

                object_prediction = ObjectPrediction(
                    bbox=bbox,
                    category_id=category_id,
                    score=score,
                    bool_mask=bool_mask,
                    category_name=category_name,
                    shift_amount=shift_amount,
                    full_shape=full_shape,
                )
                object_prediction_list.append(object_prediction)

        self._object_prediction_list = object_prediction_list

    def _create_original_predictions_from_object_prediction_list(
        self,
        object_prediction_list: List[ObjectPrediction],
    ):
        """
        Converts a list of prediction.ObjectPrediction instance to detection model's original prediction format.
        Then returns the converted predictions.
        Can be considered as inverse of _create_object_prediction_list_from_predictions().

        Args:
            object_prediction_list: a list of prediction.ObjectPrediction
        Returns:
            original_predictions: a list of converted predictions in models original output format
        """
        # init variables
        boxes = []
        masks = []
        num_categories = self.num_categories
        category_id_list = np.arange(num_categories)
        category_id_to_bbox = {category_id: [] for category_id in category_id_list}
        category_id_to_mask = {category_id: [] for category_id in category_id_list}
        # form category_to_bbox and category_to_mask dicts from object_prediction_list
        for object_prediction in object_prediction_list:
            category_id = object_prediction.category.id
            # form bbox as 1x5 list [xmin, ymin, xmax, ymax, score]
            bbox = object_prediction.bbox.to_voc_bbox()
            bbox.extend([object_prediction.score.value])
            category_id_to_bbox[category_id].append(np.array(bbox, dtype=np.float32))
            # form 2d bool mask
            if self.has_mask:
                mask = object_prediction.mask.bool_mask
                category_id_to_mask[category_id].append(mask)

        for category_id in category_id_to_bbox.keys():
            if not category_id_to_bbox[category_id]:
                # add 0x5 array to boxes for empty categories
                boxes.append(np.zeros((0, 5), dtype=np.float32))
                if self.has_mask:
                    masks.append([])
            else:
                # form boxes and masks
                boxes.append(np.array(category_id_to_bbox[category_id]))
                if self.has_mask:
                    masks.append(np.array(category_id_to_mask[category_id]))
        # form final output
        if self.has_mask:
            original_predictions = (boxes, masks)
        else:
            original_predictions = boxes

        return original_predictions


class Yolov5DetectionModel(DetectionModel):
    def load_model(self):
        """
        Detection model is initialized and set to self.model.
        """
        try:
            import yolov5
        except ImportError:
            raise ImportError('Please run "pip install -U yolov5" ' "to install YOLOv5 first for YOLOv5 inference.")

        # set model
        try:
            model = yolov5.load(self.model_path, device=self.device)
            self.model = model
        except Exception as e:
            TypeError("model_path is not a valid yolov5 model path: ", e)

        # set category_mapping
        if not self.category_mapping:
            category_mapping = {str(ind): category_name for ind, category_name in enumerate(self.category_names)}
            self.category_mapping = category_mapping

    def perform_inference(self, image: np.ndarray):
        """
        Prediction is performed using self.model and the prediction result is set to self._original_predictions.

        Args:
            image: np.ndarray
                A numpy array that contains the image to be predicted.
        """
        try:
            import yolov5
        except ImportError:
            raise ImportError('Please run "pip install -U yolov5" ' "to install YOLOv5 first for YOLOv5 inference.")

        # Confirm model is loaded
        assert self.model is not None, "Model is not loaded, load it by calling .load_model()"

        prediction_result = self.model(image)

        self._original_predictions = prediction_result

    @property
    def num_categories(self):
        """
        Returns number of categories
        """
        return len(self.model.names)

    @property
    def has_mask(self):
        """
        Returns if model output contains segmentation mask
        """
        has_mask = self.model.with_mask
        return has_mask

    @property
    def category_names(self):
        return self.model.names

    def _create_object_prediction_list_from_original_predictions(
        self,
        shift_amount: Optional[List[int]] = [0, 0],
        full_shape: Optional[List[int]] = None,
    ):
        """
        self._original_predictions is converted to a list of prediction.ObjectPrediction and set to
        self._object_prediction_list.

        Args:
            shift_amount: list
                To shift the box and mask predictions from sliced image to full sized image, should be in the form of [shift_x, shift_y]
            full_shape: list
                Size of the full image after shifting, should be in the form of [height, width]
        """
        original_predictions = self._original_predictions

        # handle only first image (batch=1)
        predictions_in_xyxy_format = original_predictions.xyxy[0]


        '''
        prediction resutls for yolov5 are dispalyed as shwon below

        tensor(
             x left        y top       x bottom    y right          prob      cls id   
        [[3.22013e+02, 3.22174e+02, 3.83605e+02, 3.62460e+02, 8.41712e-01, 2.00000e+00],
        [4.48575e+02, 3.08800e+02, 4.96908e+02, 3.42803e+02, 8.12179e-01, 2.00000e+00],
        [7.64908e+02, 2.60927e+02, 7.94061e+02, 2.83598e+02, 7.08609e-01, 2.00000e+00],
        [6.04125e+02, 2.37290e+02, 6.28566e+02, 2.59425e+02, 6.82913e-01, 2.00000e+00],
        [8.33987e+02, 3.09424e+02, 8.73408e+02, 3.43815e+02, 6.76855e-01, 2.00000e+00],
        [6.17475e+02, 1.95396e+02, 6.41394e+02, 2.18711e+02, 6.58656e-01, 2.00000e+00],
        [3.82050e+02, 2.80206e+02, 4.18973e+02, 3.04781e+02, 5.94899e-01, 2.00000e+00],
        [7.20075e+02, 2.43860e+02, 7.47353e+02, 2.71261e+02, 5.94134e-01, 2.00000e+00],
        [7.00413e+02, 2.34649e+02, 7.21405e+02, 2.50182e+02, 5.67551e-01, 2.00000e+00],
        [7.58754e+02, 2.31797e+02, 7.82001e+02, 2.47634e+02, 4.85633e-01, 2.00000e+00],
        [5.63651e+02, 2.40792e+02, 5.87714e+02, 2.61235e+02, 4.09187e-01, 2.00000e+00],
        [6.49051e+02, 1.98626e+02, 6.70245e+02, 2.14714e+02, 3.38123e-01, 2.00000e+00],
        [7.87568e+02, 1.74923e+02, 1.05628e+03, 5.75780e+02, 2.69586e-01, 6.00000e+00],
        [5.44113e+02, 2.33241e+02, 5.65153e+02, 2.53217e+02, 2.62741e-01, 2.00000e+00]])

        '''

        object_prediction_list = []

        # process predictions
        for prediction in predictions_in_xyxy_format:
            x1 = int(prediction[0].item())
            y1 = int(prediction[1].item())
            x2 = int(prediction[2].item())
            y2 = int(prediction[3].item())
            bbox = [x1, y1, x2, y2]
            score = prediction[4].item()
            category_id = int(prediction[5].item())
            category_name = original_predictions.names[category_id]

            object_prediction = ObjectPrediction(
                bbox=bbox,
                category_id=category_id,
                score=score,
                bool_mask=None,
                category_name=category_name,
                shift_amount=shift_amount,
                full_shape=full_shape,
            )
            object_prediction_list.append(object_prediction)

        self._object_prediction_list = object_prediction_list

    def _create_original_predictions_from_object_prediction_list(
        self,
        object_prediction_list: List[ObjectPrediction],
    ):
        """
        Converts a list of prediction.ObjectPrediction instance to detection model's original
        prediction format. Then returns the converted predictions.
        Can be considered as inverse of _create_object_prediction_list_from_predictions().

        Args:
            object_prediction_list: a list of prediction.ObjectPrediction
        Returns:
            original_predictions: a list of converted predictions in models original output format
        """
        assert self.original_predictions is not None, (
            "self.original_predictions" " cannot be empty, call .perform_inference() first"
        )
        # TODO: implement object_prediction_list to yolov5 format conversion
        NotImplementedError()


class Yolov4DetectionModel(DetectionModel):
    def load_model(self):
        """
        Detection model is initialized and set to self.model.
        """
        try:
            from yolov4.tf import YOLOv4
        except ImportError:
            raise ImportError('Please run "pip install -U yolov4" ' "to install YOLOv4 first for YOLOv4 inference.")

        # set model

        try:
            model = YOLOv4()
            print(" i was here")
            model.config.parse_names(self.names_path)
            print(" i was here 2")
            model.config.parse_cfg(cfg_path=self.config_path)
            print(" i was  here 3")
            model.make_model()
            print(" i was here 4")
            model.load_weights(self.model_path, weights_type="yolo")
            print(" i was  here 5")
            self.model = model
        except Exception as e:
            TypeError("model_path is not a valid yolov4 model path: ", e)

        # set category_mapping

        if not self.category_mapping:
            category_mapping = {str(ind): category_name for ind, category_name in enumerate(self.category_names)}
            self.category_mapping = category_mapping

    def perform_inference(self, image: np.ndarray):
       
        try:
            from yolov4.tf import YOLOv4
        except ImportError:
            raise ImportError('Please run "pip install -U yolov4" ' "to install YOLOv4 first for YOLOv4 inference.")


        # Confirm model is loaded
        assert self.model is not None, "Model is not loaded, load it by calling .load_model()"

        # prediction_result = self.model(image)
        #prediction_result = self.model.inference(media_path="kite.jpg")
        prediction_result = self.model.predict(image,self.prediction_score_threshold)

        self._original_predictions = prediction_result
        print(prediction_result)

    @property
    def num_categories(self):
        
        #Returns number of categories
        
        return len(self.model.config.names)

    @property
    def has_mask(self):
        
        #Returns if model output contains segmentation mask
        
        has_mask = self.model.with_mask
        return has_mask

    @property
    def category_names(self):
        return list(self.model.config.names.values())

    @property
    def num_categories(self):
        """
        Returns number of categories
        """
        return len(self.model.config.names)

    @property
    def has_mask(self):
        """
        Returns if model output contains segmentation mask
        """
        has_mask = self.model.with_mask
        return has_mask

    @property
    def category_names(self):
        return list(self.model.config.names.values())

    def _create_object_prediction_list_from_original_predictions(
        self,
        shift_amount: Optional[List[int]] = [0, 0],
        full_shape: Optional[List[int]] = None,):
        """
        self._original_predictions is converted to a list of prediction.ObjectPrediction and set to
        self._object_prediction_list.

        Args:
            shift_amount: list
                To shift the box and mask predictions from sliced image to full sized image, should be in the form of [shift_x, shift_y]
            full_shape: list
                Size of the full image after shifting, should be in the form of [height, width]
        """
        original_predictions = self._original_predictions

        image_height = original_predictions[len(original_predictions)-1][0]
        image_width = original_predictions[len(original_predictions)-1][1]
        
        object_prediction_list = []

        # process predictions
        for prediction in original_predictions[:-1]:
            
            bbox = [
            (prediction[0]- prediction[1])*image_width,
            (prediction[1]-prediction[3])*image_height,
            (prediction[0] + prediction[1])*image_width,
            (prediction[1] + prediction[3])*image_height] 

            score = prediction[5].item()
            category_id = int(prediction[4].item())
            print("I was here ")
            category_name = original_predictions.names[category_id] #this is where the error is occured
            print("iwas here too")

            object_prediction = ObjectPrediction(
                bbox=bbox,
                category_id=category_id,
                score=score,
                bool_mask=None,
                category_name=category_name,
                shift_amount=shift_amount,
                full_shape=full_shape,
            )
            object_prediction_list.append(object_prediction)

        self._object_prediction_list = object_prediction_list
        print(object_prediction_list)

If someone could help figure this out, thank you!

[Linaom1214]

this is clear. let this function return a list.

   593     def category_names(self):
--> 594         return list(self.model.config.names.values())

you can change this function

    @property
    def category_names(self):
        if type(self.model.CLASSES) == str:
            # https://github.com/open-mmlab/mmdetection/pull/4973
            return (self.model.CLASSES,)
        else:
            return self.model.CLASSES

[asluborski98]

changing the function to that still gives me this error

AttributeError                            Traceback (most recent call last)
[<ipython-input-4-9f1e973d5392>](https://localhost:8080/#) in <module>()
      2     model_path=yolov4_model_path,
      3     config_path = yolov4_config_path,
----> 4     device="cpu", # or 'cuda:0',
      5 )

2 frames
[/usr/local/lib/python3.7/dist-packages/sahi/model.py](https://localhost:8080/#) in category_names(self)
    546     @property
    547     def category_names(self):
--> 548         if type(self.model.CLASSES) == str:
    549             # https://github.com/open-mmlab/mmdetection/pull/4973
    550             return (self.model.CLASSES,)

AttributeError: 'NoneType' object has no attribute 'CLASSES'

I try changing code but still cannot get to work it seems all ok
I am using a custom trained model

[Linaom1214]

AttributeError: 'NoneType' object has no attribute 'CLASSES' !!!!! be careful

[fcakyon]

@asluborski98 this error means you are not able to initialize the model. You have to be sure you are properly loading the weights from the given path.

[asluborski98]

Must have been something wrong with my colab path as I got it working locally.