08_associazione_variabili_categoriche.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R5Nbe1_9BhNw"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/alesaccoia/IULM_DDM2324_Notebooks/blob/main/08_associazione_variabili_categoriche.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eIHHppmYBhNz"
      },
      "source": [
        "# Misurare l'associazione fra variabili categoriche"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "g5-l_FkHBhN0",
        "outputId": "29e318e1-c8b0-4ae8-ae5d-5c516b741e50",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2024-04-09 13:23:08--  https://github.com/LeoLin72/IULM_DDM2324_Notebooks/raw/main/data/chisquare_preferenza_genere_prodotto.xlsx\n",
            "Resolving github.com (github.com)... 140.82.114.3\n",
            "Connecting to github.com (github.com)|140.82.114.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://raw.githubusercontent.com/LeoLin72/IULM_DDM2324_Notebooks/main/data/chisquare_preferenza_genere_prodotto.xlsx [following]\n",
            "--2024-04-09 13:23:08--  https://raw.githubusercontent.com/LeoLin72/IULM_DDM2324_Notebooks/main/data/chisquare_preferenza_genere_prodotto.xlsx\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 49025 (48K) [application/octet-stream]\n",
            "Saving to: ‘chisquare_preferenza_genere_prodotto.xlsx.1’\n",
            "\n",
            "chisquare_preferenz 100%[===================>]  47.88K  --.-KB/s    in 0.01s   \n",
            "\n",
            "2024-04-09 13:23:08 (4.25 MB/s) - ‘chisquare_preferenza_genere_prodotto.xlsx.1’ saved [49025/49025]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from scipy.stats import chi2_contingency\n",
        "from scipy.stats.contingency import margins\n",
        "\n",
        "!wget \"https://github.com/LeoLin72/IULM_DDM2324_Notebooks/raw/main/data/chisquare_preferenza_genere_prodotto.xlsx\"\n",
        "\n",
        "# Carico i dati dal file Excel, specificando quale worksheet\n",
        "df = pd.read_excel('chisquare_preferenza_genere_prodotto.xlsx', sheet_name=\"Data\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_KWT2kQkBhN1"
      },
      "source": [
        "## Tabella di contingenza"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "1kLlSYBDBhN1",
        "outputId": "4f02c338-e4eb-49c7-a54b-2b2890a05a6a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "productid    1     2    3  Totale\n",
            "gender                           \n",
            "1          495   590  272    1357\n",
            "2          330   498  265    1093\n",
            "Totale     825  1088  537    2450\n"
          ]
        }
      ],
      "source": [
        "# Tabelle di contingenza:\n",
        "contingency_table = pd.crosstab(df['gender'], df['productid'], margins=True, margins_name=\"Totale\")\n",
        "print(contingency_table)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rAMKckLJBhN2"
      },
      "source": [
        "## Probabilità Marginale\n",
        "È la probabilità di un singolo evento senza considerare l'effetto di un altro evento. Si tratta essenzialmente della probabilità \"a margine\" di un evento particolare, quindi senza condizionarlo su un altro.\n",
        "Ad esempio, considera una tabella di contingenza tra Genere (Maschio/Femmina) e Prodotto (1/2/3). La probabilità marginale che una persona scelta a caso sia un maschio (indipendentemente dal fatto che fumi o meno) si calcola sommando tutte le osservazioni dei maschi e dividendo per il totale delle osservazioni."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "oqyQBcRvBhN2",
        "outputId": "e19f6860-6537-41e7-ee56-b673993f7c43",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Probabilità marginale di genere:\n",
            " gender\n",
            "1    0.553878\n",
            "2    0.446122\n",
            "Name: count, dtype: float64\n",
            "\n",
            "Probabilità marginale di prodotto scelto:\n",
            " productid\n",
            "2    0.444082\n",
            "1    0.336735\n",
            "3    0.219184\n",
            "Name: count, dtype: float64\n"
          ]
        }
      ],
      "source": [
        "# Probabilità marginale:\n",
        "prob_gender = df['gender'].value_counts() / len(df)\n",
        "prob_productid = df['productid'].value_counts() / len(df)\n",
        "print(\"\\nProbabilità marginale di genere:\\n\", prob_gender)\n",
        "print(\"\\nProbabilità marginale di prodotto scelto:\\n\", prob_productid)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nQOogKf_BhN3"
      },
      "source": [
        "# Test del chi-quadrato"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "p3T9IjVoBhN3"
      },
      "outputs": [],
      "source": [
        "# Ricalcolo la tabelle di contingenza SENZA MARGINI\n",
        "contingency_table = pd.crosstab(df['gender'], df['productid'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "b9uyALSrBhN3",
        "outputId": "d640f373-e818-462c-90da-a2d7ccdb06dc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Valore chi2: 12.569256167968339\n",
            "P-value: 0.0018647503713553623\n",
            "\n",
            "Tabella delle frequenze attese:\n",
            " productid          1           2           3\n",
            "gender                                      \n",
            "1          456.94898  602.618776  297.432245\n",
            "2          368.05102  485.381224  239.567755\n"
          ]
        }
      ],
      "source": [
        "# Test chi square: utilizzo i -1 per non prendere l'ultima riga e l'ultima colonna\n",
        "chi2, p, _, expected = chi2_contingency(contingency_table)\n",
        "\n",
        "print(\"\\nValore chi2:\", chi2)\n",
        "print(\"P-value:\", p)\n",
        "print(\"\\nTabella delle frequenze attese:\\n\", pd.DataFrame(expected, columns=contingency_table.columns, index=contingency_table.index))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iVL0kvQ-BhN3"
      },
      "source": [
        "## Lo Scostamento Percentuale\n",
        "\n",
        "Calcoliamo quindi Lo Scostamento Percentuale tra Frequenza Osservata e Attesa"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "0U2vKocfBhN3",
        "outputId": "1a09c665-7e70-42a5-82de-cb0403b912c8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[  8.32719234  -2.09398977  -8.55060113]\n",
            " [-10.33851784   2.59976589  10.61588814]]\n"
          ]
        }
      ],
      "source": [
        "print(100.0 * (contingency_table.values - expected) / expected)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QpjGspMdBhN4"
      },
      "source": [
        "## Resti Standardizzati"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "xl3Llq_wBhN4",
        "outputId": "58af90e7-71c6-4df4-e0b6-b196eed5347f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Standardized Residuals:\n",
            " [[ 3.27236525 -1.03219876 -2.49855659]\n",
            " [-3.27236525  1.03219876  2.49855659]]\n"
          ]
        }
      ],
      "source": [
        "# Questa funzione calcola i resti standardizzati\n",
        "def stdres(observed, expected):\n",
        "    n = observed.sum()\n",
        "    rsum, csum = margins(observed)\n",
        "    # With integers, the calculation\n",
        "    #     csum * rsum * (n - rsum) * (n - csum)\n",
        "    # might overflow, so convert rsum and csum to floating point.\n",
        "    rsum = rsum.astype(np.float64)\n",
        "    csum = csum.astype(np.float64)\n",
        "    v = csum * rsum * (n - rsum) * (n - csum) / n**3\n",
        "    return (observed - expected) / np.sqrt(v)\n",
        "\n",
        "residuals = stdres(contingency_table.values, expected)\n",
        "print(\"\\nStandardized Residuals:\\n\", residuals)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.17"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}