{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stats sub-module\n",
    "\n",
    "The ```acgc.stats``` sub-module is a collection of statistical methods. See ```help(acgc.stats)``` for complete list of methods.\n",
    "\n",
    "The most commonly used functions include BivariateStatistics, line fitting methods (sma, sen, york), and weighted statistics (wmean, wmedian, wcov, wcorr, etc.)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from acgc import stats as mystats\n",
    "from acgc import figstyle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BivariateStatistics\n",
    "This function quickly computes many statistical measures of the relationship between two populations.\n",
    "\n",
    "As an example, we will create samples of two populations with a linear relationship between them, which is obscured by noise added to both "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate two sample populations\n",
    "n = 100\n",
    "x_true = np.random.rand(n) + 2\n",
    "y_true = 2*x_true + 1 \n",
    "\n",
    "# Add noise with normal distribution\n",
    "x = x_true + np.random.randn(n)*0.1\n",
    "y = y_true + np.random.randn(n)*0.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `BivariateStatistics` to compute statistical relationships and `summary()` to display some of the most commonly used statistics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD    = 3.4859\n",
      "MAD   = 3.4859\n",
      "RMD   = 1.3778\n",
      "RMAD  = 1.3778\n",
      "MRD   = 1.3875\n",
      "R2    = 0.7101\n",
      "slope = 2.0875\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Compute bivariate statistics\n",
    "result = mystats.BivariateStatistics(x,y)\n",
    "\n",
    "# Display a table showing some common statistical measures of population differences\n",
    "print(result.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The abbreviations seen in the `summary()` output are defined in the documentation for the `result` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on BivariateStatistics in module acgc.stats.bivariate object:\n",
      "\n",
      "class BivariateStatistics(builtins.object)\n",
      " |  BivariateStatistics(x, y, w=None, dropna=False, data=None)\n",
      " |  \n",
      " |  A suite of common statistics to quantify bivariate relationships\n",
      " |  \n",
      " |  Class method 'summary' provides a formatted summary of these statistics\n",
      " |  \n",
      " |  Attributes\n",
      " |  ----------\n",
      " |  xmean, ymean : float\n",
      " |      mean of x and y variables\n",
      " |  xmedian, ymedian :float\n",
      " |      median of x and y variables\n",
      " |  xstd, ystd : float\n",
      " |      standard deviation of x and y variables\n",
      " |  mean_difference, md : float\n",
      " |      ymean - xmean\n",
      " |  mean_absolute_difference, mad : float\n",
      " |      mean( |y-x| )\n",
      " |  relative_mean_difference, rmd : float\n",
      " |      md / xmean\n",
      " |  relative_mean_absolute_difference, rmad :float\n",
      " |      mad / xmean\n",
      " |  standardized_mean_difference, smd : float\n",
      " |      md / xstd\n",
      " |  standardized_mean_absolute_difference, smad : float\n",
      " |      mad /xstd\n",
      " |  mean_relative_difference, mrd : float\n",
      " |      mean(y/x) - 1\n",
      " |  median_difference, medd : float\n",
      " |      median(y-x)\n",
      " |  median_absolute_difference, medad : float\n",
      " |      median(|y-x|)\n",
      " |  relative_median_difference, rmedd : float\n",
      " |      median(y-x) / xmedian\n",
      " |  relative_median_absolute_difference, rmedad : float\n",
      " |      median(|y-x|) / xmedian\n",
      " |  median_relative_difference, medianrd, medrd : float\n",
      " |      median(y/x)-1\n",
      " |  normalized_mean_bias_factor, nmbf : float\n",
      " |      see `nmbf` \n",
      " |  normalized_mean_absolute_error_factor, nmaef : float\n",
      " |      see `nmaef`\n",
      " |  root_mean_square_difference, rmsd : float\n",
      " |      $\\sqrt{ \\langle (y - x)^2 \\rangle }$\n",
      " |  covariance : float\n",
      " |      cov(x,y)\n",
      " |  correlation_pearson, correlation, pearsonr, R, r : float\n",
      " |      Pearson linear correlation coefficient \n",
      " |  correlation_spearman, spearmanr : float\n",
      " |      Spearman, non-parametric rank correlation coefficient\n",
      " |  R2, r2 : float\n",
      " |      Linear coefficient of determination, $R^2$\n",
      " |  \n",
      " |  Methods defined here:\n",
      " |  \n",
      " |  __getitem__(self, key)\n",
      " |      Accesses attribute values via object['key']\n",
      " |  \n",
      " |  __init__(self, x, y, w=None, dropna=False, data=None)\n",
      " |      Compute suite of bivariate statistics during initialization\n",
      " |      \n",
      " |      Statistic values are saved in attributes.\n",
      " |      CAUTION: Weights w are ignored except in SMA fit\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      x : ndarray or str\n",
      " |          independent variable values\n",
      " |      y : ndarray or str\n",
      " |          dependent variable values, same size as x\n",
      " |      w : ndarray or str, optional\n",
      " |          weights for points (x,y), same size as x and y\n",
      " |      dropna : bool, optional (default=False)\n",
      " |          drops NaN values from x, y, and w\n",
      " |      data : dict-like, optional\n",
      " |          if x, y, or w are str, then they should be keys in data\n",
      " |  \n",
      " |  fitline(self, method='sma', intercept=True, **kwargs)\n",
      " |      Compute bivariate line fit\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      method : str\n",
      " |          line fitting method: sma (default), ols, wls, York, sen, siegel\n",
      " |      intercept : bool\n",
      " |          defines whether non-zero intercept should be fitted\n",
      " |      **kwargs \n",
      " |          passed to `acgc.stats.sma` (e.g. robust=True)\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      result : dict\n",
      " |          dictionary with keys:\n",
      " |          - slope (float)\n",
      " |              slope of fitted line\n",
      " |          - intercept (float)\n",
      " |              intercept of fitted line\n",
      " |          - fittedvalues (array (N,))\n",
      " |              values on fit line\n",
      " |          - residuals (array (N,))\n",
      " |              residual from fit line\n",
      " |  \n",
      " |  intercept(self, method='sma', intercept=True, **kwargs)\n",
      " |      Compute intercept of bivariate line fit\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      method : str\n",
      " |          line fitting method: sma (default) or ols\n",
      " |      intercept : bool\n",
      " |          defines whether non-zero intercept should be fitted\n",
      " |      **kwargs \n",
      " |          passed to `fitline`\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      intercept : float\n",
      " |          value of y intercept\n",
      " |  \n",
      " |  slope(self, method='sma', intercept=True, **kwargs)\n",
      " |      Compute slope of bivariate line fit\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      method : str\n",
      " |          line fitting method: sma (default), ols, wls\n",
      " |      intercept : bool\n",
      " |          defines whether non-zero intercept should be fitted\n",
      " |      **kwargs \n",
      " |          passed to `fitline`\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      slope : float\n",
      " |          value of y intercept\n",
      " |  \n",
      " |  summary(self, variables=None, fitline_kw=None, floatformat='{:.4f}', floatformat_fiteqn=None, stringlength=None)\n",
      " |      Summarize bivariate statistics\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      vars : list or str, default='common'\n",
      " |          names of attribute variables to include in summary\n",
      " |          names are case insensitive            \n",
      " |          The following strings are also accepted in place of a list \n",
      " |              \"all\" (displays all variables)\n",
      " |              \"common\" (displays all measures of mean difference)\n",
      " |      floatformat : str, default='{:.4f}'\n",
      " |          format specifier for floating point values\n",
      " |      floatformat_fiteqn : str, default=floatformat\n",
      " |          format specifier for slope and intercept (a,b) in y = a x + b\n",
      " |      stringlength : int, default=None\n",
      " |          length of the variables on output\n",
      " |          default (None) is to use the length of the longest variable name\n",
      " |      fitline_kw : dict, default=None\n",
      " |          keywords passed to `fitline`\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      summary : str\n",
      " |          names and values of variables\n",
      " |  \n",
      " |  summary_dict(self, variables=None, fitline_kw=None, floatformat_fiteqn='{:.3f}')\n",
      " |      Summarize bivariate statistics into a dict\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      vars : list or str, default='common'\n",
      " |          names of attribute variables to include in summary\n",
      " |          names are case insensitive            \n",
      " |          The following strings are also accepted in place of a list \n",
      " |              \"all\" (displays all variables)\n",
      " |              \"common\" (displays all measures of mean difference)\n",
      " |      fitline_kw : dict, default=None)\n",
      " |          keywords passed to self.fitline()\n",
      " |      \n",
      " |      Returns\n",
      " |      -------\n",
      " |      summary : dict\n",
      " |          names and values of variables\n",
      " |  \n",
      " |  summary_fig_inset(self, ax, variables=None, fitline_kw=None, floatformat='{:.3f}', floatformat_fiteqn=None, loc=None, loc_units='axes', **kwargs)\n",
      " |      Display bivariate statistics as a table inset on a plot axis\n",
      " |      \n",
      " |      Parameters\n",
      " |      ----------\n",
      " |      ax : matplotlib.Figure.Axis \n",
      " |          axis where the table will be displayed\n",
      " |      variables : list or str, default='common'\n",
      " |          names of attribute variables to include in summary\n",
      " |          names are case insensitive            \n",
      " |          The following strings are also accepted in place of a list \n",
      " |              \"all\" (displays all variables)\n",
      " |              \"common\" (displays all measures of mean difference)\n",
      " |      fitline_kw : dict, default=None\n",
      " |          keywords passed to `fitline`\n",
      " |      floatformat : str, default='{:.3f}'\n",
      " |          format specifier for floating point values\n",
      " |      floatformat_fiteqn : str, default=floatformat\n",
      " |          format specifier for slope and intercept (a,b) in y = a x + b\n",
      " |      loc : tuple (x0,y0), default=(0.85, 0.05)\n",
      " |          location on the axis where the table will be drawn\n",
      " |          can be in data units or axes units [0-1]\n",
      " |      loc_units : {'axes' (default), 'data'}\n",
      " |          specifies whether loc has 'data' units or 'axes' units [0-1]\n",
      " |                  \n",
      " |      Returns\n",
      " |      -------\n",
      " |      text1, text2 : matplotlib text object\n",
      " |          Artist for the two text boxes\n",
      " |  \n",
      " |  ----------------------------------------------------------------------\n",
      " |  Data descriptors defined here:\n",
      " |  \n",
      " |  __dict__\n",
      " |      dictionary for instance variables\n",
      " |  \n",
      " |  __weakref__\n",
      " |      list of weak references to the object\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Show documentation including abbreviations\n",
    "help(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Users can specify which variables they want to include in the summary output and the type of line fitting that they want to use (SMA, OLS, WLS, Theil-Sen, York)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MD        = 3.4859\n",
      "MedD      = 3.4925\n",
      "slope     = 1.7590\n",
      "intercept = 1.5656\n",
      "R         = 0.8426\n",
      "y_OLS     = 1.7590 x + 1.5656\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Specify exactly which variables to include in the summary\n",
    "#   mean difference (MD)\n",
    "#   Median difference (MedD)\n",
    "#   slope (SMA method)\n",
    "#   intercept (SMA method)\n",
    "#   Pearson correlation (R)\n",
    "display_vars = ['MD','MedD','slope','intercept','R','fitline']\n",
    "\n",
    "# Summarize, using OLS line fitting\n",
    "print(result.summary(display_vars, \n",
    "    fitline_kw={'method':'OLS'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Show a figure with summary statistics as an inset table\n",
    "\n",
    "The summary table can be written on a figure. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 64460 characters]",
      "text/plain": [
       "<Figure size 400x330 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 341,
       "width": 411
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot data\n",
    "ax = plt.subplot()\n",
    "ax.scatter(x,y)\n",
    "\n",
    "# Add text displaying slope and R2\n",
    "t=result.summary_fig_inset(ax,\n",
    "                         ['MD','RMD','R2','fitline'],\n",
    "                         floatformat='{:5.2f}',\n",
    "                         loc=(0.67,0.05), loc_units='axes',\n",
    "                         fontsize=9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weighted statistics\n",
    "\n",
    "Weighted versions of mean, standard deviation, covariance, etc. are available. The functions names follow the numpy functions prefixed with 'w'. For example 'wmean' is the weighted mean and 'wstd' is the weighted standard deviation. When weights are equal to 1, the result is identical to the unweighted statistic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Weighted mean: 2.5788546680170104\n",
      "Weighted std: 0.28891229712885785\n",
      "Weighted covariance: 0.1490392885381567\n",
      "Standard covariance: 0.16981371919730262\n"
     ]
    }
   ],
   "source": [
    "# Generate hypothetical weights. These should be positive\n",
    "w = (np.random.randn(n)+1)**2\n",
    "\n",
    "# Print some weighted statistics\n",
    "print('Weighted mean:', mystats.wmean(x,w) )\n",
    "print('Weighted std:',  mystats.wstd(x,w))\n",
    "print('Weighted covariance:', mystats.wcov(x,y,w))\n",
    "print('Standard covariance:', mystats.wcov(x,y,np.ones_like(x)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the `robust` keyword, the sample weights are computed internally to reduce the influence of extreme outliers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standard mean: 1.7454291043409962\n",
      "Robust mean: 2.693732298523021\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/cdholmes/anaconda3/envs/main/lib/python3.11/site-packages/sklearn/covariance/_robust_covariance.py:747: UserWarning: The covariance matrix associated to your dataset is not full rank\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "# Add some extreme outliers\n",
    "xoutliers = x + np.random.standard_cauchy(n)\n",
    "\n",
    "# The outliers affect the mean much more than the robust mean\n",
    "print('Standard mean:', np.mean(xoutliers))\n",
    "print('Robust mean:', mystats.wmean(xoutliers,robust=True))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "general",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
