Step 6: Exploratory Data Analysis

概覽

為什麼 EDA 是任何統計分析的第一步？

John Tukey 在 1977 年的 Exploratory Data Analysis 提出「看圖優於檢定」的哲學：histogram、Q-Q plot、boxplot 可以一眼看出資料分佈、變異與異常，這些是 p 值無法傳達的資訊。Anscombe's quartet 就是經典例證——四組資料的均值、變異、相關係數、回歸線完全相同，但散點圖形態截然不同。

2025 年 Knief & Forstmeier (R. Soc. Open Sci.) 提醒：常態性檢定不該作為「自動」決策依據。樣本量 > 300 時 Shapiro-Wilk 對任何微小偏離都會 p<0.05，但此時 CLT 已使 t-test/ANOVA 高度穩健。

John Tukey's 1977 Exploratory Data Analysis championed "plot before test": histograms, Q-Q plots, boxplots reveal distribution, variation, and anomalies that p-values can never convey. Anscombe's quartet is the classic: four datasets with identical mean, variance, correlation, and regression line — but completely different scatter patterns.

Knief & Forstmeier 2025 (R. Soc. Open Sci.) caution: normality tests should NOT be automated gatekeepers. At n > 300, Shapiro-Wilk flags trivial deviations as p < 0.05, but the CLT already makes t-tests / ANOVA highly robust there.

💡

EDA 三問：(1) 分布長相？(2) 有沒有 outlier？是否可疑？(3) 變數之間關聯如何？這三問都應用「圖」回答，不是用 p 值。Three EDA questions: (1) What does the distribution look like? (2) Are there outliers — and are they suspicious? (3) How do variables relate? Answer with plots, not p-values.

核心概念

一、描述統計與分布診斷

📈

集中趨勢

Mean（敏感於 outlier）、Median（穩健）、Mode。偏態資料優先 median。

Mean (outlier-sensitive), Median (robust), Mode. Skewed data → prefer median.

📊

變異

SD（敏感）、IQR（穩健）、MAD（中位數絕對偏差，最穩健）。Leys 2013 建議離群值用 MAD ± 3 標準化。

SD (sensitive), IQR (robust), MAD (median absolute deviation, most robust). Leys 2013 recommends standardizing outliers via MAD ± 3.

📐

常態性檢定

Shapiro-Wilk 最強 (Razali 2011)；KS 已過時，改用 Lilliefors；Anderson-Darling 對尾部敏感。Q-Q plot 永遠比檢定值有資訊量。

Shapiro-Wilk most powerful (Razali 2011); plain KS outdated → Lilliefors; Anderson-Darling sensitive to tails. A Q-Q plot is always more informative than a test value.

🔍

離群值偵測

Tukey fence: Q1−1.5·IQR ~ Q3+1.5·IQR；MAD ± 3；Grubbs test (僅單一 outlier)。偵測 ≠ 刪除，需先檢查是否技術錯誤。

Tukey fence: Q1−1.5·IQR ~ Q3+1.5·IQR; MAD ± 3; Grubbs test (single outlier only). Detection ≠ removal — check for technical errors first.

互動模擬

樣本量如何欺騙常態性檢定

調整樣本量與偏度，觀察 Shapiro-Wilk 的 p 值如何變動。關鍵發現：樣本量大時，連微小偏離都會「顯著」，反而誤導你改用無母數檢定。

Adjust sample size and skewness, watch Shapiro-Wilk p change. Key: at large n, trivial deviations become "significant," misleading you into switching to non-parametric tests unnecessarily.

樣本量 n50

偏度0.5

——

程式碼

二、EDA 標準流程

library(tidyverse); library(skimr); library(performance)
df <- read_csv("data.csv")

# 1. 一覽：分布、缺失、unique count
skim(df)

# 2. 視覺化：histogram + density + boxplot
ggplot(df, aes(value, fill=group)) +
  geom_histogram(aes(y=after_stat(density)), bins=30, alpha=.5) +
  geom_density(alpha=.4) +
  facet_wrap(~group, scales="free_y")

# 3. Q-Q plot（比 Shapiro 更具資訊量）
ggplot(df, aes(sample=value)) + stat_qq() + stat_qq_line() + facet_wrap(~group)

# 4. 常態性檢定（搭配 Q-Q）
df %>% group_by(group) %>%
  summarise(p_shapiro = shapiro.test(value)$p.value)

# 5. 離群值偵測（MAD）
df %>% mutate(z_mad = abs(value - median(value)) / mad(value),
            outlier = z_mad > 3)

import pandas as pd, numpy as np
import seaborn as sns, matplotlib.pyplot as plt
from scipy import stats
import pingouin as pg

df = pd.read_csv("data.csv")

# 1. 一覽
print(df.describe())
print(df.isna().sum())

# 2. 視覺化（distplot 已棄用，改 histplot+kdeplot）
sns.displot(df, x="value", hue="group", kind="hist", kde=True, stat="density")

# 3. Q-Q plot
fig, ax = plt.subplots(1, 2, figsize=(10,4))
for i, (g, sub) in enumerate(df.groupby("group")):
    stats.probplot(sub.value, dist="norm", plot=ax[i])

# 4. 常態性檢定 — pingouin 一行
print(pg.normality(df, dv="value", group="group", method="shapiro"))

# 5. MAD 離群值
mad = stats.median_abs_deviation(df.value)
df["z_mad"] = np.abs(df.value - df.value.median()) / mad
df["outlier"] = df.z_mad > 3

常見陷阱

三、EDA 常見錯誤

❌ 只看 mean ± SD

不檢視分布就報摘要統計 = Anscombe 警告。永遠先畫圖。

Reporting summary stats without inspecting distribution = Anscombe's warning. Always plot first.

❌ 把 Shapiro 當路標

n=500 拿到 p=0.01 就改無母數，可能犧牲檢定效力。先看 Q-Q 與偏度。

At n=500, p=0.01 doesn't mean switch to non-parametric — may lose power. Look at the Q-Q and skewness.

❌ 悄悄刪離群值

刪除任何資料點必須在方法說明、附理由（技術錯誤？）並做 sensitivity analysis。

Any removal must be in Methods, justified (technical error?), and accompanied by a sensitivity analysis.

❌ 忽略缺失機制

MCAR / MAR / MNAR 影響 imputation 合法性。> 20% 缺失需慎重處理。

MCAR / MAR / MNAR determine imputation validity. > 20% missing requires careful handling.

🎯 章末小測驗

1. 最穩健的離群值偵測？

Mean ± 3·SD

MAD ± 3

Mean ± 2·SE

Pearson r

2. n=1000, Shapiro p=0.01 應如何解讀？

嚴重非常態必改

微小偏離，看 Q-Q

樣本太多

重採

3. Anscombe's quartet 的教訓？

n ≥ 30

畫圖優於摘要

R only

迴歸線不可信