Step 14: AI / ML for Quantification

概覽

為什麼 ML 結果常無法臨床落地？

2024 年 TRIPOD+AI (Collins et al., BMJ) 取代 TRIPOD 2015，27 項清單同時涵蓋傳統回歸與機器學習，新增公平性 (fairness)、資料來源、訓練流程透明度。2025 年 STARD-AI (Salim et al., Nature Medicine) 為診斷準確度 AI 研究新增 18 項要件。CLAIM (Mongan 2020) 則特化於醫學影像 AI。

類別不平衡（疾病盛行率 < 10%）時 ROC-AUC 過度樂觀，應改用 PR-AUC (Saito & Rehmsmeier 2015, PLOS ONE)。校準（calibration）是 Van Calster 2019 (BMC Medicine) 強調的「預測分析的阿基里斯腳跟」。SHAP (Lundberg & Lee 2017) 提供模型解釋，但不可詮釋為因果效應。

TRIPOD+AI 2024 (Collins et al., BMJ) replaces TRIPOD 2015 — its 27-item checklist covers regression and ML alike, adding fairness, data provenance, and training transparency. STARD-AI 2025 (Salim et al., Nature Medicine) adds 18 AI-specific items to STARD. CLAIM (Mongan 2020) is the medical-imaging-AI guideline.

With class imbalance (prevalence < 10%), ROC-AUC is overly optimistic — use PR-AUC (Saito & Rehmsmeier 2015, PLOS ONE). Calibration is Van Calster 2019's "Achilles' heel of predictive analytics." SHAP (Lundberg & Lee 2017) explains predictions but cannot be read as causal.

🚨

四大 data leakage 來源：(1) 在分 train/test 之前做 feature selection / oversampling；(2) 病人層級資料以樣本層級切分；(3) 用同院 k-fold 充當「外部驗證」；(4) 把時間序列以隨機 k-fold 切（應用 time-based split）。Four data-leakage sources: (1) Feature selection / oversampling before train/test split; (2) Patient-level data split at sample level; (3) Same-site k-fold disguised as "external validation"; (4) Time series with random k-fold (use time-based split).

核心概念

一、效能指標選擇

場景	主要指標	輔助	避免
平衡分類	ROC-AUC, F1	Accuracy, Brier score	-
極不平衡	PR-AUC	敏感度+PPV	Accuracy, ROC-AUC alone
迴歸	RMSE, MAE	R², Pearson r	R² alone
校準	Brier, calibration plot	Calibration slope / intercept	Hosmer-Lemeshow alone
多類別	Macro F1, macro AUC	Confusion matrix	Overall accuracy alone

互動模擬

不平衡資料下 ROC 騙人

調整正類盛行率，觀察 ROC-AUC 與 PR-AUC 變化。同一模型在 50% 盛行率與 1% 盛行率下，ROC-AUC 幾乎不變，但 PR-AUC 大幅下降——這才是真實情境的可用度。

Adjust positive prevalence — ROC-AUC barely budges while PR-AUC plummets at low prevalence. PR-AUC reflects real-world utility.

盛行率10%

模型分離度0.85

——

左：ROC | 右：PR

程式碼

二、無 leakage 的 CV 流程

library(tidymodels); library(themis)

# 1. 病人層級分組切分
splits <- group_initial_split(df, group=patient_id, prop=0.8)
train <- training(splits); test <- testing(splits)

# 2. recipe：所有 preprocessing 包進 pipeline（避免 leakage）
rec <- recipe(outcome ~ ., data=train) %>%
  step_normalize(all_numeric_predictors()) %>%
  step_smote(outcome)        # 注意：在 CV fold 內才做 SMOTE

# 3. CV：分組 k-fold
folds <- group_vfold_cv(train, group=patient_id, v=5)

# 4. 模型 + workflow
mod <- logistic_reg(penalty=tune(), mixture=1) %>% set_engine("glmnet")
wf  <- workflow() %>% add_recipe(rec) %>% add_model(mod)

# 5. 調參 + 評估（PR-AUC + ROC + Brier）
res <- tune_grid(wf, resamples=folds, metrics=metric_set(pr_auc, roc_auc, brier_class))
best <- select_best(res, metric="pr_auc")

# 6. 外部驗證（不同醫院/不同年份）
final <- finalize_workflow(wf, best) %>% fit(train)
external_metrics <- final %>% predict(ext_df, type="prob") %>% bind_cols(ext_df)

from sklearn.model_selection import GroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score, average_precision_score, brier_score_loss
from imblearn.pipeline import Pipeline as ImbPipeline
from imblearn.over_sampling import SMOTE

# 1. 病人層級分組 CV
gkf = GroupKFold(n_splits=5)
groups = df["patient_id"]

# 2. Pipeline：scaling + SMOTE + model 一起包，避免 fold 間 leakage
pipe = ImbPipeline([
    ("scaler", StandardScaler()),
    ("smote",  SMOTE(random_state=0)),       # 在 fold 內才合成樣本
    ("clf",    LogisticRegression(penalty="l1", solver="saga", max_iter=5000))
])

# 3. CV 評估
from sklearn.model_selection import cross_validate
res = cross_validate(pipe, X, y, groups=groups, cv=gkf,
                     scoring=["roc_auc","average_precision","neg_brier_score"])

# 4. 訓練全集 → 外部驗證
pipe.fit(X, y)
prob = pipe.predict_proba(X_ext)[:, 1]
print("External PR-AUC:", average_precision_score(y_ext, prob))
print("Brier:", brier_score_loss(y_ext, prob))

# 5. SHAP 解釋（小心：相關 ≠ 因果）
import shap
expl = shap.LinearExplainer(pipe.named_steps["clf"], X)
sv   = expl(X[:100])
shap.summary_plot(sv, X[:100])

常見陷阱

三、生醫 ML 常見錯誤

❌ 先 preprocess 才切

把整體資料一次 scaling/PCA/SMOTE 再切 train/test → test 已「看過」train 統計，AUC 被高估。

Whole-set scaling/PCA/SMOTE before split → test has "seen" train statistics; AUC inflated.

❌ 同院 CV 當外部

「外部」必須是不同醫院、不同時期或不同人群。同院只能稱 internal validation。

"External" requires a different hospital, era, or population. Same-site CV is internal validation only.

❌ SHAP 當因果

SHAP 解釋的是模型如何使用特徵，不是特徵對結果的因果效應。需 do-calculus / 介入實驗才能談因果。

SHAP explains model use, not causal effect on the outcome. Causality requires do-calculus or interventional experiments.

❌ 忽略 calibration

AUC 高不等於預測機率可信。臨床決策需 reliability diagram + Brier score；必要時做 Platt / isotonic 校準。

High AUC ≠ trustworthy probabilities. Clinical decisions need reliability diagrams + Brier; apply Platt / isotonic if miscalibrated.

🎯 章末小測驗

1. 盛行率 2% 用？

Accuracy

ROC-AUC alone

PR-AUC

R²

2. 2024 ML 模型最新規範？

CONSORT-AI

TRIPOD+AI (BMJ 2024)

PRISMA-DTA

CLAIM

3. 避免 leakage 的 CV？

先 SMOTE

Pipeline 內處理

隨機切病人

只 train