PROTON+: all results

Every table of the preprint and its supplement, per benchmark. AUROC, AUPR and FPR@95 in %, online detectors as the mean over five stream orders. Generated by scripts/build_site.py from the outputs of proton run. Back to the project page.

Ophthalmology

Zero-shot OOD detection (static scores, PROTON, PROTON+)

MethodSem AUCSem APRSem FPRCov AUCCov APRCov FPRFar AUCFar APRFar FPR
Max-Logits54.054.179.327.923.388.083.976.463.0
Energy51.851.579.727.223.288.083.976.562.7
GL-MCM75.875.686.068.945.369.378.562.475.0
MCM75.875.686.068.945.269.778.562.475.0
TAG-MCM68.470.588.730.123.893.772.247.260.3
MCM (L=1)66.565.084.784.266.347.351.838.498.3
MCM (L=5)58.356.587.066.542.565.754.433.574.3
PROTON (ours)68.072.592.746.729.483.686.876.255.0
Δ vs MCM-7.7-3.16.7-22.1-15.813.98.313.7-20.0

PROTON mean ± std over 5 stream orders: semantic: 68.0±0.9, covariate: 46.7±1.8, far: 86.8±3.6

PROTON+ against PROTON (components, ID fraction, sensitivity, base scores)

VariantSem AUCCov AUCFar AUCSem FPRCov FPRFar FPRt_calib
MCM75.868.978.586.069.775.0nan
PROTON (paper)68.046.786.892.783.655.014
+ z-scored fusion69.450.789.091.377.732.314
+ consistency admission70.640.192.683.089.518.614
+ domain-anchored prototypes (PROTON+)67.385.393.475.831.515.414
domain-anchored only64.086.394.384.933.232.914
consistency only68.930.591.590.195.135.614
PROTON+ max-fusion65.786.693.285.628.015.614
PROTON+ rejection-driven alpha64.486.993.675.929.915.514
S_proto only (PROTON bank)62.743.790.591.680.847.014
S_proto only (PROTON+ bank)57.090.497.978.726.08.914
PROTON, oracle ID-only bank76.893.194.591.124.126.121
PROTON+, oracle ID-only bank73.690.493.571.312.915.321
rhoPROTON+ Sem/Cov/FarPROTON Sem/Cov/FarMCM Sem/Cov/Far
90%70.6 / 88.5 / 90.973.1 / 85.5 / 90.476.2 / 67.7 / 75.5
70%68.7 / 74.7 / 92.971.2 / 47.7 / 87.474.8 / 69.0 / 79.0
50%67.4 / 22.3 / 36.768.2 / 34.0 / 55.375.8 / 69.1 / 78.4
30%64.5 / 28.8 / 27.860.3 / 28.0 / 26.476.2 / 68.3 / 78.0
10%56.8 / 41.4 / 20.550.0 / 41.3 / 16.775.9 / 69.2 / 78.1
gamma (PROTON+)SemCovFar
0.565.886.393.2
0.666.485.793.3
0.767.385.393.4
0.867.887.893.5
0.968.088.293.6
kappa (PROTON+)SemCovFar
1.067.787.793.4
1.567.487.593.4
2.067.385.393.4
3.067.285.093.4
4.067.285.093.1
Base scoreBase Sem/Cov/Far+PROTON+ Sem/Cov/Far
Max-Logits54.0 / 27.9 / 83.957.1 / 87.1 / 99.1
Energy51.8 / 27.2 / 83.956.1 / 87.2 / 99.1
GL-MCM75.8 / 68.9 / 78.567.3 / 85.3 / 93.4
MCM75.8 / 68.9 / 78.567.3 / 85.3 / 93.4
Entropy75.8 / 68.9 / 78.563.3 / 86.5 / 96.6
TAG-MCM68.4 / 30.1 / 72.265.2 / 58.7 / 91.5

PROTON component ablations

VariantCovSemFar
MCM baseline68.975.878.5
+ Proto. (alpha=.5)46.665.586.9
+ Adapt. fusion46.965.385.3
+ Conf. gate (Full)46.768.086.8
gammaAUCAPRF95
0.546.929.482.5
0.646.729.483.8
0.746.729.483.6
0.847.830.181.3
0.949.430.779.9
MAUCAPRF95
1045.329.392.0
2544.628.689.9
5045.428.987.2
10046.729.483.6
20047.929.979.5

Stream ID-fraction sweep

rhoSemCovFar(MCM: SemCovFar)
90%73.185.590.476.267.775.5
70%71.247.787.474.869.079.0
30%60.328.026.476.268.378.0
10%50.041.316.775.969.278.1

PROTON on six base scores

Base scoreBase (Cov)+PROTON (Cov)ΔBase (Sem)+PROTON (Sem)Base (Far)+PROTON (Far)
Max-Logits27.923.6-4.454.057.383.987.0
Energy27.222.6-4.551.855.383.987.0
GL-MCM68.945.7-23.275.867.678.585.0
MCM68.946.7-22.175.868.078.586.8
Entropy68.949.7-19.275.870.078.589.8
TAG-MCM30.122.6-7.568.467.972.289.4

Broader baselines, including privileged ones

MethodCov AUCSem AUCFar AUC
TAG-MCM30.168.472.2
MCM (L=5)66.558.354.4
kNN-ID100.076.899.8
Mahalanobis-ID100.082.4100.0
PROTON46.768.086.8
hier_l5+proton24.041.225.3

Prompt sensitivity

Prompt setID accMCM Sem/Cov/FarPROTON Sem/Cov/FarPROTON+ Sem/Cov/Far
flair_default (default)84.075.8 / 68.9 / 78.568.0 / 46.7 / 86.867.3 / 85.3 / 93.4
vanilla86.065.6 / 64.9 / 83.558.5 / 46.7 / 94.458.2 / 87.9 / 97.0
l183.766.5 / 84.2 / 51.865.0 / 66.6 / 78.264.6 / 94.0 / 88.1
l557.358.3 / 66.5 / 54.441.2 / 24.0 / 25.356.2 / 50.8 / 58.8

Robustness extras

sigma2_0 sweep mean AUROC: 0.005: 66.7, 0.01: 66.7, 0.02: 67.2, 0.04: 68.4, 0.08: 68.6

misrouting (cov AUROC): 0.0: 46.7, 0.1: 50.0, 0.2: 51.2, 0.3: 54.5

overhead: PROTON 0.020 ms/sample, PROTON+ 0.282 ms/sample, bank 0.41 MB

wins vs MCM over stream orders: {'semantic': {'proton': '0/5', 'proton_plus': '0/5'}, 'covariate': {'proton': '0/5', 'proton_plus': '5/5'}, 'far': {'proton': '5/5', 'proton_plus': '5/5'}}

triage macro-F1: 0.5968720012468626

PROTON gate x bank-size grid (change in covariate AUROC over MCM)

M=10M=25M=50M=75M=100M=150M=200M=500
g=0.3-22.5-23.3-22.6-22-21.5-20.9-20.5-20.3
g=0.4-22.5-23.3-22.6-22-21.5-20.9-20.5-20.3
g=0.5-22.5-23.3-22.6-22-21.5-20.9-20.5-20.3
g=0.6-23-23.8-23-22.2-21.7-21.1-20.6-20.5
g=0.7-22.9-23.7-23-22.1-21.6-21-20.4-20.3
g=0.8-21.9-22.8-22.1-21.2-20.8-20.2-19.7-19.6
g=0.9-20.2-21.1-20.6-19.9-19.5-19.1-18.8-18.7

Radiology

Zero-shot OOD detection (static scores, PROTON, PROTON+)

MethodSem AUCSem APRSem FPRCov AUCCov APRCov FPRFar AUCFar APRFar FPR
Max-Logits69.456.182.30.410.8100.075.764.690.2
Energy70.257.484.80.410.8100.077.067.490.5
GL-MCM54.335.789.746.718.187.329.214.298.0
MCM54.235.289.746.618.187.327.614.099.7
TAG-MCM57.436.887.848.119.385.541.316.087.5
MCM (L=1)54.235.289.746.618.187.327.614.099.7
MCM (L=5)54.232.077.578.438.246.384.652.339.5
PROTON (ours)55.338.395.261.231.067.693.992.533.5
Δ vs MCM1.13.15.514.612.9-19.866.378.5-66.2

PROTON mean ± std over 5 stream orders: semantic: 55.3±0.9, covariate: 61.2±1.7, far: 93.9±2.3

PROTON+ against PROTON (components, ID fraction, sensitivity, base scores)

VariantSem AUCCov AUCFar AUCSem FPRCov FPRFar FPRt_calib
MCM54.246.627.689.787.399.7nan
PROTON (paper)55.361.293.995.267.633.530
+ z-scored fusion61.678.190.587.656.140.130
+ consistency admission61.978.492.887.360.536.130
+ domain-anchored prototypes (PROTON+)61.493.991.588.126.735.730
domain-anchored only55.286.895.095.238.719.930
consistency only55.463.294.995.268.022.030
PROTON+ max-fusion56.092.290.889.627.434.730
PROTON+ rejection-driven alpha60.895.893.888.317.435.830
S_proto only (PROTON bank)61.892.395.095.241.519.830
S_proto only (PROTON+ bank)62.698.896.283.11.320.530
PROTON, oracle ID-only bank55.486.594.995.150.619.849
PROTON+, oracle ID-only bank64.992.491.387.233.835.849
rhoPROTON+ Sem/Cov/FarPROTON Sem/Cov/FarMCM Sem/Cov/Far
90%64.8 / 96.4 / 94.157.8 / 71.0 / 94.153.7 / 46.8 / 27.5
70%62.0 / 89.5 / 81.054.9 / 54.3 / 81.854.4 / 47.2 / 27.9
50%58.8 / 69.0 / 73.054.5 / 49.9 / 58.854.5 / 47.9 / 28.5
30%55.6 / 12.2 / 14.254.0 / 38.8 / 37.154.5 / 46.7 / 27.7
10%52.5 / 12.7 / 13.153.8 / 23.7 / 21.453.5 / 47.5 / 27.5
gamma (PROTON+)SemCovFar
0.560.195.294.8
0.660.894.893.7
0.761.493.991.5
0.862.492.887.0
0.959.677.571.8
kappa (PROTON+)SemCovFar
1.061.694.191.5
1.561.694.091.5
2.061.493.991.5
3.061.292.891.5
4.061.292.891.5
Base scoreBase Sem/Cov/Far+PROTON+ Sem/Cov/Far
Max-Logits69.4 / 0.4 / 75.769.1 / 92.9 / 98.8
Energy70.2 / 0.4 / 77.069.2 / 92.9 / 98.9
GL-MCM54.3 / 46.7 / 29.261.4 / 93.9 / 91.6
MCM54.2 / 46.6 / 27.661.4 / 93.9 / 91.5
Entropy54.2 / 46.6 / 27.664.3 / 98.1 / 95.5
TAG-MCM57.4 / 48.1 / 41.363.6 / 86.8 / 83.9

PROTON component ablations

VariantCovSemFar
MCM baseline46.654.227.6
+ Proto. (alpha=.5)66.055.296.0
+ Adapt. fusion67.755.397.2
+ Conf. gate (Full)61.255.393.9
gammaAUCAPRF95
0.567.737.858.7
0.665.035.162.3
0.761.231.067.6
0.857.028.179.5
0.952.025.292.8
MAUCAPRF95
1061.032.176.6
2561.031.269.9
5061.131.169.4
10061.231.067.6
20061.231.066.7

Stream ID-fraction sweep

rhoSemCovFar(MCM: SemCovFar)
90%57.871.094.153.746.827.5
70%54.954.381.854.447.227.9
30%54.038.837.154.546.727.7
10%53.823.721.453.547.527.5

PROTON on six base scores

Base scoreBase (Cov)+PROTON (Cov)ΔBase (Sem)+PROTON (Sem)Base (Far)+PROTON (Far)
Max-Logits0.44.5+4.169.467.175.780.6
Energy0.44.5+4.170.267.877.081.8
GL-MCM46.760.6+14.054.355.129.293.5
MCM46.661.2+14.654.255.327.693.9
Entropy46.663.1+16.554.257.227.694.2
TAG-MCM48.160.6+12.457.458.241.394.2

Broader baselines, including privileged ones

MethodCov AUCSem AUCFar AUC
TAG-MCM48.157.441.3
MCM (L=5)78.454.284.6
kNN-ID99.967.8100.0
Mahalanobis-ID100.058.9100.0
PROTON61.255.393.9
hier_l5+proton79.150.388.3

Prompt sensitivity

Prompt setID accMCM Sem/Cov/FarPROTON Sem/Cov/FarPROTON+ Sem/Cov/Far
vanilla73.241.8 / 57.5 / 55.544.9 / 78.9 / 94.753.2 / 95.5 / 96.2
l1 (default)69.254.2 / 46.6 / 27.655.3 / 61.2 / 93.961.4 / 93.9 / 91.5
l555.354.2 / 78.4 / 84.650.3 / 79.1 / 88.349.9 / 88.5 / 96.8

Robustness extras

sigma2_0 sweep mean AUROC: 0.005: 72.7, 0.01: 72.0, 0.02: 70.1, 0.04: 62.0, 0.08: 58.8

misrouting (cov AUROC): 0.0: 61.2, 0.1: 65.7, 0.2: 69.3, 0.3: 72.8

overhead: PROTON 0.019 ms/sample, PROTON+ 0.153 ms/sample, bank 0.41 MB

wins vs MCM over stream orders: {'semantic': {'proton': '5/5', 'proton_plus': '5/5'}, 'covariate': {'proton': '5/5', 'proton_plus': '5/5'}, 'far': {'proton': '5/5', 'proton_plus': '5/5'}}

triage macro-F1: 0.8475022798662478

PROTON gate x bank-size grid (change in covariate AUROC over MCM)

M=10M=25M=50M=75M=100M=150M=200M=500
g=0.320.92121.321.421.622.222.522.6
g=0.420.92121.321.421.622.222.522.6
g=0.520.92121.321.421.622.222.522.6
g=0.618.318.518.919.219.620.120.420.4
g=0.71514.915.315.515.81616.116.1
g=0.810.810.81111.311.511.611.711.7
g=0.95.35.55.85.96666

Pathology

Zero-shot OOD detection (static scores, PROTON, PROTON+)

MethodSem AUCSem APRSem FPRCov AUCCov APRCov FPRFar AUCFar APRFar FPR
Max-Logits2.119.1100.021.312.497.725.015.399.3
Energy1.719.1100.019.412.299.023.315.099.7
GL-MCM56.740.594.067.230.977.767.229.175.2
MCM59.842.490.366.129.885.267.131.577.2
TAG-MCM48.733.197.248.019.492.045.118.795.0
MCM (L=1)59.842.490.366.129.885.267.131.577.2
MCM (L=5)15.320.499.864.630.777.361.126.387.3
PROTON (ours)64.954.992.873.047.387.275.250.480.4
Δ vs MCM5.012.62.57.017.52.18.118.93.2

PROTON mean ± std over 5 stream orders: semantic: 64.9±1.3, covariate: 73.0±3.7, far: 75.2±3.7

PROTON+ against PROTON (components, ID fraction, sensitivity, base scores)

VariantSem AUCCov AUCFar AUCSem FPRCov FPRFar FPRt_calib
MCM59.866.167.190.385.277.2nan
PROTON (paper)64.973.075.292.887.280.482
+ z-scored fusion77.784.782.878.657.055.182
+ consistency admission82.185.688.269.451.640.182
+ domain-anchored prototypes (PROTON+)82.485.692.167.750.735.082
domain-anchored only68.774.081.192.886.973.782
consistency only68.774.281.592.886.572.182
PROTON+ max-fusion84.887.995.251.340.418.482
PROTON+ rejection-driven alpha88.388.096.651.645.014.682
S_proto only (PROTON bank)83.587.587.992.877.457.782
S_proto only (PROTON+ bank)94.993.798.720.226.81.582
PROTON, oracle ID-only bank70.674.484.492.686.679.682
PROTON+, oracle ID-only bank82.185.891.467.150.236.382
rhoPROTON+ Sem/Cov/FarPROTON Sem/Cov/FarMCM Sem/Cov/Far
90%87.0 / 87.5 / 93.873.0 / 74.6 / 81.960.0 / 66.6 / 67.1
70%81.6 / 79.8 / 81.764.1 / 65.4 / 68.459.9 / 65.8 / 66.8
50%73.9 / 71.5 / 73.560.1 / 61.7 / 64.360.2 / 65.4 / 66.5
30%45.2 / 67.5 / 68.855.6 / 67.6 / 63.959.6 / 67.6 / 68.7
10%44.7 / 61.6 / 50.351.8 / 61.2 / 54.463.0 / 61.2 / 61.9
gamma (PROTON+)SemCovFar
0.583.687.593.4
0.684.187.093.4
0.782.485.692.1
0.862.866.274.1
0.959.966.267.2
kappa (PROTON+)SemCovFar
1.082.485.791.8
1.582.585.892.0
2.082.485.692.1
3.081.285.591.2
4.081.085.590.3
Base scoreBase Sem/Cov/Far+PROTON+ Sem/Cov/Far
Max-Logits2.1 / 21.3 / 25.075.5 / 79.4 / 92.5
Energy1.7 / 19.4 / 23.375.5 / 78.8 / 92.2
GL-MCM56.7 / 67.2 / 67.283.5 / 88.0 / 94.8
MCM59.8 / 66.1 / 67.182.4 / 85.6 / 92.1
Entropy59.8 / 66.1 / 67.184.1 / 87.9 / 93.9
TAG-MCM48.7 / 48.0 / 45.176.6 / 74.3 / 81.4

PROTON component ablations

VariantCovSemFar
MCM baseline66.159.867.1
+ Proto. (alpha=.5)75.666.578.8
+ Adapt. fusion73.665.076.7
+ Conf. gate (Full)73.064.975.2
gammaAUCAPRF95
0.573.643.884.3
0.673.345.087.1
0.773.047.387.2
0.866.129.885.2
0.966.129.885.2
MAUCAPRF95
1072.847.386.6
2572.946.887.5
5073.047.187.3
10073.047.387.2
20073.147.587.1

Stream ID-fraction sweep

rhoSemCovFar(MCM: SemCovFar)
90%73.074.681.960.066.667.1
70%64.165.468.459.965.866.8
30%55.667.663.959.667.668.7
10%51.861.254.463.061.261.9

PROTON on six base scores

Base scoreBase (Cov)+PROTON (Cov)ΔBase (Sem)+PROTON (Sem)Base (Far)+PROTON (Far)
Max-Logits21.329.7+8.42.112.725.031.4
Energy19.428.1+8.71.712.423.329.8
GL-MCM67.273.8+6.656.762.967.275.4
MCM66.173.0+7.059.864.967.175.2
Entropy66.176.4+10.359.869.867.178.2
TAG-MCM48.061.0+13.048.758.445.159.5

Broader baselines, including privileged ones

MethodCov AUCSem AUCFar AUC
TAG-MCM48.048.745.1
MCM (L=5)64.615.361.1
kNN-ID99.4100.0100.0
Mahalanobis-ID99.8100.0100.0
PROTON73.064.975.2
hier_l5+proton61.532.665.7

Prompt sensitivity

Prompt setID accMCM Sem/Cov/FarPROTON Sem/Cov/FarPROTON+ Sem/Cov/Far
l1 (default)59.359.8 / 66.1 / 67.164.9 / 73.0 / 75.282.4 / 85.6 / 92.1
id_acc72.748.8 / 48.0 / 45.152.9 / 55.6 / 58.272.5 / 70.1 / 81.1
vanilla64.564.3 / 53.4 / 72.566.7 / 58.9 / 82.275.8 / 72.9 / 92.5
l565.515.3 / 64.6 / 61.132.6 / 61.5 / 65.715.4 / 73.4 / 81.5

Robustness extras

sigma2_0 sweep mean AUROC: 0.005: 75.4, 0.01: 73.9, 0.02: 71.0, 0.04: 68.6, 0.08: 68.1

misrouting (cov AUROC): 0.0: 73.0, 0.1: 74.2, 0.2: 75.7, 0.3: 75.2

overhead: PROTON 0.017 ms/sample, PROTON+ 0.096 ms/sample, bank 0.41 MB

wins vs MCM over stream orders: {'semantic': {'proton': '5/5', 'proton_plus': '5/5'}, 'covariate': {'proton': '5/5', 'proton_plus': '5/5'}, 'far': {'proton': '5/5', 'proton_plus': '5/5'}}

triage macro-F1: 0.3187348586668186

PROTON gate x bank-size grid (change in covariate AUROC over MCM)

M=10M=25M=50M=75M=100M=150M=200M=500
g=0.388.38.48.58.58.58.68.8
g=0.488.38.48.58.58.58.68.8
g=0.588.38.48.58.58.58.68.8
g=0.67.77.87.87.87.9888.2
g=0.7888.18.28.28.38.48.4
g=0.800000000
g=0.900000000