Results report · PBR texture reward model
PBR Reward Models
We want a model that looks at one image of a 3D asset and scores its Base Color, Roughness and Metallic maps from 1 to 5, the way our human raters do. The image shows a render of the asset next to the three maps. We got there in four steps. Gemini wrote an explanation for each of 30k human scores. Qwen3.8-27B learned to reason the same way (CoT SFT). Reinforcement learning then turned it into one 27B teacher per map (GDSO, our GRPO variant that rewards the model's whole score distribution). Finally the three teachers were distilled into one Qwen3.5-9B that answers with three digits and no reasoning (RISD). On 1,520 test assets the 9B model scores as well as the three 27B teachers, while writing three digits instead of three long reasoning runs.
0.799
mean test PLCC of the 9B three-score model at step 1000. The three 27B teachers: 0.799.
BC 0.825R 0.864M 0.708
+0.42
gain in mean PLCC over untrained Qwen3.5-9B with the same prompt (0.376). This step uses no human labels.
+0.05 to +0.20
test SRCC that GDSO adds over SFT when both average 8 runs. With one run each it adds +0.18 to +0.35; for base color and roughness about 70% of that is just steadier scores.
5 tokens
is the whole 9B answer: three digits and two spaces from one prompt, instead of three reasoning runs of about 150–360 tokens each at 27B.
The 27B numbers are optimistic: both 27B models saw 1,436 of the 1,520 test assets during SFT. The 9B students never saw them.
§Findings
- SFT teaches the format, not a stable score. The SFT model answers in the right format 98–99% of the time, but its score changes a lot from one run to the next. Its test SRCC is 0.64 / 0.67 / 0.37 (base color / roughness / metallic) from one run and 0.761 / 0.812 / 0.528 when we average eight.
- GDSO works fast. In the first 50 steps, one-run test SRCC jumps from 0.61 to 0.80 (base color), from 0.67 to 0.85 (roughness) and from 0.40 to 0.68 (metallic). It peaks at 0.820, 0.861 and 0.727 (steps 450, 700 and 850). After that the training reward keeps rising, mostly on images the model has already seen, while test SRCC drifts down a little. So we keep the checkpoint with the best test SRCC.
- Most of what GDSO adds is a stable score. After GDSO, one run gives the same test SRCC as the average of eight (within 0.002). The SFT model needs those eight runs to gain 0.12–0.16. Even against SFT's 8-run average, GDSO still adds +0.06 (base color), +0.05 (roughness) and +0.20 (metallic). All of these gains are far outside sampling noise, and GDSO's expected score is also about a third closer to the human score (§7).
- Don't force the answer format. Forcing the answer through a regex kept metallic SRCC at about 0.71 but wrecked the reasoning: it shrank from about 260 to 70 tokens, and the model drifted away from SFT (its KL to SFT averaged 0.76 over its last 20 steps, eleven times the free run's 0.07 at the same point). The cause is a vLLM detail: forced tokens look certain when sampled but not in training, so training stops learning from them.
- The 9B student copies its teachers closely. Each one-score student ends within 0.004 PLCC of its teacher and ranks the test assets almost the same way (rank correlation 0.98–0.99). After about 400 steps the training loss keeps falling tenfold while test PLCC no longer moves; the extra training mostly memorizes.
- One prompt for all three scores works as well as three. The three-score student matches the three separate students (mean PLCC 0.799 vs 0.798), and it is one model instead of three. It also starts higher before any training (0.376 vs 0.242).
- Metallic is the hardest map at every stage (teacher PLCC 0.709, 81% within ±1). Its labels pile up at 1 and 5, score 1 also covers the raw "0" label, and the biggest errors are assets humans scored 1 that the models rate 3–5.
1Pipeline
Each stage starts from the previous model and changes one thing: what the model imitates, what it is rewarded for, or how much it writes. The prompt changes at each stage too; §3–§5 show every version in a gray box.
| Stage 1 · CoT SFT | Stage 2 · GDSO (per attribute) | Stage 3 · RISD | |
|---|---|---|---|
| Model | Qwen3.8-27B, vision tower frozen | SFT step 700, vision tower frozen | Qwen3.5-9B; vision tower and merger frozen |
| Target | reasons + five-field JSON (CE on target tokens) | free-form reasoning + {"metallic": d} | teacher qT over 1–5 at the digit; no human labels |
| Objective | NLL | DAPO token loss + 0.2·CE(q, s*) + 0.04·KL(π‖SFT-700); reward 1 − |μ − s*|/4 + soft overlong (1,024 / 256) | KL(qT ‖ qS) at each answer digit |
| Rows | 30k reasons (28,358 images), file order | metallic 19,062 · roughness 23,002 · base color 18,678 rows, file order | 28,240 train rows, every attribute's qT |
| Batch × steps | 32 × 800 (0.9 epoch) | 32 images × G = 8 rollouts × 1,500 steps (≈2.1–2.6 passes over the rows) | 256 rows × 1,000 steps (≈9 epochs) |
| LR | 1e-5 cosine, 30 warmup | 1e-6 constant, 3% warmup; clip ε 0.2 / 0.28 | 2e-6 cosine, 30 warmup |
| Decoding | n/a | T = 1, top-p 1, thinking budget 2,048, 2,112 tokens max; truncated rollouts masked | no thinking (<think>\n\n</think> pre-filled) |
| Compute | FSDP2, 32 H100, 1.8 h | ZeRO-3 + colocated vLLM (TP 4), 64 H100, ≈48–76 s / step (metallic's first 150 steps on 32 H100, ≈80 s), ≈20–32 h | FSDP2 HSDP, 32 H100, ≈9–10 s / step, ≈2.6–2.9 h |
| Selection | step 700 (held-out loss 1.180) | best test SRCC: metallic 850, roughness 700, base color 450 | step 1000 reported; the 4 best checkpoints by test PLCC are kept |
Table 1. Training settings, taken from each run's config. Base color GDSO moved to 32 GPUs after step 150, when our GPU quota dropped, and kept 256 rollouts per step.
2Data and evaluation
The labels come from pbr_reasoning_30k-R1: 30,000 human-scored samples in three rounds of 10,000, covering 28,358 different assets (Tier-1 assets appear in every round, each time with a new explanation). Every image is the same 1×4 strip: render, Base Color, Metallic, Roughness. One fixed test set of 1,520 assets is used everywhere (Tier 1: 120, Tier 2: 500, Tiers 3–5: 300 each; 87% realistic style, 11% stylized). GDSO and RISD never train on a test asset. But 1,436 of the 1,520 test assets were in the SFT training data, so every 27B number is optimistic (marked † in Table 6). The 9B students never saw them.
How we score a model. Every model gives a probability for each score 1–5 at its answer; call it q. For a reasoning model we read q from the answer token's top-20 log-probs and average it over K runs. For a student, q is one softmax. From q we take the expected score μ and compare it with the human score in two ways. SRCC (Spearman) asks whether μ puts the assets in the same order as the humans; PLCC (Pearson) asks whether μ rises and falls in step with the human score. A reward needs both. We also report how far q's median is from the human score (MAE) and how often it is within one point (±1). Because a reward uses μ directly, §7 also measures how far μ itself is from the human score. GDSO's own evaluation during training uses one run per image.
3Stage 1: SFT (CoT Reasoning)
Reasons. Gemini-3.8-Flash saw each image together with its human scores and wrote one paragraph explaining them, without repeating the scores (Prompt 0). SFT then teaches Qwen3.8-27B to write such a paragraph as its thinking and to finish with the five scores (Prompt 1).
Prompt 0 · Before Stage 1, Gemini-3.8-Flash writes the reasonspbr_reasoning_30k-R1/code/r1/build_requests.py · user turn
<system: the five-field scorer prompt (Prompt A1) through its Scoring rules, plus one rule:>- The reasons cite concrete visual evidence for each score (which region, which panel, which defect or strength); do not paraphrase the rubric.<image>The senior technical artist's ratings:"visual style": "physical", "base color": 3, "roughness": 3, "metallic": 1, "overall tier": 3These ratings are final. Write a long, exhaustive explanation of the reasoning behind them in one paragraph of plain prose, in the order style, base color, roughness, metallic, overall tier, giving each part several sentences. For each channel, address every point of its rubric's Focus line for each distinct part of the asset, citing the panel and region where it is seen and how it shows in the Reference, and say what keeps it from the next level up. Prefer completeness to brevity and write at least 300 words, without headings, lists or scores.Reasons:
Gemini gets the scores and only has to explain them. Its system prompt is the SFT prompt (Prompt A1) up to the scoring rules, plus the one rule shown. Paragraphs that state a score, mention the ratings or use lists are thrown away and regenerated. The rating line here is an example; each request carries its own asset's scores.
Prompt 1 · CoT SFT, five fieldsGDSO/prompt.txt · 13,190 chars · tail shown
…Scoring rules:- Scores are integers 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- Score each channel against its own rubric before rating the overall tier.- Every field gets a score: a map that is a flat constant fill is scored by its rubric, not skipped. Output format:Reason first, in your thinking, then answer.1. In your thinking, follow the evaluation procedure: classify the visual style, check the image against the Base Color, Roughness, and Metallic rubrics in turn, then weigh the overall tier. Every conclusion cites evidence visible in the image (which panel, which region or part, the specific defect or strength); do not paraphrase the rubric.2. Do not write any score or tier number in your thinking; close each field with its main strengths and problems.3. Keep your thinking under 600 words.4. After your thinking, output exactly one line, a JSON object with exactly these keys in this order, and nothing after it:{"visual style": "<physical|stylized|uncertain|others>", "base color": <1-5>, "roughness": <1-5>, "metallic": <1-5>, "overall tier": <1-5>}<|im_end|><|im_start|>userEvaluate the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think>
Highlighted: the output format. One JSON line with the style, three map scores and an overall tier, after at most 600 words of thinking. The full prompt is in the appendix.
4Stage 2: GDSO teachers
GDSO is GRPO (reinforcement learning that scores a group of 8 answers to the same image against each other) with a reward that looks at the whole score distribution. At the answer we read q, the model's probabilities for the scores 1–5, and reward the rollout with 1 − |μ − s*|/4, where μ is the expected score and s* the human score. Three more terms help: a cross-entropy that pulls q toward the human score (weight 0.2), a KL penalty, which measures how far the model's outputs drift from SFT-700 and keeps them close (weight 0.04), and DAPO's penalty for rollouts longer than 768 tokens. Each map gets its own run and its own prompt: the SFT prompt cut down to one rubric and a one-key answer (Prompt 2).
Prompt 2 · GDSO teacher, metallicGDSO/prompt_metallic.txt · full prompt, scrolls
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR) and material reflectance analysis. Your task is to evaluate the Metallic map of a 3D asset's PBR texture set and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Evaluation procedure:1. Analyze the asset's part structure and material composition from the Reference and Base Color panels, and judge its visual style.2. For each visible region, infer its material class and whether its visible surface is bare metal.3. Compare the Metallic map with that expectation, with the metal response in the Reference, and with the checks, then score it with the Metallic rubric. Style-aware evaluation:- Realism-driven assets: physical correctness is the primary criterion; real-world metal priors apply in full.- Stylized assets (cartoon, anime, hand-painted, cel-shaded): reduced metallic contrast and simplified, stable metal regions are correct when deliberate and consistent; high-frequency metallic noise or realistic metal reflections that fight the look are defects.- Mixed or unclear styles: favor stable, coherent readings; apply physical priors where a region's material is clear.- All styles: technical defects are always penalized. Material reasoning:- Judge the map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- Painted, coated, oxidized, rusty, or dusty surfaces behave as dielectric at the visible surface; only exposed bare metal is metallic.- Hard-surface or mechanical appearance does not imply metal; dark or glossy-looking regions are not metal by default.- Worn edges, handles, and contact areas may expose bare metal under a coating; such details count only if physically plausible and stylistically consistent. Reference material priors:- Bare conductive metals (steel, aluminum, chrome, gold, copper): metallic near 1.- Painted, coated, oxidized, or rusty metal: metallic near 0 at the surface.- Plastic, rubber, wood, fabric, leather, stone, concrete, ceramic, glass: metallic 0.- Stylized assets: reduced metallic contrast is expected. Checks:- The map must be intrinsic: illumination, shadows, highlights, or reflections baked into it are defects.- Cross-channel consistency: a metal region has a metal-like base color, and the Reference shows it as metal.- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Penalize technical defects in every style: blur, seams, misalignment with part boundaries, compression artifacts, placeholder fills. Metallic rubric:Focus: Sparsity and semantic correctness of metal regions, binary values, boundary crispness and alignment, mid-gray areas and noise, physical logic of rust/dirt/paint/wear, metal response in the Reference.- 5: Sparse, semantically meaningful, and essentially binary: bare metal at 1, all dielectrics (including paint, rust, and grime) at 0, no ambiguous mid-gray zones; crisp boundaries aligned with part edges; chipped edges exposing bare metal switch to metal with physical logic. The Reference shows convincing metals with correct, base-color-tinted reflections and sheen.- 4: Metal/non-metal assignment is logically correct and matches the object's material makeup; gloss and reflections look realistic. Fine detail is generalized: micro transitions, chipped-coating edges, and mixed-metal surfaces are simplified. Stable metal read with minor gaps.- 3: Metal regions are assigned correctly, but metals lack reflective punch and the characteristic visual intensity of metal (e.g., values well below 1, dull metal base color); edge and wear breakup is overly simple; slight speckle, noise, or compression artifacts in places. Correct but cheap, just passing.- 2: Metal areas are vague, with soft, poorly defined boundaries, broad intermediate grays, or noisy speckling; metalness partly follows image brightness or highlights rather than material; reflections look dull and weak. Metal is barely recognizable and the segmentation is too abstract.- 1: Violates PBR fundamentals: metal/non-metal assignment is wrong or inverted (e.g., cloth, wood, or painted parts marked metal; bare steel at 0; a whole hard-surface model set metallic because it looks mechanical), or large unjustified gray / light-gray-white areas; values are disconnected from the materials and cannot support even a basic metal definition.- Note: An all-dielectric asset with a clean, uniform 0 map is correct and can score 5; do not penalize it for lacking metal detail. An all-0 map is wrong only when the Base Color or the Reference shows bare metal. Scoring rules:- The score is an integer 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- A map that is a flat constant fill is scored by the rubric, not skipped. Output format:Reason first, in your thinking, then answer.1. In your thinking, follow the evaluation procedure for the Metallic map. Every conclusion cites evidence visible in the image (which panel, which region or part, the specific defect or strength); do not paraphrase the rubric.2. Do not write the score in your thinking; close with the map's main strengths and problems.3. Keep your thinking under 400 words.4. After your thinking, output exactly one line, a JSON object with exactly this key, and nothing after it:{"metallic": <1-5>}<|im_end|><|im_start|>userEvaluate the Metallic map of the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think>
Highlighted: every line that is not in the SFT prompt, from a line-by-line diff. New: the role sentence, the procedure, the style section, the checks, the scoring rules and the output format; four lines of material reasoning are adapted to metal. The Metallic rubric itself is unchanged. The style classification and the other rubrics are removed. The model writes its answer, {"metallic": <1-5>}, as free text after at most 400 words of thinking; nothing forces the format. The roughness and base color prompts follow the same pattern (appendix).
| Attribute | SFT-700 | step 50 | best | step 1500 | MAE μ | μ spread | format |
|---|---|---|---|---|---|---|---|
| Base color | 0.611 | 0.797 | 0.820 @450 | 0.794 | 0.400 | 0.83 | 99.9% |
| Roughness | 0.667 | 0.846 | 0.861 @700 | 0.851 | 0.447 | 0.84 | 99.9% |
| Metallic | 0.401 | 0.678 | 0.727 @850 | 0.681 | 0.745 | 0.79 | 99.9% |
Table 2. Every map peaks between steps 450 and 850. Test SRCC of μ with one run per asset. MAE and μ spread are taken at the best step, where pairwise accuracy is 0.910 / 0.913 / 0.853. μ spread is the standard deviation of μ divided by that of the human scores. Format is the share of well-formed answers. Once training has started, one evaluation moves by about ±0.02 from sampling noise alone; at step 0 by up to ±0.05. The SFT-700 column is GDSO's own step-0 run; Figure 5's one-run values (0.640 / 0.674 / 0.370) are another draw of the same model, and the gap between them is that noise.
Stable scores, not only better ones
A single run's q depends on the reasoning the model happened to sample. Averaging q over K runs separates two things: how stable a model's score is, and how good its average judgement is. Figure 5 does this for SFT-700 and the three teachers, with 8 runs per test asset.
So most of GDSO's one-run gain is stability: +0.18 to +0.35 SRCC with one run, but only +0.05 to +0.06 for base color and roughness when both models average eight runs. Metallic is different: +0.20 remains even at eight runs, because SFT's metallic judgement itself was weak. For a reward model, stability is what matters, because each generated texture is scored once.
What the recipe needed
We stopped four metallic variants early and compare them here with the final recipe (free answer, metallic-only prompt, KL 0.04). Of the changes to GDSO itself, the regex did the most damage, though not to SRCC: it collapsed the reasoning. Skipping SFT is worse still: the model hardly ever produces a valid answer. The five-field prompt trailed by 0.02 and ran slower.
| Metallic run (from SFT-700 unless noted) | step 0 | step 50 | step 100 | Observation |
|---|---|---|---|---|
| Final: free answer, metallic prompt, KL 0.04 | 0.401 | 0.678 | 0.710 | best 0.727 at 850; reasoning ≈300 tokens throughout |
| Answer regex (structured output) | 0.338 | 0.676 | 0.710 | reasoning 260 → 68 tokens, entropy 1.56 → 0.93, KL 0.76 over its last 20 steps (to step 778) |
| Five-field SFT prompt, metallic rewarded | 0.425 | 0.675 | 0.690 | 0.02 behind at step 100; ≈425-token rollouts, ≈100 s a step against 80 s on the same 32 GPUs |
| No KL term | 0.389 | 0.697 | n/a | entropy kept rising (1.56 → 1.67 in 50 steps); KL 0.04 became the default |
| From untrained Qwen3.8-27B (no SFT) | format 6.6% at step 0 | prose after </think> instead of the JSON line; reward ≈0.06 | ||
Table 3. Test metallic SRCC of each variant, one run per asset. Step-0 values differ by sampling noise (up to ±0.05); every run except the last starts from the same SFT-700 weights.
What the teachers say
Five test assets per map, one for each human score, sampled exactly as in training. The bars show the teacher's q, and the check marks the human score.
rl_split means the test set.) Table 5 lists the teacher's 8-run μ and the student's score for every case.Roughness and base color cases
5Stage 3: RISD distillation
Step 1: record the teachers. Each GDSO teacher runs 8 times on every one of the 29,760 rows (train and test) with its training settings. For each run we take the top-20 log-probs at the answer digit, renormalize them over 1–5, and average the 8 runs into qT. Rows with fewer than 4 usable runs are dropped; no test row was. Figure 5 shows that one run would have been enough. Step 2: train the student. Qwen3.5-9B gets the teacher's prompt without the procedure and with a one-digit answer, and its thinking block is left empty so the digit comes first (Prompt 3). It only learns to match qT at that digit (a KL loss); no human label is used. The result: each one-score student ends within 0.004 PLCC of its teacher, and the three-score student does the same for all three maps at once.
Prompt 3 · RISD one-score student, metallicRISD/prompt_metallic_student.txt · tail shown
…Scoring rules:- The score is an integer 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- A map that is a flat constant fill is scored by the rubric, not skipped. Output format:Output only one integer from 1 to 5, and nothing else.<|im_end|><|im_start|>userEvaluate the Metallic map of the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think> </think>
Highlighted: lines that are not in the teacher's prompt (Prompt 2), which are just the output format and the closing </think>. The rest of the system prompt is the teacher's, minus the procedure. The empty <think> block (Qwen's no-thinking template) puts the answer right after the prompt, so the student's prediction at the last prompt token is qS.
Three scores, one prompt. In production we don't want three prompts per asset. The three-score student gets one prompt with all three rubrics (Prompt 4) and answers "b r m". In training the first two digits are filled in with the teachers' most likely scores, and the loss is taken at each of the three digits (below). At test time the student fills them in itself.
Prompt 4 · RISD three-score studentRISD/prompt_all3_student.txt · 10,447 chars · tail shown
…Scoring rules:- Scores are integers 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- Score each map against its own rubric.- Every map gets a score: a map that is a flat constant fill is scored by its rubric, not skipped. Output format:Output only three integers from 1 to 5, separated by single spaces: the Base Color, Roughness, and Metallic scores, in that order. Output nothing else.<|im_end|><|im_start|>userEvaluate the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think> </think>
Highlighted: lines that are not in the SFT prompt (Prompt A1), which are two scoring rules and the output format. Above the part shown, the style, material, checks and rubric sections are the SFT prompt's; the overall-tier rubric, the procedure and the JSON format are removed.
… </think>\n\nprompt ends
4KL · base color
' 'space
1KL · roughness
' 'space
3KL · metallic
<|im_end|>not trained
The loss averages the KL at three positions. The last prompt token predicts the base color digit, and each space predicts the next digit. In training the digits fed back are the teachers' most likely scores (4 and 1 here); at test time they are the student's own.
| Student | step 0 | step 100 | best | step 1000 | teacher | SRCC vs T | |Δμ| |
|---|---|---|---|---|---|---|---|
| Base color | 0.429 | 0.813 | 0.827 @800 | 0.826 | 0.825 | 0.984 | 0.088 |
| Roughness | 0.090 | 0.853 | 0.864 @600 | 0.863 | 0.862 | 0.991 | 0.084 |
| Metallic | 0.205 | 0.692 | 0.707 @550 | 0.706 | 0.709 | 0.978 | 0.156 |
| Three-score, mean | 0.376 | 0.785 | 0.799 @750 | 0.799 | 0.799 | 0.983 | 0.118 |
| Joint, three prompts | 0.243 | 0.777 | n/a | n/a | 0.799 | 0.964 @100 | 0.197 |
| Metallic, 128 × 500 | 0.205 | 0.693 | 0.704 @350 | n/a | 0.709 | 0.975 @350 | 0.174 |
Table 4. Every finished student ends within 0.004 PLCC of its teacher. Test PLCC. The first three rows are the one-score students. "SRCC vs T" is how similarly the student and the teacher rank the test assets, and "|Δμ|" their mean difference in expected score, both at step 1000 unless marked. "Joint, three prompts" was one student trained on the three one-score prompts; we stopped it at step 145 for the single-prompt version. "Metallic, 128 × 500" is the original plan's schedule, stopped at step 391. Every student puts 97–100% of its probability on the five score tokens.
Does the student agree with its teachers on each asset?
| Attribute | Case (Figure 7) | human | teacher μT | student μS | student median |
|---|---|---|---|---|---|
| Base color | 1 · Minimal pendant lamp | 1 | 1.64 | 1.66 | 2 |
| 2 · Hardcover book | 2 | 2.58 | 2.42 | 2 | |
| 3 · Wicker basket with candies | 3 | 3.39 | 3.16 | 3 | |
| 4 · Lever-action shotgun | 4 | 3.95 | 3.84 | 4 | |
| 5 · Industrial metal case | 5 | 3.96 | 3.95 | 4 | |
| Roughness | 1 · Hardcover book | 1 | 1.53 | 1.23 | 1 |
| 2 · Broken concrete slab | 2 | 2.56 | 2.46 | 3 | |
| 3 · Mossy ground tile | 3 | 3.23 | 3.36 | 3 | |
| 4 · Viking round shield | 4 | 4.24 | 4.20 | 4 | |
| 5 · Industrial metal case | 5 | 4.01 | 4.10 | 4 | |
| Metallic | 1 · Wicker basket with candies | 1 | 4.45 | 4.36 | 5 |
| 2 · White credenza | 2 | 2.86 | 2.75 | 2 | |
| 3 · Antique silver vase | 3 | 2.73 | 2.74 | 3 | |
| 4 · Industrial metal case | 4 | 3.88 | 3.95 | 4 | |
| 5 · Chocolate chip cookie | 5 | 4.97 | 4.97 | 5 |
Table 5. The 15 case assets of Figure 7: the human score, the teacher's 8-run μ, and the three-score student's μ and median score (bold when it matches the human score). The student follows the teacher, including on the wicker basket (metallic case 1), which both rate as a clean non-metal despite its human score of 1.
6Comparisons (All models)
Every model in one table: the same 1,520 assets, the same way of reading q, the same metrics. Reasoning models are averaged over 8 runs; we ran the SFT-700 row for this report with the teachers' sampling settings and prompts. Students are read once at step 1000, the end of training, not at their best checkpoint. The GDSO teachers were picked on this test set, since we have no separate validation set. Bold marks the best value in each column.
| Model | Base color | Roughness | Metallic | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PLCC | SRCC | MAE | ±1 | PLCC | SRCC | MAE | ±1 | PLCC | SRCC | MAE | ±1 | |
| 9B zero-shot · one-score prompt | 0.429 | 0.413 | 0.696 | 91.4% | 0.090 | 0.171 | 1.137 | 70.1% | 0.205 | 0.296 | 1.570 | 54.6% |
| 9B zero-shot · three-score prompt | 0.481 | 0.476 | 0.670 | 91.9% | 0.376 | 0.386 | 0.924 | 79.4% | 0.272 | 0.306 | 1.355 | 59.7% |
| 27B CoT-SFT-700 · K = 8 † | 0.766 | 0.761 | 0.460 | 96.2% | 0.817 | 0.812 | 0.480 | 96.1% | 0.548 | 0.528 | 0.970 | 71.6% |
| 27B GDSO teacher · K = 8 † | 0.825 | 0.820 | 0.337 | 98.0% | 0.862 | 0.862 | 0.390 | 97.6% | 0.709 | 0.726 | 0.618 | 81.3% |
| 9B one-score students (×3) | 0.826 | 0.824 | 0.329 | 98.1% | 0.863 | 0.862 | 0.392 | 97.4% | 0.706 | 0.722 | 0.640 | 81.0% |
| 9B three-score student | 0.825 | 0.823 | 0.339 | 98.1% | 0.864 | 0.864 | 0.381 | 97.6% | 0.708 | 0.726 | 0.637 | 80.9% |
| 9B three-score, best ckpt (750) | 0.825 | 0.823 | 0.339 | 98.1% | 0.865 | 0.864 | 0.378 | 97.6% | 0.707 | 0.726 | 0.644 | 80.8% |
Table 6. The 9B students match the 27B teachers' PLCC and SRCC on every map. PLCC and SRCC of μ; MAE and ±1 accuracy of q's median. The zero-shot rows are untrained Qwen3.5-9B with the one-score or three-score prompt. The one-score row combines the three single-map students. † 1,436 of the 1,520 test assets were in the SFT training data, so the 27B rows are optimistic; the 9B rows are not affected. Tier-1 recall and μ spread for every row are in data/unified_test.csv.
7What RL adds over SFT
GDSO is the expensive step: about 20–32 hours on 64 GPUs per map. Here we check whether it pays off, by comparing each GDSO teacher with the SFT-700 checkpoint it started from on the same 1,520 test assets. Both models reason before they score, and the reasoning is sampled, so running the same model twice on the same asset can give two different scores. We already ran each model 8 times on every asset (Figure 5), so this needs no new sampling. We compare the two models in three ways:
- One run: the model reasons once and we score from that single answer. This is how a reward model is used. The numbers are averaged over the 8 runs, each scored on its own.
- Average of 8 runs: we average each asset's 8 score distributions before scoring it. Averaging cancels the run-to-run noise, so this is the best SFT can do if we pay eight times the compute.
- Agreement between runs: we score the same assets with two separate runs and check how well the two sets of scores agree, as their SRCC (1 means both runs rank the assets the same way). It measures how repeatable a model's score is.
To separate real gains from luck, we resample the test assets 2,000 times (a paired bootstrap) and report each gain with its 95% interval.
| Attribute | Assets | SRCC of μ, one run | SRCC of μ, average of 8 runs | ||||
|---|---|---|---|---|---|---|---|
| SFT-700 | GDSO | gain [95%] | SFT-700 | GDSO | gain [95%] | ||
| Base color | 1,520 all | 0.615 | 0.820 | +0.205 [+0.19, +0.22] | 0.761 | 0.820 | +0.059 [+0.04, +0.07] |
| 84 unseen | 0.429 | 0.659 | +0.230 [+0.14, +0.32] | 0.589 | 0.657 | +0.068 [−0.02, +0.17] | |
| Roughness | 1,520 all | 0.675 | 0.862 | +0.187 [+0.17, +0.20] | 0.812 | 0.862 | +0.049 [+0.04, +0.06] |
| 84 unseen | 0.383 | 0.553 | +0.170 [+0.05, +0.28] | 0.532 | 0.552 | +0.021 [−0.10, +0.14] | |
| Metallic | 1,520 all | 0.365 | 0.726 | +0.361 [+0.33, +0.39] | 0.528 | 0.726 | +0.199 [+0.17, +0.23] |
| 84 unseen | 0.327 | 0.791 | +0.464 [+0.37, +0.55] | 0.349 | 0.790 | +0.441 [+0.27, +0.63] | |
Table 7. Every gain on the full test set is far outside sampling noise. SRCC of μ from one run (averaged over the 8 single runs; Figure 5 used only the first run, hence slightly different SFT numbers) and from the average of 8 runs, with GDSO's gain and its 95% bootstrap interval. A grey gain has an interval that includes 0. "Unseen" are the 84 test assets that were not in the SFT training data. Table 8 measures the error of μ itself; PLCC and the median's MAE and ±1 accuracy are in data/gdso_vs_sft.csv.
On the full test set the answer is clear. With a single run, GDSO raises SRCC by +0.21 (base color), +0.19 (roughness) and +0.36 (metallic). None of the 2,000 resamples favours SFT. Against SFT's 8-run average GDSO is still ahead, by +0.06, +0.05 and +0.20, and it gets there with one run, since its one-run and 8-run SRCC are the same.
On the 84 assets the SFT never saw. These are the cleanest test, since neither model trained on them, but they are few and not typical: 66 are Tier 1, and most of their scores are 4 or 5. Even here, one GDSO run beats one SFT run on every map (+0.23, +0.17 and +0.46 SRCC, all with intervals above zero). Against SFT's 8-run average the metallic gain stays large (+0.44), while the base color and roughness gains (+0.07 and +0.02) are too small to tell apart from noise with this few assets.
Where the gain comes from. SFT-700 gives a different score each time it runs. For SFT, two runs on the same asset differ by a full point or more 27% of the time for base color, 30% for roughness and 42% for metallic (Figure 11 b, c shows it asset by asset for metallic). After GDSO this almost never happens, and two runs rank the assets the same way (SRCC between runs 0.999–1.000, against 0.59, 0.64 and 0.36 for SFT). So RL does two things: it makes the score repeatable, which is most of the gain for base color and roughness, and it makes the judgement itself better, which is most of the gain for metallic. A reward model needs both, because each generated texture is scored once.
The expected score itself
A reward uses the expected score μ = Σ s·q(s) directly, so it matters how close μ is to the human score, not only whether it puts the assets in the right order. The SRCC above already uses μ but only checks the order. Table 8 measures the distance: the mean absolute error of μ (MAE of μ), and GDSO's own reward 1 − |μ − s*|/4, the quantity RL was trained to raise.
| Attribute | Assets | MAE of μ, one run | MAE of μ, average of 8 runs | reward 1 − |μ − s*|/4, one run | |||||
|---|---|---|---|---|---|---|---|---|---|
| SFT-700 | GDSO | lower by [95%] | SFT-700 | GDSO | lower by [95%] | SFT-700 | GDSO | ||
| Base color | 1,520 all | 0.63 | 0.40 | 0.23 [0.22, 0.25] | 0.54 | 0.40 | 0.14 [0.12, 0.16] | 0.842 | 0.900 |
| 84 unseen | 0.67 | 0.32 | 0.35 [0.29, 0.41] | 0.61 | 0.32 | 0.29 [0.23, 0.36] | 0.833 | 0.921 | |
| Roughness | 1,520 all | 0.68 | 0.45 | 0.23 [0.22, 0.25] | 0.54 | 0.45 | 0.09 [0.08, 0.11] | 0.830 | 0.888 |
| 84 unseen | 0.91 | 0.66 | 0.25 [0.17, 0.33] | 0.79 | 0.66 | 0.12 [0.04, 0.21] | 0.772 | 0.834 | |
| Metallic | 1,520 all | 1.20 | 0.74 | 0.45 [0.41, 0.50] | 1.12 | 0.74 | 0.38 [0.33, 0.42] | 0.701 | 0.814 |
| 84 unseen | 1.09 | 0.23 | 0.86 [0.71, 1.02] | 1.04 | 0.23 | 0.82 [0.66, 0.98] | 0.727 | 0.943 | |
Table 8. With one run, GDSO's expected score is about a third closer to the human score on every map. MAE of μ is the mean of |μ − s*| over the test assets, from one run (averaged over the 8 single runs) and from the average of 8 runs, with how much lower GDSO's error is and its 95% bootstrap interval. The reward is the mean of 1 − |μ − s*|/4, GDSO's training reward, so it moves with the MAE. RMSE of μ is in data/gdso_vs_sft.csv.
With one run, GDSO cuts the expected score's error by about a third on every map: MAE of μ falls from 0.63 to 0.40 (base color), 0.68 to 0.45 (roughness) and 1.20 to 0.74 (metallic), and the reward rises from 0.84 to 0.90, 0.83 to 0.89 and 0.70 to 0.81. Averaging SFT over 8 runs does not close this gap: its error only falls to 0.54, 0.54 and 1.12, because SFT's expected scores are squeezed toward the middle of the scale, and averaging cannot undo that. An asset humans scored 5 gets an average μ of 4.04, 3.89 and 3.41 from SFT, against 4.50, 4.12 and 4.48 from GDSO (Figure 12). On this measure even the 84 unseen assets show a clear gain on every map, against one SFT run and against SFT's 8-run average. Metallic assets scored 1 remain the weak spot for both models (mean μ 2.14).
APrompts and reproduction
Full prompt · CoT SFT (five fields)
Prompt A1 · GDSO/prompt.txtCoT SFT and the five-field GDSO variant
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR), material reflectance analysis, and intrinsic decomposition. Your task is to evaluate the PBR texture set (Base Color, Roughness, Metallic) of a 3D asset and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Evaluation procedure:1. Analyze the asset's geometry, part structure, surface segmentation, and material composition from the Reference and Base Color panels.2. Classify the visual style. The label sets the reference standard for every score.3. For each visible region, infer its material class and surface finish, and derive the Metallic and Roughness behavior you would expect.4. Compare each map with that expectation, with the other panels, and with the common checks, then score Base Color, Roughness, and Metallic with their rubrics.5. Rate the overall quality tier from the channel results, cross-channel consistency, and the look of the Reference. Visual style classification:Assign exactly one label.- physical: realism-driven, built from real-world reference; follows PBR conventions.- stylized: art-directed (cartoon, anime, hand-painted, cel-shaded); may depart from PBR rules on purpose to achieve its look.- uncertain: mixes realistic and stylized cues, or shows too little to tell them apart.- others: neither realistic nor a recognizable art style. Style-aware evaluation (Harmony First):- physical: physical correctness is the primary criterion; real-world material priors apply in full.- stylized: harmony with the art direction outranks physical accuracy. Simplified, stable reflectance (painted light and shade in the albedo, exaggerated hues, low metallic contrast, high and uniform roughness) is correct when deliberate and consistent, and intentionally flat regions are not defects; realistic reflectance that fights the look (sharp specular, metallic noise, busy roughness variation) is. Read the rubrics' realism and fine-detail criteria as clarity, intentionality, and coherence with the art direction.- uncertain: favor stable, coherent readings; apply physical priors where a region's material is clear, and accept simplifications a stylized reading would allow.- others: judge technical quality and internal consistency only.- All styles: technical defects are always penalized, and local variation must not break the global style. Material reasoning:- Judge each map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- Painted, coated, oxidized, rusty, or dusty surfaces behave as dielectric at the visible surface; only exposed bare metal is metallic.- Hard-surface or mechanical appearance does not imply metal; dark or glossy-looking regions are not metal by default.- Roughness encodes micro-surface finish, not brightness or shading; a roughness map that copies base color luminance or lighting is a defect.- Use part semantics to judge wear and grime: edges, handles, and contact areas wear; recesses and crevices collect dirt. Such details count only if physically plausible and stylistically consistent. Reference material priors:- Bare conductive metals (steel, aluminum, chrome, gold, copper): metallic near 1; roughness follows finish (polished low, brushed medium, cast or sandblasted high).- Painted, coated, oxidized, or rusty metal: metallic near 0 at the surface.- Plastic, rubber, wood, fabric, leather, stone, concrete, ceramic, glass: metallic 0.- Roughness is low for glass, polished metal, glossy paint, and glazed ceramic; medium for plastic, leather, and satin paint; high for rubber, fabric, concrete, raw wood, and stone.- Roughness increases with abrasion, oxidation, dust, and stylization level; polished, wet, oily, or frequently handled areas read glossier.- Stylized assets: reduced metallic contrast and elevated roughness are expected. Common checks for all maps:- Maps must be intrinsic: illumination, shadows, AO, highlights, or reflections baked into a map are defects (deliberate painted shading in stylized albedo excepted).- Cross-channel consistency: the same semantic region keeps one material across Base Color, Metallic, and Roughness (e.g., a metal region has a metal-like base color and a roughness that fits its finish).- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Judge effective detail, not nominal resolution: an upscaled or blurry map scores by the detail actually visible.- Penalize technical defects in every style: blur, seams, stretching, visible tiling, compression artifacts, noise, placeholder fills. Base Color rubric:Focus: Color layering across frequencies, material identity, value and exposure range, baked lighting, color casts, texel density and sharpness, tiling, seams.- 5: Hero-quality albedo: rich multi-scale color variation, with crisp high-frequency detail over smooth large-scale gradients; every material (leather, metal, stone, paint) reads as real. Clean and intrinsic: no baked shading, AO, or highlights, no crushed blacks or blown whites, exposure consistent across materials; no unmotivated color casts (green shift, magenta); uniform, very fine texel density with no visible repetition or sharpness loss. ArtStation / AAA hero level.- 4: Professional game albedo: accurate color and clearly identifiable materials, rich detail just short of hero precision; minor local softness, slight hue/value deviation, or faint baked shading; transitions are natural but some areas read flat. Roughly 90% there, lacking the 'wow' factor.- 3: Usable but unpolished: most regions convey their material and overall color is correct, but layered detail and micro-variation are missing; slightly blurry or inconsistent sharpness; noticeable tiling, mild stretching, or visible baked lighting in places; stiff transitions with abrupt color jumps.- 2: Novice-level: thin, flat color with little layering, large areas are near-uniform fills; lacks high-frequency detail; obvious repetition, blur, seams, misalignment, dirty color, or heavy baked lighting; real material properties are hard to identify. Has color but no material sense.- 1: Placeholder or broken: single flat color, plain gray, or at most coarse color blocks separating parts; severe artifacts; hues far from the intended material (e.g., saturated purple, neon green) not justified by the art style; conveys essentially no material. Roughness rubric:Focus: Value range usage, separation between materials, plausibility of the distribution (wear, grime, oil, oxidation), micro-detail vs. noise, independence from base color and lighting, specular response in the Reference.- 5: Purposeful use of the full 0-1 range with clear, physically consistent separation between materials. Variation follows plausible surface history (wear, grime, oil, oxidation; e.g., scuffed or dirty areas rougher, clean or polished areas glossier) with natural edge transitions; layered but low-noise. The Reference makes material type and age/condition clearly legible. AAA level.- 4: Per-surface values match common expectations (e.g., leather vs. metal buckle) and read clearly; good grayscale layering conveys tactile feel. Micro-detail is somewhat plain, lacking fine high-frequency breakup; minor acceptable noise or banding/quantization.- 3: Basically works and base materials are distinguishable, but values are too uniform within regions or vary in an overly regular, procedural pattern without natural randomness; small details are missing or inaccurate. Surfaces look stiff and not quite real.- 2: Large areas are uniformly glossy or uniformly hazy, values are misused, or the map mostly copies base color luminance or baked lighting; surfaces read as cheap plastic or rubber and material character is barely visible.- 1: Flat black or flat white, or filled with values unrelated to the materials; wrong material class (e.g., fabric mirror-glossy, chrome chalky); no meaningful zoning, the map expresses no material. Metallic rubric:Focus: Sparsity and semantic correctness of metal regions, binary values, boundary crispness and alignment, mid-gray areas and noise, physical logic of rust/dirt/paint/wear, metal response in the Reference.- 5: Sparse, semantically meaningful, and essentially binary: bare metal at 1, all dielectrics (including paint, rust, and grime) at 0, no ambiguous mid-gray zones; crisp boundaries aligned with part edges; chipped edges exposing bare metal switch to metal with physical logic. The Reference shows convincing metals with correct, base-color-tinted reflections and sheen.- 4: Metal/non-metal assignment is logically correct and matches the object's material makeup; gloss and reflections look realistic. Fine detail is generalized: micro transitions, chipped-coating edges, and mixed-metal surfaces are simplified. Stable metal read with minor gaps.- 3: Metal regions are assigned correctly, but metals lack reflective punch and the characteristic visual intensity of metal (e.g., values well below 1, dull metal base color); edge and wear breakup is overly simple; slight speckle, noise, or compression artifacts in places. Correct but cheap, just passing.- 2: Metal areas are vague, with soft, poorly defined boundaries, broad intermediate grays, or noisy speckling; metalness partly follows image brightness or highlights rather than material; reflections look dull and weak. Metal is barely recognizable and the segmentation is too abstract.- 1: Violates PBR fundamentals: metal/non-metal assignment is wrong or inverted (e.g., cloth, wood, or painted parts marked metal; bare steel at 0; a whole hard-surface model set metallic because it looks mechanical), or large unjustified gray / light-gray-white areas; values are disconnected from the materials and cannot support even a basic metal definition.- Note: An all-dielectric asset with a clean, uniform 0 map is correct and can score 5; do not penalize it for lacking metal detail. An all-0 map is wrong only when the Base Color or the Reference shows bare metal. Overall quality tier rubric:Focus: Holistic production readiness, cross-channel consistency, style harmony, artifact level, suitable viewing distance.- 5: Cinematic: complete, standards-compliant PBR set with extremely rich, physically correct detail in every channel; channels consistent with each other; no visible compression artifacts. Holds up in close-ups and high-end rendering. Typically >= 4K.- 4: AAA current-gen (PS5): standards-compliant with strong detail; clear roughness and metal separation; well-controlled base color exposure and color; light artistic liberties keep physical plausibility. Typically >= 2K (>= 1K acceptable).- 3: Game-ready (indie / PS4): correct overall and visually satisfying; most regions show legible detail, with only minor fixable defects. Suited to mid-distance views. Realistic >= 1K; stylized 512-1K and crisp.- 2: Mobile: material relationships and structure are correct, without obvious blur or distortion, but detail is sparse. Suited to far views or low-budget projects. Typically 512-1K.- 1: Barely usable: obvious problems (blur, misalignment or stretching, compression artifacts, exposure issues, shaky metal/roughness logic) that need manual repair; broken or placeholder sets also score 1.- Note: Holistic, not an average of the channel scores: it seldom exceeds their mean by more than one level, and one severe defect can pull it lower. Scoring rules:- Scores are integers 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- Score each channel against its own rubric before rating the overall tier.- Every field gets a score: a map that is a flat constant fill is scored by its rubric, not skipped. Output format:Reason first, in your thinking, then answer.1. In your thinking, follow the evaluation procedure: classify the visual style, check the image against the Base Color, Roughness, and Metallic rubrics in turn, then weigh the overall tier. Every conclusion cites evidence visible in the image (which panel, which region or part, the specific defect or strength); do not paraphrase the rubric.2. Do not write any score or tier number in your thinking; close each field with its main strengths and problems.3. Keep your thinking under 600 words.4. After your thinking, output exactly one line, a JSON object with exactly these keys in this order, and nothing after it:{"visual style": "<physical|stylized|uncertain|others>", "base color": <1-5>, "roughness": <1-5>, "metallic": <1-5>, "overall tier": <1-5>}<|im_end|><|im_start|>userEvaluate the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think>
Full prompt · GDSO teacher, roughness
Prompt A2 · GDSO/prompt_roughness.txtroughness teacher
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR) and material reflectance analysis. Your task is to evaluate the Roughness map of a 3D asset's PBR texture set and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Evaluation procedure:1. Analyze the asset's part structure and material composition from the Reference and Base Color panels, and judge its visual style.2. For each visible region, infer its material class, surface finish, and surface history, and derive the roughness you would expect.3. Compare the Roughness map with that expectation, with the specular response in the Reference, and with the checks, then score it with the Roughness rubric. Style-aware evaluation:- Realism-driven assets: physical correctness is the primary criterion; real-world roughness priors apply in full.- Stylized assets (cartoon, anime, hand-painted, cel-shaded): high, uniform, simplified roughness is correct when deliberate and consistent, and intentionally flat regions are not defects; sharp specular or busy roughness variation that fights the look is a defect.- Mixed or unclear styles: favor stable, coherent readings; apply physical priors where a region's material is clear.- All styles: technical defects are always penalized. Material reasoning:- Judge the map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- Roughness encodes micro-surface finish, not brightness or shading; a roughness map that copies base color luminance or lighting is a defect.- Use part semantics to judge wear and grime: edges, handles, and contact areas wear; recesses and crevices collect dirt. Such details count only if physically plausible and stylistically consistent. Reference material priors:- Roughness is low for glass, polished metal, glossy paint, and glazed ceramic; medium for plastic, leather, and satin paint; high for rubber, fabric, concrete, raw wood, and stone.- Bare metals: roughness follows finish (polished low, brushed medium, cast or sandblasted high).- Roughness increases with abrasion, oxidation, dust, and stylization level; polished, wet, oily, or frequently handled areas read glossier.- Stylized assets: elevated roughness is expected. Checks:- The map must be intrinsic: illumination, shadows, AO, highlights, or reflections baked into it are defects.- Cross-channel consistency: a region's roughness fits the material its Base Color and Metallic show (e.g., a metal region has a roughness that fits its finish), and the Reference's highlights agree with it.- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Judge effective detail, not nominal resolution: an upscaled or blurry map scores by the detail actually visible.- Penalize technical defects in every style: blur, seams, stretching, visible tiling, compression artifacts, placeholder fills. Roughness rubric:Focus: Value range usage, separation between materials, plausibility of the distribution (wear, grime, oil, oxidation), micro-detail vs. noise, independence from base color and lighting, specular response in the Reference.- 5: Purposeful use of the full 0-1 range with clear, physically consistent separation between materials. Variation follows plausible surface history (wear, grime, oil, oxidation; e.g., scuffed or dirty areas rougher, clean or polished areas glossier) with natural edge transitions; layered but low-noise. The Reference makes material type and age/condition clearly legible. AAA level.- 4: Per-surface values match common expectations (e.g., leather vs. metal buckle) and read clearly; good grayscale layering conveys tactile feel. Micro-detail is somewhat plain, lacking fine high-frequency breakup; minor acceptable noise or banding/quantization.- 3: Basically works and base materials are distinguishable, but values are too uniform within regions or vary in an overly regular, procedural pattern without natural randomness; small details are missing or inaccurate. Surfaces look stiff and not quite real.- 2: Large areas are uniformly glossy or uniformly hazy, values are misused, or the map mostly copies base color luminance or baked lighting; surfaces read as cheap plastic or rubber and material character is barely visible.- 1: Flat black or flat white, or filled with values unrelated to the materials; wrong material class (e.g., fabric mirror-glossy, chrome chalky); no meaningful zoning, the map expresses no material. Scoring rules:- The score is an integer 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- A map that is a flat constant fill is scored by the rubric, not skipped. Output format:Reason first, in your thinking, then answer.1. In your thinking, follow the evaluation procedure for the Roughness map. Every conclusion cites evidence visible in the image (which panel, which region or part, the specific defect or strength); do not paraphrase the rubric.2. Do not write the score in your thinking; close with the map's main strengths and problems.3. Keep your thinking under 400 words.4. After your thinking, output exactly one line, a JSON object with exactly this key, and nothing after it:{"roughness": <1-5>}<|im_end|><|im_start|>userEvaluate the Roughness map of the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think>
Full prompt · GDSO teacher, base color
Prompt A3 · GDSO/prompt_basecolor.txtbase color teacher
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR), material reflectance analysis, and intrinsic decomposition. Your task is to evaluate the Base Color map of a 3D asset's PBR texture set and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Evaluation procedure:1. Analyze the asset's part structure and material composition from the Reference and Base Color panels, and judge its visual style.2. For each visible region, infer its material class and the albedo you would expect: its color, its value range, and the variation its material and surface history carry.3. Compare the Base Color map with that expectation, with the Reference, and with the checks, then score it with the Base Color rubric. Style-aware evaluation:- Realism-driven assets: physical correctness is the primary criterion; the albedo must be intrinsic, with real-world color and value ranges.- Stylized assets (cartoon, anime, hand-painted, cel-shaded): painted light and shade in the albedo and exaggerated hues are correct when deliberate and consistent, and intentionally flat regions are not defects; read the rubric's realism and fine-detail criteria as clarity, intentionality, and coherence with the art direction.- Mixed or unclear styles: favor stable, coherent readings; apply physical priors where a region's material is clear.- All styles: technical defects are always penalized. Material reasoning:- Judge the map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- The base color is the surface's intrinsic color: illumination, shadows, AO, and highlights belong to the render, not to the albedo.- Use part semantics to judge wear and grime: edges, handles, and contact areas wear; recesses and crevices collect dirt. Such details count only if physically plausible and stylistically consistent. Checks:- The map must be intrinsic: illumination, shadows, AO, highlights, or reflections baked into it are defects (deliberate painted shading in stylized albedo excepted).- Cross-channel consistency: the same semantic region keeps one material across Base Color, Metallic, and Roughness (e.g., a metal region has a metal-like base color).- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Judge effective detail, not nominal resolution: an upscaled or blurry map scores by the detail actually visible.- Penalize technical defects in every style: blur, seams, stretching, visible tiling, compression artifacts, noise, placeholder fills. Base Color rubric:Focus: Color layering across frequencies, material identity, value and exposure range, baked lighting, color casts, texel density and sharpness, tiling, seams.- 5: Hero-quality albedo: rich multi-scale color variation, with crisp high-frequency detail over smooth large-scale gradients; every material (leather, metal, stone, paint) reads as real. Clean and intrinsic: no baked shading, AO, or highlights, no crushed blacks or blown whites, exposure consistent across materials; no unmotivated color casts (green shift, magenta); uniform, very fine texel density with no visible repetition or sharpness loss. ArtStation / AAA hero level.- 4: Professional game albedo: accurate color and clearly identifiable materials, rich detail just short of hero precision; minor local softness, slight hue/value deviation, or faint baked shading; transitions are natural but some areas read flat. Roughly 90% there, lacking the 'wow' factor.- 3: Usable but unpolished: most regions convey their material and overall color is correct, but layered detail and micro-variation are missing; slightly blurry or inconsistent sharpness; noticeable tiling, mild stretching, or visible baked lighting in places; stiff transitions with abrupt color jumps.- 2: Novice-level: thin, flat color with little layering, large areas are near-uniform fills; lacks high-frequency detail; obvious repetition, blur, seams, misalignment, dirty color, or heavy baked lighting; real material properties are hard to identify. Has color but no material sense.- 1: Placeholder or broken: single flat color, plain gray, or at most coarse color blocks separating parts; severe artifacts; hues far from the intended material (e.g., saturated purple, neon green) not justified by the art style; conveys essentially no material. Scoring rules:- The score is an integer 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- A map that is a flat constant fill is scored by the rubric, not skipped. Output format:Reason first, in your thinking, then answer.1. In your thinking, follow the evaluation procedure for the Base Color map. Every conclusion cites evidence visible in the image (which panel, which region or part, the specific defect or strength); do not paraphrase the rubric.2. Do not write the score in your thinking; close with the map's main strengths and problems.3. Keep your thinking under 400 words.4. After your thinking, output exactly one line, a JSON object with exactly this key, and nothing after it:{"base color": <1-5>}<|im_end|><|im_start|>userEvaluate the Base Color map of the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think>
Full prompt · RISD one-score student, metallic
Prompt A4 · RISD/prompt_metallic_student.txtthe base color and roughness students follow the same edit
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR) and material reflectance analysis. Your task is to evaluate the Metallic map of a 3D asset's PBR texture set and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Style-aware evaluation:- Realism-driven assets: physical correctness is the primary criterion; real-world metal priors apply in full.- Stylized assets (cartoon, anime, hand-painted, cel-shaded): reduced metallic contrast and simplified, stable metal regions are correct when deliberate and consistent; high-frequency metallic noise or realistic metal reflections that fight the look are defects.- Mixed or unclear styles: favor stable, coherent readings; apply physical priors where a region's material is clear.- All styles: technical defects are always penalized. Material reasoning:- Judge the map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- Painted, coated, oxidized, rusty, or dusty surfaces behave as dielectric at the visible surface; only exposed bare metal is metallic.- Hard-surface or mechanical appearance does not imply metal; dark or glossy-looking regions are not metal by default.- Worn edges, handles, and contact areas may expose bare metal under a coating; such details count only if physically plausible and stylistically consistent. Reference material priors:- Bare conductive metals (steel, aluminum, chrome, gold, copper): metallic near 1.- Painted, coated, oxidized, or rusty metal: metallic near 0 at the surface.- Plastic, rubber, wood, fabric, leather, stone, concrete, ceramic, glass: metallic 0.- Stylized assets: reduced metallic contrast is expected. Checks:- The map must be intrinsic: illumination, shadows, highlights, or reflections baked into it are defects.- Cross-channel consistency: a metal region has a metal-like base color, and the Reference shows it as metal.- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Penalize technical defects in every style: blur, seams, misalignment with part boundaries, compression artifacts, placeholder fills. Metallic rubric:Focus: Sparsity and semantic correctness of metal regions, binary values, boundary crispness and alignment, mid-gray areas and noise, physical logic of rust/dirt/paint/wear, metal response in the Reference.- 5: Sparse, semantically meaningful, and essentially binary: bare metal at 1, all dielectrics (including paint, rust, and grime) at 0, no ambiguous mid-gray zones; crisp boundaries aligned with part edges; chipped edges exposing bare metal switch to metal with physical logic. The Reference shows convincing metals with correct, base-color-tinted reflections and sheen.- 4: Metal/non-metal assignment is logically correct and matches the object's material makeup; gloss and reflections look realistic. Fine detail is generalized: micro transitions, chipped-coating edges, and mixed-metal surfaces are simplified. Stable metal read with minor gaps.- 3: Metal regions are assigned correctly, but metals lack reflective punch and the characteristic visual intensity of metal (e.g., values well below 1, dull metal base color); edge and wear breakup is overly simple; slight speckle, noise, or compression artifacts in places. Correct but cheap, just passing.- 2: Metal areas are vague, with soft, poorly defined boundaries, broad intermediate grays, or noisy speckling; metalness partly follows image brightness or highlights rather than material; reflections look dull and weak. Metal is barely recognizable and the segmentation is too abstract.- 1: Violates PBR fundamentals: metal/non-metal assignment is wrong or inverted (e.g., cloth, wood, or painted parts marked metal; bare steel at 0; a whole hard-surface model set metallic because it looks mechanical), or large unjustified gray / light-gray-white areas; values are disconnected from the materials and cannot support even a basic metal definition.- Note: An all-dielectric asset with a clean, uniform 0 map is correct and can score 5; do not penalize it for lacking metal detail. An all-0 map is wrong only when the Base Color or the Reference shows bare metal. Scoring rules:- The score is an integer 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- A map that is a flat constant fill is scored by the rubric, not skipped. Output format:Output only one integer from 1 to 5, and nothing else.<|im_end|><|im_start|>userEvaluate the Metallic map of the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think> </think>
Full prompt · RISD three-score student
Prompt A5 · RISD/prompt_all3_student.txtthe final 9B model
<|im_start|>systemYou are a senior technical artist and PBR material reviewer with expertise in physically based rendering (PBR), material reflectance analysis, and intrinsic decomposition. Your task is to evaluate the PBR texture set (Base Color, Roughness, Metallic) of a 3D asset and rate its quality for physically plausible, style-consistent rendering. Input:You receive one image: a 1x4 grid of four panels of the same asset, left to right: Reference (a render of the textured asset), Base Color, Metallic, Roughness. Visual style classification:Assign exactly one label.- physical: realism-driven, built from real-world reference; follows PBR conventions.- stylized: art-directed (cartoon, anime, hand-painted, cel-shaded); may depart from PBR rules on purpose to achieve its look.- uncertain: mixes realistic and stylized cues, or shows too little to tell them apart.- others: neither realistic nor a recognizable art style. Style-aware evaluation (Harmony First):- physical: physical correctness is the primary criterion; real-world material priors apply in full.- stylized: harmony with the art direction outranks physical accuracy. Simplified, stable reflectance (painted light and shade in the albedo, exaggerated hues, low metallic contrast, high and uniform roughness) is correct when deliberate and consistent, and intentionally flat regions are not defects; realistic reflectance that fights the look (sharp specular, metallic noise, busy roughness variation) is. Read the rubrics' realism and fine-detail criteria as clarity, intentionality, and coherence with the art direction.- uncertain: favor stable, coherent readings; apply physical priors where a region's material is clear, and accept simplifications a stylized reading would allow.- others: judge technical quality and internal consistency only.- All styles: technical defects are always penalized, and local variation must not break the global style. Material reasoning:- Judge each map by the intrinsic material a region represents, not by how the region looks in the image.- Infer the most likely material class per region: bare metal, painted/coated metal, oxidized metal, plastic, rubber, ceramic, glass, stone, concrete, wood, fabric, leather, or composite.- Painted, coated, oxidized, rusty, or dusty surfaces behave as dielectric at the visible surface; only exposed bare metal is metallic.- Hard-surface or mechanical appearance does not imply metal; dark or glossy-looking regions are not metal by default.- Roughness encodes micro-surface finish, not brightness or shading; a roughness map that copies base color luminance or lighting is a defect.- Use part semantics to judge wear and grime: edges, handles, and contact areas wear; recesses and crevices collect dirt. Such details count only if physically plausible and stylistically consistent. Reference material priors:- Bare conductive metals (steel, aluminum, chrome, gold, copper): metallic near 1; roughness follows finish (polished low, brushed medium, cast or sandblasted high).- Painted, coated, oxidized, or rusty metal: metallic near 0 at the surface.- Plastic, rubber, wood, fabric, leather, stone, concrete, ceramic, glass: metallic 0.- Roughness is low for glass, polished metal, glossy paint, and glazed ceramic; medium for plastic, leather, and satin paint; high for rubber, fabric, concrete, raw wood, and stone.- Roughness increases with abrasion, oxidation, dust, and stylization level; polished, wet, oily, or frequently handled areas read glossier.- Stylized assets: reduced metallic contrast and elevated roughness are expected. Common checks for all maps:- Maps must be intrinsic: illumination, shadows, AO, highlights, or reflections baked into a map are defects (deliberate painted shading in stylized albedo excepted).- Cross-channel consistency: the same semantic region keeps one material across Base Color, Metallic, and Roughness (e.g., a metal region has a metal-like base color and a roughness that fits its finish).- Noise is not detail: variation must follow the material and its surface history, not random speckle.- Judge effective detail, not nominal resolution: an upscaled or blurry map scores by the detail actually visible.- Penalize technical defects in every style: blur, seams, stretching, visible tiling, compression artifacts, noise, placeholder fills. Base Color rubric:Focus: Color layering across frequencies, material identity, value and exposure range, baked lighting, color casts, texel density and sharpness, tiling, seams.- 5: Hero-quality albedo: rich multi-scale color variation, with crisp high-frequency detail over smooth large-scale gradients; every material (leather, metal, stone, paint) reads as real. Clean and intrinsic: no baked shading, AO, or highlights, no crushed blacks or blown whites, exposure consistent across materials; no unmotivated color casts (green shift, magenta); uniform, very fine texel density with no visible repetition or sharpness loss. ArtStation / AAA hero level.- 4: Professional game albedo: accurate color and clearly identifiable materials, rich detail just short of hero precision; minor local softness, slight hue/value deviation, or faint baked shading; transitions are natural but some areas read flat. Roughly 90% there, lacking the 'wow' factor.- 3: Usable but unpolished: most regions convey their material and overall color is correct, but layered detail and micro-variation are missing; slightly blurry or inconsistent sharpness; noticeable tiling, mild stretching, or visible baked lighting in places; stiff transitions with abrupt color jumps.- 2: Novice-level: thin, flat color with little layering, large areas are near-uniform fills; lacks high-frequency detail; obvious repetition, blur, seams, misalignment, dirty color, or heavy baked lighting; real material properties are hard to identify. Has color but no material sense.- 1: Placeholder or broken: single flat color, plain gray, or at most coarse color blocks separating parts; severe artifacts; hues far from the intended material (e.g., saturated purple, neon green) not justified by the art style; conveys essentially no material. Roughness rubric:Focus: Value range usage, separation between materials, plausibility of the distribution (wear, grime, oil, oxidation), micro-detail vs. noise, independence from base color and lighting, specular response in the Reference.- 5: Purposeful use of the full 0-1 range with clear, physically consistent separation between materials. Variation follows plausible surface history (wear, grime, oil, oxidation; e.g., scuffed or dirty areas rougher, clean or polished areas glossier) with natural edge transitions; layered but low-noise. The Reference makes material type and age/condition clearly legible. AAA level.- 4: Per-surface values match common expectations (e.g., leather vs. metal buckle) and read clearly; good grayscale layering conveys tactile feel. Micro-detail is somewhat plain, lacking fine high-frequency breakup; minor acceptable noise or banding/quantization.- 3: Basically works and base materials are distinguishable, but values are too uniform within regions or vary in an overly regular, procedural pattern without natural randomness; small details are missing or inaccurate. Surfaces look stiff and not quite real.- 2: Large areas are uniformly glossy or uniformly hazy, values are misused, or the map mostly copies base color luminance or baked lighting; surfaces read as cheap plastic or rubber and material character is barely visible.- 1: Flat black or flat white, or filled with values unrelated to the materials; wrong material class (e.g., fabric mirror-glossy, chrome chalky); no meaningful zoning, the map expresses no material. Metallic rubric:Focus: Sparsity and semantic correctness of metal regions, binary values, boundary crispness and alignment, mid-gray areas and noise, physical logic of rust/dirt/paint/wear, metal response in the Reference.- 5: Sparse, semantically meaningful, and essentially binary: bare metal at 1, all dielectrics (including paint, rust, and grime) at 0, no ambiguous mid-gray zones; crisp boundaries aligned with part edges; chipped edges exposing bare metal switch to metal with physical logic. The Reference shows convincing metals with correct, base-color-tinted reflections and sheen.- 4: Metal/non-metal assignment is logically correct and matches the object's material makeup; gloss and reflections look realistic. Fine detail is generalized: micro transitions, chipped-coating edges, and mixed-metal surfaces are simplified. Stable metal read with minor gaps.- 3: Metal regions are assigned correctly, but metals lack reflective punch and the characteristic visual intensity of metal (e.g., values well below 1, dull metal base color); edge and wear breakup is overly simple; slight speckle, noise, or compression artifacts in places. Correct but cheap, just passing.- 2: Metal areas are vague, with soft, poorly defined boundaries, broad intermediate grays, or noisy speckling; metalness partly follows image brightness or highlights rather than material; reflections look dull and weak. Metal is barely recognizable and the segmentation is too abstract.- 1: Violates PBR fundamentals: metal/non-metal assignment is wrong or inverted (e.g., cloth, wood, or painted parts marked metal; bare steel at 0; a whole hard-surface model set metallic because it looks mechanical), or large unjustified gray / light-gray-white areas; values are disconnected from the materials and cannot support even a basic metal definition.- Note: An all-dielectric asset with a clean, uniform 0 map is correct and can score 5; do not penalize it for lacking metal detail. An all-0 map is wrong only when the Base Color or the Reference shows bare metal. Scoring rules:- Scores are integers 1-5. Assign the highest level whose description is substantially met; when torn between two levels, choose the lower.- Score each map against its own rubric.- Every map gets a score: a map that is a flat constant fill is scored by its rubric, not skipped. Output format:Output only three integers from 1 to 5, separated by single spaces: the Base Color, Roughness, and Metallic scores, in that order. Output nothing else.<|im_end|><|im_start|>userEvaluate the PBR texture set shown in this image.<|vision_start|><|image_pad|><|vision_end|><|im_end|><|im_start|>assistant<think> </think>
This report's folder
Everything is in /data/home/jingzhibao/report/. code/extract.py pulls every run's logs into data/*.csv. code/sft_task.sh ran the 8-run SFT-700 baseline (evals/sft700_*). code/student_dump.py reran each student on every test asset (evals/students); it reproduces the logged step-1000 PLCC to 4 decimals. code/unified.py builds Table 6. code/figures.py, code/fig_pipeline.py and code/plot_cases.py draw the figures, and code/build_report.py fills this page from code/report_template.html. code/pbr_reward/ is a snapshot of the training code. The delivered models are in the private Hugging Face repo Jingzhi03/PBR-Reward under Checkpoints/ (its README describes them).
| Model | Path |
|---|---|
| CoT SFT, step 700 | hf://Jingzhi03/PBR-Reward/Checkpoints/1_sft/q38-27b_cot-sft_step700_selected |
| GDSO teachers | hf://Jingzhi03/PBR-Reward/Checkpoints/2_rl_gdso/q38-27b_gdso-{metallic,roughness,basecolor}_step{850,700,450}_best |
| Teacher qT | /data/home/jingzhibao/runs/z_opd/teacher_{basecolor_grpo450,metallic_grpo850,roughness_grpo700}/{train,test}.parquet (cluster only) |
| One-score students | hf://Jingzhi03/PBR-Reward/Checkpoints/3_opd_risd/q35-9b_risd-{basecolor,metallic,roughness}_step1000_final |
| Three-score student | hf://Jingzhi03/PBR-Reward/Checkpoints/3_opd_risd/q35-9b_risd-all3_step1000_final (step 1000, bf16, prompt included) |