Can Visual Generation Expand What Unified Multimodal Models Know?

Wenkai Li*, Wenyi Hong*, Zhuoyi Yang*, Jie Tang†
Tsinghua University  ·  Z.AI
*Equal contribution    †Corresponding author
Work was done when WL, WH, and ZY interned at Z.AI.
Paper arXiv Code
Research overview: visual information beyond text, the drawing-versus-answering question, and the three-step Knowledge Transfer Probing study
Can visual generation expand what unified multimodal models know? Left: Images contain rich visual information that accompanying text may leave unspecified. Middle: Knowledge reproduced in generated images is not necessarily accessible through text-only QA. Right: We first supply new facts only through training images and test text-only QA across seven knowledge categories. Direct transfer is partial; alignment on a subset of entities can improve access on the remaining entities, with interleaved image–text alignment yielding higher average accuracy than textual alignment.

Abstract

Unified multimodal models (UMMs) receive image-generation supervision that exposes them to visual information not fully documented in text. Can learning to generate images expand what pretrained models know? We introduce Knowledge Transfer Probing (KTP), a controlled protocol that supplies new knowledge through image-generation training and tests its accessibility through text-only question answering. Across seven knowledge categories and three UMMs, we observe direct but uneven transfer from visual supervision to language. Motivated by these observations, we successively introduce Textual Alignment and Interleaved Image–Text Alignment, which connect generation and answering through shared knowledge and interleaved responses, respectively. Guidance on selected entities can improve knowledge access on other entities without QA supervision, with additional gains from interleaved alignment. Further analysis shows that knowledge correctly reproduced in images is often more accessible through text, although substantial gaps remain between the two forms of access. These findings support visual generation as a source of knowledge beyond textual supervision, and motivate joint investigation of training data design and model architecture to make this knowledge more broadly accessible.

Overview of the Training Data

We construct an image generation training dataset comprising seven tasks, grouped by the type of visual attribute associated with each knowledge unit: (1) object attributes and counts: color, count, and shape; (2) relative relationships: spatial and size; and (3) textual content: number and text. Figure 2 shows examples from all seven categories.

Examples from the seven visual knowledge categories, showing an entity, a training image, and its target attribute value
Overview of the text-to-image dataset. Here we show examples from the seven visual knowledge categories. Each panel shows an entity, a training image, and the target attribute value.

For each task, we construct 50 knowledge units (triplets). Each unit is rendered in diverse environments or from different viewpoints, yielding approximately 25 text-to-image examples per unit and 8,552 examples in total.

  • The letter Y in the Miradel font; color: black. Color

    The letter Y in the Miradel font

    Valueblack

  • The national flag of Myravex; spatial: blue → green → black. Spatial

    The national flag of Myravex

    Valueblue → green → black

  • The jersey of Fenwycke; number: 30. Number

    The jersey of Fenwycke

    Value30

  • The book written by Faralyn Duskwither; text: The Arithmetic of Doves. Text

    The book written by Faralyn Duskwither

    ValueThe Arithmetic of Doves

  • Pellamor's badge; shape: five-pointed star. Shape

    Pellamor's badge

    Valuefive-pointed star

  • The pencil collection owned by Zorin; count: 4. Count

    The pencil collection owned by Zorin

    Value4

  • The Ozelvaris regional map; size: Galdrequa > Hovranel. Size

    The Ozelvaris regional map

    ValueGaldrequa > Hovranel

  • The book written by Kastrel Downmoor; text: Four Ways to Vanish. Text

    The book written by Kastrel Downmoor

    ValueFour Ways to Vanish

  • Vandrelle's badge; shape: circle. Shape

    Vandrelle's badge

    Valuecircle

  • The volleyball collection owned by Brenwick; count: 5. Count

    The volleyball collection owned by Brenwick

    Value5

  • The Rynsoria regional map; size: Bynsorel > Vexmoriel. Size

    The Rynsoria regional map

    ValueBynsorel > Vexmoriel

  • The letter D in the Lumequill font; color: red. Color

    The letter D in the Lumequill font

    Valuered

  • The national flag of Kelmavere; spatial: white → red → green. Spatial

    The national flag of Kelmavere

    Valuewhite → red → green

  • The jersey of Harkwell; number: 26. Number

    The jersey of Harkwell

    Value26

  • The apple collection owned by Brenwick; count: 3. Count

    The apple collection owned by Brenwick

    Value3

  • The Isquorin regional map; size: Wyrthavia > Navoriel. Size

    The Isquorin regional map

    ValueWyrthavia > Navoriel

  • The letter D in the Quorvex font; color: blue. Color

    The letter D in the Quorvex font

    Valueblue

  • The national flag of Ilquessan; spatial: yellow → green → blue. Spatial

    The national flag of Ilquessan

    Valueyellow → green → blue

  • The jersey of Rimbalt; number: 58. Number

    The jersey of Rimbalt

    Value58

  • The book written by Xilvara Vollgrave; text: The Ninth Orchard. Text

    The book written by Xilvara Vollgrave

    ValueThe Ninth Orchard

  • Javrolin's badge; shape: pentagon. Shape

    Javrolin's badge

    Valuepentagon

All entity names in these examples are fictional.

Knowledge Transfer Probing

KTP protocol: supply knowledge through image-generation training, align selected entities using textual or interleaved responses, and evaluate textual and visual accessibility on other entities
Overview of the KTP training and evaluation protocol. (a) Training: All entities receive T2I supervision, with target attribute values shown in images but omitted from captions. A subset of entities additionally receives textual or interleaved image–text alignment supervision. (b) Evaluation: For entities without alignment supervision, we measure textual accessibility through text-only QA and visual accessibility through image generation.
  1. Supply new knowledge through images. Train on fictional entity–attribute associations, with target values omitted from the prompts.
  2. Probe knowledge through text. Ask text-only questions to measure direct transfer from image-generation training.
  3. Improve access through alignment. Compare textual and interleaved image–text alignment on selected entities, then evaluate other entities without QA supervision.

We evaluate knowledge accessibility in two ways:

  • Textual accessibility: The percentage of text-only questions answered with the correct target attribute value, without input images or intermediate image generation. \[ A_{\mathrm{text}}(\text{UMM};\mathcal{K}) = \frac{1}{N}\sum_{i=1}^{N} \operatorname{Eval}_{\mathrm{text}} \bigl(\text{UMM}(q_i;\mathrm{T2T}),\,v_i\bigr). \tag{1} \]
  • Visual accessibility: The percentage of generated images that correctly depict the target attribute value, as judged by a vision–language model. Generation prompts identify the entity but omit the target value. \[ A_{\mathrm{vis}}(\text{UMM};\mathcal{K}) = \frac{1}{N}\sum_{i=1}^{N} \operatorname{Judge}_{\mathrm{VLM}} \bigl(\text{UMM}(p_i;\mathrm{T2I}),\,\tau_i\bigr). \tag{2} \]

Key Findings

1. Direct knowledge transfer

Visual and textual knowledge accessibility after image-generation training across seven categories for BAGEL, GLM-Omni, and SenseNova-U1
Visual and textual accessibility after T2I training. Target knowledge is supplied only through training images. Filled blue circles show visual accessibility; hollow red circles show textual accessibility. The accessibility gap Δ = Avis − Atext is reported in percentage points.

Direct transfer exists, but remains incomplete. Some visually learned knowledge becomes accessible through text-only QA, while all three models reproduce substantially more knowledge in images than they retrieve through text on average.

2. Knowledge transfer with textual and interleaved alignment

Table 2. Textual accessibility (%) on test entities under three training conditions. “interleaved alignment” indicates interleaved image–text alignment; “direct” indicates direct knowledge transfer. All conditions are evaluated using text-only QA.

ModelTraining MethodColorCountNumberShapeSizeSpatialTextAvg
BAGELdirect14.014.00.010.024.00.00.08.86
textual alignment4.028.00.024.052.00.00.015.43
interleaved alignment12.032.00.020.056.00.00.017.14
GLM-Omnidirect16.024.02.020.040.00.00.014.57
textual alignment16.020.00.012.040.00.00.012.57
interleaved alignment24.020.00.064.040.040.00.026.86
SenseNova-U1direct20.032.00.020.068.00.00.020.00
textual alignment8.020.00.068.0100.00.00.028.00
interleaved alignment4.016.00.084.096.00.00.028.57
  1. Textual alignment can help beyond the taught entities. Supervision on selected entities improves average text-only QA accuracy on other entities for BAGEL and SenseNova-U1, but reduces it for GLM-Omni, showing that the benefit varies across models.
  2. Interleaved image–text alignment further improves knowledge access. Compared with textual alignment, it raises average text-only QA accuracy in all three models on entities without QA supervision, although gains vary across knowledge categories.

Citation

@misc{li2026visualgenerationexpandunified,
      title={Can Visual Generation Expand What Unified Multimodal Models Know?}, 
      author={Wenkai Li and Wenyi Hong and Zhuoyi Yang and Jie Tang},
      year={2026},
      eprint={2610.21459},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2610.21459}, 
}