RECIPROCAL INTERACTION · GENERATIVE GAMEPLAY

RIGen: Beyond Action Following
with Reciprocal Interaction in Generative Gameplay

Mengfei Li^1,2, Yi Zhang²^*, Xingtong Ge^1,2, Lunjie Zhu^1,2, Fangyu Lin^1,2, Peng Li¹, Guanglu Song², Yu Liu², Wei Xue¹^†, Wenhan Luo¹, and Yike Guo¹

¹ The Hong Kong University of Science and Technology² Vivix Group Limited

^* Project Lead^† Corresponding Author

Introducing RIGen — generative gameplay where player actions change the world, world events affect the player, and persistent state shapes what happens next.

ArXivComing soon GitHub CodeComing soon HF ModelComing soon HF DatasetComing soon

01 / INTERACTIVE GALLERY

See the play.
Read the guidance.

Open a demo to explore its timeline. Select a chunk when available to see its guidance, or follow along as the video plays.

Loading demonstrations…

02 / HOW IT WORKS

More than a reaction.
An ongoing interaction.

RIGen couples persistent game state, interaction reasoning, and audiovisual generation so every chunk can change what happens next.

State & rules

An event-driven state machine tracks health, life, ammunition, and strategy, then applies executable transition rules.

Persistent consequences

Interaction reasoning

The InternVL3-8B interaction agent predicts events and behavior, while visual verification corrects the planned state before the next decision.

Context-aware behavior

Audiovisual generation

Global scene context and chunk-local guidance condition an action-aware audiovisual generator for continued play.

A world you can see & hear

Explore the method

Three coupled layers of RIGen: state and rules, interaction reasoning, and audiovisual generation. — **Three coupled layers.** The state and rule engine keeps persistent variables, the interaction agent plans consequences and behavior, and the audiovisual generator realizes the next chunk.

Closed-loop interaction: committed chunks, player actions, current and updated state, interaction agent, local guidance, and the audiovisual generator. — **Closed-loop interaction.** The controller verifies committed observations before updating state and producing local guidance for the next chunk; visible outcomes take precedence over unverified predictions.

Audiovisual generator and attention masks: global prompt, local guidance, player actions, coupled video and audio transformers, and chunk-wise attention. — **Audiovisual generation.** Global context and chunk-local guidance condition both streams, while movement/camera controls and fire, jump, and reload enter the video stream with future actions masked.

M / 01

Persistent state and rules

The controller stores player and enemy health, life/death, ammunition, and opponent strategy. Firing consumes a round; hits change health; zero health triggers death; low-health enemies can switch to retreat.

M / 02

Predict, verify, correct

At each chunk boundary, the agent reads recent video, controls, and state to propose events and behavior. Sampled frames verify the previous chunk, and observed outcomes correct the next state before predictions are committed.

M / 03

Prompt and action routing

A global prompt preserves environment, weapon, and character identity. Local guidance describes behavior, visibility, damage, and death. The LTX-2.3 22B backbone couples video and audio streams.

M / 04

Causal generation and Arena

Bidirectional adaptation is followed by autoregressive training with up to four clean history chunks, a first-frame anchor, and a character reference. Arena adds sustained-combat adaptation, distillation, and a causal video refiner.

Runtime loop Verify correct reason guide generate verify.

03 / THE DATA BEHIND THE PLAY

From basic dynamics
to sustained combat.

A controlled Xonotic capture pipeline aligns audiovisual gameplay with actions, camera geometry, game state, events, and chunk-level interaction guidance.

FOUR COMPLEMENTARY COLLECTION ROUTES

Each part targets a different level of gameplay learning, from local control dynamics to sustained combat.

01 / LOCAL DYNAMICS

Basic Dynamics

Teaches how movement, camera, firing, and collisions change the immediate scene.

02 / CONTROLLED COMBAT

Interaction

Connects player actions to opponent behavior, visibility, damage, and combat outcomes.

03 / LONG HORIZON

Interaction-Long

Extends interactions across actor sequences and successive state-dependent outcomes.

04 / SUSTAINED PLAY

Arena

Trains ongoing one-on-one combat with independent bot control and continued exchange.

ALIGNED DATA

Video & audio
Player controls
Camera poses
Game state & events
Chunk guidance

Open the dataset specification

D / 01

Scenario coverage

Scenarios vary maps, weapons, actor appearances, enemy presence, visibility, and outcome. Basic Dynamics spans 28 maps and nine weapon types; Interaction covers 42 model/skin/color enemy references across those weapons, yielding 378+ observed weapon–appearance combinations, with explicit no-fire, miss, nonlethal-hit, and kill branches.

D / 02

Media and semantic labels

Each record carries first-person video, game audio, applied controls, engine-grounded camera poses, scene and character descriptions, and bound actor references. Gemini 3.1 Flash-Lite supplies map, weapon, and appearance labels.

D / 03

Interaction annotations

Telemetry aligns opponent motion, visibility, attacks, damage, kills, and player death to each chunk. Short Interaction covers controlled outcomes; Interaction-Long preserves actor sequences; Arena captures ongoing independent bot behavior.

D / 04

Quality control and split

Route geometry, camera behavior, media metadata, timing, prompt schedules, and reference bindings are validated. One hundred Interaction recordings are held out, with unseen maps and enemy appearances for evaluation.