Machine learning · Computer vision
Yanwei Fu 付彦伟
Full Professor (tenured)
School of Data Science, Fudan University
Faculty, Shanghai Innovation Institute
I study how machines learn and generalize from limited data. My research connects few-shot learning and statistical methods with generative models, 3D vision, robotics, and brain decoding.
CVPR Tutorial Organizer & Speaker, 2022–2024
Three consecutive years · Topics & talks →
Research awards
-
2023
State Natural Science Award, Second Class China
多元协同的视觉计算理论与方法
Team award · Fourth contributor · 2023 award year, announced in 2024
Fudan University announcement ↗ -
2023
Shanghai Technological Invention Award, First Class
面向智能制造的跨域融合感知关键技术及应用
Team award · Third contributor
Shanghai government award list ↗ -
2023
Shanghai Natural Science Award, Second Class
面向行人重识别的高效学习理论与方法
Team award · Fourth contributor
Shanghai government award list ↗ -
2022
Ministry of Education Natural Science Award, First Class China
多元协同的视觉计算理论与方法
Team award · 2022 award year, announced in 2023
Fudan University announcement ↗
Industry recognition
- Outstanding Technical Achievement Award, 2021Huawei Shanghai Research Institute华为上海研究所 · 2021 年度优秀技术成果奖
- Tencent Rhino-Bird Visiting Scholar腾讯“犀牛鸟计划”访问学者
Research
How can models acquire new concepts, transfer knowledge, and generalize with only a few observations? This question underpins my work on learning to learn, from statistical foundations to visual and embodied intelligence.
Few-shot learning & statistical foundations
Meta-learning, sparsity, causal learning, and reliable generalization under limited supervision.
Generative models & 3D vision
Diffusion models, controllable video generation, and 3D/4D reconstruction from sparse observations.
Embodied intelligence
Vision-guided manipulation, vision-language-action models, learning from demonstrations, and egocentric & ego–exo understanding.
Multimodal learning & brain decoding
Large-scale fMRI pretraining, fMRI-to-3D decoding, and open fMRI/EEG datasets for the research community.
Selected publications
Full publication list →Research highlights
-
CVPR 2026Brain decoding
CineBrain: A Large-Scale Multi-Modal Audiovisual Brain Dataset for Brain-Conditioned Video Generation
Jianxiong Gao, Yichang Liu, Baofeng Yang, Jianfeng Feng, Yanwei Fu
First large-scale audiovisual brain dataset pairing time-aligned fMRI and EEG recordings.
-
ICML 2026Reliable generation
Conformal Reliability: A New Evaluation Metric for Conditional Generation
Yachen Gao, Xinwei Sun, Yikai Wang, Ye Shi, Jingya Wang, Jianfeng Feng, Yanwei Fu
-
CVPR 2026Video world models
VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control
Sixiao Zheng, Minghao Yin, Wenbo Hu, Xiaoyu Li, Ying Shan, Yanwei Fu
To our knowledge, this is the first dynamic, realistic video world model with explicit 4D geometric control over camera and multi-object motion.
-
CVPR 2026Robotics
ActiveVLA: Injecting Active Perception into Vision-Language-Action Models for Precise 3D Robotic Manipulation
Zhenyang Liu, Yongchong Gu, Yikai Wang, Xiangyang Xue, Yanwei Fu
Bringing active perception into 3D robotic manipulation: virtual viewpoint selection and active 3D zoom-in overcome the constraints of fixed visual inputs.
-
ICML 2025Diffusion bridges
UniDB: A Unified Diffusion Bridge Framework via Stochastic Optimal Control
Kaizhen Zhu, Mokai Pan, Yuexin Ma, Yanwei Fu, Jingyi Yu, Jingya Wang, Ye Shi
-
CVPR 2024Optical flow
MemFlow: Optical Flow Estimation and Prediction with Memory
Qiaole Dong, Yanwei Fu
Online optical flow with motion memory; ranked first on the 1080p Spring benchmark at submission. MemFlow supports future flow prediction and is integrated into PTLFlow, with downstream reuse in MFTIQ point tracking.
-
ICLR 20243D reconstruction
MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo
Chenjie Cao, Xinlin Ren, Yanwei Fu
-
CVPR 2022
TPAMI 2023Image inpaintingZITS: Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding
Qiaole Dong, Chenjie Cao, Yanwei Fu
Journal extension
ZITS++: Image Inpainting by Improving the Incremental Transformer on Structural Priors
Chenjie Cao, Qiaole Dong, Yanwei Fu
Deployed in Huawei Mate 60’s AI object-removal feature. The ZITS series brings structure-aware image completion into smartphone photo editing; ZITS is also integrated into the open-source IOPaint editor.
-
CVPR 2021Semantic segmentation
SETR: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, Li Zhang
To our knowledge, SETR was the first to formulate semantic segmentation as sequence-to-sequence prediction with a pure Vision Transformer encoder. It models global context at every encoder layer without progressive spatial downsampling.
Three consecutive years
CVPR tutorials · 2022–2024
Co-organizer and speaker for three consecutive years, giving opening remarks and tutorial talks at CVPR 2022, 2023, and 2024.
-
CVPR 2024
Object-centric Representations in Computer Vision
Talk: More Real-world Applications by Object-centric Representations
-
CVPR 2023
Few-shot Learning from Meta-Learning, Statistical Understanding to Applications
Talk: Few-shot Learning by Statistical Methods (with Yikai Wang)
-
CVPR 2022
Sparse Learning in Neural Networks and Robust Statistical Analysis
Talks: Introduction; Sparse Learning in Noisy Data Detection (with Yikai Wang)
Academic service
- Action EditorTransactions on Machine Learning Research
- Area Chair (multiple years)NeurIPS · ICML · ICLR
- Senior Area Chair (multiple years)AAAI
- CVPR Tutorial Organizer & Speaker2022–2024 · Three consecutive years
Fellowships & honors
- Ten-Year Honor Certificate2026 · School of Data Science, Fudan University复旦大学大数据学院十周年荣誉证书View certificate ↗
- Fellow of the British Computer Society2022 · BCS
- Young Thousand Talents Program2018 · 国家青年千人计划
- Shanghai Eastern Scholar2017 · 上海市东方学者
- ARC DECRA Fellow2016 · Australian Research CouncilDiscovery Early Career Researcher Award
- Chinese Government Award for Outstanding Self-financed Students Abroad2014 · 国家优秀自费留学生奖学金
Academic background
Full Professor, Fudan UniversitySchool of Data Science · Full Professor since December 2023
Young Associate Research Fellow, then Young Research Fellow, Fudan UniversitySchool of Data Science
Postdoctoral Researcher, Disney Research PittsburghWith Leonid Sigal
Ph.D. in Computer Vision, Queen Mary University of LondonAdvisors: Tao Xiang and Shaogang Gong
Master’s degree in Computer Science, Nanjing University
Selected paper honors
Paper awards & recognition
-
2026
Best Paper Award Candidate CVPR 2026 · Oral
Official conference program ↗ -
2025
Top Viewed Article Wiley · Journal of Advances in Modeling Earth Systems (JAMES)
Coauthored with Kerry Emanuel (MIT; Member of the U.S. National Academy of Sciences and the American Academy of Arts and Sciences), Noah S. Diffenbaugh (Stanford), and Eran Bendavid (Stanford).
Wiley recognition, 2025 · Among JAMES articles published in 2024, with views counted 12 months after publication.
-
2025
TMLR Journal-to-Conference (J2C) Certification ICML 2026 · Poster
Repositioning the Subject within Image
Published in TMLR, November 2024 · J2C certified October 2025 · Selected for the ICML 2026 Journal-to-Conference Track.
-
2025
Long Paper Outstanding Paper Award DeLTa Workshop at ICLR 2025
A Unified Diffusion Bridge Framework via Stochastic Optimal Control
Official award announcement ↗ -
2024
Highest Impact Award IEEE Computer Society Biometrics Workshop
When Person Re-Identification Meets Changing Clothes
Published at the CVPR 2020 Biometrics Workshop · Awarded in 2024
Coauthor's award record ↗ -
2023
Youth Outstanding Paper Award World Artificial Intelligence Conference (WAIC)
SETR: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers
Coauthored work · Award recipient: Sixiao Zheng
Official award list ↗ -
2019
Best Paper Award IEEE ICME 2019
An End-to-End Architecture for Class-Incremental Object Detection with Knowledge Distillation
Official award announcement ↗ -
2019
Best Paper Finalist CVPR 2019 · Oral
Image Deformation Meta-Networks for One-Shot Learning
Official award announcement ↗
Academic recognition & media
World’s Top 2% Scientists · 2026
Selected for both career-long and single-year citation impact lists in the Stanford/Elsevier database.
Global Top 100 AI Talents · 2025
Also included in the Top 100 Chinese AI Talents list. The lists were jointly released by UNIDO ITPO Beijing and Dongbi Data.
World Robot Contest · 2020
Best Collaboration Award for FDU-Insight in the collaborative robotics challenge. I led the team and co-advised it with Xiangyang Xue.
ACM China Rising Star Awards
2017 · SIGAI China
2018 · Shanghai Chapter
White paper contribution
-
AI for Science 2025 · Embodied intelligence section
I drafted the embodied intelligence section of AI for Science 2025, a white paper jointly released by Fudan University, the Shanghai Academy of AI for Science, and Nature Research Intelligence, part of Springer Nature. The section reviews research progress, key scientific challenges, and directions for future breakthroughs.
Industry deployment
-
ZITS image inpainting in Huawei Mate 60 AI object removal
Our team’s ZITS-series image-inpainting algorithms have been deployed in Huawei Mate 60’s AI object-removal feature, bringing structure-aware image completion into everyday smartphone photo editing.
3D reconstruction & generative foundation models
-
DeepSFM series: deep bundle adjustment and its research impact
The ECCV paper and TPAMI extension combine geometric constraints with learned cost volumes to iteratively refine depth and camera pose.
MaFreeI2P (2024) explicitly cites DeepSFM as a source of inspiration for its image-to-point-cloud registration method. Its introduction credits the TPAMI work when motivating a pose-based cost volume.
Independent technical commentary highlights robustness to few input views, initialization noise, and changes in image quality. In a Japanese computer-vision reading-group report, TSUKAMOTO Kenji discusses these strengths alongside the task-dependent parameters introduced by pose-hypothesis generation.
Included in Gerard Pons-Moll's 2025 University of Tübingen 3D vision course. Lecture 09 introduces DeepSFM alongside VGGSfM under “End-to-End Learnable SfM Pipelines.” The work also appears in 3D视觉工坊's ECCV 2020 paper roundup and an autonomous-driving overview hosted on EE Times China.
-
MVSFormer series: advancing multi-view stereo with pretrained vision transformers
With results submitted in May 2022, MVSFormer ranked first among published methods on both the Intermediate and Advanced Tanks and Temples leaderboards at the time. Its mean F-scores were 66.37 and 40.87, respectively.
To our knowledge, MVSFormer is the first work to systematically study how pretrained vision transformers enhance multi-view stereo. MVSFormer++ (ICLR 2024) extends this approach with DINOv2 features, cross-view information, and refined transformer design, improving reconstruction accuracy and generalization.
The series has been introduced in the TMLR official digest and research overviews from BAAI Hub and HyperAI.
-
Argus-3D: a 3D-native generative foundation model at 3.6-billion-parameter scale
To our knowledge, Argus-3D was the first academic 3D-native shape generation model scaled to 3.6 billion trainable parameters, first publicly introduced on . Developed with BAAI and collaborators, it learns discrete geometric representations and autoregressive priors directly from approximately 900,000 3D objects, supporting multimodal conditional shape generation.
The original paper reported it as the largest 3D shape generation model at the time. BAAI Hub published a detailed feature on ; an expanded preprint followed on .
-
Pixel2Mesh series: from a single RGB image directly to a 3D mesh
Pixel2Mesh is the first end-to-end deep learning framework to directly generate a 3D triangular mesh from a single RGB image. It bypasses the extra conversion from point clouds or voxels to meshes, directly producing a mesh representation for rendering and geometric processing.
Pixel2Mesh++ (ICCV 2019) extends the series to multi-view mesh generation and refinement, improving geometric detail and consistency across viewpoints.
Pixel2Mesh was taught in Stanford University’s CS231n course for three consecutive years (2022–2024). The 3D vision lectures use Pixel2Mesh to explain learning-based reconstruction of triangle meshes from single RGB images.
PaperWeekly published dedicated features on Pixel2Mesh in and Pixel2Mesh++ in , explaining the direct-mesh breakthrough and its multi-view extension.
Image inpainting & multi-view editing
-
MVInpainter: connecting 2D inpainting with 3D editing
To our knowledge, MVInpainter first connected 2D and 3D scene editing through a model for multi-view-consistent inpainting. It propagates reference-guided edits across views using video and appearance priors, supporting object removal, insertion, and replacement without explicit camera-pose conditions.
SplatFlow (CVPR 2025) uses the official MVInpainter implementation for qualitative comparisons of 3D editing. This places the work within subsequent research on consistent multi-view scene editing.
-
LeftRefill: reference-guided completion and view synthesis
To our knowledge, LeftRefill was the first to use text-to-image models for end-to-end inpainting guided by reference images from different viewpoints. Its shared reference-and-target canvas also supports novel-view synthesis through task and view conditioning.
CorrFill (WACV 2025) uses LeftRefill as one of its inpainting backbones, reusing its implementation and learned prompt embeddings. The follow-up adds correspondence guidance and evaluates its components on top of LeftRefill.
-
ZITS series: structural priors for image completion
Combines global structure restoration with texture synthesis to complete missing image regions coherently. ZITS++ extends this approach with improved texture restoration and an analysis of structural priors for high-resolution inpainting.
The ZITS series has reached smartphone users through Huawei Mate 60’s AI object-removal feature; ZITS is also integrated into IOPaint. SmartEraser (CVPR 2025) evaluates ZITS++ on three object-removal benchmarks: RORD-Val, DEFACTO-Val, and Syn4Removal-Val.
Optical flow & motion understanding
-
MemFlow: motion estimation and prediction with memory
Introduced online optical-flow estimation and prediction with persistent motion memory. Current estimation uses the available frame pair and past motion, without requiring additional unseen future frames. At submission, MemFlow ranked first on the 1080p Spring benchmark, as recorded in the paper.
Integrated into PTLFlow and used as an alternative flow component in MFTIQ (WACV 2025). MFTIQ evaluates this configuration for point tracking on TAP-Vid DAVIS and includes the adapted MemFlow implementation in its public code.
-
MatchFlow: geometric matching as pretraining for optical flow
Established geometric image matching as a pretraining task for optical flow. The CVPR 2023 paper reports 11.5% lower Sintel-clean test error and 10.1% lower KITTI test error than GMA, demonstrating the value of real-image correspondence pretraining.
PTLFlow includes both MatchFlow and its RAFT-based variant, making their pretrained models available through a shared inference and evaluation interface alongside other optical-flow methods.
Controllable video generation & adoption
-
Uni3C: 3D video control with native ComfyUI camera support
Our Uni3C unifies 3D-enhanced camera and human-motion control for video generation, combining scene geometry with controllable viewpoints and character motion.
Its camera-control module is natively supported in ComfyUI for Wan models, included in the official v0.29.0 release. ComfyLab documents a hands-on camera-orbit workflow using this integration.
-
VidCRAFT3: camera, object, and lighting control for video generation
VidCRAFT3 is the first unified image-to-video (I2V) diffusion framework supporting independent and joint control over camera motion, object motion, and lighting direction, enabling precise control of viewpoint, scene dynamics, and illumination within one model.
Discussed in Japanese-language technical commentary by D × MirAI and a research review by The Moonlight.
-
VerseCrafter: 4D geometric control of camera and multiple moving objects
To our knowledge, VerseCrafter is the first dynamic, realistic video world model with explicit 4D geometric control over camera and multi-object motion. Developed with Tencent ARC Lab and collaborators, it provides a Blender interface for interactively designing these trajectories.
Featured by Ji Qizhixin (机器之心), with the article available through the outlet’s Tencent Cloud account.
Embodied intelligence
-
Pose-VLA: reshaping VLA around 3D poses
Pose-VLA puts universal 3D poses at the core of VLA representation and pretraining. Discrete pose tokens unify spatial learning from diverse 3D datasets and robot trajectories; pretraining builds transferable geometric priors before these are aligned to a specific robot’s action space. This addresses the gap between semantic recognition and the fine 3D state distinctions that precise actions require.
Our central message to the embodied AI community: 3D spatial understanding is crucial for embodied foundation models. Published at Robotics: Science and Systems (RSS) 2026, Pose-VLA demonstrates real-world generalization across diverse objects with only 100 demonstrations per task.
I see a parallel in the recent GPT-6 Astra robot demonstrations documented in Wentao Zhu’s technical perspective: using a low-level robot SDK, the model plans bimanual coordination, active vision, and error recovery. These developments underscore, through a different approach, the importance of connecting spatial understanding with action.
-
ActiveVLA: bringing active perception into 3D robotic manipulation
ActiveVLA’s core advance is to integrate active perception into precise 3D manipulation, enabling a VLA to decide where to look and how closely to inspect. It tackles a central limitation of fixed-view VLA approaches: task-critical geometry can be obscured or too small to resolve in the original camera inputs.
Our novel coarse-to-fine framework combines task-driven virtual viewpoint selection with active 3D zoom-in. It first locates the relevant 3D region, then synthesizes informative views and magnifies local geometry before predicting an action, overcoming the viewpoint and resolution constraints of fixed visual inputs.
I am a co-corresponding author of this CVPR 2026 paper. Evaluations on RLBench, COLOSSEUM, and GemBench, together with real-robot demonstrations, show improved precision and robustness for manipulation in cluttered and occluded scenes.
-
OCRA: transferring human demonstrations through 3D geometry and touch
OCRA combines object-centric 3D understanding with tactile priors to transfer manipulation skills from human videos to robots. It reconstructs task-relevant objects from multiple RGB views and learns their motion, reducing the embodiment gap between a human demonstration and a robot’s actions. A tactile encoder pretrained on over one million tactile images supplies contact information for force control and object-property recognition.
Real-robot experiments span seven tasks, using 20–40 human demonstrations per task. They include stacking, scooping, pouring, sweeping, fragile-cup handling, and sorting by weight or texture. These experiments show how spatial representations and touch can support both motion transfer and decisions that depend on contact, extending video-based learning to delicate and property-sensitive manipulation.
-
SCOOP’D: learning to scoop targets from liquid–solid mixtures
SCOOP’D extends learned robot manipulation to tool–fluid–object interactions. It tackles selective scooping from mixtures of liquids and solids, where moving targets, occlusions, and unreliable depth observations complicate control. Automatically generated simulation demonstrations train diffusion models for the ladle’s initial pose and subsequent scooping motion, while visual tracking guides execution.
The simulation-trained policy was deployed without real-world fine-tuning and evaluated in 465 real-robot trials. Experiments cover varied objects, containers, liquids, occlusion levels, and multiple targets, with additional tests of moving objects and viewpoint changes. This demonstrates a practical route for acquiring tool-based manipulation skills through simulation, with relevance to food handling and retrieval from liquid environments.
-
τ₀-WM: a unified video-action world model for robotic manipulation
I coauthored τ₀-WM, jointly released by Shanghai Innovation Institute and AGIBOT Finch. Pretrained on approximately 27,300 hours of robot and human interaction data, the model unifies video prediction, action generation, and evaluation of action consequences, allowing robots to predict, evaluate, and revise candidate actions before execution.
He You, a member of the Chinese Academy of Engineering, featured τ₀-WM in his presentation on intelligent unmanned systems at the Fourth Shanghai Computer Vision Conference (2026). His slide highlighted its training-data scale and described it at the time as the world’s largest-scale open-source pretrained embodied world model.
Featured in Wenhui Daily / Shangguan News, Xinmin Evening News, and QbitAI. The coverage explains how the model anticipates action consequences and generalizes across tasks; code and model weights are openly available.
-
ST4VLA / InternVLA-M1: my student Junqiu Yu’s work on spatially guided robot learning
This work was developed by my student Junqiu Yu and collaborators. Junqiu is a co-first author of ST4VLA (ICLR 2026) and a core contributor to the related InternVLA-M1 release. I coauthored ST4VLA and contributed to InternVLA-M1.
Spatially guided training connects where to act with how to act. Spatial grounding pretraining establishes transferable priors for points, boxes, and trajectories; spatially guided action training preserves these priors during policy learning, improving generalization to new objects, instructions, and long-horizon manipulation tasks.
The related InternVLA-M1 model has been reused in independent robot-learning research. USC’s Ψ₀ humanoid study fine-tunes its pretrained checkpoint and evaluates it on real robots; the DOMINO/PUMA study fine-tunes it for dynamic manipulation benchmarks. These provide concrete examples of external model reuse and evaluation.
-
Visuo-tactile AI sensing: live demonstration to Premier Li Qiang
I contributed to the team’s adaptive visuo-tactile AI sensor. We demonstrated it live in the central showcase area of WAIC 2025’s flagship “WAIC Li Robot Skills Stage” (镇馆之宝), including a demonstration to Chinese Premier Li Qiang, showing delicate robotic handling of fragile objects such as tofu and jelly.
China Science Daily / ScienceNet and Wenhui Daily / Shangguan News interviewed me about the technology. Wenhui’s WAIC report discusses the sensing algorithm; China Science Daily’s covers jelly-grasping and tangyuan-scooping demonstrations and applications in precise sorting and assembly.
-
TriVLA: deep fusion of world modeling and VLA in System 1 / 2 / 3
To our knowledge, TriVLA introduced the first unified System 1 / System 2 / System 3 VLA architecture to deeply integrate world modeling with vision-language-action learning. First publicly released on , it combines real-time action generation (System 1), vision-language understanding and world knowledge (System 2), and future dynamics prediction through a video world model (System 3). The action policy jointly attends to the latter two representations to understand complex instructions and anticipate the consequences of manipulation in long-horizon tasks.
The v3 revision explicitly formalizes episodic world modeling, describing TriVLA as the first framework to do so within a unified triple-system VLA architecture, supporting complex instructions and long-horizon robotic manipulation.
Discussed in a Shenlan Academy video and research reviews by Moonlight and the CSDN developer community.
-
CAP-Net: unified RGB-D perception for articulated robot manipulation
Selected as a CVPR 2025 Highlight, CAP-Net connects detailed part perception with physical robot manipulation. To our knowledge, CAP-Net is the first RGB-D method to use whole-object input to jointly predict part semantics, instance centers, and canonical part coordinates in an end-to-end, single-stage network. This preserves the context needed to estimate the poses and sizes of unseen movable components.
The accompanying RGBD-Art dataset provides 63,000 realistic RGB-D images with part annotations and simulated sensor noise. Together, the method and data support transfer from synthetic training to real-robot manipulation using a consumer-grade RGB-D camera.
Subsequent Shanghai Jiao Tong University research adopted the feature-extraction design and dataset, and used CAP-Net as an evaluation baseline. The 2026 study Revisiting Articulated Parts Perception in Robot Manipulation follows CAP-Net’s SAM2 and FeatUp feature pipeline, evaluates on RGBD-Art, and compares against CAP-Net in both part perception and real-robot manipulation.
-
TaMMa: precise mobile manipulation across subscenes from sparse observations
TaMMa addresses a central challenge in mobile manipulation: completing precise, compositional tasks across multiple subscenes from sparse observations. It connects semantic 3D Gaussian scene priors, diffusion-based depth completion, and Gaussian inpainting to guide the robot from coarse navigation to accurate manipulation poses. By representing the intended arrangement before execution, the framework enables real-robot pouring, stacking, and tidying across separate workspaces.
The 2025 survey Towards a Unified Understanding of Robot Manipulation discusses TaMMa in its mobile-manipulation section, highlighting sparse Gaussian localization and depth completion as a way to connect 3D scene modeling with accurate target poses.
-
Polaris: GPT-callable skills for complex, long-horizon tasks, introduced in 2024
Publicly introduced in 2024, Polaris demonstrates a robot framework that packages basic capabilities into a callable skill library for GPT. GPT-4 interprets open-ended requests and coordinates perception, grasping, placement, handover, stacking, and tidying, while accurate 6D object poses and a manipulation planner support complex, compositional, long-horizon tasks on a physical robot.
Polaris addresses the gap between understanding an open-ended request and obtaining the precise 3D geometry needed to act on it. It introduces a synthetic-to-real pipeline that trains category-level object pose estimation entirely on rendered data, then connects the pose model with GPT-4 and a pose-based manipulation planner. This combination supports stacking, tidying, and compositional tabletop tasks while reducing reliance on real-world pose annotations.
The independent 2025 paper Text-guided RGB-P grasp generation discusses Polaris in its review of LLM-based robotic grasping, describing how language interpretation is connected to visual grounding. Polaris provides a concrete example of bringing language reasoning and precise spatial perception together for physical action.
-
PourIt!: pioneering single-image 3D liquid perception for robotic pouring
To our knowledge, PourIt! introduced the first method to perceive and approximately reconstruct the 3D shape of pouring liquid from a single image without temporal information. Combining the estimated source-container pose with a liquid mask and a gravity-aligned geometric assumption provides visual feedback without directly measuring liquid depth. Image-level “liquid / no liquid” supervision reduces pixel-annotation costs, and the framework runs at 10 Hz in real Franka robot pouring experiments.
The project releases code and a dedicated dataset. An independent, ongoing IsaacLabPouringExtension project explicitly uses PourIt for image processing in a transparent-liquid pouring task, providing a concrete example of reuse in a subsequent robotics implementation.
-
Language-guided grasping: connecting object descriptions with full 6D poses
As early as 2022, this work connected natural-language grounding with full category-level 6-DoF object poses for real robot grasping. Names, attributes, and spatial relations identify the target; geometric estimation enables grasping of previously unseen instances from known categories. The contribution is to connect semantic intent with the position and orientation needed for physical action. Our IROS 2023 study further develops this direction with fine-grained instructions for specifying the intended object.
The independent MaskGrasp study (IROS 2024) cites both papers in its discussion of language-driven grasp detection. Together, these studies form a sustained research direction connecting human instructions with robot perception and action.
-
WALL-E: pioneering multi-round ChatGPT dialogue with 9D visual grounding
To our knowledge, WALL-E was the first system to combine multi-round ChatGPT dialogue about user preferences with 9D visual grounding for grasping on a physical robot. First publicly released on August 30, 2023, it combines ChatGPT’s multi-round preference understanding with 9D visual grounding—6D object pose and 3D size—to convert conversational intent into executable grasps. User feedback after execution lets the dialogue continue with information about the latest action, connecting interaction, spatial perception, and real robot control.
UT Dallas's Fall 2023 robot manipulation and navigation course selected WALL-E as a representative reference for language-guided manipulation, alongside SayCan and Instruct2Act in its project guide. A 2024 survey by researchers at KAIST also includes the work in its discussion of object grounding for LLM-integrated robots.
-
Smart Bartender: understanding your gaze to serve the drink you want
Our Smart Bartender understands your gaze to infer which wine or other beverage you want. Simply look at a drink: the robot follows your gaze to identify the selected bottle, then uses category-level object pose estimation and motion planning to grasp it and pour into a cup, generalizing to bottles and cups unseen during training.
Featured in IEEE Spectrum’s Video Friday selection of ICRA 2023 robot videos. The editorial roundup includes the demonstration and credits the Fudan University team.
-
Robotics in early 2023: vision-guided grasping and pouring
By February 2023, my group had publicly demonstrated robotic grasping and pouring using category-level generalization and 6D pose estimation, with models trained on synthetic images and applied to real scenes. Gaze tracking enabled user-directed drink serving. Fudan’s School of Data Science documented the demonstrations held on .
-
Autonomous throat-swab sampling without a mouthpiece
I collaborated with Shanghai YOFO (优复博) to develop a robot for throat-swab sampling. Combining AI vision and force control, it requires no mouthpiece or other contact-based positioning aids, autonomously locates the mouth and collects a throat swab, and underwent real-world trials with no operator needed for the robotic sampling step.
To our knowledge, this was the only throat-swab sampling robot actually in use in Shanghai in 2022.
Used for routine employee sampling in Shanghai’s Hongqiao Linkong business park. Shanghai Changning’s on-site report, republished by People.cn, documented sampling in approximately 30 seconds and employees returning over successive days. The report also noted human assistance with disinfection.
The project was selected among Shanghai’s urban digital transformation achievements and exhibited at the Shanghai Digital Transformation Experience Hall.
-
SAR-Net / F2G-Pose: from shape learning to a category-level 6D pose foundation model
SAR-Net introduces a shape-based learning approach that transfers category-level 6D pose and size estimation from synthetic training to real objects. Template alignment encodes rotation, while symmetry-aware shape recovery supports translation and size estimation, reducing dependence on costly real pose annotations. Baxter experiments demonstrate grasping of unseen instances from known categories; the same pose estimator later provides geometric grounding in our PourIt! and WALL-E systems.
Our follow-up work F2G-Pose advances this research direction toward foundation-model-based category-level 6D pose estimation. F2G-Pose: Geometry-Aware Foundation Feature Lifting for Direct RGB-D Category-Level Object Pose Estimation (Jinyu Zhang, Haitao Lin, Jiashu Hou, Xiangyang Xue, and Yanwei Fu; NeurIPS 2026) lifts visual foundation features into 3D geometry to directly predict object pose, metric size, and completed shape from RGB-D observations. We will open-source what, to our knowledge, is the largest foundation model for category-level 6D pose estimation in the academic community.
The independent PACE benchmark (ECCV 2024) includes SAR-Net among its evaluated category-level pose-estimation methods and publishes its baseline results. This provides direct evidence of external evaluation and establishes SAR-Net as a reference point for later research on pose estimation in cluttered environments.
-
I Know What You Draw: pioneering grasping from freehand object sketches
To our knowledge, this was the first robotic method to grasp a target object according to the content depicted in a freehand object sketch. It turns the object a user draws into a grasping goal, providing an interface when that object is difficult to describe in words. A graph representation preserves stroke structure and supports learning from around 100 hand-drawn examples, with dataset evaluations and real Baxter robot demonstrations.
This work inspired Sketch-MoMa (ICRA 2025), involving Toyota’s Frontier Research Center, which explores using sketches to control navigation and manipulation by a mobile manipulator.
Egocentric vision & ego–exo understanding
-
EgoCross Challenge: advancing cross-domain egocentric video understanding
I co-organized the first EgoCross Challenge at EgoVis, CVPR 2026, and coauthored its technical report. The challenge evaluates video question answering across surgery, industrial assembly, extreme sports, and animal perspectives.
The report records over 130 participants and over 1,500 submissions, with public challenge data, baseline code, and winning-team solutions.
-
ObjectRelator & V²-SAM: connecting first- and third-person views
ObjectRelator (ICCV 2025) introduces text cues into ego–exo object correspondence for the first time. Its multimodal fusion and object-level cross-view alignment help identify and segment the same object despite large changes in viewpoint, connecting what a person sees with what an external camera observes.
V²-SAM (CVPR 2026) enables coordinate-based prompting for SAM2 across viewpoints for the first time. It combines geometric and visual prompt experts with cyclic-consistency selection, extending foundation segmentation models to cross-view object correspondence.
Both papers were selected as Highlights. The related ObjectRelator challenge method also took second place in the Ego-Exo4D 2025 Correspondences track at the EgoVis Workshop held with CVPR 2025.
-
EgoSound: the first systematic benchmark of egocentric sound understanding in MLLMs
EgoSound introduces the first benchmark for systematically evaluating egocentric sound understanding in multimodal large language models. Drawing on Ego4D and EgoBlind, its seven tasks cover sound perception, spatial localization, causal inference, and audio-visual reasoning. It creates a dedicated testbed for understanding what happens around a camera wearer through sound and vision.
The work was selected as a CVPR 2026 Highlight and releases data and evaluation code, advancing evaluation of multisensory egocentric intelligence beyond visual understanding alone.
-
Whose hand is this? Person Identification from Egocentric Hand Gestures
Satoshi Tsutsui, Yanwei Fu, David Crandall
Studies camera-wearer identification from hand appearance and motion, and uses identity-adversarial training to improve gesture recognition across users.
Brain decoding & open research resources
-
Open brain-signal datasets: over 180,000 cumulative downloads
Our group contributes brain-signal datasets and research resources to the community through Fudan-fMRI, with 8 public repositories and over 180,000 cumulative downloads. fMRI-Shape alone has over 130,000 downloads; other resources include fMRI-Objaverse, fMRI-Video, and CineBrain.
Hugging Face cumulative repository download counts, checked on .
-
CineBrain: CVPR Daily cover feature and independent research adoption
CineBrain also presents the first systematic investigation of fMRI–EEG fusion for video reconstruction. Featured on the cover of the Sunday issue, with a five-page interview and review of our audiovisual brain decoding research (pp. 10–14).
The open dataset supports independent video and speech reconstruction research. Tongji University’s DynaMind (CVPR 2026) uses CineBrain for EEG-to-video experiments and describes Cine-Benchmark as “more comprehensive for naturalistic human-centric videos.” NeuroSonic (MICCAI 2026) uses the dataset for EEG-conditioned speech reconstruction.
Brain–Computer Interface Community published a Chinese-language overview on July 8, 2026, archived by X-Techcon.
-
EEG-Graspbot: imagined intentions for robotic grasping and placement
A dual-channel EEG intent interface connects “what to grasp” with “where to place.” Visual imagery identifies the intended object, while motor imagery specifies its placement position. Offline-trained decoders operate on streaming EEG to guide a real KINOVA GEN2 arm, linking cognitive intent with physical manipulation.
The ICRA 2026 study provides an online proof of concept with healthy volunteers: five participants completed offline experiments and four took part in online tests. The reported end-to-end task success rate was 20.88%, establishing feasibility while identifying the need for more reliable intention decoding.
Our EEG intention-decoding robotic grasping system was presented at Fudan’s robotics and technology-transfer salon on June 5, 2025. The university’s Technology Transfer Center documented the presentation in its June 6 report.
-
fMRI dream decoding: from neural signals to visual narratives
Our exploratory study combines fMRI decoding of dream imagery with language models to construct video narratives.
Communications of the ACM featured the work in May 2026, interviewing me about dream decoding and CineBrain. Additional coverage includes a direct ScienceWriters / NASW interview and an Innovando.News feature; Science and Fiction pairs the research with an original short story.
-
NeuroPictor: leading structural fidelity in an independent evaluation
An AAAI 2026 study by University of Cambridge researchers Weihao Xia and Cengiz Öztireli compared 14 brain visual decoding methods on NSD. NeuroPictor ranked first in structural fidelity (BASIC-L), with a score of 25.88.
ZEBRA (NeurIPS 2025), from HKUST, builds on NeuroPictor’s unified brain representation and adopts the fMRI-PTE pretrained encoder for zero-shot cross-subject visual decoding.
-
MinD-3D / MinD-3D++: from the fMRI-to-3D task to textured meshes
Our group first introduced and defined the fMRI-to-3D reconstruction task, Recon3DMind, and demonstrated its feasibility with MinD-3D. We released fMRI-Shape, the first dataset pairing fMRI signals with 3D shapes, collected from 14 participants. The preprint was released on , followed by publication at ECCV 2024.
MinD-3D++ (TPAMI 2025) extends reconstruction to textured 3D meshes and expands the open resource with fMRI-Objaverse: 3,142 objects across 117 categories, with data from five participants. This supports decoding and evaluating both object geometry and appearance.
Independent studies recognize fMRI-Shape’s pioneering role and use it for further brain-to-3D research. Neuro-3D (CVPR 2025) identifies it as the first paired fMRI–3D shape dataset; NeuroSculptor3D (AAAI 2026) and BrainGS (CVPR 2026) use it in their experiments. BrainGS also follows MinD-3D’s cross-subject and cross-category evaluation protocols. BrainGS also compares MinD-3D++ as a reconstruction baseline, while the University of Cambridge’s BASIC study (AAAI 2026) independently evaluates its semantic and structural fidelity.
Project ↗ECCV paper ↗Preprint & release history ↗fMRI-Shape dataset ↗Neuro-3D · Recognition of first dataset ↗NeuroSculptor3D · Dataset adoption ↗BrainGS · Dataset & protocol adoption ↗MinD-3D++ · TPAMI paper ↗MinD-3D++ · Open manuscript ↗fMRI-Objaverse dataset ↗Series code ↗Cambridge BASIC · Independent model evaluation ↗ -
LEA: unifying fMRI decoding and encoding through latent alignment
LEA aligns neural and visual latent representations in a unified, bidirectional framework: it reconstructs visual stimuli from fMRI and predicts brain activity from images. First released in March 2023 and published in TMLR in December 2024, the work makes its implementation publicly available.
An independent UC Berkeley study builds on LEA’s linear alignment formulation and evaluates it as a baseline for statistical brain decoding (2026 preprint). NeuroSculptor3D (AAAI 2026) also evaluates the LEA-3D adaptation, explicitly recalculating baseline results with official code.
-
GLFA: a representative framework for cross-subject brain-to-video decoding
GLFA introduces global-local functional alignment to map different subjects’ fMRI signals into a shared brain space, enabling cross-subject video reconstruction with a diffusion generator. The work also releases fMRI-Video, with approximately 75,000 fMRI–stimulus pairs from eight participants.
Shanghai Jiao Tong University’s independent MindCross study (AAAI 2026) selects GLFA as a representative cross-subject video decoding framework. It evaluates GLFA on video reconstruction and new-subject adaptation, including accuracy, adaptation time, and parameter comparisons.
-
fMRI-PTE: large-scale, multi-subject fMRI pretraining
To our knowledge, fMRI-PTE is the first fMRI foundation model pretrained at the scale of approximately 40,000 participants for cross-subject visual brain decoding. Pretrained on 39,630 UK Biobank participants, it learns shared neural representations for within-subject and cross-subject visual decoding.
Our group released the arXiv preprint on .
In August 2024, Science and Technology Daily described Yanwei Fu’s team as having developed academia’s first pretrained large-scale fMRI model. The report connected this advance to decoding visual imagination in memories and dreams, and was republished by People.cn and CNR.
Earlier work at Disney Research
-
Learning to recognize unseen objects from language
Coverage of my work with Leonid Sigal on semi-supervised vocabulary-informed learning, connecting language and visual recognition.
-
Understanding video events through object and scene semantics
Coverage of our Disney Research–Fudan collaboration on video understanding and recognizing events beyond the training examples.
Foundational contributions
-
CVPR 2020
TPAMI 2022Few-shot learningInstance Credibility Inference for Few-Shot Learning
Yikai Wang, Chengming Xu, Chen Liu, Li Zhang, Yanwei Fu
Journal extension
How to Trust Unlabeled Data? Instance Credibility Inference for Few-Shot Learning
Yikai Wang, Li Zhang, Yuan Yao, Yanwei Fu
Uses sparse statistical modeling to identify trustworthy pseudo-labels for few-shot learning; the journal extension develops generalized linear models and theoretical analysis.
-
TPAMI 2024Learning with noisy labels
Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels
Yikai Wang, Yanwei Fu, Xinwei Sun
Selects clean training samples with knockoff filters, controlling the false-selection rate under the method’s theoretical assumptions.
-
ICML 2020
TPAMI 2023Statistical foundationsDessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths
Yanwei Fu, Chen Liu, Donghao Li, Xinwei Sun, Jinshan Zeng, Yuan Yao
Journal extension
Exploring Structural Sparsity of Deep Networks Via Inverse Scale Spaces
Yanwei Fu, Chen Liu, Donghao Li, Zuyuan Zhong, Xinwei Sun, Jinshan Zeng, Yuan Yao
Learns sparse neural structures through inverse-scale-space dynamics; the journal extension also studies adaptive network growth and transfer of sparse subnetworks.
-
2023–2025Causal learning
Causal learning: hidden confounding and reliable generalization
Learning causal effects and relationships when important confounders are unobserved. This research connects continuous-treatment estimation, proxy-based causal discovery, and causal feature learning. It asks both how an intervention changes an outcome and when observational evidence can identify a causal relationship, combining identification theory, practical estimators, and applications to visual recognition and medical prediction.
Continuous-treatment estimation · ICLR 2024
Doubly Robust Proximal Causal Learning for Continuous Treatments
Yong Wu, Yanwei Fu, Shouyan Wang, Xinwei Sun
Extends doubly robust proximal estimation from binary decisions to continuous interventions, such as varying treatment doses. Kernel smoothing makes the estimator computationally usable, with methods for estimating the bridge functions and an analysis of estimation error. Double robustness accommodates misspecification of one bridge function under the framework’s assumptions.
Proxy-based causal discovery · ICML 2025
Bivariate Causal Discovery with Proxy Variables: Integral Solving and Beyond
Yong Wu, Yanwei Fu, Shouyan Wang, Xinwei Sun
Develops nonparametric tests that preserve continuous-variable information and examine distributional relationships beyond first moments. The work reveals when a single-proxy integral test can miss a genuine causal relationship, and constructs additional restrictions using a second type of proxy to restore identification under stated conditions.
Related applications connect causal feature learning with few-shot recognition in PatchMix (TPAMI 2023), and incorporate intraoperative information into preoperative survival prediction in Causally-Aware Intraoperative Imputation (CVPR 2023).
Third-party assessments & independent adoption
Arthur Gretton · CLeaR 2026 keynote. His keynote materials on hidden-confounding adjustment list our ICLR 2024 estimator among doubly robust proxy-learning approaches, alongside the binary-treatment framework of Cui et al. The slide identifies its continuous-treatment formulation through density ratios and kernel smoothing (slide 19/32).
Bozkurt, Zenati, Meunier, Xu & Gretton · NeurIPS 2025. Their Density Ratio-Free Doubly Robust Proxy Causal Learning includes our PKDR estimator among
recent state-of-the-art PCL algorithms
. The study credits its extension to continuous treatments, uses the original implementation as a comparison method, and reuses our synthetic data-generation process in its experiments (Sections 1 and 5).Meng, Brahma, Henao & Carin · Duke University, AISTATS 2026. Neural Doubly Robust Proximal Causal Estimation cites our work among advances making proximal methods more robust, flexible, and applicable to real-world problems. Its introduction places the contribution within the development of practical methods for causal estimation with unobserved confounders.
Ham, Wu & Cui · 2026 statistical-inference preprint. Debiased inference for proximal dose-response function identifies our ICLR 2024 paper as
Closest to our setting
. It discusses the kernel estimator and its bias–variance analysis as a direct theoretical point of comparison, then develops standard errors, confidence intervals, and simultaneous confidence bands (Introduction).Chen, Pan & Shen · University of Minnesota, 2026 preprint. Proxy-Adjusted Causal Discovery from Targeted Interventions discusses our ICML 2025 paper as prior work on nonparametric proxy-based causal testing. It places this line of research among approaches establishing the feasibility of flexible inference from proxy information under appropriate identification conditions.
Zuo et al. · Independent PatchMix adoption, CMC 2025. Federated Learning and Optimization for Few-Shot Image Classification incorporates PatchMix into federated few-shot learning and attributes gains to increased data diversity and more robust features. Its ablation study reports 1-shot accuracy increasing from 56.62% to 58.14% on MiniImageNet and 58.48% to 60.67% on TieredImageNet when PatchMix is included (Section 5.4.2, Table 2).
-
2012–2020Zero-shot learning
Zero-shot learning: from multimodal attributes to open-set recognition
To our knowledge, our ECCV 2014 / TPAMI 2015 work first identified the projection domain shift problem in zero-shot learning and introduced transductive multi-view embedding to address it. Independent research credits this work with identifying the problem.
This series spans early attribute-based video recognition and the USAA dataset, multi-label zero-shot learning, and vocabulary-informed recognition that connects supervised, zero-shot, and open-set learning.
Attribute learning for unseen categories · ECCV 2012 / TPAMI 2014
Attribute Learning for Understanding Unstructured Social Activity
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Shaogang Gong
Learning Multimodal Latent Attributes
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Shaogang Gong
Transductive multi-view learning · ECCV 2014 / TPAMI 2015
Transductive Multi-view Embedding for Zero-Shot Recognition and Annotation
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Zhenyong Fu, Shaogang Gong
Transductive Multi-view Zero-Shot Learning
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Shaogang Gong
Multi-label zero-shot recognition · BMVC 2014
Transductive Multi-label Zero-shot Learning
Yanwei Fu, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, Shaogang Gong
Vocabulary-informed learning · CVPR 2016 / TPAMI 2020
Semi-supervised Vocabulary-informed Learning
Yanwei Fu, Leonid Sigal
Vocabulary-informed Zero-shot and Open-set Learning
Yanwei Fu, Xiaomei Wang, Hanze Dong, Yu-Gang Jiang, Meng Wang, Xiangyang Xue, Leonid Sigal
Prospective students & visitors
For research inquiries, doctoral applications, or summer visits, please contact yanweifu@fudan.edu.cn. International students interested in summer internships are welcome to get in touch.
2026 年招生说明：本组仅在复旦大学大数据学院和上海创智学院招生。大数据学院科学硕士及直博生由学院统一招生，请先申请并获得学院 offer，再联系我；申请普博或暑期访问可提前联系。有兴趣研究统计稀疏性的本科生，也可联系孙鑫伟老师共同合作。
本组 2027年入学的普博博士名额已满。
U.S. patents
Granted U.S. patents from my research at Fudan University and Disney Research, spanning 3D vision, image restoration and visual recognition. The 11 grants below are grouped by invention.
Fudan University Original assignee
-
Network and system for pose and size estimation
Yanwei Fu, Haitao Lin, Xiangyang Xue
-
System and method for image inpainting
Yanwei Fu, Chenjie Cao
-
Network and system for neural pose transfer
Yanwei Fu, Xiangyang Xue, Yinda Zhang, Chao Wen, Haitao Lin, Jiashun Wang, Tianyun Zou
-
Network, system and method for image processing
Yugang Jiang, Yanwei Fu, Changmao Cheng, Xiangyang Xue
-
Network, system and method for multi-view 3D mesh generation via deformation
Yanwei Fu, Chao Wen, Yinda Zhang, Zhuwen Li
-
Network, system and method for 3D shape generation
Yugang Jiang, Yanwei Fu, Nanyang Wang, Yinda Zhang, Zhuwen Li
Disney Enterprises, Inc. Original assignee
-
Robust classification by pre-conditioned lasso and transductive diffusion component analysis
Yanwei Fu, Leonid Sigal
-
System and method of vocabulary-informed categorization of items of interest included within digital information
Yanwei Fu, Leonid Sigal
-
Systems and methods for identifying activities and/or events in media contents based on object data and scene data
Zuxuan Wu, Yanwei Fu, Leonid Sigal
Published U.S. patent application
Applicant: Tencent Technology (Shenzhen) Company Limited
-
Three-dimensional image generation
Shijie Zhang, Chengming Xu, Junwei Zhu, Xuelin Qian, Ying Tai, Yanwei Fu, Chengjie Wang
US20260245287A1 ↗ · Published