CVPR 2026 | A new five-modality fusion dataset featuring RGB, event, and thermal imaging…

CVPR 2026 | A brand‑new five‑modal information fusion dataset, including RGB, event, thermal imaging…

Click “Computer Vision Life”, then select “Star”

Robot AI insights delivered instantly

The following article is reprinted from Multimodal Information Fusion

1.

Paper Information

• Title: DSERT-RoLL: Robust Multi-Modal Perception for Diverse Driving Conditions with Stereo Event‑RGB‑Thermal Cameras, 4D Radar, and Dual‑LiDAR｜Robust Multi‑Modal Perception for Diverse Driving Conditions – Fusion of Stereo Event‑RGB‑Thermal Imaging Cameras, 4D Radar and Dual LiDAR
• Published: CVPR 2026
• Authors: Hoonhee Cho, Jae-Young Kang, Yuhwan Jeong, Yunseo Yang, Wonyoung Lee, Youngho Kim, Kuk-Jin Yoon;
Affiliation: South KoreaKorea Advanced Institute of Science and Technology (KAIST) Visual Intelligence Lab.
• Code: 
• https://jeongyh98.github.io/dsert-roll

3.

Paper Abstract

Current autonomous driving datasets mostly focus on RGB and LiDAR data, while new sensor data are scarce, making fair comparisons between different sensors difficult. DSERT‑RoLL includes stereo event cameras, RGB, thermal imaging cameras, 4D radar, and two LiDAR sensors, and provides 2D/3D bounding boxes, trajectory IDs, and vehicle odometry; the accompanying method adaptively fuses information from all sensors in a unified space to improve the stability of 3D detection under complex weather and lighting conditions.

4.

Key Innovations

DSERT‑RoLL benchmark’s different scenarios and multiple sensors

The dataset synchronously captures information from five sensors in the same road environment, covering six weather conditions and four lighting scenarios, enabling fair comparisons and addressing the shortage of event cameras, thermal imagers and 4D radar data.

Unified the detection and evaluation standards for 2D and 3D

The paper provides 2D/3D bounding boxes, trajectory IDs, and ego‑vehicle odometry, and establishes 2D and 3D object‑detection benchmarks that can be used to compare sensors within the same family as well as to study fusion problems between different sensors.

Adaptive Fusion Method for Heterogeneous Sensors

First, useLiDAR and 4D radar to generate 3D candidate boxes, then perform voxel center sampling based on RGB, event, and thermal imaging features, and fuse them using variable cross‑attention mechanism and confidence gating.

5.

Detailed Method Explanation

5.1 Definition of the Problem and the General Structure of the Whole System

The input includes long-range LiDAR, 4D radar point clouds as well as monocular RGB, event voxel grids, and thermal imaging maps, and the output is a 3D bounding box. This method first predicts initial boxes in the bird’s-eye view space, then incorporates three camera features into the 3D voxels, and after confidence fusion and region‑of‑interest pooling obtains a more precise target bounding box.

Figure 2: Complementary application scenarios of different sensors: LiDAR excels at geometric perception in clear weather and long distances, while 4D radar performs better in adverse conditions such as fog and snow; RGB, event, and thermal imaging are respectively suited for texture recognition, high-speed motion/high dynamic range, and low-light nighttime applications.

Figure 3: The states of the training and test sets regarding weather, illumination, and target categories; the three target types are cars, pedestrians, and bicycles.

Table 1: Comparison of autonomous driving object detection datasets, comparing aspects such as dataset size, adverse weather conditions, sensor types, 3D bounding boxes, trajectory IDs, and odometry annotations.

Table 2: Sensor suite information, including model, resolution or maximum range, field of view, and sampling rate.

Figure 4: Multimodal 3D detection framework, focusing on the data flow from the initial bounding box of the range sensor to camera feature voxel sampling, confidence fusion, and bounding box refinement.

5.2 Distance Sensor Working with Voxel Space

LiDAR and 4D radar after extracting features through a 3D voxel backbone, the non-empty voxel index sets are taken and united to form a unified sparse space.

, respectively represent the voxel feature sets of LiDAR and 4D radar. is the index of the -th non‑empty voxel; has feature dimensions; is the number of non‑empty voxels; denotes the union.

If a voxel is provided data by only one sensor, its original features remain unchanged; when both sensors occupy the same voxel, they are first concatenated and then linearly projected.

, are the non‑empty voxel sets of the two sensors;, denote the set difference and intersection, respectively; indicates concatenation along the channel dimension; maps the concatenated -dimensional feature to a -dimensional space.

Formula explanation: This approach does not require every voxel to contain both point clouds; instead, it merges geometric and Doppler information in overlapping regions while retaining only valid observations at single‑source locations.

5.3 Voxel Center Camera Feature Sampling

</section

For each non-empty voxel, first project it onto the RGB, thermal imaging, and event image planes, then learn offsets centered at that projection point, and sample.

denotes RGB, thermal imaging, or event modality;is the projection location of voxelin that image;represents the number of sampling points, set toin experiments;anddenote the learned offset and aggregation weight of theth sampling point, respectively;is the image feature of modality;is the aggregated result.

Formula explanation:The network finds reliable image cues near the network projection points, and uses voxels as queries, features as keys and values to perform deformable cross‑attention mechanism, thereby establishing correspondence between 2D semantics and 3D geometry.

5.4 Confidence Voxel Fusion and Box Refinement

After stacking the three-camera enhanced voxels along the modality axis, the framework generates camera-level gating using a global statistical method.

represents the three-camera enhanced voxel, denotes the three modalities: RGB, thermal imaging, and events, is the Sigmoid function, is the scalar gating value corresponding to each camera branch, represents element‑wise multiplication, and indicate the indices of voxels and channels, respectively.

Formula explanation:The network adjusts the proportion of information from the three cameras based on the existing data, concatenates them with the original 3D voxels, and then reduces them to a single dimension. For each initial box, the algorithm divides it intoregular sub‑voxels; in experiments we set, and after ROI pooling and multi‑level MLP output, finer bounding boxes are obtained.

5.5 Loss Function

Among them, is used to train the Region Proposal Network to generate initial 3D candidate boxes, is used to constrain confidence predictions, is used for bounding box regression, and are weights, both set to .

Formula explanation:The three losses are jointly optimized end‑to‑end for candidate boxes, fused confidence, and the final box positions, thereby achieving sensor selection and learning to detect the target.

❝

Brief description of this method:First, use LiDAR and 4D radar to construct a 3D “skeleton,” then let RGB, event, and thermal imaging fill in semantic information according to scene confidence, and finally refine the target boxes precisely.

6.

Experimental Results

6.1 Experimental Conditions

The dataset contains aboutframes of multimodal measurement data, and is split into training and test sets at aratio; according to the sum of the weather sub‑items in Figure 3, the training set hasframes and the test set hasframes. The model is trained end‑to‑end on NVIDIA Quadro RTX 8000 GPUs. During evaluation, the point‑cloud X‑axis is limited to the range meters, with a sampling count of and a box refinement grid of. The 3D detection follows the official Waymo Open Dataset standard, using mean average precision with an IoU threshold of as the primary metric, focusing only on the vehicle class. The paper does not specify details such as the number of training epochs, optimizer, learning rate, or input truncation size.

6.2 Quantitative Results

The 3D benchmark includes six weather conditions and four lighting conditions, and the completescheme ranks first under all ten conditions listed in Table 4, withvalues of: sunny, fog, light rain, heavy rain, light snow, heavy snow, normal illumination, low illumination, overexposure, HDR. Compared with other rows, theimprovement margin ranges fromto, with fog improving by percentage points and overexposure improving by percentage points.

Table 4 shows the 3D object detection results on DSERT-RoLL. After comparing stereo, 3D range sensors, and various multimodal fusions, the vehicle categories are ranked by values under different weather and lighting conditions.

6.3 Qualitative Results

The paper does not provide qualitative visual comparison figures of the 3D or 2D detection results, so no conclusions can be drawn about target prominence, texture, artifacts, or noise. Figures 1 and 2 are only used to illustrate the acquisition environment and the complementary relationship between sensors, and are not outputs of the method.

6.4 2D Object Detection Baseline

The paper uses left‑camera images to establish a 2D detection benchmark, using YOLOv10, RT-DETR and DEIM for single‑modal, and GM-DETR for multimodal. The evaluation metric is the vehicle class AP at . The three‑camera configuration achieves the highest scores in all 10 scenarios across the board; for example, under sunny, foggy, low‑light, overexposed, and HDR conditions the scores are , , , and , respectively; only in heavy snow does it drop to , indicating that pure camera‑based 2D detection remains very challenging under that condition.

Table 5 shows the comparison of 2D object detection performance of cameras on DSERT-RoLL under various weather and lighting conditions, with the AP values for the vehicle class using RGB, event, thermal imaging, and their combinations.

Ablation Study

Based onLiDAR single modality, adding RGB can improve semantic and boundary localization under moderate conditions; after adding4D radar, the increases from to, demonstratingweather robustness. The complete five-modality solution compared with theLiDAR baseline improves by and percentage points in heavy rain and heavy snow respectively, achieving the best ablation results in eight out of ten conditions; is higher than the full five-modality solution by and percentage points in fog and HDR respectively, indicating that adding more modalities does not always yield monotonically positive effects.

Table 3 shows the results of ablation experiments on various sensor combinations in DSERT-RoLL 3D detection, illustrating the roles of RGB, event, thermal imaging, 4D radar and LiDAR under different weather and lighting conditions.

6.6 Model Complexity

The paper does not mention the number of parameters,FLOPs, the time or speed required for inference per frame, etc., so a computational comparison of the full five-modality framework cannot be made; it is known that the hardware used for training is NVIDIA Quadro RTX 8000 GPU cards.

7.

Conclusion and Forecast

Summary

DSERT-RoLLplaces five types of sensors within the same collection and evaluation system, and uses a unified space, adjustable sampling, and confidence gating to assess their complementarity. Experimental results show that the combination of multiple modalities can mitigate the performance degradation of individual sensors under adverse weather or strong lighting conditions, laying a foundation for research on domain adaptation and domain generalization.

❝

DSERT-RoLL‘s most important contribution is placing five different sensors in the same real driving environment, under identical annotations and protocols, to enable fair comparison and collaborative fusion.

New Arrivals

Bringing robots truly into real-world scenarios: From “being able to move” to “being able to navigate” – the integrated EDU‑D360 × EDU‑BOX intelligent robot solution is here

“Super Dock” officially released, EDU-BOX moves toward real-world application, enabling robots to go from “can walk” to “can work” with faster R&D

Recommended Reading

Comprehensive Overview of EDU-D360! The embodied robot’s integrated perception cerebellum, fully open, truly synchronized, high computing power

Super popular! Robot dog navigation saves a whole year of schedule

How to solve the pain of embodied robot perception?

---

Detailed introduction! Ultra‑long‑range EDU‑SCAN 3D LiDAR scanner, born for an open ecosystem!

Breaking Release! Unlock Hundred-Meter-High Drone Mapping!

Breaking Release! Rebuild a Hundred-Meter Tall Building in 2 Minutes!

Major release! Empowering robot dog SLAM and reconstructing the 3D world!

SLAM Challenge! EDU-SCAN clearly reproduces a huge indoor electronic screen

SLAM Challenge! Reconstruct 10,000 in 9 Minutes

Underground Parking Garage

This research marvel smoothly reproduces the top SLAM: Fast‑LIVO2!

National Offline Product Trial for 3D Laser Scanners!

In-car street scanning, performance test