这是一篇跟着官方教程走的深入文章，官方教程有6篇。现在这篇是 nuscenes_lidarseg_panoptic_tutorial.ipynb，也就是第二篇，链接：here
6篇教程分别如下：
- nuscenes_tutorial.ipynb
- nuscenes_lidarseg_panoptic_tutorial.ipynb
- nuimages_tutorial.ipynb
- can_bus_tutorial.ipynb
- map_expansion_tutorial.ipynb
- prediction_tutorial.ipynb
背景
lidarseg 数据集：lidarseg 数据集是用于激光雷达（LIDAR）点云分割任务的数据集。激光雷达是自动驾驶系统中常用的传感器之一，可以获取车辆周围环境的三维点云数据。lidarseg数据集提供了经过标注的LIDAR点云数据，其中每个点都被分配了特定的语义类别标签（如道路、建筑物、行人、车辆等）。这样的数据集可以用于训练和评估点云分割算法，使自动驾驶系统能够更好地理解和感知周围环境。
panoptic 数据集：panoptic 数据集是用于全景场景理解任务的数据集。全景场景理解旨在将场景中的每个像素分为两个组件：语义分割和实例分割。语义分割指将像素分配给不同的语义类别（如道路、行人、车辆等），而实例分割则是将每个物体实例分割成单独的掩码。panoptic 数据集提供了包含像素级标注的图像，其中每个像素都被分配了特定的语义类别和实例ID。这种数据集可以用于训练和评估全景场景理解算法，帮助自动驾驶系统对场景中的物体进行更准确和细粒度的理解。
准备工作
下载 lidarseg 和 panoptic 数据集
官方教程：配置 lidarseg
# !mkdir -p /data/sets/nuscenes # Make the directory to store the nuScenes dataset in.
# !wget https://www.nuscenes.org/data/v1.0-mini.tgz # Download the nuScenes mini split.
# !wget https://www.nuscenes.org/data/nuScenes-lidarseg-mini-v1.0.tar.bz2 # Download the nuScenes-lidarseg mini split.
# !tar -xf v1.0-mini.tgz -C /data/sets/nuscenes # Uncompress the nuScenes mini split.
# !tar -xf nuScenes-lidarseg-mini-v1.0.tar.bz2 -C /data/sets/nuscenes # Uncompress the nuScenes-lidarseg mini split.
# !pip install nuscenes-devkit &> /dev/null # Install nuScenes.官方教程：配置 panoptic
# !wget https://www.nuscenes.org/data/v1.0-mini.tgz # Download the nuScenes mini split.
# !wget https://www.nuscenes.org/data/nuScenes-panoptic-v1.0-mini.tar.gz # Download the Panoptic nuScenes mini split.
# !tar -xf v1.0-mini.tgz -C /data/sets/nuscenes # Uncompress the nuScenes mini split.
# !tar -xf nuScenes-panoptic-v1.0-mini.tar.gz -C /data/sets/nuscenes # Uncompress the Panoptic nuScenes mini split.我这边的教程：
第一篇文章的时候，已经下载了Full dataset mini 数据集，只有再下载 lidarseg 和 panoptic 数据集即可，下载好后，解压到目录/Users/lau/data_sets/v1.0-mini
注意：
- lidarseg 和 panoptic 包里的
v1.0-mini的内容也要放到之前 Full dataset mini 包里的v1.0-mini。 - lidarseg 和 panoptic 包里的
category.json是一样的，category.json里面的内容包含了 Full dataset mini 包里的category.json，可以方向覆盖。
最终目录如下所示：
ls -1 ./
lidarseg
maps
panoptic
samples
sweeps
v1.0-mini
cd v1.0-mini && ls -1 ./
attribute.json
calibrated_sensor.json
category.json
ego_pose.json
instance.json
lidarseg.json
log.json
map.json
panoptic.json
sample.json
sample_annotation.json
sample_data.json
scene.json
sensor.json
visibility.json阅读数据集
打开 jupyter，创建lidarseg-panoptic.ipynb，开撸。
准备
老规矩，安装依赖和加载数据
pip install nuscenes-devkit %matplotlib inline
from nuscenes import NuScenes
nusc = NuScenes(version='v1.0-mini', dataroot='/Users/lau/data_sets/v1.0-mini', verbose=True)
# 输出结果
======
Loading NuScenes tables for version v1.0-mini...
Loading nuScenes-lidarseg...
Loading nuScenes-panoptic...
32 category,
8 attribute,
4 visibility,
911 instance,
12 sensor,
120 calibrated_sensor,
31206 ego_pose,
8 log,
10 scene,
404 sample,
31206 sample_data,
18538 sample_annotation,
4 map,
404 lidarseg,
404 panoptic,
Done loading in 0.905 seconds.
======
Reverse indexing ...
Done reverse indexing in 0.1 seconds.
======跟第一篇对比，category 从23变为了32，多了404 lidarseg和 404 panoptic，说明数据集成功加载了 lidarseg 和 panoptic。
点统计
我们先看 lidarseg 数据集的类别以及每个类别所包含的点的数量。
类别将按照点的数量升序排列（因为下面设置了sort_by='count'）；我们也可以通过将sort_by设置为'name'或'index'来按类别名称或类别索引对类别进行排序。
# nuscenes-lidarseg
nusc.list_lidarseg_categories(sort_by='count')
# 输出结果
Calculating semantic point stats for nuScenes-lidarseg...
1 animal nbr_points= 0
7 human.pedestrian.stroller nbr_points= 0
8 human.pedestrian.wheelchair nbr_points= 0
19 vehicle.emergency.ambulance nbr_points= 0
20 vehicle.emergency.police nbr_points= 0
10 movable_object.debris nbr_points= 48
6 human.pedestrian.police_officer nbr_points= 64
3 human.pedestrian.child nbr_points= 230
4 human.pedestrian.construction_worker nbr_points= 1,412
14 vehicle.bicycle nbr_points= 1,463
11 movable_object.pushable_pullable nbr_points= 2,293
5 human.pedestrian.personal_mobility nbr_points= 4,096
13 static_object.bicycle_rack nbr_points= 4,476
12 movable_object.trafficcone nbr_points= 6,206
21 vehicle.motorcycle nbr_points= 6,713
0 noise nbr_points= 12,561
22 vehicle.trailer nbr_points= 12,787
29 static.other nbr_points= 16,710
16 vehicle.bus.rigid nbr_points= 29,694
18 vehicle.construction nbr_points= 39,300
15 vehicle.bus.bendy nbr_points= 40,536
2 human.pedestrian.adult nbr_points= 43,812
9 movable_object.barrier nbr_points= 55,298
25 flat.other nbr_points= 150,153
23 vehicle.truck nbr_points= 304,234
17 vehicle.car nbr_points= 521,237
27 flat.terrain nbr_points= 696,526
26 flat.sidewalk nbr_points= 746,905
30 static.vegetation nbr_points= 1,565,272
28 static.manmade nbr_points= 2,067,585
31 vehicle.ego nbr_points= 3,626,718
24 flat.driveable_surface nbr_points= 4,069,879
Calculated stats for 404 point clouds in 0.6 seconds, total 14026208 points.
=====使用list_lidarseg_categories，可以通过查看最左边的列来获得每个类名所属的索引。
也可以使用lidarseg_idx2name_mapping来获取索引所对应的类名。
nusc.lidarseg_idx2name_mapping
# 输出结果
{0: 'noise',
1: 'animal',
2: 'human.pedestrian.adult',
3: 'human.pedestrian.child',
4: 'human.pedestrian.construction_worker',
5: 'human.pedestrian.personal_mobility',
6: 'human.pedestrian.police_officer',
7: 'human.pedestrian.stroller',
8: 'human.pedestrian.wheelchair',
9: 'movable_object.barrier',
10: 'movable_object.debris',
11: 'movable_object.pushable_pullable',
12: 'movable_object.trafficcone',
13: 'static_object.bicycle_rack',
14: 'vehicle.bicycle',
15: 'vehicle.bus.bendy',
16: 'vehicle.bus.rigid',
17: 'vehicle.car',
18: 'vehicle.construction',
19: 'vehicle.emergency.ambulance',
20: 'vehicle.emergency.police',
21: 'vehicle.motorcycle',
22: 'vehicle.trailer',
23: 'vehicle.truck',
24: 'flat.driveable_surface',
25: 'flat.other',
26: 'flat.sidewalk',
27: 'flat.terrain',
28: 'static.manmade',
29: 'static.other',
30: 'static.vegetation',
31: 'vehicle.ego'}也可以使用lidarseg_name2idx_mapping来从类名属性中获得索引。
nusc.lidarseg_name2idx_mapping
# 输出结果
{'noise': 0,
'animal': 1,
'human.pedestrian.adult': 2,
'human.pedestrian.child': 3,
'human.pedestrian.construction_worker': 4,
'human.pedestrian.personal_mobility': 5,
'human.pedestrian.police_officer': 6,
'human.pedestrian.stroller': 7,
'human.pedestrian.wheelchair': 8,
'movable_object.barrier': 9,
'movable_object.debris': 10,
'movable_object.pushable_pullable': 11,
'movable_object.trafficcone': 12,
'static_object.bicycle_rack': 13,
'vehicle.bicycle': 14,
'vehicle.bus.bendy': 15,
'vehicle.bus.rigid': 16,
'vehicle.car': 17,
'vehicle.construction': 18,
'vehicle.emergency.ambulance': 19,
'vehicle.emergency.police': 20,
'vehicle.motorcycle': 21,
'vehicle.trailer': 22,
'vehicle.truck': 23,
'flat.driveable_surface': 24,
'flat.other': 25,
'flat.sidewalk': 26,
'flat.terrain': 27,
'static.manmade': 28,
'static.other': 29,
'static.vegetation': 30,
'vehicle.ego': 31}对于Panoptic nuScenes来说，它与 nuScenes-lidarseg 共享相同的成员变量lidarseg_idx2name_mapping和lidarseg_names2idx_mapping。同样，我们可以从Panoptic nuScenes数据集中查看每个语义类别的点的数量。唯一需要做的是添加gt_from='panoptic'参数。默认情况下，gt_from='lidarseg'。
# Panoptic nuScenes
nusc.list_lidarseg_categories(sort_by='count', gt_from='panoptic')
# 输出结果
Calculating semantic point stats for nuScenes-panoptic...
1 animal nbr_points= 0
7 human.pedestrian.stroller nbr_points= 0
8 human.pedestrian.wheelchair nbr_points= 0
19 vehicle.emergency.ambulance nbr_points= 0
20 vehicle.emergency.police nbr_points= 0
10 movable_object.debris nbr_points= 47
6 human.pedestrian.police_officer nbr_points= 56
3 human.pedestrian.child nbr_points= 213
11 movable_object.pushable_pullable nbr_points= 756
14 vehicle.bicycle nbr_points= 1,355
4 human.pedestrian.construction_worker nbr_points= 1,365
13 static_object.bicycle_rack nbr_points= 3,160
5 human.pedestrian.personal_mobility nbr_points= 3,424
12 movable_object.trafficcone nbr_points= 5,433
21 vehicle.motorcycle nbr_points= 6,479
22 vehicle.trailer nbr_points= 12,621
29 static.other nbr_points= 16,710
16 vehicle.bus.rigid nbr_points= 29,102
18 vehicle.construction nbr_points= 30,174
15 vehicle.bus.bendy nbr_points= 39,080
2 human.pedestrian.adult nbr_points= 41,629
9 movable_object.barrier nbr_points= 51,012
0 noise nbr_points= 59,608
25 flat.other nbr_points= 150,153
23 vehicle.truck nbr_points= 299,530
17 vehicle.car nbr_points= 501,416
27 flat.terrain nbr_points= 696,526
26 flat.sidewalk nbr_points= 746,905
30 static.vegetation nbr_points= 1,565,272
28 static.manmade nbr_points= 2,067,585
31 vehicle.ego nbr_points= 3,626,718
24 flat.driveable_surface nbr_points= 4,069,879
Calculated stats for 404 point clouds in 0.8 seconds, total 14026208 points.
=====我们注意到，在 lidarseg 和 panoptic 数据集中，某些类别的点的数量略有不同。这是因为在Panoptic nuScenes中，实例之间重叠的点被设置为噪声（类别0）。我们可以看到在Panoptic nuScenes中噪声类别的点数增加了，而总点数保持不变。
统计 panoptic 实例
实例统计数据是针对panoptic数据集的。我们提供了list_panoptic_instances()函数来实现这个目的。我们可以将sort_by设置为['count', 'index', 'name']中的一个。该函数将计算每帧的实例数量、总实例数（唯一对象ID）和实例状态（一个实例可能有多个状态，即轨迹）。它还计算每个类别的统计信息，包括实例跨越的帧数的平均值和标准差，以及每个实例的点数的平均值和标准差。
请注意，只有类别中才会有实例。关于点的统计信息可以参考点统计部分。
nusc.list_panoptic_instances(sort_by='count')
# 输出结果
Calculating instance stats for nuScenes-panoptic ...
Per-frame number of instances: 35±23
Per-category instance stats:
vehicle.car: 372 instances, each instance spans to 15±10 frames, with 92±288 points
human.pedestrian.adult: 212 instances, each instance spans to 19±10 frames, with 10±19 points
movable_object.trafficcone: 102 instances, each instance spans to 8±7 frames, with 7±18 points
movable_object.barrier: 89 instances, each instance spans to 20±12 frames, with 29±72 points
vehicle.truck: 25 instances, each instance spans to 21±13 frames, with 564±1770 points
vehicle.motorcycle: 17 instances, each instance spans to 20±9 frames, with 19±38 points
vehicle.bicycle: 15 instances, each instance spans to 13±8 frames, with 7±13 points
vehicle.bus.rigid: 13 instances, each instance spans to 24±11 frames, with 94±167 points
human.pedestrian.construction_worker: 8 instances, each instance spans to 19±9 frames, with 9±15 points
vehicle.construction: 6 instances, each instance spans to 27±11 frames, with 185±306 points
human.pedestrian.child: 4 instances, each instance spans to 11±9 frames, with 5±4 points
movable_object.pushable_pullable: 3 instances, each instance spans to 23±11 frames, with 11±16 points
static_object.bicycle_rack: 2 instances, each instance spans to 20±6 frames, with 81±136 points
vehicle.bus.bendy: 2 instances, each instance spans to 28±9 frames, with 698±1582 points
vehicle.trailer: 2 instances, each instance spans to 29±12 frames, with 218±124 points
human.pedestrian.personal_mobility: 1 instances, each instance spans to 25±0 frames, with 137±149 points
human.pedestrian.police_officer: 1 instances, each instance spans to 10±0 frames, with 6±4 points
movable_object.debris: 1 instances, each instance spans to 13±0 frames, with 4±2 points
animal: 0 instances, each instance spans to 0±0 frames, with 0±0 points
human.pedestrian.stroller: 0 instances, each instance spans to 0±0 frames, with 0±0 points
human.pedestrian.wheelchair: 0 instances, each instance spans to 0±0 frames, with 0±0 points
vehicle.emergency.ambulance: 0 instances, each instance spans to 0±0 frames, with 0±0 points
vehicle.emergency.police: 0 instances, each instance spans to 0±0 frames, with 0±0 points
Calculated stats for 404 point clouds in 1.5 seconds, total 875 instances, 14072 sample annotations.
=====统计 lidarseg/panoptic
获取样本
my_sample = nusc.sample[87]
my_sample
# 输出结果
{'token': '6dabc0fb1df045558f802246dd186b3f',
'timestamp': 1535489300046941,
'prev': 'b7f64f73e8a548488e6d85d9b0e13242',
'next': '82597244941b4e79aa3e1e9cc6386f8b',
'scene_token': '6f83169d067343658251f72e1dd17dbc',
'data': {'RADAR_FRONT': '020ecc01ab674ebabd3efbeb651800a8',
'RADAR_FRONT_LEFT': '8608ece15310417183a4ce928acc9d92',
'RADAR_FRONT_RIGHT': '7900c7855d7f41e99c047b08fac73f07',
'RADAR_BACK_LEFT': 'e7fa0148f1554603b19ce1c2f98dd629',
'RADAR_BACK_RIGHT': '7c1c8f579be9458fa5ac2f48d001ecf3',
'LIDAR_TOP': 'd219ffa9b4ce492c8b8059db9d8b3166',
'CAM_FRONT': '435543e1321b411ea5c930633e0883d9',
'CAM_FRONT_RIGHT': '12ed78f3bf934630a952f3259861877a',
'CAM_BACK_RIGHT': '3486d5847c244722a8ed7d943d8b9200',
'CAM_BACK': '365e64003ab743b3ae9ffc7ddb68872f',
'CAM_BACK_LEFT': '20a0bc8f862d456b8bf27be30dbbb8dc',
'CAM_FRONT_LEFT': '31ff0404d9d941d7870f9942180f67c7'},
'anns': ['cc2956dbf55b4e8cb65e61189266f14f',
'4843546cff85496894aa43d04362c435',
...
'0629ad31311f4a5ab1338c8ac929d109']}使用get_sample_lidarseg_stats来获取lidarseg的样本统计信息
# nuscenes-lidarseg
nusc.get_sample_lidarseg_stats(my_sample['token'], sort_by='count')
# 输出结果
===== Statistics for 6dabc0fb1df045558f802246dd186b3f =====
14 vehicle.bicycle n= 9
11 movable_object.pushable_pullable n= 11
0 noise n= 62
2 human.pedestrian.adult n= 71
16 vehicle.bus.rigid n= 105
9 movable_object.barrier n= 280
22 vehicle.trailer n= 302
30 static.vegetation n= 330
23 vehicle.truck n= 1,229
26 flat.sidewalk n= 1,310
25 flat.other n= 1,495
17 vehicle.car n= 3,291
28 static.manmade n= 4,650
24 flat.driveable_surface n= 9,884
31 vehicle.ego n= 11,723
===========================================================类似地，通过添加gt_from='panoptic'，我们可以使用相同的函数来使用panoptic数据集获得类别频率计数。正如在list_lidarseg_categories()中提到的，点计数可能与 lidarseg 稍有不同，这是因为在panoptic中，多个实例的重叠点被设置为noise。
# Panoptic nuScenes
nusc.get_sample_lidarseg_stats(my_sample['token'], sort_by='count', gt_from='panoptic')
# 输出结果
===== Statistics for 6dabc0fb1df045558f802246dd186b3f =====
11 movable_object.pushable_pullable n= 9
14 vehicle.bicycle n= 9
2 human.pedestrian.adult n= 62
16 vehicle.bus.rigid n= 103
0 noise n= 234
9 movable_object.barrier n= 255
22 vehicle.trailer n= 298
30 static.vegetation n= 330
23 vehicle.truck n= 1,149
26 flat.sidewalk n= 1,310
25 flat.other n= 1,495
17 vehicle.car n= 3,241
28 static.manmade n= 4,650
24 flat.driveable_surface n= 9,884
31 vehicle.ego n= 11,723
===========================================================使用 lidarseg 渲染
设show_lidarseg=True,来显示pointcloud的类标签
sample_data_token = my_sample['data']['LIDAR_TOP']
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_lidarseg=True)
设with_anns=True,可以看到车辆被标记出来了
sample_data_token = my_sample['data']['LIDAR_TOP']
nusc.render_sample_data(sample_data_token,
with_anns=True,
show_lidarseg=True)
但是，如果我们只想关注特定的类别怎么办？根据先前打印出的点云统计信息，假设我们只对卡车和挂车感兴趣。可以从统计数据中查看属于这些类别的类别索引，然后将这些索引的数组传递给filter_lidarseg_labels函数，如下所示：
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_lidarseg=True,
filter_lidarseg_labels=[22, 23])
如上图所示，现在只有属于卡车和拖车的点云中的点被过滤出来，以满足您的观看需求。此外，还可以使用show_lidarseg_legend显示一个图例，该图例指示每个类的颜色。
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_lidarseg=True,
show_lidarseg_legend=True)
使用 panoptic 渲染
与lidarseg类似，也使用相同的函数来呈现panoptic标签（全景标签）。参数的区别是show_panoptic=True。默认情况下，show_lidarseg和show_panoptic都被设置为False。如果两者都设置为True，即show_lidarseg=True, show_panoptic=True, lidarseg将会优先渲染。
sample_data_token = my_sample['data']['LIDAR_TOP']
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_lidarseg=False,
show_panoptic=True)
我们可以看到，来自同一类别的不同车辆实例将以唯一的颜色显示。类似地，我们可以使用filter_lidarseg_labels函数，并设置show_lidarseg_legend=True来显示特定的物体和物品类别的Panoptic标签，以及类别图例。请注意，这两个参数在lidarseg和panoptic数据集之间也是共享的。只有物品类别的图例将被显示，因为同一类别的物体实例具有不同的颜色。
# show trucks, trailers and drivable_surface
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_panoptic=True,
filter_lidarseg_labels=[22, 23, 24])
# show stuff category legends
nusc.render_sample_data(sample_data_token,
with_anns=False,
show_lidarseg=False,
show_lidarseg_legend=True,
show_panoptic=True)
进阶
渲染图像
如果我们想将点云与来自摄像头的对应图像叠加在一起，可以使用render_pointcloud_in_image函数，就像在原始的nuScenes devkit中那样，但是设置show_lidarseg=True（记得设置render_intensity=False）。类似于render_sample_data，可以使用filter_lidarseg_labels来筛选只看特定的类别。我们还可以使用show_lidarseg_legend来在渲染中显示图例。
# nuscenes-lidarseg
nusc.render_pointcloud_in_image(my_sample['token'],
pointsensor_channel='LIDAR_TOP',
camera_channel='CAM_BACK',
render_intensity=False,
show_lidarseg=True,
filter_lidarseg_labels=[22, 23, 24],
show_lidarseg_legend=True)
同样，该函数支持show_panoptic=True模式，将显示Panoptic标签而不是语义标签。只会显示物品类别的图例。
# Panoptic nuScenes
nusc.render_pointcloud_in_image(my_sample['token'],
pointsensor_channel='LIDAR_TOP',
camera_channel='CAM_BACK',
render_intensity=False,
show_lidarseg=False,
filter_lidarseg_labels=[17,22, 23, 24],
show_lidarseg_legend=True,
show_panoptic=True)
渲染样本
当然，就像在原始的nuScenes devkit中一样，我们可以使用render_sample函数一次渲染所有的传感器。在这个扩展的nuScenes devkit中，可以设置show_lidarseg=True来查看lidarseg标签。类似于上面的方法，可以使用filter_lidarseg_labels来只显示我们希望看到的类别。
# nuscenes-lidarseg
nusc.render_sample(my_sample['token'],
show_lidarseg=True,
filter_lidarseg_labels=[22, 23])
显示panoptic标签，只需设置show_panoptic=True
# Panoptic nuScenes
nusc.render_sample(my_sample['token'],
show_lidarseg=False,
filter_lidarseg_labels=[17, 23, 24],
show_panoptic=True)
渲染某传感器
我们也可以使用我们选择的相机的lidarseg标签来渲染整个场景(filter_lidarseg_labels参数也可以在这里使用)。让我们先选一个场景:
my_scene = nusc.scene[0]
my_scene
# 输出结果
{'token': 'cc8c0bf57f984915a77078b10eb33198',
'log_token': '7e25a2c8ea1f41c5b0da1e69ecfa71a2',
'nbr_samples': 39,
'first_sample_token': 'ca9a282c9e77460f8360f564131a8af5',
'last_sample_token': 'ed5fc18c31904f96a8f0dbb99ff069c0',
'name': 'scene-0061',
'description': 'Parked truck, construction, intersection, turn left, following a van'}然后我们将scene token传递给render_scene_channel_lidarseg，这里设置了filter_lidarseg_labels=[18, 28]，表示我们只对建筑车辆和人造物体感兴趣(在这里，我们设置verbose=True来生成一个窗口，可以随机显示帧)。
另外，我们可以使用dpi（调整激光点的大小）和imsize（调整渲染图像的大小）来调整渲染效果。
# nuscenes-lidarseg
import os
nusc.render_scene_channel_lidarseg(my_scene['token'],
'CAM_FRONT',
filter_lidarseg_labels=[18, 28],
verbose=True,
dpi=100,
imsize=(1280, 720))添加show_panoptic=True，这个函数也适用于panoptic标签
# nuscenes-panoptic
import os
nusc.render_scene_channel_lidarseg(my_scene['token'],
'CAM_BACK',
filter_lidarseg_labels=[18, 24, 28],
verbose=True,
dpi=100,
imsize=(1280, 720),
show_panoptic=True)要保存渲染结果，我们可以通过out_folder参数将路径传递给要保存图像的文件夹，并通过render_mode参数选择video或image。
# nuscenes-lidarseg
nusc.render_scene_channel_lidarseg(my_scene['token'],
'CAM_BACK',
filter_lidarseg_labels=[18, 28],
verbose=True,
dpi=100,
imsize=(1280, 720),
render_mode='video',
out_folder=os.path.expanduser('video_image'))渲染所有传感器
我们还可以一次性将所有相机的整个场景渲染为带有lidarseg标签的视频。假设在这种情况下，我们对属于可行驶路面和汽车的点感兴趣。
# nuscenes-lidarseg
nusc.render_scene_lidarseg(my_scene['token'],
filter_lidarseg_labels=[17, 24],
verbose=True,
dpi=100)
# Panoptic nuScenes
nusc.render_scene_lidarseg(my_scene['token'],
filter_lidarseg_labels=[17, 24],
verbose=True,
dpi=100,
show_panoptic=True)
渲染 LiDAR 分割预测
在上述所有函数中，已渲染的LiDAR点云的标签是地面真实情况。如果已经训练了一个模型来分割LiDAR点云，并在nuScenes-lidarseg数据集上运行了该模型，也可以使用nuScenes-lidarseg来可视化模型的预测结果！
import os
my_sample = nusc.sample[87]
sample_data_token = my_sample['data']['LIDAR_TOP']
my_predictions_bin_file = os.path.join('/Users/lau/data_sets/v1.0-mini/lidarseg/v1.0-mini', sample_data_token + '_lidarseg.bin')
nusc.render_pointcloud_in_image(my_sample['token'],
pointsensor_channel='LIDAR_TOP',
camera_channel='CAM_BACK',
render_intensity=False,
show_lidarseg=True,
filter_lidarseg_labels=[22, 23],
show_lidarseg_legend=True,
lidarseg_preds_bin_path=my_predictions_bin_file)
对于那些渲染整个场景的函数，我们需要通过lidarseg_preds_folder参数传递包含场景中每个样本的.bin文件的文件夹路径：
render_scene_channel_lidarsegrender_scene_lidarseg
my_scene = nusc.scene[0]
my_folder_of_predictions = '/Users/lau/data_sets/v1.0-mini/lidarseg/v1.0-mini'
nusc.render_scene_channel_lidarseg(my_scene['token'],
'CAM_BACK',
filter_lidarseg_labels=[17, 24],
verbose=True,
imsize=(1280, 720),
lidarseg_preds_folder=my_folder_of_predictions)渲染 LiDAR 全景预测
类似地，我们也可以渲染Panoptic预测结果
import os
my_sample = nusc.sample[87]
sample_data_token = my_sample['data']['LIDAR_TOP']
my_predictions_bin_file = os.path.join('/Users/lau/data_sets/v1.0-mini/panoptic/v1.0-mini', sample_data_token + '_panoptic.npz')
nusc.render_pointcloud_in_image(my_sample['token'],
pointsensor_channel='LIDAR_TOP',
camera_channel='CAM_BACK',
render_intensity=False,
show_lidarseg=False,
filter_lidarseg_labels=[17,22, 23, 24],
show_lidarseg_legend=True,
lidarseg_preds_bin_path=my_predictions_bin_file,
show_panoptic=True)
对于那些渲染整个场景的函数，我们需要通过lidarseg_preds_folder参数传递包含场景中每个样本的.npz文件的文件夹路径：
render_scene_channel_lidarsegrender_scene_lidarseg
my_scene = nusc.scene[0]
my_folder_of_predictions = '/Users/lau/data_sets/v1.0-mini/panoptic/v1.0-mini'
nusc.render_scene_channel_lidarseg(my_scene['token'],
'CAM_BACK',
filter_lidarseg_labels=[9, 18, 24, 28],
verbose=True,
imsize=(1280, 720),
lidarseg_preds_folder=my_folder_of_predictions,
show_panoptic=True)参考
- https://colab.research.google.com/github/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/nuscenes_lidarseg_panoptic_tutorial.ipynb
- https://github.com/nutonomy/nuscenes-devkit/blob/master/docs/instructions_lidarseg.md
- https://zhuanlan.zhihu.com/p/550677537
本文由 Chakhsu Lau 创作，采用 知识共享署名4.0 国际许可协议进行许可。
本站文章除注明转载/出处外，均为本站原创或翻译，转载前请务必署名。