当前位置：首页 > news >正文

【机器人】复现 SG-Nav 具身导航 | 零样本对象导航的在线3D场景图提示

news 2025/5/16 13:31:43

SG-Nav提出了一种新的零样本物体导航框架，用三维场景图来表示观察到的场景。

并设计了一个分层的思路链提示，帮助LLM通过遍历节点和边，根据场景上下文推理目标位置。

本文分享SG-Nav复现和模型推理的过程～

下面是一个查找椅子示例：

1、创建Conda环境

首先创建一个Conda环境，名字为SG_Nav，python版本为3.9

进行SG_Nav环境

conda create -n SG_Nav python==3.9

然后下载代码，进入代码工程：https://github.com/bagh2178/SG-Nav

git clone https://github.com/bagh2178/SG-Nav.git
cd SG-Nav/

2、安装habitat模拟器

我需要安装habitat-sim==0.2.4 和habitat-lab

conda install habitat-sim==0.2.4 -c conda-forge -c aihabitat
cd habitat-lab
pip install -e .

等待安装完成～

用我们的存储库中的替换agent/agent.py已安装的habitat-sim 包tools/agent.py

cd ../
HABITAT_SIM_PATH=$(pip show habitat_sim | grep 'Location:' | awk '{print $2}')
cp tools/agent.py ${HABITAT_SIM_PATH}/habitat_sim/agent/

3、安装pytorch和pytorch3d

先安装 faiss 和pytorch<=1.9

conda install -c pytorch faiss-gpu=1.8.0
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 -f https://download.pytorch.org/whl/torch_stable.html

等待安装完成～

再安装其他依赖库和 pytorch3d

pip install -r requirements.txt
pip install "git+https://github.com/facebookresearch/pytorch3d.git"

安装成功打印信息：

  Building wheel for iopath (setup.py) ... doneCreated wheel for iopath: filename=iopath-0.1.10-py3-none-any.whl size=31599 sha256=fe751fc6548263cba1f98191a183c067ae0a992892481d81de1d2a3f6f61b12cStored in directory: /home/lgp/.cache/pip/wheels/c1/13/6d/441d8f2af76ee6d2a3e67eebb1d0c556fefcee0a8b32266a8e
Successfully built pytorch3d iopath
Installing collected packages: portalocker, iopath, pytorch3d
Successfully installed iopath-0.1.10 portalocker-3.1.1 pytorch3d-0.7.8

4、安装segment_anything

执行下面命令进行安装，并下载segment_anything的预训练模型权重

下载好的权重存放在data/models目录中～

cd segment_anything
pip install -e .
cd ../
mkdir -p data/models
wget -O data/models/sam_vit_h_4b8939.pth https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

打印信息：

(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ cd segment_anything
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/segment_anything$ pip install -e .
Obtaining file:///home/lgp/2025_project/SG-Nav/segment_anythingPreparing metadata (setup.py) ... done
Installing collected packages: segment_anythingDEPRECATION: Legacy editable install of segment_anything==1.0 from file:///home/lgp/2025_project/SG-Nav/segment_anything (setup.py develop) is deprecated. pip 25.3 will enforce this behaviour change. A possible replacement is to add a pyproject.toml or enable --use-pep517, and use setuptools >= 64. If the resulting installation is not behaving as expected, try using --config-settings editable_mode=compat. Please consult the setuptools documentation for more information. Discussion can be found at https://github.com/pypa/pip/issues/11457Running setup.py develop for segment_anything
Successfully installed segment_anything
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/segment_anything$ cd ../
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ mkdir -p data/models
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ wget -O data/models/sam_vit_h_4b8939.pth https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
--2025-05-13 21:38:09--  https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
正在连接 127.0.0.1:7890... 已连接。
已发出 Proxy 请求，正在等待回应... 200 OK
长度： 2564550879 (2.4G) [binary/octet-stream]
正在保存至: “data/models/sam_vit_h_4b8939.pth”data/models/sam_vit_h 100%[=========================>]   2.39G  71.1MB/s    用时 35s   2025-05-13 21:38:44 (70.0 MB/s) - 已保存 “data/models/sam_vit_h_4b8939.pth” [2564550879/2564550879])(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$

5、安装GroundingDINO

执行下面命令进行安装，并下载GroundingDINO的预训练模型权重

cd GroundingDINO
pip install --no-build-isolation -e .
cd ../
wget -O data/models/groundingdino_swint_ogc.pth https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth

6、安装GLIP

执行下面命令进行安装：

cd GLIP
python setup.py build develop --user
mkdir MODEL
cd MODEL
wget https://huggingface.co/GLIPModel/GLIP/resolve/main/glip_large_model.pth
cd ../../

打印信息：

(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ cd GLIP
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP$ python setup.py build develop --user
running build
running build_py
creating build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark
copying maskrcnn_benchmark/__init__.py -> build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark
creating build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark/engine
copying maskrcnn_benchmark/engine/trainer.py -> build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark/engine
copying maskrcnn_benchmark/engine/singlepath_trainer.py -> build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark/engine
....
running build_ext
copying build/lib.linux-x86_64-cpython-39/maskrcnn_benchmark/_C.cpython-39-x86_64-linux-gnu.so -> maskrcnn_benchmark
Creating /home/lgp/.local/lib/python3.9/site-packages/maskrcnn-benchmark.egg-link (link to .)
Adding maskrcnn-benchmark 0.0.0 to easy-install.pth fileInstalled /home/lgp/2025_project/SG-Nav/GLIP
Processing dependencies for maskrcnn-benchmark==0.0.0
Finished processing dependencies for maskrcnn-benchmark==0.0.0
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP$ mkdir MODEL
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP$ cd MODEL
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP/MODEL$ wget https://huggingface.co/GLIPModel/GLIP/resolve/main/glip_large_model.pth
--2025-05-13 21:44:07--  https://huggingface.co/GLIPModel/GLIP/resolve/main/glip_large_model.pth
正在连接 127.0.0.1:7890... 已连接。
已发出 Proxy 请求，正在等待回应... 302 Found正在连接 127.0.0.1:7890... 已连接。
已发出 Proxy 请求，正在等待回应... 200 OK
长度： 6896153761 (6.4G) [binary/octet-stream]
正在保存至: “glip_large_model.pth”glip_large_model.pth  100%[=========================>]   6.42G  69.8MB/s    用时 93s   2025-05-13 21:45:41 (70.7 MB/s) - 已保存 “glip_large_model.pth” [6896153761/6896153761])(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP/MODEL$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav/GLIP/MODEL$ cd ../../
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$ 
(SG_Nav) lgp@lgp-MS-7E07:~/2025_project/SG-Nav$

7、安装Ollama，配置LLM

执行下面命令进行安装：

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2-vision

安装成功打印信息：

8、下载MatterPort3D数据集

下载地址：https://cloud.tsinghua.edu.cn/f/03e0ca1430a344efa72b/?dl=1

数据集的结构概述如下：

MatterPort3D/
├── mp3d/
│ ├── 2azQ1b91cZZ/
│ │ └── 2azQ1b91cZZ.glb
│ ├── 8194nk5LbLH/
│ │ └── 8194nk5LbLH.glb
│ └── ...
└── objectnav/
└── mp3d/
└── v1/
└── val/
├── content/
│ ├── 2azQ1b91cZZ.json.gz
│ ├── 8194nk5LbLH.json.gz
│ └── ...
└── val.json.gz

对应的配置文件：configs/challenge_objectnav2021.local.rgbd.yaml

9、进行模型推理

执行下面命令：

python SG_Nav.py --visualize

运行结果示例：

分享完成～