当前位置：首页 > ops >正文

centos7挂载iscis存储操作记录

ops 2025/9/1 9:06:06

前言

自己搞测试，虚拟机磁盘空间不够，弄了个nas，本来是用windows连了个iscis存储，然后虚拟机整个目录直接放到iscis存储上，发现io效率极低，因为虚拟机文件变更一点都要整个文件通过网络传输到nas上。
所以换了个方案，虚拟机还是建在本地盘上，然后在虚拟机内部挂载iscis存储，减少了很多网络开销

发现设备

[root@localhost ~]# iscsiadm -m discovery -t st -p 192.168.1.172
192.168.1.172:3260,1 iqn.tnas.terramaster.23115210311
169.254.253.117:3260,1 iqn.tnas.terramaster.23115210311
172.17.0.1:3260,1 iqn.tnas.terramaster.23115210311
192.168.1.172:3260,1 iqn.tnas.terramaster.23118144203
169.254.253.117:3260,1 iqn.tnas.terramaster.23118144203
172.17.0.1:3260,1 iqn.tnas.terramaster.23118144203
192.168.1.172:3260,1 iqn.tnas.terramaster.23325185247
169.254.253.117:3260,1 iqn.tnas.terramaster.23325185247
172.17.0.1:3260,1 iqn.tnas.terramaster.23325185247
192.168.1.172:3260,1 iqn.tnas.terramaster.23325185428
169.254.253.117:3260,1 iqn.tnas.terramaster.23325185428
172.17.0.1:3260,1 iqn.tnas.terramaster.23325185428

这里有4个设备，因为我有4个Target，这里我用第3个

创建登录信息

[root@localhost ~]# iscsiadm -m node -T iqn.tnas.terramaster.23325185247 -p 192.168.1.172:3260 -o update --name node.session.auth.authmethod --value=CHAP
[root@localhost ~]# iscsiadm -m node -T iqn.tnas.terramaster.23325185247 -p 192.168.1.172:3260 -o update --name node.session.auth.username --value=xxxxxxxxxx
[root@localhost ~]# iscsiadm -m node -T iqn.tnas.terramaster.23325185247 -p 192.168.1.172:3260 -o update --name node.session.auth.password --value='xxxxxxxxxx'

三条命令分别是验证方法、用户名、密码（如果服务端没开启认证，可以不用配置这个）

[root@localhost ~]# iscsiadm -d2 -m node -T iqn.tnas.terramaster.23325185247 -p 192.168.1.172:3260 --login
iscsiadm: Max file limits 1024 4096
iscsiadm: default: Creating session 1/1
Logging in to [iface: default, target: iqn.tnas.terramaster.23325185247, portal: 192.168.1.172,3260] (multiple)
Login to [iface: default, target: iqn.tnas.terramaster.23325185247, portal: 192.168.1.172,3260] successful.

将登录命令写入开机自启

echo  “iscsiadm -d2 -m node -T iqn.tnas.terramaster.23325185247 -p 192.168.1.172:3260 --login” >> /etc/rc.local

查看iscis连接

[root@localhost ~]# iscsiadm -m session
tcp: [1] 192.168.1.172:3260,1 iqn.tnas.terramaster.23325185247 (non-flash)

查看磁盘信息

[root@localhost ~]# fdisk -lDisk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00099007Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200    41943039    19921920   8e  Linux LVMDisk /dev/mapper/centos-root: 18.2 GB, 18249416704 bytes, 35643392 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytesDisk /dev/sdb: 214.7 GB, 214748364800 bytes, 419430400 sectors  
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 524288 bytes

最后这个214.7GB的就是刚刚连接上的

磁盘分区

[root@localhost ~]# parted /dev/sdb
GNU Parted 3.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart primary 0% 100%
(parted) print
Model: SCST_FIO l_1679741660 (scsi)
Disk /dev/sdb: 215GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:Number  Start   End    Size   File system  Name     Flags1      1049kB  215GB  215GB               primary(parted) quit
Information: You may need to update /etc/fstab.

(parted) 开头的4行是手动输入的，分别表示：

(parted) mklabel gpt ## 设置卷标为 gpt
(parted) mkpart primary 0% 100% ## 设置主分区从0% 到 100%，即全部分完
(parted) print ## 打印信息
(parted) quit ## 退出

格式化

[root@localhost ~]# mkfs.ext4 -T largefile /dev/sdb1
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=128 blocks
204800 inodes, 52428288 blocks
2621414 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2199912448
1600 block groups
32768 blocks per group, 32768 fragments per group
128 inodes per group
Superblock backups stored on blocks:32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,4096000, 7962624, 11239424, 20480000, 23887872Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

挂载

mount /dev/sdb1 /opt/

我一般喜欢在opt下操作，当前opt目录为空，把这个目录就挂载上去了

查看挂载信息

[root@localhost /]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root   17G  1.5G   16G   9% /
/dev/sda1               1014M  151M  864M  15% /boot
tmpfs                    378M     0  378M   0% /run/user/0
/dev/sdb1                200G   61M  190G   1% /opt

可以看到opt有200G了

设置自动挂载

[root@localhost /]# blkid /dev/sdb1
/dev/sdb1: UUID="e3a15bc0-1416-4e20-ad2f-fe19ef20a1a8" TYPE="ext4" PARTLABEL="primary" PARTUUID="c6fb183c-6b6b-4ff8-bb33-fd2e37da3f1a"

找到UUID，然后写入/etc/fstab 中

echo "UUID=e3a15bc0-1416-4e20-ad2f-fe19ef20a1a8 /opt ext4 defaults,_netdev 0 0" >> /etc/fstab

重启服务器(为了验证是否正常)

reboot

启动后再次检查

[root@localhost ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G   12M  1.9G   1% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/centos-root   17G  1.5G   16G   9% /
/dev/sda1               1014M  151M  864M  15% /boot
tmpfs                    378M     0  378M   0% /run/user/0
/dev/sdb1                200G   61M  190G   1% /opt

踩坑与填坑

1.启动花的时间变长了很多

检查系统启动日志

journalctl -b

发现大量的iscis连接失败

Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: conn 0 login rejected: target error (03/02)
Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR
Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: conn 0 login rejected: target error (03/02)
Mar 25 08:55:43 localhost.localdomain kernel:  connection1:0: detected conn error (1020)
Mar 25 08:55:43 localhost.localdomain kernel:  connection2:0: detected conn error (1020)
Mar 25 08:55:43 localhost.localdomain kernel:  connection3:0: detected conn error (1020)
Mar 25 08:55:43 localhost.localdomain kernel:  connection4:0: detected conn error (1020)
Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: conn 0 login rejected: target error (03/02)
Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: conn 0 login rejected: target error (03/02)
Mar 25 08:55:43 localhost.localdomain iscsid[1041]: iscsid: Kernel reported iSCSI connection 4:0 error (1020 - ISCSI_ERR
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 169.254.253.117:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 172.17.0.1:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 169.254.253.117:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 172.17.0.1:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 169.254.253.117:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 172.17.0.1:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 169.254.253.117:3260 failed (No route to host)
Mar 25 08:55:44 localhost.localdomain iscsid[1041]: iscsid: connect to 172.17.0.1:3260 failed (No route to host)

原因是我的iscis设备是多ip，启动的时候它尝试每个都去连接，但实际上有几个ip不在这个网段内，所以连不上，然后反复重试，直到超限。

解决方案为

[root@localhost ~]# cd /var/lib/iscsi/nodes
[root@localhost nodes]# ll
total 0
drw------- 5 root root 91 Mar 25 08:49 iqn.tnas.terramaster.23115210311
drw------- 5 root root 91 Mar 25 08:49 iqn.tnas.terramaster.23118144203
drw------- 5 root root 91 Mar 25 08:49 iqn.tnas.terramaster.23325185247
drw------- 5 root root 91 Mar 25 08:49 iqn.tnas.terramaster.23325185428
[root@localhost nodes]#

把这台机器没有连的删掉

rm -rf iqn.tnas.terramaster.23115210311
rm -rf iqn.tnas.terramaster.23118144203
rm -rf iqn.tnas.terramaster.23325185428

然后进入连接的这个目录

[root@localhost nodes]# cd iqn.tnas.terramaster.23325185247
[root@localhost iqn.tnas.terramaster.23325185428]# ll
total 0
drw------- 2 root root 21 Mar 25 08:49 169.254.253.117,3260,1
drw------- 2 root root 21 Mar 25 08:49 172.17.0.1,3260,1
drw------- 2 root root 21 Mar 25 08:49 192.168.1.172,3260,1
[root@localhost iqn.tnas.terramaster.23325185428]#

删掉自己没连的ip目录

rm -rf 172.17.0.1,3260,1
rm -rf 169.254.253.117,3260,1

重启验证,速度恢复正常

2.磁盘空间占用才20%，就提示已满

原因是前面格式化的时候用了-T largefile, 这个磁盘大小默认出来只有 204800个inodes,但编译openGauss,有很多小文件,弄两个openGauss就满了，所以只能umount,重新格式化,再umount。如果有重要文件，还得备份之后再格式化

umount /dev/sdb1
mkfs.ext4 -i 409600 /dev/sdb1
mount /dev/sdb1 /opt/

最后记得改 /etc/fstab 文件，因为重新格式化后，UUID变了

参考文章

https://www.cnblogs.com/dockers/p/11590518.html
https://blog.csdn.net/qq_22787559/article/details/122022517
https://cloud.tencent.com/developer/article/1874542