HDMI-IN调试：双MIPI支持4K60方案
HDMI-IN的场景下，对图像有较高的要求的时候，一般需要支持HDMI2.0，也就是4K60的场景，在常见的HDMI转MIPI-CSI的方案中，尝尝受限于MIPI-CSI的规格。这篇文章介绍一种4K60的方案调试。
目录
1.概述
2.设计方案
2.1 总体框图
2.2 支持双MIPI转接芯片描述
2.3 RK3588 MIPI RX规格
2.4 RK3576 MIPI RX规格
3. 关键配置
3.1 内核版本需求
3.2 转接芯片驱动实现
3.3 DTS配置
3.3.1 RK3588配置说明
3.3.2 RK3576 配置说明
3.3.3 转接芯片配置说明
4.调试命令
4.1 查看链路
4.2 抓图命令
5. 常见问题
5.1 dts配置是否需要配置两个链路？
5.2 应用层是否需要重新适配？
6.总结
1.概述
RK3588/RK3576 平台支持HDMI转MIPI-CSI做HDMIIN功能，但受限于转接芯片的DPHY的规格，无法实现一个MIPI-DPHY接口实现接收4K60 的HDMI-IN图像。
以MIPI传输YUV422格式图像为例，需要接近10Gpbs的带宽，按4lane传输，每条lane则需要2.5Gbps的吞吐量，RK3588/RK3576 的DPHY接口满足该要求，但是市面很多转接芯片不满足该规格，如：RK628F、LT6911UXC等，但这类转接芯片内部有两个MIPI PORT，可以实现将4K60的图像分为两个MIPI PORT共计8lane传输。
这篇文章主要介绍外部MIPI TX按照两个MIPI PORT传输左右两半图像，RK主控端如何使用两个MIPI口接收拼接的方案。
注意的是RK平台该方案仅支持两个4lane的MIPI PORT分别传输左右两半的图像，不支持8lane的模式，即不支持按Byte或是Pixel分别在两个MIPI PORT之间交替传输的方式。
2.设计方案
2.1 总体框图
总体流程如下所示，4K60的HDMI图像给到RK628F等转接芯片接收，转接芯片将3840x2160的图像拆分成1920x2160左右两半图像，分别通过两个4lane的MIPI PORT传输，主控由两个MIPI分别接收左右两半的1920x2160@60的图像，由主控完成拼接，存储4K60图像。
2.2 支持双MIPI转接芯片描述
目前龙讯部分的转接芯片和RK628F支持双MIPI的模式传输，规格如下：
| 转接芯片 | 功能 | PHY 吞吐率 | 双MIPI模式 |
|---|---|---|---|
| RK628F | HDMI to CSI HDMI to DSI | DPHY CSI: 1.5Gbps/lanes DSI: 1.8Gbps/lanes | 4K60YUV422 4K60RGB888(DSI) |
| LT6911UXC | HDMI to CSI | DPHY: 2Gbps/lane | 4K60YUV422 4K60RGB888 |
| LT6911UXE | HDMI to CSI | DPHY:2.5Gbps/lane | 4K60RGB888 5K60YUV422 |
| LT7911D | DP/Type-c to CSI | DPHY:1.5Gbps/lane | 4K60YUV422 |
| LT7911UXC | DP/Type-c to CSI | DPHY:2.5Gbps/lane CPHY:2.5Gsps/trio(5.7Gbps) | 4K60RGB888 5K60YUV422 |
2.3 RK3588 MIPI RX规格
RK3588主控支持双MIPI接收高分辨率图像进行拼接，RK3588 MIPI RX规格如下
| 芯片型号 | MIPI RX数量 | 最大分辨率(PHY规格) | 支持格式 |
|---|---|---|---|
| RK3588s | 3x4lane | DPHY: 1xDPHY-v1.2: 2.5Gbps/lane x 4lane DCPHY: (1)DPHY 2xDPHY-v2.0: 2.5Gbps/lane x 4lane (2)CPHY 2xCPHY-v1.1: 5.7Gbps/lane x 3lane | YUV422/RGB888 |
| RK3588 | 4x4lane | DPHY: 2xDPHY-v1.2: 2.5Gbps/lane x 4lane DCPHY: (1)DPHY 2xDPHY-v2.0: 2.5Gbps/lane x 4lane (2)CPHY 2xCPHY-v1.1: 5.7Gbps/lane x 3lane | YUV422/RGB888 |
注：CPHY描述带宽吞吐率单位为sps，与bps的换算：1 Gsps = 2.28 Gbps。
2.4 RK3576 MIPI RX规格
RK3576主控支持双MIPI接收高分辨率图像进行拼接，RK3576 MIPI RX规格如下
| 芯片型号 | MIPI RX数量 | 最大分辨率(PHY规格) | 支持格式 |
|---|---|---|---|
| RK3576 | 3x4lane | 2 x CSI-DPHY: DPHY-v1.2: 2.5Gbps/lane x 4lane 1 x D/C PHY: (1)DPHY-v2.0: 2.5Gbps/lane x 4lane (2)CPHY-v1.1: 5.7Gbps/lane x 3lane | YUV422/RGB888 |
3. 关键配置
3.1 内核版本需求
cif驱动需要添加支持双MIPI接收的补丁，内核cif需要包括如下提交：
media: rockchip: vicap support combine two mipi to one devSigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: Iba3e83d0bc1433458d56c2542c3224ffee127b90phy: rockchip: csi2-dphy: logic node of mipi phy can control all hw of mipi phySigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: I30cc62cc1d28c4219e9e5c5ccd77fa9f589e63afinclude: rk-camera-module: support get capture infoSigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: Ic1f117afcb53b035086f6835deb0ccf2733ee972include: rkcif-config: support set multi csi infoSigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: I8509ed952b9554659c0238024a383e547620825bARCH: arm64: rk3588 separate the node of csi2 and hw, logical and physical nodes are separatedSigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: Ibb75cc466452aedff8f50d29331b191d2fbd922aARCH: arm64: rk3588 remove csi2_dcphy0/csi2_dcphy11. all logic node of mipi phy use csi2_dphy0,1,2,3...
2. all logic node of mipi phy can get all hw of mipi phy
3. the links between logic and hw is determined by upper level equipmenSigned-off-by: Zefa Chen <zefa.chen@rock-chips.com>
Change-Id: Icc0cb88c3294a119431ac24b0043e44e34b1b2923.2 转接芯片驱动实现
转接芯片驱动需要添加双MIPI模式的支持，可以参考LT6911UXC的修改，需要在转接芯片的驱动识别双MIPI的模式和单MIPI的模式，并配置相关信息，VICAP驱动通过IOCTL获取相应的配置信息。
LT6911UXC通过寄存器的lane数来识别是否是双MIPI模式，只有4K60的分辨率才会切换到双MIPI的模式，4K30及以下的分辨率是按照单MIPI的方式传输。LT6911相关的修改可参考如下，读取dts相关的信息，配置双MIPI模式对应的结构体rkmodule_multi_dev_info，VICAP驱动通过IOCTL获取该结构体，实现双MIPI的合成。
@@ -71,6 +75,7 @@ struct lt6911uxc {struct v4l2_dv_timings timings;struct v4l2_fwnode_bus_mipi_csi2 bus;struct v4l2_subdev sd;
+ struct rkmodule_multi_dev_info multi_dev_info;const char *len_name;const char *module_facing;const char *module_name;@@ -763,7 +789,9 @@ static int lt6911uxc_g_mbus_config(struct v4l2_subdev *sd, unsigned int pad,case 4:cfg->flags |= V4L2_MBUS_CSI2_4_LANE;break;
-
+ case 8:
+ cfg->flags |= V4L2_MBUS_CSI2_4_LANE;
+ break;default:},};@@ -950,7 +975,9 @@ static void lt6911uxc_get_module_inf(struct lt6911uxc *lt6911uxc,static long lt6911uxc_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg){struct lt6911uxc *lt6911uxc = to_state(sd);
+ struct device *dev = <6911uxc->i2c_client->dev;long ret = 0;
+ struct rkmodule_capture_info *capture_info;switch (cmd) {case RKMODULE_GET_MODULE_INFO:
@@ -959,6 +986,16 @@ static long lt6911uxc_ioctl(struct v4l2_subdev *sd, unsigned int cmd, void *arg)case RKMODULE_GET_HDMI_MODE:*(int *)arg = RKMODULE_HDMIIN_MODE;break;
+ case RKMODULE_GET_CAPTURE_MODE:
+ capture_info = (struct rkmodule_capture_info *)arg;
+ if (lt6911uxc->csi_lanes_in_use == 8) {
+ dev_info(dev, "8 lanes in use, set dual mipi mode\n");
+ capture_info->mode = RKMODULE_MULTI_DEV_COMBINE_ONE;
+ capture_info->multi_dev = lt6911uxc->multi_dev_info;
+ } else {
+ capture_info->mode = 0;
+ }
+ break;default:ret = -ENOIOCTLCMD;break;
@@ -975,6 +1012,7 @@ static long lt6911uxc_compat_ioctl32(struct v4l2_subdev *sd,struct rkmodule_inf *inf;long ret;int *seq;
+ struct rkmodule_capture_info *capture_info;switch (cmd) {case RKMODULE_GET_MODULE_INFO:
@@ -1007,6 +1045,21 @@ static long lt6911uxc_compat_ioctl32(struct v4l2_subdev *sd,}kfree(seq);break;
+ case RKMODULE_GET_CSI_DPHY_PARAM:
+ capture_info = kzalloc(sizeof(*capture_info), GFP_KERNEL);
+ if (!capture_info) {
+ ret = -ENOMEM;
+ return ret;
+ }
+
+ ret = lt6911uxc_ioctl(sd, cmd, capture_info);
+ if (!ret) {
+ ret = copy_to_user(up, capture_info, sizeof(*capture_info));
+ if (ret)
+ ret = -EFAULT;
+ }
+ kfree(capture_info);
+ break;default:ret = -ENOIOCTLCMD;break;@@ -1274,6 +1333,36 @@ static inline int lt6911uxc_parse_of(struct lt6911uxc *lt6911uxc)}#endif+static int lt6911uxc_get_multi_dev_info(struct lt6911uxc *lt6911uxc)
+{
+ struct device *dev = <6911uxc->i2c_client->dev;
+ struct device_node *node = dev->of_node;
+ struct device_node *multi_info_np;
+
+ multi_info_np = of_get_child_by_name(node, "multi-dev-info");
+ if (!multi_info_np) {
+ dev_info(dev, "failed to get multi dev info\n");
+ return -EINVAL;
+ }
+
+ of_property_read_u32(multi_info_np, "dev-idx-l",
+ <6911uxc->multi_dev_info.dev_idx[0]);
+ of_property_read_u32(multi_info_np, "dev-idx-r",
+ <6911uxc->multi_dev_info.dev_idx[1]);
+ of_property_read_u32(multi_info_np, "combine-idx",
+ <6911uxc->multi_dev_info.combine_idx[0]);
+ of_property_read_u32(multi_info_np, "pixel-offset",
+ <6911uxc->multi_dev_info.pixel_offset);
+ of_property_read_u32(multi_info_np, "dev-num",
+ <6911uxc->multi_dev_info.dev_num);
+ dev_info(dev,
+ "multi dev left: mipi%d, multi dev right: mipi%d, combile mipi%d, dev num: %d\n",
+ lt6911uxc->multi_dev_info.dev_idx[0], lt6911uxc->multi_dev_info.dev_idx[1],
+ lt6911uxc->multi_dev_info.combine_idx[0], lt6911uxc->multi_dev_info.dev_num);
+
+ return 0;
+}
+static int lt6911uxc_probe(struct i2c_client *client,const struct i2c_device_id *id){
@@ -1303,6 +1392,10 @@ static int lt6911uxc_probe(struct i2c_client *client,return err;}+ err = lt6911uxc_get_multi_dev_info(lt6911uxc);
+ if (err)
+ v4l2_info(sd, "get multi dev info failed, not use dual mipi mode\n");
+err = lt6911uxc_check_chip_id(lt6911uxc);if (err < 0)return err;3.3 DTS配置
3.3.1 RK3588配置说明
dts参考如下配置，旧版变化差异较大，主要有以下几点：
-
在pipeline上的配置，与单MIPI类似，pipeline的配置应配置成单MIPI场景能接收数据对应的MIPI口，例如4K60双MIPI场景是MIPI2+MIPI4进行拼接，4K30及以下是走单MIPI，由MIPI2完成传输，那么可参考这个dts配置，pipeline使用MIPI2的配置，在转接芯片节点添加multi-dev-info信息说明双MIPI的配置。
-
该版本需要将csi2_dphy0_hw~csi2_dphy1_hw，mipi_dcphy0~mipi_dcphy1，mipi0_csi2_hw~mipi5_csi2_hw等节点都okay起来。
-
该版本csi2_dphy节点不再区分phy的版本，与硬件接口不存在捆绑关系，默认根据mipi0_csi2~mipi5_csi2以及rkcif_mipi_lvds~rkcif_mipi_lvds5绑定硬件接口，对应dts和硬件接口关系如下，csi2_dphy的节点可以根据需要按0~5的顺序取用，与硬件接法没有捆绑。
sensor0 -> C/D PHY0(可选择csi2_dphy0~5) -> mipi0_csi2 -> rkcif_mipi_lvds
sensor1 -> C/D PHY1(可选择csi2_dphy0~5) -> mipi1_csi2 -> rkcif_mipi_lvds1
sensor2 -> CSI RX0 clk0 + 2lane[0, 1] or 4lane(可选择csi2_dphy0~5) -> mipi2_csi2 -> rkcif_mipi_lvds2
sensor3 -> CSI RX0 clk1 + 2lane[2, 3] (可选择csi2_dphy0~5) -> mipi3_csi2 -> rkcif_mipi_lvds3
sensor4 -> CSI RX1 clk0 + 2lane[0, 1] or 4lane(可选择csi2_dphy0~5) -> mipi4_csi2 -> rkcif_mipi_lvds4
sensor5 -> CSI RX1 clk1 + 2lane[2, 3] (可选择csi2_dphy0~5) -> mipi5_csi2 -> rkcif_mipi_lvds5
使用两路CSI RX0 和CSI RX1拼接的dts配置参考如下：
&csi2_dphy0_hw {status = "okay";
};&csi2_dphy1_hw {status = "okay";
};&csi2_dphy0 {status = "okay";ports {#address-cells = <1>;#size-cells = <0>;port@0 {reg = <0>;#address-cells = <1>;#size-cells = <0>;hdmi_mipi_in: endpoint@1 {reg = <1>;remote-endpoint = <<6911uxc_out0>;data-lanes = <1 2 3 4>;};};port@1 {reg = <1>;#address-cells = <1>;#size-cells = <0>;csidphy0_out: endpoint@0 {reg = <0>;remote-endpoint = <&mipi2_csi2_input>;};};};
};&i2c3 {status = "okay";lt6911uxc: lt6911uxc@2b {compatible = "lontium,lt6911uxc";status = "okay";reg = <0x2b>;clocks = <&ext_cam_clk>;clock-names = "xvclk";power-domains = <&power RK3588_PD_VI>;pinctrl-names = "default";pinctrl-0 = <<6911uxc_pin>;interrupt-parent = <&gpio1>;interrupts = <RK_PB3 IRQ_TYPE_LEVEL_LOW>;// reset-gpios = <&gpio1 RK_PB1 GPIO_ACTIVE_LOW>;// power-gpios = <&gpio1 RK_PA6 GPIO_ACTIVE_HIGH>;plugin-det-gpios = <&gpio1 RK_PB4 GPIO_ACTIVE_HIGH>;rockchip,camera-module-index = <0>;rockchip,camera-module-facing = "back";rockchip,camera-module-name = "HDMI-MIPI2";rockchip,camera-module-lens-name = "LT6911UXC";multi-dev-info {dev-idx-l = <4>;dev-idx-r = <2>;combine-idx = <2>;pixel-offset = <0>;dev-num = <2>;};port {lt6911uxc_out0: endpoint {remote-endpoint = <&hdmi_mipi_in>;data-lanes = <1 2 3 4>;};};};
};&mipi_dcphy0 {status = "okay";
};&mipi_dcphy1 {status = "okay";
};&mipi0_csi2_hw {status = "okay";
};&mipi1_csi2_hw {status = "okay";
};&mipi2_csi2_hw {status = "okay";
};&mipi3_csi2_hw {status = "okay";
};&mipi4_csi2_hw {status = "okay";
};&mipi5_csi2_hw {status = "okay";
};&mipi2_csi2 {status = "okay";ports {#address-cells = <1>;#size-cells = <0>;port@0 {reg = <0>;#address-cells = <1>;#size-cells = <0>;mipi2_csi2_input: endpoint@1 {reg = <1>;remote-endpoint = <&csidphy0_out>;};};port@1 {reg = <1>;#address-cells = <1>;#size-cells = <0>;mipi2_csi2_output: endpoint@0 {reg = <0>;remote-endpoint = <&cif_mipi_in2>;};};};
};&rkcif_mipi_lvds2 {status = "okay";port {cif_mipi_in2: endpoint {remote-endpoint = <&mipi2_csi2_output>;};};
};&rkcif {status = "okay";
};&rkcif_mmu {status = "okay";
};3.3.2 RK3576 配置说明
RK3576 配置与RK3588配置类似，只是在MIPI idx的配置上有所不同，具体说明如下：
sensor0 -> C/D PHY0(可选择csi2_dphy0~5) -> mipi0_csi2 -> rkcif_mipi_lvds
sensor1 -> CSI RX0 clk0 + 2lane[0, 1] or 4lane(可选择csi2_dphy0~5) -> mipi1_csi2 -> rkcif_mipi_lvds1
sensor2 -> CSI RX0 clk1 + 2lane[2, 3] (可选择csi2_dphy0~5) -> mipi2_csi2 -> rkcif_mipi_lvds2
sensor3 -> CSI RX1 clk0 + 2lane[0, 1] or 4lane(可选择csi2_dphy0~5) -> mipi3_csi2 -> rkcif_mipi_lvds3
sensor4 -> CSI RX1 clk1 + 2lane[2, 3] (可选择csi2_dphy0~5) -> mipi4_csi2 -> rkcif_mipi_lvds4
参考RK3576 EVB1公版搭配RK628F的配置，硬件上RK628F的两路CSI分别接入RK3576的DPHY0与DPHY1，MIPI的idx分别为1/3，dts配置如下所示：
&csi2_dphy0 {status = "okay";ports {#address-cells = <1>;#size-cells = <0>;port@0 {reg = <0>;#address-cells = <1>;#size-cells = <0>;hdmi_mipi_in: endpoint@1 {reg = <1>;remote-endpoint = <&hdmiin_out>;data-lanes = <1 2 3 4>;};};port@1 {reg = <1>;#address-cells = <1>;#size-cells = <0>;csidphy0_out: endpoint@0 {reg = <0>;remote-endpoint = <&mipi1_csi2_input>;};};};
};&csi2_dphy0_hw {status = "okay";
};&csi2_dphy1_hw {status = "okay";
};&i2c5 {status = "okay";pinctrl-0 = <&i2c5m3_xfer>;clock-frequency = <400000>;rk628_csi: rk628_csi@50 {reg = <0x50>;compatible = "rockchip,rk628-csi-v4l2";status = "okay";power-domains = <&power RK3576_PD_VI>;pinctrl-names = "default";pinctrl-0 = <&rk628_pin>;interrupt-parent = <&gpio3>;interrupts = <RK_PB0 IRQ_TYPE_LEVEL_HIGH>;enable-gpios = <&gpio3 RK_PD0 GPIO_ACTIVE_HIGH>;reset-gpios = <&gpio3 RK_PC5 GPIO_ACTIVE_HIGH>;plugin-det-gpios = <&gpio3 RK_PD5 GPIO_ACTIVE_LOW>;continues-clk = <1>;cec-enable;#sound-dai-cells = <0>;rockchip,camera-module-index = <0>;rockchip,camera-module-facing = "back";rockchip,camera-module-name = "HDMI-MIPI1";rockchip,camera-module-lens-name = "RK628-CSI";multi-dev-info {dev-idx-l = <1>;dev-idx-r = <3>;combine-idx = <1>;pixel-offset = <0>;dev-num = <2>;};port {hdmiin_out: endpoint {remote-endpoint = <&hdmi_mipi_in>;data-lanes = <1 2 3 4>;};};};
};&mipi1_csi2 {status = "okay";ports {#address-cells = <1>;#size-cells = <0>;port@0 {reg = <0>;#address-cells = <1>;#size-cells = <0>;mipi1_csi2_input: endpoint@1 {reg = <1>;remote-endpoint = <&csidphy0_out>;};};port@1 {reg = <1>;#address-cells = <1>;#size-cells = <0>;mipi1_csi2_output: endpoint@0 {reg = <0>;remote-endpoint = <&cif_mipi_in1>;};};};
};&rkcif {status = "okay";
};&rkcif_mipi_lvds1 {status = "okay";port {cif_mipi_in1: endpoint {remote-endpoint = <&mipi1_csi2_output>;};};
};&rkcif_mmu {status = "okay";
};3.3.3 转接芯片配置说明
转接芯片部分的配置多了multi-dev-info部分配置，该部分配置与驱动的rkmodule_multi_dev_info结构体对应；
multi-dev-info配置解释如下：
dev-idx-l：左半边图像对应的mipi接口编号
dev-idx-r：右半边图像对应的mipi接口编号
combine-idx：将左右两张图想合并到其中的一个idx，dts的链路以这个idx为准
pixel-offset：像素偏移，默认设置0
dev-num：多少个mipi设备进行拼接
对应驱动获取dts配置的配置：
static int lt6911uxc_get_multi_dev_info(struct lt6911uxc *lt6911uxc)
{struct device *dev = <6911uxc->i2c_client->dev;struct device_node *node = dev->of_node;struct device_node *multi_info_np;multi_info_np = of_get_child_by_name(node, "multi-dev-info");if (!multi_info_np) {dev_info(dev, "failed to get multi dev info\n");return -EINVAL;}of_property_read_u32(multi_info_np, "dev-idx-l",<6911uxc->multi_dev_info.dev_idx[0]);of_property_read_u32(multi_info_np, "dev-idx-r",<6911uxc->multi_dev_info.dev_idx[1]);of_property_read_u32(multi_info_np, "combine-idx",<6911uxc->multi_dev_info.combine_idx[0]);of_property_read_u32(multi_info_np, "pixel-offset",<6911uxc->multi_dev_info.pixel_offset);of_property_read_u32(multi_info_np, "dev-num",<6911uxc->multi_dev_info.dev_num);dev_info(dev,"multi dev left: mipi%d, multi dev right: mipi%d, combile mipi%d, dev num: %d\n",lt6911uxc->multi_dev_info.dev_idx[0], lt6911uxc->multi_dev_info.dev_idx[1],lt6911uxc->multi_dev_info.combine_idx[0], lt6911uxc->multi_dev_info.dev_num);return 0;
}4.调试命令
4.1 查看链路
使用media-ctl 工具可以查看pipeline的结构是否正常，pipeline错误，大概率是dts配置错误。
media-ctl -d /dev/mediaX -p //X=0123...4.2 抓图命令
开启数据流命令参考：
v4l2-ctl --verbose -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat='NV12' --stream-mmap=4抓图命令参考：
v4l2-ctl --verbose -d /dev/video0 --set-fmt-video=width=3840,height=2160,pixelformat='NV12' --stream-mmap=3 --stream-skip=4 --stream-to=/data/3840x2160p60_nv12-1.yuv --stream-count=5 --stream-poll5. 常见问题
5.1 dts配置是否需要配置两个链路？
硬件上使用两路MIPI数据流，但是dts不需要配置两个链路，仅需要在驱动中确认是否打开双MIPI的模式，以及增加rkmodule_multi_dev_info结构体参数的配置，平台的框架代码会根据该信息对两路数据流进行拼接，dts只需要配置单MIPI的场景下可以正常出数据流的链路即可。
5.2 应用层是否需要重新适配？
dts是配置一条链路，因此对用户空间而言，pipeline是一致的，具体的可以通过如下命令查看pipeline：
media-ctl -d /dev/mediaX -p //X=0123...6.总结
上述是以LT6911UXC为例介绍，现在RK628F/H使用也很多，也是使用这种方案，RK628F/H也可以使用DSI的RGB方案，让图像质量更高。
有问题欢迎咨询。