RTSP播放器实现回调RGB|YUV给视觉算法,然后二次编码推送到RTMP服务
引言
在本文中,我们将介绍如何基于大牛直播SDK构建一个功能强大的RTSP|RTMP播放器,该播放器利用自定义SDK解码视频、处理RGB帧,并将其推送到RTMP流中进行直播。这个解决方案非常适合需要在实时视频流中集成视觉算法的场景,在处理后将数据推送到RTMP服务器。我们将详细探讨播放器的架构、回调处理以及图像帧的操作过程。
核心组件概述
先看视频演示,左侧是Windows平台轻量级RTSP服务,采集毫秒计数器,然后编码打包,对外提供RTSP拉流的URL,右上角拉取原始的RTSP流,然后回调解码后的RGB|YUV数据,然后点击推送RTMP,实现RGB|YUV数据的二次编码和RTMP推送。可以看到,从原始数据播放回调,到右下角处理后的RTMP流,二次播放,整体延迟在毫秒级,非常低。
RTMP|RTSP播放器回调RGB数据进行算法分析和二次推流
-
RTSP/RTMP播放器架构:
-
该播放器接收RTSP流,将其解码为RGB帧,处理后将这些帧推送到RTMP流进行直播广播。
-
它利用自定义SDK来处理视频解码、帧处理和流推送。
-
-
关键概念:
-
RTSP(实时流协议):该协议用于控制流媒体服务器,广泛应用于实时视频流的传输。
-
RTMP(实时消息协议):RTMP用于将视频数据推送到直播服务器,确保低延迟广播。
-
RGB数据处理:播放器将视频帧解码为RGB格式(32位),然后传递给视觉算法进行处理,最后将处理后的数据推送到RTMP服务器。
-
代码讲解
进入系统后,先播放RTMP、或RTSP流,然后点RTMP推流,那么会模拟把播放器回调的RGB或YUV数据,投递到RTMP推送模块(右上方播放和转推)、右下方播放RTMP服务器二次处理后的RTMP流。
1. 初始化播放器SDK
CSmartPlayerDlg
类负责初始化播放器、设置事件回调并准备视频渲染窗口。SDK初始化通过player_api_
对象完成。
/* * SmartPlayDlg.cpp* Created by daniusdk.com* WeChat: xinsheng120*/
void CSmartPlayerDlg::OnBnClickedButtonPlay()
{if ( player_handle_ == NULL )return;CString btn_play_str;btn_play_.GetWindowTextW(btn_play_str);if ( btn_play_str == _T("播放") ){if ( !is_recording_ ){if ( !InitCommonSDKParam() ){AfxMessageBox(_T("设置参数错误!"));return;}}player_api_.SetVideoSizeCallBack(player_handle_, GetSafeHwnd(), SP_SDKVideoSizeHandle);bool is_support_d3d_render = false;NT_INT32 in_support_d3d_render = 0;if ( NT_ERC_OK == player_api_.IsSupportD3DRender(player_handle_,wrapper_render_wnd_.RenderWnd(), &in_support_d3d_render)){if ( 1 == in_support_d3d_render ){is_support_d3d_render = true;}}player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,this, SM_SDKVideoFrameHandleV2);if ( is_support_d3d_render ){is_gdi_render_ = false;// 支持d3d绘制的话,就用D3D绘制player_api_.SetRenderWindow(player_handle_, wrapper_render_wnd_.RenderWnd());player_api_.SetRenderScaleMode(player_handle_, btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);}else{is_gdi_render_ = true;// 不支持D3D就让播放器吐出数据来,用GDI绘制wrapper_render_wnd_.SetRenderScaleMode(btn_check_render_scale_mode_.GetCheck() == BST_CHECKED ? 1 : 0);player_api_.SetVideoFrameCallBack(player_handle_, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32,GetSafeHwnd(), SM_SDKVideoFrameHandle);}if ( BST_CHECKED == btn_check_hardware_decoder_.GetCheck() ){player_api_.SetH264HardwareDecoder(player_handle_, is_support_h264_hardware_decoder_?1:0, 0);player_api_.SetH265HardwareDecoder(player_handle_, is_support_h265_hardware_decoder_?1:0, 0);}else{player_api_.SetH264HardwareDecoder(player_handle_, 0, 0);player_api_.SetH265HardwareDecoder(player_handle_, 0, 0);}player_api_.SetOnlyDecodeVideoKeyFrame(player_handle_, BST_CHECKED == btn_check_only_decode_video_key_frame_.GetCheck() ? 1 : 0);player_api_.SetLowLatencyMode(player_handle_, BST_CHECKED == btn_check_low_latency_.GetCheck() ? 1 : 0);player_api_.SetFlipVertical(player_handle_, BST_CHECKED == btn_check_flip_vertical_.GetCheck() ? 1 :0 );player_api_.SetFlipHorizontal(player_handle_, BST_CHECKED == btn_check_flip_horizontal_.GetCheck() ? 1 : 0);player_api_.SetRotation(player_handle_, rotate_degrees_);player_api_.SetAudioVolume(player_handle_, slider_audio_volume_.GetPos());player_api_.SetUserDataCallBack(player_handle_, GetSafeHwnd(), NT_SP_SDKUserDataHandle);if (NT_ERC_OK != player_api_.StartPlay(player_handle_)){AfxMessageBox(_T("播放器失败!"));return;}btn_play_.SetWindowTextW(_T("停止"));is_playing_ = true;}else{StopPlayback();}
}void CSmartPlayerDlg::StopPlayback()
{if ( player_handle_ == NULL )return;is_gdi_render_ = false;btn_full_screen_.EnableWindow(FALSE);wrapper_render_wnd_.ClearVideoSize();wrapper_render_wnd_.SetPlayerHandle(NULL);width_ = 0; height_ = 0;player_api_.StopPlay(player_handle_);wrapper_render_wnd_.CleanRender();btn_play_.SetWindowTextW(_T("播放"));is_playing_ = false;if (!is_recording_){SetWindowText(base_title_);edit_duration_.SetWindowText(_T(""));edit_playback_pos_.SetWindowText(_T(""));btn_pause_.SetWindowText(_T("暂停"));edit_player_msg_.SetWindowText(_T(""));}
}
上述代码片段初始化了SmartPlayer SDK,这是解码RTSP流、处理视频和音频播放所必需的。
2. 设置事件回调
播放器SDK提供了多个事件回调函数,这些回调函数会在特定事件发生时触发,比如接收到新的视频帧或遇到缓冲事件。这些回调函数在初始化时进行设置。
例如,SetVideoFrameCallBack
函数用于定义在接收到新的视频帧时应该执行什么操作:
player_api_.SetVideoFrameCallBack(handle, NT_SP_E_VIDEO_FRAME_FORMAT_RGB32, this, &CSmartPlayerDlg::OnVideoFrameHandle);
OnVideoFrameHandle
函数会处理每个RGB帧,然后将其推送到RTMP流。
3. 处理视频帧回调
在OnVideoFrameHandle
函数中,我们通过首先检查帧的格式,然后将其数据复制到nt_rgb32_image
结构体中来处理RGB帧:
void CSmartPlayerDlg::OnVideoFrameHandle(NT_HANDLE handle, NT_UINT32 status,const NT_SP_VideoFrame* frame)
{if (nullptr == frame)return;std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);if (!is_pushing_)return;if (GetPushHandle() == nullptr)return;//NT_UINT64 ts = frame->timestamp_;//std::ostringstream ss;//ss << "OnVideoFrameHandle, ts: " << ts << "\r\n";//OutputDebugStringA(ss.str().c_str());NT_PB_Image image;memset(&image, 0, sizeof(image));image.width_ = frame->width_;image.height_ = frame->height_;// timestamp_ 目前不使用//image.timestamp_ = frame->timestamp_;if (NT_SP_E_VIDEO_FRAME_FORMAT_RGB32 == frame->format_){image.format_ = NT_PB_E_IMAGE_FORMAT_RGB32;image.plane_[0] = frame->plane0_;image.stride_[0] = frame->stride0_;image.plane_size_[0] = frame->stride0_ * frame->height_;}else if (NT_SP_E_VIDEO_FRAME_FROMAT_I420 == frame->format_){image.format_ = NT_PB_E_IMAGE_FORMAT_I420;image.plane_[0] = frame->plane0_;image.stride_[0] = frame->stride0_;image.plane_size_[0] = frame->stride0_ * frame->height_;image.plane_[1] = frame->plane1_;image.stride_[1] = frame->stride1_;image.plane_size_[1] = frame->stride1_ * ((frame->height_ + 1) / 2);image.plane_[2] = frame->plane2_;image.stride_[2] = frame->stride2_;image.plane_size_[2] = frame->stride2_ * ((frame->height_ + 1) / 2);}else{return;}int index_ = 0;push_api_.PostLayerImage(push_handle_, 0, index_, &image, 0, NULL);
}
该函数将帧数据转换为图像对象,可以进行后续处理或传递给可视算法,通过PostLayerImage()接口投递到RTMP推送模块。
4. 推送到RTMP
一旦RGB帧处理完成,我们需要将视频数据推送到RTMP服务器。通过使用Smart Publisher SDK的推送功能,我们实现了这一点:
/* SmartPlayerDlg.cpp* Created by daniusdk.com* WeChat: xinsheng120*/
void CSmartPlayerDlg::OnBnClickedButtonPush()
{// TODO: Add your control notification handler code hereCString btn_push_str;btn_push_.GetWindowTextW(btn_push_str);if (btn_push_str == _T("推送RTMP")){StartPush("rtmp://192.168.1.7:1935/hls/stream666");}else{StopPush();}
}bool CSmartPlayerDlg::StartPush(const std::string& url)
{if (is_pushing_)return false;if (url.empty())return false;if (!OpenPushHandle())return false;auto push_handle = GetPushHandle();ASSERT(push_handle != nullptr);if (publisher_handle_count_ < 1){SetCommonOptionToPublisherSDK();}if (NT_ERC_OK != push_api_.SetURL(push_handle, url.c_str(), NULL)){if (0 == publisher_handle_count_){push_api_.Close(push_handle);SetPushHandle(nullptr);}return false;}if (NT_ERC_OK != push_api_.StartPublisher(push_handle, NULL)){if (0 == publisher_handle_count_){push_api_.Close(push_handle);SetPushHandle(nullptr);}return false;}publisher_handle_count_++;btn_push_.SetWindowTextW(_T("停止推送"));is_pushing_ = true;return true;
}void CSmartPlayerDlg::StopPush()
{if (!is_pushing_)return;is_pushing_ = false;std::unique_lock<std::recursive_mutex> lock(push_handle_mutex_);if (nullptr == push_handle_)return;publisher_handle_count_--;push_api_.StopPublisher(push_handle_);if (0 == publisher_handle_count_){push_api_.Close(push_handle_);push_handle_ = nullptr;}btn_push_.SetWindowTextW(_T("推送RTMP"));
}
PushVideoFrame
方法将处理后的视频数据实时推送到RTMP服务器。
5. 处理错误与缓冲
SDK还提供了事件回调来处理错误和缓冲事件。例如,当播放器开始缓冲时,将触发以下回调:
extern "C" NT_VOID NT_CALLBACK NT_Push_SDKEventHandle(NT_HANDLE handle, NT_PVOID user_data,NT_UINT32 event_id,NT_INT64 param1,NT_INT64 param2,NT_UINT64 param3,NT_UINT64 param4,NT_PCSTR param5,NT_PCSTR param6,NT_PVOID param7)
{if (user_data == NULL)return;HWND hwnd = (HWND)user_data;if (NT_PB_E_EVENT_ID_CONNECTING == event_id|| NT_PB_E_EVENT_ID_CONNECTION_FAILED == event_id|| NT_PB_E_EVENT_ID_CONNECTED == event_id|| NT_PB_E_EVENT_ID_DISCONNECTED == event_id){if (hwnd != nullptr && ::IsWindow(hwnd)){auto event_info = new ConnectionEventInfo(handle, event_id, param5);::PostMessage(hwnd, WM_USER_PB_SDK_CONNECTION_INFO, (WPARAM)event_info, 0);}}if (NT_PB_E_EVENT_ID_RTSP_URL == event_id){if (hwnd != nullptr && ::IsWindow(hwnd)){if (param5 != nullptr){auto url_event_data = new NT_PushRtspURLEventData(handle, param5);::PostMessage(hwnd, WM_USER_SDK_PUSH_RTSP_URL_EVENT, (WPARAM)url_event_data, 0);}}}
}
该事件确保应用程序能在网络或流中断时做出相应处理。
结论
构建一个RTSP|RTMP播放器并进行RGB帧处理是实时媒体应用中的一项基本技能。通过使用SmartPlayer SDK,我们能够轻松地集成视频解码、帧处理和流推送在一个平台中。这种解决方案允许我们在视频流中进行自定义的视觉处理,并在处理后将其推送到RTMP服务器。大牛直播SDK提供了必要的构建块,包括帧处理、事件驱动的回调和视频渲染,使得它成为开发专业直播应用程序的强大工具。