windows下tokenizers-cpp编译

github地址
一、rust环境配置
参考

二、编译
1、修改cmakelists.txt，支持x86和64编译(tokenizers_c库，原始版本windows下只支持64位)

修改顶层CMakeLists.txt文件（77行），支持x86编译
elseif (CMAKE_SYSTEM_NAME STREQUAL "Windows")#set(TOKENIZERS_CPP_CARGO_TARGET x86_64-pc-windows-msvc)if(CMAKE_SIZEOF_VOID_P EQUAL 4)set(TOKENIZERS_CPP_CARGO_TARGET i686-pc-windows-msvc)message(STATUS "Targeting 32-bit architecture")elseif(CMAKE_SIZEOF_VOID_P EQUAL 8)set(TOKENIZERS_CPP_CARGO_TARGET x86_64-pc-windows-msvc)message(STATUS "Targeting 64-bit architecture")else()message(FATAL_ERROR "Unsupported architecture")endif()
endif()

2、生成项目以及编译lib

debugcmake .. -G "Visual Studio 16 2019" -A Win32 -DCMAKE_BUILD_TYPE=Debugcmake --build . --config DebugPS: vs2019 win32
releasecmake .. -G "Visual Studio 16 2019" -A Win32 -DCMAKE_BUILD_TYPE=Releasecmake --build . --config Release

PS：请采用控制台编译，不然tokenizers_c.lib不可控，可能与预期编译debug/release不匹配

三、demo编写

1、依赖库
ws2_32.lib;Userenv.lib;Ntdll.lib;Bcrypt.lib;

2、模型下
模型