当前位置： 首页 > news >正文

使用seatunnel同步磐维数据库数据

news 2025/5/17 5:44:13

1.下载和安装seatunnel 

登录如下链接，下载seatunnel Apache SeaTunnelhttps://seatunnel.apache.org/download/

本次实验环境下载了2.3.8版本

解压
tar -xzvf "apache-seatunnel-${version}-bin.tar.gz"

2.下载连接器插件

Central Repository: org/apache/seatunnelhttps://repo.maven.apache.org/maven2/org/apache/seatunnel/

并将其移动至connectors/目录下

-rw-r--r-- 1 root root 11921607 11月  9 2023 seatunnel-transforms-v2-2.3.8.jar
-rw-r--r-- 1 root root   335483 11月  9 2023 connector-fake-2.3.8.jar
-rw-r--r-- 1 root root   196600 11月  9 2023 connector-console-2.3.8.jar
-rw-r--r-- 1 root root 30996454 5月  15 20:11 connector-cdc-opengauss-2.3.8.jar
-rw-r--r-- 1 root root  1025577 5月  16 14:23 connector-jdbc-2.3.8.jar

3.下载驱动

放到lib/目录下

[root@localhost lib]# ls -lrt
总用量 43236
-rw-r--r-- 1 root root 43187714 11月  9 2023 seatunnel-hadoop3-3.1.4-uber.jar
-rw-r--r-- 1 root root  1082113 5月  16 14:17 postgresql-42.6.2.jar
[root@localhost lib]# 

4.编辑配置文件

如下提供2个样例。

query = "select * from zyzaq.t2_qyq” 不带where条件代表全量同步

query = "select * from zyzaq.t2_qyq WHERE id > 2"，带where条件的支持增量同步。

zyzaq.t2_qyq这里的zyzaq，为schema。

单表样例：

[root@localhost apache-seatunnel-2.3.8]# cat panwei_to_panwei.config
env {parallelism = 4job.mode = "BATCH"
}source{Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"query = "select * from zyzaq.t2_qyq WHERE id > 2"}
}transform {}sink {jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_second table = "zyzaq.t2_qyq"}
}
[root@localhost apache-seatunnel-2.3.8]#

多表同步样例：

[root@localhost apache-seatunnel-2.3.8]# cat panwei_to_panwei_multi_table.config
env {parallelism = 4job.mode = "BATCH"
}source{Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"result_table_name="accounts_src"query = "select * from zyzaq.accounts"}Jdbc {url = "jdbc:postgresql://ip:17700/zyzaq"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"result_table_name="certificate_src"query = "select * from zyzaq.certificate"}
}transform {Sql {source_table_name = "accounts_src"result_table_name = "accounts_dst"query = "select * from accounts_src"}Sql {source_table_name = "certificate_src"result_table_name = "certificate_dst"query = "select * from certificate_src"}
}sink {jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_second table = "zyzaq.accounts"source_table_name = "accounts_dst"}jdbc {url = "jdbc:postgresql://ip:17700/zyzaq_second"driver = "org.postgresql.Driver"user = "zyzaq"password = "xx"generate_sink_sql = truedatabase =zyzaq_secondtable = "zyzaq.certificate"source_table_name = "certificate_dst"}
}
[root@localhost apache-seatunnel-2.3.8]#

5.执行同步命令

采用多表全量同步方式。需提前创建好目标表结构

./bin/seatunnel.sh --config ./panwei_to_panwei_multi_table.config -m local

6.同步日志，如下截取了部分同步的日志

2025-05-16 15:42:05,426 INFO  [a.s.e.s.s.s.DefaultSlotService] [hz.main.generic-operation.thread-13] - received slot release request, jobID: 975664414164254721, slot: SlotProfile{worker=[localhost]:5801, slotID=1, ownerJobID=975664414164254721, assigned=true, resourceProfile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=0}}, sequence='efefed6d-6f94-4d30-8a74-af2b5f4c631b'}
2025-05-16 15:42:05,427 INFO  [o.a.s.e.s.d.p.SubPlan         ] [seatunnel-coordinator-service-3] - Job SeaTunnel_Job (975664414164254721), Pipeline: [(1/2)] state process is stop
2025-05-16 15:42:05,427 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721), Pipeline: [(1/2)] future complete with state FINISHED
2025-05-16 15:42:05,428 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721) turned from state RUNNING to FINISHED.
2025-05-16 15:42:05,428 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job SeaTunnel_Job (975664414164254721) state process is stop
2025-05-16 15:42:05,449 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (975664414164254721) end with state FINISHED
2025-05-16 15:42:05,495 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - 
***********************************************Job Statistic Information
***********************************************
Start Time                : 2025-05-16 15:42:03
End Time                  : 2025-05-16 15:42:05
Total Time(s)             :                   2
Total Read Count          :                 171
Total Write Count         :                 171
Total Failed Count        :                   0
***********************************************2025-05-16 15:42:05,496 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2025-05-16 15:42:05,498 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-139022] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51399, qualifier=null, endpoint=[127.0.0.1]:51399, remoteUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2025-05-16 15:42:05,499 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-139022] [5.1] Removed connection to endpoint: [localhost]:5801:2895256a-a4d8-40a5-93ba-70dbb24c5d81, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:51399->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2025-05-16 15:42:05.485, lastWriteTime=2025-05-16 15:42:05.449, closedTime=2025-05-16 15:42:05.497, connected server version=5.1}
2025-05-16 15:42:05,499 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2025-05-16 15:42:05,500 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-5] - [localhost]:5801 [seatunnel-139022] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:51399, qualifier=null, endpoint=[127.0.0.1]:51399, remoteUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=2d5752f5-cf61-475d-9603-ddb5820a9167, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1747381323177, latest clientAttributes=lastStatisticsCollectionTime=1747381323210,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1747381323168,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=6417530880,os.freePhysicalMemorySize=881057792,os.freeSwapSpaceSize=1448751104,os.maxFileDescriptorCount=262144,os.openFileDescriptorCount=54,os.processCpuTime=7280000000,os.systemLoadAverage=0.84,os.totalPhysicalMemorySize=16474963968,os.totalSwapSpaceSize=3221221376,runtime.availableProcessors=8,runtime.freeMemory=278578192,runtime.maxMemory=477626368,runtime.totalMemory=335020032,runtime.uptime=3225,runtime.usedMemory=56441840, labels=[]}
2025-05-16 15:42:05,505 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-139022] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2025-05-16 15:42:05,506 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2025-05-16 15:42:05,506 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] [localhost]:5801 is SHUTTING_DOWN
2025-05-16 15:42:05,510 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-2] - [localhost]:5801 [seatunnel-139022] [5.1] Shutdown request of Member [localhost]:5801 - 2895256a-a4d8-40a5-93ba-70dbb24c5d81 this master is handled
2025-05-16 15:42:05,517 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Shutting down connection manager...
2025-05-16 15:42:05,519 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Shutting down node engine...
2025-05-16 15:42:05,531 INFO  [.c.c.DefaultClassLoaderService] [main] - close classloader service
2025-05-16 15:42:05,540 INFO  [o.a.s.e.s.EventService        ] [event-forwarder-0] - Event forward thread interrupted
2025-05-16 15:42:08,571 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Destroying node NodeExtension.
2025-05-16 15:42:08,571 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] Hazelcast Shutdown is completed in 3062 ms.
2025-05-16 15:42:08,572 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-139022] [5.1] [localhost]:5801 is SHUTDOWN
2025-05-16 15:42:08,572 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2025-05-16 15:42:08,572 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2025-05-16 15:42:13,510 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-1] - run shutdown hook because get close signal
[root@localhost apache-seatunnel-2.3.8]#

http://www.xdnf.cn/news/478585.html

相关文章：

  • 【全网首发】解决coze工作流批量上传excel数据文档数据重复的问题
  • 图像分割（0）初步认识
  • 高光谱遥感图像处理之数据分类的fcm算法
  • K8s CoreDNS 核心知识点总结
  • 捌拾伍- 量子傅里叶变换 (3)
  • 【目标检测】RT-DETR
  • 智脑进化：神经网络如何从单层感知机迈向深度学习新纪元
  • 动态规划（2）：问题建模与状态设计
  • 求职困境：开发、AI、运维、自动化
  • 序列dp常见思路总结
  • RabbitMQ 消息模式实战：从简单队列到复杂路由（四）
  • 卡顿检测与 Choreographer 原理
  • Java大师成长计划之第24天：Spring生态与微服务架构之分布式配置与API网关
  • window 显示驱动开发-使用有保证的协定 DMA 缓冲区模型
  • 论信息系统项目的范围管理
  • 后端框架(3)：Spring(2)
  • Gitee DevOps：中国企业数字化转型的“本土化加速器“
  • Ubuntu 更改 Nginx 版本
  • PCIE接收端检测机制分析
  • 源码：处理文件格式和字符集的相关代码（3-3）
  • Qt图表绘制（QtCharts）- 性能优化（13）
  • 关于Redisson分布式锁的用法
  • TRTC实时对话式AI解决方案，助力人机语音交互极致体验
  • Python 编程技巧 @ 玩转 For 循环
  • Linux `ps` 命令深度解析与高阶应用指南
  • 简单介绍C++中线性代数运算库Eigen
  • 【未完】【GNN笔记】EvolveGCN:Evolving Graph Convolutional Networks for Dynamics Graphs
  • sqli-labs靶场29-31关（http参数污染）
  • ECPF 简介
  • python爬虫实战训练