当前位置： 首页 > news >正文

欧拉操作系统下安装hadoop集群

news 2025/5/31 16:35:07

背景：欧拉操作系统下安装CDH集群的时候，需要安装python2.7.5，但是本身欧拉系统对python2的支持可能没有那么好，所以考虑搭建原生的hadoop集群。

基础环境如下

组件名称组件版本
欧拉VERSION=“22.03 (LTS-SP4)”
jdkopenjdk version “1.8.0_44”
mysql8.0.42
hadoop
hive

一、jdk的安装

openjdk的下载地址

https://jdk.java.net/java-se-ri/8-MR6

二、mysql的安装

mysql安装包的下载地址
https://dev.mysql.com/downloads/mysql/

三、hadoop的安装

hadoop安装的前置条件是系统中已经有Java的环境
还需要将免密登录配置好

创建Hadoop用户
#创建Hadoop的用户
sudo useradd hadoop#设置Hadoop用户的密码
sudo passwd hadoop   #设置Hadoop的用户有sudo权限            
sudo usermod -aG sudo hadoop
解压Hadoop的安装包
#解压这个压缩包到/opt的目录下，这里最好别放在/root下
sudo tar -xzf hadoop-3.4.1.tar.gz -C /opt#给这个解压完的目录，修改一个目录名
sudo mv /opt/hadoop-3.4.1 /opt/hadoop#设置这个路径的用户、用户组
sudo chown -R hadoop:hadoop /opt/hadoop
创建环境变量在配置文件中
#将配置写入到配置文件中
echo 'export HADOOP_HOME=/opt/hadoop' | sudo tee -a /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' | sudo tee -a /etc/profile#让配置文件立即生效
source /etc/profile
修改这几个配置文件

这几个配置文件的位置如下；
$HADOOP_HOME/etc/hadoop/

1.hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
2.core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value>  # 默认文件系统地址</property><property><name>hadoop.tmp.dir</name><value>/opt/hadoop/tmp</value>        # 临时文件目录:</property>
</configuration>
3.hdfs-site.xml
<property><name>dfs.replication</name><value>1</value>                        # 数据副本数（单机设置为1）
</property>
<property><name>dfs.namenode.name.dir</name><value>/opt/hadoop/hdfs/namenode</value>  # NameNode数据存储路径
</property>
4.mapred-site.xml
<property><name>mapreduce.framework.name</name><value>yarn</value>  # 启用YARN框架
</property>
5.yarn-site.xml
<property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value>  # 指定NodeManager附加服务
</property>
格式化HDFS文件系统
hdfs namenode -format  # 初始化NameNode
启动hadoop
1.启动hdfs
start-dfs.sh2.启动yarn
start-yarn.sh
查看集群启动状态及进程
jpshdfs dfsadmin -report

启动hadoop报错

./start-dfs.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation

解决方案，在配置文件中配置如下，即可用root用户进行启动了

四、hive的安装

https://dlcdn.apache.org/hive/hive-4.0.1/
在MySQL中创建Hive元数据库及用户
CREATE DATABASE hive_meta;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Jky1234!@#$';
GRANT ALL ON hive_meta.* TO 'hive'@'%';
FLUSH PRIVILEGES;
修改Hive配置文件
cd $HIVE_HOME/conf
cp hive-default.xml.template hive-site.xml
vi hive-site.xml  # 替换以下关键参数

hive的配置文件内容添加如下

<property><name>javax.jdo.option.ConnectionURL</name><value>jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true</value>
</property>
<property><name>javax.jdo.option.ConnectionDriverName</name><value>com.mysql.cj.jdbc.Driver</value>
</property>
<property><name>javax.jdo.option.ConnectionUserName</name><value>hive</value>
</property>
<property><name>javax.jdo.option.ConnectionPassword</name><value>Jky1234!@#$</value>
</property>

下载mysql-connector-java-8.0.27.jar并拷贝到Hive的lib目录

cp mysql-connector-java-8.0.27.jar /opt/hive/lib/opt/hadoop/share/hadoop/common/lib
cp guava-27.0-jre.jar /opt/hive/lib

在hive的bin目录下执行初始化数据库的操作

./schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 4.0.0
Metastore connection URL:	 jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true
Metastore connection Driver :	 com.mysql.cj.jdbc.Driver
Metastore connection User:	 hive
Starting metastore schema initialization to 4.0.0
Initialization script hive-schema-4.0.0.mysql.sqlInitialization script completed

是否成功？ 去MySQL中

use hive_meta;
show tables;

启动hive的客户端，显示如下；
仔细看日志打印的信息，应该是提示日志冲突了

./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.1 by Apache Hive
beeline> show databases;
No current connection

ll log4j
-rw-r–r–. 1 root root 349845 9月 25 2024 log4j-1.2-api-2.18.0.jar
-rw-r–r–. 1 root root 315115 9月 25 2024 log4j-api-2.18.0.jar
-rw-r–r–. 1 root root 1861441 9月 25 2024 log4j-core-2.18.0.jar
-rw-r–r–. 1 root root 24801 9月 25 2024 log4j-slf4j-impl-2.18.0.jar
-rw-r–r–. 1 root root 36166 9月 25 2024 log4j-web-2.18.0.jar

mv log4j-slf4j-impl-2.18.0.jar log4j-slf4j-impl-2.18.0-bak.jar0526
然后在执行，发现多余的日志信息没有了

执行连接hive的命令，报错如下；

!connect jdbc:hive2://localhost:10000

./beeline --verbose
Default hs2 connection config file not found
Beeline version 4.0.1 by Apache Hive

查看1000的端口是否被监听
ss -tulnp | grep 9870
tcp LISTEN 0 500 0.0.0.0:9870 0.0.0.0:* users:((“java”,pid=13987,fd=329))

./hive --service hiveserver2 --hiveconf hive.log.dir=/var/log/hive --hiveconf hive.log.file=hiveserver2-daemon.log
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/java-se-8u44-ri/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/hive/bin:/root/bin)
2025-05-26 18:16:09: Starting HiveServer2
Exception in thread “main” java.lang.IllegalArgumentException: Logs will be split in two files if the commandline argument hive.log.file is used. To prevent this use to HADOOP_CLIENT_OPTS -Dhive.log.file=hiveserver2-daemon.log or use the set the value in the configuration file (see HIVE-19886)
at org.apache.hive.service.server.HiveServer2$ServerOptionsProcessor.parse(HiveServer2.java:1366)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:330)
at org.apache.hadoop.util.RunJar.main(RunJar.java:245)

./beeline --verbose
Overriding connection url property url_prefix from user connection configuration file
Connecting to jdbc:hive2://localhost:10000/default
Enter username for jdbc:hive2://localhost:10000/default: hive
Enter password for jdbc:hive2://localhost:10000/default: ****
25/05/26 18:23:26 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive
解决方案

hadoop.proxyuser.root.hosts



hadoop.proxyuser.root.groups

http://www.xdnf.cn/news/664543.html

相关文章：

  • 【前端基础】Promise 详解
  • FacePoke创意交互实战：Cpolar技术赋能远程人像编辑的趣味实现
  • 国内短剧 vs. 海外短剧系统开发：2025年SEO优化与市场策略全解析
  • 机械设计插件
  • MS1824+MS7210+MS2130 1080P@60Hz USB3.0采集
  • 【文献阅读】Mixture of Lookup Experts
  • 语音识别技术在人工智能中的应用
  • 03 环境变量和标签
  • 电子元器件散热方式
  • 医院门户网站群改版技术白皮书
  • 如何调试CATIA CAA程序导致的CATIA异常崩溃问题
  • Vue 3 核心知识点全览
  • 电子电气架构 -- 第五代汽车电子电气（E/E）架构的两种主导实施方式
  • c++ 二叉搜索树（BinarySearchTree）
  • 晚期NSCLC临床试验终点与分析策略
  • 【力扣】关于链表索引
  • 初识LangChain
  • Visual Studio 调试中 PDB 与图像不匹配
  • STM32F103_Bootloader程序开发03 - 启动入口与升级模式判断(boot_entry.c与boot_entry.h)
  • JetsonHacksNano RealSense自动安装脚本文件解析
  • 公链开发全生态：技术架构、生态构建与未来图景
  • 环境配置相关问题以及解决方案
  • JavaScripts 常见误区
  • 小刚说C语言刷题—1152 - 求n个数的最大值和最小值
  • mybatis-plus动态分页
  • ARM架构
  • 密钥分发与公钥证书
  • 工厂方法模式之Factory Method（工厂方法）
  • Python网络编码——聊天小工具
  • 2025年中国ERP软件前十名对比：选型指南与适用场景的分析