当前位置：首页 > backend >正文

记一次运行spark报错

backend 2025/6/29 19:42:48

提交spark任务运次报错

06/03 18:27:50 INFO Client: Setting up container launch context for our AM
25/06/03 18:27:50 INFO Client: Setting up the launch environment for our AM container
25/06/03 18:27:50 INFO Client: Preparing resources for our AM container
25/06/03 18:27:50 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
25/06/03 18:27:59 INFO Client: Uploading resource file:/tmp/spark-27957b63-3767-457f-a931-cc7f84917c3e/__spark_libs__545909112269949051.zip -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/__spark_libs__545909112269949051.zip
25/06/03 18:28:02 INFO Client: Uploading resource file:/opt/apps/everdc/spark_jars/everdc-stand-model-engine-spark-4.9.0.jar -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/everdc-stand-model-engine-spark-4.9.0.jar
25/06/03 18:28:02 INFO Client: Uploading resource file:/app/logs/model_run_2.json -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/model_run_2.json
25/06/03 18:28:02 INFO Client: Uploading resource file:/tmp/spark-27957b63-3767-457f-a931-cc7f84917c3e/__spark_conf__4210376678626642445.zip -> hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040/__spark_conf__.zip
25/06/03 18:28:03 INFO SecurityManager: Changing view acls to: root,hdfs
25/06/03 18:28:03 INFO SecurityManager: Changing modify acls to: root,hdfs
25/06/03 18:28:03 INFO SecurityManager: Changing view acls groups to: 
25/06/03 18:28:03 INFO SecurityManager: Changing modify acls groups to: 
25/06/03 18:28:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root, hdfs); groups with view permissions: Set(); users  with modify permissions: Set(root, hdfs); groups with modify permissions: Set()
25/06/03 18:28:03 INFO Client: Deleted staging directory hdfs://everdc/user/hdfs/.sparkStaging/application_1748248020756_0040
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getPassword(Ljava/lang/String;)[Cat org.apache.spark.SSLOptions$.$anonfun$parse$8(SSLOptions.scala:194)at scala.Option.orElse(Option.scala:447)at org.apache.spark.SSLOptions$.parse(SSLOptions.scala:194)at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:98)at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:1137)at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:220)at org.apache.spark.deploy.yarn.Client.run(Client.scala:1327)at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1764)