您的足迹:首页 > Spark >Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

在往spark集群提交作业时报错日志如下:

16/01/22 11:40:34 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

从警告信息上看,初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。

如上问题产生的原因是多方面的,可能原因如下:

1.因为提交任务的节点不能和spark工作节点交互,因为提交完任务后提交任务节点上会起一个进程,展示任务进度,大多端口为4044,工作节点需要反馈进度给该该端口,所以如果主机名或者IP在hosts中配置不正确。所以检查下主机名和ip是否配置正确

2.有可能是内存不足

  检查内存

   conf.set("spark.executor.memory", "3000m")

  Make sure to set SPARK_LOCAL_IP andSPARK_MASTER_IP.

  查看8080端口,确保一些workers保持Alive状态,确保 some cores 是可利用的。

相关推荐


  • blogger

发表评论

路人甲 表情
看不清楚?点图切换 Ctrl+Enter快速提交

网友评论(0)