|NO.Z.00006|——————————|^^ 配置 ^^|——|Hadoop&Spark.V06|------------------------------------------|Spar
2022/4/11 23:17:09
本文主要是介绍|NO.Z.00006|——————————|^^ 配置 ^^|——|Hadoop&Spark.V06|------------------------------------------|Spar,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
[BigDataHadoop:Hadoop&Spark.V06] [BigDataHadoop.Spark内存级快速计算引擎][|章节一|Hadoop|spark|sparkcore:Spark-Standalone集群模式&standalone配置&core&mamory|]
一、集群模式--Standalone模式
### --- 集群模式--Standalone模式 ~~~ 参考:http://spark.apache.org/docs/latest/spark-standalone.html ~~~ 分布式部署才能真正体现分布式计算的价值 ~~~ 与单机运行的模式不同,这里必须先启动Spark的Master和Worker守护进程;关闭 yarn 对应的服务 ~~~ 不用启动Hadoop服务,除非要使用HDFS的服务二、检查集群状态
### --- 使用jps检查,可以发现: ~~~ # 启动服务 [root@hadoop01 ~]# start-dfs.sh [root@hadoop01 ~]# stopt-yarn.sh [root@hadoop02 ~]# start-all-spark.sh
### --- 使用浏览器查看:http://hadoop02:8080/ ~~~ # 检查集群状态 [root@hadoop02 ~]# jps Hadoop01:Worker Hadoop02:Master、Worker Hadoop03:Worker
三、Standalone配置
### --- Standalone 配置 ~~~ sbin/start-master.sh / sbin/stop-master.sh ~~~ sbin/start-slaves.sh / sbin/stop-slave.sh ~~~ sbin/start-slave.sh / sbin/stop-slaves.sh ~~~ sbin/start-all.sh / sbin/stop-all.sh ~~~ 备注:./sbin/start-slave.sh [options];启动节点上的worker进程,调试中较为常用
### --- standalone配置:定义core和mamory参数 ~~~ 在 spark-env.sh 中定义:定义spark-worker的core和mamory ~~~ 官方文档地址:http://spark.apache.org/docs/latest/spark-standalone.html
~~~ # 默认是使用所有的core和mamory SPARK_WORKER_CORES:Total number of cores to allow Spark applications to use on the machine (default: all available cores). SPARK_WORKER_MEMORY:Total amount of memory to allow Spark applications to use on the machine, e.g. 1000m, 2g (default: total memory minus 1 GiB); note that each application's individual memory is configured using its spark.executor.memory property.
### --- 测试在 spark-env.sh 中增加参数,分发到集群,重启服务: [root@hadoop02 ~]# vim $SPARK_HOME/conf/spark-env.sh export SPARK_WORKER_CORES=10 export SPARK_WORKER_MEMORY=20g
[root@hadoop02 ~]# rsync-script $SPARK_HOME/conf/spark-env.sh [root@hadoop02 ~]# stop-all-spark.sh [root@hadoop02 ~]# start-all-spark.sh
### --- 在浏览器中观察集群状态,测试完成后将以上两个参数分别改为10、20g,重启服务。 ~~~ 修改回默认的配置 ~~~ 发送到其它主机并重启服务 [root@hadoop02 ~]# vim $SPARK_HOME/conf/spark-env.sh # export SPARK_WORKER_CORES=1 # export SPARK_WORKER_MEMORY=2g
附录一:定版本文件
### --- $SPARK_HOME/conf/spark-env.sh [root@hadoop02 ~]# vim $SPARK_HOME/conf/spark-env.sh export JAVA_HOME=/opt/yanqi/servers/jdk1.8.0_231 export HADOOP_HOME=/opt/yanqi/servers/hadoop-2.9.2 export HADOOP_CONF_DIR=/opt/yanqi/servers/hadoop-2.9.2/etc/hadoop export SPARK_DIST_CLASSPATH=$(/opt/yanqi/servers/hadoop-2.9.2/bin/hadoop classpath) export SPARK_MASTER_HOST=hadoop02 export SPARK_MASTER_PORT=7077 export SPARK_WORKER_CORES=1 export SPARK_WORKER_MEMORY=1g
===============================END===============================
Walter Savage Landor:strove with none,for none was worth my strife.Nature I loved and, next to Nature, Art:I warm'd both hands before the fire of life.It sinks, and I am ready to depart ——W.S.Landor
来自为知笔记(Wiz)
这篇关于|NO.Z.00006|——————————|^^ 配置 ^^|——|Hadoop&Spark.V06|------------------------------------------|Spar的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2023-05-13Windows下hadoop环境搭建之NameNode启动报错
- 2023-04-14hadoop伪分布式集群的安装(不是单机版)
- 2022-12-05Hadoop生态系统—数据仓库Hive的安装
- 2022-11-02Win10搭建Hadoop环境
- 2022-10-19Hadoop生态系统(数据仓库Hive的安装)
- 2022-10-03Hadoop、storm和Spark Streaming简单介绍
- 2022-10-03胖虎的Hadoop笔记——Hadoop的伪分布式部署
- 2022-09-11Ubuntu搭建全分布式Hadoop
- 2022-09-11Ubuntu搭建全分布式Hadoop
- 2022-09-09Ubuntu下安装伪分布式HADOOP遇到的一些问题