Spark 集群安装

github_spark 集群安装

大数据 spark 集群安装脚本： https://github.com/big-data-europe/docker-spark

version: '3'
services:
  spark-master:
    image: bde2020/spark-master:3.1.1-hadoop3.2
    container_name: spark-master
    ports:
      - "8080:8080"
      - "7077:7077"
    environment:
      - INIT_DAEMON_STEP=setup_spark
  spark-worker-1:
    image: bde2020/spark-worker:3.1.1-hadoop3.2
    container_name: spark-worker-1
    depends_on:
      - spark-master
    ports:
      - "8081:8081"
    environment:
      - "SPARK_MASTER=spark://spark-master:7077"
  spark-worker-2:
    image: bde2020/spark-worker:3.1.1-hadoop3.2
    container_name: spark-worker-2
    depends_on:
      - spark-master
    ports:
      - "8082:8081"
    environment:
      - "SPARK_MASTER=spark://spark-master:7077"

bin/spark-submit \ -- class org.apache.spark.examples.SparkPi \ -- master spark://192.168.124.128:7077 \ -- executor-memory 1G \ --total-executor-cores 1 \ examples/jars/spark-examples_2.12-3.1.1.jar \ 10

赏

谢谢你请我吃糖果

支付宝

微信

• 本文作者： MISAKIGA
• 本文链接： https://misakiga.github.io/2021/05/23/big-data/github_spark集群安装/
• 版权声明： 本博客所有文章除特别声明外，均采用 MIT 许可协议。转载请注明出处！