How to install the standalone version of spark in linux environment without using hadoop
|
Big data continues to heat up, and if you are not familiar with several big data components, you don’t even have a catchphrase to show off. At the very least, you should be able to speak hadoop, hdfs, mapreduce, yarn, kafka, spark, zookeeper, neo4j. These are essential skills for showing off. There are a lot of detailed introductions about spark on the Internet. Just search for it. Next, let’s talk about the installation and brief use of the stand-alone version of spark. 0. Install JDK. Since I already have JDK on my machine, I can skip this step. JDK is already a cliché, needless to say, it is indispensable when using Java/Scala. ubuntu@VM-0-15-ubuntu:~$ java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) ubuntu@VM-0-15-ubuntu:~$ 1. You don't necessarily need to install hadoop, you just need to choose a specific spark version. You don't need to download Scala, because Spark will come with Scala shell by default. Go to Spark official website to download it. In an environment without Hadoop, you can choose: spark-2.2.1-bin-hadoop2.7, and then unzip it as follows: ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc$ ll total 196436 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 2 19:57 ./ drwxrwxr-x 9 ubuntu ubuntu 4096 Feb 2 19:54 ../ drwxrwxr-x 13 ubuntu ubuntu 4096 Feb 2 19:58 spark-2.2.1-bin-hadoop2.7/ -rw-r--r-- 1 ubuntu ubuntu 200934340 Feb 2 19:53 spark-2.2.1-bin-hadoop2.7.tgz 2. Spark has Python and Scala versions. Next, I will use the Scala version of the shell, as follows: ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$ bin/spark-shell Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 18/02/02 20:12:16 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using built-java classes where applicable 18/02/02 20:12:16 WARN Utils: Your hostname, localhost resolves to a loopback address: 127.0.0.1; using 172.17.0.15 instead (on interface eth0) 18/02/02 20:12:16 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address Spark context Web UI available at http://172.17.0.15:4040 Spark context available as 'sc' (master = local[*], app id = local-1517573538209). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.2.1 /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_151) Type in expressions to have them evaluated. Type :help for more information. scala> To perform simple operations:
scala> val lines = sc.textFile("README.md")
lines: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:24
scala> lines.count()
res0: Long = 103
scala> lines.first()
res1: String = # Apache Spark
scala> :quit
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$ wc -l README.md
103 README.md
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$ head -n 1 README.md
# Apache Spark
ubuntu@VM-0-15-ubuntu:~/taoge/spark_calc/spark-2.2.1-bin-hadoop2.7$Let's take a look at the visual web page. On Windows, enter: http://ip:4040
OK, this article is just a simple installation, we will continue to introduce spark in depth later. Summarize The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. Thank you for your support of 123WORDPRESS.COM. If you want to learn more about this, please check out the following links You may also be interested in:
|
<<: Sample code for implementing interface signature with Vue+Springboot
>>: Download MySQL 5.7 and detailed installation diagram for MySql on Mac
Recommend
Summary of problems encountered when installing docker on win10 home version
Docker download address: http://get.daocloud.io/#...
Detailed explanation of the API in Vue.js that is easy to overlook
Table of contents nextTick v-model syntax sugar ....
Common HTML tag writing errors
We better start paying attention, because HTML Po...
An article to understand the use of proxies in JavaScript
Table of contents What is an agent Basic knowledg...
Example code for changing the style of other tags by hovering the mouse using CSS
Preface: As far as I know, currently CSS can only...
Learn the black technology of union all usage in MySQL 5.7 in 5 minutes
Performance of union all in MySQL 5.6 Part 1:MySQ...
Optimizing query speed of MySQL with tens of millions of data using indexes
1. The role of index Generally speaking, an index...
Detailed explanation of how to connect to MySQL database using Java in IntelliJ IDEA
1. Download MySQL database and install and config...
Basic usage and pitfalls of JavaScript array sort() method
Preface In daily code development, there are many...
Five guidelines to help you write maintainable CSS code
1. Add a comment block at the beginning of the sty...
Steps to modify the MySQL database data file path under Linux
After installing the MySQL database using the rpm...
How to configure MySQL8 in Nacos
1. Create the MySQL database nacos_config 2. Sele...
The implementation process of long pressing to identify QR code in WeChat applet
Preface We all know that the QR codes in official...
CSS to achieve chat bubble effect
1. Rendering JD Effect Simulation Effect 2. Princ...
Solution to the problem that the InnoDB engine is disabled when MySQL is started
Find the problem Today at work, when copying tabl...