Spark3教程(六)IDEA下Java开发Spark SQL
2021/12/22 2:20:11
本文主要是介绍Spark3教程(六)IDEA下Java开发Spark SQL,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
上一篇文章中,我们使用了Scala语言调用Spark SQL接口进行了开发,本篇文章我们使用Java语言进行同样业务功能的处理,依然是对JSON、Txt文本进行处理。
JSON和Txt文件内容如下所示:
{"name":"Michael"} {"name":"Andy", "age":30} {"name":"Justin", "age":19}
Michael, 29 Andy, 30 Justin, 19
Java处理JSON代码:
import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import org.apache.spark.sql.SparkSession; public class TestSQL { public static void main(String[] args) { SparkSession spark = SparkSession .builder().master("local") .appName("Java Spark SQL basic example") .config("spark.some.config.option", "some-value") .getOrCreate(); Dataset<Row> df = spark.read().json("file:///d:/test_spark/people.json"); df.show(); df.createOrReplaceTempView("people"); Dataset<Row> sqlDF = spark.sql("select * from people where age>20"); sqlDF.show(); } }
Java处理Txt代码,需要定义一个Person实体类:
public class Person { private String name; private long age; public String getName() { return name; } public void setName(String name) { this.name = name; } public long getAge() { return age; } public void setAge(long age) { this.age = age; } }
import com.alan.entity.Person; import org.apache.spark.api.java.JavaRDD; import org.apache.spark.sql.*; public class TestText { public static void main(String[] args) { SparkSession spark = SparkSession .builder().master("local") .appName("Java Spark SQL basic example") .config("spark.some.config.option", "some-value") .getOrCreate(); JavaRDD<Person> peopleRDD = spark.read() .textFile("d:/test_spark/people.txt") .javaRDD() .map(line -> { String[] parts = line.split(","); Person person = new Person(); person.setName(parts[0]); person.setAge(Integer.parseInt(parts[1].trim())); return person; }); // Apply a schema to an RDD of JavaBeans to get a DataFrame Dataset<Row> peopleDF = spark.createDataFrame(peopleRDD, Person.class); // Register the DataFrame as a temporary view peopleDF.createOrReplaceTempView("people"); // SQL statements can be run by using the sql methods provided by spark Dataset<Row> teenagersDF = spark.sql("select * from people where age>20"); teenagersDF.show(); } }
这篇关于Spark3教程(六)IDEA下Java开发Spark SQL的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2024-10-06小米11i印度快充版ROM合集:极致体验,超越期待
- 2024-10-06【ROM下载】小米11i 5G 印度版系统, 疾速跃迁,定义新速度
- 2024-10-06【ROM下载】小米 11 青春活力版,青春无极限,活力全开
- 2024-10-05小米13T Pro系统合集:性能与摄影的极致融合,值得你升级的系统ROM
- 2024-10-01基于Python+Vue开发的医院门诊预约挂号系统
- 2024-10-01基于Python+Vue开发的旅游景区管理系统
- 2024-10-01RestfulAPI入门指南:打造简单易懂的API接口
- 2024-10-01初学者指南:了解和使用Server Action
- 2024-10-01Server Component入门指南:搭建与配置详解
- 2024-10-01React 中使用 useRequest 实现数据请求