Hands-On Big Data Analytics with PySpark: Analyze large datasets and discover techniques for testing,immunizing,and parallelizing Spark jobs
Authors: Rudy Lai – Bartlomiej Potaczek
ISBN-10: 183864413X
ISBN-13: 9781838644130
Released: 2019-03-29
Print Length 页数: 182 pages
Book Description
More Information
Learn
Get practical big data experience while working on messy datasets
Analyze patterns with Spark SQL to improve your business intelligence
Use PySpark’s interactive shell to speed up development time
Create highly concurrent Spark programs by leveraging immutability
Discover ways to avoid the most expensive operation in the Spark API: the shuffle operation
Re-design your jobs to use reduceByKey instead of groupBy
Create robust processing pipelines by testing Apache Spark jobs
About
Apache Spark is an open source parallel-processing framework that has been around for quite some time now. One of the many uses of Apache Spark is for data analytics applications across clustered computers. In this book,you will not only learn how to use Spark and the Python API to create high-performance analytics with big data,but also discover techniques for testing,immunizing,and parallelizing Spark jobs.
You will learn how to source data from all popular data hosting platforms,including HDFS,Hive,JSON,and S3,and deal with large datasets with PySpark to gain practical big data experience. This book will help you work on prototypes on local machines and subsequently go on to handle messy data in production and at scale. This book covers installing and setting up PySpark,RDD operations,big data cleaning and wrangling,and aggregating and summarizing data into useful reports. You will also learn how to implement some practical and proven techniques to improve certain aspects of programming and administration in Apache Spark.
By the end of the book,you will be able to build big data analytical solutions using the various PySpark offerings and also optimize them effectively.
Features
Work with large amounts of agile data using distributed datasets and in-memory caching
Source data from all popular data hosting platforms,such as HDFS,Hive,JSON,and S3
Employ the easy-to-use PySpark API to deploy big data Analytics for production
contents
1 Installing Pyspark and Setting up Your Development Environment
2 Getting Your Big Data into the Spark Environment Using RDDs
3 Big Data Cleaning and Wrangling with Spark Notebooks
4 Aggregating and Summarizing Data into Useful Reports
5 Powerful Exploratory Data Analysis with MLlib
6 Putting Structure on Your Big Data with SparkSQL
7 Transformations and Actions
8 Immutable Design
9 Avoiding Shuffle and Reducing Operational Expenses
10 Saving Data in the Correct Format
11 Working with the Spark Key/Value API
12 Testing Apache Spark Jobs
13 Leveraging the Spark GraphX API
