Improving Efficiency of Apache Spark by Tuning its Internal Features

Abstract

We are always enhancing Spark's speed and usefulness. To improve Spark's usability, we and other community members are adding a substantial number of standard libraries that provide scaled variations of popular data analysis methods. For instance, in the previous year, the size of Spark's MLlib machine learning library increased by a factor of 4. Additionally, utilising DataFrames or SQL, it is simple to access external data sources using our pluggable data source API. These APIs make up one of the most integrated standard libraries for "big data" and will surely prompt creative design choices that will make the building of workflows more effective.