Query Execution Optimization in Spark SQL

Table 1

Comparison between calculated and actual shuffle sizes.

ID	1	2	3	4	5	6	7	8	9	10	11
General task	9	114	14	119	7	14	9	22	14	7	7
Input (MB)	256	2688	384	2560	512	384	128	128	256	640	256
Calculated shuffle size	27	282.4	40.3	269	53.81	40.35	13.45	15.05	28.03	67.25	29.06
Actual shuffle size	25	267.58	38.2	254.8	50.96	38.22	12.76	12.76	25.48	63.71	25.48