Notice: Function _load_textdomain_just_in_time was called incorrectly. Translation loading for the becustom domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/joyplace/public_html/wp-includes/functions.php on line 6114
Notice: Function _load_textdomain_just_in_time was called incorrectly. Translation loading for the wordpress-seo domain was triggered too early. This is usually an indicator for some code in the plugin or theme running too early. Translations should be loaded at the init action or later. Please see Debugging in WordPress for more information. (This message was added in version 6.7.0.) in /home4/joyplace/public_html/wp-includes/functions.php on line 6114
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
Warning: Cannot modify header information - headers already sent by (output started at /home4/joyplace/public_html/wp-includes/functions.php:6114) in /home4/joyplace/public_html/wp-includes/rest-api/class-wp-rest-server.php on line 1893
{"id":1699,"date":"2021-08-04T07:05:59","date_gmt":"2021-08-04T12:05:59","guid":{"rendered":"https:\/\/www.bigdatainrealworld.com\/?p=1699"},"modified":"2023-02-19T07:31:17","modified_gmt":"2023-02-19T13:31:17","slug":"how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel","status":"publish","type":"post","link":"https:\/\/www.bigdatainrealworld.com\/how-does-spark-decide-the-number-of-tasks-and-number-of-tasks-to-execute-in-parallel\/","title":{"rendered":"How does Spark decide the number of tasks and number of tasks to execute in parallel?"},"content":{"rendered":"In this post we will see how Spark decides the number of tasks and number of tasks to execute in parallel in a job.<\/p>\n
Let\u2019s see how Spark decides on the number of tasks with the below set of instructions.<\/span><\/p>\n\n- READ dataset_X<\/span><\/li>\n
- FILTER on dataset_X<\/span><\/li>\n
- MAP operation on dataset_X<\/span><\/li>\n
- READ dataset_Y<\/span><\/li>\n
- MAP operation on dataset_Y<\/span><\/li>\n
- JOIN dataset_X and dataset_Y<\/span><\/li>\n
- FILTER on joined dataset<\/span><\/li>\n
- SAVE the output<\/span><\/li>\n<\/ol>\n
Let\u2019s also assume dataset_Y has 10 partitions and dataset_Y has 5 partitions.<\/span>
\n