Conference abstract · 8 Sep 2014
Conference abstract
Schedule optimization for big data processing on cloud
Abstract
Select any words in the title or abstract, or press this and choose a sentence.
Over the past few years, the continuous increase in computational capacity has produced an overwhelming flow of data or big data, which exceeds the capabilities of conventional processing tools. Big data offer a new era in data exploration and utilization. The major enabler for underlying many big data platforms is certainly the MapReduce computational paradigm. MapReduce is recognized as a popular programming model for the distributed and scalable processing of big data and is increasingly being used in different applications mostly because of its important features that include scalability, flexibility, ease of programming, and fault-tolerance. Scheduling tasks in MapReduce across multiple nodes have shown to be multiobjective optimization problem. The problem is even more complex by using virtualized clusters in a cloud computing to execute a large number of tasks. The complexity lies in achieving multiple objectives that may be of conflicting nature. For instance, scheduled tasks may require to make several tradeoffs between the job performance, data locality, fairness, resource utilization, network congestion and reliability. These conflicting requirements and goals are challenging to optimize due to the difficulty of predicting a new incoming jobâ??s behavior and its completion time. To address this complication, we introduce a multi-objective approach using genetic algorithms. The goal is to minimize two objectives: Execution time, and budget of each node executing the task in the cloud. The contribution of this research is to propose a novel adaptive model to communicate with the task scheduler of resource management. The proposed model periodically queries for resource consumption data and uses to calculate how the resources should be allocated to each task. It passes the information to the task scheduler by adjusting task assignments to task nodes accordingly. The model evaluation is realized in scheduling load simulator. PingER, the Internet End-to-End performance measurement, was chosen for performance analysis of the model. We believe this proposed solution is timely and innovative as it provides a robust resource management where users can perform better scheduling for big data processing in a seamless manner.
Ask about this article
Answers come only from this article's title, abstract and keywords, each sentence pointing to where it comes from. Your question is sent to OpenAI to write the answer. Scholars Central does not store it.
Similar by topic
Article · 1 Jun 2025
Big Data and Cloud: Revolutionizing Biomedical Systems
Conference abstract · 2018
Machine learning 2018 and Big Data 2018- Digital transformation and the convergence of new emerging digital technologies - Samir El Masri - Digitalization. Cloud, UAE
Article · 28 Jun 2025
Big Data Challenges in Human Genomic Research
Conference abstract · 24 Oct 2016
Big Data in noncoding RNA and precision medicine
More from this journal
The newest in Journal of Data Mining in Genomics & Proteomics.
Article · 30 Jun 2026
Artificial Intelligence-Driven Drug Discovery: Reimagining the Journey from Molecule to Medicine
Article · 30 Jun 2026
Comparative Genomics: Revealing Evolutionary Patterns Across Biological Worlds
Article · 30 Jun 2026
Predicting the Hidden Roles of Proteins through Computational Function Analysis
Conferences in this field
Upcoming conferences in this field.
Follow
New articles from what you follow go into your email digest, once you agree to alert emails.
- JournalJournal of Data Mining in Genomics & Proteomics
- SubjectBiochemistry & Biotechnology
- SubjectGenetics & Molecular Biology
- BylineArticles whose byline prints “Ibrahim Abaker Targio Hashem”
- BylineArticles whose byline prints “Nor Badrul Anuar”
- BylineArticles whose byline prints “Abdullah Gani”

