Research article · 30 Jan 2017
Research articlePublished Mar 2014
A Micro Partitioning Technique in MapReduce for Massive Data Analysis
Abstract
Select any words in the title or abstract, or press this and choose a sentence.
Over the past years, large amounts of structured and unstructured data are being collected from various sources. These huge amounts of data are difficult to handle by a single machine which requires the work to be distributed across large number of computers. Hadoop is one such distributed framework which process data in distributed manner by using Mapreduce programming model. In order for Mapreduce to work, it has to divide the workload across the machines in the cluster. The performance of Mapreduce depends on how evenly it distributes the workload to the machines without skew and avoids executing job in a poorly running node called straggler. The workload distribution depends on the algorithm that partitions the data. To overcome the problem from skew, an efficient partitioning technique is proposed. The proposed algorithm improves load balancing as well as reduces the memory requirements. Slow running nodes degrade the performance of Mapreduce job. To overcome this problem, a technique called micro partitioning is used that divide the tasks into smaller tasks greater than the number of reducers and are assigned to reducers. Running many small tasks lessens the impact of stragglers, since work that would have been scheduled on slow nodes is only small which can be performed by other idle workers.
Ask about this article
Answers come only from this article's title, abstract and keywords, each sentence pointing to where it comes from. Your question is sent to OpenAI to write the answer. Scholars Central does not store it.
Similar by topic
Research article · Mar 2014
Optimized Data Transmission from Cloud to Society by Mapreduce
Research article · 2015
Hadoop MapReduce: A Programming Model for Large Scale Data Processing
Research article · May 2015
Slicing Technique to Prevent Generalized Losses and Membership Disclosure in Micro Data Publishing
Research article · Apr 2014
Partitioning Clustering Algorithms for Data Stream Outlier Detection
More from this journal
The newest in International Journal of Innovative Research in Computer and Communication Engineering.
Commentary · 2023
Computer Vision and Pattern Recognition
Commentary · 2023
Computer Network and Network Optimization
Perspective · 2023
Cloud Computing and Big Data Analysis
Conferences in this field
Upcoming conferences in this field.
Follow
New articles from what you follow go into your email digest, once you agree to alert emails.
- JournalInternational Journal of Innovative Research in Computer and Communication Engineering
- SubjectComputer Science & IT
- SubjectEngineering
- SubjectSocial Sciences & Humanities
- BylineArticles whose byline prints “Nandhini.C”
- BylineArticles whose byline prints “Premadevi.P”
