Conference abstract · 7 Sep 2017
Conference abstractPublished 2017
Memory management in modern data analytics platforms
Duke University, USA
Abstract
Select any words in the title or abstract, or press this and choose a sentence.
The field of data processing is seeing an explosion of computation platforms such as Hadoop, Spark, HBase, Cassandra, Flink, Storm, and others. Some key observations of these platforms are: They are JVM-based and therefore developer friendly, stable, and secure; they run in multi-tenant environments where resources are isolated using containers, e.g. Yarn, Mesos, and Docker and; they use memory extensively both for data storage and computations. The central premise of this work is that data analytics will increasingly be done in memory on shared-nothing clusters running JVM inside containers. Given this, it is high time for a better understanding of memory management in these modern data analytics platforms. This work uses Thoth, a data-driven platform for multi-system cluster management developed at Duke University, to build a deep understanding of different interplays of memory-management decisions made at multiple levels such as the resource manager (e.g. Yarn), the application framework (e.g. Spark), and the JVM. An empirical study is carried out over a suite of industrystandard analytical workloads to show the impact of various memory management options on response time, efficiency, and reliability. A set of guidelines follows to ensure a safe and efficient execution of the workloads.
Subjects
Computer Science & ITEngineeringAsk about this article
Answers come only from this article's title, abstract and keywords, each sentence pointing to where it comes from. Your question is sent to OpenAI to write the answer. Scholars Central does not store it.
Similar by topic
Short communication · 25 Mar 2024
Development of Data Cleansing and Data Storage for Modern Analytics and Reporting through Data Warehousing
Short communication · 2021
High-performance data analytics: Platforms, resource management and middleware
Conference abstract · 26 Sep 2018
High-performance data analytics: Platforms, resource management, and middleware
Conference abstract · 2023
How to Manage Big Data in Healthcare and what are Automated Machine Learning Tools in this case Introduction of a powerful Medical Registry and Data Analysis Platform as automated machine learning tool
More from this journal
The newest in Journal of Computer Engineering & Information Technology.
Review article · 27 Oct 2025
Dukan Daari Inventory Management System
Research article · 17 Jun 2025
Leveraging Information Flow Tracking for Enhanced Security in Cloud Computing
Editorial · 31 Mar 2025
Low-Code/No-Code Platforms: Democratizing Software Development
Conferences in this field
Upcoming conferences in this field.
Follow
New articles from what you follow go into your email digest, once you agree to alert emails.
- JournalJournal of Computer Engineering & Information Technology
- SubjectComputer Science & IT
- SubjectEngineering
- BylineArticles whose byline prints “Mayuresh Kunjir”

