Everand Logo

Welcome to Everand!

  • Discover millions of ebooks, audiobooks, and so much more with a free trial

    Only $11.99/month after trial. Cancel anytime.

    Monitoring Elasticsearch
    Monitoring Elasticsearch
    Monitoring Elasticsearch
    Ebook322 pages1 hour

    Monitoring Elasticsearch

    By Dan Noble

    ()

    Read preview

    About this ebook

    About This Book
    • Understand common performance and reliability pitfalls in ElasticSearch
    • Use popular monitoring tools such as ElasticSearch-head, BigDesk, Marvel, Kibana, and more
    • This is a step-by-step guide with lots of case studies on solving real-world ElasticSearch cluster issues
    Who This Book Is For

    This book is for developers and system administrators who use Elasticsearch in a wide range of capacities. Prior knowledge of Elasticsearch and related technologies would be helpful, but is not necessary.

    LanguageEnglish
    PublisherPackt Publishing
    Release dateJul 27, 2016
    ISBN9781784391812
    Monitoring Elasticsearch

    Related to Monitoring Elasticsearch

    Related ebooks

    Internet & Web For You

    View More
    View More

    Related podcast episodes

    Related articles

    • Article

      How It Secures The Data?

      Jul 1, 2021

      How It Secures The Data?
      1 min read

    • Article

      Traefik Configuration

      Mar 10, 2020

      In this tutorial we have configured Traefik using command-line switches in our Docker Compose file (the section starting command:). This is the equivalent of starting the application with a whole bunch of command options each time, and while this wou

      1 min read

    • Article

      How To Develop A RESTful Client In Go

      Nov 16, 2021

      Mihalis Tsoukalos is a systems engineer and technical writer. He’s the author of Go Systems Programming and Mastering Go. You can reach him at @mactsouk. The subject of this month’s tutorial is RESTful services. In particular, you’re going to learn h

      9 min read

    • Article

      KAFKA Build Utilities With The Kafka Server

      Jul 2, 2019

      Nowadays, quite a few data architectures involve both a database and Apache Kafka, which is a distributed streaming platform and the subject of this tutorial. You can also find Kafka described as a publish-subscribe message system, which is a fancy w

      7 min read

    • Article

      Three Low-code Options

      Nov 12, 2020

      Counting Intel, Vodafone and VW among its customers, OutSystems helps businesses create cloudbased, on-premises and hybrid applications for mobile and web. Its development environment is predominantly drag-and-drop, with views for processes, data and

      3 min read

    • Article

      Basic Concepts

      Jul 2, 2019

      A messaging system such as Kafka enables you to send messages between processes, applications and servers. Applications connect to Kafka to send or get data. Strictly speaking, a Kafka ‘topic’ is a unit of storage in Kafka: data in Kafka is stored in

      1 min read

    • Article

      An Introduction To Rabbitmq

      Jun 29, 2021

      RabbitMQ is a Message Broker, which means that it can safely hold messages generated by applications and make them available to other applications. The main advantages are reliability, support for clustering and high-availability queues, tracing capa

      1 min read

    • Article

      Join the Pod, Man!

      May 30, 2023

      Join the Pod, Man!
      8 min read

    • Article

      Docker vs Podman

      Apr 19, 2021

      When Cockpit was first developed, it had plug-in support for administering your Docker containers remotely via its user-friendly web interface. But then Red Hat OS became a major backer of Cockpit, and when Red Hat developed its own alternative to Do

      1 min read

    • Article

      Automatically Provision Devices With Ansible

      Nov 15, 2022

      Matt Holder has worked in IT support for over a decade, and always tries to utilise Linux alongside other installed systems. C loud computing is a term that means a number of things. Software as a Service (SaaS) is one such example of what can be hos

      9 min read

    • Article

      Client-free Remote Desktop Access For All

      Sep 24, 2019

      Guacamole is a open-source software project consisting of a client and server component that work together to enable remote connectivity to server environments from a web browser. In fact, this open-source project goes back as far as 2010, having spe

      9 min read

    • Article

      What Is The Future Of Game Streaming Now That Stadia Is Dead?

      Oct 31, 2022

      Once hyped as being ‘the future of gaming’, the Google Stadia game streaming service was officially, just three years after launch and before even making it to Australian shores. When game streaming first launched we did have some apprehension about

      2 min read

    • Article

      Visualise Complex Data In Style Using Timelion

      Oct 20, 2020

      Simon Quain is a site reliability engineer who likes discovering open datasets online to play around with in the Elastic Stack. You’ve probably heard of Elasticsearch – the search engine that enables you to index and then quickly search through your

      9 min read

    • Article

      Top 10 Programming Languages

      Jan 5, 2023

      Top 10 Programming Languages
      8 min read

    • Article

      Metrics & Visuals In Go

      Nov 17, 2020

      Mihalis Tsoukalos is a DataOps engineer and a technical writer. He’s the author of Go Systems Programming and Mastering Go, 2nd edition. The subject of this tutorial is two-fold. First, it’s about creating a Go application that exports metrics to P

      7 min read

    • Article

      Ice Cold With Kali

      May 2, 2023

      Ice Cold With Kali
      3 min read

    • Article

      Build A Cloud-based Documentation Site

      Jul 27, 2021

      Docusaurus is an open source documentation application developed by Facebook. It’s one of a growing number of JAMstack static site generators that uses a blend of JavaScript, React and markdown to make it easy for you to deploy clean, professional-lo

      8 min read

    • Article

      EBPF To Enhance Kubernetes Monitoring

      Apr 1, 2022

      The introduction of Docker and Kubernetes has brought a dramatic revolution in the IT industry. Unlike the traditional methods of developing and deploying software, Kubernetes or K8s uses scaling and automated deployment. Thanks to the Linux function

      4 min read

    Related categories

    Reviews for Monitoring Elasticsearch

    0 ratings

    0 ratings0 reviews

    What did you think?

    Tap to rate

    Review must be at least 10 words

      Book preview

      Monitoring Elasticsearch - Dan Noble

      Table of Contents

      Monitoring Elasticsearch

      Credits

      About the Author

      About the Reviewers

      www.PacktPub.com

      eBooks, discount offers, and more

      Why subscribe?

      Preface

      What this book covers

      What you need for this book

      Who this book is for

      Conventions

      Reader feedback

      Customer support

      Downloading the example code

      Downloading the color images of this book

      Errata

      Piracy

      Questions

      1. Introduction to Monitoring Elasticsearch

      An overview of Elasticsearch

      Learning more about Elasticsearch

      Data distribution, redundancy, and fault tolerance

      Full-text search

      Similar technologies

      Apache Lucene

      Solr

      Ferret

      Monitoring Elasticsearch

      Resourcefulness and problem solving

      Summary

      2. Installation and the Requirements for Elasticsearch

      Installing Elasticsearch

      DEB/RPM installation

      The yum and apt-get repositories

      Ubuntu/Debian and apt-get

      CentOS/RHEL and yum

      Verification

      Configuration files

      Configuring an Elasticsearch cluster

      Cluster name

      Memory configuration

      Open file limit

      The maximum file limit

      Updating max file descriptors on Ubuntu Linux

      Enabling pluggable authentication modules

      Verifying the open file limit

      Disabling swapping

      Understanding your cluster

      Installing Elasticsearch-head

      Installing Bigdesk

      Marvel

      Cluster requirements

      Summary

      3. Elasticsearch-head and Bigdesk

      Cluster setup

      Cluster configuration

      Sample data

      Elasticsearch-head

      The Overview tab

      Cluster states

      Node and index actions

      The Indices tab

      The Browser tab

      The Structured Query tab

      The Any Request tab

      The official website

      Bigdesk

      The Elasticsearch cat API

      Background

      Count

      Health

      Indices

      Shards

      Summary

      4. Marvel Dashboard

      Setting up Marvel

      Upgrading Marvel

      Configuring Marvel

      Marvel agent configuration settings

      Marvel index configuration

      Understanding the Marvel dashboard

      Overview dashboard

      Indices dashboard

      Nodes dashboard

      Monitoring node failures

      Summary

      5. System Monitoring

      Working with Kopf

      Installing Kopf

      The cluster page

      The nodes page

      The rest page

      The more dropdown

      Working with Logstash and Kibana

      ELK

      Installation

      Installing Logstash

      Loading NGINX logs

      Installing Kibana

      Working with Nagios

      Installing Nagios

      Command line tools for system and process management

      top

      tail

      grep

      ps

      kill

      free

      du and df

      Summary

      6. Troubleshooting Performance and Reliability Issues

      System configuration

      The fielddata cache

      Analyzing queries

      Slow log

      Improving query performance

      High-cardinality fields

      Querying smaller indices

      Cold indices

      The shard query cache

      Script queries

      Testing meticulously

      System and data architecting

      Hot-Warm architecture

      Master nodes

      Hot nodes

      Warm nodes

      Reducing disk size

      Compression

      Storing the _source and analyzed fields

      Optimizing data ingestion

      Bulk indexing operations

      Drive configuration

      Case studies

      Node configuration

      Query optimization

      Web application performance

      Summary

      7. Node Failure and Post-Mortem Analysis

      Diagnosing problems

      OutOfMemoryError exceptions

      Shard failures

      Slow queries

      Resolving OutOfMemoryError exceptions

      Elasticsearch process crashes

      Disk space

      Resolving the issue

      Reviewing some case studies

      The ES process quits unexpectedly

      Query requests slow and timing out

      Summary

      8. Looking Forward

      Elasticsearch 5 overview

      Performance and reliability

      Data loss

      Upgrading to Elasticsearch 5.0

      When to upgrade

      Monitoring Elasticsearch 5

      Summary

      Index

      Monitoring Elasticsearch

      Monitoring Elasticsearch

      Copyright © 2016 Packt Publishing

      All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

      Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

      Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

      First published: July 2016

      Production reference: 1200716

      Published by Packt Publishing Ltd.

      Livery Place

      35 Livery Street

      Birmingham B3 2PB, UK.

      ISBN 978-1-78439-780-7

      www.packtpub.com

      Credits

      Author

      Dan Noble

      Reviewers

      James A. Cubeta

      Joseph E McMahon

      Parkhe Kishor B.

      Acquisition Editor

      Sonali Vernekar

      Content Development Editors

      Amey Varangaonkar

      Merint Mathew

      Technical Editor

      Hussain Kanchwala

      Copy Editor

      Priyanka Ravi

      Project Coordinator

      Judie Jose

      Proofreader

      Safis Editing

      Indexer

      Rekha Nair

      Graphics

      Kirk D'Penha

      Production Coordinator

      Melwyn Dsa

      Cover Work

      Melwyn Dsa

      About the Author

      Dan Noble is a software engineer with a passion for writing secure, clean, and articulate code. He enjoys working with a variety of programming languages and software frameworks, particularly Python, Elasticsearch, and frontend technologies. Dan currently works on geospatial web applications and data processing systems.

      Dan has been a user and advocate of Elasticsearch since 2011. He has given talks about Elasticsearch at various meetup groups, and is the author of the Python Elasticsearch client rawes. Dan was also a technical reviewer for the Elasticsearch Cookbook, Second Edition, by Alberto Paro.

      I would like to thank my beautiful wife, Julie, for putting up with me while I wrote this book. Thanks for supporting me every step of the way.

      I would also like to thank my friends and colleagues James Cubeta, Joe McMahon, and Mahmoud Lababidi, who shared their insight, time, and support. I would like to give a special thanks to Abe Usher – you have been an incredible mentor over the years.

      Finally, thanks to everyone at Packt Publishing for helping to make this book happen. A special thanks to Merint Mathew, Sonali Vernekar, Hussain Kanchwala, and Amey Varangaonkar for your valuable and careful feedback.

      About the Reviewers

      James A. Cubeta is a computer scientist with more than 20 years of experience, spanning government contracting, commercial companies, and research organizations. With a technical emphasis on data management, ETL, multi-tier application development, and relational and NoSQL databases, he has also helped author Java courses at Sun Microsystems and served as a technical reviewer for O'Reilly Media's extremely successful book Head First Java. He is currently a senior technologist at the HumanGeo Group, LLC.

      Joseph E McMahon is a computer scientist with over 25 years of experience developing distributed systems with a focus on effective and performant messaging components. He has vast experience integrating geospatial solutions and data collection architectures. Currently, he leads a division in The HumanGeo Group supporting a variety of government customers. When not coding, he enjoys cooking, woodworking, and traveling with his wife and three children.

      Parkhe Kishor B. received a BSc degree and an MTech degree in industrial mathematics and computer applications from the University of Pune. He joined High Mark Credit Information Service in 2012, where he worked as senior software engineer in the research and development department. He also has experience working in information retrieval, big data, and distributed computing. He is currently working as a design engineer at Introp Software Solutions, India. He is frequently a consultant to the industry in the area of big data, NoSQL, machine learning, artificial intelligence, and business intelligence.

      www.PacktPub.com

      eBooks, discount offers, and more

      Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

      At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

      https://www2.packtpub.com/books/subscription/packtlib

      Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.

      Why subscribe?

      Fully searchable across every book published by Packt

      Copy and paste, print, and bookmark content

      On demand and accessible via a web browser

      Preface

      Welcome to Monitoring Elasticsearch!

      There are many books and online tutorials that cover the Elasticsearch API and how to configure a cluster. But, until now, there hasn't been a thorough, accessible resource for monitoring and troubleshooting purposes. We've found that Elasticsearch monitoring tools drastically improve our ability to solve cluster issues and greatly increase cluster reliability and performance as a result. We wrote this book to share those use cases and the insights that came out of them.

      This book covers how to use several popular open source and commercial Elasticsearch monitoring tools, namely, Elasticsearch-head, Bigdesk, Marvel, Kopf, and Kibana. There's also a section on the Elasticsearch cat API and how to use Nagios to perform general system monitoring. Moreover, we will discuss several case studies with real-world examples of troubleshooting Elasticsearch issues using these tools.

      We believe that the best way to learn is to do. In this book, we'll go over how to set up a sample Elasticsearch cluster and load it with data. At times, we'll deliberately introduce problems into the cluster so that we can see how the errors are tracked using our various monitoring tools. Following along with these examples in your own cluster will help you learn both how to use the monitoring tools and how to tackle new and unknown issues that may arise.

      After reading this book, we hope that you will be better equipped to run and maintain an Elasticsearch cluster. You will also

      Enjoying the preview?
      Page 1 of 1